Page Comparison

This is our test strategy for the GAEN solution, which consists of a Mobile App (iOS and Android), and a set of servers, which can be delivered as a SaaS offering for HDs, or deployed in their own chosen infrastructure.

Later sections of this article detail the specifics of our current plans & status.

Project Scope

The project scope is to deliver an implementation of the Google / Apple Exposure Notifications solution for deployment by Health Departments.

...

• These are Apple/Googles responsibilities to get right, not ours.

• Our implementation willl be no better, and no worse, in these regards than any other GAEN solution

• If we do find problems in these areas, we doubt we will have very much success in driving Google / Apple to make things better in any short timescale.

Impediments

There are some significant impediments to testing GAEN, which have a signficiant influence on our approach

• The mobile app for test must be built and distributed using the Apple Developer account of an HD with GAEN entitlements

• Until we get Apple approval for an open Beta (still a work in progress), this limits us to a total of 25 testers per HD (including any HD staff who want to test)

• Testing of customizations will need to be done in the context of HD accounts. This may prove difficult if we want to test customizations that are not of direct interest to that HD: building good relationships so that these partners are willing to support us is key.

• The Android App is significantly behind iOS App (but coming soon!)
  

Testing Priorities

We have identified the following as the top priorities for testing

• OS testing: basic testing across all supported iOS & Android versions

• …

• See Jira board: https://pathcheck.atlassian.net/jira/software/projects/GAEN/boards/33/backlog

• Goal is for high priority items to be tested before launch.

Testing Non-Priorities

...

• On both iOS and Android, we are very limited in terms of our ability to control which Bluetooth Random IDs get stored on the phone. There is no way to populate these programmatically - the only way to do so is to have a real Bluetooth device nearby, broadcasting keys. This makes it very hard to set up a phone with a realistic set of Bluetooth Random IDs that a phone might pick up over a 14 day period.

• Testing Exposure Notifications e2e can be very slow because it can take a full day for the necessary client-server interactions to get an EN to a phone, after a set of infected keys are shared. The fact that all this function is handled opaquely by the OS means that there is no easy way around this. This means that test cases take a very long (and unpredictable) elapsed time to complete.
  

Testing Priorities

At the start of testing, we went through a process with Bob Mallon, where we

• Identified all the gaps we might need to fill to take the solution from “pilot-ready” to “community-ready”

• Discussed and agreed priorities, where

  • High = needed for launch

  • Medium = needed 6-8 weeks after launch

  • Low = some stage beyond that.

The output of that process can be found here:

https://docs.google.com/spreadsheets/d/1wHepFi2Km38I1mCgJo56as8V6Ubn0sA2CdUeBncHGgU/edit#gid=0

Going forward, we have moved that content into Jira, where it is now maintained, using the same High/Medium/Low priorities.

• See Jira board: https://pathcheck.atlassian.net/jira/software/projects/GAEN/boards/33/backlog

• Medium & Low priority items

• Goal is for medium items to be tested ~6-8 weeks after launch

Resourcing Plan

We Note that this board tracks testing activities only. Development activities are still tracked in Trello.

Resourcing Plan

Resourcing testing for the GAEN project has been a challenge due to:

• The fact that testers need access to multiple physical devices to test ENs, combined with the fact that initially the app was available on iOS (with the app now available on Android, this last point is now moot).

• General shortage of testing volunteers at PathCheck.

To help with the resourcing, we have had 2 full-time testers + a manager assigned to the project from Winjit in India

...

• Security & Privacy testing (other than some basics) → TBD. We don’t yet have a plan here.

• Language testing → Applause, probably. But concerns about TF access.

Expected Schedule

Work in progress - aim to clarify with Winjit on Tuesday

• Concern about load testing - sounds like we are ready and it would be valuable to start asap

  • Sherif thinks this is important testing and has not yet been done by anyone (not Google either).

  • Not sure how soon Winjit can work on this

  • Concern about attenuation of comms.

Tracking & Bug Reporting

• Dev team uses Jira

• Test team activities tracked in Jira

• Bugs to be tracked in Jira

• Processes for Beta testers / HDs to raise issues still TBD

• Processes for Support / ZenDesk to raise issues still TBD

...

With a lot of pressure on elapsed time & the winding down of GPS testing, we hope to supplement the Winjit team with some testing work from other PathCheck volunteers, particular in test areas that do not requre EN generation (and therefore don’t require the use of multiple physical devices; which most testers don’t have access to).

Current Status and Schedule

This table shows the status of the high priority items (see Jira links above), and current view of these (as of 4 Aug).

For the Winjit team, we have the following planned schedule of activities, which we will maintain and revise as we move forwards.

https://docs.google.com/spreadsheets/d/14T3jJDibuIOYob0HMcOCYpLM-HUo_ED4_9F7c3rBltA/edit?usp=sharing

Tracking & Bug Reporting

Process documented here (still undergoing revisions & improvements)

Bug tracking for GAEN

Key open questions & risks

• Who in PathCheck is worrying about general GAEN/BT deficiencies. Can we get a research project kicked off on this?

• Can we develop tools that can simulate many phones being near another phone over Bluetooth (are there tools from Trinity College Dublin we can re-use here)?

• Can we find a way to reduce the elapsed time required for an EN test (currently ~a day due to client-server GAEN protocol delays)?

• Privacy & Security - need clear overall plan

• Customization: what’s the plan for MN & can we deliver? What’s the plan for “productized” customization for the next wave of deployments after MN?

• Analytics: what are we doing & can we deliver?

• What’s the schedule for languages? Presumably depends on copy changes? And gets scuppered by HDs modifying text - do we have a plan for that?

• What’s coming fast on the heels of MN that we need to watch out for?

• Security: how can we get BugCrowd engaged with GAEN?

• What is the risk of Accessibility and/or Security becoming hard blockers at MN or other HAs?

• What are our requirements on HD tuning of controls of risk parameters?