GAEN Test Strategy & Plan

This is our test strategy for the GAEN solution, which consists of a Mobile App (iOS and Android), and a set of servers, which can be delivered as a SaaS offering for HDs, or deployed in their own chosen infrastructure.

Later sections of this article detail the specifics of our current plans & status.

Project Scope

The project scope is to deliver an implementation of the Google / Apple Exposure Notifications solution for deployment by Health Departments.

Deficincies in the Google / Apple EN solution itself, whether due to bugs in Google / Apple code, or more fundamental limitations with BT / RF technology are of significant interest to PathCheck in general, but do not necessarily impact our ability to deliver against the above goal.

As a whole, the PathCheck organization is very keen to understand the nature of deficiencies that exist in GAEN, and to figure out how they can be overcome in order to build better DCT solutions, that investigation is not a part of the scope of this project.

As a result of this scope limitation, the following testing areas are not considered priorities for the GAEN implementation project (we hope that this testing will be picked up by other PathCheck initiatives):

• Bluetooth TC4TL accuracy: Correctness of Exposure Notifications vs. relative positions of phones

• Non-interferences of other BT applications on the phone (e.g. BT headset)

• Non-interference of other BT / RF applications in theenvironment, including large numbers of App users.

• Battery consumption (which we expect to be dominated by the BLE function, which is a Google / Apple responsibility)

• Bluetooth Interoperability between a wide range of iOS & Android devices.

These aspects of the product do all matter, but testing in these areas is deprioritized on the basis that:

• These are Apple/Googles responsibilities to get right, not ours.

• Our implementation willl be no better, and no worse, in these regards than any other GAEN solution

• If we do find problems in these areas, we doubt we will have very much success in driving Google / Apple to make things better in any short timescale.

Impediments

There are some significant impediments to testing GAEN, which have a signficiant influence on our approach

• The mobile app for test must be built and distributed using the Apple Developer account of an HD with GAEN entitlements

• Until we get Apple approval for an open Beta (still a work in progress), this limits us to a total of 25 testers per HD (including any HD staff who want to test)

• Testing of customizations will need to be done in the context of HD accounts. This may prove difficult if we want to test customizations that are not of direct interest to that HD: building good relationships so that these partners are willing to support us is key.

• On both iOS and Android, we are very limited in terms of our ability to control which Bluetooth Random IDs get stored on the phone. There is no way to populate these programmatically - the only way to do so is to have a real Bluetooth device nearby, broadcasting keys. This makes it very hard to set up a phone with a realistic set of Bluetooth Random IDs that a phone might pick up over a 14 day period.

• Testing Exposure Notifications e2e can be very slow because it can take a full day for the necessary client-server interactions to get an EN to a phone, after a set of infected keys are shared. The fact that all this function is handled opaquely by the OS means that there is no easy way around this. This means that test cases take a very long (and unpredictable) elapsed time to complete.
  
  

Testing Priorities

At the start of testing, we went through a process with Bob Mallon, where we

• Identified all the gaps we might need to fill to take the solution from “pilot-ready” to “community-ready”

• Discussed and agreed priorities, where

  • High = needed for launch

  • Medium = needed 6-8 weeks after launch

  • Low = some stage beyond that.

The output of that process can be found here:

https://docs.google.com/spreadsheets/d/1wHepFi2Km38I1mCgJo56as8V6Ubn0sA2CdUeBncHGgU/edit#gid=0

Going forward, we have moved that content into Jira, where it is now maintained, using the same High/Medium/Low priorities.

• See Jira board: https://pathcheck.atlassian.net/jira/software/projects/GAEN/boards/33/backlog

Note that this board tracks testing activities only. Development activities are still tracked in Trello.

Resourcing Plan

Resourcing testing for the GAEN project has been a challenge due to:

• The fact that testers need access to multiple physical devices to test ENs, combined with the fact that initially the app was available on iOS (with the app now available on Android, this last point is now moot).

• General shortage of testing volunteers at PathCheck.

To help with the resourcing, we have had 2 full-time testers + a manager assigned to the project from Winjit in India

• They have access to a good range of physical devices

• They are able to take on test planning & test execution

• They will send test plans to us for review, before they begin execution

• They have access to Minnesota TF (as of 29/7), and are working on validating the basic mainline EN flow.

They also have access to specialist testers in:

• Accessibility

• UX

• Scale / load testing

What WinJit can’t do…

• Security & Privacy testing (other than some basics) → TBD. We don’t yet have a plan here.

• Language testing → Applause, probably. But concerns about TF access.

With a lot of pressure on elapsed time & the winding down of GPS testing, we hope to supplement the Winjit team with some testing work from other PathCheck volunteers, particular in test areas that do not requre EN generation (and therefore don’t require the use of multiple physical devices; which most testers don’t have access to).

Current Status and Schedule

This table shows the status of the high priority items (see Jira links above), and current view of these (as of 4 Aug).

For the Winjit team, we have the following planned schedule of activities, which we will maintain and revise as we move forwards.

https://docs.google.com/spreadsheets/d/14T3jJDibuIOYob0HMcOCYpLM-HUo_ED4_9F7c3rBltA/edit?usp=sharing

Tracking & Bug Reporting

Process documented here (still undergoing revisions & improvements)

Bug tracking for GAEN

Key open questions & risks

• Who in PathCheck is worrying about general GAEN/BT deficiencies. Can we get a research project kicked off on this?

• Can we develop tools that can simulate many phones being near another phone over Bluetooth (are there tools from Trinity College Dublin we can re-use here)?

• Can we find a way to reduce the elapsed time required for an EN test (currently ~a day due to client-server GAEN protocol delays)?

• Privacy & Security - need clear overall plan

• Customization: what’s the plan for MN & can we deliver? What’s the plan for “productized” customization for the next wave of deployments after MN?

• Analytics: what are we doing & can we deliver?

• What’s the schedule for languages? Presumably depends on copy changes? And gets scuppered by HDs modifying text - do we have a plan for that?

• What’s coming fast on the heels of MN that we need to watch out for?

• Security: how can we get BugCrowd engaged with GAEN?

• What is the risk of Accessibility and/or Security becoming hard blockers at MN or other HAs?

• What are our requirements on HD tuning of controls of risk parameters?