Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

This is our test strategy for the GAEN solution, which consists of a Mobile App (iOS and Android), and a set of servers, which can be delivered as a SaaS offering for HDs, or deployed in their own chosen infrastructure.

Project Scope

The project scope is to deliver an implementation of the Google / Apple Exposure Notifications solution for deployment by Health Departments.

Deficincies in the Google / Apple EN solution itself, whether due to bugs in Google / Apple code, or more fundamental limitations with BT / RF technology are of significant interest to PathCheck in general, but do not necessarily impact our ability to deliver against the above goal.

As a whole, the PathCheck organization is very keen to understand the nature of deficiencies that exist in GAEN, and to figure out how they can be overcome in order to build better DCT solutions, that investigation is not a part of the scope of this project.

As a result of this scope limitation, the following testing areas are not considered priorities for the GAEN implementation project (we hope that this testing will be picked up by other PathCheck initiatives):

  • Bluetooth TC4TL accuracy: Correctness of Exposure Notifications vs. relative positions of phones

  • Non-interferences of other BT applications on the phone (e.g. BT headset)

  • Non-interference of other BT / RF applications in theenvironment, including large numbers of App users.

  • Battery consumption (which we expect to be dominated by the BLE function, which is a Google / Apple responsibility)

  • Bluetooth Interoperability between a wide range of iOS & Android devices.

These aspects of the product do all matter, but testing in these areas is deprioritized on the basis that:

  • These are Apple/Googles responsibilities to get right, not ours.

  • Our implementation willl be no better, and no worse, in these regards than any other GAEN solution

  • If we do find problems in these areas, we doubt we will have very much success in driving Google / Apple to make things better in any short timescale.

Impediments

There are some significant impediments to testing GAEN, which have a signficiant influence on our approach

  • The mobile app for test must be built and distributed using the Apple Developer account of an HD with GAEN entitlements

  • Until we get Apple approval for an open Beta (still a work in progress), this limits us to a total of 25 testers per HD (including any HD staff who want to test)

  • Testing of customizations will need to be done in the context of HD accounts. This may prove difficult if we want to test customizations that are not of direct interest to that HD: building good relationships so that these partners are willing to support us is key.

  • The Android App is significantly behind iOS App (but coming soon!)

Testing Priorities

We have identified the following as the top priorities for testing

Testing Non-Priorities

See also: https://pathcheck.atlassian.net/jira/software/projects/GAEN/boards/33/backlog

  • Medium & Low priority items

  • Goal is for medium items to be tested ~6-8 weeks after launch

Resourcing Plan

We have had 2 full-time testers + a manager assigned to the project from Winjit in India

  • They have access to a good range of physical devices

  • They are able to take on test planning & test execution

  • They will send test plans to us for review, before they begin execution

  • They have access to Minnesota TF (as of 29/7), and are working on validating the basic mainline EN flow.

They also have access to specialist testers in:

  • Accessibility

  • UX

  • Scale / load testing

What WinJit can’t do…

  • Security & Privacy testing (other than some basics) → TBD. We don’t yet have a plan here.

  • Language testing → Applause, probably. But concerns about TF access.

Expected Schedule

Work in progress - aim to clarify with Winjit on Tuesday

  • Concern about load testing - sounds like we are ready and it would be valuable to start asap

    • Sherif thinks this is important testing and has not yet been done by anyone (not Google either).

    • Not sure how soon Winjit can work on this

    • Concern about attenuation of comms.

Tracking & Bug Reporting

  • Dev team uses Jira

  • Test team activities tracked in Jira

  • Bugs to be tracked in Jira

  • Processes for Beta testers / HDs to raise issues still TBD

  • Processes for Support / ZenDesk to raise issues still TBD

Key open questions

  • No labels