Testability of Safe Places - PathCheck GPS Mobile App interactions

Owned by Diarmid Mackenzie
Last updated: Jun 24, 2020
9 min read

This document reviews requirements around testing interactions between Safe Places & the Pathcheck GPS mobile App, the various technical solutions available (either e2e, or tool-based), and some options for how we could improve things.

Apologies for length - I wanted to get broad coverage of all aspects of this, which has led to a rather long doc!

It does not draw any firm conclusions yet - I’m hoping to circulate and collect feedback first, particularly any use cases I have missed, or any technical options I have overlooked, or issues with the options I have proposed. Depending on level of async feedback, I may schedule a call to gather this input.

Once I have some more info on those aspects, then I’ll try to map out a proposed set of improvements that we should make here, and figure out a schedule for them.

Scope

Functionality - 2 key areas

  • Location Data upload function

  • Points of Concern download function

We are interested in cases where:

  • Focus is on testing the App

  • Focus is on testing Safe Places

  • Tester is interested in the whole end-to-end flow for a single user

  • Tester is interested in the whole end-to-end flow including aggregating data from multiple users.

Specific use cases of interest:

  • Testing by a project volunteer (in test or Dev)

  • Testing by a crowdsourced engineer (e.g. Applause)

  • Testing by an App Store, trying to decide whether to accept our App

  • Demonstrations by PathCheck Implementation team - to prospects, for videos etc.

  • Testing by interested Health Department prospects, wanting to explore the solution

  • Functional Automated Tests (for App, SPL Backend or SPL Front End)

Possible Solutions

The high-level solutions break down as follows:

  • Test e2e - use a real SPL instance, and one or more real Mobile Apps.

  • Mock out the App (or Apps) - may be useful when focus of interest is SPL

  • Mock out SPL - may be useful when focus of interest is the App.

What solutions do we have today? Let’s analyse each of the key flows…

Location Data upload

Approaches available.

  • e2e

  • Mock out the App using Postman (currently used for SPL Backend automated tests).

  • Dev FF to allow output of location data to email.

How do these stack up against our needs?

When testing the App

e2e - using Shared SPL instance

  • Quick & convenient - readily available: https://safeplaces.extremesolution.com/trace

  • Allows testing of full e2e flow

  • Lack of privacy - shared location data is available to all on the project

  • Data storage is unreliable (back end often blown away)

  • Does not provide access to raw data for analysis (e.g. checking reliability/accuracy of data output).

Output location data to email via FF

  • Convenient solution for getting data for analysis

  • Gives access to raw data, but not an e2e test

  • Only available on Dev builds with Feature Flags.

Whose needs do these meet / fail to meet?

  • Project volunteer (in test or Dev) - just about OK, but with reservations about privacy

  • Testing by a crowdsourced engineer (e.g. Applause) - Privacy is probably an issue - input from Applause is they don’t mind their testers having access to each others' data while they are testing, but unknown people having access to their testers data, or data being stored beyond the test session duration is a concern.

  • Testing by an App Store, trying to decide whether to accept our App

    • Having to navigate the Safe Places UI is probably not something an App Store reviewer wants to take on. The ideal would be being able to be pre-briefed on a viable access code for data sharing, which will work for them, without interaction with any system, or person.

  • Testing by interested Health Department prospects, wanting to explore the solution

    • Privacy is an issue, both in terms of their data, and access they have to other data through the SPL login.

    • Lack of access to raw data for analysis is also an issue, particularly given our history of poor quality data here.

  • Demonstrations by PathCheck Implementation team - OK, but with reservations about privacy

  • Functional Automated Tests - Need not met. We want to have an easy way for test scripts to get a viable access code, and push data to an endpoint (and ideally validate that data).

Possible improvements: some combination of the following…

  • Multiple Safe Places deployments to create partitions to help with data privacy - e.g. dedicated Safe Places instance for Applause.

  • Safe Places deployment per-tester. This would solve data privacy but cost may be too high.

  • Reduce privacy risk through ability to delete data sets in Safe Places

  • Encryption of data sets in Safe Places so that only the Contact Tracer who created the original record can read the data (See PLACES-358 for a related proposed security enhancement)

  • Some other mechanism so that Contact Tracers can only access the records they created.

  • Mock HTTPS server that can be used for testing sharing of location data, instead of Safe Places. This could be particularly useful to accommodate the App Store tester case.

When testing Safe Places

e2e - using App

  • Minimal expertise needed

  • Requires availability of a device or emulator

  • Hard to drive programatically (e.g. for automation)

  • Limited flexibility on data sets (can only use data in the app; which takes time to build up and can only be in your actual location).

Mock out the App using Postman

  • Well suited for automation

  • No need to have a device (real or emulated)

  • High flexibility on data sets, but lack of supportive tooling means it’s time-consuming to build a particular data set.

Whose needs do these meet / fail to meet?

  • Project volunteer (in test or Dev) - OK for basic testing, but testing with large/custom data sets requires time & expertise.

  • Testing by a crowdsourced engineer (e.g. Applause) - OK for basic testing, but testing with large/custom data sets requires time & expertise.

  • Testing by interested Health Department prospects, wanting to explore the solution

    • OK for basic testing but testing with large/custom data sets requires unfeasible levels of time & expertise..

  • Demonstrations by PathCheck Implementation team - Difficult to demonstrate a “canned” data set.

  • Functional Automated Tests - OK (using Postman)

Possible improvements

  • Build on top of Postman to make it much more usable / acessible

  • Add (Dev-only, FF controlled) ability to upload custom set of data points into the Mobile App (from where you can drive an import into Safe Places).

  • (both probably desirable to match all above use cases).

Points of Concern Download

Approaches available.

  • e2e

  • Mock out Safe Places using Infection Data Flask Server: https://diarmidmackenzie.pythonanywhere.com/infection-data

  • Mock out the App: Explicit checking of hashed data points using hash matching tools.

How do these stack up against our needs?

When testing the App

e2e - using Safe Places

  • Easy to use: Data points can be manually entered in Safe Places as required

  • Staging YAML file includes details of shared Safe Places cursor file

  • Other users data points are not really an issue - they will typically fail to generate exposure notifications and simply be ignored.

  • Can a shared Safe Places instance serve data in a wide range of geographic locations? (probably it can, subject to suitable YAML file boundaries - needs analysis)

  • Hard to generate large data sets (e.g. for scale testing)

  • Possible privacy concern: the points of concern you generate may betray your own location (since it is likely you are trying to generate exposure notifications).

Mock out Safe Places using Infection Data Flask Server

  • Works well for scale testing where you care primarily about volume of data.

  • OK to generate trails in a particular location

  • Full support for hashing & cursor files.

  • App support for custom cursor files is limited (work in progress with Feature Flag to set a custom YAML file; YAML file support recently added to Flask Server)

  • In theory could be viable for automated functional & scale tests - but unproven.

  • Not high-availability (no current means of maintaining operation while adding new function)

  • Limited control of data sets. Can control volume & location, but not more specific distribution of points (random distribution in a set area).

Whose needs do these meet / fail to meet?

  • Project volunteer (in test or Dev) - Good enough, if we can sort out issues with getting the App to download data from both Safe Places shared instance cursor file, and from Flask Server.

  • Testing by a crowdsourced engineer (e.g. Applause) - Solutions probably viable, but need better documentation. A dedicated Safe Places server instance for e.g. Applause might be desirable to address privacy & collision/contention issues.

  • Testing by an App Store, trying to decide whether to accept our App

    • Our hope is that being able to select an HD is enough for this tester - they won’t need to explicitly generate exposure notifications. If they did, we’d have a great difficulty….

    • Flask Server does not seem like a suitable solution for App Store testers since it depends on Feature Flags to set the custom YAML.

    • Data download from a HD will have to come from one of the production HDs since those are all that will be available - very unlikely that we could generate an exposure notification from this, as the locations don’t match (and even if we had an HD covering Apple HQ, we’d not know where the points of concern were!).

  • Testing by interested Health Department prospects, wanting to explore the solution

    • HD tester having to download a whole load of points of concern from our internal testers is not ideal. Much preferable for HD to be able to just publish/download points of concern they want to test. But that would require a dedicated publishing endpoint for them.

    • Flask Server does not seem like a suitable solution for HDs since it depends on Feature Flags to set the custom YAML.

  • Demonstrations by PathCheck Implementation team

    • Shared Safe Places cursor file may be viable for demos, but involves risk because of shared usage. A dedicated instance would be safer (allowing fulll control of published data).

    • Flask Server may be suitable for demos, with some care needed on presentation. Unproven for this use case.

  • Functional Automated Tests

    • Safe Places server should not be introduced into Mobile App Test Automation - too much complexity.

    • Flask server could meet this requirement. May need some minor enhancements to meet requirements of specific test cases. We’d also need to address lack of High Availability.

Possible improvements

  • Clarify & document the various ways that YAML files can be used to direct test apps to particular cursor files

  • Dedicated Safe Places servers for external organizations like Applause, for each HD trialing the app, or even per-tester.

  • Add support for bulk data generation though Safe Places (tools layers in top of Front End or Back End?)

  • Enhancements to Flask Server to support Test Automation

    • High Availability and/or Availability coupled to test environment - i.e. run in test environment, not PythonAnywhere

    • More control in terms of what points of concern are generated (e.g. to hit specific functional TCs).

  • More analysis of demo requirements, and determine best technology fit, including possibly a dedictaed demo Safe Places instance.

When testing Safe Places

  • e2e - using App to pull & validate data

    • Difficult to actually validate data - only visibility is via exposure notifications, which provide minimal info (for privacy reasons) and depend on matching data being present on the app.

    • Ability to load a custom set of location points into the App in advance would help here (but we don’t have this yet)

    • Ability to control exposure_notification_timeframe (reducing to 5 mins) can help make exposure notification more sensitive in matching specific locations. (but this is not available in Safe Places UI yet)

  • Mock out the App and check the hashes using manual hash checking tools.

    • Hashes cannot be decoded - you must know what the hash should be, in order to check it (not really a flaw in our test tools - rather it’s a characteristic of the solution).

    • Requires significant expertise to manipulate the hash checking tools. We could provide better documentation here, and maybe even make available as a web-based service

Whose needs do these meet / fail to meet?

  • Project volunteer (in test or Dev) - Not really good enough to properly validate Safe Places data. We can spot gross errors, but spotting subtle / one-off errors needs more tooling than we have.

  • Testing by a crowdsourced engineer (e.g. Applause) - Similar to project volunteers. Very basic testing is OK, but more tooling desirable to support deeper testing.

  • Testing by interested Health Department prospects, wanting to explore the solution

    • If an HD has their own Safe Places solution, using the App alongside to validate published data should be adequate for most purposes.

  • Demonstrations by PathCheck Implementation team

    • Presentation of a single exposure notification is adequate for Demo purposes.

  • Functional Automated Tests

    • These can use known specific locations/times are simply check against known hash values, or re-compute in the scripts. Such approach already established in Mobile App Unit Tests.

Possible improvements

  • Develop a web-based service that allows generation of hashes for a data point, or set of data points, for use by testers wanting to validate hash files.

  • Add support for loading a custom set of location points into the app, for the purpose of testing intersection.

    • Also tooling to develop these data sets to match a given target set of points of concern.

  • Feature Flag to enable some sort of intersection debugging mode, which shows exactly which location points match (or did not match) which points of concern.

  • Add control for exposure_notification_timeframe into Safe Places Publishing UI.

    • But if testers are to use this, we’d need to consider how that will play with a shared Safe Places Publisher… One tester could mess up other testers testing! Maybe could only be used when testing with a dedicated Safe Places instance?