This document reviews requirements around testing interactions between Safe Places & the Pathcheck GPS mobile App, the various technical solutions available (either e2e, or tool-based), and some options for how we could improve things.
Apologies for length - I wanted to get broad coverage of all aspects of this, which has led to a rather long doc!
It does not draw any firm conclusions yet - I’m hoping to circulate and collect feedback first, particularly any use cases I have missed, or any technical options I have overlooked, or issues with the options I have proposed. Depending on level of async feedback, I may schedule a call to gather this input.
Once I have some more info on those aspects, then I’ll try to map out a proposed set of improvements that we should make here, and figure out a schedule for them.
Scope
Functionality - 2 key areas
Location Data upload function
Points of Concern download function
We are interested in cases where:
Focus is on testing the App
Focus is on testing Safe Places
Tester is interested in the whole end-to-end flow.
Specific use cases of interest:
Testing by a project volunteer (in test or Dev)
Testing by a crowdsourced engineer (e.g. Applause)
Demonstrations by PathCheck Implementation team - to prospects, for videos etc.
Testing by interested Health Department prospects, wanting to explore the solution
Functional Automated Tests (for App, SPL Backend or SPL Front End)
Possible Solutions
The high-level solutions break down as follows:
Test e2e - use a real SPL instance, and a real Mobile App
Mock out the App - may be useful when focus of interest is SPL
Mock out SPL - may be useful when focus of interest is the App.
What solutions do we have today? Let’s analyse each of the key flows…
Location Data upload
Approaches available.
e2e
Mock out the App using Postman (currently used for SPL Backend automated tests).
Dev FF to allow output of location data to email.
How do these stack up against our needs?
When testing the App
e2e - using Shared SPL instance
Quick & convenient - readily available: https://safeplaces.extremesolution.com/trace
Allows testing of full e2e flow
Lack of privacy - shared location data is available to all on the project
Data storage is unreliable (back end often blown away)
Does not provide access to raw data for analysis (e.g. checking reliability/accuracy of data output).
Output location data to email via FF
Convenient solution for getting data for analysis
Gives access to raw data, but not an e2e test
Only available on Dev builds with Feature Flags.
Whose needs do these meet / fail to meet?
Project volunteer (in test or Dev) - just about OK, but with reservations about privacy
Testing by a crowdsourced engineer (e.g. Applause) - Privacy is probably an issue - input from Applause is they don’t mind their testers having access to each others' data while they are testing, but unknown people having access to their testers data, or data being stored beyond the test session duration is a concern.
Testing by interested Health Department prospects, wanting to explore the solution
Privacy is an issue, both in terms of their data, and access they have to other data through the SPL login.
Lack of access to raw data for analysis is also an issue, particularly given our history of poor quality data here.
Demonstrations by PathCheck Implementation team - OK, but with reservations about privacy
Functional Automated Tests - Need not met. We want to have an easy way for test scripts to get a viable access code, and push data to an endpoint (and ideally validate that data).
Possible improvements: some combination of the following…
Multiple Safe Places deployments to create partitions to help with data privacy - e.g. dedicated Safe Places instance for Applause.
Safe Places deployment per-user. This would solve data privacy but cost may be too high.
Reduce privacy risk through ability to delete data sets in Safe Places
Encryption of data sets in Safe Places so that only the Contact Tracer who created the original record can read the data (See PLACES-358 for a related proposed security enhancement)
Some other mechanism so that Contact Tracers can only access the records they created.
Mock HTTPS server that can be used for testing sharing of location data, instead of Safe Places.
When testing Safe Places
e2e - using App
Minimal expertise needed
Requires availability of a device or emulator
Hard to drive programatically (e.g. for automation)
Limited flexibility on data sets (can only use data in the app; which takes time to build up and can only be in your actual location).
Mock out the App using Postman
Well suited for automation
No need to have a device (real or emulated)
High flexibility on data sets, but lack of supportive tooling means it’s time-consuming to build a particular data set.
Whose needs do these meet / fail to meet?
Project volunteer (in test or Dev) - OK for basic testing, but testing with large/custom data sets requires time & expertise.
Testing by a crowdsourced engineer (e.g. Applause) - OK for basic testing, but testing with large/custom data sets requires time & expertise.
Testing by interested Health Department prospects, wanting to explore the solution
OK for basic testing but testing with large/custom data sets requires unfeasible levels of time & expertise..
Demonstrations by PathCheck Implementation team - Difficult to demonstrate a “canned” data set.
Functional Automated Tests - OK (using Postman)
Possible improvements
Build on top of Postman to make it much more usable / acessible
Add ability to upload custom set of data points into the Mobile App (from where you can drive an import into Safe Places.
(both probably desirable to match all above use cases).
Points of Concern Download
Approaches available.
e2e
Mock out Safe Places using Infection Data Flask Server: https://diarmidmackenzie.pythonanywhere.com/infection-data
Mock out the App: Explicit checking of hashed data points using hash matching tools.
How do these stack up against our needs?
When testing the App
e2e - using Safe Places
Easy to use: Data points can be manually entered in Safe Places as required
Staging YAML file includes details of shared Safe Places cursor file
Other users data points are not really an issue - they will typically fail to generate exposure notifications and simply be ignored.
Can a shared Safe Places instance serve data in a wide range of geographic locations? (probably it can, subject to suitable YAML file boundaries - needs analysis)
Hard to generate large data sets (e.g. for scale testing)
Possible privacy concern: the points of concern you generate may betray your own location (since it is likely you are trying to generate exposure notifications).
Mock out Safe Places using Infection Data Flask Server
Works well for scale testing where you care primarily about volume of data.
OK to generate trails in a particular location
Full support for hashing & cursor files.
App support for custom cursor files is limited (work in progress with Feature Flag to set a custom YAML file; YAML file support recently added to Flask Server)
In theory could be viable for automated functional & scale tests - but unproven.
Not high-availability (no current means of maintaining operation while adding new function)
Limited control of data sets. Can control volume & location, but not more specific distribution of points (random distribution in a set area).
Whose needs do these meet / fail to meet?
Project volunteer (in test or Dev) - Good enough, if we can sort out issues with getting the App to download data from both Safe Places shared instance cursor file, and from Flask Server.
Testing by a crowdsourced engineer (e.g. Applause) - Solutions probably viable, but need better documentation. A dedicated Safe Places server instance for e.g. Applause might be desirable to address privacy & collision/contention issues.
Testing by interested Health Department prospects, wanting to explore the solution
HD tester having to download a whole load of points of concern from our internal testers is not ideal. Much preferable for HD to be able to just publish/download points of concern they want to test. But that would require a dedicated publishing endpoint for them.
Flask Server does not seem like a suitable solution for HDs since it depends on Feature Flags to set the custom YAML.
Demonstrations by PathCheck Implementation team
Shared Safe Places cursor file may be viable for demos, but involves risk because of shared usage. A dedicated instance would be safer (allowing fulll control of published data).
Flask Server may be suitable for demos, with some care needed on presentation. Unproven for this use case.
Functional Automated Tests
Safe Places server should not be introduced into Mobile App Test Automation - too much complexity.
Flask server could meet this requirement. May need some minor enhancements to meet requirements of specific test cases. We’d also need to address lack of High Availability.
Possible improvements
Clarify & document the various ways that YAML files can be used to direct test apps to particular cursor files
Dedicated Safe Places servers for external organizations like Applause, for each HD trialing the app, or even per-tester.
Add support for bulk data generation though Safe Places (tools layers in top of Front End or Back End?)
Enhancements to Flask Server to support Test Automation
High Availability and/or Availability coupled to test environment - i.e. run in test environment, not PythonAnywhere
More control in terms of what points of concern are generated (e.g. to hit specific functional TCs).
More analysis of demo requirements, and determine best technology fit, including possibly a dedictaed demo Safe Places instance.
When testing Safe Places
e2e - using App to pull & validate data
Difficult to actually validate data - only visibility is via exposure notifications, which provide minimal info (for privacy reasons) and depend on matching data being present on the app.
Ability to load a custom set of location points into the App in advance would help here (but we don’t have this yet)
Ability to control exposure_notification_timeframe (reducing to 5 mins) can help make exposure notification more sensitive in matching specific locations. (but this is not available in Safe Places UI yet)
Mock out the App and check the hashes using manual hash checking tools.
Hashes cannot be decoded - you must know what the hash should be, in order to check it (not really a flaw in our test tools - rather it’s a characteristic of the solution).
Requires significant expertise to manipulate the hash checking tools. We could provide better documentation here, and maybe even make available as a web-based service
Whose needs do these meet / fail to meet?
Project volunteer (in test or Dev) - Not really good enough to properly validate Safe Places data. We can spot gross errors, but spotting subtle / one-off errors needs more tooling than we have.
Testing by a crowdsourced engineer (e.g. Applause) - Similar to project volunteers. Very basic testing is OK, but more tooling desirable to support deeper testing.
Testing by interested Health Department prospects, wanting to explore the solution
If an HD has their own Safe Places solution, using the App alongside to validate published data should be adequate for most purposes.
Demonstrations by PathCheck Implementation team
Presentation of a single exposure notification is adequate for Demo purposes.
Functional Automated Tests
These can use known specific locations/times are simply check against known hash values, or re-compute in the scripts. Such approach already established in Mobile App Unit Tests.
Possible improvements
Develop a web-based service that allows generation of hashes for a data point, or set of data points, for use by testers wanting to validate hash files.
Add support for loading a custom set of location points into the app, for the purpose of testing intersection.
Also tooling to develop these data sets to match a given target set of points of concern.
Feature Flag to enable some sort of intersection debugging mode, which shows exactly which location points match (or did not match) which points of concern.
Add control for exposure_notification_timeframe into Safe Places Publishing UI.
But if testers are to use this, we’d need to consider how that will play with a shared Safe Places Publisher… One tester could mess up other testers testing! Maybe could only be used when testing with a dedicated Safe Places instance?