Context - Haiti
Haiti is a small country on the Carribbean island of Hispaniola. It has a population of 11 million, and an area of 28,000 sq km - about the same area as Massachussetts, or Belgium.
The capital is Port-au-Prince, with a population of 1.2M in the city itself, and about 2.8M in the metro area around it.
Cap-Haitien (534k) and Port-de-Paix (250k) are the only large cities outside the Port-au-Prince metro area.
Main languages are Haitian Creole and French.
As of 1 May, there have been 85 recorded cases of COVID-19 and 8 deaths. The first cases were recorded 6 weeks ago, on March 20, and growth has been roughly linear since then. 848 people have been tested for COVID-19.
Social distancing measures are in place, most people wear face masks when in public, and there is a curfew from 8pm to 5am, but industries are still operating at reduced capacity, and there are no restrictions on movements outside curfew hours.
Rollout Plan & anticipated scale
The Haitian Health Authorities are keen to roll out the Safe Paths solution as soon as possible. There are some blockign issues relating to them being able to push the App to the Play store under their own account, but we expect these to be resolved within a few days.
Once the app is available, we expect Haiti to move forward with implementation straight away.
We can’t predict how fast the app will roll out, but the fastest likely growth would be 12 cases/week logged in Safe Places. This assumes all cases have smartphones & use the app (both unlikely), but it also assumes no sudden surge in case rates (either due to increased viral spread, or increased testing).
We assume historical cases will not be netered into Safe Places. Even if they are, the historical cases for the last 2 weeks would only amount to about 24.
Since new cases will age out after 2 or 3 weeks, this suggests that the total number of cases tracked by the HA at any point in time should be no more than 40, unless there is a sudden surge in cases, combined with high adoption of the App.
What are we trying to learn from a Haiti Field Trial?
From a field trial, we hope to learn about the following:
Any issues with localization of the app: either with language, or with local information provided by the Haitian Health Authorities.
Possible issues with GPS detection / accuracy in the Haitian environment
Any particular movement patterns in Haiti (e.g. public transpor) that might not interact well with our algorithms for exposure detection
(non-Haiti-specific) General information about how it is to use the App in a real-world situation.
Overall approach
The proposed approach is as follows:
We have 5 or 6 testers in Haiti
We give them a series of missions, which involve movements, and interactions with ther other testers. For example, multiple testers travel on the same bus together, or meet together in a common location. (it will be OK for them to social distance with a 2m gap, because GPS is not that sensitive)
We collect all the location data from their phones, as recorded by the App.
We then run a series of analyses of it, to determine which contacts our algorithms would have detected, and which we missed.
To cover the UX angle, we also create a series of mock HA servers with selected subsets of the data published (and all the rest of the config, e.g. news pages, matching the Haitian HA). We ask the testers to enter each of these HA servers by URL in turn, and give us feedback on their experience.
Note that we will aim to use Safe Places in processing and generating the HA data sets. We may ask to run this as an “interview” with the tester to better test the flow - but we can only do that if they speak English.
If we are quick, it may be possible to roll out fixes to the algorithms, and repeat this UX test with the same data, but an updated software load, to get feedback on whether the behaviour is improved.
Key questons with this approach…
Can we get hold of 5-6 testers in Haiti? Applause think so, but we need to confirm
Will they be willing to travel around Haiti, given the risks involved at the moment? We need to discuss.
Are we comfortable asking them to do so? Needs internal discussion.
Are the testers close enough to each other to be able to cross paths as demanded by the plan? Needs discussion with Applause.
Will the testers be willing to share their location histories with us? Needs discussion with Applause.
Will the testers speak English & be willing to conduct a mock Redaction interview with us?
Most of this work isn’t really speciaist software testing work. Do we need to go through Applause for the testers, or should we try another channel? Applause is a siple & appealing option, and we don’t obviously have other channels available. Let’s use Applause as 1st choice, and raise this question if they can’t deliver.
Rationale behind this approach
Using real people on the ground in Haiti. While we could simulate movements on the groun in Haiti using the Perfecto environment, we feel this would risk valuable insights on all of points 1-4 above, vs using people on the ground in Haiti.
Using fake HA data, rather than the real production data. We want the field trial to be “ahead” of the production roll-out in terms of number of reported cases, so that our testers have a reasonable chance of hitting false positive, false negative and true position scenarios.
Using real data from teh App as this test data. We could generate synthetic data artificially, either by hand (plotting some routed through Haiti on a map), or using ML based on soem real Haitian Google Takeout data. However, the ideal case would be that we publish data generated by our own App, running on real phones in Haiti, as this is where the production data will be sourced from. If we use data from other sources, we risk missing subtle issues (e.g. with out own App’s GPS detection differing from Google Takeout).
Pre-requisites
We are under time pressure here, as we want to do this testing before production ramps to significant levels of cases. However there is no point executing this testing if there are known issues in the Apps that will obviously interfere with the chaces of success.
Therefore we need some fixes before we go ahead with this:
GPS data reliability. The app must record data reliably, and not lock up & stop recording. In the case that there are some known scenarios (e.g. battery saving mode) that interfere with GPS recording, it will be OK to brief the testers on workarounds, e.g. having to open the app every few hours to.
GPS data accuracy. We must have fixed the app’s current issues with GPS accuracy, and got to the point where we believe the accuracy is good enough.
OS version support. We must make sure we support the necessary Android & iOS versions, bearing in mind that many in Haiti may have older models of phone.
We should have the ability to roll out fixes, without loss of recorded location data.
Non-dependencies: a couple of other important app issues, that we don’t need to fix ahead of field trial
Scalability of data sets. RIght now we can only support ~50 cases (with a full 14 day history). That won’t actually be a constraint for this field trial.
Appropriateness of algorithms. We are concerned that our algorithms don’t work well in scenarios involving movement (e.g. public transport). This need not be a big deal, as we can modify algorithms, roll out fixes & repeat tests, without having to recapture the location data again.
Next Steps
1) Review internally:
Do we believe this plan will give us valuable additional information ahead of the Haiti launch? It will take some effort, and involve some risk on the part of our testers - so is it worth it?
When can we deliver the necessary pre-requisites on the App? For discussion with PM / Dev.
Who will run this effort from Path Check? We don’t have a good volunteer within the Testign team - but there isn’t a lot of specialist testing expertise needed here, so we may be able to find someone else to run with this.
Do we want to try to cover both iOS & Android? Or shall we just focus on Android?
2) Discuss with Applause
Can they provide the testers?
Can they do what we need them to do?
Can they work to our timescales?
3) Build a detailed plan & execute!