Draft version - Release Candidate is not even available yet, and we have not yet begun testing ###
We are under a lot of pressure to deliver an App via the App Store to one of our prospects.
They are in a country outside the US, so we had considered just releasing to the App Store in that country. However that would entail additional delays, and therefore the current plan is to publish an initial release globally on the App Store.
This document assesses the latest App build (v0.9.6 ??) against suitability for this purpose, and highlights the key risks & quality issues that we have been able to identify. This draws on a combination of testing performed on the latest build,and previous project knowledge about the state of the App.
Expected Usage
The key uses we anticipate for the App are:
Evaluation & testing by one prospect in a small country outside the US.
Use in marketing demos with other prospects
Download and use by people across the world who have heard of the project, and want to begin storing their location data on their phone, in case they are later diagnosed with COVID-19
Other people who are interested in the project, which may include goverment researchers, other COVID-19 projects, Hackers, journalists etc. etc.
Note that usages 3 & 4 are usages that we are exposed to by the decision to release the app globally. If we limited release to a single small country, our exposure to these usages would be massively reduced.
Key issues and concerns
Security, Privacy & misinformation.
We know we have issues with Security & Privacy vs. where we want to get to - e.g. data stored unencrypted on users' phones. This is a concern given that the app may become widely used to store personal data in the US & elsewhere due to the profile of the project. Our Privacy-first messaging & non-profit status + MIT backing means that users will be inclined to trust us by default.
Risks therefore include…
Personal data breaches
Published analyses of security deficiencies by hackers/journalists
Reputational damage from both of the above.
Pubicly derivable Security Report indicates various “Medium” vulnerabilities, and suggests we link to Twitter, Facebook, Pinteres,t, Google and Canadian Mind Products.
Manipulation of naive users. Is there a risk that naive users could be manipulated into sharing their location data with organizations other than Healthcare Agencies? Can / should we do more to discourage / prevent this?
Exposure to fake sources of COVID-19 data. Having provided an app that consumes a certain kind of data, but not provided the data itself (the HAs), do we encourage unscrupulous 3rd parties to fill that gap with fake data?
Exposure to possible JSON injection attacks. We have not fuzz / security tested the HA JSON interface.
Theoretical risk of high mobile data bills in the event users are persuaded to download data from an unscrupulous 3rd party who offers a very large JSON file. Not clear what the motivation of such a 3rd party might be, but there is no known limit to the amount the app will try to download, and it may do so over a mobile data network.
Upgrade
We have not tested upgrade from this release to a future release. Therefore we don’t know that this app is upgradable-from without loss of data. Desirable to test this before shipping, else we might have to manage a lot of users through a delete & re-install process.
Upgrade which changes data format (e.g. encrypts data) may be particularly problematic in future. Should that be a problem, we can probably roll out a patch with fixes, that doesn’t lose data, followed by the release that changes data format. So this particular point is probably not a major issue.
Confidentiality
If the App is published to the global App store with support for just 2 languages, does this leak confidential information about the location of a key prospect?
General functional deficiencies
Previously known defects:
#516 - We expect quite a few issues around false negatives in scenarios where both parties are moving > 10m / minute (i.e. anything but stationary). This is due to the tight 20m radius for detection, and the likelihood that phone’s timers for logging GPS data will be out of sync. This is a theoretical problem: we have not yet established how much of a problem this is in realistic scenarios.
Error messages are known to be a weak area - e.g. when configuring a Health Authority, or when network is unavailable (HAs? Newsfeed? Terms). Untested, but it is likely that storage capacity issues also do not produce clear error messages. This can all lead to user confusion, and waste a lot of the Safe Paths' team time in troubleshooting.
New defects found testing the latest app.
### to be completed.
### Check that Haitian Creole text at least fits on all screens, headings etc. at least on a normal form-factor phone.
New function: Select Health Authority during setup
### To be tested. Note any issues identified.
Country-specific issues, risks & limitations
We have not tested the translation in the target language with a native speaker. Therefore we do not know whether there are issues with context, quality of translation, mis-spellings etc.
The target language has not been tested with a wide range of screen form factors. Therefore there is a risk that some text or headings does not fit the available space, on some smaller devices.
When changing language, it is very easy to get into a confusing state where the dashboard shows the wrong language. It is then not possible to get out of this state. # 609.
Guidance issued to users may not be locally tailored - e.g. reference to Mayo Clinic; advice to ask to be tested. #486
Issues with deployments in non-target countries
The fact that no HAs are available could be confusing to users who do not understand that this infrastructure still needs to be rolled out. It may simply look as though the app is incomplete in their region, which may lead to confusion, negative reviews and damage to our reputation.
Suitability for demonstrations with prospects in other countries
These will only be issues if we have to demo the Play STore app. If we can continue to use APKs (which I suspect is the case) then these are non-issues
Restrictions in available languages may be an issue.
Restrictions in function intended to make the release as robust as possible may cause problems (E.g. Google Data Import has been removed)
Unknowns
Reliability of location logging. We have seen this spontaneously stop on occasions. We have not systematically tested for this yet, so there may still be weaknesses here.
Scalability. We have not tested how the app scales with large data sets being served from HAs. We do not know what guidance to give HAs in terms of how much data they can server to their communities.
General quality. We have executed a very short test cycle over the last build before shipping. There is inevitably some risk of problems with the app that would have been identified if we had tested a bit longer.
Issues believed to be non-impactful
The app cannot be installed on multiple user accounts on a single device.
Minor cosmetic issues
Top of text trimmed in setup pages #
Wrong shade of white in HA page # 620
Off center ripple on some phones: #617
Some text doesn’t fit available space on some smaller phones #618 (not known if similar issues exist in Haitian Creole as not tested on many different sized screens)
The app always has a permanent notification, which cannot be dismissed. #473
Brief flash of red on status screen before settling to green. # Not raised (seen on Moto G7/Android 9 - what about other phones?)
Glitch when navigating back from Dashboard to Main screen. #???