...
Note that usages 3 & 4 are usages that we are exposed to by the decision to release the app globally. If we limited release to a single small country, our exposure to these usages would be massively reduced.
Key issues and concerns
New function: Select Health Authority during setup
Unfortunately this has led to some significant navigation issues during setup. The key is that to progress past HA setup, you have not go back, to go forward again. And the back button is in the top left hand corner, which doesn't match any of the navigation patterns the user has seen up to that point. I anticipate most users will fail to set the app up, and have to try again a 2nd time to succeed.
There are some other related navigation issues too - see : https://github.com/tripleblindmarket/covid-safe-paths/issues/675
Additionally, the new flow does not cater at all well for the majority of App installers who will not have any local HAs to set up.
Overall, while I completely agree that configuring HAs belongs in the initial setup sequence, these changes have actually made setup of the App more complicated & confusing.
Security, Privacy & misinformation.
We know we have issues with Security & Privacy vs. where we want to get to - e.g. data stored unencrypted on users' phones. This is a concern given that the app may become widely used to store personal data in the US & elsewhere due to the profile of the project. Our Privacy-first messaging & non-profit status + MIT backing means that users will be inclined to trust us by default.
Risks therefore include…
Personal data breaches
Published analyses of security deficiencies by hackers/journalists
Reputational damage from both of the above.
Pubicly derivable Security Report indicates various “Medium” vulnerabilities, and suggests we link to Twitter, Facebook, Pinteres,t, Google and Canadian Mind Products.
Manipulation of naive users. Is there a risk that naive users could be manipulated into sharing their location data with organizations other than Healthcare Agencies? Can / should we do more to discourage / prevent this?
Exposure to fake sources of COVID-19 data. Having provided an app that consumes a certain kind of data, but not provided the data itself (the HAs), do we encourage unscrupulous 3rd parties to fill that gap with fake data?
Exposure to possible JSON injection attacks. We have not fuzz / security tested the HA JSON interface.
Theoretical risk of high mobile data bills in the event users are persuaded to download data from an unscrupulous 3rd party who offers a very large JSON file. Not clear what the motivation of such a 3rd party might be, but there is no known limit to the amount the app will try to download, and it may do so over a mobile data network.
Upgrade
We have not tested upgrade from this release to a future release. Therefore we don’t know that this app is upgradeableupgradable-from without loss of data. Desirable to test this before shipping, else we might have to manage a lot of users through a delete & re-install process.
Upgrade which changes data format (e.g. encrypts data) may be particularly problematic in future. Should that be a problem, we can probably roll out a patch with fixes, that doesn’t lose data, followed by the release that changes data format. So this particular point is probably not a major issue.
Confidentiality
If the App is published to the global App store with support for just 2 languages, does this leak confidential information about the location of a key prospect?
Corporate Transparency
The Terms + About pages refer to a umber of corporate entities without providing enough clarity about who they are or their role in the project.
The new EULA raises some questions that aren’t easy to answer. When this app goes live, the EULA is sure to get some scrutiny. “Pathcheck Inc” has zero public profile, no website explaining what it is etc. It does not appear on google at all, even if you search for it alongside Safe Paths. I had to look up in a Massachussetts business directory to find any info at all. At least when I did, some names were recognizably part of the Safe Paths project.
Are we happy with the Team section on the About page? It refers to TripleBlind, EyeNetra, Lin Ventures. For TripleBlind, at least there are some traceable connections to the project. The connections and relevance of EyeNetra and Link Ventures are unclear and un-googleable as far as I can tell. This all might appear (to someone of a suspicious mind) a bit shady. Together with the fact that Pathcheck Inc. is not mentioned here. Given that there is already skepticism about the role of various private entities in the project, this just seems to create more mystery & fuel for suspicion, when we could provide some much clearer information about what the various corporate entities involved are, what their roles are etc.
General functional deficiencies
...
#516 - We expect quite a few issues around false negatives in scenarios where both parties are moving > 10m / minute (i.e. anything but stationary). This is due to the tight 20m radius for detection, and the likelihood that phone’s timers for logging GPS data will be out of sync. This is a theoretical problem: we have not yet established how much of a problem this is in realistic scenarios.
New defects found testing the latest app.
### to be compleed.
New function: Select Health Authority during setup
Note any issues identifiedError messages are known to be a weak area - e.g. when configuring a Health Authority, or when network is unavailable (HAs? Newsfeed? Terms). Untested, but it is likely that storage capacity issues also do not produce clear error messages. This can all lead to user confusion, and waste a lot of the Safe Paths' team time in troubleshooting.
Unfinished External Content
The app links in several bits of external content that are unfinished, or just “test” pages (and look exactly like that). The overall impression is not just of an unfinished product, but also that we don’t even know it is unfinished, or we forgot to finish it.
The App still has an HA called “Example Test Authority for Testing”. This looks terrible. I understand this data is external to the app, but it should be cleared up before we go live on the App Store.
The News pages still look terrible. Again, I recognize they are hosted outside the App, but we should not go live with the App with the pages as they are. A blank page that says “coming soon” would be better.
Country-specific issues, risks & limitations
We have not tested the translation in the target language with a native speaker. Therefore we do not know whether there are issues with context, quality of translation, mimis-spellings etc.
When changing language, it is very easy to get into a confusing state where the dashboard shows the wrong language. It is then not possible to get out of this state. Raised here, but the bug report is not clear about how easy it is to hit, and hard to then escape from https://github.com/tripleblindmarket/covid-safe-paths/issues/609 We found two issues in the second language:
the Android pop-up requests to access the device’s location & access photos, media & files, are both still in English (GitHub issue on the way). #676. (though Dev view is this cannot be fixed)
In the Exposure Noritication page, days of the week, months, and time units are not localized #677.
The target language has not been tested with a wide range of screen form factors. Therefore there is a risk that some text or headings does not fit the available space, on some smaller devices.
Guidance issued to users may not be locally tailored - e.g. reference to Mayo Clinic; advice to ask to be tested. #486
...
Issues with deployments in non-target countries
The fact that no HAs are available could be confusing to users who do not understand that this infrastructure still needs to be rolled out. It may simply look as though the app is incomplete in their region, which may lead to confusion, negative reviews and damage to our reputation.
Further, when no HA is set up, the App reports “no known contact” rather than “No data”. This seems to be a bug, contrary to the MVP spec, and it specifically impacts users who do not have a local HA.
Suitability for demonstrations with prospects in other countries
These will only be issues if we have to demo the Play STtre app. If we can continue to use APKs (which I suspect is the case) then these are non-issues
Restrictions in available languages may be an issue.
Restrictions in function intended to make the release as robust as possible may cause problems
...
(E.g. Google Data Import has been removed)
Unknowns
Reliability of location logging. We have seen this spontaneously stop on occasions. We have not systematically tested for this yet, so there may still be weaknesses here.
Scalability. We have not tested how the app scales with large data sets being served from HAs. We do not know what guidance to give HAs in terms of how much data they can server to their communities.
General quality. We have executed a very short test cycle over the last build before shipping. There is inevitably some risk of problems with the app that would have been identified if we had tested a bit longer.
Issues believed to be non-impactful
The app cannot be installed on multiple user accounts on a single device.
Minor cosmetic issues
...
Wrong shade of white in HA page # 620
Off center ripple on some phones: #617
Some text doesn’t fit available space on some smaller phones #618 (not known if similar issues exist in Haitian Creole as not tested on many different sized screens
Unknowns
Reliability of location logging. We have seen this spontaneously stop on occasions. We have not systematically tested for this yet, so there may still be weaknesses here.
...
)
The app always has a permanent notification, which cannot be dismissed. #473
Brief flash of red on status screen before settling to green. # Not raised (seen on Moto G7/Android 9 - what about other phones?)
Glitch when navigating back from Dashboard to Main screen. (not new, but not in GitHub yet - to be added)
Exposure notification says March/April when all data points are in April. (also not new, but still to be added to GitHub)