V1.0 App Store Release Candidate - Quality Assessment

Owned by Diarmid Mackenzie (Unlicensed)
Last updated: Apr 24, 2020 by Diarmid Mackenzie
7 min read

We are under a lot of pressure to deliver an App via the App Store to one of our prospects.

They are in a country outside the US, so we had considered just releasing to the App Store in that country. However that would entail additional delays, and therefore the current plan is to publish an initial release globally on the App Store.

This document assesses the latest App build (v0.9.6 ??) against suitability for this purpose, and highlights the key risks & quality issues that we have been able to identify. This draws on a combination of testing performed on the latest build,and previous project knowledge about the state of the App.

Expected Usage

The key uses we anticipate for the App are:

  1. Evaluation & testing by one prospect in a small country outside the US.

  2. Use in marketing demos with other prospects

  3. Download and use by people across the world who have heard of the project, and want to begin storing their location data on their phone, in case they are later diagnosed with COVID-19

  4. Other people who are interested in the project, which may include goverment researchers, other COVID-19 projects, Hackers, journalists etc. etc.

Note that usages 3 & 4 are usages that we are exposed to by the decision to release the app globally. If we limited release to a single small country, our exposure to these usages would be massively reduced.

Key issues and concerns

New function: Select Health Authority during setup

Security, Privacy & misinformation.

  • We know we have issues with Security & Privacy vs. where we want to get to - e.g. data stored unencrypted on users' phones. This is a concern given that the app may become widely used to store personal data in the US & elsewhere due to the profile of the project. Our Privacy-first messaging & non-profit status + MIT backing means that users will be inclined to trust us by default.

  • Risks therefore include…

  • Personal data breaches

  • Published analyses of security deficiencies by hackers/journalists

  • Reputational damage from both of the above.

  • Pubicly derivable Security Report indicates various “Medium” vulnerabilities, and suggests we link to Twitter, Facebook, Pinteres,t, Google and Canadian Mind Products.

  • Manipulation of naive users. Is there a risk that naive users could be manipulated into sharing their location data with organizations other than Healthcare Agencies? Can / should we do more to discourage / prevent this?

  • Exposure to fake sources of COVID-19 data. Having provided an app that consumes a certain kind of data, but not provided the data itself (the HAs), do we encourage unscrupulous 3rd parties to fill that gap with fake data?

  • Exposure to possible JSON injection attacks. We have not fuzz / security tested the HA JSON interface.

  • Theoretical risk of high mobile data bills in the event users are persuaded to download data from an unscrupulous 3rd party who offers a very large JSON file. Not clear what the motivation of such a 3rd party might be, but there is no known limit to the amount the app will try to download, and it may do so over a mobile data network.

Upgrade

  • We have not tested upgrade from this release to a future release. Therefore we don’t know that this app is upgradable-from without loss of data. Desirable to test this before shipping, else we might have to manage a lot of users through a delete & re-install process.

  • Upgrade which changes data format (e.g. encrypts data) may be particularly problematic in future. Should that be a problem, we can probably roll out a patch with fixes, that doesn’t lose data, followed by the release that changes data format. So this particular point is probably not a major issue.

Confidentiality

If the App is published to the global App store with support for just 2 languages, does this leak confidential information about the location of a key prospect?


Corporate Transparency

The Terms + About pages refer to a umber of corporate entities without providing enough clarity about who they are or their role in the project.

  • The new EULA raises some questions that aren’t easy to answer. When this app goes live, the EULA is sure to get some scrutiny. “Pathcheck Inc” has zero public profile, no website explaining what it is etc. It does not appear on google at all, even if you search for it alongside Safe Paths. I had to look up in a Massachussetts business directory to find any info at all. At least when I did, some names were recognizably part of the Safe Paths project.

  • Are we happy with the Team section on the About page? It refers to TripleBlind, EyeNetra, Lin Ventures. For TripleBlind, at least there are some traceable connections to the project. The connections and relevance of EyeNetra and Link Ventures are unclear and un-googleable as far as I can tell. This all might appear (to someone of a suspicious mind) a bit shady. Together with the fact that Pathcheck Inc. is not mentioned here. Given that there is already skepticism about the role of various private entities in the project, this just seems to create more mystery & fuel for suspicion, when we could provide some much clearer information about what the various corporate entities involved are, what their roles are etc.

General functional deficiencies

Previously known defects:

  • #516 - We expect quite a few issues around false negatives in scenarios where both parties are moving > 10m / minute (i.e. anything but stationary). This is due to the tight 20m radius for detection, and the likelihood that phone’s timers for logging GPS data will be out of sync. This is a theoretical problem: we have not yet established how much of a problem this is in realistic scenarios.

  • Error messages are known to be a weak area - e.g. when configuring a Health Authority, or when network is unavailable (HAs? Newsfeed? Terms). Untested, but it is likely that storage capacity issues also do not produce clear error messages. This can all lead to user confusion, and waste a lot of the Safe Paths' team time in troubleshooting.

Unfinished External Content

The app links in several bits of external content that are unfinished, or just “test” pages (and look exactly like that). The overall impression is not just of an unfinished product, but also that we don’t even know it is unfinished, or we forgot to finish it.

  • The App still has an HA called “Example Test Authority for Testing”. This looks terrible. I understand this data is external to the app, but it should be cleared up before we go live on the App Store.

  • The News pages still look terrible. Again, I recognize they are hosted outside the App, but we should not go live with the App with the pages as they are. A blank page that says “coming soon” would be better.

Country-specific issues, risks & limitations

We have not tested the translation in the target language with a native speaker. Therefore we do not know whether there are issues with context, quality of translation, mis-spellings etc.

We found two issues in the second language:

  • the Android pop-up requests to access the device’s location & access photos, media & files, are both still in English (GitHub issue on the way). #676. (though Dev view is this cannot be fixed)

  • In the Exposure Noritication page, days of the week, months, and time units are not localized #677.

The target language has not been tested with a wide range of screen form factors. Therefore there is a risk that some text or headings does not fit the available space, on some smaller devices.

Guidance issued to users may not be locally tailored - e.g. reference to Mayo Clinic; advice to ask to be tested. #486

Issues with deployments in non-target countries

The fact that no HAs are available could be confusing to users who do not understand that this infrastructure still needs to be rolled out. It may simply look as though the app is incomplete in their region, which may lead to confusion, negative reviews and damage to our reputation.

Further, when no HA is set up, the App reports “no known contact” rather than “No data”. This seems to be a bug, contrary to the MVP spec, and it specifically impacts users who do not have a local HA.

 

Suitability for demonstrations with prospects in other countries

These will only be issues if we have to demo the Play STtre app. If we can continue to use APKs (which I suspect is the case) then these are non-issues

  • Restrictions in available languages may be an issue.

  • Restrictions in function intended to make the release as robust as possible may cause problems (E.g. Google Data Import has been removed)

 

Unknowns

Reliability of location logging. We have seen this spontaneously stop on occasions. We have not systematically tested for this yet, so there may still be weaknesses here.

Scalability. We have not tested how the app scales with large data sets being served from HAs. We do not know what guidance to give HAs in terms of how much data they can server to their communities.

General quality. We have executed a very short test cycle over the last build before shipping. There is inevitably some risk of problems with the app that would have been identified if we had tested a bit longer.

 

Issues believed to be non-impactful

The app cannot be installed on multiple user accounts on a single device.

Minor cosmetic issues

  • Wrong shade of white in HA page # 620

  • Off center ripple on some phones: #617

  • Some text doesn’t fit available space on some smaller phones #618 (not known if similar issues exist in Haitian Creole as not tested on many different sized screens)

  • The app always has a permanent notification, which cannot be dismissed. #473

  • Brief flash of red on status screen before settling to green. # Not raised (seen on Moto G7/Android 9 - what about other phones?)

  • Glitch when navigating back from Dashboard to Main screen. (not new, but not in GitHub yet - to be added)

  • Exposure notification says March/April when all data points are in April. (also not new, but still to be added to GitHub)