Test Report - 9 April 2020 v0.9.1

Covid Safe Paths – Diarmid Mackenzie Test Session #1 – 9/4/2020 

Test Session Charter: General explore of app & hunt for potential issues.  Includes updates 11 April to link to GitHub issues raised.

Approach

My approach is not 100% aligned with the tester brief here: https://docs.google.com/document/d/1HN9XkS-C7X0rJUZ5wom9wE8-7gBygHT8PLOEAEFiYOU/edit#

Rather than following the test plan step by step, I have taken a holistic approach towards testing the App, evaluating the App in the context of its overall goals and purpose (in so far as I understand them, informed by reading the available documentation published on the Google Drive).

My approach to testing is inspired by the Context Driven school of testing, and by the “Rapid Software Testing” approach taught by James Bach & Michael Bolton.

I am available on Slack, so please reach out with any questions.  Alternatively, add comments in this doc & I will aim to respond.

Summary Assessment

• For the most part a slick, well-presented, good looking app.

• Testing was significantly limited by the non-availability of supporting network services (Healthcare Authorities / Trusted Sources with exposure data)

• Top concerns:

• 1. That the app doesn’t present clear enough info/messages to users to effect large-scale positive changes in user behaviour

• 2. Privacy concerns with email as the default mechanism for data sharing

• 3. I was unable to use the “Import Locations” function (unclear whether usability or functional issue).

• 4. The app does not seem to give a positive user experience for the scenario where a patient has been diagnosed with COVID-19 and the app is to be used for contact tracing purposes.

Limitations of this Test Report

I have also not included screenshots from the app within this report.  In most cases, I hope the meaning of my comments is clear without the support of screenshots, but I am happy to provide screenshots to further explain any particular issue observed – please let me know.

This document was originally drafted in Word, then moved to Google Docs.  Hence the dodgy formatting in places.

Background assumptions

Key assumptions made that informed my testing and this report (these are based on background reading in the Safe Paths Google Drive space, but may be incorrect)

• The app has 2 key intended use cases:

o   Use by a concerned party, to help assess & reduce their risk of (a) exposure to COVID-19 and (b) passing on COVID-19 if they have been infected.

o   Use by a patient who is known to have been infected with COVID-19, under the direction of a local Health Authority, for Contact Tracing purposes.

• The key goal of the app is the reduce the spread of COVID-19.  Therefore the app creates value, in as far as it results in changes to behaviour of users, that leads to this goal.  If the app does not lead to behaviour changes by its users, then it has little value.

Details of my top concern

My top concern is that the app as currently implemented doesn’t present clear enough info/messages to effect large-scale positive changes in user behaviour.

While these points are made within the detailed commentary on the app, I have pulled them out here to avoid the overall message being lost.

Specific concerns as follows:

• [DHM-S1-1 New 442] The app generates false positives by default.  It defaults to a “green” status when in fact no data has been collected.  This risks giving people false confidence that they are uninfected and could have a negative impact on their behaviour.

• Due to absence of Healthcare Authority network services, I have not been able to test the scenario in which the app reports a “red” status.  However from the screenshots (below), I do not see any clear guidance to the user that they should self-isolate (or any other advice or actions that they should take).  It is not clear to me what behaviour is desired from end-users when the app reports a “red” status, or how the App is intended to effect that behaviour. [DHM-S1-2 New 443]

Open

Open

1^st screenshot from “Figma screen designs” https://docs.google.com/document/d/1DXFNwjS0hnYy25PnAkXBH5KW9OoW2tw3C9oV2xsX-Ww/edit

2^nd screenshot from “Beta Test Plan” https://docs.google.com/spreadsheets/d/1-xiqmGNoqBUmJiXVX3vhEgpjp4eYDy1gbnunOIHZzG0/edit#gid=1606249405

• [DHM-S1-3 New 444] I am unsure that the purely binary red/green nature of the indicator is optimal for driving appropriate end-user behaviour.  My understanding is that a single contact with a possible point of exposure will lead to “red” status.  If so, there is a risk that users that are given a “red” status dismiss it, on the basis that it is still highly probable that they are not infected.  Once they become habituated to ignoring a “red” status, this makes it harder to drive behaviour change later, when a greater number of contacts with points of exposure have been made.

o   Computing risk of exposure from contact data, and communicating this to the user (whether by a binary red/green status, or via a more detailed measure, e.g. % risk) seems to be a complex topic that needs careful design, and optimal policy may vary due to local outbreak conditions.

o   Assuming the App today implements a binary red/green status based on a single contact with a point of exposure, it is not obvious to me that this is an optimal, or even effective, solution to this problem.

• [DHM-S1-4 New 445] As far as I can tell, the app does not provide any information to a user regarding how to avoid exposure to COVID-19 – it only provides information once they have crossed a path, and therefore might be exposed.  This is important for 2 reasons:

o   First, it seems like a missed opportunity to positively influence user behaviour

o   Second, it means that the app does not provide any “selfish” benefit to an end-user, but only “altruistic” benefits.  It allows them to help avoid spreading COVID-19, but it does not provide any way to help them to avoid catching COVID-19.  The absence of these “selfish” benefits could be a major barrier to adoption & use of the app.

For details of my other top concerns, see detailed testing notes on individual sections of the app.

Detailed testing notes

I worked through each of the main areas of app function, as detailed in the “Test Plan”.  Observations are broken down by area, and writing that in order of severity.

https://docs.google.com/document/d/1HN9XkS-C7X0rJUZ5wom9wE8-7gBygHT8PLOEAEFiYOU/edit#

• [DHM-S1-5 New 446] The intro screens are oriented towards an undiagnosed user.  As a diagnosed user, being directed to use this app by a Healthcare Authority for Contact Tracing, I would not find the messaging reassuring, and could find it upsetting e.g “The way back to normal starts here”, “If you cross paths with someone who already has COVID-19….”

• [DHM-S1-1 New 442] At the end of the setup process, the green “No known contact” screen is a misleading false positive generated in the absence of any real data.  Giving someone a green status where in fact you have zero information about their status is leading, is misinformation, and could lead to the user taking unreasonable risks in the belief they are not infected.

o   The app should not display a green status until a user has some significant movement history, and has registered with a relevant local Hath Authority for contact data.

o   Prior to this point it should be explicit that status is unknown.

o   Implementation suggestion: a % indicator that shows what proportion of possible information the user has access to, could provide a nice incentive for them to add additional config (e.g. import location history, or add more Healthcare Authorities).

o   (but need to also be careful about the potential for misleading if you tell a user they have 100% information).

• [DHM-S1-6 New 447] The “All finished” screen implies to the user they have done all the setup needed to use the app.  This is misleading, as for the app to provide benefits, they also need to add Healthcare Authorities (and also, ideally, import their location history)

o   Suggestion: incorporate both these steps into the initial setup flow.

• Error cases:

o   Good handling for the case where the user does not grant access to location info.  App is persistent and generates notifications until it gets this permission

o   [DHM-S1-7 New 448] Poor user experience if the user chooses “don’t ask me again” when denying permission – user just gets repeated notifications that the app needs to access location info, but gets no guidance how to get out of this situation (they need to go deep into the settings menu & explicitly adjust the permissions).

• [DHM-S1-8 - New 449] I was not able to get this function to work, despite following all on-screen instructions.

o   Within the embedded Google Account pane, I was able to export & download my location history.

o   After that, instructions were to “open this screen again.  The data will import automatically”.  I tried to do this many times, but did not succeed.  I don’t know whether that’s a functional issue or a UX issue.

• [DHM-S1-9 - Existing 69] I see a lot of problems with the embedded Google panel.  It creates a complex UI, that I think many users will not be able to successfully navigate.  If possible, it would be much preferable to have this function available natively in the App, with a simplified UX.

o   There are lots of options exposed, and it is not obvious which steps to take

o   General google navigation is available, which allows the user to navigate all over their google account, and get completely lost.

o   There is a “Help” link, but it links to help for Google Account, rather than help for the operation the user is actually trying to perform.

o   Likewise with the “Send feedback” link – this is available, but for Google.

o   Lack of control of the content Google display is a risk, e.g.

§  If you create user documentation of this process (online, in-app, tutorial videos etc.) there is a risk of changing their UI.

§  Google Help warned me that they were running with a “limited team in light of COVID-19” – not a message you’d want misinterpreted as coming from Safe Paths.

o   Android “Back” button moves back in the SafePaths app, not in the embedded Google page.

o   Reduced screen available meant that some Google pages I was able to navigate to were cropped on the right.

o   Confusing options available for data export like “Export every 2 months for a year”.

o   “File type & size” – should it be .zip or .tgz?  2GB?

o   I am in Italy, and the embedded google panel was in Italian, even though the rest of the app was in English.

• Other notes

o   [DHM-S1-10 - New 450] Since I am already logged into my google account in my Android phone, I had expected my google account email address to be auto-populated.

o    [DHM-S1-11 - New 451] Google warns “the process can take a long time, possibly hours or days”.  I don’t know whether this is true, but it’s not consistent with the messaging in the SafePaths app.

o    [DHM-S1-12 - New 452] On the Dashboard screen: “Safepaths has no affiliation with Google & never shares your data.”  This claim is not completely accurate since one of the key functions of this app is to share the user’s data.  “… without your consent” should be added, perhaps?

• Unexplored topic: what happens if I import a Google Location history some time after installing the app, so that the Google location history overlaps with data the app has already logged?

• [DHM-S1-13 - New 453] Privacy concern.  I am offered many options for how to share my data, but the default option is via email.

o   Having shared via email, my unencrypted, unredacted location history is now stored indefinitely in both my “sent items” folder, and in the mailbox of the Healthcare Authority I sent it to.

o   This seems inconsistent with the stated privacy aims of the project (and e.g. with the policy to delete location data after 28 days).

• [DHM-S1-14 - New 454] Gaps between records in the location data were highly variable: 300, 308, 318, even 518 seconds.

• [DHM-S1-15 - Existing #350] Location & timestamp data is excessively verbose, with spurious accuracy

o   Millisecond timestamps, but they all end in 000

o   8 decimal places on location data.  However even when stationary, I see my location vary significantly in the 4^th decimal place on both longitude & latititude.

• [DHM-S1-16 - Existing #264] Longitude & latitude data is not very accurate

o   I am indoors, in a rural location.  Phone has Wifi + SIM card with OK signal.

o   While stationary, over an hour, data varied as follows:

§  Lat: -0.0000012 to + 0.00015 (total variation: 0.00016)

§  Long: -0.0002873 to + 0.0007226 (total variation: 0.00101)

o   On the basis that 1 degree is approx. 100km, that’s ~100m variation.

• [DHM-S1-17 - New #455] Sharing by email adds some useful context to the message “Here is my location log from Private Kit”.  However, sharing by other means (WhatsApp, Google Drive) just dumps an JSON file with a cryptic filename, with zero context).

• Noted that although the App does not request background permissions from Android, it remains able to record location data even when the app is closed & the phone is put to sleep.  (see notes on Notifications, under “other observations” below).

• I also observed that when the app is uninstalled, location data stored appears to be deleted: on re-install on the app, only location data since the most recent setup is available.  From a privacy pov, this is good.

• Limitation: I have not yet been able to run long-duration tests to confirm that location data is correctly aged out after 28 days.

• Difficult to test this properly, because (I believe) the available “Health Authorities” are dummy entries, and there are not yet any real “Health authorities” to connect to.  Hence testing so far of this area is limited.

• [DHM-S1-18 - New #456] There is apparently no policing of what text can be entered as a URL.  I was able to paste 1000s of lines of text here.

o   I haven’t explored this, but if there is no policing here, I’d expect it may be possible to attack the app here using SQL injections, very long text etc.

• [DHM-S1-19 - Existing #430] As well as no policing of URLs, there is also no validation, so easy for a user to mistype a URL, and end up with the App not working correctly.

o   See also point above where zero available Health Authority data leads to a false positive “OK” report to the user.  In combination these create a substantial risk that the App spreads misinformation.

• [DHM-S1-23 - New #460] Not clear whether the app will mandate the use of HTTPS for Health Authority URLs, and insist on trusted security certificates.  Clear security risks if not.

• Several UX aspects that I found unintuitive [DHM-S1-20 - New #457]

o   “Save” (floppy disk) icon used to add an entry.  This doesn’t match any UX mechanic that I am familiar with in mobile apps for adding entries to a list.

o   No clear separation of the part of screen used to select a Health Authority from the list of already configured Has above (both plain white background)

o   Health Authorities that I have already added continue to appear as options in the list.  But I can’t add them twice, so I’d expect them to disappear from the selection list once selected.

o   “Paste your URL here”.  I’d prefer “Type or Paste your URL here”

o   When I enter a 2^nd URL, the previous URL I entered is offered as the default value.  I don’t consider this helpful: whatever URL I am trying to add, it is not the one I already added.

• [DHM-S1-21 - New #458] When there are many Health Authorities, it is not clear what UX will be to allow me to choose the ones relevant to me.

o   Are they listed alphabetically?

o   How does the app work with a list that is too long to fit on the page?

o   Can I filter for Health Authorities that are local to my area etc.?

o   Can I search for a given Health Authority name?

• [DHM-S1-22 - New #459] Screen title “Choose health authority” suggests I should just choose one.  “Trusted Sources” + the overall UI design, suggests that I can pick more than one.  Desirable to avoid this ambiguity.

• [DHM-S1-23 - New #460] There could be a security risk of an attack where users are encouraged to enter a malicious URL in the app.

• [DHM-S1-24 - New #461] The 1

^st time I use the “Add Trusted Source” button, only “Add authority via URL” is visible – the other 2 available entries take a few seconds to populate.

• [DHM-S1-25 - New #464] When removing an authority, there is a typo in the word “PROCCEED”

I have not yet tested this function (insufficient elapsed time).

In any case, At this point without any real (or synthetic) Health Authority data in my area, any testing at this stage would be very limited.

Other screens

Exposure History

• Even when I have already configured several “Trusted Sources” on the “Choose Health Authority” screen, I am still told “No exposure data is available”, and guided back to the “Choose health authority” screen

o   Presumably this is because none of the Trusted Sources I have configured is actually live.

• [DHM-S1-26 # 469] Unnecessary variation of terminology:

o   “Select Healthcare Authorities” vs “Choose Healthcare Authorities”

o   “Healthcare Authorities” vs. “Trusted Sources” vs. “trusted healthcare authorities”

o   As well as helping to avoid user confusion, having single, consistent terms will also help with Internationalization.

Legal

• [DHM-S1-27 #470] Terms of Use.  This is not my area of expertise, but I am surprised that the Terms of Use are hidden away,.  SafePaths seems to infer that by using the App I consent to the Terms of User, but at no point during installation was I invited to read these, nor did I explicitly consent to them.

• (I ave not read/reviewed the full Terms of Use – this is not my area of speciality).

• [DHM-S1-28 #471] It seems odd to report the Version & my android OS in the Legal page.  The “About” screen would make more sense to me.

Other observations

 [DHM-S1-29 #472] Analytics

• My guess (maybe incorrect) is that we have avoided instrumenting the app with Analytics, due to Privacy concerns.

• That’s valid, but the downside is that it will be very difficult to evaluate how the app is actually working.

• E.g. suppose our contact detection algorithm is so sensitive that 99% of users have a “red” status.  I don’t think we’d know or get any feedback that might allow us to tune the contact algorithm.

• As noted earlier, computing risk of exposure from contact data, and appropriately communicating this to the user seems to be a complex topic.  I think the chances of getting this correct without analytics showing e.g. % of users red vs. green, average location history stored by each user, number of Trusted Sources configured etc. seem to me very slim indeed.

Notifications:

[DHM-S1-30 New #473] “COVID Safe Paths Enabled” notification about storing GPS data cannot be dismissed.

• My only option is to “minimize” the notification, but this seems to have no effect.

• (I believe as a result) the SafePaths app always has a small purple dot in the top right corner indicating notifications.

I was interested to note that the App doesn’t seek permissions to run in background.  However it seems to be able to continue to log my location info every 5 minutes even when I close the app and put my phone to sleep.

• My guess is that it uses this “notification” mechanism to keep running somehow (rather than relying on having permission to run in the background)?

• If so, that’s a clever workaround.  But the disadvantage is that this “sticky” notification might annoy users (some may delete the app in frustration), and prevent the app being able to clearly signal a notification when it wants to.

[DHM-S1-31 New #463] Setup instructions here don’t mention that you must delete prior installations of SafePaths or Private Kit.  If you don’t do this, install fails with an unclear error message.

https://docs.google.com/document/d/1HN9XkS-C7X0rJUZ5wom9wE8-7gBygHT8PLOEAEFiYOU/edit#

(given the onboarding instructions for the project ask you to install the app from the Play Store, almost everyone will hit this)

[DHM-S1-32 #474] Language support.  When I set my phone to Italian, the UI remains in English.  I  have seen PMO documents that suggest we are exploring rollouts in Andorra, Ethiopia etc.

• It’s not clear to me that the rollout strategy & the product roadmap in terms of localization support are fully aligned.

Future testing

Currently testing is significantly hampered by the non-availability of Trusted Sources of contact data for infected patients.

A key priority for testing should be the availability of test Trusted Sources, with synthetic data, that can be synchronized by the App.

• It would be desirable for App testers to have the ability to control the synthetic data, to generate specific interesting test conditions.

These capabilities would enable more complete testing of:

• Trusted Sources pages

• Exposure History page

• Flows where user is notified that they may have been exposed to an infection.

• Performance at scale, where the question  of how the app will cope with a data set consisting of 1000s of users, with 100s of location history data points, is of key concern.

In the absence of progress in these areas, the other areas that would be valuable targets for further test sessions at this point.

• Testing across multiple makes / models of device, and form factors

• Security testing, particularly around the URL for Trusted Sources

• Long-duration testing.  So far I only have tested a few hours of location data.

• Location data import (once I have understood why I could not make this work).