Quality Targets - Safe Paths App MVP

This is a consultation document intended for use between Test, Dev and PM regarding quality targets for the MVP. Early Draft, under review - do not take any content here as final.

BRIEF UPDATE: After initial conversation with @Sam Zimmerman his feedback is that we are going to have to push forward the quality on Security & Privacy faster than we are at the moment - i.e. not acceptable for them to be Red and we need to get them moving towards Green. Anyone with a particular interst or expertise in Security or Privacy, please get in touch with @Diarmid Mackenzie The rest is probably about right. Some details on overall stratregy and why this is the right prioritization of wuality to follow shortly….

It is generated from the latest Quality Map (Apr 21 2020) by Test.

Based on Test’s understanding of the current levels of function in tha App it represents a best case in terms of quality targets that could be achieved for MVP, subject to fixes to address the bugs we will inevitably find in Test The need to ship the MVP quickly and pragmatically means that it is quite likely that the MVP we ship will in fact fall short of what is outlined in this document.

This is not a recommendation for the quality targets we should aim for in MVP (that is a matter for PLM).

It is not an assessment of the quality currently delivered by the App (our expectation is that the app falls short of this positon today on multiple aspects, but assessment is ongoing).

By highlighting certain cells red & yellow we aim to provide::

Direction for PM, that our current plan is expected to leave deficiencies in these areas vs. the eventual requirements for the product.
Direction for Test, that it would be unhelpful to raise bugs in these areas (except for bugs of extremely high severity) since it is acknoowledged and expected that these aspects of the product will be deficient.
Direction for Dev in terms of which bug fixes to target for MVP, and which bugs can be left.

Limitation: currently this document sets targets for External stakeholders only. Internal stakeholders are important too, but for the sake of efficiency while we assess the value of this exercise, we will leave them out.

If this document proves useful, we can extend or create a parallel doc for Internal Stakeholders.

This document is focussed on the Mobile App, but also flags where an attribute is dependent on other elements of the solution. A similar document could be generated for Safe Places as well, but this does not yet exist.

Users (Undiagnosed)

Attribute	Illustrative statement (not comprehensive)	Safe Paths	Non-App Dependencies
Precision of Info	When I overlapped with someone I want to get specific information (arrow time window, specific place)	We should meet this expectation
Accuracy (False Positives)	I don’t want to be warned about encounters where there was no plausible risk of infection	There will be False positives (e.g. when alone in a car)	Also dependent on redaction practices
Accuracy (False Negatives)	If I have an encounter with someone that risks infection, I want to be notified	There will be many false negatives, but at least some true positves	Also depends on other users sharing data
Reliability (foreground)	I want the app to work when I try to use it. No hangs, crashes etc.	We should meet this expectation
Reliability (background)	Background operations of the app should work all the time - no silent failures	We should meet this expectation
Clarity	I want it to be easy to understand what the app is telling me, and what actions I should take.	We should meet this expectation	May be additonal supporting guidance on websites
Usability	It should be easy to do what I want to do in the app without too much thought.	We should meet this expectation
Privacy	My personal data should not be exposed without my consent	There are known shortcomings here: see Privacy Roadmap - Input from Testing team & Software Testing Community	Also depends on overall privacy architecture
Security	The app should not expose me to any source of harm	There are known shortcomings here	Also depends on overall security architecture
Trust	I’m putting my trust in this app. I don’t want it undermining my confidence in it. I want it to feel like a high quality product..	We should meet this expectation	Dependency on quality of data available to App, availablity of HAs
Cost	The app should not cost me much to run & use (ideally it should be free).	Data volume downloaded is not optimized	Free requires agreements with network operators
Complete data	I want most infected users to share their data (actually a specific category of “False Negatives” but important enough to pull out separately)	N/A	Depends on public campaigning

Users (Diagnosed)

All these attributes are attributes for undiagnosed users too, We pull them out because explicitly considering these attributes in the context of a diagnosed user leads to different insights - and the threats to quality are fundamentally different.

Note: some of the attributes that are important to Undiagnosed users are not important to diagnosed users, because they use the app differently - so not all rows are duplicated here.

Attribute	Illustrative statement (not comprehensive)	Safe Paths	Non-App Dependencies
Reliability	I want the app to work when I try to use it. No hangs, crashes etc.	We should meet this expectation
Non-intrusivity	When I am not trying to use the app, I don’t want it to bother me, or cause me any issues	We should meet this expectation
Clarity	I want it to be easy to understand what the app is telling me, and what actions I should take.	We should meet this expectation
Usability	It should be easy to do what I want to do in the app without too much thought.	We should meet this expectation
Privacy	My personal data should not be exposed without my consent	There are known shortcomings here. See: Privacy Roadmap - Input from Testing team & Software Testing Community	Also depends on overall privacy architecture
Security	The app should not expose me to any source of harm	There are known shortcomings here	Also depends on overall security architecture
Trust	I’m putting my trust in this app. I don’t want it undermining my confidence in it. I want it to feel like a high quality product..	We should meet this expectation
Cost	The app should not cost me much to run & use (ideally it should be free).	Data volume uploaded is not optimized	“Free” requires agreements with network operators

Health Authority

Attribute	Illustrative statement (not comprehensive)	Safe Paths	Non-App Dependencies
Usability	It should be easy to guide a diagnosed use through the process of sending data.	We should meet this expectation	Depends on HA infra to receive data (e.g. email account in MVP)
Clarity	The app should provide users with clear direction about what to do	We should meet this expectation
Accuracy	Information provided by the app must be accurate	We should meet this expectation
Reassurance	Patients directed to the Health authority by the app should not have been distressed, agitated or angered by their experience with the app.	Quality of data & matching in MVP means we don’t expect to meet this
Speed	Individuals at risk of COVID-19 exposure are notified promptly, so they can quickly reach out to the Health Authority for guidance	We should meet this expectation	Depends on broader siolution as well
Retention	A high proportion of people who install the app keep it installed	We should meet this expectation	Depends on broader siolution as well
Utilized	A high proportion of people who install the app regularly use it.	We should meet this expectation	Depends on broader siolution as well
Scalability	There should be no issues with the app as the number of infected patients tracked increased, or as the number of users of the app increases.	We should meet this expectation	Depends on broader siolution as well

Community

The wider community in which the app is deployed. They may or may not be users of the app.

Attribute	Illustrative statement (not comprehensive)	Safe Paths	Non-App Dependencies
Impactful	Reduces the spread of COVD-19	Impact limited in MVP	Depends on broader siolution as well
Enabling	Enables a higher level activity within the community than would otherwise be possible.	Impact limited in MVP	Depends on broader siolution as well
Not harmful	App does not lead to harmful behaviour in the community (e.g. due to misinformation)	Risk of misinformation and other harm with MVP	Depends on broader siolution as well

Local Businesses

Businesses such as restaurants could be significantly damaged by information that connects their location to users diagnosed positive.

Attribute	Illustrative statement (not comprehensive)	Safe Paths	Non-App Dependencies
Privacy	I don’t want the fact that people who have visited my business, and are later diagnosed positive with COVID-19, to become public knowledge.	N/A	Depends on Redaction
Misinformation	I don’t want to suffer damage to my business from misinformation spread by the App.	N/A	Depends on

Internet Service Providers

Attribute	Illustrative statement (not comprehensive)	Safe Paths	Non-App Dependencies
Low impact	Safe Paths should not disrupt my ability to deliver internet services to other users (either in terms of sustained bandwidth, or spikes)	We should meet this expectation
Dfferentiable	I’d like to understand what traffic in my network is driven by Safe Paths. I may want to apply different policy to it (throttling, billing etc.).	We may meet this expectation	Depends on network traffic analysis capabilities

Partners

Partners are any organizations that have a relationship with Safe Paths, and are therefore stakeholders in our success.

Attribute	Illustrative statement (not comprehensive)	Safe Paths	Non-App Dependencies
Privacy & Security	The app must be perceived as delivering its Privacy & Security claims	We won’t meet this expectation.	Depends on messagng about future improvements
Impactful	The app should be having an impact	Impact limited in MVP	Depends on messagng about future improvements
Impact perception	The app should be seen to be having an impact	Impact limited in MVP	Depends on messagng about future improvements
Reputation & brand	The app should have a strong brand and reputation. Among other things, we must avoid “ridiculous” errors, that could make entertaining (but embarrassing) news headlines. E.g. “Car driving at 100mph along freeway past rush hour traffic in the other direction racks up 100 possible COVID-19 infections” Or E.g. “Every resident of 100 storey tower block asked to self-isolate, due to a single COVID-19 postive patient self-isolating in his own apartment”	Risk of exactly these kinds of problems in MVP	Depends on messagng about future improvements

Privacy Advocates

By positioning ourselves as privacy-first, we encourage the support of privacy advocates, campaigners & influencers.

By supporting us, they become stakeholders in our success, and could be harmed if we underdeliver against our promises.

Attribute	Illustrative statement (not comprehensive)
Privacy & Security	The app must be perceived as delivering its Privacy & Security claims	We won’t meet this expectation.	Depends on messagng about future improvements
Misinformation	The app must not develop a reputation for spreading misinformation.	We may not meet this expectation.	Depends on messagng about future improvements
Reputation & brand	The app should have a strong brand and reputation	We may not meet this expectation.	Depends on messagng about future improvements