Risks of Harm

Owned by Diarmid Mackenzie (Unlicensed)
Last updated: Apr 28, 2020
2 min read

A place to log possible risks of harm that could result from the Safe Paths App. Intention is that we grow this list as we encounter new considerations about how the product could lead to harm. Also that we regularly review this list, and use it to help product direction such that we mitigate these risks as far as possible.

All project participants are invited to add concerns to this list

Identified Risks of Harm

Increased social inequality

  • Poorer communities have less access to smartphones

  • Poorer communities live more densely, and may be more prone to false positives & false negatives than more privileged communities

  • COVID-19 is disproportionately hitting ethnic minority communities in the US. Poor people will use this app (and therefore be at risk of any harms it may cause), while rich people never will.

  • Misuse / abuse of data by Health Authorities of other Government Agencies. E.g. Concern that data heat maps could lead to “kettling” of deprived communities to contain the spread.

Corruption

  • Contact tracer may redact certain business from traces, in exchange for compensation from them.

Harm to small businesses

  • Concern about harm to small businesses, because of information being spread that there was a COVID infection there. Larger businesses will be able to cope, but small businesses will have less resilience.

Health status stamps (if implemented)

  • Could be problematic, and lead to the App becoming de facto mandatory, even if that was not our original vision.

Unregulated Health Authorities (mostly outside the US)

  • May not respect our vision that private data should only be shared with consent.

Reckless behaviour due to misinformation (False Negatives)

  • Particularly if the app is not very careful to indicate the limits of information it has (e.g. perhaps there is no HA data downloaded, or perhaps no location data has been recorded by the App).

Fatigue due to excess False Positives.

  • If the App “cried wolf” this could lead to disregarding not only info from the app, but also possibly from other sources of guidance as well.

Reckless behaviour due to false sense of safety

  • Even if the app does not have high levels of false positives or false negatives, there is a risk that people substitute use of the APp for other more-effective controls of spread, e.g. hand-washing.

Problems caused by leaks of personal data, e.g. location information

  • Individuals attacked or discriminated against having been identified as spreaders of the disease

  • Individuals exposed to harm following leaks of location info to another person (e.g. Domestic Violence)

  • Leaked location information used inappropriately by security services or law enforcement.

“Griefing” - deliberate injection of false COVID infection reports into the system, for any number of reasons:

  • Business competition

  • Entertainment

  • Civil disobedience / disruption

  • Foreign attackers

  • Organized crime

 

Actions we could take to identify further Risks of Harm

STRIDE threat modelling - see “Security” pages.

“Abusability Testing” -

https://www.wired.com/story/abusability-testing-ashkan-soltani/

https://www.usenix.org/node/226468

Diversity - Open Source Software has a diversity issue. More diversity in our project volunteers might help identify more risks of harm.

Other Risk-related brainstorming activities