28 April - Some basic scale testing

Owned by Diarmid Mackenzie
Last updated: Apr 29, 2020
4 min read

Headline

Headlines of HA data set scalability of V1.0 on a mid-range Android phone.

  • Becomes unstable at around 50 cases per HA, where cases overlap with phones location data (i.e. exposure occurs)

  • Becomes unstable at 75 -100 cases per HA

  • With data sets from multiple HAs below this limit, it is possible to load up to 300 cases. The app becomes unstable at around 500 cases.

This assumed 14d of data per case, w/ 288 data points/day (288 x 5 mins = 24 hours). With redaction of home address and other locations, true data volumes may be half this size, or smaller - in which case we might assume (not yet explicitly tested) that the numbers of cases supported could be 2x larger or more.

Data set sizes

Some size info for various data sets referenced in this testing.

Number of cases

Number of days

Data points per day

Total Data points

Approx size of JSON

Number of cases

Number of days

Data points per day

Total Data points

Approx size of JSON

10

14

288

40k

3.4MB

50

14

288

200k

17MB

100

14

288

400k

34MB

300

14

288

1.2M

102MB

500

14

288

2M

170MB

Testing Details

Testign on a Motorola G7 Power, Android 9.

Using tools I described here:

https://github.com/tripleblindmarket/covid-safe-paths/issues/398

to test scale as described here:

https://github.com/tripleblindmarket/covid-safe-paths/issues/325

Previously I had successfully downloaded data sets with 1 case, and 1 day (288 data poiints) = 24kB.

When I pushed this up…

  • 10 cases / 14d (40k points) seemed OK. Brief flash of white screen before rendering Exposure data screen.

  • 100 cases / 14d (400k points) caused the app to crash within a few seconds. It restarted, but crashed repeatedly, until Android eventually flagged up a warning about repeated crashes.

  • Building up the set of data points with a smaller set of data sets, the app seemed stable at 300 cases /1.2M data points (each individual data set up to 50 cases/14 days / 200k data points).

  • I pushed up to a total of 500 cases (2M data points), across about 13 HAs (mostly 50s, a few smaller ones). The app seemed stable for a while, but then crashed. This time the app did not automatically restart.

  • After that, the app seemed stable for a while. I decided to clear out the HAs. I got rid of most of them. With about 5 left, I noticed that some of the delete operations were taking a notivceable time to process (maybe a second). Then on one delete (with just 2 x HAs left) the app scrashed. The app did not restart automatically.

  • I restarted the app, cleared out the last 2 x HAs, and started working out what the individual number oof cases was that I could add (with 14d history)…

  • I added an HA with 75 cases / 14 days (300k data points). This seemed stable (not sure if it’s significant, but there were no conflicts with this data set). However, when I then deleted the HA, the app crashed.

  • I tried again with an HA with 75 cases / 14 days that did overlap. This caused a crash when viewing exposure history.

  • Starting the app up again, this time iI was able to delete the 75 cases data set.

  • But strangely, when I did this, and had no HA data in the app, I was still receiving notifications about crossing paths, and my location history still showed exposures. This cleared itself out within the time it took to type the previous sentence.

  • Having seen that a data set with an exposure seems to hit problems at a lower limit, I tried with a set of 50 case / 14 days again, but all data points close to my logged location. Compared to what I see with smaller data sets, it took a lot longer for the exposure to be determined (probably jsut the time to download the file), and as soon as I got the notification of exposure, the app crashed.

  • On 2nd attempt, withthe same size data set, the App coped with the Exposure detection without crashing.

  • Tried again with 75/14. Instant crash on exposure detection.

  • Then crashed again shortly after HA deletion.

  • Then again: seems we have a reliable repro at 75/14.

  • When the data does not clash with my location history, I can load 75/14 and be stable, but 100/14 is reliably too much.

 

So summing up the above….

  • For a single HA data set with lots of exposure points, some instability seen at 50/14 (200k data points), reliably unstable at 75/14 (300k data points)

  • For a single HA data set with no exposure points, some instability seen at 75/14 (300k data points), reliably unstable at 100/14 (400k data points)

  • If individual HA data sets are small (< 50/14 = 200k data points), then the system is stable with up to 300 cases, 14d history each (1.2M total data points). The system becomes unstable at around 500 cases (2M data points).

Instability seemed to be triggered at multiple points

  • On exposure detection / notification

  • On deletion of an HA

  • At random moments not obviously dorrelated with any activity.

Some crashes led to immediate restarts. For other crashes the app did not restart automatically & had to be restarted manually.