v1.0.1 is a quick follow-up to v1.0, primarily intended to fix late-breaking issues in that release, which weren’t severe enough to block roll-out, but which we wanted to solve as soon as possible afterwards.
Headline
There are 2 obvious bugs with location settings bugs in v1.0.1, such that on balance it feels like a step back, rather than a step forward vs. v1.0.
At least one, and possibly both, of these bugs seem have been introduced by function to skip location settings during the setup flow. We understand this function was not supposed to have been included in v1.0.1 - but it is present nonetheless.
To get to a shippable v1.0.1, we recommend either
we fix the bugs with the location settings
or: we remove the function to skip location settings during the setup flow.
Expected Usage
As v1.0 - see V1.0 App Store Release Candidate - Quality Assessment
Changes in v1.0.1 vs. v1.0
Changes are described here:
https://github.com/tripleblindmarket/covid-safe-paths/releases/tag/v1.0.1
Remove healthcare selection to onboarding flow (#679)
Minor EULA updates (#671)
Fix settings icon on iOS (#687)
Fix issue where news from the healthcare authority was not shown (#704)
As always, lots of back end work
English and Haitian Creole Translation
However, there is evidence in testing of other changes - in particular there is a new function during on-boarding to skip the decision to enable location services. It is not clear why this changes is in the release, as it does not appear to be part of the intentions for the release
Test Approach
We have run through a standard set of tests, covering all the screens in the App, on both Android and iOS.
Our detailed test plan has not been kept up-to-date with various changes to onboarding workflows, so our testers have been using the test plan as a general guide, but evaluating the app “as a user”, rather than comparing the fine details against exactly what is documented in the test plan.
Because the expected changes in the test plan are fairly minor, we have kept testign lightweight and have not retested against a large range of different devices.
New Problems
Suspected new in v1.0.1…
“Enable Location” button does not work on iOS: https://github.com/tripleblindmarket/covid-safe-paths/issues/717
After Skipping setting Location on Android, Location is incorrectly reported as “Active”: https://github.com/tripleblindmarket/covid-safe-paths/issues/719
Probably existed in v1.0 as well…
On iOS, Haitian Creole version, some text is still in English: https://github.com/tripleblindmarket/covid-safe-paths/issues/718
Existing Concerns
Since v1.0.1 is a very small delta on v1.0, most of teh concerns flagged wot the v1.0 release also apply to v1.0.1
Also, since the v1.0 release, we have developed a better understanding of problems in a couple of key areas:
GPS location logging is highly unreliable, with logging regularly stopping for 10s of minutes, and occasionally stopping for hours, or even days. See https://github.com/tripleblindmarket/covid-safe-paths/issues/454
GPS location logging is inaccurate, often by 50+m, sometimes by 100s of m. See https://github.com/tripleblindmarket/covid-safe-paths/issues/691
The app becomes unstable at modest levels of case data downloaded from an HA. 50 cases w/ full 14 day history (approx 200k data points) is about the limit the App can cope with. See 28 April - Some basic scale testing
Since these problems are not regressions, there would be no reason to block roll-out of v1.01 on the basis of these.
They are included here as input to future product direction.
Lessons to Learn
There appear to be 2 key lessons to learn from the above.
It appears that changes not intended for v1.0.1 (option to skip location settings) have been included in the release. Some (and possibly all) of the regressions in the release appear to be related to the includion of this unintended change.
The very detailed test plan initially developed for the app has quickly become outdated, because there have been a lot of changes to the UI, and there hasn’t been the necessary communication & action to update the test plan. Testing need to decide whether to move to a less tightly defined test plan, or to continue with a very explicit test plan, and put in place processes to keep this up to date as the product UI develops.