v1.0.1 Quality Report

Owned by Diarmid Mackenzie
Last updated: Apr 30, 2020
3 min read

v1.0.1 is a quick follow-up to v1.0, primarily intended to fix late-breaking issues in that release, which weren’t severe enough to block roll-out, but which we wanted to solve as soon as possible afterwards.

Headline

There are 2 obvious bugs with location settings bugs in v1.0.1, such that on balance it feels like a step back, rather than a step forward vs. v1.0.

At least one, and possibly both, of these bugs seem have been introduced by function to skip location settings during the setup flow. We understand this function was not supposed to have been included in v1.0.1 - but it is present nonetheless.

To get to a shippable v1.0.1, we recommend either

  • we fix the bugs with the location settings

  • or: we remove the function to skip location settings during the setup flow.

Expected Usage

As v1.0 - see V1.0 App Store Release Candidate - Quality Assessment

Changes in v1.0.1 vs. v1.0

Changes are described here:

https://github.com/tripleblindmarket/covid-safe-paths/releases/tag/v1.0.1

  • Remove healthcare selection to onboarding flow (#679)

  • Minor EULA updates (#671)

  • Fix settings icon on iOS (#687)

  • Fix issue where news from the healthcare authority was not shown (#704)

  • As always, lots of back end work

  • English and Haitian Creole Translation

However, there is evidence in testing of other changes - in particular there is a new function during on-boarding to skip the decision to enable location services. It is not clear why this changes is in the release, as it does not appear to be part of the intentions for the release

Test Approach

We have run through a standard set of tests, covering all the screens in the App, on both Android and iOS.

Our detailed test plan has not been kept up-to-date with various changes to onboarding workflows, so our testers have been using the test plan as a general guide, but evaluating the app “as a user”, rather than comparing the fine details against exactly what is documented in the test plan.

Because the expected changes in the test plan are fairly minor, we have kept testign lightweight and have not retested against a large range of different devices.

New Problems

Suspected new in v1.0.1…

“Enable Location” button does not work on iOS: https://github.com/tripleblindmarket/covid-safe-paths/issues/717

After Skipping setting Location on Android, Location is incorrectly reported as “Active”: https://github.com/tripleblindmarket/covid-safe-paths/issues/719

Unknown whether new in v1.0.1…

Some (but not all) iOS devices see a 3-5 second hang on pressing the “Finish Setup” buttonhttps://github.com/tripleblindmarket/covid-safe-paths/issues/727

Probably existed in v1.0 as well…

On iOS, Haitian Creole version, some text is still in English: https://github.com/tripleblindmarket/covid-safe-paths/issues/718

Existing Concerns

Since v1.0.1 is a very small delta on v1.0, most of teh concerns flagged wot the v1.0 release also apply to v1.0.1

Also, since the v1.0 release, we have developed a better understanding of problems in a couple of key areas:

Since these problems are not regressions, there would be no reason to block roll-out of v1.01 on the basis of these.

They are included here as input to future product direction.

Lessons to Learn

There appear to be 2 key lessons to learn from the above.

  • It appears that changes not intended for v1.0.1 (option to skip location settings) have been included in the release. Some (and possibly all) of the regressions in the release appear to be related to the includion of this unintended change.

  • The very detailed test plan initially developed for the app has quickly become outdated, because there have been a lot of changes to the UI, and there hasn’t been the necessary communication & action to update the test plan. Testing need to decide whether to move to a less tightly defined test plan, or to continue with a very explicit test plan, and put in place processes to keep this up to date as the product UI develops.