Page Comparison

...

This is a short document and is not comprehensive. If something important is not mentioned, or highlighted, that may be concealing an issue, so please do flag it.

Table of Contents

Background

1.2.0 is a new release that we hope to push to the App Stores on 16 July 2020.

...

iOS GPS changes are isolated from the rest of the codebase and have been extensively tested.
endEnd-to-end location data sharing and exposure notifications has been a recent focus of test, and there has been little (if any) codebase change since this testing was performed.
PR Spanish is probably the highest risk new feature. The new translation comes from a native speaker, but it has not been tested by a native speaker, so there is a risk that minor errors will have crept through. There are also risks of alignment issues, text not fitting etc.

With regard to regressions in pre-existing functionfunctionality, the risks are moderate to low.

There has been no active change control on the codebase, with plenty of checkins fom the GAEN team etc., so there is a definite risk that some function has been indvertently inadvertently broken, and this has not been discovered yet.
However the vast majority of the product function functionality will have been exercised during the iOS location testing, exposure notification testing, and the Guam lab evaluation which has been conducted using these builds (or very similar ones). This significantly mitigates the risk of serious regressions.

...

A light retest of location data reliability on both iOS and ANdroidAndroid
A light test of Exposure Notification function
A test by a non-speaker of the language of the device using PR Spanish, checking for issues where text does not fit, English text is not translated etc.
A broad general regression test of the App across both iOS and Android.

...

On 2 different Android devices, and 2 different iOS devices, covering multiple OS levels, we onboard the app, collect location data for 6 hours, and analyze the output JSON data (this tool or similar):

https://diarmidmackenzie.pythonanywhere.com/upload

...

This will be performed using the Staging version of Safe Places.

The general approach to e2e testing will be as described here: How to Test GPS End to End

...

However, these 52 tests are somewhat out of date, as they have not been maintained since they were written. Therefore testers should expect to update and adjust these test cases to fit current functionality.

There is no requirement to report test results in PractiTest if this is not convenient - for example the test cases can be exported, pasted into Confluence, and then annotated inline with test results.

Due to the desire to ship the release quickly, some long duration tests in the test plan may be skipped, or replaced with a shorter best-effort equivalent.

Schedule

Test builds for 1.2.0 have been available from early on Wednesday Eastern time.

Our goal is to push the release to the App Stores by close of business Thursday. Therefore our intention is to perform this testing on Wednesday and Thursday, concluding early afternoon in the US.

We are deliberately aiming to bring a broader set of testers into this test effort, to help broaden our testing capacity. As a result, test execution may be less tightly compressed than what we have been able to achieve on previous releases.

However, if we make slow progress on Wednesday, we will set all hands to the pump to complete the necessary testing in time for Thursday afternoon.

Known Defects & Untested Areas

...

The intention of this section is to highlight known quality shortcomings with the Mobile App that are believed to be acceptable for 1.2.0, and will not be raised by testers during QA.

Known issues

There are known to be text display issues on certain Android phones, certainly the OnePlus 7T, and possibly others.
Battery consumption is known to be higher than we would ike (up to 25% per day) on some iOS and Android devices
Location gathering is known not to be 100% reliable, though we believe it is at least 90% reliable on Android, and at least 80% reliable on iOS.
The app does not handle the condition in which Location services are disabled at a whole-device scope. In this case, the error messages and direction to the user are unhelpful.
We know that there are several issues wit Accessibility even at the WCAG A standard (and certainly at the AA standard).

Testing limitations

We have not tested the mobile app with Android versions < 7.0.
While we have tested a range of different devices, we have not yet tested the very broad range that we would want to test ahead of a broad community rollout. So there remains a risk of problems with certain specific makes / models of device.
Our Exposure Notification testing has only covered mainline cases so far. We have not tested a full range of edge conditions, we have not proven that JSON files are correctly cached etc.
We have performed limited testing with error conditionsand conditions and adverse conditions, such as network unavailable, Airplane mode, Battery saver etc.
Scalability testing of the Mobile App has not yet been performed. In particular the performance of Exposure Notification calculations with 14d of data stored on the phone remains unknown.
Prvacy Privacy and Security Testing of the Mobile App are still in progress, with various key areas of testing not yet covered.