This is a Test Plan for the 1.2.0 release. It aims to achieve alignment between the PM, Dev & QA teams about the risks inherent in this release, the testing steps we are planning to take to mitigate those risks, and the overall quality goals that the 1.2.0 release is (and is not) expected to meet.
This is a short document and is not comprehensive. If something important is not mentioned, or highlighted, that may be concealing an issue, so please do flag it.
Background
1.2.0 is a new release that we hope to push to the App Stores on 16 July 2020.
The primary goals of the release are:
To fix iOS GPS reliability
To deliver full end-to-end location data sharing and exposure notifications
Puerto Rican Spanish (in addition to English and Haitian Creole that are already supported).
Some further items still under consideration for the release (though unlikely to be included given how close we are to the wire) are Tagalog and Chamorro (both languages).
The 1.2.0 codebase is a recent (July 9) branch from the develop branch, so there will be other changes that have been made to the codebase aside from the changes specifically targeted above. In particular there has been active ongoing development of the GAEN app on the same codebase, along with various other bugs fixes, testability improvements etc.
The 1.1.X series of releases targeted a quality bar that was good enough for demonstrations at a live launch, betas, and pilots. It was not intended to be suitable for large-scale rollout, and does not have the levels of polish, security, and testing across a very broad range of device types and OSs that would be appropriate for such a roll-out.
1.2.0 does not deliver any specific improvements in this regard, though it may contain some bug fixes.
Risks
With regard to the new function, the risks are fairly low:
iOS GPS changes are isolated from the rest of the codebase and have been extensively tested.
End-to-end location data sharing and exposure notifications has been a recent focus of test, and there has been little (if any) codebase change since this testing was performed.
PR Spanish is probably the highest risk new feature. The new translation comes from a native speaker, but it has not been tested by a native speaker, so there is a risk that minor errors will have crept through. There are also risks of alignment issues, text not fitting etc.
With regard to regressions in pre-existing functionality, the risks are moderate to low.
There has been no active change control on the codebase, with plenty of checkins fom the GAEN team etc., so there is a definite risk that some function has been inadvertently broken, and this has not been discovered yet.
However the vast majority of the product functionality will have been exercised during the iOS location testing, exposure notification testing, and the Guam lab evaluation which has been conducted using these builds (or very similar ones). This significantly mitigates the risk of serious regressions.
As discussed above, 1.2.0 is not intended to deliver any specific improvements in terms of polish, security or testing across a very broad range of devices and OSsm vs the 1.1.X series of releases. Therefore, while there are likely to be issues in this area, they would not be considered release blockers unless they were regressions vs 1.1.X (which is unlikely as per above).
Test Plan
The test plan for the release comprises:
A light retest of location data reliability on both iOS and Android
A light test of Exposure Notification function
A test by a non-speaker of the language of the device using PR Spanish, checking for issues where text does not fit, English text is not translated etc.
A broad general regression test of the App across both iOS and Android.
In more detail….
Location Data Reliability
On 2 different Android devices, and 2 different iOS devices, covering multiple OS levels, we onboard the app, collect location data for 6 hours, and analyze the output JSON data (this tool or similar):
We expect to see a minimum of 80% of the expected number of data points on iOS, and a minimum of 90% on Android.
Location Data Sharing and Exposure Notification
We perform a complete e2e flow including data sharing, publishing, and consumption of the same data leading to an Exposure Notification, on both iOS and Android devices.
This will be performed using the Staging version of Safe Places.
The general approach to e2e testing will be as described here: How to Test GPS End to End
Puerto Rican Spanish
The app will be operated in Puerto Rican Spanish mode, covering all of the screens documented here:
GPS Mobile App - Overview of Function
The tester need not have Spanish language skills, but they should be looking for:
Any text that appears to be not translated (still in English)
And text that is misaligned, wraps unexpectedly, or appears problematic otherwise.
General Regression Test
We perform this full regression test on both iOS and Android, but only on a single device of each type.
We follow the regression test plan defined in PractiTest:
However, these 52 tests are somewhat out of date, as they have not been maintained since they were written. Therefore testers should expect to update and adjust these test cases to fit current functionality.
There is no requirement to report test results in PractiTest if this is not convenient - for example the test cases can be exported, pasted into Confluence, and then annotated inline with test results.
Due to the desire to ship the release quickly, some long duration tests in the test plan may be skipped, or replaced with a shorter best-effort equivalent.
Schedule
Test builds for 1.2.0 have been available from early on Wednesday Eastern time.
Our goal is to push the release to the App Stores by close of business Thursday. Therefore our intention is to perform this testing on Wednesday and Thursday, concluding early afternoon in the US.
We are deliberately aiming to bring a broader set of testers into this test effort, to help broaden our testing capacity. As a result, test execution may be less tightly compressed than what we have been able to achieve on previous releases.
However, if we make slow progress on Wednesday, we will set all hands to the pump to complete the necessary testing in time for Thursday afternoon.
Known Defects & Untested Areas
This section provides an incomplete lists of some of the known defects and untested areas in the Mobile App.
The intention of this section is to highlight known quality shortcomings with the Mobile App that are believed to be acceptable for 1.2.0, and will not be raised by testers during QA.
Known issues
There are known to be text display issues on certain Android phones, certainly the OnePlus 7T, and possibly others.
Battery consumption is known to be higher than we would ike (up to 25% per day) on some iOS and Android devices
Location gathering is known not to be 100% reliable, though we believe it is at least 90% reliable on Android, and at least 80% reliable on iOS.
The app does not handle the condition in which Location services are disabled at a whole-device scope. In this case, the error messages and direction to the user are unhelpful.
We know that there are several issues wit Accessibility even at the WCAG A standard (and certainly at the AA standard).
Testing limitations
We have not tested the mobile app with Android versions < 7.0.
While we have tested a range of different devices, we have not yet tested the very broad range that we would want to test ahead of a broad community rollout. So there remains a risk of problems with certain specific makes / models of device.
Our Exposure Notification testing has only covered mainline cases so far. We have not tested a full range of edge conditions, we have not proven that JSON files are correctly cached etc.
We have performed limited testing with error conditions and adverse conditions, such as network unavailable, Airplane mode, Battery saver etc.
Scalability testing of the Mobile App has not yet been performed. In particular the performance of Exposure Notification calculations with 14d of data stored on the phone remains unknown.
Privacy and Security Testing of the Mobile App are still in progress, with various key areas of testing not yet covered.
Quick Links:
How to raise bugs found in Testing