Testing of the PathCheck GPS Production iOS IPS file, on Perfecto Mobile (real devices, driven through a browser).
Testing Scope
Key aims of testing:
Make sure app works & looks good on a range of iOS devices and iOS version levels.
Look out for anything that looks unpolished, unfinished, hard to use.
Cover all mainline paths (e.g. include testing “Maybe later” options on permissions)
Build will point to a production YAML file with no HDs available, so there will be no upload of location data, or download of points of concern. Verify that user experience is sensible.
Restrictions on testing (due to nature of the build)
No validation of location recordig function - with no HDs this data can’t be shared and will be inaccessible.
No testing of exposure notification logic, though we will review at the empty exposure notification screen.
The smallest screen available to test on is the iPhone SE (2nd from left). I don’t have an iPhone 4 / 4S available to test with. It is likely that the iPhone 4 / 4S will experience additional issues with content size, that are not covered in this test plan.
Other restrictions:
No testing of error conditions, robustness conditions, stress testing of App - to follow in a later testing session, but lower priority.
Test Results #1 - iPhone SE on iOS 12.3.1
This is the iPhone with the smallest screen that I can test with. Though not as small as the iPhone 4 (which I don’t have available). I have separated out issues into those believed to be generic, and those that seem to be related to screen size.
Generic Issues identified:
Languages: only English & Haitian Creole offered. MVP1 scope includes Latin American Spanish, Tagalog, and Chamorro - No bug raised - I believe this is still under development, SAF-391 & child stories. But we need clarification on where we expect to get to on langauge for tha App Store launch.
Changing language to Haitian Crole after onboarding only changes the “More” screen - other screens remain in English - SAF-739.
Note that SAF-554, seen on Android, where the language picker itself often rolls back to English, was not seen on iOS.
Choosing “Maybe Later” option on Location Permissions does not work - I am still asked for Location Permissions anyway. SAF-731.
Choosing “Maybe Later” option on Notifications appears to work at first, but actually you still get asked for Notification Permissions right at the end of onboarding. SAF-731 (I documented in the same bug as it may well be the same issue - can raise a separate bug of preffered).
Having chosen “Maybe Later” option for Notifications, and then allowed them at the end of onboarding, I have managed to get into a state where the home screen has an “Allow Notifications” button, even though Notifications are already allowed in the App. SAF-738.
The “Health Departments” screen includes a “Manually Add via URL” option, which should not be present in a production build. SAF-732.
On “Terms of Use” screen, “Continue” button is closer to the bottom of the screen than all other onboarding buttons. Minor inconsistency. SAF-733.
On “Legal” screen, privacy policy still points to an outdated Google Doc. SAF-695 (this is the same issue raised against Android).
Unexpected navigation with the “About” and “Legal” screens: if you select More, then Legal, then Home, then More, you end up on the Legal screen, not the More screen. Similar to problems seen on Android. SAF-729, I added notes there.
See details below for Exposure History Page. 2 bugs: SAF-735 & SAF-736.
Possible enhancement / improvement: In a couple of places we report “Your phone is currently outside any areas covered by Health Departments”. That’s fine, but it would be even better to refer users to where they can find information about what areas are covered by PathCheck Health Departments, and where coverage might be coming soon - ideally a reference to such a page on our website if it exists. I’ve not raised a “bug” for this, but happy to raise a “bug” or “story” to track, if one doesn’t already exist.
Screen-size related issues
In “Enable Notifications” screen, English content overruns the space available: SAF-730. Other languages not yet tested (see above for why)
On Home screen, “Allow Notifications” button does not fit on the screen. SAF-734.
On the More screen, when changing languages, the language picker barely fits on the screen. SAF-737.
On the Health Departments screen, the image takes up a vast amount of limited real-estate. It’s an inconsistent design (no images elsewhere), and pushes the useful content off the bottom of the screen. Do we really want / need this image? SAF-740.
One positive bit of feedback on handling small screens: On the “Share your anonymized location history…” screen, the content doesn’t quite fit in the space available, but the screen is scrollable, and the button at the bottom is in the correct place. This design pattern could be a good solution for SAF-730, and maybe other points above.
Exposure History
Much improved from what I have seen on recent builds.
? icon by heading now moves to a Help Page
Legend not present (SAF-735). However better to have it not present, than present but with no pop-up.
Dot indicating today’s date is available.
Dates and days appear to be aligned correctly (I hope that’s not just a 1 in 7 chance!)
Dates are bold if & only if I have recorded location data for that date.
A remaining problem: the text for non-bold dates where I have no data seems to be incorrect - it is the same as the text I see for 21st where I do have data. I believe the text here should indicate “no data”. SAF-736.
For Dates in the future, there is no supporting text / commentary. The Figma design did not cover this case, so I am not certain what is “correct”, but that seems reasonable.
Minor consemtic issue: dingle digit dates are renedered as “jun 07” where I think “Jun 7” would be more normal. SAF-741.
Test Results #2 - More iOS devices to follow
Having tested with the small-screen iPhone SE already, I’m not expecting to find many more device-specific problems - larger screens ought to be less problematic - but further testing will follow and I’ll add notes here.
But publishing this now, so we have an initial set of test reporting to share.