Basic iOS Testing

Testing of the PathCheck GPS Production iOS IPS file, on Perfecto Mobile (real devices, driven through a browser).

Testing Scope

Key aims of testing:

• Make sure app works & looks good on a range of iOS devices and iOS version levels.

• Look out for anything that looks unpolished, unfinished, hard to use.

• Cover all mainline paths (e.g. include testing “Maybe later” options on permissions)

• Build will point to a production YAML file with no HDs available, so there will be no upload of location data, or download of points of concern. Verify that user experience is sensible.

Restrictions on testing (due to nature of the build)

• No validation of location recordig function - with no HDs this data can’t be shared and will be inaccessible.

• No testing of exposure notification logic, though we will review at the empty exposure notification screen.

• The smallest screen available to test on is the iPhone SE (2nd from left). I don’t have an iPhone 4 / 4S available to test with. It is likely that the iPhone 4 / 4S will experience additional issues with content size, that are not covered in this test plan.
  
  

  Open

  

Other restrictions:

• No testing of error conditions, robustness conditions, stress testing of App - to follow in a later testing session, but lower priority.

Test Results #1 - iPhone SE on iOS 12.3.1

This is the iPhone with the smallest screen that I can test with. Though not as small as the iPhone 4 (which I don’t have available). I have separated out issues into those believed to be generic, and those that seem to be related to screen size.

Generic Issues identified:

• Languages: only English & Haitian Creole offered. MVP1 scope includes Latin American Spanish, Tagalog, and Chamorro - No bug raised - I believe this is still under development, SAF-391 & child stories. But we need clarification on where we expect to get to on langauge for tha App Store launch.

• Changing language to Haitian Creole after onboarding only changes the “More” screen - other screens remain in English - SAF-739.

  • Note that SAF-554, seen on Android, where the language picker itself often rolls back to English, was not seen on iOS.

• Request for Notifications has a mis-spelling: “receive” spelled as “recieve”(twice). SAF-742.

• Choosing “Maybe Later” option on Location Permissions does not work - I am still asked for Location Permissions anyway. SAF-731.

• Choosing “Maybe Later” option on Notifications appears to work at first, but actually you still get asked for Notification Permissions right at the end of onboarding. SAF-731 (I documented in the same bug as it may well be the same issue - can raise a separate bug of preffered).

• Having chosen “Maybe Later” option for Notifications, and then allowed them at the end of onboarding, I have managed to get into a state where the home screen has an “Allow Notifications” button, even though Notifications are already allowed in the App. SAF-738.

• The “Health Departments” screen includes a “Manually Add via URL” option, which should not be present in a production build. SAF-732.

• On “Terms of Use” screen, “Continue” button is closer to the bottom of the screen than all other onboarding buttons. Minor inconsistency. SAF-733.

• On “Legal” screen, privacy policy still points to an outdated Google Doc. SAF-695 (this is the same issue raised against Android).

• See details below for Exposure History Page. 3 bugs: SAF-735, SAF-736 & SAF-741

• Possible enhancement / improvement: In a couple of places we report “Your phone is currently outside any areas covered by Health Departments”. That’s fine, but it would be even better to refer users to where they can find information about what areas are covered by PathCheck Health Departments, and where coverage might be coming soon - ideally a reference to such a page on our website if it exists. I’ve not raised a “bug” for this, but happy to raise a “bug” or “story” to track, if one doesn’t already exist.

Screen-size related issues

• In “Enable Notifications” screen, English content overruns the space available: SAF-730. Other languages not yet tested (see above for why)

• On Home screen, “Allow Notifications” button does not fit on the screen. SAF-734.

• On the More screen, when changing languages, the language picker barely fits on the screen. SAF-737.

• On the Health Departments screen, the image takes up a vast amount of limited real-estate. It’s an inconsistent design (no images elsewhere), and pushes the useful content off the bottom of the screen. Do we really want / need this image? SAF-740.

• One positive bit of feedback on handling small screens: On the “Share your anonymized location history…” screen, the content doesn’t quite fit in the space available, but the screen is scrollable, and the button at the bottom is in the correct place. This design pattern could be a good solution for SAF-730, and maybe other points above.

Exposure History

Open

• Much improved from what I have seen on recent builds.

• ? icon by heading now moves to a Help Page

• Legend not present (SAF-735). However better to have it not present, than present but with no pop-up.

• Dot indicating today’s date is available.

• Dates and days appear to be aligned correctly (I hope that’s not just a 1 in 7 chance!)

• Dates are bold if & only if I have recorded location data for that date.

• A remaining problem: the text for non-bold dates where I have no data seems to be incorrect - it is the same as the text I see for 21st where I do have data. I believe the text here should indicate “no data”. SAF-736.

• For Dates in the future, there is no supporting text / commentary. The Figma design did not cover this case, so I am not certain what is “correct”, but that seems reasonable.

• Minor consemtic issue: dingle digit dates are renedered as “jun 07” where I think “Jun 7” would be more normal. SAF-741.
  
  
  

Test Results #2 - More iOS devices to follow

Having tested with the small-screen iPhone SE already, I’m not expecting to find many more device-specific problems - larger screens ought to be less problematic - but further testing will follow and I’ll add notes here.

But publishing this now, so we have an initial set of test reporting to share.

iPhone XS Max, iOS 13.2

Largest screen iPhone (obviously iPads are bigger!)

• No new bugs revealed by testing on this device.

• SAF-733 does not manifest on this device. On the XS Max, the “Continue” button alignment on “Terms of Use” screen, matches other buttons on other screens.

iPad Pro 11, iOS 13.4.1

Largest screen of all.

App renders in portrait mode, with a large black border, regardless of device orientation:

In spite of the massive screen, the App experiences problems seen on smaller screens, similar to iPhone 6 - e.g. SAF-733, SAF-730

Because the link to the Privacy Policy launches a different App, it ends up in Landscape mode, not Portait mode (also Safari on this device can’t access Google docs, not sure why that is). The same happens with other browser links, e.g. links to covidsafepaths.org. (I suspect this depends on the rotation lock settings on the device - with rotation lock off, the browser will probably render in portrait mode).

“About Screen” shows this is rendered as a 375 x 667 screen, same as an iPhone 7 or 8 - see above.

Summary: some odditiesm, but as long as we are happy that it has the appearance of a Phone App running on an iPad, nothing that seems like a must-fix issue.

iPhone 7 Plus, iOS 12.2

Didn’t exspect to see anything special with this one. However, by chance I picked a device on which device-level Location services had been disabled. As a result, this is what I saw….

• During onboarding, I requested to enable Location. But I didn’t get the expected pop-up - I went straight through to a home screen, asking me to enable Location Access.
  
  

  
  It appears this device is hard-coded to not allow location access, and I don’t even have permissions to change that in the Settings (greyed out).
  
  

In device Settings, under “Privacy”, overall location services were turned off: