Basic Android Testing

Owned by Diarmid Mackenzie
Last updated: Jun 24, 2020
6 min read

Similar to Basic iOS testing:

Basic iOS Testing

Testing Scope

Key aims of testing:

  • Make sure app works & looks good on a range of Android devices and Android version levels.

  • Look out for anything that looks unpolished, unfinished, hard to use.

  • Cover all mainline paths (e.g. include testing “Maybe later” options on permissions)

  • Build will point to a production YAML file with no HDs available, so there will be no upload of location data, or download of points of concern. Verify that user experience is sensible.

Restrictions on testing (due to nature of the build)

  • No validation of location recording function - with no HDs this data can’t be shared and will be inaccessible.

  • No testing of exposure notification logic, though we will review at the empty exposure notification screen.

  • Will try to test small & large screen devices - using real devices or devices in Perfecto Cloud. TBC what limits we can achieve.

Other restrictions:

  • No testing of error conditions, robustness conditions, stress testing of App - to follow in a later testing session, but lower priority.

Test Results #1 - Android 9 on Moto G7 Power

This is the device on which I’ve done most previous testing.

Problems identified

These are all problems that also exist on iOS, see: Basic iOS Testing or have been raised before, so keeping details here very light.

Problems in common with iOS

  • Languages not implemented yet - SAF-391

  • When you do change language, most screens remainin English - SAF-739

  • The “Health Departments” screen includes a “Manually Add via URL” option, which should not be present in a production build. SAF-732.

  • On “Terms of Use” screen during onboarding, “Continue” button is closer to the bottom of the screen than all other onboarding buttons. Minor inconsistency. SAF-733.

  • On “Legal” screen, privacy policy still points to an outdated Google Doc. SAF-695

  • Guidance to enable Location capabilities does not work if Location is disabled at device scope - SAF-239

  • Issues with Exposure History as per iOS Page. 2 bugs: SAF-735, SAF-736 & SAF-741.

  • Would be nice to add a link to where people can learn more about HDs that will be deploying soon (no bug raised).

  • Health Department screen - image means that important content is off the bottom of the screen, and it’s not even obvious that there is anything hidden - SAF-740.

Problems that are unique to Android - though not new.

  • App is still called “PathCheck BT” - SAF-681

  • Changing language is unreliable/random - SAF-554

  • When you say “Maybe later” on location during onboarding, the later flow to enable location is more complicated than the original flow that was skipped - SAF-701. (this may exist on iOS as well, but this would be blocked behind SAF-731).

  • App sometimes gets into hung state requiring re-install. SAF-710. Intermittently, if I leave my phone and come back to it, I will see the App does not start up, it just hangs on the opening splash screen. I have been seeing this on a few builds, but haven’t yet managed to isolate a clear reprot scenario. Latest is that this may be related to USB debugging, in which case we can live with for MVP1 App Store, for sure. I will continue to monitor.

New problems

  • There is no “Back” control on the “Select your Health Department” screen (and the Android Back button does not work either) - SAF-745

  • Version number is 1.0.8, rather than 1.1.0 (I believe this is known, due to the way the build was built, hence no bug raised yet)

Upgrade

  • I attempted to simulate upgrade, but it failed. SAF-744. ADB logcat suggests a version numbering issue (14 < 68).


Test Results #2 - Samsung Galaxy S6 - Android 7.0

Oldest Android OS version available on Perfecto.

  • As you might expect from a 360 x 640 screen, some of the problems seen on small iOS screens also occur on this device. - SAF-730.


Test Results #3 - OnePlus 7T - Android 10.0


”Started” mising on “Get Started” button. Very odd…! Also “t” is missing on “Next” button… and “phone” missing on “Set up my phone” button.

Note: Android 10 offers a new location control: “allow only when using the app”. There is something similar on iOS. Do we have function to detect and corrct when the user uses those settings? For investigation…

In Exposure screen, the “1” is missing form tiday’s date (“21”)

 

The “No Health Departments” text does not render, just the “refreshing” arrow. Does the device lack internet access, perhaps?

Unlike on Android 9, Back button twice does not exit from this screen. I was stuck, so I tried closing & re-opening the app. When I did that it got hung on this screen. Just like what I have seen in SAF-710, which I had started to speculate might be related to USB debugging. I guess devices in Perfecto labs are hooked up for USB debugging…?

What a mess! - I’m going to test with more Android 10 devices, to figure out if this is a device-specific problem or an OS problem.


Test Results #4 - Google Pixel 4 - Android 10.0


Let’s sum up the problems seen on OnePlus 7T ANdroid 10.0, and see what did / did not replicate on Google Pixel 4, Android 10.0.

  1. Truncated text on buttons.

  2. Location control “Allow only when using the app”.

  3. Missing digits on “Exposure” screen

  4. “No Health Departments” text does not render

  5. Pressing Back button twice does not exit from “Select your Health Department” screen.

  6. Hang when closing app on “Select your Health Department” screen after pressing back button twice, closing and re-opening the app.

On the Pixel phone, we did not see 1. or 3. See an example screenshot below.

We do see 2., and I did some testing around this: looks like we have no function to detect when a user chooses this option, and encourage them to do change it.

I did not see 4 the green spinner was replaced with the Health Departments.

I did see both 5. (back button pressed twice did not close the screen) and 6. the hang on app restart.

Here’s a screenshot showing a correctly rendered blue button….

So, let’s raise some bugs…

  • SAF-746 - Android 10 has fine-grained location controls (like iOS). PathCheck does not accommodate these yet.

  • Pressing Back button twice does not return to home screen on Android 10.0. Added as a note on SAF-745. Not a separate bug here as the key fix is to get a proper on-screen back button.

  • SAF-747 - App hangs on restart after viewing Health Departments screen.

  • In testing around SAF-747 on my Moto G7 Power, Android 9, I found anoher bug….

  • SAF-748 - Android 9 - app crashes when pressing back button (intermittent - occurs in bursts, then stops)

I’m going to test with the OnPlus 7T again before raising bugs for 1, 3 & 4.

 

 

 

 

Test Results #5 - OnePlus 7T - Android 10.0 - Revisited

I have worked with PErfecto to confirm that the display issues are a geneuine bug on this device (this covers 1 & 3).

https://pathcheck.atlassian.net/browse/SAF-758

Regarding 4, still not re-tested yet. Guessing a network connectivity issue, but we’ll see…

Test Results #6 - Armor X3 - Android 9

My daughter’s phone.

Screen was 360 x 720, which is the same as I have tested with, but she has Font size set to “Largest“ on her phone.

There were a lot of problems with the display as a result - raised as SAF-749. Screenshots are attached in that issue.

How far do we need to go with this “small screen” testing? Our requirements are not clearly defined here. SAF-750 raised (and assigned to Jen) with a proposed requirements definition.