GPS Mobile App Automation

Owned by Diarmid Mackenzie
Last updated: Aug 26, 2020 by Stella Nelson
5 min read

This page is no longer in use, for up to date meeting minutes please see here:

GPS Solution - Testing

24th August 2020 - 12pm EST/5pm BST

Note: From next week we will combine all GPS meetings into the Monday meeting.

18 August 2020 - 12pm Eastern

Attendees: Liz, Jeri, Stella, Dave, Diarmid

  • There’s been some progress on the tests, but mostly we talked about pivoting our automation efforts towards the GAEN app.

  • We seeem to have converged on BrowserStack as the place to test, since we have fewest impediments there (compared to SauceLabs & Perfecto).

  • There are some major challenges with e2e testign with GAEN, but Diarmid’s view is that can can build some tests that validate about 60% of the end-to-end flow, and that these will be more valuable to PathCheck right now than GPS App automation.

  • A lot of the building blocks we need for this are the same as what we have already developed.

  • Initial sketch of a plan here

  • (old) E2E Automation Thoughts

  • Next steps:

    • Liz to engage with Dev team, understand their build pipeline (GAEN is different from GPS) and figure out how to integrate

    • Jeri to get started with the GAEN app (probably on Android)m get familiar with it, and start with anything that’s not blocked on Liz’s work.

    • Diarmid to get Dev team (John Schoeman) to provide input on the proposed test automation.

  • Also: Diarmid is rolling off the project. Dave will be leading on teh GAEN app & hence this automation effort. Stella will still be around to help out as needed.

11 August 2020 - 12pm Eastern

Available test environments: updates from Stella:

“Hello world” Mobile test has made good progress (Liz):

  • Basic “hello world” integration is up with BrowserStack

  • For SauceLabs, open request still for webdriver.io - no useful support yet.

Tech stack (Liz)

  • Assertion library is Jasmine, similar to Jest.

  • Outstanding tooling choices: webdriver vs. nightwatch vs. testcafe vs. cypress other options.

  • Not a priority to have aligned stacks between mobile app & Safe Places, as they willl have a completely separate set of test cases.

  • We are using webdriver with Appium, as we don’t believe other options work with Appium. Choice still wide open for Web / Selenium.

Next steps on Mobile Automation:

  • Test succeeds but still returns an error code: this is step #1

  • step #2: get an iOS “hello world” working.

  • Not yet automated into pipeline - step #3.

  • Then we could begin implementing basic test cases.

  • Liz also suggests we look at WDIO image comparison service & native app compare - screen comparison tools. Easy to set up & powerful. Jeri will have a look at this over the next week.

This week:

  • Get Jeri up & running to match Liz’s level

  • Steps #1, #2 above: Android & iOS working

  • Start looking at pipeline integration & writing more tests.

Other items (Deven):

  • Z Attack Proxy (ZAP) - can analyze Selenium tests & spot any insecure practices on the fly.

  • ZAP HUD, feedback to developers in real-time.

  • Deven has some training materials Stella will take a look at this.

4 August 2020 - 12pm Eastern

Liz trying to get Appium running against Sauce Labs, but not yet working.

  • Stella is working on set-up with Sauce Labs.

  • Suspects that some of the automation pieces are not yet set up.

  • (also Perfecto is not working at the moment, but we hope it will come back).

  • Liz should route issues through Stella to get attention from SL - send errors etc.

With Web Driver, Perfecto has a Java driver, but no JS driver. SauceLabs has both.

  • We prefer to develop in JS for consistency with Dev team.

BrowserStack is another option Liz could look at (also Jonathon’s private device cloud if he can provide some docs on this).

We don’t have iOS build automatically generated at the moment. We’re not sure why not

  • Liz to focus on Android for now.

  • Then will look at fixing up iOS builds later.

“Hello world” test just checks whether the app is running.

  • Focus on running locally first - yarn install to set up environment

  • Sauce Connect tunnel is the only fiddly bit, currently a few manual steps. Liz looking at using a Docker container to streamline this. Some input from Sauce Labs would be useful here.

  • Then wire this into CI once it is running manually.

Goals for next week:

  • Hello world test, launch app, close app. With one of these cloud options (Perfecto, SL, BrowserStack or Jonathon’s setup).

  • Should work on both Windows & Mac.

  • Jeri to be able to run the tests as well (Mac)

  • Diarmid should be able to run it on Windows as well.

 

28 July 2020 - 12pm Eastern

  • Consensus that we won’t move forwards with Eggplant. Significant investment needed to fix up the model to match the latest app, and we don’t see much value in return.

  • 21labs also needs significant work to update models to work with the latest app. We think there is more value here, but we’re unsure whether we wan to move forwards with 21labs, or whether Native Appium would be better,

  • 21labs vs. Native Appium

    • 21labs is more accessible for folks who can’t code. But for people comfortable with coding, it’s slower, fiddlier, and harder to learn than coding with Appium (we suspect).

    • 21labs automatic maintenance of scripts is a neat feature, but if we design code for Appium scripts well, with good re-use of code, the problem it solves won’t manifest in the first place.

    • Currently, we have 3 people (Liz, Jeri, Kallie) all comfortable with coding, and happy to learn Appium - and have found 21labs frustrating, fiddly & slow.

    • We agreed to have a spike where we try to push forwards with test scripts in Appium directly (running against Perfecto). After a week, we’ll re-assess. If things go well we’ll likely stick with Appium. If they go badly, we may pivot back to 21labs.

    • Native Appium may run a bit faster than 21labs, which may help us with CI costs - see later point.

  • Plan for the week

    • Goal for next Tues (4 August) was to have a basic Appium script that we could run that gets us through onboarding.

    • (once we can do that, the rest of the app is just more button pushes, so should be pretty easy).

    • (we didn’t explicitly specify iOS or Android. Either would do. Both would be awesome!)

  • Other points: e2e testing

    • For e2e testing, we’ll need to figure out how to extract codes for location data sharing from Safe Places.

    • Various options:

      • Direct access to Safe Places Back End API - simplest if we can.

      • Using Appium + browser on the Mobile App to access Safe Places Front End.

      • Invoke some Selenium scripts (which will probably be developed anyway by the Selenium team)

    • Solving this problem is a priority for next week.

  • Other points: CI/CD

    • Once we have some working scripts, we’ll want to integrate into the CI/CD pipeline.

    • Let’s do this as soon as we have something of some minimal value.

    • TBD how often this runs. Bare minimum would be once a week, manually.

    • Ideally it would run much more frequently, potentially every build.

    • If the tests take a long time, this could increase out GitHub CI costs, as the VM that does the builds sits waiting for the results of the automated tests to return. This is a future challenge we will need to address if we want to get these tests running on a very frequent basis.