(old) E2E Automation Thoughts

Owned by Diarmid Mackenzie
Last updated: Dec 17, 2020 by Ben St John
4 min read

Rough document exploring a possible approach here.

So far on the GAEN project we’ve not pushed forward on e2e test automation for the EN function.

The rationale here was twofold

  • Test phones for would need to be within BT range of each other. Most automation targets (e.g. in a Mobile Device Cloud) won’t be.

  • The overall elapsed time for the test would be ~24 hours due to the various delays involved in pushing keys, and pulling keys down again.

However, recent events have shown that it’s quite easy for us to regress this critical EN function, so a fast-feedback e2e test for this would be extremely valuable.

We have also been making good progress on Test Automation, both Mobile App (Appium) and Web (Selenium) on the GPS solution, and have a team of people working on both of these. But given teh priority of GAEN over GPS at the moment, it’s not clear we are putting our automation effort in the right place.

 

Can we do better? Here’s an idea… not quite an “end to end” test, but about 66% of the way there, and with no need for Bluetooth.

Our “e2e” test could:

  • Install the app and onboard on a single device (Appium)

  • The device generates its firstly daily TEK.

  • Get the device to output this TEK (debug option in the app) (Appium)

  • Gather a verification code from the verification server (Selenium)

  • Enter the code in the app, and report as infected.

  • After a suitable wait (TBC how long this needs to be)….

  • … go to the publically available URL where exposure keys are published

  • download the relevant zip file from this URL.

  • decode the binary key files, and confirm that the TEK is present in the file.


This test misses out a few things:

  • It does not test the BT flows that detect whether devices are in proximity. But as per above that is always going to be a challenge to test with automation.

  • It does not test that the app correctly downloads the keys in a reasonable timeframe, and matches. But to include this function in the test would add up to 4 hours elapsed time (because the app won’t download more frequently than that), and would only give any meaningful results if there had been a BT RPID exchange between the two phones, which is hard to engineer.

But it does cover the full onboarding and infection submission flow on one side, and also the full function of both the Verification and ENS Servers.

In terms of how much of the function the above e2e test covers, it’s worth noting that it would have caught 2 out of 2 recent problems hit with e2e EN flows at Guam.

 

Net of this, the above test:

  • Is something we can do fairly easily, in the near term, using technology we mostly have ready to go.

  • Would give us valuable and timely feedback on the main functionality of the GAEN app.


Even if it’s not a perfect e2e test, then it’s still probably worth implementing.

 

Key implementation questions (let’s try to ensure these are all resolvable before we start building…)

  • Which app build do we run on? This needs to be a build created for an HD partner with GAEN entitlements. Guam are probably a good bet here, as they are friendly and happy to work with us on things like this.

    • We are already looking at deploying a 2nd non-production GAEN server at Guam, which we will use for pen testing. Not 100% sure this can be achieved but we are trying to make it happen. This same server (or the same technique to create a 3rd server) could potentially be used to keep this test infra separate from Guam production.

  • Can we get access to the Verfication Server? Yes, just needs an email + password. Art can set this up.

  • Can we install the Guam GAEN App on both iOS and Android, in a Mobile Cloud like BorwserStack or SauceLabs?

    • No problem for Android.

    • TBC for iOS - may be issues with App signing (as we won’t know the UDID in advance), but this may not be a problem in a Mobile Cloud (e.g. Perfecto seemed to have re-signing technology to work around this).

  • Can we get visibility into the TEK allocated by the App? I think this is available as a debug option - need to check.

  • How long do we need to wait after reporting exposure, before we can check for presence of keys at the published location? Not sure - check with Dev for a view on this.

  • How do we decode the binary data file to search for the correct TEK? Not sure, but presumably this can be done - just need to understand the binary data format. Dev may be able to help…

  • How will this test be affected when we add the analytics server into the flow? TBC - needs input from people who understand plans there….