GAEN Testing Meeting Minutes

Owned by Diarmid Mackenzie
Last updated: Sept 02, 2020 by Dave Runkle (Deactivated)
6 min read

August 24 2020 - 7am Eastern

  • GAEN-192/194 are still blockers, but we suspect an issue with APHL integration (Dave & Devs to meet with APHL later today)

  • Some introductory testing of OS security (report to follow from Vinay)

  • Next actions/priorities TBD after APHL meeting today

  • No known access issues among Winjit team

August 20 2020 - 1pm Eastern

  • Triage of bugs. John has spent some hours triaging bugs. Delegating some of this. Backlog of bugs is mostly acceptable, so they wiill stick around for a while.

  • Moving Dev to Jira. Project larger, more complex - integrate with ProdPad. Timescale TBD.

  • e2e - answers to questions posed in Diarmid’s e2e write-up.

    • Create a sub-team.

      • John to work with Dave to set this up.

      • Focus is to buld this e2e test.

  • Current issues with Winjit inability to submit keys

    • John setting up BugSnag

    • Improve error messages from server.

    • One key issue is that the Google server just is not designed to support multiple submissions from a single device.

  • Android - can clear out keys manually - make this part of our script.

    • Diarmid: Brief Winjit on this point…

  • iOS - problem that there is no way to clear out old keys, and submitting old keys causes problems.

    • There is a button “Delete Exposure Logs” but it seems not to do anything

    • Cloud iOS devics, emulated devices - can we come up with a way to blow away an iOS.

  • There may be issues submitting keys to our server:

    • When App A & App B are both on the phone at the same time

    • When App A is installed, then deleted & B is installed.

    • When A & B are both PathCheck hosted

    • When A & B are both PathCheck but maybe not hosted

    • When A are PathCheck & B are not.

    • And more…?

    • Worth articulating all the different cases, so we can get clear on the priorities.

    • This is particularly important for Guam, as they get a lot of visitors from the US.

    • Of concern but not top priority vs. other items…



August 13 2020 - 1pm Eastern

For discussion…

  • Upded ticket flow: still with Diarmid / Stella.

  • Whitesource progress - Diarmid to take a 1st look.

  • Error paths around diagnoss submission - questions

    • Just raise bugs & go from there…

  • Bug volume

    • Quite a few now

    • Do we need more process around this? Triage / prioritization?

      • With John S, but a bit overloaded.

      • Need a backfill for Bob Spectra/Sam/Adam discussing.

      • John to try to take a pass this afternoon.

  • EN problems on iOS-iOS and iOS-Android

  • Version naming? Some confusion. (0.0.1 (112) vs. 1.0.0 (12)).

    • Most recent pushed to MN is 0.0.1 (112).

    • Sounds like Guam is the anomaly.

    • Reason ENs not working at WinJit most likely to be due to recent server changes…

  • 4 new Winjit testers to onboard

    • Diarmid to drive, but steps with Matt, Dick for iOS & Android.

  • UDIDs for all iOS devices - needed for new AppCenter distribution.

    • These are dripping in from Winjit.

  • Diarmid rolling off - David Runkle joining next week.

  • Easier navigation to all app screens for accessibility / language testing?

    • Cut as a Jira card - waiting to get to top of priority.

    • Focus still on base function & UI updates.

  • Pen testing with Guam account? Target is pen testing with Cobalt.io starting 18 August, but not clear whether we are on track. Also to be used for BugCrowd, I hope.

  • QA of HD customizations? Discussed last week - process definition needed?

    • Still discussion regarding scope etc.

    • Current differences are very small - e.g. external ink to privacy policy; is self-assessment in/out?

      • WIth current scope, should be within Dev capabilitiy to get stuff right.

    • No need for specific QA right now - revisit when updated scope is defined.

 

August 6 2020 - 1pm Eastern

Updates as follows:

For discussion (will update with further notes in-call).

  • We have some bugs raised now (mostly from Winjit). What flow do we want to use to handle these?

    • John S to Triage bugs & put them into the Trello board

    • We need a R4T status in the workflow - Diarmid/Stella.

    • Need to define how frequently builds will be pushed to Test. In 48 hours or so, we should get daily builds - Matt

    • Investigate moving iOS to AppCenter (Debug builds only; Release builds have to be loaded via TestFlight) - Matt

    • OK for some testing to work on Debug builds, as long as some testing happens on the Release builds before we ship - hence we still need some testers on TF. Current pattern is once a week on Thursdays.

  • Setting up new Guam environment for Pen testing (and probably also BugCrowd).

    • Who can help me with this?

      • Server set up

      • Apple TF & Google Test Group set-up.

      • Art & Sherif to help me with this. But uncertain whether it’s all going to work w/ GAEN entitlements.

  • Moving forwards with Whitesource…

  • Some specific technical queries:

    • Do we have an API spec for the GAEN servers (Verification & ENS)? I though Sherif shared a spec with me, but can’t find it. Needed for scale testing.

      • Matt to dig out Swagger doc.

    • How does the GAEN app interact with multiple user accounts on Android? Are keys, ENs etc. per-user or per-device?

      • Per-app install, exposures are unique

      • But keys are the same per-device

      • Do some testing & see where we stand. Our storage is Realm (per-App), rather than per-user.

    • Testability for Accessibility, customization, languages…

      • Very valuable to have an easy means to generate all possible screens in the app without having to drive full e2e flows.

        • Not supported today. Could be handled as a requirement.

        • Drive as a new requirement - drive conversation to agree reqs. John to create a Trello ticket.

        • Questions about language testing scope & responsibilities that also need to be characterized - Diarmid to consider further.

    • Scale testing the Device’s local RPID store

      • Guam have offered to get hold of 20 or so phones, set them all up in close proximity, and leave for a week or more. Worth doing?

        • Edging into “research project” territory.

        • Probably worth doing

        • What’s the app actually exposed to here? (as opposed to the OS) - focus on this if possible.

          • Very large key files?

          • Connectivity problems with very large key files - restart etc.

          • Download processsing happens only max 10 times a day, and if it hits a failure will abort and only start again some time later.

          • Very large numbers of ENs (but hard to imagine this being more than 100 or so of these…)

  • Upcoming function changes…?

    • Customization - In App name now shipped. PM to decide what else we need to do for MN & others (colours etc.)

    • UX changes - Onboarding flow rework is now in the code. Will be in next build. Some other minor changes (e.g. an extra link).

    • Analytics - Just getting started, release plans etc., still to be defined.

    • What else? - Who owns QA for per-HD deliverables of customization? Should be QA for now…

 

July 30 2020 - 12pm Eastern

  • Attendees: Diarmid, Bob (PM), John Sch, Matt B (Dev)

  • Diarmid presented early version of test strategy here: GAEN Test Strategy & Plan

  • Consensus on limited interpretation of project scope. But agreement that “evaluation of GAEN” intiative should be progressed in parallel as a research train - Diarmid to discuss with Sam.

  • Various challenges with build distribution

    • A given HD build will only enable certain languages

    • Unclear how to test customizability function. We’d like to test customization functionality genericaly, not just as used by a given HD - so we need a friendly HD that can create an extra APp version that we can use for testing.

    • Diarmid to reach out to Jeff/Jen for a siggestion (DONE: Jeff advised Guam).

  • Ramesh has started gathering physical phones

    • Nobody has a clear plan for what to do with these

    • John Sch would like a couple

    • Diarmid to consider testers who could take them - but concern that most testers are only contributing a few hours: hard to make good use of these.

    • We discussed making the phones available remotely. That requires some set-up, but might be useful - especially if they were in a known physical configuration. Still TBC exactly where they should be located, and do we need a server to connect in to them? (and if so, who provides & sets up). Jonathan Wright has built a similar setup with 4 phones for use by testers. That might be enough.

  • Overall - good to see testing starting to move forwards, but we need more clarity on how quickly we think we can progress from this point. Diarmid working on that with Vinay & others.