July 30 2020 - 12pm Eastern
Attendees: Diarmid, Bob (PM), John Sch, Matt B (Dev)
Diarmid presented early version of test strategy here: GAEN Test Strategy & Plan
Consensus on limited interpretation of project scope. But agreement that “evaluation of GAEN” intiative should be progressed in parallel as a research train - Diarmid to discuss with Sam.
Various challenges with build distribution
A given HD build will only enable certain languages
Unclear how to test customizability function. We’d like to test customization functionality genericaly, not just as used by a given HD - so we need a friendly HD that can create an extra APp version that we can use for testing.
Diarmid to reach out to Jeff/Jen for a siggestion (DONE: Jeff advised Guam).
Ramesh has started gathering physical phones
Nobody has a clear plan for what to do with these
John Sch would like a couple
Diarmid to consider testers who could take them - but concern that most testers are only contributing a few hours: hard to make good use of these.
We discussed making the phones available remotely. That requires some set-up, but might be useful - especially if they were in a known physical configuration. Still TBC exactly where they should be located, and do we need a server to connect in to them? (and if so, who provides & sets up). Jonathan Wright has built a similar setup with 4 phones for use by testers. That might be enough.
Overall - good to see testing starting to move forwards, but we need more clarity on how quickly we think we can progress from this point. Diarmid working on that with Vinay & others.
August 6 2020 - 1pm Eastern
Updates as follows:
Substantial updates to GAEN Test Strategy & Plan including a bunch more detail.
Plan for WinJit testing here: https://docs.google.com/spreadsheets/d/14T3jJDibuIOYob0HMcOCYpLM-HUo_ED4_9F7c3rBltA/edit?usp=sharing
Targetting 13 August “Tech Release” for MN, but not clear how real a deadline this is - awaiting update from Jen (PM) on this.
Current priorities for WInJit are:
OS levels
Device settings interactions
Accessibility
Server scale (not expecting to be done for 13 August)
Later plans:
Customization testing (when our plans are clarified)
More expensive testing with device settings / states interactions.
Usability (after UX rework)
Some key blockers for MN on Privacy/Security
Thomas Donnelly leading on this
Key items are:
DPIA (Thomas)
OWASP Top 10 (static analysis / pen testing (Diarmid)
Policies for how we handle incidents & vulnerabilities (Thomas to draft a proposal)
Clarify how we ensure the integrity of our software, given our broad community of contributors (Diarmid to draft materials; no process changes expected).
Composition Analysis (Diarmid) - kicking off conversations with WhiteSource.
For discussion (will update with further notes in-call).
We have some bugs raised now (mostly from Winjit). What flow do we want to use to handle these?
John S to Triage bugs & put them into the Trello board
We need a R4T status in the workflow - Diarmid/Stella.
Need to define how frequently builds will be pushed to Test. In 48 hours or so, we should get daily builds - Matt
Investigate moving iOS to AppCenter (Debug builds only; Release builds have to be loaded via TestFlight) - Matt
OK for some testing to work on Debug builds, as long as some testing happens on the Release builds before we ship - hence we still need some testers on TF. Current pattern is once a week on Thursdays.
Setting up new Guam environment for Pen testing (and probably also BugCrowd).
Who can help me with this?
Server set up
Apple TF & Google Test Group set-up.
Art & Sherif to help me with this. But uncertain whether it’s all going to work w/ GAEN entitlements.
Moving forwards with Whitesource…
Video on their free solution here: https://resources.whitesourcesoftware.com/product-videos/whitesource-bolt-for-github
Raises GitHub issues. Is that OK?
John to take a look at this.
Some specific technical queries:
Do we have an API spec for the GAEN servers (Verification & ENS)? I though Sherif shared a spec with me, but can’t find it. Needed for scale testing.
Matt to dig out Swagger doc.
How does the GAEN app interact with multiple user accounts on Android? Are keys, ENs etc. per-user or per-device?
Per-app install, exposures are unique
But keys are the same per-device
Do some testing & see where we stand. Our storage is Realm (per-App), rather than per-user.
Testability for Accessibility, customization, languages…
Very valuable to have an easy means to generate all possible screens in the app without having to drive full e2e flows.
Not supported today. Could be handled as a requirement.
Drive as a new requirement - drive conversation to agree reqs. John to create a Trello ticket.
Questions about language testing scope & responsibilities that also need to be characterized - Diarmid to consider further.
Scale testing the App’s local RPID store
Guam have offered to get hold of 20 or so phones, set them all up in close proximity, and leave for a week or more. Worth doing?
Edging into “research project” territory.
Probably worth doing
What’s the app actually exposed to here? (as opposed to the OS) - focus on this if possible.
Very large key files?
Connectivity problems with very large key files - restart etc.
Download processsing happens only max 10 times a day, and if it hits a failure will abort and only start again some time later.
Very large numbers of ENs (but hard to imagine this being more than 100 or so of these…)
Upcoming function changes…?
Customization - In App name now shipped. PM to decide what else we need to do for MN & others (colours etc.)
UX changes - Onboarding flow rework is now in the code. Will be in next build. Some other minor changes (e.g. an extra link).
Analytics - Just getting started, release plans etc., still to be defined.
What else? - Who owns QA for per-HD deliverables of customization? Should be QA for now…