July 30 2020 - 12pm Eastern
Attendees: Diarmid, Bob (PM), John Sch, Matt B (Dev)
Diarmid presented early version of test strategy here: GAEN Test Strategy & Plan
Consensus on limited interpretation of project scope. But agreement that “evaluation of GAEN” intiative should be progressed in parallel as a research train - Diarmid to discuss with Sam.
Various challenges with build distribution
A given HD build will only enable certain languages
Unclear how to test customizability function. We’d like to test customization functionality genericaly, not just as used by a given HD - so we need a friendly HD that can create an extra APp version that we can use for testing.
Diarmid to reach out to Jeff/Jen for a siggestion (DONE: Jeff advised Guam).
Ramesh has started gathering physical phones
Nobody has a clear plan for what to do with these
John Sch would like a couple
Diarmid to consider testers who could take them - but concern that most testers are only contributing a few hours: hard to make good use of these.
We discussed making the phones available remotely. That requires some set-up, but might be useful - especially if they were in a known physical configuration. Still TBC exactly where they should be located, and do we need a server to connect in to them? (and if so, who provides & sets up). Jonathan Wright has built a similar setup with 4 phones for use by testers. That might be enough.
Overall - good to see testing starting to move forwards, but we need more clarity on how quickly we think we can progress from this point. Diarmid working on that with Vinay & others.
August 6 2020 - 1pm Eastern
Updates as follows:
Substantial updates to GAEN Test Strategy & Plan including a bunch more detail.
Plan for WinJit testing here: https://docs.google.com/spreadsheets/d/14T3jJDibuIOYob0HMcOCYpLM-HUo_ED4_9F7c3rBltA/edit?usp=sharing
Targetting 13 August “Tech Release” for MN, but not clear how real a deadline this is - awaiting update from Jen (PM) on this.
Current priorities for WInJit are:
OS levels
Device settings interactions
Accessibility
Server scale (not expecting to be done for 13 August)
Later plans:
Customization testing (when our plans are clarified)
More expensive testing with device settings / states interactions.
Usability (after UX rework)
Some key blockers for MN on Privacy/Security
Thomas Donnelly leading on this
Key items are:
DPIA (Thomas)
OWASP Top 10 (static analysis / pen testing (Diarmid)
Policies for how we handle incidents & vulnerabilities (Thomas to draft a proposal)
Clarify how we ensure the integrity of our software, given our broad community of contributors (Diarmid to draft materials; no process changes expected).
Composition Analysis (Diarmid) - kicking off conversations with WhiteSource.
For discussion (will update with further notes in-call).
We have some bugs raised now (mostly from Winjit). What flow do we want to use to handle these?
Setting up new Guam environment for Pen testing (and probably also BugCrowd).
Who can help me with this?
Server set up
Apple TF & Google Test Group set-up.
Moving forwards with Whitesource…
Video on their free solution here: https://resources.whitesourcesoftware.com/product-videos/whitesource-bolt-for-github
Raises GitHub issues. Is that OK?
Some specific technical queries:
Do we have an API spec for the GAEN servers (Verification & ENS)? I though Sherif shared a spec with me, but can’t find it. Needed for scale testing.
How does the GAEN app interact with multiple user accounts on Android? Are keys, ENs etc. per-user or per-device?
Testability for Accessibility, customization, languages…
Very valuable to have an easy means to generate all possible screens in the app without having to drive full e2e flows.
Scale testing the App’s local RPID store
Guam have offered to get hold of 20 or so phones, set them all up in close proximity, and leave for a week or more. Worth doing?
Upcoming function changes…?
Customization
UX changes
Analytics
What else?