...
August 24 2020 -
...
7am Eastern
Attendees: Diarmid, Bob (PM), John Sch, Matt B (Dev)
Diarmid presented early version of test strategy here: GAEN Test Strategy & Plan
Consensus on limited interpretation of project scope. But agreement that “evaluation of GAEN” intiative should be progressed in parallel as a research train - Diarmid to discuss with Sam.
Various challenges with build distribution
A given HD build will only enable certain languages
Unclear how to test customizability function. We’d like to test customization functionality genericaly, not just as used by a given HD - so we need a friendly HD that can create an extra APp version that we can use for testing.
Diarmid to reach out to Jeff/Jen for a siggestion (DONE: Jeff advised Guam).
Ramesh has started gathering physical phones
Nobody has a clear plan for what to do with these
John Sch would like a couple
Diarmid to consider testers who could take them - but concern that most testers are only contributing a few hours: hard to make good use of these.
We discussed making the phones available remotely. That requires some set-up, but might be useful - especially if they were in a known physical configuration. Still TBC exactly where they should be located, and do we need a server to connect in to them? (and if so, who provides & sets up). Jonathan Wright has built a similar setup with 4 phones for use by testers. That might be enough.
Overall - good to see testing starting to move forwards, but we need more clarity on how quickly we think we can progress from this point. Diarmid working on that with Vinay & othersGAEN-192/194 are still blockers, but we suspect an issue with APHL integration (Dave & Devs to meet with APHL later today)
Some introductory testing of OS security (report to follow from Vinay)
Next actions/priorities TBD after APHL meeting today
No known access issues among Winjit team
August 20 2020 - 1pm Eastern
Triage of bugs. John has spent some hours triaging bugs. Delegating some of this. Backlog of bugs is mostly acceptable, so they wiill stick around for a while.
Moving Dev to Jira. Project larger, more complex - integrate with ProdPad. Timescale TBD.
e2e - answers to questions posed in Diarmid’s e2e write-up.
Create a sub-team.
John to work with Dave to set this up.
Focus is to buld this e2e test.
Current issues with Winjit inability to submit keys
John setting up BugSnag
Improve error messages from server.
One key issue is that the Google server just is not designed to support multiple submissions from a single device.
Android - can clear out keys manually - make this part of our script.
Diarmid: Brief Winjit on this point…
iOS - problem that there is no way to clear out old keys, and submitting old keys causes problems.
There is a button “Delete Exposure Logs” but it seems not to do anything
Cloud iOS devics, emulated devices - can we come up with a way to blow away an iOS.
There may be issues submitting keys to our server:
When App A & App B are both on the phone at the same time
When App A is installed, then deleted & B is installed.
When A & B are both PathCheck hosted
When A & B are both PathCheck but maybe not hosted
When A are PathCheck & B are not.
And more…?
Worth articulating all the different cases, so we can get clear on the priorities.
This is particularly important for Guam, as they get a lot of visitors from the US.
Of concern but not top priority vs. other items…
August 13 2020 - 1pm Eastern
For discussion…
Upded ticket flow: still with Diarmid / Stella.
Whitesource progress - Diarmid to take a 1st look.
Error paths around diagnoss submission - questions
Just raise bugs & go from there…
Bug volume
Quite a few now
Do we need more process around this? Triage / prioritization?
With John S, but a bit overloaded.
Need a backfill for Bob Spectra/Sam/Adam discussing.
John to try to take a pass this afternoon.
EN problems on iOS-iOS and iOS-Android
Version naming? Some confusion. (0.0.1 (112) vs. 1.0.0 (12)).
Most recent pushed to MN is 0.0.1 (112).
Sounds like Guam is the anomaly.
Reason ENs not working at WinJit most likely to be due to recent server changes…
4 new Winjit testers to onboard
Diarmid to drive, but steps with Matt, Dick for iOS & Android.
UDIDs for all iOS devices - needed for new AppCenter distribution.
These are dripping in from Winjit.
Diarmid rolling off - David Runkle joining next week.
Easier navigation to all app screens for accessibility / language testing?
Cut as a Jira card - waiting to get to top of priority.
Focus still on base function & UI updates.
Pen testing with Guam account? Target is pen testing with Cobalt.io starting 18 August, but not clear whether we are on track. Also to be used for BugCrowd, I hope.
QA of HD customizations? Discussed last week - process definition needed?
Still discussion regarding scope etc.
Current differences are very small - e.g. external ink to privacy policy; is self-assessment in/out?
WIth current scope, should be within Dev capabilitiy to get stuff right.
No need for specific QA right now - revisit when updated scope is defined.
August 6 2020 - 1pm Eastern
Updates as follows:
Substantial updates to GAEN Test Strategy & Plan including a bunch more detail.
Plan for WinJit testing here: https://docs.google.com/spreadsheets/d/14T3jJDibuIOYob0HMcOCYpLM-HUo_ED4_9F7c3rBltA/edit?usp=sharing
Targetting 13 August “Tech Release” for MN, but not clear how real a deadline this is - awaiting update from Jen (PM) on this.
Current priorities for WInJit are:
OS levels
Device settings interactions
Accessibility
Server scale (not expecting to be done for 13 August)
Later plans:
Customization testing (when our plans are clarified)
More expensive testing with device settings / states interactions.
Usability (after UX rework)
Some key blockers for MN on Privacy/Security
Thomas Donnelly leading on this
Key items are:
DPIA (Thomas)
OWASP Top 10 (static analysis / pen testing (Diarmid)
Policies for how we handle incidents & vulnerabilities (Thomas to draft a proposal)
Clarify how we ensure the integrity of our software, given our broad community of contributors (Diarmid to draft materials; no process changes expected).
Composition Analysis (Diarmid) - kicking off conversations with WhiteSource.
...
We have some bugs raised now (mostly from Winjit). What flow do we want to use to handle these?
John S to Triage bugs & put them into the Trello board
We need a R4T status in the workflow - Diarmid/Stella.
Need to define how frequently builds will be pushed to Test. In 48 hours or so, we should get daily builds - Matt
Investigate moving iOS to AppCenter (Debug builds only; Release builds have to be loaded via TestFlight) - Matt
OK for some testing to work on Debug builds, as long as some testing happens on the Release builds before we ship - hence we still need some testers on TF. Current pattern is once a week on Thursdays.
Setting up new Guam environment for Pen testing (and probably also BugCrowd).
Who can help me with this?
Server set up
Apple TF & Google Test Group set-up.
Art & Sherif to help me with this. But uncertain whether it’s all going to work w/ GAEN entitlements.
Moving forwards with Whitesource…
Video on their free solution here: https://resources.whitesourcesoftware.com/product-videos/whitesource-bolt-for-github
Raises GitHub issues. Is that OK?
John to take a look at this.
Some specific technical queries:
Do we have an API spec for the GAEN servers (Verification & ENS)? I though Sherif shared a spec with me, but can’t find it. Needed for scale testing.
Matt to dig out Swagger doc.
How does the GAEN app interact with multiple user accounts on Android? Are keys, ENs etc. per-user or per-device?
Per-app install, exposures are unique
But keys are the same per-device
Do some testing & see where we stand. Our storage is Realm (per-App), rather than per-user.
Testability for Accessibility, customization, languages…
Very valuable to have an easy means to generate all possible screens in the app without having to drive full e2e flows.
Not supported today. Could be handled as a requirement.
Drive as a new requirement - drive conversation to agree reqs. John to create a Trello ticket.
Questions about language testing scope & responsibilities that also need to be characterized - Diarmid to consider further.
Scale testing the App’s Device’s local RPID store
Guam have offered to get hold of 20 or so phones, set them all up in close proximity, and leave for a week or more. Worth doing?
Edging into “research project” territory.
Probably worth doing
What’s the app actually exposed to here? (as opposed to the OS) - focus on this if possible.
Very large key files?
Connectivity problems with very large key files - restart etc.
Download processsing happens only max 10 times a day, and if it hits a failure will abort and only start again some time later.
Very large numbers of ENs (but hard to imagine this being more than 100 or so of these…)
Upcoming function changes…?
Customization - In App name now shipped. PM to decide what else we need to do for MN & others (colours etc.)
UX changes - Onboarding flow rework is now in the code. Will be in next build. Some other minor changes (e.g. an extra link).
Analytics - Just getting started, release plans etc., still to be defined.
What else? - Who owns QA for per-HD deliverables of customization? Should be QA for now…
July 30 2020 - 12pm Eastern
Attendees: Diarmid, Bob (PM), John Sch, Matt B (Dev)
Diarmid presented early version of test strategy here: GAEN Test Strategy & Plan
Consensus on limited interpretation of project scope. But agreement that “evaluation of GAEN” intiative should be progressed in parallel as a research train - Diarmid to discuss with Sam.
Various challenges with build distribution
A given HD build will only enable certain languages
Unclear how to test customizability function. We’d like to test customization functionality genericaly, not just as used by a given HD - so we need a friendly HD that can create an extra APp version that we can use for testing.
Diarmid to reach out to Jeff/Jen for a siggestion (DONE: Jeff advised Guam).
Ramesh has started gathering physical phones
Nobody has a clear plan for what to do with these
John Sch would like a couple
Diarmid to consider testers who could take them - but concern that most testers are only contributing a few hours: hard to make good use of these.
We discussed making the phones available remotely. That requires some set-up, but might be useful - especially if they were in a known physical configuration. Still TBC exactly where they should be located, and do we need a server to connect in to them? (and if so, who provides & sets up). Jonathan Wright has built a similar setup with 4 phones for use by testers. That might be enough.
Overall - good to see testing starting to move forwards, but we need more clarity on how quickly we think we can progress from this point. Diarmid working on that with Vinay & others.