...
August 13 2020 -
...
1pm Eastern
...
Attendees: Diarmid, Bob (PM), John Sch, Matt B (Dev)
...
Diarmid presented early version of test strategy here: GAEN Test Strategy & Plan
...
Consensus on limited interpretation of project scope. But agreement that “evaluation of GAEN” intiative should be progressed in parallel as a research train - Diarmid to discuss with Sam.
...
Various challenges with build distribution
A given HD build will only enable certain languages
Unclear how to test customizability function. We’d like to test customization functionality genericaly, not just as used by a given HD - so we need a friendly HD that can create an extra APp version that we can use for testing.
Diarmid to reach out to Jeff/Jen for a siggestion (DONE: Jeff advised Guam).
...
Ramesh has started gathering physical phones
Nobody has a clear plan for what to do with these
John Sch would like a couple
Diarmid to consider testers who could take them - but concern that most testers are only contributing a few hours: hard to make good use of these.
We discussed making the phones available remotely. That requires some set-up, but might be useful - especially if they were in a known physical configuration. Still TBC exactly where they should be located, and do we need a server to connect in to them? (and if so, who provides & sets up). Jonathan Wright has built a similar setup with 4 phones for use by testers. That might be enough.
...
For discussion…
Upded ticket flow: still with Diarmid / Stella.
Whitesource progress - Diarmid to take a 1st look.
Error paths around diagnoss submission - questions
Just raise bugs & go from there…
Bug volume
Quite a few now
Do we need more process around this? Triage / prioritization?
With John S, but a bit overloaded.
Need a backfill for Bob Spectra/Sam/Adam discussing.
John to try to take a pass this afternoon.
EN problems on iOS-iOS and iOS-Android
Version naming? Some confusion. (0.0.1 (112) vs. 1.0.0 (12)).
Most recent pushed to MN is 0.0.1 (112).
Sounds like Guam is the anomaly.
Reason ENs not working at WinJit most likely to be due to recent server changes…
4 new Winjit testers to onboard
Diarmid to drive, but steps with Matt, Dick for iOS & Android.
UDIDs for all iOS devices - needed for new AppCenter distribution.
These are dripping in from Winjit.
Diarmid rolling off - David Runkle joining next week.
Easier navigation to all app screens for accessibility / language testing?
Cut as a Jira card - waiting to get to top of priority.
Focus still on base function & UI updates.
Pen testing with Guam account? Target is pen testing with Cobalt.io starting 18 August, but not clear whether we are on track. Also to be used for BugCrowd, I hope.
QA of HD customizations? Discussed last week - process definition needed?
Still discussion regarding scope etc.
Current differences are very small - e.g. external ink to privacy policy; is self-assessment in/out?
WIth current scope, should be within Dev capabilitiy to get stuff right.
No need for specific QA right now - revisit when updated scope is defined.
August 6 2020 - 1pm Eastern
...
We have some bugs raised now (mostly from Winjit). What flow do we want to use to handle these?
John S to Triage bugs & put them into the Trello board
We need a R4T status in the workflow - Diarmid/Stella.
Need to define how frequently builds will be pushed to Test. In 48 hours or so, we should get daily builds - Matt
Investigate moving iOS to AppCenter (Debug builds only; Release builds have to be loaded via TestFlight) - Matt
OK for some testing to work on Debug builds, as long as some testing happens on the Release builds before we ship - hence we still need some testers on TF. Current pattern is once a week on Thursdays.
Setting up new Guam environment for Pen testing (and probably also BugCrowd).
Who can help me with this?
Server set up
Apple TF & Google Test Group set-up.
Art & Sherif to help me with this. But uncertain whether it’s all going to work w/ GAEN entitlements.
Moving forwards with Whitesource…
Video on their free solution here: https://resources.whitesourcesoftware.com/product-videos/whitesource-bolt-for-github
Raises GitHub issues. Is that OK?
John to take a look at this.
Some specific technical queries:
Do we have an API spec for the GAEN servers (Verification & ENS)? I though Sherif shared a spec with me, but can’t find it. Needed for scale testing.
Matt to dig out Swagger doc.
How does the GAEN app interact with multiple user accounts on Android? Are keys, ENs etc. per-user or per-device?
Per-app install, exposures are unique
But keys are the same per-device
Do some testing & see where we stand. Our storage is Realm (per-App), rather than per-user.
Testability for Accessibility, customization, languages…
Very valuable to have an easy means to generate all possible screens in the app without having to drive full e2e flows.
Not supported today. Could be handled as a requirement.
Drive as a new requirement - drive conversation to agree reqs. John to create a Trello ticket.
Questions about language testing scope & responsibilities that also need to be characterized - Diarmid to consider further.
Scale testing the Device’s local RPID store
Guam have offered to get hold of 20 or so phones, set them all up in close proximity, and leave for a week or more. Worth doing?
Edging into “research project” territory.
Probably worth doing
What’s the app actually exposed to here? (as opposed to the OS) - focus on this if possible.
Very large key files?
Connectivity problems with very large key files - restart etc.
Download processsing happens only max 10 times a day, and if it hits a failure will abort and only start again some time later.
Very large numbers of ENs (but hard to imagine this being more than 100 or so of these…)
Upcoming function changes…?
Customization - In App name now shipped. PM to decide what else we need to do for MN & others (colours etc.)
UX changes - Onboarding flow rework is now in the code. Will be in next build. Some other minor changes (e.g. an extra link).
Analytics - Just getting started, release plans etc., still to be defined.
What else? - Who owns QA for per-HD deliverables of customization? Should be QA for now…
July 30 2020 - 12pm Eastern
Attendees: Diarmid, Bob (PM), John Sch, Matt B (Dev)
Diarmid presented early version of test strategy here: GAEN Test Strategy & Plan
Consensus on limited interpretation of project scope. But agreement that “evaluation of GAEN” intiative should be progressed in parallel as a research train - Diarmid to discuss with Sam.
Various challenges with build distribution
A given HD build will only enable certain languages
Unclear how to test customizability function. We’d like to test customization functionality genericaly, not just as used by a given HD - so we need a friendly HD that can create an extra APp version that we can use for testing.
Diarmid to reach out to Jeff/Jen for a siggestion (DONE: Jeff advised Guam).
Ramesh has started gathering physical phones
Nobody has a clear plan for what to do with these
John Sch would like a couple
Diarmid to consider testers who could take them - but concern that most testers are only contributing a few hours: hard to make good use of these.
We discussed making the phones available remotely. That requires some set-up, but might be useful - especially if they were in a known physical configuration. Still TBC exactly where they should be located, and do we need a server to connect in to them? (and if so, who provides & sets up). Jonathan Wright has built a similar setup with 4 phones for use by testers. That might be enough.
Overall - good to see testing starting to move forwards, but we need more clarity on how quickly we think we can progress from this point. Diarmid working on that with Vinay & others.