Versions Compared

Old Version 11

changes.mady.by.user Diarmid Mackenzie

Saved on

compared with

New Version Current

changes.mady.by.user Diarmid Mackenzie

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Last updated by: Diarmid Mackenzie, Saturday 19 April, 5am EDTFriday 22 May

Testing Activity

OverallSafe Paths Android: v1.0 now delivered to tha App Store. This release is not without issues, and there will be a number of quick follow-up releases over the coming days.

  • See V1.0 App Store Release Candidate - Quality Assessment for the input I gave on this release - thanks to everyone who provided input

  • Vedha Kannan continues to manage volunteers working with the Test Case library in PractiTest.

  • PM will be laying out a pla for the next release & roadmap - should be opportunities for Testers to get involved in the Scrum team to discuss user storeis etc. in advance of delivery of a final build into Test.

Safe Paths iOS: We have still not got through Beta App Review, so we are limited in our TestFLight distribution.

  • Eran Kinsbruner has provided a private tenant for iOS testing. Contact Jonathon Wright if you would like access. Perfecto.io - Remote Mobile Testing (Automated & Manual)

  • Jonathan Wright can repackage / resign the IPA file to deploy to any iOS devices (just need peoples device IDs (uid) and apple ids). Onboarding (TiP)

  • Luca Dusi team of ~20 testers are testing iOS at the moment,

Safe Places

  • Adam Leon Smith has been developing an Test Plan:  Safe Places < this includes notes on how to contribute

  • We have had delays on the PQA Test team, that were going to work on this.

  • The PQA Test team may still come through, but in the meantime, there is a definite need for testers to work on Safe Places testing, as per Adam’s Test Plan - please help if you can!

Safe Paths Automation

...

  • Team from PQA Test has not started yet.

...

  • Adam Leon Smith is looking at a number of solutions for Test Data

    • His own Google location history.  5 years of data rebased into 86 different versions of March 2020.  He has tools to do this manipulation if you want to (see the test plan doc) 

      • If you are using your own long-term location history for testing, be mindful of privacy concerns, and keep the data to yourself (do not share with the project) unless you have explicit agreement otherwise.

    • He has a ML contact who can use this as seed data to generate many more similar trails.

      • Huw Price previously solved this problem for smart cities project, Jonathan Wright has reached out for the 5billion historical journeys previously generated (GPX format).

  • We have a lead on a tech that allows a user to program alternate GPS coordinates into their phone, to simulate being in different locations.  Could be a powerful enabler, as testers will be able to use test data from anywhere in the world.

    • Jonathon Wright is qualifying this opportunity. Working with Eran Kinsbruner around using the same stack we used to test smart city mobile apps (https://youtu.be/wuQt97rTa1Y)

  • Test Data / Mock HAs for Automation.  Problem that will need solving. Probably give to PQA Test as part of their Automation mission -

  • Diarmid is looking with Dev at whether we can bypass the complexities of the Ap’s interfaces on location data by isolating the relevant components of the App in a pure software harness, and simply feeding location data in directly. Test Strategy for Location Data Filtering / Presentation Algorithms. Would be keen to find technical testers keen to work on this - needs some programmind / data skills.

  • Security / Privacy

    • Privacy & Security - Current Status
      Pritesh Patel + colleagues from BioConnect are experts in this space & starting to pull together an overall strategy, based on the previous bullet as a starting point.

    • PM + Dev are now engaged with the importance of Privacy, and the need for a cleear plan to close existing gaps.

    • Dev are working with Salesforce, who have offered to

    • Adam Leon Smith has raised a feature request for a brief architectural deployment/security note on this for Safe Places

Real world testing

  • This is challenging in lockdown, but vitally important, and now urgent as we will soon have prospects doing their own real-rold testing & we need to be ahead of the prblems they are going to find.

  • See; 25 April 2020 - Real-world GPS test #1 for an example of some testing that has shown up a bunch of significant concerns. We need a more global spread on this, so anyone who would be able to conduct similar testing, please do so.

...

  • We are working on the MVP1 delivery of Safe Paths & Safe Places

  • This will ship to HAs on 1 June, but there will be more testing to be completed before it is ready for launch, which we will hope to do over the following week (including getting fixes for any problems).

  • Goal for 1 June should be that all features are code complete, and the basic function of the solution works.

  • In parallel, we have kicked off Project Aurora to build a GAEN Bluetooth app (without GPS). That will be coming into test soon as well.

  • Loads of activity around Privacy, Security & Transparency - we hope to have a robust public posiiton launched on all these points by 1 June.. Diarmid & Adam both heavily involved in this.


Safe Paths Mobile App:

  • Right now we are in a big Dev push for MVP1, but not much code to test yet. - we expect lots of new function to be coming into Test over the next week or so.

  • Follow this link to see what’s in MVP1.

  • There’s quite a few detaied items still to be specced, but some of the big changes coming soon, that will need significant testing. Lots of planning work to be done for these…

  • MVP1#1: Location lgging reliability

  • MVP1#2: Location lgging accuracy

  • MVP1#16: Secure transmission of location data to HAs.

  • MVP1#17: Scalability

  • MVP1#18: Moving points of concern from plain text to hashes

  • Other areas of testing that we can be driving forward in the meantime.

  • Regression testing - Dev is ongoing, and we have access to regular Dev builds via TestFlight & Google Beta. It is helpful for us to keep doing some testing with these. Defined set of test cases in PractiTest here, but not yet run through this: https://stage.practitest.com/p/5023/sets/26387/edit

  • Deeper testing - There are some existing functions that have not been very deeply tested yet. Some examples: Accessibility, early Android versions, some aspects of Security - talk to Diarmid if you are interested in exploring any of these. Some known gaps here: Test Session Charters

  • Test Automation. We have some automated test scripts from 21labs and Eggplant, and we are looking to extend these.

Safe Places

  • Adam Leon Smith has been developing an Test Plan:  Safe Places < this includes notes on how to contribute

  • We have a team of about 5 testers engaged in a mix of manual testign and automated tests (using Postman and Selenium).

  • Adam Leon Smith is test lead - talk to him if you have any questions.

Security / Privacy

  • Lots of progress going on here

  • Much of it at the level of principles / reqs / spec - which has fed into MVP1

  • A lot of the MVP1 features are intended to improve Privacy and Security.

  • Still some items where we are behind (e.g. a proper Threat Model, penetration testing etc.

  • Anyone with experience in these areas who is able to contribute - please talk to DIarmid or Adam..

Testing in Production

  • This is being managed by the UX group,

  • Ali Raizin is lead, please contact here if you;d like to help.

Haiti customer testing

  • We have an opportunity to work closely with a lead prospect in Hait, to define what testing they should do, and support them through it.

  • An important opportunity to shape our first customer deployment & ensure we

  • Nobody yet assigned to this - volunteers please contact Diarmid Mackenzie

Testing Community support

...

    • Way forward is expected to be primarily based on analytics from Safe Places rather than Safe Paths, for privacy reasons.

    • MVP1 story MVP1#19 covers some work in this area.

Testing Community support

  • Sherry Heinze is continuing to be a point of contact for any new testers who want some support working out what to work on; or for anyone who wants to find something different to work on. Sherry will be looking for new volunteers, and will probably reach out to you, but if she doesn’t, please do reach out to her. She’s on Mountain Time.

Key impediments for the teamThese include a raised date (baselined at 4/19), to provide some accountability in terms of resolving impediments quickly

Key issues right now.

  1. We need code to test! Loads of function coming in for MVP1 but not ready for test yet.

  2. Not enough volunteers. Although we are keeping up now , when the wave of MVP1 work + the GAEN BT app becomes ready for test, we will be struggling to keep up.

    1. It seems we have a large number of volunteers, who are only able to contribute a small amount of time.

    2. We’re not managing to create a wrokflow that works with this volunteer team, leading to poor distribution of work.

  3. Test Automation for Mobile clients - we are working on this with 21labs, to try to build a range automated regression tests.

Updates on previously reported items - mostly solved.

  1. (4/19) Too many gaps in documentation of requirements, and detailed product behaviour

    1. Diarmid hoping to drive something here, but anyone else who could take this on would be amazing.
      (Diarmid making slow progress with PM here, on varous threads)MVP1 spreadsheet here: https://docs.google.com/spreadsheets/d/1VTSnUOrfKBKXLkButvZ4B8MJzfV8gw-4JYv4lkG6XHA/edit#gid=0

    2. More detailes specs linked off there / Jira.

    3. ALso some good info on details in teh UI / UX space: https://pathcheck.atlassian.net/wiki/spaces/UIUX/overview

  2. (4/19) No available domain expert for Contact Tracing

    1. Kyle is working on a contact through AlinaIs one enough?  Or do we need more?has been collecting lots of info from HAs

    2. We have some info here How are Health Authorities actually going to use Safe Paths?

    3. We are still hoping to get an epidemiological adviser onboard within the project (with Kyle, I think).

  3. (4/19) No signatory to make a contract with Applause

    1. Du Tri will take it forward for Partnership to action - have pushed again to try to make this happenAll sorted - though we have not yet come up with a good use for Applause (Haiti field tiral not yet a priority).

  4. (4/19) No clear path for escalation of very high-level issues identified in Test - e.g. Privacy / Security concerns

    1. Diarmid now has a comms channel established with Sam, when neededAll resolved, and improved organization of the PM team means it’s much clearer where to go with specific issuesloads of good progress on Privacy / Security / Transparency / EThics, which shoudl all bear fruit in the MVP1 timeframe.

  5. (4/19) PractiTest not yet embedded as a tool.

    1. Limited test cases documented in PractiTest

    2. No clear patterns for using PractiTest for more exploratory forms of testing and for overall Requirements Tracing - not clear whether or not we want to invest in this, much of this is happening in Confluence today, and that may be good enough.

  6. (4/19) No current process for raising reqs/spec issues uncovered in Test

    1. We’ve been asked not to file these in GitHub issues.

    2. We believe we’ll be moving to Jira, but not quite ready yet

    3. We need to update guidance for testers.All covered in Jira: How to raise bugs found in Testing

  7. (4/19) It’s still proving difficult for even experienced testers to engage with this project, when they first arrive, despite us now having a broad set of resources now available in Confluence. We are tryng to get a better understanding of what else can we do. Are further resources neeed? What? Do we need to run in a much more explicitly directed manner? May Moorefield outreach + Lean Coffee session will hopefully provide some answers.

  8. ??? what else - please add your impediments here

...

    1. Sherry Heinze acting as community liaison. Feedback is that the resources available are mostly good (or at least OK). Reasons for individuals not engaging are highly mixed & personal.

Other useful docs:

Testing Resources

...