Testing in Production (TiP)

Goals

Testing in production is an important core competency for mitigating risks that it exposes.

The following page will document provides context, use cases and how to get involved.

Use Cases / KPIs

How many actually complete set-up?

Utilizing Firebase: Analytics > Conversions Event to track event cadence for the app 'onboarding process' during startup (not including meta data unless opt-in).
How many turn on location data

Unable to capture telemetry / instrumentation meta data without implementing ‘set user properties’

firebase.analytics().setUserProperties({location_data: TRUE});
How many subscribe to an HA?

Utilizing Firebase: Analytics > Conversions Event to track event cadence for the ‘subscribe to HA' (not including meta data unless opt-in)
How many open the app after they install it?

Utilizing Firebase: Analytics > Retentions to track high level ‘engagement’.
How many get an alert that they may be infected?

Utilizing Firebase (FCM) Cloud / In-App Messaging: Push notifications would need to be implemented.
How many location data points does their app log

Depending on the device this calculation can be made during the export function (then passed to custom user properties).
How many infection data points do they have from HAs they are subscribed to.

Unable to track this information from device (PII / GDPR)
How is performance is effected by network location (NV)

Utilizing Firebase: Quality > Performance we can look at both Network Response Latency (NRL) and Device Performance (Duration Traces). Custom Traces (CPU, Memory, Device Attributes) and Data Aggregation (Network / URL).
How can we A/B test / Canary Rollout

Utilizing Firebase: Audiences can be defined with a subset of user behaviors that can be flagged for rollout.

Epic / User Story

Target release	MVP
Epic / User Story	https://pathcheck.atlassian.net/browse/TEST-7
Document status	IN PROGRESS
Document owner	Jonathon Wright
Designer
Tech lead
Technical writers
QA

Task / Work in Progress (WIP)

Task / Goal / WIP	Status
Provide Access to TestFlight	https://pathcheck.atlassian.net/browse/TEST-3
Provide Access to Remote Cloud Devices	https://pathcheck.atlassian.net/browse/TEST-4
Provide Access to Local Mobile Devices	https://pathcheck.atlassian.net/browse/TEST-5

Apple Build (iOS / IPA) - COVID Safe Paths

Check out the pre-requisites section for any dependencies.

Pre-requisites

Clone the GIT repositories

Download the latest IPA / APK files

https://github.com/tripleblindmarket/covid-safe-paths/releases/tag/v0.9.4

Background

1. Make layers – like a stack of pancakes

The idea of “testing in production” can actually mean different things. Are you testing a bunch of test servers from within your production data center? Or are your test applications running separately on top of your production platform? Or are you truly running live tests against 100% production-deployed code? The answer should be all of these. Layer your production testing to give you the ability to test different aspects of the production environment in different ways. Then match up your test cases so as to minimize the impact that your testing – and maintenance of the test environment – has on production users.

2. Time your tests when usage is light

Non-functional testing can have an impact on your entire user base if you let it. It can make a server environment sluggish, and that’s something no one wants. Study your analytics and determine the best time to schedule your tests. For example, look for the lowest levels of:

Number of users on the site
Resource-intensive processes within the environment

3. Collect real traffic data and replay it to your test systems

Make sure to use actual traffic data you have been collecting in production (such as user workflows, user behavior and resources) to drive the generation of load for test cases. That way when you exercise your tests within your production environment, you’ll have confidence that the simulated behavior is realistic.

4. Introduce a chaos monkey

According to Netflix engineers Cory Bennett and Ariel Tseitlin, “The best defense against major unexpected failures is to fail often. By frequently causing failures, we force our services to be built in a way that is more resilient.” Netflix built what’s called a Chaos Monkey into their production environment. This code actually introduces failures into the production environment randomly, forcing engineers to design recovery systems and develop a stronger, more adaptive platform. You can put your own chaos monkey in place because Netflix released their code to GitHub.

5. Monitor like crazy

When you are running a production test, keep your eye on key user performance metrics so that you know if the test is having any kind of unacceptable impact on the user experience. Be prepared to shut the test down if that’s the case.

6. Create an “Opt-in” experience for experimental testing

A great way to test how your application performs with real users is to have some “opt-in” to new feature releases. This will allow you to monitor and collect data from real time users and make adjustments to your testing strategy accordingly, without as much concern about impacting their experience. After all – they’ve already agreed to become test subjects, so a little hiccup here and there won’t come as a surprise.

Open Questions

Question	Answer	Date Answered