25 April 2020 - Real-world GPS test #1

This was my 1st real world test - it was intended to be a simple baseline to use as a foundation for future tests. It turns out that even this simple test was a little too ambitious for where the tech is right now.

The plan was:

3 phones, each in a separate pocket
Synchronized logging to reduce risk of problems with non-synchronization + movement (see #516)
(I hoped to achieve this syncrhonization by disabling/re-enabling location tracking on all 3 phones at the same time at the start of the test)
Walk around my local area for about 40 mins.
Analyze the location data, and see how close the GPS stayed.

In a perfect world, our algorithms would detect a 40 minute duration match for each pair of phones. I knew there were risks of problems if the phones GPS pings drifted out of sync (because I was moving, as per #516 concerns). I was also aware there were risks of their GPS signals not being in alignment with each other even when they did ping at the same time.

In detail, here’s what I did:

I had 3 phones, all equipped with SIM cards and Mobile service with Wind-3.
All have COVID Safe Paths Installed
A is a Mototola G7 Power (Android 9)
B is Samsung Galaxy J6 (Android 9)
C is a Redmi 6A (Android 8.1)
At 6:52 (+/- a few seconds), I briefly disabled, then re-enable location tracking on each phone using the Safe Paths in-app control.
I then went for a walk, about 10 mins in one direction, then back, then a 20 minute loop in another direction.
The first 20 minutes was fairly tree-covered (including a section in quite deep woods). The 2nd 20 minutes was deliberately out in the open air.
At the end, I exported the JSON data from each app and analyzed the results.
This is in rural Italy.

Results - GPS ping timings

Ping timestamps for the phone were as follows. These are in seconds, relative 6:52,which is approximately when I turned location tracking on/off.

What do we see?

No strong evidence that disabling / enabling location settings has any impact on the ping timings
Reasonable synchronization in the next ping, 5 minutes later, so maybe it did have an impact
Of the 3 phones, only A pinged reliably every ~5 minutes through the test.
This is despite the fact that for the 2nd half of the test, approx t = 1200 to 2400, I was out of the woods, in the open, with good all round visibility to the sky

Because I got no data in the 2nd half od the walk, despite being in the open, I suspect a software problem is behind the absence of pings for B & C. Next step would be to run some more tests with anoter app used as a Control (e.g. Google Maps).

Results - GPS postions

Given the small set of data points actually obtained from the phones, I wasn’t able to do as full an analysis of the GPS position data.

I had hoped to have multiple synchronized pings from the phones as I walked around, but there were few of those. But here’s the analysis of the data I have. I have lined up rows of data with the closest ping times, hence the gaps in teh data for the 4-5 pings that A made, while B & C were silent.

DIstance in degrees uses basic Pythagoras Theorem to calulate a direct distance. My conversion to metres uses a very crude 1 degree = 100km.

The first row is data from befre I started the walk (Row 2 in the timestamps table above). I was stationary & indoors while I was setting up the phones, hence the different time intervals won’t have made much difference.

A and C were close enough to have triggered a Safe Paths Match (< 20m), but B was a long way off.

The 2nd row is the closest in time pings we had - all within 18 seconds of each other. I was walking at a gentle pace, so in 18 seconds, I may have covered 20m (~4km/h). A and were aligned on position (although B & C were actually the closest in terms of when the pings occurred). This time, C seems to have been the outlier.

The final set of pings came when I had finished the walk & was inside again. So the times are off by a lot, but that shouldn’t matter too mcuh as I was stationary. B & C were close. A was just over 20m away, but given the ping was also out by 200 seconds, maybe it was genuinely taken from a slightly different position, perhaps just before I came indoors.

To the left, I have includes here’s a map of the overwall walk, relative to the start position.

Units are degrees, so each square (0.001 degrees) is approximately a 100m x 100m square.

What have I learned?

Key learnings:

Reliable generation of GPS location pings seems to be a major issue. This is a major concern in terms of he service itself, but it’s also going to be a major impediment to testing. We must resolve this as a top priority.
Consistency of GPS positioning for phones in the same place looks questionable. However it’s hard to evaluate this in a test where I am moving, and phones are not pinging at the same time.
A control would be very useful in evaluating the data. I will look at running Google Maps in parallel, to distinguish “no GPS signal” cases from cases where there is a viable GPS signal, but the software is nevertheless not logging anything.
Assuming we can get to the point where we have consistent (to within ~10-20m) location data from multiple phones inthe same place and the same time, then concerns about missing matches due to non-synchronized GPS pings (#516) is actually very real. Whereas I phrased #516 in terms of a pathological case with 2 phones in exact antiphase, the reality is that 1m/second is a gentle walking pace, so if you are walking with someone, then your location pings actually have to be synchronized to within 20 seconds of each other, or you won’t get a match. The odds of 2 x 5 minute timers being within 20 sceonds of each other are ~10%.
I was over-optimistic & I tried to run before I could walk in terms of testing. Or more specifically, I tried to walk before I could stand still…!

What do I not yet know?

Key things I don’t yet know and I am keen to find out:

Does disabling & re-enabling location data in the app actually reset the GPS ping cycle & therefore synchronize the timers? Not yet clear.
How cosistent will the location data be for 2 phones in the same place?
Does that location data become more consistent, if the 2 phones ping more closely in time together?
What is a good approach to use to provide a “control” for GPS data? Tracing location data with google in parallel? Will doing so interfere with my test results (e.g. by “priming” the GPS receiver in some way?
Do services like Google Maps, Pokemon Go & Geocaching use some sort of noise reducation or averaging techniques to create a higher quality GPS signal? Are we being naive in simply assuming the raw GPS data we get back from the API is good enough for this service?
Is there already established expertise on this topic that I am unaware of, and therefore failing to take advantage of.
How representative is my geographical context, of other contexts in which we might deploy? What will be the impacts of latitude, cities, weather etc. etc.

What Next?

Share this Test Report - I think there are some very improtant questions raised here, which are threats to the viability of this service. If we don’t resolve these, we will immediately be exposed in field trials.

Try to get answers from Dev on a couple of key questions

Does disabling & re-enabling location data in the app actually reset the GPS ping cycle & therefore synchronize the timers? If not, is there another way to achieve this effect?
Do services like Google Maps, Pokemon Go & Geocaching use some sort of noise reducation or averaging techniques to create a higher quality GPS signal? Are we being naive in simply assuming the raw GPS data we get back from the API is good enough for this service?

Conduct further testing focussed on GPS locations of phones in static positions

Indoord & outdoors
GPS pings synchronized & non-syncrhonized

Experiment with using “GPS controls” such as Google Maps, to generate data that I can use to validate the data generated by Safe Paths.

Encourage other testers, in other geographical environments to conduct similar tests & report results.

Try to connect with others in the project who are interested in this topic:

Some of these issues go beyond software dev, and we may have other folks who have expertise, and can point to previous research findings etc.
There may have been some in-project research in this area already that I am unaware of.
The impacts of these issues are also far-reaching, and may extend far beyond the Dev team.