Folow-on from this morning’s testing.
25 April 2020 - Real-world GPS test #1
Continued testing with 3 phones:
A is a Mototola G7 Power (Android 9)
B is Samsung Galaxy J6 (Android 9)
C is a Redmi 6A (Android 8.1)
From about 07:40 (05:40 UTC) to 10:21 (08:21 UTC), they were all on a desk, inside, next to me while I worked on my computer.
From 10:24 (08:24 UTC) to 12:29 (10:29 UTC) they were on a table outside, with clear view of teh sky, all within a metre of each other.
From 12:30 (10:30 UTC) to 14:16 (12:16 UTC) they were in another indoors location.
Except for B, which was picked up at 13:12 (11:12 UTC), and taken away (mon son wanted his phone back!).
I did not handle (i.e. wake the screen, etc.) the other two phones until 13:50 (11:50 UTC), when I used the app to export location data by email.
Due to some surpises with the location data, I then performed another data export at 14:50 (12:50 UTC) on both phones A & B.
Despite my notes in my previous report, I did not set up any sort of additional “control” GPS logging app on these phones.
Timestamps for GPS data
All times in UTC, as that matches the data…
Phone A | Phone B | Phone C | |
|---|---|---|---|
05:40 to 08:21 - desk inside | Highly variable intervals from 5:00 up to 52:47 (between 6:06 and 6:59) | Most intervals solid at 5:00. Just 3 exceptions: 6:46, 8:24 & 8:20 | 5:13, then 5:08, then 30 consecutive logs at 5:00 intervals exactly. |
08:24 to 10:29 | 9:49 gap at up to 8:32, but after that mostly settled at 5:00 intevals. 3 intervals almost exactly 10:00. | Zero data points in this period | 26 consecutive logs at 5:00 intervals exactly. |
10:30 to 12:16 | Variable again, including one interval of 13:35. Most intervals in the 5-6 min or 9-10 min ranges | Zer data points until 11:12 when the phone was woken. Then started logging again (2:55:05 since last log). Variable intervals from 5:18 to 7:08 | 20 consecurive logs at exact 5:00 intervals, then one at 6:58 |
Data export content | Both data exports contaned complete data | Data export contained complete data | Data export at 11:50 contained incomplete data, only up to 6:08:48. Later data export at 12:50 contained all data up to that point. |
Summing, up we seem to see a variety of behaviours, many of which appear to be bugs of one sort or another.
Location tracking locks up for extended periods but sometimes seems to recover automatically (e.g. A, up to 52 mins).
Location tracking locks up for extended period, but only eventually recovered when a user unlocks the phone (only on B: locked up for 2h55).
For logs in the 5-10 minute range, there seems to be different types of noise - sometimes it takes the form of 2 bands: 5 minute or 10 minute intervals exactly (+/-1 second). Sometimes the slower intervals are more evenly spread across the space between 5mins and 10 mins.
Possible variations are per-phone model, or per-Android version. Would need more test data to establish that.
Plus the one odd instance where location data had been recorded, but was not included in the Export file.
Here’s all the timing data, plotted on a log scale, so the big outliers can be fitted in.
GPS location data
Also of interest is the GPS location data itself. Recall that the phones were sitting stationary for ~2 hours in each of 3 locations, 2 indoors, one outdoors.
These plots show the vriation in distance, in all cases from the average of all points recorded - calculated separately for each of the 3 test locations..
What these seem to show is that
GPS position of A is highly variable when indoors, more consistent outdoors (but there were still variability
GPS position of B is consistent and stable indoors (we got no outdoors data points due to the GPS logs hang)
GPS position of C is very consistent, both indoors and outdoors.
On all devices these appears to be a notable correlation between variability of position data, and variability of gaps beweetn GPS logs.
Disagreement from the mean position of the phones was consistently at the level of 20 metres or above - which implied phones disagreeing by up to 40 metrex on GPS location.
Individual data points, particularly for phone A were consistently more variable than that, with variation of up to 72 meters from the mean position of all phones. The effects appear to be worse indoors. They could potentiallty be influenced by the presence of WiFi, since there was a WiFi hotspot close to both of the indoor test locations.
What have I learned?
We seem to see wuite different behaviour between different devices
In some conditions, GPS data can be consistent, and logs can be produced reliably every 5 minutes. However this does not always happen.
Variation of GPS readings on one device does not appear to correlate with GPS readings from another device. Hence there is little (if any) sign that synchronizing GPS location logs will reduce differences in position on static phones. (it does seem likely to help with moving phones, for obvious reasons, as per #516)
There appear to be multiple distinct manners by which a phone can log GPS data irregularly.
It is possible that movement interferes with reliable GPS log times. In our earlier test, with movement, timings were always variable; in this static test there were some very regular timings.
It is possible that operating a phone (bringing it out of a sleeping state) will bring locked up GPS logging back to life. We saw this happen with phone B when it locked up.
It is possible that WiFi affects the stability of GPS data recorded by some phones.
Even with static phones in the open air, the assumption that phones will agree on GPS to within 10m is unrealistic - the reality appears to be closer to 40m (with outlying data points that are much worse).
What do I not yet know?
Key things I don’t yet know, which may be relevant
Does WiFi destabilize location data when indoors? Would we get more stable data with WiFi disabled?
How much of the problems are with the App vs. GPS signals? As stated before, a control in the form of another GPS app would be valuable.
What are the causes of the variations in timing of GPS logs? Given the multiple patterns observed, there appear to be multiple causes.
Is there anything we can do in software to improve the reliability of GPS data? E.g. gather multiple readings and discard any outliers?
Plus some more unansered questions carried over from the last session…
Does disabling & re-enabling location data in the app actually reset the GPS ping cycle & therefore synchronize the timers? Not yet clear.
What is a good approach to use to provide a “control” for GPS data? Tracing location data with google in parallel? Will doing so interfere with my test results (e.g. by “priming” the GPS receiver in some way?
Do services like Google Maps, Pokemon Go & Geocaching use some sort of noise reducation or averaging techniques to create a higher quality GPS signal? Are we being naive in simply assuming the raw GPS data we get back from the API is good enough for this service?
Is there already established expertise on this topic that I am unaware of, and therefore failing to take advantage of.
How representative is my geographical context, of other contexts in which we might deploy? What will be the impacts of latitude, cities, weather etc. etc.
What Next?
See previous Test Report - most of that is still valid.
This test session has covered the following item
Conduct further testing focussed on GPS locations of phones in static positions
Indoord & outdoorsGPS pings synchronized & non-syncrhonized