1 June - 3d testing with latest Android Build
- Diarmid Mackenzie
Summary
Latest Android develop build (delivered on 28 May: https://tinyurl.com/safepah-app-285 ) has delivered substantial improvements on both reliability & accuracy.
Broadly, I would say that reliability & accyracy appear good enough for MVP1 - however we should bear in mind that the assessment is limited to one make/model of phone one Android version, and one geographic location, and one particular user’s usage pattern. Other test results may vary.
Other than “further testing needed”, specific concerns that have arisen from the testing are:
Must fix / analyze for MVP1
Battery usage appears to have increased ~3x from previous builds, and is now at a concerning level. We should look at what battery optimizations we can make without compromising accuracy. https://pathcheck.atlassian.net/browse/SAF-556
The latest build does not appear to be stable. https://pathcheck.atlassian.net/browse/SAF-546 has been raised. This is the only problem I have been able to isolate so far, but I don’t believe that this explains all the instability I saw over the 3 days. Nevertheless the instability does not seem to have significantly interfered with location logging reliability.
The latest implementation logs sensitive info into the Android system log. This should be fixes for privacy reasons: https://pathcheck.atlassian.net/browse/SAF-494
To look at, but may not be needed for MVP1:
There are still a significant number of gaps > 6 minutes (16% of gaps). While addressing these may not be critical for MVP1 (particularly compared to e.g. getting iOS up to the same level as Android), there is still an opportunity for further reliability improvements here. Looking at the latest ADB logcat traces, https://pathcheck.atlassian.net/browse/SAF-493 appears to still exist.
There was one occasion where a gap of 49 minutes occurred. Again, it would be valuable to understand and fix this problem. https://pathcheck.atlassian.net/browse/SAF-557
Details of Test
Testing on an Android 9 Moto G7 Power phone, in a location in rural Italy, from 29 May to 1 June, a total duration of 67 hours.
The phone has access to GPS, cell network (cell coverage is not great but OK), and my home WiFi network). Much of the time was spent indoors.
The app had no special permissions regarding Battery Saver settings (Battery Optimization was enabled for the App, as per default settings).
Testing was performed with this build https://tinyurl.com/safepah-app-285, and compared to previous testing done from 23-26 May in the same location, with the same phone as described here: 23 May & 26 May - Testing with updated Android build
.The key changes in this build were described by Dev as follows:
GPS Accuracy: Using the getLastKnownLocation method (with GPS provider) of LocationManager API along with the Fused Location Provider API to retrieve the locations.
Reliability: Using Fused Location Provider API that ensures the location updates in a given interval. Current interval is 5 minutes
Note: We are taking 5 locations with 3 seconds apart every 5 minutes and storing the 5th location in the realm DB that retrieves at 15th second every 5 minutes
In both cases, I was moving around at home, and in my garden, sometimes with my phone in my pocket, sometimes it was left at my desk. I used my phone normally during the period.
I’d made one brief trip away from the house during the 23-26 May testing, but disregarded these points in assessing accuracy.
I assessed reliability by measuring the gaps between consecutive data points.
I assessed accuracy using a “by eye” assessment of the spread of data points vs. my rough recollection of where I had been + my knowledge of the local topography that enables me to identify certain locations that I definitely had not been (dense woods, or steep slopes, which I’d certainly have recalled had I been in those locations).
Reliability
The left shows the previous trace from 23-26 May. The tight shows the latest trace (29 May -1 June).
There is still some variability in gaps between logs, but only one instance over 14 minutes.
Analyzing the data we can compare as follows:
Metric | 23 to 26 May | 30 May to 1 June |
|---|---|---|
% of expected data points | 61% | 91% |
% > 6 mins | 22% | 16% |
% > 9 mins | 17% | 2.7% |
% > 14 mins | 3.2% | 0.1% |
% > 29 mins | 2.1% | 0.1% |
In the latest trace, there is just one occasion where there was a long gap between points.
From 08:03 to 08:53 on 31 May, there was a 49 min gap with no data. I don’t have any info on what was happening at that time.
Accuracy
Just looking at the portion of the trace that is in & around my home.
Previous (about 500 data points) is on the left, latest (about 730 data points) is on the right.
While my movements about the site will not have been absolutely consistent between the two scenarios, they will have been broadly similar. The right-hand picture has just 4 “implausible” data points (0.5%), whereas the the left-hand picture is much more spread out, and contains at least 30 “implausible” data points (in deep woods, on steep hillsides, places I know I would not have been).
Overall, accuracy on the more recent trace looks to be far more accurate than previously.
Battery
I have just 2 data points on battery consumption:
on 29 May: 8% battery usage over 20 hours.
on 1 June: 6% battery usage over 18 hours 15 mins
Compared to previous measures (2% in 16 hours), it does appear that battery usage has increased significantly.
We should look at what we can do to reduce battery usage.