Fortunately you don’t have to make do with my wild guesses and assumptions as Smashing Magazine have a done a comprehensive critical analysis of testing done by app and game developers using the TestDroid cloud-based testing suite. Over 17 million tests were run on 288 different devices over 3 months early in 2014. Depending on region and measurement, the 288 devices represent somewhere between 92% and 97% of Android phones. With the credentials laid out, let’s take a look and see what the testing revealed.
Most of us will have seen the version stats from Google, showing the relative percentage of Android versions. Put simply, Froyo is less than 1%, Gingerbread is around 10% and ditto for Ice Cream Sandwich. Jelly Bean takes the lion’s share at nearly 55% and KitKat comes in second at about 25%.
Smashing Magazine’s testing showed that on average 23% of apps exhibit a problem when moving between versions of Android. The biggest problems arose with Gingerbread (30%) followed by KitKat at 21%. Jelly Bean and ancient Honeycomb were next. Interestingly, although Gingerbread is the oldest version with significant market share, 40% of tested apps still work with this version.
The figures also reveal that ICS is the most stable version of Android with a low failure rate that broadly continues through Jelly Bean, though KitKat was more problematic with an increased error rate.
While the OS can cause problems, the hardware’s not blameless either. The research looked at screen resolution and the impact of memory on apps as well. Devices with resolutions of 2560 × 1600, 1280 × 800 and 1280 × 720 pixels gave the fewest problems, typically 1.5% or less. Small screen resolutions were the worst with 400 × 240 and 320 × 240 pixels being particularly bad.
On the RAM front, 512 MB seems to be a significant cut-off point and it’s no surprise that Google recommends this as a minimum. With this amount of memory or less, around 40% of tested apps exhibited problems. At 768 MB and above, the error rate falls to 16% and by 1 GB RAM, it’s down to 1%.
Overall, this is all interesting stuff and a fascinating insight into what app developers have to put up with. I’ve only covered a few of the areas and there’s additional analysis on drivers, OEM customisations and chipsets. I thoroughly recommend that you read the whole article over at Smashing Magazine to understand more.
Taking a slightly different view from a user perspective, if you want a really stable device, you should be buying a high resolution device, with 1 GB RAM and running Ice Cream Sandwich. Hmm.