track this

The internet seems to have figured out how Assisted GPS works, and it doesn’t like it. It doesn’t like it one bit.

The first thing to point out, I think, because a certain contingent of the internet punditry brigade have decided that this revelation must immediately be turned into a stick with which to further beat the iOS vs Android horse, is that Android caches precisely the same data.

Here’s the output of Android’s location data cache:
$ ./parse.py cache.cell
db version:  1
total:       41

key                     accuracy  conf.   latitude    longitude  time
240:5:15:983885             1186     75   57.704031   11.910801  04/11/11 20:03:14 +0200
240:5:15:983882              883     75   57.706322   11.911692  04/13/11 01:41:29 +0200
240:5:75:4915956             678     75   57.700175   11.976824  04/13/11 11:52:16 +0200
240:5:75:4915953             678     75   57.700064   11.976629  04/13/11 11:53:09 +0200
240:7:61954:58929           1406     75   57.710205   11.921849  04/15/11 19:46:31 +0200
240:7:15:58929                -1      0    0.000000    0.000000  04/15/11 19:46:32 +0200
240:5:75:4915832             831     75   57.690024   11.998419  04/15/11 16:13:53 +0200
Compare and contrast with the iPhone’s location data cache :
CI		 Timestamp	Latitude	Longitude	AccuracyConfidence
196769687	306858899.8	-33.79464816	151.1914875	1500	90
78808909	307497296.2	-33.89795403	151.2098288	1500	90
78808911	307506560.8	-33.89293759	151.2039136	1500	90
78723747	308008772	-33.79670207	151.1811041	1500	90
78783294	308104094	-33.8842238	151.2066338	1500	90
78742991	308217181.1	-33.79704956	151.1957413	1500	90
78723098	308301684.8	-33.80716884	151.1686881	1500	90

The iPhone’s cache data is slightly truncated to fit within the available width here, but if you’d like to see the full schema for the cell tower location table click here for a PNG or here for an Excel spreadsheet.

So it’s worth noting that the contents of the logs kept on both Android and iOS devices is identical. In fact, it’s suspiciously identical. Perhaps it’s just because these are the obvious and only metrics you might look for in a location database, or perhaps it’s because both are most likely loosely based on the database schema Skyhook used (both Android and iOS, until relatively recently, relied on Skyhook for cached geolocation services).

Regardless, Google have openly stated that the purpose of their database is to provide aGPS functionality. Given that the contents of the databases are identical it seems entirely reasonable to assume that iOS’s consolidated.db is, indeed, a geolocation cache used for the purposes of aiding aGPS.

The biggest meaningful difference between how Google and Apple handle this functionality is that Android only retains locally the last 50 cell towers you’ve seen. This is a fairly basic trade-off – the contents of the cache on Android is potentially less interesting but as a result the cache itself is less useful. An Android device needs to get aGPS data from the internet more frequently than an iOS device because its local cell tower location cache is less exhaustive. More internet access means more battery usage and slower GPS lookups.

Google also state that they collect a device ID allowing their own database of cell tower records to theoretically be tracked back to an individual handset. The data appears to be anonymised. I don’t view this as a significant problem either way. I certainly don’t think Google are actually using this data to track individual Android owners and I think doing so would be difficult even though it appears to be possible.

Consolidated.db logs cell tower locations - not your phone’s location
The next thing to make clear is that this cache contains the location of cellular access towers and wifi access points. It does not directly track the location of the phone (and by extension, the user). The following image is pulled from my own consolidated.db file. It’s centered on my home address. Note that there is no marker over my home. The large blob on the left-hand edge is Vodafone’s Australian head office, and the cell tower I was most frequently joined to when my phone was with Vodafone. The various other blobs are nowhere near my house, nor are they spots along my daily commute. They’re simply other towers that the phone has seen:

This bears repeating: The contents of the cache are the locations of cellular access towers. Not the location of the phone. It’s certainly possible to interrogate the contents of the file and determine my rough whereabouts to within about a 2km radius. It’s largely useless for figuring out my whereabouts to any great level of accuracy. You aren’t going to be able to look at the contents of consolidated.db and determine my home or office address.

It is true that the data is sent to Apple. In addition to providing an offline, locally accessible location cache this data is also sent to Apple and used to let Apple build their own central cell tower location database. It’s further used, as is set out in the iOS Terms of Service, in the same was as your GPS-determined location – that is, to provide location-aware services for apps. This is non-controversial though – we already knew this, and within the iOS License Agreement it is made clear that your location will be shared for specific purposes, and the OS provides the ability to turn this functionality off. (That first link is worth reading, incidentally, for anyone who was actually surprised to discover that iOS has an offline location cache – it’s Apple’s response to Congress on precisely when, how and why they collect user’s location data. Pages six and seven in particular deal with the use of cell tower location tracking to enable assisted GPS capabilities.)

When location services are enabled on an iOS device the contents of consolidated.db is batched and sent to Apple twice a day. This ceases when location services are disabled – although, of course, the ability to use location-aware apps ceases along with it. Again, this is non-controversial. Android behaves in the same manner (although it seems updates are sent almost in real-time rather than batched). Neither OS shares your location with the OS vendor if location services are turned off.

With location services disabled, iOS ceases to maintain the local location cache. It does not delete the existing consolidated.db, which has, I think, lead to some confusion in commentary on this issue. But you can readily test this yourself – examine your own consolidated.db file, disable location services, and take your phone out and about with you. When you return, make another backup and examine the contents of consolidated.db again. The database will not have been updated beyond the point at which location services were disabled.

So with location services disabled there is, indeed, no tracking of your location occurring, either by Apple or by the device itself.

Broader implications of having an offline location cache
It’s certainly true that if I now need to determine your phone’s general location (within the ~2km radius of accuracy set out above) at a specific point in time, say for the purposes of a law suit, I can subpoena you for the contents of your consolidated.db file. But then I could always have subpoenaed your mobile phone company for the same information. It’s hard to see how this is a significant problem, or indicative of a privacy issue. It’s also true that if I steal or hack your computer I can probably interrogate your iOS backups to look determine the same information – your phone’s general location to within a couple of kilometres. This is completely circumvented by ticking the “encrypt iPhone backups” checkbox within iTunes, so if this is a significant concern to you then you should do just that and worry no more.

If I’ve hacked your computer and am now trying to figure out your home address then consolidated.db is useless to me – the data simply isn’t accurate enough. Much better, instead, for me to check your wireless access point’s MAC address against Google’s publicly query-able wireless AP location database. For my own home router this gives a location accurate to within a few feet.

So given all of the above, what are the actual, real-world problems raised by the existence of iOS’s cell tower location cache? From what I can see, there are none. In fact, I’m so convinced that there are none that I’m sharing my own consolidated.db file for any and all of you to download.

Further I don’t think it’s sensible to paint this as an iOS vs Android issue – both platforms are doing this, and it seems to be a good engineering solution to the problem of how to rapidly provide location data without the need to resort to internet access or to firing up the GPS radio. Trying to turn this into an issue of platform contention seems, to me, to be misguided.

That said, if there are privacy issues I’ve overlooked then let me know in the comments and I’ll try to address them.

Posted in android, apple, iphone | 1 Comment

UI issues in Twitter for Mac/Tweetie 2

Twitter for OS X, as made available via the Mac App Store and hereafter referred to as “Tweetie 2 for Mac” or “Tweetie 2″, is absolutely the best Twitter client for OS X available today. That isn’t necessarily saying a lot – I’m no fan of the state of desktop Twitter clients in general – but Tweetie 2, like Tweetie for Mac before it, manages to be more polished than any other Mac Twitter client. That said, Tweetie 2 has some major issues relating to missing features and UI quirks, a few of which I’ll detail here.

It completely ignores the HIG. I’m largely OK with this though. The interface is certainly very stylish and largely usable. However, a side effect of the completely non-standard window design is that the drag area is impossibly small, occupying only the space between the traffic light widgets.

Update: This is incorrect – the entire black sidebar area is actually draggable.


There’s also an odd issue with window positioning – it’s not possible to position the window in such a manner that the lower edge is below the notional top of the OS X dock. I’m not aware of any other application that behaves this way – the original Tweetie certainly didn’t.

Hiding the dock, or placing it on another screen edge, allows the window to be positioned on the bottom screen edge. Tweetie 2 seems to be ensuring it’s never positioned off the edge of the screen – a noble goal, perhaps, but it’d be nice if it’d let me position it in such a manner as to not waste a gutter of a pixels at the bottom of the display.

The video below shows the behaviour of the original and new versions of the app with regard to window placement.

Bizarrely Tweetie 2 no longer responds correctly to the unversal “Close Window” OS X keyboard shortcut, Cmd-W. Instead, issuing Cmd-W causes Tweetie 2 to hide – the functionality is identical to issuing Cmd-H. This is easily demonstrated by opening the Compose Tweet window, selecting the main Twitter window, then issuing Cmd-W, which causes the whole application to hide. Clicking its icon then restores both windows. The same occurs when using the menu commands for Close/Hide in place of the keyboard shortcuts. This is shown in the following video.

Tweetie 2, at long last, brings official retweet support to the OS X client – but actually using it spawns a modal dialogue. Worse, if the Tweetie 2 window’s width is less than that of the resulting dialogue box then the entire Tweetie 2 window is repositioned until the dialogue is dismissed. Retweeting when window is less wide than the retweet dialogue repositions the window, as shown in the following video.

There’s also some feature regression. It’s no longer possible to determine whether or not a user is following you – functionality previously available in the original Tweetie for OS X, and in both the iPhone and iPad Twitter applications.

Disappointingly, there are obvious and useful features from the iPhone client which haven’t been implemented, such as the ability to view your retweets, the ability to report accounts as spam, or add or remove users from a list. There’s no geolocation support, despite Snow Leopard’s inclusion of the same Core Location wifi positioning capabilities as the iOS devices, and the Profile view doesn’t display the total number of tweets for any account, including your own.

Update: It turns out it’s possible to report a user a spam via the user’s Profile page, though it’s odd this hasn’t been added to the (already overloaded) context menu.

The right-click/context menu options are much more comprehensive than they used to be but the lumping of a dozen or more actions into these menus seems unwieldy.

The only way to post an image is to drag and drop it into the New Tweet window. The ability to do so is a welcome addition, but the removal of the ability to browse to an image is a net usability loss.

URL shortening is now enabled by default, removing an extra step when posting but making it impossible to deliberately post a full link – something I do infrequently but occasionally have need for regardless. It’s also no longer possible to select which shortening service to use – all URLs are shortened via t.co, Twitter’s own URL shortening service.

Worse, it shortens everything. URL’s which are shorter than the resulting shortened URL are shortened anyway. URL’s to other shortening services such as bit.ly, for example, are re-wrapped as t.co URLs.

It’s such a shame – after the long, long wait for Tweetie to be updated to bring it up to the same standards of functionality and usability as the simple iPhone client, Twitter for Mac seems to miss the mark in such a large number of ways I wonder if it was rushed out to meet the Mac App Store launch deadline. While I’d normally hold out hope that these issues will eventually be resolved the glacial pace of development for the Mac version of Tweetie leaves little reason to be hopeful they will be fixed soon.

Posted in apple, mac, Twitter | 4 Comments

why I don’t run a food blog (and why I don’t read yours)

I’m widely known as being a bit of a foodie. And it’s true, I am. But it’s always been very clear in my mind that this blog shouldn’t be a food blog. I’m not sure exactly what this blog actually is (beyond being mostly dull, self-serving and pointless, as is the nature of blogs) – but a food blog it most definitely is not.

This doesn’t mean that I won’t occasionally write about food. But I firmly believe the world needs fewer food writers, not more. It needs fewer food websites, not another one. And it is perhaps, but only perhaps, less in need of another recipe site than it is of another site devoted to allegedly funny photoshops of cats.

It’s not that I don’t read food blogs – there are a handful that I do – but rather that the internet is already so full to bursting with food blogs of such overwhelming mediocrity that adding another may well cause the entire thing to breakdown completely, unleashing a maelstrom of lolcats and cookie recipes of apocalyptic proportions.

Attempting today to use the internet as a culinary resource is already hellish. A search on nearly any subject will yield a dozen conflicting pieces of advice. Recipes published online tend to be wholesale and unauthorised copies of those published in cookbooks. Where they aren’t, they’re often untested, or obviously strewn with errors, or, worse, non-obviously strewn with errors.

My own utterly unscientific observations are that 99% of food blogs suffer from being laden with impossibly awful food photography, excruciatingly awful writing, or, most frequently, both. This doesn’t mean that as a foodie’s resource the internet is useless – that remaining 1% can be worth its weight in gold. Rather it’s just a recognition that the overall quality of the internet’s food writing is dire and an acceptance that I do not cook, photograph food, or write well enough to do anything other than make the problem worse.

Despite this, I may still occasionally write about food – after all, I spend a lot of my life cooking, or eating, or thinking about what to cook next, or what to eat next, or reminiscing about the last time I ate something truly extraordinary. Never writing about something that dominates so much of my time and thought would be strange indeed.

But while I may occasionally write about food, as I am now in a meta kind of way, I’ll offer up this pledge – I’ll never post a recipe, I’ll never post a food photograph, and I’ll never post a restaurant review, though I reserve the right to continue to do all these things, badly, on my Twitter account.

And now, a list of truly invaluable internet food resources and food blogs – my pick for the 1% of the writing worth reading:

  1. eGullet – Probably the single best food community on the internet
  2. Chowhound – The discussion boards at chow.org, and a fantastic reminder that lively and engaging discussion is still possible on the internet without immediately devolving to the level of YouTube comments
  3. Ruhlman – An opinionated and polarising figure, but probably the most talented food writer I’m aware of today. Neither a chef nor even a particularly good cook, but an extremely talented author with a palpable obsession with food, cooking and the craft of the kitchen.
Posted in food | Leave a comment
  • Apologies, but no results were found.