This weekend at AAG (Association of American Geographers) Annual Meeting. I presented the following paper in a session led by Seth Spielman (who does some amazing work identifying “personal cities” through geopaths). And I got to hang out with the imitable Sarah Williams.
OpenPaths: A new approach to aggregating personal geographic data
The collection of personal geographic data from mobile devices is a ubiquitous practice of service providers and application developers. These data are being stored, analyzed, and monetized primarily by corporate interests; there is limited agency for individuals over their own data. Awareness among the public regarding the value of their personal data is nascent. OpenPaths, created by the Research and Development Lab at the New York Times Company, is a platform and a model and a platform that demonstrates the collective value of personal data sovereignty. It was developed in response to widespread media coverage of the obfuscated but accessible location record generated by all Apple iOS devices. OpenPaths participants store their encrypted geographic data in a cloud infrastructure while maintaining ownership and programatic control. Projects of many kinds, from mobility research to expressive artwork, petition individuals for access to their data in exchange for a stake in the outcome of the project. Ultimately, we would like to activate the practice of “participatory sensing” on a large scale in a way that self-regulates the creation of ad-hoc geographic datasets. Furthermore, within a theoretical context, OpenPaths moves beyond locative media’s primary concerns with connectivity, the coupling of data to place, and spatial representation to address the components of an ethical implementation of crowd-sourced geographic systems in the age of “big data”. How can we seat the individual in a mode of control over personal geographic narratives in a society in which locative media has become banal?
Some OpenPaths news: in the last couple of weeks we’ve revamped the service and launched iPhone and Android applications — it’s been exciting to hear about people using the platform again (OP on Twitter).
Flowing data posts:
“There are a lot of ways to collect your location, whether it’s for journaling and personal reflection or for sharing with others, but it can be tricky making use of your data once it’s stored behind company servers.”
OpenPaths is different.
NYC VERTICES:
In several recent posts, I’ve talked about experiments with personal geographic data collected via OpenPaths. In those examples, location is treated in absolute terms, latitude and longitude.
However, I am working toward something here. Most of my past work has been concerned with the relative qualities of place, the psychogeography that isn’t necessarily keyable to coordinates (see our article in Urban Omnibus). Presently, I’m developing some analytics to try and bridge that gap.
The first order of business is to begin thinking about location in terms of place. Place is a concept that is relative to the context of the individual — but using geodata we can at least identify significant patterns that suggest loci of activity.
Starting from my path data, I used a clustering algorithm (I’ll post the code next) to construct a network graph of my arrivals and departures around the city. What you see here are the locations of all of my “significant” places around the city, over the last 6 months or so. NYT Labs is the big green point up top — home is purple toward the bottom. The lines show the strength of the connections between them (eg, from home I’m most likely to go to the lab).
The conceptual shift here, and I think it’s an important one, is to begin to treat location as behavior. More to come.
(drawn with python)
One thing I wanted to try with OpenPaths data is to reconstruct the paths I have taken. I’m using the as yet unreleased iPhone app, which records a point every time there is a “significant” location change, as determined by the iOS API (this runs in the background without much battery drain — the reason we aren’t using continuous GPS is that such an app would quickly burn through it).
This image shows all of my points around the city in a given time period (without a base map, to preserve some privacy, and it’s kind of more interesting that way). The lines are determined by grouping series of points that are within 10 minutes of each other and inferring the start point.
Additionally, I estimated the maximum speed of each path by looking at the fastest few segments. By clustering the speeds, using the k-means algorithm, the paths are classified by mode of transportation. Red is car or Amtrak, purple is bike (my primary mode), and green is walking.
The arc of my frequent rides between Brooklyn and midtown are pretty clear, and in general you get a sense of the areas that I habitually cover with the different modes. Note Prospect Park at the bottom. One thing that’s a significant omission is the bulk of my subway travel — any ideas on how I could infer this would be appreciated. And I certainly walk around more than is reflected here, but the location changes aren’t big or sustained enough to register.
Had the pleasure of having lunch at OpenPlans today, at Merran’s invitation. Theyre an incredibly cool organization combining software development with open government and livable streets. Some exciting discussion around applications for OpenPaths and the nature of personal data.
DATA REENACTMENT: STREET VIEW VIDEO FROM A STOLEN PHONE
My close friend and collaborator, Sue, had her iPhone stolen earlier this month. The thief had it for 5 days, after which he ransomed it back to her. In the meantime, he had it with him as he drove around LA, presumably looking for other opportunities to be an asshole.
Our phones, clearly, are really personal devices. When we talk about personal data, the mobile phone is as physical an embodiment of this as anything, a data-sensory appendage if you will. What does it mean, then, when we’ve been separated from the device? It feels like identity theft as much as the loss of valuable electronics.
So when Sue got it back, she felt a bit estranged from it. We wondered about the life her device had had away from her, which led her to use OpenPaths to take a look at where it had been. Sure enough, the thief’s home and haunts were pretty readily identifiable.
Sue had also seen the last video Id made with OpenPaths and Google Street View, and we decided to make another one with her data. However, I wanted to take it a bit further. As fun as my first video attempt had been, it’s a bit impressionistic — you just get this blitz of unconnected images. However, Sue’s data had a very clear narrative behind it. We had a collection of points that the thief had visited with the phone, so I thought we should be able to get a smooth path between them.
First, I used the Google Directions API to map the likely route that the thief would have taken between known locations, as well as filling in some intermediary points, which was @blprnt’s idea from our earlier brainstorms. One of the cool things about the Street View panorama data (described by @jaimethompson) is that it shows the linkages between consecutive images taken by the Google car. So by calculating the heading from one point to the next and heuristically choosing links between panoramas headed in the right direction, we can access all the images taken along the way. Again using heading we can point the camera in the right direction, download the tiles we want, and stitch a frame together. Applying this to the thief’s route, we got a complete reconstructed path that plays back much more like a continuous video than my previous experiment (it evens out after the frantic first 30 seconds).
It’s a bit like if Google was driving the getaway car, starting downtown where the phone was stolen, and traveling over the city until it’s finally given back. Of course, we’re leaving out the pauses when he wasnt moving, and the temporal displacement of Street View images make this a kind of a weird frankendata — while the video retains some relationship to the truth of the human interaction behind it, it remains a kind of data fiction.
Oh, and for those who prefer the written word, theres always the driving directions.
Edit: some press love from Gizmodo and Flowing Data
STREET VIEW VIDEO VIA OPENPATHS API
[code] Python (2.6), GPL
Brainstorming with @blprnt this morning about what people might do with the new OpenPaths API, we thought it would be pretty awesome to see every place you’ve ever been via Google Street View.
Loading all of that up through the Google Maps interface seemed overly burdensome, so we figured there must be a way to pull the static tiles. Turns out there is (though it’s unofficial). @jaimethompson breaks it down for you.
From there, it was pretty straightforward to pull the points, scrape the images, and assemble the video. It includes points from September ‘10 to the present and a dozen or so cities, beginning in LA I think, but NYC clearly dominates. Non-urban spots arent captured well, and in Googleland it’s never winter. You might also notice that the granularity of the video increases at the end. That’s because at a certain point I start using the forthcoming OpenPaths app, which samples periodically, rather than the data from iTunes backups, which only looks at novel locations. The API pulls from both.
Want your own? I did this with python as usual — you can grab the code here if you’re interested (youll need PIL and the latest OpenCV bindings installed to export the video). This is a bit of a soft launch for the OP API as we gradually work in new features. Let me know if anyone gives it a try (especially if youre using a different language).
Noncoders fear not — we’ll hopefully be integrating something like this (but cooler and more blprnty) directly into the OpenPaths interface in the near future.
INTRODUCING OPENPATHS.CC
So last night a few of us at the lab pushed last night to launch openpaths.cc, a service that securely collects iPhone location data that people can later share with research projects. The basic idea is that Apple collects this data, but it’s really yours and you should be able to use it in awesome projects as you see fit. The site gives you a way to store it securely, and then to share it (on a case by case basis) with research/art/project proposals, as well as download it in friendly formats for your personal use. We also have some awesome map visualizations by Jer Thorp so that you can see where you’ve been, which is really payoff enough.
Posts by Jake and Jer and an awesome interview with Michael that gets into the themes we have been discussing, as well as other press.
Edit: iOS update 4.3.3. ends the collection of this data. However, old files have not been deleted. Further, our uploader app will scan Time Machine backups, so there’s still a lot of data to be gathered.
