PUCK: PUCK of Ubiquitous Contextual Knowledge™
Internet-of-things / ubiquitous computing … these concepts have been central to NYTLabs thinking over the last year (and related to much of my past work 1, 2). They are old ideas, but we are finally seeing their fruition in interesting consumer products on the market.
I’m into two concepts in particular: Twine and GreenGoose. Both are looking at dirt simple, generalized hardware (particularly the mighty accelerometer) that wirelessly spit data up to the cloud. Server-side software can then contextualize that data to serve any number of social purposes, like alerting you when the mail comes (Twine) or keeping tabs on whether the dog is getting fed (GreenGoose). It hits that middle range of Greenfield’s notion of scale — the level of the room (as opposed to the individual, building, or street) — which I think is a particularly ripe area.
So in tribute I wanted to create our own version at NYTLabs (after acronym brainstorming w/ @mboggie). The idea is a small, generalized piece of hardware that you can attach to a physical thing in your world which can be programmatically assigned to monitor interesting events. It should have long battery life and report data from the accelerometer and other sensors wirelessly. Its logic should be on the server somewhere, not burned into the device, and it should have HTTP and UDP interfaces. And finally, in my twist on the concept, I want it to respond to touch, so that if you pick it up, it turns into a high-resolution controller, like a Nintendo Wii. The idea is to then have hundreds of these all over everything. I’ve been picking at this the last few months, and finally have a proof of concept running.
The core component of PUCK is the XBee. XBees rule. A network of lightweight components is in itself aesthetically appealing, and in essence, all I want to do here is hook a sensor up to one. The XBee is smart enough that this can be done without the help of an Arduino or other microcontroller, which is essential to keeping down the cost.
The path I went down and should not have is trying to figure out how to program them directly in python over the serial port. Instead, Andrew Rapp has an awesome java library and a wealth of documentation that is the way to go. I got hung up in that I didn’t just want to collect data from the XBee, I wanted to reprogram its behavior on the fly, using ‘API’ mode — this library has all that hard work in place.
In any case, the rest of the design here comprises an accelerometer, tilt switch, and LED. The tilt switch is able to wake up an XBee — it’s quite sensitive, so moving the device at all can take it from a mode where its just periodically reporting data and put it into high-frequency reporting mode. Finally, the LED just indicates when the puck is awake. When we get a MakerBot I’ll design a housing.
On the server side, I’m pushing the data into Redis and rebroadcasting it via OSC. The next step is to figure out how to architect the signal processing and the nature of the interface.
But I’m far from a hardware ninja, so I’m pleased to have something together that I can experiment with. Updates to come.
It’s been an unexpected honor to have my reflected mug appear in this week’s issue of TIME magazine. The NYT Labs Reveal project, for which I wrote the software but to which Alexis, Ted, Justin, Matt, Agnes, Will and Michael Z all contributed concept, design, and production, has been chosen as one of the 50 best inventions of the year. In print it’s page 72; online here (subscription required). I love that they’ve organized things by size — the mirror is 76 centimeters.
Here’s the clipping
Slashdot picked up on a project Ive been working on at the lab, originally described by the Nieman Jouralism Lab. Nieman didnt show what I think are the better features of the mirror (visualization of sensor-based health stats, gesture-controlled video), and the Slashdot thread, though hilarious, misses the point behind the project. I hope to articulate the project better in our own documentation and a new version of the software which we’ll roll out in the coming months.
DATA REENACTMENT: STREET VIEW VIDEO FROM A STOLEN PHONE
My close friend and collaborator, Sue, had her iPhone stolen earlier this month. The thief had it for 5 days, after which he ransomed it back to her. In the meantime, he had it with him as he drove around LA, presumably looking for other opportunities to be an asshole.
Our phones, clearly, are really personal devices. When we talk about personal data, the mobile phone is as physical an embodiment of this as anything, a data-sensory appendage if you will. What does it mean, then, when we’ve been separated from the device? It feels like identity theft as much as the loss of valuable electronics.
So when Sue got it back, she felt a bit estranged from it. We wondered about the life her device had had away from her, which led her to use OpenPaths to take a look at where it had been. Sure enough, the thief’s home and haunts were pretty readily identifiable.
Sue had also seen the last video Id made with OpenPaths and Google Street View, and we decided to make another one with her data. However, I wanted to take it a bit further. As fun as my first video attempt had been, it’s a bit impressionistic — you just get this blitz of unconnected images. However, Sue’s data had a very clear narrative behind it. We had a collection of points that the thief had visited with the phone, so I thought we should be able to get a smooth path between them.
First, I used the Google Directions API to map the likely route that the thief would have taken between known locations, as well as filling in some intermediary points, which was @blprnt’s idea from our earlier brainstorms. One of the cool things about the Street View panorama data (described by @jaimethompson) is that it shows the linkages between consecutive images taken by the Google car. So by calculating the heading from one point to the next and heuristically choosing links between panoramas headed in the right direction, we can access all the images taken along the way. Again using heading we can point the camera in the right direction, download the tiles we want, and stitch a frame together. Applying this to the thief’s route, we got a complete reconstructed path that plays back much more like a continuous video than my previous experiment (it evens out after the frantic first 30 seconds).
It’s a bit like if Google was driving the getaway car, starting downtown where the phone was stolen, and traveling over the city until it’s finally given back. Of course, we’re leaving out the pauses when he wasnt moving, and the temporal displacement of Street View images make this a kind of a weird frankendata — while the video retains some relationship to the truth of the human interaction behind it, it remains a kind of data fiction.
Oh, and for those who prefer the written word, theres always the driving directions.
Edit: some press love from Gizmodo and Flowing Data
STREET VIEW VIDEO VIA OPENPATHS API
[code] Python (2.6), GPL
Brainstorming with @blprnt this morning about what people might do with the new OpenPaths API, we thought it would be pretty awesome to see every place you’ve ever been via Google Street View.
Loading all of that up through the Google Maps interface seemed overly burdensome, so we figured there must be a way to pull the static tiles. Turns out there is (though it’s unofficial). @jaimethompson breaks it down for you.
From there, it was pretty straightforward to pull the points, scrape the images, and assemble the video. It includes points from September ‘10 to the present and a dozen or so cities, beginning in LA I think, but NYC clearly dominates. Non-urban spots arent captured well, and in Googleland it’s never winter. You might also notice that the granularity of the video increases at the end. That’s because at a certain point I start using the forthcoming OpenPaths app, which samples periodically, rather than the data from iTunes backups, which only looks at novel locations. The API pulls from both.
Want your own? I did this with python as usual — you can grab the code here if you’re interested (youll need PIL and the latest OpenCV bindings installed to export the video). This is a bit of a soft launch for the OP API as we gradually work in new features. Let me know if anyone gives it a try (especially if youre using a different language).
Noncoders fear not — we’ll hopefully be integrating something like this (but cooler and more blprnty) directly into the OpenPaths interface in the near future.
The Zeo is actually a surprisingly effective sleep monitor, an EEG sensor you wear when you sleep that syncs with a base station. By recording your brainwaves, it breaks down the night into periods of deep, REM, and light sleep, and stores the data on an SD card which you upload to a website for analytics (they should really give the base station some wifi). I say surprisingly, because I was startled at the degree to which I could reconstruct a bleary narrative of my night — unconsciousness, sequences of dreams, car alarms, etc — by looking the graph. I typically remember my dreams, and occasionally lucid dream spontaneously — people have already been hacking the Zeo to aid in their lucid dreaming practices.
Zeo has their own graphs, which are plenty nice, but I wanted to create a visualization which would allow me to see all of my nights at once. I was hoping that a radial arrangement would reveal ‘ripples’ of dream states that occurred at relatively constant amounts of time after falling asleep. Try as I might to cluster the sleep data in different ways to show the patterns, that idea turned out to be false (Ive since learned more about sleep and the roll that body temperature cycles may play, which perhaps prevents these from lining up.
In any case, this is an example of the visualizations Ive been making. It shows two weeks of sleeping, with a wedge for each night. The center of the image is the point of falling asleep, and you move later into the night as you move outward. Bright blue shows proper “deep” sleep and the greenish rings are REM, with everything else being fairly crappy. The number on each night is the “ZQ” from Zeo — a measure of sleep quality taking into account time asleep and the duration of each phase. Created in python with my drawing module.
VISUALIZING RELATIVE ENGAGEMENT LEVELS AT EYEO WITH THE AFFECTIVA Q-SENSOR
Affectiva is a spin-off of the Affectiva Computing group at MIT. Their Q-sensor measures skin conductance, which increases with “excitement, attention, or anxiety.” It also has an accelerometer and logs data at 32hz for over 24hrs, which is rad. I’ve been playing around with one in the lab to see if the metrics are useful as a measure of engagement in various of our prototypes, as well as working with the NYT customer insight group to see if they can play a role in evaluating site UX. Soon I hope to include it in my behavior modeling research.
In any case, I’m at the eyeo festival in Minneapolis. I thought it would be fun to see what would happen if I wore the sensor my entire time here, and to correlate the data with events during the day.
Im pulling in the raw Q-sensor data, and processing it to produce a visualization of two features in the signal: a low frequency “tonic”, which shows the general level excitement, and “phasic” features which are more discrete moments of engagement, eg, lots of social interaction. I made a quick web app to record events (although Ive done a terrible job of using it).
The huge caveat here is that the raw signal is very rich, and this visualization strategy is a very rough take at interpreting it. I’m making no claims as to what this actually represents. And I’ve labeled almost nothing. So if you find the graphic unsatisfying, it’s a reflection on me and not Affectiva’s great hardware. I’ve used my python drawing module.
I will also state that everyone’s presentations have been awesome, and I’ve remained fully engaged throughout, regardless of what the sensor says. Ill try to improve things in the coming days.
OK, off to bike down the Mississippi.
DAY 0 (6/26)
DAY 1 (6/27)
DAY 2 (6/28)
DAY 3 (6/29)
AN ACTIVITY HEATMAP FROM FITBIT DATA
The Fitbit is a very nicely designed device, basically an accelerometer plus clock in a belt clip that will wirelessly sync to a base station. It’s marketed as a fitness aid — on their site, you can check out charts and graphs that give you some sense whether you’re moving as much as you should be, set goals, and add subjectively reported measures like diet and mood. It’s a perfect example of a tool for a quantified self practice.
Though fitness is interesting, I’m more fascinated with the possibilities of using movement data for behavior analysis. My intuition is that the patterns of how much one moves over the course of a day or week or month is enough to infer quite a bit about an individual’s lifestyle and to potentially drive a predictive model.
Fitbit data, once gathered by the base station, is sent to your account on their site, which is accessible through an oauth-based API. Which is awesome. However, I was disappointed that the data they expose is very much geared toward their fitness market, and is preprocessed and interpreted along these lines. Significantly, unless I’ve just missed something, the raw, minute-by-minute data is not available, and everything is aggregated by day. While I can understand the motivation behind this, there is lot of power in the raw stream that’s not exposed.
However, prior to the release of the Fitbit API, the developer Eric Blue published a hack in perl that scrapes data from your Fitbit profile page — turns out that the Flash charts they display are powered by an XML feed with significantly more resolution than what is offered via the API. Based on this discovery, I wrote a python script that pulls the granular data.
Using numpy and my nascent drawing module, I created a heatmap of my average week in terms of intensity of movement (since 2011-01-01, wearing the Fitbit somewhat consistently). Behavior clearly emerges, and my bike to work each morning dominates.
From here, there are a lot of possibilities. Beginning to cluster days by type and to cross-reference this with other vectors, such as location, are my next steps. We’ve ordered like 10 of these things for the lab, so I’m psyched to see what we can do with them.
INTRODUCING OPENPATHS.CC
So last night a few of us at the lab pushed last night to launch openpaths.cc, a service that securely collects iPhone location data that people can later share with research projects. The basic idea is that Apple collects this data, but it’s really yours and you should be able to use it in awesome projects as you see fit. The site gives you a way to store it securely, and then to share it (on a case by case basis) with research/art/project proposals, as well as download it in friendly formats for your personal use. We also have some awesome map visualizations by Jer Thorp so that you can see where you’ve been, which is really payoff enough.
Posts by Jake and Jer and an awesome interview with Michael that gets into the themes we have been discussing, as well as other press.
Edit: iOS update 4.3.3. ends the collection of this data. However, old files have not been deleted. Further, our uploader app will scan Time Machine backups, so there’s still a lot of data to be gathered.
