Processing
Random Etc. Notes to self. Work, play, and the rest.
Design.Yahoo: Web 2.0 meets War Games?
Whilst I wrestle with my reaction to the reception of Twitter Blocks, it's interesting to look at what other people in information visualisation are working on.
Yahoo's new design research outfit, apparently also known as yhaus, have just put up a site outlining their work so far. The first thing I noticed is that they've snagged an amazing subdomain: design.yahoo.com. The second thing I noticed is that compared to other teams I'm aware of in the field (including Stamen) they have a pretty good gender balance (a thorny issue but I'm noting it anyway). The third thing I'll note is the guest appearance of non-Yahoo work in the portraits of the team: I see Torrent Raiders, Fidg't... what else?
Sadly, all but one of the demos there so far is a big Quicktime movie. I know that with millions of users Yahoo has to be a stickler for browser support and compatibility, but I hope they get a chance to take this work live on the web as well as demo it in movie form. There's some solid realtime Flash and Processing work hiding in there, and people (OK, I) want to see it in its interactive entirety.
There's clearly some healthy collaboration and influence going on there (much as in our work, e.g. Ben Fry's zipdecode looms large over the interactive version of our Trulia search animation). Yahoo's Aaron Koblin is best known for his Flight Patterns piece, and this visualisation by Michael Chang of Yahoo trip planner data is very similar:
Likewise, Aaron's work on traffic patterns bears a close resemblance to Flight Patterns:
I don't want to pick on them too much, because it's really beautiful work I admire a great deal (and it might seem like sour grapes), but both the pieces I've highlighted do suffer from something we've tried to avoid at Stamen: animated information graphics on top of black backgrounds and vector maps can easily look like screenshots from a modern-day War Games:
I'm glad other people as well as us are experimenting in public, and I'm glad sites like Infosthetics and Visual Complexity are cataloguing our efforts. We need our own visual language around this kind of visualisation that doesn't resonate with the imagery of war.
Apple's iphone has made a strong impression with slick transitions in its interface design, but the maps application still borrows from Google's pseudo-shadows and static pins. The playful interfaces of the Nintendo's Wii games certainly offer a different path, but the rest of the games (and movie) industry's cinematic-realism aesthetic exerts a strong influence over our generation of designers and it isn't going to meet these goals any time soon.
It's a fairly regular topic of conversation at Stamen: how can you make a visualisation of e.g. 911 calls actually look like emergencies, and not birthday parties or toilet flushes, without freaking people out and without making it mundane? Is it possible to use great circles to connect air travel destinations without it looking like missiles? Can you animate growing and shrinking red and yellow circles on an aerial map without it looking like Gulf War I weapons company propaganda?
It's a bloggy weekend here at Random Etc, if you're reading along I'm definitely interested to hear your thoughts.
Criticism for Twitter Blocks
We finally kicked Twitter Blocks out of the door yesterday. It's been in development for about a month, mainly by Ryan Alexander and myself (but all Ryan whilst I was working on Oakland Crimespotting, he's a star). It's the first time we've done 3D things using Flash and it's amazing what's being done with Papervision3D at the moment. On the other hand, having 3D shoe-horned into the 2D Flash engine means the learning curve is a lot steeper than the native-3D Processing/OpenGL worlds Ryan and I are used to.
Whenever Stamen launches a new thing, my immediate reaction is a sigh of relief followed by a slightly obsessive-compulsive trawl of what people are saying about it online. (I use a combination of alerts from Technorati, Google Blog Seach and Bloglines to keep track).
The interesting thing about working with companies like Digg and Twitter is that your work inherits all the criticisms and detractors of those sites as well. Digg's users are clamouring for a picture section, so when we launched Digg Arc many of the responses ignored the piece entirely and chastised Digg for paying attention to visualisation and not to the main site. The same argument is already being used against Twitter Blocks, even though the amount of time Twitter's developers put into it was tiny compared to the amount of time they're putting into stability and new features.
Don't get me wrong: some of the early feedback we're getting is very positive, the team at Twitter have been very receptive and we're proud of our work. This much is good. Some of the feedback we're getting points out that the work isn't immediately understandable (I agree, and maybe we could do some more explaining, but I think we're OK for now).
However, there is also a strong and steady flow of negative comments that I've gathered here so I can think about them all in one place.
“Pretty visualization but I doubt its practicality.” PoppuPot
“Twitter Blocks is the kind of thing that demos well at conferences. Not too useful in real life.” Dave Winer
“exploring myneighbourhood : fun 3D view but so what? not sure i will do that everyday.” jean-michel gobet
“Well its interesting that its a new Twitter toy, but I just don't get it. Functional?” programwitch
“Puzzled but Entertained” … “Not really got the slightest idea why this is anything other than an interesting folly.” Tom Coates
“What is the point? Beats me.” Russel Heimlich
“I have to say I was absolutely gobsmacked by how utterly pointless it is.” EirePreneur
(The last one is, short of a personal insult, pretty much the harshest thing anyone has ever said about work I've been involved in.)
To address Tom Coates' point, if you're entertained we've done our job. If you're puzzled then maybe we can help explain things better next time. But I don't mind if some of our work is seen as a folly (“an extravagant, frivolous or fanciful building, designed more for artistic expression than for practicality”). Not everything that everyone does has to be useful or profound. (Nevermind that we've personally found Twitter Blocks a useful way to explore the Twitter network in the last few weeks, with frequent remarks of “I didn't know X was on Twitter”).
Jim Bumgardner (aka KrazyDad, author of O'Reilly's Flickr Hacks) addressed the “so what?” response to frivolous work in a blog post called Utility is Overrated a couple of months back. In the comments there's a comparison with Susan Sontag's Against Interpretation. In it, she states that “interpretation is the revenge of the intellect upon art”. I don't want to point at Twitter Blocks and say “art” (the Motorola sponsorship in particular makes that tricky) but I think that people are thinking too hard about things if they're looking for the “point” of it.
I'm not asking that people stop casting a critical eye over what's presented to them, especially when it's being hyped to death and it's commercially branded. It's fine to ask “what's it for?”, especially of new tools or things that aim to improve the efficiency or effectiveness of this or that. But why not also accept that some things might just be for entertainment and ask “am I having fun” once in a while instead of looking for a problem to be solved or an important statement to be read? Some things just are.
Processing Hacks Fixed
Sadly (due to my neglect as sys-admin) processinghacks.com hasn't been accepting new edits/users since I moved hosts a few months ago. I finally got around to fixing it today, so everything should be working now.
The site hasn't quite reached the ambitious heights I had in mind for it when toxi and I first put the wiki together and drafted the contents. Some of the hack suggestions are out of date now, and many of the existing hacks don't work with the latest release yet. That said, I hope to put some time towards kicking it into shape over the upcoming weeks and I'd be delighted if you'd join me.
I should also mention that there's a good incentive to get your hacks written up, because we're planning to move Processing Hacks to processing.org - you might have seen a note about that in the Learning section. Fame and fortune (or at least some geeky glory) await!
A quick (less certain) note on using Lucene in Processing
In the spirit of continuing our impromptu database week on Processing Blogs (my post, toxi's post, then Florian Jennet updating the sql library), I thought I'd post another quick example using Lucene.
Last week Ryan and I needed a reliable way to search inside a data set we were working with. I had previously tried and failed to write my own useful search routine for the same data, so I wanted to take a look at Lucene instead. (This week, I might have used SQLite, but I hadn't tried it last week!).
Lucene isn't a relational database like MySQL or SQLite, although it has a few similarities with the way most database engines speed up queries using indexes. That's because Lucene is "just" the indexing part. You tell Lucene about your data, one part at a time, and then construct queries to ask it which parts of your data match the query. The key thing is that you keep hold of your data yourself and structure it any way you like, Lucene keeps its own representation of the data for searching. Because of this it can index much more data than you can store in memory.
Anyway, I put up an simple example Lucene applet here that indexes the text of Time Machine by H.G. Wells from Project Gutenberg and lets you type queries against it. The text itself and the Lucene index it builds are both quite small (tens of KB, compressed), but the applet is around 750KB. This is because Lucene's core jar file is about 500KB so it's more suited to standalone projects and applications.
The code is kind of documented, but Lucene was really too much for me to understand properly in just one day. Nevertheless, again, I hope people find it useful!
A Quick Note on using SQLite in Processing
SQLite is a great standalone SQL database engine - not ideal for every situation (particularly large websites), but more than good enough to have already made its way into desktop projects from Apple, and more recently Adobe and Google.
My colleague Mike recently used it as a way to distribute data from a 511.org transit website scraping project he's been doing. I wanted to see if I could download his data (gathered and processed using python) and access it using Processing.
Web searches for SQLite and Java (java wrapper sqlite, etc.) turn up lots of matches, including this promising tutorial from Tim Anderson, but sadly the most prominent matches didn't work for me. It's a real pain to get up and running and installed in a reliable way, mainly due to the need to compile SQLite natively for each platform and then talk to the native code using Java.
Enter this amazing project, a JDBC library for SQLite written in pure java that uses NestedVM to compile the C code for SQLite into something any Java VM can use. It's frightening to think about the amount of misdirection, abstraction and interpretation going on here, but it worked first time and plenty fast enough for my purposes.
Take a look after the jump for the code I ended up with. I hope people find it useful:
Trulia Hindsight - Processing Prototypes
A few people have asked if I'm still using Processing now that I've joined Stamen (best known for their Flash work). Whilst it's true that I've been quiet on the Processing front, hard at work learning Actionscript 3 for Flash 9 to enable us to deliver Trulia Hindsight, much of that piece was informed by early sketches I wrote in Processing. (The graphs we made of a day of diggs were also made with Processing.)
Here are four movies we made from Trulia's data to get across the ideas we wanted to develop into Hindsight. It's a lucky thing that Flash 9 can shift many more points around the screen than Flash 8, otherwise we'd have been stuck. That said, it's still way behind Processing with OpenGL for this kind of visualisation, so choose your tools wisely when building a proof-of-concept in a different language to the one your project will be delivered in!
In the first few weeks of working with Trulia, we did some initial work exploring non-geographic views of their data such as tree maps and node graphs and so on. In the end though, the most compelling thing we came up with was to explore the different dimensions of the Trulia database in the form of animated maps.
Here are two movies, the first is San Francisco and the second is San Jose, showing the properties animated along an intuitive axis: the year they were built.
San Francisco by Year (Trulia Hindsight Prototype) from Stamen on Vimeo
San Jose by Year (Trulia Hindsight Prototype) from Stamen on Vimeo
And here are two more movies, also made with Processing, that show the properties that were sold in the last 10 years (under $2m), this time animated by sale price:
San Francisco by Price (Trulia Hindsight Prototype) from Stamen on Vimeo
San Jose by Price (Trulia Hindsight Prototype) from Stamen on Vimeo
The animations by sale price aren't available in Trulia Hindsight (yet) but we hope to work more on these less-intuitive dimensions for animation in the future.
No doubt whichever direction we go in next I'll still reach for Processing to try things out. It's the tool I find it easiest to think in, and although Flex Builder (based on Eclipse) is a great IDE, I still find myself wanting to bend Actionscript to be more like Processing when it comes to prototyping my ideas - it seems I'm not the only one!
Stamen is hiring a designer
Stamen, the company I work for in San Francisco, is hiring a designer.
If you're "someone who is a designer first and foremost, a coder a distant second, and who's interested in where these areas interleave" then please read the post on our site and consider getting in touch. I can't recommend us highly enough.
Some Attempts at Twitter Visualisation
Twitter is a website that asks only one thing, "what are you doing?" and aggregates your responses intermingled with the responses of your friends over the last 24 hours. If you let it (I don't) it will SMS you every time your friends update, or if you prefer (I do) it will send you an instant message instead. It will also let you update by web, IM or SMS. It's certainly an easy way to SMS a group of people and only pay for one message, but the IM and web integration mean it's more than just group SMS.
So it's not IM, SMS or the web, but it talks to all three. I like it. I want to hate it. I suppose I cheat, because I don't let it SMS me very often. And maybe because most of my contacts are a continent away, so I only get a few messages a day (they're all asleep). But there it is: I'm not stressed out by it, I'm still Getting Things Done (though that system's not for me, yet). Continuous Partial Attention be damned.
Of course, it's fully Web 2.0 buzz-word compliant, so it has an API that you can use to get data in and out. Not a super-useful one for visualisation, but useful enough to get started. Knocking some ideas back and forth at Stamen with Eric yesterday I decided it was worth trying his idea of plotting twitter activity on a circle. I started with a circle representing the previous 24 hours, rather than a 12 hour clock face, for several reasons:
- we wondered if there would be obvious patterns to spot, for example between our European and American friends
- the Twitter API only gives you the previous 24 hours of activity
- if a whole day fits in one circle, then circles can be overlaid to compare days
- Eric wants to make coasters of our visualisations... shh!
That's it really.
Given 24 hours of statuses I assigned each user a colour and plotted the status at an angle corresponding to how much of the day had elapsed. I joined each message to the previous message from that person, if there was one. Here is how my first pass turned out:
And here's another variation, still with a colour per person but ditching the arcs and instead using concentric rings for status messages. There are small dots again mapped to time of day. Moire be damned.
The top of the circle is midnight (PST), the bottom is noon. The data was sampled at about 4pm. I'm not sure where this circular/spiral visualisation is going, but if I revisit it I will probably unroll them into a rectangle in the hope that there is space to draw and read the messages. After all, the messages are what it's all about.
These were built with Processing, using the now-built-in XML support and the gorgeous PDF library. I haven't posted the applet because it doesn't work online with the Twitter API, sorry.
Travel Time Tube Map mentioned in New Statesman
My Travel Time Tube Map is mentioned at the end of Peter Ackroyd's City of Illusions article for the New Statesman. Here's a direct link if you've just arrived from there:
Processing 0121 is ready
A new version of Processing is available now.
If you haven't looked at Processing for a while, it's definitely worth picking up again. It's maturing very nicely - this week I've been using it at work, and the ability to create a single app with full-screen OpenGL graphics, PDF and movie output (thanks to Dan Shiffman's movie maker library) is absolutely great.
← Back to Home
© Random Etc.. Powered by WordPress using the DePo Skinny Theme.