Words and Actions
We can already see some early examples of video mixed with text in book-like packages. For example, Songs of Imagination and DigitisationΦ, In Search of Lost TimΦ and Inanimate AliceΦ are all interesting attempts to test these waters.
I won't go into these works in any detail here (I urge you instead to go and experience them yourself). But to my mind their biggest shortcoming is relevant to our theme and happens to be common to all of them: none is readily avalable to enjoy on mobile devices.
Exceptions are at last emerging. I've already mentioned the iPhone version of The Death of Bunny MunroΦ, which, among other things, features videos of the author, Nick Cave, reading his story. There's also VookΦ, a startup that's focused specifically on the creation of electronic books that mix text with video, and these too are available for the iPhone. Sadly, none of the works published so far are particularly imaginative or engaging. But the first examples of any new form rarely are, and at least they're a start.
There are perhaps also early signs of movies themselves moving towards a similar goal from another direction. RageΦ by Sally Potter is the first mass-market film to be released initially to mobile phones, and surely also the first to be created explicitly with this medium in mind. And though it isn't particularly good, it probably won't be the last of its kind.
Of course, by far the most visual form of written storytelling is the comic, and one idea I would like to see pursued is that of animated comic strips. KamikazeΦ is an iPhone app for displaying ordinary static comic strips (along with some basic navigational interactivity).
Photo: GenusApps.com
But how much more interesting for some of the frames to be animated (like the split screen scenes in the TV series, 24Φ) or responsive to the reader's touch (like in a video game).
Aside from comics, what about a blend of text and video in which the video acts as a backdrop (like a visual form of background music) to accompany shifts in location, perspective or tone? As in the earlier Shakespearean example, the results would in some ways resemble a subtitled film, but now with the video element firmly subordinated.
And so on... you get the idea.
This short section has necessarily only scratched the surface of what's possible with video, and the truth is that we won't know the true potential of any of these approaches until we try them. Yet given the mutually complementary strengths of prose and moving pictures it seems very likely to me that, in the right creative hands, the combination will become as important to narrative fiction as has been the marriage of celluloid and sound.
The reason that authors and publishers haven't done this before is not that this wasn't (sometimes) desirable, it's because it simply wasn't possible. Now it is.