I was reading Brenda Michelson’s recent post on the Real Time Web this morning and came cross this piece of enlightenment from Richard Tibbetts over at StreamBase.
… “Richard Tibbetts, CTO of StreamBase, explains that financial markets make up about 80 percent of his company’s customers today. Web companies are just starting to adopt the technology.
“You’re going to see real-time Web mashups, where data is integrated from multiple sources,” Tibbetts says. Such a mashup could, for example, monitor second-to-second fluctuations in the price of airline tickets and automatically purchase one when it falls below a certain price.”
… Real-time applications, whether using traditional database technology or Hadoop, stand to become much more sophisticated going forward. “When people say real-time Web today, they have a narrow view of it–consumer applications like Twitter, Facebook, and a little bit of search,” says StreamBase’s Tibbetts.”
I agree with Richard – most people’s view of the real time web is severely limited. Also, remember not all that long ago, concepts like Twitter and Facebook were new and uncomfortable – mention either in mixed company and you were likely to hear, “What would I use that for?” How quickly things change.
There’s another use case for real-time Web mashups that I’d like to illustrate a bit.
Imagine a terrorist suspect receiving a phone call. Who’s the phone call from? What news hit the web moments before that call? Who was the suspect emailing or chatting with at that time? What happened after the call – did the suspect then call someone else? Wouldn’t it be neat if all of this was presented in a mashup – related news, video, chat, email, phone calls, etc. in one place? Sound far fetched?
It isn’t. Intelligence agencies are doing this today – by combining this type of data with payments data from First Data and Western Union, terrorists are being targeted and arrested every day.
A real-time mashup is a step in the direction of providing context. CEP platforms today are dependent upon the provision of context. What the CEP engine is searching for must be defined, the relationships must be defined, the queries written, etc. How much information are organizations not gleaning from their data simply because they’re unaware of underlying relationships – or context?
By using advanced statistical methods, relationships between data sets can be predicted – by working backwards using Bayesian models one is able to highlight abnormally high covariances or contingent probabilities. Once a particular threshold is set and crossed, the events and context that triggered the abnormally high covariance can be presented for analysis. But the computer power required for this is nothing short of massive and the software must be concurrent and parallel. And the current product set in CEP land isn’t up to this task.
Yet.
I’m looking forward to 2010. It’s a Star Trek world out there; so much just waiting to be discovered.