Opher wrote a good post on what functions an Event Processing System should have.
Of the four general (and comprehensive) functions, I like #4 (the first three are a given and rather pedestrian):
4. Situation discovery / event pattern discovery: This function is to discover that some situation occurs without having a predefined patterns, using intelligent techniques. While the first three types of functions are more investigated (although I can’t say that all issues are figured out), the fourth one is still a challenge, since there are some experiments, but generally it is not well established yet.
If we look at what functionality the current vendors provide in this area, the most advanced functionality seems to be some type of OLAP capability. So if we were going to lay down a roadmap of how we get to #4 above, first we’ve got to figure out where we are, what’s required for #4 above, and figure out how to get there from here.
Live OLAP is certainly a good step in the right direction. Real-time mashups may help as well. Behind all of this is the internal operational data store – this is where all the transactions for an entity are processed. It may be called the Online Transaction Processing function within a company. Adjunct to this might be a large, denormalized Dimensional Data Mart to facilitate traditional OLAP. Also important is what’s going on in the world outside of the organization – external events certainly can and do influence customers and companies in unique and different ways. Consider for example, the power that Google wields over shopping decisions today. If you’re looking for mistletoe belt buckles, there may be a couple hundred vendors who share your same depraved sense of humor but if they’re not on Google, you won’t find them.
It’s in this area of the Dimensional Data Mart that what is referred to as CEP today could provide the foundation for dynamic and event driven OLAP. I don’t like the terms real-time and OLAP for this, but it’s the best the marketers could come with to describe event driven dimensional data analysis. So we’ll live with it.
Visualization plays a large role in this – a CEP engine can slice and dice data to your processor’s content, but if you can’t view it, it don’t mean much. So whatever tool a user decides to install on their desktop must be able to flexibly provide information in a condensed and meaning way; allowing dimensions, filters, slices, etc. to be moved around at will. This means that those resulting queries need to be coordinated with the CEP engine – to both provide a snapshot or benchmark and event driven incremental updates.
I’m familiar with what a number of firms have done in this area – the most complex and meaningful solution being at Citi. But, it’s ‘hard-coded.’ Is there any CEP vendor out there today that has abstracted these processes to a level where they might be generally applicable? I doubt it. But I bet Aleri’s Live OLAP comes close though.
But no CEP vendor today can claim anything remotely close to the vision that #4 above entails. I look forward to Opher keeping us all on our toes.