Currently, the only organization offering benchmarks of CEP platforms is the Securities Technology Analysis Center (STAC). The benchmark is named the A1.
To date, Aleri is the only vendor to have published a benchmark performed in conjunction with STAC. And although the benchmark was not the A1, the information provided by the benchmark and attendant results is both meaningful and important. An article discussing Aleri’s benchmark can be found here and here.
So that means that no other CEP vendor has published a STAC A1 benchmark since it’s announcement, over a year ago.
Why?
I believe that it’s only possible to benchmark the lowest common denominator of CEP engines – each product has it’s own unique strengths but more importantly, each has differences in approach and development environment. Those differences can hardly be summed up in a benchmark.
If I put my sales guy hat on for a moment, I probably wouldn’t be too much into the STAC A1 benchmark for a couple of reasons – 1) it introduces additional variables into the sales process, and 2) it further removes me from a prospective client – what if the prospect simply reviewed the benchmark and never called?
If, instead, I put on my technologist hat for a moment, I know that benchmarks can be gamed (anyone remember Oracle?). And that my situation is probably going to be a little different than the scenario described in the benchmark. This can be attributed to adapters (what I call on and off ramps – I like the highway analogy for CEP), depth of attendant analysis, languages used, etc. As a technologist, I have a deep seated distrust of whether or not standardized benchmarks are important in anything but the most generic and commodity offerings.
And if I put on my investor hat, I’m not so sure that I would want to have a Capital Markets only benchmark – in Crossing the Chasm, the Geoffrey Moore(author) describes how some customers might not buy from a vendor just because of the customers the vendor already has. If I was looking for world domination, I’d hardly want to give the impression that our product was a ‘Capital Markets Only’ offering. Especially in today’s environment of ‘hi fi trading is bad’ and economic woes which have been broadly, and incorrectly, attributed to the unleashing of machines upon the markets. Just for the record, Goldman, which accounts (pre-theft) for a large majority of that trading, has been doing it for years. Given the majority of vendors in CEP focusing on Capital Markets, these probably aren’t important issues.
My Conclusion
I won’t be using STAC to facilitate benchmarks for my efforts; primarily for the reasons described above. But there’s also one more – all of the CEP vendors are reporting increased sales, and all with out STAC’s A1 benchmark.