Author Archives: colin

It’s Time to Kill the Elephant

Posted on July 3, 2011 | 4 Comments

Google started using MapReduce about 10 years ago.  Somewhere between there and now, Doug Cutting decided that he could copy it while at Yahoo and Hadoop was born.  Doug now works at a company named Cloudera who bills themselves as providing the “only solution that manages Apache Hadoop across the enterprise.”  Hadoop has been around for so long that even leading analyst firms are covering it, claiming that if your organization is an early adopter, you need to be looking at Hadoop.  Hear that Luddites?  Time to get moving.

Hadoop Is Picking Up Speed

MAYBE THERE’S A REASON FOR THAT

Recently, Google announced their move away from batch based MapReduce to something a little more real time.  Seams like it was taking days to update search results with something that you might be interested in.  Google never open sourced their implementation of MapReduce, which is said to be at least one or two orders of magnitude faster than Hadoop.  But still not fast enough.

EVEN YAHOO IS GETTING INTO THE ACT

Yahoo used to have a substantial relationship with Cloudera, at least according to Cloudera.  But now even Yahoo have started a company to distribute and support Hadoop.  Yahoo calls their company hortonworks.

WHAT THIS MEANS TO YOU

Without getting into things like how much data and corresponding analysis you need to do before Hadoop makes any sense to use at all (most companies are not going to see any benefit at all), let’s recognize something.  All of these recent shifts from companies like Google, Yahoo, and others no longer see a competitive advantage in batch based MapReduce.  The future has arrived, let’s look at some evidence.

REAL TIME HADOOP

MapReduce

There have been more than a handful of releases in this space – like S4 from Yahoo, HStreaming, Storm, and several NoSQL databases now supporting this, it means that for competitive advantage, you’d best be getting some real-time.  And getting it soon.

WHAT IS REAL-TIME?

Database vendors like DataStax, who support Cassandra, claim to be real-time.  They’re not.  They say that they’re real time because as soon as you commit data to the database, it’s available for query.  That’s supported by just about every database and hardly a new and exciting feature of NoSQL.  Even one of their big shots left to start a real time company named Platfora.

CONTINUOUS QUERY OR EVENT-DRIVEN

Rather than thinking about what real-time is or is not, let’s worry about event-driven.  Let’s use an example:

I’m a manager, and I want to know when the average time on my website dips below 2 minutes.  Using the ‘my database is real time because the data I send to it can be queried after I write it’ means that I would have to run this query repeatedly at regular intervals to catch this mounting exodus from my web properties.

THERE’S GOT TO BE A BETTER WAY

And there is, it’s called continuous query.  I ask the same question as above, and there’s some process somewhere that’s sessionizing data from my web logs and injecting that into that server – the same server that I sent the query above to.  And when that process finds a web session that lasted less than 2 minutes, it sends another ‘row’ to the program that submitted that query.

ABRACADABRA

Waiting for Hadoop Query

And then I’ve got it on my dashboard, and can switch out the really badly designed page the marketing department A/B’d this morning.  That’s continuous query, or event-driven.  The term real-time didn’t even need to be mentioned.  If I was running batch based Hadoop, that notification could have taken hours, or days.  How much money would your company lose if that happened to you?

BACK TO MAP/REDUCE

I am Node of Cluster...

So if I can do the above, why do I need MapReduce?  MapReduce is an algorithm for splitting work up, distributing the work out to nodes where the data lives that needs to be analyzed, and then gathering the results.  If you’re problem is big enough, MapReduce might help you get it done faster than using just one machine.

BUT EITHER WAY

If you’re running batch processes, like some well known web properties are and think that Hadoop holds an answer to your ever dwindling ad revenue, you’re mistaken.  And if you’re that CIO, the other thing you need to be working on is most likely your resume.

GET YOURSELF SOME CONTINUOUS QUERY, AND GET COMPETITIVE!

and thanks for reading!

→ 4 Comments

Posted in Opinion

Hope, FPGA’s, High Frequency Trading and the New Market Access Rules

Posted on February 3, 2011 | 4 Comments

I recently became aware of an emerging practice most likely being implemented by clearing companies at the low end of the capitalization sprectrum offering a unique solution to the recent Market Access Rules.

NO UN-FILTERED DIRECT ACCESS

What the SEC is trying to do is remove, or reduce the opportunity for either crooks, idiots, or algo’s gone wild from doing bad things to the market. Under the new rules, order flow needs to be monitored. This is not something that the HFT crowd like to hear, because it slows them down. So a couple of innovative idiots got together and came up with the solution that I’m going to describe here.

HOPE IS NOT A STRATEGY

Let alone a comprehensive compliance or surveillance strategy. What the idiots are doing is putting a ‘black box’ between the HFT firms FIX engines and the execution venues. The box, most likely powered by an FPGA device, scans the outbound order flow, and if it finds something it doesn’t like, it messes up the payload of the FIX order so that the execution venue (hopefully) rejects the message. Why is this done this way? Because the ‘black box’ is both out of process – both the source of orders and resulting executions, etc. are behind FIX engines, and because the ‘black box’ isn’t actually maintaining connections between the HFT firm’s order generators and execution venue.

A PICTURE IS WORTH A MILLION REJECTS

This is a little complicated, so let’s look at this picture:

In the diagram above, the ‘black box’ isn’t maintaining FIX connections to either the HFT’s order generators or the execution venue.  So, the ‘black box’ can’t just reject the order if it’s out of bounds back to the order generator because then the FIX sequence #’s get all mixed up.  There’s a little more to this, but you get the general idea.

YES, THIS IS REAL, AND I’M NOT KIDDING

So, this whole thing is designed so that an examiner can come into the Olde Thyme Highe Frequency Trading Shoppey and be escorted into the back room and shown the shiny box.  Wow.  Are you serious?  ”Look, we’re making sure that this firm isn’t doing anything wrong – we’re actively monitoring the flow and if they do something we don’t like, we shut them down.”  Right, they shut down the order flow attached to the box.  What about the order generators that the examiner doesn’t see? There’s a host of issues here, but we’re going to focus on one – and it’s a doozy.

DENIAL OF SERVICE ATTACKS

So, we’ve installed the OMICRON 5000 monitoring device and our HFT/algo team is ready to do business.  And everything is fine.  They’re trustworthy chaps and have no intention of gaming the system.  (cough cough).  But their first algo goes completely nuts.  And gets shut down by the clearing firm.  But it doesn’t really get shut down.  Instead, it’s sending 1000′s of malformed FIX messages to an execution venue per second.  Or maybe 10,000′s of malformed FIX messages to many execution venues.  Wow.  In the internet world, we call this a denial of service attack – flood a destination with more traffic that it can handle.  And while the execution venues can handle normal traffic, what about rejecting every message? Is every execution venue out there ready for this?  I don’t think so.  I’ve been involved with FIX longer than I’ll admit to in public, and I’ve seen a lot of testing  - “Yeah, reject worked.  It worked fine.  I mean, we never thought they’d be sending 1,000′s of orders a second that would all reject…”

I DON’T KNOW

What should be done about this.   I have lots of ideas about surveillance and how it should be done.  But I don’t have any thoughts about this.  Mostly because I never thought anyone would be so stupid as to ever actually deploy this type of ‘solution.’  Where’s the SEC when you need them?

THANKS FOR READING

→ 4 Comments

Posted in Capital Markets, HFT, Surveillance

Cassandra’s Data Model

Posted on January 27, 2011 | 4 Comments

As we prepare to implement our Market Data repository to facilitate algo development and back-testing, you should have downloaded Cassandra and installed it by now.  What, you haven’t?  Well, click here, get it done and then come back for some fun.  To get things up and running once you’ve downloaded Cassandra, click here for some guidance (this assumes you’re running Linux but should point you in the right direction if you’re running Windoze).

CONFUSION

Most of the explanations I’ve read about Cassandra’s data model first extol the virtues of NoSQL and the evils of Relational Databases.  And so while getting the reader caught up in this mythic struggle that summons images from Tolkien’s middle earth, the point is lost.  And that point is?

IT’S ALL ACTUALLY QUITE EASY

Cassandra thinks about data the way we think about data.  Most of us think about data in rows and columns.  So does Cassandra.  But it also alleviates some extra stuff we don’t need while adding some stuff that we do need.  And that can be a little disconcerting initially.  To make things easier, let’s first describe a goal for our exercise.  We’d like to get a day’s worth of market data, by symbol, in ascending time order.  Also, we might like to get the data for a slice of time within that day.  Like, “give me all the BBO’s for American Airlines for May 20th, 2010,” or, “I’d like to see the BBO’s for American Airlines for May 20th, 2010 between 1 and 2pm.”  Let’s jump right in.

LET’S GET OUR DATA

As we subscribe to our favorite market data feed, we receive something like:

  • Symbol,
  • Bid,
  • Offer,
  • Bid Size,
  • Offer Size.
  • Time Stamp, and
  • Seq # (most quote vendors provide a Sequence # because multiple quotes can occur for any given Time Stamp)

We’re going to call this a column family.  Cassandra’s analog for a table is a Column Family.  You can see why this fits so well above – the columns that belong to the symbol AA comprise a family of related information.  I’d like to store this data by symbol, so later, I can retrieve it.  Using the Cassandra client (cassandra-cli – it’s in the bin directory where you installed Cassandra), let’s create the BBO Column Family.  It looks like this:

create column family bbo with comparator = UTF8Type
and column_metadata = [
{column_name: symbol, validation_class:UTF8Type},
{column_name: bb, validation_class: UTF8Type},
{column_name: bo, validation_class: UTF8Type},
{column_name: bbSize, validation_class: UTF8Type},
{column_name: bSize, validation_class: UTF8Type},
{column_name: timeStamp, validation_class: LongType},
{column_name: seqNum, validation_class: LongType},
];

And now that we’ve created the schema, let’s insert some quotes.

Set bbo[‘AA’][‘symbol’]=’AA’;
Set bbo[‘AA’][‘bb’]=’123.34’;
Set bbo[‘AA’][‘bo’]=’123.84’;
Set bbo[‘AA’][‘bbSize’]=’100’;
Set bbo[‘AA’][‘boSize’]=’200’;
Set bbo[‘AA’][‘timeStamp’]=1234;

What happens when you execute a list bbo command now?  So, that’s easy enough.   So what happens as we get the next quote?  Well, we go to insert our data like this:

Set bbo[‘AA’][‘symbol’=’AA’;
Set bbo[‘AA’][‘bb’]=’125.34’;
Set bbo[‘AA’][‘bo’]=’125.84’;
Set bbo[‘AA’][‘bbSize’]=’100’;
Set bbo[‘AA’][‘boSize’]=’200’;
Set bbo[‘AA’][‘timeStamp’]=1235;

And then to see our data, enter this command (again):

List bbo;

When we use the ‘list bbo’ command, we’re only go see that data last inserted for that row key.  What happened to the previous data?  It was over-written with the new data.  So if we wanted to save each quote, we could combine the timestamp with the column name and then we’d be inserting unique columns each time and we’d be fine.  But there’s a different way to do this.

BIG DEAL, I DON’T SEE ANYTHING DIFFERENT HERE

And you don’t, because we haven’t started introducing the special sauce yet.  Well, we kind of did.  In the schema definitions above, you’ll notice we didn’t say that much about what we could or couldn’t insert into a row.  We just started adding columns dynamically.  So, each row, which is identified by a key, can have different columns in it and even a different number of columns.

WELL, THAT’S NOT GOING TO WORK

So, how do we keep track of all the quotes for our symbol?  First a little clarification, the Column Family above is really BBO, and we’ve inserted a row identified by the key, ‘AA” and some associated tag/data value pairs.  Think of this as a map of maps.  So now, we need to insert the bits that change for a given symbol over time.  How could we do that?  We create a Super Column Family of course.  A Super Column Family contains Super Columns.  A Super Column is kind of like another row of data – so using our example above, the Super Column we’ll be inserting consists of the BB, BO, BB Size, etc. The data above gets inserted using [AA] as our row key, and we need to pick a key for the Super Column that contains the quote data.    Let’s pick Seq# as our Super Column key.  Our row key is still Symbol, and I’ve prepended the date to it.  This way, all the data for a day’s worth of AA will be in the same row.  This is called a compound, or aggregate, key.  It looks like this:

create column family sbbo with column_type = 'Super' and comparator = ‘BytesType’
and column_metadata = [
{column_name: bb, validation_class: UTF8Type},
{column_name: bo, validation_class: UTF8Type},
{column_name: bbSize, validation_class: UTF8Type},
{column_name: bSize, validation_class: UTF8Type},
{column_name: timeStamp, validation_class: LongType},
];

And the insert statements look like this (we’re using the Seq# as the key – that’s the Super Column key right after the row key or, ’20100124:AA’ below):

Set sbbo[‘20100124:AA’][1234][‘bb’]=’100.00’;
Set sbbo[‘20100124:AA’][1234][‘bo’]=’101.00’;
Set sbbo[‘20100124:AA’][1235][‘bb’]=’101.00’;
Set sbbo[‘20100124:AA’][1235][‘bo’]=’102.00’;
Set sbbo[‘20100125:AA’][1234][‘bb’]=’100.00’;
Set sbbo[‘20100125:AA’][1234][‘bo’]=’101.00’;
Set sbbo[‘20100125:AA’][1235][‘bb’]=’101.00’;
Set sbbo[‘20100125:AA’][1235][‘bo’]=’102.00’;

Now let’s see what’s in the column family:

List sbbo;

So now it looks like we’re able to store a set of quotes for a symbol for any given day.  Bingo.

All we’ve really done here is add another map – so we now have a map (Date, Symbol) that contains a map (Symbol, Quote) that contains another map (Quote, QuoteField).  Or, what we’ve done is figured out a way to represent the potentially sparse fact tables resulting from large data analysis (OLAP) projects in a concise and easily addressable fashion.  Told you it wasn’t that hard.

GIVE ME MY DATA

So, now that we’ve inserted a couple of rows of data, let’s see how to get our data.  From above, we want to:

  1. Get all the data for a day’s worth of a symbol, and
  2. Get all the data for a slice of time during a day for a symbol

Assuming you’ve entered the statements above to insert the data, we can retrieve an entire day’s worth of AA with this simple statement:

get sbbo[‘20100124:AA’];

Now that we’ve gone over some of Cassandra’s basics, we’ll get a little more into it in upcoming posts.  That’s where we’ll cover the goal in #2.

THANKS FOR READING

→ 4 Comments

Posted in NoSQL

Building a Back Testing Platform for Algorithmic Trading

Posted on January 16, 2011 | Leave a comment

On this continuing series, I am examining thoughts and specific implementation details around building a back-testing platform for algo trading.  Eventually, we’ll see where complex event processing plays and how to implement it.

Appendix to Part One – The Data Format

Rather than looking at various database solutions first and then trying to define the problem in terms of those solutions, let’s first examine what market data looks like.  In its most simple form, market data looks like this (there’s usually a little more, but this is fine for our purposes):

  • Date: The date of the market data,
  • Time: When did the quote occur during the date,
  • Sequence #: Most quote or trade streams include a sequence #,
  • Symbol: What security is this data for?
  • Best Bid: The best bid (we’re going to concern ourselves with BBO data for this series, it’s easier),
  • Best Bid Size: How much does someone want to buy at the Best Bid,
  • Best Offer: The best offer,
  • Best Offer Size: How much does someone want to sell at the Best Offer.

Consider this chart:

Market Data


If we break down data, we can successively see how data might be arranged on disk for subsequent reading.  We want to read the data very quickly.  If we were using a standard relational database, it’s easy to see that we might be replicating some unnecessary data during the reads.  And we if use a typical columnar database, we can see that there are chunks that could be read together increasing throughput.

For example, for any given millisecond (Time) in a quote feed, there may be more than one symbol with a quote.  In fact, that’s quite common.  So replicating the time stamp is superfluous.  So if we had a table for a date’s worth of data, then we’d have a Time column that was replicated throughout the table.  No reason to do that.

Looking again at the data, we can see that, for a given time, there might be multiple quotes available for multiple symbols.  We’d like to read those in order as a little group.  By organizing the data on disk as a flattened multi-dimensional map of maps, we would:

  1. Start with a given day (our table),
  2. Start with a time (our row),
  3. Read each quote in sequence # order (our column)
  4. Process (do something)
  5. Increment the time, and go to #2 above until we run out of data (lather, rise, repeat)
  6. Put the $ in the bank

If we could write this data structure to disk as we get it from the quote feed, and had fast enough disk, we could keep up with the feed.  If we needed to create some indexes on the data, we could easily do that as well.  We’d simply create another table that would hold an inverted list of time and sequence #’s by symbol.  If we want to process a day’s worth of data, we’re all set.  If we want to process a symbol, or group of symbols, we’re all set.

So, to summarize, we need a hybrid approach.  In some places, we want rows of data – storing columns of data via a unique key.  In our case, that’s the Time column above for a given day.  The row above is Time, the column (or Super Column) is the Quote for a Symbol.  The Super Column’s key is the Sequence #.  Can anyone guess which database might fit nicely for this use case?

In my next post, I’ll describe a formalized data structure and it’s implementation.  I might even include a little code for all you #NoSQL guys and gals out there.

Thanks for reading!

→ Leave a comment

Posted in Algo Trading, Capital Markets, CEP, NoSQL

Tagged

Building a Back Testing Platform for Algorithmic Trading

Posted on January 14, 2011 | 3 Comments

In this series, I’m going to outline in general, how to build a back-testing platform for the creation, tweaking, and subsequent execution of algorithms used in electronic trading.

Part One – The Data

I recently made some comments on Vertica’s blog in regards to what I considered to be a fairly bold claim.  They said that Vertica was the only real column store.  But even if they are, so what? In my comments, I alluded to my belief that we optimize problems to solutions – we try to fix stuff using what we’ve got in our toolbox without having to run to Home Depot.

The real test is when the rubber hits the road – how do you actually solve a problem in a new way that’s motivating.  And by motivating I mean the solution addresses the issues, enables new capabilities, and is economically attractive.

So rather than tell you that DarkStar and our approach to processing both real-time and historical data (there’s a difference?) is the Real Enchilada, I thought I would illustrate a real world use case.

Let’s say you want to store a bunch of market data.  And I mean a bunch.  You want to store every piece of market data for the whole US Equities market.

And you’d like to have this data so that you can run analytics on it.  Or maybe even back-test strategies for buying and selling stocks.  So let’s assume that you’ve got some java code lying around to do that.

For our example, we are interested in seeing whether or not using volume average weighted price strategies actually work.   In our example, we will pretend that we are buying a lot of stock, and the theory we want test is whether or not buying that stock during the day when it’s lower than it’s weighted average price will give us a better average price during the day than just going with the flow (often referred to as volume participation).

We are all familiar with how relational databases work, and anyone who’s been in capital markets for a while knows how futile it would be to use something like Oracle for this due to cost, hardware and just how difficult it is to get the data into the database in the first place,

Oh that’s right, I forgot to tell you, we are going to have to load this data first.

I am not going to go into the relevant benefits of a column store here either, you can check out many other websites for that.

Instead, let’s look at some issues.  First, I would rather load the data directly into the database as it happens.  Staging the data separately is costly and error prone. In addition, what happens when you decide to load that data and encounter a problem that can’t be fixed in time for the next market day?  What if you actually run out of space or compute to get caught up?  Well then you can’t back test the next day to further refine your algorithms.  Algo’s should be tweaked every day.  New algo’s need to be developed to remain competitive.  So here, a database error costs real money.

So I need a fast data store.

As I am loading the data, what happens if one of my disks goes boom? Or one of my machines go boom? Well, now I have a problem.  If I fail over to another datacenter, how do i reconcile? What a nightmare!

So I need a data store that we can take a sledgehammer to and it will keep running.

Hey, if I have this big historical data store, I still need to query it while it is being updated.  Ideally, I would like to also be running analysis and back testing during the day.  Scheduling jobs to run at night is so very ’90′s.

So my data store has to facilitate both interactive query and batch analysis.

But wait, doing all of this means that I am going to have to figure out how to use the same code for back-testing that i use to generate orders during market hours.  It’s either that or use some visual, script based or different harnesses for my java or C++ code.  Yet another nightmare.

So, I would Iike to run the same code against my historical data store that I also use to generate orders during the day.

There’s a bunch of other stuff too, management, instrumentation, removing old data that I don’t need for back-testing, all the stuff we associate with normal day to day big data operations. We need to know what’s going on during the day so that we can be proactive.  There’s gold in that data!

And one last thing, it would be really cool if most of this technology wasn’t proprietary.  I mean let’s face it, firms that talk more about their investors on their websites than their clients can’t possibly have my best interests at heart.

This is a tall list.  Let’s knock it down, one by one.

Here is a diagram for your consideration.

The diagram isn’t very technical, and that’s on purpose – I’m outlining an algorithm, or methodology that may or may not solve our problem.

In the diagram, I’ve depicted the database as a cluster of machines.  Instead of using one big machine backed by a SAN, I’m going to use a number of machines.  Each of those machines is going to connect to the Market Data source and get data.

As we receive the data, we’re going to take a peek at it, and determine where in the cluster that data needs to live and while we’re doing that, we’re going to right it to disk.  A background process will make sure that the data ends up on the node we want it on.  More of why that’s so incredibly important in Part Two – Analyzing the Data in this series.

Also, I’m going to ask the cluster to replicate everything we’re writing to it – we’re going to end up writing the data a total of 2 times in this example.  I might usually suggest 3, but we’ve got two data centers running the same solution, so I’ll actually have 4 copies of the data.

Why write the data to three nodes in the cluster?  First, if a node goes down, I still want to be able to write data.  If the node that goes down is the primary node, I’m going to remember that and when that node comes back up, I’m going to write all the data to it as part of its “Welcome Back to the Cluster Party!”  And second, if I’m reading data from the cluster (remember, we’ve got algo’s running and users querying this data), I want my data.  If a node goes down, your users don’t really care – they just want their data.  By replicating the data across multiple nodes, I achieve high availability without having to fail-over to another instance or data center.

Ok, we’ve got the Sledge Hammer test handled, which is cool, but everything I’ve described above sounds like it’s going to take a lot of time and that the system is going to very slow.

Not true.  Each node in the cluster above is subscribing to market data.  So if one machine can ingest X messages per second, then a cluster of 10 machines should be able to ingest 10 * X messages per second, right?  Let see what that means in a real world example:

On May 20, 2010, there were about 1.1 billion BBO messages as published via SuperFeed (NYSE’s market data platform), those quotes represent the best bid, bid size, offer, and offer size for each stock at any given time.  In terms of messages per second, that’s about 50,000.  In terms of size per second, that’s about 4,500k per second.  Hmmm, chunky!

These are intimidating numbers.  But if we divide the problem up a bit, and use 10 nodes in a cluster, each node only needs to ingest about 450k per second across 5,000 messages.  All of a sudden, we’re dealing with something quite reasonable.

So, now we’ve got a cluster that can load the entire market real time and it’s redundant.  What about analyzing the data?  That’s in Part Two – Analyzing the Data which I’ll post next week.

→ 3 Comments

Posted in Algo Trading, CEP

Predictions for 2011

Posted on December 19, 2010 | 2 Comments

Some predictions for 2011.  In no particular order or importance.

1. CEP – The Feature

There’s a couple of things going on here.  The most important being that Mark Palmer is writing blog posts about Richard Tibbetts writing blog posts on the Tabb Group’s site about writing better software on Wall Street (because software startups write better code and deal with bigger problems than firms on Wall Street, especially exchanges, do, right?…).  No customer win.  No ‘yet another use for CEP.’  Just good old fashion buzz-word copy designed to remind people that there’s still one stand-alone CEP vendor. (click click click, is this link working?)

CEP will become a feature of larger, more established, horizontal offerings.  Because once the opportunities in Canada and South America dry up (you know those hotbeds for financial engineering, right? Canada and South America?  You don’t?) the reality will sink in.  What’s that reality?  That no one in NYC is buying CEP engines for HFT trading anymore.  Why?  The CEP vendors don’t know why.  Even Apama has seen the light.  You can now process events one at a time, or within the context of their CEP engine.  Stunning.  They’re pushing the ‘platform.’  Good.  It’s about time Tibco got some competition.

So this coming year won’t be the year of CEP, it will be the yesteryear of CEP, like, “Remember yesteryear, when we all thought CEP was going to be really hot?”  CEP will become a feature found in Event Processing Platforms.  And we’ll finally start to see adoption of those platforms in large, house hold names.

2. Hadoop & Analytical Databases

People are going to begin realizing that databases that incorporate map/reduce into their architecture will be *no faster* than Hadoop.  Why?  Go buy a book about Hadoop and then sit down with a piece of paper and pencil.  Databases are designed to support multitudes of users all asking different questions.  And vendors would like to have us believe that they’re also all running long lasting jobs taking advantage of their shiny map/reduce implementations.  Long lasting.  As in, not interactive.  As in, batch jobs running on a map/reduce framework.  The bigger the job, the less increase in throughput the analytical database will be able to offer.  It’s physics.  Save your money.  So this coming year, we’ll probably see further consolidation and some unexpected exits in this area.

3.  And Speaking About Hadoop

Batch is dead as competitive advantage.  As Jeff Jonas loves to point out, and points out really well, data velocity is growing.  And the rate at which data velocity is growing is increasing.  And companies can’t process the data they have today.  And many companies are actually making bad decisions with more data, not better decisions, why?  Because the data has lost most of its value once it’s been crunched.  You can only take a batch system like Hadoop so far.  But right now (at least for the near future) you actually still need to incorporate some ideas from the batch world for everything to come together.  So this coming year, we’ll see more people start treating Hadoop as either a must have to compete (minimal cost of entry) or “How f(*)(* much does that cluster cost to run?  There’s got to be a better way!”  It’s time to outsource your Hadoop cluster.

4. Real Time & Batch

I’m not saying that Hadoop or Map/Reduce is a waste of time and money like some vendors who make outrageous claims like, “Google has stopped using map/reduce.”  That’s idiotic.  Please put the kool-aid down, you’ve had enough to drink.  What’s important is the ability to analyze data in flight, to make decisions while there’s still the opportunity to have an impact.  How does one accomplish this?  By having a context in which events are analyzed.  How is context built?  Via the constant processing of events in flight, constructing and augmenting context, and supplementing that context with the result of monster jobs run on gazilla-bytes of data (like that? gazilla- it’s mine).  So this coming year, we’ll see a focus on moving analytics to real time.  (deeper analytics than VWAP-please!)

5. The Big Picture

There have been some really neato-keeno entries in the visualization space.  Things like Tableau, Spotfire, etc.  But they’re great for analysis of relatively static data sources.  They’re not for real time stuff.  Even offerings from vendors like Panopticon, which can provide some insight into multidimensional data sources updating in real time, really offer quite limited analysis tools for big data.  So in the coming year, we should see more focus on real time data mining.

6.  Did He Say, “Big Data?”

Yup.  I said it.  I’ve heard people define big data as more data than will fit on one machine.  Those people haven’t worked on the machines I’ve worked on.  I define big data as “when you can’t turn your data into actionable intelligence fast enough to have an impact during the window of opportunity.”  Or something like that.  I’m not in marketing.  Common patterns for analyzing big data are emerging that tie as-it-happens analysis with context and historical data.  The lines are blurring.  It’s all just becoming data.  And business wants it all.  Even my father-in-law, in Germany, asked, “Ja, SAP wants me to store my data in ze cloud.”  Everyone knows about big data and 2011 is going to be all about it.  Getting it.  Storing it.  Analyzing it.  Visualizing it.  And then we’ll see the real emergence of privacy issues, like what happens when our respective governments start using simple tools like the ‘People You May Know’ from Linkedin during child pornography investigations?  It’s going to happen.  Our government isn’t ready.  The legal infrstructure isn’t there.  There will be the formation of chaos in this regard in 2011.

7.  Computer Network Attack Platforms (CNA)

In 2011, something important is going to happen.  The general populace will be made aware that, in addition to all the traditional, ground-based, we can’t win engagements in places like Afghanistan and Iraq, that we’re also involved in a different kind of war.  One that rages on every day, one that runs 24×7 and involves facets of technology present both on earth and in space and in the Internet.  And that’s cyber warfare.  World War III is here already – China has already ‘stolen’ the Internet from the United States for about 15 minutes, diverting the majority of our most important Internet based traffic through their country for storage and analysis.  Doesn’t that raise an eyebrow?  It should.  One of the biggest things that happened in 2010 went off without a hitch and without a great deal of coverage.  Iran’s nuclear power plant, the one everyone was afraid of, was rendered inoperable by a virus.  Because that plant is inoperable, Russia is going to continue to make big money.  And Israel is going to sleep easier.  Odd bedfellows, wouldn’t you say?  We’re going to learn more about CNA’s in 2011.

7.  What about NoSQL?

Yawn.  It’s going to be a TWO BILLION DOLLAR market.  Just like CEP was.  Really.

Happy New Year!

→ 2 Comments

Posted in Opinion

Tagged

What's Wrong With Complex Event Processing?

Posted on November 20, 2010 | 6 Comments

I spend a significant amount of my time keeping up with advances in processing high velocity big data.  Over the last year, I’ve watched the NoSQL camp grow a lot.  And now, some folks are even forecasting a market approaching $2 Billion USD by 2015. The last time I saw that kind of trajectory for a new software category was for Complex Event Processing.  So without casting any undue aspersion on the NoSQL camp, let’s talk about why CEP has so dramatically failed to generate the returns venture capital firms were so sure they were going to achieve.

WHAT IS EVENT PROCESSING?

Event Processing, or Event Driven Architectures, means nothing more than processing an event one event at a time; preferably sometime shortly after they occur.  The opposite of this is Batch Processing, which means batching events, or messages, or what most of the world would call a row, of data and processing them together.  In batches.  Sounds simple enough, right?  All of you reading this blog post have used an Event Driven Architecture.  In fact, you’re using one now – it’s in your browser.  Can you imagine what the user experience would be if your browser ‘batched’ up all of your mouse clicks and submitted them every 30 seconds?  Event Driven Architectures promise the same type of agility and increased user experience for line of business and consumer applications that you’re experiecing right now.  In fact, it’s probably hard to think about using the web in batch mode – it just doesn’t make sense.

WHAT IS COMPLEX EVENT PROCESSING?

For the most part, a marketing phrase.  That’s right – and again, for the most part, it’s completely meaningless.  As an early and continuing contributor to this particular area of technology, I remember when StreamBase, Apama, myself, and others called this field Event Stream Processing.  Then one of those firms marketing departments decided to differentiate.  I’ll leave the specific firm to your intuition.   So, what is Event Stream Processing?  That’s much easier to answer.  Event Stream Processing is Event Processing with four additional key components:

1. Continuous Query

Rather than having to poll a server for an event, using ESP , the user of the system issues a query and is subsequently informed with events, aggregations, or patterns that satisfy the specifics of the query.  This happens continuously, until you stop the query.

2. Windows (Time and/or Length)

Using ESP, the user can ask, as an example, for an average value of some key over either a time or length window.  Something like, ‘Give me the average amount of time people have spent on the homepage in the last 10 minutes.’  This query would provide an updated average either continuously, or perhaps at regular intervals.

3. Pattern Matching

With Pattern Matching, I’m able to define a series of events that fit a pattern, and then be notifified when that pattern is observed.  Usually within some Time or Length Window.  So, I might ask, “How many users are going from the “Home Page” to the “About Company Page” and then clicking on “My Profile” during a rolling 10 minute window”.”

4. A Language

Tying all of the above together in a neat little language is a cool idea – it makes using these features easier. At least, that’s the theory.  And this is one place where CEP has gone wrong and is not the general computing revolution that myself and others have hoped for.  I’ll expound upon this after a brief distraction in the next 2 paragraphs.  Please bear with me.

WHERE IS COMPLEX EVENT PROCESSING USED?

Even Mark Palmer, who is usually extremely bullish about CEP and probably sprinkles it on his breakfast cereal, has recently admitted the CEP is only hot in Capital Markets.  While I might disagree a bit with Mark, which is nothing new, I think we can all agree that CEP, at the $200M total market size is far less than we had all hoped for.  Frankly, it reminds me of the FIX engine vendor battles – I was an early provider there too – and we all ended up fighting over an ever shrinking market place.

WHAT IS THAT MARKET ANYWAY?

The current vendor set of CEP tends to focus on Capital Markets.  But not really.  It focuses on an even smaller slice of Capital Markets called High Frequency Trading.   Seems more people know more about High Frequency Trading today than CEP. The important thing here is the what all the smart analysts are calling the “CEP Market” really isn’t the “CEP Market” at all.  It’s the HFT software market.  And again, looking at the impressively long list of clients that Mr. Palmer has cited in his recent blog post, many of those firms aren’t actually using CEP for HFT, but an even smaller subset of functionality.  That’s why the market is so small – if any VC firm thought the total addressable market for this technology was going to be $200M in 2010, no CEP startup would have received funding.  And when HFT finds the Next Big Thing, the CEP market, as defined today, will evaporate.  And along with it, any CEP vendor who has concentrated solely upon that market.

SO WHAT HAPPENED?

The idea was that we were ushering in a New Way To Compute Things.  Like all technologists who spend way too much time thinking about this stuff, we thought everyone would immediately see how smart we were, run out and buy one of the CEP based products, and join is in revolutionizing how data is turned into information and used by business folk to make money and pay our salaries.  The only problem is, we forgot 2 things; 1) who would be using our software to do this work, and 2) who would subsequently be using the applications developed by 1.

DEVELOPERS – A FINICKY BREED

I used to be a Real Developer – I wrote in C++.  Then Sun decided that the Internet was the Computer and we all started to learn Java.  Java is cool – Java makes it easy for anyone to write bad code whereas C++ really took some effort to mess things up.  More and more people started using Java for everything; servers, clients, web stuff, etc.  And now, I’m not sure what people use anymore – perhaps coders are using NoJava for all of their no shiny NoSQL apps.  I still use Java.  And I’m loathe to learn another language.  See #4 above in ‘What’s CEP?”  I don’t want to learn another language.  And I certainly don’t want to move all of my work; servers, clients, webapps, etc. to a new and unproven language.  And no matter which vendor you don’t choose for your CEP application because you write it all yourself anyway, none of their languages can claim to be broadly or generally adopted.  Proof?  Try to buy a book on one of them.  There are umpteen books out on NoSQL in less time than it took some CEP vendors to go out of business.  CEP vendors have failed to appeal to core IT departments.  Period.  And core IT departments are the folks who have to assemble all the crap they buy from vendors into something that business users get to complain about when it doesn’t work.

BUSINESS USERS – “SHOW ME SOMETHING!”

Business users want to see information.  They want to see information presented crisply; ready for decision making.  And today, more than ever, they want to see it on their web browser, iPad, iPhone, Droid, Apple TV, disconnected lap top, on flat panels on the front of their refrigerator in the kitchen and on the heads up display in their car while commuting to work.  In short, they want information any time they want information so that they can function in what has become, and will continue to become, an ever faster and more connected world.  Even Progress Apama, who I think is doing really well, uses Flash based instrumentation.  No iPad for you!  There is no CEP environment that let’s the IT folks build a complete application for the business user.  So the business user never ‘SEE’S” CEP.  So they’re not impressed.  They don’t get it.  And they don’t provide budget for stuff they don’t get.

AND IN CONCLUSION

CEP has failed to achieve the multi-billion dollar market forecasts that we all went out and raised money based upon because most vendors have failed to provide the education and tools necessary to create the complete user experience.  Most of the CEP vendors don’t even have their own visualization products – they partner with other vendors to provide things like Tree Maps or Dash Boards.  How they expect to revolutionize the world by outsourcing their interaction with the Business User is beyond me.

THE LESSON?

If the NoSQL camp would like to come anywhere close to realizing the crack smoking analysts’ estimates of a $2B market, they should 1) make the technology readily accessible to the IT department (which they’re doing) and 2) make sure that the business users knows why it’s making a difference.  If they can reach out and touch the business user, or consumer, all the better.

AND THANKS FOR READING

→ 6 Comments

Posted in CEP, Opinion

Flash Crash – HFT Not To Blame so What Next?

Posted on October 27, 2010 | 3 Comments

Much to the SEC’s consternation, the recent report detailing the causes of the May 6th, 2010 Flash Crash has failed to indict High Frequency Trading as the cause.

BUT WE WANT THE MONEY

How does all of this tie in with the SEC’s bid to build a huge consolidated audit trail? (CAT) – well, after a lot of thought, I don’t really know. And I haven’t read a lot from anyone that purports to know either. All I know is the SEC wants a lot of money to build it because, well, we need it to watch all of those evil HFT firms out there disrupting the market.

REG NMS IS BROKEN

So here’s a relevant question, if HFT wasn’t to blame for the Flash Crash, then what was? I thought Reg NMS was supposed to ensure that, no matter how many exchanges enter the fray, we were all supposed to get the best price – regardless of where the order executed. And that all of the exchanges would cooperate and there would be peace and harmony in the valley. Guess what, that’s not working.  IT would seem that the ‘structure’ of the market needs a little attention.

IF IT’S BROKEN, WHAT ARE THEY REGULATING?

So, if HFT isn’t to blame for the crash, and it boils down to some idiot shorting billions of dollars of futures in one fell swoop, then something else must be broken. And the SEC’s CAT will fix it, right? Wrong. All the SEC’s CAT proposal will do once it’s live in 4 years after billions of dollars is yet again confirm that HFT isn’t responsible for any other flash crashes as well. So why spend the money?

BECAUSE I WANT AN EMPIRE

Could it be, that the SEC, who didn’t even have the expertise required to answer the basic question, “WTF happened to the market please” could be looking to build an empire? Is this the same organization who seems to dispense different levels of disciplinary action based upon how deep the offending firms pockets are? Say it isn’t so!  Asking for billions of dollars represents a multiple over the SEC’s current operating budget.  Just what exactly do they plan on doing with all that money?  And how many firms have offered to build the capabilities for a lot less?  If the SEC doesn’t even know what the problem is, how do they know how much money to ask for and what system needs to be built?  Madness.

I AM NOT IMPRESSED

I’d really like to hear how the SEC’s going to fix the market’s structure and prevent this from happening again.  Then I’d like to see actual enforcement of the laws on the books and see people who violate these laws go to jail instead of write checks.  In short, what I’d like to see is the SEC start doing their job before asking for billions of dollars to do something that wouldn’t have prevented what happened in the first place.  Please.  Does the SEC and Mary Schapiro think we’re all just that stupid?

THAT’S RIGHT, I SAID IT

So not only am I not going to go on record with the above (go back and read it again – the SEC isn’t doing their job) but I’m always going to say that event processing technology can’t prevent future occurrences of Flash Crash (and CEP vendors who think they can really don’t know what they’re talking about).  Several vendors  (YOU KNOW WHO YOU ARE) are all too happy to not point out that the emperors new clothes, well, need some additional tailoring as they jump on the SEC and CFTC’s bandwagon, hoping for additional, regulation sourced revenue.

THE SEC COULD BE A HERO

I believe that if the SEC started doing their job again, and investor confidence returned to the equity markets, maybe we’d see a return to previous volumes.  Because right now, at the volume levels out there today, there’s going to be blood in the streets as firms collapse under the lack of trading.  All the SEC has to do is their job.  And be 1/10th as vocal about it as their request for billions of hard earned tax payers dollars.

Thanks for reading!

→ 3 Comments

Posted in Opinion, RegNMS

Tagged

It's Deja Vu – All Over Again

Posted on September 8, 2010 | 6 Comments

I was recently asked “What problems does CEP solve that cannot be solved with smart coding, a columnar database or a whopping great grid?” via a Linkedin group for Complex Event Processing.  Here’s the link.  I think I understand the question, and if I do, it’s really the wrong question.  Which means of course that my answer is probably going to cause a bit of a ruckus.

WHAT IS CEP ANYWAY?

There are a couple of things that a system needs to say that it incorporates CEP – the ability to continously query an event stream, pattern matching semantics, and sliding windows over data.  For example:

SELECT PHASORID, AVG(FREQUENCY)  FROM SMARTGRID.WIN:TIME(30 MINUTES) GROUP BY PHASORID;

This sets up a continuous query – it listens for events that are named ‘SMARTGRID’ and calculates the average frequency by phasorid.  This is an example of a time based window.  The window always refers to the last 30 minutes.  Another example:

SELECT A.* FROM PATTERN(A=SMARTGRID(FREQUENCY>130) -> (TIME.INTERVAL=30 MINUTES) AND B=SMARTGRID(FREQUENCY<110 AND A.PHASORID=B.PHASORID));

This statement says, “I want to know when a phasor measures a  high frequency followed by a lower frequency.  This is an example of pattern matching.  Both examples demonstrate a continuous query – those queries just keep running, returning a continuous stream of results.

THAT’S A LOT OF CEP

I’ve been focused primarily on solving problems using CEP for the last 6 years, and using event processing for the last 20+.  So, I’ve seen lots of examples of the right use of CEP and some wrong uses of CEP as well.  We’ll talk about the right uses here.  Mark Piper asks about, “smart coding.”  Let’s answer that one first.

SMART CODERS

Sure, smart coders can get along without CEP.  Just like they can get along without a database.  Or a messaging bus.  And if you’re a really smart coder, you don’t even need an operating system.  You’ll just write all of those using your copious amount of spare time that your employer provides you because you don’t have any deliverables that are time sensitive.  And since you don’t have any customers, you can do whatever you want anyway.  Right?  Wrong!

There’s a difference between an implementation of CEP and CEP itself – maybe you want the whole development environment ala StreamBase, Apama or Sybase.  Maybe you want an API.  That’s most likely up to your individual/team coding style and whether or not you actually buy into the, “Use our platform for everything – it will save you time and money!” argument.

But the gist of this argument is that if you want to have a CEP source of functionality to use when solving problems by writing code, you probably want a general library, or system, that you can use to do so instead of writing it yourself.  And I’ve had the privilege of working with some very smart coders over the years and it took even them slightly longer than a month or two to build something that could be used in a number of different scenarios.

CHEAPER CODE

So that’s the tech side.  What about the business side?  That’s even easier – I don’t want to hire a bunch of C++ coders to write infrastructure – unless that’s my business.  And most businesses don’t fall into that category.  A lot of big financial firms do get a bigger bang for buck by writing their own stuff – they’ve got the staff (because they can afford the really smart coders it takes to build this stuff), they’ve got performance requirements that are extremely stringent and it’s actually cheaper for them to build vs buy given scope and breadth of deployment.  So, unless it’s your business to build infrastructure by using very expensive coders, it’s cheaper and just as effective to use third party libraries or systems.  And that’s just not for CEP but generally applicable across the board.

COLUMNAR DATABASES AND GRIDS?

Not sure if I understand the last part of Mark’s question, but I’ll take a stab anyway.  Mark mentions grid – and the use of grid vs stream processing typically finds itself used in different parts of the same problem.  For example, if I’m running a big credit derivatives trading operation, I’ve probably got a grid running some fairly heavy compute and updating a shared cache of stuff that the bank needs for a variety of applications.  Then let’s imagine that I’m receiving client orders and want to do something based upon data in that grid, I might just use a system that incorporates components of CEP (notice that I did not say a CEP system here – that’s important) to streamline the event stream processing characteristics of that application using the grid cache to grab parameters for me as they change.  So the use of grid and CEP here is complimentary – not dynamically opposed.  Two different compute problems, two different technologies applied to a very common pattern.  There are similar patterns involving the use of CEP for data-in-flight and columnar, or any database really, for historical data.

SO IN CONCLUSION

I recommend the use of existing code over writing new code just because you’ve got smart coders if it makes sense and is economically attractive.  And, just like in the 90′s when we were selling EAI applications, all the C++ guys whined back then, complaining that, “We can do it better and faster.”  While that may have been true (very few ever actually proved that to me), it was usually far more expensive and very brittle.  Tools exist for a reason – it’s what separates us from the animals.

AND AS ALWAYS

Thanks for reading!

→ 6 Comments

Posted in Opinion

Tagged

DarkStar Filters Twitter Stream in Real Time

Posted on September 1, 2010 | 3 Comments

I show how to use DarkStar to filter the Twitter stream.

→ 3 Comments

Posted in Twitter

Tagged , ,