We had data yesterday, we have data today, and we’ll have data tomorrow.  In fact, we’ll have a lot more data tomorrow.  We’re adding data to our lives at what seems to be an exponential rate.

I belong to the Low Latency, Big Data group on Linkedin, here’s the link – Low Latency, Big Data.  There are some heavy hitters out there and we’ve been working together on a definition for big data.

WHAT EXACTLY IS BIG DATA?

There are a couple of working definitions in progress in the group, but the one I like that seems to be emerging really doesn’t have that much to do with data at all.  It’s about architecture.

THE SCALES OF BIG DATA

How about scaling to handle all this data.  I think one of the core tenants of big data is that it doesn’t fit on one machine.  Or maybe a better way to say that is that it won’t scale on one machine.  A seemingly natural progression to this logic is that if you want to play with big data, your architecture has to scale.

GRAVITY IS HEAVY MAN

Data has gravity.  Hadoop, Yahoo’s slow implementation of Google’s Map/Reduce typically runs on top of HDFS or Hadoop Distributed File System.  And when you run Hadoop Jobs, the code is shipped out to the nodes in a Hadoop cluster and run against the data.  That scales – it scales because if you want to store more data and run more processes against it, you just add nodes.  For the most part, this means that Hadoop is a shared-nothing architecture.  That’s important if you want to scale.  (go read about it elsewhere, then see the error I made above, then continue reading).  Running code on the node with data means you don’t have to ship the data around the network.  See the gravity?

BIG DATA DOESN’T REWARD SHARING

It would seem that by sharing nothing between nodes (things like state), we can run processes in parallel.  Running things in parallel against the same size data instead of running serially means we should get done sooner.  That’s fairly obvious, right?

WHAT’S NOT OBVIOUS THEN?

Big data has nothing to do with data.  Big data is the advent of grid and parallel processing.  Big data represents the democratization of tools and processing power (cloud) made available to anyone with a credit card.  Grid and parallel processing have been around for a while.  Elastic resource is relatively new.  Free software for VC’s to pimp is almost entirely new.  (we call that ‘open source’)

VOLUME, VARIETY, VELOCITY

Whatever.

THE TAKE AWAY

The take away here is that if you’re a vendor and your solution only runs on one machine, you don’t scale well enough to handle big data.  Since big data is really about scaling out.  If you haven’t solved this not-so-easy problem, then using the phrase big data in your marketing is nothing more than a big lie.

AS ALWAYS

Thanks for reading.  And Happy Holidays!

Without getting into a rehash of the hash over there, let’s look at things a bit differently.  Let’s talk about what CEP is not.

I left trading to join a firm called NEON.  I was an early investor in this company, my mentor had started the firm, and to me it looked like the cat’s meow.  It was a great time, a lot of people made a ton of money, and I was introduced into the world of software via Enterprise Application Integration (EAI).  Using EAI, one could centralize the business logic associated with plugging different systems into each other, convert the format of one system into another, and pass messages around seamlessly between these applications.  There weren’t a lot of competitors at that time, we were arguable #1 in the space, and the technology became MQ Series Integrator (think Websphere, like I said, we made a lot of money.)

Well, CEP isn’t EAI because there’s no concept of format libraries – sure CEP engines use input/output adapters but sure does every program ever written (I’m waiting for the first salesperson to licence the keyboard/screen adapter set – available in different languages soon!).  We’re going to come back to EAI in a moment.

Throughout my career, the teams I’ve worked with have used a variety of 4th generation languages.  Stuff like Powerbuilder, SQL Windows, Paradox, etc.  Each of those environments had some common elements, screen designers, a domain specific language designed to make bizapp dev faster, and abstractions for common data stores.  Often times, our groups wrote servers that integrated with these front ends via RPC, a streaming connection, or databases.

CEP isn’t a 4th generation bizapp dev environment – there’s no facility for building gui’s.  Although some of the CEP platforms out there do have DSL’s, some also use SQL derivations.  I’ve used the SQL derivations (I’ve worked at two of those co’s) and guess what?  The people in those firms hated using the language themselves.  ”Yes, you could do a covariance matrix with <insert proprietary get-me-sued-for-naming-it-here> but I could do it faster and easier in a different language.

I’ve also used many databases.  But you don’t use CEP to store data – you only process the data in flight.

So, CEP isn’t EAI, it’s not a database, and it’s not an application development environment.  Where, then, did CEP come from?  Let’s look at a couple.

The work out of Berkeley and the work out of Brown, Brandeis and MIT focused on event stream processing.  Here’s a blurb about Berkeley’s Telegraph:

Telegraph is an adaptive data-flow system, which allows individuals and institutions to access, combine, analyze, and otherwise benefit from this data wherever it resides.  As a data-flow system, Telegraph can tap into pooled data stored on the network, and harness streams of live data coming out of networked sensors, software, and smart devices.  In order to operate robustly in this volatile, inter-networked world, Telegraph is adaptive – it uses new data-flow technologies to route unpredictable and bursty data-flows through computing resources on a network, resulting in manageable streams of useful information.

And here’s one about Aurora (Brown, Brandeis, & MIT):

Aurora addresses three broad application types in a single, unique framework:

1. Real-time monitoring applications continuously monitor the present state of the world and are, thus, interested in the most current data as it arrives from the environment. In these applications, there is little or no need (or time) to store such data.
2. Archival applications are typically interested in the past. They are primarily concerned with processing large amounts of finite data stored in a time-series repository.
3. Spanning applications involve both the present and past states of the world, requiring combining and comparing incoming live data and stored historical data. These applications are the most demanding as there is a need to balance real-time requirements with efficient processing of large amounts of disk-resident data.

Hmm.  I’ve worked with both of those packages – no mention of Complex Event Processing in there at all.  So where did that phrase even come from?  Well, that’s the title of David Luckham’s book, “The Power of the Event” in which the good professor describes not so much an implementation, but a set of processes designed to help us all run our businesses and missions more effectively.  In the book though, David references a language that deals with streaming data.  Oh oh….

Around 2005-2006, a couple of firms were struggling trying to describe what Event Stream Processing was and why it was important and more importantly, why you should be spending money on it.  I was the CTO of one of those firms.  We competed mostly against Streambase at the time.  Somewhere during that time frame, the phrase Complex Event Processing was adapted in an effort to differentiate.  At that time, Aleri wasn’t CEP – they were OLAP.  Streambase, formerly Grassy Brook, probably choose that name in homage to Stream Processing.  Kaskad is Swedish for waterfall, or where a bunch of rivers and/or streams collide.  I don’t think Apama ever used the phrase ESP, they were focused on trading from the start.  Starting to get the picture?

So, if CEP isn’t EAI, and it’s not a 4th generation bizapp tool, what is it?  I’ve probably kicked this dead horse enough, but one more time and it’s not going to notice.  CEP needs 4 things to be called CEP (or ESP…):

1. Domain Specific Language
2. Continuous Query
3. Time or Length Windows
4. Temporal Pattern Matching

These 4 things, in my opinion, don’t make up a separate space, let alone a market.  What they describe is Event Stream Processing.  What they describe are features found in larger, more complete event processing environments from IBM, SAP, TIBCO, and Progress.  And TIBCO, for example, just added the missing features described above to their Business Events Platform, and had instant CEP (sarcasm mine).  Those offerings look a lot like the traditional EAI platforms – or where all of this began roughly 20 years ago.

So, I don’t think what a couple of vendors sell as Complex Event Processing is really CEP at all.  If you want an idea of what CEP is really all about, read David’s book to get started.  Then take a look at Tim Bass’s blog thecepblog.com.

For now, let’s just drop the phrase CEP (because it’s mostly just Stream Processing) because it means so little to so many and fails to impart any meaningful message to the people who actually write checks for this stuff.

Thanks for reading!

Hadoop Is Picking Up Speed

MAYBE THERE’S A REASON FOR THAT

Recently, Google announced their move away from batch based MapReduce to something a little more real time.  Seams like it was taking days to update search results with something that you might be interested in.  Google never open sourced their implementation of MapReduce, which is said to be at least one or two orders of magnitude faster than Hadoop.  But still not fast enough.

EVEN YAHOO IS GETTING INTO THE ACT

Yahoo used to have a substantial relationship with Cloudera, at least according to Cloudera.  But now even Yahoo have started a company to distribute and support Hadoop.  Yahoo calls their company hortonworks.

WHAT THIS MEANS TO YOU

Without getting into things like how much data and corresponding analysis you need to do before Hadoop makes any sense to use at all (most companies are not going to see any benefit at all), let’s recognize something.  All of these recent shifts from companies like Google, Yahoo, and others no longer see a competitive advantage in batch based MapReduce.  The future has arrived, let’s look at some evidence.

REAL TIME HADOOP

MapReduce

There have been more than a handful of releases in this space – like S4 from Yahoo, HStreaming, Storm, and several NoSQL databases now supporting this, it means that for competitive advantage, you’d best be getting some real-time.  And getting it soon.

WHAT IS REAL-TIME?

Database vendors like DataStax, who support Cassandra, claim to be real-time.  They’re not.  They say that they’re real time because as soon as you commit data to the database, it’s available for query.  That’s supported by just about every database and hardly a new and exciting feature of NoSQL.  Even one of their big shots left to start a real time company named Platfora.

CONTINUOUS QUERY OR EVENT-DRIVEN

Rather than thinking about what real-time is or is not, let’s worry about event-driven.  Let’s use an example:

I’m a manager, and I want to know when the average time on my website dips below 2 minutes.  Using the ‘my database is real time because the data I send to it can be queried after I write it’ means that I would have to run this query repeatedly at regular intervals to catch this mounting exodus from my web properties.

THERE’S GOT TO BE A BETTER WAY

And there is, it’s called continuous query.  I ask the same question as above, and there’s some process somewhere that’s sessionizing data from my web logs and injecting that into that server – the same server that I sent the query above to.  And when that process finds a web session that lasted less than 2 minutes, it sends another ‘row’ to the program that submitted that query.

ABRACADABRA

Waiting for Hadoop Query

And then I’ve got it on my dashboard, and can switch out the really badly designed page the marketing department A/B’d this morning.  That’s continuous query, or event-driven.  The term real-time didn’t even need to be mentioned.  If I was running batch based Hadoop, that notification could have taken hours, or days.  How much money would your company lose if that happened to you?

BACK TO MAP/REDUCE

I am Node of Cluster...

So if I can do the above, why do I need MapReduce?  MapReduce is an algorithm for splitting work up, distributing the work out to nodes where the data lives that needs to be analyzed, and then gathering the results.  If you’re problem is big enough, MapReduce might help you get it done faster than using just one machine.

BUT EITHER WAY

If you’re running batch processes, like some well known web properties are and think that Hadoop holds an answer to your ever dwindling ad revenue, you’re mistaken.  And if you’re that CIO, the other thing you need to be working on is most likely your resume.

GET YOURSELF SOME CONTINUOUS QUERY, AND GET COMPETITIVE!

and thanks for reading!

NO UN-FILTERED DIRECT ACCESS

What the SEC is trying to do is remove, or reduce the opportunity for either crooks, idiots, or algo’s gone wild from doing bad things to the market. Under the new rules, order flow needs to be monitored. This is not something that the HFT crowd like to hear, because it slows them down. So a couple of innovative idiots got together and came up with the solution that I’m going to describe here.

HOPE IS NOT A STRATEGY

Let alone a comprehensive compliance or surveillance strategy. What the idiots are doing is putting a ‘black box’ between the HFT firms FIX engines and the execution venues. The box, most likely powered by an FPGA device, scans the outbound order flow, and if it finds something it doesn’t like, it messes up the payload of the FIX order so that the execution venue (hopefully) rejects the message. Why is this done this way? Because the ‘black box’ is both out of process – both the source of orders and resulting executions, etc. are behind FIX engines, and because the ‘black box’ isn’t actually maintaining connections between the HFT firm’s order generators and execution venue.

A PICTURE IS WORTH A MILLION REJECTS

This is a little complicated, so let’s look at this picture:

In the diagram above, the ‘black box’ isn’t maintaining FIX connections to either the HFT’s order generators or the execution venue.  So, the ‘black box’ can’t just reject the order if it’s out of bounds back to the order generator because then the FIX sequence #’s get all mixed up.  There’s a little more to this, but you get the general idea.

YES, THIS IS REAL, AND I’M NOT KIDDING

So, this whole thing is designed so that an examiner can come into the Olde Thyme Highe Frequency Trading Shoppey and be escorted into the back room and shown the shiny box.  Wow.  Are you serious?  ”Look, we’re making sure that this firm isn’t doing anything wrong – we’re actively monitoring the flow and if they do something we don’t like, we shut them down.”  Right, they shut down the order flow attached to the box.  What about the order generators that the examiner doesn’t see? There’s a host of issues here, but we’re going to focus on one – and it’s a doozy.

DENIAL OF SERVICE ATTACKS

So, we’ve installed the OMICRON 5000 monitoring device and our HFT/algo team is ready to do business.  And everything is fine.  They’re trustworthy chaps and have no intention of gaming the system.  (cough cough).  But their first algo goes completely nuts.  And gets shut down by the clearing firm.  But it doesn’t really get shut down.  Instead, it’s sending 1000′s of malformed FIX messages to an execution venue per second.  Or maybe 10,000′s of malformed FIX messages to many execution venues.  Wow.  In the internet world, we call this a denial of service attack – flood a destination with more traffic that it can handle.  And while the execution venues can handle normal traffic, what about rejecting every message? Is every execution venue out there ready for this?  I don’t think so.  I’ve been involved with FIX longer than I’ll admit to in public, and I’ve seen a lot of testing  - “Yeah, reject worked.  It worked fine.  I mean, we never thought they’d be sending 1,000′s of orders a second that would all reject…”

I DON’T KNOW

What should be done about this.   I have lots of ideas about surveillance and how it should be done.  But I don’t have any thoughts about this.  Mostly because I never thought anyone would be so stupid as to ever actually deploy this type of ‘solution.’  Where’s the SEC when you need them?

THANKS FOR READING

CONFUSION

Most of the explanations I’ve read about Cassandra’s data model first extol the virtues of NoSQL and the evils of Relational Databases.  And so while getting the reader caught up in this mythic struggle that summons images from Tolkien’s middle earth, the point is lost.  And that point is?

IT’S ALL ACTUALLY QUITE EASY

Cassandra thinks about data the way we think about data.  Most of us think about data in rows and columns.  So does Cassandra.  But it also alleviates some extra stuff we don’t need while adding some stuff that we do need.  And that can be a little disconcerting initially.  To make things easier, let’s first describe a goal for our exercise.  We’d like to get a day’s worth of market data, by symbol, in ascending time order.  Also, we might like to get the data for a slice of time within that day.  Like, “give me all the BBO’s for American Airlines for May 20th, 2010,” or, “I’d like to see the BBO’s for American Airlines for May 20th, 2010 between 1 and 2pm.”  Let’s jump right in.

LET’S GET OUR DATA

As we subscribe to our favorite market data feed, we receive something like:

• Symbol,
• Bid,
• Offer,
• Bid Size,
• Offer Size.
• Time Stamp, and
• Seq # (most quote vendors provide a Sequence # because multiple quotes can occur for any given Time Stamp)

We’re going to call this a column family.  Cassandra’s analog for a table is a Column Family.  You can see why this fits so well above – the columns that belong to the symbol AA comprise a family of related information.  I’d like to store this data by symbol, so later, I can retrieve it.  Using the Cassandra client (cassandra-cli – it’s in the bin directory where you installed Cassandra), let’s create the BBO Column Family.  It looks like this:

create column family bbo with comparator = UTF8Type
and column_metadata = [
{column_name: symbol, validation_class:UTF8Type},
{column_name: bb, validation_class: UTF8Type},
{column_name: bo, validation_class: UTF8Type},
{column_name: bbSize, validation_class: UTF8Type},
{column_name: bSize, validation_class: UTF8Type},
{column_name: timeStamp, validation_class: LongType},
{column_name: seqNum, validation_class: LongType},
];


And now that we’ve created the schema, let’s insert some quotes.

Set bbo[‘AA’][‘symbol’]=’AA’;
Set bbo[‘AA’][‘bb’]=’123.34’;
Set bbo[‘AA’][‘bo’]=’123.84’;
Set bbo[‘AA’][‘bbSize’]=’100’;
Set bbo[‘AA’][‘boSize’]=’200’;
Set bbo[‘AA’][‘timeStamp’]=1234;


What happens when you execute a list bbo command now?  So, that’s easy enough.   So what happens as we get the next quote?  Well, we go to insert our data like this:

Set bbo[‘AA’][‘symbol’=’AA’;
Set bbo[‘AA’][‘bb’]=’125.34’;
Set bbo[‘AA’][‘bo’]=’125.84’;
Set bbo[‘AA’][‘bbSize’]=’100’;
Set bbo[‘AA’][‘boSize’]=’200’;
Set bbo[‘AA’][‘timeStamp’]=1235;


And then to see our data, enter this command (again):

List bbo;

When we use the ‘list bbo’ command, we’re only go see that data last inserted for that row key.  What happened to the previous data?  It was over-written with the new data.  So if we wanted to save each quote, we could combine the timestamp with the column name and then we’d be inserting unique columns each time and we’d be fine.  But there’s a different way to do this.

BIG DEAL, I DON’T SEE ANYTHING DIFFERENT HERE

And you don’t, because we haven’t started introducing the special sauce yet.  Well, we kind of did.  In the schema definitions above, you’ll notice we didn’t say that much about what we could or couldn’t insert into a row.  We just started adding columns dynamically.  So, each row, which is identified by a key, can have different columns in it and even a different number of columns.

WELL, THAT’S NOT GOING TO WORK

So, how do we keep track of all the quotes for our symbol?  First a little clarification, the Column Family above is really BBO, and we’ve inserted a row identified by the key, ‘AA” and some associated tag/data value pairs.  Think of this as a map of maps.  So now, we need to insert the bits that change for a given symbol over time.  How could we do that?  We create a Super Column Family of course.  A Super Column Family contains Super Columns.  A Super Column is kind of like another row of data – so using our example above, the Super Column we’ll be inserting consists of the BB, BO, BB Size, etc. The data above gets inserted using [AA] as our row key, and we need to pick a key for the Super Column that contains the quote data.    Let’s pick Seq# as our Super Column key.  Our row key is still Symbol, and I’ve prepended the date to it.  This way, all the data for a day’s worth of AA will be in the same row.  This is called a compound, or aggregate, key.  It looks like this:

create column family sbbo with column_type = 'Super' and comparator = ‘BytesType’
and column_metadata = [
{column_name: bb, validation_class: UTF8Type},
{column_name: bo, validation_class: UTF8Type},
{column_name: bbSize, validation_class: UTF8Type},
{column_name: bSize, validation_class: UTF8Type},
{column_name: timeStamp, validation_class: LongType},
];


And the insert statements look like this (we’re using the Seq# as the key – that’s the Super Column key right after the row key or, ’20100124:AA’ below):

Set sbbo[‘20100124:AA’][1234][‘bb’]=’100.00’;
Set sbbo[‘20100124:AA’][1234][‘bo’]=’101.00’;
Set sbbo[‘20100124:AA’][1235][‘bb’]=’101.00’;
Set sbbo[‘20100124:AA’][1235][‘bo’]=’102.00’;
Set sbbo[‘20100125:AA’][1234][‘bb’]=’100.00’;
Set sbbo[‘20100125:AA’][1234][‘bo’]=’101.00’;
Set sbbo[‘20100125:AA’][1235][‘bb’]=’101.00’;
Set sbbo[‘20100125:AA’][1235][‘bo’]=’102.00’;

Now let’s see what’s in the column family:

List sbbo;

So now it looks like we’re able to store a set of quotes for a symbol for any given day.  Bingo.

All we’ve really done here is add another map – so we now have a map (Date, Symbol) that contains a map (Symbol, Quote) that contains another map (Quote, QuoteField).  Or, what we’ve done is figured out a way to represent the potentially sparse fact tables resulting from large data analysis (OLAP) projects in a concise and easily addressable fashion.  Told you it wasn’t that hard.

GIVE ME MY DATA

So, now that we’ve inserted a couple of rows of data, let’s see how to get our data.  From above, we want to:

1. Get all the data for a day’s worth of a symbol, and
2. Get all the data for a slice of time during a day for a symbol

Assuming you’ve entered the statements above to insert the data, we can retrieve an entire day’s worth of AA with this simple statement:

get sbbo[‘20100124:AA’];

Now that we’ve gone over some of Cassandra’s basics, we’ll get a little more into it in upcoming posts.  That’s where we’ll cover the goal in #2.

THANKS FOR READING

Appendix to Part One – The Data Format

Rather than looking at various database solutions first and then trying to define the problem in terms of those solutions, let’s first examine what market data looks like.  In its most simple form, market data looks like this (there’s usually a little more, but this is fine for our purposes):

• Date: The date of the market data,
• Time: When did the quote occur during the date,
• Sequence #: Most quote or trade streams include a sequence #,
• Symbol: What security is this data for?
• Best Bid: The best bid (we’re going to concern ourselves with BBO data for this series, it’s easier),
• Best Bid Size: How much does someone want to buy at the Best Bid,
• Best Offer: The best offer,
• Best Offer Size: How much does someone want to sell at the Best Offer.

Consider this chart:

Market Data

If we break down data, we can successively see how data might be arranged on disk for subsequent reading.  We want to read the data very quickly.  If we were using a standard relational database, it’s easy to see that we might be replicating some unnecessary data during the reads.  And we if use a typical columnar database, we can see that there are chunks that could be read together increasing throughput.

For example, for any given millisecond (Time) in a quote feed, there may be more than one symbol with a quote.  In fact, that’s quite common.  So replicating the time stamp is superfluous.  So if we had a table for a date’s worth of data, then we’d have a Time column that was replicated throughout the table.  No reason to do that.

Looking again at the data, we can see that, for a given time, there might be multiple quotes available for multiple symbols.  We’d like to read those in order as a little group.  By organizing the data on disk as a flattened multi-dimensional map of maps, we would:

1. Start with a given day (our table),
2. Start with a time (our row),
3. Read each quote in sequence # order (our column)
4. Process (do something)
5. Increment the time, and go to #2 above until we run out of data (lather, rise, repeat)
6. Put the $ in the bank

If we could write this data structure to disk as we get it from the quote feed, and had fast enough disk, we could keep up with the feed.  If we needed to create some indexes on the data, we could easily do that as well.  We’d simply create another table that would hold an inverted list of time and sequence #’s by symbol.  If we want to process a day’s worth of data, we’re all set.  If we want to process a symbol, or group of symbols, we’re all set.

So, to summarize, we need a hybrid approach.  In some places, we want rows of data – storing columns of data via a unique key.  In our case, that’s the Time column above for a given day.  The row above is Time, the column (or Super Column) is the Quote for a Symbol.  The Super Column’s key is the Sequence #.  Can anyone guess which database might fit nicely for this use case?

In my next post, I’ll describe a formalized data structure and it’s implementation.  I might even include a little code for all you #NoSQL guys and gals out there.

Thanks for reading!

Part One – The Data

I recently made some comments on Vertica’s blog in regards to what I considered to be a fairly bold claim.  They said that Vertica was the only real column store.  But even if they are, so what? In my comments, I alluded to my belief that we optimize problems to solutions – we try to fix stuff using what we’ve got in our toolbox without having to run to Home Depot.

The real test is when the rubber hits the road – how do you actually solve a problem in a new way that’s motivating.  And by motivating I mean the solution addresses the issues, enables new capabilities, and is economically attractive.

So rather than tell you that DarkStar and our approach to processing both real-time and historical data (there’s a difference?) is the Real Enchilada, I thought I would illustrate a real world use case.

Let’s say you want to store a bunch of market data.  And I mean a bunch.  You want to store every piece of market data for the whole US Equities market.

And you’d like to have this data so that you can run analytics on it.  Or maybe even back-test strategies for buying and selling stocks.  So let’s assume that you’ve got some java code lying around to do that.

For our example, we are interested in seeing whether or not using volume average weighted price strategies actually work.   In our example, we will pretend that we are buying a lot of stock, and the theory we want test is whether or not buying that stock during the day when it’s lower than it’s weighted average price will give us a better average price during the day than just going with the flow (often referred to as volume participation).

We are all familiar with how relational databases work, and anyone who’s been in capital markets for a while knows how futile it would be to use something like Oracle for this due to cost, hardware and just how difficult it is to get the data into the database in the first place,

Oh that’s right, I forgot to tell you, we are going to have to load this data first.

I am not going to go into the relevant benefits of a column store here either, you can check out many other websites for that.

Instead, let’s look at some issues.  First, I would rather load the data directly into the database as it happens.  Staging the data separately is costly and error prone. In addition, what happens when you decide to load that data and encounter a problem that can’t be fixed in time for the next market day?  What if you actually run out of space or compute to get caught up?  Well then you can’t back test the next day to further refine your algorithms.  Algo’s should be tweaked every day.  New algo’s need to be developed to remain competitive.  So here, a database error costs real money.

So I need a fast data store.

As I am loading the data, what happens if one of my disks goes boom? Or one of my machines go boom? Well, now I have a problem.  If I fail over to another datacenter, how do i reconcile? What a nightmare!

So I need a data store that we can take a sledgehammer to and it will keep running.

Hey, if I have this big historical data store, I still need to query it while it is being updated.  Ideally, I would like to also be running analysis and back testing during the day.  Scheduling jobs to run at night is so very ’90′s.

So my data store has to facilitate both interactive query and batch analysis.

But wait, doing all of this means that I am going to have to figure out how to use the same code for back-testing that i use to generate orders during market hours.  It’s either that or use some visual, script based or different harnesses for my java or C++ code.  Yet another nightmare.

So, I would Iike to run the same code against my historical data store that I also use to generate orders during the day.

There’s a bunch of other stuff too, management, instrumentation, removing old data that I don’t need for back-testing, all the stuff we associate with normal day to day big data operations. We need to know what’s going on during the day so that we can be proactive.  There’s gold in that data!

And one last thing, it would be really cool if most of this technology wasn’t proprietary.  I mean let’s face it, firms that talk more about their investors on their websites than their clients can’t possibly have my best interests at heart.

This is a tall list.  Let’s knock it down, one by one.

Here is a diagram for your consideration.

The diagram isn’t very technical, and that’s on purpose – I’m outlining an algorithm, or methodology that may or may not solve our problem.

In the diagram, I’ve depicted the database as a cluster of machines.  Instead of using one big machine backed by a SAN, I’m going to use a number of machines.  Each of those machines is going to connect to the Market Data source and get data.

As we receive the data, we’re going to take a peek at it, and determine where in the cluster that data needs to live and while we’re doing that, we’re going to right it to disk.  A background process will make sure that the data ends up on the node we want it on.  More of why that’s so incredibly important in Part Two – Analyzing the Data in this series.

Also, I’m going to ask the cluster to replicate everything we’re writing to it – we’re going to end up writing the data a total of 2 times in this example.  I might usually suggest 3, but we’ve got two data centers running the same solution, so I’ll actually have 4 copies of the data.

Why write the data to three nodes in the cluster?  First, if a node goes down, I still want to be able to write data.  If the node that goes down is the primary node, I’m going to remember that and when that node comes back up, I’m going to write all the data to it as part of its “Welcome Back to the Cluster Party!”  And second, if I’m reading data from the cluster (remember, we’ve got algo’s running and users querying this data), I want my data.  If a node goes down, your users don’t really care – they just want their data.  By replicating the data across multiple nodes, I achieve high availability without having to fail-over to another instance or data center.

Ok, we’ve got the Sledge Hammer test handled, which is cool, but everything I’ve described above sounds like it’s going to take a lot of time and that the system is going to very slow.

Not true.  Each node in the cluster above is subscribing to market data.  So if one machine can ingest X messages per second, then a cluster of 10 machines should be able to ingest 10 * X messages per second, right?  Let see what that means in a real world example:

On May 20, 2010, there were about 1.1 billion BBO messages as published via SuperFeed (NYSE’s market data platform), those quotes represent the best bid, bid size, offer, and offer size for each stock at any given time.  In terms of messages per second, that’s about 50,000.  In terms of size per second, that’s about 4,500k per second.  Hmmm, chunky!

These are intimidating numbers.  But if we divide the problem up a bit, and use 10 nodes in a cluster, each node only needs to ingest about 450k per second across 5,000 messages.  All of a sudden, we’re dealing with something quite reasonable.

So, now we’ve got a cluster that can load the entire market real time and it’s redundant.  What about analyzing the data?  That’s in Part Two – Analyzing the Data which I’ll post next week.

1. CEP – The Feature

There’s a couple of things going on here.  The most important being that Mark Palmer is writing blog posts about Richard Tibbetts writing blog posts on the Tabb Group’s site about writing better software on Wall Street (because software startups write better code and deal with bigger problems than firms on Wall Street, especially exchanges, do, right?…).  No customer win.  No ‘yet another use for CEP.’  Just good old fashion buzz-word copy designed to remind people that there’s still one stand-alone CEP vendor. (click click click, is this link working?)

CEP will become a feature of larger, more established, horizontal offerings.  Because once the opportunities in Canada and South America dry up (you know those hotbeds for financial engineering, right? Canada and South America?  You don’t?) the reality will sink in.  What’s that reality?  That no one in NYC is buying CEP engines for HFT trading anymore.  Why?  The CEP vendors don’t know why.  Even Apama has seen the light.  You can now process events one at a time, or within the context of their CEP engine.  Stunning.  They’re pushing the ‘platform.’  Good.  It’s about time Tibco got some competition.

So this coming year won’t be the year of CEP, it will be the yesteryear of CEP, like, “Remember yesteryear, when we all thought CEP was going to be really hot?”  CEP will become a feature found in Event Processing Platforms.  And we’ll finally start to see adoption of those platforms in large, house hold names.

2. Hadoop & Analytical Databases

People are going to begin realizing that databases that incorporate map/reduce into their architecture will be *no faster* than Hadoop.  Why?  Go buy a book about Hadoop and then sit down with a piece of paper and pencil.  Databases are designed to support multitudes of users all asking different questions.  And vendors would like to have us believe that they’re also all running long lasting jobs taking advantage of their shiny map/reduce implementations.  Long lasting.  As in, not interactive.  As in, batch jobs running on a map/reduce framework.  The bigger the job, the less increase in throughput the analytical database will be able to offer.  It’s physics.  Save your money.  So this coming year, we’ll probably see further consolidation and some unexpected exits in this area.

3.  And Speaking About Hadoop

Batch is dead as competitive advantage.  As Jeff Jonas loves to point out, and points out really well, data velocity is growing.  And the rate at which data velocity is growing is increasing.  And companies can’t process the data they have today.  And many companies are actually making bad decisions with more data, not better decisions, why?  Because the data has lost most of its value once it’s been crunched.  You can only take a batch system like Hadoop so far.  But right now (at least for the near future) you actually still need to incorporate some ideas from the batch world for everything to come together.  So this coming year, we’ll see more people start treating Hadoop as either a must have to compete (minimal cost of entry) or “How f(*)(* much does that cluster cost to run?  There’s got to be a better way!”  It’s time to outsource your Hadoop cluster.

4. Real Time & Batch

I’m not saying that Hadoop or Map/Reduce is a waste of time and money like some vendors who make outrageous claims like, “Google has stopped using map/reduce.”  That’s idiotic.  Please put the kool-aid down, you’ve had enough to drink.  What’s important is the ability to analyze data in flight, to make decisions while there’s still the opportunity to have an impact.  How does one accomplish this?  By having a context in which events are analyzed.  How is context built?  Via the constant processing of events in flight, constructing and augmenting context, and supplementing that context with the result of monster jobs run on gazilla-bytes of data (like that? gazilla- it’s mine).  So this coming year, we’ll see a focus on moving analytics to real time.  (deeper analytics than VWAP-please!)

5. The Big Picture

There have been some really neato-keeno entries in the visualization space.  Things like Tableau, Spotfire, etc.  But they’re great for analysis of relatively static data sources.  They’re not for real time stuff.  Even offerings from vendors like Panopticon, which can provide some insight into multidimensional data sources updating in real time, really offer quite limited analysis tools for big data.  So in the coming year, we should see more focus on real time data mining.

6.  Did He Say, “Big Data?”

Yup.  I said it.  I’ve heard people define big data as more data than will fit on one machine.  Those people haven’t worked on the machines I’ve worked on.  I define big data as “when you can’t turn your data into actionable intelligence fast enough to have an impact during the window of opportunity.”  Or something like that.  I’m not in marketing.  Common patterns for analyzing big data are emerging that tie as-it-happens analysis with context and historical data.  The lines are blurring.  It’s all just becoming data.  And business wants it all.  Even my father-in-law, in Germany, asked, “Ja, SAP wants me to store my data in ze cloud.”  Everyone knows about big data and 2011 is going to be all about it.  Getting it.  Storing it.  Analyzing it.  Visualizing it.  And then we’ll see the real emergence of privacy issues, like what happens when our respective governments start using simple tools like the ‘People You May Know’ from Linkedin during child pornography investigations?  It’s going to happen.  Our government isn’t ready.  The legal infrstructure isn’t there.  There will be the formation of chaos in this regard in 2011.

7.  Computer Network Attack Platforms (CNA)

In 2011, something important is going to happen.  The general populace will be made aware that, in addition to all the traditional, ground-based, we can’t win engagements in places like Afghanistan and Iraq, that we’re also involved in a different kind of war.  One that rages on every day, one that runs 24×7 and involves facets of technology present both on earth and in space and in the Internet.  And that’s cyber warfare.  World War III is here already – China has already ‘stolen’ the Internet from the United States for about 15 minutes, diverting the majority of our most important Internet based traffic through their country for storage and analysis.  Doesn’t that raise an eyebrow?  It should.  One of the biggest things that happened in 2010 went off without a hitch and without a great deal of coverage.  Iran’s nuclear power plant, the one everyone was afraid of, was rendered inoperable by a virus.  Because that plant is inoperable, Russia is going to continue to make big money.  And Israel is going to sleep easier.  Odd bedfellows, wouldn’t you say?  We’re going to learn more about CNA’s in 2011.

7.  What about NoSQL?

Yawn.  It’s going to be a TWO BILLION DOLLAR market.  Just like CEP was.  Really.

Happy New Year!

WHAT IS EVENT PROCESSING?

Event Processing, or Event Driven Architectures, means nothing more than processing an event one event at a time; preferably sometime shortly after they occur.  The opposite of this is Batch Processing, which means batching events, or messages, or what most of the world would call a row, of data and processing them together.  In batches.  Sounds simple enough, right?  All of you reading this blog post have used an Event Driven Architecture.  In fact, you’re using one now – it’s in your browser.  Can you imagine what the user experience would be if your browser ‘batched’ up all of your mouse clicks and submitted them every 30 seconds?  Event Driven Architectures promise the same type of agility and increased user experience for line of business and consumer applications that you’re experiecing right now.  In fact, it’s probably hard to think about using the web in batch mode – it just doesn’t make sense.

WHAT IS COMPLEX EVENT PROCESSING?

For the most part, a marketing phrase.  That’s right – and again, for the most part, it’s completely meaningless.  As an early and continuing contributor to this particular area of technology, I remember when StreamBase, Apama, myself, and others called this field Event Stream Processing.  Then one of those firms marketing departments decided to differentiate.  I’ll leave the specific firm to your intuition.   So, what is Event Stream Processing?  That’s much easier to answer.  Event Stream Processing is Event Processing with four additional key components:

1. Continuous Query

Rather than having to poll a server for an event, using ESP , the user of the system issues a query and is subsequently informed with events, aggregations, or patterns that satisfy the specifics of the query.  This happens continuously, until you stop the query.

2. Windows (Time and/or Length)

Using ESP, the user can ask, as an example, for an average value of some key over either a time or length window.  Something like, ‘Give me the average amount of time people have spent on the homepage in the last 10 minutes.’  This query would provide an updated average either continuously, or perhaps at regular intervals.

3. Pattern Matching

With Pattern Matching, I’m able to define a series of events that fit a pattern, and then be notifified when that pattern is observed.  Usually within some Time or Length Window.  So, I might ask, “How many users are going from the “Home Page” to the “About Company Page” and then clicking on “My Profile” during a rolling 10 minute window”.”

4. A Language

Tying all of the above together in a neat little language is a cool idea – it makes using these features easier. At least, that’s the theory.  And this is one place where CEP has gone wrong and is not the general computing revolution that myself and others have hoped for.  I’ll expound upon this after a brief distraction in the next 2 paragraphs.  Please bear with me.

WHERE IS COMPLEX EVENT PROCESSING USED?

Even Mark Palmer, who is usually extremely bullish about CEP and probably sprinkles it on his breakfast cereal, has recently admitted the CEP is only hot in Capital Markets.  While I might disagree a bit with Mark, which is nothing new, I think we can all agree that CEP, at the $200M total market size is far less than we had all hoped for.  Frankly, it reminds me of the FIX engine vendor battles – I was an early provider there too – and we all ended up fighting over an ever shrinking market place.

WHAT IS THAT MARKET ANYWAY?

The current vendor set of CEP tends to focus on Capital Markets.  But not really.  It focuses on an even smaller slice of Capital Markets called High Frequency Trading.   Seems more people know more about High Frequency Trading today than CEP. The important thing here is the what all the smart analysts are calling the “CEP Market” really isn’t the “CEP Market” at all.  It’s the HFT software market.  And again, looking at the impressively long list of clients that Mr. Palmer has cited in his recent blog post, many of those firms aren’t actually using CEP for HFT, but an even smaller subset of functionality.  That’s why the market is so small – if any VC firm thought the total addressable market for this technology was going to be $200M in 2010, no CEP startup would have received funding.  And when HFT finds the Next Big Thing, the CEP market, as defined today, will evaporate.  And along with it, any CEP vendor who has concentrated solely upon that market.

SO WHAT HAPPENED?

The idea was that we were ushering in a New Way To Compute Things.  Like all technologists who spend way too much time thinking about this stuff, we thought everyone would immediately see how smart we were, run out and buy one of the CEP based products, and join is in revolutionizing how data is turned into information and used by business folk to make money and pay our salaries.  The only problem is, we forgot 2 things; 1) who would be using our software to do this work, and 2) who would subsequently be using the applications developed by 1.

DEVELOPERS – A FINICKY BREED

I used to be a Real Developer – I wrote in C++.  Then Sun decided that the Internet was the Computer and we all started to learn Java.  Java is cool – Java makes it easy for anyone to write bad code whereas C++ really took some effort to mess things up.  More and more people started using Java for everything; servers, clients, web stuff, etc.  And now, I’m not sure what people use anymore – perhaps coders are using NoJava for all of their no shiny NoSQL apps.  I still use Java.  And I’m loathe to learn another language.  See #4 above in ‘What’s CEP?”  I don’t want to learn another language.  And I certainly don’t want to move all of my work; servers, clients, webapps, etc. to a new and unproven language.  And no matter which vendor you don’t choose for your CEP application because you write it all yourself anyway, none of their languages can claim to be broadly or generally adopted.  Proof?  Try to buy a book on one of them.  There are umpteen books out on NoSQL in less time than it took some CEP vendors to go out of business.  CEP vendors have failed to appeal to core IT departments.  Period.  And core IT departments are the folks who have to assemble all the crap they buy from vendors into something that business users get to complain about when it doesn’t work.

BUSINESS USERS – “SHOW ME SOMETHING!”

Business users want to see information.  They want to see information presented crisply; ready for decision making.  And today, more than ever, they want to see it on their web browser, iPad, iPhone, Droid, Apple TV, disconnected lap top, on flat panels on the front of their refrigerator in the kitchen and on the heads up display in their car while commuting to work.  In short, they want information any time they want information so that they can function in what has become, and will continue to become, an ever faster and more connected world.  Even Progress Apama, who I think is doing really well, uses Flash based instrumentation.  No iPad for you!  There is no CEP environment that let’s the IT folks build a complete application for the business user.  So the business user never ‘SEE’S” CEP.  So they’re not impressed.  They don’t get it.  And they don’t provide budget for stuff they don’t get.

AND IN CONCLUSION

CEP has failed to achieve the multi-billion dollar market forecasts that we all went out and raised money based upon because most vendors have failed to provide the education and tools necessary to create the complete user experience.  Most of the CEP vendors don’t even have their own visualization products – they partner with other vendors to provide things like Tree Maps or Dash Boards.  How they expect to revolutionize the world by outsourcing their interaction with the Business User is beyond me.

THE LESSON?

If the NoSQL camp would like to come anywhere close to realizing the crack smoking analysts’ estimates of a $2B market, they should 1) make the technology readily accessible to the IT department (which they’re doing) and 2) make sure that the business users knows why it’s making a difference.  If they can reach out and touch the business user, or consumer, all the better.

AND THANKS FOR READING

BUT WE WANT THE MONEY

How does all of this tie in with the SEC’s bid to build a huge consolidated audit trail? (CAT) – well, after a lot of thought, I don’t really know. And I haven’t read a lot from anyone that purports to know either. All I know is the SEC wants a lot of money to build it because, well, we need it to watch all of those evil HFT firms out there disrupting the market.

REG NMS IS BROKEN

So here’s a relevant question, if HFT wasn’t to blame for the Flash Crash, then what was? I thought Reg NMS was supposed to ensure that, no matter how many exchanges enter the fray, we were all supposed to get the best price – regardless of where the order executed. And that all of the exchanges would cooperate and there would be peace and harmony in the valley. Guess what, that’s not working.  IT would seem that the ‘structure’ of the market needs a little attention.

IF IT’S BROKEN, WHAT ARE THEY REGULATING?

So, if HFT isn’t to blame for the crash, and it boils down to some idiot shorting billions of dollars of futures in one fell swoop, then something else must be broken. And the SEC’s CAT will fix it, right? Wrong. All the SEC’s CAT proposal will do once it’s live in 4 years after billions of dollars is yet again confirm that HFT isn’t responsible for any other flash crashes as well. So why spend the money?

BECAUSE I WANT AN EMPIRE

Could it be, that the SEC, who didn’t even have the expertise required to answer the basic question, “WTF happened to the market please” could be looking to build an empire? Is this the same organization who seems to dispense different levels of disciplinary action based upon how deep the offending firms pockets are? Say it isn’t so!  Asking for billions of dollars represents a multiple over the SEC’s current operating budget.  Just what exactly do they plan on doing with all that money?  And how many firms have offered to build the capabilities for a lot less?  If the SEC doesn’t even know what the problem is, how do they know how much money to ask for and what system needs to be built?  Madness.

I AM NOT IMPRESSED

I’d really like to hear how the SEC’s going to fix the market’s structure and prevent this from happening again.  Then I’d like to see actual enforcement of the laws on the books and see people who violate these laws go to jail instead of write checks.  In short, what I’d like to see is the SEC start doing their job before asking for billions of dollars to do something that wouldn’t have prevented what happened in the first place.  Please.  Does the SEC and Mary Schapiro think we’re all just that stupid?

THAT’S RIGHT, I SAID IT

So not only am I not going to go on record with the above (go back and read it again – the SEC isn’t doing their job) but I’m always going to say that event processing technology can’t prevent future occurrences of Flash Crash (and CEP vendors who think they can really don’t know what they’re talking about).  Several vendors  (YOU KNOW WHO YOU ARE) are all too happy to not point out that the emperors new clothes, well, need some additional tailoring as they jump on the SEC and CFTC’s bandwagon, hoping for additional, regulation sourced revenue.

THE SEC COULD BE A HERO

I believe that if the SEC started doing their job again, and investor confidence returned to the equity markets, maybe we’d see a return to previous volumes.  Because right now, at the volume levels out there today, there’s going to be blood in the streets as firms collapse under the lack of trading.  All the SEC has to do is their job.  And be 1/10th as vocal about it as their request for billions of hard earned tax payers dollars.

Thanks for reading!