Dealing with the data tsunami

The approach is well known – there is no debate -- parallelism is the answer. This answer is a result of hard physics, heat and power.

We don’t need to think about this, but we do need to implement the systems.

Systems consist of hardware and software. The hardware is there, the software isn’t.

Note that the software is there for transactional work, so half of the problem is solved. We are glad that it is, because that half is creating:
• a huge business in multicore machines
• a huge effort to collect, store, and distribute the data

So with the hardware platform at hand (go to Fry’s and slap down a credit card) and the data available, the missing piece required to realize the hidden value and translate raw data into meaningful intelligence is the software architecture.

Why do we think this? Some might disagree about one or more of these points, but they are our view. And we are not alone, the big-time analysts (Bloor, IDC, etc.) agree with these statements.

Assertion #1 Developers aren’t ready. This is a known truth in the general developer community. Although classes, seminars, books, and articles are all trying to help, it is a really hard problem and everyone knows it. If a programmer thinks it is easy, then they don’t know what they are doing or haven’t yet even tried.

Assertion #2 Languages aren't ready. We see lots of interest in ‘new’ alternatives such as Erlang, Haskell, Threading Building Blocks, etc., because programmers know that the low level functions are just that, too low level.

Assertion #3 This isn't going to solve itself - there is no silver bullet coming. No free lunch, Intel isn’t going to ride in and save the day, Amazon computing in the sky isn’t going to make the problems go away

We are certain that DataRush can help with this challenge - how are you dealing with it today?