Every year or so Google comes out with an interesting piece of infrastructure, always backed by claims that it’s being used by thousands of people on thousands of servers and processes petabytes or exabytes of web data. That alone makes Google papers interesting reading.

This latest piece of research just came out on Google’s Research Buzz page. It’s about a system called Dremel (note: Dremel is a company building hardware tools which I happened to use a lot when I was building model R/C airplanes as a kid). Dremel is an interesting move by Google which provides a system for interactive analysis of data. It was created because it was thought that native MapReduce has too much latency for for fast interactive querying/analysis. It uses data that sits on different storage systems like GFS or BigTable. Data is modeled in a columnar, semi-structured format and the query language is SQL-like with extensions to handle the non-relational data model. I find this interesting - below is my analysis of what Dremel is and the big conclusion.

Main characteristics of the system:

Data & Storage Model
• Data is stored in a semi-structured format. This is not XML, rather it uses Google’s Protocol Buffers. Protocol Buffers (PB) allow developers to define schemas that are nested.
• Every field is stored in its own file, i.e. every element of the Protocol Buffers schema is columnar-ized. Columnar modeling is especially important for Dremel for two specific reasons:
- Protocol Buffer data structures can be huge (> 1000 fields).
- Dremel does not offer any data modeling tools to help break these data structures down. E.g. there’s nothing in the paper that explains how you can take a Protocol Buffers data structure and break it down to 5 different tables.
• Data is stored in a way that makes it possible to recreate the orignial “flat” schema from the columnar representation. This however requires a full pass over the data - the paper doesn’t explain how point or indexed queries would be executed.
• There’s almost no information about how data gets in the right format, how is it stored, deleted, replicated, etc. My best guess is that when someone defines a Dremel table, data is copied from the underlying storage to the local storage of Dremel nodes (“leaf nodes”) and at the same time is replicated across the leaf nodes. Since data in Dremel cannot be updated (it seems to be a write-once-read-many model), design & implementation of the replication subsystem should be significantly simplified.

Interface
• Query interface is SQL-like but with extensions to handle the semi-structured, nested nature of data. Input of queries is semi-structured, and output is semi-structured as well. One needs to get used to this since it’s significantly different from the relational model.
• Tables can be defined from files, e.g. stored in GFS by means of a “DEFINE TABLE” command.
• The data model and query language makes Dremel appropriate for developers; for Dremel to be used by analysts or database folks, a different/simpler data model and a good number of tools (for loading, changing the data model etc) would be needed.

Query Execution
• Queries do NOT use MapReduce, unlike Hadoop query tools like Pig & Hive.
• Dremel provides optimizations for sequential data access, such as async I/O & prefetching.
• Dremel supports approximate results (e.g. return partial results after reading X% of data - this speeds up processing in systems with 100s of servers or more since you don’t have to wait for laggards).
• Dremel can use replicas to speed up execution if a server becomes too slow. This is similar to the “backup copies” idea from the original Google MapReduce paper.
• There seems to be a tree-like model of executing queries, meaning that there are intermediate layers of servers between the leaf nodes and the top node (which receives the user query). This is useful for very large deployments (e.g. thousands of servers) since it provides some intermediate aggregation points that reduce the amount of data that needs to flow to any single node.

Performance & Scale
• Compared to Google’s native MapReduce implementation, Dremel  is two orders of magnitude faster in terms of query latency. As mentioned above, part of the reason is that the Protocol Buffers are usually very large and Dremel doesn’t have a way to break those down except for its columnar modeling. Another reason is the high startup cost of Google’s MapReduce implementation.
• Following Google’s tradition, Dremel was shown to scale reasonably well to thousands of servers although this was demonstrated only over a single query that parallelizes nicely and from what I understand doesn’t reshuffle much data. To really understand scalability, it’d be interesting to see benchmarks with a more complex workload collection.
• The paper mentions little to nothing about how data is partitioned across the cluster. Scalability of the system will probably be sensitive to partitioning strategies, so that seems like a significant omission IMO.

So the big question – can MapReduce itself handle fast, interactive querying?
• There’s a difference between the MapReduce paradigm, as an interface for writing parallel applications, and a MapReduce implementation (two examples are Google’s own MapReduce implementation, which is mentioned in the Dremel paper, and open-source Hadoop). MapReduce implementations have unique performance characteristics.
• It is well known that Google’s MapReduce implementation & Hadoop’s MapReduce implementation are optimized for batch processing and not fast, interactive analysis. Besides the Dremel paper, look at this Berkeley paper for some Hadoop numbers and an effort to improve the situation.
• Native MapReduce execution is not fundamentally slow; however Google’s MapReduce and Hadoop happen to be oriented more towards batch processing. Dremel tries to overcome that by building a completely different system that speeds interactive querying. Interestingly, Aster Data’s SQL-MapReduce came about to address this in the first place and offers very fast interactive queries even though it uses MapReduce. So the idea that one needs to get rid of MapReduce to achieve fast interactivity is something I disagree with - we’ve shown this is not the case with SQL-MapReduce.

As the market around big data heats up, it’s great to see the ecosystem for Hadoop, MapReduce, and massively parallel databases expanding. This includes events for education and networking around big data.

As such, Aster Data is co-sponsoring our first official “unconference” the night before the 2010 Hadoop Summit. It’s called BigDataCamp and will be June 28th at the TechMart from 5:00-9:30PM (adjacent to the Hyatt where Hadoop Summit is taking place). Similar to our ScaleCamp event last year where we heard from companies like LinkedIn and ShareThis and industry practitioners like Chris Wensel (author of Cascading), there will be a lineup of great talks, including hands-on workshops led by Amazon Web Services, Karmasphere, and more. In addition, we’re lucky to have Dave Nielsen as the moderator/organizer of the event as he’s chaired similar unconferences such as CloudCamp, and is an expert at facilitating content and discussions to best fit attendee interest.

It’s very fitting to have the more open/dynamic agenda style of an unconference given the audience will be more of the “analytic scientists” - a title which I’ve seen LinkedIn use when describing the rise in  job roles dedicated to tackling big data in companies to tease out insights and develop data-driven products and applications. The analytic scientist-customers I speak with who use Aster Data together with Hadoop challenge the norms and move quickly - not unlike an unconference agenda. I expect a night of free thinking (and free drinks/food), big ideas, and a practical look at emerging technologies and techniques to tackle big data. Best of all, the networking portion is a great chance to meet folks to hear what they’re up to and exchange ideas.

Check out the agenda at www.bigdatacamp.org and note that seats are limited and we expect to sell out, so please REGISTER NOW. Hope to see you there!

We’re a couple of days away from our second Big Data Summit event in as many calendar quarters, and it’s shaping up to be jam-packed with good presentations and conversation around innovations in data management and advanced analytics.  For people not aware, we’re conducting Big Data Summits regionally in North America and Europe with an eye on helping educate organizations who are looking for ways to tackle the enormous amount of data growing in (and outside) of their four walls. More importantly, we’re helping people answer the question, “what do I do with all this data?”.  If you can’t make it to the Bay Area on Feb 18th, look for one coming soon to a venue near you.  Here are some previews of what’s coming on Thursday:

1) Intuit (formerly mint.com) will be talking about how they anonymize consumer data and provide financial benchmarks and insights to individuals.  e.g., Do you spend more on dinner than the average Joe or Jane in California?

2)  Mobclix, a mobile advertising network, will discuss how they enable application developers for the iPhone, Android, BlackBerry, etc. to make more money by providing targeting advertising based on better analytics. (And how they’ve deployed it on Amazon Web Services to scale up & down on demand)

3) Merv Adrian from IT Market Strategy will keynote to talk about trends in data management and the questions you need to ask yourself as you consider ways to tackle big data problems. He’s also moderating a panel that I’m hearing will have some cameo appearances from other industry analysts with big brains.

4) Dell is our platinum sponsor and they’ll be talking about some new hardware they’re rolling out for big data computing and putting on display at the show.

5) Our CEO and co-founder will be talking about our approach to harnessing the power of big data and giving a glimpse of the future from Aster Data.

6) Other sponsors include Amazon Web Services, Informatica, and a new partner of ours, Impetus.

Hope to see you at the show.

Aster Data 4.0 is here and for those of you who subscribe to Aster Data’s blog, “Winning with Data”, you may have noticed that we’ve changed things up a bit.  This blog is now called the “Big Data Blog” and will continue to be a mash-up of opinions and news from the team at Aster Data. Topics will continue to be a mix of technical deep-dives as well as company announcements and content.

At the same time, our CEO and co-founder Mayank will be sharing his thoughts on a separate blog called “Winning with Data” where he will talk about his perspectives of the market trends, customer use-cases, technology evolution and company growth. You can find all of his previous posts there, as well as fresh content starting with the announcement of Aster Data’s massively parallel data-application server.

On Friday we had series of events and announcements around our new Aster-Hadoop Data Connector which utilizes key new SQL-MapReduce functions to provide ultra-fast, two-way data loading between HDFS (Hadoop Distributed File System) and Aster Data’s MPP data warehouse.

In addition to the Big Data Summit we held in New York City (which we’ll detail in a separate post), Colin White presented on a Webcast with Aster on the various use-cases for Hadoop within data warehouse environments. Colin does a great job summarizing what Hadoop is, how it’s different from an RDBMS, the different types of users for each, and how they co-exist nicely in customer environments.

Below are the slides to view if you weren’t able to attend the event, which will also be available for on-demand viewing soon in our resource library.

After a long thinking process, I have decided to step down from my day-to-day operating role as Chief Scientist. I will be checking in on the team every now and then, and I will continue to serve Aster on the board of directors, where we shape the strategic direction of the company and products.

Aster is now in a new phase, when maximal focus on product delivery and responding to customers’ every need is taking center stage. We have a fantastic product that is growing quickly and is being deployed at many, many customer sites. The level of product focus in our team is astounding. Yes, my friends, the garage days are now officially over! … And I’m delighted we’ve come this far.

One should always invest their energy in those activities that most benefit from that energy. Aster is a grown-up now and is solidly set on a path with great momentum toward executing our “big data” vision. At the same time, I have a budding team of incredibly bright students back in my lab at EPFL, who are hungry for my help and scientific direction.  For me, it’s time for the next “garage project.”

As you might imagine, this is not an easy decision for a founder. Perhaps this is how parents feel when sending their kid to college away from home… but time does not stand still.

It has been a thrill to start Aster with Mayank and Tasso, see her emerge from 5 PCs cooled by a portable fan in my living room, and then working for her all these years. I’m proud of what we’ve achieved so far, both in terms of product and market impact, as well as our published research papers. At a future stage in Aster’s growth, it may make sense for me to resume my role in leading the company’s long-term research. Until then, I look forward to helping the company from behind the scenes and to watch Aster grow and develop the best data management platform in the market.

Carpe datum!

Aster Data has seen tremendous growth in North America. We announced today that we have opened a Europe office in West London, England. The office will be headed by Bob Pearson, our newly appointed Europe Area Director. Bob is an entrepreneurial industry leader and had earlier introduced Opsware into Europe, eventually propelling Opsware to be #1 in Europe in its market. We had been in conversations with Bob for 12 months - understanding the European market - before we opened our office this summer.

We also announced today that our first customer in Europe is the #1 online poker gaming site in the world, Full Tilt Poker. We have been working with Full Tilt Poker for 8 months now helping deploy Aster nCluster to power their fraud prevention systems and provide enhanced customer service to their players.

It is no surprise that data size growth is a world-wide phenomenon, and certainly occurs across “the pond” as well. We have noticed that European customers in numerous industries, such as financial services and insurance, online retailing, social networking, communications, and gaming are deploying new (and sometimes custom) applications to leverage big data.

Aster Data is certainly the most application friendly big-data infrastructure in the market, and we look forward to working with our European customers in the coming years!

Netezza pre-announced last week that they will be moving to a new architecture - one based around IBM blades (Linux + Intel + RAM) with commodity SAS disks, RAID controllers, and NICs. The product will continue to rely on an FPGA, but that would sit much further from the disks & RAID controller, beyond the RAM but adjacent to the Intel CPU, in contrast to their previous product line.

In assembling a new hardware stack, Netezza calls this re-architecture as a change but not really a change - the FPGA will continue to offload data compression/decompression, selection and projection from the Intel CPU; the Intel CPU will be used to push-down joins and group bys; the RAM will be used to enable caching (thus helping improve mixed workload performance).

I think this is a pretty significant change for Netezza.

Clearly, Netezza would not have invested in this change - assemble & ship a new hardware stack to share revenue with IBM vs. a 3rd party hardware assembler - if Netezza’s old FPGA-dominant hardware was not being out-priced and out-performed by our Intel-based commodity hardware.

It was a matter of time before the market realized that FPGA’s had reached their end-of-life status in the data warehousing market. In realizing the writing on the wall, and responding to it early, Netezza has made a bold decision to change - and yet, clung to the warm familiarity of an FPGA as a “side car”.

Netezza, and the rest of the market, will soon become aware that a change in hardware stack is not a free lunch. The richness of CPU and RAM resources in an IBM commodity blade come at a cost that a resource-starved FPGA-based architecture never had to account for.

In 2009, after having engineered its software for an FPGA over the last 9 years, Netezza will need to come to terms with commodity hardware in production systems and demonstrate that they can:

- Manage processes and memory spawned by a single query across 100s of blade servers

- Maintain consistent caches across 100s of blade servers - after all, it is Oracle’s Cache Fusion technology that is the bane of scaling Oracle RAC beyond 8 blade servers

- Tolerate the higher frequency of failures that a commodity Linux + RAID Controller/driver + Network driver stack incur when put under rigorous data movement (e.g., allocation/de-allocation of memory contributing to memory leaks)

- Add a new IBM blade and ensure incremental scaling of their appliance

- Upgrade the software stack in place - unlike an FPGA-based hardware stack that customers are OK to floor-sweep in their upgrade

- Contain run-away queries from allocating the abundant CPU and RAM resources and starving other concurrent queries in the workload

- Reduce network traffic for a blade with 2 NICs that is managing 8 disks vs. a Power-PC/FPGA that had 1 NIC for 1 disk

- …

If you take a quick pulse of the market, apart from our known installations of 100+ servers, there is no other vendor - mature or new-age - who has demonstrated that 100’s of commodity servers can be made to work together to run a single database.

And I believe that there is a fundamental reason for this lack of proof-point even a decade after Linux has matured and commodity servers have been used for computing - software not built from the ground-up to leverage the richness and contain the limitations of commodity hardware is incapable of scaling. Aster nCluster has been built ground up to have these capabilities on a commodity stack. Netezza’s software written for proprietary hardware cannot be retrofitted to work on commodity hardware (else, Netezza would have completely taken the FPGAs out, now that they have powerful CPUs!). Netezza has its work cut-out - they have taken a dramatic shift that has the ability to bring the company and its production customers to its knees. And there-in lies Netezza’s challenge - they must succeed while supporting their current customers on an FPGA-based platform while moving resources to build out a commodity-based platform.

And we have not even touched upon the extension of SQL with MapReduce to power big data manipulation using arbitrary user-written procedures.

If a system is not fundamentally designed to leverage commodity servers, it’s only going to be a band-aid on seams that are bursting. Overall, we will curiously watch how long it takes Netezza to eliminate their FPGAs completely and move to a real commodity stack so that the customers can have the freedom to choose their own hardware and not be locked down to Netezza-supplied custom hardware.

 Rajeev was a close friend and a cherished mentor. We were saddened to hear the news today and we will miss him dearly. Our thoughts are with his family.

I’m delighted to welcome Specific Media to the quickly-growing family of Aster customers! I had the pleasure of briefly meeting the folks from Specific Media in our offices last week. Similar to Aster, Specific Media is incredibly focused on doing more with data to increase the value they provide to their customers: advertisers which represent 300 of the top Fortune 500 brands.

They’re also really smart and humble about what they do, which makes it a pleasure to work with them. And what you wouldn’t know from a brief introduction is how cutting-edge their analytic methodologies and capabilities are.  We’re just starting our partnership together and hope to have some success metrics to share later about how they are using the Aster nCluster database for their data warehouse. They have some interesting ideas for using the Aster In-Database MapReduce framework to perform rich analysis of data efficiently for improved ad targeting and relevancy.