Today there is a lot of innovation in the big data market. However, we see a “chasm” between the SQL technologies—which are very enterprise-friendly—and the new wave of open source big data or “NoSQL” software which is used extensively by engineering organizations. In the middle is a very large number of enterprises trying to understand how they can use these new technologies to push their analytical capabilities beyond purely SQL, while at the same time utilizing their existing investments in technologies and people. This is the problem that Aster solves.

With last week’s announcement, the launch of our Teradata Aster MapReduce solutions which include Aster Database 5.0 software (formerly Aster nCluster) and our new Aster MapReduce Appliance, we bring to market the best answer for the organizations that are “caught in the middle.”  Unlike SQL-only systems focused primarily on analyzing structured data, our database and appliance provide support for native MapReduce which enables a new generation of analytics, such as digital marketing optimization, social graph analysis, fraud detection based on customer behavior, etc. Our newly extended libraries of pre-built MapReduce analytical functions allows such applications to be developed with significantly less time and cost versus other MapReduce technologies. And, unlike other MapReduce-based systems, we offer full SQL support, integration with all major BI and ETL vendors and a data adapter to EDW systems that allows enterprises to utilize existing tools and skills to bring big data analytics to their businesses. Finally, with our new appliance, we leverage Teradata’s strength and engineering to provide a proven and performance-optimized system for businesses to start analyzing untapped diverse data while cutting down on time, cost and frustration!

As we move forward, Aster is committed to being the leader in SQL and MapReduce analytics for multi-structured data. Having spent 6 years in this market, we believe that it’s not just the coolest technologies that will win, but the ones that make it easier for business analysts and data scientists within organizations to solve their business problems and innovate with analytics. With the launch of our new Teradata Aster solutions — including the revamped SQL-MapReduce interfaces and the new Aster MapReduce appliance—we are pushing the state of the art towards this direction (or as my marketing team likes to say – “bringing the science of data to the art of business”).

It enables multi-structured data to be stored natively in the database. Therefore, we can store relational data as tables with rows and columns. We can store PDF documents as PDF documents, HTML pages as HTML pages – and the same with Java objects, JPG files, Word documents, GIS data, and others.

It enables multi-structured data to be automatically (dynamically) interpreted natively in the database. For example, we can process PDF data to retrieve the various text blocks in that document, HTML pages to retrieve its content, and JPG files to render images or extract features. In other words, we can interact with the data in its native form to leverage the structure inherent in the stored data.

The final piece is that it enables a human or application user to step across the different structures seamlessly. For example, you can write a query that:

This can all be done in one workflow and one session. Impressive?

We live in interesting times. The future is opening up in front of us.

Thus a search engine can search for words, propose completion of partially typed words, do spell checking, and suggest grammar corrections “automatically”.

In the last 30 years, we’ve built specialized systems to handle each data structure differently at scale. We index a large corpus of documents in a dedicated search engine for searches, we arrange lots of words in a publishing framework to compose documents, we store relational data in a RDBMS to do reporting, we store emails in an e-discovery platform to identify emails that satisfy a certain pattern, we build and store cubes in a MOLAP engine to do interactive analysis, and so on.

Each such system is a silo – it imposes a particular structure on big data, and then it leverages that structure to do its tasks efficiently at scale.

The silo approach imposes fragmentation of data assets. It is expensive to maintain these silos. It is inefficient for humans and programs to master these silos – they have to learn the nuances of each silo to become an expert in exploiting it. As a result, we have all kinds of data administrators – a cube expert, a text expert, a spreadsheet expert, and so on.

The state of data fragmentation reminds me of the “dedicated function machines” that pre-dated the “Personal Computer”. We used to have electronic type-writers that would create documents, calculators that would calculate formulae, fax machines that would transmit documents, even tax machines that would calculate taxes. All of these machines were booted to relic-status at a museum by a general-purpose computer – the functions were ported on top of its computing framework and the data was stored in its file system. The unity of all of these functions and its data on the general-purpose computer gave rise to “integration” benefits. It made tasks easier: we can now fill our tax forms in (structured form-based) PDF documents, do tax calculations, and file taxes by transmitting the document – all on one platform. Our productivity has gone up. Indeed, the assimilation of data is leading to net new tasks that were not possible before. We can let programs search for previous year’s filings, read the entries, and populate this year’s forms from previous year’s filing to minimize data-entry errors.

We have the same opportunity in front of us now in the field of big data. For too long, have we relegated functions that work on big data to isolated “dedicated function machines.” These dedicated function machines are bad because they are not “open.” Data in a search engine can only be “searched” – it cannot be analyzed for sentiments or plagiarism or edited to insert or remove references. The data is the same, but each of these tasks requires a “dedicated function machine.”

We have the option to build a general purpose machine for big data – a multi-structured big data platform – that allows multiple structures of data to co-exist on a single platform that is flexible enough to perform multiple functions on data.

Such a platform, for example, would allow us to analyze structured payments data to identify our valuable customers, interpret sentiments of calls they made to us, analyze the most common problem across negative sentiment interactions, and predict the loss in revenue that can be prevented by solving that problem and the cost of acquiring net new customers to overcome the losses. Without a multi-structure big data platform, the above workflow is a 12-18 month cycle performed by a cross-functional team of “dedicated function experts” (CFO group, Customer Support group, Products group, Marketing group) – a bureaucratic mess of project management that produces results too expensively, too infrequently and too inaccurately, making simplifying assumptions at each step as they cannot agree on even basic metrics.

An open “Multi-Structured Big Data Platform” would be hugely enabling and open up vast efficiency and functionality that we can’t imagine today.

In our data-driven economy, business models are being created (and destroyed) and shifting based on the ability to compete on data and analytics. The winners realize the advantage of having platforms,  that allow data to be stored in multiple structures and (more importantly) allow data to be processed in multiple structures. This allows companies to more easily 1) harness and 2) quickly process ALL of the data about their business to better understand customers, behaviors, and opportunities/threats in the market. We call this “multi-structured” data, which has been a topic of discussion lately with IDC Research (where we first saw the term referenced) and other industry analysts. It is also the upcoming topic of a webcast we’re doing with the IDC on June 15^th.

To us, multi-structured data means “a variety of data formats and types.” This could include any data “structured” or “unstructured”  - “relational” or “non-relational”. Curt Monash has blogged about naming such data Poly-structured or Multi-structured. At the core is the ability for an analytic platform to both 1) store and 2) process a diversity of formats in the most efficient means possible.

Handling Multi-structured Data

We in the industry use the term “structured” data to mean “relational” data. And data that is not “relational” is called “unstructured” or “semi-structured.”

Unfortunately, this definition lumps text, csv, pdf, doc, mpeg, jpeg, html, log files as unstructured data. Clearly, all of these forms of data have an implicit “structure” to them!

My first observation is that Relational is one way of manifesting the data. Text is another way of expressing the data – Jpeg, gif, bmp and other formats are structured forms of expressing images. For example, (Mayank, Aster Data, San Carlos, 6/1/2011) is a relational row stored in a table (Name, Company Visited, City Visited, Date Visited) – the same data can be expressed in text as “Mayank visited Aster Data, based in San Carlos, on June 1, 2011.” A geo-tagged photograph of Mayank entering the Aster Data office in San Carlos on June 1, 2011 will also capture the same information.

My second observation is that “structure” of data is what makes applications understand the data and know what to do with it. For example, a SQL-based application can issue the right SQL queries to process its logic; an image viewer can interpret JPG/GIF/BMP files to interpret the data; a text-engine can parse subject-object-verbs to interpret the data; etc.

Each application leverages the structure of data to do its processing in the most efficient manner. Thus, search engines recognize the white-space structure in English and can build inverted indexes on words to do fast searches. Relational engines recognize row headers and tuple boundaries to build indexes that can be used to retrieve selected rows very quickly. And so on.

My third observation is that each application produces data in a structure that is most efficient for its use. Thus, applications produce logs; cameras produce images; business applications produce relational rows; Web content engines produce HTML pages; etc. It is very hard to “Transform” data from one structure to the other. ETL tools have their hands full in just doing transformations from a relational schema to another relational schema. And semantic engines have a hard time “transforming” text to relational forms. All such “across structure” transforms are lost in the information.

Relational databases handle relational structure and relational processing very efficiently, but they are severely limiting in their capabilities to store and process other structures (e.g., text, xml, jpg, pdf, doc). In these engines, relations are a first-class citizen; every other structure is a distant second-class citizen.

Hadoop is exciting in the “Big Data” world because it doesn’t pre-suppose any structure. Data in any structure can be stored in plain files. Applications can read the files and build their own structures on the fly. It is liberating. However, it is not efficient – precisely because it reduces all data to its base form of files and robs the data of its structure – the structure that would allow for efficient processing or storage by applications! Each application has to redo its work from scratch.

What would it take for a platform to treat multiple structures of data as first class citizens? How could it natively support each format, yet provide a unified way to express queries or analytic logic at the end-user level to as to abstract away the complexity/diversity of the data and provide insights more quickly?  It’d be liberating as well as efficient!

The Developer Portal gives our customers and partners a community for collaborating with peers to leverage the flexibility and power of SQL-MapReduce for analytics that were previously impossible or impractical. Data scientists, quantitative analysts, and developers from customers, partners, and Aster Data are using the portal to highlight insights and best practices, share analytic functions, and leverage the experience and knowledge of the community to easily harness the power of SQL-MapReduce for big data analytics.

The portal enables collaboration that is key in making it easy for our customers to become SQL-MapReduce experts so they can solve core business challenges. As Navdeep Alam, director of data architecture at Mzinga, said, the portal “will allow us the ability to share and leverage insights with others in using big data analytics to attain a deeper understanding of customers’ behavior and create competitive advantage for our business.”

We’re seeing strong interest in the Developer Portal from our current customers. Early activity and content on the portal includes discussions about using the GSL libraries, programming in .NET, and writing sessionization and sampling functions. We plan to expand on this with tutorials for additional functions over the next few months.

If you aren’t already a customer, we encourage you to get started at the Aster Data Developer Center, where you can get your hands on SQL-MapReduce by downloading Aster Data Developer Express for free and find links to other resources like www.mapreduce.org.  If you are an Aster Data customer, we encourage you to also register for access to the new SQL-MapReduce Developer Portal for additional content and learning.

We’re always interested in your feedback as to how we can better help developers learn about and use MapReduce and Aster Data’s SQL-MapReduce.  If you have any suggestions, please feel free to add them below in the comments.

The combined goal is big, as said on Teradata’s web site home page:

Today marks a major milestone in our continuing journey, and we are thrilled to join forces with the market leader in data management. Our company has achieved a lot since our inception just 5 years ago, and we look forward to accelerating our innovation and market reach even further – with the market strength of Teradata and the speed of our combined cultures. In 5 years, we’ve played a big role in shaping the Big Data Analytics Platform market and innovated on new technologies that enable customers to store diverse, granular data and process it in diverse ways. The big data opportunity as we see it is more about extracting insights from your diverse data than just finding cost-effective ways to store it. Processing and extracting deep insights from diverse and big data is where we’ve innovated and broken new ground, and with this merger we will accelerate it further.

Our journey started when we realized that (a) it was hard and expensive to manage big data, and (b) it was nearly impossible to process and analyze diverse (non-relational) data types like Web clicks, social connections, and text files at scale. The two worlds of data management and data processing were separate – RDBMSs would store and manage data in their world; however, applications and tools would do analytics outside of the database. This division severely restricted the types of analytics possible on large amounts of data. We discussed this in more detail on an earlier blog post from January 26.

The real impact of the above two restrictions was that organizations were drawing in a flood of data and couldn’t make any sense out of it. For instance, organizations couldn’t analyze enough data to understand their customers at an individual level, and thus they couldn’t improve their products and customer experience. Or, they couldn’t detect advanced fraud schemes because the offenders were hiding in terabytes of data (the outliers) and complicated money network schemes, resulting in huge losses.

Foreseeing this opportunity, we decided to change the enterprise data infrastructure and build a platform that (a) uses commodity hardware to scale at unprecedented levels while keeping costs low, (b) combines data management and data processing in one platform to allow much deeper analysis of data at much larger scale, and (c) accommodates the processing of diverse data types (e.g. machine generated data, social network data, text data, etc.) in a single platform.

Over the past 5 years we have been aggressively building our technology and developing this big new market. We’ve had continuous and increasing success – one recognition of this was Gartner’s recent Magic Quadrant. And looking forward, we were seeing a 2011 where the new market we were creating would become mainstream reality across organizations. As a Gartner press release recently stated: “2011 will be the year when data warehousing reaches what could well be its most-significant inflection point since its inception… The biggest, and possibly most-elaborate data management system in the IT house is changing. The new data warehouse will introduce new scope for flexibility in adding new information types and change detection.”

And this execution now sets the stage for our joining forces with Teradata. We love this merger for 3 reasons:

• First, we love that Teradata is by far the most successful data warehousing and data-driven applications company in the world. As founders, we understood that Teradata will accelerate our vision and will back us in realizing the full potential of the Big Data Analytics Platform.
• Second, we have always had a big and ambitious technology vision. A bold vision needs time and resources to execute to its full potential. As part of Teradata, we will have the resources and support needed to accelerate our technology. We will also have access to a global sales organization and channel to accelerate the adoption of the Big Data Analytics Platform, and ultimately bring more benefits to our customers, more quickly.
• Third, Aster Data nCluster is very complementary to Teradata’s existing product portfolio. By combining products from both companies, we can come to market with solutions that solve a very wide range of diverse data management and data analysis problems using “best of breed” components. We expect both Aster Data and Teradata customers to find our joint offerings very unique and valuable for their business, thus increasing their opportunities and decreasing their costs.

In closing, we want to re-iterate that we have never been more excited about our market, our company and our opportunity! Our vision has proven to be right early on and we’ve watch other players in our market try to follow suit – that’s just one external validation of our direction, and there have been many more as our customers use our products to break new ground in analytics insights on diverse and big data. As we innovated and as we delivered on the vision for big and diverse data management, our team’s execution has truly defined and helped shape the market. And in this evolution, we are more confident and tremendously excited as we write the next chapter of this market.

Upon close of this transaction, the merger with Teradata is about taking our products, our innovations, our IP, and the Aster Data team, and accelerating our lead in the big data and big data analytics market. Or simply put it’s about ‘going big.’

We really want to thank our customers that believed in us and drove key input into our product roadmap and see the big data opportunity. We promise that our commitment and support to you all will only increase in the future. Also our team, who joined a small company and have worked hard to make it so successful. And finally, our investors who understood the opportunity and believed they were going to be part of something new, valuable and exciting.

For more information on today’s announcement read the full press release at www.asterdata.com and also visit Teradata’s web site www.teradata.com

- Mayank Bawa & Tasso Argyros

In this process, you will see significant movement from vendors, first in their marketing collateral (as it is always the case for followers in a technology space) and then scrambling to cover their product gaps in the 5 categories that define a true analytic platform that I mentioned in Part I of 2011: – The Year of the Analytics Platform.

What took Aster Data 6+ years to build is impossible to be done overnight, or over a few releases (side note: if you are interested in software product development and haven’t read the Mythical Man-Month, now is a good time – it’s an all-time classic and explains this point very clearly), and especially if the fundamental architecture is not there from day one.

But the momentum for the analytic platform category is there and, at this point, is irreversible. Part of this powerful trend is derived from the central place that analytics is taking in the enterprise and government. Analytics today is not a luxury, but a necessity for competitiveness. Every industry today is thinking how to employ analytics to better understand its customers, cut costs, and increase revenues. For example, companies in the financial services sector, a fiercely competitive space, want to use the wealth of data they have to become more relevant to their customers, increase customer satisfaction and retention rates. Governments’ use of data and analytics is one of few last resorts against terrorism and cyber threats. In retail, the advent of Internet, social networks, and globalization has increased competition and reduced margins. Using analytics to understand cross-channel behavior and preferences of consumers improves the returns of marketing campaigns and optimizes product pricing and placement, and can make the difference between red and black ink at the bottom of the balance sheet.

I believe I’m not alone in thinking the analytic platform revolution is here to stay. Probably the strongest statement about this came October 2010 when Merv Adrian and Colin White released their research report through the BEyeNETWORK, “Analytic Platforms: Beyond the Traditional Data Warehouse” (registration required, but you’ll only be contacted by Aster Data vs the other 8 sponsors) . They did what no one vendor could do, which is build consensus and a broad view of the market forces at work which unequivocally put analytic platforms on the map as a distinct category with defined edges. The abstract says it all:

“The once staid and settled database market has been disrupted by an upwelling of new entrants targeting use cases that have nothing to do with transaction processing. Focused on making more sophisticated, real-time business analysis available to more simultaneous users on larger, richer sets of data, these analytic database management system (ADBMS) players have sought to upend the notion that one database is sufficient for all storage and usage of corporate information. They have evangelized and successfully introduced the analytic platform and proven its value.”

If you read nothing else on the topic, read this report. Colin and Merv (who has since gone on to work for Gartner) did fantastic primary research on the core capabilities lacking in traditional data warehouse systems, the business needs analytic platforms meet, and much, much more – all backed with research and statistics.

Many others are thinking along the same lines, as pointed out in a recent Information Management Newsletter:

“With enterprises becoming more aggressive in reducing their go-to-market time, alternate solutions to relational database management system products and solutions are emerging that focus on performance, efficient data storage and in-database analytical capabilities. It typically takes a fairly long time to build analytics out of transactional data and convert those insights into any tangible marketing action in any organization. The primary reason for this delay is that analytics is traditionally kept separate from the database layer. This can result in data replication, and with big data, agility does get compromised to an extent. The reason analytics is typically externalized (from the DB layer) is possibly because of the inadequacy and non-procedural nature of SQL, the language that has evolved as a standard to manipulate structured data. SQL in its native form is not meant for analytics but is intended for data storage, retrieval and creation of simple summaries.”

While this article does not talk about analytic platforms as a category, it talks about the technical reasons why critical new capabilities such as advanced in-database analytic techniques which support both declarative-based SQL as well as procedural-based languages to process data without replication outside the system, are necessary.

Curt Monash wrote a post on January 24 saying:

“I’m going to refer to an analytic RDBMS that has been extended by advanced-analytics functionality as an analytic computing system, rather than as some kind of “platform,” although I suspect the latter term is more likely to wind up winning…”

While some haven’t fully adopted the idea of an analytic platform, several are already recognizing Aster Data’s potential as a leader of the category. For instance, on January 21, CIO pointed to Aster Data as one of twenty companies to watch in 2011:

“Using a combination of embedded analytics and a high performance, large scale data management platform, Aster Data supports complex and data intensive applications such as those for web analytics, customer behaviour analytics and fraud detection. With organisations looking to find new ways to use and analyse the vast amount of data they have been collecting but not exploiting in 2011, Aster Data and its analytic database platform is well positioned to make the most of this opportunity.”

I’m interested in hearing your thoughts about this. Leave a comment below to share your own insights…

Fast forward to 2008 when Aster Data led the way in putting massive parallel processing inside a MPP database, using MapReduce, to advance how you process massive amounts of diverse data. While this was fully aligned with our vision for managing hoards of diverse data and allowing deep data processing in a single platform, most thought it was intriguing but couldn’t quite see the light in terms of where the future was going. At one point, we thought of naming our product XAP – “extreme analytic platform” or “extreme analytic processing” as that’s what it was designed to do from day one. However, we thought better of it since we thought we would have to educate people too much on what an “analytic platform” was and how it was different from a traditional DBMS for data warehousing. Since we also were serving the data architects in organizations as well as the front-line business that demands better, faster analytics, we needed to use terminology that resonated with both.

Then, in the fall of 2009, with our flagship product Aster Data nCluster 4.0, we made further strides in running advanced analytics inside the database by including all the built-in application services (e.g. like dynamic WLM, backup, monitoring, etc) to go with it. At that time, we referred to it as a Data-Application Server – which our customers quickly started calling a Data-Analytics Server.  I remember when analyst Jim Kobielus at Forrester said,

“It’s really innovative and I don’t use those terms lightly. Moving application logic into the data warehousing environment is ‘a logical next step’.”

And others saying,

“The platform takes a different approach from traditional data warehouses, DBMS and data analytics solutions by housing data and applications together in one system, fully parallelizing both. This eradicates the need for movements of massive amounts of data and the problems with latency and restricted access that creates.”

What they started to fully appreciate and realize is that big data is not just about storing hoards of data, but rather, cracking the code on how to process all of it in deep ways, at blazing fast speeds.

By 2010, everyone whose roots were in cost-effective data warehousing suddenly started claiming to be a platform for deep analytics. How ironic — since none of the other vendors had even built any fundamental architecture that lent itself to being a true analytics machine.  Aster Data did so – from the very beginning – with a deep in-database analytics engine; SQL-MapReduce analytics processing; data-application services so you could really run all procedural code in-database and support it with all the key application services; out-of-the-box analytics modules (that now number over 1000); and a visual development environment for point-and-click development of apps and push down of apps into our platform.

By late 2010, the term “analytic platform” started to take shape. The definition of it fit exactly with what Aster Data has built. And now, traditional DW appliances are claiming to be analytic platforms. Even Netezza is taking the same box they had before and calling it “An Appliance for Deep Business Analytics,” and pure columnar MPP DBMS’s like Vertica and ParAccel overnight went from being ‘the world’s fastest database’ to ALL claiming to be an analytic(s) platform.  This is a typical marketing trajectory if you now see where the future lies in big data management.  The market as a whole is gravitating to accept that if you truly want to manage big, diverse data, you ultimately want to analyze all of it, and for that you’re really in need of a big data analytic platform – not just a big data store. Recently, Curt Monash supports a similar notion when describing choices in analytic computing system design.

I predict this year will be the year where the analytic platform – which we at Aster Data started to talk about and deliver in 2008 – will now be a distinct and unique category: distinct from an enterprise data warehouse (EDW); distinct from traditional DBMSs; distinct from even some pure MPP DBMSs; and distinct from even Hadoop.

An analytic platform, put simply, must have the following:

1. Native in-database processing engine for application embedding that provides the capability to run applications inside an MPP database with high performance and high reliability and provides necessary services so that applications can process the right data at the right time.  This is not UDFs, which due to architectural limitations have always been a small “niche” in the RDBMS world, or Stored Procedures which can’t match the performance and flexibility needed to push applications inside in a large scale MPP system.

2. Native support for MapReduce.  Just like the world needs a language for basic data management (SQL), it also needs a framework for writing and deploying applications inside the data analytics platform.  MapReduce is by far the most prominent and promising interface for doing exactly that. Hadoop’s open source popularity will only propel MapReduce’s significance and already MapReduce has become the de-facto standard for writing large, data intensive parallel applications inside the database.  “Native” means that MapReduce is not built on top of UDFs or Stored Procedures, nor is it a side-implementation on the DBMS, nor is it a simple Hadoop connector as every other DBMS vendor has done.  All these approaches make MapReduce totally unusable for big data applications & analytics.

3. Tight integration with SQL. A great Analytics Platform for diverse data management, built for the Enterprise needs to respect SQL or otherwise risk running into objections in the Enterprise world where the majority of the skill set is around SQL. As of now, Aster Data’s patent-pending SQL-MapReduce is the only technology that manages to tightly blend two different worlds – SQL and MapReduce – together.  Integration means that SQL analysts can seamlessly use MR applications inside the database, and MR data scientists can use the flexibility and power of SQL to complement their MR applications.

4. Enterprise feature support for in-database applications – High Availability; Backup/Restore; Access and Security; Resource Partitioning and Prioritization for concurrent operations (WLM). Several database systems offer these capabilities when it comes to SQL queries; but none offers all the same when it comes to in-database applications.  Having the same support for both interfaces is critical because otherwise in-database apps are naturally forced into a second-citizen status making DBAs reluctant to let users run in-db apps and Enterprise security and compliance groups reserved about risks of going beyond SQL. Aster Data’s platform is unique in that SQL is just another interface, just like MapReduce.  All platform capabilities have been built as services and treat each interface in the same way, resulting in 100% enterprise feature support for both interfaces.

5. Making in-database application development EASY and cost-effective.  An otherwise great Analytical Platform that’s too hard and too expensive to use will end up doing no analytics and thus does not deserve to be called an Analytical Platform.  This is why elements like the following are critical: libraries of pre-packaged SQL/MR analytic functions; 3rd party in-database application integrations; integrated, graphical, development environment for in-database apps and MapReduce; data visualization tools; unified monitoring and debugging of SQL and in-database application processes; and integrations with front-end visualization tools that can distribute the power of MapReduce to 1000s of analysts and business users.

So from what I see in organizations I talk to every week, their need for a new Analytics Platform for managing and processing big, diverse data is loud and clear. I think this is the year when it is validated and legitimized as a new and distinct category. If you agree – or disagree for that matter – leave a comment or point me to a resource that argues the counter points.  I look forward to hearing from you.

MapReduce (the framework). MapReduce is a framework that allows programmers to develop analytical applications that run on (usually large) clusters of commodity hardware and process (usually large) amounts of data. It was first introduced by Google and it is language independent. It is abstract in the sense that an application that uses MapReduce doesn’t have to care about things like the number of servers/processes, fault tolerance, etc. MapReduce is supported by multiple implementations including the open source project Hadoop and Aster Data. Google also has its own proprietary implementation which, unfortunately, is also called MapReduce and sometimes creates confusion.

MapReduce (the Google implementation of MapReduce framework). As mentioned above, Google has its own implementation of MapReduce. This was described in the 2004 OSDI paper and it was the theoretical basis upon which Hadoop was developed. Google’s MapReduce was a processing framework and it was using Google’s GFS (Google File System) for data storage.

Aster Data’s SQL-MapReduce. Aster Data has a patent-pending implementation of MapReduce that (a) uses a database for data persistence, (b) is tightly integrated with SQL, i.e. an analyst or BI tool can invoke MapReduce via SQL queries, thus making MapReduce accessible to the enterprise. It supports multiple programming languages such as Java and C and it is accessible through standard interfaces such as ODBC and JDBC.

Hadoop. Hadoop is an Apache “umbrella” project that hosts many sub-projects, including Hadoop MapReduce and HDFS, Hadoop’s version of the Google File System which Hadoop MapReduce uses for data storage. Hadoop is the core open source project – however, there are many distributions for Hadoop, just as there are many distributions for Linux. These distributions contain Hadoop binaries together with other utilities and tools. The most popular distributions are the Cloudera distribution, the Yahoo distribution and the baseline Apache distribution.

HDFS. HDFS is Hadoop’s version of GFS and it is a distributed file system. HDFS can exist without Hadoop MapReduce, but usually Hadoop MapReduce requires HDFS. Aster Data’s MapReduce does not require HDFS as it uses an extensible MPP database for data storage and persistence.

Cloudera. Cloudera usually means either (a) the company, (b) Cloudera’s Distribution for Hadoop.

Sqoop. Sqoop which is short for “SQL to Hadoop” is an open source project that provides a framework for connecting to SQL data stores for data exchange.

NoSQL. NoSQL started as a term to describe a collection of products that did not support or rely on SQL. This included Hadoop and other products like Cassandra. However, as more people realized that SQL is a necessary interface  for many data management systems, the term evolved to mean (N)ot (o)nly SQL. These days there are attempts to port SQL on top of Hadoop and other NoSQL products.

Are there any MapReduce-related terms I omitted? Please add them in the comments below and include a definition and links to good resources if you’d like.

Today at a press conference in San Francisco we announced the general availability of our Dell cloud solutions. One of the solutions we debuted was the Dell Cloud Solution for Data Analytics, a combination of our PowerEdge C servers with Aster Data’s nCluster, a massively parallel processing database with an integrated analytics engine.

Earlier this week I stopped by Aster Data‘s headquarters in San Carlos, CA and met up with their EVP of marketing, Sharmila Mulligan. I recorded this video where Sharmila discusses the Dell and Aster solution and the fantastic results a customer is seeing with it.

Some of the ground Sharmila covers:

• What customer pain points and problems does this solution address (hint: organizations trying to manage huge amounts of both structured and unstructured data)
• How Aster’s nCluster software is optimized for Dell PowerEdge C2100 and how it provides very high performance analytics as well as a cost effective way to store very large data.
• (2:21) InsightExpress, a leading provider of digital marketing research solutions, has deployed the Dell and Aster analytics solution and has seen great results:
  • Up and running w/in 6 weeks
  • Queries that took 7-9 minutes now run in 3 seconds

Pau for now…

Extra-credit reading

• Aster Data and Dell Partner to Provide Customers with Integrated Solutions for Large Scale Data Management and Advanced Analytics
• Aster Data and Dell Selected by InsightExpress to Enable Advanced Data Analytics for Digital Marketing Research
• Talking to Aster Data’s brand new CEO
• An Overview: Aster’s Big Data Architecture