In a previous post, I outlined a design pattern that we call “The Automated Feedback Loop.” In this post, I want to outline a design pattern that we call “The (Iterative) Analytics Data Warehouse”.

The traditional well-understood design pattern of a data warehouse is a central (for Enterprise Data Warehouse) or departmental (for Data Marts) repository of data. Data is fed into the warehouse from ETL processes that pull data from a variety of sources. The data is organized in a data model that caters to 3 use-cases of the warehouse:

1. Reports - A set of BI queries are run with regular frequency to monitor the state of the business. The target of the reports are business users who want to understand what happened. The goal is to keep them in touch with the pulse of the business.
2. Exports - A set of export jobs are run with adhoc frequency to provide data sets for further analysis. The target of the exports are business analysts who want to optimize business practices. The goal is to provide them with true, quality-stamped data so that they can make confident optimization recommendations.
3. Adhoc - A set of queries are run with adhoc frequency to detect or verify patterns that influence business events. The source of the queries are data scientists who want to understand and optimize business practices. The goal is to provide them with computation capabilities (good query interfaces, enough processing, memory and storage resources) to allow them to interact with the data.

The exports and adhoc tasks are transient tasks. Once the data analysts or data scientists find a pattern valuable to the business, that pattern is incorporated into a report so that business users can monitor that pattern on a frequent repeatable practice.

In a typical data warehouse, the bulk of tasks (~80%) are from [1] Reports. The remainder of 20% is from [2] Exports and [3] Adhoc.

Since Reports are frequent and generate known queries, the design of the data warehouse is done to cater to reporting. This includes data models, indexes, materialized views or derived tables – and other optimizations – to make the known Reporting queries go fast.

Since exports and adhoc tasks are infrequent and generate unknown queries, the design of the data warehouse is unable to cater to them upfront. This means that exports and adhoc tasks generate queries that are harder to satisfy (since they have to fight the data modeling decisions made for reporting) and therefore impose more load on the data warehouse.

The net result is that reports run fast while exports and adhocs are slow. In fact, exports and adhocs consume so much resources, that reporting starts running slower. And that is not good – reports are distributed widely and reach a wide variety of business users. They are unhappy and put pressure on the data warehousing team to “get the reports in time.” At the same time, the export and adhoc users are unhappy because they can’t get to the data fast enough to benefit the business.

The poor data warehouse team now looks for solutions invariably prioritizing reports. Historically, the answer was to prioritize via workload management – constrain adhoc & export usage to devote resources to reporting. If workload management didn’t work, the answer was to create walled gardens by enforcing rules -

“thou shalt not report when data is loading; thou shalt not adhoc query when reports are being generated; thou shalt not export except in the evening”.

Let’s call this design pattern of data warehouse to be a “Reporting Data Warehouse”.

The first-class citizen of a “Reporting Data Warehouse” is reports. The exports and adhoc are second-class citizens – they do not get dedicated data models, they do not get large chunks of resources, and their protests are answered by asking them to constrain their requirements (use samples, use rolled up aggregates that were built to make reports faster, use smaller timeframes of history that were retained to just satisfy reporting requirements, phrase queries that are simpler even though they may be compromises on the pattern sought, …).

This motivates the definition of a different design pattern of a data warehouse whose use is to be an “Analytical (Iterative) Data Warehouse“.

The first-class citizens of an “Analytical Data Warehouse” are exports and adhoc analytics, and the primary users of the Analytical Data Warehouse are business analysts and data scientists. The data models are built to support their adhoc usage – fine-granularity data is retained, rich dimension tables are frequently imported, derived views and tables are created promptly, interfaces are opened up to express their patterns in a computationally simple and natural manner, scale-out is used to create resources for the tasks to finish interactively, enough storage is allocated for several exports to proceed simultaneously.

In other words, the infrastructure and the team exist to support export and adhoc usage as their primary customers.

Once an insight is confirmed, it can be added with careful design to the Reporting Data Warehouse – with carefully defined data models, indexes and materialized views and support to maintain it during the ETL process.

The infrastructure can play a significant role in enabling Analytical Data Warehouses.

1. Query interface to support in-database computations: The export and adhoc queries want to manipulate data in rich ways, and often SQL is not enough. The infrastructure should support an easy-to-express interface for rich computations (e.g., SQL/MapReduce). This is important because the correct perspective of data, amenable to downstream manipulation, cannot be defined upfront in the ETL process.
2. Incremental scale-out MPP capabilities: The infrastructure should allow for an ability to scale out both storage and computing resources incrementally and easily (i.e., without months of planning). This is important because temporary storage requirements (as data is transformed for analysis) or temporary processing requirements (as several models are generated to validate insights) at the peak of the analysis can be much higher than normal use.
3. Cheap hardware: The size of data demanded by exports and adhoc users may be large and the computations may be rich. The infrastructure must enable analysis of data at a cheap operating point.
4. Workload manage both export and analysis: Some data analysts are comfortable in manipulating data in their preferred tools (e.g., Excel, SAS, Matlab) – others are comfortable writing in-database queries (SQL, Java, C++, Perl, Python). The infrastructure should elegantly manage all tasks, and not require a “walled garden” to favor queries over export, or vice-versa.

My observation has been that the design methodology of an Analytical Data Warehouse is substantially different from a Reporting Data Warehouse. Understanding the primary customer of a data warehouse can often help simplify operations of the data warehouse and help lower the operating point costs substantially by making priorities clearer.

Both Aster Data and Greenplum made announcements this week that illustrated different approaches. At the same time that Aster Data announced the Aster Analytics Center, Greenplum announced an upcoming product named Chorus. I wanted to take a moment to compare and contrast what these announcements say about the direction of the two companies.

Greenplum’s approach speaks to two traditional problem areas i) access to data, from provisioning of data marts to connectivity to data across marts, and ii) some level of collaboration among certain developers and analysts. Their approach is to create a tool for provisioning, unified data access, and sharing of annotations and data among different developers and analysts. Interestingly, this is not an entirely new concept; these are well-known problems for which a number of companies and tools have already developed best-of-breed solutions over the last 15 years. For example, the capabilities for data access are another version of Export/Copy primitives that already exist in all databases and that have been built upon by common ETL and EII tools for cases in which richer support than Export & Copy are needed – for instance, when data has to be transformed, correlated or cleaned while being moved from one context (mart) to another (mart).

This approach is indicative of a product direction in which the primary focus is on adding another option to the list of tools available to customers to address these problems. It’s really not a ground-breaking innovation that evolves the world of analytics. New types of analytics, or ‘data-driven applications,’ is where the enormous opportunity lies. The Greenplum approach of data collaboration is interesting in a test environment or sandbox. When it comes to real production value however, it effectively increases the functions available to the end user, but at a big cost due to significant increases in complexity, security issues and extra administrative overhead. What does this mean exactly?

• The spin-up of marts and moving data around can result in “data sprawl” which ultimately increases administrative overhead and is dangerous in these days of compliance and sensitivity to privacy and data leaks.
• Adding a new toolset into the data processing stack creates difficult and painful work to either manage and administer multiple tool sets for similar purposes or to eliminate and transition away from investments in existing toolsets.
• To enable effective communication and sharing, users need strong processes and features for source identification of data, data collection, data transformation, rule administration, error detection & correction, data governance and security. The quality and security policies around meta-data are especially important as free-form annotations can lead to propagation of errors or leaks in the absence of strong oversight.

In contrast, Aster Data’s recent announcements support our long-standing investments in our unique advanced in-database architecture where applications run fully inside Aster Data’s platform with complete application services essential for complex analytic applications. The announcements highlight that our vision is not to create a new set of tools and layers in the data stack that recreate capabilities currently available from a number of leading vendors, but rather to deliver a new Analytics Platform, a Data-Application Server, to uniquely enable analytics professionals to create data-rich applications that were impossible or impractical before – namely, to create and use advanced analytics for rich, rapid, and scalable insights into their data. This focus is complemented by our partners, who offer proven best-of-breed solutions for collaboration and data transformation.

A key illustration of the investments that Aster is making in this vision is the formation of the new Aster Analytics Center: a center of excellence; ready-to-use analytics solutions that leverage MapReduce; and best practices for advanced analytics on big data. The Center’s charter is to develop products and provide insights that help organizations use data in clever ways to enable data-driven decisions. The Center is headed by Dr. Jonathan Goldman, our Director of Analytics and Applications, and a team of analytics experts. Jonathan joined us from LinkedIn, where as their Principal Scientist he led a team of analytics researchers to build cutting-edge products with the rich data sets LinkedIn has amassed. His team’s focus was on driving growth and user engagement for the LinkedIn social network. His team developed a successful model to build, ship, and iterate – to deliver value to LinkedIn effectively and sustainably. Across 3 years, he and his team delivered several industry-first features that surprised and delighted LinkedIn’s users – “People You May Know,” “Who Viewed My Profile?” “Jobs that are Similar to Mine,” and several others.

One of the first product solutions from the Aster Analytics Center is a suite of advanced analytics modules built on SQL and MapReduce called ‘Aster Data Analytic Foundation.’ The suite makes it easy for data analysts to leverage large volumes of diverse data effectively. This package, which made its debut with our nCluster 4.5 release, announced in February, provides a suite of rich analytic functions that enable data scientists and users to manipulate data easily rather than building primitives from scratch.

The second aspect of the Analytics Center’s charter, the methodology of using data, is being addressed by their work to create analytics best practices that provide blueprints for data analysts to develop their insights into an operational data product that can be delivered repeatably.

From what we see already with customers, the Aster Analytics Center – the Aster Analytics Foundation solution, big data analytics best practices, and deep analytics expertise – will be a catalyst accelerating a chain reaction that will revolutionize data usage across industries.

We are very excited to announce a strategic partnership between Aster Data and SAS Institute to further accelerate the “SAS In-Database Processing” initiative.

The objective of the partnership is to integrate SAS software capabilities within our MPP database which Aster Data’s 4.0 release uniquely supports. Last week we announced the capability to fully push down analytics application logic inside our MPP database so applications can now inside the database allowing analytics to be performed on massive data scales with very fast response.  We call this a Massively Parallel Data-Application Server. We had earlier presented more details on this unique implementation of SAS software inside Aster Data’s nCluster software at a co-hosted session with SAS at M2009.

Our architecture enables SAS software procs to run natively inside the database thereby preserving the statistical integrity of SAS software computations while giving unprecedented performance increases during analysis of large data sets. SAS Institute partners in this initiative with other databases too – but the difference is that each of these databases require the re-implementation of SAS software procs as proprietary UDFs or Stored Procedures.

We also allow dynamic workload management capabilities to enable graceful resource sharing between SAS software computations, SQL queries, loads, backups and scale-outs – all of which may be going on concurrently. The workload management enables administrators to dial-up or dial-down resources to the data mining operations based on the criticality of the mining and other tasks being performed.

Our fast loading and trickle feed capabilities ensure that SAS software procs have access to fresh data for modeling and scoring, ensuring a timely and accurate analysis. This avoids the need to export snapshots (or samples) of data to an external SAS server for analysis, saving analysts valuable time in their iterations and discovery cycles.

We’ve been working with SAS Institute for a while now, and it is very evident why SAS has been the market leader in analytic applications for three decades. The technology team is very sharp, driven to innovate and execute. And as a result we’ve achieved a lot working together in a short time.

We look forward to working with SAS Institute to dramatically advance analytics for big data!

The traditional 20-year old data pipeline of Operational Data Stores (to pool data), Data Warehouses (to store data), Data Marts (to farm out data), Application Servers (to process data) and UI (to present data) are under severe strain – because we are expecting a lot of data to move from one tier to the other. Application Servers pull data from Databases for computations and push the results of the computation to the UI servers. But data is like a boulder – the larger the data, the more the inertia, and therefore the larger the time and effort needed to move it from one system to another.

The resulting performance problems of moving ‘big data’ are so severe that application writers unconsciously compromise the quality of their analysis by avoiding “big data computations” – they first reduce the “big data” to “small data” (via SQL-based aggregations/windowing/sampling) and then perform computations on “small data” or data samples.

The problem of ‘big data’ analysis will continue to grow severe in the next 10 years as data volumes grow and applications demand more data granularity to model behavior and identify patterns so as to better understand and service their customers. To do this, you have to analyze all your available data. For the last 5 years, companies have routinely upgraded their data infrastructure every 12-18 months as data sizes double and the traditional data pipeline buckles under the weight of larger data movement – and they will be forced to continue doing this in the next 10 years if nothing fundamental changes.

Clearly, we need a new, sustainable solution to address this state of affairs.

The ‘aha!’ for big data management is to realize that traditional data pipeline suffers from an architecture problem – of moving data to applications – that must change to allow applications to move to the data.

I am very pleased to announce a new version of Aster Data nCluster that addresses this challenge head-on.

Moving applications to the data requires a fundamental change in the traditional database architecture where applications are co-located inside the database engine so that they can iteratively read, write and update all data. The new infrastructure acts as a ‘Data-Application Server’ managing both data and applications as first-class citizens. Like a traditional database, it provides a very strong data management layer. Like a traditional application server, it provides a very strong application processing framework. It co-locates applications with data, thus eliminating data movement from the Database to the Application server. At the same time, it keeps the two layers separate to ensure the right fault-tolerance and resource-management models – bad data will not crash the application, and vice-versa a bad application will not crash the database.

Our architecture and implementation ensures that apps should not have to be re-written to make this transition. The application is pushed down into the Aster 4.0 system and transparently parallelized across the servers that store the relevant data. As a result, Aster Data nCluster 4.0 simultaneously also delivers 10x-100x boost in performance and scalability.

Those using Aster Data’s solution, including comScore, Full Tilt Poker, Telefonica I+D, Enquisite – are testament to the benefits of this fundamental change. In each case, it was the embedding of the application with the data that enables them to scale seamlessly and perform ultra-fast analysis.

The new release brings to fruition a major product roadmap milestone that we’ve been working on for the last 4 years. There is a lot more innovation coming – and this milestone is significant enough that we issue a clarion call to all persons working on “big data applications” – we need to move applications to the data because the other way round is unsustainable in this new era.

We also announced today that our first customer in Europe is the #1 online poker gaming site in the world, Full Tilt Poker. We have been working with Full Tilt Poker for 8 months now helping deploy Aster nCluster to power their fraud prevention systems and provide enhanced customer service to their players.

It is no surprise that data size growth is a world-wide phenomenon, and certainly occurs across “the pond” as well. We have noticed that European customers in numerous industries, such as financial services and insurance, online retailing, social networking, communications, and gaming are deploying new (and sometimes custom) applications to leverage big data.

Aster Data is certainly the most application friendly big-data infrastructure in the market, and we look forward to working with our European customers in the coming years!

In assembling a new hardware stack, Netezza calls this re-architecture as a change but not really a change – the FPGA will continue to offload data compression/decompression, selection and projection from the Intel CPU; the Intel CPU will be used to push-down joins and group bys; the RAM will be used to enable caching (thus helping improve mixed workload performance).

I think this is a pretty significant change for Netezza.

Clearly, Netezza would not have invested in this change – assemble & ship a new hardware stack to share revenue with IBM vs. a 3rd party hardware assembler – if Netezza’s old FPGA-dominant hardware was not being out-priced and out-performed by our Intel-based commodity hardware.

It was a matter of time before the market realized that FPGA’s had reached their end-of-life status in the data warehousing market. In realizing the writing on the wall, and responding to it early, Netezza has made a bold decision to change – and yet, clung to the warm familiarity of an FPGA as a “side car”.

Netezza, and the rest of the market, will soon become aware that a change in hardware stack is not a free lunch. The richness of CPU and RAM resources in an IBM commodity blade come at a cost that a resource-starved FPGA-based architecture never had to account for.

In 2009, after having engineered its software for an FPGA over the last 9 years, Netezza will need to come to terms with commodity hardware in production systems and demonstrate that they can:

- Manage processes and memory spawned by a single query across 100s of blade servers

- Maintain consistent caches across 100s of blade servers – after all, it is Oracle’s Cache Fusion technology that is the bane of scaling Oracle RAC beyond 8 blade servers

- Tolerate the higher frequency of failures that a commodity Linux + RAID Controller/driver + Network driver stack incur when put under rigorous data movement (e.g., allocation/de-allocation of memory contributing to memory leaks)

- Add a new IBM blade and ensure incremental scaling of their appliance

- Upgrade the software stack in place – unlike an FPGA-based hardware stack that customers are OK to floor-sweep in their upgrade

- Contain run-away queries from allocating the abundant CPU and RAM resources and starving other concurrent queries in the workload

- Reduce network traffic for a blade with 2 NICs that is managing 8 disks vs. a Power-PC/FPGA that had 1 NIC for 1 disk

- …

If you take a quick pulse of the market, apart from our known installations of 100+ servers, there is no other vendor – mature or new-age – who has demonstrated that 100’s of commodity servers can be made to work together to run a single database.

And I believe that there is a fundamental reason for this lack of proof-point even a decade after Linux has matured and commodity servers have been used for computing – software not built from the ground-up to leverage the richness and contain the limitations of commodity hardware is incapable of scaling. Aster nCluster has been built ground up to have these capabilities on a commodity stack. Netezza’s software written for proprietary hardware cannot be retrofitted to work on commodity hardware (else, Netezza would have completely taken the FPGAs out, now that they have powerful CPUs!). Netezza has its work cut-out – they have taken a dramatic shift that has the ability to bring the company and its production customers to its knees. And there-in lies Netezza’s challenge – they must succeed while supporting their current customers on an FPGA-based platform while moving resources to build out a commodity-based platform.

And we have not even touched upon the extension of SQL with MapReduce to power big data manipulation using arbitrary user-written procedures.

If a system is not fundamentally designed to leverage commodity servers, it’s only going to be a band-aid on seams that are bursting. Overall, we will curiously watch how long it takes Netezza to eliminate their FPGAs completely and move to a real commodity stack so that the customers can have the freedom to choose their own hardware and not be locked down to Netezza-supplied custom hardware.

The appliance is powered by Dell hardware and Aster’s nCluster SQL/ MR database, with optional software for BI platform from Microstrategy and data modeling software from Aqua Data Studio.

Our product portfolio now allows our customers to get the benefits of our flagship Aster nCluster SQL/MR database in the packaging that they are most comfortable with – on-premise software, in-cloud service, or pre-packaged appliance.

The appliance offering packs a lot of punch compared to other data warehousing appliances in the market – it has the highest ratio of compute & memory to data sizes, allowing you to run rich queries on the appliance without breaking a sweat.

We are especially proud of the open nature of our appliance – the hardware is from Dell built from industry-standard components, the BI server is from Microstrategy, and the data modeling tool is from AquaFold (Aqua Data Studio). The appliance brings together industry-leading components of a full data warehouse stack together – all pre-tested and configured for optimal performance.

Even the programming of our appliance is open – our SQL/MR framework allows applications to push computation into the appliance using industry standard SQL augmented with MapReduce in the language of your choice (Java, C#, Perl, Python, etc.).

We have been approached by a number of customers seeking a get-started-quickly system, especially those groups of users and departments seeking a Hadoop framework to build their solutions upon.

In response to the requests, we are proud to announce an Express Edition of the appliance that is designed to work for upto 1TB of user data. And it comes in an even more attractive price – that of $50K only – complete with hardware and software!

Give us a call – we’ll get your warehouse setup on our appliance to ensure that the time-to-first-query is measured in hours, not months!

Tasso’s hard work, dedication, confidence and vision have been a huge part of our success to date, and we know they will be a big part of great things ahead for Aster. Congratulations to you, and to all the other great companies that made the list as well; it’s an honor for them to be recognized alongside you.