OLAP is not dead. Today OLAP is in high demand. One of the most practical big data analytics technologies that I have reviewed in recent years is the innovation around OLAP on Hadoop. In this article I will share what I find compelling about using OLAP with Hadoop to maximize the value of big data across the enterprise.

As an expert in traditional OLAP, I am keenly aware of the benefits it brings to the business. Fast, simple, drag-and-drop reporting on easy to understand, governed data appeals to both the business and IT. User-friendly OLAP reporting layers buffer the business user from complex native database structures, joins and sensitive calculation logic. Standard OLAP XMLA connections allow reporting from just about any popular enterprise or self-service BI tool such as SAP Business Objects, Microstrategy, IBM Cognos, Tableau, TIBCO Spotfire, or even Excel. Even if in-memory technology reduces the need for pre-aggregation with smaller data sources, OLAP user experience is still highly valued.

Traditional OLAP can’t handle Big Data

Exponential growth in data sources, varieties and types rapidly surpassed what traditional, rigid solutions were designed to handle. Reporting directly on big data sources such as Hadoop entailed long, slow batch processes and extremely complex code. In an attempt to simplify and make big data useful to the business, early adopters attempted to move big data back into traditional databases or deploy traditional OLAP on top of Hadoop data sources. Those efforts failed. Connector cartridge nuances, distinct count pains, monster dimension processing limitations and many other issues with those use case immediately arose. Thus new OLAP on Hadoop technologies were invented.

OLAP limitations

Introducing Kyvos Insights

Kyvos Insights is a comprehensive OLAP solution specifically designed for the leading Hadoop distributions including Cloudera, Hortonworks and MapR, as well as Apache Hadoop. It delivers unprecedented scalability that traditional OLAP solutions cannot handle. Without Java programming of Hadoop expertise, BI professionals can design OLAP cubes for the business to fully enjoy the value of big data with tools they already use.

Kyvos architecture

Instant Insights on Massive Data

Imagine being able to analyze big data at the speed-of-thought. Calculate formulas and build drag-and-drop reports on a billion-member OLAP cube dimension with trillions of facts instantaneously. That is what modern OLAP on Hadoop with Kyvos Insights can deliver in seconds. It is amazing.

reports on a billion-member OLAP cube dimension with trillions of facts instantaneously

As one of the market leaders in this space, Kyvos Insights, developed patent pending OLAP on Hadoop technology. Kyvos Insights can return most Hadoop big data queries in sub-seconds. Dimension sizes are no longer limited to traditional OLAP specifications. You no longer need to move big data – you can process data in place. Speaking of processing, you can process data at any scale, granularity or complexity. Without code, Kyvos Insights automatically creates Map Reduce jobs.

Kyvos works with hundreds of billions of rows, hundreds of dimensions and measures, dimensions with cardinality in the hundreds of millions, and still delivers response times in seconds. Kyvos is also designed to scale for concurrent requests with minimal degradation. Current Kyvos’ customers report are handling ~100 concurrent requests.

Another benefit Kyvos Insights brings to big data analytics on Hadoop is enterprise grade security. OLAP data sources created with Kyvos Insights support single-sign on authentication along with role-based, column and row level data security at the lowest granular level. Role based access control is supported with LDAP, Active Directory, and Kerberos integration.

Easy to Design and Deploy

A historical challenge when working with Hadoop is development difficulty. Lengthy Java code, bizarrely named script utilities, immature open-source tools and awkward user experiences are daunting for most data scientists and BI developers tasked to generate value from collected big data. Kyvos Insights addressed those pains be applying a familiar OLAP design user experience on top of Hadoop. Without any Java code, I could build an OLAP on Hadoop cube in minutes.

familiar OLAP design user experience on top of Hadoop without any Java code

When I reviewed this solution, I felt immediately comfortable with it. Kyvos Insights supports connections to HDFS (Hadoop File System) and HCatalog, a table and storage management layer for Hadoop.

ConnectingtoHadoop

Connecting to Hadoop data

After connecting to available Hadoop and non-Hadoop data sources, I was shown options to visually transform data without any coding required. Kyvos Insights provided a rich set of functions including joins, unions, filters, pivots and aggregations.

Transformation

Transforming big data visually

Kyvos Insights also intelligently suggested OLAP dimensions and measures based on my data source field types. Time intelligence comes already baked in and available in calculation formulas. With mere point-and-click ease, I could further customize and add my own calculations.

OLAP_design

Designing OLAP cubes on Hadoop

After my OLAP on Hadoop cube was designed, I could add and schedule jobs for incremental data loads. One of the key differences in Kyvos Insights approach is to use pre-aggregation and caching for optimal reporting on huge data sets.

Deliver Big Data to the Business

For the business to get value from my new OLAP cube on Hadoop, I simply share the connection information. Delightfully this technology uses the same XMLA standards traditional OLAP uses. That means anyone can use their favorite reporting tool since most of them already support OLAP data sources.

uses the same XMLA standards traditional OLAP uses

For my demonstration, Tableau was tested. In Tableau, you select Microsoft Analysis Services as the connection type and point to the Kyvos installation on-premises or in the cloud. Kyvos also provides several white papers for Tableau use cases if you want to learn more about this specific solution combination for big data analytics.

Connecting Tableau

Connecting Tableau to Kyvos Insights OLAP on Hadoop cube

Just like traditional OLAP cubes, OLAP on Hadoop with Kyvos Insights can be navigated in Tableau using drag-and-drop ease. MDX queries are issued to Kyvos via the direct connection – no data is copied making it efficient and highly performant.

The dashboard you see below is a 30 billion row cube with about 25 attributes and dimensions, 6 measures including Distinct Customer Count and the query was returned in a few seconds.

Tableau Kyvos

Fast insights from Hadoop

Note Kyvos Insights also comes with a dashboard application. Here is an example dashboard built with it.

Kyvos dashboard

Kyvos dashboard app

Who is using Kyvos

When I review innovative technologies, I like to get a sense for who is already using it and what they are doing. Impressively, Kyvos Insights has already been adopted by one of the top global financial institutions and a well-known telecom.

For More Information

If you are interested in learning more about OLAP on Hadoop and Kyvos Insights technology, please check out the following resources.