As I gaze outside my quiet home office window watching an alligator in the pond, I am reminded of my mostly isolated virtual reality. Not even a rare splurge on chocolate chip cookie dough cheered me up. Thank goodness for IT/Dev Connections next week! Even introverts need to socialize a little bit to stay sane. Speaking of sanity, I continue to hear insane expectations that self-service BI tools can replace the need for a data warehouse. Three groups from different areas of the world discussed that topic with me recently. Who is selling that nonsense? Did you buy a self-service BI fantasy?
“I like nonsense, it wakes up brain cells. Fantasy is a necessary ingredient in living.”
– Dr. Seuss
Accurate Reporting Data Models have Not Changed
While contemporary self-service BI tools are totally amazing and revolutionizing business intelligence everywhere with rapid, simple, visual reporting for everyone, data model design patterns for reporting accurate values over time, across multiple data sources in an organization have not changed. I am not alone in delivering this sobering but sorely needed message to help you succeed. Top industry thought leaders also cite the same warnings and tips. There are a few packaged applications, enterprise BI solutions, ETL tools and offerings like WhereScape that can partially automate the development of a data warehouse. However, I have not seen a single self-service BI or data visualization tool yet that has data warehouse-killer capabilities despite their magical sales claims. If anything, I have seen those delightful, empowering, data visualization solutions make the case even stronger for investing in a real data warehouse or modernized reporting data management strategy.
Self-service BI tools in the market today are ideal for personal and team level reporting, quick prototypes and fire-drill, on the spot insights. The appeal of rapid, agile analytics, freeing yourself from long IT queues is absolutely irresistible. Fast insight for decision-making and immediate results are why self-service BI has saturated the market since the economy crashed in October 2008. These tools can and truly do make a significant, positive impact in an organization…when used properly with a little IT guidance and a data warehouse. When misused or improperly rolled out, self-service BI initiatives can quickly create an expensive data mess to fix that is not immediately apparent to non-technical business users.
Typically what I see with today’s awesome self-service BI tools (Microsoft Power BI, Tableau, Qlik, Spotfire, SAP Lumira or others) is a business unit secretly by-passing established reporting channels and over-confidently growing a shadow-IT initiative that eventually starts to falter. After a speedy proof of concept project that successfully delivers compelling, gorgeous dashboards or new insights in one area, the business goes “all in” and expands their self-reliant reporting joys broadly. As more data sources get added over time, data volumes increase, data quality decreases, data redundancy occurs and self-service maintenance pains begin to surface. Then reporting inaccuracy issues start to creep in and undermine credibility of your self-service BI project.
Why? Few business users understand or appreciate data modeling and the foundation of Kimball dimensional reporting design concepts. Most self-service BI tools mask that complexity and auto-magically build a reporting data model that can only provide an AS-IS (Slowly Changing Dimension Type 1) view of your data over time. Rarely will the non-technical business user connect, ingest or extract, transform and store data with Slowly Changing Dimension Type 2 support for AS-IS and AS-WAS reporting in a manner that is efficient, highly scalable, secure or even accurate as the data or organizational structures in source applications evolve over time.
A data warehouse uses proven, tried and true, reporting data model design patterns that are optimized for slice, dice, and drill-down by any of the self-service BI tools. In addition to doing a much better job of combining data from numerous data sources across an organization, concepts like slowly changing dimensions are implemented to ensure accurate values are saved as source systems get updated or entities change. For example, company business structures, managers, field teams and even names change over time. To be able to correctly compare year over year metrics, AS-IS to AS-WAS, you can’t only use the current AS-IS values. You should be able to report across both the current and historical values correctly regardless of the date range being examined. In regulated industries, education, healthcare, finance, manufacturing, retail, government, etc., reporting data accurately over time is not optional. In fact, you may legally be accountable for the values in your reports when faced with audits or in lawsuits. Managing data across an organization and responsibly reporting that data is not a trivial matter even if the tools to create beautiful interactive dashboards or reports are now super simple for anyone to use.
Top 10 Tell-Tale Signs that you might need a Data Warehouse for Self-Service BI to Succeed
As soon as the business self-service BI reporting pain starts to get unbearable, sponsors recognize that they may need a little expert help and hire a consultant to avoid IT reviews of the project. The consultant is then faced with figuring out what will keep their customer happy, what needs triage and how to level set expectations if there is a self-service BI data disaster brewing. No one wants to hear that their dearly beloved self-service BI baby is ugly or that they need a data warehouse after they bought into a self-service BI fantasy tool. When you come across a data mess, keep in mind that the folks trying to build reports and dashboards already know it. How the consultant and sponsors choose to navigate and resolve that situation ultimately determines long-term organizational self-service BI success or failure. Don’t spend countless hours hiding self-service BI data problems over and over again. Consider wisely investing in a data warehouse approach one time. Quick data fixes cover up underlying data model problems that almost always get worse.
Here are my Top 10 clues that indicate a self-service BI project is in trouble and needs IT or professional BI guidance and/or a data warehouse to ensure continued success.
- If you see multiple copies of the same entities or data sets, often with values that are no longer in sync. For example, multiple customer, person, product, location or time dimensions.
- If you can only report on today’s organizational structure and values. In other words, if you are unable to compare last year over this year metrics in organizations that experience changes to business structures, managers, field teams or people.
- If you do not have a shared reporting calendar across data from different data sources.
- If rolled up detail numbers are not matching your summary numbers.
- If you start trying to join, merge or blend more than three different application data sources together in your self-service BI tool.
- If you are unable to change your reports and still get the right numbers.
- If you start to make different copies of formulas or reports to keep snapshots of the right numbers.
- If you find manually typed in values in formulas or reports that should be dynamic variable values.
- If you have no security on your reports but should have role-based or data level security on them.
- If your reports start getting too complex to author or your queries start taking way too much time to finish.
If you find yourself answering yes to these questions, call a few reporting and data warehouse consultants to provide a quick assessment sooner than later. When you have a healthy self-service BI project, you will be really happy and not feeling much if any reporting pain. In fact, you will most likely truly enjoy the fantastic advances in this area of modern business intelligence for the masses.