Like many other industries today, life sciences face a data deluge. Astronomical amounts of healthcare data double every 73 days. Extreme levels of data proliferation overwhelm top research organizations, existing talent, and infrastructure. How can your organization efficiently analyze data in the gigantic data era? Let’s explore the enterprise data mesh pattern.
Please join me and tag.bio on March 12 to explore how you can use a decentralized data mesh to integrate multimodal life sciences data in more efficient, modernized analytics architecture. After introducing the new design pattern, we show a live demonstration of running distributed queries with it. Even if you are not in research or life sciences, the data mesh pattern might be of interest to you.
What is a data mesh?
To deliver robust analytics programs in a flexible, highly adaptable manner, organizational leaders need to look beyond data lakes and data warehouses. Rather than consolidate and move all data to a centralized data lake or data warehouse, a data mesh allows you to overlay app access across numerous data sources using a product thinking philosophy. By applying a product thinking approach to data, you can overcome many limitations of the past.
In a data mesh, your data lakes and data warehouses become nodes in a network. You can still apply data lake best practices, such as making immutable data available for explorations and analytical usage, to the source-oriented domain data products. However, apps and data products will drive your design and delivery focus. Distributed data queries will be optimized by defined apps.
With an enterprise data mesh approach, apps and data products will drive your design and delivery focus.
The following diagram is an example of how tag.bio uses an enterprise data mesh to deliver life sciences product-based data experiences. A similar approach could be used in other industries.
A data mesh “future-proofs” your analytics architecture by providing a semantic buffer over data acquisition, storage, distribution, and underlying brittle data sources. If you change data sources, your data mesh can mitigate the swap without breaking apps or requiring downtime. You can also elegantly query data across disparate data silos without data movement or other painful issues we endured in the past. A data mesh is similar to data fabric concepts and data virtualization technologies but it is designed differently.
Overcoming past architecture failures
To delve deeper into the data mesh pattern, check out How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh by Zhamak Dehghani. In her brilliant article, Zhamak Dehghani explains how previous generations of proprietary enterprise data warehouse and business intelligence platforms with large price tags left companies with equally hefty amounts of technical debt. First era failures resulted in unmaintainable ETL jobs, data copies, and reports that only a small group of specialized people could build or understand. As data volumes grew, so did the maintenance chaos.
Second generation dig data ecosystems such as data lakes only brought more difficulty. Cryptic scripts and long-running batch jobs operated by a central team of expensive big data engineers created data lake monsters. With the introduction of the data lake, we added data silos to navigate. To further exacerbate analytical pain, we add streaming for real-time data, cloud managed services, and constantly evolving machine learning frameworks into the mix.
Achieving analytical nirvana
According to a recent Accenture survey of 190 executives in the United States, many firms struggle with data management basics. Top cited challenges extend beyond technology to cultural and operational issues. Only 32% of companies reported being able to realize tangible and measurable value from data. Merely 27% said data and analytics projects produce insights and recommendations that are highly actionable.
To get actionable insights, empower domain experts with trusted data.
If you are struggling to close the data to value gap, try moving towards a product-focused data strategy powered by an enterprise data mesh. By productizing and democratizing data using an enterprise data mesh, you can maximize the value of your data sources by enabling domain experts to answer their own questions from trusted data sources. Reduce analytical architectural complexity with an elegant data mesh UX.
For more information
In conclusion, if you’d like to learn more about enterprise data mesh technology, please review the following resources.