Data Mesh has been seeing a lot of hype over the last couple of years because it is a new way of managing data, aiming to solve the problem of scalability of monolithic data architecture. It was also one of the most discussed topics during this year’s Data Innovation Summit.
At this event, Daniel Tidström, Senior Partner at Data Edge, explained the challenges and benefits organisations should be aware of when deploying Data Mesh. But most importantly, he pointed out why Data Mesh should not be seen as data architecture and why that can lead to problems.
“Data Mesh addressed the real problem that many organisations have with the continued increase of the number of data sources and data volumes of the upstream side, but also more consumers of data, more applications needing data to fuel the business.”, adds Tidström.
The Four Pillars of Data Mesh
How is the Data Mesh built up? According to Tidström, four main pillars should be taken into consideration by organisations:
- Domain – driven ownership of the data – This is an important pillar so that the ecosystem creating and consuming data can scale out as the number of sources of data, number of use cases, and diversity of access models to the data increase, simply increase the autonomous nodes of the mesh. “This is one of the core pillars to re-organize data from one centralized, often monolithic platform into domains. And do that both upstream when it comes to who produces the data, who owns the data and move the responsibility to being taken care of making that data available for consumption in the organisation. But also it is important to remember that it can be done downstream. We have product analytics team, marketing analytics team, product teams themself that consume a lot of data, so also managing on the consumption side of the data and owning that more locally.”, adds Tidström.
- Data as a product – This pillar means that data users can easily discover, understand and securely use high quality data with a delightful experience; data that is distributed across many domains. “Managing data as a product adoptes data to who the consumer is, it needs to be discoverable, managed as a product and important as a product, not a by-product which is often the case.”, additionally explains Tidström.
- Safe-served data platform – Based on this pillar, the domain teams can create and consume data products autonomously using the platform abstractions, hiding the complexity of building, executing and maintaining secure and interoperable data products. “To build something sustainable and something that can scale we need to abstract and move from data team to platform team, to put easy and fit for purpose tools like provisioning infrastructure, for storage or compute, for running pipelines, for managing orchestration, for providing shared services such as data catalog etc.”, says Tidström.
- Federated governance – In order data users can get value from aggregation and correlation of independent data products, the mesh is behaving as an ecosystem following global interoperability standards: standards that are baked computationally into the platform. “How do we avoid building silos? How do we manage to integrate one dataset from one domain with a dataset from another domain? Do we have information and standards that work together? How do we manage regulatory requirements? This is important, otherwise we will end up in a big chaotic situation.”, notes Tidström.
Adopting of Data Mesh by Organisations (Or Why Data Mesh Is Not Architecture)
There are several points and steps that organisations need to consider when they will start adopting Data Mesh and establishing its pillars. It begins on the organisational and leadership level, with the requirement of having a domain-driven development and establishing a bounded context in which the organisation will know what teams will work together, what is the good constellation of teams or what constitutes the domain. Afterwards, the organisation should ensure that the teams are empowered and feel accountable and ownership of the solutions. Also, organisations need to treat data as an essential part of what they do and be aware that there is a difference between building a feature and building a data product. The process continues with putting these abstractions and automation in place, and governance, standards and regulations play an important role. Here is a list of detailed steps and points for organisations when they will start thinking about adopting Data Mesh:
- Establish bounded context
- Empower teams
- Accountability and ownership
- The cultural shift to see data as a first-class citizen
- New roles: data product owners and data engineers
- Product discovery and delivery
- Data product infrastructure code, data and metadata
- Security and access control
- Change management and SLAs
- The transition from ownership to enablement (platform team)
- Finding the right levels of abstraction and standards
- Designing a scalable architecture
- A high degree of automation
- Establish a federated governance organisation
- Establish policies and guidelines for data quality, compliance and managing regulatory frameworks
- Managing cross-domain interoperability
- Automation of rules, policies and decisions
“When I group these things, the data and technology is one part of this, but it is equally important how we look at leadership and organisation structure when it comes to building up a Data Mesh. There will be new people and new skills that we need to build up. We need different processes, and governance is a key part. What is the collective term for these kinds of pillars? To me, it is not architecture. I see it as a data operation model. There is a lot of complexity”, says Tidström.
Challenges for Organisations With Data Mesh
How can organisations know if it is the right thing for them to adopt Data Mesh? We’ve talked about this before, but this time we’ll highlight some of the challenges and opportunities for the organisations. Let’s explore first the challenges:
- Management – Adopting Data Mesh requires a strong to-down mandate, and lack of management buy-in is one of the challenges. It is important for the whole organisation to view data as an important asset, and without that approach, it will be hard if not impossible to succeed with a transformation of this magnitude. A strong business case is needed to achieve this.
- Scalability – The objective of Data Mesh is to create value from data at scale. If organisations are not operating near full efficiency with highly automated processes it is likely more effective to close these gaps first.
- Approach – There is no one correct way of implementing Data Mesh. Finding the right approach requires a careful analysis of the problem to solve and an iterative approach that delivers value incrementally. The challenge is that organisations treat Data Mesh as a technical solution to a vaguely defined problem.
- Accountability and empowerment – Without an already established culture of empowered and accountable teams it will be extremely difficult to make a Data Mesh paradigm work. The fundamental ideas behind Data Mesh are deeply rooted in domain-driven design with clear bounded context, strong ownership and empowerment. Often the challenge is that organisations are not empowered and accountable for outcomes.
- Processes and roles – Does organisations have established roles, responsibilities, processes and incentive structure for distributed data teams? Without having these teams in place it will be very hard to create effective teams. This risks leading to confusion, inefficiencies and prioritizing other more familiar work.
- Data talent – Does organisations have a critical mass of data talent? Do they have a sufficient amount of data savvy engineers and analytics to sustain a Data Mesh model?
- Engineering maturity – In order to build and operate Data Mesh the platform team needs to have a high level of maturity. Often as a challenge it is seen that the organisation’s data team has low engineering maturity. Having the right abstractions and services, highly automated process and computational scalability is crucial.
- Security and privacy – Decentralizing the ownership of these aspects of working with data can be very scary and often organisations do not have buy-in to distribute security, privacy and compliance. Without having clear buy-in and involvement of legal and compliance officers the Data Mesh will not see the light of the day.
- Data governance – Creating a federated governance structure is on the core pillars of Data Mesh. Without acknowledging the importance of this it is very likely that organisations will move from a working monolith to a distributed silo model ultimately backtracking 20 years. That’s why it is important data governance to be seen as a core activity.
Additional on the challenges and the right time for organisations to think about Data Mesh, you can find helpful content here.
Opportunities With Data Mesh for Organisations
After knowing what obstacles organisations need to be aware of when adopting Data Mesh, it is valid to address the opportunities. Here are some of them:
- Data is an asset with high potential value for most companies. If it is a strategic asset, organisations should consider a chief data officer, as they have a chief financial officer, chief HR officer and chief product officer.
- Data is inherently distributed – Almost everyone in the organisation either produces or consumes data or probably both. So, managing it like a monolith makes less sense because it will not scale.
- The amount of data and the number of sources will grow. There are more analytics teams, data scientists, and ML engineers in organisations, so the complexity will only increase. Re-thinking the monolith is a good thing, and organisations should consider the approach Data Mesh proposes.
- Increased need for innovation and experimentation is essential to avoid situations where data teams become the bottleneck that has to bring data in, integrate it and transform it to suit everyone’s purposes since the same data can be used differently in finance, marketing, and product aspects.
- Data democratization and empowerment – This follows ownership and accountability.
- Data engineers are unicorns. We need to find other ways of scaling – Finding a data engineer to scale a centralized team is not a strategy that will work. Data engineers are unicorns or the best job of the work today because of the demand.
Featured image: Claudio Schwarz on Unsplash