Harnessing and understanding data is paramount in the ever-evolving technological landscape (sound familiar?). Centralized data monoliths have been the mainstay, but as organizations scale and diversify, the centralized approach to data becomes a bottleneck (equally not new). Also not new is the concept that IT is a metaphor for a roadblock in the business user’s mind.
Enter the Data Mesh data platform architecture. This architecture helps business users create and manage data products without depending on IT resources (at least, that’s the theory). This novel concept sounds like another ebb and flow of centralized to decentralized and back again. Yet, if we reframe this situation slightly, a new idea emerges. Is data mesh a data platform architecture or an approach that enables IT to help shape the business’s organizational design?
What is a Data Mesh architecture?
Data platform architectures seem like fashion trends: ever-changing and attempting to satisfy different purposes. And on the face of it, choosing data platform architectures is often touted as a technology decision. And that tends to generally hold until you start venturing into federated, decentralized data platform architectures like the Data Mesh architecture. As you begin to federate, you move beyond traditional data engineering teams into the business, and organizational design becomes a front-and-center consideration.
In short, think of the data mesh as a city. City planners and engineers build and maintain the infrastructure, and its citizens use it at will, building houses, office buildings, retail outlets, restaurants, etc. In the case of the data mesh, IT builds and maintains the data platform but enables the business to leverage it without depending on IT resources to create valuable data products.
This article isn’t meant to explore the details of Data Mesh Architecture, as that’s done quite well by Zhamak Dehghani in her post Data Mesh Principles and Logical Architecture. However, let’s briefly define Data Mesh architecture for those less familiar.
Dehghani describes a Data Mesh architecture as “an intentionally designed distributed data architecture, under centralized governance and standardization for interoperability, enabled by a shared and harmonized self-serve data infrastructure.” I like to explain it more simply by saying centralized infrastructure and governance with decentralized data products. To put it differently, the data platform and its governance are centrally managed, while data product creation and ownership are decentralized.
Federated and decentralized
The Data Mesh architecture relies heavily on data product constructs. Ask Perplexity to “define data product,” and you’ll get:
A data product is a structured and reusable asset that combines data, metadata, and processing capabilities to deliver valuable insights or functionalities for specific business purposes.
For me, “asset” and “value” are the standout words when considering Data Mesh constructs. A data product is an asset that provides business value. Therefore, this asset should belong to the business function(s) that extract value from it (vs. IT). Expanding this further, you begin to understand the federated and decentralized nature of the Data Mesh architecture and why data products are essential. We need a centrally managed but broadly available platform (federated) on which a business unit’s data products can live (decentralized).
If I, as a business unit, were to ask the data platform team to own the data products, they would need to understand my business unit’s domain as well as the data platform architecture and technology. If I decentralize the data platform, the business units must understand and own its architecture, technology, and business domain as they will own and operate their data platform. In either scenario, owning everything creates unnecessary burdens and unrealistic expectations. Oh, and bottlenecks. Lots of bottlenecks – capacity constraints galore.
Organizational impacts of Data Mesh architectures
Understanding the Federated and Decentralized nature of Data Mesh Architecture helps us understand why it is an organizational design decision. In short, Data Mesh distributes parts of IT/Data engineering-managed ecosystem to other business units while keeping the core platform pieces within its remit.
So, what is “distributed,” and why did I use quotes around “distributed?” Let’s begin with some of the critical roles you’ll find in a Data Mesh implementation. Let’s lead with “tomato” “tomato,” in that the role name isn’t as crucial as its accountabilities. That said, this article would more than double in length if I were to break down all possible roles and their accountabilities, so for today’s purpose, I will list some of the more relevant ones. Contact me if you want to engage in a lengthier discussion on broader data platform roles and accountabilities.
- Data Product Owner
- Data Product Manager
- Data Architect
- Data Product Developer
- Data Scientist
- Machine Learning Engineer
Those familiar with data platforms will immediately notice I excluded data platform and infrastructure roles. The centralized nature of data platform management within Data Mesh architectures means these roles stay in the IT/Data Engineering team, so I’m leaving those alone for now.
The listed roles, however, might have historically lived purely in the IT/Data Engineering team(s). With Data Mesh, these roles become “distributed” into the business functions that own the data products. There are those quotes again. Data Mesh presumes those who know their business can best design, build, and manage their data products and should be free to do so.
However, despite our best efforts as technologists, we have not yet constructed a way for non-technical users to build an exhaustive and robust set of data products. Amazon Q with QuickSight is a monumental step forward in enabling analytics dashboarding using natural language and Generative AI. Still, Q does not yet capture the full range of business users’ data analytics and AI needs. Nor do existing no-code/low-code platforms (though these are continuously getting better and better). So, where does this leave us? Back to our city planning metaphor, the city is built by IT, and its business citizens become citizen developers imagining and creating the data products they need without suffering the delays of heavily contended IT team members (supply and demand ring a bell?). If only the story ended here. It starts to unravel with one question:
What happens when I exceed the skills of a citizen data product developer?
Today’s data product needs span the gamut from dashboards to building (not using) machine learning models. Somewhere along the way, the business will find that its decentralized autonomy hits a skillset wall, leaving the business to source critical roles (see above) in data product design and development. This is where IT dons a new hat and leans into advising the business on organizational design.
IT’s role in advising on Data Mesh organizational design
IT has been the gatekeeper for data and data platforms for decades. In a Data Mesh architecture, its role shifts from direct control to advisor and enabler, helping the organization adopt decentralized, federated data practices. IT’s extensive experience in building data platforms and products positions it uniquely to support the organizational changes necessary for a successful data mesh implementation.
Remembering the organizational impacts, we revisit that most business units lack the experience to fully understand the intricacies of data architecture and product development, and this is where IT steps in. IT’s role evolves to advise business units on designing their internal teams effectively, ensuring they can leverage the data platform independently while adhering to centralized governance and interoperability standards.
Advising on team design and role definition
One of IT’s critical advisory roles in a data mesh implementation is helping business units design their teams for data product ownership. This involves determining which roles are necessary, such as Data Product Owners, Data Engineers, and Data Scientists, and advising on whether these roles should be fully integrated into the business unit or shared across multiple units. IT can guide which skill sets are essential for these roles and provide recommendations on the demand thresholds that justify full-time data product roles.
IT’s experience with various data projects allows it to recognize patterns in team composition and role distribution, making it an invaluable advisor in establishing a federated yet cohesive approach. By advising on best practices for role hydration (assigning and developing roles as demand grows), IT ensures that each business unit is positioned to manage its data products successfully without unnecessary dependencies.
Facilitating shared resources and collaboration
Data mesh architectures introduce decentralized ownership, but this does not mean business units should operate in isolation. IT is crucial in fostering collaboration and encouraging shared resource models where it makes sense. For example, suppose the demand for a data product role is insufficient to justify a full-time position within a single business unit. In that case, IT can help facilitate a shared resource arrangement between units. This ensures efficient use of resources while maintaining the quality and responsiveness needed for data product development.
IT’s centralized oversight enables it to monitor platform usage and identify opportunities for cross-functional synergies. With observability and telemetry built into the data mesh infrastructure, IT can recognize when similar needs arise across business units and proactively recommend shared solutions or coordinated efforts, optimizing productivity and reducing redundancy.
Establishing outsourced IT teams for business units
Not all business units can fully support their data product development needs due to resource constraints, budget limitations, or skill shortages. In such cases, IT can establish dedicated data product teams that operate within the business unit but remain managed by IT. These teams provide the business with the capabilities to develop data products while benefiting from IT’s experience and best practices in data management, governance, and platform integration.
By providing a managed outsourcing model, IT ensures that even the least-equipped business units can participate in the data mesh without compromising quality or governance. This approach maintains a separation of concerns, allowing IT to focus on platform reliability and governance while enabling business units to innovate and create value from their data.
With Shared Resources and Outsourced teams, you can see why I put “distributed” in quotes above. It’s not quite as simple as wishing team members into existence within business units – nor is it always practical.
Conclusion
Implementing a Data Mesh architecture represents a significant organizational design decision, not just a technological one. IT is uniquely positioned to help guide this transformation by advising business units on team structures, facilitating shared resource models, and providing managed support where needed. By embracing this advisory role, IT can ensure that the organization fully realizes the potential of a Data Mesh—empowering business units while maintaining the necessary structure and governance to drive effective, scalable data product development.
Data Mesh is still an evolving concept, and its implementation can vary based on organizational needs and technology selection. Please contact me if you’d like to discuss how your organization can successfully leverage a data mesh architecture across technology and organizational design.
