{"id":9484,"date":"2023-10-16T08:00:00","date_gmt":"2023-10-16T06:00:00","guid":{"rendered":"https:\/\/www.hibernian-recruitment.com\/?p=9484"},"modified":"2023-10-17T18:24:30","modified_gmt":"2023-10-17T16:24:30","slug":"how-to-understand-data-mesh","status":"publish","type":"post","link":"https:\/\/www.hibernian-recruitment.com\/en\/how-to-understand-data-mesh\/","title":{"rendered":"Demystifying Data Mesh: A Game-Changer for Future-Proof Data Architecture"},"content":{"rendered":"\n

Demystifying Data Mesh: A Game-Changer for Future-proof Data Architecture<\/h1>\n\n\n\n
\n

We need to shift to a paradigm that draws from modern distributed architecture: considering domains as the first class concern, applying platform thinking to create self-serve data infrastructure, and treating data as a product.<\/p>\nZhamak Deghani<\/cite><\/blockquote>\n\n\n\n

The concept of data mesh was first coined by Zhamak Deghani in 2019, when she was a director of emerging technologies at Thoughtworks. In an in-depth write-up she presented a problem underlying current data architecture: infrastructure was centralised and monolithic, alienating data producers from consumers.<\/p>\n\n\n\n

But as data was becoming more abundant, and our use cases for it more complex and variable. Previously data was mainly used for business intelligence; now we were using it to design products, optimise workflows and empower employees – existing architecture lacked the flexibility optimally serve all those consumers.<\/p>\n\n\n\n

\"\"
The four core principles of data mesh. Source<\/a><\/figcaption><\/figure>\n\n\n\n

Her solution is a paradigm shift, which involves decentralising your data architecture and reframing data ownership and processing. A decentralised system allows for local data ownership and governance, improving data quality and security. Almost more importantly, presenting data as a product and creating self-service platforms allows wider access and consumption of the data, means it becomes easier to embed it in all levels of an organisation.<\/p>\n\n\n\n

The Core Principles<\/h1>\n\n\n\n

Domain Ownership:<\/h3>\n\n\n\n

Traditionally data has been centralised in data warehouses or data lakes, with one team being responsible for its quality and distribution, as well as the maintenance of the infrastructure. This can create bottlenecks in distribution and places pressure on the data team. While they are responsible for data quality and delivery, they might lack the specific expertise about its collection and application \u2013 creating a disconnect.<\/p>\n\n\n\n

One of the core ideas within a data mesh is decentralisation in the form of domain ownership. This cuts out the middleman and means that the people responsible for data production, are also the ones who ensure its quality and prepare it for application. In short, each domain that produces data will also be tasked with transforming it into a product ready for consumption.<\/p>\n\n\n\n

\"\"
Traditional approaches to data can create a disconnect between data owners and users. Source<\/a><\/figcaption><\/figure>\n\n\n\n

Data as a Product:<\/h3>\n\n\n\n

The second principle of data mesh is to reframe data as a product. Essentially, the domain team which produces the data needs apply product thinking and prepare the data in ways which are most valuable to its consumers.  <\/p>\n\n\n\n

The data product goes beyond the mere data set; it encompasses code for data pipelines, metadata, and infrastructure too. To have any value the data product must also fulfil certain conditions: it must be discoverable, addressable, self-describing, secure, trustworthy, and interoperable.<\/p>\n\n\n\n

Self-Serve Data Platforms:<\/h3>\n\n\n\n

Another pillar of data mesh is the idea of a self-serve data infrastructure. This infrastructure should be domain agnostic and hide the complexities of data provisioning from its users, which would allow even users with less specialised knowledge to access and use the data.<\/p>\n\n\n\n

Deghani envisioned that within such platforms, there could be multiple planes which provide access to specialised capabilities, catering to different consumer\u2019s needs. Essentially, through abstraction the infrastructure removes barriers to accessing the data and creates autonomy for users.<\/p>\n\n\n\n

Federated Governance:<\/h3>\n\n\n\n

This last principle safeguards interoperability within the data mesh. One of the benefits of the data mesh is that it confers autonomy and responsibility to the domains and consumers within the mesh. However, if the domains, products, and interfaces are not governed by a standardised set of rules, they will soon cease to be interoperable. The governance challenge is to balance local autonomy with global conventions and standards.<\/p>\n\n\n\n

A New Philosophy<\/h1>\n\n\n\n

This write-up is surface-level, but for anyone looking to dive deeper into the concept, there are many excellent articles available online \u2013 including articles by Deghani herself, exploring the principles <\/a>and architecture<\/a>.<\/p>\n\n\n\n

\"\"
Potential key advantages of having a data mesh over more traditional architecture. Source<\/a><\/figcaption><\/figure>\n\n\n\n

There are compelling reasons to adopting a data mesh over a more traditionally centralised architecture. For starters, it promises improved data quality and scalability, as well as higher resilience to outages. It would also make it easier for anyone within an organisation to access and harness the data, regardless of their data engineering skills.  There is also no need for central maintenance and hierarchy, as each domain is responsible for maintenance and control of its own node.<\/p>\n\n\n\n

The most important takeaway from this, or any other article about data mesh, is that its not a singular technology. It\u2019s almost a design philosophy which affects how you build your infrastructure and implement technology.  <\/p>\n\n\n\n

Sources and Further Reading<\/h2>\n\n\n\n

Data Mesh: Should you try this at home? | ZDNET<\/a><\/p>\n\n\n\n

Data Mesh Principles and Logical Architecture<\/a><\/p>\n\n\n\n

Driving Faster Insights with a Data Mesh – RTInsights<\/a><\/p>\n\n\n\n

Four Principles of Data Mesh | Thoughtworks<\/a><\/p>\n\n\n\n

How Data Mesh helps you capture greater value from your data? | nasscom<\/a><\/p>\n\n\n\n

How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh<\/a><\/p>\n\n\n\n

It’s not just about tech, it’s about ownership and communication | Thoughtworks<\/a><\/p>\n\n\n\n

What is a data mesh? | IBM<\/a><\/p>\n\n\n\n

What Is Data Mesh, and How Does it Work? ft. Zhamak Dehghani<\/a><\/p>\n\n\n\n

Why We Started Nextdata. When I first developed the concept of\u2026 | by zhamak dehghani | Medium<\/a> <\/p>\n","protected":false},"excerpt":{"rendered":"

A layman\u2019s attempt at demystifying data mesh: what are its core principles, its implications and why do we even need it in the first place?<\/p>\n","protected":false},"author":31,"featured_media":9489,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[908,909,910],"class_list":["post-9484","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-general","tag-data-architecture","tag-data-mesh","tag-decentralised-infrastructure"],"acf":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.hibernian-recruitment.com\/en\/wp-json\/wp\/v2\/posts\/9484","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.hibernian-recruitment.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hibernian-recruitment.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hibernian-recruitment.com\/en\/wp-json\/wp\/v2\/users\/31"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hibernian-recruitment.com\/en\/wp-json\/wp\/v2\/comments?post=9484"}],"version-history":[{"count":6,"href":"https:\/\/www.hibernian-recruitment.com\/en\/wp-json\/wp\/v2\/posts\/9484\/revisions"}],"predecessor-version":[{"id":9505,"href":"https:\/\/www.hibernian-recruitment.com\/en\/wp-json\/wp\/v2\/posts\/9484\/revisions\/9505"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.hibernian-recruitment.com\/en\/wp-json\/wp\/v2\/media\/9489"}],"wp:attachment":[{"href":"https:\/\/www.hibernian-recruitment.com\/en\/wp-json\/wp\/v2\/media?parent=9484"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hibernian-recruitment.com\/en\/wp-json\/wp\/v2\/categories?post=9484"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hibernian-recruitment.com\/en\/wp-json\/wp\/v2\/tags?post=9484"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}