Hype is a double-edged sword

While hype is good to popularize new technology, especially when it entails a better way of doing things, it’s bad in one way: people rush to get it implemented without fully understanding what real problem it solves and whether it truly is useful given their current state. I don’t blame anybody—the pull of shiny new things in technology is truly hard to resist. And of course, Resume-driven development is a strong motivation. It is this aspect of hype that can make everyone very suspicious of the very technology. Vendors, consultants, implementers, proponents of the technology are all suspect.

Up until a point, hype has great benefits. It helps the why behind the technology disseminate easily and get people to pay attention. Beyond that point, however, hype only serves to increase suspicion in the technology. It is like one of those mysterious movie characters. They remain enigmatic until they talk less and remain poker-faced. Once they remain too much about themselves, there’s nothing interesting about them. Examples abound in the tech industry. Blockchain, Web 3, Kubernetes, AI/ML and I’m afraid Data Mesh might join the list.

Popping fake Data Mesh bubbles

Hope is a thing with feathers and is not a strategy when it comes to implementing a Data Mesh. It is quick to fly away very far. Without a good understanding of what Data Mesh is, you might be tempted to implement something in that direction, but here are some pointers as to why you still might not end up with a Data Mesh.

Fallacy #1: I’ll buy a solution from vendor X and I’ll have Data Mesh

This is the primary fallacy buyers of “Data Mesh” solutions face today. The key thing to understand here is that Data Mesh is a socio-technical construct. Without the social or cultural aspects implemented, you have no Data Mesh. In fact, it is the cultural aspect of Data Mesh that is arguably more difficult to implement.

There are parallels here with the DevOps movement. With DevOps, there was a unified team that did both development and operations, like the name implies. This was suggested since the development team wanted to push new features all the time and the operations team did not want to introduce potential instability with pushing out new features. In a way, these teams had conflicting interests and the only plausible solution was to make them one single team. While you can get a ton of solutions from DevOps tools vendors, if you do not make the cultural change of unifying the development and operations teams, you will never have a true DevOps culture in your organization. Data Mesh has strong parallels here. If you do not manage to convince domain leaders to take up the ownership of data products, large, centralized data engineering and data ownership will continue no matter how many solutions from a myriad vendors you deploy.

Data Mesh will require some sort of top-down push for this reason, so that everyone is aligned in the same direction. After all, this is extra responsibility albeit one that is for the betterment of the whole organization.

Having buy-ins from domain leaders for them to own data products is a crucial step towards implementation of a Data Mesh and it is then that you are ready to get them enabled with the right technology. Having “Data Mesh” solutions without this first step does nothing in the direction of building a data-driven organization.

Fallacy #2: I’ll deploy a single solution and have Data Mesh

There are many aspects of the Data Mesh that require help from technology, cataloging systems, self-serve capability, common data platform, computational governance, etc. While there may be a single vendor in the future who may offer all the pieces that might help you implement a Data Mesh, there’s nothing stopping you from implementing one with solutions from different vendors. In fact, once you understand what needs to be done, you can probably leverage solutions currently in use in your organization to achieve a Data Mesh implementation.

It is very important to keep your focus on what needs to be accomplished and for this reason it is important to understand what benefits Data Mesh offers—the why behind it. You can then figure out how you might want to go about achieving it. It is safe to say that no two Data Mesh implementations will look the same. Also, textbook Data Mesh is an ideal that no one will probably achieve. Having said that, with the right culture and solutions in place, there are several benefits one can realize by simply beginning to, in earnest, go in the direction.

‍

Fallacy #3: I’ll just introduce self-serve on the existing infrastructure and I’ll have Data Mesh

If you’ve persisted this far, you probably already know why this is a fallacy. Data Mesh is not just about self-serve. You can have self-service today with a centralized data team and centralized infrastructure. Don’t get me wrong here—self-service capability in this situation will still be useful. But the most important aspect of a Data Mesh is its decentralized ownership of data products by domain teams. And like we discussed before, this is the largest chasm organizations will have to cross and the challenges that any technology implementation brings in will be pale in comparison.

But seriously, go for it

Data Mesh’s hype is a response to two things: the sheer challenge of building data-driven organizations and the oncoming deluge of data. It is easy to get lost in the loud and confusing marketing that promises Data Mesh nirvana. It’s easy to forget that no technology solution can help alleviate the pain and will that is required to go from a centralized data engineering culture to a decentralized one. When implementing a Data Mesh, it is important to completely forget about the technical aspects for a bit and  understand what problems it solves. Similarly, think deeply about whether those problems exist in an organization of your size in the first place. It is important to understand the why behind Data Mesh, put in place a cultural infrastructure, map its various benefits with technology solutions you might have today or will have to acquire anew. But—once you peel away the hype and look at Data Mesh for the benefits it brings in, the pain will be worth your organization’s new found ability to democratize data-driven decision making.

Helping you get the benefits of Data Mesh

At Zipstack, our mission is to make it easy for anyone to create data products that everyone can use. If you want to create a data-driven culture by letting domain experts build data products, it is not fair to expect them to do it without first enabling them with world class platforms and tools. We’re also bringing the power of low-code/no-code to a new generation of citizen data engineers, expanding the circle of who can participate in building high-quality data products, speeding up your organization’s journey towards decentralization of both data engineering and decision making.

‍