No fundamental changes

There’s no question there have been several improvements in the Data Warehouse area since it has been in use in the past several decades. I think the most important one is something that made Snowflake popular: the decoupling of compute and storage. The most important feature is not really a feature, but the fact that when you hear people talking about a Data Warehouse, they’re most probably referring to a Cloud Data Warehouse. One of the triumvirate: Snowflake, Big Query or Red Shift. Data Warehouses have moved to the cloud and are available on a usage-based pricing model.

While these are improvements in the way Data Warehouses work and are deployed or commissioned, there has been no change in the way they are used in the workflow of data engineering teams.

The main bottleneck in creating a decentralized, data culture is this: Data Warehouses are still centrally managed. There’s typically a team (or a person) whose job it is to “maintain” the Data Warehouse. They’re stewards of what gets in, who has access to it, how to remove cruft that might form over time, creating org-side models and marts etc.

Previously, we discussed how, if you’re looking to implement something like a Data Mesh, the cultural changes are arguably more difficult to implement compared to the technology.

Come to think of it, Data Mesh is a suggested set of new workflows more than it is a set of suggested technologies. You can use a lot of different technologies to achieve a Data Mesh implementation.

Does Data Warehouse have a place in Data Mesh?

This is an important question to answer. A road has been laid tunneling through mountains and bridging wild rivers, but who are we planning to take along to the promised land? Let’s see how we can go about this.

If your data engineering and data analytics workflow has the now common, steward secured centralized Data Warehouse-based workflow at the heart of it, you are far from realizing your Data Mesh implementation goals. But does that mean the Data Warehouse is incompatible with Data Mesh? The answer is: it depends.

‍

Let’s look at a few different aspects:

• Teams that are creating Data Products are reaching out to the Data Warehouse management team: For Data Mesh to work, there needs to be a self-serve data platform. It should not be a situation where whenever Data Products need to be created, tickets are being raised in order to get access to the Data Warehouse.
• Data Warehouse trying to be the “one database” for analytics use cases in the organization: This is what made Data Warehouses popular earlier. But, with Data Mesh, you have a structure similar to a micro-services architecture where each service has its own logical database. It has to be the same case if a Data Warehouse is used to implement a Data Mesh. The physical database can be the same, but each Data Product has to have its own logical database.
• You can even leverage some aspects of the Data Warehouse to implement some subsystems of the Data Mesh like access control.
• You can also leverage the Data Warehouse for aspects of the Data Mesh like federated computational governance since this is where you try and automate various aspects of operations and govern data for the whole organization.
• Another aspect that can be leveraged is if the Data Warehouse has a catalog feature like Snowflake’s built-in catalog. This helps in discovery and self-serving data products.

Start by leveraging what you have

When you want to implement a greenfield Data Mesh, it is not just a technical, but more often a political undertaking. Moreover, we’ve discussed how making the required cultural changes to make the Data Mesh work is probably more than half the work compared to what needs to be achieved from a technical perspective.

It is indeed a good idea to then seek and enable champions within the organization who are willing to create and maintain Data Products and by quickly leveraging existing tools in order to get there. The main idea is to organically convince the whole organization about the power of Data Products and how transformational they can be in setting up a data-driven culture. The same frugal approach that startups adopt then needs to be engaged in the creation of data products by leveraging whatever is available without first having to jump through the hoops of budget approvals for something as new as Data Mesh.

Once the various teams in the organization actually use a couple of Data Products and experience their benefits first hand, it then becomes not just easy to get budgets and resources to work on Data Mesh, it also makes it easy to convince domain owners to take up sponsorship and ownership of Data Products from their domains.

‍