The State of Data Engineering in 2022

RudderStack previously wrote about both the history of the data engineering “megatrend” and a few major trends in data engineering that they saw taking shape in 2021.

In this post, they share their thoughts on the data industry in 2022, first by reviewing the predictions made in early 2021, then outlining their view of the state of data engineering with new predictions.

Revisiting 2021 predictions

It’s been a year since RudderStack made their last set of predictions, so let’s see how they did.

The first prediction was that RudderStack would see dedicated data engineering support for each team. Now, there are different structures emerging. For example, organizations might have one team of data engineers supporting all other teams, in other cases each team might have its own data engineer. More companies are realizing the need for dedicated data support across the organization, and it is expected this trend will keep accelerating in 2022.

RudderStack also said that they would see more unicorns solving data problems. They were right on this. Even in an environment where the VC market is extremely inflated, some of the biggest rounds focused on data related companies. Clickhouse raising $250M and Airbyte’s $150M are just two recent examples.

The commoditization of data platforms continues, and it will accelerate. The spearhead here is still pipelines and ELT, but they see more and more categories growing. For example, anything that has to do with data quality and observability. As the market is still young this effect will continue. It’s still early, but there are plenty of signs here.

Real time and streaming infrastructure will become standard. It’s hard to tell if real time data has become more important than batch data, but companies are investing heavily in real time infrastructure and continue to do so. Two good examples of this trend are the amazing IPO of Confluent and new products like Apache Hudi that offer real time capabilities over data lakes.

The state of data engineering in 2022

Let’s take a look at both the current state and a few predictions about the data space in 2022.

Read more

_{*Source: RudderStack}