Photo by Anete Lusina from Pexels

We could start this article by sharing statistics that show how quickly the amount of data we generate every day is growing. We could delve into how Big Data and AI are rapidly changing so many areas of our lives. But we no longer have to explain it. Everybody intuitively knows it.

So what’s missing? The right people with the right skill set.

Nearly every company these days strives to become more data-driven. To accomplish that, many firms are hiring software engineers under the assumption that the challenge is purely on a technical level. But this assumption is overly simplistic and does not acknowledge the specialized skills required to handle data projects on a large scale.

Having worked on both data and software engineering projects, I believe that they require people with a different background. Both roles are needed, and in a wider sense, they complement each other.

In software engineering projects, we need programmers who know how to implement well-constructed abstractions and applications. In contrast, data engineering projects involve extracting and transforming vast amounts of data at scale — often in real-time — and consolidating this data into a broader data ecosystem. Those two types of projects entail considerably different processes, tooling, and skill sets.

There is no denying that code that processes your data must be thoroughly tested and well-engineered. However, software engineers often place more importance on code quality than data quality. Code is a vehicle to provide value, but it doesn’t provide value by itself. In the end, all the tooling, refactoring, modularity, continuous deployment, and full test coverage don’t help that much if the data is wrong because the code’s ultimate objective is not satisfied.

This is not to say that those aspects are irrelevant in data projects. Quite the opposite. Poorly written code can lead to many problems down the road — especially with respect to long-term maintainability. But data quality should have priority over purely technical aspects.

Software engineers pioneered unit testing and test-driven development. By leveraging automation, we can ensure that our code works as expected at all times.

Still, for data projects, unit tests are not enough. Without testing your data, it only covers a part of the equation. Imagine that your code is perfectly designed and does what it needs to do, but your data suddenly comes in corrupt, stale, or incomplete due to a failure in a source system that generates this data. While unit tests pass, data tests will fail, thereby alerting you about a data quality issue. Additionally, unit tests are evaluated only upon a new build and deployment (e.g. when you commit and push changes to your code), while your data tests are running continuously in your data pipelines.

Therefore, we need both. Unit tests protect from code and program logic failures, and data tests protect from ingesting bad data into our system.

If you want to learn more about data testing, there is a great open source Python library built for this purpose.

A deployment artifact in software engineering can be a package, an API, a website, a mobile app, or a container image. In contrast, data projects deal with different artifacts. The outcome of a data pipeline can be a table in a data warehouse, a parquet file in a data lake(house), an automated report, or a productionized data science model.

It’s hard to draw a line between the responsibilities of a data engineer vs. a software engineer. There are data engineers who do a lot of software engineering (e.g. when doing “reverse ETL” and loading processed data from the data warehouse back to application databases). Similarly, in many companies, software engineers are also responsible for doing data processing.

Even though it’s challenging to separate responsibilities, it’s beneficial to carefully consider whom should you hire or assign for each project. Software engineers can overcomplicate things at times when working on data projects. For instance, something that could be solved by a data engineer by means of a single SQL query may end up becoming a full-fledged backend API with an ORM layer if the same problem is tackled by a software engineer. Building APIs is great (as is the API-first approach), but often we may be better served by implementing a simple solution first.

We often hear the statement to use “the right tool for the job.” But we also need to consider the right person with the right skill set for the job. Following the same example, the API may be a better solution if the use case requires an audit trail of each request, different user access permissions, or would need to be connected to a frontend. But if you need to regularly move data from A to B at scale and bring it to the right format along the way, a data engineer (and SQL) will likely fit the task better.

Imagine the following scenario: You need to build a custom analytical application that should display specific visualizations and needs to be embedded in a mobile and web application. The underlying data has to be consolidated since the charts require a combination of data from various source systems, and specific business logic needs to be applied when processing this data.

What’s a possible solution? You could assign a data engineer to the task of building a data mart in your cloud data warehouse. They would be responsible for extracting the source data, transforming it into a set of metrics and dimensions, and doing it all at scale and on schedule by building processes that ensure the data stays up to date.

In contrast, the software engineer can take care of building and deploying a responsive React application that will be properly designed, secured, tested, and will present data from a data mart to the end-users. Ideally, you would also have an API layer between the data-serving layer and the application presentation layer.

In this article, we looked at use cases when a software or data engineer might be the better fit for the task at hand. We looked at the differences between software engineering and data projects and how we can reduce friction between them by using the API-based Business Intelligence platform provided by GoodData.

Thank you for reading!