Data Lake vs Data Warehouse: What’s the Difference?
16th September 2020
Data lakes and data warehouses are both widely used for storing big data, but they are not interchangeable terms. To understand where you need to invest, you need to understand this difference.
A data lake is a repository of raw data, both structured and unstructured, the purpose for which is not yet defined. We have written an explainer article here to go more in depth into what a data lake is. The difference for a data warehouse is that the data is filtered and has already been processed for a defined purpose.
This distinction is important because the two serve different purposes. While a data lake may work for one company, a data warehouse may be a better fit for another. Similarly, a single company may find that it needs to use both at different times.
Below we lay out the four main differences between a data lake and a data warehouse.
1. Structure: raw vs. processed
The biggest difference between data lakes and data warehouses is the varying structure of raw vs. processed data.
Data lakes store raw, unprocessed data, while data warehouses store processed and refined data. When we talk about raw data, we mean data that has not yet been processed for a specific purpose. Raw data is flexible, meaning it can be quickly analysed for any purpose, which makes it ideal for machine learning.
The risk with raw data is that, without strict data quality management and governance, data lakes can become polluted with bad quality data and become a data swamp. The key to avoiding a swamp rests with appropriate data quality and data governance measures being in place.
On the other hand. data warehouses, by storing only processed data, save on pricey storage space by not maintaining data that may never be used. The downside to this is the loss of flexibility when analysing data outside of a pre-determined scope.
2. Purpose: undetermined vs pre-determined
Raw data flows into a data lake, similarly to how water flows into a lake. Sometimes this is done with a specific future use in mind, but sometimes it is just to have on hand. We therefore define this data as undetermined – the specific purpose of the individual pieces of data is not fixed. This means that data lakes generally have less organisation and less filtration of data than their counterpart.
Processed data is raw data that has been put to a specific use. This means that all data found in a data warehouse has to be processed either manually or, more commonly, through a tool before it can be stored.
3. Use Case: data scientists vs business users
Data lakes, due to the raw nature of the data, can be difficult to navigate by those unfamiliar with unprocessed data. Therefore they usually require a data scientist and specialised tools to understand and translate it for any specific business use.
Alternatively, data preparation tools are available to create self-service access to the information stored in data lakes. This can reduce the need for a specialist in an organisation to handle data processing.
Processed data is generally seen by business users through charts, spreadsheets, and tables. This means that employees in an organisation can read it and gain valuable insights.
4. Accessibility: flexibility vs security
Accessibility and ease of use refers to the use of data repository, not the data within them. Data lake architecture has no structure and is therefore easy to access and easy to change. Plus, any changes that are made to the data can be done quickly since data lakes have very few limitations.
Data warehouses are, by design, more structured. One major benefit of data warehouse architecture is that the processing and structure of data makes the data itself easier to decipher, the limitations of structure make data warehouses difficult and costly to manipulate.
Both data lakes and data warehouses can be useful to organisations in different contexts. To find out which would benefit your organisation most, talk to our data experts today.
