In this post we are going to cover the difference between a data lake, a data warehouse and a lakehouse. Interviewers love this one, because the lakehouse is newer and many candidates can only say "it's a mix of both".
A data lake stores raw files of any type on cheap storage. A data warehouse stores cleaned, structured tables built for fast SQL. A lakehouse keeps the data as files on cheap lake storage, but adds a table format with a transaction log on top, so those files behave like warehouse tables.
We will start with the problem the lakehouse solves, then look at how the transaction log actually works, what it gives you, and how to choose between the three. There is no SQL demo in this one.
Question 6 of the Real DWH Question and Answers series on TechBrothersIT.
The interview answer
A data lake stores raw files of any type cheaply, with schema-on-read. A data warehouse stores cleaned, structured tables with schema-on-write, fast SQL and strong governance. A lakehouse adds an open table format with a transaction log (Delta Lake, Apache Iceberg, Apache Hudi) on top of lake files, giving ACID transactions and warehouse-style SQL directly on cheap storage.
The problem: two systems, two copies
To understand the lakehouse, start with the problem it solves. Many companies ended up running both a lake and a warehouse. Raw data lands in the lake, then a pipeline copies the useful part again, into the warehouse.
Now the same data lives in two places:
- Two copies to pay for.
- Two pipelines to keep in sync.
- Two security models to manage.
Machine learning reads the lake, dashboards read the warehouse, and sooner or later their numbers disagree. For example, the machine learning team counts 1,204 orders while the dashboard shows 1,189.
The lakehouse idea is simple: what if the lake itself could behave like a warehouse? Then we need only one copy.
How a lakehouse works: files plus a transaction log
The data itself is still plain Parquet files in object storage. Next to them sits a folder with a transaction log. In Delta Lake it is called the delta log. Every change to the table is written as a new, numbered commit in that log.
Let's follow a small example.
| version | action | file |
|---|---|---|
| 0 | add | part-001.parquet |
| 1 | add | part-002.parquet |
| 2 | remove | part-001.parquet |
| 2 | add | part-003.parquet |
- Version 0 added the first file,
part-001.parquet. - Version 1 added a second file,
part-002.parquet, for new orders. - Version 2 is an update. The files are never edited in place, so the log records two actions: remove
part-001.parquet, and addpart-003.parquet, a rewritten copy of it with the corrected rows.
A reader opens the log first and sees that the current version, version 2, is made of part-002 and part-003. It reads only those two files. Half-written files are never part of a commit, so nobody sees them.
Apache Iceberg and Apache Hudi work on the same idea: metadata tracks which files belong to each version of the table.
What the transaction log gives you
That log is what turns a folder of files into a real table:
- ACID transactions - a write either fully succeeds or does not appear at all, and readers always see a consistent version.
- Schema enforcement - rows that don't match the table's columns are rejected, so the lake stops filling with surprises.
- UPDATE, DELETE and MERGE - on lake data, which plain files never supported well. That is what makes slowly changing dimensions practical on a lake.
- Time travel - because old versions are recorded, you can query the table as it was yesterday, or roll back a bad load.
The three-way comparison
Here is the table to walk through in the interview.
| Data lake | Data warehouse | Lakehouse | |
|---|---|---|---|
| DATA | Any format, raw files | Structured tables | Both: files managed as tables |
| SCHEMA | On read | On write | Enforced on write |
| STORAGE COST | Low (object storage) | Higher | Low (object storage) |
| TRANSACTIONS / BI | Weak | Strong | Strong (log + engine) |
| EXAMPLES | S3, ADLS, GCS | Snowflake, BigQuery, Redshift, Synapse | Databricks + Delta, Iceberg tables |
Notice that the lake and the lakehouse share the same cheap object storage. The difference is the log and the engine on top, which give the lakehouse the strong transactions and BI performance that used to need a warehouse.
Which one to choose
- Cloud warehouse - you mainly need BI and SQL on structured data, and want a fully managed service.
- Lakehouse - you have lots of semi-structured data and machine learning, and want one copy of the data on open formats.
- Both - very common in practice, and the line keeps blurring: warehouses like Snowflake and BigQuery can read Iceberg tables too.
A lakehouse does not remove data modeling. The curated, or gold, layer still holds facts and dimensions in a star schema.
Follow-up questions interviewers ask
Is the data warehouse dead?
No. Many companies run both, and warehouses such as Snowflake and BigQuery can read open table formats like Iceberg.
What makes a lakehouse table different from Parquet files?
The transaction log (Delta Lake, Iceberg, Hudi metadata) that records which files form each table version.
Does a lakehouse replace data modeling?
No. The curated / gold layer still uses facts and dimensions.
Key points
- Lake: raw files of any type, cheap, schema-on-read
- Warehouse: structured tables, schema-on-write, fast SQL, governance
- Lakehouse: lake files + a transaction log (Delta, Iceberg, Hudi)
- The log gives ACID, schema enforcement, MERGE and time travel
- A lakehouse still needs modeling: facts and dimensions
Question 7 — Row store vs column store What is the difference between a row store and a column store?
Part of the Real DWH Question and Answers series on TechBrothersIT. File names and examples on this page are invented for teaching.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.