The Difference Between Parquet and Delta Lake

Parquet is a columnar file format and Delta Lake is the system on top of it. What each one does, what plain Parquet misses and why Fabric uses both.

In this article 6 sections
  1. Parquet is a file format, Delta is a system
  2. Parquet stores data column by column
  3. Delta Lake turns files into tables
  4. What plain Parquet files are missing
  5. Delta in Microsoft Fabric
  6. Parquet or Delta?

Parquet and Delta Lake come up together all the time, and they are easy to mix up. Every Delta table stores its data in Parquet files, so where does one end and the other begin? The biggest difference, I think, is that Parquet is a file format and Delta is a system.

This is part of my series on Delta tables. In the first part I explained what Delta is and why Fabric uses it, and in the anatomy of a Delta table I opened up a real table on OneLake. This time I put plain Parquet next to Delta: what they have in common, how they differ and when you would use one or the other.

The Difference Between Parquet and Delta Lake Watch on YouTube

Parquet is a file format, Delta is a system

Parquet is just a way to store tabular data in a file on disk, the same way you can with CSV or JSON files.

Delta works one level higher. It takes the files on a data lake and gives you tables on top of them.

Parquet stores data column by column

Parquet is an open source file format. It is column oriented, so it compresses data column by column instead of storing it row by row like a CSV file does. If you know how the VertiPaq engine behind Power BI semantic models works, or how clustered columnstore indexes work in SQL Server, Parquet files are a bit similar.

This makes Parquet very efficient for storage, and efficient for analytical reads as well. When you retrieve data for analytics you usually don’t need every column on a row. You need the customer ID, the date and the revenue amount, and the other columns don’t matter to you. So why would you retrieve them? A columnar format makes it easy to read only the columns you need, and that speeds up retrieval enormously.

Put in human terms, Parquet is a great file format for analytics.

Delta Lake turns files into tables

Delta Lake, on the other hand, is an open source storage framework that lets you build a lakehouse architecture on top of a data lake. A data lake usually only stores files. Delta gives you tables on top of those files, with ACID transactions and the other goodies this series covers.

Delta tables store their data in Parquet files. The Delta part is the layer that brings table behaviour to your lakehouse, and the difference is easy to see in how a table is laid out on storage:

  • The Delta table itself is a folder.
  • That folder holds one or more Parquet files, which physically store the data.
  • Next to them sits a subfolder called _delta_log, with JSON files for the transactions, or commits, on the table: the writes, updates, deletes and so on.

A plain Parquet file is just that: a Parquet file. So Parquet is the backbone of a Delta table, and the difference is a lone Parquet file on one side and, on the other, one or more Parquet files combined with the Delta log, which together make up the Delta table format.

What plain Parquet files are missing

So why don’t we build lakehouses on a data lake with plain Parquet files? They don’t give you table-like behaviour, and a lot is missing: there are no transactions, no safe writes with concurrency and no history.

The storage itself is very efficient, and so is reading from the file format. That is the reason Delta chose Parquet. Plain Parquet falls short once you need to write and change the data as a table.

Say you want several processes writing into the same table at the same time. Parquet cannot do that. Delta can.

Because the Delta log tracks every transaction, Delta also builds up history, and you can query the table as it was at any point in that history. Query it as of yesterday and you get yesterday’s state of the table and of the data inside it.

Parquet is excellent at reading data. Delta is excellent at managing the change in that data.

Side by side:

ParquetDelta Lake
What it isA columnar file formatA storage framework for lakehouse tables
On storageA fileA folder with Parquet files and a _delta_log folder of JSON commits
ReadsEfficient: read only the columns you needThe data sits in Parquet, so you keep those efficient reads
TransactionsNoneACID transactions
Several writers at onceNoYes
HistoryNoneQuery the table as of any point in its history

Delta in Microsoft Fabric

Delta is an extension on top of Parquet, and Parquet is still the file format inside it. In Fabric, Delta is the default table storage for building lakehouses. Even when you use a Fabric Warehouse with T-SQL, you are still using Delta.

You see this behaviour across the workloads. In data warehousing and data engineering we store data in Delta tables, and then a lot of different workloads consume those tables: data engineering, data science, Power BI and more.

Parquet or Delta?

To recap: Parquet is a columnar file format optimised for analytics. It stores data column by column instead of row by row and lets you read only the columns your query needs, which makes reading fast and efficient.

Delta is a storage framework for lakehouse tables. It brings ACID transactions and the Delta log, which points to the different states of the table and its data over time, and that is what lets you build a lakehouse architecture on top of Parquet.

So in Fabric you don’t have to choose. Do we store our data in Parquet, for its efficient reads? Yes. In Delta, for the transactions, the history and the write concurrency? Also yes. On Microsoft Fabric we use Delta, powered by Parquet files.

Bas Land · Microsoft Data Platform MVP

Bas has spent more than 12 years in the Microsoft data stack. He builds data platforms on Microsoft Fabric, teaches as a Microsoft Certified Trainer and speaks at conferences such as Power BI Gebruikersdagen.

↑↓to move ↵to open escto close