CSV, JSON, Parquet, and Delta: which one should you use?

A practical guide to storage formats in Databricks, with runnable examples

Priya inherited a pipeline that worked fine for a year. Then it stopped being fine.

The job that used to finish in four minutes was taking thirty-five. Nothing about the logic had changed. No new joins, no new tables, no new business rules. The only thing that changed was time. The source folder had grown from a few thousand CSV files to a few hundred thousand.

Her first instinct was to ask for a bigger cluster. That is almost always the first instinct, and it is almost always the wrong one.

She spent an evening looking at the Spark UI instead. One number stood out. For a query that needed three columns, the job was reading 42 GB. The three columns she actually needed were maybe 2 GB of data.

The pipeline was not slow because the cluster was small. It was slow because the storage format made Spark read everything, every time.

That is the whole story of file formats in one paragraph. How you store data decides how much work every future query has to do.

https://youtu.be/xNVh3No_SDM

Four formats, one question

Most teams end up with four choices in a Databricks workspace: CSV, JSON, Parquet, and Delta. People argue about them as if one is objectively best. That is not the right frame.

A better question is this: who or what reads this data next?

If a human opens it in Excel, CSV wins. If an API produces it with nested fields, JSON is what you get. If a query engine reads it thousands of times, Parquet wins. If your pipeline needs to update, delete, or rerun safely, Delta wins.

Let us walk through each one properly, with examples you can run in Databricks Free Edition.

CSV: simple, honest, and quietly expensive

CSV stores data row by row. Every row is a line of text, every value is a string until someone decides otherwise.

order_id,customer_id,amount,order_date
1001,C100,150.00,2026-08-01
1002,C101,200.00,2026-08-01
1003,C100,75.00,2026-08-02

Read that file in PySpark and you can watch the cost appear.

orders_csv = (
    spark.read
    .option("header", True)
    .option("inferSchema", True)
    .csv("/Volumes/workspace/default/book_data/orders.csv")
)

orders_csv.printSchema()
orders_csv.count()

Two things are happening that you should notice.

First, inferSchema reads the file once just to guess the types, then reads it again to load the data. On a small file that is invisible. On a large folder it doubles your read cost. The fix is to declare the schema yourself.

from pyspark.sql.types import StructType, StructField, StringType, DoubleType, DateType

orders_schema = StructType([
    StructField("order_id", StringType(), True),
    StructField("customer_id", StringType(), True),
    StructField("amount", DoubleType(), True),
    StructField("order_date", DateType(), True),
])

orders_csv = (
    spark.read
    .option("header", True)
    .schema(orders_schema)
    .csv("/Volumes/workspace/default/book_data/orders.csv")
)

Second, CSV has no column boundaries that a reader can skip. If you want one column out of forty, the engine still walks past the other thirty-nine. That is what happened to Priya.

CSV is not bad. It is the best format in the world for handing a file to a person. It is a poor format for a query engine to read a million times.

Use CSV at the edge of your system, where humans and other tools live. Do not use it as the storage for tables you query every day.

JSON: flexible, and flexible costs something

JSON arrives when systems talk to each other. Event streams, webhooks, application logs, API responses.

{"event_id":"E1","user":{"id":"U9","country":"IN"},"items":[{"sku":"A1","qty":2}],"ts":"2026-08-01T10:04:00Z"}

Its strength is nesting. One record can hold a whole object graph without you designing a join first. Its weakness is that every single record repeats every field name, so files are large, and there is no schema to trust.

Reading it looks easy.

events = spark.read.json("/Volumes/workspace/default/book_data/ecommerce_events.json")
events.printSchema()

Then reality shows up. One day a producer sends qty as "2" instead of 2, and your numbers quietly change. This is the failure mode we cover in the schema evolution lesson, and it is worth reading before you build on raw JSON.

The practical pattern is to treat JSON as arrival, not storage. Land it, flatten what you need, write the result as a table.

from pyspark.sql import functions as F

flattened_events = events.select(
    F.col("event_id"),
    F.col("user.id").alias("user_id"),
    F.col("user.country").alias("country"),
    F.explode("items").alias("item"),
    F.col("ts").cast("timestamp").alias("event_time"),
)

typed_events = flattened_events.select(
    "event_id",
    "user_id",
    "country",
    F.col("item.sku").alias("sku"),
    F.col("item.qty").cast("int").alias("quantity"),
    "event_time",
)

typed_events.write.format("delta").mode("overwrite").saveAsTable("bronze_events")

If you keep semi-structured fields that genuinely vary, Databricks now has a better home for them than a string column. The VARIANT type stores JSON with real query performance, and we wrote about it separately in Databricks VARIANT explained.

Parquet: the moment reading gets cheap

Parquet stores data column by column instead of row by row.

Same three rows as before, stored differently:

[order_id]:    1001, 1002, 1003
[customer_id]: C100, C101, C100
[amount]:      150.00, 200.00, 75.00
[order_date]:  2026-08-01, 2026-08-01, 2026-08-02

Two things follow from that layout, and they are the reason Parquet exists.

Column pruning. Ask for two columns and the reader physically skips the rest. Nothing clever is required from you.

Compression that actually works. Values in a column are similar to each other, so they compress far better than a mixed row of text. Ten times smaller than CSV is a normal result, not a best case.

Parquet also stores min and max statistics per column per row group. So when you filter, the reader can skip entire chunks of a file without decoding them. That is predicate pushdown.

You can see all of it in one small experiment.

from pyspark.sql.functions import expr

sample = spark.range(1000000).select(
    expr("id as order_id"),
    expr("concat('C', cast(rand() * 10000 as int)) as customer_id"),
    expr("rand() * 1000 as amount"),
    expr("date_add('2026-01-01', cast(rand() * 365 as int)) as order_date"),
)

base = "/Volumes/workspace/default/book_data/format_test"

sample.write.mode("overwrite").csv(f"{base}/csv")
sample.write.mode("overwrite").json(f"{base}/json")
sample.write.mode("overwrite").parquet(f"{base}/parquet")
sample.write.format("delta").mode("overwrite").save(f"{base}/delta")

Then compare what each one costs on disk.

def folder_size_mb(path):
    files = dbutils.fs.ls(path)
    total = sum(f.size for f in files if not f.name.startswith("_"))
    return total / 1024 / 1024

for fmt in ["csv", "json", "parquet", "delta"]:
    print(f"{fmt:8} {folder_size_mb(f'{base}/{fmt}'):8.2f} MB")

You will see something close to this shape. Your exact numbers will differ, and that is fine. The ratio is the lesson.

Now time a filtered query against CSV and against Parquet.

import time

def timed_sum(path, fmt):
    start = time.time()
    (
        spark.read.format(fmt).load(path)
        .filter("customer_id = 'C5000'")
        .selectExpr("sum(amount)")
        .collect()
    )
    return time.time() - start

print(f"csv     {timed_sum(f'{base}/csv', 'csv'):.2f}s")
print(f"parquet {timed_sum(f'{base}/parquet', 'parquet'):.2f}s")

The gap is not a tuning trick. It is the layout doing the work. This is the core idea behind the file formats and optimization lesson, where we go deeper into row groups and statistics.

So why not stop at Parquet?

Because Parquet is a file format, not a table format. It has no idea what a table is. There is no transaction log, so a reader can catch a half-finished write. There is no update or delete. There is no history. If a job fails halfway through overwriting a folder, you are left with whatever made it to disk.

Delta: Parquet with a memory

Delta Lake stores Parquet files and adds a transaction log beside them. That log is the entire difference.

Because there is a log:

• A reader always sees a complete version of the table, never a partial write
• You can UPDATE, DELETE, and MERGE
• You can read the table as it was yesterday
• A failed job leaves no mess, because the version was never committed
• Schema changes are recorded instead of silently reinterpreted

That last point matters more than people expect. Formats do not just decide speed, they decide whether a rerun is safe. We wrote a whole piece about why safe reruns decide if a pipeline is production ready, and Delta is what makes those reruns possible.

Here is the practical shape of it.

-- Upsert instead of full reload
MERGE INTO silver_orders AS target
USING staged_orders AS source
  ON target.order_id = source.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

-- Look at what happened and when
DESCRIBE HISTORY silver_orders;

-- Read the table as it was at an earlier version
SELECT COUNT(*) FROM silver_orders VERSION AS OF 12;

And when a bad load gets in, you do not rebuild from source. You go back.

RESTORE TABLE silver_orders TO VERSION AS OF 11;

Delta also gives you maintenance you can reason about.

-- Compact small files
OPTIMIZE silver_orders;

-- Remove old, unreferenced files (keep a safety window)
VACUUM silver_orders;

For layout, the modern default in Databricks is liquid clustering rather than hand-picked partitions. We compared the two in Z-ORDER or liquid clustering, and the short version is that liquid clustering adapts as your query patterns change.

ALTER TABLE silver_orders
CLUSTER BY (customer_id, order_date);

The Delta Lake lesson walks through the transaction log properly, and the partitioning and performance lesson covers layout decisions with real file counts.

A side note on Iceberg

Delta is not the only table format. Apache Iceberg does the same job with a different log design, and Unity Catalog can act as an Iceberg catalog.

The choice between them is not about which is faster. It is about who else reads your tables. If engines outside Databricks need to read the same data without copying it, Iceberg is the interoperable answer. If your work lives in Databricks, Delta stays the simplest default, and every idea in this article still applies.

How to actually decide

Put the four formats in the place where each one is strongest instead of picking a winner.

Mapped onto a medallion pipeline, the answer is almost boring, and boring is good.

Raw CSV and JSON land in a landing area. Bronze reads them once and writes Delta. Silver and Gold are Delta all the way through. Exports back out to CSV or Parquet happen at the edge, for whoever needs them.

If that structure is new to you, the medallion architecture lesson is the right next stop.

Try it tonight

Reading about this is worth maybe twenty percent of the value. Running it is the rest.

1. Load orders.csv in Databricks Free Edition with an explicit schema
2. Write the same DataFrame as CSV, JSON, Parquet, and Delta
3. Compare folder sizes and note the ratio
4. Time the same filtered aggregation against CSV and Parquet
5. Append a few rows to the Delta table, then run DESCRIBE HISTORY
6. Change one row with MERGE, then read the previous version with VERSION AS OF

Step 6 is the one that changes how people think. The moment you read yesterday's table on purpose, "storage format" stops being a detail and becomes a design decision.

If you want a full pipeline to practice on instead of loose snippets, your first data engineering project walks from a messy CSV to a Gold table in one sitting, using the exact patterns above.

What to remember

Formats are not a matter of taste. They are a decision about how much work every future query has to do, and about whether a failed job leaves damage behind.

CSV is for people. JSON is for arrival. Parquet is for reading cheaply. Delta is for tables you have to trust.

Priya's fix, in the end, was one afternoon of work. She landed the CSV files once, wrote a Delta table, and clustered it on the two columns her queries filtered on. The thirty-five minute job went back under five. The cluster stayed exactly the same size.

Continue learning

• File formats and optimization
• Delta Lake fundamentals
• Partitioning and performance
• Schema evolution
• Medallion architecture
• Your first data engineering project