When data is not big: what DuckDB changes about clusters

Short answer

Do you need a distributed cluster for analytics?

Often you do not. People who ran large cloud warehouses say much of what was called big data fits on one modern machine. Part of that evidence comes from companies that benefit from it. If a typical question touches data that one machine can hold or scan, a single-machine engine such as DuckDB fits in my view. A cluster still earns its place for very large tables, heavy continuous loading and many simultaneous writers. Size the tool to the data a question touches.

§ 1 The claim

Since Google published its MapReduce paper in 2004, the standard answer to "we have a lot of data" has been a distributed system. That means many machines sharing one job.[1] The counter-claim has been made most loudly since 2023. It makes three points. Most organisations never had that much data. The hardware in one machine has grown faster than their datasets. A well-built single-machine engine now covers most analytical work.

§ 2 What a single-machine engine is

DuckDB is the best-known example. Its documentation describes an analytical database that runs embedded inside the program that uses it. There is no separate server to install or maintain and no external dependencies.[2] It is built for analytical queries (OLAP). Those are long-running questions that scan and aggregate large parts of a dataset. It processes columns of values in batches to cut the processor time spent on each value. That technique is called vectorised execution.[2]

As of 2 October 2026 the current release is 1.5.6, a bug-fix release published on 28 September 2026. The project says version 2.0.0 is due in October.[3]

§ 3 The case that most data is not big

The essay

In February 2023 Jordan Tigani published an essay titled "Big Data is Dead".[1] Tigani was a founding engineer on Google BigQuery. He published the essay on the blog of MotherDuck. MotherDuck sells a data warehouse built on DuckDB.[4] So the essay comes with both inside knowledge and a commercial stake. His main points:

• Most BigQuery customers had less than a terabyte in total storage. Among heavy users the median was much less than 100 GB. He says he cannot share the underlying figures, so this rests on his account.
• Queries are smaller than storage. In his analysis of customers spending more than 1,000 US dollars a year, 90% of queries processed less than 100 MB.
• Recent data gets most of the attention, so the working set is far smaller than the archive.
• Single machines have grown. One of his definitions of big data is whatever does not fit on one machine. By that definition fewer workloads qualify each year.

The public dataset

In 2024 Amazon researchers published a study of the Redshift fleet. With it they published a dataset of query statistics from 200 serverless and 200 provisioned instances over three months.[5] A MotherDuck analysis of that dataset was published on 7 August 2024. It reports that about 3 in every 10,000 queries touched more than 10 TB. It also reports that roughly 5% of databases held that much.[6] Two cautions apply. The analysis comes from a vendor with a stake in the answer. It notes that the dataset is not intended to represent Redshift as a whole.

The research

The idea predates DuckDB. In a 2015 workshop paper, Frank McSherry, Michael Isard and Derek Murray proposed a measure they called COST. It is the hardware a distributed system needs before it outperforms a competent single-threaded program. They reported that many published systems needed hundreds of cores to get there. Some never did.[7]

In May 2025 Hannes Mühleisen wrote on the DuckDB project's own blog. He reported running complex analytical queries over a 265 GB database on a 2012 MacBook Pro with 16 GB of memory. He argued that the industry's move to distributed systems came earlier than it needed to.[8]

§ 4 The case for clusters

None of these sources says distributed systems are pointless. Several supply the counter-evidence themselves.

• Some organisations do have very large data. Tigani's own account includes customers with hundreds of petabytes.[1] The Redshift paper describes workload properties that follow very long-tailed distributions. That means a small number of users sit far beyond the typical case.[5]
• Many writers at once. DuckDB's concurrency documentation describes a model where either one process reads and writes or several processes only read. As of October 2026, writing from several processes is offered through a remote protocol that the documentation still labels beta. For production use it points to the DuckLake format with a PostgreSQL catalog.[9]
• Memory limits. DuckDB can spill to disk for work that is larger than memory. Its own tuning guide lists cases that can still fail, such as queries that combine several memory-hungry operations.[10]
• Pipelines as well as queries. The Redshift study found that write-heavy data pipelines are a prominent part of real workloads.[5]

A distributed engine is designed around those needs. Apache Spark, for example, splits a job between a driver program, a cluster manager and executor processes running on worker nodes.[11] That design carries coordination overhead, which is what the COST paper measured. It is also what lets one job use more machines than any single one could replace.

The essay was also contested in public. MotherDuck's site carries a debate, dated 20 April 2023, between Tigani and Aditya Parameswaran of Ponder.[12] I could not find a written version of Parameswaran's case, so I have not summarised his argument here.

§ 5 The two designs side by side

Each design as its own documentation describes it.

§ 6 How to choose

This section is opinion. The useful questions for a marketing or analytics team are about the workload itself. The label on the tool matters less.

1. How much data does a typical question touch? That is different from how much you store. Both Tigani's account and the Redshift analysis suggest the gap between the two is wide.[1][6]
2. How many people or processes write at the same time? If the answer is many, check the engine's concurrency model before anything else.[9]
3. Does it need to be an always-on shared service? An embedded engine is a library. Scheduling, access control and backups still have to come from somewhere.
4. What can your team run? A cluster nobody can operate is slower than a laptop somebody understands.

The choice is also less binary than it was. DuckDB can read Iceberg tables and write to them through a catalog.[13] The DuckDB team's own DuckLake format was announced in May 2025. It keeps catalog metadata in an ordinary SQL database with the data in Parquet files.[14] So a team can keep its tables in an open format, as described in Open table formats explained. It can query them from one machine today. It can bring in a distributed engine later if the workload grows into it.

Either way, the answers are only as sound as the inputs. See Data quality before AI.