All Posts

Data & Analytics

What ClickBench Actually Tells You About Warehouse Speed

What ClickBench Actually Tells You About Warehouse Speed

TechShorts Newsroom

Editorial Team

What ClickBench Is

ClickBench is an open benchmark maintained on GitHub by the team behind ClickHouse, the analytical database vendor (ClickHouse/ClickBench, 2026). It runs 43 analytical SQL queries against a fixed dataset: roughly 100 GB of anonymized web clickstream data, about 99.9 million rows, known in the project as the "Hits" table (ClickHouse/ClickBench, 2026). By default, every engine is tested on the same AWS instance type, a c6a.4xlarge with 500 GB of gp2 storage, so the hardware variable is held constant and only software performance differs (ClickHouse/ClickBench, 2026).

Results are published as a public leaderboard covering more than 60 database engines, from open-source projects like DuckDB and ClickHouse itself to commercial cloud warehouses like Snowflake and BigQuery (ClickHouse/ClickBench, 2026). Contributors submit their own results following a documented methodology, and the configuration files are checked into the repository so a result can, in principle, be reproduced by anyone else.

Why It Exists

Before ClickBench, most warehouse and OLAP engine comparisons came from a vendor's own benchmark, run on hardware and query patterns the vendor chose, published with methodology that was rarely reproducible by an outside party. ClickBench was built to narrow that specific problem: one dataset, one set of queries, one hardware baseline, published configuration. It has become the reference leaderboard the analytical database industry points to when discussing raw query speed, cited on vendor comparison pages and engineering blogs across the category.

What the Rankings Have Shown Recently

The leaderboard has changed hands twice in the past year. DuckDB, the embedded analytical database originally built for single-machine use, reached the top spot among open-source engines in the "hot run" category on October 9, 2025, on the strength of its v1.4 LTS release, trailing only Umbra, a closed-source research prototype not available as a commercial product (DuckDB, 2025). ClickBench separates "hot" runs, where a query repeats against data already cached in memory, from "cold" runs starting from disk, because the two measure different things.

Then on May 5, 2026, MotherDuck, the company that sells a managed cloud version of DuckDB, posted a result for its largest instance size, called Mega, that ranked first overall: it completed the full ClickBench query set in 5.9 seconds, ahead of ClickHouse Cloud at 18.7 seconds and self-hosted ClickHouse at 32.3 seconds (MotherDuck, 2026). MotherDuck disclosed pricing alongside the result: a Mega instance costs $12 per hour, compared with roughly $128 per hour for a Snowflake 3XL warehouse, which finished the same benchmark in 14.1 seconds (MotherDuck, 2026). That comparison is a vendor's own reading of its product against a competitor's public list price, not an independent audit, and it should be read that way.

What the Benchmark Does Not Measure

ClickBench's own documentation and independent reviewers point to the same three gaps. First, the Hits dataset is a single flat table; none of the 43 queries involve a join, so the benchmark says nothing about how an engine performs on the multi-table queries that make up most real analytics workloads, and joins are frequently where query planners struggle most. Second, queries run sequentially, one at a time, by a single user. It measures nothing about concurrency, meaning how an engine holds up when dozens of analysts and dashboards query it at once, which is a different engineering problem than single-query latency. Third, at 100 GB the dataset is small by production data warehouse standards, and a small dataset can flatter engines whose advantage comes from fitting more of the working set in memory, an advantage that can narrow at multi-terabyte scale.

None of this makes ClickBench useless. It is a controlled, reproducible measure of one thing: how fast an engine executes single-table analytical queries against a fixed dataset on fixed hardware. That is a real and useful signal. It is not the same thing as "which warehouse should we buy," and vendors publishing a ClickBench win are, understandably, not the ones pointing out what the number leaves out.

What to Look at When Evaluating a Warehouse or OLAP Engine

A ClickBench ranking is a reasonable first filter, not a purchase decision. Three questions get closer to an actual buying decision. Does the benchmark's query shape resemble your workload, meaning do you run mostly single-table scans or heavy multi-table joins. What happens under your expected concurrency, which requires either a vendor-provided concurrency benchmark or your own load test, since ClickBench does not cover it. And what does total cost look like at your actual data volume and query pattern, since a per-hour instance price at benchmark scale does not necessarily hold at production scale.

The Broader Context

The ClickBench churn is happening alongside continued activity in the DuckDB ecosystem more broadly. DuckCon #7, the project's community conference, ran in Amsterdam on June 24, 2026, with a program previewing planned changes for a future DuckDB 2.0 release, including a new VARIANT data type for semi-structured data and asynchronous I/O for object storage reads (DuckDB, 2026). None of that has shipped yet, so evaluating it belongs in a future post once it is released, not this one.

For now, the practical takeaway is narrower. When a vendor cites a ClickBench result, ask which run type, what hardware, what the dataset was, and what the number does not cover. The benchmark's own documentation answers all four questions. Most vendor marketing pages built around it do not.