Every data warehouse vendor has a benchmark chart, and most of them show the vendor winning. The charts are not necessarily false. They usually answer a narrower question than the buyer is asking. Reading one well means knowing which benchmark was run, who ran it, under what rules, and what the workload leaves out.
What an analytical benchmark is
An analytical benchmark is a fixed dataset, a fixed set of queries and a fixed procedure for running them, so two systems can be compared on the same work. The two long-standing standards come from the Transaction Processing Performance Council (TPC), an industry body that publishes specifications and audits results.
TPC-H is the older one. It uses a normalized product, order and supplier schema and 22 parameterized ad hoc queries (TPC, TPC-H Specification). TPC-DS models a retailer's web, catalog and store sales across 24 tables in a snowflake schema and runs 99 queries, each executed twice during the performance test (TPC, TPC-DS Specification; Fivetran, 2022). Its headline metric, QphDS, is queries per hour for decision support, calculated from the scale factor and the number of concurrent query streams.
Scale factor matters. The TPC-DS specification says a result is only comparable to other TPC-DS results under the same major revision and at the same scale factor (TPC, TPC-DS Specification). A 1TB run and a 100TB run are different tests.
Why vendors rarely name each other
Buyers need to compare systems before committing, and a shared workload is the cheapest way to do it. For decades vendors made that harder. The "DeWitt clause", a licence term barring customers from publishing benchmarks that name the product without the vendor's approval, traces back to Oracle's response to a 1982 study by database researcher David DeWitt that showed Oracle performing poorly (Brent Ozar Unlimited, 2018). The clause spread across the industry, which is why academic papers often compare anonymous systems labelled "DBMS-X".
Databricks removed the clause from its service terms in November 2021 and added a reciprocal provision: a competitor that benchmarks Databricks loses its own DeWitt protection, so Databricks can benchmark it back by name (Databricks, 2021). The right to publish results is not universal, and it varies by product and contract.
How the main approaches differ
Audited TPC results. A vendor submits a full run to the TPC, which reviews it against the specification. In November 2021 Databricks announced an audited 100TB TPC-DS result of 32,941,245 QphDS, against a previous record of 14,861,137 QphDS held by an Alibaba system (Databricks, 2021). An audited result says the run followed the rules. It does not say the configuration resembles what a customer would buy.
Derived benchmarks. Anyone may build a benchmark from TPC materials under the TPC's fair use policy if it is labelled as derived. The TPC does not audit derived results and does not consider them comparable to official TPC-DS results (TPC, Fair Use Policy). Most TPC-DS numbers in vendor blogs fall into this category.
The Fivetran Cloud Data Warehouse Benchmark, run with consultancy Brooklyn Data Co. between May and October 2022, is a derived test done openly. It generated TPC-DS data at 1TB, ran the 99 queries once each to avoid result caching, and compared Redshift, Snowflake, BigQuery, Databricks and Synapse on cost and speed. Fivetran states it is not an official TPC-DS benchmark because it used one scale, modified queries slightly and did not tune the warehouses (Fivetran, 2022). Fivetran sells pipelines into all five platforms, so it is a vendor, though not one of the warehouses being ranked.
Open, reproducible benchmarks. ClickBench, maintained in a public repository under the ClickHouse organisation, publishes its dataset, queries and a script per system covering installation, loading and query runs. It uses a single table of 99,997,497 rows derived from anonymised web analytics traffic, and 43 queries (ClickHouse, ClickBench). Its documentation states the limits: one flat table does not represent classical warehouses built on star or snowflake schemas, those systems may be disadvantaged, and the dataset is small by modern standards. Join-heavy workloads are outside its scope.
What to check when a benchmark lands in a sales deck
- Who ran it, and whether it was audited. A vendor's own run of a competitor is the weakest evidence.
- Which benchmark and which scale factor. Numbers across TPC-H, TPC-DS and ClickBench are not comparable.
- Whether both systems were tuned equally. Clustering, compression, file layout and sizing each move results.
- How the data was loaded. Pre-built or re-optimised datasets can flatter a system.
- Cold or warm runs. Result caches make repeat queries near-instant.
- Price basis. Price-performance depends on list price, region and edition on the test date, so the date belongs next to the number.
What commonly goes wrong
The best documented failure is the two-sided dispute. When Databricks published its 2021 record alongside a comparison with Snowflake, Snowflake responded that the Snowflake figures were not transparent, audited or reproducible, and published its own TPC-DS run (Snowflake, 2021). Databricks replied that Snowflake had used a pre-built dataset rather than raw TPC-DS generator output, and that on the official dataset its fastest Snowflake run took 7,276 seconds, 1.9 times longer than Snowflake reported (Databricks, 2021). Neither set of numbers was independently audited as a head-to-head comparison. A buyer reading only one side would have been misled either way.
The deeper problem is that standard workloads may not resemble production. Researchers at Amazon analysed query statistics from the Redshift fleet and found that write-heavy pipelines are prominent, workloads vary over time, and in half of clusters 80% of queries were exact repeats of earlier ones (van Renen et al., VLDB 2024). The data comes from one vendor's fleet, but it echoes earlier academic work on a Cloud Analytics Benchmark, which found a higher share of data manipulation statements and much wider variation in table and query sizes than TPC-H assumes (van Renen and Leis, VLDB 2023).
TPC-style tests mostly measure how fast an engine answers a fixed set of large, unique analytical queries. Many production warehouses spend much of their time on ingestion, small repeated dashboard queries and concurrency. That gap is why a proof of concept on a sample of real queries, run on the buyer's own data at the buyer's own scale, remains the only benchmark that answers the buyer's actual question.
Sources
- Transaction Processing Performance Council. "TPC Benchmark DS Standard Specification, Version 3.2.0." https://www.tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.2.0.pdf
- Transaction Processing Performance Council. "TPC Benchmark H Standard Specification, Revision 2.17.1." https://www.tpc.org/tpc_documents_current_versions/pdf/tpc-h_v2.17.1.pdf
- Transaction Processing Performance Council. "Fair Use of TPC Benchmarks, Quick Reference." 2019. https://www.tpc.org/TPC_Documents_Current_Versions/pdf/Fair_Use_Quick_Reference_v1.0.0.pdf
- Brent Ozar Unlimited. "The DeWitt Clause: Why You Rarely See Database Benchmarks." 2018. https://www.brentozar.com/archive/2018/05/the-dewitt-clause-why-you-rarely-see-database-benchmarks/
- Databricks. "Eliminating the DeWitt Clause for Database Benchmarking." 2021. https://www.databricks.com/blog/2021/11/08/eliminating-the-dewitt-clause-for-database-benchmarking.html
- Databricks. "Databricks Sets Official Data Warehousing Performance Record." 2021. https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html
- Snowflake. "Industry Benchmarks and Competing with Integrity." 2021. https://www.snowflake.com/en/blog/industry-benchmarks-and-competing-with-integrity/
- Databricks. "Snowflake Claims Similar Price/Performance to Databricks, But Not So Fast!" 2021. https://www.databricks.com/blog/2021/11/15/snowflake-claims-similar-price-performance-to-databricks-but-not-so-fast.html
- Fivetran. "Cloud Data Warehouse Benchmark." 2022. https://www.fivetran.com/blog/warehouse-benchmark
- ClickHouse. "ClickBench: a Benchmark for Analytical Databases." https://github.com/ClickHouse/ClickBench
- van Renen, A. et al. "Why TPC Is Not Enough: An Analysis of the Amazon Redshift Fleet." Proceedings of the VLDB Endowment 17(11), 2024. https://www.vldb.org/pvldb/vol17/p3694-saxena.pdf
- van Renen, A. and Leis, V. "Cloud Analytics Benchmark." Proceedings of the VLDB Endowment 16(6), 2023. https://www.vldb.org/pvldb/vol16/p1413-renen.pdf