Performance Benchmarking Ethics

Headline TPS is the vanity metric of Layer-1 marketing. We have seen protocols claim 100,000 TPS while their finality latency is undefined, their state size is unbounded, and their benchmark transactions are empty transfers between two pre-funded accounts. This note explains how we measure protocols so that the numbers mean something.

The dishonest benchmark playbook

Before describing the honest method, it is worth listing the tricks we see repeatedly:

• Empty transactions. No payload, no state reads, no signatures verified on the critical path.
• Single sender/receiver. Removes contention in the mempool and the account trie.
• No consensus finality. Reporting optimistic execution throughput rather than committed, finalised throughput.
• Closed validator set. Fewer validators than a production deployment; often one node per machine.
• No network faults. Zero latency, no packet loss, no partitions.
• Vertical scaling. Running the benchmark on a single 128-core machine and calling it distributed.
• Cherry-picked metrics. Reporting peak throughput while hiding p99 latency and failed transactions.

Any of these can inflate TPS by one or two orders of magnitude. Our job is to deflate it.

The honest method

Define the workload

A benchmark workload must resemble production: mixed transaction types, realistic payload sizes, signature verification, state reads and writes, and contract calls if the protocol supports them. We publish the transaction mix and the genesis state so that the benchmark can be reproduced.

Measure finality, not execution

The only number that matters for settlement is committed throughput: transactions that are finalised and cannot be reverted. We measure end-to-end latency from client submission to finality notification, and we report p50, p95, and p99 separately. A protocol with a low median and a 10-second tail is not fast for everyone.

Use realistic network conditions

Consensus performance lives on the network. We run benchmarks with:

• Inter-validator latency drawn from a real geodistribution.
• Packet loss between 0.1% and 1%.
• Occasional partitions to test recovery time.
• Byzantine validators if the protocol claims BFT tolerance.

Report resource utilisation

Throughput per dollar is more useful than throughput per node. We report CPU, memory, disk I/O, and bandwidth per validator. A protocol that hits 10,000 TPS on 64 cores is not necessarily better than one that hits 5,000 TPS on 16 cores.

State growth and replay

A benchmark is incomplete if it does not run long enough to observe state growth. We measure:

• Storage growth per day at sustained throughput.
• Time to sync a new node from genesis.
• Time to replay history for state validation.

A fast protocol that becomes unsyncable in six months is not a fast protocol.

The adversarial dimension

Performance under honest load is necessary but not sufficient. We also measure:

• Censorship resistance. Can a minority of validators suppress specific transactions?
• Fairness. Does the same transaction type receive consistent latency, or does it depend on the sender?
• Recovery time. How long does the protocol take to return to steady state after a partition or crash?
• Spam resistance. Does a burst of low-value transactions degrade service for high-value ones?

Reporting standards

When we publish benchmark results, we include:

1. The exact protocol version and commit hash.
2. The hardware and network topology.
3. The workload generator and transaction mix.
4. The definition of finality used for the metric.
5. Raw latency histograms and throughput time series.
6. All anomalies, dropped transactions, and restart events.

Closing note

Honest benchmarking is expensive. It requires more hardware, more time, and more courage than marketing benchmarks. But it is the only way to know whether a protocol will survive its first production stress test.