A comparison is only useful later if someone can tell what was compared and how. This template is the minimum we would want to see in a report, from any vendor or any team, ours included.
Header
- Dates and times of the runs, with timezone, and the run order.
- Product and plan tested for each provider, and the session mode used.
- Client software, version and configuration, including timeouts and retries.
- Where the client ran, because network location changes timing.
Per target
- The URL pattern and why it was chosen.
- The content check that defines success, applied identically to every provider.
- Requests attempted per provider, and the number that passed.
- Median and 95th percentile time to first byte.
- Bytes per success, including failed attempts.
- Failure mix: 429, 403, soft blocks and timeouts.
Raw data
Publish one row per request, so a reader can recompute anything in the summary and check your arithmetic.
run_id,provider,target,started_at_utc,status,passed_check,ttfb_ms,bytes,attempt