Daniel Hiltgen 79c1e93c00 bench: improve benchmarking tool (#14240) ...

New features:
- Warmup phase to eliminate cold-start outliers
- time-to-first-token measured in each epoch
- VRAM/memory tracking to identify CPU spillover
- Controlled prompt length
- Defaults to 6 epochs and 200 tokens max

Benchstat fixes:
- ns/request instead of ns/op — non-standard unit created a separate group instead of grouping with timing metrics
- Token count as the N field — benchstat interprets N as iteration count for statistical weighting, not as a token count

2026-03-15 11:47:31 -07:00

37 KiB

Raw Permalink Blame History

View Raw