The AWS Machine Learning Blog argues that evaluating models solely on dollars per million tokens overlooks what production workloads actually cost, since the relevant metric is outcomes rather than raw token pricing. To address this, the post presents an open-source benchmarking harness for comparing OpenAI models on Amazon Bedrock.
The harness measures cost per correct answer, agent trajectory cost, and rubric-graded deliverable quality, providing metrics tied to results instead of unit token price.
Why it matters
Token pricing is a common basis for comparing models, but it may not reflect the total cost of achieving correct or high-quality results. Metrics such as cost per correct answer and trajectory cost offer a way to assess models against workload objectives.
Who should care
Teams selecting OpenAI models on Amazon Bedrock for production use, and those building agent-based workflows who need to weigh cost against output quality.