How to measure LLM API cost per conversation, then cut it

You cannot cut LLM API spend until you know what a single conversation costs, and the API already returns the numbers you need. Every OpenAI and Anthropic response, and every Amazon Bedrock Converse response, carries a usage object with input, output and cached token counts (per each provider’s API reference); log those counts with a conversation ID and multiply each one by its rate on the provider’s pricing page. Cached input is billed at a different rate from regular input, so a calculation that ignores the cache fields gets cost per conversation wrong.

Why is my LLM API spend rising even though usage hasn’t changed?

Request count can stay flat while tokens per request climb. Chat APIs are stateless, so every turn resends the system prompt, the tool definitions and the whole conversation so far, and later turns cost more than early ones. Agent loops make several billed calls per user message, and reasoning models bill their hidden reasoning tokens as output. Chart input tokens per call by turn number before you blame volume or price.

How do I find the cost per conversation in my logs?

Each OpenAI, Anthropic and Bedrock Converse response carries a usage object. Log it on every call; the field names differ by provider.

For OpenAI:

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "model": "gpt-5-mini",
  "usage": {
    "prompt_tokens": 1247,
    "completion_tokens": 189,
    "total_tokens": 1436,
    "prompt_tokens_details": { "cached_tokens": 1152 },
    "completion_tokens_details": { "reasoning_tokens": 0 }
  }
}

On OpenAI, prompt_tokens includes the cached tokens. Subtract cached_tokens and price that part at the cached-input rate. reasoning_tokens are already counted inside completion_tokens. The Responses API uses input_tokens, input_tokens_details.cached_tokens and output_tokens instead.

For Anthropic:

{
  "id": "msg_123",
  "type": "message",
  "model": "claude-sonnet-5-5",
  "usage": {
    "input_tokens": 123,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 1000,
    "output_tokens": 204
  }
}

On Anthropic, input_tokens counts only the uncached part of the prompt. Cache writes and cache reads are separate fields with their own rates, so price all three input fields.

For Amazon Bedrock (Converse API):

{
  "output": { "message": { "role": "assistant", "content": [{ "text": "..." }] } },
  "stopReason": "end_turn",
  "usage": {
    "inputTokens": 987,
    "outputTokens": 156,
    "totalTokens": 1143,
    "cacheReadInputTokens": 0,
    "cacheWriteInputTokens": 0
  }
}

Bedrock InvokeModel returns each model’s native response body instead. For Anthropic models on Bedrock, that body carries the Anthropic usage object shown above.

Write one log line per API call with conversation_id, turn, provider, model, task and every token field above. Use your own session or thread ID as conversation_id. The provider’s request ID changes on every call, so grouping by it gives cost per call, not per conversation.

Load current rates into a model_prices table from https://openai.com/api/pricing/, https://www.anthropic.com/pricing and https://aws.amazon.com/bedrock/pricing/, and record the month you checked each page. Then calculate cost per conversation in BigQuery or Athena:

-- llm_logs: one row per API call, token fields normalized when you write the log
--   input_tokens       = uncached input (OpenAI: prompt_tokens - cached_tokens)
--   cache_read_tokens  = OpenAI cached_tokens, Anthropic cache_read_input_tokens,
--                        Bedrock cacheReadInputTokens
--   cache_write_tokens = Anthropic cache_creation_input_tokens,
--                        Bedrock cacheWriteInputTokens (0 on OpenAI)
--   output_tokens      = completion/output tokens, reasoning tokens included
-- model_prices: USD per million tokens from each pricing page, with checked_month
SELECT
  l.conversation_id,
  COUNT(*) AS api_calls,
  SUM(l.input_tokens * p.input_per_mtok
      + l.cache_read_tokens * p.cache_read_per_mtok
      + l.cache_write_tokens * p.cache_write_per_mtok
      + l.output_tokens * p.output_per_mtok) / 1e6 AS total_cost_usd
FROM llm_logs AS l
JOIN model_prices AS p
  ON l.provider = p.provider
 AND l.model = p.model
GROUP BY l.conversation_id
ORDER BY total_cost_usd DESC
LIMIT 10;

Run the same query grouped by task, and by turn number within a conversation. That shows which step and which turn drive the cost. Save the output; it is your before number.

Check your own bill in 2 minutes: https://revelations.consulting/checker?utm_source=blog&utm_medium=organic&utm_content=blog-llm-api-cost-per-conversation

How do I cut LLM API cost without hurting quality?

Two levers, each checked against an evaluation set before rollout: cache the stable prefix of every prompt, and route each step to the smallest model that passes.

Prompt caching: serve static content from cache

You resend system prompts and tool definitions on every call, and pay for them every time. Prompt caching bills a repeated prefix at the provider’s lower cached-input rate when the start of the prompt exactly matches a recent request to the same model.

Order every request from most stable to least stable: tool definitions, then the system prompt, then fixed examples, then conversation history, then the new user message. Keep timestamps, user names and request IDs out of the stable part. One changed character early in the prompt breaks the cache match for everything after it.

OpenAI caches long prompts automatically. Anthropic caches only what you mark with a cache_control breakpoint. Bedrock caches only what you mark with a cachePoint block, on models that support caching.

Caching can raise your bill. Anthropic, and Claude models on Bedrock, charge more than the base input rate to write a cache entry, so caching a prefix that is rarely reused costs more than not caching it. Caches also expire after minutes without traffic and are scoped to one model. Reordering a prompt can change model output, so run your evaluation set after the change.

Check for cache hits in the response:

OpenAI     usage.prompt_tokens_details.cached_tokens
Anthropic  usage.cache_read_input_tokens   (writes: cache_creation_input_tokens)
Bedrock    usage.cacheReadInputTokens      (writes: cacheWriteInputTokens)

Track cache reads as a share of input tokens, per task. If reads stay at zero after the first call of a conversation, look for a changing value near the top of the prompt, a prefix below the provider’s minimum cacheable length, or a missing breakpoint. Cache writes on every call with no reads mean you pay the write premium for nothing.

Model routing: assign the right model to the right task

TaskStart withMove up a tier only if
Classification, intent detectionSmallest tier: Claude Haiku, OpenAI mini or nano models, Gemini Flash-LiteAccuracy on your evaluation set misses your threshold
Extraction to a fixed schemaSmall tier, with structured outputs or a JSON schemaField-level accuracy misses your threshold
SummarizationSmall tierReviewers flag dropped or invented facts
Multi-step reasoning, planning, hard agent stepsLargest tier: Claude Opus, OpenAI’s flagship models—

Model names and prices change often. Check each provider’s models page and pricing page the week you change routing.

Route by step, not by conversation. A support conversation can classify intent and extract order details on the small model, then send only the policy decision to the largest one. Log task on every call so each step is priced on its own.

What can go wrong: quality drops you don’t see

A cheaper model can return answers that look fine and are wrong more often. Build an evaluation set before you change anything: a few hundred real, anonymized inputs per task with expected outputs, hard cases included. Score each task with a check that fits it: exact match for classification, field-level accuracy for extraction, and a written rubric with human spot checks for summaries. BLEU and ROUGE only measure word overlap, so they miss a summary that drops the key fact.

Run the set on your current setup to get a baseline. Run it again on every change: a new model, a reordered prompt for caching, a trimmed history. Ship only if the score holds.

Roll out behind a feature flag to a small share of traffic. Compare error rates, retries and user escalations against the baseline, and keep the old route one flag flip away. Each week, review cost per conversation by task, cache reads as a share of input tokens, and evaluation score side by side.

What it usually looks like

Run the cost-per-conversation query and read the most expensive conversations first. Check them for these patterns:

  • Input tokens per call that climb every turn because the full history is resent, with no trimming or summarizing of old turns
  • Long system prompts and tool definitions on every call, with cache reads (cached_tokens, cache_read_input_tokens or cacheReadInputTokens) at zero
  • One large model handling every step, including high-volume classification and summarization
  • Agent loops and retries that make several billed calls for one user message

I cut a ~$1M/yr cloud bill by 26% at a former employer. LLM spend responds to the same discipline: measure the unit cost first, then fix the largest driver before the next one.

Start this week with the ten most expensive conversations from the query. Check what is in each prompt, whether the cache was read, and whether a smaller model passes the same steps on your evaluation set. Multiply the per-conversation difference by your daily conversation count to size the saving.

If you want a second pair of eyes on those numbers, book a free 30-minute consult: https://revelations.consulting/?utm_source=blog&utm_medium=organic&utm_content=blog-llm-api-cost-per-conversation#contact. A senior engineer goes through your cost per conversation with you. No slide deck.

Frequently asked questions

What is a cached token in OpenAI, Anthropic or Bedrock responses?

A cached token is an input token the provider served from its prompt cache because the start of the prompt exactly matched a recent request to the same model, and it is billed at a lower rate than regular input. OpenAI reports cached tokens in usage.prompt_tokens_details.cached_tokens, Anthropic in usage.cache_read_input_tokens and the Amazon Bedrock Converse API in usage.cacheReadInputTokens. Default cache lifetimes are minutes without use, with longer retention options on some providers and models, so low-traffic prompts may miss the cache.

Can I use self-hosted models to cut LLM costs?

Self-hosted open-weight models cut LLM costs only when GPU utilization stays high, because you pay for GPU instance hours whether or not requests arrive. A 70B-parameter model usually needs several GPUs, or quantization, to serve. Compare that hourly cost at your real traffic against your measured API cost per conversation, plus the engineering time to run, patch and evaluate the model.

How do I avoid overpaying for long context windows?

Context window size does not set the price of an LLM API call: providers bill per token sent and received, so a short prompt costs nothing extra on a long-context model. Long-context cost comes from long inputs, most often the full conversation history resent every turn, so trim or summarize old turns and cache the stable prefix. Some models charge a higher per-token rate above an input-size threshold; check the provider’s pricing page before you send very long prompts.

Want this kind of thing weekly? Get Cloud Bill Weekly. Want us to find it in your bill? We're taking three founding clients for a free Cost Assessment.