What an AI API Really Costs: A Practical Budget Breakdown

A single user request can trigger several model calls, tool calls, or image processing. Long responses and retries also increase spending. When budgeting for an AI API, it helps to measure cost per successfully completed task: how much the service spent to give the user the result they needed.

What belongs in the budget

First, check what your provider charges for and which billing units it uses. Alongside API fees, account for the application that uses the API. Include the following components where they apply to your service:

• Input data: system instructions, conversation history, documents, and the current request.
• Output data: model responses, including overly long or discarded results.
• Cached input: the portion of input billed at a separate rate, if the provider supports this pricing.
• Additional operations: search, code execution, file handling, image and audio processing, and other tool calls.
• Application infrastructure: databases, vector search, storage, network traffic, monitoring, and queues.
• Repeated calls: retries after errors, timeouts, or invalid responses, and user-requested regenerations.

A basic cost model

For a text API billed by token usage, add the costs of regular input, cached input, and output. For each component, divide the token volume by the billing unit in the rate sheet, then multiply by the corresponding rate. Add the cost of tools and other billed operations.

Add retry costs only if those calls are not already included in the token and operation volumes. Then add infrastructure costs. Divide the total by the number of API calls and, separately, by the number of successfully completed tasks. Cost per task helps show whether inexpensive calls are worthwhile when their results often need another attempt or manual correction.

A hypothetical calculation

Suppose a service makes 10,000 API calls per month, excluding retries. Each call averages 1,200 input tokens and 300 output tokens. Of the input, 400 tokens form a repeated context; they are billed as cached input in half of the calls. Assume hypothetical rates per million tokens of 2 currency units for regular input, 0.5 for cached input, and 8 for output.

• Regular input: 8 million tokens that are not cached, plus 2 million tokens from cache misses. Total: 10 million tokens, costing 20 currency units.
• Cached input: 400 tokens × 5,000 calls = 2 million tokens. Cost: 1 currency unit.
• Output: 300 tokens × 10,000 calls = 3 million tokens. Cost: 24 currency units.
• Total token cost: 45 currency units.

If a tool is called once in each of 1,000 of these API calls and costs 0.01 currency unit per tool call, it adds another 10 currency units. The preliminary total is 55 currency units before retries and infrastructure, including storage.

What commonly distorts a forecast

• Average usage hides expensive calls. Check the median and upper percentiles of context and response length as well.
• Conversation history increases input volume with every message if the application resends it in full.
• A response limit is a ceiling, not an expected volume. Base forecasts on actual response lengths and their variation.
• Automatic retries can turn one user request into several billed API calls.
• Sending every task to one model ignores the differences between simple and complex requests.
• Experiments, tests, and internal requests also consume budget. Track them separately from user traffic.

How to reduce costs and check quality

Start with the largest cost component and change one factor at a time. For each change, check whether quality holds up on the tasks your service handles.

• Remove duplicates, outdated messages, and irrelevant documents from the context.
• Check whether caching is available when the beginning of the context stays unchanged across many calls.
• Set the response format and length to suit the task. When shortening answers, check that they still contain enough information.
• Route simple and complex tasks to different models after testing quality on your own set of examples.
• Cap retries and record the reason for every repeated call.
• Set daily and monthly limits by product, customer, feature, and environment.
• Compare alternatives by cost per successful task, error rate, latency, and result quality.

The minimum useful set of metrics

For each call, record the model identifier, input and output volume, cache hit status, number of tool calls, retries, final status, and estimated cost. Grouping these records by feature and customer helps compare spending and identify the share attributable to retries and unsuccessful calls.

Build a small scenario table for the forecast: traffic volume, average and high usage per call, the expected share of successful tasks, and applicable rates. Compare the forecast with actual spending, and assess application changes by their effect on cost and quality.

For an initial estimate, use our AI API cost calculator: https://achugunov.ru/tools/ai-api-cost