RAG cost guide

By Seif Amara · Use cases · Updated 2026-08-21

The anatomy of a RAG bill

Retrieval-augmented generation adds a retrieval step before the model answers: a query fetches relevant chunks, those chunks are prepended to the prompt, and the model reads all of it. The bill is driven by tokens per query (system prompt, retrieved chunks, the question, and the answer) multiplied by query volume.

Why context dominates

The retrieved context is usually the largest input component, and it is billed on every single query. A question that retrieves 3,000 tokens of chunks sends those 3,000 tokens each time, even if the question itself is short. This is why input cost, not output cost, tends to dominate a RAG workload.

How to cut RAG cost

Retrieve fewer, better chunks. Tightening retrieval so only the most relevant context is sent cuts input tokens directly and often improves answer quality at the same time. Summarizing or compressing chunks, and caching any stable prefix, help further. Model the numbers on the RAG cost calculator.

Key takeaways

In RAG, retrieved context tokens are usually the biggest cost driver.
Fewer, higher-quality chunks cost far less than many mediocre ones.
Low input price and strong instruction-following matter most for RAG models.

FAQ

What drives RAG cost?

Retrieved context tokens dominate, because every query prepends the chunks you retrieve. Cutting chunk count or size reduces input tokens sharply.

Which model is most cost-effective for RAG?

Models with low input prices and strong instruction-following. Input dominates a RAG bill, so a cheap input rate matters more than a cheap output rate.

Try it now

Put this into practice with the free calculator. No sign-up, runs in your browser.

Open the token calculator