LLM Cost and Deployment Fundamentals: The Model Gives You Capability, the Workflow Decides the Bill

For a web developer, wiring up an LLM looks like “calling an API,” but it’s really about managing the product of four variables: capability, context, number of calls, and output length.

LLM usage cost = model unit price × token volume × workflow amplification factor

The model gives you capability; the workflow decides the bill. Every turn of an agent loop is a billable unit — managing the harness and context is managing cost.

Running Example: Three Versions of a Code Review Assistant

V1 — Single-Turn Chat (Paste the Diff)

input ~3.2K + output ~500
Claude Sonnet 4.6: about $0.01 per call

V2 — Automatic RAG (Retrieve Relevant Files)

input ~25K + output ~800
About $0.09 per call (≈ 9× V1)

Input is the biggest cost driver in RAG; you also need to count embedding/indexing fees.

V3 — Agent Loop (Reads the Repo, Runs Tests, Edits Code)

15 turns × avg. 40K input + 3K output per turn
Total input ~600K, output ~45K
Claude Sonnet 4.6: about $2.48 per call
DeepSeek V4-Pro (80% cache hit): about $0.09–0.12 per call

You didn’t choose a pricier model name — you chose a heavier workflow. Design the thinnest workflow that still clears your quality bar.

Token Billing Essentials

Rule	Explanation
Input/output priced separately	Output is often 3–8× pricier; under agent + thinking, output can be 60–80% of the bill
The entire prompt counts as input	system prompt, history, RAG, tool results, tool schema
Prompt caching	Cached reads are much cheaper when a stable prefix is reused
Long-context tiered pricing	A single request over a threshold gets billed at a higher tier
Batch API	Non-real-time tasks usually get ~50% discount

Reading `usage` in Code

const u = response.usage;
const cost =
  (u.prompt_tokens - (u.prompt_tokens_details?.cached_tokens ?? 0)) * inputPrice +
  (u.prompt_tokens_details?.cached_tokens ?? 0) * cachedPrice +
  u.completion_tokens * outputPrice;

metrics.record('llm.cost_usd', cost, { model, feature: 'code-review' });

Tracking cost per task is more meaningful for the business than cost per token.

Five Cost Knobs

Knob	Cost impact
`max_tokens`	Set 64–256 for classification tasks, not the 4096 default
System prompt length	Billed every turn; should be stable, lean, and cacheable
History strategy	Full history → input grows linearly
RAG top-k / chunk size	The first lever going from V1 to V2
Model routing	Simple tasks → Flash/Turbo; complex → Sonnet/Pro

Hidden agent costs: tool schemas count as input; tool results get fed back in; retries double the cost; thinking mode inflates output tokens.

Pricing Snapshot (2026-06, $/1M tokens)

Tier	International representative	Input/Output
Flagship	GPT-5.5	$5 /$ 30
Workhorse	Claude Sonnet 4.6	$3 /$ 15
Routing	Gemini Flash-Lite	$0.10 /$ 0.40
Best value	DeepSeek V4-Flash	$0.14 /$ 0.28
China workhorse	Qwen3.5-Plus	~ $0.11 / ~$ 0.67
China coding	GLM-5 / Kimi K2.7	~ $0.56–0.95 / ~$ 2.5–4.0

Domestic platforms often price by context-length tiers; check each vendor’s official pricing page before shipping.

Five Optimization Levers (by impact)

Model routing — use small models for classification/extraction
Control context — lower RAG top-k, truncate history, summarize tool results
Constrain output — max_tokens, disable thinking, structured output
Prompt caching — keep the system prefix stable, put dynamic content later
Instrumentation and alerts — alert when cost_per_task exceeds a threshold

Anti-Patterns

Defaulting to max_tokens=4096 for intent classification
Stuffing the entire codebase into a prompt
Agents with no turn cap and no token budget
Sending every request to the flagship model
Never reading usage, only checking the bill at month-end
Assuming a Cursor/Copilot subscription means the API is free

Self-Hosted Deployment: When to Consider It

Driver	Signal
Data compliance	Code/user data can’t leave the network
Cost scale	Monthly API spend stably exceeds ¥30–50K and the model can be fixed
Model can be fixed	An open-source Qwen/GLM works; you don’t need the latest closed model

A hybrid architecture is most common: primary inference locally, with fallback to a cloud API for complex tasks.

Rough VRAM Estimation

GPU VRAM ≈ quantized weights + KV cache + framework overhead (0.5–2 GB)
Doubling context ≈ doubles the KV cache

Config	Minimum GPU
7–8B Q4, 8K	12–16 GB
32B Q4, 8K	24 GB (4090)
32B Q4, 32K	48 GB

OOMs are often caused by KV cache, not weights. Use vLLM/SGLang in production; Ollama for trying things out.

API vs. Self-Hosted TCO

Break-even: monthly API cost > monthly TCO (hardware + ops + power) → worth a POC

A rough monthly TCO for a single 4090 server is about ¥570; if monthly API spend is only ¥8,000 and a DeepSeek-tier API is usable, self-hosting is hard to justify — unless the data must stay on-prem.

Connecting to AI4SE

Concept	Cost implication
Agent loop	Every loop turn is a billable unit
Harness	Context management = cost management
Coding agent benchmarks	Look at token/task, not just the Index
Tool calling	Every tool result fed back in = input tokens

See Reading the Coding Agent Market and Its Benchmarks for benchmark interpretation.