The Hidden Ways Your AI Product Leaks Money — a $50–70 Tuition, Written Down

You don’t get billed for models—you get billed for mistakes you never see.

If you run an AI product in India—WhatsApp support, lead‑qual flows, document Q&A—your cloud invoice is a tuition fee for design decisions you didn't know you were making. The model is almost never the problem. The orchestration around it is.

This post writes down the leaks we keep fixing for founders. None of this is theoretical. These are real issues from audits and internal builds—turned into a checklist you can act on this week.

What “tuition” looks like in rupees

Here are three leaks that cost real money before we fixed them. I’m sharing them as learning, not as a case-study pitch:

• $8.70 → $3 per video pipeline: A “book‑to‑video” generator quietly called three different LLMs per scene, transcribed audio twice, and rendered captions with no caching. Routing and caching took it from roughly ₹720 to ~₹250 per video. Same output quality, less orchestration nonsense.

• $0.08/image silent overspend: Vision prompts sent full‑resolution images base64‑inlined into requests. Resizing to model-preferred dimensions and compressing thumbnails eliminated ≈₹6–7 per image on volume workloads.

• Billed 3× for one video: A retry race between the queue and a webhook triggered three identical transcoding + LLM summary jobs. Adding idempotency keys and a single source of truth removed the triple charge with zero code changes to the models.

None of these required a new model. They required engineering guardrails.

Where the leaks hide (and how to stop them)

1) One‑size‑fits‑all model usage

Founder symptom: “Everything hits the premium LLM because it’s safest.”

Why it happens: Shipping fast means defaulting to a single endpoint. Then the edge cases (5–10%) justify the cost for the 90% easy queries.

₹ impact: You pay top-tier rates for greetings, FAQs, and repeats that a small model or cached answer could handle.

Fix this week: Define a 3‑tier router: cheap small model by default; mid‑tier when context >N tokens or confidence drops; premium only on escalations. Add a cold cache for known FAQs.

2) Prompt bloat and no max_tokens

Founder symptom: “Long responses, long bills.”

Why it happens: System prompts include policy, brand tone, full tool lists, and entire chat history. No output cap.

₹ impact: You buy tokens you didn’t need to generate.

Fix this week: Trim system prompts, summarize chat history, and set max_tokens per use case. For WhatsApp replies, 120–180 tokens is usually plenty.

3) Function/tool-call loops

Founder symptom: “It’s thinking forever.”

Why it happens: Tool-calling agents call themselves into loops (fetch → plan → fetch). Each turn is another bill.

₹ impact: 2–5× invocation multiplier on heavy days.

Fix this week: Enforce a hard cap on tool calls per request (e.g., 2), and add a “finish or fallback” rule. If not done by cap, return a partial result or escalate.

4) Duplicate work from retries

Founder symptom: “Random spikes at midnight.”

Why it happens: Queue, webhook, and client each retry with their own backoff. No idempotency key.

₹ impact: Same job billed 2–3×.

Fix this week: Add an idempotency key at the very edge and deduplicate in your worker. One job, one bill.

5) Vision payloads too large

Founder symptom: “Image queries feel pricey.”

Why it happens: Full‑size images embedded as base64, or sent to a VLM when a cheap classifier would do.

₹ impact: You pay for pixels you don’t need.

Fix this week: Resize to model‑recommended dimensions, compress, and add a pre‑classifier: “Is this an invoice?” → yes/no on a tiny model before calling a VLM.

6) Audio/video billed more than once

Founder symptom: “Per‑minute costs are off.”

Why it happens: Mis-segmentation and overlapping chunks; transcription + summary pipelines re‑read the same bytes.

₹ impact: Double or triple billing on minutes you already paid for.

Fix this week: Segment deterministically, persist transcripts, and reuse them downstream (don’t re‑transcribe to summarize).

7) Over‑embedding your knowledge base

Founder symptom: “Search is fast, bills are not.”

Why it happens: Embedding every field of every doc every time. No incremental updates or dedup.

₹ impact: You pay to store synonyms.

Fix this week: Hash chunks and embed only changed content. Use smaller‑dimension embeddings if recall is stable.

8) Context window misuse

Founder symptom: “We keep the whole chat for quality.”

Why it happens: Fear of losing nuance → you pass 20 previous turns.

₹ impact: Input tokens explode; latency rises.

Fix this week: Summarize prior turns into a short rolling memory. Keep raw history in your DB, not in every prompt.

9) No caching for deterministic answers

Founder symptom: “Volume is killing us during spikes.”

Why it happens: Same top 100 questions asked across 10,000 users. You recompute every time.

₹ impact: Paying to regenerate known text.

Fix this week: Add a content-addressed cache: prompt+normalized question → cached answer for N days, with a manual invalidate on policy change.

10) Instrumentation gaps

Founder symptom: “I can’t tell which feature costs what.”

Why it happens: Cost is logged globally (one monthly number) not per feature or tenant.

₹ impact: You can’t kill what you can’t see.

Fix this week: Tag every request with feature, user tier, and request_id. Emit tokens_in, tokens_out, model, latency, cache_hit. Roll it up per week.

Quick comparison: common leaks and fast fixes

A simple routing plan you can adopt this week

You don’t need an AI research team. You need sensible gates.

• Tier 0: Cache – If question+context matches a known answer within your tolerance, return cached. Zero tokens.

• Tier 1: Small model – Default path for routine queries, template fills, intent detection. Set low max_tokens.

• Tier 2: Mid‑tier model – Kick in when context length >N, when confidence from Tier 1 falls below threshold, or when the user is a paid tier requiring higher precision.

• Tier 3: Premium model – Only on: financial/compliance impact, VIP users, or when Tier 2 flags uncertainty. Log every Tier 3 invocation for weekly review.

Escalation rules to encode now:

• If cache miss AND intent in [FAQ, greeting, operating hours] → Tier 1.

• If context tokens > X OR doc count > Y → Tier 2.

• If confidence < C OR user_tier = enterprise → Tier 3.

• If tool_calls >= 2 → stop and summarize, don’t loop.

Fallbacks: Prefer a precise short answer + link over a rambling essay. On WhatsApp, latency beats verbosity.

Rupee guardrails for Indian products

Not everything needs a spreadsheet. Set three numbers and make them visible in your weekly review:

• Target cost per meaningful unit: Pick the unit your customer values: per resolved WhatsApp conversation, per qualified lead, per processed document, per generated video. Decide your target COGS per unit (e.g., ≤10–20% of what you charge).

• Escalation budget: Decide how many Tier 3 calls per 1,000 units you can afford (e.g., single digits). If you breach it, adjust routing or cache.

• Latency SLO: Define acceptable 95th percentile times. Slow paths often equal expensive paths.

These are product numbers, not infra numbers. Your team should know them like MRR.

What to instrument this week (no new vendors required)

• Per‑request tags: feature, tenant, user_tier, request_id, conversation_id.

• Core metrics: model, tokens_in, tokens_out, total_cost_hint (provider estimates if available), cache_hit, retries, tool_calls, latency_ms.

• Weekly rollups: cost per unit (conversation/lead/doc/video), Tier 3 rate, cache hit rate, P95 latency.

• Dashboards & alerts: Alert when Tier 3 rate or retries jump 2× week‑over‑week. That’s your early‑warning for leaks.

Founder checklist: fix leaks without breaking UX

• Set max_tokens per flow. Short answers win on WhatsApp and email.

• Add a content‑addressed cache for repeat prompts.

• Introduce 3‑tier routing with explicit escalation criteria.

• Make idempotency keys mandatory across queues/webhooks.

• Resize/compress vision inputs and reuse transcripts downstream.

• Embed only changed chunks, on a schedule, not on every save.

• Summarize prior turns; don’t pass full history into every prompt.

• Review Tier 3 logs weekly and push common cases down the stack.

Who should own AI cost

If everything is “infra,” no one owns unit economics. Give product the dashboard and a budget per unit. Engineering enforces guardrails; finance verifies trends; product decides when to spend extra for delight.

When you need outside eyes

If you’re founder‑led without a full‑time tech leader, get a short, sharp pass from someone who thinks like a builder and a GM. Our approach and templates are in the Fractional CTO Operating Manual. It’s the exact way we align routing, quality, and rupee spend without slowing shipping.

The point

You don’t need a new model. You need a better contract with the one you have: stricter prompts, clear caps, sane routing, and logs that let you kill leaks on Monday, not quarter‑end.

If you want a second opinion on your tech, here’s how we think about it: book a free strategy call. We’ll look at your routing, prompts, and logs—and show you where the rupees are escaping.