Semantic Caching: Cutting LLM Cost Without Cutting Corners

Two users ask your support assistant the same thing:

"How do I reset my password?"
"i forgot my password, how to reset it"

An ordinary cache sees two different strings, hashes them to two different keys, and pays for two full model calls. A semantic cache sees two vectors pointing in nearly the same direction, serves the second one from memory, and charges you for an embedding lookup instead of an inference.

That is the whole mechanism, and it is not complicated. What makes it worth writing about is that the economics are almost absurdly favourable while the correctness risk is real. Most write-ups get that balance backwards, leading with the savings and skipping the part that can embarrass you in production.

What it actually is

A normal cache is keyed on exact equality: hash(prompt) → response. One character different and it is a miss.

A semantic cache is keyed on proximity in meaning. You embed the incoming prompt, search for the nearest prompt you have already answered, and if the similarity clears a threshold you return the stored answer. So the lookup changes shape completely:

	Exact cache	Semantic cache
Key	Hash of the string	Embedding vector
Lookup	O(1) hash get	Approximate nearest neighbour
Hit condition	Byte-identical	Similarity above a threshold
Cost per lookup	Effectively zero	One embedding plus one vector search
Can it be wrong?	No	Yes

That last row is the engineering problem in its entirety. An exact cache either has your answer or it does not. A semantic cache can hand you a confidently wrong answer to a question nobody asked.

Why exact caching fails on this traffic

Natural language has enormous surface variance for identical intent. Word order, politeness, typos, filler, contractions, and in India and South East Asia especially, code-mixing between English and a local language inside one sentence.

The consequence is that exact-match hit rates on real support and Q&A traffic sit in the low single digits. People phrase things their own way. Meanwhile the underlying intent distribution is heavily concentrated, with a handful of questions accounting for most of the volume. Exact caching cannot see that concentration at all, and capturing it is the reason semantic caching exists.

The cost arithmetic

Rather than quoting a savings percentage, here are the terms.

C_llm is the cost of one full model call, input tokens plus output tokens.
C_emb is the cost of embedding one query.
C_search is the cost of one vector search, which is compute you already own.
h is the cache hit rate, between 0 and 1.

Without a cache, N requests cost N × C_llm. With one, every request pays the embedding and search cost while only misses pay for inference:

total = N × (C_emb + C_search) + N × (1 - h) × C_llm

Which makes the cache worth having when:

C_emb + C_search < h × C_llm

Now put real magnitudes in. An embedding call is one to three orders of magnitude cheaper than a chat completion, because it uses a small model, takes short input and generates no output. Take a conservative 100× and the break-even hit rate falls out:

h > C_emb / C_llm  ≈  1 / 100  =  1%

So a semantic cache starts paying for itself somewhere around a 1% hit rate, and on concentrated traffic you will typically see 30% to 60%. This is not a close call, which is why I would push back on anyone still debating whether semantic caching is worth it on cost grounds. The real question is whether you can keep the false-hit rate low enough to trust the thing.

For reference, Redis publishes figures for its managed LangCache service of up to 15× faster responses on hits and up to 73% lower inference cost on high-repetition workloads. Treat vendor numbers as the top of the range and measure your own, but the shape is right.

There are two places the savings do not show up the way you expect. If your prompt is a 50,000-token document and the answer is one line, the embedding cost of that prompt stops being negligible and you should be caching at a different layer entirely. And provider prompt caching is a separate thing that stacks with this rather than replacing it: several providers now discount repeated prefixes within a request, which is exact-match caching on the prefix done server-side. It reduces C_llm. It does nothing for "same question, different words". Use both.

The performance argument

Cost gets the headlines but latency is what users feel.

Path	Typical latency
Exact cache hit	under 1 ms
Semantic cache hit	20–60 ms, mostly the embedding call
Model call	500 ms to 10 s

A hit is one to two orders of magnitude faster. Look at what dominates it, though: the embedding call, not the vector search. Search over a cache-sized index is sub-millisecond, and the 20 to 60 ms is a network round trip to an embedding API.

That has an architectural consequence people miss. If you want fast hits, the embedding model is what to optimise, whether that means hosting a small one locally or picking a provider with a region near you. Tuning the vector store while the embedding hop is 95% of your hit latency is effort in the wrong place.

There is a second-order benefit as well. Every hit is a request that never reaches your model provider, so it never consumes rate limit or queue capacity. Under a load spike the cache is not only making things faster, it is the reason you are not being throttled.

Architecture

Three tiers, cheapest check first. The exact-match layer in front is not redundant: identical repeats are common and serving them costs nothing at all.

Cheapest check first. Every request pays for normalisation; only L1 misses pay for an embedding; only L2 misses pay for inference.

Implementing it

The core loop is about thirty lines. Redis here because vector search and key-value lookup live in the same place, which keeps a hit down to one hop.

THRESHOLD = 0.93          # cosine similarity, tuned per intent class

def answer(prompt, tenant):
    norm = normalise(prompt)                     # trim, collapse space, strip PII
    ns   = f"cache:{tenant}"                     # never share across tenants

    # L1: exact match. Free, so always try it first.
    if hit := kv.get(f"{ns}:exact:{sha256(norm)}"):
        metrics.hit("l1")
        return hit

    # L2: semantic. One embedding, one vector search.
    vec  = embed(norm)
    near = vector_search(ns, vec, top_k=1)

    if near and near.score >= threshold_for(norm):
        metrics.hit("l2", score=near.score)
        return near.response

    # L3: the expensive path.
    metrics.miss()
    resp = llm(prompt)

    # Write back to both tiers, same TTL, tagged for invalidation.
    kv.set(f"{ns}:exact:{sha256(norm)}", resp, ttl=TTL)
    vector_upsert(ns, vec, response=resp, prompt=norm, tags=tags_for(norm), ttl=TTL)
    return resp

Four details matter more than the shape of that loop.

Normalise before you hash or embed. Trimming and case-folding lifts the L1 hit rate for nothing, and stripping obvious PII before it enters a shared cache is not optional.

Namespace by tenant, in the key rather than in a filter you might forget to apply somewhere.

Store the original prompt next to the response. You cannot audit a false hit if all you kept was a vector, and this is the difference between diagnosing a threshold problem in an hour and guessing at it for a fortnight.

Tag entries so you can invalidate them. An entry derived from a document that has since changed is now wrong, and a TTL alone will not save you.

Choosing the threshold

Everything above is mechanical. This part decides whether semantic caching helps you or humiliates you, and it is where I would spend the effort.

Similarity is a continuum and you are forcing a binary decision onto it, so there are two failure modes sitting either side of your threshold.

Set it too low and you get false hits: the cache answers a question the user did not ask. "How do I cancel my subscription?" gets served the answer to "How do I change my subscription?" This is the dangerous direction, because the response is fluent, confident and wrong, and nothing in your logs resembles an error.

Set it too high and you waste money on inference for questions you had already answered. Annoying, cheap, self-correcting.

Given that asymmetry, start strict and loosen with evidence rather than the other way round.

It should not be a single number either. Consider what tolerance these deserve:

"what are your office hours"          → loose is fine
"what is my current account balance"  → should never be cached
"how do I reset my password"          → moderate
"is dosage X safe with medication Y"  → strict, or no cache

Intent class determines tolerance. A generic FAQ can sit around 0.90. Anything touching money, health, legal matters or identity should be strict or excluded outright. If you make only one refinement beyond a global threshold, make it a per-class policy.

Tune it with data rather than intuition. Collect a few hundred real query pairs from logs and label them as same intent or different intent. Compute similarity for every pair and plot the two distributions. Your threshold goes where they separate, and the size of the overlap region tells you your irreducible error rate. If the distributions overlap badly, the problem is your embedding model rather than your threshold, and a domain-specific model will separate domain queries far better than a general one.

Then keep measuring in production. Sample cache hits and have a second model, or a human once a week, judge whether the served answer actually addressed the question. False hit rate needs to be a number on a dashboard.

What not to cache

A short list that will save you an incident.

Anything personalised. "What is my order status" has a different correct answer per user, and semantically those queries are nearly identical, which makes this exactly the case a semantic cache gets wrong.

Anything time-sensitive: prices, availability, live status.

Anything a user can act on financially or medically without a human in between.

Anything with a per-request authorisation dimension. If two users are entitled to different answers then the entitlement has to be part of the cache key, or you do not have a cache, you have a data leak. This is the one that turns a performance optimisation into a security incident, so I will state it directly: a shared semantic cache across tenants will leak data across tenants as soon as somebody asks a similar-enough question. Namespace per tenant, and treat any cross-tenant sharing as a deliberate decision that needs a reason.

Invalidation

Semantic caches inherit every hard part of cache invalidation and add one of their own, which is that you cannot enumerate the keys depending on a given fact, because the keys are vectors.

Three things work in practice. TTL is the floor rather than the strategy, short enough that stale answers age out and long enough to keep the hit rate, which usually means hours rather than days. Tagging on write handles content changes: when you store an answer derived from document D, tag the entry with D, and drop by tag when D changes. That requires knowing what fed each answer, which is a good discipline to have anyway.

The third is versioning the whole cache. Put a version in the namespace, cache:v3:{tenant}, and bump it whenever the prompt template, the model or the embedding model changes, since all three alter what stored entries mean. Bumping a prefix is the cheapest invalidation available.

That last one catches a failure worth understanding. Change your embedding model and every stored vector becomes meaningless, because it now lives in a different vector space and similarity scores against the old vectors are noise. This has to be a version bump. It cannot be a rolling change.

Metrics

Metric	Why
Hit rate, split L1 and L2	If L2 adds little over L1, the threshold is too strict or the traffic is not concentrated
False-hit rate	The only metric that tells you whether the cache is safe
p50 and p99 latency by path	Confirms hits are fast, exposes a slow embedding hop
Cost per thousand requests	The number you will be asked about
Score distribution of served hits	A pile-up just above the threshold means you are skating close to the edge

If you can only track two, track hit rate and false-hit rate. Either one without the other gives you a cache you cannot justify or a cache you cannot trust.

When to skip it

Low-repetition traffic, like creative generation or unique document processing, has nothing to hit. Fully personalised workloads make the cache pure overhead. Domains with near-zero tolerance for a wrong answer and no capacity to tune and monitor a threshold are better off without one, because an unmeasured semantic cache is a liability. And output-heavy, prompt-light workloads may already be covered by a provider's own prefix caching.

Where I would start

The unusual thing about semantic caching is that the cost argument is trivially won while the correctness argument takes real work. That inverts the normal engineering conversation, and it explains why all the interesting decisions turn out to be about the threshold, the tenant boundary, and what you refuse to cache at all.

If you are implementing it, I would do the exact-match tier first because it is free, then add semantic with a deliberately strict threshold, then measure false hits before loosening anything. Namespace by tenant from the first commit, since retrofitting that is painful and the failure mode is a data leak rather than a performance regression.

Do that and you have a cache that makes the system cheaper and faster without ever making it wrong, which is the only version of this worth shipping.