Deduplication at Scale: Building a strict L1-L2-L3 cache pipeline

High-volume AI products rarely fail because of one dramatic engineering mistake. They usually fail slowly through quiet duplication: repeated prompts, retried requests, and semantically equivalent questions that trigger full-price inference again and again. In production, this "duplication tax" can account for 30% to 60% of total inference spend if left unmanaged.

"Caching for LLMs is fundamentally different from caching for REST APIs. You aren't just matching keys; you are matching intent across a sea of noise."

If your serving layer cannot identify and reuse prior work safely, your cost profile scales linearly with traffic, not value. To break this link, we need more than a simple Redis KV-store. We need a multi-layer pipeline designed for trust, normalization, and semantic awareness.

Normalization: The Foundation of Cache Hit Rate

Most teams struggle with cache hits because they hash the raw request body. This is a mistake. AI requests are full of non-semantic variance: whitespace differences, varying system prompts, slightly different temperature settings, or metadata fields that change with every user session.

A production-grade gateway must implement a Canonicalization Layer before reaching the cache. This involves:

Without this step, your cache is at the mercy of the client's formatting. With it, you can achieve a "Hard Hit" rate that remains stable even as your frontend code evolves.

The L1-L2-L3 Pipeline: Tiered Decision Making

Deduplication should follow a pipeline of increasing complexity (and cost). We categorize hits into three distinct architectural layers:

Semantic Matching: Trust over Hit Rate

Semantic caching is where most teams fail. They use a broad cosine similarity threshold (e.g., 0.85) and find that the cache returns "hallucinated" answers that don't quite match the user's intent. In a production environment, trust is more important than hit rate.

We recommend a "Strict Semantic" approach. This means using high-dimensional embeddings (like text-embedding-3-large) paired with second-order validation logic:

• Length Ratio Check: If the candidate answer is 50% shorter than the average response for this prompt, reject it as a potential incomplete or error-state cache entry.
• Lexical Anchor Matching: Ensure that key entities (names, numbers, specific nouns) present in the prompt also appear in the cached result.
• Adaptive Thresholds: Lower-cost models can have looser similarity requirements, but premium reasoning models must have a strict 0.98+ threshold.

Write-Path Stability: Async or Bust

The performance gains of a cache are erased if the "Write" path blocks the "Read" path. Generating an embedding for every request is expensive (both in latency and provider cost).

"The right architecture is synchronous read, asynchronous write. Serve the token, then enqueue the cache update."

By moving cache population—including normalization, embedding generation, and vector indexing—to a background worker (or a Go-routine in a high-throughput gateway), you keep your p99 latency flat even when the cache itself is under heavy write pressure.

What to measure beyond 'Hit Rate'

High hit rates can be misleading. To understand if your deduplication strategy is actually working, you need to track Value-Adjusted Hit Rate.

This metric weights each hit by the cost of the model it avoided. A hit on a gpt-5.2 request is worth 50x more than a hit on Llama-3-8B. By focusing on the "expensive misses," engineering teams can refine their normalization logic where it has the highest financial impact.

Deduplication at scale is not a set-it-and-forget-it feature. It is a core serving discipline. Done correctly, it transforms an AI product from a cost-center into a highly efficient, predictable engine of value.