Production AI Architecture Study Guide — AI Advanced

Production AI Architecture

This domain tests your ability to design, optimize, and operate AI systems that run reliably and cost-effectively at scale. It covers the full stack from streaming user experiences to self-hosted inference, rate limiting, and cost modeling. At 18% of the exam, it is the second-largest domain — expect 8–11 questions. Questions emphasize architectural trade-offs, exact configuration details, provider-specific rules, and failure-mode reasoning rather than coding syntax.

Streaming UX: SSE, useChat, and Token-by-Token Rendering

Server-Sent Events (SSE) is a one-way HTTP/1.1 protocol where the server pushes text/event-stream chunks to the client over a persistent connection. It is the standard transport for streaming LLM tokens in production Next.js applications.

Key SSE rules:

• Content-Type header must be text/event-stream
• Each event is separated by a blank line (\n\n)
• Data lines are prefixed with data:
• The special data: [DONE] sentinel closes the stream (OpenAI convention)
• SSE uses HTTP, not WebSockets — no upgrade handshake required

Next.js API routes support SSE via the Response object with a ReadableStream. In the App Router (/app/api/route.ts), you return a streaming Response; in the Pages Router (/pages/api/), you set res.flushHeaders() and write chunks manually. The App Router approach is preferred because it integrates with the Vercel AI SDK.

Vercel AI SDK (ai package):

• useChat hook: manages message state, streaming, input, and submission client-side
• StreamingTextResponse (SDK v2) / streamText with .toDataStreamResponse() (SDK v3): wraps a provider stream into a compliant SSE response
• useCompletion: for single-turn completions (no message history)
• Default behavior: tokens render progressively as they arrive (token-by-token rendering)

Common traps:

• Confusing SSE with WebSockets — SSE is unidirectional; WebSockets are not needed for LLM streaming
• Forgetting to disable body parsing in Pages Router (export const config = { api: { bodyParser: false } } is NOT required for SSE — bodyParser only affects request, not response)
• Assuming useChat works without the matching API route format — it expects the AI SDK data stream protocol
• Vercel's default function timeout is 10 seconds; streaming requires enabling maxDuration (up to 800s on Pro) to prevent premature cutoff

Semantic Caching

Semantic caching stores LLM responses keyed not by exact prompt string but by embedding similarity. If a new prompt is within a configurable cosine similarity threshold of a cached prompt, the cached response is returned without calling the LLM.

GPTCache is the canonical open-source semantic cache library:

• Embeds incoming prompts using a configurable encoder (e.g., text-embedding-ada-002 or local ONNX models)
• Stores embeddings in a vector store (FAISS, Milvus, Redis, Qdrant)
• Default similarity threshold: 0.8 (cosine similarity) — tune upward (0.9+) for precision, downward (0.7) for higher hit rate
• Cache backend stores both the vector and the raw response
• Supports exact match caching as a first-pass lookup before semantic search

Cache hit flow: Incoming prompt → embed → vector similarity search → if score ≥ threshold → return cached response (no LLM call). If below threshold → call LLM → store embedding + response → return response.

Worked scenario: A customer support chatbot receives "How do I reset my password?" and "What's the process to change my password?" — semantically near-identical. With threshold 0.85, the second query hits the cache and saves one LLM call. With threshold 0.95, it misses (strings differ enough in embedding space at high precision).

Common traps:

• Assuming semantic caching works without an embedding model — embedding the query is always required
• Setting threshold too low (0.5) and returning wrong cached answers for semantically different questions
• Forgetting that the embedding call itself has cost/latency — choose a fast/cheap encoder for cache lookups
• GPTCache is not a managed service; it requires self-hosted infrastructure