Question 1

What is a Large Language Model and how does it generate text?

Accepted Answer

A Large Language Model is a transformer neural network trained on huge text corpora to predict the next token. At inference it generates text one token at a time, feeding each generated token back in as input (autoregressive decoding). Capabilities like reasoning, coding and instruction-following emerge from this objective plus post-training (instruction tuning, RLHF).

Question 2

What are embeddings and what are they used for?

Accepted Answer

An embedding is a dense vector (hundreds to thousands of floats) produced by a model such that semantically similar texts map to nearby points. Because similarity becomes geometry, embeddings power semantic search (query vs document vectors), RAG retrieval, deduplication, clustering, classification and recommendations. Key practical facts: they come from dedicated embedding models (not the chat model), similarity is usually cosine, and vectors from different models are not comparable.

Question 3

What belongs in the system prompt versus the user message?

Accepted Answer

The system prompt sets durable behaviour: role/persona, rules and constraints, output format, tool-use policy — things the app controls and the user should not override. User messages carry the task and data for each turn. Keeping the split clean improves instruction-following (models weight system instructions heavily), enables prompt caching of the stable prefix, and is your first line of defense against prompt injection since untrusted content stays in user/tool messages.

Question 4

What is a vector database and when do you actually need one?

Accepted Answer

A vector database stores embeddings with ANN indexes plus filtering, CRUD and scaling (Pinecone, Weaviate, Qdrant, Milvus). But you often do not need a dedicated one: under ~100k vectors a flat in-memory search is fine, and pgvector/SQLite extensions let you keep vectors next to your relational data with transactional consistency. Reach for a dedicated vector DB at millions of vectors, high QPS, or when you need heavy metadata filtering combined with ANN at scale.

Question 5

Context windows in 2025-26: how do you use 128k-1M tokens without drowning?

Accepted Answer

128k is table stakes, Gemini/Claude offer 1M. Do not stuff — retrieve + place key info at start/end, compress middle, and test needle recall.

Question 6

What is RAG and why use it instead of fine-tuning for knowledge?

Accepted Answer

Retrieval-Augmented Generation retrieves relevant documents for a query (usually via embeddings) and puts them in the prompt so the model answers from them. Versus fine-tuning it wins for knowledge because it is updateable instantly (reindex, no retraining), auditable (citations), per-tenant scoped (retrieval filters enforce access), and cheaper. Fine-tuning is for behaviour and style, not facts — the standard architecture is RAG for knowledge, prompting/fine-tuning for form.

Question 7

What are tokens and why do they matter for cost and limits?

Accepted Answer

Tokens are the sub-word units a model reads and writes (roughly 4 characters or 0.75 words of English each). Everything is denominated in tokens: API pricing (per input/output token), the context-window limit, and generation latency (one forward pass per output token). Tokenization also explains quirks like poor character counting and inflated costs for non-English text or JSON-heavy payloads.

Question 8

What are evals and why does every LLM feature need them?

Accepted Answer

Evals are automated tests for model behaviour: a dataset of realistic inputs, expected outcomes or grading criteria, and graders (code checks or LLM judges) producing scores. They exist because LLM behaviour is nondeterministic and prompt/model changes have non-local effects — without evals every change is vibes-tested on three examples. A minimal loop: collect real cases (especially failures), define graders, run on every prompt/model change, track scores over time. Evals are the highest-levera

Question 9

What are hallucinations and what causes them?

Accepted Answer

A hallucination is fluent output that is factually wrong or fabricated — invented citations, APIs that do not exist, confident wrong answers. They happen because the model optimizes plausibility (next-token likelihood), not truth: when the training data or provided context lacks the answer, the most statistically plausible continuation may be false. Mitigations include grounding with retrieval, allowing the model to abstain, and verifying claims downstream.

Question 10

AI observability: how do you trace and log LLM calls in prod?

Accepted Answer

Emit OTEL gen_ai spans for every LLM call (model, tokens, latency, cost, prompt version). Centralize via AI Gateway/LangSmith and replay failures into evals.

AI ENGINEERINGDECODED.

Transmission log

What is a Large Language Model and how does it generate text?

What are embeddings and what are they used for?

What belongs in the system prompt versus the user message?

What is a vector database and when do you actually need one?

Context windows in 2025-26: how do you use 128k-1M tokens without drowning?

What is RAG and why use it instead of fine-tuning for knowledge?

What are tokens and why do they matter for cost and limits?

What are evals and why does every LLM feature need them?

What are hallucinations and what causes them?

AI observability: how do you trace and log LLM calls in prod?

What is the difference between zero-shot, one-shot and few-shot prompting?

What is a context window and what happens when you exceed it?

AI SDK 5: when to use streamText vs generateText vs generateObject?

How do you choose the right model for a task?

When should you use batch APIs instead of real-time LLM calls?

How do you build resilient LLM calls with rate limits, retries and fallbacks?

How does streaming work in LLM APIs and why does it matter for UX?

What is the difference between a base model, an instruction-tuned model and a chat model?

How do you get reliable structured JSON output from an LLM?

How do you build a golden dataset for evaluating an LLM feature?

What is the Model Context Protocol (MCP) and what problem does it solve?

Embeddings in 2025-26: how do you choose model, dimensions, and distance?

When do you use code-based graders versus model-based graders?

How do multimodal models handle images, audio and other non-text inputs?

What is the difference between fine-tuning, LoRA and prompt engineering?

How do you estimate and control LLM costs in production?

Beyond Ragas: how do you eval LLMs with LLM-as-judge, DeepEval, and human eval?

Cost optimization for LLM apps: how do you keep the bill predictable?

How does similarity search work — cosine, dot product and Euclidean?

Tool calling vs function calling vs structured output — what changed in 2025?

How do you manage conversations that outgrow the context window?

Hybrid search: why combine BM25 with vectors and how does RRF work?

How do temperature, top-p and other sampling parameters affect output?

How do you design human-in-the-loop approval for agent actions?

Reranking in RAG: when and how to add a cross-encoder?

What chunking strategies exist and how does chunk size affect RAG quality?

What is prompt caching and how does it reduce cost and latency?

Prompt caching in 2025: how do you cut cost with Anthropic & OpenAI caches?

What is context engineering and how does it differ from prompt engineering?

What is an AI agent and how does the agent loop work?

What is chain-of-thought prompting and when does it actually help?

How do you choose between open-weight models and proprietary API models?

How do you reduce hallucinations in a production LLM application?

What is reranking and why add it after retrieval?

How do you design good tool definitions for an LLM?

What is LLM-as-judge and what are its known biases?

How do you scope retrieval with metadata filtering and multi-tenancy?

Evaluating RAG with Ragas: faithfulness, answer relevancy, context precision?

How do you handle PII and sensitive data when calling LLM APIs?

Chunking in RAG: how do you pick size, overlap, and strategy?

Streaming LLM responses to the UI: SSE, chunking, abort, and resume?

What is hybrid search and why combine BM25 with vector search?

What are guardrails and how do you implement them around a model?

How do you catch regressions when changing a prompt or upgrading a model?

How does function calling (tool use) work under the hood?

What is the ReAct pattern and why did it become the default for agents?

Vercel AI SDK 5 streamText: how do you stream LLM chat to React?

What should you log and trace in an LLM application?

How should an agent handle tool errors, timeouts and retries?

How do you sandbox an agent and enforce least-privilege tool access?

How do you evaluate a RAG pipeline end to end?

Guardrails for LLM apps: how do you implement input/output safety in 2025?

What is semantic caching and when is it safe to use?

When should you split work across multiple agents instead of one?

How do you evaluate and debug agent trajectories?

How would you design a production RAG pipeline end-to-end in 2025-26?

MCP authoring: how do you build a Model Context Protocol server in 2025-26?

Why are BLEU and ROUGE poor metrics for LLM outputs, and what replaced them?

How do you version, test and safely roll out prompt changes?

What are common RAG failure modes and how do you debug them?

What is prompt injection and how do you defend against it?

Guardrails for AI agents: rate limits, budget caps, and HITL?

How do you validate and repair structured LLM output at runtime?

pgvector vs Pinecone vs Qdrant: how do you choose a vector DB in 2025?

Fine-tune vs RAG vs prompt engineering: how do you choose in 2025-26?

How do you monitor LLM quality drift in production?

What techniques reduce LLM latency at each stage of a request?

What is quantization and when would you use a quantized model?