Six AI projects: a field guide

The original gives five steps per project. Here every step is expanded: why it is needed, what decisions it involves, where it usually breaks, how to check that it works, and how to carry it over to my stack. Facts about the Claude API were checked against the official documentation in October 2026.

Overview

§ 1. Hybrid Search RAG over Internal Docs

In short: answers over internal documentation, grounded in sources. Two retrievers (by meaning and by keywords) cover each other’s weak spots, a reranker picks the best, and the model answers only from what was found.

Original stack: Python · LangChain · Claude API · Pinecone · rank_bm25 · Cohere Rerank · FastAPI

On my stack: Go · PostgreSQL + pgvector · Postgres full-text search · Claude API (Go SDK) · reranker via API

Steps

1. Split documents into chunks, compute embeddings, store them in Pinecone.

   Split by document structure (headings, paragraphs, lists), not by a fixed number of characters. Rule of thumb: 300-800 tokens per chunk, 10-20% overlap. Every chunk has a stable id, doc_id, section heading, url, update date and access permissions. A useful trick: before embedding, prepend the document and section titles to the chunk. Otherwise a fragment like “increase the timeout to 30 seconds” loses its context: the timeout of what?

2. Build a BM25 index over the same corpus.

   The same set of chunks and the same ids as in the vector index, otherwise the merge in step 3 breaks. BM25 catches what embeddings blur: error codes, SKUs, names, abbreviations, ticket numbers. Russian and Kazakh need normalization (lowercasing, stemming or lemmatization), otherwise “договор” and “договора” (“contract” and “of the contract”) become different tokens.

3. Run both retrievers and merge the results with Reciprocal Rank Fusion.

   Each retriever returns its top-N (around 50). RRF adds up positions, not scores (dense and BM25 scores live on incomparable scales): score(d) = Σ 1 / (k + rank_i(d)), usually k = 60. The winner is a document ranked high in both lists. No score normalization is needed, and that is the whole beauty of the method.

4. Rerank the merged list with Cohere Rerank and keep the top-k.

   A reranker (cross-encoder) reads the query and the chunk together, so it is more accurate than embeddings, but slower and more expensive. Feed it only the candidates after RRF (20-50 of them), and keep 5-10 for the model’s context. This is the main quality lever: retrieval is responsible for recall, the reranker for precision.

5. Generate an answer with inline citations and check that every claim is grounded in a source.

   The model may answer only from the chunks it was given, and when there is nothing to rely on, it says honestly “the documentation does not cover this”. The Claude API has a native Citations feature: chunks are passed as documents with citations.enabled, and the answer comes back in blocks that point to specific places in the sources. The API guarantees that citations point to documents that were actually passed, and the cited text is not billed as output. Custom content documents suit RAG chunks: the API does not split them further. A limitation: Citations cannot be combined with structured outputs. Checking that a claim really follows from its source is still on you: spot-check it with an LLM judge.

Key ideas

• Dense and sparse fail in different places: embeddings understand synonyms but confuse codes and names; BM25 is the opposite. A hybrid is cheaper than squeezing the maximum out of one of them.
• Recall first, then precision: wide retrieval, narrow rerank.
• Access permissions are filtered before retrieval (a metadata filter), not after generation.
• Less context is often better: 6 good chunks beat 30 mediocre ones.

Code

def rrf(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

candidates = rrf([dense_ids, bm25_ids])[:50]   # then into the reranker

-- chunks: id, content, embedding vector(...), tsv tsvector
-- tsv: GENERATED ALWAYS AS (to_tsvector('russian', content)) STORED + GIN index
-- $1: query embedding, $2: query text
WITH semantic AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS rnk
  FROM chunks
  ORDER BY embedding <=> $1
  LIMIT 50
),
keyword AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS rnk
  FROM chunks, websearch_to_tsquery('russian', $2) AS q
  WHERE tsv @@ q
  ORDER BY ts_rank_cd(tsv, q) DESC
  LIMIT 50
)
SELECT COALESCE(s.id, k.id) AS id,
       COALESCE(1.0 / (60 + s.rnk), 0) + COALESCE(1.0 / (60 + k.rnk), 0) AS rrf_score
FROM semantic s
FULL OUTER JOIN keyword k ON s.id = k.id
ORDER BY rrf_score DESC
LIMIT 30;

Pitfalls

• The indexes drift out of sync: a document was updated in the vector index but not in BM25. You need one ingestion pipeline and one source of truth for chunks.
• rank_bm25 keeps the index in memory and cannot update incrementally: fine for a prototype, not for production.
• Chunks without context: “see above”, a table cut in half, code without a description.
• Stale documents beat fresh ones. Take the update date into account or mark outdated content.
• The built-in Postgres ranking (ts_rank_cd) is not BM25. For real BM25 there are extensions (for example, pg_search from ParadeDB) or a separate search engine.
• Postgres has no counterpart of the russian configuration for Kazakh: for a mixed corpus use simple plus your own normalization.

How to tell it works

• A golden set of 50-100 real questions with known correct chunks (built by project 04).
• Retrieval: recall@k (did the right chunk make the top-k), MRR, nDCG. Measure dense, BM25, hybrid and hybrid + rerank separately to see what each layer contributes.
• Generation: faithfulness (is everything backed by sources), answer relevance, the share of honest “I don’t know” answers to questions the corpus has no answer for.

On my stack

Everything except the reranker lives in one Postgres: pgvector for vectors, tsvector + GIN for keywords, RRF in a single SQL query. Fewer moving parts, and the index drift problem disappears: a chunk and both of its indexes are updated in one transaction. Anthropic does not ship its own embedding model and its documentation points to Voyage AI (who also offer rerankers), but choose the provider for your language and domain by running your golden set. The Go SDK for Claude supports Citations.

§ 2. Text-to-SQL with Guardrails

In short: a question in natural language turns into SQL and a result. The model writes the query, but permission to run it comes only from deterministic code and database privileges.

Original stack: Python · Claude API · SQLAlchemy · Postgres · sqlglot · Pydantic · FastAPI

On my stack: Go · pg_query_go · pgx · read-only role on a replica · Claude API with structured outputs

Steps

1. Give the model the schema and the user’s question.

   Not bare DDL, but a schema with meaning: comments on tables and columns, foreign keys, sample values for enum-like columns (status: 'paid' | 'refunded'), business definitions (“revenue = sum of paid minus refunded”), 5-10 “question → reference SQL” pairs. If there are dozens of tables, start with schema linking: a separate step picks the relevant tables and passes only those.

2. Parse the generated SQL with sqlglot and block anything destructive.

   Work with the parse tree (AST), not with regexes. An allowlist, not a denylist: exactly one statement is allowed, and it is a SELECT (including WITH and UNION). Plus a tree walk: no INSERT/UPDATE/DELETE inside a CTE (Postgres has data-modifying CTEs), only allowed tables and columns, no dangerous functions (pg_sleep, pg_read_file, dblink and the like), a forced LIMIT.

3. Execute it read-only on an isolated replica first.

   The parser is the first line of defense, but only the database gives a guarantee: a separate role with SELECT on the tables it needs, a READ ONLY transaction, statement_timeout, a replica instead of the primary. For multi-tenant data, Row Level Security, so the query physically cannot see other tenants’ rows. Run EXPLAIN before executing: reject plans with a huge estimated cost.

4. Attach a confidence score and ask for confirmation when it is low.

   Signal sources: self-consistency (generate 3-5 SQL variants and compare the results: if they agree, confidence is higher), validation errors and the number of retries, ambiguity in the question itself (“last month” in which time zone?). Below the threshold, do not execute silently: show the SQL with its assumptions or ask a clarifying question.

5. Return the result together with the SQL so the user can check it.

   The response contains the data, the SQL, a plain-language explanation and a list of assumptions (“only counted orders with status paid”). A user who sees the SQL catches errors that no parser will.

Key ideas

• Defense in depth: prompt, parser, database privileges, timeouts. Each layer can be broken through, but all of them together almost cannot.
• The most common error is not a dangerous query but a wrong one: syntactically valid, with the wrong granularity, duplicates from a JOIN or the wrong filter. What protects against it is the schema description and the eval set, not guardrails.
• Ask the model to return the structure {sql, assumptions, needs_clarification}, not text with SQL inside.

Code

import sqlglot
from sqlglot import exp

ALLOWED_TABLES = {"orders", "customers", "products"}
READ_ROOTS = (exp.Select, exp.Union, exp.Intersect, exp.Except)
WRITE_NODES = (exp.Insert, exp.Update, exp.Delete, exp.Drop, exp.Create, exp.Command)

def guard(sql: str) -> exp.Expression:
    statements = sqlglot.parse(sql, read="postgres")
    if len(statements) != 1 or statements[0] is None:
        raise ValueError("exactly one statement required")
    tree = statements[0]
    if not isinstance(tree, READ_ROOTS):
        raise ValueError("only SELECT is allowed")
    if next(tree.find_all(*WRITE_NODES), None) is not None:
        raise ValueError("write inside the query")
    cte_names = {cte.alias_or_name for cte in tree.find_all(exp.CTE)}
    for table in tree.find_all(exp.Table):
        if table.name not in ALLOWED_TABLES | cte_names:
            raise ValueError(f"table not in allowlist: {table.name}")
    return tree

CREATE ROLE nl2sql_reader LOGIN;          -- password via secrets
GRANT USAGE ON SCHEMA analytics TO nl2sql_reader;
GRANT SELECT ON analytics.orders, analytics.customers, analytics.products TO nl2sql_reader;
ALTER ROLE nl2sql_reader SET default_transaction_read_only = on;
ALTER ROLE nl2sql_reader SET statement_timeout = '5s';

Pitfalls

• Prompt injection through data: values from tables end up in the model’s context on later iterations and can contain instructions.
• Personal data: exclude PII columns both from the schema the model sees and from the allowlist.
• Expensive queries: a Cartesian product without WHERE is safe and valid, but it takes down the replica. Hence the timeout and EXPLAIN.
• Dialect: the model writes MySQL syntax for Postgres. State the dialect explicitly in the prompt; sqlglot can transpile, but that is a crutch.

How to tell it works

• Execution accuracy on a “question → reference SQL” set: compare results, not query text (the same answer can come from different SQL).
• The share of blocked queries and how many of them were blocked needlessly.
• The share of answers where the user edited the SQL or asked again.

On my stack

pg_query_go uses the real Postgres parser, so the parse tree matches how the database itself will understand the query: for a single dialect this is more reliable than sqlglot. Pydantic is replaced by Go structs and JSON Schema for structured outputs or strict tool use in the Claude API. In pgx, open the transaction with pgx.TxOptions{AccessMode: pgx.ReadOnly} on top of a role with default_transaction_read_only: two independent locks.

§ 3. LoRA Fine-Tuning Pipeline

In short: fine-tune an open model for your domain, usually training less than 1% of the parameters, on a single GPU. You need it when prompting and RAG do not give you the behavior, format or price you need.

Original stack: Python · PyTorch · Hugging Face · PEFT · bitsandbytes · Weights & Biases

On my stack: training stays in Python (that is where the ecosystem is) · serving via vLLM behind a Go service

Steps

1. Collect a clean instruction dataset in your domain.

   Quality matters more than volume: a few hundred or a few thousand verified examples beat tens of thousands of noisy ones. A single format (the target model’s chat template), deduplication, balance across task types, a held-out eval split with no overlap with train (near-duplicates included). A good source: real logs from project 04 plus manual review. Synthetic data from a strong model is acceptable if someone has checked it.

2. Load the base model in 4-bit with bitsandbytes.

   This is QLoRA: the base weights are quantized to NF4 and frozen, computation runs in bf16, and only the adapters are trained. A 7-8B parameter model fits on a single 24 GB GPU in this mode. Out of memory: gradient checkpointing.

3. Attach LoRA adapters with PEFT, choose rank and alpha.

   LoRA does not touch the original matrix W; it adds a low-rank correction: W' = W + (α / r) · B·A, where B is d×r and A is r×k, both small. r (rank) is the adapter’s capacity: 8-16 for style and format, 32-64 for harder tasks. alpha scales the contribution; a common heuristic is alpha = r or alpha = 2r. target_modules: the attention projections (q, k, v, o), often the MLP layers as well, which usually improves quality.

4. Train while tracking loss and metrics in W&B.

   Log train loss, eval loss, learning rate and, most importantly, the task metric on the eval set every N steps. Rules of thumb for LoRA: learning rate around 1e-4 ... 2e-4, 1-3 epochs. Eval loss rising while train loss falls means overfitting: take the best checkpoint. Compute the loss only on the response tokens and mask the instruction tokens.

5. Merge the adapters into the base and compare against the base model.

   Merge the adapter into a base loaded in bf16/fp16, not in 4-bit: merging into a quantized model introduces errors. Compare three variants: the base as is, the base with the best prompt, the fine-tuned model. Plus a general benchmark for regressions: narrow fine-tuning can degrade general abilities (catastrophic forgetting).

Key ideas

• Fine-tuning changes behavior, not knowledge. Keep facts that go stale in RAG.
• The only fair comparison is against the base with its best prompt, not the base with no prompt.
• An adapter is small (megabytes) and the base is shared: you can keep several adapters for different tasks.

Code

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, quantization_config=bnb, device_map="auto"
)
model = prepare_model_for_kbit_training(model)

lora = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()   # expect under 1% of parameters

from peft import PeftModel

# load the base in bf16, not 4-bit (older transformers versions call the parameter torch_dtype)
base = AutoModelForCausalLM.from_pretrained(BASE_MODEL, dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(base, ADAPTER_DIR).merge_and_unload()
merged.save_pretrained("merged-model")

Pitfalls

• The chat template differs between training and inference: in production the model suddenly gets “dumber”.
• Eval leaking into train: beautiful metrics and zero benefit.
• The base model’s license: not all open weights allow commercial use.
• A large rank on a small dataset: overfitting arrives sooner than you think.

How to tell it works

• The task metric (exact match, F1, JSON validity, an LLM judge with a rubric) on the held-out set is higher than for the base with the best prompt.
• General benchmarks have not dropped.
• Inference latency and cost are calculated: the quality gain is not eaten up by hosting costs.

On my stack

vLLM can serve several LoRA adapters on top of one base model (multi-LoRA) and exposes an OpenAI-compatible HTTP API that a Go service calls with an ordinary client. One GPU instance for several narrow tasks. This project is about open models: Claude’s behavior is tuned with prompts, examples and tools.

§ 4. Automated Eval Dataset Generator

In short: turn production logs into a versioned test set that runs automatically on every new version of the model, prompt or pipeline and catches regressions before release. Without it, any “improvements” in the other five projects are feelings, not facts.

Original stack: Python · Claude API · Pandas · Pydantic · DuckDB · Streamlit

On my stack: Kafka → Parquet in Object Storage · DuckDB on top of Parquet · Go runner in CI · results in Postgres

Steps

1. Pull real prompts and responses from production logs.

   Synthetic tests check what you thought of; logs show what people actually ask. Scrub PII first. Log these along with the prompt: the model and prompt versions, the context passed in, user signals (a dislike, a repeated question, a manual edit of the answer), latency, tokens. DuckDB reads Parquet and JSONL directly, without loading them into a database.

2. Cluster them to find the cases that really matter.

   Query embeddings → clustering (k-means or HDBSCAN) → the model gives the clusters human-readable names. Then a stratified sample: every large cluster is represented; rare but costly cases (money, legal matters, data deletion) are weighted up; every known failure goes into the set without exception. The goal is to cover the types of queries, not to repeat the most frequent ones.

3. Generate reference answers with a model, then review them by hand.

   The model drafts the expected answer, a human edits and approves it: the manual pass is exactly what turns the set into a golden set. For open-ended answers the reference is not text but a rubric: which facts must be present, what must not appear, what format to follow. Streamlit serves here as the labeling interface: the query, the draft, and “accept / fix / reject” buttons.

4. Automatically score every new model version on this set.

   Deterministic checks wherever possible: valid JSON, matching fields, exact match, presence of required facts. LLM-as-judge for the rest: the judge gets the rubric and gives a score with a rationale. The judge is calibrated: on 50-100 examples, compare its scores with human ones. Known judge biases: they favor long answers, the first position in pairwise comparison, and answers from their own model. For nightly runs where latency does not matter, the Message Batches API fits: it is cheaper than regular calls.

5. Catch regressions before they reach users.

   Compare per cluster, not just the average: +2% on average can hide −15% in one important cluster. Account for noise: several runs or a confidence interval, and a regression threshold larger than the noise. Wire it in as a CI gate: a PR that changes a prompt does not get merged if an important cluster dropped.

Key ideas

• The set is alive: every production incident becomes a test, like a regression test after a bug.
• Version the set like code: releases are compared on a frozen version.
• The set must not leak into prompts and few-shot examples, otherwise you measure memory, not quality.

Code

from typing import Literal
from pydantic import BaseModel

class EvalCase(BaseModel):
    id: str
    cluster: str
    input: str
    expected: str | None = None
    rubric: list[str] = []      # what the answer must contain
    must_not: list[str] = []    # what must not appear
    source: Literal["prod_log", "incident", "synthetic"]
    reviewed_by: str | None = None

SELECT prompt_version,
       count(*)                                              AS requests,
       avg(CASE WHEN feedback = 'down' THEN 1 ELSE 0 END)    AS dislike_rate,
       quantile_cont(latency_ms, 0.95)                       AS p95_ms
FROM read_parquet('logs/*.parquet')
WHERE ts >= now() - INTERVAL 7 DAY
GROUP BY prompt_version
ORDER BY prompt_version;

Pitfalls

• A set built only from the most frequent queries: a high score and zero information about the edge cases.
• Unreviewed references from a model: the system is graded against its own mistakes.
• A judge from the same model as the one under test: inflated scores.
• The set has gone stale: the product changed, and the tests check old behavior.

How to tell it works

• The gate catches a known regression: deliberately roll back a good prompt version and make sure CI turns red.
• The judge’s agreement with humans is measured and acceptable.
• New incidents make it into the set regularly, not just “someday”.

On my stack

The logs already go through Kafka: a sink writes Parquet to the S3-compatible Object Storage of Yandex Cloud, and DuckDB reads straight from there via the httpfs extension. A runner in Go: runs the cases in parallel through the Claude API with a concurrency limit (errgroup.SetLimit), with results and run history in Postgres. DuckDB has a Go driver if you need the analysis inside the service.

§ 5. Semantic Caching Layer

In short: do not call the model if a similar question has already been asked. A layer in front of the LLM returns a stored answer based on semantic similarity: it saves money and hundreds of milliseconds at the risk of returning an unsuitable answer.

Original stack: Python · FastAPI · Redis · OpenAI embeddings · FAISS · Claude API

On my stack: Go middleware in front of the LLM client · Redis with vector search or pgvector · any embedding provider

Steps

1. Compute an embedding for every incoming prompt before calling the model.

   Normalize the text before embedding (trim whitespace, fold case). An embedding costs money and tens of milliseconds on every request, misses included, so the layer pays off only with a high enough hit rate. Put a cheap exact cache keyed by hash in front of the semantic layer: identical requests should not even reach the embedding step.

2. Search the cache for a semantically similar past prompt.

   The nearest neighbor by cosine similarity through an ANN index (HNSW in Redis or FAISS in memory). Search only within the request’s “space”: the key includes the model, the system prompt version, the generation parameters, the tool set, and for personalized answers also the user or tenant. Otherwise user A gets an answer generated for user B.

3. If the similarity is above the threshold, return the cached answer.

   The threshold is the main parameter of the project, and it depends on the embedding model. Start conservatively (0.95 and above) and tune it on labeled “same / different” pairs. The dangerous near-duplicates, where embeddings show high similarity despite a different meaning: negation (“how to cancel a subscription” and “how to keep a subscription from being canceled”), numbers and dates (“report for March” and “report for May”), entities (“taxes for a sole proprietor” and “taxes for an LLP”).

4. On a miss, call the model and store the embedding together with the answer.

   Store with a TTL by data type: reference material lives longer than answers about prices and statuses. Do not cache: error responses, refusals, answers that depend on the current time or on earlier messages in the conversation. What gets cached is mostly single self-contained questions; multi-turn context almost never matches.

5. Put the hit rate and the money saved on a simple dashboard.

   Metrics: hit rate (exact and semantic separately), p50/p95 latency on a hit and on a miss, savings = hits × average call cost − cost of all embeddings. And the main thing the original lacks: the false hit rate. Send a sample of hits for review (to a human or an LLM judge: does the cached answer fit the new question).

Key ideas

• A cache without scope is a data leak between users.
• A high hit rate means nothing without a known false hit rate.
• The system prompt version in the cache key solves invalidation when the prompt changes; the TTL solves stale data.

Code

def cached_completion(req):
    scope = (req.model, req.system_prompt_version, req.tenant_id)
    norm = normalize(req.prompt)

    if hit := exact_cache.get(hash((scope, norm))):
        return hit                                    # 1. exact match

    vec = embed(norm)
    match = vector_index.nearest(vec, scope=scope)
    if match and match.similarity >= THRESHOLD:
        metrics.semantic_hit(match.similarity)
        return match.response                         # 2. semantic match

    resp = llm.complete(req)                          # 3. miss
    if is_cacheable(req, resp):
        ttl = ttl_for(req)
        vector_index.add(vec, resp, scope=scope, ttl=ttl)
        exact_cache.set(hash((scope, norm)), resp, ttl=ttl)
    return resp

Pitfalls

• FAISS in process memory is not shared between replicas and is lost on restart.
• The documentation changed, but the cache keeps serving old answers: you need a TTL or explicit event-driven invalidation.
• Concurrent identical misses: ten requests in a row, ten model calls.
• A threshold picked “by eye” on five examples.

How to tell it works

• The false hit rate on a sample of hits is measured and acceptable.
• The savings are positive after subtracting the cost of embeddings.
• Latency on a hit is noticeably lower than on a miss, and latency on a miss has grown only slightly.

On my stack

An almost ideal project for Go: it is middleware in the HTTP client to the LLM. Redis is already in the stack, and it has vector search with an HNSW index; the alternative is a Postgres table with pgvector. Concurrent identical misses are collapsed by golang.org/x/sync/singleflight: the model is called once, and the others wait for the same result. Export the hit/miss/false hit metrics to the same place as the rest of the service’s metrics.

§ 6. Agent Orchestration System

In short: an agent as an explicit state graph: it plans, calls tools, checks the result and decides what to do next. The goal of the project is not a “smart” agent but a predictable one: every action is typed, every state is saved, a human confirms anything risky, and any failure can be reproduced.

Original stack: Python · LangGraph · Claude API · Redis · Postgres · FastAPI · Celery

On my stack: Go · an explicit state machine or Temporal instead of LangGraph + Celery · Redis for hot state · Postgres for history · Kafka for events

Steps

1. Describe the agent graph with plan, act and reflect nodes.

   The state is a typed object: the goal, the plan, the step history, tool results, counters. plan breaks the goal into steps. act picks and calls a tool. reflect compares the result with the goal and decides: continue, replan, finish or escalate to a human. Transitions are conditional, but with hard limits: maximum iterations, a token and money budget, a timeout for the whole run. Without them, the first “let me try again” loop will burn through the budget.

2. Register every tool with a typed input schema.

   JSON Schema (from Pydantic or Go structs) and validation before execution. In the Claude API, strict tool use guarantees that the call arguments match the schema. Return errors to the model in a structured form (“the date field must be in YYYY-MM-DD format”) so it can correct itself. A tool description is also a prompt: when to use it, when not to, what it returns. Five clear tools are better than thirty overlapping ones. Make tools with side effects idempotent (an idempotency key on every call): retries are inevitable.

3. Save memory to Redis between every step.

   Two kinds of memory. The state of the current run: a checkpoint after every step, so that after a worker crash you continue from the same place rather than from the start. Long-term memory: facts across sessions. In LangGraph this is a checkpointer with a Redis or Postgres backend. Redis for hot state with a TTL, Postgres for durable history and audit.

4. Add a human-in-the-loop checkpoint for risky actions.

   Split the tools by risk: reads (free), reversible writes (log them), irreversible or external actions (a payment, sending an email, a deletion, a deploy): pause and confirm. The graph stops, the state is saved, the human gets a request with specifics (what, with which parameters, why), and after the answer the graph continues from the same point. In LangGraph this is interrupt together with a checkpointer. A queue like Celery is needed because a run can wait for a human for hours and must not hold an HTTP request open.

5. Log every step so you can replay and debug failures.

   For every step: the input state, the model’s prompt and response, tokens and cost, tool calls with arguments and results, latency, the reflect decision. Replay: restore the state from a checkpoint and rerun it with the recorded tool responses, otherwise the outside world has already changed and the bug does not reproduce. The tracing standard: OpenTelemetry, one trace per run, one span per step.

Key ideas

• Tool results are data, not instructions: a web page or an email saying “ignore previous instructions” must not change the agent’s behavior.
• Errors compound: 95% reliability per step gives about 60% over ten steps. Fewer steps, with checks between them.
• Multiple agents for their own sake add coordination and points of failure, not quality.

Code

type Step string

const (
	StepPlan    Step = "plan"
	StepAct     Step = "act"
	StepReflect Step = "reflect"
	StepApprove Step = "await_approval"
	StepDone    Step = "done"
)

type RunState struct {
	RunID     string
	Goal      string
	Plan      []string
	History   []StepRecord
	Iteration int
	SpentUSD  float64
	Next      Step
}

func (o *Orchestrator) Run(ctx context.Context, s *RunState) error {
	for s.Next != StepDone && s.Next != StepApprove {
		if s.Iteration >= o.MaxIterations || s.SpentUSD >= o.BudgetUSD {
			return ErrBudgetExceeded
		}
		var err error
		switch s.Next {
		case StepPlan:
			err = o.plan(ctx, s)
		case StepAct:
			err = o.act(ctx, s) // a risky tool sets s.Next = StepApprove
		case StepReflect:
			err = o.reflect(ctx, s)
		}
		if err != nil {
			return err
		}
		s.Iteration++
		if err := o.store.Checkpoint(ctx, s); err != nil { // after every step
			return err
		}
	}
	return nil
}

Pitfalls

• Infinite loops and cost explosion when there are no limits.
• Many tools with similar descriptions: the model gets confused about which to pick.
• A non-idempotent tool plus a retry: a double payment, two emails.
• Replay without recorded tool responses does not reproduce the bug.

How to tell it works

• Task success rate on a set of scenarios with expected outcomes (this is project 04, just for agents).
• The average number of steps and the cost per task.
• The share of runs that hit a limit.
• The share of escalations to a human and how many of them were actually needed.

On my stack

Temporal (it has a native Go SDK) covers steps 3-5 out of the box: durable execution, retries, waiting for a signal from a human even for days, a full event history for replay. LangGraph + Celery + a Redis checkpointer essentially assemble the same thing by hand. Kafka carries step events to analytics and audit. The agent’s tools: RAG from project 01 and Text-to-SQL from project 02.

How the projects fit into one system

The order in which to build them:

1. 01 RAG. The foundation: it teaches you retrieval and answers grounded in sources.
2. 04 Evals. Right after the first working pipeline, so that from then on you improve by the numbers, not by feel.
3. 02 Text-to-SQL. A second kind of retrieval (structured data) and a culture of guardrails.
4. 05 Semantic cache. Infrastructure work, quick, and it shows the money right away.
5. 06 Agent. Brings in 01 and 02 as tools and is unmanageable without evals.
6. 03 LoRA. A separate skill set (ML and GPUs). It makes sense once data has piled up from the logs and it is proven that prompting is not enough.

Cross-cutting rules

• The model proposes, code verifies.
• Measure first, optimize second.
• Log everything needed for reproduction: the model version, the prompt version, the context passed in, the tool responses.
• Count the cost of a single request from day one.
• Access permissions and scope apply before the model, not after it.
• Start with the simplest option and add complexity only when the numbers say it is time.

Glossary

Embedding

a vector that captures the meaning of a text; texts with similar meanings yield nearby vectors.

Dense / sparse retrieval

search by embeddings / search by word overlap (BM25).

BM25

a keyword ranking formula: it accounts for how often a word occurs in the document, how rare it is in the corpus, and the document’s length.

RRF (Reciprocal Rank Fusion)

merging several ranked lists by position: Σ 1 / (k + rank).

Reranker / cross-encoder

a model that scores a “query + document” pair as a whole; more accurate than embeddings, but slower.

Recall@k

the share of questions for which the right document made it into the first k results.

MRR, nDCG

ranking metrics that take into account the position of the correct answer.

Faithfulness (groundedness)

whether every claim in the answer is backed by sources.

AST

the parse tree of a query; it lets you check structure rather than text.

Schema linking

selecting the relevant tables and columns before generating SQL.

Execution accuracy

whether the result of the generated query matches the result of the reference one.

LoRA

fine-tuning through low-rank additions to the weights while the base stays frozen.

QLoRA

LoRA on top of a base quantized to 4 bits.

NF4

a 4-bit quantization format designed for normally distributed weights.

Rank (r) and alpha

the adapter’s capacity and the scale of its contribution.

Catastrophic forgetting

the loss of a model’s general abilities after narrow fine-tuning.

Golden set

a human-reviewed set of inputs with references or rubrics.

LLM-as-judge

scoring an answer with another model against a rubric.

Cosine similarity

the cosine of the angle between vectors, a measure of semantic similarity.

HNSW

a graph index for approximate nearest neighbor search, the standard in vector databases.

Prompt caching

a cache of the prompt prefix on the provider side; the model is still called, but a repeat is cheaper and faster.

Checkpointer

a mechanism that saves the agent’s state after every step.

Human-in-the-loop (HITL)

a pause in the system’s work until a human decides.

Idempotency key

a key that keeps a repeated call from performing the action a second time.

Durable execution

execution in which workflow progress survives process crashes (Temporal).

Source

Original: a carousel of six cards: Hybrid Search RAG over Internal Docs, Text-to-SQL with Guardrails, LoRA Fine-Tuning Pipeline, Automated Eval Dataset Generator, Semantic Caching Layer, Agent Orchestration System. Author: Bashiri Smith, Baswe AI.

Taken from the original: the project names, the stacks and the five steps of each project (the steps are translated and expanded).

Added: explanations of the steps, key ideas, code, pitfalls, metrics, the “On my stack” sections, the system diagram, the learning order, the glossary.

Further reading:

• Claude API: Citations
• Claude API: Prompt caching
• Claude API: Embeddings
• Claude API: Structured outputs
• Claude API: Strict tool use
• Claude API: Batch processing
• Anthropic: Building effective agents

Compiled in October 2026. Code is marked as a sketch where library APIs change between versions.