Learning the system¶

Purpose. This document explains, from first principles, the concepts the rest of the codebase depends on — language models, tokens, embeddings, vector stores, LoRA, GGUF, quantization, and the evaluation gate. It is written as an educational reference: each machine-learning concept is introduced on its own terms and then related to an established software-engineering idea, so the complete ecosystem can be understood whether the goal is to operate the platform or to develop it. No prior exposure to machine-learning infrastructure is assumed; familiarity with ordinary backend systems is sufficient to follow every section.

Companion pages

Stack & decisions covers which components were chosen and why. This page covers how those components work. Read both for the full picture, or use the Architecture page as the map once the concepts are clear.

Most of this platform is built from conventional components: an HTTP service, a relational database, a job queue, and a background worker. The machine-learning-specific parts are comparatively few, and each one corresponds to a familiar infrastructure concept. The sections below introduce those parts individually, then assemble them into the full picture in §6.

1. What a language model is¶

Set aside the term "artificial intelligence" for a moment. Mechanically, a language model is a pure function:

f(text_so_far) → probability distribution over the next token

Given some text, the model returns how likely each possible "next chunk of text" is. Generating a sentence is therefore an iterative process: predict the next chunk, append it, and predict again. There is no database lookup and no separate reasoning engine — only a very large mathematical function evaluated repeatedly.

Comparable to predictive text, scaled up

A smartphone keyboard suggests the next word from the preceding few words. A language model performs the same operation, but conditioned on the entire conversation and trained on a very large corpus. "Generating a response" is that prediction run forward, one chunk at a time, until a designated "stop" token is produced.

The "very large function" is defined by its weights — billions of numbers fixed during training. Evaluating the function is called inference. A useful correspondence: compiling a program is expensive and performed once, whereas running it happens continuously. Training is analogous to compilation; inference is analogous to execution.

Tokens — the unit of text¶

Models do not operate on characters or whole words; they operate on tokens — sub-word fragments. A tokenizer splits text into these fragments, each mapped to an integer.

Common words are usually a single token: "the" is one token.
Rare or invented words are split into several. An invented word such as "Zorptania" might cost four tokens, because the model has never encountered it and must assemble it from smaller fragments.

Tokens resemble syllables; rare words resemble spelling aloud

A familiar word is read as a single unit, while an unfamiliar one is spelled out fragment by fragment. A model behaves the same way. This has practical consequences: token count drives cost, context limits, and — as §5 describes — how difficult an invented answer is to learn.

Every response in this platform reports prompt_tokens, completion_tokens, and total_tokens for precisely this reason: the token is the unit of accounting.

GGUF and quantization — the model as a deployable artifact¶

A model's weights must be stored in a file. This platform serves models in the GGUF format, executed by the llama-cpp engine.

GGUF is a single-file packaging of the weights together with metadata. It is comparable to a compiled binary of the model; llama-cpp is the runtime that loads and executes it, much as a virtual machine executes a compiled artifact.
Quantization stores each weight at lower numeric precision (for example, four bits instead of sixteen) to reduce file size and memory footprint, trading a small amount of accuracy for a large reduction in size. A model labeled Q4_K_M is quantized to four bits.

Quantization is comparable to lossy image compression

A raw photograph is large and pixel-perfect; a compressed (JPEG) version is a fraction of the size and nearly indistinguishable. Quantization applies the same principle to model weights: a three-billion-parameter model that would require roughly 12 GB at full precision fits in approximately 4 GB at four-bit precision, with only minor quality loss.

This is why a consumer GPU with 8 GB of memory can serve a model that nominally requires far more: it executes the compressed form rather than the full-precision one.

Inference: one chef, one cutting board¶

The llama-cpp model object maintains mutable internal state during generation (a "KV cache" — effectively a scratchpad that records the conversation so far). A single model instance is not safe to call concurrently.

One chef, one cutting board

Two cooks sharing a single cutting board mid-task will collide. Likewise, two requests reaching one model instance's scratchpad simultaneously corrupt each other, producing garbage output or a crash. The remedy is not to rewrite the engine but to require that callers take turns.

That is exactly what the implementation does: a per-model asyncio.Lock serializes generation on each model instance (adapta/core/model_manager.py, adapta/core/inference.py), generation itself runs on a bounded thread pool so it does not block the asynchronous event loop, and a timeout caps each call. The model cache is a bounded LRU, so a large number of endpoints cannot exhaust memory by each pinning a full model in RAM. None of this is specific to machine learning; it is ordinary concurrency engineering around a non-thread-safe resource, identical in spirit to guarding any shared mutable handle.

2. Two ways to specialize a model¶

A base model is a generalist, whereas most deployments require a specialist. There are two fundamentally different ways to obtain one, and conflating them is the most common conceptual error:

Question	Give it knowledge (RAG)	Change its behavior (adapter)
What it addresses	"What do the documents say?"	"How should the model act?"
Modifies the weights?	No	Yes (through an adapter)
Comparable to	An open-book examination	Practice until a behavior is habitual
Input	Documents	A LoRA adapter (imported and gated on CPU) and an eval set of prompt → response examples
Cost	Seconds, CPU	Training happens outside the platform; gating and serving are CPU
Reversibility	Delete a file and re-index	Swap the adapter

A working distinction

Facts the model should look up are a matter of knowledge (RAG). A manner in which the model should behave is a matter of an adapter. "Answer from a 500-page handbook" calls for RAG; "always reply in a terse, bulleted support voice" calls for an adapter.

3. Knowledge (RAG), from first principles¶

RAG stands for Retrieval-Augmented Generation. The model does not memorize the documents. Instead, at the moment a question is asked, the relevant passages are located and supplied to the model as context, and the model is asked to answer using them.

An open-book examination with a capable research assistant

Rather than memorizing a textbook, an examinee whose assistant instantly locates the three most relevant pages and places them in view can answer from what is on the page. The model is the examinee; retrieval is the assistant. The model's effective knowledge is whatever has been placed in front of it.

Making "locate the relevant passages" work requires two machine-learning building blocks.

Embeddings — coordinates for meaning¶

An embedding is a function that converts a piece of text into a vector (a list of roughly 384 numbers) such that texts with similar meaning are positioned near one another in that vector space.

Coordinates for meaning

Consider a map on which every sentence has a location and sentences about the same topic cluster together — "refund policy" and "money-back guarantee" end up adjacent even though they share no words. An embedding model assigns those coordinates. This is the opposite of a hash function, which deliberately scatters similar inputs to unrelated outputs; an embedding maps similar inputs to nearby outputs.

In this codebase, adapta/services/embeddings.py wraps a sentence-transformers model that produces these vectors, loaded lazily as a singleton.

Vector store — a database indexed by meaning¶

A conventional database index retrieves rows by exact value (for example, a B-tree on user_id). A vector store retrieves by nearest neighbor in meaning — "return the chunks whose embeddings are closest to this query's embedding." This platform uses ChromaDB, with one collection per project.

A catalog organized by topic rather than by title

A title index locates a book only when its exact name is known. A topic-organized catalog locates books about a subject, even when different words are used. The vector store is that topic catalog; the closeness search (approximate nearest-neighbor) is the act of walking to the correct shelf.

Hybrid retrieval — meaning and words¶

Nearest-neighbor search on embeddings has one predictable failure: it matches meaning, so an exact token that carries the whole question — an error code, a SKU, a surname — can be smoothed into the neighborhood of merely similar things and lost. The classical fix is the older technology: BM25, keyword scoring that rewards rare terms appearing often in a short passage.

The topic catalog and the index at the back of the book

The topic catalog finds books about a subject even in different words. The back-of-book index finds the exact term and nothing else. A researcher uses both, and trusts a passage that appears in both lists more than one that appears in either alone.

Retrieval here uses both, then adds a third opinion:

Vector search fetches a wide pool of candidates (top_k × 4).
BM25 scores that same pool by keyword relevance.
Reciprocal Rank Fusion merges the two rankings. It deliberately ignores the scores and uses only the positions — a cosine similarity and a BM25 score are not on a shared scale, but "third in one list, first in the other" is meaningful without any calibration.
A cross-encoder reranker re-scores the shortlist. The embedding model is a bi-encoder — it reads query and passage separately, which is what makes the vectors precomputable and search fast. A cross-encoder reads the pair together, so it can weigh them against each other. It is much more accurate and much slower, which is exactly why it runs last, on a handful of candidates rather than the whole collection.

The reranker is optional (ADAPTA_RAG_RERANKER_MODEL="" disables it, leaving the fused order), and the citation scores reported to the client come from whichever stage ranked last.

The complete RAG cycle¶

Combining these pieces (adapta/services/rag.py, adapta/services/chat.py):

INDEX TIME (once, when a file is uploaded):
  document → parse → split into chunks → embed each chunk → store vectors in Chroma

QUERY TIME (on every chat request):
  question → embed it → vector search in Chroma (top_k × 4 candidates)
           → BM25 score the same candidates → fuse both rankings (RRF)
           → cross-encoder rerank the shortlist → keep top_k
           → inject them as context
           → model generates an answer grounded in those chunks → return with citations

The citations follow directly from the mechanism: because the system selects which chunks to place in front of the model, it can return those same chunks. The model is not asked to invent sources; it is given them.

Two practical safeguards in the code are worth noting, both of which are ordinary engineering. Retrieval runs off the event loop and under a timeout, so a stalled vector store degrades to a 504 rather than freezing every request; and the assembled prompt is fitted to the model's context window, dropping the least-relevant chunks first if it would not otherwise fit, rather than silently truncating the question.

4. LoRA adapters, from first principles¶

Fine-tuning changes how a model behaves by adjusting its weights from examples. Fully retraining a model with billions of parameters is prohibitively expensive, however. LoRA (Low-Rank Adaptation) is the technique that makes the operation affordable.

Patches applied over the original, rather than rewriting it

Full fine-tuning rewrites the entire model to instill one new behavior, which is enormous and wasteful. LoRA leaves the original weights untouched and instead produces a small set of corrective adjustments applied on top of them. These adjustments — the "adapter" — are tiny, a few megabytes against a multi-gigabyte model, yet sufficient to shift behavior.

In software terms, the base model is a large read-only dependency, and a LoRA adapter is a patch or diff applied over it at load time — comparable to a plugin, a version-control patch, or a style override. The small patch is shipped, not a new copy of the whole model.

Bring your own adapter¶

Adapta does not train adapters: in-product training (and with it the CUDA worker) was removed on 2026-10-10. Training an adapter is done with a LoRA trainer elsewhere. What the platform owns is everything after: import the adapter (PEFT, converted to GGUF by adapta/core/adapter_conversion.py, or GGUF directly), gate it on CPU against the base on the project's eval set, and serve it with RAG. The import flow is live (E4.2–E4.4, 2026-10-10); the import guide says what an adapter must match.

The eval set — labeled examples¶

The eval set is an instruction dataset: a JSONL file in which each line is a {prompt, response} pair (optionally with a system message), validated by adapta/services/datasets.py against specs/schemas/eval_dataset.schema.json. It is labeled data — input paired with desired output — the same shape as any supervised-learning task. The platform can also synthesize one from indexed documents (converting chunks into question/answer pairs), removing the need to author it by hand.

The job queue and worker¶

Long-running work is never performed inside a request handler, for the same reason that resizing ten thousand images synchronously within an HTTP request would be unacceptable.

An order ticket on a kitchen rail

The app is the front of house taking orders: it writes a ticket (a job, pushed onto the Redis queue). The worker is the kitchen: it takes the next ticket (via BLPOP), looks up the handler for its kind, and updates the ticket's status. The front of house never blocks while the kitchen works.

The worker (the same image as app) keeps the loop — heartbeat, healthcheck, requeue of the in-flight job on SIGTERM — but its handler table is empty today; an unknown kind fails the job with a typed error.

5. The evaluation gate — continuous integration for a model¶

An adapter introduces a specific risk: it can silently make a model worse. Unlike a code change, the difference cannot be read in a diff — the change is a shift across billions of weights. The question, then, is how to determine whether an adapter is safe to serve.

The answer mirrors how code is determined to be safe to ship: a test suite is run, and deployment is refused if it fails. That is the role of the evaluation gate.

A taste test on a fresh portion before a dish leaves the kitchen

A cook does not judge a dish by the spoon used to prepare it but by a fresh portion. The gate scores the adapter on a held-out eval set it never trained on, comparable to assessing a student with questions that were not on the study sheet. Scoring on the training data would be equivalent to grading an examination with the answer key in view, which reveals nothing about generalization.

Status

The gate that shipped with in-product training was removed on 2026-10-10 (E4.1). Its CPU replacement scores adapter and base with a task metric (E4.3, services/eval_gate.py); the adapter job runs it on every import (E4.4). An adapter that fails it never backs an endpoint. The design below is the contract the rebuilt gate keeps.

How the score is computed:

Held-out rows. Only rows the adapter never trained on are scored.
Response-only loss. The score measures how well the model produces the target answer; the prompt tokens are masked, so the answer is graded rather than the question. The mask is the shared prefix of the tokenized prompt and the full text, so a single-token answer (a bare classification label) is still graded.
Score = exp(-loss). This converts average per-token "surprise" (perplexity) into a value between 0 and 1, where higher is better. A score of 1.0 indicates that the held-out answers were predicted perfectly.
The gate. An adapter is permitted to serve if either it clears an absolute bar (default 0.6) or it clears a low sanity floor and surpasses the base model on the same held-out split by a margin.

A gate is only meaningful if it scores the model on the conditioning it will actually be served with, so scoring and serving must render every prompt through the same function — adapta.core.chat_templates.render. Otherwise an adapter could pass the gate on a conditioning no request ever sends.

The reason for the second path is itself instructive about calibrating a gate to reality. A small base model cannot reach the absolute bar even on an ideal task (the invented answer tokens carry an irreducible per-token cost — see the discussion of tokens), yet an adapter that reliably outperforms its own starting point has demonstrably learned the target behavior. The gate therefore measures improvement, not only an absolute threshold, which makes it a stronger signal rather than a weaker one: a non-improving or degenerate adapter still fails both paths.

In practice the gate rewards the tasks LoRA is genuinely good at — a repeatable shape in the answer: structured extraction (text → a fixed JSON schema), classification (message → one label from a closed set), and fixed format/voice (replies in a house template). These are exactly the cases where the held-out answer is predictable enough for the response-only loss to drop. A dataset with no learnable pattern (every answer different, or facts the adapter would have to memorize) cannot move the held-out score and is correctly blocked. Two fixture adapters in tests/fixtures/adapters/ (classify-pass and garbage-control) are the pass and fail cases the rebuilt gate is checked against.

An adapter that fails the gate raises EvalGateFailed (422), is not registered, and cannot back an endpoint. This is the platform's central safety guarantee: an unverified adapter never serves.

6. The complete picture¶

With the preceding sections in place, the Architecture page can be read straightforwardly. The platform consists of:

a web service (app) performing CRUD operations over PostgreSQL, plus two specialized flows;
RAG, an open-book-examination model powered by embeddings (coordinates for meaning) and a vector store (a database indexed by meaning);
adapters, in which LoRA adapters — patches over the frozen base — are imported and gated by a continuous-integration-style evaluation before they may serve (import and gate: E4.2–E4.4); a Redis queue and worker handle background jobs;
a single OpenAI-compatible API for serving, with llama-cpp executing quantized GGUF models under a per-model lock.

Every machine-learning term in that summary now has a plain engineering meaning.

Operating versus developing the ecosystem¶

The same concepts support two activities, and the documentation separates them deliberately:

Operating the platform — creating projects, supplying documents or datasets, observing the evaluation gate, and issuing keys — is described in the User Guide. The operator's mental model is the left-hand column of the §2 table (knowledge vs. behavior) plus the gate in §5.
Developing the platform — changing the API, the schema, the adapter path, or the serving path — is described in the Workflow page and the Architecture page. The developer's mental model adds the internal mechanisms in §1, §3, and §4, and the contract-driven process that protects them.

Glossary — machine-learning term to engineering equivalent¶

Term	Engineering equivalent
Inference	Executing a compiled function; the hot path
Training	Compilation — expensive, performed offline (outside Adapta)
Token	A syllable or word fragment; the unit of text and of cost
Weights	The billions of constants that define the model function
GGUF	A compiled binary of the model; `llama-cpp` is its runtime
Quantization	Lossy compression of weights — smaller, slightly less accurate
Embedding	Coordinates for meaning (similar text yields nearby vectors)
Vector store (Chroma)	A database indexed by meaning rather than exact value
BM25	Classical keyword scoring — rewards rare terms in short passages
RRF	Merges two rankings by position, so their scores need no shared scale
Cross-encoder	Reads query and passage together; slow but accurate — used to rerank a shortlist
RAG	An open-book examination: retrieve the relevant passages, then answer
Chunk	One indexed passage of a document (a "page" placed in view)
Citation	The chunks supplied to the model, returned for transparency
Fine-tuning	Practice until a behavior is habitual (modifies weights); done elsewhere, then the adapter is imported
LoRA adapter	A small patch or diff applied over the frozen base weights
Instruction dataset	Labeled examples: `prompt → desired response` (here, the eval set)
Held-out split	The fresh portion that is tasted / the unseen examination questions
Perplexity	Average per-token "surprise"; lower indicates a better fit
Evaluation gate	Continuous integration for a model — it cannot ship unless it passes
Base model	A large read-only dependency on which every project builds
Endpoint / slug	A deployed model instance and the name applications use to call it

Related pages: Architecture for how these pieces are assembled, and Workflow for how they are changed safely.