Skip to content

Architecture overview

About these pages

The Internals section walks through how the llm package works inside, for contributors and the curious. The public API is documented under LLM; this section is about the implementation.

or/llm is a stateless translation layer. It decides what to send for one request and how to interpret the streamed response, and leaves history storage, context compaction, and tool-loop orchestration to the caller. The same conversation can target any model on any supported wire protocol, and the target can change between turns; the library re-adapts the history for each request.

Package layout

There is no separate "facade" and "core": the public types and the implementation live in one package, llm. Protocol adapters live in their own sub-packages and register themselves on import, so an application links only the vendor SDKs it actually uses.

Path Role
llm/ The whole neutral core and public API: models, messages, options, streaming, transform, the adapter and provider registries, and the default client
llm/openai/ The openai-completions and openai-responses adapters; registers both from init
llm/openai/internal/chatcompletions/ Chat Completions request conversion, compatibility dialects, and stream state
llm/openai/internal/responses/ Responses input-item conversion and event state machine
llm/openai/internal/transport/ HTTP client setup, request hooks, and shared SSE filtering
llm/anthropic/ The anthropic-messages adapter; registers itself from init
llm/all/ Blank-imports both provider packages, for callers that want every built-in protocol
llm/internal/ jsonx (lenient JSON helpers) and genmodels (the catalog generator)

An adapter is pulled in for its side effects:

import (
    "github.com/ktsoator/or/llm"
    _ "github.com/ktsoator/or/llm/anthropic" // registers anthropic-messages
)

The registries, adapter, and client

Dispatch is built from a few small pieces, all in the core package:

  • ProtocolAdapter — an interface with Protocol() (its registry key) and Stream() (the request lifecycle for one protocol). See adapters.
  • AdapterRegistry — a concurrency-safe map[Protocol]ProtocolAdapter. Provider init functions call llm.Register to add themselves to the package default registry; a caller that prefers explicit wiring builds its own with NewAdapterRegistry and AdapterRegistry.Register.
  • ProviderRegistry — a concurrency-safe map of per-vendor configuration: credential sources, static headers, and any ProviderOverride. Its ResolveRequest fills the API key and applies overrides to each request before dispatch; the default is NewBuiltInProviderRegistry, populated from the catalog. See providers.
  • Client — holds both registries and routes each request: it resolves provider configuration through the ProviderRegistry, then dispatches to the adapter for the model's protocol. llm.Stream and llm.Complete are thin wrappers over a default client bound to the default registries.
flowchart LR
    subgraph core["package llm"]
        R["AdapterRegistry"]
        P["ProviderRegistry"]
        C["Client"]
    end
    OA["llm/openai · init()"] -->|Register| R
    AN["llm/anthropic · init()"] -->|Register| R
    C -->|"ResolveRequest(model, options)"| P
    C -->|"Get(model.Protocol)"| R

Request data flow

flowchart TD
    A["llm.Complete / Stream"] --> B["Client.Stream"]
    B --> V["options.Validate(protocol, tools)"]
    V --> RR["providers.ResolveRequest<br/>key · override · headers"]
    RR --> C{"adapters.Get(model.Protocol)"}
    C -->|anthropic-messages| D["Anthropic adapter"]
    C -->|openai-completions| E["OpenAI adapter"]
    C -->|openai-responses| F["OpenAI Responses adapter"]
    D --> T["TransformMessages → convert → SDK request"]
    E --> T
    F --> T
    T --> G["StreamWriter: Emit / Done / Fail"]
    G --> H["chan Event → caller"]

The Protocol field on the model is the discriminator: Client.Stream uses it to pick an adapter from the registry. Everything left of the adapter is provider-neutral; everything inside it may speak one concrete wire protocol.

Reading a request end to end

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
func (c *Client) Stream(ctx context.Context, model Model, input Context, options StreamOptions) (<-chan Event, error) {
    if c.adapters == nil {
        return nil, errors.New("adapter registry is nil")
    }
    if err := options.Validate(model.Protocol, input.Tools); err != nil { // (1)!
        return nil, err
    }

    // A nil provider registry still resolves the legacy environment API key.
    model, options = c.providers.ResolveRequest(model, options) // (2)!

    adapter, ok := c.adapters.Get(model.Protocol) // (3)!
    if !ok {
        return nil, fmt.Errorf(
            "no adapter registered for protocol %q",
            model.Protocol,
        )
    }

    return adapter.Stream(ctx, model, input, options)
}
  1. Protocol-specific options are checked against the target protocol before any HTTP request is built, so a mismatch fails fast.
  2. ProviderRegistry.ResolveRequest fills the API key — from StreamOptions, a provider override, or the provider's environment variables, in that order — and applies any per-provider base-URL and header override. A model whose provider is not registered falls back to the legacy environment lookup.
  3. Protocol selects the adapter. The same conversation can target either protocol; the library re-adapts the history per request.

Source: llm/client.go, llm/adapters.go, llm/default.go.

Where to go next