# CHUNK #8 — AI Provider Layer

**Duration:** ~2 days · **Prereqs:** CHUNK #2 (storage, `packages/core` token budget) · **Follows:** CHUNK #7 · **Unblocks:** CHUNK #9 (roles + suggestions build on this)

**Purpose:** Build the provider abstraction that talks to any OpenAI-compatible endpoint (local Ollama/llama.cpp/LM Studio/vLLM) plus Anthropic, Gemini, and OpenRouter via the Vercel AI SDK v5 — with streaming, thinking/reasoning params, and the token-budget engine that keeps every prompt inside the context window.

---

## Steps

### 1. Provider config (`packages/ai/src/providers/`)
1. **Provider type** (`packages/ai/src/providers/types.ts`):
   ```ts
   interface ProviderConfig {
     id: string;
     kind: 'openai-compatible' | 'anthropic' | 'google' | 'openrouter';
     baseUrl?: string;      // e.g. http://localhost:11434/v1, https://api.openai.com/v1
     apiKey?: string;       // optional for local
     model: string;
     temperature: number;
     maxTokens: number;
     thinking?: { enabled: boolean; budget?: number; effort?: 'low'|'medium'|'high' };
     contextWindow: number;
   }
   ```
2. **Discovery** (`packages/ai/src/providers/discover.ts`): auto-detect local providers by probing common endpoints (`http://localhost:11434/v1/models`, `http://localhost:1234/v1/models`). Show "detected local models" in settings.
3. **Storage**: providers persisted per-user (server) or per-machine (local) in the settings store.

### 2. Adapter layer (`packages/ai/src/adapters/`)
1. **OpenAI-compatible adapter** (`openaiCompatible.ts`): uses `@ai-sdk/openai` with a custom `apiBase` + `headers`. Handles streaming (`generateText`/`streamText`) and non-streaming.
2. **Anthropic adapter** (`anthropic.ts`): `@ai-sdk/anthropic`; maps `thinking.budget` → the SDK's `thinking` param.
3. **Google adapter** (`google.ts`): `@ai-sdk/google` (Gemini).
4. **OpenRouter adapter** (`openrouter.ts`): `@ai-sdk/openai-compatible` with `https://openrouter.ai/api/v1` base + special headers (`HTTP-Referer`, `X-Title`).
5. **Factory** (`getProvider(config)`): returns the right adapter; unknown `kind` throws a clear error.

### 3. Thinking / reasoning (`packages/ai/src/thinking.ts`)
1. Normalize `thinking` across providers:
   - o-series (OpenAI) → `reasoningEffort`.
   - Anthropic → `thinking` budget object.
   - Local (e.g. Qwen3 via Ollama) → extra body params / `chat_template_kwargs`.
2. Parse streamed reasoning tokens when present (strip from final content, expose separately for debugging).

### 4. Token budget (`packages/ai/src/budget.ts`)
1. **Count** (`countTokens(text)`): wrap `gpt-tokenizer` (o200k_base), lazy-loaded. Already stubbed in Chunk 2 — flesh out with real tokenizer.
2. **Allocate** (`allocateBudget({ system, context, history, output, total })`): returns per-slot token caps (defaults: system+role 10%, context pack 40%, history 30%, output 20%). Adjustable per provider config.
3. **Guard** (`enforceBudget(prompt, budget)`): throws or truncates if the assembled prompt exceeds the context window; logs which slot overflowed.

### 5. Streaming client (`packages/ai/src/client.ts`)
1. `streamCompletion(provider, messages, opts)` → async iterator of `{ type: 'text'|'reasoning'|'tool-call'|'done', text? }`.
2. `complete(provider, messages, opts)` → full text (for non-streaming tool use).
3. Error handling: 429 → retry with backoff; auth errors → clear message; timeout → abort.
4. **Usage tracking**: capture `promptTokens`/`completionTokens` per call → stored in `AISession` (Chunk 9) + shown in settings.

### 6. Tests
1. Unit: `countTokens` on known strings (compare to gpt-tokenizer reference).
2. Unit: `allocateBudget` + `enforceBudget` edge cases (empty context, overflow).
3. Unit: thinking-param normalization for each provider.
4. Integration (mock): a fake OpenAI-compatible server returns a streamed response; adapter yields correct chunks. Use `msw` or a local Express stub.

---

## Acceptance criteria (all must pass)
- [ ] Configure an OpenAI-compatible provider (local Ollama URL) → streaming works.
- [ ] Configure OpenAI/Anthropic/Gemini/OpenRouter with API keys → streaming works (mock keys in tests).
- [ ] Auto-detect a local model running on `:11434`.
- [ ] `thinking` enabled → correct provider param sent; reasoning tokens parsed when present.
- [ ] `countTokens` matches gpt-tokenizer within tolerance.
- [ ] Prompt exceeding context window → clear error, no crash.
- [ ] Streaming chunks arrive in order; usage tracked per call.
- [ ] CI green (unit + mocked integration).

## Definition of done
Every provider type is reachable through one streaming client, the token budget keeps prompts in-window, and local models are auto-discovered. Chunk 9 wires roles and suggestions on top.

## Risks & fallbacks
- **AI SDK v5 API churn** → pin exact versions; wrap all SDK calls in `packages/ai` so the app never imports `@vercel/ai` directly.
- **Local model quirks** (non-standard `/v1` paths, missing streaming) → tolerate missing `stream` by falling back to non-streaming; document required Ollama flags (`--jinja`, etc.).
- **Tokenizer drift** → gpt-tokenizer is the source of truth; re-run token tests when the model family changes.