PubScript/CHUNK#8.md

CHUNK #8 — AI Provider Layer

Duration: ~2 days · Prereqs: CHUNK #2 (storage, packages/core token budget) · Follows: CHUNK #7 · Unblocks: CHUNK #9 (roles + suggestions build on this)

Purpose: Build the provider abstraction that talks to any OpenAI-compatible endpoint (local Ollama/llama.cpp/LM Studio/vLLM) plus Anthropic, Gemini, and OpenRouter via the Vercel AI SDK v5 — with streaming, thinking/reasoning params, and the token-budget engine that keeps every prompt inside the context window.

---

Steps

1. Provider config (packages/ai/src/providers/)

1. Provider type (packages/ai/src/providers/types.ts):

   interface ProviderConfig {
     id: string;
     kind: 'openai-compatible' | 'anthropic' | 'google' | 'openrouter';
     baseUrl?: string;      // e.g. http://localhost:11434/v1, https://api.openai.com/v1
     apiKey?: string;       // optional for local
     model: string;
     temperature: number;
     maxTokens: number;
     thinking?: { enabled: boolean; budget?: number; effort?: 'low'|'medium'|'high' };
     contextWindow: number;
   }
   

2. Discovery (packages/ai/src/providers/discover.ts): auto-detect local providers by probing common endpoints (http://localhost:11434/v1/models, http://localhost:1234/v1/models). Show "detected local models" in settings.
3. Storage: providers persisted per-user (server) or per-machine (local) in the settings store.

2. Adapter layer (packages/ai/src/adapters/)

1. OpenAI-compatible adapter (openaiCompatible.ts): uses @ai-sdk/openai with a custom apiBase + headers. Handles streaming (generateText/streamText) and non-streaming.
2. Anthropic adapter (anthropic.ts): @ai-sdk/anthropic; maps thinking.budget → the SDK's thinking param.
3. Google adapter (google.ts): @ai-sdk/google (Gemini).
4. OpenRouter adapter (openrouter.ts): @ai-sdk/openai-compatible with https://openrouter.ai/api/v1 base + special headers (HTTP-Referer, X-Title).
5. Factory (getProvider(config)): returns the right adapter; unknown kind throws a clear error.

3. Thinking / reasoning (packages/ai/src/thinking.ts)

1. Normalize thinking across providers:
   • o-series (OpenAI) → reasoningEffort.
   • Anthropic → thinking budget object.
   • Local (e.g. Qwen3 via Ollama) → extra body params / chat_template_kwargs.
2. Parse streamed reasoning tokens when present (strip from final content, expose separately for debugging).

4. Token budget (packages/ai/src/budget.ts)

1. Count (countTokens(text)): wrap gpt-tokenizer (o200k_base), lazy-loaded. Already stubbed in Chunk 2 — flesh out with real tokenizer.
2. Allocate (allocateBudget({ system, context, history, output, total })): returns per-slot token caps (defaults: system+role 10%, context pack 40%, history 30%, output 20%). Adjustable per provider config.
3. Guard (enforceBudget(prompt, budget)): throws or truncates if the assembled prompt exceeds the context window; logs which slot overflowed.

5. Streaming client (packages/ai/src/client.ts)

1. streamCompletion(provider, messages, opts) → async iterator of { type: 'text'|'reasoning'|'tool-call'|'done', text? }.
2. complete(provider, messages, opts) → full text (for non-streaming tool use).
3. Error handling: 429 → retry with backoff; auth errors → clear message; timeout → abort.
4. Usage tracking: capture promptTokens/completionTokens per call → stored in AISession (Chunk 9) + shown in settings.

6. Tests

1. Unit: countTokens on known strings (compare to gpt-tokenizer reference).
2. Unit: allocateBudget + enforceBudget edge cases (empty context, overflow).
3. Unit: thinking-param normalization for each provider.
4. Integration (mock): a fake OpenAI-compatible server returns a streamed response; adapter yields correct chunks. Use msw or a local Express stub.

---

Acceptance criteria (all must pass)

• Configure an OpenAI-compatible provider (local Ollama URL) → streaming works.
• Configure OpenAI/Anthropic/Gemini/OpenRouter with API keys → streaming works (mock keys in tests).
• Auto-detect a local model running on :11434.
• thinking enabled → correct provider param sent; reasoning tokens parsed when present.
• countTokens matches gpt-tokenizer within tolerance.
• Prompt exceeding context window → clear error, no crash.
• Streaming chunks arrive in order; usage tracked per call.
• CI green (unit + mocked integration).

Definition of done

Every provider type is reachable through one streaming client, the token budget keeps prompts in-window, and local models are auto-discovered. Chunk 9 wires roles and suggestions on top.

Risks & fallbacks

• AI SDK v5 API churn → pin exact versions; wrap all SDK calls in packages/ai so the app never imports @vercel/ai directly.
• Local model quirks (non-standard /v1 paths, missing streaming) → tolerate missing stream by falling back to non-streaming; document required Ollama flags (--jinja, etc.).
• Tokenizer drift → gpt-tokenizer is the source of truth; re-run token tests when the model family changes.