Schwitzz
79 lines
5.1 KiB
Markdown
79 lines
5.1 KiB
Markdown
# CHUNK #8 — AI Provider Layer
|
|
|
|
**Duration:** ~2 days · **Prereqs:** CHUNK #2 (storage, `packages/core` token budget) · **Follows:** CHUNK #7 · **Unblocks:** CHUNK #9 (roles + suggestions build on this)
|
|
|
|
**Purpose:** Build the provider abstraction that talks to any OpenAI-compatible endpoint (local Ollama/llama.cpp/LM Studio/vLLM) plus Anthropic, Gemini, and OpenRouter via the Vercel AI SDK v5 — with streaming, thinking/reasoning params, and the token-budget engine that keeps every prompt inside the context window.
|
|
|
|
---
|
|
|
|
## Steps
|
|
|
|
### 1. Provider config (`packages/ai/src/providers/`)
|
|
1. **Provider type** (`packages/ai/src/providers/types.ts`):
|
|
```ts
|
|
interface ProviderConfig {
|
|
id: string;
|
|
kind: 'openai-compatible' | 'anthropic' | 'google' | 'openrouter';
|
|
baseUrl?: string; // e.g. http://localhost:11434/v1, https://api.openai.com/v1
|
|
apiKey?: string; // optional for local
|
|
model: string;
|
|
temperature: number;
|
|
maxTokens: number;
|
|
thinking?: { enabled: boolean; budget?: number; effort?: 'low'|'medium'|'high' };
|
|
contextWindow: number;
|
|
}
|
|
```
|
|
2. **Discovery** (`packages/ai/src/providers/discover.ts`): auto-detect local providers by probing common endpoints (`http://localhost:11434/v1/models`, `http://localhost:1234/v1/models`). Show "detected local models" in settings.
|
|
3. **Storage**: providers persisted per-user (server) or per-machine (local) in the settings store.
|
|
|
|
### 2. Adapter layer (`packages/ai/src/adapters/`)
|
|
1. **OpenAI-compatible adapter** (`openaiCompatible.ts`): uses `@ai-sdk/openai` with a custom `apiBase` + `headers`. Handles streaming (`generateText`/`streamText`) and non-streaming.
|
|
2. **Anthropic adapter** (`anthropic.ts`): `@ai-sdk/anthropic`; maps `thinking.budget` → the SDK's `thinking` param.
|
|
3. **Google adapter** (`google.ts`): `@ai-sdk/google` (Gemini).
|
|
4. **OpenRouter adapter** (`openrouter.ts`): `@ai-sdk/openai-compatible` with `https://openrouter.ai/api/v1` base + special headers (`HTTP-Referer`, `X-Title`).
|
|
5. **Factory** (`getProvider(config)`): returns the right adapter; unknown `kind` throws a clear error.
|
|
|
|
### 3. Thinking / reasoning (`packages/ai/src/thinking.ts`)
|
|
1. Normalize `thinking` across providers:
|
|
- o-series (OpenAI) → `reasoningEffort`.
|
|
- Anthropic → `thinking` budget object.
|
|
- Local (e.g. Qwen3 via Ollama) → extra body params / `chat_template_kwargs`.
|
|
2. Parse streamed reasoning tokens when present (strip from final content, expose separately for debugging).
|
|
|
|
### 4. Token budget (`packages/ai/src/budget.ts`)
|
|
1. **Count** (`countTokens(text)`): wrap `gpt-tokenizer` (o200k_base), lazy-loaded. Already stubbed in Chunk 2 — flesh out with real tokenizer.
|
|
2. **Allocate** (`allocateBudget({ system, context, history, output, total })`): returns per-slot token caps (defaults: system+role 10%, context pack 40%, history 30%, output 20%). Adjustable per provider config.
|
|
3. **Guard** (`enforceBudget(prompt, budget)`): throws or truncates if the assembled prompt exceeds the context window; logs which slot overflowed.
|
|
|
|
### 5. Streaming client (`packages/ai/src/client.ts`)
|
|
1. `streamCompletion(provider, messages, opts)` → async iterator of `{ type: 'text'|'reasoning'|'tool-call'|'done', text? }`.
|
|
2. `complete(provider, messages, opts)` → full text (for non-streaming tool use).
|
|
3. Error handling: 429 → retry with backoff; auth errors → clear message; timeout → abort.
|
|
4. **Usage tracking**: capture `promptTokens`/`completionTokens` per call → stored in `AISession` (Chunk 9) + shown in settings.
|
|
|
|
### 6. Tests
|
|
1. Unit: `countTokens` on known strings (compare to gpt-tokenizer reference).
|
|
2. Unit: `allocateBudget` + `enforceBudget` edge cases (empty context, overflow).
|
|
3. Unit: thinking-param normalization for each provider.
|
|
4. Integration (mock): a fake OpenAI-compatible server returns a streamed response; adapter yields correct chunks. Use `msw` or a local Express stub.
|
|
|
|
---
|
|
|
|
## Acceptance criteria (all must pass)
|
|
- [ ] Configure an OpenAI-compatible provider (local Ollama URL) → streaming works.
|
|
- [ ] Configure OpenAI/Anthropic/Gemini/OpenRouter with API keys → streaming works (mock keys in tests).
|
|
- [ ] Auto-detect a local model running on `:11434`.
|
|
- [ ] `thinking` enabled → correct provider param sent; reasoning tokens parsed when present.
|
|
- [ ] `countTokens` matches gpt-tokenizer within tolerance.
|
|
- [ ] Prompt exceeding context window → clear error, no crash.
|
|
- [ ] Streaming chunks arrive in order; usage tracked per call.
|
|
- [ ] CI green (unit + mocked integration).
|
|
|
|
## Definition of done
|
|
Every provider type is reachable through one streaming client, the token budget keeps prompts in-window, and local models are auto-discovered. Chunk 9 wires roles and suggestions on top.
|
|
|
|
## Risks & fallbacks
|
|
- **AI SDK v5 API churn** → pin exact versions; wrap all SDK calls in `packages/ai` so the app never imports `@vercel/ai` directly.
|
|
- **Local model quirks** (non-standard `/v1` paths, missing streaming) → tolerate missing `stream` by falling back to non-streaming; document required Ollama flags (`--jinja`, etc.).
|
|
- **Tokenizer drift** → gpt-tokenizer is the source of truth; re-run token tests when the model family changes.
|