mirror of
https://github.com/Gitlawb/openclaude.git
synced 2026-08-24 02:34:15 -05:00
fix(ollama): preserve chat history with native context (#1805)
* fix(ollama): preserve chat history with native context Route Ollama chat requests through the native /api/chat endpoint so OpenClaude can send request-level options.num_ctx instead of relying on Ollama's OpenAI-compatible shim. Default the Ollama request context to 32768 tokens, support OPENCLAUDE_OLLAMA_NUM_CTX and OLLAMA_CONTEXT_LENGTH overrides, and map max tokens/temperature/top_p into native Ollama options. Adapt native Ollama streaming and non-streaming responses back into the existing OpenAI-shaped conversion pipeline, including usage, text, structured tool calls, and tool_use stop reasons. Normalize native Ollama request messages for images and historical tool calls, avoiding OpenAI-only image_url/id/type payload fields in /api/chat requests. Add Ollama context diagnostics, loopback-only ollama ps status checks, regression coverage, and documentation for verifying active context length. * fix(ollama): address native routing review feedback * fix(ollama): restrict loopback host matching * fix(ollama): exclude wildcard bind address * fix(ollama): keep https localhost proxies on chat completions --------- Co-authored-by: jatmn <12479882+jatmn@users.noreply.github.com>
This commit is contained in:
@@ -198,6 +198,13 @@ $env:OPENAI_MODEL="qwen2.5-coder:7b"
|
||||
openclaude
|
||||
```
|
||||
|
||||
For Ollama, OpenClaude uses Ollama's native chat API and requests a 32768-token
|
||||
context window on each chat request so same-session history is not silently
|
||||
truncated by Ollama's OpenAI-compatible shim. Set `OPENCLAUDE_OLLAMA_NUM_CTX`
|
||||
or `OLLAMA_CONTEXT_LENGTH` if you need a different request-level context size.
|
||||
See [Advanced Setup](docs/advanced-setup.md#ollama-context-length) for
|
||||
verification with `ollama ps`.
|
||||
|
||||
## Setup Guides
|
||||
|
||||
Beginner-friendly guides:
|
||||
|
||||
@@ -140,6 +140,53 @@ export OPENAI_BASE_URL=http://localhost:11434/v1
|
||||
export OPENAI_MODEL=llama3.3:70b
|
||||
```
|
||||
|
||||
#### Ollama Context Length
|
||||
|
||||
OpenClaude sends the current conversation history to Ollama on each turn and
|
||||
uses Ollama's native chat API for Ollama endpoints. Native chat lets OpenClaude
|
||||
send `options.num_ctx` with each request, so Ollama receives a 32768-token
|
||||
context window by default instead of falling back to the smaller context often
|
||||
used by Ollama's OpenAI-compatible `/v1/chat/completions` shim.
|
||||
|
||||
To choose a different request-level context size, set
|
||||
`OPENCLAUDE_OLLAMA_NUM_CTX` before launching OpenClaude:
|
||||
|
||||
```bash
|
||||
export OPENCLAUDE_OLLAMA_NUM_CTX=65536
|
||||
```
|
||||
|
||||
You can also start Ollama with a global context length:
|
||||
|
||||
macOS / Linux:
|
||||
|
||||
```bash
|
||||
# Stop any existing Ollama app/server first, then run:
|
||||
OLLAMA_CONTEXT_LENGTH=32768 ollama serve
|
||||
```
|
||||
|
||||
Windows PowerShell:
|
||||
|
||||
```powershell
|
||||
# Quit any existing Ollama app/server first, then run:
|
||||
$env:OLLAMA_CONTEXT_LENGTH="32768"
|
||||
ollama serve
|
||||
```
|
||||
|
||||
After a chat request, verify the loaded model is using the requested context:
|
||||
|
||||
```bash
|
||||
ollama ps
|
||||
```
|
||||
|
||||
Check the `CONTEXT` column. If it still shows a small value such as `4K` after a
|
||||
new OpenClaude request, stop the existing Ollama app/server, start it again, and
|
||||
retry the request.
|
||||
|
||||
Use a concrete recall test after changing the setting, such as asking the model
|
||||
to repeat the first topic from the current chat. Questions like "do you remember our
|
||||
conversation?" can trigger generic local-model disclaimers even when history is
|
||||
present.
|
||||
|
||||
### Atomic Chat (local, Apple Silicon)
|
||||
|
||||
```bash
|
||||
@@ -329,6 +376,7 @@ The **OpenClaude VS Code extension** can store the key in Secret Storage and set
|
||||
| `OPENAI_MODEL` | OpenAI-compatible only | Model name such as `gpt-4o`, `deepseek-v4-flash`, or `llama3.3:70b` |
|
||||
| `OPENAI_BASE_URL` | No | API endpoint, defaulting to `https://api.openai.com/v1` |
|
||||
| `OPENAI_API_BASE` | No | Compatibility alias for `OPENAI_BASE_URL` |
|
||||
| `OPENCLAUDE_OLLAMA_NUM_CTX` | Ollama only | Request-level Ollama context window. Defaults to `32768`; set a larger value for longer same-session history if your model and hardware can handle it. |
|
||||
| `CLAUDE_CODE_OPENAI_CONTEXT_WINDOWS` | No | JSON map of OpenAI-compatible model names to context windows, such as `{"custom-model":1000000}`. Use this when a custom provider does not expose context metadata from `/v1/models`. |
|
||||
| `OPENCODE_API_KEY` | OpenCode Zen / Go | Shared API key for OpenCode Zen (pay-as-you-go) and OpenCode Go (subscription); get yours from https://opencode.ai |
|
||||
| `MIMO_API_KEY` | Xiaomi MiMo route | Xiaomi MiMo API key for `https://api.xiaomimimo.com/v1`; mirrored into the OpenAI-compatible auth env when the MiMo route is active |
|
||||
|
||||
@@ -72,6 +72,19 @@ openclaude
|
||||
|
||||
No API key is needed for Ollama local models.
|
||||
|
||||
OpenClaude asks Ollama for a 32768-token context window on each chat request.
|
||||
If you need a different size, set `OPENCLAUDE_OLLAMA_NUM_CTX` before launching
|
||||
OpenClaude, or start Ollama with a global context setting:
|
||||
|
||||
```bash
|
||||
# Stop any existing Ollama app/server first, then run:
|
||||
OLLAMA_CONTEXT_LENGTH=32768 ollama serve
|
||||
```
|
||||
|
||||
After a chat request, run `ollama ps` in another terminal and check the
|
||||
`CONTEXT` column. It should show the requested size. If it still shows a small
|
||||
value such as `4K`, restart the Ollama app/server and try again.
|
||||
|
||||
### Option D: LM Studio
|
||||
|
||||
Install LM Studio first from:
|
||||
@@ -132,6 +145,8 @@ Check the basics:
|
||||
- make sure Ollama is installed
|
||||
- make sure Ollama is running
|
||||
- make sure the model was pulled successfully
|
||||
- if same-session chat history appears missing, verify the active `CONTEXT`
|
||||
value with `ollama ps`; OpenClaude requests 32K by default
|
||||
|
||||
### For LM Studio
|
||||
|
||||
|
||||
@@ -68,6 +68,20 @@ openclaude
|
||||
|
||||
No API key is needed for Ollama local models.
|
||||
|
||||
OpenClaude asks Ollama for a 32768-token context window on each chat request.
|
||||
If you need a different size, set `OPENCLAUDE_OLLAMA_NUM_CTX` before launching
|
||||
OpenClaude, or start Ollama with a global context setting:
|
||||
|
||||
```powershell
|
||||
# Quit any existing Ollama app/server first, then run:
|
||||
$env:OLLAMA_CONTEXT_LENGTH="32768"
|
||||
ollama serve
|
||||
```
|
||||
|
||||
After a chat request, run `ollama ps` in another PowerShell window and check the
|
||||
`CONTEXT` column. It should show the requested size. If it still shows a small
|
||||
value such as `4K`, restart the Ollama app/server and try again.
|
||||
|
||||
### Option D: LM Studio
|
||||
|
||||
Install LM Studio first from:
|
||||
@@ -145,6 +159,8 @@ Check the basics:
|
||||
- make sure Ollama is installed
|
||||
- make sure Ollama is running
|
||||
- make sure the model was pulled successfully
|
||||
- if same-session chat history appears missing, verify the active `CONTEXT`
|
||||
value with `ollama ps`; OpenClaude requests 32K by default
|
||||
|
||||
### For LM Studio
|
||||
|
||||
|
||||
@@ -82,6 +82,23 @@ describe('Session timeout fix', () => {
|
||||
})
|
||||
})
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Fix 2b: Ollama context history preservation
|
||||
// ---------------------------------------------------------------------------
|
||||
describe('Ollama context history fix', () => {
|
||||
test('openaiShim uses native Ollama chat with request-level num_ctx', async () => {
|
||||
const content = await file('services/api/openaiShim.ts').text()
|
||||
|
||||
expect(content).toContain('buildOllamaChatUrl')
|
||||
expect(content).toContain('/api/chat')
|
||||
expect(content).toContain('useNativeOllamaChat')
|
||||
expect(content).toContain('num_ctx: getOllamaNumCtx()')
|
||||
expect(content).toContain('normalizeOllamaNativeMessages(body.messages)')
|
||||
expect(content).toContain('convertOllamaStreamingResponse')
|
||||
expect(content).toContain('convertOllamaNonStreamingResponse')
|
||||
})
|
||||
})
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Fix 3: Agent loop continuation nudge
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -12,6 +12,8 @@ import {
|
||||
buildLocalModelContextLoad,
|
||||
checkLocalModelContextLoad,
|
||||
isActiveProviderLocalModel,
|
||||
isLoopbackOllamaEndpoint,
|
||||
parseOllamaPsContextWarning,
|
||||
} from '../utils/statusNoticeLocalModel.js'
|
||||
|
||||
const emptyPermissionContext = async () => getEmptyToolPermissionContext()
|
||||
@@ -167,6 +169,89 @@ describe('buildLocalModelContextLoad', () => {
|
||||
'planner: ~9,000 tokens',
|
||||
])
|
||||
})
|
||||
|
||||
test('includes Ollama context length contributor without other warnings', () => {
|
||||
const result = buildLocalModelContextLoad(null, [
|
||||
{
|
||||
id: 'ollama_context_length',
|
||||
message: 'Ollama context length is too small',
|
||||
details: ['llama3.1:8b: active CONTEXT is 4K'],
|
||||
summary: 'Ollama CONTEXT: 4K (OpenClaude requests 32K)',
|
||||
},
|
||||
])
|
||||
|
||||
expect(result?.lines).toEqual([
|
||||
'Ollama CONTEXT: 4K (OpenClaude requests 32K)',
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('parseOllamaPsContextWarning', () => {
|
||||
test('warns when a loaded Ollama model has a small active context', () => {
|
||||
const result = parseOllamaPsContextWarning(`
|
||||
NAME ID SIZE PROCESSOR UNTIL CONTEXT
|
||||
llama3.1:8b 46e0c10c039e 6.7 GB 100% GPU 4 minutes from now 4K
|
||||
`)
|
||||
|
||||
expect(result).toMatchObject({
|
||||
id: 'ollama_context_length',
|
||||
summary: 'Ollama CONTEXT: 4K (OpenClaude requests 32K)',
|
||||
})
|
||||
expect(result?.details.join('\n')).toContain('OpenClaude requests 32768')
|
||||
})
|
||||
|
||||
test('does not warn when the active context is already large enough', () => {
|
||||
const result = parseOllamaPsContextWarning(`
|
||||
NAME ID SIZE PROCESSOR UNTIL CONTEXT
|
||||
llama3.1:8b 46e0c10c039e 6.7 GB 100% GPU 4 minutes from now 32K
|
||||
`)
|
||||
|
||||
expect(result).toBeNull()
|
||||
})
|
||||
|
||||
test('only warns for the active model when multiple Ollama models are loaded', () => {
|
||||
const output = `
|
||||
NAME ID SIZE PROCESSOR UNTIL CONTEXT
|
||||
tinyllama:1b 46e0c10c039e 1.1 GB 100% GPU 4 minutes from now 4K
|
||||
llama3.1:8b 7e0c10c039e46 6.7 GB 100% GPU 4 minutes from now 32K
|
||||
`
|
||||
|
||||
expect(parseOllamaPsContextWarning(output, 'llama3.1:8b')).toBeNull()
|
||||
expect(parseOllamaPsContextWarning(output, 'tinyllama:1b')).toMatchObject({
|
||||
summary: 'Ollama CONTEXT: 4K (OpenClaude requests 32K)',
|
||||
})
|
||||
})
|
||||
|
||||
test('ignores older ollama ps output without a context column', () => {
|
||||
const result = parseOllamaPsContextWarning(`
|
||||
NAME ID SIZE PROCESSOR UNTIL
|
||||
llama3.1:8b 46e0c10c039e 6.7 GB 100% GPU 4 minutes from now
|
||||
`)
|
||||
|
||||
expect(result).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
describe('isLoopbackOllamaEndpoint', () => {
|
||||
test('allows local Ollama endpoints for local ollama ps diagnostics', () => {
|
||||
expect(isLoopbackOllamaEndpoint('http://localhost:11434/v1')).toBe(true)
|
||||
expect(isLoopbackOllamaEndpoint('http://127.0.0.1:11434/v1')).toBe(true)
|
||||
expect(isLoopbackOllamaEndpoint('http://[::1]:11434/v1')).toBe(true)
|
||||
})
|
||||
|
||||
test('skips remote Ollama endpoints so local ollama ps is not misleading', () => {
|
||||
expect(isLoopbackOllamaEndpoint('http://10.0.0.5:11434/v1')).toBe(false)
|
||||
expect(isLoopbackOllamaEndpoint('http://0.0.0.0:11434/v1')).toBe(false)
|
||||
expect(isLoopbackOllamaEndpoint('http://ollama.lan:11434/v1')).toBe(false)
|
||||
expect(isLoopbackOllamaEndpoint('https://localhost:11434/v1')).toBe(false)
|
||||
expect(isLoopbackOllamaEndpoint('https://127.0.0.1:11434/v1')).toBe(false)
|
||||
expect(isLoopbackOllamaEndpoint('https://ollama.example.com/v1')).toBe(false)
|
||||
expect(isLoopbackOllamaEndpoint('http://127.0.0.1.nip.io:11434/v1')).toBe(false)
|
||||
})
|
||||
|
||||
test('skips localhost proxies whose path merely contains ollama', () => {
|
||||
expect(isLoopbackOllamaEndpoint('http://localhost:8080/ollama/v1')).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('checkLocalModelContextLoad', () => {
|
||||
|
||||
@@ -45,6 +45,7 @@ export function StatusNotices(t0) {
|
||||
const [memoryFiles, setMemoryFiles] = React.useState(cachedMemoryFiles);
|
||||
const [localModelContextLoad, setLocalModelContextLoad] = React.useState<LocalModelContextWarning | null | undefined>(undefined);
|
||||
const isLocalModel = isActiveProviderLocalModel();
|
||||
const mainLoopModel = useAppState(s => s.mainLoopModel);
|
||||
let t1;
|
||||
if ($[0] === Symbol.for("react.memo_cache_sentinel")) {
|
||||
t1 = () => {
|
||||
@@ -72,6 +73,8 @@ export function StatusNotices(t0) {
|
||||
agentDefinitions,
|
||||
memoryFiles,
|
||||
async () => toolPermissionContext,
|
||||
undefined,
|
||||
mainLoopModel ?? undefined,
|
||||
).then(warning => {
|
||||
if (!cancelled) {
|
||||
setLocalModelContextLoad(warning);
|
||||
@@ -84,10 +87,9 @@ export function StatusNotices(t0) {
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, [agentDefinitions, isLocalModel, memoryFiles, toolPermissionContext, tools]);
|
||||
}, [agentDefinitions, isLocalModel, mainLoopModel, memoryFiles, toolPermissionContext, tools]);
|
||||
const t2 = getGlobalConfig();
|
||||
const permissionMode = useAppState(s => s.toolPermissionContext.mode);
|
||||
const mainLoopModel = useAppState(s => s.mainLoopModel);
|
||||
const context: StatusNoticeContext = {
|
||||
config: t2,
|
||||
agentDefinitions,
|
||||
|
||||
@@ -128,7 +128,7 @@ test('redacts credentials in transport diagnostic URL logs', async () => {
|
||||
|
||||
expect(transportLog).toBeDefined()
|
||||
const logLine = String(transportLog?.[0])
|
||||
expect(logLine).toContain('url=http://redacted:redacted@localhost:11434/v1/chat/completions')
|
||||
expect(logLine).toContain('url=http://redacted:redacted@localhost:11434/api/chat')
|
||||
expect(logLine).not.toContain('user:supersecret')
|
||||
expect(logLine).not.toContain('supersecret@')
|
||||
})
|
||||
@@ -201,8 +201,8 @@ test('logs self-heal localhost fallback with redacted from/to URLs', async () =>
|
||||
|
||||
expect(fallbackLog).toBeDefined()
|
||||
const logLine = String(fallbackLog?.[0])
|
||||
expect(logLine).toContain('from=http://redacted:redacted@localhost:11434/v1/chat/completions')
|
||||
expect(logLine).toContain('to=http://redacted:redacted@127.0.0.1:11434/v1/chat/completions')
|
||||
expect(logLine).toContain('from=http://redacted:redacted@localhost:11434/api/chat')
|
||||
expect(logLine).toContain('to=http://redacted:redacted@127.0.0.1:11434/api/chat')
|
||||
expect(logLine).not.toContain('supersecret')
|
||||
})
|
||||
|
||||
|
||||
@@ -26,7 +26,7 @@ type OpenAIShimClient = {
|
||||
}
|
||||
}
|
||||
|
||||
function makeSseResponse(lines: string[]): Response {
|
||||
function makeOllamaNativeStreamingResponse(lines: string[]): Response {
|
||||
const encoder = new TextEncoder()
|
||||
return new Response(
|
||||
new ReadableStream({
|
||||
@@ -35,12 +35,12 @@ function makeSseResponse(lines: string[]): Response {
|
||||
controller.close()
|
||||
},
|
||||
}),
|
||||
{ headers: { 'Content-Type': 'text/event-stream' } },
|
||||
{ headers: { 'Content-Type': 'application/x-ndjson' } },
|
||||
)
|
||||
}
|
||||
|
||||
function makeChunks(chunks: unknown[]): string[] {
|
||||
return [...chunks.map(c => `data: ${JSON.stringify(c)}\n\n`), 'data: [DONE]\n\n']
|
||||
function makeNdjsonChunks(chunks: unknown[]): string[] {
|
||||
return chunks.map(c => `${JSON.stringify(c)}\n`)
|
||||
}
|
||||
|
||||
describe('parseTextToolCalls', () => {
|
||||
@@ -193,17 +193,32 @@ describe('parseTextToolCalls', () => {
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const ollamaChunk = (content: string, finishReason?: string) => ({
|
||||
id: 'chatcmpl-1',
|
||||
object: 'chat.completion.chunk',
|
||||
model: 'qwen2.5:7b',
|
||||
choices: [{ index: 0, delta: { content }, finish_reason: finishReason ?? null }],
|
||||
message: { role: 'assistant', content },
|
||||
done: Boolean(finishReason),
|
||||
...(finishReason ? { done_reason: finishReason } : {}),
|
||||
})
|
||||
|
||||
const ollamaToolChunk = (toolCalls: unknown[], finishReason?: string) => ({
|
||||
id: 'chatcmpl-1',
|
||||
object: 'chat.completion.chunk',
|
||||
model: 'qwen2.5:7b',
|
||||
choices: [{ index: 0, delta: { tool_calls: toolCalls }, finish_reason: finishReason ?? null }],
|
||||
message: {
|
||||
role: 'assistant',
|
||||
content: '',
|
||||
tool_calls: toolCalls.map((toolCall) => {
|
||||
const call = toolCall as {
|
||||
function?: { name?: string; arguments?: unknown }
|
||||
}
|
||||
const args = call.function?.arguments
|
||||
return {
|
||||
function: {
|
||||
name: call.function?.name,
|
||||
arguments: typeof args === 'string' ? JSON.parse(args) : args,
|
||||
},
|
||||
}
|
||||
}),
|
||||
},
|
||||
done: Boolean(finishReason),
|
||||
...(finishReason ? { done_reason: finishReason } : {}),
|
||||
})
|
||||
|
||||
describe('Ollama streaming — think-tag filtering on text-tool fallback (P1)', () => {
|
||||
@@ -225,8 +240,8 @@ describe('Ollama streaming — think-tag filtering on text-tool fallback (P1)',
|
||||
// Repro: model emits <think>private plan</think> followed by tool-call JSON.
|
||||
// accumulatedText is raw; stripRanges leaves the <think> block unless we filter it.
|
||||
globalThis.fetch = (async () =>
|
||||
makeSseResponse(
|
||||
makeChunks([
|
||||
makeOllamaNativeStreamingResponse(
|
||||
makeNdjsonChunks([
|
||||
ollamaChunk('<think>private plan</think>{"name":"Bash","arguments":{"command":"ls"}}'),
|
||||
ollamaChunk('', 'stop'),
|
||||
]),
|
||||
@@ -277,8 +292,8 @@ describe('Ollama streaming — plain text response with no tool calls', () => {
|
||||
|
||||
test('plain text in two chunks (content then stop) is emitted as text_delta', async () => {
|
||||
globalThis.fetch = (async () =>
|
||||
makeSseResponse(
|
||||
makeChunks([
|
||||
makeOllamaNativeStreamingResponse(
|
||||
makeNdjsonChunks([
|
||||
ollamaChunk('Hello from Ollama.'),
|
||||
ollamaChunk('', 'stop'),
|
||||
]),
|
||||
@@ -309,8 +324,8 @@ describe('Ollama streaming — plain text response with no tool calls', () => {
|
||||
|
||||
test('plain text in single chunk (content + stop) is emitted as text_delta', async () => {
|
||||
globalThis.fetch = (async () =>
|
||||
makeSseResponse(
|
||||
makeChunks([ollamaChunk('Hello from Ollama.', 'stop')]),
|
||||
makeOllamaNativeStreamingResponse(
|
||||
makeNdjsonChunks([ollamaChunk('Hello from Ollama.', 'stop')]),
|
||||
)) as unknown as FetchType
|
||||
|
||||
const client = createOpenAIShimClient({}) as OpenAIShimClient
|
||||
@@ -336,8 +351,8 @@ describe('Ollama streaming — plain text response with no tool calls', () => {
|
||||
|
||||
test('multi-chunk plain text (no tool calls) assembles correctly', async () => {
|
||||
globalThis.fetch = (async () =>
|
||||
makeSseResponse(
|
||||
makeChunks([
|
||||
makeOllamaNativeStreamingResponse(
|
||||
makeNdjsonChunks([
|
||||
ollamaChunk('Hello '),
|
||||
ollamaChunk('from '),
|
||||
ollamaChunk('Ollama.'),
|
||||
@@ -386,14 +401,13 @@ describe('Ollama streaming — visible text before real structured tool_calls (P
|
||||
// Repro: Ollama endpoint emits visible prose first, then real structured tool_calls.
|
||||
// Before fix: ollamaTextBuffer was discarded when the text block closed.
|
||||
globalThis.fetch = (async () =>
|
||||
makeSseResponse(
|
||||
makeChunks([
|
||||
makeOllamaNativeStreamingResponse(
|
||||
makeNdjsonChunks([
|
||||
ollamaChunk('Let me check that.'),
|
||||
ollamaToolChunk([
|
||||
{ index: 0, id: 'call_1', type: 'function', function: { name: 'Bash', arguments: '{"command":"ls"}' } },
|
||||
]),
|
||||
{ id: 'chatcmpl-1', object: 'chat.completion.chunk', model: 'qwen2.5:7b',
|
||||
choices: [{ index: 0, delta: {}, finish_reason: 'tool_calls' }] },
|
||||
ollamaChunk('', 'stop'),
|
||||
]),
|
||||
)) as unknown as FetchType
|
||||
|
||||
@@ -444,8 +458,8 @@ describe('Ollama streaming — visible prose before text-based tool-call fallbac
|
||||
// Before fix: hasEmittedContentStart === false in the fallback branch, so the prose in
|
||||
// ollamaTextBuffer was discarded — only the synthetic tool_use block was emitted.
|
||||
globalThis.fetch = (async () =>
|
||||
makeSseResponse(
|
||||
makeChunks([
|
||||
makeOllamaNativeStreamingResponse(
|
||||
makeNdjsonChunks([
|
||||
ollamaChunk('I will inspect the file.\n'),
|
||||
ollamaChunk('{"name":"Read","arguments":{"file_path":"/tmp/foo.ts"}}'),
|
||||
ollamaChunk('', 'stop'),
|
||||
@@ -513,8 +527,8 @@ describe('Ollama streaming — non-stop terminal finish reasons flush buffer', (
|
||||
|
||||
test('buffered text is flushed when finish_reason is "length"', async () => {
|
||||
globalThis.fetch = (async () =>
|
||||
makeSseResponse(
|
||||
makeChunks([
|
||||
makeOllamaNativeStreamingResponse(
|
||||
makeNdjsonChunks([
|
||||
ollamaChunk('Partial response cut off by'),
|
||||
ollamaChunk('', 'length'),
|
||||
]),
|
||||
@@ -542,8 +556,8 @@ describe('Ollama streaming — non-stop terminal finish reasons flush buffer', (
|
||||
|
||||
test('text-tool JSON is extracted when finish_reason is "length" but finish_reason stays "length"', async () => {
|
||||
globalThis.fetch = (async () =>
|
||||
makeSseResponse(
|
||||
makeChunks([
|
||||
makeOllamaNativeStreamingResponse(
|
||||
makeNdjsonChunks([
|
||||
ollamaChunk('{"name":"Bash","arguments":{"command":"ls"}}'),
|
||||
ollamaChunk('', 'length'),
|
||||
]),
|
||||
@@ -576,8 +590,8 @@ describe('Ollama streaming — non-stop terminal finish reasons flush buffer', (
|
||||
|
||||
test('buffered text is flushed when finish_reason is "content_filter"', async () => {
|
||||
globalThis.fetch = (async () =>
|
||||
makeSseResponse(
|
||||
makeChunks([
|
||||
makeOllamaNativeStreamingResponse(
|
||||
makeNdjsonChunks([
|
||||
ollamaChunk('Text stopped by content filter'),
|
||||
ollamaChunk('', 'content_filter'),
|
||||
]),
|
||||
@@ -609,8 +623,8 @@ describe('Ollama streaming — non-stop terminal finish reasons flush buffer', (
|
||||
|
||||
test('buffered text is flushed when finish_reason is "safety"', async () => {
|
||||
globalThis.fetch = (async () =>
|
||||
makeSseResponse(
|
||||
makeChunks([
|
||||
makeOllamaNativeStreamingResponse(
|
||||
makeNdjsonChunks([
|
||||
ollamaChunk('Text stopped by safety check'),
|
||||
ollamaChunk('', 'safety'),
|
||||
]),
|
||||
|
||||
@@ -1231,28 +1231,21 @@ test('uses max_tokens instead of max_completion_tokens for local providers', asy
|
||||
|
||||
globalThis.fetch = (async (_input, init) => {
|
||||
const body = JSON.parse(String(init?.body))
|
||||
expect(body.max_tokens).toBe(64)
|
||||
expect(body.max_completion_tokens).toBeUndefined()
|
||||
expect(body.options?.num_predict).toBe(64)
|
||||
expect(body.options?.num_ctx).toBe(32768)
|
||||
expect(body.stream_options).toBeUndefined()
|
||||
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
id: 'chatcmpl-1',
|
||||
model: 'llama3.1:8b',
|
||||
choices: [
|
||||
{
|
||||
message: {
|
||||
role: 'assistant',
|
||||
content: 'hello',
|
||||
},
|
||||
finish_reason: 'stop',
|
||||
},
|
||||
],
|
||||
usage: {
|
||||
prompt_tokens: 5,
|
||||
completion_tokens: 1,
|
||||
total_tokens: 6,
|
||||
message: {
|
||||
role: 'assistant',
|
||||
content: 'hello',
|
||||
},
|
||||
done: true,
|
||||
done_reason: 'stop',
|
||||
prompt_eval_count: 5,
|
||||
eval_count: 1,
|
||||
}),
|
||||
{
|
||||
headers: {
|
||||
@@ -5707,11 +5700,11 @@ test('self-heals localhost resolution failures by retrying local loopback base U
|
||||
}),
|
||||
).resolves.toBeDefined()
|
||||
|
||||
expect(requestUrls[0]).toBe('http://localhost:11434/v1/chat/completions')
|
||||
expect(requestUrls).toContain('http://127.0.0.1:11434/v1/chat/completions')
|
||||
expect(requestUrls[0]).toBe('http://localhost:11434/api/chat')
|
||||
expect(requestUrls).toContain('http://127.0.0.1:11434/api/chat')
|
||||
})
|
||||
|
||||
test('self-heals local endpoint_not_found by retrying with /v1 base URL', async () => {
|
||||
test('uses native Ollama chat endpoint when local base URL omits /v1', async () => {
|
||||
process.env.OPENAI_BASE_URL = 'http://localhost:11434'
|
||||
|
||||
const requestUrls: string[] = []
|
||||
@@ -5719,33 +5712,17 @@ test('self-heals local endpoint_not_found by retrying with /v1 base URL', async
|
||||
const url = typeof input === 'string' ? input : input.url
|
||||
requestUrls.push(url)
|
||||
|
||||
if (url === 'http://localhost:11434/chat/completions') {
|
||||
return new Response('Not Found', {
|
||||
status: 404,
|
||||
headers: {
|
||||
'Content-Type': 'text/plain',
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
id: 'chatcmpl-1',
|
||||
model: 'qwen2.5-coder:7b',
|
||||
choices: [
|
||||
{
|
||||
message: {
|
||||
role: 'assistant',
|
||||
content: 'hello from /v1',
|
||||
},
|
||||
finish_reason: 'stop',
|
||||
},
|
||||
],
|
||||
usage: {
|
||||
prompt_tokens: 5,
|
||||
completion_tokens: 2,
|
||||
total_tokens: 7,
|
||||
message: {
|
||||
role: 'assistant',
|
||||
content: 'hello from native Ollama',
|
||||
},
|
||||
done: true,
|
||||
done_reason: 'stop',
|
||||
prompt_eval_count: 5,
|
||||
eval_count: 2,
|
||||
}),
|
||||
{
|
||||
status: 200,
|
||||
@@ -5767,9 +5744,66 @@ test('self-heals local endpoint_not_found by retrying with /v1 base URL', async
|
||||
}),
|
||||
).resolves.toBeDefined()
|
||||
|
||||
expect(requestUrls).toEqual(['http://localhost:11434/api/chat'])
|
||||
})
|
||||
|
||||
test('keeps remote Ollama-named gateways on chat completions', async () => {
|
||||
process.env.OPENAI_BASE_URL = 'https://ollama-gateway.example.com/v1'
|
||||
|
||||
const requestUrls: string[] = []
|
||||
globalThis.fetch = (async (input, init) => {
|
||||
const url = typeof input === 'string' ? input : input.url
|
||||
requestUrls.push(url)
|
||||
const body = JSON.parse(String(init?.body)) as Record<string, unknown>
|
||||
expect(body.max_tokens).toBe(64)
|
||||
expect(body.options).toBeUndefined()
|
||||
|
||||
return makeChatCompletionResponse('llama3.1:8b')
|
||||
}) as unknown as FetchType
|
||||
|
||||
const client = createOpenAIShimClient({}) as OpenAIShimClient
|
||||
|
||||
await expect(
|
||||
client.beta.messages.create({
|
||||
model: 'llama3.1:8b',
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
max_tokens: 64,
|
||||
stream: false,
|
||||
}),
|
||||
).resolves.toBeDefined()
|
||||
|
||||
expect(requestUrls).toEqual([
|
||||
'http://localhost:11434/chat/completions',
|
||||
'http://localhost:11434/v1/chat/completions',
|
||||
'https://ollama-gateway.example.com/v1/chat/completions',
|
||||
])
|
||||
})
|
||||
|
||||
test('keeps HTTPS localhost Ollama-port proxies on chat completions', async () => {
|
||||
process.env.OPENAI_BASE_URL = 'https://localhost:11434/v1'
|
||||
|
||||
const requestUrls: string[] = []
|
||||
globalThis.fetch = (async (input, init) => {
|
||||
const url = typeof input === 'string' ? input : input.url
|
||||
requestUrls.push(url)
|
||||
const body = JSON.parse(String(init?.body)) as Record<string, unknown>
|
||||
expect(body.max_tokens).toBe(64)
|
||||
expect(body.options).toBeUndefined()
|
||||
|
||||
return makeChatCompletionResponse('llama3.1:8b')
|
||||
}) as unknown as FetchType
|
||||
|
||||
const client = createOpenAIShimClient({}) as OpenAIShimClient
|
||||
|
||||
await expect(
|
||||
client.beta.messages.create({
|
||||
model: 'llama3.1:8b',
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
max_tokens: 64,
|
||||
stream: false,
|
||||
}),
|
||||
).resolves.toBeDefined()
|
||||
|
||||
expect(requestUrls).toEqual([
|
||||
'https://localhost:11434/v1/chat/completions',
|
||||
])
|
||||
})
|
||||
|
||||
|
||||
@@ -77,6 +77,7 @@ import {
|
||||
getLocalFastPathConfig,
|
||||
getLocalProviderRetryBaseUrls,
|
||||
getGithubEndpointType,
|
||||
isDirectLocalOllamaEndpoint,
|
||||
isLikelyOllamaEndpoint,
|
||||
isLocalProviderUrl,
|
||||
resolveRuntimeCodexCredentials,
|
||||
@@ -109,6 +110,7 @@ import {
|
||||
hasInvalidCredentialPlaceholder,
|
||||
parseCredentialList,
|
||||
} from './credentialPool.js'
|
||||
import { MIN_RECOMMENDED_OLLAMA_CONTEXT_TOKENS } from '../../utils/ollamaContext.js'
|
||||
|
||||
const GITHUB_429_MAX_RETRIES = 3
|
||||
const GITHUB_429_BASE_DELAY_SEC = 1
|
||||
@@ -300,6 +302,399 @@ interface OpenAITool {
|
||||
}
|
||||
}
|
||||
|
||||
type OllamaChatResponse = {
|
||||
model?: string
|
||||
message?: {
|
||||
role?: string
|
||||
content?: string
|
||||
tool_calls?: Array<{
|
||||
function?: {
|
||||
name?: string
|
||||
arguments?: unknown
|
||||
}
|
||||
}>
|
||||
}
|
||||
done?: boolean
|
||||
done_reason?: string
|
||||
prompt_eval_count?: number
|
||||
eval_count?: number
|
||||
}
|
||||
|
||||
type OllamaChatMessage = Omit<OpenAIMessage, 'content' | 'tool_calls'> & {
|
||||
content?: string
|
||||
images?: string[]
|
||||
tool_calls?: Array<{
|
||||
function: {
|
||||
name: string
|
||||
arguments: Record<string, unknown>
|
||||
}
|
||||
}>
|
||||
}
|
||||
|
||||
function parsePositiveIntegerEnv(value: string | undefined): number | null {
|
||||
if (!value?.trim()) {
|
||||
return null
|
||||
}
|
||||
const parsed = Number(value.trim())
|
||||
if (!Number.isInteger(parsed) || parsed <= 0) {
|
||||
return null
|
||||
}
|
||||
return parsed
|
||||
}
|
||||
|
||||
function getOllamaNumCtx(): number {
|
||||
return (
|
||||
parsePositiveIntegerEnv(process.env.OPENCLAUDE_OLLAMA_NUM_CTX) ??
|
||||
parsePositiveIntegerEnv(process.env.OLLAMA_CONTEXT_LENGTH) ??
|
||||
MIN_RECOMMENDED_OLLAMA_CONTEXT_TOKENS
|
||||
)
|
||||
}
|
||||
|
||||
function buildOllamaChatUrl(baseUrl: string): string {
|
||||
const parsed = new URL(baseUrl)
|
||||
parsed.pathname = parsed.pathname.replace(/\/+$/, '').replace(/\/v1$/i, '')
|
||||
parsed.pathname = `${parsed.pathname.replace(/\/+$/, '')}/api/chat`
|
||||
parsed.search = ''
|
||||
parsed.hash = ''
|
||||
return parsed.toString()
|
||||
}
|
||||
|
||||
function extractOllamaImageData(url: string): string | null {
|
||||
const match = url.match(/^data:[^;,]+;base64,(.+)$/i)
|
||||
if (!match) {
|
||||
return null
|
||||
}
|
||||
return match[1]
|
||||
}
|
||||
|
||||
function normalizeOllamaNativeToolCalls(
|
||||
toolCalls: OpenAIMessage['tool_calls'],
|
||||
): OllamaChatMessage['tool_calls'] {
|
||||
if (!Array.isArray(toolCalls) || toolCalls.length === 0) {
|
||||
return undefined
|
||||
}
|
||||
|
||||
const normalized = toolCalls
|
||||
.map(toolCall => {
|
||||
const name = toolCall.function?.name
|
||||
if (!name) {
|
||||
return null
|
||||
}
|
||||
|
||||
let args: Record<string, unknown> = {}
|
||||
try {
|
||||
const parsed = JSON.parse(toolCall.function.arguments || '{}')
|
||||
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
|
||||
args = parsed as Record<string, unknown>
|
||||
}
|
||||
} catch {
|
||||
args = {}
|
||||
}
|
||||
|
||||
return {
|
||||
function: {
|
||||
name,
|
||||
arguments: args,
|
||||
},
|
||||
}
|
||||
})
|
||||
.filter((toolCall): toolCall is NonNullable<typeof toolCall> => toolCall !== null)
|
||||
|
||||
return normalized.length > 0 ? normalized : undefined
|
||||
}
|
||||
|
||||
function normalizeOllamaNativeMessages(messages: unknown): OllamaChatMessage[] {
|
||||
if (!Array.isArray(messages)) {
|
||||
return []
|
||||
}
|
||||
|
||||
return messages.map(message => {
|
||||
const openAIMessage = message as OpenAIMessage
|
||||
const content = openAIMessage.content
|
||||
const toolCalls = normalizeOllamaNativeToolCalls(openAIMessage.tool_calls)
|
||||
if (!Array.isArray(content)) {
|
||||
return {
|
||||
...openAIMessage,
|
||||
content,
|
||||
...(toolCalls ? { tool_calls: toolCalls } : { tool_calls: undefined }),
|
||||
}
|
||||
}
|
||||
|
||||
const textParts: string[] = []
|
||||
const images: string[] = []
|
||||
|
||||
for (const part of content) {
|
||||
if (part.type === 'text') {
|
||||
if (part.text) {
|
||||
textParts.push(part.text)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
if (part.type === 'image_url') {
|
||||
const imageUrl = part.image_url.url
|
||||
const imageData = extractOllamaImageData(imageUrl)
|
||||
if (imageData) {
|
||||
images.push(imageData)
|
||||
} else {
|
||||
textParts.push(`[Image: ${imageUrl}]`)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
...openAIMessage,
|
||||
content: textParts.join('\n'),
|
||||
...(images.length > 0 ? { images } : {}),
|
||||
...(toolCalls ? { tool_calls: toolCalls } : { tool_calls: undefined }),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
function mapOllamaDoneReason(doneReason: unknown): string | null {
|
||||
if (doneReason === 'length') return 'length'
|
||||
if (doneReason === 'stop') return 'stop'
|
||||
if (typeof doneReason === 'string' && doneReason) return doneReason
|
||||
return null
|
||||
}
|
||||
|
||||
function normalizeOllamaToolCalls(
|
||||
toolCalls: NonNullable<OllamaChatResponse['message']>['tool_calls'],
|
||||
): Array<{
|
||||
id: string
|
||||
type: 'function'
|
||||
function: { name: string; arguments: string }
|
||||
}> | undefined {
|
||||
if (!Array.isArray(toolCalls) || toolCalls.length === 0) {
|
||||
return undefined
|
||||
}
|
||||
|
||||
const normalized = toolCalls
|
||||
.map(toolCall => {
|
||||
const name = toolCall.function?.name
|
||||
if (!name) {
|
||||
return null
|
||||
}
|
||||
const args = toolCall.function?.arguments
|
||||
return {
|
||||
id: `call_${crypto.randomUUID().replace(/-/g, '').slice(0, 24)}`,
|
||||
type: 'function' as const,
|
||||
function: {
|
||||
name,
|
||||
arguments:
|
||||
typeof args === 'string' ? args : JSON.stringify(args ?? {}),
|
||||
},
|
||||
}
|
||||
})
|
||||
.filter((toolCall): toolCall is NonNullable<typeof toolCall> => toolCall !== null)
|
||||
|
||||
return normalized.length > 0 ? normalized : undefined
|
||||
}
|
||||
|
||||
function buildOpenAIUsageFromOllama(data: OllamaChatResponse) {
|
||||
const promptTokens = data.prompt_eval_count ?? 0
|
||||
const completionTokens = data.eval_count ?? 0
|
||||
return {
|
||||
prompt_tokens: promptTokens,
|
||||
completion_tokens: completionTokens,
|
||||
total_tokens: promptTokens + completionTokens,
|
||||
}
|
||||
}
|
||||
|
||||
function convertOllamaChatResponseToOpenAI(
|
||||
data: OllamaChatResponse,
|
||||
fallbackModel: string,
|
||||
): Record<string, unknown> {
|
||||
const toolCalls = normalizeOllamaToolCalls(data.message?.tool_calls)
|
||||
return {
|
||||
id: makeMessageId(),
|
||||
object: 'chat.completion',
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
model: data.model ?? fallbackModel,
|
||||
choices: [
|
||||
{
|
||||
index: 0,
|
||||
message: {
|
||||
role: 'assistant',
|
||||
content: data.message?.content ?? '',
|
||||
...(toolCalls ? { tool_calls: toolCalls } : {}),
|
||||
},
|
||||
finish_reason: mapOllamaDoneReason(data.done_reason),
|
||||
},
|
||||
],
|
||||
usage: buildOpenAIUsageFromOllama(data),
|
||||
}
|
||||
}
|
||||
|
||||
function responseWithPreservedUrl(
|
||||
body: BodyInit | null,
|
||||
init: ResponseInit,
|
||||
url: string,
|
||||
): Response {
|
||||
const response = new Response(body, init)
|
||||
try {
|
||||
Object.defineProperty(response, 'url', {
|
||||
value: url,
|
||||
configurable: true,
|
||||
})
|
||||
} catch {
|
||||
/* some runtimes lock the property; downstream has transport fallback */
|
||||
}
|
||||
return response
|
||||
}
|
||||
|
||||
async function convertOllamaNonStreamingResponse(
|
||||
response: Response,
|
||||
fallbackModel: string,
|
||||
): Promise<Response> {
|
||||
const data = await response.json() as OllamaChatResponse
|
||||
return responseWithPreservedUrl(
|
||||
JSON.stringify(convertOllamaChatResponseToOpenAI(data, fallbackModel)),
|
||||
{
|
||||
status: response.status,
|
||||
statusText: response.statusText,
|
||||
headers: { 'content-type': 'application/json' },
|
||||
},
|
||||
response.url,
|
||||
)
|
||||
}
|
||||
|
||||
function openAIStreamChunk(
|
||||
id: string,
|
||||
model: string,
|
||||
delta: Record<string, unknown>,
|
||||
finishReason: string | null = null,
|
||||
): string {
|
||||
return `data: ${JSON.stringify({
|
||||
id,
|
||||
object: 'chat.completion.chunk',
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
model,
|
||||
choices: [{ index: 0, delta, finish_reason: finishReason }],
|
||||
})}\n\n`
|
||||
}
|
||||
|
||||
function convertOllamaStreamingResponse(
|
||||
response: Response,
|
||||
fallbackModel: string,
|
||||
): Response {
|
||||
const body = response.body
|
||||
if (!body) {
|
||||
return response
|
||||
}
|
||||
|
||||
const decoder = new TextDecoder()
|
||||
const encoder = new TextEncoder()
|
||||
const reader = body.getReader()
|
||||
const streamId = makeMessageId()
|
||||
let buffer = ''
|
||||
let hasEmittedRole = false
|
||||
let hasEmittedToolCall = false
|
||||
|
||||
const stream = new ReadableStream<Uint8Array>({
|
||||
async pull(controller) {
|
||||
while (true) {
|
||||
const { done, value } = await reader.read()
|
||||
if (done) {
|
||||
if (buffer.trim()) {
|
||||
enqueueOllamaLineAsOpenAI(buffer.trim(), controller)
|
||||
buffer = ''
|
||||
}
|
||||
controller.enqueue(encoder.encode('data: [DONE]\n\n'))
|
||||
controller.close()
|
||||
return
|
||||
}
|
||||
|
||||
buffer += decoder.decode(value, { stream: true })
|
||||
const lines = buffer.split(/\r?\n/)
|
||||
buffer = lines.pop() ?? ''
|
||||
|
||||
let emittedLine = false
|
||||
for (const line of lines) {
|
||||
if (line.trim()) {
|
||||
enqueueOllamaLineAsOpenAI(line.trim(), controller)
|
||||
emittedLine = true
|
||||
}
|
||||
}
|
||||
if (emittedLine) {
|
||||
return
|
||||
}
|
||||
}
|
||||
},
|
||||
cancel(reason) {
|
||||
return reader.cancel(reason)
|
||||
},
|
||||
})
|
||||
|
||||
function enqueueOllamaLineAsOpenAI(
|
||||
line: string,
|
||||
controller: ReadableStreamDefaultController<Uint8Array>,
|
||||
): void {
|
||||
let data: OllamaChatResponse
|
||||
try {
|
||||
data = JSON.parse(line) as OllamaChatResponse
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
|
||||
const model = data.model ?? fallbackModel
|
||||
const chunks: string[] = []
|
||||
const delta: Record<string, unknown> = {}
|
||||
if (!hasEmittedRole) {
|
||||
delta.role = 'assistant'
|
||||
hasEmittedRole = true
|
||||
}
|
||||
if (data.message?.content) {
|
||||
delta.content = data.message.content
|
||||
}
|
||||
const toolCalls = normalizeOllamaToolCalls(data.message?.tool_calls)
|
||||
if (toolCalls) {
|
||||
hasEmittedToolCall = true
|
||||
delta.tool_calls = toolCalls.map((toolCall, index) => ({
|
||||
index,
|
||||
id: toolCall.id,
|
||||
type: toolCall.type,
|
||||
function: toolCall.function,
|
||||
}))
|
||||
}
|
||||
if (Object.keys(delta).length > 0) {
|
||||
chunks.push(openAIStreamChunk(streamId, model, delta))
|
||||
}
|
||||
if (data.done) {
|
||||
chunks.push(openAIStreamChunk(
|
||||
streamId,
|
||||
model,
|
||||
{},
|
||||
hasEmittedToolCall
|
||||
? 'tool_calls'
|
||||
: mapOllamaDoneReason(data.done_reason),
|
||||
))
|
||||
chunks.push(`data: ${JSON.stringify({
|
||||
id: streamId,
|
||||
object: 'chat.completion.chunk',
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
model,
|
||||
choices: [],
|
||||
usage: buildOpenAIUsageFromOllama(data),
|
||||
})}\n\n`)
|
||||
}
|
||||
|
||||
for (const chunk of chunks) {
|
||||
controller.enqueue(encoder.encode(chunk))
|
||||
}
|
||||
}
|
||||
|
||||
return responseWithPreservedUrl(
|
||||
stream,
|
||||
{
|
||||
status: response.status,
|
||||
statusText: response.statusText,
|
||||
headers: { 'content-type': 'text/event-stream' },
|
||||
},
|
||||
response.url,
|
||||
)
|
||||
}
|
||||
|
||||
function convertSystemPrompt(
|
||||
system: unknown,
|
||||
): string {
|
||||
@@ -2676,6 +3071,11 @@ class OpenAIShimMessages {
|
||||
: shimConfig.endpointPath?.startsWith('/models/gemini-')
|
||||
? 'gemini'
|
||||
: request.transport
|
||||
const useNativeOllamaChat =
|
||||
effectiveTransport === 'chat_completions' &&
|
||||
!shimConfig.endpointPath &&
|
||||
isDirectLocalOllamaEndpoint(request.baseUrl) &&
|
||||
isLikelyOllamaEndpoint(request.baseUrl)
|
||||
const openaiMessages = convertMessages(compressedMessages, params.system, {
|
||||
preserveReasoningContent: shimConfig.preserveReasoningContent,
|
||||
reasoningContentFallback: shimConfig.reasoningContentFallback,
|
||||
@@ -3317,6 +3717,9 @@ class OpenAIShimMessages {
|
||||
if (shimConfig.endpointPath) {
|
||||
return `${baseUrl}${shimConfig.endpointPath}`
|
||||
}
|
||||
if (useNativeOllamaChat) {
|
||||
return buildOllamaChatUrl(baseUrl)
|
||||
}
|
||||
return request.transport === 'responses' || request.transport === 'responses_compat'
|
||||
? `${baseUrl}/responses`
|
||||
: buildChatCompletionsUrl(baseUrl)
|
||||
@@ -3381,9 +3784,31 @@ class OpenAIShimMessages {
|
||||
// Local backends do not implement prefix caching, so the deep key-sort
|
||||
// is pure CPU overhead per request (issue #1016). Drop to the native
|
||||
// `JSON.stringify` fast path when the fast-path config opts out.
|
||||
const buildOllamaChatBody = (): Record<string, unknown> => {
|
||||
const options: Record<string, unknown> = {
|
||||
num_ctx: getOllamaNumCtx(),
|
||||
}
|
||||
if (body.max_tokens !== undefined) {
|
||||
options.num_predict = body.max_tokens
|
||||
} else if (body.max_completion_tokens !== undefined) {
|
||||
options.num_predict = body.max_completion_tokens
|
||||
}
|
||||
if (params.temperature !== undefined) options.temperature = params.temperature
|
||||
if (params.top_p !== undefined) options.top_p = params.top_p
|
||||
|
||||
return {
|
||||
model: request.resolvedModel,
|
||||
messages: normalizeOllamaNativeMessages(body.messages),
|
||||
stream: params.stream ?? false,
|
||||
options,
|
||||
...(body.tools ? { tools: body.tools } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
const serializeBody = (): string => {
|
||||
const payload =
|
||||
effectiveTransport === 'responses' || effectiveTransport === 'responses_compat' ? buildResponsesBody()
|
||||
useNativeOllamaChat ? buildOllamaChatBody()
|
||||
: effectiveTransport === 'responses' || effectiveTransport === 'responses_compat' ? buildResponsesBody()
|
||||
: effectiveTransport === 'anthropic_messages' ? buildAnthropicMessagesBody()
|
||||
: effectiveTransport === 'gemini' ? buildGeminiBody()
|
||||
: body
|
||||
@@ -3551,6 +3976,11 @@ class OpenAIShimMessages {
|
||||
|
||||
if (response.ok) {
|
||||
credentialPool?.reportSuccess(credentialLease)
|
||||
if (useNativeOllamaChat) {
|
||||
response = params.stream
|
||||
? convertOllamaStreamingResponse(response, request.resolvedModel)
|
||||
: await convertOllamaNonStreamingResponse(response, request.resolvedModel)
|
||||
}
|
||||
let tokensIn = 0
|
||||
let tokensOut = 0
|
||||
// Skip clone() for streaming responses - it blocks until full body is received,
|
||||
|
||||
@@ -54,13 +54,16 @@ afterEach(() => {
|
||||
test('treats localhost endpoints as local', () => {
|
||||
expect(isLocalProviderUrl('http://localhost:11434/v1')).toBe(true)
|
||||
expect(isLocalProviderUrl('http://127.0.0.1:11434/v1')).toBe(true)
|
||||
expect(isLocalProviderUrl('http://0.0.0.0:11434/v1')).toBe(true)
|
||||
// Full 127.0.0.0/8 loopback range should be treated as local
|
||||
expect(isLocalProviderUrl('http://127.0.0.2:11434/v1')).toBe(true)
|
||||
expect(isLocalProviderUrl('http://127.1.2.3:11434/v1')).toBe(true)
|
||||
expect(isLocalProviderUrl('http://127.255.255.255:11434/v1')).toBe(true)
|
||||
})
|
||||
|
||||
test('does not treat wildcard bind addresses as local endpoints', () => {
|
||||
expect(isLocalProviderUrl('http://0.0.0.0:11434/v1')).toBe(false)
|
||||
})
|
||||
|
||||
test('treats private IPv4 endpoints as local', () => {
|
||||
expect(isLocalProviderUrl('http://10.0.0.1:11434/v1')).toBe(true)
|
||||
expect(isLocalProviderUrl('http://172.16.0.1:11434/v1')).toBe(true)
|
||||
|
||||
@@ -419,7 +419,7 @@ export function isLocalProviderUrl(baseUrl: string | undefined): boolean {
|
||||
hostname = hostname.slice(0, zoneIdIndex)
|
||||
}
|
||||
|
||||
if (LOCALHOST_HOSTNAMES.has(hostname) || hostname === '0.0.0.0') {
|
||||
if (LOCALHOST_HOSTNAMES.has(hostname)) {
|
||||
return true
|
||||
}
|
||||
if (hostname.endsWith('.local')) {
|
||||
@@ -543,6 +543,33 @@ export function isLikelyOllamaEndpoint(baseUrl: string | undefined): boolean {
|
||||
}
|
||||
}
|
||||
|
||||
export function isDirectLocalOllamaEndpoint(baseUrl: string | undefined): boolean {
|
||||
if (!baseUrl) return false
|
||||
try {
|
||||
const parsed = new URL(baseUrl)
|
||||
let hostname = parsed.hostname.toLowerCase()
|
||||
if (hostname.startsWith('[') && hostname.endsWith(']')) {
|
||||
hostname = hostname.slice(1, -1)
|
||||
}
|
||||
const ipv4Octets = hostname.split('.')
|
||||
const isLoopbackIpv4 =
|
||||
ipv4Octets.length === 4 &&
|
||||
ipv4Octets.every(octet => /^\d{1,3}$/.test(octet) && Number(octet) <= 255) &&
|
||||
ipv4Octets[0] === '127'
|
||||
return (
|
||||
parsed.protocol === 'http:' &&
|
||||
parsed.port === '11434' &&
|
||||
(
|
||||
hostname === 'localhost' ||
|
||||
hostname === '::1' ||
|
||||
isLoopbackIpv4
|
||||
)
|
||||
)
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
export function getLocalProviderRetryBaseUrls(baseUrl: string): string[] {
|
||||
if (!isLocalProviderUrl(baseUrl)) {
|
||||
return []
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
import { execFileNoThrow } from './execFileNoThrow.js'
|
||||
|
||||
export const MIN_RECOMMENDED_OLLAMA_CONTEXT_TOKENS = 32_768
|
||||
|
||||
export type OllamaContextWarning = {
|
||||
modelName: string
|
||||
contextValue: string
|
||||
contextTokens: number
|
||||
}
|
||||
|
||||
function parseContextTokenValue(value: string | undefined): number | null {
|
||||
const normalized = value?.trim().replace(/,/g, '')
|
||||
if (!normalized) {
|
||||
return null
|
||||
}
|
||||
|
||||
const match = normalized.match(/^(\d+(?:\.\d+)?)([kKmM]?)$/)
|
||||
if (!match) {
|
||||
return null
|
||||
}
|
||||
|
||||
const amount = Number(match[1])
|
||||
if (!Number.isFinite(amount) || amount <= 0) {
|
||||
return null
|
||||
}
|
||||
|
||||
const suffix = match[2].toLowerCase()
|
||||
if (suffix === 'm') {
|
||||
return Math.round(amount * 1_000_000)
|
||||
}
|
||||
if (suffix === 'k') {
|
||||
return Math.round(amount * 1_024)
|
||||
}
|
||||
return Math.round(amount)
|
||||
}
|
||||
|
||||
function normalizeOllamaModelName(modelName: string | undefined): string | null {
|
||||
const normalized = modelName?.trim().toLowerCase().split('?')[0]
|
||||
return normalized || null
|
||||
}
|
||||
|
||||
export function parseOllamaPsContextWarning(
|
||||
output: string,
|
||||
activeModelName?: string,
|
||||
): OllamaContextWarning | null {
|
||||
const normalizedActiveModel = normalizeOllamaModelName(activeModelName)
|
||||
const lines = output
|
||||
.split(/\r?\n/)
|
||||
.map(line => line.trim())
|
||||
.filter(Boolean)
|
||||
|
||||
if (lines.length < 2) {
|
||||
return null
|
||||
}
|
||||
|
||||
const header = lines[0]
|
||||
const contextColumnStart = header.toLowerCase().indexOf('context')
|
||||
if (contextColumnStart === -1) {
|
||||
return null
|
||||
}
|
||||
|
||||
for (const line of lines.slice(1)) {
|
||||
const modelName = line.split(/\s+/)[0] ?? 'loaded model'
|
||||
if (
|
||||
normalizedActiveModel &&
|
||||
normalizeOllamaModelName(modelName) !== normalizedActiveModel
|
||||
) {
|
||||
continue
|
||||
}
|
||||
|
||||
const contextValue = line.slice(contextColumnStart).trim().split(/\s+/)[0]
|
||||
const contextTokens = parseContextTokenValue(contextValue)
|
||||
if (
|
||||
contextTokens !== null &&
|
||||
contextTokens < MIN_RECOMMENDED_OLLAMA_CONTEXT_TOKENS
|
||||
) {
|
||||
return {
|
||||
modelName,
|
||||
contextValue,
|
||||
contextTokens,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
export async function checkOllamaPsContextWarning(
|
||||
activeModelName?: string,
|
||||
): Promise<OllamaContextWarning | null> {
|
||||
const result = await execFileNoThrow('ollama', ['ps'], {
|
||||
timeout: 1000,
|
||||
preserveOutputOnError: true,
|
||||
useCwd: false,
|
||||
})
|
||||
if (result.code !== 0 || !result.stdout.trim()) {
|
||||
return null
|
||||
}
|
||||
|
||||
return parseOllamaPsContextWarning(result.stdout, activeModelName)
|
||||
}
|
||||
@@ -1,5 +1,8 @@
|
||||
import { resolveActiveRouteIdFromEnv } from '../integrations/routeMetadata.js'
|
||||
import { isLocalProviderUrl } from '../services/api/providerConfig.js'
|
||||
import {
|
||||
isDirectLocalOllamaEndpoint,
|
||||
isLocalProviderUrl,
|
||||
} from '../services/api/providerConfig.js'
|
||||
import type { Tool, ToolPermissionContext } from '../Tool.js'
|
||||
import type { AgentDefinitionsResult } from '../tools/AgentTool/loadAgentsDir.js'
|
||||
import type { MemoryFileInfo } from './claudemd.js'
|
||||
@@ -11,8 +14,17 @@ import {
|
||||
import { formatTokens } from './format.js'
|
||||
import { isEnvTruthy } from './envUtils.js'
|
||||
import { plural } from './stringUtils.js'
|
||||
import {
|
||||
type OllamaContextWarning,
|
||||
checkOllamaPsContextWarning,
|
||||
parseOllamaPsContextWarning as parseOllamaPsContext,
|
||||
} from './ollamaContext.js'
|
||||
|
||||
type ContributorId = 'mcp_tools' | 'agent_descriptions' | 'claudemd_files'
|
||||
type ContributorId =
|
||||
| 'mcp_tools'
|
||||
| 'agent_descriptions'
|
||||
| 'claudemd_files'
|
||||
| 'ollama_context_length'
|
||||
|
||||
export type LocalModelContextContributor = {
|
||||
id: ContributorId
|
||||
@@ -56,24 +68,64 @@ function summarizeContextWarning(
|
||||
}
|
||||
}
|
||||
|
||||
export function buildLocalModelContextLoad(
|
||||
warnings: ContextWarnings | null | undefined,
|
||||
): LocalModelContextWarning | null {
|
||||
if (!warnings) {
|
||||
function summarizeOllamaContextWarning(
|
||||
warning: OllamaContextWarning,
|
||||
): LocalModelContextContributor {
|
||||
return {
|
||||
id: 'ollama_context_length',
|
||||
message: 'Ollama context length is too small',
|
||||
details: [
|
||||
`${warning.modelName}: active CONTEXT is ${warning.contextValue}`,
|
||||
'OpenClaude requests 32768 tokens for Ollama chats. If `ollama ps` keeps showing a smaller CONTEXT after a new request, restart Ollama and verify with `ollama ps`.',
|
||||
],
|
||||
summary: `Ollama CONTEXT: ${warning.contextValue} (OpenClaude requests 32K)`,
|
||||
}
|
||||
}
|
||||
|
||||
export function isLoopbackOllamaEndpoint(baseUrl: string | undefined): boolean {
|
||||
return isDirectLocalOllamaEndpoint(baseUrl)
|
||||
}
|
||||
|
||||
export function parseOllamaPsContextWarning(
|
||||
output: string,
|
||||
activeModelName?: string,
|
||||
): LocalModelContextContributor | null {
|
||||
const warning = parseOllamaPsContext(output, activeModelName)
|
||||
return warning ? summarizeOllamaContextWarning(warning) : null
|
||||
}
|
||||
|
||||
async function checkOllamaContextLength(
|
||||
baseUrl: string | undefined,
|
||||
activeModelName?: string,
|
||||
): Promise<LocalModelContextContributor | null> {
|
||||
if (!isLoopbackOllamaEndpoint(baseUrl)) {
|
||||
return null
|
||||
}
|
||||
|
||||
const warning = await checkOllamaPsContextWarning(activeModelName)
|
||||
if (!warning) {
|
||||
return null
|
||||
}
|
||||
|
||||
return summarizeOllamaContextWarning(warning)
|
||||
}
|
||||
|
||||
export function buildLocalModelContextLoad(
|
||||
warnings: ContextWarnings | null | undefined,
|
||||
extraContributors: LocalModelContextContributor[] = [],
|
||||
): LocalModelContextWarning | null {
|
||||
const contributors = [
|
||||
warnings.mcpWarning,
|
||||
warnings.agentWarning,
|
||||
warnings.claudeMdWarning,
|
||||
warnings?.mcpWarning,
|
||||
warnings?.agentWarning,
|
||||
warnings?.claudeMdWarning,
|
||||
]
|
||||
.filter((warning): warning is ContextWarning => warning !== null)
|
||||
.filter((warning): warning is ContextWarning => warning != null)
|
||||
.map(summarizeContextWarning)
|
||||
.filter(
|
||||
(contributor): contributor is LocalModelContextContributor =>
|
||||
contributor !== null,
|
||||
)
|
||||
.concat(extraContributors)
|
||||
|
||||
if (contributors.length === 0) {
|
||||
return null
|
||||
@@ -142,21 +194,29 @@ export async function checkLocalModelContextLoad(
|
||||
memoryFiles: MemoryFileInfo[],
|
||||
getToolPermissionContext: () => Promise<ToolPermissionContext>,
|
||||
baseUrl?: string,
|
||||
activeModelName?: string,
|
||||
): Promise<LocalModelContextWarning | null> {
|
||||
const resolvedBaseUrl = baseUrl ?? resolveActiveProviderBaseUrl()
|
||||
if (!isLocalProviderUrl(resolvedBaseUrl)) {
|
||||
return null
|
||||
}
|
||||
|
||||
const warnings = await checkContextWarnings(
|
||||
tools,
|
||||
agentDefinitions ?? null,
|
||||
getToolPermissionContext,
|
||||
{
|
||||
memoryFiles,
|
||||
mcpTokenStrategy: 'estimate',
|
||||
includeUnreachableRules: false,
|
||||
},
|
||||
const [warnings, ollamaContextContributor] = await Promise.all([
|
||||
checkContextWarnings(
|
||||
tools,
|
||||
agentDefinitions ?? null,
|
||||
getToolPermissionContext,
|
||||
{
|
||||
memoryFiles,
|
||||
mcpTokenStrategy: 'estimate',
|
||||
includeUnreachableRules: false,
|
||||
},
|
||||
),
|
||||
checkOllamaContextLength(resolvedBaseUrl, activeModelName),
|
||||
])
|
||||
|
||||
return buildLocalModelContextLoad(
|
||||
warnings,
|
||||
ollamaContextContributor ? [ollamaContextContributor] : [],
|
||||
)
|
||||
return buildLocalModelContextLoad(warnings)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user