fix(ollama): preserve chat history with native context (#1805)

* fix(ollama): preserve chat history with native context

Route Ollama chat requests through the native /api/chat endpoint so OpenClaude can send request-level options.num_ctx instead of relying on Ollama's OpenAI-compatible shim.

Default the Ollama request context to 32768 tokens, support OPENCLAUDE_OLLAMA_NUM_CTX and OLLAMA_CONTEXT_LENGTH overrides, and map max tokens/temperature/top_p into native Ollama options.

Adapt native Ollama streaming and non-streaming responses back into the existing OpenAI-shaped conversion pipeline, including usage, text, structured tool calls, and tool_use stop reasons.

Normalize native Ollama request messages for images and historical tool calls, avoiding OpenAI-only image_url/id/type payload fields in /api/chat requests.

Add Ollama context diagnostics, loopback-only ollama ps status checks, regression coverage, and documentation for verifying active context length.

* fix(ollama): address native routing review feedback

* fix(ollama): restrict loopback host matching

* fix(ollama): exclude wildcard bind address

* fix(ollama): keep https localhost proxies on chat completions

---------

Co-authored-by: jatmn <12479882+jatmn@users.noreply.github.com>
This commit is contained in:
JATMN
2026-06-29 18:04:42 +08:00
committed by GitHub
co-authored by jatmn
parent b15660e388
commit 259c7ec27a
15 changed files with 963 additions and 104 deletions
+7
View File
@@ -198,6 +198,13 @@ $env:OPENAI_MODEL="qwen2.5-coder:7b"
openclaude
```
For Ollama, OpenClaude uses Ollama's native chat API and requests a 32768-token
context window on each chat request so same-session history is not silently
truncated by Ollama's OpenAI-compatible shim. Set `OPENCLAUDE_OLLAMA_NUM_CTX`
or `OLLAMA_CONTEXT_LENGTH` if you need a different request-level context size.
See [Advanced Setup](docs/advanced-setup.md#ollama-context-length) for
verification with `ollama ps`.
## Setup Guides
Beginner-friendly guides:
+48
View File
@@ -140,6 +140,53 @@ export OPENAI_BASE_URL=http://localhost:11434/v1
export OPENAI_MODEL=llama3.3:70b
```
#### Ollama Context Length
OpenClaude sends the current conversation history to Ollama on each turn and
uses Ollama's native chat API for Ollama endpoints. Native chat lets OpenClaude
send `options.num_ctx` with each request, so Ollama receives a 32768-token
context window by default instead of falling back to the smaller context often
used by Ollama's OpenAI-compatible `/v1/chat/completions` shim.
To choose a different request-level context size, set
`OPENCLAUDE_OLLAMA_NUM_CTX` before launching OpenClaude:
```bash
export OPENCLAUDE_OLLAMA_NUM_CTX=65536
```
You can also start Ollama with a global context length:
macOS / Linux:
```bash
# Stop any existing Ollama app/server first, then run:
OLLAMA_CONTEXT_LENGTH=32768 ollama serve
```
Windows PowerShell:
```powershell
# Quit any existing Ollama app/server first, then run:
$env:OLLAMA_CONTEXT_LENGTH="32768"
ollama serve
```
After a chat request, verify the loaded model is using the requested context:
```bash
ollama ps
```
Check the `CONTEXT` column. If it still shows a small value such as `4K` after a
new OpenClaude request, stop the existing Ollama app/server, start it again, and
retry the request.
Use a concrete recall test after changing the setting, such as asking the model
to repeat the first topic from the current chat. Questions like "do you remember our
conversation?" can trigger generic local-model disclaimers even when history is
present.
### Atomic Chat (local, Apple Silicon)
```bash
@@ -329,6 +376,7 @@ The **OpenClaude VS Code extension** can store the key in Secret Storage and set
| `OPENAI_MODEL` | OpenAI-compatible only | Model name such as `gpt-4o`, `deepseek-v4-flash`, or `llama3.3:70b` |
| `OPENAI_BASE_URL` | No | API endpoint, defaulting to `https://api.openai.com/v1` |
| `OPENAI_API_BASE` | No | Compatibility alias for `OPENAI_BASE_URL` |
| `OPENCLAUDE_OLLAMA_NUM_CTX` | Ollama only | Request-level Ollama context window. Defaults to `32768`; set a larger value for longer same-session history if your model and hardware can handle it. |
| `CLAUDE_CODE_OPENAI_CONTEXT_WINDOWS` | No | JSON map of OpenAI-compatible model names to context windows, such as `{"custom-model":1000000}`. Use this when a custom provider does not expose context metadata from `/v1/models`. |
| `OPENCODE_API_KEY` | OpenCode Zen / Go | Shared API key for OpenCode Zen (pay-as-you-go) and OpenCode Go (subscription); get yours from https://opencode.ai |
| `MIMO_API_KEY` | Xiaomi MiMo route | Xiaomi MiMo API key for `https://api.xiaomimimo.com/v1`; mirrored into the OpenAI-compatible auth env when the MiMo route is active |
+15
View File
@@ -72,6 +72,19 @@ openclaude
No API key is needed for Ollama local models.
OpenClaude asks Ollama for a 32768-token context window on each chat request.
If you need a different size, set `OPENCLAUDE_OLLAMA_NUM_CTX` before launching
OpenClaude, or start Ollama with a global context setting:
```bash
# Stop any existing Ollama app/server first, then run:
OLLAMA_CONTEXT_LENGTH=32768 ollama serve
```
After a chat request, run `ollama ps` in another terminal and check the
`CONTEXT` column. It should show the requested size. If it still shows a small
value such as `4K`, restart the Ollama app/server and try again.
### Option D: LM Studio
Install LM Studio first from:
@@ -132,6 +145,8 @@ Check the basics:
- make sure Ollama is installed
- make sure Ollama is running
- make sure the model was pulled successfully
- if same-session chat history appears missing, verify the active `CONTEXT`
value with `ollama ps`; OpenClaude requests 32K by default
### For LM Studio
+16
View File
@@ -68,6 +68,20 @@ openclaude
No API key is needed for Ollama local models.
OpenClaude asks Ollama for a 32768-token context window on each chat request.
If you need a different size, set `OPENCLAUDE_OLLAMA_NUM_CTX` before launching
OpenClaude, or start Ollama with a global context setting:
```powershell
# Quit any existing Ollama app/server first, then run:
$env:OLLAMA_CONTEXT_LENGTH="32768"
ollama serve
```
After a chat request, run `ollama ps` in another PowerShell window and check the
`CONTEXT` column. It should show the requested size. If it still shows a small
value such as `4K`, restart the Ollama app/server and try again.
### Option D: LM Studio
Install LM Studio first from:
@@ -145,6 +159,8 @@ Check the basics:
- make sure Ollama is installed
- make sure Ollama is running
- make sure the model was pulled successfully
- if same-session chat history appears missing, verify the active `CONTEXT`
value with `ollama ps`; OpenClaude requests 32K by default
### For LM Studio
+17
View File
@@ -82,6 +82,23 @@ describe('Session timeout fix', () => {
})
})
// ---------------------------------------------------------------------------
// Fix 2b: Ollama context history preservation
// ---------------------------------------------------------------------------
describe('Ollama context history fix', () => {
test('openaiShim uses native Ollama chat with request-level num_ctx', async () => {
const content = await file('services/api/openaiShim.ts').text()
expect(content).toContain('buildOllamaChatUrl')
expect(content).toContain('/api/chat')
expect(content).toContain('useNativeOllamaChat')
expect(content).toContain('num_ctx: getOllamaNumCtx()')
expect(content).toContain('normalizeOllamaNativeMessages(body.messages)')
expect(content).toContain('convertOllamaStreamingResponse')
expect(content).toContain('convertOllamaNonStreamingResponse')
})
})
// ---------------------------------------------------------------------------
// Fix 3: Agent loop continuation nudge
// ---------------------------------------------------------------------------
@@ -12,6 +12,8 @@ import {
buildLocalModelContextLoad,
checkLocalModelContextLoad,
isActiveProviderLocalModel,
isLoopbackOllamaEndpoint,
parseOllamaPsContextWarning,
} from '../utils/statusNoticeLocalModel.js'
const emptyPermissionContext = async () => getEmptyToolPermissionContext()
@@ -167,6 +169,89 @@ describe('buildLocalModelContextLoad', () => {
'planner: ~9,000 tokens',
])
})
test('includes Ollama context length contributor without other warnings', () => {
const result = buildLocalModelContextLoad(null, [
{
id: 'ollama_context_length',
message: 'Ollama context length is too small',
details: ['llama3.1:8b: active CONTEXT is 4K'],
summary: 'Ollama CONTEXT: 4K (OpenClaude requests 32K)',
},
])
expect(result?.lines).toEqual([
'Ollama CONTEXT: 4K (OpenClaude requests 32K)',
])
})
})
describe('parseOllamaPsContextWarning', () => {
test('warns when a loaded Ollama model has a small active context', () => {
const result = parseOllamaPsContextWarning(`
NAME ID SIZE PROCESSOR UNTIL CONTEXT
llama3.1:8b 46e0c10c039e 6.7 GB 100% GPU 4 minutes from now 4K
`)
expect(result).toMatchObject({
id: 'ollama_context_length',
summary: 'Ollama CONTEXT: 4K (OpenClaude requests 32K)',
})
expect(result?.details.join('\n')).toContain('OpenClaude requests 32768')
})
test('does not warn when the active context is already large enough', () => {
const result = parseOllamaPsContextWarning(`
NAME ID SIZE PROCESSOR UNTIL CONTEXT
llama3.1:8b 46e0c10c039e 6.7 GB 100% GPU 4 minutes from now 32K
`)
expect(result).toBeNull()
})
test('only warns for the active model when multiple Ollama models are loaded', () => {
const output = `
NAME ID SIZE PROCESSOR UNTIL CONTEXT
tinyllama:1b 46e0c10c039e 1.1 GB 100% GPU 4 minutes from now 4K
llama3.1:8b 7e0c10c039e46 6.7 GB 100% GPU 4 minutes from now 32K
`
expect(parseOllamaPsContextWarning(output, 'llama3.1:8b')).toBeNull()
expect(parseOllamaPsContextWarning(output, 'tinyllama:1b')).toMatchObject({
summary: 'Ollama CONTEXT: 4K (OpenClaude requests 32K)',
})
})
test('ignores older ollama ps output without a context column', () => {
const result = parseOllamaPsContextWarning(`
NAME ID SIZE PROCESSOR UNTIL
llama3.1:8b 46e0c10c039e 6.7 GB 100% GPU 4 minutes from now
`)
expect(result).toBeNull()
})
})
describe('isLoopbackOllamaEndpoint', () => {
test('allows local Ollama endpoints for local ollama ps diagnostics', () => {
expect(isLoopbackOllamaEndpoint('http://localhost:11434/v1')).toBe(true)
expect(isLoopbackOllamaEndpoint('http://127.0.0.1:11434/v1')).toBe(true)
expect(isLoopbackOllamaEndpoint('http://[::1]:11434/v1')).toBe(true)
})
test('skips remote Ollama endpoints so local ollama ps is not misleading', () => {
expect(isLoopbackOllamaEndpoint('http://10.0.0.5:11434/v1')).toBe(false)
expect(isLoopbackOllamaEndpoint('http://0.0.0.0:11434/v1')).toBe(false)
expect(isLoopbackOllamaEndpoint('http://ollama.lan:11434/v1')).toBe(false)
expect(isLoopbackOllamaEndpoint('https://localhost:11434/v1')).toBe(false)
expect(isLoopbackOllamaEndpoint('https://127.0.0.1:11434/v1')).toBe(false)
expect(isLoopbackOllamaEndpoint('https://ollama.example.com/v1')).toBe(false)
expect(isLoopbackOllamaEndpoint('http://127.0.0.1.nip.io:11434/v1')).toBe(false)
})
test('skips localhost proxies whose path merely contains ollama', () => {
expect(isLoopbackOllamaEndpoint('http://localhost:8080/ollama/v1')).toBe(false)
})
})
describe('checkLocalModelContextLoad', () => {
+4 -2
View File
@@ -45,6 +45,7 @@ export function StatusNotices(t0) {
const [memoryFiles, setMemoryFiles] = React.useState(cachedMemoryFiles);
const [localModelContextLoad, setLocalModelContextLoad] = React.useState<LocalModelContextWarning | null | undefined>(undefined);
const isLocalModel = isActiveProviderLocalModel();
const mainLoopModel = useAppState(s => s.mainLoopModel);
let t1;
if ($[0] === Symbol.for("react.memo_cache_sentinel")) {
t1 = () => {
@@ -72,6 +73,8 @@ export function StatusNotices(t0) {
agentDefinitions,
memoryFiles,
async () => toolPermissionContext,
undefined,
mainLoopModel ?? undefined,
).then(warning => {
if (!cancelled) {
setLocalModelContextLoad(warning);
@@ -84,10 +87,9 @@ export function StatusNotices(t0) {
return () => {
cancelled = true;
};
}, [agentDefinitions, isLocalModel, memoryFiles, toolPermissionContext, tools]);
}, [agentDefinitions, isLocalModel, mainLoopModel, memoryFiles, toolPermissionContext, tools]);
const t2 = getGlobalConfig();
const permissionMode = useAppState(s => s.toolPermissionContext.mode);
const mainLoopModel = useAppState(s => s.mainLoopModel);
const context: StatusNoticeContext = {
config: t2,
agentDefinitions,
@@ -128,7 +128,7 @@ test('redacts credentials in transport diagnostic URL logs', async () => {
expect(transportLog).toBeDefined()
const logLine = String(transportLog?.[0])
expect(logLine).toContain('url=http://redacted:redacted@localhost:11434/v1/chat/completions')
expect(logLine).toContain('url=http://redacted:redacted@localhost:11434/api/chat')
expect(logLine).not.toContain('user:supersecret')
expect(logLine).not.toContain('supersecret@')
})
@@ -201,8 +201,8 @@ test('logs self-heal localhost fallback with redacted from/to URLs', async () =>
expect(fallbackLog).toBeDefined()
const logLine = String(fallbackLog?.[0])
expect(logLine).toContain('from=http://redacted:redacted@localhost:11434/v1/chat/completions')
expect(logLine).toContain('to=http://redacted:redacted@127.0.0.1:11434/v1/chat/completions')
expect(logLine).toContain('from=http://redacted:redacted@localhost:11434/api/chat')
expect(logLine).toContain('to=http://redacted:redacted@127.0.0.1:11434/api/chat')
expect(logLine).not.toContain('supersecret')
})
@@ -26,7 +26,7 @@ type OpenAIShimClient = {
}
}
function makeSseResponse(lines: string[]): Response {
function makeOllamaNativeStreamingResponse(lines: string[]): Response {
const encoder = new TextEncoder()
return new Response(
new ReadableStream({
@@ -35,12 +35,12 @@ function makeSseResponse(lines: string[]): Response {
controller.close()
},
}),
{ headers: { 'Content-Type': 'text/event-stream' } },
{ headers: { 'Content-Type': 'application/x-ndjson' } },
)
}
function makeChunks(chunks: unknown[]): string[] {
return [...chunks.map(c => `data: ${JSON.stringify(c)}\n\n`), 'data: [DONE]\n\n']
function makeNdjsonChunks(chunks: unknown[]): string[] {
return chunks.map(c => `${JSON.stringify(c)}\n`)
}
describe('parseTextToolCalls', () => {
@@ -193,17 +193,32 @@ describe('parseTextToolCalls', () => {
// ---------------------------------------------------------------------------
const ollamaChunk = (content: string, finishReason?: string) => ({
id: 'chatcmpl-1',
object: 'chat.completion.chunk',
model: 'qwen2.5:7b',
choices: [{ index: 0, delta: { content }, finish_reason: finishReason ?? null }],
message: { role: 'assistant', content },
done: Boolean(finishReason),
...(finishReason ? { done_reason: finishReason } : {}),
})
const ollamaToolChunk = (toolCalls: unknown[], finishReason?: string) => ({
id: 'chatcmpl-1',
object: 'chat.completion.chunk',
model: 'qwen2.5:7b',
choices: [{ index: 0, delta: { tool_calls: toolCalls }, finish_reason: finishReason ?? null }],
message: {
role: 'assistant',
content: '',
tool_calls: toolCalls.map((toolCall) => {
const call = toolCall as {
function?: { name?: string; arguments?: unknown }
}
const args = call.function?.arguments
return {
function: {
name: call.function?.name,
arguments: typeof args === 'string' ? JSON.parse(args) : args,
},
}
}),
},
done: Boolean(finishReason),
...(finishReason ? { done_reason: finishReason } : {}),
})
describe('Ollama streaming — think-tag filtering on text-tool fallback (P1)', () => {
@@ -225,8 +240,8 @@ describe('Ollama streaming — think-tag filtering on text-tool fallback (P1)',
// Repro: model emits <think>private plan</think> followed by tool-call JSON.
// accumulatedText is raw; stripRanges leaves the <think> block unless we filter it.
globalThis.fetch = (async () =>
makeSseResponse(
makeChunks([
makeOllamaNativeStreamingResponse(
makeNdjsonChunks([
ollamaChunk('<think>private plan</think>{"name":"Bash","arguments":{"command":"ls"}}'),
ollamaChunk('', 'stop'),
]),
@@ -277,8 +292,8 @@ describe('Ollama streaming — plain text response with no tool calls', () => {
test('plain text in two chunks (content then stop) is emitted as text_delta', async () => {
globalThis.fetch = (async () =>
makeSseResponse(
makeChunks([
makeOllamaNativeStreamingResponse(
makeNdjsonChunks([
ollamaChunk('Hello from Ollama.'),
ollamaChunk('', 'stop'),
]),
@@ -309,8 +324,8 @@ describe('Ollama streaming — plain text response with no tool calls', () => {
test('plain text in single chunk (content + stop) is emitted as text_delta', async () => {
globalThis.fetch = (async () =>
makeSseResponse(
makeChunks([ollamaChunk('Hello from Ollama.', 'stop')]),
makeOllamaNativeStreamingResponse(
makeNdjsonChunks([ollamaChunk('Hello from Ollama.', 'stop')]),
)) as unknown as FetchType
const client = createOpenAIShimClient({}) as OpenAIShimClient
@@ -336,8 +351,8 @@ describe('Ollama streaming — plain text response with no tool calls', () => {
test('multi-chunk plain text (no tool calls) assembles correctly', async () => {
globalThis.fetch = (async () =>
makeSseResponse(
makeChunks([
makeOllamaNativeStreamingResponse(
makeNdjsonChunks([
ollamaChunk('Hello '),
ollamaChunk('from '),
ollamaChunk('Ollama.'),
@@ -386,14 +401,13 @@ describe('Ollama streaming — visible text before real structured tool_calls (P
// Repro: Ollama endpoint emits visible prose first, then real structured tool_calls.
// Before fix: ollamaTextBuffer was discarded when the text block closed.
globalThis.fetch = (async () =>
makeSseResponse(
makeChunks([
makeOllamaNativeStreamingResponse(
makeNdjsonChunks([
ollamaChunk('Let me check that.'),
ollamaToolChunk([
{ index: 0, id: 'call_1', type: 'function', function: { name: 'Bash', arguments: '{"command":"ls"}' } },
]),
{ id: 'chatcmpl-1', object: 'chat.completion.chunk', model: 'qwen2.5:7b',
choices: [{ index: 0, delta: {}, finish_reason: 'tool_calls' }] },
ollamaChunk('', 'stop'),
]),
)) as unknown as FetchType
@@ -444,8 +458,8 @@ describe('Ollama streaming — visible prose before text-based tool-call fallbac
// Before fix: hasEmittedContentStart === false in the fallback branch, so the prose in
// ollamaTextBuffer was discarded — only the synthetic tool_use block was emitted.
globalThis.fetch = (async () =>
makeSseResponse(
makeChunks([
makeOllamaNativeStreamingResponse(
makeNdjsonChunks([
ollamaChunk('I will inspect the file.\n'),
ollamaChunk('{"name":"Read","arguments":{"file_path":"/tmp/foo.ts"}}'),
ollamaChunk('', 'stop'),
@@ -513,8 +527,8 @@ describe('Ollama streaming — non-stop terminal finish reasons flush buffer', (
test('buffered text is flushed when finish_reason is "length"', async () => {
globalThis.fetch = (async () =>
makeSseResponse(
makeChunks([
makeOllamaNativeStreamingResponse(
makeNdjsonChunks([
ollamaChunk('Partial response cut off by'),
ollamaChunk('', 'length'),
]),
@@ -542,8 +556,8 @@ describe('Ollama streaming — non-stop terminal finish reasons flush buffer', (
test('text-tool JSON is extracted when finish_reason is "length" but finish_reason stays "length"', async () => {
globalThis.fetch = (async () =>
makeSseResponse(
makeChunks([
makeOllamaNativeStreamingResponse(
makeNdjsonChunks([
ollamaChunk('{"name":"Bash","arguments":{"command":"ls"}}'),
ollamaChunk('', 'length'),
]),
@@ -576,8 +590,8 @@ describe('Ollama streaming — non-stop terminal finish reasons flush buffer', (
test('buffered text is flushed when finish_reason is "content_filter"', async () => {
globalThis.fetch = (async () =>
makeSseResponse(
makeChunks([
makeOllamaNativeStreamingResponse(
makeNdjsonChunks([
ollamaChunk('Text stopped by content filter'),
ollamaChunk('', 'content_filter'),
]),
@@ -609,8 +623,8 @@ describe('Ollama streaming — non-stop terminal finish reasons flush buffer', (
test('buffered text is flushed when finish_reason is "safety"', async () => {
globalThis.fetch = (async () =>
makeSseResponse(
makeChunks([
makeOllamaNativeStreamingResponse(
makeNdjsonChunks([
ollamaChunk('Text stopped by safety check'),
ollamaChunk('', 'safety'),
]),
+78 -44
View File
@@ -1231,28 +1231,21 @@ test('uses max_tokens instead of max_completion_tokens for local providers', asy
globalThis.fetch = (async (_input, init) => {
const body = JSON.parse(String(init?.body))
expect(body.max_tokens).toBe(64)
expect(body.max_completion_tokens).toBeUndefined()
expect(body.options?.num_predict).toBe(64)
expect(body.options?.num_ctx).toBe(32768)
expect(body.stream_options).toBeUndefined()
return new Response(
JSON.stringify({
id: 'chatcmpl-1',
model: 'llama3.1:8b',
choices: [
{
message: {
role: 'assistant',
content: 'hello',
},
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 5,
completion_tokens: 1,
total_tokens: 6,
message: {
role: 'assistant',
content: 'hello',
},
done: true,
done_reason: 'stop',
prompt_eval_count: 5,
eval_count: 1,
}),
{
headers: {
@@ -5707,11 +5700,11 @@ test('self-heals localhost resolution failures by retrying local loopback base U
}),
).resolves.toBeDefined()
expect(requestUrls[0]).toBe('http://localhost:11434/v1/chat/completions')
expect(requestUrls).toContain('http://127.0.0.1:11434/v1/chat/completions')
expect(requestUrls[0]).toBe('http://localhost:11434/api/chat')
expect(requestUrls).toContain('http://127.0.0.1:11434/api/chat')
})
test('self-heals local endpoint_not_found by retrying with /v1 base URL', async () => {
test('uses native Ollama chat endpoint when local base URL omits /v1', async () => {
process.env.OPENAI_BASE_URL = 'http://localhost:11434'
const requestUrls: string[] = []
@@ -5719,33 +5712,17 @@ test('self-heals local endpoint_not_found by retrying with /v1 base URL', async
const url = typeof input === 'string' ? input : input.url
requestUrls.push(url)
if (url === 'http://localhost:11434/chat/completions') {
return new Response('Not Found', {
status: 404,
headers: {
'Content-Type': 'text/plain',
},
})
}
return new Response(
JSON.stringify({
id: 'chatcmpl-1',
model: 'qwen2.5-coder:7b',
choices: [
{
message: {
role: 'assistant',
content: 'hello from /v1',
},
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 5,
completion_tokens: 2,
total_tokens: 7,
message: {
role: 'assistant',
content: 'hello from native Ollama',
},
done: true,
done_reason: 'stop',
prompt_eval_count: 5,
eval_count: 2,
}),
{
status: 200,
@@ -5767,9 +5744,66 @@ test('self-heals local endpoint_not_found by retrying with /v1 base URL', async
}),
).resolves.toBeDefined()
expect(requestUrls).toEqual(['http://localhost:11434/api/chat'])
})
test('keeps remote Ollama-named gateways on chat completions', async () => {
process.env.OPENAI_BASE_URL = 'https://ollama-gateway.example.com/v1'
const requestUrls: string[] = []
globalThis.fetch = (async (input, init) => {
const url = typeof input === 'string' ? input : input.url
requestUrls.push(url)
const body = JSON.parse(String(init?.body)) as Record<string, unknown>
expect(body.max_tokens).toBe(64)
expect(body.options).toBeUndefined()
return makeChatCompletionResponse('llama3.1:8b')
}) as unknown as FetchType
const client = createOpenAIShimClient({}) as OpenAIShimClient
await expect(
client.beta.messages.create({
model: 'llama3.1:8b',
messages: [{ role: 'user', content: 'hello' }],
max_tokens: 64,
stream: false,
}),
).resolves.toBeDefined()
expect(requestUrls).toEqual([
'http://localhost:11434/chat/completions',
'http://localhost:11434/v1/chat/completions',
'https://ollama-gateway.example.com/v1/chat/completions',
])
})
test('keeps HTTPS localhost Ollama-port proxies on chat completions', async () => {
process.env.OPENAI_BASE_URL = 'https://localhost:11434/v1'
const requestUrls: string[] = []
globalThis.fetch = (async (input, init) => {
const url = typeof input === 'string' ? input : input.url
requestUrls.push(url)
const body = JSON.parse(String(init?.body)) as Record<string, unknown>
expect(body.max_tokens).toBe(64)
expect(body.options).toBeUndefined()
return makeChatCompletionResponse('llama3.1:8b')
}) as unknown as FetchType
const client = createOpenAIShimClient({}) as OpenAIShimClient
await expect(
client.beta.messages.create({
model: 'llama3.1:8b',
messages: [{ role: 'user', content: 'hello' }],
max_tokens: 64,
stream: false,
}),
).resolves.toBeDefined()
expect(requestUrls).toEqual([
'https://localhost:11434/v1/chat/completions',
])
})
+431 -1
View File
@@ -77,6 +77,7 @@ import {
getLocalFastPathConfig,
getLocalProviderRetryBaseUrls,
getGithubEndpointType,
isDirectLocalOllamaEndpoint,
isLikelyOllamaEndpoint,
isLocalProviderUrl,
resolveRuntimeCodexCredentials,
@@ -109,6 +110,7 @@ import {
hasInvalidCredentialPlaceholder,
parseCredentialList,
} from './credentialPool.js'
import { MIN_RECOMMENDED_OLLAMA_CONTEXT_TOKENS } from '../../utils/ollamaContext.js'
const GITHUB_429_MAX_RETRIES = 3
const GITHUB_429_BASE_DELAY_SEC = 1
@@ -300,6 +302,399 @@ interface OpenAITool {
}
}
type OllamaChatResponse = {
model?: string
message?: {
role?: string
content?: string
tool_calls?: Array<{
function?: {
name?: string
arguments?: unknown
}
}>
}
done?: boolean
done_reason?: string
prompt_eval_count?: number
eval_count?: number
}
type OllamaChatMessage = Omit<OpenAIMessage, 'content' | 'tool_calls'> & {
content?: string
images?: string[]
tool_calls?: Array<{
function: {
name: string
arguments: Record<string, unknown>
}
}>
}
function parsePositiveIntegerEnv(value: string | undefined): number | null {
if (!value?.trim()) {
return null
}
const parsed = Number(value.trim())
if (!Number.isInteger(parsed) || parsed <= 0) {
return null
}
return parsed
}
function getOllamaNumCtx(): number {
return (
parsePositiveIntegerEnv(process.env.OPENCLAUDE_OLLAMA_NUM_CTX) ??
parsePositiveIntegerEnv(process.env.OLLAMA_CONTEXT_LENGTH) ??
MIN_RECOMMENDED_OLLAMA_CONTEXT_TOKENS
)
}
function buildOllamaChatUrl(baseUrl: string): string {
const parsed = new URL(baseUrl)
parsed.pathname = parsed.pathname.replace(/\/+$/, '').replace(/\/v1$/i, '')
parsed.pathname = `${parsed.pathname.replace(/\/+$/, '')}/api/chat`
parsed.search = ''
parsed.hash = ''
return parsed.toString()
}
function extractOllamaImageData(url: string): string | null {
const match = url.match(/^data:[^;,]+;base64,(.+)$/i)
if (!match) {
return null
}
return match[1]
}
function normalizeOllamaNativeToolCalls(
toolCalls: OpenAIMessage['tool_calls'],
): OllamaChatMessage['tool_calls'] {
if (!Array.isArray(toolCalls) || toolCalls.length === 0) {
return undefined
}
const normalized = toolCalls
.map(toolCall => {
const name = toolCall.function?.name
if (!name) {
return null
}
let args: Record<string, unknown> = {}
try {
const parsed = JSON.parse(toolCall.function.arguments || '{}')
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
args = parsed as Record<string, unknown>
}
} catch {
args = {}
}
return {
function: {
name,
arguments: args,
},
}
})
.filter((toolCall): toolCall is NonNullable<typeof toolCall> => toolCall !== null)
return normalized.length > 0 ? normalized : undefined
}
function normalizeOllamaNativeMessages(messages: unknown): OllamaChatMessage[] {
if (!Array.isArray(messages)) {
return []
}
return messages.map(message => {
const openAIMessage = message as OpenAIMessage
const content = openAIMessage.content
const toolCalls = normalizeOllamaNativeToolCalls(openAIMessage.tool_calls)
if (!Array.isArray(content)) {
return {
...openAIMessage,
content,
...(toolCalls ? { tool_calls: toolCalls } : { tool_calls: undefined }),
}
}
const textParts: string[] = []
const images: string[] = []
for (const part of content) {
if (part.type === 'text') {
if (part.text) {
textParts.push(part.text)
}
continue
}
if (part.type === 'image_url') {
const imageUrl = part.image_url.url
const imageData = extractOllamaImageData(imageUrl)
if (imageData) {
images.push(imageData)
} else {
textParts.push(`[Image: ${imageUrl}]`)
}
}
}
return {
...openAIMessage,
content: textParts.join('\n'),
...(images.length > 0 ? { images } : {}),
...(toolCalls ? { tool_calls: toolCalls } : { tool_calls: undefined }),
}
})
}
function mapOllamaDoneReason(doneReason: unknown): string | null {
if (doneReason === 'length') return 'length'
if (doneReason === 'stop') return 'stop'
if (typeof doneReason === 'string' && doneReason) return doneReason
return null
}
function normalizeOllamaToolCalls(
toolCalls: NonNullable<OllamaChatResponse['message']>['tool_calls'],
): Array<{
id: string
type: 'function'
function: { name: string; arguments: string }
}> | undefined {
if (!Array.isArray(toolCalls) || toolCalls.length === 0) {
return undefined
}
const normalized = toolCalls
.map(toolCall => {
const name = toolCall.function?.name
if (!name) {
return null
}
const args = toolCall.function?.arguments
return {
id: `call_${crypto.randomUUID().replace(/-/g, '').slice(0, 24)}`,
type: 'function' as const,
function: {
name,
arguments:
typeof args === 'string' ? args : JSON.stringify(args ?? {}),
},
}
})
.filter((toolCall): toolCall is NonNullable<typeof toolCall> => toolCall !== null)
return normalized.length > 0 ? normalized : undefined
}
function buildOpenAIUsageFromOllama(data: OllamaChatResponse) {
const promptTokens = data.prompt_eval_count ?? 0
const completionTokens = data.eval_count ?? 0
return {
prompt_tokens: promptTokens,
completion_tokens: completionTokens,
total_tokens: promptTokens + completionTokens,
}
}
function convertOllamaChatResponseToOpenAI(
data: OllamaChatResponse,
fallbackModel: string,
): Record<string, unknown> {
const toolCalls = normalizeOllamaToolCalls(data.message?.tool_calls)
return {
id: makeMessageId(),
object: 'chat.completion',
created: Math.floor(Date.now() / 1000),
model: data.model ?? fallbackModel,
choices: [
{
index: 0,
message: {
role: 'assistant',
content: data.message?.content ?? '',
...(toolCalls ? { tool_calls: toolCalls } : {}),
},
finish_reason: mapOllamaDoneReason(data.done_reason),
},
],
usage: buildOpenAIUsageFromOllama(data),
}
}
function responseWithPreservedUrl(
body: BodyInit | null,
init: ResponseInit,
url: string,
): Response {
const response = new Response(body, init)
try {
Object.defineProperty(response, 'url', {
value: url,
configurable: true,
})
} catch {
/* some runtimes lock the property; downstream has transport fallback */
}
return response
}
async function convertOllamaNonStreamingResponse(
response: Response,
fallbackModel: string,
): Promise<Response> {
const data = await response.json() as OllamaChatResponse
return responseWithPreservedUrl(
JSON.stringify(convertOllamaChatResponseToOpenAI(data, fallbackModel)),
{
status: response.status,
statusText: response.statusText,
headers: { 'content-type': 'application/json' },
},
response.url,
)
}
function openAIStreamChunk(
id: string,
model: string,
delta: Record<string, unknown>,
finishReason: string | null = null,
): string {
return `data: ${JSON.stringify({
id,
object: 'chat.completion.chunk',
created: Math.floor(Date.now() / 1000),
model,
choices: [{ index: 0, delta, finish_reason: finishReason }],
})}\n\n`
}
function convertOllamaStreamingResponse(
response: Response,
fallbackModel: string,
): Response {
const body = response.body
if (!body) {
return response
}
const decoder = new TextDecoder()
const encoder = new TextEncoder()
const reader = body.getReader()
const streamId = makeMessageId()
let buffer = ''
let hasEmittedRole = false
let hasEmittedToolCall = false
const stream = new ReadableStream<Uint8Array>({
async pull(controller) {
while (true) {
const { done, value } = await reader.read()
if (done) {
if (buffer.trim()) {
enqueueOllamaLineAsOpenAI(buffer.trim(), controller)
buffer = ''
}
controller.enqueue(encoder.encode('data: [DONE]\n\n'))
controller.close()
return
}
buffer += decoder.decode(value, { stream: true })
const lines = buffer.split(/\r?\n/)
buffer = lines.pop() ?? ''
let emittedLine = false
for (const line of lines) {
if (line.trim()) {
enqueueOllamaLineAsOpenAI(line.trim(), controller)
emittedLine = true
}
}
if (emittedLine) {
return
}
}
},
cancel(reason) {
return reader.cancel(reason)
},
})
function enqueueOllamaLineAsOpenAI(
line: string,
controller: ReadableStreamDefaultController<Uint8Array>,
): void {
let data: OllamaChatResponse
try {
data = JSON.parse(line) as OllamaChatResponse
} catch {
return
}
const model = data.model ?? fallbackModel
const chunks: string[] = []
const delta: Record<string, unknown> = {}
if (!hasEmittedRole) {
delta.role = 'assistant'
hasEmittedRole = true
}
if (data.message?.content) {
delta.content = data.message.content
}
const toolCalls = normalizeOllamaToolCalls(data.message?.tool_calls)
if (toolCalls) {
hasEmittedToolCall = true
delta.tool_calls = toolCalls.map((toolCall, index) => ({
index,
id: toolCall.id,
type: toolCall.type,
function: toolCall.function,
}))
}
if (Object.keys(delta).length > 0) {
chunks.push(openAIStreamChunk(streamId, model, delta))
}
if (data.done) {
chunks.push(openAIStreamChunk(
streamId,
model,
{},
hasEmittedToolCall
? 'tool_calls'
: mapOllamaDoneReason(data.done_reason),
))
chunks.push(`data: ${JSON.stringify({
id: streamId,
object: 'chat.completion.chunk',
created: Math.floor(Date.now() / 1000),
model,
choices: [],
usage: buildOpenAIUsageFromOllama(data),
})}\n\n`)
}
for (const chunk of chunks) {
controller.enqueue(encoder.encode(chunk))
}
}
return responseWithPreservedUrl(
stream,
{
status: response.status,
statusText: response.statusText,
headers: { 'content-type': 'text/event-stream' },
},
response.url,
)
}
function convertSystemPrompt(
system: unknown,
): string {
@@ -2676,6 +3071,11 @@ class OpenAIShimMessages {
: shimConfig.endpointPath?.startsWith('/models/gemini-')
? 'gemini'
: request.transport
const useNativeOllamaChat =
effectiveTransport === 'chat_completions' &&
!shimConfig.endpointPath &&
isDirectLocalOllamaEndpoint(request.baseUrl) &&
isLikelyOllamaEndpoint(request.baseUrl)
const openaiMessages = convertMessages(compressedMessages, params.system, {
preserveReasoningContent: shimConfig.preserveReasoningContent,
reasoningContentFallback: shimConfig.reasoningContentFallback,
@@ -3317,6 +3717,9 @@ class OpenAIShimMessages {
if (shimConfig.endpointPath) {
return `${baseUrl}${shimConfig.endpointPath}`
}
if (useNativeOllamaChat) {
return buildOllamaChatUrl(baseUrl)
}
return request.transport === 'responses' || request.transport === 'responses_compat'
? `${baseUrl}/responses`
: buildChatCompletionsUrl(baseUrl)
@@ -3381,9 +3784,31 @@ class OpenAIShimMessages {
// Local backends do not implement prefix caching, so the deep key-sort
// is pure CPU overhead per request (issue #1016). Drop to the native
// `JSON.stringify` fast path when the fast-path config opts out.
const buildOllamaChatBody = (): Record<string, unknown> => {
const options: Record<string, unknown> = {
num_ctx: getOllamaNumCtx(),
}
if (body.max_tokens !== undefined) {
options.num_predict = body.max_tokens
} else if (body.max_completion_tokens !== undefined) {
options.num_predict = body.max_completion_tokens
}
if (params.temperature !== undefined) options.temperature = params.temperature
if (params.top_p !== undefined) options.top_p = params.top_p
return {
model: request.resolvedModel,
messages: normalizeOllamaNativeMessages(body.messages),
stream: params.stream ?? false,
options,
...(body.tools ? { tools: body.tools } : {}),
}
}
const serializeBody = (): string => {
const payload =
effectiveTransport === 'responses' || effectiveTransport === 'responses_compat' ? buildResponsesBody()
useNativeOllamaChat ? buildOllamaChatBody()
: effectiveTransport === 'responses' || effectiveTransport === 'responses_compat' ? buildResponsesBody()
: effectiveTransport === 'anthropic_messages' ? buildAnthropicMessagesBody()
: effectiveTransport === 'gemini' ? buildGeminiBody()
: body
@@ -3551,6 +3976,11 @@ class OpenAIShimMessages {
if (response.ok) {
credentialPool?.reportSuccess(credentialLease)
if (useNativeOllamaChat) {
response = params.stream
? convertOllamaStreamingResponse(response, request.resolvedModel)
: await convertOllamaNonStreamingResponse(response, request.resolvedModel)
}
let tokensIn = 0
let tokensOut = 0
// Skip clone() for streaming responses - it blocks until full body is received,
@@ -54,13 +54,16 @@ afterEach(() => {
test('treats localhost endpoints as local', () => {
expect(isLocalProviderUrl('http://localhost:11434/v1')).toBe(true)
expect(isLocalProviderUrl('http://127.0.0.1:11434/v1')).toBe(true)
expect(isLocalProviderUrl('http://0.0.0.0:11434/v1')).toBe(true)
// Full 127.0.0.0/8 loopback range should be treated as local
expect(isLocalProviderUrl('http://127.0.0.2:11434/v1')).toBe(true)
expect(isLocalProviderUrl('http://127.1.2.3:11434/v1')).toBe(true)
expect(isLocalProviderUrl('http://127.255.255.255:11434/v1')).toBe(true)
})
test('does not treat wildcard bind addresses as local endpoints', () => {
expect(isLocalProviderUrl('http://0.0.0.0:11434/v1')).toBe(false)
})
test('treats private IPv4 endpoints as local', () => {
expect(isLocalProviderUrl('http://10.0.0.1:11434/v1')).toBe(true)
expect(isLocalProviderUrl('http://172.16.0.1:11434/v1')).toBe(true)
+28 -1
View File
@@ -419,7 +419,7 @@ export function isLocalProviderUrl(baseUrl: string | undefined): boolean {
hostname = hostname.slice(0, zoneIdIndex)
}
if (LOCALHOST_HOSTNAMES.has(hostname) || hostname === '0.0.0.0') {
if (LOCALHOST_HOSTNAMES.has(hostname)) {
return true
}
if (hostname.endsWith('.local')) {
@@ -543,6 +543,33 @@ export function isLikelyOllamaEndpoint(baseUrl: string | undefined): boolean {
}
}
export function isDirectLocalOllamaEndpoint(baseUrl: string | undefined): boolean {
if (!baseUrl) return false
try {
const parsed = new URL(baseUrl)
let hostname = parsed.hostname.toLowerCase()
if (hostname.startsWith('[') && hostname.endsWith(']')) {
hostname = hostname.slice(1, -1)
}
const ipv4Octets = hostname.split('.')
const isLoopbackIpv4 =
ipv4Octets.length === 4 &&
ipv4Octets.every(octet => /^\d{1,3}$/.test(octet) && Number(octet) <= 255) &&
ipv4Octets[0] === '127'
return (
parsed.protocol === 'http:' &&
parsed.port === '11434' &&
(
hostname === 'localhost' ||
hostname === '::1' ||
isLoopbackIpv4
)
)
} catch {
return false
}
}
export function getLocalProviderRetryBaseUrls(baseUrl: string): string[] {
if (!isLocalProviderUrl(baseUrl)) {
return []
+101
View File
@@ -0,0 +1,101 @@
import { execFileNoThrow } from './execFileNoThrow.js'
export const MIN_RECOMMENDED_OLLAMA_CONTEXT_TOKENS = 32_768
export type OllamaContextWarning = {
modelName: string
contextValue: string
contextTokens: number
}
function parseContextTokenValue(value: string | undefined): number | null {
const normalized = value?.trim().replace(/,/g, '')
if (!normalized) {
return null
}
const match = normalized.match(/^(\d+(?:\.\d+)?)([kKmM]?)$/)
if (!match) {
return null
}
const amount = Number(match[1])
if (!Number.isFinite(amount) || amount <= 0) {
return null
}
const suffix = match[2].toLowerCase()
if (suffix === 'm') {
return Math.round(amount * 1_000_000)
}
if (suffix === 'k') {
return Math.round(amount * 1_024)
}
return Math.round(amount)
}
function normalizeOllamaModelName(modelName: string | undefined): string | null {
const normalized = modelName?.trim().toLowerCase().split('?')[0]
return normalized || null
}
export function parseOllamaPsContextWarning(
output: string,
activeModelName?: string,
): OllamaContextWarning | null {
const normalizedActiveModel = normalizeOllamaModelName(activeModelName)
const lines = output
.split(/\r?\n/)
.map(line => line.trim())
.filter(Boolean)
if (lines.length < 2) {
return null
}
const header = lines[0]
const contextColumnStart = header.toLowerCase().indexOf('context')
if (contextColumnStart === -1) {
return null
}
for (const line of lines.slice(1)) {
const modelName = line.split(/\s+/)[0] ?? 'loaded model'
if (
normalizedActiveModel &&
normalizeOllamaModelName(modelName) !== normalizedActiveModel
) {
continue
}
const contextValue = line.slice(contextColumnStart).trim().split(/\s+/)[0]
const contextTokens = parseContextTokenValue(contextValue)
if (
contextTokens !== null &&
contextTokens < MIN_RECOMMENDED_OLLAMA_CONTEXT_TOKENS
) {
return {
modelName,
contextValue,
contextTokens,
}
}
}
return null
}
export async function checkOllamaPsContextWarning(
activeModelName?: string,
): Promise<OllamaContextWarning | null> {
const result = await execFileNoThrow('ollama', ['ps'], {
timeout: 1000,
preserveOutputOnError: true,
useCwd: false,
})
if (result.code !== 0 || !result.stdout.trim()) {
return null
}
return parseOllamaPsContextWarning(result.stdout, activeModelName)
}
+80 -20
View File
@@ -1,5 +1,8 @@
import { resolveActiveRouteIdFromEnv } from '../integrations/routeMetadata.js'
import { isLocalProviderUrl } from '../services/api/providerConfig.js'
import {
isDirectLocalOllamaEndpoint,
isLocalProviderUrl,
} from '../services/api/providerConfig.js'
import type { Tool, ToolPermissionContext } from '../Tool.js'
import type { AgentDefinitionsResult } from '../tools/AgentTool/loadAgentsDir.js'
import type { MemoryFileInfo } from './claudemd.js'
@@ -11,8 +14,17 @@ import {
import { formatTokens } from './format.js'
import { isEnvTruthy } from './envUtils.js'
import { plural } from './stringUtils.js'
import {
type OllamaContextWarning,
checkOllamaPsContextWarning,
parseOllamaPsContextWarning as parseOllamaPsContext,
} from './ollamaContext.js'
type ContributorId = 'mcp_tools' | 'agent_descriptions' | 'claudemd_files'
type ContributorId =
| 'mcp_tools'
| 'agent_descriptions'
| 'claudemd_files'
| 'ollama_context_length'
export type LocalModelContextContributor = {
id: ContributorId
@@ -56,24 +68,64 @@ function summarizeContextWarning(
}
}
export function buildLocalModelContextLoad(
warnings: ContextWarnings | null | undefined,
): LocalModelContextWarning | null {
if (!warnings) {
function summarizeOllamaContextWarning(
warning: OllamaContextWarning,
): LocalModelContextContributor {
return {
id: 'ollama_context_length',
message: 'Ollama context length is too small',
details: [
`${warning.modelName}: active CONTEXT is ${warning.contextValue}`,
'OpenClaude requests 32768 tokens for Ollama chats. If `ollama ps` keeps showing a smaller CONTEXT after a new request, restart Ollama and verify with `ollama ps`.',
],
summary: `Ollama CONTEXT: ${warning.contextValue} (OpenClaude requests 32K)`,
}
}
export function isLoopbackOllamaEndpoint(baseUrl: string | undefined): boolean {
return isDirectLocalOllamaEndpoint(baseUrl)
}
export function parseOllamaPsContextWarning(
output: string,
activeModelName?: string,
): LocalModelContextContributor | null {
const warning = parseOllamaPsContext(output, activeModelName)
return warning ? summarizeOllamaContextWarning(warning) : null
}
async function checkOllamaContextLength(
baseUrl: string | undefined,
activeModelName?: string,
): Promise<LocalModelContextContributor | null> {
if (!isLoopbackOllamaEndpoint(baseUrl)) {
return null
}
const warning = await checkOllamaPsContextWarning(activeModelName)
if (!warning) {
return null
}
return summarizeOllamaContextWarning(warning)
}
export function buildLocalModelContextLoad(
warnings: ContextWarnings | null | undefined,
extraContributors: LocalModelContextContributor[] = [],
): LocalModelContextWarning | null {
const contributors = [
warnings.mcpWarning,
warnings.agentWarning,
warnings.claudeMdWarning,
warnings?.mcpWarning,
warnings?.agentWarning,
warnings?.claudeMdWarning,
]
.filter((warning): warning is ContextWarning => warning !== null)
.filter((warning): warning is ContextWarning => warning != null)
.map(summarizeContextWarning)
.filter(
(contributor): contributor is LocalModelContextContributor =>
contributor !== null,
)
.concat(extraContributors)
if (contributors.length === 0) {
return null
@@ -142,21 +194,29 @@ export async function checkLocalModelContextLoad(
memoryFiles: MemoryFileInfo[],
getToolPermissionContext: () => Promise<ToolPermissionContext>,
baseUrl?: string,
activeModelName?: string,
): Promise<LocalModelContextWarning | null> {
const resolvedBaseUrl = baseUrl ?? resolveActiveProviderBaseUrl()
if (!isLocalProviderUrl(resolvedBaseUrl)) {
return null
}
const warnings = await checkContextWarnings(
tools,
agentDefinitions ?? null,
getToolPermissionContext,
{
memoryFiles,
mcpTokenStrategy: 'estimate',
includeUnreachableRules: false,
},
const [warnings, ollamaContextContributor] = await Promise.all([
checkContextWarnings(
tools,
agentDefinitions ?? null,
getToolPermissionContext,
{
memoryFiles,
mcpTokenStrategy: 'estimate',
includeUnreachableRules: false,
},
),
checkOllamaContextLength(resolvedBaseUrl, activeModelName),
])
return buildLocalModelContextLoad(
warnings,
ollamaContextContributor ? [ollamaContextContributor] : [],
)
return buildLocalModelContextLoad(warnings)
}