Connect a local model

Embedder can send its chat requests to a model server you run yourself instead of to a platform model. Those requests go straight from your machine to that server — they never pass through Embedder’s backend. Connect a local model when you want to keep your code on hardware you control, work on a restricted network, or evaluate open-weight models against your own firmware.

Local models are an organization feature. Local models appears in /connect only after an Embedder admin enables Local Models for your organization. If you don’t see it, ask your admin.

Supported servers

Any server that speaks the OpenAI /v1/chat/completions API works. Embedder recognizes four by name and applies server-specific handling to each:

Server	Default base URL
Ollama	`http://localhost:11434`
LM Studio	`http://localhost:1234`
vLLM	`http://localhost:8000`
llama.cpp	`http://localhost:8080`
Other OpenAI-compatible	—

If your server also implements /v1/models, Embedder lists everything it is serving automatically. If it doesn’t, name the model yourself when you add the endpoint.

Step 1: Start your server

Start the server with the context window you intend to use. Whatever you choose here is what you tell Embedder in the next step.

Ollama
LM Studio
vLLM
llama.cpp

OLLAMA_CONTEXT_LENGTH=32768 ollama serve
ollama pull qwen3:32b

vllm serve Qwen/Qwen3-32B --max-model-len 32768

llama-server -m qwen3-32b.gguf -c 32768 --port 8080

Ollama’s default context window is 4096 tokens, which is far too small for an agentic turn carrying a system prompt, tool schemas, and file contents. Always start ollama serve with OLLAMA_CONTEXT_LENGTH set to a realistic value.

Step 2: Add the endpoint

Run /connect

In an Embedder session, run the /connect command to open the connections view.

Choose Local models

Select Local models, described as “Ollama, LM Studio, vLLM, llama.cpp, or any OpenAI-compatible server”.

Fill in the form

Field	Example	What it does
Name	`ollama`	Prefixes this endpoint’s models everywhere in Embedder, so the model above becomes `ollama/qwen3:32b`
Server	Ollama	Enables server-specific handling and suggests the default port
Base URL	`http://localhost:11434`	With or without the trailing `/v1` — both work. Supports `${ENV_VAR}`
Model	`qwen3:32b`	Leave blank to list whatever the server reports
Context window	`32768`	The number of tokens the server was actually started with
API key	blank	Optional. Most local servers accept unauthenticated requests. Supports `${ENV_VAR}`
Response timeout	`300000`	How long to wait for the first response while a model cold-loads, in milliseconds

Save

Embedder confirms with “ollama is ready to use” and switches the session onto the endpoint’s model straight away. That choice survives a restart until you pick something else.

Step 3: Verify it works

Ask the model something small, such as “what files are in this project?”, and confirm you get a reply. Two things tell you the request went to your own server:

Running /model shows qwen3:32b (ollama) as the selected model.
Your server logs a /v1/chat/completions request.

Set the context window correctly

This is the one setting that causes trouble, so it is worth getting right. Local servers do not report their context window over the OpenAI-compatible API, so Embedder believes the number you type. Declaring more than the server actually has is the dangerous direction: the server quietly drops the oldest turns and answers as though they never happened. It reads as the model forgetting rather than as an error, and Embedder’s automatic compression never runs because it thinks there is still room. Two things reduce the risk:

For Ollama, Embedder asks the server what the model supports and caps your declared value at it. Declaring 128k against an 8k model becomes 8k.
For every other server, your declared value is used as-is. Match it to the flag you started the server with (--max-model-len, -c, and so on).

Ollama applies a second ceiling of its own, OLLAMA_CONTEXT_LENGTH, which no endpoint reports and which defaults to 4096. If your model starts forgetting earlier turns despite a large declared window, this is almost always why.

Choose a model

Run /model at any time. Local models appear as qwen3:32b (ollama) alongside platform models. To pin one for a project, reference it as local/<endpoint>/<model> in .embedder/models.json:

.embedder/models.json

{ "main": "local/ollama/qwen3:32b" }

A main entry in .embedder/models.json outranks the model you pick interactively. If you connect an endpoint while a pin is in place, the pin wins again on the next start — Embedder won’t rewrite a file you checked in.

Connect several endpoints

You can configure as many endpoints as you like. Each contributes models under its own prefix, so a laptop running Ollama and a lab machine running vLLM coexist without colliding:

ollama/qwen3:32b
lab/Qwen/Qwen3-32B

Run /connect again to add, edit, or remove any of them. To keep an endpoint configured but hidden from the model picker, set enabled to false in the configuration file described below.

Keep API keys out of config files

If your server requires a key, reference an environment variable instead of typing the secret:

${LLM_API_KEY}
${LLM_API_KEY:-fallback-value}

Embedder resolves the variable on every request and stores only the literal ${LLM_API_KEY} on disk, so the secret never lands in a config file. If you type the key itself instead, it is stored in plain text — the reference is what protects it.

Where the reference goes

Type it into the field exactly as written, braces included:

Field	Where to find it
API key	`/connect` → the endpoint form, under Advanced
Base URL	The same form, e.g. `http://${LAB_HOST}:8000`
Custom headers	No field for these — add them to `~/.embedder/local-endpoints.json`

The API key field masks what you type, so you will see dots rather than the reference. It is stored correctly; reopen the endpoint to confirm.

Where the variable goes

The simplest place is a .env file in the folder you have open:

.env

LLM_API_KEY=sk-your-key-here

Embedder runs with that folder as its working directory and reads the file on startup, so no shell configuration is needed.

Add .env to your .gitignore. Referencing a variable keeps the key out of Embedder’s config, but a .env file sits inside your repository — without the ignore rule you have moved the secret rather than protected it.

This works for the folder you have open. If you use a different working directory, or run embedder from a terminal, set the variable in that environment instead:

export LLM_API_KEY=sk-your-key-here

On macOS, an editor started from Spotlight or the Dock does not read your shell profile, so an export in ~/.zshrc is invisible to it. A .env file in the open folder avoids the problem; otherwise use launchctl setenv LLM_API_KEY … and restart the editor. VS Code’s terminal.integrated.env setting does not help here — it applies to integrated terminals, not to Embedder.

If a variable is unset, that endpoint is skipped with a warning rather than taking your whole model list down, and a request to it fails with a message naming the variable.

What still reaches Embedder

Your prompts and the model’s replies go directly between your machine and your server. Embedder’s backend is still involved in a few things:

Session setup. The first time you use a local model, Embedder contacts its backend and sends the model identifier, which includes the endpoint name you chose but not its URL or key. The result is cached, so a brief loss of connectivity afterwards is harmless.
Conversation compaction, which runs on the platform even when your chat model is local.
Tools backed by Embedder services, such as web search and document search.

Advanced configuration

Endpoints live in ~/.embedder/local-endpoints.json. The /connect form covers the common fields; edit the file directly for the rest. Your edits are preserved when you later change something through the form.

~/.embedder/local-endpoints.json

{
  "version": 1,
  "endpoints": [
    {
      "name": "ollama",
      "label": "Workstation",
      "kind": "ollama",
      "baseUrl": "http://localhost:11434",
      "apiKey": "${LLM_API_KEY}",
      "headers": { "x-custom-header": "value" },
      "timeoutMs": 300000,
      "discover": true,
      "enabled": true,
      "defaultContext": 32768,
      "defaultOutput": 4096,
      "dropParams": ["stream_options"],
      "models": [
        { "id": "qwen3:32b", "displayName": "Qwen3 32B", "context": 32768, "output": 4096 }
      ]
    }
  ]
}

Field	Purpose
`label`	Friendly name shown in the model picker instead of `name`
`headers`	Extra headers sent on every request. Supports `${ENV_VAR}`
`discover`	Set to `false` to skip `/v1/models` and use only the models you declare
`enabled`	Set to `false` to keep an endpoint configured but hidden
`defaultContext` / `defaultOutput`	Limits applied to models that don’t declare their own
`dropParams`	Request fields to strip before sending, for servers that reject fields they don’t recognize
`models`	Declared models, each with an optional `displayName`, `context`, and `output`

This file is separate from connected-providers.json, which stores sign-in details for hosted providers. Your base URL, API key, and custom headers stay on your machine.

Limitations

Images and PDFs are not sent to local vision models.
Compaction runs on the platform, even when the chat model is local.
Only /v1/chat/completions is supported. No embeddings, no /v1/completions, no reranking.
Tool-call quality is the model’s responsibility. Embedder repairs malformed tool calls, but a small model that cannot follow a tool schema will struggle regardless of configuration.

Troubleshooting

Local models doesn't appear in /connect

The feature is not enabled for your organization. Ask an Embedder admin to turn on Local Models, then restart Embedder so it picks up the change.

Local endpoint is not reachable, is the server running?

Embedder could not open a connection. Check that the server is running, that the base URL and port are correct, and that nothing between the two is blocking the request. Confirm independently with curl http://localhost:11434/v1/models.

The model forgets earlier turns

The declared context window is larger than what the server actually enforces, so the server is silently dropping the oldest messages. For Ollama, check OLLAMA_CONTEXT_LENGTH. For other servers, match your declared value to the flag you started the server with.

A 400 error naming an unknown field

Your server rejects a request field that the OpenAI client always sends. Add that field name to dropParams for the endpoint and try again.

The first request times out

Large models can take minutes to cold-load. Raise Response timeout. The timeout bounds only the wait for the first response, so raising it never cuts a long answer short.

Local model is missing its endpoint prefix

The saved model selection is stale, usually after an endpoint was renamed. Run /model and pick the model again.

A local model won't start a turn on first use

Embedder needs a working connection to itself the first time you use a local model. Check your connection and try again. The result is cached, so later sessions are unaffected by short outages.

Core Concepts

Customizations

Debug Mode

Headless

Integrations

Product Updates

Connect a local model

Supported servers

Step 1: Start your server

Step 2: Add the endpoint

Step 3: Verify it works

Set the context window correctly

Choose a model

Connect several endpoints

Keep API keys out of config files

Where the reference goes

Where the variable goes

What still reaches Embedder

Advanced configuration

Limitations

Troubleshooting

​Supported servers

​Step 1: Start your server

​Step 2: Add the endpoint

​Step 3: Verify it works

​Set the context window correctly

​Choose a model

​Connect several endpoints

​Keep API keys out of config files

​Where the reference goes

​Where the variable goes

​What still reaches Embedder

​Advanced configuration

​Limitations

​Troubleshooting

Supported servers

Step 1: Start your server

Step 2: Add the endpoint

Step 3: Verify it works

Set the context window correctly

Choose a model

Connect several endpoints

Keep API keys out of config files

Where the reference goes

Where the variable goes

What still reaches Embedder

Advanced configuration

Limitations

Troubleshooting