> ## Documentation Index
> Fetch the complete documentation index at: https://docs.embedder.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Connect a local model
> Run Embedder against Ollama, LM Studio, vLLM, llama.cpp, or any OpenAI-compatible server you host yourself
Embedder can send its chat requests to a model server you run yourself instead of to a platform model. Those requests go straight from your machine to that server — they never pass through Embedder's backend.
Connect a local model when you want to keep your code on hardware you control, work on a restricted network, or evaluate open-weight models against your own firmware.
Local models are an organization feature. **Local models** appears in `/connect` only after an Embedder admin enables **Local Models** for your organization. If you don't see it, ask your admin.
## Supported servers
Any server that speaks the OpenAI `/v1/chat/completions` API works. Embedder recognizes four by name and applies server-specific handling to each:
| Server | Default base URL |
| ----------------------- | ------------------------ |
| Ollama | `http://localhost:11434` |
| LM Studio | `http://localhost:1234` |
| vLLM | `http://localhost:8000` |
| llama.cpp | `http://localhost:8080` |
| Other OpenAI-compatible | — |
If your server also implements `/v1/models`, Embedder lists everything it is serving automatically. If it doesn't, name the model yourself when you add the endpoint.
## Step 1: Start your server
Start the server with the context window you intend to use. Whatever you choose here is what you tell Embedder in the next step.
```bash theme={"system"}
OLLAMA_CONTEXT_LENGTH=32768 ollama serve
ollama pull qwen3:32b
```
Load your model, open the **Developer** tab, and start the local server. It listens on port 1234 by default.
Set the context length in the model's load settings before you start the server.
```bash theme={"system"}
vllm serve Qwen/Qwen3-32B --max-model-len 32768
```
```bash theme={"system"}
llama-server -m qwen3-32b.gguf -c 32768 --port 8080
```
Ollama's default context window is 4096 tokens, which is far too small for an agentic turn carrying a system prompt, tool schemas, and file contents. Always start `ollama serve` with `OLLAMA_CONTEXT_LENGTH` set to a realistic value.
## Step 2: Add the endpoint
In an Embedder session, run the `/connect` command to open the connections view.
Select **Local models**, described as "Ollama, LM Studio, vLLM, llama.cpp, or any OpenAI-compatible server".
| Field | Example | What it does |
| -------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------- |
| **Name** | `ollama` | Prefixes this endpoint's models everywhere in Embedder, so the model above becomes `ollama/qwen3:32b` |
| **Server** | Ollama | Enables server-specific handling and suggests the default port |
| **Base URL** | `http://localhost:11434` | With or without the trailing `/v1` — both work. Supports `${ENV_VAR}` |
| **Model** | `qwen3:32b` | Leave blank to list whatever the server reports |
| **Context window** | `32768` | The number of tokens the server was actually started with |
| **API key** | blank | Optional. Most local servers accept unauthenticated requests. Supports `${ENV_VAR}` |
| **Response timeout** | `300000` | How long to wait for the first response while a model cold-loads, in milliseconds |
Embedder confirms with **"ollama is ready to use"** and switches the session onto the endpoint's model straight away. That choice survives a restart until you pick something else.
## Step 3: Verify it works
Ask the model something small, such as "what files are in this project?", and confirm you get a reply. Two things tell you the request went to your own server:
* Running `/model` shows `qwen3:32b (ollama)` as the selected model.
* Your server logs a `/v1/chat/completions` request.
## Set the context window correctly
This is the one setting that causes trouble, so it is worth getting right.
Local servers do not report their context window over the OpenAI-compatible API, so Embedder believes the number you type. Declaring more than the server actually has is the dangerous direction: the server quietly drops the oldest turns and answers as though they never happened. It reads as the model forgetting rather than as an error, and Embedder's automatic compression never runs because it thinks there is still room.
Two things reduce the risk:
* **For Ollama**, Embedder asks the server what the model supports and caps your declared value at it. Declaring 128k against an 8k model becomes 8k.
* **For every other server**, your declared value is used as-is. Match it to the flag you started the server with (`--max-model-len`, `-c`, and so on).
Ollama applies a second ceiling of its own, `OLLAMA_CONTEXT_LENGTH`, which no endpoint reports and which defaults to 4096. If your model starts forgetting earlier turns despite a large declared window, this is almost always why.
## Choose a model
Run `/model` at any time. Local models appear as `qwen3:32b (ollama)` alongside platform models.
To pin one for a project, reference it as `local//` in `.embedder/models.json`:
```json .embedder/models.json theme={"system"}
{ "main": "local/ollama/qwen3:32b" }
```
A `main` entry in `.embedder/models.json` outranks the model you pick interactively. If you connect an endpoint while a pin is in place, the pin wins again on the next start — Embedder won't rewrite a file you checked in.
## Connect several endpoints
You can configure as many endpoints as you like. Each contributes models under its own prefix, so a laptop running Ollama and a lab machine running vLLM coexist without colliding:
```text theme={"system"}
ollama/qwen3:32b
lab/Qwen/Qwen3-32B
```
Run `/connect` again to add, edit, or remove any of them. To keep an endpoint configured but hidden from the model picker, set `enabled` to `false` in the configuration file described below.
## Keep API keys out of config files
If your server requires a key, reference an environment variable instead of typing the secret:
```text theme={"system"}
${LLM_API_KEY}
${LLM_API_KEY:-fallback-value}
```
Embedder resolves the variable on every request and stores only the literal `${LLM_API_KEY}` on disk, so the secret never lands in a config file. If you type the key itself instead, it is stored in plain text — the reference is what protects it.
### Where the reference goes
Type it into the field exactly as written, braces included:
| Field | Where to find it |
| ------------------ | ------------------------------------------------------------------- |
| **API key** | `/connect` → the endpoint form, under **Advanced** |
| **Base URL** | The same form, e.g. `http://${LAB_HOST}:8000` |
| **Custom headers** | No field for these — add them to `~/.embedder/local-endpoints.json` |
The API key field masks what you type, so you will see dots rather than the reference. It is stored correctly; reopen the endpoint to confirm.
### Where the variable goes
The simplest place is a `.env` file in the folder you have open:
```bash .env theme={"system"}
LLM_API_KEY=sk-your-key-here
```
Embedder runs with that folder as its working directory and reads the file on startup, so no shell configuration is needed.
Add `.env` to your `.gitignore`. Referencing a variable keeps the key out of Embedder's config, but a `.env` file sits inside your repository — without the ignore rule you have moved the secret rather than protected it.
This works for the folder you have open. If you use a different working directory, or run `embedder` from a terminal, set the variable in that environment instead:
```bash theme={"system"}
export LLM_API_KEY=sk-your-key-here
```
On macOS, an editor started from Spotlight or the Dock does not read your shell profile, so an `export` in `~/.zshrc` is invisible to it. A `.env` file in the open folder avoids the problem; otherwise use `launchctl setenv LLM_API_KEY …` and restart the editor. VS Code's `terminal.integrated.env` setting does not help here — it applies to integrated terminals, not to Embedder.
If a variable is unset, that endpoint is skipped with a warning rather than taking your whole model list down, and a request to it fails with a message naming the variable.
## What still reaches Embedder
Your prompts and the model's replies go directly between your machine and your server. Embedder's backend is still involved in a few things:
* **Session setup.** The first time you use a local model, Embedder contacts its backend and sends the model identifier, which includes the endpoint name you chose but not its URL or key. The result is cached, so a brief loss of connectivity afterwards is harmless.
* **Conversation compaction**, which runs on the platform even when your chat model is local.
* **Tools backed by Embedder services**, such as web search and document search.
## Advanced configuration
Endpoints live in `~/.embedder/local-endpoints.json`. The `/connect` form covers the common fields; edit the file directly for the rest. Your edits are preserved when you later change something through the form.
```json ~/.embedder/local-endpoints.json theme={"system"}
{
"version": 1,
"endpoints": [
{
"name": "ollama",
"label": "Workstation",
"kind": "ollama",
"baseUrl": "http://localhost:11434",
"apiKey": "${LLM_API_KEY}",
"headers": { "x-custom-header": "value" },
"timeoutMs": 300000,
"discover": true,
"enabled": true,
"defaultContext": 32768,
"defaultOutput": 4096,
"dropParams": ["stream_options"],
"models": [
{ "id": "qwen3:32b", "displayName": "Qwen3 32B", "context": 32768, "output": 4096 }
]
}
]
}
```
| Field | Purpose |
| ---------------------------------- | ------------------------------------------------------------------------------------------- |
| `label` | Friendly name shown in the model picker instead of `name` |
| `headers` | Extra headers sent on every request. Supports `${ENV_VAR}` |
| `discover` | Set to `false` to skip `/v1/models` and use only the models you declare |
| `enabled` | Set to `false` to keep an endpoint configured but hidden |
| `defaultContext` / `defaultOutput` | Limits applied to models that don't declare their own |
| `dropParams` | Request fields to strip before sending, for servers that reject fields they don't recognize |
| `models` | Declared models, each with an optional `displayName`, `context`, and `output` |
This file is separate from `connected-providers.json`, which stores sign-in details for hosted providers. Your base URL, API key, and custom headers stay on your machine.
## Limitations
* **Images and PDFs are not sent** to local vision models.
* **Compaction runs on the platform**, even when the chat model is local.
* **Only `/v1/chat/completions` is supported.** No embeddings, no `/v1/completions`, no reranking.
* **Tool-call quality is the model's responsibility.** Embedder repairs malformed tool calls, but a small model that cannot follow a tool schema will struggle regardless of configuration.
## Troubleshooting
The feature is not enabled for your organization. Ask an Embedder admin to turn on **Local Models**, then restart Embedder so it picks up the change.
Embedder could not open a connection. Check that the server is running, that the base URL and port are correct, and that nothing between the two is blocking the request. Confirm independently with `curl http://localhost:11434/v1/models`.
The declared context window is larger than what the server actually enforces, so the server is silently dropping the oldest messages. For Ollama, check `OLLAMA_CONTEXT_LENGTH`. For other servers, match your declared value to the flag you started the server with.
Your server rejects a request field that the OpenAI client always sends. Add that field name to `dropParams` for the endpoint and try again.
Large models can take minutes to cold-load. Raise **Response timeout**. The timeout bounds only the wait for the first response, so raising it never cuts a long answer short.
The saved model selection is stale, usually after an endpoint was renamed. Run `/model` and pick the model again.
Embedder needs a working connection to itself the first time you use a local model. Check your connection and try again. The result is cached, so later sessions are unaffected by short outages.