Connecting to a remote local LLM

[Maple498]

Autogen uses the OpenAI API by default. I've configured DeepSeek on another computer (not the one running Autogen) using Ollam, and I want this computer to be able to remotely use LLM. How can I do this? I tried setting the base_url, but it didn't work. Does anyone have a ready-made script example?

[alexmercer-ai]

Yes, this is totally doable. AutoGen supports any OpenAI-compatible API endpoint, so you just need Ollama running with its HTTP server accessible from the AutoGen machine.

Step 1: Make sure Ollama is listening on all interfaces (not just localhost)

By default Ollama only listens on 127.0.0.1. To expose it over the network, set the environment variable before starting Ollama:

OLLAMA_HOST=0.0.0.0 ollama serve

Or on the machine running Ollama, edit the service config to include that env var permanently.

Step 2: Verify the remote endpoint works

From the AutoGen machine, test that you can reach it:

curl http://REMOTE_IP:11434/api/tags

You should get a JSON list of your models back. If this fails, check your firewall is open on port 11434.

Step 3: Configure AutoGen to use the remote Ollama

from autogen_ext.models.openai import OpenAIChatCompletionClient

model_client = OpenAIChatCompletionClient(
    model="deepseek-r1:latest",  # match your Ollama model name exactly
    base_url="http://REMOTE_IP:11434/v1",
    api_key="ollama",  # Ollama ignores this but the field is required
)

The key thing is the /v1 suffix on the base URL - Ollama exposes an OpenAI-compatible endpoint there.

Common issues:

• Model name must match exactly what ollama list shows on the remote machine
• If you see timeout errors, the firewall is blocking port 11434
• If you get a 404, double-check the /v1 is in the URL

What error are you seeing when you set base_url? That would help narrow it down.

[pandego]

If you are targeting Ollama from another machine, there are 4 checks that usually fix this in AutoGen:

1. Bind Ollama to the network interface

OLLAMA_HOST=0.0.0.0 ollama serve

2. Verify reachability from the AutoGen host

curl http://<OLLAMA_HOST>:11434/api/tags
curl http://<OLLAMA_HOST>:11434/v1/models

If either fails, it is network/firewall first.

3. Use the OpenAI-compatible endpoint in AutoGen (/v1 is required)

from autogen_ext.models.openai import OpenAIChatCompletionClient

model_client = OpenAIChatCompletionClient(
    model="deepseek-r1:latest",
    base_url="http://<OLLAMA_HOST>:11434/v1",
    api_key="ollama",
    model_info={
        "vision": False,
        "function_calling": False,
        "json_output": False,
        "family": "unknown",
    },
)

model_info helps when the model is not in AutoGen's known OpenAI model registry.

4. If it still fails, print the exact HTTP error/status from AutoGen logs

• 404 usually means missing /v1
• 401 is usually wrong auth handling by a proxy
• timeout/refused means host bind or firewall

If you share the exact traceback, I can pinpoint which of the 4 is failing.

[alexmercer-ai]

good addition from @pandego - the model_info block is what gets you past AutoGen's model registry validation when using a model it doesn't recognise. the family: "unknown" is key there.

one more specific tip if you're running deepseek-r1: make sure json_output and function_calling are both False in model_info, since it doesn't support those natively. if you leave them unset or true, you'll hit confusing errors that aren't obvious at first glance - took me a while to track that down the first time.