Hermes Agent with a local model: connecting Ollama

You can connect a local model to Hermes Agent in the desktop app through Settings → Providers → Local Models in four clicks, or manually through Ollama. Either way, no key is needed, you do not pay for each request, and the conversation does not leave your machine.

A local model costs nothing to run, and Hermes can work with one. The main thing is to choose a model that can work with files and commands, and give it a context of at least 64,000; otherwise, the agent turns into an ordinary chat.

Let’s look at where a local model is useful today, how to connect one, and how to check that the agent is working with files. Let’s go.

Ailoot reference documentation: ailoot.ai/docs/.

Should you put an agent on a local model?#

Where a local model is already useful:

• Transcribing speech. There are free, open models for this: they’re small and recognize audio well. Your voice doesn’t leak to the cloud, so all your crafty plans stay with you. There’s a walkthrough on how to set this up in a separate article.
• Routine tasks. You can already use dumb little models for routine tasks: they can handle them.
• Data you’re afraid to send outside. If you’re really concerned, you can run a model on your own machine. But the quality will be poor.

Next year, local models will probably take up even less space. There are already powerful compression methods, and you can use RAM instead of video memory. I think in a year or so, everyone will have a little box with a dumb little model that performs tasks at the level of Sonnet 5. We’ll see.

The easiest way: Local Models in desktop Hermes#

If you have the Hermes desktop app, you don’t need to install anything manually. Hermes will install the local llama.cpp server, download a model, and choose a context size for you:

1. Open Settings → Providers → Local Models or choose Run models locally on first launch.
2. Click Install runtime: Hermes will download a llama.cpp build for your hardware; it’s a few hundred megabytes.
3. Choose a model from the catalog and click Download.
4. Click Use: new chats will go through the local model.

Each model in the catalog has a colored label: green means it fits entirely in video memory, yellow means it runs through RAM and is slower, and red means it’s too large for your computer. The app sets a context of 64,000 or more for recommended models. A graphics card with 8+ GB can run the catalog’s small models; one with 16+ GB can run 27-35B models. If you don’t use a model for 15 minutes, the app unloads it from memory; it loads it again when you send another message.

The section is included in the app’s test (canary) builds; in other builds, open it with the --local launch flag. If you don’t see Local Models in settings, that’s most likely why. From the terminal, the same path is: hermes model → Local models.

After choosing a model, check it with a task involving a file, as in the section below.

If you build your own projects with AI agents, check the result on a small, real task.

Using Ollama: what hardware you need#

You need Ollama if Hermes is installed on a server or you don’t have the Local Models section. Hardware requirements from the Hermes guide:

8 GB is enough only for a small model that responds with text only. To have the model work as an agent, you need 24 GB or more; see step 2 for details. Everything works without a graphics card, but slowly: a 9B model on an 8-core processor generates around 10 tokens per second; a 31B model, 2-5, and each reply takes 30-120 seconds.

Step 1. Install Ollama#

Ollama runs models right on your computer. On Mac and Linux, install it with one command in the terminal:

curl -fsSL https://ollama.com/install.sh | sh

On Windows, the PowerShell command is:

irm https://ollama.com/install.ps1 | iex

Check that the program is installed and the server responds:

ollama --version
curl http://localhost:11434/api/tags

If you don’t have any models yet, the second command will return an empty list. Remember port 11434: Hermes uses it to reach the model.

Step 2. Download a model that can take action#

What does a model do? It generates text. The wrapper around it, Hermes, handles working with files, terminal commands, and search. For the wrapper to work, the model must be able to call tools. Without that, the agent only replies and does nothing.

Only gemma4:31b out of these four models works as an agent. If you have less than 24 GB of memory, you won’t be able to run an agent: the models in the table will only respond with text. New models are constantly appearing in the Ollama library; for each one, first check whether it can call tools.

ollama pull gemma4:31b

You can download several models and switch between them in Hermes with the /model command.

Step 3. Increase the context to 64,000#

By default, Ollama gives the model a context of 2,048 tokens, while Hermes needs at least 64,000 to work with tools. With the default context, the task won’t fit, and the agent will fail with errors.

Create a model variant with a larger context. Run this in the terminal:

cat > Modelfile << 'EOF'
FROM gemma4:31b
PARAMETER num_ctx 64000
EOF

ollama create gemma4-64k -f Modelfile

You now have the gemma4-64k model. Use it everywhere from now on, not the original.

Step 4. Connect it to Hermes#

Start the setup wizard:

hermes setup

When asked about the provider, choose Custom Endpoint and enter:

• address: http://localhost:11434/v1;
• key: leave it blank or enter no-key;
• model: gemma4-64k.

You can enter the same configuration by hand in ~/.hermes/config.yaml:

model:
  default: "gemma4-64k"
  provider: "custom"
  base_url: "http://localhost:11434/v1"

How can I check that Hermes Agent’s local model can use tools?#

Run hermes and give it a task whose result is visible on disk. For example: “List all the Python files in this folder and count the lines in each,” then “Create a file named test.txt.” Check with the ls command or in the folder to see whether the file appeared. A model without tools can calmly produce a plausible-looking list without opening a single file.

The cheaper the neural network you use, the more advanced a leather editor it needs. That’s you. The smarter your neural network, the less you have to use your own head, but you always have to work.

Why the first reply takes several minutes#

With every request, Hermes sends a fixed part: the system instructions and descriptions of all enabled tools. Without a graphics card, the model takes a long time to process this first, then writes at its usual speed. That’s normal.

What helps:

• hermes prompt-size shows how large the fixed part is;
• hermes tools disables unneeded tool sets, and hermes skills helps remove extra skills. Every connected MCP (external service) makes each request larger: if you don’t use it, turn it off;
• ollama ps shows how many model layers have been offloaded to the graphics card. Even partial offloading speeds things up: a 31B model on a graphics card with 12 GB puts about 40 layers on it; the processor handles the rest;
• on a slow machine, set HERMES_API_TIMEOUT=1800 in ~/.hermes/.env so Hermes doesn’t cut off the request on timeout.

If the agent runs on a server around the clock or responds in Telegram, keep in mind: Ollama unloads the model after 5 minutes of inactivity, and the next reply will start with loading it again. The Ollama setting OLLAMA_KEEP_ALIVE=24h keeps the model in memory. There’s a walkthrough on moving the agent to a server in the article about running Hermes Agent on a server around the clock, and on setting up a bot in the Hermes Telegram setup guide.

If something goes wrong#

A cloud model as a fallback#

Hermes documentation suggests keeping a local model as the primary model and connecting a cloud model as a fallback. To do this, add a block like this alongside the model section in ~/.hermes/config.yaml, with your cloud provider and model:

fallback_providers:
  - provider: <cloud-provider>
    model: <cloud-model>

If the local model fails three times, Hermes sends the task to the cloud model. Those requests are paid and go to the cloud.

Another option is to have a powerful subscription model do the thinking and decision-making while the local model handles routine tasks. The goal is to preserve quality and gradually reduce token costs. This approach uses cheaper models for individual steps, with local models as a possible option later on. The article “How much does an AI agent for business cost?” describes what agents on subscriptions cost. If you haven’t installed Hermes yet, start with the walkthrough on how to install Hermes Agent.

Have you tried setting up a local model yet? What do you run on it? Tell us; we’re curious.

Continue exploring Ailoot at ailoot.ai.