Air-Gapped AI SRE Agent: Run AURA on a Local LLM with llama.cpp

The model, the serving layer, and the agent asking questions all sit on local hardware. The first thing it debugs is the AI stack it is running on.

Learn more about AURA → https://www.mezmo.com/aura
Get started today → https://github.com/mezmo/aura
Get deployment help → https://www.mezmo.com/contact

No external inference provider in the path. Start with a model close to the frontier and a little above 30B: Qwen 3.6 or 3.8 27B, Qwen 3.6 or 3.8 35B A3B, Gemma 4 31B or 26B A4B, or GLM 4.7 30B A3B.

For serving it, llama.cpp or LM Studio over Ollama. Ollama is compatible, but it does not give you the control you want starting out. Either of the first two works through AURA's OpenAI provider, which already carries the reasoning settings the model needs.

Point the provider at your host and port, then reference the model by alias rather than inlining every argument. Arguments that do have to go on the request belong in agent.llm.additional_params. On the llama.cpp side the settings live in an INI file, and the ones shown here are sized for 24 gigs of VRAM.

The config in this video is built for research plus probing a large Compose stack for debugging and general health. The run at the end points the AURA CLI at a local homelab server.

AURA reads and reports here. It returns a diagnosis and does not act on the environment. This video also does not cover installing or running llama.cpp itself.

For more detail, visit
https://www.mezmo.com/videos/air-gapped-ai-sre-agent-run-aura-on-your-own-local-llm

Chapters

0:00 Air-gapped AURA against a local model

0:11 Picking a model: 30B and up

0:46 The AURA config

0:59 llama.cpp and LM Studio over Ollama

1:19 Running through the OpenAI provider

1:35 Host, port, and configuring the model by alias

2:03 The llama.cpp config and 24 gigs of VRAM

2:36 AURA diagnosing its own AI stack

#AISREAgent #LocalLLM #AirGapped