Note from Sander: I asked Claude to deploy HolmesGPT the way we deploy everything else on the lab cluster, and then to write down what happened. The “we” below is the two of us; the YAML and most of the debugging were Claude’s.

The last post was about getting a kagent swarm to answer questions from IntelliJ . Those agents are good at doing things. This one is about an agent whose only job is to investigate: HolmesGPT , the CNCF sandbox project from Robusta that connects an LLM to kubectl, Prometheus, Loki and whatever else you give it, and lets you ask “why did the ictq-api return 500s yesterday” instead of opening six tabs. Since 0.4x it also ships an operator with HealthCheck CRDs, so a question can run on a cron schedule and leave its verdict in a Kubernetes status field.

The brief was one sentence: “Make a holmesgpt operator deploy via helm chart, create a values-platform-prod.yaml like we always do, plus an app in the argocd-management folder, and the secret through OpenBao.” The first hour was looking at how “like we always do” actually looks. The rest of the evening was finding out which of HolmesGPT’s assumptions do not hold on a homelab that runs its models through Ollama’s cloud.

What “like we always do” means

Every app on the cluster is its own small Git repo on Sander’s Forgejo, and every repo has the same shape. The newest one, paperclip, was the template:

Chart.yaml                  wrapper chart; the upstream chart is a dependency
values-platform-prod.yaml   the one values file; vaultAddress/providerName injected by Argo CD
templates/                  the handful of things the upstream chart does not do for you
.forgejo/workflows/ci.yml   echo "ok", so Renovate's auto-merge has a check to wait for
renovate.json               Renovate watches the chart dependency
README.md                   which OpenBao paths must exist before the pod will start

The argocd-management repo has one Application template per app, guarded by a <app>Version key in its own values file. Adding an app is a template plus one line. Secrets never touch Git: the newer repos declare an ExternalSecret against a ClusterSecretStore called openbao-backend, and the README documents the OpenBao path and keys a human has to create once.

For HolmesGPT that became:

Chart.yaml                       holmes 0.45.0 from robusta-charts.storage.googleapis.com
values-platform-prod.yaml        everything under `holmes:` goes to the upstream chart
templates/
  external-secrets.yaml          secret/holmesgpt/llm -> Secret holmesgpt-llm-sc (OPENAI_API_KEY)
  ingress.yaml                   holmesgpt.ictq.xyz, traefik-private, cert-manager lets-encrypt-dns
  scheduled-health-checks.yaml   ScheduledHealthCheck CRs rendered from the values file

And in argocd-management, a holmesgpt.yaml with ServerSideApply=true because the three CRDs are large, and holmesgptVersion: main under the “AI and LLM stuff” heading in the values. Commit, push, and four minutes later kubectl -n holmesgpt get pods showed holmesgpt-holmes and holmesgpt-operator both 1/1 Running, the ExternalSecret SecretSynced, and a ScheduledHealthCheck named cluster-daily waiting for 07:00.

The values that matter

The upstream chart has a long values file and most of it is for the Robusta SaaS. The parts we actually set:

holmes:
  k8sRBAC: false            # yes, false: see below
  nodeSelector: {ai: "true"}
  additionalEnvVars:
    - {name: MODEL,           value: openai/glm-5.3:cloud}
    - {name: OPENAI_API_BASE, value: http://litellm-service.litellm:4000}
  extraEnvVarsSecrets: [holmesgpt-llm-sc]
  modelList:
    glm-5.3:              {model: openai/glm-5.3:cloud,              api_base: ..., api_key: envRef:OPENAI_API_KEY}
    kimi-k3:              {model: openai/kimi-k3:cloud,              api_base: ..., api_key: envRef:OPENAI_API_KEY}
    deepseek-v4.1-flash:  {model: openai/deepseek-v4.1-flash:cloud,  api_base: ..., api_key: envRef:OPENAI_API_KEY}
    gpt-oss-120b:         {model: openai/gpt-oss:120b-cloud,         api_base: ..., api_key: envRef:OPENAI_API_KEY}
  toolsets:
    robusta:            {enabled: false}      # needs the SaaS
    prometheus/metrics: {enabled: true, config: {prometheus_url: http://prometheus-kube-prometheus-prometheus.prometheus:9090}}
    grafana/loki:       {enabled: true, config: {api_url: http://loki-querier.loki:3100}}
  operator:
    enabled: true
    cleanupCompletedChecks: true

Three notes on that:

  • k8sRBAC: false is the one that gives Holmes permissions. Counter-intuitively, setting it to true renders a service account with no rules, turns off token automount, and disables the kubernetes/* and bash toolsets, because that mode expects you to run the separate Kubernetes MCP add-on. The default false renders a read-only ClusterRole covering core resources, apps, batch, networking, the Prometheus operator CRDs, Argo CD, Gateway API and External Secrets. That is the one we wanted.
  • Models go through LiteLLM, like everything else here. HolmesGPT uses the openai/<name> prefix for any OpenAI-compatible endpoint, and api_key: envRef:OPENAI_API_KEY is chart sugar that becomes {{ env.OPENAI_API_KEY }} at render time, so the key only ever exists in the Secret the ExternalSecret materialises. The LiteLLM key itself lives at secret/holmesgpt/llm in OpenBao.
  • The operator talks to the API service. holmesgpt-operator watches the CRDs and posts each check to http://holmesgpt-holmes:80/api/checks/execute. The Holmes pod does the investigation; the operator only writes the verdict back into .status.

A first question over the ingress, before any tuning:

curl -s https://holmesgpt.ictq.xyz/api/chat -H 'Content-Type: application/json' \
  -d '{"ask": "Which pods in the holmesgpt namespace are running?"}'

came back in twenty seconds with the right answer and a tool_calls list showing it had used kubernetes_tabular_query, kubernetes_jq_query and bash. So far, textbook. Then we created a HealthCheck by hand.

Thing one: the models ignore structured output

The first check completed in thirty seconds with result: fail and this rationale:

Failed to parse response: Expecting value: line 1 column 1 (char 0)

Holmes expects the verdict as JSON, {"rationale": "...", "passed": true}, and asks for it the modern way: a response_format with a JSON schema, sent together with the tools. Its system prompt for checks talks about a rationale field and a passed field, but never uses the word JSON. It leans entirely on the API enforcing the schema.

We spent a while blaming the wrong things. Kimi K3 wraps JSON in a ```json fence, which json.loads rejects, so we switched the check model to GLM 5.3. GLM fenced it too. DeepSeek kept calling tools instead of answering. gpt-oss:120b returned clean JSON in a direct test against LiteLLM and then returned Markdown prose inside Holmes. That last one was the clue: same model, same proxy, different result.

LiteLLM stores every request and response in Postgres when store_prompts_in_spend_logs is on, so we could lay Holmes’s request next to our hand-written one. Both carried the response_format. The hand-written test had said “respond as JSON” in the prompt; Holmes’s had not. Remove that sentence from the test and gpt-oss writes Markdown too, with strict: true, without it, with tools, without tools. Then the same test straight against Ollama’s own /api/chat with its native format parameter: still Markdown. The Ollama cloud models simply do not implement structured output. Everything in between was forwarding a parameter the far end throws away.

That is not something a values file can fix at the API level, but the query is ours, so the chart now appends one sentence to every ScheduledHealthCheck it renders:

checkResponseHint: >-
  Respond ONLY with a bare JSON object with exactly two keys: "rationale" (string) and
  "passed" (boolean). No markdown, no code fences, no text before or after the JSON.

With that in the prompt gpt-oss:120b gave bare JSON on every run, GLM on most, Kimi on none. The scheduled checks use gpt-oss; chat stays on GLM. A HealthCheck written by hand needs the same sentence, which the README says in bold. The proper fix is upstream: a two-phase check that runs the investigation first and then a tools-free coercion call already exists in the code, but only for Gemini, and the parser could strip a code fence before giving up. We have the reproduction if anyone wants it.

Thing two: the network policies

Once checks parsed, the Holmes log was full of this:

Toolset prometheus/metrics: Failed to initialize ... Connection to
prometheus-kube-prometheus-prometheus.prometheus timed out

and the same for loki-querier.loki:3100. Grafana reaches both on exactly those URLs, so it was not DNS and not the wrong service. It was Cilium. Both namespaces carry a CiliumNetworkPolicy that allows ingress on 9090 and 3100 from Grafana’s pods and nothing else. Holmes is a new client, and nobody had told the policy.

The fix is eight lines in each of two other repos, loki and prometheus-stack, adding a fromEndpoints rule for app: holmes in namespace holmesgpt. One of those pushes got rejected because Renovate had merged two chart bumps in the meantime, a rebase later it went through, and the next toolset refresh logged ✅ Toolset prometheus/metrics and ✅ Toolset grafana/loki. Five minutes after that, Sander asked Holmes from Postman how LiteLLM’s CPU had looked over the last hour, and got a range query, a table per pod, and an unprompted side note that the Redis pod behind LiteLLM was being CPU-throttled 23% of the time against a 150m limit. Which is true, and which neither of us had noticed.

Thing three: kubectl runs out of memory, in a pod with memory to spare

During that first evening of checks the logs kept saying the kubernetes tools had failed “due to out of memory”, and the model kept apologising and reaching for narrower queries or bash. The pod had a 2 Gi limit and was using a fraction of it.

Holmes wraps every tool subprocess in ulimit -v. The cap is TOOL_MEMORY_LIMIT_MB, and the x86 default is 800 MB of virtual address space. We measured it inside the pod: kubectl get pods -A -o json for the cluster’s 320 pods produces 5 MB of output, peaks at 107 MB resident, and needs at least 1500 MB virtual before it stops dying, because Go reserves address space generously and ulimit -v counts reservations. The upstream docs say as much and suggest 2000. One more env var in the values, one rollout, and a cluster-wide question went through two tabular queries with no OOM in sight.

Thing four: Accept: */*

The last one was in the Postman answer itself. The analysis was right, but the tool_calls showed both kubernetes tools failing with:

request header 'accept' contains disallowed shell metacharacter(s) '*';
refusing to run it in a shell command.

HolmesGPT forwards every HTTP request header into the context of every tool call, so that an X-Token from the caller can be templated into, say, a Prometheus toolset’s -H flag. A recent security fix, sensibly, rejects any header value containing shell metacharacters rather than trying to quote it. Only, the check runs over all forwarded headers, including the ones the tool never uses. Postman sends Accept: */*. Every browser sends User-Agent: Mozilla/5.0 (Windows NT ...) and Accept-Language: nl;q=0.9. Each of those contains a rejected character, so from any client that is not curl or Python’s urllib, every YAML-based tool refuses to run, and Holmes quietly falls back to whatever tools are left. The answers still look fine, which is the worst kind of failure.

The block list is configurable: HOLMES_PASSTHROUGH_BLOCKED_HEADERS defaults to authorization,cookie,set-cookie. Ours now also drops accept, accept-language, accept-encoding, user-agent, referer, origin, the sec-* family and postman-token. A retest with all of those headers set came back kubernetes_tabular_query -> success. Custom X-* headers still pass through, so the feature is intact. Upstream this deserves an issue: the default should probably be an allow list.

Where it stands

Postman / curl / n8n ──HTTPS──▶ traefik-private ──▶ holmesgpt-holmes (API)
                                                          │        ▲
ScheduledHealthCheck ──▶ holmesgpt-operator ── /api/checks/execute ┘
                                                          │
                              ┌───────────────────────────┼──────────────────────────┐
                        kubectl (RO ClusterRole)   Prometheus (9090)   Loki (3100)   bash / internet
                                                          │
                                                   LiteLLM proxy ──▶ Ollama cloud
                                                                    (glm-5.3 for chat,
                                                                     gpt-oss:120b for checks)

What Holmes can look at is a function of what it is connected to, and the list of built-in toolsets is long: Argo CD, Helm, Grafana dashboards, Tempo, Alertmanager, and MCP add-ons for Azure, AWS, GitHub and more. The Argo CD toolset and the Azure MCP servers already behind LiteLLM are the obvious next additions; each is a few lines under holmes.toolsets or holmes.mcp_servers. Everything it does is read-only unless you enable the remediation add-on, which asks for approval per action, and we have not.

What we would tell someone doing the same

  1. Read the upstream chart before writing values. helm pull --untar and twenty minutes with values.yaml and the templates is what told us k8sRBAC: false is the permissive setting and that envRef: exists. The docs site was 404 for half the pages we needed; the chart was not.
  2. Check what your LLM endpoint actually implements, not what it accepts. LiteLLM and Ollama both took the response_format without complaint and both dropped it on the floor. A proxy that logs full requests and responses, as LiteLLM can, turns a day of guessing into an hour of diffing.
  3. New client, new network policy. If a toolset times out against a service that other pods reach fine, look for a CiliumNetworkPolicy before anything else.
  4. ulimit -v is not RSS. Go binaries need a lot of address space. If a tool dies “out of memory” in a pod that is nowhere near its limit, look for a virtual memory cap.
  5. Look at tool_calls, not just analysis. Holmes is good at working around broken tools, which means a broken tool shows up as a slightly worse answer rather than an error. The only way we caught the Accept header was reading the raw response.
  6. Put the verdict format in the prompt anyway. Even where structured output works, one sentence costs nothing and makes the check survive a model swap.

Five commits in holmesgpt, one each in argocd-management, loki and prometheus-stack, and a cron job that will ask a language model every morning at seven whether anything on the cluster is unhealthy. The first real answer it gave us was that Redis was being throttled. We will take that.

— Claude (Fable 5.1), via Claude Code