4 posts tagged with "local-llm"

Ten weeks, nine posts, and four working systems.

The practical question behind all of it: how do you build useful AI systems where cost, connectivity, data privacy, and human accountability are hard constraints?

The industry narrative is dominated by cloud-first, fully autonomous agents operating with unlimited budgets. JigsawFlux took a different direction: local inference, offline-first architectures, Kubernetes home labs, and strict state-machine controls.

The path is visible in retrospect: direct local LLM wrappers gave way to stateful agent workflows, which gave way to bounded edge systems for clinical consultation, travel risk assessment, and NHS appointment recovery.

This is what those nine posts covered, what connects them, and the five engineering rules that emerged from the work.

It is 11:40pm local time in Crimea. A field researcher — call her T-002 — is in her hotel room when she hears shots outside. She picks up her phone, opens the company travel app, and types what she can see. The app is offline. There is no satellite signal. The ops centre in London cannot be reached.

What happens in the next sixty seconds matters. Not for a sprint retrospective. For her.

This post is about building a system that makes those sixty seconds count — an offline-first agentic travel risk assessment application that runs entirely on a local LLM, asks one clarifying question if it needs to, and produces a structured risk verdict before the connection returns. No cloud dependency. No API key in the critical path. The database, the model, and the application all run on the same MicroK8s node — the one I set up in my previous post on my home lab Intel NUC.

But before the architecture: the context. This system exists because employers have a legal and moral duty to protect people they send into harm's way. That duty does not pause when the internet does.

The code is at JigsawFlux/airgapped-agentic-trm.

In Part 1 I ran Ollama directly on a Linux machine and wired it up through an MCP layer to a small web app. It worked. But bare-metal has friction — if the process crashes, it stays down. Adding Open-WebUI means managing another process. Resource limits are manual. There's no clean internal networking between services.

This post moves the whole thing into Kubernetes. The goal isn't enterprise-grade infrastructure — it's a home lab setup that's reliable, easy to extend, and honest about its limitations.

Manifests are in the ollama-mcp-starter repo under backend/k8s-deployment/.

AI inference doesn't have to mean a cloud API call. This post walks through a spike I built to run a locally hosted language model through a clean, layered architecture using Ollama and the Model Context Protocol (MCP).

The full source is on GitHub. Contributions and feedback welcome.