© 2026 Medoya LLC
Powered by voilamia
Subscribe to receive new insights
Medoya Labs
One OpenAI-compatible endpoint in front of every AI model your organisation runs on its own hardware. Your tools and agents get one address and one key with a budget; your security lead gets one place where access, rate limits and spend are decided. Patient data stays on your network unless you route it out on purpose.
zzRouter is one address in front of every model your organisation runs itself. Behind it sit the engines that actually run the weights: Ollama, vLLM, llama.cpp, and MLX on Apple silicon. They live on whichever Linux, macOS or Windows machines you already own with the memory and the GPUs, and zzRouter installs on each as a system service. In front of it, a coding assistant, an agent framework or a plain SDK speaks the OpenAI API or the Ollama API and never learns which machine answered.
It exists because our clients want inference on premises for data that cannot leave the building, on a mix of hardware nobody planned, with agents talking to one address and one credential whose limits someone can set. An engine alone runs one model on one machine and knows nothing about who is asking or what it costs. A gateway such as LiteLLM answers that part, and assumes the engines are already running and that someone else keeps them up. zzRouter is also the thing that keeps them up. It installs the engine, downloads the weights, starts the model on the machine that has the GPU when the first request for it arrives, stops it once it has sat idle, and patches itself. There is no database behind it: keys, teams and spend are three files in the configuration directory, and a node is one static Go binary. The same key can reach a hosted provider when a task needs one, so the cloud is a routing choice rather than a rewrite.
Change the base URL in the OpenAI SDK and keep the same calls: chat, embeddings and the rest of the OpenAI routes, or the Ollama ones if that is what the tool speaks. A refusal never arrives as a silent drop: a rate limit is a 429 that says when to retry, a bad request a 400 with a stable code a program can branch on. Behind the one key sits one budget, and every call leaves one line in the ledger with the model, the tokens and the cost.
Static keys are for operators. Everyone else gets a virtual key: hashed with Argon2id, owned by exactly one team, suspendable, expiring, carrying its own limits and the team's. Every request is checked in one fixed order, suspended, expired, model access, requests per minute, tokens per minute, budget, concurrency, and a failure anywhere down the chain releases whatever the earlier steps reserved.
Budget is the interesting one because the cost of a request is not known until the answer has arrived, so the check reserves an estimate before the model runs and the real token count settles it afterwards. A reservation whose request vanished mid-flight is reaped after five minutes, long enough to outlast a cold model load, rather than left to be settled against the next request on the same key. A spend limit of zero means the limit is not enforced, and the code says so where it is checked, because "unlimited" and "nothing left" must never share a value.
The default is worth stating plainly. Out of the box the inference surface answers anonymously for local models, the way Ollama does, so nothing breaks on day one. One setting closes it, a team whose model access has been set closes it by itself, and a hosted model needs a key unless an operator has opted it open, because it bills someone's upstream account. Cloud providers appear as models like any other, so a key that has not been granted a cloud model cannot reach one, and that access list is what a security review asks to see. The real provider credentials live in the coordinator's configuration and are never copied to a worker.
The ledger deserves the same plainness. It is an in-memory ring of the last thousand calls holding the model, the node, the tokens, the latency, the cost, the key and the team. Prompt capture is on by default for the last hundred so an operator can see what a misbehaving agent actually sent; it sits behind the administrator key, nothing is written to disk, and one line of configuration turns it off, which is the line a hospital sets before go-live. Every call is also exported as standard metrics your monitoring already reads.
An agent driving the management side gets the same treatment. Every failure carries a stable code with its recovery written next to it in the catalog, the route list can be read at runtime, and the quickstart is written as request templates an agent can paste into a tool definition.
One machine, the coordinator, fronts the others, the workers, over mutual TLS on a port of its own, separate from the administration port. A worker asks to pair and prints a one-time code that dies in fifteen minutes; someone holding the administrator key accepts it at the coordinator, and the worker then holds a thirty-day certificate it renews itself, which only the coordinator can sign. A worker running an incompatible version is refused outright, with an error that says which versions would be accepted.
What a worker does not have is the point. It holds no administration key and serves no inference on its administration port at all; the only way to its models is the cluster port, and the only thing that authorises a request there is a certificate the coordinator issued. That makes the coordinator the single place where access is decided for anything routed through the cluster. Nothing on the worker enforces any of it, on purpose, because two enforcement points disagree the moment one of them is a version behind. The trade is that the certificate and the check on who issued it are the whole story. Loosen either and a certificate becomes unmetered access to every model on every worker.
A request goes to a machine that already has the model loaded, or launches it where the hardware fits. A model name can also stand for a group of deployments chosen by priority, by speed or by load, with a breaker that takes a failing one out of rotation.
One more thing a security lead should hear from us rather than find. The provider configuration names the binary an engine runs and its flags, so whoever can write that directory runs code as the node. The directory belongs to the service user and nothing else on the machine may write it; launch parameters arriving over the API are screened character by character, and the schema rejects any flag it does not know. Our threat model document says this in its first paragraph, and it is available on request.
Everything here runs in our own lab before it is recommended to anyone: a Mac as coordinator, a Debian container with an RTX PRO 6000 as the worker that does the heavy lifting, and a Windows laptop as the worker that goes to sleep when it feels like it. That last machine is the most useful of the three, because anything that only works when every node is awake is found out within a week. The record of what has and has not been proven is the coverage tracker, which lists every route the end-to-end suites have never driven, and it lists a lot of them. A tracker that only records the wins is not a tracker.
On a service install the node cannot replace its own executable, because a service account that can do that can replace the one an administrator runs under sudo. Instead it writes a request naming only an action and a version, no path and no address, and a root task does the rest: fetch the release, check the checksum and a signature tied to our build identity, run the new binary once to prove it executes here, install it beside the old one, and roll back on its own if the node does not come up healthy. Where the node does update itself, on a laptop say, three failed boots roll the release back without anyone present.
This is the problem a clinic or a hospital has when it decides that some questions must be answered by a model that never leaves the building. Your engineers want agents on those models. Your security review wants to know which key can reach which model, what the monthly ceiling is and who set it, how a worker in a closet is prevented from being an open door, and how it gets patched without handing a service account the keys to the host. zzRouter is where we work those answers out on our own hardware and at our own risk before we recommend them on yours, down to the append-only audit log that answers "who set it" with the key that did it and a time, which is a person's name as soon as each administrator holds their own key rather than the shared one. In a readiness review we map your models to keys and teams, set the first budgets, and write down the evidence your security review will ask for.
Running in our lab since September 2025 and used daily: about 1,800 commits, 600 test files, and an end-to-end coverage report generated from real traffic rather than from claims. Go, Apache 2.0 licence, pre-1.0: breaking changes are still allowed between minor versions and the changelog says so, the cluster wire contract carries its own version number so a mismatched worker is refused rather than half-served, and the OpenAI-compatible routes are the surface we change last. The source is not yet public; releases will be Sigstore-signed when it is.
Book a free 30-minute readiness review. Bring your team's current AI tool usage and one workflow you want agents in. Leave with the three blockers and the first step.