Serving: endpoints, AI gateway, tracing and evaluations
Serve only what was approved.
Model endpoints on your own GPUs or at EU providers, behind one gateway that routes each request to the release that passed its evaluation and its approvals. Every call is traced, priced and counted against the calling key's budget.
Endpoints on your hardware
An endpoint is a running model server. Managed endpoints run on your compute targets through the same agent that runs training jobs, so serving needs no second deployment tool and no inbound access to the cluster from EUVYRA's control plane.
| Engine | What runs | What you provide |
|---|---|---|
| vLLM | The vLLM OpenAI-compatible server | A Hugging Face repository such as Qwen/Qwen2.5-7B-Instruct, or a path in the image |
| llama.cpp | The llama.cpp server, for GGUF models on CPUs or small GPUs | A GGUF repository or file |
| Container | Any image that serves HTTP | The image, and optionally command, port and health path |
| Provider | Nothing: an external, OpenAI-compatible API | URL, model name and the provider's key; the host must be on your allowlist |
The server keeps the desired state and the agent reconciles it. On Kubernetes an endpoint becomes a Deployment, a Service and a Secret holding a token, so the model server refuses requests that did not come through the gateway. On a Docker host the endpoint gets whole GPUs that no job holds.
Autoscaling
Replicas can follow the gateway's traffic, measured as requests in flight or requests per minute per replica, between bounds you set. With a minimum of zero an idle endpoint scales to zero, and the next request wakes it: another endpoint of the release answers meanwhile, or the request waits a configurable time for the cold start. Every change is recorded with its reason. An endpoint can also run in a pool of cloud machines that is ordered on demand.
One gateway, routed by your release process.
Applications point an OpenAI client at EUVYRA and name an AI system as the model. The gateway looks at that system's deployed releases on every request, so the release workflow is the only control there is:
- the direct release takes the traffic, a canary takes its share, and a shadow release gets a copy whose answer is traced and discarded;
- promoting a canary makes it primary, rolling back removes it, and a new release supersedes the old one;
radiology-triage@stagingreaches the same system's staging release;endpoint/namereaches an unreleased endpoint for tests, only with a key scoped to its project, so applications cannot bypass approval;- a system classified as prohibited is never served.
from openai import OpenAI
client = OpenAI(
base_url="https://euvyra.example.eu/v1",
api_key="euv_...",
)
reply = client.chat.completions.create(
model="radiology-triage",
messages=[{"role": "user", "content": "Hallo"}],
)What every call gets
Redaction
Endpoints set to redact have emails, IBANs, card and phone numbers, IP addresses and similar data masked before the request leaves. The trace records how many items of each type.
A trace
Input, answer, tokens, cost, latency and time to first token, in the endpoint's project and under its privacy settings. Shadow copies are traced too.
A price
From the endpoint's own prices per million tokens or the organization's price list, with token counts from the model's usage report, streamed calls included.
Budgets and limits
Each API key has a monthly budget in euros and a request rate limit. Raising either is a spending decision for a person, not for an AI assistant.
Familiar errors
Errors use OpenAI's format and status codes, so clients raise their usual exceptions; answers while an endpoint wakes up carry
Retry-After.More than one client format
OpenAI-compatible chat, completions and embeddings, and the Anthropic Messages format for clients built on it.
Tracing, evaluations and prompts.
Trace agents and retrieval pipelines from the Python SDK, or send OpenTelemetry: EUVYRA reads the GenAI semantic conventions and OpenInference over OTLP/HTTP. Spans carry kinds such as llm, retrieval, tool, agent and guardrail, with tokens, cost and errors.
Evaluations run versioned test suites against the exact model or prompt version you plan to release. The server scores every output with built-in evaluators for correctness, groundedness, PII leakage, refusal, bias and robustness, or with an LLM judge on an approved EU endpoint, and a gate decides whether the release may go to approval. euvyra eval run fails a CI job when quality regresses.
People stay in the loop: blind pairwise review, annotation queues with a rubric, and online scoring of sampled production traffic. A playground runs one prompt against up to four models side by side through the same gateway, and any trace can become a test case. Prompts are versioned with labels, and the production label only moves when an approved release is deployed.
Questions about serving
Can we serve a model that is not approved yet?
For testing, yes: endpoint/name reaches an endpoint directly with a key scoped to its project, and those calls are traced with the tag unreleased. Production traffic under an AI system's name only reaches deployed releases.
Which EU model providers can we use?
Any provider with an OpenAI-compatible API. Its host has to be on your outbound allowlist; otherwise the gateway refuses the call with provider_blocked.
What does the gateway need from the network?
The EUVYRA server calls endpoints directly, so their address must be reachable from it, for example because EUVYRA runs in the same cluster. Machines in cloud pools use the account's private network for this.
Does the gateway store prompts and answers?
Each call becomes a trace, and the project's privacy settings apply: redaction, pseudonymized user ids, or metadata only without content. Retention is set per project.