euvyra
DeutschRun it yourself

Training: experiments, compute and cloud machines

Track every run. Train on GPUs you control.

Experiment tracking with the depth of a dedicated tracker, and a scheduler that runs the same code on your Kubernetes cluster, a single GPU workstation or machines rented by the hour. Every job ends up as a run with its cost and its full log.

Run it yourself
The runs page of a chest X-ray triage project: training loss and validation AUC charted across the ten newest runs.

Experiment tracking

euvyra.init() records the configuration, git commit and diff hash, package versions and system load. Everything else you log on top.

  • Runs and metrics

    Live metrics per step, GPU and CPU load, tags, notes, groups and job types. Runs that crash are marked as crashed; runs resumed from a checkpoint continue counting steps.

  • Media

    Images, audio, video, sandboxed HTML, histograms, plots, tables and confusion matrices, plus 3D meshes, point clouds with boxes, and molecules.

  • Charts across runs

    Compare any runs side by side and keep chart layouts as workspaces, shared with the project or private to you. Custom Vega-Lite charts cannot fetch anything from a URL.

  • Reports

    Documents of markdown, charts, run tables, media, sweeps and lineage graphs that show current numbers when opened, edited by several people at once.

  • Sweeps

    Grid, random and Bayesian search (Tree-structured Parzen Estimator) with Hyperband-style early stopping. Parameter files use the W&B format, so existing sweeps carry over.

  • Artifacts and lineage

    Versioned artifacts with aliases such as best or production, and a lineage graph from dataset version to run to model version to release.

  • Data that stays in your buckets

    Reference files in S3, Google Cloud Storage, Azure, HTTPS or a file system. EUVYRA stores the manifest with checksums and never reads the files.

  • Model graphs

    The structure of PyTorch, Keras 3 and ONNX models, or of a safetensors checkpoint read from its header alone, drawn with the data flow between layers.

  • Offline and integrations

    Log on a machine without network access and sync later. Callbacks for Hugging Face, Lightning, Keras, XGBoost, LightGBM and scikit-learn.

Your GPUs pull the work. Nothing pushes into your network.

An agent runs next to the hardware: in a dedicated Kubernetes namespace with namespaced rights only, or as a systemd service on a single Docker host such as a DGX Spark. It sends a heartbeat with its free capacity, claims queued jobs that fit, starts them and reports state and logs. The cluster needs no inbound access and no VPN.

  • Queues and quotas. Priorities per job, GPU hours per month and parallel jobs per project, and which targets a project may use.
  • Cost per run. Duration times the target's hourly and per-GPU price, on the job and in the run summary.
  • Multi-node training. A job with nodes: 4 becomes four pods with NNODES, NODE_RANK and MASTER_ADDR set for torchrun.
  • Sweeps on compute. Queue a sweep as jobs; each takes its configuration when it starts, so a Bayesian search keeps learning.
  • A page per cluster. Nodes, GPUs with utilisation, memory, temperature and power, seven days of history, maintenance mode and excluded nodes.
A GPU machine's page: one NVIDIA GB10 at 61 percent utilisation, used by a model endpoint, with CPU and memory in use.

What happens when you launch a job

The training code is the same as on a laptop. The agent hands the container everything euvyra.init() needs to attach to the job's run.

  1. Queue it

    From the CLI, the Python SDK, the console or an AI assistant.

    euvyra launch --project medical-ai \
  --image registry.example.eu/ml/train:1.4 \
  --gpus 1 --wait -- python train.py

  2. An agent claims it

    The first agent with enough free capacity, or the target you named. A job asking for a GPU type only goes to targets that have it.

  3. EUVYRA prepares the run

    It creates the run with the job's configuration and an API key limited to that project, which expires with the job's timeout and is revoked when the job ends.

  4. The container trains

    On Kubernetes as a Job whose key lives in a Secret, with requests equal to limits, no privilege escalation and no service account token. On Docker with whole GPUs per job.

  5. The record is complete

    Status, exit code, GPU hours, cost and the complete container log are attached to the run, and the change is in the audit log.

Cloud machines inside a budget

When your own GPUs are busy, EUVYRA can order machines itself. An ordered machine installs Docker, the NVIDIA driver and the agent on first boot, then behaves like any other target. A pool orders machines while jobs or endpoints sent to it wait and deletes them once they have been idle, so a GPU costs money only while there is work for it.

Providers and the regions offered by default
ProviderDefault regionsGPUs
Hetzner CloudFalkenstein, Nuremberg, HelsinkiCPU only through the cloud API; Hetzner's dedicated GPU servers join with the install script
ScalewayParis, Amsterdam, WarsawL4, L40S, H100, RTX 6000 and others
OVHcloudGravelines, Strasbourg, Frankfurt, WarsawL4, L40S, A100, H100, V100
AWSFrankfurt, Ireland, Paris, Stockholm, Milan, Spaing4dn, g5, g6, g6e, p4d, p5 and newer
Google CloudNetherlands, Belgium, Frankfurt, Paris, Finland, WarsawL4, A100, H100

Limits a person sets

  • A monthly budget per cloud account. Before every order EUVYRA adds up this month's spend and refuses an order that would exceed it; once the budget is used up, machines drain and are deleted.
  • A maximum number of machines per account and per pool, an idle time after which a machine goes, and an age limit.
  • Regions outside the EU can be added to an account, but are never offered by default.
  • Ordering machines, connecting accounts and resizing pools need an organization admin, and none of it is allowed to AI assistants.

Each machine joins with a bootstrap token that works once and expires after two hours, so what a provider's metadata service shows of the setup data is useless afterwards. Provider credentials are stored encrypted, never returned by the API and never sent to a machine.

Questions about training

Does our cluster need to be reachable from EUVYRA?

No. The agent connects out to EUVYRA and pulls work. Only model endpoints that the gateway calls must be reachable from the EUVYRA server.

Do we have to change our training code?

Not to run it as a job: containers that do not use the SDK still get a run with status, duration, exit code, cost and log. Adding euvyra.init() gives you metrics, media and lineage on top.

Can we keep our W&B sweep files?

Yes. Sweep parameters use the W&B format, including log_uniform_values and Hyperband early termination settings, and euvyra sweep create sweep.yaml reads them.

Which hardware has it run on?

The Kubernetes path ran end to end on a k3s cluster, and the Docker path ran GPU jobs on an NVIDIA DGX Spark (GB10) while vLLM served from the same machine. Agent images are built for amd64 and arm64.