Run an image workload

The Python realhw CLI can execute an existing OCI or Docker image on the fleet without a GitLab CI pipeline. The image is pulled by Podman on the selected runner; RealHW does not build a Dockerfile or accept a build context in this workflow.

The web app currently manages GitLab project runners and interactive hardware leases. It does not yet submit this deployment file as a remote workload.

Remote workload API (preview)

The workload API supports signed input uploads, hardware admission, status, cancellation, logs and artifact downloads. Confirm workload.submit in realhw capabilities --json before use. Workers must report workload:io:v1 in addition to the hardware you request. The local command below remains a separate interface.

OAuth approval issues the project-scoped token directly:

realhw auth login --project-id PROJECT_UUID \
  --scope runners:read --scope workloads:read --scope workloads:write
realhw hardware list --json
realhw workloads upload bundle.tar \
  --account-id ACCOUNT_UUID --project-id PROJECT_UUID --json

Save a request as run.json, replacing the identifiers and image digest:

{
  "account_id": "ACCOUNT_UUID",
  "project_id": "PROJECT_UUID",
  "idempotency_key": "candidate-001-i4",
  "image": "registry.gitlab.com/PROJECT/qemu@sha256:IMAGE_DIGEST",
  "required_labels": ["arch:x86_64", "kvm", "saas-linux-medium-amd64"],
  "preemptible": true,
  "cpu_count": 4,
  "memory_bytes": 8589934592,
  "disk_bytes": 8589934592,
  "timeout_seconds": 900,
  "inputs": [{"id": "UPLOADED_INPUT_UUID", "name": "bundle.tar"}]
}

preemptible: true is what makes the nested-KVM spot shape eligible. gce-kvm-spot (n2-standard-4 / n2-standard-8) guarantees arch:x86_64, kvm, and saas-linux-medium-amd64. Leave the field off for an on-demand unit job (saas-linux-medium-amd64 on gce-generic-od, or saas-linux-medium-arm64 with pool gce-arm-od).

realhw workloads submit --file run.json --json
realhw workloads wait RUN_UUID --json
realhw workloads logs RUN_UUID --after -1 --json
realhw workloads artifacts RUN_UUID --output artifacts.zip --json

Each input is mounted read-only at /in/NAME. Write evidence under /out. The runner collects that directory after execution, including failed tests. A host crash or transfer failure may prevent collection; missing evidence must never count as a passing test. ZIP files are downloaded without extraction.

For a private image, set REALHW_REGISTRY_USER and REALHW_REGISTRY_PASSWORD to a GitLab credential with read_registry. Keep it out of the request file and source control. Image references must pin a SHA-256 digest. The image runs its default entrypoint and command unless command is supplied. Custom environment keys must start with WORKLOAD_.

A successful CLI invocation means the API request succeeded. Inspect the returned outcome: success, failed or cancelled. For a failure, distinguish gitlab_reason: script_failure from runner_system_failure; the field name reflects the shared job lifecycle, even for runs submitted without GitLab CI. Log pages have chunks, a decimal-string next cursor and truncated; follow next until the page is empty. No output envelope is added by --json.

Run a pushed image on a GPU

GPU workloads use the same image submission API. Request the device with hardware labels and put the CUDA user-space libraries your application needs in the workload image:

{
  "account_id": "ACCOUNT_UUID",
  "project_id": "PROJECT_UUID",
  "idempotency_key": "gpu-check-001",
  "image": "registry.example.com/acme/gpu-check@sha256:IMAGE_DIGEST",
  "required_labels": ["gpu:nvidia", "gpu:count:1"],
  "cpu_count": 4,
  "memory_bytes": 17179869184,
  "timeout_seconds": 900
}

Use gpu:model:<name> when a specific accelerator is required, for example gpu:model:nvidia-tesla-a100. The host VM and RealHW runner provide the NVIDIA kernel driver, /dev/nvidia*, and NVIDIA Container Toolkit; the pushed image provides the application and CUDA user-space libraries. RealHW does not build the image. Check realhw hardware list --json first, because GPU capacity is operator-enabled and may be busy or unavailable in a particular zone.

The image must be digest-pinned. A private registry requires REALHW_REGISTRY_USER and REALHW_REGISTRY_PASSWORD in the runner’s secret environment; do not put registry credentials in run.json.

Select an on-demand ARM pool

To require on-demand ARM capacity, add "pool": "gce-arm-od" and use "required_labels": ["arch:aarch64", "saas-linux-medium-arm64"] with an ARM64 image. For Cargo tests, use the cargo-unit image; the x86 QEMU/KVM image does not run on T2A. Inspect realhw hardware list --json for current pool names, shapes, reported hardware and provisionable capacity.

pool is an exact placement constraint for existing hosts and new machines. RealHW keeps the labels, resource limits and quota checks, and does not fall back to another pool. Omit it (or use null) for automatic pool selection. A syntactically valid pool with no matching capacity waits until capacity is available or the workload times out. Pool selection is part of the idempotent request: use a new idempotency key when changing it.

T2A provisioning tries the configured Iowa zones in order, skipping us-central1-c; the production order is us-central1-a, us-central1-b, then us-central1-f. On-demand avoids Spot preemption but does not guarantee stock or successful runner enrollment. Check the workload’s provisioning attempts.

Understand a provisioning wait

realhw workloads status RUN_UUID --json
realhw workloads status RUN_UUID --json | jq '.provisioning'

Status includes provisioning: {schema_version: 1, attempts: [], truncated: false}. It requires workloads:read and the same account/project authorization as the workload itself. Only attempts created for that workload and owned by its account and project are included. This is not a fleet-wide machine listing.

Each attempt has id, provider, nullable shape and spot, status, created_at, updated_at, nullable completed_at, and nullable reason_code and message. spot: false means on-demand; null means unspecified. The latest 20 attempts are returned newest first, ordered by creation time and then ID; truncated: true means older attempts were omitted. An empty array means no recorded provisioning attempts for this workload, not that capacity is impossible; a workload can run on an existing host without provisioning a new one.

Attempt states include requested, creating_vm, waiting_enrollment, active, succeeded, failed, suppressed, cancelled, reaping, and reaped. An active provisioning attempt does not prove the workload process has started; check the workload’s started_at and status. A reaped attempt has been cleaned up; its original failure reason can still be present. Failed attempts do not necessarily mean the overall workload has failed: another attempt may follow.

Public failure codes are provider_capacity_unavailable, runner_enrollment_timeout, and provisioning_failed. Messages are sanitized; raw provider errors, credentials, cloud instance identifiers, and other workloads’ machine details are never included in this section. Unknown failures use the generic code; contact support with the workload and attempt IDs.

Provisioning history is included in single-workload status and the final wait response, not the workload list. Older servers may omit provisioning; treat that as unavailable diagnostics. A local wait timeout does not cancel the workload.

GCE ARM on-demand pools can provide a fallback to spot capacity. Switching to on-demand does not by itself resolve a runner enrollment failure; that means the machine did not register before its deadline.

Reuse the same request and idempotency key after an ambiguous submission. A changed request with that key returns HTTP 409. A local wait timeout does not cancel the run: use realhw workloads cancel RUN_UUID, then poll status. Queue time consumes the workload deadline. Disk is an admission requirement, not a filesystem quota. CPU and memory use the existing reservation machinery.

Limits: eight inputs of at most 512 MiB each; output contents at most 512 MiB and 10,000 regular files; logs at most 8 MiB; runtime up to eight hours. Storage objects expire after 14 days. Artifact download URLs expire after one hour. Compose and cloud-init remain unsupported by this remote API.

QEMU image inputs and results

The repository’s images/qemu runtime expects /in/bundle.tar. The tar contains regular files and a manifest.json describing the command to run inside the container:

{
  "command": ["bash", "scripts/frozen-test.sh"],
  "env": {"SERIAL_LOG": "/out/serial.log"},
  "timeout_seconds": 210,
  "provenance": {"revision": "PRODUCT_COMMIT"}
}

Supply the exact kernel, firmware shim and trusted scripts as bundle inputs. The image checks the KVM device, unpacks under /workload, and records console.log and result.json; the test script writes serial.log. The result contains the bundle’s SHA-256, command exit code and execution kind (process_exit, timeout or infrastructure_failure). The container exits 1 for a failed command; inspect result.json for the original command exit code.

The harness remains responsible for the test verdict. A successful process exit or successful artifact download does not prove a kernel test passed.

Local client command

Use a normalized deployment file with the runner CLI:

cargo run -p hw-runner -- run-deployment \
  examples/client-smoke.yaml \
  --output-dir /tmp/realhw-client-smoke

The current client path accepts one service and one replica. It reserves the requested CPU and memory, starts the image with Podman, forwards its logs, and returns the container exit code.

Collect artifacts

The output directory is mounted into the container at /out. Declare required files or globs in the service definition:

name: qemu-test
services:
  qemu:
    image: ghcr.io/acme/qemu-runner@sha256:...
    command: ["/usr/local/bin/run-tests"]
    resources:
      cpu_count: 4
      memory_bytes: 4294967296
    artifacts:
      - path: /out/**/*.log
        name: logs
        required: true
      - path: /out/results/**
        name: results

After the process exits, the runner prints each matching artifact. A required artifact that is missing makes the client command fail.

Request KVM

QEMU workloads request capabilities and an allowlisted device explicitly:

hardware:
  requires: ["arch:x86_64", "kvm", "vmx"]
  devices: [kvm]

The runner exposes /dev/kvm only when inventory and admission confirm that it is available. A GCE VM can provide this when nested virtualization and the guest device configuration are enabled; an ordinary SaaS Docker runner does not provide it.

Cloud-init and multi-service deployments

cloud_init is guest-only configuration for a VM-backed workload. It is not a host startup script and is not applied to an ordinary container. The local one-shot client rejects cloud-init, multiple services, and replicas. Remote submission for those deployment features is planned and is not part of the current web app.

On macOS, choose an output directory shared with the Podman machine, such as a path under the checked-out repository.

Bound the capacity wait

Add "queue_timeout_seconds": 120 to a workload submission to stop further placement attempts after two minutes waiting in queued or waiting_capacity. The optional value must be 30–28,800 seconds and no greater than timeout_seconds. Omitting it preserves the existing overall timeout behavior. It is part of the idempotent request: changing it requires a new idempotency key. Status/list JSON includes the configured value (null when omitted) and assigned_at separately from started_at.

The dispatcher checks expiry when scanning and again under its job-row lock. Cleanup queues a terminal infrastructure outcome with outcome.internal: "queue_wait_timed_out"; the normal completion worker records failure, cancels orphan provisioning and releases resources. Settlement occurs on the cleanup/completion cycle, not exactly at the deadline instant. This is not a script failure, and a never-started workload records zero compute usage.

A placement attempt claimed before the cutoff may finish after it. This setting does not interrupt an assigned/dispatching job or extend the overall timeout. Assignment handoff and execution use their existing watchdogs. It is not a promise that a user process will start by the cutoff. A CLI wait timeout is a separate local timeout and does not cancel a workload.