# Run an image workload > Run a one-shot OCI image with RealHW and collect logs and artifacts. Source: https://docs.realhw.dev/guides/run-image-workload/ Documentation index: https://docs.realhw.dev/llms.txt The Python `realhw` CLI can execute an existing OCI or Docker image on the fleet without a GitLab CI pipeline. The image is pulled by Podman on the selected runner; RealHW does not build a Dockerfile or accept a build context in this workflow. The web app currently manages GitLab project runners and interactive hardware leases. It does not yet submit this deployment file as a remote workload. ## Remote workload API (preview) The workload API supports signed input uploads, hardware admission, status, cancellation, logs and artifact downloads. Confirm `workload.submit` in `realhw capabilities --json` before use. Workers must report `workload:io:v1` in addition to the hardware you request. The local command below remains a separate interface. OAuth approval issues the project-scoped token directly: ```sh realhw auth login --project-id PROJECT_UUID \ --scope runners:read --scope workloads:read --scope workloads:write realhw hardware list --json realhw workloads upload bundle.tar \ --account-id ACCOUNT_UUID --project-id PROJECT_UUID --json ``` Save a request as `run.json`, replacing the identifiers and image digest: ```json { "account_id": "ACCOUNT_UUID", "project_id": "PROJECT_UUID", "idempotency_key": "candidate-001-i4", "image": "registry.gitlab.com/PROJECT/qemu@sha256:IMAGE_DIGEST", "required_labels": ["arch:x86_64", "kvm", "saas-linux-medium-amd64"], "preemptible": true, "cpu_count": 4, "memory_bytes": 8589934592, "disk_bytes": 8589934592, "timeout_seconds": 900, "inputs": [{"id": "UPLOADED_INPUT_UUID", "name": "bundle.tar"}] } ``` `preemptible: true` is what makes the nested-KVM spot shape eligible. `gce-kvm-spot` (`n2-standard-4` / `n2-standard-8`) guarantees `arch:x86_64`, `kvm`, and `saas-linux-medium-amd64`. Leave the field off for an on-demand unit job (`saas-linux-medium-amd64` on `gce-generic-od`, or `saas-linux-medium-arm64` with pool `gce-arm-od`). ```sh realhw workloads submit --file run.json --json realhw workloads wait RUN_UUID --json realhw workloads logs RUN_UUID --after -1 --json realhw workloads artifacts RUN_UUID --output artifacts.zip --json ``` Each input is mounted read-only at `/in/NAME`. Write evidence under `/out`. The runner collects that directory after execution, including failed tests. A host crash or transfer failure may prevent collection; missing evidence must never count as a passing test. ZIP files are downloaded without extraction. For a private image, set `REALHW_REGISTRY_USER` and `REALHW_REGISTRY_PASSWORD` to a GitLab credential with `read_registry`. Keep it out of the request file and source control. Image references must pin a SHA-256 digest. The image runs its default entrypoint and command unless `command` is supplied. Custom environment keys must start with `WORKLOAD_`. A successful CLI invocation means the API request succeeded. Inspect the returned `outcome`: `success`, `failed` or `cancelled`. For a failure, distinguish `gitlab_reason: script_failure` from `runner_system_failure`; the field name reflects the shared job lifecycle, even for runs submitted without GitLab CI. Log pages have `chunks`, a decimal-string `next` cursor and `truncated`; follow `next` until the page is empty. No output envelope is added by `--json`. ### Run a pushed image on a GPU GPU workloads use the same image submission API. Request the device with hardware labels and put the CUDA user-space libraries your application needs in the workload image: ```json { "account_id": "ACCOUNT_UUID", "project_id": "PROJECT_UUID", "idempotency_key": "gpu-check-001", "image": "registry.example.com/acme/gpu-check@sha256:IMAGE_DIGEST", "required_labels": ["gpu:nvidia", "gpu:count:1"], "cpu_count": 4, "memory_bytes": 17179869184, "timeout_seconds": 900 } ``` Use `gpu:model:` when a specific accelerator is required, for example `gpu:model:nvidia-tesla-a100`. The host VM and RealHW runner provide the NVIDIA kernel driver, `/dev/nvidia*`, and NVIDIA Container Toolkit; the pushed image provides the application and CUDA user-space libraries. RealHW does not build the image. Check `realhw hardware list --json` first, because GPU capacity is operator-enabled and may be busy or unavailable in a particular zone. The image must be digest-pinned. A private registry requires `REALHW_REGISTRY_USER` and `REALHW_REGISTRY_PASSWORD` in the runner's secret environment; do not put registry credentials in `run.json`. ### Select an on-demand ARM pool To require on-demand ARM capacity, add `"pool": "gce-arm-od"` and use `"required_labels": ["arch:aarch64", "saas-linux-medium-arm64"]` with an ARM64 image. For Cargo tests, use the cargo-unit image; the x86 QEMU/KVM image does not run on T2A. Inspect `realhw hardware list --json` for current pool names, shapes, reported hardware and provisionable capacity. `pool` is an exact placement constraint for existing hosts and new machines. RealHW keeps the labels, resource limits and quota checks, and does not fall back to another pool. Omit it (or use null) for automatic pool selection. A syntactically valid pool with no matching capacity waits until capacity is available or the workload times out. Pool selection is part of the idempotent request: use a new idempotency key when changing it. T2A provisioning tries the configured Iowa zones in order, skipping `us-central1-c`; the production order is `us-central1-a`, `us-central1-b`, then `us-central1-f`. On-demand avoids Spot preemption but does not guarantee stock or successful runner enrollment. Check the workload's provisioning attempts. ### Understand a provisioning wait ```sh realhw workloads status RUN_UUID --json realhw workloads status RUN_UUID --json | jq '.provisioning' ``` Status includes `provisioning: {schema_version: 1, attempts: [], truncated: false}`. It requires `workloads:read` and the same account/project authorization as the workload itself. Only attempts created for that workload and owned by its account and project are included. This is not a fleet-wide machine listing. Each attempt has `id`, `provider`, nullable `shape` and `spot`, `status`, `created_at`, `updated_at`, nullable `completed_at`, and nullable `reason_code` and `message`. `spot: false` means on-demand; null means unspecified. The latest 20 attempts are returned newest first, ordered by creation time and then ID; `truncated: true` means older attempts were omitted. An empty array means no recorded provisioning attempts for this workload, not that capacity is impossible; a workload can run on an existing host without provisioning a new one. Attempt states include `requested`, `creating_vm`, `waiting_enrollment`, `active`, `succeeded`, `failed`, `suppressed`, `cancelled`, `reaping`, and `reaped`. An `active` provisioning attempt does not prove the workload process has started; check the workload's `started_at` and `status`. A `reaped` attempt has been cleaned up; its original failure reason can still be present. Failed attempts do not necessarily mean the overall workload has failed: another attempt may follow. Public failure codes are `provider_capacity_unavailable`, `runner_enrollment_timeout`, and `provisioning_failed`. Messages are sanitized; raw provider errors, credentials, cloud instance identifiers, and other workloads' machine details are never included in this section. Unknown failures use the generic code; contact support with the workload and attempt IDs. Provisioning history is included in single-workload status and the final `wait` response, not the workload list. Older servers may omit `provisioning`; treat that as unavailable diagnostics. A local `wait` timeout does not cancel the workload. GCE ARM on-demand pools can provide a fallback to spot capacity. Switching to on-demand does not by itself resolve a runner enrollment failure; that means the machine did not register before its deadline. Reuse the same request and idempotency key after an ambiguous submission. A changed request with that key returns HTTP 409. A local wait timeout does not cancel the run: use `realhw workloads cancel RUN_UUID`, then poll status. Queue time consumes the workload deadline. Disk is an admission requirement, not a filesystem quota. CPU and memory use the existing reservation machinery. Limits: eight inputs of at most 512 MiB each; output contents at most 512 MiB and 10,000 regular files; logs at most 8 MiB; runtime up to eight hours. Storage objects expire after 14 days. Artifact download URLs expire after one hour. Compose and cloud-init remain unsupported by this remote API. ## QEMU image inputs and results The repository's `images/qemu` runtime expects `/in/bundle.tar`. The tar contains regular files and a `manifest.json` describing the command to run inside the container: ```json { "command": ["bash", "scripts/frozen-test.sh"], "env": {"SERIAL_LOG": "/out/serial.log"}, "timeout_seconds": 210, "provenance": {"revision": "PRODUCT_COMMIT"} } ``` Supply the exact kernel, firmware shim and trusted scripts as bundle inputs. The image checks the KVM device, unpacks under `/workload`, and records `console.log` and `result.json`; the test script writes `serial.log`. The result contains the bundle's SHA-256, command exit code and execution kind (`process_exit`, `timeout` or `infrastructure_failure`). The container exits 1 for a failed command; inspect `result.json` for the original command exit code. The harness remains responsible for the test verdict. A successful process exit or successful artifact download does not prove a kernel test passed. ## Local client command Use a normalized deployment file with the runner CLI: ```sh cargo run -p hw-runner -- run-deployment \ examples/client-smoke.yaml \ --output-dir /tmp/realhw-client-smoke ``` The current client path accepts one service and one replica. It reserves the requested CPU and memory, starts the image with Podman, forwards its logs, and returns the container exit code. ## Collect artifacts The output directory is mounted into the container at `/out`. Declare required files or globs in the service definition: ```yaml name: qemu-test services: qemu: image: ghcr.io/acme/qemu-runner@sha256:... command: ["/usr/local/bin/run-tests"] resources: cpu_count: 4 memory_bytes: 4294967296 artifacts: - path: /out/**/*.log name: logs required: true - path: /out/results/** name: results ``` After the process exits, the runner prints each matching artifact. A required artifact that is missing makes the client command fail. ## Request KVM QEMU workloads request capabilities and an allowlisted device explicitly: ```yaml hardware: requires: ["arch:x86_64", "kvm", "vmx"] devices: [kvm] ``` The runner exposes `/dev/kvm` only when inventory and admission confirm that it is available. A GCE VM can provide this when nested virtualization and the guest device configuration are enabled; an ordinary SaaS Docker runner does not provide it. ## Cloud-init and multi-service deployments `cloud_init` is guest-only configuration for a VM-backed workload. It is not a host startup script and is not applied to an ordinary container. The local one-shot client rejects cloud-init, multiple services, and replicas. Remote submission for those deployment features is planned and is not part of the current web app. On macOS, choose an output directory shared with the Podman machine, such as a path under the checked-out repository. ### Bound the capacity wait Add `"queue_timeout_seconds": 120` to a workload submission to stop further placement attempts after two minutes waiting in `queued` or `waiting_capacity`. The optional value must be 30–28,800 seconds and no greater than `timeout_seconds`. Omitting it preserves the existing overall timeout behavior. It is part of the idempotent request: changing it requires a new idempotency key. Status/list JSON includes the configured value (null when omitted) and `assigned_at` separately from `started_at`. The dispatcher checks expiry when scanning and again under its job-row lock. Cleanup queues a terminal infrastructure outcome with `outcome.internal: "queue_wait_timed_out"`; the normal completion worker records failure, cancels orphan provisioning and releases resources. Settlement occurs on the cleanup/completion cycle, not exactly at the deadline instant. This is not a script failure, and a never-started workload records zero compute usage. A placement attempt claimed before the cutoff may finish after it. This setting does not interrupt an assigned/dispatching job or extend the overall timeout. Assignment handoff and execution use their existing watchdogs. It is not a promise that a user process will start by the cutoff. A CLI `wait` timeout is a separate local timeout and does not cancel a workload.