# RealHW documentation

Generated from the published documentation on each build. Start with https://docs.realhw.dev/get-started/agent-setup/index.md for CLI setup. Current hardware comes from the authenticated fleet API, not these examples. For GitLab event onboarding, see the GitLab event onboarding guide and discover runners.webhook.inspect/configure in the live capabilities API.

# Get started

> Connect RealHW to GitLab and run jobs on selected hardware.

Source: https://docs.realhw.dev/get-started/
Documentation index: https://docs.realhw.dev/llms.txt

RealHW supports GitLab jobs and a REST client for inspecting hardware and reserving interactive consoles.

For CLI setup, start with [Set up with your agent](https://docs.realhw.dev/get-started/agent-setup/). For CI, follow the GitLab steps below.

You do not build RealHW or run RealHW services yourself in the normal customer path. You connect a GitLab project, choose runner tags, and let RealHW place each job on matching capacity.

## Start here

1. [Connect a GitLab project](https://docs.realhw.dev/get-started/connect-gitlab-project/)
2. [Run your first job](https://docs.realhw.dev/get-started/run-your-first-job/)
3. [Choose a runner](https://docs.realhw.dev/get-started/choose-a-runner/)
4. [Review the marker catalog](https://docs.realhw.dev/reference/marker-catalog/)
5. [Run an image workload](https://docs.realhw.dev/guides/run-image-workload/)

## The model

RealHW exposes GitLab runner tags:

- GitLab-compatible SaaS tags such as `saas-linux-medium-amd64`
- RealHW hardware markers such as `gpu:nvidia`, `nic:tier:25g`, and `numa:nodes:2`

GitLab uses those tags to select the RealHW project runner. RealHW then reserves matching hardware before the job starts.


---

# Set up with your agent

> Install the checkout-based RealHW client, authenticate, and verify read-only access.

Source: https://docs.realhw.dev/get-started/agent-setup/
Documentation index: https://docs.realhw.dev/llms.txt

## Set up with your agent

Set up the RealHW CLI on this machine using https://docs.realhw.dev/get-started/agent-setup/index.md as the canonical guide. Read it first and follow its pinned checkout installation instructions; the CLI is not published on PyPI and repository access is required. Preserve existing checkouts and credentials. Use an existing environment token without displaying it, or guide me through browser login with read-only permissions. Never ask me to paste credentials into this conversation. Verify the CLI version, supported operations, and current hardware with JSON commands. Report any access or setup blocker clearly. Stop after verification; do not reserve hardware or start workloads.


## Before you start

The RealHW Python CLI is in early access. **It is not published on PyPI.** Install
from the authorized GitLab repository below; do not install a similarly named package
from a package index. Your GitLab identity must have access to the private repository.
If access is denied, request access from your RealHW administrator and stop setup.

You need Git, Python 3.11 or newer, and [uv](https://docs.astral.sh/uv/getting-started/installation/).
Saved credentials require macOS/Linux file permissions. On other platforms, use an
environment token. Your network must also be allowed through the RealHW ingress rules.

## Install the verified checkout

This guide targets **CLI 0.4.0**, commit
`4ca1bc7472b7d9aafb087e416f85eb8a3b9d9256`. Use a new directory; preserve existing
checkouts and uncommitted work. Use your normal GitLab SSH authentication:

```sh
git clone git@gitlab.com:pvinci/realhw-cli.git realhw-cli-0.4.0
cd realhw-cli-0.4.0
git switch --detach 4ca1bc7472b7d9aafb087e416f85eb8a3b9d9256
uv tool install .
realhw --version
```

Expected version: `realhw, version 0.4.0`. If your installation uses HTTPS GitLab
authentication, use `https://gitlab.com/pvinci/realhw-cli.git` with your configured
credential helper. Never embed a GitLab token in a clone URL or prompt.

For an existing **uv-managed Python client**, use `uv tool install --force .` from the
verified checkout to update it. First inspect `uv tool list` and `command -v realhw`.
If `realhw` is a different program, such as the older Rust CLI, preserve it and resolve
the executable conflict before proceeding. If the executable is missing from PATH,
follow uv's installation output or run `uv tool update-shell` and reopen your terminal.

## Authenticate

Use an existing `REALHW_API_TOKEN` (or `REALHW_API_KEY`) if the user has already supplied
one through the environment. Do not print its value, read it into a prompt, or paste it
into command arguments. The token needs `runners:read` for hardware discovery.

Otherwise, use browser login:

```sh
realhw auth login
```

Default login permissions are read-only: `runners:read consoles:read`. The user signs
in on the web page, checks the displayed device code against the terminal, and approves
access. For a headless terminal, run `realhw auth login --no-browser` and give the user
the displayed verification URL and code. The user completes approval; the agent should
not approve on their behalf. Login expires after 24 hours and can be revoked in Settings.

Environment tokens take precedence over saved browser login. If an environment token
is expired or insufficiently scoped, replace it through the user's secret management
or explicitly unset it before choosing browser login. Never silently change API origin
or reuse a token against a different service.

## Verify without allocating hardware

```sh
realhw --version
realhw auth status --json
realhw capabilities --json
realhw hardware list --json
```

`auth status` checks local configuration only. A successful `hardware list` request
verifies authenticated access with `runners:read`. `capabilities` is public and does
not by itself verify authentication. Both discovery responses have `schema_version: 1`.
An empty `labels` array is a valid current fleet report, not an authentication failure.
Report the CLI version, API origin, verification results, and any blocker without
including credentials. **Stop here.** Reserve hardware only when the user requests it.

## When setup fails

| Failure | Next step |
| --- | --- |
| GitLab clone denied or repository not found | Check the user's GitLab identity and repository membership. Do not substitute another package. |
| CLI missing or wrong version | Check PATH, `command -v realhw`, and `uv tool list`; preserve unrelated installations. |
| Exit 3, HTTP 401 | Supply a valid token or log in again. Check for an expired environment token overriding login. |
| Exit 3, HTTP 403 | Check token scopes and project restrictions. If ingress blocks the request, ask the RealHW administrator to check network access. |
| Exit 4 | Check DNS, connectivity, API origin, and request timeout. |
| HTTP 404 | Confirm the API origin and whether that server supports the requested endpoint. |
| JSON/protocol error or an HTML response | Check proxy/ingress access and the selected API origin. |

Use `--json` for automation; successful results go to stdout, errors to stderr.
See the [JSON contract and exit codes](https://docs.realhw.dev/guides/cli/#json-output-contract-v1).
Do not automatically retry a reservation after a timeout: inspect state first to avoid
allocating twice.

## What is available

The Python CLI discovers hardware and supported operations, lists project runners, and
reserves, lists, and closes interactive console sessions. CLI 0.4.0 adds preview image-workload submission, status, cancellation, logs and
artifact retrieval when the server advertises those operations. Cloud-init and
Compose remain unsupported. Read `realhw capabilities --json` for the server's current contract.

Hardware is discovered live through the API; documentation examples do not promise
current capacity or that multiple labels coexist on one available machine.

For agents reading documentation, start with [/llms.txt](https://docs.realhw.dev/llms.txt), retrieve individual
Markdown pages as needed, or download [/llms-full.txt](https://docs.realhw.dev/llms-full.txt) for the complete
published documentation. These files are regenerated with every docs build.

## Runner event onboarding

After installing CLI 0.4.0, use `realhw capabilities --json` to check for
`runners.webhook.inspect` and `runners.webhook.configure`. These operations inspect
and configure GitLab events for an existing registration; they are separate from
creating a runner. Follow [GitLab event onboarding](https://docs.realhw.dev/guides/gitlab-webhook/) for
preview/apply, required scopes, injected credentials, audit IDs, and delivery checks.
The installation pin above includes these commands; check live capabilities to verify
that the matching backend has been deployed.


---

# Choose a runner

> Pick GitLab-compatible SaaS tags and RealHW hardware markers.

Source: https://docs.realhw.dev/get-started/choose-a-runner/
Documentation index: https://docs.realhw.dev/llms.txt

Choose RealHW capacity in three layers:

1. Start with a GitLab-compatible SaaS tag for the generic runner shape.
2. Add RealHW hardware markers when the job needs specialized hardware.
3. Add `HW_*` variables only when the job needs an explicit reservation or placement override.

## Generic job

```yaml
unit-tests:
  tags:
    - saas-linux-medium-amd64
  script:
    - cargo test
```

## Hardware-specific job

```yaml
xdp-tests:
  tags:
    - saas-linux-large-amd64
    - nic:tier:25g
    - nic:driver:mlx5_core
  variables:
    HW_NIC_TIER: "25g"
  script:
    - ./run-xdp-tests.sh
```

## When to use variables

Tags describe eligible runner capacity. `HW_*` variables describe what this job must reserve now.

Use variables for explicit CPU or memory overrides, NIC tier reservation, SR-IOV VF count, NUMA policy, hugepage size, and placement preference.


---

# Connect a GitLab project

> Add RealHW as a project runner for GitLab.

Source: https://docs.realhw.dev/get-started/connect-gitlab-project/
Documentation index: https://docs.realhw.dev/llms.txt

Connect each GitLab project to RealHW with one project runner. After the runner is connected, jobs choose hardware with `tags` in `.gitlab-ci.yml`.

## Prerequisites

- A GitLab project where you can manage project runners
- A RealHW account with available units or an assigned billing group
- A GitLab personal access token if you want RealHW to create the runner for you

## Managed setup

Use managed setup when your GitLab policy allows RealHW to create the runner.

1. Open the RealHW dashboard.
2. Select **Create project runner**.
3. Enter the GitLab instance URL, usually `https://gitlab.com`.
4. Enter the GitLab project ID or project path.
5. Choose the initial runner tags for this project.
6. Provide a GitLab personal access token with runner creation permissions.
7. Save the runner.

RealHW stores the GitLab runner token server-side. You should not need to copy `glrt-*` runner tokens in the normal path.

## Manual import

Use manual import when your organization does not allow third-party runner creation with a personal access token.

1. Create a project runner in GitLab.
2. Copy the runner authentication token.
3. Open the RealHW dashboard.
4. Select **Import runner**.
5. Paste the token and choose the tags RealHW should expose.

Manual import can be more limited. RealHW may not be able to update GitLab-side tags, pause the runner, delete the runner, or rotate the runner token.

## Verify

In GitLab, confirm the project has an active runner with the tags you selected. In RealHW, confirm the runner status is active.

Next, [run your first job](https://docs.realhw.dev/get-started/run-your-first-job/).


---

# Run your first job

> Use a GitLab-compatible SaaS runner tag with RealHW.

Source: https://docs.realhw.dev/get-started/run-your-first-job/
Documentation index: https://docs.realhw.dev/llms.txt

Start with a normal GitLab-compatible SaaS runner tag. This validates the project runner before you add specialized hardware markers.

Add this job to `.gitlab-ci.yml`:

```yaml
stages:
  - test

unit-tests:
  stage: test
  tags:
    - saas-linux-medium-amd64
  script:
    - echo "running on RealHW"
    - uname -m
```

Push a commit and open the GitLab pipeline.

## Expected result

The job is picked up by the RealHW project runner and runs on compatible Linux capacity.

## Add a hardware marker

After the basic job works, add a marker for the hardware you need:

```yaml
gpu-smoke:
  stage: test
  tags:
    - saas-linux-large-amd64
    - gpu:nvidia
    - gpu:count:1
  script:
    - nvidia-smi
```

GitLab sees the job as a tagged runner job. RealHW uses the hardware markers to place it on a host with matching GPU capacity.


---

# GitLab CI examples

> Copy-paste RealHW jobs for common hardware targets.

Source: https://docs.realhw.dev/gitlab/gitlab-ci-examples/
Documentation index: https://docs.realhw.dev/llms.txt

## Generic Linux

```yaml
unit-tests:
  stage: test
  tags:
    - saas-linux-medium-amd64
  script:
    - cargo test
```

## ARM64

```yaml
arm64-tests:
  stage: test
  tags:
    - saas-linux-medium-arm64
  script:
    - uname -m
    - ./test-arm64.sh
```

## NVIDIA GPU

```yaml
gpu-tests:
  stage: test
  tags:
    - saas-linux-large-amd64
    - gpu:nvidia
    - gpu:count:1
  script:
    - nvidia-smi
    - ./test-gpu.sh
```

## 25G NIC

```yaml
xdp-tests:
  stage: test
  tags:
    - saas-linux-large-amd64
    - nic:tier:25g
    - nic:driver:mlx5_core
  variables:
    HW_NIC_TIER: "25g"
  script:
    - ./run-xdp-tests.sh
```

## SR-IOV VF

```yaml
sriov-tests:
  stage: test
  tags:
    - saas-linux-large-amd64
    - nic:tier:25g
  variables:
    HW_NIC_TIER: "25g"
    HW_SRIOV_VF_COUNT: "1"
  script:
    - ./run-sriov-tests.sh
```

## Hugepages

```yaml
hugepage-tests:
  stage: test
  tags:
    - saas-linux-large-amd64
  variables:
    HW_HUGEPAGE_BYTES: "1073741824"
    HW_NUMA_POLICY: "single-node"
  script:
    - ./run-hugepage-tests.sh
```


---

# Managed runner setup

> Let RealHW create and manage the GitLab project runner.

Source: https://docs.realhw.dev/gitlab/managed-runner-setup/
Documentation index: https://docs.realhw.dev/llms.txt

Managed setup is the recommended path. RealHW creates the GitLab project runner and stores the runner token server-side.

## Inputs

- GitLab instance URL
- GitLab project ID or project path
- Initial runner tags
- GitLab personal access token with runner creation permissions

## Result

RealHW creates a GitLab project runner and starts routing tagged jobs for that project.

Use [GitLab-compatible tags](https://docs.realhw.dev/hardware-markers/gitlab-compatible-tags/) for generic jobs, then add [hardware markers](https://docs.realhw.dev/hardware-markers/) as needed.


---

# Manual runner import

> Import an existing GitLab project runner token.

Source: https://docs.realhw.dev/gitlab/manual-runner-import/
Documentation index: https://docs.realhw.dev/llms.txt

Manual import is for organizations that do not allow RealHW to create runners with a GitLab personal access token.

## Steps

1. Create a project runner in GitLab.
2. Copy the runner authentication token.
3. Import that token in the RealHW dashboard.
4. Configure the tags RealHW should expose for the project.

## Tradeoffs

Manual import can still run jobs, but RealHW may not be able to update GitLab-side tags, pause the runner, delete the runner, or rotate the token.


---

# Project runners

> How RealHW appears to GitLab.

Source: https://docs.realhw.dev/gitlab/project-runner/
Documentation index: https://docs.realhw.dev/llms.txt

RealHW uses one GitLab project runner identity per customer project. GitLab sees a normal project runner with tags. RealHW is the placement layer behind that runner.

```text
GitLab project job
  -> RealHW project runner
  -> RealHW placement and reservation
  -> matching hardware host
```

The runner tags are the customer-facing contract. Jobs select tags in `.gitlab-ci.yml`; RealHW maps those tags to compatible capacity and reserves resources before container execution.

## What GitLab sees

- A project runner
- A runner token
- A list of runner tags
- Tagged jobs from `.gitlab-ci.yml`

## What RealHW handles

- Hardware inventory
- Job admission
- Fleet placement
- CPU, memory, GPU, NIC, NUMA, SR-IOV, and hugepage reservations
- Usage accounting


---

# Troubleshooting

> Diagnose runner selection, disk capacity, and KVM execution failures.

Source: https://docs.realhw.dev/gitlab/troubleshooting/
Documentation index: https://docs.realhw.dev/llms.txt

## Job is stuck

Check that the job tags match tags exposed by the RealHW project runner.

```yaml
tags:
  - saas-linux-medium-amd64
```

If the job has a hardware marker, confirm your project has access to that hardware class.

## Job runs on the wrong shape

Use one GitLab-compatible SaaS tag for the generic shape:

```yaml
tags:
  - saas-linux-large-amd64
```

Avoid mixing multiple generic size tags unless you intentionally allow any of those sizes.

## Hardware job waits

Hardware markers describe host capability, not immediate availability. A valid GPU, NIC, or SR-IOV request can wait if matching hardware is busy.

## Job fails before container start

The request may be unsatisfiable or blocked by policy. Check unsupported markers, unavailable project entitlements, excessive `HW_CPU` or `HW_MEMORY`, and unsupported `HW_*` variables.

## Disk capability probe fails

Compare the reported free space with the job's floor on the build filesystem.
A script that prints “infra failure” and exits still produces GitLab
`script_failure`; the message alone does not change the failure category.
RealHW disk rejection before execution uses `runner_system_failure`.

Ask the fleet operator to inspect disk capacity, drain the affected host, and
reclaim unused managed caches or expand capacity before retrying. RealHW
admission reserves a default 10 GiB floor plus 4 GiB of growth headroom per
active job. A workload can require more; this check is not a disk quota.

## KVM is missing or unusable

Request the `kvm` capability in addition to the desired size or architecture.
A SaaS-compatible size tag describes a shape and does not guarantee KVM.
The host must provide usable KVM, and the executor must expose `/dev/kvm`
to the job container. Check both layers when the probe fails.

RealHW supports owned KVM hosts and a dedicated GCE N2 nested-KVM pool.
The GCE worker creates a KVM VM during bootstrap before enrolling, but the
job must still verify that its container can use the device.

## QEMU starts but the guest times out

Once disk and KVM probes pass, inspect the serial log for the expected boot
markers and the first panic. A QEMU timeout is not evidence of the original
disk failure. Preserve the job revision and artifacts so kernel failures can
be investigated separately from runner-capacity problems.


---

# API keys

> Authenticate scripts with scoped RealHW credentials.

Source: https://docs.realhw.dev/guides/api-keys/
Documentation index: https://docs.realhw.dev/llms.txt

Use **Settings → API keys** to create a personal key for a script or client.
Choose a name, permissions, an optional project restriction, and an expiry.
Save the secret when it is shown: RealHW cannot display it again.

Keys belong to your user and account. A project restriction narrows access further.
You can view each key’s prefix, permissions, expiration and last-used time, and
revoke it from Settings. Revocation blocks new authenticated requests; it does
not terminate existing workloads or an already-open console connection.

## Permissions

| Permission | Access |
| --- | --- |
| `runners:read` | List project runners in your account, limited to the selected project |
| `consoles:read` | List your console sessions within the key’s account and project scope |
| `consoles:write` | Reserve and close your console sessions |
| `consoles:connect` | Open a WebSocket connection to your console sessions |
| `workloads:read` | Read workload status, logs and artifacts (preview) |
| `workloads:write` | Upload inputs, submit and cancel workloads (preview) |

Keys cannot access platform administration, billing changes, or key management.
OAuth device approval directly issues a scoped token; that token cannot create
additional tokens. Request new permissions through a new browser approval.
For remote execution, verify the deployed operations with `realhw capabilities
--json` and follow [Run an image workload](https://docs.realhw.dev/guides/run-image-workload/).

## Use a key

Store the secret in your shell’s environment or your secret manager as
`REALHW_API_KEY`. Supply your account ID as `REALHW_ACCOUNT_ID`.

```sh
curl --fail-with-body \
  -H "Authorization: Bearer $REALHW_API_KEY" \
  "https://realhw.dev/api/v1/accounts/$REALHW_ACCOUNT_ID/virtual-runners"
```

Reserve a console with `consoles:write` using your RealHW account and project UUIDs:

```sh
curl --fail-with-body -X POST \
  -H "Authorization: Bearer $REALHW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "account_id": "YOUR_ACCOUNT_UUID",
    "project_id": "YOUR_PROJECT_UUID",
    "protocol": "shell",
    "required_labels": ["arch:x86_64"],
    "image": "alpine:3.20",
    "ttl_seconds": 3600
  }' \
  https://realhw.dev/api/v1/console/sessions
```

List sessions with `GET /api/v1/console/sessions`. Close one with
`DELETE /api/v1/console/sessions/{id}` and a JSON body such as
`{"reason":"finished"}`. A WebSocket client can connect to
`/api/v1/console/sessions/{id}/ws` with the same Authorization header and
`consoles:connect` permission. Browser consoles continue to use your sign-in session.

Expired, revoked, or invalid keys return `401`. An operation outside the key’s
permissions returns `403`; inaccessible individual sessions may return `404`.


---

# Python client and CLI

> Use RealHW from a terminal, Python script, or LLM agent.

Source: https://docs.realhw.dev/guides/cli/
Documentation index: https://docs.realhw.dev/llms.txt

The `realhw-cli` repository provides a Python REST client and the Click-based `realhw`
command. For the pinned checkout installation, browser login, and read-only verification,
start with [Set up with your agent](https://docs.realhw.dev/get-started/agent-setup/).

The client is in early access and is **not published on PyPI**. Repository access is
required. Supported operations include discovery, runner listing, and interactive
console reservation. CLI 0.3.0 adds [preview image workloads](https://docs.realhw.dev/guides/run-image-workload/),
including logs and artifacts, when the server advertises `workload.submit`.

## Authenticate

For automation, inject `REALHW_API_TOKEN` from your secret manager. Create the token in
**Settings → API keys** with the permissions and project restriction you need.
`REALHW_API_KEY` is also accepted. Do not include tokens in command arguments or prompts.

For an interactive terminal:

```sh
realhw auth login
realhw consoles list
```

Login opens RealHW in your browser. Sign in, review the requested permissions, confirm
that the displayed code matches your terminal, then authorize the CLI. Never approve a
code sent by someone else. For SSH or a terminal without a browser, use
`realhw auth login --no-browser` and open the printed URL on another device.

This is the OAuth device authorization flow (RFC 8628), using the public client ID
`realhw-cli`. The default permissions are `runners:read` and `consoles:read`. Access lasts
24 hours; log in again after expiry. No refresh token is issued. The credential appears
in **Settings → API keys**, where you can revoke it.

```sh
realhw auth login --scope consoles:read --scope consoles:write --project-id <project-uuid>
realhw auth status
realhw auth logout
```

Logout forgets the local credential; it does not revoke remote access or unset environment
tokens. Status shows local metadata without exposing the token. Saved credentials use
private files (0600) in `~/.config/realhw`, separated by API origin. This storage requires
macOS/Linux file permissions; other platforms can use environment tokens.

## Reserve a console

```sh
realhw runners list --account-id <account-uuid>
realhw consoles reserve --account-id <account-uuid> --project-id <project-uuid> \
  --label arch:x86_64 --label kvm --image ubuntu:24.04 --ttl 3600
realhw consoles list
realhw consoles close <session-uuid> --reason 'Finished debugging'
```

Use current labels returned by `hardware list --json`. Reservation may allocate billable hardware.
This creates an interactive shell reservation, not a one-shot image job. Console
WebSocket streaming remains available through the web app.

## Scripts and Python

Commands return JSON on stdout and errors on stderr, with nonzero exit codes on failure.
Use `--compact` before the subcommand for single-line JSON. Login instructions go to
stderr. There are no hidden confirmation prompts.

```python
import os
from realhw_cli import RealHWClient

with RealHWClient(token=os.environ["REALHW_API_TOKEN"]) as client:
    sessions = client.consoles()
```

`REALHW_API_URL` overrides the default `https://realhw.dev`. Existing ingress/IP access
restrictions still apply; tokens do not bypass them. HTTPS is required except for
loopback development. Redirects are disabled and mutations are never automatically
retried. After a timeout, inspect state before repeating a reservation.

Browser login requires the matching server OAuth endpoints and `/device` page. An older
server returns 404; use an API token until that server is upgraded.

## JSON output contract v1

The CLI uses Click. `--json` is accepted before, between, or after subcommands:

```sh
realhw --json capabilities
realhw hardware list --json
realhw consoles --json list
```

Successful commands write exactly one JSON value followed by a newline to stdout.
`--json` emits compact single-line JSON; `--compact` is an alias. Without either flag,
output is still JSON, formatted for people. REST responses pass through without a
CLI envelope, renamed fields, or type conversions. Unknown server fields are retained;
empty HTTP responses become `null`. Help and version are plain text.

Errors write `{"error":{"code":"…","message":"…","status":null}}` to stderr and leave
stdout empty. `status` is an HTTP status integer when applicable. Parse `code` and
`status`; messages are for people and may change. Login instructions also use stderr.
Local authentication commands return credential metadata, never the token.

Exit codes: 0 success; 1 API/configuration/protocol error; 2 usage error;
3 authentication/permission error; 4 network/timeout; 130 interrupted.
Consumers must ignore unknown fields. Discovery responses carry `schema_version: 1`;
additive fields are compatible, while breaking changes require a new contract version.
The CLI preserves each API endpoint's response schema rather than imposing its own.

## Current hardware and supported operations

```sh
realhw capabilities --json
realhw hardware list --json
```

`GET /api/v1/capabilities` is public and enumerates supported token-authenticated
operations, methods, paths, permissions, CLI equivalents, and mutation request schemas.
The console connection uses WebSocket and has no CLI streaming command yet.

`GET /api/v1/fleet/hardware` requires `runners:read`. It returns:

```json
{"schema_version":1,"observed_at":"2026-09-22T12:00:00Z","host_freshness_seconds":120,"labels":["arch:x86_64","kvm"]}
```

`labels` remains the sorted, deduplicated union from online, non-decommissioned
hosts seen within 120 seconds. A reported `kvm` label requires the host's explicit
`/dev/kvm` report. Busy or draining hosts can still report capabilities.

The additive `provisionable` array lists enabled cloud shapes even when no host is
running. Each entry has `provider`, `pool`, `shape`, sorted `labels`, `cpu_count`,
`memory_bytes` (a decimal string), and `boot_disk_size_gb`. Resource fields can be
null when unspecified. These are configured shape contracts, not measured hosts;
boot disk size is not a guarantee of free workspace disk.

For example, an enabled GCP N2 KVM pool advertises `arch:x86_64` and `kvm`, while
ARM64 cloud pools advertise `arch:aarch64`. Disabled/unpriced shapes and providers
excluded by platform policy are omitted. The server reads the scheduler's catalog
and rate enablement; the CLI does not maintain its own machine list.

`availability_checked_at` records the provider's last availability publication.
`quota_fits` is null if there is no report or it is older than five minutes;
false indicates the latest fresh quota check failed. `cooldown_until` is non-null
while a capacity retry cooldown is active. These fields do not guarantee provider
stock, active provider credentials, project authorization, or immediate placement.
Admission makes those decisions. An empty running fleet can therefore return
`labels: []` with a nonempty `provisionable` array. Older servers may omit the array;
clients must treat that as unknown, not proof that cloud capacity is unavailable.

ARM QEMU can run with `-accel tcg` without KVM on ARM64 or x86 hosts. The container
image must match the host architecture; the emulated guest architecture can differ.
Request `kvm` only for a workload that requires hardware acceleration, and use a
compatible host/guest architecture. Never silently substitute TCG for a KVM test.

The older public `hardware labels` command includes cached shape labels and is not a
current hardware snapshot. Use `hardware list` for current reports.
Preview workload operations appear in discovery after the API is deployed.
Cloud-init and Compose remain unsupported; do not infer them from image execution.

## GitLab webhook onboarding (CLI 0.4.0)

Use `realhw runners webhook inspect RUNNER_UUID` and
`realhw runners webhook configure RUNNER_UUID --reason REASON` to inspect and preview.
Add `--apply` after reviewing the result. Configuration needs `runners:write`;
inspection needs `runners:read`. Optional `--gitlab-token-env VARIABLE` names a secret
environment variable, never a token value. Ops equivalents are under
`realhw ops runners webhook`, with separate Ops credentials and policy discovery.
See [GitLab event onboarding](https://docs.realhw.dev/guides/gitlab-webhook/) for the API, safe credential
handling, audit IDs, server prerequisites, and actual delivery verification.


---

# Debug runner selection

> Diagnose why a GitLab job did not run on the expected hardware.

Source: https://docs.realhw.dev/guides/debug-runner-selection/
Documentation index: https://docs.realhw.dev/llms.txt

## Check the tags

The job must use tags exposed by the RealHW project runner.

```yaml
tags:
  - saas-linux-large-amd64
  - gpu:nvidia
```

## Check marker spelling

Markers are lowercase and use `:` separators.

Good:

```text
nic:tier:25g
gpu:count:1
arch:aarch64
```

## Check variables

Use `HW_*` variables for explicit reservations. A tag like `nic:tier:25g` selects eligible hosts; `HW_NIC_TIER=25g` asks RealHW to reserve a NIC tier for the job.

## Check availability

A valid hardware request can wait while matching capacity is busy. If the request can never be satisfied by your project's capacity, the job fails before container start.


---

# GitLab event onboarding

> Inspect and configure project-scoped GitLab job and pipeline event delivery through the API or CLI.

Source: https://docs.realhw.dev/guides/gitlab-webhook/
Documentation index: https://docs.realhw.dev/llms.txt

GitLab job events create durable demand that can start cold RealHW capacity. A runner
registration alone does not establish event delivery. CLI 0.4.0 adds the commands below;
check `realhw capabilities --json` for server support before applying changes. A 404
means the matching API has not been deployed. The retired Rust CLI has no onboarding
support; use the Python `realhw-cli` repository.

## Customer workflow

Use an existing runner UUID from `realhw runners list --account-id ACCOUNT_UUID`.
Inspection requires `runners:read`; configuration requires `runners:write` and the
registration's account/project scope. Browser login defaults to read-only permissions.
Request the necessary permissions explicitly or provision a restricted API key.

```sh
realhw capabilities --json
realhw runners webhook inspect RUNNER_UUID --json
realhw runners webhook configure RUNNER_UUID --reason 'Enable pending-job demand' --json
realhw runners webhook configure RUNNER_UUID --reason 'Enable pending-job demand' --apply --json
```

Configure defaults to preview: it reads GitLab but does not create/update a hook or
persist a supplied credential. It records an audit operation, including previews and
failures. Review the callback, action, blockers, and limitations before applying.
An apply creates one hook or updates the existing exact-callback hook in place. Multiple
matching hooks block the operation; unrelated hooks are left intact.

If the registration has no usable stored project API credential, inject a token from
your secret manager into an environment variable, then name that variable:

```sh
realhw runners webhook configure RUNNER_UUID --reason 'Restore project event delivery' \
  --gitlab-token-env PROJECT_GITLAB_TOKEN --json
realhw runners webhook configure RUNNER_UUID --reason 'Restore project event delivery' \
  --gitlab-token-env PROJECT_GITLAB_TOKEN --apply --json
```

Never put the token value in an argument, prompt, JSON file, or documentation. GitLab
requires project read access and sufficient API permissions/role to manage hooks
(normally Maintainer). Successful apply encrypts the supplied project credential for
subsequent job metadata reads. Failed apply and preview do not replace it. Tokens are
never returned. Use the narrowest project-scoped token GitLab supports.

## Operator workflow

Set `REALHW_OPS_API_BASE` and `REALHW_OPS_TOKEN` through your approved secret environment.
Ops commands never fall back to customer credentials and read the Ops policy before
webhook operations.

```sh
realhw ops policy --json
realhw ops runners webhook inspect RUNNER_UUID --json
realhw ops runners webhook configure RUNNER_UUID --reason 'Restore event onboarding' --json
realhw ops runners webhook configure RUNNER_UUID --reason 'Restore event onboarding' --apply --json
```

The customer endpoints are `GET` and `POST`
`/api/v1/virtual-runners/{runner_id}/gitlab-webhook`. Typed Ops equivalents use
`/admin/realhw/virtual-runners/{runner_id}/gitlab-webhook`. POST accepts `reason`,
`dry_run` (defaults true), and optional write-only `gitlab_access_token`. Configure
returns `operation_id`, `dry_run`, `action` (`create` or `reconcile`), and `status`.
Rust DTOs generate the browser contract; the Python client preserves the API JSON.

## Delivery and recovery

RealHW derives a per-project hook verifier from its server-only root. Only the derived
verifier goes to that project's GitLab instance. The receiver continues accepting
retained legacy verifiers during migration. The callback is derived from the trusted
server `REALHW_GITLAB_WEBHOOK_ORIGIN` HTTPS origin, never from a request Host header. The dashboard origin may be IP restricted and must not be used here. Operators must
ensure that the configured public API origin routes `/api/v1/gitlab/webhooks/` to this API.

A successful configuration means provider readback verified the hook identity, job and
pipeline subscriptions, enabled status and TLS verification. GitLab versions may omit
`token_present`; `verification_token_present: null` remains unknown. Configuration is
not evidence that an event arrived or that a job ran. `provider_inspected: false` means
hook presence is unknown; an empty `hooks` array in that state proves nothing.

After applying, trigger an authorized real pipeline and verify retained demand for
that project, scheduling, and a running job. `last_demand_received_at` is historical,
not proof of current delivery. Do not cancel productive jobs just to generate events.
After a timeout, inspect first: an external GitLab write may have completed even if
RealHW could not finish its database transaction. The retained audit intent records
that uncertainty; rerunning reconciliation uses the same exact callback identity.

Developers: run API provider/receiver tests, isolated PostgreSQL scope/audit tests,
CLI contract tests, typeshare, and the docs build. Do not validate against production
by creating arbitrary hooks or writing credentials to disk.


---

# Run an image workload

> Run a one-shot OCI image with RealHW and collect logs and artifacts.

Source: https://docs.realhw.dev/guides/run-image-workload/
Documentation index: https://docs.realhw.dev/llms.txt

The Python `realhw` CLI can execute an existing OCI or Docker image on the
fleet without a GitLab CI pipeline. The image is pulled by Podman on the selected
runner; RealHW does not build a Dockerfile or accept a build context in this
workflow.

The web app currently manages GitLab project runners and interactive hardware
leases. It does not yet submit this deployment file as a remote workload.

## Remote workload API (preview)

The workload API supports signed input uploads, hardware admission, status,
cancellation, logs and artifact downloads. Confirm `workload.submit` in
`realhw capabilities --json` before use. Workers must report `workload:io:v1`
in addition to the hardware you request.
The local command below remains a separate interface.

OAuth approval issues the project-scoped token directly:

```sh
realhw auth login --project-id PROJECT_UUID \
  --scope runners:read --scope workloads:read --scope workloads:write
realhw hardware list --json
realhw workloads upload bundle.tar \
  --account-id ACCOUNT_UUID --project-id PROJECT_UUID --json
```

Save a request as `run.json`, replacing the identifiers and image digest:

```json
{
  "account_id": "ACCOUNT_UUID",
  "project_id": "PROJECT_UUID",
  "idempotency_key": "candidate-001-i4",
  "image": "registry.gitlab.com/PROJECT/qemu@sha256:IMAGE_DIGEST",
  "required_labels": ["arch:x86_64", "kvm", "saas-linux-medium-amd64"],
  "preemptible": true,
  "cpu_count": 4,
  "memory_bytes": 8589934592,
  "disk_bytes": 8589934592,
  "timeout_seconds": 900,
  "inputs": [{"id": "UPLOADED_INPUT_UUID", "name": "bundle.tar"}]
}
```

`preemptible: true` is what makes the nested-KVM spot shape eligible. `gce-kvm-spot` (`n2-standard-4` / `n2-standard-8`) guarantees `arch:x86_64`, `kvm`, and `saas-linux-medium-amd64`. Leave the field off for an on-demand unit job (`saas-linux-medium-amd64` on `gce-generic-od`, or `saas-linux-medium-arm64` with pool `gce-arm-od`).

```sh
realhw workloads submit --file run.json --json
realhw workloads wait RUN_UUID --json
realhw workloads logs RUN_UUID --after -1 --json
realhw workloads artifacts RUN_UUID --output artifacts.zip --json
```

Each input is mounted read-only at `/in/NAME`. Write evidence under `/out`.
The runner collects that directory after execution, including failed tests.
A host crash or transfer failure may prevent collection; missing evidence must
never count as a passing test. ZIP files are downloaded without extraction.

For a private image, set `REALHW_REGISTRY_USER` and `REALHW_REGISTRY_PASSWORD`
to a GitLab credential with `read_registry`. Keep it out of the request file
and source control. Image references must pin a SHA-256 digest. The image runs
its default entrypoint and command unless `command` is supplied. Custom
environment keys must start with `WORKLOAD_`.

A successful CLI invocation means the API request succeeded. Inspect the
returned `outcome`: `success`, `failed` or `cancelled`. For a failure, distinguish
`gitlab_reason: script_failure` from `runner_system_failure`; the field name
reflects the shared job lifecycle, even for runs submitted without GitLab CI.
Log pages have `chunks`, a decimal-string `next` cursor and `truncated`; follow
`next` until the page is empty. No output envelope is added by `--json`.

### Run a pushed image on a GPU

GPU workloads use the same image submission API. Request the device with
hardware labels and put the CUDA user-space libraries your application needs in
the workload image:

```json
{
  "account_id": "ACCOUNT_UUID",
  "project_id": "PROJECT_UUID",
  "idempotency_key": "gpu-check-001",
  "image": "registry.example.com/acme/gpu-check@sha256:IMAGE_DIGEST",
  "required_labels": ["gpu:nvidia", "gpu:count:1"],
  "cpu_count": 4,
  "memory_bytes": 17179869184,
  "timeout_seconds": 900
}
```

Use `gpu:model:<name>` when a specific accelerator is required, for example
`gpu:model:nvidia-tesla-a100`. The host VM and RealHW runner provide the NVIDIA
kernel driver, `/dev/nvidia*`, and NVIDIA Container Toolkit; the pushed image
provides the application and CUDA user-space libraries. RealHW does not build
the image. Check `realhw hardware list --json` first, because GPU capacity is
operator-enabled and may be busy or unavailable in a particular zone.

The image must be digest-pinned. A private registry requires
`REALHW_REGISTRY_USER` and `REALHW_REGISTRY_PASSWORD` in the runner's secret
environment; do not put registry credentials in `run.json`.

### Select an on-demand ARM pool

To require on-demand ARM capacity, add `"pool": "gce-arm-od"` and use
`"required_labels": ["arch:aarch64", "saas-linux-medium-arm64"]` with an ARM64
image. For Cargo tests, use the cargo-unit image; the x86 QEMU/KVM image does
not run on T2A. Inspect `realhw hardware list --json` for current pool names,
shapes, reported hardware and provisionable capacity.

`pool` is an exact placement constraint for existing hosts and new machines.
RealHW keeps the labels, resource limits and quota checks, and does not fall
back to another pool. Omit it (or use null) for automatic pool selection.
A syntactically valid pool with no matching capacity waits until capacity is
available or the workload times out. Pool selection is part of the idempotent
request: use a new idempotency key when changing it.

T2A provisioning tries the configured Iowa zones in order, skipping
`us-central1-c`; the production order is `us-central1-a`, `us-central1-b`, then
`us-central1-f`. On-demand avoids Spot preemption but does not guarantee stock
or successful runner enrollment. Check the workload's provisioning attempts.

### Understand a provisioning wait

```sh
realhw workloads status RUN_UUID --json
realhw workloads status RUN_UUID --json | jq '.provisioning'
```

Status includes `provisioning: {schema_version: 1, attempts: [], truncated: false}`.
It requires `workloads:read` and the same account/project authorization as the
workload itself. Only attempts created for that workload and owned by its account
and project are included. This is not a fleet-wide machine listing.

Each attempt has `id`, `provider`, nullable `shape` and `spot`, `status`,
`created_at`, `updated_at`, nullable `completed_at`, and nullable `reason_code`
and `message`. `spot: false` means on-demand; null means unspecified. The latest
20 attempts are returned newest first, ordered by creation time and then ID;
`truncated: true` means older attempts were omitted. An empty array means no
recorded provisioning attempts for this workload, not that capacity is impossible;
a workload can run on an existing host without provisioning a new one.

Attempt states include `requested`, `creating_vm`, `waiting_enrollment`, `active`,
`succeeded`, `failed`, `suppressed`, `cancelled`, `reaping`, and `reaped`.
An `active` provisioning attempt does not prove the workload process has started;
check the workload's `started_at` and `status`. A `reaped` attempt has been cleaned
up; its original failure reason can still be present. Failed attempts do not
necessarily mean the overall workload has failed: another attempt may follow.

Public failure codes are `provider_capacity_unavailable`,
`runner_enrollment_timeout`, and `provisioning_failed`. Messages are sanitized;
raw provider errors, credentials, cloud instance identifiers, and other workloads'
machine details are never included in this section. Unknown failures use the
generic code; contact support with the workload and attempt IDs.

Provisioning history is included in single-workload status and the final `wait`
response, not the workload list. Older servers may omit `provisioning`; treat that
as unavailable diagnostics. A local `wait` timeout does not cancel the workload.

GCE ARM on-demand pools can provide a fallback to spot capacity. Switching to
on-demand does not by itself resolve a runner enrollment failure; that means the
machine did not register before its deadline.

Reuse the same request and idempotency key after an ambiguous submission.
A changed request with that key returns HTTP 409. A local wait timeout does not
cancel the run: use `realhw workloads cancel RUN_UUID`, then poll status.
Queue time consumes the workload deadline. Disk is an admission requirement,
not a filesystem quota. CPU and memory use the existing reservation machinery.

Limits: eight inputs of at most 512 MiB each; output contents at most 512 MiB
and 10,000 regular files; logs at most 8 MiB; runtime up to eight hours. Storage
objects expire after 14 days. Artifact download URLs expire after one hour.
Compose and cloud-init remain unsupported by this remote API.

## QEMU image inputs and results

The repository's `images/qemu` runtime expects `/in/bundle.tar`. The tar contains
regular files and a `manifest.json` describing the command to run inside the
container:

```json
{
  "command": ["bash", "scripts/frozen-test.sh"],
  "env": {"SERIAL_LOG": "/out/serial.log"},
  "timeout_seconds": 210,
  "provenance": {"revision": "PRODUCT_COMMIT"}
}
```

Supply the exact kernel, firmware shim and trusted scripts as bundle inputs.
The image checks the KVM device, unpacks under `/workload`, and records
`console.log` and `result.json`; the test script writes `serial.log`. The result
contains the bundle's SHA-256, command exit code and execution kind
(`process_exit`, `timeout` or `infrastructure_failure`). The container exits 1
for a failed command; inspect `result.json` for the original command exit code.

The harness remains responsible for the test verdict. A successful process
exit or successful artifact download does not prove a kernel test passed.

## Local client command

Use a normalized deployment file with the runner CLI:

```sh
cargo run -p hw-runner -- run-deployment \
  examples/client-smoke.yaml \
  --output-dir /tmp/realhw-client-smoke
```

The current client path accepts one service and one replica. It reserves the
requested CPU and memory, starts the image with Podman, forwards its logs, and
returns the container exit code.

## Collect artifacts

The output directory is mounted into the container at `/out`. Declare required
files or globs in the service definition:

```yaml
name: qemu-test
services:
  qemu:
    image: ghcr.io/acme/qemu-runner@sha256:...
    command: ["/usr/local/bin/run-tests"]
    resources:
      cpu_count: 4
      memory_bytes: 4294967296
    artifacts:
      - path: /out/**/*.log
        name: logs
        required: true
      - path: /out/results/**
        name: results
```

After the process exits, the runner prints each matching artifact. A required
artifact that is missing makes the client command fail.

## Request KVM

QEMU workloads request capabilities and an allowlisted device explicitly:

```yaml
hardware:
  requires: ["arch:x86_64", "kvm", "vmx"]
  devices: [kvm]
```

The runner exposes `/dev/kvm` only when inventory and admission confirm that it
is available. A GCE VM can provide this when nested virtualization and the
guest device configuration are enabled; an ordinary SaaS Docker runner does
not provide it.

## Cloud-init and multi-service deployments

`cloud_init` is guest-only configuration for a VM-backed workload. It is not a
host startup script and is not applied to an ordinary container. The local
one-shot client rejects cloud-init, multiple services, and replicas. Remote
submission for those deployment features is planned and is not part of the
current web app.

On macOS, choose an output directory shared with the Podman machine, such as a
path under the checked-out repository.

### Bound the capacity wait

Add `"queue_timeout_seconds": 120` to a workload submission to stop further
placement attempts after two minutes waiting in `queued` or `waiting_capacity`.
The optional value must be 30–28,800 seconds and no greater than
`timeout_seconds`. Omitting it preserves the existing overall timeout behavior.
It is part of the idempotent request: changing it requires a new idempotency key.
Status/list JSON includes the configured value (null when omitted) and
`assigned_at` separately from `started_at`.

The dispatcher checks expiry when scanning and again under its job-row lock.
Cleanup queues a terminal infrastructure outcome with
`outcome.internal: "queue_wait_timed_out"`; the normal completion worker records
failure, cancels orphan provisioning and releases resources. Settlement occurs
on the cleanup/completion cycle, not exactly at the deadline instant. This is
not a script failure, and a never-started workload records zero compute usage.

A placement attempt claimed before the cutoff may finish after it. This setting
does not interrupt an assigned/dispatching job or extend the overall timeout.
Assignment handoff and execution use their existing watchdogs. It is not a
promise that a user process will start by the cutoff. A CLI `wait` timeout is a
separate local timeout and does not cancel a workload.


---

# Run on a 25G NIC

> Select a host with a 25G network interface.

Source: https://docs.realhw.dev/guides/run-on-25g-nic/
Documentation index: https://docs.realhw.dev/llms.txt

Use NIC tier and driver markers for packet, XDP, DPDK, or driver tests.

```yaml
xdp-tests:
  stage: test
  tags:
    - saas-linux-large-amd64
    - nic:tier:25g
    - nic:driver:mlx5_core
  variables:
    HW_NIC_TIER: "25g"
    HW_NIC_DRIVER: "mlx5_core"
  script:
    - ./run-xdp-tests.sh
```

The selected NIC name is exposed to the job environment when the runner attaches a NIC to the container.


---

# Run on ARM64

> Select ARM64 RealHW capacity from GitLab CI.

Source: https://docs.realhw.dev/guides/run-on-arm64/
Documentation index: https://docs.realhw.dev/llms.txt

Use a GitLab-compatible ARM64 tag for ARM jobs.

```yaml
arm64-tests:
  stage: test
  tags:
    - saas-linux-medium-arm64
  script:
    - test "$(uname -m)" = "aarch64"
    - ./test-arm64.sh
```

If you need only architecture matching and not a SaaS shape, use `arch:aarch64`.


---

# Run on a GPU

> Select GPU hardware from GitLab CI.

Source: https://docs.realhw.dev/guides/run-on-gpu/
Documentation index: https://docs.realhw.dev/llms.txt

Add GPU markers to a normal SaaS-compatible shape.

```yaml
gpu-tests:
  stage: test
  tags:
    - saas-linux-large-amd64
    - gpu:nvidia
    - gpu:count:1
  script:
    - nvidia-smi
    - ./test-gpu.sh
```

Use `gpu:model:<name>` when the job requires a specific model.


---

# Run with hugepages

> Reserve hugepages for a GitLab job.

Source: https://docs.realhw.dev/guides/run-with-hugepages/
Documentation index: https://docs.realhw.dev/llms.txt

Set `HW_HUGEPAGE_BYTES` for jobs that need hugepage-backed memory.

```yaml
hugepage-tests:
  stage: test
  tags:
    - saas-linux-large-amd64
  variables:
    HW_HUGEPAGE_BYTES: "1073741824"
    HW_NUMA_POLICY: "single-node"
  script:
    - ./run-hugepage-tests.sh
```

Use `HW_NUMA_POLICY` when hugepage locality matters.


---

# Run with SR-IOV

> Reserve SR-IOV virtual functions for a GitLab job.

Source: https://docs.realhw.dev/guides/run-with-sriov/
Documentation index: https://docs.realhw.dev/llms.txt

Request NIC capacity with tags, then reserve VFs with `HW_SRIOV_VF_COUNT`.

```yaml
sriov-tests:
  stage: test
  tags:
    - saas-linux-large-amd64
    - nic:tier:25g
  variables:
    HW_NIC_TIER: "25g"
    HW_SRIOV_VF_COUNT: "1"
  script:
    - ./run-sriov-tests.sh
```

Use this for jobs that need a VF assigned to the test container.


---

# Service boundaries and pilot readiness

> What RealHW currently supports and what requires an agreed pilot configuration.

Source: https://docs.realhw.dev/guides/service-boundaries/
Documentation index: https://docs.realhw.dev/llms.txt

RealHW offers complete GitLab execution offload: connect a project and keep its
existing pipeline and supported job definitions in GitLab. RealHW supplies
execution capacity for ordinary jobs and hardware-specific jobs, returning traces,
results and artifacts to GitLab. You do not need to convert each job into an API
request. Runner settings and compatible tag coverage must match your project;
validate the CI features your pipeline uses before switching execution over.

You can also run an existing container image directly through the workload
API/CLI, without a pipeline. That path selects reported hardware and returns
status, logs and declared outputs; it does not build your Dockerfile. GitLab
pipelines can retain their existing build jobs.

The current system operates trusted workloads. A general-purpose public service
for arbitrary untrusted code is not currently offered. API authentication and
container packaging do not establish tenant isolation. External pilots require
an agreed isolation boundary, supported hardware profile and operational review.

## One service, configurable execution

Hardware requirements (x86, ARM64, GPU and devices), fleet ownership
(RealHW-managed or customer cloud/hardware), and capacity policy (best effort or
reserved) are dimensions of the same service. GitLab, API and CLI are entry
points. A configuration must be supported and validated as a whole; these choices
do not imply that every combination is available. In particular, ARM KVM needs
hardware that exposes compatible virtualization. Reported capabilities and
provisionable catalog entries are distinct from a verified workload result.

## Capacity and execution

Hardware discovery reports online and provisionable hardware; a matching label
is not a reservation or an immediate-start guarantee. Any reserved-capacity offer
requires an explicit agreement identifying capacity, concurrency and replacement
behavior. A successful provisioning probe does not establish that your program
or guest kernel boots.

A workload's `started_at` currently means a runner accepted it. The runner may
still be preparing the workspace, inputs or image. Do not use that timestamp as
proof that QEMU or your application executed.

Use `queue_timeout_seconds` to bound further dispatch attempts while a workload
is queued or waiting for capacity. See [image workloads](../run-image-workload/)
for the cutoff semantics and the separate overall timeout.

## Evidence and failures

A script failure and an infrastructure failure have different meanings. A job
that never reached execution is not a product test verdict. Status and
provisioning history identify the available evidence.

Final collection prioritizes artifacts over cache uploads and attempts to retain
the workspace when collection fails. A host that disappears can still prevent
final serial/artifact collection. Permanent off-host partial-artifact recovery is
not currently a general guarantee. A job that never booted a guest cannot supply
a guest serial log.

## Images, identity and billing

Remote image submissions currently accept digest-pinned images from the GitLab
Container Registry. Scope API tokens to the project and required operations;
never put credentials in a prompt, issue, repository or workload log. Follow the
[API key guide](../api-keys/) and current private-image submission documentation.

Prepaid credits exist. They do not themselves purchase reserved capacity or an
availability SLA. Never-started jobs have zero compute usage under the existing
metering path. Started infrastructure failures and retries require an explicit
commercial credit policy; do not assume automatic refunds or free retries.

Public availability commitments, reserved pricing, retention and support terms
must be agreed explicitly. A healthy service snapshot or successful deployment
is not proof that a configuration is ready for a paid production workload.


---

# Hardware markers

> Select RealHW capacity with GitLab runner tags.

Source: https://docs.realhw.dev/hardware-markers/
Documentation index: https://docs.realhw.dev/llms.txt

RealHW exposes hardware markers as GitLab runner tags. You already know the interface: put tags on a GitLab CI job.

There are two tag families:

- GitLab-compatible SaaS tags for generic Linux runner shapes
- RealHW hardware markers for specialized hardware

## Start simple

```yaml
unit-tests:
  tags:
    - saas-linux-medium-amd64
  script:
    - cargo test
```

## Add hardware when needed

```yaml
gpu-tests:
  tags:
    - saas-linux-large-amd64
    - gpu:nvidia
    - gpu:count:1
  script:
    - nvidia-smi
```

## Tags versus variables

Tags select eligible runner capacity. `HW_*` variables request an explicit reservation for the job.

For most generic jobs, use only a GitLab-compatible SaaS tag. For specialized jobs, use hardware markers plus variables when the job needs a specific reservable resource.


---

# Advanced reservations

> Use HW variables for explicit resource requests.

Source: https://docs.realhw.dev/hardware-markers/advanced-reservations/
Documentation index: https://docs.realhw.dev/llms.txt

Use `HW_*` variables when tags are not specific enough.

Tags select eligible capacity. Variables request concrete resources for this job.

## Common variables

| Variable | Use |
|---|---|
| `HW_CPU` | Override the CPU count inferred from the SaaS tag. |
| `HW_MEMORY` | Override the memory inferred from the SaaS tag. |
| `HW_GPU_COUNT` | Reserve a specific number of GPUs. |
| `HW_GPU_MODEL` | Request a specific GPU model. |
| `HW_NIC_TIER` | Reserve a NIC tier such as `25g`. |
| `HW_NIC_DRIVER` | Request a NIC driver such as `mlx5_core`. |
| `HW_SRIOV_VF_COUNT` | Reserve SR-IOV virtual functions. |
| `HW_NUMA_POLICY` | Use `any`, `single-node`, or `spread`. |
| `HW_HUGEPAGE_BYTES` | Reserve hugepage bytes. |
| `HW_PREEMPTIBLE` | Allow preemptible capacity for generic jobs. |
| `HW_PLACEMENT` | Use `auto`, `bare-metal`, or `cloud`. |

## Example

```yaml
packet-test:
  tags:
    - saas-linux-large-amd64
    - nic:tier:25g
  variables:
    HW_CPU: "8"
    HW_MEMORY: "16Gi"
    HW_NIC_TIER: "25g"
    HW_SRIOV_VF_COUNT: "1"
    HW_NUMA_POLICY: "single-node"
  script:
    - ./run_packet_test.sh
```


---

# CPU and architecture

> Select CPU architecture and SIMD capabilities.

Source: https://docs.realhw.dev/hardware-markers/cpu-and-architecture/
Documentation index: https://docs.realhw.dev/llms.txt

Use architecture markers when the job must run on a specific CPU architecture.

```yaml
arm64-tests:
  tags:
    - arch:aarch64
  script:
    - uname -m
    - ./test-arm64.sh
```

You can also use GitLab-compatible ARM64 tags:

```yaml
tags:
  - saas-linux-medium-arm64
```

SIMD markers select CPU feature support:

```yaml
simd-tests:
  tags:
    - saas-linux-large-amd64
    - simd:avx512f
  script:
    - ./test-avx512.sh
```


---

# GitLab-compatible tags

> Use familiar GitLab SaaS runner tags on RealHW.

Source: https://docs.realhw.dev/hardware-markers/gitlab-compatible-tags/
Documentation index: https://docs.realhw.dev/llms.txt

RealHW exposes GitLab-compatible SaaS runner tags for convenience. If your job already uses GitLab hosted runner tags, start there.

## AMD64

- `saas-linux-small-amd64`
- `saas-linux-medium-amd64`
- `saas-linux-large-amd64`
- `saas-linux-xlarge-amd64`
- `saas-linux-2xlarge-amd64`

## ARM64

- `saas-linux-small-arm64`
- `saas-linux-medium-arm64`
- `saas-linux-large-arm64`

## Compatibility behavior

Larger hosts can advertise smaller compatible tags. For example, a host that can run `saas-linux-xlarge-amd64` can also advertise `saas-linux-small-amd64`, `saas-linux-medium-amd64`, and `saas-linux-large-amd64`.

Use the smallest tag that fits the job. Add RealHW hardware markers only when the job needs specialized hardware.


---

# GPU markers

> Select GPU vendor, model, and count.

Source: https://docs.realhw.dev/hardware-markers/gpu/
Documentation index: https://docs.realhw.dev/llms.txt

Use GPU markers when a job needs accelerator hardware.

```yaml
gpu-tests:
  tags:
    - saas-linux-large-amd64
    - gpu:nvidia
    - gpu:count:1
  script:
    - nvidia-smi
    - ./test-gpu.sh
```

## Marker forms

- `gpu:<vendor>`
- `gpu:model:<name>`
- `gpu:count:<n>`

Examples:

```yaml
tags:
  - gpu:nvidia
  - gpu:model:a100
  - gpu:count:2
```

The runner exposes selected GPU information to the job environment.


---

# Network markers

> Select NIC driver, speed, tier, and SR-IOV-capable capacity.

Source: https://docs.realhw.dev/hardware-markers/network/
Documentation index: https://docs.realhw.dev/llms.txt

Use network markers when a job needs real NIC hardware.

```yaml
xdp-tests:
  tags:
    - saas-linux-large-amd64
    - nic:tier:25g
    - nic:driver:mlx5_core
  variables:
    HW_NIC_TIER: "25g"
  script:
    - ./run-xdp-tests.sh
```

## Marker forms

- `nic:<driver>`
- `nic:driver:<driver>`
- `nic:speed:<mbps>`
- `nic:tier:1g`
- `nic:tier:10g`
- `nic:tier:25g`
- `nic:tier:40g`
- `nic:tier:100g`

For SR-IOV jobs, pair NIC markers with `HW_SRIOV_VF_COUNT`.


---

# NUMA and hugepages

> Select NUMA topology and reserve hugepages.

Source: https://docs.realhw.dev/hardware-markers/numa-and-hugepages/
Documentation index: https://docs.realhw.dev/llms.txt

Use NUMA markers when a job needs a host with a specific topology.

```yaml
numa-tests:
  tags:
    - saas-linux-large-amd64
    - numa:nodes:2
  variables:
    HW_NUMA_POLICY: "single-node"
  script:
    - ./run-numa-tests.sh
```

Use hugepage variables when the job must reserve hugepages:

```yaml
hugepage-tests:
  tags:
    - saas-linux-large-amd64
  variables:
    HW_HUGEPAGE_BYTES: "1073741824"
    HW_NUMA_POLICY: "single-node"
  script:
    - ./run-hugepage-tests.sh
```


---

# RealHW

> Developer documentation for hardware-aware GitLab runners.

Source: https://docs.realhw.dev/
Documentation index: https://docs.realhw.dev/llms.txt

RealHW offers complete GitLab execution offload: connect your project and keep
your existing pipeline and supported job definitions. GitLab orchestrates the
pipeline; RealHW supplies execution capacity and returns job results, logs and
artifacts. You do not need to rewrite every job into a new API workflow. Keep using familiar GitLab tags for ordinary jobs, then ask for GPUs,
KVM, NICs, SR-IOV, NUMA placement, hugepages, or a specific CPU architecture
when the workload needs them. The web app manages project runners, pipeline
placement, usage, and interactive hardware leases.

```yaml
qemu-tests:
  tags:
    - realhw
    - arch:x86_64
    - kvm
  script:
    - ./run-qemu-tests.sh
  artifacts:
    when: always
    paths:
      - results/
```

The runner admits the request before the job starts, projects only allowlisted
devices, streams logs through the lifecycle, and returns the job result to
GitLab.

## Set up with your agent

Set up the RealHW CLI on this machine using https://docs.realhw.dev/get-started/agent-setup/index.md as the canonical guide. Read it first and follow its pinned checkout installation instructions; the CLI is not published on PyPI and repository access is required. Preserve existing checkouts and credentials. Use an existing environment token without displaying it, or guide me through browser login with read-only permissions. Never ask me to paste credentials into this conversation. Verify the CLI version, supported operations, and current hardware with JSON commands. Report any access or setup blocker clearly. Stop after verification; do not reserve hardware or start workloads.


## Choose your path


  ### [Open the dashboard](https://realhw.dev/dashboard)

    Manage project runners, inspect placement, and review usage in the RealHW app.
  
  ### [Select hardware](https://docs.realhw.dev/hardware-markers/)

    Browse the marker catalog and compose a request from stable capability tags.
  
  ### [Run an image locally](https://docs.realhw.dev/guides/run-image-workload/)

    Use the local runner CLI to launch an existing OCI image and collect artifacts.
  
  ### [Debug placement](https://docs.realhw.dev/guides/debug-runner-selection/)

    Trace tags, capacity, admission, and unsupported requests when a job waits.
  


## Developer reference

| Need | Start here |
| --- | --- |
| GitLab job examples | [CI examples](https://docs.realhw.dev/gitlab/gitlab-ci-examples/) |
| CPU and architecture | [CPU and architecture](https://docs.realhw.dev/hardware-markers/cpu-and-architecture/) |
| GPUs and device counts | [GPU markers](https://docs.realhw.dev/hardware-markers/gpu/) |
| KVM, artifacts, and cloud-init | [Run an image workload](https://docs.realhw.dev/guides/run-image-workload/) |
| Runner selection failures | [Debug runner selection](https://docs.realhw.dev/guides/debug-runner-selection/) |


---

# GitLab tags

> RealHW tag behavior for GitLab CI jobs.

Source: https://docs.realhw.dev/reference/gitlab-tags/
Documentation index: https://docs.realhw.dev/llms.txt

RealHW uses GitLab runner tags as the customer-facing selection API.

## Generic shape tags

- `saas-linux-small-amd64`
- `saas-linux-medium-amd64`
- `saas-linux-large-amd64`
- `saas-linux-xlarge-amd64`
- `saas-linux-2xlarge-amd64`
- `saas-linux-small-arm64`
- `saas-linux-medium-arm64`
- `saas-linux-large-arm64`

## Hardware marker tags

- `arch:<value>`
- `simd:<flag>`
- `gpu:<vendor>`
- `gpu:model:<name>`
- `gpu:count:<n>`
- `nic:<driver>`
- `nic:driver:<driver>`
- `nic:speed:<mbps>`
- `nic:tier:<tier>`
- `numa:nodes:<n>`

Tags describe host capability. They do not guarantee that capacity is idle at the moment the job is queued.


---

# HW variables

> Resource reservation variables for RealHW jobs.

Source: https://docs.realhw.dev/reference/hw-variables/
Documentation index: https://docs.realhw.dev/llms.txt

Use `HW_*` variables for explicit per-job reservations.

| Variable | Example |
|---|---|
| `HW_CPU` | `"8"` |
| `HW_MEMORY` | `"16Gi"` |
| `HW_GPU_COUNT` | `"1"` |
| `HW_GPU_MODEL` | `"a100"` |
| `HW_NIC_TIER` | `"25g"` |
| `HW_NIC_DRIVER` | `"mlx5_core"` |
| `HW_SRIOV_VF_COUNT` | `"1"` |
| `HW_NUMA_POLICY` | `"single-node"` |
| `HW_HUGEPAGE_BYTES` | `"1073741824"` |
| `HW_PREEMPTIBLE` | `"true"` |
| `HW_PLACEMENT` | `"auto"` |

Generic jobs should usually rely on GitLab-compatible SaaS tags. Use variables for hardware-specific requests and explicit overrides.

`HW_PREEMPTIBLE` defaults to `false`. Only an explicit `true` permits spot
placement; job names and runtime estimates never grant interruption permission.
Pre-lease webhook warm-up uses verified on-demand catalog capacity because job
events do not carry the workload’s interruption opt-in.


---

# Job examples

> RealHW GitLab CI job snippets.

Source: https://docs.realhw.dev/reference/job-examples/
Documentation index: https://docs.realhw.dev/llms.txt

## Basic shape

```yaml
unit-tests:
  tags:
    - saas-linux-medium-amd64
  script:
    - cargo test
```

## Hardware shape

```yaml
packet-test:
  tags:
    - saas-linux-large-amd64
    - nic:tier:25g
  variables:
    HW_NIC_TIER: "25g"
    HW_SRIOV_VF_COUNT: "1"
  script:
    - ./run_packet_test.sh
```

## Explicit placement

```yaml
bare-metal-cert:
  tags:
    - saas-linux-large-amd64
    - gpu:nvidia
  variables:
    HW_PLACEMENT: "bare-metal"
    HW_GPU_COUNT: "1"
  script:
    - ./run-certification.sh
```


---

# Marker catalog

> Complete RealHW marker families.

Source: https://docs.realhw.dev/reference/marker-catalog/
Documentation index: https://docs.realhw.dev/llms.txt

| Family | Form | Example |
|---|---|---|
| GitLab-compatible AMD64 | `saas-linux-<size>-amd64` | `saas-linux-large-amd64` |
| GitLab-compatible ARM64 | `saas-linux-<size>-arm64` | `saas-linux-medium-arm64` |
| Architecture | `arch:<value>` | `arch:aarch64` |
| SIMD | `simd:<flag>` | `simd:avx512f` |
| GPU vendor | `gpu:<vendor>` | `gpu:nvidia` |
| GPU model | `gpu:model:<name>` | `gpu:model:a100` |
| GPU count | `gpu:count:<n>` | `gpu:count:1` |
| NIC driver | `nic:<driver>` or `nic:driver:<driver>` | `nic:driver:mlx5_core` |
| NIC speed | `nic:speed:<mbps>` | `nic:speed:25000` |
| NIC tier | `nic:tier:<tier>` | `nic:tier:25g` |
| NUMA nodes | `numa:nodes:<n>` | `numa:nodes:2` |

Marker availability depends on the hardware attached to your RealHW project.


---

# Known limitations

> Current RealHW limitations customers should know.

Source: https://docs.realhw.dev/status/known-limitations/
Documentation index: https://docs.realhw.dev/llms.txt

## GitLab support

RealHW is focused on GitLab project runners. GitHub Actions and Jenkins adapters are not part of the customer path yet.

## Hardware availability

Hardware markers select eligible capacity. They do not guarantee immediate availability. A valid request can wait when matching hardware is busy.

## Workspaces

Interactive workspace wire types exist, but workspace leases are not part of the supported customer path yet.

## Manual runner import

Manual runner import requires an existing GitLab runner authentication token. RealHW can store and use the imported token, but there is no token rotation flow in the API today.

## Production service readiness

Current operation assumes trusted workloads. Public hostile-code isolation,
account-exclusive reserved capacity, true user-process start timings, host-loss
artifact checkpoints and commercial support/credit policies require additional
validation. See [service boundaries](../../guides/service-boundaries/).


---

# Supported capabilities

> Current customer-facing RealHW capabilities.

Source: https://docs.realhw.dev/status/supported-capabilities/
Documentation index: https://docs.realhw.dev/llms.txt

## Supported

- GitLab project runner integration through RealHW runner personalities
- GitLab-compatible SaaS runner tags for generic Linux shapes
- Architecture markers
- SIMD markers
- GPU vendor, model, and count markers
- NIC driver, speed, and tier markers
- NUMA node markers
- CPU and memory reservations
- GPU reservations
- NIC reservations
- SR-IOV VF reservations
- Hugepage reservations
- Placement preference with `HW_PLACEMENT`