# Troubleshooting

> Diagnose runner selection, disk capacity, and KVM execution failures.

Source: https://docs.realhw.dev/gitlab/troubleshooting/
Documentation index: https://docs.realhw.dev/llms.txt

## Job is stuck

Check that the job tags match tags exposed by the RealHW project runner.

```yaml
tags:
  - saas-linux-medium-amd64
```

If the job has a hardware marker, confirm your project has access to that hardware class.

## Job runs on the wrong shape

Use one GitLab-compatible SaaS tag for the generic shape:

```yaml
tags:
  - saas-linux-large-amd64
```

Avoid mixing multiple generic size tags unless you intentionally allow any of those sizes.

## Hardware job waits

Hardware markers describe host capability, not immediate availability. A valid GPU, NIC, or SR-IOV request can wait if matching hardware is busy.

## Job fails before container start

The request may be unsatisfiable or blocked by policy. Check unsupported markers, unavailable project entitlements, excessive `HW_CPU` or `HW_MEMORY`, and unsupported `HW_*` variables.

## Disk capability probe fails

Compare the reported free space with the job's floor on the build filesystem.
A script that prints “infra failure” and exits still produces GitLab
`script_failure`; the message alone does not change the failure category.
RealHW disk rejection before execution uses `runner_system_failure`.

Ask the fleet operator to inspect disk capacity, drain the affected host, and
reclaim unused managed caches or expand capacity before retrying. RealHW
admission reserves a default 10 GiB floor plus 4 GiB of growth headroom per
active job. A workload can require more; this check is not a disk quota.

## KVM is missing or unusable

Request the `kvm` capability in addition to the desired size or architecture.
A SaaS-compatible size tag describes a shape and does not guarantee KVM.
The host must provide usable KVM, and the executor must expose `/dev/kvm`
to the job container. Check both layers when the probe fails.

RealHW supports owned KVM hosts and a dedicated GCE N2 nested-KVM pool.
The GCE worker creates a KVM VM during bootstrap before enrolling, but the
job must still verify that its container can use the device.

## QEMU starts but the guest times out

Once disk and KVM probes pass, inspect the serial log for the expected boot
markers and the first panic. A QEMU timeout is not evidence of the original
disk failure. Preserve the job revision and artifacts so kernel failures can
be investigated separately from runner-capacity problems.