# GPU Reliability Lab

An open tool and case library for testing the evidence behind GPU reliability decisions.

## The larger direction

GPU engineers and incident-response agents need to distinguish a hardware failure from incomplete telemetry, a changed workload, or a measurement error. The goal is a shared set of inspectable checks and replayable cases that makes those distinctions testable.

Ezgi's contribution is field knowledge, controlled hardware experiments, and upstream fixes. The first public evidence is her H200 CUDA launch-count validation and GB300 measurements. Each future case must carry a concrete source and state what was actually observed.

## Working today

- Two deterministic browser checks: CUDA counter coverage and sampled GPU-power integrity.
- Native Prometheus and NVIDIA CSV input, processed locally.
- Five examples with explicit provenance: one measured capture, two reconstructions of one issue, and two synthetic fixtures.
- Downloadable rule reports, exported assumptions, and an open case manifest.
- The smaller GB300 comparison page as a linked article companion.

## Next: checks engineers can use on their own incidents

Add adapters in response to real evidence, starting with NCCL rank/configuration completeness and DCGM/Xid context. Prefer consuming existing NVIDIA diagnostics over rebuilding them. Unknown signatures and absent ranks must remain unknown, not become guessed causes.

For each adapter, require supported-format fixtures, failure cases, a clear boundary between observed facts and hypotheses, and an independently checkable expected result. A contributor must be able to reproduce the check without contacting production.

## Then: a credible GPU incident evaluation set

Collect independently reviewed, publicly shareable cases covering observability gaps, correctness failures, degraded fabric, clock/power constraints, and recovery verification. Keep synthetic examples separate from reproduced hardware cases. Group variants from the same incident so they cannot inflate the number of independent cases.

Before publishing agent scores, define a versioned rubric, held-out incidents, evidence attribution, unsafe-action penalties, and inter-reviewer agreement. Evaluate diagnosis and next-check quality separately from whether a workload recovered. Public seed cases are for development, not a valid held-out ranking.

## Acceptance before calling this useful or better

Three independent GPU practitioners should bring their own artifacts and complete a concrete diagnostic task. Record whether the tool found a real evidence problem, reduced time to a verified conclusion, or produced a misleading result. Compare those same tasks with their existing tools. Do not substitute likes or a polished page for this evidence.

No external outreach or artifact collection has been performed yet.

## Contribution contract

Each case needs a source URL or a licensed reproduction, a capture/reconstruction/synthetic label, relevant versions and units, expected outcomes with reasoning, unsupported conclusions, and a replay test. Keep original values intact. Corrections must preserve the previous result and explain what changed.

No customer logs or private incident records are required for the initial public case set.