Vendor management — SLA tracking, benchmarks on delivery, RMA flow

A GPU cluster operator deals with vendors at three layers: the hardware OEM that built the nodes, the colocation provider that hosts them, and the network gear vendors. Each layer has its own SLA, its own RMA process, its own escalation path. Mishandle any of them and your operations slip from reactive to chaotic — small problems compound into outages because the replacement part took 14 days when the contract said 5.

This page is the operator's view: what to track, what to send, what to expect, and how to be the customer the vendor wants to help (without being a doormat about contractual terms).

What you track

Three SLAs matter day-to-day:

Most contracts are weak on the third — vendors will sign uptime SLAs because they're easy to measure, but balk at performance SLAs because performance depends on the operator's stack as much as the hardware. Push hard for written performance commitments at delivery, even if they're informal "best effort" terms.

Tracking uptime per vendor

Source the data from your own monitoring, not the vendor's:

# Node-up minutes per month, per node
# (Prometheus query, simplified)
sum_over_time(up{job="node-exporter", node=~"gpu-.*"}[30d])

# Convert to %
(sum_over_time(up{node=~"gpu-.*"}[30d]) / (30 * 1440)) * 100

If a node was intentionally down (planned reboot, your maintenance), that doesn't count against the vendor. Tag your downtime windows in the metrics:

# subtract scheduled-maintenance windows
up{node=~"gpu-.*"} unless on() planned_maintenance_window == 1

Monthly report: each node, hours up, hours down (split: vendor-attributable vs your-side), uptime %. Anything below contracted SLA, escalate.

Tracking time-to-replace

Two timestamps:

1. RMA submitted (your ticket creation time at the vendor portal).
2. Part on-site and node back to operational state.

The contract may carve up "delivery" vs "installation" vs "validation" — read carefully. A 4-hour NBD contract that says "part delivered to loading dock" doesn't help you if it then takes 2 days to get from loading dock to rack.

Track per-RMA:

RMA-2026-04-12-001
  Component:  H100 GPU (gpu-04 GPU index 3)
  Failure:    XID 79 + ECC double-bit
  Submitted:  2026-04-12 09:00 UTC
  Vendor ack: 2026-04-12 09:42 UTC
  Part on-site: 2026-04-12 21:30 UTC  (T+12h30m)
  Part installed: 2026-04-13 14:00 UTC  (T+29h)
  Validated:  2026-04-13 16:00 UTC  (T+31h, post-NCCL re-benchmark)
  Contracted: NBD (next business day)  -> met

Aggregate: median, p95, max time-to-replace per quarter. Vendors with rising p95s are the early signal of supply-chain issues; raise it in QBR before it becomes a fleet-wide problem.

Tracking benchmarks on delivery

Every new node passes a network validation runbook before it sees a tenant. The output is the baseline. Save it.

gpu-25  delivered 2026-04-30
  pre-tenant validation:
    nccl all_reduce 8GB busbw: 195.4 GB/s    [tier expectation: 190+ GB/s, PASS]
    ib_write_bw single port: 392 Gb/s        [400G port, PASS]
    nvidia-smi p2p r same-baseboard: OK     [PASS]
  signed off by:  ops-team-foo

Store these in a lightweight database. They are evidence. When the same node six months later is delivering 160 GB/s busbw, you have a baseline to compare against — and a vendor case for "this is degraded since delivery, not how we accepted it".

Periodic re-test

Quarterly, re-benchmark a sample (not the whole fleet — too expensive in tenant downtime). Pick 5-10% of nodes, drain them, re-run NCCL all_reduce, compare to delivery baseline.

# A scheduled drain window, the operator runs:
kubectl cordon gpu-25
kubectl drain gpu-25 --ignore-daemonsets --grace-period=600

# SSH or kubectl debug to run the standard 2-node NCCL test
# Compare busbw to delivery baseline; allow a 5% drift, escalate at >10%

Drift above ~10% on bandwidth without an obvious cause (firmware change, kernel upgrade) means something has degraded. Common causes: optical module starting to fail, switch port marginal, cable seating problem. Often vendor-side, sometimes operator-side. Investigation feeds the RDMA debugging flow.

Communicating root cause to vendors

When a problem is the vendor's, you have two communication tasks: open the ticket with enough information that they can triage in one round-trip, and follow up with factual prose that doesn't accidentally sound like marketing.

The ticket: what to include

Subject: <node ID>: <symptom> — request for investigation

Cluster: <slug>
Node: gpu-04
Component (if known): GPU index 3 (UUID: ...)
Symptom (one line): NCCL hangs after ~3 minutes on multi-node training; XID 79 in dmesg

Timeline:
  T-3 days: node was healthy, validated busbw 195 GB/s on standard test
  T-1 day: tenant began running training; first failure observed
  T-0:    isolated to gpu-04 GPU index 3 via [link to triage notes]

Evidence:
  dmesg snippet: <attached: dmesg-gpu-04-2026-05-04.log>
  nvidia-smi -q output: <attached>
  XID list: 79 (3 events, all on GPU 3) and 109 (1 event)
  NCCL test result: 8GB all_reduce, 60 GB/s (vs 195 GB/s baseline)
  Hardware info: <attached: dmidecode + lspci -vvv>

Actions taken:
  Ran [/docs/operations/triage-decision-tree] to confirm hardware vs software origin
  Ruled out driver mismatch (driver pinned, no recent change)
  Replaced cable on suspect link; symptom unchanged
  Cordoned the node; tenant moved off

Request: please confirm whether this is a GPU failure or backplane / NVLink fault.
We are pursuing replacement under contract <contract-id>.

What this gives the vendor: the symptom, the evidence, and the question. They can dispatch a tech with the right part on the first trip.

What you don't include: speculation. "I think it might be the NVLink switch" is fine in a phone conversation, not in the ticket. Stick to what the logs show.

The follow-up: factual prose, not marketing

When you write up the resolution (for your own records, for the vendor's case file, for a customer-facing comms about the affected window), it should read like an engineer wrote it.

Bad:

Working closely with our trusted vendor partners, we identified and resolved a complex hardware issue affecting a subset of our cutting-edge GPU infrastructure. We thank our customers for their patience and continue to deliver world-class performance.

Good:

A GPU on gpu-04 (index 3) reported XID 79 (NVLink fatal error) and XID 109 (uncorrectable ECC) over a 26-hour window. The GPU was replaced under RMA on 2026-04-13. Post-replacement validation showed busbw restored to 198 GB/s (baseline: 195 GB/s).

The good version contains exactly the facts. No adjectives. No "world-class". No "trusted partners". This style is for vendor communication and for customer comms — see incident response for the customer-facing comms templates that follow the same discipline.

This is the style customers and vendors trust. They can see you understand the problem and they can see what was done.

What to ask in QBRs

Quarterly business reviews with vendors are when the relationship gets renegotiated. Bring data, ask specific questions:

• Per-RMA: time-to-on-site distribution (median, p95, worst). Are we slipping?
• Failure rates: how many GPUs failed this quarter, by failure mode? Compare to expected MTBF.
• Spares pool: where are spares stocked? How many of each component? When does the pool refresh?
• Firmware: what's the recommended firmware roadmap; what's compatibility-tested with our driver pin?
• Roadmap: when does next-gen hardware enter our procurement window? Lead time?

A QBR without specific data is the vendor presenting to you. A QBR with data is a working session about gaps.

Capacity contracts: committed vs on-demand

Two procurement modes for hardware and for cloud-equivalent capacity:

Committed capacity — you commit to a multi-year purchase or a multi-month reservation. Lower per-unit price; you eat the cost regardless of utilization.

On-demand — you pay for what you use. Higher per-unit price; can scale down to zero.

For physical hardware purchases, "committed" is the default — you bought the box. For colo space and power, you usually have a mix of committed (the rack contract) and overage (extra power draw billed monthly). For cloud burst capacity, you pick the model.

Operator levers:

• Stagger commitments. Don't sign a 3-year power contract the same week as a 3-year hardware lease. If demand drops, both come due simultaneously.
• Negotiate flex clauses. Some contracts allow you to swap committed capacity for newer-gen at refresh time. Worth more than a 5% discount.
• Track committed-vs-used monthly. Same shape as the capacity planning report, but viewed through the procurement lens. If committed power is 90% utilized but the rack is at 65% physical occupancy, you have a power-density problem before you have a capacity problem.

Invoicing reconciliation

Vendor invoices have errors. Not always; not even mostly. But often enough that monthly reconciliation pays for itself.

The categories of error you will see:

• Stale spares pool count. Vendor still billing for 8 spares; the pool has 6 because two were consumed in RMA last month and not replenished.
• Wrong RMA crediting. A node was down for 11 days under RMA; the contract says vendor credits the customer for downtime over 24h; the credit doesn't appear.
• Power overage miscounted. Colo bill includes "overage power" but the PDU readings don't agree.
• Bandwidth overage. Network bill claims you exceeded 95th-percentile threshold; your own NetFlow says you didn't.
• Tax category errors. Equipment vs services taxed differently; vendor mis-classifies; you overpay sales tax.

Don't blindly approve. Even a 5-minute scan against your own utilization records catches most of these. For colo and network bills, demand the underlying meter data, not just a summary.

The reconciliation cadence: monthly, by an operator who knows the cluster (not pure finance). The operator-finance handoff is critical — finance sees totals, the operator sees patterns.

RMA flow

The RMA process should be muscle memory by the time you need it for the third time. Document the steps, train the on-call.

RMA: dead GPU

Trigger: XID 79, XID 95, XID 109 — the fatal-error class. Or DCGM diagnostic failure on individual subsystem.

# 1. Confirm the GPU is the failing component, not a downstream symptom
#    (See triage decision tree)
nvidia-smi -q -i 3 | grep -E 'XID|ECC|Persistence'
dmesg -T | grep -i 'NVRM' | tail -50

# 2. Cordon the node, drain tenant workloads
kubectl cordon gpu-04
kubectl drain gpu-04 --ignore-daemonsets --delete-emptydir-data --force

# 3. Capture evidence for the RMA package
sudo nvidia-smi -q > /tmp/rma-gpu-04-nvsmi-$(date +%F).log
sudo dmesg -T > /tmp/rma-gpu-04-dmesg-$(date +%F).log
sudo journalctl --since "24 hours ago" > /tmp/rma-gpu-04-journal-$(date +%F).log

# 4. Submit RMA with the package attached, note start time

# 5. Wait. Don't poke.

When the part arrives:

# 6. Coordinate with the colo for physical access (if it's a hands-on swap)
# 7. After swap: power on, validate
nvidia-smi
nvidia-smi --query-gpu=name,uuid,driver_version,vbios_version --format=csv

# 8. Re-run [/docs/operations/network-validation-by-speed]
# 9. Compare to baseline (delivery benchmark)
# 10. Uncordon, return to pool
kubectl uncordon gpu-04

If post-replacement bandwidth is below baseline by > 5%, do not put the node back in the tenant pool yet. Investigate further — sometimes the replacement is itself defective, or the tech reseated something else and broke a different thing.

RMA: dead HCA / NIC

Trigger: link won't come up, port flapping, or ibstat shows Down for an extended period; PCIe link state degraded.

# 1. Confirm it's the HCA, not the cable or switch port
ibstat | grep -A3 'Port 1'    # State, Rate, LinkLayer
sudo ethtool <iface>          # link state at IP layer
sudo lspci -vvv | grep -A20 ConnectX | grep LnkSta

# 2. Eliminate cable: ask the colo to replace cable, or swap to a known-good port
# 3. Eliminate switch: try a different switch port
# 4. If still down — RMA the HCA

# 5. Capture evidence
sudo flint -d /dev/mst/mt4129_pciconf0 q > /tmp/rma-hca-flint-$(date +%F).log
sudo mlxlink -d <pciid> --show_eye --show_module > /tmp/rma-hca-mlxlink-$(date +%F).log
dmesg -T | grep -i mlx5 > /tmp/rma-hca-dmesg-$(date +%F).log

The mlxlink output (eye diagram, module SFP/QSFP info) is what the vendor uses to confirm the HCA itself is bad vs marginal optics. Always capture before submitting RMA.

Replacement on an HCA includes refirmware-to-current step:

# Post-install: flash to the contracted firmware version
sudo flint -d /dev/mst/mt4129_pciconf0 -i fw-ConnectX7-rel-28_39_1002.bin burn
# Reboot or reload mlx5_core
sudo modprobe -r mlx5_core; sudo modprobe mlx5_core
ibstat | head -30

Then re-run validation: ib_write_bw to a peer, then full NCCL re-test.

RMA: dead switch port

Trigger: a single port on a leaf switch is dead, link won't come up, or flapping. Often discovered when one node's HCA is fine on a different port.

This is a colo or network-vendor problem, not the GPU vendor. Different ticket queue, different SLA. Open with:

• The switch model and port number.
• Evidence the port is dead (your end's HCA is good, swap to alt port works).
• Optic info (SFP/QSFP model on both ends).

Switch ports usually get reactivated by a remote-hands operation (replace the optic in the switch end, or the cable). Sometimes the port itself is dead and the switch needs a board swap, which is more disruptive.

If the affected port hosts a node that's part of an active tenant: communicate. The node is offline; the tenant has lost N GPUs from their reservation; ETA depends on the colo's response.

RMA: PSU, fan, RAM, drive

Less interesting from an operator-cluster standpoint — these are commodity hardware-vendor RMAs. Same template:

• Capture evidence (IPMI logs, dmesg, smartctl for drives).
• Open the RMA.
• Coordinate physical replacement.
• Validate (boot, no errors, performance unchanged).

Drives in the GPU node typically don't hold tenant data (that's on Weka), so a failed drive is operationally minor. Replace at the next maintenance window unless it triggers an alert about boot-disk failure.

When the vendor isn't responding

Sometimes vendors slip. The escalation path:

1. Day 1-2 over SLA: polite ping in the ticket. Reference the contract clause. Ask for ETA.
2. Day 3-4: phone the account manager. Reference the ticket and the contract.
3. Day 5+: escalate via your VP / CTO to their VP. The first time you do this it's awkward; if you don't, the vendor learns you won't escalate.

Document each escalation in the operator log. Reference these in QBRs. A vendor with a recurring escalation pattern needs renegotiation or replacement.

Spares pool management

You don't want to be RMAing your only failed GPU during a customer-facing incident. Maintain a spares pool — vendor-managed (they ship to your colo) or your-managed (you bought hot spares).

Reasonable defaults for a 1024-GPU cluster:

• 8-16 GPU spares (1-2% of fleet)
• 4-8 HCA spares
• 4 PSU spares per chassis-class
• 2-4 NVMe drives per chassis-class
• 0-2 server-board spares (rare failure, long lead time)
• Cable + optic spares: ~5% of in-use count

Track spares-in-pool monthly. Replenish proactively when below 50% of nominal.

Common operator mistakes

Letting the vendor lead the diagnosis. If you submit a ticket without evidence, the vendor's first reply is "please collect logs and reseat" — costing you a day. Do the diagnosis first, then submit.

Accepting "no problem found" without re-validation. A node returned from RMA "no problem found" but with the same symptom on the next test run is still broken. Don't put it back into a tenant pool until you've reproduced or the vendor explains the no-problem-found credibly.

Not capturing the delivery baseline. Six months in, you can't tell if a node has degraded — you have nothing to compare against. Always benchmark on delivery; always store the result.

Treating the colo and the OEM as one vendor. They aren't. Different SLAs, different ticket queues, different escalation paths. Know which problem belongs to which.

Letting capacity contracts auto-renew. Re-negotiate. Hardware prices drop. Power contracts shift. Auto-renewing a 3-year-old contract at original rates is leaving money on the table.

Marketing-speak in postmortems. Don't write "our trusted vendor partner". Write "the GPU was replaced under RMA on 2026-04-13". Customers, auditors, and vendors all prefer the latter.

Skipping QBRs. "Things are fine, why meet?" Because the data trends matter and they only show up in aggregate. Skipping a QBR means missing the early signal of supply-chain pressure or rising failure rates.

Worked example: a real RMA cycle

The following is what an actual RMA looks like end-to-end, sanitized but not fictional in shape. It shows the rhythm — how long things take, what the comms look like, when problems escalate.

Day 0 — symptom

2026-04-12 09:00 UTC

Customer ticket:  "Training job on tenant-bar fails after 3 minutes"
                  Job ID: 12345 (their accounting)
                  Pattern: NCCL hang at all-reduce, no progress
                  Ran fine yesterday on same configuration

Operator triage:
  squeue:    job 12345 was on gpu-04, gpu-05, gpu-06, gpu-07
  dmesg gpu-04: NVRM: Xid (PCI:0000:18:00): 79, pid=27361, GPU has fallen off the bus
  dmesg gpu-05: clean
  dmesg gpu-06: clean
  dmesg gpu-07: clean

Conclusion:  gpu-04 hardware fault. Isolated.

Day 0 — containment

09:18:  kubectl cordon gpu-04
09:18:  kubectl drain gpu-04 --ignore-daemonsets --delete-emptydir-data
09:35:  scancel running tenant-bar jobs that included gpu-04
        (tenant notified, jobs requeued onto gpu-08 within minutes)
09:42:  customer ack: training restarted, completing on schedule

Tenant impact: 1 job lost, ~40 minutes of training time
              (their checkpoint cadence was 30 minutes — recovered cleanly)

Day 0 — RMA submitted

10:00:  Evidence package assembled
        - dmesg full output (last 24h)
        - nvidia-smi -q -i 3 (target GPU)
        - dcgmi diag -r 3 results (FAILED on PCI subsystem)
        - lspci -vvv (link state, device IDs)
        - 7-day metrics graph (utilization, power, thermal)

10:18:  RMA-2026-0412-001 submitted to OEM-foo portal
        Severity: P1 (production node down)
        Contract reference: HW-2024-0815, NBD coverage
        Evidence: package attached

10:42:  Vendor ack received
        "Engineering reviewing; will dispatch replacement; on-site contact pending"

11:30:  Vendor confirms part shipped from regional spares depot
        ETA: 2026-04-12 21:00 UTC (T+11h)
        Replacement: same H100 80GB SXM5, manufacturing batch <newer>

13:45:  Operator updates customer:
        "The affected node is in RMA. ETA on replacement: ~T+11h.
         Your subsequent training runs are routing around. We'll re-validate
         the node before returning it to your reservation."

Day 0 — receipt

21:18:  Vendor part arrives at colo loading dock
21:42:  Colo remote-hands receives, escorts to rack
22:30:  Replacement GPU installed (board-level swap)
22:45:  Node powered on, BIOS clean, OS boots
22:55:  nvidia-smi shows 8 GPUs, no XID, ECC clean
23:10:  validation runbook started (network-validation-by-speed)

Day 1 — validation and re-acceptance

00:30:  ib_write_bw single-link: 392 Gb/s   [PASS, baseline 393]
00:45:  nccl all_reduce 8GB busbw: 197.4 GB/s  [PASS, baseline 195.4 — slightly higher!]
01:00:  GPU 3 individual stress (dcgmi diag -r 4): clean
01:15:  Node release decision: APPROVED
01:18:  kubectl uncordon gpu-04

Total downtime: T+0 to T+16h18m (16 hours, 18 minutes)
Contracted SLA: NBD (next business day)
Result: WITHIN SLA

Day 1 — postmortem to vendor

04:00 UTC summary email to vendor account manager:

  RMA-2026-0412-001 closed.
  Cause: GPU index 3 of gpu-04 (s/n: ...) reported XID 79 plus uncorrectable
  ECC over a 26-hour window. Replacement received at T+11h, installed at
  T+13h, validated at T+16h. Post-replacement bandwidth: 197.4 GB/s
  (baseline 195.4). No further issues.

  Two follow-ups:
    1. Failure analysis on the returned GPU — please share when complete.
    2. We saw three similar XID 79 events in the last 6 months across the fleet.
       MTBF in our deployment is trending shorter than the spec sheet.
       Want to schedule a call to review.

The follow-up is the substance. One incident is noise. Three is a pattern. The vendor either has an answer or they don't, and either response is data.

Quarter-end QBR

2026-Q2 QBR with OEM-foo:

Items raised:
  - 4 GPU RMAs completed, all within SLA (3 NBD-met, 1 same-day)
  - Average time-to-on-site: 11h (median); 14h (p95)
  - 3 of 4 RMAs were same XID 79 pattern. Pattern recognition.

Vendor response:
  - Acknowledged pattern; engineering team has it open
  - Suspected firmware issue; recommended FW upgrade to <version>
  - Compatibility verified against our pinned driver
  - Schedule a fleet-wide rolling FW upgrade in maintenance windows

Operator action items:
  [ ] Test FW upgrade on 2 nodes (validate in /docs/operations/network-validation-by-speed)
  [ ] Plan rolling upgrade across 96 nodes over 4 weeks (4 nodes / week)
  [ ] Re-test failure rate post-upgrade

The point: this rhythm only works if you're tracking the data. A QBR with no specific RMA-frequency stats is the vendor presenting their roadmap; a QBR with stats is a working session about your fleet.

Operating with multiple vendors at once

Most clusters have at minimum: an OEM (the boxes), a colo (rack, power, cooling), a network-gear vendor (switches, optics), a storage vendor (Weka, DDN), and one or more software vendors (NVIDIA AI Enterprise, Slurm support, etc.). Each has its own SLA and ticket queue.

The operational risk: a problem that touches multiple vendor boundaries. Examples:

• GPU-to-switch link is flapping. Could be HCA (OEM RMA), cable (colo / network vendor), switch port (network vendor), or optic (network vendor or your own). Each vendor will deny first, fix second.
• NCCL bandwidth degraded across the rack. GPU? Driver? Switch? Operator-side OFED upgrade? Each vendor has plausible deniability.
• Filesystem write bursts cause network congestion. Storage vendor says "our writes look normal"; network vendor says "yes those writes saturate this port"; nobody's wrong, but the customer sees slow training.

Operating principle: drive triage centrally. Don't open a ticket with each vendor and see which fixes it first. Diagnose internally to a specific component, then open the ticket with the right vendor with the right evidence. See triage decision tree.

The exception: when triage is genuinely ambiguous and each test you can run takes 4 hours, parallel-open is OK — but communicate to each vendor that the others are also engaged ("this is also being investigated as a potential X with Y vendor"). They'll coordinate or at least not blame each other in writing.

Spares pool: managing through wear-out

The spares pool is not static. Components fail; you replace them. The pool refills (vendor ships from a regional depot, or you buy new). Track pool levels weekly:

SPARES POOL — 2026-04 WEEK 4
  GPU H100 80GB:           pool 12 / nominal 16   (4 RMAs used, 0 received)
  HCA ConnectX-7 400G:     pool 6  / nominal 8    (2 used, 1 in transit)
  PSU (server-class):      pool 8  / nominal 8    (no failures)
  NVMe boot drive 1TB:     pool 4  / nominal 4
  Active optical 400G:     pool 14 / nominal 20   (failures: 6)
  IB cable 5m:             pool 10 / nominal 10

ALERT: optical 400G below replenishment threshold (70% nominal)
       order placed 2026-04-26, ETA 2026-05-08

Operator triggers:

• Pool < 50% nominal — order replenishment immediately.
• Failure rate trending up — schedule conversation with vendor about batch defect.
• Failures concentrated on specific component generation — flag for re-evaluation.

The trap: a vendor-managed spares pool that nobody has visibility into. If your contract has the vendor stocking spares at their site, you have less visibility into pool health and you're at their mercy on response time. Self-stock the highest-failure-rate components even if the contract covers them; you'll hit a faster RMA cycle.

Procurement vocabulary the operator should own

Even if procurement is a separate team, the operator should be able to read the contract.

• MTBF (Mean Time Between Failures) — vendor-claimed; specific to the component class. Compare against your observed rate; report deltas in QBR.
• NBD (Next Business Day) — contractual SLA term. Defines when the part is on-site, not when it's installed. Read carefully.
• DOA (Dead on Arrival) — failure within first hours of operation. Different RMA path; usually instant replacement.
• Replacement-in-kind — vendor commits to replacing with same model. Important if your stack is firmware-pinned to that exact spec.
• Field upgrade — vendor swaps a sub-component without removing the chassis. Less disruptive than full unit swap.
• EOL / EOSL — End of Life / End of Service Life. After EOSL the vendor stops supporting that hardware; you're on your own. Track and plan replacements.
• Liquidated damages — contractual financial penalty when SLA breached. Usually capped; usually not commensurate with actual operational cost. Negotiate at signing.
• Force majeure — clause excusing performance under extraordinary events. Vendors love it; ensure scope is bounded.

Read your contracts. The operator who knows their contract escalates with confidence.

Validation: are you ready for the next RMA?

Once a quarter, run a tabletop exercise: pick a node, simulate "GPU 3 just reported XID 79". Walk the on-call through the runbook. If they can't:

• Find the RMA template in 60 seconds — fix the docs.
• Identify the contract that covers it in 60 seconds — fix the contract registry.
• State the contracted SLA from memory — train the rota.

The first time you do this, it goes badly. That's the point.

See also

• Network validation by speed — the benchmarks you run on delivery and re-test
• Incident response — customer-facing comms during vendor-side incidents
• Triage decision tree — narrowing failures to specific components before RMA
• RDMA debugging — fabric-side diagnosis preceding HCA RMA
• Driver / firmware mismatch — eliminate driver causes before blaming hardware
• DCGM — dcgmi diag for pre-RMA evidence
• Capacity planning — committed-vs-on-demand procurement
• Cost optimization — datacenter PUE and procurement-side levers

External:

• NVIDIA RMA portal documentation
• DDN / Weka SLA reference templates
• NIST 800-88 (data sanitization) — relevant when returning failed drives