Summary
✨ AI‑Generated
A reliability engineering role focused on operating and improving advanced AI compute infrastructure. Responsibilities include maintaining uptime, handling complex hardware and network failures, and developing recovery strategies for critical systems.
Highlights
Work on mission-critical AI computing environments, ensuring reliability and resilience of advanced infrastructure at large scale.
Description
The AI Systems Support Engineer is responsible for the reliability, availability, and resilience of DAMAC AI's GPU cluster infrastructure — ensuring Blackwell-based DGX SuperPOD and HGX systems achieve and exceed defined SLA and uptime commitments.
This role combines traditional site reliability engineering with deep expertise in GPU infrastructure failure modes, disaster recovery, business continuity, and the operational resilience of mission-critical AI compute environments.
Experience Requirements
7+ years of experience in Site Reliability Engineering, infrastructure operations, or HPC systems administration in data center or cloud environments
Deep understanding of GPU infrastructure failure modes — GPU hardware faults, NVLink errors, InfiniBand fabric partitions, and storage path failures
Proven experience designing and executing Disaster Recovery plans and Business Continuity exercises for mission-critical infrastructure
Strong experience with SLO/SLI frameworks, error budget management, and reliability engineering practices
Hands-on experience with observability platforms: Prometheus, Grafana, NVIDIA DCGM, ELK/Loki, PagerDuty/OpsGenie
Experience leading P1/P2 incident response and facilitating blameless postmortems in high-availability environments
Proficiency in Python, Bash, or Go for automation, self-healing tooling, and observability integrations
Familiarity with NVIDIA DGX SuperPOD architecture, NVLink/NVSwitch fabric design, and NVIDIA DCGM health monitoring
Incident Management & Root Cause Analysis
Lead P1 and P2 incident response for GPU infrastructure events — detection, triage, escalation, containment, recovery, and post-incident review
Drive root cause analysis (RCA) and blameless post-mortem process for all P1 incidents — ensuring findings are documented and remediation actions tracked to closure
Maintain MTTR targets: ≤2 hours for P1 GPU infrastructure events, ≤4 hours for P2 events
Build and maintain on-call runbooks, escalation playbooks, and incident response procedures for AI infrastructure teams
Operate the incident management workflow in ServiceNow or PagerDuty — ensuring all incidents are tracked, communicated, and closed with full documentation
Establish a weekly reliability review meeting to assess open incidents, SLO burn, and upcoming maintenance risk
