Software Engineer, Hardware Health

Skills

About the Role

You will build critical infrastructure that keeps OpenAI's largest compute clusters healthy. You will define health signals, build health checks, investigate hardware failures, own node lifecycle workflows and build automation for global cluster management.

Requirements

• 7+ years in software or infrastructure engineering
• Strong Python and shell scripting
• Experience building large-scale distributed systems or infrastructure platforms
• Comfort with SQL, PromQL or similar tooling
• Strong systems debugging skills

Responsibilities

• Define and maintain health signals across GPUs, CPUs, networking and platform infrastructure
• Build health checks that detect, remediate and verify failures at scale
• Ensure health checks run with minimal latency
• Investigate hardware and system-level failures
• Own node lifecycle workflows including drain, quarantine, repair, RMA and return-to-service
• Build automation and tooling for global cluster management
• Partner with workload, reliability and provider teams

Benefits

• Equity
• Performance-related bonus
• Medical, dental, and vision insurance
• Health Savings Account employer contributions
• Health FSA, Dependent Care FSA, commuter pre-tax accounts
• 401(k) with employer match
• Paid parental leave
• Paid medical and caregiver leave
• Flexible PTO
• 13+ paid company holidays
• Mental health and wellness support
• Employer-paid life and disability coverage
• Learning and development stipend
• Daily meals in offices
• Relocation support