Site Reliability Engineer Post Training

Skills

About the Role

You will own the reliability, performance, and uptime of large-scale post-training and reinforcement-learning jobs. You will support active model runs, debug full-stack failures, build monitoring and recovery automation, improve checkpointing and scheduling, reduce operational toil, join on-call rotations, and write postmortems.

Requirements

• 4+ years of experience as a production engineer, site reliability engineer, or infrastructure engineer operating large-scale distributed systems in production
• Experience debugging complex distributed-system failures
• Software engineering skills in Python, Go, or C++
• Knowledge of Linux systems internals and networking fundamentals
• Experience owning production systems and participating in on-call rotations

Responsibilities

• Own reliability, performance, and uptime for large-scale post-training and reinforcement-learning jobs
• Partner with research teams during active model runs
• Debug failures across accelerators, networking, storage, schedulers, and training frameworks
• Build monitoring, alerting, and automated recovery
• Improve checkpointing, fault tolerance, and job scheduling
• Build internal tools that reduce toil and improve cluster utilization
• Participate in an on-call rotation
• Write postmortems and implement permanent fixes for recurring failures

Benefits

• Visa sponsorship
• Health benefits
• Dental benefits
• Vision benefits
• Unlimited PTO
• Paid parental leave
• Relocation support