Training Performance Engineer

Skills

About the Role

You will drive efficiency improvements across the distributed training stack by profiling training runs, finding bottlenecks in compute, communication and storage, and optimizing GPU utilization. You will implement model graph transforms, build monitoring tools, and partner with researchers. The role is hybrid in San Francisco, three days per week.

Requirements

• Strong programming skills in Python and C++
• Experience running distributed training jobs on multi-GPU systems or HPC clusters
• Experience debugging complex distributed systems
• Exposure to PyTorch, JAX or TensorFlow
• Ability to translate profiling data into engineering improvements

Responsibilities

• Profile end-to-end training runs to identify bottlenecks across compute, communication and storage
• Optimize GPU utilization and throughput for distributed training
• Improve kernel efficiency, scheduling and collective communication performance
• Implement model graph transforms
• Build tooling to monitor and visualize MFU, throughput and uptime
• Partner with researchers to scale new model architectures
• Contribute to infrastructure decisions for reliability and efficiency

Benefits

• Equity
• Medical, dental, and vision insurance with employer HSA contributions
• Pre-tax Health FSA, Dependent Care FSA and commuter accounts
• 401(k) with employer match
• Paid parental leave up to 24 weeks for birth parents and 20 weeks for non-birthing parents
• Paid medical and caregiver leave up to 8 weeks
• Flexible PTO for exempt employees
• 13+ paid company holidays and coordinated office closures
• Paid sick or safe time
• Mental health and wellness support
• Employer-paid basic life and disability coverage
• Annual learning and development stipend
• Daily meals in offices and meal delivery credits
• Relocation support
• Charitable donation matching and wellness stipends
• Performance-related bonus