Senior ML Systems Engineer Frameworks and Tooling

Skills

About the Role

You will build and own systems for large-scale language-model training. You will design distributed-training abstractions, improve throughput and stability on multi-node clusters, create monitoring and debugging tooling, investigate bottlenecks, and deliver reliable and reproducible training runs.

Requirements

• Engineering experience with large-scale distributed training or HPC systems
• Familiarity with JAX internals, distributed-training libraries, or custom kernels and fused operations
• Experience with Slurm, Ray, Kubernetes, or similar cluster orchestration
• Ability to debug CUDA, NCCL, networking, IO, and data-pipeline performance
• Experience with Docker or Singularity/Apptainer
• Experience building tools for ML teams
• Strong collaboration skills

Responsibilities

• Build and own the training framework for large-scale language-model training
• Design distributed-training abstractions and checkpointing systems
• Improve training throughput and stability on multi-node clusters
• Develop monitoring, logging, debugging, and developer tooling
• Work with infrastructure teams on clusters, containers, and hardware configurations
• Investigate and resolve ML-systems performance bottlenecks
• Build reproducible and debuggable large-scale training systems

Benefits

• Equity
• Weekly lunch stipend of $75/£75 or equivalent local currency
• Health and dental benefits
• Mental health budget
• RRSP matching, 401K, and pension scheme
• Parental leave top-up for up to 6 months
• Arts, culture, fitness, wellness, quality-time, and workspace-improvement benefits
• Education and learning stipend
• 6 weeks of paid vacation
• Remote travel budget and annual company offsite
• Co-working benefit
• $500 home office stipend