Software Engineer, Training Infrastructure
2 days agoSeniorSalary: 300000 - 400000 USD / yearSan Francisco, California or New York, New YorkOnsiteFull TimeAiJobs by Thinking Machines Lab
Thinking Machines LabVisit Thinking Machines Lab website
Artificial-intelligence research and product company building customizable AI systems, including the Tinker training API and Inkling open-weight models.
Distributed
Funding history
About Thinking Machines Lab
Thinking Machines Lab develops AI products that let researchers and developers fine-tune and use models, while also releasing open-weight multimodal models.
Skills
Candidate Availability
About the Role
You will build the distributed training stack that runs supervised and reinforcement learning jobs reliably across large GPU clusters. You'll work on scheduling, orchestration, checkpointing and recovery, optimize throughput and cost, bring up new model architectures including mixture-of-experts, build reliability and observability, support multi-tenant LoRA fine-tuning, and collaborate with API and product teams.
Requirements
- 4+ years of software engineering including building and operating production distributed systems
- Strong Python plus experience in C++, Rust, or Go
- Hands-on experience with large-scale model training or inference infrastructure using frameworks such as PyTorch
- Track record of debugging complex performance and reliability issues in distributed environments
Responsibilities
- Design, build, and operate distributed training infrastructure spanning scheduling, orchestration, checkpointing, and recovery
- Improve training throughput and cost efficiency through profiling, parallelism, memory optimization, and kernel- and communication-level tuning
- Bring up and optimize training support for new model architectures including mixture-of-experts
- Build reliability: failure detection, job recovery, multi-tenant isolation, and observability
- Support multi-tenant LoRA fine-tuning and coupled sampling and training for reinforcement learning
- Collaborate with Tinker API, frameworks, and product teams
Benefits
- Health, dental, and vision benefits
- Unlimited PTO
- Paid parental leave
- Relocation support
- Visa sponsorship
