We launched recruiters.rip, let your AI apply for you. First month free with code onemonthfree

Software Engineer, Training Infrastructure

Artificial-intelligence research and product company building customizable AI systems, including the Tinker training API and Inkling open-weight models.

Seed8 current maintainers6 active leads
Distributed
About Thinking Machines Lab

Thinking Machines Lab develops AI products that let researchers and developers fine-tune and use models, while also releasing open-weight multimodal models.

View jobs by Thinking Machines Lab

Skills

Candidate Availability

About the Role

You will build the distributed training stack that runs supervised and reinforcement learning jobs reliably across large GPU clusters. You'll work on scheduling, orchestration, checkpointing and recovery, optimize throughput and cost, bring up new model architectures including mixture-of-experts, build reliability and observability, support multi-tenant LoRA fine-tuning, and collaborate with API and product teams.

Requirements

  • 4+ years of software engineering including building and operating production distributed systems
  • Strong Python plus experience in C++, Rust, or Go
  • Hands-on experience with large-scale model training or inference infrastructure using frameworks such as PyTorch
  • Track record of debugging complex performance and reliability issues in distributed environments

Responsibilities

  • Design, build, and operate distributed training infrastructure spanning scheduling, orchestration, checkpointing, and recovery
  • Improve training throughput and cost efficiency through profiling, parallelism, memory optimization, and kernel- and communication-level tuning
  • Bring up and optimize training support for new model architectures including mixture-of-experts
  • Build reliability: failure detection, job recovery, multi-tenant isolation, and observability
  • Support multi-tenant LoRA fine-tuning and coupled sampling and training for reinforcement learning
  • Collaborate with Tinker API, frameworks, and product teams

Benefits

  • Health, dental, and vision benefits
  • Unlimited PTO
  • Paid parental leave
  • Relocation support
  • Visa sponsorship