Technical Lead - Training Runtime, Data(set) Movement

Skills

About the Role

You will own datasets across training infrastructure, setting direction for how training jobs read data. You will design a unified read platform, define APIs and versioning, build reliability into the read path, create data visualizers and write production code while aligning researchers and infrastructure teams.

Requirements

• Built or owned dataset, data loading, storage or distributed training infrastructure at large scale
• Care about API design, debugging, performance and correctness
• Understanding of failure modes of large distributed training jobs
• Experience with stateful iterators, checkpoint/restart semantics, caching, remote services or high-throughput storage reads
• Comfortable working across Python and lower-level systems code
• Experience with multimodal, video, reinforcement learning or pretraining data pipelines

Responsibilities

• Design and build a unified dataset read platform for multiple training frameworks
• Define dataset APIs, storage-format expectations, registration/versioning and migration paths
• Build reliability into the read path, including stateful iteration, caching, fast restart and recovery
• Build terminal and web-based visualizers for text, multimodal and reinforcement learning data
• Write and review production code in data loading, service, caching and reliability paths
• Partner with training framework, reinforcement learning, multimodal, storage, runtime and cluster infrastructure teams

Benefits

• Equity
• Performance-related bonus
• Medical, dental, and vision insurance with HSA employer contributions
• Health FSA, Dependent Care FSA, and commuter pre-tax accounts
• 401(k) with employer match
• Paid parental leave (up to 24 weeks birth parents, 20 weeks non-birthing parents)
• Paid medical and caregiver leave (up to 8 weeks)
• Flexible PTO for exempt employees
• 13+ paid company holidays and company office closures
• Paid sick or safe time
• Mental health and wellness support
• Employer-paid basic life and disability coverage
• Annual learning and development stipend
• Daily meals in offices and meal delivery credits
• Relocation support
• Charitable donation matching and wellness stipends