Member of Technical Staff Distributed Training Systems
Bagel Labs is an artificial-intelligence research lab developing distributed training methods for frontier diffusion models on commodity hardware. Its work focuses on generative models for robotics, video, and world modelling.
Projects
About Bagel Labs
Bagel Labs develops Distributed Diffusion Models (DDMs), which use independently trained smaller expert models and a lightweight router rather than a single tightly coupled diffusion model. The organization has publicly released Paris, a DDM, and describes Paris-2 as a video DDM pretrained from scratch.
Skills
About the Role
You will build and operate the systems that support distributed training across heterogeneous compute. You will create reliable experiment infrastructure, benchmark and evaluation harnesses, and traceable data and model pipelines. You will add observability, debug training workflows, and turn fragile research prototypes into repeatable runs and trustworthy artifacts.
Requirements
- Hands-on experience with distributed training, GPU workloads, experiment infrastructure, or large-scale machine learning systems.
- Ability to debug performance, reliability, and reproducibility problems in complex training and evaluation workflows.
- Clear communication.
- Strong ownership.
Responsibilities
- Build and operate distributed training for diffusion-heavy workloads across heterogeneous compute.
- Make experiments reliable with launchers, configurations, checkpointing, logging, metrics, run comparison, and reproducibility.
- Build benchmark and evaluation harnesses for physical AI research, including robotics and world-model experiments.
- Own data and model pipelines so results trace back to a dataset, version, and configuration.
- Add observability for GPU utilization, failure modes, data quality, routing behavior, model quality, and training stability.
- Turn fragile research prototypes into repeatable runs and trustworthy artifacts.
Benefits
- Meaningful equity.
- Paid travel to top machine learning and systems conferences around the world.