Staff and Senior Software Engineer, Inference Infrastructure

Skills

About the Role

You will design, build, and maintain distributed inference systems. You will develop request routing, load balancing, autoscaling, fleet orchestration, deployment pipelines, and support for new accelerator platforms and model architectures.

Requirements

• Significant software engineering experience, particularly with distributed systems
• Desire to learn machine learning systems and infrastructure
• Experience with high-performance large-scale distributed systems
• Experience implementing and deploying machine learning systems at scale
• Experience with load balancing, request routing, or traffic management
• Familiarity with LLM inference optimization, batching, and caching
• Experience with Kubernetes and cloud infrastructure
• Proficiency in Python or Rust

Responsibilities

• Design, build, and maintain distributed systems that serve Claude
• Develop resilient systems that adapt to real-world events
• Develop request routing, load balancing, and traffic-management systems
• Maximize fleet compute efficiency through autoscaling and workload orchestration
• Build and operate production deployment pipelines
• Provide high-performance inference infrastructure for researchers
• Integrate AI accelerator platforms and support new model architectures

Benefits

• Equity donation matching
• Generous vacation
• Parental leave
• Flexible working hours