Software Engineer Inference

Skills

About the Role

You will operate and scale production inference systems serving live traffic. You will manage safe model rollouts, improve observability and capacity planning, lead incident response, productionize serving techniques, and design resilient failover and redundancy while balancing infrastructure cost and capacity.

Requirements

• Experience operating large-scale, latency-sensitive production systems
• Proficiency in Python and Go or another systems language
• Experience with observability, monitoring, and incident response for production services
• Strong understanding of distributed systems and failure modes at scale
• Experience running production inference for large language models or large-scale ML systems
• Experience with deployment and rollout systems such as canarying, blue/green deployments, or feature flags
• Experience with capacity planning and cost optimization for GPU or TPU infrastructure
• Familiarity with batching, caching, or quantization for inference

Responsibilities

• Operate and scale production inference systems serving live traffic
• Own safe, incremental rollouts of new models, model versions, and inference optimizations
• Build and improve observability, alerting, and capacity planning
• Productionize new serving techniques while maintaining reliability
• Lead incident response, root cause analysis, and durable fixes for production issues
• Design graceful degradation, failover, and redundancy
• Manage capacity and cost tradeoffs for serving infrastructure

Benefits

• Health benefits
• Dental benefits
• Vision benefits
• Unlimited PTO
• Paid parental leave
• Relocation support as needed