Engineering Manager, Inference Router

Skills

About the Role

You will lead the team responsible for the intelligent control plane of an AI inference platform. You'll own the architecture of the routing layer across data centers, guide design of multi-region routing, P/D disaggregation and cache-aware routing, drive delivery of cross-functional initiatives, scale the platform for latency and reliability, and build, mentor and grow your team of engineers.

Requirements

• 8+ years of experience in large-scale software engineering, distributed systems or infrastructure
• 2+ years of engineering management experience
• Strong C++ and/or Go programming skills
• Deep experience operating production services on Kubernetes
• Track record building highly available, scalable systems
• Experience designing request routing, load balancing, scheduling or traffic-management systems
• Experience with cloud infrastructure and scalable microservices

Responsibilities

• Own the architecture and evolution of the inference routing layer across data centers and third-party integrations
• Guide design of global request routing, heterogeneous compute routing, prefix-aware routing and low-TTFT scheduling
• Drive routing algorithms balancing cache locality, queueing, load and latency
• Build simulation and load-generation frameworks to validate routing policies
• Build, manage and grow a team of distributed systems, infrastructure and ML systems engineers
• Provide technical direction, mentorship and career development
• Recruit top talent
• Drive cross-functional execution and ensure high-quality releases
• Scale the routing layer and improve TTFT, throughput and cache-hit rate
• Design for high availability, failover and redundancy
• Partner with inference runtime, platform, SRE, ML and third-party teams