Engineering Manager - Inference Performance
7 hours agoSalary: 240K - 270KSan Francisco; New YorkHybridFull TimeEngineering ManagementJobs by Baseten
BasetenVisit Baseten website
Baseten is an AI inference platform for deploying, optimizing, and scaling custom, open-source, and fine-tuned models in production.
San Francisco, United States
Funding history
About Baseten
Baseten provides model runtimes, inference infrastructure, developer workflows, and deployment options including managed cloud, self-hosted, and hybrid environments.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will provide hands-on technical leadership for inference performance work. You will manage, mentor, and hire engineers; set the technical roadmap; review designs; guide profiling and optimization; and improve GPU utilization, latency, cost, and runtime performance.
Requirements
- Experience managing engineers, including hiring, mentoring, feedback, and performance reviews
- Experience leading or supporting GPU optimization teams
- Understanding of GPU architecture and performance tradeoffs
- Familiarity with PyTorch, TensorRT, or TensorRT-LLM
- Experience driving roadmaps and shipping complex technical projects
- Written and verbal communication skills
Responsibilities
- Lead, mentor, and grow inference performance engineers
- Hire GPU and inference engineering talent
- Own the technical roadmap and execution for runtime performance work
- Review designs and guide profiling and optimization efforts
- Drive productionization of quantization, speculative decoding, KV-cache reuse, chunked prefill, and custom scheduling
- Measure tokens per GPU-hour, utilization, latency, and cost
- Bring up and tune new model architectures on new hardware
- Coordinate launches and priorities with partner functions
- Set standards for engineering quality, benchmarking, operational excellence, and incident response
Benefits
- Equity
- Medical, dental, and vision insurance for U.S. employees and dependents
- Flexible PTO
- Company-wide Winter Break
- Paid parental leave
- Fertility and family-building stipend through Carrot
- Company-facilitated 401(k) for U.S. employees