Site Reliability Engineer Fleet Command and Control
EtchedVisit Etched website
Etched is an AI-hardware company building rack-scale frontier inference clusters.
San Jose, United States
Funding history
About Etched
Etched co-designs chips, racks, software, and manufacturing systems for efficient inference of frontier AI models, targeting throughput, latency, cost, and power efficiency across prefill and decode workloads.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will define fleet health signals, SLOs, alerting standards, and automated recovery workflows. You will route incidents, shape telemetry and tracing standards, run incident management and on-call operations, review reliability designs, debug production and hardware failures, and coordinate on-site technical work.
Requirements
- 2+ years in site reliability, production engineering, or infrastructure for large-scale production systems
- Experience defining alerting standards using Prometheus, Grafana, or similar tools
- Knowledge of SLOs, error budgets, and alert design
- On-call experience for external-user services, including incident leadership and blameless postmortems
- Experience debugging production incidents from logs, metrics, traces, and core dumps
- Experience with application and scripting languages such as Python, Go, or Rust
- Hands-on experience with Linux, containers, Kubernetes, and infrastructure as code
- Fluency with Linux debugging tools including perf, strace, gdb, dmesg, journalctl, lspci, and tcpdump
- Comfort diagnosing hardware and physical-layer failures
- Clear writing and ability to establish shared standards across teams
Responsibilities
- Define health signals and automated actions for inference systems
- Set SLOs and common alerting standards
- Reduce alert noise through suppression, correlation, and retirement of ineffective pages
- Automate recovery workflows with guardrails
- Route events to the appropriate engineer and dispatch on-site technicians
- Shape fleet telemetry and establish OpenTelemetry tracing standards
- Run incident management, postmortems, corrective-action tracking, and sustainable on-call operations
- Review service designs for failure modes and graceful degradation
- Debug production incidents, request latency, hardware faults, and alerting issues
- Create runbooks and agent-executable recovery tasks
Benefits
- Medical, dental, and vision packages with generous premium coverage
- $500 per month credit for waiving medical benefits
- Housing subsidy of $2,500 per month for employees living within walking distance of the office
- Relocation support for employees moving to San Jose
- Wellness benefits covering fitness and mental health
- Daily lunch and dinner in the office
- Unlimited compute budget subject to ROI justification