Staff Engineer Distributed Storage HPC and AI Infrastructure
Together AI operates an AI-native cloud platform for open and custom AI models.
About Together AI
Together AI provides production AI infrastructure spanning inference, accelerated compute, model training and fine-tuning, and secure code sandboxes for AI development.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will design and deliver multi-petabyte storage systems for AI training and inference workloads. You will architect parallel filesystems and object stores, integrate storage technologies, optimize costs and data paths, and build Kubernetes-native storage operators. You will improve reliability through monitoring, disaster recovery, automation, and performance tuning while partnering with ML and SRE stakeholders.
Requirements
- 8+ years of storage engineering experience, including 3+ years managing distributed storage at multi-petabyte scale.
- Experience deploying and operating high-performance storage for GPU or HPC clusters.
- Production experience with Kubernetes and cloud-native storage.
- Strong Go and Python programming skills for production tools.
- Technical leadership experience improving performance, reliability, or cost efficiency.
- Expertise with distributed storage systems such as WekaFS, Lustre, GPFS, or BeeGFS.
- Production object-storage experience with S3, MinIO, Ceph, or R2.
- Knowledge of Kubernetes CSI drivers, StatefulSets, PersistentVolumes, storage operators, and custom controllers.
- Experience with GPU storage optimization, RDMA or InfiniBand, and parallel filesystem optimization.
- Experience with Terraform, Ansible, Helm, GitOps, and ArgoCD.
- Advanced Linux storage knowledge, including filesystems, LVM, NVMe, and RAID.
- Experience with Prometheus, Grafana, and Thanos.
Responsibilities
- Design multi-petabyte AI and ML storage systems and optimize capacity and costs.
- Design and optimize RDMA, InfiniBand, 400GbE, NVMe-oF, iSCSI, and TCP/IP storage networking.
- Build Kubernetes storage operators and controllers for automated provisioning, multi-tenancy, and quotas.
- Optimize caching, parallel filesystems, and data paths for GPU nodes at scale.
- Build multi-tier caches and implement data-locality, model-weight distribution, prefetching, and eviction strategies.
- Implement monitoring, alerting, SLOs, disaster recovery, backups, runbooks, and chaos engineering.
- Partner with ML and SRE teams, mentor on storage practices, contribute to open source, and write documentation and postmortems.
Benefits
- Hybrid working with two days per week at the Amsterdam office.
