AI Fabric Architect
fal is an active generative-media AI platform for developers, providing optimized model APIs, serverless deployment, and GPU compute.
Funding history
About fal
Founded in 2021 by Burkay Gur and Gorkem Yurtseven, fal provides infrastructure for production generative-media applications, including image, video, audio, 3D, and multimodal models.
Skills
About the Role
You will be the architect of the network that GPU clusters run on, owning the fabric end to end from the NIC in each GPU server through the spine and leaf. You will define the fabric architecture each site is built from, then get hands-on to make sure it is built and performs. That means writing port maps, tracing bad optics, tuning congestion control, running performance acceptance before capacity goes live, debugging fabric issues, selecting vendors and hardware, building monitoring and automation with colleagues, and mentoring future network engineers.
Requirements
- 7+ years in data center or HPC networking, including direct responsibility for production GPU cluster fabrics
- Track record of taking GPU fabrics from empty racks to production with hands-on work
- Deep expertise in InfiniBand (NDR/XDR, UFM, subnet management) and/or RoCEv2 lossless Ethernet
- Strong command of rail-optimized and Clos/fat-tree designs, BGP/ECMP, and EVPN/VXLAN
- Understanding of how fabric design and tuning affect collective communication and workload performance
- Comfortable automating with Python, Ansible, or similar
- Strong Linux skills and ability to debug across host, NIC, and fabric
- Sound judgment on tradeoffs between cost, performance, reliability, and time to deploy
- NVIDIA Quantum/Spectrum-X, ConnectX-7/8, BlueField-3 SuperNICs, and/or Broadcom Tomahawk 5-class Ethernet
- 400G/800G OSFP optics, AOC/DAC, MPO trunking
Responsibilities
- Own the fabric architecture for every site: GPU backend fabric (InfiniBand and/or RoCEv2) and GPU frontend fabric
- Choose topologies and make key calls: rail-optimized vs. other designs, InfiniBand vs. Ethernet, oversubscription ratios, and scaling from one pod to many
- Turn the architecture into a repeatable network blueprint: cabling plans, port maps, fabric addressing and routing design, and fabric bill of materials
- Lead hands-on bring-up at each site: validate cabling and firmware, configure the fabric manager and switches, and run performance acceptance (NCCL tests, burn-in)
- Tune workload performance: congestion control, PFC/ECN, adaptive routing, and NCCL settings for inference and training traffic
- Debug fabric problems from flapping links to slow jobs
- Evaluate and select AI fabric vendors and hardware, and push back with data when specs are not met
- Collaborate on network monitoring, telemetry, and automation, and act as top escalation point for AI fabric incidents
- Document processes and help hire and mentor network engineers
Benefits
- Equity
- Relocation assistance to San Francisco
- Health, dental, and vision insurance (US)
- Regular team events and offsite
