Member of Technical Staff (Web Access Engineer, Crawler)

Skills

About the Role

You will build reliable fetching and rendering systems that supply public web content to indexing and retrieval systems. You'll diagnose network errors, page-loading issues and incomplete rendering, trace them from network protocols through client-side execution, tell technical failures apart from intentional access restrictions, and ship production fixes that improve content quality and efficiency at scale.

Requirements

• 5+ years of software engineering experience in web infrastructure, browsers, CDNs or crawling
• Strong programming skills in at least one of Go, Python, Rust or C++
• Deep understanding of HTTP/1.1, HTTP/2 and HTTP/3, including status semantics, caching, conditional requests, content encoding and redirects
• Strong knowledge of TLS, DNS, cookies and sessions
• Understanding of CDN caching, HTTP error handling, retry backoff and rate limits
• Knowledge of page lifecycle, service workers, client-side rendering and shadow DOM
• Experience debugging JavaScript; fluency with Chrome DevTools, CDP, HAR files, curl and packet captures
• Comfort working independently when the path is not fully defined

Responsibilities

• Own fetching and rendering improvements end to end, from investigation through production rollout, partnering with crawler engineers and SREs
• Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects and client-side rendering, and classify root causes
• Improve page loading, completion detection and content capture to handle dynamic pages reliably
• Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors
• Build tools to reproduce failures, compare crawler output against expected content and create regression tests
• Define metrics for fetching reliability, content completeness and error classification to prioritize improvements