Staff Software Engineer, Lakeflow Pipelines DR
Databricks is a data and AI platform that lets organizations build analytics, AI agents, and applications on a unified, governed lakehouse.
Funding history
Investors
About Databricks
Data engineers, analysts, and AI teams use Databricks to process large datasets, build reliable pipelines, and train models on a single governed platform. Users can run SQL analytics, serve ML predictions in real time, and deploy AI agents grounded in enterprise data. Its open lakehouse architecture provides consistent security and governance across analytical and operational workloads.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will design and implement distributed systems that replicate and recover pipelines across regions. You will address consistency, idempotency, causal ordering, failover, failback, recovery, metadata reconciliation, observability, and failure-injection testing to prevent data loss or duplication.
Requirements
- Distributed system
- Database
- Storage system
- Streaming system
- Reliability engineering
- Java, Scala, C++, Go, Python, or similar production language
- Consistency
- Transaction
- Idempotency
- Replication
- Checkpointing
- Data lineage
Responsibilities
- Design and implement distributed systems for cross-region pipeline replication and recovery
- Solve consistency, idempotency, causal-ordering, failover, failback, and recovery problems
- Build recovery for pipelines, streaming tables, and materialized views
- Reconcile metadata and implement deep-clone and observability capabilities
- Conduct failure-injection testing and recovery simulations
- Perform game-day testing and reason about failure modes
Benefits
- Annual performance bonus eligibility
- Equity eligibility