Senior Staff Data Engineer Databricks
Shield AI is a U.S. defense-technology company developing mission-autonomy software and autonomous aircraft for military and allied operations.
Funding history
Investors
About Shield AI
Founded in 2015, Shield AI builds Hivemind autonomy software and V-BAT and X-BAT aircraft for operations in contested, GPS- and communications-denied environments. Its current site also presents Aechelon synthetic-reality simulation and Vision Systems detection and tracking products.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will build and operate Databricks ingestion and transformation pipelines across Bronze and Silver lakehouse layers. You will create reusable engineering standards, implement secure platform constructs and CI/CD, enforce data-quality and compliance controls, document data processing, and partner with domain teams on trusted data delivery.
Requirements
- 12+ years of data engineering experience with hands-on ownership of production data pipelines
- Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, Spark, PySpark, and/or Spark SQL
- Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control
- Experience with batch, CDC, and/or streaming ingestion patterns
- Experience with CI/CD and deployment automation for data pipelines and platform assets
- Strong SQL skills and data-modeling fundamentals
- Experience applying data classification, security, retention, or compliance requirements in regulated or security-sensitive workflows
- Ability to design pipelines with awareness of data sensitivity, ownership, permitted use, and downstream impact
Responsibilities
- Build batch and streaming ingestion pipelines into the Databricks lakehouse
- Own Bronze-layer ingestion, metadata capture, load traceability, and recoverable ingestion
- Build Silver-layer cleansing, standardization, deduplication, conformance, and quality-enforcement pipelines
- Define reusable ingestion and transformation patterns and standards
- Implement Databricks catalogs, schemas, service principals, job orchestration, and deployment patterns
- Build CI/CD pipelines for data-platform assets
- Apply data-classification, segregation, and handling requirements
- Build data-quality controls and route problematic records according to defined rules
- Maintain documentation for source objects, transformations, quality rules, and limitations
- Collaborate with analytics and domain engineering teams on data governance and onboarding
Benefits
- Bonus
- Benefits
- Equity