Senior Data Engineer
Skills
About the Role
You will support data scientists, analysts and software engineers by building maintainable infrastructure and tooling they can use to deliver end-to-end solutions to business problems. You'll work with terabytes to petabyte-scale data in a complex environment supporting multiple products and stakeholders across the US and Kuala Lumpur. Using Python and/or Java, you'll design and implement an analytical environment with in-house and third-party tools to automate data activities and enable efficient processing of growing data volume and complexity. You'll design and implement complex data pipelines and data models for analytical consumption, working with Redshift, Snowflake, EMR, Kubernetes and Airflow as your main tools. You'll write scalable, performant SQL queries over billions of rows and help simplify processing so insights are easier to extract. You should bring deep experience designing and managing large datasets and pipelines, and be an authority at building scalable, stable and cost-efficient solutions.
Requirements
- Expert at writing and optimising SQL queries
- Proficiency in Python, Java or similar languages
- Familiarity with data warehousing concepts
- Experience in Airflow or other workflow orchestrators
- Familiarity with basic principles of distributed computing
- Experience with big data technologies like Spark, Delta Lake or others
- Proven ability to innovate and leading delivery of a complex solution
- Excellent verbal and written communication - proven ability to communicate with technical teams and summarise complex analyses in business terms
- Ability to work with shifting deadlines in a fast-paced environment
- Authoritative in ETL optimisation, designing, coding, and tuning big data processes using Spark
- Knowledge of big data architecture concepts like Lambda or Kappa
- Experience with streaming workflows to process datasets at low latencies
- Experience in managing data - ensuring data quality, tracking lineages, improving data discovery and consumption
- Sound knowledge of distributed systems - able to optimise partitioning, distribution and MPP of high-level data structures
- Experience in working with large databases, efficiently moving billions of rows, and complex data modelling
- Familiarity with AWS is a big plus
- Experience in planning day to day tasks, knowing how and what to prioritise and overseeing their execution
Responsibilities
- Design, implement, operate and improve the analytics platform
- Design data solutions using various big data technologies and low latency architectures
- Collaborate with data scientists, business analysts, product managers, software engineers and other data engineers to develop, implement and validate deployed data solutions
- Maintain the data warehouse with timely and quality data
- Build and maintain data pipelines from internal databases and SaaS applications
- Understand and implement data engineering best practices
- Improve, manage, and teach standards for code maintainability and performance in code submitted and reviewed
- Mentor and provide guidance to junior engineers on the job