Core Data Engineer
Flow Traders is a global liquidity provider that delivers continuous liquidity in financial markets, with deep expertise in Exchange Traded Products (ETPs). The firm lowers overall trading costs for market participants by providing higher execution quality across global exchanges and trading platforms.
About Flow Traders
Flow Traders is a technology-driven liquidity provider operating in global financial markets, with a particular focus on Exchange Traded Products (ETPs). The company combines deep ETP expertise, cutting-edge technology, and strong risk management to increase transparency and efficiency across the financial ecosystem, serving market participants such as exchanges and trading platforms across Europe, the U.S. and Asia-Pacific. Beyond its core trading business, Flow Traders operates a dedicated strategic capital investment unit that partners with companies aligned to its mission of transparency and efficiency in financial markets, and it runs the Flow Traders Foundation, a philanthropic initiative focused on fighting poverty.
Skills
About the Role
You will own the end-to-end lifecycle of some of the most critical real-time data products at Flow Traders, bringing low-latency analytical capabilities to end users across the organization. You will architect and execute the blueprint for these data products, enhancing the Data Platform to develop and monitor automated pipelines so stakeholders always have immediate access to reliable data. You will become an expert in dynamic data sets drawn from market data, network traffic, and application logs, optimizing their use and exploring new sources to unlock value in real time. You will collaborate closely with teams across the firm to ensure systems are designed with real-time data considerations at their core, streamlining workflows and maximizing the insights drawn from real-time analytics. As steward of these vital data products, you will be accountable for delivering them at exceptional quality with a focus on low latency, providing benefits felt across all divisions daily.
Requirements
- Excellent programming skills in Python
- Working proficiency with Linux, SQL and relational databases, and version control systems
- Proven experience with real-time OLAP databases like ClickHouse, Apache Pinot, StarRocks required
- Experience with Airflow, Kafka, and GCP (especially GKE, Dataproc, and Big Query) highly desired
- Experience with Pandas, Spark, Apache Beam, Apache NiFi a plus
- Domain literacy in exchange market data, network traffic data, security reference/pricing data, ETF compositions, application log data
- Extreme attention to detail and obsession with quality
- Strong interpersonal skills, works well with technical and non-technical stakeholders
- Able to balance individual stakeholder demands while preserving data quality and applicability
- Organized and able to define, manage, and execute multiple projects in parallel while triaging operational concerns
Responsibilities
- Deliver core data products including market data, trades/orders/quotes/inquiries, and application logs with a focus on low-latency analytics
- Build and maintain pipelines, develop quality checks, correct day-to-day issues, and automate manual processing steps
- Define and manage project plans to assume ownership of data products
- Create and refine a common processing model such as fast vs slow path and use of alternative T3 solutions
- Investigate currently available data sources to maximize information and value extracted
- Explore, evaluate, and onboard additional data sources independently and upon stakeholder request
- Automate existing workflows and develop real-time capabilities as a platform offering
- Ensure pipelines leverage standard Flow technologies
- Provide guidance on data-related architecture and design topics
- Ensure licensing and access control are appropriate
Benefits
- Flow Academy for continuous learning and opportunities to attend domain-related conferences
- Comprehensive health insurance coverage
- In-house lounge with a bar, pool table and console games
- Daily catered breakfast and lunch with healthy snacks and drinks
- In-house hairdresser and massage therapist
- Personal trainers, weekly boot camps and a subsidized gym membership
- Annual company trip and a variety of events throughout the year