Senior Data Scientist AI Evaluation
Alpaca is a brokerage infrastructure company providing developer APIs for stocks, options, fixed income, and crypto trading, plus tokenization infrastructure for TradFi–DeFi markets.
Funding history
Investors
About Alpaca
Alpaca provides API-first brokerage infrastructure for fintechs, institutions, algorithmic traders, and developers. Its offerings include Broker API, Trading API, market data, self-clearing brokerage services, crypto services, and the Instant Tokenization Network for in-kind minting and redemption of tokenized assets.
Skills
About the Role
You will define ground truth, metrics, and scoring methods for models and agents. You will build repeatable evaluation loops, operationalize evaluation infrastructure, translate results into system improvements, and establish measurable AI quality standards with cross-functional partners.
Requirements
- Quantitative measurement rigor in model evaluation, metric validation, or experimentation
- Statistical and machine learning foundation
- Experience with sample sizing, confidence intervals, significance, non-determinism, and human ground truth validation
- Python proficiency
- SQL proficiency
- Experience evaluating production models
- Judgment defining quality metrics and ground truth for ambiguous outputs
- Communication and cross-functional collaboration skills
- Problem-solving ability
- 6–10 years of quantitative data science or machine learning experience with measurement or evaluation focus
Responsibilities
- Define ground truth, metrics, and scoring methods for models and agents
- Build repeatable evaluation loops and catch regressions before release
- Partner with engineering and analytics engineering to operationalize evaluation harnesses
- Translate evaluation results into recommendations for system improvements
- Establish evaluation guidelines, documentation, and review practices
- Mentor stakeholders on evaluation best practices and measurable AI quality
Benefits
- Stock options
- Health benefits
- One-time USD 500 home-office setup
- USD 150 monthly stipend via Brex Card
