Principal AI Data & Evaluation Strategist
Braintrust is a decentralized talent network that connects skilled professionals and freelancers with leading companies. The platform is user-owned and controlled, utilizing the BTRST token on the Ethereum blockchain for governance, payments, and other network activities. Braintrust also offers an AI-powered recruiting platform, Braintrust AIR, to help businesses streamline their hiring process by reducing costs, eliminating bias, and identifying top talent more efficiently.
Projects
About Braintrust
Braintrust operates as a decentralized, user-owned talent network connecting skilled knowledge workers with top companies globally. The platform is built on the Ethereum blockchain and uses its native BTRST token for governance, staking, and payments, allowing talent to retain 100% of their earnings. Braintrust offers several products and services, including a Talent Marketplace and an AI-powered recruiting platform called Braintrust AIR. Braintrust AIR uses conversational AI video interviews to screen candidates, providing detailed scorecards to help companies save time, reduce costs, and eliminate bias in hiring. The platform caters to various industries, including healthcare, technology, and financial services. Braintrust fosters a strong community through its Braintrust Academy, which offers free courses, and various community programs and events. The network is governed by its community, ensuring it serves the needs of its users rather than a centralized corporation.
Skills
About the Role
You will serve as a consultative expert and thought partner alongside program managers, providing strategic guidance during project discovery, solution design, data planning, evaluation-framework creation, quality-rubric development, and pilot execution. You'll work directly with stakeholders, engineering teams, researchers, and program leadership to define project requirements before execution begins. This is not an operational-delivery role; you'll bring subject-matter expertise, challenge assumptions, identify risks early, and ensure projects are designed for quality, scalability, and measurable outcomes. It is a remote, flexible engagement contributing to frontier AI work.
Requirements
- 10+ years in AI, data science, machine learning, research, or AI operations
- Experience designing large-scale AI data programs
- Experience with LLM evaluation, human feedback systems, AI benchmarking, and dataset development
- Strong written English and the ability to communicate complex ideas clearly
- Comfortable working independently in a remote environment
- Based in the United States
Responsibilities
- Lead project discovery and translate business goals into AI data requirements
- Define project success metrics and identify risks and failure modes early
- Design training and evaluation datasets and define taxonomy structures
- Recommend data-sourcing methodologies and establish ground-truth standards
- Identify coverage gaps and edge cases
- Develop human-evaluation frameworks and quality-measurement methodologies
- Establish acceptance criteria and design benchmarking approaches
- Build calibration mechanisms
- Challenge assumptions that may impact project quality
- Recommend industry best practices and guide teams through ambiguity
- Support executive and client reviews