Attorney, Evaluation Operations (Contract)

Skills

About the Role

You will work as a contract attorney evaluating and grading AI outputs before they reach customers. You will compare AI-generated outputs for the same legal task, test new features in pre-release environments, review grading rubrics, and build evaluation datasets. Projects typically run 2 to 20 hours over multiple days, some with 24-48 hour turnaround, and you may decline any project. The work is not client-facing.

Requirements

• JD and active license to practice in at least one U.S. jurisdiction
• 3+ years of practice experience at an AmLaw 100 or comparable firm, or in-house legal department
• Experience in commercial litigation or transactional law
• Experience using AI tools
• Clear, concise legal writing
• Availability for short-notice projects and occasional weekends
• US-based

Responsibilities

• Compare AI-generated outputs for the same legal task and evaluate quality against a written rubric
• Test new product features end-to-end in a pre-release environment and provide qualitative feedback
• Review and assess grading rubrics for accuracy and completion
• Build evaluation datasets including realistic legal queries, contract markups, and synthetic legal documents

Benefits

• Laptop provided
• Remote work with own schedule