Senior LLM Systems Engineer
Skills
About the Role
As a Senior LLM Systems Engineer, you will own the LLM driven components of the oracle automation stack and ensure accuracy, performance, resilience, and operational quality for model powered decisions. You will build evaluations, observability, tooling, fallbacks, and feedback loops to make LLM behavior measurable and dependable in real world conditions. You will improve prompts, model selection, tooling usage, structured outputs, retrieval, and evaluation coverage. You will design validation, retries, fallbacks, uncertainty handling, and human review paths for ambiguous inputs. You will build datasets, dashboards, traces, and review loops to surface model quality. You will enhance agent orchestration and tool use across internal services APIs search workflows databases and external data sources. You will debug live issues, investigate regressions, improve runbooks, and reduce operator friction. You will be measured by broader coverage, latency and cost improvements while preserving quality.
Requirements
- 3+ years of professional software engineering experience in Python TypeScript or similar production languages.
- Hands-on experience building production systems that use LLMs agents retrieval structured outputs or model-powered workflows.
- Experience designing evaluations test datasets regression checks quality metrics or manual review loops for AI systems.
- Strong debugging ability across APIs databases queues logs model outputs and external data sources.
- Practical understanding of prompt engineering tool calling structured output validation retrieval and common LLM failure modes.
- Ability to reason carefully about correctness in uncertain or adversarial environments.
- High agency strong ownership and clear written communication.
- Experience with oracle systems prediction markets DeFi protocols or other crypto infrastructure.
- Experience with UMA optimistic oracle mechanisms Polymarket or similar systems.
- Experience building agentic systems that use tools search browser automation APIs or database queries.
- Experience with LLM tracing model monitoring evaluation frameworks or AI observability tools.
- Experience optimizing model cost and latency at scale.
- Experience with Postgres data pipelines queue-based systems background jobs or event-driven architectures.
- Familiarity with blockchain operational constraints especially RPC limits indexing event logs finality and chain-specific behavior.
- Experience with GCP Cloud Run GitHub Actions Terraform or similar infrastructure.
Responsibilities
- Own and improve the LLM driven components of the oracle automation stack.
- Improve LLM accuracy by refining prompts model selection tool usage and retrieval.
- Improve system performance by reducing latency token usage and cost while preserving decision quality.
- Enhance resilience with validation retries fallbacks uncertainty handling and human review paths.
- Build evaluations datasets dashboards traces and review loops to make model quality visible.
- Improve agent orchestration and tool use across internal services APIs search workflows databases and external data sources.
- Support production operations by debugging live issues improving runbooks and reducing operator friction.
Benefits
- Meaningful long term equity participation.
- 100% remote
- Flexible vacation and family care
- Training and development
- Remote work options
- At least two team wide offsites a year