BENCH is looking for an experienced AI Engineer who has built and shipped AI systems in the past.
You should have hands-on experience building AI agents and AI workflows, working with different models, and taking AI systems into production. You should also have a good understanding of evaluations and how to use them to measure and improve AI systems.
What you’ll do
- Build and maintain AI systems and agents used in production
- Build agent harnesses and workflows around LLMs
- Work with different models and understand which models work best for different tasks
- Build and maintain evaluation systems for AI agents and workflows
- Use evals to identify failure cases and improve system performance
- Work with tool calling, context management, structured outputs, retrieval, and other parts of agent systems
- Build feedback loops and self-improvement systems
- Test new models and approaches and evaluate whether they improve existing systems
- Debug and improve AI systems running in production
What we’re looking for
- 2 to 3+ years of experience working with AI systems
- Strong software engineering skills
- Experience building and shipping AI agents or AI workflows in production
- Good knowledge of LLM evaluations and evaluation frameworks
- Experience designing eval datasets, metrics, graders, and testing approaches
- Experience working with agent harnesses and orchestration
- Experience with tool calling, retrieval, context management, and structured outputs
- Experience working with multiple LLMs and model providers
- Understanding of common failure modes in AI agents and how to debug them
- Experience using production data and feedback to improve AI systems
- Comfortable experimenting with new models and approaches and measuring whether they actually work
We’re looking for someone who has actually built AI systems, understands how they behave in production, and knows how to systematically evaluate and improve them.