Licorne Society a été missionné par une startup IA en pleine croissance pour les aider à trouver leur Lead LLM Engineer.
What You Will Own
You will be responsible for one thing:
Make our AI outputs reliable, fast, and indispensable in real workflows.
Concretely
- Design and evolve our LLM / agent architecture
- Own output quality across key use cases (emails, document analysis, etc.)
- Build evaluation systems (datasets, metrics, regression detection)
- Drive fast iteration loops from production data
- Improve retrieval, reasoning, and tool usage
- Ensure production reliability (latency, failure modes, fallback)
- Work directly with product + founders on what to build and why
What This Role Is Really About
Most teams fail because:
- they don’t know what “good output” means
- they don’t have evals
- they iterate randomly
- they overuse agents
Your job is to fix that.
You Will Turn
- vague user problems
- → into structured AI systems
- → with measurable performance
- → that improve every week
What You Need To Be Excellent At
- Shipping real LLM systems
- You’ve built systems used in production (not demos)
- You understand RAG, tools, agents, structured outputs
- You can design full pipelines, not just prompts
- Evaluation-driven development
- You know how to define quality metrics
- You build datasets from real usage
- You run continuous evals to prevent regressions
- Debugging complex failures
- You can trace issues across:
- retrieval
- prompts
- model behavior
- You don’t guess — you isolate and fix
- Speed of iteration
- You move from problem → improvement in hours or days, not weeks
- You use logs, traces, and data — not intuition alone
- Strong judgment
- You know when to:
- use an agent vs a pipeline
- add complexity vs simplify
- You optimize for reliability and user value, not novelty
What We Don’t Care About
- Number of years of experience
- Whether you’ve used a specific framework
- Fancy research credentials
If you can build, debug, and improve real systems, you’re a fit.
What Success Looks Like (first 90 Days)
- Clear eval framework for core use cases
- Measurable improvement in output quality
- Faster iteration cycles across the team
- Reduced hallucinations / failures
- Stronger system architecture decisions
Stack (context, Not Requirements)
- Python (FastAPI)
- Postgres
- Google Cloud
- LangGraph / LangChain (evolving)
- PostHog (product analytics)
- Langfuse (LLM traces)
- LLM APIs (Azure OpenAI)