Licorne Society a été missionné par une startup IA en pleine croissance pour les aider à trouver leur Lead LLM Engineer.
What you will own
You will be responsible for one thing:
Make our AI outputs reliable, fast, and indispensable in real workflows.
Concretely:
-
Design and evolve our LLM / agent architecture
- Own output quality across key use cases (emails, document analysis, etc.)
-
Build evaluation systems (datasets, metrics, regression detection)
-
Drive fast iteration loops from production data
-
Improve retrieval, reasoning, and tool usage
- Ensure production reliability (latency, failure modes, fallback)
-
Work directly with product + founders on what to build and why
What this role is really about
Most teams fail because:
- they don’t know what “good output” means
- they don’t have evals
- they iterate randomly
- they overuse agents
Your job is to fix that.
You will turn:
-
vague user problems
-
- into structured AI systems
-
- with measurable performance
-
- that improve every week
What you need to be excellent at
1. Shipping real LLM systems
- You’ve built systems used in production (not demos)
- You understand RAG, tools, agents, structured outputs
- You can design full pipelines, not just prompts
2. Evaluation-driven development
- You know how to define quality metrics
- You build datasets from real usage
- You run continuous evals to prevent regressions
3. Debugging complex failures
- You can trace issues across:
-
- retrieval
- prompts
- model behavior
- You don’t guess — you isolate and fix
4. Speed of iteration
You move from problem
- improvement in hours or days, not weeks
- You use logs, traces, and data — not intuition alone
5. Strong judgment
- You know when to:
-
- use an agent vs a pipeline
- add complexity vs simplify
- You optimize for reliability and user value , not novelty
What we don’t care about
- Number of years of experience
- Whether you’ve used a specific framework
- Fancy research credentials
If you can build, debug, and improve real systems , you’re a fit.
What success looks like (first 90 days)
- Clear eval framework for core use cases
-
Measurable improvement in output quality
- Faster iteration cycles across the team
-
Reduced hallucinations / failures
- Stronger system architecture decisions
Stack (context, not requirements)
- Python (FastAPI)
- Postgres
- Google Cloud
- LangGraph / LangChain (evolving)
- PostHog (product analytics)
- Langfuse (LLM traces)
- LLM APIs (Azure OpenAI)