A Software Engineer - Technical Advisor evaluates frontier AI coding models to uncover precisely where and why they break when working on real engineering tasks. Candidates spend their time reviewing model-generated pull requests, evaluating full agent session logs, and constructing hard, container-based test problems. Rather than writing code from scratch to ship product features, this role acts as a high-bar technical authority—using deep code review skills and rigorous written rationale to separate genuinely good software engineering from plausible, superficial AI code.
Daily Tasks
- Audit model-generated pull requests (PRs) against real production repositories, documenting every identified issue alongside its severity and detailed technical rationale.
- Evaluating full coding-agent sessions to analyze what the model investigated, verified, assumed, or skipped.
- Designing and building container-based (Docker) technical benchmarks used to test AI models.
- Writing clear, original technical rationales explaining why code fails (all written work must be self-authored without AI text generators).
- Collaborating asynchronously with AI researchers to share findings and refine evaluation criteria.
Required Skills
- 8+ years of production engineering experience preferred (exceptions only for clearly exceptional profiles).
- Backgrounds including Senior, Staff, or Principal Software Engineers, Tech Leads, or Open-Source Maintainers.
- Experience working in production codebases with a strict code review culture (startups, big tech, or open source are all acceptable).
- Polyglot adaptability: Heavy experience with Python and TypeScript is common, but must be comfortable dropping into unfamiliar languages weekly.
- Core Tooling: Demonstrated comfort using Docker, git, and the Command Line Interface (CLI) to reproduce, isolate, and debug results locally.
- Cross-Layer Fluency: Comfort working across backend, frontend, APIs, data, testing, or developer tooling.
- Strong written communication skills—able to articulate why code fails, not just how to fix it quickly.
Pay: $120.00 - $150.00 per hour
Expected hours: 30.0 – 40.0 per week
Work Location: Remote