Role Overview:
We are seeking an experienced AI Test Lead to lead end-to-end testing and quality assurance of Generative AI applications. This role bridges AI quality engineering, test automation, and customer engagement. You will collaborate with QA Managers, AI/ML Engineers, Prompt Engineers, Full Stack Developers, and DevOps Engineers to validate production-ready, scalable AI applications against industry AI testing standards, while serving as the primary testing point of contact for customers.
Key Responsibilities
- Define and execute AI testing strategy, planning, execution, defect management, and release sign-off.
- Lead functional, automation, API, integration, performance, and AI-specific testing for GenAI applications.
- Design test approaches for LLM evaluation, prompt regression, hallucination detection, bias and fairness testing, robustness validation, and output quality assessment.
- Manage sprint-level test planning, estimation, quality metrics, and release readiness in Agile environments.
- Act as the primary QA contact for customers, driving requirement discussions, status reviews, defect triage, and UAT support.
- Build and enhance Python-based automation frameworks using Pytest, Playwright, Selenium, and API automation tools.
- Implement automated AI evaluation pipelines using tools such as DeepEval, Ragas, LangSmith, or promptfoo.
- Integrate test automation into CI/CD pipelines and establish quality governance, documentation, and reporting standards.
- Mentor QA engineers and promote best practices in AI testing and automation.
Required skills
- 8–15+ years in software testing and QA, with experience leading testing teams.
- Hands-on experience in AI/GenAI testing, LLM validation, or intelligent application testing.
- Strong Python programming and test automation expertise.
- Experience with API testing, automation frameworks, and CI/CD pipelines.
- Strong customer communication, stakeholder management, and delivery leadership skills.
- Experience with Agile/Scrum methodologies and test management tools.
Preferred qualifications
- Experience with RAG systems, conversational AI, AI agents, or LLM-based applications.
- Familiarity with AI governance frameworks such as ISO/IEC 42001, NIST AI RMF, or EU AI Act.
- Knowledge of AI quality metrics including accuracy, relevance, groundedness, consistency, and toxicity evaluation.
What This Role Offers
- Lead cutting-edge GenAI testing programs with high visibility to customers and senior leadership
- Ownership of AI quality strategy from test planning through production sign-off
- Work with talented AI architects, engineers, and domain experts
- Continuous learning and growth in the rapidly evolving AI testing and quality engineering space