The client is looking at hiring a AI Quality & Model Evaluation Lead for their team at Bangalore.
Experience : 5+ Years
Mandatory Skills:
- QA Strategy, Functional/API/Regression Testing
- AI/ML & LLM Evaluation
- Healthcare RCM & Audit Processes
- SQL & Data Analysis
- Statistical Sampling & Quality Assessment
- Test Management & Defect Tracking
- Leadership & Stakeholder Management
Role Summary :
-QA & Software Testing (Functional, integration, regression, API, data validation, test planning and execution)
-AI/ML Model Evaluation (Model accuracy, precision, recall, F1-score, sensitivity, specificity, benchmark and ground-truth validation)
-LLM & Generative AI Evaluation (Response accuracy, relevance, groundedness, hallucination, factuality, consistency, and prompt evaluation)
-Healthcare RCM (Medical coding validation, clinical documentation review, AR denial management, Prior Authorization and audit methodology etc)
-Statistical Sampling & Audit (Random/stratified sampling, sample size determination, quality audits, confidence levels, and error analysis)
-Data Analysis & SQL (Data extraction, validation, reconciliation, quality analysis, and complex SQL queries)
-Python (Data analysis, statistical analysis, test automation, and evaluation scripts)
-LLM Evaluation Tools (DeepEval, Ragas, LangSmith, or equivalent evaluation frameworks)
-API Testing (Postman, REST APIs, request/response validation, authentication, and API automation)
-Test Management & Defect Tracking (JIRA, TestRail, TestLink, defect lifecycle, test execution, and reporting)
-QA Automation (Automation frameworks, regression automation, data validation, and test execution optimization)
-Quality Metrics & Reporting (Test coverage, defect trends, model accuracy, audit results, quality KPIs, and dashboards)
-Go/No-Go & Risk Assessment (Release readiness, quality risk evaluation, acceptance criteria, and data-driven decisions)
-Agile & QA Process (Sprint planning, estimation, QA processes, continuous improvement, and cross-functional collaboration)
-Leadership & Stakeholder Management (Team mentoring, task planning, communication, issue escalation, and stakeholder coordination)
Mandatory Skills:
-QA & Test Strategy (Test planning, test case design, functional/API/regression testing, defect management, and QA processes) — Mandatory
-AI/ML & LLM Evaluation (Model validation, accuracy metrics, ground-truth comparison, LLM response evaluation, hallucination, relevance, and groundedness) — Mandatory
-Healthcare RCM & Audit (Medical coding validation, clinical documentation review, AR denial management, Prior Authorization and audit methodology etc) — Mandatory
-SQL & Data Analysis (Data validation, EDA, cleaning, aggregation, reconciliation, quality analysis, and model-result analysis) — Mandatory
-Statistical Sampling & Quality Decision-Making (Sampling methodology, statistical analysis, quality thresholds, risk assessment, and Go/No-Go decisions) — Mandatory