Company Description The organization is in the process of preparing a formal announcement and will share more details soon. At this stage, the company, backed with private funding is focused on building a strong foundation for advanced AI initiatives and assembling a core technical team. Prospective applicants can expect to join at an early, formative phase, with significant opportunity to influence both technology direction and product vision. The role will be based in Indiranagar, Bengaluru, a growing hub for technology and innovation. Further information about the company’s mission, products, and culture will be communicated as the announcement is made.
- Role Description
- As a Lead AI Engineer, you will architect, design, and implement AI solutions that power core products and internal platforms in a full-time, on-site role.
- You own the generation stack (RAG, prompting, model selection, orchestration) and the voice path (ASR tuning, endpointing, TTS output)
- More details on role and responsibilities below;
- Experience
- 2+ years building LLM systems in production, serving real user traffic.
- You have built an evaluation harness that caught a regression before it reached users, and then used it to improve the system. This is the primary assessment topic.
- Strong backend engineering: you own the service, the queue, p95 latency, and on-call for the AI path.
- Comfortable blocking a release, with the data to support the call.
- Compensation
- Competitive with market benchmarks (Cash + bonus + equity)
Response Quality Ownership
Measure:
- Build the golden-set regression suite from production transcripts: human-labelled, stratified by skill, intent, language, and user cohort. Scored on groundedness, numeric fidelity, compliance, persona adherence, and refusal correctness.
- Maintain per-skill compliance rubrics, instead of a single aggregate score. RBI fair practice norms for credit and lending, IRDAI norms for insurance, CICRA constraints on bureau data. An aggregate pass rate across skills hides violations in the smallest one.
- Implement LLM-as-judge grading calibrated against human labels. Report per-axis agreement (Cohen's κ), re-validate on every judge-model or judge-prompt change, maintain a running human spot-audit sample.
- Score sampled production traffic continuously. Slice by skill, cohort and language. Alert on output score drift and input distribution drift.
- Build the red-team suite: prompt injection through user-supplied fields (name, transaction memo), jailbreaks that elicit specific rate, premium or eligibility commitments, cross-skill leakage, multi-turn escalation, PII exfiltration.
Improve:
- Maintain an error taxonomy over failed cases and work the top categories: retrieval misses, misrouting, prompt ambiguity, model selection, template gaps. Every fix lands with a regression case.
- Own prompt and model versioning, offline→shadow→canary rollout, and rollback.
- Run the regression suite as a required status check on the release branch. You define the thresholds per axis and per skill, what a failure blocks, and the exemption path.
Grounded Generation
- Enforce the separation between deterministic systems and the generative layer across every skill: the credit engine emits Reason Codes, pricing services emit APR and eligibility, the product catalogue emits terms and premiums, the policy corpus supplies rules. The LLM renders these values and never originates one.
- Build the post-generation verifier: extract every numeral, currency amount, rate, date, tenure and product name from model output, assert membership in that turn's source payload, block or regenerate on mismatch.
- Add claim-level entailment (NLI) checks for non-numeric statements against retrieved context - recommendations and comparisons, where the failure is a wrong claim rather than a wrong number.
- Track verifier block rate as a generator-quality signal, not just a safety net. Block rate should fall because generation improved; falling because thresholds loosened is a regression, and the metric should be built so the two are distinguishable.
- Own retrieval quality as a measured system: labelled query→document sets per corpus (Reason Codes, product catalogue, policy documents), recall@k and MRR tracked per release.
- Implement abstention and human handoff as designed outputs with their own eval criteria.
Voice Experience Quality
Voice is where most of Rahul's UX risk sits, and it is the part conventional NLP metrics do not capture.
Measure:
- Turn-taking: false-endpoint rate (Rahul cutting the user off mid-sentence), late-endpoint delay (p50/p95 silence before Rahul starts), barge-in success rate, and TTS stop latency once the user speaks over it.
- Recognition: WER on code-switched Hinglish, and separately entity error rate on amounts, rates, dates, tenures and product names. These are the errors that change meaning - a 2% WER that transcribes ₹2,000 as ₹20,000 is a product failure that aggregate WER will hide.
- Synthesis: automated MOS estimation (UTMOS / NISQA / DNSMOS or equivalent) on sampled output, plus periodic human listening panels. A pronunciation regression set covering ₹ amounts, percentages, dates, tenures in months and years, English loanwords in Hindi prosody, and bank, insurer and product names.
- Conversation-level: repeat and rephrase rate (the user re-asking the same thing is the clearest signal something was not understood or not clear), task completion rate over voice versus the same intent in text, mid-conversation abandonment, and escalation-to-human rate.
- Degradation: measured behaviour under packet loss and on 2G/3G connections, not assumed.
Improve:
- Tune VAD and endpointing against the explicit tradeoff between false cuts and response delay. Both are user-visible; the balance point is an empirical question per language.
- Build and maintain the pronunciation lexicon and SSML rules for financial vocabulary across all skills. Regression-test them like code.
- Bring time-to-first-audio down against the budget: streaming at every stage, speculative first-token generation, prefetching likely payloads during ASR.
- A 200 ms end-to-end response is not achievable with ASR → routing → retrieval → backend call → generation → TTS in the loop. 200 ms for barge-in stop latency is achievable, and matters more for how responsive Rahul feels. Targets below reflect that.
Multi-Skill Orchestration
- Build intent routing across Rahul's skills: credit coaching, loan application, insurance, credit card discovery, savings. Routing accuracy is measured per skill against a labelled set, not tuned by inspection.
- Weight misroutes by cost, not just count. Routing a user in debt distress into card discovery is a different class of error from the reverse, and a single accuracy number treats them as equal.
- Handle mid-conversation skill handoff and context carry, and measure it - a user moving from "why is my score down" to "should I take this loan" is one conversation, not two sessions.
- Implement the Text-to-SQL layer over user financial data, with query sandboxing and result validation before any generated summary. Measured on execution accuracy against a labelled query set.
- Integrate with the Ventile Credit Model (CDSO's team) so explanations are anchored to actual model outputs rather than inferred.
Explicitly Not In Scope
- Fraud and risk modelling - device scoring, liveness detection, VPN flagging, account-takeover guardrails. Separate discipline, separate hire.
- Fine-tuning as a starting point. Run it when it wins on your evals. Prompting and retrieval are the baseline it has to beat.
- On-device / SLM inference in year one. Roadmapped, and you will shape it, but not the first-year problem.
First 90 Days
- Day 30 - 300-case golden set built and human-labelled, stratified across skills and covering both text and voice. Current pipeline baselined against it. Voice metrics instrumented end to end so turn-taking and entity error rate are visible for the first time. Results published internally.
- Day 60 - Judge calibration at ≥0.8 agreement with human labels on groundedness and compliance. Verifier deployed in the request path. Regression suite running as a required check on the release branch, with per-skill thresholds.
- Day 90 - Production traffic sampled and scored on a dashboard shared by engineering and compliance, sliced by skill. First improvement cycle closed: top three error categories identified, fixed, and shown to have moved the metric. A documented, auditable answer to "how do we know Rahul does not state an incorrect rate, premium or eligibility?"
Ideal Background
- Engineers from voice or conversational AI teams who have run ASR/TTS or NLP pipelines in production.
- ML platform and backend engineers at consumer fintech who have owned model serving through to the API layer.
- Early engineers at AI startups who shipped LLM products end to end and built their own eval tooling.
We do not score publications, competition rankings, or framework preference. You should be able to justify using - or not using - LangChain, LlamaIndex, or CrewAI.
Interview Process
- Technical discussion: a production system you built, its failure modes after launch, and what you changed as a result.
- Design exercise: given a spoken user question that could plausibly route to two different skills, design the routing, grounding, verification and evaluation path.
- Written exercise: review 10 real Rahul voice sessions across different skills. Classify the failures, propose what to fix first and what to gate on.
No timed algorithm rounds.