Key Responsibilities
Run and Monitor Inferences
- Execute batch inference jobs on Databricks and other environments.
- Troubleshoot inference issues, log results, and escalate model anomalies.
Pipeline & Evaluation Support
- Assist with running DSPy pipelines and loading custom models/configs.
- Support evaluation runs using internal metrics
- Track and summarize performance metrics for inference experiments.
Data Handling & Experimentation
- Help prepare and clean legal text datasets for inference and evaluation.
- Annotate outputs to support error analysis and benchmarking.
- Maintain reproducible scripts/notebooks for inference runs.
Qualifications
- Background in Computer Science, Data Science, or AI/ML (student or recent grad is good).
- Familiarity with Python and libraries like pandas, scikit-learn, or PyTorch.
- Exposure to LLM frameworks (DSPy, LangChain, Hugging Face, or equivalent).
- Comfortable working with Databricks / MLflow or other cloud ML environments.
- Detail-oriented with strong organizational skills for tracking experiments.
Evaluation:
- Technical Skills - 50%
- LLM frameworks - 25%
- Databricks/MLflow - 25%
Must have:
- must have experience Azure data tools - azure, databricks, MLflow, python
- Exposure to LLM frameworks (DSPy, LangChain, Hugging Face, or equivalent).
- Familiarity with Python and libraries like pandas, scikit-learn, or PyTorch.,