Job Overview
We are looking for a skilled Azure Databricks & GenAI Engineer to design, develop, and maintain scalable data and AI solutions on Microsoft Azure. The ideal candidate will have strong hands-on experience with Azure Databricks, PySpark, Python, SQL, data engineering, Generative AI, and LLM-based applications.
The candidate will work on building modern data pipelines, lakehouse architectures, AI/ML workflows, and GenAI solutions that leverage enterprise data.
Key Responsibilities
- Design and develop scalable data engineering solutions using Azure Databricks.
- Build and optimize data pipelines using PySpark, Python, SQL, and Delta Lake.
- Develop and maintain Databricks notebooks, workflows, jobs, and Delta tables.
- Implement Medallion Architecture (Bronze, Silver, Gold) for data processing and transformation.
- Develop Generative AI solutions using Large Language Models (LLMs), embeddings, and vector search.
- Build RAG (Retrieval-Augmented Generation) applications using enterprise data.
- Integrate Azure AI services and Azure OpenAI with Databricks-based solutions.
- Develop data ingestion and transformation pipelines using Azure Data Factory (ADF) and other Azure services.
- Implement data preprocessing, feature engineering, and model-ready datasets for AI/ML applications.
- Work with MLflow for experiment tracking, model management, and deployment.
- Optimize Databricks workloads, Spark jobs, SQL queries, and data pipelines for performance and cost.
- Implement data quality, governance, security, and access controls across data platforms.
- Collaborate with Data Scientists, AI Engineers, Software Developers, and business stakeholders.
- Develop CI/CD pipelines for data and AI workloads using Azure DevOps or GitHub Actions.
- Troubleshoot production data and GenAI solutions and ensure high availability and reliability.
Required Skills
- Strong hands-on experience with Azure Databricks.
- Strong knowledge of Python, PySpark, and SQL.
- Experience with Delta Lake and Lakehouse Architecture.
- Practical experience with Generative AI and LLM applications.
- Knowledge of RAG, embeddings, vector databases/vector search, and prompt engineering.
- Experience with Azure OpenAI or equivalent LLM platforms.
- Experience with Azure Data Factory (ADF).
- Knowledge of MLflow and MLOps concepts.
- Understanding of data modeling, ETL/ELT, and data warehousing concepts.
- Experience with Git and CI/CD practices.
- Strong analytical and problem-solving skills.
Preferred Skills
- Experience with Azure AI Search and vector search.
- Experience developing AI applications using frameworks such as LangChain or LlamaIndex.
- Knowledge of Databricks Mosaic AI and Model Serving.
- Experience with Unity Catalog and Databricks governance.
- Knowledge of REST APIs and microservices.
- Experience with Azure services such as ADLS Gen2, Azure Functions, Key Vault, and Event Hubs.
- Understanding of responsible AI, data privacy, and enterprise AI security.
Work Location: Remote