Job Title: Senior Azure Databricks Engineer
Job Location: Dublin, Ireland (Onsite)
Job Type: Contract
Role Overview
We are looking for an experienced Azure Databricks Engineer with 7+ years of hands-on experience in data engineering to join our data platform team in Dublin.
The successful candidate will be responsible for designing, developing, and optimizing scalable enterprise data platforms and pipelines using Azure Databricks, Apache Spark, PySpark, SQL, Azure Data Lake Storage, and Azure Data Factory.
You will work closely with data architects, analysts, application teams, and business stakeholders to deliver high-quality data solutions supporting analytics, reporting, and AI/ML use cases.
Key Responsibilities
- Design and develop scalable data pipelines and data engineering solutions using Azure Databricks.
- Develop high-performance PySpark, Spark SQL, and Python applications for large-scale data processing.
- Build and maintain batch and near-real-time data pipelines.
- Implement Delta Lake based data architectures and medallion/lakehouse patterns.
- Develop data ingestion and transformation pipelines using Azure Data Factory (ADF) and Databricks.
- Work with Azure Data Lake Storage Gen2 (ADLS Gen2) for enterprise-scale data storage.
- Design and implement data models for analytical and reporting workloads.
- Implement data quality, validation, reconciliation, monitoring, and error-handling frameworks.
- Optimize Spark jobs, cluster configurations, SQL queries, and Delta tables for performance and cost.
- Implement data governance and access controls using Unity Catalog.
- Develop and manage Databricks workflows/jobs and production deployments.
- Implement CI/CD pipelines using Azure DevOps/GitHub Actions and Infrastructure as Code where applicable.
- Monitor production pipelines and troubleshoot performance, data quality, and availability issues.
- Collaborate with architects and engineering teams to establish data engineering standards and best practices.
- Mentor junior engineers and contribute to technical design and code reviews.
Required Skills & Experience
Azure Databricks
*7+ years of overall data engineering experience, with significant hands-on experience in *Azure Databricks.
- Strong experience with Apache Spark and PySpark.
- Strong proficiency in SQL and Python.
- Hands-on experience with Delta Lake and lakehouse architecture.
- Experience with Databricks Workflows/Jobs, notebooks, clusters, and job orchestration.
- Experience with Unity Catalog data governance, permissions, and secure data access.
- Strong understanding of Spark performance tuning and optimization.
Microsoft Azure
- Strong experience with Azure Data Lake Storage Gen2 (ADLS).
- Hands-on experience with Azure Data Factory.
- Experience with Azure Key Vault, Azure Monitor and other Azure data/platform services.
- Good understanding of Azure networking, security, IAM and enterprise cloud architecture.
Data Engineering
- Strong understanding of ETL/ELT architecture and data warehousing concepts.
- Experience implementing Medallion Architecture Bronze, Silver and Gold layers.
- Experience with batch and streaming data processing.
- Knowledge of data modelling, dimensional modelling, slowly changing dimensions and CDC.
- Experience working with large-scale datasets and distributed processing.
- Strong understanding of data quality, lineage, observability and governance.
DevOps & Engineering Practices
- Experience with Git-based development and CI/CD.
- Experience with Azure DevOps or GitHub Actions.
- Knowledge of automated testing and deployment practices.
- Experience with Infrastructure as Code such as Terraform is desirable.
- Strong understanding of Agile/Scrum delivery methodologies.
Desirable Skills
- Databricks certification such as Databricks Data Engineer Associate/Professional.
- Microsoft Azure certifications such as Azure Data Engineer Associate (DP-203) or equivalent.
- Experience with Kafka or Azure Event Hubs.
- Experience with Power BI and analytical data platforms.
- Experience with Microsoft Purview or enterprise data governance platforms.
- Experience with Terraform.
- Experience with Databricks REST APIs, CLI, SDKs or automation.
- Experience with machine learning/AI data pipelines is a plus.
Technical Skills
Must Have:
`Azure Databricks | PySpark | Apache Spark | Python | SQL | Delta Lake | ADLS Gen2 | Azure Data Factory | Unity Catalog | Data Engineering | ETL/ELT | Data Modelling | CI/CD | Git`
Good to Have:
`Azure DevOps | Terraform | Kafka | Azure Event Hubs | Power BI | Microsoft Purview | Databricks Workflows | Structured Streaming | Azure Monitor`
Education
- Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related discipline.
Key Competencies
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder-management skills.
- Ability to work independently and as part of a distributed engineering team.
- Strong ownership of production data platforms and pipelines.
- Ability to translate business requirements into robust technical solutions.
- Experience working in enterprise environments with high standards for security, quality, and reliability.