Data Engineering – UK
Location: United Kingdom
Work Arrangement: Remote
We are hiring on behalf of one of our clients for a Data Engineering role focused on designing, developing, and maintaining reliable data infrastructure, pipelines, and scalable data solutions.
Key Responsibilities
- Design, develop, and maintain scalable and reliable data pipelines for batch and real-time data processing.
- Build and optimise ETL and ELT workflows for collecting, transforming, and loading data from multiple sources.
- Develop data integration solutions to support business intelligence, analytics, reporting, and operational requirements.
- Work with structured, semi-structured, and unstructured datasets across multiple data sources.
- Design and implement data ingestion frameworks for internal and external data sources.
- Develop and maintain data warehouses, data lakes, lakehouses, and other modern data storage solutions.
- Create efficient data models and schemas to support analytical and operational use cases.
- Perform data extraction, transformation, validation, cleansing, and quality checks.
- Implement automated processes for data validation, monitoring, and pipeline management.
- Monitor data pipelines and proactively identify and resolve performance, integration, and data quality issues.
- Optimise queries, data pipelines, and processing workflows to improve performance and resource utilisation.
- Develop reusable data engineering components and frameworks to improve efficiency and scalability.
- Integrate data from APIs, databases, cloud platforms, applications, and third-party systems.
- Collaborate with Data Scientists, Data Analysts, Software Engineers, DevOps teams, and business stakeholders to understand data requirements.
- Translate business and technical requirements into effective data engineering solutions.
- Implement appropriate data governance, security, privacy, and access-control practices.
- Support data lineage, metadata management, and documentation initiatives.
- Participate in the design and implementation of cloud-based data architecture.
- Automate repetitive data engineering and operational processes wherever appropriate.
- Contribute to testing, deployment, and continuous improvement of data pipelines and data platforms.
- Maintain clear technical documentation covering data models, pipelines, workflows, integrations, and infrastructure.
- Stay informed about emerging data engineering technologies, cloud platforms, and modern data architecture practices.
Required Qualifications and Skills
- Degree or equivalent qualification in Computer Science, Data Engineering, Information Technology, or a related field.
- Strong understanding of data structures, databases, data processing, and data architecture concepts.
- Proficiency in SQL and experience working with relational databases.
- Strong programming skills in Python, Java, Scala, or another programming language commonly used in data engineering.
- Understanding of ETL and ELT processes and data pipeline architecture.
- Knowledge of data warehousing, data modelling, and database design principles.
- Strong problem-solving, analytical, and troubleshooting skills.
- Understanding of data quality, validation, governance, and security principles.
- Ability to work with large and complex datasets.
- Strong communication and collaboration skills.
- Ability to work independently while contributing effectively within cross-functional teams.
Preferred Skills
- Experience with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
- Hands-on experience with Apache Spark, Databricks, Kafka, Apache Airflow, or similar technologies.
- Experience working with cloud-based data warehouses such as Snowflake, BigQuery, Amazon Redshift, or Azure Synapse.
- Knowledge of modern data lake, lakehouse, and distributed data processing architectures.
- Experience designing and implementing batch and real-time data processing solutions.
- Familiarity with streaming technologies and event-driven data architectures.
- Experience working with REST APIs, web services, and third-party data integrations.
- Knowledge of Docker, Kubernetes, or other containerisation technologies.
- Familiarity with CI/CD practices and DevOps methodologies.
- Experience using Git or other version-control systems.
- Understanding of infrastructure-as-code tools and cloud deployment practices.
- Knowledge of data orchestration and workflow management tools.
- Experience implementing data quality monitoring and observability solutions.
- Familiarity with data governance, data cataloguing, metadata management, and data lineage.
- Understanding of security and compliance considerations within data platforms.
- Experience with automated testing for data pipelines and data transformation processes.
- Knowledge of machine learning data pipelines and feature engineering workflows is an advantage.
- Exposure to Agile development methodologies and collaborative software engineering practices.
- Ability to evaluate new technologies and recommend appropriate tools and approaches for data engineering requirements.
What We Offer
- Remote working opportunity within the UK.
- Opportunity to work on real-world data engineering projects.
- Exposure to modern data platforms, cloud technologies, and data infrastructure.
- Collaborative and professional working environment.
- Opportunity to work across data engineering, cloud, analytics, and technology functions.