Overview
Join our innovative team as a Data Engineer and play a pivotal role in transforming raw data into actionable insights that drive strategic decision-making. In this dynamic role, you will design, develop, and maintain robust data pipelines and architectures that support large-scale data processing and analytics. Your expertise will empower our organization to harness the full potential of big data technologies, ensuring data quality, security, and accessibility across various platforms. This is an exciting opportunity for passionate professionals eager to make a tangible impact through cutting-edge data solutions.
Responsibilities
- Design, develop, and optimize scalable data pipelines using tools such as Apache Spark, Hadoop, and Informatica to handle vast datasets efficiently.
- Build and maintain data warehouses and databases utilizing Oracle, Microsoft SQL Server, and other relational database systems with a focus on effective database design principles.
- Implement ETL (Extract, Transform, Load) processes to consolidate data from diverse sources into centralized repositories for analysis.
- Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions leveraging SQL, Python, Java, Bash scripting, and shell scripting.
- Ensure data integrity, security, and compliance by establishing best practices for data governance across cloud environments like AWS.
- Develop automation scripts and workflows to streamline data processing tasks using VBA, Bash scripts, or other scripting languages.
- Monitor system performance, troubleshoot issues promptly, and optimize workflows for efficiency and reliability.
- Stay current with emerging big data technologies such as Apache Hive and Spark to continuously enhance our data infrastructure.
Requirements
- Proven experience in designing and managing large-scale data architectures using big data tools like Hadoop, Spark, or Hive.
- Strong proficiency in programming languages including Python and Java for developing scalable data solutions.
- Hands-on experience with cloud platforms such as AWS for deploying and managing data services.
- Solid understanding of relational databases including Oracle and Microsoft SQL Server; expertise in database design is essential.
- Familiarity with ETL tools like Informatica or similar platforms to facilitate seamless data integration processes.
- Knowledge of SQL scripting for querying and managing databases effectively.
- Ability to perform analysis using various tools; strong analytical skills are critical for interpreting complex datasets.
- Experience with Unix/Linux shell scripting (Bash) for automation tasks is highly desirable.
- Excellent problem-solving skills with a keen eye for detail to ensure high-quality data outputs.
- Effective communication skills to collaborate with technical teams and translate business needs into technical specifications.
Embark on a journey where your technical prowess fuels innovation! We’re committed to fostering an environment where your skills grow alongside exciting challenges—empowering you to shape the future of our data landscape while enjoying comprehensive benefits designed to support your well-being at every stage of your career.
Pay: £40,000.00-£50,000.00 per year
Work Location: In person