Job Overview
We are seeking a highly skilled and energetic Sr. Databricks Engineer to lead the design, development, and optimization of large-scale data systems within a cloud-based environment. In this pivotal role, you will harness your expertise in big data technologies, cloud platforms, and software development to build robust data pipelines, enable advanced analytics, and support business intelligence initiatives. Your innovative approach will drive data-driven decision-making and empower our organization to unlock the full potential of its data assets.
Duties
- Develop, implement, and maintain scalable ETL (Extract, Transform, Load) pipelines using Databricks, Spark, and other big data tools to process vast datasets efficiently.
- Design and optimize data models and schemas for data warehouses, ensuring high performance and scalability aligned with dimensional modeling principles.
- Collaborate with cross-functional teams to integrate diverse data sources such as AWS cloud services, Azure Data Lake, Hadoop ecosystems, and SQL databases like Oracle and Microsoft SQL Server.
- Build and manage data workflows utilizing Apache Hive, Informatica, Talend, and RESTful APIs to facilitate seamless data movement and transformation.
- Implement best practices for data management integration, query management, and database design across cloud databases and on-premises systems.
- Support model training, analysis efforts, and business intelligence tools like Looker by providing reliable data infrastructure and access.
- Participate actively in Agile development cycles to deliver continuous improvements in big data systems while adhering to security standards and compliance requirements.
Skills
- Extensive experience with cloud platforms such as AWS, Azure (including Data Lake), and Public Cloud environments.
- Strong proficiency in programming languages including Java and Python for software development and automation tasks.
- Deep understanding of big data systems like Hadoop, Spark, Apache Hive, and related ecosystem components.
- Proven expertise in data warehousing design, dimensional modeling, ETL pipeline development, and query management using SQL databases such as Oracle or SQL Server.
- Familiarity with business intelligence tools like Looker for visual analytics and reporting purposes.
- Knowledge of data modeling concepts including linked data principles for semantic interoperability across datasets.
- Skilled in shell scripting (Bash/Unix shell) for automation tasks within Linux environments.
- Experience with data engineering practices including RESTful API integration, model training support, and analysis skills to derive insights from complex datasets.
- Ability to work effectively within Agile teams while managing multiple priorities in a fast-paced environment.
Join us as a Sr. Databricks Engineer to shape the future of our enterprise data landscape! Bring your passion for innovative big data solutions—your expertise will fuel smarter decisions across our organization while advancing your career in a dynamic setting that values your technical mastery!
Pay: $101,556.33 - $122,304.39 per year
Work Location: In person