Lead AWS Data Engineer
Experience: 8 Years
Location: Hyderabad(On-Site)
Notice Period: Immediate Joiner Only
Role Overview
We are looking for a Lead AWS Data Engineer with 8+ years of experience to lead the design and development of scalable, reliable data pipelines on AWS. The candidate should be highly hands-on with PySpark, SQL, Terraform, AWS data services, and Lakehouse/Medallion architecture.
Key Responsibilities
- Lead the design and development of scalable AWS data pipelines using PySpark/Spark, Python, and SQL.
- Implement Bronze/Silver/Gold (Medallion) architecture.
- Design incremental processing, CDC, SCD Type 2, Merge/Upsert, and historical data solutions.
- Handle deduplication, late-arriving/out-of-sequence data, updates, and deletes.
- Build idempotent, resilient, and re-processable pipelines.
- Provision and manage AWS infrastructure using Terraform.
- Work with S3, Glue, EMR, Lambda, Step Functions, and related AWS services.
- Drive data quality, performance optimization, monitoring, and production support.
- Provide technical leadership, conduct code/design reviews, and mentor data engineers.
Required Skills
- 8+ years of Data Engineering experience, with data engineering.
- Strong PySpark/Spark, Python, and Advanced SQL.
- Strong AWS data engineering experience.
- Hands-on Terraform / Infrastructure as Code.
- Strong knowledge of Lakehouse/Medallion architecture.
- Experience with CDC, SCD Type 2, incremental processing, deduplication, and historical processing.
- Strong understanding of idempotency, merge keys, ordering columns, event time vs. ingestion time, and reprocessing.
Good communication, leadership, and problem-solving skills.