Key Responsibilities
- Implement and operationalize enterprise-scale Lakehouse platforms, data products, and data marketplace capabilities.
- Design, develop, test, and maintain scalable batch, streaming, CDC, and API-based data ingestion pipelines.
- Develop multimodal data ingestion pipelines, including:
o Content extraction from various file formats.
o Regex-based extraction of specific fields.
o Content extraction from embedded images.
o Frame extraction from video files.
o Transcript extraction from audio files.
- Build and maintain foundation and business data products with defined data contracts, SLAs, data quality controls, and governance standards.
- Implement and work with modern open table formats, including Iceberg, Hudi, and Delta Lake.
- Develop and support data pipelines for RAG, vector search, Generative AI (GenAI), and agentic AI use cases.
- Perform performance tuning, optimization, production support, troubleshooting, and root cause analysis across data platforms and pipelines.
- Implement data ingestion, transformation, reconciliation, and data quality frameworks.
- Develop data architectures supporting NLP and AI-driven analytics, including the ingestion, curation, governance, and management of structured and unstructured data.
- Support ML platforms and workflows, including model development, deployment, and operationalization.
- Develop internal engineering tools and full-stack applications using Python, shell scripting, and modern web frameworks.
- Expose and integrate data through APIs, event streams, dashboards, and BI platforms.
- Create and maintain technical documentation, deployment guides, and
- operational runbooks.
- Ensure compliance with engineering standards, DevSecOps controls, CI/CD
- practices, security requirements, and software delivery standards.
- Collaborate effectively with distributed engineering, data, technology, and
- business teams across multiple projects.
- Drive continuous improvement in data platform reliability, scalability,
- automation, and operational excellence.
Key Requirements
- Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related discipline.
- 8–12 years of experience in Data Engineering, Big Data, Data Lake, Lakehouse, or large-scale data platform implementations.
- Strong hands-on experience with enterprise data platforms such as Databricks, Snowflake, Cloudera, Azure, AWS, GCP, Huawei, or Alibaba Cloud.
- Hands-on experience designing and developing data products and data marketplace capabilities.
- Strong expertise in Spark, PySpark, SQL, Python, and/or Scala, with strong overall programming skills.
- Proven experience building scalable data ingestion, transformation, streaming, CDC, reconciliation, and data quality pipelines.
- Hands-on experience with Iceberg, Hudi, Delta Lake, and object storage platforms.
- Experience with modern data technologies such as Kafka, Flink, Spark Streaming, Airflow, Trino, Dremio, Hive, and/or Impala.
- Strong hands-on experience with Kubernetes, OpenShift, Docker, Terraform, Jenkins, Git, CI/CD, MLflow, and observability tools.
- Experience designing data architectures and pipelines supporting NLP, AI-driven analytics, RAG, vector search, GenAI, and agentic AI use cases.
- Experience handling and governing unstructured and multimodal data, including documents, images, audio, and video.
- Experience with ML platforms and frameworks such as CML, Spark MLlib, scikitlearn, and XGBoost, including model deployment. Strong knowledge of data modelling, metadata management, data lineage, data governance, and data quality.
- Experience exposing data through APIs, event streams, dashboards, and BI platforms.
- Experience with full-stack/internal engineering tool development using Python, shell scripting, Flask, React, or similar technologies is advantageous.
- Knowledge or experience with Teradata, Netezza, Greenplum, or MPP migration programmes is advantageous.
- Strong engineering, automation, troubleshooting, and performance optimization mindset.
- Strong communication and stakeholder management skills, with the ability to work effectively across distributed teams and multiple projects.
- Experience working in Agile delivery environments and enterprise-scale technology platforms.
- Strong commitment to quality, operational excellence, automation, and continuous improvement.
- Relevant certifications such as Databricks Certified Data Engineer, Azure Data Engineer Associate, AWS Data Analytics Specialty, Google Professional Data Engineer, SnowPro, or DAMA CDMP would be an advantage.