Coherent Pixel Systems is Hiring!
Consultant – Databricks Architect
Location: Fully Remote
Work Mode: Remote
Experience: 10–15 Years
Shift: 3:00 PM – 11:00 PM IST
Job DescriptionWe are looking for an experienced Databricks Architect with strong expertise in designing modern data platforms, data ingestion frameworks, and scalable lakehouse solutions. The ideal candidate should have deep experience with Databricks and enterprise ingestion tools, along with a good understanding of AI/ML and Generative AI capabilities within the Databricks ecosystem.
Responsibilities
- Design and implement scalable Databricks Lakehouse architectures.
- Define batch, streaming, CDC, and real-time data ingestion patterns.
- Architect data pipelines using Databricks and tools such as ADF, Kafka, Fivetran, Informatica, Airbyte, or similar platforms.
- Establish best practices for Delta Lake, Medallion Architecture, Unity Catalog, governance, security, and performance optimization.
- Provide technical leadership across data engineering and analytics teams.
- Design solutions supporting analytics, machine learning, and Generative AI/RAG use cases.
- Evaluate Databricks AI capabilities such as Mosaic AI, MLflow, Model Serving, and Vector Search.
- Define reusable architecture standards, frameworks, and implementation guidelines.
Requirements
- 10–15 years of industry experience in Data Engineering/Data Architecture with strong hands-on Databricks expertise.
- Strong knowledge of Apache Spark, PySpark, SQL, Python, Delta Lake, and Unity Catalog.
- Extensive experience with data ingestion, ETL/ELT, streaming, APIs, CDC, and integration tools.
- Experience with Azure, AWS, or GCP data platforms.
- Strong understanding of Lakehouse, Data Lake, Data Warehouse, and Medallion Architecture concepts.
- Knowledge of AI/ML, Generative AI, RAG, Vector Search, MLflow, and Databricks Mosaic AI.
- Strong architecture, stakeholder management, and technical leadership skills.
- Databricks and cloud certifications are preferred.
Work Location: In person