Title: Senior Data Quality Engineer
Location: Chicago, IL
Job Description:
- Data Quality Engineering & Frameworks
- Design and implement enterprise-wide data quality frameworks aligned to Lakehouse architecture (bronze, silver, gold layers)
- Define and enforce data quality rules including completeness, accuracy, consistency, timeliness, and validity
- Develop reusable data validation, reconciliation, and monitoring patterns within Databricks pipelines
- Establish automated data quality checks embedded within ELT/ETL workflows
Databricks & Pipeline Integration
- Integrate data quality controls directly into Databricks (Spark/Delta Lake) pipelines and workflows
- Develop scalable validation processes for batch and event-driven ingestion pipelines
- Partner with Data Engineers to ensure quality gates are enforced across ingestion, transformation, and consumption layers
- Optimize data quality processes for performance and scalability within large distributed datasets
Monitoring, Observability & Issue Management
- Implement and manage data observability frameworks, including metrics, alerts, and dashboards
- Monitor data pipelines and proactively identify anomalies, failures, and quality degradation
- Lead root cause analysis (RCA) efforts for data quality issues and drive remediation
- Develop and maintain quality scorecards and reporting for stakeholders
Data Governance & Compliance
- Ensure adherence to enterprise data governance standards, including metadata, lineage, and auditability
- Partner with Data Governance teams (e.g., Collibra) to align data definitions, ownership, and controls
- Support regulatory requirements (e.g., SOX, GLBA, data integrity standards) through auditable data quality controls
- Define and enforce data quality SLAs and data contracts across domains
Automation & DevOps
- Implement CI/CD practices for data quality rules, validations, and monitoring
- Automate testing frameworks for validating data transformations and pipelines
- Develop reusable libraries and frameworks for enterprise-scale data quality enforcement
Collaboration & Leadership
- Partner with Data Engineers, Data Architects, BI teams, and business stakeholders to embed quality-by-design principles
- Provide technical leadership and mentorship on data quality best practices
- Act as a subject matter expert (SME) for data quality across the organization
- Drive continuous improvement and innovation in data quality tooling and methodologies
Required Qualifications
- 7+ years of experience in data engineering, data quality engineering, or related roles
- Strong hands-on experience with Databricks, Spark (PySpark), and Delta Lake
- Proven experience implementing data quality frameworks and controls in modern data platforms
- Advanced SQL and data profiling/validation skills
- Experience working with large-scale datasets in cloud environments (AWS or Azure)
- Experience integrating data quality into ELT/ETL pipelines and orchestration tools
- Strong understanding of data governance and data lifecycle management
Preferred Qualifications
- Experience in financial services or regulated environments
- Familiarity with data governance tools (e.g., Collibra)
- Experience with data observability or quality tooling (e.g., Monte Carlo, Great Expectations, Deequ, or similar)
- Experience with real-time data quality validation (streaming pipelines)
- Knowledge of regulatory reporting and data controls frameworks
- Cloud or Databricks certifications
Technical Skills
- Databricks (Lakehouse, Unity Catalog, workflows)
- Spark / PySpark
- SQL (advanced)
- Delta Lake
- Data quality frameworks (rule engines, validation patterns)
- Data observability and monitoring
- Cloud platforms (AWS or Azure)
- Orchestration tools (Airflow, Control-M)
- APIs and data integration
- CI/CD and DevOps
- Data modelling and lineage concepts
Professional Competencies
- Strong analytical and problem-solving skills with a focus on data integrity
- High attention to detail and commitment to data accuracy
- Strong communication skills across technical and non-technical stakeholders
- Ability to influence standards and drive enterprise adoption
- Collaborative mindset with a focus on continuous improvement