Senior Data Engineer – Private Markets Data
Location: Boston, MA
About the Opportunity
A leading global investment management firm is seeking a highly motivated Senior Data Engineer to join its Private Markets Data team. This individual will be responsible for designing, implementing, and maintaining sophisticated data engineering systems and governance processes while helping develop next-generation data solutions.
The ideal candidate is a well-rounded technologist with strong data engineering expertise, hands-on programming experience, and a passion for building scalable, high-performance data platforms.
This role works closely with data product managers, business analysts, technology teams, and business stakeholders and provides significant exposure to the investment management process.
Key Responsibilities
- Design, develop, and maintain on-premises and cloud-based data ingestion and processing pipelines.
- Administer and enhance data engineering tools including Airflow, DBT Cloud, Snowflake, and Python.
- Support and enhance data governance processes and data quality initiatives.
- Develop production-quality code and scalable data solutions supporting key business initiatives.
- Conduct architecture and code reviews to ensure security, scalability, performance, and quality.
- Partner with cloud migration, information security, and business analysis teams to develop new applications and migrate existing systems to cloud-native technologies.
- Provide technical leadership and mentorship to onshore and offshore engineering team members.
- Help define and execute the technical roadmap while identifying opportunities to improve efficiency, processes, and data quality.
- Optimize data pipelines and improve overall job performance and runtime.
Technology & Technical Experience
The ideal candidate will bring experience across a modern data engineering environment, including:
- Python for data processing, automation, and production-grade development.
- Advanced SQL, data modeling, and data warehouse design.
- Snowflake, including experience building, deploying, and optimizing Snowflake Cortex AI.
- Data orchestration and transformation tools such as Airflow and DBT Cloud.
- Cloud-native AWS serverless data and integration technologies.
- Docker and Kubernetes.
- Deployment and infrastructure tools including Terraform, GitHub Actions, and ECR.
- Monitoring and observability platforms such as Datadog, Grafana, or Amazon CloudWatch.
- Data platforms including Snowflake, Redshift, BigQuery, or Athena.
- BI/reporting tools such as Power BI or Tableau.
- Data governance, data quality, metadata management, and lineage frameworks.
- Web scraping, APIs, and data integration best practices.
- Agile software development environments.
AI & Emerging Technology Experience
Candidates should be comfortable incorporating modern AI technologies into their day-to-day engineering workflow, including:
- Large language models such as Claude and ChatGPT for data analysis and automation.
- Prompt engineering and agentic/workflow automation frameworks.
- AI-assisted development tools such as Claude Code, OpenAI Codex, Cursor, or Windsurf.
- Familiarity with Model Context Protocol (MCP) and reusable AI skills is a plus.
- Exposure to vector databases and Retrieval-Augmented Generation (RAG) pipelines such as Pinecone, pgvector, or Snowflake Cortex Search is a plus.
- Modern lakehouse technologies such as Delta Lake, Apache Iceberg, or Apache Hudi are a plus.
- Data quality and observability tools such as Great Expectations, Monte Carlo, or Soda are a plus.
- Data catalog and lineage platforms such as Alation, Collibra, or Unity Catalog are a plus.
- Streaming and real-time processing technologies such as Kafka, Flink, or Spark Streaming are a plus.
Qualifications
- BS or MS in Computer Science, Engineering, or a related technical discipline.
- 7+ years of professional software or data engineering experience.
- Strong foundation in software design and architectural patterns.
- Proven ability to translate complex business requirements into scalable technical solutions.
- Excellent communication, analytical, and problem-solving skills.
- Experience with Big Data and streaming technologies preferred.
- Demonstrated ability to take ownership of projects from concept through completion.
- Ability to collaborate effectively with both technical and non-technical stakeholders.
- Self-starter who thrives in a fast-paced, highly collaborative environment.
- Strong commitment to continuous learning and emerging technologies.