Senior Data & GenAI Engineer, ML, Azure Databricks / Medallion Architecture – Hybrid/Dallas, TX (Contract to Hire)
We are looking for a Senior Data & Gen AI Engineer to architect and build our next-generation data platform from scratch and no legacy refactoring. You set the patterns, frameworks, and architecture!
Work directly at the intersection of Azure Databricks Lakehouse engineering and Generative AI innovation. Your pipelines will directly power the AI capabilities defining the product roadmap.
Instead of maintaining legacy tech debt, you will have a clean slate to design scalable data pipelines within an Azure Databricks Medallion Architecture (Bronze, Silver, Gold). You will directly power enterprise Generative AI applications, RAG frameworks, and LLM pipelines, bridging the gap between raw data ingestion and cutting-edge AI systems.
Why should you apply here?
- A trailblazing leader, this company redefines what it means to set the bar for excellence and innovation.
- Fueled by over 800 passionate professionals, this company pulses with the energy and agility of a mid-sized powerhouse.
- Here, you’ll help build from the ground up; your ideas shape the future!
- Imagine a fast-growing tech team that combines startup spirit with the rock-solid foundation of an industry leader.
- Every initiative is built on cutting-edge automation and bulletproof security.
- Almost every project is a fresh opportunity, powered by the latest tools and resources to launch big ideas.
- IT champions open dialogue, seamless teamwork, and a culture where every voice is heard and everyone thrives.
- Your journey here is packed with chances to learn, grow, and rocket your career forward.
- Every contribution you make matters, and you’ll see your impact every day.
What will you be doing?
- Build data pipelines from the ground up for a generative AI data and machine learning platform.
- Work on greenfield pipeline engineering, including setting up and optimizing high-performance data pipelines from scratch using Azure Databricks to ingest both structured and unstructured data.
- Implement the Medallion Architecture by preparing data for flow through the Bronze, Silver, and Gold stores using Delta Lake.
- Be involved in the foundation design of generative AI and RAG by creating and providing datasets that are suitable for use with machine learning and for generating embeddings, as well as for LLM fine-tuning and RAG architectures, with the aim of achieving high-context, low-latency retrievability.
- Develop robust RESTful APIs so that processed data and feature layers can be served directly to Gen AI microservices and to downstream applications.
- Write extensive complex queries and models for customer and market segments.
- Prepare advanced SQL queries and PySpark jobs and optimize partitioning, indexing, and cluster configurations for both cost and execution speed.
- Develop and maintain ML-ready datasets and a Feature Store.
- Engineer robust ETL/ELT and automated workflows.
- Collaborate to ensure ML solutions deliver business impact.
- Create dashboards and models to turn ML insights into action.
What are the minimum qualifications?
- 5+ years of hands-on data engineering experience, with proven experience building pipelines from the ground up.
- Deep expertise designing end-to-end Lakehouse architectures (Bronze, Silver, Gold) using PySpark, Delta Lake, and Azure cloud services.
- Skilled in developing and maintaining dynamic ETL/ELT processes using cutting-edge tools like Azure Data Factory and Databricks.
- Practical experience preparing data for LLM applications, RAG pipelines, vector search, or embeddings management.
- Master-level ability to craft complex SQL queries, analytical functions, and data transformations alongside clean, modular Python.
- Experience designing, building, and consuming RESTful APIs to connect pipeline outputs with external services and applications.
What will make your résumé stand out?
- Skilled in building impactful statistical models that drive insights and decisions.
- Hands-on expertise with TensorFlow.
- Experience with Vector Databases (Pinecone, Qdrant, ChromaDB, or Azure AI Search).
- Hands-on experience with orchestration tools (Azure Data Factory, Databricks Workflows, Airflow).
- Familiarity with MLOps frameworks (MLflow, LangChain, LlamaIndex).
Location: Hybrid/Dallas, TX. Work onsite 2-3 days per week.
Salary: $80 – $140 per hour
Citizenship: U.S. Citizens or Permanent Residents
Employment Type: Contract-to-hire