Summary
✨ AI‑Generated
A technology services organization is seeking a Senior Data Engineer to design scalable data solutions and lead migrations to a modern cloud data platform. The role covers pipelines, ETL/ELT, Spark-based processing, performance tuning, CI/CD, cloud infrastructure, architecture, stakeholder collaboration, and emerging generative AI capabilities.
Highlights
Lead sophisticated data engineering and platform migration initiatives, work across major cloud environments, provide architectural guidance, and explore emerging generative AI capabilities within a modern data ecosystem.
Description
Key Responsibilities
● Design, develop, and implement scalable data engineering solutions using Databricks,
Apache Spark, PySpark, and SQL.
● Lead and execute legacy/non-Databricks platform migration projects to Databricks.
● Develop and optimize data pipelines, ETL/ELT workflows, and data processing
frameworks.
● Apply Databricks architecture, clustering techniques, performance optimization,
and workload tuning best practices.
● Work with cloud platforms including Azure, AWS, or GCP.
● Implement and maintain CI/CD pipelines for data engineering and Databricks
workloads.
● Collaborate with clients and stakeholders to understand requirements and translate them
into technical solutions.
● Provide technical guidance, architecture recommendations, and mentorship to other data
engineers.
● Evaluate and improve existing data platforms, pipelines, and processing frameworks.
● Work with emerging GenAI capabilities within the Databricks ecosystem, including
Databricks Genie and Databricks Apps where applicable.
● Troubleshoot complex data processing and performance-related issues.
● Ensure data solutions meet standards for scalability, reliability, security, and
maintainability.
Required Qualifications & Experience
Senior Data Engineer
● 7+ years of overall Data Engineering experience.
● 3+ years of consulting experience.
● Successfully delivered 5+ projects using Databricks.
● Strong hands-on experience with Databricks and Apache Spark.
● Mandatory experience in legacy/non-Databricks platform to Databricks migration
projects.
● Deep understanding of Databricks architecture, clustering techniques, performance
optimization, and workload tuning.
● Strong proficiency in PySpark and SQL.
● Experience with at least one major cloud platform: Azure, AWS, or GCP.
● Experience with CI/CD and DevOps practices.
● Strong client-facing, communication, problem-solving, and leadership skills.
Technical Skills
Must Have
● Databricks
● Apache Spark
● PySpark
● SQL
● Legacy-to-Databricks Migration
● Databricks architecture and optimization
● Clustering and performance tuning
● Cloud platforms – Azure / AWS / GCP
● CI/CD
Good to Have
● Databricks Genie
● Databricks Apps
● GenAI
● Advanced Databricks ecosystem knowledge
Behavioral & Leadership Competencies
● Strong consulting and client-facing skills.
● Excellent analytical and problem-solving abilities.
● Ability to independently own and deliver complex projects.
● Strong communication and stakeholder management skills.
● Ability to lead technical discussions and provide architecture recommendations.
● Mentoring and knowledge-sharing capabilities.
● Adaptability to new technologies and evolving client requirements.
