Databricks Data Engineer

We are looking for an experienced Azure Data Engineer to join a long-term data engineering programme based in London. You will be responsible for designing, developing and optimising scalable data solutions using Azure Databricks, Azure Data Factory and Scala. The role will involve processing large-scale datasets, building reliable ETL pipelines and working closely with technical and business stakeholders to translate requirements into effective data solutions. This is a hands-on engineering position suited to someone with strong Azure data platform experience and a proven background delivering high-performance data pipelines within complex enterprise environments. Contract Information Job title: Databricks Data EngineerLocation: LondonWork model: Hybrid (3 days on-site)Start date: ASAPContract length: 12 MonthsContract type: Inside IR35 Key Responsibilities Design and develop scalable data engineering solutions using Azure Databricks and Azure Data Factory.Build and maintain complex ETL and data transformation pipelines.Process and transform large datasets containing millions of records.Optimise data pipelines and distributed processing workloads for performance, reliability and scalability.Participate in customer discussions, requirement-gathering sessions and technical solution design.Translate business and technical requirements into practical data engineering solutions.Develop clean, maintainable and reusable code in line with engineering best practices.Perform code reviews, performance tuning and technical troubleshooting.Monitor delivery progress and communicate project updates to stakeholders.Identify and escalate technical risks, blockers and critical issues.Collaborate with engineers, architects and wider project teams within an Agile delivery environment. Essential Skills and Experience 6–8 years of commercial experience within data engineering or data software engineering.Strong hands-on experience developing solutions using Azure Databricks.Proven experience processing and transforming large-scale datasets.Strong working knowledge of Azure Data Factory, including the development of complex ETL workflows.Experience optimising data pipelines and distributed data-processing environmens.Hands-on development experience using Scala.Strong understanding of relevant Microsoft Azure data and cloud services.Experience designing scalable, reliable and maintainable cloud-based data solutions.Strong SQL, data transformation and troubleshooting skills.Excellent analytical and problem-solving ability.Strong communication skills and experience working with technical and non-technical stakeholders.Previous experience working within Agile delivery teams.