Senior Data Engineer

Senior Data Engineer — Data Operations Support Onsite UK (with offshore team coordination) 8+ years in data engineering / data operations; 3+ years hands-on Databricks Reports to : Data Lead / Service Lead (AMS + Data) Role summary An experienced Senior Data Engineer to lead day-to-day data operations support across the data estate, combining strong business understanding with deep hands-on engineering. The role owns the health and stability of ADF and Databricks pipelines, leads incident triage and root cause analysis, resolves complex issues, drives minor enhancements, and mentors the offshore data engineers. The successful candidate is the technical escalation point for the team and the trusted interface to Client stakeholders. Core skills (must have) Data engineering & platform Azure Data Factory (ADF) — pipeline design, orchestration, monitoring and troubleshooting.Databricks — PySpark, Spark SQL, Delta Lake, Workflows, performance tuning and optimisation.Python — production-grade scripting, automation and reusable utilities.SQL — advanced querying, stored procedures, indexing, partitioning and query optimisation (Azure SQL / SQL Server).Azure Cloud — Azure Monitor, ADLS, Key Vault; CI/CD and Git-based development.Operations & service management Incident & problem management — lead triage, resolution and permanent remediation of recurring issues.Root cause analysis — structured RCA feeding the Known Error Database (KEDB) and SOPs.SLA management — own SLA adherence for P1–P4 incidents and service requests.Data quality — reconciliation, validation frameworks and exception handling.Key responsibilities Own end-to-end health and stability of ADF and Databricks pipelines across the legacy estate and the new GDS.Lead incident triage and resolution, acting as the L3 escalation point for complex issues.Perform and document root cause analysis; drive permanent fixes and preventive actions.Deliver minor enhancements and performance optimisation (incremental loads, indexing, cluster/pipeline tuning).Build and maintain monitoring, alerting and SLA dashboards; automate routine operational tasks.Maintain SOPs, run books and the KEDB; lead knowledge transfer from the incumbent teams.Mentor and guide the offshore data engineers; review their work and uplift team capability.Engage Client stakeholders, business users and third parties; represent the team in service governance.Business & behavioral skills Strong stakeholder communication — able to translate technical issues into business impact.Ownership & accountability — drives issues to closure and sets the standard for the team.Insurance / financial services domain awareness is an advantage.Documentation discipline — clear, reusable knowledge assets for scalability.Nice to have Databricks and/or Azure certifications (Data Engineer Associate/Professional).Experience with data quality tooling (DQPro, Ataccama) and Power BI support.Experience in managed services / transition and takeover of data support.