Summary
✨ AI‑Generated
Design and deliver scalable Lakehouse platforms across AWS and/or Azure for a range of client requirements. You will architect ingestion, storage, processing, and consumption layers; automate infrastructure with Terraform; establish secure environments and governance; build CI/CD processes; and optimize monitoring, performance, and costs. The role also includes designing batch and streaming pipelines and working directly with clients.
Highlights
Broad platform engineering role covering cloud architecture, data pipelines, governance, security, automation, CI/CD, monitoring, and optimization. The position provides direct client engagement and opportunities to design scalable Lakehouse platforms across major cloud environments.
Description
Platform Architecture & Engineering responsibilities:
Design and implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirementsArchitect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layersBuild and configure cloud infrastructure using infrastructure-as-code (e.g.
Terraform & Declarative Automation Bundles(DAB's)Establish secure, compliant environments including networking (VNet/VPC, Private Link), identity (IAM/Entra ID), data governance (Unity Catalog), and access controlsDefine environment strategies (dev/test/prod), CI/CD pipelines, and release processes for Databricks deploymentsImplement monitoring, logging, cost optimisation, and performance tuning across the platformDesign and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workload
Client Delivery & Enablement responsibilities:
Work directly with clients to translate business and technical requirements into scalable platform designsLead technical workshops, architecture sessions, and whiteboarding engagements with client stakeholdersSupport rapid prototyping and proof-of-concept builds within Databricks to demonstrate platform capabilities and accelerate client adoptionProvide best practice guidance on Lakehouse architecture, data modelling, workload optimisation, and cost managementProduce high-quality technical documentation including architecture diagrams, architecture decision records (ADRs), runbooks, and deployment guidesEnable client teams through structured knowledge transfer, training, and platform handoverCollaborate with data engineers, data scientists, and product teams to ensure successful delivery outcomes
Responsibilities
Governance & SecurityImplement Unity Catalog for centralised data governance, including access control (RBAC/ABAC), data lineage, audit logging, and compliance enforcementApply security best practices across platform design: network isolation, secret management, encryption at rest and in transit, and identity federationEnsure platform designs meet client regulatory and compliance requirements (e.g.
GDPR, ISO 27001, sector-specific standards)
What success looks like in the role
Delivery of robust, secure, and scalable Databricks platforms that meet client performance and cost expectationsClear, well-architected solutions that balance flexibility, governance, and operational efficiencyStrong client relationships built on trust, technical credibility, and effective communicationAccelerated client adoption of the Lakehouse platform through well-designed enablement and documentationReduced deployment time through reusable infrastructure patterns and automationProactive identification of risks, trade-offs, and optimisation opportunities across platform design and deliveryContribution to the organisation’s growing body of reusable platform accelerators, reference architectures, and internal knowledge
Qualifications
3+ years experience in data platform engineering, cloud engineering, or similar roles
Required Skills
Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, WorkflowsProven experience designing and deploying data platforms on AWS and/or Azure (e.g.
ADLS, S3, VNet/VPC, IAMExperience with infrastructure-as-code tools (e.g.
Terraform preferred) and CI/CD pipelines (e.g.
Azure DevOps, GitHub Actions)Solid understanding of data architecture concepts including Lakehouse medallion architecture and dimensional modellingFamiliarity with security and governance frameworks (e.g.
RBAC, ABAC, data masking, audit, compliance standards)Excellent communication skills with the ability to explain complex technical concepts to non-technical stakeholdersComfortable working in a client-facing consultancy environment with multiple concurrent engagementsProactive, self-driven, and able to take ownership of end-to-end platform deliveryWillingness to travel within the UK as required
