Azure Databricks Technical Consultant

We are looking for an experienced Azure Databricks Technical Consultant to design and deliver secure, scalable data and analytics solutions within a complex, highly regulated environment. The consultant will have a strong background in Azure Databricks and modern data engineering, working across large-scale data ingestion, transformation, analytics and AI workloads. The role will involve integrating structured and unstructured data from multiple sources, including telemetry, operational datasets, OSINT and SOCMINT. Key ResponsibilitiesDesign, build and maintain scalable data pipelines using Azure Databricks, Spark/PySpark and Azure Data Lake StorageIngest, transform and process large volumes of structured, semi-structured and unstructured data from multiple sourcesDevelop secure Lakehouse architectures and reusable data engineering frameworks within AzureDesign and optimise data models, analytical datasets and feature stores supporting advanced analytics and AI use casesIntegrate Databricks with Azure Machine Learning, Azure OpenAI and wider Azure data servicesSupport AI/ML and agentic AI workloads including RAG, LLMs and autonomous agent architecturesDevelop analytical solutions incorporating machine learning, NLP, graph analytics, clustering, entity resolution and pattern analysisImplement MLOps, AIOps and SecML processes covering model monitoring, governance, audit logging, explainability, drift detection and rollbackWork with graph technologies including Neo4j, Azure Cosmos DB Gremlin API and Microsoft Graph Data ConnectBuild reporting, dashboards and analytical outputs using Power BI and associated Microsoft technologiesEnsure data platforms meet stringent requirements around security, governance, Zero Trust, compliance and Responsible AIConduct functional, system, performance and non-functional testing across data and AI solutionsProduce solution designs, technical documentation, implementation plans and operational support documentationCollaborate with data scientists, cloud architects, software engineers, analysts and business stakeholders within Agile delivery teamsEssential Skills & ExperienceStrong hands-on experience with Azure DatabricksExcellent Python/PySpark and SQL data engineering skillsStrong experience designing and developing ETL/ELT and large-scale data pipelinesExperience with Apache Spark and Databricks Lakehouse architecturesStrong knowledge of Azure Data Lake Storage (ADLS)Experience integrating Databricks with wider Azure data and analytics servicesUnderstanding of data modelling, data quality, governance and performance optimisationExperience working with Azure Machine Learning and/or Azure OpenAIStrong understanding of cloud security, identity, access controls and Zero Trust principlesExperience working within Agile/Scrum delivery environmentsDesirable ExperienceAgentic AI, autonomous agents, RAG and Large Language ModelsMLOps and productionisation of machine learning workloadsGraph databases and graph analytics including Neo4j or Cosmos DB Gremlin APIPower BI and analytical visualisationFeature stores and ML data pipelinesNLP, clustering, entity resolution and causal inferenceResponsible AI and AI governance frameworksPrevious experience within Defence, Government, National Security or similarly regulated environmentsThis is an Inside IR35 PAYE contract role so you must be resident in the UK to be considered for this role and you must hold ACTIVE SC CLEARANCE