Senior Data Engineer

Role Purpose The Senior Data Engineer is responsible for the design, development, testing, maintenance, and documentation of data pipelines and data platforms in accordance with client requirements, system specifications, and defined technical architecture. The role undertakes analysis of system requirements and evaluates existing data systems, identifying limitations and deficiencies in processes and methods. The position is accountable for the full lifecycle of data solutions, including requirements gathering, design, implementation, testing, deployment, and ongoing production support. Key Responsibilities Data Architecture & DesignAnalyse and evaluate client data system requirementsDesign data architectures, including Lakehouse and medallion patternsDefine data models, schema structures, partitioning strategies, and storage layoutsDevelop solutions primarily using Databricks within AWS, Azure, and GCP environmentsData Pipeline DevelopmentDesign, develop, and maintain data pipeline code using Python, Scala, and SQLImplement data ingestion, transformation, and delivery processes in accordance with system specificationsTest, debug, and resolve defects in line with established standards and protocolsUtilise AI development tooling such as Genie Code, Copilot or ClaudeData Quality & GovernanceDevelop and implement data quality frameworks, validation processes, and monitoring mechanismsEnsure data platforms operate in accordance with defined specifications and governance requirementsPrepare and maintain technical documentation and operational proceduresInfrastructure & Platform EngineeringDevelop and manage infrastructure using Infrastructure as Code tools (e.g. Terraform, CDK, CloudFormation)Maintain version-controlled infrastructure definitionsSupport CI/CD processes for automated deployment of data systemsDesign and implement integrations with enterprise systems, including identity, security, and compliance frameworksSystem Integration and ImprovementIdentify and assess limitations in existing systems and platformsDesign and implement solutions to address integration, performance, and scalability requirementsTechnical Strategy & AdvisoryProvide technical input into data system design, platform selection, and architecture decisionsContribute to the development of proposals, including cost estimation and financial evaluation of technology optionsClient Enablement and Knowledge TransferWork collaboratively with client teams to support implementation and operation of data platformsDevelop and maintain documentation to support ongoing system use and maintenanceFacilitate knowledge transfer to enable client self-sufficiency Multi-Cloud Capability The role operates across multi-cloud environments and requires the design of data systems that are portable and scalable. Platforms include: AWS: Databricks, S3, Glue, Lambda, EventBridge, Redshift, IAM, VPC, KMSAzure: Databricks, ADLS Gen2, Synapse, Data Factory, Event Hubs, Key Vault, Entra ID, ExpressRouteGCP: Databricks, GCS, BigQuery, Dataflow, Pub/Sub, IAM Required Qualifications and Experience Bachelor’s degree in Computer Science, Information Technology, Software Engineering, or a related disciplineOR at least 5 years’ relevant professional experience with appropriate vendor certificationsDatabricks Data Engineer Associate and/or Data Engineer Professional certificationExperience with Databricks in a production environment, including Delta Lake and Unity CatalogProficiency in SQL and Python or ScalaExperience in designing and implementing data quality and validation frameworksDemonstrated ability to communicate technical concepts clearly to non-technical stakeholdersStrong software engineering practices, including testing, version control (Git), and CI/CDExperience with Infrastructure as Code tools (e.g. Terraform, CloudFormation)Understanding of distributed systems and data pipeline designAbility to produce clear and comprehensive technical documentationDemonstrated experience delivering and maintaining production data systems Bonus Points Experience with real-time or streaming data systems (e.g. Kafka)Knowledge of data governance, privacy, and compliance frameworks (e.g. APRA CPG234, ISO 27001, CIS Benchmarks)Machine learning infrastructure and pipelines (on Databricks or cloud platform)Background in platform engineering or Site Reliability EngineeringCloud platform certifications (AWS, Azure, or GCP)Experience contributing to technical knowledge sharing or capability development within teams