DontApply.Club
Login Register

Data Platform Engineer – GCP & Airflow

Thrive It Systems Ltd — United Kingdom · Posted ~5 hours ago

Senior

Skills

Python Apache Airflow GCP BigQuery GKE Cloud Run ETL/ELT REST APIs CI/CD Terraform Infrastructure as Code Automated testing Performance tuning Data warehousing Cloud architecture Reliability engineering

🔓 Log in to save this job, tailor your resume & track your apply process — 7 days free, no card needed.

Log in to add to target list

Summary ✨ AI‑Generated

A technically demanding data platform engineering role focused on designing and operating scalable, production-grade data integration pipelines. You will lead technical delivery, build reliable ETL/ELT workflows, work extensively with cloud data services and container platforms, and establish robust CI/CD and infrastructure automation. Strong Python and software engineering skills are essential, along with an ability to design for reliability, communicate complex technical concepts, and coordinate engineering efforts across teams.

Highlights

Lead technical delivery and strategy for a scalable data platform, with strong ownership of architecture, reliability, automation, and engineering standards. The role offers broad exposure to cloud infrastructure, data engineering, and cross-team technical leadership.

Description

Lead engineering delivery and technical strategy for a multitenant data integration platform Build and maintain production grade ETLELT pipelines orchestrated with Apache Airflow Utilize container platforms such as GKE and Cloud Run Apply strong software engineering skills in Python APIs configuration management testing and performance tuning Implement CICD pipelines release controls automated quality gates and InfrastructureasCode Terraform Design for reliability including retries idempotency dead letter patterns and operational resilience Translate complex technical concepts to diverse audiences and align cross team efforts Work with GCP BigQuery for largescale data warehousing and analytics Architect cloud solutions using GCP Cloud Architecture best practices Desirable Experience with data lineage metadata management tools multitenant architectures data governance and vendor system integrations Responsibilities Own and drive the technical roadmap aligning engineering with business priorities and operational goalsLead solution design for new integrations and platform enhancements ensuring maintainability and scalabilityEstablish and enforce engineering standards for coding testing documentation observability and securityContribute hands-on to critical ETLs framework components design spikes and performance optimizationsDesign and develop robust Airflow ETL pipelines with effective scheduling dependency management and error handlingDevelop onboarding patterns for new source and vendor systems ensuring reliability and traceabilityDrive data quality controls including validation reconciliation and exception handlingEnhance the multitenant architecture focusing on tenant isolation configuration metadata driven pipelines and runtime scalabilityOptimize platform performance cost and reliability across environments with capacity planning and SLAsImprove automation for build deployment and release processes with CICD best practices and self-service toolingImplement comprehensive data lineage and observability with logging metrics ing and dashboardsPartner with stakeholders including product owners architects vendors and governance teams to manage requirements and deliveryLead agile ceremonies remove blockers and maintain delivery momentumMentor engineers through reviews and coaching to foster continuous improvement and capability growth Required Skills Apache AirflowETL ConceptsGCP BigQueryGCP Cloud ArchitecturePython Preferred Skills Desirable Experience with data lineage metadata management tools multitenant architectures data governance and vendor system integrations