Senior Distributed Data Engineer

Senior Data Infrastructure Engineer Berlin, Germany | Hybrid - 3 days onsite weekly Join a highly technical AI research and engineering organisation building the data infrastructure required to support next-generation AI models at massive scale. As a Senior Data Infrastructure Engineer, you will architect, build and operate the distributed systems that process petabytes of data and connect large-scale data workflows with cutting-edge AI research. The Senior Data Infrastructure Engineer will focus on scalability, reliability, throughput, resource utilisation and cost efficiency, ensuring that complex data-processing workloads can run consistently across large distributed environments. This Senior Data Infrastructure Engineer role is ideal for someone who enjoys solving difficult distributed-systems problems and has experience owning data platforms under real-world throughput, reliability and cost pressures. Key Responsibilities Design, build and scale distributed data-processing infrastructure capable of handling petabyte-scale workloads.Develop execution layers for pipeline stages with different computational requirements, selecting appropriate processing engines and technologies.Optimise compute utilisation across large-scale workloads, including GPU-accelerated data-processing stages.Build highly reliable pipelines with graceful failure recovery, restart-ability, monitoring and comprehensive observability.Profile large-scale workloads to identify throughput bottlenecks and optimise compute utilisation and infrastructure costs.Establish robust systems for dataset versioning, lineage, reproducibility and traceability across the data-processing lifecycle.Ensure intermediate and final datasets can be reliably reproduced against specific research requirements.Build and maintain data infrastructure that integrates effectively with large-scale machine learning training pipelines.Evaluate emerging technologies and architectural approaches to improve scalability, reliability, performance and cost efficiency.Establish technical documentation, standards and best practices across the data platform. Required Experience & Skills Bachelor's degree or higher in Computer Science, Computer Engineering, Software Engineering or a related discipline.At least 5+ years of full-time experience designing, implementing and operating large-scale distributed data-processing systems or ML data infrastructure.Demonstrated ownership of production data-processing systems operating under significant throughput, reliability and cost pressures.Strong experience with distributed processing frameworks such as Ray, Apache Spark or Flink.Experience with workflow orchestration and large-scale pipeline execution.Strong understanding of performance profiling, throughput optimisation and distributed computing.Experience working with Apache Arrow, Parquet or comparable high-performance data formats.Experience with dataset versioning, lineage, reproducibility and traceability.Experience with workload managers or schedulers such as Ray, Kubernetes or Slurm.Familiarity with containerisation technologies including Docker or Enroot.Experience designing systems for fault tolerance, observability and failure recovery.Strong software engineering skills and the ability to make sustainable technical decisions in rapidly evolving environments.Desired: Experience supporting GPU-accelerated data-processing workloads; Experience working with cloud infrastructure and large-scale compute environments; Familiarity with vector databases and modern data infrastructure; Experience with infrastructure-as-code; Experience working directly with ML training infrastructure or frontier AI workloads. Why Apply? Build the core data infrastructure supporting large-scale AI research and model development.Work at petabyte scale, solving complex distributed-systems and data-platform challenges.Have genuine architectural ownership over how large-scale workloads are executed, monitored and optimised.Work closely with highly technical Research and Engineering teams.Evaluate and introduce new technologies rather than simply maintaining an established data platform.Solve problems spanning distributed computing, GPU utilisation, reliability, observability, throughput and infrastructure cost. Apply now or send a copy of your CV, referencing the title and location, and with a short intro to cw@eu-recruit.com. By applying to this role you understand that we may collect your personal data and store and process it on our systems. For more information please see our Privacy Notice (https://eu-recruit.com/about-us/privacy-notice/)