Summary
✨ AI‑Generated
Join a senior data engineering role focused on building scalable cloud-based data solutions. You will design and maintain batch and streaming pipelines, develop large-scale processing solutions with Python and SQL, manage data workflows, and contribute to modern data architectures. The role provides opportunities to work with distributed processing, workflow orchestration, streaming technologies, and cloud-native data platforms.
Highlights
Work on scalable cloud data platforms using modern big data technologies, with exposure to both batch and streaming workloads, optimization and modernization initiatives, and collaboration across technical and business teams.
Description
About The Role
In this role, you will contribute to developing scalable cloud-based data solutions on AWS using Databricks and modern big data technologies.
You will work on batch and streaming data pipelines, support optimization and modernization initiatives, and collaborate with cross-functional teams to deliver reliable and efficient data platforms across different stages of the project lifecycle.
Responsibilities
Design, develop, and maintain scalable batch and streaming data pipelinesBuild and optimize data processing solutions using Python (PySpark) and SQLDevelop and manage data workflows using Databricks on AWSWork with Apache Spark and Databricks for large-scale data processingImplement and maintain Delta Lake-based data architecturesUtilize orchestration tools such as Apache Airflow or MWAA to manage workflowsSupport implementation of streaming solutions using Apache Kafka, Amazon MSK, or KinesisCollaborate with technical and business stakeholders to deliver effective data solutionsParticipate in architecture discussions and contribute to continuous improvement initiatives within the teamParticipate in the full project lifecycle, from PoC and MVP stages to production implementation
Requirements
4+ years of experience in Big Data or Data EngineeringStrong proficiency in Python (PySpark) and SQLHands-on experience with AWS cloud services for data engineering solutionsPractical experience with Databricks on AWS, including building and managing data pipelinesStrong knowledge of Apache Spark and large-scale data processingExperience with batch and streaming data processingFamiliarity with Delta Lake and Databricks components such as Workflows and JobsExperience with orchestration tools such as Apache Airflow or MWAAKnowledge of streaming technologies such as Apache Kafka, Amazon MSK, or KinesisUpper-intermediate or higher level of English
SoftServe is an equal opportunity employer.
Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.
