Summary
✨ AI‑Generated
A technology-focused organization is looking for a skilled data engineering professional to build, optimize, and maintain scalable data solutions. The role involves developing data pipelines, improving processing performance, implementing modern data management practices, and collaborating with engineers, architects, analysts, and business stakeholders to deliver reliable analytics capabilities.
Highlights
Opportunity to work on large-scale data engineering and analytics solutions using modern cloud technologies. The role offers collaboration with cross-functional technical teams and involvement in designing, optimizing, and supporting high-performance data platforms.
Description
We are seeking a skilled Databricks Application Engineer to design, develop, optimize, and support scalable data engineering and analytics solutions on the Databricks platform.
The ideal candidate will have strong expertise in Spark, Databricks, cloud platforms, and data pipeline development.
This role involves working closely with data engineers, architects, data scientists, and business stakeholders to deliver high-performance data solutions.
Key Responsibilities:
Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.Build and optimize ETL/ELT processes for large-scale structured and unstructured data.Develop data ingestion frameworks from various sources including APIs, databases, flat files, and streaming platforms.Implement Delta Lake architecture and data management best practices.Optimize Spark jobs for performance, scalability, and cost efficiency.Collaborate with business and technical teams to gather requirements and translate them into technical solutions.Develop notebooks, workflows, and automated jobs within Databricks.Monitor and troubleshoot production data pipelines and workflows.Implement data quality checks, governance, and security standards.Support CI/CD deployment processes and DevOps practices.Participate in code reviews and contribute to technical design discussions.Ensure compliance with organizational data management policies and standards.Required Qualifications:
Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.3+ years of experience in Data Engineering or Big Data technologies.Hands-on experience with Databricks platform.Strong programming skills in Python, PySpark, or Scala.Experience with Apache Spark and distributed data processing.Strong SQL development and database concepts.Experience with Delta Lake, Databricks Workflows, and Unity Catalog.Knowledge of Data Warehousing concepts and dimensional modeling.Experience working with cloud platforms:Azure Databricks (Preferred)AWS DatabricksGCP DatabricksExperience with Git, CI/CD pipelines, and version control systems.Strong analytical and problem-solving skills.Preferred Qualifications:
Databricks Certified Associate or Professional certification.Experience with Azure Data Factory, Azure Synapse, or Microsoft Fabric.Knowledge of Kafka, Event Hubs, or streaming data architectures.Experience with Terraform or Infrastructure as Code (IaC).Familiarity with Airflow, DBT, or orchestration tools.Exposure to machine learning workflows within Databricks.Experience in Agile/Scrum environments.Technical Skills:
Must Have:
DatabricksPySparkPythonSQLDelta LakeData EngineeringApache SparkETL/ELTGood to Have:
Azure Data FactoryAzure Synapse AnalyticsMicrosoft FabricKafkaAirflowDBTTerraformGitHub Actions / Azure DevOpsNice to Have:
Machine Learning Operations (MLOps)Data GovernanceUnity CatalogData Security & ComplianceSoft Skills:
Strong communication and stakeholder management.Ability to work independently and collaboratively.Excellent troubleshooting and analytical skills.Strong documentation and knowledge-sharing mindset.
