Senior Data Engineer

Job Responsibilities:  Develop workflows and ELT data pipelines using Python, Spark/PySpark, and Databricks  Onboard enterprise datasets into Palmos, including ingestion, transformation, and validation of data assets  Build, test, and maintain scalable data pipelines and data architectures that support enterprise controls and analytics use cases  Build buisness controls algorithms for communications data to identify anomalies  Build data completeness and integrity controls for the pipelines  Apply data engineering best practices for performance optimization, reliability, and maintainability  Use SQL extensively and work with both relational and NoSQL data stores  Partner with producers to understand data requirements and translate them into production-ready solutions  Apply SDLC practices including CI/CD, testing, and operational monitoring to ensure pipeline stability  Contribute to reusable frameworks and standards to accelerate onboarding and pipeline delivery  Identify data issues, anomalies, and optimization opportunities to improve data quality and performance  Leverage enterprise-authorized AI coding assist tools within the work environment to improve code quality, delivery speed, and productivity across complex deliverables (e.g., code generation/refactoring, unit test creation, documentation), while validating outputs through peer review, automated testing, and secure coding standards; contribute learnings and reusable patterns to improve broader team effectiveness Required Qualifications, Capabilities, and Skills:  Hands-on experience with Databricks, Spark/PySpark, Python, and SQL  Experience developing and maintaining data pipelines and data processing systems  Understanding of the data lifecycle, including ingestion, transformation, storage, and consumption  Knowledge of cloud platforms (AWS) and distributed data processing  Experience with SDLC practices including CI/CD, testing, and deployment  Strong problem-solving skills and ability to troubleshoot data and pipeline issues  Ability to collaborate effectively within agile teams and across stakeholders  Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (e.g., for coding, test creation, troubleshooting, or documentation) with demonstrated ability to critically evaluate, validate, and refine AI- generated outputs for correctness, performance, and security  Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; ability to guide peers on safe and effective usage within team practices Preferred Qualifications, Capabilities, and Skills:  Experience with Databricks lakehouse, Databricks Genie, Delta Lake, and medallion architecture  Familiarity with enterprise data platforms and data mesh principles  Exposure to data quality, observability, and metadata management tools  Experience supporting analytics, reporting, or AI/ML workloads  Experience working on regulatory controls