GCP Data Engineer

Role Description & Requirements Role and Responsibilities Key cloud engineer for customer engagementsCollaborate with lead Data Consultant on customer engagementsDefine solution architecture and detailed design for data platforms on GCPBuild solutions, i.e., perform hands on coding for the designed solutionInteract with customer data engineering team to provide sound advice and guidance on solution architecture as well as hands on coding solutionsPerform pair programming with customer’s data engineers and handover the developed solution to customer engineers. Minimum Qualifications Solution Architecture Knowledge and Expertise Required ○ Designing data platform architecture and design for migrating open source or other public cloud based data platforms to Cloud (pref. GCP) Designing green field data platforms on cloud ○ Ability to perform research on tools and technology, identify important considerations for technology selection in the architecture, provide recommendations on tool selection ○ Define solution architecture and detailed design for batch ingestion and processing of data from sources such as files, cloud and on-prem databases, streaming solutions for real time data ingestion and processing (such as log analytics, click-stream analytics, real time data ingestion from other types of sources) ○ Build data pipelines and CI/CD pipelines ○ In-depth knowledge of at least one public cloud, preferably GCP Tools experience Required ○ GCP (GCP preferred, or equivalent in other public clouds acceptable) ○ GCP Dataflow, Pub/Sub, Cloud Composer, BigQuery, Cloud Run, Cloud ○ Functions,Spanner, CloudBuild, Vertex AI, Terraform Coding expertise ○ Programming knowledge and willingness to be hands-on as part of day to day work ○ Knowledge of at least one (preferably both): Python, Java Preferred Qualifications Solution Architecture Knowledge and Expertise Required ○ Experience with open source ecosystem and distributions such as Hadoop, Spark, Cloudera/Hortonworks ○ Understanding and experience with NoSQL databases such as HBASE, MongoDB Tools experience Required ○ Other public clouds ○ Open source ecosystem and frameworks (Hadoop, SPARK, Oozie, Kafka, HBASE) Certification At least one of below: ○ GCP Data engineering (preferred) ○ AWS Customer Experience Experience in large enterprise environments, navigating architectural review and approval frameworks, lobbying design submissions, engaging cross-functional stakeholders (Preferably one of the Big Four AU Banks)