Data Engineer 2

Position Summary The Data Engineer 2 builds and operates the pipelines and datasets that the business runs on. This is a hands-on delivery role: the engineer owns pipelines end to end (ingestion, transformation, publishing, monitoring, and support) and is expected to contribute to design decisions above the level of an individual job, particularly in orchestration, data lineage, and how curated data is modeled and exposed to downstream consumers. Beyond conventional ETL, this role builds the curated, well-described datasets that analytics tools and AI systems consume. A successful candidate can explain not only how to move data, but why a given storage format, partition layout, or table design is the right one, and what breaks downstream when it is wrong. Responsibilities Build and maintain batch, incremental, and near-real-time pipelines that extract from transactional databases, SaaS and internal APIs, file feeds, and event streams, and load to databases, data lakes, and data warehouses.Implement change data capture (CDC) on high-volume source tables, handling schema drift, late-arriving and out-of-order records, soft deletes, and replays.Write idempotent, restartable jobs: re-running a pipeline for a given partition or window produces the same result without duplicating or losing rows.Apply the correct load pattern for the volume and business need, whether full refresh, watermark-based incremental, merge/upsert, or slowly changing dimension (Type 1 and Type 2), and justify the choice.Write and tune complex SQL: window functions, multi-CTE transformations, set operations. Read the query plan and address the actual cause of slowness (scan volume, distribution skew, spill to disk, poor join order) rather than adding hardware.Apply business rules and integration logic in a testable, documented layer rather than embedded in ad-hoc scripts.Designing the orchestration layer, task dependencies, and observability. Responsibilities include implementing backfill mechanisms, managing failure strategies, and creating thorough documentation.Modeling data for consumption with clear definitions of grain, keys, and measures, alongside an understanding of columnar storage mechanics like Parquet and ORC. Responsibilities include applying physical design principles for columnar warehouses, managing file layouts in object storage, and working with non-transactional stores while evaluating and defending the appropriate storage solution for various workloads.Designing automated lineage capture at table and column levels, ensuring that documentation stays current with orchestration and transformation changes. Responsibilities include maintaining dataset catalogs with essential details, performing impact analysis before schema changes, and implementing data quality gates to prevent erroneous data from being published. Additionally, focuses on supporting auditability and reproducibility while applying least-privilege access and data retention rules.Thorough testing before code promotion, including unit tests and validation against source counts, along with ensuring the target environment is ready for initial data loads. Providing production support for pipelines, managing failures with a focus on root causes, participating in an on-call rotation for incidents, and working within a Git-based workflow with peer reviews. Additionally, comprehensive documentation is essential to ensure that datasets and pipelines can be operated by others without assistance.Translating business questions from various stakeholders into clear data requirements while suggesting alternative solutions when appropriate. Responsibilities include participating in design and code reviews to address concerns regarding maintainability, cost, and data accuracy, as well as mentoring Level 1 engineers in SQL, pipeline patterns, and debugging techniques.Provide production support for the pipelines you own: triage and root-cause failures, communicate impact and ETA to affected consumers, then eliminate the class of failure rather than only the instance. Required Qualifications Bachelor’s degree in Computer Science, Information Technology, or a related field, or equivalent practical experience.At least 3 years building and operating production data pipelines (not solely analytics or reporting work).Strong Python at production quality: modular code, error handling, logging, configuration, and tests, well beyond one-off scripts.Advanced SQL, including window functions, multi-step CTE transformations, and set operations; able to read an execution plan and explain why a query is slow and what to change.Hands-on AWS data services: S3, Glue, Lambda, and Redshift, plus at least one of DMS, Kinesis, EMR, Step Functions, or Athena.Demonstrable understanding of columnar storage and columnar analytical databases: file formats, encoding and compression, partitioning and pruning, and their effect on query cost.Experience loading and querying non-transactional stores, including a data lake on object storage; able to explain how lake and warehouse workloads differ.Production experience with at least one orchestration framework, and the ability to reason about workflow design, dependencies, idempotency, retries, and backfills rather than only clicking through a UI.Data modeling ability: dimensional and/or wide-table designs, grain and key definition, and slowly changing dimensions.Data quality as routine practice: automated tests and reconciliation as part of delivery, not an afterthought.Experience with Git branching, pull requests, and CI-based deployment.Clear written and verbal communication. Able to document a dataset so that someone who did not build it can use it correctly, and able to explain a data issue to a non-technical stakeholder.Ability to analyze broad, ambiguous data needs and decompose them into deliverable work. Preferred Qualifications Open table formats (Apache Iceberg, Delta Lake, Hudi) and lakehouse architecture in production.A transformation framework with built-in testing, documentation, and lineage, such as dbt.Asset-based or flow-based orchestration (Dagster, Prefect) in addition to AWS-native workflow services.Lineage and catalog tooling: AWS Glue Data Catalog, OpenLineage/Marquez, DataHub, OpenMetadata, Amundsen, Collibra, or Alation.Streaming and CDC at scale: Kinesis, Kafka/MSK, Debezium, or equivalent, including exactly-once and ordering concerns.Distributed query and processing engine tuning: Spark, Trino, or Athena.Exposure to serving data to LLMs or AI agents, such as MCP servers, semantic layers, text-to-SQL, or retrieval and vector stores, plus awareness of the failure modes (ambiguous columns, unbounded scans, leaked sensitive fields, stale context).Infrastructure as code (Terraform, CloudFormation, or CDK) and containerized workloads.Cost awareness and optimization: attributing and reducing storage, scan, and compute cost.Familiarity with the downstream BI layer and how modeling decisions constrain it.Domain exposure to e-commerce, print or custom manufacturing, or high-volume order data: orders, SKUs and product configuration, pricing, fulfillment, and marketing attribution.