Observability Engineer

Observability Engineer (KUBERNETES & CONTAINERS, GITOPS & AUTOMATION, QUERY LANGUAGES, CLOUD & INFRASTRUCTURE) Toronto, On Hybrid - 4 days ABOUT THE ROLE We are seeking an experienced Observability Engineer to join our Enterprise Kubernetes Platform team at a leading financial services organization. You'll own the complete observability stack across 50+ production Kubernetes clusters, providing metrics, logging, tracing, and alerting capabilities that ensure exceptional reliability and performance for mission-critical applications. This role combines deep technical expertise in modern observability tools with emerging AI/ML capabilities to build intelligent monitoring solutions, predictive alerting, and self-healing infrastructure. ======================================================================== WHAT YOU'LL DO ======================================================================== OBSERVABILITY STACK OWNERSHIP -------------------------------------------------------------------------------- • Design, deploy, and maintain enterprise-scale observability infrastructure including Prometheus, Grafana, Thanos, Loki, and modern collection agents • Manage observability deployments using GitOps principles and infrastructure as code • Implement long-term metrics storage solutions with cloud object storage • Maintain and upgrade observability components across development, QA, UAT, production, and DR environments • Configure distributed observability architecture spanning multiple data centers and cloud providers METRICS & MONITORING -------------------------------------------------------------------------------- • Design and implement Prometheus monitoring strategies for Kubernetes infrastructure and containerized applications • Create ServiceMonitors, PodMonitors for automated metrics collection • Develop rules for intelligent alerting with minimal false positives • Configure multi-cluster metrics federation and aggregation • Optimize metrics cardinality, storage e_iciency, and query performance • Implement recording rules for pre-aggregated metrics and SLI calculations DASHBOARDS & VISUALIZATION -------------------------------------------------------------------------------- • Build comprehensive Grafana dashboards for infrastructure health, application performance, and business metrics • Create reusable dashboard templates and libraries for development teams • Implement dashboard-as-code • Configure multiple datasources • Design executive dashboards with SLO/SLI tracking and business KPIs • Implement role-based access control and multi-tenancy in Grafana LOGGING INFRASTRUCTURE -------------------------------------------------------------------------------- • Deploy and manage centralized logging solutions (Loki, ELK, Splunk, or similar) • Configure log collection agents (Promtail, Fluentd, FluentBit, Vector, etc.) • Design log retention policies balancing cost, compliance, and operational needs • Create LogQL/Lucene queries and log-based alerts • Implement log correlation with metrics and traces for unified troubleshooting • Build log aggregation pipelines with parsing, filtering, and enrichment ALERTING & INCIDENT MANAGEMENT -------------------------------------------------------------------------------- • Configure intelligent alerting with Alertmanager or equivalent platforms • Design alert rules with appropriate severity levels, thresholds, and SLOs • Implement alert routing to Slack, PagerDuty, ServiceNow, email, and webhooks • Create automated runbooks and remediation workflows • Develop alert inhibition, silencing, and grouping strategies • Tune alerting to achieve signal-to-noise ratio improvements • Integrate with incident management and on-call rotation systems DISTRIBUTED TRACING -------------------------------------------------------------------------------- • Implement distributed tracing solutions using OpenTelemetry, Jaeger, or Tempo • Instrument applications for trace collection and correlation • Configure trace sampling strategies for cost and performance optimization • Build trace-based dashboards for latency analysis and dependency mapping • Integrate tracing with metrics and logs for comprehensive observability AI/ML FOR OBSERVABILITY -------------------------------------------------------------------------------- • Implement AI-powered anomaly detection for metrics and logs • Build predictive alerting using machine learning models to forecast issues before they occur • Develop intelligent alert correlation and root cause analysis systems • Integrate LLM-based tools for log analysis and troubleshooting assistance • Implement AIOps capabilities for automated incident triage and resolution • Use AI to optimize alert thresholds and reduce false positives • Build natural language query interfaces for observability data • Implement Model Context Protocol (MCP) for AI agent integration with observability platforms AUTOMATION & PLATFORM INTEGRATION -------------------------------------------------------------------------------- • Automate observability deployment using GitOps workflows (FluxCD, ArgoCD) • Integrate with CI/CD pipelines for automated testing and validation • Build self-service portals for teams to create dashboards and alerts • Develop APIs and CLIs for observability automation • Integrate with secrets management solutions (Vault, AWS Secrets Manager) • Configure LDAP/AD/SSO authentication for observability platforms • Automate compliance reporting and audit logging ENABLEMENT & COLLABORATION -------------------------------------------------------------------------------- • Onboard application teams to observability platforms • Provide guidance on instrumentation best practices • Create documentation, training materials, and self-service guides • Conduct workshops • Support development teams during incidents with observability insights • Collaborate with SRE, DevOps, and platform engineering teams PERFORMANCE & COST OPTIMIZATION -------------------------------------------------------------------------------- • Monitor and optimize observability stack resource consumption • Implement autoscaling for stateless observability components • Tune data retention, compaction, and downsampling strategies • Conduct capacity planning for metrics and log storage growth • Optimize query performance and dashboard response times • Implement cost allocation and chargeback for multi-tenant environments ======================================================================= REQUIRED QUALIFICATIONS ======================================================================= EXPERIENCE -------------------------------------------------------------------------------- • 4-6 years of experience in observability, monitoring, SRE, or platform engineering roles • 3+ years hands-on production experience with Prometheus and Grafana • 2+ years working with Kubernetes and containerized environments • Strong expertise with PromQL for metrics querying and alerting • Experience deploying and managing observability stacks at scale (1000+ nodes) • Proven track record of reducing MTTR through e_ective observability CORE TECHNICAL SKILLS -------------------------------------------------------------------------------- OBSERVABILITY PLATFORMS: • Prometheus (including Prometheus Operator) • Grafana (dashboards, alerting, plugins) • Thanos, Cortex, or Mimir for long-term storage • Alertmanager or equivalent alerting platforms LOGGING: • Loki, Elasticsearch/ELK Stack, Splunk, or CloudWatch Logs • Log collection agents (Promtail, Fluentd, FluentBit, Vector) • LogQL, Lucene, or equivalent query languages TRACING: • OpenTelemetry (OTEL Collector, instrumentation) • Jaeger, Zipkin, Tempo, or AWS X-Ray • Trace sampling and correlation strategies KUBERNETES & CONTAINERS: • Kubernetes architecture and operations (1.24+) • Custom Resource Definitions (CRDs) and Operators • ServiceMonitor, PodMonitor, PrometheusRule resources • Kubernetes metrics (kube-state-metrics, node-exporter, cAdvisor) GITOPS & AUTOMATION: • GitOps workflows (FluxCD, ArgoCD, or similar) • Infrastructure as Code (Kustomize, Helm, Terraform) • CI/CD platforms (GitHub Actions, GitLab CI, Jenkins) • Scripting (Python, Bash, Go) QUERY LANGUAGES: • PromQL (expert level required) • LogQL or Lucene • Basic SQL for data analysis CLOUD & INFRASTRUCTURE: • AWS, Azure, or GCP cloud platforms • Object storage (S3, GCS, Azure Blob) • Multi-cloud and hybrid architectures • vSphere or on-premises virtualization (nice to have) AI/ML & EMERGING TECHNOLOGIES -------------------------------------------------------------------------------- • Experience with AI/ML frameworks for observability (Prophet, TensorFlow, PyTorch) • Anomaly detection algorithms and time-series forecasting • LLM integration for log analysis and troubleshooting (GPT, Claude, etc.) • Model Context Protocol (MCP) for AI agent integration • AIOps platforms (Moogsoft, BigPanda, Datadog Watchdog, etc.) • Natural language processing for log parsing and analysis • Familiarity with vector databases for semantic search (Pinecone, Weaviate) • Experience with AI-powered root cause analysis tools • Knowledge of prompt engineering for observability use cases PREFERRED QUALIFICATIONS -------------------------------------------------------------------------------- • Kubernetes certifications (CKA, CKAD, or CKS) • Grafana certification or equivalent training • Knowledge of service mesh observability ( • Experience with SRE practices, SLIs, SLOs, and error budgets • Background in financial services or regulated industries • Familiarity with compliance requirements (SOX, PCI-DSS, etc.) • Contributions to open-source observability projects • Experience with APM tools