Lead Site Reliability Engineer – Reliability Platform

The problem you'd own Imunify360 is a multi-layer Linux server security suite — WAF, IDS/IPS, malware scanning and cleanup, proactive defence, patch management, reputation — running as an agent on hundreds of thousands of customer servers, backed by a cloud estate of scanning, correlation and signature-delivery services on our own bare metal. Roughly 70 components currently ship without a defined service level indicator. Some are internal services we can scrape. Many are agent-side subsystems running on machines we do not own, reporting through a heartbeat we designed for something else. There is monitoring, and there are dashboards, and there is no coherent answer to the question "is this component doing its job right now, and how would we know if it stopped?". We know the cost of that gap precisely, because we recently paid it: a security control was silently disabled across a large fraction of the fleet for 61 days. Every dashboard was green. The telemetry reported a ruleset version but not whether the control that consumed it was switched on, so a configuration change was indistinguishable from a broken updater. Three independent safety mechanisms existed and all three were gated behind the same condition that caused the failure. Your job is to make that class of failure detectable in hours instead of months, across the whole product line, and to build the system that keeps it detectable as the product changes. This is a greenfield charter inside a brownfield estate. You are not inheriting an SRE team, an SLO framework or a paging culture. You are defining them, with the engineering leads, and then making them stick. What you'll do Define what "working" means for :70 componentsRun SLI definition with squad leads and senior engineers. You facilitate and hold the standard; the owning squad signs the SLI. Build the taxonomy this product actually needs, which is broader than availability and latency:Service SLIs — availability, latency, error rate for cloud-side servicesFleet SLIs — heartbeat reachability, version and configuration convergence across the installed baseControl-efficacy SLIs — the differentiator. What fraction of protected units have the control effectively enabled and current , not merely installed. Ruleset generation drift, signature age, scan coverage, enforcement-mode distributionDelivery SLIs — artifact publish success, rule-to-fleet lead time, hotfix time-to-convergencePipeline SLIs — ingest lag, verdict latency, queue age, backlog burnEnforce one non-negotiable design rule: an SLI must be measurable from outside the gate of the thing it measures. If the control being off also switches off the signal that would tell you it is off, the SLI is invalid. This is the lesson of the incident above and it is the reason this role existsAttach an SLO, an error budget and an owning squad to each. Tiering is expected — not every component earns a 99.9% target or a pager Build the collection systemDesign and build the pipeline that gets these indicators off the fleet and into a queryable store: push-based, sampled, privacy-constrained, and with a cardinality budget you set and defendExtend agent-side and service-side instrumentation where the signal does not exist yet, in Python, Go and Rust, working with the owning squadsConsolidate the current sprawl of dashboards, ad-hoc queries and reporting paths into a defensible set of instruments, and retire what does not earn its keep Build alerting and alert managementSymptom-based, SLO-anchored alerting with multi-window burn-rate semantics. Not threshold soupA three-tier taxonomy — page / ticket / dashboard — with an explicit rule for what is allowed to page a human at 03:00Every alert ships with an owner, a runbook and a documented failure mode, or it does not shipAlert hygiene as a standing practice: quarterly review, deletion counted as a win, actionable-rate tracked. A persistent inability to perform a security-relevant refresh should page. It currently logs a warning Build escalationComponent → owning squad ownership map, kept current, machine-readable, and wired into routing so an alert reaches the right seven people rather than a shared channelSeverity matrix, acknowledgement SLAs, follow-the-sun rota design across UTC−5 ... UTC+8, and clean handoff protocolIncident command practice and blameless postmortems within 24 hours. We already do postmortems and do them honestly, including publicly retracting our own wrong findings; you raise the floor on the mechanical parts — timelines, ownership, action-item follow-throughDesign the escalation system so that squads carry their own pagers. You build and operate the platform and coach on the practice; you are not the buffer that absorbs everyone else's alerts Requirements What you'll bring Required (Must-haves): Substantial production-engineering or SRE experience, including at least one environment where you defined the SLO framework rather than inherited it. We will ask you to walk through SLIs you personally wrote and how you negotiated them with resistant teamsStrong Python. Comfortable reading and modifying Go or Rust — our agents are written in them and instrumentation lands thereDeep practical grip on time-series and event telemetry at scale: Prometheus/OpenMetrics, Grafana, an Alertmanager-class routing layer, and a columnar store for high-cardinality fleet data (ClickHouse or equivalent)Distributed systems debugging on bare metal and long-lived hosts. Most of this estate is not Kubernetes, and the reflexes that assume an orchestrator will not transfer cleanlyConfiguration management and CI at production scale — Ansible, GitLab CI, Jenkins or close equivalentsThe judgement to design measurement for machines you do not own and cannot scrape: push telemetry, sampling, clock skew, partial reporting, and the privacy constraints that come with running on a customer's serverWritten communication that holds up async. This role is 40% telemetry engineering and 40% getting sixty engineers to agree on what "healthy" means; the remaining 20% is refusing to let the answer be a dashboard nobody reads Valuable (Nice-to-haves): Security product background — WAF, EDR, AV, vulnerability management — and the instinct that a security control's SLI is about enforcement, not uptimeMonitoring under audit: SOC 2 CC7.x, ISO 27001 A.8.16, NIST SP 800-137 continuous monitoring. Some of this work is audit evidence and it helps if you have written for that audienceOpenTelemetry, eBPF, SentryCost- and cardinality-aware telemetry designFluency with agentic development tooling — we run a Cursor/Claude-first SDLC with internal and third-party MCP servers, and engineers here are assessed on how well they work with itKubernetes, for the one workload that is on it Not this role Not a DevOps ticket queue, not build-system ownership, not cloud cost management, not the on-call rota for other squads' services First year, in outcomes 30 days: Component inventory with named owners. SLI taxonomy and tiering agreed. 3 pilot components fully instrumented end to end as the reference implementation. 90 days: Collection pipeline in production. Tier-1 components (the ones whose failure is a customer security exposure) carry SLO, alert, runbook, owner. Escalation routing live for tier-1. 180 days: All :70 components have a defined SLI and an owner. Alert taxonomy enforced; page volume and actionable-rate measured and published. Squad on-call operating. 365 days: Mean time to detect a silent control-degradation is under 24 hours, measured, against a 61-day baseline. Error-budget policy influences release decisions. The function is documented well enough that hire #2 and #3 are additive, not archaeological. How we work Remote-first and async across nine time zones. Weekly PO sync and architecture sync; monthly demo and OKR review; quarterly architecture summit. Decisions land as ADRs. Every output carries an owner and a due date. Postmortems are blameless and published, and we correct ourselves on the record when we get something wrong. Benefits What's in it for you? A strong focus on professional development with opportunities for learning and growth:Interesting and challenging projects,Mentor and other knowledge-exchange programs;Fully remote work with flexible working hours, that allows you to schedule your day and work from any location worldwide;Paid 24 days of vacation per year, 10 days of national holidays, and unlimited sick leaves to ensure you maintain a healthy work-life balance;Compensation for private medical insurance;Co-working and gym/sports reimbursement;The opportunity to receive a reward for the most innovative idea that the company can patent, fostering a culture of creativity and innovation By applying for this position, you consent to the processing of your personal data as described in our Privacy Policy (https://cloudlinux.com/candidate-privacy-notice), which provides detailed information on how we maintain and handle your data.