Compare Push-Based CI/CD vs. Pull-Based GitOps for Kubernetes cluster security and drift detection.

Push-Based CI/CD (e.g., GitHub Actions / Jenkins running `kubectl apply` or `helm upgrade`): Requires storing powerful cluster-admin Kubernetes credentials inside the external CI server and exposing the K8s API server to CI runners; furthermore, CI only runs when a commit happens, so it never detects manual `kubectl edit` drift in the cluster! Pull-Based GitOps (ArgoCD / Flux running INSIDE the cluster): The agent pulls outbound from Git, so external CI never needs cluster credentials (CI only builds the image and opens a PR updating the image digest in the GitOps config repo), and the agent continuously detects and reverts live cluster drift.

What causes a 'High Cardinality Explosion' in Prometheus TSDB, and why must you never put `user_id`, `email`, or full URLs into metric labels?

In Prometheus's Time-Series Database (TSDB), every unique combination of metric name + label key-value pairs creates a separate physical time series stored in memory/index! If a metric has `method` (5 values) $\times$ `status` (10 values) = 50 series (safe). If you add a label `user_id` with 1,000,000 users, Prometheus creates $50 \times 1,000,000 = 50,000,000$ distinct time series, immediately exhausting Prometheus RAM (`OOMKilled`). High-cardinality identifiers (`user_id`, `request_id`, `ip`) belong in Logs or Trace Spans (or Prometheus Exemplars), NEVER in metric labels.

Compare the RED Method (Tom Wilkie) and the USE Method (Brendan Gregg), and state when to apply each.

RED Method (for request-driven Microservices / APIs): Measure for every service: 1. Rate (requests per second), 2. Errors (number of failing requests per second), and 3. Duration (distribution/histogram of request latency: P50, P95, P99). USE Method (for underlying Hardware / Infrastructure resources like CPU, Memory, Disks, Network NICs): Measure for every hardware resource: 1. Utilization (% of time the resource was busy), 2. Saturation (degree of queued work that cannot be serviced immediately, e.g., runqueue length), and 3. Errors (count of hardware/driver error events).

What is an Error Budget in SRE, how is it calculated, and how does it resolve the organizational conflict between Product Developers and Operations?

Error Budget = $100\% - \text{SLO}$ (e.g., for a $99.9\%$ SLO over 30 days, the Error Budget is $0.1\%$ of requests or 43.2 minutes). Instead of Product wanting maximum feature velocity and Ops wanting zero changes to protect stability, both teams agree on an Error Budget policy: so long as the service has remaining Error Budget, developers can release rapidly and take calculated risks; if outages burn through 100% of the Error Budget in the window, feature launches freeze (except P0 security bugs) and engineering effort pivots to reliability improvements until the budget replenishes.

How does Prometheus's Pull-based architecture work, and when should you use the Prometheus `Pushgateway`?

Prometheus periodically scrapes (HTTP `GET /metrics`) configured targets discovered via Kubernetes Service Discovery (`kubernetes_sd_configs`), Consul, or EC2/GCP APIs. Pull makes it trivial to tell if a target is down (`up == 0`) and prevents overloaded targets from DDoS-ing the monitoring server. The `Pushgateway` should ONLY be used for short-lived, ephemeral batch/cron jobs that finish and exit before Prometheus's scrape interval (`15s–60s`) can pull from them (never use Pushgateway for long-running services).

What is OpenTelemetry (OTel) and how does the OpenTelemetry Collector pipeline (`Receivers` -> `Processors` -> `Exporters`) work?

OpenTelemetry is the CNCF vendor-neutral standard combining APIs, SDKs, semantic conventions, and the OTLP (OpenTelemetry Protocol over gRPC `4317` / HTTP `4318`) for Traces, Metrics, and Logs. The OTel Collector decouples apps from observability backends via a 3-stage pipeline: 1. `Receivers` (ingest telemetry via OTLP, Prometheus scrape, Host metrics, Filelog); 2. `Processors` (batching, memory limiting, PII redaction, tail-based sampling, adding k8s metadata via `k8sattributes`); 3. `Exporters` (fan-out to Prometheus, Jaeger/Tempo, Loki, Datadog, or Cloud Operations).

Explain the architecture of ArgoCD (`argocd-server`, `argocd-repo-server`, `argocd-application-controller`) and the `Application` vs. `ApplicationSet` CRDs.

1. `argocd-server`: API/gRPC server and Web UI. 2. `argocd-repo-server`: Clones Git repositories and renders Helm charts / Kustomize overlays / Jsonnet into plain Kubernetes YAML manifests. 3. `argocd-application-controller`: Continuously compares the rendered target manifests against live Kubernetes cluster objects, reporting `Synced` vs `OutOfSync` and `Healthy` vs `Degraded`. The `Application` CRD maps ONE Git `source` to ONE cluster/namespace `destination`; the `ApplicationSet` CRD uses Generators (Git directory, Cluster, Matrix, Pull Request) to template dozens of `Applications` automatically.

What is Multi-Window, Multi-Burn-Rate SLO Alerting, and why is it far superior to alerting on simple instantaneous error rates?

Burn Rate measures how fast a service is consuming its 30-day Error Budget relative to a steady $1\times$ rate (a burn rate of $14.4\times$ exhausts the entire 30-day budget in 50 hours / 2% in 1 hour!). Recommended by the Google SRE Workbook, Multi-Window Multi-Burn-Rate alerting pairs a Long Window with a Short Window (typically $1/12$th the long window): e.g., Page on-call ONLY if the 1-hour burn rate $> 14.4\times$ AND the 5-minute burn rate $> 14.4\times$. The 1-hour window guarantees high precision (significant budget is actually burning), while the 5-minute window ensures the alert clears within minutes once the outage ends!

How does decoupling 'Deployment' from 'Release' using Feature Flags (Feature Toggles) improve CI/CD safety and trunk-based development?

Deployment is the technical act of pushing and running new binary/container code in production; Release is the business decision to expose a new feature to end users. With Feature Flags (e.g., LaunchDarkly, OpenFeature, Unleash), engineers merge incomplete or high-risk code into `main` daily deployed behind an `OFF` flag, then progressively enable the feature at runtime for internal QA users, 5% of beta customers, or specific regions—and can kill a broken feature in milliseconds via a config toggle without rolling back or redeploying containers!

What are the principles of a Blameless Postmortem (Incident Retrospective) in SRE culture?

A Blameless Postmortem assumes that engineers Involved in an incident acted with good intentions based on the information and tools available to them at the time. Instead of blaming or punishing 'human error' (which incentivizes engineers to hide mistakes), the postmortem asks: 'Why did the system's guardrails, UI, CI checks, or peer review allow a single human command to cause a production outage?' Every postmortem documents a timeline of events, impact (Error Budget burned), root/contributing technical causes, what went well/poorly, and tracked Action Items with owners and SLAs.

How does Software Supply Chain Security work in modern CI/CD pipelines (SBOM, Cosign image signing, SLSA provenance, Admission Verification)?

1. Generate an SBOM (Software Bill of Materials in CycloneDX or SPDX format via `syft`) listing every transitive dependency in the container image, and scan for CVEs (`trivy`/`grype`). 2. Generate attested SLSA (Supply-chain Levels for Software Artifacts) build provenance proving the image was built by a trusted CI runner from a specific Git commit. 3. Cryptographically sign the OCI image digest (and attach the SBOM/attestation) using Sigstore `cosign` (supporting keyless OIDC signing with Fulcio + Rekor transparency log). 4. Enforce signature verification at Kubernetes deploy time via Kyverno or Sigstore `policy-controller`.

How does Prometheus `Alertmanager` handle alert Grouping, Inhibition, and Silences, and what does the `for:` clause in a Prometheus Alert Rule do?

In a Prometheus Alert Rule, `for: 5m` keeps an alert in `Pending` state until the PromQL expression remains continuously true for 5 full minutes before transitioning to `Firing` and sending it to `Alertmanager` (preventing flapping pager noise). `Alertmanager` then provides: 1. Grouping: Bundles dozens of simultaneous pod alerts from the same cluster/service into a single consolidated Slack/PagerDuty notification; 2. Inhibition: Suppresses lower-level alerts if a root-cause alert is already firing (e.g., mute 50 pod alerts on Node X if `NodeDown` is firing for Node X); 3. Silences: Time-window mutes based on label matchers during maintenance.

Why should CI/CD pipelines (GitHub Actions, GitLab CI) use OpenID Connect (OIDC) federated tokens instead of stored long-lived Cloud IAM Secrets?

Storing static `AWS_SECRET_ACCESS_KEY`, `AZURE_CREDENTIALS`, or `GCP_SA_KEY` in CI repository secrets creates a permanent credential exfiltration target and requires manual rotation. With OIDC federation, the CI runner requests a short-lived signed JWT from the CI platform's OIDC provider containing claims (`repo:org/app:ref:refs/heads/main`, `environment:prod`) and exchanges it directly with AWS STS / Azure Entra ID / GCP STS for a temporary 15–60 minute cloud access token—zero static secrets stored in CI!

What are the four DORA (DevOps Research and Assessment) key metrics that measure software delivery and operational performance?

Velocity metrics: 1. Deployment Frequency (how often an organization successfully releases to production); 2. Lead Time for Changes (the time it takes a code commit to reach running production). Stability metrics: 3. Change Failure Rate (the percentage of deployments causing a failure in production requiring remediation/rollback); 4. Failed Deployment Recovery Time / MTTR (how long it takes to restore service after a production incident). Elite teams optimize BOTH velocity and stability simultaneously through small batch sizes and automated testing.

How does Grafana Loki differ architecturally from Elasticsearch for log aggregation in Kubernetes?

Elasticsearch parses and builds a full-text inverted index of the entire content of every log message at ingestion time (fast arbitrary full-text search, but high RAM/CPU/disk index overhead). Grafana Loki ('like Prometheus, but for logs') indexes ONLY the small set of metadata labels attached to each log stream (e.g., `{namespace="prod", app="checkout", pod="checkout-7b9"}`, matching your Prometheus metric labels!) and stores the compressed raw log chunks in cheap object storage (S3/GCS/Blob), using parallel grep (`LogQL`) over matching time/label chunks.

Compare Mean Time Between Failures (MTBF), Mean Time To Detect (MTTD), and Mean Time To Recover/Resolve (MTTR), and why modern DORA/SRE metrics prioritize MTTR.

MTBF measures average time between system failures; MTTD measures how long it takes observability/alerting to detect an anomaly after it begins; MTTR measures the total time from failure onset to full restoration of customer service. In complex cloud-native systems changing hundreds of times a day, preventing 100% of failures is impossible; minimizing MTTR via automated canary rollbacks, feature flags, chaos engineering, and fast observability reduces customer impact far more reliably than trying to freeze changes to maximize MTBF.

What are the Three Pillars of Observability, and what distinct question does each pillar answer during an incident?

1. Metrics (numeric time-series aggregations stored in Prometheus/VictoriaMetrics/Mimir): Answers 'WHAT is broken and WHEN did it start?' (fast, constant-overhead alerting on symptom trends like error rate or P99 latency). 2. Logs (structured JSON discrete event records stored in Loki/Elasticsearch/ClickHouse): Answers 'WHY did this specific instance fail?' (exact stack trace, error message, and context). 3. Distributed Traces (end-to-end request spans across microservices stored in Tempo/Jaeger): Answers 'WHERE in the distributed call graph is the latency bottleneck or failure occurring?'

How do Sync Waves and Resource Hooks (`PreSync`, `Sync`, `PostSync`, `SyncFail`) control deployment ordering in ArgoCD?

Sync Waves (`argocd.argoproj.io/sync-wave: "-1"`) order how resources are applied: lower (including negative) integers are applied first, and ArgoCD waits for all resources in wave $N$ to become `Healthy` before applying wave $N+1$ (e.g., Wave `-1`: Namespace & CRDs -> Wave `0`: DB Deployment -> Wave `1`: Backend App). Resource Hooks (`argocd.argoproj.io/hook: PreSync`) run one-off Kubernetes `Jobs` at specific lifecycle phases—for example, running a database schema migration Job in `PreSync` before new application pods roll out in `Sync`, or sending a Slack alert on `SyncFail`.

How does W3C Trace Context propagation (`traceparent` HTTP header) link spans across microservices, and how do Head Sampling vs. Tail Sampling differ?

Every instrumented service injects and extracts the W3C standard `traceparent` HTTP/gRPC header (`00- - - `), linking all downstream child spans to the same `trace_id`. Head Sampling makes the keep/drop decision at the very first ingress span (e.g., randomly sample 1% of requests—cheap and stateless, but misses 99% of rare error traces!). Tail Sampling buffers spans of a trace in an OTel Collector tier and makes the sampling decision AFTER the request finishes—keeping 100% of traces containing an HTTP 5xx error or latency >2s, while sampling only 1% of boring 200 OK traces.

What are the Four Golden Signals of monitoring defined in the Google SRE Book?

1. Latency: The time it takes to service a request—crucially distinguishing between the latency of successful requests vs. the latency of failed requests (a fast `500 Internal Server Error` in 1ms will artificially lower your average latency if not separated!). 2. Traffic: How much demand is being placed on the system (e.g., HTTP requests/sec, Kafka messages/sec). 3. Errors: The rate of requests that fail—explicitly (HTTP `5xx`), implicitly (HTTP `200 OK` returning wrong/empty payload), or by policy (exceeding SLA latency target). 4. Saturation: How 'full' your most constrained resource is (CPU run queue, memory, disk I/O, DB connection pool at 95%).

Compare the four progressive CI/CD deployment strategies: Rolling Update, Blue/Green, Canary, and Shadow (Dark Traffic Mirroring).

1. Rolling Update: Gradually replaces old instances with new ones in-place (zero extra infra cost, but slow rollback and mixed versions serve live traffic). 2. Blue/Green: Provisions a full duplicate 'Green' environment alongside live 'Blue', runs smoke tests, then flips the Load Balancer/router 100% from Blue to Green (instant rollback by flipping back; 2x infra cost during cutover). 3. Canary: Routes a small fraction (e.g., 1% -> 5% -> 25% -> 100%) of live traffic to the new version while monitoring error/latency SLIs (Argo Rollouts / Flagger). 4. Shadow / Dark Launch: Mirrors a duplicate copy of live production requests to v2 asynchronously without returning v2's responses to users.

What is the difference between `prune: true` and `selfHeal: true` in an ArgoCD `Application` automated `syncPolicy`?

`prune: true`: When a manifest file is DELETED from the Git repository, ArgoCD automatically deletes the corresponding live resource from the Kubernetes cluster (by default `prune` is `false` to prevent accidental deletions). `selfHeal: true`: When someone manually modifies or deletes a live resource directly inside the Kubernetes cluster (`kubectl edit`/`delete`) causing live drift from Git, ArgoCD automatically overwrites the manual change and reconciles the cluster back to the exact state in Git.

Compare Secret Scanning, SAST, DAST, and SCA in a DevSecOps ('Shift-Left') CI/CD pipeline.

1. Secret Scanning (`gitleaks`, `trufflehog`): Pre-commit and CI hooks blocking accidental commits of API keys, private keys, or tokens. 2. SAST (Static Application Security Testing, e.g., Semgrep, SonarQube, CodeQL): Analyzes proprietary source code/AST for SQLi, XSS, and insecure coding patterns without running the app. 3. SCA (Software Composition Analysis, e.g., Dependabot, Snyk, Trivy): Checks third-party open-source packages (`package-lock.json`, `go.sum`, OS layers) against CVE databases. 4. DAST (Dynamic Application Security Testing, e.g., OWASP ZAP): Probes the running staging application from the outside for runtime vulnerabilities.

How much maximum downtime per 30-day month and per 365-day year corresponds to `99.9%` ('Three Nines'), `99.99%` ('Four Nines'), and `99.999%` ('Five Nines') availability?

`99.9%` (Three Nines, $0.1\%$ error budget): ~43.2 minutes/month or ~8.76 hours/year. `99.95%`: ~21.6 minutes/month or ~4.38 hours/year. `99.99%` (Four Nines, $0.01\%$ error budget): ~4.32 minutes/month or ~52.6 minutes/year. `99.999%` (Five Nines, $0.001\%$ error budget): ~25.9 seconds/month or ~5.26 minutes/year.

How do you calculate the 95th percentile (P95) request latency across all Pods of a service in PromQL using `histogram_quantile()`?

`histogram_quantile(0.95, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))`. Critical PromQL rules: 1. Always apply `rate()` to the `_bucket` counter first over a time window; 2. When aggregating across pods with `sum by (...)`, you MUST include the `le` (less-than-or-equal bucket upper bound) label inside the `by (le, ...)` clause, or `histogram_quantile()` cannot interpolate across buckets!

What are the Four Core Principles of GitOps as defined by the OpenGitOps / CNCF specification?

1. Declarative: A system managed by GitOps must have its desired state expressed declaratively (e.g., K8s YAML, Helm, Kustomize). 2. Versioned and Immutable: Desired state is stored in Git in a way that enforces immutability, versioning, and retains a complete audit trail. 3. Pulled Automatically: Software agents (ArgoCD / Flux) automatically pull the desired state declarations from the source. 4. Continuously Reconciled: Software agents continuously observe actual system state and attempt to apply the desired state (closed-loop drift remediation).

Compare `rate()` and `irate()` in PromQL when querying a Prometheus `Counter` metric.

Both calculate the per-second average rate of increase of a `Counter` over a range vector (e.g., `[5m]`) and automatically handle counter resets to `0` on container restarts. `rate(http_requests_total[5m])` averages over ALL data points between the first and last sample in the 5-minute window—smoothing out brief spikes and ideal for Alerting Rules and slow-moving SLO trends. `irate(http_requests_total[5m])` ('instant rate') looks ONLY at the last TWO data points inside the window—producing a highly responsive, volatile graph for zooming in on rapid spikes, but unsuitable for alerting.

Define and distinguish SLI (Service Level Indicator), SLO (Service Level Objective), and SLA (Service Level Agreement) in Site Reliability Engineering.

SLI (Indicator): A quantitative measurement of a specific aspect of the level of service provided (usually expressed as a ratio: $\frac{\text{Good Events}}{\text{Total Valid Events}} \times 100\%$, e.g., 'fraction of HTTP GET requests returning non-5xx in $< 300\text{ms}). SLO (Objective): The internal target value or range for an SLI over a rolling time window (e.g., '$99.9\%$ of requests over 28 days meet the SLI'). SLA (Agreement): An external legal/business contract with customers defining financial penalties (service credits) if availability drops below a looser threshold (e.g., $99.5\%$, giving engineering an early warning buffer below the internal $99.9\%$ SLO).

Why are Average (Mean) Latency metrics misleading for SREs, and why should P50, P95, and P99 (Tail Latency) percentiles be used instead?

Latency distributions in web systems are heavily right-skewed (non-normal). If 99 out of 100 requests take `10ms` and 1 request hits a GC pause or database lock taking `10,000ms` (10 seconds), the average latency is ~`110ms`—looking healthy while hiding the fact that 1% of requests are timing out (and in a microservice fan-out where one user page load calls 50 backend services, a 1% per-call tail latency results in $1 - 0.99^{50} = 39.5\%$ of user page loads experiencing that 10s slow tail!). P95 and P99 directly measure the worst user experiences.

Compare the four Prometheus Metric Types: `Counter`, `Gauge`, `Histogram`, and `Summary`.

1. `Counter`: Cumulative metric that ONLY monotonically increases (or resets to `0` on process restart), e.g., `http_requests_total`, `errors_total`. 2. `Gauge`: Point-in-time numeric value that can go BOTH up and down, e.g., `node_memory_Active_bytes`, `cpu_temperature`, `queue_depth`. 3. `Histogram`: Samples observations (like request duration) into configurable server-side cumulative buckets (`_bucket{le="0.5"}`), plus `_sum` and `_count`; CAN be aggregated across multiple Pod instances! 4. `Summary`: Calculates streaming quantiles (`phi`) on the client side; CANNOT be mathematically aggregated across multiple instances!

How is 'Toil' defined in Site Reliability Engineering, and what is the Google SRE target cap for toil?

Toil is operational work tied to running a production service that tends to be: 1. Manual, 2. Repetitive, 3. Automatable, 4. Tactical (interrupt-driven rather than strategic), 5. Devoid of enduring long-term value, and 6. Scales linearly as the service/traffic grows ($O(n)$). Google SRE targets keeping toil below 50% of each SRE's time, dedicating at least 50% of time to software engineering projects that permanently eliminate toil through automation and self-healing systems.

How do you safely store encrypted secrets inside a public or private GitOps repository using Sealed Secrets vs. SOPS vs. External Secrets Operator (ESO)?

1. Bitnami Sealed Secrets: Asymmetric crypto (`kubeseal` CLI encrypts a Secret using the cluster controller's public key into a safe-to-commit `SealedSecret` CRD; only the in-cluster controller holds the private key to decrypt it). 2. Mozilla SOPS (Secrets OPerationS): Encrypts only the YAML/JSON *values* (leaving keys readable for git diffs) backed by AWS KMS, GCP KMS, Azure Key Vault, or `age` keys. 3. External Secrets Operator (ESO): Stores NO secret ciphertext in Git at all—Git only stores an `ExternalSecret` CRD containing the key name/path reference, which ESO fetches dynamically at runtime from Vault/AWS Secrets Manager/Key Vault.

CI/CD Pipelines, GitOps (ArgoCD), Prometheus/Grafana Observability & SRE SLIs/SLOs

Senior DevOps and Site Reliability Engineering (SRE) mastery covering deployment strategies (Blue/Green, Canary, Shadow, Feature Flags), GitOps principles, ArgoCD and Flux architecture, the Three Pillars of Observability, OpenTelemetry, Prometheus TSDB and PromQL, the Four Golden Signals, RED/USE methods, and SRE SLIs/SLOs/Error Budgets.

Level: advanced
Languages: en → en
Cards: 32
Markdown: CI/CD Pipelines, GitOps (ArgoCD), Prometheus/Grafana Observability & SRE SLIs/SLOs (.md)

Flashcards (32)

#	Front (Term / Question)	Back (Definition / Answer)
1	Compare Push-Based CI/CD vs. Pull-Based GitOps for Kubernetes cluster security and drift detection.	Push-Based CI/CD (e.g., GitHub Actions / Jenkins running `kubectl apply` or `helm upgrade`): Requires storing powerful cluster-admin Kubernetes credentials inside the external CI server and exposing the K8s API server to CI runners; furthermore, CI only runs when a commit happens, so it never detects manual `kubectl edit` drift in the cluster! Pull-Based GitOps (ArgoCD / Flux running INSIDE the cluster): The agent pulls outbound from Git, so external CI never needs cluster credentials (CI only builds the image and opens a PR updating the image digest in the GitOps config repo), and the agent continuously detects and reverts live cluster drift.
2	What causes a 'High Cardinality Explosion' in Prometheus TSDB, and why must you never put `user_id`, `email`, or full URLs into metric labels?	In Prometheus's Time-Series Database (TSDB), every unique combination of metric name + label key-value pairs creates a separate physical time series stored in memory/index! If a metric has `method` (5 values) $\times$ `status` (10 values) = 50 series (safe). If you add a label `user_id` with 1,000,000 users, Prometheus creates $50 \times 1,000,000 = 50,000,000$ distinct time series, immediately exhausting Prometheus RAM (`OOMKilled`). High-cardinality identifiers (`user_id`, `request_id`, `ip`) belong in Logs or Trace Spans (or Prometheus Exemplars), NEVER in metric labels.
3	Compare the RED Method (Tom Wilkie) and the USE Method (Brendan Gregg), and state when to apply each.	RED Method (for request-driven Microservices / APIs): Measure for every service: 1. Rate (requests per second), 2. Errors (number of failing requests per second), and 3. Duration (distribution/histogram of request latency: P50, P95, P99). USE Method (for underlying Hardware / Infrastructure resources like CPU, Memory, Disks, Network NICs): Measure for every hardware resource: 1. Utilization (% of time the resource was busy), 2. Saturation (degree of queued work that cannot be serviced immediately, e.g., runqueue length), and 3. Errors (count of hardware/driver error events).
4	What is an Error Budget in SRE, how is it calculated, and how does it resolve the organizational conflict between Product Developers and Operations?	Error Budget = $100\% - \text{SLO}$ (e.g., for a $99.9\%$ SLO over 30 days, the Error Budget is $0.1\%$ of requests or 43.2 minutes). Instead of Product wanting maximum feature velocity and Ops wanting zero changes to protect stability, both teams agree on an Error Budget policy: so long as the service has remaining Error Budget, developers can release rapidly and take calculated risks; if outages burn through 100% of the Error Budget in the window, feature launches freeze (except P0 security bugs) and engineering effort pivots to reliability improvements until the budget replenishes.
5	How does Prometheus's Pull-based architecture work, and when should you use the Prometheus `Pushgateway`?	Prometheus periodically scrapes (HTTP `GET /metrics`) configured targets discovered via Kubernetes Service Discovery (`kubernetes_sd_configs`), Consul, or EC2/GCP APIs. Pull makes it trivial to tell if a target is down (`up == 0`) and prevents overloaded targets from DDoS-ing the monitoring server. The `Pushgateway` should ONLY be used for short-lived, ephemeral batch/cron jobs that finish and exit before Prometheus's scrape interval (`15s–60s`) can pull from them (never use Pushgateway for long-running services).
6	What is OpenTelemetry (OTel) and how does the OpenTelemetry Collector pipeline (`Receivers` -> `Processors` -> `Exporters`) work?	OpenTelemetry is the CNCF vendor-neutral standard combining APIs, SDKs, semantic conventions, and the OTLP (OpenTelemetry Protocol over gRPC `4317` / HTTP `4318`) for Traces, Metrics, and Logs. The OTel Collector decouples apps from observability backends via a 3-stage pipeline: 1. `Receivers` (ingest telemetry via OTLP, Prometheus scrape, Host metrics, Filelog); 2. `Processors` (batching, memory limiting, PII redaction, tail-based sampling, adding k8s metadata via `k8sattributes`); 3. `Exporters` (fan-out to Prometheus, Jaeger/Tempo, Loki, Datadog, or Cloud Operations).
7	Explain the architecture of ArgoCD (`argocd-server`, `argocd-repo-server`, `argocd-application-controller`) and the `Application` vs. `ApplicationSet` CRDs.	1. `argocd-server`: API/gRPC server and Web UI. 2. `argocd-repo-server`: Clones Git repositories and renders Helm charts / Kustomize overlays / Jsonnet into plain Kubernetes YAML manifests. 3. `argocd-application-controller`: Continuously compares the rendered target manifests against live Kubernetes cluster objects, reporting `Synced` vs `OutOfSync` and `Healthy` vs `Degraded`. The `Application` CRD maps ONE Git `source` to ONE cluster/namespace `destination`; the `ApplicationSet` CRD uses Generators (Git directory, Cluster, Matrix, Pull Request) to template dozens of `Applications` automatically.
8	What is Multi-Window, Multi-Burn-Rate SLO Alerting, and why is it far superior to alerting on simple instantaneous error rates?	Burn Rate measures how fast a service is consuming its 30-day Error Budget relative to a steady $1\times$ rate (a burn rate of $14.4\times$ exhausts the entire 30-day budget in 50 hours / 2% in 1 hour!). Recommended by the Google SRE Workbook, Multi-Window Multi-Burn-Rate alerting pairs a Long Window with a Short Window (typically $1/12$th the long window): e.g., Page on-call ONLY if the 1-hour burn rate gt; 14.4\times$ AND the 5-minute burn rate gt; 14.4\times$. The 1-hour window guarantees high precision (significant budget is actually burning), while the 5-minute window ensures the alert clears within minutes once the outage ends!
9	How does decoupling 'Deployment' from 'Release' using Feature Flags (Feature Toggles) improve CI/CD safety and trunk-based development?	Deployment is the technical act of pushing and running new binary/container code in production; Release is the business decision to expose a new feature to end users. With Feature Flags (e.g., LaunchDarkly, OpenFeature, Unleash), engineers merge incomplete or high-risk code into `main` daily deployed behind an `OFF` flag, then progressively enable the feature at runtime for internal QA users, 5% of beta customers, or specific regions—and can kill a broken feature in milliseconds via a config toggle without rolling back or redeploying containers!
10	What are the principles of a Blameless Postmortem (Incident Retrospective) in SRE culture?	A Blameless Postmortem assumes that engineers Involved in an incident acted with good intentions based on the information and tools available to them at the time. Instead of blaming or punishing 'human error' (which incentivizes engineers to hide mistakes), the postmortem asks: 'Why did the system's guardrails, UI, CI checks, or peer review allow a single human command to cause a production outage?' Every postmortem documents a timeline of events, impact (Error Budget burned), root/contributing technical causes, what went well/poorly, and tracked Action Items with owners and SLAs.
11	How does Software Supply Chain Security work in modern CI/CD pipelines (SBOM, Cosign image signing, SLSA provenance, Admission Verification)?	1. Generate an SBOM (Software Bill of Materials in CycloneDX or SPDX format via `syft`) listing every transitive dependency in the container image, and scan for CVEs (`trivy`/`grype`). 2. Generate attested SLSA (Supply-chain Levels for Software Artifacts) build provenance proving the image was built by a trusted CI runner from a specific Git commit. 3. Cryptographically sign the OCI image digest (and attach the SBOM/attestation) using Sigstore `cosign` (supporting keyless OIDC signing with Fulcio + Rekor transparency log). 4. Enforce signature verification at Kubernetes deploy time via Kyverno or Sigstore `policy-controller`.
12	How does Prometheus `Alertmanager` handle alert Grouping, Inhibition, and Silences, and what does the `for:` clause in a Prometheus Alert Rule do?	In a Prometheus Alert Rule, `for: 5m` keeps an alert in `Pending` state until the PromQL expression remains continuously true for 5 full minutes before transitioning to `Firing` and sending it to `Alertmanager` (preventing flapping pager noise). `Alertmanager` then provides: 1. Grouping: Bundles dozens of simultaneous pod alerts from the same cluster/service into a single consolidated Slack/PagerDuty notification; 2. Inhibition: Suppresses lower-level alerts if a root-cause alert is already firing (e.g., mute 50 pod alerts on Node X if `NodeDown` is firing for Node X); 3. Silences: Time-window mutes based on label matchers during maintenance.
13	Why should CI/CD pipelines (GitHub Actions, GitLab CI) use OpenID Connect (OIDC) federated tokens instead of stored long-lived Cloud IAM Secrets?	Storing static `AWS_SECRET_ACCESS_KEY`, `AZURE_CREDENTIALS`, or `GCP_SA_KEY` in CI repository secrets creates a permanent credential exfiltration target and requires manual rotation. With OIDC federation, the CI runner requests a short-lived signed JWT from the CI platform's OIDC provider containing claims (`repo:org/app:ref:refs/heads/main`, `environment:prod`) and exchanges it directly with AWS STS / Azure Entra ID / GCP STS for a temporary 15–60 minute cloud access token—zero static secrets stored in CI!
14	What are the four DORA (DevOps Research and Assessment) key metrics that measure software delivery and operational performance?	Velocity metrics: 1. Deployment Frequency (how often an organization successfully releases to production); 2. Lead Time for Changes (the time it takes a code commit to reach running production). Stability metrics: 3. Change Failure Rate (the percentage of deployments causing a failure in production requiring remediation/rollback); 4. Failed Deployment Recovery Time / MTTR (how long it takes to restore service after a production incident). Elite teams optimize BOTH velocity and stability simultaneously through small batch sizes and automated testing.
15	How does Grafana Loki differ architecturally from Elasticsearch for log aggregation in Kubernetes?	Elasticsearch parses and builds a full-text inverted index of the entire content of every log message at ingestion time (fast arbitrary full-text search, but high RAM/CPU/disk index overhead). Grafana Loki ('like Prometheus, but for logs') indexes ONLY the small set of metadata labels attached to each log stream (e.g., `{namespace="prod", app="checkout", pod="checkout-7b9"}`, matching your Prometheus metric labels!) and stores the compressed raw log chunks in cheap object storage (S3/GCS/Blob), using parallel grep (`LogQL`) over matching time/label chunks.
16	Compare Mean Time Between Failures (MTBF), Mean Time To Detect (MTTD), and Mean Time To Recover/Resolve (MTTR), and why modern DORA/SRE metrics prioritize MTTR.	MTBF measures average time between system failures; MTTD measures how long it takes observability/alerting to detect an anomaly after it begins; MTTR measures the total time from failure onset to full restoration of customer service. In complex cloud-native systems changing hundreds of times a day, preventing 100% of failures is impossible; minimizing MTTR via automated canary rollbacks, feature flags, chaos engineering, and fast observability reduces customer impact far more reliably than trying to freeze changes to maximize MTBF.
17	What are the Three Pillars of Observability, and what distinct question does each pillar answer during an incident?	1. Metrics (numeric time-series aggregations stored in Prometheus/VictoriaMetrics/Mimir): Answers 'WHAT is broken and WHEN did it start?' (fast, constant-overhead alerting on symptom trends like error rate or P99 latency). 2. Logs (structured JSON discrete event records stored in Loki/Elasticsearch/ClickHouse): Answers 'WHY did this specific instance fail?' (exact stack trace, error message, and context). 3. Distributed Traces (end-to-end request spans across microservices stored in Tempo/Jaeger): Answers 'WHERE in the distributed call graph is the latency bottleneck or failure occurring?'
18	How do Sync Waves and Resource Hooks (`PreSync`, `Sync`, `PostSync`, `SyncFail`) control deployment ordering in ArgoCD?	Sync Waves (`argocd.argoproj.io/sync-wave: "-1"`) order how resources are applied: lower (including negative) integers are applied first, and ArgoCD waits for all resources in wave $N$ to become `Healthy` before applying wave $N+1$ (e.g., Wave `-1`: Namespace & CRDs -> Wave `0`: DB Deployment -> Wave `1`: Backend App). Resource Hooks (`argocd.argoproj.io/hook: PreSync`) run one-off Kubernetes `Jobs` at specific lifecycle phases—for example, running a database schema migration Job in `PreSync` before new application pods roll out in `Sync`, or sending a Slack alert on `SyncFail`.
19	How does W3C Trace Context propagation (`traceparent` HTTP header) link spans across microservices, and how do Head Sampling vs. Tail Sampling differ?	Every instrumented service injects and extracts the W3C standard `traceparent` HTTP/gRPC header (`00-<128-bit-trace-id>-<64-bit-parent-span-id>-<trace-flags>`), linking all downstream child spans to the same `trace_id`. Head Sampling makes the keep/drop decision at the very first ingress span (e.g., randomly sample 1% of requests—cheap and stateless, but misses 99% of rare error traces!). Tail Sampling buffers spans of a trace in an OTel Collector tier and makes the sampling decision AFTER the request finishes—keeping 100% of traces containing an HTTP 5xx error or latency >2s, while sampling only 1% of boring 200 OK traces.
20	What are the Four Golden Signals of monitoring defined in the Google SRE Book?	1. Latency: The time it takes to service a request—crucially distinguishing between the latency of successful requests vs. the latency of failed requests (a fast `500 Internal Server Error` in 1ms will artificially lower your average latency if not separated!). 2. Traffic: How much demand is being placed on the system (e.g., HTTP requests/sec, Kafka messages/sec). 3. Errors: The rate of requests that fail—explicitly (HTTP `5xx`), implicitly (HTTP `200 OK` returning wrong/empty payload), or by policy (exceeding SLA latency target). 4. Saturation: How 'full' your most constrained resource is (CPU run queue, memory, disk I/O, DB connection pool at 95%).
21	Compare the four progressive CI/CD deployment strategies: Rolling Update, Blue/Green, Canary, and Shadow (Dark Traffic Mirroring).	1. Rolling Update: Gradually replaces old instances with new ones in-place (zero extra infra cost, but slow rollback and mixed versions serve live traffic). 2. Blue/Green: Provisions a full duplicate 'Green' environment alongside live 'Blue', runs smoke tests, then flips the Load Balancer/router 100% from Blue to Green (instant rollback by flipping back; 2x infra cost during cutover). 3. Canary: Routes a small fraction (e.g., 1% -> 5% -> 25% -> 100%) of live traffic to the new version while monitoring error/latency SLIs (Argo Rollouts / Flagger). 4. Shadow / Dark Launch: Mirrors a duplicate copy of live production requests to v2 asynchronously without returning v2's responses to users.
22	What is the difference between `prune: true` and `selfHeal: true` in an ArgoCD `Application` automated `syncPolicy`?	`prune: true`: When a manifest file is DELETED from the Git repository, ArgoCD automatically deletes the corresponding live resource from the Kubernetes cluster (by default `prune` is `false` to prevent accidental deletions). `selfHeal: true`: When someone manually modifies or deletes a live resource directly inside the Kubernetes cluster (`kubectl edit`/`delete`) causing live drift from Git, ArgoCD automatically overwrites the manual change and reconciles the cluster back to the exact state in Git.
23	Compare Secret Scanning, SAST, DAST, and SCA in a DevSecOps ('Shift-Left') CI/CD pipeline.	1. Secret Scanning (`gitleaks`, `trufflehog`): Pre-commit and CI hooks blocking accidental commits of API keys, private keys, or tokens. 2. SAST (Static Application Security Testing, e.g., Semgrep, SonarQube, CodeQL): Analyzes proprietary source code/AST for SQLi, XSS, and insecure coding patterns without running the app. 3. SCA (Software Composition Analysis, e.g., Dependabot, Snyk, Trivy): Checks third-party open-source packages (`package-lock.json`, `go.sum`, OS layers) against CVE databases. 4. DAST (Dynamic Application Security Testing, e.g., OWASP ZAP): Probes the running staging application from the outside for runtime vulnerabilities.
24	How much maximum downtime per 30-day month and per 365-day year corresponds to `99.9%` ('Three Nines'), `99.99%` ('Four Nines'), and `99.999%` ('Five Nines') availability?	`99.9%` (Three Nines, $0.1\%$ error budget): ~43.2 minutes/month or ~8.76 hours/year. `99.95%`: ~21.6 minutes/month or ~4.38 hours/year. `99.99%` (Four Nines, $0.01\%$ error budget): ~4.32 minutes/month or ~52.6 minutes/year. `99.999%` (Five Nines, $0.001\%$ error budget): ~25.9 seconds/month or ~5.26 minutes/year.
25	How do you calculate the 95th percentile (P95) request latency across all Pods of a service in PromQL using `histogram_quantile()`?	`histogram_quantile(0.95, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))`. Critical PromQL rules: 1. Always apply `rate()` to the `_bucket` counter first over a time window; 2. When aggregating across pods with `sum by (...)`, you MUST include the `le` (less-than-or-equal bucket upper bound) label inside the `by (le, ...)` clause, or `histogram_quantile()` cannot interpolate across buckets!
26	What are the Four Core Principles of GitOps as defined by the OpenGitOps / CNCF specification?	1. Declarative: A system managed by GitOps must have its desired state expressed declaratively (e.g., K8s YAML, Helm, Kustomize). 2. Versioned and Immutable: Desired state is stored in Git in a way that enforces immutability, versioning, and retains a complete audit trail. 3. Pulled Automatically: Software agents (ArgoCD / Flux) automatically pull the desired state declarations from the source. 4. Continuously Reconciled: Software agents continuously observe actual system state and attempt to apply the desired state (closed-loop drift remediation).
27	Compare `rate()` and `irate()` in PromQL when querying a Prometheus `Counter` metric.	Both calculate the per-second average rate of increase of a `Counter` over a range vector (e.g., `[5m]`) and automatically handle counter resets to `0` on container restarts. `rate(http_requests_total[5m])` averages over ALL data points between the first and last sample in the 5-minute window—smoothing out brief spikes and ideal for Alerting Rules and slow-moving SLO trends. `irate(http_requests_total[5m])` ('instant rate') looks ONLY at the last TWO data points inside the window—producing a highly responsive, volatile graph for zooming in on rapid spikes, but unsuitable for alerting.
28	Define and distinguish SLI (Service Level Indicator), SLO (Service Level Objective), and SLA (Service Level Agreement) in Site Reliability Engineering.	SLI (Indicator): A quantitative measurement of a specific aspect of the level of service provided (usually expressed as a ratio: $\frac{\text{Good Events}}{\text{Total Valid Events}} \times 100\%$, e.g., 'fraction of HTTP GET requests returning non-5xx in lt; 300\text{ms}#39;). SLO (Objective): The internal target value or range for an SLI over a rolling time window (e.g., '$99.9\%$ of requests over 28 days meet the SLI'). SLA (Agreement): An external legal/business contract with customers defining financial penalties (service credits) if availability drops below a looser threshold (e.g., $99.5\%$, giving engineering an early warning buffer below the internal $99.9\%$ SLO).
29	Why are Average (Mean) Latency metrics misleading for SREs, and why should P50, P95, and P99 (Tail Latency) percentiles be used instead?	Latency distributions in web systems are heavily right-skewed (non-normal). If 99 out of 100 requests take `10ms` and 1 request hits a GC pause or database lock taking `10,000ms` (10 seconds), the average latency is ~`110ms`—looking healthy while hiding the fact that 1% of requests are timing out (and in a microservice fan-out where one user page load calls 50 backend services, a 1% per-call tail latency results in $1 - 0.99^{50} = 39.5\%$ of user page loads experiencing that 10s slow tail!). P95 and P99 directly measure the worst user experiences.
30	Compare the four Prometheus Metric Types: `Counter`, `Gauge`, `Histogram`, and `Summary`.	1. `Counter`: Cumulative metric that ONLY monotonically increases (or resets to `0` on process restart), e.g., `http_requests_total`, `errors_total`. 2. `Gauge`: Point-in-time numeric value that can go BOTH up and down, e.g., `node_memory_Active_bytes`, `cpu_temperature`, `queue_depth`. 3. `Histogram`: Samples observations (like request duration) into configurable server-side cumulative buckets (`_bucket{le="0.5"}`), plus `_sum` and `_count`; CAN be aggregated across multiple Pod instances! 4. `Summary`: Calculates streaming quantiles (`phi`) on the client side; CANNOT be mathematically aggregated across multiple instances!
31	How is 'Toil' defined in Site Reliability Engineering, and what is the Google SRE target cap for toil?	Toil is operational work tied to running a production service that tends to be: 1. Manual, 2. Repetitive, 3. Automatable, 4. Tactical (interrupt-driven rather than strategic), 5. Devoid of enduring long-term value, and 6. Scales linearly as the service/traffic grows ($O(n)$). Google SRE targets keeping toil below 50% of each SRE's time, dedicating at least 50% of time to software engineering projects that permanently eliminate toil through automation and self-healing systems.
32	How do you safely store encrypted secrets inside a public or private GitOps repository using Sealed Secrets vs. SOPS vs. External Secrets Operator (ESO)?	1. Bitnami Sealed Secrets: Asymmetric crypto (`kubeseal` CLI encrypts a Secret using the cluster controller's public key into a safe-to-commit `SealedSecret` CRD; only the in-cluster controller holds the private key to decrypt it). 2. Mozilla SOPS (Secrets OPerationS): Encrypts only the YAML/JSON values (leaving keys readable for git diffs) backed by AWS KMS, GCP KMS, Azure Key Vault, or `age` keys. 3. External Secrets Operator (ESO): Stores NO secret ciphertext in Git at all—Git only stores an `ExternalSecret` CRD containing the key name/path reference, which ESO fetches dynamically at runtime from Vault/AWS Secrets Manager/Key Vault.