Ensure solution and operations reliability Study Guide — Professional Cloud Architect

Ensure Solution and Operations Reliability

This domain accounts for 12% of the GCP Professional Cloud Architect exam and tests your ability to design, implement, and operate reliable systems on Google Cloud. You must demonstrate practical knowledge of observability tooling (monitoring, logging, tracing), SRE principles such as SLOs and error budgets, incident management practices, and reliability patterns for GKE, Pub/Sub, networking, and data persistence. Expect scenario-based questions where you must choose the right service, configuration, or architecture to meet a stated reliability requirement.

Cloud Monitoring: Custom Metrics, Uptime Checks, and Alerting Policies

Core Concepts

Cloud Monitoring (formerly Stackdriver) ingests metrics from GCP services, VMs, Kubernetes, and custom sources. It is the primary tool for observability and alerting.

Custom metrics allow you to push application-specific data (e.g., queue depth, business KPIs) using the Cloud Monitoring API or OpenTelemetry. Custom metrics use the custom.googleapis.com/ namespace. The API supports up to 10,000 custom metric descriptors per project.

Uptime checks verify availability of external URLs, GCP resources, or internal endpoints (via the Monitoring agent). Checks run from multiple global locations at intervals as low as 1 minute. A resource is considered down when checks from a configurable number of regions fail simultaneously.

Alerting policies define conditions (metric threshold, uptime check failure, log-based metric) and notification channels (email, PagerDuty, Pub/Sub, Slack, webhook). Key sub-components:

• Condition: metric threshold, absence, rate-of-change, or forecast
• Notification channels: must be created before referencing in a policy
• Alert duration: how long a condition must be violated before firing (default-configurable; 0 = immediate)
• Auto-close: alerts auto-close after 7 days if not manually resolved

Common Traps

• Confusing uptime checks (external black-box probing) with health checks used by load balancers (backend routing decisions) — they are separate resources serving different purposes.
• Thinking custom metrics require an agent — they can be written directly via the Cloud Monitoring API or OpenTelemetry SDK.
• Assuming alerting fires instantly — the alert duration window must elapse first.
• Forgetting that notification channels must exist before an alerting policy can reference them via Terraform or API.

Worked Scenario

Requirement: An e-commerce team wants to alert when the number of items in a Redis queue exceeds 10,000 for more than 5 minutes.

Reasoning: Redis queue depth is not a built-in GCP metric. You must instrument the application to publish a custom metric (custom.googleapis.com/redis/queue_depth). Create an alerting policy with a metric threshold condition set to > 10,000 and a duration of 5 minutes. Choose Pub/Sub as the notification channel to trigger a Cloud Function for auto-remediation.

Cloud Logging: Log-Based Metrics, Log Sinks, and Log Retention

Core Concepts

Cloud Logging collects, stores, and queries logs from GCP services, VMs (via the Logging agent), and applications. Logs are stored in log buckets within a project's log router.

Log-based metrics convert log entries matching a filter into Cloud Monitoring metrics (counter or distribution). Useful for alerting on log patterns (e.g., error rate, specific exception count). They live in the logging.googleapis.com/user/ namespace.

Log sinks export log entries to destinations outside Cloud Logging storage:

Log retention defaults and configuration:

The _Required bucket holds Admin Activity and System Event audit logs and cannot be deleted or have retention reduced.

Log exclusions prevent specific log entries from being ingested (reducing cost). Applied at the log router level before storage.

Common Traps

• Confusing log-based metrics (Cloud Monitoring metrics derived from logs) with log analytics (BigQuery-style queries in Logging) — they are different features.
• Assuming the _Required bucket retention can be extended — it cannot be configured.
• Forgetting that log sinks are not retroactive — they only export logs generated after the sink is created.
• Mixing up exclusion filters (prevent ingestion) vs. sink filters (select what to export) — both use the same filter syntax but serve opposite purposes.
• Thinking you can reduce _Required bucket retention to save costs — you cannot.