Compare Google Cloud Persistent Disks (Standard, Balanced, Performance/SSD, Extreme), Hyperdisk, Local SSD, and Cloud Filestore.

Persistent Disk (PD) / Hyperdisk: Network-attached durable block storage (`pd-standard` HDD, `pd-balanced`, `pd-ssd`, `pd-extreme` / Hyperdisk Extreme with configurable IOPS); survives VM termination, supports snapshots and regional replication (`Regional PD` across 2 zones). Local SSD: Physically attached NVMe/SCSI disk on the host server; ultra-high IOPS/low latency, but EPHEMERAL (data lost if VM stops). Filestore: Managed NAS (NFSv3/v4.1) shared across multiple VMs/GKE pods (`RWX`).

Compare Table Partitioning and Table Clustering in BigQuery, and explain how they work together.

Partitioning divides a table into distinct physical segments based on: 1. Ingestion time (`_PARTITIONTIME`), 2. A `DATE`/`TIMESTAMP`/`DATETIME` column, or 3. An `INTEGER` range (up to 4,000 partitions per table); partition pruning provides a hard upfront guarantee on bytes scanned. Clustering sorts data WITHIN each partition by up to 4 specified columns (e.g., `customer_id`, `country`), colocating related values into storage blocks for dynamic block pruning on high-cardinality filters and aggregations.

What are BigQuery Materialized Views, BI Engine, and BigLake?

Materialized Views: Precomputed views that periodically cache aggregation results and automatically stay fresh when base tables change; BigQuery's smart tuning transparently rewrites base-table queries to hit the materialized view if applicable. BI Engine: Fast in-memory analysis service integrated with Looker Studio/BigQuery for sub-second dashboard queries. BigLake: Unified storage engine enabling fine-grained security across multi-cloud/GCS open formats (Parquet, Iceberg, Delta, ORC).

How does High Availability (HA) work in Google Cloud SQL, and how does it differ from Read Replicas?

Cloud SQL HA (Regional configuration) uses a Primary instance in one zone and a Standby instance in a second zone within the same region, kept in sync via synchronous replication to a Regional Persistent Disk. If the primary zone or instance fails, Cloud SQL automatically fails over to the Standby (typically <60s) on the same IP. Read Replicas use asynchronous replication to offload read queries (within the same or different region) and do not provide automatic HA failover.

How does Google Cloud Spanner achieve external consistency (linearizability) across global datacenters without traditional lock contention bottlenecking reads?

Cloud Spanner uses Google's hardware-assisted `TrueTime` API (combining GPS receivers and atomic rubidium clocks in every datacenter) to bound clock uncertainty $\epsilon$ to a few milliseconds. By waiting out the uncertainty window ($2\epsilon$) during commit ('commit wait') along with Paxos consensus groups and Multi-Version Concurrency Control (MVCC), Spanner guarantees that transaction timestamps strictly reflect real-world causal ordering, enabling lock-free consistent reads across the globe.

Compare Cloud Data Fusion, Datastream, and Database Migration Service (DMS) in Google Cloud.

Cloud Data Fusion: Managed visual, code-free ETL/ELT data integration platform built on open-source CDAP. Datastream: Serverless Change Data Capture (CDC) and replication service that streams real-time inserts/updates/deletes from Oracle, MySQL, PostgreSQL, SQL Server, and Salesforce into BigQuery, Cloud Storage, or Spanner. Database Migration Service (DMS): Serverless service specifically focused on migrating databases (homogeneous or heterogeneous) to Cloud SQL or AlloyDB with minimal downtime.

Why and how should applications use the Cloud SQL Auth Proxy to connect to Cloud SQL?

The Cloud SQL Auth Proxy is a lightweight sidecar/binary that establishes an encrypted TLS 1.3 tunnel to the Cloud SQL instance using IAM permissions (`roles/cloudsql.client`), removing the need to manage static SSL certificates or whitelist changing client IP addresses in Authorized Networks. It also supports automatic IAM database authentication using short-lived OAuth tokens.

How do you achieve High Availability and zero-downtime resizing in Cloud Bigtable?

Each Bigtable instance can have 1 to N Clusters located in different zones or regions, with all clusters sharing data via automatic asynchronous eventual replication. Adding or removing nodes in a Bigtable cluster rebalances tablet pointers (since actual SSTable data lives on Google's Colossus filesystem, decoupled from Bigtable compute nodes) in a few minutes with zero downtime. Routing profiles (Single-cluster vs Multi-cluster routing) manage failover.

Explain the core architecture of Google Cloud Pub/Sub: Topics, Subscriptions, Delivery Semantics, and Message Ordering.

Publishers send messages to a `Topic`. To receive messages, consumers attach to a `Subscription` on that topic. If 1 Subscription has 5 consumers, each message goes to only 1 consumer (competing consumers / load balancing). If 3 Subscriptions exist on 1 Topic, every Subscription gets a copy of every message (fan-out). Pub/Sub provides at-least-once delivery by default (plus optional Exactly-Once delivery and Ordering Keys for FIFO per key).

Compare Customer-Managed Encryption Keys (CMEK), Customer-Supplied Encryption Keys (CSEK), and Google-Managed Default Encryption in GCP.

Google-Managed Default Encryption: Google automatically encrypts all data at rest using AES-256 without customer setup. CMEK: You create and control key rotation/revocation inside Cloud KMS while Google services manage encryption/decryption transparently. CSEK: You generate and store raw AES-256 keys on-premises and pass the key in every API call; Google never stores the key on disk (only in RAM during the operation).

Compare Google Cloud Dataflow and Cloud Dataproc for data processing pipelines.

Cloud Dataflow: Fully serverless, autoscaling stream and batch processing service based on Apache Beam (Java, Python, Go, SQL); best for new cloud-native unified streaming + batch pipelines with zero cluster management. Cloud Dataproc: Managed Apache Hadoop, Spark, Hive, and Presto/Trino cluster service (clusters spin up in <90 seconds, supports Ephemeral clusters, GCS connector instead of HDFS, and Dataproc Serverless for Spark); best for migrating existing Spark/Hadoop workloads.

What are Signed URLs and Signed Policy Documents in Google Cloud Storage?

A Signed URL (`gcloud storage sign-url` or V4 signing with a Service Account) grants time-limited read (`GET`) or write (`PUT`) permission to a specific GCS object for anyone possessing the URL, even if they don't have a Google account. A Signed Policy Document is used with HTML forms (`POST`) to allow users to upload files directly to GCS while enforcing constraints on file size, content-type, and key prefix.

What are Authorized Views, Authorized Datasets, Row-Level Security, and Column-Level Security (Policy Tags) in BigQuery?

Authorized View / Dataset: Allows sharing the query results of a specific SQL View with users/groups WITHOUT granting them permission to read the underlying source tables directly. Row-Level Security (`CREATE ROW ACCESS POLICY`): Filters which rows a user can see based on their IAM principal (e.g., `SESSION_USER()`). Column-Level Security: Uses Data Catalog Policy Tags and IAM `Fine-Grained Reader` roles to mask or restrict PII/sensitive columns.

Compare Firestore (Native Mode vs. Datastore Mode) and Memorystore in Google Cloud.

Firestore in Native Mode: Serverless document NoSQL database featuring real-time client listeners, offline mobile/web SDK sync, ACID multi-document transactions, and scale-to-zero. Firestore in Datastore Mode: Backwards-compatible with legacy App Engine Datastore API for server-side high-throughput backends (no real-time client listeners). Memorystore: Fully managed in-memory cache service for Redis / Valkey and Memcached providing sub-millisecond data access.

Why is `SELECT *` an anti-pattern in BigQuery, and how does BigQuery on-demand pricing calculate query cost?

Because BigQuery uses Capacitor columnar storage, on-demand pricing bills strictly by the total bytes read from the SPECIFIC columns referenced in your query ($6.25/TB after the first 1 TB/month free). Running `SELECT *` forces BigQuery to scan every column in the table, multiplying cost. Note: Adding a `LIMIT 10` clause does NOT reduce the bytes scanned or cost of a BigQuery query!

How do `ackDeadline`, Dead-Letter Topics (DLQ), and Message Retention / Seek work in Cloud Pub/Sub?

When a subscriber receives a message, it must acknowledge (`Ack`) it before the `ackDeadline` (10–600 seconds) expires, or Pub/Sub redelivers it. If a poison message fails repeatedly (after 5–100 configured delivery attempts), Pub/Sub forwards it to a configured Dead-Letter Topic for inspection. Enabling Message Retention on a subscription (plus `retain_acked_messages`) allows using `Pub/Sub Seek` to replay already-acknowledged messages back to a past timestamp or Snapshot.

Compare BigQuery On-Demand pricing vs. BigQuery Editions (Capacity / Slot Reservations with Autoscaling).

On-Demand pricing: Pay per TB of data scanned ($6.25/TB) with a shared pool typically up to 2,000 slots per project; best for spiky, unpredictable workloads. BigQuery Editions (Standard, Enterprise, Enterprise Plus): Pay per capacity 'Slot-hour' (compute capacity) with a baseline reservation and automatic slot autoscaling (plus 1- or 3-year commitments); provides predictable monthly budgeting and workload management across teams.

What are Windowing (Fixed, Sliding, Session), Watermarks, and Triggers in Apache Beam / Google Cloud Dataflow?

Windowing groups unbounded streaming data by Event Time: 1. Fixed (tumbling, non-overlapping intervals, e.g., every 5m); 2. Sliding (overlapping, e.g., 10m window every 1m); 3. Session (per-key dynamic windows closed after an inactivity gap). A Watermark is the system's heuristic estimate of when all data up to a given event timestamp has arrived. Triggers control when window panes are emitted, including how late-arriving data (`withAllowedLateness`) is handled.

What is the #1 cause of hotspots when designing schemas in Cloud Spanner and Cloud Bigtable, and how do you avoid it?

Monotonically increasing primary keys or row keys (such as auto-incrementing integers `1, 2, 3...` or leading timestamps `2026-11-02...`). Because Spanner and Bigtable sort rows lexicographically by key across key-range splits/tablets, sequential keys route 100% of new writes to a single tablet server. Mitigations: Use UUIDv4, hash prefixing, bit-reversal of sequential IDs, or put high-cardinality attributes (like `user_id` or `sensor_id`) BEFORE the timestamp in a composite key.

Compare Cloud SQL, AlloyDB for PostgreSQL, and Cloud Spanner for relational database workloads in GCP.

Cloud SQL: Managed MySQL, PostgreSQL, and SQL Server (up to 64 TB, vertical scaling, regional HA). AlloyDB for PostgreSQL: Google-engineered PostgreSQL-compatible engine with decoupled storage/compute, columnar engine for hybrid OLTP+OLAP, 4x faster transactional throughput than standard Postgres, 99.99% SLA. Cloud Spanner: Globally distributed, horizontally scalable relational database with ACID transactions, SQL queries, TrueTime external consistency, and up to 99.999% multi-regional SLA.

Compare Regional, Dual-Region, and Multi-Region location types in Google Cloud Storage.

Regional: Data stored redundantly across zones within a single GCP region (lowest cost, co-located with Compute Engine/GKE for lowest latency and zero inter-region network charges). Dual-Region: Data replicated across a specific pair of regions (supports Turbo Replication with a 15-minute RPO SLA). Multi-Region: Data replicated across a large geographic area (`us`, `eu`, `asia`) with 99.95% SLA (Standard) and geo-redundancy.

What is Google Cloud Bigtable and what workloads is it designed for?

Cloud Bigtable is Google's sparsely populated, wide-column NoSQL database (the system underlying HBase) designed for petabyte-scale workloads requiring consistent single-digit millisecond latency and high throughput (millions of ops/sec). Ideal for IoT time-series telemetry, financial market ticks, ad-tech clickstreams, and geospatial/graph datasets. It indexes data by a single lexicographically sorted Row Key (no multi-row SQL joins).

What is Uniform Bucket-Level Access vs. Fine-Grained Access Control in Google Cloud Storage?

Uniform Bucket-Level Access (recommended): Disables legacy object-level Access Control Lists (ACLs) and manages all access uniformly via GCP IAM policies at the bucket level (also required to use IAM Conditions and Workload Identity cleanly). Fine-Grained Access: Combines IAM policies with individual object-level and bucket-level POSIX-style ACLs, which is harder to audit centrally.

Compare Pull, StreamingPull, Push, and Export (BigQuery / Cloud Storage) subscriptions in Cloud Pub/Sub.

Pull / StreamingPull: Subscribers initiate requests (HTTP/gRPC) to fetch messages at their own pace and send `Ack` IDs; best for high-throughput workers on GKE/GCE/Dataflow. Push: Pub/Sub sends an HTTPS `POST` webhook for each message to a public/Cloud Run endpoint; HTTP `2xx` acts as `Ack`, while `4xx/5xx` triggers exponential backoff retry. Export Subscriptions: Write messages directly from Pub/Sub into BigQuery tables or Cloud Storage buckets WITHOUT needing a Dataflow pipeline.

How does Object Lifecycle Management work in Google Cloud Storage, and what actions and conditions are supported?

Object Lifecycle Management applies declarative JSON rules to a bucket to automatically transition or delete objects. Supported Actions: `Delete`, `SetStorageClass` (downgrade from Standard -> Nearline -> Coldline -> Archive), and `AbortIncompleteMultipartUpload`. Supported Conditions: `age` (days since creation), `createdBefore`, `isLive` (versioned objects), `numNewerVersions`, `matchesPrefix`/`matchesSuffix`, and `daysSinceCustomTime`.

How can you preview data in BigQuery without incurring query scan costs?

Instead of running `SELECT * FROM dataset.table LIMIT 10`, use: 1. The 'Preview' tab in the BigQuery Cloud Console UI (free, uses `tabledata.list` API); 2. The CLI command `bq head -n 10 project:dataset.table`; or 3. Run a dry-run (`--dry_run`) first to estimate exact bytes processed before executing a query.

How does Sensitive Data Protection (formerly Cloud DLP) de-identify PII in Google Cloud pipelines?

Sensitive Data Protection scans structured (BigQuery, Spanner) and unstructured (GCS) data using 150+ built-in `InfoTypes` (credit cards, SSNs, emails). It supports four main de-identification techniques: 1. Redaction/Masking (replacing characters with `*`); 2. Replacement (replacing with `[EMAIL]`); 3. Crypto-based Deterministic Tokenization (AES-SIV or Format-Preserving Encryption FPE-FFX, preserving referential integrity for SQL joins); 4. Bucketing / Date Shifting.

Compare the four Google Cloud Storage (GCS) storage classes: Standard, Nearline, Coldline, and Archive.

All four classes offer the same millisecond first-byte latency, high durability (11 nines), and global API! They differ by pricing and minimum storage duration: 1. Standard: Hot data, no minimum duration, no retrieval fee; 2. Nearline: Accessed <1x/month, 30-day min duration, per-GB retrieval fee; 3. Coldline: Accessed <1x/quarter, 90-day min duration, higher retrieval fee; 4. Archive: Accessed <1x/year, 365-day min duration, lowest storage cost and highest retrieval fee.

Compare Cloud Composer, Cloud Workflows, and Cloud Scheduler for orchestration in GCP.

Cloud Composer: Managed Apache Airflow running on GKE; best for complex, multi-step big data ETL/ELT DAGs across GCP and multi-cloud. Cloud Workflows: Serverless, low-latency HTTP/API state machine orchestrator (YAML/JSON) that coordinates microservices, Cloud Run, and GCP APIs with sub-second steps and scales to zero. Cloud Scheduler: Managed cron service that triggers a single HTTP endpoint, Pub/Sub topic, or App Engine target on a schedule.

How do Log Buckets, Log Routers, and Log Sinks work in Google Cloud Logging?

Every log entry ingested into a GCP Project passes through the Log Router, which evaluates inclusion and exclusion filters. Matching logs are routed via Log Sinks to: 1. `_Required` Log Bucket (Admin Activity, System Event, Access Transparency; fixed 400-day retention, free); 2. `_Default` Log Bucket (30-day default retention, configurable); 3. Custom Log Buckets (with Log Analytics / BigQuery linked datasets); 4. Cloud Storage (archival); 5. BigQuery (SQL analysis); or 6. Pub/Sub (external SIEM export).

How does BigQuery's decoupled architecture work under the hood (Capacitor, Colossus, Borg/Dremel, Jupiter)?

BigQuery completely separates storage from compute: 1. Colossus (Google's global distributed filesystem) stores data in Capacitor (a proprietary columnar format that compresses and operates directly on encoded columns); 2. Dremel (running on Borg) allocates dynamic trees of stateless execution units called 'Slots' to execute SQL in parallel; 3. Jupiter (petabit-per-second datacenter network) shuffles intermediate data between slots directly in memory.

What is Google Cloud Operations Suite (formerly Stackdriver) and what are the four primary types of Cloud Audit Logs?

Cloud Operations Suite includes Cloud Logging, Cloud Monitoring, Cloud Trace, Cloud Profiler, and Error Reporting. The 4 Cloud Audit Log types are: 1. Admin Activity logs (WHO modified resource configurations; always on, free, retained 400 days); 2. Data Access logs (WHO read/wrote data inside resources; disabled by default except BigQuery because of high volume/cost); 3. System Event logs (Google-initiated maintenance; always on, free, 400 days); 4. Policy Denied logs (VPC-SC or IAM rejections).

GCP Storage, BigQuery, Pub/Sub, Dataflow & Cloud SQL

Advanced Google Cloud data engineering, storage, and database architecture covering Cloud Storage classes and lifecycle rules, Persistent Disks vs Filestore, Cloud SQL vs Cloud Spanner vs Bigtable vs Firestore, BigQuery internals (partitioning, clustering, slots), Pub/Sub, and Dataflow vs Dataproc.

Level: advanced
Languages: en → en
Cards: 32
Markdown: GCP Storage, BigQuery, Pub/Sub, Dataflow & Cloud SQL (.md)

Flashcards (32)

#	Front (Term / Question)	Back (Definition / Answer)
1	Compare Google Cloud Persistent Disks (Standard, Balanced, Performance/SSD, Extreme), Hyperdisk, Local SSD, and Cloud Filestore.	Persistent Disk (PD) / Hyperdisk: Network-attached durable block storage (`pd-standard` HDD, `pd-balanced`, `pd-ssd`, `pd-extreme` / Hyperdisk Extreme with configurable IOPS); survives VM termination, supports snapshots and regional replication (`Regional PD` across 2 zones). Local SSD: Physically attached NVMe/SCSI disk on the host server; ultra-high IOPS/low latency, but EPHEMERAL (data lost if VM stops). Filestore: Managed NAS (NFSv3/v4.1) shared across multiple VMs/GKE pods (`RWX`).
2	Compare Table Partitioning and Table Clustering in BigQuery, and explain how they work together.	Partitioning divides a table into distinct physical segments based on: 1. Ingestion time (`_PARTITIONTIME`), 2. A `DATE`/`TIMESTAMP`/`DATETIME` column, or 3. An `INTEGER` range (up to 4,000 partitions per table); partition pruning provides a hard upfront guarantee on bytes scanned. Clustering sorts data WITHIN each partition by up to 4 specified columns (e.g., `customer_id`, `country`), colocating related values into storage blocks for dynamic block pruning on high-cardinality filters and aggregations.
3	What are BigQuery Materialized Views, BI Engine, and BigLake?	Materialized Views: Precomputed views that periodically cache aggregation results and automatically stay fresh when base tables change; BigQuery's smart tuning transparently rewrites base-table queries to hit the materialized view if applicable. BI Engine: Fast in-memory analysis service integrated with Looker Studio/BigQuery for sub-second dashboard queries. BigLake: Unified storage engine enabling fine-grained security across multi-cloud/GCS open formats (Parquet, Iceberg, Delta, ORC).
4	How does High Availability (HA) work in Google Cloud SQL, and how does it differ from Read Replicas?	Cloud SQL HA (Regional configuration) uses a Primary instance in one zone and a Standby instance in a second zone within the same region, kept in sync via synchronous replication to a Regional Persistent Disk. If the primary zone or instance fails, Cloud SQL automatically fails over to the Standby (typically <60s) on the same IP. Read Replicas use asynchronous replication to offload read queries (within the same or different region) and do not provide automatic HA failover.
5	How does Google Cloud Spanner achieve external consistency (linearizability) across global datacenters without traditional lock contention bottlenecking reads?	Cloud Spanner uses Google's hardware-assisted `TrueTime` API (combining GPS receivers and atomic rubidium clocks in every datacenter) to bound clock uncertainty $\epsilon$ to a few milliseconds. By waiting out the uncertainty window ($2\epsilon$) during commit ('commit wait') along with Paxos consensus groups and Multi-Version Concurrency Control (MVCC), Spanner guarantees that transaction timestamps strictly reflect real-world causal ordering, enabling lock-free consistent reads across the globe.
6	Compare Cloud Data Fusion, Datastream, and Database Migration Service (DMS) in Google Cloud.	Cloud Data Fusion: Managed visual, code-free ETL/ELT data integration platform built on open-source CDAP. Datastream: Serverless Change Data Capture (CDC) and replication service that streams real-time inserts/updates/deletes from Oracle, MySQL, PostgreSQL, SQL Server, and Salesforce into BigQuery, Cloud Storage, or Spanner. Database Migration Service (DMS): Serverless service specifically focused on migrating databases (homogeneous or heterogeneous) to Cloud SQL or AlloyDB with minimal downtime.
7	Why and how should applications use the Cloud SQL Auth Proxy to connect to Cloud SQL?	The Cloud SQL Auth Proxy is a lightweight sidecar/binary that establishes an encrypted TLS 1.3 tunnel to the Cloud SQL instance using IAM permissions (`roles/cloudsql.client`), removing the need to manage static SSL certificates or whitelist changing client IP addresses in Authorized Networks. It also supports automatic IAM database authentication using short-lived OAuth tokens.
8	How do you achieve High Availability and zero-downtime resizing in Cloud Bigtable?	Each Bigtable instance can have 1 to N Clusters located in different zones or regions, with all clusters sharing data via automatic asynchronous eventual replication. Adding or removing nodes in a Bigtable cluster rebalances tablet pointers (since actual SSTable data lives on Google's Colossus filesystem, decoupled from Bigtable compute nodes) in a few minutes with zero downtime. Routing profiles (Single-cluster vs Multi-cluster routing) manage failover.
9	Explain the core architecture of Google Cloud Pub/Sub: Topics, Subscriptions, Delivery Semantics, and Message Ordering.	Publishers send messages to a `Topic`. To receive messages, consumers attach to a `Subscription` on that topic. If 1 Subscription has 5 consumers, each message goes to only 1 consumer (competing consumers / load balancing). If 3 Subscriptions exist on 1 Topic, every Subscription gets a copy of every message (fan-out). Pub/Sub provides at-least-once delivery by default (plus optional Exactly-Once delivery and Ordering Keys for FIFO per key).
10	Compare Customer-Managed Encryption Keys (CMEK), Customer-Supplied Encryption Keys (CSEK), and Google-Managed Default Encryption in GCP.	Google-Managed Default Encryption: Google automatically encrypts all data at rest using AES-256 without customer setup. CMEK: You create and control key rotation/revocation inside Cloud KMS while Google services manage encryption/decryption transparently. CSEK: You generate and store raw AES-256 keys on-premises and pass the key in every API call; Google never stores the key on disk (only in RAM during the operation).
11	Compare Google Cloud Dataflow and Cloud Dataproc for data processing pipelines.	Cloud Dataflow: Fully serverless, autoscaling stream and batch processing service based on Apache Beam (Java, Python, Go, SQL); best for new cloud-native unified streaming + batch pipelines with zero cluster management. Cloud Dataproc: Managed Apache Hadoop, Spark, Hive, and Presto/Trino cluster service (clusters spin up in <90 seconds, supports Ephemeral clusters, GCS connector instead of HDFS, and Dataproc Serverless for Spark); best for migrating existing Spark/Hadoop workloads.
12	What are Signed URLs and Signed Policy Documents in Google Cloud Storage?	A Signed URL (`gcloud storage sign-url` or V4 signing with a Service Account) grants time-limited read (`GET`) or write (`PUT`) permission to a specific GCS object for anyone possessing the URL, even if they don't have a Google account. A Signed Policy Document is used with HTML forms (`POST`) to allow users to upload files directly to GCS while enforcing constraints on file size, content-type, and key prefix.
13	What are Authorized Views, Authorized Datasets, Row-Level Security, and Column-Level Security (Policy Tags) in BigQuery?	Authorized View / Dataset: Allows sharing the query results of a specific SQL View with users/groups WITHOUT granting them permission to read the underlying source tables directly. Row-Level Security (`CREATE ROW ACCESS POLICY`): Filters which rows a user can see based on their IAM principal (e.g., `SESSION_USER()`). Column-Level Security: Uses Data Catalog Policy Tags and IAM `Fine-Grained Reader` roles to mask or restrict PII/sensitive columns.
14	Compare Firestore (Native Mode vs. Datastore Mode) and Memorystore in Google Cloud.	Firestore in Native Mode: Serverless document NoSQL database featuring real-time client listeners, offline mobile/web SDK sync, ACID multi-document transactions, and scale-to-zero. Firestore in Datastore Mode: Backwards-compatible with legacy App Engine Datastore API for server-side high-throughput backends (no real-time client listeners). Memorystore: Fully managed in-memory cache service for Redis / Valkey and Memcached providing sub-millisecond data access.
15	Why is `SELECT *` an anti-pattern in BigQuery, and how does BigQuery on-demand pricing calculate query cost?	Because BigQuery uses Capacitor columnar storage, on-demand pricing bills strictly by the total bytes read from the SPECIFIC columns referenced in your query ($6.25/TB after the first 1 TB/month free). Running `SELECT *` forces BigQuery to scan every column in the table, multiplying cost. Note: Adding a `LIMIT 10` clause does NOT reduce the bytes scanned or cost of a BigQuery query!
16	How do `ackDeadline`, Dead-Letter Topics (DLQ), and Message Retention / Seek work in Cloud Pub/Sub?	When a subscriber receives a message, it must acknowledge (`Ack`) it before the `ackDeadline` (10–600 seconds) expires, or Pub/Sub redelivers it. If a poison message fails repeatedly (after 5–100 configured delivery attempts), Pub/Sub forwards it to a configured Dead-Letter Topic for inspection. Enabling Message Retention on a subscription (plus `retain_acked_messages`) allows using `Pub/Sub Seek` to replay already-acknowledged messages back to a past timestamp or Snapshot.
17	Compare BigQuery On-Demand pricing vs. BigQuery Editions (Capacity / Slot Reservations with Autoscaling).	On-Demand pricing: Pay per TB of data scanned ($6.25/TB) with a shared pool typically up to 2,000 slots per project; best for spiky, unpredictable workloads. BigQuery Editions (Standard, Enterprise, Enterprise Plus): Pay per capacity 'Slot-hour' (compute capacity) with a baseline reservation and automatic slot autoscaling (plus 1- or 3-year commitments); provides predictable monthly budgeting and workload management across teams.
18	What are Windowing (Fixed, Sliding, Session), Watermarks, and Triggers in Apache Beam / Google Cloud Dataflow?	Windowing groups unbounded streaming data by Event Time: 1. Fixed (tumbling, non-overlapping intervals, e.g., every 5m); 2. Sliding (overlapping, e.g., 10m window every 1m); 3. Session (per-key dynamic windows closed after an inactivity gap). A Watermark is the system's heuristic estimate of when all data up to a given event timestamp has arrived. Triggers control when window panes are emitted, including how late-arriving data (`withAllowedLateness`) is handled.
19	What is the #1 cause of hotspots when designing schemas in Cloud Spanner and Cloud Bigtable, and how do you avoid it?	Monotonically increasing primary keys or row keys (such as auto-incrementing integers `1, 2, 3...` or leading timestamps `2026-11-02...`). Because Spanner and Bigtable sort rows lexicographically by key across key-range splits/tablets, sequential keys route 100% of new writes to a single tablet server. Mitigations: Use UUIDv4, hash prefixing, bit-reversal of sequential IDs, or put high-cardinality attributes (like `user_id` or `sensor_id`) BEFORE the timestamp in a composite key.
20	Compare Cloud SQL, AlloyDB for PostgreSQL, and Cloud Spanner for relational database workloads in GCP.	Cloud SQL: Managed MySQL, PostgreSQL, and SQL Server (up to 64 TB, vertical scaling, regional HA). AlloyDB for PostgreSQL: Google-engineered PostgreSQL-compatible engine with decoupled storage/compute, columnar engine for hybrid OLTP+OLAP, 4x faster transactional throughput than standard Postgres, 99.99% SLA. Cloud Spanner: Globally distributed, horizontally scalable relational database with ACID transactions, SQL queries, TrueTime external consistency, and up to 99.999% multi-regional SLA.
21	Compare Regional, Dual-Region, and Multi-Region location types in Google Cloud Storage.	Regional: Data stored redundantly across zones within a single GCP region (lowest cost, co-located with Compute Engine/GKE for lowest latency and zero inter-region network charges). Dual-Region: Data replicated across a specific pair of regions (supports Turbo Replication with a 15-minute RPO SLA). Multi-Region: Data replicated across a large geographic area (`us`, `eu`, `asia`) with 99.95% SLA (Standard) and geo-redundancy.
22	What is Google Cloud Bigtable and what workloads is it designed for?	Cloud Bigtable is Google's sparsely populated, wide-column NoSQL database (the system underlying HBase) designed for petabyte-scale workloads requiring consistent single-digit millisecond latency and high throughput (millions of ops/sec). Ideal for IoT time-series telemetry, financial market ticks, ad-tech clickstreams, and geospatial/graph datasets. It indexes data by a single lexicographically sorted Row Key (no multi-row SQL joins).
23	What is Uniform Bucket-Level Access vs. Fine-Grained Access Control in Google Cloud Storage?	Uniform Bucket-Level Access (recommended): Disables legacy object-level Access Control Lists (ACLs) and manages all access uniformly via GCP IAM policies at the bucket level (also required to use IAM Conditions and Workload Identity cleanly). Fine-Grained Access: Combines IAM policies with individual object-level and bucket-level POSIX-style ACLs, which is harder to audit centrally.
24	Compare Pull, StreamingPull, Push, and Export (BigQuery / Cloud Storage) subscriptions in Cloud Pub/Sub.	Pull / StreamingPull: Subscribers initiate requests (HTTP/gRPC) to fetch messages at their own pace and send `Ack` IDs; best for high-throughput workers on GKE/GCE/Dataflow. Push: Pub/Sub sends an HTTPS `POST` webhook for each message to a public/Cloud Run endpoint; HTTP `2xx` acts as `Ack`, while `4xx/5xx` triggers exponential backoff retry. Export Subscriptions: Write messages directly from Pub/Sub into BigQuery tables or Cloud Storage buckets WITHOUT needing a Dataflow pipeline.
25	How does Object Lifecycle Management work in Google Cloud Storage, and what actions and conditions are supported?	Object Lifecycle Management applies declarative JSON rules to a bucket to automatically transition or delete objects. Supported Actions: `Delete`, `SetStorageClass` (downgrade from Standard -> Nearline -> Coldline -> Archive), and `AbortIncompleteMultipartUpload`. Supported Conditions: `age` (days since creation), `createdBefore`, `isLive` (versioned objects), `numNewerVersions`, `matchesPrefix`/`matchesSuffix`, and `daysSinceCustomTime`.
26	How can you preview data in BigQuery without incurring query scan costs?	Instead of running `SELECT * FROM dataset.table LIMIT 10`, use: 1. The 'Preview' tab in the BigQuery Cloud Console UI (free, uses `tabledata.list` API); 2. The CLI command `bq head -n 10 project:dataset.table`; or 3. Run a dry-run (`--dry_run`) first to estimate exact bytes processed before executing a query.
27	How does Sensitive Data Protection (formerly Cloud DLP) de-identify PII in Google Cloud pipelines?	Sensitive Data Protection scans structured (BigQuery, Spanner) and unstructured (GCS) data using 150+ built-in `InfoTypes` (credit cards, SSNs, emails). It supports four main de-identification techniques: 1. Redaction/Masking (replacing characters with `*`); 2. Replacement (replacing with `[EMAIL]`); 3. Crypto-based Deterministic Tokenization (AES-SIV or Format-Preserving Encryption FPE-FFX, preserving referential integrity for SQL joins); 4. Bucketing / Date Shifting.
28	Compare the four Google Cloud Storage (GCS) storage classes: Standard, Nearline, Coldline, and Archive.	All four classes offer the same millisecond first-byte latency, high durability (11 nines), and global API! They differ by pricing and minimum storage duration: 1. Standard: Hot data, no minimum duration, no retrieval fee; 2. Nearline: Accessed <1x/month, 30-day min duration, per-GB retrieval fee; 3. Coldline: Accessed <1x/quarter, 90-day min duration, higher retrieval fee; 4. Archive: Accessed <1x/year, 365-day min duration, lowest storage cost and highest retrieval fee.
29	Compare Cloud Composer, Cloud Workflows, and Cloud Scheduler for orchestration in GCP.	Cloud Composer: Managed Apache Airflow running on GKE; best for complex, multi-step big data ETL/ELT DAGs across GCP and multi-cloud. Cloud Workflows: Serverless, low-latency HTTP/API state machine orchestrator (YAML/JSON) that coordinates microservices, Cloud Run, and GCP APIs with sub-second steps and scales to zero. Cloud Scheduler: Managed cron service that triggers a single HTTP endpoint, Pub/Sub topic, or App Engine target on a schedule.
30	How do Log Buckets, Log Routers, and Log Sinks work in Google Cloud Logging?	Every log entry ingested into a GCP Project passes through the Log Router, which evaluates inclusion and exclusion filters. Matching logs are routed via Log Sinks to: 1. `_Required` Log Bucket (Admin Activity, System Event, Access Transparency; fixed 400-day retention, free); 2. `_Default` Log Bucket (30-day default retention, configurable); 3. Custom Log Buckets (with Log Analytics / BigQuery linked datasets); 4. Cloud Storage (archival); 5. BigQuery (SQL analysis); or 6. Pub/Sub (external SIEM export).
31	How does BigQuery's decoupled architecture work under the hood (Capacitor, Colossus, Borg/Dremel, Jupiter)?	BigQuery completely separates storage from compute: 1. Colossus (Google's global distributed filesystem) stores data in Capacitor (a proprietary columnar format that compresses and operates directly on encoded columns); 2. Dremel (running on Borg) allocates dynamic trees of stateless execution units called 'Slots' to execute SQL in parallel; 3. Jupiter (petabit-per-second datacenter network) shuffles intermediate data between slots directly in memory.
32	What is Google Cloud Operations Suite (formerly Stackdriver) and what are the four primary types of Cloud Audit Logs?	Cloud Operations Suite includes Cloud Logging, Cloud Monitoring, Cloud Trace, Cloud Profiler, and Error Reporting. The 4 Cloud Audit Log types are: 1. Admin Activity logs (WHO modified resource configurations; always on, free, retained 400 days); 2. Data Access logs (WHO read/wrote data inside resources; disabled by default except BigQuery because of high volume/cost); 3. System Event logs (Google-initiated maintenance; always on, free, 400 days); 4. Policy Denied logs (VPC-SC or IAM rejections).