diff --git a/skills/cloud/google-cloud-logging-configuration/SKILL.md b/skills/cloud/google-cloud-logging-configuration/SKILL.md new file mode 100644 index 0000000000..8a3e63454e --- /dev/null +++ b/skills/cloud/google-cloud-logging-configuration/SKILL.md @@ -0,0 +1,288 @@ +--- +name: google-cloud-logging-configuration +description: >- + Configures cross-project logging in GCP using either Centralized Storage (routing + logs to a central regional bucket) or Read-Time Aggregation (querying logs across + projects using Log Scopes and Log Views). Use when setting up telemetry aggregation, + configuring regional log buckets, creating logging sinks, configuring Log Scopes, + creating custom Log Views, or managing permissions for cross-project log access. + Don't use for querying, searching, or analyzing existing logs (use log query skills + or Log Explorer tools instead). +--- + +# Configuring Cross-Project Logging (Centralized vs. Read-Time) + +This skill describes how to configure and verify cross-project logging structures +in Google Cloud Platform (GCP) using `gcloud` commands, covering both Centralized +Storage and Read-Time Aggregation approaches. + +## Decision Matrix: Centralized vs. Read-Time Aggregation + +Use this skill to configure **Centralized Storage** (routing logs to a central +bucket). If your requirements differ, consider **Read-Time Aggregation** (using +Observability Scopes) instead. + +| Criterion | Centralized Storage | Read-Time Aggregation | +: : (This Skill) : (Observability Scopes) : +| :-------------------- | :----------------------- | :----------------------- | +| **GCP Project Scale** | Scales to thousands of | Best for < 375 projects. | +: : projects. : : +| **Log Storage** | Consolidated in a single | Resides in originating | +: : central project. : child projects. : +| **SQL Analytics** | Easy; unified querying | Hard; requires querying | +: : via Log Analytics : multiple locations. : +: : (BigQuery). : : +| **Access Control** | Scoped access via | Requires IAM access to | +: : central Log Views. : all child projects. : +| **Cost** | Potential for | Cost-effective; no data | +: : duplication if : replication. : +: : exclusions aren't set. : : + +### Why We Chose Centralized Storage: + +1. **Unified SQL Analytics:** Essential for querying across all logs from a + single surface using BigQuery SQL. +2. **Centralized Access Control:** Allows managing access to logs in one place + without granting permissions to all source projects. + +## Architecture + +### Centralized Storage (Log Routing) +```mermaid +graph LR + subgraph "Source Project(s)" + Log[Resource Logs] --> Sink["Sink: route-to-regional-telemetry-bucket"] + end + + subgraph "Destination Project" + Sink --> Bucket["Bucket: regional-telemetry-bucket (us-central1)"] + Bucket --> BQ["BigQuery Dataset (via Link)"] + end +``` + +### Read-Time Aggregation (Log Scopes) +```mermaid +graph LR + subgraph "Source Project 1" + Log1[Resource Logs] --> Bucket1["Bucket: _Default"] + end + + subgraph "Source Project 2" + Log2[Resource Logs] --> Bucket2["Bucket: _Default"] + end + + subgraph "Scoping Project (No Log Storage)" + Scope["Log Scope: central-query-scope"] + Scope -.-> Bucket1 + Scope -.-> Bucket2 + end +``` + +## Setup Steps: Centralized Storage (Log Routing) + +Use these steps to route logs from one or more source projects to a central bucket in a destination project. + +### 1. Create regional log bucket in destination project +Create a custom log bucket with Log Analytics enabled (required for BigQuery integration). + +> [!TIP] Prefer regional buckets (e.g., `us-central1`) over `global` to ensure compatibility with Log Analytics and SQL querying. + +```bash +gcloud logging buckets create {bucket_id} \ + --project={destination_project_id} \ + --location={region} \ + --retention-days={retention_days} \ + --enable-analytics +``` + +### 2. Create log sink in source project +Create a project-level sink in the source project pointing to the destination bucket. + +> [!IMPORTANT] **Gotcha:** Standard filter expressions like `logName:abc` can fail to match. **Always** use `LOG_ID("{log_id}")` for precise matching. + +```bash +gcloud logging sinks create {sink_name} \ + logging.googleapis.com/projects/{destination_project_id}/locations/{region}/buckets/{bucket_id} \ + --project={source_project_id} \ + --log-filter='LOG_ID("{log_id}")' +``` + +### 3. Grant IAM permissions to sink writer +Retrieve the `writerIdentity` of the created sink and grant it `roles/logging.bucketWriter` on the destination project. + +```bash +# Get the writer identity (service account) +gcloud logging sinks describe {sink_name} \ + --project={source_project_id} \ + --format="value(writerIdentity)" + +# Grant permissions on the destination project +gcloud projects add-iam-policy-binding {destination_project_id} \ + --member={writer_identity} \ + --role=roles/logging.bucketWriter +``` + +--- + +## Setup Steps: Read-Time Aggregation (Log Scopes) + +Use these steps to query logs across multiple projects without moving them, using Log Scopes. + +### 1. Create a custom Log View (Optional but recommended) +Create a Log View in the source project to restrict which logs are accessible. + +> [!IMPORTANT] **Gotcha:** Log View filters are restricted. They can **only** contain restrictions on log source (`SOURCE()`), valid resource types (`resource.type`), or Log ID (`LOG_ID()`). You **cannot** filter by severity (e.g., `severity>=ERROR` is invalid inside a view filter; this must be done at query time). + +```bash +gcloud logging views create {view_id} \ + --bucket={bucket_id} \ + --location={region} \ + --project={source_project_id} \ + --log-filter='LOG_ID("{log_id}")' +``` +* `{bucket_id}`: e.g., `_Default` +* `{region}`: e.g., `global` +* `{view_id}`: e.g., `app-logs-view` +* If you want to allow access to all logs in the bucket, omit the `--log-filter` flag. + +### 2. Create a Log Scope in the scoping project +Create a Log Scope in the central project (scoping project) listing the source resources (projects or specific log views). + +```bash +gcloud logging scopes create {log_scope_id} \ + --project={scoping_project_id} \ + --resource-names={resource_names} +``` +* `{log_scope_id}`: e.g., `central-query-scope` +* `{resource_names}`: Comma-separated list of resources (e.g., `projects/source-project-1,projects/source-project-2/locations/global/buckets/_Default/views/app-logs-view`). Max 50 projects, 100 resources total. + +### 3. Update Default Observability Scope (Optional) +Link the Log Scope to the project's default observability scope so it is used by default in Logs Explorer. + +```bash +gcloud observability scopes update Default \ + --project={scoping_project_id} \ + --location=global \ + --log-scope=//logging.googleapis.com/projects/{scoping_project_id}/locations/global/logScopes/{log_scope_id} +``` + +### 4. Grant IAM permissions to users +Unlike centralized routing, permissions are checked at query time. Users running queries must have: +* `roles/logging.viewAccessor` granted on the specific Log View resource (with IAM conditions) or `roles/logging.viewer` on the source projects. +* Access to the scoping project. + +--- + +## Restricting Access to Sensitive Logs (Security) + +If you have sensitive logs (e.g., security logs) in a source project's `_Default` bucket, anyone with `roles/logging.viewer` on that project can see them via the default `_Default` view. To restrict them: + +### 1. Exclude from Default View (Keep logs, hide from general users) +Update the `_Default` view filter in the source project to explicitly exclude the sensitive logs: + +```bash +gcloud logging views update _Default \ + --bucket=_Default \ + --location=global \ + --project={source_project_id} \ + --log-filter='NOT LOG_ID("cloudaudit.googleapis.com/data_access") AND NOT LOG_ID("externalaudit.googleapis.com/data_access") AND NOT LOG_ID("{sensitive_log_id}")' +``` + +### 2. Create a custom view for Security Team +Create a custom view (e.g., `security-view`) that includes the sensitive logs, and grant the security team `roles/logging.viewAccessor` restricted to that view using IAM conditions. + +### 3. Exclude from Default Bucket (Discard logs) +If you want to completely prevent these logs from being stored in the `_Default` bucket: + +> [!CAUTION] This will delete the logs immediately unless you have another sink routing them to a secure destination. + +```bash +gcloud logging sinks update _Default \ + --project={source_project_id} \ + --add-exclusion=name=exclude-sensitive,filter='LOG_ID("{sensitive_log_id}")' +``` + +--- + +## Cost Optimization (Reducing Logging Costs) + +Logging costs in GCP are based on the volume of data ingested and stored. You can significantly reduce costs by excluding high-volume, low-value logs or by sampling them. + +> [!IMPORTANT] +> **Key Cost Considerations:** +> * **Log Analytics Downgrade Limitation (Mandatory Warning):** Whenever providing guidance or commands on logging cost optimization or exclusions, you **must** proactively warn the user that once a log bucket has been upgraded to use Log Analytics, it **cannot be downgraded** to remove the analytics capability. Explain that if they want to stop incurring analytics costs, they must delete the bucket or route logs away from it. + + +### 1. Exclude high-volume logs entirely +To completely stop ingesting a specific type of log (e.g., Load Balancer logs or specific info logs) to the default bucket: + +```bash +gcloud logging sinks update _Default \ + --project={project_id} \ + --add-exclusion=name=exclude-lb-logs,filter='resource.type="http_load_balancer"' +``` + +### 2. Sample high-volume logs (Keep a percentage) +If you need some logs for analysis but want to reduce volume, use the `sample()` function in the exclusion filter. + +> [!IMPORTANT] +> The `sample(field, fraction)` function matches a `fraction` of logs. When used in an **exclusion filter**, the matched logs are **discarded**. +> Therefore, to **keep** a certain percentage of logs, you must **exclude** the remaining percentage. +> * To **keep 10%** (exclude 90%), use `sample(insertId, 0.9)` in the exclusion filter. +> * To **keep 25%** (exclude 75%), use `sample(insertId, 0.75)` in the exclusion filter. + +To keep only 10% of `DEBUG` severity logs (excluding 90%): + +```bash +gcloud logging sinks update _Default \ + --project={project_id} \ + --add-exclusion=name=sample-debug-logs,filter='severity=DEBUG AND sample(insertId, 0.9)' +``` + +-------------------------------------------------------------------------------- + +## Verification + +### 1. Write test log in source project +```bash +gcloud logging write {log_id} "Test log message" --project={source_project_id} +``` + +### 2. Verify Centralized Storage (Read from destination bucket) +Verify the log arrived in the destination bucket. Note that you **must** specify the `--view` flag. + +```bash +gcloud logging read 'LOG_ID("{log_id}")' \ + --bucket={bucket_id} \ + --location={region} \ + --project={destination_project_id} \ + --view=_AllLogs +``` + +### 3. Verify Read-Time Aggregation (Read from multiple resources) +Verify you can read logs using the scope or by specifying the resource names directly. + +```bash +gcloud logging read 'LOG_ID("{log_id}")' \ + --resource-names=projects/{source_project_id}/locations/{region}/buckets/{bucket_id}/views/{view_id} +``` + +### 4. Query logs via BigQuery SQL (Optional, Centralized only) +If Log Analytics is enabled, you can link the bucket to BigQuery to query logs using SQL. + +```bash +# Create a link (creates linked BQ dataset) +gcloud logging links create {link_id} \ + --bucket={bucket_id} \ + --location={region} \ + --project={destination_project_id} + +# Query via bq CLI +bq query \ + --project_id={destination_project_id} \ + --use_legacy_sql=false \ + --location={region} \ + "SELECT log_id, text_payload, timestamp FROM \`{destination_project_id}.{link_id}._AllLogs\` WHERE log_id = '{log_id}'" +``` + +* `{link_id}`: e.g., `bqlink` (use lowercase, alphanumeric, no hyphens)