google · franciscovalentecastro · Jul 23, 2026
diff --git a/skills/cloud/gboc-3p-app-log-config/SKILL.md b/skills/cloud/gboc-3p-app-log-config/SKILL.md
@@ -0,0 +1,115 @@
+---
+name: gboc-3p-app-log-config
+description: >-
+  Configures the Google-Built OpenTelemetry Collector (GBOC) to collect and
+  parse logs from well-known third-party applications (Nginx, Apache, MySQL,
+  MongoDB, Kafka, PostgreSQL, Cassandra, IIS, and more) on a VM, replicating the
+  Ops Agent's parsing (regexes, timestamp and severity mapping, httpRequest
+  structure) by referencing the upstream ops-agent source. Use when setting up
+  GBOC log collection for a specific supported third-party app, when you need the
+  exact Ops Agent regex or field mapping for an app, or to make GBOC log output
+  match the Ops Agent format. Don't use for generic or custom (non-standard) log
+  formats (use generate-gboc-log-config) or for metrics collection.
+---
+
+# Configure GBOC for Third-Party App Logs (Ops Agent parity)
+
+## What this skill is for
+
+The Ops Agent ships built-in log parsers for dozens of third-party apps. GBOC
+(a stock OpenTelemetry Collector distribution) does **not** ship these parsers,
+but you can reproduce their output by reading the Ops Agent source and
+translating it into stock GBOC components (`filelog` receiver + `transform`
+processor + `googlecloud` exporter).
+
+This skill is **reference-driven**: rather than embedding every app's intricate
+regex (they change and are easy to get subtly wrong), it points you to the
+authoritative ops-agent source and gives you the rules to translate it faithfully.
+
+**Relationship to other skills:**
+
+-   Use [generate-gboc-log-config](../generate-gboc-log-config/SKILL.md) for the
+    generic mechanics (choosing OS, reading the log file, editing
+    `config.yaml`, restarting the service, verifying ingestion) and for custom /
+    non-standard log formats. This skill focuses only on the app-specific
+    parsing that must match the Ops Agent.
+
+## Supported apps
+
+Only apps that the Ops Agent defines a **logging** parser for are supported.
+The full list, the receiver types each app exposes, and the exact GitHub
+reference links are in
+[references/ops-agent-references.md](references/ops-agent-references.md).
+
+Quick check: if the app is not in that table (i.e. it has no logging processor
+in the ops-agent `apps/` directory), treat the log as a custom format and use
+[generate-gboc-log-config](../generate-gboc-log-config/SKILL.md) instead.
+
+## Workflow
+
+### 1. Identify the app and its receiver type
+
+Ask which app (e.g. `nginx`) and which log stream (e.g. access vs. error). Map it
+to the ops-agent app name and receiver type using the table in
+[references/ops-agent-references.md](references/ops-agent-references.md)
+(e.g. `nginx` → `nginx_access`, `nginx_error`).
+
+### 2. Fetch the authoritative Ops Agent definition
+
+Read two sources for that app:
+
+1.  **`apps/<app>.go`** — defines the receiver types, the regex (often via a
+    shared helper like `genericAccessLogParser`), the `TimeFormat`, field type
+    casts, and the severity `ModifyFields` mapping.
+2.  **The OTel golden** `logging-otel-receiver_<app>` (only some apps have one)
+    — shows the exact `transform` statements and final field layout the Ops
+    Agent produces for the OTel pipeline.
+
+> [!TIP]
+> `read_url_content` truncates long files. Fetch raw sources with `curl` and
+> grep them instead:
+> ```bash
+> curl -sSL https://raw.githubusercontent.com/GoogleCloudPlatform/ops-agent/master/apps/nginx.go
+> curl -sSL https://raw.githubusercontent.com/GoogleCloudPlatform/ops-agent/master/apps/common_logging_processors.go
+> ```
+> Access-log parsers (apache, nginx, jetty, tomcat, couchbase, iis) live in the
+> shared `genericAccessLogParser` in `apps/common_logging_processors.go`.
+
+### 3. Translate Ops Agent constructs into stock GBOC
+
+The Ops Agent goldens use custom OTTL functions (`ExtractPatternsRubyRegex`,
+`IsMatchRubyRegex`) that are **not** compiled into GBOC. Do not copy goldens
+verbatim. Instead translate using the rules in
+[references/ops-agent-to-gboc-mapping.md](references/ops-agent-to-gboc-mapping.md),
+which covers regex syntax conversion, the `httpRequest` structure, timestamp and
+severity mapping, `jsonPayload` vs. labels, and includes a full worked Nginx
+example.
+
+### 4. Merge into the VM's config and deploy
+
+Hand off to [generate-gboc-log-config](../generate-gboc-log-config/SKILL.md) for
+reading the existing `config.yaml`, backing it up, merging the new receiver /
+processor / pipeline, restarting `otelcol-google`, and verifying with
+`gcloud logging read`.
+
+### 5. Verify parity
+
+Confirm the emitted `LogEntry` matches the Ops Agent shape: correct `logName`
+(e.g. `nginx_access`), populated `httpRequest` where applicable, `severity`
+translated, and no leftover `-` placeholder fields in `jsonPayload`.
+
+## Key gotchas (see the mapping reference for detail)
+
+-   **Regex flavor**: Ops Agent regexes use Ruby `(?<name>...)`; RE2 (Go /
+    OTTL / stanza `regex_parser`) needs `(?P<name>...)`.
+-   **httpRequest**: build the `attributes["gcp.http_request"]` map — the
+    `googlecloud` exporter maps it to `LogEntry.httpRequest`. Cast `status` to an
+    integer; keep `responseSize` as a string.
+-   **jsonPayload vs. labels**: the exporter turns the log record **body** (a
+    map) into `jsonPayload`, and **attributes** into labels (except the special
+    `gcp.*` keys). Put user-facing fields in the body.
+-   **Placeholder values**: apache/nginx-style parsers omit fields equal to `-`.
+-   **Component availability**: some operators (`csv`, `windowseventlog`) may not
+    be compiled into a given GBOC build; fall back to `regex_parser`. Verify with
+    the interactive-run trick in
+    [generate-gboc-log-config](../generate-gboc-log-config/SKILL.md).
diff --git a/skills/cloud/gboc-3p-app-log-config/references/ops-agent-references.md b/skills/cloud/gboc-3p-app-log-config/references/ops-agent-references.md
@@ -0,0 +1,72 @@
+# Ops Agent references for third-party app logging
+
+Use these upstream `ops-agent` sources to reproduce an app's log parsing in GBOC.
+All links point at `master`; pin to a release tag if you need reproducibility.
+
+## How to fetch (avoid truncation)
+
+`read_url_content` truncates long files. Prefer `curl` on the raw host and grep:
+
+```bash
+# App definition (receiver types, regex, TimeFormat, severity mapping)
+curl -sSL https://raw.githubusercontent.com/GoogleCloudPlatform/ops-agent/master/apps/<app>.go
+
+# Shared access-log parser (apache/nginx/jetty/tomcat/couchbase/iis access logs)
+curl -sSL https://raw.githubusercontent.com/GoogleCloudPlatform/ops-agent/master/apps/common_logging_processors.go
+
+# OTel golden output (only some apps): the exact transform + field layout
+curl -sSL https://raw.githubusercontent.com/GoogleCloudPlatform/ops-agent/master/confgenerator/testdata/goldens/logging-otel-receiver_<app>/golden/linux/otel.yaml
+```
+
+- App source: `https://github.com/GoogleCloudPlatform/ops-agent/blob/master/apps/<app>.go`
+- Golden dir: `https://github.com/GoogleCloudPlatform/ops-agent/tree/master/confgenerator/testdata/goldens/logging-otel-receiver_<app>`
+
+## Apps with a logging parser
+
+Derived from the ops-agent `apps/` directory (files defining a logging
+processor/receiver). "OTel golden" marks apps that also have a
+`logging-otel-receiver_<app>` golden — the most complete reference. For the
+others, read `apps/<app>.go` (and `common_logging_processors.go` for access
+logs) to extract the regex, timestamp, and severity mapping. Receiver-type names
+below are indicative; always confirm the exact names in the source.
+
+| App | Indicative receiver types | Source (`apps/`) | OTel golden |
+| --- | --- | --- | --- |
+| nginx | `nginx_access`, `nginx_error` | `nginx.go` | ✅ `logging-otel-receiver_nginx` |
+| mysql | `mysql_error`, `mysql_general`, `mysql_slow` | `mysql.go` | ✅ `logging-otel-receiver_mysql` |
+| mongodb | `mongodb` | `mongodb.go` | ✅ `logging-otel-receiver_mongodb` |
+| kafka | `kafka` | `kafka.go` | ✅ `logging-otel-receiver_kafka` |
+| apache | `apache_access`, `apache_error` | `apache.go` | — |
+| iis | `iis_access` | `iis.go` | — |
+| jetty | `jetty_access` | `jetty.go` | — |
+| tomcat | `tomcat_access`, `tomcat_system` | `tomcat.go` | — |
+| couchbase | `couchbase_general`, `couchbase_http_access` | `couchbase.go` | — |
+| couchdb | `couchdb` | `couchdb.go` | — |
+| elasticsearch | `elasticsearch_json`, `elasticsearch_gc` | `elasticsearch.go` | — |
+| postgresql | `postgresql_general` | `postgresql.go` | — |
+| oracledb | `oracledb_audit`, `oracledb_alert` | `oracledb.go` | — |
+| vault | `vault_audit` | `vault.go` | — |
+| zookeeper | `zookeeper_general` | `zookeeper.go` | — |
+| cassandra | `cassandra_system`, `cassandra_debug`, `cassandra_gc` | `cassandra.go` | — |
+| hadoop | `hadoop` | `hadoop.go` | — |
+| hbase | `hbase` | `hbase.go` | — |
+| flink | `flink` | `flink.go` | — |
+| solr | `solr_system` | `solr.go` | — |
+| rabbitmq | `rabbitmq` | `rabbitmq.go` | — |
+| redis | `redis` | `redis.go` | — |
+| saphana | `saphana` | `saphana.go` | — |
+| varnish | `varnishlog` | `varnish.go` | — |
+| wildfly | `wildfly` | `wildfly.go` | — |
+| active_directory_ds | `active_directory_ds` (Windows event log) | `active_directory_ds.go` | — |
+
+> The shared access-log parser used by apache/nginx/jetty/tomcat/couchbase/iis
+> lives in `apps/common_logging_processors.go` (`genericAccessLogParser`).
+
+## Generic / infrastructure log references
+
+These goldens are useful patterns even outside a specific app:
+
+- `logging-otel-receiver_systemd` — journald / systemd.
+- `logging-otel-receiver_syslog_type_multiple_receivers` — syslog (tcp/udp).
+- `logging-otel-receiver_forward` — Fluent Forward protocol.
+- `logging-otel-receiver_files_refresh_interval` — plain file tailing options.
diff --git a/skills/cloud/gboc-3p-app-log-config/references/ops-agent-to-gboc-mapping.md b/skills/cloud/gboc-3p-app-log-config/references/ops-agent-to-gboc-mapping.md
@@ -0,0 +1,131 @@
+# Translating Ops Agent parsing into stock GBOC
+
+The Ops Agent generates OTel configs that rely on **custom OTTL functions**
+(`ExtractPatternsRubyRegex`, `IsMatchRubyRegex`) which are **not** compiled into
+GBOC. Read the goldens to understand the *target output*, but rebuild it with
+stock components: the `filelog` receiver (`regex_parser` operator) plus a
+`transform` processor, exporting via `googlecloud`.
+
+## How the `googlecloud` exporter maps a log record
+
+Confirmed from `exporter/collector/logs.go` in `opentelemetry-operations-go`:
+
+| Log record part | Becomes | Notes |
+| --- | --- | --- |
+| Body (a **map**) | `jsonPayload` | If body is a string → `textPayload`. |
+| Attributes | `labels` | Except the special `gcp.*` keys below. |
+| `attributes["gcp.http_request"]` (map) | `LogEntry.httpRequest` | Consumed & removed from labels. |
+| `attributes["gcp.log_name"]` | `logName` | |
+| `attributes["gcp.source_location"]` | `sourceLocation` | |
+| `attributes["gcp.trace_sampled"]` | `traceSampled` | |
+| `severity_number` / `severity_text` | `severity` | |
+
+So: put user-facing fields in the **body map** (→ `jsonPayload`), and build the
+`gcp.http_request` map for the HTTP structure.
+
+## Translation rules
+
+1.  **Regex flavor**: convert Ruby `(?<name>...)` to RE2 `(?P<name>...)`. RE2
+    supports non-greedy `*?`/`+?`. Keep the character classes as-is.
+
+2.  **Timestamp**: use the app's `TimeFormat` (strptime directives) directly in
+    the `regex_parser` `timestamp` block (`layout:`) — e.g. nginx/apache use
+    `%d/%b/%Y:%H:%M:%S %z`.
+
+3.  **`httpRequest`** (access logs): build `attributes["gcp.http_request"]` with
+    the LogEntry HttpRequest field names: `remoteIp`, `requestMethod`,
+    `requestUrl`, `protocol`, `status`, `responseSize`, `referer`, `userAgent`.
+    -   Cast `status` to an integer (`Int(...)`).
+    -   Leave `responseSize` a **string** (matches Ops Agent; the exporter
+        accepts it).
+
+4.  **Type casts**: mirror the app's `ParserShared.Types` map. Fields marked
+    `integer` there get `Int(...)`; everything else stays a string.
+
+5.  **Severity**: if `apps/<app>.go` has a `LoggingProcessorModifyFields` block
+    mapping a parsed level to `severity` (e.g. `error` → `ERROR`), reproduce it
+    by setting `severity_text` (and `severity_number` to 0 so the exporter
+    re-derives the number from the text). Access-log parsers usually set **no**
+    severity — don't invent one.
+
+6.  **Omit `-` placeholders**: apache/nginx-style parsers drop fields whose value
+    is `-` (e.g. `referer`, `host`, `user`). Use
+    `delete_key(..., "x") where IsMatch(x, "^-$")`.
+
+7.  **Log name**: set the exporter `default_log_name` (or
+    `attributes["gcp.log_name"]`) to the receiver type, e.g. `nginx_access`.
+
+## Worked example: `nginx_access`
+
+Ops Agent regex (`apps/common_logging_processors.go`, `genericAccessLogParser`),
+converted to RE2, plus the transform that produces Ops-Agent-equivalent output.
+
+```yaml
+receivers:
+  filelog/nginx:
+    include:
+      - /var/log/nginx/access.log   # replace with the actual path
+    start_at: end                   # use "beginning" only for testing
+    operators:
+      - type: regex_parser
+        parse_from: body
+        parse_to: body
+        regex: '^(?P<http_request_remoteIp>[^ ]*) (?P<host>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<http_request_requestMethod>\S+)(?: +(?P<http_request_requestUrl>[^"]*?)(?: +(?P<http_request_protocol>\S+))?)?" (?P<http_request_status>[^ ]*) (?P<http_request_responseSize>[^ ]*)(?: "(?P<http_request_referer>[^"]*)" "(?P<http_request_userAgent>[^"]*)")?(?: "(?P<gzip_ratio>[^"]*)")?$'
+        timestamp:
+          parse_from: body.time
+          layout: '%d/%b/%Y:%H:%M:%S %z'
+
+processors:
+  transform/nginx_access:
+    error_mode: ignore
+    log_statements:
+      - context: log
+        statements:
+          - set(body["http_request_status"], Int(body["http_request_status"])) where body["http_request_status"] != nil
+          - set(attributes["gcp.http_request"]["remoteIp"], body["http_request_remoteIp"]) where body["http_request_remoteIp"] != nil
+          - set(attributes["gcp.http_request"]["requestMethod"], body["http_request_requestMethod"]) where body["http_request_requestMethod"] != nil
+          - set(attributes["gcp.http_request"]["requestUrl"], body["http_request_requestUrl"]) where body["http_request_requestUrl"] != nil
+          - set(attributes["gcp.http_request"]["protocol"], body["http_request_protocol"]) where body["http_request_protocol"] != nil
+          - set(attributes["gcp.http_request"]["status"], body["http_request_status"]) where body["http_request_status"] != nil
+          - set(attributes["gcp.http_request"]["responseSize"], body["http_request_responseSize"]) where body["http_request_responseSize"] != nil
+          - set(attributes["gcp.http_request"]["userAgent"], body["http_request_userAgent"]) where body["http_request_userAgent"] != nil
+          - set(attributes["gcp.http_request"]["referer"], body["http_request_referer"]) where body["http_request_referer"] != nil
+          - delete_key(attributes["gcp.http_request"], "referer") where (attributes["gcp.http_request"] != nil and attributes["gcp.http_request"]["referer"] != nil and IsMatch(body["http_request_referer"], "^-$"))
+          - delete_key(body, "http_request_remoteIp")
+          - delete_key(body, "http_request_requestMethod")
+          - delete_key(body, "http_request_requestUrl")
+          - delete_key(body, "http_request_protocol")
+          - delete_key(body, "http_request_status")
+          - delete_key(body, "http_request_responseSize")
+          - delete_key(body, "http_request_referer")
+          - delete_key(body, "http_request_userAgent")
+          - delete_key(body, "time")
+          - delete_key(body, "host") where (body["host"] == nil or IsMatch(body["host"], "^-$"))
+          - delete_key(body, "user") where (body["user"] == nil or IsMatch(body["user"], "^-$"))
+          - delete_key(body, "gzip_ratio") where (body["gzip_ratio"] == nil or body["gzip_ratio"] == "" or IsMatch(body["gzip_ratio"], "^-$"))
+
+exporters:
+  googlecloud:
+    log:
+      default_log_name: nginx_access
+
+service:
+  pipelines:
+    logs/nginx:
+      receivers: [filelog/nginx]
+      processors: [memory_limiter, transform/nginx_access, resourcedetection, batch]
+      exporters: [googlecloud]
+```
+
+### Adapting to other apps
+
+-   **Other access logs** (apache/jetty/tomcat/couchbase/iis): same
+    `genericAccessLogParser` regex and `httpRequest` construction; only the
+    include path and `default_log_name` change.
+-   **Error/system logs** (nginx_error, mysql_error, etc.): usually a different
+    regex and a severity `ModifyFields` mapping instead of `httpRequest`. Copy
+    the regex from `apps/<app>.go`, keep parsed fields in the body
+    (`jsonPayload`), and translate the severity mapping to `severity_text`.
+-   **JSON logs** (elasticsearch_json): replace `regex_parser` with a
+    `json_parser` operator (`parse_to: body`); map the timestamp and severity
+    fields the app specifies.