diff --git a/docs/contributor/arch/037-ops-scrape-diagnostic-metrics.md b/docs/contributor/arch/037-ops-scrape-diagnostic-metrics.md new file mode 100644 index 0000000000..60f3e92ff6 --- /dev/null +++ b/docs/contributor/arch/037-ops-scrape-diagnostic-metrics.md @@ -0,0 +1,173 @@ +--- +title: Ops Scrape Diagnostic Metrics +status: Proposed +date: 2026-06-12 +--- + +# Ops Scrape Diagnostic Metrics + +## Context and Problem Statement + +The metric agent's Prometheus receivers produce per-target diagnostic metrics for every scrape operation. These metrics report the health and behavior of individual scrape targets: + +| Metric | Description | +|--------|-------------| +| `up` | Whether the target is reachable (1 = healthy, 0 = failed) | +| `scrape_duration_seconds` | Time taken to scrape the target | +| `scrape_samples_scraped` | Number of samples the target exposed (pre-relabeling) | +| `scrape_samples_post_metric_relabeling` | Samples remaining after `metric_relabel_configs` | +| `scrape_series_added` | Approximate number of new series added per scrape | +| `scrape_body_size_bytes` | Response body size (-1 = body size limit exceeded, 0 = other failure) | +| `scrape_timeout_seconds` | Configured scrape timeout (static config value) | +| `scrape_sample_limit` | Configured sample limit (static config value) | + +`scrape_timeout_seconds` and `scrape_sample_limit` do not provide any value since they are based on static configuration and are excluded from further consideration. + +### The Problem + +These metrics are currently dropped before reaching user backends (controlled by `diagnosticMetrics.enabled` in the MetricPipeline spec) and are not exposed for internal monitoring. We have no visibility into whether scrape targets are healthy, hitting limits, or timing out. + +See [#2955](https://github.com/kyma-project/telemetry-manager/issues/2955). + +### Cardinality + +The scrape diagnostic metrics themselves are not a cardinality concern in absolute terms. At 6 metrics × n targets, even 300 pods produce only 1,800 series — manageable for any Prometheus instance. + +However, we do not control the amount of workload. It is impossible to predict how many scrape targets exist, and with increasing pod count, the metric agent's Prometheus exporter must hold more series in memory. The series count scales linearly with cluster size, and each series carries multiple labels from `resource_to_telemetry_conversion` (`k8s_pod_name`, `k8s_namespace_name`, `k8s_node_name`, etc.), which inflates per-series memory cost. + +### Per-Metric Assessment + +The metric agent has three Prometheus scrape jobs: `app-pods`, `app-services`, and `istio`. + +| Metric | Decision | Purpose | +|------------------------------------------|----------|---------------------------------------------------------------------------------------------------| +| `up` | Keep | Health/alerting baseline | +| `scrape_samples_scraped` | Keep | Finds the offending target; the metric `sample_limit: 50000` is enforced against (pre-relabeling) | +| `scrape_series_added` | Keep | Churn signal: detects cardinality spikes | +| `scrape_duration_seconds` | Keep | Catches targets slow to serialize large metrics pages (approaching timeout) | +| `scrape_body_size_bytes` | Keep | Sentinel: alert on -1 (body size limit hit) or 0 (failure) | +| `scrape_samples_post_metric_relabeling` | Drop | Only meaningful where `metric_relabel_configs` exist. | + +`scrape_samples_post_metric_relabeling` is equal to `scrape_samples_scraped` in `prometheus` input scrape jobs. It is only useful in `istio` input scrape jobs for identifying how much metric series are discarded after relabeling. It does not provide much information as to why a scrape failed, therefore we can safely discard this metric. + +Using a combination of these metrics, ops can identify the root cause of scrape failures: + +| Failure mode | `up` | `scrape_body_size_bytes` | `scrape_duration_seconds ` | `scrape_samples_scraped` | +|---------------------------------|--------|---------------------------|----------------------------|---------------------------| +| Target unreachable | 0 | 0 | low | 0 | +| Scrape timeout | 0 | 0 | ≈ scrape_interval | 0 | +| Body size limit exceeded (20MB) | 0 | -1 | varies | varies | +| Sample limit exceeded (50000) | 0 | 0 | varies | ≥ 50000 | +| Healthy | 1 | > 0 | low | > 0 | + +`scrape_body_size_bytes` is only interesting when its value is 0 or -1, because these values indicate that a scrape failed due to exceeding the body size limit or some other error. We can filter the metrics so that we only expose unhealthy metrics for `scrape_body_size_bytes`. + +### Aggregation Considerations + +You cannot aggregate at scrape time — Prometheus `metric_relabel_configs` only drops, keeps, or rewrites labels. Aggregation happens downstream in the OTel Collector using the `metricstransform` or `transform` processor. + +Per-metric aggregation guidance: + +| Metric | Function | Rationale | +|--------|----------|-----------| +| `scrape_samples_scraped` | `max` | Hunting the outlier (the target with the most series); `sum` smears it | +| `scrape_series_added` | `max` | Churn concentrates in one target | +| `scrape_duration_seconds` | `max` | The outlier approaching timeout is what you keep this metric to catch; `p95`/`p99` suppress the exact outlier because it's a gauge | +| `up` | `count` / `min` | `count(up == 0)` for failure count; `min` as an "any target down?" boolean | +| `scrape_body_size_bytes` | Never aggregate | Aggregation destroys the -1 sentinel meaning | +| `scrape_samples_post_metric_relabeling` | `max` | Same as `scrape_samples_scraped` | + +Always aggregate using a function — never a bare `labeldrop` that leaves live replicas with identical label identities, causing series collisions. + +**Advantages of aggregation:** +- Bounded cardinality regardless of cluster size (series count = number of jobs × aggregation groups, not number of pods) +- Safe for self-monitor ingestion — no risk of OOM from workload growth +- Reduces metric agent memory since the Prometheus exporter holds fewer series +- `max` aggregation surfaces the worst target per group, which is the actionable signal + +**Disadvantages of aggregation:** +- Lose per-pod attribution — you know "some target in the istio job has 48,000 samples" but not *which* pod +- For churn debugging, per-pod detail is required to find the specific proxy — requires switching to unaggregated mode +- Adds investigation latency: must flip override, wait for next scrape cycle, then observe +- Requires `metricstransform` processor in the collector image (dependency) +- Aggregation dimension choice is non-obvious (by job? by namespace? by node?) — each choice loses different information +- `max` across all targets hides multi-target degradation (5 targets at 40,000 samples looks the same as 1 target at 40,000) + +## Considered Options + +### Option A: Full Per-Target Exposure (No Aggregation) + +Add a `metrics/ops-scrape-metrics` pipeline that: +1. Receives metrics from the enrichment routing connector (prometheus + istio sources) +2. Filters by metric name to keep only diagnostic metrics +3. Exports all datapoints via a Prometheus exporter on port 9090 + +### Option B: Combined Value-Based Filtering and Aggregation + +Apply different strategies per metric based on their nature: +- **Value-based filtering** for metrics where only unhealthy values are interesting: + - `up`: drop when value == 1 (healthy targets), keep only failures + - `scrape_body_size_bytes`: drop when value > 0 (normal body size), keep only sentinel values (-1, 0) +- **Aggregation** for numeric diagnostics where the worst case is the actionable signal: + - `max(scrape_duration_seconds)` by job + - `max(scrape_samples_scraped)` by job + - `max(scrape_series_added)` by job + +Per-target detail can be restored on demand using the `telemetry-overrides` ConfigMap to bypass aggregation and expose unaggregated metrics for debugging. But this requires a restart of the Metric Agent. + +### Comparison + +| Criteria | Option A | Option B | +|---------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------| +| Per-target visibility | Full — can identify exactly which pod/proxy is problematic | Partial — `up == 0` and `scrape_body_size_bytes` retain per-target identity; aggregated metrics lose it | +| Target counting | `count(up)` gives total targets; `count(up == 0)` gives failure count | Cannot count total targets (healthy ones are dropped) | +| Preventive measures | Can detect when scrape samples approach the sample limit for a specific target | Cannot — only surfaces the `max` per job, loses per-target trending | +| Cardinality | Scales linearly with cluster size (5 metrics × n targets) — unpredictable memory cost | Bounded — filtered metrics scale with number of *unhealthy* targets only; aggregated metrics produce one series per job | +| Self-monitor risk | Risk of OOM in large clusters with many scrape targets | Safe — cardinality is bounded regardless of cluster size | +| Multi-target degradation | Visible — each target reports independently | Hidden — `max` across all targets hides the case where multiple targets degrade simultaneously | +| Implementation complexity | Simple — no aggregation logic, no extra processors | Requires `metricstransform` or `transform` processor for aggregation | +| Failure-mode distinction | Can distinguish "target not found" from "target unhealthy" from "target slow" per pod | Can distinguish for filtered metrics (`up`, `scrape_body_size_bytes`); aggregated metrics lose this | +| Switching to full detail | Already full detail | Requires override config change and pod restart | + +## Decision + +*To be decided after cardinality testing with 100+ pods.* + +## Implementation Notes + +### Architecture + +The ops scrape metrics pipeline sits after the enrichment pipeline and before the output pipelines: + +``` +routing/enrichment ──┬──> metrics/output-{user-pipeline} (existing) + └──> metrics/ops-scrape-metrics (new) + ├─ filter/ops-keep-scrape-metrics + ├─ filter/ops-drop-healthy-scrape-metrics (Option B only) + └─ prometheus/ops-scrape-metrics (:9090) +``` + +The pipeline receives from the `routing/enrichment` connector. Scrape diagnostic metrics always pass through enrichment because they never match the skip-enrichment criteria (which applies only to runtime resource metrics like `node*`, `deployment*`). + +### Prometheus Exporter Configuration + +The Prometheus exporter on port 9090 uses: +- `resource_to_telemetry_conversion.enabled: true` — promotes OTel resource attributes (like `k8s.pod.name`) to Prometheus metric labels for target identification +- `metric_expiration: 5m` — automatically removes stale series when a target disappears, preventing unbounded growth from pod churn + +### Network Policy and Istio + +Port 9090 is included in the metric agent's network policy ingress rules and excluded from Istio sidecar interception through `traffic.sidecar.istio.io/excludeInboundPorts`. + +### Self-Monitor Integration + +The self-monitor uses `role: endpoints` service discovery, which scrapes every pod behind a Service individually. If we add port 9090 to the existing `telemetry-metric-agent-metrics` Service (or create a dedicated Service), all DaemonSet pods are scraped — giving the full union of scrape diagnostics across all nodes. + +### Future: Conditional Per-Target Detail via Overrides + +If aggregation is chosen (Option B), the override mechanism uses the existing `telemetry-overrides` ConfigMap: +1. Add a `metricstransform` processor to the ops pipeline that aggregates by default +2. Gate per-target mode behind the overrides ConfigMap +3. Use hot reload (ConfigMap watch) — no pod restart required +4. Scope the override to also restore trimmed Istio peer dimensions +5. Template it as a per-cluster boolean flip ahead of time diff --git a/internal/otelcollector/config/common/component_ids.go b/internal/otelcollector/config/common/component_ids.go index de1aaaa6a8..2bcecc6d9e 100644 --- a/internal/otelcollector/config/common/component_ids.go +++ b/internal/otelcollector/config/common/component_ids.go @@ -127,6 +127,8 @@ const ComponentIDDropSkipEnrichmentAttributeProcessor ComponentID = "transform/d const ComponentIDSetInstrumentationScopePrometheusProcessor ComponentID = "transform/set-instrumentation-scope-prometheus" const ComponentIDSetInstrumentationScopeIstioProcessor ComponentID = "transform/set-instrumentation-scope-istio" const ComponentIDInsertSkipEnrichmentAttributeProcessor ComponentID = "transform/insert-skip-enrichment-attribute" +const ComponentIDOpsKeepScrapeMetricsProcessor ComponentID = "filter/ops-keep-scrape-metrics" +const ComponentIDOpsDropHealthyScrapeMetricsProcessor ComponentID = "filter/ops-drop-healthy-scrape-metrics" // TRACE-SPECIFIC PROCESSORS ====================================================== @@ -148,6 +150,8 @@ func ComponentIDOTLPExporter(protocol telemetryv1beta1.OTLPProtocol, pipelineRef return fmt.Sprintf("otlp_grpc/%s", pipelineRef.QualifiedName()) } +const ComponentIDOpsScrapeMetricsExporter ComponentID = "prometheus/ops-scrape-metrics" + // ================================================================================ // CONNECTORS // ================================================================================ diff --git a/internal/otelcollector/config/common/types.go b/internal/otelcollector/config/common/types.go index 29af4b5e67..97e8cb4776 100644 --- a/internal/otelcollector/config/common/types.go +++ b/internal/otelcollector/config/common/types.go @@ -185,6 +185,16 @@ type Auth struct { Authenticator string `yaml:"authenticator"` } +type PrometheusExporterConfig struct { + Endpoint string `yaml:"endpoint"` + MetricExpiration string `yaml:"metric_expiration,omitempty"` + ResourceToTelemetryConversion *ResourceToTelemetryConversion `yaml:"resource_to_telemetry_conversion,omitempty"` +} + +type ResourceToTelemetryConversion struct { + Enabled bool `yaml:"enabled"` +} + // ============================================================================= // PROCESSOR TYPES // ============================================================================= diff --git a/internal/otelcollector/config/metricagent/config_builder.go b/internal/otelcollector/config/metricagent/config_builder.go index 48a7c0edfb..9af0c27e91 100644 --- a/internal/otelcollector/config/metricagent/config_builder.go +++ b/internal/otelcollector/config/metricagent/config_builder.go @@ -13,6 +13,7 @@ import ( operatorv1beta1 "github.com/kyma-project/telemetry-manager/apis/operator/v1beta1" telemetryv1beta1 "github.com/kyma-project/telemetry-manager/apis/telemetry/v1beta1" "github.com/kyma-project/telemetry-manager/internal/otelcollector/config/common" + "github.com/kyma-project/telemetry-manager/internal/otelcollector/ports" "github.com/kyma-project/telemetry-manager/internal/pipelines" commonresources "github.com/kyma-project/telemetry-manager/internal/resources/common" metricpipelineutils "github.com/kyma-project/telemetry-manager/internal/utils/metricpipeline" @@ -22,6 +23,7 @@ import ( const ( maxStalenessMultiplier = 4 //nolint:mnd // Tolerate max 3 scrape failures and additional timing jitter enrichmentServicePipelineID = "metrics/enrichment-conditional" + opsScrapeMetricsPipelineID = "metrics/ops-scrape-metrics" podMetricPattern = `^k8s[.]pod[.].*` containerMetricPattern = `(^k8s[.]container[.].*)|(^container[.].*)` nodeMetricPattern = `^k8s[.]node[.].*` @@ -32,7 +34,11 @@ const ( jobMetricPattern = `^k8s[.]job[.].*` ) -var diagnosticMetricNames = []string{"up", "scrape_duration_seconds", "scrape_samples_scraped", "scrape_samples_post_metric_relabeling", "scrape_series_added"} +var diagnosticMetricNames = []string{"up", "scrape_duration_seconds", "scrape_timeout_seconds", "scrape_samples_scraped", "scrape_samples_post_metric_relabeling", "scrape_sample_limit", "scrape_series_added", "scrape_body_size_bytes"} + +// opsScrapeMetricNames is a subset of diagnosticMetricNames excluding static config values +// (scrape_timeout_seconds and scrape_sample_limit) that are the same for all targets within a scrape job. +var opsScrapeMetricNames = []string{"up", "scrape_duration_seconds", "scrape_samples_scraped", "scrape_samples_post_metric_relabeling", "scrape_series_added", "scrape_body_size_bytes"} type buildComponentFunc = common.BuildComponentFunc[*telemetryv1beta1.MetricPipeline] @@ -184,11 +190,23 @@ func (b *Builder) Build(ctx context.Context, pipelines []telemetryv1beta1.Metric b.addK8sAttributesProcessor(opts), b.addRestoreOtelServiceAttrsProcessor(opts), b.addServiceEnrichmentProcessor(opts), - b.addExporterForEnrichmentRouter(pipelinesWithRuntimeInput, pipelinesWithPrometheusInput, pipelinesWithIstioInput), + b.addExporterForEnrichmentRouter(pipelinesWithRuntimeInput, pipelinesWithPrometheusInput, pipelinesWithIstioInput, inputs.prometheus || inputs.istio), ); err != nil { return nil, nil, fmt.Errorf("failed to add enrichment service pipeline: %w", err) } + // Ops scrape metrics pipeline (always present when prometheus or istio input is active) + if inputs.prometheus || inputs.istio { + if err := b.AddServicePipeline(ctx, nil, opsScrapeMetricsPipelineID, + b.addReceiverForEnrichmentRouter(pipelinesWithRuntimeInput, pipelinesWithPrometheusInput, pipelinesWithIstioInput, inputs.prometheus || inputs.istio), + b.addOpsKeepScrapeMetricsProcessor(), + b.addOpsDropHealthyScrapeMetricsProcessor(), + b.addOpsScrapeMetricsExporter(), + ); err != nil { + return nil, nil, fmt.Errorf("failed to add ops scrape metrics pipeline: %w", err) + } + } + // Output pipelines for _, pipeline := range pipelines { outputPipelineID := formatOutputMetricServicePipelineID(&pipeline) @@ -207,7 +225,7 @@ func (b *Builder) Build(ctx context.Context, pipelines []telemetryv1beta1.Metric // Receivers // Metrics are received from either the enrichment pipeline or directly from input pipelines, // depending on whether they have the skip enrichment attribute set. - b.addReceiverForEnrichmentRouter(pipelinesWithRuntimeInput, pipelinesWithPrometheusInput, pipelinesWithIstioInput), + b.addReceiverForEnrichmentRouter(pipelinesWithRuntimeInput, pipelinesWithPrometheusInput, pipelinesWithIstioInput, inputs.prometheus || inputs.istio), b.addReceiverForInputRouter(common.ComponentIDRuntimeInputRoutingConnector, pipelinesWithRuntimeInput, runtimeInputEnabled), b.addReceiverForInputRouter(common.ComponentIDPrometheusInputRoutingConnector, pipelinesWithPrometheusInput, prometheusInputEnabled), b.addReceiverForInputRouter(common.ComponentIDIstioInputRoutingConnector, pipelinesWithIstioInput, istioInputEnabled), @@ -989,6 +1007,46 @@ func dropDiagnosticMetricsFilterProcessor(inputSource common.InputSourceType) *c }) } +func (b *Builder) addOpsKeepScrapeMetricsProcessor() buildComponentFunc { + return b.AddProcessor( + b.StaticComponentID(common.ComponentIDOpsKeepScrapeMetricsProcessor), + func(mp *telemetryv1beta1.MetricPipeline) any { + return keepScrapeMetricsFilterProcessor() + }, + ) +} + +func keepScrapeMetricsFilterProcessor() *common.FilterProcessorConfig { + metricNameConditions := nameConditions(opsScrapeMetricNames) + dropNonScrapeMetricsExpr := common.Not(common.JoinWithOr(metricNameConditions...)) + + return common.MetricFilterProcessor([]telemetryv1beta1.FilterSpec{ + { + Conditions: []string{dropNonScrapeMetricsExpr}, + }, + }) +} + +func (b *Builder) addOpsDropHealthyScrapeMetricsProcessor() buildComponentFunc { + return b.AddProcessor( + b.StaticComponentID(common.ComponentIDOpsDropHealthyScrapeMetricsProcessor), + func(mp *telemetryv1beta1.MetricPipeline) any { + return dropHealthyScrapeMetricsFilterProcessor() + }, + ) +} + +func dropHealthyScrapeMetricsFilterProcessor() *common.FilterProcessorConfig { + return common.MetricFilterProcessor([]telemetryv1beta1.FilterSpec{ + { + Conditions: []string{ + `metric.name == "up" and datapoint.value_int == 1`, + `metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0`, + }, + }, + }) +} + func nameConditions(names []string) []string { var nameConditions []string for _, name := range names { @@ -1069,6 +1127,21 @@ func (b *Builder) addOTLPExporter(queueSize int) buildComponentFunc { ) } +func (b *Builder) addOpsScrapeMetricsExporter() buildComponentFunc { + return b.AddExporter( + b.StaticComponentID(common.ComponentIDOpsScrapeMetricsExporter), + func(ctx context.Context, mp *telemetryv1beta1.MetricPipeline) (any, common.EnvVars, error) { + return &common.PrometheusExporterConfig{ + Endpoint: fmt.Sprintf("${%s}:%d", common.EnvVarCurrentPodIP, ports.OpsScrapeMetrics), + MetricExpiration: "5m", + ResourceToTelemetryConversion: &common.ResourceToTelemetryConversion{ + Enabled: true, + }, + }, nil, nil + }, + ) +} + // Connector builders func (b *Builder) addExporterForInputRouter(componentID string, outputPipelines []telemetryv1beta1.MetricPipeline) buildComponentFunc { @@ -1093,16 +1166,16 @@ func (b *Builder) addReceiverForInputRouter(componentID string, outputPipelines ) } -func (b *Builder) addExporterForEnrichmentRouter(runtimePipelines, prometheusPipelines, istioPipelines []telemetryv1beta1.MetricPipeline) buildComponentFunc { +func (b *Builder) addExporterForEnrichmentRouter(runtimePipelines, prometheusPipelines, istioPipelines []telemetryv1beta1.MetricPipeline, includeOpsScrapeMetrics bool) buildComponentFunc { return b.AddExporter( b.StaticComponentID(common.ComponentIDEnrichmentRoutingConnector), func(ctx context.Context, mp *telemetryv1beta1.MetricPipeline) (any, common.EnvVars, error) { - return enrichmentRoutingConnector(runtimePipelines, prometheusPipelines, istioPipelines), nil, nil + return enrichmentRoutingConnector(runtimePipelines, prometheusPipelines, istioPipelines, includeOpsScrapeMetrics), nil, nil }, ) } -func (b *Builder) addReceiverForEnrichmentRouter(runtimePipelines, prometheusPipelines, istioPipelines []telemetryv1beta1.MetricPipeline) buildComponentFunc { +func (b *Builder) addReceiverForEnrichmentRouter(runtimePipelines, prometheusPipelines, istioPipelines []telemetryv1beta1.MetricPipeline, includeOpsScrapeMetrics bool) buildComponentFunc { return b.AddReceiver( b.StaticComponentID(common.ComponentIDEnrichmentRoutingConnector), func(mp *telemetryv1beta1.MetricPipeline) any { @@ -1110,24 +1183,34 @@ func (b *Builder) addReceiverForEnrichmentRouter(runtimePipelines, prometheusPip return nil } - return enrichmentRoutingConnector(runtimePipelines, prometheusPipelines, istioPipelines) + return enrichmentRoutingConnector(runtimePipelines, prometheusPipelines, istioPipelines, includeOpsScrapeMetrics) }, ) } -func enrichmentRoutingConnector(runtimePipelines, prometheusPipelines, istioPipelines []telemetryv1beta1.MetricPipeline) common.RoutingConnectorConfig { +func enrichmentRoutingConnector(runtimePipelines, prometheusPipelines, istioPipelines []telemetryv1beta1.MetricPipeline, includeOpsScrapeMetrics bool) common.RoutingConnectorConfig { tableEntries := []common.RoutingConnectorTableEntry{} if len(runtimePipelines) > 0 { - tableEntries = append(tableEntries, enrichmentRoutingConnectorTableEntry(runtimePipelines, common.KymaInputNameEquals(common.InputSourceRuntime))) + tableEntries = append(tableEntries, enrichmentRoutingConnectorTableEntry(runtimePipelines, common.KymaInputNameEquals(common.InputSourceRuntime), nil)) } if len(prometheusPipelines) > 0 { - tableEntries = append(tableEntries, enrichmentRoutingConnectorTableEntry(prometheusPipelines, common.KymaInputNameEquals(common.InputSourcePrometheus))) + var extraPipelines []string + if includeOpsScrapeMetrics { + extraPipelines = []string{opsScrapeMetricsPipelineID} + } + + tableEntries = append(tableEntries, enrichmentRoutingConnectorTableEntry(prometheusPipelines, common.KymaInputNameEquals(common.InputSourcePrometheus), extraPipelines)) } if len(istioPipelines) > 0 { - tableEntries = append(tableEntries, enrichmentRoutingConnectorTableEntry(istioPipelines, common.KymaInputNameEquals(common.InputSourceIstio))) + var extraPipelines []string + if includeOpsScrapeMetrics { + extraPipelines = []string{opsScrapeMetricsPipelineID} + } + + tableEntries = append(tableEntries, enrichmentRoutingConnectorTableEntry(istioPipelines, common.KymaInputNameEquals(common.InputSourceIstio), extraPipelines)) } return common.RoutingConnectorConfig{ @@ -1136,11 +1219,14 @@ func enrichmentRoutingConnector(runtimePipelines, prometheusPipelines, istioPipe } } -func enrichmentRoutingConnectorTableEntry(pipelines []telemetryv1beta1.MetricPipeline, routingCondition string) common.RoutingConnectorTableEntry { +func enrichmentRoutingConnectorTableEntry(pipelines []telemetryv1beta1.MetricPipeline, routingCondition string, extraPipelines []string) common.RoutingConnectorTableEntry { + pipelineIDs := formatOutputPipelineIDs(pipelines) + pipelineIDs = append(pipelineIDs, extraPipelines...) + return common.RoutingConnectorTableEntry{ Context: "metric", Statement: fmt.Sprintf("route() where %s", routingCondition), - Pipelines: formatOutputPipelineIDs(pipelines), + Pipelines: pipelineIDs, } } diff --git a/internal/otelcollector/config/metricagent/prometheus_receiver.go b/internal/otelcollector/config/metricagent/prometheus_receiver.go index c76c3494b5..0d3731d552 100644 --- a/internal/otelcollector/config/metricagent/prometheus_receiver.go +++ b/internal/otelcollector/config/metricagent/prometheus_receiver.go @@ -40,6 +40,7 @@ func prometheusPodsReceiverConfig(collectionInterval time.Duration) *PrometheusR ScrapeInterval: collectionInterval, SampleLimit: sampleLimit, BodySizeLimit: bodySizeLimit, + ExtraScrapeMetrics: true, KubernetesDiscoveryConfigs: discoveryConfigWithNodeSelector(RolePod), JobName: appPodsJobName, RelabelConfigs: prometheusPodsRelabelConfigs(), @@ -61,6 +62,7 @@ func prometheusServicesReceiverConfig(opts BuildOptions, collectionInterval time ScrapeInterval: collectionInterval, SampleLimit: sampleLimit, BodySizeLimit: bodySizeLimit, + ExtraScrapeMetrics: true, KubernetesDiscoveryConfigs: discoveryConfigWithNodeSelector(RoleEndpoints), } @@ -158,6 +160,7 @@ func prometheusIstioReceiverConfig(envoyMetricsEnabled bool, collectionInterval BodySizeLimit: bodySizeLimit, MetricsPath: "/stats/prometheus", ScrapeInterval: collectionInterval, + ExtraScrapeMetrics: true, KubernetesDiscoveryConfigs: discoveryConfigWithNodeSelector(RolePod), RelabelConfigs: []Relabel{ keepIfRunningOnSameNode(NodeAffiliatedPod), diff --git a/internal/otelcollector/config/metricagent/testdata/delta-temporality-multiple-pipelines.yaml b/internal/otelcollector/config/metricagent/testdata/delta-temporality-multiple-pipelines.yaml index 283d130e8c..a291622877 100644 --- a/internal/otelcollector/config/metricagent/testdata/delta-temporality-multiple-pipelines.yaml +++ b/internal/otelcollector/config/metricagent/testdata/delta-temporality-multiple-pipelines.yaml @@ -49,6 +49,14 @@ service: - transform/set-kyma-input-name-runtime exporters: - routing/runtime-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test1: receivers: - routing/enrichment @@ -214,6 +222,7 @@ receivers: - regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 action: labelmap + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -269,6 +278,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -286,7 +296,7 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-envoy-metrics-if-disabled: error_mode: ignore metric_conditions: @@ -302,6 +312,17 @@ processors: metric_conditions: - conditions: - IsMatch(metric.name, "^k8s.node.network.*") and not(IsMatch(datapoint.attributes["interface"], "^(eth|en).*")) + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") k8s_attributes: auth_type: serviceAccount passthrough: false @@ -432,6 +453,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -444,6 +470,7 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "prometheus" pipelines: - metrics/output-test2 + - metrics/ops-scrape-metrics context: metric routing/prometheus-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/istio-diagnostic.yaml b/internal/otelcollector/config/metricagent/testdata/istio-diagnostic.yaml index 88f32b3855..9fe14a3c41 100644 --- a/internal/otelcollector/config/metricagent/testdata/istio-diagnostic.yaml +++ b/internal/otelcollector/config/metricagent/testdata/istio-diagnostic.yaml @@ -34,6 +34,14 @@ service: - transform/set-kyma-input-name-istio exporters: - routing/istio-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -92,6 +100,7 @@ receivers: - source_labels: [__name__] regex: istio_.* action: keep + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -107,6 +116,17 @@ processors: metric_conditions: - conditions: - IsMatch(metric.name, "^envoy_.*") and resource.attributes["kyma.input.name"] == "istio" + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") istio_noise_filter: {} k8s_attributes: auth_type: serviceAccount @@ -203,6 +223,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -211,6 +236,7 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "istio" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/istio-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/istio-envoy.yaml b/internal/otelcollector/config/metricagent/testdata/istio-envoy.yaml index f91c41c3f0..8793b5d96d 100644 --- a/internal/otelcollector/config/metricagent/testdata/istio-envoy.yaml +++ b/internal/otelcollector/config/metricagent/testdata/istio-envoy.yaml @@ -34,6 +34,14 @@ service: - transform/set-kyma-input-name-istio exporters: - routing/istio-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -92,6 +100,7 @@ receivers: - source_labels: [__name__] regex: envoy_.*|istio_.* action: keep + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -106,7 +115,18 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") istio_noise_filter: {} k8s_attributes: auth_type: serviceAccount @@ -205,6 +225,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -213,6 +238,7 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "istio" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/istio-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/istio-installed-and-disabled.yaml b/internal/otelcollector/config/metricagent/testdata/istio-installed-and-disabled.yaml index 1685ee9b72..23b3eeee1d 100644 --- a/internal/otelcollector/config/metricagent/testdata/istio-installed-and-disabled.yaml +++ b/internal/otelcollector/config/metricagent/testdata/istio-installed-and-disabled.yaml @@ -49,6 +49,14 @@ service: - transform/set-kyma-input-name-runtime exporters: - routing/runtime-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -201,6 +209,7 @@ receivers: - regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 action: labelmap + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -256,6 +265,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -308,6 +318,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -327,7 +338,7 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-envoy-metrics-if-disabled: error_mode: ignore metric_conditions: @@ -343,6 +354,17 @@ processors: metric_conditions: - conditions: - IsMatch(metric.name, "^k8s.node.network.*") and not(IsMatch(datapoint.attributes["interface"], "^(eth|en).*")) + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") k8s_attributes: auth_type: serviceAccount passthrough: false @@ -464,6 +486,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -476,6 +503,7 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "prometheus" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/prometheus-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/istio-installed-and-enabled.yaml b/internal/otelcollector/config/metricagent/testdata/istio-installed-and-enabled.yaml index 7bf1f996a8..60cc4fab83 100644 --- a/internal/otelcollector/config/metricagent/testdata/istio-installed-and-enabled.yaml +++ b/internal/otelcollector/config/metricagent/testdata/istio-installed-and-enabled.yaml @@ -61,6 +61,14 @@ service: - transform/set-kyma-input-name-runtime exporters: - routing/runtime-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -214,6 +222,7 @@ receivers: - regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 action: labelmap + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -269,6 +278,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -321,6 +331,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -356,6 +367,7 @@ receivers: - source_labels: [__name__] regex: envoy_.*|istio_.* action: keep + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -370,12 +382,12 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-diagnostic-metrics-if-input-source-prometheus: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-non-pvc-volumes-metrics: error_mode: ignore metric_conditions: @@ -386,6 +398,17 @@ processors: metric_conditions: - conditions: - IsMatch(metric.name, "^k8s.node.network.*") and not(IsMatch(datapoint.attributes["interface"], "^(eth|en).*")) + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") istio_noise_filter: {} k8s_attributes: auth_type: serviceAccount @@ -519,6 +542,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -531,10 +559,12 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "prometheus" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric - statement: route() where resource.attributes["kyma.input.name"] == "istio" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/istio-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/istio-namespace-filters.yaml b/internal/otelcollector/config/metricagent/testdata/istio-namespace-filters.yaml index 9b6fb6303a..9b8dfc170a 100644 --- a/internal/otelcollector/config/metricagent/testdata/istio-namespace-filters.yaml +++ b/internal/otelcollector/config/metricagent/testdata/istio-namespace-filters.yaml @@ -34,6 +34,14 @@ service: - transform/set-kyma-input-name-istio exporters: - routing/istio-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -94,6 +102,7 @@ receivers: - source_labels: [__name__] regex: istio_.* action: keep + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -108,12 +117,23 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-envoy-metrics-if-disabled: error_mode: ignore metric_conditions: - conditions: - IsMatch(metric.name, "^envoy_.*") and resource.attributes["kyma.input.name"] == "istio" + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/test-filter-by-namespace-istio-input: error_mode: ignore metric_conditions: @@ -216,6 +236,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -224,6 +249,7 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "istio" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/istio-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/istio-namespace-no-filters.yaml b/internal/otelcollector/config/metricagent/testdata/istio-namespace-no-filters.yaml index b897d0dcee..7c2f2d5854 100644 --- a/internal/otelcollector/config/metricagent/testdata/istio-namespace-no-filters.yaml +++ b/internal/otelcollector/config/metricagent/testdata/istio-namespace-no-filters.yaml @@ -34,6 +34,14 @@ service: - transform/set-kyma-input-name-istio exporters: - routing/istio-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -93,6 +101,7 @@ receivers: - source_labels: [__name__] regex: istio_.* action: keep + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -107,12 +116,23 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-envoy-metrics-if-disabled: error_mode: ignore metric_conditions: - conditions: - IsMatch(metric.name, "^envoy_.*") and resource.attributes["kyma.input.name"] == "istio" + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") istio_noise_filter: {} k8s_attributes: auth_type: serviceAccount @@ -209,6 +229,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -217,6 +242,7 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "istio" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/istio-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/istio-not-installed-and-disabled.yaml b/internal/otelcollector/config/metricagent/testdata/istio-not-installed-and-disabled.yaml index 618b041e1d..ad17b34a57 100644 --- a/internal/otelcollector/config/metricagent/testdata/istio-not-installed-and-disabled.yaml +++ b/internal/otelcollector/config/metricagent/testdata/istio-not-installed-and-disabled.yaml @@ -49,6 +49,14 @@ service: - transform/set-kyma-input-name-runtime exporters: - routing/runtime-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -201,6 +209,7 @@ receivers: - regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 action: labelmap + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -256,6 +265,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -270,7 +280,7 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-envoy-metrics-if-disabled: error_mode: ignore metric_conditions: @@ -286,6 +296,17 @@ processors: metric_conditions: - conditions: - IsMatch(metric.name, "^k8s.node.network.*") and not(IsMatch(datapoint.attributes["interface"], "^(eth|en).*")) + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") k8s_attributes: auth_type: serviceAccount passthrough: false @@ -407,6 +428,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -419,6 +445,7 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "prometheus" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/prometheus-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/istio-not-installed-and-enabled.yaml b/internal/otelcollector/config/metricagent/testdata/istio-not-installed-and-enabled.yaml index 1111be1223..93fe9ee74a 100644 --- a/internal/otelcollector/config/metricagent/testdata/istio-not-installed-and-enabled.yaml +++ b/internal/otelcollector/config/metricagent/testdata/istio-not-installed-and-enabled.yaml @@ -61,6 +61,14 @@ service: - transform/set-kyma-input-name-runtime exporters: - routing/runtime-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -214,6 +222,7 @@ receivers: - regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 action: labelmap + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -269,6 +278,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -299,6 +309,7 @@ receivers: - source_labels: [__name__] regex: envoy_.*|istio_.* action: keep + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -313,12 +324,12 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-diagnostic-metrics-if-input-source-prometheus: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-non-pvc-volumes-metrics: error_mode: ignore metric_conditions: @@ -329,6 +340,17 @@ processors: metric_conditions: - conditions: - IsMatch(metric.name, "^k8s.node.network.*") and not(IsMatch(datapoint.attributes["interface"], "^(eth|en).*")) + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") istio_noise_filter: {} k8s_attributes: auth_type: serviceAccount @@ -462,6 +484,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -474,10 +501,12 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "prometheus" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric - statement: route() where resource.attributes["kyma.input.name"] == "istio" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/istio-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/istio-only.yaml b/internal/otelcollector/config/metricagent/testdata/istio-only.yaml index 865374d459..b289efe71d 100644 --- a/internal/otelcollector/config/metricagent/testdata/istio-only.yaml +++ b/internal/otelcollector/config/metricagent/testdata/istio-only.yaml @@ -34,6 +34,14 @@ service: - transform/set-kyma-input-name-istio exporters: - routing/istio-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -93,6 +101,7 @@ receivers: - source_labels: [__name__] regex: istio_.* action: keep + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -107,12 +116,23 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-envoy-metrics-if-disabled: error_mode: ignore metric_conditions: - conditions: - IsMatch(metric.name, "^envoy_.*") and resource.attributes["kyma.input.name"] == "istio" + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") istio_noise_filter: {} k8s_attributes: auth_type: serviceAccount @@ -211,6 +231,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -219,6 +244,7 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "istio" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/istio-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/multiple-inputs-mixed.yaml b/internal/otelcollector/config/metricagent/testdata/multiple-inputs-mixed.yaml index 7cb9ff130f..2b08e17ad3 100644 --- a/internal/otelcollector/config/metricagent/testdata/multiple-inputs-mixed.yaml +++ b/internal/otelcollector/config/metricagent/testdata/multiple-inputs-mixed.yaml @@ -61,6 +61,14 @@ service: - transform/set-kyma-input-name-runtime exporters: - routing/runtime-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test1: receivers: - routing/enrichment @@ -240,6 +248,7 @@ receivers: - regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 action: labelmap + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -295,6 +304,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -325,6 +335,7 @@ receivers: - source_labels: [__name__] regex: istio_.* action: keep + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -339,12 +350,12 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-diagnostic-metrics-if-input-source-prometheus: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-envoy-metrics-if-disabled: error_mode: ignore metric_conditions: @@ -360,6 +371,17 @@ processors: metric_conditions: - conditions: - IsMatch(metric.name, "^k8s.node.network.*") and not(IsMatch(datapoint.attributes["interface"], "^(eth|en).*")) + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/test1-filter-by-namespace-prometheus-input: error_mode: ignore metric_conditions: @@ -523,6 +545,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -536,10 +563,12 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "prometheus" pipelines: - metrics/output-test1 + - metrics/ops-scrape-metrics context: metric - statement: route() where resource.attributes["kyma.input.name"] == "istio" pipelines: - metrics/output-test2 + - metrics/ops-scrape-metrics context: metric routing/istio-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/prometheus-diagnostic.yaml b/internal/otelcollector/config/metricagent/testdata/prometheus-diagnostic.yaml index bed6ed94c0..1a27f0ca42 100644 --- a/internal/otelcollector/config/metricagent/testdata/prometheus-diagnostic.yaml +++ b/internal/otelcollector/config/metricagent/testdata/prometheus-diagnostic.yaml @@ -34,6 +34,14 @@ service: - transform/set-kyma-input-name-prometheus exporters: - routing/prometheus-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -107,6 +115,7 @@ receivers: - regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 action: labelmap + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -162,6 +171,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -177,6 +187,17 @@ processors: metric_conditions: - conditions: - IsMatch(metric.name, "^envoy_.*") and resource.attributes["kyma.input.name"] == "istio" + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") k8s_attributes: auth_type: serviceAccount passthrough: false @@ -272,6 +293,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -280,6 +306,7 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "prometheus" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/prometheus-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/prometheus-namespace-filters.yaml b/internal/otelcollector/config/metricagent/testdata/prometheus-namespace-filters.yaml index cd6ae47638..4a2cfb0b92 100644 --- a/internal/otelcollector/config/metricagent/testdata/prometheus-namespace-filters.yaml +++ b/internal/otelcollector/config/metricagent/testdata/prometheus-namespace-filters.yaml @@ -34,6 +34,14 @@ service: - transform/set-kyma-input-name-prometheus exporters: - routing/prometheus-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -109,6 +117,7 @@ receivers: - regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 action: labelmap + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -164,6 +173,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -178,12 +188,23 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-envoy-metrics-if-disabled: error_mode: ignore metric_conditions: - conditions: - IsMatch(metric.name, "^envoy_.*") and resource.attributes["kyma.input.name"] == "istio" + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/test-filter-by-namespace-prometheus-input: error_mode: ignore metric_conditions: @@ -285,6 +306,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -293,6 +319,7 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "prometheus" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/prometheus-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/prometheus-only.yaml b/internal/otelcollector/config/metricagent/testdata/prometheus-only.yaml index 940dc66c53..aeabe8e191 100644 --- a/internal/otelcollector/config/metricagent/testdata/prometheus-only.yaml +++ b/internal/otelcollector/config/metricagent/testdata/prometheus-only.yaml @@ -34,6 +34,14 @@ service: - transform/set-kyma-input-name-prometheus exporters: - routing/prometheus-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -108,6 +116,7 @@ receivers: - regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 action: labelmap + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -163,6 +172,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -177,12 +187,23 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "prometheus" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-envoy-metrics-if-disabled: error_mode: ignore metric_conditions: - conditions: - IsMatch(metric.name, "^envoy_.*") and resource.attributes["kyma.input.name"] == "istio" + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") k8s_attributes: auth_type: serviceAccount passthrough: false @@ -280,6 +301,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -288,6 +314,7 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "prometheus" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/prometheus-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/testdata/setup-comprehensive.yaml b/internal/otelcollector/config/metricagent/testdata/setup-comprehensive.yaml index 7f40f6fa98..04fc53172f 100644 --- a/internal/otelcollector/config/metricagent/testdata/setup-comprehensive.yaml +++ b/internal/otelcollector/config/metricagent/testdata/setup-comprehensive.yaml @@ -61,6 +61,14 @@ service: - transform/set-kyma-input-name-runtime exporters: - routing/runtime-input + metrics/ops-scrape-metrics: + receivers: + - routing/enrichment + processors: + - filter/ops-keep-scrape-metrics + - filter/ops-drop-healthy-scrape-metrics + exporters: + - prometheus/ops-scrape-metrics metrics/output-test: receivers: - routing/enrichment @@ -216,6 +224,7 @@ receivers: - regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 action: labelmap + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -271,6 +280,7 @@ receivers: - source_labels: [__meta_kubernetes_service_name] target_label: service action: replace + extra_scrape_metrics: true kubernetes_sd_configs: - role: endpoints selectors: @@ -301,6 +311,7 @@ receivers: - source_labels: [__name__] regex: envoy_.*|istio_.* action: keep + extra_scrape_metrics: true kubernetes_sd_configs: - role: pod selectors: @@ -315,7 +326,7 @@ processors: error_mode: ignore metric_conditions: - conditions: - - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added") + - resource.attributes["kyma.input.name"] == "istio" and (metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_timeout_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_sample_limit" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/drop-non-pvc-volumes-metrics: error_mode: ignore metric_conditions: @@ -326,6 +337,17 @@ processors: metric_conditions: - conditions: - IsMatch(metric.name, "^k8s.node.network.*") and not(IsMatch(datapoint.attributes["interface"], "^(eth|en).*")) + filter/ops-drop-healthy-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - metric.name == "up" and datapoint.value_int == 1 + - metric.name == "scrape_body_size_bytes" and datapoint.value_double > 0 + filter/ops-keep-scrape-metrics: + error_mode: ignore + metric_conditions: + - conditions: + - not(metric.name == "up" or metric.name == "scrape_duration_seconds" or metric.name == "scrape_samples_scraped" or metric.name == "scrape_samples_post_metric_relabeling" or metric.name == "scrape_series_added" or metric.name == "scrape_body_size_bytes") filter/test-filter-by-namespace-prometheus-input: error_mode: ignore metric_conditions: @@ -474,6 +496,11 @@ exporters: initial_interval: 5s max_interval: 30s max_elapsed_time: 300s + prometheus/ops-scrape-metrics: + endpoint: ${MY_POD_IP}:9090 + metric_expiration: 5m + resource_to_telemetry_conversion: + enabled: true connectors: routing/enrichment: default_pipelines: [] @@ -486,10 +513,12 @@ connectors: - statement: route() where resource.attributes["kyma.input.name"] == "prometheus" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric - statement: route() where resource.attributes["kyma.input.name"] == "istio" pipelines: - metrics/output-test + - metrics/ops-scrape-metrics context: metric routing/istio-input: default_pipelines: diff --git a/internal/otelcollector/config/metricagent/types.go b/internal/otelcollector/config/metricagent/types.go index 693f8d6fed..10e39e65bc 100644 --- a/internal/otelcollector/config/metricagent/types.go +++ b/internal/otelcollector/config/metricagent/types.go @@ -246,14 +246,14 @@ type PrometheusScrape struct { } type Scrape struct { - JobName string `yaml:"job_name"` - SampleLimit int `yaml:"sample_limit,omitempty"` - BodySizeLimit string `yaml:"body_size_limit,omitempty"` - ScrapeInterval time.Duration `yaml:"scrape_interval,omitempty"` - MetricsPath string `yaml:"metrics_path,omitempty"` - RelabelConfigs []Relabel `yaml:"relabel_configs,omitempty"` - MetricRelabelConfigs []Relabel `yaml:"metric_relabel_configs,omitempty"` - + JobName string `yaml:"job_name"` + SampleLimit int `yaml:"sample_limit,omitempty"` + BodySizeLimit string `yaml:"body_size_limit,omitempty"` + ScrapeInterval time.Duration `yaml:"scrape_interval,omitempty"` + MetricsPath string `yaml:"metrics_path,omitempty"` + RelabelConfigs []Relabel `yaml:"relabel_configs,omitempty"` + MetricRelabelConfigs []Relabel `yaml:"metric_relabel_configs,omitempty"` + ExtraScrapeMetrics bool `yaml:"extra_scrape_metrics,omitempty"` KubernetesDiscoveryConfigs []KubernetesDiscovery `yaml:"kubernetes_sd_configs,omitempty"` TLS *TLS `yaml:"tls_config,omitempty"` diff --git a/internal/otelcollector/ports/ports.go b/internal/otelcollector/ports/ports.go index 5ef7e73370..cce5f5924e 100644 --- a/internal/otelcollector/ports/ports.go +++ b/internal/otelcollector/ports/ports.go @@ -4,6 +4,7 @@ const ( OTLPHTTP int32 = 4318 OTLPGRPC int32 = 4317 Metrics int32 = 8888 + OpsScrapeMetrics int32 = 9090 HealthCheck int32 = 13133 Pprof int32 = 1777 IstioEnvoyTelemetry int32 = 15090 diff --git a/internal/resources/otelcollector/agent.go b/internal/resources/otelcollector/agent.go index 55406ede26..78871f6267 100644 --- a/internal/resources/otelcollector/agent.go +++ b/internal/resources/otelcollector/agent.go @@ -49,6 +49,7 @@ type AgentApplierDeleter struct { baseName string extraPodLabels map[string]string + extraMetricsPorts []int32 makeAnnotationsFunc func(configChecksum string, opts AgentApplyOptions) map[string]string image string rbac rbac @@ -125,6 +126,7 @@ func NewMetricAgentApplierDeleter(globals config.Global, image, priorityClassNam globals: globals, baseName: names.MetricAgent, extraPodLabels: extraLabels, + extraMetricsPorts: []int32{ports.OpsScrapeMetrics}, makeAnnotationsFunc: makeMetricAgentAnnotations, image: image, rbac: makeMetricAgentRBAC(globals.TargetNamespace()), @@ -172,7 +174,7 @@ func (aad *AgentApplierDeleter) ApplyResources(ctx context.Context, c client.Cli configChecksum := configchecksum.Calculate([]corev1.ConfigMap{*configMap}, secretsInChecksum) - networkPolicies := makeAgentNetworkPolicies(name, opts.IstioEnabled) + networkPolicies := makeAgentNetworkPolicies(name, opts.IstioEnabled, aad.extraMetricsPorts) for _, np := range networkPolicies { if err := k8sutils.CreateOrUpdateNetworkPolicy(ctx, labelerClient, np); err != nil { @@ -291,7 +293,7 @@ func (aad *AgentApplierDeleter) makeAgentDaemonSet(configChecksum string, opts A ) } -func makeAgentNetworkPolicies(name types.NamespacedName, istioEnabled bool) []*networkingv1.NetworkPolicy { +func makeAgentNetworkPolicies(name types.NamespacedName, istioEnabled bool, extraMetricsPorts []int32) []*networkingv1.NetworkPolicy { metricsNetworkPolicy := commonresources.MakeNetworkPolicy( name, commonresources.DefaultSelector(name.Name), @@ -300,7 +302,7 @@ func makeAgentNetworkPolicies(name types.NamespacedName, istioEnabled bool) []*n map[string]string{ commonresources.LabelKeyTelemetryMetricsScraping: commonresources.LabelValueTelemetryMetricsScraping, }, - agentIngressMetricsPorts(istioEnabled)), + agentIngressMetricsPorts(istioEnabled, extraMetricsPorts)), ) agentNetworkPolicy := commonresources.MakeNetworkPolicy( name, @@ -325,7 +327,7 @@ func makeMetricAgentAnnotations(configChecksum string, opts AgentApplyOptions) m annotations := map[string]string{commonresources.AnnotationKeyChecksumConfig: configChecksum} if opts.IstioEnabled { - annotations[commonresources.AnnotationKeyIstioExcludeInboundPorts] = strconv.Itoa(int(ports.Metrics)) + annotations[commonresources.AnnotationKeyIstioExcludeInboundPorts] = fmt.Sprintf("%d,%d", ports.Metrics, ports.OpsScrapeMetrics) // Provision Istio certificates for Prometheus Receiver running as a part of MetricAgent by injecting a sidecar which will rotate SDS certificates and output them to a volume. annotations[commonresources.AnnotationKeyIstioProxyConfig] = fmt.Sprintf(`# configure an env variable OUTPUT_CERTS to write certificates to the given folder proxyMetadata: @@ -399,8 +401,9 @@ func makeFileLogCheckPointVolumeMount() corev1.VolumeMount { } } -func agentIngressMetricsPorts(istioEnabled bool) []int32 { +func agentIngressMetricsPorts(istioEnabled bool, extraPorts []int32) []int32 { metricsPorts := []int32{ports.Metrics} + metricsPorts = append(metricsPorts, extraPorts...) if istioEnabled { metricsPorts = append(metricsPorts, ports.IstioEnvoyTelemetry) diff --git a/internal/resources/otelcollector/testdata/metric-agent-fips-enabled.yaml b/internal/resources/otelcollector/testdata/metric-agent-fips-enabled.yaml index 94fdd1a0ae..20631cba22 100644 --- a/internal/resources/otelcollector/testdata/metric-agent-fips-enabled.yaml +++ b/internal/resources/otelcollector/testdata/metric-agent-fips-enabled.yaml @@ -204,6 +204,8 @@ spec: ports: - port: 8888 protocol: TCP + - port: 9090 + protocol: TCP podSelector: matchLabels: app.kubernetes.io/name: telemetry-metric-agent diff --git a/internal/resources/otelcollector/testdata/metric-agent-istio.yaml b/internal/resources/otelcollector/testdata/metric-agent-istio.yaml index 70aafd48a6..b7ca65b7c5 100644 --- a/internal/resources/otelcollector/testdata/metric-agent-istio.yaml +++ b/internal/resources/otelcollector/testdata/metric-agent-istio.yaml @@ -79,7 +79,7 @@ spec: proxyMetadata: OUTPUT_CERTS: /etc/istio-output-certs sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-output-certs"}]' - traffic.sidecar.istio.io/excludeInboundPorts: "8888" + traffic.sidecar.istio.io/excludeInboundPorts: 8888,9090 traffic.sidecar.istio.io/includeOutboundIPRanges: "" traffic.sidecar.istio.io/includeOutboundPorts: 4317,9090 labels: @@ -226,6 +226,8 @@ spec: ports: - port: 8888 protocol: TCP + - port: 9090 + protocol: TCP - port: 15090 protocol: TCP podSelector: diff --git a/internal/resources/otelcollector/testdata/metric-agent-vpa-zero-max-memory.yaml b/internal/resources/otelcollector/testdata/metric-agent-vpa-zero-max-memory.yaml index 252f55e009..87c14e2ea8 100644 --- a/internal/resources/otelcollector/testdata/metric-agent-vpa-zero-max-memory.yaml +++ b/internal/resources/otelcollector/testdata/metric-agent-vpa-zero-max-memory.yaml @@ -248,6 +248,8 @@ spec: ports: - port: 8888 protocol: TCP + - port: 9090 + protocol: TCP podSelector: matchLabels: app.kubernetes.io/name: telemetry-metric-agent diff --git a/internal/resources/otelcollector/testdata/metric-agent-vpa.yaml b/internal/resources/otelcollector/testdata/metric-agent-vpa.yaml index 1f8e684556..2d568ef66d 100644 --- a/internal/resources/otelcollector/testdata/metric-agent-vpa.yaml +++ b/internal/resources/otelcollector/testdata/metric-agent-vpa.yaml @@ -248,6 +248,8 @@ spec: ports: - port: 8888 protocol: TCP + - port: 9090 + protocol: TCP podSelector: matchLabels: app.kubernetes.io/name: telemetry-metric-agent diff --git a/internal/resources/otelcollector/testdata/metric-agent.yaml b/internal/resources/otelcollector/testdata/metric-agent.yaml index 24c82933cc..96176a75c2 100644 --- a/internal/resources/otelcollector/testdata/metric-agent.yaml +++ b/internal/resources/otelcollector/testdata/metric-agent.yaml @@ -218,6 +218,8 @@ spec: ports: - port: 8888 protocol: TCP + - port: 9090 + protocol: TCP podSelector: matchLabels: app.kubernetes.io/name: telemetry-metric-agent