Skip to content

Reduce OpenTelemetry ingestion volume #2414

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ejsmith wants to merge 8 commits into
base: main
Choose a base branch
from
+48 −5
Open

Reduce OpenTelemetry ingestion volume #2414

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
6f93f3c
Reduce OpenTelemetry ingestion volume
ejsmith Jul 28, 2026
24f3e55
Merge branch 'main' into feature/reduce-otel-volume
ejsmith Jul 29, 2026
2bc894e
Drop EntityChanged messaging spans
ejsmith Jul 29, 2026
443f6ac
Upgrade SigNoz k8s-infra chart to 0.16.0
ejsmith Jul 29, 2026
12111d8
Fix event-post Redis span filter
ejsmith Jul 29, 2026
c7c6152
Merge branch 'main' into feature/reduce-otel-volume
ejsmith Jul 29, 2026
fb99bd9
Drop unused metrics at the SDK source
ejsmith Jul 29, 2026
4c5f28f
Keep metric filtering in Collector config
ejsmith Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion k8s/ex-prod-tasks.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ helm upgrade vpa fairwinds-stable/vpa --namespace vpa -f vpa-values.yaml --reset

# upgrade signoz
helm repo update
helm upgrade --reset-values signoz-collector signoz/k8s-infra -f signoz.yaml --set "signozApiKey=$SIGNOZ_KEY" --dry-run
helm upgrade --reset-values signoz-collector signoz/k8s-infra --version 0.16.0 -f signoz.yaml --set "signozApiKey=$SIGNOZ_KEY" --dry-run

# upgrade elasticsearch metrics exporter
helm upgrade --reset-values es-exporter prometheus-community/prometheus-elasticsearch-exporter `
Expand Down
2 changes: 1 addition & 1 deletion k8s/ex-setup.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ kubectl apply -f ex-$ENV-redis.yaml -n ex-$ENV

# install signoz otel collector
helm repo add signoz https://charts.signoz.io
helm install signoz-collector signoz/k8s-infra -f signoz.yaml --set "signozApiKey=$SIGNOZ_KEY"
helm install signoz-collector signoz/k8s-infra --version 0.16.0 -f signoz.yaml --set "signozApiKey=$SIGNOZ_KEY"

# install elasticsearch metrics exporter for signoz (prod only)
helm install es-exporter prometheus-community/prometheus-elasticsearch-exporter `
Expand Down
42 changes: 39 additions & 3 deletions k8s/signoz.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,12 @@ global:
cloud: aks
clusterName: ex-k8s-v6
deploymentEnvironment: prod
otelCollectorEndpoint: ingest.us.signoz.cloud:443
otelCollectorEndpoint: https://ingest.us.signoz.cloud:443
otelInsecure: false
presets:
otlpExporter:
enabled: false
otlphttpExporter:
enabled: true
loggingExporter:
enabled: false
Expand Down Expand Up @@ -33,9 +35,43 @@ otelAgent:
endpoint: http://apm.elastic-system.svc:8200
tls:
insecure: true
processors:
filter/drop_unused_metrics:
error_mode: ignore
metrics:
exclude:
match_type: strict
metric_names:
- http.client.request.duration
- http.client.request.time_in_queue
- http.client.connection.duration
- http.client.open_connections
- http.client.active_requests
- dns.lookup.duration
- kestrel.connection.duration
- aspnetcore.authentication.authenticate.duration
- aspnetcore.routing.match_attempts
- dotnet.exceptions
- dotnet.gc.collections
- dotnet.gc.last_collection.heap.size
- dotnet.gc.last_collection.heap.fragmentation.size
- dotnet.process.cpu.time
- foundatio.lock.wait.time
- ex.posts.size
- ex.events.parse.jsoneventparserplugin
- ex.events.field.count
- ex.eventpipeline.markascriticalaction
filter/drop_cache_spans:
error_mode: ignore
traces:
span:
- 'attributes["db.system"] == "redis" and status.code != STATUS_CODE_ERROR and (resource.attributes["service.name"] == "exceptionless-web" or resource.attributes["service.name"] == "job-event-posts")'

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore the prefixed event-post service matcher

Fresh evidence after the prior thread is that fb99bd9 reverted 12111d8's correction in the reviewed target tree. The production job loads Apm.ServiceName: exceptionless from src/Exceptionless.Job/appsettings.yml, while Program.cs passes job-event-posts to ApmConfig, whose concatenation emits service.name=exceptionless-job-event-posts; therefore successful Redis spans from this high-volume worker never match this predicate and continue to both exporters. Restore the prefixed matcher.

Useful? React with 👍 / 👎.

- 'name == "Message: EntityChanged"'
service:
pipelines:
traces:
exporters: [otlp, otlp/elastic]
processors: [filter/drop_cache_spans, batch]
exporters: [otlphttp, otlp/elastic]
metrics:
exporters: [otlp, otlp/elastic]
processors: [filter/drop_unused_metrics, batch]
Comment thread
ejsmith marked this conversation as resolved.
exporters: [otlphttp, otlp/elastic]
7 changes: 7 additions & 0 deletions src/Exceptionless.Web/ApmExtensions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,13 @@ public static IHostBuilder AddApm(this IHostBuilder builder, ApmConfig config)
b.AddRuntimeInstrumentation();
b.AddProcessInstrumentation();

b.AddView(
"http.server.request.duration",
new ExplicitBucketHistogramConfiguration
{
Boundaries = [0.005, 0.025, 0.1, 0.5, 1, 5]
});

if (config.Console)
b.AddConsoleExporter((_, metricReaderOptions) =>
{
Expand Down
Loading