Is your question related to a specific component?
VictoriaTraces single-node v0.11.0, trace ID indexing and Jaeger trace-by-ID API.
Describe the question in detail
Does VictoriaTraces support storing spans with the same Trace ID in different tenants, with each tenant querying only its own spans?
For example, two applications belonging to different tenants may propagate the same trace context during an inter-service request and export their respective spans to their own tenants. We are not asking for cross-tenant queries or access to another tenant's spans.
The documentation describes multitenancy via AccountID/ProjectID headers. In an isolated reproduction using the official v0.11.0 image, two tenants can store spans with the same Trace ID and different Span IDs. LogsQL returns each tenant's own span, but Jaeger trace-by-ID returns HTTP 200 for the first tenant and HTTP 404 for the second.
Questions:
- Is this use case supported?
- Is the behavior below a bug or an intentional limitation?
- Is there a design reason why the in-memory trace index key excludes the tenant ID? If this is intentional, what configuration or usage pattern is recommended, and where is the limitation documented?
Reproduction
Start a fresh instance (no mounted data). The shortened index flush interval makes the example faster; the original observation also occurred with the default interval.
docker run --rm --name vt-tenant-repro -p 127.0.0.1:19429:10428 victoriametrics/victoria-traces:v0.11.0 -storageDataPath=/tmp/traces -insert.indexFlushInterval=2s
Run this Python 3 script in another terminal. It uses only the standard library, synthetic data, different Span IDs, and timestamps two minutes in the past. It waits 15 seconds after ingestion, longer than the configured index buffering and default data flush intervals.
import json, time, urllib.request, urllib.parse, urllib.error
BASE = 'http://127.0.0.1:19429'
TRACE = '1234567890abcdef1234567890abcdef'
def request(path, account, data=None):
headers = {'AccountID': str(account), 'ProjectID': '7'}
if data is not None:
headers['Content-Type'] = 'application/json'
data = json.dumps(data).encode()
try:
with urllib.request.urlopen(urllib.request.Request(BASE + path, data=data, headers=headers), timeout=15) as r:
return r.status, r.read().decode()
except urllib.error.HTTPError as e:
return e.code, e.read().decode()
start = time.time_ns() - 120_000_000_000
for account, span in [(101, '1111111111111111'), (102, '2222222222222222')]:
payload = {'resourceSpans': [{'resource': {'attributes': [{'key': 'service.name', 'value': {'stringValue': 'service-' + str(account)}}]}, 'scopeSpans': [{'spans': [{'traceId': TRACE, 'spanId': span, 'name': 'example', 'kind': 2, 'startTimeUnixNano': str(start), 'endTimeUnixNano': str(start + 100_000_000)}]}]}]}
print('INSERT', account, request('/insert/opentelemetry/v1/traces', account, payload), flush=True)
# Wait longer than both index buffers and the default data flush interval.
time.sleep(15)
for account in [101, 102]:
query = urllib.parse.urlencode({'query': 'trace_id:="' + TRACE + '"', 'limit': '10'})
print('RAW', account, request('/select/logsql/query?' + query, account), flush=True)
print('JAEGER', account, request('/select/jaeger/api/traces/' + TRACE, account), flush=True)Observed:
| Tenant (AccountID, ProjectID) |
OTLP insert |
LogsQL span query |
Jaeger trace-by-ID |
| (101, 7) |
200 |
own span 1111111111111111 |
200, own span |
| (102, 7) |
200 |
own span 2222222222222222 |
404, trace not found |
If supported, the expected behavior is that both tenants can retrieve their own spans independently, even when the Trace ID is identical. No cross-tenant span disclosure was observed in this reproduction.
Code investigation
In v0.11.0 index_helper.go, mustPushIndex keys the maps only by the Trace ID. An existing entry updates its time range and returns without comparing tenant IDs. The tenant stored in the entry remains that of the initial insertion. flushIndexInMap then writes the index to that tenant.
This appears to explain the observation, but we would appreciate clarification of the intended multitenancy contract. We reviewed #21, #71 and #81; they discuss indexing performance and accuracy, but we did not find discussion of this cross-tenant case. #109 concerns a recent-trace indexing delay; this reproduction waits beyond the shortened index flush interval.
Documentation consulted
Is your question related to a specific component?
VictoriaTraces single-node v0.11.0, trace ID indexing and Jaeger trace-by-ID API.
Describe the question in detail
Does VictoriaTraces support storing spans with the same Trace ID in different tenants, with each tenant querying only its own spans?
For example, two applications belonging to different tenants may propagate the same trace context during an inter-service request and export their respective spans to their own tenants. We are not asking for cross-tenant queries or access to another tenant's spans.
The documentation describes multitenancy via AccountID/ProjectID headers. In an isolated reproduction using the official v0.11.0 image, two tenants can store spans with the same Trace ID and different Span IDs. LogsQL returns each tenant's own span, but Jaeger trace-by-ID returns HTTP 200 for the first tenant and HTTP 404 for the second.
Questions:
Reproduction
Start a fresh instance (no mounted data). The shortened index flush interval makes the example faster; the original observation also occurred with the default interval.
Run this Python 3 script in another terminal. It uses only the standard library, synthetic data, different Span IDs, and timestamps two minutes in the past. It waits 15 seconds after ingestion, longer than the configured index buffering and default data flush intervals.
Observed:
If supported, the expected behavior is that both tenants can retrieve their own spans independently, even when the Trace ID is identical. No cross-tenant span disclosure was observed in this reproduction.
Code investigation
In v0.11.0 index_helper.go,
mustPushIndexkeys the maps only by the Trace ID. An existing entry updates its time range and returns without comparing tenant IDs. The tenant stored in the entry remains that of the initial insertion.flushIndexInMapthen writes the index to that tenant.This appears to explain the observation, but we would appreciate clarification of the intended multitenancy contract. We reviewed #21, #71 and #81; they discuss indexing performance and accuracy, but we did not find discussion of this cross-tenant case. #109 concerns a recent-trace indexing delay; this reproduction waits beyond the shortened index flush interval.
Documentation consulted