feat: organization-scoped event custom fields with management UI and filter integration - #1815
feat: organization-scoped event custom fields with management UI and filter integration#1815ejsmith wants to merge 12 commits into
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds custom field support for events by updating the event model and repository logic, removing legacy query visitor code, and updating test infrastructure to support the new behavior.
- Removed obsolete EventFieldsQueryVisitor usage and file.
- Introduced automatic custom field creation in the event repository.
- Updated the PersistentEvent model to implement virtual custom fields and modified test configurations accordingly.
Reviewed Changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/Exceptionless.Tests/Search/PersistentEventQueryValidatorTests.cs | Removed legacy query visitor call in tests. |
| tests/Exceptionless.Tests/Search/EventIndexTests.cs | Updated repository query to include organization filtering. |
| tests/Exceptionless.Tests/Migrations/FixDuplicateStacksMigrationTests.cs | Set log level to Trace for migration tests. |
| tests/Exceptionless.Tests/AppWebHostFactory.cs | Added Kibana container configuration. |
| src/Exceptionless.Core/Repositories/Queries/Visitors/EventFieldsQueryVisitor.cs | Removed unused visitor implementation. |
| src/Exceptionless.Core/Repositories/EventRepository.cs | Added auto-creation of custom fields and related tenant logic. |
| src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs | Removed legacy dynamic mapping for event index custom fields. |
| src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs | Added index creation for custom fields. |
| src/Exceptionless.Core/Models/PersistentEvent.cs | Updated Idx property type and implemented IHaveVirtualCustomFields. |
Files not reviewed (1)
- src/Exceptionless.Core/Exceptionless.Core.csproj: Language not supported
niemyjski
changed the title
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…filter integration Add organization-level custom field management that allows premium users to define typed, searchable event fields (keyword, string, numeric, date, boolean) with full lifecycle management including soft-delete, index slot reuse, and saved view protection. Key changes: - Custom field CRUD on organization controller with validation and plan gating - EventCustomFieldService for document change handling and field lifecycle - Premium feature detection in saved views (UsesPremiumFeatures flag) - Filter integration with operator support per field type - Angular UI: management dialog, filter picker integration - Svelte UI: custom fields management and filter components - Comprehensive test coverage (107+ custom field tests) - Max 20 active fields per organization, slot-based index allocation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
0a8855a to
b277880
Compare
…seline, fix Prettier - Restore accidentally deleted ControllerManifestTests.cs (was removed during squash) - Generate controller-manifest.json baseline with custom fields endpoints included - Fix Prettier formatting in sidebar.svelte Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…heck Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…finedAsync The previous baseline was missing PostPredefinedAsync because UPDATE_SNAPSHOTS ran against a stale build. Rebuilt and regenerated correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…es, ContainsKey pattern - PersistentEvent.GetCustomFields(): consolidate value-type guard into the .Where() predicate - EventCustomFieldService: use .OfType<PersistentEvent>() to replace the outer .Where() null filter and the redundant inner .Where(d => d is not null) in the document loop - CustomFieldIndexingTests: fix regex type names (integer→int, boolean→bool; add string and float) - CustomFieldApiTests: fix 'integer' index type to 'int' (integer is not a supported type) - CustomFieldApiTests: replace ContainsKey+indexer with TryGetValue for SessionHasError assertion The 'integer' index type bug was silent: ConvertValue returns null for unknown types so the field would be stored but never actually indexed, with no error surfaced to the user. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ring
ConvertToKeyword and ConvertToString were calling value.ToString() without
a format provider. On servers with a non-en-US locale (e.g. de-DE), float,
double, and decimal values produce '1,5' instead of '1.5', breaking
Elasticsearch keyword-field searches because the stored value and the query
value use different decimal separators. DateTime values were also formatted
with the current thread culture instead of ISO 8601.
Fix: introduce a FormatInvariant helper that uses CultureInfo.InvariantCulture
for numeric types and the roundtrip 'O' ISO-8601 format for DateTime/DateTimeOffset.
The bool.ToString() behaviour ("True"/"False") is preserved for backwards
compatibility with already-indexed data.
Add regression tests that explicitly set CultureInfo to de-DE and assert
that the output always uses an invariant decimal separator. Also add tests
for long coercion in ConvertToBool, and for DateTime/DateTimeOffset keyword
formatting.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Focused custom-field and session directionAfter re-reading the current implementation, I think the session compatibility concern is real, but the larger proposed redesign is more machinery than this PR needs. Contract to preserve
Changes required before merge
Scope I do not think is required hereI would not require an administrative preflight feature, a global lifecycle lock, a persisted compatibility cutoff, an eight-stage deployment, new operator policy, or a general backfill/reindex system. Foundatio already serializes slot allocation per entity/tenant/type, and this branch already serializes user creation for quota enforcement. Exact invariant validation plus dual-read compatibility is the proportional fix. Verification needed
This keeps the useful part of the feedback—session continuity and deterministic reserved slots—without turning this PR into a broader custom-field control plane. |
|
Follow-up audit for the focused custom-field and session direction is complete. Feedback disposition
RCA and final fixThe session continuity concern was real: pooled system slots replaced retained legacy session indexes. The branch now centralizes the three system descriptors, validates their exact reserved invariants, and performs legacy/current dual reads with correct positive, exists, missing, range, and negation semantics. The remaining valid issue was the public contract, not another migration framework: automatic arbitrary extended-data indexing was removed, but the PR description and filtering docs still called the change additive and even suggested a
The PR description now reflects the current four endpoints, three system fields, synchronous soft-delete policy, known limitations, and breaking search behavior. Verification
Thermo-nuclear structural review found no PR-caused file crossing 1,000 lines, no justified code-judo simplification beyond the existing small canonical system descriptor/visitor, and no reason to add a general backfill, dual-write, or migration control plane for this accepted one-time cutover. |
|
Summary
Adds opt-in, organization-scoped custom event fields backed by pooled Elasticsearch slots, with Svelte management UI and event-filter integration.
This is a forward-only indexing cutover. User-defined fields are indexed only after an organization creates a definition; historical events are not backfilled or reindexed.
What changed
Backend
PersistentEventexposes primitiveDatavalues through Foundatio's virtual custom-field contract.EventCustomFieldServicevalidates and converts values, strips client-supplied managed slots, enforces organization-scoped creation locks and quotas, and provisions protected system fields.EventRepositoryresolvesdata.{name}andidx.{name}through the organization's active field definition and blocks raw pooled-slot access.System-field compatibility
@ref:session,sessionend, andhaserrorare Exceptionless-owned system fields. They:Positive, range, and exists filters use legacy-or-current semantics; missing uses legacy-and-current; negation wraps the combined expression. Premium gating for Sessions functionality remains unchanged.
API
User-defined fields require a paid plan and are limited to 20 active fields by default. System fields do not consume that quota.
Deletion is a synchronous soft-delete. It is blocked when a saved view references the field, and its slot is not reclaimed or reused while retained events may still contain values.
Frontend
Upgrade and customer impact
Before this change, paid organizations automatically copied primitive extended-data values into named
Idxfields and arbitrarydata.foofilters resolved to those fields. This PR intentionally replaces that behavior with explicit definitions:Self-hosted operators should inventory saved views and integrations using arbitrary
data.*filters, create the required definitions before resuming ingestion when continuity matters, and re-ingest retained events only when historical search is worth the one-time cost.For the hosted rollout, the proportional plan is a read-only usage audit plus targeted notice to affected organizations, with manual seeding only for confirmed fields if the affected set is small.
Verification
dotnet build tests/Exceptionless.Tests/Exceptionless.Tests.csproj --no-restore --nologo --disable-build-servers -m:1EXCEPTIONLESS_TEST_SCOPE=pr1815-7d71 dotnet test tests/Exceptionless.Tests/Exceptionless.Tests.csproj --no-build -- --filter-class Exceptionless.Tests.CustomFields.CustomFieldIndexingTests— 24 passeddeno task checkindocsdeno task verifyindocs— build, 4 tests, telemetry validation, and local-link validation passed513270fd8— website, version, API, client, E2E, docker, and CLA checks passed; deploy/publish jobs skipped as expected.Focused coverage includes CRUD/auth/quota behavior, tenant isolation, type conversion, concurrency, raw-slot blocking, system-slot conflicts, session legacy/current Boolean semantics, free-plan system indexing, organization teardown, and the explicit forward-only legacy cutover.
Breaking behavior
This PR changes search behavior for arbitrary extended-data fields that do not have an organization custom-field definition. Existing API shapes remain compatible, but existing saved views and integrations that depend on automatic
data.*indexing require a one-time field-definition rollout.Known limitations