Skip to content

[evcc] Add evcc integration for site/loadpoint metrics and logs - #20323

Open
smnschndr wants to merge 1 commit into
elastic:mainfrom
smnschndr:add-evcc-integration
Open

[evcc] Add evcc integration for site/loadpoint metrics and logs#20323
smnschndr wants to merge 1 commit into
elastic:mainfrom
smnschndr:add-evcc-integration

Conversation

@smnschndr

Copy link
Copy Markdown
Contributor

Proposed commit message

Adds a new evcc integration collecting metrics and logs from evcc, the open-source solar charging and energy management system.

Three data streams:

  • site (metrics): home power, grid power, solar (PV) power/energy, home battery power/SoC/capacity, and cost/CO2/solar-share statistics for several reporting periods. Collected via CEL input polling GET /api/state.
  • loadpoint (metrics): one event per configured charge point — mode, charging/connected/enabled state, charge power/energy/duration, effective current/SoC limits, and connected vehicle name/SoC/range. Same endpoint as site, split by loadpoint.
  • log (logs): evcc's own application logs, collected via journald and parsed into ECS fields (event.action/category/type/outcome, destination.ip/port, url.*, http.response.status_code, event.duration). Charging session lifecycle events, vehicle API errors, device read failures, and charger logic warnings are parsed into dedicated fields; high-volume DEBUG-level telemetry is kept as plain message text since the same readings are already collected in structured form by site/loadpoint.

The evcc /api/state endpoint returns a very large, deeply nested response (including forecast arrays and device certificates) and is not formally documented upstream ("the actual state structure is not documented yet... details may change between releases", per evcc's own OpenAPI spec). The CEL programs curate only the fields listed above rather than shipping the raw payload, and the battery field is normalized in the ingest pipeline to handle a real shape difference observed between evcc versions (older releases publish it as a bare array, newer releases as a single object with a nested device list) — found by running the system tests against a live evcc instance rather than only the public demo.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices (no dashboards included in this initial version)

Author's Checklist

  • elastic-package build / lint pass
  • elastic-package test pipeline passes for all three data streams
  • elastic-package test system passes for site and loadpoint against both a mock server and a live evcc instance
  • Log parsing derived from analysis of ~9.5M real journald lines and validated against ~38.5k real historical log lines with zero pipeline errors

How to test this PR locally

elastic-package stack up -d
elastic-package test pipeline -d site,loadpoint,log
elastic-package test system -d site
elastic-package test system -d loadpoint

Related issues

Screenshots

No dashboards yet in this initial version.

@smnschndr
smnschndr requested a review from a team as a code owner July 23, 2026 10:39
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Reviewers

Buildkite won't run for external contributors automatically; you need to add a comment:

  • /test : will kick off a build in Buildkite.

NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details.

smnschndr added a commit to smnschndr/integrations that referenced this pull request Jul 23, 2026
@smnschndr
smnschndr force-pushed the add-evcc-integration branch 2 times, most recently from 56c1e95 to 7be3327 Compare July 23, 2026 10:42
@smnschndr

Copy link
Copy Markdown
Contributor Author

/test

@andrewkroh andrewkroh added New Integration Issue or pull request for creating a new integration package. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Jul 23, 2026
@smnschndr

Copy link
Copy Markdown
Contributor Author

Ran a full self-review pass against this package (package-spec, ecs-field-mappings, cel-programs, ingest-pipelines, integration-testing, anonymize-logs) and pushed fixes for everything it found:

  • CEL (site, loadpoint): want_more is now explicit on the success path too (was error-path only); redact.fields: ~ declared for completeness.
  • Pipeline: on_failure append processors now tagged; log pipeline now appends destination.ip to related.ip.
  • Fields: split each data stream's base-fields.yml down to the canonical 6 entries and moved everything else into a new fields/ecs.yml.
  • Manifest: filled in missing description/required on stream variables.
  • Tests: added preserve_original_event to every pipeline test's fields.tags; renamed fixtures to the test-<package>-<stream>-<type>-sample.log convention; regenerated all sample_event.json via elastic-package test system --generate instead of hand-authoring; swapped a private-range IP placeholder for an RFC 5737 documentation address; removed the stale scaffold sample_event.json at the package root.
  • Tests (log): added a journald-based system test (Dockerfile + docker-compose building a binary journal via systemd-journal-remote, mirroring packages/journald's pattern).

elastic-package build/lint are clean, all three pipeline tests pass deterministically, and the site/loadpoint system tests pass against both the mock and a live evcc instance. The new log system test's mock is independently verified correct (image builds, journal imports cleanly, and the exact journalctl invocation the agent constructs returns correctly-matched events when run manually), but the end-to-end run is flaky in my local macOS/Podman setup in a way consistent with known container-orchestration limitations for the journald input (#10998) rather than a package issue — flagging in case it needs a second look in CI.

@smnschndr
smnschndr force-pushed the add-evcc-integration branch from d817b86 to fe94a12 Compare July 23, 2026 13:22
Adds a new evcc integration collecting metrics and logs from evcc, the
open-source solar charging and energy management system.

Three data streams:
- site (metrics): home power, grid power, solar (PV) power/energy, home
  battery power/SoC/capacity, and cost/CO2/solar-share statistics, collected
  via CEL input polling GET /api/state.
- loadpoint (metrics): one event per configured charge point - mode,
  charging/connected/enabled state, charge power/energy/duration, effective
  current/SoC limits, and connected vehicle name/SoC/range.
- log (logs): evcc's own application logs, collected via journald and parsed
  into ECS fields (event.action/category/type/outcome, destination.ip/port,
  url.*, http.response.status_code, event.duration), derived from analysis
  of real production journald data and validated against ~38.5k historical
  log lines with zero pipeline errors. High-volume DEBUG-level telemetry is
  kept as plain message text since the same readings are already collected
  in structured form by site/loadpoint.

The battery field is normalized in the site ingest pipeline to handle a
real shape difference between evcc versions (older releases publish it as
a bare array, newer releases as a single object with a nested device list)
- found by running the system tests against a live evcc instance rather
  than only the public demo.

Also includes fixes from a full self-review pass (package-spec,
ecs-field-mappings, cel-programs, ingest-pipelines, integration-testing,
anonymize-logs): explicit want_more on CEL success paths, related.ip
population, base-fields.yml split to the canonical 6 entries plus a new
fields/ecs.yml per data stream, tagged on_failure processors, complete
manifest variable documentation, preserve_original_event coverage in test
fixtures, system-test-generated sample events, RFC 5737 test IP addresses,
and a journald-based system test for the log data stream mirroring
packages/journald's mock pattern.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019CcHjGhVCunZ6cewt2WyrX
@smnschndr
smnschndr force-pushed the add-evcc-integration branch from fe94a12 to 01030c0 Compare July 23, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. New Integration Issue or pull request for creating a new integration package.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants