Skip to content

Update influencer fields of Security ML integrations #20335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sodhikirti07 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Update influencer fields of Security ML integrations #20335

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4e6f317
Update influencer fields for DED, LMD, DGA and ProblemChild jobs
sodhikirti07 Jul 23, 2026
3da6f1e
Update transform source index for windows PAD
sodhikirti07 Jul 23, 2026
2d215c6
Update changelog.yml
sodhikirti07 Jul 23, 2026
ebde7a2
Update ded transform and README.md
sodhikirti07 Jul 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/ded/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
- version: "3.4.0"
changes:
- description: Add file and process influencers to ML jobs and extend the network transform group_by with process and network fields.
type: enhancement
link: https://github.com/elastic/integrations/pull/20335
- version: "3.3.0"
changes:
- description: Update transform source index to scope to the relevant data streams.
Expand Down
8 changes: 6 additions & 2 deletions packages/ded/elasticsearch/transform/pivot_transform_ea/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@
name: host.id
- external: ecs
name: user.name
- external: ecs
name: user.id
- external: ecs
name: event.module
- external: ecs
Expand All @@ -22,6 +20,12 @@
name: source.ip
- external: ecs
name: process.name
- external: ecs
name: network.transport
- external: ecs
name: process.executable
- external: ecs
name: process.parent.name
- external: ecs
name: process.code_signature.subject_name
- external: ecs
Expand Down
26 changes: 18 additions & 8 deletions packages/ded/elasticsearch/transform/pivot_transform_ea/transform.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@

dest:
index: ml_network_ded_ea-3.3.0
index: ml_network_ded_ea-3.4.0
aliases:
- alias: ml_network_ded_ea.latest
move_on_creation: true
- alias: ml_network_ded_ea.all
move_on_creation: false
pipeline: 3.3.0-ml_ded_ingest_pipeline
pipeline: 3.4.0-ml_ded_ingest_pipeline
description: This transform runs every 30 minutes and collects network logs to detect data exfiltration in your environment for the past month up to the runtime.
frequency: 30m
pivot:
Expand All @@ -26,10 +26,9 @@ pivot:
field: host.id
'user.name':
terms:
field: user.name
user.id:
terms:
field: user.id
script:
source: "return doc.containsKey('user.name') && !doc['user.name'].isEmpty() ? doc['user.name'].value : ''"
lang: painless

@jmcarlock jmcarlock Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use missing_bucket: true for these instead?

@sodhikirti07 sodhikirti07 Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this option instead of the script

event.module:
terms:
field: event.module
Expand Down Expand Up @@ -66,6 +65,17 @@ pivot:
'process.name':
terms:
field: process.name
'network.transport':
terms:
field: network.transport
'process.executable':
terms:
field: process.executable

@jmcarlock jmcarlock Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other thing I picked up, if you are using Network Packet Capture as the data source, do you need to enable the monitor_processes setting in the integration policy?
https://github.com/elastic/integrations/blob/63384ae307ea445e86c064c37c84eebf24815e97/packages/network_traffic/_dev/build/docs/README.md#monitor_processes

@sodhikirti07 sodhikirti07 Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good find, added this under a note in README

'process.parent.name':
terms:
script:
source: "return doc.containsKey('process.parent.name') && !doc['process.parent.name'].isEmpty() ? doc['process.parent.name'].value : ''"
lang: painless
'process.code_signature.subject_name':
terms:
script:
Expand All @@ -77,7 +87,7 @@ pivot:
source: "return doc.containsKey('process.code_signature.trusted') && !doc['process.code_signature.trusted'].isEmpty() ? doc['process.code_signature.trusted'].value.toString() : ''"
lang: painless
source:
index: "logs-endpoint.events.network-*,logs-network_traffic.*"
index: "logs-endpoint.events.network-*,logs-network_traffic*"
query:
bool:
filter:
Expand Down Expand Up @@ -113,5 +123,5 @@ sync:
delay: 120s
field: "@timestamp"
_meta:
fleet_transform_version: 3.3.0
fleet_transform_version: 3.4.0
run_as_kibana_system: false
35 changes: 25 additions & 10 deletions packages/ded/kibana/ml_module/ded-ml.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,6 @@
"host.id",
"user.name",
"event.module",
"user.id",
"process.name",
"source.ip",
"destination.ip",
Expand All @@ -81,7 +80,9 @@
"destination.geo.country_iso_code",
"source.bytes",
"process.code_signature.subject_name",
"process.code_signature.trusted"
"process.code_signature.trusted",
"process.executable",
"process.parent.name"
]
},
"data_description": {
Expand Down Expand Up @@ -123,13 +124,16 @@
"host.id",
"user.name",
"event.module",
"user.id",
"process.name",
"source.ip",
"destination.ip",
"source.bytes",
"process.code_signature.subject_name",
"process.code_signature.trusted"
"process.code_signature.trusted",
"destination.port",
"network.transport",
"process.executable",
"process.parent.name"
]
},
"data_description": {
Expand Down Expand Up @@ -171,14 +175,16 @@
"host.id",
"user.name",
"event.module",
"user.id",
"process.name",
"source.ip",
"destination.ip",
"destination.port",
"source.bytes",
"process.code_signature.subject_name",
"process.code_signature.trusted"
"process.code_signature.trusted",
"network.transport",
"process.executable",
"process.parent.name"
]
},
"data_description": {
Expand Down Expand Up @@ -222,15 +228,16 @@
"host.id",
"user.name",
"event.module",
"user.id",
"process.name",
"source.ip",
"destination.ip",
"destination.geo.city_name",
"destination.geo.region_name",
"source.bytes",
"process.code_signature.subject_name",
"process.code_signature.trusted"
"process.code_signature.trusted",
"process.executable",
"process.parent.name"
]
},
"data_description": {
Expand Down Expand Up @@ -279,7 +286,11 @@
"process.name",
"file.size",
"process.code_signature.subject_name",
"process.code_signature.trusted"
"process.code_signature.trusted",
"file.extension",
"process.executable",
"process.parent.name",
"file.hash.sha256"
]
},
"data_description": {
Expand Down Expand Up @@ -377,7 +388,11 @@
"process.name",
"file.size",
"process.code_signature.subject_name",
"process.code_signature.trusted"
"process.code_signature.trusted",
"file.extension",
"process.executable",
"process.parent.name",
"file.hash.sha256"
]
},
"data_description": {
Expand Down
2 changes: 1 addition & 1 deletion packages/ded/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.0
name: ded
title: "Data Exfiltration Detection"
version: 3.3.0
version: 3.4.0
source:
license: "Elastic-2.0"
description: "ML package to detect data exfiltration in your network and file data."
Expand Down
5 changes: 5 additions & 0 deletions packages/dga/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "3.2.0"
changes:
- description: Add runtime_mappings for process signature influencers on the ML datafeed.
type: enhancement
link: https://github.com/elastic/integrations/pull/20335
- version: "3.1.0"
changes:
- description: Add process signature fields as influencers to ML jobs.
Expand Down
18 changes: 16 additions & 2 deletions packages/dga/kibana/ml_module/dga-ml.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@
"host.name",
"host.id",
"process.name",
"process.code_signature.subject_name",
"process.code_signature.trusted"
"process_code_signature_subject_name",
"process_code_signature_trusted"
]
},
"data_description": {
Expand Down Expand Up @@ -113,6 +113,20 @@
}
]
}
},
"runtime_mappings": {
"process_code_signature_subject_name": {
"type": "keyword",
"script": {
"source": "if (doc.containsKey('process.code_signature.subject_name') && doc['process.code_signature.subject_name'].size() != 0) { emit(doc['process.code_signature.subject_name'].value); } else { emit('None'); }"
}
},
"process_code_signature_trusted": {
"type": "keyword",
"script": {
"source": "if (doc.containsKey('process.code_signature.trusted') && doc['process.code_signature.trusted'].size() != 0) { emit(doc['process.code_signature.trusted'].value.toString()); } else { emit('None'); }"
}
}
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion packages/dga/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.4
name: dga
title: "Domain Generation Algorithm Detection"
version: 3.1.0
version: 3.2.0
source:
license: "Elastic-2.0"
description: "ML solution package to detect domain generation algorithm (DGA) activity in your network data."
Expand Down
5 changes: 5 additions & 0 deletions packages/lmd/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "3.4.0"
changes:
- description: Add file and session influencers to ML jobs and restrict RDP transform source to Windows hosts.
type: enhancement
link: https://github.com/elastic/integrations/pull/20335
- version: "3.3.0"
changes:
- description: Update transform source index to scope to the relevant data streams.
Expand Down
5 changes: 4 additions & 1 deletion packages/lmd/elasticsearch/transform/pivot_transform_ea/transform.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ source:
- term:
'process.Ext.session_info.logon_type':
value: RemoteInteractive
- term:
'host.os.type':
value: windows
runtime_mappings:
'destination.ip':
type: ip
Expand Down Expand Up @@ -86,5 +89,5 @@ sync:
delay: 60s
field: '@timestamp'
_meta:
fleet_transform_version: 3.3.0
fleet_transform_version: 3.4.0
run_as_kibana_system: false
32 changes: 25 additions & 7 deletions packages/lmd/kibana/ml_module/lmd-ml.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,11 @@
"event.action",
"process.name",
"process.code_signature.subject_name",
"process.code_signature.trusted"
"process.code_signature.trusted",
"file.path",
"file.extension",
"file.name",
"file.hash.sha256"
]
},
"data_description": {
Expand Down Expand Up @@ -113,7 +117,11 @@
"process.name",
"file.size",
"process.code_signature.subject_name",
"process.code_signature.trusted"
"process.code_signature.trusted",
"file.path",
"file.extension",
"file.name",
"file.hash.sha256"
]
},
"data_description": {
Expand Down Expand Up @@ -163,7 +171,9 @@
"file.name",
"process.name",
"process.code_signature.subject_name",
"process.code_signature.trusted"
"process.code_signature.trusted",
"file.path",
"file.hash.sha256"
]
},
"data_description": {
Expand Down Expand Up @@ -211,7 +221,10 @@
"file.path",
"process.name",
"process.code_signature.subject_name",
"process.code_signature.trusted"
"process.code_signature.trusted",
"file.extension",
"file.name",
"file.hash.sha256"
]
},
"data_description": {
Expand Down Expand Up @@ -264,7 +277,9 @@
"user.id",
"source.ip",
"destination.ip",
"session.duration"
"session.duration",
"number_processes_per_session",
"total_length_process_args"
]
},
"data_description": {
Expand Down Expand Up @@ -318,7 +333,8 @@
"user.id",
"source.ip",
"destination.ip",
"session.duration"
"session.duration",
"number_processes_per_session"
]
},
"data_description": {
Expand Down Expand Up @@ -417,7 +433,9 @@
"event.module",
"user.id",
"destination.ip",
"source.ip"
"source.ip",
"session.duration",
"total_length_process_args"
]
},
"data_description": {
Expand Down
2 changes: 1 addition & 1 deletion packages/lmd/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.0
name: lmd
title: "Lateral Movement Detection"
version: 3.3.0
version: 3.4.0
source:
license: "Elastic-2.0"
description: "ML package to detect lateral movement based on file transfer activity and Windows RDP events."
Expand Down
5 changes: 5 additions & 0 deletions packages/pad/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
- version: "2.4.0"
changes:
- description: Update Windows privilege transform source index to `logs-system.security-*` only.
type: enhancement
link: https://github.com/elastic/integrations/pull/20335
- version: "2.3.0"
changes:
- description: Update transform source index to scope to the relevant data streams.
Expand Down
4 changes: 2 additions & 2 deletions packages/pad/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,9 +87,9 @@ To inspect the installed assets, you can navigate to **Stack Management > Data >
| Transform name | Purpose | Source index | Destination index | Alias | Supported Platform |
|-------------------------------------------|-------------------------------------------------------------------|--------------|-------------------------------------------------|-------------------------------------------|--------------------|
| pad.pivot_transform_okta_sessions_ea | Collects user session information for Okta events | `logs-okta.system-*` | ml_okta_multiple_user_sessions_pad_ea-[version] | ml_okta_multiple_user_sessions_pad_ea.all | Okta |
| pad.pivot_transform_win_privilege_list_ea | Collects special privileges assigned to a user for Windows events | `logs-system.security-*,logs-windows.*` | ml_windows_privilege_type_pad_ea-[version] | ml_windows_privilege_type_pad_ea.all | Windows |
| pad.pivot_transform_win_privilege_list_ea | Collects special privileges assigned to a user for Windows events | `logs-system.security-*` | ml_windows_privilege_type_pad_ea-[version] | ml_windows_privilege_type_pad_ea.all | Windows |
**Source index patterns**: `logs-okta.system-*` holds Okta system logs collected by the [Okta](https://www.elastic.co/docs/reference/integrations/okta) integration, `logs-system.security-*` holds Windows security event logs collected by the [System](https://www.elastic.co/docs/reference/integrations/system) integration, and `logs-windows.*` holds Windows event logs collected by the [Windows](https://www.elastic.co/docs/reference/integrations/windows) integration.
**Source index patterns**: `logs-okta.system-*` holds Okta system logs collected by the [Okta](https://www.elastic.co/docs/reference/integrations/okta) integration, and `logs-system.security-*` holds Windows security event logs collected by the [System](https://www.elastic.co/docs/reference/integrations/system) integration.
When querying the destination indices for Okta and Windows logs, we advise using the alias for the destination index (`ml_okta_multiple_user_sessions_pad_ea.all` and `ml_windows_privilege_type_pad_ea.all`). In the event that the underlying package is upgraded, the alias will aid in maintaining the previous findings.
Expand Down
Loading
Loading