[m365_defender] Fix alert and incident pagination

[kcreddy]

Proposed commit message

[m365_defender] Fix alert and incident pagination

The `alert` and `incident` data streams paginated by incrementing `$skip`
by `batch_size` on each page while carrying the original `$filter`. Under
high volume this stalled catch-up: past a certain `$skip` offset the Graph
security API stopped returning further pages, so a backlog was never fully
drained within a single interval (reported in #20234).

Switch both streams to time-boundary pagination: hold `$skip` at 0 and, on a
full page, advance `$filter` to `lastUpdateDateTime gt <max lastUpdateDateTime
in the page>` (the last element, since `$orderby` is ascending). A non-full
page renders an empty filter value, which stops pagination. The stored cursor
is unchanged, so restart behaviour is preserved.

Also correct the `alert` `batch_size` default from 2000 to 1000 and clarify the
`batch_size` field descriptions.

Update the alert and incident system tests (and their mock configs) to
exercise multi-page pagination driven by the `$filter` boundary.

Reference: 
- https://github.com/microsoftgraph/microsoft-graph-docs-contrib/blob/main/api-reference/beta/resources/security-error-codes.md#constraints

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

System tests pass successfully (with updated pagination)

--- Test results for package: m365_defender - START ---
╭───────────────┬─────────────┬───────────┬──────────────┬────────┬───────────────╮
│ PACKAGE       │ DATA STREAM │ TEST TYPE │ TEST NAME    │ RESULT │  TIME ELAPSED │
├───────────────┼─────────────┼───────────┼──────────────┼────────┼───────────────┤
│ m365_defender │ alert       │ system    │ default      │ PASS   │ 37.080671208s │
│ m365_defender │ alert       │ system    │ oauth-params │ PASS   │    44.789865s │
│ m365_defender │ incident    │ system    │ default      │ PASS   │ 37.706307792s │
│ m365_defender │ incident    │ system    │ no-alerts    │ PASS   │ 41.069887291s │
│ m365_defender │ incident    │ system    │ oauth-params │ PASS   │ 41.240890125s │
╰───────────────┴─────────────┴───────────┴──────────────┴────────┴───────────────╯
--- Test results for package: m365_defender - END   ---
Done

Related issues

• Closes m365_defender: alert/incident pagination is fragile under high volume #20234

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package m365_defender 👍(3) 💚(1) 💔(4)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
alert	973.71	739.92	-233.79 (-24.01%)	💔
incident	1236.09	752.05	-484.04 (-39.16%)	💔
vulnerability	11363.64	6939.63	-4424.01 (-38.93%)	💔
event	621.89	524.66	-97.23 (-15.63%)	💔

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 22065f0

History

• 💔 Build #46606 failed 8d7e34a
• 💔 Build #46601 failed 2a8cf7d

cc @kcreddy

[ShourieG]

@vera-review-bot review

[vera-review-bot]

⚠️ @kcreddy I couldn't complete the review for commits 22065f0 after several attempts. A maintainer can re-run it with @vera-review-bot review.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request