feat(group-ib): migrate connector to manager-supported mode (#7282) - #7330
Open
Hugo Dupras (jabesq) wants to merge 9 commits into
Open
feat(group-ib): migrate connector to manager-supported mode (#7282)#7330Hugo Dupras (jabesq) wants to merge 9 commits into
Hugo Dupras (jabesq) wants to merge 9 commits into
Conversation
…on warnings (#7282) The manager-supported migration flattened the deeply-nested ti_api config into single-underscore fields, which renamed every TI_API__* env var. Add a backward-compatibility shim on GroupIBTIApiConfig that maps legacy double-underscore variables to the new fields and emits a DeprecationWarning, so existing deployments keep working. The canonical variable wins when both are set. Covered by unit tests.
Replace the eager _expose_settings_as_attributes/_unwrap shim, which generated flat attributes for every settings field (including unused opencti_* and connector id/name/scope/log_level), with explicit properties for the values actually consumed and helpers that read the nested settings directly. Removes dead attribute generation while preserving the historical ConfigConnector interface (secret unwrapping, ISO-8601 duration, and None-for-unknown collection/extra settings).
…nings (#7282) The manager-supported migration flattened the nested ti_api config, which also broke deployments still using a nested config.yml (proxy, extra_settings and slash-keyed collections sub-sections were silently ignored). Extend the GroupIBTIApiConfig migration validator to flatten those nested sub-sections onto the canonical fields (in addition to the legacy double-underscore env vars), emitting a DeprecationWarning per section. The flattened (canonical) value takes precedence when both forms are provided. Covered by unit tests.
Hugo Dupras (jabesq)
requested review from
Pauline Eustachy (Powlinett) and
Maxime (maximerafaillac)
August 20, 2026 09:54
|
🔴 Connector Linter errors detected
|
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request migrates the external-import/group-ib connector to manager-supported mode (deployable via XTM Composer) by moving configuration loading/validation to connectors-sdk + Pydantic settings, while preserving the connector’s existing runtime structure and behavior.
Changes:
- Introduces a validated
ConnectorSettingsmodel (with backward-compatible migration for legacyti_apienv vars and nestedconfig.ymlshapes) and wiresOpenCTIConnectorHelperto the generated helper config. - Updates runtime/config artifacts for manager support: manifest flag, generated config JSON schema + markdown docs, and flattened
.env.sample/config.yml.sample. - Adds unit tests covering settings validation, connector wiring, and legacy-config migration paths.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| external-import/group-ib/src/settings.py | Adds Pydantic/connectors-sdk settings models, including legacy config flattening and validation. |
| external-import/group-ib/src/config.py | Refactors configuration access to come from ConnectorSettings and exposes compatibility properties/helpers. |
| external-import/group-ib/src/lib/external_import.py | Instantiates OpenCTIConnectorHelper using the settings-derived helper config. |
| external-import/group-ib/src/requirements.txt | Adds runtime dependencies for settings/manager-supported mode (Pydantic + connectors-sdk). |
| external-import/group-ib/src/init.py | Re-exports ConnectorSettings for schema generation tooling. |
| external-import/group-ib/metadata/connector_manifest.json | Sets manager_supported: true. |
| external-import/group-ib/metadata/connector_config_schema.json | Adds generated manager-supported config schema. |
| external-import/group-ib/metadata/CONNECTOR_CONFIG_DOC.md | Adds generated configuration documentation for Composer/users. |
| external-import/group-ib/.env.sample | Updates env var layout to flattened single-underscore variables with new defaults. |
| external-import/group-ib/src/config.yml.sample | Updates sample YAML to the flattened ti_api field structure. |
| external-import/group-ib/README.md | Points users to the generated configuration documentation. |
| external-import/group-ib/tests/conftest.py | Adds test-time import path wiring for src/. |
| external-import/group-ib/tests/test_main.py | Adds wiring/smoke tests for settings -> helper -> connector initialization. |
| external-import/group-ib/tests/tests_connector/test_settings.py | Adds settings validation + legacy migration unit tests. |
| external-import/group-ib/tests/test-requirements.txt | Adds pytest + references main connector requirements for tests. |
Suppressed comments (1)
external-import/group-ib/src/settings.py:138
- This warning uses
DeprecationWarning, which is ignored by default, so operators may never see the env-var migration guidance. Consider usingUserWarning(or logging) so the deprecation message is visible by default.
warnings.warn(
f"Environment variable 'TI_API_{legacy_key.upper()}' is deprecated and "
f"will be removed in a future release; use "
f"'TI_API_{canonical_key.upper()}' instead.",
DeprecationWarning,
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+105
to
+109
| warnings.warn( | ||
| f"Deprecated nested 'ti_api.{section}' configuration detected; " | ||
| f"use the flattened '{section}_*' settings instead.", | ||
| DeprecationWarning, | ||
| stacklevel=2, |
Helene Nguyen (helene-nguyen)
self-requested a review
August 20, 2026 12:19
Helene Nguyen (helene-nguyen)
approved these changes
Aug 21, 2026
Helene Nguyen (helene-nguyen)
left a comment
Helene Nguyen (helene-nguyen)
left a comment
Member
There was a problem hiding this comment.
Hugo Dupras (@jabesq) Thanks for this huge migration
Tested on my side with both configuration files .env and config.yml
- Each config is read properly when enable it
- Thanks for adding tests
Worth to know
- Following our discussion, the
config.ymlis now read properly - Config is now loaded and validated once at startup, and removing the
os.environscan andyaml.loadis a net good reduction
LGTM :)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
ConfigConnector,CustomConnector,ExternalImportConnector), scheduling loop, and logging are unchanged.ConnectorSettingsmodel (src/settings.py) exposed viato_helper_config(), replacing the legacyyaml.load+pycti.get_config_variablepattern.manager_supported: truein__metadata__/connector_manifest.json(while preservingverified: true).__metadata__/connector_config_schema.jsonand__metadata__/CONNECTOR_CONFIG_DOC.md, and regenerate.env.sample/src/config.yml.sampleto the flat single-underscore variable structure.ti_apiconfig into single-underscore fields (the connectors-sdk settings loader only resolves two-levelSECTION_FIELDenv vars).GroupIBTIApiConfig(model_validator(mode="before")) that migrates legacy double-underscore env vars (e.g.TI_API__PROXY__IP,TI_API__COLLECTIONS__APT_THREAT__ENABLE) to the flattened fields, emitting aDeprecationWarning.config.ymlsub-sections (proxy,extra_settings, and slash-keyedcollectionslikeapt/threat) onto the same fields, also emitting aDeprecationWarning; the new flattened value wins when both forms are present.ConfigConnector: replace the eager attribute-generation shim with explicit@propertyaccessors for the values actually consumed, and read the collection/extra-settings helpers directly from the settings model (preservingNone-for-unknown behavior thecyberintegrationslib relies on).pydanticandconnectors-sdktosrc/requirements.txtand asrc/__init__.pyre-exportingConnectorSettings.tests/test_main.py,tests/tests_connector/test_settings.py, plustests/conftest.pyandtests/test-requirements.txt).Related issues
Checklist
Further comments
Backward compatibility. Env-var and nested-config renames are backward-compatible via two deprecation shims in
GroupIBTIApiConfig(warn only), so no forced config migration is required. Legacy double-underscore env vars (TI_API__PROXY__IP,TI_API__COLLECTIONS__APT_THREAT__ENABLE, …) and legacy nestedconfig.ymlsub-sections (proxy,extra_settings, slash-keyedcollectionssuch asapt/threat) are automatically flattened onto the new single-underscore fields, each emitting aDeprecationWarning. When both the old and new forms are present, the new flattened value wins. This was verified end-to-end by loading the full old nestedconfig.yml.Breaking changes / behavior notes:
log_levelis now validated against lowercase{debug, info, warn, warning, error}; non-standard casing/values that pycti previously tolerated will now fail validation.update_existing_datanow defaults totruewhen unset (the old effective default was disabled), so deployments that omitted it will start updating existing entities. Both sample files now set it explicitly.OPENCTI_URL/OPENCTI_TOKEN,TI_API_USERNAME/TI_API_TOKEN) now fail fast at startup if entirely absent.CONNECTOR_CONFIDENCE_LEVEL,CONNECTOR_MQ_*, andCONNECTOR_DOCKER_CONTAINER_NAME(pycti sources MQ settings from platform registration).