Concurrent config applies cause UpdateOverview to never be sent

[valyria257]

Bug Overview

When two identical ConfigApplyRequest messages are delivered to the agent close together, the UpdateOverview call that should be sent by the FileManagerService after the first apply is dropped, preventing the management plane from learning about changes.

Expected Behavior

If changes are made to the nginx config context, I expect an UpdateOverview call to be sent to the management plane.

Steps to Reproduce the Bug

1. Trigger two identical ConfigApplyRequest messages to be delivered to the agent in rapid succession
2. Observe in nginx-agent logs: the first ConfigApplyRequest applies successfully and NGINX reloads, but no "Sending file overview update due to NGINX configuration updates" log appears after it completes. The second ConfigApplyRequest applies the same config with the same result — no UpdateOverview is sent by either.

Environment Details

• Version of this project or specific commit: 3.11.2

Additional Context

Sequence diagram of observed bug:

sequenceDiagram
    participant DPC as dataplane-ctrl
    participant NP as NginxPlugin
    participant WP as WatcherPlugin

    DPC->>NP: ConfigApplyRequest (A)
    NP->>WP: Add instanceID to instancesWithConfigApplyInProgress
    NP->>NP: Download external file (cert1), write to disk
    NP->>NP: Reload NGINX
    NP->>DPC: "Config apply successful" (A)
    NP-->>WP: EnableWatchersTopic{newContext} [async bus message]
    NP->>WP: Remove instanceID from instancesWithConfigApplyInProgress

    DPC->>NP: ConfigApplyRequest (B - same files as A)
    NP->>WP: Add instanceID to instancesWithConfigApplyInProgress

    Note over WP: handleEnableWatchers goroutine<br/>dequeues Config Apply A's EnableWatchersTopic

    WP->>WP: slices.Contains(instancesWithConfigApplyInProgress, instanceID)?
    Note over WP: TRUE
    WP->>WP: ⚠️ "Not sending updated NGINX config context<br/>since config apply is in progress"
    Note over WP: NginxConfigUpdateTopic suppressed<br/>UpdateOverview never fired ❌

    NP->>NP: Download external file (cert1), write to disk
    NP->>NP: Reload NGINX
    NP->>DPC: "Config apply successful" (B)
    NP-->>WP: EnableWatchersTopic{sameContext} [async bus message]
    NP->>WP: Remove instanceID from instancesWithConfigApplyInProgress

    Note over WP: handleEnableWatchers goroutine<br/>dequeues Config Apply B's EnableWatchersTopic

    WP->>WP: slices.Contains(instancesWithConfigApplyInProgress, instanceID)?
    Note over WP: FALSE
    WP->>WP: cache.Equal(newContext)?
    Note over WP: TRUE — cert unchanged from Config Apply A
    WP->>WP: ⚠️ NginxConfigUpdateTopic suppressed<br/>No cache change to report ❌

Loading

Logs:

1786981770059	2026-08-17T15:49:30.059Z	time=2026-08-17T15:49:30.059Z level=INFO msg="Found NGINX Plus API" url=http://localhost:49151/api correlation_id=3bbcd45e-9a53-11f1-af17-8912484f7359
1786981770029	2026-08-17T15:49:30.029Z	time=2026-08-17T15:49:30.029Z level=INFO msg="Found NGINX Plus API" url=http://localhost:49151/api correlation_id=3bbb0c0c-9a53-11f1-af17-8912484f7359
1786981766274	2026-08-17T15:49:26.274Z	time=2026-08-17T15:49:26.274Z level=INFO msg="Sending data plane response message" message="Config apply successful" status=COMMAND_STATUS_OK error="" correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981766273	2026-08-17T15:49:26.273Z	time=2026-08-17T15:49:26.273Z level=INFO msg="Found NGINX Plus API" url=http://localhost:49151/api correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981766272	2026-08-17T15:49:26.272Z	time=2026-08-17T15:49:26.272Z level=INFO msg="Finished monitoring NGINX error logs after reload" correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981766272	2026-08-17T15:49:26.272Z	time=2026-08-17T15:49:26.272Z level=INFO msg="No errors found in NGINX error logs after reload" correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756315	2026-08-17T15:49:16.315Z	time=2026-08-17T15:49:16.315Z level=INFO msg="Finished reloading NGINX master process" process_id=773 correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756315	2026-08-17T15:49:16.315Z	time=2026-08-17T15:49:16.315Z level=INFO msg="NGINX workers have been reloaded" correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756285	2026-08-17T15:49:16.285Z	time=2026-08-17T15:49:16.285Z level=INFO msg="Waiting for NGINX worker processes to be reloaded" correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756274	2026-08-17T15:49:16.274Z	time=2026-08-17T15:49:16.271Z level=INFO msg="Starting to monitor NGINX error log for errors" log_file=/var/log/nginx/error.log correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756274	2026-08-17T15:49:16.274Z	time=2026-08-17T15:49:16.271Z level=INFO msg="Starting to monitor NGINX error log for errors" log_file=/var/log/nginx/stream-error.log correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756260	2026-08-17T15:49:16.260Z	time=2026-08-17T15:49:16.260Z level=INFO msg="NGINX configuration tested" output="nginx: the configuration file /etc/nginx/nginx.conf syntax is ok\nnginx: configuration file /etc/nginx/nginx.conf test is successful\n" correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756260	2026-08-17T15:49:16.260Z	time=2026-08-17T15:49:16.260Z level=INFO msg="Reloading NGINX master process" pid=773 correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756241	2026-08-17T15:49:16.241Z	time=2026-08-17T15:49:16.241Z level=INFO msg="Validating NGINX configuration" correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756241	2026-08-17T15:49:16.241Z	time=2026-08-17T15:49:16.241Z level=INFO msg="Found NGINX Plus API" url=http://localhost:49151/api correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756239	2026-08-17T15:49:16.239Z	time=2026-08-17T15:49:16.239Z level=INFO msg="Renaming file /etc/nginx/.secret-cert.crt.agent.tmp to /etc/nginx/secret-cert.crt" correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756230	2026-08-17T15:49:16.230Z	time=2026-08-17T15:49:16.229Z level=INFO msg="Successfully downloaded file content" file_name=/etc/nginx/secret-cert.crt size=973 correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756230	2026-08-17T15:49:16.230Z	time=2026-08-17T15:49:16.230Z level=INFO msg="Successfully downloaded external file" event_tag=ID-1310 hash="UCGKXib8HG/mzYp45mqYv7Bn5+5Z9jpbGqHx3EoTOxw=" correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756187	2026-08-17T15:49:16.187Z	time=2026-08-17T15:49:16.186Z level=INFO msg="Downloading external file" correlation_id=185fa0c6-c513-499c-bb27-941b1942d3f1 server_type=command
1786981756186	2026-08-17T15:49:16.186Z	time=2026-08-17T15:49:16.184Z level=INFO msg="Sending data plane response message" message="Config apply successful" status=COMMAND_STATUS_OK error="" correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981756186	2026-08-17T15:49:16.186Z	time=2026-08-17T15:49:16.185Z level=INFO msg="Received management plane config apply request" server_type=command
1786981756184	2026-08-17T15:49:16.184Z	time=2026-08-17T15:49:16.183Z level=INFO msg="Found NGINX Plus API" url=http://localhost:49151/api correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981756182	2026-08-17T15:49:16.182Z	time=2026-08-17T15:49:16.182Z level=INFO msg="Finished monitoring NGINX error logs after reload" correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981756182	2026-08-17T15:49:16.182Z	time=2026-08-17T15:49:16.182Z level=INFO msg="No errors found in NGINX error logs after reload" correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746828	2026-08-17T15:49:06.828Z	time=2026-08-17T15:49:06.828Z level=INFO msg="Finished reloading NGINX master process" process_id=773 correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746828	2026-08-17T15:49:06.828Z	time=2026-08-17T15:49:06.828Z level=INFO msg="NGINX workers have been reloaded" correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746193	2026-08-17T15:49:06.193Z	time=2026-08-17T15:49:06.193Z level=INFO msg="Waiting for NGINX worker processes to be reloaded" correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746182	2026-08-17T15:49:06.182Z	time=2026-08-17T15:49:06.182Z level=INFO msg="Starting to monitor NGINX error log for errors" log_file=/var/log/nginx/error.log correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746182	2026-08-17T15:49:06.182Z	time=2026-08-17T15:49:06.182Z level=INFO msg="Starting to monitor NGINX error log for errors" log_file=/var/log/nginx/stream-error.log correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746168	2026-08-17T15:49:06.168Z	time=2026-08-17T15:49:06.168Z level=INFO msg="NGINX configuration tested" output="nginx: the configuration file /etc/nginx/nginx.conf syntax is ok\nnginx: configuration file /etc/nginx/nginx.conf test is successful\n" correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746168	2026-08-17T15:49:06.168Z	time=2026-08-17T15:49:06.168Z level=INFO msg="Reloading NGINX master process" pid=773 correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746135	2026-08-17T15:49:06.135Z	time=2026-08-17T15:49:06.135Z level=INFO msg="Validating NGINX configuration" correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746134	2026-08-17T15:49:06.134Z	time=2026-08-17T15:49:06.134Z level=INFO msg="Found NGINX Plus API" url=http://localhost:49151/api correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746130	2026-08-17T15:49:06.130Z	time=2026-08-17T15:49:06.130Z level=INFO msg="Renaming file /etc/nginx/.secret-cert.crt.agent.tmp to /etc/nginx/secret-cert.crt" correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746129	2026-08-17T15:49:06.129Z	time=2026-08-17T15:49:06.129Z level=INFO msg="Successfully downloaded external file" event_tag=ID-1310 hash="UCGKXib8HG/mzYp45mqYv7Bn5+5Z9jpbGqHx3EoTOxw=" correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746129	2026-08-17T15:49:06.129Z	time=2026-08-17T15:49:06.129Z level=INFO msg="Successfully downloaded file content" file_name=/etc/nginx/secret-cert.crt size=973 correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746051	2026-08-17T15:49:06.051Z	time=2026-08-17T15:49:06.049Z level=INFO msg="Downloading external file" correlation_id=e98dd36c-3846-4a4b-ba4f-56a86ef4ed14 server_type=command
1786981746046	2026-08-17T15:49:06.046Z	time=2026-08-17T15:49:06.046Z level=INFO msg="Received management plane config apply request" server_type=command
1786981741248	2026-08-17T15:49:01.248Z	time=2026-08-17T15:49:01.247Z level=INFO msg="Connection created" response=response:{status:COMMAND_STATUS_OK} correlation_id=a9d57be1-9830-11f1-af17-8912484f7359 server_type=command
1786981741248	2026-08-17T15:49:01.248Z	time=2026-08-17T15:49:01.248Z level=INFO msg="Agent connected" correlation_id=a9d57be1-9830-11f1-af17-8912484f7359 server_type=command
1786981741099	2026-08-17T15:49:01.099Z	time=2026-08-17T15:49:01.097Z level=ERROR msg="Failed to receive message from subscribe stream. Trying create connection rpc again" error="rpc error: code = Canceled desc = terminated signal received" correlation_id=a9d57be1-9830-11f1-af17-8912484f7359 server_type=command

[valyria257]

Proposed Solution

There is a guard here that suppresses a message from being sent on NginxConfigUpdateTopic if there is a pending config apply. This ensures that stale NginxConfigContext does not interfere with the current config apply. However, if a second identical ConfigApplyRequest arrives before the first has sent out the NginxConfigUpdateTopic message, it prevents an UpdateOverview from ever being sent.

To fix, instead of ignoring the NginxConfigContext, we will buffer and flush it once the config apply completes. We will only send to the NginxConfigUpdateTopic if the previous pending NginxConfigContext matches the current context to ensure there is no stale, incorrect context.

Before

sequenceDiagram
    participant DPC as dataplane-ctrl
    participant NP as NginxPlugin
    participant WP as WatcherPlugin
    participant FMS as FileManagerService

    DPC->>NP: ConfigApplyRequest A
    NP->>WP: Add instance to inProgress[]
    NP->>NP: Write files, reload NGINX
    NP->>DPC: Config apply successful
    NP-->>WP: EnableWatchersTopic{contextA} [async]
    NP->>WP: Remove instance from inProgress[]

    DPC->>NP: ConfigApplyRequest B (identical)
    NP->>WP: Add instance to inProgress[]

    WP->>WP: handleEnableWatchers (A's message)
    WP->>WP: inProgress[].contains(instance)? TRUE
    WP->>WP: ❌ Drop NginxConfigUpdateTopic

    NP->>NP: Write files, reload NGINX
    NP->>DPC: Config apply successful
    NP-->>WP: EnableWatchersTopic{contextA} [async]
    NP->>WP: Remove instance from inProgress[]

    WP->>WP: handleEnableWatchers (B's message)
    WP->>WP: inProgress[].contains(instance)? FALSE
    WP->>WP: cache.Equal(contextA)? TRUE
    WP->>WP: ❌ Drop NginxConfigUpdateTopic

    note over FMS: UpdateOverview never sent

Loading

After

sequenceDiagram
    participant DPC as dataplane-ctrl
    participant NP as NginxPlugin
    participant WP as WatcherPlugin
    participant FMS as FileManagerService

    DPC->>NP: ConfigApplyRequest A
    NP->>WP: Add instance to inProgress[]
    NP->>NP: Write files, reload NGINX
    NP->>DPC: Config apply successful
    NP-->>WP: EnableWatchersTopic{contextA} [async]
    NP->>WP: Remove instance from inProgress[]

    DPC->>NP: ConfigApplyRequest B (identical)
    NP->>WP: Add instance to inProgress[]

    WP->>WP: handleEnableWatchers (A's message)
    WP->>WP: inProgress[].contains(instance)? TRUE
    WP->>WP: ✅ Buffer contextA

    NP->>NP: Write files, reload NGINX
    NP->>DPC: Config apply successful
    NP-->>WP: EnableWatchersTopic{contextA} [async]
    NP->>WP: Remove instance from inProgress[]
    WP->>WP: Buffered context exists?  TRUE
    WP->>WP: buffered == completed context? TRUE
    WP->>WP: ✅ Flush — send NginxConfigUpdateTopic

    WP->>FMS: NginxConfigUpdateTopic{contextA}
    FMS->>FMS: Sending file overview update...
    FMS->>DPC: UpdateOverview ✅

Loading

[pooyanazad]

Hi @valyria257 , thanks for the detailed report and logs. I’d like to work on this.

From my initial review, it looks like the issue may be related to the ordering between the first config apply completing and the watcher processing its config-context update. A second apply may mark the same instance as in progress before the first update is handled, causing that update to be dropped.

I haven’t fully reproduced this yet, so I’d like to validate the behavior with a regression test and investigate the flow more carefully before deciding on the final fix. I’ll do my best to address it cleanly without changing the expected config-apply behavior.

Please let me know if someone is already working on this or if you have a preferred direction.