feat: Make the task handler log level configurable

Closes #650.

The task handler is what feeds the log view in the Airflow web UI. Its
level was pinned to INFO with no way to change it, so asking for DEBUG
logs meant getting them in the files and the Vector aggregator but not in
the UI, and there was no way to quiet the UI either.

Render the level from the new task appender in the CRD. It defaults to
INFO, which is what Airflow itself defaults to, so nothing changes for
existing deployments.

A task level below INFO also has to open up the logger it is wired to,
otherwise the logger drops the records before the handler can filter
them. A level above INFO must not raise the logger, since other handlers
still want those records, so the logger level is the lower of the two.

Both the stdlib template used by 2.x and 3.0.x and the structlog template
used by 3.1 and later are covered.

Author: Churi12

CHANGELOG.md

@@ -2,6 +2,14 @@
 
 ## [Unreleased]
 
+### Added
+
+- The log level of the `task` handler, which is what the Airflow UI displays, is now
+  configurable through the `task` appender in the CRD, alongside `console` and `file`.
+  It defaults to `INFO`, matching Airflow's own default, so existing deployments are
+  unaffected. This makes it possible to send DEBUG to the log files and the Vector
+  aggregator while showing only INFO in the UI ([#829]).
+
 ### Changed
 
 - Internal operator refactoring: introduce a build() step in the reconciler that
@@ -13,6 +21,7 @@
 [#814]: https://github.com/stackabletech/airflow-operator/pull/814
 [#821]: https://github.com/stackabletech/airflow-operator/pull/821
 [#827]: https://github.com/stackabletech/airflow-operator/pull/827
+[#829]: https://github.com/stackabletech/airflow-operator/pull/829
 
 ## [26.7.0] - 2026-07-21
 

Cargo.lock

@@ -34,4 +34,6 @@ tracing = "0.1"
 
 [patch."https://github.com/stackabletech/operator-rs.git"]
 # stackable-operator = { git = "https://github.com/stackabletech//operator-rs.git", branch = "main" }
-# stackable-operator = { path = "../operator-rs/crates/stackable-operator" }
+# TODO: Point back at a released tag once the task appender is released.
+# Depends on https://github.com/stackabletech/operator-rs/pull/1255
+stackable-operator = { git = "https://github.com/Churi12/operator-rs.git", branch = "feat/task-appender-log-config" }

Cargo.toml

@@ -4,9 +4,8 @@
 The logs can be forwarded to a Vector log aggregator by providing a discovery ConfigMap for the aggregator and by enabling the log agent:
 
 NOTE: The `task` handler is responsible for showing the task logs in the UI.
-Unfortunately, the log level of the `task` handler cannot be specified.
-To avoid that all logs are emitted to the UI, the log level of the `airflow.task` logger is set explicitly to `INFO`.
-You can change the log level as shown below.
+Its log level is configured with the `task` appender, alongside `console` and `file`, and defaults to `INFO` as it does in Airflow itself.
+Because it is a separate appender, the UI can be given a different log level than the log files and the Vector aggregator, as shown below.
 
 [source,yaml]
 ----
@@ -29,11 +28,14 @@ spec:
         enableVectorAgent: true
         containers:
           airflow:
+            # Write DEBUG to the log files and the Vector aggregator, but only show INFO in the UI.
+            file:
+              level: DEBUG
+            task:
+              level: INFO
             loggers:
               "airflow.processor":
                 level: INFO
-              "airflow.task":
-                level: DEBUG
   schedulers:
     config:
       logging: