docs: add missing 3.0.0 feature documentation

docs/Binary-distributions.md

@@ -0,0 +1,60 @@
+---
+title: Binary Distributions
+layout: documentation
+documentation: true
+---
+
+Storm 3.0.0 ships two binary distributions. Both contain identical core functionality; they differ only in which optional plugins are bundled.
+
+| Distribution | Filename | Approx. size |
+|---|---|---|
+| **Full** | `apache-storm-<version>.tar.gz` | ~395 MB of jars |
+| **Lite** | `apache-storm-<version>-lite.tar.gz` | ~207 MB of jars (~47% smaller) |
+
+## What the lite distribution omits
+
+### storm-autocreds (−79 MB)
+
+`storm-autocreds` provides Nimbus and Supervisor with the ability to populate and renew HDFS and HBase delegation tokens on **Kerberos-secured clusters**. It pulls in the full Hadoop and HBase client trees. The plugin is off by default and is only needed on secure Hadoop deployments.
+
+In the lite distribution, only the `external/storm-autocreds/README` ships. To install the plugin on demand, run:
+
+```
+bin/storm-autocreds-fetch
+```
+
+This uses Maven to resolve `org.apache.storm:storm-autocreds` and its runtime dependencies from Maven Central and copies them into `extlib-daemon/`. Restart Nimbus and Supervisor afterwards.
+
+Pass `--version` and `--dest` to override the detected Storm version or target directory. Pass `--` to forward arguments to Maven (e.g. to use an internal mirror or an offline local repository):
+
+```
+bin/storm-autocreds-fetch -- -s /path/to/settings.xml
+bin/storm-autocreds-fetch -- -Dmaven.repo.local=/path/to/offline-repo -o
+```
+
+### storm-kafka-monitor (−38 MB)
+
+`storm-kafka-monitor` displays Kafka spout consumer lag in the Storm UI and powers the `bin/storm-kafka-monitor` command. It pulls in the Kafka client library and is only needed when running Kafka spouts and wanting lag metrics. When it is absent, the UI degrades gracefully: the lag column shows an actionable message rather than failing, and the `bin/storm-kafka-monitor` wrapper prints a hint to run the fetch command.
+
+In the lite distribution, only the `external/storm-kafka-monitor/README` ships. To install it on demand, run:
+
+```
+bin/storm-kafka-monitor-fetch
+```
+
+This resolves `org.apache.storm:storm-kafka-monitor` and its runtime dependencies into `lib-tools/storm-kafka-monitor/`. No daemon restart is required; the UI picks it up on the next request.
+
+The same `--version`, `--dest`, and `--` passthrough options are available as for `storm-autocreds-fetch`.
+
+## lib-common: shared jar de-duplication
+
+Both distributions include a structural change that is transparent to users: jars that were previously duplicated across the daemon classpath (`lib/`) and the worker classpath (`lib-worker/`) are now kept in a single `lib-common/` directory. `bin/storm.py` adds `lib-common` to both classpaths, so the runtime behaviour is unchanged. Only byte-identical jars (same filename and SHA-256) were de-duplicated; no version was silently merged. This accounts for approximately 71 MB of the reduction.
+
+## Which distribution should I use?
+
+Use the **lite distribution** unless you specifically need one of the unbundled plugins:
+
+- If you run topologies on a **Kerberos-secured Hadoop or HBase cluster**, run `bin/storm-autocreds-fetch` after installing from the lite distribution, or use the full distribution.
+- If you want **Kafka spout lag** in the Storm UI or the `storm-kafka-monitor` command, run `bin/storm-kafka-monitor-fetch` after installing from the lite distribution, or use the full distribution.
+
+For all other use cases — including running Kafka spout topologies without the lag UI feature — the lite distribution is sufficient and carries no functional difference from the full distribution.

docs/JitterAwareStreamGrouping.md

@@ -0,0 +1,54 @@
+---
+title: JitterAwareStreamGrouping
+layout: documentation
+documentation: true
+---
+
+`JitterAwareStreamGrouping` is a stream grouping that selects downstream tasks based on observed per-task execution jitter. It steers traffic away from tasks experiencing backpressure or execution variance, producing a more even distribution of work over time.
+
+## Background: the jitter metric
+
+Storm 3.0.0 introduces a per-task jitter metric based on the EWMA algorithm in [RFC 1889 Appendix A](https://www.rfc-editor.org/rfc/rfc1889#appendix-A). The metric tracks inter-arrival time variance in each task's processing loop. A lightweight control loop propagates these measurements to upstream components at regular intervals without touching the data path.
+
+A task with a steady processing rate accumulates near-zero jitter. A task that stalls, GC-pauses, or falls behind its input queue accumulates a rising jitter estimate, which the grouping uses as a signal to route tuples elsewhere.
+
+## Usage
+
+Use `customGrouping` with a `JitterAwareStreamGrouping` instance in place of `shuffleGrouping`:
+
+```java
+import org.apache.storm.grouping.JitterAwareStreamGrouping;
+
+TopologyBuilder builder = new TopologyBuilder();
+builder.setSpout("spout", new MySpout(), 1);
+builder.setBolt("bolt", new MyBolt(), 4)
+       .customGrouping("spout", new JitterAwareStreamGrouping());
+```
+
+`JitterAwareStreamGrouping` is serializable and safe across workers. Before any jitter data arrives (e.g., at topology startup) it falls back to random task selection.
+
+## Benchmarking
+
+The `storm-perf` module includes a dedicated benchmark topology:
+
+```
+storm jar storm-perf/target/storm-perf-*.jar \
+  org.apache.storm.perf.JitterAwareGroupingTopology
+```
+
+Run it alongside the equivalent shuffle-grouping topology to measure the effect in your cluster. The [local development cluster](Local-dev-cluster.html) provides Grafana dashboards with per-task jitter visibility.
+
+## When to use it
+
+- Bolts with variable execution time (external I/O, cache misses, non-uniform key distributions) where a subset of tasks may consistently lag.
+- Topologies where a single hot task creates cascading backpressure upstream.
+- As a drop-in replacement for `shuffleGrouping` on any stream where tasks are interchangeable and execution uniformity matters.
+
+## When not to use it
+
+- **Data-locality requirements**: use `fieldsGrouping` where tuples must reach a specific task.
+- **Low parallelism** (2 tasks): the grouping has fewer candidates to choose between and the benefit diminishes.
+- **Order-sensitive streams**: like any non-deterministic grouping, this does not preserve tuple order across tasks.
+- **Intra-worker traffic**: for streams where both components run in the same worker, `localOrShuffleGrouping` avoids serialization entirely and is preferable.
+
+The jitter control loop runs on all topologies regardless of whether `JitterAwareStreamGrouping` is in use. The grouping simply reads measurements the loop already produces; enabling it carries no additional overhead on the data path.

docs/Local-dev-cluster.md

@@ -0,0 +1,64 @@
+---
+title: Local Development Cluster
+layout: documentation
+documentation: true
+---
+
+Storm ships a Docker Compose configuration under `docker/` that provisions a fully distributed cluster on a single machine. Unlike [local mode](Local-mode.html), this setup forces inter-worker traffic across real sockets, making it suitable for benchmarking serialization, compression, and network-sensitive behaviours that only appear with genuine inter-process communication.
+
+## What's included
+
+- **Nimbus**, **ZooKeeper**, and two **Supervisors** on an isolated Docker network
+- **Prometheus** scraping Storm's Metrics V2 endpoint
+- **Grafana** with a pre-built per-task dashboard covering throughput, latency, and jitter
+- **`netsim.sh`** — a utility that injects controlled latency and jitter between Supervisors using `tc netem`
+
+## Starting the cluster
+
+```
+cd docker/
+docker compose up -d
+```
+
+Once healthy, Grafana is available at `http://localhost:3000` (default credentials: `admin` / `admin`). The Storm UI is at `http://localhost:8080`.
+
+## Submitting a topology
+
+The `storm-perf` module contains topologies designed for this environment:
+
+```
+storm jar storm-perf/target/storm-perf-*.jar \
+  org.apache.storm.perf.FileReadWordCountTopo \
+  -c nimbus.seeds='["nimbus"]' \
+  -c storm.zookeeper.servers='["zookeeper"]'
+```
+
+Any topology JAR works. Point `nimbus.seeds` and `storm.zookeeper.servers` at the Docker hostnames as shown above.
+
+## Network simulation
+
+`netsim.sh` wraps `tc netem` to shape traffic between the two Supervisor containers:
+
+```
+# Add 3 ms latency and 1 ms jitter (typical intra-datacenter profile)
+./netsim.sh apply 3ms 1ms
+
+# Remove all shaping
+./netsim.sh reset
+
+# Show current tc settings on both supervisors
+./netsim.sh status
+
+# Round-trip ping between supervisors
+./netsim.sh ping
+```
+
+Changes take effect immediately and are visible in the Grafana latency panels in real time.
+
+## Teardown
+
+```
+docker compose down
+```
+
+Persistent volumes are not configured; all topology data and metrics are lost on teardown. This cluster is intended for development and benchmarking, not production use.

docs/Storm-Scheduler.md

@@ -25,3 +25,33 @@ Any topologies submitted to the cluster not listed there will not be isolated. N
 
 The isolation scheduler solves the multi-tenancy problem – avoiding resource contention between topologies – by providing full isolation between topologies. The intention is that "productionized" topologies should be listed in the isolation config, and test or in-development topologies should not. The remaining machines on the cluster serve the dual role of failover for isolated topologies and for running the non-isolated topologies.
 
+When choosing which hosts to assign an isolated topology to, the scheduler sorts eligible hosts by three criteria in order:
+
+1. **Assignable slots (descending)** — prefer hosts with greater total slot capacity.
+2. **Free slots (descending)** — among hosts of equal capacity, prefer the one with more currently free slots, minimising the number of worker evictions needed.
+3. **Hostname (ascending)** — alphabetical tiebreaker for deterministic assignments.
+
+The secondary and tertiary sorts were added in Storm 3.0.0. Previously only the primary sort applied, which could cause unnecessary evictions when two hosts had the same total capacity but different numbers of occupied slots.
+
+## EvenScheduler
+
+`EvenScheduler` (used directly or via `DefaultScheduler`) distributes workers as evenly as possible across available supervisors. It is the default when no custom scheduler is configured and `IsolationScheduler` is not in use.
+
+### Idle supervisor rebalance
+
+*Available since Storm 3.0.0. Disabled by default.*
+
+When a supervisor returns from maintenance, its slots are normally left idle: each topology's desired worker count is already satisfied on the surviving supervisors, so `needsScheduling` reports nothing to do. Workers only migrate back if an operator manually rebalances or restarts every affected topology.
+
+Setting `nimbus.even.rebalance.idle.supervisor.enabled: true` adds an opt-in pass that relocates workers onto the returned supervisor automatically in a single scheduling round, distributed across topologies in round-robin order.
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `nimbus.even.rebalance.idle.supervisor.enabled` | `false` | Master switch. When false the pass never runs. |
+| `nimbus.even.rebalance.max.free.per.topology` | `0` | Per-topology cap on workers relocated per round. `0` means no additional cap beyond the even-distribution budget. |
+| `nimbus.even.rebalance.idle.supervisor.min.stable.rounds` | `3` | Number of consecutive monitoring rounds a supervisor must be seen as stable before it is eligible as a relocation target. Set to `0` to disable the flap guard. |
+
+A relocation is a worker JVM restart, involving brief tuple replay, JIT re-warmup, and possible windowed or stateful bolt state restore. Keep this off for topologies with stateful or windowed bolts that have a non-trivial restore cost, latency-sensitive topologies sensitive to JIT re-warmup, and clusters where supervisors are prone to flapping.
+
+**Scope:** `EvenScheduler` and `DefaultScheduler` only. `ResourceAwareScheduler` is unaffected.
+

docs/index.md

@@ -57,6 +57,9 @@ We're also notifying it via annotating classes with marker interface `@Interface
 * [Daemon Metrics/Monitoring](ClusterMetrics.html)
 * [Windows users guide](windows-users-guide.html)
 * [Classpath handling](Classpath-handling.html)
+* [Binary Distributions (full vs. lite)](Binary-distributions.html)
+* [Cluster State Serialization](Cluster-State-Serialization.html)
+* [Local Development Cluster](Local-dev-cluster.html)
 
 ### Intermediate
 
@@ -71,6 +74,7 @@ We're also notifying it via annotating classes with marker interface `@Interface
 * [State Checkpointing](State-checkpointing.html)
 * [Windowing](Windowing.html)
 * [Joining Streams](Joins.html)
+* [JitterAwareStreamGrouping](JitterAwareStreamGrouping.html)
 * [Blobstore(Distcache)](distcache-blobstore.html)
 
 ### Debugging