Optimize GroupNorm

README.md

@@ -54,16 +54,18 @@ The model is trained from a random initialization until convergence, which is de
 1. If running the benchmark for the first time, or running with different fractal parameters (`n_categories`, `variance_threshold`) than previously, generate fractal classes and instances:  
     `scaffold generate_fractals -c ScaFFold/configs/benchmark_default.yml`
 
-    Note that the benchmark ships with an initial set of 50 fractal classes.
+    Fractal category libraries are generated deterministically from the
+    configured seed (under `fract_base_dir/var<...>/seed<...>/`) and reused by
+    later runs with the same seed.
 
 1. Once fractal generation completes, run the benchmark:  
     `torchrun-hpc -N 1 -n 4 --gpus-per-proc 1 $(which scaffold) benchmark -c ScaFFold/configs/benchmark_default.yml`
 
 ScaFFold benchmark training always uses PyTorch distributed execution with DistConv spatial parallelism. For a singleton run, launch one distributed rank rather than disabling distributed execution.
 
-`benchmark` creates a folder for the benchmark run(s) at `base_run_dir` set in the config file. For reproducibility, we store a copy of the benchmark run config yml. Within each run subfolder, `benchmark` creates a yml config for that specific run.
+Each `benchmark` invocation performs exactly one benchmark run, in a run folder created under `base_run_dir` set in the config file. Every run parameter must be single-valued; a list (e.g. `problem_scale: [6, 7]`) is rejected by name, since parameter sweeps are not supported. To compare parameter settings, launch one benchmark run per setting. For reproducibility, the run folder holds a copy of the benchmark config yml as `base_config.yaml` plus the fully merged `config.yaml` for that run.
 
-After each run completes, statistics from the run are stored in `train_stats.csv`. Additionally, users can inspect plots of the training and validation losses over time in `<base_run_dir/figures`.
+After the run completes, statistics from the run are stored in `train_stats.csv`. Additionally, users can inspect plots of the training and validation losses over time in `<base_run_dir/figures`.
 
 Parameters are set in a `.yml` config file and can be modified by the user. See
 [`ScaFFold/configs/benchmark_default.yml`](ScaFFold/configs/benchmark_default.yml)
@@ -143,7 +145,7 @@ The weights we use to scale IFS parameters look like the following:
 
 ### Dataset generation
 
-Finally,  we are ready to generate a dataset for training our model. Each sample in the dataset is composed of several fractal instances (default=3), randomly selected from any category, overlain with eachother in a 3D voxel grid. Each fractal instance in a sample is placed in a random, non-centered location in the voxel grid if the `scale` parameter set in the sweep config file is <1; otherwise, each fractal instance is centered on the center of the voxel grid. Below is an outline of the data generation process:
+Finally,  we are ready to generate a dataset for training our model. Each sample in the dataset is composed of several fractal instances (default=3), randomly selected from any category, overlain with eachother in a 3D voxel grid. Each fractal instance in a sample is placed in a random, non-centered location in the voxel grid if the `scale` parameter set in the benchmark config file is <1; otherwise, each fractal instance is centered on the center of the voxel grid. Below is an outline of the data generation process:
 ```
 For n  in n_volumes:
 
@@ -172,7 +174,7 @@ For n  in n_volumes:
 
 #### 1. Profiling with the PyTorch Profiler
 
-Set `PROFILE_TORCH=ON` to generate a PyTorch profiling trace that can be read into [Perfetto](https://ui.perfetto.dev/).
+Set `PROFILE_TORCH=ON` to generate a PyTorch profiling trace that can be read into [Perfetto](https://ui.perfetto.dev/). The trace is written into the run directory. `1`, `true`, `on` and `yes` (any case) enable profiling; every other value, including `0`, `false`, `no` and `off`, leaves it disabled.
 
 #### 2. Profiling with Caliper & Adiak
 

ScaFFold/benchmark.py

@@ -12,45 +12,18 @@
 #
 # SPDX-License-Identifier: (Apache-2.0)
 
-import itertools
 import shutil
 from argparse import Namespace
 from pathlib import Path, PosixPath
 
-import yaml
 from mpi4py import MPI
 
 from ScaFFold import worker
-from ScaFFold.utils.config_utils import Config
 from ScaFFold.utils.distributed import get_world_rank
 from ScaFFold.utils.perf_measure import adiak_init, adiak_value
 from ScaFFold.utils.utils import setup_mpi_logger
 
 
-def expand_sweep_combinations(config_dict):
-    """
-    Expand list-valued sweep parameters into per-run scalar configs.
-
-    Scalar-typed config fields holding lists are treated as sweep dimensions;
-    the cross product of their values yields one config dict per combination.
-    An all-scalar config yields exactly one combination. Legitimately
-    list-typed fields (e.g. dc_num_shards) are never treated as sweeps.
-    """
-    sweep_keys = sorted(
-        k for k in Config._SCALAR_KEYS if isinstance(config_dict.get(k), list)
-    )
-    if not sweep_keys:
-        return [dict(config_dict)]
-
-    combinations = []
-    value_lists = [config_dict[k] for k in sweep_keys]
-    for values in itertools.product(*value_lists):
-        combo = dict(config_dict)
-        combo.update(dict(zip(sweep_keys, values)))
-        combinations.append(combo)
-    return combinations
-
-
 def main(kwargs_dict: dict = {}):
     args = Namespace(**kwargs_dict)
     log = setup_mpi_logger(__file__, args.verbose)
@@ -60,65 +33,42 @@ def main(kwargs_dict: dict = {}):
     rank = get_world_rank(required=True)
     log.debug("args found: %s", args)
 
-    run_dicts = None
-    # Now set up and start benchmark run(s)
+    run_dict = None
+    # Now set up and start the benchmark run. Each invocation runs exactly one
+    # benchmark run, in the run directory the CLI resolved.
+    benchmark_run_dir = args.benchmark_run_dir
     if args.restart:
-        # Resume the run in the directory the CLI resolved. The worker path
-        # reads config.run_dir/run_iter directly (a missing run_dir would crash
-        # BaseTrainer), so fill both from the resolved benchmark_run_dir here
-        # just as the fresh, single-combination path does below.
-        benchmark_run_dir = args.benchmark_run_dir
-        kdict = {k: v for k, v in vars(args).items() if k not in ["command"]}
-        kdict["run_dir"] = str(benchmark_run_dir)
-        kdict["run_iter"] = Path(f"{benchmark_run_dir}/run")
-        run_dicts = [kdict]
+        # Resume the run in that directory. The worker path reads
+        # config.run_dir/run_iter directly (a missing run_dir would crash
+        # BaseTrainer), so fill both in here just as the fresh path does below.
+        run_dict = {k: v for k, v in vars(args).items() if k not in ["command"]}
+        run_dict["run_dir"] = str(benchmark_run_dir)
+        run_dict["run_iter"] = Path(f"{benchmark_run_dir}/run")
     elif rank == 0:
-        # Get run dir
-        benchmark_run_dir = args.benchmark_run_dir
-
-        # Save copy of benchmark config yml to run dir
+        # Save a copy of the benchmark config yml in the run dir, under a fixed
+        # name of its own. Copying it in under its original name overwrites the
+        # merged config.yaml the CLI just wrote whenever the base config file
+        # happens to be called config.yaml -- and restart.sh points -c at
+        # $RUN_DIR/config.yaml, so the restart would reload the raw base config
+        # with none of the run's overrides or metadata.
         bench_config_path = Path(args.config)
-        shutil.copy(bench_config_path, benchmark_run_dir)
-
-        base_dict = {k: v for k, v in vars(args).items() if k not in ["command"]}
-        combinations = expand_sweep_combinations(base_dict)
+        shutil.copy(bench_config_path, Path(benchmark_run_dir) / "base_config.yaml")
 
-        # One run (and run directory) per parameter combination. The single
-        # all-scalar combination keeps the original flat layout.
-        run_dicts = []
-        if len(combinations) == 1:
-            kdict = combinations[0]
-            kdict["run_dir"] = str(benchmark_run_dir)
-            kdict["run_iter"] = Path(f"{benchmark_run_dir}/run")
-            run_dicts.append(kdict)
-        else:
-            for i, kdict in enumerate(combinations):
-                run_dir = Path(benchmark_run_dir) / f"param_set_{i}"
-                run_dir.mkdir(parents=True, exist_ok=True)
-                kdict["run_dir"] = str(run_dir)
-                kdict["run_iter"] = run_dir / "run"
-                with open(run_dir / "run_config.yaml", "w") as file:
-                    yaml.dump(
-                        {
-                            k: str(v) if isinstance(v, PosixPath) else v
-                            for k, v in kdict.items()
-                        },
-                        file,
-                    )
-                run_dicts.append(kdict)
+        run_dict = {k: v for k, v in vars(args).items() if k not in ["command"]}
+        run_dict["run_dir"] = str(benchmark_run_dir)
+        run_dict["run_iter"] = Path(f"{benchmark_run_dir}/run")
 
     comm.Barrier()
-    run_dicts = comm.bcast(run_dicts, root=0)
+    run_dict = comm.bcast(run_dict, root=0)
 
     adiak_init(comm)
-    for kdict in run_dicts:
-        # Add all config params as metadata
-        for key, value in kdict.items():
-            if isinstance(value, dict):
-                log.debug("Adiak: skipping key with dict value '%s'", key)
-                continue
-            if isinstance(value, PosixPath):
-                value = str(value)
-            adiak_value(key, value)
-
-        worker.main(kwargs_dict=kdict)
+    # Add all config params as metadata
+    for key, value in run_dict.items():
+        if isinstance(value, dict):
+            log.debug("Adiak: skipping key with dict value '%s'", key)
+            continue
+        if isinstance(value, PosixPath):
+            value = str(value)
+        adiak_value(key, value)
+
+    worker.main(kwargs_dict=run_dict)