fs/vfs: Add ioctldir for volume ioctls via the mountpoint directory.

[casaroli]

Summary

Some ioctl commands act on a volume rather than on a file — FIOC_REFORMAT,
FIOC_OPTIMIZE, FIOC_INTEGRITY, FIOC_DUMP. The only route into a file
system, though, has been the per-file ioctl method, so a caller has to open
an unrelated file just to name the volume it means.

For NXFFS that is not merely awkward, it is a dead end. nxffs_ioctl()
refuses FIOC_REFORMAT while any file on the volume is open:

/* We cannot reformat the volume if there are any open inodes */

if (volume->ofiles)
  {
    ferr("ERROR: Open files\n");
    ret = -EBUSY;
    goto errout_with_lock;
  }

and nxffs_open() puts every opened file on exactly that list:

ofile->flink   = volume->ofiles;
volume->ofiles = ofile;

The descriptor used to issue the command is itself an open file on the
volume, so the check can never pass. FIOC_REFORMAT is documented,
implemented, and impossible to invoke.

This adds an optional ioctldir method to struct mountpt_operations,
reached by issuing the ioctl on a descriptor for the mountpoint directory:

int fd = open("/mnt/nxffs", O_RDONLY | O_DIRECTORY);
ioctl(fd, FIOC_REFORMAT, 0);
close(fd);

CODE int (*ioctldir)(FAR struct inode *mountpt,
                     FAR struct fs_dirent_s *dir,
                     int cmd, unsigned long arg);

The method takes the same (mountpt, dir) pair as opendir/readdir/
rewinddir, so it reads as a member of the directory-operations family; a
file system recovers the volume from the mountpoint inode and may ignore
dir (NXFFS does). It is placed at the end of the structure rather than
beside the other directory operations on purpose — see Impact.

dir_ioctl() forwards any command it does not handle itself to that method,
when the directory belongs to a mounted volume and the file system supplies
one. NXFFS implements it, which is what makes its FIOC_REFORMAT reachable.

Why a new method rather than reusing ioctl. The obvious alternative —
have dir_ioctl() call the existing per-file method — is not safe. That
method takes a struct file, and the implementations dereference
filep->f_priv after asserting on it:

fs/fat/fs_fat32.c:1733          DEBUGASSERT(filep->f_priv != NULL);
fs/romfs/fs_romfs.c:597         DEBUGASSERT(filep->f_priv != NULL);
fs/tmpfs/fs_tmpfs.c:2007        DEBUGASSERT(filep->f_priv != NULL);
fs/spiffs/src/spiffs_vfs.c:945  DEBUGASSERT(filep->f_priv != NULL);

A directory descriptor has no open file behind it, so forwarding into those
would assert or fault. A separate entry point keeps that contract intact and
makes support explicit rather than assumed.

File	Change
include/nuttx/fs/fs.h	ioctldir added to struct mountpt_operations
fs/vfs/fs_dir.c	dir_ioctl() forwards unhandled commands to it
fs/nxffs/nxffs_ioctl.c	shared implementation; per-file and per-directory entry points
fs/nxffs/nxffs_initialize.c	registers the method
fs/nxffs/nxffs.h	declaration
Documentation/components/filesystem/index.rst	documents the new VFS method
Documentation/components/filesystem/nxffs.rst	documents how to reach the two ioctls, and why FIOC_REFORMAT needs this route

Impact

Users: additive. A new way to reach volume-wide ioctls; nothing that
works today changes. NXFFS keeps its per-file path, and both entry points
share one implementation. FIOC_REFORMAT on NXFFS becomes usable for the
first time.

File systems: the new member is appended at the end of
struct mountpt_operations, so the positional initialisers every file
system uses are unchanged — several already stop short of the end of the
structure today and continue to compile untouched. A file system that leaves
it NULL behaves exactly as before: the VFS answers -ENOTTY for any
command on a directory descriptor it does not handle itself.

Although ioctldir belongs conceptually with the directory operations
(opendir…rewinddir), it is placed at the end rather than beside them
precisely for this reason: all 21 in-tree filesystems initialise
mountpt_operations positionally, so a member inserted mid-structure would
force every one of them to add a NULL slot for a method it does not
implement. Appending keeps the change to the single filesystem that uses it.

Build / hardware / security: no build system changes, no hardware
dependency, no new configuration options. No change to any syscall
signature or user-visible ABI.

Documentation: updated in this PR, both the VFS method reference and the
NXFFS page.

Compatibility: no behavioural change for any existing caller. The
forwarding only happens for commands that previously returned -ENOTTY.

Testing

Host: macOS 15 (Darwin 25.5.0), Apple silicon.
Target: sim:nxffs (simulator).

A test was written that exercises both routes on one volume: create a file,
issue FIOC_REFORMAT through a descriptor on that file, then through a
descriptor on the mountpoint directory, then issue an unrecognised command
on the directory to confirm it is still refused.

On master — the directory route does not exist, so the command is
unreachable by any means:

via file fd:      ret=-1 errno=16 (EBUSY, as expected)
via directory fd: ret=-1 errno=25
bogus cmd on dir: ret=-1 errno=25

With this change, same test, same configuration:

via file fd:      ret=-1 errno=16 (EBUSY, as expected)
via directory fd: ret=0 errno=0 (reformatted)
bogus cmd on dir: ret=-1 errno=25

The first line is unchanged, confirming the per-file path still behaves as
it always did. The second shows the command now succeeds. The third confirms
an unrecognised command on a directory descriptor is still refused with
-ENOTTY, so the forwarding did not turn dir_ioctl() into a catch-all.

The existing NXFFS test (CONFIG_TESTING_NXFFS, 100 loops of fill/delete/
verify against the simulated MTD) was run on sim:nxffs and passes
unchanged.

tools/checkpatch.sh -f passes on every changed file.

Follow-up, not in this PR

SPIFFS implements FIOC_INTEGRITY, FIOC_REFORMAT, FIOC_OPTIMIZE and
FIOC_DUMP with the same shape and can adopt the method the same way. It is
left out here to keep the change reviewable, and because NXFFS is the case
where the command is currently unreachable rather than merely awkward.

[github-actions]

MemBrowse Memory Report

esp32-devkitc

• ROM: .flash.rodata +4 B, .flash.text +40 B (+0.0%, 123,672 B / 4,194,272 B, total: 3% used)
• drom0_0_seg: .flash.rodata +4 B (+0.0%, 13,352 B / 4,194,272 B, total: 0% used)
• irom0_0_seg: .flash.text +40 B (+0.0%, 87,984 B / 3,342,304 B, total: 3% used)

hifive1-revb

• flash: .text +56 B (+0.1%, 83,348 B / 4,194,304 B, total: 2% used)

qemu-armv8a

• Code: .rodata +32 B, .text.dir_ioctl +64 B (+0.0%, 316,830 B)

qemu-intel64

• Code: .text +61 B (+0.0%, 8,656,786 B)
• Data: .rodata +32 B (+0.0%, 120,551 B)

rx65n-rsk2mb

• ROM: .text +48 B (+0.1%, 86,444 B / 2,097,152 B, total: 4% used)

s698pm-dkit

• Code: .text +160 B (+0.0%, 361,040 B)

stm32-nucleo-f103rb

• flash: .text +40 B (+0.1%, 33,924 B / 131,072 B, total: 26% used)
  No memory changes detected for:
• arduino-mega2560
• mirtoo