fs/xipfs: writable execute-in-place file system for downloadable modules

Documentation/applications/examples/nxflatxip/index.rst

@@ -0,0 +1,87 @@
+=================================================
+``nxflatxip`` NXFLAT Executed In Place from XIPFS
+=================================================
+
+Writes an NXFLAT module into a :doc:`XIPFS </components/filesystem/xipfs>`
+volume at run time, the way a download would, and runs two instances of it
+concurrently. The module's text is executed directly out of flash and shared
+between the instances; each instance gets its own data.
+
+The same module in a ROMFS image would execute in place just as well. What it
+could not do is arrive after the firmware was built, which is the point of
+the exercise.
+
+Usage::
+
+    nxflatxip [bench | defrag]
+
+With no argument it stages the module, reports the extent it landed in and
+the flash address the file system will map it at, then runs both instances::
+
+    nsh> nxflatxip
+    === NXFLAT module executed in place from xipfs ===
+
+    staged 268 bytes to /mnt/xipfs/xipmod
+    extent: block 6 x1, size 268, flash addr 0x10106000
+
+    running two concurrent instances...
+
+    [while running] pin count on the shared text = 2
+
+      instance seed 1: text 0x10106070, stack 0x20006860, sum 2080 -- data private and intact
+      instance seed 2: text 0x10106070, stack 0x200074b0, sum 4160 -- data private and intact
+    [after exit]    pin count on the shared text = 2 (released when this task exits)
+
+The two instances report the same text address, inside the flash window and
+equal to where the file lies on the media, and different stack addresses.
+While they run the extent carries one pin per instance, which is what stops
+the defragmenter relocating code that is executing.
+
+The pins outlive the instances: ``binfmt`` maps the text in the context of
+whoever called ``exec()``, so the mappings belong to the demo's address space
+and are released when it exits. Run ``xipfs -n`` afterwards to see the count
+back at zero.
+
+Subcommands
+===========
+
+``bench``
+  Times a bulk read out of the mapped flash before and after a write. On a
+  target whose driver has to tear down and restore execute-in-place around
+  each erase, the difference is the cost of whichever restore path it used.
+  Needs a large file to read; the command says how to create one.
+
+``defrag``
+  Fills the volume, deletes every other file until an allocation genuinely
+  cannot be satisfied, compacts, retries the same allocation, then verifies
+  every relocated file byte for byte and again after a remount. Prints a
+  block map at each step.
+
+Building the module
+===================
+
+The module is built from ``module/xipmod.c`` at build time, exactly the way
+:doc:`../nxflat/index` builds its test programs, so it needs the same two
+host tools from the NuttX toolchain, ``mknxflat`` and ``ldnxflat``, and the
+board's ``Make.defs`` must name them.
+
+The module has no static data and no string constants, and reports through a
+callback into the firmware rather than formatting its own output. The comment
+at the top of ``module/xipmod.c`` records why: the ``ldnxflat`` in
+circulation miscomputes both of those cases. The callback also exercises the
+other direction of the interface, firmware code entered with the module's
+data base still live in its PIC register.
+
+Configuration
+=============
+
+``CONFIG_EXAMPLES_NXFLATXIP``
+  Enable the example. Requires ``CONFIG_FS_XIPFS``, ``CONFIG_NXFLAT`` and
+  ``CONFIG_LIBC_EXECFUNCS``.
+
+``CONFIG_EXAMPLES_NXFLATXIP_MOUNTPT``
+  The XIPFS volume to use.
+
+``CONFIG_EXAMPLES_NXFLATXIP_MTD``
+  The MTD device backing that volume, used only by the ``defrag``
+  subcommand, which remounts to show that what it did survives.

Documentation/applications/system/xipfs/index.rst

@@ -0,0 +1,68 @@
+=========================================
+``xipfs`` XIPFS Compaction and Block Map
+=========================================
+
+Compacts a :doc:`XIPFS </components/filesystem/xipfs>` volume and reports how
+its erase blocks are laid out.
+
+Because a XIPFS file is a single contiguous extent, an allocation can fail
+for lack of a contiguous run while plenty of free space remains. Compaction
+coalesces the holes that deletes leave behind. It is never automatic: an
+allocation simply fails with ``ENOSPC`` and the caller decides whether
+compacting is worth its cost in erases. This command is that decision made
+by hand.
+
+Usage::
+
+    xipfs [-n] [-t <ms>] [<mountpoint>]
+
+``-n``
+  Survey and report only; move nothing.
+
+``-t <ms>``
+  Give the pass a time budget. It stops at the first extent boundary after
+  the budget is spent, which is always a consistent layout.
+
+``<mountpoint>``
+  The volume to act on. Defaults to ``CONFIG_SYSTEM_XIPFS_MOUNTPOINT``.
+
+The compaction itself is asked for through a descriptor for the mountpoint
+directory, so no file inside the volume is open while it runs and one pass
+can reach every extent. Without ``-n`` the command surveys, compacts, and
+surveys again, so the map it prints last is the one on the media::
+
+    nsh> xipfs -n
+    /mnt/xipfs: 250 blocks of 4096 bytes (1000 KB)
+
+      file                             start  blocks    bytes  pin
+      xipmod                               0       1      268    0
+
+         0 |#.................................................|
+        50 |..................................................|
+
+      '.' free   '#' in use   'P' pinned by a live mapping
+
+      used 1, free 249, largest free run 249 blocks (996 KB)
+      1 free run, fragmentation 0%
+
+The fragmentation figure is the share of free space a single allocation
+cannot reach, ``(free - largest_run) / free``. At 0% the largest possible
+file already fits however scattered the map looks, which is the question a
+caller facing ``ENOSPC`` actually has.
+
+Files are listed by their path relative to the mount: the command walks the
+volume's directories, which hold no blocks of their own and so do not appear
+in the map.
+
+A ``P`` marks an extent held by a live execute-in-place mapping. Compaction
+skips those, because relocating them would move code that is executing, so
+they bound what any pass can reclaim.
+
+Configuration
+=============
+
+``CONFIG_SYSTEM_XIPFS``
+  Enable the command. Requires ``CONFIG_FS_XIPFS``.
+
+``CONFIG_SYSTEM_XIPFS_MOUNTPOINT``
+  The volume used when none is named on the command line.

Documentation/applications/testing/xipfs/index.rst

@@ -0,0 +1,98 @@
+==================================
+``xipfs`` XIPFS File System Test
+==================================
+
+Exercises the :doc:`XIPFS </components/filesystem/xipfs>` file system: the
+routine VFS paths, the write-once rules, both flavours of mapping, and then
+the two properties that are easy to get wrong and quiet when they are --
+release of execute-in-place pins, and power-loss atomicity of the metadata
+commit.
+
+Usage::
+
+    xipfs_test [<section>]
+
+With no argument every section runs. The power-loss sweeps take minutes while
+everything else takes seconds, hence the selector::
+
+    nsh> xipfs_test xip
+    XIPFS test suite on /mnt/xipfs (/dev/rpflash)
+    -- XIP mapping --
+      PASS  create module image
+      PASS  media is memory mapped
+      PASS  strict XIP mmap succeeds
+      PASS  mapping points into flash
+      PASS  mapped bytes match file contents
+      PASS  mapping does not cost RAM proportional to the file
+      PASS  mapping takes a pin
+      PASS  munmap
+      PASS  munmap drops the pin
+    ==== 9 passed, 0 failed ====
+
+Sections
+========
+
+``basic``
+  Create, read back, stat, readdir, unlink, and that all of it survives a
+  remount.
+
+``writeonce``
+  That reopening a closed file for writing, appending, and seeking during a
+  write are refused, and that a sequential write still works afterwards.
+
+``strict``
+  That a mapping past the end of a file is refused and a valid one is not.
+
+``xip``
+  That a mapping lands inside the media window rather than the heap, that the
+  mapped bytes are the file's, that it costs no RAM proportional to the file,
+  and that it takes and releases a pin.
+
+``multimap``
+  That N mappings of one file produce N pins on one extent and alias the same
+  address.
+
+``crosstask``
+  That one task dying does not release another task's pin, and that the last
+  holder does.
+
+``teardown``
+  That a task killed without calling ``munmap`` still has its pin released.
+  This is the one that matters for a module that faults.
+
+``dirs``
+  That directories behave like a real tree: an empty one exists and survives
+  a remount, ``readdir`` reports each level, ``rmdir`` refuses a directory
+  that still holds something and succeeds once emptied, and a path through a
+  missing directory or through a file fails rather than being created.
+
+``defrag``
+  That compaction relocates extents, keeps their contents, survives a
+  remount, distinguishes an open file from a pinned one, and never relocates
+  a pinned extent.
+
+``powerloss``, ``powertorn``, ``unlink``, ``defraglos``
+  Fail the Nth flash operation, remount, and check the volume is consistent
+  and every committed file byte-for-byte intact -- sweeping N across create,
+  unlink and compaction. The ``torn`` variant leaves the failing write
+  partially programmed rather than cleanly refused, which is what a real
+  power loss mid-program leaves behind. These need
+  ``CONFIG_FS_XIPFS_FAULT_INJECT``.
+
+Configuration
+=============
+
+``CONFIG_TESTING_FS_XIPFS``
+  Enable the test. Requires ``CONFIG_FS_XIPFS``.
+
+``CONFIG_TESTING_FS_XIPFS_MOUNTPT``
+  The volume to test. The suite remounts it, so nothing else should be using
+  it.
+
+``CONFIG_TESTING_FS_XIPFS_MTD``
+  The MTD device backing that volume, needed for the remounts.
+
+.. note::
+
+   The suite writes and erases the volume continuously, the power-loss
+   sections most of all. Point it at a volume whose contents do not matter.

Documentation/components/filesystem/index.rst

@@ -587,6 +587,7 @@ NuttX provides support for a variety of file systems out of the box.
   tmpfs.rst
   unionfs.rst
   userfs.rst
+  xipfs.rst
   zipfs.rst
   inotify.rst
   nuttxfs.rst
@@ -603,7 +604,7 @@ they require:
 
 1. They require a block device driver. They include vfat, romfs, smartfs, and
    littlefs.
-2. They require MTD drivers. They include romfs, spiffs, littlefs.
+2. They require MTD drivers. They include romfs, spiffs, littlefs, xipfs.
 3. They require neither block nor MTD drivers. They include nxffs, tmpfs, nfs
    binfs, procfs, userfs, hostfs, cromfs, unionfs, rpmsgfs, and zipfs.