docs: update FSNotify documentation

- Detail the challenges of finding the 'watchee' path
- Explain the use of open_by_handle_at() and Irmap
- Explicitly document that pending events are dropped with a warning
- Explain how spurious events are generated during restore (ghost files)
- Add details for Fanotify inode and mount marks

Signed-off-by: Andrei Vagin <avagin@google.com>

Documentation/under-the-hood/fsnotify.md

@@ -1,39 +1,38 @@
-# Fsnotify
+# FSNotify (Inotify and Fanotify)
 
-## Challenges in Dumping and Restoring Fsnotify
+CRIU supports checkpointing and restoring `inotify` and `fanotify` instances. These mechanisms allow applications to monitor filesystem events (like file creation, modification, or deletion).
 
-The implementation of `fsnotify` is relatively straightforward—we can identify watched items by their handles from the `procfs` output:
+## The Challenges of FSNotify C/R
 
-```
-pos:	0
-flags:	02000000
-inotify wd:3 ino:9e7e sdev:800013 mask:800afce ignored_mask:0 fhandle-bytes:8 fhandle-type:1 f_handle:7e9e0000640d1b6d
-```
+Restoring an fsnotify instance is inherently difficult because the kernel does not provide a direct way to retrieve the original path of a watched object (the "watchee"). Furthermore, the event queues themselves pose consistency risks.
 
-During a dump, CRIU records the watched file handle and, during restoration, reopens it by retrieving the path from the file descriptor link provided by `procfs`.
+### 1. Identifying the Watchee
+When an application adds a watch (via `inotify_add_watch`), the kernel associates the watch with an **inode**, but it does not store the **path** used to create it. To restore the watch, CRIU must find a valid path to that specific inode.
+*   **Open by Handle**: CRIU first attempts to use `open_by_handle_at()`. If the filesystem supports file handles, CRIU captures the handle during the dump and uses it to re-open the inode during restoration without needing the original path.
+*   **Irmap (Inode Reverse Mapping)**: If file handles are unavailable, CRIU uses the [Irmap](irmap.md) engine to scan the filesystem and find a path that leads to the target inode.
 
-This works well until watched items are descendants of another watch descriptor. Consider a directory `dir` containing two files, `a` and `b`:
+### 2. Event Queue Consistency
+If there are pending events in the fsnotify queue at the time of the dump, CRIU cannot currently "peek" at them or safely migrate them.
+*   **Dropped Events**: During a dump, CRIU checks if the fsnotify file descriptor has data. If it does, CRIU emits a warning: `The ... inotify events will be dropped`. These events are lost, and the application must be prepared to handle this gap in its event stream.
+*   **Spurious Events**: The process of checkpointing and restoring itself can trigger new filesystem events. For example, creating or deleting **ghost files** (temporary files used to restore unlinked but open files) can generate `IN_CREATE` or `IN_DELETE` events that the application will see upon resumption.
 
-```
-dir
- `- a
- `- b
-```
+### 3. Ghost Files and Circular Dependencies
+A "ghost file" is a file that was deleted by the application but is still held open. During restoration, CRIU must recreate these files. This action itself generates notify events, potentially confusing applications that monitor the directories where these ghost files are temporarily placed.
 
-If a program sets up an `fsnotify` mark on `dir` and both files, and then opens and unlinks `a` and `b`, notify events are generated. If these events are still in the queue (i.e., the program has not read them yet) when a dump begins, a problem arises. Because the kernel currently lacks an API to "peek" at events in the queue (reading them without removing them), we must either ignore these events or refuse the dump.
+## Support for Fanotify
 
-Refusing the dump is sometimes necessary because of CRIU's current design, where a dump attempt might force CRIU to generate its own events, leading to an endless cycle. This is primarily due to "ghost" files—files that have been removed by the application but whose file descriptors remain open. For these, CRIU generates a hard link to the deleted file during dumping, which triggers notify events. A similar situation occurs during restoration when ghost files are unlinked, again causing the kernel to generate events.
+CRIU also supports `fanotify`, including:
+*   **Inode Marks**: Similar to inotify, these target specific files or directories.
+*   **Mount Marks**: Fanotify can monitor entire mount points. CRIU identifies the mount ID and restores the mark on the corresponding mount in the restored namespace.
 
-Until the `fsnotify` dump/restore procedure is redesigned, we must ignore non-empty notify queues during a dump and accept that CRIU will generate its own events during restoration.
+## Current Strategy: "Chopping the Knot"
 
-## Potential Solutions
-
-Possible ways to resolve this include:
-
-- **Late-stage processing**: When dumping, collect `fsnotify` and ghost file descriptors into separate lists and process them at the very end, reading out notify events from the `fsnotify` descriptors.
-- **Deferred restoration**: During restoration, collect `fsnotify` descriptors in the root CRIU task and defer their restoration until all other files from every child process are restored. Then, restore the notifies and read out all generated events.
-
-Both approaches would require significant changes to CRIU's architecture. For now, CRIU simply prints a warning if an `fsnotify` queue is not empty and continues processing.
+Due to the complexity of perfectly migrating event queues, CRIU's current strategy is:
+1.  **Warn and Drop**: Acknowledge that pending events are lost.
+2.  **Restore the Watches**: Ensure the application continues to receive *new* events after restoration.
+3.  **Namespace Integration**: Correctly map mount-level fanotify marks within their respective mount namespaces.
 
 ## See also
-- [Irmap](irmap.md)
+* [Irmap](irmap.md)
+* [Invisible Files](invisible-files.md)
+* [Mount Points](mount-points.md)