The Scanner Bottleneck: From FileManager to POSIX

One syscall changed everything.

The first working scanner used FileManager.enumerator. You probably would have written the same thing. It is the obvious choice: idiomatic Swift, clean API, handles symlinks and error cases, does exactly what the documentation says. You call it, you get a sequence of URLs, you read attributes off each one. A few dozen lines of code and the whole filesystem is yours.

It worked. It returned correct results. And it was slow enough to make me question my life choices.

FileManager does this for each file: it crosses from user space into the kernel, asks for attributes, gets a dictionary back, and returns to user space. One round trip per file. For 5.2 million files, that is 5.2 million kernel boundary crossings. The kernel already has all the information you need, sitting in memory, ready to go. But the API forces you to ask for it one file at a time, like ordering a million items from a warehouse and insisting on a separate delivery truck for each one.

getattrlistbulk is the answer Apple has had since OS X 10.10. One system call per directory. The kernel returns all file attributes for every entry in that directory in a single packed buffer. Name, type, size, modification date: all of it, in one read. This is how Finder does it. This is how Spotlight does it. It is documented in the BSD layer of the macOS SDK, . The man page is detailed, the attribute list constants are well-named. It is just not what you reach for when you are writing Swift and feeling civilized about it.

graph LR
    subgraph FM["FileManager approach"]
        FM1[Directory entry] -->|"attributesOfItem()<br/>1 syscall per file"| FM2[Single attribute dict]
        FM2 -->|repeat per file| FM3[FileNode]
    end
    subgraph PX["getattrlistbulk approach"]
        PX1[Directory] -->|"getattrlistbulk()<br/>1 syscall per directory"| PX2["Packed buffer<br/>(all entries, all attrs)"]
        PX2 -->|walk buffer| PX3[FileNode per entry]
    end
    FM3 -.->|"5.2M nodes"| OUT[ScanResult]
    PX3 -.->|"5.2M nodes: 1.7x faster"| OUT

The implementation lives in POSIXDirectoryScanner. It drops down to C-level system calls, which means working with raw pointers.

getattrlistbulk writes into a buffer you provide. The buffer contains a tightly packed sequence of variable-length attribute records. You walk it by reading a 4-byte record length at the start of each record, then advancing the pointer by that length. Inside each record, the attribute fields are packed in the order you requested them, with alignment requirements that vary by type.

One rule the Apple documentation is explicit about: do not dereference these pointers directly for multi-byte integers. The buffer is not guaranteed to be aligned. On architectures that enforce alignment, you get a bus error. On those that do not, you get silent data corruption. Use memcpy into a local variable instead.

// Correct: memcpy before reading a 4-byte field
uint32_t size;
memcpy(&size, ptr, sizeof(size));
ptr += sizeof(size);

// Wrong: direct dereference may fault on unaligned reads
uint32_t size = *((uint32_t *)ptr);

In practice, unaligned reads work on modern Apple Silicon. You follow the spec anyway, because “works on my machine” is not a memory safety argument.

Two edge cases needed specific handling.

The first is FAT32 volumes. External drives formatted as FAT32 behave differently under getattrlistbulk: some attribute combinations are not supported, and the call returns ENOTSUP or truncated records. The scanner detects this and falls back gracefully rather than producing corrupt data.

The second is cloud drives. iCloud files evicted to the cloud have two size attributes: ATTR_FILE_DATALENGTH (the logical size) and ATTR_FILE_ALLOCSIZE (the bytes actually on disk). For a disk analyzer, you want bytes on disk. Reading DATALENGTH on an evicted file can trigger a network fetch, pulling the file body back down from the cloud. That is wrong in three ways: it is slow, it burns bandwidth, and the number it returns is a lie about local storage.

The cloud drive path detection checks for /Library/CloudStorage/ and /Library/Mobile Documents/ in the volume path. When those paths are detected, the scanner requests ALLOCSIZE instead of DATALENGTH. The disk usage numbers stay grounded in what is actually consuming space on your machine.

The pipeline after the change looks like this:

graph TD
    V[Volume root] --> D[Directory]
    D -->|getattrlistbulk<br/>one syscall| B[Attribute buffer]
    B -->|parse packed struct| N[FileNode]
    N -->|child dir?| D
    N -->|file| T[Tree accumulation]
    T --> SR[ScanResult]

Each directory is one syscall. The buffer parsing is a tight loop over contiguous memory. The tree accumulates in a depth-first traversal driven by a work stack.

The benchmark after switching:

Debug build, M3 Pro. Release builds would be faster, but I measured what I had. The warm/cold distinction matters: filesystem cache can shift these numbers by 2-4x depending on whether the OS has recently traversed the same directories. The warm numbers are after three prior scans of the same disk. A user’s first scan hits cold cache.

The 1.7x improvement on warm cache is almost entirely syscall reduction. The work being done is the same: visit every entry, record name and size and type. The difference is how many times you cross the kernel boundary to get there.

The POSIXDirectoryScanner is a drop-in replacement via the DirectoryScannerProtocol. The rest of the app does not know or care which scanner it is using. The protocol boundary kept the change local.

Every platform has the idiomatic API tier and the one the system actually uses.

For most code, the idiomatic tier is the right choice. But when you are doing something the platform does thousands of times per second, it is worth asking what the platform actually uses. Finder scans directories with getattrlistbulk. Spotlight uses it. The API has been there since OS X 10.10.

FileManager was right for prototyping. getattrlistbulk was right for shipping. The 1.7x improvement was real but humbling: I had targeted 2.5-3x. The remaining scan time is dominated by tree construction, 5.2 million FileNode allocations and String constructions that no syscall change can fix. Sometimes the bottleneck moves. You follow it.

References

• getattrlistbulk(2) man page: Apple Developer Library
• File System Programming Guide: Apple Developer Library
• POSIX: Wikipedia