Validation¶
hfsplus-forensic parses untrusted Apple HFS+/HFSX structures from potentially
compromised disk images. Correctness is therefore established the way forensic
tooling must be: against independent oracles (a different tool, or a different
code path, that already decodes the same bytes correctly) on real third-party
or real macOS-produced corpora with known ground truth — never against fixtures
we hand-encoded and then graded ourselves.
This page records exactly which oracle and which corpus back each capability, so
the claim is independently re-checkable. Per-file provenance (source, generator
command, hashes, license) lives in tests/data/README.md;
the fleet-wide machine index is issen/docs/corpus-catalog.md. This page
cross-references both rather than duplicating them.
How to read the evidence tiers¶
Each validation below is tagged with the trustworthiness of its check, not whether the data is "synthetic":
- Tier 1 — an independent third party authored the artifact and the answer key, or it is real-world data decoded by an independent tool. The strongest claim.
- Tier 2 — real engine output whose ground truth is derivable from the documented construction, or confirmed by an independent code path on real data. Genuinely checked, but we chose the scenario.
- Tier 3 — fixture and expected answer both authored here, nothing independent vouching. Used only for per-branch coverage, never as a correctness claim: a self-consistent round trip proves internal consistency, not correctness against real-world bytes.
Independent oracles¶
| Oracle | Independent of us? | Validates | Tier |
|---|---|---|---|
Apple AppleFSCompression (ditto --hfsCompression) |
Yes — Apple's own compressor wrote the real LZVN forks | LZVN (type 8) resource-fork decode: our output must equal the original pre-compression file that Apple compressed | 1 |
Apple AppleFSCompression (afsctool -c -T ZLIB / -T LZFSE) |
Yes — drives Apple's real framework | zlib (types 3/4) and LZFSE (types 11/12) decode: output must equal the original pre-compression file | 1 |
Apple compression_decode_buffer (COMPRESSION_LZVN) |
Yes — the OS's own decoder | The macOS 26.5 (Tahoe) type-8 .expected answer key (equals the kernel's transparent read) for a real LZVN block with trailing bytes |
1 |
lzvn crate (lzvn-core) |
Yes — vetted third-party codec we reuse | The LZVN decode itself (length-tolerant, reads real decmpfs blocks with trailing bytes after end-of-stream) | 1 |
flate2 / lzfse_rust crates |
Yes — vetted third-party codecs we reuse | The zlib / LZFSE decode itself | 1 |
macOS hdiutil (HFS+ filesystem builder) |
Yes — Apple's own filesystem writer | HFS+ volume-header geometry, catalog B-tree directory listing, and data-fork extraction: ground truth is the layout Apple's builder wrote (the files/contents we placed) | 2 |
Independent test corpora¶
All fixtures are produced by Apple's own tools (hdiutil, ditto,
afsctool, the kernel's transparent-compression read path) on real macOS,
including two captured from a clean macOS 26.5 (Tahoe) system. They are small,
committed (the parser is byte-buffer-driven and the tests include_bytes! them),
and carry ground truth derivable from the documented construction or from an
independent Apple decoder. Hashes and full provenance are in
tests/data/README.md.
| Corpus | Source | Used for | License / redistribution |
|---|---|---|---|
HFS+ volume + header + nested (hfs_plus_header.bin, hfs_plus_volume.bin, hfs_plus_nested.bin) |
macOS hdiutil create -fs HFS+ -layout NONE |
Volume-header geometry, root + nested directory listing, data-fork extraction | REAL-self; committed |
LZVN resource fork (lzvn.rsrc + lzvn.expected) |
macOS ditto --hfsCompression (decmpfs type 8) |
Type-8 LZVN resource-fork decode vs the original file | REAL-self; committed |
decmpfs end-to-end volume (hfs_decmpfs_volume.bin) |
hdiutil + ditto --hfsCompression (4 MiB HFS+ volume, comp.bin + plain.bin control) |
End-to-end read_file transparent decompression |
REAL-self; committed |
zlib / LZFSE resource forks + inline payloads (real_zlib_rsrc.rsrc, real_lzfse_rsrc.rsrc, real_zlib_inline.payload, real_lzfse_inline.payload, zlib.expected, real_zlib_inline.expected) |
afsctool -c -T ZLIB / -T LZFSE over /usr/share/dict/words |
Type 3/4 zlib and type 11/12 LZFSE decode vs the original files | REAL-self; committed |
Tahoe type-8 + type-9 (tahoe_type8.rsrc/.expected, tahoe_type9.decmpfs/.expected) |
Real files (/usr/bin/loads.d, /usr/bin/pp) on macOS 26.5 (build 25F71), read-only mount |
Real LZVN with trailing bytes; real type-9 inline marker; .expected for type-8 from Apple compression_decode_buffer |
REAL-self; committed |
type-3 0xFF-stored inline (zlib_type3_stored.payload + zlib_inline.expected) |
Hand-built (Apple's compressor never emits this marker) | The decmpfs "stored" remainder branch | SYNTHETIC (the sole synthetic fixture); committed |
Per-capability validation¶
decmpfs LZVN (type 8) decompression — Tier 1¶
src/decmpfs.rs:350 (decodes_real_macos_lzvn_resource_fork) decodes a real
ditto --hfsCompression resource fork (lzvn.rsrc, 2 × 64 KiB chunks) and
asserts the output equals lzvn.expected — the original 80 000-byte file Apple
compressed. Apple wrote the compressed bytes; the answer key is the file before
compression. The end-to-end path is validated by tests/decmpfs_integration.rs:40
(read_file_transparently_decompresses_decmpfs_lzvn), which reads comp.bin from
a real 4 MiB HFS+ volume and requires it to read back as the original 262 144-byte
payload (regenerated byte-for-byte from the documented LCG, so no expected fixture
is committed), with plain.bin as the uncompressed control. The codec itself is
the vetted third-party lzvn crate.
decmpfs zlib (types 3/4) decompression — Tier 1¶
src/decmpfs.rs:360 (decodes_real_macos_zlib_resource_fork) decodes a real
afsctool -T ZLIB type-4 resource fork and matches zlib.expected (150 KB of
real text). src/decmpfs.rs:370 (decodes_real_macos_inline_zlib) does the same
for a real type-3 inline (xattr) payload. Real data earned its keep here: it
exposed that zlib block offsets are relative to headerSize+4, a bug
self-consistent synthetic fixtures had passed. The codec is flate2.
decmpfs LZFSE (types 11/12) decompression — Tier 1¶
src/decmpfs.rs:416 (decodes_real_macos_lzfse_resource_fork) decodes a real
afsctool -T LZFSE type-12 resource fork against the same 150 KB original;
src/decmpfs.rs:426 (decodes_real_macos_inline_lzfse) validates a real
type-11 inline payload. Real data exposed that LZFSE forks zero-pad their chunk
table. The codec is lzfse_rust.
decmpfs on real macOS 26.5 (Tahoe) — Tier 1¶
src/decmpfs.rs:467 (decodes_real_tahoe_type8_lzvn_with_trailing_bytes) decodes
a real type-8 LZVN block from /usr/bin/loads.d that carries trailing bytes
after the LZVN end-of-stream opcode — the case strict whole-stream decoders
reject. Its .expected answer key is produced by Apple's own
compression_decode_buffer (COMPRESSION_LZVN), an independent OS decoder
equal to the kernel's transparent read. src/decmpfs.rs:477
(decodes_real_tahoe_type9_inline_marker) validates a real type-9 inline xattr
from /usr/bin/pp with its 1-byte storage marker. These two real samples
exposed two decoder bugs that synthetic ditto fixtures had masked (strict
trailing-byte reject, and an unstripped type-9 marker); the wider Tahoe capture
moved from 0/35 to 35/35 real samples decoding correctly.
decmpfs uncompressed / fail-loud branches — Tier 3 (per-branch coverage)¶
The inline-uncompressed (src/decmpfs.rs:390), chunked-uncompressed
(src/decmpfs.rs:436), and type-9-shape (src/decmpfs.rs:451) cases, plus the
fail-loud arms — bad magic (:497), truncated header (:507), unknown type
(:512), unsupported LZBITMAP/dedup (:518, :527), missing resource fork
(:536), and length-mismatch (:486) — use hand-built inputs. They establish
that each branch behaves and that a malformed or under-length input fails loud
(DecmpfsError) rather than returning a short or wrong buffer; they are not a
correctness claim against real-world bytes. The sole synthetic fixture, the
type-3 0xFF-stored payload (src/decmpfs.rs:380), covers a remainder branch
Apple's real compressor never emits.
HFS+ volume header, directory listing, file extraction — Tier 2¶
tests/catalog.rs parses real hdiutil-created HFS+ output. The header test
(parses_real_volume_header, tests/catalog.rs:24) asserts version 4, 4096-byte
allocation blocks, 512 total blocks, 2 MiB volume size. Listing and extraction are
checked on a populated volume: lists_real_root_directory
(tests/catalog.rs:40) requires HELLO.TXT / READ.ME / SUBDIR with correct
dir/file flags; reads_real_file_contents (tests/catalog.rs:62) requires
HELLO.TXT to read back as b"hello hfs"; walk_lists_nested_paths
(tests/catalog.rs:97) requires the nested path SUB/NESTED.TXT and its
b"nested data" contents. Ground truth is the layout Apple's hdiutil wrote and
the files we placed — derivable from the documented construction, hence Tier 2.
This is not yet cross-checked against an independent HFS+ decoder.
HFS+ anomaly analyzer (findings::audit) — Tier 2, TSK-cross-checked¶
src/findings.rs grades structural and metadata anomalies over the parsed
volume — five HFS-* finding families — and is validated two ways in
tests/findings.rs:
True-negative (clean real volumes are silent). audit() returns zero
anomalies on all three real hdiutil/ditto-created images
(clean_real_volume_has_no_anomalies, clean_real_nested_volume_has_no_anomalies,
clean_real_decmpfs_volume_has_no_anomalies). The analyzer's own catalog
readings were cross-checked against The Sleuth Kit on the same images and
agree exactly:
| Image | TSK fsstat / istat |
findings::audit reading |
Anomalies |
|---|---|---|---|
hfs_plus_volume.bin |
HFS+, 2 files, 3 folders, block size 4096; HELLO.TXT (CNID 18) size 9, created == modified | 2 files, 3 folders, block size 4096; 1 block allocated covers a 9-byte fork; create == content-mod | 0 ✅ |
hfs_plus_nested.bin |
HFS+; SUB/NESTED.TXT reachable |
2 files, 3 folders; nested path walked | 0 ✅ |
hfs_decmpfs_volume.bin |
comp.bin (CNID 20) compressed flag set, com.apple.decmpfs xattr 16 B, resource fork 34796 B present |
type-8 LZVN, resource-fork storage, non-empty resource fork → no missing-resource finding | 0 ✅ |
TSK commands used as the oracle: fsstat <img> (file/folder counts, block size,
volume dates), istat <img> <inode> (per-file size, timestamps, resource-fork
size, compressed flag), fls -r <img> (catalog walk / CNIDs).
Positive (crafted corruption of those same real volumes). Each anomaly
family is driven by patching real volume bytes and asserting the matching graded
finding fires (and that clean siblings stay silent), e.g.
corrupt_btree_node_link_is_flagged, leaf_height_anomaly_is_flagged,
header_kind_on_nonzero_node_is_flagged → HFS-BTREE-NODE-INVALID;
catalog_extents_mismatch_is_flagged → HFS-CATALOG-EXTENTS-MISMATCH;
deleted_but_referenced_is_flagged → HFS-DELETED-BUT-REFERENCED;
create_after_modify_is_flagged / timestamp_before_epoch_is_flagged /
timestamp_after_volume_is_flagged → HFS-TIME-ANOMALY;
decmpfs_missing_resource_is_flagged / decmpfs_empty_resource_logical_is_flagged
/ decmpfs_inline_without_payload_is_flagged / decmpfs_truncated_header_is_flagged
/ decmpfs_unknown_type_is_flagged → HFS-DECMPFS-MISSING-RESOURCE. Ground
truth is the byte we flipped, derivable from the documented construction — hence
Tier 2.
Each finding is an observation (note() reads "consistent with …"), never a
verdict — the analyst/tribunal concludes. The analyzer is panic-free and
bounds-checked like the reader; findings.rs carries 100% line coverage
(cargo llvm-cov) except a handful of defensive guards annotated
// cov:unreachable (node-bounds and extent-record guards kept for
defence-in-depth under their dominating invariant).
Robustness — never panic, never over-read¶
Production code is #![forbid(unsafe_code)] (enforced via [lints.rust]
unsafe_code = "forbid"). Every length and offset is bounds-checked, and a decmpfs
file that cannot be decoded returns None/Err rather than the (empty) data
fork — decmpfs never degrades to silently-wrong output.
Gaps and honest caveats¶
- HFS+ catalog readings are TSK-cross-checked; the reader's tests are not.
The analyzer (
findings::audit) has its file/folder counts, sizes, and timestamps reconciled against The Sleuth Kitfsstat/istat/flson the same images (see the analyzer section above) — a genuine independent oracle for those readings. The reader's own integration tests (tests/catalog.rs) still assert against thehdiutil-written layout only, without a TSK comparison in the test body; wiring TSK output intotests/catalog.rswould lift the reader tests themselves to Tier 1. - No fuzzing harness yet. CI runs
cargo fmt --check,cargo clippy --all-targets -D warnings, andcargo test.findings.rsholds 100% line coverage (cargo llvm-cov, defensive guards annotated// cov:unreachable), but CI does not yet gate oncargo llvm-cov, and there is nofuzz/cargo-fuzz workspace. Both remain recommended fleet backstops to add (the Paranoid-Gatekeeper standard).
Reproducing the validation¶
All fixtures are committed, so every test runs with a plain cargo test — no
environment variables, no large-image download, no --ignored gate:
# Everything (decmpfs codec unit tests + HFS+ reader integration tests)
cargo test
# Just the decmpfs codec unit tests (real LZVN / zlib / LZFSE + Tahoe + fail-loud)
cargo test --lib
# Just the HFS+ reader integration tests (real hdiutil volumes)
cargo test --test catalog
# Just the HFS+ anomaly analyzer (clean true-negatives + crafted positives)
cargo test --test findings
# Just the end-to-end decmpfs read_file decompression
cargo test --test decmpfs_integration
Cross-check the analyzer's catalog readings against The Sleuth Kit on the same images (oracle):
fsstat tests/data/hfs_plus_volume.bin # file/folder counts, block size
istat tests/data/hfs_plus_volume.bin 18 # HELLO.TXT size + timestamps
fls -r tests/data/decmpfs/hfs_decmpfs_volume.bin # catalog walk / CNIDs
istat tests/data/decmpfs/hfs_decmpfs_volume.bin 20 # comp.bin: compressed flag + resource fork