Index format

Each index lives in its own directory under ~/.local/share/quarry/<index-name>/. This page covers what is inside, in enough detail to know what is safe to inspect, back up, or delete.

Layout

~/.local/share/quarry/notes/
├── postings       # inverted index: term to document mapping
├── documents.db   # document table: paths, sizes, mtimes, checksums
└── snippets/      # extracted text, chunked per document

The postings file

postings is the inverted index itself: for every term Quarry has seen, a list of which documents contain it and where. This is what a query is actually matched against, and it is the largest of the three files for most indexes, since it stores a position for every occurrence of every term.

The document table

documents.db is a small embedded database mapping an internal document ID to its file path, size, last modified time, and a content checksum. The checksum is how Quarry decides, on a rescan, whether a file’s content actually changed or just its metadata did, avoiding unnecessary re-extraction for a file that was merely touched.

The snippet store

snippets/ holds extracted, chunked text used to build the preview shown around a match in search results. It is derived entirely from the source files and rebuilt whenever a document is reindexed.

Everything is rebuildable

The entire index directory can be deleted at any time. The next scan, triggered manually with quarry reindex <name> or automatically the next time the daemon starts and finds a missing index, rebuilds it from the original source files.

rm -rf ~/.local/share/quarry/notes
quarry reindex notes

This is the standard way to recover from a corrupted index, and it is also how Quarry handles a version upgrade that changes the internal format: rather than migrating the old files in place, it rebuilds from scratch.

Size, roughly

For a folder of plain text and Markdown, the index directory typically runs 15 to 30 percent of the size of the original content, dominated by the postings file. Binary formats with smaller extractable text, like PDFs, produce a smaller index relative to the file’s size on disk.

Locking

While a scan or reindex is in progress, the index directory is locked to prevent two processes from writing to it at once. A search against a locked index still works, reading the last consistent state; only concurrent writes are prevented. See Exit codes for the code returned when a command hits a locked index.