ingest

Turn a folder into sections

Give cairn a directory of notes. It walks the folder, splits each file at its headings, and stores one memory per section. Search can then hit the section instead of the whole file.

Files it reads: .md, .markdown, and .txt. It skips hidden paths, empty files, and any file larger than 1 MB. A heading is a line that starts with one, two, or three # marks. A section longer than 20,000 characters is split again on blank lines. The heading is kept on each piece.

Run the same folder again. Sections whose text has not changed are counted as unchanged and are not written a second time. A section that is almost the same as one already stored under that file name is flagged. It is not dropped. You decide whether to replace the old one.

From the terminal

cairn ingest ./docs --team acme --json

--type defaults to document. The JSON report has files, created, unchanged, flagged, and flagged_keys. Each flagged row names the file, the heading, and the nearby memory keys.

From an editor agent

The tool is cairn_ingest. It reads a directory on the machine that runs cairn-mcp. Pass dir and team. memory_type is optional.

{
  "dir": "/home/me/project/docs",
  "team": "acme",
  "memory_type": "document"
}

From Python

The same function serves the command and the tool. Call it when a script should load a corpus and then inspect the report.

from cairn.ingest import ingest_dir

report = ingest_dir(client, "acme", "docs")
for hit in report["flagged_keys"]:
    print(hit["file"], hit["heading"], hit["near"])

client is a CairnClient opened on the project vault. New sections in one file are embedded together, then stored. The relative path is saved as provenance, so you can see which file a section came from.

File names share a task

The task id is the file name, lowercased, with punctuation turned into dashes. docs/api/README.md and docs/setup/README.md both become the task readme. Their text stays separate, and the relative path is still on the memory. Near-duplicate checks for those two files see each other, because they share a task. Give files distinct names when each document should stay in its own task.