catena

Small programs that each do one part of turning a pile of files into a searchable corpus, and hand the next one text on stdout.

A catena is a commentary assembled out of excerpts drawn from many sources and chained in order, each one carrying the name of who said it and where. That is both what these programs produce and how they run.

Nothing here needs a database, a GPU, a queue or a config file. Each tool is a filter: bytes in, records out, diagnostics on stderr, and an exit code that means something.

Install

cargo install --git https://git.pumpin.online/funkpower/catena <tool>

Each tool installs by name, so you can take one without taking the rest.

The tools

booksniff	What a file actually is, whatever its name claims
bookmeta	The metadata a file already carries
colophon	The record a book prints in its own front matter
tocfind	A book's own table of contents
langid	What language a document is in
pdfgutter	Books scanned two pages to a sheet, split back into pages
pdftriage	Whether a PDF's text layer is worth reading, or whether it has to be OCRed
bibclean	Catalogue records audited and repaired, with a reason for every change
vocabmerge	The near-duplicate headings in a catalogue's vocabulary
seriesof	The series a book belongs to, and which volume this is
idnorm	DOIs, ISSNs, ISBNs and arXiv ids, read and checked
bibfetch	What a register says about an identifier
citefind	The references in a bibliography or a page of notes
mdscrub	Converted markdown in, clean markdown out
mdsplit	One markdown file into the works inside it
chunkmd	Markdown into passages that carry their page, chapter and role
cuenorm	Any subtitle format in, one normalised cue stream out
cuepack	Cues into passages worth embedding, keeping the times they were said at
cueseg	A timed transcript into topic chapters
emb	Lines into vectors, against an embedding server
rrf	Several ranked lists into one, without the database
avplan	Whether a recording plays, and the one ffmpeg call that fixes it

Each links to its own page: what it does, when you would use it, worked examples, options and exit codes.

→ Examples — programs that run. A book to passages and references, a recording to chapter marks and a copy that plays on a phone, and a hybrid search fused, in shell, Rust, Python and TypeScript.

→ Recipes — using these tools together. A scanned book to a searchable passage, a transcript to a citation with a timestamp, vectors into pgvector, and hybrid search fused on top.

The libraries underneath

Every tool is a library crate with a thin command on top, so anything a command does can be done in process instead.

corpus-core	The record types and the stream plumbing everything shares
textkit	Token counting, sentence and paragraph splitting
cuekit	The cue model, and the three subtitle formats it is read from
catena-cli	The parts of a filter every command does the same way
bindings	The same libraries from Python, C and JavaScript

From another language

The tools are filters first, and a subprocess is the right answer most of the time. When the caller is already holding the text, the same libraries are callable in process from three languages:

	Install	Import
Python	pip install catena-corpus	import catena
JavaScript	npm install catena-corpus	import { chunkMarkdown } from "catena-corpus"
C	build the capi feature	#include <catena.h>

chunks = catena.chunk_markdown(markdown, target=512)
chunks[0]["page"]        # 41

Each hands back the same record shapes the commands write — dictionaries in Python, plain objects in JavaScript, JSON in C — so a stage that was parsing NDJSON from a subprocess can call the function and get what it was already parsing. Details in bindings.

A worked pipeline

Converted markdown in, retrievable passages out:

$ pandoc -t commonmark chapter.xhtml | mdscrub | chunkmd --min-tokens 20
{"v":1,"idx":0,"text":"The obligation ran both ways, and contemporaries said so plainly\nwhenever they were asked.4 A lord who could not\nprotect was a lord who could not command…","tokens":56,"page":41,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":1,"text":"What the rolls do not say, and what has to be inferred, is how often the\nobligation was actually tested…","tokens":143,"page":42,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}

A recording instead of a book, with chapter marks a scrubber can use:

$ cuenorm episode.vtt | cueseg --duration 7200
{"v":1,"start":0.0,"end":1190.6,"title":"The Mill Stood At The Northern End","excerpt":"the mill stood at the northern end of the valley where…"}
{"v":1,"start":1190.6,"end":2381.3,"title":"Parliament Dissolved Itself Again","excerpt":"parliament dissolved itself again amid furious objections…"}

And before any of that, whether a scanned PDF is worth reading at all:

$ pdftriage scan.pdf
garbled, needs OCR
  1820 characters on the opening pages
  1820 characters over 5 sample pages, ascii text score 0.15
  letters 0.13, latin-1 0.29

$ for f in *.pdf; do pdftriage --quiet "$f" && ocrmypdf "$f" "ocr/$f"; done

What they promise

Eight things hold for all of them, and together they are what lets any of these go into a pipeline without first reading how it works:

• Each does a single job, with no mode switch that changes what it is.
• Input on stdin, results on stdout, diagnostics on stderr — never mixed.
• Anything list-shaped is NDJSON, one record per line, flushed as produced.
• Exit codes carry the answer: 0 yes, 1 no, 2 could not tell, and failures in the sysexits range, so "no" and "broke" are never the same number.
• Nothing reaches the network, a database, a config file or any stored state, bar two exceptions that say so in --help.
• Nothing is assumed about where it is running: no hostnames, no paths, no private vocabularies built in.
• The same input gives the same output, on every run and every machine.
• --json is the stable form, and does not change shape without a major version.
• --man writes the program's manual page, generated from the same definition --help is.

The whole of it, with the reasoning: what these programs promise.

The records

Five shapes travel between the tools, versioned, defined in corpus-core and described in the records.

$ cargo run -p corpus-core -- schema all

Build

./check.sh

Formatting, lints, tests and documentation, which is the same gate CI runs.

Licence

MPL-2.0, library crates and binaries alike. The notice lives in LICENSE and the license field of every manifest rather than in a header on every file, which is what MPL-2.0 Exhibit A allows for.