aegean.db¶

SQLite persistence for a Corpus: a faithful, queryable round-trip (documents + tokens + provenance) with an optional FTS5 full-text index, plus lazy streaming. Stdlib sqlite3 only. See also Corpus.to_sql / Corpus.from_sql.

db ¶

SQLite persistence for a Corpus — stdlib sqlite3 only, so the core stays dependency-free (the same precedent as aegean.cache).

to_sqlite / from_sqlite are a faithful, queryable round-trip: documents and tokens are normalized into rows (so SQL and full-text search work over them), with the nested structure (signs, alternate readings, annotations, line groupings, image refs, notes) kept in JSON columns. Provenance and the sign inventory live in a small key/value meta table. search matches a whole token by default (an FTS5 phrase when available, else an indexed exact-match query on tokens(text)); mode="substring" is the opt-in within-token search (a casefolded Python scan, since SQLite's LIKE folds ASCII case only). stream yields documents lazily for a large DB-backed corpus without materializing it.

A corpus out of the database cites exactly like a corpus out of JSON — the provenance round- trips, and a database grown with append=True records every appended corpus's provenance, so the reloaded corpus cites them all. Corpus.to_sql / Corpus.from_sql are thin wrappers over these functions.

to_sqlite ¶

to_sqlite(corpus: Corpus, path: str | Path, *, fts: bool = True, append: bool = False, progress: Callable[[int, int], None] | None = None) -> None

Write corpus to a SQLite database at path.

By default this overwrites any existing file. With append=True it instead upserts corpus's documents into an existing database (by document id — a document with a matching id is replaced, others are added), keeping the rest intact; the FTS5 index is refreshed, and the appended corpus's provenance is recorded alongside the original's, so the reloaded corpus cites every source that went in (see from_sqlite). Documents and tokens become queryable rows; provenance and the sign inventory live in a meta table. Round-trips losslessly via from_sqlite.

progress (optional) is called as progress(done, total) after each document is written, counting documents — the hook a minutes-long write (a DDbDP-sized corpus is ~57k documents) reports through; the CLI paints a live line from it. The written database is identical with or without it; the final call is (total, total), made before the FTS index build (one bulk statement, not per-document).

from_sqlite ¶

from_sqlite(path: str | Path, *, progress: Callable[[int, int], None] | None = None) -> Corpus

Reconstruct the Corpus written by to_sqlite — documents, sign inventory, and provenance, byte-for-byte equivalent to the original. A database grown with to_sqlite(append=True) stores one provenance per appended source; those come back as a single combined provenance naming every source, so Corpus.cite stays truthful.

progress (optional) is called as progress(done, total) after each document is materialized, counting documents — the hook the ~100 s DDbDP whole-corpus load (aegean.load("ddbdp")) reports through. The returned corpus is identical with or without it; an empty database makes no calls, and the final call is (total, total).

search ¶

search(path: str | Path, query: str, *, limit: int = 50, mode: str = 'token') -> list[tuple[str, int | None, str]]

Search a SQLite corpus's tokens; returns (doc_id, position, text) hits.

position is the token's stored position, or None for a token saved without one (a supported state since 0.19.4: an appended token keeps position=None).

mode="token" (default) matches a whole token literally: the query must equal the token, so a transliteration with hyphens (KU-RO, A-DU) matches only that token and never a longer token that merely contains it (KU-RO does not match PO-TO-KU-RO). It uses the FTS5 index when present (then confirms the exact match); without FTS5 it falls back to an indexed exact-match query for an ASCII query, or a Python scan for a non-ASCII one (SQLite's NOCASE folds ASCII case only).

mode="substring" matches the query as a substring of a token, so KU-RO also finds PO-TO-KU-RO — useful for tracing every token a sign-group occurs in. It folds case in Python (SQLite's LIKE also folds ASCII only), scanning the token table: about 4 ms on the bundled 1,721-document Linear A corpus, linear in the token count.

Both modes fold case, Greek included (ku-ro finds KU-RO; λόγος finds ΛΌΓΟΣ, final sigma folding with the rest). Diacritics still have to match: λογος does not find λόγος.

limit caps the number of hits (default 50); zero or a negative value returns every match. The database is opened read-only (a sqlite mode=ro URI), so a search can never create or modify a file: a missing path or a non-SQLite file raises sqlite3.OperationalError / sqlite3.DatabaseError instead of leaving an empty database behind.

stream ¶

stream(path: str | Path) -> Iterator[Document]

Yield the documents of a SQLite corpus one at a time, without materializing the corpus.

A separate read cursor fetches each document's tokens on demand, so memory stays flat for arbitrarily large databases — the DB-backed counterpart to a streamed load (item: large corpora). Pairs with from_sqlite when random access is wanted instead.