aegean.db¶
SQLite persistence for a Corpus: a faithful, queryable round-trip (documents + tokens +
provenance) with an optional FTS5 full-text index, plus lazy streaming. Stdlib sqlite3
only. See also Corpus.to_sql / Corpus.from_sql.
Since 0.61.0, replacing an existing database uses SQLite's online backup transaction. Existing WAL readers retain their snapshot until their transaction ends; subsequent reads see the replacement. A competing writer can cause a bounded timeout, leaving the previous database intact. External file renames/unlinks while connections are open remain outside this contract.
Creating a new database uses an atomic, no-clobber hard link on the destination filesystem. Filesystems without hard-link support fail explicitly. Non-SQLite destinations and orphaned journal sidecars are refused. Users of 0.60.0 should close all connections before replacing a WAL database.
Index lexical searches¶
Build an optional lexical index on a writable local database to accelerate exact surface, normalized, and lemma searches:
from aegean import db
db.build_lexical_index("corpus.db")
print(db.lexical_index_status("corpus.db"))
hits = db.search("corpus.db", "λόγος", view="lemma", statuses=["certain"])
Index keys use the same normalization and reading-status rules as corpus queries. Missing lemmas remain excluded, and hits preserve their original text and order. Building again refreshes the index transactionally. Token edits, including direct SQL edits, invalidate the index; searches use the scan path until it is rebuilt. Search and index-status inspection open the database read-only. Substring searches retain their scan behavior. Existing databases need no index to remain usable.
The CLI equivalents are aegean db index corpus.db and
aegean db index corpus.db --status --json. Index creation takes a local path,
not a hosted corpus identifier; make a writable working copy of a fetched corpus
before indexing it.
db ¶
SQLite persistence for a Corpus — stdlib sqlite3 only, so the core stays
dependency-free (the same precedent as aegean.cache).
to_sqlite / from_sqlite are a faithful, queryable round-trip: documents and tokens
are normalized into rows (so SQL and full-text search work over them), with the nested
structure (signs, alternate readings, annotations, line groupings, image refs, notes) kept
in JSON columns. Provenance and the sign inventory live in a small key/value meta table.
search matches a whole token by default (an FTS5 phrase when available, else an indexed
exact-match query on tokens(text)); mode="substring" is the opt-in within-token search
(a casefolded Python scan, since SQLite's LIKE folds ASCII case only). stream yields
documents lazily for a large DB-backed corpus without materializing it.
Explicit lexical searches can use materialized comparison keys built or refreshed with
build_lexical_index. Token writes invalidate these keys in the same transaction;
lexical_index_status reports whether they can be used. Searches never build an index.
A corpus out of the database cites exactly like a corpus out of JSON — the provenance round-
trips, and a database grown with append=True records every appended corpus's provenance,
so the reloaded corpus cites them all. Corpus.to_sql / Corpus.from_sql are thin
wrappers over these functions.
to_sqlite ¶
to_sqlite(corpus: Corpus, path: str | Path, *, fts: bool = True, append: bool = False, progress: Callable[[int, int], None] | None = None) -> None
Write corpus to a SQLite database at path.
By default this overwrites any existing SQLite database. Replacement uses a
SQLite transaction, including when committed pages live in an active WAL: readers
retain their old snapshot until their transaction ends. A busy writer is given five
seconds to finish before replacement raises sqlite3.OperationalError and keeps
the existing database intact. Non-SQLite files are refused without overwriting them.
With append=True it instead
upserts corpus's documents into an existing database (by document id — a document
with a matching id is replaced, others are added), keeping the rest intact; the FTS5
index is updated transactionally, and the appended corpus's provenance is recorded
alongside the original's, so the reloaded corpus cites every source that went in
(see from_sqlite).
Documents and tokens become queryable rows; provenance and the sign inventory live in
a meta table. Round-trips losslessly via from_sqlite.
progress (optional) is called as progress(done, total) after each document is
written, counting documents — the hook a minutes-long write (a DDbDP-sized corpus is
~57k documents) reports through; the CLI paints a live line from it. The written
database is identical with or without it; the final call is (total, total),
made before the FTS index build (one bulk statement, not per-document).
from_sqlite ¶
from_sqlite(path: str | Path, *, progress: Callable[[int, int], None] | None = None) -> Corpus
Reconstruct the Corpus written by to_sqlite — documents, sign inventory, and
provenance, byte-for-byte equivalent to the original. A database grown with
to_sqlite(append=True) stores one provenance per appended source; those come back
as a single combined provenance naming every source, so Corpus.cite stays truthful.
progress (optional) is called as progress(done, total) after each document is
materialized, counting documents — the hook the ~100 s DDbDP whole-corpus load
(aegean.load("ddbdp")) reports through. The returned corpus is identical with or
without it; an empty database makes no calls, and the final call is (total, total).
lexical_index_status ¶
Report an exact-search index's state without writing or scanning token rows.
state is absent before an explicit build, ready when its source
generation and lexical/Unicode semantics agree, stale after source/index
changes, or unsupported for an unrecognized storage structure. Only a
ready index is used. indexed_tokens describes the last complete build,
including nonword tokens retained for status-only searches.
build_lexical_index ¶
build_lexical_index(path: str | Path, *, progress: Callable[[int, int], None] | None = None) -> dict[str, Any]
Explicitly build or refresh exact lexical keys on an existing writable database.
Uses the same surface/normalized/lemma and whole-token/segment status rules
as search. A single write transaction replaces derived rows and their
generation marker; rollback, interruption, and competing writers cannot
publish partial keys. Live schemas are retained during refresh. Triggers
invalidate the index on every token INSERT/UPDATE/DELETE, including external
SQL; append/update requires another explicit refresh. Full corpus replacement
produces a database without this optional index.
Existing hosted databases continue to scan until explicitly indexed on a
writable copy. progress(done, total) counts tokens; an exception from the
callback rolls back the entire build. Returns lexical_index_status for the
committed build. Read-only searches never call this function.
search ¶
search(path: str | Path, query: str, *, limit: int = 50, mode: str = 'token', view: WordView | None = None, statuses: Iterable[ReadingStatus | str] | None = None) -> list[tuple[str, int | None, str]]
Search a SQLite corpus's tokens; returns (doc_id, position, text) hits.
position is the token's stored position, or None for a token saved without
one (a supported state since 0.19.4: an appended token keeps position=None).
mode="token" (default) matches a whole token literally: the query must equal the
token, so a transliteration with hyphens (KU-RO, A-DU) matches only that token and
never a longer token that merely contains it (KU-RO does not match PO-TO-KU-RO). It
uses the FTS5 index when present (then confirms the exact match); without FTS5 it falls
back to an indexed exact-match query for an ASCII query, or a Python scan for a non-ASCII
one (SQLite's NOCASE folds ASCII case only).
mode="substring" matches the query as a substring of a token, so KU-RO also
finds PO-TO-KU-RO — useful for tracing every token a sign-group occurs in. It folds
case in Python (SQLite's LIKE also folds ASCII only) and therefore has to scan the
token table: on the bundled 1,721-document Linear A corpus (6,406 tokens) that is
around 9 ms warm, against around 2 ms for the same query in token mode. Both
figures are machine-dependent; the substring cost is linear in the token count, so it
scales with the corpus while the indexed whole-token match does not.
Both modes fold case, Greek included (ku-ro finds KU-RO; λόγος finds
ΛΌΓΟΣ, final sigma folding with the rest). Diacritics still have to match:
λογος does not find λόγος.
Explicit view='surface', 'normalized', or 'lemma' searches lexical
WORD tokens using the same keys as Corpus.word_frequencies and
Corpus.query. Surface is case-sensitive; normalized/lemma use NFC,
lowercase, and final-sigma folding, preserving diacritics/iota subscript.
Lemma reads existing annotations only; missing lemmas do not match. Returned
hits still contain the original token text. statuses selects whole tokens
and every typed editorial segment. Exact searches use a ready index built with
build_lexical_index, otherwise scan. Substring searches always scan. Omitting
both options preserves the legacy indexed, case-insensitive search across all
token kinds.
limit caps the number of hits (default 50); zero or a negative value returns
every match. The database is opened read-only (a sqlite mode=ro URI), so a
search can never create or modify a file: a missing path or a non-SQLite file raises
sqlite3.OperationalError / sqlite3.DatabaseError instead of leaving an empty
database behind.
stream ¶
stream(path: str | Path) -> Iterator[Document]
Yield the documents of a SQLite corpus one at a time, without materializing the corpus.
A separate read cursor fetches each document's tokens on demand, so memory stays flat for
arbitrarily large databases — the DB-backed counterpart to a streamed load (item: large
corpora). Pairs with from_sqlite when random access is wanted instead.