aegean.core¶

core ¶

Script-agnostic core: data model, script plugin contract, corpus, numerals.

Corpus ¶

Corpus(documents: list[Document], sign_inventory: SignInventory | None = None, provenance: Provenance | None = None, script_id: str = '')

A collection of Document s plus shared inventory + provenance.

load `classmethod` ¶

load(script_id: str, *, version: str | None = None) -> 'Corpus'

Load a registered corpus by name, e.g. Corpus.load("lineara") (bundled) or Corpus.load("damos") (fetched to the local data store on first use).

Loaders may cache one built instance per process (the bundled corpora do), so the result is returned as a copy: mutate it freely, and documents added, dropped, or edited never leak into later load calls.

version (optional) loads a kept historical release of a fetched corpus that the project still hosts, for reproducing an earlier analysis byte-for-byte (e.g. version="v1" for a pre-0.29.0 epigraphy corpus). It applies only to the corpora with kept historical pins (isicily, iip, iospe, igcyr, edh, ddbdp; see aegean.data.historical_versions) and fetches into a separate version-suffixed cache entry, leaving the current data untouched. The default (version=None) is unchanged.

get ¶

get(doc_id: str) -> Document | None

The document with id doc_id, or None if there is no such document.

fingerprint ¶

fingerprint() -> str

A stable content hash of this corpus, covering everything a token-level analysis can see: the script id, the provenance data_version, each document's id, every token's text, TokenKind, ReadingStatus, decomposed signs, Unicode glyphs, alternate readings (alt), and annotations (hashed as sorted key/value pairs), and any subset: / merged: / appended: provenance note. Document metadata (site, period, ...) is deliberately excluded. Cheap relative to the analyses it keys: one pass over the tokens, no model build. Two corpora with the same fingerprint have the same analysable content, so it's the cache key for aegean.cache-memoised analyses (including the sign-level dispersion/keyness that read signs).

cache_key ¶

cache_key() -> str

Alias for fingerprint, the protocol aegean.cache keys on.

copy ¶

copy() -> 'Corpus'

A structurally independent copy: a fresh document list and, per document, fresh token/line/translation containers, so mutating the copy (or the original) never affects the other. Token and Sign are frozen value objects but each carries a mutable per-element dict (annotations / attrs) for user analysis, so the copy rebuilds those with independent dicts, keeping a per-token annotation edit (or a per-sign attr edit) from leaking into the original, a sibling copy, or a later load of the same cached corpus. The remaining fields (DocumentMeta, Provenance, the immutable Token/Sign scalars) are shared. One pass over the tokens — a fraction of a second even for the largest corpora (the annotation-rich NT is the slowest); the copy fingerprints identically to the original.

filter ¶

filter(**meta: Any) -> 'Corpus'

Return a new Corpus whose documents match all given metadata fields (AND-combination), e.g. corpus.filter(site="HT", period="LMIB").

The subset's provenance records what was filtered (a subset: note), so cite on the result cites the exact subset used.

subset ¶

subset(ids: Iterable[str]) -> 'Corpus'

A new Corpus of just the documents whose id is in ids (original order kept).

The id-based counterpart to filter; records a subset: provenance note so cite on the result names the slice.

merge ¶

merge(*others: 'Corpus', dedupe: str = 'error') -> 'Corpus'

Merge this corpus with others into one (documents concatenated in order).

dedupe controls duplicate document ids across the inputs: "error" (raise, listing the collisions — the safe default), "first" / "last" (keep that occurrence), or "suffix" (rename later collisions id#2, id#3, …). The merged script_id is the common value, or "mixed" when the inputs differ; the sign inventory is the first input's when scripts agree, else None. A fresh provenance names every source, so cite on the result stays truthful.

See also aegean.combine, the module-level form that takes a list.

cite ¶

cite(style: str = 'plain') -> str

Cite this corpus — or the exact filtered subset — in one call.

style: "plain" (one line), "bibtex" (a @misc entry), or "apa". Filtered subsets (see filter) carry a subset: note that all three styles include, so the citation states exactly what was used.

iter_documents ¶

iter_documents() -> Iterator[Document]

Iterate documents (the explicit-name form of iter(corpus)).

iter_tokens ¶

iter_tokens() -> Iterator[Token]

Every Token, in document then in-document order — a memory-friendly stream that never builds an all-tokens list (useful on a large corpus).

iter_words ¶

iter_words() -> Iterator[str]

Every lexical (WORD) token's text, in order, lazily. The unit word_frequencies counts — stream it to feed your own Counter or a running statistic without materialising a list.

word_frequencies ¶

word_frequencies() -> list[tuple[str, int]]

(word, count) for every lexical word, sorted by descending count.

diagnose ¶

diagnose(level: str = 'quick') -> 'DiagnoseReport'

A descriptive corpus-health report: reading-status profile, provenance / citation completeness, Aegean accounting reconciliation, numeral anomalies, annotation review state, and (level="full") sign-frequency outliers.

Composes existing public machinery into one aegean.core.diagnose.DiagnoseReport (.print() / .to_markdown() / .to_dataframe()); every check degrades gracefully on a corpus it does not apply to. Descriptive, not a verdict.

to_dataframe ¶

to_dataframe(level: str = 'document')

A pandas DataFrame at document, token, or word level.

pandas is an optional dependency — install with pip install 'pyaegean[data]'.

to_dict ¶

to_dict() -> dict[str, Any]

A compact, lossy export (_meta + per-document words/metadata) for quick interop. For a complete, reversible serialization use to_json/from_json.

to_json ¶

to_json(path: str | Path | None = None, *, indent: int | None = 2) -> str | None

Serialize the whole corpus to JSON losslessly — every token (with its kind, signs, glyphs, line/position), the physical lines, full document metadata, the sign inventory, and provenance all survive. from_json reverses it exactly.

Returns the JSON string, or writes it to path and returns None when path is given. (Unlike to_dict, which is a compact lossy summary.)

from_json `classmethod` ¶

from_json(source: str | Path) -> 'Corpus'

Reconstruct a Corpus from to_json output: a JSON string, a Path to a .json file, or a path-like string (anything not beginning with {).

from_records `classmethod` ¶

from_records(records: Sequence[dict[str, Any]], *, script_id: str = 'custom', provenance: Provenance | None = None, sign_inventory: SignInventory | None = None) -> 'Corpus'

Build a corpus from plain dict records — your own inscriptions get the full API (filter, query, DataFrames, citation, export).

Each record needs an "id" and its text as one of:

"lines": a list of physical lines, each a list of tokens;
"words": a flat token list (treated as one line);
"text": a whitespace-tokenized string (one line).

A token is a string, or a dict {"text": …} with optional "kind" (a TokenKind value; inferred when omitted — numerals by parseability, the rest words), "status" (a ReadingStatus value), and "alt" (alternate readings). Hyphenated tokens get their signs split. Optional record keys: "meta" (site/period/scribe/support/findspot/ name), "translations". Example::

corpus = Corpus.from_records([
    {"id": "X1", "text": "KU-RO 10", "meta": {"site": "My site"}},
    {"id": "X2", "lines": [["A-DU", {"text": "5", "status": "unclear"}]]},
], script_id="lineara")

To make it loadable by name, register a loader: aegean.core.corpus.register_loader("myfind", lambda: corpus).

from_dict `classmethod` ¶

from_dict(data: dict[str, Any]) -> 'Corpus'

Reconstruct a Corpus from the dict to_json serializes (its json.loads).

Raises ValueError when the file records a schema version newer than this release understands (a file from a future pyaegean), naming the fix; a missing or older version loads normally.

to_sql ¶

to_sql(path: str | Path, *, fts: bool = True, append: bool = False) -> None

Write this corpus to a SQLite database (stdlib only). Documents and tokens become queryable rows with an optional FTS5 text index; provenance round-trips. With append=True the documents are upserted into an existing database (by id) instead of overwriting it. Reload with Corpus.from_sql; search with aegean.db.search.

from_sql `classmethod` ¶

from_sql(path: str | Path) -> 'Corpus'

Reconstruct a Corpus from a SQLite database written by to_sql — the lossless counterpart to from_json. For huge databases, aegean.db.stream yields documents one at a time instead of loading them all.

query ¶

query(filters: Sequence[FilterRow], output: Output = 'inscriptions', *, annotated_ids: set[str] | None = None) -> QueryResults

Run the compound-query predicate engine over this corpus.

filters is a sequence of aegean.analysis.FilterRow rows (a field id, a value, and optional connector/negate); output selects "inscriptions" or "words". Returns aegean.analysis.QueryResults (.inscriptions and .words) carrying this corpus's provenance and a summary of the filters, so results.cite() cites the exact result set. In .words each count is the word's document frequency (how many distinct inscriptions it occurs in), not its token frequency; for token counts use word_frequencies. The available fields are in aegean.analysis.FIELDS. Unlike filter (exact metadata match), this supports text/prefix/sign-pattern/co-occurrence predicates with AND/OR/NOT.

Document `dataclass` ¶

Document(id: str, script_id: str, tokens: list[Token], lines: list[list[int]], glyphs: str = '', transcription: str = '', translations: list[str] = list(), meta: DocumentMeta = DocumentMeta())

One inscription / tablet / text.

line_tokens `property` ¶

line_tokens: list[list[Token]]

Tokens regrouped by physical line.

DocumentMeta `dataclass` ¶

DocumentMeta(site: str = '', support: str = '', scribe: str = '', findspot: str = '', period: str = '', name: str = '', images: tuple[str, ...] = (), notes: tuple[str, ...] = ())

Bibliographic / archaeological metadata for a document.

ReadingStatus ¶

Bases: str, Enum

Editorial certainty of a token's reading (Leiden / EpiDoc conventions).

CERTAIN is the default. The others mark the apparatus an epigraphic edition must preserve — damaged, restored, or lost text. The bundled loaders decode each edition's apparatus into these statuses (the Leiden underdots, brackets, and erasure marks of the Cypriot and Linear A corpora); a bring-your-own EpiDoc corpus populates them from <unclear> / <supplied> / <gap> markup, and the EpiDoc writer emits them back.

Sign `dataclass` ¶

Sign(label: str, glyph: str | None = None, codepoint: int | None = None, phonetic: str | None = None, script_id: str = '', attrs: dict[str, Any] = dict())

One graphic unit of a script (syllabogram, letter, or logogram).

SignInventory ¶

SignInventory(signs: list[Sign], script_id: str = '')

The set of signs for a script, indexed by label / glyph / codepoint.

copy ¶

copy() -> 'SignInventory'

An independent copy: each Sign is rebuilt with its own attrs dict.

Sign is a frozen value object but its attrs is a mutable per-sign dict for user analysis; a shared/cached inventory (the @lru_cache-d *_inventory() accessors) would otherwise let one caller's attrs edit leak into every later reader and into a subsequent corpus load. Mirrors Corpus.copy for the sign layer.

Token `dataclass` ¶

Token(text: str, kind: TokenKind, signs: tuple[str, ...] = (), glyphs: str | None = None, line_no: int | None = None, position: int | None = None, status: ReadingStatus = CERTAIN, alt: tuple[str, ...] = (), annotations: dict[str, str] = dict())

One unit in a document's transliterated text stream.

TokenKind ¶

Bases: str, Enum

The role a token plays in a document's text stream.

Provenance `dataclass` ¶

Provenance(source: str, license: str = '', citation: str = '', url: str = '', schema_version: int = SCHEMA_VERSION, notes: tuple[str, ...] = tuple(), data_version: str = '', edition_fidelity: str = '')

Where a corpus came from and how to cite it.

cite ¶

cite() -> str

A one-line citation string for papers / logs.

bibtex ¶

bibtex(key: str = 'aegean-corpus') -> str

A BibTeX @misc entry for this source.

Best-effort formatting of the recorded free-text provenance: only fields actually known are emitted; the first year found in the citation string (if any) becomes year; the license and any provenance notes (e.g. the subset note Corpus.filter records) go into note.

apa ¶

apa() -> str

An APA-style reference line (n.d. when no year is recoverable).

Best-effort formatting of the recorded free-text provenance; notes (e.g. the subset note Corpus.filter records) follow in brackets.

Script ¶

Bases: ABC

A writing system the package can read and analyse.

sign_inventory `abstractmethod` `property` ¶

sign_inventory: SignInventory

The script's SignInventory.

tokenize `abstractmethod` ¶

tokenize(raw: str) -> list[Token]

Split a raw transliteration string into typed tokens.

register_loader ¶

register_loader(script_id: str, fn: Callable[[], 'Corpus']) -> None

Register a corpus loader so Corpus.load(script_id) / aegean.load(script_id) works.

get_script ¶

get_script(script_id: str) -> Script

Return the registered Script for script_id (raises KeyError if unknown).

register ¶

register(script: Script) -> None

Register a script plugin under its id (each built-in plugin calls this on import).

registered_scripts ¶

registered_scripts() -> list[str]

The sorted ids of all registered scripts, e.g. ['cypriot', 'cyprominoan', 'greek', 'lineara', 'linearb'].

aegean.core¶

core ¶

Corpus ¶

load classmethod ¶

get ¶

fingerprint ¶

cache_key ¶

copy ¶

filter ¶

subset ¶

merge ¶

cite ¶

iter_documents ¶

iter_tokens ¶

iter_words ¶

word_frequencies ¶

diagnose ¶

to_dataframe ¶

to_dict ¶

to_json ¶

from_json classmethod ¶

from_records classmethod ¶

from_dict classmethod ¶

to_sql ¶

from_sql classmethod ¶

query ¶

Document dataclass ¶

line_tokens property ¶

DocumentMeta dataclass ¶

ReadingStatus ¶

Sign dataclass ¶

SignInventory ¶

copy ¶

Token dataclass ¶

TokenKind ¶

Provenance dataclass ¶

cite ¶

bibtex ¶

apa ¶

Script ¶

sign_inventory abstractmethod property ¶

tokenize abstractmethod ¶

register_loader ¶

get_script ¶

register ¶

registered_scripts ¶

load `classmethod` ¶

from_json `classmethod` ¶

from_records `classmethod` ¶

from_dict `classmethod` ¶

from_sql `classmethod` ¶

Document `dataclass` ¶

line_tokens `property` ¶

DocumentMeta `dataclass` ¶

Sign `dataclass` ¶

Token `dataclass` ¶

Provenance `dataclass` ¶

sign_inventory `abstractmethod` `property` ¶

tokenize `abstractmethod` ¶