aegean¶

aegean ¶

pyaegean — a specialist Python toolkit for Ancient Greek and the Aegean syllabic scripts (Linear A, Linear B, Cypriot, and Cypro-Minoan).

A script-agnostic corpus data layer; a deep Greek NLP track (opt-in Perseus-treebank lemmas/POS, LSJ glossing, a dependency parser, an unseen-form POS tagger, edit-tree and neural lemmatizers, and a benchmark harness); a multi-provider AI/translation layer; and optional geographic mapping (aegean.geo, the [geo] extra). The core is dependency-free and imports instantly; the heavier backends are opt-in and fetched to cache.

Corpus ¶

Corpus(documents: list[Document], sign_inventory: SignInventory | None = None, provenance: Provenance | None = None, script_id: str = '')

A collection of Document s plus shared inventory + provenance.

load `classmethod` ¶

load(script_id: str, *, version: str | None = None) -> 'Corpus'

Load a registered corpus by name, e.g. Corpus.load("lineara") (bundled) or Corpus.load("damos") (fetched to the local data store on first use).

Loaders may cache one built instance per process (the bundled corpora do), so the result is returned as a copy: mutate it freely, and documents added, dropped, or edited never leak into later load calls.

version (optional) loads a kept historical release of a fetched corpus that the project still hosts, for reproducing an earlier analysis byte-for-byte (e.g. version="v1" for a pre-0.29.0 epigraphy corpus). It applies only to the corpora with kept historical pins (isicily, iip, iospe, igcyr, edh, ddbdp; see aegean.data.historical_versions) and fetches into a separate version-suffixed cache entry, leaving the current data untouched. The default (version=None) is unchanged.

get ¶

get(doc_id: str) -> Document | None

The document with id doc_id, or None if there is no such document.

fingerprint ¶

fingerprint() -> str

A stable content hash of this corpus, covering everything a token-level analysis can see: the script id, the provenance data_version, each document's id, every token's text, TokenKind, ReadingStatus, decomposed signs, Unicode glyphs, alternate readings (alt), and annotations (hashed as sorted key/value pairs), and any subset: / merged: / appended: provenance note. Document metadata (site, period, ...) is deliberately excluded. Cheap relative to the analyses it keys: one pass over the tokens, no model build. Two corpora with the same fingerprint have the same analysable content, so it's the cache key for aegean.cache-memoised analyses (including the sign-level dispersion/keyness that read signs).

cache_key ¶

cache_key() -> str

Alias for fingerprint, the protocol aegean.cache keys on.

copy ¶

copy() -> 'Corpus'

A structurally independent copy: a fresh document list and, per document, fresh token/line/translation containers, so mutating the copy (or the original) never affects the other. Token and Sign are frozen value objects but each carries a mutable per-element dict (annotations / attrs) for user analysis, so the copy rebuilds those with independent dicts, keeping a per-token annotation edit (or a per-sign attr edit) from leaking into the original, a sibling copy, or a later load of the same cached corpus. The remaining fields (DocumentMeta, Provenance, the immutable Token/Sign scalars) are shared. One pass over the tokens — a fraction of a second even for the largest corpora (the annotation-rich NT is the slowest); the copy fingerprints identically to the original.

filter ¶

filter(**meta: Any) -> 'Corpus'

Return a new Corpus whose documents match all given metadata fields (AND-combination), e.g. corpus.filter(site="HT", period="LMIB").

The subset's provenance records what was filtered (a subset: note), so cite on the result cites the exact subset used.

subset ¶

subset(ids: Iterable[str]) -> 'Corpus'

A new Corpus of just the documents whose id is in ids (original order kept).

The id-based counterpart to filter; records a subset: provenance note so cite on the result names the slice.

merge ¶

merge(*others: 'Corpus', dedupe: str = 'error') -> 'Corpus'

Merge this corpus with others into one (documents concatenated in order).

dedupe controls duplicate document ids across the inputs: "error" (raise, listing the collisions — the safe default), "first" / "last" (keep that occurrence), or "suffix" (rename later collisions id#2, id#3, …). The merged script_id is the common value, or "mixed" when the inputs differ; the sign inventory is the first input's when scripts agree, else None. A fresh provenance names every source, so cite on the result stays truthful.

See also aegean.combine, the module-level form that takes a list.

cite ¶

cite(style: str = 'plain') -> str

Cite this corpus — or the exact filtered subset — in one call.

style: "plain" (one line), "bibtex" (a @misc entry), or "apa". Filtered subsets (see filter) carry a subset: note that all three styles include, so the citation states exactly what was used.

iter_documents ¶

iter_documents() -> Iterator[Document]

Iterate documents (the explicit-name form of iter(corpus)).

iter_tokens ¶

iter_tokens() -> Iterator[Token]

Every Token, in document then in-document order — a memory-friendly stream that never builds an all-tokens list (useful on a large corpus).

iter_words ¶

iter_words() -> Iterator[str]

Every lexical (WORD) token's text, in order, lazily. The unit word_frequencies counts — stream it to feed your own Counter or a running statistic without materialising a list.

word_frequencies ¶

word_frequencies() -> list[tuple[str, int]]

(word, count) for every lexical word, sorted by descending count.

diagnose ¶

diagnose(level: str = 'quick') -> 'DiagnoseReport'

A descriptive corpus-health report: reading-status profile, provenance / citation completeness, Aegean accounting reconciliation, numeral anomalies, annotation review state, and (level="full") sign-frequency outliers.

Composes existing public machinery into one aegean.core.diagnose.DiagnoseReport (.print() / .to_markdown() / .to_dataframe()); every check degrades gracefully on a corpus it does not apply to. Descriptive, not a verdict.

to_dataframe ¶

to_dataframe(level: str = 'document')

A pandas DataFrame at document, token, or word level.

pandas is an optional dependency — install with pip install 'pyaegean[data]'.

to_dict ¶

to_dict() -> dict[str, Any]

A compact, lossy export (_meta + per-document words/metadata) for quick interop. For a complete, reversible serialization use to_json/from_json.

to_json ¶

to_json(path: str | Path | None = None, *, indent: int | None = 2) -> str | None

Serialize the whole corpus to JSON losslessly — every token (with its kind, signs, glyphs, line/position), the physical lines, full document metadata, the sign inventory, and provenance all survive. from_json reverses it exactly.

Returns the JSON string, or writes it to path and returns None when path is given. (Unlike to_dict, which is a compact lossy summary.)

from_json `classmethod` ¶

from_json(source: str | Path) -> 'Corpus'

Reconstruct a Corpus from to_json output: a JSON string, a Path to a .json file, or a path-like string (anything not beginning with {).

from_records `classmethod` ¶

from_records(records: Sequence[dict[str, Any]], *, script_id: str = 'custom', provenance: Provenance | None = None, sign_inventory: SignInventory | None = None) -> 'Corpus'

Build a corpus from plain dict records — your own inscriptions get the full API (filter, query, DataFrames, citation, export).

Each record needs an "id" and its text as one of:

"lines": a list of physical lines, each a list of tokens;
"words": a flat token list (treated as one line);
"text": a whitespace-tokenized string (one line).

A token is a string, or a dict {"text": …} with optional "kind" (a TokenKind value; inferred when omitted — numerals by parseability, the rest words), "status" (a ReadingStatus value), and "alt" (alternate readings). Hyphenated tokens get their signs split. Optional record keys: "meta" (site/period/scribe/support/findspot/ name), "translations". Example::

corpus = Corpus.from_records([
    {"id": "X1", "text": "KU-RO 10", "meta": {"site": "My site"}},
    {"id": "X2", "lines": [["A-DU", {"text": "5", "status": "unclear"}]]},
], script_id="lineara")

To make it loadable by name, register a loader: aegean.core.corpus.register_loader("myfind", lambda: corpus).

from_dict `classmethod` ¶

from_dict(data: dict[str, Any]) -> 'Corpus'

Reconstruct a Corpus from the dict to_json serializes (its json.loads).

Raises ValueError when the file records a schema version newer than this release understands (a file from a future pyaegean), naming the fix; a missing or older version loads normally.

to_sql ¶

to_sql(path: str | Path, *, fts: bool = True, append: bool = False) -> None

Write this corpus to a SQLite database (stdlib only). Documents and tokens become queryable rows with an optional FTS5 text index; provenance round-trips. With append=True the documents are upserted into an existing database (by id) instead of overwriting it. Reload with Corpus.from_sql; search with aegean.db.search.

from_sql `classmethod` ¶

from_sql(path: str | Path) -> 'Corpus'

Reconstruct a Corpus from a SQLite database written by to_sql — the lossless counterpart to from_json. For huge databases, aegean.db.stream yields documents one at a time instead of loading them all.

query ¶

query(filters: Sequence[FilterRow], output: Output = 'inscriptions', *, annotated_ids: set[str] | None = None) -> QueryResults

Run the compound-query predicate engine over this corpus.

filters is a sequence of aegean.analysis.FilterRow rows (a field id, a value, and optional connector/negate); output selects "inscriptions" or "words". Returns aegean.analysis.QueryResults (.inscriptions and .words) carrying this corpus's provenance and a summary of the filters, so results.cite() cites the exact result set. In .words each count is the word's document frequency (how many distinct inscriptions it occurs in), not its token frequency; for token counts use word_frequencies. The available fields are in aegean.analysis.FIELDS. Unlike filter (exact metadata match), this supports text/prefix/sign-pattern/co-occurrence predicates with AND/OR/NOT.

Document `dataclass` ¶

Document(id: str, script_id: str, tokens: list[Token], lines: list[list[int]], glyphs: str = '', transcription: str = '', translations: list[str] = list(), meta: DocumentMeta = DocumentMeta())

One inscription / tablet / text.

line_tokens `property` ¶

line_tokens: list[list[Token]]

Tokens regrouped by physical line.

DocumentMeta `dataclass` ¶

DocumentMeta(site: str = '', support: str = '', scribe: str = '', findspot: str = '', period: str = '', name: str = '', images: tuple[str, ...] = (), notes: tuple[str, ...] = ())

Bibliographic / archaeological metadata for a document.

Provenance `dataclass` ¶

Provenance(source: str, license: str = '', citation: str = '', url: str = '', schema_version: int = SCHEMA_VERSION, notes: tuple[str, ...] = tuple(), data_version: str = '', edition_fidelity: str = '')

Where a corpus came from and how to cite it.

cite ¶

cite() -> str

A one-line citation string for papers / logs.

bibtex ¶

bibtex(key: str = 'aegean-corpus') -> str

A BibTeX @misc entry for this source.

Best-effort formatting of the recorded free-text provenance: only fields actually known are emitted; the first year found in the citation string (if any) becomes year; the license and any provenance notes (e.g. the subset note Corpus.filter records) go into note.

apa ¶

apa() -> str

An APA-style reference line (n.d. when no year is recoverable).

Best-effort formatting of the recorded free-text provenance; notes (e.g. the subset note Corpus.filter records) follow in brackets.

ReadingStatus ¶

Bases: str, Enum

Editorial certainty of a token's reading (Leiden / EpiDoc conventions).

CERTAIN is the default. The others mark the apparatus an epigraphic edition must preserve — damaged, restored, or lost text. The bundled loaders decode each edition's apparatus into these statuses (the Leiden underdots, brackets, and erasure marks of the Cypriot and Linear A corpora); a bring-your-own EpiDoc corpus populates them from <unclear> / <supplied> / <gap> markup, and the EpiDoc writer emits them back.

Script ¶

Bases: ABC

A writing system the package can read and analyse.

sign_inventory `abstractmethod` `property` ¶

sign_inventory: SignInventory

The script's SignInventory.

tokenize `abstractmethod` ¶

tokenize(raw: str) -> list[Token]

Split a raw transliteration string into typed tokens.

Sign `dataclass` ¶

Sign(label: str, glyph: str | None = None, codepoint: int | None = None, phonetic: str | None = None, script_id: str = '', attrs: dict[str, Any] = dict())

One graphic unit of a script (syllabogram, letter, or logogram).

SignInventory ¶

SignInventory(signs: list[Sign], script_id: str = '')

The set of signs for a script, indexed by label / glyph / codepoint.

copy ¶

copy() -> 'SignInventory'

An independent copy: each Sign is rebuilt with its own attrs dict.

Sign is a frozen value object but its attrs is a mutable per-sign dict for user analysis; a shared/cached inventory (the @lru_cache-d *_inventory() accessors) would otherwise let one caller's attrs edit leak into every later reader and into a subsequent corpus load. Mirrors Corpus.copy for the sign layer.

Token `dataclass` ¶

Token(text: str, kind: TokenKind, signs: tuple[str, ...] = (), glyphs: str | None = None, line_no: int | None = None, position: int | None = None, status: ReadingStatus = CERTAIN, alt: tuple[str, ...] = (), annotations: dict[str, str] = dict())

One unit in a document's transliterated text stream.

TokenKind ¶

Bases: str, Enum

The role a token plays in a document's text stream.

DiagnoseReport `dataclass` ¶

DiagnoseReport(script_id: str, source: str, level: str, n_documents: int, n_tokens: int, reading_status: StatusProfile, provenance: ProvenanceProfile, accounting: AccountingProfile, numerals: NumeralProfile, review: ReviewProfile, signs: SignProfile)

A descriptive health report for one corpus. Sections are always present; a section that does not apply carries applicable=False and an explanatory note.

Render with print (rich table when installed, plain text otherwise), to_markdown (a shareable summary), or to_dataframe (needs the [data] extra).

to_markdown ¶

to_markdown() -> str

A shareable corpus-health summary in Markdown. Wherever a discrepancy is reported, the standard Aegean-metrology caveat travels with it.

to_text ¶

to_text() -> str

A plain-text rendering (the fallback used by print when rich is absent).

print ¶

print(*, console: Any = None) -> None

Render to the terminal: a rich table per section when rich is importable, a plain-text fallback otherwise (the core never requires rich).

to_dataframe ¶

to_dataframe()

A pandas DataFrame with one row per (section, check).

pandas is an optional dependency — install with pip install 'pyaegean[data]'.

set_verbosity ¶

set_verbosity(level: str | int) -> _Verbosity

Turn on pyaegean's library logging at level and route it to stderr.

level is "debug", "info", or "warning" (case-insensitive), or a logging level int (logging.INFO). Usable two ways:

as a plain call — aegean.set_verbosity("info") — which persists until changed;
as a context manager — with aegean.set_verbosity("debug"): ... — which restores the previous level (and removes the stderr handler if it added one) on exit.

The library attaches nothing and prints nothing until this is called (or PYAEGEAN_LOG is set); it never touches the root logger or logging.basicConfig, so it does not interfere with an application's own logging configuration.

Raises ValueError naming the valid levels for an unrecognized level.

get_script ¶

get_script(script_id: str) -> Script

Return the registered Script for script_id (raises KeyError if unknown).

register ¶

register(script: Script) -> None

Register a script plugin under its id (each built-in plugin calls this on import).

registered_scripts ¶

registered_scripts() -> list[str]

The sorted ids of all registered scripts, e.g. ['cypriot', 'cyprominoan', 'greek', 'lineara', 'linearb'].

load ¶

load(script_id: str, *, version: str | None = None) -> Corpus

Load a registered corpus by id, e.g. aegean.load("lineara").

The Aegean corpora and the Greek sample texts are bundled and load offline; the rest (nt, damos, sigla, the epigraphy corpora, ddbdp) fetch to the local data store on first use.

version (optional) loads a kept historical release of a fetched corpus the project still hosts, for reproducing an earlier analysis (e.g. aegean.load("isicily", version="v1") for the pre-0.29.0 I.Sicily data). It is accepted only for corpora with kept historical pins (isicily, iip, iospe, igcyr, edh, ddbdp); the default loads the current data.

Each call returns an independent copy (see Corpus.copy), so mutating the result never leaks into a later load() of the same corpus.

read_corpus ¶

read_corpus(spec: str) -> 'Corpus'

Resolve a corpus from spec, in this order of precedence:

"-" reads a JSON corpus from stdin; a string starting with { is parsed as inline JSON (the output of :meth:Corpus.to_json).
an exact registered id ("lineara", "damos", "nt", …) → the bundled or fetched corpus. A registered id always wins over a same-named file; case is forgiven as a last resort ("LINEARA" loads lineara when nothing else matches).
a Greek work id like "tlg0012.tlg001" → fetched from Perseus / First1KGreek via :func:aegean.greek.load_work (network on first use, then cached).
a file path: .json → :meth:Corpus.from_json; .db / .sqlite / .sqlite3 → :meth:Corpus.from_sql.

Raises :class:CorpusNotFound (a ValueError) listing the accepted forms when nothing matches, with a did-you-mean suggestion when the spec is close to a registered id. This is what aegean.load is to a single id, generalized to any source.

diagnose ¶

diagnose(corpus: 'Corpus', level: str = 'quick') -> DiagnoseReport

A descriptive health report for corpus.

level is "quick" (the default: reading status, provenance, accounting, numeral anomalies, review state) or "full" (adds the sign-frequency scan for Aegean corpora). Composes only existing public machinery; every check that does not apply to this corpus (a Greek prose corpus has no accounting) is marked not-applicable rather than raising.

combine ¶

combine(corpora: 'Iterable[Corpus]', *, dedupe: str = 'error') -> Corpus

Merge several corpora into one (see Corpus.merge).

corpora is any iterable of Corpus; dedupe handles duplicate document ids ("error" (default), "first", "last", or "suffix").

aegean¶

aegean ¶

Corpus ¶

load classmethod ¶

get ¶

fingerprint ¶

cache_key ¶

copy ¶

filter ¶

subset ¶

merge ¶

cite ¶

iter_documents ¶

iter_tokens ¶

iter_words ¶

word_frequencies ¶

diagnose ¶

to_dataframe ¶

to_dict ¶

to_json ¶

from_json classmethod ¶

from_records classmethod ¶

from_dict classmethod ¶

to_sql ¶

from_sql classmethod ¶

query ¶

Document dataclass ¶

line_tokens property ¶

DocumentMeta dataclass ¶

Provenance dataclass ¶

cite ¶

bibtex ¶

apa ¶

ReadingStatus ¶

Script ¶

sign_inventory abstractmethod property ¶

tokenize abstractmethod ¶

Sign dataclass ¶

SignInventory ¶

copy ¶

Token dataclass ¶

TokenKind ¶

DiagnoseReport dataclass ¶

to_markdown ¶

to_text ¶

print ¶

to_dataframe ¶

set_verbosity ¶

get_script ¶

register ¶

registered_scripts ¶

load ¶

read_corpus ¶

diagnose ¶

combine ¶

load `classmethod` ¶

from_json `classmethod` ¶

from_records `classmethod` ¶

from_dict `classmethod` ¶

from_sql `classmethod` ¶

Document `dataclass` ¶

line_tokens `property` ¶

DocumentMeta `dataclass` ¶

Provenance `dataclass` ¶

sign_inventory `abstractmethod` `property` ¶

tokenize `abstractmethod` ¶

Sign `dataclass` ¶

Token `dataclass` ¶

DiagnoseReport `dataclass` ¶