aegean.core¶

core ¶

Script-agnostic core: data model, script plugin contract, corpus, numerals.

Corpus ¶

Corpus(documents: list[Document], sign_inventory: SignInventory | None = None, provenance: Provenance | None = None, script_id: str = '')

A collection of Document s plus shared inventory + provenance.

load `classmethod` ¶

load(script_id: str, *, version: str | None = None) -> 'Corpus'

Load a registered corpus by name, e.g. Corpus.load("lineara") (bundled) or Corpus.load("damos") (fetched to the local data store on first use).

Loaders may cache one built instance per process (the bundled corpora do), so the result is returned as a copy: mutate it freely, and documents added, dropped, or edited never leak into later load calls.

version (optional) loads a kept historical release of a fetched corpus that the project still hosts, for reproducing an earlier analysis byte-for-byte (e.g. version="v1" for a pre-0.29.0 epigraphy corpus). It applies only to the corpora with kept historical pins (isicily, iip, iospe, igcyr, edh, ddbdp; see aegean.data.historical_versions) and fetches into a separate version-suffixed cache entry, leaving the current data untouched. The default (version=None) is unchanged.

get ¶

get(doc_id: str) -> Document | None

The document with id doc_id, or None if there is no such document.

fingerprint ¶

fingerprint() -> str

A stable content hash of this corpus, covering everything a token-level analysis can see: the script id, the dataset identity (the provenance data_version, when it names the data rather than a package release), each document's id, every token's text, TokenKind, ReadingStatus, decomposed signs, Unicode glyphs, alternate readings (alt), and annotations (hashed as sorted key/value pairs), and any subset: / merged: / appended: provenance note. Document metadata (site, period, ...) and the installed package version are deliberately excluded. Cheap relative to the analyses it keys: one pass over the tokens, no model build. Two corpora with the same fingerprint have the same analysable content, so it's the cache key for aegean.cache-memoised analyses (including the sign-level dispersion/keyness that read signs).

cache_key ¶

cache_key() -> str

Alias for fingerprint, the protocol aegean.cache keys on.

copy ¶

copy() -> 'Corpus'

A structurally independent copy: a fresh document list and, per document, fresh token/line/translation containers, so mutating the copy (or the original) never affects the other. Token and Sign are frozen value objects but each carries a mutable per-element dict (annotations / attrs) for user analysis, so the copy rebuilds those with independent dicts, keeping a per-token annotation edit (or a per-sign attr edit) from leaking into the original, a sibling copy, or a later load of the same cached corpus. The remaining fields (DocumentMeta, Provenance, the immutable Token/Sign scalars) are shared. One pass over the tokens — a fraction of a second even for the largest corpora (the annotation-rich NT is the slowest); the copy fingerprints identically to the original.

filter ¶

filter(**meta: Any) -> 'Corpus'

Return a new Corpus whose documents match all given metadata fields (AND-combination), e.g. corpus.filter(site="HT", period="LMIB").

A name that is not a DocumentMeta field raises ValueError listing the fields that are. A mistyped name would otherwise match nothing and return an empty corpus whose provenance note reads exactly like a real filter, leaving "no documents match" indistinguishable from "that field does not exist".

The subset's provenance records what was filtered (a subset: note), so cite on the result cites the exact subset used.

subset ¶

subset(ids: Iterable[str]) -> 'Corpus'

A new Corpus of just the documents whose id is in ids (original order kept).

The id-based counterpart to filter; records a subset: provenance note so cite on the result names the slice.

merge ¶

merge(*others: 'Corpus', dedupe: str = 'error') -> 'Corpus'

Merge this corpus with others into one (documents concatenated in order).

dedupe controls duplicate document ids across the inputs: "error" (raise, listing the collisions — the safe default), "first" / "last" (keep that occurrence), or "suffix" (rename later collisions id#2, id#3, …). The merged script_id is the common value, or "mixed" when the inputs differ; the sign inventory is the first input's when scripts agree, else None. A fresh provenance names every source, so cite on the result stays truthful.

See also aegean.combine, the module-level form that takes a list.

cite ¶

cite(style: str = 'plain') -> str

Cite this corpus — or the exact filtered subset — in one call.

style: "plain" (one line), "bibtex" (a @misc entry), or "apa". Filtered subsets (see filter) carry a subset: note that all three styles include, so the citation states exactly what was used.

iter_documents ¶

iter_documents() -> Iterator[Document]

Iterate documents (the explicit-name form of iter(corpus)).

iter_tokens ¶

iter_tokens() -> Iterator[Token]

Every Token, in document then in-document order — a memory-friendly stream that never builds an all-tokens list (useful on a large corpus).

iter_words ¶

iter_words() -> Iterator[str]

Every lexical (WORD) token's text, in order, lazily. The unit word_frequencies counts — stream it to feed your own Counter or a running statistic without materialising a list.

word_frequencies ¶

word_frequencies() -> list[tuple[str, int]]

(word, count) for every lexical word, sorted by descending count.

diagnose ¶

diagnose(level: str = 'quick') -> 'DiagnoseReport'

A descriptive corpus-health report: reading-status profile, provenance / citation completeness, Aegean accounting reconciliation, numeral anomalies, annotation review state, and (level="full") sign-frequency outliers.

Composes existing public machinery into one aegean.core.diagnose.DiagnoseReport (.print() / .to_markdown() / .to_dataframe()); every check degrades gracefully on a corpus it does not apply to. Descriptive, not a verdict.

to_dataframe ¶

to_dataframe(level: str = 'document')

A pandas DataFrame at document, token, or word level.

Token and word rows lead with the identity columns (doc_id, line_no, position, text, kind, status, site, period), then the token's own annotations. The typed form_* and alignment_* blocks follow, and each is present only when at least one exported token carries that state, so a corpus that records neither does not carry 25 empty columns. A canonical column keeps its own value when an annotation shares its name.

pandas is an optional dependency — install with pip install 'pyaegean[data]'.

to_dict ¶

to_dict() -> dict[str, Any]

A compact, lossy export (_meta + per-document words/metadata) for quick interop. For a complete, reversible serialization use to_json/from_json.

to_json ¶

to_json(path: str | Path | None = None, *, indent: int | None = 2) -> str | None

Serialize the whole corpus to JSON losslessly — every token (with its kind, signs, glyphs, line/position), the physical lines, full document metadata, the sign inventory, and provenance all survive. from_json reverses it exactly.

Returns the JSON string, or writes it to path and returns None when path is given. (Unlike to_dict, which is a compact lossy summary.)

from_json `classmethod` ¶

from_json(source: str | Path) -> 'Corpus'

Reconstruct a Corpus from to_json output: a JSON string, a Path to a .json file, or a path-like string.

A string that opens with { or [ is read as JSON, and otherwise as a filename. Those two forms overlap — "[draft].json" is a legal relative path — so a string that opens like JSON but does not parse is read as a path when a file of that name exists; a payload that is merely malformed keeps reporting its own decode error, since no such file exists.

Files are read as utf-8-sig, so a corpus file saved with a leading UTF-8 BOM (an editor or a Windows tool that adds one) loads like any other UTF-8 file; a BOM on a JSON string is stripped for the same reason. A malformed file raises ValueError naming what is wrong (see from_dict).

from_records `classmethod` ¶

from_records(records: Sequence[dict[str, Any]], *, script_id: str = 'custom', provenance: Provenance | None = None, sign_inventory: SignInventory | None = None) -> 'Corpus'

Build a corpus from plain dict records — your own inscriptions get the full API (filter, query, DataFrames, citation, export).

Each record needs an "id" and its text as one of:

"lines": a list of physical lines, each a list of tokens;
"words": a flat token list (treated as one line);
"text": a whitespace-tokenized string (one line).

A token is a string, or a dict {"text": …} with optional "kind" (a TokenKind value; inferred when omitted — numerals by parseability, the rest words), "status" (a ReadingStatus value), and "alt" (alternate readings). Hyphenated tokens get their signs split. Optional record keys: "meta" (site/period/scribe/support/findspot/ name), "translations". Example::

corpus = Corpus.from_records([
    {"id": "X1", "text": "KU-RO 10", "meta": {"site": "My site"}},
    {"id": "X2", "lines": [["A-DU", {"text": "5", "status": "unclear"}]]},
], script_id="lineara")

To make it loadable by name, register a loader: aegean.core.corpus.register_loader("myfind", lambda: corpus).

from_dict `classmethod` ¶

from_dict(data: dict[str, Any]) -> 'Corpus'

Reconstruct a Corpus from the dict to_json serializes (its json.loads).

This is the interchange entry point for every saved corpus, so a source that cannot be read is reported, never half-loaded: a ValueError names the document and the field at fault (document 'HT 13', token 4 is missing 'kind') instead of surfacing a bare KeyError key name.

Raises ValueError when the file records a schema version newer than this release understands (a file from a future pyaegean), naming the fix; a missing version loads as the pre-form-state schema.

to_sql ¶

to_sql(path: str | Path, *, fts: bool = True, append: bool = False) -> None

Write this corpus to a SQLite database (stdlib only). Documents and tokens become queryable rows with an optional FTS5 text index; provenance round-trips. With append=True the documents are upserted into an existing database (by id) instead of overwriting it. Reload with Corpus.from_sql; search with aegean.db.search.

from_sql `classmethod` ¶

from_sql(path: str | Path) -> 'Corpus'

Reconstruct a Corpus from a SQLite database written by to_sql — the lossless counterpart to from_json. For huge databases, aegean.db.stream yields documents one at a time instead of loading them all.

query ¶

query(filters: Sequence[FilterRow], output: Output = 'inscriptions', *, annotated_ids: set[str] | None = None) -> QueryResults

Run the compound-query predicate engine over this corpus.

filters is a sequence of aegean.analysis.FilterRow rows (a field id, a value, and optional connector/negate); output selects "inscriptions" or "words". Returns aegean.analysis.QueryResults (.inscriptions and .words) carrying this corpus's provenance and a summary of the filters, so results.cite() cites the exact result set. In .words each count is the word's document frequency (how many distinct inscriptions it occurs in), not its token frequency; for token counts use word_frequencies. The available fields are in aegean.analysis.FIELDS. Unlike filter (exact metadata match), this supports text/prefix/sign-pattern/co-occurrence predicates with AND/OR/NOT.

Document `dataclass` ¶

Document(id: str, script_id: str, tokens: list[Token], lines: list[list[int]], glyphs: str = '', transcription: str = '', translations: list[str] = list(), meta: DocumentMeta = DocumentMeta(), source_text: str | None = None)

One inscription / tablet / text.

line_tokens `property` ¶

line_tokens: list[list[Token]]

Tokens regrouped by physical line.

validate_source_alignment ¶

validate_source_alignment() -> None

Validate all token mappings against this document's exact source.

Legacy documents (no source snapshot and no alignments) remain valid. A source-bearing document is intentionally all-or-nothing: every token must have a mapping, mappings must point to this document, be ordered without overlap, use unique IDs, and record the exact whitespace gap preceding each source slice.

DocumentMeta `dataclass` ¶

DocumentMeta(site: str = '', support: str = '', scribe: str = '', findspot: str = '', period: str = '', name: str = '', images: tuple[str, ...] = (), notes: tuple[str, ...] = ())

Bibliographic / archaeological metadata for a document.

FormSegment `dataclass` ¶

FormSegment(text: str, status: ReadingStatus = CERTAIN, source_ref: SourceMarkupRef | None = None)

One ordered piece of a token form and its editorial reading status.

An empty segment is permitted only for ReadingStatus.LOST. That models a lacuna without inventing replacement text; non-empty restored text remains visibly marked as editor-supplied by its status and optional source reference.

role `property` ¶

role: ReadingStatus

Compatibility spelling for code that calls the status a segment role.

source `property` ¶

source: SourceMarkupRef | None

Compatibility alias for source_ref.

ReadingStatus ¶

Bases: str, Enum

Editorial certainty of a token's reading (Leiden / EpiDoc conventions).

CERTAIN is the default. The others mark the apparatus an epigraphic edition must preserve — damaged, restored, or lost text. The bundled loaders decode each edition's apparatus into these statuses (the Leiden underdots, brackets, and erasure marks of the Cypriot and Linear A corpora); a bring-your-own EpiDoc corpus populates them from <unclear> / <supplied> / <gap> markup, and the EpiDoc writer emits them back.

Sign `dataclass` ¶

Sign(label: str, glyph: str | None = None, codepoint: int | None = None, phonetic: str | None = None, script_id: str = '', attrs: dict[str, Any] = dict())

One graphic unit of a script (syllabogram, letter, or logogram).

SignInventory ¶

SignInventory(signs: list[Sign], script_id: str = '')

The set of signs for a script, indexed by label / glyph / codepoint.

copy ¶

copy() -> 'SignInventory'

An independent copy: each Sign is rebuilt with its own attrs dict.

Sign is a frozen value object but its attrs is a mutable per-sign dict for user analysis; a shared/cached inventory (the @lru_cache-d *_inventory() accessors) would otherwise let one caller's attrs edit leak into every later reader and into a subsequent corpus load. Mirrors Corpus.copy for the sign layer.

SourceAlignment `dataclass` ¶

SourceAlignment(document_id: str, sentence_id: str | None, source_token_id: str, original_text: str, start_char: int, end_char: int, whitespace_before: str, normalized_text: str, normalization_ops: tuple[str, ...] = ())

Lossless provenance for one token in an exact source snapshot.

Character positions are half-open Python string offsets, not encoded-byte positions. original_text is the exact source slice and normalized_text is the tokenizer's normalized view of that slice. For a typed editorial token, :class:TokenFormState separately records the later regularized/normalized selection and the exact value handed to an analyzer. The alignment value is intentionally immutable so annotation and export code cannot silently alter the source mapping after it has been created.

validate_source ¶

validate_source(source_text: str, document_id: str | None = None) -> None

Validate the owning document and exact source slice.

ValueError identifies either a document mismatch or a changed source snapshot. This is deliberately a strict check: a mapping must never be projected against a merely similar normalized string.

SourceMarkupRef `dataclass` ¶

SourceMarkupRef(source_id: str, path: str, tag: str, attrs: tuple[tuple[str, str], ...] = ())

A semantic reference to an element in a source edition.

This is deliberately not a raw XML-byte or offset claim. path is a semantic locator supplied by the importer. The EpiDoc reader uses unique document-order labels such as "reg[17]"; the locator is stable for that parsed source tree, not across arbitrary edits to the XML. attrs preserves source attribute order for faithful diagnostics.

attributes `property` ¶

attributes: tuple[tuple[str, str], ...]

Alias for attrs used by callers that prefer the expanded name.

to_dict ¶

to_dict() -> dict[str, Any]

Return the canonical JSON-safe representation.

from_dict `classmethod` ¶

from_dict(value: dict[str, Any]) -> 'SourceMarkupRef'

Decode and validate one canonical source reference mapping.

Token `dataclass` ¶

Token(text: str, kind: TokenKind, signs: tuple[str, ...] = (), glyphs: str | None = None, line_no: int | None = None, position: int | None = None, status: ReadingStatus = CERTAIN, alt: tuple[str, ...] = (), annotations: dict[str, str] = dict(), alignment: SourceAlignment | None = None, form_state: TokenFormState | None = None)

One unit in a document's transliterated text stream.

TokenFormState `dataclass` ¶

TokenFormState(diplomatic: str, regularized: str | None = None, normalized: str | None = None, model_input: str | None = None, segments: tuple[FormSegment, ...] = (), model_input_ops: tuple[str, ...] = (), model_input_source: Literal['diplomatic', 'regularized', 'normalized', 'explicit'] | None = None)

Lossless form states for one token, independent of display Token.text.

diplomatic is the explicitly supplied/original form. regularized and normalized are optional editorial/model forms; model_input records the exact string actually sent to a model. model_input_source is a constrained provenance label rather than an unverifiable free-form claim. segments retain ordered supplied/unclear/lost pieces and may encode source choices, so their concatenation is intentionally not required to equal diplomatic.

original `property` ¶

original: str

Alias for the explicit diplomatic/original form.

model_input_provenance `property` ¶

model_input_provenance: str | None

Read-only alias for the constrained model_input_source label.

operations `property` ¶

operations: tuple[str, ...]

Read-only alias for the ordered model-input operations.

editorial_status `property` ¶

editorial_status: ReadingStatus

Most severe segment status, ordered LOST > RESTORED > UNCLEAR > CERTAIN.

status `property` ¶

status: ReadingStatus

Compatibility alias for the derived editorial status.

TokenKind ¶

Bases: str, Enum

The role a token plays in a document's text stream.

Provenance `dataclass` ¶

Provenance(source: str, license: str = '', citation: str = '', url: str = '', schema_version: int = SCHEMA_VERSION, notes: tuple[str, ...] = tuple(), data_version: str = '', edition_fidelity: str = '')

Where a corpus came from and how to cite it.

cite ¶

cite() -> str

A one-line citation string for papers / logs.

bibtex ¶

bibtex(key: str = 'aegean-corpus') -> str

A BibTeX @misc entry for this source.

Best-effort formatting of the recorded free-text provenance: only fields actually known are emitted; the first year found in the citation string (if any) becomes year; the license and any provenance notes (e.g. the subset note Corpus.filter records) go into note.

apa ¶

apa() -> str

An APA-style reference line (n.d. when no year is recoverable).

Best-effort formatting of the recorded free-text provenance; notes (e.g. the subset note Corpus.filter records) follow in brackets.

Script ¶

Bases: ABC

A writing system the package can read and analyse.

sign_inventory `abstractmethod` `property` ¶

sign_inventory: SignInventory

The script's SignInventory.

tokenize `abstractmethod` ¶

tokenize(raw: str) -> list[Token]

Split a raw transliteration string into typed tokens.

register_loader ¶

register_loader(script_id: str, fn: Callable[[], 'Corpus']) -> None

Register a corpus loader so Corpus.load(script_id) / aegean.load(script_id) works.

get_script ¶

get_script(script_id: str) -> Script

Return the registered Script for script_id (raises KeyError if unknown).

register ¶

register(script: Script) -> None

Register a script plugin under its id (each built-in plugin calls this on import).

registered_scripts ¶

registered_scripts() -> list[str]

The sorted ids of all registered scripts, e.g. ['cypriot', 'cyprominoan', 'greek', 'lineara', 'linearb'].

aegean.core¶

core ¶

Corpus ¶

load classmethod ¶

get ¶

fingerprint ¶

cache_key ¶

copy ¶

filter ¶

subset ¶

merge ¶

cite ¶

iter_documents ¶

iter_tokens ¶

iter_words ¶

word_frequencies ¶

diagnose ¶

to_dataframe ¶

to_dict ¶

to_json ¶

from_json classmethod ¶

from_records classmethod ¶

from_dict classmethod ¶

to_sql ¶

from_sql classmethod ¶

query ¶

Document dataclass ¶

line_tokens property ¶

validate_source_alignment ¶

DocumentMeta dataclass ¶

FormSegment dataclass ¶

role property ¶

source property ¶

ReadingStatus ¶

Sign dataclass ¶

SignInventory ¶

copy ¶

SourceAlignment dataclass ¶

validate_source ¶

SourceMarkupRef dataclass ¶

attributes property ¶

to_dict ¶

from_dict classmethod ¶

Token dataclass ¶

TokenFormState dataclass ¶

original property ¶

model_input_provenance property ¶

operations property ¶

editorial_status property ¶

status property ¶

TokenKind ¶

Provenance dataclass ¶

cite ¶

bibtex ¶

apa ¶

Script ¶

sign_inventory abstractmethod property ¶

tokenize abstractmethod ¶

register_loader ¶

get_script ¶

register ¶

registered_scripts ¶

load `classmethod` ¶

from_json `classmethod` ¶

from_records `classmethod` ¶

from_dict `classmethod` ¶

from_sql `classmethod` ¶

Document `dataclass` ¶

line_tokens `property` ¶

DocumentMeta `dataclass` ¶

FormSegment `dataclass` ¶

role `property` ¶

source `property` ¶

Sign `dataclass` ¶

SourceAlignment `dataclass` ¶

SourceMarkupRef `dataclass` ¶

attributes `property` ¶

from_dict `classmethod` ¶

Token `dataclass` ¶

TokenFormState `dataclass` ¶

original `property` ¶

model_input_provenance `property` ¶

operations `property` ¶

editorial_status `property` ¶

status `property` ¶

Provenance `dataclass` ¶

sign_inventory `abstractmethod` `property` ¶

tokenize `abstractmethod` ¶