aegean.io¶
io ¶
I/O adapters — move the corpus model to and from interchange formats.
Import your own material with from_text / from_text_file / from_text_dir /
from_csv (plain text, a folder of texts, or a CSV → a Corpus), or from_epidoc
(any EpiDoc TEI edition → a Corpus); export with the CSV/Parquet/EpiDoc writers. The
Linear B-specific EpiDoc reader (DAMOS-style files, text-derived Aegean token kinds) lives
in aegean.scripts.linearb and Corpus.load("linearb"). For pyaegean's own lossless
archive format, use Corpus.to_json / Corpus.from_json.
MergedReview
dataclass
¶
MergedReview(rows: tuple[dict[str, str], ...], conflicts: tuple[ReviewConflict, ...], reviewers: tuple[str, ...], source_paths: tuple[str, ...] = tuple())
The result of merging several reviewers' corrected copies of one export.
rows is the clean, agreed subset as review-table rows (ready to write or to apply with
apply_merged); conflicts are the disagreements held back for a human; reviewers is
every reviewer whose corrections are in rows; source_paths records the merged
tables. Use to_csv to write the agreed subset back out as a review table.
to_csv ¶
Write the agreed (clean) subset as a review CSV (UTF-8 BOM, formula-guarded).
ReviewConflict
dataclass
¶
ReviewConflict(doc_id: str, position: int, token: str, field: str, options: tuple[ReviewerValue, ...])
One field of one token where reviewers proposed different corrections.
field is the human field name (lemma / pos / morph); options lists each
reviewer's proposed value (and note). A conflict is never resolved silently — it is surfaced
for a human to settle.
ReviewerValue
dataclass
¶
One reviewer's proposed value for a field (with their note, if any).
from_epidoc ¶
from_epidoc(source: str | Path, *, script_id: str = 'greek') -> Corpus
Load EpiDoc TEI (a file or a directory of *.xml) into a Corpus.
The inverse of write_epidoc: round-trips the id, find-place, token/line stream,
editorial certainty, and alternate readings. script_id labels the corpus
(default "greek"). pyaegean parses your files locally and never re-hosts them.
read_epidoc ¶
read_epidoc(source: str | Path, *, script_id: str = 'greek') -> list[Document]
Parse an EpiDoc TEI file — or a directory of *.xml files — into Documents.
script_id labels the result: EpiDoc's xml:lang can't disambiguate (say) Linear A
from Cypro-Minoan, so the caller names the script. Uses the stdlib XML parser only.
Raises FileNotFoundError if source does not exist (or a directory holds no
*.xml files), and ValueError if nothing in it is EpiDoc (no <div type="edition">
or <body> in the TEI namespace) — rather than silently returning an empty list. A
malformed file inside a directory raises an xml.etree.ElementTree.ParseError whose
message names the offending file, so a single bad inscription in a large corpus folder
is identifiable (a directory has no line/column of its own).
to_epidoc ¶
to_epidoc(document: Document) -> str
Serialize a single Document to an EpiDoc TEI XML string.
The transliteration lives in a TEI <div type="edition"> (EpiDoc's required edition
division), as <lb/>-delimited lines of <w>/<num>/<g> tokens. A token whose
aegean.ReadingStatus is not CERTAIN is wrapped in the matching EpiDoc apparatus
element (<unclear> or <supplied>), so editorial certainty survives the round trip
through aegean.scripts.linearb.parse_epidoc. The output validates against the EpiDoc
RelaxNG schema (see tests/test_io.py).
Token text round-trips subject to standard XML text normalization: a carriage return becomes a line feed, leading/trailing whitespace on a token is trimmed, and a token (or alternate reading) whose text is only whitespace does not survive the parse. Real transliteration tokens are never whitespace-only, so this affects only synthetic input; the transliteration content itself is preserved.
Token.annotations (lemma, morphology, evidence class, review stamps) are NOT
serialized to EpiDoc — the format carries the edition text and its apparatus, not an
analysis layer. Use Corpus.to_json or aegean.db.to_sqlite for the full record.
write_epidoc ¶
Write EpiDoc TEI XML to disk.
A single Document is written to the file path; a
Corpus is written as one {id}.xml file per document into the
directory path (created if needed) — the layout
aegean.scripts.linearb.parse_epidoc reads back. Ids are sanitized for
the filesystem (anything outside [A-Za-z0-9-_.] becomes _), which
can conflate distinct ids: when two ids sanitize to the same filename, the
later ones (in id order) get a -2, -3, ... suffix and a warning
names the colliding ids, so no document silently overwrites another.
Token.annotations are not serialized (see to_epidoc).
to_rdf ¶
to_rdf(corpus: Corpus, path: str | Path, *, fmt: str = 'turtle', base_uri: str | None = None) -> None
Write corpus to disk as Linked Open Data.
fmt is "turtle" (aliases "ttl") or "jsonld" (alias "json-ld"). Each
document becomes a subject with a stable URI minted from its authoritative identifiers
(a papyri.info DDbDP document URI, else Trismegistos / I.Sicily / a base_uri fragment;
see the module docstring for the priority order and the DDbDP map), typed
dctype:Text, carrying its title, identifiers, the corpus license (dcterms:license,
NonCommercial included), source, place / date, and its reading text as an rdf:value
literal (language-tagged grc for the Greek corpora). base_uri defaults to the
non-resolvable urn:aegean: namespace.
The write is atomic (temp file + os.replace), so a failed or interrupted write never
truncates a prior export. Raises ValueError for an unknown fmt, or for a base_uri
that cannot appear literally in an IRI (a space, a control character, or an IRIREF-forbidden
character) since that would make the Turtle and JSON-LD subjects disagree.
RDF is an export only: there is no reader and no round-trip guarantee (use
aegean.core.corpus.Corpus.to_json / aegean.db.to_sqlite for lossless persistence).
apply_merged ¶
apply_merged(merged: MergedReview, corpus: 'Corpus') -> 'Corpus'
Land a MergedReview's agreed corrections onto corpus, returning a NEW corpus.
Runs the agreed (clean) subset through the same apply core as from_review_table, so every
guard still fires (each row's token text is verified against the corpus; a wrong-corpus
mismatch or an orphaned row raises). Each corrected field keeps the machine value under
<field>__pred; every contributing reviewer is stamped on the token (reviewed_by) and
listed in the review: provenance note, which records that it came from a merge.
from_review_table ¶
Read reviewer corrections from path back onto corpus, returning a NEW corpus.
Rows are matched to tokens by doc_id + position, and each matched row's exported
token text is verified against the token it matched: a mismatch (the corpus changed
between export and apply, or the wrong corpus was passed) raises ValueError naming the
rows rather than silently landing a correction on the wrong word. Duplicate rows for one
token with conflicting corrections, corrections whose row matches no token, and a
malformed CSV also raise ValueError.
For each row whose correct_* differs from the machine value the reviewer saw (the
row's own pred_* cell, falling back to the token's current annotation), the token's
annotation for that field is set to the corrected value, the machine value is preserved
under <field>__pred, and the token is stamped reviewed_by /
review_status="corrected" (plus review_note when the reviewer left one). A
morphology correction lands on the same key that supplied the displayed prediction
(morph or UD feats). Rows left blank change nothing. The stamped reviewer is
reviewer when given, else each row's own reviewer column. A review: provenance
note records how many tokens were corrected. The input corpus is not mutated.
merge_review_tables ¶
merge_review_tables(paths: Iterable[str | Path], corpus: 'Corpus', *, on_conflict: str = 'error') -> MergedReview
Merge several reviewers' corrected copies of the SAME review export.
Each table in paths is a corrected copy of one to_review_table export. Corrections the
reviewers agree on (or that only one reviewer made) are combined into MergedReview.rows;
where two reviewers give different values for the same field of the same token, the
disagreement is surfaced as a ReviewConflict and never silently resolved. With
on_conflict="error" any conflict raises ValueError listing them all; with
on_conflict="report" the conflicts are returned in MergedReview.conflicts and the
agreed subset stays applicable (apply_merged).
Reviewer identity comes from each table's reviewer column (or, when blank, the file
name). The tables must be copies of one export: a table whose token text disagrees at a
shared (doc_id, position) raises (a wrong-corpus mix-up), as does a reviewer name that
appears in more than one table (identities must be distinct to attribute a conflict).
corpus is used to verify the export's shape; the corrections are landed by
apply_merged.
needs_review_flag ¶
Whether a token's annotation should be verified by a human.
True when its evidence class (annotations[source_key], e.g. from
aegean.greek.annotate.annotate_corpus) is a low-confidence class (identity /
unresolved); failing that, when lemma_known is the string "false". A token
that carries neither signal (for example a gold-annotated corpus) is not flagged.
to_review_table ¶
to_review_table(corpus: 'Corpus', path: str | Path, *, source_key: str = 'lemma_source', only_needs_review: bool = False, reviewer: str = '') -> int
Write one reviewable row per WORD token of corpus to path (CSV, UTF-8 BOM).
Each row carries the token's identity (doc_id/position/line_no/ref), the
machine pred_lemma/pred_pos/pred_morph from its annotations, the
evidence_class and a needs_review flag, the corpus citation, and blank
correct_* / reviewer_note columns for the reviewer. With only_needs_review only
the flagged rows are written. Pass reviewer to pre-stamp every row's reviewer
column (hand a named copy to each reviewer when the corrected copies will be merged with
merge_review_tables); it is left blank by default. Returns the number of rows written.
A token without a position is not exported: the apply join key is
doc_id + position, so a correction on such a row could never be applied. Cells
that would open as a live formula in a spreadsheet are neutralized with a leading
apostrophe (stripped again by from_review_table).
to_csv ¶
to_csv(corpus: Corpus, path: str | Path, *, level: str = 'document', progress: Callable[[int, int], None] | None = None) -> None
Write the corpus's level DataFrame ("document"/"token"/"word") to CSV.
progress, when given, is called progress(done, total) once per document as the
rows are generated (total is the document count) so a very large export is not
silent; the default (None) is the byte-identical original path.
to_parquet ¶
to_parquet(corpus: Corpus, path: str | Path, *, level: str = 'document', progress: Callable[[int, int], None] | None = None) -> None
Write the corpus's level DataFrame to Parquet (needs a parquet engine).
progress (progress(done, total) per document, total = document count) covers
the row-generation phase only: Parquet buffers the whole DataFrame before its single
write call, so the final progress call lands at (total, total) and the write follows.
The default (None) is the byte-identical original path.
from_csv ¶
from_csv(path: str | Path, *, text_col: str = 'text', id_col: str | None = None, script_id: str = 'greek', meta_cols: Sequence[str] = (), encoding: str = 'utf-8-sig') -> 'Corpus'
Build a Corpus from a CSV file. text_col holds each row's text; id_col
(optional) holds its document id (otherwise ids are <stem>:<row>). meta_cols
names columns to carry into document metadata (recognized: site/period/scribe/support/
findspot/name). Raises ValueError if text_col is absent.
The default utf-8-sig encoding transparently strips a leading UTF-8 BOM (Excel
writes one), so the first column name is not silently prefixed with it; it reads a
BOM-less UTF-8 file identically.
from_text ¶
from_text(text: str, *, script_id: str = 'greek', doc_id: str = 'text', split: str = 'whole', meta: dict[str, str] | None = None) -> 'Corpus'
Build a Corpus from a raw string.
split controls how the text becomes documents: "whole" (default, one document),
"paragraph" (one per blank-line-separated block), or "line" (one per line).
Line breaks are preserved as physical lines. script_id picks the tokenizer ("greek"
by default). Raises ValueError if the text has no content.
from_text_dir ¶
from_text_dir(path: str | Path, *, script_id: str = 'greek', glob: str = '*.txt', split: str = 'whole', encoding: str = 'utf-8') -> 'Corpus'
Build one Corpus from a folder of text files (one or more documents per file,
per split). Document ids come from each file's stem (de-duplicated with a #n
suffix on collision). Raises NotADirectoryError / FileNotFoundError as appropriate.
from_text_file ¶
from_text_file(path: str | Path, *, script_id: str = 'greek', split: str = 'whole', doc_id: str | None = None, encoding: str = 'utf-8', meta: dict[str, str] | None = None) -> 'Corpus'
Build a Corpus from a plain-text file. The document id defaults to the file's
stem. See :func:from_text for split. Raises FileNotFoundError if missing.
from_workbench_export ¶
from_workbench_export(source: str | Path | dict[str, Any] | list[Any], *, script_id: str = 'lineara') -> Corpus
Load a workbench corpus export into a Corpus.
source is a path to a JSON file, a JSON string, or already-parsed
JSON. Both forms the workbench produces are accepted: the schema-v1
export object (records under "inscriptions", provenance under
"_meta", per-record "derived" analyses — ignored here) and a
plain array of inscription records. script_id defaults to lineara
(the workbench's own corpus); pass the real script when re-importing an
export of some other corpus, so the documents are not rebranded.
The workbench schema carries token text only: any editorial
ReadingStatus or Token.annotations the original corpus had were not
in the export, so every re-imported token is CERTAIN and unannotated
(see to_workbench).
Token kinds are inferred the Corpus.from_records way (numerals by
parseability, everything else a word); glyphs, transcription, and image
references are carried onto the documents. The export's own metadata
(app version, generation time, scope) lands in the corpus provenance.
Both field spellings the workbench has used are read: the schema-v1
export writes the dating period as period and nests imagery under an
images object (facsimile/photograph/rights/rightsUrl),
while the plain-array shape (and to_workbench) uses context and the
flat facsimileImages/images lists.
to_workbench ¶
to_workbench(corpus: Corpus, path: str | Path | None = None) -> list[dict[str, Any]]
Emit workbench-shaped inscription records (optionally writing JSON).
Each document becomes one record with the fields the workbench renders:
id/site/support/scribe/findspot/context (its name
for the dating period)/name, the flat words list, per-line
lines, translations, glyphs, transcription, and image
references. Image files are never embedded — the workbench treats the
references as paths under its own mirror, so corpora without one simply
show no imagery.
Content the format does NOT preserve: the workbench schema carries token
text only, so per-token editorial ReadingStatus (UNCLEAR/RESTORED/LOST)
and Token.annotations (lemma, morphology, evidence class, review stamps)
are not written, and a re-import reads every token as CERTAIN and
unannotated. For a lossless round-trip use Corpus.to_json or
aegean.db.to_sqlite.
With path, the records are also written as UTF-8 JSON — the file the
app loads via ?corpus=<url> or its corpus file picker.