aegean.data¶
data ¶
Bundled-data access + a download-to-cache layer.
Compact text data ships in the wheel (read via importlib.resources). Large or
license-restricted assets — notably the Linear A facsimile mirror (~116 MB) — are
NOT bundled; they are fetched on demand from upstream into a user cache. This
is how the package stays small regardless of how large the source corpora are.
The "cache" is a permanent local store, not an evicting one: a fetched dataset
is downloaded once in full and stays on disk (never re-fetched, evicted, or
expired) until you remove it (aegean data remove).
Downloads are sha256-verified (when a checksum is pinned), atomic (written to a
.part file then renamed), idempotent (a present, valid cache file is a
no-op), and resumable: a transfer cut off by a network failure keeps its
.part file, and the next attempt (an in-call retry, or a later fetch)
continues from the bytes already on disk via an HTTP Range request rather than
restarting a multi-hundred-MB asset from zero. A dataset's URL can be
overridden without a code change via
PYAEGEAN_<NAME>_URL (e.g. PYAEGEAN_LINEARA_IMAGES_URL), so a researcher
can point at their own mirror before an official release is pinned.
DataNotAvailableError ¶
Bases: RuntimeError
Raised when a non-bundled dataset has not been fetched (or can't be).
HistoricalPin
dataclass
¶
A superseded release pin for a dataset that the project still hosts.
When an asset is rebuilt and re-hosted under a new tag (the 0.29.0 epigraphy
re-host: isicily-corpus-v2 etc.), the previous release is kept on GitHub so
an earlier analysis can be reproduced byte-for-byte. Each HistoricalPin records
that kept release: its version (the release tag suffix, "v1"), the exact
url and sha256 the pin was published with, and superseded (the version
that replaced it). extract mirrors the dataset's archive kind (a tar to unpack)
for that historical asset. fetch(name, version=...) resolves these.
FetchAborted ¶
Bases: DataNotAvailableError
Raised when a fetch is canceled through its abort hook (e.g. the TUI's
download worker being cancelled). The .part file is kept, so a later fetch
resumes instead of restarting.
load_bundled_json ¶
Load a JSON file shipped inside the wheel, e.g.
load_bundled_json("lineara", "signs.json").
load_gzip_json ¶
gzip-decompress and parse a fetched .json.gz index, capping the decompressed size.
A prebuilt index is sha256-pinned when fetched from the project release, but a
PYAEGEAN_<NAME>_URL override disables that check, so a swapped mirror could serve a
tiny gzip that inflates to gigabytes and exhausts memory. Decompress in chunks and stop
with a clear error past max_bytes instead of loading the whole stream blindly.
cache_dir ¶
Where fetched datasets are stored (override with PYAEGEAN_CACHE).
A permanent local store, not an evicting cache: entries stay until
explicitly removed (aegean data remove, or deleting this directory).
historical_versions ¶
historical_versions(name: str) -> list[HistoricalPin]
The kept superseded release pins for dataset name (newest-recorded first),
or [] when the dataset has none. Each pin is a HistoricalPin carrying the
version tag, its published url + sha256, and the version that superseded it —
the reproducibility record fetch(name, version=...) resolves against.
available_versions ¶
Every fetchable version of dataset name: the current pin first (current:
True), then each kept historical pin. Each entry is
{"version", "current", "sha256", "url", "superseded"}. Empty when name is
not a registered dataset.
on_disk_paths ¶
The cache paths that would exist if spec is present, whether or not
they do. Defaults to a single root/name entry (the fetch path); a
spec.on_disk override lists the real artifact names for datasets a
backend writes under a different filename (the prebuilt lexicon indexes,
the agdt-derived members). See DataSpec.on_disk.
present_paths ¶
Which of spec's on-disk artifacts actually exist under root.
versioned_entry_paths ¶
Cache paths of dataset name's versioned fetches, for byte accounting and removal.
A fetch(name, version=...) (the kept-release path) lands in a
<name>@<version> cache entry beside the current pin: a file, or a directory for
an extract dataset, plus, transiently, its download/extraction siblings
(.part, .part.info, .extract, .old, .sha256, .lock), all of
which share the <name>@<version> prefix. The current-pin probes (on_disk_paths,
present_paths) live at a separate location, so these versioned entries were
invisible to data list byte accounting and unreachable by data remove; this
enumerates them.
version restricts the result to that one kept release (<name>@<version> and
its own siblings); None returns every version's entries. The match is anchored on
the exact <name>@ prefix, and for a single version on the exact entry name or
that entry plus one of the known download/extraction suffixes, so a sibling dataset is
never swept in and, e.g., v1 never matches v11 or the distinct version v1.2
(a version tag can itself contain dots, so a plain entry-plus-dot prefix is unsafe;
dataset names contain no @). Only entries that actually exist are returned.
versioned_bytes ¶
Total on-disk size of dataset name's versioned cache entries (0 if none).
See versioned_entry_paths; version narrows it to a single kept release.
is_downloaded ¶
Whether any real on-disk artifact of spec's CURRENT pin is present under root.
This is the corrected downloaded-probe: a dataset a backend fetched under a
different filename (lsj-index -> lsj-perseus-index.json.gz, an
agdt-derived member) counts as downloaded, where a bare
(root/name).exists() check missed it. It reports the CURRENT pin only: a
store holding only a kept historical version (<name>@<version>) reads not
downloaded, since the current asset is what a plain fetch/load uses.
Use versioned_entry_paths / versioned_bytes for the kept-version footprint.
downloaded_bytes ¶
Total real on-disk size of spec's artifacts (0 if none).
Counts the current pin's present artifacts plus any kept versioned entries
(<name>@<version> from fetch(name, version=...)) that also occupy the store,
so a dataset's reported footprint reflects every reclaimable byte it holds. The
current-pin subset alone is sum(_dir_bytes(p) for p in present_paths(spec, root));
the versioned subset alone is versioned_bytes.
bundled_data_version ¶
The version of the bundled datasets.
Bundled data ships inside the wheel and is immutable for a given release, so
its version is the package version; versions gives per-file sha256s.
versions ¶
A reproducibility manifest of every dataset pyaegean can touch.
Returns {"package": …, "bundled": {…}, "fetched": {…}}: each bundled
JSON file with its sha256 + size (hashed from the installed wheel contents),
and each registered fetchable asset with its pinned URL/sha256, license, and
whether it is present in the local cache.
Pinning for papers: record aegean.__version__ and this manifest
(e.g. json.dump(aegean.data.versions(), f)) alongside your results;
anyone with the same package version and matching sha256s is analyzing
byte-identical data. Fetched assets are sha256-verified on download, so a
matching pin in this manifest is the byte-level guarantee.
When a dataset's URL is env-overridden (PYAEGEAN_<NAME>_URL, a user's own
mirror), fetch does not enforce the pinned sha256 against that other
source, so the manifest reports sha256_enforced: false and blanks the
sha256 for that entry: it would be dishonest to advertise a checksum the
download did not verify.
sha256_file ¶
Streaming sha256 of a file (won't load a 500 MB asset into memory).
download_file ¶
Download a single URL to dest atomically (a .part temp then rename),
optionally sha256-verified. A transfer cut off by a network failure keeps
its .part, and the next call resumes it with an HTTP Range request.
Returns dest; raises DataNotAvailableError on a network failure or
checksum mismatch. Shared by fetch and the on-demand dataset downloaders
(e.g. the Greek treebank).
fetch ¶
fetch(name: str, *, version: str | None = None, force: bool = False, abort: Callable[[], bool] | None = None, progress: Callable[[int, int], None] | None = None) -> Path
Download a registered remote dataset into the cache and return its path.
Verifies the sha256 when one is pinned, downloads atomically, and is a no-op
when the cache already holds it. An interrupted download keeps its .part
file and the next call resumes from it (an HTTP Range request) instead of
restarting from zero. For extract datasets the download is a
tar archive that is unpacked into a cache directory (returned); otherwise the
downloaded file path is returned. Concurrent fetches of the same dataset
(other threads or processes) are serialized on a per-dataset lock: the later
caller waits, then returns the completed artifact. abort is an optional
zero-argument callable polled during the transfer; when it returns true the
fetch stops with FetchAborted, keeping the partial file resumable (how the
TUI cancels a download worker).
version (optional) fetches a specific kept release of the dataset instead
of the current pin — the current version tag ("v2") or a superseded one the
project still hosts ("v1"; see historical_versions). A versioned fetch
lands in a separate version-suffixed cache entry (<name>@<version>), so
the default (version=None) path is completely unaffected — byte-for-byte the
same download to the same location as before. Use it to reproduce an analysis
pinned to an earlier release; available_versions lists what exists.
progress (optional) reports the run's movement as progress(done, total):
during the download it is bytes — the absolute byte position and the full
file size, or total == -1 when the remote declares no Content-Length (the
[int, int] signature is preserved by using -1 rather than None, and
-1 is unambiguous where an empty file's 0 would not be); resume continues
from the kept .part offset. For an extract dataset it is then called with
tar members — progress(members_done, total_members) — while unpacking. A
fresh, already-cached fetch makes no download calls (nothing to report). If
progress raises, the transfer's resumable .part is kept and the error is
surfaced unwrapped. Raises DataNotAvailableError for unknown datasets, un-pinned
URLs, an unknown version, checksum mismatches, unsafe archives, or network
failures — never silently, and never blocking import.
fetch_prebuilt ¶
Place a hosted prebuilt artifact at dest; return True on success.
Lets an opt-in backend prefer a small hosted index/model over a slow local
build (a ~270 MB download, or minutes of training), while keeping
build-from-source as the fallback: any failure — no pinned URL, network
error, checksum mismatch — returns False instead of raising, so the
caller proceeds to build. member names a file inside an extract
dataset's unpacked directory.
fetch_text ¶
fetch_text(name: str, dest: str | Path, *, max_bytes: int = _MAX_GZIP_JSON_BYTES, download: bool = True, expect_gzip: bool | None = None) -> Path
Fetch dataset name and materialize it at dest, gunzipping and stamping it.
The shared fetch-then-materialize helper. It fetches the registered dataset name
(sha256-pinned, via fetch), gunzip-decompresses it under a max_bytes cap, and
writes the result to dest. expect_gzip declares the caller's knowledge of the
asset: True means the asset is always a gzip archive, so a source failing the
1f 8b magic-byte check is a corrupt or swapped archive and raises rather than
materializing garbage; False copies through without decompressing; the default
None sniffs the content, decompressing gzip and copying plain sources through,
but still refuses a non-gzip source whose own name says .gz (the same
corrupt-archive reasoning). The max_bytes cap guards a decompression-bomb (or
otherwise oversized) payload served through a PYAEGEAN_<NAME>_URL mirror override
that disables the pinned-sha check, exactly as load_gzip_json documents.
Contract:
- Atomic write.
destis written to a temp sibling then swapped into place, so an interrupted write never leaves a partial file to be served, and a failed write leaves any priordestintact. - Re-pin means re-extract. A
<dest>.sha256sidecar records the sha256 of the fetched source. On a later calldestis reused only when that sidecar matches the current source, so a re-pinned asset (new content at the same name) re-materializes. A missing or unreadable stamp also re-materializes: unlikefetch's heavy-archive extract path, there is no legacy-trust carve-out here, because these are small files, so a fresh decompress is cheap and correctness wins.
download=False returns dest without fetching, for referencing the cached path
offline. Returns dest. Raises DataNotAvailableError for an unknown or un-pinned
dataset, a network or checksum failure (from fetch), or a source that exceeds
max_bytes.
load_corpus_version ¶
load_corpus_version(script_id: str, version: str, *, progress: Callable[[int, int], None] | None = None) -> 'Corpus'
Load a kept historical release of a fetched corpus as a Corpus.
Backs aegean.load(script_id, version=...) for the corpora whose loaders go
through fetch and have kept historical pins (_VERSIONED_CORPORA): the five
epigraphy JSON corpora reload via Corpus.from_json; ddbdp materialises from
its SQLite database (heavy). The versioned asset lands in its own cache entry, so
the current corpus is untouched. Raises DataNotAvailableError for a corpus with
no kept historical pins.