Benchmark results
Sphragis
19 models · 8 baselines and floors
Select a row to highlight its line in the plot; rows with diagnostics also compare accuracy by author
Sentence. Main in-work track, 28 authors. The test split holds out random sentences of every work; the same works appear in training. Athenaeus is a compilation of quotation and is retained on this track only.
- Representation:
- Combination selected before test
- GreBERTa
- KainoBERTa
- KainoBERT
- LUAR encoder
- Qwen3.5
- Char n-gram
- Lemma TF-IDF
- Function words
- Cosine Delta
- Syntax (human trees)
- Syntax (machine trees)
- Syntax
- Burrows
- ALM
- Metrical line
- Stacked ensemble
- Floor (no Greek seen)
| Sentence | ||||||
|---|---|---|---|---|---|---|
| Char n-gram+KainoBERT (end-to-end)+lemma TF-IDF1-unit pick; larger sizes differpre-committed pick | Best selection-split combination among the pre-registered candidates | 74.472.9–75.9 | 96.795.2–97.9 | 98.496.5–99.5 | posteriors or feature blocks of 2 or 3 arms | The pre-committed combination: for each task, the candidate combination with the best macro-F1 on the selection split (validation+validation2 / validation2), chosen among the candidates fixed before any result was read; within a candidate, the combination method is chosen per task on validation2 macro-F1 among concatenation of the arms' prepared feature blocks (where every arm has them) and five training-free posterior combiners (hard vote, soft vote, geometric mean, mean rank, validation2-weighted vote), and test is read once per cell after that choice. Only this cell read test; no other combination's test score exists. The pick differs from task to task: hover a cell for its recipe, or select the row for the full list. |
| Qwen3.5-27B (LoRA, tuned on train) | Linear softmax head on the last token | 72.571.0–73.7 | 97.095.6–98.0 | 98.796.9–99.7 | sentence 108,933,120 trainable parameters; work-held-out 108,851,200 trainable parameters; tragedians 108,805,120 trainable parameters; orators 108,810,240 trainable parameters; historians 108,815,360 trainable parameters; archaic epic 108,805,120 trainable parameters; tragedians (works held out) 108,800,000 trainable parameters; orators (works held out) 108,805,120 trainable parameters | Qwen3.5-27B in bfloat16, frozen, with LoRA adapters on every attention, gated-delta-net and MLP projection (sentence r=16, alpha=32; work-held-out r=16, alpha=32; tragedians r=16, alpha=32; orators r=16, alpha=32; historians r=16, alpha=32; archaic epic r=16, alpha=32; tragedians (works held out) r=16, alpha=32; orators (works held out) r=16, alpha=32) and a trained linear head read at the last token; trained on the track's one-unit task with the epoch selected on validation macro-F1 and early stopping, validation2 never opened; larger-task logits are means over exact constituents; test evaluated once. The reloaded model reproduced the Trainer's validation macro-F1 before test was read (training/reloaded: sentence 72.9/73.0; work-held-out 54.4/54.3; tragedians 71.8/71.9; orators 71.4/71.5; historians 85.8/85.8; archaic epic 75.5/75.4; tragedians (works held out) 67.5/67.3; orators (works held out) 53.0/53.3) |
| Char n-gram Dirichlet-multinomial | Dirichlet compound multinomial posterior predictive | 67.165.6–68.6 | 94.392.6–95.7 | 97.095.1–98.2 | Validation-selected n: sentence 4/4/3; work-held-out 3/3/3; tragedians 4/4/3; orators 4/3/3; historians 6/3/2; archaic epic 6/5/5; tragedians (works held out) 2/2/2; orators (works held out) 6/2/2 | Character n-gram counts with a uniform author prior fixed a priori; validation-selected n and Dirichlet smoothing alpha (sentence 0.1/0.1/0.1; work-held-out 0.1/0.1/0.1; tragedians 0.1/0.1/0.01; orators 0.1/0.1/1.0; historians 0.01/1.0/1.0; archaic epic 0.1/0.1/0.1; tragedians (works held out) 0.1/1.0/0.1; orators (works held out) 0.1/1.0/1.0); vocabulary from the atomic training split only; train+validation refit; test evaluated once; validation-selected Dirichlet concentration (sentence 100.0/10000.0/1000.0; work-held-out raw/raw/raw; tragedians raw/100.0/raw; orators 10000.0/raw/100.0; historians 1000.0/raw/raw; archaic epic raw/raw/raw; tragedians (works held out) raw/1000.0/raw; orators (works held out) 1000.0/1000.0/1000.0), where the raw posterior is the non-bursty limit that reproduces naive Bayes |
| Char n-gram naive Bayes | Multinomial naive Bayes | 67.065.5–68.4 | 94.392.5–95.6 | 97.195.1–98.3 | Validation-selected n: sentence 4/4/3; work-held-out 3/3/3; tragedians 4/4/3; orators 4/3/3; historians 6/3/2; archaic epic 6/5/5; tragedians (works held out) 2/3/2; orators (works held out) 6/2/2 | Character n-gram counts with a uniform author prior fixed a priori; validation-selected n and Dirichlet smoothing alpha (sentence 0.1/0.1/0.1; work-held-out 0.1/0.1/0.1; tragedians 0.1/0.1/0.01; orators 0.1/0.1/1.0; historians 0.01/1.0/1.0; archaic epic 0.1/0.1/0.1; tragedians (works held out) 0.1/1.0/0.1; orators (works held out) 0.1/0.1/0.1); vocabulary from the atomic training split only; train+validation refit; test evaluated once |
| KainoBERT (end-to-end) | Linear softmax head | 65.964.5–67.0 | 92.290.5–93.5 | 96.294.2–97.4 | ~136M trainable parameters per track | Fine-tuned on the track's one-unit task with validation-loss early stopping; larger-task logits are means over exact constituents; test evaluated once; the published score is the mean over 3 seeds and its interval resamples rows and averages the seeds. KainoBERT is pretrained from the second corpus derivative, from which every benchmark work, passage (any six-word window) and line was removed; its memorisation probe places the test sentences at its unseen level. The first KainoBERT, whose corpus removed exact lines only and whose probe placed them at its seen level, was withdrawn on 2026-09-24 |
| Char n-gram | PyTorch multinomial logistic regression | 65.864.1–67.2 | 95.594.0–96.7 | 97.595.7–98.6 | sentence 137,215 / 27,377 / 27,377; work-held-out 20,501 / 20,501 / 20,501; tragedians 15,482 / 15,482 / 52,944; orators 12,291 / 12,291 / 1,761; historians 51,049 / 14,563 / 1,923; archaic epic 69,599 / 18,680 / 18,680; tragedians (works held out) 13,312 / 1,913 / 1,913; orators (works held out) 11,686 / 11,686 / 1,717 | Validation-selected n (sentence 4/3/3; work-held-out 3/3/3; tragedians 3/3/4; orators 3/3/2; historians 4/3/2; archaic epic 4/3/3; tragedians (works held out) 3/2/2; orators (works held out) 3/3/2) among orders 2-6; Five-point learning-rate search on validation macro-F1; validation-loss early stopping; train+validation refit; test evaluated once |
| KainoBERT (trained on chunks) | Linear softmax head | 64.262.7–65.4 | 92.390.7–93.5 | 96.093.8–97.3 | ~136M trainable parameters per task size | One classifier per task size, fine-tuned on that size's whole chunks ([CLS] unit [SEP] ... up to 1,024 tokens) rather than on single sentences whose logits are then averaged, which is what every other encoder row does; the larger sizes also see author-pure chunks resampled from the atomic training rows afresh every epoch. Each size selects its own epoch on validation macro-F1 with early stopping; test evaluated once. Macro-F1 that chunk training gains over averaging the one-unit model's logits, on the 5- and 10-unit tasks: sentence +0.13/-1.56; work-held-out +4.49/+16.44; tragedians +9.81/+3.00; orators -1.02/-5.00; historians -0.23/-4.22; archaic epic -1.52/-0.31; tragedians (works held out) +5.80/-2.06; orators (works held out) +33.01/+25.71. The published score is the mean over 3 seeds. KainoBERT is pretrained from the second corpus derivative, from which every benchmark work, passage (any six-word window) and line was removed; its memorisation probe places the test sentences at its unseen level. The first KainoBERT, whose corpus removed exact lines only and whose probe placed them at its seen level, was withdrawn on 2026-09-24 |
| KainoBERTa (end-to-end) | Linear softmax head | 63.662.0–64.9 | 92.190.3–93.5 | 96.694.7–97.8 | ~112M trainable parameters per track | Fine-tuned on the track's one-unit task with validation-loss early stopping; larger-task logits are means over exact constituents; test evaluated once; the published score is the mean over 3 seeds and its interval resamples rows and averages the seeds. Architecture control for KainoBERT: pretrained from the same second corpus derivative, the same packed blocks, tokenizer, masking and schedule, stopped by the same validation plateau rule; the one difference besides the architecture is a twelve-epoch high-rate phase where KainoBERT's is four, since four left the roberta architecture on its initial plateau. The first KainoBERTa, pretrained from the derivative that removed exact lines only, was withdrawn on 2026-09-25 |
| Siamese KainoBERT (SupCon + CE) | Nearest author centroid and linear softmax head | 63.562.0–65.0 | 84.981.6–87.1 | 87.983.2–90.9 | ~136M trainable parameters; 256-d L2-normalized embedding | Shared GreBERTa encoder trained with supervised contrastive plus cross-entropy on author-balanced batches; epoch and decision rule (sentence centroid; work-held-out centroid; tragedians centroid; orators head; historians centroid; archaic epic head; tragedians (works held out) centroid; orators (works held out) centroid; epoch sentence 24; work-held-out 19; tragedians 25; orators 15; historians 17; archaic epic 19; tragedians (works held out) 31; orators (works held out) 8) both selected on the full validation split; larger-task scores are means over exact constituents; test evaluated once. KainoBERT is pretrained from the second corpus derivative, from which every benchmark work, passage (any six-word window) and line was removed; its memorisation probe places the test sentences at its unseen level. The first KainoBERT, whose corpus removed exact lines only and whose probe placed them at its seen level, was withdrawn on 2026-09-24 |
| OLMo-1B authorial language models | Lowest mean per-token surprisal | 63.061.1–64.3 | 83.379.9–85.6 | 83.579.4–86.2 | 1.18B parameters per author; sentence 28 models; work-held-out 12 models; tragedians 3 models; orators 4 models; historians 5 models; archaic epic 3 models; tragedians (works held out) 2 models; orators (works held out) 3 models | One OLMo-1B model further pretrained per author on the second edition with validation-loss early stopping; attribution is the lowest mean per-token surprisal; no base-model or epoch search; test evaluated once |
| Lemma TF-IDF | PyTorch multinomial logistic regression | 62.961.2–64.5 | 94.392.2–95.8 | 98.596.8–99.5 | sentence 27,489 / 27,489 / 27,489; work-held-out 14,017 / 14,017 / 14,017; tragedians 7,333 / 7,333 / 7,333; orators 4,271 / 4,271 / 4,271; historians 8,341 / 8,341 / 8,341; archaic epic 8,364 / 8,364 / 8,364; tragedians (works held out) 5,606 / 5,606 / 5,606; orators (works held out) 3,854 / 3,854 / 3,854 | Five-point learning-rate search on validation macro-F1; validation-loss early stopping; train+validation refit; test evaluated once |
| KainoBERT | PyTorch multinomial logistic regression | 59.257.4–60.8 | 90.287.9–91.9 | 95.793.2–97.2 | 768 frozen features | Five-point learning-rate search on validation macro-F1; validation-loss early stopping; train+validation refit; test evaluated once. KainoBERT is pretrained from the second corpus derivative, from which every benchmark work, passage (any six-word window) and line was removed; its memorisation probe places the test sentences at its unseen level. The first KainoBERT, whose corpus removed exact lines only and whose probe placed them at its seen level, was withdrawn on 2026-09-24 |
| KainoBERTa | PyTorch multinomial logistic regression | 57.355.6–58.7 | 88.585.9–90.5 | 95.292.5–96.9 | 768 frozen features | Five-point learning-rate search on validation macro-F1; validation-loss early stopping; train+validation refit; test evaluated once. Architecture control for KainoBERT: pretrained from the same second corpus derivative, the same packed blocks, tokenizer, masking and schedule, stopped by the same validation plateau rule; the one difference besides the architecture is a twelve-epoch high-rate phase where KainoBERT's is four, since four left the roberta architecture on its initial plateau. The first KainoBERTa, pretrained from the derivative that removed exact lines only, was withdrawn on 2026-09-25 |
| LUAR-style encoder+linear probe | PyTorch multinomial logistic regression | 56.655.0–58.1 | 79.976.3–82.4 | 85.681.2–88.8 | 512 frozen features | The same frozen LUAR-style embeddings read by a linear classifier instead of by neighbours; selected learning rate and epoch: sentence 0.0003 at epoch 1725; work-held-out 0.01 at epoch 27; tragedians 0.003 at epoch 232; orators 0.001 at epoch 787; historians 0.01 at epoch 269; archaic epic 0.0003 at epoch 1941; tragedians (works held out) 0.01 at epoch 26; orators (works held out) 0.01 at epoch 31. Five-point learning-rate search on validation macro-F1; validation-loss early stopping; train+validation refit; test evaluated once |
| LUAR-style encoder+kNN | Similarity-weighted vote of the k nearest training rows | 47.946.3–49.4 | 69.465.7–71.9 | 72.567.6–75.7 | 512-d unit-norm embedding, frozen | Frozen LUAR-style encoder (KainoBERT backbone, attention-mask mean pooling and a two-layer projection onto the unit sphere, trained with a supervised contrastive episode loss on pool authors the benchmark never scores); a row is attributed by the similarity-weighted vote of its nearest training embeddings, k selected per task on validation macro-F1 among 1/5/10/25 (k = sentence 10/5/5; work-held-out 5/1/1; tragedians 5/1/1; orators 1/1/1; historians 25/10/1; archaic epic 1/1/1; tragedians (works held out) 10/5/5; orators (works held out) 10/5/1); at test the reference set is train plus validation, the kNN analogue of the repo's refit; larger-task scores are means over exact constituents; test evaluated once |
| Burrows lemma z-scores | PyTorch multinomial logistic regression | 42.540.7–44.1 | 84.381.2–86.7 | 94.291.0–96.5 | Validation-selected MFW n: sentence 1000/1000/1000; work-held-out 1000/500/500; tragedians 1000/1000/1000; orators 1000/1000/500; historians 1000/500/500; archaic epic 1000/1000/1000; tragedians (works held out) 500/100/100; orators (works held out) 1000/500/1000 | Most frequent lemmata and z-score reference from the atomic training split only; n selected per task among 100/500/1000; Five-point learning-rate search on validation macro-F1; validation-loss early stopping; train+validation refit; test evaluated once |
| Cosine Delta (lemma frequencies) | Nearest author profile by cosine distance | 40.438.9–41.7 | 80.477.4–82.7 | 92.389.2–94.4 | Validation-selected n: sentence 2000/2000/2000; work-held-out 2000/500/500; tragedians 2000/2000/2000; orators 2000/2000/2000; historians 2000/1000/500; archaic epic 2000/2000/2000; tragedians (works held out) 500/500/500; orators (works held out) 2000/1000/1000; vocabulary sentence 2,000 / 2,000 / 2,000; work-held-out 2,000 / 500 / 500; tragedians 2,000 / 2,000 / 2,000; orators 2,000 / 2,000 / 1,659; historians 2,000 / 1,000 / 500; archaic epic 2,000 / 2,000 / 2,000; tragedians (works held out) 500 / 500 / 500; orators (works held out) 1,056 / 1,000 / 1,000 | The most frequent lemmata of the atomic training split, relative frequencies z-scored and attributed to the nearest author profile by cosine distance; n is selected per task on validation macro-F1 among 500/1000/2000 and the culling threshold (the fraction of training authors that must use a lemma) among 0/0.2/0.5 (culling: sentence 0/0/0; work-held-out 0/0.2/0.2; tragedians 0/0/0; orators 0/0/0.5; historians 0/0/0; archaic epic 0/0/0; tragedians (works held out) 0/0/0; orators (works held out) 0.5/0.5/0); the lemma list, the z-score reference and the author profiles come from the atomic training split only and are refit on train plus validation; test evaluated once |
| Syntax feature rates (machine trees) | PyTorch multinomial logistic regression | 37.135.4–38.6 | 75.371.5–78.0 | 86.782.2–89.3 | Validation-selected d: sentence 3/3/3; work-held-out 3/2/2; tragedians 3/4/2; orators 4/2/2; historians 2/4/3; archaic epic 4/4/3; tragedians (works held out) 4/3/2; orators (works held out) 4/2/2; features sentence 153,894 / 153,894 / 153,894; work-held-out 137,675 / 6,619 / 6,619; tragedians 120,442 / 1,201,665 / 6,282; orators 1,151,691 / 5,724 / 5,724; historians 5,663 / 1,069,387 / 104,374; archaic epic 1,131,993 / 1,131,993 / 112,065; tragedians (works held out) 1,022,060 / 106,623 / 5,854; orators (works held out) 1,120,209 / 5,578 / 5,578 | Rates of syntactic atoms from the uniform machine parse (conllu_machine) with binned dependency distance; combination dimension 1-4 selected per task on validation macro-F1; Five-point learning-rate search on validation macro-F1; validation-loss early stopping; train+validation refit; test evaluated once |
| Qwen3.5-27B (zero-shot) | Greedy decoding, the answer parsed to one author | 26.124.9–27.3 | 49.345.6–52.0 | 58.753.8–61.9 | 27B frozen parameters; nothing trained and nothing selected | Reads the test split and nothing else: Qwen/Qwen3.5-27B is prompted once per test row with the family's author inventory and the row's plain text (prompt sphragis-2026-09-closed-set-v1), and the greedy answer is parsed to one label. Nothing is trained and nothing is selected, so train, validation and validation2 are never opened and there is no refit. An answer naming no author, or two, counts as wrong; such answers numbered sentence 5/1/0; work-held-out 0/1/1; tragedians 0/0/0; orators 0/0/0; historians 0/0/0; archaic epic 0/0/0; tragedians (works held out) 0/0/0; orators (works held out) 0/0/0 |
| Siamese GreBERTa (SupCon + CE)floor | Nearest author centroid and linear softmax head | 74.172.7–75.5 | 95.293.3–96.6 | 99.198.1–99.8 | ~127M trainable parameters; 256-d L2-normalized embedding | Shared GreBERTa encoder trained with supervised contrastive plus cross-entropy on author-balanced batches; epoch and decision rule (sentence head; work-held-out head; tragedians head; orators centroid; historians head; archaic epic head; tragedians (works held out) head; orators (works held out) centroid; epoch sentence 62; work-held-out 15; tragedians 49; orators 30; historians 26; archaic epic 24; tragedians (works held out) 6; orators (works held out) 12) both selected on the full validation split; larger-task scores are means over exact constituents; test evaluated once. Baseline, not a contender: GreBERTa was pretrained by its authors on Ancient Greek corpora that were not screened for the benchmark's works, so its scores are not controlled for memorisation; shown greyed for reference |
| GreBERTa (end-to-end)floor | Linear softmax head | 71.670.2–72.8 | 95.393.6–96.5 | 97.494.8–98.7 | ~126M trainable parameters per track | Fine-tuned on the track's one-unit task with validation-loss early stopping; larger-task logits are means over exact constituents; test evaluated once; the published score is the mean over 3 seeds and its interval resamples rows and averages the seeds. Baseline, not a contender: GreBERTa was pretrained by its authors on Ancient Greek corpora that were not screened for the benchmark's works, so its scores are not controlled for memorisation; shown greyed for reference |
| GreBERTafloor | PyTorch multinomial logistic regression | 64.763.1–66.2 | 91.489.2–93.1 | 95.593.2–97.1 | 768 frozen features | Five-point learning-rate search on validation macro-F1; validation-loss early stopping; train+validation refit; test evaluated once. Baseline, not a contender: GreBERTa was pretrained by its authors on Ancient Greek corpora that were not screened for the benchmark's works, so its scores are not controlled for memorisation; shown greyed for reference |
| Syntax feature rates (human trees)floor | PyTorch multinomial logistic regression | 40.638.8–42.1 | 76.673.0–79.1 | 86.181.5–89.1 | Validation-selected d: sentence 3/3/3; work-held-out 3/2/2; tragedians 4/2/4; orators 4/2/2; historians 3/2/2; archaic epic 4/2/2; tragedians (works held out) 3/2/4; orators (works held out) 3/3/2; features sentence 136,186 / 136,186 / 136,186; work-held-out 120,613 / 5,568 / 5,568; tragedians 1,033,234 / 4,971 / 1,033,234; orators 1,055,027 / 4,691 / 4,691; historians 92,599 / 4,736 / 4,736; archaic epic 1,032,386 / 4,763 / 4,763; tragedians (works held out) 87,640 / 4,638 / 896,206; orators (works held out) 92,384 / 92,384 / 4,593 | Floor, not a contender: rates of syntactic atoms from the merged human annotation (conllu_human, UD heads since the third edition) with binned dependency distance; combination dimension 1-4 selected per task on validation macro-F1; Five-point learning-rate search on validation macro-F1; validation-loss early stopping; train+validation refit; test evaluated once. Published as a floor because the human annotation still carries each project's rate of choosing a relation, and a project names its authors |
| Markup-only (machine trees)floor | scikit-learn logistic regression | 19.9 | 34.1 | 33.1 | sentence 82 / 82 / 82; work-held-out 80 / 80 / 80; tragedians 79 / 79 / 79; orators 78 / 78 / 78; historians 79 / 79 / 79; archaic epic 78 / 78 / 78; tragedians (works held out) 78 / 78 / 78; orators (works held out) 78 / 78 / 78 | Floor: non-lexical rates over UPOS, DEPREL and FEATS values, head direction, binned head distance and function words heading anything, plus token count of the machine annotation and no Greek at all; a model that does not clearly beat it has demonstrated annotation-scheme identification, not authorship attribution. From the dataset's metadata/leakage_report_conllu_machine.json |
| Cosine Delta (function words only)floor | Nearest author profile by cosine distance | 18.017.0–19.0 | 43.339.8–46.1 | 61.255.7–64.9 | Validation-selected n: sentence all/200/200; work-held-out 200/200/200; tragedians 100/100/200; orators 100/200/100; historians 100/100/100; archaic epic 200/200/100; tragedians (works held out) 100/100/100; orators (works held out) 100/100/100; vocabulary sentence 222 / 200 / 133; work-held-out 184 / 184 / 113; tragedians 100 / 100 / 130; orators 100 / 111 / 100; historians 100 / 100 / 100; archaic epic 122 / 148 / 100; tragedians (works held out) 100 / 100 / 100; orators (works held out) 99 / 99 / 100 | Floor, not a contender: the style-not-topic reference, closed-class lemmas only, no content words at all. The inventory is majority upos in ['ADP', 'PRON'] over the atomic training documents, or a curated particle/conjunction/negation/article lemma, or one of the 50 most frequent ADV-majority lemmas, which leaves sentence 222 lemmas; work-held-out 184 lemmas; tragedians 130 lemmas; orators 130 lemmas; historians 125 lemmas; archaic epic 148 lemmas; tragedians (works held out) 122 lemmas; orators (works held out) 128 lemmas; n is selected per task on validation macro-F1 among 100/200/all of that inventory and the culling threshold among 0/0.2/0.5 (culling: sentence 0/0/0.2; work-held-out 0/0/0.5; tragedians 0/0/0; orators 0/0.5/0.5; historians 0/0/0; archaic epic 0.5/0/0; tragedians (works held out) 0/0/0; orators (works held out) 0.5/0.5/0). Otherwise the lemma-frequency row's Cosine Delta: relative frequencies z-scored and attributed to the nearest author profile by cosine distance; the lemma list, the z-score reference and the author profiles come from the atomic training split only and are refit on train plus validation; test evaluated once |
| Markup-only (human trees)floor | scikit-learn logistic regression | 17.7 | 33.0 | 31.1 | sentence 70 / 70 / 70; work-held-out 69 / 69 / 69; tragedians 65 / 65 / 65; orators 66 / 66 / 66; historians 67 / 67 / 67; archaic epic 64 / 64 / 64; tragedians (works held out) 64 / 64 / 64; orators (works held out) 64 / 64 / 64 | Floor: non-lexical rates over UPOS, DEPREL and FEATS values, head direction, binned head distance and function words heading anything, plus token count of the human annotation and no Greek at all; a model that does not clearly beat it has demonstrated annotation-scheme identification, not authorship attribution. From the dataset's metadata/leakage_report.json |
| Majority classfloor | Constant: the commonest training author | 0.8 | 0.8 | 0.8 | 0 features | Floor: every test row is assigned the author with the most training rows; sees neither Greek nor annotation. Recomputed from the official splits |
Scores are held-out test macro-F1 (%) with a 95% bootstrap interval under each, over test rows on the in-work tracks and over held-out works on the work-held-out tracks; the bold value is a column's best only when its interval lies wholly above every other contender's, so a column may have no bold value. On the work-held-out tracks the score is over the authors with a same-genre rival; the singletons' figure beside it is a genre floor. Grey floor rows see no Greek (majority class, markup-only probes), or are contenders the third edition publishes as floors (the human-tree syntax rates, the function-word Cosine Delta), pinned at the bottom. The pre-committed pick row is the one combination chosen per task before its test score was read, and the only combination whose test score exists; hover one of its cells for that task's recipe, or select the row for the full list.
Plotted results
Score by task size
Held-out test macro-F1 (%); points are linearly connected. Select a line, a table row, or a legend entry to highlight one model.
Has the encoder seen the test sentences?
| Encoder | Pretraining corpus | Benchmark test | Benchmark train | Corpus seen | Corpus unseen | Seen − unseen | Test position on seen scale |
|---|---|---|---|---|---|---|---|
| greberta | not this corpus (AncientGreek-no-sphragis-v1 used as reference text) | 46.1 | 45.7 | 46.8 | 43.2 | 3.6 | n/a |
| kainobert | AncientGreek-no-sphragis-v2 | 32.9 | 32.0 | 38.0 | 38.8 | -0.8 | 7.61 |
| kainoberta | AncientGreek-no-sphragis-v2 | 12.8 | 12.2 | 16.8 | 19.1 | -2.4 | 2.69 |
Top-1 masked-token accuracy (%). A clean encoder scores the benchmark test sentences at its unseen level. KainoBERT is pretrained from the second corpus derivative, which removes the benchmark's works and passages, and sits below its unseen level. The first KainoBERT, pretrained from a derivative that removed exact lines only, scored the test sentences at its seen level (0.460 against 0.434 unseen and 0.463 seen) and was withdrawn on 2026-09-24. GreBERTa was pretrained elsewhere on corpora not screened for the benchmark and is shown as a baseline only.