Loci Similes: A Benchmark for Extracting Intertextualities
in Latin Literature

Previous research on Latin intertextuality has mainly relied on matching word-level embeddings. DBLP:conf/naacl/BurnsBLCD21 utilized static Word2Vec models trained on lemmatized text to rank potential intertextual phrases, evaluating performance against a dataset of 945 parallels from Valerius Flaccus’ Argonautica dexter_ldquodatabase_2024. Although this approach outperformed traditional lexical methods in capturing semantic similarity, it is limited by the inability of static embeddings to handle context-dependent polysemy.

Addressing this limitation, gong_augmented_2025 substituted static vectors with LatinBERT DBLP:journals/corr/abs-2009-10053, a transformer-based model generating context-aware embeddings. By computing similarities between source words and target bigrams, their system successfully identified allusions in Lucan’s Pharsalia, with a user study confirming its utility for philological interpretation compared to alternative tools. DBLP:conf/latech/ManjavacasLK19 modeled the detection of biblical allusions in Latin sermons as an information retrieval task, finding that semantic FastText embeddings improve ranking over lexical baselines. DBLP:conf/chr/ManjavacasKK20 moved beyond pure retrieval to analyze the contextual drivers of intertextuality in the Patrologia Latina by modeling lexical similarity and thematic embedding as separate axes.

revellio2022zitate introduced a mixed digital-hermeneutic approach, designed to uncover previously undetected Vergilian citations and allusions in Jerome’s letter corpus and, in doing so, to examine Jerome’s patterns of quotation and adaption while probing the outer limits of quotation itself. Building on this foundation, DBLP:journals/dhq/SchroppKRF24 further elaborate this procedure at the level of automated pre-selection: starting from “Simillima’s” raw two-word matches, they expand and recalibrate the cascade of rule-based filters (e.g., stop-lists, part-of-speech constraints, and lemma distance criteria) in order to make the identification of very short “micro-quotations” feasible at scale.

Both approaches are restricted to n-grams. In contrast, DBLP:conf/acl/Riemenschneider23 applied the Sentence Transformer architecture DBLP:conf/emnlp/ReimersG19 to ancient languages. They introduced SPhilBerta, a multilingual model fine-tuned on parallel sentences in Ancient Greek, Latin, and English, framing the task as binary classification over sentence pairs.

DBLP:journals/corr/abs-2109-00601 used Latin stylometry by generating contextual embeddings with LatinBERT for a corpus of 39 classical and medieval texts. Their unsupervised clustering approach in authorship attribution offered quantitative evidence connecting the anonymous Gesta Principum Polonorum to the Monk of Lido.

Beyond the specific domain of Latin literature, automated detection of text reuse has been explored across languages and tasks.

Existing general-purpose frameworks draw on literary theory to distinguish between explicit citations and implicit commentary using graph-based approaches DBLP:journals/coling/KuznetsovBEG22, and take advantage of LLM-assisted metadata extraction and vector similarity to detect explicitly marked and implicit cross-document influences DBLP:journals/corr/abs-2410-15145. In general, current trends indicate that traditional lexical methods are outperformed by contextual models; for example, DBLP:conf/starsem/MacLaughlinXS21 modeled reuse detection as a binary classification task using fine-tuned BERT and Longformer models.

Within computational literary studies, researchers have applied contextual models to track reuse between genres and periods. In the bibliometric domain, DBLP:conf/ecir/BertinA18 constructed InTeReC, a corpus of more than 300,000 sentences with citations extracted from English-language PLOS articles. For French fiction, DBLP:journals/corr/abs-2410-17759 applied language models to a corpus of 12,000 novels from the 18th to early 20th centuries. In the Scandinavian context, tangherlini2024travels mapped links between Hans Christian Andersen’s travelogs and fairy tales using Sentence-Transformers and BERTopic to uncover latent motifs. Additionally, liebl-burghardt-2020-shakespeare identified Shakespearean quotes and paraphrases within contemporary fiction.

In addition to general-purpose methods, specialized approaches address low-resource languages and their unique challenges. Miller2025Alignment enhanced reuse detection in Hebrew and Aramaic manuscripts by incorporating fastText embeddings, while gorman2016approaching modeled authorial style in Ancient Greek by analyzing “syntax words” via unsupervised hierarchical clustering. Additionally, DBLP:journals/talip/SharjeelMNNR23 detected cross-lingual reuse in English-Urdu news articles by combining machine translation with classifiers.

The detection of intertextualities shares concepts with other NLP tasks that identify dependencies or similarities between texts: quote detection DBLP:conf/nodalida/JanickiKM23, paraphrase identification DBLP:journals/es/VrbanecM23, and document alignment DBLP:conf/eacl/MolfeseBTCN24.

Quote detection typically targets direct, explicitly marked quotations within contemporary corpora DBLP:conf/konvens/PetersenFreyB24; DBLP:conf/lrec/ZhangL22; DBLP:conf/ranlp/PapayP19; DBLP:conf/wsdm/VaucherSC021. This contrasts with our focus on historical intertextuality, which often relies on subtle, unmarked allusions rather than verbatim reuse. Similarly, paraphrase detection identifies semantic similarity often in the context of plagiarism DBLP:conf/paclic/MahmoudZ22; DBLP:conf/emnlp/WahleRKG22; DBLP:journals/corr/abs-2303-13989; DBLP:conf/emnlp/WahleGR23. However, intertextuality involves recontextualization, requiring the detection of stylistic echoes where the semantic core has shifted to serve a new argument. Finally, document alignment focuses on matching entire documents between parallel texts or translations DBLP:conf/ranlp/RajithaPSR21; DBLP:journals/corr/abs-2510-15577. In contrast, intertextuality detection operates at a granular level, identifying isolated references between documents that are otherwise structurally and semantically distinct.

We divide the corpus into a separate Query Corpus and a Source Corpus. The former comprises works by the Late Antique authors Jerome and Lactantius, totaling $\sim$83k text segments. The latter, which serves as the retrieval target, consists of $\sim$88k segments drawn from ten canonical classical Latin authors: Cicero, Lucretius, Catullus, Virgil, Horace, Tibullus, Propertius, Ovid, Lucan, and Martial. A detailed breakdown of segment counts and token statistics for each author is presented in Table 1. The texts were aggregated from three primary digital repositories: Corpus Corporum, the Tesserae Project, and the OpenGreekandLatin Project. For a complete listing of the specific works, critical editions utilized, and their respective provenance, see the detailed breakdown in Appendix A (Table 2). Since we exclusively rely on publicly available texts to compile the corpus, we released the corpus files alongside the ground-truth dataset.

To construct our ground truth dataset of 545 confirmed intertextual links, we draw from two complementary sources.

First, we sourced 270 instances of references to Virgil and Cicero in Jerome’s Epistulae from the dataset¹¹1https://doi.org/10.11588/data/FVCULR established by DBLP:journals/dco/SchroppWKRF24, which generally fall into three categories: verbatim quote, paraphrase, and allusion (see Figure 2). Notably, we do not retain all entries from the original dataset on a one-to-one basis. While DBLP:journals/dco/SchroppWKRF24 treat each poetic verse as a separate unit, our dataset operates at an approximate sentence level to ensure alignment between source and reused material. Consequently, we consolidated multiple entries into a single instance in cases where citations span two or more verses. Furthermore, our expert annotators rejected links from previous scholarship that were deemed controversial or lacking sufficient evidence of intertextuality.

Second, we augmented these explicitly attested references by adding 275 additional intertextual links. We first identify candidates through a rule-based n-gram matching approach published by DBLP:journals/dhq/SchroppKRF24.²²2https://github.com/MWittweiler/citation_analysis Their pipeline identifies candidate intertextual pairs by scanning for shared, non-contiguous tokens within a defined window. To ensure meaningful matches, candidates are then refined through a cascade of filters that remove high-frequency stopwords, enforce part-of-speech constraints, and exclude generic collocations based on embedding similarity. Subsequently, the candidates are manually evaluated by domain experts, and only those confirmed to constitute meaningful intertextual links are retained in the dataset.

Although the filtering process of the candidate generation pipeline removed many false positives, numerous candidates still involved common collocations (e.g., puncto temporis “in a moment”) and required manual exclusion. Therefore, candidates identified by the $n$-gram pipeline were annotated by a team of four experts in Latin literature (two pre-PhD and two post-PhD researchers), who assessed whether the lexical overlap constituted a meaningful intertextual reference or merely a coincidental overlap or common phrase. Conflicts in the annotation were resolved through group discussion. Our expert annotators used the following three criteria to guide their annotation in identifying meaningful intertextual links:

1. 1.

   Use of Uncommon Vocabulary: If the lexical overlap consists of rare or marked words, the likelihood increases that these elements were deliberately borrowed.

2. 2.

   Attested Frequency: A specialized Latin corpus database³³3http://clt.brepolis.net/llta/Search served to help determine the frequency of overlapping expression in Latin literature. Expressions appearing exclusively in the candidate source and target passages were treated as strong indicators of a unique intertextual relationship.

3. 3.

   Conduit Function: Most importantly, if the cited expression contributes semantic, rhetorical, or cultural information from the source text that cannot be derived from the target passage in isolation, thereby implicitly enriching the interpretation, it is viewed as intertextual.

We release the complete dataset including the query texts, source corpus, and annotated intertextual links as a HuggingFace dataset.⁴⁴4https://huggingface.co/collections/julian-schelb/datasets-for-latin-intertextuality-search.

We define the task of intertextuality detection as the identification of directional dependencies between two texts: a query document (the chronologically later text) and a source document (the earlier text). We formulate this as a retrieval and alignment problem where the objective is to map each text segment in the query document to a candidate set of text segments in the source document. This mapping is inherently one-to-many; a single query sentence may correspond to zero, one, or multiple distinct segments in the source text, exhibiting either semantic equivalence or stylistic imitation.

Since we compare all text segments of the query and source documents, and given that many queries will correspond to no intertextual links, we require metrics that effectively account for the correct rejection of non-links (true negatives). We therefore employ error-based metrics normalized by the total number of sentence pairs ($N$): Segment-Misclassification Rate (SMR), which serves as a global error rate; Global False-Positive Rate (FPR), measuring spurious matches; and Global False-Negative Rate (FNR), measuring missed references. The formal definitions and equations for these metrics are provided in Appendix B.1.

The locisimiles package streamlines the end-to-end workflow by allowing users to load query and source documents split into text segments, which are then compared to generate a list of potential intertextual links. The library provides reference implementations for three core architectures: a classification-only pipeline, a retrieval-only pipeline, and a hybrid retrieve-and-rerank pipeline. Beyond these baselines, the framework is designed to be extensible, allowing researchers to implement and test custom approaches by adding their own pipelines. When a ground-truth dataset is provided, the package evaluates a given pipeline using the error-based metrics described above, which are selected to reflect the reality of scholarly editorial work and the standard classification metrics. The software is released as a standard Python package that includes a graphical user interface (GUI) component for ease of use and is available as the open-source package locisimiles⁵⁵5https://github.com/julianschelb/locisimiles. See Appendix D for details on how to use the package.

Unlike traditional methods that rely on sparse lexical overlap, we leverage language models to encode text segments into dense vector representations. Specifically, we treat text segments from the author under investigation (e.g., Jerome) as queries to be matched against an index of source texts (e.g., Virgil). We employ sentence transformer models to generate embeddings for both the query and source segments. During inference, the bi-encoder is used to obtain the cosine similarity between the query embedding and every source embedding in the database. The candidates are then ranked by their similarity scores, identifying the most likely intertextual references.

Following the initial retrieval stage, we employ a cross-encoder architecture to re-rank the top-$k$ candidates. Unlike the bi-encoder, which generates static embeddings for query and source segments independently, the cross-encoder processes the query and candidate as a single paired input DBLP:conf/emnlp/ReimersG19. This allows the self-attention mechanism to directly compare the two segments at the token level and capture interactions and dependencies between them.

To ensure statistical robustness, we employ $k$-fold cross-validation ($k=5$) on the set of 545 verified positive text segment pairs. For each training fold, we generate contrastive examples by augmenting the assigned positive pairs with $n$ negative samples. In the evaluation phase, we reconstruct a realistic retrieval scenario to test the model’s ability to distinguish genuine citations from background noise: the query document combines the fold’s held-out positive instances with a subset of the query corpus (totaling 937 segments), while the source document consists of 880 source segments.

To identify the optimal pipeline architecture, we conducted extensive independent evaluations of the dense retrieval and binary classification stages. Details on base model evaluation, negative sampling ablation studies, and hyperparameter configurations are provided in Appendix B.

• •

  Base Models: We evaluated a range of pre-trained language models to isolate the effects of domain specificity versus model size.

• •

  Negative Sampling: We compared three distinct training strategies (random pairs, random negatives, and hard negatives) while varying the positive-to-negative ratio from $1{:}1$ to $1{:}10$ to optimize the model’s discrimination capability.

• •

  Combined Pipeline: For the end-to-end pipeline, we varied the retrieval depth $k$ to identify the operating point that minimizes recall loss while maximizing computational efficiency.

We evaluated the impact of training data imbalance by varying the positive-to-negative sample ratio from $1{:}1$ to $1{:}10$. Performance improved consistently with more negative examples, and the $1{:}10$ ratio yielded optimal results, suggesting potential gains from further increasing the negative sample proportion. Comparing base models showed that large multilingual models outperformed native Latin baselines. E5-large proved to be the most effective, retrieving approximately 61% of relevant references at Recall@10 (72% at Recall@100, 83% at Recall@1000). For a detailed evaluation of negative sampling and base models, see Appendix C.1.

Training with hard negatives ($\langle\text{qry},\text{sim}\rangle$) minimized error rates more effectively than random pairing. This approach enforces finer semantic distinctions, allowing the model to better distinguish genuine intertextuality from mere topical similarity. Second, we analyzed the impact of the imbalance in the training data by varying the ratio of negative-to-positive samples from $1{:}1$ to $1{:}10$. Consistent with our findings in the retrieval stage, increasing the volume of negative examples improves the robustness of the model in imbalanced scenarios. Finally, in terms of base model architectures, XLM-RoBERTa Large achieves an average F1-score of 0.5 along with the lowest average SMR of 35 misclassified links per 10,000 candidates. For detailed evaluation results, see Appendix C.2.

We evaluate the full two-stage retrieve-and-rerank pipeline, which first generates $k$ candidates via embedding cosine similarity, followed by a binary classification stage to label pairs as Reference or No Reference. As a baseline, we consider a “retrieval only” approach where the top $k$ candidates are treated as positive predictions, simulating the workload of a scholar manually reviewing the top results. Experiments are averaged across 5 folds. For this experiment, we use a larger corpus of 937 query and 880 source segments ($\sim$109 ground-truth references per fold). As shown in Figure 5, the “retrieve-and-rerank” pipeline reduces false positives compared to the retrieval-only approach. At $k=5$, it achieves an F1-score of 0.66 versus 0.26, and maintains an F1 of 0.55 at $k=100$ where pure retrieval drops to 0.02. Crucially, the SMR remains orders of magnitude lower (0.0019 vs. 0.57 at $k=500$), indicating that the classifier rejects the vast majority of spurious candidates.

In practical terms, with a reasonable retrieval depth of $k=100$, the retrieve-and-rerank pipeline recovers 79% of true intertextual references (85 out of 108). A scholar utilizing this system would need to manually review 780 candidate passages to verify the 85 genuine intertextualities, instead of reviewing the prohibitively large number of 93,700 cases with the retrieval-only pipeline (see Table 3). This corresponds to a 99% reduction in workload while retaining nearly 80% of the relevant data.