AGREE: a new benchmark for the evaluation of distributional semantic models of ancient Greek

OnderzoeksoutputAcademicpeer review

2 Citaten (Scopus)
38 Downloads (Pure)

Samenvatting

The last years have seen the application of Natural Language Processing, in particular, language models, to the study of the Semantics of ancient Greek, but only a little work has been done to create gold data for the evaluation of such models. In this contribution we introduce AGREE, the first benchmark for intrinsic evaluation of semantic models of ancient Greek created from expert judgements. In the absence of native speakers, eliciting expert judgements to create a gold standard is a way to leverage a competence that is the closest to that of natives. Moreover, this method allows for collecting data in a uniform way and giving precise instructions to participants. Human judgements about word relatedness were collected via two questionnaires: in the first, experts provided related lemmas to some proposed seeds, while in the second, they assigned relatedness judgements to pairs of lemmas. AGREE was built from a selection of the collected data.
Originele taal-2English
TijdschriftDigital Scholarship in the Humanities
DOI's
StatusE-pub ahead of print - 15-jan.-2024

Vingerafdruk

Duik in de onderzoeksthema's van 'AGREE: a new benchmark for the evaluation of distributional semantic models of ancient Greek'. Samen vormen ze een unieke vingerafdruk.

Citeer dit