A study of analogical density in various corpora at various granularity

Rashel Fam*, Yves Lepage

*この研究の対応する著者

研究成果: Article査読

1 被引用数 (Scopus)

抄録

In this paper, we inspect the theoretical problem of counting the number of analogies between sentences contained in a text. Based on this, we measure the analogical density of the text. We focus on analogy at the sentence level, based on the level of form rather than on the level of semantics. Experiments are carried on two different corpora in six European languages known to have various levels of morphological richness. Corpora are tokenised using several tokenisation schemes: character, sub-word and word. For the sub-word tokenisation scheme, we employ two popular sub-word models: unigram language model and byte-pair-encoding. The results show that the corpus with a higher Type-Token Ratio tends to have higher analogical density. We also observe that masking the tokens based on their frequency helps to increase the analogical density. As for the tokenisation scheme, the results show that analogical density decreases from the character to word. However, this is not true when tokens are masked based on their frequencies. We find that tokenising the sentences using sub-word models and masking the least frequent tokens increase analogical density.

本文言語English
論文番号314
ジャーナルInformation (Switzerland)
12
8
DOI
出版ステータスPublished - 2021 8月

ASJC Scopus subject areas

  • 情報システム

フィンガープリント

「A study of analogical density in various corpora at various granularity」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル