Compression-Based Document Length Prior for Language Models
The inclusion of document length factors has been a major topic in the development of retrieval models. We believe that current models can be further improved by more refined estimations of the document’s scope. In this poster we present a new document length prior that uses the size of the compressed document. This new prior is introduced in the context of Language Modeling with Dirichlet smoothing. The evaluation performed on several collections shows significant improvements in effectiveness.
keywords:
Publication: Congress
1624015003769
June 18, 2021
/research/publications/compression-based-document-length-prior-for-language-models
The inclusion of document length factors has been a major topic in the development of retrieval models. We believe that current models can be further improved by more refined estimations of the document’s scope. In this poster we present a new document length prior that uses the size of the compressed document. This new prior is introduced in the context of Language Modeling with Dirichlet smoothing. The evaluation performed on several collections shows significant improvements in effectiveness. - J. Parapar, David E. Losada, A. Barreiro - 10.1145/1571941.1572061
publications_en