Improving robustness and reproducibility in validation of Natural Language Generation systems.

The rapid advancement of Large Language Models (LLMs) has revolutionised Natural Language Generation (NLG), but has also given rise to significant challenges in their evaluation. Automatic metrics are insufficient to capture the true quality of the text, whilst manual evaluation is too resource-intensive. This thesis analyses the foundations of NLG evaluation. Firstly, it investigates semantic errors such as hallucinations and omissions, demonstrating that expert human validation is strictly necessary due to the low agreement amongst non-expert evaluators. It then establishes empirical guidelines for experimental design based on robust statistical frameworks and assesses the reproducibility of current evaluations. Finally, it proposes a hybrid methodology that combines the scalability of LLMs with the reliability of on-demand human intervention, ensuring more rigorous, transparent and reproducible evaluations.

Palabras clave: Natural Language Generation, NLG Evaluation, Human Evaluation, Reproducibility