CHATTY: A Tool for Building and Evaluating Customizable LLM Assistants
In this work, we present Chatty, a tool for customizing Large Language Model (LLM) assistants through structured role biasing. Chatty offers a library of predefined system roles and prompt templates, allowing users to create assistants with distinct behaviors. It includes multiple LLM families and features a parallel chat interface that supports synchronous chat with multiple assistants. Users can compare the respective completions side by side, download specific conversations, or freeze some of them. Chatty can be exploited for research in AI conversational agents and LLM evaluation.
To assess the impact of role biasing in a specific domain, we conducted an initial study in which an expert psychologist experimented with Chatty and prompted it with general health questions. This allowed for a straightforward comparison of three assistant roles: a generic assistant (with no specific customization), a specialized medical assistant (designed for a doctor), and a misleading assistant (specifically designed to deceive).
The results of this expert evaluation showed that the doctor's role produced the most reliable responses. Chatty contributes with a role customization library,
a conversational interface that seamlessly connects different assistant prompts and LLM families,
and the possibility of building conversational datasets for research in this area.
Palabras clave: Large Language Models, Role Assistants, Biasing Evaluation
Publicación: Congreso
1788261079112
1 de septiembre de 2026
/research/publications/chatty-a-tool-for-building-and-evaluating-customizable-llm-assistants
In this work, we present Chatty, a tool for customizing Large Language Model (LLM) assistants through structured role biasing. Chatty offers a library of predefined system roles and prompt templates, allowing users to create assistants with distinct behaviors. It includes multiple LLM families and features a parallel chat interface that supports synchronous chat with multiple assistants. Users can compare the respective completions side by side, download specific conversations, or freeze some of them. Chatty can be exploited for research in AI conversational agents and LLM evaluation.
To assess the impact of role biasing in a specific domain, we conducted an initial study in which an expert psychologist experimented with Chatty and prompted it with general health questions. This allowed for a straightforward comparison of three assistant roles: a generic assistant (with no specific customization), a specialized medical assistant (designed for a doctor), and a misleading assistant (specifically designed to deceive).
The results of this expert evaluation showed that the doctor's role produced the most reliable responses. Chatty contributes with a role customization library,
a conversational interface that seamlessly connects different assistant prompts and LLM families,
and the possibility of building conversational datasets for research in this area. - Manuel Couto, Marcos Fernández-Pichel, Mario Ezra Aragón, David Gallego and David E. Losada
publications_es