Original Paper: https://arxiv.org/abs/2407.18416

By: Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik Narasimhan, Vishvak Murahari

Abstract:

Persona agents, which are LLM agents that act according to an assigned persona, have demonstrated impressive contextual response capabilities across various applications. These persona agents offer significant enhancements across diverse sectors, such as education, healthcare, and entertainment, where model developers can align agent responses to different user requirements thereby broadening the scope of agent applications. However, evaluating persona agent performance is incredibly challenging due to the complexity of assessing persona adherence in free-form interactions across various environments that are relevant to each persona agent. We introduce PersonaGym, the first dynamic evaluation framework for assessing persona agents, and PersonaScore, the first automated human-aligned metric grounded in decision theory for comprehensive large-scale evaluation of persona agents. Our evaluation of 6 open and closed-source LLMs, using a benchmark encompassing 200 personas and 10,000 questions, reveals significant opportunities for advancement in persona agent capabilities across state-of-the-art models. For example, Claude 3.5 Sonnet only has a 2.97% relative improvement in PersonaScore than GPT 3.5 despite being a much more advanced model. Importantly, we find that increased model size and complexity do not necessarily imply enhanced persona agent capabilities thereby highlighting the pressing need for algorithmic and architectural invention towards faithful and performant persona agents.

Summary Notes

PersonaGym: A Breakthrough in Evaluating Persona Agents and Large Language Models

Introduction

In the rapidly evolving world of AI, Large Language Models (LLMs) are pushing boundaries in diverse applications, from customer service to advanced robotics. Amongst the many innovations, persona agents—LLMs that respond according to an assigned persona—have emerged as a powerful tool to create highly personalized user experiences. However, evaluating these persona agents' adherence to their personas in varied and dynamic environments has been a significant challenge. Enter PersonaGym, a groundbreaking dynamic evaluation framework, and PersonaScore, the first automated metric to comprehensively assess persona agents.

Key Methodologies

PersonaGym operates through a series of methodical steps designed to evaluate the capabilities of persona agents:

1. Dynamic Environment Selection: An LLM reasoner selects relevant environments from a diverse pool of 150 environments based on the persona description. This ensures that the persona agent is tested in scenarios pertinent to its role.
2. Question Generation: For each evaluation task, specific questions are generated to probe the persona agent's responses in the selected environments. This step ensures that the questions are contextually appropriate and challenging.