During the last year and a half, the healthcare industry, including the mental health field, has been upended by the crisis. As people were prevented from seeking in-person from their therapists, many turned to online solutions and mental health apps to help them through difficult times. Mempathy is one such video game narrative experience where a human player creates a conversation with an Artificial Intelligence agent in order to help them change their relationship with anxiety and overcome unrealistic standards of perfection.
Advancing through the game, the player has a feeling of advancement and companionship, taken through a narrative clicking experience with constellations and words, amplified by the artistic design of watercolours in ascending shades of blue.
What is more fascinating is the AI and ML in the background that enables this interactive experience and feeling of companionship. From a technical point of view, the Mempathy game is built using controlled language generation with Plug and Play Models (PPLM) for NPC (Non-playable character) design.
Gema Parreño, Lead Data Scientist at Apium Hub, is joining us at the Data Innovation Summit 2021 to tell more about the technical aspects of the game, presenting the Mempathy serious game as an AI Safety and Alignment opportunity and the results and lessons learnt in implementing controlled language generation using GPT with Plug and Play Models for NPC design. Gema indulged in an inspiring discussion with Hyperight on the topics of serious games, using machine learning models to create personalised player experiences, the challenges with language agent implementation and a sneak peek into the lessons learnt from implementing Plug and Play Models for NPC design.
Hyperight: Hi Gema, we are thrilled to have you joining us as a speaker at the Data Innovation Summit 2021. Let’s begin with a few words about yourself and your background.
Gema Parreño: Hej There! I´m really excited to take part in this year´s Data Innovation Summit edition. I´m Gema, Lead Data Scientist at a software development company – Apiumhub, where I develop Data-Driven and Machine Learning Solutions. Besides, I’m passionate about the intersection of machine learning and games, and have had my own startup, contributed to the open-source space in StarCraft II machine learning project, and had an amazing experience at Google Brain for Stadia.
Hyperight: Your Data Innovation Summit topic focuses on a pretty captivating topic Mempathy serious game as an AI Safety and Alignment opportunity. Could you please share a bit more about what the Mempathy game presents and how the idea arose to use it for the implementation of Safety and Alignment techniques?
Gema Parreño: Serious games are a subfield inside the game industry designed with a purpose distinct from pure fun that might include a huge variety of topics ranging from education to mental health. Mempathy is a video game narrative experience that transforms the relationship with anxiety and helps to overcome unrealistic standards of perfection. It can be resumed as a ‘conversation in the stars’ sentence with an NPC (Non-playable character), from now on, the agent. This fascinating creative mission has several challenges, from the design and technology perspective: the principles of designing a companion that could really react and adapt to the player, with the aim of going out from traditional game cyclic narratives that might be not enough to fulfil the purpose.
The road to using machine learning is an ongoing, rigorous, and creative process. Previous works have explored Reinforcement Learning with a system that focuses on reward design based on responses and Imitation Learning , in which the agent gives responses based on examples that a human would make. However, in these explorations, concepts such as scale and bias came in. Using Large Language models would make the agent more reactive to the player and create a really personalised experience for a player. Therefore, the idea of aligning the agent with the intention of the video game came in. Inside the evolution of the project and from a machine learning perspective, Mempathy explores the research question of behaviour alignment problem  with GPT models from a Game Design and Content generation perspective, aiming to solve the question of how do we create agents that behave in accordance with the designer’s intention?
On a higher level, we could say that the idea came out from the iteration of several machine learning techniques, putting the focus on solving Mempathy higher mission.
Hyperight: What challenges have you come across in using Mempathy for Safety and Alignment of Language Agents?
Gema Parreño: The development of this project has been fascinating so far, but not in need of challenges. One of the key topics here regarding the language agent implementation is currently studying and implementing research that might be useful for producing the project: the ability to analyse different solutions is a skill that, even not visible, shouldn´t pass unnoticed. Glad that I have the opportunity to learn how to deal with this challenge.
Another important key point is creating experiments that could help to benchmark different solutions at scale. The vector thought of ‘how do we design an experiment to actually test a hypothesis’ aligned with Mempathy NPC game design. There has been an ongoing evolution from designing the system from scratch to actually exploring the fascinating topic of controlling large language models.
Last but not least, I must cite that producing a full prototype taking into account all the game design principles together with the AI system has been hard, as it required me to wear multiple hats with a ‘good enough result’ to actually test the idea from a holistic point of view. However, I wouldn’t change anything about it, as I’m growing a lot!
Hyperight: What other opportunities do AI and ML offer for game development, with a focus on NPC design?
Gema Parreño: NPC design has been an active field of research and study inside the videogames industry. If you are willing to have a controlled exploration, I recommend this book  for giving a general overview of this field.
From the NPC design perspective with the focus on AI, there is a full opportunity to automate all design processes from animation to content generation. I’m sure many new ideas are yet to come, adapting research coming from robotics, simulation, and natural language processing.
Hyperight: Could you reveal to us in short what were the lessons learnt in implementing controlled language generation using GPT with Plug and Play Models for NPC design?
Gema Parreño: Plug and Play Models combine a large, pre-trained large language model and an attribute model that guide text generation without any further training, allowing flexible, controlled text generation while maintaining fluency . This solution has narrowed down content generation quite consistently with respect to content generation with GPT from 35% to 1.5% misaligned and harmful content. I’m looking forward to sharing details about the key parameter search for the proposal at the Data Innovation Summit this year.
It might also be important to note that proposing cooperative interactivity from the game design perspective has been proven key with a symbolic representation of language, as unrestricted dialogue methods remain hard for evaluation .
In conclusion, we could say that the lessons learnt come from successfully implementing game design mechanics that could be optimal for the system and choosing a strategic content generation technique aligned with NPC motivations.
I’m looking forward to connecting with you at the Data Innovation Summit edition and connecting with you. In the meantime, if you would like to know more about what we do at Apium regarding data-driven approaches, you can have a look at it here.
 Benchmarking Reinforcement Learning and Imitation Learning for serious games. When language meets games. Wordplay workshop NeuRIPs 2020
 Scalable agent alignment via reward modeling: a research direction. J.Leike et al. 2018
 Artificial intelligence and Games. G. Yannakakis & J. Togelius . Chapter 5 . Modeling players
 Plug and Play Language Models: A Simple Approach to Controlled Text Generation’ 2020 S.Dathathi et al.
 ‘Towards an automatic Turing Test: Learning to evaluate dialogue responses’ 2017 R.Lowe et al.