Machine Learning is already being used at high speed in game development. Thanks to the algorithms’ ability to learn and improve from experience, ML is a powerful tool for game developers to create more realistic worlds, fascinating challenges, and unique content, as well as better understand player behaviour.
For video game fans, it’s a real treat to look more closely at how machine learning is applied in creating those imaginary worlds they are indulging in with pleasure. To understand how it helps development teams build better games, we talked to John Wordsworth, Chief Architect at Paradox Interactive, about the various applications and aspects of machine learning in the game development process, more specifically about the opportunities ML provides for content generation in games, and we touched upon trends of ML in game development that game enthusiasts eagerly expect.
John will indulge us with an inspirational talk on 3 direct applications for machine learning for content generation in game development and present some of the opportunities and challenges ahead for their adoption at the NDSML Summit 2021.
Hyperlight: Hi John, welcome to the NDSML Summit 2021, we are really excited to have you with us. Let’s begin with a few words about yourself and what you do.
John Wordsworth: Thank you Ivana! I am very excited to be taking part in the NDSML Summit 2021.
As the Chief Architect at Paradox Interactive, a large part of my role is to explore new techniques that can enable our game development teams to build new and interesting experiences for our players. Machine learning is particularly interesting right now as it has applications across several different parts of the gaming industry – such as analysing user behaviour, helping development teams build better games more efficiently and real-time applications that directly improve the gaming experience.
Before my current role, I spent several years working with the incredibly talented core tech team who built the Clausewitz engine used by several of our games. As a programmer and then a lead, I got to understand that game development is a difficult process with several disciplines having to work closely together. As such, it became clear there was value in experimenting with machine learning techniques to find ways to improve how game developers of all disciplines can build games differently. In the past, it was difficult to move from experiments to production tools, but it’s now looking very likely that machine learning is lining up to change how development teams will build games in the coming few years.
Before working with game development, I studied a PhD in Applied Mathematics at the University of Exeter in the UK. My research focused on systems of coupled oscillators and included using genetic algorithms to adapt and train the coupling parameters. During my research work, I explored and compared GAs with neural networks and it is very interesting for me now to see the applications of machine learning flourish outside of research as well.
Hyperlight: Your NDSML Summit session will focus on Content Generation using Machine Learning in Game Development. How can Machine Learning help in game development?
John Wordsworth: We are already seeing several interesting applications of machine learning in the gaming industry. From a business standpoint, machine learning is valuable in helping us better understand player behaviour. From a gamer’s standpoint, we can already see neural networks being used for real-time effects, such as dynamic upscaling of content to improve visual quality. From a development perspective, we are just starting to see tooling appear to enable development teams to build better games in more efficient ways, which will only grow in the coming years.
The process of building a video game involves several different disciplines working together to produce a complex product which often consists of thousands (or even tens of thousands) of 2D/3D art assets, audio files, world design scripts, source code and much more. Each release and update must also go through extensive testing processes where QA teams test a huge number of different ways to play (and break) the game.
Using machine learning to improve the content generation process provides a number of unique opportunities. Here are just a few of them.
Assisted Artwork Generation: Games often consist of hundreds of assets that are all produced in a similar fashion – for instance, while a game might have ‘cartoony’ textures on their 3D models, they could be hand-drawn over real-world photos. Machine learning techniques can help optimise workflows so that artists can spend more time on the creative part of their work and less time on the ‘mechanical’ parts. For example, using style-transfer techniques to assist with the aforementioned task would allow the artist to spend more time adding customisations and flourishes to elaborate on the game’s setting.
Dynamic Audio Edits: Some parts of the development pipeline can be incredibly time-consuming and hard to change after being produced. For instance, in-game dialogue requires a voice actor to spend days in a recording studio and, if the script is changed after recording, it is difficult and costly to re-record. In the short term, speech generation with machine learning can help ‘patch’ changed audio to allow for script changes or insert the player’s name into pre-recorded dialogue. In the long-term, AI voice actors could even replace real voice actors, especially for secondary characters.
Personalised User-Content: Machine learning techniques provide an interesting opportunity to build systems that can be used directly by users to generate content that fits in with the style of the game itself. For instance, they open opportunities for having the player take a photo of themselves and have their likeness transferred into the game.
We are just starting to see companies who are commercialising machine learning techniques in content generation for game development, but there will clearly be many innovations in the coming years to change how video games are developed.
Hyperlight: What challenges have you come across in the adoption of machine learning applications in content generation in game development?
John Wordsworth: The tooling available to explore machine learning techniques today is extensive. A large body of learning materials is also readily available and the number of pre-trained models for solving specific problems is constantly growing. All of this makes it possible for interested programmers to experiment with machine learning in-game production pipelines.
While experimenting with machine learning is accessible today, taking an experiment into production can be an organisational challenge – especially if you do not have a team geared to take experiments into production. The field and related tech is changing fast, which can make it fragile when moving prototypes into the hands of designers, and the fact that the programmers experimenting with these new tools are often not the target day-to-day users means that it can be difficult for an experiment to graduate into a usable tool.
Model training today can also be challenging as the process can require large amounts of data and time, even on high-end hardware. For instance, training reasonable models for speech synthesis can take several days and requires hours of pre-recorded, tagged audio, which is time-consuming to collect. Obviously, you also won’t get everything right the first time and so you might need to iterate on your data, scripts or settings and go through this long training process several times.
What also makes it tough to increase the adoption of these custom tools is that the hype from a couple of years ago has set unrealistic expectations on machine learning in some cases. With these high expectations, when a first draft shows result only 60-80% of the quality of the content already being produced, there is a feeling of disappointment – making it difficult to generate buy-in from a product team to take the experiment further. In the future, I hope we see creators building games with machine learning techniques in mind – adapting their expectations and artistic styles to these techniques.
At the end of the day, none of these challenges are particularly different to those of other emerging technologies. Over time, these difficulties will be overcome as more and more companies look to release products around these techniques.
Hyperlight: What are the machine learning trends that will mark 2021 and beyond?
John Wordsworth: From a game-development perspective, it is exciting to see some machine learning techniques already being commercialised for easier access and general use. For instance, Adobe Photoshop 2021 saw the introduction of Neural Filters, which have brought image colourisation and skin smoothing to the general user. Equally, we are also witnessing a range of companies working on using machine learning to produce more and more realistic results with speech generation. While these might not be quite ready to replace the full gamut of emotions that can be produced by voice actors, I can already see that it would be viable to use these applications to create much better game prototypes or for speech synthesis for accessibility purposes – like in-game text-chat from other players.
Over the coming few years, I expect to see a lot more innovation from research being done in the machine learning industry at large. Many of these will have interesting applications for game development. I also expect that a large number of game development tools will appear across all of the different disciplines required to build a game, and these solutions will come with more and more customisation options for developers.
Outside of Game Development, I’m afraid I have no idea – but the two things I do know are that it’ll be exciting and it’ll move fast!
Add comment