Unity is the leading platform for creating and operating real-time 3D (RT3D) content. It helps creators of all kinds, from individuals to professional studios, to build immersive games and experiences. But Unity is more than just a platform. It’s also a pioneer in using artificial intelligence (AI) to enhance the creation and operation of RT3D content.
In this blog post, we will explore how Unity leverages AI in four different domains: simulation, content creation, game optimization, and interaction. We will also see how Unity is building an open and accessible AI ecosystem that will enable creators to do more with less and unleash their imagination.
AI for Simulation: Creating Realistic Virtual Environments
Simulation is the process of creating realistic virtual environments that mimic real-world scenarios. Simulation can be used for various purposes, such as testing, training, education, entertainment, and research.
Unity has been using AI for simulation for years, especially in the fields of automotive, aerospace, robotics, and healthcare. One of the examples of how Unity uses AI for simulation is AirSim, an open-source simulator for autonomous vehicles that Unity has partnered with Microsoft to create. AirSim uses Unity’s physics engine and graphics capabilities to generate realistic scenarios that can train and test self-driving cars. AirSim also uses AI to generate synthetic data, such as images and sensor readings, that can augment real-world data and improve the performance of machine learning models.
Another example is Unity Simulation, a cloud-based service that allows creators to run millions of simulations in parallel. Unity Simulation can be used to test various hypotheses, optimize parameters, or generate data for machine learning models. For instance, Unity Simulation was used by Synthesis AI, a company that creates synthetic faces for computer vision applications. Synthesis AI used Unity Simulation to generate diverse and high-quality facial images that can train facial recognition systems.
AI for Content Creation: Generating New Content from Existing Data
Content creation is the process of making assets, such as models, textures, animations, sounds, or scripts, that are used in games and other RT3D applications. Content creation can be time-consuming, costly, and challenging for many creators.
That’s why Unity is developing AI-powered tools that can help creators generate new content based on existing data. One of the most exciting branches of AI that can enable content creation is generative AI. Generative AI is a type of AI that can create new content from scratch or modify existing content based on some input or criteria. For example, generative AI can create realistic images, text, audio, or code.
Unity is working on natural-language generative tools that can help creators make code, animations, physical effects, or other RT3D content using simple commands or descriptions. These tools will complement existing deterministic tools and workflows , helping creators fill gaps in skill sets and resources . For example, a creator could use a natural-language generative tool to create a character animation by describing the desired motion in plain English.
Unity is also building technology in the Unity Editor to better define what AI draws from . This includes using appropriate and licensable datasets for generating content and integrating AI techniques that are customized to specific content . For example, Unity has partnered with NVIDIA to integrate NVIDIA’s Omniverse Kaolin into the Unity Editor . Omniverse Kaolin is a toolkit that uses generative AI to create 3D models from 2D sketches or images.
AI for Game Optimization: Improving Performance and Quality
Game optimization is the process of improving the performance and quality of games and other RT3D applications. Game optimization can involve various aspects, such as graphics , physics , networking , or analytics .
Unity has been using AI for game optimization for years , especially in its cloud services that help customers improve user acquisition , engagement , and revenue . One of the examples of how Unity uses AI for game optimization is GameTune, a service that uses AI to automatically test different game configurations and find the optimal one for each player segment . GameTune helps developers optimize their games without manual testing or coding.
Another example is Barracuda, a technology that allows designers to build game loops that rely on inference on devices from mobile to console to web and PC . Inference is the process of feeding data through a machine learning model to produce an output. For example, inference can be used to generate realistic animations , facial expressions , speech synthesis , or dialogue for NPCs based on their personality traits or emotions .
AI for Interaction: Enabling New Forms of Engagement
Interaction is the process of engaging users with games and other RT3D applications. Interaction can involve various elements, such as user interface , input devices , feedback , or gameplay .
Unity sees AI as not just a tool for creation but also an opportunity for new forms of interaction . With their experience and technology , they are well-positioned to help creators succeed while adopting generative AI .
One of the examples of how Unity uses AI for interaction is ML-Agents Toolkit, an open-source project that enables games and simulations to serve as environments for training intelligent agents . ML-Agents uses reinforcement learning , imitation learning , neuroevolution , or other machine learning methods to teach agents how to behave and learn from their own actions and experiences. For instance, ML-Agents was used by Electronic Arts to create intelligent football players for FIFA .
Another example is Unity’s vision for using AI in game development . They envision NPCs that come to life , diffusion content as a gameplay mechanism , and boundary-less user-generated content . But they know that their creators will do far more with this technology than they could ever even dream .
How Unity Empowers Creators with AI
Unity is a company that empowers creators with tools and services to succeed. From individuals building their first games to professional studios working on major franchises, Unity’s AI and ML-driven techniques reduce complexity, accelerate creation, and unlock new ideas
Unity has been using AI in simulation, content creation, game optimization, and interaction for years. The current explosion of generative AI presents an opportunity to go even further. With the Unity Editor, runtime, data, and network, Unity is uniquely positioned to help creators succeed with generative AI.
Most digital content today is 2D and linear – sprites, photos, film frames, building floor plans, or source code. AI data models train on this information to learn and create content. Unity enables real-time training of models based on unique datasets produced in the creation and operation of RT3D experiences. This allows them to build ever-richer services on top of Unity and provide extraordinary capabilities for their partners.
The Unity Network consists of analytics tools, ad networks, publishing systems, and cloud services, reaching a combined total of over 4 billion users each month. Each service field yields data that can be used to help customers improve user acquisition, engagement, and revenue. Unity has been using neural networks to optimize these systems for over three years.
Video games have always used some kind of generative AI to create content. This technology can be a powerful ally for developers who want to do more with less. Unity has a vision for AI that will enable creators to realize their dreams.