Integrating Cortana with Unity (Unite 2015)

The Unite Europe session is 6/25 Track 4, at 12-12:30 in WesterLiefde
The Unite Europe session is 6/25 Track 4, at 12-12:30 in WesterLiefde

IMPORTANT: This demo shows Unity Windows 10 integration. We demonstrated and shared integration at Game Jam 10, if you attended that demo you can download Unity Windows 10 beta integration setup. This example requires Unity Windows 10 integration beta.

This is the sample I’ll be presenting at Unite Europe 2015 event 6/25 (see Track 4) to demonstrate how to integrate Unity with Cortana in Windows 10, among other cool things.¬†Fortunately, it’s very easy to do: if you are a Unity developer, integrating Unity with Cortana can certainly add flavor to your Windows version of the app. And for users it just means more fun! So let’s get started…

The Idea: Mecanim and Cortana

I’ve been thinking of combining Unity mecanim animation with Cortana to make my animated character react to human voice commands. Cortana, besides being a personal assistant and just a friendly artificial intelligence inside my Windows devices can also help with basic speech things: parsing human voice commands and as it turns out, with Unity games. Here’s what my app looks like:

Cortana integrated app built with Unity
Cortana integrated app built with Unity

I’m using a free asset from Mixamo for my animation character (in fact you can use any character with animations from Unity Asset Store). I want to activate Cortana with standard Windows “Hey, Cortana” and then make my character dance, walk, run etc, by speaking a voice command. Easy?

Dancing avatar
Dancing avatar

You can find the code and also instructions for this example at the bottom of this post. Instructions:

1. After loading the Unity project, open the MainScene scene.

2. Now let’s build a Windows specific project. Open Build Settings. Since we will use Cortana in our project, click Player Settings and make sure Microphone is checked.

3.¬†Now, select Windows Store output, then choose Universal 10 as a target. If you do not see this target, then you don’t have a Windows 10 integrated Unity. Make sure you select XAML as an output project type.

4. Click Build…, Unity will prompt you for output folder, this is where a native Visual Studio project will be generated.

5. Make sure to make the following changes to App.xaml.cs

Make AppCallbacks public instead of private:

public AppCallbacks appCallbacks;

Change OnLaunched event as follows:

protectedasyncoverridevoid OnLaunched(LaunchActivatedEventArgs args)

{

splashScreen = args.SplashScreen;

try

{

StorageFile vcdStorageFile = awaitPackage.Current.InstalledLocation.GetFileAsync(@”CortanaCommands.xml”);

await Windows.ApplicationModel.VoiceCommands.VoiceCommandDefinitionManager.InstallCommandDefinitionsFromStorageFileAsync(vcdStorageFile);

awaitSpeechHelper.InitialiseSpeechRecognition();

}

catch (Exception ex)

{

System.Diagnostics.Debug.WriteLine(“Installing Voice Commands Failed: “ + ex.ToString());

}

InitializeUnity(args.Arguments);

}

Then, update OnActivated event to include voice command activation:

protectedoverridevoid OnActivated(IActivatedEventArgs args)

{

string appArgs = “”;

switch (args.Kind)

{

caseActivationKind.Protocol:

ProtocolActivatedEventArgs eventArgs = args asProtocolActivatedEventArgs;

splashScreen = eventArgs.SplashScreen;

appArgs += string.Format(“Uri={0}”, eventArgs.Uri.AbsoluteUri);

break;

//Add VoiceCommand start detection and use the SpeechHelper handler

caseActivationKind.VoiceCommand:

SpeechHelper.HandleSpeechCommand(args);

break;

}

InitializeUnity(appArgs);

}

6. Now, drop CortanaCommands.xml, SpeechHelper.cs, ToastNotificationHelper.cs from WindowsSpecificCode folder to your project, build and run it. For your convenience I also added App.xaml.cs file.

7. When the project starts, you can use Windows 10 Cortana to give it commands such as “Avatar dance” or “Avatar run”.

 

 

Download code