This repository contains a template scene for interfacing with an AI-based NPC using OpenAI API, Azure Voice API, Google Cloud Speech to Text, and Oculus Lip Sync. The project uses Unity 2021.3.x. Demonstration here
Before running the scene, you'll need to set up the following services and create a configuration file for the application to read at runtime:
Review the setup instructions for the following repositories that are used in this project:
In the StreamingAssets folder, create a services_config.json file with the following template, and replace the placeholder values with your own API keys and region information:
{
"OpenAI_APIKey": "your_openai_api_key",
"AzureVoiceSubscriptionKey": "your_azure_voice_subscription_key",
"AzureVoiceRegion": "your_azure_voice_region"
}Create a gcp_credentials.json file for Google Cloud runtime to read configuration properties from using the following template:
{
"type": "service_account",
"project_id": "YOUR PROJECT ID",
"private_key_id": "YOUR PRIVATE KEY ID",
"private_key": "YOUR PRIVATE KEY",
"client_email": "YOUR CLIENT EMAIL",
"client_id": "YOUR CLIENT ID",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "YOUR CLIENT CERT URL"
}Sometimes on the first time opening the project, you might get the following error message: "Multiple precompiled assemblies with the same name Newtonsoft.Json.dll included on the current platform." This error occurs because Unity enforces a Newtonsoft import due to its services.core dependency. To resolve this error:
- Close the project
- Open the file explorer and navigate to
\Library\PackageCache\[email protected]\Plugins - Delete the
Newtonsoft.dllfile - Reopen the project and hit "Ignore"
- Delete
Newtonsoft.dllagain from the same location - The import should now complete.
Once you've set up the configuration files, you can run the scene and begin interacting with the NPC by asking it questions. The NPC will respond within seconds.
Models Used: This project currently uses the TextDavinciV3 model provided by OpenAI's GPT-3 API. Additionally, there is a separate branch attempting to update to the latest publicly available model, ChatGpt3_5Turbo.
In addition to the APIs and packages mentioned above, this project also uses the Meta Movement SDK. More information on this SDK can be found in its GitHub repository at https://github.com/oculus-samples/Unity-Movement.