LiveLink UE MoCap

The purpose of the LiveLink UE MoCap IOS app is to stream facial trans­formations from your iPhone / iPad into your Unreal Engine animation.

LiveLink UE MoCap is based on the Apple© ARKit ARFaceTracking API, which provides 51 realtime blendshape values of your face. Blendshape values describe the relative movement of defined areas from your face in float values in the range of 0 … 1. 

This page will guide you through the steps needed for your first animation generated with your IOS device.

Download on the App Store

Prerequesites

iPhone / iPad devices:

You will need an iPhone / iPad with FaceID capability and IOS >= 12.1 to get things working.
These devices are equipped with the Apple© TrueDepth camera, which is intentionally used for unlocking your device by scanning your face.
But Apple© also provides a SDK, which give you access the the captured 3D data and which can be used for animating 3D modelled faces.

iPhone
iPhone 11 Pro Max
iPhone 11 Pro
iPhone 11
iPhone XS Max
iPhone XS
iPhone XR
iPhone X
iPad
iPad Pro (12,9″, 3. Generation)
iPad Pro (11″)

Unreal Engine requirements

LiveLink UE MoCap is tested with Unreal Engine 4.24, the current internal protocol version is 5 which means, that it might not work with older versions.

Enable the following plugins in UE4 (menu: Edit -> Plugins): LiveLink, Apple ARKit, and Apple ARKit face Support.

Restart your Unreal Engine after plugin activation.

Connecting your mobile with the Unreal Engine

Connectivity is handled using UDP. With your activated Live Link plugin in the UE4 application, an UDP server is launched on every UE4 startup.
Now start your mobile device, lauch the LiveLink UE MoCap app, edit the IP in the apps settings to the IP of the computer UE4 is running and switch the connection on.
Please make sure, that your mobile device and the UE4’s computer are in the same network.

If you now open the LiveLink dialog (menu: Window -> LiveLink), you should see the active Apple AR Face Tracking source.

If you don’t see any entry in the list, check the IP in the LiveLink UE Mocap app again, and make sure the connect switch is on. If this doesn’t help, check your firewall options, and check if the LiveLink plugin version in your UE4 is the latest one.

Using the blendshape values in Unreal Engine

LiveLink UE MoCap sends an array of float values in a fixed order to the Live Link plugin, which was launched on start of your UE application.
There is a fixed name mapping to 61 blendshape values inside the Live Link plugin, from that 52 are implemented in LiveLink UE MoCap app.
The remaining 9 values are send with the value 0 and describe additional eye / lid values and head / neck transformation. They will be implemented in one of the next versions.

A simple way to get the blendshape values into your project, is creating an animation blueprint and select Live Link Preview Controller in the selectbox on the right.

If you need more control over the data (e.g. removing noise), you can place a Live Link Pose node in your AnimGraph, and do whatever is necessary between Live Link Pose node and Output Pose node.

The names assigned to the blendshape values match the Face AR Sample, for reference or just a try you can install the project using the Epic installer. You can find detailed information here: Face AR Sample

Most probably your morph targets have different names compared to the Face AR Sample, and you want to remap / transform / modify them before you apply them to your mesh. Remapping can be simply done creating a LiveLinkRemapAsset.

There are select boxes in both above mentioned LiveLink integration methods where you can select your own remap asset.

Following the complete list of names, which are currently supported by the LiveLink UE MoCap app:

eyeBlinkLeft mouthClose
eyeLookDownLeft mouthFunnel
eyeLookInLeft mouthPucker
eyeLookOutLeft mouthLeft
eyeLookUpLeft mouthRight
eyeSquintLeft mouthSmileLeft
eyeWideLeft mouthSmileRight
eyeBlinkRight mouthFrownLeft
eyeLookDownRight mouthFrownRight
eyeLookInRight mouthDimpleLeft
eyeLookOutRight mouthDimpleRight
eyeLookUpRight mouthStretchLeft
eyeSquintRight mouthStretchRight
eyeWideRight mouthRollLower
jawForward mouthRollUpper
jawLeft mouthShrugLower
jawRight mouthShrugUpper
jawOpen mouthPressLeft