// Copyright (c) 2017-2024, The Khronos Group Inc.
//
// SPDX-License-Identifier: CC-BY-4.0

include::{generated}/meta/XR_FB_face_tracking2.adoc[]

*Last Modified Date*::
    2023-10-06

*IP Status*::
    No known IP claims.

*Contributors*::
    Jaebong Lee, Meta +
    Dikpal Reddy, Meta +
    Igor Tceglevskii, Meta +
    Bill Orr, Meta +
    Scott Ramsby, Meta

==== Overview

This extension enables applications to get weights of blend shapes.
It also enables applications to render facial expressions in XR experiences.

It is recommended to choose this extension over the
apiext:XR_FB_face_tracking extension, if it is supported by the runtime,
because this extension provides the following two additional capabilities to
the application:

* This extension provides additional seven blend shapes that estimate tongue
  movement.
* This extension allows an application and the runtime to communicate about
  the data sources that are used to estimate facial expression in a
  cooperative manner.

Face tracking data is sensitive personal information and is closely linked
to personal privacy and integrity.
Applications storing or transferring face tracking data should: always ask
the user for active and specific acceptance to do so.

If the runtime supports a permission system to control application access to
the face tracker, then the runtime must: set the pname:isValid field to
ename:XR_FALSE on the supplied slink:XrFaceExpressionWeights2FB structure
until the user allows the application to access the face tracker.
When the application access has been allowed, the runtime should: set
pname:isValid on the supplied slink:XrFaceExpressionWeights2FB structure to
ename:XR_TRUE.

Some permission systems may: control access to the eye tracking separately
from access to the face tracking, even though the eyes are part of the face.
In case the user denied tracking of the eyes, yet, allowed tracking of the
face, then the runtime must: set the pname:isEyeFollowingBlendshapesValid
field to ename:XR_FALSE on the supplied slink:XrFaceExpressionWeights2FB for
indicating that eye tracking data is not available, but at the same time
may: set the pname:isValid field to ename:XR_TRUE on the supplied
slink:XrFaceExpressionWeights2FB for indicating that another part of the
face is tracked properly.

==== Inspect system capability

[open,refpage='XrSystemFaceTrackingProperties2FB',type='structs',desc='System property for face tracking',xrefs='']
--

include::{generated}/api/structs/XrSystemFaceTrackingProperties2FB.adoc[]

.Member Descriptions
****
* pname:type is the elink:XrStructureType of this structure.
* pname:next is code:NULL or a pointer to the next structure in a structure
  chain.
  No such structures are defined in core OpenXR or this extension.
* pname:supportsVisualFaceTracking is an basetype:XrBool32, indicating if
  the current system is capable of receiving face tracking input that is
  estimated based on visual data source.
* pname:supportsAudioFaceTracking is an basetype:XrBool32, indicating if the
  current system is capable of receiving face tracking input that is
  estimated based on audio data source.
****

An application can: inspect whether the system is capable of receiving face
tracking input by extending the slink:XrSystemProperties with
slink:XrSystemFaceTrackingProperties2FB structure when calling
flink:xrGetSystemProperties.

If an application calls flink:xrCreateFaceTracker2FB only with unsupported
slink:XrFaceTrackerCreateInfo2FB::pname:requestedDataSources, the runtime
must: return ename:XR_ERROR_FEATURE_UNSUPPORTED from
flink:xrCreateFaceTracker2FB.
For example, if an application calls flink:xrCreateFaceTracker2FB only with
ename:XR_FACE_TRACKING_DATA_SOURCE2_AUDIO_FB in
slink:XrFaceTrackerCreateInfo2FB::pname:requestedDataSources when the
runtime returns ename:XR_FALSE for pname:supportsAudioFaceTracking, the
runtime must: return ename:XR_ERROR_FEATURE_UNSUPPORTED from
flink:xrCreateFaceTracker2FB.

include::{generated}/validity/structs/XrSystemFaceTrackingProperties2FB.adoc[]

--

==== Create a face tracker handle

[open,refpage='XrFaceTracker2FB',type='handles',desc='Represents a tracker for a face blend shapes.']
--
The slink:XrFaceTracker2FB handle represents the resources for face
tracking.

include::{generated}/api/handles/XrFaceTracker2FB.adoc[]

This handle is used to obtain blend shapes using the
flink:xrGetFaceExpressionWeights2FB function.

--

[open,refpage='xrCreateFaceTracker2FB',type='protos',desc='Create a face tracking handle.',xrefs='']
--
The flink:xrCreateFaceTracker2FB function is defined as:

include::{generated}/api/protos/xrCreateFaceTracker2FB.adoc[]

.Parameter Descriptions
****
* pname:session is an slink:XrSession in which the face tracker will be
  active.
* pname:createInfo is the slink:XrFaceTrackerCreateInfo2FB used to specify
  the face tracker.
* pname:faceTracker is the returned slink:XrFaceTracker2FB handle.
****

An application can: create an slink:XrFaceTracker2FB handle using
flink:xrCreateFaceTracker2FB function.

If the system does not support face tracking, the runtime must: return
ename:XR_ERROR_FEATURE_UNSUPPORTED from flink:xrCreateFaceTracker2FB.
In this case, the runtime must: return ename:XR_FALSE for both
slink:XrSystemFaceTrackingProperties2FB::pname:supportsVisualFaceTracking
and slink:XrSystemFaceTrackingProperties2FB::pname:supportsAudioFaceTracking
when the function flink:xrGetSystemProperties is called, so that the
application can: avoid creating a face tracker.

include::{generated}/validity/protos/xrCreateFaceTracker2FB.adoc[]

--

[open,refpage='XrFaceTrackerCreateInfo2FB',type='structs',desc='Information to create a facial expression handle',xrefs='']
--
The slink:XrFaceTrackerCreateInfo2FB structure is described as follows:

include::{generated}/api/structs/XrFaceTrackerCreateInfo2FB.adoc[]

.Member Descriptions
****
* pname:type is the elink:XrStructureType of this structure.
* pname:next is code:NULL or a pointer to the next structure in a structure
  chain.
  No such structures are defined in core OpenXR or this extension.
* pname:faceExpressionSet is an elink:XrFaceExpressionSet2FB that describes
  the set of blend shapes to retrieve.
* pname:requestedDataSourceCount is the number of elements in the
  pname:requestedDataSources array.
* pname:requestedDataSources is an array of
  elink:XrFaceTrackingDataSource2FB that the application accepts.
  The order of values in the array has no significance.
****

The slink:XrFaceTrackerCreateInfo2FB structure describes the information to
create an slink:XrFaceTracker2FB handle.

Runtimes may: support a variety of data sources for estimations of facial
expression, and some runtimes and devices may: use data from multiple data
sources.
The application tells the runtime all data sources that the runtime may: use
to provide facial expressions for the application.

Because the device setting may: change during a running session, the runtime
may: return a valid slink:XrFaceTracker2FB handle even if the device is
unable to estimate facial expression using the data sources requested by the
application's call to flink:xrCreateFaceTracker2FB.
The runtime must: instead return ename:XR_ERROR_FEATURE_UNSUPPORTED from
flink:xrCreateFaceTracker2FB, if for example the runtime believes it will
never be able to satisfy the request.

If pname:requestedDataSourceCount is `0`, the runtime may: choose any
supported data source, preferably one that is more expressive than the
others.

If any value in pname:requestedDataSources is duplicated the runtime must:
return ename:XR_ERROR_VALIDATION_FAILURE from the call to
flink:xrCreateFaceTracker2FB.

include::{generated}/validity/structs/XrFaceTrackerCreateInfo2FB.adoc[]
--


[open,refpage='XrFaceExpressionSet2FB',type='enums',desc='The set of blend shapes of a facial expression to track.',xrefs='XrFaceTrackerCreateInfo2FB']
--
The elink:XrFaceExpressionSet2FB enum describes the set of blend shapes of a
facial expression to track when creating an slink:XrFaceTracker2FB.

include::{generated}/api/enums/XrFaceExpressionSet2FB.adoc[]

include::{generated}/api/enums/XrFaceExpressionSet2FB.comments-box.adoc[]
--

[open,refpage='XrFaceTrackingDataSource2FB',type='enums',desc='Data source that is used to estimate facial expression.',xrefs='XrFaceTrackerCreateInfo2FB']
--
The elink:XrFaceTrackingDataSource2FB enumeration is defined as:

include::{generated}/api/enums/XrFaceTrackingDataSource2FB.adoc[]

.Enumerant Descriptions
****
* ename:XR_FACE_TRACKING_DATA_SOURCE2_VISUAL_FB - This value indicates that
  the face tracking data source supports using visual data to estimate
  facial expression.
  The runtime may: also use audio to further improve the quality of the
  tracking.
* ename:XR_FACE_TRACKING_DATA_SOURCE2_AUDIO_FB - This value indicates that
  the face tracking data source supports using audio data to estimate facial
  expression.
  The runtime must: not use visual data for this data source.
****

--

==== Delete a face tracker handle

[open,refpage='xrDestroyFaceTracker2FB',type='protos',desc='Destroy a facial expressions handle',xrefs='']
--
The flink:xrDestroyFaceTracker2FB function is defined as:

include::{generated}/api/protos/xrDestroyFaceTracker2FB.adoc[]

.Parameter Descriptions
****
* pname:faceTracker is an slink:XrFaceTracker2FB previously created by
  flink:xrCreateFaceTracker2FB.
****

The flink:xrDestroyFaceTracker2FB function releases the pname:faceTracker
and the underlying resources when face tracking experience is over.

include::{generated}/validity/protos/xrDestroyFaceTracker2FB.adoc[]
--

==== Obtain facial expressions

[open,refpage='xrGetFaceExpressionWeights2FB',type='protos',desc='Obtain blend shapes of facial expression',xrefs='']
--
The flink:xrGetFaceExpressionWeights2FB function is defined as:

include::{generated}/api/protos/xrGetFaceExpressionWeights2FB.adoc[]

.Parameter Descriptions
****
* pname:faceTracker is an slink:XrFaceTracker2FB previously created by
  flink:xrCreateFaceTracker2FB.
* pname:expressionInfo is a pointer to slink:XrFaceExpressionInfo2FB
  describing information to obtain face expression.
* pname:expressionWeights is a pointer to slink:XrFaceExpressionWeights2FB
  receiving the returned facial expression weights.
****

The flink:xrGetFaceExpressionWeights2FB function return blend shapes of
facial expression at a given time.

include::{generated}/validity/protos/xrGetFaceExpressionWeights2FB.adoc[]
--

[open,refpage='XrFaceExpressionInfo2FB',type='structs',desc='Describes the information to obtain facial expression',xrefs='xrGetFaceExpressionWeights2FB']
--
The slink:XrFaceExpressionInfo2FB structure is defined as:

include::{generated}/api/structs/XrFaceExpressionInfo2FB.adoc[]

.Member Descriptions
****
* pname:type is the elink:XrStructureType of this structure.
* pname:next is code:NULL or a pointer to the next structure in a structure
  chain.
  No such structures are defined in core OpenXR or this extension.
* pname:time is an basetype:XrTime at which the facial expression weights
  are requested.
****

The slink:XrFaceExpressionInfo2FB structure describes the information to
obtain facial expression.
The application should: pass a time equal to the predicted display time for
the rendered frame.
The system must: employ appropriate modeling to provide expressions for this
time.

include::{generated}/validity/structs/XrFaceExpressionInfo2FB.adoc[]
--

[open,refpage='XrFaceExpressionWeights2FB',type='structs',desc='Returns the facial expression',xrefs='xrGetFaceExpressionWeights2FB']
--
The slink:XrFaceExpressionWeights2FB structure is defined as:

include::{generated}/api/structs/XrFaceExpressionWeights2FB.adoc[]

.Member Descriptions
****
* pname:type is the elink:XrStructureType of this structure.
* pname:next is code:NULL or a pointer to the next structure in a structure
  chain.
  No such structures are defined in core OpenXR or this extension.
* pname:weightCount is a code:uint32_t describing the count of elements in
  pname:weights array.
* pname:weights is a pointer to an application-allocated array of code:float
  that will be filled with weights of facial expression blend shapes.
* pname:confidenceCount is a code:uint32_t describing the count of elements
  in pname:confidences array.
* pname:confidences is a pointer to an application-allocated array of
  code:float that will be filled with confidence of tracking specific parts
  of a face.
* pname:isValid is an basetype:XrBool32 which indicates that the tracked
  expression weights are valid.
* pname:isEyeFollowingBlendshapesValid is an basetype:XrBool32 which
  indicates if the 8 expression weights with prefix
  `XR_FACE_EXPRESSION2_EYES_LOOK_*` are valid.
* pname:dataSource is an elink:XrFaceTrackingDataSource2FB which indicates
  the data source that was used to estimate the facial expression.
* pname:time is an basetype:XrTime time at which the returned expression
  weights are tracked or extrapolated to.
  Equals the time at which the expression weights were requested if the
  extrapolating at the time was successful.
****

slink:XrFaceExpressionWeights2FB structure returns the facial expression.

The runtime must: return ename:XR_ERROR_VALIDATION_FAILURE if
pname:weightCount is not equal to the number of blend shapes defined by the
elink:XrFaceExpressionSet2FB used to create the slink:XrFaceTracker2FB.

The runtime must: return ename:XR_ERROR_VALIDATION_FAILURE if
pname:confidenceCount is not equal to the number of confidence areas defined
by the elink:XrFaceExpressionSet2FB used to create the
slink:XrFaceTracker2FB.

The runtime must: return pname:weights representing the weights of blend
shapes of current facial expression.

The runtime must: update the pname:weights array ordered so that the
application can: index elements using the corresponding facial expression
enum (e.g. elink:XrFaceExpression2FB) as described by
elink:XrFaceExpressionSet2FB when creating the slink:XrFaceTracker2FB.
For example, when the slink:XrFaceTracker2FB is created with
ename:XR_FACE_EXPRESSION_SET2_DEFAULT_FB, the application sets the
pname:weightCount to ename:XR_FACE_EXPRESSION2_COUNT_FB, and the runtime
must: fill the pname:weights array ordered so that it can: be indexed by the
elink:XrFaceExpression2FB enum.

The runtime must: update the pname:confidences array ordered so that the
application can: index elements using the corresponding confidence area enum
(e.g. elink:XrFaceConfidence2FB) as described by
elink:XrFaceExpressionSet2FB when creating the slink:XrFaceTracker2FB.
For example, when the slink:XrFaceTracker2FB is created with
ename:XR_FACE_EXPRESSION_SET2_DEFAULT_FB, the application sets the
pname:confidenceCount to ename:XR_FACE_CONFIDENCE2_COUNT_FB, and the runtime
must: fill the pname:confidences array ordered so that it can: be indexed by
the elink:XrFaceConfidence2FB enum.

The runtime must: set pname:isValid to ename:XR_FALSE and it must: also set
all elements of pname:weights to zero, if one of the following is true:

* the face tracker failed to track or lost track of the face
* the application lost focus
* the consent for face tracking was denied
* the runtime is unable to estimate facial expression from the data sources
  specified when flink:xrCreateFaceTracker2FB function was called

If the returned pname:isValid is ename:XR_TRUE, the runtime must: return all
weights (or all weights except eyes related weights, see
pname:isEyeFollowingBlendshapesValid).

The runtime must: set pname:isEyeFollowingBlendshapesValid to ename:XR_FALSE
and it must: also set 8 expression weights with prefix
`XR_FACE_EXPRESSION2_EYES_LOOK_*` to zero, if one of the following is true:

* the eye tracking driving blendshapes with prefix
  `XR_FACE_EXPRESSION2_EYES_LOOK_*` lost track
* the consent for eye tracking was denied

include::{generated}/validity/structs/XrFaceExpressionWeights2FB.adoc[]
--

==== Example code for obtaining facial expression

The following example code demonstrates how to obtain all weights for facial
expression blend shapes.

[source,C++]
----
XrInstance instance; // previously initialized
XrSystemId systemId; // previously initialized
XrSession session;   // previously initialized

// Confirm face tracking system support.
XrSystemFaceTrackingProperties2FB faceTrackingSystemProperties{
    XR_TYPE_SYSTEM_FACE_TRACKING_PROPERTIES2_FB};
XrSystemProperties systemProperties{XR_TYPE_SYSTEM_PROPERTIES,
                                    &faceTrackingSystemProperties};
CHK_XR(xrGetSystemProperties(instance, systemId, &systemProperties));
if (!faceTrackingSystemProperties.supportsVisualFaceTracking &&
    !faceTrackingSystemProperties.supportsAudioFaceTracking) {
    // The system does not support face tracking
    return;
}

// Get function pointer for xrCreateFaceTracker2FB.
PFN_xrCreateFaceTracker2FB pfnCreateFaceTracker2FB;
CHK_XR(xrGetInstanceProcAddr(instance, "xrCreateFaceTracker2FB",
                             reinterpret_cast<PFN_xrVoidFunction*>(
                             &pfnCreateFaceTracker2FB)));

// Create a face tracker for default set of facial expressions.
XrFaceTracker2FB faceTracker = {};
{
    XrFaceTrackerCreateInfo2FB createInfo{XR_TYPE_FACE_TRACKER_CREATE_INFO2_FB};
    createInfo.faceExpressionSet = XR_FACE_EXPRESSION_SET2_DEFAULT_FB;
    // This tells the runtime that the application can take
    // facial expression from any of two data sources.
    createInfo.requestedDataSourceCount = 2;
    XrFaceTrackingDataSource2FB dataSources[2] = {
        XR_FACE_TRACKING_DATA_SOURCE2_VISUAL_FB,
        XR_FACE_TRACKING_DATA_SOURCE2_AUDIO_FB};
    createInfo.requestedDataSources = dataSources;
    CHK_XR(pfnCreateFaceTracker2FB(session, &createInfo, &faceTracker));
}

// Allocate buffers to receive facial expression data before frame
// loop starts.
float weights[XR_FACE_EXPRESSION2_COUNT_FB];
float confidences[XR_FACE_CONFIDENCE2_COUNT_FB];

XrFaceExpressionWeights2FB expressionWeights{XR_TYPE_FACE_EXPRESSION_WEIGHTS2_FB};
expressionWeights.weightCount = XR_FACE_EXPRESSION2_COUNT_FB;
expressionWeights.weights = weights;
expressionWeights.confidenceCount = XR_FACE_CONFIDENCE2_COUNT_FB;
expressionWeights.confidences = confidences;

// Get function pointer for xrGetFaceExpressionWeights2FB.
PFN_xrGetFaceExpressionWeights2FB pfnGetFaceExpressionWeights;
CHK_XR(xrGetInstanceProcAddr(instance, "xrGetFaceExpressionWeights2FB",
                             reinterpret_cast<PFN_xrVoidFunction*>(
                             &pfnGetFaceExpressionWeights)));
while (1) {
    // ...
    // For every frame in the frame loop
    // ...
    XrFrameState frameState; // previously returned from xrWaitFrame
    const XrTime time = frameState.predictedDisplayTime;

    XrFaceExpressionInfo2FB expressionInfo{XR_TYPE_FACE_EXPRESSION_INFO2_FB};
    expressionInfo.time = time;

    CHK_XR(pfnGetFaceExpressionWeights(faceTracker, &expressionInfo, &expressionWeights));

    if (expressionWeights.isValid) {
        // If you want to do something depending on the data source.
        if (expressionWeights.dataSource == XR_FACE_TRACKING_DATA_SOURCE2_VISUAL_FB) {
            // do something when visual or audiovisual data source was used.
        } else if (expressionWeights.dataSource == XR_FACE_TRACKING_DATA_SOURCE2_AUDIO_FB) {
            // do something when audio data source was used.
        }

        for (uint32_t i = 0; i < XR_FACE_EXPRESSION2_COUNT_FB; ++i) {
            // weights[i] contains a weight of specific blend shape
        }
    }
}
----

==== Conventions of blend shapes

This extension defines 70 blend shapes for tracking facial expressions.

[open,refpage='XrFaceExpression2FB',type='enums',desc='The name of blend shapes that are trackable',xrefs='']
--
include::{generated}/api/enums/XrFaceExpression2FB.adoc[]

--

[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Brow_Lowerer_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_BROW_LOWERER_L_FB knits and lowers
the left brow area and lowers central forehead.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Brow_Lowerer_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_BROW_LOWERER_R_FB knits and lowers
the right brow area and lowers central forehead.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Cheek_Puff_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_CHEEK_PUFF_L_FB fills the left cheek
with air causing them to round and extend outward.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Cheek_Puff_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_CHEEK_PUFF_R_FB fills the right cheek
with air causing them to round and extend outward.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Cheek_Raiser_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_CHEEK_RAISER_L_FB tightens the outer rings
of the left eye orbit and squeezes the lateral left eye corners.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Cheek_Raiser_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_CHEEK_RAISER_R_FB tightens the outer rings
of the right eye orbit and squeezes the lateral right eye corners.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Cheek_Suck_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_CHEEK_SUCK_L_FB sucks the left cheek inward
and against the teeth to create a hollow effect in the cheek.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Cheek_Suck_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_CHEEK_SUCK_R_FB sucks the right cheek inward
and against the teeth to create a hollow effect in the cheek.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Chin_Raiser_B.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_CHIN_RAISER_B_FB pushes the skin of the chin
and the lower lip upward.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Chin_Raiser_T.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_CHIN_RAISER_T_FB pushes up the top lip.
This is induced by the upward force from
ename:XR_FACE_EXPRESSION2_CHIN_RAISER_B_FB.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Dimpler_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_DIMPLER_L_FB pinches the left lip corner against
the teeth, drawing them slightly backward and often upward in the process.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Dimpler_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_DIMPLER_R_FB pinches the right lip corner against
the teeth, drawing them slightly backward and often upward in the process.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Eyes_Closed_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_EYES_CLOSED_L_FB lowers the top eyelid to cover
the left eye.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Eyes_Closed_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_EYES_CLOSED_R_FB lowers the top eyelid to cover
the right eye.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Eyes_Look_Down_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_EYES_LOOK_DOWN_L_FB moves the left eyelid consistent
with downward gaze.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Eyes_Look_Down_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_EYES_LOOK_DOWN_R_FB moves the right eyelid consistent
with downward gaze.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Eyes_Look_Left_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_EYES_LOOK_LEFT_L_FB moves the left eyelid consistent
with leftward gaze.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Eyes_Look_Left_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_EYES_LOOK_LEFT_R_FB moves the right eyelid consistent
with leftward gaze.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Eyes_Look_Right_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_EYES_LOOK_RIGHT_L_FB moves the left eyelid consistent
with rightward gaze.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Eyes_Look_Right_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_EYES_LOOK_RIGHT_R_FB moves the right eyelid consistent
with rightward gaze.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Eyes_Look_Up_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_EYES_LOOK_UP_L_FB moves the left eyelid consistent
with upward gaze.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Eyes_Look_Up_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_EYES_LOOK_UP_R_FB moves the right eyelid consistent
with upward gaze.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Inner_Brow_Raiser_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_INNER_BROW_RAISER_L_FB lifts the left medial brow
and forehead area.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Inner_Brow_Raiser_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_INNER_BROW_RAISER_R_FB lifts the right medial brow
and forehead area.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Jaw_Drop.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_JAW_DROP_FB moves the lower mandible downward and
toward the neck.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Jaw_Sideways_Left.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_JAW_SIDEWAYS_LEFT_FB moves the lower mandible leftward.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Jaw_Sideways_Right.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_JAW_SIDEWAYS_RIGHT_FB moves the lower mandible rightward.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Jaw_Thrust.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_JAW_THRUST_FB projects the lower mandible forward.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lid_Tightener_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LID_TIGHTENER_L_FB tightens the rings around the
left eyelid and pushes the lower eyelid skin toward the inner eye corners.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lid_Tightener_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LID_TIGHTENER_R_FB tightens the rings around the
right eyelid and pushes the lower eyelid skin toward the inner eye corners.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Corner_Depressor_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_CORNER_DEPRESSOR_L_FB draws the left lip corner downward.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Corner_Depressor_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_CORNER_DEPRESSOR_R_FB draws the right lip corner downward.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Corner_Puller_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_CORNER_PULLER_L_FB draws the left lip corners up,
back, and laterally.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Corner_Puller_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_CORNER_PULLER_R_FB draws the right lip corners up,
back, and laterally.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Funneler_LB.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_FUNNELER_LB_FB fans the left bottom lip outward
in a forward projection, often rounding the mouth and separating the lips.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Funneler_LT.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_FUNNELER_LT_FB fans the left top lip outward
in a forward projection, often rounding the mouth and separating the lips.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Funneler_RB.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_FUNNELER_RB_FB fans the right bottom lip outward
in a forward projection, often rounding the mouth and separating the lips.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Funneler_RT.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_FUNNELER_RT_FB fans the right top lip outward
in a forward projection, often rounding the mouth and separating the lips.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Pressor_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_PRESSOR_L_FB presses the left upper and
left lower lips against one another.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Pressor_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_PRESSOR_R_FB presses the right upper and
right lower lips against one another.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Pucker_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_PUCKER_L_FB draws the left lip corners medially
causing the lips protrude in the process.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Pucker_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_PUCKER_R_FB draws the right lip corners medially
causing the lips protrude in the process.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Stretcher_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_STRETCHER_L_FB draws the left lip corners
laterally, stretching the lips and widening the jawline.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Stretcher_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_STRETCHER_R_FB draws the right lip corners
laterally, stretching the lips and widening the jawline.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Suck_LB.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_SUCK_LB_FB sucks the left bottom lip toward
the inside of the mouth.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Suck_LT.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_SUCK_LT_FB sucks the left top lip toward
the inside of the mouth.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Suck_RB.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_SUCK_RB_FB sucks the right bottom lip toward
the inside of the mouth.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Suck_RT.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_SUCK_RT_FB sucks the right top lip toward
the inside of the mouth.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Tightener_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_TIGHTENER_L_FB narrows or constricts the
left lips on a horizontal plane.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lip_Tightener_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIP_TIGHTENER_R_FB narrows or constricts the
right lips on a horizontal plane.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lips_Toward.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LIPS_TOWARD_FB forces contact between top and
bottom lips to keep the mouth closed regardless of the position of the jaw.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lower_Lip_Depressor_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LOWER_LIP_DEPRESSOR_L_FB draws the left lower lip
downward and slightly laterally.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Lower_Lip_Depressor_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_LOWER_LIP_DEPRESSOR_R_FB draws the right lower lip
downward and slightly laterally.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Mouth_Left.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_MOUTH_LEFT_FB pulls the left lip corner leftward
and pushes the right side of the mouth toward the left lip corner.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Mouth_Right.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_MOUTH_RIGHT_FB pulls the right lip corner rightward
and pushes the left side of the mouth toward the right lip corner.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Nose_Wrinkler_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_NOSE_WRINKLER_L_FB lifts the left sides of
the nose, nostrils, and central upper lip area. Often pairs with brow
lowering muscles to lower the medial brow tips.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Nose_Wrinkler_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_NOSE_WRINKLER_R_FB lifts the right sides of
the nose, nostrils, and central upper lip area. Often pairs with brow
lowering muscles to lower the medial brow tips.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Outer_Brow_Raiser_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_OUTER_BROW_RAISER_L_FB lifts the lateral left brow
and forehead areas.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Outer_Brow_Raiser_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_OUTER_BROW_RAISER_R_FB lifts the lateral right brow
and forehead areas.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Upper_Lid_Raiser_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_UPPER_LID_RAISER_L_FB pulls the top left eyelid up
and back to widen eyes.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Upper_Lid_Raiser_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_UPPER_LID_RAISER_R_FB pulls the top right eyelid up
and back to widen eyes.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Upper_Lip_Raiser_L.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_UPPER_LIP_RAISER_L_FB lifts the top left lip
(in a more lateral manner than nose wrinkler).
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Upper_Lip_Raiser_R.png[Static,width=460]|
ename:XR_FACE_EXPRESSION2_UPPER_LIP_RAISER_R_FB lifts the top right lip
(in a more lateral manner than nose wrinkler).
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Tongue_Tip_Interdental.png[Static,width=240]|
ename:XR_FACE_EXPRESSION2_TONGUE_TIP_INTERDENTAL_FB raises the tip of
the tongue to touch the top teeth like with the viseme "TH".
The tongue is visible and slightly sticks out past the teeth line.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Tongue_Tip_Alveolar.png[Static,width=240]|
ename:XR_FACE_EXPRESSION2_TONGUE_TIP_ALVEOLAR_FB raises the tip of tongue
to touch the back of the top teeth like in the viseme "NN".
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Tongue_Front_Dorsal_Palate.png[Static,width=240]|
ename:XR_FACE_EXPRESSION2_TONGUE_FRONT_DORSAL_PALATE_FB makes the front
part of the tongue to press against the palate like in the viseme "CH".
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Tongue_Mid_Dorsal_Palate.png[Static,width=240]|
ename:XR_FACE_EXPRESSION2_TONGUE_MID_DORSAL_PALATE_FB presses the middle
of the tongue against the palate like in the viseme "DD".
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Tongue_Back_Dorsal_Velar.png[Static,width=240]|
ename:XR_FACE_EXPRESSION2_TONGUE_BACK_DORSAL_VELAR_FB presses the back of
the tongue against the palate like in the viseme "KK".
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Tongue_Out.png[Static,width=240]|
ename:XR_FACE_EXPRESSION2_TONGUE_OUT_FB sticks the tongue out.
|====
[width="100%"]
|====
^.^a|image::images/XR_FB_Face_Expression_2_Tongue_Retreat.png[Static,width=240]|
ename:XR_FACE_EXPRESSION2_TONGUE_RETREAT_FB pulls the tongue back
in the throat and makes the tongue to stay down like in the viseme "AA".
|====

==== Conventions of confidence areas

This extension defines two separate areas of confidence.

[open,refpage='XrFaceConfidence2FB',type='enums',desc='The name of face areas where confidence is known',xrefs='']
--
include::{generated}/api/enums/XrFaceConfidence2FB.adoc[]

The "upper face" area represents everything above the upper lip, including
the eyes and eyebrows.
The "lower face" area represents everything under the eyes, including the
mouth and chin.
Cheek and nose areas contribute to both "upper face" and "lower face" areas.

--

*New Object Types*

* slink:XrFaceTracker2FB

*New Flag Types*

*New Enum Constants*

elink:XrObjectType enumeration is extended with:

* ename:XR_OBJECT_TYPE_FACE_TRACKER2_FB

elink:XrStructureType enumeration is extended with:

* ename:XR_TYPE_SYSTEM_FACE_TRACKING_PROPERTIES2_FB
* ename:XR_TYPE_FACE_TRACKER_CREATE_INFO2_FB
* ename:XR_TYPE_FACE_EXPRESSION_INFO2_FB
* ename:XR_TYPE_FACE_EXPRESSION_WEIGHTS2_FB

*New Enums*

* elink:XrFaceExpression2FB
* elink:XrFaceExpressionSet2FB
* elink:XrFaceConfidence2FB
* elink:XrFaceTrackingDataSource2FB

*New Structures*

* slink:XrSystemFaceTrackingProperties2FB
* slink:XrFaceTrackerCreateInfo2FB
* slink:XrFaceExpressionInfo2FB
* slink:XrFaceExpressionWeights2FB

*New Functions*

* flink:xrCreateFaceTracker2FB
* flink:xrDestroyFaceTracker2FB
* flink:xrGetFaceExpressionWeights2FB

*Issues*

* Should we add the tongue shapes to apiext:XR_FB_face_tracking as a new
  enum value in elink:XrFaceExpressionSetFB?
** **Resolved.** We expect that all applications should use
   apiext:XR_FB_face_tracking2 in the future and that
   apiext:XR_FB_face_tracking will ultimately be replaced by this extension.

*Version History*

* Revision 1, 2023-10-06 (Jaebong Lee)
** Initial extension description
