// Copyright (c) 2017-2024, The Khronos Group Inc.
//
// SPDX-License-Identifier: CC-BY-4.0

include::{generated}/meta/XR_EXT_eye_gaze_interaction.adoc[]


*Last Modified Date*::
    2020-02-20

*IP Status*::
    No known IP claims.

*Contributors*::
    Denny Rönngren, Tobii +
    Yin Li, Microsoft +
    Alex Turner, Microsoft +
    Paul Pedriana, Oculus +
    Rémi Arnaud, Varjo +
    Blake Taylor, Magic Leap +
    Lachlan Ford, Microsoft +
    Cass Everitt, Oculus

*Overview*

This extension provides an basetype:XrPath for getting eye gaze input from
an eye tracker to enable eye gaze interactions.

The intended use for this extension is to provide:

* system properties to inform if eye gaze interaction is supported by the
  current device.

* an basetype:XrPath for real time eye tracking that exposes an accurate and
  precise eye gaze pose to be used to enable eye gaze interactions.

* a structure slink:XrEyeGazeSampleTimeEXT that allows for an application to
  retrieve more information regarding the eye tracking samples.

With these building blocks, an application can discover if the XR runtime
has access to an eye tracker, bind the eye gaze pose to the action system,
determine if the eye tracker is actively tracking the users eye gaze, and
use the eye gaze pose as an input signal to build eye gaze interactions.

==== Eye tracker

An eye tracker is a sensory device that tracks eyes and accurately maps what
the user is looking at.
The main purpose of this extension is to provide accurate and precise eye
gaze for the application.

Eye tracking data can be sensitive personal information and is closely
linked to personal privacy and integrity.
It is strongly recommended that applications that store or transfer eye
tracking data always ask the user for active and specific acceptance to do
so.

If a runtime supports a permission system to control application access to
the eye tracker, then the runtime must: set the pname:isActive field to
ename:XR_FALSE on the supplied slink:XrActionStatePose structure, and must:
clear ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT,
ename:XR_SPACE_LOCATION_POSITION_VALID_BIT,
ename:XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT and
ename:XR_SPACE_LOCATION_ORIENTATION_VALID_BIT when locating using the
tracked space until the application has been allowed access to the eye
tracker.
When the application access has been allowed, the runtime may: set
pname:isActive on the supplied slink:XrActionStatePose structure to
ename:XR_TRUE and may: set ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT,
ename:XR_SPACE_LOCATION_POSITION_VALID_BIT
ename:XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT and
ename:XR_SPACE_LOCATION_ORIENTATION_VALID_BIT when locating using the
tracked space.

==== Device enumeration

When the eye gaze input extension is enabled an application may: pass in a
slink:XrSystemEyeGazeInteractionPropertiesEXT structure in next chain
structure when calling flink:xrGetSystemProperties to acquire information
about the connected eye tracker.

The runtime must: populate the slink:XrSystemEyeGazeInteractionPropertiesEXT
structure with the relevant information to the slink:XrSystemProperties
returned by the flink:xrGetSystemProperties call.

[open,refpage='XrSystemEyeGazeInteractionPropertiesEXT',type='structs',desc='Eye gaze interaction system properties']
--
include::{generated}/api/structs/XrSystemEyeGazeInteractionPropertiesEXT.adoc[]

.Member Descriptions
****
* pname:type is the elink:XrStructureType of this structure.
* pname:next is code:NULL or a pointer to the next structure in a structure
  chain.
  No such structures are defined in core OpenXR or this extension.
* pname:supportsEyeGazeInteraction the runtime must: set this value to
  ename:XR_TRUE when eye gaze sufficient for use cases such as aiming or
  targeting is supported by the current device, otherwise the runtime must:
  set this to ename:XR_FALSE.
****

include::{generated}/validity/structs/XrSystemEyeGazeInteractionPropertiesEXT.adoc[]
--

==== Eye gaze input

This extension exposes a new interaction profile path
pathname:/interaction_profiles/ext/eye_gaze_interaction that is valid for
the user path

* pathname:/user/eyes_ext

for supported input source

* subpathname:/input/gaze_ext/pose

// This extension does not meet the current style guide requirements:
// the path name must be vendor-suffixed in new extensions.
:legacy-path: /interaction_profiles/ext/eye_gaze_interaction
:properly-suffixed-path: /interaction_profiles/ext/eye_gaze_interaction_ext
include::{config}/interaction_profile_missing_suffix_warning.adoc[]

The eye gaze pose is natively oriented with +Y up, +X to the right, and -Z
forward and not gravity-aligned, similar to the
ename:XR_REFERENCE_SPACE_TYPE_VIEW.
The eye gaze pose may originate from a point positioned between the user's
eyes.
At any point of time both the position and direction of the eye pose is
tracked or untracked.
This means that the runtime must: set both
ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT and
ename:XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT or clear both
ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT and
ename:XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT.

One particularity for eye trackers compared to most other spatial input is
that the runtime may not have the capability to predict or interpolate eye
gaze poses.
Runtimes that cannot predict or interpolate eye gaze poses must: clamp the
gaze pose requested in the flink:xrLocateSpace call to the value nearest to
pname:time requested in the call.
To allow for an application to reason about high accuracy eye tracking, the
application can: chain in an slink:XrEyeGazeSampleTimeEXT to the next
pointer of the slink:XrSpaceLocation structure passed into the
flink:xrLocateSpace call.
The runtime must: set pname:time in the slink:XrEyeGazeSampleTimeEXT
structure to the clamped, predicted or interpolated time.
The application should: inspect the pname:time field to understand when in
time the pose is expressed.
The pname:time field may: be in the future if a runtime can predict gaze
poses.
The runtime must: set the pname:time field to 0 if the sample time is not
available.

When the runtime provides a nominal eye gaze pose, the
ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT must: be set if the eye
otherwise has a fully-tracked pose relative to the other space.
A runtime can: provide a sub-nominal eye-gaze pose but must: then clear the
ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT.
An application can expect that a nominal eye gaze pose can be used for use
cases such as aiming or targeting, while a sub-nominal eye gaze pose has
degraded performance and should not be relied on for all input scenarios.
Applications should be very careful when using sub-nominal eye gaze pose,
since the behavior can vary considerably for different users and
manufacturers, and some manufacturers may: not provide sub-nominal eye gaze
pose at all.

With current technology, some eye trackers may: need to undergo an explicit
calibration routine to provide a nominal accurate and precise eye gaze pose.
If the eye tracker is in an uncalibrated state when the first call to
flink:xrSyncActions is made with an eye gaze action enabled, then the
runtime should: request eye tracker calibration from the user if it has not
yet been requested.

[open,refpage='XrEyeGazeSampleTimeEXT',type='structs',desc='Eye gaze sample time structure']
--
include::{generated}/api/structs/XrEyeGazeSampleTimeEXT.adoc[]

.Member Descriptions
****
* pname:type is the elink:XrStructureType of this structure.
* pname:next is code:NULL or a pointer to the next structure in a structure
  chain.
  No such structures are defined in core OpenXR or this extension.
* pname:time is when in time the eye gaze pose is expressed.
****

include::{generated}/validity/structs/XrEyeGazeSampleTimeEXT.adoc[]
--

==== Sample code

The following example code shows how to bind the eye pose to the action
system.

[source,c++]
----
extern XrInstance instance;
extern XrSession session;
extern XrPosef pose_identity;

// Create action set
XrActionSetCreateInfo actionSetInfo{XR_TYPE_ACTION_SET_CREATE_INFO};
strcpy(actionSetInfo.actionSetName, "gameplay");
strcpy(actionSetInfo.localizedActionSetName, "Gameplay");
actionSetInfo.priority = 0;
XrActionSet gameplayActionSet;
CHK_XR(xrCreateActionSet(instance, &actionSetInfo, &gameplayActionSet));

// Create user intent action
XrActionCreateInfo actionInfo{XR_TYPE_ACTION_CREATE_INFO};
strcpy(actionInfo.actionName, "user_intent");
actionInfo.actionType = XR_ACTION_TYPE_POSE_INPUT;
strcpy(actionInfo.localizedActionName, "User Intent");
XrAction userIntentAction;
CHK_XR(xrCreateAction(gameplayActionSet, &actionInfo, &userIntentAction));

// Create suggested bindings
XrPath eyeGazeInteractionProfilePath;
CHK_XR(xrStringToPath(instance, "/interaction_profiles/ext/eye_gaze_interaction", &eyeGazeInteractionProfilePath));

XrPath gazePosePath;
CHK_XR(xrStringToPath(instance, "/user/eyes_ext/input/gaze_ext/pose", &gazePosePath));

XrActionSuggestedBinding bindings;
bindings.action = userIntentAction;
bindings.binding = gazePosePath;

XrInteractionProfileSuggestedBinding suggestedBindings{XR_TYPE_INTERACTION_PROFILE_SUGGESTED_BINDING};
suggestedBindings.interactionProfile = eyeGazeInteractionProfilePath;
suggestedBindings.suggestedBindings = &bindings;
suggestedBindings.countSuggestedBindings = 1;
CHK_XR(xrSuggestInteractionProfileBindings(instance, &suggestedBindings));

XrSessionActionSetsAttachInfo attachInfo{XR_TYPE_SESSION_ACTION_SETS_ATTACH_INFO};
attachInfo.countActionSets = 1;
attachInfo.actionSets = &gameplayActionSet;
CHK_XR(xrAttachSessionActionSets(session, &attachInfo));

XrActionSpaceCreateInfo createActionSpaceInfo{XR_TYPE_ACTION_SPACE_CREATE_INFO};
createActionSpaceInfo.action = userIntentAction;
createActionSpaceInfo.poseInActionSpace = pose_identity;
XrSpace gazeActionSpace;
CHK_XR(xrCreateActionSpace(session, &createActionSpaceInfo, &gazeActionSpace));

XrReferenceSpaceCreateInfo createReferenceSpaceInfo{XR_TYPE_REFERENCE_SPACE_CREATE_INFO};
createReferenceSpaceInfo.referenceSpaceType = XR_REFERENCE_SPACE_TYPE_LOCAL;
createReferenceSpaceInfo.poseInReferenceSpace = pose_identity;
XrSpace localReferenceSpace;
CHK_XR(xrCreateReferenceSpace(session, &createReferenceSpaceInfo, &localReferenceSpace));

while(true)
{
  XrActiveActionSet activeActionSet{gameplayActionSet, XR_NULL_PATH};
  XrTime time;

  XrActionsSyncInfo syncInfo{XR_TYPE_ACTIONS_SYNC_INFO};
  syncInfo.countActiveActionSets = 1;
  syncInfo.activeActionSets = &activeActionSet;
  CHK_XR(xrSyncActions(session, &syncInfo));

  XrActionStatePose actionStatePose{XR_TYPE_ACTION_STATE_POSE};
  XrActionStateGetInfo getActionStateInfo{XR_TYPE_ACTION_STATE_GET_INFO};
  getActionStateInfo.action = userIntentAction;
  CHK_XR(xrGetActionStatePose(session, &getActionStateInfo, &actionStatePose));

  if(actionStatePose.isActive){
    XrEyeGazeSampleTimeEXT eyeGazeSampleTime{XR_TYPE_EYE_GAZE_SAMPLE_TIME_EXT};
    XrSpaceLocation gazeLocation{XR_TYPE_SPACE_LOCATION, &eyeGazeSampleTime};
    CHK_XR(xrLocateSpace(gazeActionSpace, localReferenceSpace, time, &gazeLocation));

    // Do things
  }
}

----

*Version History*

* Revision 1, 2020-02-20 (Denny Rönngren)
** Initial version
* Revision 2, 2022-05-27 (Bryce Hutchings)
** Remove error-prone `XrEyeGazeSampleTimeEXT` validation requirement
