Allow WebXR pages to advertise their Focus Awareness, add blurred visibility state

[kearwood]

There are some use cases where the browser or OS may wish to display system UI on top of the WebXR rendered content without interrupting the site.

In this case, the headposes would continue to be updated with the RAF loop; however, the page would be expected to stop rendering controllers and would not be able to retrieve poses for the controllers while the page is in the "blurred" state.

This "blurred" state could be added to the XRVisibilityState enum. The page could perhaps advertise its awareness of Focus by passing a flag to the options argument of navigator.xr.requestSession.

[kearwood]

As an example for such focus awareness at a platform / OS level:

https://developer.oculus.com/documentation/native/android/mobile-overlays/

[kearwood]

Such "blurred" state may be used to display information (eg, media transport controls) that are not security sensitive. The UA will ensure that any displayed UX during such a blurred state is designed in such a way that no security sensitive information from the headpose can be inferred. (eg, a permission prompt asking for microphone access would randomize the order of the "Accept" and "Reject" buttons).

[cabanier]

Doesn't the spec already offer this with the "visibility-blurred" event?
Oculus is using this to display system dialogs with the immersive session still drawing in the background without controllers.

[kearwood]

It seems that this could be implemented as a refinement of visibility-blurred, with some additions:

• While blurred, the page is expected not to display controllers. (The spec currently only says, "Input is not processed by the XRSession.")
• XRInputSource XRPose's will not be updated while in visibility-blurred.
• WebXR sessions would have an options flag or feature to specify that they will respond to the "visibility-blurred" state. (The UA may choose a different UX if a page does not).

Alternately, we could require pages to handle visibility-blurred always and reject the XRSession.getPose promise when an XRInputSource's XRSpace is passed in.

[kearwood]

Something else to consider.. If we ask WebXR pages not to show controllers when visibility-blurred, this may need to be more clearly defined...

Some pages may not actually show a "controller" that looks like an object held in the hand. It could, for example, drive an IK rig with fingers, a hand, and elbow. In this case, it would look odd to see the avatar's arm disappear. Rather, the text should say something about tracking movement or representing a "pointer"...

[kearwood]

Perhaps if a page could advertise that it supports visibility-blurred in the session creation options, we could then activate otherwise compatibility breaking behavior, such as rejecting the XRSession.getPose promise when used to get the pose of a controller during visibility-blurred.

[cabanier]

Perhaps if a page could advertise that it supports visibility-blurred in the session creation options, we could then activate otherwise compatibility breaking behavior, such as rejecting the XRSession.getPose promise when used to get the pose of a controller during visibility-blurred.

Have you seen experiences that break? So far, I've only seen experiences that show the controllers as frozen instead of not rendering them in the visibility-blurred state .

[kearwood]

Have you seen experiences that break? So far, I've only seen experiences that show the controllers as frozen instead of not rendering them in the visibility-blurred state .

No sites that break; although, it would be perhaps a recommendation for sites to hide the controllers so it doesn't look broken or confuse users that may see two sets of controllers.

Perhaps we could resolve this by clarifying the "Input is not processed by the XRSession.". One way would be to just say that the XRPose returned by getPose for a XRInputSource would continue to return the same transform for the duration of the visibility-blurred state, but still resolve the promise successfully. (IIUC, this is the Oculus browser behavior currently?)

We could then add non-normative text informing sites that the best practice is to present the controllers in a way that indicates that they are no longer tracked. (eg, hide anything that looks like the physical controller, or switch to default animations or pose for rigged avatars...) When the visibility-blurred state ends, the site should be aware that there may be a large jump of movement for the controllers as they resume tracking (and to avoid it affecting physics, gameplay, etc...).

[kearwood]

Should we agenda this? If others agree, I would like to make a PR to just update the spec text to be more explicit on what "Input is not processed by the XRSession." does. The result would effectively be the same as what the Oculus browser does currently, with non-normative recommendations on hiding controller models that may be confused with the UA's rendered models.

[cabanier]

You don't need to put this on the agenda since I'm pretty sure that we're all in agreement.
If you can add recommendations to the spec and the explainer, I'm sure that will be enough.

Maybe we should also file bugs against three.js and other frameworks so they add listeners and behavior for these events.

[kearwood]

This sounds great to me. I'll prepare a PR on Monday with the proposed text changes.

I've talked about this internally at Mozilla and it seems that it can work for us. If we really do need pages to advertise focus awareness, that can always be done additively later on. I hope that it wouldn't be necessary; however.

[cabanier]

Perhaps we could resolve this by clarifying the "Input is not processed by the XRSession.". One way would be to just say that the XRPose returned by getPose for a XRInputSource would continue to return the same transform for the duration of the visibility-blurred state, but still resolve the promise successfully. (IIUC, this is the Oculus browser behavior currently?)

Yes, that is what the Oculus browser does. I'm pretty certain that other Chromium based implementations would do the same.

[Manishearth]

getPose() isn't a promise, but yes, in this case getPose() is supposed to return null.

[Manishearth]

• While blurred, the page is expected not to display controllers. (The spec currently only says, "Input is not processed by the XRSession.")

• XRInputSource XRPose's will not be updated while in visibility-blurred.

I believe this is the intent of the current spec text. When input is not processed; input poses are not extant.

• WebXR sessions would have an options flag or feature to specify that they will respond to the "visibility-blurred" state. (The UA may choose a different UX if a page does not).

Not sure if this is strictly necessary? If the input source pose is null, there's not much you can do other than stop rendering the controller.

You can already trigger this case by sticking your controller out of tracking range. I would definitely expect applications to handle this properly, especially as the failure mode of this is that you get a null error.

I don't think there needs to be special handling for the blurred state aside from the app taking it as a hint to pause things if necessary.

I would suggest we fix this by expanding the note about input not being processed.