AR-enhanced Widgets for Smartphone-centric Interaction

Eugénie Brasier, Emmanuel Pietriga, Caroline Appert
MobileHCI 2021 -- Supplemental Material
AR-enhanced widgets for smartphones use AR technology not only to offload widgets from the phone to the air around it, but also to give users more control on input precision.
This page provides 1) the full description of the preliminary experiments we ran during our iterative design process; and 2) the data we collected in the three main experiments that compare different AR-enhanced widget designs against their state-of-the-art touch-only counterparts.

Preliminary Experiment A: Mid-air Input Accuracy (Section 3.4)

Participants and Apparatus

Six participants from the pool described in the paper. Same apparatus as for the main experiments, described in Section 5. The system considers a hand pinch posture as released when the distance between the index finger's and the thumb's tips becomes greater than 2cm.

The measured offset Δrelease corresponds to the distance between the target position and the cursor's actual position at the time the system detects the change of hand posture.


Participants have to position a cursor indirectly controlled using hand movements in the air. They have to put it in a precise location, and then switch from a pinch to a release posture. We measure the offset between the target position and the cursor's actual position at the time the system detects this change of hand posture, as illustrated on the right. The graphical scene consists of a crosshair cursor and a black circular target 1.5mm in diameter, both displayed by the phone. Again, the input coordinates correspond to the point halfway in-between the thumb's and index finger's tips. Hand control is indirect and relative, allowing participants to clutch if needed. The Control-Display gain is constant (1:1). Participants are instructed to bring the cursor over the target. This causes the background to switch from mid-gray to yellow for 500ms. Participants then release their pinch posture, which ends the trial.

Design and Procedure

We follow a within-subject design. The only factor is the TARGET POSITION (10 positions evenly spread across the screen) which we introduce for ecological reasons. Each condition is repeated twice, leading to a total of 20 trials per participant. The presentation order of the different positions is random. Our measure is Δrelease, the offset between the cursor position and the target at the time the system detects the release of the pinch posture.


Averaging all collected values, we obtain Δrelease = 1.4 +/- 2.2 < 4mm.

Preliminary Experiment B: Effective use of the Air as a Display (Section 3.5)

Participants and Apparatus

Same six participants as in preliminary experiment A. Same apparatus.

The two conditions preferred by participants: North for buttons, and East for sliders.


Participants had to perform two types of tasks: pressing a series of Buttons and adjusting a Slider's value.

Design and Procedure

We follow a within-subject design, with two factors: WIDGET TYPE (Button, Slider), and WIDGET LOCATION (North, East, South, West). Trials are blocked by WIDGET TYPE. Each WIDGET TYPE x WIDGET LOCATION is replicated twice in series, for a total of 16 trials.

Participants make a qualitative assessment of the workload in each condition using the six NASA-TLX scales. They are also asked to rank the four sides in order of overall preference. Participants are instructed to focus on issues of comfort rather than performance.


A majority of participants (5) ranked East as their preferred location for interacting with Sliders, North coming second. One participant ranked South as the preferred location. Results are more nuanced for Buttons. Half of the participants ranked North first, while the other half preferred East. They all ranked West as the least-preferred location, reporting issues due to their non-dominant hand being in that region and interfering with the task. South was ranked either second (2) or third (4), participants reporting that they had to adopt uncomfortable head postures in this condition as the widgets were getting out of the ARHMD's field of view when holding the phone in a natural position.

Main Experiments (Section 4)

Sanitary Protocol

All experiments reported in this article were run during the summer of 2020. The particular situation due to the COVID-19 pandemic led us to adopt specific sanitary precautions. This document details our sanitary protocol.

Experiment 1: Command trigger

CSV file of collected data.

Experiment 2: Parameter value adjustment

CSV file of collected data.

Experiment 3: Precise 2D selection

CSV file of collected data.