Haptic PIVOT: On-Demand Handhelds in VR

Robert Kovacs, Eyal Ofek, Mar Gonzalez Franco, Alexa Fay Siu, Sebastian Marwecki, Christian Holz, and Mike Sinclair

In Proceedings of the 33th Annual ACM Symposium on User Interface Software and Technology (UIST ’20)

paper: PDF
ACM Digital Library


ABSTRACT
Hapric PIVOT is a wrist-worn haptic device that renders virtual objects into the user’s hand on demand. Its simple design comprises a single actuated joint that pivots a haptic handle into and out of the user’s hand, rendering the haptic sensations of grasping, catching, or throwing an object – anywhere in space.



Unlike existing hand-held haptic devices and haptic gloves, PIVOT leaves the user’s palm free when not in use, allowing users to make unencumbered use of their hand. PIVOT also enables rendering forces acting on the held virtual objects, such as gravity, inertia, or air-drag, by actively driving its motor while the user is firmly holding the handle. When wearing a PIVOT device on both hands, they can add haptic feedback to bimanual interaction, such as lifting larger objects. In our user study, participants (n=12) evaluated the realism of grabbing and releasing objects of different shape and size with mean score 5.19 on a scale from 1 to 7, rated the ability to catch and throw balls in different directions with different velocities (mean=5.5), and verified the ability to render the comparative weight of held objects with 87% accuracy for ~100g increments.

Presentation at UIST’20

PIVOT’s handle acts as a generic proxy while additionally offering similar ergonomics and functionality as conventional VR controllers, including touch sensing, a trigger button, and vibrotactile feedback. The device design includes a back-drivable motorized hinge for the flexing wrist movement and an additional passive radioulnar hinge that enables the hand to move sideways (here: up-down) while holding the handle.

Touching and grasping virtual objects

PIVOT’s main capability is rendering haptic sensations for acquiring, grasping and releasing virtual objects. When the user reaches for an object, PIVOT moves its handle towards the user’s hand proportionally with the distance to the virtual object. As a result, PIVOT’s handle touches the user’s hand in synchrony with the virtual object.

Catching and throwing

Using its quick actuation mechanisms, PIVOT can naturally render haptic feedback in response to throwing and catching virtual objects. The main difference to grasping stationary objects is in PIVOT’s process loop that predicts the contact with the flying object. When a potential object is flying towards the hand, PIVOT starts moving the handle in advance, accounting for the latency of the system and placing it in the user’s hand at the time the user expects the object to make contact.

Force rendering

In addition to touch feedback, PIVOT can produce a sensation of dynamic forces acting on the handheld virtual object, like gravity, springiness, inertia or drag. It does this by continuously actuating its handle motor, while it is firmly grasped by the user.


An example for rendering force feedback for dynamic animated objects, such as a wiggling bunny. User perceives the inertia of the wiggling bunny through quickly actuating PIVOT’s handle back and forth.


PIVOT also supports two-handed operation to render haptic feedback in response to larger and heavier virtual objects. Because the device cannot physically constrain the two hands to each other, it renders large objects as compliant and applies coupled force feedback on both hands.

Free-hand use and summoning on-demand

One of PIVOT’s key benefits is that it lets users use their hands freely. As shown in Figure 8a, the folded idle state affords free‑hand interaction, such as resting the hand on the table, operating tangible objects, such as a keyboard and mouse, or placing up the headphones


Hardware
The servo actively pivots the handle around the slanted vertical axis, while the handle affords passive radioulnar motion to accommodate natural hand tilting. A sliding rail mount on the hand-cuff enables adjustment to individual hand sizes.
The passive, friction based radioulnar hinge enables the hand to move when holding the handle (here: up-down).


The device is equipped with a 32 bit, 180 MHz Teensy 3.6 microcontroller that controls the servo motor, gathers the sensory information and communicates with the PC.


The inside of the handle containing the trigger, capacitive sensing, and the VCA. We use the Teensy’s built-in active loading capacitive sensing functionality to detect touch on the copper-based patches placed on the inside of the handle at four distinct locations. The handle also contains a VCA to render vibrotactile feedback as well as a trigger button as commonly found in VR controllers.



User study

We conducted a user study to evaluate PIVOT’s ability to simulate: (1) grasping sensation for a variety of objects, (2) catching & throwing of objects at different speeds and directions, and (3) perceptual illusion of weight of an object. We recruited twelve participants (ages 19-26, mean=22). Each participant performed a series of tasks inside VR using PIVOT. After each task, participants also completed an offline questionnaire.

Task 1: Grasping objects of various shape and size

In the first task, we measured PIVOT’s ability to render a grasping sensation. We asked participants to reach towards, grab, move and release virtual objects of different size (Figure 15, 1-4) and shape (5-9). Note the size range includes the diameter of the handle of the wrench and the handle of the cup, but not the full object.

Participants rated their experience in response to the question “How realistic does the grab & hold haptic experience feel?” on a 1 (not real at all) to 5 (very realistic) Likert scale directly inside the HMD.
Participants found it realistic (average score 3.9) to hold spheres of 5 cm (the most similar shape to PIVOT’s handle, o3) and within a tolerance of ±2cm in diameter (o2). Objects beyond this range (o1 and o4) were not perceived as realistic (V = 66, p < 0.01). This indicates that users accept slight discrepancies between the visual and actual size of objects.

Task 2: Catching and throwing virtual objects

In the second task, we measured PIVOT’s ability to enable catching and throwing. In addition, were aiming to identify potential ergonomic differences while catching and throwing a ball coming from different directions. Even though PIVOT moves together with the user’s hand in space, there are certain anatomic differences in catching and throwing when the palm is facing upwards, downwards, to the front or to the side. We tasked participants to catch a virtual ball thrown at them from two meters away coming from four different directions (front, side, down, up).

Catching: Participants found the haptic experience for catching realistic (Figure 17), with an average score of 3.5 ±1 (mean, sd). No significant differences were found for the direction of the ball, despite the balls coming from the front were found a bit the less realistic (avg score = 3.3), most likely because of the ergonomics of the device. Participants found catching faster balls more realistic (Pearson correlation of r=0.25, p < 0.01). We did not simulate gravity, which, with the slow balls, might have been more apparent.

Throwing: Participants found the haptic experience of throwing more realistic than that of catching (Wilcoxon signed rank paired test p < 0.001), with an average score of 4 ±1 (mean, sd). The reason for this might be the previously described importance of self-generated actions for haptic acceptance [5]. No significant differences were found in the realism between throwing to the side, up and to the front. Only for the down direction we found a significant negative correlation (r=-0.31, p < .05). Further investigation would be necessary to understand this effect. For the three other directions, the correlation of speed and realism was maintained: the fastest the throw the more realistic it seemed (Pearson correlation r=0.24, p < 0.01).