To create a truly accessible and inclusive society, we need to take the more than 2.2 billion people with vision impairments worldwide into account when we design our cities, buildings, and everyday objects. This requires sympathy and empathy, as well as a certain level of understanding of the impact of vision impairments on perception. In this study, we explore the potential of an extended version of our vision-impairment simulation system XREye to increase sympathy and empathy and evaluate its educational value in an expert study with 56 educators and education students. We include data from a previous study in related work on sympathy and empathy as a baseline for comparison with our data. Our results show increased sympathy and empathy after experiencing XREye and positive feedback regarding its educational value. Hence, we believe that vision-impairment simulations, such as XREye, have merit to be used for educational purposes in order to increase awareness for the challenges people with vision impairments face in their everyday lives.

Three-dimensional geospatial thinking is an important skillset used by earth scientists and students to analyze and interpret data. This method of inquiry is useful in glaciology, where traditional geophysical survey techniques have been adapted to map three-dimensional (3D) ice sheet structures and inform studies of ice flow, mass change, and history in both Greenland and Antarctica. Ice-penetrating radar images the ice in two-dimensional (2D) cross-sections from the surface to the base. Analysis of this data often requires visual inspection and 3D interpretation, but is hindered by data visualization tools and techniques that rarely transcend the two-dimensionality of the computer screen. Recent advances in Augmented Reality (AR) and Virtual Reality (VR), together referred to as Extended Reality (XR), offer a glimpse into the future of 3D ice-sheet data analysis. These technologies offer users an immersive experience where 3D geospatial datasets can be understood more immediately than with 2D maps, and gestural user interfaces can enhance understanding. Here we present Pol-XR, an XR application that supports both visualization and interpretation of ice-penetrating radar in Antarctica and Greenland.

Measurements of the interior structure of ice shelves give us valuable clues to understand their history, as distinct ice bodies may be traced from the grounding line toward the calving front. On the Ross Ice Shelf, this trajectory from the grounding line to the calving front takes ~1000 years, meaning that the ice shelf holds a 1000-year-long record of processes including snow accumulation, crevassing, basal melting, stretching, and basal freeze-on. Recently acquired radar data from the ROSETTA-Ice project show the internal structure of the ice in unprecedented detail. Two types of radar data, a shallow-ice radar (SIR) and a deep-ice radar (DICE), were acquired through aerogeophysical surveys flown in a grid at 10-km line spacing from 20152017. Radar data analysis entails careful tracking/picking of visible layers within the images. Images are displayed on a 2D screen and manually picked, using in-house MATLAB software. The high-resolution, 10-km line spacing of the Rosetta-Ice survey lends itself to a new method for data visualization, interpretation and analysis using the Microsoft HoloLens, an augmented reality headset that allows users to interact with 3D virtual objects using gestures. We build on previous work using the HoloLens to display ROSETTA-Ice data by creating a new user interface for measuring and comparing ROSETTA-Ice radar data. In this environment, radar images and identified horizons can be dragged in space to facilitate comparison between lines, as well as transformed (e.g., squashed, stretched, and tilted). Through this interface, we can more easily interact with data and determine how the ice changes as it flows downstream. Here we present our analysis of an ice body that appears predominantly in the SIR, and occasionally in the DICE radar data, located between the flow streaks that emanate from the Beardmore glacier. The feature is defined by the basal and near-basal reflections in the image, and can be traced in at least 26 ROSETTA-Ice lines oriented across the feature. This ice feature has a unique arcing shape and is intermittently visible for ~547 km. By comparing radar profiles of this feature, we track its volume and morphological change along flow.

Augmented Reality (AR) and Virtual Reality (VR) provide powerful, natural, and robust ways to interact with digital content, across a number of different domains. AR and VR, collectively known as Extended Reality (XR), can facilitate the execution of surgical procedures, aid in maintenance and repair of mechanical equipment, provide novel visualization paradigms for data analysis, and even empower new ways to experience video games. These experiences are built on rich, complex real-time interactive systems (RISs) that require the integration of numerous components supporting everything from rendering of virtual content to tracking of objects and people in the real world. There are decades of research on the development of robust RISs, utilizing different software engineering modalities, which facilitate the creation of these systems. While in the past, developers would frequently write all of the components and the “logical glue” themselves (often built with graphics suites such as OpenGL and DirectX), with the the rise of popular 3D game creation engines, such as Unity and Unreal, new development modalities have begun to emerge. While the underlying game engines provide a significantly easier pipeline to integrate different subsystems of AR/VR applications, there are a number of development questions that arise when considering how interaction, visualization, rendering, and application logic should interact, as developers are often left to create the “logical glue” on their own, leading to software components with low reusability. As the needs of users of these systems increase and become more complex, and as the software and hardware technology improves and becomes more sophisticated, the underlying subsystems must also evolve to help meet these needs. In this work, I present a new software design pattern, the Relay & Responder (R&R) pattern, that attempts to address the concerns found with many traditional object-oriented approaches in XR systems. The R&R pattern simplifies the design of these systems by separating logical components from the communication infrastructure that connects them, while minimizing coupling and facilitating the creation of logical hierarchies that can improve XR application design and module reuse. Additionally, I explore how this pattern can, across a number of different research development efforts, aid in the creation of powerful and rich XR RISs. I first present related work in XR system design and introduce the R&R pattern. Then I discuss how XR development can be eased by utilizing modular building blocks and present the Mercury Messaging framework, which implements the R&R pattern. Next I delve into three new XR systems that explore complex XR RIS designs (including user study management modules) using the pattern and framework. I then address the creation of multi-user, networked XR RISs using R&R and Mercury. Finally I end with a discussion on additional considerations, advantages, and limitations of the pattern and framework, in addition to prospective future work that will help improve both..

Work on VR and AR task interaction and visualization paradigms has typically focused on providing information about the current step (a cue) immediately before or during its performance. Some research has also shown benefits to simultaneously providing information about the next step (a precue). We explore whether it would be possible to improve efficiency by precueing information about multiple upcoming steps before completing the current step. To accomplish this, we developed a remote VR user study comparing task completion time and subjective metrics for different levels and styles of precueing in a path-following task. Our visualizations vary the precueing level (number of steps precued in advance) and style (whether the \textit{path} to a target is communicated through a line to the target, and whether the \textit{place} of a target is communicated through graphics at the target). Participants in our study performed best when given two to three precues for visualizations using lines to show the path to targets. However, performance degraded when four precues were used. On the other hand, participants performed best with only one precue for visualizations without lines, showing only the places of targets, and performance degraded when a second precue was given. In addition, participants performed better using visualizations with lines than ones without lines.

Augmented Reality (AR) provides opportunities to create exciting new kinds of digital entertainment, such as watching movies on a large virtual screen or playing games that interact with a real physical room. While a number of AR games have been built, many do not build on the control innovations found in modern console, PC, and mobile gaming. To explore the space of immersive multiplayer experiences with support for control innovations found in common non-immersive video games, we present HoloFight, a multiplayer fighting game using two or more Microsoft HoloLens 2s, two or more Xbox controllers, and the various natural user interfaces supported by the Microsoft HoloLens 2.

This position paper describes CognitiveAR, a system that seamlessly interfaces AR devices with smart city environments. Edge computing nodes distributed throughout the city enable multi-user cognitive assistance applications that require (1) real-time sensor data from the environment, such as approaching cars, and (2) computing resources for low-latency video processing. We discuss three such applications to elicit requirements for a platform to support them, and present our system design.

For our society to be more inclusive and accessible, the more than 2.2 billion people worldwide with limited vision should be considered more frequently in design decisions, such as architectural planning. To help architects in evaluating their designs and give medical personnel some insight on how patients experience cataracts, we worked with ophthalmologists to develop the first medically-informed, pilot-studied simulation of cataracts in eye-tracked augmented reality (AR). To test our methodology and simulation, we conducted a pilot study with cataract patients between surgeries of their two cataract-affected eyes. Participants compared the vision of their corrected eye, viewing through simulated cataracts, to that of their still affected eye, viewing an unmodified AR view. In addition, we conducted remote experiments via video call, live adjusting our simulation and comparing it to related work, with participants who had cataract surgery a few months before. We present our findings and insights from these experiments and outline avenues for future work.

Musicians face a number of issues when performing live, including organizing and annotating sheet music. This can be an unwieldy process, as musicians need to simultaneously read and manipulate sheet music and interact with the conductor and other musicians. Augmented Reality can provide a way to ease some of the more cumbersome aspects of live performance and practice. We present MiXR, a novel interactive system that combines an AR headset, a smartphone, and a tablet to allow performers to intuitively and efficiently manage and annotate virtual sheet music in their physical environment. We discuss our underlying motivation, the interaction techniques supported, and the system architecture.

Certain embodiments include a method for assisting a clinician in performing a medical procedure on a patient using augmented reality guidance. The method can include obtaining a three-dimensional model of an anatomic part of the patient. The method can also include aligning the three-dimensional model with data to form augmented reality guidance for the medical procedure. In addition, the method can include presenting the augmented reality guidance to the clinician during the medical procedure using an augmented reality three-dimensional display.

Many people suffer from visual impairments, which can be difficult for patients to describe and others to visualize. To aid in understanding what people with visual impairments experience, we demonstrate a set of medically informed simulations in eye-tracked XR of several common conditions that affect visual perception: refractive errors (myopia, hyperopia, and presbyopia), cornea disease, and age-related macular degeneration (wet and dry).

Many AR and VR task domains involve manipulating virtual objects; for example, to perform 3D geometric transformations. These operations are typically accomplished with tracked hands or hand-held controllers. However, there are some activities in which the user's hands are already busy with another task, requiring the user to temporarily stop what they are doing to perform the second task, while also taking time to disengage and reengage with the original task (e.g., putting down and picking up tools). To avoid the need to overload the user's hands this way in an AR system for guiding a physician performing a surgical procedure, we developed a hands-free approach to performing 3D transformations on patient-specific virtual organ models. Our approach uses small head motions to accomplish first-order and zero-order control, in conjunction with voice commands to establish the type of transformation. To show the effectiveness of this approach for translating, scaling, and rotating 3D virtual models, we conducted a within-subject study comparing the hands-free approach with one based on conventional manual techniques, both running on a Microsoft HoloLens and using the same voice commands to specify transformation type. Independent of any additional time to transition between tasks, users were significantly faster overall using the hands-free approach, significantly faster for hands-free translation and scaling, and faster (although not significantly) for hands-free rotation.

Vision impairments, such as cataracts, affect the way many people interact with their environment, yet are rarely considered by architects and lighting designers because of a lack of design tools. To address this, we present a method to simulate vision impairments, in particular cataracts, graphically in virtual reality (VR), using eye tracking for gaze-dependent effects. We also conduct a VR user study to investigate the effects of lighting on visual perception for users with cataracts. In contrast to existing approaches, which mostly provide only simplified simulations and are primarily targeted at educational or demonstrative purposes, we account for the user's vision and the hardware constraints of the VR headset. This makes it possible to calibrate our cataract simulation to the same level of degraded vision for all participants. Our study results show that we are able to calibrate the vision of all our participants to a similar level of impairment, that maximum recognition distances for escape route signs with simulated cataracts are significantly smaller than without, and that luminaires visible in the field of view are perceived as especially disturbing due to the glare effects they create. In addition, the results show that our realistic simulation increases the understanding of how people with cataracts see and could therefore also be informative for health care personnel or relatives of cataract patients.

In the real world, we are surrounded by potentially important data. For example, military personnel and first responders may need to understand the layout of an environment, including the locations of designated assets, specified in latitude and longitude. However, many augmented reality (AR) systems cannot associate absolute geographic coordinates with the coordinate system in which they track. We describe a simple approach for developing a wide-area outdoor wearable AR system that uses RTK GNSS position tracking to align together and georegister multiple smaller maps from an existing SLAM tracking system.

Augmented reality (AR) can improve transcatheter interventions by providing 3D visualization of patient anatomy, with potential to reduce contrast and procedure time. We present a novel AR guidance system that displays virtual, patient-specific 3D anatomic models and assess intraprocedural impact during TAVR with CEP in six patients.

Astronauts currently require extensive, near-instantaneous guidance and instruction by ground-based crew to efficiently and successfully conduct flight operations. As missions take astronauts farther away from earth and real-time communication between spacecraft and earthbound crew becomes impossible, astronauts will need technology that can help them execute flight operations with limited support. While research has shown that Augmented Reality (AR) can feasibly perform as an aid for completing certain flight tasks, there is little evidence that AR can assist in completing entire flight operations or improve flight performance metrics such as completion time. This work addresses stowage operations to investigate how AR can impact flight performance. During stowage operations, flight crew members transfer cargo items to and from different spacecraft modules. A recent stowage operation aboard the International Space Station (ISS) took 60 hours to complete with real-time ground crew support. The prolonged duration of stowage operations and the necessity for crewmembers to travel significant distances make it an appropriate domain for this investigation. StowageApp is a prototype AR application deployed on Microsoft HoloLens, and developed to assist astronauts in completing stowage operations. This paper describes the design of StowageApp and present the results of a user study comparing its performance to that of the current method of delivering stowage instructions on a handheld tablet device. This within-subject user study was performed in the ISS Node 2 Harmony, Japanese Experiment Module "Kibo," MultiPurpose Logistics Module "Leonardo," and Columbus mockups at National Aeronautics and Space Administration (NASA) Johnson Space Center in Houston, TX, USA. Each participant completed as many of a set of predetermined stowage tasks as they could in two hours in their assigned condition. Task completion time was measured, along with the number of errors attributed to the participant. Participants also completed an unweighted NASA TLX survey and provide their opinions in a free-form exit interview. Results did not reveal significant differences in task completion time, errors committed, or TLX responses between cargo message content conveyed via StowageApp and via electronic document on a tablet handheld device. However, user interviews showed that all but one participant would prefer to use StowageApp over the handheld device.

We present hybrid user interfaces that facilitate interaction with music content in 3D, using a combination of 2D and 3D input and display devices. Participants will explore an online music library, some wearing AR or VR head-worn displays used alone or in conjunction with touch screens, and others using only touch screens. They will select genres, artists, albums and songs, interacting through a combination of 3D hand-tracking and 2D multi-touch technologies.

In collaborative virtual environments, users must often perform tasks requiring coordinated action between multiple parties. Some cases are symmetric, in which users work together on equal footing, while others are asymmetric, in which one user may have more experience or capabilities than another (e.g., one may guide another in completing a task). We present a multi-user virtual reality system that supports interactions of both these types. Two collaborating users, whether co-located or remote, simultaneously manipulate the same virtual objects in a physics simulation, in tasks that require low latency networking to perform successfully. We are currently applying this approach to motor rehabilitation, in which a therapist and patient work together.

During a vascular intervention (a type of minimally invasive surgical procedure), physicians maneuver catheters and wires through a patient's blood vessels to reach a desired location in the body. Since the relevant anatomy is typically not directly visible in these procedures, virtual reality and augmented reality systems have been developed to assist in 3D navigation. Because both of a physician's hands may already be occupied, we developed an augmented reality system supporting hands-free interaction techniques that use voice and head tracking to enable the physician to interact with 3D virtual content on a head-worn display while leaving both hands available intraoperatively. We demonstrate how a virtual 3D anatomical model can be rotated and scaled using small head rotations through first-order (rate) control, and can be rigidly coupled to the head for combined translation and rotation through zero-order control. This enables easy manipulation of a model while it stays close to the center of the physician's field of view.

Built at NASA Johnson Space Center (JSC) and Columbia University and tested in JSC's full-scale mockup of the International Space Station (ISS), StowageApp is a prototype for the future of conducting cargo operations in space. StowageApp dynamically guides astronauts as they complete stowage tasks, packing and unpacking cargo.

From emergency planning to real estate, many domains can benefit from collaborative exploration of urban environments in VR and AR. We have created an interactive experience that allows multiple users to explore live datasets in context of an immersive scale model of the urban environment with which they are related.

In recent years, the entity--component--system pattern has become a fundamental feature of the software architectures of game-development environments such as Unity and Unreal, which are used extensively in developing 3D user interfaces. In these systems, UI components typically respond to events, requiring programmers to write application-specific callback functions. In some cases, components are organized in a hierarchy that is used to propagate events among vertically connected components. When components need to communicate horizontally, programmers must connect those components manually and register/unregister events as needed. Moreover, events and callback signatures may be incompatible, making modular UIs cumbersome to build and share within or across applications. To address these problems, we introduce a messaging framework, Mercury, to facilitate communication among components. We provide an overview of Mercury, outline its underlying protocol and how it propagates messages to responders using relay nodes, describe a reference implementation in Unity, and present example systems built using Mercury to explain its advantages.

Augmented reality (AR) holds great potential for IR byintegrating virtual 3D anatomic models into the real world.1In thispilot study, we developed an AR guidance system for cerebralangiography, evaluated its impact on radiation, contrast, and fluo-roscopy time, and assessed physician response.

Vascular interventions are minimally invasive surgical procedures in which a physician navigates a catheter through a patient's vasculature to a desired destination in the patient's body. Since perception of relevant patient anatomy is limited in procedures of this sort, virtual reality and augmented reality systems have been developed to assist in 3D navigation. These systems often require user interaction, yet both of the physician's hands may already be busy performing the procedure. To address this need, we demonstrate hands-free interaction techniques that use voice and head tracking to allow the physician to interact with 3D virtual content on a head-worn display while making both hands available intraoperatively. Our approach supports rotation and scaling of 3D anatomical models that appear to reside in the surrounding environment through small head rotations using first-order control, and rigid body transformation of those models using zero-order control. This allows the physician to easily manipulate a model while it stays close to the center of their field of view.

In many complex tasks, a remote subject-matter expert may need to assist a local user, to guide their actions on objects in the local user's environment. However, effective spatial referencing and action demonstration in a remote physical environment can be challenging. We demonstrate an approach that uses Virtual Reality (VR) or Augmented Reality (AR) for the remote expert, and AR for the local user, each wearing a stereo head-worn display (HWD). Our approach allows the remote expert to create and manipulate virtual replicas of physical objects in the local environment to refer to parts of those physical objects and to indicate actions on them. This can be especially useful for parts that are occluded or difficult to access. The remote expert can demonstrate actions in 3D by manipulating virtual replicas, supported by constraints and annotations, and point in 3D to portions of virtual replicas to annotate them.

We demonstrate an interaction technique that allows a user to point at a world-in-miniature representation of a city-scale virtual environment and perform efficient and precise teleportation by pre-orienting an avatar. A preview of the post-teleport view of the full-scale virtual environment updates interactively as the user adjusts the position, yaw, and pitch of the avatar's head with a pair of 6DoF-tracked controllers. We describe design decisions and contrast with alternative approaches to virtual travel.

Systems and methods for providing assistance for manipulating objects using virtual proxies and virtual replicas are provided. Disclosed systems enable a remote or co-located subject matter expert (SME) to provide guidance to a local user on how to manipulate physical or virtual objects in the local user's environment. The local user views an augmented reality (AR) display of the local environment. The SME views a virtual reality (VR) or AR display of the local environment. The SME manipulates virtual proxies and virtual replicas to demonstrate to the local user how physical or virtual objects in the local environment should be manipulated. In other scenarios, a local user is provided instruction by using a display from which the local user can view the local environment. Manipulations of objects are tracked and the system provides the user feedback on whether the objects are properly oriented.

Many tasks require that a user rotate an object to match a specific orientation in an external coordinate system. This includes tasks in which one object must be oriented relative to a second prior to assembly and tasks in which objects must be held in specific ways to inspect them. Research has investigated guidance mechanisms for some 6DOF tasks, using wide--field-of-view, stereoscopic virtual and augmented reality head-worn displays (HWDs). However, there has been relatively little work directed toward smaller field-of-view lightweight monoscopic HWDs, such as Google Glass, which may remain more comfortable and less intrusive than stereoscopic HWDs in the near future. We have designed and implemented a novel visualization approach and three additional visualizations representing different paradigms for guiding unconstrained manual 3DOF rotation, targeting these monoscopic HWDs. We describe our exploration of these paradigms and present the results of a user study evaluating the relative performance of the visualizations and showing the advantages of our new approach.

Game engines have become popular development platforms for real-time interactive systems. Contemporary game engines, such as Unity and Unreal, feature component-based architectures, in which an object's appearance and behavior is determined by a collection of component scripts added to that object. This design pattern allows common functionality to be contained within component scripts and shared among different types of objects. In this paper, we describe a flexible framework that enables programmers to design modular, reusable widgets for real-time interactive systems using a collection of component scripts. We provide a reference implementation written in C# for the Unity game engine. Making an object, or a group of objects, part of our managed widget framework can be accomplished with just a few drag-and-drop operations in the Unity Editor. While our framework provides hooks and default implementations for common widget behavior (e.g., initialization, refresh, and toggling visibility), programmers can also define custom behavior for a particular widget or combine simple widgets into a hierarchy and build arbitrarily rich ones. Finally, we provide an overview of an accompanying library of scripts that support functionality for testing and networking.

In many complex tasks, a remote subject-matter expert may need to assist a local user to guide actions on objects in the local user's environment. However, effective spatial referencing and action demonstration in a remote physical environment can be challenging. We introduce two approaches that use Virtual Reality (VR) or Augmented Reality (AR) for the remote expert, and AR for the local user, each wearing a stereo head-worn display. Both approaches allow the expert to create and manipulate virtual replicas of physical objects in the local environment to refer to parts of those physical objects and to indicate actions on them. This can be especially useful for parts that are occluded or difficult to access. In one approach, the expert points in 3D to portions of virtual replicas to annotate them. In another approach, the expert demonstrates actions in 3D by manipulating virtual replicas, supported by constraints and annotations. We performed a user study of a 6DOF alignment task, a key operation in many physical task domains, comparing both approaches to an approach in which the expert uses a 2D tablet-based drawing system similar to ones developed for prior work on remote assistance. The study showed the 3D demonstration approach to be faster than the others. In addition, the 3D pointing approach was faster than the 2D tablet in the case of a highly trained expert.

Assembly or repair tasks often require objects to be held in specific orientations to view or fit together. Research has addressed the use of AR to assist in these tasks, delivered as registered overlaid graphics on stereoscopic head-worn displays. In contrast, we are interested in using monoscopic head-worn displays, such as Google Glass. To accommodate their small monoscopic field of view, off center from the user's line of sight, we are exploring alternatives to registered overlays. We describe four interactive rotation guidance visualizations for tracked objects intended for these displays.

Many tasks in real or virtual environments require users to view a target object or location from one of a set of strategic viewpoints to see it in context, avoid occlusions, or view it at an appropriate angle or distance. We introduce ParaFrustum, a geometric construct that represents this set of strategic viewpoints and viewing directions. ParaFrustum is inspired by the look-from and look-at points of a computer graphics camera specification, which precisely delineate a location for the camera and a direction in which it looks. We generalize this approach by defining a ParaFrustum in terms of a look-from volume and a look-at volume, which establish constraints on a range of acceptable locations for the user's eyes and a range of acceptable angles in which the user's head can be oriented. Providing tolerance in the allowable viewing positions and directions avoids burdening the user with the need to assume a tightly constrained 6DoF pose when it is not required by the task. We describe two visualization techniques for virtual or augmented reality that guide a user to assume one of the poses defined by a ParaFrustum, and present the results of a user study measuring the performance of these techniques. The study shows that the constraints of a tightly constrained ParaFrustum (e.g., approximating a conventional camera frustum) require significantly more time to satisfy than those of a loosely constrained one. The study also reveals interesting differences in participant trajectories in response to the two techniques.

There has been considerable interest in producing grasping platforms using non-invasive, low bandwidth brain computer interfaces(BCIs). Most of this work focuses on low level control of simple hands. Using complex hands improves the versatility of a grasping platform at the cost of increasing its complexity. In order to control more complex hands with these low bandwidth signals, we need to use higher level abstractions. Here, we present a user interface which allows the user to combine the speed and convenience of offline preplanned grasps with the versatility of an online planner. This system incorporates a database of pre-planned grasps with the ability to refine these grasps using an online planner designed for arbitrarily complex hands. Only four commands are necessary to control the entire grasping pipeline, allowing us to use a low cost, noninvasive commercial BCI device to produce robust grasps that reflect user intent. We demonstrate the efficacy of this system with results from five subjects and present results using this system to grasp unknown objects.