User Interface Surgical Navigation HoloLens 2 Application
I created the User Interface for an augmented reality surgical navigation application. I was the only designer and I worked closely with the development team. My experience includes the following:
Maximizing use of user inputs such as hand and eye tracking and voice interactions to create 2D and 3D interface for surgical navigation application for Hololens 2.
Creating designs in Figma using MRTK templates to create MRTK UI for unity application.
Creating UI designs through sketching, storyboarding, prototyping, visual effects, low-fi and high-fi wireframes.
Design and implementation of 3D special effect Splash Screen for HoloLens 2 application. I created this by prototyping particle effects and logo elements in Unity 3D.
Prototype screens and 3D models in Unity 3D.
Gathering feedback from surgeons for product iterations.
Composing literature and compiling images for Instructions For Use for the Surgical Navigation HoloLens 2 Application.
Collaborating on FDA submission documentation.
Collaboration on documentation for FDA submission including User Risk Analysis.
Testing Unity 3D branches.
Usability Testing of the surgical navigation application.
Simulation Testing of the surgical navigation application.
Explored viability of other hmd’s for our application.
Design Process
My experience in designing for augmented reality and virtual reality applications and my experience and knowledge of healthcare allowed me to contribute to the team of engineers and orthopedic specialists.
Need: Create a surgical navigation system that is as accurate as a robot but has a smaller footprint and lower cost for the patient and hospital.
Problem: Robots for Surgical Navigation systems are extremely accurate but remove much of the decision making for the surgeon. Robots are expensive and occupy a large footprint in the surgical suite. They do not allow the surgeon to have complete control of the operation.
Solution: Use an Augmented Reality headset in surgery combined with other tracking devices and our software to create a system that can measure as accurately as a surgical robot but give the surgeon complete control over the procedure.
My Role: I worked with a team of software engineers to create a HoloLens 2 Surgical Navigation application that can be used in surgery instead of a Robot for orthopedic surgery
I designed the user interface for the application by maximizing use of user inputs such as hand and eye tracking, audio, and voice interactions to create the 2D and 3D interface for a surgical navigation application for HoloLens 2.
Early Sketching and Ideation
Storyboards
Initial Storyboards for ideation were sketched on paper. These concepts were taken into virtual reality Sketchbox 3D and iterated on. Sketchbox 3D allows early testing of designs in an immersive environment without the hours required to create a design in Unity 3D. I created storyboards from the Sketchbox 3D scene. Software engineers used the final storyboards to set up scenes in Unity 3D.
UI Screen Development Process
UI Screens for the applications were initially created as sketches, Sketches were turned into Low-fi wireframes. Low-fi wireframes were developed further in Figma as Hi-fi wireframes. We contiuned to iterate on the designs and test in augmented reality. The software developers were able to created a basic UI for testing from the High-fi wireframes created in Figma. I continued to increase the fidelity of the screens in Figma to match the branding for the company. I worked with marketing and stakeholders to create the final screen designs.
Initial Screen Wireframes
Final Screen Design
Final screen designs. These are placed in augmented reality and the surgeon can interact with these in space. I maximized the use of user inputs such as hand, eye tracking, and voice interactions to capitalize on the benefits of immersive design. The screens respond to verbal cues as well as eye interaction and distance hand interactions. The screen can be moved in space to a location that works best for the surgeon. The panel can also be hidden and shown with voice commands. The benefits of augmented reality is having a screen available when needed but hidden when not needed.
I also created an intro effect for the application with audio, special effects, and logo to represent the company brand. The splash screen serves as confirmation to the surgeon that the application is running.
