Apr. 2024 | Arizona State University | Applied Project, Graduate Second Year
Individual Project | Supervisor : Nicholas Pilarski
Project statement
Augmented Cockpit is an innovative AR experience designed to showcase the concept of seamless integration of Augmented Reality technology and hand tracking within the vehicle’s center console system. By leveraging advanced AR capabilities and intuitive hand tracking, this project aims to redefine the in-car experience, providing users with an immersive and intuitive interface for accessing essential vehicle controls and information.
Through Augmented Cockpit, users can interact with virtual displays and controls overlaid onto the physical environment, enhancing safety, convenience, and overall driving experience. This groundbreaking project represents a significant step forward in the evolution of automotive technology, offering a glimpse into the future of connected and interactive vehicle interiors.
Motivation
Automobiles have always fascinated me, and this project has been brewing in my mind since the beginning of my master’s program. I’ve noticed a prevalent issue among car reviewers regarding the use of touch screens and the elimination of physical buttons in car interior designs. While touch screens contribute to a cleaner and simpler aesthetic, they often lack the intuitive functionality of traditional buttons and can divert the driver’s attention from the road for longer periods.
As time progresses, it’s unrealistic to expect consumers to revert to cars with solely button-based interfaces. However, with the advent of the technology, there may be alternative solutions to this problem. As a designer, I’m eager to explore and present my insights on this matter through this project.
Goals
The objective of this project is to showcase a safer and more intuitive user interface utilizing Augmented Reality (AR) technology and hand tracking. The aim is to empower users to control various functions seamlessly by directly pointing towards objects like air conditioning vents and speakers. All necessary information will be displayed without necessitating the user to divert their attention away from the road, thereby enhancing safety and user experience.
Hypotheses
In traditional center console designs with physical buttons, each area is distinctly allocated for specific functions, enabling users to locate and operate them effortlessly. With the touch screens however, these designated areas have merged into one, requiring users to navigate through menus to access desired functions, resulting in increased time and concentration. Through the approach of enabling users to directly point towards functions to access them, we reintroduce the concept of dedicated areas, simplifying user interaction and improving overall user experience.
Final Outcome
Design
Given the requirement for users to interact by pointing at various functions, it’s essential to ensure that each function is mounted separately on the center console. Additionally, without the constraints of a physical interface, there’s no longer a need for excessive space to house additional machinery and wiring. Hence, I’ve opted for a design with a more unconventional approach, one that includes only the necessary components.
Fabrication
For the smaller parts such as the steering wheel, I utilize 3D printing with PLA, a material known for its speed in printing, albeit being more brittle. As for the larger parts, they are crafted using Foamular NGX F-150 foams and carved with the CNC machine.
Experience Design
The primary objective of this project is to reduce driver distractions. To achieve this, information will be conveyed to the user through colors displayed on the edge of the windshield rather than text and numbers.
Additionally, all controls will be operated via gestures, as listed below.
Hand Tracking
Hand tracking in this project is achieved using the Leap Motion Controller 2, with support from the Ultraleap Unreal Engine Plugin.
To set up the hand tracking, we'll need to download and run the Gemini: Ultraleap Hand Tracking Software, and then activate the Ultraleap Tracking Plugin.
VR Passthrough
Augmented Reality in this project is achieved through the Passthrough Mode on the Oculus Quest 3. To enable both Passthrough and Leap hand tracking on a Meta headset, we need to connect the headset to a PC via Quest Link, activate developer mode on the headset, enable Developer runtime feature and passthrough in Oculus Desktop APP. Additionally, activating the MetaXR plugin in Unreal Engine is necessary.
Current Capabilities and Possible Improvements
Presently, the project demonstrates the use of AR technology in creating an intuitive interface that allows drivers to control vehicle functions without diverting their attention from the road. It provides a foundational model that can inspire further research and development. Overcoming the existing hardware limitations and refining software integration are the next steps in making this technology widely accessible.
For current practical use, relocating the motion sensor from the front of the headset to the area where the rearview mirror is located could provide a wider sensing range. In terms of visual feedback, while AR offers a better experience, alternatives such as LED panels or reflections on the edge of the windshield can achieve similar effects. Additionally, developing a set of more intuitive gesture languages, as outlined below, would further enhance the interface's usability
Future Potential in the Next Ten Years
Looking ahead, the practical applications of the Augmented Cockpit project over the next decade could be transformative. As AR and hand-tracking technologies evolve, we can anticipate the development of more sophisticated and responsive interfaces that integrate seamlessly with various in-car systems. Future advancements might include:
Enhanced Safety Features:
Real-time hand tracking and AR interfaces could be integrated with advanced driver-assistance systems (ADAS), providing drivers with contextual information and alerts without requiring them to take their eyes off the road.
Augmented Navigation:
AR could project navigation instructions directly onto the windshield, highlighting lanes, turns, and points of interest in real-time, vastly improving driving ease and reducing cognitive load.
Broader Implication
The findings and methodologies from this project can significantly contribute to the fields of AR and human-computer interaction, serving as a benchmark for future innovations in automotive technology and beyond. Industries such as aerospace, marine navigation, and even consumer electronics can benefit from the advancements demonstrated in this project.
In summary, the Augmented Cockpit holds promise for safer, more intuitive driving experiences and sets a foundation for ongoing innovation in the integration of advanced technologies within vehicles. Over the next ten years, the advancements in this field could fundamentally alter how we interact with our cars, making driving safer, more efficient, and more enjoyable.
Reflection
Taking on the Augmented Cockpit project solo brought a mix of excitement and challenges. It was the largest model I’d ever tackled, demanding a hefty investment of time, resources, and technical skill. Navigating the fabrication process, with its intricacies and tight timelines, was especially tough. But seeing the project gradually take shape before my eyes was incredibly fulfilling.
One of the toughest tasks was implementing pass-through mode and hand tracking in Unreal Engine on a tight schedule. It meant diving into unfamiliar territory and wrestling with the complexities of AR tech. Despite facing errors and instability with Oculus features and plugins, I stayed determined, seeking help online and from my supervisor.
Working solo on such a big project had its share of challenges, but it also offered unmatched opportunities for growth. I honed my resilience, adaptability, and problem-solving skills along the way. And it made me aware of my tendency to over-fixate on a single solution, and reminded me of the importance of seeking input from others when necessary.
Looking back on the Augmented Cockpit project, it’s more than just about creating a high-tech interface. It’s been a journey of personal and professional growth. Plus, it’s deepened my passion for innovative design and tech-driven solutions that enhance human experiences.
Photo Gallery
