“From driver to passenger”

The driving experience, or immersing yourself in in-car entertainment – what will be more important in future when we are traveling by car? What does the interaction between human and car look like in the future? And what characterizes a human-machine interface (HMI) as an automotive industry technology? All this is discussed in the IAA MOBILITY Visionary Club. Because with ever more elaborate navigation systems and more in-car entertainment offerings, the question of whether we will turn from “car drivers” into pure travelers in future arises – and especially if fully-autonomous driving becomes part of our daily lives.

The seamless merging of reality, driving experience and the vehicle’s environment – that awaits us in the future, through the further development of the automotive human-machine interface (HMI). In the car, this controls the operation of entertainment systems, the air-conditioning, navigation, head-up displays or voice recognition technologies, along with safely-relevant applications. HMIs are generally housed in the central console or a as central display in the dashboard.

Ulrich Lüders, Director Strategy and Portfolio, UX Business Area at Continental, describes the seamless interplay in the interaction of human and vehicle: On the one hand, it is essential for a consistent experience to be possible from getting into the car through to driving – one that is not interrupted, for instance, by connecting your mobile. “The interaction with the car will be multi-modal – a mix of easy-to-operate touch functions and buttons, but also new experiences such as Augmented Reality and XR.” XR, short for Extended Reality, links reality with an artificial environment.

Cars and metaverse – a unique connection

Augmented Reality (AR) connects digital content with the real world, and allows users to immerse themselves partially in a virtual world. XR and AR make the interaction with the car more intuitive and more immersive, Lüders is convinced. Immersive – this term from the gaming world is increasingly gaining a foothold in the automotive sector too. In computer games, it means that the players immerse themselves fully in the fictitious imaginary world of the game. In the car, too, the virtual and the real world can be linked together – for instance, with the navigation system displaying restaurants or supermarkets, arrows showing driving directions in the head-up display as if they were marked on the road, or games on the entertainment system incorporating the movements of the actual vehicle.

Nils Wollny has developed a Virtual Reality (VR) game that does precisely that:  The car’s movement data is calculated into the game by linking the vehicle sensors with a proprietary VR headset. The car’s speed, braking and bends are transmitted to the headset in real time. For instance, a dinosaur in the game starts to dive if the car accelerates as it descends a hill. Wollney, CEO and co-founder of the Munich-based startup Holoride, is convinced that “the next decade will be dominated by the metaverse moving into cars – and cars into the metaverse.”

New content at your fingertips

Fabian Beste, co-founder and CEO of 4.screen, is convinced that the future will be characterized by car passengers and the ecosystem around the vehicle interacting with one another. The platform developed by 4.screen builds on the navigation system. Additional information, such as “third-party provider content and stores are displayed for the driver, at the tap of a finger,” he explains. And matched to the driver’s profile too. “The possibilities for interaction will develop massively,” Beste believes. Linking infrastructure components and the physical environment, such as local stores, is one of the key factors in making an experience truly seamless, he says. 

The vital basis for all these developments? A stable data connection. That’s because, as Wollny says, “real-time data is essential to enable a highly vivid and immersive experience.” For the human-car interaction, this means more than just the data linked with the world outside. “The vehicle data is unique,” Beste notes. Using sensors that cannot be recorded by the smartphone, the car’s status – in all aspects, from its current speed to the necessary service interval – is revealed. “The situation the driver currently finds himself in is the key to a better user interaction in the car,” says Beste.

These are the biggest challenges right now for implementation

When it comes to HMI, the car industry is in the middle of change. “The technologies are now all coming together,” Lüders explains. “Most vehicles with devices and screens are being launched in the next few years.” On the other hand, the consumer goods industry has a big influence too. It is already a lot further down that road, says Wollny. In the automotive sector, developments are still very screen-based. “The car industry should understand that it has everything needed to transform cars into metaverse devices.” The manufacturers have relatively long product lifecycles, in part due to the necessary tests on safety and security. “But they are increasingly opening up for third-party applications, and this openness will speed up development.”

The most important thing is to design the interaction to be easy, says Beste. That’s because while the driver needs to steer the car, he or she cannot interact much with the infotainment. “Integrating it seamlessly into an existing product is one of the challenges for us.” Transforming the car into an “extended habitat” is a key factor. Pleasant, comfortable, simple, and safe. With an ‘at-home’ feeling during the journey. 

HMI in the next five years: A focus on passengers, and more services

There will be a distinction drawn between driver and passenger, Lüders believes. More entertainment for passengers, more concentration for drivers – at least, for as long as autonomous driving is not yet a reality. Wollny takes a similar view: “In the past decade, the car industry has concentrated on the driver and neglected the passengers.” For that reason, he believes that in future the focus will be on the passengers, from design to use cases. Moreover, integration of the surroundings will become more relevant, Beste is convinced: “Based on the good interaction, many services will become possible – for instance, restaurants sponsoring battery charging while you eat there.”

The one-time car driver, whose vehicle mainly served as a means of transport, has become a user equipped with screens and information offerings. Ulrich Lüders concedes that “driver distraction is a big issue.” For that reason, operating the vehicle needs to be intuitive, and immersive offerings should not distract the person behind the wheel. Nils Wollny refers to the different roles assumed by the driver and the passengers. During the trip, passengers have entirely different needs.

Accordingly, one solution could be to play the offerings for passengers via devices such as a VR headset, instead of via screens that might distract the driver. “For safety reasons, [the entertainment] is only for the relaxed zone in the back seats, where you can immerse yourself in entertainment content,” Wollny says. Despite this, the business potential here is huge: “Worldwide, there are at least 1.4 billion journeys per day with a passenger on board.” And as soon as cars are fully self-driving, this development will speed up, “because we will all be passengers”.

HMI is creating new business models for stores and restaurants

In-car entertainment via HMI means more than simply passengers consuming films, games, or music. Fabian Beste outlines how relevant information, such as searching for charging points or stores, can become a new business model. Customers are attracted by incorporating information, for instance in games or navigation. How important is this digital experience for tomorrow’s consumer? “Essential,” says Nils Wollny, speaking in the IAA MOBILITY Visionary Club. “The consumer always makes comparisons with the consumer goods industry, where the trends emerge from,” adds Lüders. “Consumers expect the same seamless experience in the car.” Flat screens similar to the smartphone are just one example of this. Consumers wanted to see them in vehicles too.

Fabian Beste takes that further: The foundations need to be in place first, and then you can create better experiences for the user “on top” – by incorporating unique car data supplied by sensors. Data such as speed, the gap to the car in front, or interactions between driver and car. Wollny’s hope is that the way we perceive mobility will change: The car not purely as a means of transport that lags behind the consumer goods industry, but the car as somewhere you experience something. The car not as an extension to the living-room, but as a unique place moving through the world.

Viewed like that, mobility can influence decisions – an idea that Beste teases out: If you are able to watch a good film during a trip, traveling with low emissions, then maybe you are more likely to visit a friend living further away than you would have been.

Car manufacturers or startups – who does it better?

When it comes to HMI and entertainment systems, many startups and third-party providers are providing fresh impetus. But is it not better for this field to be left to the car manufacturers? “Pressure is good,” says Lüders, emphasizing that the increasing competition in current developments is positive. He sees the future of the digital in-car experience not solely in the hands of the Californian tech giants. Fabian Beste draws a distinction between entertainment and functions concerned with driving the car. Music apps, for example, should be seamlessly integrated from outside. But there is also data focused on mobility that needs to be provided from the car – such as the battery charge display, for example.

Wollny similarly sees two levels: One concerns the general operating system, around which there is set to be a “massive struggle in the years ahead”. The second concerns the applications: How can genuine value-added be created for passengers? And how can manufacturers be part of the value chain? Here, too, the previous mind-set needs to change: What happens if you extend the car into the dominant ecosystems for consumers? And if you develop applications that only the automotive industry can contribute?

Through all this discussion of the entertainment and HMI systems of the future, one thing above all becomes clear: The car is a unique world that cannot be compared with anything else – and its further development and future integration into our daily lives is one of the most exciting projects right now for the car industry.