Self-driving cars could better understand their passengers with ChatGPT, study finds

Imagine simply telling your vehicle, “I’m in a hurry,” and it automatically takes you on the most efficient route to where you need to be.

Engineers at Purdue University have discovered that an autonomous vehicle (AV) can do this with the help of ChatGPT or other chatbots made possible by artificial intelligence algorithms called large language models.

The study, which appears on the preprint server arXivis due to be presented on September 25 at the 27th IEEE International Conference on Intelligent Transportation Systems. It could be one of the first experiments testing the ability of a true autonomous vehicle to use large language models to interpret a passenger’s commands and drive accordingly.

Ziran Wang, an assistant professor at Purdue University’s Lyles School of Civil and Construction Engineering who led the study, says that for vehicles to ever become fully autonomous, they will need to understand everything their passengers command them to do, even when the command is implicit. A taxi driver, for example, would know what you need when you say you’re in a hurry without you having to tell them how to avoid traffic.

While current AVs have features that allow you to communicate with them, they require you to be more clear than if you were talking to a human. In contrast, large language models can interpret and provide responses in a more human-like way because they are trained to make connections from massive amounts of text data and continue to learn over time.

“Traditional systems in our vehicles have a user interface where you have to press buttons to convey what you want, or an audio recognition system that requires you to speak very explicitly for the vehicle to understand you,” Wang said. “But the power of large language models is that they can understand more naturally all kinds of things that you say. I don’t think any other existing system can do that.”

Conducting a new type of study

In this study, the large language models did not drive an autonomous vehicle. Instead, they assisted the autonomous vehicle in driving by using its existing features. Wang and his students found that by incorporating these models, an autonomous vehicle could not only better understand its passenger, but also personalize its ride to their satisfaction.

Before starting their experiments, the researchers trained ChatGPT with commands that ranged from more direct commands (e.g., “Drive faster”) to more indirect commands (e.g., “I’m feeling a little car sick right now”). As ChatGPT learned to respond to these commands, the researchers gave its large language models parameters to follow, asking it to take into account traffic rules, road conditions, weather, and other information detected by the vehicle’s sensors, such as cameras and light detection and ranging.

The researchers then made these large language models accessible via the cloud to an experimental vehicle with Level 4 autonomy as defined by SAE International. Level 4 is a step away from what the industry considers a fully autonomous vehicle.

When the vehicle’s voice recognition system detected a passenger’s command during the experiments, large language models in the cloud reasoned about the command using parameters the researchers set. These models then generated instructions for the vehicle’s electrical drive system (which is connected to the accelerator, brakes, gears and steering) about how to drive based on that command.

For some experiments, Wang’s team also tested a memory module they installed in the system that allowed the large language models to store data about the passenger’s historical preferences and learn to take them into account in a response to a command.

The researchers conducted most of the experiments on a proving ground in Columbus, Indiana, which was formerly an airport runway. This environment allowed them to safely test the vehicle’s responses to a passenger’s commands while driving at highway speeds on the runway and negotiating two-way intersections. They also tested the vehicle’s ability to park in accordance with a passenger’s commands in the parking lot of Purdue’s Ross-Ade Stadium.

Study participants used both the commands that the large language models had learned and the new commands while riding in the vehicle. Based on their survey responses after their rides, participants expressed lower levels of discomfort with the decisions made by the autonomous vehicle compared to data on how people tend to feel when riding in a Level Four autonomous vehicle without the help of the large language models.

The team also compared the autonomous vehicle’s performance to benchmarks created from data on what people on average consider safe and comfortable driving, such as how long it takes for the vehicle to react to avoid a rear-end collision and how quickly the vehicle accelerates and decelerates. The researchers found that the autonomous vehicle in this study outperformed all benchmarks when using the large language models to drive, even when responding to commands the models hadn’t yet learned.

Future directions

The large language models in this study took an average of 1.6 seconds to process a passenger’s command, which is considered acceptable in non-time-critical scenarios but should be improved in situations where an autonomous vehicle needs to respond more quickly, Wang said. This is a problem that plagues large language models in general and is being addressed by industry as well as academic researchers.

While not the focus of this study, large language models like ChatGPT are known to “hallucinate,” meaning they can misinterpret something they’ve learned and react incorrectly. Wang’s study was conducted in a setup with a failsafe mechanism that allowed participants to safely ride when the large language models misunderstood commands. The models improved their understanding throughout the participant’s ride, but hallucinations remain a problem that needs to be addressed before vehicle manufacturers consider integrating large language models into autonomous vehicles.

Automakers should also conduct more tests with large language models, in addition to studies conducted by academic researchers. Regulatory approval would also be needed to integrate these models into the autonomous vehicle’s controls so they can actually drive the vehicle, Wang said.

In the meantime, Wang and his students continue to conduct experiments that could help the industry explore adding large language models to AVs.

Since their study of ChatGPT, the researchers have evaluated other public and private chatbots based on large language models, such as Google’s Gemini and Meta’s Llama AI series of assistants. So far, they’ve found that ChatGPT performs best on indicators of safe and efficient autonomous vehicle travel. The results will be published soon.

The next step is to see if it would be possible for the large language models in each autonomous vehicle to communicate with each other, for example to help autonomous vehicles determine which one should go first at a four-way stop. Wang’s lab is also launching a project to study using large vision models to help autonomous vehicles drive in extreme winter conditions, which are common throughout the Midwest. These models are similar to large language models, but trained on images rather than text.