Autonomous Taxi AI Innovation: Waymo Gemini In-Car Assistant System Unveiled

Waymo is developing an important new feature for its driverless taxis—a built-in AI assistant system based on Google Gemini. This discovery was revealed by well-known security researcher Jane Manchun Wong, providing a rare glimpse into the AI strategy of this Alphabet subsidiary autonomous vehicle company. As competition in the driverless taxi market intensifies, in-car AI experiences are becoming a new battleground for differentiation.

Hidden AI Assistant System Uncovered

Wong discovered this unreleased Gemini integration while deeply analyzing the Waymo mobile app. She shared her findings on her personal blog, noting that the system prompt file is internally codenamed “Waymo Ride Assistant Meta-Prompt.” This extensive technical document, over 1,200 lines long, details how the AI assistant should operate within Waymo vehicles.

According to Wong, this is more than just a simple chatbot. The system carefully designs the entire interaction framework for the AI assistant, covering everything from greetings to feature restrictions. A Waymo spokesperson, Julia Ilina, later confirmed that the company is testing various features to improve passenger experience but did not disclose specifics.

Beyond Simple Conversations in System Design

This AI assistant, created for driverless taxis, has a clear identity. According to the system prompt, Gemini is set as “a friendly and helpful AI companion integrated into Waymo autonomous vehicles,” with the main goal to “provide useful information and assistance in a safe, reassuring, and non-intrusive manner to enhance passenger experience.”

The system requires the AI to use concise, clear language, avoid complex technical jargon, and keep responses to one to three sentences. When passengers activate the assistant via the in-car screen, Gemini can choose from pre-approved greetings or even personalize greetings based on the passenger’s riding history. This design demonstrates Waymo’s careful consideration of the passenger experience in autonomous vehicles.

Carefully Designed Boundaries for In-Vehicle Controls

According to the system prompt, this AI assistant can access and control certain in-car functions, including temperature, lighting, and music. Notably, the designers deliberately excluded control over some features—volume, route changes, seat adjustments, and window controls are all on a “prohibited list.”

When passengers request Gemini to perform functions it cannot execute, the robot responds gently, such as: “That’s not something I can do yet.” This reflects Waymo’s focus on safety and managing passenger expectations. The system also explicitly instructs the AI to distinguish itself as Gemini AI robot from Waymo’s autonomous driving system (Waymo Driver). Therefore, when asked questions like “What do you think about the road?” the AI should cite “Waymo Driver uses multiple sensors…” rather than respond in the first person.

Special Instructions for Handling Sensitive Topics

The system prompt also includes detailed instructions for handling sensitive situations. The AI is explicitly told to avoid speculating, explaining, confirming, or denying topics related to real-time driving behavior, specific driving incidents, or competitors. If a passenger asks to see a video of Waymo hitting something, the AI is instructed not to respond directly and to steer clear.

The prompt emphasizes: “Your role is not to be a spokesperson for the driving system, and you must not adopt defensive or apologetic tones.” This indicates Waymo’s high regard for brand risk management when designing its autonomous vehicle AI assistant.

Meanwhile, the system allows Gemini to answer general knowledge questions—such as weather forecasts, historical facts, local store hours, or sports scores. However, it is prohibited from performing real-world actions like ordering food, making reservations, or handling emergencies, which are outside its scope.

Strategic Differences from Competitor Grok

Waymo is not the only company integrating AI assistants into autonomous vehicles. However, different implementations reflect varying understandings of the driverless taxi experience. In contrast, Grok (Elon Musk’s X.com AI assistant) in some autonomous applications functions more like a long-term companion, capable of extended conversations and remembering previous questions’ context.

Waymo’s Gemini, on the other hand, is designed to be more pragmatic and focused. It emphasizes information delivery and basic function assistance rather than deep conversational abilities. This design choice reflects Waymo’s understanding of the use case for driverless taxis—most passengers in short trips need quick, accurate help rather than lengthy dialogues.

The Bigger Picture of Waymo’s AI Strategy

It’s important to note that Gemini is not new to Waymo’s ecosystem. The company has already been using Gemini’s “world knowledge” to train perception and decision-making systems for autonomous driving, helping to handle complex, rare, and high-risk driving scenarios. As driverless taxis move toward mainstream adoption, AI not only powers vehicle autonomy but also shapes the overall passenger experience. As this technology matures, similar AI assistant systems could become a key factor in evaluating the competitiveness of driverless taxis.