Gemini Live rolls out AI features for screen and camera interaction

Officially confirmed by Google spokesperson Alex Joseph, the features are part of Google’s broader AI initiative, 'Project Astra'.

Google has introduced AI-driven features to its Gemini Live platform, revolutionizing the way users interact with their smartphone screens and cameras. This latest Gemini Live AI update brings real-time screen-reading and live video interpretation, enabling users to receive instant, context-aware responses about on-screen content or objects captured through their smart camera features.

With this update, Gemini Live can seamlessly interpret texts, images, and videos displayed on a user’s device. Whether analyzing a document, identifying an object in a live feed, or explaining an unfamiliar interface, these AI features enhance real-time assistance. This marks a significant step in integrating AI in smartphones, making interactions more intuitive and efficient.

Project Astra and the Future of AI-Driven Features

These upgrades are part of Google’s broader AI initiative, Project Astra, which aims to develop advanced AI-driven features for seamless, multimodal user experiences. Screen interaction capabilities allow users to ask Gemini Live questions about emails, documents, or webpages without needing to copy and paste text manually. Meanwhile, smart camera features empower users to get instant AI-powered insights about objects, landmarks, and even product recommendations based on live video feeds.

According to Alex Joseph, the screen-reading feature enables users to ask Gemini questions about any content visible on their smartphone screen, offering contextual responses. Meanwhile, the live video feature leverages a smartphone’s camera to provide real-time analysis of whatever is being viewed. For instance, users can ask Gemini to identify objects, suggest aesthetic decisions, or even guide them through tasks like choosing a paint colour for freshly-glazed pottery.

Initially, these Gemini Live AI updates will be accessible only to Gemini Advanced subscribers under the Google One AI Premium plan. Google has confirmed that the rollout will expand gradually, ensuring a smooth transition for users worldwide. As AI in smartphones continues to evolve, these innovations could redefine the way users engage with their devices, making everyday tasks more efficient and interactive. 

Also Read: Gemini Live will now be accessible to Hindi speakers