Google Previewed Gemini’s AI-Powered Features at Google I/O in May
Google’s Gemini AI is rolling out two advanced features for Android users, enhancing the way they interact with on-screen content. The new tools, ‘Ask About This Screen’ and ‘Ask About This Video’, leverage AI to analyze real-time content from a screenshot or active YouTube video. When invoked, the AI processes the visual information on the screen, allowing users to pose context-aware queries directly related to the content they are viewing. For videos, it can even generate AI-driven summaries, provided that captions are available.
The ‘Ask About This Screen’ feature works by capturing a screenshot of the user’s display, interpreting the text, images, or any other data present, and delivering relevant insights or information when prompted. This allows users to interact dynamically with the content without needing to manually research or navigate away. For instance, if a user is browsing a webpage or reading an article, they can ask Gemini for clarifications, background information, or even follow-up details, all based on the content currently on display.
For videos, the ‘Ask About This Video’ feature functions similarly, but with a focus on summarizing video content. At present, this capability is limited to YouTube videos that have captions. The AI analyzes the captions to generate concise summaries or respond to user queries about specific parts of the video. This feature is particularly useful for users who want to quickly grasp the essence of a video without watching it in full or need more detailed information about a specific segment.
These features were first observed by Gadgets 360 in the Google app version 15.33.38.28.arm64 across various Android devices. Once activated, a floating Gemini AI assistant window appears, offering the user a rectangular prompt labeled “Ask about this screen,” or, in the case of YouTube videos, “Ask about this video.” This seamless integration makes interacting with on-screen content far more intuitive and efficient.
This new functionality builds on the AI capabilities that Google initially previewed at Google I/O 2023. These tools are part of a broader strategy to embed AI-enhanced user interfaces within Google’s ecosystem, enabling a more context-sensitive user experience. As these features roll out, more users will likely see enhanced capabilities in how they interact with everyday media, making multitasking and information retrieval faster and more streamlined.
Given the growing importance of contextual AI in enhancing productivity and user interaction, these additions to Gemini could reshape the way users engage with content, merging on-demand information processing with real-time applications.