Capabilities
Most of what the assistant can do happens inside a conversation:
your user asks, and the assistant decides what to use — a search, a look at an attached photo, a
note it keeps for later — and streams the result back, with each step as a
task.progress frame. Media, voice and transcription can also be
called on their own.
In a conversation
You do not call these one by one: the assistant uses them as the conversation needs, and the console's calls page lists each use.
Search and research
| Capability | What it gives your user |
|---|---|
| Web search | An answer drawn from current web pages, with its sources. |
| News | Recent news on a topic. |
| Public discussions | What people are saying about a topic on public social platforms. |
| Academic papers | Papers and their details: authors, year, abstract. |
| Products | Products with prices and merchant links. |
| Images | Images from the web that match a description. |
| Videos | Videos on video platforms that match a topic. |
| Places | Places by name or kind, near a location or anywhere. |
| Reading a web page | The main content of a page at a public address. |
| Deep research | A cited report on a question, from many sources, delivered as a file with a summary. It takes minutes; the assistant follows it to the end on its own. |
Understanding
| Capability | What it gives your user |
|---|---|
| Images | What is in a photo the user attached: objects, text, details — and answers to questions about it. |
| Video | What happens in a video the user attached, and answers to questions about it. |
Memory
The assistant keeps what a user shares that is worth keeping — a preference, a plan, a fact about them — and recalls or searches it in later conversations. Each end user's memory is their own: no other user, in your project or any other, can reach it.
Creating
In a conversation, the assistant makes and edits images, videos and music when the user asks, and
the result arrives as a media.updated frame. The same
capabilities have operations of their own — see below.
Choosing a session's capabilities
When you create a session — a text or a voice one — you choose what the assistant may use in it. Send one of:
capabilities— open only these. A capability added later never reaches the session.exclude_capabilities— open every capability but these. A capability added later opens in the sessions you create after it is added.
Send neither to open them all. Either way the session's capabilities are settled when it is
created and hold for its whole life; the session reports them in its capabilities field. When a
user asks for something the session left out, the assistant says it is not available, and
nothing is charged for it. Memory is not a capability: every session keeps it.
| Capability | What it covers |
|---|---|
search | Every search above, and reading a web page. |
deep_research | Deep research reports. |
understanding | A close look at attached images, and what happens in an attached video. Without it the assistant still sees an attached photo. |
image_generation | Making and editing images in the conversation. |
video_generation | Making and editing videos in the conversation. |
music_generation | Making music in the conversation. |
documents | Files the assistant builds for the user — a PDF, a document, a spreadsheet, slides — and heavier work on the files a user attaches. |
To keep a session to a fixed set — what your app can show today, say — list it in
capabilities. To take each new capability as it comes, name only what you do not want in
exclude_capabilities. Generating images, video or music, building files and deep research all
produce media or files: leave out the ones your app cannot show.
More capabilities may be added. A request is refused with 422 validation_failed when it names a
capability not in this list (param names the entry, such as capabilities[1]), sends both
fields (param: exclude_capabilities), or sends either field to rejoin a voice session, which
keeps the capabilities it was created with (param names the field). A capability left out of a
session still has its operations of its own.
With operations of their own
| Capability | Operations |
|---|---|
| Images: make one from a description, or edit one | Generations |
| Video: from a description, a first or last frame, or reference images, video and audio; edit part of a video, shoot a new take, transfer motion, reframe, add a soundtrack | Generations |
| Music: a song from a description | Generations |
| Voice: a spoken conversation in real time | Realtime voice |
| Transcription: a recording turned into text | Transcription |
| A brief of a voice conversation | Session briefs |
| UI your app renders, in the open A2UI format | A2UI replies |
Every capability is charged in credits from the project's balance — see Credits and billing.