A Practical Analysis of the Gemini API's URL Context Tool

NodeTips / jstips
Gemini / Node.js / javascript

Gists

Introduction

The Gemini API recently introduced the URL context tool, a feature designed to allow the model to directly fetch and utilize content from specified URLs to ground its responses. Ref

This report provides a practical demonstration of this tool’s capabilities. We will investigate its impact on two critical aspects of AI model interaction: the accuracy of the generated response and the total token consumption, which directly affects API costs.

Experimental Setup

The analysis consists of two comparative tests:

  1. Response Accuracy: We compare the answers generated by Gemini with and without the URL context tool for the same complex query.
  2. Token Consumption: We compare the number of tokens required to process the query when providing context manually versus using the URL context tool.

Prompt

The following prompt is used for all tests to ensure consistency. It asks Gemini to determine if a Google Apps Script Web App can function as an A2A (Agent-to-Agent) server, a task that requires understanding a specific, non-obvious technical capability.

const prompt = [
  `Read and understand all the content of all sites from "URLs". Follow "Mission".`,
  `<URLs>`,
  `- https://github.com/tanaikech/taking-advantage-of-Web-Apps-with-google-apps-script`,
  `</URLs>`,
  `<Mission>`,
  `- You are required to implement the A2A server using the Web Apps of Google Apps Script.`,
  `- In order to use the Web Apps as the A2A server, it is required to access the path "/.well-known/agent.json". Consider whether this can be achieved using the Web Apps.`,
  `- As a result, consider whether the Web Apps of Google Apps Script can be used as the A2A server.`,
  `- Return a simplest possible answer.`,
  `</Mission>`,
].join("\n");

Experiment 1: Impact on Response Accuracy

Without the URL Context Tool

In this scenario, we provide the URL in the prompt and enable Google Search for grounding, but do not use the specific urlContext tool.

import { GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: "###your API key###" });
const response = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: prompt,
  config: { tools: [{ googleSearch: {} }] },
});
console.log(response.text);

This script returns the following incorrect result:

No. Google Apps Script Web Apps have a fixed URL structure (e.g., https://script.google.com/macros/s/EXECUTION_ID/exec) and cannot serve content directly from the domain root or a specific path like /.well-known/agent.json.

Analysis: Without the urlContext tool, the model relies on its general training data and standard Google Search results. This information correctly identifies the typical limitations of Web Apps but fails to find the specific technique described in the provided GitHub repository. The model makes an educated but ultimately wrong assumption.

With the URL Context Tool

Here, we add the urlContext tool to the request, instructing Gemini to fetch and analyze the content from the URL directly.

import { GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: "###your API key###" });
const response = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: prompt,
  config: { tools: [{ urlContext: {} }, { googleSearch: {} }] },
});
console.log(response.text);

This script returns the following correct result:

Yes, Google Apps Script Web Apps can be used as an A2A server. They can handle requests to the /.well-known/agent.json path by utilizing the pathInfo property within the event object (e) of the doGet(e) or doPost(e) functions. This allows the Web App to extract and process the /.well-known/agent.json segment from the request URL to serve the required content.

The urlMetadata in the API response confirms the source was successfully retrieved:

[
  {
    "retrievedUrl": "https://github.com/tanaikech/taking-advantage-of-Web-Apps-with-google-apps-script",
    "urlRetrievalStatus": "URL_RETRIEVAL_STATUS_SUCCESS"
  }
]

Analysis: The urlContext tool forces the model to use the GitHub page as a primary source of truth. The repository’s README file contains the exact technical explanation of how to use pathInfo to handle custom URL paths. By directly grounding its response in this specific, relevant content, the model provides a nuanced and accurate answer.

Experiment 2: Impact on Token Consumption and Cost

Without the URL Context Tool (Manual Context)

To achieve a correct answer without the tool, the developer must manually fetch the content of the URL and include it directly in the prompt. This adds complexity to the client-side code and significantly increases the prompt size.

const res = await fetch(
  "https://raw.githubusercontent.com/tanaikech/taking-advantage-of-Web-Apps-with-google-apps-script/refs/heads/master/README.md"
);
const text = await res.text();
const prompt = [
  `Read and understand all the content of "Text". Follow "Mission".`,
  `<Text>${text}</Text>`,
  // Mission remains the same...
].join("\n");

const response = await ai.models.countTokens({
  model: "gemini-2.5-flash",
  contents: prompt,
});
console.log(response.totalTokens);

This script reports a total token count of 42,911.

With the URL Context Tool

When using the tool, the prompt remains small and clean, as the content retrieval is handled by the API.

// The prompt is the original, short version.
const response = await ai.models.countTokens({
  model: "gemini-2.5-flash",
  contents: prompt,
});
console.log(response.totalTokens);

This script reports a total token count of 162.

Analysis: The URL context tool resulted in a 99.6% reduction in token usage for this task. This is a dramatic difference that directly translates to significant cost savings and enables the processing of documents that would otherwise exceed the model’s context window limits.

Summary

This investigation demonstrates that the URL context tool is a transformative feature for the Gemini API.

  • Improves Accuracy: By grounding responses in developer-specified sources, the tool ensures the model provides accurate, context-aware answers, effectively mitigating the risk of hallucinations or reliance on outdated general knowledge.
  • Reduces Cost and Complexity: It fundamentally shifts the paradigm from “prompting with data” to “prompting with a reference to data.” This dramatically reduces input token counts, leading to substantial cost savings and simplifying application logic by offloading content retrieval to the API.

The results clearly indicate the importance and power of the urlContext tool for building sophisticated and reliable applications with the Gemini API.

 Share!