Connect Agents to Structured Web Data: Mastra, ADK & MCP

MCP gives an agent one way to discover and call tools, but it does not make every source return the same fields. This guide connects Crawlora to Mastra, Google Agent Development Kit (ADK), Gemini CLI, and common MCP clients, then shows how to preserve the source and completeness signals in the response.

The details below were checked against the linked vendor documentation on October 21, 2026. Setup times are author estimates for an existing app with the relevant framework installed and a Crawlora API key available; they are not timed tests or promises.

Integration matrix (checked 2026-10-21)

These estimates assume the application/client and model are already working. They cover adding the connection and confirming tool discovery, not creating an account, provisioning credentials, building the agent, or testing a full workflow. For details that change by client version, use the client's current setup docs. The Crawlora MCP README lists current client-specific configuration examples.

Mastra

Mastra's MCPClient can connect to a remote server and expose discovered tools with getTools(). Keep the API key in the server runtime environment. This example adds the bearer header in a custom fetch function:

import { MCPClient } from "@mastra/mcp";

const crawlora = new MCPClient({
  servers: {
    crawlora: {
      url: new URL("https://mcp.crawlora.net/mcp"),
      fetch: async (url, init) => {
        const key = process.env.CRAWLORA_API_KEY;
        if (!key) throw new Error("CRAWLORA_API_KEY is not configured");

        const headers = new Headers(init?.headers);
        headers.set("Authorization", `Bearer ${key}`);
        return fetch(url, { ...init, headers });
      },
    },
  },
});

const crawloraTools = await crawlora.getTools();
// Pass crawloraTools to the `tools` option on your Mastra Agent.

The MCPClient install and getTools() flow follow Mastra's MCP guide; Mastra documents per-server credentials through request headers or a custom fetch in its MCP authentication guide.

Google ADK

ADK treats a remote MCP server as a toolset in the agent's tools list. This Python example uses the current StreamableHTTPConnectionParams pattern:

import os

from google.adk.agents import LlmAgent
from google.adk.tools.mcp_tool import McpToolset
from google.adk.tools.mcp_tool.mcp_session_manager import StreamableHTTPConnectionParams

crawlora_tools = McpToolset(
    connection_params=StreamableHTTPConnectionParams(
        url="https://mcp.crawlora.net/mcp",
        headers={
            "Authorization": f"Bearer {os.environ['CRAWLORA_API_KEY']}"
        },
    ),
)

root_agent = LlmAgent(
    model="gemini-flash-latest",
    name="web_data_agent",
    instruction="Use Crawlora tools for supported public web data, and report source and completeness.",
    tools=[crawlora_tools],
)

Use your chosen model and the version-specific imports from the ADK MCP tools guide. For a multi-user app, ADK also supports a header_provider for request-scoped credentials; do not put one user's access token into a shared static tool configuration.

Gemini CLI and other MCP clients

Gemini CLI uses httpUrl for a Streamable HTTP server. This is a local configuration example; supply the credential through a private local setting or a supported secret mechanism, and do not commit a populated settings file:

{
  "mcpServers": {
    "crawlora": {
      "httpUrl": "https://mcp.crawlora.net/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_CRAWLORA_API_KEY"
      }
    }
  }
}

The config key differs across clients: Gemini CLI uses httpUrl; VS Code uses a servers root and type: "http"; Windsurf uses serverUrl; Cursor and Claude clients use their own settings formats. Some clients expand environment variables in headers and some don't. Check that behavior before relying on a placeholder such as ${CRAWLORA_API_KEY}. Store the real key outside source control and keep it out of browser code.

Codex version note: in a test on Oct 19, 2026, Codex CLI 0.156.1 stopped after the first 1,000 tools from the full /mcp catalog. That version-specific test used Crawlora's /mcp/agent entry point to reach the catalog through tool search and dispatch. The public MCP README documents /mcp as the standard endpoint; verify current Codex behavior and follow its client docs before choosing an endpoint. See the Codex and Cursor plugin test notes for the dated observation.

What the data looks like

An MCP call is the transport and tool envelope. It is not a guarantee that every tool returns a native object in the same shape. The MCP TypeScript SDK exposes content, checks isError, and may expose structuredContent when a tool declares an output schema. An application should check the MCP result first, then parse and validate the endpoint payload it actually receives.

At the REST layer, each endpoint has its own documented schema. For example, the current Reddit search endpoint returns public post records, the Reddit source URL/type, and a pagination cursor when another page is available. The catalog's example shape is shown below; its values are illustrative, not a live result:

{
  "code": 200,
  "msg": "OK",
  "data": {
    "query": "gpt",
    "subreddit": "OpenAI",
    "posts": [
      {
        "id": "1abcxyz",
        "title": "OpenAI discussion thread",
        "created": "2024-04-19T00:00:00Z",
        "permalink": "https://www.reddit.com/r/OpenAI/comments/1abcxyz/openai_discussion_thread/",
        "source_feed_url": "https://www.reddit.com/r/OpenAI/search?q=gpt&restrict_sr=on&sort=relevance"
      }
    ],
    "pagination": { "limit": 25, "after": "t3_1abcxyz" },
    "source": {
      "type": "feed",
      "url": "https://www.reddit.com/r/OpenAI/search?q=gpt&restrict_sr=on&sort=relevance"
    }
  }
}

created is when the post was published; it is not when your application fetched it. Record your own observed_at timestamp alongside the response. Keep the source.url (or source_feed_url) and the after cursor with the result so a later agent or user can audit the source and see whether more pages remained.

This small TypeScript wrapper keeps those details and leaves error outcomes distinct from search results:

type RedditSearchResponse = {
  code?: number;
  msg?: string;
  data?: {
    posts?: Array<{
      id?: string;
      title?: string;
      created?: string;
      permalink?: string;
      source_feed_url?: string;
    }>;
    pagination?: { limit?: number; after?: string };
    source?: { type?: string; url?: string };
  };
};

export async function searchPublicReddit(query: string) {
  const apiKey = process.env.CRAWLORA_API_KEY;
  if (!apiKey) throw new Error("CRAWLORA_API_KEY is not configured");

  const url = new URL("https://api.crawlora.net/api/v1/reddit/search");
  url.search = new URLSearchParams({ q: query, sort: "new", limit: "25" }).toString();
  const response = await fetch(url, { headers: { "x-api-key": apiKey } });
  const observedAt = new Date().toISOString();

  if (!response.ok) {
    const body = await response.json().catch(() => null);
    return {
      outcome: "error" as const,
      httpStatus: response.status,
      retryAfter: response.headers.get("Retry-After"),
      observedAt,
      body,
    };
  }

  const payload = (await response.json()) as RedditSearchResponse;
  if (payload.code !== 200 || !payload.data) {
    throw new Error("Unexpected Reddit search response; check the endpoint schema");
  }

  const posts = payload.data.posts ?? [];
  return {
    outcome: posts.length === 0 ? "empty" as const
      : payload.data.pagination?.after ? "partial" as const
      : "complete" as const,
    observedAt,
    source: payload.data.source,
    nextCursor: payload.data.pagination?.after,
    posts: posts.map((post) => ({
      id: post.id,
      title: post.title,
      publishedAt: post.created,
      url: post.permalink,
      sourceUrl: post.source_feed_url,
    })),
  };
}

This sample follows the Reddit search endpoint schema. Its 503 throttle response carries Retry-After; for that specific endpoint, the docs say to wait for the indicated delay before retrying.

Keep empty, partial, and blocked outcomes separate

The API's general error guide gives the usual retry posture, but the individual endpoint docs take precedence. A Reddit 503 throttle is different from a genuine empty result; an MCP tool-level error is different from a transport timeout. Capture the status, retry hint, endpoint, query, and request identifier when present, but never pass credentials into the model's tool result.

A platform API or a general crawler?

These are different scopes, not a reliability ranking. A general crawler gives you flexibility over pages and extraction. A hosted endpoint removes some fetching and parsing work for the platforms it supports, but it does not make every site or field available. Compare the self-hosting trade-off in Web Scraping with Crawl4AI; Crawlora's API catalog lists each supported platform and endpoint schema.

Next steps

Start with the Getting Started guide, inspect the Reddit search schema, and try a request in the Reddit search Playground. For the protocol overview, read Give Your AI Agent Live Web Data with MCP. For client plugin packaging, see Best Web Data Plugins for Codex and Cursor.

Start with one structured endpoint

Connect an MCP client or call the REST API, then inspect the exact source and response fields before building the agent workflow.

Read the quickstartTry Reddit search

Sources

Related reading

• Give Your AI Agent Live Web Data with MCP — hosted MCP basics and a first tool call.
• Best Web Data Plugins for Codex and Cursor in 2026 — installation and auth differences in those clients.
• Web Scraping with Crawl4AI in 2026 — self-hosted arbitrary-URL crawling and its operational trade-offs.