plans/web-fetch-local-agent.md
Generated by swarm planning session on 2026-02-25
Add a new web_fetch tool to the local agent that fetches and reads website content when users share URLs for reference. Unlike the existing Pro-only web_crawl tool (which uses Firecrawl for visual cloning with screenshots), web_fetch performs a direct local HTTP fetch from the user's machine, making it available to all users (free + Pro) at zero infrastructure cost.
When users paste a URL into the Dyad chat (e.g., "Help me integrate this API: https://docs.stripe.com/api"), the agent cannot access the content behind that URL. Users must manually copy-paste page content, breaking their flow. This is especially painful for developers building with APIs, following tutorials, or referencing documentation — the most common use cases for Dyad's target audience. The existing web_crawl tool only activates for "clone/copy/replicate" intent and requires Dyad Pro, leaving a gap for the broader "read this page for context" use case.
web_fetch tool that fetches a URL and returns content as markdownisDyadPro gateturndown + @mozilla/readability for content extractionhttp: and https: only; block file:, ftp:, data:, javascript:, blob: schemes)"ask" defaultMAX_TEXT_SNIPPET_LENGTH)AbortController<dyad-web-fetch> tagFetch page content: "https://docs.stripe.com/api"<dyad-web-fetch> card appears in the chat showing the URL being fetched with a loading stateDyadStateIndicator pattern)application/json badge + URL, expandable content as code blocktext/plain badge + URL, content displayed as-is<dyad-output type="warning">)Fetch page content: "https://..." (action-focused, not implementation-detail-focused)Link from lucide-react (differentiated from Globe for web_search and ScanQrCode for web_crawl)purple to differentiate from the blue used by web_search and web_crawlDyadCard pattern)New tool following the established ToolDefinition<T> pattern. Performs a direct HTTP fetch from the Electron main process using Node.js fetch(), processes HTML through @mozilla/readability for content extraction, then converts to markdown via turndown. Returns the markdown string as the tool result. No changes to existing tools or the agent handler.
Dependency pipeline: fetch(url) → linkedom.parseHTML(html) → new Readability(doc).parse() → new TurndownService().turndown(article.content) → truncateText(markdown)
linkedom is required because both @mozilla/readability and turndown need a DOM document, and Electron's main process doesn't have one. linkedom is lightweight (~50KB) and much faster than JSDOM.
src/pro/main/ipc/handlers/local_agent/tools/web_fetch.ts — Tool implementationsrc/pro/main/ipc/handlers/local_agent/tool_definitions.ts — Import and register webFetchTool in TOOL_DEFINITIONS arraypackage.json — Add turndown, @types/turndown, linkedom, @mozilla/readability (or defuddle)DyadWebFetch component for rendering the <dyad-web-fetch> XML tag in chatweb_crawl.ts, engine_fetch.ts, local_agent_handler.ts, types.tsNone. The tool returns a string result via the existing ToolResult type. No schema or storage changes.
No external API changes. Internally:
web_fetch added to TOOL_DEFINITIONS array<dyad-web-fetch> for rendererThe tool description guides LLM behavior and is the single biggest factor in feature success:
Fetch and read content from a URL. Works with web pages (returns cleaned markdown) and API endpoints (returns JSON).
### When to Use
Use this tool when the user shares a URL and wants you to reference, understand, or use information from that page. Examples:
- User shares API documentation and asks you to integrate it
- User shares a tutorial or blog post and wants you to follow it
- User shares a web page and asks about its content
- User shares an API endpoint URL and wants you to understand the response
### When NOT to Use
- User wants to CLONE / COPY / REPLICATE / RECREATE a website's visual design — use web_crawl instead
- User mentions a URL in passing without wanting you to read it
- You need to search the web for information (no specific URL) — use web_search instead
### Limitations
- Cannot render JavaScript — some dynamic/SPA pages may return limited content
- Content is truncated to ~16,000 characters for very long pages
- PDF and image files are not supported
// web_fetch.ts - Core structure
const webFetchSchema = z.object({
url: z.string().describe("URL to fetch"),
});
// URL validation: only http: and https: schemes
// No private IP blocking (user decision: allow with consent)
// Timeout: 10-15 seconds via AbortController
// User-Agent: set a reasonable browser-like string
// Content-Type handling:
// text/html → Readability extraction → Turndown markdown → truncate
// application/json → return as ```json code block → truncate
// text/plain, text/markdown → return as-is → truncate
// application/pdf, image/* → return "not supported" message
// other → attempt text extraction, fall back to "not supported"
// Truncation: reuse MAX_TEXT_SNIPPET_LENGTH (16,000 chars) pattern
export const webFetchTool: ToolDefinition<z.infer<typeof webFetchSchema>> = {
name: "web_fetch",
description: DESCRIPTION,
inputSchema: webFetchSchema,
defaultConsent: "ask",
// No isEnabled gate — available to all users
getConsentPreview: (args) => `Fetch page content: "${args.url}"`,
buildXml: (args, isComplete) => {
if (!args.url) return undefined;
let xml = `<dyad-web-fetch url="${escapeXmlContent(args.url)}">`;
if (isComplete) xml += "</dyad-web-fetch>";
return xml;
},
execute: async (args, ctx) => {
// 1. Validate URL scheme (http/https only)
// 2. Fetch with timeout (AbortController, 15s)
// 3. Check Content-Type header
// 4. For HTML: parse with Readability, convert with Turndown
// 5. For JSON: wrap in code block
// 6. For text: return as-is
// 7. For unsupported: return clear message
// 8. Truncate to MAX_TEXT_SNIPPET_LENGTH
// 9. Return markdown string as tool result
},
};
turndown, @types/turndown, linkedom, @mozilla/readability (evaluate defuddle as alternative)src/pro/main/ipc/handlers/local_agent/tools/web_fetch.ts with:
webFetchTool in tool_definitions.ts TOOL_DEFINITIONS arrayDyadWebFetch component to render <dyad-web-fetch> XML tagsbuildAgentToolSet output (no isEnabled gate)file://, ftp://, data: are rejected; http:// and https:// are acceptedwebFetchTool is included in tool set for both Pro and non-Pro contexts| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| JS-rendered SPAs return minimal content | Medium | Medium | Clear tool description noting limitation; LLM can explain to user; Pro users can use web_crawl |
| LLM confuses web_fetch with web_crawl or web_search | Low | Medium | Precise, mutually-exclusive tool descriptions with explicit when/when-not guidance |
| Large HTML pages block Electron main process during conversion | Low | Medium | Truncate raw HTML before processing; move to worker thread in follow-up if needed |
| Content quality varies across sites (paywalls, anti-bot) | Medium | Low | Return clear error messages; user can fall back to manual copy-paste |
| New dependencies (turndown, readability) introduce maintenance burden | Low | Low | Both are mature, stable libraries with large install bases |
| "Accept always" consent enables unbounded fetch loops | Low | Medium | Monitor; consider per-turn fetch limit in follow-up if abuse is observed |
defuddle (by Jina AI) as a potential alternative to @mozilla/readability. Defuddle may offer better extraction for modern web pages. Decision can be made during implementation based on testing.linkedom is included as the DOM implementation since both @mozilla/readability and turndown require a DOM document and Electron's main process doesn't provide one. linkedom is lightweight (~50KB) and much faster than JSDOM.web_fetch multiple times. Each triggers a separate consent dialog. If this proves disruptive, consider batch consent UI in a follow-up.| Decision | Reasoning |
|---|---|
New tool (web_fetch) rather than extending web_crawl | Use cases are fundamentally different (read vs. clone). Separate tools = cleaner code, clearer LLM descriptions, independent consent settings. All 3 roles agreed independently. |
| Available to all users (free + Pro) | Local fetch has zero infrastructure cost. Differentiates free tier. Natural upsell to Pro for enhanced crawl+screenshot. |
| LLM-triggered, not auto-detected | Consistent with existing tool architecture. Auto-detection would require new handler-layer logic and might fetch URLs users didn't intend. |
| Allow private/localhost IPs | Dyad runs locally; SSRF is a server-side threat model. Fetching localhost:3000 or internal docs is a legitimate use case. Consent dialog provides sufficient protection. |
| Include @mozilla/readability in v1 | Dramatically better content extraction (strips nav, footer, ads). Small marginal cost (one extra dependency). All roles agreed. |
| Handle Content-Type gracefully | ~15 lines of code prevents confusing failures for JSON, text, PDF URLs. Better UX for minimal effort. |
| Consent default: "ask" | Consistent with web_crawl and web_search. Network requests to arbitrary external URLs warrant explicit approval. |
| Truncation at 16K characters | Matches existing MAX_TEXT_SNIPPET_LENGTH. Prevents context window overflow while providing substantial content. |
Tool name: web_fetch | Consistent with web_search, web_crawl naming convention. Clear, concise, action-oriented. |
Generated by dyad:swarm-to-plan