Update robots.txt to enable agent discovery of documentation and examples

[guyoron1]

Note

This issue was automatically generated from meeting notes by the scribe agent.
Please review, edit, and add any missing context before prioritizing.

Problem

The current robots.txt configuration is too restrictive for agent use cases. Agents cannot properly discover and read documentation and example content in the repository, which limits their effectiveness when they need to reference documentation for context during code generation, review, or triage operations. The configuration should be updated to allow agents to crawl and read documentation paths.

Options considered

• Permissive robots.txt for documentation paths: Update robots.txt to explicitly allow crawling of documentation directories, example paths, and sales documentation.
• Selective path allowlist: Allow specific paths (docs, examples, guides) while maintaining restrictions on other content to limit scope.

Acceptance criteria

• robots.txt updated to allow agent access to documentation paths
• Agents can discover and read documentation and example content
• Configuration allows full reads of docs and sales/example materials
• No unintended exposure of sensitive or internal-only paths

Related

• Source: Meeting notes

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 8:44 AM UTC · Completed 8:47 AM UTC
Commit: c887fc4 · View workflow run →

[guyoron1]

/fs-triage

[fullsend-ai-triage]

Triage Summary

Severity: Medium · Category: Feature

Problem

The robots.txt at cloudflare_site/public/robots.txt blocks all training/agent crawlers site-wide with blanket Disallow: / rules. This prevents agents from discovering documentation hosted on fullsend.sh during autonomous operations.

Current State

Blocked user-agents: ClaudeBot, GPTBot, CCBot, Google-Extended, Bytespider, meta-externalagent, Applebot-Extended — all have Disallow: /.

Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot) and the default User-agent: * are allowed.

Recommended Fix

Add path-specific Allow directives before the Disallow: / line for each blocked user-agent. Use the llms.txt file to derive the documentation paths to allow (at minimum /docs/). The selective allowlist approach is recommended to meet the acceptance criterion of not exposing sensitive paths.

Example pattern:

User-agent: ClaudeBot
Allow: /docs/
Disallow: /

Repeat for each blocked agent.

Proposed Test

Parse the updated robots.txt and verify:
1. Previously-blocked agents are allowed on /docs/ and subpaths
2. Those agents remain disallowed on non-documentation paths
3. Search bots remain allowed on all paths
4. Default User-agent: * continues to allow all other bots
5. No sensitive or internal-only paths are newly exposed

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

Previous run

Triage Summary

Severity: Medium · Category: Feature

Problem

The robots.txt at cloudflare_site/public/robots.txt blocks all agent/training crawlers site-wide with blanket Disallow: / rules. This prevents agents from discovering documentation hosted on fullsend.sh.

Current State

The following user-agents are fully blocked:

• ClaudeBot, GPTBot, CCBot, Google-Extended, Bytespider, meta-externalagent, Applebot-Extended

Meanwhile, search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot) and all other bots are allowed.

Recommended Fix

For each blocked user-agent, add path-specific Allow directives before the Disallow: / line. The documentation paths to allow can be derived from the existing llms.txt file:

User-agent: ClaudeBot
Allow: /docs/
Disallow: /

Repeat for each blocked agent. Consider which of the two options from the issue to follow — permissive for all doc paths or a selective allowlist.

Proposed Test

Parse the updated robots.txt and verify:
1. Previously-blocked agents (ClaudeBot, GPTBot, etc.) are allowed on /docs/ and subpaths
2. Those agents remain disallowed on non-documentation paths
3. Search bots remain allowed on all paths
4. Default User-agent: * continues to allow all other bots

---

Labels: This is a feature request to update documentation-related configuration, matching both the enhancement and documentation labels.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 8:53 AM UTC · Completed 8:55 AM UTC
Commit: c887fc4 · View workflow run →

[guyoron1]

/fs-code

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 9:28 AM UTC · Completed 9:33 AM UTC
Commit: c887fc4 · View workflow run →