fix(seo): fetch robots.txt with broken-link crawler user agent

[tongchen92]

Feedback source

• NotFair feedback ID: 257
• Misrouted MCP tracker: nowork-studio/notfair-mcp#673
• Affected source: seo/broken-link-checker/scripts/checker.py

Verified defect

Current main calls urllib.robotparser.RobotFileParser.read() after set_url(). That standard-library helper fetches robots.txt using Python's default urllib user agent, while the crawler itself identifies as NotFairBrokenLinkChecker/1.0. Cloudflare-fronted sites that block Python's default agent can therefore block the robots fetch even though the crawler has a compatible user agent.

Expected behavior

Fetch and parse robots.txt with the crawler's explicit user agent and existing bounded timeout. If it cannot be fetched or decoded, preserve the current fail-open behavior: emit the warning and continue with rp = None.

Scope

• Replace only the robots fetch path; do not broaden crawling, request policy, or link-check behavior.
• Keep rp.can_fetch("NotFairBrokenLinkChecker/1.0", current_url) semantics intact.

Acceptance checks

1. A focused unit test proves the robots request carries User-Agent: NotFairBrokenLinkChecker/1.0.
2. A test proves a robots fetch failure remains fail-open and the crawl proceeds.
3. Existing broken-link checker tests pass.
4. No credentials, account state, or external production mutations are involved.