Open
Conversation
Contributor
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
🟡 Acceptable - Solid concept and structure, but fix the criteria.md duplication before merge. "Bad programmers worry about the code. Good programmers worry about data structures."
plugins/onboarding/skills/agent-onboarding/references/criteria.md
Outdated
Show resolved
Hide resolved
plugins/onboarding/skills/agent-readiness/scripts/scan_agent_instructions.sh
Show resolved
Hide resolved
neubig
reviewed
Mar 23, 2026
|
|
||
| --- | ||
|
|
||
| ## Pillar 2 · Feedback Loops |
Contributor
There was a problem hiding this comment.
We will want to replace this later when @xingyaoww is done with our recommended verification stack.
| | 3 | **Multi-model support** | Instructions that work across different AI models/tools, not locked to one vendor | 2+ distinct agent config types from features 1–2 present in same repo | | ||
| | 4 | **Agent skills or capabilities** | Packaged, reusable abilities the agent can invoke | `.claude/skills/`, `.factory/skills/`, `skill.md` files, tool definition files | | ||
| | 5 | **Tool server configuration** | Config for agent tool protocols (lets agents use external tools) | `.mcp.json`, `mcp.config.js`, tool server manifests | | ||
| | 6 | **Agent prompt library** | Pre-built prompts for common tasks in this repo | `.github/prompts/`, `prompts/` directory, prompt template files | |
Contributor
There was a problem hiding this comment.
Maybe not necessary if we have skills?
|
|
||
| | # | Feature | What to look for | Evidence | | ||
| |---|---------|------------------|----------| | ||
| | 1 | **Agent instruction file** | A dedicated file telling agents how to work in this repo — conventions, banned patterns, common commands | `AGENTS.md`, `CLAUDE.md`, `COPILOT.md`, `CONVENTIONS.md` at root | |
Contributor
There was a problem hiding this comment.
AGENTS.md is the standard now, probably just need that.
| | # | Feature | What to look for | Evidence | | ||
| |---|---------|------------------|----------| | ||
| | 1 | **Agent instruction file** | A dedicated file telling agents how to work in this repo — conventions, banned patterns, common commands | `AGENTS.md`, `CLAUDE.md`, `COPILOT.md`, `CONVENTIONS.md` at root | | ||
| | 2 | **AI IDE configuration** | Settings or rules for AI-powered editors/IDEs | `.cursor/rules/`, `.cursorrules`, `.github/copilot-instructions.md`, `.github/instructions/`, `.claude/settings.json` | |
Contributor
There was a problem hiding this comment.
Can probably remove, we don't necessarily need to recommend that people use IDEs.
| |---|---------|------------------|----------| | ||
| | 1 | **Agent instruction file** | A dedicated file telling agents how to work in this repo — conventions, banned patterns, common commands | `AGENTS.md`, `CLAUDE.md`, `COPILOT.md`, `CONVENTIONS.md` at root | | ||
| | 2 | **AI IDE configuration** | Settings or rules for AI-powered editors/IDEs | `.cursor/rules/`, `.cursorrules`, `.github/copilot-instructions.md`, `.github/instructions/`, `.claude/settings.json` | | ||
| | 3 | **Multi-model support** | Instructions that work across different AI models/tools, not locked to one vendor | 2+ distinct agent config types from features 1–2 present in same repo | |
Contributor
There was a problem hiding this comment.
Not necessary, since the purpose of this is mainly to tell that this is ready for openhands.
| | 4 | **Agent skills or capabilities** | Packaged, reusable abilities the agent can invoke | `.claude/skills/`, `.factory/skills/`, `skill.md` files, tool definition files | | ||
| | 5 | **Tool server configuration** | Config for agent tool protocols (lets agents use external tools) | `.mcp.json`, `mcp.config.js`, tool server manifests | | ||
| | 6 | **Agent prompt library** | Pre-built prompts for common tasks in this repo | `.github/prompts/`, `prompts/` directory, prompt template files | | ||
| | 7 | **Component-level agent guidance** | Different parts of the codebase have their own agent instructions | `AGENTS.md` or instruction files in subdirectories (e.g. `frontend/AGENTS.md`, `api/CLAUDE.md`) | |
Contributor
There was a problem hiding this comment.
Suggested change
| | 7 | **Component-level agent guidance** | Different parts of the codebase have their own agent instructions | `AGENTS.md` or instruction files in subdirectories (e.g. `frontend/AGENTS.md`, `api/CLAUDE.md`) | | |
| | 7 | **Component-level agent guidance** | Different parts of the codebase have their own agent instructions | `AGENTS.md` or instruction files in subdirectories (e.g. `frontend/AGENTS.md`, `api/AGENTS.md`) | |
| | 11 | **API documentation** | Reference docs for the project's interfaces | `openapi.yaml`, generated HTML docs, `doc.go` files, Swagger UI, `api-docs/` | | ||
| | 12 | **Inline code documentation** | Doc comments, docstrings — agents read these to understand intent | JSDoc `/** */` blocks, Python docstrings, GoDoc comments, RDoc `#` blocks, Rust `///` | | ||
| | 13 | **Runnable examples** | Working example code the agent can study and imitate | `examples/` directory, `_examples/`, example apps with their own READMEs | | ||
| | 14 | **Changelog** | History of what changed and how entries should be written | `CHANGELOG.md`, `CHANGES.md`, `HISTORY.md`, release notes in GitHub Releases | |
Contributor
There was a problem hiding this comment.
This can also just be tracked in github, so maybe not necessary.
| | 46 | **Label automation** | Automatic PR/issue labeling based on paths or content | `.github/labeler.yml`, label-sync config, auto-label workflows | | ||
| | 47 | **Multi-platform CI** | CI matrix covering multiple OS, arch, or runtime versions | `matrix:` in CI with `os: [ubuntu, macos, windows]` or multiple language versions | | ||
| | 48 | **Deployment automation** | Automated deployment pipeline | Deploy workflow triggered on merge/tag, staging + production environments in CI | | ||
| | 49 | **Automated code review checks** | Bot-driven review checks beyond CI | Danger.js config, review bot config, required review assignments, CODEOWNERS + required reviews | |
Contributor
There was a problem hiding this comment.
This should recommend the openhands review?
| | 57 | **Security scanning** | Automated vulnerability scanning in CI | `.github/workflows/codeql.yml`, Snyk config, `gosec` in CI, Trivy, Dependabot security alerts | | ||
| | 58 | **Git attributes** | Line endings, diff drivers, LFS, linguist overrides | `.gitattributes` with `text=auto`, `linguist-generated`, LFS tracking patterns | | ||
| | 59 | **Contributor agreement** | DCO sign-off or CLA process | DCO bot config, `Signed-off-by` requirement in contributing guide, CLA-assistant config | | ||
| | 60 | **Governance model** | Documented maintainer roles, decision-making process | `GOVERNANCE.md`, `MAINTAINERS.md`, governance section in docs, team/role descriptions | |
Contributor
There was a problem hiding this comment.
This seems duplicated with codeowners
|
|
||
| ## Five Pillars | ||
|
|
||
| | Pillar | Question | Features | |
Contributor
There was a problem hiding this comment.
If we change anything based on my above comments, change here too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a plugin to define a set of skills intended for repository onboarding. It includes:
readiness-report-- which evaluates the repository against a large number of features that all indicate the repository is ready for agentic software developmentagent-onboarding-- an agent specialized in reading the readiness report and identifying high-impact fixesagents-md-- contains advice for creating reasonableAGENTS.mdfiles from scratchThe readiness report was a stand-alone skill, the other two are brand new. All three have been evaluated and developed by analyzing 125 open-source repositories across a variety of languages, domains, sizes, and AI readiness levels.
Design Process
The previous readiness report was focused on web-based applications and had some very specific tools called out in the rubric. One challenge in updating the rubric is figuring out what people are actually doing to get their repositories AI-ready.
To that end, I scraped a lot of open-source repositories, some of which had no sign of AI readiness (not event a
agents.md) and others which clearly had a huge amount of effort put in to AI readiness. Manual inspection revealed four main categories of AI readiness:agents.mdfiles,.cursorconfigs, skills, etc.OpenHands scanned all the repos and produce >1.2k features that fit into these categories (and an "Other" category) as evidence (or lack thereof) that a repo was AI ready. These features were analyzed using hierarchical clustering algorithms to collapse duplicates and identify broad features. Manual tweaking produced the existing rubric from these relevant features.
To ensure these features actually indicate AI readiness, we evaluated on a handful of repos and showed that the rubric scores cleanly separated those that were AI ready from those that weren't.