🖥️ Computer-Use Agents: An Overview of Terminal-Based and GUI-Based Approaches

A curated list of papers, tools, and benchmarks on computer-use agents: systems that let LLMs operate computers through terminals or graphical interfaces.

We organize this collection into two families:

• Terminal / CLI-Based Agents interact with computers through text interfaces: shell commands, file I/O, and code execution. This category spans five stages of development: code completion, chat assistants, autonomous coding agents, general-purpose sandbox agents, and personal AI assistants.
• GUI-Based Agents perceive and act on graphical interfaces through mouse clicks, keyboard input, and screen observation. We cover web agents, desktop/OS agents, and mobile agents, each with distinct perception strategies and challenges.

Note: This is a preliminary collection and is by no means exhaustive. Many important papers and tools are likely missing. If you know of work that should be included, please open a PR or issue. Contributions of all kinds are welcome!

We also wrote a short report discussing the development and architecture of these systems: report (PDF).

Contributions welcome! Please open a PR.

---

Table of Contents

• Terminal / CLI-Based Agents
  • Code Completion
  • Chat-Based Coding Assistants
  • Autonomous Coding Agents
  • General Agents
  • Personal AI Assistants
• GUI-Based Agents
  • Web Agents
  • Desktop / OS Agents
  • Mobile Agents
• Benchmarks
• Foundational Work

---

Terminal / CLI-Based Agents

Code Completion

Name	Year	Type	Links
Codex	2021	Model	Paper
GitHub Copilot	2021	Product	Website
Codeium	2022	Product	Website

Chat-Based Coding Assistants

Name	Year	Type	Links
ChatGPT	2022	Product	Website
Cursor	2024	Product	Website
Aider	2024	Open Source	GitHub

Autonomous Coding Agents

Research Frameworks

Name	Year	Type	Links
SWE-Agent	2024	Paper + Code	Paper · GitHub
OpenHands (OpenDevin)	2024	Paper + Code	Paper · GitHub
AutoCodeRover	2024	Paper + Code	Paper · GitHub
Agentless	2024	Paper + Code	Paper · GitHub
CodeAct	2024	Paper + Code	Paper

Product Agents

Name	Year	Type	Links
Devin	2024	Product	Website
Claude Code	2025	Product	Docs
Codex CLI	2025	Product	GitHub
Gemini CLI	2025	Product	GitHub

IDE-Integrated

Name	Year	Type	Links
Cursor Agent Mode	2025	Product	Website
Windsurf (Codeium)	2025	Product	Website

General Agents

Name	Year	Type	Links
Open Interpreter	2023	Open Source	Website · GitHub
Manus	2025	Product	Website
Agent SDK (Anthropic)	2025	Product	Docs
LLM-in-Sandbox	2026	Paper + Code	Paper · GitHub

Personal AI Assistants

Name	Year	Type	Links
OpenClaw	2025	Open Source	GitHub · Website

---

GUI-Based Agents

Web Agents

Name	Year	Type	Links
Mind2Web	2023	Paper + Data	Paper · GitHub
SeeAct	2024	Paper + Code	Paper · GitHub
WebVoyager	2024	Paper + Code	Paper
WebArena	2023	Benchmark + Paper	Paper · GitHub
Browser Use	2025	Open Source	GitHub
Operator (OpenAI)	2025	Product	Blog
Project Mariner (Google)	2025	Product	Blog
BrowserTrace	2026	Open Source Tool	GitHub
ClawBench	2026	Benchmark + Paper	Paper · GitHub · Live

Desktop / OS Agents

Name	Year	Type	Links
Computer Use (Anthropic)	2024	API	Blog
CogAgent	2024	Paper + Code	Paper · GitHub
ScreenAgent	2024	Paper	Paper
UFO (Microsoft)	2024	Paper + Code	Paper · GitHub
Set-of-Mark (SoM)	2023	Paper	Paper
OmniParser	2024	Paper + Code	Paper · GitHub
OS-Copilot	2024	Paper + Code	Paper
UI-TARS (ByteDance)	2025	Paper	Paper
UI-TARS-2	2025	Paper	Paper
axcli	2026	Open Source	GitHub

Mobile Agents

Name	Year	Type	Links
AppAgent	2024	Paper + Code	Paper
Mobile-Agent-v3	2025	Paper + Code	Paper

---

Benchmarks

Name	Domain	Year	Links
GAIA	General AI assistant	2023	Paper
SWE-bench	Coding (Python)	2024	Paper · Website
WebArena	Web browsing	2023	Paper · Website
VisualWebArena	Visual web tasks	2024	Paper
OSWorld	Desktop OS	2024	Paper · GitHub
AndroidWorld	Mobile (Android)	2024	Paper
TheAgentCompany	Multi-tool (company sim)	2024	Paper · Website
Terminal-Bench	CLI tasks (diverse)	2026	Paper · Website

---

Foundational Work

Name	Year	Links
ReAct: Synergizing Reasoning and Acting in Language Models	2023	Paper
Toolformer: Language Models Can Teach Themselves to Use Tools	2023	Paper
Codex: Evaluating Large Language Models Trained on Code	2021	Paper
GPT-4 Technical Report	2023	Paper
Plan-and-Solve Prompting	2023	Paper
AdaPlanner: Adaptive Planning from Feedback	2023	Paper
Tree of Thoughts	2023	Paper
LLM Debate: Improving Factuality through Multiagent Debate	2024	Paper

---

Contributing

PRs welcome! When adding an entry, please include:

• Name, year, and type (Paper / Product / Open Source)
• Links to paper (arXiv preferred), code (GitHub), and website

License

MIT

Citation

If you find this list helpful, please give it a ⭐!

@misc{cheng2026awesomecomputeruseagents,
  title={Awesome Computer-Use Agents},
  author={Cheng, Daixuan},
  year={2026},
  howpublished={\url{https://github.com/cdxeve/awesome-computer-use-agents}},
}