Potato Annotation Tool screenshot thumbnail

Potato Annotation Tool

Collect trustworthy human labels and judgments for your datasets, models, and AI agents from one self-hosted annotation server, whether the item is a sentence, a bird call, a microscope image, or a long agent trace. A study is one YAML file plus your data, written by hand, designed in the browser, or generated by a coding agent through the Potato skill or an MCP server. For agent and LLM evaluation, Potato reads traces from OpenAI, Claude, LangChain, LangFuse, SWE-bench, OpenTelemetry, CrewAI, AutoGen, and LangGraph, then lets reviewers rate whole trajectories, single steps, text spans, or two agents side by side. Web-agent runs replay with click and scroll overlays, LLM judges are calibrated against human labels, and pytest evaluations fail CI when scores regress. Models from OpenAI, Anthropic, Gemini, Ollama, vLLM, and OpenRouter pre-label items for people to correct, active learning puts the most uncertain items first, models train on your annotations from the admin page, and a labeling engine certifies machine labels against a blind human sample. Solo Mode lets one person hand labeling to an LLM as agreement grows. Attention checks, gold items, adjudication, Krippendorff's alpha, and annotator timing keep quality visible. Recruit through Prolific, CloudResearch Connect, SONA, or invite links, or code interview transcripts with a living codebook, in-vivo codes, and memos. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. GPL-3.0-or-later licensed.

Deploy