Potato Annotation Tool

Collect trustworthy human labels and judgments for your datasets, models, and AI agents from one self-hosted annotation server, whether the item is a sentence, a bird call, a microscope image, or a long agent trace. A study is one YAML file plus your data, written by hand, designed in the browser, or generated by a coding agent through the Potato skill or an MCP server. For agent and LLM evaluation, Potato reads traces from OpenAI, Claude, LangChain, LangFuse, SWE-bench, OpenTelemetry, CrewAI, AutoGen, and LangGraph, then lets reviewers rate whole trajectories, single steps, text spans, or two agents side by side. Web-agent runs replay with click and scroll overlays, LLM judges are calibrated against human labels, and pytest evaluations fail CI when scores regress. Models from OpenAI, Anthropic, Gemini, Ollama, vLLM, and OpenRouter pre-label items for people to correct, active learning puts the most uncertain items first, models train on your annotations from the admin page, and a labeling engine certifies machine labels against a blind human sample. Solo Mode lets one person hand labeling to an LLM as agreement grows. Attention checks, gold items, adjudication, Krippendorff's alpha, and annotator timing keep quality visible. Recruit through Prolific, CloudResearch Connect, SONA, or invite links, or code interview transcripts with a living codebook, in-vivo codes, and memos. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. GPL-3.0-or-later licensed.

Potato Annotation Tool
Potato Annotation Tool
Potato Annotation Tool
Potato Annotation Tool
Potato Annotation Tool
Potato Annotation Tool
Potato Annotation Tool
Potato Annotation Tool
Potato Annotation Tool
Potato Annotation Tool
Potato Annotation Tool

Benefits

  • Whole study lives in one file
  • A task is a settings file plus your data, so it can be shared, versioned in git, reused, and published next to a paper for others to rerun.
  • Evaluate agents beyond pass or fail
  • Reviewers score a whole trajectory, each step, or a span of text inside a step, and LLM judges are calibrated against their labels before gating CI.
  • Label less with machine help
  • LLMs pre-annotate items for people to correct, and the labeling engine tests machine labels against a blind human sample, accepts those that pass your error limits, and sends the rest to people.
  • Private servers or crowd recruiting
  • Keep sensitive data on your own server, or open the study to Prolific, CloudResearch Connect, and SONA participants, with attention checks, gold items, and agreement metrics watching quality.

Features

  • Agent Trace Evaluation
  • Import traces from OpenAI, Anthropic, LangChain, LangFuse, WebArena, SWE-bench, OpenTelemetry, CrewAI, AutoGen, LangGraph, MCP, and Claude Code, then rate trajectories, steps, spans, or two agents side by side.
  • Human-AI Labeling Loop
  • LLM label suggestions via OpenAI, Anthropic, Gemini, Ollama, vLLM, and OpenRouter, plus active learning (BADGE, BALD, uncertainty), in-app model training, and Solo Mode that hands labeling to an LLM.
  • Multimodal Annotation Schemes
  • Classification, span, pairwise, and best-worst scales for text, plus waveforms, spectrograms, video, bounding boxes, polygons, PDFs, point clouds, CT volumes, microscopy, and genome or protein viewers.
  • Quality Control and Agreement
  • Attention checks, gold items, adjudication, Krippendorff's alpha, Cohen's kappa, and live item response theory scoring, all surfaced in an admin dashboard with annotator timing and export.
  • Qualitative Coding Mode
  • Turn on QDA mode for a living markdown codebook with version history, in-vivo codes, memos, cases, and full-text search across the corpus, in the style of NVivo and ATLAS.ti.
  • One-Command Deployment
  • Run potato deploy up for Hetzner, DigitalOcean, Vultr, Linode, or AWS servers with HTTPS, or use the Docker image behind nginx or Caddy, with S3 or Hugging Face backups.

Apps Similar to Potato Annotation Tool