GPUWorks

When your GPUs are spread across a rack in the office, rented cloud instances, and a Mac on a desk, GPUWorks pools them into one cluster that serves open models behind a single OpenAI-compatible endpoint. Pick a model from the built-in catalog of launch specs for vLLM and SGLang across NVIDIA, AMD, Huawei Ascend, T-Head PPU, Hygon, and Apple Silicon hardware, and the deploy page estimates the memory needed for weights, KV cache, and engine overhead, shows which GPUs the scheduler chose, and explains why it rejected the others. Several models can share one card through memory quotas or take turns with sleep and wake, while replicas, multi-node parallelism, and prefill and decode disaggregation scale a deployment out. The gateway handles chat, embeddings, rerank, audio, image, and realtime requests with API keys, per-key token usage, and failover routing. A monitor center charts time to first token, throughput, concurrency, and KV cache hit rates, usage analytics break tokens down by key and model, and a built-in benchmark sweeps concurrency to find your latency limit. GPU health checks and an AI assistant help diagnose failing cards, and a Playground lets you try any deployment in the browser. Nodes connect outbound, so they need no open ports, and models can be imported offline for air-gapped sites. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

GPUWorks
GPUWorks
GPUWorks
GPUWorks
GPUWorks
GPUWorks
GPUWorks
GPUWorks
GPUWorks
GPUWorks

Benefits

  • Pool every GPU you can reach
  • Join bare-metal servers, rented RunPod or Vast.ai instances, Kubernetes clusters, and Apple Silicon Macs into one cluster, mixing NVIDIA, AMD, Ascend, PPU, and Hygon accelerators under a single scheduler.
  • See memory needs before deploying
  • The configuration evaluation estimates GPU memory for weights, KV cache, and engine overhead, shows the cards it picked, and explains each rejected placement, so you can fix a problem before a replica starts.
  • Keep every application on one API
  • Applications call one OpenAI-compatible endpoint under /v1 with API keys and per-key token usage, while routing spreads traffic across replicas by least request, prefix affinity, or failover.
  • Trace slow requests and sick GPUs
  • Per-request timelines, searchable logs, GPU and engine metrics, card-level health checks, and an AI assistant help you find why a request was slow or a GPU keeps failing.

Features

  • Heterogeneous Accelerator Scheduler
  • Matches each deployment to hardware its launch spec supports across NVIDIA, AMD, Ascend, PPU, Hygon, and Apple Silicon, with dGPU memory quotas and sleep and wake to share cards.
  • Model Catalog and Templates
  • Built-in launch specs pin Hugging Face and ModelScope revisions, images, engines, and flags per GPU type, covering chat, vision, embedding, rerank, speech, image, and video models.
  • OpenAI-Compatible Gateway
  • Serves chat, completions, embeddings, rerank, audio, images, video, and realtime under /v1, passes through /v1/messages and /v1/responses, and issues API keys with per-key token usage.
  • Monitoring, Traces, and Benchmarks
  • GPU and engine metrics, searchable logs, per-request trace timelines with OTLP export, usage analytics, and a benchmark that sweeps concurrency to find the SLA limit.
  • Scale-Out Deployments
  • Run replicas across nodes, multi-node tensor and pipeline parallelism, and prefill and decode disaggregation (vLLM with NIXL, vLLM-Ascend with Mooncake) within one host or over RDMA.
  • AI Assistant and MCP
  • Ask questions about cluster health in the web console, or connect Claude Code or any MCP client to the read-only MCP endpoint, which an administrator enables first.

Apps Similar to GPUWorks