
openeye
io.github.aw3-technology/openeye
Vision for agents: run YOLO, depth, segmentation and detection models locally or via hosted API.
Documentation
OpenEye
Open-source eyes for the agent era.
Website · Docs · Models · Blog · PyPI
OpenEye is a CLI-first perception engine that turns raw video into structured world models for robots and autonomous agents. Pull, run, and serve computer-vision models from your terminal — like Ollama, but for vision AI.
Features
- Object Detection — YOLOv8, YOLO26 (currently ships YOLO11n weights), Grounding DINO, RF-DETR, and custom adapters
- Depth Estimation — Monocular depth via Depth Anything V2
- Segmentation — SAM2 segment anything
- Scene Understanding — Spatial relationships, scene graphs, natural language queries
- VLM Reasoning — Vision-language model integration (Nebius, OpenRouter)
- Agentic Loop — Continuous perception → reasoning → planning pipeline
- Safety Monitoring — Zone-based awareness with real-time human detection and robot halt
- CLI-Native — Every capability accessible from the terminal
- REST + WebSocket API — Serve models over HTTP with a single command
- Fleet Management — Register, deploy, and monitor edge devices at scale
- MLOps — Model registry, versioning, A/B testing, batch inference
- Governance — Policy enforcement, audit logging, safety rule management
- Hosted API —
https://openeye.sh/v1for object detection without running anything locally (pay per call in credits) - Web Dashboard — openeye.sh/dashboard: API keys, credits, history, fleet management, and — connected to your own
openeye serve— inference, live streams, and more
Quick Install
pipx install openeye-sh
With YOLO support:
pipx install "openeye-sh[yolo]"
Quick Start
# Pull a model
openeye pull yolov8
# Run inference on an image
openeye run yolov8 photo.jpg
# Start the server with REST API, WebSocket, and browser dashboard
openeye serve yolov8
# Live camera feed with real-time detections
openeye watch --models yolov8
# Live camera feed with safety monitoring
openeye watch --models yolov8 --safety
Give Your AI Agent Eyes (MCP)
OpenEye ships an MCP server, so Claude, Cursor, or any MCP client can pull models and run detection, depth, and segmentation on your machine:
pipx install "openeye-sh[mcp]"
claude mcp add openeye -- openeye-mcp
Then ask: "Run yolov8 on ~/Pictures/desk.jpg and tell me what's on my desk." Setup for Claude Desktop and other clients: docs/claude-skill.
Hosted API Quick Start
No GPU or local install needed for detection:
- Sign in at openeye.sh (email/password or 1Profile).
- Create a key in Dashboard → Settings → API Keys (keys start with
oe_and are shown once). - Buy credits in Dashboard → Settings → Credits.
- Call the API:
curl -X POST https://openeye.sh/v1/detect \
-H "Authorization: Bearer oe_your_key_here" \
-F "file=@photo.jpg"
# or with the CLI
export OPENEYE_API_KEY=oe_your_key_here
openeye api detect photo.jpg
openeye api usage
| Endpoint | Status | Credits |
|---|---|---|
POST /v1/detect | Live (YOLOv8n, JPEG/PNG under 4 MB) | 1 |
POST /v1/describe | Returns 503 until the VLM is enabled on the hosted API | 3 |
POST /v1/depth | Returns 501 — self-hosted only (openeye serve depth-anything) | 2 |
GET /v1/models, GET /v1/usage | Live | free |
The hosted API has no WebSocket streaming; for streaming, run openeye serve. See the docs for details.
Use the Dashboard with a Local Server
openeye serve allows requests from https://openeye.sh by default, so the hosted dashboard can drive a server on your machine:
openeye serve yolov8 # http://localhost:8000
Open openeye.sh/dashboard, click Server on the Overview page (the "OpenEye Server URL" dialog), and enter http://localhost:8000. Inference, Live, Scene Graph, Metrics, Benchmark, Config, Governance, MLOps, and the live agent loop then run against your server. If your browser blocks the connection from an HTTPS page to localhost, serve OpenEye over HTTPS instead.
Interactive CLI Demo
An interactive walkthrough that runs real OpenEye commands against sample images. Great for evaluating the tool, onboarding, or live presentations.
Prerequisites
- Python 3.12+
- jq (optional, for JSON pipeline steps —
brew install jq) - curl (for the API server step)
Setup
# Install the CLI with YOLO and camera support
pipx install -e "./cli[yolo,camera]"
# Pull the YOLOv8 model (6 MB)
openeye pull yolov8
Run the Demo
./demo.sh
demo.sh is a bash script: run it on macOS, Linux, WSL, or Git Bash on Windows. It uses openeye from your PATH (pipx), falling back to ./.venv or cli/.venv. In plain PowerShell or cmd, run the same steps yourself:
openeye list
openeye pull yolov8
openeye run yolov8 src/assets/demo/scene-warehouse.jpg --pretty
openeye run yolov8 src/assets/demo/scene-warehouse.jpg --visualize
openeye bench yolov8 --warmup 2 --runs 5
openeye serve yolov8 --port 8111
The demo is interactive — press Enter to advance between steps. Each step shows the command being run and its live output.
What the Demo Covers
| Step | Command | What it demonstrates |
|---|---|---|
| 1 | openeye list | Browse the model registry (7 vision models) |
| 2 | openeye pull yolov8 | Download a model — like ollama pull |
| 3 | openeye run yolov8 scene-warehouse.jpg --pretty | Object detection with structured JSON output |
| 4 | openeye run yolov8 scene-kitchen.jpg --pretty | Detection on a different scene (apple, cup, donut) |
| 5 | openeye run ... --visualize | Save an annotated image with bounding boxes |
| 6 | openeye run ... | jq | Unix pipeline composability — pipe JSON to jq |
| 7 | openeye bench yolov8 | Benchmark inference speed (latency, FPS, P95) |
| 8 | openeye serve yolov8 | Launch a REST API server and query it with curl |
| 9 | Batch loop | Process all 4 demo images and summarize results |
Demo Images
Four sample images are included in src/assets/demo/:
| Image | Scene | Typical detections |
|---|---|---|
scene-warehouse.jpg | Warehouse with people and vehicles | person, truck, tv |
scene-kitchen.jpg | Kitchen countertop | apple, cup, donut, bottle |
scene-workshop.jpg | Workshop environment | person, dining table, remote, knife |
safety-workspace.jpg | Safety-critical workspace | (used for safety guardian demos) |
Output
Annotated images are saved to demo_output/. The API server step temporarily runs on port 8111 and shuts down automatically.
Beyond the Demo
Commands not covered in the scripted demo that you can try interactively:
# Live webcam detection with terminal overlay
openeye watch --models yolov8
# Safety Guardian — zone-based human proximity alerts
openeye watch --models yolov8 --safety --danger-m 0.5 --caution-m 1.5
# Unitree G1 robot safety demo (dry-run, no robot needed)
openeye g1-demo --control-mode dry_run --demo
# Agentic perception loop — perceive, reason, act
openeye agent run
# Edge device fleet management
openeye fleet ls
# Open-vocabulary detection with text prompts (requires grounding-dino)
openeye pull grounding-dino
openeye run grounding-dino photo.jpg --prompt "person. hard hat. forklift."
Project Structure
openeye.sh/
├── cli/ # openeye-sh CLI & inference server (Python)
├── backend/ # Perception engine runtime (Python)
├── src/ # React web frontend (TypeScript)
├── api/ # Vercel functions (auth, data, credits, /v1 API)
├── db/ # Postgres schema
└── docs/ # MkDocs documentation site
| Package | Description | Language |
|---|---|---|
cli/ | CLI tool & FastAPI inference server (port 8000), MCP server (openeye-mcp) | Python |
backend/ | Perception pipeline, self-hosted fleet control plane (port 8001), governance engine | Python |
src/ | Website and dashboard at openeye.sh | TypeScript/React |
api/ | Vercel functions: Better Auth (/api/auth), dashboard data (/api/data), credits proxy (/api/cred), hosted inference API (/v1) | TypeScript |
db/ | Postgres schema (schema.sql) shared by the website and the fleet control plane | SQL |
Tech Stack
CLI & Backend: Python, Typer, Rich, FastAPI, Uvicorn, Pydantic
Frontend: React, TypeScript, Vite, Tailwind CSS, shadcn/ui, TanStack Query
Website & API: Vercel (static SPA + functions in api/), Neon Postgres, Vercel Blob (avatars)
Auth: Better Auth — email/password and "Continue with 1Profile"; Google/Apple sign-in when their OAuth credentials are configured
Credits: Cred, proxied through /api/cred
AI/ML: Ultralytics (YOLO), Transformers, ONNX Runtime, TensorRT, LangChain
Development
Frontend
npm install
npm run dev # Vite only, http://localhost:8080 (no api/ functions)
To run the api/ functions too (sign-in, dashboard data, credits, /v1), use the Vercel CLI:
vercel env pull .env.local # DATABASE_URL, BETTER_AUTH_SECRET, ... (see .env.example)
npm run db:migrate # apply db/schema.sql to that database
vercel dev
Without the server functions you can still use Continue as Demo User on the login page.
CLI
cd cli
pip install -e ".[all]" --group dev
Backend
cd backend
pip install -e . --group dev
The fleet control plane in backend/src/fleet is self-hosted: point DATABASE_URL at a Postgres database (Neon works) with db/schema.sql applied. See backend/.env.example.
Documentation
- openeye.sh/docs — the main docs (source:
src/data/docs/) docs/— MkDocs Material site with longer guides and tutorials
License
See LICENSE for details.
openeye-shpip install openeye-sh