Repository avatar
AI Tools
v0.3.0
active

openeye

io.github.aw3-technology/openeye

Vision for agents: run YOLO, depth, segmentation and detection models locally or via hosted API.

Documentation

OpenEye

Open-source eyes for the agent era.

PyPI Downloads Python License: MIT GitHub stars

Website · Docs · Models · Blog · PyPI

OpenEye is a CLI-first perception engine that turns raw video into structured world models for robots and autonomous agents. Pull, run, and serve computer-vision models from your terminal — like Ollama, but for vision AI.

Features

  • Object Detection — YOLOv8, YOLO26 (currently ships YOLO11n weights), Grounding DINO, RF-DETR, and custom adapters
  • Depth Estimation — Monocular depth via Depth Anything V2
  • Segmentation — SAM2 segment anything
  • Scene Understanding — Spatial relationships, scene graphs, natural language queries
  • VLM Reasoning — Vision-language model integration (Nebius, OpenRouter)
  • Agentic Loop — Continuous perception → reasoning → planning pipeline
  • Safety Monitoring — Zone-based awareness with real-time human detection and robot halt
  • CLI-Native — Every capability accessible from the terminal
  • REST + WebSocket API — Serve models over HTTP with a single command
  • Fleet Management — Register, deploy, and monitor edge devices at scale
  • MLOps — Model registry, versioning, A/B testing, batch inference
  • Governance — Policy enforcement, audit logging, safety rule management
  • Hosted API — https://openeye.sh/v1 for object detection without running anything locally (pay per call in credits)
  • Web Dashboard — openeye.sh/dashboard: API keys, credits, history, fleet management, and — connected to your own openeye serve — inference, live streams, and more

Quick Install

pipx install openeye-sh

With YOLO support:

pipx install "openeye-sh[yolo]"

Quick Start

# Pull a model
openeye pull yolov8

# Run inference on an image
openeye run yolov8 photo.jpg

# Start the server with REST API, WebSocket, and browser dashboard
openeye serve yolov8

# Live camera feed with real-time detections
openeye watch --models yolov8

# Live camera feed with safety monitoring
openeye watch --models yolov8 --safety

Give Your AI Agent Eyes (MCP)

OpenEye ships an MCP server, so Claude, Cursor, or any MCP client can pull models and run detection, depth, and segmentation on your machine:

pipx install "openeye-sh[mcp]"
claude mcp add openeye -- openeye-mcp

Then ask: "Run yolov8 on ~/Pictures/desk.jpg and tell me what's on my desk." Setup for Claude Desktop and other clients: docs/claude-skill.

Hosted API Quick Start

No GPU or local install needed for detection:

  1. Sign in at openeye.sh (email/password or 1Profile).
  2. Create a key in Dashboard → Settings → API Keys (keys start with oe_ and are shown once).
  3. Buy credits in Dashboard → Settings → Credits.
  4. Call the API:
curl -X POST https://openeye.sh/v1/detect \
  -H "Authorization: Bearer oe_your_key_here" \
  -F "file=@photo.jpg"

# or with the CLI
export OPENEYE_API_KEY=oe_your_key_here
openeye api detect photo.jpg
openeye api usage
EndpointStatusCredits
POST /v1/detectLive (YOLOv8n, JPEG/PNG under 4 MB)1
POST /v1/describeReturns 503 until the VLM is enabled on the hosted API3
POST /v1/depthReturns 501 — self-hosted only (openeye serve depth-anything)2
GET /v1/models, GET /v1/usageLivefree

The hosted API has no WebSocket streaming; for streaming, run openeye serve. See the docs for details.

Use the Dashboard with a Local Server

openeye serve allows requests from https://openeye.sh by default, so the hosted dashboard can drive a server on your machine:

openeye serve yolov8          # http://localhost:8000

Open openeye.sh/dashboard, click Server on the Overview page (the "OpenEye Server URL" dialog), and enter http://localhost:8000. Inference, Live, Scene Graph, Metrics, Benchmark, Config, Governance, MLOps, and the live agent loop then run against your server. If your browser blocks the connection from an HTTPS page to localhost, serve OpenEye over HTTPS instead.

Interactive CLI Demo

An interactive walkthrough that runs real OpenEye commands against sample images. Great for evaluating the tool, onboarding, or live presentations.

Prerequisites

  • Python 3.12+
  • jq (optional, for JSON pipeline steps — brew install jq)
  • curl (for the API server step)

Setup

# Install the CLI with YOLO and camera support
pipx install -e "./cli[yolo,camera]"

# Pull the YOLOv8 model (6 MB)
openeye pull yolov8

Run the Demo

./demo.sh

demo.sh is a bash script: run it on macOS, Linux, WSL, or Git Bash on Windows. It uses openeye from your PATH (pipx), falling back to ./.venv or cli/.venv. In plain PowerShell or cmd, run the same steps yourself:

openeye list
openeye pull yolov8
openeye run yolov8 src/assets/demo/scene-warehouse.jpg --pretty
openeye run yolov8 src/assets/demo/scene-warehouse.jpg --visualize
openeye bench yolov8 --warmup 2 --runs 5
openeye serve yolov8 --port 8111

The demo is interactive — press Enter to advance between steps. Each step shows the command being run and its live output.

What the Demo Covers

StepCommandWhat it demonstrates
1openeye listBrowse the model registry (7 vision models)
2openeye pull yolov8Download a model — like ollama pull
3openeye run yolov8 scene-warehouse.jpg --prettyObject detection with structured JSON output
4openeye run yolov8 scene-kitchen.jpg --prettyDetection on a different scene (apple, cup, donut)
5openeye run ... --visualizeSave an annotated image with bounding boxes
6openeye run ... | jqUnix pipeline composability — pipe JSON to jq
7openeye bench yolov8Benchmark inference speed (latency, FPS, P95)
8openeye serve yolov8Launch a REST API server and query it with curl
9Batch loopProcess all 4 demo images and summarize results

Demo Images

Four sample images are included in src/assets/demo/:

ImageSceneTypical detections
scene-warehouse.jpgWarehouse with people and vehiclesperson, truck, tv
scene-kitchen.jpgKitchen countertopapple, cup, donut, bottle
scene-workshop.jpgWorkshop environmentperson, dining table, remote, knife
safety-workspace.jpgSafety-critical workspace(used for safety guardian demos)

Output

Annotated images are saved to demo_output/. The API server step temporarily runs on port 8111 and shuts down automatically.

Beyond the Demo

Commands not covered in the scripted demo that you can try interactively:

# Live webcam detection with terminal overlay
openeye watch --models yolov8

# Safety Guardian — zone-based human proximity alerts
openeye watch --models yolov8 --safety --danger-m 0.5 --caution-m 1.5

# Unitree G1 robot safety demo (dry-run, no robot needed)
openeye g1-demo --control-mode dry_run --demo

# Agentic perception loop — perceive, reason, act
openeye agent run

# Edge device fleet management
openeye fleet ls

# Open-vocabulary detection with text prompts (requires grounding-dino)
openeye pull grounding-dino
openeye run grounding-dino photo.jpg --prompt "person. hard hat. forklift."

Project Structure

openeye.sh/
├── cli/              # openeye-sh CLI & inference server (Python)
├── backend/          # Perception engine runtime (Python)
├── src/              # React web frontend (TypeScript)
├── api/              # Vercel functions (auth, data, credits, /v1 API)
├── db/               # Postgres schema
└── docs/             # MkDocs documentation site
PackageDescriptionLanguage
cli/CLI tool & FastAPI inference server (port 8000), MCP server (openeye-mcp)Python
backend/Perception pipeline, self-hosted fleet control plane (port 8001), governance enginePython
src/Website and dashboard at openeye.shTypeScript/React
api/Vercel functions: Better Auth (/api/auth), dashboard data (/api/data), credits proxy (/api/cred), hosted inference API (/v1)TypeScript
db/Postgres schema (schema.sql) shared by the website and the fleet control planeSQL

Tech Stack

CLI & Backend: Python, Typer, Rich, FastAPI, Uvicorn, Pydantic

Frontend: React, TypeScript, Vite, Tailwind CSS, shadcn/ui, TanStack Query

Website & API: Vercel (static SPA + functions in api/), Neon Postgres, Vercel Blob (avatars)

Auth: Better Auth — email/password and "Continue with 1Profile"; Google/Apple sign-in when their OAuth credentials are configured

Credits: Cred, proxied through /api/cred

AI/ML: Ultralytics (YOLO), Transformers, ONNX Runtime, TensorRT, LangChain

Development

Frontend

npm install
npm run dev          # Vite only, http://localhost:8080 (no api/ functions)

To run the api/ functions too (sign-in, dashboard data, credits, /v1), use the Vercel CLI:

vercel env pull .env.local   # DATABASE_URL, BETTER_AUTH_SECRET, ... (see .env.example)
npm run db:migrate           # apply db/schema.sql to that database
vercel dev

Without the server functions you can still use Continue as Demo User on the login page.

CLI

cd cli
pip install -e ".[all]" --group dev

Backend

cd backend
pip install -e . --group dev

The fleet control plane in backend/src/fleet is self-hosted: point DATABASE_URL at a Postgres database (Neon works) with db/schema.sql applied. See backend/.env.example.

Documentation

  • openeye.sh/docs — the main docs (source: src/data/docs/)
  • docs/ — MkDocs Material site with longer guides and tutorials

License

See LICENSE for details.

PYPI
openeye-sh
Install Command
pip install openeye-sh
Runtime: uvx