Repository avatar
Other Tools
v1.0.2
active

documents-to-data

io.github.owlabee/documents-to-data

Turn PDFs, documents, podcasts and drone photos into Markdown, structured JSON, summaries, reports.

Documentation

owlabee

owlabee publishes 20 pay-per-result Apify Actors that turn public data sources (permit portals, government and procurement APIs, court feeds, license registries, DNS, sitemaps, storefronts) into normalized JSON.

This repo has runnable examples for each Actor: curl, Python, Node.js and an MCP config for AI agents. Each Actor is billed per result to your own Apify account; see its Store page for current pricing.

Guides to the data sources: owlabee.com

Made by Jacob Hite.

Actors

ActorWhat it returnsExamples
Building Permits Scraper — Roofing & Construction LeadsBuilding permits from 120 open-data datasets in 97 US and Canadian cities, counties and states: type, status, dates, cost, address, contractorexamples/building-permits-scraper
Accela Citizen Access & EnerGov Permit Scraper (Any City)Permits from Accela Citizen Access and Tyler EnerGov portals in 31 cities and counties, in the same schema as the open-data scraperexamples/accela-energov-permit-scraper
Grants.gov API & SAM.gov Scraper — Federal Grants FeedGrants.gov grant opportunities and SAM.gov contract notices by keyword, plus optional USAspending awards and six state and local bid datasetsexamples/grants-gov-api
Grant Matching & RFP Bid/No-Bid Scorer (AI) — Grants.govGrants.gov, SAM.gov and USAspending opportunities ranked by fit to your organization, with reasons and disqualifiersexamples/grant-matching
Tenders Scraper: EU TED, UK Find a Tender & AusTenderPublic tenders and contract award notices from EU TED, UK Find a Tender, UK Contracts Finder and AusTender in one record shapeexamples/tenders-scraper
PACER RSS Docket Monitor — New Federal Court FilingsNew docket entries from the public PACER RSS feeds of federal district, bankruptcy, appellate and national courts, filtered by party, case or entry typeexamples/pacer-docket-monitor
OSHA Inspections by Company + EPA Enforcement ReportOSHA inspections, violations and penalties and EPA ECHO enforcement cases by company, facility, state or ZIP, plus opt-in MSHA, Wage and Hour, FMCSA and OSHA accident recordsexamples/osha-epa-inspections
Product Recall Feed — FDA, NHTSA & CPSC Recalls APIFDA drug, device and food recalls, NHTSA vehicle recalls and CPSC consumer product recalls in one normalized record, plus opt-in Canada, UK and EU recall feedsexamples/product-recall-feed
License Verification API — 7 States + NPI RegistryProfessional license records by number, or by name within one board, from 10 boards in 7 states (TX, FL, CO, WA, CT, NJ, MI) and the NPI Registryexamples/license-verification-api
SPF, DMARC & DKIM Checker — Bulk Domain WHOIS, DNS & SSLOne record per domain: SPF, DMARC, DKIM, MTA-STS and BIMI with a 0-100 email security score, RDAP WHOIS, DNS, TLS and HTTP headersexamples/spf-dmarc-dkim-checker
Website Change Monitor — Sitemap Monitor & Page DiffPages discovered through a site's sitemaps, and new, removed, updated and content-changed pages between runs with diff previewsexamples/website-change-monitor
Shopify Catalog Monitor & Price Tracker — Products ScraperEvery product or variant of a public Shopify store from products.json, or price, stock and new or removed product changesexamples/shopify-catalog-monitor
E-commerce Scraper & Price Monitor — WooCommerce, MagentoProducts and variants of WooCommerce, Magento, Shopify and other stores (schema.org Product JSON-LD), or price, stock and new or removed product changesexamples/ecommerce-scraper
Bluesky Monitor & Scraper — Posts, Keywords, FollowersBluesky posts by keyword or hashtag, profiles, author feeds, follower lists, post threads and quotes, custom feeds and lists from the public AT Protocol APIexamples/bluesky-monitor
Hiring Signals & ATS Jobs Scraper — Greenhouse, Lever, AshbyOpen roles from Greenhouse, Lever, Ashby, SmartRecruiters and Workable job boards, or new, removed and changed roles per companyexamples/job-postings-api
Podcast RSS Scraper — Episodes, Transcripts & Apple PodcastsShow and episode records from podcast RSS feeds, Apple Podcasts links, Apple charts, directory searches or Podcast Index GUIDs, with Podcasting 2.0 transcripts and chaptersexamples/podcast-rss-scraper
Podcast Episode Summarizer (AI) — Transcript to SummaryPer-episode abstract, key topics, chapters, quotes, guests and action items from Podcasting 2.0 transcripts or show notesexamples/podcast-summarizer
PDF to Markdown for RAG — DOCX, OCR & ChunksClean Markdown from PDF, Office, OpenDocument, EPUB, HTML, text, CSV and image URLs, with optional RAG chunks, OCR in 21 languages, table records and imagesexamples/pdf-to-markdown
PDF & Document to JSON Extractor (AI) — Invoices, FormsOne JSON object per PDF, Office, EPUB, HTML or scanned document that matches the JSON Schema you supply, with per-field confidenceexamples/pdf-to-json
AI Drone Roof Inspection Report — Findings & Repair ScopePer-photo roof defect findings, per-building condition grade and repair scope, a Markdown report and a contact sheetexamples/drone-roof-inspection-report

Quick start

  1. Create an Apify account and copy your API token from Console → Settings → API & Integrations.

  2. Put it in your environment. Never commit it.

    export APIFY_TOKEN=<your Apify API token>
    
  3. Run any example. Every folder has an input.json taken from that Actor's own quick-start input:

    cd examples/product-recall-feed
    ./curl.sh
    

    Or call the API directly. run-sync-get-dataset-items starts a run, waits for it (up to 300 seconds) and returns the dataset items:

    curl -sS -X POST \
      "https://api.apify.com/v2/acts/owlabee~product-recall-feed/run-sync-get-dataset-items" \
      -H "Authorization: Bearer ${APIFY_TOKEN}" \
      -H "Content-Type: application/json" \
      -d '{"agencies": ["fda-drug", "nhtsa", "cpsc"], "daysBack": 30, "maxResultsPerSource": 5}'
    

Python (pip install apify-client) and Node.js (npm install apify-client) versions are in each folder as python.py and node.mjs. To cap spend on a run, set its maximum total charge in the Console or pass maxTotalChargeUsd as a run option.

Use from AI agents (MCP)

Apify hosts an MCP server at mcp.apify.com. Add ?tools= to expose only the Actors you want. This config gives an agent the recall feed and the license lookup:

{
  "mcpServers": {
    "owlabee": {
      "url": "https://mcp.apify.com/?tools=owlabee/product-recall-feed,owlabee/license-verification-api",
      "headers": { "Authorization": "Bearer ${APIFY_TOKEN}" }
    }
  }
}

Claude Code expands ${APIFY_TOKEN} from your environment; Cursor uses ${env:APIFY_TOKEN}. Each examples/<actor>/mcp.json is the same config for one Actor, and its README shows the mcp-remote variant for Claude Desktop.

Use without code (Zapier, Make, n8n)

Apify has official apps on Zapier, Make and n8n, so any owlabee Actor can run from them with your own Apify account. Step-by-step recipes with ready inputs:

Repository layout

PathWhat it is
examples/<actor>/input.json, curl.sh, python.py, node.mjs, mcp.json and a README per Actor
docs/The static guide site, served by GitHub Pages (generated, plain HTML and one CSS file)
tools/Zero-dependency Node scripts that generate examples/ and docs/, refresh the free data snapshots (tools/data-pages/refresh.mjs) and check links

To change the site's base URL, edit tools/site-config.mjs, then run node tools/build-examples.mjs && node tools/build-site.mjs and commit the output. Check the result with node tools/check-links.mjs (internal links, anchors, sitemap.xml and llms.txt; add --external to request every outside link once).

Free data and comparison pages

Besides the guides, the site has:

  • Free data (owlabee.com/data/): weekly snapshots of product recalls, grant and bid opportunities, and building permits, each made by one or two small runs of our own Actors and published as a page plus docs/data/<id>.json. Records are cut to an allowlist of fields: no person names or contact details, permits at street level (no house number), and every licensed source's attribution line on its page. Rendered by tools/data-pages.mjs.
  • Comparisons (owlabee.com/compare/): dated, sourced comparisons of the ways to get permit, PACER alert, recall and grant data, ours included, each saying where the alternatives are better. Content and sources live in tools/compare-pages.mjs; re-check the numbers when you edit it and update its CHECKED date.
  • llms.txt at the site root: a plain-text map of the Actors and pages for AI assistants, generated with the site.

tools/discovery-pages.mjs registers all of these with tools/build-site.mjs (nav, home-page section, sitemap). The sitemap lists them, so the IndexNow workflow submits them on the next push to main that changes docs/.

Weekly refresh. The mmm coordinator re-runs the snapshots once a week (its daily run does it on Mondays) and lands the result through a PR:

NODE_USE_ENV_PROXY=1 APIFY_TOKEN=<token from the environment> node tools/data-pages/refresh.mjs
node tools/check-links.mjs

The script starts eight small runs on the owlabee Apify account, each capped with maxTotalChargeUsd 0.10, rewrites docs/data/*.json and rebuilds the site; --build-only re-renders from the saved snapshots, and recalls, grants or permits refreshes one. It reads the token from the environment and never prints or stores it. There is deliberately no GitHub Actions workflow for it, since that would need the token as a repository secret.

Notes

  • Data comes from public sources as each source publishes it. Coverage, freshness and field completeness vary by source; check the linked source record before relying on a value.
  • owlabee is an independent developer on Apify Store. It is not affiliated with or endorsed by Apify, or by any agency, court, registry or company named in this repo.
  • Example code is MIT licensed (see LICENSE).