Ripperoni

Web ingestion engine. Point it at a wiki, a site, or a URL list — it grinds raw pages into clean cuts of structured JSON, one prime cut per record.

Get startedWalk-throughArchitectureGitHub

Feed it a domain, hand it a plugin (a module that extracts structured data from a page), and Ripperoni fetches pages, runs your plugin against each one, and writes clean JSON records to disk.

A run has three artifacts:

1. Plugin in plugins/<namespace>/: ScalarNode subclasses + *.dag.jsonld documents + index.ts that exports register(dispatcher).
2. Orchestration <name>.dag.jsonld: one dagonizer DAG (JSON-LD) that wires the run — embedding the built-in crawl:discover DAG and scattering over collected URLs via the plugin's per-page DAG.
3. State <name>.state.json: run parameters (baseUrl, cache, output, headers, crawler block, rate limits, parallelism) validated at startup by RunStateSchema.

Run with: ripperoni run <orchestration>.dag.jsonld --state <run>.state.json

• DAG execution. Every scrape runs as a directed acyclic graph (DAG) — a structured execution plan where steps run in dependency order — powered by @studnicky/dagonizer. Placement types (ScalarNode, ScatterNode, EmbeddedDAGNode, TerminalNode) compose at the document level; add or rearrange stages without touching anything else.
• HTML scraper. Native fetch + cheerio. Returns a CheerioAPI handle; work with selectors you already know.
• Link crawler. The built-in crawl:discover DAG walks pages matching on domain/target/delimiter regexes — follows links all the way to the end of the casing. Deduplicates, sorts naturally, respects rate limits. Embed it in any orchestration via EmbeddedDAGNode { dag: "crawl:discover" }.
• Retry + backoff. Exponential backoff with decorrelated jitter. Respects Retry-After headers. Classifies errors as NETWORK / THROTTLED / TIMEOUT / TRANSIENT / PERMANENT and retries until the record lands clean.

Quick install

git clone https://github.com/Studnicky/Ripperoni.git
cd Ripperoni && npm install && npm run build

Next steps

• Walk-through: end-to-end example with a real URL, orchestration, plugin, and output record
• Getting started: install, scaffold, and first run
• Architecture: DAG topology, package boundaries, extension points