pagesift 0.73.0

High-performance, memory-bounded web scraping and extraction engine built for AI agents and LLM pipelines.
Turn any website, document, or archive into clean Markdown and structured data — one Rust binary, native SDKs, dual Firecrawl v1/v2 & Jina Reader APIs, at a fraction of the memory.

English · 简体中文 · 日本語 · Español

All cores again, hardened inputs, fewer false positives. 0.73.0 fixes what a three-way code review and ~150 adversarial input tests found.

• Extraction uses every core again. Since 0.70.0 scrape, crawl and serve extracted on one core; the API server now serves 5–7× more requests per second (16 clients: ~310 → up to 2,196 req/s, p50 40 ms → 4.6 ms).
• No more crashes or DoS paths found by the review: a CJK panic in the feed sniffer, UTF-8 pages re-decoded into mojibake, quadratic attribute parsing (40,000 attributes: 3 s → 0.02 s), slowloris clients, javascript: links from RSS items.
• Heuristics narrowed: short Substack posts were reported as age gates; author widgets, cloaked app roots, <template> content and CSS variables (as OpenAI keys) are handled correctly.
• API limits and timeouts on every parameter, with the standard JSON error body everywhere.
• Crawl scope decided from evidence (--scope auto, default): /docs/python stays in /docs/python/*; a leaf seed such as a blog post widens to its section, and says so.

Quality holds or improves (WCXB F1 0.8522 → 0.8533, Firecrawl 0.680 → 0.682, AEB 0.958), and extraction is 3–5% faster on real pages. Full detail: 0.73.0 release notes.

Truthful signals, complete allow-lists, better extraction. 0.72.8 closes the fourth-round audit and the fifth-to-eighth-round field audits.

• 🔤 Charset headers are verified — a lying Content-Type no longer produces silent mojibake; Pagesift re-decodes the body and reports every correction in metadata.pagesift.warnings.
• 🌐 --allow-cidr everywhere — crawl, map, download, session-fetch, collect and serve now accept the same private-network allow-list as scrape; invalid CIDRs fail loudly.
• ⚙️ Default config discovery — --config > PAGESIFT_CONFIG > the platform config file (~/.config/pagesift/pagesift.toml, $XDG_CONFIG_HOME, or %APPDATA% on Windows). PAGESIFT_CONFIG= disables config files for Docker/CI.
• 🩺 doctor tells the truth — it prints the config file that was actually loaded, or the default path to create.
• 🚦 No more fake success — map returns the seed error instead of a one-URL result, and crawl exits 1 when it collected zero pages and at least one error.
• 🍪 Recipes keep the recipe — content asides such as ingredient lists are no longer pruned; preview blocks (quickview/teaser) no longer replace the real article.
• 💻 GitHub code pages are code again — line numbers are dropped and code rows are merged into one fenced block.
• 👥 Authors survive — arXiv-style author lists stay in the body, and citation_author fills metadata.author.
• 🧱 Barriers say so — German/French/Spanish block pages are ACCESS_BARRIER, age gates are interstitial, and thin extraction emits thin_content.
• 📡 Feeds are parsed — RSS/Atom become Markdown item lists; jsonLd is passed through.
• 🖼️ Image titles and video metadata — title attributes are kept, and video pages fall back to their JSON-LD VideoObject instead of returning an empty document.

Full detail: 0.72.8 release notes.

A bug-fix release for the 0.70.0 field audit. The extraction engine and API contract are unchanged; failures are now reported honestly and authorization applies consistently.

• 🧭 Crawl scope fixed — /index.html-style seeds cover the whole site instead of one page.
• 🔐 Proxy and allow-lists — NO_PROXY is honored; --allow-cidr / allow_hosts also apply to browser rendering, not just HTTP.
• 🧾 Honest content reporting — binary responses return UNSUPPORTED_OPTION, empty pages carry empty_content, and bilibili captcha pages return ACCESS_BARRIER.
• 🔒 TLS errors are classified — TLS_CERT_EXPIRED, TLS_HOSTNAME_MISMATCH, TLS_UNTRUSTED_ROOT and TLS_ERROR replace the generic NETWORK_ERROR.
• 🩺 Root and auto fallback explained — doctor flags the root sandbox problem, and --engine auto prints why rendering was unavailable.
• 🚦 Exit codes match page status — HTTP 5xx and empty 404 responses exit with code 2.
• 🧹 Hidden templates filtered — x-cloak and similar pre-hydration templates no longer leak into Markdown.
• 📚 Charset handling unified — convert detects local file charsets and accepts --charset.

Full detail: 0.71.0 release notes.

The engine underneath the numbers was replaced. 0.60 proved PageSift could finish 100,000,000 tasks; 0.70 rebuilds the data plane that does it and puts the whole system under one closed-loop controller.

• ⚙️ Hand-written NUMA-local HTTP data plane — fixed-slot NUMA executors, a minimal HTTP/1.1 reactor, hybrid reactor↔content scheduling, per-origin adaptive connections and sharded intake with bounded work stealing. The general-purpose async client is gone from the hot path.
• 📈 Scaling measured 1 → 192 vCPU, fitted with the Universal Scalability Law — γ = 8,631.8 pages/s/core, σ = 1.34 %, κ = 0.00 % (no measurable coherency penalty), R² = 0.839.
• 🧠 Memory is now an input. The engine reads cgroup v2 memory.max and sizes itself to it: at 768 MiB it settles on 380 connections and still sustains 817,346 pages/s with zero OOM.
• 🏁 200M saturation on 192 vCPU — 200,000,000 / 200,000,000, 0 failures, 947,611 pages/s, 211.1 s wall, 888.1 MiB peak RSS.
• 📊 Seven public benchmarks, published including the losses — WebMainBench 0.6414 ROUGE-5 (level with Trafilatura, above Resiliparse), WCEB 0.8613, WCXB 0.8472, robots.txt spec 0.9621, olmOCR-bench 0.4172, Common Crawl 5,165 p/s, Crawl Maze 0.0500.
• 🔧 Four production-grade defects fixed, all found by the stress suite: header-wait ignoring timeout_secs, 429 penalties that never recovered, one broken URL freezing a whole host, and a runtime panic under tight ulimit -v.
• ⚖️ Faster and cheaper per page than 0.61: 2.22× absolute throughput (947,611 p/s on 192 vCPU), +24.4 % per core, and −19.6 % CPU per page — while also lifting WebMainBench ROUGE-5 by +0.055. Per-core output on a 16-core box sits within 4.8 % of the 192-core run.

Full detail: 0.70.0 release notes.

Every number below is a full-set run on the same machine, with the suite's own scorer, under a 10 GiB memory cap. Competitors run in the same containers. Where we lose, the loss is printed.

Measured on Intel i7-10700 (8C/16T), 15 GB RAM, Ubuntu 26.04. Reproduce with bench/scripts/ — every report links to the exact binaries, dataset checksums and scorer commits.

Seven domains measured, four won. The three we did not win are reported as-is:

• D1: we trade recall for precision — 0.8854 precision (highest of all 17 entrants) vs 0.5989 recall. WCXB's reference text includes boilerplate, so whole-page output "wins" on this metric.
• D9: we render SPAs correctly (client-side and hydration both hit) and finish in 0.47 s vs Playwright's 2.55 s — 5× faster — but we don't wait for setTimeout-delayed injections.
• D4: rank 2nd overall, but we win outright on adversarial decoys (100 % vs 86.4 %). The gap comes from a category where 14.6 % of the dataset's labels are wrong (a "price" slot contains a person's name).

Speed where it matters: 1,403 PDFs in 0.83 s (31× faster than pypdf, 296× faster than pymupdf4llm), 3,128 encoding detections at 11,239 files/s, 21 MiB peak for structured data.

Machine-verifiable: full reports under docs/benchmarks/0.70_showdown_D*.md, raw outputs under out/ (gitignored), harnesses under bench/scripts/.

Hardware Testbed: Google Cloud n4d-highcpu-80 (80 vCPUs AMD EPYC 9B45, 157 GiB RAM, 96 GB NVMe SSD) Target Dataset: Tranco Top 1,000,000 Global Live Domains dynamically expanded to 100,000,000 Real Public Web Pages under 4,096 concurrent sockets across 32 Share-Nothing shards.

+-----------------------------------------------------------------------------------------+
| PageSift 0.60 WILD-X 2 Showdown (100,000,000 Real Public Web Pages on 80-Core 157GB VPS) |
+-----------------------------------------------------------------------------------------+
| Metric                       | PageSift 0.60 Performance                                |
+------------------------------+----------------------------------------------------------+
| Total Web Pages Processed    | 100,000,000 (One Hundred Million) Pages                 |
| Total Execution Wall Time    | 2,373.9 seconds (39.56 minutes / 0.66 hours)            |
| Sustained Live Public QPS    | 42,124.1 pages/sec (Peak > 51,900 p/s)                   |
| 128 GB Memory Defense        | Peak RSS: 11.59 GB (flat line, 8.8% of 128GB ceiling)   |
| Zero-Disk Retention Delta    | Physical NVMe disk increase < 100 MB (85 GB free)        |
| Genuine 200 OK Raw Fetches   | 314,170 pages (real-world diverse public web)           |
| Pure Rich Markdown Distilled | 199,422 documents (2.31 GB pure content, zero garbage)   |
| Resilience & Auto-Recovery   | 615k DNS errs, 244k timeouts, 205k WAF 403s handled     |
| Multi-Million Scalability    | 100% OOM-Proof, Zero Leaks, Zero Hangs, Zero Panics      |
+-----------------------------------------------------------------------------------------+

Run the standalone container (includes headless browser renderer):

docker run -d -p 3002:3002 -v pagesift-data:/data \
  -e PAGESIFT_API_KEYS=sk-test ghcr.io/yolow92/pagesift:0.73.0

Point your existing Firecrawl application directly to PageSift — zero code refactoring:

from firecrawl import FirecrawlApp

# Drop-in local PageSift instance
app = FirecrawlApp(api_url="http://localhost:3002", api_key="sk-test")
res = app.scrape_url("https://example.com", params={"formats": ["markdown"]})
print(res["markdown"])

Point any LLM Agent or simple HTTP client to PageSift's /r/ endpoint:

# Direct Markdown output
curl -s http://localhost:3002/r/https://en.wikipedia.org/wiki/Rust_(programming_language)

# Streamed SSE output
curl -s -N -H "Accept: text/event-stream" http://localhost:3002/r/https://news.ycombinator.com

pip install pagesift

import pagesift

# Single-page fast extraction (< 2ms)
page = pagesift.scrape("https://example.com", formats=["markdown", "metadata"])
print(page.markdown)
print(page.metadata.title, page.metadata.published_date)

# Batch scraping with adaptive bounded memory
results = pagesift.scrape_many(["https://site1.com", "https://site2.com"], concurrency=8)

npm install @pagesift/node
# or
npm install pagesift

import { scrape, scrapeMany } from '@pagesift/node'

// Scrape page to clean LLM markdown
const page = await scrape('https://example.com', {
  formats: ['markdown', 'links']
})
console.log(page.markdown)

// Concurrent stream processing
const stream = scrapeMany(['https://a.com', 'https://b.com'], { concurrency: 4 })
for await (const page of stream) {
  console.log(`Scraped: ${page.url} in ${page.elapsedMs}ms`)
}

# Install standalone binary via cargo
cargo install pagesift --locked

# Scrape single URL
pagesift scrape "https://news.ycombinator.com" --format markdown

# High-concurrency polite crawl with adaptive hardware tuning
pagesift crawl "https://docs.rs" --limit 5000 --tuning-mode adaptive --out ./data/

# Config is discovered automatically from ~/.config/pagesift/pagesift.toml
# (or $XDG_CONFIG_HOME / %APPDATA%). PAGESIFT_CONFIG= disables config files.
pagesift doctor

# One local service for every agent on the machine (127.0.0.1 only)
pagesift mcp http --port 3003 --audit-log ~/.local/state/pagesift/audit.jsonl
claude mcp add --scope user --transport http pagesift http://127.0.0.1:3003/mcp

Shared browsers and cache, an audit line per fetch, HTML snapshots of suspicious results and a pagesift_report_issue tool. See Running pagesift as a shared MCP service.

• Segmented Bounded Frontier: Memory does not scale with crawl depth. Even with 100,000 URLs, backpressure and bounded queues prevent out-of-memory disasters.
• Dynamic Hardware Adaptive Tuning: Detects system resources and calculates the safe theoretical saturation point without crashing or exhausting sockets.
• Selective Rendering Pipeline: Automatically detects static vs dynamic pages; executes headless Chrome only when dynamic JavaScript hydration provides net content gains.
• SSRF & Sandbox Defense: Strict private IP filtering, DNS-rebinding guards, and perpetual browser sandboxing ensure zero local network exposure.
• Ethical Politeness: Per-host token buckets, automatic robots.txt compliance, and RFC-compliant User-Agent identification.

• Open Source: Licensed under AGPL-3.0.
• Commercial Licensing: Need proprietary linking or private deployment without AGPL obligations? Check out our Commercial Licensing Guide or contact licensing@pagesift.dev.

• NUMA-local HTTP data plane (0.70)
• cgroup-aware adaptive capacity (0.70)
• Seven public extraction benchmarks with reproducible harnesses (0.70)
• Resumable crawl checkpoints — restart without re-fetching
• OCR fallback for scanned documents
• Resource-URL enumeration mode for security crawlers
• Distributed shard coordination across nodes

Vote or propose: Issues · Discussions

If PageSift is useful in research or a product, cite it as:

@software{pagesift2026,
  title  = {PageSift: NUMA-Local Rust Web Extraction Engine for AI Pipelines},
  author = {YOLOW92},
  year   = {2026},
  url    = {https://github.com/YOLOW92/pagesift},
  note   = {Version 0.73.0}
}