Your Data Science
Resource Deck

LangExtract is a package that extracts structured information from documents and maps it back to the exact location in the source text — then renders the result as an interactive HTML viewer. What made it succeed is how cleanly it taps into a real anxiety: when GPT gives you an answer without telling you where it came from, the first instinct is to doubt it. So you predefine the fields you want, the LLM pulls them out, and every extraction shows up alongside the evidence — the exact span of source it came from.

EdgeQuake is a package that implements the LightRAG algorithm in async Rust. Six query modes cover the spectrum — naive (keyword-style vector search), local (entity-centered local graph), global (community-based, thematic), the default hybrid (local + global combined), mix (weighted combination), and bypass (skip RAG, query the LLM directly) — each picked for the speed/cost tradeoff you need. On top of that, results are easy to inspect through the built-in web visualization, and an MCP server is included so agents like Claude and Cursor can call EdgeQuake directly.

Most document-intelligence stacks force you into Python. Kreuzberg refuses: the engine is Rust, but it ships as native bindings for Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, R, C, and TypeScript (Node/Bun/WASM/Deno) — and if none of those fit, it runs as a CLI, REST server, or MCP server that any agent host can call. The output shape is unified: every input — PDF, .docx, .epub, .hwp, .ipynb, an entire codebase — comes back as one `ExtractionResult` with `code_intelligence` and `semantic_chunking` fields, so a 'PDF for RAG' pipeline and a 'codebase for an agent' pipeline use the same call. OCR is pluggable across Tesseract (incl. WASM), PaddleOCR, EasyOCR, and VLM-OCR through 146 providers (GPT-4o, Claude, Gemini, Ollama, vLLM, llama.cpp) — stay local or route through a frontier model with one config flip. Two design quirks worth borrowing: a TOON wire format that's 30–50% smaller than JSON for the same payload, and HTML→Markdown via direct conversion instead of an AST round-trip.

On professional documents (10-K filings, contracts, technical manuals), similarity ≠ relevance — what you actually need isn't embedding-based distance, it's logical reasoning. PageIndex strips out the vector pipeline entirely: it extracts a hierarchical table-of-contents tree from the PDF, then an agent reads node summaries and walks the tree to the right section, leaving a justification at every step. The retrieval trail itself becomes evidence for *why* a particular section was chosen, so results carry a credibility log instead of just a similarity score. The cost is honest — every query takes at least one LLM call, often several, so latency and per-query cost go up. Even so, the performance is real: PageIndex hit 98.7% on FinanceBench, beating vector-based RAG.

OpenDataLoader sits in the same slot as MinerU, Docling, or LlamaParse, with two unusual bets that change how it feels in production. First, the default mode is fully local and deterministic: `pip install`, call `convert(...)`, and get reproducible Markdown / HTML / JSON-with-bboxes from an XY-Cut++ reading-order pass — no LLM, no network. Second, the same layout engine doubles as a PDF accessibility auto-tagger (the team claims the first OSS end-to-end Tagged PDF generator), so the structure you extract for RAG is the same structure that satisfies WCAG-style remediation. The hybrid mode adds OCR (80+ languages), formula extraction, and chart description by routing pages through a local AI backend, and it's the configuration that wins the team's 200-PDF benchmark at 0.907 overall. JSON output ships per-element bounding boxes, so downstream chunks can carry source-citation rectangles without any post-hoc layout inference.

GLM-OCR is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance across diverse document layouts. Only 0.9B parameters.