eb46e739e7
Playwright-based crawler that renders pages from api.intra-mart.jp and document.intra-mart.jp, extracts main content, and converts to Markdown via Pandoc. SQLite-backed queue drives a resumable sequential pipeline across crawler, extractor, converter, and storage modules. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
84 lines
4.1 KiB
Markdown
84 lines
4.1 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Overview
|
|
|
|
Documentation crawler that renders pages from `api.intra-mart.jp` and `document.intra-mart.jp` with Playwright, extracts the main content, and converts it to Markdown via Pandoc. Output is structured for Obsidian / MkDocs / Docusaurus / RAG ingestion.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
# Install (Python >= 3.11)
|
|
pip install -r requirements.txt
|
|
playwright install chromium
|
|
# pandoc must be on PATH (apt install pandoc / https://pandoc.org/installing.html)
|
|
|
|
# Run the crawler
|
|
python main.py
|
|
|
|
# Lint / format / typecheck / test (dev extras)
|
|
pip install -e ".[dev]"
|
|
ruff check .
|
|
black .
|
|
mypy .
|
|
pytest # asyncio_mode=auto is preset
|
|
pytest tests/test_x.py::test_name # single test
|
|
```
|
|
|
|
Re-running `python main.py` resumes from the SQLite queue — finished URLs are skipped, `processing` rows are reset to `retry` on startup.
|
|
|
|
## Architecture
|
|
|
|
The pipeline lives in `main.py` as a single `DocumentationCrawler` class that orchestrates per-module components. Flow per page:
|
|
|
|
```
|
|
URLManager.normalize/is_allowed
|
|
↓
|
|
PlaywrightClient.fetch_page (Chromium, JS-rendered, retries via tenacity)
|
|
↓
|
|
HTMLCleaner.clean (drop REMOVE_SELECTORS)
|
|
↓
|
|
HTMLExtractor.extract_main_content (first match of CONTENT_SELECTORS)
|
|
↓
|
|
AssetDownloader (download + rewrite asset links to local paths)
|
|
↓
|
|
MarkdownConverter.html_to_markdown (Pandoc → GFM, with YAML frontmatter)
|
|
↓
|
|
FileWriter / PathMapper (output/markdown/<domain>/<sanitized path>.md)
|
|
↓
|
|
extract_links + SphinxDiscovery + NavigationParser → enqueue new URLs
|
|
```
|
|
|
|
### State is in SQLite, not in memory
|
|
|
|
`storage/metadata_db.py` (`data/metadata.db`) is the source of truth. Tables:
|
|
|
|
- `crawl_queue` — drives the loop (`pending` / `processing` / `retry` / `failed` / `done`); `get_next_pending_url` orders by `priority DESC, depth ASC, discovered_at ASC`.
|
|
- `pages` — successful crawl metadata + markdown output path.
|
|
- `assets`, `page_links`, `discovery_sources`, `failures` — provenance and the site graph.
|
|
|
|
The crawl loop is **sequential single-worker** even though README mentions concurrency — there is no asyncio worker pool. `BROWSER_RECYCLE_EVERY = 500` restarts Playwright periodically to bound memory.
|
|
|
|
### Discovery is multi-source
|
|
|
|
Links are discovered three ways and merged before enqueue: (1) generic `<a href>` extraction in `DocumentationCrawler.extract_links` (also targets Sphinx-specific selectors like `a.reference.internal`, `.toctree-wrapper a`, `link[rel=next/prev/up]`), (2) `SphinxDiscovery` (probes `searchindex.js`, `genindex.html`, etc.), (3) `NavigationParser` (builds a nav tree for `SUMMARY.md` / `navigation_tree.json` export).
|
|
|
|
### URL → filesystem mapping
|
|
|
|
`storage/path_mapper.py` produces `<output_dir>/<domain>/<sanitized path>.md`. Sanitizes Windows-invalid chars + reserved names, hashes query strings into a `__<hash>` suffix, and shortens names > 180 chars with an MD5 tail. Asset link rewriting in `extractor/asset_downloader.py` computes paths *relative to the current page's output path*, so don't change `PathMapper` output layout without also revisiting `rewrite_asset_links`.
|
|
|
|
### Configuration
|
|
|
|
`config.yaml` exists but **`main.py` does not read it** — the live configuration is the module-level constants at the top of `main.py` (`START_URLS`, `ALLOWED_DOMAINS`, `REMOVE_SELECTORS`, `CONTENT_SELECTORS`, `MAX_PAGES_PER_RUN`, `BROWSER_RECYCLE_EVERY`, `MAX_ATTEMPTS`). Update those, not the YAML.
|
|
|
|
### Logging
|
|
|
|
`utils/logger.py` configures loguru on import (just `from utils.logger import logger`). Writes rotating logs to `logs/crawler.log`, `logs/errors.log`, `logs/debug.log` — no extra setup needed.
|
|
|
|
## Conventions
|
|
|
|
- Line length 79 (`black` and `ruff` both pinned to this). Target Python 3.11.
|
|
- `pytest-asyncio` runs in `auto` mode — async test functions need no decorator.
|
|
- Output dirs (`output/`, `data/`, `logs/`) are runtime artifacts; `output/assets/`, `output/markdown/`, `data/` are gitignored.
|