crawler-intra-mart/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Overview

Documentation crawler that renders pages from api.intra-mart.jp and document.intra-mart.jp with Playwright, extracts the main content, and converts it to Markdown via Pandoc. Output is structured for Obsidian / MkDocs / Docusaurus / RAG ingestion.

Commands

# Install (Python >= 3.11)
pip install -r requirements.txt
playwright install chromium
# pandoc must be on PATH (apt install pandoc / https://pandoc.org/installing.html)

# Run the crawler
python main.py

# Lint / format / typecheck / test (dev extras)
pip install -e ".[dev]"
ruff check .
black .
mypy .
pytest                            # asyncio_mode=auto is preset
pytest tests/test_x.py::test_name # single test

Re-running python main.py resumes from the SQLite queue — finished URLs are skipped, processing rows are reset to retry on startup.

Architecture

The pipeline lives in main.py as a single DocumentationCrawler class that orchestrates per-module components. Flow per page:

URLManager.normalize/is_allowed
        ↓
PlaywrightClient.fetch_page    (Chromium, JS-rendered, retries via tenacity)
        ↓
HTMLCleaner.clean              (drop REMOVE_SELECTORS)
        ↓
HTMLExtractor.extract_main_content  (first match of CONTENT_SELECTORS)
        ↓
AssetDownloader                (download + rewrite asset links to local paths)
        ↓
MarkdownConverter.html_to_markdown  (Pandoc → GFM, with YAML frontmatter)
        ↓
FileWriter / PathMapper        (output/markdown/<domain>/<sanitized path>.md)
        ↓
extract_links + SphinxDiscovery + NavigationParser → enqueue new URLs

State is in SQLite, not in memory

storage/metadata_db.py (data/metadata.db) is the source of truth. Tables:

• crawl_queue — drives the loop (pending / processing / retry / failed / done); get_next_pending_url orders by priority DESC, depth ASC, discovered_at ASC.
• pages — successful crawl metadata + markdown output path.
• assets, page_links, discovery_sources, failures — provenance and the site graph.

The crawl loop is sequential single-worker even though README mentions concurrency — there is no asyncio worker pool. BROWSER_RECYCLE_EVERY = 500 restarts Playwright periodically to bound memory.

Discovery is multi-source

Links are discovered three ways and merged before enqueue: (1) generic <a href> extraction in DocumentationCrawler.extract_links (also targets Sphinx-specific selectors like a.reference.internal, .toctree-wrapper a, link[rel=next/prev/up]), (2) SphinxDiscovery (probes searchindex.js, genindex.html, etc.), (3) NavigationParser (builds a nav tree for SUMMARY.md / navigation_tree.json export).

URL → filesystem mapping

storage/path_mapper.py produces <output_dir>/<domain>/<sanitized path>.md. Sanitizes Windows-invalid chars + reserved names, hashes query strings into a __<hash> suffix, and shortens names > 180 chars with an MD5 tail. Asset link rewriting in extractor/asset_downloader.py computes paths relative to the current page's output path, so don't change PathMapper output layout without also revisiting rewrite_asset_links.

Configuration

config.yaml exists but main.py does not read it — the live configuration is the module-level constants at the top of main.py (START_URLS, ALLOWED_DOMAINS, REMOVE_SELECTORS, CONTENT_SELECTORS, MAX_PAGES_PER_RUN, BROWSER_RECYCLE_EVERY, MAX_ATTEMPTS). Update those, not the YAML.

Logging

utils/logger.py configures loguru on import (just from utils.logger import logger). Writes rotating logs to logs/crawler.log, logs/errors.log, logs/debug.log — no extra setup needed.

Conventions

• Line length 79 (black and ruff both pinned to this). Target Python 3.11.
• pytest-asyncio runs in auto mode — async test functions need no decorator.
• Output dirs (output/, data/, logs/) are runtime artifacts; output/assets/, output/markdown/, data/ are gitignored.