Playwright-based crawler that renders pages from api.intra-mart.jp and document.intra-mart.jp, extracts main content, and converts to Markdown via Pandoc. SQLite-backed queue drives a resumable sequential pipeline across crawler, extractor, converter, and storage modules. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| crawler.py | ||
| http_client.py | ||
| navigation_parser.py | ||
| playwright_client.py | ||
| sitemap_parser.py | ||
| sphinx_discovery.py | ||
| url_manager.py | ||