eb46e739e7
Playwright-based crawler that renders pages from api.intra-mart.jp and document.intra-mart.jp, extracts main content, and converts to Markdown via Pandoc. SQLite-backed queue drives a resumable sequential pipeline across crawler, extractor, converter, and storage modules. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
34 lines
508 B
YAML
34 lines
508 B
YAML
seed_urls:
|
|
- "https://api.intra-mart.jp/iap/index.html"
|
|
- "https://document.intra-mart.jp/library/"
|
|
|
|
allowed_domains:
|
|
- "api.intra-mart.jp"
|
|
- "document.intra-mart.jp"
|
|
|
|
output_dir: "output"
|
|
|
|
crawl:
|
|
concurrency: 5
|
|
delay_seconds: 1
|
|
timeout: 30000
|
|
max_retries: 3
|
|
|
|
playwright:
|
|
headless: true
|
|
|
|
selectors:
|
|
main_content:
|
|
- "main"
|
|
- "article"
|
|
- ".content"
|
|
- ".document"
|
|
|
|
remove:
|
|
- "nav"
|
|
- "footer"
|
|
- "header"
|
|
- ".sidebar"
|
|
- ".toc"
|
|
- "script"
|
|
- "style" |