duyd/crawler-intra-mart
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

Add intra-mart documentation crawler

Browse source 
Playwright-based crawler that renders pages from api.intra-mart.jp and
document.intra-mart.jp, extracts main content, and converts to Markdown
via Pandoc. SQLite-backed queue drives a resumable sequential pipeline
across crawler, extractor, converter, and storage modules.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Do Duy 2026-05-20 16:29:54 +07:00
parent 982391c0f0
commit eb46e739e7
36 changed files with 9366 additions and 84 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

107
.gitignore vendored
View file

@ -1,7 +1,6 @@
# ---> Python
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*.py[codz]
*$py.class
# C extensions
@ -47,7 +46,8 @@ htmlcov/
nosetests.xml
coverage.xml
*.cover
*.py,cover
*.py.cover
*.lcov
.hypothesis/
.pytest_cache/
cover/
@ -93,35 +93,65 @@ ipython_config.py
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# Pipfile.lock
# UV
# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# uv.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# poetry.lock
# poetry.toml
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
# https://pdm-project.org/en/latest/usage/project/#working-with-version-control
# pdm.lock
# pdm.toml
.pdm-python
.pdm-build/
# pixi
# Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
# pixi.lock
# Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
# in the .venv directory. It is recommended not to include this directory in version control.
.pixi/*
!.pixi/config.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat-schedule*
celerybeat.pid
# Redis
*.rdb
*.aof
*.pid
# RabbitMQ
mnesia/
rabbitmq/
rabbitmq-data/
# ActiveMQ
activemq-data/
# SageMath parsed files
*.sage.py
# Environments
.env
.envrc
.venv
env/
venv/
@ -158,31 +188,44 @@ cython_debug/
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
# .idea/
# ---> Java
# Compiled class file
*.class
# Abstra
# Abstra is an AI-powered process automation framework.
# Ignore directories containing user credentials, local state, and settings.
# Learn more at https://abstra.io/docs
.abstra/
# Log file
*.log
# Visual Studio Code
# Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore
# that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
# and can be added to the global gitignore or merged into this file. However, if you prefer,
# you could uncomment the following to ignore the entire vscode folder
# .vscode/
# Temporary file for partial code execution
tempCodeRunnerFile.py
# BlueJ files
*.ctxt
# Ruff stuff:
.ruff_cache/
# Mobile Tools for Java (J2ME)
.mtj.tmp/
# PyPI configuration file
.pypirc
# Package Files #
*.jar
*.war
*.nar
*.ear
*.zip
*.tar.gz
*.rar
# Marimo
marimo/_static/
marimo/_lsp/
__marimo__/
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
hs_err_pid*
replay_pid*
# Streamlit
.streamlit/secrets.toml
# Output directories
output/
data/
# Claude Code local settings
.claude/
# Log archives
logs/*.zip

1
.python-version Normal file
View file

@ -0,0 +1 @@
3.11

83
CLAUDE.md Normal file
View file

@ -0,0 +1,83 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Overview
Documentation crawler that renders pages from `api.intra-mart.jp` and `document.intra-mart.jp` with Playwright, extracts the main content, and converts it to Markdown via Pandoc. Output is structured for Obsidian / MkDocs / Docusaurus / RAG ingestion.
## Commands
```bash
# Install (Python >= 3.11)
pip install -r requirements.txt
playwright install chromium
# pandoc must be on PATH (apt install pandoc / https://pandoc.org/installing.html)
# Run the crawler
python main.py
# Lint / format / typecheck / test (dev extras)
pip install -e ".[dev]"
ruff check .
black .
mypy .
pytest # asyncio_mode=auto is preset
pytest tests/test_x.py::test_name # single test
```
Re-running `python main.py` resumes from the SQLite queue — finished URLs are skipped, `processing` rows are reset to `retry` on startup.
## Architecture
The pipeline lives in `main.py` as a single `DocumentationCrawler` class that orchestrates per-module components. Flow per page:
```
URLManager.normalize/is_allowed
↓
PlaywrightClient.fetch_page (Chromium, JS-rendered, retries via tenacity)
↓
HTMLCleaner.clean (drop REMOVE_SELECTORS)
↓
HTMLExtractor.extract_main_content (first match of CONTENT_SELECTORS)
↓
AssetDownloader (download + rewrite asset links to local paths)
↓
MarkdownConverter.html_to_markdown (Pandoc → GFM, with YAML frontmatter)
↓
FileWriter / PathMapper (output/markdown/<domain>/<sanitized path>.md)
↓
extract_links + SphinxDiscovery + NavigationParser → enqueue new URLs
```
### State is in SQLite, not in memory
`storage/metadata_db.py` (`data/metadata.db`) is the source of truth. Tables:
- `crawl_queue` — drives the loop (`pending` / `processing` / `retry` / `failed` / `done`); `get_next_pending_url` orders by `priority DESC, depth ASC, discovered_at ASC`.
- `pages` — successful crawl metadata + markdown output path.
- `assets`, `page_links`, `discovery_sources`, `failures` — provenance and the site graph.
The crawl loop is **sequential single-worker** even though README mentions concurrency — there is no asyncio worker pool. `BROWSER_RECYCLE_EVERY = 500` restarts Playwright periodically to bound memory.
### Discovery is multi-source
Links are discovered three ways and merged before enqueue: (1) generic `<a href>` extraction in `DocumentationCrawler.extract_links` (also targets Sphinx-specific selectors like `a.reference.internal`, `.toctree-wrapper a`, `link[rel=next/prev/up]`), (2) `SphinxDiscovery` (probes `searchindex.js`, `genindex.html`, etc.), (3) `NavigationParser` (builds a nav tree for `SUMMARY.md` / `navigation_tree.json` export).
### URL → filesystem mapping
`storage/path_mapper.py` produces `<output_dir>/<domain>/<sanitized path>.md`. Sanitizes Windows-invalid chars + reserved names, hashes query strings into a `__<hash>` suffix, and shortens names > 180 chars with an MD5 tail. Asset link rewriting in `extractor/asset_downloader.py` computes paths *relative to the current page's output path*, so don't change `PathMapper` output layout without also revisiting `rewrite_asset_links`.
### Configuration
`config.yaml` exists but **`main.py` does not read it** — the live configuration is the module-level constants at the top of `main.py` (`START_URLS`, `ALLOWED_DOMAINS`, `REMOVE_SELECTORS`, `CONTENT_SELECTORS`, `MAX_PAGES_PER_RUN`, `BROWSER_RECYCLE_EVERY`, `MAX_ATTEMPTS`). Update those, not the YAML.
### Logging
`utils/logger.py` configures loguru on import (just `from utils.logger import logger`). Writes rotating logs to `logs/crawler.log`, `logs/errors.log`, `logs/debug.log` — no extra setup needed.
## Conventions
- Line length 79 (`black` and `ruff` both pinned to this). Target Python 3.11.
- `pytest-asyncio` runs in `auto` mode — async test functions need no decorator.
- Output dirs (`output/`, `data/`, `logs/`) are runtime artifacts; `output/assets/`, `output/markdown/`, `data/` are gitignored.

210
LICENSE
View file

@ -1,73 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices stating that You changed the files; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2026 duyd
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

373
README.md
View file

@ -1,6 +1,373 @@
# crawler-intra-mart
# Intra-Mart Documentation Crawler → Markdown Exporter
Xây dựng một hệ thống crawl tài liệu từ:
## 🚀 Quickstart với [`uv`](https://docs.astral.sh/uv/) (khuyến nghị)
* [https://api.intra-mart.jp/iap/index.html](https://api.intra-mart.jp/iap/index.html)
Project đã được lock vào `uv.lock` + ghim Python qua `.python-version`. `uv` sẽ tự cài đúng phiên bản Python, dựng `.venv`, và sync deps từ lock file — không cần `pip`, không cần `pyenv`.
### 1. Cài `uv`
| OS | Lệnh |
| ------------------- | --------------------------------------------------------------------------------- |
| **macOS / Linux** | `curl -LsSf https://astral.sh/uv/install.sh \| sh` |
| **Windows (PowerShell)** | `powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 \| iex"` |
| **macOS (Homebrew)** | `brew install uv` |
| **Bất kỳ OS nào có pipx** | `pipx install uv` |
Xác nhận đã cài: `uv --version`.
### 2. Cài Pandoc (bắt buộc, không nằm trong `uv`)
`MarkdownConverter` gọi binary `pandoc` qua subprocess, nên Pandoc phải có sẵn trên `PATH`.
| OS | Lệnh |
| ------------------------ | --------------------------------------------------- |
| **macOS (Homebrew)** | `brew install pandoc` |
| **Ubuntu / Debian** | `sudo apt install pandoc` |
| **Fedora / RHEL** | `sudo dnf install pandoc` |
| **Arch** | `sudo pacman -S pandoc` |
| **Windows (winget)** | `winget install --id JohnMacFarlane.Pandoc` |
| **Windows (Chocolatey)** | `choco install pandoc` |
| **Khác** | Tải installer tại <https://pandoc.org/installing.html> |
Xác nhận: `pandoc --version`.
### 3. Clone & sync deps
```bash
git clone https://github.com/IricsDo/intra_mart_doc_crawler.git
cd intra_mart_doc_crawler
uv sync # tạo .venv, cài đúng Python 3.11 + toàn bộ deps từ uv.lock
uv run playwright install chromium # tải Chromium cho Playwright
```
> Trên **Linux**, có thể `playwright` báo thiếu system lib (libnss3, libatk, …). Fix:
> `uv run playwright install-deps chromium` (cần sudo, dùng apt). Trên **macOS/Windows** thường không cần.
### 4. Chạy crawler
```bash
uv run python main.py
```
Hoặc activate venv rồi chạy như Python thường:
```bash
# macOS / Linux
source .venv/bin/activate
python main.py
# Windows (PowerShell)
.venv\Scripts\Activate.ps1
python main.py
# Windows (cmd.exe)
.venv\Scripts\activate.bat
python main.py
```
Chạy lại = **resume** từ SQLite queue (`data/metadata.db`). URL đã `done` bị skip.
### 5. Dev tools (lint / format / type-check / test)
```bash
uv sync --extra dev # cài thêm black, ruff, mypy, pytest
uv run ruff check .
uv run black .
uv run mypy .
uv run pytest # asyncio_mode = auto đã preset trong pyproject.toml
```
### Lệnh `uv` thường dùng
| Tác dụng | Lệnh |
| ------------------------------------- | ------------------------------------- |
| Sync env đúng theo `uv.lock` | `uv sync` |
| Sync + cài optional dep `dev` | `uv sync --extra dev` |
| Thêm 1 package mới | `uv add <package>` |
| Bỏ 1 package | `uv remove <package>` |
| Update lock theo `pyproject.toml` | `uv lock` |
| Upgrade 1 package | `uv lock --upgrade-package <package>` |
| Chạy lệnh trong venv mà không activate | `uv run <command>` |
> ℹ️ Vẫn dùng `pip` được — xem mục **Cài đặt môi trường (pip)** ở dưới. Nhưng `uv` nhanh hơn 10–100× và đảm bảo cùng phiên bản deps trên mọi máy nhờ `uv.lock`.
---
## Mục tiêu project
Crawl tài liệu từ:
* [https://api.intra-mart.jp/iap/](https://api.intra-mart.jp/iap/)
* [https://document.intra-mart.jp/library/](https://document.intra-mart.jp/library/)
rồi convert toàn bộ sang Markdown có:
* giữ nguyên hierarchy theo URL path
* giữ internal links (rewrite sang đường dẫn local)
* tải assets (ảnh, css, file đính kèm) về local
* hỗ trợ JS-rendered docs (Playwright + Chromium)
* resume được khi chạy lại (state nằm trong SQLite)
* output sạch để dùng với:
* Obsidian
* MkDocs
* Docusaurus
* RAG / LLM
* GitBook
---
## Kiến trúc tổng thể
Toàn bộ pipeline nằm trong `main.py` (`DocumentationCrawler`), điều phối các module con. Flow mỗi page:
```text
URLManager.normalize / is_allowed
↓
PlaywrightClient.fetch_page (Chromium, JS-rendered, retry qua tenacity)
↓
HTMLCleaner.clean (loại REMOVE_SELECTORS)
↓
HTMLExtractor.extract_main_content (match CONTENT_SELECTORS đầu tiên)
↓
AssetDownloader (download + rewrite asset link sang local)
↓
MarkdownConverter.html_to_markdown (Pandoc → GFM, kèm YAML frontmatter)
↓
FileWriter / PathMapper (output/markdown/<domain>/<sanitized path>.md)
↓
extract_links + SphinxDiscovery + NavigationParser → enqueue URL mới
```
### State nằm ở SQLite, không phải in-memory
`storage/metadata_db.py` (file `data/metadata.db`) là source of truth.
Bảng chính:
* `crawl_queue` — drive vòng lặp crawl: `pending` / `processing` / `retry` / `failed` / `done`. Lấy URL kế tiếp theo `priority DESC, depth ASC, discovered_at ASC`.
* `pages` — metadata page crawl thành công + đường dẫn markdown output.
* `assets`, `page_links`, `discovery_sources`, `failures` — provenance + site graph.
Chạy lại `python main.py` sẽ resume: URL `done` bị bỏ qua, URL đang `processing` bị reset về `retry` khi khởi động.
> ⚠️ Crawl loop hiện chạy **sequential single-worker** (chưa có asyncio worker pool). Để giới hạn memory, Playwright được restart mỗi `BROWSER_RECYCLE_EVERY = 500` page.
### Discovery đa nguồn
URL được phát hiện từ 3 nguồn, merge lại trước khi enqueue:
1. `DocumentationCrawler.extract_links` — generic `<a href>` + selector đặc thù Sphinx (`a.reference.internal`, `.toctree-wrapper a`, `link[rel=next/prev/up]`).
2. `SphinxDiscovery` — probe `searchindex.js`, `genindex.html`, …
3. `NavigationParser` — dựng nav tree để export `SUMMARY.md` / `navigation_tree.json`.
### URL → filesystem mapping
`storage/path_mapper.py` map sang `<output_dir>/<domain>/<sanitized path>.md`:
* sanitize ký tự Windows-invalid + reserved names
* hash query string thành suffix `__<hash>`
* rút gọn tên > 180 ký tự bằng MD5 tail
Asset link rewrite trong `extractor/asset_downloader.py` tính path **tương đối với output path của page hiện tại** — nếu đổi layout output của `PathMapper`, phải sửa luôn `rewrite_asset_links`.
---
## Công nghệ sử dụng
| Thành phần | Công nghệ |
| ------------------- | ------------------ |
| Rendering | Playwright (Chromium) |
| HTTP fast-path | httpx |
| HTML Parse | BeautifulSoup4 + lxml |
| URL handling | urllib.parse |
| Markdown conversion | Pandoc (qua pypandoc / markdownify) |
| State & queue | SQLite |
| Logging | loguru |
| Retry | tenacity |
| Config | constants ở đầu `main.py` (xem mục Configuration) |
---
## Cấu trúc project
```text
intra-mart-crawler/
│
├── README.md
├── CLAUDE.md
├── requirements.txt
├── pyproject.toml
├── config.yaml ← KHÔNG được main.py đọc (xem Configuration)
├── main.py ← entrypoint + DocumentationCrawler
│
├── crawler/
│ ├── crawler.py
│ ├── http_client.py
│ ├── playwright_client.py
│ ├── url_manager.py
│ ├── sitemap_parser.py
│ ├── sphinx_discovery.py
│ └── navigation_parser.py
│
├── extractor/
│ ├── html_extractor.py
│ ├── html_cleaner.py
│ └── asset_downloader.py
│
├── converter/
│ ├── markdown_converter.py
│ └── link_rewriter.py
│
├── storage/
│ ├── file_writer.py
│ ├── metadata_db.py ← SQLite source of truth
│ └── path_mapper.py
│
├── models/
│ ├── page.py
│ └── crawl_result.py
│
├── utils/
│ ├── logger.py
│ ├── retry.py
│ └── helpers.py
│
├── output/ ← runtime artifact (gitignored)
│ ├── markdown/
│ └── assets/
│
├── data/ ← runtime artifact (gitignored)
│ └── metadata.db
│
└── logs/ ← rotating logs (gitignored)
├── crawler.log
├── errors.log
└── debug.log
```
---
## Cài đặt môi trường (pip — cách cũ)
> Nếu bạn dùng `uv` thì có thể bỏ qua section này — xem **Quickstart với `uv`** ở đầu file.
Yêu cầu: **Python >= 3.11** và **Pandoc** có trên PATH.
```bash
# 1. Python deps
pip install -r requirements.txt
# 2. Browser cho Playwright
playwright install chromium
# 3. Pandoc
sudo apt install pandoc # Ubuntu/Debian
# macOS: brew install pandoc
# Windows: winget install --id JohnMacFarlane.Pandoc
# hoặc https://pandoc.org/installing.html
```
### Dev environment (lint / format / type-check / test)
```bash
pip install -e ".[dev]"
ruff check .
black .
mypy .
pytest # asyncio_mode = auto đã preset
pytest tests/test_x.py::test_name # chạy 1 test
```
Line length pinned ở **79** cho cả `black` và `ruff`. Target Python 3.11.
---
## Configuration
> ⚠️ `config.yaml` **không được `main.py` đọc**. Đây là tàn dư từ thiết kế cũ — sẽ refactor sau.
Cấu hình "live" là các hằng số ở đầu `main.py`:
| Constant | Ý nghĩa |
| ----------------------- | ------------------------------------------------------- |
| `START_URLS` | Seed URLs để crawl |
| `ALLOWED_DOMAINS` | Whitelist domain — ngoài list này sẽ không enqueue |
| `OUTPUT_DIR` | Thư mục output gốc (mặc định `output/`) |
| `REMOVE_SELECTORS` | CSS selectors bị drop trước khi extract |
| `CONTENT_SELECTORS` | Selectors thử lần lượt để lấy main content |
| `MAX_PAGES_PER_RUN` | Số page tối đa mỗi lần chạy |
| `MAX_ATTEMPTS` | Số lần retry trước khi mark `failed` |
| `BROWSER_RECYCLE_EVERY` | Restart Chromium sau mỗi N page (giới hạn memory leak) |
Muốn đổi behaviour → sửa các constant này, **không phải `config.yaml`**.
---
## Chạy project
```bash
python main.py
```
Chạy lại lệnh trên = resume từ SQLite queue. URL đã `done` bị skip, URL `processing` reset về `retry`.
---
## Output ví dụ
```text
output/
├── markdown/
│ ├── api.intra-mart.jp/
│ │ └── iap/
│ │ ├── index.md
│ │ ├── javadoc/
│ │ │ └── ...
│ │ └── apilist-ssjs/
│ │ └── ...
│ │
│ └── document.intra-mart.jp/
│ └── library/
│ ├── getting-started.md
│ └── workflow.md
│
└── assets/
└── ... (ảnh, css, file đính kèm)
```
Mỗi file `.md` có YAML frontmatter (source URL, crawl time, ...) và đã rewrite internal link + asset link sang relative path.
Ngoài ra:
* `SUMMARY.md` — index theo nav tree (cho GitBook / mdBook).
* `navigation_tree.json` — nav tree dạng JSON (cho MkDocs / Docusaurus / custom sidebar).
---
## Logging
`utils/logger.py` cấu hình loguru ngay khi import (chỉ cần `from utils.logger import logger`). Ghi rotating log vào `logs/crawler.log`, `logs/errors.log`, `logs/debug.log` — không cần setup gì thêm.
---
## Hướng phát triển tiếp theo
### Trong scope crawler
1. **Async concurrent crawling** — hiện crawl sequential, có thể thêm asyncio worker pool + semaphore + rate limiting.
2. **Better extraction fallback** — bổ sung `trafilatura` / `readability-lxml` cho page không match `CONTENT_SELECTORS` nào.
3. **Duplicate detection** — hash HTML/markdown content để skip page trùng nội dung khác URL.
4. **Đọc `config.yaml` thực sự** (hoặc bỏ hẳn file này) — đồng bộ với constants trong `main.py`.
5. **Incremental update** — re-crawl chỉ những page có `ETag` / `Last-Modified` thay đổi.
### Hệ sinh thái sau crawler
1. Build MkDocs / Docusaurus site từ output.
2. Build vector database + embedding docs.
3. Tạo RAG chatbot trên docs đã embed.
4. Semantic search UI.
5. Git sync pipeline cho output markdown.

34
config.yaml Normal file
View file

@ -0,0 +1,34 @@
seed_urls:
- "https://api.intra-mart.jp/iap/index.html"
- "https://document.intra-mart.jp/library/"
allowed_domains:
- "api.intra-mart.jp"
- "document.intra-mart.jp"
output_dir: "output"
crawl:
concurrency: 5
delay_seconds: 1
timeout: 30000
max_retries: 3
playwright:
headless: true
selectors:
main_content:
- "main"
- "article"
- ".content"
- ".document"
remove:
- "nav"
- "footer"
- "header"
- ".sidebar"
- ".toc"
- "script"
- "style"

0
converter/__init__.py Normal file
View file

201
converter/link_rewriter.py Normal file
View file

@ -0,0 +1,201 @@
from pathlib import Path
from urllib.parse import urlparse
from bs4 import BeautifulSoup
class LinkRewriter:
"""
Rewrite internal intra-mart links
from online URLs -> local markdown paths
Example:
https://api.intra-mart.jp/iap/auth/login.html
=>
../auth/login.md
"""
def __init__(self, output_dir: str):
self.output_dir = Path(output_dir)
def rewrite(
self,
markdown: str,
current_url: str,
current_output_path: Path
):
"""
Rewrite markdown links
Parameters
----------
markdown : str
markdown content
current_url : str
original page url
current_output_path : Path
local markdown file path
"""
lines = markdown.splitlines()
rewritten_lines = []
for line in lines:
rewritten = self.rewrite_markdown_links(
line,
current_output_path
)
rewritten_lines.append(rewritten)
return "\n".join(rewritten_lines)
def rewrite_markdown_links(
self,
line: str,
current_output_path: Path
):
"""
Rewrite markdown inline links
Example:
[API](https://api.intra-mart.jp/iap/auth.html)
->
[API](../auth.md)
"""
import re
pattern = r"\[([^\]]+)\]\(([^)]+)\)"
matches = re.findall(pattern, line)
if not matches:
return line
rewritten_line = line
for text, url in matches:
if not self.is_internal_url(url):
continue
local_path = self.url_to_markdown_path(url)
relative_path = self.make_relative_path(
current_output_path,
local_path
)
old = f"[{text}]({url})"
new = f"[{text}]({relative_path})"
rewritten_line = rewritten_line.replace(
old,
new
)
return rewritten_line
def is_internal_url(self, url: str):
"""
Check if URL belongs to intra-mart docs
"""
parsed = urlparse(url)
allowed_domains = [
"api.intra-mart.jp",
"document.intra-mart.jp"
]
return any(
domain in parsed.netloc
for domain in allowed_domains
)
def url_to_markdown_path(self, url: str):
"""
Convert URL -> local markdown path
Example:
https://api.intra-mart.jp/iap/auth/login.html
=>
output/markdown/iap/auth/login.md
"""
parsed = urlparse(url)
path = parsed.path.strip("/")
if not path:
path = "index"
if path.endswith(".html"):
path = path[:-5]
local_path = (
self.output_dir /
(path + ".md")
)
return local_path
def make_relative_path(
self,
current_file: Path,
target_file: Path
):
"""
Generate relative markdown path
Example:
current:
output/markdown/iap/index.md
target:
output/markdown/iap/auth/login.md
=>
auth/login.md
"""
current_dir = current_file.parent
relative = target_file.relative_to(
self.output_dir
)
target_absolute = self.output_dir / relative
relative_path = Path(
target_absolute.relative_to(current_dir)
)
try:
relative_path = target_absolute.relative_to(
current_dir
)
return str(relative_path).replace("\\", "/")
except Exception:
import os
relative_path = os.path.relpath(
target_absolute,
current_dir
)
return relative_path.replace("\\", "/")

460
converter/markdown_converter.py Normal file
View file

@ -0,0 +1,460 @@
import shutil
import tempfile
import subprocess
from pathlib import Path
from bs4 import BeautifulSoup
try:
from markdownify import markdownify as _markdownify
except ImportError: # pragma: no cover - optional dep gate
_markdownify = None
from utils.logger import logger
class MarkdownConverter:
"""
HTML -> Markdown converter.
Two engines are available:
- ``markdownify`` (default): pure Python, no subprocess, ~10x
faster per page. Fidelity is good for documentation HTML.
- ``pandoc``: spawns the pandoc binary; higher fidelity for
pathological HTML but pays a process-start cost per page.
"""
def __init__(
self,
engine="markdownify",
pandoc_path="pandoc",
timeout=120,
media_dir="output/assets/pandoc"
):
self.engine = engine
self.pandoc_path = pandoc_path
self.timeout = timeout
self.media_dir = Path(media_dir)
self.media_dir.mkdir(parents=True, exist_ok=True)
if self.engine == "pandoc":
self.validate_pandoc()
elif self.engine == "markdownify":
if _markdownify is None:
raise RuntimeError(
"markdownify is not installed. "
"pip install markdownify, or pass "
"engine='pandoc'."
)
else:
raise ValueError(
f"Unknown markdown engine: {self.engine!r}"
)
def validate_pandoc(self):
"""
Ensure pandoc exists.
"""
if not shutil.which(
self.pandoc_path
):
raise RuntimeError(
"Pandoc not found. "
"Please install pandoc."
)
def html_to_markdown(
self,
html: str,
title=None,
source_url=None,
media_dir=None
):
"""
Convert HTML -> Markdown using the configured engine.
"""
html = self.preprocess_html(html)
if self.engine == "markdownify":
markdown = self._convert_markdownify(html)
else:
markdown = self._convert_pandoc(html, media_dir)
markdown = self.postprocess_markdown(markdown)
markdown = self.add_frontmatter(
markdown,
title=title,
source_url=source_url,
)
logger.debug("Markdown conversion completed.")
return markdown
def _convert_markdownify(self, html: str) -> str:
"""
In-process HTML -> Markdown via the markdownify
library. Roughly 10x faster per page than spawning
pandoc.
"""
def code_lang_cb(el):
# HTMLCleaner.normalize_code_blocks already sets
# data-language on <pre> when it can detect one.
return el.get("data-language") or ""
try:
return _markdownify(
html,
heading_style="ATX",
code_language_callback=code_lang_cb,
bullets="-",
strip=["script", "style"],
)
except Exception as e:
logger.exception(f"markdownify failed: {e}")
raise
def _convert_pandoc(self, html: str, media_dir) -> str:
"""
Fallback engine: shell out to pandoc.
"""
temp_html_path = None
temp_md_path = None
try:
with tempfile.NamedTemporaryFile(
suffix=".html",
mode="w",
encoding="utf-8",
delete=False,
) as temp_html:
temp_html.write(html)
temp_html_path = temp_html.name
temp_md_path = temp_html_path + ".md"
extract_media_dir = (
Path(media_dir) if media_dir else self.media_dir
)
extract_media_dir.mkdir(parents=True, exist_ok=True)
command = [
self.pandoc_path,
temp_html_path,
"-f", "html",
"-t", "gfm",
"--wrap=none",
"--markdown-headings=atx",
f"--extract-media={extract_media_dir}",
"-o", temp_md_path,
]
result = subprocess.run(
command,
capture_output=True,
text=True,
timeout=self.timeout,
)
if result.returncode != 0:
logger.error(f"Pandoc failed: {result.stderr}")
raise RuntimeError(result.stderr)
with open(
temp_md_path,
"r",
encoding="utf-8",
) as f:
return f.read()
except subprocess.TimeoutExpired:
logger.error("Pandoc conversion timeout.")
raise
finally:
self.cleanup_temp_file(temp_html_path)
self.cleanup_temp_file(temp_md_path)
def preprocess_html(
self,
html: str
):
"""
Clean HTML before conversion.
"""
soup = BeautifulSoup(
html,
"lxml"
)
# Remove scripts/styles
for tag in soup.find_all(
[
"script",
"style",
"noscript"
]
):
tag.decompose()
# Normalize code blocks
for pre in soup.find_all("pre"):
code = pre.find("code")
if not code:
continue
classes = code.get(
"class",
[]
)
for cls in classes:
if cls.startswith(
"language-"
):
lang = cls.replace(
"language-",
""
)
pre["data-language"] = lang
return str(soup)
def postprocess_markdown(
self,
markdown: str
):
"""
Normalize markdown output.
"""
lines = markdown.splitlines()
cleaned = []
previous_empty = False
for line in lines:
stripped = line.rstrip()
# Collapse excessive empty lines
if not stripped:
if previous_empty:
continue
previous_empty = True
else:
previous_empty = False
cleaned.append(
stripped
)
markdown = "\n".join(cleaned)
markdown = self.fix_code_fences(
markdown
)
markdown = self.fix_tables(
markdown
)
return markdown.strip()
def fix_code_fences(
self,
markdown: str
):
"""
Improve fenced code blocks.
"""
lines = markdown.splitlines()
output = []
in_code = False
for line in lines:
if line.startswith("```"):
in_code = not in_code
output.append(line)
# Close unclosed code block
if in_code:
output.append("```")
return "\n".join(output)
def fix_tables(
self,
markdown: str
):
"""
Fix malformed markdown tables.
"""
# Future enhancement hook
return markdown
def add_frontmatter(
self,
markdown,
title=None,
source_url=None
):
"""
Add YAML frontmatter.
Values are escaped so that titles or URLs containing
quotes / backslashes / newlines do not produce
broken YAML.
"""
metadata = []
if title:
metadata.append(
f'title: {self._yaml_quote(title)}'
)
if source_url:
metadata.append(
f'source_url: {self._yaml_quote(source_url)}'
)
if not metadata:
return markdown
frontmatter = "---\n"
frontmatter += "\n".join(
metadata
)
frontmatter += "\n---\n\n"
return frontmatter + markdown
def _yaml_quote(
self,
value
):
"""
Safely double-quote a value for YAML frontmatter.
Escapes backslashes, double quotes and control chars
(newline / carriage return / tab) which are otherwise
illegal inside a double-quoted YAML scalar.
"""
text = str(value)
text = (
text
.replace("\\", "\\\\")
.replace('"', '\\"')
.replace("\n", "\\n")
.replace("\r", "\\r")
.replace("\t", "\\t")
)
return f'"{text}"'
def cleanup_temp_file(
self,
path
):
"""
Safely remove temp file.
"""
if not path:
return
try:
Path(path).unlink(
missing_ok=True
)
except Exception as e:
logger.warning(
f"Temp cleanup failed "
f"{path}: {e}"
)
def batch_convert(
self,
html_documents
):
"""
Batch conversion helper.
html_documents:
[
{
"html": "...",
"title": "...",
"url": "..."
}
]
"""
results = []
for doc in html_documents:
try:
markdown = self.html_to_markdown(
html=doc["html"],
title=doc.get("title"),
source_url=doc.get("url")
)
results.append({
"success": True,
"markdown": markdown
})
except Exception as e:
results.append({
"success": False,
"error": str(e)
})
return results

0
crawler/__init__.py Normal file
View file

248
crawler/crawler.py Normal file
View file

@ -0,0 +1,248 @@
import asyncio
from typing import Set
from bs4 import BeautifulSoup
from crawler.playwright_client import PlaywrightClient
from crawler.url_manager import URLManager
from extractor.html_extractor import HTMLExtractor
from extractor.html_cleaner import HTMLCleaner
from converter.markdown_converter import MarkdownConverter
from converter.path_mapper import PathMapper
from storage.file_writer import FileWriter
from utils.logger import logger
from converter.link_rewriter import LinkRewriter
from extractor.asset_downloader import AssetDownloader
class DocumentationCrawler:
def __init__(self, config):
self.config = config
self.client = PlaywrightClient(
timeout=config["crawl"]["timeout"],
headless=config["playwright"]["headless"]
)
self.url_manager = URLManager(
config["allowed_domains"]
)
self.extractor = HTMLExtractor(
config["selectors"]["main_content"]
)
self.cleaner = HTMLCleaner(
config["selectors"]["remove"]
)
self.converter = MarkdownConverter()
self.mapper = PathMapper(
config["output_dir"] + "/markdown"
)
self.writer = FileWriter()
self.link_rewriter = LinkRewriter(
config["output_dir"] + "/markdown"
)
self.asset_downloader = AssetDownloader(
output_dir=config["output_dir"] + "/assets"
)
self.delay_seconds = config["crawl"]["delay_seconds"]
self.max_retries = config["crawl"]["max_retries"]
async def crawl(self):
"""
Main crawling entry point
"""
logger.info("Starting crawler...")
await self.client.start()
queue = asyncio.Queue()
for url in self.config["seed_urls"]:
await queue.put(url)
try:
while not queue.empty():
current_url = await queue.get()
current_url = self.url_manager.normalize(current_url)
if not self.url_manager.should_visit(current_url):
continue
self.url_manager.mark_visited(current_url)
logger.info(f"Crawling: {current_url}")
try:
await self.process_page(current_url, queue)
except Exception as e:
logger.exception(
f"Failed processing {current_url}: {e}"
)
await asyncio.sleep(self.delay_seconds)
finally:
await self.client.stop()
logger.info("Crawler stopped.")
async def process_page(
self,
url: str,
queue: asyncio.Queue
):
"""
Process single page:
- render html
- extract content
- clean html
- convert markdown
- save markdown
- discover links
"""
page_data = await self.fetch_with_retry(url)
if not page_data:
return
html = page_data["html"]
await self.asset_downloader.download_assets_from_html(
base_url=url,
html=html
)
logger.info(f"Extracting content: {url}")
main_html = self.extractor.extract_main_content(html)
clean_html = self.cleaner.clean(main_html)
logger.info(f"Converting markdown: {url}")
markdown = self.converter.html_to_markdown(
clean_html
)
output_path = self.mapper.url_to_path(url)
markdown = self.link_rewriter.rewrite(
markdown=markdown,
current_url=url,
current_output_path=output_path
)
self.writer.write_text(
output_path,
markdown
)
logger.info(f"Saved markdown: {output_path}")
await self.discover_links(
base_url=url,
html=html,
queue=queue
)
async def fetch_with_retry(
self,
url: str
):
"""
Retry wrapper for page fetching
"""
for attempt in range(1, self.max_retries + 1):
try:
logger.info(
f"Fetching ({attempt}/{self.max_retries}): {url}"
)
return await self.client.fetch_page(url)
except Exception as e:
logger.warning(
f"Retry {attempt} failed for {url}: {e}"
)
await asyncio.sleep(2)
logger.error(f"Max retries exceeded: {url}")
return None
async def discover_links(
self,
base_url: str,
html: str,
queue: asyncio.Queue
):
"""
Extract all links from page
and push valid links into queue
"""
soup = BeautifulSoup(html, "lxml")
links_found = 0
for a in soup.find_all("a", href=True):
href = a["href"].strip()
if not href:
continue
if href.startswith("#"):
continue
if href.startswith("mailto:"):
continue
if href.startswith("javascript:"):
continue
try:
next_url = self.url_manager.resolve(
base_url,
href
)
next_url = self.url_manager.normalize(
next_url
)
if self.url_manager.should_visit(next_url):
await queue.put(next_url)
links_found += 1
except Exception as e:
logger.warning(
f"Invalid URL {href} from {base_url}: {e}"
)
logger.info(
f"Discovered {links_found} new links from {base_url}"
)

157
crawler/http_client.py Normal file
View file

@ -0,0 +1,157 @@
import httpx
from utils.logger import logger
from crawler.playwright_client import is_binary_file
def _http2_available() -> bool:
"""
httpx requires the optional 'h2' package for HTTP/2. Detect
its presence so we transparently fall back to HTTP/1.1 +
keep-alive when h2 isn't installed (no hard dependency).
"""
try:
import h2 # noqa: F401
return True
except ImportError:
return False
class HTTPClient:
"""
Lightweight HTTP fetcher used as the fast-path for
static documentation pages. Returns a dict shaped like
PlaywrightClient.fetch_page so process_page can consume
either source.
"""
USER_AGENT = (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0.0.0 Safari/537.36"
)
def __init__(
self,
timeout=20,
max_keepalive=20,
max_connections=40,
):
self.timeout = timeout
self.client = httpx.AsyncClient(
timeout=timeout,
follow_redirects=True,
http2=_http2_available(),
headers={"User-Agent": self.USER_AGENT},
limits=httpx.Limits(
max_connections=max_connections,
max_keepalive_connections=max_keepalive,
),
)
async def aclose(self):
await self.client.aclose()
async def fetch_page(
self,
url: str,
etag: str | None = None,
last_modified: str | None = None,
):
if is_binary_file(url):
return {
"url": url,
"title": url.split("/")[-1],
"html": "",
"status_code": None,
"headers": {},
"is_binary": True,
"content_type": "",
}
# Build conditional-GET headers when we have a prior
# ETag / Last-Modified for this URL. A 304 response
# lets the caller skip re-fetching, re-parsing, and
# re-writing markdown — pure savings on resumed runs.
headers = {}
if etag:
headers["If-None-Match"] = etag
if last_modified:
headers["If-Modified-Since"] = last_modified
try:
response = await self.client.get(url, headers=headers)
except Exception as e:
logger.warning(f"HTTP fetch failed {url}: {e}")
return None
content_type = response.headers.get("content-type", "")
if response.status_code == 304:
return {
"url": str(response.url),
"title": "",
"html": "",
"status_code": 304,
"headers": dict(response.headers),
"is_binary": False,
"content_type": content_type,
"not_modified": True,
}
if response.status_code >= 400:
return {
"url": str(response.url),
"title": "",
"html": "",
"status_code": response.status_code,
"headers": dict(response.headers),
"is_binary": False,
"content_type": content_type,
"ok": False,
}
if "html" not in content_type.lower():
# Treat non-HTML responses (e.g. application/pdf,
# application/javascript) as binary so the caller
# skips parsing them as documentation pages.
return {
"url": str(response.url),
"title": url.split("/")[-1],
"html": "",
"status_code": response.status_code,
"headers": dict(response.headers),
"is_binary": True,
"content_type": content_type,
}
html = response.text
title = self._extract_title(html)
return {
"url": str(response.url),
"title": title,
"html": html,
"status_code": response.status_code,
"headers": dict(response.headers),
"is_binary": False,
"content_type": content_type,
"ok": True,
"etag": response.headers.get("etag"),
"last_modified": response.headers.get("last-modified"),
}
@staticmethod
def _extract_title(html: str) -> str:
# Avoid full BS4 parse for just a title — the next
# stage will parse the HTML anyway.
lower = html.lower()
start = lower.find("<title")
if start < 0:
return ""
gt = lower.find(">", start)
end = lower.find("</title", gt)
if gt < 0 or end < 0:
return ""
return html[gt + 1 : end].strip()

557
crawler/navigation_parser.py Normal file
View file

@ -0,0 +1,557 @@
import json
from urllib.parse import urljoin
from bs4 import BeautifulSoup
from utils.logger import logger
def _ensure_soup(html_or_soup):
if hasattr(html_or_soup, "select_one") and hasattr(
html_or_soup, "find_all"
):
return html_or_soup
return BeautifulSoup(html_or_soup, "lxml")
class NavigationParser:
"""
Parse documentation navigation/sidebar
to preserve hierarchy structure.
Supports:
- sidebar menus
- nested navigation trees
- table of contents
- navigation JSON
"""
def __init__(self, url_manager=None):
self.discovered_urls = set()
self.url_manager = url_manager
def parse_navigation(
self,
base_url: str,
html
):
"""
Main entry point. Accepts an HTML string or a parsed
BeautifulSoup/Tag.
Returns:
-------
{
"urls": [...],
"tree": [...]
}
"""
soup = _ensure_soup(html)
urls = set()
tree = []
# Parse sidebar menus
sidebar_tree = self.parse_sidebar_navigation(
soup,
base_url
)
tree.extend(sidebar_tree)
urls.update(
self.extract_urls_from_tree(sidebar_tree)
)
sphinx_tree = self.parse_sphinx_toctree(
soup,
base_url
)
tree.extend(sphinx_tree)
urls.update(
self.extract_urls_from_tree(sphinx_tree)
)
# Parse TOC
toc_tree = self.parse_toc_navigation(
soup,
base_url
)
tree.extend(toc_tree)
urls.update(
self.extract_urls_from_tree(toc_tree)
)
breadcrumb_tree = self.parse_breadcrumbs(
soup,
base_url
)
tree.extend(breadcrumb_tree)
urls.update(
self.extract_urls_from_tree(breadcrumb_tree)
)
# Parse generic nav blocks
nav_tree = self.parse_generic_navigation(
soup,
base_url
)
tree.extend(nav_tree)
urls.update(
self.extract_urls_from_tree(nav_tree)
)
logger.info(
f"Navigation parser found {len(urls)} URLs"
)
tree = self.dedupe_tree(tree)
return {
"urls": list(urls),
"tree": tree
}
def parse_sidebar_navigation(
self,
soup,
base_url
):
"""
Parse sidebar-based docs navigation
"""
selectors = [
"nav",
".sidebar",
".navigation",
".menu",
".toc",
".md-nav",
".wy-menu",
".theme-doc-sidebar-container",
".theme-doc-sidebar-menu",
".sphinxsidebar",
".sphinxsidebarwrapper",
".toctree-wrapper",
".globaltoc",
".localtoc",
".relations",
".related",
".body .toctree-wrapper",
".document .toctree-wrapper",
".contents.topic"
]
results = []
for selector in selectors:
nodes = soup.select(selector)
for node in nodes:
tree = self.parse_list_structure(
node,
base_url
)
if tree:
results.extend(tree)
return results
def parse_toc_navigation(
self,
soup,
base_url
):
"""
Parse table-of-contents sections
"""
selectors = [
".toc",
".table-of-contents",
"#table-of-contents",
".contents",
".markdown-toc"
]
results = []
for selector in selectors:
nodes = soup.select(selector)
for node in nodes:
tree = self.parse_list_structure(
node,
base_url
)
if tree:
results.extend(tree)
return results
def parse_generic_navigation(
self,
soup,
base_url
):
"""
Fallback parser:
extract all nav-like links
"""
results = []
for a in soup.find_all("a", href=True):
href = a["href"].strip()
title = a.get_text(" ", strip=True)
if not title:
continue
if not href:
continue
if href.startswith("#"):
continue
if href.startswith("javascript:"):
continue
if href.startswith("mailto:"):
continue
full_url = self.resolve_url(base_url, href)
if not full_url:
continue
item = {
"title": title,
"url": full_url,
"children": []
}
results.append(item)
return results
def parse_list_structure(
self,
node,
base_url
):
"""
Parse nested UL/LI navigation tree
"""
results = []
root_lists = node.find_all(
["ul", "ol"],
recursive=False
)
if not root_lists:
root_lists = [node]
for ul in root_lists:
for li in ul.find_all("li", recursive=False):
item = self.parse_list_item(
li,
base_url
)
if item:
results.append(item)
return results
def parse_list_item(
self,
li,
base_url
):
"""
Parse single navigation item
"""
a = li.find("a", href=True)
if not a:
return None
href = a["href"].strip()
if not href:
return None
full_url = self.resolve_url(base_url, href)
if not full_url:
return None
title = a.get_text(" ", strip=True)
if not title:
title = full_url.rsplit("/", 1)[-1]
item = {
"title": title,
"url": full_url,
"children": []
}
nested_lists = li.find_all(
["ul", "ol"],
recursive=False
)
for nested in nested_lists:
for child_li in nested.find_all(
"li",
recursive=False
):
child_item = self.parse_list_item(
child_li,
base_url
)
if child_item:
item["children"].append(
child_item
)
return item
def extract_urls_from_tree(
self,
tree
):
"""
Flatten navigation tree into URL list
"""
urls = set()
for item in tree:
url = item.get("url")
if url:
urls.add(url)
children = item.get("children", [])
child_urls = self.extract_urls_from_tree(
children
)
urls.update(child_urls)
return urls
def export_tree_json(
self,
tree,
output_path
):
"""
Export navigation hierarchy to JSON
"""
with open(
output_path,
"w",
encoding="utf-8"
) as f:
json.dump(
tree,
f,
ensure_ascii=False,
indent=2
)
logger.info(
f"Navigation tree saved: {output_path}"
)
def generate_summary_markdown(
self,
tree
):
"""
Generate SUMMARY.md
for mdBook / docs navigation
"""
lines = ["# Summary", ""]
self.build_summary_lines(
tree,
lines,
level=0
)
return "\n".join(lines)
def build_summary_lines(
self,
tree,
lines,
level=0
):
"""
Recursive markdown tree builder
"""
indent = " " * level
for item in tree:
title = item.get("title", "Untitled")
url = item.get("url", "#")
lines.append(
f"{indent}- [{title}]({url})"
)
children = item.get("children", [])
if children:
self.build_summary_lines(
children,
lines,
level + 1
)
def resolve_url(self, base_url, href):
if not href:
return None
href = href.strip()
if href.startswith("#"):
return None
if href.startswith(("javascript:", "mailto:", "tel:")):
return None
if self.url_manager:
resolved = self.url_manager.resolve(base_url, href)
if not resolved:
return None
if not self.url_manager.is_allowed(resolved):
return None
return resolved
return urljoin(base_url, href)
def parse_sphinx_toctree(
self,
soup,
base_url
):
selectors = [
".toctree-wrapper",
".sphinxsidebar",
".sphinxsidebarwrapper",
".globaltoc",
".localtoc",
".contents.topic"
]
results = []
for selector in selectors:
for node in soup.select(selector):
tree = self.parse_list_structure(node, base_url)
if tree:
results.extend(tree)
return results
def dedupe_tree(self, tree):
seen = set()
result = []
for item in tree:
url = item.get("url")
if url and url in seen:
continue
if url:
seen.add(url)
children = item.get("children", [])
item["children"] = self.dedupe_tree(children)
result.append(item)
return result
def parse_breadcrumbs(
self,
soup,
base_url
):
selectors = [
".breadcrumb",
".breadcrumbs",
"nav[aria-label='breadcrumb']",
".wy-breadcrumbs",
".related"
]
items = []
for selector in selectors:
for node in soup.select(selector):
for a in node.find_all("a", href=True):
title = a.get_text(" ", strip=True)
href = a.get("href")
url = self.resolve_url(base_url, href)
if not url:
continue
items.append({
"title": title or url.rsplit("/", 1)[-1],
"url": url,
"children": []
})
return items

429
crawler/playwright_client.py Normal file
View file

@ -0,0 +1,429 @@
import asyncio
from playwright.async_api import (
async_playwright,
TimeoutError as PlaywrightTimeoutError,
Error as PlaywrightError
)
from utils.logger import logger
from utils.retry import retry_async
from urllib.parse import urlparse
def is_binary_file(url: str):
path = urlparse(url).path.lower()
binary_ext = (
".xls", ".xlsx",
".pdf",
".zip",
".tar",
".gz",
".doc",
".docx",
".ppt",
".pptx",
".xlsm",
".csv"
)
return any(path.endswith(ext) for ext in binary_ext)
class PlaywrightClient:
"""
Production-ready Playwright wrapper.
"""
def __init__(
self,
timeout=45000,
headless=True,
browser_type="chromium",
content_selectors=None,
selector_wait_ms=3000,
blocked_resource_types=None,
):
self.timeout = timeout
self.headless = headless
self.browser_type = browser_type
# Selectors the caller considers "main content". If
# provided, we wait for any of them to appear instead
# of doing a blind fixed sleep — saves seconds per
# page on docs that DOM-render quickly.
self.content_selectors = content_selectors or []
self.selector_wait_ms = selector_wait_ms
# Resource types we abort at the network layer to
# avoid downloading bytes Chromium will never use for
# extraction. Images and stylesheets are also fetched
# separately by AssetDownloader from the raw HTML, so
# the browser doesn't need them.
self.blocked_resource_types = set(
blocked_resource_types
if blocked_resource_types is not None
else ("image", "stylesheet", "media", "font")
)
self.playwright = None
self.browser = None
self.context = None
async def start(self):
"""
Start Playwright browser.
"""
logger.info(
"Starting Playwright..."
)
self.playwright = await async_playwright().start()
browser_launcher = getattr(
self.playwright,
self.browser_type
)
self.browser = await browser_launcher.launch(
headless=self.headless,
args=[
"--disable-blink-features=AutomationControlled",
"--disable-dev-shm-usage",
"--no-sandbox",
"--disable-setuid-sandbox",
"--disable-gpu"
]
)
self.context = await self.browser.new_context(
viewport={
"width": 1600,
"height": 900
},
user_agent=(
"Mozilla/5.0 "
"(Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 "
"(KHTML, like Gecko) "
"Chrome/124.0.0.0 "
"Safari/537.36"
),
locale="en-US",
java_script_enabled=True,
ignore_https_errors=True
)
logger.info(
"Playwright started."
)
async def stop(self):
"""
Shutdown browser safely.
"""
logger.info(
"Stopping Playwright..."
)
try:
if self.context:
await self.context.close()
if self.browser:
await self.browser.close()
if self.playwright:
await self.playwright.stop()
except Exception as e:
logger.warning(
f"Playwright stop error: {e}"
)
logger.info(
"Playwright stopped."
)
@retry_async(
attempts=3,
min_wait=2,
max_wait=10
)
async def fetch_page(self, url: str):
"""
Render and fetch page safely (production version).
"""
logger.info(f"Fetching page: {url}")
if is_binary_file(url):
logger.info(f"Detected binary file, skipping as page: {url}")
return {
"url": url,
"title": url.split("/")[-1],
"html": "",
"status_code": None,
"headers": {},
"is_binary": True,
"content_type": ""
}
page = None
try:
page = await self.context.new_page()
# Apply routing (ONLY HTML requests should be intercepted)
await page.route(
"**/*",
self.route_interceptor
)
response = await page.goto(
url,
wait_until="domcontentloaded",
timeout=self.timeout
)
# Wait only as long as it takes for actual content
# to appear, instead of the previous fixed 5s
# load-state wait + 0.5s sleep. If the caller did
# not configure content selectors, fall back to
# the cheaper load-state wait.
if self.content_selectors:
joined = ",".join(self.content_selectors)
try:
await page.wait_for_selector(
joined,
timeout=self.selector_wait_ms,
)
except PlaywrightTimeoutError:
logger.debug(
f"Content selector not seen within "
f"{self.selector_wait_ms}ms, "
f"continuing anyway: {url}"
)
else:
try:
await page.wait_for_load_state(
"load",
timeout=self.selector_wait_ms,
)
except PlaywrightTimeoutError:
logger.debug(
f"Load state timeout, continuing "
f"anyway: {url}"
)
title = await page.title()
html = await page.content()
final_url = page.url
status = response.status if response else None
headers = response.headers if response else {}
logger.info(f"Fetched page successfully: {url}")
return {
"url": final_url,
"title": title,
"html": html,
"status_code": status,
"headers": headers,
"is_binary": False
}
except PlaywrightTimeoutError:
logger.error(f"Timeout while fetching: {url}")
raise
except PlaywrightError as e:
if self.is_transient_error(e):
logger.warning(
f"Transient Playwright error for {url}: {e}"
)
else:
logger.exception(
f"Non-transient Playwright error for {url}: {e}"
)
raise
except Exception as e:
logger.exception(
f"Unexpected fetch failed {url}: {e}"
)
raise
finally:
if page is not None:
try:
await page.close()
except Exception as close_err:
logger.warning(
f"Page close error for {url}: {close_err}"
)
async def route_interceptor(
self,
route
):
"""
Block unnecessary resources to speed up crawling.
Image and stylesheet bytes never affect HTML
extraction; assets are downloaded separately by
AssetDownloader from the raw HTML. Aborting them at
the network layer drops the per-page wire weight by
a large factor on doc sites.
"""
if route.request.resource_type in self.blocked_resource_types:
await route.abort()
return
await route.continue_()
async def screenshot(
self,
url,
output_path
):
"""
Debug screenshot utility.
"""
page = None
try:
page = await self.context.new_page()
await page.goto(
url,
wait_until="domcontentloaded",
timeout=self.timeout
)
await page.screenshot(
path=output_path,
full_page=True
)
logger.info(
f"Screenshot saved: "
f"{output_path}"
)
finally:
if page is not None:
await page.close()
async def evaluate_js(
self,
url,
script
):
"""
Execute JS on page.
"""
page = None
try:
page = await self.context.new_page()
await page.goto(
url,
wait_until="domcontentloaded",
timeout=self.timeout
)
result = await page.evaluate(
script
)
return result
finally:
if page is not None:
await page.close()
async def fetch_navigation_links(
self,
url
):
"""
Extract rendered navigation links.
"""
page = None
try:
page = await self.context.new_page()
await page.goto(
url,
wait_until="domcontentloaded",
timeout=self.timeout
)
links = await page.evaluate("""
() => {
return Array.from(
document.querySelectorAll('a')
).map(a => a.href)
}
""")
return links
finally:
if page is not None:
await page.close()
def is_transient_error(self, error: Exception):
message = str(error).lower()
transient_keywords = [
"err_connection_refused",
"err_connection_reset",
"err_connection_closed",
"err_timed_out",
"timeout",
"net::err",
"target closed",
"browser has been closed",
"page crashed"
]
return any(
keyword in message
for keyword in transient_keywords
)

333
crawler/sitemap_parser.py Normal file
View file

@ -0,0 +1,333 @@
import gzip
import xml.etree.ElementTree as ET
import httpx
from utils.logger import logger
class SitemapParser:
"""
Parse sitemap.xml files
to discover documentation URLs.
Supports:
- sitemap.xml
- sitemap index
- gzipped sitemap
"""
def __init__(self, timeout=30):
self.timeout = timeout
async def parse(
self,
sitemap_url: str
):
"""
Main entry point
Returns:
-------
List[str]
"""
logger.info(
f"Parsing sitemap: {sitemap_url}"
)
content = await self.fetch_sitemap(
sitemap_url
)
if not content:
return []
root = ET.fromstring(content)
tag = self.clean_tag(root.tag)
# sitemap index
if tag == "sitemapindex":
urls = await self.parse_sitemap_index(
root
)
logger.info(
f"Sitemap index found {len(urls)} URLs"
)
return urls
# regular sitemap
elif tag == "urlset":
urls = self.parse_urlset(root)
logger.info(
f"Sitemap contains {len(urls)} URLs"
)
return urls
else:
logger.warning(
f"Unknown sitemap type: {tag}"
)
return []
async def fetch_sitemap(
self,
sitemap_url: str
):
"""
Download sitemap content
"""
try:
async with httpx.AsyncClient(
timeout=self.timeout,
follow_redirects=True
) as client:
response = await client.get(
sitemap_url
)
response.raise_for_status()
content = response.content
# gzip sitemap
if sitemap_url.endswith(".gz"):
content = gzip.decompress(
content
)
return content
except Exception as e:
logger.exception(
f"Failed fetching sitemap {sitemap_url}: {e}"
)
return None
async def parse_sitemap_index(
self,
root
):
"""
Parse sitemap index recursively
Example:
<sitemapindex>
<sitemap>
<loc>...</loc>
</sitemap>
</sitemapindex>
"""
all_urls = []
for sitemap in root.findall(".//*"):
tag = self.clean_tag(sitemap.tag)
if tag != "loc":
continue
child_sitemap_url = sitemap.text
if not child_sitemap_url:
continue
logger.info(
f"Found child sitemap: {child_sitemap_url}"
)
try:
urls = await self.parse(
child_sitemap_url
)
all_urls.extend(urls)
except Exception as e:
logger.warning(
f"Failed child sitemap "
f"{child_sitemap_url}: {e}"
)
return all_urls
def parse_urlset(
self,
root
):
"""
Parse regular sitemap URL list
Example:
<urlset>
<url>
<loc>...</loc>
</url>
</urlset>
"""
urls = []
for url_node in root.findall(".//*"):
tag = self.clean_tag(url_node.tag)
if tag != "loc":
continue
url = url_node.text
if not url:
continue
url = url.strip()
if url not in urls:
urls.append(url)
return urls
def clean_tag(
self,
tag: str
):
"""
Remove XML namespace
Example:
{http://www.sitemaps.org/schemas/sitemap/0.9}url
->
url
"""
if "}" in tag:
return tag.split("}", 1)[1]
return tag
async def discover_common_sitemaps(
self,
base_url: str
):
"""
Try common sitemap locations
Returns:
-------
List[str]
"""
common_paths = [
"/sitemap.xml",
"/sitemap_index.xml",
"/sitemap-index.xml",
"/sitemap.gz",
"/robots.txt"
]
discovered = []
for path in common_paths:
url = base_url.rstrip("/") + path
try:
async with httpx.AsyncClient(
timeout=self.timeout
) as client:
response = await client.get(url)
if response.status_code == 200:
logger.info(
f"Found sitemap resource: {url}"
)
discovered.append(url)
except Exception:
pass
return discovered
async def parse_robots_for_sitemaps(
self,
robots_url: str
):
"""
Extract sitemap URLs from robots.txt
Example:
Sitemap: https://example.com/sitemap.xml
"""
sitemap_urls = []
try:
async with httpx.AsyncClient(
timeout=self.timeout
) as client:
response = await client.get(
robots_url
)
response.raise_for_status()
text = response.text
for line in text.splitlines():
line = line.strip()
if line.lower().startswith(
"sitemap:"
):
sitemap_url = line.split(
":",
1
)[1].strip()
sitemap_urls.append(
sitemap_url
)
logger.info(
f"Found {len(sitemap_urls)} "
f"sitemaps in robots.txt"
)
except Exception as e:
logger.warning(
f"Failed parsing robots.txt "
f"{robots_url}: {e}"
)
return sitemap_urls

329
crawler/sphinx_discovery.py Normal file
View file

@ -0,0 +1,329 @@
import json
import re
from pathlib import Path
from urllib.parse import urljoin
import httpx
from bs4 import BeautifulSoup
from utils.logger import logger
def _ensure_soup(html_or_soup):
if hasattr(html_or_soup, "select_one") and hasattr(
html_or_soup, "find_all"
):
return html_or_soup
return BeautifulSoup(html_or_soup, "lxml")
class SphinxDiscovery:
"""
Discover hidden / structured pages from Sphinx-like docs.
Sources:
- searchindex.js
- genindex.html
- global index pages
- toctree/sidebar links
"""
def __init__(
self,
url_manager,
timeout=30
):
self.url_manager = url_manager
self.timeout = timeout
async def discover_from_base(
self,
base_url: str
):
"""
Main discovery entry point.
"""
discovered = set()
candidates = self.common_sphinx_urls(
base_url
)
for url in candidates:
urls = await self.discover_from_url(url)
discovered.update(urls)
logger.info(
f"Sphinx discovery found {len(discovered)} URLs from {base_url}"
)
return sorted(discovered)
def common_sphinx_urls(
self,
base_url: str
):
"""
Common Sphinx-generated files.
"""
base = base_url.rstrip("/") + "/"
return [
urljoin(base, "searchindex.js"),
urljoin(base, "genindex.html"),
urljoin(base, "py-modindex.html"),
urljoin(base, "search.html"),
urljoin(base, "contents.html"),
urljoin(base, "index.html"),
]
async def discover_from_url(
self,
url: str
):
"""
Dispatch discovery by file type.
"""
if url.endswith("searchindex.js"):
return await self.parse_searchindex(url)
if url.endswith(".html"):
return await self.parse_html_index(url)
return []
async def fetch_text(
self,
url: str
):
try:
async with httpx.AsyncClient(
timeout=self.timeout,
follow_redirects=True
) as client:
response = await client.get(url)
if response.status_code >= 400:
return None
return response.text
except Exception as e:
logger.debug(
f"Sphinx discovery fetch failed {url}: {e}"
)
return None
async def parse_searchindex(
self,
url: str
):
"""
Parse Sphinx searchindex.js.
Typical format:
Search.setIndex({...})
"""
text = await self.fetch_text(url)
if not text:
return []
try:
payload = self.extract_searchindex_json(text)
if not payload:
return []
docnames = payload.get("docnames", [])
base_url = url.rsplit("/", 1)[0] + "/"
discovered = set()
for docname in docnames:
page_url = urljoin(
base_url,
docname + ".html"
)
normalized = self.url_manager.normalize(page_url)
if (
normalized
and self.url_manager.is_allowed(normalized)
):
discovered.add(normalized)
logger.info(
f"Parsed {len(discovered)} URLs from searchindex: {url}"
)
return sorted(discovered)
except Exception as e:
logger.warning(
f"Failed parsing searchindex {url}: {e}"
)
return []
def extract_searchindex_json(
self,
text: str
):
"""
Extract JSON-like object from Sphinx searchindex.js.
"""
match = re.search(
r"Search\.setIndex\((.*)\)\s*;?\s*$",
text,
re.DOTALL
)
if not match:
return None
raw = match.group(1)
try:
return json.loads(raw)
except Exception:
pass
# Fallback for slightly non-standard JS object
raw = raw.replace("'", '"')
try:
return json.loads(raw)
except Exception:
logger.debug(
"searchindex.js is not valid JSON after normalization."
)
return None
async def parse_html_index(
self,
url: str
):
"""
Parse Sphinx index-like HTML files.
"""
html = await self.fetch_text(url)
if not html:
return []
soup = BeautifulSoup(html, "lxml")
discovered = set()
selectors = [
"a.reference.internal",
".toctree-wrapper a",
".wy-menu a",
".sphinxsidebar a",
".globaltoc a",
".localtoc a",
"nav a",
"a[href]"
]
for selector in selectors:
for a in soup.select(selector):
href = a.get("href")
resolved = self.url_manager.resolve(
url,
href
)
if not resolved:
continue
if not self.url_manager.is_allowed(resolved):
continue
discovered.add(resolved)
logger.info(
f"Parsed {len(discovered)} URLs from HTML index: {url}"
)
return sorted(discovered)
async def discover_from_rendered_html(
self,
base_url: str,
html
):
"""
Discover Sphinx links from already-rendered page HTML.
Accepts either a string or a parsed soup.
"""
soup = _ensure_soup(html)
discovered = set()
selectors = [
"a.reference.internal",
".toctree-wrapper a.reference.internal",
".section a.reference.internal",
".body a.reference.internal",
".document a.reference.internal",
".wy-menu a",
".sphinxsidebar a",
".globaltoc a",
".localtoc a",
"link[rel='next']",
"link[rel='prev']",
"link[rel='up']"
]
for selector in selectors:
for node in soup.select(selector):
href = node.get("href")
if not href:
continue
resolved = self.url_manager.resolve(
base_url,
href
)
if not resolved:
continue
if not self.url_manager.is_allowed(resolved):
continue
discovered.add(resolved)
return sorted(discovered)
def build_site_graph_edges(
self,
source_url: str,
target_urls: list[str]
):
"""
Prepare graph edges.
"""
edges = []
for target in target_urls:
edges.append({
"source": source_url,
"target": target,
"type": "sphinx_internal_link"
})
return edges

328
crawler/url_manager.py Normal file
View file

@ -0,0 +1,328 @@
from urllib.parse import (
urlparse,
urljoin,
parse_qs,
urlencode
)
from pathlib import Path
from utils.logger import logger
class URLManager:
"""
Production-ready URL manager.
Handles:
- normalization
- deduplication
- filtering
- crawl scope
- asset filtering
"""
def __init__(
self,
allowed_domains,
allowed_schemes=None,
blocked_extensions=None,
remove_query_params=True
):
self.allowed_domains = set(
allowed_domains
)
self.allowed_schemes = (
allowed_schemes
or {"http", "https"}
)
self.remove_query_params = (
remove_query_params
)
self.blocked_extensions = (
blocked_extensions
or {
".jpg",
".jpeg",
".png",
".gif",
".svg",
".webp",
".pdf",
".xls",
".xlsx",
".doc",
".docx",
".ppt",
".pptx",
".zip",
".tar",
".gz",
".mp4",
".mp3",
".avi",
".woff",
".woff2",
".ttf",
".eot"
}
)
def normalize(
self,
url: str
):
"""
Normalize URL consistently.
"""
if not url:
return None
parsed = urlparse(url)
# Remove fragments
parsed = parsed._replace(
fragment=""
)
# Normalize scheme/domain
scheme = parsed.scheme.lower()
netloc = parsed.netloc.lower()
# Remove default ports
netloc = netloc.replace(
":80",
""
).replace(
":443",
""
)
# Normalize path
path = parsed.path or "/"
if path != "/" and path.endswith("/"):
path = path.rstrip("/")
# Remove tracking query params
query = ""
if (
not self.remove_query_params
and parsed.query
):
query = parsed.query
normalized = parsed._replace(
scheme=scheme,
netloc=netloc,
path=path,
query=query
)
return normalized.geturl()
def resolve(
self,
base_url: str,
href: str
):
"""
Resolve relative URL.
"""
if not href:
return None
href = href.strip()
# Ignore anchors
if href.startswith("#"):
return None
# Ignore JS/mail links
ignored = [
"javascript:",
"mailto:",
"tel:"
]
if any(
href.startswith(prefix)
for prefix in ignored
):
return None
try:
resolved = urljoin(
base_url,
href
)
return self.normalize(
resolved
)
except Exception as e:
logger.warning(
f"URL resolve failed "
f"{href}: {e}"
)
return None
def is_allowed(
self,
url: str
):
"""
Check crawl scope.
"""
if not url:
return False
parsed = urlparse(url)
# Scheme check
if (
parsed.scheme
not in self.allowed_schemes
):
return False
domain = parsed.netloc.lower()
# Domain check
if not any(
domain == allowed or domain.endswith("." + allowed)
for allowed in self.allowed_domains
):
return False
# Extension filter
extension = Path(
parsed.path
).suffix.lower()
if extension in self.blocked_extensions:
return False
return True
def should_enqueue(
self,
url: str
):
"""
Determine whether URL
should be crawled.
"""
normalized = self.normalize(url)
if not normalized:
return False
if not self.is_allowed(normalized):
return False
return True
def extract_domain(
self,
url: str
):
"""
Extract domain.
"""
parsed = urlparse(url)
return parsed.netloc.lower()
def extract_path(
self,
url: str
):
"""
Extract path.
"""
parsed = urlparse(url)
return parsed.path
def remove_query_tracking(
self,
url: str
):
"""
Remove tracking params.
"""
parsed = urlparse(url)
query = parse_qs(
parsed.query
)
blocked_params = {
"utm_source",
"utm_medium",
"utm_campaign",
"fbclid",
"gclid"
}
clean_query = {
k: v
for k, v in query.items()
if k not in blocked_params
}
encoded_query = urlencode(
clean_query,
doseq=True
)
parsed = parsed._replace(
query=encoded_query
)
return parsed.geturl()
def is_binary_url(self, url: str):
parsed = urlparse(url)
extension = Path(parsed.path).suffix.lower()
binary_extensions = {
".pdf",
".xls",
".xlsx",
".doc",
".docx",
".ppt",
".pptx",
".zip"
}
return extension in binary_extensions
def stats(self):
return {
"allowed_domains": list(self.allowed_domains),
"blocked_extensions": list(self.blocked_extensions),
}

0
extractor/__init__.py Normal file
View file

420
extractor/asset_downloader.py Normal file
View file

@ -0,0 +1,420 @@
import asyncio
import hashlib
from pathlib import Path
from urllib.parse import urljoin, urlparse
import aiofiles
import httpx
from bs4 import BeautifulSoup
from utils.logger import logger
import os
def _ensure_soup(html_or_soup):
if hasattr(html_or_soup, "select_one") and hasattr(
html_or_soup, "find_all"
):
return html_or_soup
return BeautifulSoup(html_or_soup, "lxml")
class AssetDownloader:
"""
Download assets from documentation pages.
Supported:
- images
- css
- js
- fonts
- downloadable files
Features:
- deduplicate downloads
- async download
- preserve structure
"""
def __init__(
self,
output_dir="output/assets",
timeout=30,
max_concurrency=10,
http_client=None
):
self.output_dir = Path(output_dir)
self.timeout = timeout
# downloaded maps url -> local Path so repeat lookups
# short-circuit without touching the filesystem
self.downloaded = {}
# Shared httpx client (keep-alive + optional http2). If
# the caller didn't pass one we create our own — but
# a shared client across pages is what actually buys
# the speedup.
self._owns_client = http_client is None
if http_client is None:
try:
import h2 # noqa: F401
_http2 = True
except ImportError:
_http2 = False
http_client = httpx.AsyncClient(
timeout=self.timeout,
follow_redirects=True,
http2=_http2,
limits=httpx.Limits(
max_connections=max_concurrency * 2,
max_keepalive_connections=max_concurrency * 2,
),
)
self.client = http_client
self.semaphore = asyncio.Semaphore(max_concurrency)
self.allowed_extensions = {
".png",
".jpg",
".jpeg",
".gif",
".svg",
".webp",
".css",
".js",
".woff",
".woff2",
".ttf",
".eot",
".pdf",
".zip",
".ico",
".bmp",
".avif",
".map",
".json",
".xls",
".xlsx",
".doc",
".docx",
".ppt",
".pptx",
".csv"
}
async def download_assets_from_html(
self,
base_url: str,
html
):
"""
Parse HTML and download all assets. Accepts either an
HTML string or a BS4 soup/Tag so process_page can pass
an already-parsed tree.
"""
soup = _ensure_soup(html)
asset_urls = set()
# Images
for img in soup.find_all("img", src=True):
src = img["src"].strip()
full_url = urljoin(base_url, src)
asset_urls.add(full_url)
# CSS
for link in soup.find_all("link", href=True):
href = link["href"].strip()
rel = link.get("rel", [])
if "stylesheet" in rel:
asset_urls.add(
urljoin(base_url, href)
)
# JS
for script in soup.find_all(
"script",
src=True
):
src = script["src"].strip()
asset_urls.add(
urljoin(base_url, src)
)
logger.info(
f"Found {len(asset_urls)} assets"
)
results = await asyncio.gather(
*(self._download_safe(u) for u in asset_urls),
return_exceptions=False,
)
downloaded_assets = [
{"url": url, "local_path": str(path)}
for url, path in results
if path is not None
]
return downloaded_assets
async def _download_safe(self, asset_url):
try:
local_path = await self.download_asset(asset_url)
return asset_url, local_path
except Exception as e:
logger.warning(
f"Asset download failed {asset_url}: {e}"
)
return asset_url, None
async def download_asset(
self,
asset_url: str
):
"""
Download single asset.
"""
asset_url = self.normalize_url(asset_url)
cached = self.downloaded.get(asset_url)
if cached is not None:
logger.debug(f"Already downloaded: {asset_url}")
return cached
parsed = urlparse(asset_url)
extension = Path(parsed.path).suffix.lower()
if extension not in self.allowed_extensions:
logger.debug(
f"Skipped unsupported asset: "
f"{asset_url}"
)
return None
local_path = self.url_to_local_path(
asset_url
)
# Skip re-downloading assets that are already on disk
# from a prior run. Saves a HEAD+GET roundtrip per
# cache-bustable asset on resume.
if local_path.exists():
self.downloaded[asset_url] = local_path
return local_path
local_path.parent.mkdir(
parents=True,
exist_ok=True
)
try:
async with self.semaphore:
response = await self.client.get(asset_url)
response.raise_for_status()
async with aiofiles.open(
local_path,
"wb"
) as f:
await f.write(response.content)
self.downloaded[asset_url] = local_path
logger.debug(
f"Downloaded asset: {local_path}"
)
return local_path
except Exception as e:
logger.warning(
f"Failed asset download "
f"{asset_url}: {e}"
)
return None
async def aclose(self):
if self._owns_client:
await self.client.aclose()
def url_to_local_path(
self,
url: str
):
"""
Convert asset URL -> local path.
Example:
https://api.intra-mart.jp/assets/logo.png
=>
output/assets/api.intra-mart.jp/assets/logo.png
"""
parsed = urlparse(url)
domain = parsed.netloc
path = parsed.path.strip("/")
if not path:
filename = self.hash_url(url)
path = f"unknown/{filename}"
local_path = (
self.output_dir /
domain /
path
)
return local_path
def hash_url(
self,
url: str
):
"""
Generate stable filename hash
"""
return hashlib.md5(
url.encode("utf-8")
).hexdigest()
def normalize_url(
self,
url: str
):
"""
Remove fragments and query strings.
Query strings are stripped so that cache-busted URLs
(e.g. logo.png?v=1 vs logo.png?v=2) deduplicate to the
same local file path -- avoiding repeated downloads
that overwrite the same file and avoiding stale dedup
keys that point to a path written by a different URL.
"""
parsed = urlparse(url)
clean = parsed._replace(fragment="", query="")
return clean.geturl()
def rewrite_asset_links(
self,
html,
base_url: str,
current_output_path: Path
):
"""
Rewrite HTML asset links to local filesystem paths.
Accepts either an HTML string or a BS4 soup/Tag.
Returns the rewritten HTML as a string.
"""
soup = _ensure_soup(html)
# Rewrite image src
for img in soup.find_all("img", src=True):
src = img["src"]
full_url = urljoin(base_url, src)
if Path(urlparse(full_url).path).suffix.lower() not in self.allowed_extensions:
continue
relative_path = self.make_relative_asset_path(
full_url,
current_output_path
)
img["src"] = relative_path
# Rewrite CSS href -- ONLY rewrite tags we actually
# download (rel=stylesheet). Other <link> tags
# (canonical, alternate, icon, preload, ...) point to
# files we never fetched, so rewriting their href
# would produce dead local references.
for link in soup.find_all(
"link",
href=True
):
rel = link.get("rel", [])
if "stylesheet" not in rel:
continue
href = link["href"]
full_url = urljoin(base_url, href)
if Path(urlparse(full_url).path).suffix.lower() not in self.allowed_extensions:
continue
relative_path = self.make_relative_asset_path(
full_url,
current_output_path
)
link["href"] = relative_path
# Rewrite JS src
for script in soup.find_all(
"script",
src=True
):
src = script["src"]
full_url = urljoin(base_url, src)
if Path(urlparse(full_url).path).suffix.lower() not in self.allowed_extensions:
continue
relative_path = self.make_relative_asset_path(
full_url,
current_output_path
)
script["src"] = relative_path
return str(soup)
def make_relative_asset_path(
self,
asset_url: str,
current_output_path: Path
):
local_path = self.url_to_local_path(asset_url)
relative = os.path.relpath(
local_path,
current_output_path.parent
)
return relative.replace("\\", "/")

227
extractor/html_cleaner.py Normal file
View file

@ -0,0 +1,227 @@
from bs4 import BeautifulSoup
from utils.logger import logger
def _ensure_soup(html_or_soup):
"""
Accept either a BeautifulSoup/Tag or an HTML string and
return a BeautifulSoup-like object. Cheap when input is
already parsed (this is the whole point — avoid reparsing).
"""
if hasattr(html_or_soup, "select_one") and hasattr(
html_or_soup, "find_all"
):
return html_or_soup
return BeautifulSoup(html_or_soup, "lxml")
class HTMLCleaner:
"""
Clean extracted HTML
before markdown conversion.
"""
def __init__(
self,
remove_selectors
):
self.remove_selectors = remove_selectors
def clean(
self,
html
):
"""
Clean HTML in place when given a soup; otherwise parse
the string, clean, and return the string. Returning
the same type the caller passed lets process_page hold
a soup across the whole pipeline without re-parsing.
"""
return_string = isinstance(html, (str, bytes))
soup = _ensure_soup(html)
self.remove_unwanted_nodes(soup)
self.normalize_code_blocks(soup)
self.normalize_tables(soup)
self.fix_relative_links(soup)
self.remove_empty_tags(soup)
self.unwrap_redundant_tags(soup)
return str(soup) if return_string else soup
def remove_unwanted_nodes(
self,
soup
):
"""
Remove navigation/UI noise.
"""
for selector in self.remove_selectors:
try:
for tag in soup.select(selector):
tag.decompose()
except Exception as e:
logger.warning(
f"Remove selector failed "
f"{selector}: {e}"
)
additional_noise = [
".sidebar",
".toc",
".breadcrumb",
".pagination",
".edit-page",
".theme-doc-version-badge",
".header",
".footer",
".ads",
".search",
".search-box",
".navigation"
]
for selector in additional_noise:
for tag in soup.select(selector):
tag.decompose()
def normalize_code_blocks(
self,
soup
):
"""
Normalize code blocks
for better markdown conversion.
"""
for pre in soup.find_all("pre"):
code = pre.find("code")
if not code:
continue
classes = code.get("class", [])
language = None
for cls in classes:
if cls.startswith("language-"):
language = cls.replace(
"language-",
""
)
if language:
pre["data-language"] = language
def normalize_tables(
self,
soup
):
"""
Improve HTML table structure.
"""
for table in soup.find_all("table"):
if not table.find("thead"):
first_row = table.find("tr")
if first_row:
thead = soup.new_tag(
"thead"
)
first_row.extract()
thead.append(first_row)
table.insert(0, thead)
def fix_relative_links(
self,
soup
):
"""
Remove javascript links.
"""
for a in soup.find_all(
"a",
href=True
):
href = a["href"].strip()
if href.startswith(
"javascript:"
):
del a["href"]
def remove_empty_tags(
self,
soup
):
"""
Remove empty useless tags.
"""
removable = [
"div",
"span",
"p"
]
for tag_name in removable:
for tag in soup.find_all(tag_name):
if tag.get_text(strip=True):
continue
if tag.find():
continue
tag.decompose()
def unwrap_redundant_tags(
self,
soup
):
"""
Flatten unnecessary wrappers.
"""
unwrap_tags = [
"span"
]
for tag_name in unwrap_tags:
for tag in soup.find_all(tag_name):
if tag.attrs:
continue
tag.unwrap()

207
extractor/html_extractor.py Normal file
View file

@ -0,0 +1,207 @@
from bs4 import BeautifulSoup
from utils.logger import logger
def _ensure_soup(html_or_soup):
if hasattr(html_or_soup, "select_one") and hasattr(
html_or_soup, "find_all"
):
return html_or_soup
return BeautifulSoup(html_or_soup, "lxml")
class HTMLExtractor:
"""
Extract main documentation content
from rendered HTML.
"""
def __init__(self, selectors):
self.selectors = selectors
def extract_main_content(
self,
html
):
"""
Extract the most relevant content block.
"""
result = self.extract_with_status(html)
return result["html"]
def extract_with_status(self, html):
"""
Like extract_main_content but reports whether a
configured selector matched. Accepts either an HTML
string or a BeautifulSoup/Tag so process_page can
avoid re-parsing.
The returned dict now also includes ``node`` — the
matched BS4 element — so downstream stages (asset
downloader, asset link rewriter) can operate directly
on the parsed tree instead of re-parsing ``html``.
"""
soup = _ensure_soup(html)
self.remove_noise(soup)
for selector in self.selectors:
try:
node = soup.select_one(selector)
if node:
text_length = len(
node.get_text(strip=True)
)
if text_length < 100:
continue
logger.info(
f"Main content found "
f"with selector: {selector}"
)
return {
"node": node,
"html": str(node),
"matched": True,
"selector": selector,
"text_length": text_length,
}
except Exception as e:
logger.warning(
f"Selector failed {selector}: {e}"
)
logger.warning(
"No selector matched. Using fallback extraction."
)
fallback = self.fallback_extract(soup)
return {
"node": fallback,
"html": str(fallback),
"matched": False,
"selector": None,
"text_length": len(fallback.get_text(strip=True))
if hasattr(fallback, "get_text")
else 0,
}
def remove_noise(
self,
soup
):
"""
Remove obvious noise
before extraction.
"""
noise_tags = [
"script",
"style",
"noscript",
"iframe",
"svg"
]
for tag_name in noise_tags:
for tag in soup.find_all(tag_name):
tag.decompose()
def fallback_extract(
self,
soup
):
"""
Heuristic fallback extraction.
"""
candidates = []
possible_selectors = [
"main",
"article",
".content",
".document",
".markdown-body",
".theme-doc-markdown",
".rst-content",
".page-content"
]
for selector in possible_selectors:
for node in soup.select(selector):
score = self.calculate_score(node)
candidates.append(
(score, node)
)
if candidates:
candidates.sort(
key=lambda x: x[0],
reverse=True
)
best = candidates[0][1]
logger.info(
"Fallback extraction selected "
"best candidate."
)
return best
# Final fallback = body
body = soup.body
if body:
return body
return soup
def calculate_score(
self,
node
):
"""
Score content relevance.
"""
text = node.get_text(" ", strip=True)
text_length = len(text)
paragraph_count = len(
node.find_all("p")
)
code_blocks = len(
node.find_all(["pre", "code"])
)
headings = len(
node.find_all(
["h1", "h2", "h3"]
)
)
score = (
text_length
+ paragraph_count * 50
+ code_blocks * 100
+ headings * 30
)
return score

815
main.py Normal file
View file

@ -0,0 +1,815 @@
import asyncio
from pathlib import Path
from bs4 import BeautifulSoup
from crawler.http_client import HTTPClient
from crawler.playwright_client import PlaywrightClient
from crawler.url_manager import URLManager
from converter.markdown_converter import MarkdownConverter
from extractor.html_cleaner import HTMLCleaner
from extractor.html_extractor import HTMLExtractor
from models.crawl_result import CrawlResult
from storage.file_writer import FileWriter
from storage.metadata_db import MetadataDB
from storage.path_mapper import PathMapper
from utils.logger import logger
from extractor.asset_downloader import AssetDownloader
from crawler.sphinx_discovery import SphinxDiscovery
from crawler.navigation_parser import NavigationParser
START_URLS = [
"https://api.intra-mart.jp/iap/",
"https://document.intra-mart.jp/library/"
]
OUTPUT_DIR = "output"
ALLOWED_DOMAINS = [
"api.intra-mart.jp",
"document.intra-mart.jp"
]
REMOVE_SELECTORS = [
"script",
"style",
"noscript",
".sidebar-ad",
".navigation-footer",
".breadcrumb",
".toc",
".page-footer",
".header",
".search",
".ads"
]
CONTENT_SELECTORS = [
"main",
"article",
".content",
".document",
".markdown-body",
".theme-doc-markdown",
"#content",
# Javadoc (api.intra-mart.jp/iap/javadoc/...) — static
# pages, by far the bulk of the queue. Matching this
# selector lets the HTTP fast-path skip Playwright.
"div.contentContainer",
# apilist-ssjs (api.intra-mart.jp/iap/apilist-ssjs/...)
# wraps its body content in <div class="main">. Use a
# body-scoped selector so we don't match `.main`
# elements nested inside navigation widgets.
"body > div.main",
# document.intra-mart.jp/library/... — most user-guide
# pages put their content in a single top-level
# <div class="container">.
"body > div.container",
]
MAX_PAGES_PER_RUN = 100000
MAX_ATTEMPTS = 3
BROWSER_RECYCLE_EVERY = 500
class _PermanentFetchFailure(Exception):
"""Raised when re-fetching the URL cannot succeed (4xx)."""
# Number of concurrent page workers. Each worker shares the
# single PlaywrightContext + single httpx client; only DB
# writes serialize through self.db_lock. The bottleneck is
# network I/O, so this is the multiplier on throughput.
CONCURRENCY = 8
class DocumentationCrawler:
"""
Main crawler orchestrator.
"""
def __init__(self):
self.client = PlaywrightClient(
headless=True,
content_selectors=CONTENT_SELECTORS,
)
self.http_client = HTTPClient()
self.url_manager = URLManager(
allowed_domains=ALLOWED_DOMAINS
)
self.sphinx_discovery = SphinxDiscovery(
url_manager=self.url_manager
)
self.navigation_parser = NavigationParser(
url_manager=self.url_manager
)
self.navigation_trees = []
self.cleaner = HTMLCleaner(
REMOVE_SELECTORS
)
self.extractor = HTMLExtractor(
CONTENT_SELECTORS
)
self.asset_downloader = AssetDownloader(
output_dir=str(Path(OUTPUT_DIR) / "assets"),
http_client=self.http_client.client,
)
self.converter = MarkdownConverter()
self.writer = FileWriter()
self.db = MetadataDB()
self.path_mapper = PathMapper(
Path(OUTPUT_DIR) / "markdown"
)
self.processed_this_run = 0
# Single lock serializing DB write blocks across
# concurrent workers. SQLite + Python's sqlite3 module
# already mutex on the connection, but a transaction()
# context spans many statements — without this lock,
# two workers could interleave statements mid-tx.
self.db_lock = asyncio.Lock()
# In-flight worker count for clean shutdown: a worker
# that sees an empty queue must wait until all peers
# finish (their process_page may enqueue new URLs).
self._in_flight = 0
self._stop = False
# Track domains we've already navigation-parsed.
# Sidebars are site-wide on Sphinx/Javadoc, so the same
# tree is rediscovered on every page — running the
# heavy parser repeatedly wastes CPU. Any per-section
# URLs we miss here will still be picked up by
# extract_links (it iterates all <a href>).
self._nav_parsed_domains = set()
async def initialize(self):
"""
Initialize crawler resources.
"""
logger.info(
"Initializing crawler..."
)
self.db.reset_stuck_processing()
await self.client.start()
for url in START_URLS:
normalized = self.url_manager.normalize(url)
if normalized and self.url_manager.is_allowed(normalized):
self.db.enqueue_url(
normalized,
parent_url=None,
depth=0,
priority=100
)
for url in START_URLS:
discovered = await self.sphinx_discovery.discover_from_base(url)
for discovered_url in discovered:
if not self.url_manager.should_enqueue(discovered_url):
continue
if self.db.is_done(discovered_url):
continue
if self.db.is_known(discovered_url):
continue
self.db.enqueue_url(
discovered_url,
parent_url=url,
depth=1,
priority=50
)
self.db.save_discovery_source(
url=discovered_url,
discovery_type="sphinx_base_discovery",
source_url=url
)
logger.info(f"Initial queue stats: {self.db.queue_stats()}")
async def shutdown(self):
"""
Cleanup resources.
"""
logger.info(
"Shutting down crawler..."
)
self.export_summary()
await self.client.stop()
await self.http_client.aclose()
self.db.close()
logger.info(
"Crawler shutdown complete."
)
async def crawl(self):
"""
Main crawl entry point. Spawns CONCURRENCY workers that
share the single browser context + httpx client and
serialize through self.db_lock for DB writes.
"""
await self.initialize()
try:
workers = [
asyncio.create_task(self._worker(i))
for i in range(CONCURRENCY)
]
await asyncio.gather(*workers)
finally:
await self.shutdown()
async def _claim_next_url(self):
"""
Atomically pick the next pending URL and mark it
processing. Returns None when the queue is empty.
"""
async with self.db_lock:
item = self.db.get_next_pending_url()
if not item:
return None
url = item["url"]
if self.db.is_done(url):
# Stale row — skip without claiming.
return "skip"
self.db.mark_processing(url)
self._in_flight += 1
return item
async def _worker(self, worker_id: int):
"""
One concurrent worker. Loops until the queue stays
empty *and* no peer worker is in-flight (peers can
enqueue new URLs as they discover links).
"""
while not self._stop:
if self.processed_this_run >= MAX_PAGES_PER_RUN:
return
item = await self._claim_next_url()
if item == "skip":
continue
if item is None:
# No work available — but a peer might enqueue
# more. Exit only if everyone else is idle too.
if self._in_flight == 0:
return
await asyncio.sleep(0.3)
continue
url = item["url"]
depth = item["depth"]
try:
result = await self.process_page(
url=url,
depth=depth,
)
async with self.db_lock:
self.db.mark_done(url)
if result and result.url != url:
self.db.mark_done(result.url)
self.processed_this_run += 1
n = self.processed_this_run
if n % 20 == 0:
logger.info(
f"Progress: {n} pages this run. "
f"Queue: {self.db.queue_stats()}"
)
if n % 100 == 0:
self.export_summary()
logger.info(
"Intermediate summary exported."
)
except asyncio.CancelledError:
raise
except _PermanentFetchFailure as e:
# Skip the retry ramp — re-fetching can't
# succeed.
logger.warning(f"Permanent failure on {url}: {e}")
async with self.db_lock:
self.db.mark_permanently_failed(url, str(e))
except Exception as e:
logger.exception(
f"Failed processing {url}: {e}"
)
async with self.db_lock:
self.db.mark_failed(
url,
str(e),
max_attempts=MAX_ATTEMPTS,
)
finally:
async with self.db_lock:
self._in_flight -= 1
async def _fetch_and_extract(self, url: str):
"""
Try a cheap HTTP fetch first. If a configured content
selector matches we're done. Otherwise fall back to a
full Playwright render (handles JS-only pages).
Returns (response_dict, extraction_dict) or (None, None)
when neither path produces usable content.
"""
# Conditional GET: ask the server whether the page
# has changed since we last cached it. A 304 lets us
# skip all downstream work.
etag, last_modified = self.db.get_cached_headers(url)
http_response = await self.http_client.fetch_page(
url,
etag=etag,
last_modified=last_modified,
)
if http_response is not None and http_response.get("is_binary"):
return http_response, None
if http_response is not None and http_response.get("not_modified"):
# Signal "nothing changed" to process_page.
return http_response, None
# Hard 4xx (404, 410, 451, ...): there is nothing
# Playwright can do that httpx cannot. Surface this so
# the worker can mark the URL permanently failed and
# stop wasting retries.
if (
http_response is not None
and http_response.get("status_code") is not None
and 400 <= http_response["status_code"] < 500
):
return http_response, None
if http_response is not None and http_response.get("ok"):
# Parse the HTML once here. The cleaner mutates
# the soup in place; the extractor uses it; assets
# later use the matched node. No re-parses.
cleaning_soup = BeautifulSoup(
http_response["html"], "lxml"
)
self.cleaner.clean(cleaning_soup)
extraction = self.extractor.extract_with_status(
cleaning_soup
)
if extraction["matched"]:
logger.info(f"HTTP fast-path success: {url}")
return http_response, extraction
logger.info(
f"HTTP fetch ok but no content selector matched "
f"— falling back to Playwright: {url}"
)
# Fallback: full browser render
pw_response = await self.client.fetch_page(url)
if pw_response.get("is_binary"):
return pw_response, None
cleaning_soup = BeautifulSoup(pw_response["html"], "lxml")
self.cleaner.clean(cleaning_soup)
extraction = self.extractor.extract_with_status(
cleaning_soup
)
return pw_response, extraction
async def process_page(self, url: str, depth: int):
"""
Process single page.
"""
logger.info(
f"Crawling: {url}"
)
if self.url_manager.is_binary_url(url):
logger.info(f"Skipping binary URL as page: {url}")
return None
response, extraction = await self._fetch_and_extract(url)
if response is None:
return None
if response.get("is_binary"):
logger.info(f"Skipping binary response: {url}")
return None
# 304 Not Modified: skip the entire pipeline. Worker
# will mark_done as usual.
if response.get("not_modified"):
logger.info(f"Not modified, skipping: {url}")
return CrawlResult(
status_code=304,
url=url,
title="",
markdown_path="",
links=[],
success=True,
)
# _fetch_and_extract returns extraction=None with a
# 4xx response when the URL is a permanent failure
# (404 etc.). Raise a typed error so the worker maps
# it to mark_permanently_failed instead of retrying.
status_code = response.get("status_code")
if extraction is None and status_code and 400 <= status_code < 500:
raise _PermanentFetchFailure(
f"HTTP {status_code} on {url}"
)
final_url = response.get("url", url)
html = response["html"]
title = response["title"]
# The extracted node lives inside the cleaning soup
# already produced in _fetch_and_extract. Use it
# directly to avoid one BS4 re-parse per page.
extracted_node = extraction["node"]
# Resolve output path first
output_path = self.path_mapper.url_to_path(
final_url
)
# Download assets — pass the already-parsed node
# rather than its serialized HTML string.
downloaded_assets = await self.asset_downloader.download_assets_from_html(
base_url=final_url,
html=extracted_node,
)
async with self.db_lock:
with self.db.transaction():
for asset in downloaded_assets:
self.db.save_asset(
page_url=final_url,
asset_url=asset["url"],
local_path=asset["local_path"]
)
# Rewrite asset links in place on the same parsed
# node — no re-parse needed.
self.asset_downloader.rewrite_asset_links(
html=extracted_node,
base_url=final_url,
current_output_path=output_path,
)
# Convert markdown
markdown = self.converter.html_to_markdown(
str(extracted_node),
title=title,
source_url=final_url,
media_dir=Path(OUTPUT_DIR) / "assets" / "pandoc",
)
self.writer.write_text(
output_path,
markdown
)
# Parse the ORIGINAL html ONCE for all three
# downstream consumers. Previously each one re-parsed
# the same string independently.
original_soup = BeautifulSoup(html, "lxml")
discovered_links = self.extract_links(
final_url,
original_soup,
)
sphinx_links = await self.sphinx_discovery.discover_from_rendered_html(
base_url=final_url,
html=original_soup,
)
# Only parse navigation the first time we visit a
# given domain. The sidebar tree is site-wide and
# repeating the parse on every page is the single
# most expensive bit of per-page CPU we still have.
nav_domain = self.url_manager.extract_domain(final_url)
if nav_domain in self._nav_parsed_domains:
navigation_result = {"urls": [], "tree": []}
else:
navigation_result = self.navigation_parser.parse_navigation(
base_url=final_url,
html=original_soup,
)
self._nav_parsed_domains.add(nav_domain)
self.navigation_trees.extend(
navigation_result["tree"]
)
for nav_url in navigation_result["urls"]:
if nav_url not in discovered_links:
discovered_links.append(nav_url)
logger.info(
f"SPHINX_LINKS: {len(sphinx_links)} from {final_url}"
)
for sphinx_link in sphinx_links:
if sphinx_link not in discovered_links:
discovered_links.append(sphinx_link)
# Batch every per-link write + the page row into one
# transaction per page — turns hundreds of fsyncs into
# one, and one acquisition of db_lock per worker turn.
async with self.db_lock:
with self.db.transaction():
for link in discovered_links:
logger.debug(f"DISCOVERED: {link}")
self.db.save_page_link(
source_url=final_url,
target_url=link,
link_type="sphinx_or_html_internal"
)
if not self.url_manager.should_enqueue(link):
continue
if self.db.is_done(link):
continue
if self.db.is_known(link):
continue
logger.debug(f"ENQUEUED: {link}")
self.db.enqueue_url(
url=link,
parent_url=final_url,
depth=depth + 1,
priority=0
)
self.db.save_discovery_source(
url=link,
discovery_type="page_internal_link",
source_url=final_url
)
self.db.save_page(
url=final_url,
title=title,
markdown_path=str(output_path),
status="success",
status_code=response.get(
"status_code",
200
),
markdown_length=len(markdown),
word_count=len(
markdown.split()
),
metadata={
"depth": depth,
"links_found": len(discovered_links),
"assets_found": len(downloaded_assets)
},
etag=response.get("etag"),
last_modified=response.get("last_modified"),
)
logger.debug(
f"Saved markdown: "
f"{output_path}"
)
return CrawlResult(
status_code=response.get("status_code"),
url=final_url,
title=title,
markdown_path=str(output_path),
links=discovered_links,
success=True
)
def extract_links(
self,
base_url,
html
):
"""
Extract crawlable links. Accepts an HTML string or a
parsed BS4 soup/Tag.
"""
if hasattr(html, "select") and hasattr(html, "find_all"):
soup = html
else:
soup = BeautifulSoup(html, "lxml")
discovered = set()
for a in soup.find_all(
"a",
href=True
):
href = a.get("href")
resolved = self.url_manager.resolve(
base_url,
href
)
if not resolved:
continue
if not self.url_manager.is_allowed(
resolved
):
continue
discovered.add(
resolved
)
selectors = [
"a.reference.internal",
".toctree-wrapper a",
".wy-menu a",
".sphinxsidebar a",
".globaltoc a",
".localtoc a",
"link[rel='next']",
"link[rel='prev']",
"link[rel='up']"
]
for selector in selectors:
for node in soup.select(selector):
href = node.get("href")
resolved = self.url_manager.resolve(
base_url,
href
)
if not resolved:
continue
if not self.url_manager.is_allowed(resolved):
continue
discovered.add(resolved)
return list(discovered)
def export_summary(self):
"""
Export crawl summary.
"""
summary_path = (
Path(OUTPUT_DIR)
/ "crawl_summary.json"
)
data = {
"processed_this_run": self.processed_this_run,
"queue_stats": self.db.queue_stats()
}
self.writer.write_json(
summary_path,
data
)
graph_path = (
Path(OUTPUT_DIR)
/ "site_graph.json"
)
self.db.export_site_graph(
graph_path
)
navigation_tree = self.navigation_parser.dedupe_tree(
self.navigation_trees
)
navigation_path = (
Path(OUTPUT_DIR)
/ "navigation_tree.json"
)
self.navigation_parser.export_tree_json(
navigation_tree,
navigation_path
)
summary_md = self.navigation_parser.generate_summary_markdown(
navigation_tree
)
summary_md_path = (
Path(OUTPUT_DIR)
/ "SUMMARY.md"
)
self.writer.write_text(
summary_md_path,
summary_md
)
logger.info(
f"Summary exported: "
f"{summary_path}"
)
async def recycle_browser(self):
"""
Restart Playwright browser/context periodically
to avoid long-running memory/socket leaks.
"""
logger.info("Recycling browser context...")
await self.client.stop()
await asyncio.sleep(2)
await self.client.start()
logger.info("Browser context recycled.")
async def async_main():
crawler = DocumentationCrawler()
await crawler.crawl()
def main():
asyncio.run(
async_main()
)
if __name__ == "__main__":
main()

0
models/__init__.py Normal file
View file

228
models/crawl_result.py Normal file
View file

@ -0,0 +1,228 @@
from dataclasses import dataclass, field
from typing import Optional, List, Dict, Any
from datetime import datetime, timezone
def _utcnow_naive() -> datetime:
"""
Replacement for the deprecated datetime.utcnow().
Returns a naive UTC datetime (no tzinfo) to preserve
the historical wire format.
"""
return datetime.now(timezone.utc).replace(tzinfo=None)
@dataclass
class CrawlResult:
"""
Result object for a crawled page.
"""
# Core
url: str
title: str
# Content
raw_html: Optional[str] = None
markdown: Optional[str] = None
markdown_path: Optional[str] = None
# Status
status_code: Optional[int] = None
success: bool = True
error_message: Optional[str] = None
# Links (ONLY ONE SOURCE OF TRUTH)
links: List[str] = field(default_factory=list)
# Timing
crawled_at: datetime = field(default_factory=_utcnow_naive)
# Assets
assets: List[Dict[str, Any]] = field(default_factory=list)
# Navigation
navigation_tree: Optional[list] = None
# Stats
content_length: int = 0
markdown_length: int = 0
word_count: int = 0
# Metadata
tags: List[str] = field(default_factory=list)
metadata: Dict[str, Any] = field(default_factory=dict)
def compute_statistics(self):
"""
Compute markdown statistics.
"""
if not self.markdown:
return
self.markdown_length = len(
self.markdown
)
self.word_count = len(
self.markdown.split()
)
self.content_length = len(
self.markdown or ""
)
def add_asset(
self,
asset_url: str,
local_path: str
):
"""
Add downloaded asset.
"""
self.assets.append({
"url": asset_url,
"local_path": local_path
})
def add_discovered_link(
self,
url: str
):
"""
Add discovered URL.
"""
if url not in self.links:
self.links.append(url)
def mark_failed(
self,
error_message: str
):
"""
Mark crawl as failed.
"""
self.success = False
self.error_message = error_message
def to_dict(self):
"""
Convert object to serializable dict.
"""
return {
"url": self.url,
"title": self.title,
"markdown_path": self.markdown_path,
"success": self.success,
"error_message": self.error_message,
"status_code": self.status_code,
"crawled_at": self.crawled_at.isoformat(),
"assets": self.assets,
"links": self.links,
"content_length": self.content_length,
"markdown_length": self.markdown_length,
"word_count": self.word_count,
"tags": self.tags,
"metadata": self.metadata
}
@classmethod
def from_dict(
cls,
data: dict
):
"""
Restore CrawlResult from dict.
"""
obj = cls(
url=data.get("url", ""),
title=data.get("title", "")
)
obj.markdown_path = data.get(
"markdown_path"
)
obj.success = data.get(
"success",
True
)
obj.error_message = data.get(
"error_message"
)
obj.status_code = data.get(
"status_code"
)
obj.assets = data.get(
"assets",
[]
)
obj.links = data.get(
"links",
[]
)
obj.content_length = data.get(
"content_length",
0
)
obj.markdown_length = data.get(
"markdown_length",
0
)
obj.word_count = data.get(
"word_count",
0
)
obj.tags = data.get(
"tags",
[]
)
obj.metadata = data.get(
"metadata",
{}
)
crawled_at = data.get(
"crawled_at"
)
if crawled_at:
obj.crawled_at = datetime.fromisoformat(
crawled_at
)
return obj
def summary(self):
"""
Human-readable summary.
"""
return (
f"CrawlResult("
f"url={self.url}, "
f"success={self.success}, "
f"markdown_length={self.markdown_length}, "
f"assets={len(self.assets)}, "
f"links={len(self.links)}"
f")"
)

264
models/page.py Normal file
View file

@ -0,0 +1,264 @@
from dataclasses import dataclass, field
from typing import Optional, List, Dict
from datetime import datetime, timezone
def _utcnow_naive() -> datetime:
"""
Replacement for the deprecated datetime.utcnow().
Returns a naive UTC datetime (no tzinfo) to preserve
the historical wire format.
"""
return datetime.now(timezone.utc).replace(tzinfo=None)
@dataclass
class Page:
"""
Raw rendered page object.
Represents a fetched documentation page
before processing pipeline.
"""
# Core metadata
url: str
title: str
# Raw content
html: str
# HTTP metadata
status_code: Optional[int] = None
headers: Dict[str, str] = field(
default_factory=dict
)
# Render metadata
rendered: bool = True
# Timing
fetched_at: datetime = field(
default_factory=_utcnow_naive
)
# Content metadata
content_type: Optional[str] = None
encoding: Optional[str] = "utf-8"
# Link extraction
discovered_links: List[str] = field(
default_factory=list
)
# Asset extraction
assets: List[str] = field(
default_factory=list
)
# Optional processed outputs
markdown: Optional[str] = None
local_path: Optional[str] = None
# Crawl state
depth: int = 0
parent_url: Optional[str] = None
# Error handling
success: bool = True
error_message: Optional[str] = None
# Extra metadata
metadata: Dict = field(
default_factory=dict
)
def add_link(
self,
url: str
):
"""
Add discovered URL.
"""
if url not in self.discovered_links:
self.discovered_links.append(url)
def add_asset(
self,
asset_url: str
):
"""
Add asset URL.
"""
if asset_url not in self.assets:
self.assets.append(asset_url)
def mark_failed(
self,
message: str
):
"""
Mark page fetch failed.
"""
self.success = False
self.error_message = message
def html_length(self):
"""
Return HTML size.
"""
return len(self.html or "")
def word_count(self):
"""
Approximate word count.
"""
text = self.html or ""
return len(text.split())
def to_dict(self):
"""
Serialize object.
"""
return {
"url": self.url,
"title": self.title,
"status_code": self.status_code,
"headers": self.headers,
"rendered": self.rendered,
"fetched_at": self.fetched_at.isoformat(),
"content_type": self.content_type,
"encoding": self.encoding,
"discovered_links": self.discovered_links,
"assets": self.assets,
"markdown": self.markdown,
"local_path": self.local_path,
"depth": self.depth,
"parent_url": self.parent_url,
"success": self.success,
"error_message": self.error_message,
"metadata": self.metadata
}
@classmethod
def from_dict(
cls,
data: dict
):
"""
Restore from serialized dict.
"""
page = cls(
url=data["url"],
title=data.get("title", ""),
html=data.get("html", "")
)
page.status_code = data.get(
"status_code"
)
page.headers = data.get(
"headers",
{}
)
page.rendered = data.get(
"rendered",
True
)
page.content_type = data.get(
"content_type"
)
page.encoding = data.get(
"encoding",
"utf-8"
)
page.discovered_links = data.get(
"discovered_links",
[]
)
page.assets = data.get(
"assets",
[]
)
page.markdown = data.get(
"markdown"
)
page.local_path = data.get(
"local_path"
)
page.depth = data.get(
"depth",
0
)
page.parent_url = data.get(
"parent_url"
)
page.success = data.get(
"success",
True
)
page.error_message = data.get(
"error_message"
)
page.metadata = data.get(
"metadata",
{}
)
fetched_at = data.get(
"fetched_at"
)
if fetched_at:
page.fetched_at = datetime.fromisoformat(
fetched_at
)
return page
def summary(self):
"""
Human-readable summary.
"""
return (
f"Page("
f"url={self.url}, "
f"title={self.title}, "
f"status={self.status_code}, "
f"links={len(self.discovered_links)}, "
f"assets={len(self.assets)}, "
f"success={self.success}"
f")"
)

64
pyproject.toml Normal file
View file

@ -0,0 +1,64 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "intra-mart-doc-crawler"
version = "1.0.0"
description = "Production-grade documentation crawler and markdown exporter"
authors = [
{ name = "Iric Do" }
]
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"playwright>=1.53.0",
"beautifulsoup4>=4.12.3",
"lxml>=5.2.2",
"httpx>=0.28.1",
"aiofiles>=24.1.0",
"loguru>=0.7.2",
"tenacity>=9.0.0",
"markdownify>=0.13.1",
"PyYAML>=6.0.2"
]
[project.optional-dependencies]
dev = [
"black>=24.4.2",
"ruff>=0.5.0",
"pytest>=8.2.2",
"pytest-asyncio>=0.23.7",
"mypy>=1.10.1"
]
[tool.setuptools]
include-package-data = true
[tool.setuptools.packages.find]
include = [
"crawler*",
"converter*",
"extractor*",
"models*",
"storage*",
"utils*"
]
exclude = [
"output*",
"data*",
"logs*"
]
[tool.black]
line-length = 79
[tool.ruff]
line-length = 79
target-version = "py311"
[tool.pytest.ini_options]
asyncio_mode = "auto"

12
requirements.txt Normal file
View file

@ -0,0 +1,12 @@
playwright
beautifulsoup4
lxml
markdownify
pypandoc
loguru
pyyaml
httpx
aiofiles
tenacity
typer
rich

0
storage/__init__.py Normal file
View file

414
storage/file_writer.py Normal file
View file

@ -0,0 +1,414 @@
import json
import os
import shutil
import tempfile
from pathlib import Path
import aiofiles
from utils.logger import logger
class FileWriter:
"""
Production-ready file writer.
Supports:
- text
- json
- binary
- async write
- atomic write
- safe overwrite
"""
def __init__(
self,
encoding="utf-8"
):
self.encoding = encoding
def ensure_parent(
self,
path: Path
):
"""
Ensure parent directory exists.
"""
path.parent.mkdir(
parents=True,
exist_ok=True
)
def write_text(
self,
path: Path,
content: str,
overwrite=True
):
"""
Write text file safely.
"""
self.ensure_parent(path)
if path.exists() and not overwrite:
logger.warning(
f"Skip existing file: {path}"
)
return
# Create the temp file in the SAME directory as the
# destination so that shutil.move can do a same-FS
# atomic rename. mkstemp on /tmp can land on a
# different mount point, forcing a non-atomic
# copy+delete fallback. Also: mkstemp returns a raw
# fd that must be closed -- the old code leaked one
# fd per write_text call.
temp_fd, temp_path = tempfile.mkstemp(
prefix=".tmp_",
suffix=path.suffix,
dir=str(path.parent)
)
try:
with os.fdopen(
temp_fd,
"w",
encoding=self.encoding
) as f:
f.write(content)
shutil.move(
temp_path,
path
)
logger.info(
f"Saved text file: {path}"
)
except Exception as e:
logger.exception(
f"Failed writing text "
f"{path}: {e}"
)
# Clean up the temp file if move never happened.
try:
Path(temp_path).unlink(missing_ok=True)
except Exception:
pass
raise
async def write_text_async(
self,
path: Path,
content: str
):
"""
Async text writer.
"""
self.ensure_parent(path)
try:
async with aiofiles.open(
path,
"w",
encoding=self.encoding
) as f:
await f.write(content)
logger.info(
f"Saved async text: {path}"
)
except Exception as e:
logger.exception(
f"Async write failed "
f"{path}: {e}"
)
raise
def write_json(
self,
path: Path,
data,
indent=2
):
"""
Write JSON file.
"""
self.ensure_parent(path)
try:
with open(
path,
"w",
encoding=self.encoding
) as f:
json.dump(
data,
f,
ensure_ascii=False,
indent=indent
)
logger.info(
f"Saved JSON: {path}"
)
except Exception as e:
logger.exception(
f"JSON write failed "
f"{path}: {e}"
)
raise
async def write_json_async(
self,
path: Path,
data,
indent=2
):
"""
Async JSON writer.
"""
self.ensure_parent(path)
try:
content = json.dumps(
data,
ensure_ascii=False,
indent=indent
)
async with aiofiles.open(
path,
"w",
encoding=self.encoding
) as f:
await f.write(content)
logger.info(
f"Saved async JSON: {path}"
)
except Exception as e:
logger.exception(
f"Async JSON write failed "
f"{path}: {e}"
)
raise
def write_binary(
self,
path: Path,
content: bytes
):
"""
Write binary file.
"""
self.ensure_parent(path)
try:
with open(
path,
"wb"
) as f:
f.write(content)
logger.info(
f"Saved binary: {path}"
)
except Exception as e:
logger.exception(
f"Binary write failed "
f"{path}: {e}"
)
raise
async def append_text(
self,
path: Path,
content: str
):
"""
Append text to file.
"""
self.ensure_parent(path)
try:
async with aiofiles.open(
path,
"a",
encoding=self.encoding
) as f:
await f.write(content)
logger.info(
f"Appended text: {path}"
)
except Exception as e:
logger.exception(
f"Append failed "
f"{path}: {e}"
)
raise
def read_text(
self,
path: Path
):
"""
Read text file.
"""
try:
with open(
path,
"r",
encoding=self.encoding
) as f:
return f.read()
except Exception as e:
logger.exception(
f"Read failed "
f"{path}: {e}"
)
raise
def exists(
self,
path: Path
):
"""
Check file existence.
"""
return path.exists()
def delete(
self,
path: Path
):
"""
Delete file safely.
"""
try:
if path.exists():
path.unlink()
logger.info(
f"Deleted file: {path}"
)
except Exception as e:
logger.exception(
f"Delete failed "
f"{path}: {e}"
)
def sanitize_filename(
self,
filename: str
):
"""
Remove invalid filesystem chars.
"""
invalid_chars = [
"<",
">",
":",
"\"",
"/",
"\\",
"|",
"?",
"*"
]
for char in invalid_chars:
filename = filename.replace(
char,
"_"
)
return filename.strip()
def write_markdown_with_metadata(
self,
path: Path,
markdown: str,
metadata: dict = None
):
"""
Write markdown with YAML frontmatter.
"""
self.ensure_parent(path)
content = ""
if metadata:
content += "---\n"
for key, value in metadata.items():
content += f"{key}: {value}\n"
content += "---\n\n"
content += markdown
self.write_text(
path,
content
)

935
storage/metadata_db.py Normal file
View file

@ -0,0 +1,935 @@
import json
import sqlite3
from contextlib import contextmanager
from pathlib import Path
from datetime import datetime, timezone
from utils.logger import logger
def _utcnow_iso() -> str:
"""
Return a naive UTC ISO timestamp.
datetime.utcnow() is deprecated in Python 3.12+; this
helper preserves the historical naive-UTC string format
(no '+00:00' suffix) used throughout the schema so
existing rows compare correctly.
"""
return datetime.now(timezone.utc).replace(
tzinfo=None
).isoformat()
class MetadataDB:
"""
Metadata storage layer
for crawler state tracking.
Stores:
- crawled pages
- crawl status
- markdown outputs
- assets
- hashes
- timestamps
- incremental update info
"""
def __init__(
self,
db_path="data/metadata.db"
):
self.db_path = Path(db_path)
self.db_path.parent.mkdir(
parents=True,
exist_ok=True
)
self.conn = sqlite3.connect(
self.db_path
)
self.conn.row_factory = sqlite3.Row
# depth > 0 means we're inside a batched transaction
# and per-row commits should be suppressed
self._tx_depth = 0
self.initialize()
def initialize(self):
"""
Create required tables.
"""
cursor = self.conn.cursor()
cursor.execute("PRAGMA journal_mode=WAL")
cursor.execute("PRAGMA synchronous=NORMAL")
cursor.execute("PRAGMA temp_store=MEMORY")
cursor.execute("PRAGMA cache_size=-65536")
# Pages table
cursor.execute("""
CREATE TABLE IF NOT EXISTS pages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
url TEXT UNIQUE,
title TEXT,
markdown_path TEXT,
status TEXT,
status_code INTEGER,
content_hash TEXT,
word_count INTEGER,
markdown_length INTEGER,
crawled_at TEXT,
updated_at TEXT,
parent_url TEXT,
depth INTEGER,
metadata_json TEXT
)
""")
# Assets table
cursor.execute("""
CREATE TABLE IF NOT EXISTS assets (
id INTEGER PRIMARY KEY AUTOINCREMENT,
page_url TEXT,
asset_url TEXT,
local_path TEXT,
downloaded_at TEXT
)
""")
# Site graph edges
cursor.execute("""
CREATE TABLE IF NOT EXISTS page_links (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_url TEXT NOT NULL,
target_url TEXT NOT NULL,
link_type TEXT DEFAULT 'internal',
discovered_at TEXT,
UNIQUE(source_url, target_url)
)
""")
# Discovery provenance
cursor.execute("""
CREATE TABLE IF NOT EXISTS discovery_sources (
id INTEGER PRIMARY KEY AUTOINCREMENT,
url TEXT NOT NULL,
discovery_type TEXT,
source_url TEXT,
discovered_at TEXT
)
""")
# Crawl queue state
cursor.execute("""
CREATE TABLE IF NOT EXISTS crawl_queue (
url TEXT PRIMARY KEY,
status TEXT NOT NULL,
priority INTEGER DEFAULT 0,
depth INTEGER DEFAULT 0,
parent_url TEXT,
discovered_at TEXT,
started_at TEXT,
finished_at TEXT,
attempts INTEGER DEFAULT 0,
last_error TEXT
)
""")
# Failed pages
cursor.execute("""
CREATE TABLE IF NOT EXISTS failures (
id INTEGER PRIMARY KEY AUTOINCREMENT,
url TEXT,
error_message TEXT,
failed_at TEXT
)
""")
self.conn.commit()
# Add the indexes that drive the hot path. IF NOT
# EXISTS makes this safe on existing databases.
cursor.execute(
"CREATE INDEX IF NOT EXISTS "
"idx_crawl_queue_pick "
"ON crawl_queue(status, priority DESC, depth, "
"discovered_at)"
)
cursor.execute(
"CREATE INDEX IF NOT EXISTS "
"idx_page_links_source "
"ON page_links(source_url)"
)
# Conditional-GET columns. SQLite has no "ADD COLUMN
# IF NOT EXISTS" so swallow the duplicate-column error
# when running against an existing database.
for column in ("etag TEXT", "last_modified TEXT"):
try:
cursor.execute(
f"ALTER TABLE pages ADD COLUMN {column}"
)
except sqlite3.OperationalError as e:
if "duplicate column" not in str(e).lower():
raise
self.conn.commit()
logger.info(
"Metadata database initialized."
)
@contextmanager
def transaction(self):
"""
Batch many writes into one commit (one fsync).
Nested usage is supported: only the outermost block
actually issues BEGIN/COMMIT. This lets save_* helpers
stay safe to call individually while also being cheap
when wrapped in a per-page transaction.
"""
if self._tx_depth == 0:
self.conn.execute("BEGIN")
self._tx_depth += 1
try:
yield
except Exception:
self._tx_depth -= 1
if self._tx_depth == 0:
self.conn.rollback()
raise
else:
self._tx_depth -= 1
if self._tx_depth == 0:
self.conn.commit()
def _commit(self):
"""
Commit unless we are inside a batched transaction.
"""
if self._tx_depth == 0:
self.conn.commit()
def save_page(
self,
url,
title="",
markdown_path="",
status="success",
status_code=200,
content_hash="",
word_count=0,
markdown_length=0,
parent_url=None,
depth=0,
metadata=None,
etag=None,
last_modified=None,
):
"""
Insert or update page metadata.
"""
cursor = self.conn.cursor()
now = _utcnow_iso()
metadata_json = json.dumps(
metadata or {},
ensure_ascii=False
)
cursor.execute(
"""
INSERT INTO pages (
url,
title,
markdown_path,
status,
status_code,
content_hash,
word_count,
markdown_length,
crawled_at,
updated_at,
parent_url,
depth,
metadata_json,
etag,
last_modified
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(url)
DO UPDATE SET
title=excluded.title,
markdown_path=excluded.markdown_path,
status=excluded.status,
status_code=excluded.status_code,
content_hash=excluded.content_hash,
word_count=excluded.word_count,
markdown_length=excluded.markdown_length,
updated_at=excluded.updated_at,
metadata_json=excluded.metadata_json,
etag=COALESCE(excluded.etag, pages.etag),
last_modified=COALESCE(
excluded.last_modified,
pages.last_modified
)
""",
(
url,
title,
markdown_path,
status,
status_code,
content_hash,
word_count,
markdown_length,
now,
now,
parent_url,
depth,
metadata_json,
etag,
last_modified,
),
)
self._commit()
logger.debug(
f"Saved page metadata: {url}"
)
def get_cached_headers(self, url):
"""
Return (etag, last_modified) for an already-crawled
URL, or (None, None) if unknown.
"""
cursor = self.conn.cursor()
cursor.execute(
"SELECT etag, last_modified FROM pages WHERE url=?",
(url,),
)
row = cursor.fetchone()
if not row:
return None, None
return row["etag"], row["last_modified"]
def save_asset(
self,
page_url,
asset_url,
local_path
):
"""
Save asset metadata.
"""
cursor = self.conn.cursor()
cursor.execute("""
INSERT INTO assets (
page_url,
asset_url,
local_path,
downloaded_at
)
VALUES (?, ?, ?, ?)
""", (
page_url,
asset_url,
local_path,
_utcnow_iso()
))
self._commit()
def save_page_link(
self,
source_url,
target_url,
link_type="internal"
):
cursor = self.conn.cursor()
cursor.execute("""
INSERT OR IGNORE INTO page_links (
source_url,
target_url,
link_type,
discovered_at
)
VALUES (?, ?, ?, ?)
""", (
source_url,
target_url,
link_type,
_utcnow_iso()
))
self._commit()
def save_discovery_source(
self,
url,
discovery_type,
source_url=None
):
cursor = self.conn.cursor()
cursor.execute("""
INSERT INTO discovery_sources (
url,
discovery_type,
source_url,
discovered_at
)
VALUES (?, ?, ?, ?)
""", (
url,
discovery_type,
source_url,
_utcnow_iso()
))
self._commit()
def get_outgoing_links(
self,
source_url
):
cursor = self.conn.cursor()
cursor.execute("""
SELECT target_url
FROM page_links
WHERE source_url=?
""", (source_url,))
return [
row["target_url"]
for row in cursor.fetchall()
]
def export_site_graph(
self,
output_path
):
cursor = self.conn.cursor()
cursor.execute("""
SELECT
source_url,
target_url,
link_type
FROM page_links
""")
rows = cursor.fetchall()
graph = [
dict(row)
for row in rows
]
with open(
output_path,
"w",
encoding="utf-8"
) as f:
json.dump(
graph,
f,
ensure_ascii=False,
indent=2
)
logger.info(
f"Exported site graph: {output_path}"
)
def save_failure(
self,
url,
error_message
):
"""
Save failed crawl.
"""
cursor = self.conn.cursor()
cursor.execute("""
INSERT INTO failures (
url,
error_message,
failed_at
)
VALUES (?, ?, ?)
""", (
url,
error_message,
_utcnow_iso()
))
self._commit()
logger.warning(
f"Saved failure: {url}"
)
def enqueue_url(
self,
url,
parent_url=None,
depth=0,
priority=0
):
"""
Save crawl queue item.
"""
cursor = self.conn.cursor()
cursor.execute("""
INSERT OR IGNORE INTO crawl_queue (
url,
status,
priority,
depth,
parent_url,
discovered_at,
attempts
)
VALUES (?, ?, ?, ?, ?, ?, ?)
""", (
url,
"pending",
priority,
depth,
parent_url,
_utcnow_iso(),
0
))
self._commit()
def update_queue_status(
self,
url,
status
):
"""
Update queue item status.
"""
cursor = self.conn.cursor()
cursor.execute("""
UPDATE crawl_queue
SET status=?
WHERE url=?
""", (
status,
url
))
self._commit()
def page_exists(
self,
url
):
"""
Check if page already crawled.
"""
cursor = self.conn.cursor()
cursor.execute("""
SELECT 1
FROM pages
WHERE url=?
LIMIT 1
""", (url,))
row = cursor.fetchone()
return row is not None
def get_page(
self,
url
):
"""
Retrieve page metadata.
"""
cursor = self.conn.cursor()
cursor.execute("""
SELECT *
FROM pages
WHERE url=?
""", (url,))
row = cursor.fetchone()
if not row:
return None
return dict(row)
def get_all_pages(self):
"""
Get all crawled pages.
"""
cursor = self.conn.cursor()
cursor.execute("""
SELECT *
FROM pages
ORDER BY crawled_at DESC
""")
rows = cursor.fetchall()
return [
dict(row)
for row in rows
]
def get_failed_pages(self):
"""
Get all failures.
"""
cursor = self.conn.cursor()
cursor.execute("""
SELECT *
FROM failures
ORDER BY failed_at DESC
""")
rows = cursor.fetchall()
return [
dict(row)
for row in rows
]
def get_pending_queue(self):
"""
Get pending URLs.
"""
cursor = self.conn.cursor()
cursor.execute("""
SELECT *
FROM crawl_queue
WHERE status='pending'
""")
rows = cursor.fetchall()
return [
dict(row)
for row in rows
]
def get_next_pending_url(self):
cursor = self.conn.cursor()
cursor.execute("""
SELECT *
FROM crawl_queue
WHERE status IN ('pending', 'retry')
ORDER BY priority DESC, depth ASC, discovered_at ASC
LIMIT 1
""")
row = cursor.fetchone()
return dict(row) if row else None
def mark_processing(self, url):
cursor = self.conn.cursor()
cursor.execute("""
UPDATE crawl_queue
SET
status='processing',
started_at=?,
attempts=attempts + 1
WHERE url=?
""", (
_utcnow_iso(),
url
))
self._commit()
def mark_done(self, url):
cursor = self.conn.cursor()
cursor.execute("""
UPDATE crawl_queue
SET
status='done',
finished_at=?
WHERE url=?
""", (
_utcnow_iso(),
url
))
self._commit()
def mark_failed(self, url, error_message, max_attempts=3):
cursor = self.conn.cursor()
cursor.execute("""
SELECT attempts
FROM crawl_queue
WHERE url=?
""", (url,))
row = cursor.fetchone()
attempts = row["attempts"] if row else 0
next_status = "failed" if attempts >= max_attempts else "retry"
cursor.execute("""
UPDATE crawl_queue
SET
status=?,
last_error=?,
finished_at=?
WHERE url=?
""", (
next_status,
error_message,
_utcnow_iso(),
url
))
cursor.execute("""
INSERT INTO failures (
url,
error_message,
failed_at
)
VALUES (?, ?, ?)
""", (
url,
error_message,
_utcnow_iso()
))
self._commit()
logger.warning(
f"Marked URL as {next_status}: {url}"
)
def mark_permanently_failed(self, url, error_message):
"""
Skip the retry ramp and mark the URL as terminally
failed. Used for hard 4xx (e.g. 404) where retrying
the same URL will produce the same response.
"""
cursor = self.conn.cursor()
cursor.execute(
"""
UPDATE crawl_queue
SET
status='failed',
last_error=?,
finished_at=?
WHERE url=?
""",
(
error_message,
_utcnow_iso(),
url,
),
)
cursor.execute(
"""
INSERT INTO failures (
url,
error_message,
failed_at
)
VALUES (?, ?, ?)
""",
(url, error_message, _utcnow_iso()),
)
self._commit()
logger.warning(
f"Marked URL permanently failed: {url} ({error_message})"
)
def reset_stuck_processing(self):
cursor = self.conn.cursor()
cursor.execute("""
UPDATE crawl_queue
SET status='retry'
WHERE status='processing'
""")
affected = cursor.rowcount
self._commit()
if affected:
logger.warning(
f"Recovered {affected} stuck processing URLs."
)
def is_done(self, url):
cursor = self.conn.cursor()
cursor.execute("""
SELECT 1
FROM crawl_queue
WHERE url=? AND status='done'
LIMIT 1
""", (url,))
return cursor.fetchone() is not None
def is_known(self, url):
cursor = self.conn.cursor()
cursor.execute("""
SELECT 1
FROM crawl_queue
WHERE url=?
LIMIT 1
""", (url,))
return cursor.fetchone() is not None
def queue_stats(self):
cursor = self.conn.cursor()
cursor.execute("""
SELECT status, COUNT(*) AS count
FROM crawl_queue
GROUP BY status
""")
return {
row["status"]: row["count"]
for row in cursor.fetchall()
}
def delete_page(
self,
url
):
"""
Remove page metadata.
"""
cursor = self.conn.cursor()
cursor.execute("""
DELETE FROM pages
WHERE url=?
""", (url,))
self._commit()
def clear_failures(self):
"""
Clear failure table.
"""
cursor = self.conn.cursor()
cursor.execute("""
DELETE FROM failures
""")
self._commit()
def export_pages_json(
self,
output_path
):
"""
Export all pages metadata.
"""
pages = self.get_all_pages()
with open(
output_path,
"w",
encoding="utf-8"
) as f:
json.dump(
pages,
f,
ensure_ascii=False,
indent=2
)
logger.info(
f"Exported metadata JSON: "
f"{output_path}"
)
def close(self):
"""
Close database connection.
"""
self.conn.close()
logger.info(
"Metadata database closed."
)

314
storage/path_mapper.py Normal file
View file

@ -0,0 +1,314 @@
import hashlib
from pathlib import Path
from urllib.parse import urlparse
from utils.logger import logger
class PathMapper:
"""
Production-ready URL -> filesystem mapper.
Handles:
- structure preservation
- collision prevention
- filename sanitization
- cross-platform safety
"""
def __init__(
self,
output_dir,
include_domain=True,
max_filename_length=180
):
self.output_dir = Path(
output_dir
)
self.include_domain = (
include_domain
)
self.max_filename_length = (
max_filename_length
)
def url_to_path(
self,
url: str,
extension=".md"
):
"""
Convert URL -> local file path.
"""
parsed = urlparse(url)
domain = parsed.netloc
path = parsed.path.strip("/")
# Homepage
if not path:
path = "index"
# Remove trailing slash
path = path.rstrip("/")
# Remove html extension
if path.endswith(".html"):
path = path[:-5]
# Handle query params
if parsed.query:
query_hash = self.hash_string(
parsed.query
)[:10]
path += f"__{query_hash}"
# Sanitize each segment
segments = []
for segment in path.split("/"):
segment = self.sanitize_filename(
segment
)
if not segment:
segment = "untitled"
segments.append(segment)
path = "/".join(segments)
# Append extension
path += extension
# Include domain
if self.include_domain:
output_path = (
self.output_dir /
domain /
path
)
else:
output_path = (
self.output_dir /
path
)
# Prevent very long filenames
output_path = self.shorten_if_needed(
output_path
)
output_path.parent.mkdir(
parents=True,
exist_ok=True
)
logger.debug(
f"Mapped URL -> path: "
f"{url} => {output_path}"
)
return output_path
def asset_url_to_path(
self,
url: str
):
"""
Map asset URL to asset path.
"""
return self.url_to_path(
url,
extension=""
)
def sanitize_filename(
self,
filename: str
):
"""
Remove filesystem-invalid chars.
"""
invalid_chars = [
"<",
">",
":",
"\"",
"/",
"\\",
"|",
"?",
"*",
"\n",
"\r",
"\t"
]
for char in invalid_chars:
filename = filename.replace(
char,
"_"
)
filename = filename.strip()
# Windows reserved names -- the OS blocks these
# regardless of extension, so e.g. "CON.txt" is
# also unusable on Windows. Check the stem (the
# portion before the first dot) against the full
# reserved set: CON, PRN, AUX, NUL, COM1..COM9,
# LPT1..LPT9.
reserved = {
"CON",
"PRN",
"AUX",
"NUL",
}
for i in range(1, 10):
reserved.add(f"COM{i}")
reserved.add(f"LPT{i}")
stem = filename.split(".", 1)[0]
if stem.upper() in reserved:
filename = "_" + filename
return filename
def shorten_if_needed(
self,
path: Path
):
"""
Prevent excessively long filenames.
"""
filename = path.name
if len(filename) <= self.max_filename_length:
return path
suffix = path.suffix
stem = path.stem
hash_part = self.hash_string(
filename
)[:12]
shortened = (
stem[:100]
+ "__"
+ hash_part
+ suffix
)
return path.with_name(
shortened
)
def hash_string(
self,
value: str
):
"""
Stable hash helper.
"""
return hashlib.md5(
value.encode("utf-8")
).hexdigest()
def relative_path(
self,
path: Path
):
"""
Get relative path from output dir.
"""
try:
return path.relative_to(
self.output_dir
)
except Exception:
return path
def markdown_path_to_url(
self,
markdown_path: Path
):
"""
Reverse mapping helper.
"""
relative = markdown_path.relative_to(
self.output_dir
)
return str(relative)
def ensure_unique_path(
self,
path: Path
):
"""
Prevent overwrite collisions.
"""
if not path.exists():
return path
counter = 1
while True:
candidate = path.with_name(
f"{path.stem}_{counter}"
f"{path.suffix}"
)
if not candidate.exists():
return candidate
counter += 1
def stats(
self
):
"""
Path mapper statistics.
"""
return {
"output_dir": str(
self.output_dir
),
"include_domain": (
self.include_domain
)
}

0
utils/__init__.py Normal file
View file

481
utils/helpers.py Normal file
View file

@ -0,0 +1,481 @@
import json
import sqlite3
from pathlib import Path
from datetime import datetime
from utils.logger import logger
class MetadataDB:
"""
Metadata storage layer
for crawler state tracking.
Stores:
- crawled pages
- crawl status
- markdown outputs
- assets
- hashes
- timestamps
- incremental update info
"""
def __init__(
self,
db_path="data/metadata.db"
):
self.db_path = Path(db_path)
self.db_path.parent.mkdir(
parents=True,
exist_ok=True
)
self.conn = sqlite3.connect(
self.db_path
)
self.conn.row_factory = sqlite3.Row
self.initialize()
def initialize(self):
"""
Create required tables.
"""
cursor = self.conn.cursor()
# Pages table
cursor.execute("""
CREATE TABLE IF NOT EXISTS pages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
url TEXT UNIQUE,
title TEXT,
markdown_path TEXT,
status TEXT,
status_code INTEGER,
content_hash TEXT,
word_count INTEGER,
markdown_length INTEGER,
crawled_at TEXT,
updated_at TEXT,
parent_url TEXT,
depth INTEGER,
metadata_json TEXT
)
""")
# Assets table
cursor.execute("""
CREATE TABLE IF NOT EXISTS assets (
id INTEGER PRIMARY KEY AUTOINCREMENT,
page_url TEXT,
asset_url TEXT,
local_path TEXT,
downloaded_at TEXT
)
""")
# Crawl queue state
cursor.execute("""
CREATE TABLE IF NOT EXISTS crawl_queue (
id INTEGER PRIMARY KEY AUTOINCREMENT,
url TEXT UNIQUE,
status TEXT,
discovered_at TEXT
)
""")
# Failed pages
cursor.execute("""
CREATE TABLE IF NOT EXISTS failures (
id INTEGER PRIMARY KEY AUTOINCREMENT,
url TEXT,
error_message TEXT,
failed_at TEXT
)
""")
self.conn.commit()
logger.info(
"Metadata database initialized."
)
def save_page(
self,
url,
title="",
markdown_path="",
status="success",
status_code=200,
content_hash="",
word_count=0,
markdown_length=0,
parent_url=None,
depth=0,
metadata=None
):
"""
Insert or update page metadata.
"""
cursor = self.conn.cursor()
now = datetime.utcnow().isoformat()
metadata_json = json.dumps(
metadata or {},
ensure_ascii=False
)
cursor.execute("""
INSERT INTO pages (
url,
title,
markdown_path,
status,
status_code,
content_hash,
word_count,
markdown_length,
crawled_at,
updated_at,
parent_url,
depth,
metadata_json
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(url)
DO UPDATE SET
title=excluded.title,
markdown_path=excluded.markdown_path,
status=excluded.status,
status_code=excluded.status_code,
content_hash=excluded.content_hash,
word_count=excluded.word_count,
markdown_length=excluded.markdown_length,
updated_at=excluded.updated_at,
metadata_json=excluded.metadata_json
""", (
url,
title,
markdown_path,
status,
status_code,
content_hash,
word_count,
markdown_length,
now,
now,
parent_url,
depth,
metadata_json
))
self.conn.commit()
logger.info(
f"Saved page metadata: {url}"
)
def save_asset(
self,
page_url,
asset_url,
local_path
):
"""
Save asset metadata.
"""
cursor = self.conn.cursor()
cursor.execute("""
INSERT INTO assets (
page_url,
asset_url,
local_path,
downloaded_at
)
VALUES (?, ?, ?, ?)
""", (
page_url,
asset_url,
local_path,
datetime.utcnow().isoformat()
))
self.conn.commit()
def save_failure(
self,
url,
error_message
):
"""
Save failed crawl.
"""
cursor = self.conn.cursor()
cursor.execute("""
INSERT INTO failures (
url,
error_message,
failed_at
)
VALUES (?, ?, ?)
""", (
url,
error_message,
datetime.utcnow().isoformat()
))
self.conn.commit()
logger.warning(
f"Saved failure: {url}"
)
def enqueue_url(
self,
url,
status="pending"
):
"""
Save crawl queue item.
"""
cursor = self.conn.cursor()
cursor.execute("""
INSERT OR IGNORE INTO crawl_queue (
url,
status,
discovered_at
)
VALUES (?, ?, ?)
""", (
url,
status,
datetime.utcnow().isoformat()
))
self.conn.commit()
def update_queue_status(
self,
url,
status
):
"""
Update queue item status.
"""
cursor = self.conn.cursor()
cursor.execute("""
UPDATE crawl_queue
SET status=?
WHERE url=?
""", (
status,
url
))
self.conn.commit()
def page_exists(
self,
url
):
"""
Check if page already crawled.
"""
cursor = self.conn.cursor()
cursor.execute("""
SELECT 1
FROM pages
WHERE url=?
LIMIT 1
""", (url,))
row = cursor.fetchone()
return row is not None
def get_page(
self,
url
):
"""
Retrieve page metadata.
"""
cursor = self.conn.cursor()
cursor.execute("""
SELECT *
FROM pages
WHERE url=?
""", (url,))
row = cursor.fetchone()
if not row:
return None
return dict(row)
def get_all_pages(self):
"""
Get all crawled pages.
"""
cursor = self.conn.cursor()
cursor.execute("""
SELECT *
FROM pages
ORDER BY crawled_at DESC
""")
rows = cursor.fetchall()
return [
dict(row)
for row in rows
]
def get_failed_pages(self):
"""
Get all failures.
"""
cursor = self.conn.cursor()
cursor.execute("""
SELECT *
FROM failures
ORDER BY failed_at DESC
""")
rows = cursor.fetchall()
return [
dict(row)
for row in rows
]
def get_pending_queue(self):
"""
Get pending URLs.
"""
cursor = self.conn.cursor()
cursor.execute("""
SELECT *
FROM crawl_queue
WHERE status='pending'
""")
rows = cursor.fetchall()
return [
dict(row)
for row in rows
]
def delete_page(
self,
url
):
"""
Remove page metadata.
"""
cursor = self.conn.cursor()
cursor.execute("""
DELETE FROM pages
WHERE url=?
""", (url,))
self.conn.commit()
def clear_failures(self):
"""
Clear failure table.
"""
cursor = self.conn.cursor()
cursor.execute("""
DELETE FROM failures
""")
self.conn.commit()
def export_pages_json(
self,
output_path
):
"""
Export all pages metadata.
"""
pages = self.get_all_pages()
with open(
output_path,
"w",
encoding="utf-8"
) as f:
json.dump(
pages,
f,
ensure_ascii=False,
indent=2
)
logger.info(
f"Exported metadata JSON: "
f"{output_path}"
)
def close(self):
"""
Close database connection.
"""
self.conn.close()
logger.info(
"Metadata database closed."
)

98
utils/logger.py Normal file
View file

@ -0,0 +1,98 @@
import os
import sys
from pathlib import Path
from loguru import logger
LOG_DIR = Path("logs")
LOG_DIR.mkdir(
parents=True,
exist_ok=True
)
def setup_logger():
"""
Configure production logger.
"""
logger.remove()
# Console logger
logger.add(
sys.stdout,
level="INFO",
colorize=True,
enqueue=True,
backtrace=True,
diagnose=True,
format=(
"<green>{time:YYYY-MM-DD HH:mm:ss}</green> | "
"<level>{level: <8}</level> | "
"<cyan>{name}</cyan>:"
"<cyan>{function}</cyan>:"
"<cyan>{line}</cyan> - "
"<level>{message}</level>"
)
)
# Main crawler log
logger.add(
LOG_DIR / "crawler.log",
level="INFO",
rotation="50 MB",
retention="30 days",
compression="zip",
enqueue=True,
encoding="utf-8",
backtrace=True,
diagnose=True,
format=(
"{time:YYYY-MM-DD HH:mm:ss} | "
"{level: <8} | "
"{name}:{function}:{line} - "
"{message}"
)
)
# Error log
logger.add(
LOG_DIR / "errors.log",
level="ERROR",
rotation="20 MB",
retention="60 days",
compression="zip",
enqueue=True,
encoding="utf-8",
backtrace=True,
diagnose=True,
format=(
"{time:YYYY-MM-DD HH:mm:ss} | "
"{level: <8} | "
"{name}:{function}:{line} - "
"{message}"
)
)
# Debug log: very chatty (every DISCOVERED/ENQUEUED, every
# save_*), so only enable when explicitly asked for.
# Writing+rotating 100 MB of debug log per run silently
# bottlenecks the crawler.
if os.environ.get("CRAWLER_DEBUG"):
logger.add(
LOG_DIR / "debug.log",
level="DEBUG",
rotation="100 MB",
retention="14 days",
compression="zip",
enqueue=True,
encoding="utf-8",
)
logger.info("Logger initialized.")
# Initialize automatically
setup_logger()

275
utils/retry.py Normal file
View file

@ -0,0 +1,275 @@
import asyncio
from functools import wraps
import random
from tenacity import (
retry,
stop_after_attempt,
wait_exponential,
retry_if_exception_type,
before_sleep_log
)
from utils.logger import logger
class RetryConfig:
"""
Global retry configuration.
"""
DEFAULT_ATTEMPTS = 3
DEFAULT_MIN_WAIT = 1
DEFAULT_MAX_WAIT = 10
DEFAULT_JITTER = 0.5
def retry_sync(
attempts=RetryConfig.DEFAULT_ATTEMPTS,
min_wait=RetryConfig.DEFAULT_MIN_WAIT,
max_wait=RetryConfig.DEFAULT_MAX_WAIT,
exceptions=(Exception,)
):
"""
Retry decorator for sync functions.
Example:
@retry_sync()
def task():
...
"""
return retry(
stop=stop_after_attempt(attempts),
wait=wait_exponential(
multiplier=1,
min=min_wait,
max=max_wait
),
retry=retry_if_exception_type(
exceptions
),
before_sleep=before_sleep_log(
logger,
"WARNING"
),
reraise=True
)
def retry_async(
attempts=RetryConfig.DEFAULT_ATTEMPTS,
min_wait=RetryConfig.DEFAULT_MIN_WAIT,
max_wait=RetryConfig.DEFAULT_MAX_WAIT,
jitter=RetryConfig.DEFAULT_JITTER,
exceptions=(Exception,)
):
"""
Retry decorator for async functions.
Example:
@retry_async()
async def fetch():
...
"""
def decorator(func):
@wraps(func)
async def wrapper(*args, **kwargs):
last_exception = None
for attempt in range(
1,
attempts + 1
):
try:
logger.debug(
f"Attempt {attempt}/"
f"{attempts}: "
f"{func.__name__}"
)
return await func(
*args,
**kwargs
)
except exceptions as e:
last_exception = e
logger.warning(
f"Retry failed "
f"{attempt}/{attempts} "
f"for {func.__name__}: {e}"
)
if attempt >= attempts:
break
base_sleep = min(
min_wait * (2 ** (attempt - 1)),
max_wait
)
sleep_time = base_sleep + random.uniform(
0,
jitter
)
logger.info(
f"Sleeping "
f"{sleep_time:.2f}s before retry"
)
await asyncio.sleep(
sleep_time
)
logger.error(
f"Max retries exceeded "
f"for {func.__name__}"
)
raise last_exception
return wrapper
return decorator
async def retry_operation(
coro,
attempts=3,
delay=1,
exceptions=(Exception,)
):
"""
Retry arbitrary async coroutine.
Example:
result = await retry_operation(
fetch_page(url)
)
"""
last_exception = None
for attempt in range(
1,
attempts + 1
):
try:
logger.debug(
f"Retry operation "
f"attempt {attempt}/{attempts}"
)
return await coro
except exceptions as e:
last_exception = e
logger.warning(
f"Retry operation failed "
f"{attempt}/{attempts}: {e}"
)
if attempt >= attempts:
break
await asyncio.sleep(delay)
logger.error(
"Retry operation exceeded "
"max attempts."
)
raise last_exception
def safe_execute(
func,
*args,
default=None,
log_error=True,
**kwargs
):
"""
Execute function safely.
Returns default value if failed.
Example:
result = safe_execute(
parse_html,
html,
default=""
)
"""
try:
return func(
*args,
**kwargs
)
except Exception as e:
if log_error:
logger.exception(
f"Safe execute failed "
f"{func.__name__}: {e}"
)
return default
async def safe_execute_async(
func,
*args,
default=None,
log_error=True,
**kwargs
):
"""
Safe async execution.
"""
try:
return await func(
*args,
**kwargs
)
except Exception as e:
if log_error:
logger.exception(
f"Safe async execute failed "
f"{func.__name__}: {e}"
)
return default

830
uv.lock Normal file
View file

@ -0,0 +1,830 @@
version = 1
revision = 3
requires-python = ">=3.11"
resolution-markers = [
"python_full_version >= '3.15'",
"python_full_version < '3.15'",
]
[[package]]
name = "aiofiles"
version = "25.1.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/41/c3/534eac40372d8ee36ef40df62ec129bee4fdb5ad9706e58a29be53b2c970/aiofiles-25.1.0.tar.gz", hash = "sha256:a8d728f0a29de45dc521f18f07297428d56992a742f0cd2701ba86e44d23d5b2", size = 46354, upload-time = "2025-10-09T20:51:04.358Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/bc/8a/340a1555ae33d7354dbca4faa54948d76d89a27ceef032c8c3bc661d003e/aiofiles-25.1.0-py3-none-any.whl", hash = "sha256:abe311e527c862958650f9438e859c1fa7568a141b22abcd015e120e86a85695", size = 14668, upload-time = "2025-10-09T20:51:03.174Z" },
]
[[package]]
name = "anyio"
version = "4.13.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "idna" },
{ name = "typing-extensions", marker = "python_full_version < '3.13'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/19/14/2c5dd9f512b66549ae92767a9c7b330ae88e1932ca57876909410251fe13/anyio-4.13.0.tar.gz", hash = "sha256:334b70e641fd2221c1505b3890c69882fe4a2df910cba14d97019b90b24439dc", size = 231622, upload-time = "2026-03-24T12:59:09.671Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/da/42/e921fccf5015463e32a3cf6ee7f980a6ed0f395ceeaa45060b61d86486c2/anyio-4.13.0-py3-none-any.whl", hash = "sha256:08b310f9e24a9594186fd75b4f73f4a4152069e3853f1ed8bfbf58369f4ad708", size = 114353, upload-time = "2026-03-24T12:59:08.246Z" },
]
[[package]]
name = "ast-serialize"
version = "0.5.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/81/9d/09e27731bd5864a9ce04e3244074e674bb8936bf62b45e0357248717adac/ast_serialize-0.5.0.tar.gz", hash = "sha256:5880091bfe6f4f986f22866375c2e884843e7a0b6343ae41aeea659613d879b6", size = 61157, upload-time = "2026-05-17T17:48:29.429Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c0/9a/13dde51ba9e15f8b97957ab7cb0120d0e381524d651c6bd630b9c359227f/ast_serialize-0.5.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:8f5c14f169eb0972c0c21bada5358b23d6047c76583b005234f865b11f1fa00a", size = 1183520, upload-time = "2026-05-17T17:47:30.831Z" },
{ url = "https://files.pythonhosted.org/packages/37/de/5a7f0a9fe68944f536632a5af84676739c7d2582be42deb082634bf3a754/ast_serialize-0.5.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7d1a2de9de5be04652f0ed60738356ef94f66db37924a9499fffe98dc491aa0b", size = 1175779, upload-time = "2026-05-17T17:47:32.551Z" },
{ url = "https://files.pythonhosted.org/packages/9c/81/0bb853e76e4f6e9a1855d569003c59e19ffac45f7079d91505d1bb212f92/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:be5173fb66f9b49026d9d5a2ff0fc7c7009077107c0eb285b2d60fdf1fe10bd1", size = 1233750, upload-time = "2026-05-17T17:47:34.731Z" },
{ url = "https://files.pythonhosted.org/packages/e5/d3/4cf705beeccc08754d0bbda99aefff26110e209b9a07ac8a6b60eec48531/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f8015cd071ac1339924ee2b8098c93e00e155f30a16f40ec9816fcf84f4753f6", size = 1235942, upload-time = "2026-05-17T17:47:36.287Z" },
{ url = "https://files.pythonhosted.org/packages/26/c8/ee097e437ea27dd2b8b227865c875492b585650a5802a22d82b304c8201b/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5499e8797edff2a9186aa313ed382c6b422e798e9332d9953badcee6e69a88f2", size = 1442517, upload-time = "2026-05-17T17:47:38.17Z" },
{ url = "https://files.pythonhosted.org/packages/ff/bd/68063442838f1ba68ec72b5436430bc75b3bb17a1a3c3063f09b0c05ae2b/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:6848f2a093fb5548751a9a09bff8fcd229e2bbeb0e3331f391b6ae6d26cd9903", size = 1254081, upload-time = "2026-05-17T17:47:39.826Z" },
{ url = "https://files.pythonhosted.org/packages/50/e2/1e520793bc6a4e4524a6ab022391e827825eaa0c3811828bfdc6852eca26/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:832d4c998e0b091fd60a6d6bceee535483c4d490de9ba85003af835225719261", size = 1259910, upload-time = "2026-05-17T17:47:41.369Z" },
{ url = "https://files.pythonhosted.org/packages/4e/e1/49b60f467979979cfe6913b43948ff25bca971ad0591d181812f163a988e/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:16db7c62ec0b8efe1d7afd283a388d8f74f2605d56032e5a37747d2de8dba027", size = 1250678, upload-time = "2026-05-17T17:47:43.702Z" },
{ url = "https://files.pythonhosted.org/packages/74/ba/66ab9555de6275677566f6574e5ef6c29cb185ea866f643bc06f8280a8ee/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:baf5eb061eb5bccade4128ad42da33787d72f6013809cd1b590376ece8b3c937", size = 1301603, upload-time = "2026-05-17T17:47:46.256Z" },
{ url = "https://files.pythonhosted.org/packages/66/42/6aca9b9abc710014b2be9059689e5dd1679339e78f567ffb4d255a9e2050/ast_serialize-0.5.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:104e4a35bd7c124173c41760ef9aaea17ddb3f86c65cb643671d59afbe3ee94c", size = 1410332, upload-time = "2026-05-17T17:47:47.899Z" },
{ url = "https://files.pythonhosted.org/packages/47/68/2f76594432a22581ecf878b5e75a9b8601c24b2241cf0bbeb1e21fcf370c/ast_serialize-0.5.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:36be371028fc1675acb38a331bde160dbab7ff907fdf00b67eb6911aa106951b", size = 1509979, upload-time = "2026-05-17T17:47:50.942Z" },
{ url = "https://files.pythonhosted.org/packages/40/ac/a93c9b58292653f6c595752f677a08e608f903b710594909e9231a389b3b/ast_serialize-0.5.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:061ee58bdb52341c8201a6df41182a977736bae3b7ded87ca7176ca25a8a47ab", size = 1505002, upload-time = "2026-05-17T17:47:54.093Z" },
{ url = "https://files.pythonhosted.org/packages/14/2e/b278f68c497ee2f1d1576cbbef8db5281cd4a5f2db040537592ac9c8862e/ast_serialize-0.5.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:b15219e9cdc9f53f6f4cb51c009203507228226148c05c5e8fe451c28b435eb3", size = 1456231, upload-time = "2026-05-17T17:47:56.311Z" },
{ url = "https://files.pythonhosted.org/packages/0b/43/419be1c566a4c504cd8fd60ce2f84e790f295495c0f327cfaeadf3d51012/ast_serialize-0.5.0-cp314-cp314t-win32.whl", hash = "sha256:842d1c004bb466c7df036f95fabef789570541922b10976b12f5592a69cf0b38", size = 1058668, upload-time = "2026-05-17T17:47:58.305Z" },
{ url = "https://files.pythonhosted.org/packages/03/6f/c9d4d549295ed05111aeb8853232d1afd9d0a179fddb01eeffbb3a4a6842/ast_serialize-0.5.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b0c06d760909b095cc466356dfccd05a1c7233a6ca191c020dca2c6a6f16c24c", size = 1101075, upload-time = "2026-05-17T17:48:00.35Z" },
{ url = "https://files.pythonhosted.org/packages/d0/8e/d00c5ab30c58222e07d62956fca86c59d91b9ad32997e633c38b526623a3/ast_serialize-0.5.0-cp314-cp314t-win_arm64.whl", hash = "sha256:787baedb0262cc49e8ce37cc15c00ae818e46a165a3b36f5e21ed174998104cb", size = 1075347, upload-time = "2026-05-17T17:48:01.753Z" },
{ url = "https://files.pythonhosted.org/packages/e0/9e/dc2530acb3a60dc6e46d65abf27d1d9f86721694757906a148d90a6860de/ast_serialize-0.5.0-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:0668aa9459cfa8c9c49ddd2163ebcf43088ba045ef7492af6fe22e0098303101", size = 1191380, upload-time = "2026-05-17T17:48:03.738Z" },
{ url = "https://files.pythonhosted.org/packages/26/0a/bd3d18a582f273d6c843d16bb9e22e9e16365ff7991e92f18f798e9f1224/ast_serialize-0.5.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:bf683d6363edf2b39eed6b6d4fe22d34b6203867a67e27134d9e2a2680c4bc4a", size = 1183879, upload-time = "2026-05-17T17:48:05.463Z" },
{ url = "https://files.pythonhosted.org/packages/40/ae/1f919100f8620887af58fcc381c61a1f218cdf89c6e155f87b213e61010a/ast_serialize-0.5.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9cc22cf0c9be65e71cf88fda130af60d61eb4a79370ad4cfe7900d48a4aa2211", size = 1244529, upload-time = "2026-05-17T17:48:07.008Z" },
{ url = "https://files.pythonhosted.org/packages/c6/ca/6376559dcce707cdbc1d0d9a13c8d3baaaa501e949ce0ebdc4230cd881aa/ast_serialize-0.5.0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f66173891548c9f2726bf27957b41cabce12fa679dc6da505ddbde4d4b3b31cf", size = 1240560, upload-time = "2026-05-17T17:48:08.46Z" },
{ url = "https://files.pythonhosted.org/packages/35/b2/a620e206b5aeb7efbf2710336df57d457cffbb3991076bbcc1147ef9abd4/ast_serialize-0.5.0-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e42d729ef2be96a14efbad355093284739e3670ece3e534f82cc8832790911d9", size = 1451172, upload-time = "2026-05-17T17:48:09.922Z" },
{ url = "https://files.pythonhosted.org/packages/fa/e0/4ad5c04c24a40481b2935ce9a0ccdb6023dc8b667167d06ae530cc3512f2/ast_serialize-0.5.0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b725026bafa801dbd7310eb13a75f0a2e370e7e51b2cb225f9d21fcfadf919ee", size = 1265072, upload-time = "2026-05-17T17:48:11.469Z" },
{ url = "https://files.pythonhosted.org/packages/b2/71/4d1d479aa56d0101c40e17720c3d6ac2af7269ea0487a80b18e7bfd1a5b7/ast_serialize-0.5.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b54f60c1d78767a53b67eaa663f0dfac3afe606aa07f1301572f588b73d64809", size = 1270488, upload-time = "2026-05-17T17:48:13.575Z" },
{ url = "https://files.pythonhosted.org/packages/6d/4f/0de1bbe06f6edef9fde4ed12ca8e7b3ec7e6e2bd4e672c5af487f7957665/ast_serialize-0.5.0-cp39-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:27d51654fc240a1e87e742d353d98eb45b75f62f129086b3596ab53df2ac2a43", size = 1260702, upload-time = "2026-05-17T17:48:15.141Z" },
{ url = "https://files.pythonhosted.org/packages/75/61/e00872439cfdddcc3c1b6cdaa6e5d904ba8e26a18807c67c4e14409d0ca8/ast_serialize-0.5.0-cp39-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2782c36237c46dd1674542f2109740ea5ea485a169bf1431939ada0434e17934", size = 1311182, upload-time = "2026-05-17T17:48:16.779Z" },
{ url = "https://files.pythonhosted.org/packages/76/8e/699a5b955f7926956c95e9e1d74132acad73c2fe7a426f94da89123c20aa/ast_serialize-0.5.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1943db345233cc7194a470f13afa9c59772c0b123dea0c9414c4d4ca54369759", size = 1421410, upload-time = "2026-05-17T17:48:18.527Z" },
{ url = "https://files.pythonhosted.org/packages/a9/ae/d5b7626874478997adc7a29ab28accf21e596fb590c944290401dfd0b29e/ast_serialize-0.5.0-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:df1c00022cbbcb064bfaa505aa9c9295362443ce5dacb459d1331d3da353f887", size = 1516587, upload-time = "2026-05-17T17:48:20.133Z" },
{ url = "https://files.pythonhosted.org/packages/0c/ce/b59e02a82d9c4244d64cde502e0b00e83e38816abe19155ceb5437402c7f/ast_serialize-0.5.0-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:cae65289fc456fde04af979a2be09302ef5d8ab92ef23e596d6746dc267ada27", size = 1515171, upload-time = "2026-05-17T17:48:21.921Z" },
{ url = "https://files.pythonhosted.org/packages/8b/38/d8d90042747d05aa08d4efcf1c99035a5f670a6bf4c214d31644392afbca/ast_serialize-0.5.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:239a4c354e8d676e9d94631d1d4a64edc6b266f86ff3a5a80aedd344f342c01d", size = 1464668, upload-time = "2026-05-17T17:48:23.544Z" },
{ url = "https://files.pythonhosted.org/packages/dd/51/5b840c4df7334104cecffa28f23904fe81ca89ca223d2450e288de39fd3c/ast_serialize-0.5.0-cp39-abi3-win32.whl", hash = "sha256:143a4ef63285a075871908fda3672dc21864b83a8ec3ee12304aa3e4c5387b9a", size = 1068311, upload-time = "2026-05-17T17:48:25.027Z" },
{ url = "https://files.pythonhosted.org/packages/41/11/ca5672c7d491825bc4cd6702dea106a6b60d928707712ec257c7833ae476/ast_serialize-0.5.0-cp39-abi3-win_amd64.whl", hash = "sha256:cf25572c526add400f26a4750dc6ce0c3bb93fc1f75e7ae0cad4ce4f2cd5c590", size = 1108931, upload-time = "2026-05-17T17:48:26.591Z" },
{ url = "https://files.pythonhosted.org/packages/45/19/cc8bd127d28a43da249aa955cfd164cf8fd534e79e42cea96c4854d72fd0/ast_serialize-0.5.0-cp39-abi3-win_arm64.whl", hash = "sha256:92a31c9c20d25a076edaeec76b128a3535d74a24f340b9a8a7e96c9b86dc9642", size = 1081181, upload-time = "2026-05-17T17:48:28.122Z" },
]
[[package]]
name = "beautifulsoup4"
version = "4.14.3"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "soupsieve" },
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/c3/b0/1c6a16426d389813b48d95e26898aff79abbde42ad353958ad95cc8c9b21/beautifulsoup4-4.14.3.tar.gz", hash = "sha256:6292b1c5186d356bba669ef9f7f051757099565ad9ada5dd630bd9de5fa7fb86", size = 627737, upload-time = "2025-11-30T15:08:26.084Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/1a/39/47f9197bdd44df24d67ac8893641e16f386c984a0619ef2ee4c51fbbc019/beautifulsoup4-4.14.3-py3-none-any.whl", hash = "sha256:0918bfe44902e6ad8d57732ba310582e98da931428d231a5ecb9e7c703a735bb", size = 107721, upload-time = "2025-11-30T15:08:24.087Z" },
]
[[package]]
name = "black"
version = "26.5.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "click" },
{ name = "mypy-extensions" },
{ name = "packaging" },
{ name = "pathspec" },
{ name = "platformdirs" },
{ name = "pytokens" },
]
sdist = { url = "https://files.pythonhosted.org/packages/c0/37/5628dd55bf2b34257fc7603f0fe97c40e3aaf24265f416a9c85c95ca1436/black-26.5.1.tar.gz", hash = "sha256:dd321f668053961824bcc1be1cc1df748b2d7e4fa28086b08331e577b0100a73", size = 679439, upload-time = "2026-05-18T16:53:36.107Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/4b/96/3c3e09f09f44a37aac36b178a279cd19aa7001bd796187a7b162a294c81f/black-26.5.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:96ae2c733b2aabdd9986e2c5df628ff3473676cd1c5faded1ff496cf6d74083c", size = 1970639, upload-time = "2026-05-18T17:05:11.461Z" },
{ url = "https://files.pythonhosted.org/packages/83/ea/5ad117b9ee3ecd933c712bcbae610006e5b7cc9f41c526cd7ed3b6c4124c/black-26.5.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0e48b87e03bf109288e55cfceadcfa15ff5470aca2851a851950ed2926f450d7", size = 1792130, upload-time = "2026-05-18T17:05:12.983Z" },
{ url = "https://files.pythonhosted.org/packages/06/3a/7c448bc623fcdfa96672531beb5a616ea5e64f6975955254d7731ffb0ad9/black-26.5.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5119fa92ae61f786e8c3662fd60aece1d0a2dd5cca5d0c79417a95e7a4272a59", size = 1846134, upload-time = "2026-05-18T17:05:14.506Z" },
{ url = "https://files.pythonhosted.org/packages/a1/5b/0b39b3a5917f0657ac014ad2edb58c139553a478adfe7f817abf1622ff6e/black-26.5.1-cp311-cp311-win_amd64.whl", hash = "sha256:30d3c14661f2792e9142cce3eeeb1cbc175b3eb5f733be0c8eeb99651e52b0c3", size = 1478883, upload-time = "2026-05-18T17:05:16.542Z" },
{ url = "https://files.pythonhosted.org/packages/4c/48/dc222692e0f95030db1bbfb6c857e76858bad09058221ea7aae815255327/black-26.5.1-cp311-cp311-win_arm64.whl", hash = "sha256:1ef92b76f7733f282fd096ea406200b5a286c42947412b0eaff3a74e3616cefe", size = 1277776, upload-time = "2026-05-18T17:05:18.029Z" },
{ url = "https://files.pythonhosted.org/packages/24/99/7744b906703228264ef73bdd534df88ec1ef3de45c4e78f6d31b9e32d0c9/black-26.5.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:4ad6fa01f941920f54f2bbb35f3df7673428a0ef98a0b0840c2eaef3b110efa8", size = 2012518, upload-time = "2026-05-18T17:05:20.108Z" },
{ url = "https://files.pythonhosted.org/packages/b7/c0/c5a3b1636dfd09c42534f2b3cf33506814f6d3e066fb0879ffa16c1ae860/black-26.5.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:3915f256e75a2d7cf88d8953d37f780455dc586cc72dee059c528fe77f581217", size = 1816016, upload-time = "2026-05-18T17:05:21.84Z" },
{ url = "https://files.pythonhosted.org/packages/1f/0e/36044316b65ca471d3bb6d3703fd06fb50c6b727c3562f6a5a3153634f88/black-26.5.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d98d4137277c75dfb898ec8d846c4fd68ba1e9cf77f95e2865c203dc18f4c3d", size = 1884150, upload-time = "2026-05-18T17:05:23.546Z" },
{ url = "https://files.pythonhosted.org/packages/b3/33/dafc5808c2af43672912111d7c3354af1615f7e2be3bed7a878461abbe4d/black-26.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:a1dca32d9f1784af512a13410ec204c6f7f0aa9797a111c42e1c03449821c264", size = 1486825, upload-time = "2026-05-18T17:05:25.004Z" },
{ url = "https://files.pythonhosted.org/packages/82/14/b965ee6ad2a311f28bdbf692def3ee9848d2ae289dab28b27657fcee3e78/black-26.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:1037d5ac7b7b310b2632ad867ec8d0e4c4819dcdb0b820f63135da746a24e418", size = 1288646, upload-time = "2026-05-18T17:05:26.477Z" },
{ url = "https://files.pythonhosted.org/packages/3f/5c/c384363980e11e25ca6b93205949bb331fbf35f4e0dbec376dfa6326cec8/black-26.5.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:2b36cf2ddf5566e205f6535f782a62194a184d33e175b64ae8c40b1737522be3", size = 2009020, upload-time = "2026-05-18T17:05:28.132Z" },
{ url = "https://files.pythonhosted.org/packages/0b/df/9f31c5e0babbfed77d505fc5d120beb98b21b33feaeded3924ea941fe360/black-26.5.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:1f7ea64ebfa01b50f693508fc39f875e264446d3b097088f84f203b9d09618a0", size = 1813335, upload-time = "2026-05-18T17:05:31.266Z" },
{ url = "https://files.pythonhosted.org/packages/fb/24/8e7b9a2fa61b0afd82209efe937557d180a1fa055bd7f6161eb9defc3719/black-26.5.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ecb3e624844c798144e9bd986954e0adc81d8911a1f30f375e1252fe26e8c294", size = 1881614, upload-time = "2026-05-18T17:05:32.718Z" },
{ url = "https://files.pythonhosted.org/packages/49/ad/b4e0d9365ba8ac34f6bbab62a4b1b2dd5d618fac3fa1b8db968c844201b5/black-26.5.1-cp313-cp313-win_amd64.whl", hash = "sha256:e1a26503279b6b310669fb0b219c39e4820b77e8189fe80f522bb511f247db0a", size = 1488925, upload-time = "2026-05-18T17:05:34.259Z" },
{ url = "https://files.pythonhosted.org/packages/a1/4b/652b859bf5df88a751c30451b09338f7fd26a77d1271c666992f836b7711/black-26.5.1-cp313-cp313-win_arm64.whl", hash = "sha256:5c34b25da232ead53a6f335b76dbea124f4d152ad568b9080d6f944bc2b34b52", size = 1289883, upload-time = "2026-05-18T17:05:36.019Z" },
{ url = "https://files.pythonhosted.org/packages/a6/16/a8da8eb208c51c7f4ce74609a45d0dcc6d8a2141e45e81ee5289d1bb0d59/black-26.5.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:e88976690a64b0af98312ca958415849cb42423423c5f2ee74af4b49a97a2168", size = 2004800, upload-time = "2026-05-18T17:05:38.182Z" },
{ url = "https://files.pythonhosted.org/packages/11/8a/a479296a19e383b70a725882a6cf3d786540601ff03cabbaaf1cce864c5a/black-26.5.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:32d5ea7f6c8bdfa6e648326ebca1f02b0764e2a029edc6f8dce2627e19d468c3", size = 1815576, upload-time = "2026-05-18T17:05:40.309Z" },
{ url = "https://files.pythonhosted.org/packages/81/6b/cfaf3d39f25132c156a068f6b805576c9103a84086019507c70e1911ee7d/black-26.5.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ea8d16dc41655aa113cd64665e7219446cd7e4ff2248d7178eaa905190c86b18", size = 1877927, upload-time = "2026-05-18T17:05:42.463Z" },
{ url = "https://files.pythonhosted.org/packages/66/76/302e313964bcff7e28df329d39f84f5270095730d85ff0acc260610a0d82/black-26.5.1-cp314-cp314-win_amd64.whl", hash = "sha256:577f21094ea469ef92ec1adaf2c9441a226d2144d01a5be2fa823cecf6543e50", size = 1511860, upload-time = "2026-05-18T17:05:43.943Z" },
{ url = "https://files.pythonhosted.org/packages/27/4e/a3827e35e0e567f9f9ee59e2a0ab979267dca98718f25547ca8c6733afd4/black-26.5.1-cp314-cp314-win_arm64.whl", hash = "sha256:ed1a20af114c301a0269bf01163d51dbef72737fd65f850001e7cbe7f3c7abae", size = 1316632, upload-time = "2026-05-18T17:05:45.521Z" },
{ url = "https://files.pythonhosted.org/packages/94/51/f975cae76d44274cc2868dc9040ac5d58d464784610234455b4e7b19c6ef/black-26.5.1-py3-none-any.whl", hash = "sha256:4ed7f7da04046d2e488437170797d3b4a4ad83906683bcb7dfc68b673bbce5e2", size = 213693, upload-time = "2026-05-18T16:53:33.964Z" },
]
[[package]]
name = "certifi"
version = "2026.4.22"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/25/ee/6caf7a40c36a1220410afe15a1cc64993a1f864871f698c0f93acb72842a/certifi-2026.4.22.tar.gz", hash = "sha256:8d455352a37b71bf76a79caa83a3d6c25afee4a385d632127b6afb3963f1c580", size = 137077, upload-time = "2026-04-22T11:26:11.191Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/22/30/7cd8fdcdfbc5b869528b079bfb76dcdf6056b1a2097a662e5e8c04f42965/certifi-2026.4.22-py3-none-any.whl", hash = "sha256:3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a", size = 135707, upload-time = "2026-04-22T11:26:09.372Z" },
]
[[package]]
name = "click"
version = "8.4.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "colorama", marker = "sys_platform == 'win32'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/23/e4/796662cd90cf80e3a363c99db2b88e0e394b988a575f60a17e16440cd011/click-8.4.0.tar.gz", hash = "sha256:638f1338fe1235c8f4e008e4a8a254fb5c5fbdcbb40ece3c9142ebb78e792973", size = 350843, upload-time = "2026-05-17T00:47:58.425Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/ee/ae/8e92f8058baf87f6c7d86ee7e457668690195cc77efedb8d3797a06e3940/click-8.4.0-py3-none-any.whl", hash = "sha256:40c50b7c6c6adac2823d411041ec84f3f103f1b280d5e9ce0d7f998995832f81", size = 116147, upload-time = "2026-05-17T00:47:56.842Z" },
]
[[package]]
name = "colorama"
version = "0.4.6"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
]
[[package]]
name = "greenlet"
version = "3.5.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/3c/3f/dbf99fb14bfeb88c28f16729215478c0e265cacd6dc22270c8f31bb6892f/greenlet-3.5.0.tar.gz", hash = "sha256:d419647372241bc68e957bf38d5c1f98852155e4146bd1e4121adea81f4f01e4", size = 196995, upload-time = "2026-04-27T13:37:15.544Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/8b/0f/a91f143f356523ff682309732b175765a9bc2836fd7c081c2c67fedc1ad4/greenlet-3.5.0-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:8f1cc966c126639cd152fdaa52624d2655f492faa79e013fea161de3e6dda082", size = 284726, upload-time = "2026-04-27T12:20:51.402Z" },
{ url = "https://files.pythonhosted.org/packages/95/82/800646c7ffc5dbabd75ddd2f6b519bb898c0c9c969e5d0473bfe5d20bcce/greenlet-3.5.0-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:362624e6a8e5bca3b8233e45eef33903a100e9539a2b995c364d595dbc4018b3", size = 604264, upload-time = "2026-04-27T12:52:39.494Z" },
{ url = "https://files.pythonhosted.org/packages/ca/ac/354867c0bba812fc33b15bc55aedafedd0aee3c7dd91dfca22444157dc0c/greenlet-3.5.0-cp311-cp311-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5ecd83806b0f4c2f53b1018e0005cd82269ea01d42befc0368730028d850ed1c", size = 616099, upload-time = "2026-04-27T12:59:39.623Z" },
{ url = "https://files.pythonhosted.org/packages/c9/ab/192090c4a5b30df148c22bf4b8895457d739a7c7c5a7b9c41e5dd7f537f2/greenlet-3.5.0-cp311-cp311-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fa94cb2288681e3a11645958f1871d48ee9211bd2f66628fdace505927d6e564", size = 623976, upload-time = "2026-04-27T13:02:37.363Z" },
{ url = "https://files.pythonhosted.org/packages/ff/b0/815bece7399e01cadb69014219eebd0042339875c59a59b0820a46ece356/greenlet-3.5.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0ff251e9a0279522e62f6176412869395a64ddf2b5c5f782ff609a8216a4e662", size = 615198, upload-time = "2026-04-27T12:25:25.928Z" },
{ url = "https://files.pythonhosted.org/packages/24/11/05eb2b9b188c6df7d68a89c99134d644a7af616a40b9808e8e6ced315d5d/greenlet-3.5.0-cp311-cp311-manylinux_2_39_riscv64.whl", hash = "sha256:64d6ac45f7271f48e45f67c95b54ef73534c52ec041fcda8edf520c6d811f4bc", size = 418379, upload-time = "2026-04-27T13:05:12.755Z" },
{ url = "https://files.pythonhosted.org/packages/10/80/3b2c0a895d6698f6ddb31b07942ebfa982f3e30888bc5546a5b5990de8b2/greenlet-3.5.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:6d874e79afd41a96e11ff4c5d0bc90a80973e476fda1c2c64985667397df432b", size = 1574927, upload-time = "2026-04-27T12:53:25.81Z" },
{ url = "https://files.pythonhosted.org/packages/44/0e/f354af514a4c61454dbc68e44d47544a5a4d6317e30b77ddfa3a09f4c5f3/greenlet-3.5.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:0ed006e4b86c59de7467eb2601cd1b77b5a7d657d1ee55e30fe30d76451edba4", size = 1642683, upload-time = "2026-04-27T12:25:23.9Z" },
{ url = "https://files.pythonhosted.org/packages/fa/6a/87f38255201e993a1915265ebb80cd7c2c78b04a45744995abbf6b259fd8/greenlet-3.5.0-cp311-cp311-win_amd64.whl", hash = "sha256:703cb211b820dbffbbc55a16bfc6e4583a6e6e990f33a119d2cc8b83211119c8", size = 238115, upload-time = "2026-04-27T12:21:48.845Z" },
{ url = "https://files.pythonhosted.org/packages/e3/f8/450fe3c5938fa737ea4d22699772e6e34e8e24431a47bf4e8a1ceed4a98e/greenlet-3.5.0-cp311-cp311-win_arm64.whl", hash = "sha256:6c18dfb59c70f5a94acd271c72e90128c3c776e41e5f07767908c8c1b74ad339", size = 235017, upload-time = "2026-04-27T12:22:26.768Z" },
{ url = "https://files.pythonhosted.org/packages/ef/32/f2ce6d4cac3e55bc6173f92dbe627e782e1850f89d986c3606feb63aafa7/greenlet-3.5.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:db2910d3c809444e0a20147361f343fe2798e106af8d9d8506f5305302655a9f", size = 286228, upload-time = "2026-04-27T12:20:34.421Z" },
{ url = "https://files.pythonhosted.org/packages/b7/aa/caed9e5adf742315fc7be2a84196373aab4816e540e38ba0d76cb7584d68/greenlet-3.5.0-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3ec9ea74e7268ace7f9aab1b1a4e730193fc661b39a993cd91c606c32d4a3628", size = 601775, upload-time = "2026-04-27T12:52:41.045Z" },
{ url = "https://files.pythonhosted.org/packages/c7/af/90ae08497400a941595d12774447f752d3dfe0fbb012e35b76bc5c0ff37e/greenlet-3.5.0-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:54d243512da35485fc7a6bf3c178fdda6327a9d6506fcdd62b1abd1e41b2927b", size = 614436, upload-time = "2026-04-27T12:59:41.595Z" },
{ url = "https://files.pythonhosted.org/packages/3f/e9/4eeadf8cb3403ac274245ba75f07844abc7fa5f6787583fc9156ba741e0f/greenlet-3.5.0-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:41353ec2ecedf7aa8f682753a41919f8718031a6edac46b8d3dc7ed9e1ceb136", size = 620610, upload-time = "2026-04-27T13:02:39.194Z" },
{ url = "https://files.pythonhosted.org/packages/2b/e0/2e13df68f367e2f9960616927d60857dd7e56aaadd59a47c644216b2f920/greenlet-3.5.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d280a7f5c331622c69f97eb167f33577ff2d1df282c41cd15907fc0a3ca198c", size = 611388, upload-time = "2026-04-27T12:25:28.008Z" },
{ url = "https://files.pythonhosted.org/packages/ee/ef/f913b3c0eb7d26d86a2401c5e1546c9d46b657efee724b06f6f4ac5d8824/greenlet-3.5.0-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:58c1c374fe2b3d852f9b6b11a7dff4c85404e51b9a596fd9e89cf904eb09866d", size = 422775, upload-time = "2026-04-27T13:05:14.261Z" },
{ url = "https://files.pythonhosted.org/packages/82/f7/393c64055132ac0d488ef6be549253b7e6274194863967ddc0bc8f5b87b8/greenlet-3.5.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1eb67d5adefb5bd2e182d42678a328979a209e4e82eb93575708185d31d1f588", size = 1570768, upload-time = "2026-04-27T12:53:28.099Z" },
{ url = "https://files.pythonhosted.org/packages/b8/4b/eaf7735253522cf56d1b74d672a58f54fc114702ceaf05def59aae72f6e1/greenlet-3.5.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2628d6c86f6cb0cb45e0c3c54058bbec559f57eaae699447748cb3928150577e", size = 1635983, upload-time = "2026-04-27T12:25:26.903Z" },
{ url = "https://files.pythonhosted.org/packages/4c/fe/4fb3a0805bd5165da5ebf858da7cc01cce8061674106d2cf5bdab32cbfde/greenlet-3.5.0-cp312-cp312-win_amd64.whl", hash = "sha256:d4d9f0624c775f2dfc56ba54d515a8c771044346852a918b405914f6b19d7fd8", size = 238840, upload-time = "2026-04-27T12:23:54.806Z" },
{ url = "https://files.pythonhosted.org/packages/cb/cb/baa584cb00532126ffe12d9787db0a60c5a4f55c27bfe2666df5d4c30a32/greenlet-3.5.0-cp312-cp312-win_arm64.whl", hash = "sha256:83ed9f27f1680b50e89f40f6df348a290ea234b249a4003d366663a12eab94f2", size = 235615, upload-time = "2026-04-27T12:21:38.57Z" },
{ url = "https://files.pythonhosted.org/packages/0c/58/fc576f99037ce19c5aa16628e4c3226b6d1419f72a62c79f5f40576e6eb3/greenlet-3.5.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:5a5ed18de6a0f6cc7087f1563f6bd93fc7df1c19165ca01e9bde5a5dc281d106", size = 285066, upload-time = "2026-04-27T12:23:05.033Z" },
{ url = "https://files.pythonhosted.org/packages/4a/ba/b28ddbe6bfad6a8ac196ef0e8cff37bc65b79735995b9e410923fffeeb70/greenlet-3.5.0-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a717fbc46d8a354fa675f7c1e813485b6ba3885f9bef0cd56e5ba27d758ff5b", size = 604414, upload-time = "2026-04-27T12:52:42.358Z" },
{ url = "https://files.pythonhosted.org/packages/09/06/4b69f8f0b67603a8be2790e55107a190b376f2627fe0eaf5695d85ffb3cd/greenlet-3.5.0-cp313-cp313-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ddc090c5c1792b10246a78e8c2163ebbe04cf877f9d785c230a7b27b39ad038e", size = 617349, upload-time = "2026-04-27T12:59:43.32Z" },
{ url = "https://files.pythonhosted.org/packages/6a/15/a643b4ecd09969e30b8a150d5919960caae0abe4f5af75ab040b1ab85e78/greenlet-3.5.0-cp313-cp313-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4964101b8585c144cbda5532b1aa644255126c08a265dae90c16e7a0e63aaa9d", size = 623234, upload-time = "2026-04-27T13:02:40.611Z" },
{ url = "https://files.pythonhosted.org/packages/8a/17/a3918541fd0ddefe024a69de6d16aa7b46d36ac19562adaa63c7fa180eff/greenlet-3.5.0-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2094acd54b272cb6eae8c03dd87b3fa1820a4cef18d6889c378d503500a1dc13", size = 613927, upload-time = "2026-04-27T12:25:30.28Z" },
{ url = "https://files.pythonhosted.org/packages/77/18/3b13d5ef1275b0ffaf933b05efa21408ac4ca95823c7411d79682e4fdcff/greenlet-3.5.0-cp313-cp313-manylinux_2_39_riscv64.whl", hash = "sha256:7022615368890680e67b9965d33f5773aade330d5343bbe25560135aaa849eae", size = 425243, upload-time = "2026-04-27T13:05:15.689Z" },
{ url = "https://files.pythonhosted.org/packages/ee/e1/bd0af6213c7dd33175d8a462d4c1fe1175124ebed4855bc1475a5b5242c2/greenlet-3.5.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5e05ba267789ea87b5a155cf0e810b1ab88bf18e9e8740813945ceb8ee4350ba", size = 1570893, upload-time = "2026-04-27T12:53:29.483Z" },
{ url = "https://files.pythonhosted.org/packages/9b/2a/0789702f864f5382cb476b93d7a9c823c10472658102ccd65f415747d2e2/greenlet-3.5.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:0ecec963079cd58cbd14723582384f11f166fd58883c15dcbfb342e0bc9b5846", size = 1636060, upload-time = "2026-04-27T12:25:28.845Z" },
{ url = "https://files.pythonhosted.org/packages/b2/8f/22bf9df92bbff0eb07842b60f7e63bf7675a9742df628437a9f02d09137f/greenlet-3.5.0-cp313-cp313-win_amd64.whl", hash = "sha256:728d9667d8f2f586644b748dbd9bb67e50d6a9381767d1357714ea6825bb3bf5", size = 238740, upload-time = "2026-04-27T12:24:01.341Z" },
{ url = "https://files.pythonhosted.org/packages/b6/b7/9c5c3d653bd4ff614277c049ac676422e2c557db47b4fe43e6313fc005dc/greenlet-3.5.0-cp313-cp313-win_arm64.whl", hash = "sha256:47422135b1d308c14b2c6e758beedb1acd33bb91679f5670edf77bf46244722b", size = 235525, upload-time = "2026-04-27T12:23:12.308Z" },
{ url = "https://files.pythonhosted.org/packages/94/5e/a70f31e3e8d961c4ce589c15b28e4225d63704e431a23932a3808cbcc867/greenlet-3.5.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:f35807464c4c58c55f0d31dfa83c541a5615d825c2fe3d2b95360cf7c4e3c0a8", size = 285564, upload-time = "2026-04-27T12:23:08.555Z" },
{ url = "https://files.pythonhosted.org/packages/af/a6/046c0a28e21833e4086918218cfb3d8bed51c075a1b700f20b9d7861c0f4/greenlet-3.5.0-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:55fa7ea52771be44af0de27d8b80c02cd18c2c3cddde6c847ecebdf72418b6a1", size = 651166, upload-time = "2026-04-27T12:52:43.644Z" },
{ url = "https://files.pythonhosted.org/packages/47/f8/4af27f71c5ff32a7fbc516adb46370d9c4ae2bc7bd3dc7d066ac542b4b15/greenlet-3.5.0-cp314-cp314-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a97e4821aa710603f94de0da25f25096454d78ffdace5dc77f3a006bc01abba3", size = 663792, upload-time = "2026-04-27T12:59:44.93Z" },
{ url = "https://files.pythonhosted.org/packages/fb/89/2dadb89793c37ee8b4c237857188293e9060dc085f19845c292e00f8e091/greenlet-3.5.0-cp314-cp314-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bf2d8a80bec89ab46221ae45c5373d5ba0bd36c19aa8508e85c6cd7e5106cd37", size = 668086, upload-time = "2026-04-27T13:02:42.314Z" },
{ url = "https://files.pythonhosted.org/packages/a3/59/1bd6d7428d6ed9106efbb8c52310c60fd04f6672490f452aeaa3829aa436/greenlet-3.5.0-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8f52a464e4ed91780bdfbbdd2b97197f3accaa629b98c200f4dffada759f3ae7", size = 660933, upload-time = "2026-04-27T12:25:33.276Z" },
{ url = "https://files.pythonhosted.org/packages/82/35/75722be7e26a2af4cbd2dc35b0ed382dacf9394b7e75551f76ed1abe87f2/greenlet-3.5.0-cp314-cp314-manylinux_2_39_riscv64.whl", hash = "sha256:1bae92a1dd94c5f9d9493c3a212dd874c202442047cf96446412c862feca83a2", size = 470799, upload-time = "2026-04-27T13:05:17.094Z" },
{ url = "https://files.pythonhosted.org/packages/83/e4/b903e5a5fae1e8a28cdd32a0cfbfd560b668c25b692f67768822ddc5f40f/greenlet-3.5.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:762612baf1161ccb8437c0161c668a688223cba28e1bf038f4eb47b13e39ccdf", size = 1618401, upload-time = "2026-04-27T12:53:31.062Z" },
{ url = "https://files.pythonhosted.org/packages/0e/e3/5ec408a329acb854fb607a122e1ee5fb3ff649f9a97952948a90803c0d8e/greenlet-3.5.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:57a43c6079a89713522bc4bcb9f75070ecf5d3dbad7792bfe42239362cbf2a16", size = 1682038, upload-time = "2026-04-27T12:25:31.838Z" },
{ url = "https://files.pythonhosted.org/packages/91/20/6b165108058767ee643c55c5c4904d591a830ee2b3c7dbd359828fbc829f/greenlet-3.5.0-cp314-cp314-win_amd64.whl", hash = "sha256:3bc59be3945ae9750b9e7d45067d01ae3fe90ea5f9ade99239dabdd6e28a5033", size = 239835, upload-time = "2026-04-27T12:24:54.136Z" },
{ url = "https://files.pythonhosted.org/packages/4e/62/1c498375cee177b55d980c1db319f26470e5309e54698c8f8fc06c0fd539/greenlet-3.5.0-cp314-cp314-win_arm64.whl", hash = "sha256:a96fcee45e03fe30a62669fd16ab5c9d3c172660d3085605cb1e2d1280d3c988", size = 236862, upload-time = "2026-04-27T12:23:24.957Z" },
{ url = "https://files.pythonhosted.org/packages/78/a8/4522939255bb5409af4e87132f915446bf3622c2c292d14d3c38d128ae82/greenlet-3.5.0-cp314-cp314t-macosx_11_0_universal2.whl", hash = "sha256:a10a732421ab4fec934783ce3e54763470d0181db6e3468f9103a275c3ed1853", size = 293614, upload-time = "2026-04-27T12:24:12.874Z" },
{ url = "https://files.pythonhosted.org/packages/15/5e/8744c52e2c027b5a8772a01561934c8835f869733e101f62075c60430340/greenlet-3.5.0-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7fc391b1566f2907d17aaebe78f8855dc45675159a775fcf9e61f8ee0078e87f", size = 650723, upload-time = "2026-04-27T12:52:45.412Z" },
{ url = "https://files.pythonhosted.org/packages/00/ef/7b4c39c03cf46ceca512c5d3f914afd85aa30b2cc9a93015b0dd73e4be6c/greenlet-3.5.0-cp314-cp314t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:680bd0e7ad5e8daa8a4aa89f68fd6adc834b8a8036dc256533f7e08f4a4b01f7", size = 656529, upload-time = "2026-04-27T12:59:46.295Z" },
{ url = "https://files.pythonhosted.org/packages/5f/5c/0602239503b124b70e39355cbdb39361ecfe65b87a5f2f63752c32f5286f/greenlet-3.5.0-cp314-cp314t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1aa4ce8debcd4ea7fb2e150f3036588c41493d1d52c43538924ae1819003f4ce", size = 657015, upload-time = "2026-04-27T13:02:43.973Z" },
{ url = "https://files.pythonhosted.org/packages/0b/b5/c7768f352f5c010f92064d0063f987e7dc0cd290a6d92a34109015ce4aa1/greenlet-3.5.0-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ddb36c7d6c9c0a65f18c7258634e0c416c6ab59caac8c987b96f80c2ebda0112", size = 654364, upload-time = "2026-04-27T12:25:35.64Z" },
{ url = "https://files.pythonhosted.org/packages/38/51/8699f865f125dc952384cb432b0f7138aa4d8f2969a7d12d0df5b94d054d/greenlet-3.5.0-cp314-cp314t-manylinux_2_39_riscv64.whl", hash = "sha256:728a73687e39ae9ca34e4694cbf2f049d3fbc7174639468d0f67200a97d8f9e2", size = 488275, upload-time = "2026-04-27T13:05:18.28Z" },
{ url = "https://files.pythonhosted.org/packages/ef/d0/079ebe12e4b1fc758857ce5be1a5e73f06870f2101e52611d1e71925ce54/greenlet-3.5.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e5ddf316ced87539144621453c3aef229575825fe60c604e62bedc4003f372b2", size = 1614204, upload-time = "2026-04-27T12:53:32.618Z" },
{ url = "https://files.pythonhosted.org/packages/6d/89/6c2fb63df3596552d20e58fb4d96669243388cf680cff222758812c7bfaa/greenlet-3.5.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:4a448128607be0de65342dc9b31be7f948ef4cc0bc8832069350abefd310a8f2", size = 1675480, upload-time = "2026-04-27T12:25:34.168Z" },
{ url = "https://files.pythonhosted.org/packages/15/32/77ee8a6c1564fc345a491a4e85b3bf360e4cf26eac98c4532d2fdb96e01f/greenlet-3.5.0-cp314-cp314t-win_amd64.whl", hash = "sha256:d60097128cb0a1cab9ea541186ea13cd7b847b8449a7787c2e2350da0cb82d86", size = 245324, upload-time = "2026-04-27T12:24:40.295Z" },
]
[[package]]
name = "h11"
version = "0.16.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" },
]
[[package]]
name = "httpcore"
version = "1.0.9"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "certifi" },
{ name = "h11" },
]
sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" },
]
[[package]]
name = "httpx"
version = "0.28.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "anyio" },
{ name = "certifi" },
{ name = "httpcore" },
{ name = "idna" },
]
sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" },
]
[[package]]
name = "idna"
version = "3.15"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/82/77/7b3966d0b9d1d31a36ddf1746926a11dface89a83409bf1483f0237aa758/idna-3.15.tar.gz", hash = "sha256:ca962446ea538f7092a95e057da437618e886f4d349216d2b1e294abfdb65fdc", size = 199245, upload-time = "2026-05-12T22:45:57.011Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/d2/23/408243171aa9aaba178d3e2559159c24c1171a641aa83b67bdd3394ead8e/idna-3.15-py3-none-any.whl", hash = "sha256:048adeaf8c2d788c40fee287673ccaa74c24ffd8dcf09ffa555a2fbb59f10ac8", size = 72340, upload-time = "2026-05-12T22:45:55.733Z" },
]
[[package]]
name = "iniconfig"
version = "2.3.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" },
]
[[package]]
name = "intra-mart-doc-crawler"
version = "1.0.0"
source = { editable = "." }
dependencies = [
{ name = "aiofiles" },
{ name = "beautifulsoup4" },
{ name = "httpx" },
{ name = "loguru" },
{ name = "lxml" },
{ name = "markdownify" },
{ name = "playwright" },
{ name = "pyyaml" },
{ name = "tenacity" },
]
[package.optional-dependencies]
dev = [
{ name = "black" },
{ name = "mypy" },
{ name = "pytest" },
{ name = "pytest-asyncio" },
{ name = "ruff" },
]
[package.metadata]
requires-dist = [
{ name = "aiofiles", specifier = ">=24.1.0" },
{ name = "beautifulsoup4", specifier = ">=4.12.3" },
{ name = "black", marker = "extra == 'dev'", specifier = ">=24.4.2" },
{ name = "httpx", specifier = ">=0.28.1" },
{ name = "loguru", specifier = ">=0.7.2" },
{ name = "lxml", specifier = ">=5.2.2" },
{ name = "markdownify", specifier = ">=0.13.1" },
{ name = "mypy", marker = "extra == 'dev'", specifier = ">=1.10.1" },
{ name = "playwright", specifier = ">=1.53.0" },
{ name = "pytest", marker = "extra == 'dev'", specifier = ">=8.2.2" },
{ name = "pytest-asyncio", marker = "extra == 'dev'", specifier = ">=0.23.7" },
{ name = "pyyaml", specifier = ">=6.0.2" },
{ name = "ruff", marker = "extra == 'dev'", specifier = ">=0.5.0" },
{ name = "tenacity", specifier = ">=9.0.0" },
]
provides-extras = ["dev"]
[[package]]
name = "librt"
version = "0.11.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/40/08/9e7f6b5d2b5bed6ad055cdd5925f192bb403a51280f86b56554d9d0699a2/librt-0.11.0.tar.gz", hash = "sha256:075dc3ef4458a278e0195cbf6ac9d38808d9b906c5a6c7f7f79c3888276a3fb1", size = 200139, upload-time = "2026-05-10T18:17:25.138Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/fe/87/2bf31fe17587b29e3f93ec31421e2b1e1c3e349b8bf6c7c313dbad1d5340/librt-0.11.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:93d95bd45b7d58343d8b90d904450a545144eec19a002511163426f8ab1fae29", size = 141092, upload-time = "2026-05-10T18:15:34.795Z" },
{ url = "https://files.pythonhosted.org/packages/cf/08/5c5bf772920b7ebac6e32bc91a643e0ab3870199c0b542356d3baa83970a/librt-0.11.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4ee278c769a713638cdacd4c0436d72156e75df3ebc0166ab2b9dc43acc386c9", size = 142035, upload-time = "2026-05-10T18:15:36.242Z" },
{ url = "https://files.pythonhosted.org/packages/06/20/662a03d254e5b000d838e8b345d83303ddb768c080fd488e40634c0fa66b/librt-0.11.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f230cb1cbc9faaa616f9a678f530ebcf186e414b6bcbd88b960e4ba1b92428d5", size = 475022, upload-time = "2026-05-10T18:15:37.56Z" },
{ url = "https://files.pythonhosted.org/packages/de/f3/aa81523e45184c6ec23dc7f63263362ec55f80a09d424c012359ecbe7e35/librt-0.11.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:5d63c855d86938d9de93e265c9bd8c705b51ec494de5738340ee93767a686e4b", size = 467273, upload-time = "2026-05-10T18:15:39.182Z" },
{ url = "https://files.pythonhosted.org/packages/6b/6f/59c74b560ca8853834d5501d589c8a2519f4184f273a085ffd0f37a1cc47/librt-0.11.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:993f028be9e96a08d31df3479ac80d99be374d17f3b78e4796b3fd3c913d4e89", size = 497083, upload-time = "2026-05-10T18:15:40.634Z" },
{ url = "https://files.pythonhosted.org/packages/fe/7b/5aa4d2c9600a719401160bf7055417df0b2a47439b9d88286ce45e56b65f/librt-0.11.0-cp311-cp311-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:258d73a0aa66a055e65b2e4d1b8cdb23b9d132c5bb915d9547d804fcaed116cc", size = 489139, upload-time = "2026-05-10T18:15:41.934Z" },
{ url = "https://files.pythonhosted.org/packages/d6/31/9143803d7da6856a69153785768c4936864430eec0fd9461c3ea527d9922/librt-0.11.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:0827efe7854718f04aaddf6496e96960a956e676fe1d0f04eb41511fd8ad06d5", size = 508442, upload-time = "2026-05-10T18:15:43.206Z" },
{ url = "https://files.pythonhosted.org/packages/2f/5a/bce08184488426bda4ccc2c4964ac048c8f68ae89bd7120082eef4233cfd/librt-0.11.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:7753e57d6e12d019c0d8786f1c09c709f4c3fcc57c3887b24e36e6c06ec938b7", size = 514230, upload-time = "2026-05-10T18:15:44.761Z" },
{ url = "https://files.pythonhosted.org/packages/89/8c/bb5e213d254b7505a0e658da199d8ab719086632ce09eef311ab27976523/librt-0.11.0-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:11bd19822431cc21af9f27374e7ae2e58103c7d98bda823536a6c47f6bb2bb3d", size = 494231, upload-time = "2026-05-10T18:15:46.308Z" },
{ url = "https://files.pythonhosted.org/packages/9d/fb/541cdad5b1ab1300398c74c4c9a497b88e5074c21b1244c8f49731d3a284/librt-0.11.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:22bdf239b219d3993761a148ffa134b19e52e9989c84f845d5d7b71d70a17412", size = 537585, upload-time = "2026-05-10T18:15:47.629Z" },
{ url = "https://files.pythonhosted.org/packages/8f/f2/464bb69295c320cb06bddb4f14a4ec67934ee14b2bffb12b19fb7ab287ba/librt-0.11.0-cp311-cp311-win32.whl", hash = "sha256:46c60b61e308eb535fbd6fa622b1ee1bb2815691c1ad9c98bf7b84952ec3bc8d", size = 100509, upload-time = "2026-05-10T18:15:49.157Z" },
{ url = "https://files.pythonhosted.org/packages/6d/e7/a17ee1788f9e4fbf548c19f4afa07c92089b9e24fef6cb2410863781ef4c/librt-0.11.0-cp311-cp311-win_amd64.whl", hash = "sha256:902e546ff044f579ff1c953ff5fce97b636fe9e3943996b2177710c6ef076f73", size = 118628, upload-time = "2026-05-10T18:15:50.345Z" },
{ url = "https://files.pythonhosted.org/packages/cc/c7/6c766214f9f9903bcfcfbef97d807af8d8f5aa3502d247858ab17582d212/librt-0.11.0-cp311-cp311-win_arm64.whl", hash = "sha256:65ac3bc20f78aa0ee5ae84baa68917f89fef4af63e941084dd019a0d0e749f0c", size = 103122, upload-time = "2026-05-10T18:15:52.068Z" },
{ url = "https://files.pythonhosted.org/packages/8b/d0/07c77e067f0838949b43bd89232c29d72efebb9d2801a9750184eb706b71/librt-0.11.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b87504f1690a23b9a2cca841191a04f83895d4fc2dd04df91d82b1a04ca2ad46", size = 144147, upload-time = "2026-05-10T18:15:53.227Z" },
{ url = "https://files.pythonhosted.org/packages/7a/24/8493538fa4f62f982686398a5b8f68008138a75086abdea19ade64bf4255/librt-0.11.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:40071fc5fe0ce8daa6de616702314a01e1250711682b0523d6ab8d4525910cb3", size = 143614, upload-time = "2026-05-10T18:15:54.657Z" },
{ url = "https://files.pythonhosted.org/packages/ff/1e/f8bad050810d9171f34a1648ed910e56814c2ba61639f2bd53c6377ae24b/librt-0.11.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:137e79445c896a0ea7b265f52d23954e05b64222ee1af69e2cb34219067cbb67", size = 485538, upload-time = "2026-05-10T18:15:56.117Z" },
{ url = "https://files.pythonhosted.org/packages/c0/fe/3594ebfbaf03084ba4b120c9ba5c3183fd938a48725e9bbe6ff0a5159ad8/librt-0.11.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:cca6644054e78746d8d4ef238681f9c34ff8b584fe6b988ecebb8db3b15e622a", size = 479623, upload-time = "2026-05-10T18:15:57.544Z" },
{ url = "https://files.pythonhosted.org/packages/b0/da/5d1876984b3746c85dbd219dbfcb73c85f54ee263fd32e5b2a632ec14571/librt-0.11.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d5b0eea49f5562861ee8d757a32ef7d559c1d35be2aaaa1ec28941d74c9ffc8a", size = 513082, upload-time = "2026-05-10T18:15:58.805Z" },
{ url = "https://files.pythonhosted.org/packages/19/6e/55bdf5d5ca00c3e18430690bf2c953d8d3ffd3c337418173d33dec985dc9/librt-0.11.0-cp312-cp312-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0d1029d7e1ae1a7e647ed6fb5df8c4ce2dffefb7a9f5fd1376a4554d96dac09f", size = 508105, upload-time = "2026-05-10T18:16:00.2Z" },
{ url = "https://files.pythonhosted.org/packages/07/10/f1f23a7c595ee90ece4d35c851e5d104b1311a887ed1b4ac4c35bbd13da8/librt-0.11.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:bc3ce6b33c5828d9e80592011a5c584cb2ce86edbc4088405f70da47dc1d1b3b", size = 522268, upload-time = "2026-05-10T18:16:01.708Z" },
{ url = "https://files.pythonhosted.org/packages/b6/02/5720f5697a7f54b78b3aefbe20df3a48cedcff1276618c4aa481177942ed/librt-0.11.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:936c5995f3514a42111f20099397d8177c79b4d7e70961e396c6f5a0a3566766", size = 527348, upload-time = "2026-05-10T18:16:03.496Z" },
{ url = "https://files.pythonhosted.org/packages/50/db/b4a47c6f91db4ff76348a0b3dd0cc65e090a078b765a810a62ff9434c3d3/librt-0.11.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:9bc0ca6ad9381cbe8e4aa6e5726e4c80c78115a6e9723c599ed1d73e092bc49d", size = 516294, upload-time = "2026-05-10T18:16:05.173Z" },
{ url = "https://files.pythonhosted.org/packages/9e/58/9384b2f4eb1ed1d273d40948a7c5c4b2360213b402ef3be4641c06299f9c/librt-0.11.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:070aa8c26c0a74774317a72df8851facc7f0f012a5b406557ac56992d92e1ec8", size = 553608, upload-time = "2026-05-10T18:16:06.839Z" },
{ url = "https://files.pythonhosted.org/packages/21/7b/5aa8848a7c6a9278c79375146da1812e695754ceec5f005e6043461a7315/librt-0.11.0-cp312-cp312-win32.whl", hash = "sha256:6bf14feb84b05ae945277395451998c89c54d0def4070eb5c08de544930b245a", size = 101879, upload-time = "2026-05-10T18:16:08.103Z" },
{ url = "https://files.pythonhosted.org/packages/37/33/8a745436944947575b584231750a41417de1a38cf6a2e9251d1065651c09/librt-0.11.0-cp312-cp312-win_amd64.whl", hash = "sha256:75672f0bc524ede266287d532d7923dbce94c7514ad07627bac3d0c6d92cc4d9", size = 119831, upload-time = "2026-05-10T18:16:09.174Z" },
{ url = "https://files.pythonhosted.org/packages/59/67/a6739ac96e28b7855808bdb0370e250606104a859750d209e5a0716fe7ab/librt-0.11.0-cp312-cp312-win_arm64.whl", hash = "sha256:2f10cf143e4a9bb0f4f5af568a00df94a2d69ef41c2579584454bb0fe5cc642c", size = 103470, upload-time = "2026-05-10T18:16:10.369Z" },
{ url = "https://files.pythonhosted.org/packages/82/61/e59168d4d0bf2bf90f4f0caf7a001bfc60254c3af4586013b04dc3ef517b/librt-0.11.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:78dc31f7fdfe9c9d0eb0e8f42d139db230e826415bbcabd9f0e9faaaee909894", size = 144119, upload-time = "2026-05-10T18:16:11.771Z" },
{ url = "https://files.pythonhosted.org/packages/61/fd/caa1d60b12f7dd79ccea23054e06eeaebe266a5f52c40a6b651069200ce5/librt-0.11.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:fa475675db22290c3158e1d42326d0f5a65f04f44a0e68c3630a25b53560fb9c", size = 143565, upload-time = "2026-05-10T18:16:13.334Z" },
{ url = "https://files.pythonhosted.org/packages/b8/a9/dc744f5c2b4978d48db970be29f22716d3413d28b14ad99740817315cf2c/librt-0.11.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:621db29691044bdeda22e789e482e1b0f3a985d90e3426c9c6d17606416205ea", size = 485395, upload-time = "2026-05-10T18:16:14.729Z" },
{ url = "https://files.pythonhosted.org/packages/8f/21/7f8e97a1e4dae952a5a95948f6f8507a173bc1e669f54340bba6ca1ca31b/librt-0.11.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:a9010e2ed5b3a9e158c5fd966b3ab7e834bb3d3aacc8f66c91dd4b57a3799230", size = 479383, upload-time = "2026-05-10T18:16:16.321Z" },
{ url = "https://files.pythonhosted.org/packages/a6/6d/d8ee9c114bebf2c50e29ec2aa940826fccb62a645c3e4c18760987d0e16d/librt-0.11.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7c39513d8b7477a2e1ed8c43fc21c524e8d5a0f8d4e8b7b074dbdbe7820a08e2", size = 513010, upload-time = "2026-05-10T18:16:17.647Z" },
{ url = "https://files.pythonhosted.org/packages/f0/43/0b5708af2bd30a46400e72ba6bdaa8f066f15fb9a688527e34220e8d6c06/librt-0.11.0-cp313-cp313-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:7aef3cf1d5af86e770ab04bfd993dfc4ae8b8c17f66fb77dd4a7d50de7bbb1a3", size = 508433, upload-time = "2026-05-10T18:16:19.309Z" },
{ url = "https://files.pythonhosted.org/packages/4a/50/356187247d09013490481033183b3532b58acf8028bcb34b2b56a375c9b2/librt-0.11.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:557183ddc36babe46b27dd60facbd5adb4492181a5be887587d57cda6e092f21", size = 522595, upload-time = "2026-05-10T18:16:20.642Z" },
{ url = "https://files.pythonhosted.org/packages/40/e7/c6ac4240899c7f3248079d5a9900debe0dadb3fdeaf856684c987105ba47/librt-0.11.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:83d3e1f72bd42f6c5c0b7daec530c3f829bd02db42c70b8ddf0c2d90a2459930", size = 527255, upload-time = "2026-05-10T18:16:22.352Z" },
{ url = "https://files.pythonhosted.org/packages/eb/b5/a81322dbeedeeaf9c1ee6f001734d28a09d8383ac9e6779bc24bbd0743c6/librt-0.11.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:4ce1f21fbe589bc1afd7872dece84fb0e1144f794a288e58a10d2c54a55c43be", size = 516847, upload-time = "2026-05-10T18:16:23.627Z" },
{ url = "https://files.pythonhosted.org/packages/ae/66/6e6323787d592b55204a42595ff1102da5115601b53a7e9ddebc889a6da5/librt-0.11.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:970b09f7044ea2b64c9da42fd3d335666518cfd1c6e8a182c95da73d0214b41e", size = 553920, upload-time = "2026-05-10T18:16:25.025Z" },
{ url = "https://files.pythonhosted.org/packages/9c/21/623f8ca230857102066d9ca8c6c1734995908c4d0d1bee7bb2ef0021cb33/librt-0.11.0-cp313-cp313-win32.whl", hash = "sha256:78fddc31cd4d3caa897ad5d31f856b1faadc9474021ad6cb182b9018793e254e", size = 101898, upload-time = "2026-05-10T18:16:26.649Z" },
{ url = "https://files.pythonhosted.org/packages/b3/1d/b4ebd44dd723f768469007515cb92251e0ae286c94c140f374801140fa74/librt-0.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:8ca8aa88751a775870b764e93bad5135385f563cb8dcee399abf034ea4d3cb47", size = 119812, upload-time = "2026-05-10T18:16:27.859Z" },
{ url = "https://files.pythonhosted.org/packages/3b/e4/b2f4ca7965ca373b491cdb4bc25cdb30c1649ca81a8782056a83850292a9/librt-0.11.0-cp313-cp313-win_arm64.whl", hash = "sha256:96f044bb325fd9cf1a723015638c219e9143f0dfbc0ca54c565df2b7fc748b44", size = 103448, upload-time = "2026-05-10T18:16:29.066Z" },
{ url = "https://files.pythonhosted.org/packages/29/eb/dbce197da4e227779e56b5735f2decc3eb36e55a1cdbf1bd65d6639d76c1/librt-0.11.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:4a017a95e5837dc15a8c5661d60e05daa96b90908b1aa6b7acdf443cd25c8ebd", size = 143345, upload-time = "2026-05-10T18:16:30.674Z" },
{ url = "https://files.pythonhosted.org/packages/76/a3/254bebd0c11c8ba684018efb8006ff22e466abce445215cca6c778e7d9de/librt-0.11.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:b1ecbd9819deccc39b7542bf4d2a740d8a620694d39989e58661d3763458f8d4", size = 143131, upload-time = "2026-05-10T18:16:32.037Z" },
{ url = "https://files.pythonhosted.org/packages/f1/3f/f77d6122d21ac7bf6ae8a7dfced1bd2a7ac545d3273ebdcaf8042f6d619f/librt-0.11.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7da327dacd7be8f8ec36547373550744a3cc0e536d54665cd83f8bcd961200e8", size = 477024, upload-time = "2026-05-10T18:16:33.493Z" },
{ url = "https://files.pythonhosted.org/packages/ac/0a/2c996dadebaa7d9bbbd43ef2d4f3e66b6da545f838a41694ef6172cebec8/librt-0.11.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:0dc56b1f8d06e60db362cc3fdae206681817f86ce4725d34511473487f12a34b", size = 474221, upload-time = "2026-05-10T18:16:34.864Z" },
{ url = "https://files.pythonhosted.org/packages/0a/7e/f5d92af8486b8272c23b3e686b46ff72d89c8169585eb61eef01a2ac7147/librt-0.11.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:05fb8fb2ab90e21c8d12ea240d744ad514da9baf381ebfa70d91d20d21713175", size = 505174, upload-time = "2026-05-10T18:16:36.705Z" },
{ url = "https://files.pythonhosted.org/packages/af/1a/cb0734fe86398eb33193ab753b7326255c74cac5eb09e76b9b16536e7adb/librt-0.11.0-cp314-cp314-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:cae74872be221df4374d10fec61f93ed1513b9546ea84f2c0bf73ab3e9bd0b03", size = 497216, upload-time = "2026-05-10T18:16:38.418Z" },
{ url = "https://files.pythonhosted.org/packages/18/06/094820f91558b66e29943c0ec41c9914f460f48dd51fc503c3101e10842d/librt-0.11.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:32bcc918c0148eb7e3d57385125bac7e5f9e4359d05f07448b09f6f778c2f31c", size = 513921, upload-time = "2026-05-10T18:16:39.848Z" },
{ url = "https://files.pythonhosted.org/packages/0b/c2/00de9018871a282f530cacb457d5ec0428f6ac7e6fedde9aff7468d9fb04/librt-0.11.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:f9743fc99135d5f78d2454435615f6dec0473ca507c26ce9d92b10b562a280d3", size = 520850, upload-time = "2026-05-10T18:16:41.471Z" },
{ url = "https://files.pythonhosted.org/packages/51/9d/64631832348fd1834fb3a61b996434edddaaf25a31d03b0a76273159d2cf/librt-0.11.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:5ba067f4aadae8fda802d91d2124c90c42195ff32d9161d3549e6d05cfe26f96", size = 504237, upload-time = "2026-05-10T18:16:43.15Z" },
{ url = "https://files.pythonhosted.org/packages/a5/ec/ae5525eb16edc827a044e7bb8777a455ff95d4bca9379e7e6bddd7383647/librt-0.11.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:de3bf945454d032f9e390b85c4072e0a0570bf825421c8be0e71209fa65e1abe", size = 546261, upload-time = "2026-05-10T18:16:44.408Z" },
{ url = "https://files.pythonhosted.org/packages/5a/09/adce371f27ca039411da9659f7430fcc2ba6cd0c7b3e4467a0f091be7fa9/librt-0.11.0-cp314-cp314-win32.whl", hash = "sha256:d2277a05f6dcb9fd13db9566aac4fabd68c3ea1ea46ee5567d4eef8efa495a2f", size = 96965, upload-time = "2026-05-10T18:16:46.039Z" },
{ url = "https://files.pythonhosted.org/packages/d6/ee/8ac720d98548f173c7ce2e632a7ca94673f74cacd5c8162a84af5b35958a/librt-0.11.0-cp314-cp314-win_amd64.whl", hash = "sha256:ab73e8db5e3f564d812c1f5c3a175930a5f9bc96ccb5e3b22a34d7858b401cf7", size = 115151, upload-time = "2026-05-10T18:16:47.133Z" },
{ url = "https://files.pythonhosted.org/packages/94/20/c900cf14efeb09b6bef2b2dff20779f73464b97fd58d1c6bccc379588ae3/librt-0.11.0-cp314-cp314-win_arm64.whl", hash = "sha256:aea3caa317752e3a466fa8af45d91ee0ea8c7fdd96e42b0a8dd9b76a7931eba1", size = 98850, upload-time = "2026-05-10T18:16:48.597Z" },
{ url = "https://files.pythonhosted.org/packages/0c/71/944bfe4b64e12abffcd3c15e1cce07f72f3d55655083786285f4dedeb532/librt-0.11.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:d1b36540d7aaf9b9101b3a6f376c8d8e9f7a9aec93ed05918f2c69d493ffef72", size = 151138, upload-time = "2026-05-10T18:16:49.839Z" },
{ url = "https://files.pythonhosted.org/packages/b6/10/99e64a5c86989357fda078c8143c533389585f6473b7439172dd8f3b3b2d/librt-0.11.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:efbb343ab2ce3540f4ecbe6315d677ed70f37cd9a72b1e58066c918ca83acbaa", size = 151976, upload-time = "2026-05-10T18:16:51.062Z" },
{ url = "https://files.pythonhosted.org/packages/21/31/5072ad880946d83e5ea4147d6d018c78eefce85b77819b19bdd0ee229435/librt-0.11.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:aa0dd688aab3f7914d3e6e5e3554978e0383312fb8e771d84be008a35b9ee548", size = 557927, upload-time = "2026-05-10T18:16:52.632Z" },
{ url = "https://files.pythonhosted.org/packages/5e/8d/70b5fb7cfbab60edbe7381614ab985da58e144fbf465c86d44c95f43cdca/librt-0.11.0-cp314-cp314t-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:f5fb36b8c6c63fdcbb1d526d94c0d1331610d43f4118cc1beb4efef4f3faacb2", size = 539698, upload-time = "2026-05-10T18:16:53.934Z" },
{ url = "https://files.pythonhosted.org/packages/fa/a3/ba3495a0b3edbd24a4cae0d1d3c64f39a9fc45d06e812101289b50c1a619/librt-0.11.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4a9a237d13addb93715b6fee74023d5ee3469b53fce527626c0e088aa585805f", size = 577162, upload-time = "2026-05-10T18:16:55.589Z" },
{ url = "https://files.pythonhosted.org/packages/f7/db/36e25fb81f99937ff1b96612a1dc9fd66f039cb9cc3aee12c01fac31aab9/librt-0.11.0-cp314-cp314t-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5ddd17bd87b2c56ddd60e546a7984a2e64c4e8eab92fb4cf3830a48ad5469d51", size = 566494, upload-time = "2026-05-10T18:16:56.975Z" },
{ url = "https://files.pythonhosted.org/packages/33/0d/3f622b47f0b013eeb9cf4cc07ae9bfe378d832a4eec998b2b209fe84244d/librt-0.11.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:bd43992b4473d42f12ff9e68326079f0696d9d4e6000e8f39a0238d482ba6ee2", size = 596858, upload-time = "2026-05-10T18:16:58.374Z" },
{ url = "https://files.pythonhosted.org/packages/a9/02/71b90bc93039c46a2000651f6ad60122b114c8f54c4ad306e0e96f5b75ad/librt-0.11.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:f8e3e8056dd674e279741485e2e512d6e9a751c7455809d0114e6ebf8d781085", size = 590318, upload-time = "2026-05-10T18:16:59.676Z" },
{ url = "https://files.pythonhosted.org/packages/04/04/418cb3f75621e2b761fb1ab0f017f4d70a1a72a6e7c74ee4f7e8d198c2f3/librt-0.11.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:c1f708d8ae9c56cf38a903c44297243d2ec83fd82b396b977e0144a3e76217e3", size = 575115, upload-time = "2026-05-10T18:17:01.007Z" },
{ url = "https://files.pythonhosted.org/packages/cc/2c/5a2183ac58dd911f26b5d7e7d7d8f1d87fcecdddd99d6c12169a258ff62c/librt-0.11.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:0add982e0e7b9fc14cf4b33789d5f13f66581889b88c2f58099f6ce8f92617bd", size = 617918, upload-time = "2026-05-10T18:17:02.682Z" },
{ url = "https://files.pythonhosted.org/packages/15/1f/dc6771a52592a4451be6effa200cbfc9cec61e4393d3033d81a9d307961d/librt-0.11.0-cp314-cp314t-win32.whl", hash = "sha256:2b481d846ac894c4e8403c5fd0e87c5d11d6499e404b474602508a224ff531c8", size = 103562, upload-time = "2026-05-10T18:17:03.99Z" },
{ url = "https://files.pythonhosted.org/packages/62/4a/7d1415567027286a75ba1093ec4aca11f073e0f559c530cf3e0a757ad55c/librt-0.11.0-cp314-cp314t-win_amd64.whl", hash = "sha256:28edb433edde181112a908c78907af28f964eabc15f4dd16c9d66c834302677c", size = 124327, upload-time = "2026-05-10T18:17:05.465Z" },
{ url = "https://files.pythonhosted.org/packages/ce/62/b40b382fa0c66fee1478073eb8db352a4a6beda4a1adccf1df911d8c289c/librt-0.11.0-cp314-cp314t-win_arm64.whl", hash = "sha256:dee008f20b542e3cd162ba338a7f9ec0f6d23d395f66fe8aeeec3c9d067ea253", size = 102572, upload-time = "2026-05-10T18:17:06.809Z" },
]
[[package]]
name = "loguru"
version = "0.7.3"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "colorama", marker = "sys_platform == 'win32'" },
{ name = "win32-setctime", marker = "sys_platform == 'win32'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/3a/05/a1dae3dffd1116099471c643b8924f5aa6524411dc6c63fdae648c4f1aca/loguru-0.7.3.tar.gz", hash = "sha256:19480589e77d47b8d85b2c827ad95d49bf31b0dcde16593892eb51dd18706eb6", size = 63559, upload-time = "2024-12-06T11:20:56.608Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/0c/29/0348de65b8cc732daa3e33e67806420b2ae89bdce2b04af740289c5c6c8c/loguru-0.7.3-py3-none-any.whl", hash = "sha256:31a33c10c8e1e10422bfd431aeb5d351c7cf7fa671e3c4df004162264b28220c", size = 61595, upload-time = "2024-12-06T11:20:54.538Z" },
]
[[package]]
name = "lxml"
version = "6.1.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/05/3b/aab6728cae887456f409b4d75e8a01856e4f04bd510de38052a47768b680/lxml-6.1.1.tar.gz", hash = "sha256:ba96ae44888e0185281e937633a743ea90d5a196c6000f82565ebb0580012d40", size = 4197430, upload-time = "2026-05-18T19:19:06.424Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/62/b0/83f481780d1548750b8ce2ec824073deef2f452d9cd1a6faff8507e3d16d/lxml-6.1.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:53b7d2b7a10b1c35c0a5e21e9224accf60c1bbfba523990732e521b2b73adef2", size = 8526461, upload-time = "2026-05-18T19:17:25.862Z" },
{ url = "https://files.pythonhosted.org/packages/b9/d5/30fa0f808002c7329397bfbb24e306789c0b29f04aa5842c07b174b4216f/lxml-6.1.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ff3f333630ab480244a1bff72043e511a91eb22e7595dead8653ee5612dd8f3d", size = 4595375, upload-time = "2026-05-18T19:17:34.555Z" },
{ url = "https://files.pythonhosted.org/packages/4f/d2/edb71cf0e561581a7c5eb2626244320eb04e9f8ce6d563184fd668b45073/lxml-6.1.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a4bbea04c97f6d78a48e3fbc1cb9116d2780b1b39e03a23f6eb9b603fd61f510", size = 4923654, upload-time = "2026-05-18T19:17:42.917Z" },
{ url = "https://files.pythonhosted.org/packages/4c/77/1bc7eeb0de4577d783fb625aa092cc9357883bba35845a3666bf1259f3dc/lxml-6.1.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:db1d75f6617a49c1c01bc7023713e0ff59ab32c9579ae62a7674c0e34f3b0b0a", size = 5067921, upload-time = "2026-05-18T19:17:49.175Z" },
{ url = "https://files.pythonhosted.org/packages/1b/3c/c0690d74bd2bc17bc03b5b0d093569ead597dd0bfa088bf99eef8c24e19c/lxml-6.1.1-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a12689be69a28ddaa0ab99a5a1137da2afd5f8f16df7b5680b66f616d3eda1d", size = 5002456, upload-time = "2026-05-18T19:17:59.715Z" },
{ url = "https://files.pythonhosted.org/packages/66/8d/d1b3271af0c0f1e27e8472a849e4d2c65bc7766884b9ad2da9e76e145c88/lxml-6.1.1-cp311-cp311-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:18b73c339ae29b90fd2d06e58ebd555a751bde9cd6bbd36cc0281b9a2c94e9d8", size = 5202776, upload-time = "2026-05-18T19:18:08.924Z" },
{ url = "https://files.pythonhosted.org/packages/7a/45/689824ffb237fd10125ad273f32b28ff04dc6203c2822c85ff65a93df65e/lxml-6.1.1-cp311-cp311-manylinux_2_28_i686.whl", hash = "sha256:752d3bbfe874715ccd0aec7f88d7fc623c0f1fd7aa7b3238a084e017bad2a009", size = 5329945, upload-time = "2026-05-18T19:18:13.673Z" },
{ url = "https://files.pythonhosted.org/packages/5d/c0/ef73af53767e958fd87d437c170f272e2f6e6c0f854939f133a895f1e711/lxml-6.1.1-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:6b1761fbf9ec984e2e9d9c589ef5f5fd684b7c19f92aadd567a26c5224958db6", size = 4659237, upload-time = "2026-05-18T19:18:18.657Z" },
{ url = "https://files.pythonhosted.org/packages/a0/5e/e1158e40397585e91cb0472374a1f63d0926a1ddeaa92f13d1a1ffe306d5/lxml-6.1.1-cp311-cp311-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d680fbcb768404c601ecb43519ecd8461f6954cb11c06a78962f666832ccfca8", size = 5265904, upload-time = "2026-05-18T19:18:24.883Z" },
{ url = "https://files.pythonhosted.org/packages/a0/16/8687e5d1400ed1c0bc41dace232ebb7553952b618ea1f2e5fb6e2cfbbe23/lxml-6.1.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:162af1091cd785f2f27e62d3547ae9bc58ec5c86dd314d67021fd02463708d83", size = 5045225, upload-time = "2026-05-18T19:17:20.073Z" },
{ url = "https://files.pythonhosted.org/packages/ca/18/d877bd1ae2e5ffdfd4836565aba350db31feb2f2656d6ce70316ed66a05e/lxml-6.1.1-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:e9308ff8241c532df3f3e570f9a5aeed6c853f888512ba4b75638d7c11c95ef6", size = 4712721, upload-time = "2026-05-18T19:17:40.512Z" },
{ url = "https://files.pythonhosted.org/packages/44/4d/1f44fd1d770b10dacbf6b5c6e520f4d6e0708744930f719dc04e67cab981/lxml-6.1.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:5f6994074ebae6ffb04447268e37dc16edc304f9859cf91acb86e0af6c1b395c", size = 5252549, upload-time = "2026-05-18T19:17:51.236Z" },
{ url = "https://files.pythonhosted.org/packages/64/5d/1d66b84f850089254c230ef6ea6b267a5a54e2e179a5d960036a05d501d7/lxml-6.1.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:80c2dfadb855da477cf73373ad29a333535dedb9b12bad02c9814c8e2b43bf08", size = 5226877, upload-time = "2026-05-18T19:18:00.875Z" },
{ url = "https://files.pythonhosted.org/packages/ad/00/84c4b5302d42a2d0184f38d538c8a197f33b52a50bd4f7bcfe990bce3036/lxml-6.1.1-cp311-cp311-win32.whl", hash = "sha256:30a89d3ac8faec007453fb541f3f46807eeec88edd5826f6e3fe001752a2c621", size = 3594072, upload-time = "2026-05-18T19:17:12.714Z" },
{ url = "https://files.pythonhosted.org/packages/61/9d/2e2f7d876349f45e0f3e29f72da311668853d59b58d473a2dea4f0160135/lxml-6.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:abbefa31eee84842140f67acef1c828e28bba8bbf0c3bc6e5492a9af88152c28", size = 4025469, upload-time = "2026-05-18T19:17:50.566Z" },
{ url = "https://files.pythonhosted.org/packages/b0/d5/570e6390e4110331e6208b2ba83d1482cc9146808ee118b22824a34c1070/lxml-6.1.1-cp311-cp311-win_arm64.whl", hash = "sha256:dcb292aa7fe485ceff7af4f92e46c5af397daec5dff64871a528f0fc47a3cc5b", size = 3667640, upload-time = "2026-05-19T19:22:48.293Z" },
{ url = "https://files.pythonhosted.org/packages/6a/6e/c4add832b6fc1e887125b96f880d7b9b70aae5248718e046b1704bcac4b9/lxml-6.1.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:104c09bda8d2a562824c0e319d0768ce26a779b7601e0931d33b09b53c392ef7", size = 8570821, upload-time = "2026-05-18T19:17:42.068Z" },
{ url = "https://files.pythonhosted.org/packages/22/00/ff3009c88e65de8011630acf8ab5a09cb2becd2aaf47fba2f3449f6224e9/lxml-6.1.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:25c6997a9a534e016695a0ba06b2f07945de682731ff01065b6d5a4474179da1", size = 4624252, upload-time = "2026-05-18T19:17:47.897Z" },
{ url = "https://files.pythonhosted.org/packages/42/95/bb63f0fd62e554fe078e1fb3c8fe9083c14ddc7ad7fa178d10e57e071ac7/lxml-6.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c921ba5c51e4e9f63b8b00267d06566e1f63407408a0496da2d1d0bfc819c7fc", size = 4930746, upload-time = "2026-05-18T19:18:29.637Z" },
{ url = "https://files.pythonhosted.org/packages/eb/99/0013e8d9b5960f4f041cf0b73e2f80c23eb5205b1f7bfb20203243651359/lxml-6.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:54a7f95e4de5fb94e2f9f4b9055c6ba33bf3d628fd77a1d647c5923caa2cdcdc", size = 5093723, upload-time = "2026-05-18T19:18:34.168Z" },
{ url = "https://files.pythonhosted.org/packages/29/91/317b332636bfc7bddcff828d41b3307f50043f4b237e40849c333d80fa1a/lxml-6.1.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96f2ec43df44b1f76249ee0a615334f9b5b060e1c8bd90e706dad2d14d02f383", size = 5005557, upload-time = "2026-05-18T19:18:39.798Z" },
{ url = "https://files.pythonhosted.org/packages/42/2f/cc9bf06afe70f9c9093ae60855d9759da9db601ec4080f7473319666ffd7/lxml-6.1.1-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:70ef8a7e102a1508f8121aae5b0867abd663f72c14f0a9c937e6554cb4587b7b", size = 5631036, upload-time = "2026-05-18T19:18:44.858Z" },
{ url = "https://files.pythonhosted.org/packages/08/f6/af32e23e563971ffb0fb86be52bc5be5c2c118858ffc119bf6a9039b173d/lxml-6.1.1-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ebe6af670449830d6d9b752c256a983291c766a1365ba5d5460048f9e33a7818", size = 5240367, upload-time = "2026-05-18T19:18:49.217Z" },
{ url = "https://files.pythonhosted.org/packages/78/83/8555d40948b09ce86f1bd0c68a7ac31d07b1929f92cc1b074006c97ef2d2/lxml-6.1.1-cp312-cp312-manylinux_2_28_i686.whl", hash = "sha256:27acc820660aaffa4f7c087f29120e12980f7779d56d8492d263170111284740", size = 5350171, upload-time = "2026-05-18T19:18:52.779Z" },
{ url = "https://files.pythonhosted.org/packages/63/75/5d92da93729b7bad783689e6496049fa40927b45bec7bf183c981de3ca70/lxml-6.1.1-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:1db753c9115ec7100d073b744d17e25e88a8f90f5c39b2f5dd878149af59671f", size = 4694874, upload-time = "2026-05-18T19:18:55.139Z" },
{ url = "https://files.pythonhosted.org/packages/c5/b5/3aad415a9a25b822e783f15deeb4dffccf5113030f1afa2222dd929313d9/lxml-6.1.1-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c4f469aebd783bb741c2ecb2a681008fd26bfe5c16a9a72ed5467f834e810df2", size = 5244492, upload-time = "2026-05-18T19:19:01.28Z" },
{ url = "https://files.pythonhosted.org/packages/f1/a1/5fcf7eb9904b80086aa47dcf0027de07b1bb990afad2e6823144c368ae04/lxml-6.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:766b010012d59470072c1816b5b6c69f1d243e5db36ea5968e94accf430a4635", size = 5048232, upload-time = "2026-05-18T19:18:12.67Z" },
{ url = "https://files.pythonhosted.org/packages/77/74/1f601b63c7a69fcdf10fa9b148c81da8442204194f6c55509cc485c786b9/lxml-6.1.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:b8d812c6011c08b8111a15e54dd990b8923692d80adf35488bee34026c35accf", size = 4777023, upload-time = "2026-05-18T19:18:15.928Z" },
{ url = "https://files.pythonhosted.org/packages/a2/b9/7a78f51aec95b1bf780d78e12705a9f6533284f8693dc5c0e6724fa53d3f/lxml-6.1.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:fe0306bd29505a9177aac19f1877174b0e7422c222a59f70b2cd41633448c3dc", size = 5645773, upload-time = "2026-05-18T19:18:23.223Z" },
{ url = "https://files.pythonhosted.org/packages/a5/6e/98a7b7ad54e4e74fa1f20fff776913980619d0ebe5558232d7da6580bdd8/lxml-6.1.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:5ba186ad207446c65d3bb3d3e0412b032b1d9f595e59861e2354798c5703d955", size = 5233088, upload-time = "2026-05-18T19:18:31.433Z" },
{ url = "https://files.pythonhosted.org/packages/65/d1/bc0ed2427bf609f2ee10da303a6a226f9c8bce94f945dc29a32ce55de6e4/lxml-6.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:aa366a1e55b8ebfe8ca8ddc3cfe75c8ebade181aeb0f661d0cb05986b647f72a", size = 5260995, upload-time = "2026-05-18T19:18:37.091Z" },
{ url = "https://files.pythonhosted.org/packages/69/8b/6772e1a4b513fc50a8d931f19edde0e13ae6918510a1e13ff67864f3e5ed/lxml-6.1.1-cp312-cp312-win32.whl", hash = "sha256:126c93f7f56f0eda92f6d8c619edc463a4f23d9252f1c9d0405a76f25fa9f11a", size = 3596382, upload-time = "2026-05-18T19:17:18.37Z" },
{ url = "https://files.pythonhosted.org/packages/1b/89/45198e9624762af2dfd2cb8782598477ceb29f6e59caab560388ae1f4ec1/lxml-6.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:26e6eda8d38c1fcab1090dd196ee87cbd13788e531937610e2589085de074e77", size = 3997255, upload-time = "2026-05-18T19:17:56.781Z" },
{ url = "https://files.pythonhosted.org/packages/90/a9/7a54b6834088d9ae528a7b780584ba6a39a9457b0ac330479f20ffbc9449/lxml-6.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:6540377fbd53fe1b629172288c464fb18db11ce1fa7dc15891da10aa9dcc3e7f", size = 3659610, upload-time = "2026-05-19T19:22:50.843Z" },
{ url = "https://files.pythonhosted.org/packages/a5/eb/7e6f37c5584ccbb2ff267f56fd0339016938c1c8684cfefab9b33ffc2f36/lxml-6.1.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:68a9198d0fc122d14bb76837de9aa80cf84caed990b5b237f532ed87d3706736", size = 8559780, upload-time = "2026-05-18T19:17:57.661Z" },
{ url = "https://files.pythonhosted.org/packages/a1/36/587c2521cf23a2cd6c9c22108aa7528f683a1f195ed7ccd23a4b1786ad36/lxml-6.1.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7d47866cb32fb503450b6edc9df355d10dc49836af2e89901bd6ac6b0896d9d9", size = 4618006, upload-time = "2026-05-18T19:18:04.452Z" },
{ url = "https://files.pythonhosted.org/packages/6e/ca/ab7bfe2bf4c972af5e7878262845ead3a24a929a9b04bc11c7c1ece6c82a/lxml-6.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb7c9811bfaa8b1ed5ed319f5d370dfbcaa59d52ea64be2a5a85e18195930354", size = 4924139, upload-time = "2026-05-18T19:19:04.873Z" },
{ url = "https://files.pythonhosted.org/packages/6b/55/a0c72851dfee5ecc689f949723a73dea457758912542cb955b108eaf0d8f/lxml-6.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:762ff394d5bd56da0cf034a23dcce4e13923f15321a2adfa2ac00201dc6d3fca", size = 5082329, upload-time = "2026-05-18T19:19:09.728Z" },
{ url = "https://files.pythonhosted.org/packages/f0/b6/0608f7d61a3b96cc67e5648a3d906e31a5082093e10e7be65b3886289938/lxml-6.1.1-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a088f287f7d8275a33c07f2cac6c50b9319309a0200a39e7e75d80c707723099", size = 4993564, upload-time = "2026-05-18T19:19:13.608Z" },
{ url = "https://files.pythonhosted.org/packages/4c/66/ae227524b066d29d55bf0b453d93d2d793c40218657d643dcbbca13b8faf/lxml-6.1.1-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e902da4b04e6b52e5893900d4b8ab46068f75f3561f01bf1080957f9fd932ed6", size = 5613467, upload-time = "2026-05-18T19:19:16.228Z" },
{ url = "https://files.pythonhosted.org/packages/a6/76/dbe4a00b50385e40194231dcfe5a12c059de7cf90e89c83407d2b085b719/lxml-6.1.1-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1d4962d4c66bf830a7e59ed6cfc17d148149898a3aefa8ec6e59763e6e3ed085", size = 5228304, upload-time = "2026-05-18T19:19:19.354Z" },
{ url = "https://files.pythonhosted.org/packages/1c/01/00b1b8442ed2041793336868ba0b9ea4b13d7da7c085c6404c207a63bf79/lxml-6.1.1-cp313-cp313-manylinux_2_28_i686.whl", hash = "sha256:581d4c8ae690a6609e64862dd6b7c2489635c2d13907fc2b20f2bc200ff1d21e", size = 5341607, upload-time = "2026-05-18T19:19:22.297Z" },
{ url = "https://files.pythonhosted.org/packages/63/36/1ad29931e9a4638bb707869f01d423a6c815f82152138d1a40dfcfde2b95/lxml-6.1.1-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:876e1ff5930ed8bf295ec5ef9a8155e9b6b1876bbf1deed8b3a8069311875a8f", size = 4700168, upload-time = "2026-05-18T19:19:25.133Z" },
{ url = "https://files.pythonhosted.org/packages/3c/d1/a9536cecf9be18a0dc72d32bead283a2332d1ffebd2dd3ac70ce444686e5/lxml-6.1.1-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9eb9b5a968f6e0f6d640092a567e14529ff8cea2e29d00da6f78a79fa49f013c", size = 5232487, upload-time = "2026-05-18T19:19:28.603Z" },
{ url = "https://files.pythonhosted.org/packages/0e/77/b4fb1e03bf5d130e879214d3100092e386418807fb74dd0adc4b0a48f351/lxml-6.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:aa49e06d94aba782c6a02eecb7e507969e7e7a41b267f1b359bb35585f295d5b", size = 5044231, upload-time = "2026-05-18T19:18:42.246Z" },
{ url = "https://files.pythonhosted.org/packages/26/4c/d00daeeb0a5530c4028a9232aa1b93db3ef4ed2158c116ea73c79a9765b3/lxml-6.1.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:70cdfd80589d59e43e18005dd7244e8895e93db8ab6a620b7e23df5445a4e3d2", size = 4769450, upload-time = "2026-05-18T19:18:48.013Z" },
{ url = "https://files.pythonhosted.org/packages/ed/6a/715a3a8d156ce42f29cf014706f5410c2ff3b02267774110fc23266409fe/lxml-6.1.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:aad9aa39483ed8ec44d6d2e59e5b98a0d80676ef0d92f44bfc374836111f62f5", size = 5635874, upload-time = "2026-05-18T19:18:51.914Z" },
{ url = "https://files.pythonhosted.org/packages/45/37/0544bc21dde2a88f3a17b504e6fc79c0e01d25a33c2f6079724e9e72b9c7/lxml-6.1.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:d49514be2f28d895c38cf9d2b72d7b9a07d00314519f456c0b50b53cfcf4c785", size = 5223987, upload-time = "2026-05-18T19:18:59.715Z" },
{ url = "https://files.pythonhosted.org/packages/4d/f8/f6a5e8185bcb28c2befae3d31f8e3df3b811cb0f47746517a81279fcafe1/lxml-6.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:47402e62c52ff5988c1e8c6c63177f5708bccf48e366dea4e3dcf1e645e04947", size = 5250276, upload-time = "2026-05-18T19:19:03.834Z" },
{ url = "https://files.pythonhosted.org/packages/c7/f2/1a2b9f1b7a49d45495369be7ef9ad05b262930f2eab3e3145706fca8083f/lxml-6.1.1-cp313-cp313-win32.whl", hash = "sha256:3483644525531e1d5762b0c44a8e18b6efba321b6dcf8a8952de10b037618bca", size = 3596903, upload-time = "2026-05-18T19:17:29.863Z" },
{ url = "https://files.pythonhosted.org/packages/e6/99/f4ffb024f238eec2131aaa09f3278fb6129cf892741bf68e1fc1afb8c100/lxml-6.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:a10bd2fd62e8ce916ececb342f348f190724a098c1faa056fdfb2a22ad5e8660", size = 3995869, upload-time = "2026-05-18T19:18:02.596Z" },
{ url = "https://files.pythonhosted.org/packages/d1/53/70eb8c5c6037f27448f1e3c54ebede9545a801ae63f0a7254afca4fe8e45/lxml-6.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:424aa57aca0897eb922aef34395bd1289b3b6f04e6bae20ea123c0c7e333cffc", size = 3658490, upload-time = "2026-05-19T19:22:53.846Z" },
{ url = "https://files.pythonhosted.org/packages/13/e2/2e325795566de01d0d7c3bb57d3c370616b2d07b01214e84eec5d3b10963/lxml-6.1.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:19b7ab10b210b0b3ad7985d9ac4eb66ab09a90b20fe6e2f7ba55d01a234345d0", size = 8577146, upload-time = "2026-05-18T19:18:17.765Z" },
{ url = "https://files.pythonhosted.org/packages/93/cf/5630b5e4be7d2e6bee8efe83865c925221103cf0221303b104ce134b01e2/lxml-6.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c08e5c694306507275f2290073350c4f32e383db15213b2c69e7ff39c1193840", size = 4623866, upload-time = "2026-05-18T19:18:30.669Z" },
{ url = "https://files.pythonhosted.org/packages/d2/51/3904907c063451cf8d4a5c9fe0cad95fa1f4ec57f4e3884fa0731bd7a305/lxml-6.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:74a9717fd0d82effef5c2854f0d917231d5324b5a3eb7275c43ac9fa32f97a14", size = 4950022, upload-time = "2026-05-18T19:19:31.958Z" },
{ url = "https://files.pythonhosted.org/packages/94/cd/9c7611a51c37a2830928405817cc5d56a97f64fab83cc3f628748b135749/lxml-6.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:efe0374196335f93b53269acd811b944f2e6bdc88e8894f214bd636455484909", size = 5086695, upload-time = "2026-05-18T19:19:34.764Z" },
{ url = "https://files.pythonhosted.org/packages/da/d6/24e3b5906abb0b674ff2ae195bc3ce59708df2bcd17cf17703b2d7dd643a/lxml-6.1.1-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ac931cdc9442c1763b8a8f6cd62c0c938737eafc5be75eff88df55fc73bc0d00", size = 5031642, upload-time = "2026-05-18T19:19:37.771Z" },
{ url = "https://files.pythonhosted.org/packages/2d/db/6ec54f99019838bff54785c51da07f189eb4676861c5f2730962b0d8d665/lxml-6.1.1-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:aee395f5d0927f947758b4ec119fd5fc8ec71f07a1c5c52077b30b04c0fa6955", size = 5647338, upload-time = "2026-05-18T19:19:40.553Z" },
{ url = "https://files.pythonhosted.org/packages/42/3d/ef4dcfffd22d27a61805d8ed9f7fb888495bc6aa88648fa07c1eaa5586b6/lxml-6.1.1-cp314-cp314-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9395002973c827b3ed67db77e6ec09f092919a587022174554096a269378fb13", size = 5239528, upload-time = "2026-05-18T19:19:43.657Z" },
{ url = "https://files.pythonhosted.org/packages/62/bb/37fb3f0dff146bdcfa78eec47879273820b2a0bf350ec236ce14bd0b1c26/lxml-6.1.1-cp314-cp314-manylinux_2_28_i686.whl", hash = "sha256:73bc2086f141224ebddb7fc5c6a36ca58b31b94b561e1dfe8e073e3270fad1e7", size = 5350730, upload-time = "2026-05-18T19:19:46.307Z" },
{ url = "https://files.pythonhosted.org/packages/90/42/43253f168388df4fae1f38c01df36ddb9bee39e2048167b54cdcbae85ea3/lxml-6.1.1-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:3779def59032b81e44a5f70096ef6bf2082f8d901937dca354474ba09782e245", size = 4697530, upload-time = "2026-05-18T19:19:49.889Z" },
{ url = "https://files.pythonhosted.org/packages/eb/a8/c5a8504f81bbdfc8e7094c2c850cdb4ed6777fc4d5ddd9e5ab819f3b0d54/lxml-6.1.1-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:86c89b9d55ebf820ad7c90bc533410f0d098054f293351f10603c0c46ff598f5", size = 5250670, upload-time = "2026-05-18T19:19:53.199Z" },
{ url = "https://files.pythonhosted.org/packages/77/b7/c7e76ab18744d75e21f320ebf9ff9d1ceae2b54dd431ea5a64caf26c9672/lxml-6.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19607c6bbff2a44cf3fe8250abccd20942d3462473e0a721d01d379ed017e462", size = 5084485, upload-time = "2026-05-18T19:19:08.422Z" },
{ url = "https://files.pythonhosted.org/packages/31/31/b35c53f8ef7b7c31cacd23d3638652fff7bcd1deb6eedb709ab43b685908/lxml-6.1.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:c6ed5141a5c7507cf3ee76bd363b0d6f801e3321adc35b5d825a23115faa5465", size = 4737635, upload-time = "2026-05-18T19:19:12.321Z" },
{ url = "https://files.pythonhosted.org/packages/d9/06/31f23c813a7fe8e0cb1b175e915b08c9bf4e86d225b210feadbdbe519667/lxml-6.1.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:62aeb7e85b5d60320b9d77eef2e773994e2c0ce10121b277e0a19804e1654a5a", size = 5670681, upload-time = "2026-05-18T19:19:15.001Z" },
{ url = "https://files.pythonhosted.org/packages/1a/bc/ce619bccc89b1fd9ad8a8e1330ee3f3beff9f2ff95b712d7bbcdd6e22fc3/lxml-6.1.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:b1b963fd8f5caa68e99dfae060d54de1fe9cba899b8718b44a00cdca53c3e590", size = 5238229, upload-time = "2026-05-18T19:19:18.131Z" },
{ url = "https://files.pythonhosted.org/packages/2f/5d/b329acbbedc0b619ebc2be6cf7ee9ed07e80892c88d4dfd612c33805789a/lxml-6.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:63876be28efefa04a1df615b46770e82042cce445cfdce55160522f57b231ccb", size = 5264191, upload-time = "2026-05-18T19:19:21.118Z" },
{ url = "https://files.pythonhosted.org/packages/d6/85/be36fb1425b30db3c3f9df75fe86343ebffb79e6320bd7f588e25bfeac39/lxml-6.1.1-cp314-cp314-win32.whl", hash = "sha256:7f7a92e8583f06b1fd49d01158143b8461cfcd135dcb10ec807270a3051bd603", size = 3657202, upload-time = "2026-05-18T19:17:39.509Z" },
{ url = "https://files.pythonhosted.org/packages/b8/ce/3cf9a827342269f54d405a6202397de63f07c69cbd6ce7d183a3f0cba1e9/lxml-6.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:b2d444f2e66624d68e9c6b211e28a76e22fff5fcabcfff4deac18b529b7d4137", size = 4064497, upload-time = "2026-05-18T19:18:14.662Z" },
{ url = "https://files.pythonhosted.org/packages/d9/3e/1a957bde8f0760039e627f94699f82caa782c9d838d86c3d28245ee67212/lxml-6.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:3fd9728a2735fda14f4e8235830c86b539e9661e849665bf926d3f867943b4bf", size = 3741991, upload-time = "2026-05-19T19:22:59.111Z" },
{ url = "https://files.pythonhosted.org/packages/78/b2/00ed55b3a2efa4658fb795c38d1090ec9b3e8a6c3683d4441fa517f09c3b/lxml-6.1.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:787b2496d0dbe8cd180984e8d29e3a6f76e7ea34db781cb3bd55e4ba1ef8b4ee", size = 8827545, upload-time = "2026-05-18T19:18:41.193Z" },
{ url = "https://files.pythonhosted.org/packages/c0/73/74573db19baa618d5f266f2407898b087ff6927115b00b71e5fc1b700847/lxml-6.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:2c8daa471358dc2d6fcf02165e80ec68f77871a286df95bc5cc3816153b0fd2c", size = 4735736, upload-time = "2026-05-18T19:18:46.761Z" },
{ url = "https://files.pythonhosted.org/packages/16/02/6f7061f4f95f51e545d48e87647c54791d204a4e881be4156e7a26ba5338/lxml-6.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:acd7d70b64c0aae0c7922cca83d288a16f5f6da523637697872253415269baef", size = 4970291, upload-time = "2026-05-18T19:19:56.215Z" },
{ url = "https://files.pythonhosted.org/packages/b0/02/55fc057d8283427dea7d6edb102e7a840239c77a64a983d92f62a304c0e9/lxml-6.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4f0dd2f01f9f8a89f565d000e03abcf0a13d692a346c8d22f628d49af098777a", size = 5102822, upload-time = "2026-05-18T19:19:59.223Z" },
{ url = "https://files.pythonhosted.org/packages/e4/48/8e1cf78d89d66850121d9255a2a24414c98f775da93b90cf976956c24b14/lxml-6.1.1-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b7e8a14c8634bf6f7a568634cb395305a6d964aeb5b7ee32248094bed3a7e2c", size = 5027923, upload-time = "2026-05-18T19:20:01.549Z" },
{ url = "https://files.pythonhosted.org/packages/ed/00/0632a0647612c8af24d26997b3b961397daa9d5b2581444805933629a4cb/lxml-6.1.1-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:86281fbdd6a8162756f8d603f37e3435bfa38043adb79c6dc6a2dfee065e7525", size = 5595843, upload-time = "2026-05-18T19:20:03.93Z" },
{ url = "https://files.pythonhosted.org/packages/bc/86/ab008a7dc360711b66858d61c80a5979a70a09f2aa2b05d9698df80b803d/lxml-6.1.1-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c5d7152ec39ca7c402d8fb9bad86140a15b9503bd0c54484e3f1bbe3dd37ceca", size = 5224515, upload-time = "2026-05-18T19:20:06.381Z" },
{ url = "https://files.pythonhosted.org/packages/75/c6/2702ff375e728e34f56d9a45339a9cf7e4427e917f542225242d63a05afa/lxml-6.1.1-cp314-cp314t-manylinux_2_28_i686.whl", hash = "sha256:88d8cb75b9d82858497a5393e3c63cfbf03035225e4b35a49ed7ccb151e4dc0e", size = 5312511, upload-time = "2026-05-18T19:20:09.308Z" },
{ url = "https://files.pythonhosted.org/packages/b7/57/a5807c98f87a86f10ef9ffab35516df7c0f0c4b6d5d33e9f608ab9c04a31/lxml-6.1.1-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f64ec5397ea6a41fc1b4af0380d79b44a755b5531dcaccd9940fb260dca93038", size = 4639206, upload-time = "2026-05-18T19:20:11.704Z" },
{ url = "https://files.pythonhosted.org/packages/1f/e1/8a0a2c35734812395f4da4eaf33748a7e5705bfb2a58b128da764339d5ec/lxml-6.1.1-cp314-cp314t-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d34bbf07dbc7ca5970671b1512e928991fb5e9d95365636c9b2d8b4f53af405e", size = 5232404, upload-time = "2026-05-18T19:20:14.064Z" },
{ url = "https://files.pythonhosted.org/packages/c2/e2/0e6a4dd5ad84d01d99aa7bae7cfefd4a760a0e0f8176818241de17d9b6c0/lxml-6.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:17e0e18d4ad8adbd0399291bc44845b69d9dd68439a3cdebdf35ff902ec05072", size = 5083769, upload-time = "2026-05-18T19:19:23.758Z" },
{ url = "https://files.pythonhosted.org/packages/a0/7e/161f33d463f6ffc1c7679104b65086dea120080d49dde4d238f015aaee2f/lxml-6.1.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:3ab541146f1f6968c462d6c2ac495148e8cdba2f8347700b2141b6ec5a75bf52", size = 4758936, upload-time = "2026-05-18T19:19:27.256Z" },
{ url = "https://files.pythonhosted.org/packages/f1/fb/2369825e3f6ca99305bf9f7b7085fda91c8b0922a89e54d900974aa3ef85/lxml-6.1.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:2a0217714657e023ef4293500f65aa20fce6164c8fd6b08fa5bd4a859fb14b9b", size = 5620296, upload-time = "2026-05-18T19:19:29.993Z" },
{ url = "https://files.pythonhosted.org/packages/30/90/d61e383146f74c5ab683947ea14dc7b82778838ab9b95ea73a23b60d0191/lxml-6.1.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:05a82eb6e1530a64f26225b55cbd178113bd0b5af1c2b625f25e5296742c26d2", size = 5228598, upload-time = "2026-05-18T19:19:33.523Z" },
{ url = "https://files.pythonhosted.org/packages/76/2d/2dafd8149e94b05bb070690efd5bb2680720681e03ff03fc57d2b70a1105/lxml-6.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9e36f163528fc50cbef305f02a5fd66d404edf7049cdaff211dbc2cba5a7013e", size = 5247845, upload-time = "2026-05-18T19:19:36.649Z" },
{ url = "https://files.pythonhosted.org/packages/ce/68/b30e913340c380ddac9580c6e6230991fc37240ec4f64704833e4f3e2769/lxml-6.1.1-cp314-cp314t-win32.whl", hash = "sha256:649dda677cf3bd6ac9ae14007ba0c824ded8ce5808b53fc7431d9140399118c1", size = 3897345, upload-time = "2026-05-18T19:17:33.562Z" },
{ url = "https://files.pythonhosted.org/packages/3c/4e/9eb2af5335545f9fbcd7af57bcf87c6025d31eaa31b14ec184a6c8675328/lxml-6.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:793033d6c5cdf33a573f910d9bea14ef8f5771820411d118da8e1182edb53d5e", size = 4393350, upload-time = "2026-05-18T19:18:10.076Z" },
{ url = "https://files.pythonhosted.org/packages/7f/2c/0f1e93c636720e8a3eb59af2bfda99d98b55891e1c53bc30c2e0e865f01b/lxml-6.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:58bb955caba94e467d2a96da17660d2d704e0675894cba21ab8a775b8621fd1c", size = 3817223, upload-time = "2026-05-19T19:22:56.823Z" },
{ url = "https://files.pythonhosted.org/packages/b5/32/86a3f0f724a3a402d4627937a7fc27b160e45e7012b4adf47f6e1e844511/lxml-6.1.1-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:31033dc34636ea6b7d5cc11b1ddbda78a14de858ba9d3e1ed4b69a3085bc521e", size = 3930127, upload-time = "2026-05-18T19:19:02.27Z" },
{ url = "https://files.pythonhosted.org/packages/40/44/d832e82af08723761556d004b1d04d281c09f9a8cecd7d3148548c9941a3/lxml-6.1.1-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3893c14c4b6ac5b2d54ba8cf03e99fe5104e592de491f19bd6b82756c09f8004", size = 4210769, upload-time = "2026-05-18T19:20:41.427Z" },
{ url = "https://files.pythonhosted.org/packages/6d/39/0dc5949f759ed7d951e0bb8c2f2d9d7aca1908d22352fa84a8afd2ea54af/lxml-6.1.1-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c07da4cebf6889f03ebac8d238f62318e29f495de0aa18a51ea14e61ae907e2e", size = 4318163, upload-time = "2026-05-18T19:20:44.702Z" },
{ url = "https://files.pythonhosted.org/packages/e6/fb/8ab3845fe046ba4cbf74536bcf6801a774b7caf4350de1c5d37f1f0a9e90/lxml-6.1.1-pp311-pypy311_pp73-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f6f0ce10945fab9c4c06ce14e22af9059d1a87493a9af4501a5b0b9187e21cf2", size = 4250945, upload-time = "2026-05-18T19:20:47.385Z" },
{ url = "https://files.pythonhosted.org/packages/68/1b/7553ab136894374ffae8851ec06f98f511cd8e66246e41b6be059d0a7289/lxml-6.1.1-pp311-pypy311_pp73-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f8844cd288697c6425c9beba919302241e3278871dc6519515e72b04e987abcf", size = 4401664, upload-time = "2026-05-18T19:20:50.489Z" },
{ url = "https://files.pythonhosted.org/packages/db/a4/441aee36c6f6b249823d20fd91f9be9ab89d7c5a8ae542a4a4ca6d342d56/lxml-6.1.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:ed21202aec73cda4d55d1ce57b389aadb90ffb044e6cd1080b8347efe1b1ec84", size = 3508989, upload-time = "2026-05-18T19:18:38.158Z" },
]
[[package]]
name = "markdownify"
version = "1.2.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "beautifulsoup4" },
{ name = "six" },
]
sdist = { url = "https://files.pythonhosted.org/packages/3f/bc/c8c8eea5335341306b0fa7e1cb33c5e1c8d24ef70ddd684da65f41c49c92/markdownify-1.2.2.tar.gz", hash = "sha256:b274f1b5943180b031b699b199cbaeb1e2ac938b75851849a31fd0c3d6603d09", size = 18816, upload-time = "2025-11-16T19:21:18.565Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/43/ce/f1e3e9d959db134cedf06825fae8d5b294bd368aacdd0831a3975b7c4d55/markdownify-1.2.2-py3-none-any.whl", hash = "sha256:3f02d3cc52714084d6e589f70397b6fc9f2f3a8531481bf35e8cc39f975e186a", size = 15724, upload-time = "2025-11-16T19:21:17.622Z" },
]
[[package]]
name = "mypy"
version = "2.1.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "ast-serialize" },
{ name = "librt", marker = "platform_python_implementation != 'PyPy'" },
{ name = "mypy-extensions" },
{ name = "pathspec" },
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/82/15/cca9d88503549ed6fedeaa1d448cdddd542ee8a490232d732e278036fbf2/mypy-2.1.0.tar.gz", hash = "sha256:81e76ad12c2d804512e9b13240d1588316531bfba07558286078bfbce9613633", size = 3898359, upload-time = "2026-05-11T18:37:36.237Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/0a/a1/639f3024794a2a15899cb90707fe02e044c4412794c39c5769fd3df2e2ef/mypy-2.1.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:a683016b16fe2f572dc04c72be7ee0504ac1605a265d0200f5cea695fb788f41", size = 14691685, upload-time = "2026-05-11T18:33:27.973Z" },
{ url = "https://files.pythonhosted.org/packages/3b/08/9a585dea4325f20d8b80dc78623fa50d1fd2173b710f6237afd6ba6ab39b/mypy-2.1.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:1a293c534adb55271fef24a26da04b855540a8c13cc07bc5917b9fd2c394f2ca", size = 13555165, upload-time = "2026-05-11T18:32:16.107Z" },
{ url = "https://files.pythonhosted.org/packages/81/dc/7c42cc9c6cb01e8eb09961f1f738741d3e9c7e9d5c5b30ec69222625cd5f/mypy-2.1.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7406f4d048e71e576f5356d317e5b0a9e666dfd966bd99f9d14ca06e1a341538", size = 13994376, upload-time = "2026-05-11T18:32:39.256Z" },
{ url = "https://files.pythonhosted.org/packages/d4/fa/285946c33bce716e082c11dfeee9ee196eaf1f5042efb3581a31f9f205e4/mypy-2.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e0210d626fc8b31ccc90233754c7bc90e1f43205e85d96387f7db1285b55c398", size = 14864618, upload-time = "2026-05-11T18:34:49.765Z" },
{ url = "https://files.pythonhosted.org/packages/2b/83/82397f48af6c27e295d57979ded8490c9829040152cf7571b2f026aeb9a0/mypy-2.1.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:3712c20deed54e814eaaa825603bada8ea1c390670a397c95b98405347acc563", size = 15102063, upload-time = "2026-05-11T18:34:05.855Z" },
{ url = "https://files.pythonhosted.org/packages/40/68/b02dec39057b88eb03dc0aa854732e26e8361f34f9d0e20c7614967d1eba/mypy-2.1.0-cp311-cp311-win_amd64.whl", hash = "sha256:fcaa0e479066e31f7cceb6a3bea39cb22b2ff51a6b2f24f193d19179ba17c389", size = 11060564, upload-time = "2026-05-11T18:35:36.494Z" },
{ url = "https://files.pythonhosted.org/packages/cf/a8/ea3dcbef31f99b634f2ee23bb0321cbc8c1b388b76a861eb849f13c347dc/mypy-2.1.0-cp311-cp311-win_arm64.whl", hash = "sha256:0b1a5260c95aa443083f9ed3592662941951bca3d4ca224a5dc517c38b7cf666", size = 9966983, upload-time = "2026-05-11T18:37:14.139Z" },
{ url = "https://files.pythonhosted.org/packages/95/b1/55861beb5c339b44f9a2ba92df9e2cb1eeb4ae1eee674cdf7772c797778b/mypy-2.1.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:244358bf1c0da7722230bce60683d52e8e9fd030554926f15b747a84efb5b3af", size = 14874381, upload-time = "2026-05-11T18:37:31.784Z" },
{ url = "https://files.pythonhosted.org/packages/0b/b3/b7f770114b7d0ac92d0f76e8d93c2780844a70488a90e91821927850da86/mypy-2.1.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4ec7c57657493c7a75534df2751c8ae2cda383c16ecc55d2106c54476b1b16f6", size = 13665501, upload-time = "2026-05-11T18:34:23.063Z" },
{ url = "https://files.pythonhosted.org/packages/b6/f3/8ae2037967e2126689a0c11d99e2b707134a565191e92c60ca2572aec60a/mypy-2.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d8161b6ff4392410023224f0969d17db93e1e154bc3e4ba62598e720723ae211", size = 14045750, upload-time = "2026-05-11T18:31:48.151Z" },
{ url = "https://files.pythonhosted.org/packages/a0/32/615eb5911859e43d054941b0d0a7d06cfa2870eba86529cf385b052b111c/mypy-2.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bf03e12003084a67395184d3eb8cbd6a489dc3655b5664b28c210a9e2403ab0b", size = 15061630, upload-time = "2026-05-11T18:37:06.898Z" },
{ url = "https://files.pythonhosted.org/packages/d4/03/4eafbfff8bfab1b87082741eae6e6a624028c984e6708b73bce2a8570c9d/mypy-2.1.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:20509760fd791c51579d573153407d226385ec1f8bcce55d730b354f3336bc22", size = 15288831, upload-time = "2026-05-11T18:31:18.07Z" },
{ url = "https://files.pythonhosted.org/packages/99/ee/919661478e5891a3c96e549c036e467e64563ab85995b10c53c8358e16a3/mypy-2.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:6753d0c1fdd6b1a23b9e4f283ce80b2153b724adcb2653b20b85a8a28ac6436b", size = 11135228, upload-time = "2026-05-11T18:34:31.23Z" },
{ url = "https://files.pythonhosted.org/packages/24/0a/6a12b9782ca0831a553192f351679f4548abc9d19a7cc93bb7feb02084c7/mypy-2.1.0-cp312-cp312-win_arm64.whl", hash = "sha256:98ebb6589bb3b6d0c6f0c459d53ca55b8091fbc13d277c4041c885392e8195e8", size = 10040684, upload-time = "2026-05-11T18:36:48.199Z" },
{ url = "https://files.pythonhosted.org/packages/6e/dd/c7191469c777f07689c032a8f7326e393ea34c92d6d76eb7ce5ba57ea66d/mypy-2.1.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:35aac3bb114e03888f535d5eb51b8bafbb3266586b599da1940f9b1be3ec5bd5", size = 14852174, upload-time = "2026-05-11T18:31:38.929Z" },
{ url = "https://files.pythonhosted.org/packages/55/8c/aed55408879043d72bb9135f4d0d19a02b886dd569631e113e3d2706cb8d/mypy-2.1.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8de55a8c861f2a49331f807be98d90caeceeef520bde13d43a160207f8af613e", size = 13651542, upload-time = "2026-05-11T18:36:04.636Z" },
{ url = "https://files.pythonhosted.org/packages/3a/8e/f371a824b1f1fa8ea6e3dbb8703d232977d572be2329554a3bc4d960302f/mypy-2.1.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5fdf2941a07434af755837d9880f7d7d25f1dacb1af9dcd4b9b66f2220a3024e", size = 14033929, upload-time = "2026-05-11T18:35:55.742Z" },
{ url = "https://files.pythonhosted.org/packages/94/21/f54be870d6dd53a82c674407e0f8eed7174b05ec78d42e5abd7b42e84fd5/mypy-2.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e195b817c13f02352a9c124301f9f30f078405444679b6753c1b96b6eed37285", size = 15039200, upload-time = "2026-05-11T18:33:10.281Z" },
{ url = "https://files.pythonhosted.org/packages/17/99/bf21748626a40ce59fd29a39386ab46afec88b7bd2f0fa6c3a97c995523f/mypy-2.1.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:5431d42af987ebd92ba2f71d45c85ed41d8e6ca9f5fd209a69f68f707d2469e5", size = 15272690, upload-time = "2026-05-11T18:32:07.205Z" },
{ url = "https://files.pythonhosted.org/packages/d6/d7/9e90d2cf47100bea550ed2bc7b0d4de3a62181d84d5e37da0003e8462637/mypy-2.1.0-cp313-cp313-win_amd64.whl", hash = "sha256:767fe8c66dc3e01e19e1737d4c38ebefead16125e1b8e58ad421903b376f5c65", size = 11147435, upload-time = "2026-05-11T18:33:56.477Z" },
{ url = "https://files.pythonhosted.org/packages/ec/46/e5c449e858798e35ffc90946282a27c62a77be743fe17480e4977374eb91/mypy-2.1.0-cp313-cp313-win_arm64.whl", hash = "sha256:ecfe70d43775ab99562ab128ce49854a362044c9f894961f68f898c23cb7429d", size = 10035052, upload-time = "2026-05-11T18:32:30.049Z" },
{ url = "https://files.pythonhosted.org/packages/b0/ca/b279a672e874aedd5498ae25f722dacc8aa86bbffb939b3f97cbb1cf6686/mypy-2.1.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:7354c5a7f69d9345c3d6e69921d57088eea3ddeeb6b20d34c1b3855b02c36ec2", size = 14848422, upload-time = "2026-05-11T18:35:45.984Z" },
{ url = "https://files.pythonhosted.org/packages/27/e6/3efe56c631d959b9b4454e208b0ac4b7f4f58b404c89f8bec7b49efdfc21/mypy-2.1.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:49890d4f76ac9e06ec117f9e09f3174da70a620a0c300953d8595c926e80947f", size = 13677374, upload-time = "2026-05-11T18:36:57.188Z" },
{ url = "https://files.pythonhosted.org/packages/84/7f/8107ea87a44fd1f1b59882442f033c9c3488c127201b1d1d15f1cbd6022e/mypy-2.1.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:761be68e023ef5d94678772396a8af1220030f80837a3afd8d0aef3b419666f4", size = 14055743, upload-time = "2026-05-11T18:35:18.361Z" },
{ url = "https://files.pythonhosted.org/packages/51/4d/b6d34db183133b83761b9199a82d31557cdbb70a380d8c3b3438e11882a3/mypy-2.1.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c90345fc182dc363b891350457ec69c35140858538f38b4540845afcc32b1aef", size = 15020937, upload-time = "2026-05-11T18:34:59.618Z" },
{ url = "https://files.pythonhosted.org/packages/ff/d7/f08360c691d758acb02f45022c34d98b92892f4ea756644e1000d4b9f3d8/mypy-2.1.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:b84802e7b5a6daf1f5e15bc9fcd7ddae77be13981ffab037f1c67bb84d67d135", size = 15253371, upload-time = "2026-05-11T18:36:41.081Z" },
{ url = "https://files.pythonhosted.org/packages/67/1b/09460a13719530a19bce27bd3bc8449e83569dd2ba7faf51c9c3c30c0b61/mypy-2.1.0-cp314-cp314-win_amd64.whl", hash = "sha256:022c771234936ceac541ebaf836fe9e2abeb3f5e09aff21588fe543ff006fe21", size = 11326429, upload-time = "2026-05-11T18:34:13.526Z" },
{ url = "https://files.pythonhosted.org/packages/40/62/75dbf0f82f7b6680340efc614af29dd0b3c17b8a4f1cd09b8bd2fd6bc814/mypy-2.1.0-cp314-cp314-win_arm64.whl", hash = "sha256:498207db725cec88829a6a5c2fc771205fd043719ef98bc49aba8fb9fc4e6d57", size = 10218799, upload-time = "2026-05-11T18:32:23.491Z" },
{ url = "https://files.pythonhosted.org/packages/b2/66/caca04ed7d972fb6eb6dd1ccd6df1de5c38fae8c5b3dc1c4e8e0d85ee6b9/mypy-2.1.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7d5e5cad0efeba72b93cd17490cc0d69c5ac9ca132994fe3fb0314808aeeb83e", size = 15923458, upload-time = "2026-05-11T18:35:28.64Z" },
{ url = "https://files.pythonhosted.org/packages/ed/52/2d90cbe49d014b13ed7ff337930c30bad35893fe38a1e4641e756bb62191/mypy-2.1.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ff715050c127d724fd260a2e666e7747fdd83511c0c47d449d98238970aef780", size = 14757697, upload-time = "2026-05-11T18:36:14.208Z" },
{ url = "https://files.pythonhosted.org/packages/ac/37/d98f4a14e081b238992d0ed96b6d39c7cc0148c9699eb71eaa68629665ea/mypy-2.1.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:82208da9e09414d520e912d3e462d454854bed0810b71540bb016dcbca7308fd", size = 15405638, upload-time = "2026-05-11T18:33:48.249Z" },
{ url = "https://files.pythonhosted.org/packages/a3/c2/15c46613b24a84fad2aea1248bf9619b99c2767ae9071fe224c179a0b7d4/mypy-2.1.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e79ebc1b904b84f0310dff7469655a9c36c7a68bddb37bdd42b67a332df61d08", size = 16215852, upload-time = "2026-05-11T18:32:50.296Z" },
{ url = "https://files.pythonhosted.org/packages/5c/90/9c16a57f482c76d25f6379762b56bbf65c711d8158cf271fb2802cfb0640/mypy-2.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:e583edc957cfb0deb142079162ae826f58449b116c1d442f2d91c69d9fced081", size = 16452695, upload-time = "2026-05-11T18:33:38.182Z" },
{ url = "https://files.pythonhosted.org/packages/0f/4c/215a4eeb63cacc5f17f516691ea7285d11e249802b942476bff15922a314/mypy-2.1.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b33b6cd332695bba180d55e717a79d3038e479a2c49cc5eb3d53603409b9a5d7", size = 12866622, upload-time = "2026-05-11T18:34:39.945Z" },
{ url = "https://files.pythonhosted.org/packages/4b/50/1043e1db5f455ffe4c9ab22747cd8ca2bc492b1e4f4e21b130a44ee2b217/mypy-2.1.0-cp314-cp314t-win_arm64.whl", hash = "sha256:4f910fe825376a7b66ef7ca8c98e5a149e8cd64c19ae71d84047a74ee060d4e6", size = 10610798, upload-time = "2026-05-11T18:36:31.444Z" },
{ url = "https://files.pythonhosted.org/packages/0d/2a/13ca1f292f6db1b98ff495ef3467736b331621c5917cad984b7043e7348d/mypy-2.1.0-py3-none-any.whl", hash = "sha256:a663814603a5c563fb87a4f96fb473eeb30d1f5a4885afcf44f9db000a366289", size = 2693302, upload-time = "2026-05-11T18:31:29.246Z" },
]
[[package]]
name = "mypy-extensions"
version = "1.1.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/a2/6e/371856a3fb9d31ca8dac321cda606860fa4548858c0cc45d9d1d4ca2628b/mypy_extensions-1.1.0.tar.gz", hash = "sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558", size = 6343, upload-time = "2025-04-22T14:54:24.164Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" },
]
[[package]]
name = "packaging"
version = "26.2"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/d7/f1/e7a6dd94a8d4a5626c03e4e99c87f241ba9e350cd9e6d75123f992427270/packaging-26.2.tar.gz", hash = "sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661", size = 228134, upload-time = "2026-04-24T20:15:23.917Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" },
]
[[package]]
name = "pathspec"
version = "1.1.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/5a/82/42f767fc1c1143d6fd36efb827202a2d997a375e160a71eb2888a925aac1/pathspec-1.1.1.tar.gz", hash = "sha256:17db5ecd524104a120e173814c90367a96a98d07c45b2e10c2f3919fff91bf5a", size = 135180, upload-time = "2026-04-27T01:46:08.907Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl", hash = "sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189", size = 57328, upload-time = "2026-04-27T01:46:07.06Z" },
]
[[package]]
name = "platformdirs"
version = "4.9.6"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/9f/4a/0883b8e3802965322523f0b200ecf33d31f10991d0401162f4b23c698b42/platformdirs-4.9.6.tar.gz", hash = "sha256:3bfa75b0ad0db84096ae777218481852c0ebc6c727b3168c1b9e0118e458cf0a", size = 29400, upload-time = "2026-04-09T00:04:10.812Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/75/a6/a0a304dc33b49145b21f4808d763822111e67d1c3a32b524a1baf947b6e1/platformdirs-4.9.6-py3-none-any.whl", hash = "sha256:e61adb1d5e5cb3441b4b7710bea7e4c12250ca49439228cc1021c00dcfac0917", size = 21348, upload-time = "2026-04-09T00:04:09.463Z" },
]
[[package]]
name = "playwright"
version = "1.60.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "greenlet" },
{ name = "pyee" },
]
wheels = [
{ url = "https://files.pythonhosted.org/packages/21/f0/832bd9677194908da118064eef20082f2791e3d18215cc6d9391ee2c5a67/playwright-1.60.0-py3-none-macosx_10_13_x86_64.whl", hash = "sha256:6a8cd0fec171fb3089e95e898c8bc8a6f35dea0b78b399e12fcc19427e91b1d7", size = 43474635, upload-time = "2026-05-18T12:00:31.969Z" },
{ url = "https://files.pythonhosted.org/packages/59/7b/e1d32ae8a3ed937ec2be3721c5f728b13d731a0b7c6442e0b3bec5094ac0/playwright-1.60.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:39b5420ba6145045b69ced4c5c47d4d9fe5bddfc8ff816c518913afcb25ec7a5", size = 42261327, upload-time = "2026-05-18T12:00:35.638Z" },
{ url = "https://files.pythonhosted.org/packages/d7/bc/23de499ded6411c188a20c5a0dea6f0cd4ed5d2b3cc6042a5dbd3ed609aa/playwright-1.60.0-py3-none-macosx_11_0_universal2.whl", hash = "sha256:2581d0e6a3392c71f91b27460c7fd093356818dc430f48153896c8aeeaef7705", size = 43474636, upload-time = "2026-05-18T12:00:39.294Z" },
{ url = "https://files.pythonhosted.org/packages/22/7b/1d679f4fced4ea94efadd17103856d8c565384f68382a1681264e46f5925/playwright-1.60.0-py3-none-manylinux1_x86_64.whl", hash = "sha256:1c2bfae7884fb3fb05b853290eab8f343d524e5016f2f1def702acbbdf14c93e", size = 47467220, upload-time = "2026-05-18T12:00:43.179Z" },
{ url = "https://files.pythonhosted.org/packages/84/c2/1528d267d4442bd2c6b8eaeab819dd52c2030bf80e89293f0ba1f687473b/playwright-1.60.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:43e66564125ee31b07a58cefb21e256d62d67d8d1713e6858df7a3019d8ed353", size = 47154856, upload-time = "2026-05-18T12:00:46.715Z" },
{ url = "https://files.pythonhosted.org/packages/bb/4e/b008b6440a7a1624378041da94829956d4b8f7ab9ef5aad22d0dc3f2e26d/playwright-1.60.0-py3-none-win32.whl", hash = "sha256:ec94e416ea320711e0ad4bf185dcbf41833672961e90773e1885255d7db7b7e7", size = 37902157, upload-time = "2026-05-18T12:00:50.374Z" },
{ url = "https://files.pythonhosted.org/packages/55/f0/0541524133104f9cc20bf900870ff4a736b76a23483f3a55295ddfa58409/playwright-1.60.0-py3-none-win_amd64.whl", hash = "sha256:9566821ce6030a1f9e7146a24e19355ab0d98805fd0f9be50bb3d8fef1750c02", size = 37902159, upload-time = "2026-05-18T12:00:53.728Z" },
{ url = "https://files.pythonhosted.org/packages/80/c8/210f282d278e4709cdd71b12a31af45a30a22ab3207b387e29b37e478713/playwright-1.60.0-py3-none-win_arm64.whl", hash = "sha256:6e4f6700a4c2250efff8e690a81d66e3855754fb587b6b87cf5c784014f91537", size = 34037981, upload-time = "2026-05-18T12:00:57.584Z" },
]
[[package]]
name = "pluggy"
version = "1.6.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" },
]
[[package]]
name = "pyee"
version = "13.0.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/8b/04/e7c1fe4dc78a6fdbfd6c337b1c3732ff543b8a397683ab38378447baa331/pyee-13.0.1.tar.gz", hash = "sha256:0b931f7c14535667ed4c7e0d531716368715e860b988770fc7eb8578d1f67fc8", size = 31655, upload-time = "2026-02-14T21:12:28.044Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/a0/c4/b4d4827c93ef43c01f599ef31453ccc1c132b353284fc6c87d535c233129/pyee-13.0.1-py3-none-any.whl", hash = "sha256:af2f8fede4171ef667dfded53f96e2ed0d6e6bd7ee3bb46437f77e3b57689228", size = 15659, upload-time = "2026-02-14T21:12:26.263Z" },
]
[[package]]
name = "pygments"
version = "2.20.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" },
]
[[package]]
name = "pytest"
version = "9.0.3"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "colorama", marker = "sys_platform == 'win32'" },
{ name = "iniconfig" },
{ name = "packaging" },
{ name = "pluggy" },
{ name = "pygments" },
]
sdist = { url = "https://files.pythonhosted.org/packages/7d/0d/549bd94f1a0a402dc8cf64563a117c0f3765662e2e668477624baeec44d5/pytest-9.0.3.tar.gz", hash = "sha256:b86ada508af81d19edeb213c681b1d48246c1a91d304c6c81a427674c17eb91c", size = 1572165, upload-time = "2026-04-07T17:16:18.027Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/d4/24/a372aaf5c9b7208e7112038812994107bc65a84cd00e0354a88c2c77a617/pytest-9.0.3-py3-none-any.whl", hash = "sha256:2c5efc453d45394fdd706ade797c0a81091eccd1d6e4bccfcd476e2b8e0ab5d9", size = 375249, upload-time = "2026-04-07T17:16:16.13Z" },
]
[[package]]
name = "pytest-asyncio"
version = "1.3.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pytest" },
{ name = "typing-extensions", marker = "python_full_version < '3.13'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/90/2c/8af215c0f776415f3590cac4f9086ccefd6fd463befeae41cd4d3f193e5a/pytest_asyncio-1.3.0.tar.gz", hash = "sha256:d7f52f36d231b80ee124cd216ffb19369aa168fc10095013c6b014a34d3ee9e5", size = 50087, upload-time = "2025-11-10T16:07:47.256Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/e5/35/f8b19922b6a25bc0880171a2f1a003eaeb93657475193ab516fd87cac9da/pytest_asyncio-1.3.0-py3-none-any.whl", hash = "sha256:611e26147c7f77640e6d0a92a38ed17c3e9848063698d5c93d5aa7aa11cebff5", size = 15075, upload-time = "2025-11-10T16:07:45.537Z" },
]
[[package]]
name = "pytokens"
version = "0.4.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/b6/34/b4e015b99031667a7b960f888889c5bd34ef585c85e1cb56a594b92836ac/pytokens-0.4.1.tar.gz", hash = "sha256:292052fe80923aae2260c073f822ceba21f3872ced9a68bb7953b348e561179a", size = 23015, upload-time = "2026-01-30T01:03:45.924Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/3d/92/790ebe03f07b57e53b10884c329b9a1a308648fc083a6d4a39a10a28c8fc/pytokens-0.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:d70e77c55ae8380c91c0c18dea05951482e263982911fc7410b1ffd1dadd3440", size = 160864, upload-time = "2026-01-30T01:02:57.882Z" },
{ url = "https://files.pythonhosted.org/packages/13/25/a4f555281d975bfdd1eba731450e2fe3a95870274da73fb12c40aeae7625/pytokens-0.4.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4a58d057208cb9075c144950d789511220b07636dd2e4708d5645d24de666bdc", size = 248565, upload-time = "2026-01-30T01:02:59.912Z" },
{ url = "https://files.pythonhosted.org/packages/17/50/bc0394b4ad5b1601be22fa43652173d47e4c9efbf0044c62e9a59b747c56/pytokens-0.4.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b49750419d300e2b5a3813cf229d4e5a4c728dae470bcc89867a9ad6f25a722d", size = 260824, upload-time = "2026-01-30T01:03:01.471Z" },
{ url = "https://files.pythonhosted.org/packages/4e/54/3e04f9d92a4be4fc6c80016bc396b923d2a6933ae94b5f557c939c460ee0/pytokens-0.4.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:d9907d61f15bf7261d7e775bd5d7ee4d2930e04424bab1972591918497623a16", size = 264075, upload-time = "2026-01-30T01:03:04.143Z" },
{ url = "https://files.pythonhosted.org/packages/d1/1b/44b0326cb5470a4375f37988aea5d61b5cc52407143303015ebee94abfd6/pytokens-0.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:ee44d0f85b803321710f9239f335aafe16553b39106384cef8e6de40cb4ef2f6", size = 103323, upload-time = "2026-01-30T01:03:05.412Z" },
{ url = "https://files.pythonhosted.org/packages/41/5d/e44573011401fb82e9d51e97f1290ceb377800fb4eed650b96f4753b499c/pytokens-0.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:140709331e846b728475786df8aeb27d24f48cbcf7bcd449f8de75cae7a45083", size = 160663, upload-time = "2026-01-30T01:03:06.473Z" },
{ url = "https://files.pythonhosted.org/packages/f0/e6/5bbc3019f8e6f21d09c41f8b8654536117e5e211a85d89212d59cbdab381/pytokens-0.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6d6c4268598f762bc8e91f5dbf2ab2f61f7b95bdc07953b602db879b3c8c18e1", size = 255626, upload-time = "2026-01-30T01:03:08.177Z" },
{ url = "https://files.pythonhosted.org/packages/bf/3c/2d5297d82286f6f3d92770289fd439956b201c0a4fc7e72efb9b2293758e/pytokens-0.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:24afde1f53d95348b5a0eb19488661147285ca4dd7ed752bbc3e1c6242a304d1", size = 269779, upload-time = "2026-01-30T01:03:09.756Z" },
{ url = "https://files.pythonhosted.org/packages/20/01/7436e9ad693cebda0551203e0bf28f7669976c60ad07d6402098208476de/pytokens-0.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5ad948d085ed6c16413eb5fec6b3e02fa00dc29a2534f088d3302c47eb59adf9", size = 268076, upload-time = "2026-01-30T01:03:10.957Z" },
{ url = "https://files.pythonhosted.org/packages/2e/df/533c82a3c752ba13ae7ef238b7f8cdd272cf1475f03c63ac6cf3fcfb00b6/pytokens-0.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:3f901fe783e06e48e8cbdc82d631fca8f118333798193e026a50ce1b3757ea68", size = 103552, upload-time = "2026-01-30T01:03:12.066Z" },
{ url = "https://files.pythonhosted.org/packages/cb/dc/08b1a080372afda3cceb4f3c0a7ba2bde9d6a5241f1edb02a22a019ee147/pytokens-0.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8bdb9d0ce90cbf99c525e75a2fa415144fd570a1ba987380190e8b786bc6ef9b", size = 160720, upload-time = "2026-01-30T01:03:13.843Z" },
{ url = "https://files.pythonhosted.org/packages/64/0c/41ea22205da480837a700e395507e6a24425151dfb7ead73343d6e2d7ffe/pytokens-0.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5502408cab1cb18e128570f8d598981c68a50d0cbd7c61312a90507cd3a1276f", size = 254204, upload-time = "2026-01-30T01:03:14.886Z" },
{ url = "https://files.pythonhosted.org/packages/e0/d2/afe5c7f8607018beb99971489dbb846508f1b8f351fcefc225fcf4b2adc0/pytokens-0.4.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:29d1d8fb1030af4d231789959f21821ab6325e463f0503a61d204343c9b355d1", size = 268423, upload-time = "2026-01-30T01:03:15.936Z" },
{ url = "https://files.pythonhosted.org/packages/68/d4/00ffdbd370410c04e9591da9220a68dc1693ef7499173eb3e30d06e05ed1/pytokens-0.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:970b08dd6b86058b6dc07efe9e98414f5102974716232d10f32ff39701e841c4", size = 266859, upload-time = "2026-01-30T01:03:17.458Z" },
{ url = "https://files.pythonhosted.org/packages/a7/c9/c3161313b4ca0c601eeefabd3d3b576edaa9afdefd32da97210700e47652/pytokens-0.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:9bd7d7f544d362576be74f9d5901a22f317efc20046efe2034dced238cbbfe78", size = 103520, upload-time = "2026-01-30T01:03:18.652Z" },
{ url = "https://files.pythonhosted.org/packages/8f/a7/b470f672e6fc5fee0a01d9e75005a0e617e162381974213a945fcd274843/pytokens-0.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:4a14d5f5fc78ce85e426aa159489e2d5961acf0e47575e08f35584009178e321", size = 160821, upload-time = "2026-01-30T01:03:19.684Z" },
{ url = "https://files.pythonhosted.org/packages/80/98/e83a36fe8d170c911f864bfded690d2542bfcfacb9c649d11a9e6eb9dc41/pytokens-0.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:97f50fd18543be72da51dd505e2ed20d2228c74e0464e4262e4899797803d7fa", size = 254263, upload-time = "2026-01-30T01:03:20.834Z" },
{ url = "https://files.pythonhosted.org/packages/0f/95/70d7041273890f9f97a24234c00b746e8da86df462620194cef1d411ddeb/pytokens-0.4.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dc74c035f9bfca0255c1af77ddd2d6ae8419012805453e4b0e7513e17904545d", size = 268071, upload-time = "2026-01-30T01:03:21.888Z" },
{ url = "https://files.pythonhosted.org/packages/da/79/76e6d09ae19c99404656d7db9c35dfd20f2086f3eb6ecb496b5b31163bad/pytokens-0.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f66a6bbe741bd431f6d741e617e0f39ec7257ca1f89089593479347cc4d13324", size = 271716, upload-time = "2026-01-30T01:03:23.633Z" },
{ url = "https://files.pythonhosted.org/packages/79/37/482e55fa1602e0a7ff012661d8c946bafdc05e480ea5a32f4f7e336d4aa9/pytokens-0.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:b35d7e5ad269804f6697727702da3c517bb8a5228afa450ab0fa787732055fc9", size = 104539, upload-time = "2026-01-30T01:03:24.788Z" },
{ url = "https://files.pythonhosted.org/packages/30/e8/20e7db907c23f3d63b0be3b8a4fd1927f6da2395f5bcc7f72242bb963dfe/pytokens-0.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:8fcb9ba3709ff77e77f1c7022ff11d13553f3c30299a9fe246a166903e9091eb", size = 168474, upload-time = "2026-01-30T01:03:26.428Z" },
{ url = "https://files.pythonhosted.org/packages/d6/81/88a95ee9fafdd8f5f3452107748fd04c24930d500b9aba9738f3ade642cc/pytokens-0.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:79fc6b8699564e1f9b521582c35435f1bd32dd06822322ec44afdeba666d8cb3", size = 290473, upload-time = "2026-01-30T01:03:27.415Z" },
{ url = "https://files.pythonhosted.org/packages/cf/35/3aa899645e29b6375b4aed9f8d21df219e7c958c4c186b465e42ee0a06bf/pytokens-0.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d31b97b3de0f61571a124a00ffe9a81fb9939146c122c11060725bd5aea79975", size = 303485, upload-time = "2026-01-30T01:03:28.558Z" },
{ url = "https://files.pythonhosted.org/packages/52/a0/07907b6ff512674d9b201859f7d212298c44933633c946703a20c25e9d81/pytokens-0.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:967cf6e3fd4adf7de8fc73cd3043754ae79c36475c1c11d514fc72cf5490094a", size = 306698, upload-time = "2026-01-30T01:03:29.653Z" },
{ url = "https://files.pythonhosted.org/packages/39/2a/cbbf9250020a4a8dd53ba83a46c097b69e5eb49dd14e708f496f548c6612/pytokens-0.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:584c80c24b078eec1e227079d56dc22ff755e0ba8654d8383b2c549107528918", size = 116287, upload-time = "2026-01-30T01:03:30.912Z" },
{ url = "https://files.pythonhosted.org/packages/c6/78/397db326746f0a342855b81216ae1f0a32965deccfd7c830a2dbc66d2483/pytokens-0.4.1-py3-none-any.whl", hash = "sha256:26cef14744a8385f35d0e095dc8b3a7583f6c953c2e3d269c7f82484bf5ad2de", size = 13729, upload-time = "2026-01-30T01:03:45.029Z" },
]
[[package]]
name = "pyyaml"
version = "6.0.3"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/6d/16/a95b6757765b7b031c9374925bb718d55e0a9ba8a1b6a12d25962ea44347/pyyaml-6.0.3-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e", size = 185826, upload-time = "2025-09-25T21:31:58.655Z" },
{ url = "https://files.pythonhosted.org/packages/16/19/13de8e4377ed53079ee996e1ab0a9c33ec2faf808a4647b7b4c0d46dd239/pyyaml-6.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824", size = 175577, upload-time = "2025-09-25T21:32:00.088Z" },
{ url = "https://files.pythonhosted.org/packages/0c/62/d2eb46264d4b157dae1275b573017abec435397aa59cbcdab6fc978a8af4/pyyaml-6.0.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c", size = 775556, upload-time = "2025-09-25T21:32:01.31Z" },
{ url = "https://files.pythonhosted.org/packages/10/cb/16c3f2cf3266edd25aaa00d6c4350381c8b012ed6f5276675b9eba8d9ff4/pyyaml-6.0.3-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00", size = 882114, upload-time = "2025-09-25T21:32:03.376Z" },
{ url = "https://files.pythonhosted.org/packages/71/60/917329f640924b18ff085ab889a11c763e0b573da888e8404ff486657602/pyyaml-6.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d", size = 806638, upload-time = "2025-09-25T21:32:04.553Z" },
{ url = "https://files.pythonhosted.org/packages/dd/6f/529b0f316a9fd167281a6c3826b5583e6192dba792dd55e3203d3f8e655a/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a", size = 767463, upload-time = "2025-09-25T21:32:06.152Z" },
{ url = "https://files.pythonhosted.org/packages/f2/6a/b627b4e0c1dd03718543519ffb2f1deea4a1e6d42fbab8021936a4d22589/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4", size = 794986, upload-time = "2025-09-25T21:32:07.367Z" },
{ url = "https://files.pythonhosted.org/packages/45/91/47a6e1c42d9ee337c4839208f30d9f09caa9f720ec7582917b264defc875/pyyaml-6.0.3-cp311-cp311-win32.whl", hash = "sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b", size = 142543, upload-time = "2025-09-25T21:32:08.95Z" },
{ url = "https://files.pythonhosted.org/packages/da/e3/ea007450a105ae919a72393cb06f122f288ef60bba2dc64b26e2646fa315/pyyaml-6.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf", size = 158763, upload-time = "2025-09-25T21:32:09.96Z" },
{ url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063, upload-time = "2025-09-25T21:32:11.445Z" },
{ url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973, upload-time = "2025-09-25T21:32:12.492Z" },
{ url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116, upload-time = "2025-09-25T21:32:13.652Z" },
{ url = "https://files.pythonhosted.org/packages/65/30/d7353c338e12baef4ecc1b09e877c1970bd3382789c159b4f89d6a70dc09/pyyaml-6.0.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c", size = 844011, upload-time = "2025-09-25T21:32:15.21Z" },
{ url = "https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc", size = 807870, upload-time = "2025-09-25T21:32:16.431Z" },
{ url = "https://files.pythonhosted.org/packages/05/c0/b3be26a015601b822b97d9149ff8cb5ead58c66f981e04fedf4e762f4bd4/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e", size = 761089, upload-time = "2025-09-25T21:32:17.56Z" },
{ url = "https://files.pythonhosted.org/packages/be/8e/98435a21d1d4b46590d5459a22d88128103f8da4c2d4cb8f14f2a96504e1/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea", size = 790181, upload-time = "2025-09-25T21:32:18.834Z" },
{ url = "https://files.pythonhosted.org/packages/74/93/7baea19427dcfbe1e5a372d81473250b379f04b1bd3c4c5ff825e2327202/pyyaml-6.0.3-cp312-cp312-win32.whl", hash = "sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5", size = 137658, upload-time = "2025-09-25T21:32:20.209Z" },
{ url = "https://files.pythonhosted.org/packages/86/bf/899e81e4cce32febab4fb42bb97dcdf66bc135272882d1987881a4b519e9/pyyaml-6.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b", size = 154003, upload-time = "2025-09-25T21:32:21.167Z" },
{ url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344, upload-time = "2025-09-25T21:32:22.617Z" },
{ url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" },
{ url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" },
{ url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" },
{ url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" },
{ url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" },
{ url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" },
{ url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" },
{ url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" },
{ url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" },
{ url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" },
{ url = "https://files.pythonhosted.org/packages/9d/8c/f4bd7f6465179953d3ac9bc44ac1a8a3e6122cf8ada906b4f96c60172d43/pyyaml-6.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac", size = 181814, upload-time = "2025-09-25T21:32:35.712Z" },
{ url = "https://files.pythonhosted.org/packages/bd/9c/4d95bb87eb2063d20db7b60faa3840c1b18025517ae857371c4dd55a6b3a/pyyaml-6.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310", size = 173809, upload-time = "2025-09-25T21:32:36.789Z" },
{ url = "https://files.pythonhosted.org/packages/92/b5/47e807c2623074914e29dabd16cbbdd4bf5e9b2db9f8090fa64411fc5382/pyyaml-6.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7", size = 766454, upload-time = "2025-09-25T21:32:37.966Z" },
{ url = "https://files.pythonhosted.org/packages/02/9e/e5e9b168be58564121efb3de6859c452fccde0ab093d8438905899a3a483/pyyaml-6.0.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788", size = 836355, upload-time = "2025-09-25T21:32:39.178Z" },
{ url = "https://files.pythonhosted.org/packages/88/f9/16491d7ed2a919954993e48aa941b200f38040928474c9e85ea9e64222c3/pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5", size = 794175, upload-time = "2025-09-25T21:32:40.865Z" },
{ url = "https://files.pythonhosted.org/packages/dd/3f/5989debef34dc6397317802b527dbbafb2b4760878a53d4166579111411e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764", size = 755228, upload-time = "2025-09-25T21:32:42.084Z" },
{ url = "https://files.pythonhosted.org/packages/d7/ce/af88a49043cd2e265be63d083fc75b27b6ed062f5f9fd6cdc223ad62f03e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35", size = 789194, upload-time = "2025-09-25T21:32:43.362Z" },
{ url = "https://files.pythonhosted.org/packages/23/20/bb6982b26a40bb43951265ba29d4c246ef0ff59c9fdcdf0ed04e0687de4d/pyyaml-6.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac", size = 156429, upload-time = "2025-09-25T21:32:57.844Z" },
{ url = "https://files.pythonhosted.org/packages/f4/f4/a4541072bb9422c8a883ab55255f918fa378ecf083f5b85e87fc2b4eda1b/pyyaml-6.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3", size = 143912, upload-time = "2025-09-25T21:32:59.247Z" },
{ url = "https://files.pythonhosted.org/packages/7c/f9/07dd09ae774e4616edf6cda684ee78f97777bdd15847253637a6f052a62f/pyyaml-6.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3", size = 189108, upload-time = "2025-09-25T21:32:44.377Z" },
{ url = "https://files.pythonhosted.org/packages/4e/78/8d08c9fb7ce09ad8c38ad533c1191cf27f7ae1effe5bb9400a46d9437fcf/pyyaml-6.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba", size = 183641, upload-time = "2025-09-25T21:32:45.407Z" },
{ url = "https://files.pythonhosted.org/packages/7b/5b/3babb19104a46945cf816d047db2788bcaf8c94527a805610b0289a01c6b/pyyaml-6.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c", size = 831901, upload-time = "2025-09-25T21:32:48.83Z" },
{ url = "https://files.pythonhosted.org/packages/8b/cc/dff0684d8dc44da4d22a13f35f073d558c268780ce3c6ba1b87055bb0b87/pyyaml-6.0.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702", size = 861132, upload-time = "2025-09-25T21:32:50.149Z" },
{ url = "https://files.pythonhosted.org/packages/b1/5e/f77dc6b9036943e285ba76b49e118d9ea929885becb0a29ba8a7c75e29fe/pyyaml-6.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c", size = 839261, upload-time = "2025-09-25T21:32:51.808Z" },
{ url = "https://files.pythonhosted.org/packages/ce/88/a9db1376aa2a228197c58b37302f284b5617f56a5d959fd1763fb1675ce6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065", size = 805272, upload-time = "2025-09-25T21:32:52.941Z" },
{ url = "https://files.pythonhosted.org/packages/da/92/1446574745d74df0c92e6aa4a7b0b3130706a4142b2d1a5869f2eaa423c6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65", size = 829923, upload-time = "2025-09-25T21:32:54.537Z" },
{ url = "https://files.pythonhosted.org/packages/f0/7a/1c7270340330e575b92f397352af856a8c06f230aa3e76f86b39d01b416a/pyyaml-6.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9", size = 174062, upload-time = "2025-09-25T21:32:55.767Z" },
{ url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" },
]
[[package]]
name = "ruff"
version = "0.15.13"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/24/21/a7d5c126d5b557715ef81098f3db2fe20f622a039ff2e626af28d674ab80/ruff-0.15.13.tar.gz", hash = "sha256:f9d89f17f7ba7fb2ed42921f0df75da797a9a5d71bc39049e2c687cf2baf44b7", size = 4678180, upload-time = "2026-05-14T13:44:37.869Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c6/61/11d458dc6ac22504fd8e237b29dfd40504c7fbbcc8930402cfe51a8e63ed/ruff-0.15.13-py3-none-linux_armv6l.whl", hash = "sha256:444b580fc72fd6887e650acd3e575e18cdc79dbcf42fb4030b491057921f61f8", size = 10738279, upload-time = "2026-05-14T13:44:18.7Z" },
{ url = "https://files.pythonhosted.org/packages/86/ca/caa871ee7be718c45256fada4e16a218ee3e33f0c4a46b729a60a24912e6/ruff-0.15.13-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:6590d009e7cb7ebf36f83dbdd44a3fa48a0994ff6f1cdc1b08006abe58f98dc7", size = 11124798, upload-time = "2026-05-14T13:44:06.427Z" },
{ url = "https://files.pythonhosted.org/packages/d3/19/43f5f2e568dddde567fc41f8471f9432c09563e19d3e617a48cfa52f8f0a/ruff-0.15.13-py3-none-macosx_11_0_arm64.whl", hash = "sha256:1c26d2f66163deeb6e08d8b39fbbe983ce3c71cea06a6d7591cfd1421793c629", size = 10460761, upload-time = "2026-05-14T13:44:04.375Z" },
{ url = "https://files.pythonhosted.org/packages/99/df/cf938cd6de3003178f03ad7c1ea2a6c099468c03a35037985070b37e76be/ruff-0.15.13-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9dbd6f94b434f896308e4d57fb7bfde0d02b99f7a64b3bdab0fdfa6a864203a5", size = 10804451, upload-time = "2026-05-14T13:44:25.221Z" },
{ url = "https://files.pythonhosted.org/packages/c7/7d/5d0973129b154ded2225729169d7068f26b467760b146493fde138415f23/ruff-0.15.13-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:bf3259f3be4d181bda591da5db2571aed6853c6a048157756448020bc6c5cd22", size = 10534285, upload-time = "2026-05-14T13:44:08.888Z" },
{ url = "https://files.pythonhosted.org/packages/1f/e3/6b999bbc66cd51e5f073842bc2a3995e99c5e0e72e16b15e7261f7abf57a/ruff-0.15.13-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ae9c17e5eb4430c154e76abc25d79a318190f5a997f38fb6b114416c5319ffc9", size = 11312063, upload-time = "2026-05-14T13:44:11.274Z" },
{ url = "https://files.pythonhosted.org/packages/af/5a/642639e9f5db04f1e97fbd6e091c6fd20725bdf072fb114d00eefb9e6eb8/ruff-0.15.13-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:2e2e39bff6c341f4b577a21b801326fab0b11847f48fcaa83f00a113c9b3cb55", size = 12183079, upload-time = "2026-05-14T13:44:01.634Z" },
{ url = "https://files.pythonhosted.org/packages/19/4c/7585735f6b53b0f12de13618b2f7d250a844f018822efc899df2e7b8295f/ruff-0.15.13-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e8d9a8e08013542e94d3220bc5b62cc3e5ef87c5f74bff367d3fac14fab013e6", size = 11440833, upload-time = "2026-05-14T13:43:59.043Z" },
{ url = "https://files.pythonhosted.org/packages/e8/31/bf1a0803d077e679cfeee5f2f67290a0fa79c7385b5d9a8c17b9db2c48f0/ruff-0.15.13-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:cc411dfebe5eebe55ce041c6ae080eb7668955e866daa2fbb16692a784f1c4ca", size = 11434486, upload-time = "2026-05-14T13:44:27.761Z" },
{ url = "https://files.pythonhosted.org/packages/e1/4e/62c9b999875d4f14db80f277c030578f5e249c9852d65b7ac7ad0b43c041/ruff-0.15.13-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:768494eb08b9cee54e2fd27969966f74db5a57f6eaa7a90fcb3306af34dfc4bd", size = 11385189, upload-time = "2026-05-14T13:44:13.704Z" },
{ url = "https://files.pythonhosted.org/packages/fc/89/7e959047a104df3eb12863447c110140191fc5b6c4f379ea2e803fcdb0e4/ruff-0.15.13-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:fb75f9a3a7e42ffe117d734494e6c5e5cb3565d66e12612cb63d0e572a41a5b6", size = 10781380, upload-time = "2026-05-14T13:43:56.734Z" },
{ url = "https://files.pythonhosted.org/packages/ff/52/5fd18f3b88cab63e88aa11516b3b4e1e5f720e5c330f8dbe5c26210f41f8/ruff-0.15.13-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:8cb74dd33bb2f6613faf7fc03b660053b5ac4f80e706d5788c6335e2a8048d51", size = 10540605, upload-time = "2026-05-14T13:44:20.748Z" },
{ url = "https://files.pythonhosted.org/packages/e8/e0/9e35f338990d3e41a82875ff7053ffe97541dae81c9d02143177f381d572/ruff-0.15.13-py3-none-musllinux_1_2_i686.whl", hash = "sha256:7ef823f817fcd191dc934e984be9cf4094f808effa16f2542ad8e821ba02bbf2", size = 11036554, upload-time = "2026-05-14T13:44:16.256Z" },
{ url = "https://files.pythonhosted.org/packages/c2/13/070fb048c24080fba188f66371e2a92785be257ad02242066dc7255ac6e9/ruff-0.15.13-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:f345a13937bd7f09f6f5d19fa0721b0c103e00e7f62bc67089a8e5e037719e0b", size = 11528133, upload-time = "2026-05-14T13:44:22.808Z" },
{ url = "https://files.pythonhosted.org/packages/6b/8c/b1e1666aef7fc6555094d73ae6cd981701781ae85b97ceefc0eebd0b4668/ruff-0.15.13-py3-none-win32.whl", hash = "sha256:4044f94208b3b05ba0fc4a4abd0558cf4d6459bd18325eead7fd8cc66f909b41", size = 10721455, upload-time = "2026-05-14T13:44:35.697Z" },
{ url = "https://files.pythonhosted.org/packages/ab/a6/870a3e8a50590bb92be184ad928c2922f088b00d9dc5c5ec7b924ee08c22/ruff-0.15.13-py3-none-win_amd64.whl", hash = "sha256:7064884d442b7d477b4e7473d12da7f08851d2b1982763c5d3f388a19468a1a4", size = 11900409, upload-time = "2026-05-14T13:44:30.389Z" },
{ url = "https://files.pythonhosted.org/packages/9b/36/9c015cd052fca743dae8cb2aeb16b551444787467db42ceab0fc968865af/ruff-0.15.13-py3-none-win_arm64.whl", hash = "sha256:2471da9bd1068c8c064b5fd9c0c4b6dddffd6369cb1cd68b29993b1709ff1b21", size = 11179336, upload-time = "2026-05-14T13:44:33.026Z" },
]
[[package]]
name = "six"
version = "1.17.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" },
]
[[package]]
name = "soupsieve"
version = "2.8.3"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/7b/ae/2d9c981590ed9999a0d91755b47fc74f74de286b0f5cee14c9269041e6c4/soupsieve-2.8.3.tar.gz", hash = "sha256:3267f1eeea4251fb42728b6dfb746edc9acaffc4a45b27e19450b676586e8349", size = 118627, upload-time = "2026-01-20T04:27:02.457Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/46/2c/1462b1d0a634697ae9e55b3cecdcb64788e8b7d63f54d923fcd0bb140aed/soupsieve-2.8.3-py3-none-any.whl", hash = "sha256:ed64f2ba4eebeab06cc4962affce381647455978ffc1e36bb79a545b91f45a95", size = 37016, upload-time = "2026-01-20T04:27:01.012Z" },
]
[[package]]
name = "tenacity"
version = "9.1.4"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/47/c6/ee486fd809e357697ee8a44d3d69222b344920433d3b6666ccd9b374630c/tenacity-9.1.4.tar.gz", hash = "sha256:adb31d4c263f2bd041081ab33b498309a57c77f9acf2db65aadf0898179cf93a", size = 49413, upload-time = "2026-02-07T10:45:33.841Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/d7/c1/eb8f9debc45d3b7918a32ab756658a0904732f75e555402972246b0b8e71/tenacity-9.1.4-py3-none-any.whl", hash = "sha256:6095a360c919085f28c6527de529e76a06ad89b23659fa881ae0649b867a9d55", size = 28926, upload-time = "2026-02-07T10:45:32.24Z" },
]
[[package]]
name = "typing-extensions"
version = "4.15.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" },
]
[[package]]
name = "win32-setctime"
version = "1.2.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/b3/8f/705086c9d734d3b663af0e9bb3d4de6578d08f46b1b101c2442fd9aecaa2/win32_setctime-1.2.0.tar.gz", hash = "sha256:ae1fdf948f5640aae05c511ade119313fb6a30d7eabe25fef9764dca5873c4c0", size = 4867, upload-time = "2024-12-07T15:28:28.314Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/e1/07/c6fe3ad3e685340704d314d765b7912993bcb8dc198f0e7a89382d37974b/win32_setctime-1.2.0-py3-none-any.whl", hash = "sha256:95d644c4e708aba81dc3704a116d8cbc974d70b3bdb8be1d150e36be6e9d1390", size = 4083, upload-time = "2024-12-07T15:28:26.465Z" },
]