bellingcat/vk-url-scraper
1
0
Fork 0
mirror of https://github.com/bellingcat/vk-url-scraper.git synced 2026-06-07 19:08:38 +03:00
Code Issues Packages Projects Releases Wiki Activity

ported vk scraper logic into lib

Browse Source
This commit is contained in:
Miguel Ramalho
2022-06-17 19:15:20 +02:00
parent b3c7ac8e5d
commit 965057619f
32 changed files with 1346 additions and 610 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.github/ISSUE_TEMPLATE/bug_report.yml vendored
View File

@@ -6,7 +6,7 @@ body:
- type: markdown - type: markdown
attributes: attributes:
value: > value: >
#### Before submitting a bug, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/allenai/python-package-template/issues?q=is%3Aissue+sort%3Acreated-desc+). #### Before submitting a bug, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/bellingcat/vk-url-scraper/issues?q=is%3Aissue+sort%3Acreated-desc+).
- type: textarea - type: textarea
attributes: attributes:
label: 🐛 Describe the bug label: 🐛 Describe the bug
@@ -17,7 +17,7 @@ body:
```python ```python
# All necessary imports at the beginning # All necessary imports at the beginning
import my_package import vk_url_scraper
# A succinct reproducing example trimmed down to the essential parts: # A succinct reproducing example trimmed down to the essential parts:
assert False is True, "Oh no!" assert False is True, "Oh no!"

4
.github/ISSUE_TEMPLATE/documentation.yml vendored
View File

@@ -1,5 +1,5 @@
name: 📚 Documentation name: 📚 Documentation
description: Report an issue related to https://my-package.readthedocs.io/latest description: Report an issue related to https://vk-url-scraper.readthedocs.io/latest
labels: 'documentation' labels: 'documentation'
body: body:
@@ -7,7 +7,7 @@ body:
attributes: attributes:
label: 📚 The doc issue label: 📚 The doc issue
description: > description: >
A clear and concise description of what content in https://my-package.readthedocs.io/latest is an issue. A clear and concise description of what content in https://vk-url-scraper.readthedocs.io/latest is an issue.
validations: validations:
required: true required: true
- type: textarea - type: textarea

4
.github/pull_request_template.md vendored
View File

@@ -10,9 +10,9 @@ Changes proposed in this pull request:
## Before submitting ## Before submitting
<!-- Please complete this checklist BEFORE submitting your PR to speed along the review process. --> <!-- Please complete this checklist BEFORE submitting your PR to speed along the review process. -->
- [ ] I've read and followed all steps in the [Making a pull request](https://github.com/allenai/beaker-py/blob/main/CONTRIBUTING.md#making-a-pull-request) - [ ] I've read and followed all steps in the [Making a pull request](https://github.com/bellingcat/vk-url-scraper/blob/main/CONTRIBUTING.md#making-a-pull-request)
section of the `CONTRIBUTING` docs. section of the `CONTRIBUTING` docs.
- [ ] I've updated or added any relevant docstrings following the syntax described in the - [ ] I've updated or added any relevant docstrings following the syntax described in the
[Writing docstrings](https://github.com/allenai/beaker-py/blob/main/CONTRIBUTING.md#writing-docstrings) section of the `CONTRIBUTING` docs. [Writing docstrings](https://github.com/bellingcat/vk-url-scraper/blob/main/CONTRIBUTING.md#writing-docstrings) section of the `CONTRIBUTING` docs.
- [ ] If this PR fixes a bug, I've added a test that will fail without my fix. - [ ] If this PR fixes a bug, I've added a test that will fail without my fix.
- [ ] If this PR adds a new feature, I've added tests that sufficiently cover my new functionality. - [ ] If this PR adds a new feature, I've added tests that sufficiently cover my new functionality.

2
.github/workflows/main.yml vendored
View File

@@ -88,7 +88,7 @@ jobs:
if: always() if: always()
run: | run: |
. .venv/bin/activate . .venv/bin/activate
pip uninstall -y my-package pip uninstall -y vk-url-scraper
release: release:
name: Release name: Release

2
.github/workflows/pr_checks.yml vendored
View File

@@ -9,7 +9,7 @@ on:
branches: branches:
- main - main
paths: paths:
- 'my_package/**' - 'vk_url_scraper/**'
jobs: jobs:
changelog: changelog:

51
.github/workflows/setup.yml vendored
View File

@@ -1,51 +0,0 @@
name: Setup
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
on:
pull_request:
branches:
- main
push:
branches:
- main
jobs:
test_personalize:
name: Personalize
runs-on: [ubuntu-latest]
timeout-minutes: 10
steps:
- uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: '3.7'
- name: Install prerequisites
run: |
pip install -r setup-requirements.txt
- name: Run personalize script
run: |
python scripts/personalize.py --github-org epwalsh --github-repo new-repo --package-name new-package --yes
- name: Verify changes
shell: bash
run: |
set -eo pipefail
# Check that 'new-package' replaced 'my-package' in some files.
grep -q 'new-package' setup.py .github/workflows/main.yml CONTRIBUTING.md
# Check that the new repo URL replaced the old one in some files.
grep -q 'https://github.com/epwalsh/new-repo' setup.py CONTRIBUTING.md
# Double check that there are no lingering mentions of old names.
for pattern in 'my[-_]package' 'https://github.com/allenai/python-package-template'; do
if find . -type f -not -path './.git/*' | xargs grep "$pattern"; then
echo "Found ${pattern} where it shouldn't be!"
exit 1
fi
done
echo "All good!"

2
.gitignore vendored
View File

@@ -1,3 +1,5 @@
.env
vk_config.v2.json
# build artifacts # build artifacts
.eggs/ .eggs/

48
CONTRIBUTING.md
View File

@@ -6,10 +6,10 @@ Thanks for considering contributing! Please read this document to learn the vari
### Did you find a bug? ### Did you find a bug?
First, do [a quick search](https://github.com/allenai/python-package-template/issues) to see whether your issue has already been reported. First, do [a quick search](https://github.com/bellingcat/vk-url-scraper/issues) to see whether your issue has already been reported.
If your issue has already been reported, please comment on the existing issue. If your issue has already been reported, please comment on the existing issue.
Otherwise, open [a new GitHub issue](https://github.com/allenai/python-package-template/issues). Be sure to include a clear title Otherwise, open [a new GitHub issue](https://github.com/bellingcat/vk-url-scraper/issues). Be sure to include a clear title
and description. The description should include as much relevant information as possible. The description should and description. The description should include as much relevant information as possible. The description should
explain how to reproduce the erroneous behavior as well as the behavior you expect to see. Ideally you would include a explain how to reproduce the erroneous behavior as well as the behavior you expect to see. Ideally you would include a
code sample or an executable test case demonstrating the expected behavior. code sample or an executable test case demonstrating the expected behavior.
@@ -21,7 +21,7 @@ We use GitHub issues to track feature requests. Before you create a feature requ
* Make sure you have a clear idea of the enhancement you would like. If you have a vague idea, consider discussing * Make sure you have a clear idea of the enhancement you would like. If you have a vague idea, consider discussing
it first on a GitHub issue. it first on a GitHub issue.
* Check the documentation to make sure your feature does not already exist. * Check the documentation to make sure your feature does not already exist.
* Do [a quick search](https://github.com/allenai/python-package-template/issues) to see whether your feature has already been suggested. * Do [a quick search](https://github.com/bellingcat/vk-url-scraper/issues) to see whether your feature has already been suggested.
When creating your request, please: When creating your request, please:
@@ -41,31 +41,31 @@ When you're ready to contribute code to address an open issue, please follow the
Then clone your fork locally with Then clone your fork locally with
git clone https://github.com/USERNAME/python-package-template.git git clone https://github.com/USERNAME/vk-url-scraper.git
or or
git clone git@github.com:USERNAME/python-package-template.git git clone git@github.com:USERNAME/vk-url-scraper.git
At this point the local clone of your fork only knows that it came from *your* repo, github.com/USERNAME/python-package-template.git, but doesn't know anything the *main* repo, [https://github.com/allenai/python-package-template.git](https://github.com/allenai/python-package-template). You can see this by running At this point the local clone of your fork only knows that it came from *your* repo, github.com/USERNAME/vk-url-scraper.git, but doesn't know anything the *main* repo, [https://github.com/bellingcat/vk-url-scraper.git](https://github.com/bellingcat/vk-url-scraper). You can see this by running
git remote -v git remote -v
which will output something like this: which will output something like this:
origin https://github.com/USERNAME/python-package-template.git (fetch) origin https://github.com/USERNAME/vk-url-scraper.git (fetch)
origin https://github.com/USERNAME/python-package-template.git (push) origin https://github.com/USERNAME/vk-url-scraper.git (push)
This means that your local clone can only track changes from your fork, but not from the main repo, and so you won't be able to keep your fork up-to-date with the main repo over time. Therefore you'll need to add another "remote" to your clone that points to [https://github.com/allenai/python-package-template.git](https://github.com/allenai/python-package-template). To do this, run the following: This means that your local clone can only track changes from your fork, but not from the main repo, and so you won't be able to keep your fork up-to-date with the main repo over time. Therefore you'll need to add another "remote" to your clone that points to [https://github.com/bellingcat/vk-url-scraper.git](https://github.com/bellingcat/vk-url-scraper). To do this, run the following:
git remote add upstream https://github.com/allenai/python-package-template.git git remote add upstream https://github.com/bellingcat/vk-url-scraper.git
Now if you do `git remote -v` again, you'll see Now if you do `git remote -v` again, you'll see
origin https://github.com/USERNAME/python-package-template.git (fetch) origin https://github.com/USERNAME/vk-url-scraper.git (fetch)
origin https://github.com/USERNAME/python-package-template.git (push) origin https://github.com/USERNAME/vk-url-scraper.git (push)
upstream https://github.com/allenai/python-package-template.git (fetch) upstream https://github.com/bellingcat/vk-url-scraper.git (fetch)
upstream https://github.com/allenai/python-package-template.git (push) upstream https://github.com/bellingcat/vk-url-scraper.git (push)
Finally, you'll need to create a Python 3 virtual environment suitable for working on this project. There a number of tools out there that making working with virtual environments easier. Finally, you'll need to create a Python 3 virtual environment suitable for working on this project. There a number of tools out there that making working with virtual environments easier.
The most direct way is with the [`venv` module](https://docs.python.org/3.7/library/venv.html) in the standard library, but if you're new to Python or you don't already have a recent Python 3 version installed on your machine, The most direct way is with the [`venv` module](https://docs.python.org/3.7/library/venv.html) in the standard library, but if you're new to Python or you don't already have a recent Python 3 version installed on your machine,
@@ -77,8 +77,8 @@ When you're ready to contribute code to address an open issue, please follow the
Then you can create and activate a new Python environment by running: Then you can create and activate a new Python environment by running:
conda create -n my-package python=3.9 conda create -n vk-url-scraper python=3.9
conda activate my-package conda activate vk-url-scraper
Once your virtual environment is activated, you can install your local clone in "editable mode" with Once your virtual environment is activated, you can install your local clone in "editable mode" with
@@ -93,7 +93,7 @@ When you're ready to contribute code to address an open issue, please follow the
<details><summary>Expand details 👇</summary><br/> <details><summary>Expand details 👇</summary><br/>
Once you've added an "upstream" remote pointing to [https://github.com/allenai/python-package-temlate.git](https://github.com/allenai/python-package-template), keeping your fork up-to-date is easy: Once you've added an "upstream" remote pointing to [https://github.com/bellingcat/vk-url-scraper.git](https://github.com/bellingcat/vk-url-scraper), keeping your fork up-to-date is easy:
git checkout main # if not already on main git checkout main # if not already on main
git pull --rebase upstream main git pull --rebase upstream main
@@ -119,7 +119,7 @@ When you're ready to contribute code to address an open issue, please follow the
<details><summary>Expand details 👇</summary><br/> <details><summary>Expand details 👇</summary><br/>
Our continuous integration (CI) testing runs [a number of checks](https://github.com/allenai/python-package-template/actions) for each pull request on [GitHub Actions](https://github.com/features/actions). You can run most of these tests locally, which is something you should do *before* opening a PR to help speed up the review process and make it easier for us. Our continuous integration (CI) testing runs [a number of checks](https://github.com/bellingcat/vk-url-scraper/actions) for each pull request on [GitHub Actions](https://github.com/features/actions). You can run most of these tests locally, which is something you should do *before* opening a PR to help speed up the review process and make it easier for us.
First, you should run [`isort`](https://github.com/PyCQA/isort) and [`black`](https://github.com/psf/black) to make sure you code is formatted consistently. First, you should run [`isort`](https://github.com/PyCQA/isort) and [`black`](https://github.com/psf/black) to make sure you code is formatted consistently.
Many IDEs support code formatters as plugins, so you may be able to setup isort and black to run automatically everytime you save. Many IDEs support code formatters as plugins, so you may be able to setup isort and black to run automatically everytime you save.
@@ -129,7 +129,7 @@ When you're ready to contribute code to address an open issue, please follow the
isort . isort .
black . black .
Our CI also uses [`flake8`](https://github.com/allenai/python-package-template/tree/main/tests) to lint the code base and [`mypy`](http://mypy-lang.org/) for type-checking. You should run both of these next with Our CI also uses [`flake8`](https://github.com/bellingcat/vk-url-scraper/tree/main/tests) to lint the code base and [`mypy`](http://mypy-lang.org/) for type-checking. You should run both of these next with
flake8 . flake8 .
@@ -137,15 +137,15 @@ When you're ready to contribute code to address an open issue, please follow the
mypy . mypy .
We also strive to maintain high test coverage, so most contributions should include additions to [the unit tests](https://github.com/allenai/python-package-template/tree/main/tests). These tests are run with [`pytest`](https://docs.pytest.org/en/latest/), which you can use to locally run any test modules that you've added or changed. We also strive to maintain high test coverage, so most contributions should include additions to [the unit tests](https://github.com/bellingcat/vk-url-scraper/tree/main/tests). These tests are run with [`pytest`](https://docs.pytest.org/en/latest/), which you can use to locally run any test modules that you've added or changed.
For example, if you've fixed a bug in `my_package/a/b.py`, you can run the tests specific to that module with For example, if you've fixed a bug in `vk_url_scraper/a/b.py`, you can run the tests specific to that module with
pytest -v tests/a/b_test.py pytest -v tests/a/b_test.py
To check the code coverage locally in this example, you could run To check the code coverage locally in this example, you could run
pytest -v --cov my_package.a.b tests/a/b_test.py pytest -v --cov vk_url_scraper.a.b tests/a/b_test.py
If your contribution involves additions to any public part of the API, we require that you write docstrings If your contribution involves additions to any public part of the API, we require that you write docstrings
for each function, method, class, or module that you add. for each function, method, class, or module that you add.
@@ -156,9 +156,9 @@ When you're ready to contribute code to address an open issue, please follow the
If the build fails, it's most likely due to small formatting issues. If the error message isn't clear, feel free to comment on this in your pull request. If the build fails, it's most likely due to small formatting issues. If the error message isn't clear, feel free to comment on this in your pull request.
And finally, please update the [CHANGELOG](https://github.com/allenai/python-package-template/blob/main/CHANGELOG.md) with notes on your contribution in the "Unreleased" section at the top. And finally, please update the [CHANGELOG](https://github.com/bellingcat/vk-url-scraper/blob/main/CHANGELOG.md) with notes on your contribution in the "Unreleased" section at the top.
After all of the above checks have passed, you can now open [a new GitHub pull request](https://github.com/allenai/python-package-template/pulls). After all of the above checks have passed, you can now open [a new GitHub pull request](https://github.com/bellingcat/vk-url-scraper/pulls).
Make sure you have a clear description of the problem and the solution, and include a link to relevant issues. Make sure you have a clear description of the problem and the solution, and include a link to relevant issues.
We look forward to reviewing your PR! We look forward to reviewing your PR!

214
LICENSE
View File

@@ -1,201 +1,21 @@
Apache License MIT License
Version 2.0, January 2004
https://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION Copyright (c) 2022 Stichting Bellingcat
1. Definitions. Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
"License" shall mean the terms and conditions for use, reproduction, The above copyright notice and this permission notice shall be included in all
and distribution as defined by Sections 1 through 9 of this document. copies or substantial portions of the Software.
"Licensor" shall mean the copyright owner or entity authorized by THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
the copyright owner that is granting the License. IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
"Legal Entity" shall mean the union of the acting entity and all AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
other entities that control, are controlled by, or are under common LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
control with that entity. For the purposes of this definition, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
"control" means (i) the power, direct or indirect, to cause the SOFTWARE.
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright {yyyy} {name of copyright owner}
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

4
Makefile
View File

@@ -1,7 +1,7 @@
.PHONY : docs .PHONY : docs
docs : docs :
rm -rf docs/build/ rm -rf docs/build/
sphinx-autobuild -b html --watch my_package/ docs/source/ docs/build/ sphinx-autobuild -b html --watch vk_url_scraper/ docs/source/ docs/build/
.PHONY : run-checks .PHONY : run-checks
run-checks : run-checks :
@@ -9,4 +9,4 @@ run-checks :
black --check . black --check .
flake8 . flake8 .
mypy . mypy .
CUDA_VISIBLE_DEVICES='' pytest -v --color=yes --doctest-modules tests/ my_package/ CUDA_VISIBLE_DEVICES='' pytest -v --color=yes --doctest-modules tests/ vk_url_scraper/

27
Pipfile Normal file
View File

@@ -0,0 +1,27 @@
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
vk-api = "*"
[dev-packages]
sphinx-copybutton = "==0.5.0"
flake8 = "*"
mypy = "==0.961"
black = "==22.3.0"
isort = "==5.10.1"
pytest = "*"
pytest-sphinx = "*"
pytest-cov = "*"
twine = ">=1.11.0"
sphinx = ">=4.3.0,<5.1.0"
furo = "==2022.6.4.1"
myst-parser = ">=0.15.2,<0.19.0"
sphinx-autobuild = "==2021.3.14"
sphinx-autodoc-typehints = "*"
python-dotenv = "*"
[requires]
python_version = "3.9"

940
Pipfile.lock generated Normal file
View File

@@ -0,0 +1,940 @@
{
"_meta": {
"hash": {
"sha256": "bab533e734f6da55647cc76a9f5a51d46c641723d485e38a16e2e31bca097130"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.9"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.org/simple",
"verify_ssl": true
}
]
},
"default": {
"certifi": {
"hashes": [
"sha256:84c85a9078b11105f04f3036a9482ae10e4621616db313fe045dd24743a0820d",
"sha256:fe86415d55e84719d75f8b69414f6438ac3547d2078ab91b67e779ef69378412"
],
"markers": "python_version >= '3.6'",
"version": "==2022.6.15"
},
"charset-normalizer": {
"hashes": [
"sha256:2857e29ff0d34db842cd7ca3230549d1a697f96ee6d3fb071cfa6c7393832597",
"sha256:6881edbebdb17b39b4eaaa821b438bf6eddffb4468cf344f09f89def34a8b1df"
],
"markers": "python_version >= '3.5'",
"version": "==2.0.12"
},
"idna": {
"hashes": [
"sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff",
"sha256:9d643ff0a55b762d5cdb124b8eaa99c66322e2157b69160bc32796e824360e6d"
],
"markers": "python_version >= '3.5'",
"version": "==3.3"
},
"requests": {
"hashes": [
"sha256:bc7861137fbce630f17b03d3ad02ad0bf978c844f3536d0edda6499dafce2b6f",
"sha256:d568723a7ebd25875d8d1eaf5dfa068cd2fc8194b2e483d7b1f7c81918dbec6b"
],
"markers": "python_version >= '3.7' and python_version < '4'",
"version": "==2.28.0"
},
"urllib3": {
"hashes": [
"sha256:44ece4d53fb1706f667c9bd1c648f5469a2ec925fcf3a776667042d645472c14",
"sha256:aabaf16477806a5e1dd19aa41f8c2b7950dd3c746362d7e3223dbe6de6ac448e"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4'",
"version": "==1.26.9"
},
"vk-api": {
"hashes": [
"sha256:11c731e214ebc7fa911db81efb021f97587493a5402b992f24748fe1cd9d7afc",
"sha256:d0ae766fa93a40d47c5da045d94201721bf766dbde122a1d2253516b35c5edf3"
],
"index": "pypi",
"version": "==11.9.8"
}
},
"develop": {
"alabaster": {
"hashes": [
"sha256:446438bdcca0e05bd45ea2de1668c1d9b032e1a9154c2c259092d77031ddd359",
"sha256:a661d72d58e6ea8a57f7a86e37d86716863ee5e92788398526d58b26a4e4dc02"
],
"version": "==0.7.12"
},
"attrs": {
"hashes": [
"sha256:2d27e3784d7a565d36ab851fe94887c5eccd6a463168875832a1be79c82828b4",
"sha256:626ba8234211db98e869df76230a137c4c40a12d72445c45d5f5b716f076e2fd"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
"version": "==21.4.0"
},
"babel": {
"hashes": [
"sha256:7614553711ee97490f732126dc077f8d0ae084ebc6a96e23db1482afabdb2c51",
"sha256:ff56f4892c1c4bf0d814575ea23471c230d544203c7748e8c68f0089478d48eb"
],
"markers": "python_version >= '3.6'",
"version": "==2.10.3"
},
"beautifulsoup4": {
"hashes": [
"sha256:58d5c3d29f5a36ffeb94f02f0d786cd53014cf9b3b3951d42e0080d8a9498d30",
"sha256:ad9aa55b65ef2808eb405f46cf74df7fcb7044d5cbc26487f96eb2ef2e436693"
],
"markers": "python_version >= '3.6'",
"version": "==4.11.1"
},
"black": {
"hashes": [
"sha256:06f9d8846f2340dfac80ceb20200ea5d1b3f181dd0556b47af4e8e0b24fa0a6b",
"sha256:10dbe6e6d2988049b4655b2b739f98785a884d4d6b85bc35133a8fb9a2233176",
"sha256:2497f9c2386572e28921fa8bec7be3e51de6801f7459dffd6e62492531c47e09",
"sha256:30d78ba6bf080eeaf0b7b875d924b15cd46fec5fd044ddfbad38c8ea9171043a",
"sha256:328efc0cc70ccb23429d6be184a15ce613f676bdfc85e5fe8ea2a9354b4e9015",
"sha256:35020b8886c022ced9282b51b5a875b6d1ab0c387b31a065b84db7c33085ca79",
"sha256:5795a0375eb87bfe902e80e0c8cfaedf8af4d49694d69161e5bd3206c18618bb",
"sha256:5891ef8abc06576985de8fa88e95ab70641de6c1fca97e2a15820a9b69e51b20",
"sha256:637a4014c63fbf42a692d22b55d8ad6968a946b4a6ebc385c5505d9625b6a464",
"sha256:67c8301ec94e3bcc8906740fe071391bce40a862b7be0b86fb5382beefecd968",
"sha256:6d2fc92002d44746d3e7db7cf9313cf4452f43e9ea77a2c939defce3b10b5c82",
"sha256:6ee227b696ca60dd1c507be80a6bc849a5a6ab57ac7352aad1ffec9e8b805f21",
"sha256:863714200ada56cbc366dc9ae5291ceb936573155f8bf8e9de92aef51f3ad0f0",
"sha256:9b542ced1ec0ceeff5b37d69838106a6348e60db7b8fdd245294dc1d26136265",
"sha256:a6342964b43a99dbc72f72812bf88cad8f0217ae9acb47c0d4f141a6416d2d7b",
"sha256:ad4efa5fad66b903b4a5f96d91461d90b9507a812b3c5de657d544215bb7877a",
"sha256:bc58025940a896d7e5356952228b68f793cf5fcb342be703c3a2669a1488cb72",
"sha256:cc1e1de68c8e5444e8f94c3670bb48a2beef0e91dddfd4fcc29595ebd90bb9ce",
"sha256:cee3e11161dde1b2a33a904b850b0899e0424cc331b7295f2a9698e79f9a69a0",
"sha256:e3556168e2e5c49629f7b0f377070240bd5511e45e25a4497bb0073d9dda776a",
"sha256:e8477ec6bbfe0312c128e74644ac8a02ca06bcdb8982d4ee06f209be28cdf163",
"sha256:ee8f1f7228cce7dffc2b464f07ce769f478968bfb3dd1254a4c2eeed84928aad",
"sha256:fd57160949179ec517d32ac2ac898b5f20d68ed1a9c977346efbac9c2f1e779d"
],
"index": "pypi",
"version": "==22.3.0"
},
"bleach": {
"hashes": [
"sha256:08a1fe86d253b5c88c92cc3d810fd8048a16d15762e1e5b74d502256e5926aa1",
"sha256:c6d6cc054bdc9c83b48b8083e236e5f00f238428666d2ce2e083eaa5fd568565"
],
"markers": "python_version >= '3.7'",
"version": "==5.0.0"
},
"certifi": {
"hashes": [
"sha256:84c85a9078b11105f04f3036a9482ae10e4621616db313fe045dd24743a0820d",
"sha256:fe86415d55e84719d75f8b69414f6438ac3547d2078ab91b67e779ef69378412"
],
"markers": "python_version >= '3.6'",
"version": "==2022.6.15"
},
"cffi": {
"hashes": [
"sha256:00c878c90cb53ccfaae6b8bc18ad05d2036553e6d9d1d9dbcf323bbe83854ca3",
"sha256:0104fb5ae2391d46a4cb082abdd5c69ea4eab79d8d44eaaf79f1b1fd806ee4c2",
"sha256:06c48159c1abed75c2e721b1715c379fa3200c7784271b3c46df01383b593636",
"sha256:0808014eb713677ec1292301ea4c81ad277b6cdf2fdd90fd540af98c0b101d20",
"sha256:10dffb601ccfb65262a27233ac273d552ddc4d8ae1bf93b21c94b8511bffe728",
"sha256:14cd121ea63ecdae71efa69c15c5543a4b5fbcd0bbe2aad864baca0063cecf27",
"sha256:17771976e82e9f94976180f76468546834d22a7cc404b17c22df2a2c81db0c66",
"sha256:181dee03b1170ff1969489acf1c26533710231c58f95534e3edac87fff06c443",
"sha256:23cfe892bd5dd8941608f93348c0737e369e51c100d03718f108bf1add7bd6d0",
"sha256:263cc3d821c4ab2213cbe8cd8b355a7f72a8324577dc865ef98487c1aeee2bc7",
"sha256:2756c88cbb94231c7a147402476be2c4df2f6078099a6f4a480d239a8817ae39",
"sha256:27c219baf94952ae9d50ec19651a687b826792055353d07648a5695413e0c605",
"sha256:2a23af14f408d53d5e6cd4e3d9a24ff9e05906ad574822a10563efcef137979a",
"sha256:31fb708d9d7c3f49a60f04cf5b119aeefe5644daba1cd2a0fe389b674fd1de37",
"sha256:3415c89f9204ee60cd09b235810be700e993e343a408693e80ce7f6a40108029",
"sha256:3773c4d81e6e818df2efbc7dd77325ca0dcb688116050fb2b3011218eda36139",
"sha256:3b96a311ac60a3f6be21d2572e46ce67f09abcf4d09344c49274eb9e0bf345fc",
"sha256:3f7d084648d77af029acb79a0ff49a0ad7e9d09057a9bf46596dac9514dc07df",
"sha256:41d45de54cd277a7878919867c0f08b0cf817605e4eb94093e7516505d3c8d14",
"sha256:4238e6dab5d6a8ba812de994bbb0a79bddbdf80994e4ce802b6f6f3142fcc880",
"sha256:45db3a33139e9c8f7c09234b5784a5e33d31fd6907800b316decad50af323ff2",
"sha256:45e8636704eacc432a206ac7345a5d3d2c62d95a507ec70d62f23cd91770482a",
"sha256:4958391dbd6249d7ad855b9ca88fae690783a6be9e86df65865058ed81fc860e",
"sha256:4a306fa632e8f0928956a41fa8e1d6243c71e7eb59ffbd165fc0b41e316b2474",
"sha256:57e9ac9ccc3101fac9d6014fba037473e4358ef4e89f8e181f8951a2c0162024",
"sha256:59888172256cac5629e60e72e86598027aca6bf01fa2465bdb676d37636573e8",
"sha256:5e069f72d497312b24fcc02073d70cb989045d1c91cbd53979366077959933e0",
"sha256:64d4ec9f448dfe041705426000cc13e34e6e5bb13736e9fd62e34a0b0c41566e",
"sha256:6dc2737a3674b3e344847c8686cf29e500584ccad76204efea14f451d4cc669a",
"sha256:74fdfdbfdc48d3f47148976f49fab3251e550a8720bebc99bf1483f5bfb5db3e",
"sha256:75e4024375654472cc27e91cbe9eaa08567f7fbdf822638be2814ce059f58032",
"sha256:786902fb9ba7433aae840e0ed609f45c7bcd4e225ebb9c753aa39725bb3e6ad6",
"sha256:8b6c2ea03845c9f501ed1313e78de148cd3f6cad741a75d43a29b43da27f2e1e",
"sha256:91d77d2a782be4274da750752bb1650a97bfd8f291022b379bb8e01c66b4e96b",
"sha256:91ec59c33514b7c7559a6acda53bbfe1b283949c34fe7440bcf917f96ac0723e",
"sha256:920f0d66a896c2d99f0adbb391f990a84091179542c205fa53ce5787aff87954",
"sha256:a5263e363c27b653a90078143adb3d076c1a748ec9ecc78ea2fb916f9b861962",
"sha256:abb9a20a72ac4e0fdb50dae135ba5e77880518e742077ced47eb1499e29a443c",
"sha256:c2051981a968d7de9dd2d7b87bcb9c939c74a34626a6e2f8181455dd49ed69e4",
"sha256:c21c9e3896c23007803a875460fb786118f0cdd4434359577ea25eb556e34c55",
"sha256:c2502a1a03b6312837279c8c1bd3ebedf6c12c4228ddbad40912d671ccc8a962",
"sha256:d4d692a89c5cf08a8557fdeb329b82e7bf609aadfaed6c0d79f5a449a3c7c023",
"sha256:da5db4e883f1ce37f55c667e5c0de439df76ac4cb55964655906306918e7363c",
"sha256:e7022a66d9b55e93e1a845d8c9eba2a1bebd4966cd8bfc25d9cd07d515b33fa6",
"sha256:ef1f279350da2c586a69d32fc8733092fd32cc8ac95139a00377841f59a3f8d8",
"sha256:f54a64f8b0c8ff0b64d18aa76675262e1700f3995182267998c31ae974fbc382",
"sha256:f5c7150ad32ba43a07c4479f40241756145a1f03b43480e058cfd862bf5041c7",
"sha256:f6f824dc3bce0edab5f427efcfb1d63ee75b6fcb7282900ccaf925be84efb0fc",
"sha256:fd8a250edc26254fe5b33be00402e6d287f562b6a5b2152dec302fa15bb3e997",
"sha256:ffaa5c925128e29efbde7301d8ecaf35c8c60ffbcd6a1ffd3a552177c8e5e796"
],
"version": "==1.15.0"
},
"charset-normalizer": {
"hashes": [
"sha256:2857e29ff0d34db842cd7ca3230549d1a697f96ee6d3fb071cfa6c7393832597",
"sha256:6881edbebdb17b39b4eaaa821b438bf6eddffb4468cf344f09f89def34a8b1df"
],
"markers": "python_version >= '3.5'",
"version": "==2.0.12"
},
"click": {
"hashes": [
"sha256:7682dc8afb30297001674575ea00d1814d808d6a36af415a82bd481d37ba7b8e",
"sha256:bb4d8133cb15a609f44e8213d9b391b0809795062913b383c62be0ee95b1db48"
],
"markers": "python_version >= '3.7'",
"version": "==8.1.3"
},
"colorama": {
"hashes": [
"sha256:854bf444933e37f5824ae7bfc1e98d5bce2ebe4160d46b5edf346a89358e99da",
"sha256:e6c6b4334fc50988a639d9b98aa429a0b57da6e17b9a44f0451f930b6967b7a4"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
"version": "==0.4.5"
},
"commonmark": {
"hashes": [
"sha256:452f9dc859be7f06631ddcb328b6919c67984aca654e5fefb3914d54691aed60",
"sha256:da2f38c92590f83de410ba1a3cbceafbc74fee9def35f9251ba9a971d6d66fd9"
],
"version": "==0.9.1"
},
"coverage": {
"extras": [
"toml"
],
"hashes": [
"sha256:01c5615d13f3dd3aa8543afc069e5319cfa0c7d712f6e04b920431e5c564a749",
"sha256:106c16dfe494de3193ec55cac9640dd039b66e196e4641fa8ac396181578b982",
"sha256:129cd05ba6f0d08a766d942a9ed4b29283aff7b2cccf5b7ce279d50796860bb3",
"sha256:145f296d00441ca703a659e8f3eb48ae39fb083baba2d7ce4482fb2723e050d9",
"sha256:1480ff858b4113db2718848d7b2d1b75bc79895a9c22e76a221b9d8d62496428",
"sha256:269eaa2c20a13a5bf17558d4dc91a8d078c4fa1872f25303dddcbba3a813085e",
"sha256:26dff09fb0d82693ba9e6231248641d60ba606150d02ed45110f9ec26404ed1c",
"sha256:2bd9a6fc18aab8d2e18f89b7ff91c0f34ff4d5e0ba0b33e989b3cd4194c81fd9",
"sha256:309ce4a522ed5fca432af4ebe0f32b21d6d7ccbb0f5fcc99290e71feba67c264",
"sha256:3384f2a3652cef289e38100f2d037956194a837221edd520a7ee5b42d00cc605",
"sha256:342d4aefd1c3e7f620a13f4fe563154d808b69cccef415415aece4c786665397",
"sha256:39ee53946bf009788108b4dd2894bf1349b4e0ca18c2016ffa7d26ce46b8f10d",
"sha256:4321f075095a096e70aff1d002030ee612b65a205a0a0f5b815280d5dc58100c",
"sha256:4803e7ccf93230accb928f3a68f00ffa80a88213af98ed338a57ad021ef06815",
"sha256:4ce1b258493cbf8aec43e9b50d89982346b98e9ffdfaae8ae5793bc112fb0068",
"sha256:664a47ce62fe4bef9e2d2c430306e1428ecea207ffd68649e3b942fa8ea83b0b",
"sha256:75ab269400706fab15981fd4bd5080c56bd5cc07c3bccb86aab5e1d5a88dc8f4",
"sha256:83c4e737f60c6936460c5be330d296dd5b48b3963f48634c53b3f7deb0f34ec4",
"sha256:84631e81dd053e8a0d4967cedab6db94345f1c36107c71698f746cb2636c63e3",
"sha256:84e65ef149028516c6d64461b95a8dbcfce95cfd5b9eb634320596173332ea84",
"sha256:865d69ae811a392f4d06bde506d531f6a28a00af36f5c8649684a9e5e4a85c83",
"sha256:87f4f3df85aa39da00fd3ec4b5abeb7407e82b68c7c5ad181308b0e2526da5d4",
"sha256:8c08da0bd238f2970230c2a0d28ff0e99961598cb2e810245d7fc5afcf1254e8",
"sha256:961e2fb0680b4f5ad63234e0bf55dfb90d302740ae9c7ed0120677a94a1590cb",
"sha256:9b3e07152b4563722be523e8cd0b209e0d1a373022cfbde395ebb6575bf6790d",
"sha256:a7f3049243783df2e6cc6deafc49ea123522b59f464831476d3d1448e30d72df",
"sha256:bf5601c33213d3cb19d17a796f8a14a9eaa5e87629a53979a5981e3e3ae166f6",
"sha256:cec3a0f75c8f1031825e19cd86ee787e87cf03e4fd2865c79c057092e69e3a3b",
"sha256:d42c549a8f41dc103a8004b9f0c433e2086add8a719da00e246e17cbe4056f72",
"sha256:d67d44996140af8b84284e5e7d398e589574b376fb4de8ccd28d82ad8e3bea13",
"sha256:d9c80df769f5ec05ad21ea34be7458d1dc51ff1fb4b2219e77fe24edf462d6df",
"sha256:e57816f8ffe46b1df8f12e1b348f06d164fd5219beba7d9433ba79608ef011cc",
"sha256:ee2ddcac99b2d2aec413e36d7a429ae9ebcadf912946b13ffa88e7d4c9b712d6",
"sha256:f02cbbf8119db68455b9d763f2f8737bb7db7e43720afa07d8eb1604e5c5ae28",
"sha256:f1d5aa2703e1dab4ae6cf416eb0095304f49d004c39e9db1d86f57924f43006b",
"sha256:f5b66caa62922531059bc5ac04f836860412f7f88d38a476eda0a6f11d4724f4",
"sha256:f69718750eaae75efe506406c490d6fc5a6161d047206cc63ce25527e8a3adad",
"sha256:fb73e0011b8793c053bfa85e53129ba5f0250fdc0392c1591fd35d915ec75c46",
"sha256:fd180ed867e289964404051a958f7cccabdeed423f91a899829264bb7974d3d3",
"sha256:fdb6f7bd51c2d1714cea40718f6149ad9be6a2ee7d93b19e9f00934c0f2a74d9",
"sha256:ffa9297c3a453fba4717d06df579af42ab9a28022444cae7fa605af4df612d54"
],
"markers": "python_version >= '3.7'",
"version": "==6.4.1"
},
"cryptography": {
"hashes": [
"sha256:093cb351031656d3ee2f4fa1be579a8c69c754cf874206be1d4cf3b542042804",
"sha256:0cc20f655157d4cfc7bada909dc5cc228211b075ba8407c46467f63597c78178",
"sha256:1b9362d34363f2c71b7853f6251219298124aa4cc2075ae2932e64c91a3e2717",
"sha256:1f3bfbd611db5cb58ca82f3deb35e83af34bb8cf06043fa61500157d50a70982",
"sha256:2bd1096476aaac820426239ab534b636c77d71af66c547b9ddcd76eb9c79e004",
"sha256:31fe38d14d2e5f787e0aecef831457da6cec68e0bb09a35835b0b44ae8b988fe",
"sha256:3b8398b3d0efc420e777c40c16764d6870bcef2eb383df9c6dbb9ffe12c64452",
"sha256:3c81599befb4d4f3d7648ed3217e00d21a9341a9a688ecdd615ff72ffbed7336",
"sha256:419c57d7b63f5ec38b1199a9521d77d7d1754eb97827bbb773162073ccd8c8d4",
"sha256:46f4c544f6557a2fefa7ac8ac7d1b17bf9b647bd20b16decc8fbcab7117fbc15",
"sha256:471e0d70201c069f74c837983189949aa0d24bb2d751b57e26e3761f2f782b8d",
"sha256:59b281eab51e1b6b6afa525af2bd93c16d49358404f814fe2c2410058623928c",
"sha256:731c8abd27693323b348518ed0e0705713a36d79fdbd969ad968fbef0979a7e0",
"sha256:95e590dd70642eb2079d280420a888190aa040ad20f19ec8c6e097e38aa29e06",
"sha256:a68254dd88021f24a68b613d8c51d5c5e74d735878b9e32cc0adf19d1f10aaf9",
"sha256:a7d5137e556cc0ea418dca6186deabe9129cee318618eb1ffecbd35bee55ddc1",
"sha256:aeaba7b5e756ea52c8861c133c596afe93dd716cbcacae23b80bc238202dc023",
"sha256:dc26bb134452081859aa21d4990474ddb7e863aa39e60d1592800a8865a702de",
"sha256:e53258e69874a306fcecb88b7534d61820db8a98655662a3dd2ec7f1afd9132f",
"sha256:ef15c2df7656763b4ff20a9bc4381d8352e6640cfeb95c2972c38ef508e75181",
"sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e",
"sha256:f8ec91983e638a9bcd75b39f1396e5c0dc2330cbd9ce4accefe68717e6779e0a"
],
"markers": "python_version >= '3.6'",
"version": "==37.0.2"
},
"docutils": {
"hashes": [
"sha256:23010f129180089fbcd3bc08cfefccb3b890b0050e1ca00c867036e9d161b98c",
"sha256:679987caf361a7539d76e584cbeddc311e3aee937877c87346f31debc63e9d06"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
"version": "==0.18.1"
},
"flake8": {
"hashes": [
"sha256:479b1304f72536a55948cb40a32dce8bb0ffe3501e26eaf292c7e60eb5e0428d",
"sha256:806e034dda44114815e23c16ef92f95c91e4c71100ff52813adf7132a6ad870d"
],
"index": "pypi",
"version": "==4.0.1"
},
"furo": {
"hashes": [
"sha256:79f2d3a61e3d971c0acd59f53d3202e7336789cd893f7bdc3cc7bc37d6ef252c",
"sha256:c927848edf3292030d0719ebdab9e16d56f1b91c68562b9be316aa5b843775ab"
],
"index": "pypi",
"version": "==2022.6.4.1"
},
"idna": {
"hashes": [
"sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff",
"sha256:9d643ff0a55b762d5cdb124b8eaa99c66322e2157b69160bc32796e824360e6d"
],
"markers": "python_version >= '3.5'",
"version": "==3.3"
},
"imagesize": {
"hashes": [
"sha256:1db2f82529e53c3e929e8926a1fa9235aa82d0bd0c580359c67ec31b2fddaa8c",
"sha256:cd1750d452385ca327479d45b64d9c7729ecf0b3969a58148298c77092261f9d"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
"version": "==1.3.0"
},
"importlib-metadata": {
"hashes": [
"sha256:5d26852efe48c0a32b0509ffbc583fda1a2266545a78d104a6f4aff3db17d700",
"sha256:c58c8eb8a762858f49e18436ff552e83914778e50e9d2f1660535ffb364552ec"
],
"markers": "python_version < '3.10'",
"version": "==4.11.4"
},
"iniconfig": {
"hashes": [
"sha256:011e24c64b7f47f6ebd835bb12a743f2fbe9a26d4cecaa7f53bc4f35ee9da8b3",
"sha256:bc3af051d7d14b2ee5ef9969666def0cd1a000e121eaea580d4a313df4b37f32"
],
"version": "==1.1.1"
},
"isort": {
"hashes": [
"sha256:6f62d78e2f89b4500b080fe3a81690850cd254227f27f75c3a0c491a1f351ba7",
"sha256:e8443a5e7a020e9d7f97f1d7d9cd17c88bcb3bc7e218bf9cf5095fe550be2951"
],
"index": "pypi",
"version": "==5.10.1"
},
"jeepney": {
"hashes": [
"sha256:5efe48d255973902f6badc3ce55e2aa6c5c3b3bc642059ef3a91247bcfcc5806",
"sha256:c0a454ad016ca575060802ee4d590dd912e35c122fa04e70306de3d076cce755"
],
"markers": "sys_platform == 'linux'",
"version": "==0.8.0"
},
"jinja2": {
"hashes": [
"sha256:31351a702a408a9e7595a8fc6150fc3f43bb6bf7e319770cbc0db9df9437e852",
"sha256:6088930bfe239f0e6710546ab9c19c9ef35e29792895fed6e6e31a023a182a61"
],
"markers": "python_version >= '3.7'",
"version": "==3.1.2"
},
"keyring": {
"hashes": [
"sha256:372ff2fc43ab779e3f87911c26e6c7acc8bb440cbd82683e383ca37594cb0617",
"sha256:3ac00c26e4c93739e19103091a9986a9f79665a78cf15a4df1dba7ea9ac8da2f"
],
"markers": "python_version >= '3.7'",
"version": "==23.6.0"
},
"livereload": {
"hashes": [
"sha256:776f2f865e59fde56490a56bcc6773b6917366bce0c267c60ee8aaf1a0959869"
],
"version": "==2.6.3"
},
"markdown-it-py": {
"hashes": [
"sha256:93de681e5c021a432c63147656fe21790bc01231e0cd2da73626f1aa3ac0fe27",
"sha256:cf7e59fed14b5ae17c0006eff14a2d9a00ed5f3a846148153899a0224e2c07da"
],
"markers": "python_version >= '3.7'",
"version": "==2.1.0"
},
"markupsafe": {
"hashes": [
"sha256:0212a68688482dc52b2d45013df70d169f542b7394fc744c02a57374a4207003",
"sha256:089cf3dbf0cd6c100f02945abeb18484bd1ee57a079aefd52cffd17fba910b88",
"sha256:10c1bfff05d95783da83491be968e8fe789263689c02724e0c691933c52994f5",
"sha256:33b74d289bd2f5e527beadcaa3f401e0df0a89927c1559c8566c066fa4248ab7",
"sha256:3799351e2336dc91ea70b034983ee71cf2f9533cdff7c14c90ea126bfd95d65a",
"sha256:3ce11ee3f23f79dbd06fb3d63e2f6af7b12db1d46932fe7bd8afa259a5996603",
"sha256:421be9fbf0ffe9ffd7a378aafebbf6f4602d564d34be190fc19a193232fd12b1",
"sha256:43093fb83d8343aac0b1baa75516da6092f58f41200907ef92448ecab8825135",
"sha256:46d00d6cfecdde84d40e572d63735ef81423ad31184100411e6e3388d405e247",
"sha256:4a33dea2b688b3190ee12bd7cfa29d39c9ed176bda40bfa11099a3ce5d3a7ac6",
"sha256:4b9fe39a2ccc108a4accc2676e77da025ce383c108593d65cc909add5c3bd601",
"sha256:56442863ed2b06d19c37f94d999035e15ee982988920e12a5b4ba29b62ad1f77",
"sha256:671cd1187ed5e62818414afe79ed29da836dde67166a9fac6d435873c44fdd02",
"sha256:694deca8d702d5db21ec83983ce0bb4b26a578e71fbdbd4fdcd387daa90e4d5e",
"sha256:6a074d34ee7a5ce3effbc526b7083ec9731bb3cbf921bbe1d3005d4d2bdb3a63",
"sha256:6d0072fea50feec76a4c418096652f2c3238eaa014b2f94aeb1d56a66b41403f",
"sha256:6fbf47b5d3728c6aea2abb0589b5d30459e369baa772e0f37a0320185e87c980",
"sha256:7f91197cc9e48f989d12e4e6fbc46495c446636dfc81b9ccf50bb0ec74b91d4b",
"sha256:86b1f75c4e7c2ac2ccdaec2b9022845dbb81880ca318bb7a0a01fbf7813e3812",
"sha256:8dc1c72a69aa7e082593c4a203dcf94ddb74bb5c8a731e4e1eb68d031e8498ff",
"sha256:8e3dcf21f367459434c18e71b2a9532d96547aef8a871872a5bd69a715c15f96",
"sha256:8e576a51ad59e4bfaac456023a78f6b5e6e7651dcd383bcc3e18d06f9b55d6d1",
"sha256:96e37a3dc86e80bf81758c152fe66dbf60ed5eca3d26305edf01892257049925",
"sha256:97a68e6ada378df82bc9f16b800ab77cbf4b2fada0081794318520138c088e4a",
"sha256:99a2a507ed3ac881b975a2976d59f38c19386d128e7a9a18b7df6fff1fd4c1d6",
"sha256:a49907dd8420c5685cfa064a1335b6754b74541bbb3706c259c02ed65b644b3e",
"sha256:b09bf97215625a311f669476f44b8b318b075847b49316d3e28c08e41a7a573f",
"sha256:b7bd98b796e2b6553da7225aeb61f447f80a1ca64f41d83612e6139ca5213aa4",
"sha256:b87db4360013327109564f0e591bd2a3b318547bcef31b468a92ee504d07ae4f",
"sha256:bcb3ed405ed3222f9904899563d6fc492ff75cce56cba05e32eff40e6acbeaa3",
"sha256:d4306c36ca495956b6d568d276ac11fdd9c30a36f1b6eb928070dc5360b22e1c",
"sha256:d5ee4f386140395a2c818d149221149c54849dfcfcb9f1debfe07a8b8bd63f9a",
"sha256:dda30ba7e87fbbb7eab1ec9f58678558fd9a6b8b853530e176eabd064da81417",
"sha256:e04e26803c9c3851c931eac40c695602c6295b8d432cbe78609649ad9bd2da8a",
"sha256:e1c0b87e09fa55a220f058d1d49d3fb8df88fbfab58558f1198e08c1e1de842a",
"sha256:e72591e9ecd94d7feb70c1cbd7be7b3ebea3f548870aa91e2732960fa4d57a37",
"sha256:e8c843bbcda3a2f1e3c2ab25913c80a3c5376cd00c6e8c4a86a89a28c8dc5452",
"sha256:efc1913fd2ca4f334418481c7e595c00aad186563bbc1ec76067848c7ca0a933",
"sha256:f121a1420d4e173a5d96e47e9a0c0dcff965afdf1626d28de1460815f7c4ee7a",
"sha256:fc7b548b17d238737688817ab67deebb30e8073c95749d55538ed473130ec0c7"
],
"markers": "python_version >= '3.7'",
"version": "==2.1.1"
},
"mccabe": {
"hashes": [
"sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
"sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
],
"version": "==0.6.1"
},
"mdit-py-plugins": {
"hashes": [
"sha256:b1279701cee2dbf50e188d3da5f51fee8d78d038cdf99be57c6b9d1aa93b4073",
"sha256:ecc24f51eeec6ab7eecc2f9724e8272c2fb191c2e93cf98109120c2cace69750"
],
"markers": "python_version ~= '3.6'",
"version": "==0.3.0"
},
"mdurl": {
"hashes": [
"sha256:6a8f6804087b7128040b2fb2ebe242bdc2affaeaa034d5fc9feeed30b443651b",
"sha256:f79c9709944df218a4cdb0fcc0b0c7ead2f44594e3e84dc566606f04ad749c20"
],
"markers": "python_version >= '3.7'",
"version": "==0.1.1"
},
"mypy": {
"hashes": [
"sha256:006be38474216b833eca29ff6b73e143386f352e10e9c2fbe76aa8549e5554f5",
"sha256:03c6cc893e7563e7b2949b969e63f02c000b32502a1b4d1314cabe391aa87d66",
"sha256:0e9f70df36405c25cc530a86eeda1e0867863d9471fe76d1273c783df3d35c2e",
"sha256:1ece702f29270ec6af25db8cf6185c04c02311c6bb21a69f423d40e527b75c56",
"sha256:3e09f1f983a71d0672bbc97ae33ee3709d10c779beb613febc36805a6e28bb4e",
"sha256:439c726a3b3da7ca84a0199a8ab444cd8896d95012c4a6c4a0d808e3147abf5d",
"sha256:5a0b53747f713f490affdceef835d8f0cb7285187a6a44c33821b6d1f46ed813",
"sha256:5f1332964963d4832a94bebc10f13d3279be3ce8f6c64da563d6ee6e2eeda932",
"sha256:63e85a03770ebf403291ec50097954cc5caf2a9205c888ce3a61bd3f82e17569",
"sha256:64759a273d590040a592e0f4186539858c948302c653c2eac840c7a3cd29e51b",
"sha256:697540876638ce349b01b6786bc6094ccdaba88af446a9abb967293ce6eaa2b0",
"sha256:9940e6916ed9371809b35b2154baf1f684acba935cd09928952310fbddaba648",
"sha256:9f5f5a74085d9a81a1f9c78081d60a0040c3efb3f28e5c9912b900adf59a16e6",
"sha256:a5ea0875a049de1b63b972456542f04643daf320d27dc592d7c3d9cd5d9bf950",
"sha256:b117650592e1782819829605a193360a08aa99f1fc23d1d71e1a75a142dc7e15",
"sha256:b24be97351084b11582fef18d79004b3e4db572219deee0212078f7cf6352723",
"sha256:b88f784e9e35dcaa075519096dc947a388319cb86811b6af621e3523980f1c8a",
"sha256:bdd5ca340beffb8c44cb9dc26697628d1b88c6bddf5c2f6eb308c46f269bb6f3",
"sha256:d5aaf1edaa7692490f72bdb9fbd941fbf2e201713523bdb3f4038be0af8846c6",
"sha256:e999229b9f3198c0c880d5e269f9f8129c8862451ce53a011326cad38b9ccd24",
"sha256:f4a21d01fc0ba4e31d82f0fff195682e29f9401a8bdb7173891070eb260aeb3b",
"sha256:f4b794db44168a4fc886e3450201365c9526a522c46ba089b55e1f11c163750d",
"sha256:f730d56cb924d371c26b8eaddeea3cc07d78ff51c521c6d04899ac6904b75492"
],
"index": "pypi",
"version": "==0.961"
},
"mypy-extensions": {
"hashes": [
"sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d",
"sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"
],
"version": "==0.4.3"
},
"myst-parser": {
"hashes": [
"sha256:4965e51918837c13bf1c6f6fe2c6bddddf193148360fbdaefe743a4981358f6a",
"sha256:739a4d96773a8e55a2cacd3941ce46a446ee23dcd6b37e06f73f551ad7821d86"
],
"index": "pypi",
"version": "==0.18.0"
},
"packaging": {
"hashes": [
"sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb",
"sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522"
],
"markers": "python_version >= '3.6'",
"version": "==21.3"
},
"pathspec": {
"hashes": [
"sha256:7d15c4ddb0b5c802d161efc417ec1a2558ea2653c2e8ad9c19098201dc1c993a",
"sha256:e564499435a2673d586f6b2130bb5b95f04a3ba06f81b8f895b651a3c76aabb1"
],
"version": "==0.9.0"
},
"pkginfo": {
"hashes": [
"sha256:848865108ec99d4901b2f7e84058b6e7660aae8ae10164e015a6dcf5b242a594",
"sha256:a84da4318dd86f870a9447a8c98340aa06216bfc6f2b7bdc4b8766984ae1867c"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'",
"version": "==1.8.3"
},
"platformdirs": {
"hashes": [
"sha256:027d8e83a2d7de06bbac4e5ef7e023c02b863d7ea5d079477e722bb41ab25788",
"sha256:58c8abb07dcb441e6ee4b11d8df0ac856038f944ab98b7be6b27b2a3c7feef19"
],
"markers": "python_version >= '3.7'",
"version": "==2.5.2"
},
"pluggy": {
"hashes": [
"sha256:4224373bacce55f955a878bf9cfa763c1e360858e330072059e10bad68531159",
"sha256:74134bbf457f031a36d68416e1509f34bd5ccc019f0bcc952c7b909d06b37bd3"
],
"markers": "python_version >= '3.6'",
"version": "==1.0.0"
},
"py": {
"hashes": [
"sha256:51c75c4126074b472f746a24399ad32f6053d1b34b68d2fa41e558e6f4a98719",
"sha256:607c53218732647dff4acdfcd50cb62615cedf612e72d1724fb1a0cc6405b378"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
"version": "==1.11.0"
},
"pycodestyle": {
"hashes": [
"sha256:720f8b39dde8b293825e7ff02c475f3077124006db4f440dcbc9a20b76548a20",
"sha256:eddd5847ef438ea1c7870ca7eb78a9d47ce0cdb4851a5523949f2601d0cbbe7f"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
"version": "==2.8.0"
},
"pycparser": {
"hashes": [
"sha256:8ee45429555515e1f6b185e78100aea234072576aa43ab53aefcae078162fca9",
"sha256:e644fdec12f7872f86c58ff790da456218b10f863970249516d60a5eaca77206"
],
"version": "==2.21"
},
"pyflakes": {
"hashes": [
"sha256:05a85c2872edf37a4ed30b0cce2f6093e1d0581f8c19d7393122da7e25b2b24c",
"sha256:3bb3a3f256f4b7968c9c788781e4ff07dce46bdf12339dcda61053375426ee2e"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
"version": "==2.4.0"
},
"pygments": {
"hashes": [
"sha256:5eb116118f9612ff1ee89ac96437bb6b49e8f04d8a13b514ba26f620208e26eb",
"sha256:dc9c10fb40944260f6ed4c688ece0cd2048414940f1cea51b8b226318411c519"
],
"markers": "python_version >= '3.6'",
"version": "==2.12.0"
},
"pyparsing": {
"hashes": [
"sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
"sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
],
"markers": "python_full_version >= '3.6.8'",
"version": "==3.0.9"
},
"pytest": {
"hashes": [
"sha256:13d0e3ccfc2b6e26be000cb6568c832ba67ba32e719443bfe725814d3c42433c",
"sha256:a06a0425453864a270bc45e71f783330a7428defb4230fb5e6a731fde06ecd45"
],
"index": "pypi",
"version": "==7.1.2"
},
"pytest-cov": {
"hashes": [
"sha256:578d5d15ac4a25e5f961c938b85a05b09fdaae9deef3bb6de9a6e766622ca7a6",
"sha256:e7f0f5b1617d2210a2cabc266dfe2f4c75a8d32fb89eafb7ad9d06f6d076d470"
],
"index": "pypi",
"version": "==3.0.0"
},
"pytest-sphinx": {
"hashes": [
"sha256:349a55eb44f78860eba8085daa3de56d7f51e537030b4a4a0aaf8cbe5c900428",
"sha256:a9e3018f6abcce1ac5293ac1de96ac0408f3edbf8d095bfd99b641c03a0f3f7f"
],
"index": "pypi",
"version": "==0.4.0"
},
"python-dotenv": {
"hashes": [
"sha256:b7e3b04a59693c42c36f9ab1cc2acc46fa5df8c78e178fc33a8d4cd05c8d498f",
"sha256:d92a187be61fe482e4fd675b6d52200e7be63a12b724abbf931a40ce4fa92938"
],
"index": "pypi",
"version": "==0.20.0"
},
"pytz": {
"hashes": [
"sha256:1e760e2fe6a8163bc0b3d9a19c4f84342afa0a2affebfaa84b01b978a02ecaa7",
"sha256:e68985985296d9a66a881eb3193b0906246245294a881e7c8afe623866ac6a5c"
],
"version": "==2022.1"
},
"pyyaml": {
"hashes": [
"sha256:0283c35a6a9fbf047493e3a0ce8d79ef5030852c51e9d911a27badfde0605293",
"sha256:055d937d65826939cb044fc8c9b08889e8c743fdc6a32b33e2390f66013e449b",
"sha256:07751360502caac1c067a8132d150cf3d61339af5691fe9e87803040dbc5db57",
"sha256:0b4624f379dab24d3725ffde76559cff63d9ec94e1736b556dacdfebe5ab6d4b",
"sha256:0ce82d761c532fe4ec3f87fc45688bdd3a4c1dc5e0b4a19814b9009a29baefd4",
"sha256:1e4747bc279b4f613a09eb64bba2ba602d8a6664c6ce6396a4d0cd413a50ce07",
"sha256:213c60cd50106436cc818accf5baa1aba61c0189ff610f64f4a3e8c6726218ba",
"sha256:231710d57adfd809ef5d34183b8ed1eeae3f76459c18fb4a0b373ad56bedcdd9",
"sha256:277a0ef2981ca40581a47093e9e2d13b3f1fbbeffae064c1d21bfceba2030287",
"sha256:2cd5df3de48857ed0544b34e2d40e9fac445930039f3cfe4bcc592a1f836d513",
"sha256:40527857252b61eacd1d9af500c3337ba8deb8fc298940291486c465c8b46ec0",
"sha256:473f9edb243cb1935ab5a084eb238d842fb8f404ed2193a915d1784b5a6b5fc0",
"sha256:48c346915c114f5fdb3ead70312bd042a953a8ce5c7106d5bfb1a5254e47da92",
"sha256:50602afada6d6cbfad699b0c7bb50d5ccffa7e46a3d738092afddc1f9758427f",
"sha256:68fb519c14306fec9720a2a5b45bc9f0c8d1b9c72adf45c37baedfcd949c35a2",
"sha256:77f396e6ef4c73fdc33a9157446466f1cff553d979bd00ecb64385760c6babdc",
"sha256:819b3830a1543db06c4d4b865e70ded25be52a2e0631ccd2f6a47a2822f2fd7c",
"sha256:897b80890765f037df3403d22bab41627ca8811ae55e9a722fd0392850ec4d86",
"sha256:98c4d36e99714e55cfbaaee6dd5badbc9a1ec339ebfc3b1f52e293aee6bb71a4",
"sha256:9df7ed3b3d2e0ecfe09e14741b857df43adb5a3ddadc919a2d94fbdf78fea53c",
"sha256:9fa600030013c4de8165339db93d182b9431076eb98eb40ee068700c9c813e34",
"sha256:a80a78046a72361de73f8f395f1f1e49f956c6be882eed58505a15f3e430962b",
"sha256:b3d267842bf12586ba6c734f89d1f5b871df0273157918b0ccefa29deb05c21c",
"sha256:b5b9eccad747aabaaffbc6064800670f0c297e52c12754eb1d976c57e4f74dcb",
"sha256:c5687b8d43cf58545ade1fe3e055f70eac7a5a1a0bf42824308d868289a95737",
"sha256:cba8c411ef271aa037d7357a2bc8f9ee8b58b9965831d9e51baf703280dc73d3",
"sha256:d15a181d1ecd0d4270dc32edb46f7cb7733c7c508857278d3d378d14d606db2d",
"sha256:d4db7c7aef085872ef65a8fd7d6d09a14ae91f691dec3e87ee5ee0539d516f53",
"sha256:d4eccecf9adf6fbcc6861a38015c2a64f38b9d94838ac1810a9023a0609e1b78",
"sha256:d67d839ede4ed1b28a4e8909735fc992a923cdb84e618544973d7dfc71540803",
"sha256:daf496c58a8c52083df09b80c860005194014c3698698d1a57cbcfa182142a3a",
"sha256:e61ceaab6f49fb8bdfaa0f92c4b57bcfbea54c09277b1b4f7ac376bfb7a7c174",
"sha256:f84fbc98b019fef2ee9a1cb3ce93e3187a6df0b2538a651bfb890254ba9f90b5"
],
"markers": "python_version >= '3.6'",
"version": "==6.0"
},
"readme-renderer": {
"hashes": [
"sha256:73b84905d091c31f36e50b4ae05ae2acead661f6a09a9abb4df7d2ddcdb6a698",
"sha256:a727999acfc222fc21d82a12ed48c957c4989785e5865807c65a487d21677497"
],
"markers": "python_version >= '3.7'",
"version": "==35.0"
},
"requests": {
"hashes": [
"sha256:bc7861137fbce630f17b03d3ad02ad0bf978c844f3536d0edda6499dafce2b6f",
"sha256:d568723a7ebd25875d8d1eaf5dfa068cd2fc8194b2e483d7b1f7c81918dbec6b"
],
"markers": "python_version >= '3.7' and python_version < '4'",
"version": "==2.28.0"
},
"requests-toolbelt": {
"hashes": [
"sha256:380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f",
"sha256:968089d4584ad4ad7c171454f0a5c6dac23971e9472521ea3b6d49d610aa6fc0"
],
"version": "==0.9.1"
},
"rfc3986": {
"hashes": [
"sha256:50b1502b60e289cb37883f3dfd34532b8873c7de9f49bb546641ce9cbd256ebd",
"sha256:97aacf9dbd4bfd829baad6e6309fa6573aaf1be3f6fa735c8ab05e46cecb261c"
],
"markers": "python_version >= '3.7'",
"version": "==2.0.0"
},
"rich": {
"hashes": [
"sha256:4c586de507202505346f3e32d1363eb9ed6932f0c2f63184dea88983ff4971e2",
"sha256:d2bbd99c320a2532ac71ff6a3164867884357da3e3301f0240090c5d2fdac7ec"
],
"markers": "python_version < '4' and python_full_version >= '3.6.3'",
"version": "==12.4.4"
},
"secretstorage": {
"hashes": [
"sha256:0a8eb9645b320881c222e827c26f4cfcf55363e8b374a021981ef886657a912f",
"sha256:755dc845b6ad76dcbcbc07ea3da75ae54bb1ea529eb72d15f83d26499a5df319"
],
"markers": "sys_platform == 'linux'",
"version": "==3.3.2"
},
"six": {
"hashes": [
"sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
"sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
"version": "==1.16.0"
},
"snowballstemmer": {
"hashes": [
"sha256:09b16deb8547d3412ad7b590689584cd0fe25ec8db3be37788be3810cbf19cb1",
"sha256:c8e1716e83cc398ae16824e5572ae04e0d9fc2c6b985fb0f900f5f0c96ecba1a"
],
"version": "==2.2.0"
},
"soupsieve": {
"hashes": [
"sha256:3b2503d3c7084a42b1ebd08116e5f81aadfaea95863628c80a3b774a11b7c759",
"sha256:fc53893b3da2c33de295667a0e19f078c14bf86544af307354de5fcf12a3f30d"
],
"markers": "python_version >= '3.6'",
"version": "==2.3.2.post1"
},
"sphinx": {
"hashes": [
"sha256:b18e978ea7565720f26019c702cd85c84376e948370f1cd43d60265010e1c7b0",
"sha256:d3e57663eed1d7c5c50895d191fdeda0b54ded6f44d5621b50709466c338d1e8"
],
"index": "pypi",
"version": "==5.0.2"
},
"sphinx-autobuild": {
"hashes": [
"sha256:8fe8cbfdb75db04475232f05187c776f46f6e9e04cacf1e49ce81bdac649ccac",
"sha256:de1ca3b66e271d2b5b5140c35034c89e47f263f2cd5db302c9217065f7443f05"
],
"index": "pypi",
"version": "==2021.3.14"
},
"sphinx-autodoc-typehints": {
"hashes": [
"sha256:20294de2a818bda04953c5cb302ec5af46138c81980ad9efa6d8fc1fc4242518",
"sha256:c04d8f8d70e988960e25b206af39a90df84e7e2c085bb24e123bc3684021b313"
],
"index": "pypi",
"version": "==1.18.3"
},
"sphinx-basic-ng": {
"hashes": [
"sha256:9aecb5345816998789ef76658a83e3c0a12aafa14b17d40e28cd4aaeb94d1517",
"sha256:bf9a8fda0379c7d2ab51c9543f2b18e014b77fb295b49d64f3c1a910c863b34f"
],
"markers": "python_version >= '3.7'",
"version": "==0.0.1a11"
},
"sphinx-copybutton": {
"hashes": [
"sha256:9684dec7434bd73f0eea58dda93f9bb879d24bff2d8b187b1f2ec08dfe7b5f48",
"sha256:a0c059daadd03c27ba750da534a92a63e7a36a7736dcf684f26ee346199787f6"
],
"index": "pypi",
"version": "==0.5.0"
},
"sphinxcontrib-applehelp": {
"hashes": [
"sha256:806111e5e962be97c29ec4c1e7fe277bfd19e9652fb1a4392105b43e01af885a",
"sha256:a072735ec80e7675e3f432fcae8610ecf509c5f1869d17e2eecff44389cdbc58"
],
"markers": "python_version >= '3.5'",
"version": "==1.0.2"
},
"sphinxcontrib-devhelp": {
"hashes": [
"sha256:8165223f9a335cc1af7ffe1ed31d2871f325254c0423bc0c4c7cd1c1e4734a2e",
"sha256:ff7f1afa7b9642e7060379360a67e9c41e8f3121f2ce9164266f61b9f4b338e4"
],
"markers": "python_version >= '3.5'",
"version": "==1.0.2"
},
"sphinxcontrib-htmlhelp": {
"hashes": [
"sha256:d412243dfb797ae3ec2b59eca0e52dac12e75a241bf0e4eb861e450d06c6ed07",
"sha256:f5f8bb2d0d629f398bf47d0d69c07bc13b65f75a81ad9e2f71a63d4b7a2f6db2"
],
"markers": "python_version >= '3.6'",
"version": "==2.0.0"
},
"sphinxcontrib-jsmath": {
"hashes": [
"sha256:2ec2eaebfb78f3f2078e73666b1415417a116cc848b72e5172e596c871103178",
"sha256:a9925e4a4587247ed2191a22df5f6970656cb8ca2bd6284309578f2153e0c4b8"
],
"markers": "python_version >= '3.5'",
"version": "==1.0.1"
},
"sphinxcontrib-qthelp": {
"hashes": [
"sha256:4c33767ee058b70dba89a6fc5c1892c0d57a54be67ddd3e7875a18d14cba5a72",
"sha256:bd9fc24bcb748a8d51fd4ecaade681350aa63009a347a8c14e637895444dfab6"
],
"markers": "python_version >= '3.5'",
"version": "==1.0.3"
},
"sphinxcontrib-serializinghtml": {
"hashes": [
"sha256:352a9a00ae864471d3a7ead8d7d79f5fc0b57e8b3f95e9867eb9eb28999b92fd",
"sha256:aa5f6de5dfdf809ef505c4895e51ef5c9eac17d0f287933eb49ec495280b6952"
],
"markers": "python_version >= '3.5'",
"version": "==1.1.5"
},
"tomli": {
"hashes": [
"sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc",
"sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f"
],
"markers": "python_version < '3.11'",
"version": "==2.0.1"
},
"tornado": {
"hashes": [
"sha256:0a00ff4561e2929a2c37ce706cb8233b7907e0cdc22eab98888aca5dd3775feb",
"sha256:0d321a39c36e5f2c4ff12b4ed58d41390460f798422c4504e09eb5678e09998c",
"sha256:1e8225a1070cd8eec59a996c43229fe8f95689cb16e552d130b9793cb570a288",
"sha256:20241b3cb4f425e971cb0a8e4ffc9b0a861530ae3c52f2b0434e6c1b57e9fd95",
"sha256:25ad220258349a12ae87ede08a7b04aca51237721f63b1808d39bdb4b2164558",
"sha256:33892118b165401f291070100d6d09359ca74addda679b60390b09f8ef325ffe",
"sha256:33c6e81d7bd55b468d2e793517c909b139960b6c790a60b7991b9b6b76fb9791",
"sha256:3447475585bae2e77ecb832fc0300c3695516a47d46cefa0528181a34c5b9d3d",
"sha256:34ca2dac9e4d7afb0bed4677512e36a52f09caa6fded70b4e3e1c89dbd92c326",
"sha256:3e63498f680547ed24d2c71e6497f24bca791aca2fe116dbc2bd0ac7f191691b",
"sha256:548430be2740e327b3fe0201abe471f314741efcb0067ec4f2d7dcfb4825f3e4",
"sha256:6196a5c39286cc37c024cd78834fb9345e464525d8991c21e908cc046d1cc02c",
"sha256:61b32d06ae8a036a6607805e6720ef00a3c98207038444ba7fd3d169cd998910",
"sha256:6286efab1ed6e74b7028327365cf7346b1d777d63ab30e21a0f4d5b275fc17d5",
"sha256:65d98939f1a2e74b58839f8c4dab3b6b3c1ce84972ae712be02845e65391ac7c",
"sha256:66324e4e1beede9ac79e60f88de548da58b1f8ab4b2f1354d8375774f997e6c0",
"sha256:6c77c9937962577a6a76917845d06af6ab9197702a42e1346d8ae2e76b5e3675",
"sha256:70dec29e8ac485dbf57481baee40781c63e381bebea080991893cd297742b8fd",
"sha256:7250a3fa399f08ec9cb3f7b1b987955d17e044f1ade821b32e5f435130250d7f",
"sha256:748290bf9112b581c525e6e6d3820621ff020ed95af6f17fedef416b27ed564c",
"sha256:7da13da6f985aab7f6f28debab00c67ff9cbacd588e8477034c0652ac141feea",
"sha256:8f959b26f2634a091bb42241c3ed8d3cedb506e7c27b8dd5c7b9f745318ddbb6",
"sha256:9de9e5188a782be6b1ce866e8a51bc76a0fbaa0e16613823fc38e4fc2556ad05",
"sha256:a48900ecea1cbb71b8c71c620dee15b62f85f7c14189bdeee54966fbd9a0c5bd",
"sha256:b87936fd2c317b6ee08a5741ea06b9d11a6074ef4cc42e031bc6403f82a32575",
"sha256:c77da1263aa361938476f04c4b6c8916001b90b2c2fdd92d8d535e1af48fba5a",
"sha256:cb5ec8eead331e3bb4ce8066cf06d2dfef1bfb1b2a73082dfe8a161301b76e37",
"sha256:cc0ee35043162abbf717b7df924597ade8e5395e7b66d18270116f8745ceb795",
"sha256:d14d30e7f46a0476efb0deb5b61343b1526f73ebb5ed84f23dc794bdb88f9d9f",
"sha256:d371e811d6b156d82aa5f9a4e08b58debf97c302a35714f6f45e35139c332e32",
"sha256:d3d20ea5782ba63ed13bc2b8c291a053c8d807a8fa927d941bd718468f7b950c",
"sha256:d3f7594930c423fd9f5d1a76bee85a2c36fd8b4b16921cae7e965f22575e9c01",
"sha256:dcef026f608f678c118779cd6591c8af6e9b4155c44e0d1bc0c87c036fb8c8c4",
"sha256:e0791ac58d91ac58f694d8d2957884df8e4e2f6687cdf367ef7eb7497f79eaa2",
"sha256:e385b637ac3acaae8022e7e47dfa7b83d3620e432e3ecb9a3f7f58f150e50921",
"sha256:e519d64089b0876c7b467274468709dadf11e41d65f63bba207e04217f47c085",
"sha256:e7229e60ac41a1202444497ddde70a48d33909e484f96eb0da9baf8dc68541df",
"sha256:ed3ad863b1b40cd1d4bd21e7498329ccaece75db5a5bf58cd3c9f130843e7102",
"sha256:f0ba29bafd8e7e22920567ce0d232c26d4d47c8b5cf4ed7b562b5db39fa199c5",
"sha256:fa2ba70284fa42c2a5ecb35e322e68823288a4251f9ba9cc77be04ae15eada68",
"sha256:fba85b6cd9c39be262fcd23865652920832b61583de2a2ca907dbd8e8a8c81e5"
],
"markers": "python_version > '2.7'",
"version": "==6.1"
},
"twine": {
"hashes": [
"sha256:42026c18e394eac3e06693ee52010baa5313e4811d5a11050e7d48436cf41b9e",
"sha256:96b1cf12f7ae611a4a40b6ae8e9570215daff0611828f5fe1f37a16255ab24a0"
],
"index": "pypi",
"version": "==4.0.1"
},
"typing-extensions": {
"hashes": [
"sha256:6657594ee297170d19f67d55c05852a874e7eb634f4f753dbd667855e07c1708",
"sha256:f1c24655a0da0d1b67f07e17a5e6b2a105894e6824b92096378bb3668ef02376"
],
"markers": "python_version < '3.10'",
"version": "==4.2.0"
},
"urllib3": {
"hashes": [
"sha256:44ece4d53fb1706f667c9bd1c648f5469a2ec925fcf3a776667042d645472c14",
"sha256:aabaf16477806a5e1dd19aa41f8c2b7950dd3c746362d7e3223dbe6de6ac448e"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4'",
"version": "==1.26.9"
},
"webencodings": {
"hashes": [
"sha256:a0af1213f3c2226497a97e2b3aa01a7e4bee4f403f95be16fc9acd2947514a78",
"sha256:b36a1c245f2d304965eb4e0a82848379241dc04b865afcc4aab16748587e1923"
],
"version": "==0.5.1"
},
"zipp": {
"hashes": [
"sha256:56bf8aadb83c24db6c4b577e13de374ccfb67da2078beba1d037c17980bf43ad",
"sha256:c4f6e5bbf48e74f7a38e7cc5b0480ff42b0ae5178957d564d18932525d5cf099"
],
"markers": "python_version >= '3.7'",
"version": "==3.8.0"
}
}
}

111
README.md
View File

@@ -1,110 +1 @@
# python-package-template # vk-url-scraper
This is a template repository for Python package projects.
## In this README :point_down:
- [Features](#features)
- [Usage](#usage)
- [Initial setup](#initial-setup)
- [Creating releases](#creating-releases)
- [Projects using this template](#projects-using-this-template)
- [FAQ](#faq)
- [Contributing](#contributing)
## Features
This template repository comes with all of the boilerplate needed for:
⚙️ Robust (and free) CI with [GitHub Actions](https://github.com/features/actions):
- Unit tests ran with [PyTest](https://docs.pytest.org) against multiple Python versions and operating systems.
- Type checking with [mypy](https://github.com/python/mypy).
- Linting with [flake8](https://flake8.pycqa.org/en/latest/).
- Formatting with [isort](https://pycqa.github.io/isort/) and [black](https://black.readthedocs.io/en/stable/).
🤖 [Dependabot](https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/) configuration to keep your dependencies up-to-date.
📄 Great looking API documentation built using [Sphinx](https://www.sphinx-doc.org/en/master/) (run `make docs` to preview).
🚀 Automatic GitHub and PyPI releases. Just follow the steps in [`RELEASE_PROCESS.md`](./RELEASE_PROCESS.md) to trigger a new release.
## Usage
### Initial setup
1. [Create a new repository](https://github.com/allenai/python-package-template/generate) from this template with the desired name of your project.
*Your project name (i.e. the name of the repository) and the name of the corresponding Python package don't necessarily need to match, but you might want to check on [PyPI](https://pypi.org/) first to see if the package name you want is already taken.*
2. Create a Python 3.7 or newer virtual environment.
*If you're not sure how to create a suitable Python environment, the easiest way is using [Miniconda](https://docs.conda.io/en/latest/miniconda.html). On a Mac, for example, you can install Miniconda using [Homebrew](https://brew.sh/):*
```
brew install miniconda
```
*Then you can create and activate a new Python environment by running:*
```
conda create -n my-package python=3.9
conda activate my-package
```
3. Now that you have a suitable Python environment, you're ready to personalize this repository. Just run:
```
pip install -r setup-requirements.txt
python scripts/personalize.py
```
And then follow the prompts.
:pencil: *NOTE: This script will overwrite the README in your repository.*
4. Commit and push your changes, then make sure all GitHub Actions jobs pass.
5. (Optional) If you plan on publishing your package to PyPI, add repository secrets for `PYPI_USERNAME` and `PYPI_PASSWORD`. To add these, go to "Settings" > "Secrets" > "Actions", and then click "New repository secret".
*If you don't have PyPI account yet, you can [create one for free](https://pypi.org/account/register/).*
6. (Optional) If you want to deploy your API docs to [readthedocs.org](https://readthedocs.org), go to the [readthedocs dashboard](https://readthedocs.org/dashboard/import/?) and import your new project.
Then click on the "Admin" button, navigate to "Automation Rules" in the sidebar, click "Add Rule", and then enter the following fields:
- **Description:** Publish new versions from tags
- **Match:** Custom Match
- **Custom match:** v[vV]
- **Version:** Tag
- **Action:** Activate version
Then hit "Save".
*After your first release, the docs will automatically be published to [your-project-name.readthedocs.io](https://your-project-name.readthedocs.io/).*
### Creating releases
Creating new GitHub and PyPI releases is easy. The GitHub Actions workflow that comes with this repository will handle all of that for you.
All you need to do is follow the instructions in [RELEASE_PROCESS.md](./RELEASE_PROCESS.md).
## Projects using this template
Here is an incomplete list of some projects that started off with this template:
- [ai2-tango](https://github.com/allenai/tango)
- [cached-path](https://github.com/allenai/cached_path)
- [beaker-py](https://github.com/allenai/beaker-py)
- [gantry](https://github.com/allenai/beaker-gantry)
☝️ *Want your work featured here? Just open a pull request that adds the link.*
## FAQ
#### Should I use this template even if I don't want to publish my package?
Absolutely! If you don't want to publish your package, just delete the `docs/` directory and the `release` job in [`.github/workflows/main.yml`](https://github.com/allenai/python-package-template/blob/main/.github/workflows/main.yml).
## Contributing
If you find a bug :bug:, please open a [bug report](https://github.com/allenai/python-package-template/issues/new?assignees=&labels=bug&template=bug_report.md&title=).
If you have an idea for an improvement or new feature :rocket:, please open a [feature request](https://github.com/allenai/python-package-template/issues/new?assignees=&labels=Feature+request&template=feature_request.md&title=).

2
RELEASE_PROCESS.md
View File

@@ -2,7 +2,7 @@
## Steps ## Steps
1. Update the version in `my_package/version.py`. 1. Update the version in `vk_url_scraper/version.py`.
3. Run the release script: 3. Run the release script:

15
docs/source/conf.py
View File

@@ -17,14 +17,15 @@ from datetime import datetime
# #
sys.path.insert(0, os.path.abspath("../../")) sys.path.insert(0, os.path.abspath("../../"))
sys.path.insert(0, os.path.abspath("../../vk_url_scraper"))
from my_package.version import VERSION, VERSION_SHORT # noqa: E402 from vk_url_scraper.version import VERSION, VERSION_SHORT # noqa: E402
# -- Project information ----------------------------------------------------- # -- Project information -----------------------------------------------------
project = "my-package" project = "vk-url-scraper"
copyright = f"{datetime.today().year}, Allen Institute for Artificial Intelligence" copyright = f"{datetime.today().year}, Bellingcat"
author = "Allen Institute for Artificial Intelligence" author = "Bellingcat"
version = VERSION_SHORT version = VERSION_SHORT
release = VERSION release = VERSION
@@ -82,7 +83,7 @@ typehints_defaults = "comma"
# #
html_theme = "furo" html_theme = "furo"
html_title = f"my-package v{VERSION}" html_title = f"vk-url-scraper v{VERSION}"
# Add any paths that contain custom static files (such as style sheets) here, # Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files, # relative to this directory. They are copied after the builtin static files,
@@ -97,7 +98,7 @@ html_theme_options = {
"footer_icons": [ "footer_icons": [
{ {
"name": "GitHub", "name": "GitHub",
"url": "https://github.com/allenai/python-package-template", "url": "https://github.com/bellingcat/vk-url-scraper",
"html": """ "html": """
<svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 16 16"> <svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 16 16">
<path fill-rule="evenodd" d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0 0 16 8c0-4.42-3.58-8-8-8z"></path> <path fill-rule="evenodd" d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0 0 16 8c0-4.42-3.58-8-8-8z"></path>
@@ -108,7 +109,7 @@ html_theme_options = {
], ],
} }
# -- Hack to get rid of stupid warnings from sphinx_autodoc_typehints -------- # -- Hack to get rid of warnings from sphinx_autodoc_typehints --------
class ShutupSphinxAutodocTypehintsFilter(logging.Filter): class ShutupSphinxAutodocTypehintsFilter(logging.Filter):

28
docs/source/index.rst
View File

@@ -1,12 +1,18 @@
.. my_package documentation master file, created by .. vk_url_scraper documentation master file, created by
sphinx-quickstart on Tue Sep 21 08:07:48 2021. sphinx-quickstart on Tue Sep 21 08:07:48 2021.
You can adapt this file completely to your liking, but it should at least You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive. contain the root `toctree` directive.
**my-package** **vk-url-scraper**
=============== ==================
.. automodule:: vk_url_scraper
:imported-members:
.. automethod:: __init__
:members:
:undoc-members:
:show-inheritance:
.. automodule:: my_package
Contents Contents
-------- --------
@@ -23,24 +29,22 @@ Contents
:hidden: :hidden:
:caption: Development :caption: Development
License <https://raw.githubusercontent.com/allenai/python-package-template/main/LICENSE> License <https://raw.githubusercontent.com/bellingcat/vk-url-scraper/main/LICENSE>
CONTRIBUTING CONTRIBUTING
GitHub Repository <https://github.com/allenai/python-package-template> GitHub Repository <https://github.com/bellingcat/vk-url-scraper>
Team Team
---- ----
**my-package** is developed and maintained by the AllenNLP team, backed by **vk-url-scraper** is developed and maintained by the Bellingcat Tech Team.
`the Allen Institute for Artificial Intelligence (AI2) <https://allenai.org/>`_.
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.
To learn more about who specifically contributed to this codebase, see To learn more about who specifically contributed to this codebase, see
`our contributors <https://github.com/allenai/python-package-template/graphs/contributors>`_ page. `our contributors <https://github.com/bellingcat/vk-url-scraper/graphs/contributors>`_ page.
License License
------- -------
**my-package** is licensed under `Apache 2.0 <https://www.apache.org/licenses/LICENSE-2.0>`_. **vk-url-scraper** is licensed under the MIT license.
A full copy of the license can be found `on GitHub <https://github.com/allenai/python-package-template/blob/main/LICENSE>`_. A full copy of the license can be found `on GitHub <https://github.com/bellingcat/vk-url-scraper/blob/main/LICENSE>`_.
Indices and tables Indices and tables
------------------ ------------------

12
docs/source/installation.md
View File

@@ -1,23 +1,23 @@
Installation Installation
============ ============
**my-package** supports Python >= 3.7. **vk-url-scraper** supports Python >= 3.7.
## Installing with `pip` ## Installing with `pip`
**my-package** is available [on PyPI](https://pypi.org/project/my-package/). Just run **vk-url-scraper** is available [on PyPI](https://pypi.org/project/vk-url-scraper/). Just run
```bash ```bash
pip install my-package pip install vk-url-scraper
``` ```
## Installing from source ## Installing from source
To install **my-package** from source, first clone [the repository](https://github.com/allenai/python-package-template): To install **vk-url-scraper** from source, first clone [the repository](https://github.com/bellingcat/vk-url-scraper):
```bash ```bash
git clone https://github.com/allenai/python-package-template.git git clone https://github.com/bellingcat/vk-url-scraper.git
cd python-package-template cd vk-url-scraper
``` ```
Then run Then run

0
my_package/__init__.py
View File

20
requirements.txt
View File

@@ -1 +1,19 @@
# Add your own dependencies to this file. #
# These requirements were autogenerated by pipenv
# To regenerate from the project's Pipfile, run:
#
# pipenv lock --requirements
#
-i https://pypi.org/simple
certifi==2022.6.15; python_version >= '3.6'
charset-normalizer==2.0.12; python_version >= '3.5'
click-help-colors==0.9.1
click==8.1.3
commonmark==0.9.1
idna==3.3; python_version >= '3.5'
pygments==2.12.0; python_version >= '3.6'
requests==2.28.0; python_version >= '3.7' and python_version < '4'
rich==12.4.4
urllib3==1.26.9; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4'
vk-api==11.9.8

175
scripts/personalize.py
View File

@@ -1,175 +0,0 @@
"""
Run this script once after first creating your project from this template repo to personalize
it for own project.
This script is interactive and will prompt you for various inputs.
"""
from pathlib import Path
from typing import Generator, List, Tuple
import click
from click_help_colors import HelpColorsCommand
from rich import print
from rich.markdown import Markdown
from rich.prompt import Confirm
from rich.syntax import Syntax
from rich.traceback import install
install(show_locals=True, suppress=[click])
REPO_BASE = (Path(__file__).parent / "..").resolve()
FILES_TO_REMOVE = {
REPO_BASE / ".github" / "workflows" / "setup.yml",
REPO_BASE / "setup-requirements.txt",
REPO_BASE / "scripts" / "personalize.py",
}
PATHS_TO_IGNORE = {
REPO_BASE / "README.md",
REPO_BASE / ".git",
REPO_BASE / "docs" / "source" / "_static" / "favicon.ico",
}
GITIGNORE_LIST = [
line.strip()
for line in (REPO_BASE / ".gitignore").open().readlines()
if line.strip() and not line.startswith("#")
]
REPO_NAME_TO_REPLACE = "python-package-template"
BASE_URL_TO_REPLACE = "https://github.com/allenai/python-package-template"
@click.command(
cls=HelpColorsCommand,
help_options_color="green",
help_headers_color="yellow",
context_settings={"max_content_width": 115},
)
@click.option(
"--github-org",
prompt="GitHub organization or user (e.g. 'allenai')",
help="The name of your GitHub organization or user.",
)
@click.option(
"--github-repo",
prompt="GitHub repository (e.g. 'python-package-template')",
help="The name of your GitHub repository.",
)
@click.option(
"--package-name",
prompt="Python package name (e.g. 'my-package')",
help="The name of your Python package.",
)
@click.option(
"-y",
"--yes",
is_flag=True,
help="Run the script without prompting for a confirmation.",
default=False,
)
@click.option(
"--dry-run",
is_flag=True,
hidden=True,
default=False,
)
def main(
github_org: str, github_repo: str, package_name: str, yes: bool = False, dry_run: bool = False
):
repo_url = f"https://github.com/{github_org}/{github_repo}"
package_actual_name = package_name.replace("_", "-")
package_dir_name = package_name.replace("-", "_")
# Confirm before continuing.
print(f"Repository URL set to: [link={repo_url}]{repo_url}[/]")
print(f"Package name set to: [cyan]{package_actual_name}[/]")
if not yes:
yes = Confirm.ask("Is this correct?")
if not yes:
raise click.ClickException("Aborted, please run script again")
# Delete files that we don't need.
for path in FILES_TO_REMOVE:
assert path.is_file(), path
if not dry_run:
path.unlink()
else:
print(f"Removing {path}")
# Personalize remaining files.
replacements = [
(BASE_URL_TO_REPLACE, repo_url),
(REPO_NAME_TO_REPLACE, github_repo),
("my-package", package_actual_name),
("my_package", package_dir_name),
]
if dry_run:
for old, new in replacements:
print(f"Replacing '{old}' with '{new}'")
for path in iterfiles(REPO_BASE):
personalize_file(path, dry_run, replacements)
# Rename 'my_package' directory to `package_dir_name`.
if not dry_run:
(REPO_BASE / "my_package").replace(REPO_BASE / package_dir_name)
else:
print(f"Renaming 'my_package' directory to '{package_dir_name}'")
# Start with a fresh README.
readme_contents = f"""# {package_actual_name}\n"""
if not dry_run:
with open(REPO_BASE / "README.md", "w+t") as readme_file:
readme_file.write(readme_contents)
else:
print("Replacing README.md contents with:\n", Markdown(readme_contents))
install_example = Syntax("pip install -e '.[dev]'", "bash")
print(
"[green]\N{check mark} Success![/] You can now install your package locally in development mode with:\n",
install_example,
)
def iterfiles(dir: Path) -> Generator[Path, None, None]:
assert dir.is_dir()
for path in dir.iterdir():
if path in PATHS_TO_IGNORE:
continue
is_ignored_file = False
for gitignore_entry in GITIGNORE_LIST:
if path.relative_to(REPO_BASE).match(gitignore_entry):
is_ignored_file = True
break
if is_ignored_file:
continue
if path.is_dir():
yield from iterfiles(path)
else:
yield path
def personalize_file(path: Path, dry_run: bool, replacements: List[Tuple[str, str]]):
with path.open("r+t") as file:
filedata = file.read()
should_update: bool = False
for old, new in replacements:
if filedata.count(old):
should_update = True
filedata = filedata.replace(old, new)
if should_update:
if not dry_run:
with path.open("w+t") as file:
file.write(filedata)
else:
print(f"Updating {path}")
if __name__ == "__main__":
main()

4
scripts/prepare_changelog.py
View File

@@ -1,7 +1,7 @@
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
from my_package.version import VERSION from vk_url_scraper.version import VERSION
def main(): def main():
@@ -27,7 +27,7 @@ def main():
lines.insert(insert_index, "\n") lines.insert(insert_index, "\n")
lines.insert( lines.insert(
insert_index + 1, insert_index + 1,
f"## [v{VERSION}](https://github.com/allenai/python-package-template/releases/tag/v{VERSION}) - " f"## [v{VERSION}](https://github.com/bellingcat/vk-url-scraper/releases/tag/v{VERSION}) - "
f"{datetime.now().strftime('%Y-%m-%d')}\n", f"{datetime.now().strftime('%Y-%m-%d')}\n",
) )

2
scripts/release.sh
View File

@@ -2,7 +2,7 @@
set -e set -e
TAG=$(python -c 'from my_package.version import VERSION; print("v" + VERSION)') TAG=$(python -c 'from vk_url_scraper.version import VERSION; print("v" + VERSION)')
read -p "Creating new release for $TAG. Do you want to continue? [Y/n] " prompt read -p "Creating new release for $TAG. Do you want to continue? [Y/n] " prompt

3
setup-requirements.txt
View File

@@ -1,3 +0,0 @@
click>=7.0,<9.0
click-help-colors>=0.9.1,<0.10
rich>=11.0,<13.0

16
setup.py
View File

@@ -27,31 +27,31 @@ def read_requirements(filename: str):
# version.py defines the VERSION and VERSION_SHORT variables. # version.py defines the VERSION and VERSION_SHORT variables.
# We use exec here so we don't import cached_path whilst setting up. # We use exec here so we don't import cached_path whilst setting up.
VERSION = {} # type: ignore VERSION = {} # type: ignore
with open("my_package/version.py", "r") as version_file: with open("vk_url_scraper/version.py", "r") as version_file:
exec(version_file.read(), VERSION) exec(version_file.read(), VERSION)
setup( setup(
name="my-package", name="vk-url-scraper",
version=VERSION["VERSION"], version=VERSION["VERSION"],
description="", description="",
long_description=open("README.md").read(), long_description=open("README.md").read(),
long_description_content_type="text/markdown", long_description_content_type="text/markdown",
classifiers=[ classifiers=[
"Intended Audience :: Developers",
"Intended Audience :: Science/Research", "Intended Audience :: Science/Research",
"Development Status :: 3 - Alpha", "Development Status :: 3 - Alpha",
"License :: OSI Approved :: Apache Software License", "License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3", "Programming Language :: Python :: 3",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
], ],
keywords="", keywords="",
url="https://github.com/allenai/python-package-template", url="https://github.com/bellingcat/vk-url-scraper",
author="Allen Institute for Artificial Intelligence", author="Bellingcat",
author_email="contact@allenai.org", author_email="tech@bellingcat.com",
license="Apache", license="Apache",
packages=find_packages( packages=find_packages(
exclude=["*.tests", "*.tests.*", "tests.*", "tests"], exclude=["*.tests", "*.tests.*", "tests.*", "tests"],
), ),
package_data={"my_package": ["py.typed"]}, package_data={"vk_url_scraper": ["py.typed"]},
install_requires=read_requirements("requirements.txt"), install_requires=read_requirements("requirements.txt"),
extras_require={"dev": read_requirements("dev-requirements.txt")}, extras_require={"dev": read_requirements("dev-requirements.txt")},
python_requires=">=3.7", python_requires=">=3.7",

3
tests/__init__.py
View File

@@ -0,0 +1,3 @@
from dotenv import load_dotenv
load_dotenv()

2
tests/hello_test.py
View File

@@ -1,2 +0,0 @@
def test_hello():
print("Hello, World!")

95
tests/scraper_test.py Normal file
View File

@@ -0,0 +1,95 @@
import os, pytest, datetime
from vk_url_scraper import VkScraper
from .util import assert_equal_lists
vks = None
def test_login_fail():
with pytest.raises(Exception) as exc_info:
VkScraper("invalid", "combination")
def test_login_success():
global vks
vks = VkScraper(os.getenv("VK_USERNAME"), os.getenv("VK_PASSWORD"))
def test_scrape_empty_urll():
assert [] == vks.scrape("something")
def test_scrape_wall_url_with_text_only():
res = vks.scrape("https://vk.com/wall-1_398461")
assert len(res) == 1
assert res[0]["id"] == "wall-1_398461"
assert res[0]["text"] == "[https://vk.com/wall-1_394596|Ранее] мы писали о жизненном цикле версий: vk.com/dev/constant_version_updates. Например, от поддержки версии API 5.50 должны были отказаться 1 сентября прошлого года, а от версии 5.80 — 14 октября. \n\nОбстоятельства сложились иначе — время отказаться от старых версий пришло только сейчас.\n\nС 19 августа 2021 года закончится срок жизни версий ниже 5.41.\nС 26 августа 2021 года перестанут поддерживаться версии ниже 5.61.\nСо 2 сентября 2021 года прекратится поддержка версий ниже 5.81.\n\nПожалуйста, успейте подготовиться к изменениям и убедиться, что в ваших приложениях ничего не сломается. Напомним, что с повышением версии у запросов может измениться формат ответов. Обо всех таких изменениях мы пишем [https://vk.com/dev/versions|здесь]."
assert res[0]["datetime"] == datetime.datetime(2021, 8, 6, 13, 32, 26)
assert len(res[0]["attachments"]) == 0
def test_scrape_wall_url_with_one_photo():
res = vks.scrape("https://vk.com/wall-1_399495")
assert len(res) == 1
assert res[0]["id"] == "wall-1_399495"
assert res[0]["text"] == "Делимся расписанием конкурса [https://vk.com/wall-1_399468|«Код Петербурга»]. Все важные этапы — на одной схеме \n\nЕсли участвуете, обязательно сохраните себе. Так будет удобнее планировать работу над проектом, и вы точно не упустите лучший момент для отправки сервиса на модерацию."
assert res[0]["datetime"] == datetime.datetime(2022, 6, 8, 13, 42)
assert len(res[0]["attachments"]) == 1
assert len(res[0]["attachments"].keys()) == 1
assert list(res[0]["attachments"].keys()) == ["photo"]
def test_scrape_wall_url_with_photos():
res = vks.scrape("https://vk.com/wall-120027872_473324")
assert len(res) == 1
assert res[0]["id"] == "wall-120027872_473324"
assert res[0]["text"] == "Хабаровск\nАллея героев\nПомолимся об укокоении воинов:\nАлександра, Игоря, Эдуарда, \nДионисия, Евгения, Александра, Артемия, Иннокентия, Андрея."
assert res[0]["datetime"] == datetime.datetime(2022, 6, 15, 12, 37, 24)
assert len(res[0]["payload"]) == 16
assert len(res[0]["attachments"].keys()) == 1
assert list(res[0]["attachments"].keys()) == ["photo"]
assert len(res[0]["attachments"]["photo"]) == 9
def test_scrape_wall_url_with_photos_inner_videos_and_links_with_inner_photos():
res = vks.scrape("https://vk.com/asdasdasd?w=wall-17315087_74182")
assert len(res) == 1
assert res[0]["id"] == "wall-17315087_74182"
assert res[0]["text"] == ""
assert res[0]["datetime"] == datetime.datetime(2022, 3, 24, 12, 1, 9)
assert len(res[0]["payload"]) == 15
assert len(res[0]["attachments"].keys()) == 3
assert_equal_lists(
list(res[0]["attachments"].keys()),
["photo", "link", "video"]
)
assert len(res[0]["attachments"]["photo"]) == 5
assert len(res[0]["attachments"]["link"]) == 1
assert len(res[0]["attachments"]["video"]) == 1
def test_scrape_photo_only():
res = vks.scrape("https://vk.com/apiclub?z=photo-1_457242435%2Falbum-1_00%2Frev")
assert len(res) == 1
assert res[0]["id"] == "photo-1_457242435"
assert res[0]["text"] == "Делимся расписанием конкурса [https://vk.com/wall-1_399468|«Код Петербурга»]. Все важные этапы — на одной схеме \n\nЕсли участвуете, обязательно сохраните себе. Так будет удобнее планировать работу над проектом, и вы точно не упустите лучший момент для отправки сервиса на модерацию."
assert res[0]["datetime"] == datetime.datetime(2022, 6, 7, 11, 43)
assert len(res[0]["payload"]) == 15
assert len(res[0]["attachments"].keys()) == 1
assert list(res[0]["attachments"].keys()) == ["photo"]
assert len(res[0]["attachments"]["photo"]) == 1
def test_scrape_video_only():
res = vks.scrape("https://vk.com/video38556806_456251917?list=ba2b77043648ff3789")
assert len(res) == 1
assert res[0]["id"] == "video38556806_456251917"
assert res[0]["datetime"] == datetime.datetime(2022, 3, 24, 6, 42, 38)
assert len(res[0]["payload"]) == 31
assert len(res[0]["attachments"].keys()) == 1
assert list(res[0]["attachments"].keys()) == ["video"]
assert 'G4YDIOBUGQ3DKMQ' in res[0]["attachments"]["video"][0]
def test_scrape_video_only2():
res = vks.scrape("https://vk.com/video-1_456239018")
print(res[0]["attachments"]["video"][0])

3
tests/util.py Normal file
View File

@@ -0,0 +1,3 @@
def assert_equal_lists(l1, l2):
assert len(l1) == len(l2)
assert str(sorted(l1)) == str(sorted(l2))

1
vk_url_scraper/__init__.py Normal file
View File

@@ -0,0 +1 @@
from .scraper import VkScraper

0
my_package/py.typed → vk_url_scraper/py.typed
View File

162
vk_url_scraper/scraper.py Normal file
View File

@@ -0,0 +1,162 @@
import re, requests
import vk_api # used to get api_token after authentication
from typing import List
from datetime import datetime
from collections import defaultdict
class VkScraper:
WALL_PATTERN = re.compile(r"(wall.{0,1}\d+_\d+)")
PHOTO_PATTERN = re.compile(r"(photo.{0,1}\d+_\d+)")
VIDEO_PATTERN = re.compile(r"(video.{0,1}\d+_\d+)")
def __init__(self, username: str, password: str, verbose: bool = True) -> None:
"""
Initializes the scraper.
This function receives a username and password and performs authentication on vk.com to then call api endpoints
Parameters
----------
username : str
Username on vk.com, can be a phone number or email
password : str
Matching password on vk.com
verbose : bool = False
If True will log debug info
Examples
--------
>>> VkScraper("+12345678", "password")
"""
self.session = vk_api.VkApi(username, password)
self.session.auth(token_only=True)
self.verbose = verbose
def scrape(self, url: str) -> List:
return self.scrape_walls(url) + self.scrape_photos(url) + self.scrape_videos(url)
def scrape_walls(self, url: str) -> List:
wall_ids = self.WALL_PATTERN.findall(url)
return self.scrape_wall_ids(wall_ids)
def scrape_wall_ids(self, wall_ids: List[str], copy_history_depth: int = 2) -> List:
"""
Receives a list of wall ids like wall123123_1231
Returns a list with one item per wall_id where each item contains:
:returns `{
"id": "wall_id",
"text": "text in this post" ,
"datetime": datetime of post,
"attachments": {
"photo": [list of urls with max quality],
"album": [list of urls with max quality],
# untested:
# "video": [list of urls with max quality],
# "link": [list of urls with max quality],
},
"payload": original response code which you can parse for more data
}
`
"""
if not len(wall_ids): return []
wall_ids = [wall_id.replace("wall", "") for wall_id in wall_ids]
# docs: https://dev.vk.com/method/wall.getById
headers = {"access_token": self.session.token["access_token"], "posts": ",".join(wall_ids), "extended": "1", "copy_history_depth": str(copy_history_depth), "v": self.session.api_version}
req = requests.get("https://api.vk.com/method/wall.getById", headers)
api_res = req.json()
res = []
for item in api_res.get("response", {}).get("items", []):
attachments_json = item.get("attachments", []) + sum([x.get("attachments", []) for x in item.get("copy_history", [])], [])
attachments = defaultdict(list)
for a in attachments_json:
try:
first_type = a["type"]
attachment = a[first_type]
if first_type == "video":
attachments["video"].extend(self.scrape_videos(f'video{attachment["owner_id"]}_{attachment["id"]}')[0].get("attachments", {}).get("video", [""]))
continue
if first_type == "link":
attachments["link"].append(attachment["url"])
if "photo" in attachment:
attachment = attachment["photo"]
first_type = "photo"
elif "video" in attachment:
attachment = attachment["video"]
attachments["video"].extend(self.scrape_videos(f'video{attachment["owner_id"]}_{attachment["id"]}')[0].get("attachments", {}).get("video", [""]))
continue
else: continue
if "thumb" in attachment:
attachment = attachment["thumb"]
if "sizes" in attachment:
try:
attachments[first_type].append(attachment["sizes"][-1]["url"])
except Exception as e:
print(f"could not get image from attachment: {e}")
except Exception as e:
print(f"Unexpected error in attachment={a}: {e}")
res.append({
"id": f'wall{item["owner_id"]}_{item["id"]}',
"text": item.get("text", ""),
"datetime": datetime.fromtimestamp(item.get("date", 0)),
"attachments": dict(attachments),
"payload": item
})
return res
def scrape_videos(self, url: str) -> List:
# TODO: https://vk.com/video-1_456239018
# TODO https://vk.com/asdasdasd?w=wall-17315087_74182 has 1 video
# https://vk.com/video38556806_456251917?list=ba2b77043648ff3789
video_ids = self.VIDEO_PATTERN.findall(url)
return self.scrape_video_ids(video_ids)
def scrape_video_ids(self, video_ids: List[str]) -> List:
if not len(video_ids): return []
video_ids = [video_id.replace("video", "") for video_id in video_ids]
headers = {"access_token": self.session.token["access_token"], "videos": ",".join(video_ids), "extended": "1", "v": self.session.api_version}
req = requests.get("https://api.vk.com/method/video.get", headers)
api_res = req.json()
res = []
for item in api_res.get("response", {}).get("items", []):
res.append({
"id": f'video{item["owner_id"]}_{item["id"]}',
"text": item.get("title", ""),
"datetime": datetime.fromtimestamp(item.get("date", 0)),
"attachments": {
"video": [item.get("player", "")],
},
"payload": item
})
return res
def scrape_photos(self, url: str) -> List:
photo_ids = self.PHOTO_PATTERN.findall(url)
return self.scrape_photo_ids(photo_ids)
def scrape_photo_ids(self, photo_ids: List[str]) -> List:
if not len(photo_ids): return []
photo_ids = [photo_id.replace("photo", "") for photo_id in photo_ids]
headers = {"access_token": self.session.token["access_token"], "photos": ",".join(photo_ids), "extended": "1", "v": self.session.api_version}
req = requests.get("https://api.vk.com/method/photos.getById", headers)
api_res = req.json()
res = []
for item in api_res.get("response", []):
res.append({
"id": f'photo{item["owner_id"]}_{item["id"]}',
"text": item.get("text", ""),
"datetime": datetime.fromtimestamp(item.get("date", 0)),
"attachments": {
"photo": [item["orig_photo"]["url"]]
},
"payload": item
})
return res

0
my_package/version.py → vk_url_scraper/version.py
View File