Posts Tagged - scrapy

Install Poetry Script Globally

# only the first time we install something 
pipx ensurepath
# close and open terminal again

# build, install and run to make sure it works
poetry build
poetry install
poetry run

# install localy with pipx
pipx install .

# now we're able to run it from anywhere
file-enlarger

in this case we invoke it as file-enlarger as the .toml declares it as such

[tool.poetry.scripts]
file-enlarger = "FileEnlarger:main"

Broken pipx?

Install your program with

# install
py -3.14 -m pip install --user .

# find installation to add to PATH
py -3.14 -m pip show fileenlarger

# add it to PATH; logout; restart console
file-enlarger

This will install it in your local /scripts or /Roaming folder. Remember to add it to your path. In this case it’s in

C:\Users\{user}\AppData\Local\Python\pythoncore-3.14-64\Scripts
C:\Users\{user}\AppData\Roaming\Python\Python314\site-packages

Read More

Python's Poetry

Prerequisites

First of all install pip and user pip to install pipx. From then on, use only pipx

install pip tools

py -m pip install --user pip-tools

# upgrade pip
py -m pip install --upgrade pip

install pipx

py -m pip install --user pipx

# adds executables to global path so you can call them without py -m ...
py -m pipx ensurepath
# close and reopen console

install poetry through pipx

py -m pipx install --user poetry

Read More

Scrapy (Python web crawler)

Scrapy is a web-scrapper & crawler.

Concepts

spider: class that you define and scrapy uses to scrape information from a website (our a group of websites). They must define the initial requests to make, optionally how to follow links in the pages and how to parse the content to extract data

item pipeline: after an item has been crawled by a spider, it’s sent to the item pipeline which processes it through several components that are executed sequentially. You can use them, for example, to save items to a database

How to use

# create a new project
scrapy startproject your_project_name  

# after writing a spider, it starts the crawl
scrapy crawl quotes

Read More