ORSR Scraper

With this application you can get all changed records in orsr for the current day.

The application consists of two parts:

1. Scraper:

• gets the data of all changed records
  • either the "aktuálna" or the "úplna" version
  • can use a socks5 proxy
• stores the data in a MongoDB

2. Flask app:

• Minimalistic flask app that has two endpoints:
  • /detail with parameter ico
    • returns a json data for the record with ico
  • /list
    • returns a paginated list of records ico and obhcodneMeno

Setup

1. Prerequisites

You need to have installed/access to:

• current python
• MongoDB
• Socks5 proxy (optional)

The installation of these is out of scope of this README

1. Download the app

Download/clone the application

2. venv and requirements

Open terminal cd to app folder and install venv

cd [appPath]
python -m venv venv

install the requirements from requirements.txt

venv/bin/pip install -r requirements.txt

for Windows:
venv\Scripts\pip.exe install -r requirements.txt

3. Config File

There is a default config file "config_base.cfg". For local changes copy this base config file and store it as "config.cfg". The config file has the following structure:

[DB]
MONGODB_URI = mongodb://localhost:27017
MONGODB_DB = softone
MONGODB_COLLECTION = orsr

[WEB]
BASE_URL = https://www.orsr.sk/
ENDPOINT = hladaj_zmeny.asp

[PROXY]
#HTTP_PROXY = socks5://user:pass@host:port
#HTTPS_PROXY = socks5://user:pass@host:port

[APP]
THREADS = 8

Setup the connection to MongoDB, number of threads being used for collecting the data and optionally also the Socks5 Proxy params.

Run the applications

1. Scraper

Run the scraper with

venv/bin/python scraper.py

for Windows:
venv\Scripts\python.exe scraper.py

It will ask you if you want to download the "aktuálny" or "úplný" record.

tqdm status bar with ThreadPool sometimes continues on newline!

2. Flask

Start flask application

venv/bin/python flaskapp.py

for Windows:
venv\Scripts\python.exe flaskapp.py

Now you can get the data from the local test server that usually runs on http://127.0.0.1:5000