reddit-downloader

Download Reddit archives from Arctic Shift and browse them locally with a Reddit-like UI powered by Datasette.

Setup

pip install -r requirements.txt

Configure subreddits

Edit config.json to set which subreddits to download:

{
  "subreddits": ["LocalLLaMA", "solarpunk", "ebikes"]
}

Download data

Torrent (preferred — bulk download from the Arctic Shift top-40k dataset):

python3 download_torrent.py

API (for subreddits not in the torrent, or as a fallback):

python3 download.py

Both scripts accept subreddit names as CLI args to override config.json:

python3 download.py solarpunk LocalLLaMA

Build the SQLite database

python3 to_sqlite.py

This auto-discovers all downloaded data files (.jsonl from the API, .zst from the torrent) and imports them into data/reddit.db with full-text search.

Browse with datasette

datasette data/reddit.db -m metadata.json --template-dir=templates --static static:static

Run everything

./run.sh

Downloads via torrent, fills gaps via API, builds the database, and launches datasette.

Project structure

config.json              # subreddit list (shared config)
download_torrent.py      # bulk download via Arctic Shift torrent
download.py              # API download (fallback / smaller subs)
to_sqlite.py             # build SQLite DB from downloaded data
metadata.json            # datasette configuration
run.sh                   # full pipeline script
templates/               # custom datasette templates (Reddit-like UI)
static/reddit.css        # custom styles
data/                    # downloaded data + SQLite DB (gitignored)