enki/kindexr
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
kindexr/spec.md
T

36 KiB
Raw Permalink Blame History

kindexr — Nostr-Native Torznab Indexer

Self-hosted bridge between NIP-35 torrent events on the Nostr relay network and the Torznab API spoken by Sonarr / Radarr / Lidarr / Readarr / Prowlarr.

Build status

  • Phase 0 — bootstrap: ✅ done
  • Phase 1 — reader, basic Torznab: ✅ done
  • Phase 2 — full *arr compatibility: ✅ done
  • Phase 3 — curation (WoT, reports, curation sets): ✅ done
  • Phase 4 — writer / publisher: ✅ done
  • Web UI — dashboard, indexed browser, publisher manager, publish queue: ✅ done
  • Phase 5 — Blossom binary bridge: long horizon
  • Phase 6 — FIPS deployment: planned

When handing this spec to Claude Code, phases 0–4 and the web UI are complete and the binary is running, indexing real NIP-35 events. Start at Phase 5 or 6 unless retrofitting earlier work.

What this is

Two halves in one daemon:

  • Reader (Phases 1-3): subscribes to kind 2003 (torrent) and kind 2004 (torrent comment) events on configured Nostr relays, indexes them locally in SQLite, and exposes a Torznab-compatible HTTP API so the *arr automation stack can query as if kindexr were any other indexer.
  • Writer (Phase 4+): watches a torrent client (qBittorrent / Transmission / Deluge) for completed downloads explicitly tagged for publishing, builds NIP-35 events, signs them via a dedicated kindexer identity, and publishes them to configured relays.

Positioning: same architectural slot as Jackett or Prowlarr. Not a frontend like dtan.xyz. Not a relay. Not a downloader. Not a DHT crawler. Middleware between Nostr and *arr.

Deployment model

Single-operator, self-hosted, personal infrastructure. Every user runs their own kindexr instance on their own hardware. There is no signup, no shared service, no multi-tenancy. The API keys exist solely to let your own Sonarr/Radarr authenticate to your own kindexr daemon.

This is intentional. A public-service deployment would re-centralize what Nostr decentralized, take on meaningful legal exposure for an indexer of torrent content, and require operational overhead (abuse handling, billing, support) that isn't the goal. The federated-curation pattern in the Identity section below provides the "share with others" value without operating a service.

Non-goals

  • No DHT crawling. Kindexr does not join the BitTorrent DHT, harvest info-hashes, or participate in BitTorrent protocol at all. The only ingest source is signed Nostr events. This is the primary structural difference from tools like Bitmagnet.
  • No website scraping. No HTML parsing of tracker sites.
  • No web browse UI for humans in v1. dtan.xyz and nostrudel/torrents serve that need. A browse UI may come later but is not the product.
  • No new download protocol. BitTorrent is unchanged. Magnet links come out exactly as they go in.
  • No relay implementation. Kindexr is a client of relays.
  • No replacement for Sonarr/Radarr. Kindexr sits underneath them in the same slot Jackett/Prowlarr occupies.
  • No multi-tenant SaaS. No user accounts, no signup, no admin panel for managing other people's access.

Architecture

  Sonarr / Radarr / Lidarr / Readarr / Prowlarr
                       │
                       │  Torznab HTTP/XML
                       ▼
                ┌──────────────┐
                │   kindexr    │
                │   reader     │ ◄── WSS ─── Nostr relays
                │   writer     │ ─── WSS ──► Nostr relays
                └──────────────┘
                       ▲
                       │ HTTP API
                       │
              qBittorrent / Transmission / Deluge
              (download client; only used by writer)

Reader and writer share the same daemon binary and the same SQLite database. They can be enabled independently — Phase 2/3 instances run with publisher.enabled: false. The reader is the default; the writer is opt-in.

Tech stack

Locked. Don't re-litigate.

Concern Choice Why
Language Rust (latest stable) Memory safety, single static binary, excellent async + SQLite support
Nostr client nostr-sdk (rust-nostr org) High-level Client API; NIP-77 negentropy, NIP-46 bunker, NIP-42 AUTH
Negentropy negentropy (rust-nostr org) NIP-77 set reconciliation for relay resync
Storage SQLite + FTS5 Single-file backup, no separate service, FTS5 fine to several million rows
SQLite driver sqlx (sqlite + bundled features) Compile-time-checked queries, async; bundled embeds SQLite, no libsqlite3.so dep
HTTP framework axum + tower Ergonomic async HTTP, tower middleware ecosystem
Torrent parsing lava_torrent Parse .torrent metainfo for writer (Phase 4); listed now
Config figment (yaml + env) + clap YAML primary, KINDEXR_* env override, CLI flag overrides via clap
Logging tracing + tracing-subscriber (json) Structured JSON for journald
TMDB tmdb-api crate Metadata enrichment (Phase 2); listed now
Torznab XML quick-xml (serialize feature) Zero-copy XML with serde derive
Async runtime tokio (multi-thread) Industry standard
CLI clap (derive feature) Subcommand-style admin CLI
Build tool just (justfile) Replaces Makefile
Deployment systemd + single static binary Same shape as before

PostgreSQL explicitly not used. SQLite for everything. Revisit only at >5M events or >100 concurrent Torznab clients.

Repository layout

kindexr/
├── Cargo.toml
├── Cargo.lock
├── src/
│   ├── main.rs
│   ├── lib.rs
│   ├── config.rs
│   ├── db/
│   │   ├── mod.rs
│   │   ├── migrations/          # numbered .sql files (same schema)
│   │   └── queries.rs
│   ├── nostr/
│   │   ├── mod.rs
│   │   ├── reader.rs
│   │   ├── writer.rs            # Phase 4
│   │   ├── signer.rs            # Phase 4
│   │   └── parser.rs
│   ├── torznab/
│   │   ├── mod.rs
│   │   ├── server.rs
│   │   ├── caps.rs
│   │   ├── search.rs
│   │   ├── xml.rs
│   │   └── categories.rs
│   ├── enrich/
│   │   ├── mod.rs
│   │   ├── tmdb.rs
│   │   └── parser.rs
│   ├── health/
│   │   ├── mod.rs
│   │   ├── tracker.rs
│   │   └── dht.rs
│   ├── publisher/
│   │   ├── mod.rs
│   │   ├── qbittorrent.rs
│   │   ├── transmission.rs
│   │   └── watcher.rs
│   └── wot/
│       ├── mod.rs
│       ├── follows.rs
│       └── trust.rs
├── src/bin/
│   └── kindexr-cli.rs
├── deploy/
│   ├── kindexr.service
│   ├── kindexr.example.yaml
│   └── nginx.conf.example
├── docs/
│   ├── ARCHITECTURE.md
│   ├── TORZNAB.md
│   ├── PUBLISHING.md
│   ├── IDENTITY.md
│   └── FIPS.md
├── archive/
│   └── go/                      # legacy Go implementation (git mv'd)
├── justfile
└── README.md

Database schema

Single SQLite file at /var/lib/kindexr/kindexr.db. Numbered migrations under internal/db/migrations/. Apply on startup; refuse to start on migration failure.

-- 001_initial.sql (applied in Phase 0/1; here for reference)

CREATE TABLE torrents (
    event_id      TEXT PRIMARY KEY,         -- nostr event id (hex)
    info_hash     TEXT NOT NULL,            -- v1 info-hash (hex, lowercase)
    pubkey        TEXT NOT NULL,            -- publisher pubkey (hex)
    created_at    INTEGER NOT NULL,
    ingested_at   INTEGER NOT NULL,
    title         TEXT NOT NULL,
    description   TEXT,
    size_bytes    INTEGER,
    category      TEXT,                     -- movie/tv/music/book/audio/xxx/other
    newznab_cat   INTEGER,
    imdb_id       TEXT,
    tmdb_id       TEXT,                     -- "movie:693134" | "tv:1396"
    tvdb_id       TEXT,
    season        INTEGER,
    episode       INTEGER,
    quality       TEXT,
    source        TEXT,
    raw_event     TEXT NOT NULL
);
CREATE UNIQUE INDEX idx_torrents_event_id ON torrents(event_id);
CREATE INDEX idx_torrents_info_hash ON torrents(info_hash);
CREATE INDEX idx_torrents_pubkey ON torrents(pubkey);
CREATE INDEX idx_torrents_imdb ON torrents(imdb_id) WHERE imdb_id IS NOT NULL;
CREATE INDEX idx_torrents_tmdb ON torrents(tmdb_id) WHERE tmdb_id IS NOT NULL;
CREATE INDEX idx_torrents_tvdb ON torrents(tvdb_id) WHERE tvdb_id IS NOT NULL;
CREATE INDEX idx_torrents_created ON torrents(created_at DESC);
CREATE INDEX idx_torrents_cat ON torrents(newznab_cat);

CREATE VIRTUAL TABLE torrents_fts USING fts5(
    title, description,
    content='torrents',
    content_rowid='rowid',
    tokenize='unicode61 remove_diacritics 2'
);
-- (triggers ai/ad/au to keep FTS in sync — already in place from Phase 1)

CREATE TABLE files (
    event_id   TEXT NOT NULL,
    idx        INTEGER NOT NULL,
    path       TEXT NOT NULL,
    size_bytes INTEGER,
    PRIMARY KEY (event_id, idx),
    FOREIGN KEY (event_id) REFERENCES torrents(event_id) ON DELETE CASCADE
);

CREATE TABLE trackers (
    event_id TEXT NOT NULL,
    url      TEXT NOT NULL,
    PRIMARY KEY (event_id, url),
    FOREIGN KEY (event_id) REFERENCES torrents(event_id) ON DELETE CASCADE
);

CREATE TABLE tags (
    event_id TEXT NOT NULL,
    tag      TEXT NOT NULL,
    PRIMARY KEY (event_id, tag),
    FOREIGN KEY (event_id) REFERENCES torrents(event_id) ON DELETE CASCADE
);
CREATE INDEX idx_tags_tag ON tags(tag);

CREATE TABLE publishers (
    pubkey      TEXT PRIMARY KEY,
    npub        TEXT,
    name        TEXT,
    trust       REAL DEFAULT 0,
    notes       TEXT,
    blocked     INTEGER DEFAULT 0,
    torrents_n  INTEGER DEFAULT 0,
    first_seen  INTEGER,
    last_seen   INTEGER
);

CREATE TABLE relays (
    url         TEXT PRIMARY KEY,
    enabled     INTEGER DEFAULT 1,
    last_event  INTEGER,
    last_sync   INTEGER,
    notes       TEXT
);

-- Simple flat API keys. No per-key visibility filtering (deferred indefinitely).
CREATE TABLE api_keys (
    key         TEXT PRIMARY KEY,
    label       TEXT NOT NULL,
    created_at  INTEGER NOT NULL,
    last_used   INTEGER
);

CREATE TABLE health (
    info_hash   TEXT PRIMARY KEY,
    seeders     INTEGER,
    leechers    INTEGER,
    checked_at  INTEGER NOT NULL
);

CREATE TABLE comments (
    event_id          TEXT PRIMARY KEY,
    torrent_event_id  TEXT NOT NULL,
    pubkey            TEXT NOT NULL,
    created_at        INTEGER NOT NULL,
    content           TEXT NOT NULL,
    raw_event         TEXT NOT NULL,
    FOREIGN KEY (torrent_event_id) REFERENCES torrents(event_id) ON DELETE CASCADE
);

Migration notes for already-completed Phase 1

The original spec had a visibility and curation_set column on api_keys for per-key filter scoping. Drop those — the personal-infrastructure model doesn't need them, and Phase 3 curation applies daemon-wide instead of per-key. If they exist in your current schema:

-- 002_drop_apikey_visibility.sql
ALTER TABLE api_keys DROP COLUMN visibility;
ALTER TABLE api_keys DROP COLUMN curation_set;

Config file

/etc/kindexr/config.yaml. Load order: defaults → file → env (KINDEXR_*) → CLI flags.

server:
  listen: "127.0.0.1:9117"
  base_url: "https://kindexr.example.com"

database:
  path: "/var/lib/kindexr/kindexr.db"

logging:
  level: "info"
  format: "json"

# Inbox relays — where to READ NIP-35 events from
relays:
  - "wss://relay.damus.io"
  - "wss://nos.lol"
  - "wss://relay.primal.net"
  - "wss://nostr.mom"
  - "wss://relay.snort.social"
  - "wss://sovbit.host"

negentropy_bootstrap: true
backfill_days: 365

# Curation (Phase 3)
curation:
  wot_only: false
  follow_depth: 2
  operator_pubkey: ""                  # the npub whose follow graph defines WoT
  allowlist: []
  blocklist: []
  curation_sets: []                    # naddr1... of subscribed kind 30004 sets
  trust_assertions: []                 # naddr1... of trusted NIP-85 assertion sources
  exclude_categories: [6000, 6010, 6020, 6030, 6040, 6050, 6060, 6070, 6080, 6090]
                                       # XXX range — always exclude regardless of WoT

# TMDB enrichment
tmdb:
  enabled: true
  api_key: "${TMDB_API_KEY}"
  cache_ttl: "168h"

# Health scraping (off by default — DHT only when enabled)
health:
  enabled: false
  method: "dht"
  refresh_interval: "30m"

# Writer side (Phase 4+) — off until explicitly configured
publisher:
  enabled: false

  # Dedicated kindexr identity. NEVER your main npub.
  signer:
    mode: "bunker"                     # bunker | ncryptsec | local
    bunker_uri: "bunker://..."         # NIP-46 connection string
    ncryptsec: ""
    nsec: ""                           # only if mode: local — discouraged

  # Outbox lists by content category. Different content can reach different
  # audiences. Categories without an explicit list fall back to default.
  outbox:
    default:
      - "wss://sovbit.host"
      - "wss://relay.damus.io"
      - "wss://nos.lol"
    # narrow audience — only your relay
    private:
      - "wss://sovbit.host"
    # wider reach for things you want public
    public:
      - "wss://sovbit.host"
      - "wss://relay.damus.io"
      - "wss://nos.lol"
      - "wss://relay.primal.net"
      - "wss://nostr.mom"

  # Download client integration
  client:
    type: "qbittorrent"                # qbittorrent | transmission | deluge | watch_dir
    qbittorrent:
      url: "http://127.0.0.1:8080"
      username: "admin"
      password: "${QBIT_PASSWORD}"

  # Per-category mapping: qBit category -> kindexr outbox + nostr metadata
  # Only torrents in a configured category get published. Everything else
  # is ignored. Default is empty — nothing publishes until you opt in.
  publish_categories:
    publish-public:
      outbox: "public"
      nostr_tags:
        - ["t", "shared-by-kindexr"]
    publish-private:
      outbox: "private"
      nostr_tags: []
    publish-books:
      outbox: "public"
      nostr_category: "book"
    publish-foss:
      outbox: "public"
      nostr_category: "software"

  # Delay between download completion and publish, for sanity checking
  publish_delay: "1h"

  # TMDB enrichment of own publishes before signing
  enrich_before_publish: true

Identity model

This is new since the original spec and deserves its own section.

Two distinct identities

Operator identity (your main npub). Your social Nostr identity. Used by kindexr only for:

  • Reading your kind 3 follow list to build the WoT
  • Reading your kind 10000 mute list to honor your blocks
  • Reading curation sets and labels you've published

Kindexr never signs anything as your operator identity.

Kindexr identity (a dedicated nsec). Generated specifically for this kindexr instance. Used for:

  • Signing kind 2003 torrent events the writer publishes
  • Signing kind 30004 curation sets ("kindexr's verified releases")
  • Signing kind 1985 labels ("this pubkey is a known good UHD encoder")
  • Signing kind 30382 trusted assertions about other publishers

If the seedbox is compromised, the worst case is "someone publishes spam under the kindexr identity for a while." You rotate, your main identity is intact, your social graph is intact.

Bootstrapping trust for the kindexr identity

A brand-new nsec has no followers and no presence. Other people's kindexr instances filtering by WoT will not see its publishes. The bootstrap sequence:

  1. Generate the kindexr nsec via NIP-49 ncryptsec or directly into the bunker. Keep the unencrypted nsec offline (paper, password manager) for recovery. Never put it on the seedbox.

  2. Set up NIP-46 bunker on a more locked-down host than the seedbox — UTS-01 is the natural choice. The bunker holds the actual nsec; kindexr connects to it over Nostr and requests signatures on demand. See docs/IDENTITY.md for the bunker walkthrough.

  3. Publish kind 0 (profile metadata) for the kindexr pubkey:

    {
  "kind": 0,
  "content": "{\"name\":\"sovbit-kindexr\",\"display_name\":\"sovbit kindexr\",\"about\":\"Automated NIP-35 publisher operated by @<your_main_npub>. Linux ISOs, FOSS releases, public-domain books. Run your own: github.com/<you>/kindexr\",\"nip05\":\"kindexr@sovbit.host\",\"lud16\":\"<lightning address>\",\"picture\":\"https://sovbit.host/kindexr-avatar.png\"}"
}

  4. Set up the NIP-05. On sovbit.host's /.well-known/nostr.json, add an entry mapping kindexer to the kindexr pubkey (hex). This makes kindexr@sovbit.host resolve and verify.

  5. From your main npub, follow the kindexr pubkey. Update your kind 3 follow list to include the kindexr pubkey. Now anyone running WoT-depth-1 or 2 filtering who follows you will automatically include kindexr's publishes.

  6. From your main npub, publish a trusted assertion (NIP-85 kind 30382) or label (NIP-32 kind 1985) explicitly vouching for the kindexr pubkey:

    {
  "kind": 30382,
  "tags": [
    ["d", "<kindexr_pubkey_hex>"],
    ["rating", "1.0"],
    ["operator-vouched", "kindexr-instance"]
  ],
  "content": "I operate this kindexr instance. Publishes are automated from my seedbox."
}

  7. Operationally, only ever auto-publish things you'd defend. Auto-publishing every torrent that crosses your seedbox is a bad idea regardless of identity. The publish-categories config in the writer section enforces this — only torrents you deliberately tagged get published. Use that gate.

Why not NIP-26 delegation?

NIP-26 lets one key sign on behalf of another but is marked unrecommended in the current NIPs list ("adds unnecessary burden for little gain"). Use the trusted-assertion/follow approach instead.

Torznab API surface

(Phase 1 already implemented t=caps and t=search. This section documents the full surface for Phase 2 completion.)

All endpoints under /api. Auth via apikey= query param.

Endpoint Status Notes
GET /api?t=caps ✅ Phase 1 Verify the <categories> block matches what Sonarr/Radarr accept. Copy structure from a working Jackett indexer if unsure.
GET /api?t=search&q=&cat= ✅ Phase 1 Generic FTS5 search
GET /api?t=tvsearch&tvdbid=&imdbid=&tmdbid=&season=&ep=&q= 🎯 Phase 2 Prefer structured ID matches over q.
GET /api?t=movie&imdbid=&tmdbid=&q= 🎯 Phase 2 Movie search
GET /api?t=music&artist=&album=&year=&q= 🎯 Phase 2 Music search
GET /api?t=audio&artist=&album=&year=&q= 🎯 Phase 2 Audio (audiobooks etc)
GET /api?t=book&author=&title=&q= 🎯 Phase 2 Book search
GET /health ✅ Phase 1 No auth. JSON.
GET /metrics Optional Prometheus text format

Response format

Standard Torznab RSS with the torznab XML namespace. Mandatory <torznab:attr> keys per item: category, size, infohash, magneturl. Optional but strongly recommended: seeders, peers, tvdbid, imdbid, tmdbid.

For events lacking peer counts, omit the seeders/peers attrs entirely rather than reporting 0 — Sonarr may drop "0-seeder" results unless explicitly configured otherwise.

Sonarr <categories> gotcha

Sonarr is silently strict about the caps response. Categories must use the exact newznab numeric IDs, must include both parent and subcat structure, and must not include unrecognized IDs. If indexer testing fails in Sonarr with no obvious error, the caps response is almost always the culprit. Diff yours against a known-working Jackett t=caps output.

NIP-35 event handling

Reference shape (kind 2003):

{
  "kind": 2003,
  "pubkey": "<hex>",
  "created_at": 1715000000,
  "content": "<long description>",
  "tags": [
    ["title", "Some.Show.S01E02.2160p.WEB-DL.x265-GRP"],
    ["x", "<40-hex info-hash>"],
    ["file", "Some.Show.S01E02.mkv", "15234567890"],
    ["tracker", "udp://tracker.opentrackr.org:1337"],
    ["i", "tcat:video,tv,4k"],
    ["i", "newznab:5045"],
    ["i", "imdb:tt15239678"],
    ["i", "tmdb:tv:693134"],
    ["i", "tvdb:355567"],
    ["t", "tv"], ["t", "4k"]
  ]
}

Parser rules

  • Reject events without ["x", <40-hex>]. Log + drop.
  • Validate signature (nostr-sdk handles this automatically; do not disable verification). Drop invalid.
  • size_bytes = sum of file tag size positions (index 2).
  • trackers table: accumulate from tracker tags.
  • i tag prefixes drive ID matching: imdb:, tmdb:movie:, tmdb:tv:, tvdb:, tcat:, newznab:.
  • newznab_cat: from ["i", "newznab:NNNN"] if present; else inferred from tcat:; else fallback to title parser.
  • t tags into tags table for filtering.
  • Honor curation.exclude_categories at ingest: events with excluded newznab_cat are stored but flagged excluded=1 (or dropped entirely — config knob, default drop).

Magnet construction

magnet:?xt=urn:btih:<info_hash>
       &dn=<url-encoded title>
       &tr=<url-encoded tracker>...

Always include event-listed trackers plus a configurable fallback list of public DHT trackers.

Category mapping (tcat → newznab)

Same map as the original spec. Extend as needed.

tcat path newznab
video,movie 2000
video,movie,hd 2040
video,movie,4k / uhd 2045
video,movie,bluray / remux 2050
video,tv 5000
video,tv,hd 5040
video,tv,4k / uhd 5045
video,tv,anime 5070
audio,music 3000
audio,music,lossless 3040
audio,audiobook 3030
book,ebook 7020
book,comic 7030
anything else 8000
anything in XXX 6000-series — excluded by default

Reader behavior

Sources

Kindexr ingests only signed Nostr events from configured relays. It does not:

  • Crawl the BitTorrent DHT
  • Scrape websites
  • Participate in BitTorrent swarms
  • Listen for peer announcements

This is the primary structural difference from Bitmagnet and similar DHT crawlers. The ingest volume and signal-to-noise ratio are categorically different — Nostr publishes are deliberate, identified, and signed; DHT announces are anonymous, automated, and dominated by garbage and porn.

Subscription model

For each configured relay, kindexr opens a persistent WebSocket and submits:

["REQ", "live-2003", {"kinds": [2003], "since": <last_seen_ts>}]
["REQ", "live-2004", {"kinds": [2004], "since": <last_seen_ts>}]

Events arrive in real time. On disconnect, exponential backoff reconnect.

Resync

Every resync_interval (default 6h), run NIP-77 negentropy against each relay to catch anything missed during disconnects or transient outages. On startup, negentropy bootstrap pulls everything within backfill_days (default 365) if negentropy_bootstrap: true.

Storage

Even ~100k events takes <500MB SQLite including FTS. Don't preemptively optimize storage; reconsider only at >5M rows.

Writer behavior

Trigger model

Default: nothing publishes. The writer is opt-in per torrent. Auto-publishing everything that crosses the seedbox is explicitly not supported.

Two trigger modes:

Category-tagged auto-publish. In qBittorrent/Transmission/Deluge, torrents get a category from publisher.publish_categories only when you deliberately set that category. The writer watches those categories; on download completion, it builds the NIP-35 event using the category's configured outbox list and nostr metadata, waits publish_delay, and publishes.

Manual CLI bulk publish. kindexr-cli publish --from-dir /media/library/foo --category publish-public walks a directory, parses torrents, publishes events. For backfilling an existing library.

Publish frequency

Exactly what you trigger. A single tagged download → one event. Could be 0/day, could be 50/day. Driven by deliberate operator action, never automatic across the whole download set.

Sign flow

  1. Build the kind 2003 event from torrent metainfo + optional TMDB enrichment.
  2. Open NIP-46 bunker connection (or use ncryptsec/local nsec per config).
  3. Send sign_event request to bunker over Nostr (encrypted kind 24133).
  4. Receive signed event back.
  5. For each relay in the resolved outbox list, open WSS and send ["EVENT", ...].
  6. Wait for OK ack; log failures, don't retry forever.
  7. Insert the event into local DB as if ingested.

Why a bunker, not a local nsec

The seedbox is the most exposed surface in your stack. An nsec on a publicly-reachable Linux host is a persistent compromise risk. NIP-46 puts the actual signing material on a separate, locked-down host (UTS-01 or your phone running Amber) and gives kindexr a session that can be revoked instantly. Same model as a hardware wallet for Bitcoin.

Junk / spam defenses

Even though Nostr is structurally cleaner than DHT, spam exists. Stacked defenses, ingest to API:

  1. WoT filter (wot_only: true): only ingest from your follow graph within N hops. Most effective single filter.

  2. Block list: kindexr ingests your kind 10000 mute list from configured relays and honors it.

  3. Curation set subscription: subscribe to kind 30004 sets from curators you trust; their endorsement boosts visibility of contained events.

  4. Trusted assertions: ingest NIP-85 kind 30382 events from operators you trust; aggregate their pubkey ratings.

  5. NIP-32 labels (kind 1985): subscribe to label sets marking known-bad pubkeys or info-hashes.

  6. NIP-56 reports (kind 1984): aggregate reports against pubkeys; auto-block at configurable threshold (default 5 reports from distinct trusted reporters).

  7. Category exclusion: exclude_categories drops events with newznab_cat in the excluded set at ingest. XXX range (6000-6099) excluded by default.

  8. Info-hash dedup: same info-hash from N different publishers collapses to one row in torrents; additional publishers tracked in a join table but don't multiply results.

  9. API-layer category filter: Sonarr asking tvsearch&tvdbid=...&cat=5040 will never return out-of-category results regardless of WoT, because the query filters by newznab_cat first.

The key insight: identity-based filtering scales. When you find a problem publisher, you mute the pubkey and they're gone permanently. Bitmagnet has no equivalent — DHT peers are anonymous and ephemeral.

Title parser

For events without imdb/tmdb/tvdb tags, parse the title to extract:

  • Title/year (4-digit pattern)
  • Season (Sxx) / Episode (Exx)
  • Quality (480p|576p|720p|1080p|2160p|SD|HD|UHD)
  • Source (WEB-DL|WEBRip|BluRay|BDRip|HDTV|DVDRip|REMUX)
  • Codec (x264|x265|HEVC|AV1)
  • Release group (after final -)

After parsing, hit TMDB's search/tv or search/movie with the cleaned name + year, store the resolved IDs, cache aggressively (TMDB IDs are stable).

Use regex-based extraction in Rust. lava_torrent handles binary .torrent file parsing for Phase 4; title string parsing uses custom regex.

CLI

kindexr-cli for admin tasks. Each subcommand should print clear status, exit non-zero on error.

kindexr-cli apikey list
kindexr-cli apikey create --label sonarr
kindexr-cli apikey revoke <key>

kindexr-cli relay list
kindexr-cli relay add wss://relay.example.com
kindexr-cli relay remove wss://relay.example.com
kindexr-cli relay enable/disable wss://...

kindexr-cli publisher list                          # list known publishers, sorted by trust
kindexr-cli publisher trust <npub> <-1.0..1.0>
kindexr-cli publisher block <npub>
kindexr-cli publisher unblock <npub>

kindexr-cli stats                                   # events total, by category, by publisher

kindexr-cli publish --from-dir <path> --category publish-public
kindexr-cli publish --magnet <magnet> --title <title> --category ...

kindexr-cli identity init                           # generate kindexr nsec, walk bunker setup
kindexr-cli identity profile-publish                # publish kind 0 for kindexr identity
kindexr-cli identity vouch                          # publish NIP-85 from main npub vouching for kindexr

kindexr-cli wot rebuild                             # force WoT recomputation
kindexr-cli wot status                              # current allowed pubkey count

Phase plan

Phase 0 — bootstrap ✅ done

Scaffolding, config loading, DB migrations, health endpoint, systemd unit.

Phase 1 — reader, basic Torznab ✅ done

Relay subscription, NIP-35 parsing, SQLite storage, FTS search, t=caps, t=search, valid Torznab RSS, Sonarr-add-as-indexer works.

Phase 2 — full *arr compatibility ✅ done

  • t=tvsearch with tvdbid/imdbid/tmdbid + season/ep filters
  • t=movie with imdbid/tmdbid
  • t=music, t=audio with artist/album/year
  • t=book with author/title
  • Title parser (regex-based: season/ep, year, quality, source)
  • TMDB enrichment — direct reqwest to TMDB v3 API, async background update
  • Newznab category map complete and applied at ingest
  • XXX category exclusion configurable via curation.exclude_categories
  • Caps XML matches Torznab spec category tree

Phase 3 — curation ✅ done

  • WoT graph builder from operator's kind 3 + follows-of-follows (level 0/1/2)
  • kind 10000 mute list ingestion
  • kind 30004 curation set subscription — curated events rank higher in search
  • kind 1984 report aggregation with configurable auto-block threshold
  • Trust score computation per publisher (WoT level + report penalty)
  • CLI commands: publisher list/info/block/unblock/mute/trust
  • kind 30382 trusted assertion ingestion — deferred
  • kind 1985 label ingestion — deferred

Acceptance: With wot_only: true, ingest from a known-spammy relay yields zero events that aren't in your WoT. Subscribing to a curation set surfaces those events at higher rank.

Phase 4 — writer / publisher ✅ done

  • Local nsec signing (bech32 or hex) via Signer
  • kindexr-cli identity init — generate or import nsec, store in DB
  • qBittorrent poll integration — completed-in-category → enqueue → publish after delay
  • Build kind 2003 from lava_torrent metainfo (files, trackers, info_hash)
  • Publish-delay queue — configurable hold before signing and broadcasting
  • TMDB inline lookup before publish (10s timeout, best-effort)
  • Signed events published to outbox relays via nostr-sdk, inserted into local DB
  • kindexr-cli publish --from <path> — bulk enqueue .torrent files
  • NIP-46 bunker signing — config field wired, implementation deferred
  • Per-category outbox routing — all categories share one outbox for now
  • CLI identity profile publish + vouch flows — deferred

Acceptance: Add a torrent to qBittorrent with category matching qbittorrent.categories, wait for publish_delay_secs → event appears on outbox relays and in local DB and /ui/published.

Web UI ✅ done

  • /ui — dashboard: indexed count, publisher count, queue stats, relay connection list, recent ingest feed
  • /ui/indexed — FTS-powered paginated torrent browser with search
  • /ui/publishers — publisher table with trust/WoT/block/mute status; vouch form accepts npub or hex pubkey; per-row block/unblock/mute actions
  • /ui/published — publish queue with pending / done tabs and error display
  • Dark theme, server-side rendered HTML, no build step, no JS framework

Phase 5 — Blossom binary bridge (long horizon)

Defer until Phase 4 is solid and ideally adopted by other operators. Requires proposing a NIP. Shape:

  • Sister kind to NIP-35 for Blossom blob releases
  • Same tag surface minus x/tracker, plus blob tags pointing to Blossom hashes on listed servers
  • Downloader sidecar that fetches blobs from Blossom servers, reassembles, hands to media library
  • Nostr-native Usenet replacement

Phase 6 — FIPS deployment

Deploy kindexr on a FIPS-networked host so peers access it over a private overlay without exposing the Torznab port to the public internet.

  • Mode A — kindexr bound to a FIPS address, peer points Sonarr at http://kindexr.fips:9117
  • Mode B — relay URLs in relays: and publisher.outbox use FIPS-resolvable WSS endpoints for private relay paths
  • Mode C — direct fips Rust crate integration (deferred; feasible once FIPS stabilizes past 0.x)

Acceptance: Sonarr on a FIPS peer resolves kindexr.fips, authenticates with an API key, and performs a successful caps + search round-trip with no public internet exposure. See docs/FIPS.md for step-by-step operator instructions.

systemd unit

(Already deployed in Phase 0; included for completeness.)

[Unit]
Description=kindexr — Nostr-native Torznab indexer
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=kindexr
Group=kindexr
ExecStart=/usr/local/bin/kindexr --config /etc/kindexr/config.yaml
Restart=on-failure
RestartSec=5

NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/kindexr
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
RestrictRealtime=true
RestrictNamespaces=true

StandardOutput=journal
StandardError=journal
SyslogIdentifier=kindexr

[Install]
WantedBy=multi-user.target

Things to flag

  • The kindexr identity should never be your main npub. Generate a dedicated nsec, sign via NIP-46 bunker on UTS-01 (or Amber). Walk the trust-bootstrap chain in docs/IDENTITY.md.
  • Auto-publish should always be opt-in per torrent via the category mechanism. Don't add an "auto-publish everything" mode.
  • Title parsing is the part most likely to suck. Budget extra time. The reference parsers have edge cases for non-English titles, anime, scene tags.
  • TMDB rate limits (50 req/sec). Implement a circuit breaker.
  • Sonarr <categories> block in caps must be exactly right or Sonarr silently rejects with no useful error.
  • Don't scrape private trackers for seeder counts. Default health.enabled: false. DHT-only when enabled.
  • Back up the SQLite file via the online backup API or sqlite3 .backup. Don't naive-cp a live database.
  • NIP-77 negentropy bootstrap against many relays is slow on first run. Show progress in logs. Consider a --skip-bootstrap flag.

Open questions

Reduced from the original spec, since several got resolved during design discussion.

  1. Show kind 2004 comments in Torznab description? Default: no in v1, store but don't surface. Could add a <description> augmentation in Phase 3.
  2. Tracker scraping (UDP/HTTP) in addition to DHT for seeder counts? Default: no, DHT only when enabled.
  3. NIP-50 search relays for query offload? Default: no, local FTS is faster and more flexible.
  4. Rate limit Torznab queries per API key? Default: yes, 60/min.

Reference material

v1.0 done criteria

  • Phases 1–4 acceptance criteria all pass
  • kindexr has been running on the operator's seedbox for 30 days without crashing or needing intervention
  • At least one external operator is running an instance
  • README is good enough that someone new can deploy in under an hour
  • A kind 30023 long-form post explaining what it is and how to run it exists, signed by the kindexr identity
  • Listed in awesome-nostr and relevant *arr community resources
Reference in New Issue View Git Blame Copy Permalink