(tested) refactor code

* add sentinel master parameter into driver config
* replace yaml double deserialization with `mapstructure` in initializers
* replace struct initializers with registered functions
* add torrent approval MD and a sanitize rest MDs
This commit is contained in:
Lawrence, Rendall
2022-04-16 00:21:47 +03:00
parent 397e106396
commit 8cd8343757
32 changed files with 473 additions and 385 deletions
+15 -5
View File
@@ -2,12 +2,22 @@
### Overview
BitTorrent clients send Announce and Scrape requests to a _Frontend_. Frontends parse requests and write responses for
the particular protocol they implement. The _TrackerLogic_ interface is used to generate responses for requests and
optionally perform a task after responding to a client. A configurable chain of _PreHook_ and _PostHook_ middleware is
used to construct an instance of TrackerLogic. PreHooks are middleware that are executed before the response has been
BitTorrent clients send Announce and Scrape requests to a _Frontend_.
Frontends parse requests and write responses for
the particular protocol they implement.
The _TrackerLogic_ interface is used to generate responses for requests and
optionally perform a task after responding to a client.
A configurable chain of _PreHook_ and _PostHook_ middleware is
used to construct an instance of TrackerLogic.
PreHooks are middleware that are executed before the response has been
written. After all PreHooks have executed, any missing response fields that are required are filled by reading out of
the configured implementation of the _Storage_ interface. PostHooks are asynchronous tasks that occur after a response
the configured implementation of the _Storage_ interface.
PostHooks are asynchronous tasks that occur after a response
has been delivered to the client. Because they are unnecessary to for generating a response, updates to the Storage for
a particular request are done asynchronously in a PostHook.
+3 -3
View File
@@ -1,10 +1,10 @@
# Frontends
A _Frontend_ is a component of Chihaya that serves a BitTorrent tracker on one protocol. The frontend accepts, parses
A _Frontend_ is a component of MoChi that serves a BitTorrent tracker on one protocol. The frontend accepts, parses
and sanitizes requests, passes them to the _Logic_ and writes responses to _Clients_.
This documentation first gives a high-level overview of Frontends and later goes into implementation specifics. Users of
Chihaya are expected to just read the first part - developers should read both.
MoChi are expected to just read the first part - developers should read both.
## Functionality
@@ -19,7 +19,7 @@ answers each of them with one response, a basic overview of the control flow is:
## Available Frontends
Chihaya ships with frontends for HTTP(S) and UDP. The HTTP frontend uses Go's `http` package. The UDP frontend
MoChi ships with frontends for HTTP(S) and UDP. The HTTP frontend uses Go's `http` package. The UDP frontend
implements both [old-opentracker-style] IPv6 and the IPv6 support specified in [BEP 15]. The advantage of the old
opentracker style is that it contains a usable IPv6 `ip` field, to enable IP overrides in announces.
+6 -6
View File
@@ -28,10 +28,10 @@ An example config might look like this:
```yaml
mochi:
prehooks:
- name: interval variation
config:
modify_response_probability: 0.2
max_increase_delta: 60
modify_min_interval: true
prehooks:
- name: interval variation
config:
modify_response_probability: 0.2
max_increase_delta: 60
modify_min_interval: true
```
+64
View File
@@ -0,0 +1,64 @@
# Approved torrents list
Package `torrentapproval` can be used for only allow or block
specified hashes or block specified hashes.
## Functionality
As said above, there are two modes of approval: white list and black list.
If mode is **white list** (`invert` set to `false`), tracker works in
_semi-private_ mode, which means, than peers can could share and receive info
about only specified list of torrents' hashes.
I.e.: if configuration contains hash `AAAA`, but peer announces hash `BBBB`
tracker will return `unapproved torrent` message back to peer.
If mode is **black list** (`invert` set to `true`), tracker will allow all hashes
**except** specified.
## Hash sources
There are two sources of hashes: `list` and `directory`.
Both of them used as **INITIAL** source for storing in storage.
If storage is not `memory`, records are persisted until _somebody_
or _something_ (different tool with access to storage) won't delete it.
`list` is the static set of hashes, specified in configuration file.
`directory` will watch for `*.torrent` files in specified path and
append/delete records from storage. This source will parse all existing
files at start and then watch for new files to add, or for delete events
to remove hash from storage.
## Configuration
This middleware provides the following parameters for configuration:
- `initial_source` - source type: `list` or `directory`
- `configuration` - options for specified source
- `list`:
- `hash_list` - list of HEX encoded hashes
- `invert` - working mode: `true` - black list, `false` - white list
- `storage_ctx` - name of storage _context_ where to store data.
It may be redis hash key, DB table name etc.
- `directory`:
- `path` - directory to watch
- `invert` and `storage_ctx` has the same meanins as `list`'s options
Configuration example:
An example config might look like this:
```yaml
mochi:
prehooks:
- name: torrent approval
options:
initial_source: list
configuration:
hash_list: ["AAA", "BBB"]
invert: false
storage_ctx: APPROVED_HASH
```
+18 -21
View File
@@ -1,17 +1,17 @@
# Redis Storage
This storage implementation separates Chihaya from its storage service. Chihaya achieves HA by storing all peer data in
Redis. Multiple instances of Chihaya can use the same redis instance concurrently. The storage service can get HA by
clustering. If one instance of Chihaya goes down, peer data will still be available in Redis.
This storage implementation separates MoChi from its storage service. MoChi achieves HA by storing all peer data in
Redis. Multiple instances of MoChi can use the same redis instance concurrently. The storage service can get HA by
clustering. If one instance of MoChi goes down, peer data will still be available in Redis.
The HA of storage service is not considered here. In case Redis runs as a single node, peer data will be unavailable if
the node is down. You should consider setting up a Redis cluster for Chihaya in production.
the node is down. You should consider setting up a Redis sentinel (or KeyDB active-active replication) for MoChi in production.
This storage implementation is currently orders of magnitude slower than the in-memory implementation.
## Use Case
When one instance of Chihaya is down, other instances can continue serving peers from Redis.
When one instance of MoChi is down, other instances can continue serving peers from Redis.
## Configuration
@@ -57,27 +57,24 @@ time as value.
Here is an example:
```
- IPv4
- IPv4_S_<infohash 1>: <modification time>
- IPv4_L_<infohash 1>: <modification time>
- IPv4_S_<infohash 2>: <modification time>
- IPv4_S_<infohash 1>
- <peer 1 key>: <modification time>
- <peer 2 key>: <modification time>
- IPv4_L_<infohash 1>
- <peer 3 key>: <modification time>
- IPv4_S_<infohash 2>
- <peer 3 key>: <modification time>
- CHI_4_I
- CHI_4_S_<HASH1>
- CHI_4_L_<HASH1>
- CHI_4_S_<HASH1>
- <peer 1 key>: <modification time in unix nanos>
- <peer 2 key>: <modification time in unix nanos>
- CHI_4_L_<HASH2>
- <peer 3 key>: <modification time in unix nanos>
...
```
In this case, prometheus would record two swarms, three seeders, and one leecher. These three keys per address family
are used to record the count of swarms, seeders, and leechers.
```
- IPv4_infohash_count: 2
- IPv4_S_count: 3
- IPv4_L_count: 1
- CHI_4_S_C: "3"
- CHI_6_L_C: "1"
```
Note: IPv4_infohash_count has a different meaning compared to the `memory` storage:
It represents the number of infohashes reported by seeder, meaning that infohashes without seeders are not counted.
Note: `CHI_4_I` set has a different meaning compared to the `memory` storage:
It represents info hashes reported by seeder, meaning that info hashes without seeders are not counted.