Mirror/mochi
1
0
Fork 0
mirror of https://github.com/sot-tech/mochi.git synced 2026-07-27 01:48:09 -07:00
Code Issues Packages Projects Releases Wiki Activity

Fix storage tests

Browse Source 
* reformat sources #JR1356
This commit is contained in:
Širhoe Biazhkovič
2021-11-26 18:43:23 +03:00
committed by Lawrence, Rendall
parent beb4736b86
commit 9122aefdd7
28 changed files with 1140 additions and 949 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+8 -8
View File
@@ -2,14 +2,14 @@
### Overview
BitTorrent clients send Announce and Scrape requests to a _Frontend_.
Frontends parse requests and write responses for the particular protocol they implement.
The _TrackerLogic_ interface is used to generate responses for requests and optionally perform a task after responding to a client.
A configurable chain of _PreHook_ and _PostHook_ middleware is used to construct an instance of TrackerLogic.
PreHooks are middleware that are executed before the response has been written.
After all PreHooks have executed, any missing response fields that are required are filled by reading out of the configured implementation of the _Storage_ interface.
PostHooks are asynchronous tasks that occur after a response has been delivered to the client.
Because they are unnecessary to for generating a response, updates to the Storage for a particular request are done asynchronously in a PostHook.
BitTorrent clients send Announce and Scrape requests to a _Frontend_. Frontends parse requests and write responses for
the particular protocol they implement. The _TrackerLogic_ interface is used to generate responses for requests and
optionally perform a task after responding to a client. A configurable chain of _PreHook_ and _PostHook_ middleware is
used to construct an instance of TrackerLogic. PreHooks are middleware that are executed before the response has been
written. After all PreHooks have executed, any missing response fields that are required are filled by reading out of
the configured implementation of the _Storage_ interface. PostHooks are asynchronous tasks that occur after a response
has been delivered to the client. Because they are unnecessary to for generating a response, updates to the Storage for
a particular request are done asynchronously in a PostHook.
### Diagram
docs/frontend.md
+45 -44
View File
@@ -1,15 +1,15 @@
# Frontends
A _Frontend_ is a component of Chihaya that serves a BitTorrent tracker on one protocol.
The frontend accepts, parses and sanitizes requests, passes them to the _Logic_ and writes responses to _Clients_.
A _Frontend_ is a component of Chihaya that serves a BitTorrent tracker on one protocol. The frontend accepts, parses
and sanitizes requests, passes them to the _Logic_ and writes responses to _Clients_.
This documentation first gives a high-level overview of Frontends and later goes into implementation specifics.
Users of Chihaya are expected to just read the first part - developers should read both.
This documentation first gives a high-level overview of Frontends and later goes into implementation specifics. Users of
Chihaya are expected to just read the first part - developers should read both.
## Functionality
A Frontend serves one protocol, for example HTTP ([BEP 3]) or UDP ([BEP 15]).
It listens for requests and usually answers each of them with one response, a basic overview of the control flow is:
A Frontend serves one protocol, for example HTTP ([BEP 3]) or UDP ([BEP 15]). It listens for requests and usually
answers each of them with one response, a basic overview of the control flow is:
1. Read the request.
2. Parse the request.
@@ -19,10 +19,9 @@ It listens for requests and usually answers each of them with one response, a ba
## Available Frontends
Chihaya ships with frontends for HTTP(S) and UDP.
The HTTP frontend uses Go's `http` package.
The UDP frontend implements both [old-opentracker-style] IPv6 and the IPv6 support specified in [BEP 15].
The advantage of the old opentracker style is that it contains a usable IPv6 `ip` field, to enable IP overrides in announces.
Chihaya ships with frontends for HTTP(S) and UDP. The HTTP frontend uses Go's `http` package. The UDP frontend
implements both [old-opentracker-style] IPv6 and the IPv6 support specified in [BEP 15]. The advantage of the old
opentracker style is that it contains a usable IPv6 `ip` field, to enable IP overrides in announces.
## Implementing a Frontend
@@ -30,8 +29,7 @@ This part is intended for developers.
### Implementation Specifics
A frontend should serve only one protocol.
It may serve that protocol on multiple transports or networks, if applicable.
A frontend should serve only one protocol. It may serve that protocol on multiple transports or networks, if applicable.
An example of that is the `http` Frontend, operating both on HTTP and HTTPS.
The typical control flow of handling announces, in more detail, is:
@@ -44,66 +42,69 @@ The typical control flow of handling announces, in more detail, is:
6. Send the response to the Client.
7. Pass the request and response to the `TrackerLogic`'s `AfterAnnounce` or `AfterScrape` method.
8. Finish, accept next request.
9. For invalid requests or errors during processing: Send an error response to the client.
This step may be skipped for suspected denial-of-service attacks.
The error response may contain information about the cause of the error.
Only errors where the Client is at fault should be explained, internal server errors should be returned without explanation.
Then finish, and accept the next request.
9. For invalid requests or errors during processing: Send an error response to the client. This step may be skipped for
suspected denial-of-service attacks. The error response may contain information about the cause of the error. Only
errors where the Client is at fault should be explained, internal server errors should be returned without
explanation. Then finish, and accept the next request.
#### Configuration
The frontend must be configurable using a single, exported struct.
The struct must have YAML annotations.
The struct must implement `log.Fielder` to be logged on startup.
The frontend must be configurable using a single, exported struct. The struct must have YAML annotations. The struct
must implement `log.Fielder` to be logged on startup.
#### Metrics
Frontends may provide runtime metrics, such as the number of requests or their duration.
Metrics must be reported using [Prometheus].
Frontends may provide runtime metrics, such as the number of requests or their duration. Metrics must be reported
using [Prometheus].
A frontend should provide at least the following metrics:
- The number of valid and invalid requests handled
- The average time it takes to handle a single request.
This request timing should be made optional using a config entry.
Requests should be separated by type, i.e. Scrapes, Announces, and other protocol-specific requests.
If the frontend serves multiple transports or networks, metrics for them should be separable.
- The number of valid and invalid requests handled
- The average time it takes to handle a single request. This request timing should be made optional using a config
entry.
Requests should be separated by type, i.e. Scrapes, Announces, and other protocol-specific requests. If the frontend
serves multiple transports or networks, metrics for them should be separable.
It is recommended to publish one Prometheus `HistogramVec` with:
- A name like `chihaya_PROTOCOL_response_duration_milliseconds`
- A value holding the duration in milliseconds of the reported request
- Labels for:
- `action` (= `announce`, `scrape`, ...)
- `address_family` (= `Unknown`, `IPv4`, `IPv6`, ...), if applicable
- `error` (= A textual representation of the error encountered during processing.)
Because `error` is expected to hold the textual representation of any error that occurred during the request, great care must be taken to ensure all error messages are static.
`error` must not contain any information directly taken from the request, e.g. the value of an invalid parameter.
This would cause this dimension of prometheus to explode, which slows down prometheus clients and reporters.
- `action` (= `announce`, `scrape`, ...)
- `address_family` (= `Unknown`, `IPv4`, `IPv6`, ...), if applicable
- `error` (= A textual representation of the error encountered during processing.)
Because `error` is expected to hold the textual representation of any error that occurred during the request,
great care must be taken to ensure all error messages are static.
`error` must not contain any information directly taken from the request, e.g. the value of an invalid parameter.
This would cause this dimension of prometheus to explode, which slows down prometheus clients and reporters.
#### Error Handling
Frontends should return `bittorrent.ClientError`s to the Client.
Frontends must not return errors that are not a `bittorrent.ClientError` to the Client.
A message like `internal server error` should be used instead.
Frontends should return `bittorrent.ClientError`s to the Client. Frontends must not return errors that are not
a `bittorrent.ClientError` to the Client. A message like `internal server error` should be used instead.
#### Request Sanitization
The `TrackerLogic` expects sanitized requests in order to function properly.
The `bittorrent` package provides the `SanitizeAnnounce` and `SanitizeScrape` functions to sanitize Announces and Scrapes, respectively.
This is the minimal required sanitization, every `AnnounceRequest` and `ScrapeRequest` must be sanitized this way.
The `bittorrent` package provides the `SanitizeAnnounce` and `SanitizeScrape` functions to sanitize Announces and
Scrapes, respectively. This is the minimal required sanitization, every `AnnounceRequest` and `ScrapeRequest` must be
sanitized this way.
Note that the `AnnounceRequest` struct contains booleans of the form `XProvided`, where `X` denotes an optional parameter of the BitTorrent protocol.
These should be set according to the values received by the Client.
Note that the `AnnounceRequest` struct contains booleans of the form `XProvided`, where `X` denotes an optional
parameter of the BitTorrent protocol. These should be set according to the values received by the Client.
#### Contexts
All methods of the `TrackerLogic` interface expect a `context.Context` as a parameter.
After a request is handled by `HandleAnnounce` without errors, the populated context returned must be used to call `AfterAnnounce`.
The same applies to Scrapes.
This way, a PreHook can communicate with a PostHook by setting a context value.
All methods of the `TrackerLogic` interface expect a `context.Context` as a parameter. After a request is handled
by `HandleAnnounce` without errors, the populated context returned must be used to call `AfterAnnounce`. The same
applies to Scrapes. This way, a PreHook can communicate with a PostHook by setting a context value.
[BEP 3]: http://bittorrent.org/beps/bep_0003.html
[BEP 15]: http://bittorrent.org/beps/bep_0015.html
[Prometheus]: https://prometheus.io/
[old-opentracker-style]: https://web.archive.org/web/20170503181830/http://opentracker.blog.h3q.com/2007/12/28/the-ipv6-situation/
docs/middleware/interval_variation.md
+8 -6
View File
@@ -4,21 +4,23 @@ This package provides the announce middleware `interval variation` which randomi
## Functionality
This middleware chooses random announces and modifies the `interval` and `min_interval` fields.
A random number of seconds are added to the `interval` field and, if desired, also to the `min_interval` field.
This middleware chooses random announces and modifies the `interval` and `min_interval` fields. A random number of
seconds are added to the `interval` field and, if desired, also to the `min_interval` field.
Note that if a response is picked for modification and `min_interval` should be changed as well, both `interval` and `min_interval` are modified by the same amount.
Note that if a response is picked for modification and `min_interval` should be changed as well, both `interval`
and `min_interval` are modified by the same amount.
## Use Case
Use this middleware to avoid recurring load spikes on the tracker.
By randomizing the announce interval, load spikes will flatten out after a few announce cycles.
Use this middleware to avoid recurring load spikes on the tracker. By randomizing the announce interval, load spikes
will flatten out after a few announce cycles.
## Configuration
This middleware provides the following parameters for configuration:
- `modify_response_probability` (float, >0, <= 1) indicates the probability by which a response will be chosen to have its announce intervals modified.
- `modify_response_probability` (float, >0, <= 1) indicates the probability by which a response will be chosen to have
its announce intervals modified.
- `max_increase_delta` (int, >0) sets an upper boundary (inclusive) for the amount of seconds added.
- `modify_min_interval` (boolean) whether to modify the `min_interval` field as well.
docs/storage/redis.md
+11 -15
View File
@@ -1,14 +1,11 @@
# Redis Storage
This storage implementation separates Chihaya from its storage service.
Chihaya achieves HA by storing all peer data in Redis.
Multiple instances of Chihaya can use the same redis instance concurrently.
The storage service can get HA by clustering.
If one instance of Chihaya goes down, peer data will still be available in Redis.
This storage implementation separates Chihaya from its storage service. Chihaya achieves HA by storing all peer data in
Redis. Multiple instances of Chihaya can use the same redis instance concurrently. The storage service can get HA by
clustering. If one instance of Chihaya goes down, peer data will still be available in Redis.
The HA of storage service is not considered here.
In case Redis runs as a single node, peer data will be unavailable if the node is down.
You should consider setting up a Redis cluster for Chihaya in production.
The HA of storage service is not considered here. In case Redis runs as a single node, peer data will be unavailable if
the node is down. You should consider setting up a Redis cluster for Chihaya in production.
This storage implementation is currently orders of magnitude slower than the in-memory implementation.
@@ -52,10 +49,10 @@ chihaya:
## Implementation
Seeders and Leechers for a particular InfoHash are stored within a redis hash.
The InfoHash is used as key, _peer keys_ are the fields, last modified times are values.
Peer keys are derived from peers and contain Peer ID, IP, and Port.
All the InfoHashes (swarms) are also stored in a redis hash, with IP family as the key, infohash as field, and last modified time as value.
Seeders and Leechers for a particular InfoHash are stored within a redis hash. The InfoHash is used as key, _peer keys_
are the fields, last modified times are values. Peer keys are derived from peers and contain Peer ID, IP, and Port. All
the InfoHashes (swarms) are also stored in a redis hash, with IP family as the key, infohash as field, and last modified
time as value.
Here is an example:
@@ -73,9 +70,8 @@ Here is an example:
- <peer 3 key>: <modification time>
```
In this case, prometheus would record two swarms, three seeders, and one leecher.
These three keys per address family are used to record the count of swarms, seeders, and leechers.
In this case, prometheus would record two swarms, three seeders, and one leecher. These three keys per address family
are used to record the count of swarms, seeders, and leechers.
```
- IPv4_infohash_count: 2