mirror of
https://github.com/bitcoinresearchkit/brk.git
synced 2026-06-13 16:33:30 -07:00
parser: price fetch fixes
This commit is contained in:
k
+2
-2
@@ -2,11 +2,11 @@
|
||||
|
||||
## v. 0.X.Y | WIP
|
||||
|
||||
|
||||
|
||||
|
||||
## v. 0.4.0 | WIP
|
||||
|
||||
|
||||
|
||||
|
||||
### Brand
|
||||
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2024 Satonomics
|
||||
Copyright (c) 2024 kibō
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
<p align="center">
|
||||
<a href="https://kibo.money" target="_blank">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/satonomics-org/satonomics/main/assets/logo-full-dark.svg">
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/satonomics-org/satonomics/main/assets/logo-full-light.svg">
|
||||
<img alt="kibō" src="https://raw.githubusercontent.com/satonomics-org/satonomics/main/assets/logo-full-light.svg" width="300" height="auto" style="max-width: 100%;">
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/kibo-money/kibo/main/assets/logo-full-dark.svg">
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/kibo-money/kibo/main/assets/logo-full-light.svg">
|
||||
<img alt="kibō" src="https://raw.githubusercontent.com/kibo-money/kibo/main/assets/logo-full-light.svg" width="300" height="auto" style="max-width: 100%;">
|
||||
</picture>
|
||||
</a>
|
||||
</p>
|
||||
@@ -12,15 +12,15 @@
|
||||
<span>Bitcoin is our only <b><i>hope</i></b> for a better future.</span>
|
||||
</p>
|
||||
|
||||
*Formerly **Satonomics***
|
||||
|
||||
## Description
|
||||
|
||||
While [mempool.space](https://mempool.space) gives a very detailed view of the network where you can follow the journey of any address, this tool is the exact opposite and very complimentary by giving you a much more global/macro view of the flow and various dynamics of the network via thousands of charts.
|
||||
**kibō** (*hope* in japanese, formerly Satonomics) is a suite of tools that aims to help you understand Bitcoin's various dynamics. To do that, there is a wide number of charts and datasets with a scale by date but also by height free for you to explore. Which allows you to verify an incredible number of things, from the number of UTXOs to the repartition of the supply between different groups over time, with many things in between and it's all made possible thanks to Bitcoin's transparency. Whether you're an enthusiast, a researcher, a miner, an analyst, a trader, a skeptic or just curious, there is something new to learn for everyone !
|
||||
|
||||
To promote even more transparency and trust in the network, this project is committed to making on-chain data accessible and verifiable by all, no matter your intentions or financial situation. That is why, the whole project is completely free, from code to services, including a real-time API with thousands and thousands of routes which can be used at will.
|
||||
While it's not the first tool trying to solve this problem, it's the first that is completely free, open-source and self-hostable. Which is very important as, just like for Bitcoin itself, I strongly believe that everyone should have access to this kind of data.
|
||||
|
||||
**Having anyone be able to easily do a health-check of the network is incredibly important and should be wanted by every single bitcoiner.**
|
||||
If you are a user of [mempool.space](https://mempool.space), you'll find this to be very complimentary, as it's a global and macro view of the chain over time instead.
|
||||
|
||||
If we want the world to move towards and, in the end, to be on a Bitcoin standard, we must have tools like this at our disposal.
|
||||
|
||||
## Donations
|
||||
|
||||
@@ -30,15 +30,15 @@ But it is a lot of work and has been worked on **full-time since November of 202
|
||||
|
||||
**At the time of writing (2024-09-12), this project has made around 2,200,000 sats, which is around $1300 or $120/month. While I'm very grateful for all donations, it's sadly unsustainable.**
|
||||
|
||||
So if you find this project useful, [please send some sats](https://geyser.fund/project/satonomics/), it would be really appreciated.
|
||||
So if you find this project useful, [please send some sats](https://geyser.fund/project/kibo/), it would be really appreciated.
|
||||
|
||||
If you're a potential sponsor, feel free to contact me in Nostr !
|
||||
|
||||
[Geyser Fund](https://geyser.fund/project/satonomics/)
|
||||
[Geyser Fund](https://geyser.fund/project/kibo/)
|
||||
|
||||
## Warning
|
||||
|
||||
This project is in a very early stage. Until more people look at the code and check the various computations in it, the datasets might be, in the worst case, completely false.
|
||||
This project is still in an early stage. Until more people look at the code and check the various computations in it, the datasets might be, in the worst case, completely false.
|
||||
|
||||
## Instances
|
||||
|
||||
@@ -55,6 +55,7 @@ This project is in a very early stage. Until more people look at the code and ch
|
||||
|
||||
### Requirements
|
||||
|
||||
- At least 16 GB of RAM
|
||||
- 1 TB of free space (will use 60-80% of that)
|
||||
- A running instance of bitcoin-core with txindex=1 and rpc credentials
|
||||
- Git
|
||||
@@ -110,7 +111,7 @@ For the first launch, the parser will need several information such as:
|
||||
|
||||
Everything will be saved in a `config.toml` file, which will allow you to simply run `./run.sh` next time
|
||||
|
||||
Here's an example
|
||||
Here's an example
|
||||
|
||||
```bash
|
||||
./run.sh --datadir=$HOME/Developer/bitcoin --rpcuser=satoshi --rpcpassword=nakamoto
|
||||
@@ -130,6 +131,17 @@ And start it also with the `run.sh` script instead of `cargo run -r`
|
||||
|
||||
Then the easiest to let others access your server is to use `cloudflared` which will also cache requests. For more information go to: https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/
|
||||
|
||||
## Roadmap
|
||||
|
||||
- **More Datasets/Charts**: If a dataset can be computed, it should exist and have its related charts
|
||||
- **Simulations**: Such as DCA simulation over a period of time with various strategies
|
||||
- **Dashboards**: For a quick and real-time view of the latest data of all the datasets
|
||||
- **Nostr integration**: First to save preferences, later to add some social functionnality
|
||||
- **Descriptions**: Add text to describe all charts and what they mean
|
||||
- **Start9 support**: By making the whole suite much easier to self-host, it's quite rough right now
|
||||
- **API Documentation**: Highly needed to explain what's what
|
||||
- **Datasets by block timestamp**: In addition to having datasets by date and height
|
||||
|
||||
## Philosophy
|
||||
|
||||
- **Best**: Replace Glassnode as the go to
|
||||
@@ -156,17 +168,6 @@ The dove (borrowed from [svgrepo](https://www.svgrepo.com/svg/351969/dove) for n
|
||||
|
||||
The orange background is a wink to Bitcoin and when in a circle, it also represents the sun, which means that while it's our hope for a better future, we still have to be careful with our collective goals and actions, to not end up like Icarus.
|
||||
|
||||
## Roadmap
|
||||
|
||||
- **More Datasets/Charts**: If a dataset can be computed, it should exist and have its related charts
|
||||
- **Simulations**: Such as DCA simulation over a period of time with various strategies
|
||||
- **Dashboards**: For a quick and real-time view of the latest data of all the datasets
|
||||
- **Nostr integration**: First to save preferences, later to add some social functionnality
|
||||
- **Descriptions**: Add text to describe all charts and what they mean
|
||||
- **Start9 support**: By making the whole suite much easier to self-host, it's quite rough right now
|
||||
- **API Documentation**: Highly needed to explain what's what
|
||||
- **Datasets by block timestamp**: In addition to having datasets by date and height
|
||||
|
||||
## Infrastructure
|
||||
|
||||
Here's the current infrastructure of the main instance and its backup.
|
||||
@@ -179,9 +180,9 @@ Though it's recommended to change to default **Browser Cache TTL** configuration
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/satonomics-org/satonomics/main/assets/infrastructure-dark.svg">
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/satonomics-org/satonomics/main/assets/infrastructure-light.svg">
|
||||
<img alt="kibō" src="https://raw.githubusercontent.com/satonomics-org/satonomics/main/assets/infrastructure-light.svg" width="768" height="auto" style="max-width: 100%;">
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/kibo-money/kibo/main/assets/infrastructure-dark.svg">
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/kibo-money/kibo/main/assets/infrastructure-light.svg">
|
||||
<img alt="kibō" src="https://raw.githubusercontent.com/kibo-money/kibo/main/assets/infrastructure-light.svg" width="768" height="auto" style="max-width: 100%;">
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ use color_eyre::eyre::Error;
|
||||
pub use ohlc::*;
|
||||
|
||||
use crate::{
|
||||
price::{Binance, Kraken, Satonomics},
|
||||
price::{Binance, Kibo, Kraken},
|
||||
structs::{
|
||||
AnyBiMap, AnyDateMap, BiMap, Date, DateMap, DateMapChunkId, Height, HeightMapChunkId,
|
||||
MapKey,
|
||||
@@ -28,8 +28,8 @@ pub struct PriceDatasets {
|
||||
binance_1mn: Option<BTreeMap<u32, OHLC>>,
|
||||
binance_daily: Option<BTreeMap<Date, OHLC>>,
|
||||
binance_har: Option<BTreeMap<u32, OHLC>>,
|
||||
satonomics_by_height: BTreeMap<HeightMapChunkId, Vec<OHLC>>,
|
||||
satonomics_by_date: BTreeMap<DateMapChunkId, BTreeMap<Date, OHLC>>,
|
||||
kibo_by_height: BTreeMap<HeightMapChunkId, Vec<OHLC>>,
|
||||
kibo_by_date: BTreeMap<DateMapChunkId, BTreeMap<Date, OHLC>>,
|
||||
|
||||
// Inserted
|
||||
pub ohlcs: BiMap<OHLC>,
|
||||
@@ -94,8 +94,8 @@ impl PriceDatasets {
|
||||
binance_har: None,
|
||||
kraken_1mn: None,
|
||||
kraken_daily: None,
|
||||
satonomics_by_height: BTreeMap::default(),
|
||||
satonomics_by_date: BTreeMap::default(),
|
||||
kibo_by_height: BTreeMap::default(),
|
||||
kibo_by_date: BTreeMap::default(),
|
||||
|
||||
ohlcs: BiMap::new_json(1, price_path),
|
||||
closes: BiMap::new_bin(1, &f("close")),
|
||||
@@ -314,7 +314,7 @@ impl PriceDatasets {
|
||||
let ohlc = self
|
||||
.get_from_daily_kraken(&date)
|
||||
.or_else(|_| self.get_from_daily_binance(&date))
|
||||
.or_else(|_| self.get_from_date_satonomics(&date))?;
|
||||
.or_else(|_| self.get_from_date_kibo(&date))?;
|
||||
|
||||
self.ohlcs.date.insert(date, ohlc);
|
||||
|
||||
@@ -322,25 +322,25 @@ impl PriceDatasets {
|
||||
}
|
||||
}
|
||||
|
||||
fn get_from_date_satonomics(&mut self, date: &Date) -> color_eyre::Result<OHLC> {
|
||||
fn get_from_date_kibo(&mut self, date: &Date) -> color_eyre::Result<OHLC> {
|
||||
let chunk_id = date.to_chunk_id();
|
||||
|
||||
#[allow(clippy::map_entry)]
|
||||
if !self.satonomics_by_date.contains_key(&chunk_id)
|
||||
if !self.kibo_by_date.contains_key(&chunk_id)
|
||||
|| self
|
||||
.satonomics_by_date
|
||||
.kibo_by_date
|
||||
.get(&chunk_id)
|
||||
.unwrap()
|
||||
.last_key_value()
|
||||
.unwrap()
|
||||
.0
|
||||
<= date
|
||||
< date
|
||||
{
|
||||
self.satonomics_by_date
|
||||
.insert(chunk_id, Satonomics::fetch_date_prices(chunk_id)?);
|
||||
self.kibo_by_date
|
||||
.insert(chunk_id, Kibo::fetch_date_prices(chunk_id)?);
|
||||
}
|
||||
|
||||
self.satonomics_by_date
|
||||
self.kibo_by_date
|
||||
.get(&chunk_id)
|
||||
.unwrap()
|
||||
.get(date)
|
||||
@@ -379,7 +379,7 @@ impl PriceDatasets {
|
||||
.last_key_value()
|
||||
.unwrap()
|
||||
.0
|
||||
<= date
|
||||
< date
|
||||
{
|
||||
self.binance_daily.replace(Binance::fetch_daily_prices()?);
|
||||
}
|
||||
@@ -428,12 +428,11 @@ impl PriceDatasets {
|
||||
.unwrap_or_else(|_| {
|
||||
self.get_from_har_binance(timestamp, previous_timestamp)
|
||||
.unwrap_or_else(|_| {
|
||||
self.get_from_height_satonomics(&height)
|
||||
.unwrap_or_else(|_| {
|
||||
let date = Date::from_timestamp(timestamp);
|
||||
self.get_from_height_kibo(&height).unwrap_or_else(|_| {
|
||||
let date = Date::from_timestamp(timestamp);
|
||||
|
||||
panic!(
|
||||
"Can't find the price for: height: {height} - date: {date}
|
||||
panic!(
|
||||
"Can't find the price for: height: {height} - date: {date}
|
||||
1mn APIs are limited to the last 16 hours for Binance's and the last 10 hours for Kraken's
|
||||
How to fix this:
|
||||
1. Go to https://www.binance.com/en/trade/BTC_USDT?type=spot
|
||||
@@ -446,8 +445,8 @@ How to fix this:
|
||||
8. Export to a har file (if there is no explicit button, click on the cog button)
|
||||
9. Move the file to 'parser/imports/binance.har'
|
||||
"
|
||||
)
|
||||
})
|
||||
)
|
||||
})
|
||||
})
|
||||
})
|
||||
});
|
||||
@@ -457,24 +456,24 @@ How to fix this:
|
||||
Ok(ohlc)
|
||||
}
|
||||
|
||||
fn get_from_height_satonomics(&mut self, height: &Height) -> color_eyre::Result<OHLC> {
|
||||
fn get_from_height_kibo(&mut self, height: &Height) -> color_eyre::Result<OHLC> {
|
||||
let chunk_id = height.to_chunk_id();
|
||||
|
||||
#[allow(clippy::map_entry)]
|
||||
if !self.satonomics_by_height.contains_key(&chunk_id)
|
||||
|| ((chunk_id.to_usize() + self.satonomics_by_height.get(&chunk_id).unwrap().len())
|
||||
if !self.kibo_by_height.contains_key(&chunk_id)
|
||||
|| ((chunk_id.to_usize() + self.kibo_by_height.get(&chunk_id).unwrap().len())
|
||||
<= height.to_usize())
|
||||
{
|
||||
self.satonomics_by_height
|
||||
.insert(chunk_id, Satonomics::fetch_height_prices(chunk_id)?);
|
||||
self.kibo_by_height
|
||||
.insert(chunk_id, Kibo::fetch_height_prices(chunk_id)?);
|
||||
}
|
||||
|
||||
self.satonomics_by_height
|
||||
self.kibo_by_height
|
||||
.get(&chunk_id)
|
||||
.unwrap()
|
||||
.get(height.to_serialized_key().to_usize())
|
||||
.cloned()
|
||||
.ok_or(Error::msg("Couldn't find height in satonomics"))
|
||||
.ok_or(Error::msg("Couldn't find height in kibo"))
|
||||
}
|
||||
|
||||
fn get_from_1mn_kraken(
|
||||
|
||||
@@ -122,7 +122,7 @@ impl Binance {
|
||||
// [timestamp, open, high, low, close, volume, ...]
|
||||
let array = value.as_array().unwrap();
|
||||
|
||||
let timestamp = (array.first().unwrap().as_u64().unwrap() / 1000) as u32;
|
||||
let timestamp = (array.first().unwrap().as_u64().unwrap() / 1_000) as u32;
|
||||
|
||||
let get_f32 = |index: usize| {
|
||||
array
|
||||
@@ -169,8 +169,9 @@ impl Binance {
|
||||
// [timestamp, open, high, low, close, volume, ...]
|
||||
let array = value.as_array().unwrap();
|
||||
|
||||
let date =
|
||||
Date::from_timestamp(array.first().unwrap().as_u64().unwrap() as u32);
|
||||
let date = Date::from_timestamp(
|
||||
(array.first().unwrap().as_u64().unwrap() / 1_000) as u32,
|
||||
);
|
||||
|
||||
let get_f32 = |index: usize| {
|
||||
array
|
||||
|
||||
@@ -12,24 +12,24 @@ use crate::{
|
||||
MapChunkId,
|
||||
};
|
||||
|
||||
pub struct Satonomics;
|
||||
pub struct Kibo;
|
||||
|
||||
const SATONOMICS_OFFICIAL_URL: &str = "https://api.satonomics.xyz";
|
||||
const SATONOMICS_OFFICIAL_BACKUP_URL: &str = "https://api-bkp.satonomics.xyz";
|
||||
const KIBO_OFFICIAL_URL: &str = "https://kibo.money/api";
|
||||
const KIBO_OFFICIAL_BACKUP_URL: &str = "https://backup.kibo.money/api";
|
||||
|
||||
const RETRIES: usize = 10;
|
||||
|
||||
impl Satonomics {
|
||||
impl Kibo {
|
||||
fn get_base_url(try_index: usize) -> &'static str {
|
||||
if try_index < RETRIES / 2 {
|
||||
SATONOMICS_OFFICIAL_URL
|
||||
KIBO_OFFICIAL_URL
|
||||
} else {
|
||||
SATONOMICS_OFFICIAL_BACKUP_URL
|
||||
KIBO_OFFICIAL_BACKUP_URL
|
||||
}
|
||||
}
|
||||
|
||||
pub fn fetch_height_prices(chunk_id: HeightMapChunkId) -> color_eyre::Result<Vec<OHLC>> {
|
||||
log("satonomics: fetch height prices");
|
||||
log("kibo: fetch height prices");
|
||||
|
||||
retry(
|
||||
|try_index| {
|
||||
@@ -62,7 +62,7 @@ impl Satonomics {
|
||||
}
|
||||
|
||||
pub fn fetch_date_prices(chunk_id: DateMapChunkId) -> color_eyre::Result<BTreeMap<Date, OHLC>> {
|
||||
log("satonomics: date height prices");
|
||||
log("kibo: fetch date prices");
|
||||
|
||||
retry(
|
||||
|try_index| {
|
||||
@@ -1,7 +1,7 @@
|
||||
mod binance;
|
||||
mod kibo;
|
||||
mod kraken;
|
||||
mod satonomics;
|
||||
|
||||
pub use binance::*;
|
||||
pub use kibo::*;
|
||||
pub use kraken::*;
|
||||
pub use satonomics::*;
|
||||
|
||||
@@ -180,7 +180,7 @@ where
|
||||
datasets.get(&ChunkId::from_usize(offseted)).map(|_| {
|
||||
let scheme = headers.get_scheme();
|
||||
let host = headers.get_host();
|
||||
format!("{scheme}://{host}{}?chunk={offseted}", route.url_path)
|
||||
format!("{scheme}://{host}/{}?chunk={offseted}", route.url_path)
|
||||
})
|
||||
};
|
||||
|
||||
|
||||
@@ -133,7 +133,7 @@ impl Routes {
|
||||
.map(|(key, route)| {
|
||||
(
|
||||
key.to_owned(),
|
||||
format!("{url}{}", route.url_path.to_owned()),
|
||||
format!("{url}/{}", route.url_path.to_owned()),
|
||||
)
|
||||
})
|
||||
.collect()
|
||||
|
||||
+4
-4
@@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html>
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
@@ -1308,7 +1308,7 @@
|
||||
localStorage.getItem(settingsThemeLocalStorageKey)
|
||||
);
|
||||
const preferredColorSchemeMatchMedia = window.matchMedia(
|
||||
"(prefers-color-scheme: dark)"
|
||||
"(prefers-color-scheme: dark)",
|
||||
);
|
||||
if (
|
||||
theme === "dark" ||
|
||||
@@ -1320,7 +1320,7 @@
|
||||
}
|
||||
|
||||
const backgroundColor = getComputedStyle(
|
||||
window.document.documentElement
|
||||
window.document.documentElement,
|
||||
).getPropertyValue("--background-color");
|
||||
const meta = window.document.createElement("meta");
|
||||
meta.name = "theme-color";
|
||||
@@ -2011,7 +2011,7 @@
|
||||
<p>
|
||||
Version:
|
||||
<a
|
||||
href="https://github.com/satonomics-org/satonomics/blob/main/CHANGELOG.md"
|
||||
href="https://github.com/kibo-money/kibo/blob/main/CHANGELOG.md"
|
||||
rel="noopener noreferrer"
|
||||
target="_blank"
|
||||
>
|
||||
|
||||
Vendored
+3
-3
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user