diff --git a/README.md b/README.md index e73680d9d..af846e1f2 100644 --- a/README.md +++ b/README.md @@ -1,75 +1,95 @@ -# SATONOMICS +

+ + + + + kibō + + +

-![Image of the Satonomics Web App](./assets/latest.jpg) +

+ A better, FOSS, Bitcoin-only, self-hostable Glassnode. +

## Description -kibō (hope) is a better, FOSS, Bitcoin-only, self-hostable Glassnode. - While [mempool.space](https://mempool.space) gives a very micro view of the network where you can follow the journey of any address, this tool is the exact opposite and very complimentary by giving you a much more global/macro view of the flow and various dynamics of the network via thousands of charts. To promote even more transparency and trust in the network, this project is committed to making on-chain data accessible and verifiable by all, no matter your intentions or financial situation. That is why, the whole project is completely free, from code to services, including a real-time API with thousands and thousands of routes which can be used at will. **Having anyone be able to easily do a health-check of the network is incredibly important and should be wanted by every single bitcoiner.** -## Warning - -This project is in a very early stage. The web app will have bugs, the API might break and the data can definitely to be false or slightly false. - ## Donations -The project is a lot of work and being worked on full-time. It doesn't have any ads and solely relies on donations. If you find this project useful, any sat would really help make it even better and would be really appreciated. +This project was started as an answer to the outrageous pricing from Glassnode (and their third tier starting at $833.33/month !). -You can donate on the project's [Geyser Fund](https://geyser.fund/project/satonomics/). +But it is a lot of work and has been worked on _**full-time since November of 2023**_ and has also been operational since then without any ads. + +_**At the time of writing (2024-09-12), this project has made around 2,200,000 sats, which is around $1300 or $120/month. It's unsustainable.**_ + +So if you find this project useful, [please send some sats](https://geyser.fund/project/satonomics/), it would be really appreciated. + +[Geyser Fund](https://geyser.fund/project/satonomics/) + +## Warning + +This project is in a very early stage. Until more people look at the code and check the various computations, the datasets might be in the worst case completely false. ## Instances -Web App: - -- [app.satonomics.xyz](https://app.satonomics.xyz) - -API: - -- [api.satonomics.xyz](https://api.satonomics.xyz) -- [api-bkp.satonomics.xyz](https://api-bkp.satonomics.xyz) +- [kibo.money](https://kibo.money) +- [backup.kibo.money](https://backup.kibo.money) ## Structure - `parser`: The backbone of the project, it does most of the work by parsing and then computing datasets from the timechain. -- `server`: A small server which automatically creates routes to access through an API all created datasets. -- `app`: A web app which displays the generated datasets in various charts and dashboards. +- `website`: A web app which displays the generated datasets in various charts and dashboards. +- `server`: A small server which will serve the -## How to run +## Setup ### Requirements -- `rustup` +- 1 TB of free space (will use 60-80% of that) +- A running instance of bitcoin-core with txindex=1 and rpc credentials -### Parser +### Docker + +Coming soon + +### Manual + +#### Hardware + +#### 1. Rust ```bash -./run.sh --datadir=$HOME/Developer/bitcoin +# https://www.rust-lang.org/tools/install +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +# https://github.com/watchexec/cargo-watch?tab=readme-ov-file#install +cargo install cargo-watch --locked ``` -### Server +#### 2. Parser + +```bash +# The first run needs several information about your bitcoin-core config +./run.sh --datadir=$HOME/Developer/bitcoin --rpcuser=satoshi --rpcpassword=nakamoto + +# Next time you can just do: ./run.sh +# As everything is saved in +``` + +#### Server ```bash -# Install rustup -# Update ./run.sh if needed ./run.sh ``` Then the easiest to let others access your server is with `cloudflared` which will also cache requests. - -## Limitations - -- Needs to stop the node to parse the files (at least for now) -- Needs a **LOT** a disk space for databases (~700 GB for data from 2009 to mid 2024) - -## Goals / Philosophy - -Adjectives that describe what this project is or strives to be, in no particular order: +## Philosophy - **Best**: Replace Glassnode as the go to - **Diverse**: Have as many charts/datasets as possible and something for everyone @@ -83,46 +103,25 @@ Adjectives that describe what this project is or strives to be, in no particular - **Versatile**: You can view the data in charts, you can download the data, you can fetch the data via an API - **Accessible**: Free Website and API with all the datasets for everyone -## Milestones +## Logo -Big features that are planned, in no particular order: +The dove (borrowed from [svgrepo](https://www.svgrepo.com/svg/351969/dove) for now) represents _**hope**_ (kibō in japanese). + +The orange background represents Bitcoin and when in a circle, it also represents the sun, which means that while it's our hope for a better future, we still have to be careful with our collective goals and actions, to not end up like Icarus. + +## Roadmap -- **Homepage**: A landing page to explains the project and what it does - **More Datasets/Charts**: If a dataset can be computed, it should exist and have its related charts - **Dashboards**: For a quick and real-time view of the latest data of all the datasets -- **NOSTR integration**: First to save preferences, later to add some social functionnality -- **Datasets by block timestamp**: In addition to having datasets by block date and block height +- **Nostr integration**: First to save preferences, later to add some social functionnality +- **Datasets by block timestamp**: In addition to having datasets by date and height - **Descriptions**: Add text to describe all charts and what they mean -- **Start9 Add-on**: By making the whole suite much easier to self-host, it's quite rough right now +- **Start9 support**: By making the whole suite much easier to self-host, it's quite rough right now - **API Documentation**: Highly needed to explain what's what -_Maybe_: +## Iterations -- A Desktop app -- A mobile app - -## Brand - -- **Name**: Willing to change if someone thinks of something better ! -- **Logo**: Most likely a placeholder - -## Collaboration - -- Repositories: - - [Github](https://github.com/satonomics-org/satonomics) - - [Codeberg](https://codeberg.org/satonomics/satonomics) -- Issues: - - [Github](https://github.com/satonomics-org/satonomics/issues) - - [NOSTR](https://gitworkshop.dev/r/naddr1qq99xct5dahx7mtfvdesz9thwden5te0wp6hyurvv4ex2mrp0yhxxmmdqgsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03srqsqqqaueek2h03/issues) -- Proposals: - - [Github](https://github.com/satonomics-org/satonomics/pulls) - - [NOSTR](https://gitworkshop.dev/r/naddr1qq99xct5dahx7mtfvdesz9thwden5te0wp6hyurvv4ex2mrp0yhxxmmdqgsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03srqsqqqaueek2h03/proposals) - -## Proof of Work - -Aka: Previous iterations - -The initial idea was totally different yet morphed over time into what it is today: a fully FOSS self-hostable on-chain data generator. +A list of all the previous versions and ideas: - https://github.com/drgarlic/satonomics - https://github.com/drgarlic/satonomics-parser diff --git a/parser/stop.sh b/parser/stop.sh deleted file mode 100755 index 20ecf25dd..000000000 --- a/parser/stop.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/usr/bin/env bash - -# if [ "$(uname)" == "Darwin" ]; then -# if [ mdutil -s / | grep "disabled" ]; then -# sudo mdutil -a -i on -# fi -# fi - -bitcoin-cli -datadir=/Users/k/Developer/bitcoin stop diff --git a/server/Cargo.lock b/server/Cargo.lock index 9a60b8dbd..2a9828552 100644 --- a/server/Cargo.lock +++ b/server/Cargo.lock @@ -2482,6 +2482,7 @@ version = "0.4.0" dependencies = [ "axum", "bincode", + "chrono", "color-eyre", "derive_deref", "itertools", @@ -2492,7 +2493,6 @@ dependencies = [ "serde_json", "swc", "swc_common", - "swc_ecma_minifier", "tokio", "tower-http", ] diff --git a/server/Cargo.toml b/server/Cargo.toml index 5d5239dca..2df390c75 100644 --- a/server/Cargo.toml +++ b/server/Cargo.toml @@ -5,17 +5,17 @@ edition = "2021" [dependencies] axum = "0.7.5" -color-eyre = "0.6.3" -itertools = "0.13.0" -regex = "1.10.6" bincode = { git = "https://github.com/bincode-org/bincode.git" } +chrono = "0.4.38" +color-eyre = "0.6.3" +derive_deref = "1.1.1" +itertools = "0.13.0" +parser = { path = "../parser" } +regex = "1.10.6" reqwest = { version = "0.12.7", features = ["json"] } serde = { version = "1.0.210", features = ["derive"] } serde_json = { version = "1.0.128" } -tokio = { version = "1.40.0", features = ["full"] } -tower-http = { version = "0.5.2", features = ["compression-full"] } -parser = { path = "../parser" } -derive_deref = "1.1.1" swc = "0.286.0" swc_common = "0.37.5" -swc_ecma_minifier = "0.205.2" +tokio = { version = "1.40.0", features = ["full"] } +tower-http = { version = "0.5.2", features = ["compression-full"] } diff --git a/server/run.sh b/server/run.sh index 7ba09fa05..e3378de35 100755 --- a/server/run.sh +++ b/server/run.sh @@ -1,3 +1,5 @@ +#!/usr/bin/env bash + if cargo watch --help &> /dev/null; then TRIGGER="./in/datasets_len.txt" diff --git a/server/src/api/handlers/dataset.rs b/server/src/api/handlers/dataset.rs index 422e20bd7..d6f0c9d59 100644 --- a/server/src/api/handlers/dataset.rs +++ b/server/src/api/handlers/dataset.rs @@ -6,7 +6,7 @@ use axum::{ response::{IntoResponse, Response}, }; use color_eyre::{eyre::eyre, owo_colors::OwoColorize}; -use reqwest::{header::HOST, StatusCode}; +use reqwest::StatusCode; use serde::Deserialize; use parser::{log, Date, DateMap, Height, HeightMap, MapChunkId, HEIGHT_MAP_CHUNK_SIZE, OHLC}; @@ -14,10 +14,11 @@ use parser::{log, Date, DateMap, Height, HeightMap, MapChunkId, HEIGHT_MAP_CHUNK use crate::{ api::structs::{Chunk, Kind, Route}, header_map::HeaderMapUtils, - response::typed_value_to_response, AppState, }; +use super::response::typed_value_to_response; + #[derive(Deserialize)] pub struct Params { chunk: Option, @@ -168,13 +169,8 @@ where let offsetted_to_url = |offseted| { datasets.get(&ChunkId::from_usize(offseted)).map(|_| { - let host = headers[HOST].to_str().unwrap(); - let scheme = if host.contains("0.0.0.0") || host.contains("localhost") { - "http" - } else { - "https" - }; - + let scheme = headers.get_scheme(); + let host = headers.get_host(); format!("{scheme}://{host}{}?chunk={offseted}", route.url_path) }) }; diff --git a/server/src/api/handlers/fallback.rs b/server/src/api/handlers/fallback.rs index cc16d7a99..eecd1b3c7 100644 --- a/server/src/api/handlers/fallback.rs +++ b/server/src/api/handlers/fallback.rs @@ -1,7 +1,9 @@ use axum::{extract::State, http::HeaderMap, response::Response}; use reqwest::header::HOST; -use crate::{response::generic_to_reponse, AppState}; +use crate::AppState; + +use super::response::generic_to_reponse; pub async fn fallback(headers: HeaderMap, State(app_state): State) -> Response { generic_to_reponse( diff --git a/server/src/api/handlers/mod.rs b/server/src/api/handlers/mod.rs index 12f024511..78a45c0da 100644 --- a/server/src/api/handlers/mod.rs +++ b/server/src/api/handlers/mod.rs @@ -1,5 +1,6 @@ mod dataset; mod fallback; +mod response; pub use dataset::*; pub use fallback::*; diff --git a/server/src/response.rs b/server/src/api/handlers/response.rs similarity index 100% rename from server/src/response.rs rename to server/src/api/handlers/response.rs diff --git a/server/src/api/structs/routes.rs b/server/src/api/structs/routes.rs index 2a6b7d36d..6a1fe2e72 100644 --- a/server/src/api/structs/routes.rs +++ b/server/src/api/structs/routes.rs @@ -101,7 +101,7 @@ impl Routes { .map(|route| format!("\"{}\"", route.url_path)) .join(" | "); - format!("// This file is auto generated by the server\n// Manual changes are forbidden\n\ntype {}Path = {};\n", name, paths) + format!("export type {}Path = {};\n", name, paths) }; let date_type = map_to_type("Date", &self.date); @@ -112,7 +112,7 @@ impl Routes { fs::write( format!("{WEBSITE_TYPES_PATH}/paths.d.ts"), - format!("{date_type}\n{height_type}\n{last_type}"), + format!("// This file is auto generated by the server\n// Manual changes are forbidden\n\n{date_type}\n{height_type}\n{last_type}"), ) .unwrap(); } diff --git a/server/src/header_map.rs b/server/src/header_map.rs index 946f0ebbd..639a66bb0 100644 --- a/server/src/header_map.rs +++ b/server/src/header_map.rs @@ -1,14 +1,23 @@ use std::path::Path; use axum::http::{header, HeaderMap}; +use chrono::{DateTime, Utc}; use parser::log; +use reqwest::header::HOST; -const STALE_IF_ERROR: u64 = 31_536_000; // 1 Year +const STALE_IF_ERROR: u64 = 30_000_000; // 1 Year ish pub trait HeaderMapUtils { + fn get_scheme(&self) -> &str; + fn get_host(&self) -> &str; + fn check_if_host_is_any_local(&self) -> bool; + fn check_if_host_is_0000(&self) -> bool; + fn check_if_host_is_localhost(&self) -> bool; + fn insert_cors(&mut self); fn insert_cache_control(&mut self, max_age: u64, stale_while_revalidate: u64); + fn insert_last_modified(&mut self, date: DateTime); fn insert_content_type(&mut self, path: &Path); fn insert_content_type_image_icon(&mut self); @@ -24,6 +33,30 @@ pub trait HeaderMapUtils { } impl HeaderMapUtils for HeaderMap { + fn get_scheme(&self) -> &str { + if self.check_if_host_is_any_local() { + "http" + } else { + "https" + } + } + + fn get_host(&self) -> &str { + self[HOST].to_str().unwrap() + } + + fn check_if_host_is_any_local(&self) -> bool { + self.check_if_host_is_localhost() || self.check_if_host_is_0000() + } + + fn check_if_host_is_0000(&self) -> bool { + self.get_host().contains("0.0.0.0") + } + + fn check_if_host_is_localhost(&self) -> bool { + self.get_host().contains("localhost") + } + fn insert_cors(&mut self) { self.insert(header::ACCESS_CONTROL_ALLOW_ORIGIN, "*".parse().unwrap()); self.insert(header::ACCESS_CONTROL_ALLOW_HEADERS, "*".parse().unwrap()); @@ -39,6 +72,12 @@ impl HeaderMapUtils for HeaderMap { ); } + fn insert_last_modified(&mut self, date: DateTime) { + let formatted = date.format("%a, %d %b %Y %H:%M:%S GMT").to_string(); + + self.insert(header::LAST_MODIFIED, formatted.parse().unwrap()); + } + // https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types fn insert_content_type(&mut self, path: &Path) { match path.extension().unwrap().to_str().unwrap() { @@ -60,24 +99,15 @@ impl HeaderMapUtils for HeaderMap { } fn insert_content_type_image_icon(&mut self) { - self.insert( - header::CONTENT_TYPE, - "image/x-icon".parse().unwrap(), - ); + self.insert(header::CONTENT_TYPE, "image/x-icon".parse().unwrap()); } fn insert_content_type_image_jpeg(&mut self) { - self.insert( - header::CONTENT_TYPE, - "image/jpeg".parse().unwrap(), - ); + self.insert(header::CONTENT_TYPE, "image/jpeg".parse().unwrap()); } fn insert_content_type_image_png(&mut self) { - self.insert( - header::CONTENT_TYPE, - "image/png".parse().unwrap(), - ); + self.insert(header::CONTENT_TYPE, "image/png".parse().unwrap()); } fn insert_content_type_application_javascript(&mut self) { @@ -92,7 +122,10 @@ impl HeaderMapUtils for HeaderMap { } fn insert_content_type_application_manifest_json(&mut self) { - self.insert(header::CONTENT_TYPE, "application/manifest+json".parse().unwrap()); + self.insert( + header::CONTENT_TYPE, + "application/manifest+json".parse().unwrap(), + ); } fn insert_content_type_text_css(&mut self) { diff --git a/server/src/main.rs b/server/src/main.rs index bbe18436e..3aa94b4f6 100644 --- a/server/src/main.rs +++ b/server/src/main.rs @@ -10,7 +10,6 @@ use website::WebsiteRoutes; mod api; mod header_map; -mod response; mod website; #[derive(Clone, Debug, Default, Serialize)] diff --git a/server/src/website/handlers/file.rs b/server/src/website/handlers/file.rs index 815a46452..bbb1f1fb6 100644 --- a/server/src/website/handlers/file.rs +++ b/server/src/website/handlers/file.rs @@ -9,6 +9,7 @@ use axum::{ http::HeaderMap, response::{IntoResponse, Response}, }; +use chrono::{DateTime, Utc}; use parser::log; use reqwest::StatusCode; @@ -38,17 +39,22 @@ pub async fn file_handler(headers: HeaderMap, path: extract::Path) -> Re } } - path_to_response(&path) + path_to_response(headers, &path) } pub async fn index_handler(headers: HeaderMap) -> Response { - path_to_response(&str_to_path("index.html")) + path_to_response(headers, &str_to_path("index.html")) } -fn path_to_response(path: &Path) -> Response { +fn path_to_response(headers: HeaderMap, path: &Path) -> Response { let mut response; - if path.extension().unwrap() == "js" { + let time = path.metadata().unwrap().modified().unwrap(); + let date: DateTime = time.into(); + + let is_localhost = headers.check_if_host_is_localhost(); + + if !is_localhost && path.extension().unwrap() == "js" { let content = minify_js(path); response = Response::new(content.into()); @@ -67,6 +73,12 @@ fn path_to_response(path: &Path) -> Response { headers.insert_cors(); headers.insert_content_type(path); + if !is_localhost { + headers.insert_cache_control(10, 50); + } + + headers.insert_last_modified(date); + response } diff --git a/website/generate-icons.sh b/website/generate-icons.sh index 02a8e4f5a..5c9bf921e 100755 --- a/website/generate-icons.sh +++ b/website/generate-icons.sh @@ -1,3 +1,5 @@ +#!/usr/bin/env bash + pwa-asset-generator "../assets/logo-dove-orange.svg" "./assets" \ --index "./assets/index.html" \ --manifest "./manifest.webmanifest" \ diff --git a/website/index.html b/website/index.html index 1b1bede17..2591cf0d0 100644 --- a/website/index.html +++ b/website/index.html @@ -1,4 +1,4 @@ - + @@ -11,22 +11,14 @@ name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" /> - -