diff --git a/.gitignore b/.gitignore index 18b8eec54..6aa23a300 100644 --- a/.gitignore +++ b/.gitignore @@ -29,3 +29,4 @@ flamegraph.svg # AI CLAUDE.md +CLAUDE*.md diff --git a/README.md b/README.md index 06b3328fd..7ac328df1 100644 --- a/README.md +++ b/README.md @@ -52,23 +52,6 @@ The primary goal of this project is to be fully-featured and accessible for ever In contrast, existing alternatives tend to be either [very costly](https://studio.glassnode.com/pricing) or missing essential features, with the vast majority being closed-source and unverifiable, which fundamentally undermines the principles of Bitcoin. -## Crates - -- [`brk`](https://crates.io/crates/brk): A wrapper around all other `brk-*` crates -- [`brk_bundler`](https://crates.io/crates/brk_bundler): A thin wrapper around [`rolldown`](https://rolldown.rs/) -- [`brk_cli`](https://crates.io/crates/brk_cli): A command line interface to run a BRK instance -- [`brk_computer`](https://crates.io/crates/brk_computer): A Bitcoin dataset computer built on top of [`brk_indexer`](https://crates.io/crates/brk_indexer) -- [`brk_error`](https://crates.io/crates/brk_error): Errors used throughout BRK -- [`brk_fetcher`](https://crates.io/crates/brk_fetcher): A Bitcoin price fetcher -- [`brk_indexer`](https://crates.io/crates/brk_indexer): A Bitcoin indexer built on top of [`brk_parser`](https://crates.io/crates/brk_parser) -- [`brk_interface`](https://crates.io/crates/brk_interface): An interface to find and format data from BRK -- [`brk_logger`](https://crates.io/crates/brk_logger): A thin wrapper around [`env_logger`](https://crates.io/crates/env_logger) -- [`brk_mcp`](https://crates.io/crates/brk_mcp): A bridge for LLMs to access BRK -- [`brk_parser`](https://crates.io/crates/brk_parser): A very fast Bitcoin block parser and iterator built on top of [`bitcoin-rust`](https://crates.io/crates/bitcoin) -- [`brk_server`](https://crates.io/crates/brk_server): A server with an API for anything from BRK -- [`brk_store`](https://crates.io/crates/brk_store): A thin wrapper around [`fjall`](https://crates.io/crates/fjall) -- [`brk_structs`](https://crates.io/crates/brk_structs): Structs used throughout BRK - ## Hosting as a service If you'd like to have your own instance hosted for you please contact [hosting@bitcoinresearchkit.org](mailto:hosting@bitcoinresearchkit.org). @@ -90,6 +73,23 @@ Deepest gratitude to the [Open Sats](https://opensats.org/) public charity. Thei Heartfelt thanks go out to every donor on [Nostr](https://primal.net/p/npub1jagmm3x39lmwfnrtvxcs9ac7g300y3dusv9lgzhk2e4x5frpxlrqa73v44) and [Geyser.fund](https://geyser.fund/project/brk) whose support has ensured the availability of the [bitcoinresearchkit.org](https://bitcoinresearchkit.org) public instance. +## Crates + +- [`brk`](https://crates.io/crates/brk): A wrapper around all other `brk-*` crates +- [`brk_bundler`](https://crates.io/crates/brk_bundler): A thin wrapper around [`rolldown`](https://rolldown.rs/) +- [`brk_cli`](https://crates.io/crates/brk_cli): A command line interface to run a BRK instance +- [`brk_computer`](https://crates.io/crates/brk_computer): A Bitcoin dataset computer built on top of [`brk_indexer`](https://crates.io/crates/brk_indexer) +- [`brk_error`](https://crates.io/crates/brk_error): Errors used throughout BRK +- [`brk_fetcher`](https://crates.io/crates/brk_fetcher): A Bitcoin price fetcher +- [`brk_indexer`](https://crates.io/crates/brk_indexer): A Bitcoin indexer built on top of [`brk_parser`](https://crates.io/crates/brk_parser) +- [`brk_interface`](https://crates.io/crates/brk_interface): An interface to find and format data from BRK +- [`brk_logger`](https://crates.io/crates/brk_logger): A thin wrapper around [`env_logger`](https://crates.io/crates/env_logger) +- [`brk_mcp`](https://crates.io/crates/brk_mcp): A bridge for LLMs to access BRK +- [`brk_parser`](https://crates.io/crates/brk_parser): A very fast Bitcoin block parser and iterator built on top of [`bitcoin-rust`](https://crates.io/crates/bitcoin) +- [`brk_server`](https://crates.io/crates/brk_server): A server with an API for anything from BRK +- [`brk_store`](https://crates.io/crates/brk_store): A thin wrapper around [`fjall`](https://crates.io/crates/fjall) +- [`brk_structs`](https://crates.io/crates/brk_structs): Structs used throughout BRK + ## Donate [`bc1q09 8zsm89 m7kgyz e338vf ejhpdt 92ua9p 3peuve`](bitcoin:bc1q098zsm89m7kgyze338vfejhpdt92ua9p3peuve) diff --git a/crates/brk/README.md b/crates/brk/README.md new file mode 100644 index 000000000..153f7f1f4 --- /dev/null +++ b/crates/brk/README.md @@ -0,0 +1,66 @@ +# brk + +Bitcoin Research Kit (BRK) is a high-performance toolchain for parsing, indexing, computing, and serving Bitcoin blockchain data. It provides an alternative to services like Glassnode and mempool.space with a focus on self-hosting and open-source transparency. + +This is the main wrapper crate that re-exports all workspace crates through feature flags. + +## Crates + +- [`brk`](https://crates.io/crates/brk): A wrapper around all other `brk-*` crates +- [`brk_bundler`](https://crates.io/crates/brk_bundler): A thin wrapper around [`rolldown`](https://rolldown.rs/) +- [`brk_cli`](https://crates.io/crates/brk_cli): A command line interface to run a BRK instance +- [`brk_computer`](https://crates.io/crates/brk_computer): A Bitcoin dataset computer built on top of [`brk_indexer`](https://crates.io/crates/brk_indexer) +- [`brk_error`](https://crates.io/crates/brk_error): Errors used throughout BRK +- [`brk_fetcher`](https://crates.io/crates/brk_fetcher): A Bitcoin price fetcher +- [`brk_indexer`](https://crates.io/crates/brk_indexer): A Bitcoin indexer built on top of [`brk_parser`](https://crates.io/crates/brk_parser) +- [`brk_interface`](https://crates.io/crates/brk_interface): An interface to find and format data from BRK +- [`brk_logger`](https://crates.io/crates/brk_logger): A thin wrapper around [`env_logger`](https://crates.io/crates/env_logger) +- [`brk_mcp`](https://crates.io/crates/brk_mcp): A bridge for LLMs to access BRK +- [`brk_parser`](https://crates.io/crates/brk_parser): A very fast Bitcoin block parser and iterator built on top of [`bitcoin-rust`](https://crates.io/crates/bitcoin) +- [`brk_server`](https://crates.io/crates/brk_server): A server with an API for anything from BRK +- [`brk_store`](https://crates.io/crates/brk_store): A thin wrapper around [`fjall`](https://crates.io/crates/fjall) +- [`brk_structs`](https://crates.io/crates/brk_structs): Structs used throughout BRK + +## Features + +- `full` - Enable all workspace crates +- `bundler` - Re-export `brk_bundler` +- `cli` - Re-export `brk_cli` (always enabled) +- `computer` - Re-export `brk_computer` +- `error` - Re-export `brk_error` +- `fetcher` - Re-export `brk_fetcher` +- `indexer` - Re-export `brk_indexer` +- `interface` - Re-export `brk_interface` +- `logger` - Re-export `brk_logger` +- `mcp` - Re-export `brk_mcp` +- `parser` - Re-export `brk_parser` +- `server` - Re-export `brk_server` +- `store` - Re-export `brk_store` +- `structs` - Re-export `brk_structs` + +## Usage + +Add to your `Cargo.toml`: + +```toml +[dependencies] +brk = { version = "0.1", features = ["full"] } +``` + +Or enable specific components: + +```toml +[dependencies] +brk = { version = "0.1", features = ["parser", "indexer", "computer"] } +``` + +## Example + +```rust +use brk::{cli, parser, indexer, computer}; + +// Use individual crates as needed +let config = cli::Config::load()?; +let blocks = parser::BlockIterator::new(&config.bitcoin_path)?; +// ... +``` diff --git a/crates/brk/examples/main.rs b/crates/brk/examples/main.rs deleted file mode 100644 index f328e4d9d..000000000 --- a/crates/brk/examples/main.rs +++ /dev/null @@ -1 +0,0 @@ -fn main() {} diff --git a/crates/brk/src/lib.rs b/crates/brk/src/lib.rs index 8816f1147..f40f966b6 100644 --- a/crates/brk/src/lib.rs +++ b/crates/brk/src/lib.rs @@ -1,4 +1,4 @@ -#![doc = include_str!(concat!("../", env!("CARGO_PKG_README")))] +#![doc = include_str!("../README.md")] #[cfg(feature = "bundler")] #[doc(inline)] diff --git a/crates/brk_bundler/README.md b/crates/brk_bundler/README.md new file mode 100644 index 000000000..67c81565a --- /dev/null +++ b/crates/brk_bundler/README.md @@ -0,0 +1,63 @@ +# brk_bundler + +Web asset bundler built on Rolldown that provides JavaScript bundling, minification, and live reloading for BRK's web interface. This crate wraps Rolldown with BRK-specific functionality including automatic file watching, version injection, and hash-based cache busting for optimal web performance. + +## Features + +- **JavaScript bundling**: Bundles and minifies JavaScript with source maps +- **Live reloading**: Automatic rebuilding and file watching during development +- **Cache busting**: Hash-based filenames for browser cache invalidation +- **Version injection**: Automatic version injection into service workers +- **Asset copying**: Copies static assets from source to distribution directory +- **Development mode**: Optional watch mode for real-time development + +## Usage + +```rust +use brk_bundler::bundle; +use std::path::Path; + +async fn build_website() -> std::io::Result<()> { + let websites_path = Path::new("./websites"); + let source_folder = "src"; + let watch = false; // Set to true for development + + // Bundle the website + let dist_path = bundle(websites_path, source_folder, watch).await?; + + println!("Website built to: {}", dist_path.display()); + Ok(()) +} +``` + +## Build Process + +1. **Clean**: Removes existing distribution directory +2. **Copy**: Copies all source files to distribution directory +3. **Bundle**: Processes JavaScript entry point with Rolldown +4. **Process**: Updates HTML to reference hashed JS files +5. **Version**: Injects version strings into service workers +6. **Watch** (optional): Monitors files for changes and rebuilds + +## File Structure + +Expected source structure: +``` +websites/ +├── src/ +│ ├── index.html # Main HTML file +│ ├── service-worker.js # Service worker (optional) +│ ├── scripts/ +│ │ └── entry.js # JavaScript entry point +│ └── ... # Other static assets +└── dist/ # Generated distribution files +``` + +## Watch Mode + +When enabled, the bundler: +- Monitors source files for changes +- Automatically rebuilds JavaScript bundles +- Updates HTML with new hashed filenames +- Reprocesses service workers with version updates +- Copies modified static assets diff --git a/crates/brk_bundler/src/lib.rs b/crates/brk_bundler/src/lib.rs index e4b9874b7..482460417 100644 --- a/crates/brk_bundler/src/lib.rs +++ b/crates/brk_bundler/src/lib.rs @@ -1,3 +1,5 @@ +#![doc = include_str!("../README.md")] + use std::{ fs, io, path::{Path, PathBuf}, diff --git a/crates/brk_cli/README.md b/crates/brk_cli/README.md index a93bfaee8..fcd70aadd 100644 --- a/crates/brk_cli/README.md +++ b/crates/brk_cli/README.md @@ -1,90 +1,73 @@ -# BRK CLI +# brk_cli -

- - GitHub Repo stars - - - License - - - Version - - - Documentation - - Size - - Dependency status - - - Discord - - - Nostr - -

+Command line interface for running BRK (Bitcoin Research Kit) instances. Orchestrates the complete pipeline: parsing Bitcoin Core blocks, indexing data, computing analytics, and serving via HTTP API. -A command line interface to run a Bitcoin Research Kit instance. +## Overview -It's very customizable with all parameters from the underlying tools (crates) used inside. +**Core Operation**: Continuous loop that waits for Bitcoin node sync, indexes new blocks, computes analytics, and serves data via HTTP. -Run `brk -h` for more information. +**Key Components**: +- **Parser**: Reads Bitcoin Core block files +- **Indexer**: Processes and stores blockchain data in vecs/stores +- **Computer**: Computes analytics across 9 specialized domains +- **Server**: HTTP API with multiple output formats (JSON, CSV, TSV, Markdown) +- **Website**: Optional web interface (none/default/custom) ## Requirements -### Hardware +- **Bitcoin Core**: Fully synced node with RPC enabled +- **Storage**: ~32% of blockchain size (~233GB currently) +- **Memory**: ~7-8GB peak during indexing, ~4-5GB steady state +- **OS**: macOS or Linux (Ubuntu: `sudo apt install libssl-dev pkg-config`) -#### Recommended - -- [Latest base model Mac mini](https://www.apple.com/mac-mini/) -- [Thunderbolt 4 SSD enclosure](https://satechi.net/products/usb4-nvme-ssd-pro-enclosure/Z2lkOi8vc2hvcGlmeS9Qcm9kdWN0VmFyaWFudC80MDE4ODQ3MDA2NzI4OA==?queryID=7961465089021ee203a60db7e62e90d2) -- [2 TB NVMe SSD](https://shop.sandisk.com/products/ssd/internal-ssd/wd-black-sn850x-nvme-ssd?sku=WDS200T2X0E-00BCA0) - -#### Minimum - -To be determined - -### Software - -- [Bitcoin](https://bitcoin.org/en/full-node) -- [Rust](https://www.rust-lang.org/tools/install) -- Unix based operating system (Mac OS or Linux) - -> [!IMPORTANT] -> Ubuntu users need to install `open-ssl` via `sudo apt install libssl-dev pkg-config` - -## Download - -### Binaries - -You can find a pre-built binary for your operating system in the [releases page](https://github.com/bitcoinresearchkit/brk/releases/latest). - -### Cargo +## Installation ```bash -# Install -cargo install brk --locked # or `cargo install brk_cli`, the result is the same +# Binary +# https://github.com/bitcoinresearchkit/brk/releases/latest -# Update -cargo install brk --locked # or `cargo install-update -a` if you have `cargo-update` installed -``` +# Via cargo +cargo install brk --locked -### Source - -```bash +# From source git clone https://github.com/bitcoinresearchkit/brk.git -cd brk/crates/brk -cargo run -r +cd brk && cargo build --release ``` ## Usage -Run `brk -h` to view each available parameter and their respective description. +```bash +# First run (set configuration) +brk --brkdir ./my_data --fetch true --website default -> [!TIP] -> Every parameter set will be saved at `~/.brk/config.toml`, which allows you to simply run `brk` next time. +# Subsequent runs (uses saved config) +brk -## Tunnel +# View all options +brk --help +``` -The easiest way to let others access your server is to use `cloudflared` which will also cache requests. For more information see [Cloudflare Tunnel](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/) documentation. +## Configuration + +All options auto-save to `~/.brk/config.toml` for subsequent runs: + +```bash +# Core paths +--bitcoindir # Bitcoin directory (default: ~/.bitcoin) +--blocksdir # Block files (default: bitcoindir/blocks) +--brkdir # BRK output directory (default: ~/.brk) + +# Data sources +-F, --fetch # Enable price data fetching (default: true) +--exchanges # Use exchange APIs for prices (default: true) + +# Server +-w, --website # Web interface: none|default|custom + +# Bitcoin RPC +--rpcconnect # RPC host (default: localhost) +--rpcport # RPC port (default: 8332) +--rpccookiefile # Cookie auth (default: bitcoindir/.cookie) +--rpcuser # Username auth (alternative to cookie) +--rpcpassword # Password auth (alternative to cookie) +``` diff --git a/crates/brk_cli/src/config.rs b/crates/brk_cli/src/config.rs index d978cfb7c..3c606b5f3 100644 --- a/crates/brk_cli/src/config.rs +++ b/crates/brk_cli/src/config.rs @@ -74,6 +74,7 @@ pub struct Config { /// DEV: Activate checking address hashes for collisions when indexing, default: false, saved #[serde(default, deserialize_with = "default_on_error")] + #[arg(skip)] check_collisions: Option, } diff --git a/crates/brk_computer/README.md b/crates/brk_computer/README.md index bba172db6..845467fde 100644 --- a/crates/brk_computer/README.md +++ b/crates/brk_computer/README.md @@ -1,28 +1,64 @@ -# BRK Computer +# brk_computer -

- - GitHub Repo stars - - - License - - - Version - - - Documentation - - Size - - Dependency status - - - Discord - - - Nostr - -

+Bitcoin analytics engine that transforms indexed blockchain data into computed datasets and metrics. Uses a modular vector architecture with lazy computation and compressed storage for optimal performance. -A dataset computer, built on top of `brk_indexer` and `brk_fetcher`. It computes any dataset you can think of and if it doesn't feel free to create an issue. +## Overview + +Computes analytics across 9 specialized domains, each implementing the compute trait pattern: + +- **indexes** - Time-based indexing (date/height mappings, epoch calculations) +- **constants** - Baseline values for calculations +- **blocks** - Block analytics (sizes, intervals, transaction counts) +- **mining** - Mining economics (hashrate, difficulty, rewards) +- **transactions** - Transaction analysis (fees, sizes, patterns, RBF) +- **stateful** - UTXO tracking and accumulated state computations +- **cointime** - Coin age and time-based value analysis +- **fetched** - External price data integration (optional) +- **price** - OHLC data across timeframes (optional, requires fetched) +- **market** - Price correlations and market metrics (optional, requires price) + +**Computation order**: Fixed dependency chain ensures data consistency (indexes → constants → blocks → mining → fetched → price → transactions → market → stateful → cointime). + +**Storage**: Uses vecdb with lazy computation and compressed format for efficient disk usage and memory management. + +## Usage + +```rust +use brk_computer::Computer; +use brk_indexer::Indexer; +use brk_fetcher::Fetcher; + +// Basic setup - computes all domains except price/market +let indexer = Indexer::forced_import("./brk_data")?; +let mut computer = Computer::forced_import("./brk_data", &indexer, None)?; + +// With price data - enables market analytics +let fetcher = Some(Fetcher::import(true, None)?); +let mut computer = Computer::forced_import("./brk_data", &indexer, fetcher)?; + +// Compute all analytics from starting point +let starting_indexes = indexer.get_starting_indexes(); +computer.compute(&indexer, starting_indexes, &exit)?; + +// Access computed vectors +let all_vecs = computer.vecs(); // Returns Vec<&dyn AnyCollectableVec> +``` + +## Key Implementation Details + +- **Forced import pattern**: Single computer instance per output directory to prevent conflicts +- **Lazy computation**: Vectors computed on-demand, cached with dependency tracking +- **Incremental updates**: Only processes new data since last computation +- **Memory efficient**: ~100MB max via compressed storage and memory mapping +- **Exit handling**: Graceful shutdown support with computation state preservation + +## Performance + +Benchmarked on MacBook Pro M3 Pro: + +- **Initial computation**: ~6-7 hours for complete Bitcoin blockchain analysis +- **Storage efficiency**: All computed datasets total only ~40GB +- **Incremental updates**: 3-5 seconds per new block +- **Memory footprint**: Peak ~7-8GB during computation, ~100MB during operation + +The initial computation processes the entire blockchain history once to generate all analytical datasets. Subsequent updates are near-instant, making BRK suitable for real-time analysis and production deployments. diff --git a/crates/brk_computer/src/lib.rs b/crates/brk_computer/src/lib.rs index d3f8e6720..e25ec0fc1 100644 --- a/crates/brk_computer/src/lib.rs +++ b/crates/brk_computer/src/lib.rs @@ -1,7 +1,4 @@ #![doc = include_str!("../README.md")] -#![doc = "\n## Example\n\n```rust"] -#![doc = include_str!("../examples/computer.rs")] -#![doc = "```"] use std::path::Path; diff --git a/crates/brk_error/README.md b/crates/brk_error/README.md index 4fe182961..861ec55b3 100644 --- a/crates/brk_error/README.md +++ b/crates/brk_error/README.md @@ -1,28 +1,57 @@ -# BRK Core +# brk_error -

- - GitHub Repo stars - - - License - - - Version - - - Documentation - - Size - - Dependency status - - - Discord - - - Nostr - -

+Centralized error handling for the Bitcoin Research Kit that provides a unified error type and result type for consistent error propagation across all BRK crates. This crate consolidates errors from external dependencies and defines domain-specific error variants used throughout the BRK ecosystem. -A list of structs that are used throughout the project as units, think of `Date`, `Height`, `Sats`, `Txindex` or anything that can be either a key and/or a value of a dataset. +## Error Types + +### External Library Errors +- **IO**: Standard I/O operations (`std::io::Error`) +- **BitcoinRPC**: Bitcoin Core RPC client errors +- **Jiff**: Date/time parsing and manipulation errors +- **Fjall**: Key-value store errors +- **VecDB/SeqDB**: Vector database errors +- **Minreq**: HTTP client errors +- **SerdeJson**: JSON serialization/deserialization errors +- **ZeroCopy**: Memory layout conversion errors +- **SystemTime**: System time errors + +### Domain-Specific Errors +- **WrongLength**: Invalid data length +- **WrongAddressType**: Unsupported Bitcoin address type +- **UnindexableDate**: Date outside indexable range (before 2009-01-03) +- **QuickCacheError**: Cache operation failures +- **Str/String**: Custom error messages + +## Usage + +```rust +use brk_error::{Error, Result}; + +fn process_bitcoin_data() -> Result<()> { + // Operations that may fail with various error types + let data = std::fs::read("blocks.dat")?; // IO error + let parsed = parse_data(&data)?; // Custom error + Ok(()) +} + +fn parse_data(data: &[u8]) -> Result { + if data.len() < 80 { + return Err(Error::WrongLength); + } + // ... parsing logic + Ok(parsed_data) +} +``` + +## Type Alias + +The crate exports `Result` as the standard result type, allowing for concise error handling: + +```rust +use brk_error::Result; + +fn my_function() -> Result { + // Automatically uses brk_error::Error as the error type + Ok("success".to_string()) +} +``` diff --git a/crates/brk_error/src/lib.rs b/crates/brk_error/src/lib.rs index 89f48632e..e40f34eaf 100644 --- a/crates/brk_error/src/lib.rs +++ b/crates/brk_error/src/lib.rs @@ -1,3 +1,5 @@ +#![doc = include_str!("../README.md")] + use std::{ fmt::{self, Debug, Display}, io, result, time, diff --git a/crates/brk_fetcher/README.md b/crates/brk_fetcher/README.md index c86905969..56c14a9bb 100644 --- a/crates/brk_fetcher/README.md +++ b/crates/brk_fetcher/README.md @@ -1,28 +1,57 @@ -# BRK Fetcher +# brk_fetcher -

- - GitHub Repo stars - - - License - - - Version - - - Documentation - - Size - - Dependency status - - - Discord - - - Nostr - -

+Bitcoin price fetcher that retrieves historical OHLC (Open, High, Low, Close) data by date or block height from multiple sources including Binance, Kraken, and the main BRK instance. This crate provides a unified interface with automatic fallback between exchanges and retry logic for reliable price data collection. -A crate that can fetch the Bitcoin price, either by date or height, from Binance, Kraken and the main instance of BRK. +## Features + +- **Multiple data sources**: Binance, Kraken APIs, and BRK instance +- **Flexible queries**: Fetch prices by date or block height with timestamp +- **Automatic fallback**: Tries sources in order (Kraken → Binance → BRK) +- **Retry logic**: Built-in retry mechanism +- **Time resolution**: 1-minute and 1-day interval support +- **HAR file import**: Import Binance chart data from browser for historical prices + +## Usage + +```rust +use brk_fetcher::Fetcher; +use brk_structs::{Date, Height}; + +fn main() -> brk_error::Result<()> { + // Initialize fetcher with exchange APIs enabled + let mut fetcher = Fetcher::import(true, None)?; + + // Fetch price by date + let price = fetcher.get_date(Date::new(2025, 1, 15))?; + println!("Price on 2025-01-15: ${}", price.close.dollars()); + + // Fetch price by block height + let price = fetcher.get_height( + Height::new(900_000), + timestamp, + previous_timestamp, + )?; + println!("Price at block 900,000: ${}", price.close.dollars()); + + Ok(()) +} +``` + +## Individual Sources + +Each exchange can be used independently: + +```rust +use brk_fetcher::{Binance, Kraken, BRK}; + +// Fetch from specific exchanges +let binance_data = Binance::fetch_1d()?; +let kraken_data = Kraken::fetch_1mn()?; +let brk_data = BRK::default().get_from_height(Height::new(800_000))?; +``` + +## Limitations + +- **1-minute data**: Limited to last 16 hours (Binance) or 10 hours (Kraken) +- **Network dependent**: Requires internet connection for exchange APIs +- **Rate limits**: Subject to exchange API rate limiting diff --git a/crates/brk_fetcher/src/lib.rs b/crates/brk_fetcher/src/lib.rs index 4fddbed87..b36e3bd14 100644 --- a/crates/brk_fetcher/src/lib.rs +++ b/crates/brk_fetcher/src/lib.rs @@ -1,7 +1,4 @@ #![doc = include_str!("../README.md")] -#![doc = "\n## Example\n\n```rust"] -#![doc = include_str!("../examples/main.rs")] -#![doc = "```"] use std::{collections::BTreeMap, path::Path, thread::sleep, time::Duration}; diff --git a/crates/brk_indexer/README.md b/crates/brk_indexer/README.md index 889d61bad..77cc96ca9 100644 --- a/crates/brk_indexer/README.md +++ b/crates/brk_indexer/README.md @@ -1,69 +1,105 @@ -# BRK Indexer +# brk_indexer -

- - GitHub Repo stars - - - License - - - Version - - - Documentation - - Size - - Dependency status - - - Discord - - - Nostr - -

+Bitcoin blockchain indexer that processes raw block data from Bitcoin Core and creates efficient storage structures using vectors and key-value stores for fast data retrieval and analysis. This crate builds the foundation for BRK's data pipeline by extracting and organizing blockchain data into optimized storage formats. -A [Bitcoin Core](https://bitcoincore.org/en/about/) node indexer which iterates over the chain (via `../brk_parser`) and creates a database of the vecs (`brk_vec`) and key/value stores ([`fjall`](https://crates.io/crates/fjall)) that can be used in your Rust code. +## Features -The crate only stores the bare minimum to be self sufficient and not have to use an RPC client (except for scripts which are not stored). If you need more data, checkout `../computer` which uses the outputs from the indexer to compute a whole range of datasets. +- **Block-by-block processing**: Iterates through blockchain using brk_parser +- **Dual storage architecture**: Combines vectors (brk_vec) for time-series data and key-value stores (brk_store) for lookups +- **Memory efficient**: ~5GB peak RAM usage during indexing +- **Collision detection**: Validates data integrity with optional collision checking +- **Incremental updates**: Supports resuming from last indexed height +- **Rollback protection**: Automatic rollback on interruption or errors -Vecs are used sparingly instead of stores for multiple reasons: +## Storage Strategy -- Only stores the relevant data since the key is an index -- Saved as uncompressed bytes and thus can be parsed manually (with any programming language) without relying on a server or library -- Easy to work with and predictable +### Vectors (brk_vec) + +Used for sequential, time-indexed data: +- Block metadata (height, timestamp, hash) +- Transaction counts and statistics +- Price data and market metrics +- Efficient for range queries and analytics + +### Key-Value Stores (brk_store) + +Used for lookup operations: +- Address mappings and balances +- Transaction and UTXO data +- Script and output type indices +- Fast point queries by hash/address ## Usage -Storage wise, the expected overhead should be around 30% of the chain itself. +```rust +use brk_indexer::Indexer; +use brk_parser::Parser; +use bitcoincore_rpc::{Auth, Client}; +use vecdb::Exit; +use std::path::Path; -Peaks at 5 GB of RAM +fn main() -> brk_error::Result<()> { + // Setup paths and RPC + let bitcoin_dir = Path::new("~/.bitcoin"); + let outputs_dir = Path::new("./brk_data"); -## Outputs + let rpc = Box::leak(Box::new(Client::new( + "http://localhost:8332", + Auth::CookieFile(bitcoin_dir.join(".cookie")), + )?)); -Vecs: `src/storage/vecs/mod.rs` + // Create parser and indexer + let parser = Parser::new( + bitcoin_dir.join("blocks"), + outputs_dir.to_path_buf(), + rpc + ); -Stores: `src/storage/stores/mod.rs` + let mut indexer = Indexer::forced_import(outputs_dir)?; -## Benchmark + // Setup exit handler + let exit = Exit::new(); + exit.set_ctrlc_handler(); -### `v0.0.21` + // Index the blockchain + let indexes = indexer.index(&parser, rpc, &exit, true)?; -- machine: `MBP M3 Pro (36GB RAM)` -- mode: `raw` -- from: `0` -- to: `892_098` -- time: `7 hours 10 min 22s` -- peak memory: `6.1GB` -- disk usage: `270 GB` -- overhead: `36%` (`270 GB / 741 GB`) + println!("Indexed up to height: {}", indexes.height); -### `v0.0.31` + Ok(()) +} +``` -- machine: `MBP M3 Pro (36GB RAM)` -- mode: `raw` -- disk usage: `208 GB` -- overhead: `28%` (`208 GB / 744 GB`) -- peak memory: `5.7GB` +## Performance + +Benchmarked on MacBook Pro M3 Pro (36GB RAM): +- **Full sync to ~892k blocks**: 7-8 hours +- **Peak memory usage**: 5-6GB +- **Storage overhead**: ~27% of Bitcoin Core `/blocks` size (193GB as of 2025/08) +- **Incremental updates**: Resumes from last height efficiently + +## Data Organization + +The indexer creates the following storage structure: +``` +brk_data/ +├── indexed/ +│ ├── vecs/ # Vector storage for time-series data +│ └── stores/ # Key-value stores for lookups +└── ... +``` + +## Requirements + +- Running Bitcoin Core node with RPC access +- Access to Bitcoin Core's block files +- Minimum 500GB free storage space +- 8GB+ RAM recommended for optimal performance + +## Incremental Indexing + +The indexer supports continuous operation: +- Automatically detects last indexed height +- Processes new blocks as they arrive +- Handles blockchain reorganizations +- Provides graceful shutdown with Ctrl+C diff --git a/crates/brk_indexer/src/lib.rs b/crates/brk_indexer/src/lib.rs index c51774db4..0f2d109a5 100644 --- a/crates/brk_indexer/src/lib.rs +++ b/crates/brk_indexer/src/lib.rs @@ -1,7 +1,4 @@ #![doc = include_str!("../README.md")] -#![doc = "\n## Example\n\n```rust"] -#![doc = include_str!("../examples/indexer.rs")] -#![doc = "```"] use std::{collections::BTreeMap, path::Path, str::FromStr, thread, time::Instant}; diff --git a/crates/brk_interface/README.md b/crates/brk_interface/README.md index dc892ec24..4e8346cdd 100644 --- a/crates/brk_interface/README.md +++ b/crates/brk_interface/README.md @@ -1,34 +1,81 @@ -# BRK Interface +# brk_interface -

- - GitHub Repo stars - - - License - - - Version - - - Documentation - - Size - - Dependency status - - - Discord - - - Nostr - -

+Data query and formatting interface that provides a unified API for accessing Bitcoin datasets from both the indexer and computer components with flexible output formats and pagination support. This crate serves as the primary data access layer for BRK's web API and MCP endpoints. -A crate that searches for datasets from either `brk_indexer` or `brk_computer` according to given parameters. +## Features -It's possible to search for one or multiple dataset if they have the same index and specify range with both the `from` and `to` being optional and supporting negative values. +- **Unified data access**: Query indexed blockchain data and computed analytics +- **Multiple output formats**: JSON, CSV, TSV, and Markdown table formatting +- **Flexible pagination**: Range queries with positive/negative indexing support +- **Multi-dataset queries**: Retrieve multiple datasets with the same index +- **Dynamic search**: Find datasets by ID with automatic fallbacks +- **Schema generation**: JSON Schema support for API documentation -The output will depend on the format choson which can be Markdown, Json, CSV or TSV and might vary if there is a one or mutiple datasets, and if one dataset one or multiple values. +## Query Parameters -In the future, it will support more features similar to a real query engine like in a Postgres databases and presets to fetch data grouped by address, transaction or blockhash/height. +### Core Parameters + +- **index**: Time frame for data retrieval (height, date, week, month, etc.) +- **ids**: Dataset identifiers (comma or space separated) +- **from**: Starting index (negative values count from end) +- **to**: Ending index (optional, exclusive) +- **count**: Maximum number of results to return +- **format**: Output format (json, csv, tsv, md) + +## Usage + +```rust +use brk_interface::{Interface, Params, ParamsOpt, Index, Format}; +use brk_indexer::Indexer; +use brk_computer::Computer; +use std::path::Path; + +fn main() -> brk_error::Result<()> { + let outputs_dir = Path::new("./brk_data"); + + // Load indexer and computer + let indexer = Indexer::forced_import(outputs_dir)?; + let computer = Computer::forced_import(outputs_dir, &indexer, None)?; + + // Create interface + let interface = Interface::build(&indexer, &computer); + + // Query latest block data + let params = Params { + index: Index::Height, + ids: vec!["date", "timestamp"].into(), + rest: ParamsOpt::default() + .set_from(-1) // Latest block + .set_format(Format::JSON), + }; + + let result = interface.search_and_format(params)?; + println!("{}", result); + + // Query price data for last 10 blocks + let params = Params { + index: Index::Height, + ids: vec!["price_usd"].into(), + rest: ParamsOpt::default() + .set_from(-10) + .set_count(10) + .set_format(Format::CSV), + }; + + let result = interface.search_and_format(params)?; + println!("{}", result); + + Ok(()) +} +``` + +## API Integration + +The interface provides methods for different use cases: + +- `search()`: Find datasets matching parameters +- `format()`: Format search results into specified output format +- `search_and_format()`: Combined search and format operation +- `get_indexes()`: List available time indices +- `get_vecids()`: List available dataset identifiers +- `get_height()`: Get current blockchain height diff --git a/crates/brk_interface/src/lib.rs b/crates/brk_interface/src/lib.rs index bdd69e2bf..8b916166a 100644 --- a/crates/brk_interface/src/lib.rs +++ b/crates/brk_interface/src/lib.rs @@ -1,7 +1,4 @@ #![doc = include_str!("../README.md")] -#![doc = "\n## Example\n\n```rust"] -#![doc = include_str!("../examples/main.rs")] -#![doc = "```"] use std::collections::BTreeMap; diff --git a/crates/brk_logger/README.md b/crates/brk_logger/README.md index e141117d6..52178706e 100644 --- a/crates/brk_logger/README.md +++ b/crates/brk_logger/README.md @@ -1,30 +1,37 @@ -# BRK Logger +# brk_logger -

- - GitHub Repo stars - - - License - - - Version - - - Documentation - - Size - - Dependency status - - - Discord - - - Nostr - -

+Colorful logging utility built on `env_logger` that provides clean, timestamped console output with optional file logging. This crate wraps `env_logger` to display logs from the `log` crate in a readable format with color-coded log levels and configurable filtering to suppress noisy third-party library logs. -A simple crate built on top of [`env_logger`](https://crates.io/crates/env_logger) to display logs from the [`log`](https://crates.io/crates/log) crate in a colorful and clean format. +## Features -It can also save logs into a file if desired. +- **Colorized output**: Log levels are color-coded (error=red, warn=yellow, info=green, debug=blue, trace=cyan) +- **Timestamps**: Each log entry includes a formatted timestamp +- **File logging**: Optional file output alongside console logging +- **Noise filtering**: Pre-configured to suppress verbose logs from Bitcoin Core RPC and other dependencies +- **Environment control**: Respects `RUST_LOG` environment variable for custom filtering + +## Usage + +```rust +use log::info; + +fn main() -> std::io::Result<()> { + // Initialize with console output only + brk_logger::init(None)?; + + // Or initialize with file logging + brk_logger::init(Some(std::path::Path::new("app.log")))?; + + info!("Application started"); + Ok(()) +} +``` + +## Default Log Filtering + +By default, the following crates are filtered to `off` to reduce noise: +- `bitcoin`, `bitcoincore-rpc` - Bitcoin Core libraries +- `fjall`, `lsm_tree` - Storage engine logs +- `rolldown`, `brk_rolldown` - Bundler logs +- `rmcp`, `brk_rmcp` - MCP protocol logs +- `tracing` - Tracing framework logs diff --git a/crates/brk_logger/src/lib.rs b/crates/brk_logger/src/lib.rs index bcf899269..140a32f76 100644 --- a/crates/brk_logger/src/lib.rs +++ b/crates/brk_logger/src/lib.rs @@ -1,7 +1,4 @@ #![doc = include_str!("../README.md")] -#![doc = "\n## Example\n\n```rust"] -#![doc = include_str!("../examples/main.rs")] -#![doc = "```"] use std::{ fmt::Display, diff --git a/crates/brk_mcp/README.md b/crates/brk_mcp/README.md index 7877a138f..e8dd6e6b0 100644 --- a/crates/brk_mcp/README.md +++ b/crates/brk_mcp/README.md @@ -1,49 +1,34 @@ -# BRK MCP +# brk_mcp -A Model Context Protocol (MCP) which gives LLMs access to all available tools in BRK +Model Context Protocol (MCP) endpoint that provides LLMs with access to Bitcoin Research Kit data and functionality. -## URLs +This crate implements a stateless MCP endpoint that integrates with `brk_server` to expose BRK's Bitcoin blockchain data through a standardized protocol, enabling LLMs like Claude to directly query blockchain metrics, transaction data, market prices, and time-series data. -- https://eu1.bitcoinresearchkit.org/mcp -- https://eu2.bitcoinresearchkit.org/mcp +The stateless design makes it compatible with load balancers by default. + +## Tools Available + +- `get_index_count` - Count of available data indexes +- `get_vecid_count` - Count of available vector identifiers +- `get_vec_count` - Total count of all vectors +- `get_indexes` - List all available indexes +- `get_accepted_indexes` - Index types and their variants +- `get_vecids` - Paginated list of vector identifiers +- `get_index_to_vecids` - Vectors supporting specific indexes +- `get_vecid_to_indexes` - Indexes supported by specific vectors +- `get_vecs` - Query vector data with flexible parameters +- `get_version` - BRK version information ## Usage -To connect to the MCP use any of the previous URL, no token or auth is needed. +The MCP server is automatically exposed at `/mcp` when BRK's HTTP server is running with MCP enabled. -This implementation has only been tested with Claude and the [MCP inspector](https://modelcontextprotocol.io/docs/tools/inspector). +### With Claude Desktop -Please be aware that the technology is evolving very rapidly, thus having issues is probably expected. If you, you can join the discord see if there is a solution. +Add the MCP endpoint to Claude Desktop. -### Claude +For example: -#### Step 1 - -First we need to connect BRK to Claude. To do that we need to go to the "Connect apps" menu from the home screen of Claude desktop. - -![Image of Claude Desktop home screen](https://github.com/bitcoinresearchkit/brk/blob/main/assets/claude-step1.png) - -#### Step 2 - -Then simply go to "Add integration". - -![Image of the Connect app" menu of Claude Desktop](https://github.com/bitcoinresearchkit/brk/blob/main/assets/claude-step2.png) - -#### Step 3 - -Claude's MCP client is (for now?) session based thus using a URL pointing to a load balancer will not work. - -Use one of the following URL instead: - -- https://eu1.bitcoinresearchkit.org/mcp -- https://eu2.bitcoinresearchkit.org/mcp - -![Image of Add Integration menu of Claude Desktop](https://github.com/bitcoinresearchkit/brk/blob/main/assets/claude-step3.png) - -#### Step 4 - -Verify that it has access to BRK's tools. - -Optionally and highly recommended, giving it unsupervised access gives a more fluid experience and prevents possible issues and errors. - -![Image of edit integration meny on Claude Desktop](https://github.com/bitcoinresearchkit/brk/blob/main/assets/claude-step4.png) +``` +https://bitcoinresearchkit.org/mcp +``` diff --git a/crates/brk_mcp/src/lib.rs b/crates/brk_mcp/src/lib.rs index c8fc47bd2..2c9177d3f 100644 --- a/crates/brk_mcp/src/lib.rs +++ b/crates/brk_mcp/src/lib.rs @@ -1,7 +1,4 @@ #![doc = include_str!("../README.md")] -// #![doc = "\n## Example\n\n```rust"] -// #![doc = include_str!("../examples/main.rs")] -// #![doc = "```"] use brk_interface::{IdParam, Interface, PaginatedIndexParam, PaginationParam, Params}; use brk_rmcp::{ diff --git a/crates/brk_parser/README.md b/crates/brk_parser/README.md index 87c499780..4949d3931 100644 --- a/crates/brk_parser/README.md +++ b/crates/brk_parser/README.md @@ -1,60 +1,87 @@ -# BRK Parser +# brk_parser -

- - GitHub Repo stars - - - License - - - Version - - - Documentation - - Size - - Dependency status - - - Discord - - - Nostr - -

+High-performance Bitcoin block parser that reads raw Bitcoin Core block files (`blkXXXXX.dat`) and provides a sequential iterator over blocks with fork filtering and XOR support. This crate processes the entire Bitcoin blockchain efficiently using parallel processing and maintains state for fast restarts. -A very fast and simple Rust library which reads raw block files (*blkXXXXX.dat*) from Bitcoin Core node and creates an iterator over all the requested blocks in sequential order (0, 1, 2, ...). +## Features -The element returned by the iterator is a tuple which includes the: -- Height: `Height` -- Block: `Block` (from `bitcoin-rust`) -- Block's Hash: `BlockHash` (also from `bitcoin-rust`) - -Tested with Bitcoin Core `v25.0..=v28.1` +- **Fast sequential parsing**: Iterates blocks in height order (0, 1, 2, ...) +- **Fork filtering**: Uses Bitcoin Core RPC to exclude orphaned blocks +- **XOR support**: Handles XOR-encrypted block files automatically +- **Parallel processing**: Multi-threaded parsing and decoding for maximum speed +- **State caching**: Saves parsing state for faster subsequent runs +- **Memory efficient**: ~500MB peak memory usage +- **Range queries**: Parse specific height ranges or single blocks ## Requirements -Even though it reads *blkXXXXX.dat* files, it **needs** `bitcoind` (with no particular parameters) to run with the RPC server to filter out forks. +- Running Bitcoin Core node with RPC enabled +- Access to Bitcoin Core's `blocks/` directory containing `blkXXXXX.dat` files +- Bitcoin Core versions v25.0 through v29.0 supported -Peak memory should be around 500MB. +## Usage -XOR-ed blocks are supported. +```rust +use brk_parser::Parser; +use brk_structs::Height; +use bitcoincore_rpc::{Auth, Client}; +use std::path::Path; -## Disclaimer +fn main() -> bitcoincore_rpc::Result<()> { + // Setup Bitcoin Core RPC client + let rpc = Box::leak(Box::new(Client::new( + "http://localhost:8332", + Auth::CookieFile(Path::new("~/.bitcoin/.cookie")), + )?)); -A state of the local chain is saved in `{bitcoindir}/blocks/blk_index_to_blk_recap.json` to allow for faster starts (see benchmark below) but doesn't yet support locking. Thus, it is highly recommended to run one instance of `brk_parser` at a time. + // Create parser + let parser = Parser::new( + Path::new("~/.bitcoin/blocks").to_path_buf(), + Path::new("./brk_data").to_path_buf(), // Output directory + rpc, + ); -## Benchmark + // Parse all blocks + parser.parse(None, None) + .iter() + .for_each(|(height, block, hash)| { + println!("Block {}: {} ({} transactions)", + height, hash, block.txdata.len()); + }); -| | [brk_parser](https://crates.io/crates/brk_parser) | [bitcoin-explorer (deprecated)](https://crates.io/crates/bitcoin-explorer) | [blocks_iterator](https://crates.io/crates/blocks_iterator)* | -| --- | --- | --- | --- | -| Runs **with** `bitcoind` | Yes ✅ | No ❌ | Yes ✅ | -| Runs **without** `bitcoind` | No ❌ | Yes ✅ | Yes ✅ | -| `0..=855_000` | 4mn 10s | 4mn 45s | > 2h | -| `800_000..=855_000` | 0mn 52s (4mn 10s if first run) | 0mn 55s | > 2h | + // Parse specific range + let start = Some(Height::new(800_000)); + let end = Some(Height::new(800_100)); -\* `blocks_iterator` is with the default config (and thus with `skip_prevout = false` which does a lot more than just iterate over blocks) so it isn't an apples to apples comparaison and the numbers are misleading. You should expect much closer times. Will update the benchmark with `skip_prevout = true` as soon as possible. + parser.parse(start, end) + .iter() + .for_each(|(height, block, hash)| { + println!("Block {}: {}", height, hash); + }); -*Benchmarked on a Macbook Pro M3 Pro* + // Get single block + let block = parser.get(Height::new(0)); // Genesis block + println!("Genesis block has {} transactions", block.txdata.len()); + + Ok(()) +} +``` + +## Output Format + +The parser returns tuples containing: +- `Height`: Block height (0, 1, 2, ...) +- `Block`: Complete block data (from `bitcoin` crate) +- `BlockHash`: Block's cryptographic hash + +## Performance + +Benchmarked on MacBook Pro M3 Pro: +- Full blockchain (0 to 855,000): **4 minutes 10 seconds** +- Recent blocks (800,000 to 855,000): **52 seconds** (4m 10s on first run) +- Peak memory usage: ~500MB + +## State Management + +The parser saves state in `{output_dir}/blk_index_to_blk_recap.json` for faster restarts. This file tracks block file indices and heights to avoid re-scanning unchanged files. + +**Note**: Only one parser instance should run at a time as the state file doesn't yet support concurrent access. diff --git a/crates/brk_parser/src/lib.rs b/crates/brk_parser/src/lib.rs index b7e8a817e..7eb93128d 100644 --- a/crates/brk_parser/src/lib.rs +++ b/crates/brk_parser/src/lib.rs @@ -1,7 +1,4 @@ #![doc = include_str!("../README.md")] -#![doc = "\n## Example\n\n```rust"] -#![doc = include_str!("../examples/main.rs")] -#![doc = "```"] use std::{cmp::Ordering, collections::BTreeMap, fs, ops::ControlFlow, path::PathBuf, thread}; diff --git a/crates/brk_server/README.md b/crates/brk_server/README.md index 5a703d781..b233ed6a4 100644 --- a/crates/brk_server/README.md +++ b/crates/brk_server/README.md @@ -1,145 +1,109 @@ -# BRK Server +# brk_server -

- - GitHub Repo stars - - - License - - - Version - - - Documentation - - Size - - Dependency status - - - Discord - - - Nostr - -

+HTTP server providing REST API access to Bitcoin analytics data. Serves computed datasets from brk_indexer and brk_computer with multiple output formats, caching, and optional web interfaces. -A crate that serves Bitcoin data and swappable front-ends, built on top of `brk_indexer`, `brk_computer` and `brk_interface`. +## Overview -The file handler, will serve the website specified by the user if any, which can be *no website*, *default* or *custom* (which is a blank folder for people to experiment). If a website is specified and the server is ran outside of the brk project and thus can't find the requested website, it will download the whole project with the correct version from Github and store it in `.brk` to be able to serve to website. This is due to the crate size limit on [crates.io](https://crates.io) and the various shenanigans that need to be done to have a website in a crate. +**Core Features**: +- **REST API**: Vector-based data access with pagination and format options +- **Multiple Formats**: JSON, CSV, TSV, Markdown output +- **HTTP Caching**: ETag-based caching with compression (Brotli, Gzip, Zstd, Deflate) +- **Weight Limits**: Request size protection (max 320,000 weight units) +- **Web Interface**: Optional website serving -The API uses `brk_interface` and so inherites all of its features including formats. +**Port**: Auto-assigns starting from 3110 -## Endpoints +## API Reference -### API +### Vector Metadata -#### [`GET /api/vecs/index-count`](https://bitcoinresearchkit.org/api/vecs/index-count) +| Endpoint | Description | +|----------|-------------| +| `GET /api/vecs/index-count` | Total number of available indexes | +| `GET /api/vecs/id-count` | Total number of vector IDs | +| `GET /api/vecs/vec-count` | Total number of vectors (sum of all index/ID combinations) | +| `GET /api/vecs/indexes` | List of all available indexes | +| `GET /api/vecs/accepted-indexes` | Object mapping indexes to their accepted variants | +| `GET /api/vecs/ids?page=N` | Paginated list of vector IDs (1000 per page, default page=0) | +| `GET /api/vecs/index-to-ids?index=INDEX&page=N` | Vector IDs supporting given index | +| `GET /api/vecs/id-to-indexes?id=ID` | Indexes supported by given vector ID | -Get the count of all existing indexes. +### Vector Data Access -#### [`GET /api/vecs/id-count`](https://bitcoinresearchkit.org/api/vecs/id-count) +#### Direct Access Pattern: `GET /api/vecs/{INDEX}-to-{ID}` -Get the count of all existing vec ids. +Access single vector with index-to-id pattern (dashes replaced with underscores internally). -#### [`GET /api/vecs/vec-count`](https://bitcoinresearchkit.org/api/vecs/vec-count) - -Get the count of all existing vecs. \ -Equals to the sum of supported Indexes of each vec id. - -#### [`GET /api/vecs/indexes`](https://bitcoinresearchkit.org/api/vecs/indexes) - -Get the list of all existing indexes. - -#### [`GET /api/vecs/accepted-indexes`](https://bitcoinresearchkit.org/api/vecs/accepted-indexes) - -Get an object which has all existing indexes as keys and a list of their accepted variants as values. - -#### [`GET /api/vecs/ids`](https://bitcoinresearchkit.org/api/vecs/ids) - -Get a paginated list of all existing vec ids. \ -There are up to 1,000 values per page. \ -If the `page` param is omitted, it will default to page `0`. - -#### [`GET /api/vecs/index-to-ids`](https://bitcoinresearchkit.org/api/vecs/index-to-ids) - -Get a paginated list of all vec ids which support a given index. -There are up to 1,000 values per page. -If the `page` param is omitted, it will default to the first page. - -#### [`GET /api/vecs/id-to-indexes`](https://bitcoinresearchkit.org/api/vecs/id-to-indexes) - -Get a list of all indexes supported by a given vec id. -The list will be empty if the vec id isn't correct. - -#### `GET /api/vecs/{INDEX}-to-{ID}` - -This endpoint retrieves data based on the specified vector index and id. - -**Parameters:** - -| Parameter | Type | Required | Description | -| --- | --- | --- | --- | -| `from` | `signed int` | No | Inclusive starting index for pagination (default is 0). | -| `to` | `signed int` | No | Exclusive ending index for pagination (default is the total number of results). Overrides `count` | -| `count` | `unsigned int` | No | The number of values requested | -| `format` | `string` | No | The format of the response. Options include `json`, `csv`, `tsv`, or `md` (default is `json`). | +**Query Parameters:** +- `from` (i64, optional): Inclusive start index. Negative values count from end (default: 0) +- `to` (i64, optional): Exclusive end index. Negative values count from end. Overrides `count` +- `count` (usize, optional): Number of values to retrieve +- `format` (string, optional): Output format - `json`, `csv`, `tsv`, `md` (default: `json`) **Examples:** +```bash +# Latest 100 price closes +curl /api/vecs/date-to-close?from=-100 -```sh -# GET /api/vecs/date-to-close -curl https://bitcoinresearchkit.org/api/vecs/date-to-close +# First 50 values as CSV +curl /api/vecs/height-to-difficulty?count=50&format=csv -# GET /api/vecs/date-to-close?from=-100 -curl https://bitcoinresearchkit.org/api/vecs/date-to-close?from=-100 - -# GET /api/vecs/date-to-close?count=100&format=csv -curl https://bitcoinresearchkit.org/api/vecs/date-to-close?count=100&format=csv +# Range from index 1000 to 1999 +curl /api/vecs/date-to-volume?from=1000&to=2000 ``` -#### `GET /api/vecs/query` +#### Multi-Vector Query: `GET /api/vecs/query` -Get one or multiple vecs depending on given parameters. -If you'd like to request multiple vec ids, simply separate them with a ','. \ -To get the last value set `-1` to the `from` parameter. \ -The response's format will depend on the given parameters, it will be: -- A value: If requested only one vec and the given range returns one value (for example: `from=-1`) -- A list: If requested only one vec and the given range returns multiple values (for example: `from=-1000&count=100` or `from=-444&to=-333`) -- A matrix: When multiple vecs are requested, even if they each return one value. +Query multiple vectors simultaneously with flexible output formats. -**Parameters:** +**Required Parameters:** +- `index`: Vector index type +- `ids`: Comma or space-separated vector IDs -| Parameter | Type | Required | Description | -| --- | --- | --- | --- | -| `index` | `VecIndex` | Yes | The vector index to query. | -| `ids` | `VecId[]` | Yes | A comma or space-separated list of vector IDs to retrieve. | -| `from` | `signed int` | No | Inclusive starting index for pagination (default is 0). | -| `to` | `signed int` | No | Exclusive ending index for pagination (default is the total number of results). Overrides `count` | -| `count` | `unsigned int` | No | The number of values requested | -| `format` | `string` | No | The format of the response. Options include `json`, `csv`, `tsv`, or `md` (default is `json`). | +**Optional Parameters:** +- `from`, `to`, `count`, `format`: Same as direct access + +**Response Types:** +- **Single value**: One vector, one result (e.g., `from=-1`) +- **Array**: One vector, multiple results +- **Matrix**: Multiple vectors (always matrix, even for single results) **Examples:** +```bash +# Single vector, latest value +curl '/api/vecs/query?index=date&ids=close&from=-1' -```sh -# GET /api/vecs/query?index=date&ids=ohlc -curl https://bitcoinresearchkit.org/api/vecs/query?index=date&ids=ohlc +# Multiple vectors, date range +curl '/api/vecs/query?index=date&ids=open,high,low,close&from=-30&format=csv' -# GET /api/vecs/query?index=week&ids=ohlc,block-interval-average&from=0&to=20&format=md -curl https://bitcoinresearchkit.org/api/vecs/query?index=week&ids=ohlc,block-interval-average&from=0&to=20&format=md +# Complex multi-vector query +curl '/api/vecs/query?index=week&ids=price-usd,volume,difficulty&count=52' ``` -### Meta +### System Endpoints -#### [`GET /version`](https://bitcoinresearchkit.org/version) +| Endpoint | Description | +|----------|-------------| +| `GET /version` | Server version (JSON string) | +| `GET /health` | Health check with timestamp | +| `GET /api` | Redirects to this documentation | +| `GET /*` | Static file serving (when website enabled) | -The version of the server and thus BRK. +### HTTP Features -### Files +**Caching:** +- ETag-based conditional requests (304 Not Modified) +- `Cache-Control: must-revalidate` headers +- In-memory cache with 50ms guard timeout -#### `GET /*` +**Compression:** +- Brotli, Gzip, Zstd, Deflate support +- Automatic content encoding negotiation -Catch all. +**CORS:** +- Cross-origin requests enabled +- Appropriate headers for web client access -When no pattern is found, the server will look for a match inside the folder of the chosen website, if any. +**Rate Limiting:** +- Request weight system (max 320,000 units) +- Weight calculated from data range size diff --git a/crates/brk_store/README.md b/crates/brk_store/README.md index 3e64ce0d1..e38509e73 100644 --- a/crates/brk_store/README.md +++ b/crates/brk_store/README.md @@ -1 +1,79 @@ -# BRK Store +# brk_store + +Thin wrapper around the Fjall embedded key-value store that provides typed, transactional storage for Bitcoin blockchain data with height-based versioning and batch operations. This crate adds BRK-specific functionality like height tracking, metadata management, and optimized configurations for Bitcoin data storage patterns. + +## Features + +- **Typed interface**: Generic over key and value types with automatic serialization +- **Height tracking**: Built-in blockchain height awareness for data versioning +- **Batch operations**: Efficient batch inserts and deletes with transaction support +- **Metadata management**: Automatic version and height metadata storage +- **Performance optimized**: Configured write buffers and memtable sizes for Bitcoin data +- **Bloom filters**: Configurable bloom filters for faster key lookups +- **Reset capability**: Clean store reset for reindexing operations + +## Usage + +```rust +use brk_store::{Store, open_keyspace}; +use brk_structs::{Height, Version}; +use std::path::Path; + +fn main() -> brk_error::Result<()> { + // Open the keyspace + let keyspace = open_keyspace(Path::new("./data"))?; + + // Create a typed store + let mut store: Store = Store::import( + &keyspace, + Path::new("./data"), + "my_store", + Version::ZERO, + Some(true), // Enable bloom filters + )?; + + // Insert data if needed at this height + store.insert_if_needed( + "key1".to_string(), + 42u64, + Height::new(800_000) + ); + + // Commit changes + store.commit(Height::new(800_000))?; + + // Persist to disk + store.persist()?; + + // Query data + if let Some(value) = store.get(&"key1".to_string())? { + println!("Value: {}", value); + } + + Ok(()) +} +``` + +## Store Lifecycle + +- **Import**: Create or open existing store with version checking +- **Insert**: Add key-value pairs with height-based conditional insertion +- **Commit**: Write batched changes to disk atomically +- **Persist**: Force sync all data to storage +- **Reset**: Clear all data for reindexing if needed + +## AnyStore Trait + +The `AnyStore` trait provides a type-erased interface for managing multiple stores: + +```rust +use brk_store::AnyStore; + +fn process_store(store: &mut dyn AnyStore) -> brk_error::Result<()> { + if store.needs(Height::new(800_000)) { + // Process this height + store.commit(Height::new(800_000))?; + } + Ok(()) +} +``` diff --git a/crates/brk_store/src/lib.rs b/crates/brk_store/src/lib.rs index cdf46c51e..34e374cd7 100644 --- a/crates/brk_store/src/lib.rs +++ b/crates/brk_store/src/lib.rs @@ -1,7 +1,4 @@ #![doc = include_str!("../README.md")] -#![doc = "\n## Example\n\n```rust"] -#![doc = include_str!("../examples/main.rs")] -#![doc = "```"] use std::{ borrow::Cow, diff --git a/crates/brk_structs/README.md b/crates/brk_structs/README.md index 4fe182961..feab38bd0 100644 --- a/crates/brk_structs/README.md +++ b/crates/brk_structs/README.md @@ -1,28 +1,93 @@ -# BRK Core +# brk_structs -

- - GitHub Repo stars - - - License - - - Version - - - Documentation - - Size - - Dependency status - - - Discord - - - Nostr - -

+Core data structures and types used throughout the Bitcoin Research Kit that provide efficient, zero-copy serializable representations of Bitcoin blockchain data. This crate defines strongly-typed wrappers around primitive types with specialized functionality for Bitcoin analysis, storage optimization, and data grouping operations. -A list of structs that are used throughout the project as units, think of `Date`, `Height`, `Sats`, `Txindex` or anything that can be either a key and/or a value of a dataset. +## Core Types + +### Blockchain Data +- **Height**: Block height with arithmetic operations +- **Date**, **Timestamp**: Time representations with Bitcoin epoch awareness +- **Txid**, **BlockHash**: Transaction and block identifiers with prefix variants +- **TxIndex**, **InputIndex**, **OutputIndex**: Transaction component indices + +### Value Types +- **Sats**: Satoshi amounts with conversion utilities +- **Bitcoin**: BTC amounts with precision handling +- **Dollars**, **Cents**: Fiat currency representations +- **OHLC**: Open/High/Low/Close price data structures + +### Address Types +- **AddressBytes**: Raw address data with type information +- **P2PKH**, **P2SH**, **P2WPKH**, **P2WSH**, **P2TR**: Address type indices +- **AnyAddressIndex**: Unified address index type + +### Storage Types +- **StoredU8/U16/U32/U64**: Optimized integer storage +- **StoredF32/F64**: Floating-point storage with compression +- **StoredBool**: Compact boolean storage + +### Time Indices +- **DateIndex**, **WeekIndex**, **MonthIndex**, **QuarterIndex** +- **SemesterIndex**, **YearIndex**, **DecadeIndex**: Time-based grouping +- **HalvingEpoch**, **DifficultyEpoch**: Bitcoin-specific time periods + +## Grouping Operations + +The crate provides powerful grouping and filtering capabilities: + +```rust +use brk_structs::*; + +// Group by address type +let p2pkh_addresses = ByAddressType::P2PKH; + +// Group by value ranges +let small_utxos = ByLtAmount::new(Sats::_1BTC); +let large_utxos = ByGeAmount::new(Sats::_10BTC); + +// Group by age ranges +let recent_utxos = ByMaxAge::new(Height::new(144)); // Last day +let old_utxos = ByMinAge::new(Height::new(52560)); // Last year + +// Combine filters +let filter = Filter::new() + .by_spendable_type() + .by_amount_range(Sats::_1K, Sats::_1M) + .by_address_type(ByAddressType::P2WPKH); +``` + +## Features + +- **Zero-copy serialization**: All types implement `FromBytes`/`IntoBytes` for efficient storage +- **Storage compression**: Built-in compression support via `StoredCompressed` trait +- **Type safety**: Strongly-typed wrappers prevent value confusion +- **Bitcoin-aware**: Constants and operations specific to Bitcoin protocol +- **Efficient grouping**: Flexible data filtering and categorization system +- **Time handling**: Comprehensive time representation with epoch support + +## Usage + +```rust +use brk_structs::*; + +// Create blockchain data types +let height = Height::new(800_000); +let amount = Sats::_1BTC; +let date = Date::new(2024, 1, 15); + +// Work with addresses +let address_data = AddressBytes::from_script(&script); +let address_index = P2WPKHAddressIndex::from_address_bytes(&address_data); + +// Price data +let ohlc = OHLCCents::new( + Open::new(45000_00), + High::new(46000_00), + Low::new(44000_00), + Close::new(45500_00) +); + +// Time-based analysis +let month_index = MonthIndex::from(date); +let halving_epoch = HalvingEpoch::from(height); +```