mirror of
https://github.com/bitcoinresearchkit/brk.git
synced 2026-04-24 06:39:58 -07:00
readmes: update
This commit is contained in:
nym21
6
Cargo.lock
generated
6
Cargo.lock
generated
@@ -3863,7 +3863,7 @@ checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc"
|
||||
|
||||
[[package]]
|
||||
name = "seqdb"
|
||||
version = "0.1.2"
|
||||
version = "0.2.0"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"log",
|
||||
@@ -4688,7 +4688,7 @@ checksum = "8f54a172d0620933a27a4360d3db3e2ae0dd6cceae9730751a036bbf182c4b23"
|
||||
|
||||
[[package]]
|
||||
name = "vecdb"
|
||||
version = "0.1.2"
|
||||
version = "0.2.0"
|
||||
dependencies = [
|
||||
"ctrlc",
|
||||
"log",
|
||||
@@ -4706,7 +4706,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "vecdb_derive"
|
||||
version = "0.1.2"
|
||||
version = "0.2.0"
|
||||
dependencies = [
|
||||
"quote",
|
||||
"syn 2.0.106",
|
||||
|
||||
@@ -1,66 +1,197 @@
|
||||
# brk
|
||||
|
||||
Bitcoin Research Kit (BRK) is a high-performance toolchain for parsing, indexing, computing, and serving Bitcoin blockchain data. It provides an alternative to services like Glassnode and mempool.space with a focus on self-hosting and open-source transparency.
|
||||
**Main wrapper crate for the Bitcoin Research Kit (BRK)**
|
||||
|
||||
This is the main wrapper crate that re-exports all workspace crates through feature flags.
|
||||
The `brk` crate serves as the primary entry point for the Bitcoin Research Kit, providing a unified interface to all BRK components through feature flags. It enables developers to selectively include only the components they need while maintaining a consistent API.
|
||||
|
||||
## Crates
|
||||
## What it provides
|
||||
|
||||
- [`brk`](https://crates.io/crates/brk): A wrapper around all other `brk-*` crates
|
||||
- [`brk_bundler`](https://crates.io/crates/brk_bundler): A thin wrapper around [`rolldown`](https://rolldown.rs/)
|
||||
- [`brk_cli`](https://crates.io/crates/brk_cli): A command line interface to run a BRK instance
|
||||
- [`brk_computer`](https://crates.io/crates/brk_computer): A Bitcoin dataset computer built on top of [`brk_indexer`](https://crates.io/crates/brk_indexer)
|
||||
- [`brk_error`](https://crates.io/crates/brk_error): Errors used throughout BRK
|
||||
- [`brk_fetcher`](https://crates.io/crates/brk_fetcher): A Bitcoin price fetcher
|
||||
- [`brk_indexer`](https://crates.io/crates/brk_indexer): A Bitcoin indexer built on top of [`brk_parser`](https://crates.io/crates/brk_parser)
|
||||
- [`brk_interface`](https://crates.io/crates/brk_interface): An interface to find and format data from BRK
|
||||
- [`brk_logger`](https://crates.io/crates/brk_logger): A thin wrapper around [`env_logger`](https://crates.io/crates/env_logger)
|
||||
- [`brk_mcp`](https://crates.io/crates/brk_mcp): A bridge for LLMs to access BRK
|
||||
- [`brk_parser`](https://crates.io/crates/brk_parser): A very fast Bitcoin block parser and iterator built on top of [`bitcoin-rust`](https://crates.io/crates/bitcoin)
|
||||
- [`brk_server`](https://crates.io/crates/brk_server): A server with an API for anything from BRK
|
||||
- [`brk_store`](https://crates.io/crates/brk_store): A thin wrapper around [`fjall`](https://crates.io/crates/fjall)
|
||||
- [`brk_structs`](https://crates.io/crates/brk_structs): Structs used throughout BRK
|
||||
- **Unified Access**: Single crate providing access to the entire BRK ecosystem
|
||||
- **Feature-based Selection**: Choose only the components you need
|
||||
- **Consistent Versioning**: All components versioned together for compatibility
|
||||
- **Simplified Dependencies**: Single dependency instead of multiple individual crates
|
||||
|
||||
## Features
|
||||
## Available Components
|
||||
|
||||
- `full` - Enable all workspace crates
|
||||
- `bundler` - Re-export `brk_bundler`
|
||||
- `cli` - Re-export `brk_cli` (always enabled)
|
||||
- `computer` - Re-export `brk_computer`
|
||||
- `error` - Re-export `brk_error`
|
||||
- `fetcher` - Re-export `brk_fetcher`
|
||||
- `indexer` - Re-export `brk_indexer`
|
||||
- `interface` - Re-export `brk_interface`
|
||||
- `logger` - Re-export `brk_logger`
|
||||
- `mcp` - Re-export `brk_mcp`
|
||||
- `parser` - Re-export `brk_parser`
|
||||
- `server` - Re-export `brk_server`
|
||||
- `store` - Re-export `brk_store`
|
||||
- `structs` - Re-export `brk_structs`
|
||||
### Core Data Pipeline
|
||||
- **`parser`** ([brk_parser](../brk_parser/)) - High-performance Bitcoin block parser
|
||||
- **`indexer`** ([brk_indexer](../brk_indexer/)) - Blockchain data indexer with dual storage
|
||||
- **`computer`** ([brk_computer](../brk_computer/)) - Analytics engine for computed datasets
|
||||
- **`interface`** ([brk_interface](../brk_interface/)) - Unified data query and formatting API
|
||||
|
||||
### Infrastructure
|
||||
- **`structs`** ([brk_structs](../brk_structs/)) - Bitcoin-aware type system and data structures
|
||||
- **`error`** ([brk_error](../brk_error/)) - Centralized error handling
|
||||
- **`store`** ([brk_store](../brk_store/)) - Blockchain-aware key-value storage
|
||||
|
||||
### External Integration
|
||||
- **`fetcher`** ([brk_fetcher](../brk_fetcher/)) - Bitcoin price data fetcher with multi-source fallback
|
||||
- **`server`** ([brk_server](../brk_server/)) - HTTP server with REST API
|
||||
- **`mcp`** ([brk_mcp](../brk_mcp/)) - Model Context Protocol for LLM integration
|
||||
|
||||
### Utilities
|
||||
- **`cli`** ([brk_cli](../brk_cli/)) - Command line interface (always enabled)
|
||||
- **`logger`** ([brk_logger](../brk_logger/)) - Logging utilities
|
||||
- **`bundler`** ([brk_bundler](../brk_bundler/)) - Asset bundling for web interfaces
|
||||
|
||||
## Usage
|
||||
|
||||
Add to your `Cargo.toml`:
|
||||
### Full Installation
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
brk = { version = "0.1", features = ["full"] }
|
||||
brk = { version = "0.0.88", features = ["full"] }
|
||||
```
|
||||
|
||||
Or enable specific components:
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
brk = { version = "0.1", features = ["parser", "indexer", "computer"] }
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
```rust
|
||||
use brk::{cli, parser, indexer, computer};
|
||||
use brk::*;
|
||||
|
||||
// Use individual crates as needed
|
||||
// Access all components
|
||||
let config = cli::Config::load()?;
|
||||
let blocks = parser::BlockIterator::new(&config.bitcoin_path)?;
|
||||
// ...
|
||||
let parser = parser::Parser::new(/* ... */);
|
||||
let indexer = indexer::Indexer::forced_import("./data")?;
|
||||
let computer = computer::Computer::forced_import("./data", &indexer, None)?;
|
||||
```
|
||||
|
||||
### Selective Components
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
brk = { version = "0.0.88", features = ["parser", "indexer", "computer"] }
|
||||
```
|
||||
|
||||
```rust
|
||||
use brk::{parser, indexer, computer};
|
||||
|
||||
// Core data pipeline only
|
||||
let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
|
||||
let mut indexer = indexer::Indexer::forced_import(output_dir)?;
|
||||
let mut computer = computer::Computer::forced_import(output_dir, &indexer, None)?;
|
||||
```
|
||||
|
||||
### Minimal Setup
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
brk = { version = "0.0.88", features = ["structs", "parser"] }
|
||||
```
|
||||
|
||||
```rust
|
||||
use brk::{structs, parser};
|
||||
|
||||
// Just parsing and types
|
||||
let height = structs::Height::new(800_000);
|
||||
let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
|
||||
```
|
||||
|
||||
## Feature Flags
|
||||
|
||||
| Feature | Description | Dependencies |
|
||||
|---------|-------------|--------------|
|
||||
| `full` | Enable all components | All crates |
|
||||
| `cli` | Command line interface | Always enabled |
|
||||
| `structs` | Core type system | Foundation for other crates |
|
||||
| `error` | Error handling | Used by most crates |
|
||||
| `parser` | Block parsing | `structs`, `error` |
|
||||
| `store` | Key-value storage | `structs`, `error` |
|
||||
| `indexer` | Blockchain indexing | `parser`, `store` |
|
||||
| `computer` | Analytics computation | `indexer`, `fetcher` (optional) |
|
||||
| `fetcher` | Price data fetching | `structs`, `error` |
|
||||
| `interface` | Data query API | `indexer`, `computer` |
|
||||
| `server` | HTTP server | `interface`, `mcp` |
|
||||
| `mcp` | LLM integration | `interface` |
|
||||
| `logger` | Logging utilities | Standalone |
|
||||
| `bundler` | Asset bundling | Standalone |
|
||||
|
||||
## Common Usage Patterns
|
||||
|
||||
### Complete BRK Instance
|
||||
|
||||
```rust
|
||||
use brk::*;
|
||||
|
||||
// Full data pipeline setup
|
||||
let config = cli::Config::load()?;
|
||||
let rpc = /* Bitcoin Core RPC client */;
|
||||
let parser = parser::Parser::new(config.blocks_dir, config.output_dir, rpc);
|
||||
let mut indexer = indexer::Indexer::forced_import(&config.output_dir)?;
|
||||
let mut computer = computer::Computer::forced_import(&config.output_dir, &indexer, None)?;
|
||||
let interface = interface::Interface::build(&indexer, &computer);
|
||||
let server = server::Server::new(interface, config.website_path);
|
||||
|
||||
// Start server
|
||||
server.serve(true).await?;
|
||||
```
|
||||
|
||||
### Data Analysis
|
||||
|
||||
```rust
|
||||
use brk::{indexer, computer, interface};
|
||||
|
||||
// Analysis-focused setup
|
||||
let indexer = indexer::Indexer::forced_import("./brk_data")?;
|
||||
let computer = computer::Computer::forced_import("./brk_data", &indexer, None)?;
|
||||
let interface = interface::Interface::build(&indexer, &computer);
|
||||
|
||||
// Query data
|
||||
let params = interface::Params {
|
||||
index: interface::Index::Height,
|
||||
ids: vec!["price_usd", "difficulty"].into(),
|
||||
rest: interface::ParamsOpt::default()
|
||||
.set_from(-100)
|
||||
.set_format(interface::Format::CSV),
|
||||
};
|
||||
|
||||
let csv_data = interface.search_and_format(params)?;
|
||||
```
|
||||
|
||||
### Custom Integration
|
||||
|
||||
```rust
|
||||
use brk::{structs, parser, error};
|
||||
|
||||
// Custom application with BRK components
|
||||
fn analyze_blocks() -> error::Result<()> {
|
||||
let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
|
||||
|
||||
parser.parse(None, None)
|
||||
.iter()
|
||||
.take(1000) // First 1000 blocks
|
||||
.for_each(|(height, block, hash)| {
|
||||
println!("Block {}: {} transactions", height, block.txdata.len());
|
||||
});
|
||||
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
## Version Compatibility
|
||||
|
||||
All BRK crates are released together with synchronized versions. When using the `brk` wrapper crate, you're guaranteed compatibility between all components.
|
||||
|
||||
- **Current version**: 0.0.88
|
||||
- **Rust MSRV**: 1.89+
|
||||
- **Bitcoin Core**: v25.0 - v29.0
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
The `brk` crate itself adds no runtime overhead - it simply re-exports the underlying crates. Performance characteristics depend on which components you use:
|
||||
|
||||
- **Full pipeline**: ~13-15 hours initial sync, ~40GB storage overhead
|
||||
- **Parser only**: ~4 minutes to parse entire blockchain
|
||||
- **Indexer only**: ~7-8 hours to index blockchain, ~5-6GB RAM usage
|
||||
- **Server**: Low latency API responses with caching and compression
|
||||
|
||||
## Dependencies
|
||||
|
||||
The `brk` crate's dependencies are determined by enabled features. Core dependencies include:
|
||||
|
||||
- `brk_cli` - Always included for configuration and CLI support
|
||||
- Individual `brk_*` crates based on enabled features
|
||||
- Transitive dependencies from enabled components
|
||||
|
||||
For specific dependency information, see individual crate READMEs.
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
@@ -1,63 +1,186 @@
|
||||
# brk_bundler
|
||||
|
||||
Web asset bundler built on Rolldown that provides JavaScript bundling, minification, and live reloading for BRK's web interface. This crate wraps Rolldown with BRK-specific functionality including automatic file watching, version injection, and hash-based cache busting for optimal web performance.
|
||||
**Asset bundling for BRK web interfaces using Rolldown**
|
||||
|
||||
## Features
|
||||
`brk_bundler` provides JavaScript/TypeScript bundling capabilities for BRK's web interfaces. It's a thin wrapper around Rolldown (Rust-based Rollup alternative) with BRK-specific configuration for building optimized web assets with file watching and automatic rebuilding.
|
||||
|
||||
- **JavaScript bundling**: Bundles and minifies JavaScript with source maps
|
||||
- **Live reloading**: Automatic rebuilding and file watching during development
|
||||
- **Cache busting**: Hash-based filenames for browser cache invalidation
|
||||
- **Version injection**: Automatic version injection into service workers
|
||||
- **Asset copying**: Copies static assets from source to distribution directory
|
||||
- **Development mode**: Optional watch mode for real-time development
|
||||
## What it provides
|
||||
|
||||
- **JavaScript Bundling**: Modern ES modules bundling with minification
|
||||
- **File Watching**: Automatic rebuilding on source file changes
|
||||
- **Asset Processing**: Copies and processes static assets
|
||||
- **Version Injection**: Automatic version string replacement in service workers
|
||||
- **Development Mode**: Live rebuilding for rapid development
|
||||
|
||||
## Key Features
|
||||
|
||||
### Bundling Capabilities
|
||||
- **ES Module Support**: Modern JavaScript bundling with tree-shaking
|
||||
- **Minification**: Automatic code minification for production builds
|
||||
- **Source Maps**: Generated source maps for debugging
|
||||
- **Entry Point Processing**: Configurable entry points with hashed output names
|
||||
|
||||
### File System Operations
|
||||
- **Directory Copying**: Copies entire source directories to distribution
|
||||
- **Selective Processing**: Special handling for specific file types
|
||||
- **Path Resolution**: Automatic path resolution and asset linking
|
||||
|
||||
### Development Features
|
||||
- **Hot Rebuilding**: Automatic rebuilds on file changes
|
||||
- **Watch Mode**: Monitors source files and triggers rebuilds
|
||||
- **Version Replacement**: Injects build version into service workers
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Bundling
|
||||
|
||||
```rust
|
||||
use brk_bundler::bundle;
|
||||
use std::path::Path;
|
||||
|
||||
async fn build_website() -> std::io::Result<()> {
|
||||
let websites_path = Path::new("./websites");
|
||||
let source_folder = "src";
|
||||
let watch = false; // Set to true for development
|
||||
// Bundle without watching (production)
|
||||
let websites_path = Path::new("./websites");
|
||||
let source_folder = "default";
|
||||
let dist_path = bundle(websites_path, source_folder, false).await?;
|
||||
|
||||
// Bundle the website
|
||||
let dist_path = bundle(websites_path, source_folder, watch).await?;
|
||||
println!("Bundled to: {:?}", dist_path);
|
||||
```
|
||||
|
||||
println!("Website built to: {}", dist_path.display());
|
||||
Ok(())
|
||||
### Development Mode with Watching
|
||||
|
||||
```rust
|
||||
// Bundle with file watching (development)
|
||||
let dist_path = bundle(websites_path, "default", true).await?;
|
||||
|
||||
// Bundler now watches for changes and rebuilds automatically
|
||||
// This will run in the background until the process exits
|
||||
```
|
||||
|
||||
### Integration with BRK CLI
|
||||
|
||||
```rust
|
||||
// Typically called from brk_cli when serving websites
|
||||
async fn setup_website(config: &Config) -> Result<PathBuf> {
|
||||
let websites_path = config.websites_path();
|
||||
let source_folder = match config.website_mode {
|
||||
WebsiteMode::Default => "default",
|
||||
WebsiteMode::Custom => "custom",
|
||||
WebsiteMode::None => return Ok(PathBuf::new()),
|
||||
};
|
||||
|
||||
// Bundle the website assets
|
||||
let dist_path = bundle(websites_path, source_folder, config.dev_mode).await?;
|
||||
|
||||
Ok(dist_path)
|
||||
}
|
||||
```
|
||||
|
||||
## Build Process
|
||||
|
||||
1. **Clean**: Removes existing distribution directory
|
||||
2. **Copy**: Copies all source files to distribution directory
|
||||
3. **Bundle**: Processes JavaScript entry point with Rolldown
|
||||
4. **Process**: Updates HTML to reference hashed JS files
|
||||
5. **Version**: Injects version strings into service workers
|
||||
6. **Watch** (optional): Monitors files for changes and rebuilds
|
||||
|
||||
## File Structure
|
||||
|
||||
Expected source structure:
|
||||
The bundler expects this directory structure:
|
||||
|
||||
```
|
||||
websites/
|
||||
├── src/
|
||||
├── default/ # Default website source
|
||||
│ ├── index.html # Main HTML file
|
||||
│ ├── service-worker.js # Service worker (optional)
|
||||
│ ├── scripts/
|
||||
│ │ └── entry.js # JavaScript entry point
|
||||
│ └── ... # Other static assets
|
||||
└── dist/ # Generated distribution files
|
||||
│ ├── service-worker.js # Service worker (version injected)
|
||||
│ ├── scripts/ # JavaScript/TypeScript source
|
||||
│ │ ├── entry.js # Main entry point
|
||||
│ │ ├── main.js # Application logic
|
||||
│ │ └── ... # Other JS modules
|
||||
│ └── assets/ # Static assets
|
||||
└── dist/ # Generated output directory
|
||||
├── index.html # Processed HTML with updated script references
|
||||
├── service-worker.js # Service worker with version injected
|
||||
├── scripts/ # Bundled and minified JavaScript
|
||||
│ └── main-[hash].js # Hashed output file
|
||||
└── assets/ # Copied static assets
|
||||
```
|
||||
|
||||
## Watch Mode
|
||||
## Bundling Process
|
||||
|
||||
When enabled, the bundler:
|
||||
- Monitors source files for changes
|
||||
- Automatically rebuilds JavaScript bundles
|
||||
- Updates HTML with new hashed filenames
|
||||
- Reprocesses service workers with version updates
|
||||
- Copies modified static assets
|
||||
1. **Clean**: Removes existing `dist/` directory
|
||||
2. **Copy**: Copies all source files to `dist/`
|
||||
3. **Bundle JavaScript**:
|
||||
- Processes `scripts/entry.js` as entry point
|
||||
- Generates minified bundle with source maps
|
||||
- Creates hashed filename for cache busting
|
||||
4. **Process HTML**: Updates script references to hashed filenames
|
||||
5. **Process Service Worker**: Injects current version string
|
||||
6. **Watch** (if enabled): Monitors for file changes and rebuilds
|
||||
|
||||
## Configuration
|
||||
|
||||
The bundler uses Rolldown with these optimized settings:
|
||||
|
||||
```rust
|
||||
BundlerOptions {
|
||||
input: Some(vec![source_entry.into()]), // scripts/entry.js
|
||||
dir: Some("./dist/scripts".to_string()), // Output directory
|
||||
cwd: Some(websites_path), // Working directory
|
||||
minify: Some(RawMinifyOptions::Bool(true)), // Enable minification
|
||||
sourcemap: Some(SourceMapType::File), // Generate source maps
|
||||
..Default::default()
|
||||
}
|
||||
```
|
||||
|
||||
## File Watching
|
||||
|
||||
In watch mode, the bundler monitors:
|
||||
|
||||
- **Source files**: Non-script files are copied on change
|
||||
- **JavaScript files**: Trigger full rebuild via Rolldown watcher
|
||||
- **HTML files**: Processed to update script references
|
||||
- **Service worker**: Version injection on changes
|
||||
|
||||
### Watch Events Handled
|
||||
|
||||
- `Create` - New files added
|
||||
- `Modify` - Existing files changed
|
||||
- Ignores `Delete` and other events
|
||||
|
||||
## Version Injection
|
||||
|
||||
Service workers get automatic version injection:
|
||||
|
||||
```javascript
|
||||
// In source service-worker.js
|
||||
const VERSION = '__VERSION__';
|
||||
|
||||
// After bundling
|
||||
const VERSION = 'v0.0.88';
|
||||
```
|
||||
|
||||
This enables proper cache invalidation across releases.
|
||||
|
||||
## Performance Features
|
||||
|
||||
- **Async Operations**: All bundling operations are async
|
||||
- **Incremental Builds**: Only rebuilds changed files in watch mode
|
||||
- **Parallel Processing**: Uses Tokio for concurrent file operations
|
||||
- **Efficient Copying**: Direct file system operations
|
||||
|
||||
## Error Handling
|
||||
|
||||
- **Graceful Failures**: Logs errors but continues watching
|
||||
- **Path Resolution**: Automatic path absolutization and validation
|
||||
- **File System Errors**: Proper error propagation with context
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `brk_rolldown` - Rust-based Rollup bundler
|
||||
- `notify` - File system watching
|
||||
- `tokio` - Async runtime for file operations
|
||||
- `sugar_path` - Path manipulation utilities
|
||||
- `log` - Error logging
|
||||
|
||||
## Integration Points
|
||||
|
||||
The bundler integrates with:
|
||||
- **brk_cli**: Called during website setup
|
||||
- **brk_server**: Serves bundled assets
|
||||
- **Development workflow**: Provides live rebuilding
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
@@ -1,73 +1,189 @@
|
||||
# brk_cli
|
||||
|
||||
Command line interface for running BRK (Bitcoin Research Kit) instances. Orchestrates the complete pipeline: parsing Bitcoin Core blocks, indexing data, computing analytics, and serving via HTTP API.
|
||||
**Command line interface for running complete BRK instances**
|
||||
|
||||
## Overview
|
||||
`brk_cli` provides the main command-line interface for operating the Bitcoin Research Kit. It orchestrates the complete data pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with automatic configuration management and graceful operation.
|
||||
|
||||
**Core Operation**: Continuous loop that waits for Bitcoin node sync, indexes new blocks, computes analytics, and serves data via HTTP.
|
||||
## What it provides
|
||||
|
||||
**Key Components**:
|
||||
- **Parser**: Reads Bitcoin Core block files
|
||||
- **Indexer**: Processes and stores blockchain data in vecs/stores
|
||||
- **Computer**: Computes analytics across 9 specialized domains
|
||||
- **Server**: HTTP API with multiple output formats (JSON, CSV, TSV, Markdown)
|
||||
- **Website**: Optional web interface (none/default/custom)
|
||||
- **Complete Pipeline Orchestration**: Coordinates parser, indexer, computer, and server components
|
||||
- **Automatic Configuration**: Saves settings to `~/.brk/config.toml` for consistent operation
|
||||
- **Continuous Operation**: Handles blockchain updates and incremental processing
|
||||
- **Web Interface Options**: Configurable website serving (none, default, custom)
|
||||
- **Graceful Shutdown**: Ctrl+C handling with proper cleanup
|
||||
|
||||
## Requirements
|
||||
## Key Features
|
||||
|
||||
- **Bitcoin Core**: Fully synced node with RPC enabled
|
||||
- **Storage**: ~32% of blockchain size (~233GB currently)
|
||||
- **Memory**: ~7-8GB peak during indexing, ~4-5GB steady state
|
||||
- **OS**: macOS or Linux (Ubuntu: `sudo apt install libssl-dev pkg-config`)
|
||||
### Pipeline Management
|
||||
- **Automatic dependency handling**: Ensures Bitcoin Core sync before processing
|
||||
- **Incremental updates**: Only processes new blocks since last run
|
||||
- **Error recovery**: Automatic retry logic and graceful error handling
|
||||
- **Resource management**: Optimized memory usage and disk I/O
|
||||
|
||||
### Configuration System
|
||||
- **Auto-save configuration**: All CLI options saved to persistent config
|
||||
- **Flexible paths**: Configurable Bitcoin directory, blocks directory, and output directory
|
||||
- **RPC authentication**: Cookie file or username/password authentication
|
||||
- **Data source options**: Configurable price fetching and exchange APIs
|
||||
|
||||
### Operation Modes
|
||||
- **Initial sync**: Full blockchain processing from genesis
|
||||
- **Continuous operation**: Real-time processing of new blocks
|
||||
- **Update mode**: Resume from last processed block
|
||||
- **Server mode**: HTTP API with optional web interface
|
||||
|
||||
## Installation
|
||||
|
||||
### Binary Release
|
||||
```bash
|
||||
# Binary
|
||||
# Download from GitHub releases
|
||||
# https://github.com/bitcoinresearchkit/brk/releases/latest
|
||||
```
|
||||
|
||||
# Via cargo
|
||||
### Via Cargo
|
||||
```bash
|
||||
cargo install brk --locked
|
||||
```
|
||||
|
||||
# From source
|
||||
### From Source
|
||||
```bash
|
||||
git clone https://github.com/bitcoinresearchkit/brk.git
|
||||
cd brk && cargo build --release
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# First run (set configuration)
|
||||
brk --brkdir ./my_data --fetch true --website default
|
||||
### First Run (Configuration Setup)
|
||||
|
||||
# Subsequent runs (uses saved config)
|
||||
```bash
|
||||
# Basic setup with default options
|
||||
brk --brkdir ./my_brk_data
|
||||
|
||||
# Full configuration
|
||||
brk --bitcoindir ~/.bitcoin \
|
||||
--brkdir ./brk_data \
|
||||
--fetch true \
|
||||
--exchanges true \
|
||||
--website default
|
||||
```
|
||||
|
||||
### Subsequent Runs
|
||||
|
||||
```bash
|
||||
# Uses saved configuration from ~/.brk/config.toml
|
||||
brk
|
||||
|
||||
# View all options
|
||||
# Override specific options
|
||||
brk --website none --fetch false
|
||||
```
|
||||
|
||||
### Command Line Options
|
||||
|
||||
```bash
|
||||
brk --help
|
||||
```
|
||||
|
||||
## Configuration
|
||||
## Configuration Reference
|
||||
|
||||
All options auto-save to `~/.brk/config.toml` for subsequent runs:
|
||||
All options are automatically saved to `~/.brk/config.toml`:
|
||||
|
||||
```bash
|
||||
# Core paths
|
||||
--bitcoindir <PATH> # Bitcoin directory (default: ~/.bitcoin)
|
||||
--blocksdir <PATH> # Block files (default: bitcoindir/blocks)
|
||||
--brkdir <PATH> # BRK output directory (default: ~/.brk)
|
||||
### Core Paths
|
||||
- `--bitcoindir <PATH>` - Bitcoin Core directory (default: `~/.bitcoin`)
|
||||
- `--blocksdir <PATH>` - Block files directory (default: `bitcoindir/blocks`)
|
||||
- `--brkdir <PATH>` - BRK output directory (default: `~/.brk`)
|
||||
|
||||
# Data sources
|
||||
-F, --fetch <BOOL> # Enable price data fetching (default: true)
|
||||
--exchanges <BOOL> # Use exchange APIs for prices (default: true)
|
||||
### Data Sources
|
||||
- `--fetch <BOOL>` - Enable price data fetching (default: `true`)
|
||||
- `--exchanges <BOOL>` - Use exchange APIs for prices (default: `true`)
|
||||
|
||||
# Server
|
||||
-w, --website <WEBSITE> # Web interface: none|default|custom
|
||||
### Web Interface
|
||||
- `--website <OPTION>` - Web interface mode:
|
||||
- `none` - API only, no web interface
|
||||
- `default` - Built-in web interface from `websites/default/`
|
||||
- `custom` - Serve custom website from `websites/custom/`
|
||||
|
||||
# Bitcoin RPC
|
||||
--rpcconnect <IP> # RPC host (default: localhost)
|
||||
--rpcport <PORT> # RPC port (default: 8332)
|
||||
--rpccookiefile <PATH> # Cookie auth (default: bitcoindir/.cookie)
|
||||
--rpcuser <USERNAME> # Username auth (alternative to cookie)
|
||||
--rpcpassword <PASSWORD> # Password auth (alternative to cookie)
|
||||
### Bitcoin Core RPC
|
||||
- `--rpcconnect <IP>` - RPC host (default: `localhost`)
|
||||
- `--rpcport <PORT>` - RPC port (default: `8332`)
|
||||
- `--rpccookiefile <PATH>` - Cookie authentication file
|
||||
- `--rpcuser <USERNAME>` - Username authentication
|
||||
- `--rpcpassword <PASSWORD>` - Password authentication
|
||||
|
||||
## Operation Flow
|
||||
|
||||
1. **Configuration Loading**: Loads saved config from `~/.brk/config.toml`
|
||||
2. **Bitcoin Core Connection**: Establishes RPC connection and waits for sync
|
||||
3. **Data Pipeline Initialization**: Sets up parser, indexer, computer, and interface
|
||||
4. **Processing Loop**:
|
||||
- Index new blocks from Bitcoin Core
|
||||
- Compute analytics on new data
|
||||
- Update cached data
|
||||
5. **Server Startup**: Launches HTTP API with optional web interface
|
||||
6. **Continuous Operation**: Monitors for new blocks and processes incrementally
|
||||
|
||||
## System Requirements
|
||||
|
||||
- **Bitcoin Core**: Fully synced node with RPC enabled
|
||||
- **Storage**: ~32% of blockchain size (~233GB as of 2025)
|
||||
- **Memory**:
|
||||
- Peak: ~7-8GB during initial indexing
|
||||
- Steady state: ~4-5GB during operation
|
||||
- **OS**: macOS or Linux
|
||||
- Ubuntu: `sudo apt install libssl-dev pkg-config`
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Initial Sync
|
||||
- **Full blockchain processing**: ~13-15 hours total
|
||||
- **Parser phase**: ~4 minutes for block parsing
|
||||
- **Indexer phase**: ~7-8 hours for data indexing
|
||||
- **Computer phase**: ~6-7 hours for analytics computation
|
||||
|
||||
### Continuous Operation
|
||||
- **New block processing**: 3-5 seconds per block
|
||||
- **API response times**: Typically <100ms with caching
|
||||
- **Memory usage**: Stable ~4-5GB during normal operation
|
||||
|
||||
## Configuration File
|
||||
|
||||
Example `~/.brk/config.toml`:
|
||||
```toml
|
||||
bitcoindir = "/Users/username/.bitcoin"
|
||||
blocksdir = "/Users/username/.bitcoin/blocks"
|
||||
brkdir = "/Users/username/brk_data"
|
||||
fetch = true
|
||||
exchanges = true
|
||||
website = "default"
|
||||
rpcconnect = "localhost"
|
||||
rpcport = 8332
|
||||
rpccookiefile = "/Users/username/.bitcoin/.cookie"
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
- **Bitcoin Core sync**: Waits for node sync before processing
|
||||
- **RPC connection**: Automatic retry logic for connection issues
|
||||
- **Processing errors**: Graceful error handling with detailed logging
|
||||
- **Graceful shutdown**: Ctrl+C handling with proper cleanup and state saving
|
||||
|
||||
## Logging
|
||||
|
||||
Logs are written to `~/.brk/brk.log` with colored console output:
|
||||
- Request/response logging with timing
|
||||
- Processing progress indicators
|
||||
- Error reporting and debugging information
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `brk_parser` - Bitcoin block parsing
|
||||
- `brk_indexer` - Blockchain data indexing
|
||||
- `brk_computer` - Analytics computation
|
||||
- `brk_interface` - Data query interface
|
||||
- `brk_server` - HTTP API server
|
||||
- `brk_logger` - Logging utilities
|
||||
- `bitcoincore_rpc` - Bitcoin Core RPC client
|
||||
- `color_eyre` - Enhanced error reporting
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
|
||||
@@ -1,64 +1,200 @@
|
||||
# brk_computer
|
||||
|
||||
Bitcoin analytics engine that transforms indexed blockchain data into computed datasets and metrics. Uses a modular vector architecture with lazy computation and compressed storage for optimal performance.
|
||||
**Bitcoin analytics engine that transforms indexed blockchain data into comprehensive metrics**
|
||||
|
||||
## Overview
|
||||
`brk_computer` is the computational layer of BRK that processes indexed blockchain data to generate analytics across multiple specialized domains. It provides comprehensive Bitcoin metrics with efficient storage and lazy computation for optimal performance.
|
||||
|
||||
Computes analytics across 9 specialized domains, each implementing the compute trait pattern:
|
||||
## What it provides
|
||||
|
||||
- **indexes** - Time-based indexing (date/height mappings, epoch calculations)
|
||||
- **constants** - Baseline values for calculations
|
||||
- **blocks** - Block analytics (sizes, intervals, transaction counts)
|
||||
- **mining** - Mining economics (hashrate, difficulty, rewards)
|
||||
- **transactions** - Transaction analysis (fees, sizes, patterns, RBF)
|
||||
- **stateful** - UTXO tracking and accumulated state computations
|
||||
- **cointime** - Coin age and time-based value analysis
|
||||
- **fetched** - External price data integration (optional)
|
||||
- **price** - OHLC data across timeframes (optional, requires fetched)
|
||||
- **market** - Price correlations and market metrics (optional, requires price)
|
||||
- **Comprehensive Analytics**: 9 specialized domains covering all aspects of Bitcoin analysis
|
||||
- **Lazy Computation**: On-demand calculation with dependency tracking and caching
|
||||
- **Incremental Updates**: Only processes new data since last computation
|
||||
- **Memory Efficiency**: ~100MB operation footprint via compressed storage and memory mapping
|
||||
- **Multi-timeframe Analysis**: Daily, weekly, monthly, quarterly, yearly perspectives
|
||||
|
||||
**Computation order**: Fixed dependency chain ensures data consistency (indexes → constants → blocks → mining → fetched → price → transactions → market → stateful → cointime).
|
||||
## Nine Analytics Domains
|
||||
|
||||
**Storage**: Uses vecdb with lazy computation and compressed format for efficient disk usage and memory management.
|
||||
The computer processes data through a fixed dependency chain:
|
||||
|
||||
1. **indexes** - Time-based indexing (date/height mappings, epoch calculations)
|
||||
2. **constants** - Baseline values and reference metrics
|
||||
3. **blocks** - Block analytics (sizes, intervals, transaction counts, weight)
|
||||
4. **mining** - Mining economics (hashrate, difficulty, rewards, epochs)
|
||||
5. **fetched** - External price data integration (optional)
|
||||
6. **price** - OHLC data across multiple timeframes (optional, requires fetched)
|
||||
7. **transactions** - Transaction analysis (fees, sizes, patterns, RBF detection)
|
||||
8. **market** - Price correlations and market metrics (optional, requires price)
|
||||
9. **stateful** - UTXO tracking and accumulated state computations
|
||||
10. **cointime** - Coin age and time-based value analysis
|
||||
|
||||
## Key Features
|
||||
|
||||
### Computation Strategy
|
||||
- **Fixed dependency chain**: Ensures data consistency across all domains
|
||||
- **Parallel processing**: Uses Rayon for performance optimization
|
||||
- **State management**: Rollback capabilities for error recovery
|
||||
- **Incremental updates**: Only computes new data since last run
|
||||
|
||||
### Analytics Capabilities
|
||||
- **Multi-timeframe analysis**: Daily, weekly, monthly, quarterly, yearly aggregations
|
||||
- **Chain-based metrics**: Height, difficulty epoch, halving epoch indexing
|
||||
- **Price correlation**: Both dollar and satoshi denominated metrics
|
||||
- **DCA analysis**: Dollar Cost Averaging with configurable periods
|
||||
- **Supply analysis**: Circulating, realized, unrealized supply metrics
|
||||
- **Address cohort tracking**: Analysis across different Bitcoin address types
|
||||
- **UTXO cohort analysis**: Realized/unrealized gains tracking
|
||||
- **Coin time analysis**: Understanding Bitcoin velocity and dormancy
|
||||
|
||||
### Storage Optimization
|
||||
- **Compressed vectors**: Efficient disk storage with lazy computation
|
||||
- **Memory mapping**: Minimal RAM usage during operation
|
||||
- **Version management**: Automatic invalidation on schema changes
|
||||
- **Dependency tracking**: Smart recomputation based on data changes
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Setup (No Price Data)
|
||||
|
||||
```rust
|
||||
use brk_computer::Computer;
|
||||
use brk_indexer::Indexer;
|
||||
use brk_fetcher::Fetcher;
|
||||
use vecdb::Exit;
|
||||
|
||||
// Basic setup - computes all domains except price/market
|
||||
// Setup without external price data
|
||||
let indexer = Indexer::forced_import("./brk_data")?;
|
||||
let mut computer = Computer::forced_import("./brk_data", &indexer, None)?;
|
||||
|
||||
// With price data - enables market analytics
|
||||
// Setup exit handler
|
||||
let exit = Exit::new();
|
||||
exit.set_ctrlc_handler();
|
||||
|
||||
// Compute all analytics
|
||||
let starting_indexes = indexer.get_starting_indexes();
|
||||
computer.compute(&indexer, starting_indexes, &exit)?;
|
||||
```
|
||||
|
||||
### Advanced Setup (With Price Data)
|
||||
|
||||
```rust
|
||||
use brk_fetcher::Fetcher;
|
||||
|
||||
// Setup with external price data for market analytics
|
||||
let fetcher = Some(Fetcher::import(true, None)?);
|
||||
let mut computer = Computer::forced_import("./brk_data", &indexer, fetcher)?;
|
||||
|
||||
// Compute all analytics from starting point
|
||||
let starting_indexes = indexer.get_starting_indexes();
|
||||
// Compute all analytics including price/market domains
|
||||
computer.compute(&indexer, starting_indexes, &exit)?;
|
||||
|
||||
// Access computed vectors
|
||||
let all_vecs = computer.vecs(); // Returns Vec<&dyn AnyCollectableVec>
|
||||
```
|
||||
|
||||
## Key Implementation Details
|
||||
### Accessing Computed Data
|
||||
|
||||
- **Forced import pattern**: Single computer instance per output directory to prevent conflicts
|
||||
- **Lazy computation**: Vectors computed on-demand, cached with dependency tracking
|
||||
- **Incremental updates**: Only processes new data since last computation
|
||||
- **Memory efficient**: ~100MB max via compressed storage and memory mapping
|
||||
- **Exit handling**: Graceful shutdown support with computation state preservation
|
||||
```rust
|
||||
// Access all computed vectors
|
||||
let all_vecs = computer.vecs(); // Returns Vec<&dyn AnyCollectableVec>
|
||||
|
||||
## Performance
|
||||
// Access specific domain data
|
||||
let block_metrics = &computer.blocks;
|
||||
let mining_data = &computer.mining;
|
||||
let transaction_stats = &computer.transactions;
|
||||
|
||||
Benchmarked on MacBook Pro M3 Pro:
|
||||
// Access price data (if available)
|
||||
if let Some(price_data) = &computer.price {
|
||||
// Use OHLC data
|
||||
}
|
||||
```
|
||||
|
||||
- **Initial computation**: ~6-7 hours for complete Bitcoin blockchain analysis
|
||||
- **Storage efficiency**: All computed datasets total only ~40GB
|
||||
### Incremental Updates
|
||||
|
||||
```rust
|
||||
// Continuous computation loop
|
||||
loop {
|
||||
// Get latest indexes from indexer
|
||||
let current_indexes = indexer.get_current_indexes();
|
||||
|
||||
// Compute only new data
|
||||
computer.compute(&indexer, current_indexes, &exit)?;
|
||||
|
||||
// Check for exit signal
|
||||
if exit.is_signaled() {
|
||||
break;
|
||||
}
|
||||
|
||||
// Wait before next update
|
||||
sleep(Duration::from_secs(60));
|
||||
}
|
||||
```
|
||||
|
||||
## Core Computer Structure
|
||||
|
||||
```rust
|
||||
pub struct Computer {
|
||||
pub indexes: indexes::Vecs, // Time indexing
|
||||
pub constants: constants::Vecs, // Baseline values
|
||||
pub blocks: blocks::Vecs, // Block analytics
|
||||
pub mining: mining::Vecs, // Mining economics
|
||||
pub market: market::Vecs, // Market metrics (optional)
|
||||
pub price: Option<price::Vecs>, // OHLC price data (optional)
|
||||
pub transactions: transactions::Vecs, // Transaction analysis
|
||||
pub stateful: stateful::Vecs, // UTXO tracking
|
||||
pub fetched: Option<fetched::Vecs>, // External data (optional)
|
||||
pub cointime: cointime::Vecs, // Coin age analysis
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
**Benchmarked on MacBook Pro M3 Pro:**
|
||||
- **Initial computation**: ~6-7 hours for complete Bitcoin blockchain
|
||||
- **Storage efficiency**: All computed datasets total ~40GB
|
||||
- **Incremental updates**: 3-5 seconds per new block
|
||||
- **Memory footprint**: Peak ~7-8GB during computation, ~100MB during operation
|
||||
- **Dependencies**: Price data domains optional (fetched, price, market)
|
||||
|
||||
The initial computation processes the entire blockchain history once to generate all analytical datasets. Subsequent updates are near-instant, making BRK suitable for real-time analysis and production deployments.
|
||||
## Domain-Specific Analytics
|
||||
|
||||
### Block Analytics
|
||||
- Block sizes, weights, transaction counts
|
||||
- Block intervals and mining statistics
|
||||
- Fee analysis per block
|
||||
|
||||
### Mining Economics
|
||||
- Hashrate estimation and difficulty tracking
|
||||
- Mining reward analysis
|
||||
- Epoch-based calculations
|
||||
|
||||
### Transaction Analysis
|
||||
- Fee rate distributions
|
||||
- RBF (Replace-By-Fee) detection
|
||||
- Output type analysis
|
||||
- Transaction size patterns
|
||||
|
||||
### Market Metrics (Optional)
|
||||
- Price correlations with on-chain metrics
|
||||
- Market cap calculations
|
||||
- DCA analysis across timeframes
|
||||
|
||||
### Stateful Analysis
|
||||
- UTXO set tracking
|
||||
- Address cohort analysis
|
||||
- Realized/unrealized gains
|
||||
- Supply distribution metrics
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Indexed data**: Requires completed `brk_indexer` output
|
||||
- **Storage space**: Additional ~40GB for computed datasets
|
||||
- **Memory**: 8GB+ RAM recommended for initial computation
|
||||
- **CPU**: Multi-core recommended for parallel processing
|
||||
- **Price data**: Optional external price feeds for market analytics
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `brk_indexer` - Source of indexed blockchain data
|
||||
- `brk_fetcher` - External price data (optional)
|
||||
- `vecdb` - Vector database with lazy computation
|
||||
- `rayon` - Parallel processing framework
|
||||
- `brk_structs` - Bitcoin-aware type system
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
@@ -1,57 +1,140 @@
|
||||
# brk_error
|
||||
|
||||
Centralized error handling for the Bitcoin Research Kit that provides a unified error type and result type for consistent error propagation across all BRK crates. This crate consolidates errors from external dependencies and defines domain-specific error variants used throughout the BRK ecosystem.
|
||||
**Centralized error handling for the Bitcoin Research Kit**
|
||||
|
||||
## Error Types
|
||||
`brk_error` provides a unified error type and result system used throughout the BRK ecosystem. It consolidates error handling from multiple external dependencies and adds Bitcoin-specific error variants.
|
||||
|
||||
### External Library Errors
|
||||
- **IO**: Standard I/O operations (`std::io::Error`)
|
||||
- **BitcoinRPC**: Bitcoin Core RPC client errors
|
||||
- **Jiff**: Date/time parsing and manipulation errors
|
||||
- **Fjall**: Key-value store errors
|
||||
- **VecDB/SeqDB**: Vector database errors
|
||||
- **Minreq**: HTTP client errors
|
||||
- **SerdeJson**: JSON serialization/deserialization errors
|
||||
- **ZeroCopy**: Memory layout conversion errors
|
||||
- **SystemTime**: System time errors
|
||||
## What it provides
|
||||
|
||||
### Domain-Specific Errors
|
||||
- **WrongLength**: Invalid data length
|
||||
- **WrongAddressType**: Unsupported Bitcoin address type
|
||||
- **UnindexableDate**: Date outside indexable range (before 2009-01-03)
|
||||
- **QuickCacheError**: Cache operation failures
|
||||
- **Str/String**: Custom error messages
|
||||
- **Unified Error Type**: Single `Error` enum that covers all error cases across BRK crates
|
||||
- **Convenient Result Type**: Pre-configured `Result<T, E = Error>` for consistent error handling
|
||||
- **External Error Integration**: Automatic conversions from common library errors
|
||||
- **Bitcoin-Specific Errors**: Domain-specific error variants for blockchain data processing
|
||||
|
||||
## Key Features
|
||||
|
||||
### Centralized Error Management
|
||||
- Single error type for the entire BRK ecosystem
|
||||
- Consistent error handling patterns across all crates
|
||||
- Reduced error type complexity in public APIs
|
||||
|
||||
### External Library Integration
|
||||
Automatic `From` implementations for errors from:
|
||||
- **I/O Operations**: `std::io::Error`
|
||||
- **Bitcoin Core RPC**: `bitcoincore_rpc::Error`
|
||||
- **Database Operations**: `fjall::Error`, `vecdb::Error`
|
||||
- **Serialization**: `serde_json::Error`
|
||||
- **Time Operations**: `jiff::Error`, `SystemTimeError`
|
||||
- **HTTP Requests**: `minreq::Error`
|
||||
- **Zero-Copy Operations**: `zerocopy` conversion errors
|
||||
|
||||
### Bitcoin-Specific Error Variants
|
||||
- `WrongAddressType` - Invalid address type for operation
|
||||
- `UnindexableDate` - Date before Bitcoin genesis (2009-01-03)
|
||||
- `WrongLength` - Invalid data length for Bitcoin structures
|
||||
- `QuickCacheError` - Cache operation failures
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Error Handling
|
||||
|
||||
```rust
|
||||
use brk_error::{Error, Result};
|
||||
|
||||
fn process_bitcoin_data() -> Result<()> {
|
||||
// Operations that may fail with various error types
|
||||
let data = std::fs::read("blocks.dat")?; // IO error
|
||||
let parsed = parse_data(&data)?; // Custom error
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn parse_data(data: &[u8]) -> Result<ParsedData> {
|
||||
if data.len() < 80 {
|
||||
return Err(Error::WrongLength);
|
||||
fn process_block() -> Result<Block> {
|
||||
let rpc_client = get_rpc_client()?; // bitcoincore_rpc::Error -> Error
|
||||
let block_data = rpc_client.get_block_info(&hash)?;
|
||||
|
||||
// Custom Bitcoin-specific validation
|
||||
if block_data.height < 0 {
|
||||
return Err(Error::Str("Invalid block height"));
|
||||
}
|
||||
// ... parsing logic
|
||||
Ok(parsed_data)
|
||||
|
||||
Ok(block_data)
|
||||
}
|
||||
```
|
||||
|
||||
## Type Alias
|
||||
|
||||
The crate exports `Result<T, E = Error>` as the standard result type, allowing for concise error handling:
|
||||
### Working with External Libraries
|
||||
|
||||
```rust
|
||||
use brk_error::Result;
|
||||
|
||||
fn my_function() -> Result<String> {
|
||||
// Automatically uses brk_error::Error as the error type
|
||||
Ok("success".to_string())
|
||||
fn save_data(data: &[u8]) -> Result<()> {
|
||||
// I/O error automatically converted
|
||||
std::fs::write("data.bin", data)?;
|
||||
|
||||
// JSON serialization error automatically converted
|
||||
let json = serde_json::to_string(&data)?;
|
||||
|
||||
// Database error automatically converted
|
||||
database.insert("key", &json)?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
### Bitcoin-Specific Validation
|
||||
|
||||
```rust
|
||||
use brk_error::{Error, Result};
|
||||
|
||||
fn validate_date(date: &Date) -> Result<()> {
|
||||
if *date < Date::new(2009, 1, 3) {
|
||||
return Err(Error::UnindexableDate);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_address_type(output_type: OutputType) -> Result<()> {
|
||||
if !output_type.is_address() {
|
||||
return Err(Error::WrongAddressType);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
### String Errors
|
||||
|
||||
```rust
|
||||
// Static string errors (zero allocation)
|
||||
Err(Error::Str("Invalid configuration"))
|
||||
|
||||
// Dynamic string errors
|
||||
Err(Error::String(format!("Block {} not found", height)))
|
||||
```
|
||||
|
||||
## Result Type
|
||||
|
||||
The crate provides a convenient `Result` type alias:
|
||||
|
||||
```rust
|
||||
pub type Result<T, E = Error> = std::result::Result<T, E>;
|
||||
```
|
||||
|
||||
This allows for clean function signatures throughout BRK:
|
||||
|
||||
```rust
|
||||
fn parse_block() -> Result<Block> { /* ... */ }
|
||||
fn index_transactions() -> Result<Vec<Transaction>> { /* ... */ }
|
||||
```
|
||||
|
||||
## Error Display
|
||||
|
||||
All errors implement `Display` and `std::error::Error`, providing:
|
||||
- Formatted error messages for debugging
|
||||
- Error chain support for nested errors
|
||||
- Integration with error handling libraries like `anyhow`
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `vecdb` - Vector database error types
|
||||
- `bitcoincore-rpc` - Bitcoin Core RPC client errors
|
||||
- `fjall` - Key-value store errors
|
||||
- `jiff` - Date/time operation errors
|
||||
- `minreq` - HTTP request errors
|
||||
- `serde_json` - JSON serialization errors
|
||||
- `zerocopy` - Zero-copy conversion errors
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
@@ -1,57 +1,186 @@
|
||||
# brk_fetcher
|
||||
|
||||
Bitcoin price fetcher that retrieves historical OHLC (Open, High, Low, Close) data by date or block height from multiple sources including Binance, Kraken, and the main BRK instance. This crate provides a unified interface with automatic fallback between exchanges and retry logic for reliable price data collection.
|
||||
**Bitcoin price data fetcher with multi-source fallback and retry logic**
|
||||
|
||||
## Features
|
||||
`brk_fetcher` provides reliable Bitcoin price data retrieval from multiple sources including Binance, Kraken, and BRK instances. It offers both date-based and block height-based price queries with automatic fallback and retry mechanisms for robust data collection.
|
||||
|
||||
- **Multiple data sources**: Binance, Kraken APIs, and BRK instance
|
||||
- **Flexible queries**: Fetch prices by date or block height with timestamp
|
||||
- **Automatic fallback**: Tries sources in order (Kraken → Binance → BRK)
|
||||
- **Retry logic**: Built-in retry mechanism
|
||||
- **Time resolution**: 1-minute and 1-day interval support
|
||||
- **HAR file import**: Import Binance chart data from browser for historical prices
|
||||
## What it provides
|
||||
|
||||
- **Multi-source fallback**: Automatic fallback between Kraken → Binance → BRK
|
||||
- **Flexible querying**: Fetch prices by date or block height with timestamps
|
||||
- **Retry logic**: Built-in retry mechanism with exponential backoff
|
||||
- **Multiple timeframes**: 1-minute and 1-day interval support
|
||||
- **HAR file import**: Import historical Binance chart data from browser
|
||||
|
||||
## Key Features
|
||||
|
||||
### Data Sources
|
||||
- **Kraken API**: Primary source for OHLC data (1-day and 1-minute intervals)
|
||||
- **Binance API**: Secondary source with additional historical data
|
||||
- **BRK instance**: Fallback source for previously cached price data
|
||||
- **HAR import**: Manual historical data import from browser sessions
|
||||
|
||||
### Query Methods
|
||||
- **Date-based queries**: Get OHLC data for specific calendar dates
|
||||
- **Height-based queries**: Get OHLC data for specific block heights with timestamps
|
||||
- **Automatic aggregation**: Combines minute-level data for block intervals
|
||||
|
||||
### Reliability Features
|
||||
- **Automatic fallback**: Tries sources in order until successful
|
||||
- **Retry mechanism**: Up to 12 hours of retries with 60-second intervals
|
||||
- **Cache clearing**: Automatic cache refresh on failures
|
||||
- **Error handling**: Graceful degradation with detailed error messages
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Setup
|
||||
|
||||
```rust
|
||||
use brk_fetcher::Fetcher;
|
||||
use brk_structs::{Date, Height};
|
||||
use brk_structs::{Date, Height, Timestamp};
|
||||
|
||||
fn main() -> brk_error::Result<()> {
|
||||
// Initialize fetcher with exchange APIs enabled
|
||||
let mut fetcher = Fetcher::import(true, None)?;
|
||||
// Initialize fetcher with exchange APIs enabled
|
||||
let mut fetcher = Fetcher::import(true, None)?;
|
||||
|
||||
// Fetch price by date
|
||||
let price = fetcher.get_date(Date::new(2025, 1, 15))?;
|
||||
println!("Price on 2025-01-15: ${}", price.close.dollars());
|
||||
// Initialize without exchange APIs (BRK-only mode)
|
||||
let mut fetcher = Fetcher::import(false, None)?;
|
||||
|
||||
// Fetch price by block height
|
||||
let price = fetcher.get_height(
|
||||
Height::new(900_000),
|
||||
timestamp,
|
||||
previous_timestamp,
|
||||
)?;
|
||||
println!("Price at block 900,000: ${}", price.close.dollars());
|
||||
|
||||
Ok(())
|
||||
}
|
||||
// Initialize with HAR file for historical Binance data
|
||||
let har_path = Path::new("./binance.har");
|
||||
let mut fetcher = Fetcher::import(true, Some(har_path))?;
|
||||
```
|
||||
|
||||
## Individual Sources
|
||||
### Date-based Price Queries
|
||||
|
||||
Each exchange can be used independently:
|
||||
```rust
|
||||
use brk_structs::Date;
|
||||
|
||||
// Fetch OHLC data for a specific date
|
||||
let date = Date::new(2024, 12, 25);
|
||||
let ohlc = fetcher.get_date(date)?;
|
||||
|
||||
println!("Bitcoin price on {}: ${:.2}", date, ohlc.close.dollars());
|
||||
println!("Daily high: ${:.2}", ohlc.high.dollars());
|
||||
println!("Daily low: ${:.2}", ohlc.low.dollars());
|
||||
```
|
||||
|
||||
### Block Height-based Price Queries
|
||||
|
||||
```rust
|
||||
use brk_structs::{Height, Timestamp};
|
||||
|
||||
// Fetch price at specific block height
|
||||
let height = Height::new(900_000);
|
||||
let timestamp = Timestamp::from_block_height(height);
|
||||
let previous_timestamp = Some(Timestamp::from_block_height(Height::new(899_999)));
|
||||
|
||||
let ohlc = fetcher.get_height(height, timestamp, previous_timestamp)?;
|
||||
println!("Bitcoin price at block {}: ${:.2}", height, ohlc.close.dollars());
|
||||
```
|
||||
|
||||
### Working with OHLC Data
|
||||
|
||||
```rust
|
||||
use brk_structs::OHLCCents;
|
||||
|
||||
// OHLC data is returned in cents for precision
|
||||
let ohlc: OHLCCents = fetcher.get_date(date)?;
|
||||
|
||||
// Convert to dollars for display
|
||||
println!("Open: ${:.2}", ohlc.open.dollars());
|
||||
println!("High: ${:.2}", ohlc.high.dollars());
|
||||
println!("Low: ${:.2}", ohlc.low.dollars());
|
||||
println!("Close: ${:.2}", ohlc.close.dollars());
|
||||
|
||||
// Access raw cent values
|
||||
println!("Close in cents: {}", ohlc.close.0);
|
||||
```
|
||||
|
||||
### Using Individual Sources
|
||||
|
||||
```rust
|
||||
use brk_fetcher::{Binance, Kraken, BRK};
|
||||
|
||||
// Fetch from specific exchanges
|
||||
let binance_data = Binance::fetch_1d()?;
|
||||
let kraken_data = Kraken::fetch_1mn()?;
|
||||
let brk_data = BRK::default().get_from_height(Height::new(800_000))?;
|
||||
// Use specific exchanges directly
|
||||
let binance = Binance::init(None);
|
||||
let kraken = Kraken::default();
|
||||
let brk = BRK::default();
|
||||
|
||||
// Fetch from specific source
|
||||
let binance_data = binance.get_from_1d(&date)?;
|
||||
let kraken_data = kraken.get_from_1mn(timestamp, previous_timestamp)?;
|
||||
let brk_data = brk.get_from_height(height)?;
|
||||
```
|
||||
|
||||
## Limitations
|
||||
### Error Handling and Retries
|
||||
|
||||
- **1-minute data**: Limited to last 16 hours (Binance) or 10 hours (Kraken)
|
||||
- **Network dependent**: Requires internet connection for exchange APIs
|
||||
- **Rate limits**: Subject to exchange API rate limiting
|
||||
```rust
|
||||
// The fetcher automatically retries on failures
|
||||
match fetcher.get_date(date) {
|
||||
Ok(ohlc) => println!("Successfully fetched: ${:.2}", ohlc.close.dollars()),
|
||||
Err(e) => {
|
||||
// After all retries and sources exhausted
|
||||
eprintln!("Failed to fetch price data: {}", e);
|
||||
}
|
||||
}
|
||||
|
||||
// Clear cache to force fresh data
|
||||
fetcher.clear();
|
||||
```
|
||||
|
||||
## Data Sources and Limitations
|
||||
|
||||
### Kraken API
|
||||
- **1-day data**: Historical daily OHLC data
|
||||
- **1-minute data**: Limited to last ~10 hours
|
||||
- **Rate limits**: Subject to Kraken API restrictions
|
||||
|
||||
### Binance API
|
||||
- **1-day data**: Historical daily OHLC data
|
||||
- **1-minute data**: Limited to last ~16 hours
|
||||
- **HAR import**: Can extend historical coverage via browser data
|
||||
|
||||
### BRK Instance
|
||||
- **Cached data**: Previously fetched price data
|
||||
- **Offline capability**: Works without internet when data is cached
|
||||
- **Height-based**: Optimized for block height queries
|
||||
|
||||
## HAR File Import
|
||||
|
||||
For historical data beyond API limits:
|
||||
|
||||
1. Visit [Binance BTC/USDT chart](https://www.binance.com/en/trade/BTC_USDT?type=spot)
|
||||
2. Set chart to 1-minute interval
|
||||
3. Open browser dev tools, go to Network tab
|
||||
4. Filter by 'uiKlines'
|
||||
5. Scroll chart to desired historical period
|
||||
6. Export network requests as HAR file
|
||||
7. Initialize fetcher with HAR path
|
||||
|
||||
## Fallback Strategy
|
||||
|
||||
The fetcher tries sources in this order:
|
||||
1. **Kraken** - Primary source for most queries
|
||||
2. **Binance** - Secondary source with extended coverage
|
||||
3. **BRK** - Fallback for cached/computed prices
|
||||
|
||||
If all sources fail, it retries up to 12 hours with 60-second intervals.
|
||||
|
||||
## Performance and Reliability
|
||||
|
||||
- **Automatic retries**: Up to 720 attempts (12 hours) with 60-second delays
|
||||
- **Cache management**: Clears cache on failures to force fresh data
|
||||
- **Error logging**: Detailed failure reporting with recovery instructions
|
||||
- **Graceful degradation**: Falls back through sources until successful
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `brk_structs` - Bitcoin-aware type system (Date, Height, OHLC types)
|
||||
- `brk_error` - Unified error handling
|
||||
- `minreq` - HTTP client for API requests
|
||||
- `serde_json` - JSON parsing for API responses
|
||||
- `log` - Logging for retry and error reporting
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
@@ -1,105 +1,190 @@
|
||||
# brk_indexer
|
||||
|
||||
Bitcoin blockchain indexer that processes raw block data from Bitcoin Core and creates efficient storage structures using vectors and key-value stores for fast data retrieval and analysis. This crate builds the foundation for BRK's data pipeline by extracting and organizing blockchain data into optimized storage formats.
|
||||
**High-performance Bitcoin blockchain indexer with dual storage architecture**
|
||||
|
||||
## Features
|
||||
`brk_indexer` processes raw Bitcoin Core block data and creates efficient storage structures using both vectors (time-series) and key-value stores (lookups). It serves as the foundation of BRK's data pipeline, organizing all blockchain data into optimized formats for fast retrieval and analysis.
|
||||
|
||||
- **Block-by-block processing**: Iterates through blockchain using brk_parser
|
||||
- **Dual storage architecture**: Combines vectors (brk_vec) for time-series data and key-value stores (brk_store) for lookups
|
||||
- **Memory efficient**: ~5GB peak RAM usage during indexing
|
||||
- **Collision detection**: Validates data integrity with optional collision checking
|
||||
- **Incremental updates**: Supports resuming from last indexed height
|
||||
- **Rollback protection**: Automatic rollback on interruption or errors
|
||||
## What it provides
|
||||
|
||||
## Storage Strategy
|
||||
- **Dual Storage Architecture**: Vectors for time-series data, key-value stores for lookups
|
||||
- **Memory Efficiency**: ~5-6GB peak RAM usage during full blockchain indexing
|
||||
- **Incremental Processing**: Resume from last indexed height with rollback protection
|
||||
- **Data Integrity**: Collision detection and validation during indexing
|
||||
- **All Bitcoin Data Types**: Complete support for blocks, transactions, inputs, outputs, and addresses
|
||||
|
||||
### Vectors (brk_vec)
|
||||
## Key Features
|
||||
|
||||
Used for sequential, time-indexed data:
|
||||
- Block metadata (height, timestamp, hash)
|
||||
- Transaction counts and statistics
|
||||
- Price data and market metrics
|
||||
### Storage Strategy
|
||||
|
||||
**Vector Storage (time-series data):**
|
||||
- Block metadata (height, timestamp, hash, difficulty, size)
|
||||
- Transaction data (version, locktime, RBF flag, indices)
|
||||
- Input/Output mappings and values
|
||||
- Address bytes for all output types
|
||||
- Efficient for range queries and analytics
|
||||
|
||||
### Key-Value Stores (brk_store)
|
||||
**Key-Value Storage (lookups):**
|
||||
- Block hash prefixes → heights
|
||||
- Transaction ID prefixes → transaction indices
|
||||
- Address byte hashes → type indices
|
||||
- Fast point queries by hash or address
|
||||
|
||||
Used for lookup operations:
|
||||
- Address mappings and balances
|
||||
- Transaction and UTXO data
|
||||
- Script and output type indices
|
||||
- Fast point queries by hash/address
|
||||
### Performance Features
|
||||
- **Parallel Processing**: Concurrent transaction and output processing using Rayon
|
||||
- **Batch Operations**: Periodic commits every 1,000 blocks for optimal I/O
|
||||
- **Memory Efficiency**: Optimized data structures minimize RAM usage
|
||||
- **Incremental Updates**: Handles blockchain reorganizations automatically
|
||||
|
||||
### Address Type Support
|
||||
Complete support for all Bitcoin address types:
|
||||
- P2PK (65-byte and 33-byte), P2PKH, P2SH
|
||||
- P2WPKH, P2WSH, P2TR, P2A
|
||||
- P2MS (multisig), OpReturn, Empty, Unknown
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Indexing
|
||||
|
||||
```rust
|
||||
use brk_indexer::Indexer;
|
||||
use brk_parser::Parser;
|
||||
use bitcoincore_rpc::{Auth, Client};
|
||||
use vecdb::Exit;
|
||||
use std::path::Path;
|
||||
|
||||
fn main() -> brk_error::Result<()> {
|
||||
// Setup paths and RPC
|
||||
let bitcoin_dir = Path::new("~/.bitcoin");
|
||||
let outputs_dir = Path::new("./brk_data");
|
||||
// Setup Bitcoin Core RPC connection
|
||||
let rpc = Box::leak(Box::new(Client::new(
|
||||
"http://localhost:8332",
|
||||
Auth::CookieFile(Path::new("~/.bitcoin/.cookie")),
|
||||
)?));
|
||||
|
||||
let rpc = Box::leak(Box::new(Client::new(
|
||||
"http://localhost:8332",
|
||||
Auth::CookieFile(bitcoin_dir.join(".cookie")),
|
||||
)?));
|
||||
// Create parser for Bitcoin Core block files
|
||||
let parser = Parser::new(
|
||||
Path::new("~/.bitcoin/blocks").to_path_buf(),
|
||||
Path::new("./brk_data").to_path_buf(),
|
||||
rpc
|
||||
);
|
||||
|
||||
// Create parser and indexer
|
||||
let parser = Parser::new(
|
||||
bitcoin_dir.join("blocks"),
|
||||
outputs_dir.to_path_buf(),
|
||||
rpc
|
||||
);
|
||||
// Create indexer with forced import (resets if needed)
|
||||
let mut indexer = Indexer::forced_import(Path::new("./brk_data"))?;
|
||||
|
||||
let mut indexer = Indexer::forced_import(outputs_dir)?;
|
||||
// Setup graceful shutdown handler
|
||||
let exit = Exit::new();
|
||||
exit.set_ctrlc_handler();
|
||||
|
||||
// Setup exit handler
|
||||
let exit = Exit::new();
|
||||
exit.set_ctrlc_handler();
|
||||
// Index the blockchain
|
||||
let indexes = indexer.index(&parser, rpc, &exit, true)?;
|
||||
println!("Indexed up to height: {}", indexes.height);
|
||||
```
|
||||
|
||||
// Index the blockchain
|
||||
### Continuous Indexing
|
||||
|
||||
```rust
|
||||
use std::time::{Duration, Instant};
|
||||
use std::thread::sleep;
|
||||
|
||||
// Continuous indexing loop for real-time updates
|
||||
loop {
|
||||
let start_time = Instant::now();
|
||||
|
||||
// Index new blocks
|
||||
let indexes = indexer.index(&parser, rpc, &exit, true)?;
|
||||
|
||||
println!("Indexed up to height: {}", indexes.height);
|
||||
|
||||
Ok(())
|
||||
|
||||
println!("Indexed to height {} in {:?}",
|
||||
indexes.height, start_time.elapsed());
|
||||
|
||||
// Check for exit signal
|
||||
if exit.is_signaled() {
|
||||
println!("Graceful shutdown requested");
|
||||
break;
|
||||
}
|
||||
|
||||
// Wait before next update cycle
|
||||
sleep(Duration::from_secs(5 * 60));
|
||||
}
|
||||
```
|
||||
|
||||
## Performance
|
||||
### Accessing Indexed Data
|
||||
|
||||
Benchmarked on MacBook Pro M3 Pro (36GB RAM):
|
||||
- **Full sync to ~892k blocks**: 7-8 hours
|
||||
```rust
|
||||
// Access the underlying storage structures
|
||||
let vecs = &indexer.vecs;
|
||||
let stores = &indexer.stores;
|
||||
|
||||
// Get block hash at specific height
|
||||
let block_hash = vecs.height_to_blockhash.get(Height::new(800_000))?;
|
||||
|
||||
// Look up transaction by prefix
|
||||
let tx_prefix = TxidPrefix::from(&txid);
|
||||
let tx_index = stores.txidprefix_to_txindex.get(&tx_prefix)?;
|
||||
|
||||
// Get address data
|
||||
let address_hash = AddressBytesHash::from(&address_bytes);
|
||||
let type_index = stores.addressbyteshash_to_anyaddressindex.get(&address_hash)?;
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
**Benchmarked on MacBook Pro M3 Pro (36GB RAM):**
|
||||
- **Full blockchain sync** (to ~892k blocks): 7-8 hours
|
||||
- **Peak memory usage**: 5-6GB
|
||||
- **Storage overhead**: ~27% of Bitcoin Core `/blocks` size (193GB as of 2025/08)
|
||||
- **Incremental updates**: Resumes from last height efficiently
|
||||
- **Storage overhead**: ~27% of Bitcoin Core block size
|
||||
- **Incremental updates**: Very fast, efficient resume from last height
|
||||
|
||||
## Data Organization
|
||||
|
||||
The indexer creates the following storage structure:
|
||||
The indexer creates this storage structure:
|
||||
```
|
||||
brk_data/
|
||||
├── indexed/
|
||||
│ ├── vecs/ # Vector storage for time-series data
|
||||
│ └── stores/ # Key-value stores for lookups
|
||||
└── ...
|
||||
│ ├── vecs/ # Vector storage
|
||||
│ │ ├── height_to_* # Height-indexed data
|
||||
│ │ ├── txindex_to_* # Transaction-indexed data
|
||||
│ │ └── outputindex_to_* # Output-indexed data
|
||||
│ └── stores/ # Key-value stores
|
||||
│ ├── hash_lookups/ # Block/TX hash mappings
|
||||
│ └── address_maps/ # Address type mappings
|
||||
└── metadata/ # Versioning and state
|
||||
```
|
||||
|
||||
## Indexes Tracking
|
||||
|
||||
The indexer maintains current indices during processing:
|
||||
|
||||
```rust
|
||||
pub struct Indexes {
|
||||
pub height: Height, // Current block height
|
||||
pub txindex: TxIndex, // Current transaction index
|
||||
pub inputindex: InputIndex, // Current input index
|
||||
pub outputindex: OutputIndex, // Current output index
|
||||
pub p2pkhaddressindex: P2PKHAddressIndex, // P2PKH address index
|
||||
// ... indices for all address types
|
||||
}
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- Running Bitcoin Core node with RPC access
|
||||
- Access to Bitcoin Core's block files
|
||||
- Minimum 500GB free storage space
|
||||
- 8GB+ RAM recommended for optimal performance
|
||||
- **Bitcoin Core node** with RPC enabled
|
||||
- **Block file access** to `~/.bitcoin/blocks/`
|
||||
- **Storage space**: Minimum 500GB (scales with blockchain growth)
|
||||
- **Memory**: 8GB+ RAM recommended
|
||||
- **CPU**: Multi-core recommended for parallel processing
|
||||
|
||||
## Incremental Indexing
|
||||
## Rollback and Recovery
|
||||
|
||||
The indexer supports continuous operation:
|
||||
- Automatically detects last indexed height
|
||||
- Processes new blocks as they arrive
|
||||
- Handles blockchain reorganizations
|
||||
- Provides graceful shutdown with Ctrl+C
|
||||
- **Automatic rollback** on interruption or blockchain reorgs
|
||||
- **State persistence** for efficient restart
|
||||
- **Version management** for storage format compatibility
|
||||
- **Graceful shutdown** with Ctrl+C handling
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `brk_parser` - Bitcoin block parsing and sequential access
|
||||
- `brk_store` - Key-value storage wrapper (fjall-based)
|
||||
- `vecdb` - Vector database for time-series storage
|
||||
- `bitcoin` - Bitcoin protocol types and parsing
|
||||
- `rayon` - Parallel processing framework
|
||||
- `bitcoincore_rpc` - Bitcoin Core RPC client
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
@@ -1,81 +1,227 @@
|
||||
# brk_interface
|
||||
|
||||
Data query and formatting interface that provides a unified API for accessing Bitcoin datasets from both the indexer and computer components with flexible output formats and pagination support. This crate serves as the primary data access layer for BRK's web API and MCP endpoints.
|
||||
**Unified data query and formatting interface for Bitcoin datasets**
|
||||
|
||||
## Features
|
||||
`brk_interface` provides a clean, unified API for accessing Bitcoin datasets from both indexer and computer components. It serves as the primary data access layer powering BRK's web API and MCP endpoints, offering flexible querying, pagination, and multiple output formats.
|
||||
|
||||
- **Unified data access**: Query indexed blockchain data and computed analytics
|
||||
- **Multiple output formats**: JSON, CSV, TSV, and Markdown table formatting
|
||||
- **Flexible pagination**: Range queries with positive/negative indexing support
|
||||
- **Multi-dataset queries**: Retrieve multiple datasets with the same index
|
||||
- **Dynamic search**: Find datasets by ID with automatic fallbacks
|
||||
- **Schema generation**: JSON Schema support for API documentation
|
||||
## What it provides
|
||||
|
||||
## Query Parameters
|
||||
- **Unified Data Access**: Single interface to query both indexed blockchain data and computed analytics
|
||||
- **Multiple Output Formats**: JSON, CSV, TSV, and Markdown table formatting
|
||||
- **Flexible Pagination**: Range queries with positive/negative indexing and automatic pagination
|
||||
- **Multi-dataset Queries**: Retrieve multiple datasets with the same time index in one call
|
||||
- **Dynamic Search**: Intelligent ID matching with automatic fallbacks and normalization
|
||||
|
||||
### Core Parameters
|
||||
## Key Features
|
||||
|
||||
- **index**: Time frame for data retrieval (height, date, week, month, etc.)
|
||||
- **ids**: Dataset identifiers (comma or space separated)
|
||||
- **from**: Starting index (negative values count from end)
|
||||
- **to**: Ending index (optional, exclusive)
|
||||
- **count**: Maximum number of results to return
|
||||
- **format**: Output format (json, csv, tsv, md)
|
||||
### Query Interface
|
||||
- **25 Time Indices**: From granular (Height, DateIndex) to aggregate (YearIndex, DecadeIndex)
|
||||
- **Bitcoin-Specific Indices**: Address types (P2PKH, P2SH, P2TR), output types, epochs
|
||||
- **Multi-dataset support**: Query multiple related datasets simultaneously
|
||||
- **Intelligent ID resolution**: Flexible matching with automatic fallbacks
|
||||
|
||||
### Pagination and Ranges
|
||||
- **Signed indexing**: Negative indices count from end (`-1` = latest, `-10` = last 10)
|
||||
- **Range queries**: `from`/`to` parameters with optional `count` limit
|
||||
- **Efficient pagination**: 1,000 items per page with proper start/end calculation
|
||||
|
||||
### Output Formatting
|
||||
- **JSON**: Single values, arrays, or matrices with automatic structure detection
|
||||
- **CSV/TSV**: Delimiter-separated values with headers for spreadsheet use
|
||||
- **Markdown**: Tables formatted for documentation and display
|
||||
- **Schema Support**: JSON Schema generation for API documentation
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Query Setup
|
||||
|
||||
```rust
|
||||
use brk_interface::{Interface, Params, ParamsOpt, Index, Format};
|
||||
use brk_indexer::Indexer;
|
||||
use brk_computer::Computer;
|
||||
use std::path::Path;
|
||||
|
||||
fn main() -> brk_error::Result<()> {
|
||||
let outputs_dir = Path::new("./brk_data");
|
||||
// Load data sources
|
||||
let indexer = Indexer::forced_import("./brk_data")?;
|
||||
let computer = Computer::forced_import("./brk_data", &indexer, None)?;
|
||||
|
||||
// Load indexer and computer
|
||||
let indexer = Indexer::forced_import(outputs_dir)?;
|
||||
let computer = Computer::forced_import(outputs_dir, &indexer, None)?;
|
||||
// Create unified interface
|
||||
let interface = Interface::build(&indexer, &computer);
|
||||
```
|
||||
|
||||
// Create interface
|
||||
let interface = Interface::build(&indexer, &computer);
|
||||
### Single Dataset Queries
|
||||
|
||||
// Query latest block data
|
||||
let params = Params {
|
||||
index: Index::Height,
|
||||
ids: vec!["date", "timestamp"].into(),
|
||||
rest: ParamsOpt::default()
|
||||
.set_from(-1) // Latest block
|
||||
.set_format(Format::JSON),
|
||||
};
|
||||
```rust
|
||||
// Get latest block data
|
||||
let params = Params {
|
||||
index: Index::Height,
|
||||
ids: vec!["date", "timestamp", "difficulty"].into(),
|
||||
rest: ParamsOpt::default()
|
||||
.set_from(-1) // Latest block
|
||||
.set_format(Format::JSON),
|
||||
};
|
||||
|
||||
let result = interface.search_and_format(params)?;
|
||||
println!("{}", result);
|
||||
let result = interface.search_and_format(params)?;
|
||||
println!("{}", result);
|
||||
```
|
||||
|
||||
// Query price data for last 10 blocks
|
||||
let params = Params {
|
||||
index: Index::Height,
|
||||
ids: vec!["price_usd"].into(),
|
||||
rest: ParamsOpt::default()
|
||||
.set_from(-10)
|
||||
.set_count(10)
|
||||
.set_format(Format::CSV),
|
||||
};
|
||||
### Range Queries
|
||||
|
||||
let result = interface.search_and_format(params)?;
|
||||
println!("{}", result);
|
||||
```rust
|
||||
// Get price data for last 30 days
|
||||
let params = Params {
|
||||
index: Index::DateIndex,
|
||||
ids: vec!["price_usd", "price_usd_high", "price_usd_low"].into(),
|
||||
rest: ParamsOpt::default()
|
||||
.set_from(-30) // Last 30 days
|
||||
.set_count(30)
|
||||
.set_format(Format::CSV),
|
||||
};
|
||||
|
||||
Ok(())
|
||||
let csv_data = interface.search_and_format(params)?;
|
||||
```
|
||||
|
||||
### Multiple Datasets
|
||||
|
||||
```rust
|
||||
// Get comprehensive block statistics
|
||||
let params = Params {
|
||||
index: Index::Height,
|
||||
ids: vec!["size", "weight", "tx_count", "fee_total"].into(),
|
||||
rest: ParamsOpt::default()
|
||||
.set_from(800_000) // Starting from block 800,000
|
||||
.set_to(800_100) // Up to block 800,100
|
||||
.set_format(Format::TSV),
|
||||
};
|
||||
|
||||
let tsv_data = interface.search_and_format(params)?;
|
||||
```
|
||||
|
||||
### Flexible ID Specification
|
||||
|
||||
```rust
|
||||
// Different ways to specify dataset IDs
|
||||
let params = Params {
|
||||
index: Index::DateIndex,
|
||||
ids: vec!["price_usd,volume_usd,market_cap"].into(), // Comma-separated
|
||||
// OR: ids: vec!["price_usd volume_usd market_cap"].into(), // Space-separated
|
||||
// OR: ids: vec!["price_usd", "volume_usd", "market_cap"].into(), // Array
|
||||
rest: ParamsOpt::default()
|
||||
.set_format(Format::JSON),
|
||||
};
|
||||
```
|
||||
|
||||
### Advanced Queries with Pagination
|
||||
|
||||
```rust
|
||||
// Query with custom pagination
|
||||
let params = Params {
|
||||
index: Index::MonthIndex,
|
||||
ids: vec!["supply_total", "supply_active"].into(),
|
||||
rest: ParamsOpt::default()
|
||||
.set_from(0) // From beginning
|
||||
.set_count(50) // Max 50 results
|
||||
.set_format(Format::Markdown),
|
||||
};
|
||||
|
||||
let markdown_table = interface.search_and_format(params)?;
|
||||
```
|
||||
|
||||
## API Methods
|
||||
|
||||
### Core Query Methods
|
||||
|
||||
```rust
|
||||
// Combined search and format
|
||||
let result = interface.search_and_format(params)?;
|
||||
|
||||
// Separate search and format
|
||||
let vecs = interface.search(params)?;
|
||||
let formatted = interface.format(vecs, params.rest.format.unwrap_or_default())?;
|
||||
|
||||
// Get metadata
|
||||
let current_height = interface.get_height();
|
||||
let available_datasets = interface.get_vecids(None)?; // Paginated
|
||||
let available_indices = interface.get_indexes();
|
||||
```
|
||||
|
||||
### Working with Results
|
||||
|
||||
```rust
|
||||
use brk_interface::{Value, Output};
|
||||
|
||||
// Handle different output types
|
||||
match interface.search_and_format(params)? {
|
||||
Output::Json(Value::Single(val)) => println!("Single value: {}", val),
|
||||
Output::Json(Value::List(arr)) => println!("Array with {} items", arr.len()),
|
||||
Output::Json(Value::Matrix(matrix)) => println!("Matrix: {}x{}", matrix.len(), matrix[0].len()),
|
||||
Output::Delimited(csv_data) => println!("CSV/TSV data:\n{}", csv_data),
|
||||
Output::Markdown(table) => println!("Markdown table:\n{}", table),
|
||||
}
|
||||
```
|
||||
|
||||
## API Integration
|
||||
## Available Indices
|
||||
|
||||
The interface provides methods for different use cases:
|
||||
### Time-based Indices
|
||||
- `Height` - Block height (0, 1, 2, ...)
|
||||
- `DateIndex` - Days since Bitcoin genesis
|
||||
- `WeekIndex`, `MonthIndex`, `QuarterIndex`, `YearIndex`, `DecadeIndex`
|
||||
- `HalvingEpoch`, `DifficultyEpoch` - Bitcoin-specific epochs
|
||||
|
||||
- `search()`: Find datasets matching parameters
|
||||
- `format()`: Format search results into specified output format
|
||||
- `search_and_format()`: Combined search and format operation
|
||||
- `get_indexes()`: List available time indices
|
||||
- `get_vecids()`: List available dataset identifiers
|
||||
- `get_height()`: Get current blockchain height
|
||||
### Bitcoin-specific Indices
|
||||
- Address types: `P2PKHAddressIndex`, `P2SHAddressIndex`, `P2WPKHAddressIndex`, etc.
|
||||
- Output types: `OutputType` classifications
|
||||
- Transaction types: `TxIndex`, `InputIndex`, `OutputIndex`
|
||||
|
||||
## Parameter Reference
|
||||
|
||||
```rust
|
||||
pub struct Params {
|
||||
pub index: Index, // Time dimension for query
|
||||
pub ids: MaybeIds, // Dataset identifiers
|
||||
pub rest: ParamsOpt, // Optional parameters
|
||||
}
|
||||
|
||||
pub struct ParamsOpt {
|
||||
pub from: Option<i64>, // Starting index (negative = from end)
|
||||
pub to: Option<i64>, // Ending index (exclusive)
|
||||
pub count: Option<usize>, // Maximum results
|
||||
pub format: Option<Format>, // Output format
|
||||
}
|
||||
```
|
||||
|
||||
## Output Formats
|
||||
|
||||
- **JSON**: Structured data (single values, arrays, matrices)
|
||||
- **CSV**: Comma-separated values with headers
|
||||
- **TSV**: Tab-separated values with headers
|
||||
- **Markdown**: Formatted tables for documentation
|
||||
|
||||
## Performance Features
|
||||
|
||||
- **Zero-copy operations**: Uses references and lifetimes to avoid cloning
|
||||
- **Memory mapping**: Leverages vecdb's memory-mapped storage
|
||||
- **Lazy evaluation**: Only processes data when formatting is requested
|
||||
- **Efficient pagination**: Smart bounds calculation and range queries
|
||||
|
||||
## Schema Support
|
||||
|
||||
The interface provides JSON Schema support for API documentation:
|
||||
|
||||
```rust
|
||||
use schemars::schema_for;
|
||||
|
||||
let schema = schema_for!(Params);
|
||||
// Use schema for API documentation
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `brk_indexer` - Indexed blockchain data source
|
||||
- `brk_computer` - Computed analytics data source
|
||||
- `vecdb` - Vector database with collection traits
|
||||
- `serde` - Serialization/deserialization support
|
||||
- `schemars` - JSON Schema generation
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
@@ -1,37 +1,192 @@
|
||||
# brk_logger
|
||||
|
||||
Colorful logging utility built on `env_logger` that provides clean, timestamped console output with optional file logging. This crate wraps `env_logger` to display logs from the `log` crate in a readable format with color-coded log levels and configurable filtering to suppress noisy third-party library logs.
|
||||
**Logging utilities with colored console output and file logging**
|
||||
|
||||
## Features
|
||||
`brk_logger` provides a thin wrapper around `env_logger` with BRK-specific defaults, colored console output, and optional file logging. It's designed to provide clear, readable logs for Bitcoin data processing operations.
|
||||
|
||||
- **Colorized output**: Log levels are color-coded (error=red, warn=yellow, info=green, debug=blue, trace=cyan)
|
||||
- **Timestamps**: Each log entry includes a formatted timestamp
|
||||
- **File logging**: Optional file output alongside console logging
|
||||
- **Noise filtering**: Pre-configured to suppress verbose logs from Bitcoin Core RPC and other dependencies
|
||||
- **Environment control**: Respects `RUST_LOG` environment variable for custom filtering
|
||||
## What it provides
|
||||
|
||||
- **Colored Console Output**: Level-based color coding for easy visual parsing
|
||||
- **File Logging**: Optional log file output with automatic rotation
|
||||
- **Sensible Defaults**: Pre-configured log levels for Bitcoin and dependency crates
|
||||
- **Timestamp Formatting**: Human-readable timestamps with system timezone
|
||||
|
||||
## Key Features
|
||||
|
||||
### Console Logging
|
||||
- **Color-coded levels**: Error (red), Warn (yellow), Info (green), Debug (blue), Trace (cyan)
|
||||
- **Formatted timestamps**: `YYYY-MM-DD HH:MM:SS` format with dimmed styling
|
||||
- **Clean output**: Minimal formatting focused on readability
|
||||
|
||||
### File Logging
|
||||
- **Optional file output**: Writes to specified file path
|
||||
- **Automatic cleanup**: Removes existing log file on initialization
|
||||
- **Append mode**: New log entries appended to file
|
||||
- **Plain text format**: No colors in file output for better compatibility
|
||||
|
||||
### Dependency Filtering
|
||||
Pre-configured to suppress noisy logs from common dependencies:
|
||||
- `bitcoin=off` - Bitcoin protocol library
|
||||
- `bitcoincore-rpc=off` - RPC client
|
||||
- `fjall=off` - Key-value store
|
||||
- `lsm_tree=off` - LSM tree implementation
|
||||
- `rolldown=off` - Bundler
|
||||
- `tracing=off` - Tracing framework
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Setup (Console Only)
|
||||
|
||||
```rust
|
||||
use brk_logger;
|
||||
|
||||
// Initialize with console output only
|
||||
brk_logger::init(None)?;
|
||||
|
||||
// Now use standard logging macros
|
||||
log::info!("BRK starting up");
|
||||
log::warn!("Bitcoin Core not fully synced");
|
||||
log::error!("Failed to connect to RPC");
|
||||
```
|
||||
|
||||
### With File Logging
|
||||
|
||||
```rust
|
||||
use std::path::Path;
|
||||
|
||||
// Initialize with both console and file output
|
||||
let log_path = Path::new("~/.brk/brk.log");
|
||||
brk_logger::init(Some(log_path))?;
|
||||
|
||||
log::info!("Logs will appear in console and file");
|
||||
```
|
||||
|
||||
### Environment Variable Control
|
||||
|
||||
```bash
|
||||
# Set log level via environment variable
|
||||
export RUST_LOG=debug
|
||||
export RUST_LOG=info,brk_parser=debug # Override for specific crates
|
||||
|
||||
# Run with custom log level
|
||||
RUST_LOG=trace brk
|
||||
```
|
||||
|
||||
### Using Color Utilities
|
||||
|
||||
The crate re-exports `OwoColorize` for consistent coloring:
|
||||
|
||||
```rust
|
||||
use brk_logger::OwoColorize;
|
||||
|
||||
println!("Success: {}", "Operation completed".green());
|
||||
println!("Warning: {}", "Low disk space".yellow());
|
||||
println!("Error: {}", "Connection failed".red());
|
||||
println!("Info: {}", "Processing block 800000".bright_black());
|
||||
```
|
||||
|
||||
## Log Format
|
||||
|
||||
### Console Output
|
||||
```
|
||||
2024-12-25 10:30:15 - info Starting BRK indexer
|
||||
2024-12-25 10:30:16 - warn Bitcoin Core still syncing (99.8% complete)
|
||||
2024-12-25 10:30:45 - info Indexed block 900000 (1.2M transactions)
|
||||
2024-12-25 10:30:46 - error Connection to RPC failed, retrying...
|
||||
```
|
||||
|
||||
### File Output
|
||||
```
|
||||
2024-12-25 10:30:15 - info Starting BRK indexer
|
||||
2024-12-25 10:30:16 - warn Bitcoin Core still syncing (99.8% complete)
|
||||
2024-12-25 10:30:45 - info Indexed block 900000 (1.2M transactions)
|
||||
2024-12-25 10:30:46 - error Connection to RPC failed, retrying...
|
||||
```
|
||||
|
||||
## Default Log Levels
|
||||
|
||||
The logger uses these default settings:
|
||||
|
||||
- **Default level**: `info` - Shows important operational information
|
||||
- **Suppressed crates**: Dependencies that produce excessive output are set to `off`
|
||||
- **Override capability**: Can be overridden via `RUST_LOG` environment variable
|
||||
|
||||
### Common Log Level Settings
|
||||
|
||||
```bash
|
||||
# Minimal output (errors and warnings only)
|
||||
RUST_LOG=warn
|
||||
|
||||
# Standard output (recommended)
|
||||
RUST_LOG=info
|
||||
|
||||
# Verbose output (for debugging)
|
||||
RUST_LOG=debug
|
||||
|
||||
# Maximum output (for development)
|
||||
RUST_LOG=trace
|
||||
|
||||
# Mixed levels (info by default, debug for specific crates)
|
||||
RUST_LOG=info,brk_indexer=debug,brk_computer=trace
|
||||
```
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### In BRK CLI
|
||||
|
||||
```rust
|
||||
use brk_logger;
|
||||
use log::info;
|
||||
|
||||
fn main() -> std::io::Result<()> {
|
||||
// Initialize with console output only
|
||||
brk_logger::init(None)?;
|
||||
|
||||
// Or initialize with file logging
|
||||
brk_logger::init(Some(std::path::Path::new("app.log")))?;
|
||||
|
||||
info!("Application started");
|
||||
fn main() -> Result<()> {
|
||||
// Initialize logging early in main
|
||||
brk_logger::init(Some(Path::new("~/.brk/brk.log")))?;
|
||||
|
||||
info!("BRK CLI starting");
|
||||
|
||||
// ... rest of application
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
## Default Log Filtering
|
||||
### In Custom Applications
|
||||
|
||||
By default, the following crates are filtered to `off` to reduce noise:
|
||||
- `bitcoin`, `bitcoincore-rpc` - Bitcoin Core libraries
|
||||
- `fjall`, `lsm_tree` - Storage engine logs
|
||||
- `rolldown`, `brk_rolldown` - Bundler logs
|
||||
- `rmcp`, `brk_rmcp` - MCP protocol logs
|
||||
- `tracing` - Tracing framework logs
|
||||
```rust
|
||||
use brk_logger::{self, OwoColorize};
|
||||
use log::{info, warn, error};
|
||||
|
||||
fn setup_logging() -> std::io::Result<()> {
|
||||
// Console only for development
|
||||
brk_logger::init(None)?;
|
||||
|
||||
info!("Application initialized");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn process_data() {
|
||||
info!("Processing Bitcoin data...");
|
||||
|
||||
// Use color utilities for progress
|
||||
println!("Progress: {}", "50%".green());
|
||||
|
||||
warn!("Large memory usage detected");
|
||||
error!("Critical error: {}", "Database connection lost".red());
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- **Minimal overhead**: Lightweight wrapper around `env_logger`
|
||||
- **Lazy evaluation**: Log messages only formatted when level is enabled
|
||||
- **File I/O**: Asynchronous file writing doesn't block main thread
|
||||
- **Memory usage**: No buffering, logs written immediately
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `env_logger` - Core logging implementation
|
||||
- `owo_colors` - Terminal color support
|
||||
- `jiff` - Modern date/time handling for timestamps
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
@@ -1,34 +1,221 @@
|
||||
# brk_mcp
|
||||
|
||||
Model Context Protocol (MCP) endpoint that provides LLMs with access to Bitcoin Research Kit data and functionality.
|
||||
**Model Context Protocol (MCP) bridge for LLM integration with BRK**
|
||||
|
||||
This crate implements a stateless MCP endpoint that integrates with `brk_server` to expose BRK's Bitcoin blockchain data through a standardized protocol, enabling LLMs like Claude to directly query blockchain metrics, transaction data, market prices, and time-series data.
|
||||
`brk_mcp` provides a Model Context Protocol server that enables Large Language Models (LLMs) to access Bitcoin blockchain data through BRK's interface layer. It implements the MCP specification to expose BRK's analytics capabilities as tools that LLMs can call.
|
||||
|
||||
The stateless design makes it compatible with load balancers by default.
|
||||
## What it provides
|
||||
|
||||
## Tools Available
|
||||
- **MCP Server Implementation**: Standards-compliant Model Context Protocol server
|
||||
- **Bitcoin Data Tools**: LLM-accessible tools for querying blockchain analytics
|
||||
- **Type-Safe Parameters**: Structured tool parameters with validation
|
||||
- **Multi-Format Output**: JSON responses for LLM consumption
|
||||
- **BRK Integration**: Direct access to all indexed and computed datasets
|
||||
|
||||
- `get_index_count` - Count of available data indexes
|
||||
- `get_vecid_count` - Count of available vector identifiers
|
||||
- `get_vec_count` - Total count of all vectors
|
||||
- `get_indexes` - List all available indexes
|
||||
- `get_accepted_indexes` - Index types and their variants
|
||||
- `get_vecids` - Paginated list of vector identifiers
|
||||
- `get_index_to_vecids` - Vectors supporting specific indexes
|
||||
- `get_vecid_to_indexes` - Indexes supported by specific vectors
|
||||
- `get_vecs` - Query vector data with flexible parameters
|
||||
- `get_version` - BRK version information
|
||||
## Available MCP Tools
|
||||
|
||||
## Usage
|
||||
Based on the actual implementation, the following tools are available:
|
||||
|
||||
The MCP server is automatically exposed at `/mcp` when BRK's HTTP server is running with MCP enabled.
|
||||
### Metadata Tools
|
||||
|
||||
### With Claude Desktop
|
||||
#### `get_index_count`
|
||||
Get the count of all existing indexes.
|
||||
|
||||
Add the MCP endpoint to Claude Desktop.
|
||||
**Parameters:** None
|
||||
**Returns:** Number of available time indices
|
||||
|
||||
For example:
|
||||
#### `get_vecid_count`
|
||||
Get the count of all existing vector IDs.
|
||||
|
||||
**Parameters:** None
|
||||
**Returns:** Number of available dataset identifiers
|
||||
|
||||
#### `get_vec_count`
|
||||
Get the count of all existing vectors (sum of supported indexes for each vector ID).
|
||||
|
||||
**Parameters:** None
|
||||
**Returns:** Total number of vector/index combinations
|
||||
|
||||
#### `get_indexes`
|
||||
Get the list of all existing indexes.
|
||||
|
||||
**Parameters:** None
|
||||
**Returns:** Array of available index names (height, date, week, month, etc.)
|
||||
|
||||
#### `get_accepted_indexes`
|
||||
Get an object with all existing indexes as keys and their accepted variants as values.
|
||||
|
||||
**Parameters:** None
|
||||
**Returns:** Object mapping indexes to their accepted variant names
|
||||
|
||||
### Discovery Tools
|
||||
|
||||
#### `get_vecids`
|
||||
Get a paginated list of all existing vector IDs.
|
||||
|
||||
**Parameters:**
|
||||
- `page` (optional number): Page number (default: 0, up to 1,000 results per page)
|
||||
|
||||
**Returns:** Array of dataset identifiers
|
||||
|
||||
#### `get_index_to_vecids`
|
||||
Get a paginated list of all vector IDs which support a given index.
|
||||
|
||||
**Parameters:**
|
||||
- `index` (string): Index name to query
|
||||
- `page` (optional number): Page number (default: 0)
|
||||
|
||||
**Returns:** Array of vector IDs that support the specified index
|
||||
|
||||
#### `get_vecid_to_indexes`
|
||||
Get a list of all indexes supported by a given vector ID.
|
||||
|
||||
**Parameters:**
|
||||
- `id` (string): Vector ID to query
|
||||
|
||||
**Returns:** Array of indexes supported by the vector ID (empty if ID doesn't exist)
|
||||
|
||||
### Data Query Tool
|
||||
|
||||
#### `get_vecs`
|
||||
Get one or multiple vectors depending on given parameters.
|
||||
|
||||
**Parameters:**
|
||||
- `index` (string): Time dimension (height, date, week, month, etc.)
|
||||
- `ids` (string): Dataset identifiers (comma or space separated)
|
||||
- `from` (optional i64): Start index (negative = from end)
|
||||
- `to` (optional i64): End index (exclusive)
|
||||
- `count` (optional usize): Maximum results
|
||||
- `format` (optional string): Output format (json, csv, tsv, md)
|
||||
|
||||
**Response format depends on parameters:**
|
||||
- **Single value**: One vector, one result (e.g., `from=-1`)
|
||||
- **Array**: One vector, multiple results (e.g., `from=-100&count=100`)
|
||||
- **Matrix**: Multiple vectors (always matrix, even for single results)
|
||||
|
||||
### System Tool
|
||||
|
||||
#### `get_version`
|
||||
Get the running version of the Bitcoin Research Kit.
|
||||
|
||||
**Parameters:** None
|
||||
**Returns:** Version string (e.g., "v0.0.88")
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Discovery Workflow
|
||||
|
||||
```
|
||||
https://bitcoinresearchkit.org/mcp
|
||||
1. Call get_indexes to see available time dimensions
|
||||
2. Call get_vecids to see available datasets (paginated)
|
||||
3. Call get_index_to_vecids to find datasets for specific timeframes
|
||||
4. Call get_vecs to query actual data
|
||||
```
|
||||
|
||||
### Basic Data Query
|
||||
|
||||
```json
|
||||
// Get latest Bitcoin price
|
||||
{
|
||||
"tool": "get_vecs",
|
||||
"parameters": {
|
||||
"index": "date",
|
||||
"ids": "close",
|
||||
"from": -1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Multi-Dataset Query
|
||||
|
||||
```json
|
||||
// Get last 30 days of OHLC data
|
||||
{
|
||||
"tool": "get_vecs",
|
||||
"parameters": {
|
||||
"index": "date",
|
||||
"ids": "open,high,low,close",
|
||||
"from": -30,
|
||||
"format": "csv"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Metadata Exploration
|
||||
|
||||
```json
|
||||
// Discover what's available
|
||||
{
|
||||
"tool": "get_accepted_indexes"
|
||||
}
|
||||
|
||||
// Find datasets for weekly analysis
|
||||
{
|
||||
"tool": "get_index_to_vecids",
|
||||
"parameters": {
|
||||
"index": "week"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Server Configuration
|
||||
|
||||
The MCP server is stateless and integrates with BRK's HTTP server:
|
||||
|
||||
```rust
|
||||
// In brk_server routes
|
||||
router.add_mcp_routes(interface, true) // Enable MCP at /mcp endpoint
|
||||
```
|
||||
|
||||
**Server Info:**
|
||||
- Protocol version: Latest MCP specification
|
||||
- Capabilities: Tools enabled
|
||||
- Instructions: Provides context about Bitcoin data access
|
||||
- Stateless mode: Compatible with load balancers
|
||||
|
||||
## Error Handling
|
||||
|
||||
The MCP server provides structured error responses following MCP specification:
|
||||
- Invalid parameters result in proper MCP error responses
|
||||
- Tool failures are handled gracefully
|
||||
- Parameter validation ensures type safety
|
||||
|
||||
## Integration
|
||||
|
||||
### With BRK Server
|
||||
MCP is exposed at the `/mcp` endpoint when enabled:
|
||||
```
|
||||
POST /mcp
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"method": "tools/call",
|
||||
"params": {
|
||||
"name": "get_vecs",
|
||||
"arguments": {
|
||||
"index": "height",
|
||||
"ids": "timestamp,size",
|
||||
"from": -1
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### LLM Instructions
|
||||
The server provides context to LLMs:
|
||||
- Explains Bitcoin data terminology (vectors, indexes, vecids)
|
||||
- Clarifies that vectors/vecs/arrays/datasets are interchangeable terms
|
||||
- Describes indexes as timeframes and vecids as dataset names
|
||||
- Encourages exploration before web browsing
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `brk_interface` - Data access and formatting layer
|
||||
- `brk_rmcp` - Rust MCP implementation
|
||||
- `axum` - HTTP router integration
|
||||
- `log` - Request logging
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
|
||||
@@ -1,87 +1,153 @@
|
||||
# brk_parser
|
||||
|
||||
High-performance Bitcoin block parser that reads raw Bitcoin Core block files (`blkXXXXX.dat`) and provides a sequential iterator over blocks with fork filtering and XOR support. This crate processes the entire Bitcoin blockchain efficiently using parallel processing and maintains state for fast restarts.
|
||||
**High-performance Bitcoin block parser for raw Bitcoin Core block files**
|
||||
|
||||
## Features
|
||||
`brk_parser` provides efficient sequential access to Bitcoin Core's raw block files (`blkXXXXX.dat`), delivering blocks in height order with automatic fork filtering and XOR encryption support. Built for blockchain analysis and indexing applications that need complete Bitcoin data access.
|
||||
|
||||
- **Fast sequential parsing**: Iterates blocks in height order (0, 1, 2, ...)
|
||||
- **Fork filtering**: Uses Bitcoin Core RPC to exclude orphaned blocks
|
||||
- **XOR support**: Handles XOR-encrypted block files automatically
|
||||
- **Parallel processing**: Multi-threaded parsing and decoding for maximum speed
|
||||
- **State caching**: Saves parsing state for faster subsequent runs
|
||||
- **Memory efficient**: ~500MB peak memory usage
|
||||
- **Range queries**: Parse specific height ranges or single blocks
|
||||
## What it provides
|
||||
|
||||
## Requirements
|
||||
- **Sequential block access**: Blocks delivered in height order (0, 1, 2, ...) regardless of physical file storage
|
||||
- **Fork filtering**: Automatically excludes orphaned blocks using Bitcoin Core RPC verification
|
||||
- **XOR encryption support**: Transparently handles XOR-encrypted block files
|
||||
- **High performance**: Multi-threaded parsing with ~500MB peak memory usage
|
||||
- **State persistence**: Caches parsing state for fast restarts
|
||||
|
||||
- Running Bitcoin Core node with RPC enabled
|
||||
- Access to Bitcoin Core's `blocks/` directory containing `blkXXXXX.dat` files
|
||||
- Bitcoin Core versions v25.0 through v29.0 supported
|
||||
## Key Features
|
||||
|
||||
### Performance Optimization
|
||||
- **Multi-threaded pipeline**: 3-stage processing (file reading, decoding, ordering)
|
||||
- **Parallel decoding**: Uses rayon for concurrent block deserialization
|
||||
- **Memory efficient**: Bounded channels prevent memory bloat
|
||||
- **State caching**: Saves parsing state to avoid re-scanning unchanged files
|
||||
|
||||
### Bitcoin Integration
|
||||
- **RPC verification**: Uses Bitcoin Core RPC to filter orphaned blocks
|
||||
- **Confirmation checks**: Only processes blocks with positive confirmations
|
||||
- **Height ordering**: Ensures sequential delivery regardless of storage order
|
||||
|
||||
### XOR Encryption Support
|
||||
- **Transparent decryption**: Automatically handles XOR-encrypted block files
|
||||
- **Streaming processing**: Applies XOR decryption on-the-fly during parsing
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Block Parsing
|
||||
|
||||
```rust
|
||||
use brk_parser::Parser;
|
||||
use brk_structs::Height;
|
||||
use bitcoincore_rpc::{Auth, Client};
|
||||
use std::path::Path;
|
||||
|
||||
fn main() -> bitcoincore_rpc::Result<()> {
|
||||
// Setup Bitcoin Core RPC client
|
||||
let rpc = Box::leak(Box::new(Client::new(
|
||||
"http://localhost:8332",
|
||||
Auth::CookieFile(Path::new("~/.bitcoin/.cookie")),
|
||||
)?));
|
||||
// Setup RPC client (must have static lifetime)
|
||||
let rpc = Box::leak(Box::new(Client::new(
|
||||
"http://localhost:8332",
|
||||
Auth::CookieFile(Path::new("~/.bitcoin/.cookie")),
|
||||
)?));
|
||||
|
||||
// Create parser
|
||||
let parser = Parser::new(
|
||||
Path::new("~/.bitcoin/blocks").to_path_buf(),
|
||||
Path::new("./brk_data").to_path_buf(), // Output directory
|
||||
rpc,
|
||||
);
|
||||
// Create parser
|
||||
let parser = Parser::new(
|
||||
Path::new("~/.bitcoin/blocks").to_path_buf(),
|
||||
Path::new("./output").to_path_buf(),
|
||||
rpc,
|
||||
);
|
||||
|
||||
// Parse all blocks
|
||||
// Parse all blocks sequentially
|
||||
parser.parse(None, None)
|
||||
.iter()
|
||||
.for_each(|(height, block, hash)| {
|
||||
println!("Block {}: {} ({} txs)", height, hash, block.txdata.len());
|
||||
});
|
||||
```
|
||||
|
||||
### Range Parsing
|
||||
|
||||
```rust
|
||||
// Parse specific height range
|
||||
let start = Some(Height::new(800_000));
|
||||
let end = Some(Height::new(800_100));
|
||||
|
||||
parser.parse(start, end)
|
||||
.iter()
|
||||
.for_each(|(height, block, hash)| {
|
||||
// Process blocks 800,000 to 800,100
|
||||
});
|
||||
```
|
||||
|
||||
### Single Block Access
|
||||
|
||||
```rust
|
||||
// Get single block by height
|
||||
let genesis = parser.get(Height::new(0));
|
||||
println!("Genesis has {} transactions", genesis.txdata.len());
|
||||
```
|
||||
|
||||
### Real-world Usage Example
|
||||
|
||||
```rust
|
||||
use brk_parser::Parser;
|
||||
use bitcoin::Block;
|
||||
|
||||
fn analyze_blockchain(parser: &Parser) {
|
||||
let mut total_transactions = 0;
|
||||
let mut total_outputs = 0;
|
||||
|
||||
parser.parse(None, None)
|
||||
.iter()
|
||||
.for_each(|(height, block, hash)| {
|
||||
println!("Block {}: {} ({} transactions)",
|
||||
height, hash, block.txdata.len());
|
||||
.for_each(|(height, block, _hash)| {
|
||||
total_transactions += block.txdata.len();
|
||||
total_outputs += block.txdata.iter()
|
||||
.map(|tx| tx.output.len())
|
||||
.sum::<usize>();
|
||||
|
||||
if height.0 % 10000 == 0 {
|
||||
println!("Processed {} blocks", height);
|
||||
}
|
||||
});
|
||||
|
||||
// Parse specific range
|
||||
let start = Some(Height::new(800_000));
|
||||
let end = Some(Height::new(800_100));
|
||||
|
||||
parser.parse(start, end)
|
||||
.iter()
|
||||
.for_each(|(height, block, hash)| {
|
||||
println!("Block {}: {}", height, hash);
|
||||
});
|
||||
|
||||
// Get single block
|
||||
let block = parser.get(Height::new(0)); // Genesis block
|
||||
println!("Genesis block has {} transactions", block.txdata.len());
|
||||
|
||||
Ok(())
|
||||
|
||||
println!("Total transactions: {}", total_transactions);
|
||||
println!("Total outputs: {}", total_outputs);
|
||||
}
|
||||
```
|
||||
|
||||
## Output Format
|
||||
|
||||
The parser returns tuples containing:
|
||||
- `Height`: Block height (0, 1, 2, ...)
|
||||
- `Block`: Complete block data (from `bitcoin` crate)
|
||||
The parser returns tuples for each block:
|
||||
- `Height`: Block height (sequential: 0, 1, 2, ...)
|
||||
- `Block`: Complete block data from the `bitcoin` crate
|
||||
- `BlockHash`: Block's cryptographic hash
|
||||
|
||||
## Performance
|
||||
## Performance Characteristics
|
||||
|
||||
Benchmarked on MacBook Pro M3 Pro:
|
||||
- Full blockchain (0 to 855,000): **4 minutes 10 seconds**
|
||||
- Recent blocks (800,000 to 855,000): **52 seconds** (4m 10s on first run)
|
||||
- Peak memory usage: ~500MB
|
||||
- **Full blockchain** (0 to 855,000): ~4 minutes
|
||||
- **Recent blocks** (800,000 to 855,000): ~52 seconds
|
||||
- **Peak memory usage**: ~500MB
|
||||
- **Restart performance**: Subsequent runs much faster due to state caching
|
||||
|
||||
## Requirements
|
||||
|
||||
- Running Bitcoin Core node with RPC enabled
|
||||
- Access to Bitcoin Core's `blocks/` directory
|
||||
- Bitcoin Core versions v25.0 through v29.0 supported
|
||||
- RPC authentication (cookie file or username/password)
|
||||
|
||||
## State Management
|
||||
|
||||
The parser saves state in `{output_dir}/blk_index_to_blk_recap.json` for faster restarts. This file tracks block file indices and heights to avoid re-scanning unchanged files.
|
||||
The parser saves parsing state in `{output_dir}/blk_index_to_blk_recap.json` containing:
|
||||
- Block file indices and maximum heights
|
||||
- File modification times for change detection
|
||||
- Restart optimization metadata
|
||||
|
||||
**Note**: Only one parser instance should run at a time as the state file doesn't yet support concurrent access.
|
||||
**Note**: Only one parser instance should run at a time as the state file doesn't support concurrent access.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `bitcoin` - Bitcoin protocol types and block parsing
|
||||
- `bitcoincore_rpc` - RPC communication with Bitcoin Core
|
||||
- `crossbeam` - Multi-producer, multi-consumer channels
|
||||
- `rayon` - Data parallelism for block decoding
|
||||
- `serde` - State serialization and persistence
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
@@ -1,109 +1,216 @@
|
||||
# brk_server
|
||||
|
||||
HTTP server providing REST API access to Bitcoin analytics data. Serves computed datasets from brk_indexer and brk_computer with multiple output formats, caching, and optional web interfaces.
|
||||
**HTTP server providing REST API access to Bitcoin analytics data**
|
||||
|
||||
## Overview
|
||||
`brk_server` provides a high-performance HTTP server that exposes BRK's indexed blockchain data and computed analytics through a comprehensive REST API. It offers multiple output formats, intelligent caching, compression, and optional web interface serving.
|
||||
|
||||
**Core Features**:
|
||||
- **REST API**: Vector-based data access with pagination and format options
|
||||
- **Multiple Formats**: JSON, CSV, TSV, Markdown output
|
||||
- **HTTP Caching**: ETag-based caching with compression (Brotli, Gzip, Zstd, Deflate)
|
||||
- **Weight Limits**: Request size protection (max 320,000 weight units)
|
||||
- **Web Interface**: Optional website serving
|
||||
## What it provides
|
||||
|
||||
**Port**: Auto-assigns starting from 3110
|
||||
- **REST API**: Vector-based data access with flexible querying and pagination
|
||||
- **Multiple Output Formats**: JSON, CSV, TSV, and Markdown table formatting
|
||||
- **Performance Features**: ETag caching, compression, and request weight limiting
|
||||
- **Web Interface**: Optional static file serving for web applications
|
||||
- **MCP Integration**: Model Context Protocol support for LLM integration
|
||||
|
||||
## Key Features
|
||||
|
||||
### API Capabilities
|
||||
- **Direct vector access**: Single vector queries with index-to-ID pattern
|
||||
- **Multi-vector queries**: Query multiple datasets simultaneously
|
||||
- **Flexible pagination**: Positive/negative indexing with range queries
|
||||
- **Format negotiation**: Multiple output formats based on use case
|
||||
|
||||
### Performance Features
|
||||
- **ETag caching**: Conditional requests with 304 Not Modified responses
|
||||
- **Compression**: Brotli, Gzip, Zstd, Deflate support with automatic negotiation
|
||||
- **Request weight limiting**: Protects against oversized queries (max 320,000 weight)
|
||||
- **In-memory caching**: 10,000-item cache with 50ms guard timeout
|
||||
|
||||
### HTTP Features
|
||||
- **Auto port assignment**: Starts from port 3110, increments if busy
|
||||
- **CORS enabled**: Cross-origin requests for web clients
|
||||
- **Tracing middleware**: Colored request/response logging
|
||||
- **Static file serving**: Optional website hosting
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Server Setup
|
||||
|
||||
```rust
|
||||
use brk_server::Server;
|
||||
use brk_interface::Interface;
|
||||
use brk_indexer::Indexer;
|
||||
use brk_computer::Computer;
|
||||
|
||||
// Load data sources
|
||||
let indexer = Indexer::forced_import("./brk_data")?;
|
||||
let computer = Computer::forced_import("./brk_data", &indexer, None)?;
|
||||
|
||||
// Create interface and server
|
||||
let interface = Interface::build(&indexer, &computer);
|
||||
let server = Server::new(interface, Some("./website".into()));
|
||||
|
||||
// Start server (with MCP support)
|
||||
server.serve(true).await?;
|
||||
```
|
||||
|
||||
### API Access Patterns
|
||||
|
||||
#### Single Vector Queries
|
||||
|
||||
```bash
|
||||
# Latest 100 price values
|
||||
curl "http://brekit.org/api/vecs/date-to-close?from=-100"
|
||||
|
||||
# First 50 difficulty values as CSV
|
||||
curl "http://brekit.org/api/vecs/height-to-difficulty?count=50&format=csv"
|
||||
|
||||
# Range from block 800,000 to 800,100
|
||||
curl "https://brekit.org/api/vecs/height-to-timestamp?from=800000&to=800100"
|
||||
```
|
||||
|
||||
#### Multi-Vector Queries
|
||||
|
||||
```bash
|
||||
# Multiple price metrics for last 30 days
|
||||
curl "http://brekit.org/api/vecs/query?index=date&ids=open,high,low,close&from=-30&format=csv"
|
||||
|
||||
# Block statistics for specific range
|
||||
curl "https://brekit.org/api/vecs/query?index=height&ids=size,weight,tx_count,fee_sum&from=800000&count=100"
|
||||
|
||||
# Weekly analytics as JSON matrix
|
||||
curl "https://brekit.org/api/vecs/query?index=week&ids=close,difficulty&from=-52"
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### Vector Metadata
|
||||
### Vector Metadata Endpoints
|
||||
|
||||
| Endpoint | Description |
|
||||
|----------|-------------|
|
||||
| `GET /api/vecs/index-count` | Total number of available indexes |
|
||||
| `GET /api/vecs/id-count` | Total number of vector IDs |
|
||||
| `GET /api/vecs/vec-count` | Total number of vectors (sum of all index/ID combinations) |
|
||||
| `GET /api/vecs/vec-count` | Total vectors (all index/ID combinations) |
|
||||
| `GET /api/vecs/indexes` | List of all available indexes |
|
||||
| `GET /api/vecs/accepted-indexes` | Object mapping indexes to their accepted variants |
|
||||
| `GET /api/vecs/ids?page=N` | Paginated list of vector IDs (1000 per page, default page=0) |
|
||||
| `GET /api/vecs/index-to-ids?index=INDEX&page=N` | Vector IDs supporting given index |
|
||||
| `GET /api/vecs/id-to-indexes?id=ID` | Indexes supported by given vector ID |
|
||||
| `GET /api/vecs/accepted-indexes` | Mapping of indexes to accepted variants |
|
||||
| `GET /api/vecs/ids?page=N` | Paginated vector IDs (1000/page) |
|
||||
| `GET /api/vecs/index-to-ids?index=INDEX&page=N` | IDs supporting given index |
|
||||
| `GET /api/vecs/id-to-indexes?id=ID` | Indexes supported by given ID |
|
||||
|
||||
### Vector Data Access
|
||||
|
||||
#### Direct Access Pattern: `GET /api/vecs/{INDEX}-to-{ID}`
|
||||
#### Direct Access Pattern
|
||||
`GET /api/vecs/{INDEX}-to-{ID}`
|
||||
|
||||
Access single vector with index-to-id pattern (dashes replaced with underscores internally).
|
||||
|
||||
**Query Parameters:**
|
||||
- `from` (i64, optional): Inclusive start index. Negative values count from end (default: 0)
|
||||
- `to` (i64, optional): Exclusive end index. Negative values count from end. Overrides `count`
|
||||
- `count` (usize, optional): Number of values to retrieve
|
||||
- `format` (string, optional): Output format - `json`, `csv`, `tsv`, `md` (default: `json`)
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Latest 100 price closes
|
||||
curl /api/vecs/date-to-close?from=-100
|
||||
|
||||
# First 50 values as CSV
|
||||
curl /api/vecs/height-to-difficulty?count=50&format=csv
|
||||
|
||||
# Range from index 1000 to 1999
|
||||
curl /api/vecs/date-to-volume?from=1000&to=2000
|
||||
# Examples
|
||||
curl "/api/vecs/height-to-timestamp"
|
||||
curl "/api/vecs/date-to-close"
|
||||
curl "/api/vecs/month-to-supply"
|
||||
```
|
||||
|
||||
#### Multi-Vector Query: `GET /api/vecs/query`
|
||||
|
||||
Query multiple vectors simultaneously with flexible output formats.
|
||||
#### Multi-Vector Query
|
||||
`GET /api/vecs/query`
|
||||
|
||||
**Required Parameters:**
|
||||
- `index`: Vector index type
|
||||
- `index`: Vector index type (height, date, week, month, etc.)
|
||||
- `ids`: Comma or space-separated vector IDs
|
||||
|
||||
**Optional Parameters:**
|
||||
- `from`, `to`, `count`, `format`: Same as direct access
|
||||
- `from` (i64): Start index (negative = from end, default: 0)
|
||||
- `to` (i64): End index (exclusive, negative = from end)
|
||||
- `count` (usize): Maximum number of results
|
||||
- `format`: Output format (`json`, `csv`, `tsv`, `md`)
|
||||
|
||||
**Response Types:**
|
||||
- **Single value**: One vector, one result (e.g., `from=-1`)
|
||||
- **Array**: One vector, multiple results
|
||||
- **Matrix**: Multiple vectors (always matrix, even for single results)
|
||||
### Response Types
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Single vector, latest value
|
||||
curl '/api/vecs/query?index=date&ids=close&from=-1'
|
||||
**Single Value** (one vector, one result):
|
||||
```json
|
||||
42.5
|
||||
```
|
||||
|
||||
# Multiple vectors, date range
|
||||
curl '/api/vecs/query?index=date&ids=open,high,low,close&from=-30&format=csv'
|
||||
**Array** (one vector, multiple results):
|
||||
```json
|
||||
[42.5, 43.1, 44.2]
|
||||
```
|
||||
|
||||
# Complex multi-vector query
|
||||
curl '/api/vecs/query?index=week&ids=price-usd,volume,difficulty&count=52'
|
||||
**Matrix** (multiple vectors):
|
||||
```json
|
||||
[
|
||||
[42.5, 43.1, 44.2],
|
||||
[1500, 1520, 1480],
|
||||
[0.05, 0.052, 0.048]
|
||||
]
|
||||
```
|
||||
|
||||
**CSV Format**:
|
||||
```csv
|
||||
index,close,supply,feerate
|
||||
0,42.5,1500,0.05
|
||||
1,43.1,1520,0.052
|
||||
2,44.2,1480,0.048
|
||||
```
|
||||
|
||||
### System Endpoints
|
||||
|
||||
| Endpoint | Description |
|
||||
|----------|-------------|
|
||||
| `GET /version` | Server version (JSON string) |
|
||||
| `GET /version` | Server version information |
|
||||
| `GET /health` | Health check with timestamp |
|
||||
| `GET /api` | Redirects to this documentation |
|
||||
| `GET /*` | Static file serving (when website enabled) |
|
||||
| `GET /api` | API documentation redirect |
|
||||
| `GET /*` | Static file serving (if enabled) |
|
||||
|
||||
### HTTP Features
|
||||
## Configuration
|
||||
|
||||
**Caching:**
|
||||
- ETag-based conditional requests (304 Not Modified)
|
||||
- `Cache-Control: must-revalidate` headers
|
||||
- In-memory cache with 50ms guard timeout
|
||||
### Server State
|
||||
|
||||
**Compression:**
|
||||
- Brotli, Gzip, Zstd, Deflate support
|
||||
- Automatic content encoding negotiation
|
||||
```rust
|
||||
pub struct AppState {
|
||||
interface: &'static Interface<'static>, // Data access interface
|
||||
path: Option<PathBuf>, // Static files path
|
||||
cache: Arc<Cache<String, Bytes>>, // Response cache
|
||||
}
|
||||
```
|
||||
|
||||
**CORS:**
|
||||
- Cross-origin requests enabled
|
||||
- Appropriate headers for web client access
|
||||
### Middleware Stack
|
||||
|
||||
**Rate Limiting:**
|
||||
- Request weight system (max 320,000 units)
|
||||
- Weight calculated from data range size
|
||||
- **Compression Layer**: Brotli, Gzip, Zstd, Deflate
|
||||
- **Response URI Layer**: Adds URI to response extensions for logging
|
||||
- **Trace Layer**: Request/response logging with colored output
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
- **ETag generation**: Based on data content and query parameters
|
||||
- **Conditional requests**: 304 Not Modified for unchanged data
|
||||
- **In-memory cache**: 10,000 items with LRU eviction
|
||||
- **Cache headers**: `Cache-Control: must-revalidate`
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Request Weight System
|
||||
- **Weight calculation**: Based on data range size and complexity
|
||||
- **Weight limit**: 320,000 units per request
|
||||
- **Protection**: Prevents oversized queries that could impact performance
|
||||
|
||||
### Compression
|
||||
- **Automatic negotiation**: Based on `Accept-Encoding` header
|
||||
- **Multiple algorithms**: Brotli (best), Zstd, Gzip, Deflate
|
||||
- **Transparent**: Applied automatically to all responses
|
||||
|
||||
### Logging
|
||||
- **Colored output**: Green for 200, red for errors, gray for redirects
|
||||
- **Request timing**: Latency measurement and display
|
||||
- **Status tracking**: HTTP status codes with appropriate colors
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `axum` - High-performance async web framework
|
||||
- `tower-http` - HTTP middleware (compression, tracing, CORS)
|
||||
- `brk_interface` - Data access and formatting layer
|
||||
- `brk_mcp` - Model Context Protocol routes
|
||||
- `quick_cache` - Fast in-memory caching
|
||||
- `tokio` - Async runtime for networking
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
|
||||
@@ -1,79 +1,147 @@
|
||||
# brk_store
|
||||
|
||||
Thin wrapper around the Fjall embedded key-value store that provides typed, transactional storage for Bitcoin blockchain data with height-based versioning and batch operations. This crate adds BRK-specific functionality like height tracking, metadata management, and optimized configurations for Bitcoin data storage patterns.
|
||||
**Blockchain-aware key-value storage wrapper for Bitcoin data**
|
||||
|
||||
## Features
|
||||
`brk_store` is a thin, typed wrapper around the [Fjall](https://github.com/fjall-rs/fjall) embedded key-value store, specifically designed for Bitcoin blockchain data storage. It provides height-based versioning, batch operations, and optimized configurations for Bitcoin data patterns.
|
||||
|
||||
- **Typed interface**: Generic over key and value types with automatic serialization
|
||||
- **Height tracking**: Built-in blockchain height awareness for data versioning
|
||||
- **Batch operations**: Efficient batch inserts and deletes with transaction support
|
||||
- **Metadata management**: Automatic version and height metadata storage
|
||||
- **Performance optimized**: Configured write buffers and memtable sizes for Bitcoin data
|
||||
- **Bloom filters**: Configurable bloom filters for faster key lookups
|
||||
- **Reset capability**: Clean store reset for reindexing operations
|
||||
## What it provides
|
||||
|
||||
- **Blockchain-aware storage**: Built-in height tracking and versioning for Bitcoin block processing
|
||||
- **Type-safe interface**: Generic over key and value types with automatic serialization
|
||||
- **Batch operations**: Efficient batched inserts/deletes with transactional support
|
||||
- **Metadata management**: Automatic version checking and height progression tracking
|
||||
- **Performance optimization**: Pre-configured for Bitcoin data patterns
|
||||
|
||||
## Key Features
|
||||
|
||||
### Height-Based Processing
|
||||
- **Height tracking**: Each store tracks the last processed blockchain height
|
||||
- **Conditional operations**: Insert only when blockchain height requires it
|
||||
- **Automatic versioning**: Combines Fjall version + user version for compatibility
|
||||
|
||||
### Batch Operations
|
||||
- **In-memory staging**: Changes staged before atomic commit
|
||||
- **Transactional safety**: All changes written atomically or none at all
|
||||
- **Efficient processing**: Minimizes disk I/O through batching
|
||||
|
||||
### Performance Optimization
|
||||
- **Optimized configurations**: 32MB write buffers, 8MB memtables for Bitcoin data
|
||||
- **Configurable bloom filters**: Optional for faster key lookups
|
||||
- **Memory efficiency**: Reused read transactions, minimal overhead
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Operations
|
||||
|
||||
```rust
|
||||
use brk_store::{Store, open_keyspace};
|
||||
use brk_structs::{Height, Version};
|
||||
use std::path::Path;
|
||||
|
||||
fn main() -> brk_error::Result<()> {
|
||||
// Open the keyspace
|
||||
let keyspace = open_keyspace(Path::new("./data"))?;
|
||||
// Open keyspace
|
||||
let keyspace = open_keyspace(Path::new("./data"))?;
|
||||
|
||||
// Create a typed store
|
||||
let mut store: Store<String, u64> = Store::import(
|
||||
&keyspace,
|
||||
Path::new("./data"),
|
||||
"my_store",
|
||||
Version::ZERO,
|
||||
Some(true), // Enable bloom filters
|
||||
)?;
|
||||
// Create typed store
|
||||
let mut store: Store<String, u64> = Store::import(
|
||||
&keyspace,
|
||||
Path::new("./data"),
|
||||
"store_name",
|
||||
Version::ZERO,
|
||||
Some(true), // Enable bloom filters
|
||||
)?;
|
||||
|
||||
// Insert data if needed at this height
|
||||
store.insert_if_needed(
|
||||
"key1".to_string(),
|
||||
42u64,
|
||||
Height::new(800_000)
|
||||
);
|
||||
// Conditional insertion based on height
|
||||
store.insert_if_needed("key1".to_string(), 42u64, Height::new(800_000));
|
||||
|
||||
// Commit changes
|
||||
store.commit(Height::new(800_000))?;
|
||||
// Batch commit changes
|
||||
store.commit(Height::new(800_000))?;
|
||||
|
||||
// Persist to disk
|
||||
store.persist()?;
|
||||
// Persist to disk
|
||||
store.persist()?;
|
||||
```
|
||||
|
||||
// Query data
|
||||
if let Some(value) = store.get(&"key1".to_string())? {
|
||||
println!("Value: {}", value);
|
||||
}
|
||||
### Querying Data
|
||||
|
||||
Ok(())
|
||||
```rust
|
||||
// Get value (checks pending puts first, then disk)
|
||||
if let Some(value) = store.get(&"key1".to_string())? {
|
||||
println!("Value: {}", value);
|
||||
}
|
||||
|
||||
// Check if store needs processing at height
|
||||
if store.needs(Height::new(800_000)) {
|
||||
// Process this height
|
||||
}
|
||||
```
|
||||
|
||||
## Store Lifecycle
|
||||
|
||||
- **Import**: Create or open existing store with version checking
|
||||
- **Insert**: Add key-value pairs with height-based conditional insertion
|
||||
- **Commit**: Write batched changes to disk atomically
|
||||
- **Persist**: Force sync all data to storage
|
||||
- **Reset**: Clear all data for reindexing if needed
|
||||
|
||||
## AnyStore Trait
|
||||
|
||||
The `AnyStore` trait provides a type-erased interface for managing multiple stores:
|
||||
### Managing Multiple Stores
|
||||
|
||||
```rust
|
||||
use brk_store::AnyStore;
|
||||
|
||||
fn process_store(store: &mut dyn AnyStore) -> brk_error::Result<()> {
|
||||
if store.needs(Height::new(800_000)) {
|
||||
// Process this height
|
||||
store.commit(Height::new(800_000))?;
|
||||
fn process_stores(stores: &mut [Box<dyn AnyStore>]) -> brk_error::Result<()> {
|
||||
let height = Height::new(800_000);
|
||||
|
||||
for store in stores {
|
||||
if store.needs(height) {
|
||||
// Process data for this store
|
||||
store.commit(height)?;
|
||||
}
|
||||
}
|
||||
|
||||
// Persist all stores
|
||||
for store in stores {
|
||||
store.persist()?;
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
### Store Reset and Reindexing
|
||||
|
||||
```rust
|
||||
// Reset store for complete reindexing
|
||||
store.reset()?;
|
||||
|
||||
// Store automatically recreated on next import
|
||||
let mut fresh_store: Store<String, u64> = Store::import(
|
||||
&keyspace,
|
||||
path,
|
||||
"store_name",
|
||||
Version::ZERO,
|
||||
Some(true)
|
||||
)?;
|
||||
```
|
||||
|
||||
## Store Lifecycle
|
||||
|
||||
1. **Import**: Create or open existing store with version checking
|
||||
2. **Insert**: Add key-value pairs with height-based conditional insertion
|
||||
3. **Commit**: Write batched changes to disk atomically
|
||||
4. **Persist**: Force sync all data to storage
|
||||
5. **Reset**: Clear all data for reindexing if needed
|
||||
|
||||
## Type Requirements
|
||||
|
||||
Keys and values must implement:
|
||||
- `Debug + Clone + From<ByteView> + Ord` for keys
|
||||
- `Debug + Clone + From<ByteView>` for values
|
||||
|
||||
This enables automatic serialization and type-safe storage operations.
|
||||
|
||||
## Version Management
|
||||
|
||||
- **Automatic migration**: Detects version mismatches and resets stores automatically
|
||||
- **File-based metadata**: Stores version and height info for persistence
|
||||
- **Compatibility checking**: Prevents data corruption from version incompatibilities
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `fjall` - Embedded key-value store engine
|
||||
- `byteview` - Zero-copy byte view operations
|
||||
- `brk_structs` - Bitcoin-aware type system
|
||||
- `brk_error` - Unified error handling
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
@@ -1,93 +1,105 @@
|
||||
# brk_structs
|
||||
|
||||
Core data structures and types used throughout the Bitcoin Research Kit that provide efficient, zero-copy serializable representations of Bitcoin blockchain data. This crate defines strongly-typed wrappers around primitive types with specialized functionality for Bitcoin analysis, storage optimization, and data grouping operations.
|
||||
**Bitcoin-aware type system and data structures for blockchain analysis**
|
||||
|
||||
## Core Types
|
||||
`brk_structs` provides the foundational type system for the Bitcoin Research Kit (BRK). It offers strongly-typed, storage-optimized data structures that encode Bitcoin protocol knowledge and enable efficient blockchain data analysis.
|
||||
|
||||
### Blockchain Data
|
||||
- **Height**: Block height with arithmetic operations
|
||||
- **Date**, **Timestamp**: Time representations with Bitcoin epoch awareness
|
||||
- **Txid**, **BlockHash**: Transaction and block identifiers with prefix variants
|
||||
- **TxIndex**, **InputIndex**, **OutputIndex**: Transaction component indices
|
||||
## What it provides
|
||||
|
||||
### Value Types
|
||||
- **Sats**: Satoshi amounts with conversion utilities
|
||||
- **Bitcoin**: BTC amounts with precision handling
|
||||
- **Dollars**, **Cents**: Fiat currency representations
|
||||
- **OHLC**: Open/High/Low/Close price data structures
|
||||
- **Type Safety**: Distinct wrapper types prevent common mistakes (e.g., confusing block height with transaction index)
|
||||
- **Storage Optimization**: Zero-copy serialization and optional compression for efficient disk storage
|
||||
- **Bitcoin Protocol Knowledge**: Built-in understanding of halving epochs, address types, and blockchain constants
|
||||
- **Comprehensive Time Indexing**: Multiple temporal granularities for time-series analysis
|
||||
- **Flexible Grouping**: Framework for categorizing and filtering blockchain data
|
||||
|
||||
### Address Types
|
||||
- **AddressBytes**: Raw address data with type information
|
||||
- **P2PKH**, **P2SH**, **P2WPKH**, **P2WSH**, **P2TR**: Address type indices
|
||||
- **AnyAddressIndex**: Unified address index type
|
||||
## Key Features
|
||||
|
||||
### Storage Types
|
||||
- **StoredU8/U16/U32/U64**: Optimized integer storage
|
||||
- **StoredF32/F64**: Floating-point storage with compression
|
||||
- **StoredBool**: Compact boolean storage
|
||||
### Core Blockchain Types
|
||||
- `Height` - Block heights with Bitcoin-specific arithmetic
|
||||
- `Txid`/`BlockHash` - Transaction and block identifiers with prefix optimization
|
||||
- `TxIndex`/`InputIndex`/`OutputIndex` - Component indices with type safety
|
||||
- `Date`/`Timestamp` - Time representations anchored to Bitcoin genesis block
|
||||
|
||||
### Time Indices
|
||||
- **DateIndex**, **WeekIndex**, **MonthIndex**, **QuarterIndex**
|
||||
- **SemesterIndex**, **YearIndex**, **DecadeIndex**: Time-based grouping
|
||||
- **HalvingEpoch**, **DifficultyEpoch**: Bitcoin-specific time periods
|
||||
### Value and Currency Types
|
||||
- `Sats` - Satoshi amounts with comprehensive arithmetic operations
|
||||
- `Bitcoin` - BTC amounts with precision handling
|
||||
- `Dollars`/`Cents` - Fiat currency for price analysis
|
||||
- OHLC types (`Open<T>`, `High<T>`, `Low<T>`, `Close<T>`) for market data
|
||||
|
||||
## Grouping Operations
|
||||
### Address System
|
||||
- `OutputType` - All Bitcoin script types (P2PKH, P2SH, P2WPKH, P2WSH, P2TR, etc.)
|
||||
- `AddressBytes` - Type-safe address data extraction
|
||||
- Address-specific indices for each output type
|
||||
- `AnyAddressIndex` - Unified address indexing
|
||||
|
||||
The crate provides powerful grouping and filtering capabilities:
|
||||
|
||||
```rust
|
||||
use brk_structs::*;
|
||||
|
||||
// Group by address type
|
||||
let p2pkh_addresses = ByAddressType::P2PKH;
|
||||
|
||||
// Group by value ranges
|
||||
let small_utxos = ByLtAmount::new(Sats::_1BTC);
|
||||
let large_utxos = ByGeAmount::new(Sats::_10BTC);
|
||||
|
||||
// Group by age ranges
|
||||
let recent_utxos = ByMaxAge::new(Height::new(144)); // Last day
|
||||
let old_utxos = ByMinAge::new(Height::new(52560)); // Last year
|
||||
|
||||
// Combine filters
|
||||
let filter = Filter::new()
|
||||
.by_spendable_type()
|
||||
.by_amount_range(Sats::_1K, Sats::_1M)
|
||||
.by_address_type(ByAddressType::P2WPKH);
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **Zero-copy serialization**: All types implement `FromBytes`/`IntoBytes` for efficient storage
|
||||
- **Storage compression**: Built-in compression support via `StoredCompressed` trait
|
||||
- **Type safety**: Strongly-typed wrappers prevent value confusion
|
||||
- **Bitcoin-aware**: Constants and operations specific to Bitcoin protocol
|
||||
- **Efficient grouping**: Flexible data filtering and categorization system
|
||||
- **Time handling**: Comprehensive time representation with epoch support
|
||||
### Temporal Indexing
|
||||
- `DateIndex` - Days since Bitcoin genesis
|
||||
- Time granularities: `WeekIndex`, `MonthIndex`, `QuarterIndex`, `YearIndex`, `DecadeIndex`
|
||||
- `HalvingEpoch` - Bitcoin halving periods (every 210,000 blocks)
|
||||
- `DifficultyEpoch` - Difficulty adjustment periods
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Types and Conversions
|
||||
|
||||
```rust
|
||||
use brk_structs::*;
|
||||
|
||||
// Create blockchain data types
|
||||
// Type-safe blockchain data
|
||||
let height = Height::new(800_000);
|
||||
let amount = Sats::_1BTC;
|
||||
let epoch = HalvingEpoch::from(height);
|
||||
let amount = Sats::_1BTC * 2;
|
||||
|
||||
// Time-based indexing
|
||||
let date = Date::new(2024, 1, 15);
|
||||
|
||||
// Work with addresses
|
||||
let address_data = AddressBytes::from_script(&script);
|
||||
let address_index = P2WPKHAddressIndex::from_address_bytes(&address_data);
|
||||
|
||||
// Price data
|
||||
let ohlc = OHLCCents::new(
|
||||
Open::new(45000_00),
|
||||
High::new(46000_00),
|
||||
Low::new(44000_00),
|
||||
Close::new(45500_00)
|
||||
);
|
||||
|
||||
// Time-based analysis
|
||||
let month_index = MonthIndex::from(date);
|
||||
let halving_epoch = HalvingEpoch::from(height);
|
||||
let month_idx = MonthIndex::from(date);
|
||||
```
|
||||
|
||||
### Address Handling
|
||||
|
||||
```rust
|
||||
// Extract address information from Bitcoin scripts
|
||||
let output_type = OutputType::from(&script);
|
||||
if output_type.is_address() {
|
||||
let address_bytes = AddressBytes::try_from((&script, output_type))?;
|
||||
}
|
||||
```
|
||||
|
||||
### Zero-Copy Storage
|
||||
|
||||
```rust
|
||||
// Efficient serialization without copying
|
||||
let height_bytes = height.as_bytes();
|
||||
let recovered_height = Height::read_from_bytes(height_bytes)?;
|
||||
```
|
||||
|
||||
### Data Grouping and Filtering
|
||||
|
||||
```rust
|
||||
// Flexible filtering for analytics
|
||||
let utxo_groups = UTXOGroups {
|
||||
all: total_utxos,
|
||||
age_range: ByAgeRange::new(age_filtered_utxos),
|
||||
epoch: ByEpoch::new(epoch_filtered_utxos),
|
||||
// ... more groupings
|
||||
};
|
||||
```
|
||||
|
||||
## Storage Optimization
|
||||
|
||||
All types implement zero-copy serialization traits:
|
||||
- **Zero overhead**: Direct memory mapping without serialization costs
|
||||
- **Optional compression**: Configurable zstd compression for space efficiency
|
||||
- **Type safety**: Compile-time guarantees about data layout and endianness
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `bitcoin` - Bitcoin protocol types and script parsing
|
||||
- `vecdb` - Vector database storage traits
|
||||
- `zerocopy` - Zero-copy serialization framework
|
||||
- `serde` - JSON serialization support
|
||||
- `jiff` - Modern date/time handling
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
Reference in New Issue
Block a user