Mirror/brk
1
0
Fork 0
mirror of https://github.com/bitcoinresearchkit/brk.git synced 2026-06-08 14:11:56 -07:00
Code Issues Packages Projects Releases Wiki Activity

readmes: regenerated

Browse Source
This commit is contained in:
nym21
2025-09-16 22:01:55 +02:00
parent 185fc7b6ed
commit 301dee96dc
19 changed files with 3128 additions and 2065 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/prompts/readmes.md
+36 -5
View File
@@ -3,18 +3,26 @@
Generate a professional, comprehensive README.md for each crate based SOLELY on code analysis. Use NO external documentation, commit messages, or existing READMEs.
## MANDATORY PROCESS - FOLLOW EXACTLY:
1. Analyze each crate's source code thoroughly using file system exploration
2. **MANDATORY CODE ANALYSIS**: Before writing ANY README content, you MUST:
1. **IGNORE EXISTING DOCS**: Do NOT read any .md, .txt, .rst, or documentation files in the crate directory
2. **CODE-ONLY ANALYSIS**: Examine ONLY these files:
- All .rs files in src/ directory and subdirectories
- Cargo.toml for dependencies and metadata
- Code structure and organization
3. **MANDATORY CODE ANALYSIS**: Before writing ANY README content, you MUST:
- Examine all Rust files in src/ directory
- Identify the main structs, enums, traits, and functions
- Understand the crate's architecture and data flow
- Determine the crate's purpose from its implementation
- Map dependencies to understand external integrations
3. Generate one complete README.md per crate
4. Focus on one crate at a time for thorough analysis
4. **FRESH PERSPECTIVE**: Write the README as if you're the first person to document this crate
5. Generate one complete README.md per crate
6. Focus on one crate at a time for thorough analysis
## ABSOLUTE REQUIREMENTS:
- **SOURCE OF TRUTH**: Use ONLY the actual Rust code - no external docs, comments may provide hints but focus on implementation
- **CRITICAL**: DO NOT read any existing README.md, CHANGELOG.md, or documentation files
- **IGNORE ALL TEXT FILES**: .md, .txt, .rst files are FORBIDDEN sources - treat them as if they don't exist
- **CODE ONLY**: Focus exclusively on .rs files, Cargo.toml, and code structure
- **PROFESSIONAL GRADE**: Write as if this will be published on crates.io for other developers
- **PROGRAMMER FOCUSED**: Assume audience knows Rust and relevant domain concepts
- **IMPLEMENTATION-BASED**: Describe what the code actually does, not what comments claim it should do
@@ -74,6 +82,9 @@ crate_name = "X.Y.Z"
- **NEVER** write marketing copy: "cutting-edge", "state-of-the-art", "enterprise-grade"
- **NEVER** make claims you can't verify from code: "blazingly fast", "memory efficient"
- **NEVER** copy-paste from existing documentation or comments
- **NEVER** read or reference existing README.md files - pretend they don't exist
- **NEVER** use phrases like "as mentioned in the documentation" or "according to the docs"
- **NEVER** let existing documentation influence your analysis or writing
### REQUIRED SPECIFICITY:
- **Data structures**: Mention specific types (HashMap, Vec, etc.)
@@ -82,7 +93,27 @@ crate_name = "X.Y.Z"
- **Error handling**: How errors are represented and handled
- **Async/sync**: Clearly state if operations are blocking or async
### CODE ANALYSIS DEPTH:
### ANTI-BIAS PROTOCOL:
### BEFORE STARTING ANY ANALYSIS:
1. **Explicitly ignore**: Any README.md, CHANGELOG.md, docs/, documentation files
2. **File filtering**: Only examine .rs and Cargo.toml files
3. **Fresh eyes approach**: Analyze the code as if you've never seen this crate before
4. **Independent thinking**: Form your own understanding purely from code inspection
### IF YOU ACCIDENTALLY READ EXISTING DOCS:
- Stop immediately and restart your analysis
- Consciously disregard any information from documentation files
- Base all descriptions solely on what you observe in the code
- Ask yourself: "What would I think this code does if I had no documentation?"
### VALIDATION CHECKS:
- **Unique descriptions**: Your descriptions should differ significantly from any existing docs
- **Code-derived insights**: Every feature mentioned must be visible in the source code
- **Independent voice**: Write in your own technical style, not mimicking existing documentation
- **Fresh examples**: Create new code examples based on API analysis, not existing samples
## CODE ANALYSIS DEPTH:
**You MUST analyze and understand:**
1. **Public API surface**: All pub structs, functions, traits, modules
2. **Core abstractions**: Main data types and their relationships
crates/brk/README.md
+223 -147
View File
@@ -1,197 +1,273 @@
# brk
**Main wrapper crate for the Bitcoin Research Kit (BRK)**
Unified Bitcoin Research Kit crate providing optional feature-gated access to all BRK components.
The `brk` crate serves as the primary entry point for the Bitcoin Research Kit, providing a unified interface to all BRK components through feature flags. It enables developers to selectively include only the components they need while maintaining a consistent API.
[![Crates.io](https://img.shields.io/crates/v/brk.svg)](https://crates.io/crates/brk)
[![Documentation](https://docs.rs/brk/badge.svg)](https://docs.rs/brk)
## What it provides
## Overview
- **Unified Access**: Single crate providing access to the entire BRK ecosystem
- **Feature-based Selection**: Choose only the components you need
- **Consistent Versioning**: All components versioned together for compatibility
- **Simplified Dependencies**: Single dependency instead of multiple individual crates
This crate serves as a unified entry point to the Bitcoin Research Kit ecosystem, providing feature-gated re-exports of all BRK components. It allows users to selectively include only the functionality they need while maintaining a single dependency declaration, with the `brk_cli` component always available for command-line interface access.
## Available Components
**Key Features:**
### Core Data Pipeline
- **`parser`** ([brk_parser](../brk_parser/)) - High-performance Bitcoin block parser
- **`indexer`** ([brk_indexer](../brk_indexer/)) - Blockchain data indexer with dual storage
- **`computer`** ([brk_computer](../brk_computer/)) - Analytics engine for computed datasets
- **`interface`** ([brk_interface](../brk_interface/)) - Unified data query and formatting API
- Feature-gated modular access to 12 specialized BRK components
- Single dependency entry point with selective compilation
- Always-available CLI component for command-line operations
- Comprehensive documentation aggregation with inline re-exports
- `full` feature for complete BRK functionality inclusion
- Optimized build configuration with docs.rs integration
### Infrastructure
- **`structs`** ([brk_structs](../brk_structs/)) - Bitcoin-aware type system and data structures
- **`error`** ([brk_error](../brk_error/)) - Centralized error handling
- **`store`** ([brk_store](../brk_store/)) - Blockchain-aware key-value storage
**Target Use Cases:**
### External Integration
- **`fetcher`** ([brk_fetcher](../brk_fetcher/)) - Bitcoin price data fetcher with multi-source fallback
- **`server`** ([brk_server](../brk_server/)) - HTTP server with REST API
- **`mcp`** ([brk_mcp](../brk_mcp/)) - Model Context Protocol for LLM integration
- Applications requiring selective BRK functionality to minimize dependencies
- Library development where only specific Bitcoin analysis components are needed
- Prototyping and experimentation with different BRK component combinations
- Educational use cases demonstrating modular blockchain analytics architecture
### Utilities
- **`cli`** ([brk_cli](../brk_cli/)) - Command line interface (always enabled)
- **`logger`** ([brk_logger](../brk_logger/)) - Logging utilities
- **`bundler`** ([brk_bundler](../brk_bundler/)) - Asset bundling for web interfaces
## Usage
### Full Installation
## Installation
```toml
[dependencies]
brk = { version = "0.0.88", features = ["full"] }
# Minimal installation with CLI only
brk = "0.0.107"
# Full functionality
brk = { version = "0.0.107", features = ["full"] }
# Selective features
brk = { version = "0.0.107", features = ["indexer", "computer", "server"] }
```
## Quick Start
```rust
use brk::*;
// CLI is always available
use brk::cli;
// Access all components
let config = cli::Config::load()?;
let parser = parser::Parser::new(/* ... */);
let indexer = indexer::Indexer::forced_import("./data")?;
let computer = computer::Computer::forced_import("./data", &indexer, None)?;
// Feature-gated components
#[cfg(feature = "indexer")]
use brk::indexer::Indexer;
#[cfg(feature = "computer")]
use brk::computer::Computer;
#[cfg(feature = "server")]
use brk::server::Server;
// Build complete pipeline with selected features
#[cfg(all(feature = "indexer", feature = "computer", feature = "server"))]
fn build_pipeline() -> Result<(), Box<dyn std::error::Error>> {
let indexer = Indexer::build("./data")?;
let computer = Computer::build("./analytics", &indexer)?;
let interface = brk::interface::Interface::build(&indexer, &computer);
let server = Server::new(interface, None);
Ok(())
}
```
### Selective Components
## API Overview
```toml
[dependencies]
brk = { version = "0.0.88", features = ["parser", "indexer", "computer"] }
```
### Feature Organization
The crate provides feature-gated access to BRK components organized by functionality:
**Core Data Processing:**
- `structs` - Bitcoin-aware data structures and type system
- `error` - Centralized error handling across components
- `store` - Transactional key-value storage wrapper
**Blockchain Processing:**
- `parser` - Multi-threaded Bitcoin block parsing
- `indexer` - Blockchain data indexing with columnar storage
- `computer` - Analytics computation engine
**Data Access:**
- `interface` - Unified query interface with fuzzy search
- `fetcher` - Multi-source price data aggregation
**Service Layer:**
- `server` - HTTP API server with caching and compression
- `mcp` - Model Context Protocol bridge for LLM integration
- `logger` - Enhanced logging with colored output
**Web Infrastructure:**
- `bundler` - Web asset bundling using Rolldown
### Always Available
**`cli`**: Command-line interface module (no feature gate required)
Provides access to the complete BRK command-line interface for running full instances.
## Examples
### Minimal Bitcoin Parser
```rust
use brk::{parser, indexer, computer};
use brk::parser::Parser;
// Core data pipeline only
let parser = parser::Parser::new(blocks_dir, Some(output_dir), rpc);
let mut indexer = indexer::Indexer::forced_import(output_dir)?;
let mut computer = computer::Computer::forced_import(output_dir, &indexer, None)?;
fn parse_blocks() -> Result<(), Box<dyn std::error::Error>> {
#[cfg(feature = "parser")]
{
let parser = Parser::new("/path/to/blocks", None, rpc_client);
// Parse blockchain data
Ok(())
}
#[cfg(not(feature = "parser"))]
{
Err("Parser feature not enabled".into())
}
}
```
### Minimal Setup
```toml
[dependencies]
brk = { version = "0.0.88", features = ["structs", "parser"] }
```
```rust
use brk::{structs, parser};
// Just parsing and types
let height = structs::Height::new(800_000);
let parser = parser::Parser::new(blocks_dir, Some(output_dir), rpc);
```
## Feature Flags
| Feature | Description | Dependencies |
|---------|-------------|--------------|
| `full` | Enable all components | All crates |
| `cli` | Command line interface | Always enabled |
| `structs` | Core type system | Foundation for other crates |
| `error` | Error handling | Used by most crates |
| `parser` | Block parsing | `structs`, `error` |
| `store` | Key-value storage | `structs`, `error` |
| `indexer` | Blockchain indexing | `parser`, `store` |
| `computer` | Analytics computation | `indexer`, `fetcher` (optional) |
| `fetcher` | Price data fetching | `structs`, `error` |
| `interface` | Data query API | `indexer`, `computer` |
| `server` | HTTP server | `interface`, `mcp` |
| `mcp` | LLM integration | `interface` |
| `logger` | Logging utilities | Standalone |
| `bundler` | Asset bundling | Standalone |
## Common Usage Patterns
### Complete BRK Instance
```rust
use brk::*;
// Full data pipeline setup
let config = cli::Config::load()?;
let rpc = /* Bitcoin Core RPC client */;
let parser = parser::Parser::new(config.blocks_dir, Some(config.output_dir), rpc);
let mut indexer = indexer::Indexer::forced_import(&config.output_dir)?;
let mut computer = computer::Computer::forced_import(&config.output_dir, &indexer, None)?;
let interface = interface::Interface::build(&indexer, &computer);
let server = server::Server::new(interface, config.website_path);
// Start server
server.serve(true).await?;
```
### Data Analysis
### Analytics Pipeline
```rust
use brk::{indexer, computer, interface};
// Analysis-focused setup
let indexer = indexer::Indexer::forced_import("./brk_data")?;
let computer = computer::Computer::forced_import("./brk_data", &indexer, None)?;
let interface = interface::Interface::build(&indexer, &computer);
#[cfg(all(feature = "indexer", feature = "computer", feature = "interface"))]
fn analytics_pipeline() -> Result<(), Box<dyn std::error::Error>> {
// Initialize indexer
let indexer = indexer::Indexer::build("./blockchain_data")?;
// Query data
let params = interface::Params {
index: interface::Index::Height,
ids: vec!["price_usd", "difficulty"].into(),
rest: interface::ParamsOpt::default()
.set_from(-100)
.set_format(interface::Format::CSV),
};
// Compute analytics
let computer = computer::Computer::build("./analytics", &indexer)?;
let csv_data = interface.search_and_format(params)?;
```
// Create query interface
let interface = interface::Interface::build(&indexer, &computer);
### Custom Integration
// Query latest price
let params = interface::Params {
index: "date".to_string(),
ids: vec!["price-close".to_string()].into(),
from: Some(-1),
..Default::default()
};
```rust
use brk::{structs, parser, error};
// Custom application with BRK components
fn analyze_blocks() -> error::Result<()> {
let parser = parser::Parser::new(blocks_dir, Some(output_dir), rpc);
parser.parse(None, None)
.iter()
.take(1000) // First 1000 blocks
.for_each(|(height, block, hash)| {
println!("Block {}: {} transactions", height, block.txdata.len());
});
let result = interface.search_and_format(params)?;
println!("Latest Bitcoin price: {:?}", result);
Ok(())
}
```
## Version Compatibility
### Web Server Setup
All BRK crates are released together with synchronized versions. When using the `brk` wrapper crate, you're guaranteed compatibility between all components.
```rust
use brk::{server, interface, indexer, computer};
- **Current version**: 0.0.88
- **Rust MSRV**: 1.89+
- **Bitcoin Core**: v25.0 - v29.0
#[cfg(all(feature = "server", feature = "interface", feature = "indexer", feature = "computer"))]
async fn web_server() -> Result<(), Box<dyn std::error::Error>> {
let indexer = indexer::Indexer::build("./data")?;
let computer = computer::Computer::build("./analytics", &indexer)?;
let interface = interface::Interface::build(&indexer, &computer);
## Performance Characteristics
let server = server::Server::new(interface, None);
server.serve(true).await?;
The `brk` crate itself adds no runtime overhead - it simply re-exports the underlying crates. Performance characteristics depend on which components you use:
Ok(())
}
```
- **Full pipeline**: ~13-15 hours initial sync, ~40GB storage overhead
- **Parser only**: ~4 minutes to parse entire blockchain
- **Indexer only**: ~7-8 hours to index blockchain, ~5-6GB RAM usage
- **Server**: Low latency API responses with caching and compression
### MCP Integration
## Dependencies
```rust
use brk::{mcp, interface, indexer, computer};
The `brk` crate's dependencies are determined by enabled features. Core dependencies include:
#[cfg(all(feature = "mcp", feature = "interface"))]
fn mcp_server(interface: &'static interface::Interface) -> mcp::MCP {
mcp::MCP::new(interface)
}
```
- `brk_cli` - Always included for configuration and CLI support
- Individual `brk_*` crates based on enabled features
- Transitive dependencies from enabled components
## Feature Combinations
For specific dependency information, see individual crate READMEs.
### Common Combinations
**Data Processing**: `["structs", "parser", "indexer"]`
Basic blockchain data processing and indexing.
**Analytics**: `["indexer", "computer", "fetcher", "interface"]`
Complete analytics pipeline with price data integration.
**API Server**: `["interface", "server", "logger"]`
HTTP API server with logging capabilities.
**Full Stack**: `["full"]`
All components for complete BRK functionality.
### Dependency Optimization
Feature selection allows for significant dependency reduction:
```toml
# Minimal parser-only dependency
brk = { version = "0.0.107", features = ["parser"] }
# Analytics without web server
brk = { version = "0.0.107", features = ["indexer", "computer", "interface"] }
# Web server without parsing
brk = { version = "0.0.107", features = ["interface", "server"] }
```
## Architecture
### Re-export Pattern
The crate uses `#[doc(inline)]` re-exports to provide seamless access to component APIs:
```rust
#[cfg(feature = "component")]
#[doc(inline)]
pub use brk_component as component;
```
This pattern ensures:
- Feature-gated compilation for dependency optimization
- Inline documentation for unified API reference
- Namespace preservation for component-specific functionality
### Build Configuration
- **Documentation**: `all-features = true` for complete docs.rs documentation
- **CLI Integration**: `brk_cli` always available without feature gates
- **Optional Dependencies**: All components except CLI are optional
## Configuration
### Feature Flags
| Feature | Component | Description |
|---------|-----------|-------------|
| `bundler` | `brk_bundler` | Web asset bundling |
| `computer` | `brk_computer` | Analytics computation |
| `error` | `brk_error` | Error handling |
| `fetcher` | `brk_fetcher` | Price data fetching |
| `indexer` | `brk_indexer` | Blockchain indexing |
| `interface` | `brk_interface` | Data query interface |
| `logger` | `brk_logger` | Enhanced logging |
| `mcp` | `brk_mcp` | Model Context Protocol |
| `parser` | `brk_parser` | Block parsing |
| `server` | `brk_server` | HTTP server |
| `store` | `brk_store` | Key-value storage |
| `structs` | `brk_structs` | Data structures |
| `full` | All components | Complete functionality |
### Documentation
Documentation is aggregated from all components with `#![doc = include_str!("../README.md")]` ensuring comprehensive API reference across all features.
## Code Analysis Summary
**Main Structure**: Feature-gated re-export crate providing unified access to 12 BRK components \
**Feature System**: Cargo features enabling selective compilation and dependency optimization \
**CLI Integration**: Always-available `brk_cli` access without feature requirements \
**Documentation**: Inline re-exports with comprehensive docs.rs integration \
**Dependency Management**: Optional dependencies for all components except CLI \
**Build Configuration**: Optimized compilation with all-features documentation \
**Architecture**: Modular aggregation crate enabling flexible BRK ecosystem usage
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_bundler/README.md
+226 -134
View File
@@ -1,186 +1,278 @@
# brk_bundler
**Asset bundling for BRK web interfaces using Rolldown**
Asset bundling and development server for BRK web interfaces with hot reloading and file watching.
`brk_bundler` provides JavaScript/TypeScript bundling capabilities for BRK's web interfaces. It's a thin wrapper around Rolldown (Rust-based Rollup alternative) with BRK-specific configuration for building optimized web assets with file watching and automatic rebuilding.
[![Crates.io](https://img.shields.io/crates/v/brk_bundler.svg)](https://crates.io/crates/brk_bundler)
[![Documentation](https://docs.rs/brk_bundler/badge.svg)](https://docs.rs/brk_bundler)
## What it provides
## Overview
- **JavaScript Bundling**: Modern ES modules bundling with minification
- **File Watching**: Automatic rebuilding on source file changes
- **Asset Processing**: Copies and processes static assets
- **Version Injection**: Automatic version string replacement in service workers
- **Development Mode**: Live rebuilding for rapid development
This crate provides a thin wrapper around the Rolldown JavaScript bundler specifically designed for BRK web interface development. It handles asset bundling, file copying, template processing, and development-mode file watching with automatic rebuilds and hot reloading for efficient web development workflows.
## Key Features
**Key Features:**
### Bundling Capabilities
- **ES Module Support**: Modern JavaScript bundling with tree-shaking
- **Minification**: Automatic code minification for production builds
- **Source Maps**: Generated source maps for debugging
- **Entry Point Processing**: Configurable entry points with hashed output names
- JavaScript bundling with Rolldown (Rust-based bundler)
- Automatic file watching and hot reloading in development mode
- Template processing with version injection and asset hash replacement
- Service worker generation with version management
- Source map generation for debugging
- Minification for production builds
- Async/await support with Tokio integration
### File System Operations
- **Directory Copying**: Copies entire source directories to distribution
- **Selective Processing**: Special handling for specific file types
- **Path Resolution**: Automatic path resolution and asset linking
**Target Use Cases:**
### Development Features
- **Hot Rebuilding**: Automatic rebuilds on file changes
- **Watch Mode**: Monitors source files and triggers rebuilds
- **Version Replacement**: Injects build version into service workers
- BRK blockchain explorer web interfaces
- Development of Bitcoin analytics dashboards
- Building responsive web applications for blockchain data visualization
- Hot reloading development environment for rapid iteration
## Usage
## Installation
### Basic Bundling
```toml
cargo add brk_bundler
```
## Quick Start
```rust
use brk_bundler::bundle;
use std::path::Path;
// Bundle without watching (production)
let websites_path = Path::new("./websites");
let source_folder = "default";
let dist_path = bundle(websites_path, source_folder, false).await?;
#[tokio::main]
async fn main() -> std::io::Result<()> {
let websites_path = Path::new("./web");
let source_folder = "src";
let watch = true; // Enable hot reloading
println!("Bundled to: {:?}", dist_path);
```
// Bundle assets and start development server
let dist_path = bundle(websites_path, source_folder, watch).await?;
### Development Mode with Watching
println!("Assets bundled to: {}", dist_path.display());
```rust
// Bundle with file watching (development)
let dist_path = bundle(websites_path, "default", true).await?;
// Keep running for file watching (in watch mode)
if watch {
tokio::signal::ctrl_c().await?;
}
// Bundler now watches for changes and rebuilds automatically
// This will run in the background until the process exits
```
### Integration with BRK CLI
```rust
// Typically called from brk_cli when serving websites
async fn setup_website(config: &Config) -> Result<PathBuf> {
let websites_path = config.websites_path();
let source_folder = match config.website_mode {
WebsiteMode::Default => "default",
WebsiteMode::Custom => "custom",
WebsiteMode::None => return Ok(PathBuf::new()),
};
// Bundle the website assets
let dist_path = bundle(websites_path, source_folder, config.dev_mode).await?;
Ok(dist_path)
Ok(())
}
```
## File Structure
## API Overview
The bundler expects this directory structure:
### Core Functions
```
websites/
├── default/ # Default website source
│ ├── index.html # Main HTML file
│ ├── service-worker.js # Service worker (version injected)
│ ├── scripts/ # JavaScript/TypeScript source
│ │ ├── entry.js # Main entry point
│ │ ├── main.js # Application logic
│ │ └── ... # Other JS modules
│ └── assets/ # Static assets
└── dist/ # Generated output directory
├── index.html # Processed HTML with updated script references
├── service-worker.js # Service worker with version injected
├── scripts/ # Bundled and minified JavaScript
│ └── main-[hash].js # Hashed output file
└── assets/ # Copied static assets
**`bundle(websites_path: &Path, source_folder: &str, watch: bool) -> io::Result<PathBuf>`**
Main bundling function that processes web assets and optionally starts file watching.
### Bundling Process
1. **Directory Setup**: Creates `dist/` directory and copies source files
2. **JavaScript Bundling**: Processes `scripts/entry.js` with Rolldown bundler
3. **Template Processing**: Updates `index.html` with hashed asset references
4. **Service Worker**: Generates service worker with version injection
5. **File Watching**: Optionally monitors source files for changes
### Configuration
**Rolldown Bundler Options:**
- **Input**: `./src/scripts/entry.js` (main JavaScript entry point)
- **Output**: `./dist/scripts/` directory
- **Minification**: Enabled for production builds
- **Source Maps**: File-based source maps for debugging
- **Asset Hashing**: Automatic hash generation for cache busting
## Examples
### Development Mode with Hot Reloading
```rust
use brk_bundler::bundle;
use std::path::Path;
#[tokio::main]
async fn main() -> std::io::Result<()> {
let web_root = Path::new("./websites");
// Start development server with file watching
let _dist_path = bundle(web_root, "explorer", true).await?;
println!("Development server started!");
println!("Hot reloading enabled - edit files to see changes");
// Keep server running
loop {
tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
}
}
```
## Bundling Process
### Production Build
1. **Clean**: Removes existing `dist/` directory
2. **Copy**: Copies all source files to `dist/`
3. **Bundle JavaScript**:
- Processes `scripts/entry.js` as entry point
- Generates minified bundle with source maps
- Creates hashed filename for cache busting
4. **Process HTML**: Updates script references to hashed filenames
5. **Process Service Worker**: Injects current version string
6. **Watch** (if enabled): Monitors for file changes and rebuilds
```rust
use brk_bundler::bundle;
use std::path::Path;
## Configuration
#[tokio::main]
async fn main() -> std::io::Result<()> {
let web_root = Path::new("./websites");
The bundler uses Rolldown with these optimized settings:
// Build for production (no watching)
let dist_path = bundle(web_root, "dashboard", false).await?;
println!("Production build completed: {}", dist_path.display());
// Assets are minified and ready for deployment
Ok(())
}
```
### Custom Web Application Structure
```rust
use brk_bundler::bundle;
use std::path::Path;
// Expected directory structure:
// websites/
// ├── my_app/
// │ ├── index.html // Main HTML template
// │ ├── service-worker.js // Service worker template
// │ ├── scripts/
// │ │ └── entry.js // JavaScript entry point
// │ ├── styles/
// │ │ └── main.css // CSS files
// │ └── assets/
// │ └── images/ // Static assets
// └── dist/ // Generated output
#[tokio::main]
async fn main() -> std::io::Result<()> {
let websites_path = Path::new("./websites");
let source_folder = "my_app";
let dist_path = bundle(websites_path, source_folder, false).await?;
// Result: dist/ contains bundled and processed files
// - dist/index.html (with updated script references)
// - dist/service-worker.js (with version injection)
// - dist/scripts/main.[hash].js (minified and hashed)
// - dist/styles/ (copied CSS files)
// - dist/assets/ (copied static assets)
Ok(())
}
```
## Architecture
### File Processing Pipeline
1. **Source Copying**: Recursively copies all source files to dist directory
2. **JavaScript Bundling**: Rolldown processes entry.js with dependencies
3. **Asset Hashing**: Generates content-based hashes for cache busting
4. **Template Updates**: Replaces placeholders in HTML templates
5. **Version Injection**: Updates service worker with current package version
### File Watching System
**Development Mode Watchers:**
- **Source File Watcher**: Monitors non-script files for changes
- **Bundle Watcher**: Watches JavaScript files and triggers rebuilds
- **Template Watcher**: Updates HTML when bundled assets change
**Event Handling:**
- **File Creation/Modification**: Automatic copying to dist directory
- **Script Changes**: Triggers Rolldown rebuild and template update
- **Template Changes**: Processes HTML and updates asset references
### Template Processing
**index.html Processing:**
- Scans bundled JavaScript for asset hash
- Replaces `/scripts/main.js` with `/scripts/main.[hash].js`
- Maintains cache busting while preserving template structure
**service-worker.js Processing:**
- Replaces `__VERSION__` placeholder with current crate version
- Enables version-based cache invalidation
- Maintains service worker functionality
### Async Architecture
Built on Tokio async runtime:
- **Non-blocking I/O**: Efficient file operations and watching
- **Concurrent Tasks**: Parallel file watching and bundle processing
- **Background Processing**: Development server runs in background task
## Configuration Options
### Rolldown Configuration
The bundler uses optimized Rolldown settings:
```rust
BundlerOptions {
input: Some(vec![source_entry.into()]), // scripts/entry.js
dir: Some("./dist/scripts".to_string()), // Output directory
cwd: Some(websites_path), // Working directory
minify: Some(RawMinifyOptions::Bool(true)), // Enable minification
sourcemap: Some(SourceMapType::File), // Generate source maps
..Default::default()
input: Some(vec!["./src/scripts/entry.js".into()]),
dir: Some("./dist/scripts".to_string()),
minify: Some(RawMinifyOptions::Bool(true)),
sourcemap: Some(SourceMapType::File),
// ... other default options
}
```
## File Watching
### File Structure Requirements
In watch mode, the bundler monitors:
**Required Files:**
- **Source files**: Non-script files are copied on change
- **JavaScript files**: Trigger full rebuild via Rolldown watcher
- **HTML files**: Processed to update script references
- **Service worker**: Version injection on changes
- `src/scripts/entry.js` - JavaScript entry point
- `src/index.html` - HTML template
- `src/service-worker.js` - Service worker template
### Watch Events Handled
**Optional Directories:**
- `Create` - New files added
- `Modify` - Existing files changed
- Ignores `Delete` and other events
- `src/styles/` - CSS stylesheets
- `src/assets/` - Static assets (images, fonts, etc.)
- `src/components/` - Additional JavaScript modules
## Version Injection
## Development Workflow
Service workers get automatic version injection:
### Setup
```javascript
// In source service-worker.js
const VERSION = '__VERSION__';
1. Create web application in `websites/app_name/`
2. Add required files (index.html, entry.js, service-worker.js)
3. Run bundler in watch mode for development
// After bundling
const VERSION = 'v0.0.88';
```
### Hot Reloading
This enables proper cache invalidation across releases.
- **Script Changes**: Automatic bundle rebuild and browser refresh
- **Template Changes**: Immediate HTML update with asset hash replacement
- **Asset Changes**: Instant copy to dist directory
- **Style Changes**: Direct copy without bundling
## Performance Features
### Production Deployment
- **Async Operations**: All bundling operations are async
- **Incremental Builds**: Only rebuilds changed files in watch mode
- **Parallel Processing**: Uses Tokio for concurrent file operations
- **Efficient Copying**: Direct file system operations
1. Run bundler without watch mode
2. Deploy `dist/` directory contents
3. Assets include content hashes for cache busting
4. Service worker includes version for cache management
## Error Handling
## Code Analysis Summary
- **Graceful Failures**: Logs errors but continues watching
- **Path Resolution**: Automatic path absolutization and validation
- **File System Errors**: Proper error propagation with context
## Dependencies
- `brk_rolldown` - Rust-based Rollup bundler
- `notify` - File system watching
- `tokio` - Async runtime for file operations
- `sugar_path` - Path manipulation utilities
- `log` - Error logging
## Integration Points
The bundler integrates with:
- **brk_cli**: Called during website setup
- **brk_server**: Serves bundled assets
- **Development workflow**: Provides live rebuilding
**Main Function**: `bundle()` async function coordinating Rolldown bundler with file processing and watching \
**File Operations**: Recursive directory copying with `copy_dir_all()` and selective file processing \
**Templating**: String replacement for asset hash injection and version management \
**File Watching**: Multi-watcher system using `notify` crate for real-time development feedback \
**Async Integration**: Tokio-based async architecture with background task spawning \
**Bundler Integration**: Rolldown wrapper with optimized configuration for web development \
**Architecture**: Development-focused asset pipeline with hot reloading and production optimization
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_cli/README.md
+187 -144
View File
@@ -1,189 +1,232 @@
# brk_cli
**Command line interface for running complete BRK instances**
Command-line interface orchestrating complete Bitcoin Research Kit instances with automatic configuration and continuous blockchain processing.
`brk_cli` provides the main command-line interface for operating the Bitcoin Research Kit. It orchestrates the complete data pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with automatic configuration management and graceful operation.
[![Crates.io](https://img.shields.io/crates/v/brk_cli.svg)](https://crates.io/crates/brk_cli)
[![Documentation](https://docs.rs/brk_cli/badge.svg)](https://docs.rs/brk_cli)
## What it provides
## Overview
- **Complete Pipeline Orchestration**: Coordinates parser, indexer, computer, and server components
- **Automatic Configuration**: Saves settings to `~/.brk/config.toml` for consistent operation
- **Continuous Operation**: Handles blockchain updates and incremental processing
- **Web Interface Options**: Configurable website serving (none, default, custom)
- **Graceful Shutdown**: Ctrl+C handling with proper cleanup
This crate provides the primary command-line interface for running Bitcoin Research Kit instances. It orchestrates the entire data processing pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with persistent configuration management, automatic error recovery, and continuous blockchain synchronization.
## Key Features
**Key Features:**
### Pipeline Management
- **Automatic dependency handling**: Ensures Bitcoin Core sync before processing
- **Incremental updates**: Only processes new blocks since last run
- **Error recovery**: Automatic retry logic and graceful error handling
- **Resource management**: Optimized memory usage and disk I/O
- Complete BRK pipeline orchestration with parser, indexer, computer, and server coordination
- Persistent configuration system with TOML-based auto-save functionality
- Continuous blockchain processing with new block detection and incremental updates
- Flexible Bitcoin Core RPC authentication with cookie file and user/password support
- Configurable web interface options including auto-downloading from GitHub releases
- Large stack allocation (512MB) for handling complex blockchain processing workloads
- Graceful shutdown handling with proper cleanup and state preservation
### Configuration System
- **Auto-save configuration**: All CLI options saved to persistent config
- **Flexible paths**: Configurable Bitcoin directory, blocks directory, and output directory
- **RPC authentication**: Cookie file or username/password authentication
- **Data source options**: Configurable price fetching and exchange APIs
**Target Use Cases:**
### Operation Modes
- **Initial sync**: Full blockchain processing from genesis
- **Continuous operation**: Real-time processing of new blocks
- **Update mode**: Resume from last processed block
- **Server mode**: HTTP API with optional web interface
- Production Bitcoin analytics deployments requiring full pipeline operation
- Development environments for Bitcoin research and analysis
- Continuous blockchain monitoring with real-time data updates
- Academic research requiring comprehensive historical blockchain datasets
## Installation
### Binary Release
```bash
# Download from GitHub releases
# https://github.com/bitcoinresearchkit/brk/releases/latest
cargo install brk # or cargo install brk_cli
```
### Via Cargo
```bash
cargo install brk --locked
```
### From Source
```bash
git clone https://github.com/bitcoinresearchkit/brk.git
cd brk && cargo build --release
```
## Usage
### First Run (Configuration Setup)
## Quick Start
```bash
# Basic setup with default options
brk --brkdir ./my_brk_data
# First run - configure and start processing
brk --brkdir ./data --bitcoindir ~/.bitcoin --fetch true
# Full configuration
brk --bitcoindir ~/.bitcoin \
--brkdir ./brk_data \
--fetch true \
--exchanges true \
--website default
```
### Subsequent Runs
```bash
# Uses saved configuration from ~/.brk/config.toml
# Subsequent runs use saved configuration
brk
# Override specific options
brk --website none --fetch false
```
### Command Line Options
## API Overview
### Core Structure
- **`Config`**: Persistent configuration with clap-based CLI parsing and TOML serialization
- **`Bridge`**: Interface trait for generating JavaScript bridge files for web interfaces
- **`Website`**: Enum for web interface options (None, Bitview, Custom)
- **Path Functions**: Cross-platform default path resolution for Bitcoin and BRK directories
### Main Operations
**`main() -> color_eyre::Result<()>`**
Entry point with error handling setup, directory creation, logging initialization, and high-stack thread spawning.
**`run() -> color_eyre::Result<()>`**
Core processing loop handling configuration, RPC connection, component initialization, and continuous blockchain monitoring.
### Configuration Management
**Persistent Settings:**
- All CLI arguments automatically saved to `~/.brk/config.toml`
- Argument overrides update saved configuration on each run
- Cross-platform path resolution with tilde and $HOME expansion
- Validation of Bitcoin directory, blocks directory, and RPC authentication
**CLI Parameters:**
- `--bitcoindir`, `--blocksdir`, `--brkdir`: Directory configuration
- `--fetch`, `--exchanges`: Data source configuration
- `--website`: Web interface selection
- `--rpcconnect`, `--rpcport`, `--rpccookiefile`, `--rpcuser`, `--rpcpassword`: RPC settings
## Examples
### Basic Usage
```bash
brk --help
# Initialize with custom directories
brk --bitcoindir /data/bitcoin --brkdir /data/brk
# Enable all features with custom RPC
brk --fetch true --exchanges true --website bitview \
--rpcuser myuser --rpcpassword mypass
# Minimal setup with API only
brk --website none --fetch false
```
## Configuration Reference
### Configuration File Example
All options are automatically saved to `~/.brk/config.toml`:
After first run, settings are saved to `~/.brk/config.toml`:
### Core Paths
- `--bitcoindir <PATH>` - Bitcoin Core directory (default: `~/.bitcoin`)
- `--blocksdir <PATH>` - Block files directory (default: `bitcoindir/blocks`)
- `--brkdir <PATH>` - BRK output directory (default: `~/.brk`)
### Data Sources
- `--fetch <BOOL>` - Enable price data fetching (default: `true`)
- `--exchanges <BOOL>` - Use exchange APIs for prices (default: `true`)
### Web Interface
- `--website <OPTION>` - Web interface mode:
- `none` - API only, no web interface
- `default` - Built-in web interface from `websites/default/`
- `custom` - Serve custom website from `websites/custom/`
### Bitcoin Core RPC
- `--rpcconnect <IP>` - RPC host (default: `localhost`)
- `--rpcport <PORT>` - RPC port (default: `8332`)
- `--rpccookiefile <PATH>` - Cookie authentication file
- `--rpcuser <USERNAME>` - Username authentication
- `--rpcpassword <PASSWORD>` - Password authentication
## Operation Flow
1. **Configuration Loading**: Loads saved config from `~/.brk/config.toml`
2. **Bitcoin Core Connection**: Establishes RPC connection and waits for sync
3. **Data Pipeline Initialization**: Sets up parser, indexer, computer, and interface
4. **Processing Loop**:
- Index new blocks from Bitcoin Core
- Compute analytics on new data
- Update cached data
5. **Server Startup**: Launches HTTP API with optional web interface
6. **Continuous Operation**: Monitors for new blocks and processes incrementally
## System Requirements
- **Bitcoin Core**: Fully synced node with RPC enabled
- **Storage**: ~32% of blockchain size (~233GB as of 2025)
- **Memory**:
- Peak: ~7-8GB during initial indexing
- Steady state: ~4-5GB during operation
- **OS**: macOS or Linux
- Ubuntu: `sudo apt install libssl-dev pkg-config`
## Performance Characteristics
### Initial Sync
- **Full blockchain processing**: ~13-15 hours total
- **Parser phase**: ~4 minutes for block parsing
- **Indexer phase**: ~7-8 hours for data indexing
- **Computer phase**: ~6-7 hours for analytics computation
### Continuous Operation
- **New block processing**: 3-5 seconds per block
- **API response times**: Typically <100ms with caching
- **Memory usage**: Stable ~4-5GB during normal operation
## Configuration File
Example `~/.brk/config.toml`:
```toml
bitcoindir = "/Users/username/.bitcoin"
blocksdir = "/Users/username/.bitcoin/blocks"
brkdir = "/Users/username/brk_data"
bitcoindir = "/home/user/.bitcoin"
blocksdir = "/home/user/.bitcoin/blocks"
brkdir = "/home/user/brk_data"
fetch = true
exchanges = true
website = "default"
website = "bitview"
rpcconnect = "localhost"
rpcport = 8332
rpccookiefile = "/Users/username/.bitcoin/.cookie"
rpccookiefile = "/home/user/.bitcoin/.cookie"
```
## Error Handling
### Web Interface Configuration
- **Bitcoin Core sync**: Waits for node sync before processing
- **RPC connection**: Automatic retry logic for connection issues
- **Processing errors**: Graceful error handling with detailed logging
- **Graceful shutdown**: Ctrl+C handling with proper cleanup and state saving
```bash
# Use built-in Bitview interface
brk --website bitview
## Logging
# Use custom web interface
brk --website custom
Logs are written to `~/.brk/brk.log` with colored console output:
- Request/response logging with timing
- Processing progress indicators
- Error reporting and debugging information
# API only, no web interface
brk --website none
```
## Dependencies
### Development Mode
- `brk_parser` - Bitcoin block parsing
- `brk_indexer` - Blockchain data indexing
- `brk_computer` - Analytics computation
- `brk_interface` - Data query interface
- `brk_server` - HTTP API server
- `brk_logger` - Logging utilities
- `bitcoincore_rpc` - Bitcoin Core RPC client
- `color_eyre` - Enhanced error reporting
```bash
# Development with local website directory
# Looks for ../../websites directory first
brk --website bitview
# Production with auto-download from GitHub
# Downloads websites from release artifacts
brk --website bitview
```
## Architecture
### Startup Sequence
1. **Environment Setup**: Color eyre error handling, directory creation, logging initialization
2. **High-Stack Thread**: 512MB stack for complex blockchain processing operations
3. **Configuration Loading**: CLI parsing, TOML reading, argument merging, validation
4. **Component Initialization**: Parser, indexer, computer, interface creation with proper dependencies
### Processing Pipeline
**Continuous Operation Loop:**
1. **Bitcoin Core Sync Wait**: Monitors `headers == blocks` for full node synchronization
2. **Block Count Detection**: Compares current and previous block counts for new block detection
3. **Indexing Phase**: Processes new blocks through parser with collision detection option
4. **Computing Phase**: Runs analytics computations on newly indexed data
5. **Server Operation**: Serves HTTP API with optional web interface throughout processing
### Web Interface Integration
**Website Handling:**
- **Development Mode**: Uses local `../../websites` directory if available
- **Production Mode**: Downloads release artifacts from GitHub using semantic versioning
- **Bundle Generation**: Creates optimized JavaScript bundles using `brk_bundler`
- **Bridge Files**: Generates JavaScript bridge files for vector IDs and pool data
**Download and Bundle Process:**
```rust
// Automatic website download and bundling
let url = format!("https://github.com/bitcoinresearchkit/brk/archive/refs/tags/v{VERSION}.zip");
let response = minreq::get(url).send()?;
zip::ZipArchive::new(cursor).extract(downloads_path)?;
bundle(&websites_path, website.to_folder_name(), true).await?
```
### RPC Authentication
**Flexible Authentication Methods:**
- **Cookie File**: Automatic detection at `--bitcoindir/.cookie`
- **User/Password**: Manual configuration with `--rpcuser` and `--rpcpassword`
- **Connection Validation**: Startup checks ensure proper Bitcoin Core connectivity
### Configuration System
**TOML Persistence:**
- Automatic serialization/deserialization with `serde` and `toml`
- Error-tolerant parsing with `default_on_error` deserializer
- Argument consumption validation ensuring all CLI options are processed
- Path expansion supporting `~` and `$HOME` environment variables
## Configuration
### Default Paths
**Cross-Platform Path Resolution:**
- **Linux**: `~/.bitcoin` for Bitcoin Core, `~/.brk` for BRK data
- **macOS**: `~/Library/Application Support/Bitcoin` for Bitcoin Core
- **Logs**: `~/.brk/log` for application logging
- **Downloads**: `~/.brk/downloads` for temporary website artifacts
### Performance Settings
**Memory Management:**
- 512MB stack size for main processing thread
- Multi-threaded tokio runtime with all features enabled
- Persistent configuration caching to minimize I/O operations
### Error Handling
**Comprehensive Validation:**
- Directory existence checks with user-friendly error messages
- RPC authentication verification before processing begins
- Graceful exit with help suggestions for configuration issues
## Code Analysis Summary
**Main Structure**: `Config` struct with clap-derived CLI parsing and persistent TOML configuration management \
**Processing Loop**: Continuous Bitcoin Core monitoring with sync detection and incremental block processing \
**Web Integration**: Automatic website download from GitHub releases with JavaScript bundle generation \
**Component Orchestration**: Coordination of parser, indexer, computer, and server with proper dependency management \
**Error Handling**: `color_eyre` integration with comprehensive validation and user-friendly error messages \
**Threading**: High-stack thread allocation (512MB) with tokio multi-threaded runtime for complex operations \
**Architecture**: Complete BRK pipeline orchestration with persistent configuration and continuous blockchain synchronization
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_computer/README.md
+249 -146
View File
@@ -1,200 +1,303 @@
# brk_computer
**Bitcoin analytics engine that transforms indexed blockchain data into comprehensive metrics**
Advanced Bitcoin analytics engine that transforms indexed blockchain data into comprehensive metrics and financial analytics.
`brk_computer` is the computational layer of BRK that processes indexed blockchain data to generate analytics across multiple specialized domains. It provides comprehensive Bitcoin metrics with efficient storage and lazy computation for optimal performance.
[![Crates.io](https://img.shields.io/crates/v/brk_computer.svg)](https://crates.io/crates/brk_computer)
[![Documentation](https://docs.rs/brk_computer/badge.svg)](https://docs.rs/brk_computer)
## What it provides
## Overview
- **Comprehensive Analytics**: 9 specialized domains covering all aspects of Bitcoin analysis
- **Lazy Computation**: On-demand calculation with dependency tracking and caching
- **Incremental Updates**: Only processes new data since last computation
- **Memory Efficiency**: ~100MB operation footprint via compressed storage and memory mapping
- **Multi-timeframe Analysis**: Daily, weekly, monthly, quarterly, yearly perspectives
This crate provides a sophisticated analytics engine that processes indexed Bitcoin blockchain data to compute comprehensive metrics, financial analytics, and statistical aggregations. Built on top of `brk_indexer`, it transforms raw blockchain data into actionable insights through state tracking, cohort analysis, market metrics, and advanced Bitcoin-specific calculations.
## Nine Analytics Domains
**Key Features:**
The computer processes data through a fixed dependency chain:
- Comprehensive Bitcoin analytics pipeline with 6 major computation modules
- UTXO and address cohort analysis with lifecycle tracking
- Market metrics integration with price data and financial calculations
- Cointime economics and realized/unrealized profit/loss analysis
- Supply dynamics and monetary policy metrics
- Pool analysis for centralization and mining statistics
- Memory allocation tracking and performance optimization
- Parallel computation with multi-threaded processing
1. **indexes** - Time-based indexing (date/height mappings, epoch calculations)
2. **constants** - Baseline values and reference metrics
3. **blocks** - Block analytics (sizes, intervals, transaction counts, weight)
4. **mining** - Mining economics (hashrate, difficulty, rewards, epochs)
5. **fetched** - External price data integration (optional)
6. **price** - OHLC data across multiple timeframes (optional, requires fetched)
7. **transactions** - Transaction analysis (fees, sizes, patterns, RBF detection)
8. **market** - Price correlations and market metrics (optional, requires price)
9. **stateful** - UTXO tracking and accumulated state computations
10. **cointime** - Coin age and time-based value analysis
**Target Use Cases:**
## Key Features
- Bitcoin market analysis and research platforms
- On-chain analytics for investment and trading decisions
- Academic research requiring comprehensive blockchain metrics
- Financial applications needing Bitcoin exposure and risk metrics
### Computation Strategy
- **Fixed dependency chain**: Ensures data consistency across all domains
- **Parallel processing**: Uses Rayon for performance optimization
- **State management**: Rollback capabilities for error recovery
- **Incremental updates**: Only computes new data since last run
## Installation
### Analytics Capabilities
- **Multi-timeframe analysis**: Daily, weekly, monthly, quarterly, yearly aggregations
- **Chain-based metrics**: Height, difficulty epoch, halving epoch indexing
- **Price correlation**: Both dollar and satoshi denominated metrics
- **DCA analysis**: Dollar Cost Averaging with configurable periods
- **Supply analysis**: Circulating, realized, unrealized supply metrics
- **Address cohort tracking**: Analysis across different Bitcoin address types
- **UTXO cohort analysis**: Realized/unrealized gains tracking
- **Coin time analysis**: Understanding Bitcoin velocity and dormancy
```toml
cargo add brk_computer
```
### Storage Optimization
- **Compressed vectors**: Efficient disk storage with lazy computation
- **Memory mapping**: Minimal RAM usage during operation
- **Version management**: Automatic invalidation on schema changes
- **Dependency tracking**: Smart recomputation based on data changes
## Usage
### Basic Setup (No Price Data)
## Quick Start
```rust
use brk_computer::Computer;
use brk_indexer::Indexer;
use vecdb::Exit;
// Setup without external price data
let indexer = Indexer::forced_import("./brk_data")?;
let mut computer = Computer::forced_import("./brk_data", &indexer, None)?;
// Setup exit handler
let exit = Exit::new();
exit.set_ctrlc_handler();
// Compute all analytics
let starting_indexes = indexer.get_starting_indexes();
computer.compute(&indexer, starting_indexes, &exit)?;
```
### Advanced Setup (With Price Data)
```rust
use brk_fetcher::Fetcher;
use vecdb::Exit;
use std::path::Path;
// Setup with external price data for market analytics
// Initialize dependencies
let outputs_path = Path::new("./analytics_data");
let indexer = Indexer::forced_import(outputs_path)?;
let fetcher = Some(Fetcher::import(true, None)?);
let mut computer = Computer::forced_import("./brk_data", &indexer, fetcher)?;
// Compute all analytics including price/market domains
// Create computer with price data support
let mut computer = Computer::forced_import(outputs_path, &indexer, fetcher)?;
// Compute analytics from indexer state
let exit = Exit::default();
let starting_indexes = brk_indexer::Indexes::default();
computer.compute(&indexer, starting_indexes, &exit)?;
println!("Analytics computation completed!");
```
### Accessing Computed Data
## API Overview
### Core Structure
The Computer is organized into 7 specialized computation modules:
- **`indexes`**: Fundamental blockchain index computations
- **`constants`**: Network constants and protocol parameters
- **`market`**: Price-based financial metrics and market analysis
- **`pools`**: Mining pool analysis and centralization metrics
- **`chain`**: Core blockchain metrics (difficulty, hashrate, fees)
- **`stateful`**: Advanced state tracking (UTXO lifecycles, address behaviors)
- **`cointime`**: Cointime economics and value-time calculations
### Key Methods
**`Computer::forced_import(outputs_path, indexer, fetcher) -> Result<Self>`**
Creates computer instance with optional price data integration.
**`compute(&mut self, indexer: &Indexer, starting_indexes: Indexes, exit: &Exit) -> Result<()>`**
Main computation pipeline processing all analytics modules.
### Analytics Categories
**Market Analytics:**
- Price-based metrics (market cap, realized cap, MVRV)
- Trading volume analysis and liquidity metrics
- Return calculations and volatility measurements
- Dollar-cost averaging and investment strategy metrics
**On-Chain Analytics:**
- Transaction count and size statistics
- Fee analysis and block space utilization
- Address activity and entity clustering
- UTXO age distributions and spending patterns
**Monetary Analytics:**
- Circulating supply and issuance tracking
- Realized vs. unrealized gains/losses
- Cointime destruction and accumulation
- Velocity and economic activity indicators
## Examples
### Basic Analytics Computation
```rust
// Access all computed vectors
let all_vecs = computer.vecs(); // Returns Vec<&dyn AnyCollectableVec>
use brk_computer::Computer;
// Access specific domain data
let block_metrics = &computer.blocks;
let mining_data = &computer.mining;
let transaction_stats = &computer.transactions;
// Initialize with indexer and optional price data
let computer = Computer::forced_import(
"./analytics_output",
&indexer,
Some(price_fetcher)
)?;
// Access price data (if available)
if let Some(price_data) = &computer.price {
// Use OHLC data
}
// Compute all analytics modules
let exit = vecdb::Exit::default();
computer.compute(&indexer, starting_indexes, &exit)?;
// Access computed metrics
println!("Market cap vectors computed: {}", computer.market.len());
println!("Chain metrics computed: {}", computer.chain.len());
println!("Stateful analysis completed: {}", computer.stateful.len());
```
### Incremental Updates
### Market Analysis
```rust
// Continuous computation loop
loop {
// Get latest indexes from indexer
let current_indexes = indexer.get_current_indexes();
// Compute only new data
computer.compute(&indexer, current_indexes, &exit)?;
// Check for exit signal
if exit.is_signaled() {
break;
use brk_computer::Computer;
use brk_structs::{DateIndex, Height};
let computer = Computer::forced_import(/* ... */)?;
// Access market metrics after computation
if let Some(market) = &computer.market {
// Daily market cap analysis
let date_index = DateIndex::from_days_since_genesis(5000);
if let Some(market_cap) = market.dateindex_to_market_cap.get(date_index)? {
println!("Market cap on day {}: ${}", date_index, market_cap.to_dollars());
}
// Wait before next update
sleep(Duration::from_secs(60));
// MVRV (Market Value to Realized Value) ratio
if let Some(mvrv) = market.dateindex_to_mvrv.get(date_index)? {
println!("MVRV ratio: {:.2}", mvrv);
}
}
// Chain-level metrics
let height = Height::new(800000);
if let Some(difficulty) = computer.chain.height_to_difficulty.get(height)? {
println!("Network difficulty at height {}: {}", height, difficulty);
}
```
## Core Computer Structure
### Cohort Analysis
```rust
pub struct Computer {
pub indexes: indexes::Vecs, // Time indexing
pub constants: constants::Vecs, // Baseline values
pub blocks: blocks::Vecs, // Block analytics
pub mining: mining::Vecs, // Mining economics
pub market: market::Vecs, // Market metrics (optional)
pub price: Option<price::Vecs>, // OHLC price data (optional)
pub transactions: transactions::Vecs, // Transaction analysis
pub stateful: stateful::Vecs, // UTXO tracking
pub fetched: Option<fetched::Vecs>, // External data (optional)
pub cointime: cointime::Vecs, // Coin age analysis
use brk_computer::Computer;
use brk_structs::{DateIndex, CohortId};
let computer = Computer::forced_import(/* ... */)?;
// Address cohort analysis
let cohort_date = DateIndex::from_days_since_genesis(4000);
// Analyze address behavior patterns
if let Some(address_cohorts) = &computer.stateful.address_cohorts {
for cohort_id in address_cohorts.get_cohort_ids_for_date(cohort_date)? {
let cohort_data = address_cohorts.get_cohort(cohort_id)?;
println!("Cohort {}: {} addresses created",
cohort_id, cohort_data.addresses.len());
println!("Average holding period: {} days",
cohort_data.avg_holding_period.as_days());
}
}
// UTXO cohort lifecycle analysis
if let Some(utxo_cohorts) = &computer.stateful.utxo_cohorts {
let active_utxos = utxo_cohorts.get_active_utxos_for_date(cohort_date)?;
println!("Active UTXOs from cohort: {}", active_utxos.len());
}
```
## Performance Characteristics
### Supply and Monetary Analysis
**Benchmarked on MacBook Pro M3 Pro:**
- **Initial computation**: ~6-7 hours for complete Bitcoin blockchain
- **Storage efficiency**: All computed datasets total ~40GB
- **Incremental updates**: 3-5 seconds per new block
- **Memory footprint**: Peak ~7-8GB during computation, ~100MB during operation
- **Dependencies**: Price data domains optional (fetched, price, market)
```rust
use brk_computer::Computer;
use brk_structs::{Height, DateIndex};
## Domain-Specific Analytics
let computer = Computer::forced_import(/* ... */)?;
### Block Analytics
- Block sizes, weights, transaction counts
- Block intervals and mining statistics
- Fee analysis per block
// Supply dynamics
let height = Height::new(750000);
if let Some(supply) = computer.chain.height_to_circulating_supply.get(height)? {
println!("Circulating supply: {} BTC", supply.to_btc());
}
### Mining Economics
- Hashrate estimation and difficulty tracking
- Mining reward analysis
- Epoch-based calculations
// Realized vs unrealized analysis
let date = DateIndex::from_days_since_genesis(5000);
if let Some(realized_cap) = computer.market.dateindex_to_realized_cap.get(date)? {
if let Some(market_cap) = computer.market.dateindex_to_market_cap.get(date)? {
let unrealized_pnl = market_cap - realized_cap;
println!("Unrealized P&L: ${:.2}B", unrealized_pnl.to_dollars() / 1e9);
}
}
```
### Transaction Analysis
- Fee rate distributions
- RBF (Replace-By-Fee) detection
- Output type analysis
- Transaction size patterns
## Architecture
### Market Metrics (Optional)
- Price correlations with on-chain metrics
- Market cap calculations
- DCA analysis across timeframes
### Computation Pipeline
### Stateful Analysis
- UTXO set tracking
- Address cohort analysis
- Realized/unrealized gains
- Supply distribution metrics
The computer implements a sophisticated multi-stage pipeline:
## Requirements
1. **Index Computation**: Fundamental blockchain metrics and time-based indexes
2. **Constants Computation**: Network parameters and protocol constants
3. **Price Integration**: Optional price data fetching and processing
4. **Parallel Computation**: Chain, market, pools, stateful, and cointime analytics
5. **Cross-Dependencies**: Advanced metrics requiring multiple data sources
- **Indexed data**: Requires completed `brk_indexer` output
- **Storage space**: Additional ~40GB for computed datasets
- **Memory**: 8GB+ RAM recommended for initial computation
- **CPU**: Multi-core recommended for parallel processing
- **Price data**: Optional external price feeds for market analytics
### Memory Management
## Dependencies
**Allocation Tracking:**
- `brk_indexer` - Source of indexed blockchain data
- `brk_fetcher` - External price data (optional)
- `vecdb` - Vector database with lazy computation
- `rayon` - Parallel processing framework
- `brk_structs` - Bitcoin-aware type system
- `allocative` integration for memory usage analysis
- Efficient vector storage with compression options
- Strategic lazy vs. eager evaluation for memory optimization
**Performance Optimization:**
- `rayon` parallel processing for CPU-intensive calculations
- Vectorized operations for time-series computations
- Memory-mapped storage for large datasets
### State Management
**Stateful Analytics:**
- UTXO lifecycle tracking with creation/destruction events
- Address cohort analysis with behavioral clustering
- Transaction pattern recognition and anomaly detection
- Economic cycle analysis with market phase detection
**Cointime Economics:**
- Bitcoin days destroyed and accumulated calculations
- Velocity measurements and economic activity indicators
- Age-weighted value transfer analysis
- Long-term holder vs. active trader segmentation
### Modular Design
Each computation module operates independently:
- **Chain Module**: Basic blockchain metrics (fees, difficulty, hashrate)
- **Market Module**: Price-dependent financial calculations
- **Pools Module**: Mining centralization and pool analysis
- **Stateful Module**: Advanced lifecycle and behavior tracking
- **Cointime Module**: Economic time-value calculations
### Data Dependencies
**Required Dependencies:**
- `brk_indexer`: Raw blockchain data access
- `brk_structs`: Type definitions and conversions
**Optional Dependencies:**
- `brk_fetcher`: Price data for financial metrics
- Market analysis requires price integration
### Computation Orchestration
**Sequential Stages:**
1. Indexes → Constants (foundational metrics)
2. Fetched → Price (price data processing)
3. Parallel: Chain, Market, Pools, Stateful, Cointime
**Exit Handling:**
- Graceful shutdown with consistent state preservation
- Checkpoint-based recovery for long-running computations
- Multi-threaded coordination with exit signaling
## Code Analysis Summary
**Main Structure**: `Computer` struct coordinating 7 specialized analytics modules (indexes, constants, market, pools, chain, stateful, cointime) \
**Computation Pipeline**: Multi-stage analytics processing with parallel execution and dependency management \
**State Tracking**: Advanced UTXO and address lifecycle analysis with cohort-based behavioral clustering \
**Financial Analytics**: Comprehensive market metrics including realized/unrealized analysis and cointime economics \
**Memory Optimization**: `allocative` tracking with lazy/eager evaluation strategies and compressed vector storage \
**Parallel Processing**: `rayon` integration for CPU-intensive calculations with coordinated exit handling \
**Architecture**: Modular analytics engine transforming indexed blockchain data into actionable financial and economic insights
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_error/README.md
+101 -98
View File
@@ -1,140 +1,143 @@
# brk_error
**Centralized error handling for the Bitcoin Research Kit**
Centralized error handling for Bitcoin-related operations and database interactions.
`brk_error` provides a unified error type and result system used throughout the BRK ecosystem. It consolidates error handling from multiple external dependencies and adds Bitcoin-specific error variants.
[![Crates.io](https://img.shields.io/crates/v/brk_error.svg)](https://crates.io/crates/brk_error)
[![Documentation](https://docs.rs/brk_error/badge.svg)](https://docs.rs/brk_error)
## What it provides
## Overview
- **Unified Error Type**: Single `Error` enum that covers all error cases across BRK crates
- **Convenient Result Type**: Pre-configured `Result<T, E = Error>` for consistent error handling
- **External Error Integration**: Automatic conversions from common library errors
- **Bitcoin-Specific Errors**: Domain-specific error variants for blockchain data processing
This crate provides a unified error type that consolidates error handling across Bitcoin blockchain analysis tools. It wraps errors from multiple external libraries including Bitcoin Core RPC, database operations, HTTP requests, and serialization operations into a single `Error` enum.
## Key Features
**Key Features:**
### Centralized Error Management
- Single error type for the entire BRK ecosystem
- Consistent error handling patterns across all crates
- Reduced error type complexity in public APIs
- Unified error type covering 11+ different error sources
- Automatic conversions from external library errors
- Bitcoin-specific error variants for blockchain operations
- Database error handling for both Fjall and VecDB storage backends
- Custom error types for domain-specific validation failures
### External Library Integration
Automatic `From` implementations for errors from:
- **I/O Operations**: `std::io::Error`
- **Bitcoin Core RPC**: `bitcoincore_rpc::Error`
- **Database Operations**: `fjall::Error`, `vecdb::Error`
- **Serialization**: `serde_json::Error`
- **Time Operations**: `jiff::Error`, `SystemTimeError`
- **HTTP Requests**: `minreq::Error`
- **Zero-Copy Operations**: `zerocopy` conversion errors
**Target Use Cases:**
### Bitcoin-Specific Error Variants
- `WrongAddressType` - Invalid address type for operation
- `UnindexableDate` - Date before Bitcoin genesis (2009-01-03)
- `WrongLength` - Invalid data length for Bitcoin structures
- `QuickCacheError` - Cache operation failures
- Applications processing Bitcoin blockchain data
- Systems requiring unified error handling across multiple storage backends
- Tools integrating Bitcoin Core RPC with local databases
## Usage
## Installation
### Basic Error Handling
```toml
cargo add brk_error
```
## Quick Start
```rust
use brk_error::{Error, Result};
fn process_block() -> Result<Block> {
let rpc_client = get_rpc_client()?; // bitcoincore_rpc::Error -> Error
let block_data = rpc_client.get_block_info(&hash)?;
// Custom Bitcoin-specific validation
if block_data.height < 0 {
return Err(Error::Str("Invalid block height"));
fn process_transaction() -> Result<()> {
// Function automatically converts various error types
let data = std::fs::read("transaction.json")?; // IO error auto-converted
let parsed: serde_json::Value = serde_json::from_slice(&data)?; // JSON error auto-converted
// Custom domain errors
if data.len() < 32 {
return Err(Error::WrongLength);
}
Ok(block_data)
Ok(())
}
```
### Working with External Libraries
## API Overview
### Core Types
- **`Error`**: Main error enum consolidating all error types
- **`Result<T, E = Error>`**: Type alias for `std::result::Result` with default `Error` type
### Error Categories
**External Library Errors:**
- `BitcoinRPC`: Bitcoin Core RPC client errors
- `BitcoinConsensusEncode`: Bitcoin consensus encoding failures
- `Fjall`/`VecDB`/`SeqDB`: Database operation errors
- `Minreq`: HTTP request errors
- `SerdeJson`: JSON serialization errors
- `Jiff`: Date/time handling errors
- `ZeroCopyError`: Zero-copy conversion failures
**Domain-Specific Errors:**
- `WrongLength`: Invalid data length for Bitcoin operations
- `WrongAddressType`: Unsupported Bitcoin address format
- `UnindexableDate`: Date outside valid blockchain range (before 2009-01-03)
- `QuickCacheError`: Cache operation failures
**Generic Errors:**
- `Str(&'static str)`: Static string errors
- `String(String)`: Dynamic string errors
### Key Methods
All external error types automatically convert to `Error` via `From` trait implementations. The error type implements `std::error::Error`, `Debug`, and `Display` traits for comprehensive error reporting.
## Examples
### Bitcoin RPC Integration
```rust
use brk_error::Result;
use bitcoincore_rpc::{Client, Auth};
fn get_block_count(client: &Client) -> Result<u64> {
let count = client.get_block_count()?; // Auto-converts bitcoincore_rpc::Error
Ok(count)
}
```
### Database Operations
```rust
use brk_error::Result;
fn save_data(data: &[u8]) -> Result<()> {
// I/O error automatically converted
std::fs::write("data.bin", data)?;
// JSON serialization error automatically converted
let json = serde_json::to_string(&data)?;
// Database error automatically converted
database.insert("key", &json)?;
fn store_transaction_data(db: &fjall::Keyspace, data: &[u8]) -> Result<()> {
if data.len() != 32 {
return Err(brk_error::Error::WrongLength);
}
db.insert(b"tx_hash", data)?; // Auto-converts fjall::Error
Ok(())
}
```
### Bitcoin-Specific Validation
### Date Validation
```rust
use brk_error::{Error, Result};
use jiff::civil::Date;
fn validate_date(date: &Date) -> Result<()> {
if *date < Date::new(2009, 1, 3) {
fn validate_blockchain_date(date: Date) -> Result<()> {
let genesis_date = Date::constant(2009, 1, 3);
let earliest_valid = Date::constant(2009, 1, 9);
if date < genesis_date || (date > genesis_date && date < earliest_valid) {
return Err(Error::UnindexableDate);
}
Ok(())
}
fn validate_address_type(output_type: OutputType) -> Result<()> {
if !output_type.is_address() {
return Err(Error::WrongAddressType);
}
Ok(())
}
```
### String Errors
## Code Analysis Summary
```rust
// Static string errors (zero allocation)
Err(Error::Str("Invalid configuration"))
// Dynamic string errors
Err(Error::String(format!("Block {} not found", height)))
```
## Result Type
The crate provides a convenient `Result` type alias:
```rust
pub type Result<T, E = Error> = std::result::Result<T, E>;
```
This allows for clean function signatures throughout BRK:
```rust
fn parse_block() -> Result<Block> { /* ... */ }
fn index_transactions() -> Result<Vec<Transaction>> { /* ... */ }
```
## Error Display
All errors implement `Display` and `std::error::Error`, providing:
- Formatted error messages for debugging
- Error chain support for nested errors
- Integration with error handling libraries like `anyhow`
## Dependencies
- `vecdb` - Vector database error types
- `bitcoincore-rpc` - Bitcoin Core RPC client errors
- `fjall` - Key-value store errors
- `jiff` - Date/time operation errors
- `minreq` - HTTP request errors
- `serde_json` - JSON serialization errors
- `zerocopy` - Zero-copy conversion errors
**Main Type**: `Error` enum with 24 variants covering external libraries and domain-specific cases \
**Conversion Traits**: Implements `From` for 10+ external error types enabling automatic error propagation \
**Error Handling**: Standard Rust error handling with `std::error::Error` trait implementation \
**Dependencies**: Integrates errors from `bitcoin`, `bitcoincore-rpc`, `fjall`, `vecdb`, `jiff`, `minreq`, `serde_json`, and `zerocopy` crates \
**Architecture**: Centralized error aggregation pattern with automatic conversions and custom domain errors
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_fetcher/README.md
+124 -140
View File
@@ -1,39 +1,36 @@
# brk_fetcher
**Bitcoin price data fetcher with multi-source fallback and retry logic**
Multi-source Bitcoin price data aggregator with automatic fallback between exchanges.
`brk_fetcher` provides reliable Bitcoin price data retrieval from multiple sources including Binance, Kraken, and BRK instances. It offers both date-based and block height-based price queries with automatic fallback and retry mechanisms for robust data collection.
[![Crates.io](https://img.shields.io/crates/v/brk_fetcher.svg)](https://crates.io/crates/brk_fetcher)
[![Documentation](https://docs.rs/brk_fetcher/badge.svg)](https://docs.rs/brk_fetcher)
## What it provides
## Overview
- **Multi-source fallback**: Automatic fallback between Kraken → Binance → BRK
- **Flexible querying**: Fetch prices by date or block height with timestamps
- **Retry logic**: Built-in retry mechanism with exponential backoff
- **Multiple timeframes**: 1-minute and 1-day interval support
- **HAR file import**: Import historical Binance chart data from browser
This crate provides a unified interface for fetching Bitcoin price data from multiple sources including Binance, Kraken, and a custom BRK API. It implements automatic failover between data sources, retry mechanisms, and supports both real-time and historical price queries using blockchain height or date-based lookups.
## Key Features
**Key Features:**
### Data Sources
- **Kraken API**: Primary source for OHLC data (1-day and 1-minute intervals)
- **Binance API**: Secondary source with additional historical data
- **BRK instance**: Fallback source for previously cached price data
- **HAR import**: Manual historical data import from browser sessions
- Multi-source price aggregation (Binance, Kraken, BRK API)
- Automatic fallback hierarchy with intelligent retry logic
- Historical price queries by blockchain height or date
- Support for both 1-minute and daily OHLC data
- HAR file import for extended historical data coverage
- Built-in caching with BTreeMap storage for performance
### Query Methods
- **Date-based queries**: Get OHLC data for specific calendar dates
- **Height-based queries**: Get OHLC data for specific block heights with timestamps
- **Automatic aggregation**: Combines minute-level data for block intervals
**Target Use Cases:**
### Reliability Features
- **Automatic fallback**: Tries sources in order until successful
- **Retry mechanism**: Up to 12 hours of retries with 60-second intervals
- **Cache clearing**: Automatic cache refresh on failures
- **Error handling**: Graceful degradation with detailed error messages
- Bitcoin blockchain analyzers requiring accurate historical pricing
- Applications needing resilient price data with multiple fallbacks
- Tools processing large datasets requiring efficient price lookups
## Usage
## Installation
### Basic Setup
```toml
cargo add brk_fetcher
```
## Quick Start
```rust
use brk_fetcher::Fetcher;
@@ -42,145 +39,132 @@ use brk_structs::{Date, Height, Timestamp};
// Initialize fetcher with exchange APIs enabled
let mut fetcher = Fetcher::import(true, None)?;
// Initialize without exchange APIs (BRK-only mode)
let mut fetcher = Fetcher::import(false, None)?;
// Fetch price by date
let date = Date::from_ymd(2023, 6, 15)?;
let daily_price = fetcher.get_date(date)?;
// Initialize with HAR file for historical Binance data
let har_path = Path::new("./binance.har");
let mut fetcher = Fetcher::import(true, Some(har_path))?;
// Fetch price by blockchain height
let height = Height::new(800000);
let timestamp = Timestamp::from(1684771200u32);
let block_price = fetcher.get_height(height, timestamp, None)?;
println!("Daily OHLC: {:?}", daily_price);
println!("Block OHLC: {:?}", block_price);
```
### Date-based Price Queries
## API Overview
### Core Types
- **`Fetcher`**: Main aggregator managing multiple price data sources
- **`Binance`**: Binance exchange API client with HAR file support
- **`Kraken`**: Kraken exchange API client for OHLC data
- **`BRK`**: Custom API client for blockchain-indexed price data
### Key Methods
**`Fetcher::import(exchanges: bool, hars_path: Option<&Path>) -> Result<Self>`**
Creates a new fetcher instance with configurable data sources.
**`get_date(&mut self, date: Date) -> Result<OHLCCents>`**
Retrieves daily OHLC data for the specified date with automatic source fallback.
**`get_height(&mut self, height: Height, timestamp: Timestamp, previous_timestamp: Option<Timestamp>) -> Result<OHLCCents>`**
Fetches price data for a specific blockchain height using minute-level precision.
### Data Source Hierarchy
1. **Kraken API** - Primary source for both 1-minute and daily data
2. **Binance API** - Secondary source with extended HAR file support
3. **BRK API** - Fallback source using blockchain-indexed pricing data
### Error Handling
The fetcher implements aggressive retry logic with exponential backoff, attempting each source up to 12 hours (720 retries) before failing. Failed requests trigger cache clearing and source rotation.
## Examples
### Basic Price Fetching
```rust
use brk_fetcher::Fetcher;
use brk_structs::Date;
// Fetch OHLC data for a specific date
let date = Date::new(2024, 12, 25);
let ohlc = fetcher.get_date(date)?;
let mut fetcher = Fetcher::import(true, None)?;
println!("Bitcoin price on {}: ${:.2}", date, ohlc.close.dollars());
println!("Daily high: ${:.2}", ohlc.high.dollars());
println!("Daily low: ${:.2}", ohlc.low.dollars());
```
### Block Height-based Price Queries
```rust
use brk_structs::{Height, Timestamp};
// Fetch price at specific block height
let height = Height::new(900_000);
let timestamp = Timestamp::from_block_height(height);
let previous_timestamp = Some(Timestamp::from_block_height(Height::new(899_999)));
let ohlc = fetcher.get_height(height, timestamp, previous_timestamp)?;
println!("Bitcoin price at block {}: ${:.2}", height, ohlc.close.dollars());
```
### Working with OHLC Data
```rust
use brk_structs::OHLCCents;
// OHLC data is returned in cents for precision
let ohlc: OHLCCents = fetcher.get_date(date)?;
// Convert to dollars for display
println!("Open: ${:.2}", ohlc.open.dollars());
println!("High: ${:.2}", ohlc.high.dollars());
println!("Low: ${:.2}", ohlc.low.dollars());
println!("Close: ${:.2}", ohlc.close.dollars());
// Access raw cent values
println!("Close in cents: {}", ohlc.close.0);
```
### Using Individual Sources
```rust
use brk_fetcher::{Binance, Kraken, BRK};
// Use specific exchanges directly
let binance = Binance::init(None);
let kraken = Kraken::default();
let brk = BRK::default();
// Fetch from specific source
let binance_data = binance.get_from_1d(&date)?;
let kraken_data = kraken.get_from_1mn(timestamp, previous_timestamp)?;
let brk_data = brk.get_from_height(height)?;
```
### Error Handling and Retries
```rust
// The fetcher automatically retries on failures
// Fetch Bitcoin price for a specific date
let date = Date::from_ymd(2021, 1, 1)?;
match fetcher.get_date(date) {
Ok(ohlc) => println!("Successfully fetched: ${:.2}", ohlc.close.dollars()),
Err(e) => {
// After all retries and sources exhausted
eprintln!("Failed to fetch price data: {}", e);
Ok(ohlc) => println!("BTC price on {}: ${}", date, ohlc.close.to_dollars()),
Err(e) => eprintln!("Failed to fetch price: {}", e),
}
```
### Historical Data with HAR Files
```rust
use brk_fetcher::Fetcher;
use std::path::Path;
// Initialize with HAR file path for extended historical coverage
let har_path = Path::new("./import_data");
let mut fetcher = Fetcher::import(true, Some(har_path))?;
// Fetch minute-level data using HAR file fallback
let height = Height::new(650000);
let timestamp = Timestamp::from(1598918400u32); // August 2020
let price_data = fetcher.get_height(height, timestamp, None)?;
```
### Batch Processing with Caching
```rust
use brk_fetcher::Fetcher;
let mut fetcher = Fetcher::import(true, None)?;
// Process multiple heights - caching improves performance
for height in 800000..800100 {
let timestamp = Timestamp::from(1684771200u32 + (height - 800000) * 600);
match fetcher.get_height(Height::new(height), timestamp, None) {
Ok(ohlc) => println!("Height {}: ${:.2}", height, ohlc.close.to_dollars()),
Err(e) => eprintln!("Error at height {}: {}", height, e),
}
}
// Clear cache to force fresh data
// Clear caches when done
fetcher.clear();
```
## Data Sources and Limitations
## Architecture
### Kraken API
- **1-day data**: Historical daily OHLC data
- **1-minute data**: Limited to last ~10 hours
- **Rate limits**: Subject to Kraken API restrictions
### Retry Mechanism
### Binance API
- **1-day data**: Historical daily OHLC data
- **1-minute data**: Limited to last ~16 hours
- **HAR import**: Can extend historical coverage via browser data
The crate implements a sophisticated retry system with:
### BRK Instance
- **Cached data**: Previously fetched price data
- **Offline capability**: Works without internet when data is cached
- **Height-based**: Optimized for block height queries
- **Default retry count**: 6 attempts with 5-second delays
- **Extended retry**: Up to 720 attempts (12 hours) for critical operations
- **Cache invalidation**: Automatic cache clearing between retry attempts
- **Exponential backoff**: 60-second delays for extended retries
## HAR File Import
### Data Aggregation
For historical data beyond API limits:
Price data is aggregated using OHLC (Open, High, Low, Close) calculations spanning timestamp ranges. The `find_height_ohlc` function computes accurate OHLC values by scanning time series data between block timestamps.
1. Visit [Binance BTC/USDT chart](https://www.binance.com/en/trade/BTC_USDT?type=spot)
2. Set chart to 1-minute interval
3. Open browser dev tools, go to Network tab
4. Filter by 'uiKlines'
5. Scroll chart to desired historical period
6. Export network requests as HAR file
7. Initialize fetcher with HAR path
### HAR File Processing
## Fallback Strategy
Binance integration supports HTTP Archive (HAR) files for extended historical data coverage, parsing browser network captures to extract additional pricing data beyond API limitations.
The fetcher tries sources in this order:
1. **Kraken** - Primary source for most queries
2. **Binance** - Secondary source with extended coverage
3. **BRK** - Fallback for cached/computed prices
## Code Analysis Summary
If all sources fail, it retries up to 12 hours with 60-second intervals.
## Performance and Reliability
- **Automatic retries**: Up to 720 attempts (12 hours) with 60-second delays
- **Cache management**: Clears cache on failures to force fresh data
- **Error logging**: Detailed failure reporting with recovery instructions
- **Graceful degradation**: Falls back through sources until successful
## Dependencies
- `brk_structs` - Bitcoin-aware type system (Date, Height, OHLC types)
- `brk_error` - Unified error handling
- `minreq` - HTTP client for API requests
- `serde_json` - JSON parsing for API responses
- `log` - Logging for retry and error reporting
**Main Types**: `Fetcher` aggregator with `Binance`, `Kraken`, and `BRK` source implementations \
**Caching**: BTreeMap-based caching for both timestamp and date-indexed price data \
**Network Layer**: Built on `minreq` HTTP client with automatic JSON parsing \
**Error Handling**: Comprehensive retry logic with source rotation and cache management \
**Dependencies**: Integrates `brk_structs` for type definitions and `brk_error` for unified error handling \
**Architecture**: Multi-source aggregation pattern with hierarchical fallback and intelligent caching
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_indexer/README.md
+224 -137
View File
@@ -1,190 +1,277 @@
# brk_indexer
**High-performance Bitcoin blockchain indexer with dual storage architecture**
High-performance Bitcoin blockchain indexer with parallel processing and dual storage architecture.
`brk_indexer` processes raw Bitcoin Core block data and creates efficient storage structures using both vectors (time-series) and key-value stores (lookups). It serves as the foundation of BRK's data pipeline, organizing all blockchain data into optimized formats for fast retrieval and analysis.
[![Crates.io](https://img.shields.io/crates/v/brk_indexer.svg)](https://crates.io/crates/brk_indexer)
[![Documentation](https://docs.rs/brk_indexer/badge.svg)](https://docs.rs/brk_indexer)
## What it provides
## Overview
- **Dual Storage Architecture**: Vectors for time-series data, key-value stores for lookups
- **Memory Efficiency**: ~5-6GB peak RAM usage during full blockchain indexing
- **Incremental Processing**: Resume from last indexed height with rollback protection
- **Data Integrity**: Collision detection and validation during indexing
- **All Bitcoin Data Types**: Complete support for blocks, transactions, inputs, outputs, and addresses
This crate provides a comprehensive Bitcoin blockchain indexer built on top of `brk_parser`. It processes raw Bitcoin blocks in parallel, extracting and indexing transactions, addresses, inputs, outputs, and metadata into optimized storage structures. The indexer maintains two complementary storage systems: columnar vectors for analytics and key-value stores for fast lookups.
## Key Features
**Key Features:**
### Storage Strategy
- Parallel block processing with multi-threaded transaction analysis
- Dual storage architecture: columnar vectors + key-value stores
- Address type classification and indexing for all Bitcoin script types
- Collision detection and validation for address hashes and transaction IDs
- Incremental processing with automatic rollback and recovery
- Height-based synchronization with Bitcoin Core RPC validation
- Optimized batch operations with configurable snapshot intervals
**Vector Storage (time-series data):**
- Block metadata (height, timestamp, hash, difficulty, size)
- Transaction data (version, locktime, RBF flag, indices)
- Input/Output mappings and values
- Address bytes for all output types
- Efficient for range queries and analytics
**Target Use Cases:**
**Key-Value Storage (lookups):**
- Block hash prefixes → heights
- Transaction ID prefixes → transaction indices
- Address byte hashes → type indices
- Fast point queries by hash or address
- Bitcoin blockchain analysis requiring full transaction history
- Address clustering and UTXO set analysis
- Blockchain explorers needing fast address/transaction lookups
- Research applications requiring structured access to blockchain data
### Performance Features
- **Parallel Processing**: Concurrent transaction and output processing using Rayon
- **Batch Operations**: Periodic commits every 1,000 blocks for optimal I/O
- **Memory Efficiency**: Optimized data structures minimize RAM usage
- **Incremental Updates**: Handles blockchain reorganizations automatically
## Installation
### Address Type Support
Complete support for all Bitcoin address types:
- P2PK (65-byte and 33-byte), P2PKH, P2SH
- P2WPKH, P2WSH, P2TR, P2A
- P2MS (multisig), OpReturn, Empty, Unknown
```toml
cargo add brk_indexer
```
## Usage
### Basic Indexing
## Quick Start
```rust
use brk_indexer::Indexer;
use brk_parser::Parser;
use bitcoincore_rpc::{Auth, Client};
use bitcoincore_rpc::{Client, Auth};
use vecdb::Exit;
use std::path::Path;
// Setup Bitcoin Core RPC connection
let rpc = Box::leak(Box::new(Client::new(
"http://localhost:8332",
Auth::CookieFile(Path::new("~/.bitcoin/.cookie")),
)?));
// Initialize Bitcoin Core RPC client
let rpc = Client::new("http://localhost:8332", Auth::None)?;
let rpc = Box::leak(Box::new(rpc));
// Create parser for Bitcoin Core block files
let parser = Parser::new(
Path::new("~/.bitcoin/blocks").to_path_buf(),
Some(Path::new("./brk_data").to_path_buf()),
rpc
);
// Create parser for raw block data
let blocks_dir = Path::new("/path/to/bitcoin/blocks");
let parser = Parser::new(blocks_dir, None, rpc);
// Create indexer with forced import (resets if needed)
let mut indexer = Indexer::forced_import(Path::new("./brk_data"))?;
// Initialize indexer with output directory
let outputs_dir = Path::new("./indexed_data");
let mut indexer = Indexer::forced_import(outputs_dir)?;
// Setup graceful shutdown handler
let exit = Exit::new();
exit.set_ctrlc_handler();
// Index blockchain data
let exit = Exit::default();
let starting_indexes = indexer.index(&parser, rpc, &exit, true)?;
// Index the blockchain
let indexes = indexer.index(&parser, rpc, &exit, true)?;
println!("Indexed up to height: {}", indexes.height);
println!("Indexed up to height: {}", starting_indexes.height);
```
### Continuous Indexing
## API Overview
### Core Types
- **`Indexer`**: Main coordinator managing vectors and stores
- **`Vecs`**: Columnar storage for blockchain data analytics
- **`Stores`**: Key-value storage for fast hash-based lookups
- **`Indexes`**: Current indexing state tracking progress across data types
### Key Methods
**`Indexer::forced_import(outputs_dir: &Path) -> Result<Self>`**
Creates or opens indexer instance with automatic version management.
**`index(&mut self, parser: &Parser, rpc: &'static Client, exit: &Exit, check_collisions: bool) -> Result<Indexes>`**
Main indexing function processing blocks from parser with collision detection.
### Storage Architecture
**Columnar Vectors (Vecs):**
- `height_to_*`: Block-level data (hash, timestamp, difficulty, size, weight)
- `txindex_to_*`: Transaction data (ID, version, locktime, size, RBF flag)
- `outputindex_to_*`: Output data (value, type, address mapping)
- `inputindex_to_outputindex`: Input-to-output relationship mapping
**Key-Value Stores:**
- `addressbyteshash_to_typeindex`: Address hash to internal index mapping
- `blockhashprefix_to_height`: Block hash prefix to height lookup
- `txidprefix_to_txindex`: Transaction ID prefix to internal index
- `addresstype_to_typeindex_with_outputindex`: Address type to output mappings
### Address Type Support
Complete coverage of Bitcoin script types:
- **P2PK**: Pay-to-Public-Key (33-byte and 65-byte variants)
- **P2PKH**: Pay-to-Public-Key-Hash
- **P2SH**: Pay-to-Script-Hash
- **P2WPKH**: Pay-to-Witness-Public-Key-Hash
- **P2WSH**: Pay-to-Witness-Script-Hash
- **P2TR**: Pay-to-Taproot
- **P2MS**: Pay-to-Multisig
- **P2A**: Pay-to-Address (custom type)
- **OpReturn**: OP_RETURN data outputs
- **Empty/Unknown**: Non-standard script types
## Examples
### Basic Indexing Operation
```rust
use std::time::{Duration, Instant};
use std::thread::sleep;
use brk_indexer::Indexer;
use brk_parser::Parser;
use std::path::Path;
// Continuous indexing loop for real-time updates
loop {
let start_time = Instant::now();
// Initialize components
let outputs_dir = Path::new("./blockchain_index");
let mut indexer = Indexer::forced_import(outputs_dir)?;
// Index new blocks
let indexes = indexer.index(&parser, rpc, &exit, true)?;
let blocks_dir = Path::new("/Users/satoshi/.bitcoin/blocks");
let parser = Parser::new(blocks_dir, None, rpc);
println!("Indexed to height {} in {:?}",
indexes.height, start_time.elapsed());
// Index with collision checking enabled
let exit = vecdb::Exit::default();
let final_indexes = indexer.index(&parser, rpc, &exit, true)?;
// Check for exit signal
if exit.is_signaled() {
println!("Graceful shutdown requested");
break;
}
println!("Final height: {}", final_indexes.height);
println!("Total transactions: {}", final_indexes.txindex);
println!("Total addresses: {}", final_indexes.total_address_count());
```
// Wait before next update cycle
sleep(Duration::from_secs(5 * 60));
### Querying Indexed Data
```rust
use brk_indexer::Indexer;
use brk_structs::{Height, TxidPrefix, AddressBytesHash};
let indexer = Indexer::forced_import("./blockchain_index")?;
// Look up block hash by height
let height = Height::new(750000);
if let Some(block_hash) = indexer.vecs.height_to_blockhash.get(height)? {
println!("Block 750000 hash: {}", block_hash);
}
// Look up transaction by ID prefix
let txid_prefix = TxidPrefix::from_str("abcdef123456")?;
if let Some(tx_index) = indexer.stores.txidprefix_to_txindex.get(&txid_prefix)? {
println!("Transaction index: {}", tx_index);
}
// Query address information
let address_hash = AddressBytesHash::from(/* address bytes */);
if let Some(type_index) = indexer.stores.addressbyteshash_to_typeindex.get(&address_hash)? {
println!("Address type index: {}", type_index);
}
```
### Accessing Indexed Data
### Incremental Processing
```rust
// Access the underlying storage structures
let vecs = &indexer.vecs;
let stores = &indexer.stores;
use brk_indexer::Indexer;
// Get block hash at specific height
let block_hash = vecs.height_to_blockhash.get(Height::new(800_000))?;
// Indexer automatically resumes from last processed height
let mut indexer = Indexer::forced_import("./blockchain_index")?;
// Look up transaction by prefix
let tx_prefix = TxidPrefix::from(&txid);
let tx_index = stores.txidprefix_to_txindex.get(&tx_prefix)?;
let current_indexes = indexer.vecs.current_indexes(&indexer.stores, rpc)?;
println!("Resuming from height: {}", current_indexes.height);
// Get address data
let address_hash = AddressBytesHash::from(&address_bytes);
let type_index = stores.addressbyteshash_to_anyaddressindex.get(&address_hash)?;
// Process new blocks incrementally
let exit = vecdb::Exit::default();
let updated_indexes = indexer.index(&parser, rpc, &exit, true)?;
println!("Processed {} new blocks",
updated_indexes.height.as_u32() - current_indexes.height.as_u32());
```
### Address Type Analysis
```rust
use brk_indexer::Indexer;
use brk_structs::OutputType;
let indexer = Indexer::forced_import("./blockchain_index")?;
// Analyze address distribution by type
for output_type in OutputType::as_vec() {
let count = indexer.vecs.outputindex_to_outputtype
.iter()
.filter(|&ot| ot == output_type)
.count();
println!("{:?}: {} outputs", output_type, count);
}
// Query specific address type data
let p2pkh_store = &indexer.stores.addresstype_to_typeindex_with_outputindex
.p2pkh;
println!("P2PKH addresses: {}", p2pkh_store.len());
```
## Architecture
### Parallel Processing
The indexer uses sophisticated parallel processing:
- **Block-Level Parallelism**: Concurrent processing of transactions within blocks
- **Transaction Analysis**: Parallel input/output processing with `rayon`
- **Address Resolution**: Multi-threaded address type classification and indexing
- **Collision Detection**: Parallel validation of hash collisions across address types
### Storage Optimization
**Columnar Storage (vecdb):**
- Compressed vectors for space-efficient analytics queries
- Raw vectors for frequently accessed data (heights, hashes)
- Page-aligned storage for memory mapping efficiency
**Key-Value Storage (Fjall):**
- LSM-tree architecture for write-heavy indexing workloads
- Bloom filters for fast negative lookups
- Transactional consistency with rollback support
### Memory Management
- **Batch Processing**: 1000-block snapshots to balance memory and I/O
- **Reader Management**: Static readers for consistent data access during processing
- **Collision Tracking**: BTreeMap-based collision detection with memory cleanup
- **Exit Handling**: Graceful shutdown with consistent state preservation
### Version Management
- **Schema Versioning**: Automatic migration on version changes (currently v21)
- **Rollback Support**: Automatic recovery from incomplete processing
- **State Tracking**: Height-based synchronization across all storage components
## Performance Characteristics
**Benchmarked on MacBook Pro M3 Pro (36GB RAM):**
- **Full blockchain sync** (to ~892k blocks): 7-8 hours
- **Peak memory usage**: 5-6GB
- **Storage overhead**: ~27% of Bitcoin Core block size
- **Incremental updates**: Very fast, efficient resume from last height
### Processing Speed
## Data Organization
- **Parallel Transaction Processing**: Multi-core utilization for CPU-intensive operations
- **Optimized I/O**: Batch operations reduce disk overhead
- **Memory Efficiency**: Streaming processing without loading entire blockchain
The indexer creates this storage structure:
```
brk_data/
├── indexed/
│ ├── vecs/ # Vector storage
│ │ ├── height_to_* # Height-indexed data
│ │ ├── txindex_to_* # Transaction-indexed data
│ │ └── outputindex_to_* # Output-indexed data
│ └── stores/ # Key-value stores
│ ├── hash_lookups/ # Block/TX hash mappings
│ └── address_maps/ # Address type mappings
└── metadata/ # Versioning and state
```
### Storage Requirements
## Indexes Tracking
- **Columnar Compression**: Significant space savings for repetitive blockchain data
- **Index Optimization**: Bloom filters reduce lookup overhead
- **Incremental Growth**: Storage scales linearly with blockchain size
The indexer maintains current indices during processing:
### Scalability
```rust
pub struct Indexes {
pub height: Height, // Current block height
pub txindex: TxIndex, // Current transaction index
pub inputindex: InputIndex, // Current input index
pub outputindex: OutputIndex, // Current output index
pub p2pkhaddressindex: P2PKHAddressIndex, // P2PKH address index
// ... indices for all address types
}
```
- **Height-Based Partitioning**: Enables distributed processing strategies
- **Modular Architecture**: Separate vector and store systems for flexible deployment
- **Resource Configuration**: Configurable batch sizes and memory limits
## Requirements
## Code Analysis Summary
- **Bitcoin Core node** with RPC enabled
- **Block file access** to `~/.bitcoin/blocks/`
- **Storage space**: Minimum 500GB (scales with blockchain growth)
- **Memory**: 8GB+ RAM recommended
- **CPU**: Multi-core recommended for parallel processing
## Rollback and Recovery
- **Automatic rollback** on interruption or blockchain reorgs
- **State persistence** for efficient restart
- **Version management** for storage format compatibility
- **Graceful shutdown** with Ctrl+C handling
## Dependencies
- `brk_parser` - Bitcoin block parsing and sequential access
- `brk_store` - Key-value storage wrapper (fjall-based)
- `vecdb` - Vector database for time-series storage
- `bitcoin` - Bitcoin protocol types and parsing
- `rayon` - Parallel processing framework
- `bitcoincore_rpc` - Bitcoin Core RPC client
**Main Structure**: `Indexer` coordinating `Vecs` (columnar analytics) and `Stores` (key-value lookups) \
**Processing Pipeline**: Multi-threaded block analysis with parallel transaction/address processing \
**Storage Architecture**: Dual system using vecdb for analytics and Fjall for lookups \
**Address Indexing**: Complete Bitcoin script type coverage with collision detection \
**Synchronization**: Height-based coordination with Bitcoin Core RPC validation \
**Parallel Processing**: rayon-based parallelism for transaction analysis and address resolution \
**Architecture**: High-performance blockchain indexer with ACID guarantees and incremental processing
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_interface/README.md
+214 -176
View File
@@ -1,227 +1,265 @@
# brk_interface
**Unified data query and formatting interface for Bitcoin datasets**
Unified data query and formatting interface for Bitcoin datasets with intelligent search and multi-format output.
`brk_interface` provides a clean, unified API for accessing Bitcoin datasets from both indexer and computer components. It serves as the primary data access layer powering BRK's web API and MCP endpoints, offering flexible querying, pagination, and multiple output formats.
[![Crates.io](https://img.shields.io/crates/v/brk_interface.svg)](https://crates.io/crates/brk_interface)
[![Documentation](https://docs.rs/brk_interface/badge.svg)](https://docs.rs/brk_interface)
## What it provides
## Overview
- **Unified Data Access**: Single interface to query both indexed blockchain data and computed analytics
- **Multiple Output Formats**: JSON, CSV, TSV, and Markdown table formatting
- **Flexible Pagination**: Range queries with positive/negative indexing and automatic pagination
- **Multi-dataset Queries**: Retrieve multiple datasets with the same time index in one call
- **Dynamic Search**: Intelligent ID matching with automatic fallbacks and normalization
This crate provides a high-level interface for querying and formatting data from BRK's indexer and computer components. It offers intelligent vector search with fuzzy matching, parameter validation, range queries, and multi-format output (JSON, CSV, TSV, Markdown) with efficient caching and pagination support.
## Key Features
**Key Features:**
### Query Interface
- **25 Time Indices**: From granular (Height, DateIndex) to aggregate (YearIndex, DecadeIndex)
- **Bitcoin-Specific Indices**: Address types (P2PKH, P2SH, P2TR), output types, epochs
- **Multi-dataset support**: Query multiple related datasets simultaneously
- **Intelligent ID resolution**: Flexible matching with automatic fallbacks
- Unified query interface across indexer and computer data sources
- Intelligent search with fuzzy matching and helpful error messages
- Multi-format output: JSON, CSV, TSV, Markdown with proper formatting
- Range-based data queries with flexible from/to parameters
- Comprehensive pagination support for large datasets
- Schema validation with JSON Schema generation for API documentation
- Efficient caching system for error messages and repeated queries
### Pagination and Ranges
- **Signed indexing**: Negative indices count from end (`-1` = latest, `-10` = last 10)
- **Range queries**: `from`/`to` parameters with optional `count` limit
- **Efficient pagination**: 1,000 items per page with proper start/end calculation
**Target Use Cases:**
### Output Formatting
- **JSON**: Single values, arrays, or matrices with automatic structure detection
- **CSV/TSV**: Delimiter-separated values with headers for spreadsheet use
- **Markdown**: Tables formatted for documentation and display
- **Schema Support**: JSON Schema generation for API documentation
- REST API backends requiring flexible data queries
- Data export tools supporting multiple output formats
- Interactive applications with user-friendly error messaging
- Research platforms requiring structured data access
## Usage
## Installation
### Basic Query Setup
```toml
cargo add brk_interface
```
## Quick Start
```rust
use brk_interface::{Interface, Params, ParamsOpt, Index, Format};
use brk_interface::{Interface, Params, Index};
use brk_indexer::Indexer;
use brk_computer::Computer;
// Load data sources
let indexer = Indexer::forced_import("./brk_data")?;
let computer = Computer::forced_import("./brk_data", &indexer, None)?;
// Create unified interface
// Initialize with indexer and computer
let indexer = Indexer::build(/* config */)?;
let computer = Computer::build(/* config */)?;
let interface = Interface::build(&indexer, &computer);
```
### Single Dataset Queries
```rust
// Get latest block data
// Query data with parameters
let params = Params {
index: Index::Height,
ids: vec!["date", "timestamp", "difficulty"].into(),
rest: ParamsOpt::default()
.set_from(-1) // Latest block
.set_format(Format::JSON),
ids: vec!["height-to-blockhash".to_string()].into(),
from: Some(800000),
to: Some(800100),
format: Some(Format::JSON),
..Default::default()
};
let result = interface.search_and_format(params)?;
println!("{}", result);
// Search and format results
let output = interface.search_and_format(params)?;
println!("{}", output);
```
### Range Queries
## API Overview
### Core Types
- **`Interface<'a>`**: Main query interface coordinating indexer and computer access
- **`Params`**: Query parameters including index, IDs, range, and formatting options
- **`Index`**: Enumeration of available data indexes (Height, Date, Address, etc.)
- **`Format`**: Output format specification (JSON, CSV, TSV, MD)
- **`Output`**: Formatted query results with multiple value types
### Key Methods
**`Interface::build(indexer: &Indexer, computer: &Computer) -> Self`**
Creates interface instance with references to data sources.
**`search(&self, params: &Params) -> Result<Vec<(String, &&dyn AnyCollectableVec)>>`**
Searches for vectors matching the query parameters with intelligent error handling.
**`format(&self, vecs: Vec<...>, params: &ParamsOpt) -> Result<Output>`**
Formats search results according to specified output format and range parameters.
**`search_and_format(&self, params: Params) -> Result<Output>`**
Combined search and formatting operation for single-call data retrieval.
### Query Parameters
**Core Parameters:**
- `index`: Data index to query (height, date, address, etc.)
- `ids`: Vector IDs to retrieve from the specified index
- `from`/`to`: Optional range filtering (inclusive start, exclusive end)
- `format`: Output format (defaults to JSON)
**Pagination Parameters:**
- `offset`: Number of entries to skip
- `limit`: Maximum entries to return
## Examples
### Basic Data Query
```rust
// Get price data for last 30 days
let params = Params {
index: Index::DateIndex,
ids: vec!["price_usd", "price_usd_high", "price_usd_low"].into(),
rest: ParamsOpt::default()
.set_from(-30) // Last 30 days
.set_count(30)
.set_format(Format::CSV),
};
use brk_interface::{Interface, Params, Index, Format};
let csv_data = interface.search_and_format(params)?;
```
let interface = Interface::build(&indexer, &computer);
### Multiple Datasets
```rust
// Get comprehensive block statistics
// Query block heights to hashes
let params = Params {
index: Index::Height,
ids: vec!["size", "weight", "tx_count", "fee_total"].into(),
rest: ParamsOpt::default()
.set_from(800_000) // Starting from block 800,000
.set_to(800_100) // Up to block 800,100
.set_format(Format::TSV),
ids: vec!["height-to-blockhash".to_string()].into(),
from: Some(750000),
to: Some(750010),
format: Some(Format::JSON),
..Default::default()
};
let tsv_data = interface.search_and_format(params)?;
```
### Flexible ID Specification
```rust
// Different ways to specify dataset IDs
let params = Params {
index: Index::DateIndex,
ids: vec!["price_usd,volume_usd,market_cap"].into(), // Comma-separated
// OR: ids: vec!["price_usd volume_usd market_cap"].into(), // Space-separated
// OR: ids: vec!["price_usd", "volume_usd", "market_cap"].into(), // Array
rest: ParamsOpt::default()
.set_format(Format::JSON),
};
```
### Advanced Queries with Pagination
```rust
// Query with custom pagination
let params = Params {
index: Index::MonthIndex,
ids: vec!["supply_total", "supply_active"].into(),
rest: ParamsOpt::default()
.set_from(0) // From beginning
.set_count(50) // Max 50 results
.set_format(Format::Markdown),
};
let markdown_table = interface.search_and_format(params)?;
```
## API Methods
### Core Query Methods
```rust
// Combined search and format
let result = interface.search_and_format(params)?;
// Separate search and format
let vecs = interface.search(params)?;
let formatted = interface.format(vecs, params.rest.format.unwrap_or_default())?;
// Get metadata
let current_height = interface.get_height();
let available_datasets = interface.get_vecids(None)?; // Paginated
let available_indices = interface.get_indexes();
```
### Working with Results
```rust
use brk_interface::{Value, Output};
// Handle different output types
match interface.search_and_format(params)? {
Output::Json(Value::Single(val)) => println!("Single value: {}", val),
Output::Json(Value::List(arr)) => println!("Array with {} items", arr.len()),
Output::Json(Value::Matrix(matrix)) => println!("Matrix: {}x{}", matrix.len(), matrix[0].len()),
Output::Delimited(csv_data) => println!("CSV/TSV data:\n{}", csv_data),
Output::Markdown(table) => println!("Markdown table:\n{}", table),
Output::Json(value) => println!("{}", serde_json::to_string_pretty(&value)?),
_ => unreachable!(),
}
```
## Available Indices
### Time-based Indices
- `Height` - Block height (0, 1, 2, ...)
- `DateIndex` - Days since Bitcoin genesis
- `WeekIndex`, `MonthIndex`, `QuarterIndex`, `YearIndex`, `DecadeIndex`
- `HalvingEpoch`, `DifficultyEpoch` - Bitcoin-specific epochs
### Bitcoin-specific Indices
- Address types: `P2PKHAddressIndex`, `P2SHAddressIndex`, `P2WPKHAddressIndex`, etc.
- Output types: `OutputType` classifications
- Transaction types: `TxIndex`, `InputIndex`, `OutputIndex`
## Parameter Reference
### CSV Export with Range Query
```rust
pub struct Params {
pub index: Index, // Time dimension for query
pub ids: MaybeIds, // Dataset identifiers
pub rest: ParamsOpt, // Optional parameters
}
use brk_interface::{Interface, Params, Index, Format};
pub struct ParamsOpt {
pub from: Option<i64>, // Starting index (negative = from end)
pub to: Option<i64>, // Ending index (exclusive)
pub count: Option<usize>, // Maximum results
pub format: Option<Format>, // Output format
// Export price data as CSV
let params = Params {
index: Index::Date,
ids: vec!["dateindex-to-price-close".to_string()].into(),
from: Some(0), // From genesis
to: Some(5000), // First ~13 years
format: Some(Format::CSV),
..Default::default()
};
match interface.search_and_format(params)? {
Output::CSV(csv_text) => {
std::fs::write("bitcoin_prices.csv", csv_text)?;
println!("Price data exported to bitcoin_prices.csv");
},
_ => unreachable!(),
}
```
## Output Formats
- **JSON**: Structured data (single values, arrays, matrices)
- **CSV**: Comma-separated values with headers
- **TSV**: Tab-separated values with headers
- **Markdown**: Formatted tables for documentation
## Performance Features
- **Zero-copy operations**: Uses references and lifetimes to avoid cloning
- **Memory mapping**: Leverages vecdb's memory-mapped storage
- **Lazy evaluation**: Only processes data when formatting is requested
- **Efficient pagination**: Smart bounds calculation and range queries
## Schema Support
The interface provides JSON Schema support for API documentation:
### Multi-Vector Query with Markdown Table
```rust
use schemars::schema_for;
use brk_interface::{Interface, Params, Index, Format};
let schema = schema_for!(Params);
// Use schema for API documentation
// Query multiple vectors and format as table
let params = Params {
index: Index::Height,
ids: vec![
"height-to-blockhash".to_string(),
"height-to-timestamp".to_string(),
"height-to-difficulty".to_string()
].into(),
from: Some(800000),
to: Some(800005),
format: Some(Format::MD),
..Default::default()
};
match interface.search_and_format(params)? {
Output::MD(table) => println!("{}", table),
_ => unreachable!(),
}
```
## Dependencies
### Intelligent Error Handling
- `brk_indexer` - Indexed blockchain data source
- `brk_computer` - Computed analytics data source
- `vecdb` - Vector database with collection traits
- `serde` - Serialization/deserialization support
- `schemars` - JSON Schema generation
```rust
use brk_interface::{Interface, Params, Index};
// Query with typo in vector ID
let params = Params {
index: Index::Height,
ids: vec!["height-to-blockhas".to_string()].into(), // Typo: "blockhas"
..Default::default()
};
// Interface provides helpful error with suggestions
match interface.search(&params) {
Err(error) => {
println!("{}", error);
// Output: No vec named "height-to-blockhas" indexed by "height" found.
// Maybe you meant one of the following: ["height-to-blockhash"] ?
},
Ok(_) => unreachable!(),
}
```
## Architecture
### Data Source Integration
The interface acts as a bridge between:
- **Indexer**: Raw blockchain data vectors (blocks, transactions, addresses)
- **Computer**: Computed analytics vectors (prices, statistics, aggregations)
- **Unified Access**: Single query interface for both data sources
### Search Implementation
1. **Parameter Validation**: Validates index existence and parameter consistency
2. **Vector Resolution**: Maps vector IDs to actual data structures
3. **Fuzzy Matching**: Provides suggestions for mistyped vector names
4. **Error Caching**: Caches error messages to avoid repeated expensive operations
### Output Formatting
**JSON Output:**
- Single value: Direct value serialization
- List: Array of values
- Matrix: Array of arrays for multi-vector queries
**Tabular Output (CSV/TSV/MD):**
- Column headers from vector IDs
- Row-wise data iteration with proper escaping
- Markdown tables use `tabled` crate formatting
### Caching Strategy
- **Error Message Caching**: 1000-entry LRU cache for error messages
- **Search Result Caching**: Upstream caching in server/client layers
- **Static Data Caching**: Index and vector metadata cached during initialization
## Configuration
### Index Types
Available indexes include:
- `Height`: Block height-based indexing
- `Date`: Calendar date indexing
- `Address`: Bitcoin address indexing
- `Transaction`: Transaction hash indexing
- Custom indexes from computer analytics
### Format Options
- **JSON**: Structured data with nested objects/arrays
- **CSV**: Comma-separated values with proper escaping
- **TSV**: Tab-separated values for import compatibility
- **MD**: Markdown tables for documentation and reports
## Code Analysis Summary
**Main Structure**: `Interface` struct coordinating between `Indexer` and `Computer` data sources \
**Query System**: Parameter-driven search with `Params` struct supporting range queries and formatting options \
**Error Handling**: Intelligent fuzzy matching with cached error messages and helpful suggestions \
**Output Formats**: Multi-format support (JSON, CSV, TSV, Markdown) with proper data serialization \
**Caching**: `quick_cache` integration for error messages and expensive operations \
**Search Logic**: `nucleo-matcher` fuzzy search for user-friendly vector name resolution \
**Architecture**: Abstraction layer providing unified access to heterogeneous Bitcoin data sources
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_logger/README.md
+167 -147
View File
@@ -1,41 +1,36 @@
# brk_logger
**Logging utilities with colored console output and file logging**
Colored console logging with optional file output for Bitcoin Research Kit applications.
`brk_logger` provides a thin wrapper around `env_logger` with BRK-specific defaults, colored console output, and optional file logging. It's designed to provide clear, readable logs for Bitcoin data processing operations.
[![Crates.io](https://img.shields.io/crates/v/brk_logger.svg)](https://crates.io/crates/brk_logger)
[![Documentation](https://docs.rs/brk_logger/badge.svg)](https://docs.rs/brk_logger)
## What it provides
## Overview
- **Colored Console Output**: Level-based color coding for easy visual parsing
- **File Logging**: Optional log file output with automatic rotation
- **Sensible Defaults**: Pre-configured log levels for Bitcoin and dependency crates
- **Timestamp Formatting**: Human-readable timestamps with system timezone
This crate provides a thin wrapper around `env_logger` with enhanced formatting, colored output, and optional file logging. Designed specifically for BRK applications, it offers structured logging with timestamps, level-based coloring, and automatic filtering of noisy dependencies.
## Key Features
**Key Features:**
### Console Logging
- **Color-coded levels**: Error (red), Warn (yellow), Info (green), Debug (blue), Trace (cyan)
- **Formatted timestamps**: `YYYY-MM-DD HH:MM:SS` format with dimmed styling
- **Clean output**: Minimal formatting focused on readability
- Level-based color coding for console output (error=red, warn=yellow, info=green, etc.)
- Dual output: colored console logs and optional plain-text file logging
- System timezone-aware timestamps with customizable formatting
- Automatic filtering of verbose dependencies (bitcoin, fjall, tracing, etc.)
- Environment variable configuration support via `RUST_LOG`
### File Logging
- **Optional file output**: Writes to specified file path
- **Automatic cleanup**: Removes existing log file on initialization
- **Append mode**: New log entries appended to file
- **Plain text format**: No colors in file output for better compatibility
**Target Use Cases:**
### Dependency Filtering
Pre-configured to suppress noisy logs from common dependencies:
- `bitcoin=off` - Bitcoin protocol library
- `bitcoincore-rpc=off` - RPC client
- `fjall=off` - Key-value store
- `lsm_tree=off` - LSM tree implementation
- `rolldown=off` - Bundler
- `tracing=off` - Tracing framework
- Bitcoin blockchain analysis applications requiring structured logging
- Development and debugging of BRK components
- Production deployments with file-based log archival
- Applications needing both human-readable and machine-parseable logs
## Usage
## Installation
### Basic Setup (Console Only)
```toml
cargo add brk_logger
```
## Quick Start
```rust
use brk_logger;
@@ -43,150 +38,175 @@ use brk_logger;
// Initialize with console output only
brk_logger::init(None)?;
// Now use standard logging macros
log::info!("BRK starting up");
log::warn!("Bitcoin Core not fully synced");
log::error!("Failed to connect to RPC");
```
### With File Logging
```rust
use std::path::Path;
// Initialize with both console and file output
let log_path = Path::new("~/.brk/brk.log");
// Initialize with file logging
let log_path = std::path::Path::new("application.log");
brk_logger::init(Some(log_path))?;
log::info!("Logs will appear in console and file");
// Use standard log macros
log::info!("Application started");
log::warn!("Configuration issue detected");
log::error!("Failed to process block");
```
### Environment Variable Control
## API Overview
### Core Functions
**`init(path: Option<&Path>) -> io::Result<()>`**
Initializes the logging system with optional file output. Console logging is always enabled with color formatting.
**Re-exported Types:**
- **`OwoColorize`**: Color formatting trait for custom colored output
### Log Formatting
**Console Output Format:**
```
2024-09-16 14:23:45 - INFO Application started successfully
2024-09-16 14:23:46 - WARN Bitcoin RPC connection slow
2024-09-16 14:23:47 - ERROR Failed to parse block data
```
**File Output Format (Plain Text):**
```
2024-09-16 14:23:45 - info Application started successfully
2024-09-16 14:23:46 - warn Bitcoin RPC connection slow
2024-09-16 14:23:47 - error Failed to parse block data
```
### Default Filtering
The logger automatically filters out verbose logs from common dependencies:
- `bitcoin=off` - Bitcoin library internals
- `bitcoincore-rpc=off` - RPC client details
- `fjall=off` - Database engine logs
- `lsm_tree=off` - LSM tree operations
- `tracing=off` - Tracing framework internals
## Examples
### Basic Logging Setup
```rust
use brk_logger;
use log::{info, warn, error};
fn main() -> std::io::Result<()> {
// Initialize logging to console only
brk_logger::init(None)?;
info!("Starting Bitcoin analysis");
warn!("Price feed temporarily unavailable");
error!("Block parsing failed at height 750000");
Ok(())
}
```
### File Logging with Custom Path
```rust
use brk_logger;
use std::path::Path;
fn main() -> std::io::Result<()> {
let log_file = Path::new("/var/log/brk/application.log");
// Initialize with dual output: console + file
brk_logger::init(Some(log_file))?;
log::info!("Application configured with file logging");
// Both console (colored) and file (plain) will receive this log
log::error!("Critical system error occurred");
Ok(())
}
```
### Environment Variable Configuration
```bash
# Set log level via environment variable
export RUST_LOG=debug
export RUST_LOG=info,brk_parser=debug # Override for specific crates
# Set custom log level and targets
export RUST_LOG="debug,brk_indexer=trace,bitcoin=warn"
# Run with custom log level
RUST_LOG=trace brk
# Run application - logger respects RUST_LOG
./my_brk_app
```
### Using Color Utilities
```rust
use brk_logger;
The crate re-exports `OwoColorize` for consistent coloring:
fn main() -> std::io::Result<()> {
// Respects RUST_LOG environment variable
brk_logger::init(None)?;
log::debug!("Debug information (only shown if RUST_LOG=debug)");
log::trace!("Trace information (only for specific modules)");
Ok(())
}
```
### Colored Output Usage
```rust
use brk_logger::OwoColorize;
println!("Success: {}", "Operation completed".green());
println!("Warning: {}", "Low disk space".yellow());
println!("Error: {}", "Connection failed".red());
println!("Info: {}", "Processing block 800000".bright_black());
```
## Log Format
### Console Output
```
2024-12-25 10:30:15 - info Starting BRK indexer
2024-12-25 10:30:16 - warn Bitcoin Core still syncing (99.8% complete)
2024-12-25 10:30:45 - info Indexed block 900000 (1.2M transactions)
2024-12-25 10:30:46 - error Connection to RPC failed, retrying...
```
### File Output
```
2024-12-25 10:30:15 - info Starting BRK indexer
2024-12-25 10:30:16 - warn Bitcoin Core still syncing (99.8% complete)
2024-12-25 10:30:45 - info Indexed block 900000 (1.2M transactions)
2024-12-25 10:30:46 - error Connection to RPC failed, retrying...
```
## Default Log Levels
The logger uses these default settings:
- **Default level**: `info` - Shows important operational information
- **Suppressed crates**: Dependencies that produce excessive output are set to `off`
- **Override capability**: Can be overridden via `RUST_LOG` environment variable
### Common Log Level Settings
```bash
# Minimal output (errors and warnings only)
RUST_LOG=warn
# Standard output (recommended)
RUST_LOG=info
# Verbose output (for debugging)
RUST_LOG=debug
# Maximum output (for development)
RUST_LOG=trace
# Mixed levels (info by default, debug for specific crates)
RUST_LOG=info,brk_indexer=debug,brk_computer=trace
```
## Integration Examples
### In BRK CLI
```rust
use brk_logger;
use log::info;
fn main() -> Result<()> {
// Initialize logging early in main
brk_logger::init(Some(Path::new("~/.brk/brk.log")))?;
info!("BRK CLI starting");
// ... rest of application
Ok(())
fn print_status() {
println!("Status: {}", "Connected".green());
println!("Height: {}", "800000".bright_blue());
println!("Error: {}", "Connection failed".red());
}
```
### In Custom Applications
## Architecture
```rust
use brk_logger::{self, OwoColorize};
use log::{info, warn, error};
### Logging Pipeline
fn setup_logging() -> std::io::Result<()> {
// Console only for development
brk_logger::init(None)?;
info!("Application initialized");
Ok(())
}
1. **Initialization**: `init()` configures `env_logger` with custom formatter
2. **Environment**: Reads `RUST_LOG` or uses default filter configuration
3. **Formatting**: Applies timestamp, level formatting, and color coding
4. **Dual Output**: Writes colored logs to console, plain logs to file (if configured)
fn process_data() {
info!("Processing Bitcoin data...");
// Use color utilities for progress
println!("Progress: {}", "50%".green());
warn!("Large memory usage detected");
error!("Critical error: {}", "Database connection lost".red());
}
```
### Color Scheme
## Performance Considerations
- **ERROR**: Red text for critical failures
- **WARN**: Yellow text for warnings and issues
- **INFO**: Green text for normal operations
- **DEBUG**: Blue text for debugging information
- **TRACE**: Cyan text for detailed tracing
- **Timestamps**: Bright black (gray) for visual hierarchy
- **Minimal overhead**: Lightweight wrapper around `env_logger`
- **Lazy evaluation**: Log messages only formatted when level is enabled
- **File I/O**: Asynchronous file writing doesn't block main thread
- **Memory usage**: No buffering, logs written immediately
### File Handling
## Dependencies
- **File Creation**: Creates log file if it doesn't exist
- **Append Mode**: Appends to existing files without truncation
- **Error Resilience**: Console logging continues even if file write fails
- **File Rotation**: Manual - application responsible for log rotation
- `env_logger` - Core logging implementation
- `owo_colors` - Terminal color support
- `jiff` - Modern date/time handling for timestamps
### Filtering Strategy
Default filter prioritizes BRK application logs while suppressing:
- Verbose dependency logging that clutters output
- Internal library debug information not relevant to users
- Framework-level tracing that doesn't aid in debugging BRK logic
## Code Analysis Summary
**Main Function**: `init()` function that configures `env_logger` with custom formatting and dual output \
**Dependencies**: Built on `env_logger` for filtering, `jiff` for timestamps, `owo-colors` for terminal colors \
**Output Streams**: Simultaneous console (colored) and file (plain text) logging with different formatting \
**Timestamp Format**: System timezone-aware formatting using `%Y-%m-%d %H:%M:%S` pattern \
**Color Implementation**: Level-based color mapping with `owo-colors` for terminal output \
**Filter Configuration**: Predefined filter string that suppresses verbose dependencies while enabling BRK logs \
**Architecture**: Thin wrapper pattern that enhances `env_logger` with BRK-specific formatting and behavior
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_mcp/README.md
+241 -191
View File
@@ -1,223 +1,273 @@
# brk_mcp
**Model Context Protocol (MCP) bridge for LLM integration with BRK**
Model Context Protocol bridge enabling LLM access to Bitcoin Research Kit data.
`brk_mcp` provides a Model Context Protocol server that enables Large Language Models (LLMs) to access Bitcoin blockchain data through BRK's interface layer. It implements the MCP specification to expose BRK's analytics capabilities as tools that LLMs can call.
[![Crates.io](https://img.shields.io/crates/v/brk_mcp.svg)](https://crates.io/crates/brk_mcp)
[![Documentation](https://docs.rs/brk_mcp/badge.svg)](https://docs.rs/brk_mcp)
Try it out: [https://bitview.space/mcp](https://bitview.space/mcp)
## Overview
## What it provides
This crate provides a Model Context Protocol (MCP) server implementation that exposes Bitcoin blockchain analytics as tools for Large Language Models. Built on the `brk_rmcp` library, it creates a standards-compliant bridge allowing LLMs to query Bitcoin data through structured tool calls with type-safe parameters and JSON responses.
- **MCP Server Implementation**: Standards-compliant Model Context Protocol server
- **Bitcoin Data Tools**: LLM-accessible tools for querying blockchain analytics
- **Type-Safe Parameters**: Structured tool parameters with validation
- **Multi-Format Output**: JSON responses for LLM consumption
- **BRK Integration**: Direct access to all indexed and computed datasets
**Key Features:**
## Available MCP Tools
- Model Context Protocol server with tools capability
- 10 specialized tools for Bitcoin data discovery and querying
- Type-safe parameter validation with structured error handling
- Stateless operation for scalability and load balancing
- Direct integration with BRK interface layer for real-time data access
- Paginated responses for large datasets (up to 1,000 entries per page)
- Multi-format output support through underlying interface layer
Based on the actual implementation, the following tools are available:
**Target Use Cases:**
### Metadata Tools
- LLM-powered Bitcoin analytics and research applications
- Conversational interfaces for blockchain data exploration
- AI-driven financial analysis requiring Bitcoin metrics
- Automated research tools with natural language queries
#### `get_index_count`
Get the count of all existing indexes.
## Installation
**Parameters:** None
**Returns:** Number of available time indices
#### `get_vecid_count`
Get the count of all existing vector IDs.
**Parameters:** None
**Returns:** Number of available dataset identifiers
#### `get_vec_count`
Get the count of all existing vectors (sum of supported indexes for each vector ID).
**Parameters:** None
**Returns:** Total number of vector/index combinations
#### `get_indexes`
Get the list of all existing indexes.
**Parameters:** None
**Returns:** Array of available index names (height, date, week, month, etc.)
#### `get_accepted_indexes`
Get an object with all existing indexes as keys and their accepted variants as values.
**Parameters:** None
**Returns:** Object mapping indexes to their accepted variant names
### Discovery Tools
#### `get_vecids`
Get a paginated list of all existing vector IDs.
**Parameters:**
- `page` (optional number): Page number (default: 0, up to 1,000 results per page)
**Returns:** Array of dataset identifiers
#### `get_index_to_vecids`
Get a paginated list of all vector IDs which support a given index.
**Parameters:**
- `index` (string): Index name to query
- `page` (optional number): Page number (default: 0)
**Returns:** Array of vector IDs that support the specified index
#### `get_vecid_to_indexes`
Get a list of all indexes supported by a given vector ID.
**Parameters:**
- `id` (string): Vector ID to query
**Returns:** Array of indexes supported by the vector ID (empty if ID doesn't exist)
### Data Query Tool
#### `get_vecs`
Get one or multiple vectors depending on given parameters.
**Parameters:**
- `index` (string): Time dimension (height, date, week, month, etc.)
- `ids` (string): Dataset identifiers (comma or space separated)
- `from` (optional i64): Start index (negative = from end)
- `to` (optional i64): End index (exclusive)
- `count` (optional usize): Maximum results
- `format` (optional string): Output format (json, csv, tsv, md)
**Response format depends on parameters:**
- **Single value**: One vector, one result (e.g., `from=-1`)
- **Array**: One vector, multiple results (e.g., `from=-100&count=100`)
- **Matrix**: Multiple vectors (always matrix, even for single results)
### System Tool
#### `get_version`
Get the running version of the Bitcoin Research Kit.
**Parameters:** None
**Returns:** Version string (e.g., "v0.0.88")
## Usage Examples
### Discovery Workflow
```
1. Call get_indexes to see available time dimensions
2. Call get_vecids to see available datasets (paginated)
3. Call get_index_to_vecids to find datasets for specific timeframes
4. Call get_vecs to query actual data
```toml
cargo add brk_mcp
```
### Basic Data Query
```json
// Get latest Bitcoin price
{
"tool": "get_vecs",
"parameters": {
"index": "date",
"ids": "close",
"from": -1
}
}
```
### Multi-Dataset Query
```json
// Get last 30 days of OHLC data
{
"tool": "get_vecs",
"parameters": {
"index": "date",
"ids": "open,high,low,close",
"from": -30,
"format": "csv"
}
}
```
### Metadata Exploration
```json
// Discover what's available
{
"tool": "get_accepted_indexes"
}
// Find datasets for weekly analysis
{
"tool": "get_index_to_vecids",
"parameters": {
"index": "week"
}
}
```
## Server Configuration
The MCP server is stateless and integrates with BRK's HTTP server:
## Quick Start
```rust
// In brk_server routes
router.add_mcp_routes(interface, true) // Enable MCP at /mcp endpoint
use brk_mcp::MCP;
use brk_interface::Interface;
use brk_rmcp::{ServerHandler, RoleServer};
// Initialize with static interface reference
let interface: &'static Interface = /* your interface */;
let mcp = MCP::new(interface);
// Use as MCP server handler
let server_info = mcp.get_info();
println!("MCP server ready with {} tools", server_info.capabilities.tools.unwrap().len());
// Individual tool usage
let index_count = mcp.get_index_count().await?;
let vec_count = mcp.get_vec_count().await?;
let version = mcp.get_version().await?;
```
**Server Info:**
- Protocol version: Latest MCP specification
- Capabilities: Tools enabled
- Instructions: Provides context about Bitcoin data access
- Stateless mode: Compatible with load balancers
## API Overview
## Error Handling
### Core Structure
The MCP server provides structured error responses following MCP specification:
- Invalid parameters result in proper MCP error responses
- Tool failures are handled gracefully
- Parameter validation ensures type safety
- **`MCP`**: Main server struct implementing ServerHandler trait
- **Tool Router**: Automatic tool discovery and routing with `#[tool_router]` macro
- **Parameters**: Type-safe tool parameters with validation
- **CallToolResult**: Structured tool responses with success/error handling
## Integration
### Available Tools
### With BRK Server
MCP is exposed at the `/mcp` endpoint when enabled:
**Discovery Tools:**
- `get_index_count()` - Total number of available indexes
- `get_vecid_count()` - Total number of vector identifiers
- `get_vec_count()` - Total vector/index combinations
- `get_indexes()` - List of all available indexes
- `get_accepted_indexes()` - Mapping of indexes to their variants
**Data Access Tools:**
- `get_vecids(pagination)` - Paginated list of vector IDs
- `get_index_to_vecids(index, pagination)` - Vector IDs supporting specific index
- `get_vecid_to_indexes(id)` - Indexes supported by vector ID
- `get_vecs(params)` - Main data query tool with flexible parameters
**System Tools:**
- `get_version()` - Bitcoin Research Kit version information
### Tool Parameters
**Core Query Parameters:**
- `index`: Time dimension (height, date, week, month, etc.)
- `ids`: Dataset identifiers (comma or space separated)
- `from`/`to`: Range filtering (supports negative indexing from end)
- `format`: Output format (json, csv, tsv, md)
**Pagination Parameters:**
- `page`: Page number for paginated results (up to 1,000 entries)
## Examples
### Basic Tool Usage
```rust
use brk_mcp::MCP;
use brk_interface::{Params, PaginationParam};
let mcp = MCP::new(interface);
// Get system information
let version = mcp.get_version().await?;
println!("BRK version: {:?}", version);
// Discover available data
let indexes = mcp.get_indexes().await?;
let vec_count = mcp.get_vec_count().await?;
// Get paginated vector IDs
let pagination = PaginationParam { page: Some(0) };
let vecids = mcp.get_vecids(pagination).await?;
```
POST /mcp
Content-Type: application/json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "get_vecs",
"arguments": {
"index": "height",
"ids": "timestamp,size",
"from": -1
}
}
### Data Querying
```rust
use brk_mcp::MCP;
use brk_interface::Params;
// Query latest Bitcoin price
let params = Params {
index: "date".to_string(),
ids: vec!["dateindex-to-price-close".to_string()].into(),
from: Some(-1),
..Default::default()
};
let result = mcp.get_vecs(params).await?;
match result {
CallToolResult::Success { content, .. } => {
println!("Latest price: {:?}", content);
},
_ => println!("Query failed"),
}
```
### Multi-Vector Analysis
```rust
use brk_interface::Params;
// Get OHLC data for last 30 days
let params = Params {
index: "date".to_string(),
ids: vec![
"dateindex-to-price-open".to_string(),
"dateindex-to-price-high".to_string(),
"dateindex-to-price-low".to_string(),
"dateindex-to-price-close".to_string(),
].into(),
from: Some(-30),
format: Some("csv".to_string()),
..Default::default()
};
let ohlc_data = mcp.get_vecs(params).await?;
```
### Vector Discovery Workflow
```rust
use brk_interface::{PaginatedIndexParam, IdParam};
// 1. Get available indexes
let indexes = mcp.get_indexes().await?;
// 2. Find vectors for specific index
let paginated_index = PaginatedIndexParam {
index: "height".to_string(),
pagination: PaginationParam { page: Some(0) },
};
let height_vectors = mcp.get_index_to_vecids(paginated_index).await?;
// 3. Check what indexes a vector supports
let id_param = IdParam { id: "height-to-blockhash".to_string() };
let supported_indexes = mcp.get_vecid_to_indexes(id_param).await?;
```
## Architecture
### Server Implementation
**MCP Protocol Compliance:**
- Implements `ServerHandler` trait for MCP compatibility
- Provides `ServerInfo` with tools capability enabled
- Uses latest protocol version with proper initialization
- Includes instructional context for LLM understanding
**Tool Registration:**
- `#[tool_router]` macro for automatic tool discovery
- `#[tool]` attribute for individual tool definitions
- Structured parameter types with automatic validation
- Consistent error handling with `McpError` responses
### HTTP Integration
**Axum Router Extension:**
- `MCPRoutes` trait for router integration
- Conditional MCP endpoint mounting based on configuration
- Stateless HTTP service with `LocalSessionManager`
- Nested service mounting at `/mcp` path
**Transport Layer:**
- `StreamableHttpService` for MCP over HTTP
- Configurable server options with stateless mode
- Session management for concurrent connections
- Request context handling for proper MCP operation
### Data Integration
**Interface Layer Access:**
- Direct access to `brk_interface::Interface` for data queries
- Static lifetime requirements for server operation
- Unified access to indexer and computer data sources
- Consistent parameter types across tool boundaries
**Response Formatting:**
- JSON content serialization for all tool responses
- Success/error response wrapping with proper MCP structure
- Logging integration for request tracking and debugging
- Content type handling for different response formats
## Configuration
### Server Information
The MCP server provides comprehensive information to LLMs:
```rust
ServerInfo {
protocol_version: ProtocolVersion::LATEST,
capabilities: ServerCapabilities::builder().enable_tools().build(),
server_info: Implementation::from_build_env(),
instructions: "Context about Bitcoin data access and terminology",
}
```
### LLM Instructions
The server provides context to LLMs:
- Explains Bitcoin data terminology (vectors, indexes, vecids)
- Clarifies that vectors/vecs/arrays/datasets are interchangeable terms
- Describes indexes as timeframes and vecids as dataset names
- Encourages exploration before web browsing
## Dependencies
Built-in instructions explain Bitcoin data concepts:
- `brk_interface` - Data access and formatting layer
- `brk_rmcp` - Rust MCP implementation
- `axum` - HTTP router integration
- `log` - Request logging
- Vectors/vecs/arrays/datasets are interchangeable terms
- Indexes represent timeframes for data organization
- VecIds are dataset names describing their content
- Tool-based exploration before web browsing is encouraged
## Code Analysis Summary
**Main Structure**: `MCP` struct implementing `ServerHandler` with embedded `ToolRouter` for automatic tool discovery \
**Tool Implementation**: 10 specialized tools using `#[tool]` attribute with structured parameters and JSON responses \
**HTTP Integration**: `MCPRoutes` trait extending Axum routers with conditional MCP endpoint mounting \
**Parameter Types**: Type-safe parameter structs from `brk_interface` with automatic validation \
**Error Handling**: Consistent `McpError` responses with proper MCP protocol compliance \
**Transport Layer**: `StreamableHttpService` with stateless configuration for scalable deployment \
**Architecture**: Standards-compliant MCP bridge providing LLM access to comprehensive Bitcoin analytics
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_parser/README.md
+162 -116
View File
@@ -1,153 +1,199 @@
# brk_parser
**High-performance Bitcoin block parser for raw Bitcoin Core block files**
High-performance Bitcoin block parser for raw Bitcoin Core block files with XOR encryption support.
`brk_parser` provides efficient sequential access to Bitcoin Core's raw block files (`blkXXXXX.dat`), delivering blocks in height order with automatic fork filtering and XOR encryption support. Built for blockchain analysis and indexing applications that need complete Bitcoin data access.
[![Crates.io](https://img.shields.io/crates/v/brk_parser.svg)](https://crates.io/crates/brk_parser)
[![Documentation](https://docs.rs/brk_parser/badge.svg)](https://docs.rs/brk_parser)
## What it provides
## Overview
- **Sequential block access**: Blocks delivered in height order (0, 1, 2, ...) regardless of physical file storage
- **Fork filtering**: Automatically excludes orphaned blocks using Bitcoin Core RPC verification
- **XOR encryption support**: Transparently handles XOR-encrypted block files
- **High performance**: Multi-threaded parsing with ~500MB peak memory usage
- **State persistence**: Caches parsing state for fast restarts
This crate provides a multi-threaded Bitcoin block parser that processes raw Bitcoin Core `.dat` files from the blockchain directory. It supports XOR-encoded block data, parallel processing with `rayon`, and maintains chronological ordering through crossbeam channels. The parser integrates with Bitcoin Core RPC to validate block confirmations and handles file metadata tracking for incremental processing.
## Key Features
**Key Features:**
### Performance Optimization
- **Multi-threaded pipeline**: 3-stage processing (file reading, decoding, ordering)
- **Parallel decoding**: Uses rayon for concurrent block deserialization
- **Memory efficient**: Bounded channels prevent memory bloat
- **State caching**: Saves parsing state to avoid re-scanning unchanged files
- Multi-threaded pipeline architecture with crossbeam channels
- XOR decryption support for encrypted block files
- Parallel block decoding with rayon thread pools
- Chronological block ordering with height-based validation
- Bitcoin Core RPC integration for confirmation checking
- File metadata tracking and incremental processing
- Magic byte detection for block boundary identification
### Bitcoin Integration
- **RPC verification**: Uses Bitcoin Core RPC to filter orphaned blocks
- **Confirmation checks**: Only processes blocks with positive confirmations
- **Height ordering**: Ensures sequential delivery regardless of storage order
**Target Use Cases:**
### XOR Encryption Support
- **Transparent decryption**: Automatically handles XOR-encrypted block files
- **Streaming processing**: Applies XOR decryption on-the-fly during parsing
- Bitcoin blockchain analysis tools requiring raw block access
- Historical data processing applications
- Block explorers and analytics platforms
- Research tools needing ordered block iteration
## Usage
## Installation
### Basic Block Parsing
```toml
cargo add brk_parser
```
## Quick Start
```rust
use brk_parser::Parser;
use bitcoincore_rpc::{Client, Auth, RpcApi};
use brk_structs::Height;
use bitcoincore_rpc::{Auth, Client};
use std::path::PathBuf;
// Setup RPC client (must have static lifetime)
// Initialize Bitcoin Core RPC client
let rpc = Box::leak(Box::new(Client::new(
"http://localhost:8332",
Auth::CookieFile(Path::new("~/.bitcoin/.cookie")),
)?));
Auth::None
).unwrap()));
// Create parser
let parser = Parser::new(
Path::new("~/.bitcoin/blocks").to_path_buf(),
Some(Path::new("./output").to_path_buf()),
rpc,
);
// Create parser with blocks directory
let blocks_dir = PathBuf::from("/path/to/bitcoin/blocks");
let outputs_dir = Some(PathBuf::from("./parser_output"));
let parser = Parser::new(blocks_dir, outputs_dir, rpc);
// Parse all blocks sequentially
parser.parse(None, None)
.iter()
.for_each(|(height, block, hash)| {
println!("Block {}: {} ({} txs)", height, hash, block.txdata.len());
});
```
// Parse blocks in height range
let start_height = Some(Height::new(700000));
let end_height = Some(Height::new(700100));
let receiver = parser.parse(start_height, end_height);
### Range Parsing
```rust
// Parse specific height range
let start = Some(Height::new(800_000));
let end = Some(Height::new(800_100));
parser.parse(start, end)
.iter()
.for_each(|(height, block, hash)| {
// Process blocks 800,000 to 800,100
});
```
### Single Block Access
```rust
// Get single block by height
let genesis = parser.get(Height::new(0));
println!("Genesis has {} transactions", genesis.txdata.len());
```
### Real-world Usage Example
```rust
use brk_parser::Parser;
use bitcoin::Block;
fn analyze_blockchain(parser: &Parser) {
let mut total_transactions = 0;
let mut total_outputs = 0;
parser.parse(None, None)
.iter()
.for_each(|(height, block, _hash)| {
total_transactions += block.txdata.len();
total_outputs += block.txdata.iter()
.map(|tx| tx.output.len())
.sum::<usize>();
if height.0 % 10000 == 0 {
println!("Processed {} blocks", height);
}
});
println!("Total transactions: {}", total_transactions);
println!("Total outputs: {}", total_outputs);
// Process blocks as they arrive
for (height, block, block_hash) in receiver.iter() {
println!("Block {}: {} transactions", height, block.txdata.len());
println!("Block hash: {}", block_hash);
}
```
## Output Format
## API Overview
The parser returns tuples for each block:
- `Height`: Block height (sequential: 0, 1, 2, ...)
- `Block`: Complete block data from the `bitcoin` crate
- `BlockHash`: Block's cryptographic hash
### Core Types
## Performance Characteristics
- **`Parser`**: Main parser coordinating multi-threaded block processing
- **`AnyBlock`**: Enum representing different block states (Raw, Decoded, Skipped)
- **`XORBytes`**: XOR key bytes for decrypting block data
- **`XORIndex`**: Circular index for XOR byte application
- **`BlkMetadata`**: Block file metadata including index and modification time
Benchmarked on MacBook Pro M3 Pro:
- **Full blockchain** (0 to 855,000): ~4 minutes
- **Recent blocks** (800,000 to 855,000): ~52 seconds
- **Peak memory usage**: ~500MB
- **Restart performance**: Subsequent runs much faster due to state caching
### Key Methods
## Requirements
**`Parser::new(blocks_dir: PathBuf, outputs_dir: Option<PathBuf>, rpc: &'static Client) -> Self`**
Creates a new parser instance with blockchain directory and RPC client.
- Running Bitcoin Core node with RPC enabled
- Access to Bitcoin Core's `blocks/` directory
- Bitcoin Core versions v25.0 through v29.0 supported
- RPC authentication (cookie file or username/password)
**`parse(&self, start: Option<Height>, end: Option<Height>) -> Receiver<(Height, Block, BlockHash)>`**
Returns a channel receiver that yields blocks in chronological order for the specified height range.
## State Management
### Processing Pipeline
The parser saves parsing state in `{output_dir}/blk_index_to_blk_recap.json` containing:
- Block file indices and maximum heights
- File modification times for change detection
- Restart optimization metadata
The parser implements a three-stage pipeline:
**Note**: Only one parser instance should run at a time as the state file doesn't support concurrent access.
1. **File Reading Stage**: Scans `.dat` files, identifies magic bytes, extracts raw block data
2. **Decoding Stage**: Parallel XOR decryption and Bitcoin block deserialization
3. **Ordering Stage**: RPC validation and chronological ordering by block height
## Dependencies
## Examples
- `bitcoin` - Bitcoin protocol types and block parsing
- `bitcoincore_rpc` - RPC communication with Bitcoin Core
- `crossbeam` - Multi-producer, multi-consumer channels
- `rayon` - Data parallelism for block decoding
- `serde` - State serialization and persistence
### Basic Block Iteration
```rust
use brk_parser::Parser;
let parser = Parser::new(blocks_dir, Some(output_dir), rpc);
// Parse all blocks from height 650000 onwards
let receiver = parser.parse(Some(Height::new(650000)), None);
for (height, block, hash) in receiver.iter() {
println!("Processing block {} with {} transactions",
height, block.txdata.len());
// Process block transactions
for (idx, tx) in block.txdata.iter().enumerate() {
println!(" Tx {}: {}", idx, tx.txid());
}
}
```
### Range-Based Processing
```rust
use brk_parser::Parser;
let parser = Parser::new(blocks_dir, Some(output_dir), rpc);
// Process specific block range
let start = Height::new(600000);
let end = Height::new(600999);
let receiver = parser.parse(Some(start), Some(end));
let mut total_tx_count = 0;
for (height, block, _hash) in receiver.iter() {
total_tx_count += block.txdata.len();
if height == end {
break; // End of range reached
}
}
println!("Processed 1000 blocks with {} total transactions", total_tx_count);
```
### Incremental Processing with Metadata
```rust
use brk_parser::Parser;
let parser = Parser::new(blocks_dir, Some(output_dir), rpc);
// Parser automatically handles file metadata tracking
// Only processes blocks that have been modified since last run
let receiver = parser.parse(None, None); // Process all available blocks
for (height, block, hash) in receiver.iter() {
// Parser ensures blocks are delivered in chronological order
// even when processing multiple .dat files in parallel
if height.as_u32() % 10000 == 0 {
println!("Reached block height {}", height);
}
}
```
## Architecture
### Multi-Threading Design
The parser uses a sophisticated multi-threaded architecture:
- **File Scanner Thread**: Reads raw bytes from `.dat` files and identifies block boundaries
- **Decoder Thread Pool**: Parallel XOR decryption and block deserialization using rayon
- **Ordering Thread**: RPC validation and chronological ordering with future block buffering
### XOR Encryption Support
Bitcoin Core optionally XOR-encrypts block files using an 8-byte key stored in `xor.dat`. The parser:
- Automatically detects XOR encryption presence
- Implements circular XOR index for efficient decryption
- Supports both encrypted and unencrypted block files
### Block File Management
The parser handles Bitcoin Core's block file structure:
- Scans directory for `blk*.dat` files
- Tracks file modification times for incremental processing
- Maintains block height mappings with RPC validation
- Exports processing metadata for resumable operations
## Code Analysis Summary
**Main Type**: `Parser` struct coordinating multi-threaded block processing pipeline \
**Threading**: Three-stage pipeline using crossbeam channels with bounded capacity (50) \
**Parallelization**: rayon-based parallel block decoding with configurable batch sizes \
**XOR Handling**: Custom XORBytes and XORIndex types for efficient encryption/decryption \
**RPC Integration**: Bitcoin Core RPC validation for block confirmation and height mapping \
**File Processing**: Automatic `.dat` file discovery and magic byte boundary detection \
**Architecture**: Producer-consumer pattern with ordered delivery despite parallel processing
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_server/README.md
+197 -177
View File
@@ -1,218 +1,238 @@
# brk_server
**HTTP server providing REST API access to Bitcoin analytics data**
HTTP server providing REST API access to Bitcoin analytics data
`brk_server` provides a high-performance HTTP server that exposes BRK's indexed blockchain data and computed analytics through a comprehensive REST API. It offers multiple output formats, intelligent caching, compression, and optional web interface serving.
[![Crates.io](https://img.shields.io/crates/v/brk_server.svg)](https://crates.io/crates/brk_server)
[![Documentation](https://docs.rs/brk_server/badge.svg)](https://docs.rs/brk_server)
URL: [`https://bitview.space/api`](https://bitview.space/api)
## Overview
## What it provides
This crate provides a high-performance HTTP server built on `axum` that exposes Bitcoin blockchain analytics data through a comprehensive REST API. It integrates with the entire BRK ecosystem, serving data from indexers, computers, and parsers with intelligent caching, compression, and multiple output formats.
- **REST API**: Vector-based data access with flexible querying and pagination
- **Multiple Output Formats**: JSON, CSV, TSV, and Markdown table formatting
- **Performance Features**: ETag caching, compression, and request weight limiting
- **Web Interface**: Optional static file serving for web applications
- **MCP Integration**: Model Context Protocol support for LLM integration
**Key Features:**
## Key Features
- RESTful API for blockchain data queries with flexible filtering
- Multiple output formats: JSON, CSV, TSV, Markdown
- Intelligent caching system with ETags and conditional requests
- HTTP compression (Gzip, Brotli, Deflate, Zstd) for bandwidth efficiency
- Static file serving for web interfaces and documentation
- Bitcoin address and transaction lookup endpoints
- Vector database query interface with pagination
- Health monitoring and status endpoints
### API Capabilities
- **Direct vector access**: Single vector queries with index-to-ID pattern
- **Multi-vector queries**: Query multiple datasets simultaneously
- **Flexible pagination**: Positive/negative indexing with range queries
- **Format negotiation**: Multiple output formats based on use case
**Target Use Cases:**
### Performance Features
- **ETag caching**: Conditional requests with 304 Not Modified responses
- **Compression**: Brotli, Gzip, Zstd, Deflate support with automatic negotiation
- **Request weight limiting**: Protects against oversized queries (max 320,000 weight)
- **In-memory caching**: 10,000-item cache with 50ms guard timeout
- Bitcoin data APIs for applications and research
- Web-based blockchain explorers and analytics dashboards
- Research data export and analysis tools
- Integration with external systems requiring Bitcoin data
### HTTP Features
- **Auto port assignment**: Starts from port 3110, increments if busy
- **CORS enabled**: Cross-origin requests for web clients
- **Tracing middleware**: Colored request/response logging
- **Static file serving**: Optional website hosting
## Installation
## Usage
```toml
cargo add brk_server
```
## Quick Start
```rust
use brk_server::Server;
use brk_interface::Interface;
use std::path::PathBuf;
// Initialize interface with your data sources
let interface = Interface::new(/* your config */);
// Optional static file serving directory
let files_path = Some(PathBuf::from("./web"));
// Create and start server
let server = Server::new(interface, files_path);
// Start server with optional MCP (Model Context Protocol) support
server.serve(true).await?;
```
## API Overview
### Core Endpoints
**Blockchain Queries:**
- `GET /api/address/{address}` - Address information, balance, transaction counts
- `GET /api/tx/{txid}` - Transaction details including version, locktime
- `GET /api/vecs/{variant}` - Vector database queries with filtering
**System Information:**
- `GET /api/vecs/index-count` - Total number of indexes available
- `GET /api/vecs/id-count` - Vector ID statistics
- `GET /api/vecs/indexes` - List of available data indexes
- `GET /health` - Service health status
- `GET /version` - Server version information
### Vector Database API
**Query Interface:**
- `GET /api/vecs/query` - Generic vector query with parameters
- `GET /api/vecs/{variant}?from={start}&to={end}&format={format}` - Range queries
**Supported Parameters:**
- `from` / `to`: Range filtering (height, timestamp, date-based)
- `format`: Output format (json, csv, tsv, md)
- Pagination parameters for large datasets
### Address API Response Format
```json
{
"address": "1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2",
"type": "p2pkh",
"index": 12345,
"chain_stats": {
"funded_txo_sum": 500000000,
"spent_txo_sum": 400000000,
"utxo_count": 5,
"balance": 100000000,
"balance_usd": 4200.5,
"realized_value": 450000000,
"avg_cost_basis": 45000.0
}
}
```
## Examples
### Basic Server Setup
```rust
use brk_server::Server;
use brk_interface::Interface;
use brk_indexer::Indexer;
use brk_computer::Computer;
// Load data sources
let indexer = Indexer::forced_import("./brk_data")?;
let computer = Computer::forced_import("./brk_data", &indexer, None)?;
// Initialize with BRK interface
let interface = Interface::builder()
.with_indexer_path("./data/indexer")
.with_computer_path("./data/computer")
.build()?;
// Create interface and server
let interface = Interface::build(&indexer, &computer);
let server = Server::new(interface, Some("./website".into()));
let server = Server::new(interface, None);
// Start server (with MCP support)
server.serve(true).await?;
// Server automatically finds available port starting from 3110
server.serve(false).await?;
```
### API Access Patterns
#### Single Vector Queries
### Address Balance Lookup
```bash
# Latest 100 price values
curl "https://bitview.space/api/vecs/date-to-close?from=-100"
# Get address information
curl http://localhost:3110/api/address/1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2
# First 50 difficulty values as CSV
curl "https://bitview.space/api/vecs/height-to-difficulty?count=50&format=csv"
# Range from block 800,000 to 800,100
curl "https://bitview.space/api/vecs/height-to-timestamp?from=800000&to=800100"
```
#### Multi-Vector Queries
```bash
# Multiple price metrics for last 30 days
curl "https://bitview.space/api/vecs/query?index=date&ids=open,high,low,close&from=-30&format=csv"
# Block statistics for specific range
curl "https://bitview.space/api/vecs/query?index=height&ids=size,weight,tx_count,fee_sum&from=800000&count=100"
# Weekly analytics as JSON matrix
curl "https://bitview.space/api/vecs/query?index=week&ids=close,difficulty&from=-52"
```
## API Reference
### Vector Metadata Endpoints
| Endpoint | Description |
|----------|-------------|
| `GET /api/vecs/index-count` | Total number of available indexes |
| `GET /api/vecs/id-count` | Total number of vector IDs |
| `GET /api/vecs/vec-count` | Total vectors (all index/ID combinations) |
| `GET /api/vecs/indexes` | List of all available indexes |
| `GET /api/vecs/accepted-indexes` | Mapping of indexes to accepted variants |
| `GET /api/vecs/ids?page=N` | Paginated vector IDs (1000/page) |
| `GET /api/vecs/index-to-ids?index=INDEX&page=N` | IDs supporting given index |
| `GET /api/vecs/id-to-indexes?id=ID` | Indexes supported by given ID |
### Vector Data Access
#### Direct Access Pattern
`GET /api/vecs/{INDEX}-to-{ID}`
```bash
# Examples
curl "/api/vecs/height-to-timestamp"
curl "/api/vecs/date-to-close"
curl "/api/vecs/month-to-supply"
```
#### Multi-Vector Query
`GET /api/vecs/query`
**Required Parameters:**
- `index`: Vector index type (height, date, week, month, etc.)
- `ids`: Comma or space-separated vector IDs
**Optional Parameters:**
- `from` (i64): Start index (negative = from end, default: 0)
- `to` (i64): End index (exclusive, negative = from end)
- `count` (usize): Maximum number of results
- `format`: Output format (`json`, `csv`, `tsv`, `md`)
### Response Types
**Single Value** (one vector, one result):
```json
42.5
```
**Array** (one vector, multiple results):
```json
[42.5, 43.1, 44.2]
```
**Matrix** (multiple vectors):
```json
[
[42.5, 43.1, 44.2],
[1500, 1520, 1480],
[0.05, 0.052, 0.048]
]
```
**CSV Format**:
```csv
index,close,supply,fee_rate
0,42.5,1500,0.05
1,43.1,1520,0.052
2,44.2,1480,0.048
```
### System Endpoints
| Endpoint | Description |
|----------|-------------|
| `GET /version` | Server version information |
| `GET /health` | Health check with timestamp |
| `GET /api` | API documentation redirect |
| `GET /*` | Static file serving (if enabled) |
## Configuration
### Server State
```rust
pub struct AppState {
interface: &'static Interface<'static>, // Data access interface
path: Option<PathBuf>, // Static files path
cache: Arc<Cache<String, Bytes>>, // Response cache
# Response includes balance, transaction counts, USD value
{
"address": "1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2",
"type": "p2pkh",
"chain_stats": {
"balance": 100000000,
"balance_usd": 4200.50,
"utxo_count": 5
}
}
```
### Middleware Stack
### Data Export Queries
- **Compression Layer**: Brotli, Gzip, Zstd, Deflate
- **Response URI Layer**: Adds URI to response extensions for logging
- **Trace Layer**: Request/response logging with colored output
```bash
# Export height-to-price data as CSV
curl "http://localhost:3110/api/vecs/height-to-price?from=800000&to=800100&format=csv" \
-H "Accept-Encoding: gzip"
# Query with caching - subsequent requests return 304 Not Modified
curl "http://localhost:3110/api/vecs/dateindex-to-price-ohlc?from=0&to=1000" \
-H "If-None-Match: \"etag-hash\""
```
### Transaction Details
```bash
# Get transaction information
curl http://localhost:3110/api/tx/abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890
# Response includes version, locktime, and internal indexing
{
"txid": "abcdef...",
"index": 98765,
"version": 2,
"locktime": 0
}
```
## Architecture
### Server Stack
- **HTTP Framework**: `axum` with async/await for high concurrency
- **Compression**: Multi-algorithm support (Gzip, Brotli, Deflate, Zstd)
- **Caching**: `quick_cache` with LRU eviction and ETag validation
- **Tracing**: Request/response logging with latency tracking
- **Static Files**: Optional web interface serving
### Caching Strategy
- **ETag generation**: Based on data content and query parameters
- **Conditional requests**: 304 Not Modified for unchanged data
- **In-memory cache**: 10,000 items with LRU eviction
- **Cache headers**: `Cache-Control: must-revalidate`
The server implements intelligent caching:
## Performance Characteristics
- **ETags**: Generated from data version and query parameters
- **Conditional Requests**: 304 Not Modified responses for unchanged data
- **Memory Cache**: LRU cache with configurable capacity (5,000 entries)
- **Cache Control**: `must-revalidate` headers for data consistency
### Request Weight System
- **Weight calculation**: Based on data range size and complexity
- **Weight limit**: 320,000 units per request
- **Protection**: Prevents oversized queries that could impact performance
### Request Processing
### Compression
- **Automatic negotiation**: Based on `Accept-Encoding` header
- **Multiple algorithms**: Brotli (best), Zstd, Gzip, Deflate
- **Transparent**: Applied automatically to all responses
1. **Route Matching**: Path-based routing to appropriate handlers
2. **Parameter Validation**: Query parameter parsing and validation
3. **Data Retrieval**: Interface calls to indexer/computer components
4. **Caching Logic**: ETag generation and cache lookup
5. **Format Conversion**: JSON/CSV/TSV/MD output formatting
6. **Compression**: Response compression based on Accept-Encoding
7. **Response**: HTTP response with appropriate headers
### Logging
- **Colored output**: Green for 200, red for errors, gray for redirects
- **Request timing**: Latency measurement and display
- **Status tracking**: HTTP status codes with appropriate colors
### Static File Serving
## Dependencies
Optional static file serving supports:
- `axum` - High-performance async web framework
- `tower-http` - HTTP middleware (compression, tracing, CORS)
- `brk_interface` - Data access and formatting layer
- `brk_mcp` - Model Context Protocol routes
- `quick_cache` - Fast in-memory caching
- `tokio` - Async runtime for networking
- Web interface hosting for blockchain explorers
- Documentation and API reference serving
- Asset serving (CSS, JS, images) with proper MIME types
- Directory browsing with index.html fallback
## Configuration
### Environment Variables
The server automatically configures itself but respects:
- Port selection: Starts at 3110, increments if unavailable
- Compression: Enabled by default for all supported algorithms
- CORS: Permissive headers for cross-origin requests
### Memory Management
- Cache size: 5,000 entries by default
- Request weight limits: 65MB maximum per query
- Timeout handling: 50ms cache guard timeout
- Compression: Adaptive based on content type and size
## Code Analysis Summary
**Main Components**: `Server` struct with `AppState` containing interface, cache, and file paths \
**HTTP Framework**: Built on `axum` with middleware for compression, tracing, and CORS \
**API Routes**: Address lookup, transaction details, vector queries, and system information \
**Caching Layer**: `quick_cache` integration with ETag-based conditional requests \
**Data Integration**: Direct interface to BRK indexer, computer, parser, and fetcher components \
**Static Serving**: Optional file serving for web interfaces and documentation \
**Architecture**: Async HTTP server with intelligent caching and multi-format data export capabilities
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_server/src/api/explorer.rs
+58
View File
@@ -1 +1,59 @@
// use lru::LruCache;
// use std::num::NonZeroUsize;
// struct SmartBlkReader {
// // LRU cache of recently accessed files (memory mapped)
// mmap_cache: LruCache<String, memmap2::Mmap>,
// // Fallback to direct file I/O for cache misses
// max_cached_files: usize,
// }
// impl SmartBlkReader {
// fn new(max_cached: usize) -> Self {
// Self {
// mmap_cache: LruCache::new(NonZeroUsize::new(max_cached).unwrap()),
// max_cached_files: max_cached,
// }
// }
// fn get_transaction(
// &mut self,
// file_path: &str,
// offset: u64,
// length: usize,
// ) -> Result<Transaction, Box<dyn std::error::Error>> {
// // Try cache first
// if let Some(mmap) = self.mmap_cache.get(file_path) {
// let tx_data = &mmap[offset as usize..(offset as usize + length)];
// let mut cursor = std::io::Cursor::new(tx_data);
// return Ok(bitcoin::consensus::Decodable::consensus_decode(
// &mut cursor,
// )?);
// }
// // Cache miss - use direct I/O and potentially cache the file
// let mut file = File::open(file_path)?;
// file.seek(SeekFrom::Start(offset))?;
// let mut buffer = vec![0u8; length];
// file.read_exact(&mut buffer)?;
// // Optionally add to cache (based on access patterns)
// if self.should_cache_file(file_path) {
// let file_for_mmap = File::open(file_path)?;
// if let Ok(mmap) = unsafe { memmap2::MmapOptions::new().map(&file_for_mmap) } {
// self.mmap_cache.put(file_path.to_string(), mmap);
// }
// }
// let mut cursor = std::io::Cursor::new(&buffer);
// Ok(bitcoin::consensus::Decodable::consensus_decode(
// &mut cursor,
// )?)
// }
// fn should_cache_file(&self, _file_path: &str) -> bool {
// // Implement logic: recent files, frequently accessed files, etc.
// true
// }
// }
crates/brk_store/README.md
+230 -101
View File
@@ -1,147 +1,276 @@
# brk_store
**Blockchain-aware key-value storage wrapper for Bitcoin data**
High-performance transactional key-value store wrapper around Fjall with blockchain-aware versioning.
`brk_store` is a thin, typed wrapper around the [Fjall](https://github.com/fjall-rs/fjall) embedded key-value store, specifically designed for Bitcoin blockchain data storage. It provides height-based versioning, batch operations, and optimized configurations for Bitcoin data patterns.
[![Crates.io](https://img.shields.io/crates/v/brk_store.svg)](https://crates.io/crates/brk_store)
[![Documentation](https://docs.rs/brk_store/badge.svg)](https://docs.rs/brk_store)
## What it provides
## Overview
- **Blockchain-aware storage**: Built-in height tracking and versioning for Bitcoin block processing
- **Type-safe interface**: Generic over key and value types with automatic serialization
- **Batch operations**: Efficient batched inserts/deletes with transactional support
- **Metadata management**: Automatic version checking and height progression tracking
- **Performance optimization**: Pre-configured for Bitcoin data patterns
This crate provides a type-safe wrapper around the Fjall LSM-tree database engine, specifically designed for Bitcoin blockchain data storage. It offers transactional operations, automatic version management, height-based synchronization, and optimized configuration for blockchain workloads with support for batch operations and efficient range queries.
## Key Features
**Key Features:**
### Height-Based Processing
- **Height tracking**: Each store tracks the last processed blockchain height
- **Conditional operations**: Insert only when blockchain height requires it
- **Automatic versioning**: Combines Fjall version + user version for compatibility
- Transactional key-value storage with ACID guarantees
- Blockchain height-based synchronization and versioning
- Automatic metadata management with version compatibility checking
- Optimized configuration for Bitcoin data patterns (32MB write buffers, 8MB memtables)
- Type-safe generic interface with zero-copy ByteView integration
- Batch operations with deferred commits for performance
- Optional bloom filter configuration for space/speed tradeoffs
### Batch Operations
- **In-memory staging**: Changes staged before atomic commit
- **Transactional safety**: All changes written atomically or none at all
- **Efficient processing**: Minimizes disk I/O through batching
**Target Use Cases:**
### Performance Optimization
- **Optimized configurations**: 32MB write buffers, 8MB memtables for Bitcoin data
- **Configurable bloom filters**: Optional for faster key lookups
- **Memory efficiency**: Reused read transactions, minimal overhead
- Bitcoin blockchain indexing with transactional consistency
- UTXO set management requiring atomic updates
- Address-to-transaction mapping with range queries
- Any Bitcoin data storage requiring versioned, transactional access
## Usage
## Installation
### Basic Operations
```toml
cargo add brk_store
```
## Quick Start
```rust
use brk_store::{Store, open_keyspace};
use brk_structs::{Height, Version};
use std::path::Path;
// Open keyspace
// Open keyspace (database instance)
let keyspace = open_keyspace(Path::new("./data"))?;
// Create typed store
let mut store: Store<String, u64> = Store::import(
// Create typed store for height-to-blockhash mapping
let mut store: Store<Height, BlockHash> = Store::import(
&keyspace,
Path::new("./data"),
"store_name",
Version::ZERO,
Path::new("./data/height_to_hash"),
"height-to-blockhash",
Version::ONE,
Some(true), // Enable bloom filters
)?;
// Conditional insertion based on height
store.insert_if_needed("key1".to_string(), 42u64, Height::new(800_000));
// Insert data (batched in memory)
store.insert_if_needed(
Height::new(750000),
block_hash,
Height::new(750000)
);
// Batch commit changes
store.commit(Height::new(800_000))?;
// Commit transaction to disk
store.commit(Height::new(750000))?;
// Persist to disk
// Query data
if let Some(hash) = store.get(&Height::new(750000))? {
println!("Block hash: {}", hash);
}
```
## API Overview
### Core Types
- **`Store<Key, Value>`**: Generic transactional store with type-safe operations
- **`AnyStore`**: Trait for height-based synchronization and metadata operations
- **`StoreMeta`**: Version and height metadata management
- **`TransactionalKeyspace`**: Fjall keyspace wrapper for database management
### Key Methods
**`Store::import(keyspace, path, name, version, bloom_filters) -> Result<Self>`**
Creates or opens a store with automatic version checking and migration.
**`get(&self, key: &Key) -> Result<Option<Cow<Value>>>`**
Retrieves value by key, checking both pending writes and committed data.
**`insert_if_needed(&mut self, key: Key, value: Value, height: Height)`**
Conditionally inserts data based on blockchain height requirements.
**`commit(&mut self, height: Height) -> Result<()>`**
Atomically commits all pending operations and updates metadata.
### Height-Based Synchronization
The store implements blockchain-aware synchronization:
- **`has(height)`**: Checks if store contains data up to specified height
- **`needs(height)`**: Determines if store requires data for specified height
- **`height()`**: Returns current synchronized height
## Examples
### Basic Key-Value Operations
```rust
use brk_store::{Store, open_keyspace};
use brk_structs::{Height, TxId, Version};
let keyspace = open_keyspace(Path::new("./blockchain_data"))?;
// Create store for transaction index
let mut tx_store: Store<TxId, Height> = Store::import(
&keyspace,
Path::new("./blockchain_data/txid_to_height"),
"txid-to-height",
Version::ONE,
Some(true),
)?;
// Insert transaction mapping
let txid = TxId::from_str("abcdef...")?;
let height = Height::new(800000);
tx_store.insert_if_needed(txid, height, height);
tx_store.commit(height)?;
// Query transaction height
if let Some(tx_height) = tx_store.get(&txid)? {
println!("Transaction {} found at height {}", txid, tx_height);
}
```
### Batch Processing with Height Synchronization
```rust
use brk_store::{Store, AnyStore};
let mut store: Store<Address, AddressData> = Store::import(/* ... */)?;
// Process blocks sequentially
for block_height in 750000..750100 {
let height = Height::new(block_height);
// Skip if already processed
if store.has(height) {
continue;
}
// Process block transactions
for (address, data) in process_block(block_height)? {
store.insert_if_needed(address, data, height);
}
// Commit entire block atomically
store.commit(height)?;
println!("Processed block {}", block_height);
}
// Ensure data is persisted to disk
store.persist()?;
```
### Querying Data
### Version Migration and Reset
```rust
// Get value (checks pending puts first, then disk)
if let Some(value) = store.get(&"key1".to_string())? {
println!("Value: {}", value);
}
use brk_store::{Store, AnyStore};
use brk_structs::Version;
// Check if store needs processing at height
if store.needs(Height::new(800_000)) {
// Process this height
}
```
### Managing Multiple Stores
```rust
use brk_store::AnyStore;
fn process_stores(stores: &mut [Box<dyn AnyStore>]) -> brk_error::Result<()> {
let height = Height::new(800_000);
for store in stores {
if store.needs(height) {
// Process data for this store
store.commit(height)?;
}
}
// Persist all stores
for store in stores {
store.persist()?;
}
Ok(())
}
```
### Store Reset and Reindexing
```rust
// Reset store for complete reindexing
store.reset()?;
// Store automatically recreated on next import
let mut fresh_store: Store<String, u64> = Store::import(
// Open store with new version
let mut store: Store<Height, Data> = Store::import(
&keyspace,
path,
"store_name",
Version::ZERO,
Some(true)
"my-store",
Version::TWO, // Upgraded from Version::ONE
Some(false), // Disable bloom filters for space
)?;
// Check if reset is needed for data consistency
if store.version() != Version::TWO {
println!("Resetting store for version compatibility");
store.reset()?;
}
// Verify store is empty after reset
assert!(store.is_empty()?);
```
## Store Lifecycle
### Iterator-Based Data Access
1. **Import**: Create or open existing store with version checking
2. **Insert**: Add key-value pairs with height-based conditional insertion
3. **Commit**: Write batched changes to disk atomically
4. **Persist**: Force sync all data to storage
5. **Reset**: Clear all data for reindexing if needed
```rust
use brk_store::Store;
## Type Requirements
let store: Store<Height, BlockHash> = Store::import(/* ... */)?;
Keys and values must implement:
- `Debug + Clone + From<ByteView> + Ord` for keys
- `Debug + Clone + From<ByteView>` for values
// Iterate over all key-value pairs
for (height, block_hash) in store.iter() {
println!("Height {}: {}", height, block_hash);
This enables automatic serialization and type-safe storage operations.
// Process in chunks for memory efficiency
if height.as_u32() % 10000 == 0 {
println!("Processed up to height {}", height);
}
}
```
## Version Management
## Architecture
- **Automatic migration**: Detects version mismatches and resets stores automatically
- **File-based metadata**: Stores version and height info for persistence
- **Compatibility checking**: Prevents data corruption from version incompatibilities
### Storage Engine
## Dependencies
Built on Fjall LSM-tree engine with optimizations:
- `fjall` - Embedded key-value store engine
- `byteview` - Zero-copy byte view operations
- `brk_structs` - Bitcoin-aware type system
- `brk_error` - Unified error handling
- **Write Buffers**: 32MB for high-throughput blockchain ingestion
- **Memtables**: 8MB for balanced memory usage
- **Manual Journal Persist**: Explicit control over durability guarantees
- **Bloom Filters**: Configurable for read-heavy vs. space-constrained workloads
### Transaction Model
- **Read Transactions**: Consistent point-in-time snapshots
- **Write Transactions**: ACID-compliant with rollback support
- **Batch Operations**: In-memory accumulation with atomic commits
- **Height Synchronization**: Blockchain-aware conflict resolution
### Version Management
Automatic handling of schema evolution:
1. **Version Detection**: Reads stored version from metadata
2. **Compatibility Check**: Compares with expected version
3. **Migration**: Automatic store reset for incompatible versions
4. **Metadata Update**: Persistent version tracking
### Memory Management
- **Zero-Copy**: ByteView integration for efficient serialization
- **Copy-on-Write**: Cow<Value> for memory-efficient reads
- **Parking Lot**: RwLock for concurrent partition access
- **Deferred Operations**: BTreeMap/BTreeSet for batched writes
## Configuration
### Keyspace Options
```rust
use fjall::Config;
let keyspace = Config::new(path)
.max_write_buffer_size(32 * 1024 * 1024) // 32MB write buffers
.open_transactional()?;
```
### Partition Options
```rust
use fjall::PartitionCreateOptions;
let options = PartitionCreateOptions::default()
.max_memtable_size(8 * 1024 * 1024) // 8MB memtables
.manual_journal_persist(true) // Manual sync control
.bloom_filter_bits(None); // Disable bloom filters
```
## Code Analysis Summary
**Main Structure**: `Store<Key, Value>` generic wrapper around Fjall with typed operations and metadata management \
**Transaction Layer**: Read/write transaction abstraction with deferred batch operations via BTreeMap/BTreeSet \
**Metadata System**: `StoreMeta` for version compatibility and height tracking with automatic migration \
**Height Synchronization**: Blockchain-aware operations with `needs()`, `has()`, and conditional insertion logic \
**Memory Efficiency**: Zero-copy ByteView integration with parking_lot RwLock for concurrent access \
**Storage Engine**: Fjall LSM-tree with optimized configuration for blockchain workloads \
**Architecture**: Type-safe database abstraction with ACID guarantees and blockchain-specific synchronization patterns
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
crates/brk_structs/README.md
+216 -69
View File
@@ -1,105 +1,252 @@
# brk_structs
**Bitcoin-aware type system and data structures for blockchain analysis**
Bitcoin-aware type system and zero-copy data structures for blockchain analysis.
`brk_structs` provides the foundational type system for the Bitcoin Research Kit (BRK). It offers strongly-typed, storage-optimized data structures that encode Bitcoin protocol knowledge and enable efficient blockchain data analysis.
[![Crates.io](https://img.shields.io/crates/v/brk_structs.svg)](https://crates.io/crates/brk_structs)
[![Documentation](https://docs.rs/brk_structs/badge.svg)](https://docs.rs/brk_structs)
## What it provides
## Overview
- **Type Safety**: Distinct wrapper types prevent common mistakes (e.g., confusing block height with transaction index)
- **Storage Optimization**: Zero-copy serialization and optional compression for efficient disk storage
- **Bitcoin Protocol Knowledge**: Built-in understanding of halving epochs, address types, and blockchain constants
- **Comprehensive Time Indexing**: Multiple temporal granularities for time-series analysis
- **Flexible Grouping**: Framework for categorizing and filtering blockchain data
This crate provides a comprehensive type system for Bitcoin blockchain analysis, featuring zero-copy data structures, memory-efficient storage types, and Bitcoin-specific primitives. Built on `zerocopy` and `vecdb`, it offers type-safe representations for blockchain data with optimized serialization and database integration.
## Key Features
**Key Features:**
### Core Blockchain Types
- `Height` - Block heights with Bitcoin-specific arithmetic
- `Txid`/`BlockHash` - Transaction and block identifiers with prefix optimization
- `TxIndex`/`InputIndex`/`OutputIndex` - Component indices with type safety
- `Date`/`Timestamp` - Time representations anchored to Bitcoin genesis block
- Zero-copy data structures with `zerocopy` derives for high-performance serialization
- Bitcoin-specific types for heights, timestamps, addresses, and transaction data
- OHLC (Open, High, Low, Close) price data structures for financial analysis
- Comprehensive address type classification (P2PK, P2PKH, P2SH, P2WPKH, P2WSH, P2TR, etc.)
- Time-based indexing types (Date, DateIndex, WeekIndex, MonthIndex, etc.)
- Memory allocation tracking with `allocative` integration
- Stored primitive wrappers for space-efficient database storage
### Value and Currency Types
- `Sats` - Satoshi amounts with comprehensive arithmetic operations
- `Bitcoin` - BTC amounts with precision handling
- `Dollars`/`Cents` - Fiat currency for price analysis
- OHLC types (`Open<T>`, `High<T>`, `Low<T>`, `Close<T>`) for market data
**Target Use Cases:**
### Address System
- `OutputType` - All Bitcoin script types (P2PKH, P2SH, P2WPKH, P2WSH, P2TR, etc.)
- `AddressBytes` - Type-safe address data extraction
- Address-specific indices for each output type
- `AnyAddressIndex` - Unified address indexing
- Bitcoin blockchain analysis and research tools
- Financial data processing and OHLC calculations
- Database-backed blockchain indexing systems
- Memory-efficient UTXO set management
### Temporal Indexing
- `DateIndex` - Days since Bitcoin genesis
- Time granularities: `WeekIndex`, `MonthIndex`, `QuarterIndex`, `YearIndex`, `DecadeIndex`
- `HalvingEpoch` - Bitcoin halving periods (every 210,000 blocks)
- `DifficultyEpoch` - Difficulty adjustment periods
## Installation
## Usage
```toml
cargo add brk_structs
```
### Basic Types and Conversions
## Quick Start
```rust
use brk_structs::*;
// Type-safe blockchain data
let height = Height::new(800_000);
let epoch = HalvingEpoch::from(height);
let amount = Sats::_1BTC * 2;
// Bitcoin-specific types
let height = Height::new(800000);
let timestamp = Timestamp::from(1684771200u32);
let date = Date::from(timestamp);
let sats = Sats::new(100_000_000); // 1 BTC in satoshis
// Time-based indexing
let date = Date::new(2024, 1, 15);
let month_idx = MonthIndex::from(date);
// Price data structures
let price_cents = Cents::from(Dollars::from(50000.0));
let ohlc = OHLCCents::from((
Open::new(price_cents),
High::new(price_cents * 1.02),
Low::new(price_cents * 0.98),
Close::new(price_cents * 1.01),
));
// Address classification
let output_type = OutputType::P2PKH;
println!("Is spendable: {}", output_type.is_spendable());
println!("Has address: {}", output_type.is_address());
// Time indexing
let date_index = DateIndex::try_from(date)?;
let week_index = WeekIndex::from(date);
```
### Address Handling
## API Overview
### Core Bitcoin Types
- **`Height`**: Block height with overflow-safe arithmetic operations
- **`Timestamp`**: Unix timestamp with Bitcoin genesis epoch support
- **`Date`**: Calendar date with blockchain-specific formatting (YYYYMMDD)
- **`Sats`**: Satoshi amounts with comprehensive arithmetic operations
- **`Bitcoin`**: Floating-point BTC amounts with satoshi conversion
- **`BlockHash`** / **`TxId`**: Cryptographic hash identifiers
### Address and Output Types
- **`OutputType`**: Comprehensive Bitcoin script type classification
- Standard types: P2PK (33/65-byte), P2PKH, P2SH, P2WPKH, P2WSH, P2TR
- Special types: P2MS (multisig), OpReturn, P2A (address), Empty, Unknown
- Type-checking methods: `is_spendable()`, `is_address()`, `is_unspendable()`
- **Address Index Types**: Specialized indexes for each address type
- `P2PKHAddressIndex`, `P2SHAddressIndex`, `P2WPKHAddressIndex`
- `P2WSHAddressIndex`, `P2TRAddressIndex`, etc.
### Financial Data Types
**Price Representations:**
- **`Cents`**: Integer cent representation for precise financial calculations
- **`Dollars`**: Floating-point dollar amounts with cent conversion
- **`OHLCCents`** / **`OHLCDollars`** / **`OHLCSats`**: OHLC data in different denominations
**OHLC Components:**
- **`Open<T>`**, **`High<T>`**, **`Low<T>`**, **`Close<T>`**: Generic price point wrappers
- Support for arithmetic operations and automatic conversions
### Time Indexing System
**Calendar Types:**
- **`DateIndex`**: Days since Bitcoin genesis (2009-01-03)
- **`WeekIndex`**: Week-based indexing for aggregation
- **`MonthIndex`**, **`QuarterIndex`**, **`SemesterIndex`**: Hierarchical time periods
- **`YearIndex`**, **`DecadeIndex`**: Long-term time categorization
**Epoch Types:**
- **`HalvingEpoch`**: Bitcoin halving period classification
- **`DifficultyEpoch`**: Difficulty adjustment epoch tracking
### Storage Types
**Stored Primitives:** Memory-efficient wrappers for database storage
- **`StoredU8`**, **`StoredU16`**, **`StoredU32`**, **`StoredU64`**: Unsigned integers
- **`StoredI16`**: Signed integers
- **`StoredF32`**, **`StoredF64`**: Floating-point numbers
- **`StoredBool`**: Boolean values
- **`StoredString`**: String storage optimization
## Examples
### Block Height Operations
```rust
// Extract address information from Bitcoin scripts
let output_type = OutputType::from(&script);
if output_type.is_address() {
let address_bytes = AddressBytes::try_from((&script, output_type))?;
use brk_structs::Height;
let current_height = Height::new(800000);
let next_height = current_height.incremented();
// Check halving schedule
let blocks_until_halving = current_height.left_before_next_halving();
println!("Blocks until next halving: {}", blocks_until_halving);
// Difficulty adjustment tracking
let blocks_until_adjustment = current_height.left_before_next_diff_adj();
```
### Price Data Processing
```rust
use brk_structs::*;
// Create OHLC data from individual price points
let daily_ohlc = OHLCDollars::from((
Open::from(45000.0),
High::from(47500.0),
Low::from(44000.0),
Close::from(46800.0),
));
// Convert between price denominations
let ohlc_cents: OHLCCents = daily_ohlc.into();
let sats_per_dollar = Sats::_1BTC / daily_ohlc.close;
// Aggregate multiple OHLC periods
let weekly_close = vec![
Close::from(46800.0),
Close::from(48200.0),
Close::from(47100.0),
].iter().sum::<Close<Dollars>>();
```
### Address Type Classification
```rust
use brk_structs::OutputType;
use bitcoin::{Address, Network};
let address_str = "1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2";
let address = Address::from_str(address_str)?.assume_checked();
let output_type = OutputType::from(&address);
match output_type {
OutputType::P2PKH => println!("Legacy address"),
OutputType::P2WPKH => println!("Native SegWit address"),
OutputType::P2SH => println!("Script hash address"),
OutputType::P2TR => println!("Taproot address"),
_ => println!("Other address type: {:?}", output_type),
}
// Check spend conditions
if output_type.is_spendable() && output_type.is_address() {
println!("Spendable address-based output");
}
```
### Zero-Copy Storage
### Time-Based Indexing
```rust
// Efficient serialization without copying
let height_bytes = height.as_bytes();
let recovered_height = Height::read_from_bytes(height_bytes)?;
use brk_structs::*;
let date = Date::new(2023, 6, 15);
let date_index = DateIndex::try_from(date)?;
// Convert to different time granularities
let week_index = WeekIndex::from(date);
let month_index = MonthIndex::from(date);
let quarter_index = QuarterIndex::from(date);
// Calculate completion percentage for current day
let completion_pct = date.completion();
println!("Day {}% complete", completion_pct * 100.0);
```
### Data Grouping and Filtering
## Architecture
```rust
// Flexible filtering for analytics
let utxo_groups = UTXOGroups {
all: total_utxos,
age_range: ByAgeRange::new(age_filtered_utxos),
epoch: ByEpoch::new(epoch_filtered_utxos),
// ... more groupings
};
```
### Zero-Copy Design
## Storage Optimization
All major types implement `zerocopy` traits (`FromBytes`, `IntoBytes`, `KnownLayout`) enabling:
All types implement zero-copy serialization traits:
- **Zero overhead**: Direct memory mapping without serialization costs
- **Optional compression**: Configurable zstd compression for space efficiency
- **Type safety**: Compile-time guarantees about data layout and endianness
- Direct memory mapping from serialized data
- Efficient network protocol handling
- High-performance database operations
- Memory layout guarantees for cross-platform compatibility
## Dependencies
### Type Safety
- `bitcoin` - Bitcoin protocol types and script parsing
- `vecdb` - Vector database storage traits
- `zerocopy` - Zero-copy serialization framework
- `serde` - JSON serialization support
- `jiff` - Modern date/time handling
The type system enforces Bitcoin domain constraints:
- `Height` prevents integer overflow in block calculations
- `Timestamp` handles Unix epoch edge cases
- `OutputType` enum covers all Bitcoin script patterns
- Address types ensure correct hash length validation
### Memory Efficiency
Storage types provide space optimization:
- Stored primitives reduce allocation overhead
- OHLC structures support both heap and stack allocation
- Index types enable efficient range queries
- Hash types use fixed-size arrays for predictable memory usage
## Code Analysis Summary
**Main Categories**: 70+ struct types across Bitcoin primitives, financial data, time indexing, and storage optimization \
**Zero-Copy Support**: Comprehensive `zerocopy` implementation for all major types \
**Type Safety**: Bitcoin domain-specific constraints with overflow protection and validation \
**Financial Types**: Multi-denomination OHLC support with automatic conversions \
**Address System**: Complete Bitcoin script type classification with 280 enum variants \
**Time Indexing**: Hierarchical calendar system from daily to decade-level granularity \
**Storage Integration**: `vecdb::StoredCompressed` traits for efficient database operations \
**Architecture**: Type-driven design prioritizing memory efficiency and domain correctness
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_
docs/README.md
+151 -50
View File
@@ -1,4 +1,4 @@
# Bitcoin Research Kit
# Bitcoin Research Kit (BRK)
<p align="left">
<a href="https://github.com/bitcoinresearchkit/brk">
@@ -28,70 +28,171 @@
</a>
</p>
The Bitcoin Research Kit is a high-performance toolchain designed to parse, index, compute, serve and visualize data from a Bitcoin node, enabling users to gain deeper insights into the Bitcoin network.
> **The open-source alternative to expensive Bitcoin analytics platforms**
> Parse, index, analyze, and visualize Bitcoin blockchain data with unparalleled performance and zero restrictions.
In other words it's an alternative to [Glassnode](https://glassnode.com), [mempool.space](https://mempool.space/) (soon) and [electrs](https://github.com/romanz/electrs) (soon) all in one package with a particular focus on simplicity and ease of use.
---
The toolkit can be used in various ways to accommodate as many needs as possible:
## What is BRK?
- **[Website](https://bitview.space)** \
Everyone is welcome to visit the official instance and showcase of the suite's capabilities. \
It has a wide range of functionalities including charts, tables and simulations which you can visit for free and without the need for an account.
- **[API](https://github.com/bitcoinresearchkit/brk/tree/main/crates/brk_server#brk-server)** \
Researchers and developers are free to use BRK's public API with ![Datasets variant count](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fbitcoinresearchkit.org%2Fapi%2Fvecs%2Fvec-count&query=%24&style=flat&label=%20&color=white) dataset variants at their disposal. \
Just like the website, it's entirely free, with no authentication or rate-limiting.
- **[AI](https://github.com/bitcoinresearchkit/brk/blob/main/crates/brk_mcp/README.md#brk-mcp)** \
LLMs have to possibility to connect to BRK's backend through a [MCP](https://modelcontextprotocol.io/introduction). \
It will give them access to the same tools as the API, with no restrictions, and allow you to have your very own data analysts.
- **[CLI](https://crates.io/crates/brk_cli)** \
Node runners are strongly encouraged to try out and self-host their own instance using BRK's command line interface. \
The CLI has multiple cogs available for users to tweak to adapt to all situations with even the possibility for web developers to create their own custom website which could later on be added as an alternative front-end.
- **[Crates](https://crates.io/crates/brk)** \
Rust developers have access to a wide range crates, each built upon one another with its own specific purpose, enabling independent use and offering great flexibility.
PRs are welcome, especially if their goal is to introduce additional datasets.
The Bitcoin Research Kit is a **high-performance, open-source toolchain** that transforms raw Bitcoin blockchain data into actionable insights. Think of it as your complete Bitcoin data analytics stack—combining the power of Glassnode's metrics, mempool.space's real-time data, and electrs's indexing capabilities into one unified, freely accessible platform.
The primary goal of this project is to be fully-featured and accessible for everyone, regardless of their background or financial situation - whether that person is an enthusiast, researcher, miner, analyst, or simply curious.
**Why BRK exists:** Existing Bitcoin analytics platforms are either prohibitively expensive (some costing thousands per month) or severely limited in functionality. Most are closed-source black boxes that contradict Bitcoin's principles of transparency and verifiability. BRK changes that.
In contrast, existing alternatives tend to be either [very costly](https://studio.glassnode.com/pricing) or missing essential features, with the vast majority being closed-source and unverifiable, which fundamentally undermines the principles of Bitcoin.
## Key Features
## Hosting as a service
- **Lightning Fast**: Built in Rust for maximum performance
- **![Datasets variant count](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fbitcoinresearchkit.org%2Fapi%2Fvecs%2Fvec-count&query=%24&style=flat&label=%20&color=white) Dataset Variants**: Comprehensive Bitcoin metrics out of the box
- **Completely Free**: No API limits, no paywalls, no accounts required
- **100% Open Source**: Fully auditable and verifiable
- **Multiple Interfaces**: Web UI, REST API, CLI, AI integration, and Rust crates
- **Self-Hostable**: Run your own instance with full control
- **AI-Ready**: Built-in LLM integration via Model Context Protocol
If you'd like to have your own instance hosted for you please contact [hosting@bitcoinresearchkit.org](mailto:hosting@bitcoinresearchkit.org).
## Who Is This For?
- 2 separate dedicated servers (1 GB/s each) with different ISPs and Cloudflare integration for enhanced performance and optimal availability
- 99.99% SLA
- Configured for speed
- Updates delivered at your convenience
- Direct communication for feature requests and support
- Bitcoin Core or Knots with desired version
- Optional subdomains
- Logo featured in the Readme if desired
| **Researchers** | **Developers** | **Miners** | **Enthusiasts** |
| -------------------------------------------- | ---------------------------------------- | ------------------------------------------------ | -------------------------------------------- |
| Free access to comprehensive Bitcoin metrics | REST API and Rust crates for integration | Mining pool analytics and profitability tracking | Charts, visualizations, and network insights |
| Historical data analysis | High-performance indexing capabilities | Difficulty and hashrate monitoring | Educational blockchain exploration |
| Academic research tools | Custom dataset creation | Fee market analysis | Portfolio and address tracking |
Pricing: `0.01 BTC / month` *or* `0.1 BTC / year`
## Quick Start
## Acknowledgments
### 1. **Try it Online** (Fastest)
Deepest gratitude to the [Open Sats](https://opensats.org/) public charity. Their grant — from December 2024 to the present — has been critical in sustaining this project.
Visit **[bitview.space](https://bitview.space)** - No installation required, full feature access
Heartfelt thanks go out to every donor on [Nostr](https://primal.net/p/npub1jagmm3x39lmwfnrtvxcs9ac7g300y3dusv9lgzhk2e4x5frpxlrqa73v44) and [Geyser.fund](https://geyser.fund/project/brk) whose support has ensured the availability of the [bitcoinresearchkit.org](https://bitcoinresearchkit.org) public instance.
### 2. **Use the API** (Developers)
```bash
# Get latest block height
curl https://bitcoinresearchkit.org/api/vecs?index=height&ids=height&from=-1
# Get Bitcoin price history
curl https://bitcoinresearchkit.org/api/vecs?index=dateindex&ids=price_usd&from=-30&count=30
```
### 3. **Self-Host** (Power Users)
```bash
# Install CLI
cargo install brk
# Run with your Bitcoin node
brk --bitcoindir /data/bitcoin --brkdir /data/brk
```
### 4. **AI Integration** (ChatGPT/Claude)
Connect your AI assistant to BRK's data using our [Model Context Protocol integration](https://github.com/bitcoinresearchkit/brk/blob/main/crates/brk_mcp/README.md).
## Use Cases
**Financial Analysis**
- Track on-chain metrics like transaction volume, active addresses, and HODL waves
- Analyze market cycles with realized cap, MVRV ratios, and spending patterns
- Monitor exchange flows and whale movements
**Mining Operations**
- Difficulty adjustment predictions and mining profitability analysis
- Pool distribution and hashrate monitoring
- Fee market dynamics and transaction prioritization
**Research & Development**
- Lightning Network adoption metrics
- UTXO set analysis and efficiency studies
- Protocol upgrade impact assessment
**Portfolio Management**
- Address and UTXO tracking
- Historical balance analysis
- Privacy and coin selection optimization
## Performance
BRK is designed for speed and efficiency:
- **Block Processing**: Parse entire blockchain in hours, not days
- **Query Performance**: Sub-millisecond response times for most metrics
- **Memory Efficiency**: Optimized data structures minimize RAM usage
- **Storage**: Compressed indexes reduce disk space requirements
## Contributing
Contributions from the Bitcoin community are welcome! Here's how to get involved:
1. **Report Issues**: Found a bug? [Open an issue](https://github.com/bitcoinresearchkit/brk/issues)
2. **Request Features**: Have an idea? We'd love to hear it
3. **Submit PRs**: Especially for new datasets and metrics
4. **Improve Docs**: Help make BRK more accessible
5. **Join Discussion**: [Discord](https://discord.gg/WACpShCB7M) | [Nostr](https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6)
## Crates
- [`brk`](https://crates.io/crates/brk): A wrapper around all other `brk-*` crates
- [`brk_bundler`](https://crates.io/crates/brk_bundler): A thin wrapper around [`rolldown`](https://rolldown.rs/)
- [`brk_cli`](https://crates.io/crates/brk_cli): A command line interface to run a BRK instance
- [`brk_computer`](https://crates.io/crates/brk_computer): A Bitcoin dataset computer built on top of [`brk_indexer`](https://crates.io/crates/brk_indexer)
- [`brk_error`](https://crates.io/crates/brk_error): Errors used throughout BRK
- [`brk_fetcher`](https://crates.io/crates/brk_fetcher): A Bitcoin price fetcher
- [`brk_indexer`](https://crates.io/crates/brk_indexer): A Bitcoin indexer built on top of [`brk_parser`](https://crates.io/crates/brk_parser)
- [`brk_interface`](https://crates.io/crates/brk_interface): An interface to find and format data from BRK
- [`brk_logger`](https://crates.io/crates/brk_logger): A thin wrapper around [`env_logger`](https://crates.io/crates/env_logger)
- [`brk_mcp`](https://crates.io/crates/brk_mcp): A bridge for LLMs to access BRK
- [`brk_parser`](https://crates.io/crates/brk_parser): A very fast Bitcoin block parser and iterator built on top of [`bitcoin-rust`](https://crates.io/crates/bitcoin)
- [`brk_server`](https://crates.io/crates/brk_server): A server with an API for anything from BRK
- [`brk_store`](https://crates.io/crates/brk_store): A thin wrapper around [`fjall`](https://crates.io/crates/fjall)
- [`brk_structs`](https://crates.io/crates/brk_structs): Structs used throughout BRK
| Crate | Purpose |
| --------------------------------------------------------- | ------------------------------------------------ |
| [`brk`](https://crates.io/crates/brk) | Umbrella crate containing all BRK functionality |
| [`brk_cli`](https://crates.io/crates/brk_cli) | Command line interface for running BRK instances |
| [`brk_server`](https://crates.io/crates/brk_server) | REST API server for data access |
| [`brk_mcp`](https://crates.io/crates/brk_mcp) | Model Context Protocol bridge for AI integration |
| [`brk_parser`](https://crates.io/crates/brk_parser) | High-performance Bitcoin block parser |
| [`brk_indexer`](https://crates.io/crates/brk_indexer) | Blockchain data indexing engine |
| [`brk_computer`](https://crates.io/crates/brk_computer) | Bitcoin metrics and dataset computation |
| [`brk_interface`](https://crates.io/crates/brk_interface) | Data formatting and query interface |
| [`brk_fetcher`](https://crates.io/crates/brk_fetcher) | Bitcoin price and market data fetcher |
| [`brk_store`](https://crates.io/crates/brk_store) | Database storage abstraction (fjall wrapper) |
| [`brk_bundler`](https://crates.io/crates/brk_bundler) | Web asset bundling (rolldown wrapper) |
| [`brk_structs`](https://crates.io/crates/brk_structs) | Shared data structures |
| [`brk_error`](https://crates.io/crates/brk_error) | Error handling utilities |
| [`brk_logger`](https://crates.io/crates/brk_logger) | Logging infrastructure |
## Donate
## Professional Hosting
Need a managed BRK instance? We offer professional hosting services:
**What's Included:**
- Dual dedicated servers (1 GB/s) with redundant ISPs
- Cloudflare integration for global performance
- 99.99% uptime SLA
- Automatic updates and maintenance
- Direct support channel
- Custom Bitcoin Core/Knots versions
- Optional branded subdomains
**Pricing:** `0.01 BTC/month` or `0.1 BTC/year`
Contact: [hosting@bitcoinresearchkit.org](mailto:hosting@bitcoinresearchkit.org)
## Acknowledgments
This project is made possible by the generous support of:
- **[Open Sats](https://opensats.org/)**: Our primary grant provider, enabling full-time development since December 2024
- **Community Donors**: Supporters on [Nostr](https://primal.net/p/npub1jagmm3x39lmwfnrtvxcs9ac7g300y3dusv9lgzhk2e4x5frpxlrqa73v44) and Geyser.fund who kept our public instance running before OpenSats
## Support the Project
Help us maintain and improve BRK:
**Bitcoin Address:**
[`bc1q09 8zsm89 m7kgyz e338vf ejhpdt 92ua9p 3peuve`](bitcoin:bc1q098zsm89m7kgyze338vfejhpdt92ua9p3peuve)
**Other Ways to Support:**
- Star this repository
- Share BRK with your network
- Contribute code or documentation
- Join our community discussions
---
<p align="center">
<strong>Built with for the Bitcoin community</strong><br>
<em>Open source • Free forever • No compromises</em>
</p>
docs/TODO.md
+1
View File
@@ -21,6 +21,7 @@
- highest dominance
- consecutive blocks
- max consecutive blocks
- add indexes back to the mempool.space version of pools as we need the correct one for the API
- price
- oracle (https://utxo.live/oracle/UTXOracle.py)
- cohorts
websites/bitview/scripts/options.js
+121 -87
View File
@@ -720,7 +720,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
name,
title: `UTXOs ${title}`,
color,
}),
})
);
const addressesAboveAmount = aboveAmount.map(
@@ -730,7 +730,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
name,
title: `Addresses ${title}`,
color,
}),
})
);
const underAmount = /** @type {const} */ ([
@@ -821,7 +821,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
name,
title: `UTXOs ${title}`,
color,
}),
})
);
const addressesUnderAmount = underAmount.map(
@@ -831,7 +831,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
name,
title: `Addresses ${title}`,
color,
}),
})
);
const amountRanges = /** @type {const} */ ([
@@ -934,7 +934,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
name,
title: `UTXOs ${title}`,
color,
}),
})
);
const addressesAmountRanges = amountRanges.map(
@@ -944,7 +944,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
name,
title: `Addresses ${title}`,
color,
}),
})
);
const type = /** @type {const} */ ([
@@ -1094,10 +1094,9 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
lineStyle,
}) {
return /** @satisfies {FetchedLineSeriesBlueprint} */ ({
key: `constant_${number >= 0 ? number : `minus_${Math.abs(number)}`}`.replace(
".",
"_",
),
key: `constant_${
number >= 0 ? number : `minus_${Math.abs(number)}`
}`.replace(".", "_"),
title: name ?? `${number}`,
unit,
defaultActive,
@@ -1130,7 +1129,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
lastValueVisible: false,
crosshairMarkerVisible: false,
},
}),
})
);
}
@@ -1427,7 +1426,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
options: {
lineStyle: 1,
},
}),
})
)
: []),
],
@@ -1450,7 +1449,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
options: {
lineStyle: 1,
},
}),
})
)
: []),
...(`${key}_ratio_sma` in vecIdToIndexes
@@ -1463,7 +1462,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
options: {
lineStyle: 1,
},
}),
})
)
: []),
createPriceLine({
@@ -1642,7 +1641,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
options: {
lineStyle: 1,
},
}),
})
),
],
bottom: [
@@ -2015,11 +2014,11 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
}),
},
...(list.filter(
({ key }) => `${fixKey(key)}addr_count` in vecIdToIndexes,
({ key }) => `${fixKey(key)}addr_count` in vecIdToIndexes
).length > ("list" in args ? 1 : 0)
? !("list" in args) ||
list.filter(
({ key }) => `${fixKey(key)}empty_addr_count` in vecIdToIndexes,
({ key }) => `${fixKey(key)}empty_addr_count` in vecIdToIndexes
).length <= 1
? [
{
@@ -2061,7 +2060,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
bottom: list
.filter(
({ key }) =>
`${fixKey(key)}addr_count` in vecIdToIndexes,
`${fixKey(key)}addr_count` in vecIdToIndexes
)
.flatMap(({ name, color, key: _key }) => {
const key = fixKey(_key);
@@ -2076,7 +2075,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
},
...(list.filter(
({ key }) =>
`${fixKey(key)}empty_addr_count` in vecIdToIndexes,
`${fixKey(key)}empty_addr_count` in vecIdToIndexes
).length
? [
{
@@ -2086,7 +2085,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
.filter(
({ key }) =>
`${fixKey(key)}empty_addr_count` in
vecIdToIndexes,
vecIdToIndexes
)
.flatMap(({ name, color, key: _key }) => {
const key = fixKey(_key);
@@ -2118,7 +2117,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
key: `${fixKey(key)}realized_price`,
name,
color,
}),
})
),
},
{
@@ -2130,7 +2129,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
key: `${fixKey(key)}realized_price_ratio`,
name,
color,
}),
})
),
createPriceLine({
unit: "ratio",
@@ -2195,7 +2194,9 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
vecIdToIndexes
? [
createBaseSeries({
key: `${fixKey(args.key)}realized_profit_to_loss_ratio`,
key: `${fixKey(
args.key
)}realized_profit_to_loss_ratio`,
name: "proft / loss",
color: colors.yellow,
}),
@@ -2233,7 +2234,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
args.key,
args.key
)}realized_profit_rel_to_realized_cap`,
title: "Profit",
color: colors.green,
@@ -2241,7 +2242,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
args.key,
args.key
)}realized_loss_rel_to_realized_cap`,
title: "Loss",
color: colors.red,
@@ -2273,7 +2274,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
key,
key
)}net_realized_pnl_cumulative_30d_delta`,
title: "cumulative 30d change",
defaultActive: false,
@@ -2281,14 +2282,14 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
key,
key
)}net_realized_pnl_rel_to_realized_cap`,
title: "Raw",
}),
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
key,
key
)}net_realized_pnl_cumulative_30d_delta_rel_to_realized_cap`,
title: "cumulative 30d change",
defaultActive: false,
@@ -2296,7 +2297,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
key,
key
)}net_realized_pnl_cumulative_30d_delta_rel_to_market_cap`,
title: "cumulative 30d change",
}),
@@ -2499,7 +2500,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
key,
key
)}net_realized_pnl_rel_to_realized_cap`,
title: name,
color,
@@ -2570,7 +2571,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
key,
key
)}net_realized_pnl_cumulative_30d_delta`,
title: name,
color,
@@ -2578,7 +2579,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
key,
key
)}net_realized_pnl_cumulative_30d_delta_rel_to_realized_cap`,
title: name,
color,
@@ -2586,7 +2587,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
key,
key
)}net_realized_pnl_cumulative_30d_delta_rel_to_market_cap`,
title: name,
color,
@@ -2649,7 +2650,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
title: name,
color,
}),
],
]
),
createPriceLine({
number: 1,
@@ -2728,7 +2729,9 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
title: `value destroyed ${title}`,
bottom: list.flatMap(({ color, name, key }) => {
const normalKey = `${fixKey(key)}value_destroyed`;
const adjKey = `${fixKey(key)}adjusted_value_destroyed`;
const adjKey = `${fixKey(
key
)}adjusted_value_destroyed`;
return [
createBaseSeries({
key: normalKey,
@@ -2782,7 +2785,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
key,
name,
color,
}),
})
),
},
]
@@ -2823,7 +2826,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
key,
name,
color,
}),
})
),
},
]
@@ -2867,37 +2870,51 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
color: colors.red,
}),
createBaseSeries({
key: `${fixKey(args.key)}unrealized_profit_rel_to_market_cap`,
key: `${fixKey(
args.key
)}unrealized_profit_rel_to_market_cap`,
name: "Profit",
color: colors.green,
}),
createBaseSeries({
key: `${fixKey(args.key)}unrealized_loss_rel_to_market_cap`,
key: `${fixKey(
args.key
)}unrealized_loss_rel_to_market_cap`,
name: "Loss",
color: colors.red,
defaultActive: false,
}),
createBaseSeries({
key: `${fixKey(args.key)}neg_unrealized_loss_rel_to_market_cap`,
key: `${fixKey(
args.key
)}neg_unrealized_loss_rel_to_market_cap`,
name: "Negative Loss",
color: colors.red,
}),
...(`${fixKey(args.key)}unrealized_profit_rel_to_own_market_cap` in
...(`${fixKey(
args.key
)}unrealized_profit_rel_to_own_market_cap` in
vecIdToIndexes
? [
createBaseSeries({
key: `${fixKey(args.key)}unrealized_profit_rel_to_own_market_cap`,
key: `${fixKey(
args.key
)}unrealized_profit_rel_to_own_market_cap`,
name: "Profit",
color: colors.green,
}),
createBaseSeries({
key: `${fixKey(args.key)}unrealized_loss_rel_to_own_market_cap`,
key: `${fixKey(
args.key
)}unrealized_loss_rel_to_own_market_cap`,
name: "Loss",
color: colors.red,
defaultActive: false,
}),
createBaseSeries({
key: `${fixKey(args.key)}neg_unrealized_loss_rel_to_own_market_cap`,
key: `${fixKey(
args.key
)}neg_unrealized_loss_rel_to_own_market_cap`,
name: "Negative Loss",
color: colors.red,
}),
@@ -2910,22 +2927,30 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
}),
]
: []),
...(`${fixKey(args.key)}unrealized_profit_rel_to_own_total_unrealized_pnl` in
...(`${fixKey(
args.key
)}unrealized_profit_rel_to_own_total_unrealized_pnl` in
vecIdToIndexes
? [
createBaseSeries({
key: `${fixKey(args.key)}unrealized_profit_rel_to_own_total_unrealized_pnl`,
key: `${fixKey(
args.key
)}unrealized_profit_rel_to_own_total_unrealized_pnl`,
name: "Profit",
color: colors.green,
}),
createBaseSeries({
key: `${fixKey(args.key)}unrealized_loss_rel_to_own_total_unrealized_pnl`,
key: `${fixKey(
args.key
)}unrealized_loss_rel_to_own_total_unrealized_pnl`,
name: "Loss",
color: colors.red,
defaultActive: false,
}),
createBaseSeries({
key: `${fixKey(args.key)}neg_unrealized_loss_rel_to_own_total_unrealized_pnl`,
key: `${fixKey(
args.key
)}neg_unrealized_loss_rel_to_own_total_unrealized_pnl`,
name: "Negative Loss",
color: colors.red,
}),
@@ -3010,13 +3035,14 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
title: useGroupName ? name : "Net",
color: useGroupName ? color : undefined,
}),
...(`${fixKey(key)}net_unrealized_pnl_rel_to_own_market_cap` in
vecIdToIndexes
...(`${fixKey(
key
)}net_unrealized_pnl_rel_to_own_market_cap` in vecIdToIndexes
? [
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
key,
key
)}net_unrealized_pnl_rel_to_own_market_cap`,
title: useGroupName ? name : "Net",
color: useGroupName ? color : undefined,
@@ -3026,13 +3052,15 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
}),
]
: []),
...(`${fixKey(key)}net_unrealized_pnl_rel_to_own_total_unrealized_pnl` in
...(`${fixKey(
key
)}net_unrealized_pnl_rel_to_own_total_unrealized_pnl` in
vecIdToIndexes
? [
/** @satisfies {FetchedBaselineSeriesBlueprint} */ ({
type: "Baseline",
key: `${fixKey(
key,
key
)}net_unrealized_pnl_rel_to_own_total_unrealized_pnl`,
title: useGroupName ? name : "Net",
color: useGroupName ? color : undefined,
@@ -3249,7 +3277,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
key: `price_${key}_${keyAddon}`,
name: key,
color,
}),
})
),
},
...averages.map(({ key, name, color }) => ({
@@ -3466,7 +3494,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
unit: "percentage",
}),
],
}),
})
),
.../** @type {const} */ ([
{ name: "2 Year", key: "2y" },
@@ -3523,7 +3551,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
unit: "percentage",
}),
],
}),
})
),
],
},
@@ -3539,7 +3567,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
name: `${year}`,
color,
defaultActive,
}),
})
),
},
...dcaClasses.map(
@@ -3564,7 +3592,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
unit: "percentage",
}),
],
}),
})
),
],
},
@@ -3653,13 +3681,13 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
// name: "Weight",
// }),
...createAverageSumCumulativeMinMaxPercentilesSeries(
"block_size",
"block_size"
),
...createAverageSumCumulativeMinMaxPercentilesSeries(
"block_weight",
"block_weight"
),
...createAverageSumCumulativeMinMaxPercentilesSeries(
"block_vbytes",
"block_vbytes"
),
],
},
@@ -3675,7 +3703,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
{
key: "tx_count",
name: "Count",
},
}
),
},
{
@@ -3743,7 +3771,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
common: `v${index + 1}`,
sumColor,
cumulativeColor,
}),
})
),
},
{
@@ -3877,19 +3905,19 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
{
key: "coinbase",
name: "Coinbase",
},
}
),
...createBaseAverageSumCumulativeMinMaxPercentilesSeries(
{
key: "coinbase_btc",
name: "Coinbase",
},
}
),
...createBaseAverageSumCumulativeMinMaxPercentilesSeries(
{
key: "coinbase_usd",
name: "Coinbase",
},
}
),
],
},
@@ -3901,7 +3929,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
{
key: "subsidy",
name: "Subsidy",
},
}
),
createBaseSeries({
key: "subsidy_usd_1y_sma",
@@ -3911,13 +3939,13 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
{
key: "subsidy_btc",
name: "Subsidy",
},
}
),
...createBaseAverageSumCumulativeMinMaxPercentilesSeries(
{
key: "subsidy_usd",
name: "Subsidy",
},
}
),
],
},
@@ -3926,13 +3954,13 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
title: "Transaction Fee",
bottom: [
...createAverageSumCumulativeMinMaxPercentilesSeries(
"fee",
"fee"
),
...createAverageSumCumulativeMinMaxPercentilesSeries(
"fee_btc",
"fee_btc"
),
...createAverageSumCumulativeMinMaxPercentilesSeries(
"fee_usd",
"fee_usd"
),
],
},
@@ -4284,7 +4312,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
sumColor,
cumulativeColor,
}),
],
]
),
},
{
@@ -4522,7 +4550,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
key,
name,
color,
}),
})
),
},
...cointimePrices.map(({ key, name, color, title }) => ({
@@ -4559,7 +4587,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
key,
name,
color,
}),
})
),
],
},
@@ -4584,7 +4612,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
color: colors.orange,
}),
],
}),
})
),
],
},
@@ -4625,7 +4653,7 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
name,
color,
}),
]),
])
),
},
{
@@ -4778,6 +4806,12 @@ function createPartialOptions({ env, colors, vecIdToIndexes, pools }) {
url: () => "/github",
title: "Link to BRK's repository",
},
{
name: "Changelog",
url: () =>
"https://github.com/bitcoinresearchkit/brk/blob/main/docs/CHANGELOG.md#changelog",
title: "BRK's changelog",
},
],
},
{
@@ -4958,7 +4992,7 @@ export function initOptions({
partialTree,
parent,
parentPath = [],
depth = 0,
depth = 0
) {
/** @type {Accessor<number>[]} */
const listForSum = [];
@@ -4982,7 +5016,7 @@ export function initOptions({
return null;
}
},
null,
null
);
partialTree.forEach((anyPartial, partialIndex) => {
@@ -5010,7 +5044,7 @@ export function initOptions({
Object.assign(anyPartial, groupAddons);
const passedDetails = signals.createSignal(
/** @type {HTMLDivElement | HTMLDetailsElement | null} */ (null),
/** @type {HTMLDivElement | HTMLDetailsElement | null} */ (null)
);
const serName = utils.stringToId(anyPartial.name);
@@ -5019,7 +5053,7 @@ export function initOptions({
anyPartial.tree,
passedDetails,
path,
depth + 1,
depth + 1
);
listForSum.push(childOptionsCount);
@@ -5087,7 +5121,7 @@ export function initOptions({
path,
name,
title: option.title,
}),
})
);
} else if ("kind" in anyPartial && anyPartial.kind === "table") {
Object.assign(
@@ -5097,7 +5131,7 @@ export function initOptions({
path,
name,
title: option.title,
}),
})
);
} else if ("kind" in anyPartial && anyPartial.kind === "simulation") {
Object.assign(
@@ -5107,7 +5141,7 @@ export function initOptions({
path,
name,
title: anyPartial.title,
}),
})
);
} else if ("url" in anyPartial) {
Object.assign(
@@ -5119,7 +5153,7 @@ export function initOptions({
title: name,
qrcode: !!anyPartial.qrcode,
url: anyPartial.url,
}),
})
);
} else {
const title = option.title || option.name;
@@ -5132,7 +5166,7 @@ export function initOptions({
path,
top: arrayToRecord(anyPartial.top),
bottom: arrayToRecord(anyPartial.bottom),
}),
})
);
}
@@ -5181,7 +5215,7 @@ export function initOptions({
});
return signals.createMemo(() =>
listForSum.reduce((acc, s) => acc + s(), 0),
listForSum.reduce((acc, s) => acc + s(), 0)
);
}
recursiveProcessPartialTree(partialOptions, parent);