diff --git a/.claude/prompts/readmes.md b/.claude/prompts/readmes.md index 62f28ed3e..55ac9b36b 100644 --- a/.claude/prompts/readmes.md +++ b/.claude/prompts/readmes.md @@ -3,18 +3,26 @@ Generate a professional, comprehensive README.md for each crate based SOLELY on code analysis. Use NO external documentation, commit messages, or existing READMEs. ## MANDATORY PROCESS - FOLLOW EXACTLY: -1. Analyze each crate's source code thoroughly using file system exploration -2. **MANDATORY CODE ANALYSIS**: Before writing ANY README content, you MUST: +1. **IGNORE EXISTING DOCS**: Do NOT read any .md, .txt, .rst, or documentation files in the crate directory +2. **CODE-ONLY ANALYSIS**: Examine ONLY these files: + - All .rs files in src/ directory and subdirectories + - Cargo.toml for dependencies and metadata + - Code structure and organization +3. **MANDATORY CODE ANALYSIS**: Before writing ANY README content, you MUST: - Examine all Rust files in src/ directory - Identify the main structs, enums, traits, and functions - Understand the crate's architecture and data flow - Determine the crate's purpose from its implementation - Map dependencies to understand external integrations -3. Generate one complete README.md per crate -4. Focus on one crate at a time for thorough analysis +4. **FRESH PERSPECTIVE**: Write the README as if you're the first person to document this crate +5. Generate one complete README.md per crate +6. Focus on one crate at a time for thorough analysis ## ABSOLUTE REQUIREMENTS: - **SOURCE OF TRUTH**: Use ONLY the actual Rust code - no external docs, comments may provide hints but focus on implementation +- **CRITICAL**: DO NOT read any existing README.md, CHANGELOG.md, or documentation files +- **IGNORE ALL TEXT FILES**: .md, .txt, .rst files are FORBIDDEN sources - treat them as if they don't exist +- **CODE ONLY**: Focus exclusively on .rs files, Cargo.toml, and code structure - **PROFESSIONAL GRADE**: Write as if this will be published on crates.io for other developers - **PROGRAMMER FOCUSED**: Assume audience knows Rust and relevant domain concepts - **IMPLEMENTATION-BASED**: Describe what the code actually does, not what comments claim it should do @@ -74,6 +82,9 @@ crate_name = "X.Y.Z" - **NEVER** write marketing copy: "cutting-edge", "state-of-the-art", "enterprise-grade" - **NEVER** make claims you can't verify from code: "blazingly fast", "memory efficient" - **NEVER** copy-paste from existing documentation or comments +- **NEVER** read or reference existing README.md files - pretend they don't exist +- **NEVER** use phrases like "as mentioned in the documentation" or "according to the docs" +- **NEVER** let existing documentation influence your analysis or writing ### REQUIRED SPECIFICITY: - **Data structures**: Mention specific types (HashMap, Vec, etc.) @@ -82,7 +93,27 @@ crate_name = "X.Y.Z" - **Error handling**: How errors are represented and handled - **Async/sync**: Clearly state if operations are blocking or async -### CODE ANALYSIS DEPTH: +### ANTI-BIAS PROTOCOL: + +### BEFORE STARTING ANY ANALYSIS: +1. **Explicitly ignore**: Any README.md, CHANGELOG.md, docs/, documentation files +2. **File filtering**: Only examine .rs and Cargo.toml files +3. **Fresh eyes approach**: Analyze the code as if you've never seen this crate before +4. **Independent thinking**: Form your own understanding purely from code inspection + +### IF YOU ACCIDENTALLY READ EXISTING DOCS: +- Stop immediately and restart your analysis +- Consciously disregard any information from documentation files +- Base all descriptions solely on what you observe in the code +- Ask yourself: "What would I think this code does if I had no documentation?" + +### VALIDATION CHECKS: +- **Unique descriptions**: Your descriptions should differ significantly from any existing docs +- **Code-derived insights**: Every feature mentioned must be visible in the source code +- **Independent voice**: Write in your own technical style, not mimicking existing documentation +- **Fresh examples**: Create new code examples based on API analysis, not existing samples + +## CODE ANALYSIS DEPTH: **You MUST analyze and understand:** 1. **Public API surface**: All pub structs, functions, traits, modules 2. **Core abstractions**: Main data types and their relationships diff --git a/crates/brk/README.md b/crates/brk/README.md index cbd2dcc2f..d5fd14bc3 100644 --- a/crates/brk/README.md +++ b/crates/brk/README.md @@ -1,197 +1,273 @@ # brk -**Main wrapper crate for the Bitcoin Research Kit (BRK)** +Unified Bitcoin Research Kit crate providing optional feature-gated access to all BRK components. -The `brk` crate serves as the primary entry point for the Bitcoin Research Kit, providing a unified interface to all BRK components through feature flags. It enables developers to selectively include only the components they need while maintaining a consistent API. +[![Crates.io](https://img.shields.io/crates/v/brk.svg)](https://crates.io/crates/brk) +[![Documentation](https://docs.rs/brk/badge.svg)](https://docs.rs/brk) -## What it provides +## Overview -- **Unified Access**: Single crate providing access to the entire BRK ecosystem -- **Feature-based Selection**: Choose only the components you need -- **Consistent Versioning**: All components versioned together for compatibility -- **Simplified Dependencies**: Single dependency instead of multiple individual crates +This crate serves as a unified entry point to the Bitcoin Research Kit ecosystem, providing feature-gated re-exports of all BRK components. It allows users to selectively include only the functionality they need while maintaining a single dependency declaration, with the `brk_cli` component always available for command-line interface access. -## Available Components +**Key Features:** -### Core Data Pipeline -- **`parser`** ([brk_parser](../brk_parser/)) - High-performance Bitcoin block parser -- **`indexer`** ([brk_indexer](../brk_indexer/)) - Blockchain data indexer with dual storage -- **`computer`** ([brk_computer](../brk_computer/)) - Analytics engine for computed datasets -- **`interface`** ([brk_interface](../brk_interface/)) - Unified data query and formatting API +- Feature-gated modular access to 12 specialized BRK components +- Single dependency entry point with selective compilation +- Always-available CLI component for command-line operations +- Comprehensive documentation aggregation with inline re-exports +- `full` feature for complete BRK functionality inclusion +- Optimized build configuration with docs.rs integration -### Infrastructure -- **`structs`** ([brk_structs](../brk_structs/)) - Bitcoin-aware type system and data structures -- **`error`** ([brk_error](../brk_error/)) - Centralized error handling -- **`store`** ([brk_store](../brk_store/)) - Blockchain-aware key-value storage +**Target Use Cases:** -### External Integration -- **`fetcher`** ([brk_fetcher](../brk_fetcher/)) - Bitcoin price data fetcher with multi-source fallback -- **`server`** ([brk_server](../brk_server/)) - HTTP server with REST API -- **`mcp`** ([brk_mcp](../brk_mcp/)) - Model Context Protocol for LLM integration +- Applications requiring selective BRK functionality to minimize dependencies +- Library development where only specific Bitcoin analysis components are needed +- Prototyping and experimentation with different BRK component combinations +- Educational use cases demonstrating modular blockchain analytics architecture -### Utilities -- **`cli`** ([brk_cli](../brk_cli/)) - Command line interface (always enabled) -- **`logger`** ([brk_logger](../brk_logger/)) - Logging utilities -- **`bundler`** ([brk_bundler](../brk_bundler/)) - Asset bundling for web interfaces - -## Usage - -### Full Installation +## Installation ```toml [dependencies] -brk = { version = "0.0.88", features = ["full"] } +# Minimal installation with CLI only +brk = "0.0.107" + +# Full functionality +brk = { version = "0.0.107", features = ["full"] } + +# Selective features +brk = { version = "0.0.107", features = ["indexer", "computer", "server"] } ``` +## Quick Start + ```rust -use brk::*; +// CLI is always available +use brk::cli; -// Access all components -let config = cli::Config::load()?; -let parser = parser::Parser::new(/* ... */); -let indexer = indexer::Indexer::forced_import("./data")?; -let computer = computer::Computer::forced_import("./data", &indexer, None)?; +// Feature-gated components +#[cfg(feature = "indexer")] +use brk::indexer::Indexer; + +#[cfg(feature = "computer")] +use brk::computer::Computer; + +#[cfg(feature = "server")] +use brk::server::Server; + +// Build complete pipeline with selected features +#[cfg(all(feature = "indexer", feature = "computer", feature = "server"))] +fn build_pipeline() -> Result<(), Box> { + let indexer = Indexer::build("./data")?; + let computer = Computer::build("./analytics", &indexer)?; + let interface = brk::interface::Interface::build(&indexer, &computer); + let server = Server::new(interface, None); + Ok(()) +} ``` -### Selective Components +## API Overview -```toml -[dependencies] -brk = { version = "0.0.88", features = ["parser", "indexer", "computer"] } -``` +### Feature Organization + +The crate provides feature-gated access to BRK components organized by functionality: + +**Core Data Processing:** +- `structs` - Bitcoin-aware data structures and type system +- `error` - Centralized error handling across components +- `store` - Transactional key-value storage wrapper + +**Blockchain Processing:** +- `parser` - Multi-threaded Bitcoin block parsing +- `indexer` - Blockchain data indexing with columnar storage +- `computer` - Analytics computation engine + +**Data Access:** +- `interface` - Unified query interface with fuzzy search +- `fetcher` - Multi-source price data aggregation + +**Service Layer:** +- `server` - HTTP API server with caching and compression +- `mcp` - Model Context Protocol bridge for LLM integration +- `logger` - Enhanced logging with colored output + +**Web Infrastructure:** +- `bundler` - Web asset bundling using Rolldown + +### Always Available + +**`cli`**: Command-line interface module (no feature gate required) +Provides access to the complete BRK command-line interface for running full instances. + +## Examples + +### Minimal Bitcoin Parser ```rust -use brk::{parser, indexer, computer}; +use brk::parser::Parser; -// Core data pipeline only -let parser = parser::Parser::new(blocks_dir, Some(output_dir), rpc); -let mut indexer = indexer::Indexer::forced_import(output_dir)?; -let mut computer = computer::Computer::forced_import(output_dir, &indexer, None)?; +fn parse_blocks() -> Result<(), Box> { + #[cfg(feature = "parser")] + { + let parser = Parser::new("/path/to/blocks", None, rpc_client); + // Parse blockchain data + Ok(()) + } + #[cfg(not(feature = "parser"))] + { + Err("Parser feature not enabled".into()) + } +} ``` -### Minimal Setup - -```toml -[dependencies] -brk = { version = "0.0.88", features = ["structs", "parser"] } -``` - -```rust -use brk::{structs, parser}; - -// Just parsing and types -let height = structs::Height::new(800_000); -let parser = parser::Parser::new(blocks_dir, Some(output_dir), rpc); -``` - -## Feature Flags - -| Feature | Description | Dependencies | -|---------|-------------|--------------| -| `full` | Enable all components | All crates | -| `cli` | Command line interface | Always enabled | -| `structs` | Core type system | Foundation for other crates | -| `error` | Error handling | Used by most crates | -| `parser` | Block parsing | `structs`, `error` | -| `store` | Key-value storage | `structs`, `error` | -| `indexer` | Blockchain indexing | `parser`, `store` | -| `computer` | Analytics computation | `indexer`, `fetcher` (optional) | -| `fetcher` | Price data fetching | `structs`, `error` | -| `interface` | Data query API | `indexer`, `computer` | -| `server` | HTTP server | `interface`, `mcp` | -| `mcp` | LLM integration | `interface` | -| `logger` | Logging utilities | Standalone | -| `bundler` | Asset bundling | Standalone | - -## Common Usage Patterns - -### Complete BRK Instance - -```rust -use brk::*; - -// Full data pipeline setup -let config = cli::Config::load()?; -let rpc = /* Bitcoin Core RPC client */; -let parser = parser::Parser::new(config.blocks_dir, Some(config.output_dir), rpc); -let mut indexer = indexer::Indexer::forced_import(&config.output_dir)?; -let mut computer = computer::Computer::forced_import(&config.output_dir, &indexer, None)?; -let interface = interface::Interface::build(&indexer, &computer); -let server = server::Server::new(interface, config.website_path); - -// Start server -server.serve(true).await?; -``` - -### Data Analysis +### Analytics Pipeline ```rust use brk::{indexer, computer, interface}; -// Analysis-focused setup -let indexer = indexer::Indexer::forced_import("./brk_data")?; -let computer = computer::Computer::forced_import("./brk_data", &indexer, None)?; -let interface = interface::Interface::build(&indexer, &computer); +#[cfg(all(feature = "indexer", feature = "computer", feature = "interface"))] +fn analytics_pipeline() -> Result<(), Box> { + // Initialize indexer + let indexer = indexer::Indexer::build("./blockchain_data")?; -// Query data -let params = interface::Params { - index: interface::Index::Height, - ids: vec!["price_usd", "difficulty"].into(), - rest: interface::ParamsOpt::default() - .set_from(-100) - .set_format(interface::Format::CSV), -}; + // Compute analytics + let computer = computer::Computer::build("./analytics", &indexer)?; -let csv_data = interface.search_and_format(params)?; -``` + // Create query interface + let interface = interface::Interface::build(&indexer, &computer); -### Custom Integration + // Query latest price + let params = interface::Params { + index: "date".to_string(), + ids: vec!["price-close".to_string()].into(), + from: Some(-1), + ..Default::default() + }; -```rust -use brk::{structs, parser, error}; - -// Custom application with BRK components -fn analyze_blocks() -> error::Result<()> { - let parser = parser::Parser::new(blocks_dir, Some(output_dir), rpc); - - parser.parse(None, None) - .iter() - .take(1000) // First 1000 blocks - .for_each(|(height, block, hash)| { - println!("Block {}: {} transactions", height, block.txdata.len()); - }); + let result = interface.search_and_format(params)?; + println!("Latest Bitcoin price: {:?}", result); Ok(()) } ``` -## Version Compatibility +### Web Server Setup -All BRK crates are released together with synchronized versions. When using the `brk` wrapper crate, you're guaranteed compatibility between all components. +```rust +use brk::{server, interface, indexer, computer}; -- **Current version**: 0.0.88 -- **Rust MSRV**: 1.89+ -- **Bitcoin Core**: v25.0 - v29.0 +#[cfg(all(feature = "server", feature = "interface", feature = "indexer", feature = "computer"))] +async fn web_server() -> Result<(), Box> { + let indexer = indexer::Indexer::build("./data")?; + let computer = computer::Computer::build("./analytics", &indexer)?; + let interface = interface::Interface::build(&indexer, &computer); -## Performance Characteristics + let server = server::Server::new(interface, None); + server.serve(true).await?; -The `brk` crate itself adds no runtime overhead - it simply re-exports the underlying crates. Performance characteristics depend on which components you use: + Ok(()) +} +``` -- **Full pipeline**: ~13-15 hours initial sync, ~40GB storage overhead -- **Parser only**: ~4 minutes to parse entire blockchain -- **Indexer only**: ~7-8 hours to index blockchain, ~5-6GB RAM usage -- **Server**: Low latency API responses with caching and compression +### MCP Integration -## Dependencies +```rust +use brk::{mcp, interface, indexer, computer}; -The `brk` crate's dependencies are determined by enabled features. Core dependencies include: +#[cfg(all(feature = "mcp", feature = "interface"))] +fn mcp_server(interface: &'static interface::Interface) -> mcp::MCP { + mcp::MCP::new(interface) +} +``` -- `brk_cli` - Always included for configuration and CLI support -- Individual `brk_*` crates based on enabled features -- Transitive dependencies from enabled components +## Feature Combinations -For specific dependency information, see individual crate READMEs. +### Common Combinations + +**Data Processing**: `["structs", "parser", "indexer"]` +Basic blockchain data processing and indexing. + +**Analytics**: `["indexer", "computer", "fetcher", "interface"]` +Complete analytics pipeline with price data integration. + +**API Server**: `["interface", "server", "logger"]` +HTTP API server with logging capabilities. + +**Full Stack**: `["full"]` +All components for complete BRK functionality. + +### Dependency Optimization + +Feature selection allows for significant dependency reduction: + +```toml +# Minimal parser-only dependency +brk = { version = "0.0.107", features = ["parser"] } + +# Analytics without web server +brk = { version = "0.0.107", features = ["indexer", "computer", "interface"] } + +# Web server without parsing +brk = { version = "0.0.107", features = ["interface", "server"] } +``` + +## Architecture + +### Re-export Pattern + +The crate uses `#[doc(inline)]` re-exports to provide seamless access to component APIs: + +```rust +#[cfg(feature = "component")] +#[doc(inline)] +pub use brk_component as component; +``` + +This pattern ensures: +- Feature-gated compilation for dependency optimization +- Inline documentation for unified API reference +- Namespace preservation for component-specific functionality + +### Build Configuration + +- **Documentation**: `all-features = true` for complete docs.rs documentation +- **CLI Integration**: `brk_cli` always available without feature gates +- **Optional Dependencies**: All components except CLI are optional + +## Configuration + +### Feature Flags + +| Feature | Component | Description | +|---------|-----------|-------------| +| `bundler` | `brk_bundler` | Web asset bundling | +| `computer` | `brk_computer` | Analytics computation | +| `error` | `brk_error` | Error handling | +| `fetcher` | `brk_fetcher` | Price data fetching | +| `indexer` | `brk_indexer` | Blockchain indexing | +| `interface` | `brk_interface` | Data query interface | +| `logger` | `brk_logger` | Enhanced logging | +| `mcp` | `brk_mcp` | Model Context Protocol | +| `parser` | `brk_parser` | Block parsing | +| `server` | `brk_server` | HTTP server | +| `store` | `brk_store` | Key-value storage | +| `structs` | `brk_structs` | Data structures | +| `full` | All components | Complete functionality | + +### Documentation + +Documentation is aggregated from all components with `#![doc = include_str!("../README.md")]` ensuring comprehensive API reference across all features. + +## Code Analysis Summary + +**Main Structure**: Feature-gated re-export crate providing unified access to 12 BRK components \ +**Feature System**: Cargo features enabling selective compilation and dependency optimization \ +**CLI Integration**: Always-available `brk_cli` access without feature requirements \ +**Documentation**: Inline re-exports with comprehensive docs.rs integration \ +**Dependency Management**: Optional dependencies for all components except CLI \ +**Build Configuration**: Optimized compilation with all-features documentation \ +**Architecture**: Modular aggregation crate enabling flexible BRK ecosystem usage --- -*This README was generated by Claude Code* +_This README was generated by Claude Code_ \ No newline at end of file diff --git a/crates/brk_bundler/README.md b/crates/brk_bundler/README.md index 9ce18be2a..4237b5d77 100644 --- a/crates/brk_bundler/README.md +++ b/crates/brk_bundler/README.md @@ -1,186 +1,278 @@ # brk_bundler -**Asset bundling for BRK web interfaces using Rolldown** +Asset bundling and development server for BRK web interfaces with hot reloading and file watching. -`brk_bundler` provides JavaScript/TypeScript bundling capabilities for BRK's web interfaces. It's a thin wrapper around Rolldown (Rust-based Rollup alternative) with BRK-specific configuration for building optimized web assets with file watching and automatic rebuilding. +[![Crates.io](https://img.shields.io/crates/v/brk_bundler.svg)](https://crates.io/crates/brk_bundler) +[![Documentation](https://docs.rs/brk_bundler/badge.svg)](https://docs.rs/brk_bundler) -## What it provides +## Overview -- **JavaScript Bundling**: Modern ES modules bundling with minification -- **File Watching**: Automatic rebuilding on source file changes -- **Asset Processing**: Copies and processes static assets -- **Version Injection**: Automatic version string replacement in service workers -- **Development Mode**: Live rebuilding for rapid development +This crate provides a thin wrapper around the Rolldown JavaScript bundler specifically designed for BRK web interface development. It handles asset bundling, file copying, template processing, and development-mode file watching with automatic rebuilds and hot reloading for efficient web development workflows. -## Key Features +**Key Features:** -### Bundling Capabilities -- **ES Module Support**: Modern JavaScript bundling with tree-shaking -- **Minification**: Automatic code minification for production builds -- **Source Maps**: Generated source maps for debugging -- **Entry Point Processing**: Configurable entry points with hashed output names +- JavaScript bundling with Rolldown (Rust-based bundler) +- Automatic file watching and hot reloading in development mode +- Template processing with version injection and asset hash replacement +- Service worker generation with version management +- Source map generation for debugging +- Minification for production builds +- Async/await support with Tokio integration -### File System Operations -- **Directory Copying**: Copies entire source directories to distribution -- **Selective Processing**: Special handling for specific file types -- **Path Resolution**: Automatic path resolution and asset linking +**Target Use Cases:** -### Development Features -- **Hot Rebuilding**: Automatic rebuilds on file changes -- **Watch Mode**: Monitors source files and triggers rebuilds -- **Version Replacement**: Injects build version into service workers +- BRK blockchain explorer web interfaces +- Development of Bitcoin analytics dashboards +- Building responsive web applications for blockchain data visualization +- Hot reloading development environment for rapid iteration -## Usage +## Installation -### Basic Bundling +```toml +cargo add brk_bundler +``` + +## Quick Start ```rust use brk_bundler::bundle; use std::path::Path; -// Bundle without watching (production) -let websites_path = Path::new("./websites"); -let source_folder = "default"; -let dist_path = bundle(websites_path, source_folder, false).await?; +#[tokio::main] +async fn main() -> std::io::Result<()> { + let websites_path = Path::new("./web"); + let source_folder = "src"; + let watch = true; // Enable hot reloading -println!("Bundled to: {:?}", dist_path); -``` + // Bundle assets and start development server + let dist_path = bundle(websites_path, source_folder, watch).await?; -### Development Mode with Watching + println!("Assets bundled to: {}", dist_path.display()); -```rust -// Bundle with file watching (development) -let dist_path = bundle(websites_path, "default", true).await?; + // Keep running for file watching (in watch mode) + if watch { + tokio::signal::ctrl_c().await?; + } -// Bundler now watches for changes and rebuilds automatically -// This will run in the background until the process exits -``` - -### Integration with BRK CLI - -```rust -// Typically called from brk_cli when serving websites -async fn setup_website(config: &Config) -> Result { - let websites_path = config.websites_path(); - let source_folder = match config.website_mode { - WebsiteMode::Default => "default", - WebsiteMode::Custom => "custom", - WebsiteMode::None => return Ok(PathBuf::new()), - }; - - // Bundle the website assets - let dist_path = bundle(websites_path, source_folder, config.dev_mode).await?; - - Ok(dist_path) + Ok(()) } ``` -## File Structure +## API Overview -The bundler expects this directory structure: +### Core Functions -``` -websites/ -├── default/ # Default website source -│ ├── index.html # Main HTML file -│ ├── service-worker.js # Service worker (version injected) -│ ├── scripts/ # JavaScript/TypeScript source -│ │ ├── entry.js # Main entry point -│ │ ├── main.js # Application logic -│ │ └── ... # Other JS modules -│ └── assets/ # Static assets -└── dist/ # Generated output directory - ├── index.html # Processed HTML with updated script references - ├── service-worker.js # Service worker with version injected - ├── scripts/ # Bundled and minified JavaScript - │ └── main-[hash].js # Hashed output file - └── assets/ # Copied static assets +**`bundle(websites_path: &Path, source_folder: &str, watch: bool) -> io::Result`** +Main bundling function that processes web assets and optionally starts file watching. + +### Bundling Process + +1. **Directory Setup**: Creates `dist/` directory and copies source files +2. **JavaScript Bundling**: Processes `scripts/entry.js` with Rolldown bundler +3. **Template Processing**: Updates `index.html` with hashed asset references +4. **Service Worker**: Generates service worker with version injection +5. **File Watching**: Optionally monitors source files for changes + +### Configuration + +**Rolldown Bundler Options:** + +- **Input**: `./src/scripts/entry.js` (main JavaScript entry point) +- **Output**: `./dist/scripts/` directory +- **Minification**: Enabled for production builds +- **Source Maps**: File-based source maps for debugging +- **Asset Hashing**: Automatic hash generation for cache busting + +## Examples + +### Development Mode with Hot Reloading + +```rust +use brk_bundler::bundle; +use std::path::Path; + +#[tokio::main] +async fn main() -> std::io::Result<()> { + let web_root = Path::new("./websites"); + + // Start development server with file watching + let _dist_path = bundle(web_root, "explorer", true).await?; + + println!("Development server started!"); + println!("Hot reloading enabled - edit files to see changes"); + + // Keep server running + loop { + tokio::time::sleep(tokio::time::Duration::from_secs(1)).await; + } +} ``` -## Bundling Process +### Production Build -1. **Clean**: Removes existing `dist/` directory -2. **Copy**: Copies all source files to `dist/` -3. **Bundle JavaScript**: - - Processes `scripts/entry.js` as entry point - - Generates minified bundle with source maps - - Creates hashed filename for cache busting -4. **Process HTML**: Updates script references to hashed filenames -5. **Process Service Worker**: Injects current version string -6. **Watch** (if enabled): Monitors for file changes and rebuilds +```rust +use brk_bundler::bundle; +use std::path::Path; -## Configuration +#[tokio::main] +async fn main() -> std::io::Result<()> { + let web_root = Path::new("./websites"); -The bundler uses Rolldown with these optimized settings: + // Build for production (no watching) + let dist_path = bundle(web_root, "dashboard", false).await?; + + println!("Production build completed: {}", dist_path.display()); + + // Assets are minified and ready for deployment + Ok(()) +} +``` + +### Custom Web Application Structure + +```rust +use brk_bundler::bundle; +use std::path::Path; + +// Expected directory structure: +// websites/ +// ├── my_app/ +// │ ├── index.html // Main HTML template +// │ ├── service-worker.js // Service worker template +// │ ├── scripts/ +// │ │ └── entry.js // JavaScript entry point +// │ ├── styles/ +// │ │ └── main.css // CSS files +// │ └── assets/ +// │ └── images/ // Static assets +// └── dist/ // Generated output + +#[tokio::main] +async fn main() -> std::io::Result<()> { + let websites_path = Path::new("./websites"); + let source_folder = "my_app"; + + let dist_path = bundle(websites_path, source_folder, false).await?; + + // Result: dist/ contains bundled and processed files + // - dist/index.html (with updated script references) + // - dist/service-worker.js (with version injection) + // - dist/scripts/main.[hash].js (minified and hashed) + // - dist/styles/ (copied CSS files) + // - dist/assets/ (copied static assets) + + Ok(()) +} +``` + +## Architecture + +### File Processing Pipeline + +1. **Source Copying**: Recursively copies all source files to dist directory +2. **JavaScript Bundling**: Rolldown processes entry.js with dependencies +3. **Asset Hashing**: Generates content-based hashes for cache busting +4. **Template Updates**: Replaces placeholders in HTML templates +5. **Version Injection**: Updates service worker with current package version + +### File Watching System + +**Development Mode Watchers:** + +- **Source File Watcher**: Monitors non-script files for changes +- **Bundle Watcher**: Watches JavaScript files and triggers rebuilds +- **Template Watcher**: Updates HTML when bundled assets change + +**Event Handling:** + +- **File Creation/Modification**: Automatic copying to dist directory +- **Script Changes**: Triggers Rolldown rebuild and template update +- **Template Changes**: Processes HTML and updates asset references + +### Template Processing + +**index.html Processing:** + +- Scans bundled JavaScript for asset hash +- Replaces `/scripts/main.js` with `/scripts/main.[hash].js` +- Maintains cache busting while preserving template structure + +**service-worker.js Processing:** + +- Replaces `__VERSION__` placeholder with current crate version +- Enables version-based cache invalidation +- Maintains service worker functionality + +### Async Architecture + +Built on Tokio async runtime: + +- **Non-blocking I/O**: Efficient file operations and watching +- **Concurrent Tasks**: Parallel file watching and bundle processing +- **Background Processing**: Development server runs in background task + +## Configuration Options + +### Rolldown Configuration + +The bundler uses optimized Rolldown settings: ```rust BundlerOptions { - input: Some(vec![source_entry.into()]), // scripts/entry.js - dir: Some("./dist/scripts".to_string()), // Output directory - cwd: Some(websites_path), // Working directory - minify: Some(RawMinifyOptions::Bool(true)), // Enable minification - sourcemap: Some(SourceMapType::File), // Generate source maps - ..Default::default() + input: Some(vec!["./src/scripts/entry.js".into()]), + dir: Some("./dist/scripts".to_string()), + minify: Some(RawMinifyOptions::Bool(true)), + sourcemap: Some(SourceMapType::File), + // ... other default options } ``` -## File Watching +### File Structure Requirements -In watch mode, the bundler monitors: +**Required Files:** -- **Source files**: Non-script files are copied on change -- **JavaScript files**: Trigger full rebuild via Rolldown watcher -- **HTML files**: Processed to update script references -- **Service worker**: Version injection on changes +- `src/scripts/entry.js` - JavaScript entry point +- `src/index.html` - HTML template +- `src/service-worker.js` - Service worker template -### Watch Events Handled +**Optional Directories:** -- `Create` - New files added -- `Modify` - Existing files changed -- Ignores `Delete` and other events +- `src/styles/` - CSS stylesheets +- `src/assets/` - Static assets (images, fonts, etc.) +- `src/components/` - Additional JavaScript modules -## Version Injection +## Development Workflow -Service workers get automatic version injection: +### Setup -```javascript -// In source service-worker.js -const VERSION = '__VERSION__'; +1. Create web application in `websites/app_name/` +2. Add required files (index.html, entry.js, service-worker.js) +3. Run bundler in watch mode for development -// After bundling -const VERSION = 'v0.0.88'; -``` +### Hot Reloading -This enables proper cache invalidation across releases. +- **Script Changes**: Automatic bundle rebuild and browser refresh +- **Template Changes**: Immediate HTML update with asset hash replacement +- **Asset Changes**: Instant copy to dist directory +- **Style Changes**: Direct copy without bundling -## Performance Features +### Production Deployment -- **Async Operations**: All bundling operations are async -- **Incremental Builds**: Only rebuilds changed files in watch mode -- **Parallel Processing**: Uses Tokio for concurrent file operations -- **Efficient Copying**: Direct file system operations +1. Run bundler without watch mode +2. Deploy `dist/` directory contents +3. Assets include content hashes for cache busting +4. Service worker includes version for cache management -## Error Handling +## Code Analysis Summary -- **Graceful Failures**: Logs errors but continues watching -- **Path Resolution**: Automatic path absolutization and validation -- **File System Errors**: Proper error propagation with context - -## Dependencies - -- `brk_rolldown` - Rust-based Rollup bundler -- `notify` - File system watching -- `tokio` - Async runtime for file operations -- `sugar_path` - Path manipulation utilities -- `log` - Error logging - -## Integration Points - -The bundler integrates with: -- **brk_cli**: Called during website setup -- **brk_server**: Serves bundled assets -- **Development workflow**: Provides live rebuilding +**Main Function**: `bundle()` async function coordinating Rolldown bundler with file processing and watching \ +**File Operations**: Recursive directory copying with `copy_dir_all()` and selective file processing \ +**Templating**: String replacement for asset hash injection and version management \ +**File Watching**: Multi-watcher system using `notify` crate for real-time development feedback \ +**Async Integration**: Tokio-based async architecture with background task spawning \ +**Bundler Integration**: Rolldown wrapper with optimized configuration for web development \ +**Architecture**: Development-focused asset pipeline with hot reloading and production optimization --- -*This README was generated by Claude Code* \ No newline at end of file +_This README was generated by Claude Code_ diff --git a/crates/brk_cli/README.md b/crates/brk_cli/README.md index ac65a0941..b7cf59cb1 100644 --- a/crates/brk_cli/README.md +++ b/crates/brk_cli/README.md @@ -1,189 +1,232 @@ # brk_cli -**Command line interface for running complete BRK instances** +Command-line interface orchestrating complete Bitcoin Research Kit instances with automatic configuration and continuous blockchain processing. -`brk_cli` provides the main command-line interface for operating the Bitcoin Research Kit. It orchestrates the complete data pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with automatic configuration management and graceful operation. +[![Crates.io](https://img.shields.io/crates/v/brk_cli.svg)](https://crates.io/crates/brk_cli) +[![Documentation](https://docs.rs/brk_cli/badge.svg)](https://docs.rs/brk_cli) -## What it provides +## Overview -- **Complete Pipeline Orchestration**: Coordinates parser, indexer, computer, and server components -- **Automatic Configuration**: Saves settings to `~/.brk/config.toml` for consistent operation -- **Continuous Operation**: Handles blockchain updates and incremental processing -- **Web Interface Options**: Configurable website serving (none, default, custom) -- **Graceful Shutdown**: Ctrl+C handling with proper cleanup +This crate provides the primary command-line interface for running Bitcoin Research Kit instances. It orchestrates the entire data processing pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with persistent configuration management, automatic error recovery, and continuous blockchain synchronization. -## Key Features +**Key Features:** -### Pipeline Management -- **Automatic dependency handling**: Ensures Bitcoin Core sync before processing -- **Incremental updates**: Only processes new blocks since last run -- **Error recovery**: Automatic retry logic and graceful error handling -- **Resource management**: Optimized memory usage and disk I/O +- Complete BRK pipeline orchestration with parser, indexer, computer, and server coordination +- Persistent configuration system with TOML-based auto-save functionality +- Continuous blockchain processing with new block detection and incremental updates +- Flexible Bitcoin Core RPC authentication with cookie file and user/password support +- Configurable web interface options including auto-downloading from GitHub releases +- Large stack allocation (512MB) for handling complex blockchain processing workloads +- Graceful shutdown handling with proper cleanup and state preservation -### Configuration System -- **Auto-save configuration**: All CLI options saved to persistent config -- **Flexible paths**: Configurable Bitcoin directory, blocks directory, and output directory -- **RPC authentication**: Cookie file or username/password authentication -- **Data source options**: Configurable price fetching and exchange APIs +**Target Use Cases:** -### Operation Modes -- **Initial sync**: Full blockchain processing from genesis -- **Continuous operation**: Real-time processing of new blocks -- **Update mode**: Resume from last processed block -- **Server mode**: HTTP API with optional web interface +- Production Bitcoin analytics deployments requiring full pipeline operation +- Development environments for Bitcoin research and analysis +- Continuous blockchain monitoring with real-time data updates +- Academic research requiring comprehensive historical blockchain datasets ## Installation -### Binary Release ```bash -# Download from GitHub releases -# https://github.com/bitcoinresearchkit/brk/releases/latest +cargo install brk # or cargo install brk_cli ``` -### Via Cargo -```bash -cargo install brk --locked -``` - -### From Source -```bash -git clone https://github.com/bitcoinresearchkit/brk.git -cd brk && cargo build --release -``` - -## Usage - -### First Run (Configuration Setup) +## Quick Start ```bash -# Basic setup with default options -brk --brkdir ./my_brk_data +# First run - configure and start processing +brk --brkdir ./data --bitcoindir ~/.bitcoin --fetch true -# Full configuration -brk --bitcoindir ~/.bitcoin \ - --brkdir ./brk_data \ - --fetch true \ - --exchanges true \ - --website default -``` - -### Subsequent Runs - -```bash -# Uses saved configuration from ~/.brk/config.toml +# Subsequent runs use saved configuration brk # Override specific options brk --website none --fetch false ``` -### Command Line Options +## API Overview + +### Core Structure + +- **`Config`**: Persistent configuration with clap-based CLI parsing and TOML serialization +- **`Bridge`**: Interface trait for generating JavaScript bridge files for web interfaces +- **`Website`**: Enum for web interface options (None, Bitview, Custom) +- **Path Functions**: Cross-platform default path resolution for Bitcoin and BRK directories + +### Main Operations + +**`main() -> color_eyre::Result<()>`** +Entry point with error handling setup, directory creation, logging initialization, and high-stack thread spawning. + +**`run() -> color_eyre::Result<()>`** +Core processing loop handling configuration, RPC connection, component initialization, and continuous blockchain monitoring. + +### Configuration Management + +**Persistent Settings:** + +- All CLI arguments automatically saved to `~/.brk/config.toml` +- Argument overrides update saved configuration on each run +- Cross-platform path resolution with tilde and $HOME expansion +- Validation of Bitcoin directory, blocks directory, and RPC authentication + +**CLI Parameters:** + +- `--bitcoindir`, `--blocksdir`, `--brkdir`: Directory configuration +- `--fetch`, `--exchanges`: Data source configuration +- `--website`: Web interface selection +- `--rpcconnect`, `--rpcport`, `--rpccookiefile`, `--rpcuser`, `--rpcpassword`: RPC settings + +## Examples + +### Basic Usage ```bash -brk --help +# Initialize with custom directories +brk --bitcoindir /data/bitcoin --brkdir /data/brk + +# Enable all features with custom RPC +brk --fetch true --exchanges true --website bitview \ + --rpcuser myuser --rpcpassword mypass + +# Minimal setup with API only +brk --website none --fetch false ``` -## Configuration Reference +### Configuration File Example -All options are automatically saved to `~/.brk/config.toml`: +After first run, settings are saved to `~/.brk/config.toml`: -### Core Paths -- `--bitcoindir ` - Bitcoin Core directory (default: `~/.bitcoin`) -- `--blocksdir ` - Block files directory (default: `bitcoindir/blocks`) -- `--brkdir ` - BRK output directory (default: `~/.brk`) - -### Data Sources -- `--fetch ` - Enable price data fetching (default: `true`) -- `--exchanges ` - Use exchange APIs for prices (default: `true`) - -### Web Interface -- `--website