Compare commits

...

15 Commits

Author SHA1 Message Date
nym21 38b8a08297 release: v0.0.89 2025-08-24 16:46:54 +02:00
nym21 c9ffd3ad99 lock: update 2025-08-24 16:46:43 +02:00
nym21 61f960de28 global: snapshot 2025-08-24 16:45:20 +02:00
nym21 da1ff2cacc computer: stateful: maybe got rollback to work, tbd 2025-08-19 23:34:05 +02:00
nym21 05036c682f global: snapshot 2025-08-17 21:38:28 +02:00
nym21 7d47bc8042 changelog: add link to releases and to changes 2025-08-16 22:23:07 +02:00
nym21 98cfd160ef changelog: vibed 2025-08-16 19:16:09 +02:00
nym21 b5e3262b67 readmes: update 2025-08-16 18:21:44 +02:00
nym21 009fb35c4c computer: cleanup 2025-08-16 16:42:01 +02:00
nym21 8648d3131a computer: convert ComputedFrom to LazyFrom 2025-08-13 10:46:28 +02:00
nym21 00c316c35d readmes: vibed 2025-08-13 00:52:23 +02:00
nym21 5f8de8e756 computer: rollback part 1 2025-08-12 22:37:16 +02:00
nym21 ee5dc8fc41 computer: refactor 2025-08-10 16:00:44 +02:00
nym21 a61926988a release: v0.0.88 2025-08-10 14:16:31 +02:00
nym21 bd8c4dfb6b website: fix options 2025-08-10 14:16:13 +02:00
150 changed files with 9773 additions and 10859 deletions
+3 -1
View File
@@ -3,7 +3,7 @@
# Builds
target
dist
websites/dist
vecid-to-indexes.js
# Copies
@@ -28,4 +28,6 @@ flamegraph.svg
*.trace
# AI
.claude
CLAUDE.md
CLAUDE*.md
+903
View File
@@ -1,3 +1,906 @@
# Changelog
*This changelog was generated by Claude Code.*
All notable changes to the Bitcoin Research Kit (BRK) project will be documented in this file.
## Unreleased
### Documentation
- **Enhanced**: Comprehensive rewrite of all crate README files for improved clarity and developer experience
- **Updated**: Main project README with better structure and documentation
- **Standardized**: README format across all workspace crates with consistent styling
- **Improved**: Documentation structure and organization across the entire project
- **Added**: More detailed descriptions for each crate's purpose and functionality
### Computer Module
- **Refactored**: Converted ComputedFrom pattern to LazyFrom pattern for improved performance
- **Added**: New LazyVecBuilder implementation for on-demand computation
- **Restructured**: Computer module stateful operations with improved organization
- **Cleaned**: Removed hardcoded format specifications, now using defaults
- **Optimized**: Vector storage operations with lazy computation patterns
- **Refactored**: Address type handling in stateful computations
- **Reorganized**: Stateful module structure for better maintainability
- **Enhanced**: Builder patterns for grouped vector operations
- **Improved**: Memory efficiency through lazy computation strategies
### Vector Storage
- **Enhanced**: Vector storage engine development workflow
- **Optimized**: Lazy vector computation patterns for better performance
- **Improved**: Vector builder architecture with lazy evaluation
### Dependencies
- **Updated**: Rayon from 1.10.0 to 1.11.0 for improved parallel processing
- **Migrated**: vecdb from external version 0.1.0 to local development path for better development workflow
- **Reorganized**: Workspace dependency structure for consistency
### Build System
- **Added**: Rust toolchain version specification (1.89) to workspace configuration
- **Enhanced**: Development environment consistency with toolchain pinning
### Server Module
- **Improved**: API interface implementations
- **Enhanced**: File serving functionality
### Indexer Module
- **Optimized**: Integration with updated computer module patterns
## [v0.0.88](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.88) - 2025-08-10
### Website
- **Fixed**: Conditional rendering of adjusted SOPR (Adjusted Spent Output Profit Ratio) charts in website options
- **Improved**: SOPR chart options now only display adjusted SOPR when the data is available in the vector indexes
- **Enhanced**: Better error handling for missing adjusted SOPR datasets
### Build System
- **Updated**: Added `rust-version.workspace = true` to all crate Cargo.toml files
- **Updated**: Bumped all crate versions from 0.0.87 to 0.0.88
- **Added**: Minimum supported Rust version constraint (1.89) to workspace configuration
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.87...v0.0.88)
## [v0.0.87](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.87) - 2025-08-10
### Major Architecture Change
- **BREAKING**: Extracted `brk_vecs` into separate external crate `vecdb`
- **Moved**: Vector storage engine to separate [seqdb](https://github.com/seqdb/seqdb) repository
- **Updated**: All crates now use `vecdb` instead of internal `brk_vecs`
- **Improved**: Better separation of concerns with vector storage as independent library
### Storage Engine
- **Renamed**: `brk_vecs``vecdb`
- **Renamed**: `brk_vecs_macros``vecdb_derive`
- **Updated**: All vector database operations now use `vecdb::Database` instead of `brk_vecs::File`
- **Enhanced**: Vector storage is now a standalone crate with its own versioning
### Dependencies
- **Updated**: `brk_rmcp` from 0.4.1 → 0.5.0
- **Updated**: `cc` from 1.2.31 → 1.2.32
- **Updated**: `hashbrown` from 0.15.4 → 0.15.5
- **Updated**: `rapidhash` from 3.0.0 → 3.1.0
- **Updated**: `rustversion` from 1.0.21 → 1.0.22
- **Updated**: `slab` from 0.4.10 → 0.4.11
### Configuration
- **Added**: `--exchanges` flag to control whether to fetch prices from exchange APIs
- **Enhanced**: Price fetching now has two levels - BRK API and exchanges
- **Improved**: Better RPC authentication error messages with troubleshooting guidance
### Build System
- **Updated**: All crate versions from 0.0.85 to 0.0.87
- **Added**: New dependency on external `vecdb` crate with derive features
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.85...v0.0.87)
## [v0.0.85](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.85) - 2025-08-07
### MCP (Model Context Protocol)
- **Updated**: Upgraded MCP implementation
- **Changed**: Made MCP stateless mode the default (was previously stateful to avoid breaking Claude)
- **Improved**: Better MCP server configuration with stateless operations
### Logging
- **Enhanced**: Updated logging utilities in `brk_logger`
### Build System
- **Updated**: Bumped version from 0.0.84 to 0.0.85
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.84...v0.0.85)
## [v0.0.84](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.84) - 2025-08-07
### Build System
- **Added**: Per-crate `build.rs` files for better build configuration
- **Improved**: Cargo.lock file management and error handling
- **Enhanced**: Build process optimization across all crates
### Vector Storage
- **Fixed**: Compressed vector functionality restored
- **Fixed**: Race condition in vector operations
- **Improved**: Vector storage stability and performance
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.83...v0.0.84)
## [v0.0.83](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.83) - 2025-07-26
### Vector Storage
- **Added**: Linux punch hole implementation for better disk space management
- **Enhanced**: Memory-mapped file handling improvements
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.82...v0.0.83)
## [v0.0.82](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.82) - 2025-07-26
### Dependencies
- **Updated**: Cleaned up Cargo dependencies
- **Removed**: Temporarily removed rayon parallelization from computer module
- **Enhanced**: Dependency management and build optimization
### Computer Module
- **Fixed**: File initialization with minimum length and regions
- **Improved**: Memory management and performance optimizations
- **Added**: Flush and punch operations for better disk usage
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.81...v0.0.82)
## [v0.0.81](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.81) - 2025-07-17
### Vector Storage
- **Fixed**: Holes export functionality in vector storage
- **Enhanced**: Vector storage reliability
### MCP Integration
- **Updated**: Now uses `rust-rmcp` instead of `brk-rmcp` for better compatibility
### Computer Module
- **Improved**: Performance optimizations
- **Fixed**: Stateful operations
- **Enhanced**: Vector-to-store conversion process (parts 1 & 2)
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.80...v0.0.81)
## [v0.0.80](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.80) - 2025-07-13
### MCP (Model Context Protocol)
- **Removed**: Claude results examples due to dead links
- **Cleaned**: MCP example documentation
### Vector Storage
- **Added**: Local and shared `stored_len` tracking to raw vector variant
- **Enhanced**: Vector storage metadata management
### Dependencies
- **Updated**: Multiple crate upgrades for better compatibility
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.79...v0.0.80)
## [v0.0.79](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.79) - 2025-07-13
### Dependencies
- **Updated**: Upgraded multiple crates for better compatibility and performance
### Website
- **Applied**: Dataset changes to chart configurations
- **Enhanced**: Chart display with updated dataset handling
### Vector Storage
- **Fixed**: Header reading functionality for existing vector files
- **Improved**: Memory management by removing mmap storage from struct
- **Enhanced**: Vector file handling reliability
### Store Operations
- **Reverted**: Bloom filters back to default settings due to slow read performance
- **Optimized**: Memory usage for future v3 improvements
- **Fixed**: Missing store-related files
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.78...v0.0.79)
## [v0.0.78](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.78) - 2025-07-13
### Build System
- **Updated**: All crate versions bumped for version consistency across workspace
- **Enhanced**: Version consistency across workspace
### Computer Module
- **Optimized**: Reduced number of ratio datasets for some cohorts to improve performance
- **Enhanced**: Performance optimizations across computation modules
- **Improved**: Resource utilization and memory management
- **Fixed**: Coarse lazy indexes functionality
- **Removed**: Debug statements for cleaner production builds
### Global Improvements
- **Added**: Semester-based time intervals
- **Changed**: Coarser intervals now computed instead of eager evaluation
- **Fixed**: Various global module issues
- **Enhanced**: Performance and resource improvements
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.76...v0.0.78)
## [v0.0.76](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.76) - 2025-07-09
### Bundler
- **Updated**: Upgraded Rolldown dependency for better bundling performance
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.75...v0.0.76)
## [v0.0.75](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.75) - 2025-07-09
### Computer Module
- **Fixed**: Reduced parallel thread count for `compute_rest_part2` to prevent external drive crashes
- **Improved**: Better handling of external storage devices during computation
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.74...v0.0.75)
## [v0.0.74](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.74) - 2025-07-09
### Computer Module
- **Fixed**: Parallel computation crashes on external drives
- **Enhanced**: Storage device compatibility improvements
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.73...v0.0.74)
## [v0.0.73](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.73) - 2025-07-09
### Website
- **Fixed**: Chart panes now maintain equal sizes
- **Removed**: Automatic saving of pane heights preference
- **Updated**: Lightweight Charts to v5.0.8
- **Added**: TODO functionality initialization
- **Enhanced**: Website development snapshots
### Dependencies
- **Updated**: Global dependency upgrades across all crates
- **Enhanced**: System-wide dependency management
### Computer Module
- **Added**: More up-to and from datasets for enhanced analysis
- **Implemented**: Store functionality (parts 2-10)
- **Enhanced**: Data storage architecture with store-based approach
- **Replaced**: Value enum with Cow for better memory efficiency
### Global System
- **Moved**: `addressindex_to_outputindex` stores from computer to indexer module
- **Renamed**: Various stores for better organization
- **Enhanced**: Global module fixes and improvements
### Indexer
- **Renamed**: Store organization for better clarity
- **Enhanced**: Store management improvements
### Documentation
- **Added**: MCP (Model Context Protocol) README documentation
- **Enhanced**: Developer documentation
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.71...v0.0.73)
## [v0.0.71](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.71) - 2025-06-25
### MCP (Model Context Protocol)
- **Enhanced**: Small improvements to MCP functionality
- **Updated**: MCP integration refinements
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.70...v0.0.71)
## [v0.0.70](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.70) - 2025-06-25
### Fetcher Module
- **Fixed**: BRK API URL endpoints
- **Enhanced**: Price fetching reliability
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.69...v0.0.70)
## [v0.0.69](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.69) - 2025-06-24
### Vector Storage
- **Reverted**: Storing file in struct (was overwhelming system resources)
- **Improved**: Memory management for vector operations
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.68...v0.0.69)
## [v0.0.68](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.68) - 2025-06-24
### Core System
- **Increased**: Maximum open files limit for better file handling
- **Enhanced**: System resource management
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.67...v0.0.68)
## [v0.0.67](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.67) - 2025-06-24
### Server Module
- **Changed**: Now uses ETag for vectors instead of date modified for better caching
- **Enhanced**: HTTP caching strategy improvements
### Vector Storage
- **Unified**: Single file with header approach for vector storage
- **Improved**: Vector file organization and management
### MCP Integration
- **Added**: MCP part 2 implementation
- **Refactored**: Server MCP integration
- **Enhanced**: MCP functionality expansion
### Global Improvements
- **Renamed**: Various indexes for better clarity
- **Updated**: Index naming conventions
- **Fixed**: Web index type imports
### CLI
- **Removed**: Duplicate 'h' short index in query functionality
### Documentation
- **Updated**: README improvements
- **Updated**: MCP documentation
- **Updated**: Changelog titles
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.66...v0.0.67)
## [v0.0.66](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.66) - 2025-06-19
### Cointime Implementation
- **Added**: Cointime functionality part 1
- **Implemented**: Cumulative destroyed coinblocks and cointime calculations
- **Enhanced**: Bitcoin time-based analytics
### Website
- **Fixed**: Options cumulative possible vecids type issue
- **Enhanced**: Chart options handling
### Documentation
- **Merged**: Pull request #18 with typo fixes from StevenBlack
- **Improved**: Code quality and documentation accuracy
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.65...v0.0.66)
## [v0.0.65](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.65) - 2025-06-17
### Website
- **Fixed**: Hang issue when candles fetched are slightly different than before
- **Enhanced**: Chart data fetching reliability
### Documentation
- **Improved**: Minor refinements to brk_cli README
- **Enhanced**: CLI documentation clarity
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.64...v0.0.65)
## [v0.0.64](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.64) - 2025-06-17
### Datasets
- **Added**: Multiple new realized datasets and charts
- **Added**: 4-year Z-score calculations
- **Added**: 200-day Simple Moving Average (SMA)
- **Added**: Mayer's Multiple indicator
- **Enhanced**: Financial analysis capabilities
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.63...v0.0.64)
## [v0.0.63](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.63) - 2025-06-16
### Website
- **Fixed**: Web interface stutter issue (follow-up fix)
- **Added**: PWA (Progressive Web App) assets
- **Enhanced**: User experience improvements
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.62...v0.0.63)
## [v0.0.62](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.62) - 2025-06-16
### Website
- **Fixed**: Stutter on update and save default chart settings to URL parameters
- **Enhanced**: Chart settings persistence
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.61...v0.0.62)
## [v0.0.61](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.61) - 2025-06-15
### Website
- **Fixed**: Error in Safari lockdown mode
- **Improved**: Charts now update instead of setData when possible for better performance
### Computer Module
- **Fixed**: OHLC data opening when fetched from different API than previous OHLC (preparation fix)
- **Enhanced**: API data source consistency
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.59...v0.0.61)
## [v0.0.59](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.59) - 2025-06-15
### Computer Module
- **Fixed**: Open of OHLC data when fetched from different API than previous OHLC
- **Enhanced**: API data consistency handling
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.58...v0.0.59)
## [v0.0.58](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.58) - 2025-06-15
### Bundler
- **Fixed**: BRK bundler usage
- **Removed**: HTML minify crate for improved bundling
- **Deployed**: brk_rolldown bundler with edge case fixes
- **Initialized**: Working bundler version
- **Added**: Basic bundling functionality
### Server
- **Updated**: Cache control for bundled websites (continued improvements)
- **Enhanced**: Website serving performance
- **Enhanced**: Static file serving performance
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.56...v0.0.58)
## [v0.0.56](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.56) - 2025-06-13
### Computer Module
- **Enhanced**: Stateful operations now reset when blockchain reorganization detected
- **Improved**: Blockchain state management reliability
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.55...v0.0.56)
## [v0.0.55](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.55) - 2025-06-13
### Global
- **Fixed**: Multiple system-wide fixes and improvements
- **Enhanced**: Overall system stability
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.54...v0.0.55)
## [v0.0.54](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.54) - 2025-06-12
### Website
- **Enhanced**: Filter possible index choices in charts (initial implementation)
- **Enhanced**: Filter possible index choices in charts
- **Fixed**: CSS styling issues
- **Added**: Auto price series type for default website
- **Added**: Live price functionality
- **Removed**: ScrollToSelected functionality
### Vector Storage
- **Fixed**: Compressed vector functionality (ongoing improvements)
- **Fixed**: Compressed vector functionality (still has slow parallel read performance)
- **Changed**: CLI defaults to raw format for performance
- **Changed**: CLI now defaults to raw format for better performance
### Indexer
- **Enhanced**: Raw format only support
- **Limited**: Now only supports raw format
- **Fixed**: Global system fixes
- **Fixed**: Various global fixes
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.52...v0.0.54)
## [v0.0.52](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.52) - 2025-06-11
### Website
- **Fixed**: Service worker and related components
- **Enhanced**: Progressive Web App functionality
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.51...v0.0.52)
## [v0.0.51](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.51) - 2025-06-11
### Website
- **Enhanced**: Default website improvements and fixes
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.50...v0.0.51)
## [v0.0.50](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.50) - 2025-06-11
### Website
- **Fixed**: Minimum bar spacing for charts
- **Enhanced**: Chart display improvements
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.49...v0.0.50)
## [v0.0.49](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.49) - 2025-06-11
### Build System
- **Updated**: Set full version of all crates for better versioning consistency
- **Enhanced**: Cargo workspace version management
### Website
- **Updated**: Multiple default website snapshots and improvements
### Documentation
- **Updated**: README improvements
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.48...v0.0.49)
## [v0.0.48](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.48) - 2025-06-09
### Website
- **Enhanced**: Default website snapshots and continued development
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.47...v0.0.48)
## [v0.0.47](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.47) - 2025-06-08
### Server
- **Added**: Documentation for index-t-value functionality
- **Added**: Support for `/api/X-to-Y` endpoints
- **Fixed**: Query CLI functionality
- **Added**: Meta API endpoints for better API discoverability
### CLI
- **Added**: Count parameter to query functionality
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.46...v0.0.47)
## [v0.0.46](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.46) - 2025-06-08
### Server
- **Added**: DDoS protection for better server security
- **Enhanced**: Request rate limiting and protection
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.45...v0.0.46)
## [v0.0.45](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.45) - 2025-06-08
### Website
- **Moved**: Service worker to root directory
- **Updated**: Service worker functionality
- **Enhanced**: PWA capabilities
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.44...v0.0.45)
## [v0.0.44](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.44) - 2025-06-07
### Server
- **Fixed**: Existing folder endpoints functionality
- **Enhanced**: File serving reliability
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.43...v0.0.44)
## [v0.0.43](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.43) - 2025-06-07
### Website
- **Updated**: Dependencies and fixed CSS issues
- **Enhanced**: Styling and layout improvements
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.42...v0.0.43)
## [v0.0.42](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.42) - 2025-06-07
### Global System
- **Fixed**: Multiple system fixes and improvements
- **Enhanced**: System-wide stability improvements
- **Resolved**: Data accuracy issues that were causing incorrect results
### Fetcher
- **Added**: Support for new API endpoints
- **Enhanced**: Data fetching capabilities
### Computer Module
- **Fixed**: Coinblocks overflow issue
- **Enhanced**: Calculation accuracy and stability
- **Increased**: Flush frequency for better data persistence
- **Fixed**: Eager mode functionality
- **Enhanced**: Performance and reliability
### Documentation
- **Reset**: Changelog for clean organization
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.40...v0.0.42)
## [v0.0.40](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.40) - 2025-05-25
### Global System
- **Fixed**: Multiple system-wide issues
- **Enhanced**: Overall stability and performance
- **Resolved**: Data accuracy issues in global computations
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.39...v0.0.40)
## [v0.0.39](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.39) - 2025-05-25
### UTXO System
- **Implemented**: UTXO dataset functionality (initial part 1)
- **Implemented**: UTXO dataset functionality (part 1)
- **Added**: Foundation for unspent transaction output analysis
- **Added**: Unspent transaction output tracking and analysis
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.37...v0.0.39)
## [v0.0.37](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.37) - 2025-05-14
### DCA Analysis
- **Added**: Dollar-Cost Averaging (DCA) classes and analysis
- **Enhanced**: Investment strategy analytics
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.36...v0.0.37)
## [v0.0.36](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.36) - 2025-05-13
### Website
- **Added**: Price line functionality to charts
- **Enhanced**: Chart visualization capabilities
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.35...v0.0.36)
## [v0.0.35](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.35) - 2025-05-12
### Financial Analytics
- **Added**: Investment return analysis (lump sum vs DCA)
- **Added**: Average and ratio datasets
- **Added**: Market Simple Moving Averages (SMAs)
- **Enhanced**: Financial analysis tools
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.34...v0.0.35)
## [v0.0.34](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.34) - 2025-05-09
### Website
- **Added**: Market charts functionality
- **Enhanced**: Chart display and interaction
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.33...v0.0.34)
## [v0.0.33](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.33) - 2025-05-08
### Vector Storage
- **Fixed**: Computed vector eager path functionality
- **Enhanced**: Path deletion for lazy vectors
- **Improved**: Vector storage reliability
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.32...v0.0.33)
## [v0.0.32](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.32) - 2025-05-08
### Vector Storage - Lazy Computation
- **Completed**: Lazy computation implementation
- **Enhanced**: On-demand vector calculation
- **Improved**: Memory efficiency for large datasets
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.31...v0.0.32)
## [v0.0.31](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.31) - 2025-04-30
### Global System
- **Fixed**: Multiple system fixes and improvements
- **Updated**: Fixed old X (Twitter) links that directed to impersonator accounts
- **Enhanced**: Link security and accuracy
### Computer Module
- **Removed**: Last indexes in favor of count-based approach
- **Enhanced**: Index management efficiency
### Vector Storage
- **Improved**: Vector iteration with `into_iter` for range collection
- **Enhanced**: Iterator performance and memory usage
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.30...v0.0.31)
## [v0.0.30](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.30) - 2025-04-23
### Website
- **Fixed**: Chart data fetching issues
- **Enhanced**: Data loading reliability
### Global System
- **Added**: Pay-to-Anchor (P2A) support
- **Enhanced**: Bitcoin script type support
- **Improved**: Transaction parsing capabilities
### Storage System
- **Unified**: Single keyspace for all stores using fjall
- **Enhanced**: Storage efficiency and organization
### Vector Storage
- **Implemented**: Lazy/eager computation model
- **Enhanced**: Flexible computation strategies
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.29...v0.0.30)
## [v0.0.29](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.29) - 2025-04-22
### Website (Kibo)
- **Fixed**: TypeScript ignore additions for better compilation
- **Enhanced**: Database functionality (part 2)
- **Improved**: Database integration and management
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.28...v0.0.29)
## [v0.0.28](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.28) - 2025-04-19
### Website (Kibo)
- **Implemented**: Database functionality (part 1)
- **Added**: Initial database integration for kibo website
### Computer Module
- **Enhanced**: Computer + kibo integration (part 14) with fixes
- **Improved**: Cross-module functionality
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.27...v0.0.28)
## [v0.0.27](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.27) - 2025-04-18
### Distribution
- **Fixed**: Tag version formatting issues
- **Enhanced**: Release packaging improvements
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.26...v0.0.27)
## [v0.0.26](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.26) - 2025-04-18
### Distribution
- **Fixed**: Ubuntu version compatibility
- **Enhanced**: Linux distribution support
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.25...v0.0.26)
## [v0.0.25](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.25) - 2025-04-18
### Distribution
- **Updated**: Ubuntu version for better compatibility
- **Added**: Manual trigger for GitHub Actions
- **Enhanced**: CI/CD pipeline flexibility
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.24...v0.0.25)
## [v0.0.24](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.24) - 2025-04-18
### Dependencies
- **Updated**: Upgraded multiple crates for better performance
- **Enhanced**: Dependency management
### Computer Module
- **Enhanced**: Computer + kibo integration (part 13)
- **Cleanup**: Kibo-related code organization
- **Continued**: Computer module development (part 12)
### Documentation
- **Updated**: Parser README documentation
- **Enhanced**: Documentation clarity and completeness
- **Updated**: Discord community links
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.23...v0.0.24)
## [v0.0.23](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.23) - 2025-04-14
### Dependencies
- **Updated**: Dependency upgrades across the workspace
- **Enhanced**: Compatibility and performance
- **Enhanced**: System compatibility
### Computer Module
- **Enhanced**: Computer + kibo integration (parts 10-11)
- **Improved**: Cross-component functionality
- **Improved**: Cross-module functionality
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.21...v0.0.23)
## [v0.0.21](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.21) - 2025-04-11
### Computer Module
- **Enhanced**: Computer + kibo integration (part 9)
- **Continued**: Ongoing computer module development
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.20...v0.0.21)
## [v0.0.20](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.20) - 2025-04-10
### Global Cleanup
- **Cleaned**: Old files and legacy code
- **Enhanced**: Codebase organization
### Vector Storage
- **Reworked**: Vector storage implementation (parts 1-4)
- **Optimized**: Performance improvements
- **Enhanced**: Memory efficiency
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.19...v0.0.20)
## [v0.0.19](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.19) - 2025-04-07
### Computer Module
- **Enhanced**: Computer functionality (parts 5-7)
- **Improved**: Data processing capabilities
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.18...v0.0.19)
## [v0.0.18](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.18) - 2025-04-05
### Computer Module
- **Enhanced**: Computer functionality (part 4)
- **Continued**: Core computer development
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.17...v0.0.18)
## [v0.0.17](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.17) - 2025-04-05
### Distribution
- **Moved**: Configuration to `config.toml` format
- **Enhanced**: Configuration management
### Storage
- **Initialized**: Disk-based storage functionality
- **Enhanced**: Data persistence capabilities
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.16...v0.0.17)
## [v0.0.16](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.16) - 2025-04-04
### Server
- **Fixed**: Repository version path issues
- **Enhanced**: Static file serving
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.15...v0.0.16)
## [v0.0.15](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.15) - 2025-04-04
### Server
- **Fixed**: Missing 'v' in version handling
- **Enhanced**: Version management
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.14...v0.0.15)
## [v0.0.14](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.14) - 2025-04-04
### Server
- **Fixed**: Downloaded repository version path
- **Enhanced**: File path resolution
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.13...v0.0.14)
## [v0.0.13](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.13) - 2025-04-04
### Global System
- **Enhanced**: System-wide improvements and snapshots
- **Fixed**: Various stability issues
### Documentation
- **Added**: Warning notices in README
- **Enhanced**: User guidance and safety information
### Website (Kibo)
- **Fixed**: Simulation functionality
- **Enhanced**: User interface improvements
- **Converted**: TypeScript types to JSDoc for better compatibility
- **Upgraded**: Signals library to tresshaked v0.2.4
### Server
- **Added**: API documentation
- **Enhanced**: Development experience
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.12...v0.0.13)
## [v0.0.12](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0.12) - 2025-04-02
### Server
- **Enhanced**: API documentation and functionality
- **Improved**: Developer experience
### Website (Kibo)
- **Enhanced**: Various improvements (parts 1-5)
- **Changed**: Font from Satoshi to Geist
- **Improved**: Typography and design
### Documentation
- **Updated**: README files with badges and shields
- **Enhanced**: Project presentation
### Dependencies
- **Updated**: Crate updates across workspace
- **Enhanced**: Compatibility and performance
[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0...v0.0.12)
## [v0.0](https://github.com/bitcoinresearchkit/brk/releases/tag/v0.0) - 2025-02-23
### Major Project Rewrite
- **BREAKING**: Complete rewrite from legacy "kibo" to modern "BRK" (Bitcoin Research Kit)
- **New**: Modular Rust workspace architecture with separate crates
- **Added**: Modern build system with Cargo workspace
- **Created**: New vector storage engine (`brk_vecs`, later became `vecdb`)
- **Established**: Clear separation between parser, indexer, computer, and server modules
- **Added**: MCP (Model Context Protocol) integration
- **Added**: RESTful API server with multiple output formats
- **Added**: Comprehensive CLI interface
- **Enhanced**: Performance and scalability improvements
---
## Previous Releases (Legacy from kibo project)
<!--
# v0.X.Y | WIP
![Image of the kibo Web App version 0.X.Y](https://github.com/bitcoinresearchkit/brk/blob/main/assets/v0.X.Y.jpg)
Generated
+233 -200
View File
File diff suppressed because it is too large Load Diff
+20 -18
View File
@@ -4,10 +4,11 @@ members = ["crates/*"]
package.description = "The Bitcoin Research Kit is a suite of tools designed to extract, compute and display data stored on a Bitcoin Core node"
package.license = "MIT"
package.edition = "2024"
package.version = "0.0.87"
package.version = "0.0.89"
package.homepage = "https://bitcoinresearchkit.org"
package.repository = "https://github.com/bitcoinresearchkit/brk"
package.readme = "README.md"
package.rust-version = "1.89"
[profile.release]
lto = "fat"
@@ -25,21 +26,19 @@ inherits = "release"
axum = "0.8.4"
bitcoin = { version = "0.32.7", features = ["serde"] }
bitcoincore-rpc = "0.19.0"
brk_bundler = { version = "0.0.87", path = "crates/brk_bundler" }
brk_cli = { version = "0.0.87", path = "crates/brk_cli" }
brk_computer = { version = "0.0.87", path = "crates/brk_computer" }
brk_structs = { version = "0.0.87", path = "crates/brk_structs" }
brk_error = { version = "0.0.87", path = "crates/brk_error" }
brk_fetcher = { version = "0.0.87", path = "crates/brk_fetcher" }
brk_indexer = { version = "0.0.87", path = "crates/brk_indexer" }
brk_interface = { version = "0.0.87", path = "crates/brk_interface" }
brk_logger = { version = "0.0.87", path = "crates/brk_logger" }
brk_mcp = { version = "0.0.87", path = "crates/brk_mcp" }
brk_parser = { version = "0.0.87", path = "crates/brk_parser" }
brk_server = { version = "0.0.87", path = "crates/brk_server" }
brk_store = { version = "0.0.87", path = "crates/brk_store" }
vecdb = { version = "0.1.0", features = ["derive"]}
# vecdb = { path = "../seqdb/crates/vecdb", features = ["derive"]}
brk_bundler = { version = "0.0.89", path = "crates/brk_bundler" }
brk_cli = { version = "0.0.89", path = "crates/brk_cli" }
brk_computer = { version = "0.0.89", path = "crates/brk_computer" }
brk_error = { version = "0.0.89", path = "crates/brk_error" }
brk_fetcher = { version = "0.0.89", path = "crates/brk_fetcher" }
brk_indexer = { version = "0.0.89", path = "crates/brk_indexer" }
brk_interface = { version = "0.0.89", path = "crates/brk_interface" }
brk_logger = { version = "0.0.89", path = "crates/brk_logger" }
brk_mcp = { version = "0.0.89", path = "crates/brk_mcp" }
brk_parser = { version = "0.0.89", path = "crates/brk_parser" }
brk_server = { version = "0.0.89", path = "crates/brk_server" }
brk_store = { version = "0.0.89", path = "crates/brk_store" }
brk_structs = { version = "0.0.89", path = "crates/brk_structs" }
byteview = "=0.6.1"
derive_deref = "1.1.1"
fjall = "2.11.2"
@@ -47,12 +46,14 @@ jiff = "0.2.15"
log = "0.4.27"
minreq = { version = "2.14.0", features = ["https", "serde_json"] }
parking_lot = "0.12.4"
rayon = "1.10.0"
rayon = "1.11.0"
serde = "1.0.219"
serde_bytes = "0.11.17"
serde_derive = "1.0.219"
serde_json = { version = "1.0.142", features = ["float_roundtrip"] }
serde_json = { version = "1.0.143", features = ["float_roundtrip"] }
tokio = { version = "1.47.1", features = ["rt-multi-thread"] }
# vecdb = { path = "../seqdb/crates/vecdb", features = ["derive"]}
vecdb = { version = "0.2.4", features = ["derive"]}
zerocopy = "0.8.26"
zerocopy-derive = "0.8.26"
@@ -67,3 +68,4 @@ cargo-dist-version = "0.29.0"
ci = "github"
installers = []
targets = ["aarch64-apple-darwin", "aarch64-unknown-linux-gnu", "x86_64-unknown-linux-gnu"]
rust-toolchain-version = "1.89"
+21 -23
View File
@@ -52,6 +52,27 @@ The primary goal of this project is to be fully-featured and accessible for ever
In contrast, existing alternatives tend to be either [very costly](https://studio.glassnode.com/pricing) or missing essential features, with the vast majority being closed-source and unverifiable, which fundamentally undermines the principles of Bitcoin.
## Hosting as a service
If you'd like to have your own instance hosted for you please contact [hosting@bitcoinresearchkit.org](mailto:hosting@bitcoinresearchkit.org).
- 2 separate dedicated servers (1 GB/s each) with different ISPs and Cloudflare integration for enhanced performance and optimal availability
- 99.99% SLA
- Configured for speed
- Updates delivered at your convenience
- Direct communication for feature requests and support
- Bitcoin Core or Knots with desired version
- Optional subdomains: `*.bitcoinresearchkit.org`, `*.brekit.org`, `*.kibo.money` and `*.satonomics.xyz`
- Logo featured in the Readme if desired
Pricing: `0.01 BTC / month` *or* `0.1 BTC / year`
## Acknowledgments
Deepest gratitude to the [Open Sats](https://opensats.org/) public charity. Their grant — from December 2024 to the present — has been critical in sustaining this project.
Heartfelt thanks go out to every donor on [Nostr](https://primal.net/p/npub1jagmm3x39lmwfnrtvxcs9ac7g300y3dusv9lgzhk2e4x5frpxlrqa73v44) and [Geyser.fund](https://geyser.fund/project/brk) whose support has ensured the availability of the [bitcoinresearchkit.org](https://bitcoinresearchkit.org) public instance.
## Crates
- [`brk`](https://crates.io/crates/brk): A wrapper around all other `brk-*` crates
@@ -68,29 +89,6 @@ In contrast, existing alternatives tend to be either [very costly](https://studi
- [`brk_server`](https://crates.io/crates/brk_server): A server with an API for anything from BRK
- [`brk_store`](https://crates.io/crates/brk_store): A thin wrapper around [`fjall`](https://crates.io/crates/fjall)
- [`brk_structs`](https://crates.io/crates/brk_structs): Structs used throughout BRK
- [`vecdb`](https://crates.io/crates/vecdb): A KISS index-value storage engine
- [`vecdb_macros`](https://crates.io/crates/vecdb_macros): Macros for [`vecdb`](https://crates.io/crates/vecdb)
## Hosting as a service
If you'd like to have your own instance hosted for you please contact [hosting@bitcoinresearchkit.org](mailto:hosting@bitcoinresearchkit.org).
- 2 separate dedicated servers (1 GB/s each) with different ISPs and Cloudflare integration for enhanced performance and optimal availability
- 99.99% SLA
- Configured for speed
- Updates delivered at your convenience
- Direct communication for feature requests and support
- Bitcoin Core or Knots with desired version
- Optional subdomains: `*.bitcoinresearchkit.org`, `*.kibo.money` and `*.satonomics.xyz`
- Logo featured in the Readme if desired
Pricing: `0.01 BTC / month` *or* `0.1 BTC / year`
## Acknowledgments
Deepest gratitude to the [Open Sats](https://opensats.org/) public charity. Their grant — from December 2024 to the present — has been critical in sustaining this project.
Heartfelt thanks go out to every donor on [Nostr](https://primal.net/p/npub1jagmm3x39lmwfnrtvxcs9ac7g300y3dusv9lgzhk2e4x5frpxlrqa73v44) and [Geyser.fund](https://geyser.fund/project/brk) whose support has ensured the availability of the [bitcoinresearchkit.org](https://bitcoinresearchkit.org) public instance.
## Donate
+1 -3
View File
@@ -7,6 +7,7 @@ homepage.workspace = true
repository.workspace = true
edition.workspace = true
version.workspace = true
rust-version.workspace = true
build = "build.rs"
[features]
@@ -23,7 +24,6 @@ full = [
"server",
"store",
"structs",
"vecs",
]
bundler = ["brk_bundler"]
computer = ["brk_computer"]
@@ -37,7 +37,6 @@ parser = ["brk_parser"]
server = ["brk_server"]
store = ["brk_store"]
structs = ["brk_structs"]
vecs = ["vecdb"]
[dependencies]
brk_bundler = { workspace = true, optional = true }
@@ -53,7 +52,6 @@ brk_parser = { workspace = true, optional = true }
brk_server = { workspace = true, optional = true }
brk_store = { workspace = true, optional = true }
brk_structs = { workspace = true, optional = true }
vecdb = { workspace = true, optional = true }
[package.metadata.docs.rs]
all-features = true
+197
View File
@@ -0,0 +1,197 @@
# brk
**Main wrapper crate for the Bitcoin Research Kit (BRK)**
The `brk` crate serves as the primary entry point for the Bitcoin Research Kit, providing a unified interface to all BRK components through feature flags. It enables developers to selectively include only the components they need while maintaining a consistent API.
## What it provides
- **Unified Access**: Single crate providing access to the entire BRK ecosystem
- **Feature-based Selection**: Choose only the components you need
- **Consistent Versioning**: All components versioned together for compatibility
- **Simplified Dependencies**: Single dependency instead of multiple individual crates
## Available Components
### Core Data Pipeline
- **`parser`** ([brk_parser](../brk_parser/)) - High-performance Bitcoin block parser
- **`indexer`** ([brk_indexer](../brk_indexer/)) - Blockchain data indexer with dual storage
- **`computer`** ([brk_computer](../brk_computer/)) - Analytics engine for computed datasets
- **`interface`** ([brk_interface](../brk_interface/)) - Unified data query and formatting API
### Infrastructure
- **`structs`** ([brk_structs](../brk_structs/)) - Bitcoin-aware type system and data structures
- **`error`** ([brk_error](../brk_error/)) - Centralized error handling
- **`store`** ([brk_store](../brk_store/)) - Blockchain-aware key-value storage
### External Integration
- **`fetcher`** ([brk_fetcher](../brk_fetcher/)) - Bitcoin price data fetcher with multi-source fallback
- **`server`** ([brk_server](../brk_server/)) - HTTP server with REST API
- **`mcp`** ([brk_mcp](../brk_mcp/)) - Model Context Protocol for LLM integration
### Utilities
- **`cli`** ([brk_cli](../brk_cli/)) - Command line interface (always enabled)
- **`logger`** ([brk_logger](../brk_logger/)) - Logging utilities
- **`bundler`** ([brk_bundler](../brk_bundler/)) - Asset bundling for web interfaces
## Usage
### Full Installation
```toml
[dependencies]
brk = { version = "0.0.88", features = ["full"] }
```
```rust
use brk::*;
// Access all components
let config = cli::Config::load()?;
let parser = parser::Parser::new(/* ... */);
let indexer = indexer::Indexer::forced_import("./data")?;
let computer = computer::Computer::forced_import("./data", &indexer, None)?;
```
### Selective Components
```toml
[dependencies]
brk = { version = "0.0.88", features = ["parser", "indexer", "computer"] }
```
```rust
use brk::{parser, indexer, computer};
// Core data pipeline only
let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
let mut indexer = indexer::Indexer::forced_import(output_dir)?;
let mut computer = computer::Computer::forced_import(output_dir, &indexer, None)?;
```
### Minimal Setup
```toml
[dependencies]
brk = { version = "0.0.88", features = ["structs", "parser"] }
```
```rust
use brk::{structs, parser};
// Just parsing and types
let height = structs::Height::new(800_000);
let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
```
## Feature Flags
| Feature | Description | Dependencies |
|---------|-------------|--------------|
| `full` | Enable all components | All crates |
| `cli` | Command line interface | Always enabled |
| `structs` | Core type system | Foundation for other crates |
| `error` | Error handling | Used by most crates |
| `parser` | Block parsing | `structs`, `error` |
| `store` | Key-value storage | `structs`, `error` |
| `indexer` | Blockchain indexing | `parser`, `store` |
| `computer` | Analytics computation | `indexer`, `fetcher` (optional) |
| `fetcher` | Price data fetching | `structs`, `error` |
| `interface` | Data query API | `indexer`, `computer` |
| `server` | HTTP server | `interface`, `mcp` |
| `mcp` | LLM integration | `interface` |
| `logger` | Logging utilities | Standalone |
| `bundler` | Asset bundling | Standalone |
## Common Usage Patterns
### Complete BRK Instance
```rust
use brk::*;
// Full data pipeline setup
let config = cli::Config::load()?;
let rpc = /* Bitcoin Core RPC client */;
let parser = parser::Parser::new(config.blocks_dir, config.output_dir, rpc);
let mut indexer = indexer::Indexer::forced_import(&config.output_dir)?;
let mut computer = computer::Computer::forced_import(&config.output_dir, &indexer, None)?;
let interface = interface::Interface::build(&indexer, &computer);
let server = server::Server::new(interface, config.website_path);
// Start server
server.serve(true).await?;
```
### Data Analysis
```rust
use brk::{indexer, computer, interface};
// Analysis-focused setup
let indexer = indexer::Indexer::forced_import("./brk_data")?;
let computer = computer::Computer::forced_import("./brk_data", &indexer, None)?;
let interface = interface::Interface::build(&indexer, &computer);
// Query data
let params = interface::Params {
index: interface::Index::Height,
ids: vec!["price_usd", "difficulty"].into(),
rest: interface::ParamsOpt::default()
.set_from(-100)
.set_format(interface::Format::CSV),
};
let csv_data = interface.search_and_format(params)?;
```
### Custom Integration
```rust
use brk::{structs, parser, error};
// Custom application with BRK components
fn analyze_blocks() -> error::Result<()> {
let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
parser.parse(None, None)
.iter()
.take(1000) // First 1000 blocks
.for_each(|(height, block, hash)| {
println!("Block {}: {} transactions", height, block.txdata.len());
});
Ok(())
}
```
## Version Compatibility
All BRK crates are released together with synchronized versions. When using the `brk` wrapper crate, you're guaranteed compatibility between all components.
- **Current version**: 0.0.88
- **Rust MSRV**: 1.89+
- **Bitcoin Core**: v25.0 - v29.0
## Performance Characteristics
The `brk` crate itself adds no runtime overhead - it simply re-exports the underlying crates. Performance characteristics depend on which components you use:
- **Full pipeline**: ~13-15 hours initial sync, ~40GB storage overhead
- **Parser only**: ~4 minutes to parse entire blockchain
- **Indexer only**: ~7-8 hours to index blockchain, ~5-6GB RAM usage
- **Server**: Low latency API responses with caching and compression
## Dependencies
The `brk` crate's dependencies are determined by enabled features. Core dependencies include:
- `brk_cli` - Always included for configuration and CLI support
- Individual `brk_*` crates based on enabled features
- Transitive dependencies from enabled components
For specific dependency information, see individual crate READMEs.
---
*This README was generated by Claude Code*
-1
View File
@@ -1 +0,0 @@
fn main() {}
+1 -5
View File
@@ -1,4 +1,4 @@
#![doc = include_str!(concat!("../", env!("CARGO_PKG_README")))]
#![doc = include_str!("../README.md")]
#[cfg(feature = "bundler")]
#[doc(inline)]
@@ -50,7 +50,3 @@ pub use brk_server as server;
#[cfg(feature = "store")]
#[doc(inline)]
pub use brk_store as store;
#[cfg(feature = "vecs")]
#[doc(inline)]
pub use vecdb as vecs;
+1
View File
@@ -6,6 +6,7 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
+186
View File
@@ -0,0 +1,186 @@
# brk_bundler
**Asset bundling for BRK web interfaces using Rolldown**
`brk_bundler` provides JavaScript/TypeScript bundling capabilities for BRK's web interfaces. It's a thin wrapper around Rolldown (Rust-based Rollup alternative) with BRK-specific configuration for building optimized web assets with file watching and automatic rebuilding.
## What it provides
- **JavaScript Bundling**: Modern ES modules bundling with minification
- **File Watching**: Automatic rebuilding on source file changes
- **Asset Processing**: Copies and processes static assets
- **Version Injection**: Automatic version string replacement in service workers
- **Development Mode**: Live rebuilding for rapid development
## Key Features
### Bundling Capabilities
- **ES Module Support**: Modern JavaScript bundling with tree-shaking
- **Minification**: Automatic code minification for production builds
- **Source Maps**: Generated source maps for debugging
- **Entry Point Processing**: Configurable entry points with hashed output names
### File System Operations
- **Directory Copying**: Copies entire source directories to distribution
- **Selective Processing**: Special handling for specific file types
- **Path Resolution**: Automatic path resolution and asset linking
### Development Features
- **Hot Rebuilding**: Automatic rebuilds on file changes
- **Watch Mode**: Monitors source files and triggers rebuilds
- **Version Replacement**: Injects build version into service workers
## Usage
### Basic Bundling
```rust
use brk_bundler::bundle;
use std::path::Path;
// Bundle without watching (production)
let websites_path = Path::new("./websites");
let source_folder = "default";
let dist_path = bundle(websites_path, source_folder, false).await?;
println!("Bundled to: {:?}", dist_path);
```
### Development Mode with Watching
```rust
// Bundle with file watching (development)
let dist_path = bundle(websites_path, "default", true).await?;
// Bundler now watches for changes and rebuilds automatically
// This will run in the background until the process exits
```
### Integration with BRK CLI
```rust
// Typically called from brk_cli when serving websites
async fn setup_website(config: &Config) -> Result<PathBuf> {
let websites_path = config.websites_path();
let source_folder = match config.website_mode {
WebsiteMode::Default => "default",
WebsiteMode::Custom => "custom",
WebsiteMode::None => return Ok(PathBuf::new()),
};
// Bundle the website assets
let dist_path = bundle(websites_path, source_folder, config.dev_mode).await?;
Ok(dist_path)
}
```
## File Structure
The bundler expects this directory structure:
```
websites/
├── default/ # Default website source
│ ├── index.html # Main HTML file
│ ├── service-worker.js # Service worker (version injected)
│ ├── scripts/ # JavaScript/TypeScript source
│ │ ├── entry.js # Main entry point
│ │ ├── main.js # Application logic
│ │ └── ... # Other JS modules
│ └── assets/ # Static assets
└── dist/ # Generated output directory
├── index.html # Processed HTML with updated script references
├── service-worker.js # Service worker with version injected
├── scripts/ # Bundled and minified JavaScript
│ └── main-[hash].js # Hashed output file
└── assets/ # Copied static assets
```
## Bundling Process
1. **Clean**: Removes existing `dist/` directory
2. **Copy**: Copies all source files to `dist/`
3. **Bundle JavaScript**:
- Processes `scripts/entry.js` as entry point
- Generates minified bundle with source maps
- Creates hashed filename for cache busting
4. **Process HTML**: Updates script references to hashed filenames
5. **Process Service Worker**: Injects current version string
6. **Watch** (if enabled): Monitors for file changes and rebuilds
## Configuration
The bundler uses Rolldown with these optimized settings:
```rust
BundlerOptions {
input: Some(vec![source_entry.into()]), // scripts/entry.js
dir: Some("./dist/scripts".to_string()), // Output directory
cwd: Some(websites_path), // Working directory
minify: Some(RawMinifyOptions::Bool(true)), // Enable minification
sourcemap: Some(SourceMapType::File), // Generate source maps
..Default::default()
}
```
## File Watching
In watch mode, the bundler monitors:
- **Source files**: Non-script files are copied on change
- **JavaScript files**: Trigger full rebuild via Rolldown watcher
- **HTML files**: Processed to update script references
- **Service worker**: Version injection on changes
### Watch Events Handled
- `Create` - New files added
- `Modify` - Existing files changed
- Ignores `Delete` and other events
## Version Injection
Service workers get automatic version injection:
```javascript
// In source service-worker.js
const VERSION = '__VERSION__';
// After bundling
const VERSION = 'v0.0.88';
```
This enables proper cache invalidation across releases.
## Performance Features
- **Async Operations**: All bundling operations are async
- **Incremental Builds**: Only rebuilds changed files in watch mode
- **Parallel Processing**: Uses Tokio for concurrent file operations
- **Efficient Copying**: Direct file system operations
## Error Handling
- **Graceful Failures**: Logs errors but continues watching
- **Path Resolution**: Automatic path absolutization and validation
- **File System Errors**: Proper error propagation with context
## Dependencies
- `brk_rolldown` - Rust-based Rollup bundler
- `notify` - File system watching
- `tokio` - Async runtime for file operations
- `sugar_path` - Path manipulation utilities
- `log` - Error logging
## Integration Points
The bundler integrates with:
- **brk_cli**: Called during website setup
- **brk_server**: Serves bundled assets
- **Development workflow**: Provides live rebuilding
---
*This README was generated by Claude Code*
+2
View File
@@ -1,3 +1,5 @@
#![doc = include_str!("../README.md")]
use std::{
fs, io,
path::{Path, PathBuf},
+4 -3
View File
@@ -6,6 +6,7 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
@@ -19,15 +20,15 @@ brk_logger = { workspace = true }
brk_parser = { workspace = true }
brk_server = { workspace = true }
vecdb = { workspace = true }
clap = { version = "4.5.43", features = ["string"] }
clap_derive = "4.5.41"
clap = { version = "4.5.45", features = ["string"] }
clap_derive = "4.5.45"
color-eyre = "0.6.5"
log = { workspace = true }
minreq = { workspace = true }
serde = { workspace = true }
tokio = { workspace = true }
toml = "0.9.5"
zip = { version = "4.3.0", default-features = false, features = ["deflate"] }
zip = { version = "4.5.0", default-features = false, features = ["deflate"] }
[[bin]]
name = "brk"
+166 -67
View File
@@ -1,90 +1,189 @@
# BRK CLI
# brk_cli
<p align="left">
<a href="https://github.com/bitcoinresearchkit/brk">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
</a>
<a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
<img src="https://img.shields.io/crates/l/brk" alt="License" />
</a>
<a href="https://crates.io/crates/brk_cli">
<img src="https://img.shields.io/crates/v/brk_cli" alt="Version" />
</a>
<a href="https://docs.rs/brk_cli">
<img src="https://img.shields.io/docsrs/brk_cli" alt="Documentation" />
</a>
<img src="https://img.shields.io/crates/size/brk_cli" alt="Size" />
<a href="https://deps.rs/crate/brk_cli">
<img src="https://deps.rs/crate/brk_cli/latest/status.svg" alt="Dependency status">
</a>
<a href="https://discord.gg/HaR3wpH3nr">
<img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
</a>
<a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
<img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
</a>
</p>
**Command line interface for running complete BRK instances**
A command line interface to run a Bitcoin Research Kit instance.
`brk_cli` provides the main command-line interface for operating the Bitcoin Research Kit. It orchestrates the complete data pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with automatic configuration management and graceful operation.
It's very customizable with all parameters from the underlying tools (crates) used inside.
## What it provides
Run `brk -h` for more information.
- **Complete Pipeline Orchestration**: Coordinates parser, indexer, computer, and server components
- **Automatic Configuration**: Saves settings to `~/.brk/config.toml` for consistent operation
- **Continuous Operation**: Handles blockchain updates and incremental processing
- **Web Interface Options**: Configurable website serving (none, default, custom)
- **Graceful Shutdown**: Ctrl+C handling with proper cleanup
## Requirements
## Key Features
### Hardware
### Pipeline Management
- **Automatic dependency handling**: Ensures Bitcoin Core sync before processing
- **Incremental updates**: Only processes new blocks since last run
- **Error recovery**: Automatic retry logic and graceful error handling
- **Resource management**: Optimized memory usage and disk I/O
#### Recommended
### Configuration System
- **Auto-save configuration**: All CLI options saved to persistent config
- **Flexible paths**: Configurable Bitcoin directory, blocks directory, and output directory
- **RPC authentication**: Cookie file or username/password authentication
- **Data source options**: Configurable price fetching and exchange APIs
- [Latest base model Mac mini](https://www.apple.com/mac-mini/)
- [Thunderbolt 4 SSD enclosure](https://satechi.net/products/usb4-nvme-ssd-pro-enclosure/Z2lkOi8vc2hvcGlmeS9Qcm9kdWN0VmFyaWFudC80MDE4ODQ3MDA2NzI4OA==?queryID=7961465089021ee203a60db7e62e90d2)
- [2 TB NVMe SSD](https://shop.sandisk.com/products/ssd/internal-ssd/wd-black-sn850x-nvme-ssd?sku=WDS200T2X0E-00BCA0)
### Operation Modes
- **Initial sync**: Full blockchain processing from genesis
- **Continuous operation**: Real-time processing of new blocks
- **Update mode**: Resume from last processed block
- **Server mode**: HTTP API with optional web interface
#### Minimum
To be determined
### Software
- [Bitcoin](https://bitcoin.org/en/full-node)
- [Rust](https://www.rust-lang.org/tools/install)
- Unix based operating system (Mac OS or Linux)
> [!IMPORTANT]
> Ubuntu users need to install `open-ssl` via `sudo apt install libssl-dev pkg-config`
## Download
### Binaries
You can find a pre-built binary for your operating system in the [releases page](https://github.com/bitcoinresearchkit/brk/releases/latest).
### Cargo
## Installation
### Binary Release
```bash
# Install
cargo install brk --locked # or `cargo install brk_cli`, the result is the same
# Update
cargo install brk --locked # or `cargo install-update -a` if you have `cargo-update` installed
# Download from GitHub releases
# https://github.com/bitcoinresearchkit/brk/releases/latest
```
### Source
### Via Cargo
```bash
cargo install brk --locked
```
### From Source
```bash
git clone https://github.com/bitcoinresearchkit/brk.git
cd brk/crates/brk
cargo run -r
cd brk && cargo build --release
```
## Usage
Run `brk -h` to view each available parameter and their respective description.
### First Run (Configuration Setup)
> [!TIP]
> Every parameter set will be saved at `~/.brk/config.toml`, which allows you to simply run `brk` next time.
```bash
# Basic setup with default options
brk --brkdir ./my_brk_data
## Tunnel
# Full configuration
brk --bitcoindir ~/.bitcoin \
--brkdir ./brk_data \
--fetch true \
--exchanges true \
--website default
```
The easiest way to let others access your server is to use `cloudflared` which will also cache requests. For more information see [Cloudflare Tunnel](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/) documentation.
### Subsequent Runs
```bash
# Uses saved configuration from ~/.brk/config.toml
brk
# Override specific options
brk --website none --fetch false
```
### Command Line Options
```bash
brk --help
```
## Configuration Reference
All options are automatically saved to `~/.brk/config.toml`:
### Core Paths
- `--bitcoindir <PATH>` - Bitcoin Core directory (default: `~/.bitcoin`)
- `--blocksdir <PATH>` - Block files directory (default: `bitcoindir/blocks`)
- `--brkdir <PATH>` - BRK output directory (default: `~/.brk`)
### Data Sources
- `--fetch <BOOL>` - Enable price data fetching (default: `true`)
- `--exchanges <BOOL>` - Use exchange APIs for prices (default: `true`)
### Web Interface
- `--website <OPTION>` - Web interface mode:
- `none` - API only, no web interface
- `default` - Built-in web interface from `websites/default/`
- `custom` - Serve custom website from `websites/custom/`
### Bitcoin Core RPC
- `--rpcconnect <IP>` - RPC host (default: `localhost`)
- `--rpcport <PORT>` - RPC port (default: `8332`)
- `--rpccookiefile <PATH>` - Cookie authentication file
- `--rpcuser <USERNAME>` - Username authentication
- `--rpcpassword <PASSWORD>` - Password authentication
## Operation Flow
1. **Configuration Loading**: Loads saved config from `~/.brk/config.toml`
2. **Bitcoin Core Connection**: Establishes RPC connection and waits for sync
3. **Data Pipeline Initialization**: Sets up parser, indexer, computer, and interface
4. **Processing Loop**:
- Index new blocks from Bitcoin Core
- Compute analytics on new data
- Update cached data
5. **Server Startup**: Launches HTTP API with optional web interface
6. **Continuous Operation**: Monitors for new blocks and processes incrementally
## System Requirements
- **Bitcoin Core**: Fully synced node with RPC enabled
- **Storage**: ~32% of blockchain size (~233GB as of 2025)
- **Memory**:
- Peak: ~7-8GB during initial indexing
- Steady state: ~4-5GB during operation
- **OS**: macOS or Linux
- Ubuntu: `sudo apt install libssl-dev pkg-config`
## Performance Characteristics
### Initial Sync
- **Full blockchain processing**: ~13-15 hours total
- **Parser phase**: ~4 minutes for block parsing
- **Indexer phase**: ~7-8 hours for data indexing
- **Computer phase**: ~6-7 hours for analytics computation
### Continuous Operation
- **New block processing**: 3-5 seconds per block
- **API response times**: Typically <100ms with caching
- **Memory usage**: Stable ~4-5GB during normal operation
## Configuration File
Example `~/.brk/config.toml`:
```toml
bitcoindir = "/Users/username/.bitcoin"
blocksdir = "/Users/username/.bitcoin/blocks"
brkdir = "/Users/username/brk_data"
fetch = true
exchanges = true
website = "default"
rpcconnect = "localhost"
rpcport = 8332
rpccookiefile = "/Users/username/.bitcoin/.cookie"
```
## Error Handling
- **Bitcoin Core sync**: Waits for node sync before processing
- **RPC connection**: Automatic retry logic for connection issues
- **Processing errors**: Graceful error handling with detailed logging
- **Graceful shutdown**: Ctrl+C handling with proper cleanup and state saving
## Logging
Logs are written to `~/.brk/brk.log` with colored console output:
- Request/response logging with timing
- Processing progress indicators
- Error reporting and debugging information
## Dependencies
- `brk_parser` - Bitcoin block parsing
- `brk_indexer` - Blockchain data indexing
- `brk_computer` - Analytics computation
- `brk_interface` - Data query interface
- `brk_server` - HTTP API server
- `brk_logger` - Logging utilities
- `bitcoincore_rpc` - Bitcoin Core RPC client
- `color_eyre` - Enhanced error reporting
---
*This README was generated by Claude Code*
+2 -41
View File
@@ -42,7 +42,7 @@ pub struct Config {
#[arg(long, value_name = "BOOL")]
exchanges: Option<bool>,
/// Website served by the server (if active), default: default, saved
/// Website served by the server, default: default, saved
#[serde(default, deserialize_with = "default_on_error")]
#[arg(short, long)]
website: Option<Website>,
@@ -72,24 +72,9 @@ pub struct Config {
#[arg(long, value_name = "PASSWORD")]
rpcpassword: Option<String>,
/// Delay between runs, default: 0, saved
#[serde(default, deserialize_with = "default_on_error")]
#[arg(long, value_name = "SECONDS")]
delay: Option<u64>,
/// Activate the Model Context Protocol (MCP) endpoint to give LLMs access to BRK (experimental), default: true, saved
#[serde(default, deserialize_with = "default_on_error")]
#[arg(long, value_name = "BOOL")]
mcp: Option<bool>,
/// DEV: Activate watching the selected website's folder for changes, default: false, saved
#[serde(default, deserialize_with = "default_on_error")]
#[arg(long, value_name = "BOOL")]
watch: Option<bool>,
/// DEV: Activate checking address hashes for collisions when indexing, default: false, saved
#[serde(default, deserialize_with = "default_on_error")]
#[arg(long, value_name = "BOOL")]
#[arg(skip)]
check_collisions: Option<bool>,
}
@@ -150,22 +135,10 @@ impl Config {
config_saved.rpcpassword = Some(rpcpassword);
}
if let Some(delay) = config_args.delay.take() {
config_saved.delay = Some(delay);
}
if let Some(check_collisions) = config_args.check_collisions.take() {
config_saved.check_collisions = Some(check_collisions);
}
if let Some(mcp) = config_args.mcp.take() {
config_saved.mcp = Some(mcp);
}
if let Some(watch) = config_args.watch.take() {
config_saved.watch = Some(watch);
}
if config_args != Config::default() {
dbg!(config_args);
panic!("Didn't consume the full config")
@@ -258,10 +231,6 @@ Finally, you can run the program with '-h' for help."
self.rpcport
}
pub fn delay(&self) -> Option<u64> {
self.delay
}
pub fn bitcoindir(&self) -> PathBuf {
self.bitcoindir
.as_ref()
@@ -334,14 +303,6 @@ Finally, you can run the program with '-h' for help."
pub fn check_collisions(&self) -> bool {
self.check_collisions.is_some_and(|b| b)
}
pub fn mcp(&self) -> bool {
self.mcp.is_none_or(|b| b)
}
pub fn watch(&self) -> bool {
self.watch.is_some_and(|b| b)
}
}
fn default_on_error<'de, D, T>(deserializer: D) -> Result<T, D::Error>
+3 -11
View File
@@ -29,7 +29,7 @@ pub fn main() -> color_eyre::Result<()> {
fs::create_dir_all(dot_brk_path())?;
brk_logger::init(Some(&dot_brk_log_path()));
brk_logger::init(Some(&dot_brk_log_path()))?;
thread::Builder::new()
.stack_size(256 * 1024 * 1024)
@@ -108,9 +108,7 @@ pub fn run() -> color_eyre::Result<()> {
interface.generate_bridge_file(website, websites_path.as_path())?;
let watch = config.watch();
Some(bundle(&websites_path, website.to_folder_name(), watch).await?)
Some(bundle(&websites_path, website.to_folder_name(), true).await?)
} else {
None
};
@@ -120,10 +118,8 @@ pub fn run() -> color_eyre::Result<()> {
bundle_path,
);
let mcp = config.mcp();
tokio::spawn(async move {
server.serve(mcp).await.unwrap();
server.serve(true).await.unwrap();
});
sleep(Duration::from_secs(1));
@@ -140,10 +136,6 @@ pub fn run() -> color_eyre::Result<()> {
computer.compute(&indexer, starting_indexes, &exit).unwrap();
if let Some(delay) = config.delay() {
sleep(Duration::from_secs(delay))
}
info!("Waiting for new blocks...");
while block_count == rpc.get_block_count()? {
+2
View File
@@ -6,6 +6,7 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
@@ -20,6 +21,7 @@ brk_parser = { workspace = true }
vecdb = { workspace = true }
derive_deref = { workspace = true }
log = { workspace = true }
pco = "0.4.6"
rayon = { workspace = true }
serde = { workspace = true }
zerocopy = { workspace = true }
+198 -26
View File
@@ -1,28 +1,200 @@
# BRK Computer
# brk_computer
<p align="left">
<a href="https://github.com/bitcoinresearchkit/brk">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
</a>
<a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
<img src="https://img.shields.io/crates/l/brk" alt="License" />
</a>
<a href="https://crates.io/crates/brk_computer">
<img src="https://img.shields.io/crates/v/brk_computer" alt="Version" />
</a>
<a href="https://docs.rs/brk_computer">
<img src="https://img.shields.io/docsrs/brk_computer" alt="Documentation" />
</a>
<img src="https://img.shields.io/crates/size/brk_computer" alt="Size" />
<a href="https://deps.rs/crate/brk_computer">
<img src="https://deps.rs/crate/brk_computer/latest/status.svg" alt="Dependency status">
</a>
<a href="https://discord.gg/HaR3wpH3nr">
<img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
</a>
<a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
<img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
</a>
</p>
**Bitcoin analytics engine that transforms indexed blockchain data into comprehensive metrics**
A dataset computer, built on top of `brk_indexer` and `brk_fetcher`. It computes any dataset you can think of and if it doesn't feel free to create an issue.
`brk_computer` is the computational layer of BRK that processes indexed blockchain data to generate analytics across multiple specialized domains. It provides comprehensive Bitcoin metrics with efficient storage and lazy computation for optimal performance.
## What it provides
- **Comprehensive Analytics**: 9 specialized domains covering all aspects of Bitcoin analysis
- **Lazy Computation**: On-demand calculation with dependency tracking and caching
- **Incremental Updates**: Only processes new data since last computation
- **Memory Efficiency**: ~100MB operation footprint via compressed storage and memory mapping
- **Multi-timeframe Analysis**: Daily, weekly, monthly, quarterly, yearly perspectives
## Nine Analytics Domains
The computer processes data through a fixed dependency chain:
1. **indexes** - Time-based indexing (date/height mappings, epoch calculations)
2. **constants** - Baseline values and reference metrics
3. **blocks** - Block analytics (sizes, intervals, transaction counts, weight)
4. **mining** - Mining economics (hashrate, difficulty, rewards, epochs)
5. **fetched** - External price data integration (optional)
6. **price** - OHLC data across multiple timeframes (optional, requires fetched)
7. **transactions** - Transaction analysis (fees, sizes, patterns, RBF detection)
8. **market** - Price correlations and market metrics (optional, requires price)
9. **stateful** - UTXO tracking and accumulated state computations
10. **cointime** - Coin age and time-based value analysis
## Key Features
### Computation Strategy
- **Fixed dependency chain**: Ensures data consistency across all domains
- **Parallel processing**: Uses Rayon for performance optimization
- **State management**: Rollback capabilities for error recovery
- **Incremental updates**: Only computes new data since last run
### Analytics Capabilities
- **Multi-timeframe analysis**: Daily, weekly, monthly, quarterly, yearly aggregations
- **Chain-based metrics**: Height, difficulty epoch, halving epoch indexing
- **Price correlation**: Both dollar and satoshi denominated metrics
- **DCA analysis**: Dollar Cost Averaging with configurable periods
- **Supply analysis**: Circulating, realized, unrealized supply metrics
- **Address cohort tracking**: Analysis across different Bitcoin address types
- **UTXO cohort analysis**: Realized/unrealized gains tracking
- **Coin time analysis**: Understanding Bitcoin velocity and dormancy
### Storage Optimization
- **Compressed vectors**: Efficient disk storage with lazy computation
- **Memory mapping**: Minimal RAM usage during operation
- **Version management**: Automatic invalidation on schema changes
- **Dependency tracking**: Smart recomputation based on data changes
## Usage
### Basic Setup (No Price Data)
```rust
use brk_computer::Computer;
use brk_indexer::Indexer;
use vecdb::Exit;
// Setup without external price data
let indexer = Indexer::forced_import("./brk_data")?;
let mut computer = Computer::forced_import("./brk_data", &indexer, None)?;
// Setup exit handler
let exit = Exit::new();
exit.set_ctrlc_handler();
// Compute all analytics
let starting_indexes = indexer.get_starting_indexes();
computer.compute(&indexer, starting_indexes, &exit)?;
```
### Advanced Setup (With Price Data)
```rust
use brk_fetcher::Fetcher;
// Setup with external price data for market analytics
let fetcher = Some(Fetcher::import(true, None)?);
let mut computer = Computer::forced_import("./brk_data", &indexer, fetcher)?;
// Compute all analytics including price/market domains
computer.compute(&indexer, starting_indexes, &exit)?;
```
### Accessing Computed Data
```rust
// Access all computed vectors
let all_vecs = computer.vecs(); // Returns Vec<&dyn AnyCollectableVec>
// Access specific domain data
let block_metrics = &computer.blocks;
let mining_data = &computer.mining;
let transaction_stats = &computer.transactions;
// Access price data (if available)
if let Some(price_data) = &computer.price {
// Use OHLC data
}
```
### Incremental Updates
```rust
// Continuous computation loop
loop {
// Get latest indexes from indexer
let current_indexes = indexer.get_current_indexes();
// Compute only new data
computer.compute(&indexer, current_indexes, &exit)?;
// Check for exit signal
if exit.is_signaled() {
break;
}
// Wait before next update
sleep(Duration::from_secs(60));
}
```
## Core Computer Structure
```rust
pub struct Computer {
pub indexes: indexes::Vecs, // Time indexing
pub constants: constants::Vecs, // Baseline values
pub blocks: blocks::Vecs, // Block analytics
pub mining: mining::Vecs, // Mining economics
pub market: market::Vecs, // Market metrics (optional)
pub price: Option<price::Vecs>, // OHLC price data (optional)
pub transactions: transactions::Vecs, // Transaction analysis
pub stateful: stateful::Vecs, // UTXO tracking
pub fetched: Option<fetched::Vecs>, // External data (optional)
pub cointime: cointime::Vecs, // Coin age analysis
}
```
## Performance Characteristics
**Benchmarked on MacBook Pro M3 Pro:**
- **Initial computation**: ~6-7 hours for complete Bitcoin blockchain
- **Storage efficiency**: All computed datasets total ~40GB
- **Incremental updates**: 3-5 seconds per new block
- **Memory footprint**: Peak ~7-8GB during computation, ~100MB during operation
- **Dependencies**: Price data domains optional (fetched, price, market)
## Domain-Specific Analytics
### Block Analytics
- Block sizes, weights, transaction counts
- Block intervals and mining statistics
- Fee analysis per block
### Mining Economics
- Hashrate estimation and difficulty tracking
- Mining reward analysis
- Epoch-based calculations
### Transaction Analysis
- Fee rate distributions
- RBF (Replace-By-Fee) detection
- Output type analysis
- Transaction size patterns
### Market Metrics (Optional)
- Price correlations with on-chain metrics
- Market cap calculations
- DCA analysis across timeframes
### Stateful Analysis
- UTXO set tracking
- Address cohort analysis
- Realized/unrealized gains
- Supply distribution metrics
## Requirements
- **Indexed data**: Requires completed `brk_indexer` output
- **Storage space**: Additional ~40GB for computed datasets
- **Memory**: 8GB+ RAM recommended for initial computation
- **CPU**: Multi-core recommended for parallel processing
- **Price data**: Optional external price feeds for market analytics
## Dependencies
- `brk_indexer` - Source of indexed blockchain data
- `brk_fetcher` - External price data (optional)
- `vecdb` - Vector database with lazy computation
- `rayon` - Parallel processing framework
- `brk_structs` - Bitcoin-aware type system
---
*This README was generated by Claude Code*
+1 -1
View File
@@ -12,7 +12,7 @@ use brk_parser::Parser;
use vecdb::Exit;
pub fn main() -> Result<()> {
brk_logger::init(Some(Path::new(".log")));
brk_logger::init(Some(Path::new(".log")))?;
let bitcoin_dir = Path::new(&std::env::var("HOME").unwrap())
.join("Library")
@@ -0,0 +1,15 @@
use std::path::Path;
use brk_computer::PriceToAmount;
use brk_error::Result;
use brk_structs::Height;
pub fn main() -> Result<()> {
let path = Path::new(&std::env::var("HOME").unwrap())
.join(".brk")
.join("computed/stateful/states");
let mut price_to_amount = PriceToAmount::create(&path, "addrs_above_1btc_under_10btc");
dbg!(price_to_amount.import_at_or_before(Height::new(890000))?);
dbg!(price_to_amount);
Ok(())
}
+6 -30
View File
@@ -6,9 +6,7 @@ use brk_structs::{
CheckedSub, DifficultyEpoch, HalvingEpoch, Height, StoredU32, StoredU64, Timestamp, Version,
Weight,
};
use vecdb::{
AnyCollectableVec, Computation, Database, EagerVec, Exit, Format, PAGE_SIZE, VecIterator,
};
use vecdb::{AnyCollectableVec, Database, EagerVec, Exit, PAGE_SIZE, VecIterator};
use crate::grouped::Source;
@@ -37,30 +35,21 @@ pub struct Vecs {
}
impl Vecs {
pub fn forced_import(
parent: &Path,
version: Version,
computation: Computation,
format: Format,
indexes: &indexes::Vecs,
) -> Result<Self> {
pub fn forced_import(parent: &Path, version: Version, indexes: &indexes::Vecs) -> Result<Self> {
let db = Database::open(&parent.join("blocks"))?;
db.set_min_len(PAGE_SIZE * 1_000_000)?;
Ok(Self {
height_to_interval: EagerVec::forced_import(
height_to_interval: EagerVec::forced_import_compressed(
&db,
"interval",
version + VERSION + Version::ZERO,
format,
)?,
timeindexes_to_timestamp: ComputedVecsFromDateIndex::forced_import(
&db,
"timestamp",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_first(),
)?,
@@ -69,8 +58,6 @@ impl Vecs {
"block_interval",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_percentiles()
@@ -82,8 +69,6 @@ impl Vecs {
"block_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -92,8 +77,6 @@ impl Vecs {
"block_weight",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -102,38 +85,31 @@ impl Vecs {
"block_size",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
height_to_vbytes: EagerVec::forced_import(
height_to_vbytes: EagerVec::forced_import_compressed(
&db,
"vbytes",
version + VERSION + Version::ZERO,
format,
)?,
indexes_to_block_vbytes: ComputedVecsFromHeight::forced_import(
&db,
"block_vbytes",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
difficultyepoch_to_timestamp: EagerVec::forced_import(
difficultyepoch_to_timestamp: EagerVec::forced_import_compressed(
&db,
"timestamp",
version + VERSION + Version::ZERO,
format,
)?,
halvingepoch_to_timestamp: EagerVec::forced_import(
halvingepoch_to_timestamp: EagerVec::forced_import_compressed(
&db,
"timestamp",
version + VERSION + Version::ZERO,
format,
)?,
db,
+1 -49
View File
@@ -3,7 +3,7 @@ use std::path::Path;
use brk_error::Result;
use brk_indexer::Indexer;
use brk_structs::{Bitcoin, CheckedSub, Dollars, StoredF64, Version};
use vecdb::{AnyCollectableVec, Computation, Database, Exit, Format, PAGE_SIZE, VecIterator};
use vecdb::{AnyCollectableVec, Database, Exit, PAGE_SIZE, VecIterator};
use super::{
Indexes,
@@ -50,8 +50,6 @@ impl Vecs {
pub fn forced_import(
parent: &Path,
version: Version,
computation: Computation,
format: Format,
indexes: &indexes::Vecs,
price: Option<&price::Vecs>,
) -> Result<Self> {
@@ -66,8 +64,6 @@ impl Vecs {
"coinblocks_created",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -76,8 +72,6 @@ impl Vecs {
"coinblocks_stored",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -86,8 +80,6 @@ impl Vecs {
"liveliness",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -96,8 +88,6 @@ impl Vecs {
"vaultedness",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -106,8 +96,6 @@ impl Vecs {
"activity_to_vaultedness_ratio",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -116,8 +104,6 @@ impl Vecs {
"vaulted_supply",
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
VecBuilderOptions::default().add_last(),
compute_dollars,
indexes,
@@ -127,8 +113,6 @@ impl Vecs {
"active_supply",
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
VecBuilderOptions::default().add_last(),
compute_dollars,
indexes,
@@ -138,8 +122,6 @@ impl Vecs {
"thermo_cap",
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -148,8 +130,6 @@ impl Vecs {
"investor_cap",
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -158,8 +138,6 @@ impl Vecs {
"vaulted_cap",
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -168,8 +146,6 @@ impl Vecs {
"active_cap",
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -178,8 +154,6 @@ impl Vecs {
"vaulted_price",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -188,8 +162,6 @@ impl Vecs {
"vaulted_price",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
true,
)?,
@@ -198,8 +170,6 @@ impl Vecs {
"active_price",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -208,8 +178,6 @@ impl Vecs {
"active_price",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
true,
)?,
@@ -218,8 +186,6 @@ impl Vecs {
"true_market_mean",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -228,8 +194,6 @@ impl Vecs {
"true_market_mean",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
true,
)?,
@@ -238,8 +202,6 @@ impl Vecs {
"cointime_value_destroyed",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -248,8 +210,6 @@ impl Vecs {
"cointime_value_created",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -258,8 +218,6 @@ impl Vecs {
"cointime_value_stored",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -268,8 +226,6 @@ impl Vecs {
"cointime_price",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -278,8 +234,6 @@ impl Vecs {
"cointime_cap",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -288,8 +242,6 @@ impl Vecs {
"cointime_price",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
true,
)?,
+192 -17
View File
@@ -2,8 +2,8 @@ use std::path::Path;
use brk_error::Result;
use brk_indexer::Indexer;
use brk_structs::{StoredU16, Version};
use vecdb::{AnyCollectableVec, AnyVec, Computation, Database, Exit, Format};
use brk_structs::{StoredI16, StoredU16, Version};
use vecdb::{AnyCollectableVec, AnyVec, Database, Exit};
use crate::grouped::Source;
@@ -21,18 +21,19 @@ pub struct Vecs {
pub constant_0: ComputedVecsFromHeight<StoredU16>,
pub constant_1: ComputedVecsFromHeight<StoredU16>,
pub constant_2: ComputedVecsFromHeight<StoredU16>,
pub constant_3: ComputedVecsFromHeight<StoredU16>,
pub constant_4: ComputedVecsFromHeight<StoredU16>,
pub constant_50: ComputedVecsFromHeight<StoredU16>,
pub constant_100: ComputedVecsFromHeight<StoredU16>,
pub constant_minus_1: ComputedVecsFromHeight<StoredI16>,
pub constant_minus_2: ComputedVecsFromHeight<StoredI16>,
pub constant_minus_3: ComputedVecsFromHeight<StoredI16>,
pub constant_minus_4: ComputedVecsFromHeight<StoredI16>,
}
impl Vecs {
pub fn forced_import(
parent: &Path,
version: Version,
computation: Computation,
format: Format,
indexes: &indexes::Vecs,
) -> Result<Self> {
pub fn forced_import(parent: &Path, version: Version, indexes: &indexes::Vecs) -> Result<Self> {
let db = Database::open(&parent.join("constants"))?;
Ok(Self {
@@ -41,8 +42,6 @@ impl Vecs {
"constant_0",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -51,8 +50,30 @@ impl Vecs {
"constant_1",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
constant_2: ComputedVecsFromHeight::forced_import(
&db,
"constant_2",
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
VecBuilderOptions::default().add_last(),
)?,
constant_3: ComputedVecsFromHeight::forced_import(
&db,
"constant_3",
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
VecBuilderOptions::default().add_last(),
)?,
constant_4: ComputedVecsFromHeight::forced_import(
&db,
"constant_4",
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -61,8 +82,6 @@ impl Vecs {
"constant_50",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -71,8 +90,38 @@ impl Vecs {
"constant_100",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
constant_minus_1: ComputedVecsFromHeight::forced_import(
&db,
"constant_minus_1",
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
VecBuilderOptions::default().add_last(),
)?,
constant_minus_2: ComputedVecsFromHeight::forced_import(
&db,
"constant_minus_2",
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
VecBuilderOptions::default().add_last(),
)?,
constant_minus_3: ComputedVecsFromHeight::forced_import(
&db,
"constant_minus_3",
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
VecBuilderOptions::default().add_last(),
)?,
constant_minus_4: ComputedVecsFromHeight::forced_import(
&db,
"constant_minus_4",
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -134,6 +183,57 @@ impl Vecs {
},
)?;
self.constant_2.compute_all(
indexer,
indexes,
starting_indexes,
exit,
|vec, _, indexes, starting_indexes, exit| {
vec.compute_to(
starting_indexes.height,
indexes.height_to_date.len(),
indexes.height_to_date.version(),
|i| (i, StoredU16::new(2)),
exit,
)?;
Ok(())
},
)?;
self.constant_3.compute_all(
indexer,
indexes,
starting_indexes,
exit,
|vec, _, indexes, starting_indexes, exit| {
vec.compute_to(
starting_indexes.height,
indexes.height_to_date.len(),
indexes.height_to_date.version(),
|i| (i, StoredU16::new(3)),
exit,
)?;
Ok(())
},
)?;
self.constant_4.compute_all(
indexer,
indexes,
starting_indexes,
exit,
|vec, _, indexes, starting_indexes, exit| {
vec.compute_to(
starting_indexes.height,
indexes.height_to_date.len(),
indexes.height_to_date.version(),
|i| (i, StoredU16::new(4)),
exit,
)?;
Ok(())
},
)?;
self.constant_50.compute_all(
indexer,
indexes,
@@ -168,6 +268,74 @@ impl Vecs {
},
)?;
self.constant_minus_1.compute_all(
indexer,
indexes,
starting_indexes,
exit,
|vec, _, indexes, starting_indexes, exit| {
vec.compute_to(
starting_indexes.height,
indexes.height_to_date.len(),
indexes.height_to_date.version(),
|i| (i, StoredI16::new(-1)),
exit,
)?;
Ok(())
},
)?;
self.constant_minus_2.compute_all(
indexer,
indexes,
starting_indexes,
exit,
|vec, _, indexes, starting_indexes, exit| {
vec.compute_to(
starting_indexes.height,
indexes.height_to_date.len(),
indexes.height_to_date.version(),
|i| (i, StoredI16::new(-2)),
exit,
)?;
Ok(())
},
)?;
self.constant_minus_3.compute_all(
indexer,
indexes,
starting_indexes,
exit,
|vec, _, indexes, starting_indexes, exit| {
vec.compute_to(
starting_indexes.height,
indexes.height_to_date.len(),
indexes.height_to_date.version(),
|i| (i, StoredI16::new(-3)),
exit,
)?;
Ok(())
},
)?;
self.constant_minus_4.compute_all(
indexer,
indexes,
starting_indexes,
exit,
|vec, _, indexes, starting_indexes, exit| {
vec.compute_to(
starting_indexes.height,
indexes.height_to_date.len(),
indexes.height_to_date.version(),
|i| (i, StoredI16::new(-4)),
exit,
)?;
Ok(())
},
)?;
Ok(())
}
@@ -175,8 +343,15 @@ impl Vecs {
[
self.constant_0.vecs(),
self.constant_1.vecs(),
self.constant_2.vecs(),
self.constant_3.vecs(),
self.constant_4.vecs(),
self.constant_50.vecs(),
self.constant_100.vecs(),
self.constant_minus_1.vecs(),
self.constant_minus_2.vecs(),
self.constant_minus_3.vecs(),
self.constant_minus_4.vecs(),
]
.into_iter()
.flatten()
@@ -36,6 +36,15 @@ where
I: StoredIndex,
T: ComputedType,
{
pub fn forced_import_compressed(
db: &Database,
name: &str,
version: Version,
options: VecBuilderOptions,
) -> Result<Self> {
Self::forced_import(db, name, version, Format::Compressed, options)
}
pub fn forced_import(
db: &Database,
name: &str,
@@ -0,0 +1,395 @@
use brk_structs::Version;
use vecdb::{
AnyBoxedIterableVec, AnyCloneableIterableVec, AnyCollectableVec, FromCoarserIndex,
LazyVecFrom2, StoredIndex,
};
use crate::grouped::{EagerVecBuilder, VecBuilderOptions};
use super::ComputedType;
#[allow(clippy::type_complexity)]
#[derive(Clone)]
pub struct LazyVecBuilder<I, T, S1I, S2T>
where
I: StoredIndex,
T: ComputedType,
S2T: ComputedType,
{
pub first: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
pub average: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
pub sum: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
pub max: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
pub min: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
pub last: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
pub cumulative: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
}
const VERSION: Version = Version::ZERO;
impl<I, T, S1I, S2T> LazyVecBuilder<I, T, S1I, S2T>
where
I: StoredIndex,
T: ComputedType + 'static,
S1I: StoredIndex + 'static + FromCoarserIndex<I>,
S2T: ComputedType,
{
#[allow(clippy::too_many_arguments)]
pub fn forced_import(
name: &str,
version: Version,
source: Option<AnyBoxedIterableVec<S1I, T>>,
source_extra: &EagerVecBuilder<S1I, T>,
len_source: AnyBoxedIterableVec<I, S2T>,
options: LazyVecBuilderOptions,
) -> Self {
let only_one_active = options.is_only_one_active();
let suffix = |s: &str| format!("{name}_{s}");
let maybe_suffix = |s: &str| {
if only_one_active {
name.to_string()
} else {
suffix(s)
}
};
Self {
first: options.first.then(|| {
Box::new(LazyVecFrom2::init(
&maybe_suffix("first"),
version + VERSION + Version::ZERO,
source_extra
.first
.as_ref()
.map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
len_source.clone(),
|i: I, source, len_source| {
if i.unwrap_to_usize() >= len_source.len() {
return None;
}
source
.next_at(S1I::min_from(i))
.map(|(_, cow)| cow.into_owned())
},
))
}),
last: options.last.then(|| {
Box::new(LazyVecFrom2::init(
name,
version + VERSION + Version::ZERO,
source_extra.last.as_ref().map_or_else(
|| {
source
.as_ref()
.unwrap_or_else(|| {
dbg!(name, I::to_string());
panic!()
})
.clone()
},
|v| v.clone(),
),
len_source.clone(),
|i: I, source, len_source| {
if i.unwrap_to_usize() >= len_source.len() {
return None;
}
source
.next_at(S1I::max_from(i, source.len()))
.map(|(_, cow)| cow.into_owned())
},
))
}),
min: options.min.then(|| {
Box::new(LazyVecFrom2::init(
&maybe_suffix("min"),
version + VERSION + Version::ZERO,
source_extra
.min
.as_ref()
.map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
len_source.clone(),
|i: I, source, len_source| {
if i.unwrap_to_usize() >= len_source.len() {
return None;
}
S1I::inclusive_range_from(i, source.len())
.flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
.min()
},
))
}),
max: options.max.then(|| {
Box::new(LazyVecFrom2::init(
&maybe_suffix("max"),
version + VERSION + Version::ZERO,
source_extra
.max
.as_ref()
.map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
len_source.clone(),
|i: I, source, len_source| {
if i.unwrap_to_usize() >= len_source.len() {
return None;
}
S1I::inclusive_range_from(i, source.len())
.flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
.max()
},
))
}),
average: options.average.then(|| {
Box::new(LazyVecFrom2::init(
&maybe_suffix("average"),
version + VERSION + Version::ZERO,
source_extra
.average
.as_ref()
.map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
len_source.clone(),
|i: I, source, len_source| {
if i.unwrap_to_usize() >= len_source.len() {
return None;
}
let vec = S1I::inclusive_range_from(i, source.len())
.flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
.collect::<Vec<_>>();
if vec.is_empty() {
return None;
}
let mut sum = T::from(0);
let len = vec.len();
vec.into_iter().for_each(|v| sum += v);
Some(sum / len)
},
))
}),
sum: options.sum.then(|| {
Box::new(LazyVecFrom2::init(
&(if !options.last && !options.average && !options.min && !options.max {
name.to_string()
} else {
maybe_suffix("sum")
}),
version + VERSION + Version::ZERO,
source_extra
.sum
.as_ref()
.map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
len_source.clone(),
|i: I, source, len_source| {
if i.unwrap_to_usize() >= len_source.len() {
return None;
}
let vec = S1I::inclusive_range_from(i, source.len())
.flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
.collect::<Vec<_>>();
if vec.is_empty() {
return None;
}
let mut sum = T::from(0);
vec.into_iter().for_each(|v| sum += v);
Some(sum)
},
))
}),
cumulative: options.cumulative.then(|| {
Box::new(LazyVecFrom2::init(
&suffix("cumulative"),
version + VERSION + Version::ZERO,
source_extra.cumulative.as_ref().unwrap().boxed_clone(),
len_source.clone(),
|i: I, source, len_source| {
if i.unwrap_to_usize() >= len_source.len() {
return None;
}
source
.next_at(S1I::max_from(i, source.len()))
.map(|(_, cow)| cow.into_owned())
},
))
}),
}
}
pub fn starting_index(&self, max_from: I) -> I {
max_from.min(I::from(
self.vecs().into_iter().map(|v| v.len()).min().unwrap(),
))
}
pub fn unwrap_first(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
self.first.as_ref().unwrap()
}
#[allow(unused)]
pub fn unwrap_average(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
self.average.as_ref().unwrap()
}
pub fn unwrap_sum(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
self.sum.as_ref().unwrap()
}
pub fn unwrap_max(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
self.max.as_ref().unwrap()
}
pub fn unwrap_min(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
self.min.as_ref().unwrap()
}
pub fn unwrap_last(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
self.last.as_ref().unwrap()
}
#[allow(unused)]
pub fn unwrap_cumulative(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
self.cumulative.as_ref().unwrap()
}
pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
let mut v: Vec<&dyn AnyCollectableVec> = vec![];
if let Some(first) = self.first.as_ref() {
v.push(first.as_ref());
}
if let Some(last) = self.last.as_ref() {
v.push(last.as_ref());
}
if let Some(min) = self.min.as_ref() {
v.push(min.as_ref());
}
if let Some(max) = self.max.as_ref() {
v.push(max.as_ref());
}
if let Some(average) = self.average.as_ref() {
v.push(average.as_ref());
}
if let Some(sum) = self.sum.as_ref() {
v.push(sum.as_ref());
}
if let Some(cumulative) = self.cumulative.as_ref() {
v.push(cumulative.as_ref());
}
v
}
}
#[derive(Default, Clone, Copy)]
pub struct LazyVecBuilderOptions {
average: bool,
sum: bool,
max: bool,
min: bool,
first: bool,
last: bool,
cumulative: bool,
}
impl From<VecBuilderOptions> for LazyVecBuilderOptions {
fn from(value: VecBuilderOptions) -> Self {
Self {
average: value.average(),
sum: value.sum(),
max: value.max(),
min: value.min(),
first: value.first(),
last: value.last(),
cumulative: value.cumulative(),
}
}
}
impl LazyVecBuilderOptions {
pub fn add_first(mut self) -> Self {
self.first = true;
self
}
pub fn add_last(mut self) -> Self {
self.last = true;
self
}
pub fn add_min(mut self) -> Self {
self.min = true;
self
}
pub fn add_max(mut self) -> Self {
self.max = true;
self
}
pub fn add_average(mut self) -> Self {
self.average = true;
self
}
pub fn add_sum(mut self) -> Self {
self.sum = true;
self
}
pub fn add_cumulative(mut self) -> Self {
self.cumulative = true;
self
}
#[allow(unused)]
pub fn rm_min(mut self) -> Self {
self.min = false;
self
}
#[allow(unused)]
pub fn rm_max(mut self) -> Self {
self.max = false;
self
}
#[allow(unused)]
pub fn rm_average(mut self) -> Self {
self.average = false;
self
}
#[allow(unused)]
pub fn rm_sum(mut self) -> Self {
self.sum = false;
self
}
#[allow(unused)]
pub fn rm_cumulative(mut self) -> Self {
self.cumulative = false;
self
}
pub fn add_minmax(mut self) -> Self {
self.min = true;
self.max = true;
self
}
pub fn is_only_one_active(&self) -> bool {
[
self.average,
self.sum,
self.max,
self.min,
self.first,
self.last,
self.cumulative,
]
.iter()
.filter(|b| **b)
.count()
== 1
}
pub fn copy_self_extra(&self) -> Self {
Self {
cumulative: self.cumulative,
..Self::default()
}
}
}
@@ -4,12 +4,9 @@ use brk_indexer::Indexer;
use brk_structs::{
DateIndex, DecadeIndex, MonthIndex, QuarterIndex, SemesterIndex, Version, WeekIndex, YearIndex,
};
use vecdb::{
AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Computation, Database, EagerVec,
Exit, Format,
};
use vecdb::{AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Database, EagerVec, Exit};
use crate::{Indexes, grouped::ComputedVecBuilder, indexes};
use crate::{Indexes, grouped::LazyVecBuilder, indexes};
use super::{ComputedType, EagerVecBuilder, Source, VecBuilderOptions};
@@ -20,12 +17,12 @@ where
{
pub dateindex: Option<EagerVec<DateIndex, T>>,
pub dateindex_extra: EagerVecBuilder<DateIndex, T>,
pub weekindex: ComputedVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
pub monthindex: ComputedVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
pub quarterindex: ComputedVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
pub semesterindex: ComputedVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
pub yearindex: ComputedVecBuilder<YearIndex, T, DateIndex, YearIndex>,
pub decadeindex: ComputedVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
pub weekindex: LazyVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
pub monthindex: LazyVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
pub quarterindex: LazyVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
pub semesterindex: LazyVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
pub yearindex: LazyVecBuilder<YearIndex, T, DateIndex, YearIndex>,
pub decadeindex: LazyVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
}
const VERSION: Version = Version::ZERO;
@@ -40,20 +37,17 @@ where
name: &str,
source: Source<DateIndex, T>,
version: Version,
format: Format,
computation: Computation,
indexes: &indexes::Vecs,
options: VecBuilderOptions,
) -> Result<Self> {
let dateindex = source.is_compute().then(|| {
EagerVec::forced_import(db, name, version + VERSION + Version::ZERO, format).unwrap()
EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO).unwrap()
});
let dateindex_extra = EagerVecBuilder::forced_import(
let dateindex_extra = EagerVecBuilder::forced_import_compressed(
db,
name,
version + VERSION + Version::ZERO,
format,
options.copy_self_extra(),
)?;
@@ -62,72 +56,54 @@ where
let dateindex_source = source.vec().or(dateindex.as_ref().map(|v| v.boxed_clone()));
Ok(Self {
weekindex: ComputedVecBuilder::forced_import(
db,
weekindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
computation,
dateindex_source.clone(),
&dateindex_extra,
indexes.weekindex_to_weekindex.boxed_clone(),
options.into(),
)?,
monthindex: ComputedVecBuilder::forced_import(
db,
),
monthindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
dateindex_source.clone(),
&dateindex_extra,
indexes.monthindex_to_monthindex.boxed_clone(),
options.into(),
)?,
quarterindex: ComputedVecBuilder::forced_import(
db,
),
quarterindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
dateindex_source.clone(),
&dateindex_extra,
indexes.quarterindex_to_quarterindex.boxed_clone(),
options.into(),
)?,
semesterindex: ComputedVecBuilder::forced_import(
db,
),
semesterindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
dateindex_source.clone(),
&dateindex_extra,
indexes.semesterindex_to_semesterindex.boxed_clone(),
options.into(),
)?,
yearindex: ComputedVecBuilder::forced_import(
db,
),
yearindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
dateindex_source.clone(),
&dateindex_extra,
indexes.yearindex_to_yearindex.boxed_clone(),
options.into(),
)?,
decadeindex: ComputedVecBuilder::forced_import(
db,
),
decadeindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
dateindex_source.clone(),
&dateindex_extra,
indexes.decadeindex_to_decadeindex.boxed_clone(),
options.into(),
)?,
),
dateindex,
dateindex_extra,
})
@@ -159,12 +135,11 @@ where
)?;
let dateindex: Option<&EagerVec<DateIndex, T>> = None;
self.compute_rest(indexes, starting_indexes, exit, dateindex)
self.compute_rest(starting_indexes, exit, dateindex)
}
pub fn compute_rest(
&mut self,
indexes: &indexes::Vecs,
starting_indexes: &Indexes,
exit: &Exit,
dateindex: Option<&impl AnyIterableVec<DateIndex, T>>,
@@ -179,42 +154,6 @@ where
.extend(starting_indexes.dateindex, dateindex, exit)?;
}
self.weekindex.compute_if_necessary(
starting_indexes.weekindex,
&indexes.weekindex_to_dateindex_count,
exit,
)?;
self.monthindex.compute_if_necessary(
starting_indexes.monthindex,
&indexes.monthindex_to_dateindex_count,
exit,
)?;
self.quarterindex.compute_if_necessary(
starting_indexes.quarterindex,
&indexes.quarterindex_to_monthindex_count,
exit,
)?;
self.semesterindex.compute_if_necessary(
starting_indexes.semesterindex,
&indexes.semesterindex_to_monthindex_count,
exit,
)?;
self.yearindex.compute_if_necessary(
starting_indexes.yearindex,
&indexes.yearindex_to_monthindex_count,
exit,
)?;
self.decadeindex.compute_if_necessary(
starting_indexes.decadeindex,
&indexes.decadeindex_to_yearindex_count,
exit,
)?;
Ok(())
}
+24 -86
View File
@@ -5,14 +5,11 @@ use brk_structs::{
DateIndex, DecadeIndex, DifficultyEpoch, Height, MonthIndex, QuarterIndex, SemesterIndex,
Version, WeekIndex, YearIndex,
};
use vecdb::{
AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Computation, Database, EagerVec,
Exit, Format,
};
use vecdb::{AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Database, EagerVec, Exit};
use crate::{
Indexes,
grouped::{ComputedVecBuilder, Source},
grouped::{LazyVecBuilder, Source},
indexes,
};
@@ -26,14 +23,14 @@ where
pub height: Option<EagerVec<Height, T>>,
pub height_extra: EagerVecBuilder<Height, T>,
pub dateindex: EagerVecBuilder<DateIndex, T>,
pub weekindex: ComputedVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
pub weekindex: LazyVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
pub difficultyepoch: EagerVecBuilder<DifficultyEpoch, T>,
pub monthindex: ComputedVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
pub quarterindex: ComputedVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
pub semesterindex: ComputedVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
pub yearindex: ComputedVecBuilder<YearIndex, T, DateIndex, YearIndex>,
pub monthindex: LazyVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
pub quarterindex: LazyVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
pub semesterindex: LazyVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
pub yearindex: LazyVecBuilder<YearIndex, T, DateIndex, YearIndex>,
// TODO: pub halvingepoch: StorableVecGeneator<Halvingepoch, T>,
pub decadeindex: ComputedVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
pub decadeindex: LazyVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
}
const VERSION: Version = Version::ZERO;
@@ -49,109 +46,86 @@ where
name: &str,
source: Source<Height, T>,
version: Version,
format: Format,
computation: Computation,
indexes: &indexes::Vecs,
options: VecBuilderOptions,
) -> Result<Self> {
let height = source.is_compute().then(|| {
EagerVec::forced_import(db, name, version + VERSION + Version::ZERO, format).unwrap()
EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO).unwrap()
});
let height_extra = EagerVecBuilder::forced_import(
let height_extra = EagerVecBuilder::forced_import_compressed(
db,
name,
version + VERSION + Version::ZERO,
format,
options.copy_self_extra(),
)?;
let dateindex = EagerVecBuilder::forced_import(
let dateindex = EagerVecBuilder::forced_import_compressed(
db,
name,
version + VERSION + Version::ZERO,
format,
options,
)?;
let options = options.remove_percentiles();
Ok(Self {
weekindex: ComputedVecBuilder::forced_import(
db,
weekindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
computation,
None,
&dateindex,
indexes.weekindex_to_weekindex.boxed_clone(),
options.into(),
)?,
monthindex: ComputedVecBuilder::forced_import(
db,
),
monthindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
None,
&dateindex,
indexes.monthindex_to_monthindex.boxed_clone(),
options.into(),
)?,
quarterindex: ComputedVecBuilder::forced_import(
db,
),
quarterindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
None,
&dateindex,
indexes.quarterindex_to_quarterindex.boxed_clone(),
options.into(),
)?,
semesterindex: ComputedVecBuilder::forced_import(
db,
),
semesterindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
None,
&dateindex,
indexes.semesterindex_to_semesterindex.boxed_clone(),
options.into(),
)?,
yearindex: ComputedVecBuilder::forced_import(
db,
),
yearindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
None,
&dateindex,
indexes.yearindex_to_yearindex.boxed_clone(),
options.into(),
)?,
decadeindex: ComputedVecBuilder::forced_import(
db,
),
decadeindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
None,
&dateindex,
indexes.decadeindex_to_decadeindex.boxed_clone(),
options.into(),
)?,
),
// halvingepoch: StorableVecGeneator::forced_import(db, name, version + VERSION + Version::ZERO, format, options)?,
height,
height_extra,
dateindex,
difficultyepoch: EagerVecBuilder::forced_import(
difficultyepoch: EagerVecBuilder::forced_import_compressed(
db,
name,
version + VERSION + Version::ZERO,
format,
options,
)?,
})
@@ -229,42 +203,6 @@ where
)?;
}
self.weekindex.compute_if_necessary(
starting_indexes.weekindex,
&indexes.weekindex_to_dateindex_count,
exit,
)?;
self.monthindex.compute_if_necessary(
starting_indexes.monthindex,
&indexes.monthindex_to_dateindex_count,
exit,
)?;
self.quarterindex.compute_if_necessary(
starting_indexes.quarterindex,
&indexes.quarterindex_to_monthindex_count,
exit,
)?;
self.semesterindex.compute_if_necessary(
starting_indexes.semesterindex,
&indexes.semesterindex_to_monthindex_count,
exit,
)?;
self.yearindex.compute_if_necessary(
starting_indexes.yearindex,
&indexes.yearindex_to_monthindex_count,
exit,
)?;
self.decadeindex.compute_if_necessary(
starting_indexes.decadeindex,
&indexes.decadeindex_to_yearindex_count,
exit,
)?;
Ok(())
}
@@ -2,7 +2,7 @@ use brk_error::Result;
use brk_indexer::Indexer;
use brk_structs::{DifficultyEpoch, Height, Version};
use vecdb::{AnyCollectableVec, Database, EagerVec, Exit, Format};
use vecdb::{AnyCollectableVec, Database, EagerVec, Exit};
use crate::{Indexes, indexes};
@@ -30,16 +30,15 @@ where
db: &Database,
name: &str,
version: Version,
format: Format,
options: VecBuilderOptions,
) -> Result<Self> {
let height = EagerVec::forced_import(db, name, version + VERSION + Version::ZERO, format)?;
let height =
EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO)?;
let height_extra = EagerVecBuilder::forced_import(
let height_extra = EagerVecBuilder::forced_import_compressed(
db,
name,
version + VERSION + Version::ZERO,
format,
options.copy_self_extra(),
)?;
@@ -48,11 +47,10 @@ where
Ok(Self {
height,
height_extra,
difficultyepoch: EagerVecBuilder::forced_import(
difficultyepoch: EagerVecBuilder::forced_import_compressed(
db,
name,
version + VERSION + Version::ZERO,
format,
options,
)?,
// halvingepoch: StorableVecGeneator::forced_import(db, name, version + VERSION + Version::ZERO, format, options)?,
+25 -84
View File
@@ -5,13 +5,13 @@ use brk_structs::{
Sats, SemesterIndex, TxIndex, Version, WeekIndex, YearIndex,
};
use vecdb::{
AnyCloneableIterableVec, AnyCollectableVec, AnyVec, CollectableVec, Computation, Database,
EagerVec, Exit, Format, GenericStoredVec, StoredIndex, VecIterator,
AnyCloneableIterableVec, AnyCollectableVec, AnyVec, CollectableVec, Database, EagerVec, Exit,
GenericStoredVec, StoredIndex, VecIterator,
};
use crate::{
Indexes,
grouped::{ComputedVecBuilder, Source},
grouped::{LazyVecBuilder, Source},
indexes, price,
};
@@ -25,14 +25,14 @@ where
pub txindex: Option<Box<EagerVec<TxIndex, T>>>,
pub height: EagerVecBuilder<Height, T>,
pub dateindex: EagerVecBuilder<DateIndex, T>,
pub weekindex: ComputedVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
pub weekindex: LazyVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
pub difficultyepoch: EagerVecBuilder<DifficultyEpoch, T>,
pub monthindex: ComputedVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
pub quarterindex: ComputedVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
pub semesterindex: ComputedVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
pub yearindex: ComputedVecBuilder<YearIndex, T, DateIndex, YearIndex>,
pub monthindex: LazyVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
pub quarterindex: LazyVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
pub semesterindex: LazyVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
pub yearindex: LazyVecBuilder<YearIndex, T, DateIndex, YearIndex>,
// TODO: pub halvingepoch: StorableVecGeneator<Halvingepoch, T>,
pub decadeindex: ComputedVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
pub decadeindex: LazyVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
}
const VERSION: Version = Version::ZERO;
@@ -48,112 +48,89 @@ where
name: &str,
source: Source<TxIndex, T>,
version: Version,
format: Format,
computation: Computation,
indexes: &indexes::Vecs,
options: VecBuilderOptions,
) -> Result<Self> {
let txindex = source.is_compute().then(|| {
Box::new(
EagerVec::forced_import(db, name, version + VERSION + Version::ZERO, format)
EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO)
.unwrap(),
)
});
let height = EagerVecBuilder::forced_import(
let height = EagerVecBuilder::forced_import_compressed(
db,
name,
version + VERSION + Version::ZERO,
format,
options,
)?;
let options = options.remove_percentiles();
let dateindex = EagerVecBuilder::forced_import(
let dateindex = EagerVecBuilder::forced_import_compressed(
db,
name,
version + VERSION + Version::ZERO,
format,
options,
)?;
Ok(Self {
weekindex: ComputedVecBuilder::forced_import(
db,
weekindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
computation,
None,
&dateindex,
indexes.weekindex_to_weekindex.boxed_clone(),
options.into(),
)?,
monthindex: ComputedVecBuilder::forced_import(
db,
),
monthindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
None,
&dateindex,
indexes.monthindex_to_monthindex.boxed_clone(),
options.into(),
)?,
quarterindex: ComputedVecBuilder::forced_import(
db,
),
quarterindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
None,
&dateindex,
indexes.quarterindex_to_quarterindex.boxed_clone(),
options.into(),
)?,
semesterindex: ComputedVecBuilder::forced_import(
db,
),
semesterindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
None,
&dateindex,
indexes.semesterindex_to_semesterindex.boxed_clone(),
options.into(),
)?,
yearindex: ComputedVecBuilder::forced_import(
db,
),
yearindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
None,
&dateindex,
indexes.yearindex_to_yearindex.boxed_clone(),
options.into(),
)?,
decadeindex: ComputedVecBuilder::forced_import(
db,
),
decadeindex: LazyVecBuilder::forced_import(
name,
version + VERSION + Version::ZERO,
format,
Computation::Lazy,
None,
&dateindex,
indexes.decadeindex_to_decadeindex.boxed_clone(),
options.into(),
)?,
),
txindex,
height,
dateindex,
difficultyepoch: EagerVecBuilder::forced_import(
difficultyepoch: EagerVecBuilder::forced_import_compressed(
db,
name,
version + VERSION + Version::ZERO,
format,
options,
)?,
// halvingepoch: StorableVecGeneator::forced_import(db, name, version + VERSION + Version::ZERO, format, options)?,
@@ -237,42 +214,6 @@ where
exit,
)?;
self.weekindex.compute_if_necessary(
starting_indexes.weekindex,
&indexes.weekindex_to_first_dateindex,
exit,
)?;
self.monthindex.compute_if_necessary(
starting_indexes.monthindex,
&indexes.monthindex_to_dateindex_count,
exit,
)?;
self.quarterindex.compute_if_necessary(
starting_indexes.quarterindex,
&indexes.quarterindex_to_monthindex_count,
exit,
)?;
self.semesterindex.compute_if_necessary(
starting_indexes.semesterindex,
&indexes.semesterindex_to_monthindex_count,
exit,
)?;
self.yearindex.compute_if_necessary(
starting_indexes.yearindex,
&indexes.yearindex_to_monthindex_count,
exit,
)?;
self.decadeindex.compute_if_necessary(
starting_indexes.decadeindex,
&indexes.decadeindex_to_yearindex_count,
exit,
)?;
self.difficultyepoch.from_aligned(
starting_indexes.difficultyepoch,
&self.height,
+4 -2
View File
@@ -1,25 +1,27 @@
mod builder_computed;
mod builder_eager;
mod builder_lazy;
mod computed;
mod from_dateindex;
mod from_height;
mod from_height_strict;
mod from_txindex;
mod ratio_from_dateindex;
mod sd_from_dateindex;
mod source;
mod value_from_dateindex;
mod value_from_height;
mod value_from_txindex;
mod value_height;
pub use builder_computed::*;
pub use builder_eager::*;
pub use builder_lazy::*;
use computed::*;
pub use from_dateindex::*;
pub use from_height::*;
pub use from_height_strict::*;
pub use from_txindex::*;
pub use ratio_from_dateindex::*;
pub use sd_from_dateindex::*;
pub use source::*;
pub use value_from_dateindex::*;
pub use value_from_height::*;
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,707 @@
use brk_error::Result;
use brk_indexer::Indexer;
use brk_structs::{CheckedSub, Date, DateIndex, Dollars, StoredF32, Version};
use vecdb::{
AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, BoxedVecIterator, CollectableVec,
Database, EagerVec, Exit, GenericStoredVec, StoredIndex,
};
use crate::{Indexes, grouped::source::Source, indexes};
use super::{ComputedVecsFromDateIndex, VecBuilderOptions};
#[derive(Clone)]
pub struct ComputedStandardDeviationVecsFromDateIndex {
days: usize,
pub sma: Option<ComputedVecsFromDateIndex<StoredF32>>,
pub sd: ComputedVecsFromDateIndex<StoredF32>,
pub _0sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub p0_5sd: ComputedVecsFromDateIndex<StoredF32>,
pub p1sd: ComputedVecsFromDateIndex<StoredF32>,
pub p1_5sd: ComputedVecsFromDateIndex<StoredF32>,
pub p2sd: ComputedVecsFromDateIndex<StoredF32>,
pub p2_5sd: ComputedVecsFromDateIndex<StoredF32>,
pub p3sd: ComputedVecsFromDateIndex<StoredF32>,
pub m0_5sd: ComputedVecsFromDateIndex<StoredF32>,
pub m1sd: ComputedVecsFromDateIndex<StoredF32>,
pub m1_5sd: ComputedVecsFromDateIndex<StoredF32>,
pub m2sd: ComputedVecsFromDateIndex<StoredF32>,
pub m2_5sd: ComputedVecsFromDateIndex<StoredF32>,
pub m3sd: ComputedVecsFromDateIndex<StoredF32>,
pub p0_5sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub p1sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub p1_5sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub p2sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub p2_5sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub p3sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub m0_5sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub m1sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub m1_5sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub m2sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub m2_5sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub m3sd_as_price: ComputedVecsFromDateIndex<Dollars>,
pub zscore: ComputedVecsFromDateIndex<StoredF32>,
}
const VERSION: Version = Version::ONE;
impl ComputedStandardDeviationVecsFromDateIndex {
#[allow(clippy::too_many_arguments)]
pub fn forced_import(
db: &Database,
name: &str,
days: usize,
source: Source<DateIndex, StoredF32>,
version: Version,
indexes: &indexes::Vecs,
) -> Result<Self> {
let options = VecBuilderOptions::default().add_last();
Ok(Self {
days,
sma: source.is_compute().then(|| {
ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_sma"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)
.unwrap()
}),
sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p0_5sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p0_5sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p1sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p1sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p1_5sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p1_5sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p2sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p2sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p2_5sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p2_5sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p3sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p3sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m0_5sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m0_5sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m1sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m1sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m1_5sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m1_5sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m2sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m2sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m2_5sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m2_5sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m3sd: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m3sd"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
_0sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_0sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p0_5sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p0_5sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p1sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p1sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p1_5sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p1_5sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p2sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p2sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p2_5sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p2_5sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
p3sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_p3sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m0_5sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m0_5sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m1sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m1sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m1_5sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m1_5sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m2sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m2sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m2_5sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m2_5sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
m3sd_as_price: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_m3sd_as_price"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
zscore: ComputedVecsFromDateIndex::forced_import(
db,
&format!("{name}_zscore"),
Source::Compute,
version + VERSION + Version::ZERO,
indexes,
options,
)?,
})
}
pub fn compute_all(
&mut self,
indexer: &Indexer,
indexes: &indexes::Vecs,
starting_indexes: &Indexes,
exit: &Exit,
source: &impl CollectableVec<DateIndex, StoredF32>,
source_as_price: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
) -> Result<()> {
let min_date = DateIndex::try_from(Date::MIN_RATIO).unwrap();
self.sma.as_mut().unwrap().compute_all(
indexer,
indexes,
starting_indexes,
exit,
|v, _, _, starting_indexes, exit| {
v.compute_sma_(
starting_indexes.dateindex,
source,
self.days,
exit,
Some(min_date),
)?;
Ok(())
},
)?;
let sma_opt: Option<&EagerVec<DateIndex, StoredF32>> = None;
self.compute_rest(
indexer,
indexes,
starting_indexes,
exit,
sma_opt,
source,
source_as_price,
)
}
#[allow(clippy::too_many_arguments)]
pub fn compute_rest(
&mut self,
indexer: &Indexer,
indexes: &indexes::Vecs,
starting_indexes: &Indexes,
exit: &Exit,
sma_opt: Option<&impl AnyIterableVec<DateIndex, StoredF32>>,
source: &impl CollectableVec<DateIndex, StoredF32>,
source_as_price: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
) -> Result<()> {
let sma = sma_opt.unwrap_or_else(|| unsafe {
std::mem::transmute(&self.sma.as_ref().unwrap().dateindex)
});
let min_date = DateIndex::try_from(Date::MIN_RATIO).unwrap();
let source_version = source.version();
self.mut_vecs().iter_mut().try_for_each(|v| -> Result<()> {
v.validate_computed_version_or_reset(
Version::ZERO + v.inner_version() + source_version,
)?;
Ok(())
})?;
let starting_dateindex = self
.mut_vecs()
.iter()
.map(|v| DateIndex::from(v.len()))
.min()
.unwrap()
.min(starting_indexes.dateindex);
let mut sorted = source.collect_range(
Some(min_date.unwrap_to_usize()),
Some(starting_dateindex.unwrap_to_usize()),
)?;
sorted.sort_unstable();
let mut sma_iter = sma.iter();
source
.iter_at(starting_dateindex)
.try_for_each(|(index, ratio)| -> Result<()> {
if index < min_date {
self.sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.p0_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.p1sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.p1_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.p2sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.p2_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.p3sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.m0_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.m1sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.m1_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.m2sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.m2_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
self.m3sd.dateindex.as_mut().unwrap().forced_push_at(
index,
StoredF32::NAN,
exit,
)?;
} else {
let ratio = ratio.into_owned();
let pos = sorted.binary_search(&ratio).unwrap_or_else(|pos| pos);
sorted.insert(pos, ratio);
let avg = sma_iter.unwrap_get_inner(index);
let population =
index.checked_sub(min_date).unwrap().unwrap_to_usize() as f32 + 1.0;
let sd = StoredF32::from(
(sorted.iter().map(|v| (**v - *avg).powi(2)).sum::<f32>() / population)
.sqrt(),
);
self.sd
.dateindex
.as_mut()
.unwrap()
.forced_push_at(index, sd, exit)?;
self.p0_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
avg + StoredF32::from(0.5 * *sd),
exit,
)?;
self.p1sd
.dateindex
.as_mut()
.unwrap()
.forced_push_at(index, avg + sd, exit)?;
self.p1_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
avg + StoredF32::from(1.5 * *sd),
exit,
)?;
self.p2sd.dateindex.as_mut().unwrap().forced_push_at(
index,
avg + 2 * sd,
exit,
)?;
self.p2_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
avg + StoredF32::from(2.5 * *sd),
exit,
)?;
self.p3sd.dateindex.as_mut().unwrap().forced_push_at(
index,
avg + 3 * sd,
exit,
)?;
self.m0_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
avg - StoredF32::from(0.5 * *sd),
exit,
)?;
self.m1sd
.dateindex
.as_mut()
.unwrap()
.forced_push_at(index, avg - sd, exit)?;
self.m1_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
avg - StoredF32::from(1.5 * *sd),
exit,
)?;
self.m2sd.dateindex.as_mut().unwrap().forced_push_at(
index,
avg - 2 * sd,
exit,
)?;
self.m2_5sd.dateindex.as_mut().unwrap().forced_push_at(
index,
avg - StoredF32::from(2.5 * *sd),
exit,
)?;
self.m3sd.dateindex.as_mut().unwrap().forced_push_at(
index,
avg - 3 * sd,
exit,
)?;
}
Ok(())
})?;
drop(sma_iter);
self.mut_vecs()
.into_iter()
.try_for_each(|v| v.safe_flush(exit))?;
[
&mut self.sd,
&mut self.p0_5sd,
&mut self.p1sd,
&mut self.p1_5sd,
&mut self.p2sd,
&mut self.p2_5sd,
&mut self.p3sd,
&mut self.m0_5sd,
&mut self.m1sd,
&mut self.m1_5sd,
&mut self.m2sd,
&mut self.m2_5sd,
&mut self.m3sd,
]
.into_iter()
.try_for_each(|v| {
v.compute_rest(starting_indexes, exit, None as Option<&EagerVec<_, _>>)
})?;
self.zscore.compute_all(
indexer,
indexes,
starting_indexes,
exit,
|vec, _, _, starting_indexes, exit| {
vec.compute_zscore(
starting_indexes.dateindex,
source,
sma,
self.sd.dateindex.as_ref().unwrap(),
exit,
)?;
Ok(())
},
)?;
let Some(price) = source_as_price else {
return Ok(());
};
let compute_as_price =
|as_price: &mut ComputedVecsFromDateIndex<Dollars>,
mut iter: BoxedVecIterator<DateIndex, StoredF32>| {
as_price.compute_all(
indexer,
indexes,
starting_indexes,
exit,
|vec, _, _, starting_indexes, exit| {
vec.compute_transform(
starting_indexes.dateindex,
price,
|(i, price, ..)| {
let multiplier = iter.unwrap_get_inner(i);
(i, price * multiplier)
},
exit,
)?;
Ok(())
},
)
};
compute_as_price(&mut self._0sd_as_price, sma.iter())?;
compute_as_price(
&mut self.p0_5sd_as_price,
self.p0_5sd.dateindex.as_ref().unwrap().iter(),
)?;
compute_as_price(
&mut self.p1sd_as_price,
self.p1sd.dateindex.as_ref().unwrap().iter(),
)?;
compute_as_price(
&mut self.p1_5sd_as_price,
self.p1_5sd.dateindex.as_ref().unwrap().iter(),
)?;
compute_as_price(
&mut self.p2sd_as_price,
self.p2sd.dateindex.as_ref().unwrap().iter(),
)?;
compute_as_price(
&mut self.p2_5sd_as_price,
self.p2_5sd.dateindex.as_ref().unwrap().iter(),
)?;
compute_as_price(
&mut self.p3sd_as_price,
self.p3sd.dateindex.as_ref().unwrap().iter(),
)?;
compute_as_price(
&mut self.m0_5sd_as_price,
self.m0_5sd.dateindex.as_ref().unwrap().iter(),
)?;
compute_as_price(
&mut self.m1sd_as_price,
self.m1sd.dateindex.as_ref().unwrap().iter(),
)?;
compute_as_price(
&mut self.m1_5sd_as_price,
self.m1_5sd.dateindex.as_ref().unwrap().iter(),
)?;
compute_as_price(
&mut self.m2sd_as_price,
self.m2sd.dateindex.as_ref().unwrap().iter(),
)?;
compute_as_price(
&mut self.m2_5sd_as_price,
self.m2_5sd.dateindex.as_ref().unwrap().iter(),
)?;
compute_as_price(
&mut self.m3sd_as_price,
self.m3sd.dateindex.as_ref().unwrap().iter(),
)?;
Ok(())
}
fn mut_vecs(&mut self) -> [&mut EagerVec<DateIndex, StoredF32>; 13] {
[
self.sd.dateindex.as_mut().unwrap(),
self.p0_5sd.dateindex.as_mut().unwrap(),
self.p1sd.dateindex.as_mut().unwrap(),
self.p1_5sd.dateindex.as_mut().unwrap(),
self.p2sd.dateindex.as_mut().unwrap(),
self.p2_5sd.dateindex.as_mut().unwrap(),
self.p3sd.dateindex.as_mut().unwrap(),
self.m0_5sd.dateindex.as_mut().unwrap(),
self.m1sd.dateindex.as_mut().unwrap(),
self.m1_5sd.dateindex.as_mut().unwrap(),
self.m2sd.dateindex.as_mut().unwrap(),
self.m2_5sd.dateindex.as_mut().unwrap(),
self.m3sd.dateindex.as_mut().unwrap(),
]
}
pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
[
self.sma.as_ref().map_or(vec![], |v| v.vecs()),
self.sd.vecs(),
self.p0_5sd.vecs(),
self.p1sd.vecs(),
self.p1_5sd.vecs(),
self.p2sd.vecs(),
self.p2_5sd.vecs(),
self.p3sd.vecs(),
self.m0_5sd.vecs(),
self.m1sd.vecs(),
self.m1_5sd.vecs(),
self.m2sd.vecs(),
self.m2_5sd.vecs(),
self.m3sd.vecs(),
self._0sd_as_price.vecs(),
self.p0_5sd_as_price.vecs(),
self.p1sd_as_price.vecs(),
self.p1_5sd_as_price.vecs(),
self.p2sd_as_price.vecs(),
self.p2_5sd_as_price.vecs(),
self.p3sd_as_price.vecs(),
self.m0_5sd_as_price.vecs(),
self.m1sd_as_price.vecs(),
self.m1_5sd_as_price.vecs(),
self.m2sd_as_price.vecs(),
self.m2_5sd_as_price.vecs(),
self.m3sd_as_price.vecs(),
self.zscore.vecs(),
]
.into_iter()
.flatten()
.collect::<Vec<_>>()
}
}
@@ -1,9 +1,7 @@
use brk_error::Result;
use brk_indexer::Indexer;
use brk_structs::{Bitcoin, DateIndex, Dollars, Sats, Version};
use vecdb::{
AnyCollectableVec, CollectableVec, Computation, Database, EagerVec, Exit, Format, StoredVec,
};
use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, StoredVec};
use crate::{
Indexes,
@@ -30,8 +28,6 @@ impl ComputedValueVecsFromDateIndex {
name: &str,
source: Source<DateIndex, Sats>,
version: Version,
format: Format,
computation: Computation,
options: VecBuilderOptions,
compute_dollars: bool,
indexes: &indexes::Vecs,
@@ -42,8 +38,6 @@ impl ComputedValueVecsFromDateIndex {
name,
source,
version + VERSION,
format,
computation,
indexes,
options,
)?,
@@ -52,8 +46,6 @@ impl ComputedValueVecsFromDateIndex {
&format!("{name}_in_btc"),
Source::Compute,
version + VERSION,
format,
computation,
indexes,
options,
)?,
@@ -63,8 +55,6 @@ impl ComputedValueVecsFromDateIndex {
&format!("{name}_in_usd"),
Source::Compute,
version + VERSION,
format,
computation,
indexes,
options,
)
@@ -116,7 +106,7 @@ impl ComputedValueVecsFromDateIndex {
) -> Result<()> {
if let Some(dateindex) = dateindex {
self.sats
.compute_rest(indexes, starting_indexes, exit, Some(dateindex))?;
.compute_rest(starting_indexes, exit, Some(dateindex))?;
self.bitcoin.compute_all(
indexer,
@@ -130,8 +120,7 @@ impl ComputedValueVecsFromDateIndex {
} else {
let dateindex: Option<&StoredVec<DateIndex, Sats>> = None;
self.sats
.compute_rest(indexes, starting_indexes, exit, dateindex)?;
self.sats.compute_rest(starting_indexes, exit, dateindex)?;
self.bitcoin.compute_all(
indexer,
@@ -1,9 +1,7 @@
use brk_error::Result;
use brk_indexer::Indexer;
use brk_structs::{Bitcoin, Dollars, Height, Sats, Version};
use vecdb::{
AnyCollectableVec, CollectableVec, Computation, Database, EagerVec, Exit, Format, StoredVec,
};
use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, StoredVec};
use crate::{
Indexes,
@@ -30,8 +28,6 @@ impl ComputedValueVecsFromHeight {
name: &str,
source: Source<Height, Sats>,
version: Version,
format: Format,
computation: Computation,
options: VecBuilderOptions,
compute_dollars: bool,
indexes: &indexes::Vecs,
@@ -42,8 +38,6 @@ impl ComputedValueVecsFromHeight {
name,
source,
version + VERSION,
format,
computation,
indexes,
options,
)?,
@@ -52,8 +46,6 @@ impl ComputedValueVecsFromHeight {
&format!("{name}_in_btc"),
Source::Compute,
version + VERSION,
format,
computation,
indexes,
options,
)?,
@@ -63,8 +55,6 @@ impl ComputedValueVecsFromHeight {
&format!("{name}_in_usd"),
Source::Compute,
version + VERSION,
format,
computation,
indexes,
options,
)
@@ -2,8 +2,8 @@ use brk_error::Result;
use brk_indexer::Indexer;
use brk_structs::{Bitcoin, Close, Dollars, Height, Sats, TxIndex, Version};
use vecdb::{
AnyCloneableIterableVec, AnyCollectableVec, CollectableVec, Computation, ComputedVecFrom3,
Database, Exit, Format, LazyVecFrom1, StoredIndex, StoredVec,
AnyCloneableIterableVec, AnyCollectableVec, CollectableVec, Database, Exit, LazyVecFrom1,
LazyVecFrom3, StoredIndex, StoredVec,
};
use crate::{Indexes, grouped::Source, indexes, price};
@@ -17,16 +17,7 @@ pub struct ComputedValueVecsFromTxindex {
pub bitcoin: ComputedVecsFromTxindex<Bitcoin>,
#[allow(clippy::type_complexity)]
pub dollars_txindex: Option<
ComputedVecFrom3<
TxIndex,
Dollars,
TxIndex,
Bitcoin,
TxIndex,
Height,
Height,
Close<Dollars>,
>,
LazyVecFrom3<TxIndex, Dollars, TxIndex, Bitcoin, TxIndex, Height, Height, Close<Dollars>>,
>,
pub dollars: Option<ComputedVecsFromTxindex<Dollars>>,
}
@@ -41,8 +32,6 @@ impl ComputedValueVecsFromTxindex {
indexes: &indexes::Vecs,
source: Source<TxIndex, Sats>,
version: Version,
computation: Computation,
format: Format,
price: Option<&price::Vecs>,
options: VecBuilderOptions,
) -> Result<Self> {
@@ -56,8 +45,6 @@ impl ComputedValueVecsFromTxindex {
name,
source.clone(),
version + VERSION,
format,
computation,
indexes,
options,
)?;
@@ -81,19 +68,14 @@ impl ComputedValueVecsFromTxindex {
&name_in_btc,
Source::None,
version + VERSION,
format,
computation,
indexes,
options,
)?;
let dollars_txindex = price.map(|price| {
ComputedVecFrom3::forced_import_or_init_from_3(
computation,
db,
LazyVecFrom3::init(
&name_in_usd,
version + VERSION,
format,
bitcoin_txindex.boxed_clone(),
indexes.txindex_to_height.boxed_clone(),
price.chainindexes_to_close.height.boxed_clone(),
@@ -115,7 +97,6 @@ impl ComputedValueVecsFromTxindex {
})
},
)
.unwrap()
});
Ok(Self {
@@ -129,8 +110,6 @@ impl ComputedValueVecsFromTxindex {
&name_in_usd,
Source::None,
version + VERSION,
format,
computation,
indexes,
options,
)
@@ -208,12 +187,6 @@ impl ComputedValueVecsFromTxindex {
if let Some(dollars) = self.dollars.as_mut() {
let dollars_txindex = self.dollars_txindex.as_mut().unwrap();
dollars_txindex.compute_if_necessary(
starting_indexes.txindex,
&indexer.vecs.txindex_to_txid,
exit,
)?;
dollars.compute_rest_from_bitcoin(
indexer,
indexes,
+97 -314
View File
@@ -11,8 +11,8 @@ use brk_structs::{
Timestamp, TxIndex, Txid, UnknownOutputIndex, Version, WeekIndex, YearIndex,
};
use vecdb::{
AnyCloneableIterableVec, AnyCollectableVec, Computation, ComputedVec, ComputedVecFrom1,
ComputedVecFrom2, Database, EagerVec, Exit, Format, PAGE_SIZE, StoredIndex, VecIterator,
AnyCloneableIterableVec, AnyCollectableVec, Database, EagerVec, Exit, LazyVecFrom1,
LazyVecFrom2, PAGE_SIZE, StoredIndex, VecIterator,
};
const VERSION: Version = Version::ZERO;
@@ -34,7 +34,7 @@ pub struct Vecs {
pub difficultyepoch_to_first_height: EagerVec<DifficultyEpoch, Height>,
pub difficultyepoch_to_height_count: EagerVec<DifficultyEpoch, StoredU64>,
pub emptyoutputindex_to_emptyoutputindex:
ComputedVecFrom1<EmptyOutputIndex, EmptyOutputIndex, EmptyOutputIndex, TxIndex>,
LazyVecFrom1<EmptyOutputIndex, EmptyOutputIndex, EmptyOutputIndex, TxIndex>,
pub halvingepoch_to_first_height: EagerVec<HalvingEpoch, Height>,
pub halvingepoch_to_halvingepoch: EagerVec<HalvingEpoch, HalvingEpoch>,
pub height_to_date: EagerVec<Height, Date>,
@@ -45,7 +45,7 @@ pub struct Vecs {
pub height_to_height: EagerVec<Height, Height>,
pub height_to_timestamp_fixed: EagerVec<Height, Timestamp>,
pub height_to_txindex_count: EagerVec<Height, StoredU64>,
pub inputindex_to_inputindex: ComputedVecFrom1<InputIndex, InputIndex, InputIndex, OutputIndex>,
pub inputindex_to_inputindex: LazyVecFrom1<InputIndex, InputIndex, InputIndex, OutputIndex>,
pub monthindex_to_dateindex_count: EagerVec<MonthIndex, StoredU64>,
pub monthindex_to_first_dateindex: EagerVec<MonthIndex, DateIndex>,
pub monthindex_to_monthindex: EagerVec<MonthIndex, MonthIndex>,
@@ -53,27 +53,27 @@ pub struct Vecs {
pub monthindex_to_semesterindex: EagerVec<MonthIndex, SemesterIndex>,
pub monthindex_to_yearindex: EagerVec<MonthIndex, YearIndex>,
pub opreturnindex_to_opreturnindex:
ComputedVecFrom1<OpReturnIndex, OpReturnIndex, OpReturnIndex, TxIndex>,
pub outputindex_to_outputindex: ComputedVecFrom1<OutputIndex, OutputIndex, OutputIndex, Sats>,
LazyVecFrom1<OpReturnIndex, OpReturnIndex, OpReturnIndex, TxIndex>,
pub outputindex_to_outputindex: LazyVecFrom1<OutputIndex, OutputIndex, OutputIndex, Sats>,
pub outputindex_to_txindex: EagerVec<OutputIndex, TxIndex>,
pub p2aaddressindex_to_p2aaddressindex:
ComputedVecFrom1<P2AAddressIndex, P2AAddressIndex, P2AAddressIndex, P2ABytes>,
LazyVecFrom1<P2AAddressIndex, P2AAddressIndex, P2AAddressIndex, P2ABytes>,
pub p2msoutputindex_to_p2msoutputindex:
ComputedVecFrom1<P2MSOutputIndex, P2MSOutputIndex, P2MSOutputIndex, TxIndex>,
LazyVecFrom1<P2MSOutputIndex, P2MSOutputIndex, P2MSOutputIndex, TxIndex>,
pub p2pk33addressindex_to_p2pk33addressindex:
ComputedVecFrom1<P2PK33AddressIndex, P2PK33AddressIndex, P2PK33AddressIndex, P2PK33Bytes>,
LazyVecFrom1<P2PK33AddressIndex, P2PK33AddressIndex, P2PK33AddressIndex, P2PK33Bytes>,
pub p2pk65addressindex_to_p2pk65addressindex:
ComputedVecFrom1<P2PK65AddressIndex, P2PK65AddressIndex, P2PK65AddressIndex, P2PK65Bytes>,
LazyVecFrom1<P2PK65AddressIndex, P2PK65AddressIndex, P2PK65AddressIndex, P2PK65Bytes>,
pub p2pkhaddressindex_to_p2pkhaddressindex:
ComputedVecFrom1<P2PKHAddressIndex, P2PKHAddressIndex, P2PKHAddressIndex, P2PKHBytes>,
LazyVecFrom1<P2PKHAddressIndex, P2PKHAddressIndex, P2PKHAddressIndex, P2PKHBytes>,
pub p2shaddressindex_to_p2shaddressindex:
ComputedVecFrom1<P2SHAddressIndex, P2SHAddressIndex, P2SHAddressIndex, P2SHBytes>,
LazyVecFrom1<P2SHAddressIndex, P2SHAddressIndex, P2SHAddressIndex, P2SHBytes>,
pub p2traddressindex_to_p2traddressindex:
ComputedVecFrom1<P2TRAddressIndex, P2TRAddressIndex, P2TRAddressIndex, P2TRBytes>,
LazyVecFrom1<P2TRAddressIndex, P2TRAddressIndex, P2TRAddressIndex, P2TRBytes>,
pub p2wpkhaddressindex_to_p2wpkhaddressindex:
ComputedVecFrom1<P2WPKHAddressIndex, P2WPKHAddressIndex, P2WPKHAddressIndex, P2WPKHBytes>,
LazyVecFrom1<P2WPKHAddressIndex, P2WPKHAddressIndex, P2WPKHAddressIndex, P2WPKHBytes>,
pub p2wshaddressindex_to_p2wshaddressindex:
ComputedVecFrom1<P2WSHAddressIndex, P2WSHAddressIndex, P2WSHAddressIndex, P2WSHBytes>,
LazyVecFrom1<P2WSHAddressIndex, P2WSHAddressIndex, P2WSHAddressIndex, P2WSHBytes>,
pub quarterindex_to_first_monthindex: EagerVec<QuarterIndex, MonthIndex>,
pub quarterindex_to_monthindex_count: EagerVec<QuarterIndex, StoredU64>,
pub quarterindex_to_quarterindex: EagerVec<QuarterIndex, QuarterIndex>,
@@ -82,12 +82,12 @@ pub struct Vecs {
pub semesterindex_to_semesterindex: EagerVec<SemesterIndex, SemesterIndex>,
pub txindex_to_height: EagerVec<TxIndex, Height>,
pub txindex_to_input_count:
ComputedVecFrom2<TxIndex, StoredU64, TxIndex, InputIndex, InputIndex, OutputIndex>,
LazyVecFrom2<TxIndex, StoredU64, TxIndex, InputIndex, InputIndex, OutputIndex>,
pub txindex_to_output_count:
ComputedVecFrom2<TxIndex, StoredU64, TxIndex, OutputIndex, OutputIndex, Sats>,
pub txindex_to_txindex: ComputedVecFrom1<TxIndex, TxIndex, TxIndex, Txid>,
LazyVecFrom2<TxIndex, StoredU64, TxIndex, OutputIndex, OutputIndex, Sats>,
pub txindex_to_txindex: LazyVecFrom1<TxIndex, TxIndex, TxIndex, Txid>,
pub unknownoutputindex_to_unknownoutputindex:
ComputedVecFrom1<UnknownOutputIndex, UnknownOutputIndex, UnknownOutputIndex, TxIndex>,
LazyVecFrom1<UnknownOutputIndex, UnknownOutputIndex, UnknownOutputIndex, TxIndex>,
pub weekindex_to_dateindex_count: EagerVec<WeekIndex, StoredU64>,
pub weekindex_to_first_dateindex: EagerVec<WeekIndex, DateIndex>,
pub weekindex_to_weekindex: EagerVec<WeekIndex, WeekIndex>,
@@ -98,52 +98,34 @@ pub struct Vecs {
}
impl Vecs {
pub fn forced_import(
parent: &Path,
version: Version,
indexer: &Indexer,
computation: Computation,
format: Format,
) -> Result<Self> {
pub fn forced_import(parent: &Path, version: Version, indexer: &Indexer) -> Result<Self> {
let db = Database::open(&parent.join("indexes"))?;
db.set_min_len(PAGE_SIZE * 10_000_000)?;
let outputindex_to_outputindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
let outputindex_to_outputindex = LazyVecFrom1::init(
"outputindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.outputindex_to_value.boxed_clone(),
|index, _| Some(index),
)?;
);
let inputindex_to_inputindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
let inputindex_to_inputindex = LazyVecFrom1::init(
"inputindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.inputindex_to_outputindex.boxed_clone(),
|index, _| Some(index),
)?;
);
let txindex_to_txindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
let txindex_to_txindex = LazyVecFrom1::init(
"txindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.txindex_to_txid.boxed_clone(),
|index, _| Some(index),
)?;
);
let txindex_to_input_count = ComputedVec::forced_import_or_init_from_2(
computation,
&db,
let txindex_to_input_count = LazyVecFrom2::init(
"input_count",
version + VERSION + Version::ZERO,
format,
indexer.vecs.txindex_to_first_inputindex.boxed_clone(),
indexer.vecs.inputindex_to_outputindex.boxed_clone(),
|index: TxIndex, txindex_to_first_inputindex_iter, inputindex_to_outputindex_iter| {
@@ -159,14 +141,11 @@ impl Vecs {
StoredU64::from((start..end).count())
})
},
)?;
);
let txindex_to_output_count = ComputedVec::forced_import_or_init_from_2(
computation,
&db,
let txindex_to_output_count = LazyVecFrom2::init(
"output_count",
version + VERSION + Version::ZERO,
format,
indexer.vecs.txindex_to_first_outputindex.boxed_clone(),
indexer.vecs.outputindex_to_value.boxed_clone(),
|index: TxIndex, txindex_to_first_outputindex_iter, outputindex_to_value_iter| {
@@ -182,116 +161,80 @@ impl Vecs {
StoredU64::from((start..end).count())
})
},
)?;
);
let p2pk33addressindex_to_p2pk33addressindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
let p2pk33addressindex_to_p2pk33addressindex = LazyVecFrom1::init(
"p2pk33addressindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.p2pk33addressindex_to_p2pk33bytes.boxed_clone(),
|index, _| Some(index),
)?;
let p2pk65addressindex_to_p2pk65addressindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
);
let p2pk65addressindex_to_p2pk65addressindex = LazyVecFrom1::init(
"p2pk65addressindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.p2pk65addressindex_to_p2pk65bytes.boxed_clone(),
|index, _| Some(index),
)?;
let p2pkhaddressindex_to_p2pkhaddressindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
);
let p2pkhaddressindex_to_p2pkhaddressindex = LazyVecFrom1::init(
"p2pkhaddressindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.p2pkhaddressindex_to_p2pkhbytes.boxed_clone(),
|index, _| Some(index),
)?;
let p2shaddressindex_to_p2shaddressindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
);
let p2shaddressindex_to_p2shaddressindex = LazyVecFrom1::init(
"p2shaddressindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.p2shaddressindex_to_p2shbytes.boxed_clone(),
|index, _| Some(index),
)?;
let p2traddressindex_to_p2traddressindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
);
let p2traddressindex_to_p2traddressindex = LazyVecFrom1::init(
"p2traddressindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.p2traddressindex_to_p2trbytes.boxed_clone(),
|index, _| Some(index),
)?;
let p2wpkhaddressindex_to_p2wpkhaddressindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
);
let p2wpkhaddressindex_to_p2wpkhaddressindex = LazyVecFrom1::init(
"p2wpkhaddressindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.p2wpkhaddressindex_to_p2wpkhbytes.boxed_clone(),
|index, _| Some(index),
)?;
let p2wshaddressindex_to_p2wshaddressindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
);
let p2wshaddressindex_to_p2wshaddressindex = LazyVecFrom1::init(
"p2wshaddressindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.p2wshaddressindex_to_p2wshbytes.boxed_clone(),
|index, _| Some(index),
)?;
let p2aaddressindex_to_p2aaddressindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
);
let p2aaddressindex_to_p2aaddressindex = LazyVecFrom1::init(
"p2aaddressindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.p2aaddressindex_to_p2abytes.boxed_clone(),
|index, _| Some(index),
)?;
let p2msoutputindex_to_p2msoutputindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
);
let p2msoutputindex_to_p2msoutputindex = LazyVecFrom1::init(
"p2msoutputindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.p2msoutputindex_to_txindex.boxed_clone(),
|index, _| Some(index),
)?;
let emptyoutputindex_to_emptyoutputindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
);
let emptyoutputindex_to_emptyoutputindex = LazyVecFrom1::init(
"emptyoutputindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.emptyoutputindex_to_txindex.boxed_clone(),
|index, _| Some(index),
)?;
let unknownoutputindex_to_unknownoutputindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
);
let unknownoutputindex_to_unknownoutputindex = LazyVecFrom1::init(
"unknownoutputindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.unknownoutputindex_to_txindex.boxed_clone(),
|index, _| Some(index),
)?;
let opreturnindex_to_opreturnindex = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
);
let opreturnindex_to_opreturnindex = LazyVecFrom1::init(
"opreturnindex",
version + VERSION + Version::ZERO,
format,
indexer.vecs.opreturnindex_to_txindex.boxed_clone(),
|index, _| Some(index),
)?;
);
Ok(Self {
emptyoutputindex_to_emptyoutputindex,
@@ -312,263 +255,220 @@ impl Vecs {
txindex_to_txindex,
unknownoutputindex_to_unknownoutputindex,
dateindex_to_date: EagerVec::forced_import(
dateindex_to_date: EagerVec::forced_import_compressed(
&db,
"date",
version + VERSION + Version::ZERO,
format,
)?,
dateindex_to_dateindex: EagerVec::forced_import(
dateindex_to_dateindex: EagerVec::forced_import_compressed(
&db,
"dateindex",
version + VERSION + Version::ZERO,
format,
)?,
dateindex_to_first_height: EagerVec::forced_import(
dateindex_to_first_height: EagerVec::forced_import_compressed(
&db,
"first_height",
version + VERSION + Version::ZERO,
format,
)?,
dateindex_to_monthindex: EagerVec::forced_import(
dateindex_to_monthindex: EagerVec::forced_import_compressed(
&db,
"monthindex",
version + VERSION + Version::ZERO,
format,
)?,
dateindex_to_weekindex: EagerVec::forced_import(
dateindex_to_weekindex: EagerVec::forced_import_compressed(
&db,
"weekindex",
version + VERSION + Version::ZERO,
format,
)?,
decadeindex_to_decadeindex: EagerVec::forced_import(
decadeindex_to_decadeindex: EagerVec::forced_import_compressed(
&db,
"decadeindex",
version + VERSION + Version::ZERO,
format,
)?,
decadeindex_to_first_yearindex: EagerVec::forced_import(
decadeindex_to_first_yearindex: EagerVec::forced_import_compressed(
&db,
"first_yearindex",
version + VERSION + Version::ZERO,
format,
)?,
difficultyepoch_to_difficultyepoch: EagerVec::forced_import(
difficultyepoch_to_difficultyepoch: EagerVec::forced_import_compressed(
&db,
"difficultyepoch",
version + VERSION + Version::ZERO,
format,
)?,
difficultyepoch_to_first_height: EagerVec::forced_import(
difficultyepoch_to_first_height: EagerVec::forced_import_compressed(
&db,
"first_height",
version + VERSION + Version::ZERO,
format,
)?,
halvingepoch_to_first_height: EagerVec::forced_import(
halvingepoch_to_first_height: EagerVec::forced_import_compressed(
&db,
"first_height",
version + VERSION + Version::ZERO,
format,
)?,
halvingepoch_to_halvingepoch: EagerVec::forced_import(
halvingepoch_to_halvingepoch: EagerVec::forced_import_compressed(
&db,
"halvingepoch",
version + VERSION + Version::ZERO,
format,
)?,
height_to_date: EagerVec::forced_import(
height_to_date: EagerVec::forced_import_compressed(
&db,
"date",
version + VERSION + Version::ZERO,
format,
)?,
height_to_difficultyepoch: EagerVec::forced_import(
height_to_difficultyepoch: EagerVec::forced_import_compressed(
&db,
"difficultyepoch",
version + VERSION + Version::ZERO,
format,
)?,
height_to_halvingepoch: EagerVec::forced_import(
height_to_halvingepoch: EagerVec::forced_import_compressed(
&db,
"halvingepoch",
version + VERSION + Version::ZERO,
format,
)?,
height_to_height: EagerVec::forced_import(
height_to_height: EagerVec::forced_import_compressed(
&db,
"height",
version + VERSION + Version::ZERO,
format,
)?,
monthindex_to_first_dateindex: EagerVec::forced_import(
monthindex_to_first_dateindex: EagerVec::forced_import_compressed(
&db,
"first_dateindex",
version + VERSION + Version::ZERO,
format,
)?,
monthindex_to_monthindex: EagerVec::forced_import(
monthindex_to_monthindex: EagerVec::forced_import_compressed(
&db,
"monthindex",
version + VERSION + Version::ZERO,
format,
)?,
monthindex_to_quarterindex: EagerVec::forced_import(
monthindex_to_quarterindex: EagerVec::forced_import_compressed(
&db,
"quarterindex",
version + VERSION + Version::ZERO,
format,
)?,
monthindex_to_semesterindex: EagerVec::forced_import(
monthindex_to_semesterindex: EagerVec::forced_import_compressed(
&db,
"semesterindex",
version + VERSION + Version::ZERO,
format,
)?,
monthindex_to_yearindex: EagerVec::forced_import(
monthindex_to_yearindex: EagerVec::forced_import_compressed(
&db,
"yearindex",
version + VERSION + Version::ZERO,
format,
)?,
quarterindex_to_first_monthindex: EagerVec::forced_import(
quarterindex_to_first_monthindex: EagerVec::forced_import_compressed(
&db,
"first_monthindex",
version + VERSION + Version::ZERO,
format,
)?,
semesterindex_to_first_monthindex: EagerVec::forced_import(
semesterindex_to_first_monthindex: EagerVec::forced_import_compressed(
&db,
"first_monthindex",
version + VERSION + Version::ZERO,
format,
)?,
weekindex_to_first_dateindex: EagerVec::forced_import(
weekindex_to_first_dateindex: EagerVec::forced_import_compressed(
&db,
"first_dateindex",
version + VERSION + Version::ZERO,
format,
)?,
yearindex_to_first_monthindex: EagerVec::forced_import(
yearindex_to_first_monthindex: EagerVec::forced_import_compressed(
&db,
"first_monthindex",
version + VERSION + Version::ZERO,
format,
)?,
quarterindex_to_quarterindex: EagerVec::forced_import(
quarterindex_to_quarterindex: EagerVec::forced_import_compressed(
&db,
"quarterindex",
version + VERSION + Version::ZERO,
format,
)?,
semesterindex_to_semesterindex: EagerVec::forced_import(
semesterindex_to_semesterindex: EagerVec::forced_import_compressed(
&db,
"semesterindex",
version + VERSION + Version::ZERO,
format,
)?,
weekindex_to_weekindex: EagerVec::forced_import(
weekindex_to_weekindex: EagerVec::forced_import_compressed(
&db,
"weekindex",
version + VERSION + Version::ZERO,
format,
)?,
yearindex_to_decadeindex: EagerVec::forced_import(
yearindex_to_decadeindex: EagerVec::forced_import_compressed(
&db,
"decadeindex",
version + VERSION + Version::ZERO,
format,
)?,
yearindex_to_yearindex: EagerVec::forced_import(
yearindex_to_yearindex: EagerVec::forced_import_compressed(
&db,
"yearindex",
version + VERSION + Version::ZERO,
format,
)?,
height_to_date_fixed: EagerVec::forced_import(
height_to_date_fixed: EagerVec::forced_import_compressed(
&db,
"date_fixed",
version + VERSION + Version::ZERO,
format,
)?,
height_to_dateindex: EagerVec::forced_import(
height_to_dateindex: EagerVec::forced_import_compressed(
&db,
"dateindex",
version + VERSION + Version::ZERO,
format,
)?,
txindex_to_height: EagerVec::forced_import(
txindex_to_height: EagerVec::forced_import_compressed(
&db,
"height",
version + VERSION + Version::ZERO,
format,
)?,
height_to_timestamp_fixed: EagerVec::forced_import(
height_to_timestamp_fixed: EagerVec::forced_import_compressed(
&db,
"timestamp_fixed",
version + VERSION + Version::ZERO,
format,
)?,
height_to_txindex_count: EagerVec::forced_import(
height_to_txindex_count: EagerVec::forced_import_compressed(
&db,
"txindex_count",
version + VERSION + Version::ZERO,
format,
)?,
dateindex_to_height_count: EagerVec::forced_import(
dateindex_to_height_count: EagerVec::forced_import_compressed(
&db,
"height_count",
version + VERSION + Version::ZERO,
format,
)?,
weekindex_to_dateindex_count: EagerVec::forced_import(
weekindex_to_dateindex_count: EagerVec::forced_import_compressed(
&db,
"dateindex_count",
version + VERSION + Version::ZERO,
format,
)?,
difficultyepoch_to_height_count: EagerVec::forced_import(
difficultyepoch_to_height_count: EagerVec::forced_import_compressed(
&db,
"height_count",
version + VERSION + Version::ZERO,
format,
)?,
monthindex_to_dateindex_count: EagerVec::forced_import(
monthindex_to_dateindex_count: EagerVec::forced_import_compressed(
&db,
"dateindex_count",
version + VERSION + Version::ZERO,
format,
)?,
quarterindex_to_monthindex_count: EagerVec::forced_import(
quarterindex_to_monthindex_count: EagerVec::forced_import_compressed(
&db,
"monthindex_count",
version + VERSION + Version::ZERO,
format,
)?,
semesterindex_to_monthindex_count: EagerVec::forced_import(
semesterindex_to_monthindex_count: EagerVec::forced_import_compressed(
&db,
"monthindex_count",
version + VERSION + Version::ZERO,
format,
)?,
yearindex_to_monthindex_count: EagerVec::forced_import(
yearindex_to_monthindex_count: EagerVec::forced_import_compressed(
&db,
"monthindex_count",
version + VERSION + Version::ZERO,
format,
)?,
decadeindex_to_yearindex_count: EagerVec::forced_import(
decadeindex_to_yearindex_count: EagerVec::forced_import_compressed(
&db,
"yearindex_count",
version + VERSION + Version::ZERO,
format,
)?,
outputindex_to_txindex: EagerVec::forced_import(
outputindex_to_txindex: EagerVec::forced_import_compressed(
&db,
"txindex",
version + VERSION + Version::ZERO,
format,
)?,
db,
@@ -596,24 +496,6 @@ impl Vecs {
// OutputIndex
// ---
self.outputindex_to_outputindex.compute_if_necessary(
starting_indexes.outputindex,
&indexer.vecs.outputindex_to_value,
exit,
)?;
self.txindex_to_output_count.compute_if_necessary(
starting_indexes.txindex,
&indexer.vecs.txindex_to_txid,
exit,
)?;
self.txindex_to_input_count.compute_if_necessary(
starting_indexes.txindex,
&indexer.vecs.txindex_to_txid,
exit,
)?;
self.outputindex_to_txindex.compute_inverse_less_to_more(
starting_indexes.txindex,
&indexer.vecs.txindex_to_first_outputindex,
@@ -621,109 +503,10 @@ impl Vecs {
exit,
)?;
self.p2pk33addressindex_to_p2pk33addressindex
.compute_if_necessary(
starting_indexes.p2pk33addressindex,
&indexer.vecs.p2pk33addressindex_to_p2pk33bytes,
exit,
)?;
self.p2pk65addressindex_to_p2pk65addressindex
.compute_if_necessary(
starting_indexes.p2pk65addressindex,
&indexer.vecs.p2pk65addressindex_to_p2pk65bytes,
exit,
)?;
self.p2pkhaddressindex_to_p2pkhaddressindex
.compute_if_necessary(
starting_indexes.p2pkhaddressindex,
&indexer.vecs.p2pkhaddressindex_to_p2pkhbytes,
exit,
)?;
self.p2shaddressindex_to_p2shaddressindex
.compute_if_necessary(
starting_indexes.p2shaddressindex,
&indexer.vecs.p2shaddressindex_to_p2shbytes,
exit,
)?;
self.p2traddressindex_to_p2traddressindex
.compute_if_necessary(
starting_indexes.p2traddressindex,
&indexer.vecs.p2traddressindex_to_p2trbytes,
exit,
)?;
self.p2wpkhaddressindex_to_p2wpkhaddressindex
.compute_if_necessary(
starting_indexes.p2wpkhaddressindex,
&indexer.vecs.p2wpkhaddressindex_to_p2wpkhbytes,
exit,
)?;
self.p2wshaddressindex_to_p2wshaddressindex
.compute_if_necessary(
starting_indexes.p2wshaddressindex,
&indexer.vecs.p2wshaddressindex_to_p2wshbytes,
exit,
)?;
self.emptyoutputindex_to_emptyoutputindex
.compute_if_necessary(
starting_indexes.emptyoutputindex,
&indexer.vecs.emptyoutputindex_to_txindex,
exit,
)?;
self.p2msoutputindex_to_p2msoutputindex
.compute_if_necessary(
starting_indexes.p2msoutputindex,
&indexer.vecs.p2msoutputindex_to_txindex,
exit,
)?;
self.opreturnindex_to_opreturnindex.compute_if_necessary(
starting_indexes.opreturnindex,
&indexer.vecs.opreturnindex_to_txindex,
exit,
)?;
self.p2aaddressindex_to_p2aaddressindex
.compute_if_necessary(
starting_indexes.p2aaddressindex,
&indexer.vecs.p2aaddressindex_to_p2abytes,
exit,
)?;
self.unknownoutputindex_to_unknownoutputindex
.compute_if_necessary(
starting_indexes.unknownoutputindex,
&indexer.vecs.unknownoutputindex_to_txindex,
exit,
)?;
// ---
// InputIndex
// ---
self.inputindex_to_inputindex.compute_if_necessary(
starting_indexes.inputindex,
&indexer.vecs.inputindex_to_outputindex,
exit,
)?;
// ---
// TxIndex
// ---
self.txindex_to_txindex.compute_if_necessary(
starting_indexes.txindex,
&indexer.vecs.txindex_to_txid,
exit,
)?;
self.height_to_txindex_count.compute_count_from_indexes(
starting_indexes.height,
&indexer.vecs.height_to_first_txindex,
+10 -53
View File
@@ -1,7 +1,4 @@
#![doc = include_str!("../README.md")]
#![doc = "\n## Example\n\n```rust"]
#![doc = include_str!("../examples/computer.rs")]
#![doc = "```"]
use std::path::Path;
@@ -10,7 +7,7 @@ use brk_fetcher::Fetcher;
use brk_indexer::Indexer;
use brk_structs::Version;
use log::info;
use vecdb::{AnyCollectableVec, Computation, Exit, Format};
use vecdb::{AnyCollectableVec, Exit, Format};
mod blocks;
mod cointime;
@@ -29,6 +26,7 @@ mod utils;
use indexes::Indexes;
pub use states::PriceToAmount;
use states::*;
#[derive(Clone)]
@@ -45,7 +43,7 @@ pub struct Computer {
pub cointime: cointime::Vecs,
}
const VERSION: Version = Version::ONE;
const VERSION: Version = Version::TWO;
impl Computer {
/// Do NOT import multiple times or things will break !!!
@@ -56,84 +54,43 @@ impl Computer {
) -> Result<Self> {
let computed_path = outputs_path.join("computed");
let computation = Computation::Lazy;
let format = Format::Compressed;
let indexes = indexes::Vecs::forced_import(
&computed_path,
VERSION + Version::ZERO,
indexer,
computation,
format,
)?;
let indexes =
indexes::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, indexer)?;
let fetched = fetcher.map(|fetcher| {
fetched::Vecs::forced_import(outputs_path, fetcher, VERSION + Version::ZERO).unwrap()
});
let price = fetched.is_some().then(|| {
price::Vecs::forced_import(
&computed_path,
VERSION + Version::ZERO,
computation,
format,
&indexes,
)
.unwrap()
price::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes).unwrap()
});
Ok(Self {
blocks: blocks::Vecs::forced_import(
&computed_path,
VERSION + Version::ZERO,
computation,
format,
&indexes,
)?,
mining: mining::Vecs::forced_import(
&computed_path,
VERSION + Version::ZERO,
computation,
format,
&indexes,
)?,
blocks: blocks::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes)?,
mining: mining::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes)?,
constants: constants::Vecs::forced_import(
&computed_path,
VERSION + Version::ZERO,
computation,
format,
&indexes,
)?,
market: market::Vecs::forced_import(
&computed_path,
VERSION + Version::ZERO,
computation,
format,
&indexes,
)?,
market: market::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes)?,
stateful: stateful::Vecs::forced_import(
&computed_path,
VERSION + Version::ZERO,
computation,
format,
Format::Compressed,
&indexes,
price.as_ref(),
&computed_path.join("states"),
)?,
transactions: transactions::Vecs::forced_import(
&computed_path,
VERSION + Version::ZERO,
indexer,
&indexes,
computation,
format,
price.as_ref(),
)?,
cointime: cointime::Vecs::forced_import(
&computed_path,
VERSION + Version::ZERO,
computation,
format,
&indexes,
price.as_ref(),
)?,
File diff suppressed because it is too large Load Diff
+2 -14
View File
@@ -3,7 +3,7 @@ use std::path::Path;
use brk_error::Result;
use brk_indexer::Indexer;
use brk_structs::{DifficultyEpoch, HalvingEpoch, StoredF64, Version};
use vecdb::{AnyCollectableVec, Computation, Database, Exit, Format, PAGE_SIZE, VecIterator};
use vecdb::{AnyCollectableVec, Database, Exit, PAGE_SIZE, VecIterator};
use crate::grouped::Source;
@@ -25,13 +25,7 @@ pub struct Vecs {
}
impl Vecs {
pub fn forced_import(
parent: &Path,
version: Version,
computation: Computation,
format: Format,
indexes: &indexes::Vecs,
) -> Result<Self> {
pub fn forced_import(parent: &Path, version: Version, indexes: &indexes::Vecs) -> Result<Self> {
let db = Database::open(&parent.join("mining"))?;
db.set_min_len(PAGE_SIZE * 1_000_000)?;
@@ -41,8 +35,6 @@ impl Vecs {
"difficulty",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -51,8 +43,6 @@ impl Vecs {
"difficultyepoch",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -61,8 +51,6 @@ impl Vecs {
"halvingepoch",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
+11 -49
View File
@@ -7,8 +7,8 @@ use brk_structs::{
OHLCDollars, OHLCSats, Open, QuarterIndex, Sats, SemesterIndex, Version, WeekIndex, YearIndex,
};
use vecdb::{
AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Computation, Database, EagerVec, Exit,
Format, GenericStoredVec, PAGE_SIZE, RawVec,
AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, EagerVec, Exit,
GenericStoredVec, PAGE_SIZE, RawVec,
};
use crate::{fetched, grouped::Source};
@@ -73,13 +73,7 @@ const VERSION: Version = Version::ZERO;
const VERSION_IN_SATS: Version = Version::ZERO;
impl Vecs {
pub fn forced_import(
parent: &Path,
version: Version,
computation: Computation,
format: Format,
indexes: &indexes::Vecs,
) -> Result<Self> {
pub fn forced_import(parent: &Path, version: Version, indexes: &indexes::Vecs) -> Result<Self> {
let db = Database::open(&parent.join("price"))?;
db.set_min_len(PAGE_SIZE * 1_000_000)?;
@@ -94,29 +88,25 @@ impl Vecs {
"ohlc_in_sats",
version + VERSION + VERSION_IN_SATS + Version::ZERO,
)?,
dateindex_to_close_in_cents: EagerVec::forced_import(
dateindex_to_close_in_cents: EagerVec::forced_import_compressed(
&db,
"close_in_cents",
version + VERSION + Version::ZERO,
format,
)?,
dateindex_to_high_in_cents: EagerVec::forced_import(
dateindex_to_high_in_cents: EagerVec::forced_import_compressed(
&db,
"high_in_cents",
version + VERSION + Version::ZERO,
format,
)?,
dateindex_to_low_in_cents: EagerVec::forced_import(
dateindex_to_low_in_cents: EagerVec::forced_import_compressed(
&db,
"low_in_cents",
version + VERSION + Version::ZERO,
format,
)?,
dateindex_to_open_in_cents: EagerVec::forced_import(
dateindex_to_open_in_cents: EagerVec::forced_import_compressed(
&db,
"open_in_cents",
version + VERSION + Version::ZERO,
format,
)?,
height_to_ohlc: RawVec::forced_import(&db, "ohlc", version + VERSION + Version::ZERO)?,
height_to_ohlc_in_sats: RawVec::forced_import(
@@ -124,37 +114,31 @@ impl Vecs {
"ohlc_in_sats",
version + VERSION + VERSION_IN_SATS + Version::ZERO,
)?,
height_to_close_in_cents: EagerVec::forced_import(
height_to_close_in_cents: EagerVec::forced_import_compressed(
&db,
"close_in_cents",
version + VERSION + Version::ZERO,
format,
)?,
height_to_high_in_cents: EagerVec::forced_import(
height_to_high_in_cents: EagerVec::forced_import_compressed(
&db,
"high_in_cents",
version + VERSION + Version::ZERO,
format,
)?,
height_to_low_in_cents: EagerVec::forced_import(
height_to_low_in_cents: EagerVec::forced_import_compressed(
&db,
"low_in_cents",
version + VERSION + Version::ZERO,
format,
)?,
height_to_open_in_cents: EagerVec::forced_import(
height_to_open_in_cents: EagerVec::forced_import_compressed(
&db,
"open_in_cents",
version + VERSION + Version::ZERO,
format,
)?,
timeindexes_to_open: ComputedVecsFromDateIndex::forced_import(
&db,
"open",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_first(),
)?,
@@ -163,8 +147,6 @@ impl Vecs {
"high",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_max(),
)?,
@@ -173,8 +155,6 @@ impl Vecs {
"low",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_min(),
)?,
@@ -183,8 +163,6 @@ impl Vecs {
"close",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -193,8 +171,6 @@ impl Vecs {
"open_in_sats",
Source::Compute,
version + VERSION + VERSION_IN_SATS + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_first(),
)?,
@@ -203,8 +179,6 @@ impl Vecs {
"high_in_sats",
Source::Compute,
version + VERSION + VERSION_IN_SATS + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_max(),
)?,
@@ -213,8 +187,6 @@ impl Vecs {
"low_in_sats",
Source::Compute,
version + VERSION + VERSION_IN_SATS + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_min(),
)?,
@@ -223,8 +195,6 @@ impl Vecs {
"close_in_sats",
Source::Compute,
version + VERSION + VERSION_IN_SATS + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -232,56 +202,48 @@ impl Vecs {
&db,
"open",
version + VERSION + Version::ZERO,
format,
VecBuilderOptions::default().add_first(),
)?,
chainindexes_to_high: ComputedVecsFromHeightStrict::forced_import(
&db,
"high",
version + VERSION + Version::ZERO,
format,
VecBuilderOptions::default().add_max(),
)?,
chainindexes_to_low: ComputedVecsFromHeightStrict::forced_import(
&db,
"low",
version + VERSION + Version::ZERO,
format,
VecBuilderOptions::default().add_min(),
)?,
chainindexes_to_close: ComputedVecsFromHeightStrict::forced_import(
&db,
"close",
version + VERSION + Version::ZERO,
format,
VecBuilderOptions::default().add_last(),
)?,
chainindexes_to_open_in_sats: ComputedVecsFromHeightStrict::forced_import(
&db,
"open_in_sats",
version + VERSION + VERSION_IN_SATS + Version::ZERO,
format,
VecBuilderOptions::default().add_first(),
)?,
chainindexes_to_high_in_sats: ComputedVecsFromHeightStrict::forced_import(
&db,
"high_in_sats",
version + VERSION + VERSION_IN_SATS + Version::ZERO,
format,
VecBuilderOptions::default().add_max(),
)?,
chainindexes_to_low_in_sats: ComputedVecsFromHeightStrict::forced_import(
&db,
"low_in_sats",
version + VERSION + VERSION_IN_SATS + Version::ZERO,
format,
VecBuilderOptions::default().add_min(),
)?,
chainindexes_to_close_in_sats: ComputedVecsFromHeightStrict::forced_import(
&db,
"close_in_sats",
version + VERSION + VERSION_IN_SATS + Version::ZERO,
format,
VecBuilderOptions::default().add_last(),
)?,
weekindex_to_ohlc: RawVec::forced_import(
@@ -4,8 +4,8 @@ use brk_error::Result;
use brk_indexer::Indexer;
use brk_structs::{Bitcoin, DateIndex, Dollars, Height, StoredU64, Version};
use vecdb::{
AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Computation, Database, EagerVec, Exit,
Format, GenericStoredVec, VecIterator,
AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, EagerVec, Exit, Format,
GenericStoredVec, VecIterator,
};
use crate::{
@@ -23,7 +23,7 @@ const VERSION: Version = Version::ZERO;
#[derive(Clone)]
pub struct Vecs {
starting_height: Height,
starting_height: Option<Height>,
pub state: Option<AddressCohortState>,
@@ -38,7 +38,6 @@ impl Vecs {
pub fn forced_import(
db: &Database,
cohort_name: Option<&str>,
computation: Computation,
format: Format,
version: Version,
indexes: &indexes::Vecs,
@@ -51,14 +50,13 @@ impl Vecs {
let suffix = |s: &str| cohort_name.map_or(s.to_string(), |name| format!("{name}_{s}"));
Ok(Self {
starting_height: Height::ZERO,
starting_height: None,
state: states_path.map(|states_path| {
AddressCohortState::default_and_import(
AddressCohortState::new(
states_path,
cohort_name.unwrap_or_default(),
compute_dollars,
)
.unwrap()
}),
height_to_address_count: EagerVec::forced_import(
db,
@@ -71,15 +69,12 @@ impl Vecs {
&suffix("address_count"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
inner: common::Vecs::forced_import(
db,
cohort_name,
computation,
format,
version,
indexes,
@@ -93,25 +88,26 @@ impl Vecs {
}
impl DynCohortVecs for Vecs {
fn starting_height(&self) -> Height {
fn min_height_vecs_len(&self) -> usize {
[
self.state.as_ref().map_or(Height::MAX, |state| {
state.height().map_or(Height::MAX, |h| h.incremented())
}),
self.height_to_address_count.len().into(),
self.inner.starting_height(),
self.height_to_address_count.len(),
self.inner.min_height_vecs_len(),
]
.into_iter()
.min()
.unwrap()
}
fn init(&mut self, starting_height: Height) {
if starting_height > self.starting_height() {
unreachable!()
}
fn reset_state_starting_height(&mut self) {
self.starting_height = Some(Height::ZERO);
}
self.starting_height = starting_height;
fn import_state(&mut self, starting_height: Height) -> Result<Height> {
let starting_height = self
.inner
.import_state(starting_height, &mut self.state.as_mut().unwrap().inner)?;
self.starting_height = Some(starting_height);
if let Some(prev_height) = starting_height.decremented() {
self.state.as_mut().unwrap().address_count = *self
@@ -120,10 +116,7 @@ impl DynCohortVecs for Vecs {
.unwrap_get_inner(prev_height);
}
self.inner.init(
&mut self.starting_height,
&mut self.state.as_mut().unwrap().inner,
);
Ok(starting_height)
}
fn validate_computed_versions(&mut self, base_version: Version) -> Result<()> {
@@ -136,7 +129,7 @@ impl DynCohortVecs for Vecs {
}
fn forced_pushed_at(&mut self, height: Height, exit: &Exit) -> Result<()> {
if self.starting_height > height {
if self.starting_height.unwrap() > height {
return Ok(());
}
@@ -7,7 +7,7 @@ use brk_structs::{
Dollars, GroupFilter, Height, Version,
};
use derive_deref::{Deref, DerefMut};
use vecdb::{AnyIterableVec, Computation, Database, Exit, Format};
use vecdb::{AnyIterableVec, Database, Exit, Format};
use crate::{
Indexes, indexes, market, price,
@@ -26,7 +26,6 @@ impl Vecs {
pub fn forced_import(
db: &Database,
version: Version,
_computation: Computation,
format: Format,
indexes: &indexes::Vecs,
price: Option<&price::Vecs>,
@@ -38,7 +37,6 @@ impl Vecs {
_0sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_with_0sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -49,7 +47,6 @@ impl Vecs {
_1sat_to_10sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_1sat_under_10sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -60,7 +57,6 @@ impl Vecs {
_10sats_to_100sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_10sats_under_100sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -71,7 +67,6 @@ impl Vecs {
_100sats_to_1k_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_100sats_under_1k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -82,7 +77,6 @@ impl Vecs {
_1k_sats_to_10k_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_1k_sats_under_10k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -93,7 +87,6 @@ impl Vecs {
_10k_sats_to_100k_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_10k_sats_under_100k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -104,7 +97,6 @@ impl Vecs {
_100k_sats_to_1m_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_100k_sats_under_1m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -115,7 +107,6 @@ impl Vecs {
_1m_sats_to_10m_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_1m_sats_under_10m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -126,7 +117,6 @@ impl Vecs {
_10m_sats_to_1btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_10m_sats_under_1btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -137,7 +127,6 @@ impl Vecs {
_1btc_to_10btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_1btc_under_10btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -148,7 +137,6 @@ impl Vecs {
_10btc_to_100btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_10btc_under_100btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -159,7 +147,6 @@ impl Vecs {
_100btc_to_1k_btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_100btc_under_1k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -170,7 +157,6 @@ impl Vecs {
_1k_btc_to_10k_btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_1k_btc_under_10k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -181,7 +167,6 @@ impl Vecs {
_10k_btc_to_100k_btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_10k_btc_under_100k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -192,7 +177,6 @@ impl Vecs {
_100k_btc_or_more: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_100k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -205,7 +189,6 @@ impl Vecs {
_10sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_10sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -216,7 +199,6 @@ impl Vecs {
_100sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_100sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -227,7 +209,6 @@ impl Vecs {
_1k_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_1k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -238,7 +219,6 @@ impl Vecs {
_10k_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_10k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -249,7 +229,6 @@ impl Vecs {
_100k_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_100k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -260,7 +239,6 @@ impl Vecs {
_1m_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_1m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -271,7 +249,6 @@ impl Vecs {
_10m_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_10m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -282,7 +259,6 @@ impl Vecs {
_1btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_1btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -293,7 +269,6 @@ impl Vecs {
_10btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_10btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -304,7 +279,6 @@ impl Vecs {
_100btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_100btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -315,7 +289,6 @@ impl Vecs {
_1k_btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_1k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -326,7 +299,6 @@ impl Vecs {
_10k_btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_10k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -337,7 +309,6 @@ impl Vecs {
_100k_btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_under_100k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -350,7 +321,6 @@ impl Vecs {
_1sat: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_1sat"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -361,7 +331,6 @@ impl Vecs {
_10sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_10sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -372,7 +341,6 @@ impl Vecs {
_100sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_100sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -383,7 +351,6 @@ impl Vecs {
_1k_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_1k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -394,7 +361,6 @@ impl Vecs {
_10k_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_10k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -405,7 +371,6 @@ impl Vecs {
_100k_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_100k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -416,7 +381,6 @@ impl Vecs {
_1m_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_1m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -427,7 +391,6 @@ impl Vecs {
_10m_sats: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_10m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -438,7 +401,6 @@ impl Vecs {
_1btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_1btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -449,7 +411,6 @@ impl Vecs {
_10btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_10btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -460,7 +421,6 @@ impl Vecs {
_100btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_100btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -471,7 +431,6 @@ impl Vecs {
_1k_btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_1k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -482,7 +441,6 @@ impl Vecs {
_10k_btc: address_cohort::Vecs::forced_import(
db,
Some("addrs_above_10k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -2,7 +2,7 @@ use brk_structs::{ByAddressType, Height};
use derive_deref::{Deref, DerefMut};
use vecdb::VecIterator;
use crate::stateful::addresstype_to_height_to_addresscount::AddressTypeToHeightToAddressCount;
use super::AddressTypeToHeightToAddressCount;
#[derive(Debug, Default, Deref, DerefMut)]
pub struct AddressTypeToAddressCount(ByAddressType<u64>);
@@ -1,9 +1,9 @@
use brk_error::Result;
use brk_structs::{ByAddressType, Height, StoredU64};
use vecdb::{EagerVec, Exit, GenericStoredVec};
use derive_deref::{Deref, DerefMut};
use vecdb::{EagerVec, Exit, GenericStoredVec};
use crate::stateful::addresstype_to_addresscount::AddressTypeToAddressCount;
use super::AddressTypeToAddressCount;
#[derive(Debug, Clone, Deref, DerefMut)]
pub struct AddressTypeToHeightToAddressCount(ByAddressType<EagerVec<Height, StoredU64>>);
@@ -3,10 +3,9 @@ use brk_structs::{ByAddressType, StoredU64};
use derive_deref::{Deref, DerefMut};
use vecdb::{AnyCollectableVec, Exit};
use crate::{
Indexes, grouped::ComputedVecsFromHeight, indexes,
stateful::addresstype_to_height_to_addresscount::AddressTypeToHeightToAddressCount,
};
use crate::{Indexes, grouped::ComputedVecsFromHeight, indexes};
use super::AddressTypeToHeightToAddressCount;
#[derive(Clone, Deref, DerefMut)]
pub struct AddressTypeToIndexesToAddressCount(ByAddressType<ComputedVecsFromHeight<StoredU64>>);
@@ -20,7 +19,6 @@ impl From<ByAddressType<ComputedVecsFromHeight<StoredU64>>> for AddressTypeToInd
impl AddressTypeToIndexesToAddressCount {
pub fn compute(
&mut self,
// height: Height,
indexes: &indexes::Vecs,
starting_indexes: &Indexes,
exit: &Exit,
@@ -76,7 +74,7 @@ impl AddressTypeToIndexesToAddressCount {
)?;
Ok(())
}
//
pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
self.0
.as_typed_vec()
@@ -0,0 +1,13 @@
mod addresscount;
mod height_to_addresscount;
mod height_to_vec;
mod indexes_to_addresscount;
mod typeindex_tree;
mod vec;
pub use addresscount::*;
pub use height_to_addresscount::*;
pub use height_to_vec::*;
pub use indexes_to_addresscount::*;
pub use typeindex_tree::*;
pub use vec::*;
@@ -1,10 +1,8 @@
use std::{collections::BTreeMap, mem};
use brk_structs::TypeIndex;
use brk_structs::{ByAddressType, TypeIndex};
use derive_deref::{Deref, DerefMut};
use super::ByAddressType;
#[derive(Debug, Deref, DerefMut)]
pub struct AddressTypeToTypeIndexTree<T>(ByAddressType<BTreeMap<TypeIndex, T>>);
@@ -1,9 +1,8 @@
use std::mem;
use brk_structs::ByAddressType;
use derive_deref::{Deref, DerefMut};
use super::ByAddressType;
#[derive(Debug, Deref, DerefMut)]
pub struct AddressTypeToVec<T>(ByAddressType<Vec<T>>);
+18 -92
View File
@@ -1,11 +1,11 @@
use brk_error::Result;
use brk_error::{Error, Result};
use brk_indexer::Indexer;
use brk_structs::{
Bitcoin, DateIndex, Dollars, Height, Sats, StoredF32, StoredF64, StoredU64, Version,
};
use vecdb::{
AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Computation,
Database, EagerVec, Exit, Format, GenericStoredVec, VecIterator,
AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database,
EagerVec, Exit, Format, GenericStoredVec, VecIterator,
};
use crate::{
@@ -128,7 +128,6 @@ impl Vecs {
pub fn forced_import(
db: &Database,
cohort_name: Option<&str>,
computation: Computation,
format: Format,
version: Version,
indexes: &indexes::Vecs,
@@ -209,8 +208,6 @@ impl Vecs {
&suffix("supply_in_profit"),
dateindex_to_supply_in_profit.as_ref().map(|v | v.boxed_clone()).into(),
version + VERSION + Version::ZERO,
format,
computation,
VecBuilderOptions::default().add_last(),
compute_dollars,
indexes,
@@ -233,8 +230,6 @@ impl Vecs {
&suffix("supply_even"),
dateindex_to_supply_even.as_ref().map(|v | v.boxed_clone()).into(),
version + VERSION + Version::ZERO,
format,
computation,
VecBuilderOptions::default().add_last(),
compute_dollars,
indexes,
@@ -257,8 +252,6 @@ impl Vecs {
&suffix("supply_in_loss"),
dateindex_to_supply_in_loss.as_ref().map(|v | v.boxed_clone()).into(),
version + VERSION + Version::ZERO,
format,
computation,
VecBuilderOptions::default().add_last(),
compute_dollars,
indexes,
@@ -281,8 +274,6 @@ impl Vecs {
&suffix("unrealized_profit"),
dateindex_to_unrealized_profit.as_ref().map(|v | v.boxed_clone()).into(),
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -322,8 +313,6 @@ impl Vecs {
&suffix("unrealized_loss"),
dateindex_to_unrealized_loss.as_ref().map(|v | v.boxed_clone()).into(),
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -345,8 +334,6 @@ impl Vecs {
&suffix("realized_cap"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -358,8 +345,6 @@ impl Vecs {
&suffix("min_price_paid"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -371,8 +356,6 @@ impl Vecs {
&suffix("max_price_paid"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -397,8 +380,6 @@ impl Vecs {
&suffix("supply"),
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
VecBuilderOptions::default().add_last(),
compute_dollars,
indexes,
@@ -414,8 +395,6 @@ impl Vecs {
&suffix("utxo_count"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -425,8 +404,6 @@ impl Vecs {
&suffix("realized_price"),
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -438,8 +415,6 @@ impl Vecs {
&suffix("realized_price"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
ratio_extended,
)
@@ -460,8 +435,6 @@ impl Vecs {
&suffix("realized_profit"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_sum()
@@ -484,8 +457,6 @@ impl Vecs {
&suffix("realized_loss"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_sum()
@@ -499,8 +470,6 @@ impl Vecs {
&suffix("negative_realized_loss"),
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)
@@ -521,8 +490,6 @@ impl Vecs {
&suffix("value_created"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum(),
)
@@ -534,8 +501,6 @@ impl Vecs {
&suffix("realized_value"),
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum(),
)
@@ -556,8 +521,6 @@ impl Vecs {
&suffix("adjusted_value_created"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum(),
)
@@ -578,8 +541,6 @@ impl Vecs {
&suffix("value_destroyed"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum(),
)
@@ -600,8 +561,6 @@ impl Vecs {
&suffix("adjusted_value_destroyed"),
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum(),
)
@@ -613,8 +572,6 @@ impl Vecs {
&suffix("realized_cap_30d_change"),
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -626,8 +583,6 @@ impl Vecs {
&suffix("net_realized_profit_and_loss"),
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_sum()
@@ -675,8 +630,6 @@ impl Vecs {
&suffix("halved_supply"),
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
VecBuilderOptions::default().add_last(),
compute_dollars,
indexes,
@@ -696,8 +649,6 @@ impl Vecs {
&suffix("negative_unrealized_loss"),
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -718,8 +669,6 @@ impl Vecs {
&suffix("net_unrealized_profit_and_loss"),
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -743,8 +692,6 @@ impl Vecs {
&suffix("net_unrealized_profit_and_loss_relative_to_market_cap"),
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -757,8 +704,6 @@ impl Vecs {
&suffix("realized_profit_relative_to_realized_cap"),
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum(),
)
@@ -770,8 +715,6 @@ impl Vecs {
&suffix("realized_loss_relative_to_realized_cap"),
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum(),
)
@@ -784,8 +727,6 @@ impl Vecs {
&suffix("net_realized_profit_and_loss_relative_to_realized_cap"),
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum(),
)
@@ -858,8 +799,6 @@ impl Vecs {
&suffix("supply_even_relative_to_own_supply"),
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -871,8 +810,6 @@ impl Vecs {
&suffix("supply_in_loss_relative_to_own_supply"),
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -884,8 +821,6 @@ impl Vecs {
&suffix("supply_in_profit_relative_to_own_supply"),
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -897,8 +832,6 @@ impl Vecs {
&suffix("supply_relative_to_circulating_supply"),
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -945,8 +878,6 @@ impl Vecs {
&suffix("supply_even_relative_to_circulating_supply"),
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -960,8 +891,6 @@ impl Vecs {
&suffix("supply_in_loss_relative_to_circulating_supply"),
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -975,8 +904,6 @@ impl Vecs {
&suffix("supply_in_profit_relative_to_circulating_supply"),
Source::Compute,
version + VERSION + Version::ONE,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)
@@ -999,8 +926,6 @@ impl Vecs {
&suffix("coinblocks_destroyed"),
Source::Compute,
version + VERSION + Version::TWO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -1009,8 +934,6 @@ impl Vecs {
&suffix("coindays_destroyed"),
Source::Compute,
version + VERSION + Version::TWO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -1020,8 +943,6 @@ impl Vecs {
&suffix("net_realized_profit_and_loss_cumulative_30d_change"),
Source::Compute,
version + VERSION + Version::new(3),
format,
computation,
indexes,
VecBuilderOptions::default().add_last()
)
@@ -1033,8 +954,6 @@ impl Vecs {
&suffix("net_realized_profit_and_loss_cumulative_30d_change_relative_to_realized_cap"),
Source::Compute,
version + VERSION + Version::new(3),
format,
computation,
indexes,
VecBuilderOptions::default().add_last()
)
@@ -1046,8 +965,6 @@ impl Vecs {
&suffix("net_realized_profit_and_loss_cumulative_30d_change_relative_to_market_cap"),
Source::Compute,
version + VERSION + Version::new(3),
format,
computation,
indexes,
VecBuilderOptions::default().add_last()
)
@@ -1056,7 +973,7 @@ impl Vecs {
})
}
pub fn starting_height(&self) -> Height {
pub fn min_height_vecs_len(&self) -> usize {
[
self.height_to_supply.len(),
self.height_to_utxo_count.len(),
@@ -1106,13 +1023,20 @@ impl Vecs {
self.height_to_satblocks_destroyed.len(),
]
.into_iter()
.map(Height::from)
.min()
.unwrap()
}
pub fn init(&mut self, starting_height: &mut Height, state: &mut CohortState) {
if let Some(prev_height) = starting_height.decremented() {
pub fn import_state(
&mut self,
starting_height: Height,
state: &mut CohortState,
) -> Result<Height> {
if let Some(mut prev_height) = starting_height.decremented() {
if self.height_to_realized_cap.as_mut().is_some() {
prev_height = state.import_at_or_before(prev_height)?;
}
state.supply.value = self
.height_to_supply
.into_iter()
@@ -1127,6 +1051,10 @@ impl Vecs {
.into_iter()
.unwrap_get_inner(prev_height);
}
Ok(prev_height.incremented())
} else {
Err(Error::Str("Unset"))
}
}
@@ -2301,7 +2229,6 @@ impl Vecs {
.as_mut()
.unwrap()
.compute_rest(
indexes,
starting_indexes,
exit,
Some(self.dateindex_to_unrealized_profit.as_ref().unwrap()),
@@ -2310,7 +2237,6 @@ impl Vecs {
.as_mut()
.unwrap()
.compute_rest(
indexes,
starting_indexes,
exit,
Some(self.dateindex_to_unrealized_loss.as_ref().unwrap()),
+234 -225
View File
@@ -1,4 +1,10 @@
use std::{cmp::Ordering, collections::BTreeMap, mem, path::Path, thread};
use std::{
cmp::Ordering,
collections::{BTreeMap, BTreeSet},
mem,
path::Path,
thread,
};
use brk_error::Result;
use brk_indexer::Indexer;
@@ -12,8 +18,8 @@ use brk_structs::{
use log::info;
use rayon::prelude::*;
use vecdb::{
AnyCollectableVec, AnyStoredVec, AnyVec, CollectableVec, Computation, Database, EagerVec, Exit,
Format, GenericStoredVec, PAGE_SIZE, RawVec, Reader, Stamp, StoredIndex, VecIterator,
AnyCollectableVec, AnyStoredVec, AnyVec, CollectableVec, Database, EagerVec, Exit, Format,
GenericStoredVec, ImportOptions, PAGE_SIZE, RawVec, Reader, Stamp, StoredIndex, VecIterator,
};
use crate::{
@@ -25,28 +31,18 @@ use crate::{
mod address_cohort;
mod address_cohorts;
mod addresstype_to_addresscount;
mod addresstype_to_height_to_addresscount;
mod addresstype_to_indexes_to_addresscount;
mod addresstype_to_typeindex_tree;
mod addresstype_to_vec;
mod addresstype;
mod common;
mod height_to_addresstype_to_vec;
mod range_map;
mod r#trait;
mod utxo_cohort;
mod utxo_cohorts;
mod withaddressdatasource;
use addresstype_to_addresscount::*;
use addresstype_to_height_to_addresscount::*;
use addresstype_to_indexes_to_addresscount::*;
pub use addresstype_to_typeindex_tree::*;
pub use addresstype_to_vec::*;
use height_to_addresstype_to_vec::*;
use addresstype::*;
use range_map::*;
use r#trait::*;
pub use withaddressdatasource::WithAddressDataSource;
use withaddressdatasource::*;
const VERSION: Version = Version::new(21);
@@ -54,19 +50,10 @@ const VERSION: Version = Version::new(21);
pub struct Vecs {
db: Database,
// ---
// States
// ---
pub chain_state: RawVec<Height, SupplyState>,
pub height_to_unspendable_supply: EagerVec<Height, Sats>,
pub indexes_to_unspendable_supply: ComputedValueVecsFromHeight,
pub height_to_opreturn_supply: EagerVec<Height, Sats>,
pub indexes_to_opreturn_supply: ComputedValueVecsFromHeight,
pub addresstype_to_height_to_address_count: AddressTypeToHeightToAddressCount,
pub addresstype_to_height_to_empty_address_count: AddressTypeToHeightToAddressCount,
pub addresstype_to_indexes_to_address_count: AddressTypeToIndexesToAddressCount,
pub addresstype_to_indexes_to_empty_address_count: AddressTypeToIndexesToAddressCount,
pub utxo_cohorts: utxo_cohorts::Vecs,
pub address_cohorts: address_cohorts::Vecs,
pub p2pk33addressindex_to_anyaddressindex: RawVec<P2PK33AddressIndex, AnyAddressIndex>,
pub p2pk65addressindex_to_anyaddressindex: RawVec<P2PK65AddressIndex, AnyAddressIndex>,
pub p2pkhaddressindex_to_anyaddressindex: RawVec<P2PKHAddressIndex, AnyAddressIndex>,
@@ -78,65 +65,74 @@ pub struct Vecs {
pub loadedaddressindex_to_loadedaddressdata: RawVec<LoadedAddressIndex, LoadedAddressData>,
pub emptyaddressindex_to_emptyaddressdata: RawVec<EmptyAddressIndex, EmptyAddressData>,
pub utxo_cohorts: utxo_cohorts::Vecs,
pub address_cohorts: address_cohorts::Vecs,
pub height_to_unspendable_supply: EagerVec<Height, Sats>,
pub height_to_opreturn_supply: EagerVec<Height, Sats>,
pub addresstype_to_height_to_address_count: AddressTypeToHeightToAddressCount,
pub addresstype_to_height_to_empty_address_count: AddressTypeToHeightToAddressCount,
// ---
// Computed
// ---
pub addresstype_to_indexes_to_address_count: AddressTypeToIndexesToAddressCount,
pub addresstype_to_indexes_to_empty_address_count: AddressTypeToIndexesToAddressCount,
pub indexes_to_unspendable_supply: ComputedValueVecsFromHeight,
pub indexes_to_opreturn_supply: ComputedValueVecsFromHeight,
pub indexes_to_address_count: ComputedVecsFromHeight<StoredU64>,
pub indexes_to_empty_address_count: ComputedVecsFromHeight<StoredU64>,
}
const SAVED_STAMPED_CHANGES: u16 = 10;
impl Vecs {
pub fn forced_import(
parent: &Path,
version: Version,
computation: Computation,
format: Format,
indexes: &indexes::Vecs,
price: Option<&price::Vecs>,
states_path: &Path,
) -> Result<Self> {
let db = Database::open(&parent.join("stateful"))?;
let db_path = parent.join("stateful");
let states_path = db_path.join("states");
let db = Database::open(&db_path)?;
db.set_min_len(PAGE_SIZE * 20_000_000)?;
db.set_min_regions(50_000)?;
let compute_dollars = price.is_some();
let chain_db = Database::open(&parent.join("chain"))?;
Ok(Self {
chain_state: RawVec::forced_import(
&chain_db,
"chain",
version + VERSION + Version::ZERO,
chain_state: RawVec::forced_import_with(
ImportOptions::new(&db, "chain", version + VERSION + Version::ZERO)
.with_saved_stamped_changes(SAVED_STAMPED_CHANGES),
)?,
height_to_unspendable_supply: EagerVec::forced_import(
height_to_unspendable_supply: EagerVec::forced_import_compressed(
&db,
"unspendable_supply",
version + VERSION + Version::ZERO,
format,
)?,
indexes_to_unspendable_supply: ComputedValueVecsFromHeight::forced_import(
&db,
"unspendable_supply",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
VecBuilderOptions::default().add_last(),
compute_dollars,
indexes,
)?,
height_to_opreturn_supply: EagerVec::forced_import(
height_to_opreturn_supply: EagerVec::forced_import_compressed(
&db,
"opreturn_supply",
version + VERSION + Version::ZERO,
format,
)?,
indexes_to_opreturn_supply: ComputedValueVecsFromHeight::forced_import(
&db,
"opreturn_supply",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
VecBuilderOptions::default().add_last(),
compute_dollars,
indexes,
@@ -146,8 +142,6 @@ impl Vecs {
"address_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -156,112 +150,94 @@ impl Vecs {
"empty_address_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
addresstype_to_height_to_address_count: AddressTypeToHeightToAddressCount::from(
ByAddressType {
p2pk65: EagerVec::forced_import(
p2pk65: EagerVec::forced_import_compressed(
&db,
"p2pk65_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2pk33: EagerVec::forced_import(
p2pk33: EagerVec::forced_import_compressed(
&db,
"p2pk33_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2pkh: EagerVec::forced_import(
p2pkh: EagerVec::forced_import_compressed(
&db,
"p2pkh_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2sh: EagerVec::forced_import(
p2sh: EagerVec::forced_import_compressed(
&db,
"p2sh_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2wpkh: EagerVec::forced_import(
p2wpkh: EagerVec::forced_import_compressed(
&db,
"p2wpkh_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2wsh: EagerVec::forced_import(
p2wsh: EagerVec::forced_import_compressed(
&db,
"p2wsh_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2tr: EagerVec::forced_import(
p2tr: EagerVec::forced_import_compressed(
&db,
"p2tr_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2a: EagerVec::forced_import(
p2a: EagerVec::forced_import_compressed(
&db,
"p2a_address_count",
version + VERSION + Version::ZERO,
format,
)?,
},
),
addresstype_to_height_to_empty_address_count: AddressTypeToHeightToAddressCount::from(
ByAddressType {
p2pk65: EagerVec::forced_import(
p2pk65: EagerVec::forced_import_compressed(
&db,
"p2pk65_empty_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2pk33: EagerVec::forced_import(
p2pk33: EagerVec::forced_import_compressed(
&db,
"p2pk33_empty_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2pkh: EagerVec::forced_import(
p2pkh: EagerVec::forced_import_compressed(
&db,
"p2pkh_empty_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2sh: EagerVec::forced_import(
p2sh: EagerVec::forced_import_compressed(
&db,
"p2sh_empty_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2wpkh: EagerVec::forced_import(
p2wpkh: EagerVec::forced_import_compressed(
&db,
"p2wpkh_empty_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2wsh: EagerVec::forced_import(
p2wsh: EagerVec::forced_import_compressed(
&db,
"p2wsh_empty_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2tr: EagerVec::forced_import(
p2tr: EagerVec::forced_import_compressed(
&db,
"p2tr_empty_address_count",
version + VERSION + Version::ZERO,
format,
)?,
p2a: EagerVec::forced_import(
p2a: EagerVec::forced_import_compressed(
&db,
"p2a_empty_address_count",
version + VERSION + Version::ZERO,
format,
)?,
},
),
@@ -272,8 +248,6 @@ impl Vecs {
"p2pk65_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -282,8 +256,6 @@ impl Vecs {
"p2pk33_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -292,8 +264,6 @@ impl Vecs {
"p2pkh_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -302,8 +272,6 @@ impl Vecs {
"p2sh_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -312,8 +280,6 @@ impl Vecs {
"p2wpkh_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -322,8 +288,6 @@ impl Vecs {
"p2wsh_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -332,8 +296,6 @@ impl Vecs {
"p2tr_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -342,8 +304,6 @@ impl Vecs {
"p2a_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -356,8 +316,6 @@ impl Vecs {
"p2pk65_empty_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -366,8 +324,6 @@ impl Vecs {
"p2pk33_empty_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -376,8 +332,6 @@ impl Vecs {
"p2pkh_empty_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -386,8 +340,6 @@ impl Vecs {
"p2sh_empty_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -396,8 +348,6 @@ impl Vecs {
"p2wpkh_empty_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -406,8 +356,6 @@ impl Vecs {
"p2wsh_empty_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -416,8 +364,6 @@ impl Vecs {
"p2tr_empty_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -426,8 +372,6 @@ impl Vecs {
"p2a_empty_address_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -436,72 +380,60 @@ impl Vecs {
utxo_cohorts: utxo_cohorts::Vecs::forced_import(
&db,
version,
computation,
format,
indexes,
price,
states_path,
&states_path,
)?,
address_cohorts: address_cohorts::Vecs::forced_import(
&db,
version,
computation,
format,
indexes,
price,
states_path,
&states_path,
)?,
p2aaddressindex_to_anyaddressindex: RawVec::forced_import(
&db,
"anyaddressindex",
version + VERSION + Version::ZERO,
p2aaddressindex_to_anyaddressindex: RawVec::forced_import_with(
ImportOptions::new(&db, "anyaddressindex", version + VERSION + Version::ZERO)
.with_saved_stamped_changes(SAVED_STAMPED_CHANGES),
)?,
p2pk33addressindex_to_anyaddressindex: RawVec::forced_import(
&db,
"anyaddressindex",
version + VERSION + Version::ZERO,
p2pk33addressindex_to_anyaddressindex: RawVec::forced_import_with(
ImportOptions::new(&db, "anyaddressindex", version + VERSION + Version::ZERO)
.with_saved_stamped_changes(SAVED_STAMPED_CHANGES),
)?,
p2pk65addressindex_to_anyaddressindex: RawVec::forced_import(
&db,
"anyaddressindex",
version + VERSION + Version::ZERO,
p2pk65addressindex_to_anyaddressindex: RawVec::forced_import_with(
ImportOptions::new(&db, "anyaddressindex", version + VERSION + Version::ZERO)
.with_saved_stamped_changes(SAVED_STAMPED_CHANGES),
)?,
p2pkhaddressindex_to_anyaddressindex: RawVec::forced_import(
&db,
"anyaddressindex",
version + VERSION + Version::ZERO,
p2pkhaddressindex_to_anyaddressindex: RawVec::forced_import_with(
ImportOptions::new(&db, "anyaddressindex", version + VERSION + Version::ZERO)
.with_saved_stamped_changes(SAVED_STAMPED_CHANGES),
)?,
p2shaddressindex_to_anyaddressindex: RawVec::forced_import(
&db,
"anyaddressindex",
version + VERSION + Version::ZERO,
p2shaddressindex_to_anyaddressindex: RawVec::forced_import_with(
ImportOptions::new(&db, "anyaddressindex", version + VERSION + Version::ZERO)
.with_saved_stamped_changes(SAVED_STAMPED_CHANGES),
)?,
p2traddressindex_to_anyaddressindex: RawVec::forced_import(
&db,
"anyaddressindex",
version + VERSION + Version::ZERO,
p2traddressindex_to_anyaddressindex: RawVec::forced_import_with(
ImportOptions::new(&db, "anyaddressindex", version + VERSION + Version::ZERO)
.with_saved_stamped_changes(SAVED_STAMPED_CHANGES),
)?,
p2wpkhaddressindex_to_anyaddressindex: RawVec::forced_import(
&db,
"anyaddressindex",
version + VERSION + Version::ZERO,
p2wpkhaddressindex_to_anyaddressindex: RawVec::forced_import_with(
ImportOptions::new(&db, "anyaddressindex", version + VERSION + Version::ZERO)
.with_saved_stamped_changes(SAVED_STAMPED_CHANGES),
)?,
p2wshaddressindex_to_anyaddressindex: RawVec::forced_import(
&db,
"anyaddressindex",
version + VERSION + Version::ZERO,
p2wshaddressindex_to_anyaddressindex: RawVec::forced_import_with(
ImportOptions::new(&db, "anyaddressindex", version + VERSION + Version::ZERO)
.with_saved_stamped_changes(SAVED_STAMPED_CHANGES),
)?,
loadedaddressindex_to_loadedaddressdata: RawVec::forced_import(
&db,
"loadedaddressdata",
version + VERSION + Version::ZERO,
loadedaddressindex_to_loadedaddressdata: RawVec::forced_import_with(
ImportOptions::new(&db, "loadedaddressdata", version + VERSION + Version::ZERO)
.with_saved_stamped_changes(SAVED_STAMPED_CHANGES),
)?,
emptyaddressindex_to_emptyaddressdata: RawVec::forced_import(
&db,
"emptyaddressdata",
version + VERSION + Version::ZERO,
emptyaddressindex_to_emptyaddressdata: RawVec::forced_import_with(
ImportOptions::new(&db, "emptyaddressdata", version + VERSION + Version::ZERO)
.with_saved_stamped_changes(SAVED_STAMPED_CHANGES),
)?,
db,
@@ -632,18 +564,16 @@ impl Vecs {
base_version + self.height_to_opreturn_supply.inner_version(),
)?;
let mut chain_state: Vec<BlockState> = vec![];
let mut chain_state_starting_height = Height::from(self.chain_state.len());
let stateful_starting_height = match separate_utxo_vecs
.par_iter_mut()
.map(|(_, v)| v.starting_height())
.map(|(_, v)| Height::from(v.min_height_vecs_len()))
.min()
.unwrap_or_default()
.min(
separate_address_vecs
.par_iter_mut()
.map(|(_, v)| v.starting_height())
.map(|(_, v)| Height::from(v.min_height_vecs_len()))
.min()
.unwrap_or_default(),
)
@@ -663,7 +593,92 @@ impl Vecs {
.cmp(&chain_state_starting_height)
{
Ordering::Greater => unreachable!(),
Ordering::Equal => {
Ordering::Equal => chain_state_starting_height,
Ordering::Less => Height::ZERO,
};
// info!("stateful_starting_height = {stateful_starting_height}");
// let stateful_starting_height = stateful_starting_height
// .checked_sub(Height::new(1))
// .unwrap_or_default();
// info!("stateful_starting_height = {stateful_starting_height}");
let starting_height = starting_indexes.height.min(stateful_starting_height);
info!("starting_height = {starting_height}");
let last_height = Height::from(indexer.vecs.height_to_blockhash.stamp());
info!("last_height = {last_height}");
if starting_height <= last_height {
info!("starting_height = {starting_height}");
let starting_height = if starting_height.is_not_zero() {
let mut set = [
self.chain_state.rollback_before(starting_height.into())?,
self.p2pk33addressindex_to_anyaddressindex
.rollback_before(starting_height.into())?,
self.p2pk65addressindex_to_anyaddressindex
.rollback_before(starting_height.into())?,
self.p2pkhaddressindex_to_anyaddressindex
.rollback_before(starting_height.into())?,
self.p2shaddressindex_to_anyaddressindex
.rollback_before(starting_height.into())?,
self.p2traddressindex_to_anyaddressindex
.rollback_before(starting_height.into())?,
self.p2wpkhaddressindex_to_anyaddressindex
.rollback_before(starting_height.into())?,
self.p2wshaddressindex_to_anyaddressindex
.rollback_before(starting_height.into())?,
self.p2aaddressindex_to_anyaddressindex
.rollback_before(starting_height.into())?,
self.loadedaddressindex_to_loadedaddressdata
.rollback_before(starting_height.into())?,
self.emptyaddressindex_to_emptyaddressdata
.rollback_before(starting_height.into())?,
]
.into_iter()
.enumerate()
.map(|(i, s)| {
let h = Height::from(s).incremented();
dbg!((i, s, h));
h
})
.collect::<BTreeSet<Height>>();
if set.len() == 1 {
set.pop_first().unwrap()
} else {
Height::ZERO
}
} else {
Height::ZERO
};
let starting_height = if starting_height.is_not_zero()
&& separate_utxo_vecs
.iter_mut()
.map(|(_, v)| v.import_state(starting_height).unwrap_or_default())
.all(|h| h == starting_height)
{
starting_height
} else {
Height::ZERO
};
info!("starting_height = {starting_height}");
let starting_height = if starting_height.is_not_zero()
&& separate_address_vecs
.iter_mut()
.map(|(_, v)| v.import_state(starting_height).unwrap_or_default())
.all(|h| h == starting_height)
{
starting_height
} else {
Height::ZERO
};
info!("starting_height = {starting_height}");
let mut chain_state: Vec<BlockState>;
if starting_height.is_not_zero() {
chain_state = self
.chain_state
.collect_range(None, None)?
@@ -682,49 +697,41 @@ impl Vecs {
}
})
.collect::<Vec<_>>();
chain_state_starting_height
} else {
info!("Starting processing utxos from the start");
// std::process::exit(0);
chain_state = vec![];
self.p2pk33addressindex_to_anyaddressindex.reset()?;
self.p2pk65addressindex_to_anyaddressindex.reset()?;
self.p2pkhaddressindex_to_anyaddressindex.reset()?;
self.p2shaddressindex_to_anyaddressindex.reset()?;
self.p2traddressindex_to_anyaddressindex.reset()?;
self.p2wpkhaddressindex_to_anyaddressindex.reset()?;
self.p2wshaddressindex_to_anyaddressindex.reset()?;
self.p2aaddressindex_to_anyaddressindex.reset()?;
self.loadedaddressindex_to_loadedaddressdata.reset()?;
self.emptyaddressindex_to_emptyaddressdata.reset()?;
separate_utxo_vecs.par_iter_mut().try_for_each(|(_, v)| {
v.reset_state_starting_height();
v.state.as_mut().unwrap().reset_price_to_amount_if_needed()
})?;
separate_address_vecs
.par_iter_mut()
.try_for_each(|(_, v)| {
v.reset_state_starting_height();
v.state.as_mut().unwrap().reset_price_to_amount_if_needed()
})?;
}
Ordering::Less => Height::ZERO,
};
let starting_height = starting_indexes.height.min(stateful_starting_height);
chain_state_starting_height = starting_height;
if starting_height.is_zero() {
info!("Starting processing utxos from the start");
starting_indexes.update_from_height(starting_height, indexes);
// TODO: rollback instead
chain_state = vec![];
chain_state_starting_height = Height::ZERO;
self.p2pk33addressindex_to_anyaddressindex.reset()?;
self.p2pk65addressindex_to_anyaddressindex.reset()?;
self.p2pkhaddressindex_to_anyaddressindex.reset()?;
self.p2shaddressindex_to_anyaddressindex.reset()?;
self.p2traddressindex_to_anyaddressindex.reset()?;
self.p2wpkhaddressindex_to_anyaddressindex.reset()?;
self.p2wshaddressindex_to_anyaddressindex.reset()?;
self.p2aaddressindex_to_anyaddressindex.reset()?;
self.loadedaddressindex_to_loadedaddressdata.reset()?;
self.emptyaddressindex_to_emptyaddressdata.reset()?;
info!("Resetting utxo price maps...");
separate_utxo_vecs
.par_iter_mut()
.flat_map(|(_, v)| v.state.as_mut())
.try_for_each(|state| state.reset_price_to_amount())?;
info!("Resetting address price maps...");
separate_address_vecs
.par_iter_mut()
.try_for_each(|(_, v)| v.state.as_mut().unwrap().reset_price_to_amount())?;
};
let last_height = Height::from(indexer.vecs.height_to_blockhash.stamp());
if starting_height <= last_height {
let inputindex_to_outputindex_reader = inputindex_to_outputindex.create_reader();
let outputindex_to_value_reader = outputindex_to_value.create_reader();
let outputindex_to_outputtype_reader = outputindex_to_outputtype.create_reader();
@@ -756,16 +763,6 @@ impl Vecs {
let mut dateindex_to_first_height_iter = dateindex_to_first_height.into_iter();
let mut dateindex_to_height_count_iter = dateindex_to_height_count.into_iter();
starting_indexes.update_from_height(starting_height, indexes);
separate_utxo_vecs
.par_iter_mut()
.for_each(|(_, v)| v.init(starting_height));
separate_address_vecs
.par_iter_mut()
.for_each(|(_, v)| v.init(starting_height));
let height_to_close_vec =
height_to_close.map(|height_to_close| height_to_close.collect().unwrap());
@@ -811,6 +808,13 @@ impl Vecs {
&mut anyaddressindex_to_anyaddressdata_reader_opt,
);
let last_height = Height::from(
height_to_date_fixed
.len()
.checked_sub(1)
.unwrap_or_default(),
);
(height.unwrap_to_usize()..height_to_date_fixed.len())
.map(Height::from)
.try_for_each(|_height| -> Result<()> {
@@ -1038,7 +1042,7 @@ impl Vecs {
}
height_to_addresstype_to_typedindex_to_data
.entry(height)
.entry(prev_height)
.or_default()
.get_mut(output_type)
.unwrap()
@@ -1208,13 +1212,13 @@ impl Vecs {
)
})?;
if height != Height::ZERO && height.unwrap_to_usize() % 10_000 == 0 {
if height != last_height && height != Height::ZERO && height.unwrap_to_usize() % 10_000 == 0 {
let _lock = exit.lock();
addresstypeindex_to_anyaddressindex_reader_opt.take();
anyaddressindex_to_anyaddressdata_reader_opt.take();
self.flush_states(height, &chain_state, mem::take(&mut addresstype_to_typeindex_to_loadedaddressdata), mem::take(&mut addresstype_to_typeindex_to_emptyaddressdata), exit)?;
self.flush_states(height, &chain_state, mem::take(&mut addresstype_to_typeindex_to_loadedaddressdata), mem::take(&mut addresstype_to_typeindex_to_emptyaddressdata), false, exit)?;
self.reset_readers_options(
&mut addresstypeindex_to_anyaddressindex_reader_opt,
@@ -1235,6 +1239,7 @@ impl Vecs {
&chain_state,
mem::take(&mut addresstype_to_typeindex_to_loadedaddressdata),
mem::take(&mut addresstype_to_typeindex_to_emptyaddressdata),
true,
exit,
)?;
}
@@ -1565,6 +1570,7 @@ impl Vecs {
mut addresstype_to_typeindex_to_emptyaddressdata: AddressTypeToTypeIndexTree<
WithAddressDataSource<EmptyAddressData>,
>,
with_changes: bool,
exit: &Exit,
) -> Result<()> {
info!("Flushing...");
@@ -1734,32 +1740,35 @@ impl Vecs {
})
})?;
let stamp = Stamp::from(height);
self.p2pk33addressindex_to_anyaddressindex
.stamped_flush(Stamp::from(height))?;
.stamped_flush_maybe_with_changes(stamp, with_changes)?;
self.p2pk65addressindex_to_anyaddressindex
.stamped_flush(Stamp::from(height))?;
.stamped_flush_maybe_with_changes(stamp, with_changes)?;
self.p2pkhaddressindex_to_anyaddressindex
.stamped_flush(Stamp::from(height))?;
.stamped_flush_maybe_with_changes(stamp, with_changes)?;
self.p2shaddressindex_to_anyaddressindex
.stamped_flush(Stamp::from(height))?;
.stamped_flush_maybe_with_changes(stamp, with_changes)?;
self.p2traddressindex_to_anyaddressindex
.stamped_flush(Stamp::from(height))?;
.stamped_flush_maybe_with_changes(stamp, with_changes)?;
self.p2wpkhaddressindex_to_anyaddressindex
.stamped_flush(Stamp::from(height))?;
.stamped_flush_maybe_with_changes(stamp, with_changes)?;
self.p2wshaddressindex_to_anyaddressindex
.stamped_flush(Stamp::from(height))?;
.stamped_flush_maybe_with_changes(stamp, with_changes)?;
self.p2aaddressindex_to_anyaddressindex
.stamped_flush(Stamp::from(height))?;
.stamped_flush_maybe_with_changes(stamp, with_changes)?;
self.loadedaddressindex_to_loadedaddressdata
.stamped_flush(Stamp::from(height))?;
.stamped_flush_maybe_with_changes(stamp, with_changes)?;
self.emptyaddressindex_to_emptyaddressdata
.stamped_flush(Stamp::from(height))?;
.stamped_flush_maybe_with_changes(stamp, with_changes)?;
self.chain_state.truncate_if_needed(Height::ZERO)?;
chain_state.iter().for_each(|block_state| {
self.chain_state.push(block_state.supply.clone());
});
self.chain_state.flush()?;
self.chain_state
.stamped_flush_maybe_with_changes(stamp, with_changes)?;
Ok(())
}
+3 -2
View File
@@ -6,9 +6,10 @@ use vecdb::{AnyCollectableVec, AnyIterableVec, Exit};
use crate::{Indexes, indexes, market, price};
pub trait DynCohortVecs: Send + Sync {
fn starting_height(&self) -> Height;
fn min_height_vecs_len(&self) -> usize;
fn reset_state_starting_height(&mut self);
fn init(&mut self, starting_height: Height);
fn import_state(&mut self, starting_height: Height) -> Result<Height>;
fn validate_computed_versions(&mut self, base_version: Version) -> Result<()>;
+17 -25
View File
@@ -3,7 +3,7 @@ use std::{ops::Deref, path::Path};
use brk_error::Result;
use brk_indexer::Indexer;
use brk_structs::{Bitcoin, DateIndex, Dollars, Height, Version};
use vecdb::{AnyCollectableVec, AnyIterableVec, Computation, Database, Exit, Format};
use vecdb::{AnyCollectableVec, AnyIterableVec, Database, Exit, Format};
use crate::{
Indexes, UTXOCohortState, indexes, market, price,
@@ -15,7 +15,7 @@ use crate::{
#[derive(Clone)]
pub struct Vecs {
starting_height: Height,
state_starting_height: Option<Height>,
pub state: Option<UTXOCohortState>,
@@ -27,7 +27,6 @@ impl Vecs {
pub fn forced_import(
db: &Database,
cohort_name: Option<&str>,
computation: Computation,
format: Format,
version: Version,
indexes: &indexes::Vecs,
@@ -40,21 +39,19 @@ impl Vecs {
let compute_dollars = price.is_some();
Ok(Self {
starting_height: Height::ZERO,
state_starting_height: None,
state: states_path.map(|states_path| {
UTXOCohortState::default_and_import(
UTXOCohortState::new(
states_path,
cohort_name.unwrap_or_default(),
compute_dollars,
)
.unwrap()
}),
inner: common::Vecs::forced_import(
db,
cohort_name,
computation,
format,
version,
indexes,
@@ -68,27 +65,22 @@ impl Vecs {
}
impl DynCohortVecs for Vecs {
fn starting_height(&self) -> Height {
[
self.state.as_ref().map_or(Height::MAX, |state| {
state.height().map_or(Height::MAX, |h| h.incremented())
}),
self.inner.starting_height(),
]
.into_iter()
.min()
.unwrap()
fn min_height_vecs_len(&self) -> usize {
self.inner.min_height_vecs_len()
}
fn init(&mut self, starting_height: Height) {
if starting_height > self.starting_height() {
unreachable!()
}
fn reset_state_starting_height(&mut self) {
self.state_starting_height = Some(Height::ZERO);
}
self.starting_height = starting_height;
fn import_state(&mut self, starting_height: Height) -> Result<Height> {
let starting_height = self
.inner
.import_state(starting_height, self.state.as_mut().unwrap())?;
self.inner
.init(&mut self.starting_height, self.state.as_mut().unwrap());
self.state_starting_height = Some(starting_height);
Ok(starting_height)
}
fn validate_computed_versions(&mut self, base_version: Version) -> Result<()> {
@@ -96,7 +88,7 @@ impl DynCohortVecs for Vecs {
}
fn forced_pushed_at(&mut self, height: Height, exit: &Exit) -> Result<()> {
if self.starting_height > height {
if self.state_starting_height.unwrap() > height {
return Ok(());
}
@@ -8,7 +8,7 @@ use brk_structs::{
Height, Timestamp, UTXOGroups, Version,
};
use derive_deref::{Deref, DerefMut};
use vecdb::{AnyIterableVec, Computation, Database, Exit, Format, StoredIndex};
use vecdb::{AnyIterableVec, Database, Exit, Format, StoredIndex};
use crate::{
Indexes, indexes, market, price,
@@ -27,7 +27,6 @@ impl Vecs {
pub fn forced_import(
db: &Database,
version: Version,
_computation: Computation,
format: Format,
indexes: &indexes::Vecs,
price: Option<&price::Vecs>,
@@ -38,7 +37,6 @@ impl Vecs {
all: utxo_cohort::Vecs::forced_import(
db,
None,
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -52,7 +50,6 @@ impl Vecs {
short: utxo_cohort::Vecs::forced_import(
db,
Some("short_term_holders"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -65,7 +62,6 @@ impl Vecs {
long: utxo_cohort::Vecs::forced_import(
db,
Some("long_term_holders"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -80,7 +76,6 @@ impl Vecs {
_0: utxo_cohort::Vecs::forced_import(
db,
Some("epoch_0"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -93,7 +88,6 @@ impl Vecs {
_1: utxo_cohort::Vecs::forced_import(
db,
Some("epoch_1"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -106,7 +100,6 @@ impl Vecs {
_2: utxo_cohort::Vecs::forced_import(
db,
Some("epoch_2"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -119,7 +112,6 @@ impl Vecs {
_3: utxo_cohort::Vecs::forced_import(
db,
Some("epoch_3"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -132,7 +124,6 @@ impl Vecs {
_4: utxo_cohort::Vecs::forced_import(
db,
Some("epoch_4"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -147,7 +138,6 @@ impl Vecs {
p2pk65: utxo_cohort::Vecs::forced_import(
db,
Some("p2pk65"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -160,7 +150,6 @@ impl Vecs {
p2pk33: utxo_cohort::Vecs::forced_import(
db,
Some("p2pk33"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -173,7 +162,6 @@ impl Vecs {
p2pkh: utxo_cohort::Vecs::forced_import(
db,
Some("p2pkh"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -186,7 +174,6 @@ impl Vecs {
p2sh: utxo_cohort::Vecs::forced_import(
db,
Some("p2sh"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -199,7 +186,6 @@ impl Vecs {
p2wpkh: utxo_cohort::Vecs::forced_import(
db,
Some("p2wpkh"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -212,7 +198,6 @@ impl Vecs {
p2wsh: utxo_cohort::Vecs::forced_import(
db,
Some("p2wsh"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -225,7 +210,6 @@ impl Vecs {
p2tr: utxo_cohort::Vecs::forced_import(
db,
Some("p2tr"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -238,7 +222,6 @@ impl Vecs {
p2a: utxo_cohort::Vecs::forced_import(
db,
Some("p2a"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -251,7 +234,6 @@ impl Vecs {
p2ms: utxo_cohort::Vecs::forced_import(
db,
Some("p2ms_outputs"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -264,7 +246,6 @@ impl Vecs {
empty: utxo_cohort::Vecs::forced_import(
db,
Some("empty_outputs"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -277,7 +258,6 @@ impl Vecs {
unknown: utxo_cohort::Vecs::forced_import(
db,
Some("unknown_outputs"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -292,7 +272,6 @@ impl Vecs {
_1w: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_1w_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -305,7 +284,6 @@ impl Vecs {
_1m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_1m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -318,7 +296,6 @@ impl Vecs {
_2m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_2m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -331,7 +308,6 @@ impl Vecs {
_3m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_3m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -344,7 +320,6 @@ impl Vecs {
_4m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_4m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -357,7 +332,6 @@ impl Vecs {
_5m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_5m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -370,7 +344,6 @@ impl Vecs {
_6m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_6m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -383,7 +356,6 @@ impl Vecs {
_1y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_1y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -396,7 +368,6 @@ impl Vecs {
_2y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_2y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -409,7 +380,6 @@ impl Vecs {
_3y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_3y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -422,7 +392,6 @@ impl Vecs {
_4y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_4y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -435,7 +404,6 @@ impl Vecs {
_5y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_5y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -448,7 +416,6 @@ impl Vecs {
_6y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_6y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -461,7 +428,6 @@ impl Vecs {
_7y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_7y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -474,7 +440,6 @@ impl Vecs {
_8y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_8y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -487,7 +452,6 @@ impl Vecs {
_10y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_10y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -500,7 +464,6 @@ impl Vecs {
_12y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_12y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -513,7 +476,6 @@ impl Vecs {
_15y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_15y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -528,7 +490,6 @@ impl Vecs {
_1d: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_1d_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -541,7 +502,6 @@ impl Vecs {
_1w: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_1w_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -554,7 +514,6 @@ impl Vecs {
_1m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_1m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -567,7 +526,6 @@ impl Vecs {
_2m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_2m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -580,7 +538,6 @@ impl Vecs {
_3m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_3m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -593,7 +550,6 @@ impl Vecs {
_4m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_4m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -606,7 +562,6 @@ impl Vecs {
_5m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_5m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -619,7 +574,6 @@ impl Vecs {
_6m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_6m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -632,7 +586,6 @@ impl Vecs {
_1y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_1y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -645,7 +598,6 @@ impl Vecs {
_2y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_2y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -658,7 +610,6 @@ impl Vecs {
_3y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_3y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -671,7 +622,6 @@ impl Vecs {
_4y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_4y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -684,7 +634,6 @@ impl Vecs {
_5y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_5y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -697,7 +646,6 @@ impl Vecs {
_6y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_6y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -710,7 +658,6 @@ impl Vecs {
_7y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_7y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -723,7 +670,6 @@ impl Vecs {
_8y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_8y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -736,7 +682,6 @@ impl Vecs {
_10y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_10y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -749,7 +694,6 @@ impl Vecs {
_12y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_12y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -764,7 +708,6 @@ impl Vecs {
up_to_1d: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_up_to_1d_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -777,7 +720,6 @@ impl Vecs {
_1d_to_1w: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_1d_up_to_1w_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -790,7 +732,6 @@ impl Vecs {
_1w_to_1m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_1w_up_to_1m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -803,7 +744,6 @@ impl Vecs {
_1m_to_2m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_1m_up_to_2m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -816,7 +756,6 @@ impl Vecs {
_2m_to_3m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_2m_up_to_3m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -829,7 +768,6 @@ impl Vecs {
_3m_to_4m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_3m_up_to_4m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -842,7 +780,6 @@ impl Vecs {
_4m_to_5m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_4m_up_to_5m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -855,7 +792,6 @@ impl Vecs {
_5m_to_6m: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_5m_up_to_6m_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -868,7 +804,6 @@ impl Vecs {
_6m_to_1y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_6m_up_to_1y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -881,7 +816,6 @@ impl Vecs {
_1y_to_2y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_1y_up_to_2y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -894,7 +828,6 @@ impl Vecs {
_2y_to_3y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_2y_up_to_3y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -907,7 +840,6 @@ impl Vecs {
_3y_to_4y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_3y_up_to_4y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -920,7 +852,6 @@ impl Vecs {
_4y_to_5y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_4y_up_to_5y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -933,7 +864,6 @@ impl Vecs {
_5y_to_6y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_5y_up_to_6y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -946,7 +876,6 @@ impl Vecs {
_6y_to_7y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_6y_up_to_7y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -959,7 +888,6 @@ impl Vecs {
_7y_to_8y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_7y_up_to_8y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -972,7 +900,6 @@ impl Vecs {
_8y_to_10y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_8y_up_to_10y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -985,7 +912,6 @@ impl Vecs {
_10y_to_12y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_10y_up_to_12y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -998,7 +924,6 @@ impl Vecs {
_12y_to_15y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_12y_up_to_15y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1011,7 +936,6 @@ impl Vecs {
from_15y: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_at_least_15y_old"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1026,7 +950,6 @@ impl Vecs {
_0sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_with_0sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1039,7 +962,6 @@ impl Vecs {
_1sat_to_10sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_1sat_under_10sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1052,7 +974,6 @@ impl Vecs {
_10sats_to_100sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_10sats_under_100sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1065,7 +986,6 @@ impl Vecs {
_100sats_to_1k_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_100sats_under_1k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1078,7 +998,6 @@ impl Vecs {
_1k_sats_to_10k_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_1k_sats_under_10k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1091,7 +1010,6 @@ impl Vecs {
_10k_sats_to_100k_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_10k_sats_under_100k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1104,7 +1022,6 @@ impl Vecs {
_100k_sats_to_1m_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_100k_sats_under_1m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1117,7 +1034,6 @@ impl Vecs {
_1m_sats_to_10m_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_1m_sats_under_10m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1130,7 +1046,6 @@ impl Vecs {
_10m_sats_to_1btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_10m_sats_under_1btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1143,7 +1058,6 @@ impl Vecs {
_1btc_to_10btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_1btc_under_10btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1156,7 +1070,6 @@ impl Vecs {
_10btc_to_100btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_10btc_under_100btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1169,7 +1082,6 @@ impl Vecs {
_100btc_to_1k_btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_100btc_under_1k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1182,7 +1094,6 @@ impl Vecs {
_1k_btc_to_10k_btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_1k_btc_under_10k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1195,7 +1106,6 @@ impl Vecs {
_10k_btc_to_100k_btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_10k_btc_under_100k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1208,7 +1118,6 @@ impl Vecs {
_100k_btc_or_more: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_100k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1223,7 +1132,6 @@ impl Vecs {
_10sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_10sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1236,7 +1144,6 @@ impl Vecs {
_100sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_100sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1249,7 +1156,6 @@ impl Vecs {
_1k_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_1k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1262,7 +1168,6 @@ impl Vecs {
_10k_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_10k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1275,7 +1180,6 @@ impl Vecs {
_100k_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_100k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1288,7 +1192,6 @@ impl Vecs {
_1m_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_1m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1301,7 +1204,6 @@ impl Vecs {
_10m_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_10m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1314,7 +1216,6 @@ impl Vecs {
_1btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_1btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1327,7 +1228,6 @@ impl Vecs {
_10btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_10btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1340,7 +1240,6 @@ impl Vecs {
_100btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_100btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1353,7 +1252,6 @@ impl Vecs {
_1k_btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_1k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1366,7 +1264,6 @@ impl Vecs {
_10k_btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_10k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1379,7 +1276,6 @@ impl Vecs {
_100k_btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_under_100k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1394,7 +1290,6 @@ impl Vecs {
_1sat: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_1sat"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1407,7 +1302,6 @@ impl Vecs {
_10sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_10sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1420,7 +1314,6 @@ impl Vecs {
_100sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_100sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1433,7 +1326,6 @@ impl Vecs {
_1k_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_1k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1446,7 +1338,6 @@ impl Vecs {
_10k_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_10k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1459,7 +1350,6 @@ impl Vecs {
_100k_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_100k_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1472,7 +1362,6 @@ impl Vecs {
_1m_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_1m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1485,7 +1374,6 @@ impl Vecs {
_10m_sats: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_10m_sats"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1498,7 +1386,6 @@ impl Vecs {
_1btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_1btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1511,7 +1398,6 @@ impl Vecs {
_10btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_10btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1524,7 +1410,6 @@ impl Vecs {
_100btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_100btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1537,7 +1422,6 @@ impl Vecs {
_1k_btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_1k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
@@ -1550,7 +1434,6 @@ impl Vecs {
_10k_btc: utxo_cohort::Vecs::forced_import(
db,
Some("utxos_above_10k_btc"),
_computation,
format,
version + VERSION + Version::ZERO,
indexes,
+5 -1
View File
@@ -1,13 +1,17 @@
use std::ops::{Add, AddAssign, SubAssign};
use brk_structs::{Dollars, Timestamp};
use serde::Serialize;
use super::SupplyState;
#[derive(Debug, Clone)]
#[derive(Debug, Clone, Serialize)]
pub struct BlockState {
#[serde(flatten)]
pub supply: SupplyState,
#[serde(skip)]
pub price: Option<Dollars>,
#[serde(skip)]
pub timestamp: Timestamp,
}
@@ -14,19 +14,15 @@ pub struct AddressCohortState {
}
impl AddressCohortState {
pub fn default_and_import(path: &Path, name: &str, compute_dollars: bool) -> Result<Self> {
Ok(Self {
pub fn new(path: &Path, name: &str, compute_dollars: bool) -> Self {
Self {
address_count: 0,
inner: CohortState::default_and_import(path, name, compute_dollars)?,
})
inner: CohortState::new(path, name, compute_dollars),
}
}
pub fn height(&self) -> Option<Height> {
self.inner.height()
}
pub fn reset_price_to_amount(&mut self) -> Result<()> {
self.inner.reset_price_to_amount()
pub fn reset_price_to_amount_if_needed(&mut self) -> Result<()> {
self.inner.reset_price_to_amount_if_needed()
}
pub fn reset_single_iteration_values(&mut self) {
@@ -8,38 +8,47 @@ use crate::{PriceToAmount, RealizedState, SupplyState, UnrealizedState};
#[derive(Clone)]
pub struct CohortState {
pub supply: SupplyState,
pub realized: Option<RealizedState>,
pub satblocks_destroyed: Sats,
pub satdays_destroyed: Sats,
price_to_amount: PriceToAmount,
price_to_amount: Option<PriceToAmount>,
}
impl CohortState {
pub fn default_and_import(path: &Path, name: &str, compute_dollars: bool) -> Result<Self> {
Ok(Self {
pub fn new(path: &Path, name: &str, compute_dollars: bool) -> Self {
Self {
supply: SupplyState::default(),
realized: compute_dollars.then_some(RealizedState::NAN),
satblocks_destroyed: Sats::ZERO,
satdays_destroyed: Sats::ZERO,
price_to_amount: PriceToAmount::forced_import(path, name),
})
price_to_amount: compute_dollars.then_some(PriceToAmount::create(path, name)),
}
}
pub fn height(&self) -> Option<Height> {
self.price_to_amount.height()
pub fn import_at_or_before(&mut self, height: Height) -> Result<Height> {
if let Some(price_to_amount) = self.price_to_amount.as_mut() {
price_to_amount.import_at_or_before(height)
} else {
Ok(height)
}
}
pub fn reset_price_to_amount(&mut self) -> Result<()> {
self.price_to_amount.reset()
pub fn reset_price_to_amount_if_needed(&mut self) -> Result<()> {
if let Some(price_to_amount) = self.price_to_amount.as_mut() {
price_to_amount.clean()?;
price_to_amount.init();
}
Ok(())
}
pub fn price_to_amount_first_key_value(&self) -> Option<(&Dollars, &Sats)> {
self.price_to_amount.first_key_value()
self.price_to_amount.as_ref().unwrap().first_key_value()
}
pub fn price_to_amount_last_key_value(&self) -> Option<(&Dollars, &Sats)> {
self.price_to_amount.last_key_value()
self.price_to_amount.as_ref().unwrap().last_key_value()
}
pub fn reset_single_iteration_values(&mut self) {
@@ -58,7 +67,10 @@ impl CohortState {
{
let price = price.unwrap();
realized.increment(supply_state, price);
self.price_to_amount.increment(price, supply_state);
self.price_to_amount
.as_mut()
.unwrap()
.increment(price, supply_state);
}
}
@@ -74,7 +86,10 @@ impl CohortState {
&& let Some(realized) = self.realized.as_mut()
{
realized.increment_(realized_cap);
self.price_to_amount.increment(realized_price, supply_state);
self.price_to_amount
.as_mut()
.unwrap()
.increment(realized_price, supply_state);
}
}
@@ -86,7 +101,10 @@ impl CohortState {
{
let price = price.unwrap();
realized.decrement(supply_state, price);
self.price_to_amount.decrement(price, supply_state);
self.price_to_amount
.as_mut()
.unwrap()
.decrement(price, supply_state);
}
}
@@ -102,7 +120,10 @@ impl CohortState {
&& let Some(realized) = self.realized.as_mut()
{
realized.decrement_(realized_cap);
self.price_to_amount.decrement(realized_price, supply_state);
self.price_to_amount
.as_mut()
.unwrap()
.decrement(realized_price, supply_state);
}
}
@@ -133,13 +154,19 @@ impl CohortState {
if let Some((price, supply)) = price_to_amount_increment
&& supply.value.is_not_zero()
{
self.price_to_amount.increment(price, supply);
self.price_to_amount
.as_mut()
.unwrap()
.increment(price, supply);
}
if let Some((price, supply)) = price_to_amount_decrement
&& supply.value.is_not_zero()
{
self.price_to_amount.decrement(price, supply);
self.price_to_amount
.as_mut()
.unwrap()
.decrement(price, supply);
}
}
}
@@ -196,12 +223,18 @@ impl CohortState {
if let Some((price, supply)) = price_to_amount_increment
&& supply.value.is_not_zero()
{
self.price_to_amount.increment(price, supply);
self.price_to_amount
.as_mut()
.unwrap()
.increment(price, supply);
}
if let Some((price, supply)) = price_to_amount_decrement
&& supply.value.is_not_zero()
{
self.price_to_amount.decrement(price, supply);
self.price_to_amount
.as_mut()
.unwrap()
.decrement(price, supply);
}
}
}
@@ -212,7 +245,7 @@ impl CohortState {
height_price: Dollars,
date_price: Option<Dollars>,
) -> (UnrealizedState, Option<UnrealizedState>) {
if self.price_to_amount.is_empty() {
if self.price_to_amount.as_ref().unwrap().is_empty() {
return (
UnrealizedState::NAN,
date_price.map(|_| UnrealizedState::NAN),
@@ -255,23 +288,30 @@ impl CohortState {
}
};
self.price_to_amount.iter().for_each(|(&price, &sats)| {
update_state(price, height_price, sats, &mut height_unrealized_state);
self.price_to_amount
.as_ref()
.unwrap()
.iter()
.for_each(|(&price, &sats)| {
update_state(price, height_price, sats, &mut height_unrealized_state);
if let Some(date_price) = date_price {
update_state(
price,
date_price,
sats,
date_unrealized_state.as_mut().unwrap(),
)
}
});
if let Some(date_price) = date_price {
update_state(
price,
date_price,
sats,
date_unrealized_state.as_mut().unwrap(),
)
}
});
(height_unrealized_state, date_unrealized_state)
}
pub fn commit(&mut self, height: Height) -> Result<()> {
self.price_to_amount.flush(height)
if let Some(price_to_amount) = self.price_to_amount.as_mut() {
price_to_amount.flush(height)?;
}
Ok(())
}
}
+4 -13
View File
@@ -1,7 +1,6 @@
use std::path::Path;
use brk_error::Result;
use brk_structs::Height;
use derive_deref::{Deref, DerefMut};
use super::CohortState;
@@ -10,19 +9,11 @@ use super::CohortState;
pub struct UTXOCohortState(CohortState);
impl UTXOCohortState {
pub fn default_and_import(path: &Path, name: &str, compute_dollars: bool) -> Result<Self> {
Ok(Self(CohortState::default_and_import(
path,
name,
compute_dollars,
)?))
pub fn new(path: &Path, name: &str, compute_dollars: bool) -> Self {
Self(CohortState::new(path, name, compute_dollars))
}
pub fn height(&self) -> Option<Height> {
self.0.height()
}
pub fn reset_price_to_amount(&mut self) -> Result<()> {
self.0.reset_price_to_amount()
pub fn reset_price_to_amount_if_needed(&mut self) -> Result<()> {
self.0.reset_price_to_amount_if_needed()
}
}
@@ -1,13 +1,13 @@
use std::{
collections::BTreeMap,
fs,
io::{Cursor, Read},
path::{Path, PathBuf},
};
use brk_error::{Error, Result};
use brk_structs::{Dollars, Height, Sats};
use derive_deref::{Deref, DerefMut};
use pco::standalone::{simple_decompress, simpler_compress};
use serde::{Deserialize, Serialize};
use zerocopy::{FromBytes, IntoBytes};
@@ -16,144 +16,155 @@ use crate::states::SupplyState;
#[derive(Clone, Debug)]
pub struct PriceToAmount {
pathbuf: PathBuf,
height: Option<Height>,
state: State,
state: Option<State>,
}
const STATE_AT_: &str = "state_at_";
const STATE_TO_KEEP: usize = 10;
impl PriceToAmount {
pub fn forced_import(path: &Path, name: &str) -> Self {
Self::import(path, name).unwrap_or_else(|_| {
// dbg!(e);
Self {
pathbuf: Self::path_(path, name),
height: None,
state: State::default(),
}
})
pub fn create(path: &Path, name: &str) -> Self {
Self {
pathbuf: path.join(format!("{name}_price_to_amount")),
state: None,
}
}
pub fn import(path: &Path, name: &str) -> Result<Self> {
let path = Self::path_(path, name);
fs::create_dir_all(&path)?;
let state = State::deserialize(&fs::read(Self::path_state_(&path))?)?;
Ok(Self {
height: Height::try_from(Self::path_height_(&path).as_path()).ok(),
pathbuf: path,
state,
})
pub fn import_at_or_before(&mut self, height: Height) -> Result<Height> {
let files = self.read_dir(None)?;
let (&height, path) = files
.range(..=height)
.next_back()
.ok_or(Error::Str("Not found"))?;
self.state = Some(State::deserialize(&fs::read(path)?)?);
Ok(height)
}
pub fn iter(&self) -> impl Iterator<Item = (&Dollars, &Sats)> {
self.state.iter()
self.state.as_ref().unwrap().iter()
}
pub fn is_empty(&self) -> bool {
self.state.is_empty()
self.state.as_ref().unwrap().is_empty()
}
pub fn first_key_value(&self) -> Option<(&Dollars, &Sats)> {
self.state.first_key_value()
self.state.as_ref().unwrap().first_key_value()
}
pub fn last_key_value(&self) -> Option<(&Dollars, &Sats)> {
self.state.last_key_value()
self.state.as_ref().unwrap().last_key_value()
}
pub fn increment(&mut self, price: Dollars, supply_state: &SupplyState) {
*self.state.entry(price).or_default() += supply_state.value;
*self.state.as_mut().unwrap().entry(price).or_default() += supply_state.value;
}
pub fn decrement(&mut self, price: Dollars, supply_state: &SupplyState) {
if let Some(amount) = self.state.get_mut(&price) {
if let Some(amount) = self.state.as_mut().unwrap().get_mut(&price) {
*amount -= supply_state.value;
if *amount == Sats::ZERO {
self.state.remove(&price);
self.state.as_mut().unwrap().remove(&price);
}
} else {
dbg!(&self.state, price, &self.pathbuf);
dbg!(price, &self.pathbuf);
unreachable!();
}
}
pub fn reset(&mut self) -> Result<()> {
self.state.clear();
self.height = None;
fs::remove_dir_all(&self.pathbuf)?;
pub fn init(&mut self) {
self.state.replace(State::default());
}
pub fn clean(&mut self) -> Result<()> {
let _ = fs::remove_dir_all(&self.pathbuf);
fs::create_dir_all(&self.pathbuf)?;
Ok(())
}
fn read_dir(&self, keep_only_before: Option<Height>) -> Result<BTreeMap<Height, PathBuf>> {
Ok(fs::read_dir(&self.pathbuf)?
.filter_map(|entry| {
let path = entry.ok()?.path();
let name = path.file_name()?.to_str()?;
let height_str = name.strip_prefix(STATE_AT_).unwrap_or(name);
if let Ok(h) = height_str.parse::<u32>().map(Height::from) {
if keep_only_before.is_none_or(|height| h < height) {
Some((h, path))
} else {
let _ = fs::remove_file(path);
None
}
} else {
None
}
})
.collect::<BTreeMap<Height, PathBuf>>())
}
pub fn flush(&mut self, height: Height) -> Result<()> {
self.height = Some(height);
height.write(&self.path_height())?;
fs::write(self.path_state(), self.state.serialize())?;
let files = self.read_dir(Some(height))?;
for (_, path) in files
.iter()
.take(files.len().saturating_sub(STATE_TO_KEEP - 1))
{
fs::remove_file(path)?;
}
fs::write(
self.path_state(height),
self.state.as_ref().unwrap().serialize()?,
)?;
Ok(())
}
pub fn height(&self) -> Option<Height> {
self.height
fn path_state(&self, height: Height) -> PathBuf {
Self::path_state_(&self.pathbuf, height)
}
fn path_(path: &Path, name: &str) -> PathBuf {
path.join(format!("{name}_price_to_amount"))
}
fn path_state(&self) -> PathBuf {
Self::path_state_(&self.pathbuf)
}
fn path_state_(path: &Path) -> PathBuf {
path.join("state")
}
fn path_height(&self) -> PathBuf {
Self::path_height_(&self.pathbuf)
}
fn path_height_(path: &Path) -> PathBuf {
path.join("height")
fn path_state_(path: &Path, height: Height) -> PathBuf {
path.join(u32::from(height).to_string())
}
}
#[derive(Clone, Default, Debug, Deref, DerefMut, Serialize, Deserialize)]
struct State(BTreeMap<Dollars, Sats>);
const COMPRESSION_LEVEL: usize = 4;
impl State {
fn serialize(&self) -> Vec<u8> {
let len = self.len();
fn serialize(&self) -> vecdb::Result<Vec<u8>> {
let keys: Vec<f64> = self.keys().cloned().map(f64::from).collect();
let mut buffer = Vec::with_capacity(8 + len * 16);
let values: Vec<u64> = self.values().cloned().map(u64::from).collect();
buffer.extend(len.as_bytes());
let compressed_keys = simpler_compress(&keys, COMPRESSION_LEVEL)?;
let compressed_values = simpler_compress(&values, COMPRESSION_LEVEL)?;
self.iter().for_each(|(key, value)| {
buffer.extend(key.as_bytes());
buffer.extend(value.as_bytes());
});
let mut buffer = Vec::new();
buffer.extend(keys.len().as_bytes());
buffer.extend(compressed_keys.len().as_bytes());
buffer.extend(compressed_keys);
buffer.extend(compressed_values);
buffer
Ok(buffer)
}
fn deserialize(data: &[u8]) -> Result<Self> {
let mut cursor = Cursor::new(data);
let mut buffer = [0u8; 8];
fn deserialize(data: &[u8]) -> vecdb::Result<Self> {
let entry_count = usize::read_from_bytes(&data[0..8])?;
let keys_len = usize::read_from_bytes(&data[8..16])?;
cursor
.read_exact(&mut buffer)
.map_err(|_| Error::Str("Failed to read entry count"))?;
let entry_count = usize::read_from_bytes(&buffer)?;
let keys: Vec<f64> = simple_decompress(&data[16..16 + keys_len])?;
let values: Vec<u64> = simple_decompress(&data[16 + keys_len..])?;
let mut map = BTreeMap::new();
let map: BTreeMap<Dollars, Sats> = keys
.into_iter()
.zip(values)
.map(|(k, v)| (Dollars::from(k), Sats::from(v)))
.collect();
for _ in 0..entry_count {
cursor.read_exact(&mut buffer)?;
let key = Dollars::read_from_bytes(&buffer)?;
cursor.read_exact(&mut buffer)?;
let value = Sats::read_from_bytes(&buffer)?;
map.insert(key, value);
}
assert_eq!(map.len(), entry_count);
Ok(Self(map))
}
+51 -193
View File
@@ -7,9 +7,8 @@ use brk_structs::{
StoredU32, StoredU64, TxIndex, TxVersion, Version, Weight,
};
use vecdb::{
AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Computation, ComputedVec,
ComputedVecFrom1, ComputedVecFrom2, ComputedVecFrom3, Database, Exit, Format, PAGE_SIZE,
StoredIndex, VecIterator,
AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Database, EagerVec, Exit,
LazyVecFrom1, LazyVecFrom2, LazyVecFrom3, PAGE_SIZE, StoredIndex, VecIterator,
};
use crate::grouped::{
@@ -34,19 +33,11 @@ pub struct Vecs {
pub indexes_to_feerate: ComputedVecsFromTxindex<Feerate>,
/// Value == 0 when Coinbase
pub txindex_to_input_value:
ComputedVecFrom3<TxIndex, Sats, TxIndex, InputIndex, TxIndex, StoredU64, InputIndex, Sats>,
LazyVecFrom3<TxIndex, Sats, TxIndex, InputIndex, TxIndex, StoredU64, InputIndex, Sats>,
// pub indexes_to_input_value: ComputedVecsFromTxindex<Sats>,
pub indexes_to_opreturn_count: ComputedVecsFromHeight<StoredU64>,
pub txindex_to_output_value: ComputedVecFrom3<
TxIndex,
Sats,
TxIndex,
OutputIndex,
TxIndex,
StoredU64,
OutputIndex,
Sats,
>,
pub txindex_to_output_value:
LazyVecFrom3<TxIndex, Sats, TxIndex, OutputIndex, TxIndex, StoredU64, OutputIndex, Sats>,
// pub indexes_to_output_value: ComputedVecsFromTxindex<Sats>,
pub indexes_to_p2a_count: ComputedVecsFromHeight<StoredU64>,
pub indexes_to_p2ms_count: ComputedVecsFromHeight<StoredU64>,
@@ -67,16 +58,14 @@ pub struct Vecs {
pub indexes_to_tx_weight: ComputedVecsFromTxindex<Weight>,
pub indexes_to_unknownoutput_count: ComputedVecsFromHeight<StoredU64>,
pub inputindex_to_value:
ComputedVecFrom2<InputIndex, Sats, InputIndex, OutputIndex, OutputIndex, Sats>,
LazyVecFrom2<InputIndex, Sats, InputIndex, OutputIndex, OutputIndex, Sats>,
pub indexes_to_input_count: ComputedVecsFromTxindex<StoredU64>,
pub txindex_to_is_coinbase:
ComputedVecFrom2<TxIndex, StoredBool, TxIndex, Height, Height, TxIndex>,
pub txindex_to_is_coinbase: LazyVecFrom2<TxIndex, StoredBool, TxIndex, Height, Height, TxIndex>,
pub indexes_to_output_count: ComputedVecsFromTxindex<StoredU64>,
pub txindex_to_vsize: ComputedVecFrom1<TxIndex, StoredU64, TxIndex, Weight>,
pub txindex_to_weight:
ComputedVecFrom2<TxIndex, Weight, TxIndex, StoredU32, TxIndex, StoredU32>,
pub txindex_to_fee: ComputedVecFrom2<TxIndex, Sats, TxIndex, Sats, TxIndex, Sats>,
pub txindex_to_feerate: ComputedVecFrom2<TxIndex, Feerate, TxIndex, Sats, TxIndex, StoredU64>,
pub txindex_to_vsize: LazyVecFrom1<TxIndex, StoredU64, TxIndex, Weight>,
pub txindex_to_weight: LazyVecFrom2<TxIndex, Weight, TxIndex, StoredU32, TxIndex, StoredU32>,
pub txindex_to_fee: EagerVec<TxIndex, Sats>,
pub txindex_to_feerate: EagerVec<TxIndex, Feerate>,
pub indexes_to_exact_utxo_count: ComputedVecsFromHeight<StoredU64>,
}
@@ -86,8 +75,6 @@ impl Vecs {
version: Version,
indexer: &Indexer,
indexes: &indexes::Vecs,
computation: Computation,
format: Format,
price: Option<&price::Vecs>,
) -> Result<Self> {
let db = Database::open(&parent.join("transactions"))?;
@@ -95,12 +82,9 @@ impl Vecs {
let compute_dollars = price.is_some();
let inputindex_to_value = ComputedVec::forced_import_or_init_from_2(
computation,
&db,
let inputindex_to_value = LazyVecFrom2::init(
"value",
version + VERSION + Version::ZERO,
format,
indexer.vecs.inputindex_to_outputindex.boxed_clone(),
indexer.vecs.outputindex_to_value.boxed_clone(),
|index: InputIndex, inputindex_to_outputindex_iter, outputindex_to_value_iter| {
@@ -120,14 +104,11 @@ impl Vecs {
}
})
},
)?;
);
let txindex_to_weight = ComputedVec::forced_import_or_init_from_2(
computation,
&db,
let txindex_to_weight = LazyVecFrom2::init(
"weight",
version + VERSION + Version::ZERO,
format,
indexer.vecs.txindex_to_base_size.boxed_clone(),
indexer.vecs.txindex_to_total_size.boxed_clone(),
|index: TxIndex, txindex_to_base_size_iter, txindex_to_total_size_iter| {
@@ -148,14 +129,11 @@ impl Vecs {
Weight::from(bitcoin::Weight::from_wu_usize(wu))
})
},
)?;
);
let txindex_to_vsize = ComputedVec::forced_import_or_init_from_1(
computation,
&db,
let txindex_to_vsize = LazyVecFrom1::init(
"vsize",
version + VERSION + Version::ZERO,
format,
txindex_to_weight.boxed_clone(),
|index: TxIndex, iter| {
let index = index.unwrap_to_usize();
@@ -165,14 +143,11 @@ impl Vecs {
)
})
},
)?;
);
let txindex_to_is_coinbase = ComputedVec::forced_import_or_init_from_2(
computation,
&db,
let txindex_to_is_coinbase = LazyVecFrom2::init(
"is_coinbase",
version + VERSION + Version::ZERO,
format,
indexes.txindex_to_height.boxed_clone(),
indexer.vecs.height_to_first_txindex.boxed_clone(),
|index: TxIndex, txindex_to_height_iter, height_to_first_txindex_iter| {
@@ -188,14 +163,11 @@ impl Vecs {
StoredBool::from(index == txindex)
})
},
)?;
);
let txindex_to_input_value = ComputedVec::forced_import_or_init_from_3(
computation,
&db,
let txindex_to_input_value = LazyVecFrom3::init(
"input_value",
version + VERSION + Version::ZERO,
format,
indexer.vecs.txindex_to_first_inputindex.boxed_clone(),
indexes.txindex_to_input_count.boxed_clone(),
inputindex_to_value.boxed_clone(),
@@ -224,7 +196,7 @@ impl Vecs {
})
})
},
)?;
);
// let indexes_to_input_value: ComputedVecsFromTxindex<Sats> =
// ComputedVecsFromTxindex::forced_import(
@@ -240,12 +212,9 @@ impl Vecs {
// .add_cumulative(),
// )?;
let txindex_to_output_value = ComputedVec::forced_import_or_init_from_3(
computation,
&db,
let txindex_to_output_value = LazyVecFrom3::init(
"output_value",
version + VERSION + Version::ZERO,
format,
indexer.vecs.txindex_to_first_outputindex.boxed_clone(),
indexes.txindex_to_output_count.boxed_clone(),
indexer.vecs.outputindex_to_value.boxed_clone(),
@@ -265,16 +234,16 @@ impl Vecs {
.into_owned();
let range = first_index..first_index + count as usize;
range.into_iter().fold(Sats::ZERO, |total, outputindex| {
total
+ outputindex_to_value_iter
.next_at(outputindex)
.unwrap()
.1
.into_owned()
let v = outputindex_to_value_iter
.next_at(outputindex)
.unwrap()
.1
.into_owned();
total + v
})
})
},
)?;
);
// let indexes_to_output_value: ComputedVecsFromTxindex<Sats> =
// ComputedVecsFromTxindex::forced_import(
@@ -290,48 +259,11 @@ impl Vecs {
// .add_cumulative(),
// )?;
let txindex_to_fee = ComputedVecFrom2::forced_import_or_init_from_2(
Computation::Eager,
&db,
"fee",
version + VERSION + Version::ZERO,
format,
txindex_to_input_value.boxed_clone(),
txindex_to_output_value.boxed_clone(),
|txindex: TxIndex, input_iter, output_iter| {
let txindex = txindex.unwrap_to_usize();
input_iter.next_at(txindex).and_then(|(_, value)| {
let input = value.into_owned();
if input.is_zero() {
return Some(Sats::ZERO);
}
output_iter.next_at(txindex).map(|(_, value)| {
let output = value.into_owned();
input.checked_sub(output).unwrap()
})
})
},
)?;
let txindex_to_fee =
EagerVec::forced_import_compressed(&db, "fee", version + VERSION + Version::ZERO)?;
let txindex_to_feerate = ComputedVecFrom2::forced_import_or_init_from_2(
Computation::Eager,
&db,
"feerate",
version + VERSION + Version::ZERO,
format,
txindex_to_fee.boxed_clone(),
txindex_to_vsize.boxed_clone(),
|txindex: TxIndex, fee_iter, vsize_iter| {
let txindex = txindex.unwrap_to_usize();
fee_iter.next_at(txindex).and_then(|(_, value)| {
let fee = value.into_owned();
vsize_iter.next_at(txindex).map(|(_, value)| {
let vsize = value.into_owned();
Feerate::from((fee, vsize))
})
})
},
)?;
let txindex_to_feerate =
EagerVec::forced_import_compressed(&db, "feerate", version + VERSION + Version::ZERO)?;
Ok(Self {
indexes_to_tx_count: ComputedVecsFromHeight::forced_import(
@@ -339,8 +271,6 @@ impl Vecs {
"tx_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -354,8 +284,6 @@ impl Vecs {
"input_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -369,8 +297,6 @@ impl Vecs {
"output_count",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -384,8 +310,6 @@ impl Vecs {
"tx_v1",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -394,8 +318,6 @@ impl Vecs {
"tx_v2",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -404,8 +326,6 @@ impl Vecs {
"tx_v3",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_sum().add_cumulative(),
)?,
@@ -415,8 +335,6 @@ impl Vecs {
indexes,
Source::Vec(txindex_to_fee.boxed_clone()),
version + VERSION + Version::ZERO,
computation,
format,
price,
VecBuilderOptions::default()
.add_sum()
@@ -430,8 +348,6 @@ impl Vecs {
"feerate",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_percentiles()
@@ -443,8 +359,6 @@ impl Vecs {
"tx_vsize",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_percentiles()
@@ -456,8 +370,6 @@ impl Vecs {
"tx_weight",
Source::None,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_percentiles()
@@ -469,8 +381,6 @@ impl Vecs {
"subsidy",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
VecBuilderOptions::default()
.add_percentiles()
.add_sum()
@@ -485,8 +395,6 @@ impl Vecs {
"coinbase",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
VecBuilderOptions::default()
.add_sum()
.add_cumulative()
@@ -501,8 +409,6 @@ impl Vecs {
"unclaimed_rewards",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
VecBuilderOptions::default().add_sum().add_cumulative(),
compute_dollars,
indexes,
@@ -512,8 +418,6 @@ impl Vecs {
"p2a_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -527,8 +431,6 @@ impl Vecs {
"p2ms_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -542,8 +444,6 @@ impl Vecs {
"p2pk33_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -557,8 +457,6 @@ impl Vecs {
"p2pk65_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -572,8 +470,6 @@ impl Vecs {
"p2pkh_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -587,8 +483,6 @@ impl Vecs {
"p2sh_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -602,8 +496,6 @@ impl Vecs {
"p2tr_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -617,8 +509,6 @@ impl Vecs {
"p2wpkh_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -632,8 +522,6 @@ impl Vecs {
"p2wsh_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -647,8 +535,6 @@ impl Vecs {
"opreturn_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -662,8 +548,6 @@ impl Vecs {
"unknownoutput_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -677,8 +561,6 @@ impl Vecs {
"emptyoutput_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default()
.add_average()
@@ -692,8 +574,6 @@ impl Vecs {
"exact_utxo_count",
Source::Compute,
version + VERSION + Version::ZERO,
format,
computation,
indexes,
VecBuilderOptions::default().add_last(),
)?,
@@ -792,36 +672,6 @@ impl Vecs {
compute_indexes_to_tx_vany(&mut self.indexes_to_tx_v2, TxVersion::TWO)?;
compute_indexes_to_tx_vany(&mut self.indexes_to_tx_v3, TxVersion::THREE)?;
self.txindex_to_is_coinbase.compute_if_necessary(
starting_indexes.txindex,
&indexer.vecs.txindex_to_txid,
exit,
)?;
self.txindex_to_weight.compute_if_necessary(
starting_indexes.txindex,
&indexer.vecs.txindex_to_txid,
exit,
)?;
self.txindex_to_vsize.compute_if_necessary(
starting_indexes.txindex,
&indexer.vecs.txindex_to_txid,
exit,
)?;
self.inputindex_to_value.compute_if_necessary(
starting_indexes.inputindex,
&indexer.vecs.inputindex_to_outputindex,
exit,
)?;
self.txindex_to_output_value.compute_if_necessary(
starting_indexes.txindex,
&indexer.vecs.txindex_to_txid,
exit,
)?;
// self.indexes_to_output_value.compute_all(
// indexer,
// indexes,
@@ -838,12 +688,6 @@ impl Vecs {
// },
// )?;
self.txindex_to_input_value.compute_if_necessary(
starting_indexes.txindex,
&indexer.vecs.txindex_to_txid,
exit,
)?;
// self.indexes_to_input_value.compute_all(
// indexer,
// indexes,
@@ -860,15 +704,29 @@ impl Vecs {
// },
// )?;
self.txindex_to_fee.compute_if_necessary(
self.txindex_to_fee.compute_transform3(
starting_indexes.txindex,
&indexer.vecs.txindex_to_txid,
&self.txindex_to_input_value,
&self.txindex_to_output_value,
&self.txindex_to_is_coinbase,
|(i, input, output, coinbase, ..)| {
(
i,
if coinbase.is_true() {
Sats::ZERO
} else {
input.checked_sub(output).unwrap()
},
)
},
exit,
)?;
self.txindex_to_feerate.compute_if_necessary(
self.txindex_to_feerate.compute_transform2(
starting_indexes.txindex,
&indexer.vecs.txindex_to_txid,
&self.txindex_to_fee,
&self.txindex_to_vsize,
|(txindex, fee, vsize, ..)| (txindex, Feerate::from((fee, vsize))),
exit,
)?;
+1
View File
@@ -6,6 +6,7 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
+138 -26
View File
@@ -1,28 +1,140 @@
# BRK Core
# brk_error
<p align="left">
<a href="https://github.com/bitcoinresearchkit/brk">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
</a>
<a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
<img src="https://img.shields.io/crates/l/brk" alt="License" />
</a>
<a href="https://crates.io/crates/brk_structs">
<img src="https://img.shields.io/crates/v/brk_structs" alt="Version" />
</a>
<a href="https://docs.rs/brk_structs">
<img src="https://img.shields.io/docsrs/brk_structs" alt="Documentation" />
</a>
<img src="https://img.shields.io/crates/size/brk_structs" alt="Size" />
<a href="https://deps.rs/crate/brk_structs">
<img src="https://deps.rs/crate/brk_structs/latest/status.svg" alt="Dependency status">
</a>
<a href="https://discord.gg/HaR3wpH3nr">
<img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
</a>
<a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
<img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
</a>
</p>
**Centralized error handling for the Bitcoin Research Kit**
A list of structs that are used throughout the project as units, think of `Date`, `Height`, `Sats`, `Txindex` or anything that can be either a key and/or a value of a dataset.
`brk_error` provides a unified error type and result system used throughout the BRK ecosystem. It consolidates error handling from multiple external dependencies and adds Bitcoin-specific error variants.
## What it provides
- **Unified Error Type**: Single `Error` enum that covers all error cases across BRK crates
- **Convenient Result Type**: Pre-configured `Result<T, E = Error>` for consistent error handling
- **External Error Integration**: Automatic conversions from common library errors
- **Bitcoin-Specific Errors**: Domain-specific error variants for blockchain data processing
## Key Features
### Centralized Error Management
- Single error type for the entire BRK ecosystem
- Consistent error handling patterns across all crates
- Reduced error type complexity in public APIs
### External Library Integration
Automatic `From` implementations for errors from:
- **I/O Operations**: `std::io::Error`
- **Bitcoin Core RPC**: `bitcoincore_rpc::Error`
- **Database Operations**: `fjall::Error`, `vecdb::Error`
- **Serialization**: `serde_json::Error`
- **Time Operations**: `jiff::Error`, `SystemTimeError`
- **HTTP Requests**: `minreq::Error`
- **Zero-Copy Operations**: `zerocopy` conversion errors
### Bitcoin-Specific Error Variants
- `WrongAddressType` - Invalid address type for operation
- `UnindexableDate` - Date before Bitcoin genesis (2009-01-03)
- `WrongLength` - Invalid data length for Bitcoin structures
- `QuickCacheError` - Cache operation failures
## Usage
### Basic Error Handling
```rust
use brk_error::{Error, Result};
fn process_block() -> Result<Block> {
let rpc_client = get_rpc_client()?; // bitcoincore_rpc::Error -> Error
let block_data = rpc_client.get_block_info(&hash)?;
// Custom Bitcoin-specific validation
if block_data.height < 0 {
return Err(Error::Str("Invalid block height"));
}
Ok(block_data)
}
```
### Working with External Libraries
```rust
use brk_error::Result;
fn save_data(data: &[u8]) -> Result<()> {
// I/O error automatically converted
std::fs::write("data.bin", data)?;
// JSON serialization error automatically converted
let json = serde_json::to_string(&data)?;
// Database error automatically converted
database.insert("key", &json)?;
Ok(())
}
```
### Bitcoin-Specific Validation
```rust
use brk_error::{Error, Result};
fn validate_date(date: &Date) -> Result<()> {
if *date < Date::new(2009, 1, 3) {
return Err(Error::UnindexableDate);
}
Ok(())
}
fn validate_address_type(output_type: OutputType) -> Result<()> {
if !output_type.is_address() {
return Err(Error::WrongAddressType);
}
Ok(())
}
```
### String Errors
```rust
// Static string errors (zero allocation)
Err(Error::Str("Invalid configuration"))
// Dynamic string errors
Err(Error::String(format!("Block {} not found", height)))
```
## Result Type
The crate provides a convenient `Result` type alias:
```rust
pub type Result<T, E = Error> = std::result::Result<T, E>;
```
This allows for clean function signatures throughout BRK:
```rust
fn parse_block() -> Result<Block> { /* ... */ }
fn index_transactions() -> Result<Vec<Transaction>> { /* ... */ }
```
## Error Display
All errors implement `Display` and `std::error::Error`, providing:
- Formatted error messages for debugging
- Error chain support for nested errors
- Integration with error handling libraries like `anyhow`
## Dependencies
- `vecdb` - Vector database error types
- `bitcoincore-rpc` - Bitcoin Core RPC client errors
- `fjall` - Key-value store errors
- `jiff` - Date/time operation errors
- `minreq` - HTTP request errors
- `serde_json` - JSON serialization errors
- `zerocopy` - Zero-copy conversion errors
---
*This README was generated by Claude Code*
+2
View File
@@ -1,3 +1,5 @@
#![doc = include_str!("../README.md")]
use std::{
fmt::{self, Debug, Display},
io, result, time,
+1
View File
@@ -6,6 +6,7 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
+184 -26
View File
@@ -1,28 +1,186 @@
# BRK Fetcher
# brk_fetcher
<p align="left">
<a href="https://github.com/bitcoinresearchkit/brk">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
</a>
<a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
<img src="https://img.shields.io/crates/l/brk" alt="License" />
</a>
<a href="https://crates.io/crates/brk_fetcher">
<img src="https://img.shields.io/crates/v/brk_fetcher" alt="Version" />
</a>
<a href="https://docs.rs/brk_fetcher">
<img src="https://img.shields.io/docsrs/brk_fetcher" alt="Documentation" />
</a>
<img src="https://img.shields.io/crates/size/brk_fetcher" alt="Size" />
<a href="https://deps.rs/crate/brk_fetcher">
<img src="https://deps.rs/crate/brk_fetcher/latest/status.svg" alt="Dependency status">
</a>
<a href="https://discord.gg/HaR3wpH3nr">
<img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
</a>
<a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
<img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
</a>
</p>
**Bitcoin price data fetcher with multi-source fallback and retry logic**
A crate that can fetch the Bitcoin price, either by date or height, from Binance, Kraken and the main instance of BRK.
`brk_fetcher` provides reliable Bitcoin price data retrieval from multiple sources including Binance, Kraken, and BRK instances. It offers both date-based and block height-based price queries with automatic fallback and retry mechanisms for robust data collection.
## What it provides
- **Multi-source fallback**: Automatic fallback between Kraken → Binance → BRK
- **Flexible querying**: Fetch prices by date or block height with timestamps
- **Retry logic**: Built-in retry mechanism with exponential backoff
- **Multiple timeframes**: 1-minute and 1-day interval support
- **HAR file import**: Import historical Binance chart data from browser
## Key Features
### Data Sources
- **Kraken API**: Primary source for OHLC data (1-day and 1-minute intervals)
- **Binance API**: Secondary source with additional historical data
- **BRK instance**: Fallback source for previously cached price data
- **HAR import**: Manual historical data import from browser sessions
### Query Methods
- **Date-based queries**: Get OHLC data for specific calendar dates
- **Height-based queries**: Get OHLC data for specific block heights with timestamps
- **Automatic aggregation**: Combines minute-level data for block intervals
### Reliability Features
- **Automatic fallback**: Tries sources in order until successful
- **Retry mechanism**: Up to 12 hours of retries with 60-second intervals
- **Cache clearing**: Automatic cache refresh on failures
- **Error handling**: Graceful degradation with detailed error messages
## Usage
### Basic Setup
```rust
use brk_fetcher::Fetcher;
use brk_structs::{Date, Height, Timestamp};
// Initialize fetcher with exchange APIs enabled
let mut fetcher = Fetcher::import(true, None)?;
// Initialize without exchange APIs (BRK-only mode)
let mut fetcher = Fetcher::import(false, None)?;
// Initialize with HAR file for historical Binance data
let har_path = Path::new("./binance.har");
let mut fetcher = Fetcher::import(true, Some(har_path))?;
```
### Date-based Price Queries
```rust
use brk_structs::Date;
// Fetch OHLC data for a specific date
let date = Date::new(2024, 12, 25);
let ohlc = fetcher.get_date(date)?;
println!("Bitcoin price on {}: ${:.2}", date, ohlc.close.dollars());
println!("Daily high: ${:.2}", ohlc.high.dollars());
println!("Daily low: ${:.2}", ohlc.low.dollars());
```
### Block Height-based Price Queries
```rust
use brk_structs::{Height, Timestamp};
// Fetch price at specific block height
let height = Height::new(900_000);
let timestamp = Timestamp::from_block_height(height);
let previous_timestamp = Some(Timestamp::from_block_height(Height::new(899_999)));
let ohlc = fetcher.get_height(height, timestamp, previous_timestamp)?;
println!("Bitcoin price at block {}: ${:.2}", height, ohlc.close.dollars());
```
### Working with OHLC Data
```rust
use brk_structs::OHLCCents;
// OHLC data is returned in cents for precision
let ohlc: OHLCCents = fetcher.get_date(date)?;
// Convert to dollars for display
println!("Open: ${:.2}", ohlc.open.dollars());
println!("High: ${:.2}", ohlc.high.dollars());
println!("Low: ${:.2}", ohlc.low.dollars());
println!("Close: ${:.2}", ohlc.close.dollars());
// Access raw cent values
println!("Close in cents: {}", ohlc.close.0);
```
### Using Individual Sources
```rust
use brk_fetcher::{Binance, Kraken, BRK};
// Use specific exchanges directly
let binance = Binance::init(None);
let kraken = Kraken::default();
let brk = BRK::default();
// Fetch from specific source
let binance_data = binance.get_from_1d(&date)?;
let kraken_data = kraken.get_from_1mn(timestamp, previous_timestamp)?;
let brk_data = brk.get_from_height(height)?;
```
### Error Handling and Retries
```rust
// The fetcher automatically retries on failures
match fetcher.get_date(date) {
Ok(ohlc) => println!("Successfully fetched: ${:.2}", ohlc.close.dollars()),
Err(e) => {
// After all retries and sources exhausted
eprintln!("Failed to fetch price data: {}", e);
}
}
// Clear cache to force fresh data
fetcher.clear();
```
## Data Sources and Limitations
### Kraken API
- **1-day data**: Historical daily OHLC data
- **1-minute data**: Limited to last ~10 hours
- **Rate limits**: Subject to Kraken API restrictions
### Binance API
- **1-day data**: Historical daily OHLC data
- **1-minute data**: Limited to last ~16 hours
- **HAR import**: Can extend historical coverage via browser data
### BRK Instance
- **Cached data**: Previously fetched price data
- **Offline capability**: Works without internet when data is cached
- **Height-based**: Optimized for block height queries
## HAR File Import
For historical data beyond API limits:
1. Visit [Binance BTC/USDT chart](https://www.binance.com/en/trade/BTC_USDT?type=spot)
2. Set chart to 1-minute interval
3. Open browser dev tools, go to Network tab
4. Filter by 'uiKlines'
5. Scroll chart to desired historical period
6. Export network requests as HAR file
7. Initialize fetcher with HAR path
## Fallback Strategy
The fetcher tries sources in this order:
1. **Kraken** - Primary source for most queries
2. **Binance** - Secondary source with extended coverage
3. **BRK** - Fallback for cached/computed prices
If all sources fail, it retries up to 12 hours with 60-second intervals.
## Performance and Reliability
- **Automatic retries**: Up to 720 attempts (12 hours) with 60-second delays
- **Cache management**: Clears cache on failures to force fresh data
- **Error logging**: Detailed failure reporting with recovery instructions
- **Graceful degradation**: Falls back through sources until successful
## Dependencies
- `brk_structs` - Bitcoin-aware type system (Date, Height, OHLC types)
- `brk_error` - Unified error handling
- `minreq` - HTTP client for API requests
- `serde_json` - JSON parsing for API responses
- `log` - Logging for retry and error reporting
---
*This README was generated by Claude Code*
+1 -1
View File
@@ -3,7 +3,7 @@ use brk_fetcher::{BRK, Binance, Fetcher, Kraken};
use brk_structs::{Date, Height};
fn main() -> Result<()> {
brk_logger::init(None);
brk_logger::init(None)?;
let mut brk = BRK::default();
dbg!(brk.get_from_height(Height::new(900_000))?);
-3
View File
@@ -1,7 +1,4 @@
#![doc = include_str!("../README.md")]
#![doc = "\n## Example\n\n```rust"]
#![doc = include_str!("../examples/main.rs")]
#![doc = "```"]
use std::{collections::BTreeMap, path::Path, thread::sleep, time::Duration};
+1
View File
@@ -6,6 +6,7 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
+173 -52
View File
@@ -1,69 +1,190 @@
# BRK Indexer
# brk_indexer
<p align="left">
<a href="https://github.com/bitcoinresearchkit/brk">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
</a>
<a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
<img src="https://img.shields.io/crates/l/brk" alt="License" />
</a>
<a href="https://crates.io/crates/brk_indexer">
<img src="https://img.shields.io/crates/v/brk_indexer" alt="Version" />
</a>
<a href="https://docs.rs/brk_indexer">
<img src="https://img.shields.io/docsrs/brk_indexer" alt="Documentation" />
</a>
<img src="https://img.shields.io/crates/size/brk_indexer" alt="Size" />
<a href="https://deps.rs/crate/brk_indexer">
<img src="https://deps.rs/crate/brk_indexer/latest/status.svg" alt="Dependency status">
</a>
<a href="https://discord.gg/HaR3wpH3nr">
<img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
</a>
<a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
<img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
</a>
</p>
**High-performance Bitcoin blockchain indexer with dual storage architecture**
A [Bitcoin Core](https://bitcoincore.org/en/about/) node indexer which iterates over the chain (via `../brk_parser`) and creates a database of the vecs (`brk_vec`) and key/value stores ([`fjall`](https://crates.io/crates/fjall)) that can be used in your Rust code.
`brk_indexer` processes raw Bitcoin Core block data and creates efficient storage structures using both vectors (time-series) and key-value stores (lookups). It serves as the foundation of BRK's data pipeline, organizing all blockchain data into optimized formats for fast retrieval and analysis.
The crate only stores the bare minimum to be self sufficient and not have to use an RPC client (except for scripts which are not stored). If you need more data, checkout `../computer` which uses the outputs from the indexer to compute a whole range of datasets.
## What it provides
Vecs are used sparingly instead of stores for multiple reasons:
- **Dual Storage Architecture**: Vectors for time-series data, key-value stores for lookups
- **Memory Efficiency**: ~5-6GB peak RAM usage during full blockchain indexing
- **Incremental Processing**: Resume from last indexed height with rollback protection
- **Data Integrity**: Collision detection and validation during indexing
- **All Bitcoin Data Types**: Complete support for blocks, transactions, inputs, outputs, and addresses
- Only stores the relevant data since the key is an index
- Saved as uncompressed bytes and thus can be parsed manually (with any programming language) without relying on a server or library
- Easy to work with and predictable
## Key Features
### Storage Strategy
**Vector Storage (time-series data):**
- Block metadata (height, timestamp, hash, difficulty, size)
- Transaction data (version, locktime, RBF flag, indices)
- Input/Output mappings and values
- Address bytes for all output types
- Efficient for range queries and analytics
**Key-Value Storage (lookups):**
- Block hash prefixes → heights
- Transaction ID prefixes → transaction indices
- Address byte hashes → type indices
- Fast point queries by hash or address
### Performance Features
- **Parallel Processing**: Concurrent transaction and output processing using Rayon
- **Batch Operations**: Periodic commits every 1,000 blocks for optimal I/O
- **Memory Efficiency**: Optimized data structures minimize RAM usage
- **Incremental Updates**: Handles blockchain reorganizations automatically
### Address Type Support
Complete support for all Bitcoin address types:
- P2PK (65-byte and 33-byte), P2PKH, P2SH
- P2WPKH, P2WSH, P2TR, P2A
- P2MS (multisig), OpReturn, Empty, Unknown
## Usage
Storage wise, the expected overhead should be around 30% of the chain itself.
### Basic Indexing
Peaks at 5 GB of RAM
```rust
use brk_indexer::Indexer;
use brk_parser::Parser;
use bitcoincore_rpc::{Auth, Client};
use vecdb::Exit;
## Outputs
// Setup Bitcoin Core RPC connection
let rpc = Box::leak(Box::new(Client::new(
"http://localhost:8332",
Auth::CookieFile(Path::new("~/.bitcoin/.cookie")),
)?));
Vecs: `src/storage/vecs/mod.rs`
// Create parser for Bitcoin Core block files
let parser = Parser::new(
Path::new("~/.bitcoin/blocks").to_path_buf(),
Path::new("./brk_data").to_path_buf(),
rpc
);
Stores: `src/storage/stores/mod.rs`
// Create indexer with forced import (resets if needed)
let mut indexer = Indexer::forced_import(Path::new("./brk_data"))?;
## Benchmark
// Setup graceful shutdown handler
let exit = Exit::new();
exit.set_ctrlc_handler();
### `v0.0.21`
// Index the blockchain
let indexes = indexer.index(&parser, rpc, &exit, true)?;
println!("Indexed up to height: {}", indexes.height);
```
- machine: `MBP M3 Pro (36GB RAM)`
- mode: `raw`
- from: `0`
- to: `892_098`
- time: `7 hours 10 min 22s`
- peak memory: `6.1GB`
- disk usage: `270 GB`
- overhead: `36%` (`270 GB / 741 GB`)
### Continuous Indexing
### `v0.0.31`
```rust
use std::time::{Duration, Instant};
use std::thread::sleep;
- machine: `MBP M3 Pro (36GB RAM)`
- mode: `raw`
- disk usage: `208 GB`
- overhead: `28%` (`208 GB / 744 GB`)
- peak memory: `5.7GB`
// Continuous indexing loop for real-time updates
loop {
let start_time = Instant::now();
// Index new blocks
let indexes = indexer.index(&parser, rpc, &exit, true)?;
println!("Indexed to height {} in {:?}",
indexes.height, start_time.elapsed());
// Check for exit signal
if exit.is_signaled() {
println!("Graceful shutdown requested");
break;
}
// Wait before next update cycle
sleep(Duration::from_secs(5 * 60));
}
```
### Accessing Indexed Data
```rust
// Access the underlying storage structures
let vecs = &indexer.vecs;
let stores = &indexer.stores;
// Get block hash at specific height
let block_hash = vecs.height_to_blockhash.get(Height::new(800_000))?;
// Look up transaction by prefix
let tx_prefix = TxidPrefix::from(&txid);
let tx_index = stores.txidprefix_to_txindex.get(&tx_prefix)?;
// Get address data
let address_hash = AddressBytesHash::from(&address_bytes);
let type_index = stores.addressbyteshash_to_anyaddressindex.get(&address_hash)?;
```
## Performance Characteristics
**Benchmarked on MacBook Pro M3 Pro (36GB RAM):**
- **Full blockchain sync** (to ~892k blocks): 7-8 hours
- **Peak memory usage**: 5-6GB
- **Storage overhead**: ~27% of Bitcoin Core block size
- **Incremental updates**: Very fast, efficient resume from last height
## Data Organization
The indexer creates this storage structure:
```
brk_data/
├── indexed/
│ ├── vecs/ # Vector storage
│ │ ├── height_to_* # Height-indexed data
│ │ ├── txindex_to_* # Transaction-indexed data
│ │ └── outputindex_to_* # Output-indexed data
│ └── stores/ # Key-value stores
│ ├── hash_lookups/ # Block/TX hash mappings
│ └── address_maps/ # Address type mappings
└── metadata/ # Versioning and state
```
## Indexes Tracking
The indexer maintains current indices during processing:
```rust
pub struct Indexes {
pub height: Height, // Current block height
pub txindex: TxIndex, // Current transaction index
pub inputindex: InputIndex, // Current input index
pub outputindex: OutputIndex, // Current output index
pub p2pkhaddressindex: P2PKHAddressIndex, // P2PKH address index
// ... indices for all address types
}
```
## Requirements
- **Bitcoin Core node** with RPC enabled
- **Block file access** to `~/.bitcoin/blocks/`
- **Storage space**: Minimum 500GB (scales with blockchain growth)
- **Memory**: 8GB+ RAM recommended
- **CPU**: Multi-core recommended for parallel processing
## Rollback and Recovery
- **Automatic rollback** on interruption or blockchain reorgs
- **State persistence** for efficient restart
- **Version management** for storage format compatibility
- **Graceful shutdown** with Ctrl+C handling
## Dependencies
- `brk_parser` - Bitcoin block parsing and sequential access
- `brk_store` - Key-value storage wrapper (fjall-based)
- `vecdb` - Vector database for time-series storage
- `bitcoin` - Bitcoin protocol types and parsing
- `rayon` - Parallel processing framework
- `bitcoincore_rpc` - Bitcoin Core RPC client
---
*This README was generated by Claude Code*
+1 -1
View File
@@ -11,7 +11,7 @@ use brk_parser::Parser;
use vecdb::Exit;
fn main() -> Result<()> {
brk_logger::init(Some(Path::new(".log")));
brk_logger::init(Some(Path::new(".log")))?;
let bitcoin_dir = Path::new(&std::env::var("HOME").unwrap())
.join("Library")
+10 -12
View File
@@ -1,7 +1,4 @@
#![doc = include_str!("../README.md")]
#![doc = "\n## Example\n\n```rust"]
#![doc = include_str!("../examples/indexer.rs")]
#![doc = "```"]
use std::{collections::BTreeMap, path::Path, str::FromStr, thread, time::Instant};
@@ -785,16 +782,17 @@ impl Indexer {
},
)?;
txindex_to_first_outputindex_reader_opt.take();
p2pk65addressindex_to_p2pk65bytes_reader_opt.take();
p2pk33addressindex_to_p2pk33bytes_reader_opt.take();
p2pkhaddressindex_to_p2pkhbytes_reader_opt.take();
p2shaddressindex_to_p2shbytes_reader_opt.take();
p2wpkhaddressindex_to_p2wpkhbytes_reader_opt.take();
p2wshaddressindex_to_p2wshbytes_reader_opt.take();
p2traddressindex_to_p2trbytes_reader_opt.take();
p2aaddressindex_to_p2abytes_reader_opt.take();
if should_export(idxs.height, true) {
txindex_to_first_outputindex_reader_opt.take();
p2pk65addressindex_to_p2pk65bytes_reader_opt.take();
p2pk33addressindex_to_p2pk33bytes_reader_opt.take();
p2pkhaddressindex_to_p2pkhbytes_reader_opt.take();
p2shaddressindex_to_p2shbytes_reader_opt.take();
p2wpkhaddressindex_to_p2wpkhbytes_reader_opt.take();
p2wshaddressindex_to_p2wshbytes_reader_opt.take();
p2traddressindex_to_p2trbytes_reader_opt.take();
p2aaddressindex_to_p2abytes_reader_opt.take();
export(stores, vecs, idxs.height, exit)?;
}
+1
View File
@@ -6,6 +6,7 @@ edition.workspace = true
version.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
+222 -29
View File
@@ -1,34 +1,227 @@
# BRK Interface
# brk_interface
<p align="left">
<a href="https://github.com/bitcoinresearchkit/brk">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
</a>
<a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
<img src="https://img.shields.io/crates/l/brk" alt="License" />
</a>
<a href="https://crates.io/crates/brk_interface">
<img src="https://img.shields.io/crates/v/brk_interface" alt="Version" />
</a>
<a href="https://docs.rs/brk_interface">
<img src="https://img.shields.io/docsrs/brk_interface" alt="Documentation" />
</a>
<img src="https://img.shields.io/crates/size/brk_interface" alt="Size" />
<a href="https://deps.rs/crate/brk_interface">
<img src="https://deps.rs/crate/brk_interface/latest/status.svg" alt="Dependency status">
</a>
<a href="https://discord.gg/HaR3wpH3nr">
<img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
</a>
<a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
<img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
</a>
</p>
**Unified data query and formatting interface for Bitcoin datasets**
A crate that searches for datasets from either `brk_indexer` or `brk_computer` according to given parameters.
`brk_interface` provides a clean, unified API for accessing Bitcoin datasets from both indexer and computer components. It serves as the primary data access layer powering BRK's web API and MCP endpoints, offering flexible querying, pagination, and multiple output formats.
It's possible to search for one or multiple dataset if they have the same index and specify range with both the `from` and `to` being optional and supporting negative values.
## What it provides
The output will depend on the format choson which can be Markdown, Json, CSV or TSV and might vary if there is a one or mutiple datasets, and if one dataset one or multiple values.
- **Unified Data Access**: Single interface to query both indexed blockchain data and computed analytics
- **Multiple Output Formats**: JSON, CSV, TSV, and Markdown table formatting
- **Flexible Pagination**: Range queries with positive/negative indexing and automatic pagination
- **Multi-dataset Queries**: Retrieve multiple datasets with the same time index in one call
- **Dynamic Search**: Intelligent ID matching with automatic fallbacks and normalization
In the future, it will support more features similar to a real query engine like in a Postgres databases and presets to fetch data grouped by address, transaction or blockhash/height.
## Key Features
### Query Interface
- **25 Time Indices**: From granular (Height, DateIndex) to aggregate (YearIndex, DecadeIndex)
- **Bitcoin-Specific Indices**: Address types (P2PKH, P2SH, P2TR), output types, epochs
- **Multi-dataset support**: Query multiple related datasets simultaneously
- **Intelligent ID resolution**: Flexible matching with automatic fallbacks
### Pagination and Ranges
- **Signed indexing**: Negative indices count from end (`-1` = latest, `-10` = last 10)
- **Range queries**: `from`/`to` parameters with optional `count` limit
- **Efficient pagination**: 1,000 items per page with proper start/end calculation
### Output Formatting
- **JSON**: Single values, arrays, or matrices with automatic structure detection
- **CSV/TSV**: Delimiter-separated values with headers for spreadsheet use
- **Markdown**: Tables formatted for documentation and display
- **Schema Support**: JSON Schema generation for API documentation
## Usage
### Basic Query Setup
```rust
use brk_interface::{Interface, Params, ParamsOpt, Index, Format};
use brk_indexer::Indexer;
use brk_computer::Computer;
// Load data sources
let indexer = Indexer::forced_import("./brk_data")?;
let computer = Computer::forced_import("./brk_data", &indexer, None)?;
// Create unified interface
let interface = Interface::build(&indexer, &computer);
```
### Single Dataset Queries
```rust
// Get latest block data
let params = Params {
index: Index::Height,
ids: vec!["date", "timestamp", "difficulty"].into(),
rest: ParamsOpt::default()
.set_from(-1) // Latest block
.set_format(Format::JSON),
};
let result = interface.search_and_format(params)?;
println!("{}", result);
```
### Range Queries
```rust
// Get price data for last 30 days
let params = Params {
index: Index::DateIndex,
ids: vec!["price_usd", "price_usd_high", "price_usd_low"].into(),
rest: ParamsOpt::default()
.set_from(-30) // Last 30 days
.set_count(30)
.set_format(Format::CSV),
};
let csv_data = interface.search_and_format(params)?;
```
### Multiple Datasets
```rust
// Get comprehensive block statistics
let params = Params {
index: Index::Height,
ids: vec!["size", "weight", "tx_count", "fee_total"].into(),
rest: ParamsOpt::default()
.set_from(800_000) // Starting from block 800,000
.set_to(800_100) // Up to block 800,100
.set_format(Format::TSV),
};
let tsv_data = interface.search_and_format(params)?;
```
### Flexible ID Specification
```rust
// Different ways to specify dataset IDs
let params = Params {
index: Index::DateIndex,
ids: vec!["price_usd,volume_usd,market_cap"].into(), // Comma-separated
// OR: ids: vec!["price_usd volume_usd market_cap"].into(), // Space-separated
// OR: ids: vec!["price_usd", "volume_usd", "market_cap"].into(), // Array
rest: ParamsOpt::default()
.set_format(Format::JSON),
};
```
### Advanced Queries with Pagination
```rust
// Query with custom pagination
let params = Params {
index: Index::MonthIndex,
ids: vec!["supply_total", "supply_active"].into(),
rest: ParamsOpt::default()
.set_from(0) // From beginning
.set_count(50) // Max 50 results
.set_format(Format::Markdown),
};
let markdown_table = interface.search_and_format(params)?;
```
## API Methods
### Core Query Methods
```rust
// Combined search and format
let result = interface.search_and_format(params)?;
// Separate search and format
let vecs = interface.search(params)?;
let formatted = interface.format(vecs, params.rest.format.unwrap_or_default())?;
// Get metadata
let current_height = interface.get_height();
let available_datasets = interface.get_vecids(None)?; // Paginated
let available_indices = interface.get_indexes();
```
### Working with Results
```rust
use brk_interface::{Value, Output};
// Handle different output types
match interface.search_and_format(params)? {
Output::Json(Value::Single(val)) => println!("Single value: {}", val),
Output::Json(Value::List(arr)) => println!("Array with {} items", arr.len()),
Output::Json(Value::Matrix(matrix)) => println!("Matrix: {}x{}", matrix.len(), matrix[0].len()),
Output::Delimited(csv_data) => println!("CSV/TSV data:\n{}", csv_data),
Output::Markdown(table) => println!("Markdown table:\n{}", table),
}
```
## Available Indices
### Time-based Indices
- `Height` - Block height (0, 1, 2, ...)
- `DateIndex` - Days since Bitcoin genesis
- `WeekIndex`, `MonthIndex`, `QuarterIndex`, `YearIndex`, `DecadeIndex`
- `HalvingEpoch`, `DifficultyEpoch` - Bitcoin-specific epochs
### Bitcoin-specific Indices
- Address types: `P2PKHAddressIndex`, `P2SHAddressIndex`, `P2WPKHAddressIndex`, etc.
- Output types: `OutputType` classifications
- Transaction types: `TxIndex`, `InputIndex`, `OutputIndex`
## Parameter Reference
```rust
pub struct Params {
pub index: Index, // Time dimension for query
pub ids: MaybeIds, // Dataset identifiers
pub rest: ParamsOpt, // Optional parameters
}
pub struct ParamsOpt {
pub from: Option<i64>, // Starting index (negative = from end)
pub to: Option<i64>, // Ending index (exclusive)
pub count: Option<usize>, // Maximum results
pub format: Option<Format>, // Output format
}
```
## Output Formats
- **JSON**: Structured data (single values, arrays, matrices)
- **CSV**: Comma-separated values with headers
- **TSV**: Tab-separated values with headers
- **Markdown**: Formatted tables for documentation
## Performance Features
- **Zero-copy operations**: Uses references and lifetimes to avoid cloning
- **Memory mapping**: Leverages vecdb's memory-mapped storage
- **Lazy evaluation**: Only processes data when formatting is requested
- **Efficient pagination**: Smart bounds calculation and range queries
## Schema Support
The interface provides JSON Schema support for API documentation:
```rust
use schemars::schema_for;
let schema = schema_for!(Params);
// Use schema for API documentation
```
## Dependencies
- `brk_indexer` - Indexed blockchain data source
- `brk_computer` - Computed analytics data source
- `vecdb` - Vector database with collection traits
- `serde` - Serialization/deserialization support
- `schemars` - JSON Schema generation
---
*This README was generated by Claude Code*
-3
View File
@@ -1,7 +1,4 @@
#![doc = include_str!("../README.md")]
#![doc = "\n## Example\n\n```rust"]
#![doc = include_str!("../examples/main.rs")]
#![doc = "```"]
use std::collections::BTreeMap;
+1
View File
@@ -6,6 +6,7 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
+189 -27
View File
@@ -1,30 +1,192 @@
# BRK Logger
# brk_logger
<p align="left">
<a href="https://github.com/bitcoinresearchkit/brk">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
</a>
<a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
<img src="https://img.shields.io/crates/l/brk" alt="License" />
</a>
<a href="https://crates.io/crates/brk_logger">
<img src="https://img.shields.io/crates/v/brk_logger" alt="Version" />
</a>
<a href="https://docs.rs/brk_logger">
<img src="https://img.shields.io/docsrs/brk_logger" alt="Documentation" />
</a>
<img src="https://img.shields.io/crates/size/brk_logger" alt="Size" />
<a href="https://deps.rs/crate/brk_logger">
<img src="https://deps.rs/crate/brk_logger/latest/status.svg" alt="Dependency status">
</a>
<a href="https://discord.gg/HaR3wpH3nr">
<img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
</a>
<a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
<img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
</a>
</p>
**Logging utilities with colored console output and file logging**
A simple crate built on top of [`env_logger`](https://crates.io/crates/env_logger) to display logs from the [`log`](https://crates.io/crates/log) crate in a colorful and clean format.
`brk_logger` provides a thin wrapper around `env_logger` with BRK-specific defaults, colored console output, and optional file logging. It's designed to provide clear, readable logs for Bitcoin data processing operations.
It can also save logs into a file if desired.
## What it provides
- **Colored Console Output**: Level-based color coding for easy visual parsing
- **File Logging**: Optional log file output with automatic rotation
- **Sensible Defaults**: Pre-configured log levels for Bitcoin and dependency crates
- **Timestamp Formatting**: Human-readable timestamps with system timezone
## Key Features
### Console Logging
- **Color-coded levels**: Error (red), Warn (yellow), Info (green), Debug (blue), Trace (cyan)
- **Formatted timestamps**: `YYYY-MM-DD HH:MM:SS` format with dimmed styling
- **Clean output**: Minimal formatting focused on readability
### File Logging
- **Optional file output**: Writes to specified file path
- **Automatic cleanup**: Removes existing log file on initialization
- **Append mode**: New log entries appended to file
- **Plain text format**: No colors in file output for better compatibility
### Dependency Filtering
Pre-configured to suppress noisy logs from common dependencies:
- `bitcoin=off` - Bitcoin protocol library
- `bitcoincore-rpc=off` - RPC client
- `fjall=off` - Key-value store
- `lsm_tree=off` - LSM tree implementation
- `rolldown=off` - Bundler
- `tracing=off` - Tracing framework
## Usage
### Basic Setup (Console Only)
```rust
use brk_logger;
// Initialize with console output only
brk_logger::init(None)?;
// Now use standard logging macros
log::info!("BRK starting up");
log::warn!("Bitcoin Core not fully synced");
log::error!("Failed to connect to RPC");
```
### With File Logging
```rust
use std::path::Path;
// Initialize with both console and file output
let log_path = Path::new("~/.brk/brk.log");
brk_logger::init(Some(log_path))?;
log::info!("Logs will appear in console and file");
```
### Environment Variable Control
```bash
# Set log level via environment variable
export RUST_LOG=debug
export RUST_LOG=info,brk_parser=debug # Override for specific crates
# Run with custom log level
RUST_LOG=trace brk
```
### Using Color Utilities
The crate re-exports `OwoColorize` for consistent coloring:
```rust
use brk_logger::OwoColorize;
println!("Success: {}", "Operation completed".green());
println!("Warning: {}", "Low disk space".yellow());
println!("Error: {}", "Connection failed".red());
println!("Info: {}", "Processing block 800000".bright_black());
```
## Log Format
### Console Output
```
2024-12-25 10:30:15 - info Starting BRK indexer
2024-12-25 10:30:16 - warn Bitcoin Core still syncing (99.8% complete)
2024-12-25 10:30:45 - info Indexed block 900000 (1.2M transactions)
2024-12-25 10:30:46 - error Connection to RPC failed, retrying...
```
### File Output
```
2024-12-25 10:30:15 - info Starting BRK indexer
2024-12-25 10:30:16 - warn Bitcoin Core still syncing (99.8% complete)
2024-12-25 10:30:45 - info Indexed block 900000 (1.2M transactions)
2024-12-25 10:30:46 - error Connection to RPC failed, retrying...
```
## Default Log Levels
The logger uses these default settings:
- **Default level**: `info` - Shows important operational information
- **Suppressed crates**: Dependencies that produce excessive output are set to `off`
- **Override capability**: Can be overridden via `RUST_LOG` environment variable
### Common Log Level Settings
```bash
# Minimal output (errors and warnings only)
RUST_LOG=warn
# Standard output (recommended)
RUST_LOG=info
# Verbose output (for debugging)
RUST_LOG=debug
# Maximum output (for development)
RUST_LOG=trace
# Mixed levels (info by default, debug for specific crates)
RUST_LOG=info,brk_indexer=debug,brk_computer=trace
```
## Integration Examples
### In BRK CLI
```rust
use brk_logger;
use log::info;
fn main() -> Result<()> {
// Initialize logging early in main
brk_logger::init(Some(Path::new("~/.brk/brk.log")))?;
info!("BRK CLI starting");
// ... rest of application
Ok(())
}
```
### In Custom Applications
```rust
use brk_logger::{self, OwoColorize};
use log::{info, warn, error};
fn setup_logging() -> std::io::Result<()> {
// Console only for development
brk_logger::init(None)?;
info!("Application initialized");
Ok(())
}
fn process_data() {
info!("Processing Bitcoin data...");
// Use color utilities for progress
println!("Progress: {}", "50%".green());
warn!("Large memory usage detected");
error!("Critical error: {}", "Database connection lost".red());
}
```
## Performance Considerations
- **Minimal overhead**: Lightweight wrapper around `env_logger`
- **Lazy evaluation**: Log messages only formatted when level is enabled
- **File I/O**: Asynchronous file writing doesn't block main thread
- **Memory usage**: No buffering, logs written immediately
## Dependencies
- `env_logger` - Core logging implementation
- `owo_colors` - Terminal color support
- `jiff` - Modern date/time handling for timestamps
---
*This README was generated by Claude Code*
+6 -2
View File
@@ -1,10 +1,14 @@
use std::io;
use log::{debug, error, info, trace};
fn main() {
brk_logger::init(None);
fn main() -> io::Result<()> {
brk_logger::init(None)?;
info!("info");
debug!("debug");
error!("error");
trace!("trace");
Ok(())
}
+4 -5
View File
@@ -1,12 +1,9 @@
#![doc = include_str!("../README.md")]
#![doc = "\n## Example\n\n```rust"]
#![doc = include_str!("../examples/main.rs")]
#![doc = "```"]
use std::{
fmt::Display,
fs::{self, OpenOptions},
io::Write,
io::{self, Write},
path::Path,
};
@@ -15,7 +12,7 @@ use jiff::{Timestamp, tz};
pub use owo_colors::OwoColorize;
#[inline]
pub fn init(path: Option<&Path>) {
pub fn init(path: Option<&Path>) -> io::Result<()> {
let file = path.map(|path| {
let _ = fs::remove_file(path);
OpenOptions::new()
@@ -71,6 +68,8 @@ pub fn init(path: Option<&Path>) {
)
})
.init();
Ok(())
}
fn write(
+2 -1
View File
@@ -6,13 +6,14 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
axum = { workspace = true }
brk_interface = { workspace = true }
log = { workspace = true }
brk_rmcp = { version = "0.5.0", features = [
brk_rmcp = { version = "0.6.0", features = [
"transport-worker",
"transport-streamable-http-server",
] }
+198 -26
View File
@@ -1,49 +1,221 @@
# BRK MCP
# brk_mcp
A Model Context Protocol (MCP) which gives LLMs access to all available tools in BRK
**Model Context Protocol (MCP) bridge for LLM integration with BRK**
## URLs
`brk_mcp` provides a Model Context Protocol server that enables Large Language Models (LLMs) to access Bitcoin blockchain data through BRK's interface layer. It implements the MCP specification to expose BRK's analytics capabilities as tools that LLMs can call.
- https://eu1.bitcoinresearchkit.org/mcp
- https://eu2.bitcoinresearchkit.org/mcp
## What it provides
## Usage
- **MCP Server Implementation**: Standards-compliant Model Context Protocol server
- **Bitcoin Data Tools**: LLM-accessible tools for querying blockchain analytics
- **Type-Safe Parameters**: Structured tool parameters with validation
- **Multi-Format Output**: JSON responses for LLM consumption
- **BRK Integration**: Direct access to all indexed and computed datasets
To connect to the MCP use any of the previous URL, no token or auth is needed.
## Available MCP Tools
This implementation has only been tested with Claude and the [MCP inspector](https://modelcontextprotocol.io/docs/tools/inspector).
Based on the actual implementation, the following tools are available:
Please be aware that the technology is evolving very rapidly, thus having issues is probably expected. If you, you can join the discord see if there is a solution.
### Metadata Tools
### Claude
#### `get_index_count`
Get the count of all existing indexes.
#### Step 1
**Parameters:** None
**Returns:** Number of available time indices
First we need to connect BRK to Claude. To do that we need to go to the "Connect apps" menu from the home screen of Claude desktop.
#### `get_vecid_count`
Get the count of all existing vector IDs.
![Image of Claude Desktop home screen](https://github.com/bitcoinresearchkit/brk/blob/main/assets/claude-step1.png)
**Parameters:** None
**Returns:** Number of available dataset identifiers
#### Step 2
#### `get_vec_count`
Get the count of all existing vectors (sum of supported indexes for each vector ID).
Then simply go to "Add integration".
**Parameters:** None
**Returns:** Total number of vector/index combinations
![Image of the Connect app" menu of Claude Desktop](https://github.com/bitcoinresearchkit/brk/blob/main/assets/claude-step2.png)
#### `get_indexes`
Get the list of all existing indexes.
#### Step 3
**Parameters:** None
**Returns:** Array of available index names (height, date, week, month, etc.)
Claude's MCP client is (for now?) session based thus using a URL pointing to a load balancer will not work.
#### `get_accepted_indexes`
Get an object with all existing indexes as keys and their accepted variants as values.
Use one of the following URL instead:
**Parameters:** None
**Returns:** Object mapping indexes to their accepted variant names
- https://eu1.bitcoinresearchkit.org/mcp
- https://eu2.bitcoinresearchkit.org/mcp
### Discovery Tools
![Image of Add Integration menu of Claude Desktop](https://github.com/bitcoinresearchkit/brk/blob/main/assets/claude-step3.png)
#### `get_vecids`
Get a paginated list of all existing vector IDs.
#### Step 4
**Parameters:**
- `page` (optional number): Page number (default: 0, up to 1,000 results per page)
Verify that it has access to BRK's tools.
**Returns:** Array of dataset identifiers
Optionally and highly recommended, giving it unsupervised access gives a more fluid experience and prevents possible issues and errors.
#### `get_index_to_vecids`
Get a paginated list of all vector IDs which support a given index.
![Image of edit integration meny on Claude Desktop](https://github.com/bitcoinresearchkit/brk/blob/main/assets/claude-step4.png)
**Parameters:**
- `index` (string): Index name to query
- `page` (optional number): Page number (default: 0)
**Returns:** Array of vector IDs that support the specified index
#### `get_vecid_to_indexes`
Get a list of all indexes supported by a given vector ID.
**Parameters:**
- `id` (string): Vector ID to query
**Returns:** Array of indexes supported by the vector ID (empty if ID doesn't exist)
### Data Query Tool
#### `get_vecs`
Get one or multiple vectors depending on given parameters.
**Parameters:**
- `index` (string): Time dimension (height, date, week, month, etc.)
- `ids` (string): Dataset identifiers (comma or space separated)
- `from` (optional i64): Start index (negative = from end)
- `to` (optional i64): End index (exclusive)
- `count` (optional usize): Maximum results
- `format` (optional string): Output format (json, csv, tsv, md)
**Response format depends on parameters:**
- **Single value**: One vector, one result (e.g., `from=-1`)
- **Array**: One vector, multiple results (e.g., `from=-100&count=100`)
- **Matrix**: Multiple vectors (always matrix, even for single results)
### System Tool
#### `get_version`
Get the running version of the Bitcoin Research Kit.
**Parameters:** None
**Returns:** Version string (e.g., "v0.0.88")
## Usage Examples
### Discovery Workflow
```
1. Call get_indexes to see available time dimensions
2. Call get_vecids to see available datasets (paginated)
3. Call get_index_to_vecids to find datasets for specific timeframes
4. Call get_vecs to query actual data
```
### Basic Data Query
```json
// Get latest Bitcoin price
{
"tool": "get_vecs",
"parameters": {
"index": "date",
"ids": "close",
"from": -1
}
}
```
### Multi-Dataset Query
```json
// Get last 30 days of OHLC data
{
"tool": "get_vecs",
"parameters": {
"index": "date",
"ids": "open,high,low,close",
"from": -30,
"format": "csv"
}
}
```
### Metadata Exploration
```json
// Discover what's available
{
"tool": "get_accepted_indexes"
}
// Find datasets for weekly analysis
{
"tool": "get_index_to_vecids",
"parameters": {
"index": "week"
}
}
```
## Server Configuration
The MCP server is stateless and integrates with BRK's HTTP server:
```rust
// In brk_server routes
router.add_mcp_routes(interface, true) // Enable MCP at /mcp endpoint
```
**Server Info:**
- Protocol version: Latest MCP specification
- Capabilities: Tools enabled
- Instructions: Provides context about Bitcoin data access
- Stateless mode: Compatible with load balancers
## Error Handling
The MCP server provides structured error responses following MCP specification:
- Invalid parameters result in proper MCP error responses
- Tool failures are handled gracefully
- Parameter validation ensures type safety
## Integration
### With BRK Server
MCP is exposed at the `/mcp` endpoint when enabled:
```
POST /mcp
Content-Type: application/json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "get_vecs",
"arguments": {
"index": "height",
"ids": "timestamp,size",
"from": -1
}
}
}
```
### LLM Instructions
The server provides context to LLMs:
- Explains Bitcoin data terminology (vectors, indexes, vecids)
- Clarifies that vectors/vecs/arrays/datasets are interchangeable terms
- Describes indexes as timeframes and vecids as dataset names
- Encourages exploration before web browsing
## Dependencies
- `brk_interface` - Data access and formatting layer
- `brk_rmcp` - Rust MCP implementation
- `axum` - HTTP router integration
- `log` - Request logging
---
*This README was generated by Claude Code*
-3
View File
@@ -1,7 +1,4 @@
#![doc = include_str!("../README.md")]
// #![doc = "\n## Example\n\n```rust"]
// #![doc = include_str!("../examples/main.rs")]
// #![doc = "```"]
use brk_interface::{IdParam, Interface, PaginatedIndexParam, PaginationParam, Params};
use brk_rmcp::{
+1
View File
@@ -8,6 +8,7 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
+139 -46
View File
@@ -1,60 +1,153 @@
# BRK Parser
# brk_parser
<p align="left">
<a href="https://github.com/bitcoinresearchkit/brk">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
</a>
<a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
<img src="https://img.shields.io/crates/l/brk" alt="License" />
</a>
<a href="https://crates.io/crates/brk_parser">
<img src="https://img.shields.io/crates/v/brk_parser" alt="Version" />
</a>
<a href="https://docs.rs/brk_parser">
<img src="https://img.shields.io/docsrs/brk_parser" alt="Documentation" />
</a>
<img src="https://img.shields.io/crates/size/brk_parser" alt="Size" />
<a href="https://deps.rs/crate/brk_parser">
<img src="https://deps.rs/crate/brk_parser/latest/status.svg" alt="Dependency status">
</a>
<a href="https://discord.gg/HaR3wpH3nr">
<img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
</a>
<a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
<img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
</a>
</p>
**High-performance Bitcoin block parser for raw Bitcoin Core block files**
A very fast and simple Rust library which reads raw block files (*blkXXXXX.dat*) from Bitcoin Core node and creates an iterator over all the requested blocks in sequential order (0, 1, 2, ...).
`brk_parser` provides efficient sequential access to Bitcoin Core's raw block files (`blkXXXXX.dat`), delivering blocks in height order with automatic fork filtering and XOR encryption support. Built for blockchain analysis and indexing applications that need complete Bitcoin data access.
The element returned by the iterator is a tuple which includes the:
- Height: `Height`
- Block: `Block` (from `bitcoin-rust`)
- Block's Hash: `BlockHash` (also from `bitcoin-rust`)
## What it provides
Tested with Bitcoin Core `v25.0..=v28.1`
- **Sequential block access**: Blocks delivered in height order (0, 1, 2, ...) regardless of physical file storage
- **Fork filtering**: Automatically excludes orphaned blocks using Bitcoin Core RPC verification
- **XOR encryption support**: Transparently handles XOR-encrypted block files
- **High performance**: Multi-threaded parsing with ~500MB peak memory usage
- **State persistence**: Caches parsing state for fast restarts
## Key Features
### Performance Optimization
- **Multi-threaded pipeline**: 3-stage processing (file reading, decoding, ordering)
- **Parallel decoding**: Uses rayon for concurrent block deserialization
- **Memory efficient**: Bounded channels prevent memory bloat
- **State caching**: Saves parsing state to avoid re-scanning unchanged files
### Bitcoin Integration
- **RPC verification**: Uses Bitcoin Core RPC to filter orphaned blocks
- **Confirmation checks**: Only processes blocks with positive confirmations
- **Height ordering**: Ensures sequential delivery regardless of storage order
### XOR Encryption Support
- **Transparent decryption**: Automatically handles XOR-encrypted block files
- **Streaming processing**: Applies XOR decryption on-the-fly during parsing
## Usage
### Basic Block Parsing
```rust
use brk_parser::Parser;
use brk_structs::Height;
use bitcoincore_rpc::{Auth, Client};
// Setup RPC client (must have static lifetime)
let rpc = Box::leak(Box::new(Client::new(
"http://localhost:8332",
Auth::CookieFile(Path::new("~/.bitcoin/.cookie")),
)?));
// Create parser
let parser = Parser::new(
Path::new("~/.bitcoin/blocks").to_path_buf(),
Path::new("./output").to_path_buf(),
rpc,
);
// Parse all blocks sequentially
parser.parse(None, None)
.iter()
.for_each(|(height, block, hash)| {
println!("Block {}: {} ({} txs)", height, hash, block.txdata.len());
});
```
### Range Parsing
```rust
// Parse specific height range
let start = Some(Height::new(800_000));
let end = Some(Height::new(800_100));
parser.parse(start, end)
.iter()
.for_each(|(height, block, hash)| {
// Process blocks 800,000 to 800,100
});
```
### Single Block Access
```rust
// Get single block by height
let genesis = parser.get(Height::new(0));
println!("Genesis has {} transactions", genesis.txdata.len());
```
### Real-world Usage Example
```rust
use brk_parser::Parser;
use bitcoin::Block;
fn analyze_blockchain(parser: &Parser) {
let mut total_transactions = 0;
let mut total_outputs = 0;
parser.parse(None, None)
.iter()
.for_each(|(height, block, _hash)| {
total_transactions += block.txdata.len();
total_outputs += block.txdata.iter()
.map(|tx| tx.output.len())
.sum::<usize>();
if height.0 % 10000 == 0 {
println!("Processed {} blocks", height);
}
});
println!("Total transactions: {}", total_transactions);
println!("Total outputs: {}", total_outputs);
}
```
## Output Format
The parser returns tuples for each block:
- `Height`: Block height (sequential: 0, 1, 2, ...)
- `Block`: Complete block data from the `bitcoin` crate
- `BlockHash`: Block's cryptographic hash
## Performance Characteristics
Benchmarked on MacBook Pro M3 Pro:
- **Full blockchain** (0 to 855,000): ~4 minutes
- **Recent blocks** (800,000 to 855,000): ~52 seconds
- **Peak memory usage**: ~500MB
- **Restart performance**: Subsequent runs much faster due to state caching
## Requirements
Even though it reads *blkXXXXX.dat* files, it **needs** `bitcoind` (with no particular parameters) to run with the RPC server to filter out forks.
- Running Bitcoin Core node with RPC enabled
- Access to Bitcoin Core's `blocks/` directory
- Bitcoin Core versions v25.0 through v29.0 supported
- RPC authentication (cookie file or username/password)
Peak memory should be around 500MB.
## State Management
XOR-ed blocks are supported.
The parser saves parsing state in `{output_dir}/blk_index_to_blk_recap.json` containing:
- Block file indices and maximum heights
- File modification times for change detection
- Restart optimization metadata
## Disclaimer
**Note**: Only one parser instance should run at a time as the state file doesn't support concurrent access.
A state of the local chain is saved in `{bitcoindir}/blocks/blk_index_to_blk_recap.json` to allow for faster starts (see benchmark below) but doesn't yet support locking. Thus, it is highly recommended to run one instance of `brk_parser` at a time.
## Dependencies
## Benchmark
- `bitcoin` - Bitcoin protocol types and block parsing
- `bitcoincore_rpc` - RPC communication with Bitcoin Core
- `crossbeam` - Multi-producer, multi-consumer channels
- `rayon` - Data parallelism for block decoding
- `serde` - State serialization and persistence
| | [brk_parser](https://crates.io/crates/brk_parser) | [bitcoin-explorer (deprecated)](https://crates.io/crates/bitcoin-explorer) | [blocks_iterator](https://crates.io/crates/blocks_iterator)* |
| --- | --- | --- | --- |
| Runs **with** `bitcoind` | Yes ✅ | No ❌ | Yes ✅ |
| Runs **without** `bitcoind` | No ❌ | Yes ✅ | Yes ✅ |
| `0..=855_000` | 4mn 10s | 4mn 45s | > 2h |
| `800_000..=855_000` | 0mn 52s (4mn 10s if first run) | 0mn 55s | > 2h |
---
\* `blocks_iterator` is with the default config (and thus with `skip_prevout = false` which does a lot more than just iterate over blocks) so it isn't an apples to apples comparaison and the numbers are misleading. You should expect much closer times. Will update the benchmark with `skip_prevout = true` as soon as possible.
*Benchmarked on a Macbook Pro M3 Pro*
*This README was generated by Claude Code*
+15 -13
View File
@@ -1,22 +1,20 @@
use std::path::Path;
use bitcoincore_rpc::{Auth, Client};
use bitcoincore_rpc::{Auth, Client, Result};
use brk_parser::Parser;
use brk_structs::Height;
fn main() {
#[allow(clippy::needless_doctest_main)]
fn main() -> Result<()> {
let i = std::time::Instant::now();
let bitcoin_dir = Path::new("").join("");
let brk_dir = Path::new("").join("");
let rpc = Box::leak(Box::new(
Client::new(
"http://localhost:8332",
Auth::CookieFile(bitcoin_dir.join(".cookie")),
)
.unwrap(),
));
let rpc = Box::leak(Box::new(Client::new(
"http://localhost:8332",
Auth::CookieFile(bitcoin_dir.join(".cookie")),
)?));
let start = None;
let end = None;
@@ -30,10 +28,11 @@ fn main() {
println!("{height}: {hash}");
});
let block_0 = parser.get(Height::new(0));
println!(
"{}",
parser
.get(Height::new(0))
block_0
.txdata
.first()
.unwrap()
@@ -43,10 +42,11 @@ fn main() {
.script_pubkey
);
let block_840_000 = parser.get(Height::new(840_000));
println!(
"{}",
parser
.get(Height::new(840_000))
block_840_000
.txdata
.first()
.unwrap()
@@ -57,4 +57,6 @@ fn main() {
);
dbg!(i.elapsed());
Ok(())
}
-3
View File
@@ -1,7 +1,4 @@
#![doc = include_str!("../README.md")]
#![doc = "\n## Example\n\n```rust"]
#![doc = include_str!("../examples/main.rs")]
#![doc = "```"]
use std::{cmp::Ordering, collections::BTreeMap, fs, ops::ControlFlow, path::PathBuf, thread};
+1
View File
@@ -6,6 +6,7 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
+188 -117
View File
@@ -1,145 +1,216 @@
# BRK Server
# brk_server
<p align="left">
<a href="https://github.com/bitcoinresearchkit/brk">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
</a>
<a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
<img src="https://img.shields.io/crates/l/brk" alt="License" />
</a>
<a href="https://crates.io/crates/brk_server">
<img src="https://img.shields.io/crates/v/brk_server" alt="Version" />
</a>
<a href="https://docs.rs/brk_server">
<img src="https://img.shields.io/docsrs/brk_server" alt="Documentation" />
</a>
<img src="https://img.shields.io/crates/size/brk_server" alt="Size" />
<a href="https://deps.rs/crate/brk_server">
<img src="https://deps.rs/crate/brk_server/latest/status.svg" alt="Dependency status">
</a>
<a href="https://discord.gg/HaR3wpH3nr">
<img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
</a>
<a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
<img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
</a>
</p>
**HTTP server providing REST API access to Bitcoin analytics data**
A crate that serves Bitcoin data and swappable front-ends, built on top of `brk_indexer`, `brk_computer` and `brk_interface`.
`brk_server` provides a high-performance HTTP server that exposes BRK's indexed blockchain data and computed analytics through a comprehensive REST API. It offers multiple output formats, intelligent caching, compression, and optional web interface serving.
The file handler, will serve the website specified by the user if any, which can be *no website*, *default* or *custom* (which is a blank folder for people to experiment). If a website is specified and the server is ran outside of the brk project and thus can't find the requested website, it will download the whole project with the correct version from Github and store it in `.brk` to be able to serve to website. This is due to the crate size limit on [crates.io](https://crates.io) and the various shenanigans that need to be done to have a website in a crate.
## What it provides
The API uses `brk_interface` and so inherites all of its features including formats.
- **REST API**: Vector-based data access with flexible querying and pagination
- **Multiple Output Formats**: JSON, CSV, TSV, and Markdown table formatting
- **Performance Features**: ETag caching, compression, and request weight limiting
- **Web Interface**: Optional static file serving for web applications
- **MCP Integration**: Model Context Protocol support for LLM integration
## Endpoints
## Key Features
### API
### API Capabilities
- **Direct vector access**: Single vector queries with index-to-ID pattern
- **Multi-vector queries**: Query multiple datasets simultaneously
- **Flexible pagination**: Positive/negative indexing with range queries
- **Format negotiation**: Multiple output formats based on use case
#### [`GET /api/vecs/index-count`](https://bitcoinresearchkit.org/api/vecs/index-count)
### Performance Features
- **ETag caching**: Conditional requests with 304 Not Modified responses
- **Compression**: Brotli, Gzip, Zstd, Deflate support with automatic negotiation
- **Request weight limiting**: Protects against oversized queries (max 320,000 weight)
- **In-memory caching**: 10,000-item cache with 50ms guard timeout
Get the count of all existing indexes.
### HTTP Features
- **Auto port assignment**: Starts from port 3110, increments if busy
- **CORS enabled**: Cross-origin requests for web clients
- **Tracing middleware**: Colored request/response logging
- **Static file serving**: Optional website hosting
#### [`GET /api/vecs/id-count`](https://bitcoinresearchkit.org/api/vecs/id-count)
## Usage
Get the count of all existing vec ids.
### Basic Server Setup
#### [`GET /api/vecs/vec-count`](https://bitcoinresearchkit.org/api/vecs/vec-count)
```rust
use brk_server::Server;
use brk_interface::Interface;
use brk_indexer::Indexer;
use brk_computer::Computer;
Get the count of all existing vecs. \
Equals to the sum of supported Indexes of each vec id.
// Load data sources
let indexer = Indexer::forced_import("./brk_data")?;
let computer = Computer::forced_import("./brk_data", &indexer, None)?;
#### [`GET /api/vecs/indexes`](https://bitcoinresearchkit.org/api/vecs/indexes)
// Create interface and server
let interface = Interface::build(&indexer, &computer);
let server = Server::new(interface, Some("./website".into()));
Get the list of all existing indexes.
#### [`GET /api/vecs/accepted-indexes`](https://bitcoinresearchkit.org/api/vecs/accepted-indexes)
Get an object which has all existing indexes as keys and a list of their accepted variants as values.
#### [`GET /api/vecs/ids`](https://bitcoinresearchkit.org/api/vecs/ids)
Get a paginated list of all existing vec ids. \
There are up to 1,000 values per page. \
If the `page` param is omitted, it will default to page `0`.
#### [`GET /api/vecs/index-to-ids`](https://bitcoinresearchkit.org/api/vecs/index-to-ids)
Get a paginated list of all vec ids which support a given index.
There are up to 1,000 values per page.
If the `page` param is omitted, it will default to the first page.
#### [`GET /api/vecs/id-to-indexes`](https://bitcoinresearchkit.org/api/vecs/id-to-indexes)
Get a list of all indexes supported by a given vec id.
The list will be empty if the vec id isn't correct.
#### `GET /api/vecs/{INDEX}-to-{ID}`
This endpoint retrieves data based on the specified vector index and id.
**Parameters:**
| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `from` | `signed int` | No | Inclusive starting index for pagination (default is 0). |
| `to` | `signed int` | No | Exclusive ending index for pagination (default is the total number of results). Overrides `count` |
| `count` | `unsigned int` | No | The number of values requested |
| `format` | `string` | No | The format of the response. Options include `json`, `csv`, `tsv`, or `md` (default is `json`). |
**Examples:**
```sh
# GET /api/vecs/date-to-close
curl https://bitcoinresearchkit.org/api/vecs/date-to-close
# GET /api/vecs/date-to-close?from=-100
curl https://bitcoinresearchkit.org/api/vecs/date-to-close?from=-100
# GET /api/vecs/date-to-close?count=100&format=csv
curl https://bitcoinresearchkit.org/api/vecs/date-to-close?count=100&format=csv
// Start server (with MCP support)
server.serve(true).await?;
```
#### `GET /api/vecs/query`
### API Access Patterns
Get one or multiple vecs depending on given parameters.
If you'd like to request multiple vec ids, simply separate them with a ','. \
To get the last value set `-1` to the `from` parameter. \
The response's format will depend on the given parameters, it will be:
- A value: If requested only one vec and the given range returns one value (for example: `from=-1`)
- A list: If requested only one vec and the given range returns multiple values (for example: `from=-1000&count=100` or `from=-444&to=-333`)
- A matrix: When multiple vecs are requested, even if they each return one value.
#### Single Vector Queries
**Parameters:**
```bash
# Latest 100 price values
curl "http://brekit.org/api/vecs/date-to-close?from=-100"
| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `index` | `VecIndex` | Yes | The vector index to query. |
| `ids` | `VecId[]` | Yes | A comma or space-separated list of vector IDs to retrieve. |
| `from` | `signed int` | No | Inclusive starting index for pagination (default is 0). |
| `to` | `signed int` | No | Exclusive ending index for pagination (default is the total number of results). Overrides `count` |
| `count` | `unsigned int` | No | The number of values requested |
| `format` | `string` | No | The format of the response. Options include `json`, `csv`, `tsv`, or `md` (default is `json`). |
# First 50 difficulty values as CSV
curl "http://brekit.org/api/vecs/height-to-difficulty?count=50&format=csv"
**Examples:**
```sh
# GET /api/vecs/query?index=date&ids=ohlc
curl https://bitcoinresearchkit.org/api/vecs/query?index=date&ids=ohlc
# GET /api/vecs/query?index=week&ids=ohlc,block-interval-average&from=0&to=20&format=md
curl https://bitcoinresearchkit.org/api/vecs/query?index=week&ids=ohlc,block-interval-average&from=0&to=20&format=md
# Range from block 800,000 to 800,100
curl "https://brekit.org/api/vecs/height-to-timestamp?from=800000&to=800100"
```
### Meta
#### Multi-Vector Queries
#### [`GET /version`](https://bitcoinresearchkit.org/version)
```bash
# Multiple price metrics for last 30 days
curl "http://brekit.org/api/vecs/query?index=date&ids=open,high,low,close&from=-30&format=csv"
The version of the server and thus BRK.
# Block statistics for specific range
curl "https://brekit.org/api/vecs/query?index=height&ids=size,weight,tx_count,fee_sum&from=800000&count=100"
### Files
# Weekly analytics as JSON matrix
curl "https://brekit.org/api/vecs/query?index=week&ids=close,difficulty&from=-52"
```
#### `GET /*`
## API Reference
Catch all.
### Vector Metadata Endpoints
When no pattern is found, the server will look for a match inside the folder of the chosen website, if any.
| Endpoint | Description |
|----------|-------------|
| `GET /api/vecs/index-count` | Total number of available indexes |
| `GET /api/vecs/id-count` | Total number of vector IDs |
| `GET /api/vecs/vec-count` | Total vectors (all index/ID combinations) |
| `GET /api/vecs/indexes` | List of all available indexes |
| `GET /api/vecs/accepted-indexes` | Mapping of indexes to accepted variants |
| `GET /api/vecs/ids?page=N` | Paginated vector IDs (1000/page) |
| `GET /api/vecs/index-to-ids?index=INDEX&page=N` | IDs supporting given index |
| `GET /api/vecs/id-to-indexes?id=ID` | Indexes supported by given ID |
### Vector Data Access
#### Direct Access Pattern
`GET /api/vecs/{INDEX}-to-{ID}`
```bash
# Examples
curl "/api/vecs/height-to-timestamp"
curl "/api/vecs/date-to-close"
curl "/api/vecs/month-to-supply"
```
#### Multi-Vector Query
`GET /api/vecs/query`
**Required Parameters:**
- `index`: Vector index type (height, date, week, month, etc.)
- `ids`: Comma or space-separated vector IDs
**Optional Parameters:**
- `from` (i64): Start index (negative = from end, default: 0)
- `to` (i64): End index (exclusive, negative = from end)
- `count` (usize): Maximum number of results
- `format`: Output format (`json`, `csv`, `tsv`, `md`)
### Response Types
**Single Value** (one vector, one result):
```json
42.5
```
**Array** (one vector, multiple results):
```json
[42.5, 43.1, 44.2]
```
**Matrix** (multiple vectors):
```json
[
[42.5, 43.1, 44.2],
[1500, 1520, 1480],
[0.05, 0.052, 0.048]
]
```
**CSV Format**:
```csv
index,close,supply,feerate
0,42.5,1500,0.05
1,43.1,1520,0.052
2,44.2,1480,0.048
```
### System Endpoints
| Endpoint | Description |
|----------|-------------|
| `GET /version` | Server version information |
| `GET /health` | Health check with timestamp |
| `GET /api` | API documentation redirect |
| `GET /*` | Static file serving (if enabled) |
## Configuration
### Server State
```rust
pub struct AppState {
interface: &'static Interface<'static>, // Data access interface
path: Option<PathBuf>, // Static files path
cache: Arc<Cache<String, Bytes>>, // Response cache
}
```
### Middleware Stack
- **Compression Layer**: Brotli, Gzip, Zstd, Deflate
- **Response URI Layer**: Adds URI to response extensions for logging
- **Trace Layer**: Request/response logging with colored output
### Caching Strategy
- **ETag generation**: Based on data content and query parameters
- **Conditional requests**: 304 Not Modified for unchanged data
- **In-memory cache**: 10,000 items with LRU eviction
- **Cache headers**: `Cache-Control: must-revalidate`
## Performance Characteristics
### Request Weight System
- **Weight calculation**: Based on data range size and complexity
- **Weight limit**: 320,000 units per request
- **Protection**: Prevents oversized queries that could impact performance
### Compression
- **Automatic negotiation**: Based on `Accept-Encoding` header
- **Multiple algorithms**: Brotli (best), Zstd, Gzip, Deflate
- **Transparent**: Applied automatically to all responses
### Logging
- **Colored output**: Green for 200, red for errors, gray for redirects
- **Request timing**: Latency measurement and display
- **Status tracking**: HTTP status codes with appropriate colors
## Dependencies
- `axum` - High-performance async web framework
- `tower-http` - HTTP middleware (compression, tracing, CORS)
- `brk_interface` - Data access and formatting layer
- `brk_mcp` - Model Context Protocol routes
- `quick_cache` - Fast in-memory caching
- `tokio` - Async runtime for networking
---
*This README was generated by Claude Code*
+1 -1
View File
@@ -12,7 +12,7 @@ use brk_server::Server;
use vecdb::Exit;
pub fn main() -> Result<()> {
brk_logger::init(Some(Path::new(".log")));
brk_logger::init(Some(Path::new(".log")))?;
let process = true;
+2 -2
View File
@@ -62,7 +62,7 @@ fn req_to_response_res(
if weight > MAX_WEIGHT {
return Err(Error::Str(
"Request is too heavy, max weight is {MAX_WEIGHT}",
"Request is too heavy, max weight is {MAX_WEIGHT} bytes",
));
}
@@ -82,7 +82,7 @@ fn req_to_response_res(
let guard_res = cache.get_value_or_guard(
&format!("{}{}{etag}", uri.path(), uri.query().unwrap_or("")),
Some(Duration::from_millis(500)),
Some(Duration::from_millis(50)),
);
let mut response = if let GuardResult::Value(v) = guard_res {
+1 -1
View File
@@ -88,7 +88,7 @@ fn path_to_response_(headers: &HeaderMap, app_state: &AppState, path: &Path) ->
let guard_res = if !must_revalidate {
Some(app_state.cache.get_value_or_guard(
&path.to_str().unwrap().to_owned(),
Some(Duration::from_millis(500)),
Some(Duration::from_millis(50)),
))
} else {
None
+1
View File
@@ -8,6 +8,7 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
+147 -1
View File
@@ -1 +1,147 @@
# BRK Store
# brk_store
**Blockchain-aware key-value storage wrapper for Bitcoin data**
`brk_store` is a thin, typed wrapper around the [Fjall](https://github.com/fjall-rs/fjall) embedded key-value store, specifically designed for Bitcoin blockchain data storage. It provides height-based versioning, batch operations, and optimized configurations for Bitcoin data patterns.
## What it provides
- **Blockchain-aware storage**: Built-in height tracking and versioning for Bitcoin block processing
- **Type-safe interface**: Generic over key and value types with automatic serialization
- **Batch operations**: Efficient batched inserts/deletes with transactional support
- **Metadata management**: Automatic version checking and height progression tracking
- **Performance optimization**: Pre-configured for Bitcoin data patterns
## Key Features
### Height-Based Processing
- **Height tracking**: Each store tracks the last processed blockchain height
- **Conditional operations**: Insert only when blockchain height requires it
- **Automatic versioning**: Combines Fjall version + user version for compatibility
### Batch Operations
- **In-memory staging**: Changes staged before atomic commit
- **Transactional safety**: All changes written atomically or none at all
- **Efficient processing**: Minimizes disk I/O through batching
### Performance Optimization
- **Optimized configurations**: 32MB write buffers, 8MB memtables for Bitcoin data
- **Configurable bloom filters**: Optional for faster key lookups
- **Memory efficiency**: Reused read transactions, minimal overhead
## Usage
### Basic Operations
```rust
use brk_store::{Store, open_keyspace};
use brk_structs::{Height, Version};
// Open keyspace
let keyspace = open_keyspace(Path::new("./data"))?;
// Create typed store
let mut store: Store<String, u64> = Store::import(
&keyspace,
Path::new("./data"),
"store_name",
Version::ZERO,
Some(true), // Enable bloom filters
)?;
// Conditional insertion based on height
store.insert_if_needed("key1".to_string(), 42u64, Height::new(800_000));
// Batch commit changes
store.commit(Height::new(800_000))?;
// Persist to disk
store.persist()?;
```
### Querying Data
```rust
// Get value (checks pending puts first, then disk)
if let Some(value) = store.get(&"key1".to_string())? {
println!("Value: {}", value);
}
// Check if store needs processing at height
if store.needs(Height::new(800_000)) {
// Process this height
}
```
### Managing Multiple Stores
```rust
use brk_store::AnyStore;
fn process_stores(stores: &mut [Box<dyn AnyStore>]) -> brk_error::Result<()> {
let height = Height::new(800_000);
for store in stores {
if store.needs(height) {
// Process data for this store
store.commit(height)?;
}
}
// Persist all stores
for store in stores {
store.persist()?;
}
Ok(())
}
```
### Store Reset and Reindexing
```rust
// Reset store for complete reindexing
store.reset()?;
// Store automatically recreated on next import
let mut fresh_store: Store<String, u64> = Store::import(
&keyspace,
path,
"store_name",
Version::ZERO,
Some(true)
)?;
```
## Store Lifecycle
1. **Import**: Create or open existing store with version checking
2. **Insert**: Add key-value pairs with height-based conditional insertion
3. **Commit**: Write batched changes to disk atomically
4. **Persist**: Force sync all data to storage
5. **Reset**: Clear all data for reindexing if needed
## Type Requirements
Keys and values must implement:
- `Debug + Clone + From<ByteView> + Ord` for keys
- `Debug + Clone + From<ByteView>` for values
This enables automatic serialization and type-safe storage operations.
## Version Management
- **Automatic migration**: Detects version mismatches and resets stores automatically
- **File-based metadata**: Stores version and height info for persistence
- **Compatibility checking**: Prevents data corruption from version incompatibilities
## Dependencies
- `fjall` - Embedded key-value store engine
- `byteview` - Zero-copy byte view operations
- `brk_structs` - Bitcoin-aware type system
- `brk_error` - Unified error handling
---
*This README was generated by Claude Code*
-3
View File
@@ -1,7 +1,4 @@
#![doc = include_str!("../README.md")]
#![doc = "\n## Example\n\n```rust"]
#![doc = include_str!("../examples/main.rs")]
#![doc = "```"]
use std::{
borrow::Cow,
+1
View File
@@ -6,6 +6,7 @@ edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
rust-version.workspace = true
build = "build.rs"
[dependencies]
+103 -26
View File
@@ -1,28 +1,105 @@
# BRK Core
# brk_structs
<p align="left">
<a href="https://github.com/bitcoinresearchkit/brk">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
</a>
<a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
<img src="https://img.shields.io/crates/l/brk" alt="License" />
</a>
<a href="https://crates.io/crates/brk_structs">
<img src="https://img.shields.io/crates/v/brk_structs" alt="Version" />
</a>
<a href="https://docs.rs/brk_structs">
<img src="https://img.shields.io/docsrs/brk_structs" alt="Documentation" />
</a>
<img src="https://img.shields.io/crates/size/brk_structs" alt="Size" />
<a href="https://deps.rs/crate/brk_structs">
<img src="https://deps.rs/crate/brk_structs/latest/status.svg" alt="Dependency status">
</a>
<a href="https://discord.gg/HaR3wpH3nr">
<img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
</a>
<a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
<img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
</a>
</p>
**Bitcoin-aware type system and data structures for blockchain analysis**
A list of structs that are used throughout the project as units, think of `Date`, `Height`, `Sats`, `Txindex` or anything that can be either a key and/or a value of a dataset.
`brk_structs` provides the foundational type system for the Bitcoin Research Kit (BRK). It offers strongly-typed, storage-optimized data structures that encode Bitcoin protocol knowledge and enable efficient blockchain data analysis.
## What it provides
- **Type Safety**: Distinct wrapper types prevent common mistakes (e.g., confusing block height with transaction index)
- **Storage Optimization**: Zero-copy serialization and optional compression for efficient disk storage
- **Bitcoin Protocol Knowledge**: Built-in understanding of halving epochs, address types, and blockchain constants
- **Comprehensive Time Indexing**: Multiple temporal granularities for time-series analysis
- **Flexible Grouping**: Framework for categorizing and filtering blockchain data
## Key Features
### Core Blockchain Types
- `Height` - Block heights with Bitcoin-specific arithmetic
- `Txid`/`BlockHash` - Transaction and block identifiers with prefix optimization
- `TxIndex`/`InputIndex`/`OutputIndex` - Component indices with type safety
- `Date`/`Timestamp` - Time representations anchored to Bitcoin genesis block
### Value and Currency Types
- `Sats` - Satoshi amounts with comprehensive arithmetic operations
- `Bitcoin` - BTC amounts with precision handling
- `Dollars`/`Cents` - Fiat currency for price analysis
- OHLC types (`Open<T>`, `High<T>`, `Low<T>`, `Close<T>`) for market data
### Address System
- `OutputType` - All Bitcoin script types (P2PKH, P2SH, P2WPKH, P2WSH, P2TR, etc.)
- `AddressBytes` - Type-safe address data extraction
- Address-specific indices for each output type
- `AnyAddressIndex` - Unified address indexing
### Temporal Indexing
- `DateIndex` - Days since Bitcoin genesis
- Time granularities: `WeekIndex`, `MonthIndex`, `QuarterIndex`, `YearIndex`, `DecadeIndex`
- `HalvingEpoch` - Bitcoin halving periods (every 210,000 blocks)
- `DifficultyEpoch` - Difficulty adjustment periods
## Usage
### Basic Types and Conversions
```rust
use brk_structs::*;
// Type-safe blockchain data
let height = Height::new(800_000);
let epoch = HalvingEpoch::from(height);
let amount = Sats::_1BTC * 2;
// Time-based indexing
let date = Date::new(2024, 1, 15);
let month_idx = MonthIndex::from(date);
```
### Address Handling
```rust
// Extract address information from Bitcoin scripts
let output_type = OutputType::from(&script);
if output_type.is_address() {
let address_bytes = AddressBytes::try_from((&script, output_type))?;
}
```
### Zero-Copy Storage
```rust
// Efficient serialization without copying
let height_bytes = height.as_bytes();
let recovered_height = Height::read_from_bytes(height_bytes)?;
```
### Data Grouping and Filtering
```rust
// Flexible filtering for analytics
let utxo_groups = UTXOGroups {
all: total_utxos,
age_range: ByAgeRange::new(age_filtered_utxos),
epoch: ByEpoch::new(epoch_filtered_utxos),
// ... more groupings
};
```
## Storage Optimization
All types implement zero-copy serialization traits:
- **Zero overhead**: Direct memory mapping without serialization costs
- **Optional compression**: Configurable zstd compression for space efficiency
- **Type safety**: Compile-time guarantees about data layout and endianness
## Dependencies
- `bitcoin` - Bitcoin protocol types and script parsing
- `vecdb` - Vector database storage traits
- `zerocopy` - Zero-copy serialization framework
- `serde` - JSON serialization support
- `jiff` - Modern date/time handling
---
*This README was generated by Claude Code*
+3 -5
View File
@@ -24,20 +24,18 @@ use super::Dollars;
)]
pub struct Cents(i64);
const SIGNIFICANT_DIGITS: i32 = 4;
impl Cents {
pub const fn mint(value: i64) -> Self {
Self(value)
}
pub fn round_to_4_digits(self) -> Self {
pub fn round_to(self, digits: i32) -> Self {
let v = self.0;
let ilog10 = v.checked_ilog10().unwrap_or(0) as i32;
Self::from(if ilog10 >= SIGNIFICANT_DIGITS {
let log_diff = ilog10 - SIGNIFICANT_DIGITS + 1;
Self::from(if ilog10 >= digits {
let log_diff = ilog10 - digits + 1;
let pow = 10.0_f64.powi(log_diff);
+6 -2
View File
@@ -39,8 +39,12 @@ impl Dollars {
Dollars((self.0 * 100.0).round() / 100.0)
}
pub fn round_to_4_digits(self) -> Self {
Self::from(Cents::from(self).round_to_4_digits())
pub fn round_to(self, digits: i32) -> Self {
Self::from(Cents::from(self).round_to(digits))
}
pub fn is_negative(&self) -> bool {
self.0 < 0.0
}
}
+8 -6
View File
@@ -12,22 +12,24 @@ use super::{Sats, StoredU64};
#[derive(
Debug, Clone, Copy, Serialize, FromBytes, Immutable, IntoBytes, KnownLayout, StoredCompressed,
)]
pub struct Feerate(f32);
pub struct Feerate(f64);
impl From<(Sats, StoredU64)> for Feerate {
fn from((sats, vsize): (Sats, StoredU64)) -> Self {
Self((f64::from(sats) / f64::from(vsize)) as f32)
let sats = u64::from(sats);
let vsize = u64::from(vsize);
Self(((sats * 1000 + vsize.checked_sub(1).unwrap()) / vsize) as f64 / 1000.0)
}
}
impl From<f64> for Feerate {
fn from(value: f64) -> Self {
Self(value as f32)
Self(value)
}
}
impl From<Feerate> for f64 {
fn from(value: Feerate) -> Self {
value.0 as f64
value.0
}
}
@@ -47,13 +49,13 @@ impl AddAssign for Feerate {
impl Div<usize> for Feerate {
type Output = Self;
fn div(self, rhs: usize) -> Self::Output {
Self((self.0 as f64 / rhs as f64) as f32)
Self(self.0 / rhs as f64)
}
}
impl From<usize> for Feerate {
fn from(value: usize) -> Self {
Self(value as f32)
Self(value as f64)
}
}
+4
View File
@@ -66,6 +66,10 @@ impl Height {
pub fn is_zero(self) -> bool {
self == Self::ZERO
}
pub fn is_not_zero(self) -> bool {
self != Self::ZERO
}
}
impl PartialEq<u64> for Height {
@@ -21,7 +21,17 @@ impl LoadedAddressData {
}
pub fn realized_price(&self) -> Dollars {
(self.realized_cap / Bitcoin::from(self.amount())).round_to_4_digits()
let p = (self.realized_cap / Bitcoin::from(self.amount())).round_to(4);
if p.is_negative() {
dbg!((
self.realized_cap,
self.amount(),
Bitcoin::from(self.amount()),
p
));
panic!("");
}
p
}
#[inline]
@@ -38,7 +48,12 @@ impl LoadedAddressData {
self.received += amount;
self.outputs_len += 1;
if let Some(price) = price {
self.realized_cap += price * amount;
let added = price * amount;
self.realized_cap += added;
if added.is_negative() || self.realized_cap.is_negative() {
dbg!((self.realized_cap, price, amount, added));
panic!();
}
}
}
@@ -49,10 +64,20 @@ impl LoadedAddressData {
self.sent += amount;
self.outputs_len -= 1;
if let Some(previous_price) = previous_price {
self.realized_cap = self
.realized_cap
.checked_sub(previous_price * amount)
.unwrap();
let subtracted = previous_price * amount;
let realized_cap = self.realized_cap.checked_sub(subtracted).unwrap();
if self.realized_cap.is_negative() || realized_cap.is_negative() {
dbg!((
self,
realized_cap,
previous_price,
amount,
previous_price * amount,
subtracted
));
panic!();
}
self.realized_cap = realized_cap;
}
Ok(())
}

Some files were not shown because too many files have changed in this diff Show More