Mirror/brk
1
0
Fork 0
mirror of https://github.com/bitcoinresearchkit/brk.git synced 2026-05-22 15:54:47 -07:00
Code Issues Packages Projects Releases Wiki Activity

readmes: regenerated

Browse Source
This commit is contained in:
nym21
2025-09-16 22:01:55 +02:00
parent 185fc7b6ed
commit 301dee96dc
19 changed files with 3128 additions and 2065 deletions
Download Patch File Download Diff File Expand all files Collapse all files

331
crates/brk_store/README.md
View File

@@ -1,147 +1,276 @@
# brk_store
**Blockchain-aware key-value storage wrapper for Bitcoin data**
High-performance transactional key-value store wrapper around Fjall with blockchain-aware versioning.
`brk_store` is a thin, typed wrapper around the [Fjall](https://github.com/fjall-rs/fjall) embedded key-value store, specifically designed for Bitcoin blockchain data storage. It provides height-based versioning, batch operations, and optimized configurations for Bitcoin data patterns.
[![Crates.io](https://img.shields.io/crates/v/brk_store.svg)](https://crates.io/crates/brk_store)
[![Documentation](https://docs.rs/brk_store/badge.svg)](https://docs.rs/brk_store)
## What it provides
## Overview
- **Blockchain-aware storage**: Built-in height tracking and versioning for Bitcoin block processing
- **Type-safe interface**: Generic over key and value types with automatic serialization
- **Batch operations**: Efficient batched inserts/deletes with transactional support
- **Metadata management**: Automatic version checking and height progression tracking
- **Performance optimization**: Pre-configured for Bitcoin data patterns
This crate provides a type-safe wrapper around the Fjall LSM-tree database engine, specifically designed for Bitcoin blockchain data storage. It offers transactional operations, automatic version management, height-based synchronization, and optimized configuration for blockchain workloads with support for batch operations and efficient range queries.
## Key Features
**Key Features:**
### Height-Based Processing
- **Height tracking**: Each store tracks the last processed blockchain height
- **Conditional operations**: Insert only when blockchain height requires it
- **Automatic versioning**: Combines Fjall version + user version for compatibility
- Transactional key-value storage with ACID guarantees
- Blockchain height-based synchronization and versioning
- Automatic metadata management with version compatibility checking
- Optimized configuration for Bitcoin data patterns (32MB write buffers, 8MB memtables)
- Type-safe generic interface with zero-copy ByteView integration
- Batch operations with deferred commits for performance
- Optional bloom filter configuration for space/speed tradeoffs
### Batch Operations
- **In-memory staging**: Changes staged before atomic commit
- **Transactional safety**: All changes written atomically or none at all
- **Efficient processing**: Minimizes disk I/O through batching
**Target Use Cases:**
### Performance Optimization
- **Optimized configurations**: 32MB write buffers, 8MB memtables for Bitcoin data
- **Configurable bloom filters**: Optional for faster key lookups
- **Memory efficiency**: Reused read transactions, minimal overhead
- Bitcoin blockchain indexing with transactional consistency
- UTXO set management requiring atomic updates
- Address-to-transaction mapping with range queries
- Any Bitcoin data storage requiring versioned, transactional access
## Usage
## Installation
### Basic Operations
```toml
cargo add brk_store
```
## Quick Start
```rust
use brk_store::{Store, open_keyspace};
use brk_structs::{Height, Version};
use std::path::Path;
// Open keyspace
// Open keyspace (database instance)
let keyspace = open_keyspace(Path::new("./data"))?;
// Create typed store
let mut store: Store<String, u64> = Store::import(
// Create typed store for height-to-blockhash mapping
let mut store: Store<Height, BlockHash> = Store::import(
&keyspace,
Path::new("./data"),
"store_name",
Version::ZERO,
Path::new("./data/height_to_hash"),
"height-to-blockhash",
Version::ONE,
Some(true), // Enable bloom filters
)?;
// Conditional insertion based on height
store.insert_if_needed("key1".to_string(), 42u64, Height::new(800_000));
// Insert data (batched in memory)
store.insert_if_needed(
Height::new(750000),
block_hash,
Height::new(750000)
);
// Batch commit changes
store.commit(Height::new(800_000))?;
// Commit transaction to disk
store.commit(Height::new(750000))?;
// Persist to disk
// Query data
if let Some(hash) = store.get(&Height::new(750000))? {
println!("Block hash: {}", hash);
}
```
## API Overview
### Core Types
- **`Store<Key, Value>`**: Generic transactional store with type-safe operations
- **`AnyStore`**: Trait for height-based synchronization and metadata operations
- **`StoreMeta`**: Version and height metadata management
- **`TransactionalKeyspace`**: Fjall keyspace wrapper for database management
### Key Methods
**`Store::import(keyspace, path, name, version, bloom_filters) -> Result<Self>`**
Creates or opens a store with automatic version checking and migration.
**`get(&self, key: &Key) -> Result<Option<Cow<Value>>>`**
Retrieves value by key, checking both pending writes and committed data.
**`insert_if_needed(&mut self, key: Key, value: Value, height: Height)`**
Conditionally inserts data based on blockchain height requirements.
**`commit(&mut self, height: Height) -> Result<()>`**
Atomically commits all pending operations and updates metadata.
### Height-Based Synchronization
The store implements blockchain-aware synchronization:
- **`has(height)`**: Checks if store contains data up to specified height
- **`needs(height)`**: Determines if store requires data for specified height
- **`height()`**: Returns current synchronized height
## Examples
### Basic Key-Value Operations
```rust
use brk_store::{Store, open_keyspace};
use brk_structs::{Height, TxId, Version};
let keyspace = open_keyspace(Path::new("./blockchain_data"))?;
// Create store for transaction index
let mut tx_store: Store<TxId, Height> = Store::import(
&keyspace,
Path::new("./blockchain_data/txid_to_height"),
"txid-to-height",
Version::ONE,
Some(true),
)?;
// Insert transaction mapping
let txid = TxId::from_str("abcdef...")?;
let height = Height::new(800000);
tx_store.insert_if_needed(txid, height, height);
tx_store.commit(height)?;
// Query transaction height
if let Some(tx_height) = tx_store.get(&txid)? {
println!("Transaction {} found at height {}", txid, tx_height);
}
```
### Batch Processing with Height Synchronization
```rust
use brk_store::{Store, AnyStore};
let mut store: Store<Address, AddressData> = Store::import(/* ... */)?;
// Process blocks sequentially
for block_height in 750000..750100 {
let height = Height::new(block_height);
// Skip if already processed
if store.has(height) {
continue;
}
// Process block transactions
for (address, data) in process_block(block_height)? {
store.insert_if_needed(address, data, height);
}
// Commit entire block atomically
store.commit(height)?;
println!("Processed block {}", block_height);
}
// Ensure data is persisted to disk
store.persist()?;
```
### Querying Data
### Version Migration and Reset
```rust
// Get value (checks pending puts first, then disk)
if let Some(value) = store.get(&"key1".to_string())? {
println!("Value: {}", value);
}
use brk_store::{Store, AnyStore};
use brk_structs::Version;
// Check if store needs processing at height
if store.needs(Height::new(800_000)) {
// Process this height
}
```
### Managing Multiple Stores
```rust
use brk_store::AnyStore;
fn process_stores(stores: &mut [Box<dyn AnyStore>]) -> brk_error::Result<()> {
let height = Height::new(800_000);
for store in stores {
if store.needs(height) {
// Process data for this store
store.commit(height)?;
}
}
// Persist all stores
for store in stores {
store.persist()?;
}
Ok(())
}
```
### Store Reset and Reindexing
```rust
// Reset store for complete reindexing
store.reset()?;
// Store automatically recreated on next import
let mut fresh_store: Store<String, u64> = Store::import(
// Open store with new version
let mut store: Store<Height, Data> = Store::import(
&keyspace,
path,
"store_name",
Version::ZERO,
Some(true)
"my-store",
Version::TWO, // Upgraded from Version::ONE
Some(false), // Disable bloom filters for space
)?;
// Check if reset is needed for data consistency
if store.version() != Version::TWO {
println!("Resetting store for version compatibility");
store.reset()?;
}
// Verify store is empty after reset
assert!(store.is_empty()?);
```
## Store Lifecycle
### Iterator-Based Data Access
1. **Import**: Create or open existing store with version checking
2. **Insert**: Add key-value pairs with height-based conditional insertion
3. **Commit**: Write batched changes to disk atomically
4. **Persist**: Force sync all data to storage
5. **Reset**: Clear all data for reindexing if needed
```rust
use brk_store::Store;
## Type Requirements
let store: Store<Height, BlockHash> = Store::import(/* ... */)?;
Keys and values must implement:
- `Debug + Clone + From<ByteView> + Ord` for keys
- `Debug + Clone + From<ByteView>` for values
// Iterate over all key-value pairs
for (height, block_hash) in store.iter() {
println!("Height {}: {}", height, block_hash);
This enables automatic serialization and type-safe storage operations.
// Process in chunks for memory efficiency
if height.as_u32() % 10000 == 0 {
println!("Processed up to height {}", height);
}
}
```
## Version Management
## Architecture
- **Automatic migration**: Detects version mismatches and resets stores automatically
- **File-based metadata**: Stores version and height info for persistence
- **Compatibility checking**: Prevents data corruption from version incompatibilities
### Storage Engine
## Dependencies
Built on Fjall LSM-tree engine with optimizations:
- `fjall` - Embedded key-value store engine
- `byteview` - Zero-copy byte view operations
- `brk_structs` - Bitcoin-aware type system
- `brk_error` - Unified error handling
- **Write Buffers**: 32MB for high-throughput blockchain ingestion
- **Memtables**: 8MB for balanced memory usage
- **Manual Journal Persist**: Explicit control over durability guarantees
- **Bloom Filters**: Configurable for read-heavy vs. space-constrained workloads
### Transaction Model
- **Read Transactions**: Consistent point-in-time snapshots
- **Write Transactions**: ACID-compliant with rollback support
- **Batch Operations**: In-memory accumulation with atomic commits
- **Height Synchronization**: Blockchain-aware conflict resolution
### Version Management
Automatic handling of schema evolution:
1. **Version Detection**: Reads stored version from metadata
2. **Compatibility Check**: Compares with expected version
3. **Migration**: Automatic store reset for incompatible versions
4. **Metadata Update**: Persistent version tracking
### Memory Management
- **Zero-Copy**: ByteView integration for efficient serialization
- **Copy-on-Write**: Cow<Value> for memory-efficient reads
- **Parking Lot**: RwLock for concurrent partition access
- **Deferred Operations**: BTreeMap/BTreeSet for batched writes
## Configuration
### Keyspace Options
```rust
use fjall::Config;
let keyspace = Config::new(path)
.max_write_buffer_size(32 * 1024 * 1024) // 32MB write buffers
.open_transactional()?;
```
### Partition Options
```rust
use fjall::PartitionCreateOptions;
let options = PartitionCreateOptions::default()
.max_memtable_size(8 * 1024 * 1024) // 8MB memtables
.manual_journal_persist(true) // Manual sync control
.bloom_filter_bits(None); // Disable bloom filters
```
## Code Analysis Summary
**Main Structure**: `Store<Key, Value>` generic wrapper around Fjall with typed operations and metadata management \
**Transaction Layer**: Read/write transaction abstraction with deferred batch operations via BTreeMap/BTreeSet \
**Metadata System**: `StoreMeta` for version compatibility and height tracking with automatic migration \
**Height Synchronization**: Blockchain-aware operations with `needs()`, `has()`, and conditional insertion logic \
**Memory Efficiency**: Zero-copy ByteView integration with parking_lot RwLock for concurrent access \
**Storage Engine**: Fjall LSM-tree with optimized configuration for blockchain workloads \
**Architecture**: Type-safe database abstraction with ACID guarantees and blockchain-specific synchronization patterns
---
*This README was generated by Claude Code*
_This README was generated by Claude Code_