mirror of
https://github.com/bitcoinresearchkit/brk.git
synced 2026-05-18 05:44:48 -07:00
73 lines
2.8 KiB
Markdown
73 lines
2.8 KiB
Markdown
# brk_indexer
|
|
|
|
Full Bitcoin blockchain indexer for fast analytics queries.
|
|
|
|
## What It Enables
|
|
|
|
Transform raw Bitcoin blockchain data into indexed vectors and key-value stores optimized for analytics. Query any block, transaction, address, or UTXO without scanning the chain.
|
|
|
|
## Key Features
|
|
|
|
- **Multi-phase block processing**: Parallel TXID computation, input/output processing, sequential finalization
|
|
- **Address indexing**: Maps addresses to their transaction history and UTXOs per address type
|
|
- **UTXO tracking**: Live outpoint→value lookups, address→unspent outputs
|
|
- **Reorg handling**: Automatic rollback to valid chain state on reorganization
|
|
- **Collision detection**: Validates rapidhash-based prefix lookups against known duplicate TXIDs
|
|
- **Incremental snapshots**: Periodic checkpoints for crash recovery
|
|
|
|
## Core API
|
|
|
|
```rust,ignore
|
|
let mut indexer = Indexer::forced_import(&outputs_dir)?;
|
|
|
|
// Index new blocks
|
|
let starting_indexes = indexer.index(&blocks, &client, &exit)?;
|
|
|
|
// Access indexed data
|
|
let height = indexer.stores.txidprefix_to_txindex.get(&txid_prefix)?;
|
|
let blockhash = indexer.vecs.block.height_to_blockhash.get(height)?;
|
|
```
|
|
|
|
## Data Structures
|
|
|
|
**Vecs** (append-only vectors):
|
|
- Block: `height_to_blockhash`, `height_to_timestamp`, `height_to_difficulty`
|
|
- Transaction: `txindex_to_txid`, `txindex_to_height`, `txindex_to_base_size`
|
|
- Input/Output: `txinindex_to_outpoint`, `txoutindex_to_value`, `txoutindex_to_outputtype`
|
|
- Address: Per-type `typeindex_to_addressbytes`
|
|
|
|
**Stores** (key-value lookups):
|
|
- `txidprefix_to_txindex` - TXID lookup via 10-byte prefix
|
|
- `blockhashprefix_to_height` - Block lookup via 4-byte prefix
|
|
- `addresshash_to_addressindex` - Address lookup per type
|
|
- `addressindex_to_unspent_outpoints` - Live UTXO set per address
|
|
|
|
## Processing Pipeline
|
|
|
|
1. **Block metadata**: Store blockhash, difficulty, timestamp
|
|
2. **Compute TXIDs**: Parallel SHA256d across transactions
|
|
3. **Process inputs**: Lookup spent outpoints, resolve address info
|
|
4. **Process outputs**: Extract addresses, assign type indexes
|
|
5. **Finalize**: Sequential store updates, UTXO set mutations
|
|
6. **Commit**: Periodic flush to disk
|
|
|
|
## Performance
|
|
|
|
| Machine | Time | Disk | Peak Disk | Memory | Peak Memory |
|
|
|---------|------|------|-----------|--------|-------------|
|
|
| MBP M3 Pro (36GB, internal SSD) | 3.1h | 233 GB | 307 GB | 5.5 GB | 11 GB |
|
|
| Mac Mini M4 (16GB, external SSD) | 4.9h | 233 GB | 303 GB | 5.4 GB | 11 GB |
|
|
|
|
Full benchmark data: [`https://github.com/bitcoinresearchkit/benches/tree/main/brk_indexer`](/benches/brk_indexer)
|
|
|
|
## Recommended: mimalloc v3
|
|
|
|
Use [mimalloc v3](https://crates.io/crates/mimalloc) as the global allocator to reduce memory usage.
|
|
|
|
## Built On
|
|
|
|
- `brk_iterator` for block iteration
|
|
- `brk_store` for key-value storage
|
|
- `brk_grouper` for address type handling
|
|
- `brk_types` for domain types
|