release: v0.0.111

.claude/prompts/changelog.md

@@ -0,0 +1,221 @@
+# Changelog Generation for Claude Code
+
+**TASK**: Update docs/CHANGELOG.md for ALL latest releases missing from the file.
+
+## ⚠️ CRITICAL FAILURE MODE TO AVOID ⚠️
+**THE #1 FAILURE**: Ignoring most changes and only documenting a few
+**ABSOLUTELY FORBIDDEN**: Skipping changes, summarizing with "and other updates", or being incomplete
+**YOU MUST DOCUMENT EVERY SINGLE MEANINGFUL CHANGE - NO EXCEPTIONS**
+
+## CORE WORKFLOW - EXECUTE EXACTLY:
+
+### Step 1: Get Release Information
+```bash
+git tag --list --sort=version:refname
+```
+
+### Step 2: Process ONE Release at a Time
+For each missing release, execute these commands to get complete information:
+
+**First, get the file list:**
+```bash
+git diff --name-only [previous-tag]..[current-tag]
+```
+
+**Then, get changes excluding Cargo.lock:**
+```bash
+git diff [previous-tag]..[current-tag] -- . ':(exclude)Cargo.lock'
+```
+
+**If output is too large, examine files individually:**
+```bash
+git diff [previous-tag]..[current-tag] -- path/to/specific/file.rs
+```
+
+**For summary of changes per file (if needed):**
+```bash
+git diff --stat [previous-tag]..[current-tag]
+```
+
+### Step 3: Analyze Before Writing
+**MANDATORY**: Before writing ANY changelog entry, analyze the diff output and explain:
+- **Use `git diff --name-only` to see ALL changed files** - this prevents truncation issues
+- **Use `git diff -- . ':(exclude)Cargo.lock'` to see actual changes** without Cargo.lock noise
+- **If output is large, examine key files individually** with `git diff [tags] -- path/to/file`
+- **Identify every functional change** - what new capabilities, fixes, or modifications were made
+- What each code change accomplishes functionally
+- The user-facing or system impact of modifications
+- **Which crate each change belongs to** (based on file paths)
+- How changes group together logically within and across crates
+- What functionality is added/removed/modified per crate
+
+**COMPLETENESS CHECK**: State "I have analyzed X files and identified Y distinct functional changes to document"
+
+### Step 4: Write Changelog Entry
+Only after analysis, update CHANGELOG.md with ONE release entry.
+
+**REQUIRED**: End each release entry with a comparison link:
+```markdown
+[View changes](https://github.com/bitcoinresearchkit/brk/compare/vPREVIOUS...vCURRENT)
+```
+Where PREVIOUS is the previous release tag and CURRENT is the current release tag.
+
+### Step 5: Stop and Confirm
+**CRITICAL**: Process only ONE release, then ask for confirmation to continue.
+
+---
+
+## STRICT RULES
+
+### FORBIDDEN - NEVER MENTION:
+- **Cargo.lock** (ignore completely)
+- **Line counts** ("added 50 lines", "removed 200 lines")
+- **Dependency version bumps** (unless enabling major new features)
+- **Local crate version changes** (0.0.61 → 0.0.62)
+- **Vague words**: "Enhanced", "Improved", "Updated", "Refactored"
+
+### FORBIDDEN PHRASES:
+- "and other changes"
+- "various updates"
+- "along with minor improvements"
+- "among other enhancements"
+- **"along with other modifications"**
+- **"plus additional changes"**
+- **"including other updates"**
+- **"and more"**
+- **ANY phrase that suggests you're skipping changes**
+
+### REQUIRED - MUST INCLUDE:
+- **Every meaningful change** in the diff (no shortcuts)
+- **Specific functionality** descriptions
+- **Business impact** of each change
+- **Complete coverage** - if 20 changes exist, document all 20
+- **Source links** for significant changes (new features, breaking changes, major bug fixes)
+
+## WORKSPACE-SPECIFIC RULES
+
+### Crate Identification:
+- **Determine crate from file paths** in the diff (e.g., `crates/brk-core/src/lib.rs` → `brk-core`)
+- **Group all changes by their crate** before writing changelog entries
+- **Use crate name as subheading** under each change type section
+- **For root-level files**: Use `workspace` as the crate name
+
+### Cross-Crate Changes:
+- **When changes span multiple crates** for one feature, mention the relationship
+- **Example**: "Added new transaction API in `brk-core` with corresponding HTTP endpoints in `brk-api`"
+
+### Crate Naming:
+- **Use backticks** around crate names: `brk-core`, `brk-api`
+- **Use workspace structure** as shown in file paths, not display names
+
+### File Header (if missing):
+```markdown
+<!-- This changelog was generated by Claude Code -->
+```
+
+### Release Entry Format:
+```markdown
+## [vX.Y.Z](https://github.com/bitcoinresearchkit/brk/releases/tag/vX.Y.Z) - YYYY-MM-DD
+
+### Breaking Changes
+#### `crate-name`
+- Specific change with functional impact explanation ([source](https://github.com/bitcoinresearchkit/brk/blob/vX.Y.Z/path/to/file.rs))
+
+### New Features
+#### `crate-name`
+- Feature description with user benefit ([source](https://github.com/bitcoinresearchkit/brk/blob/vX.Y.Z/path/to/main/file.rs))
+- Another feature with implementation details
+
+#### `another-crate`
+- Feature specific to this crate
+
+### Bug Fixes
+#### `crate-name`
+- Specific problem that was resolved ([source](https://github.com/bitcoinresearchkit/brk/blob/vX.Y.Z/path/to/file.rs))
+- Another bug fix with impact description
+
+### Internal Changes
+#### `crate-name`
+- Architectural improvement with purpose
+- Code organization change with benefit
+
+[View changes](https://github.com/bitcoinresearchkit/brk/compare/vPREVIOUS...vCURRENT)
+```
+
+---
+
+## ANALYSIS REQUIREMENTS
+
+**Before writing changelog text, you MUST state:**
+
+1. **File discovery**: "Using git diff --name-only, I see X files were modified"
+2. **Cargo.lock check**: "After excluding Cargo.lock, Y files contain actual changes"
+3. **Functional changes identified**: "I identified these distinct changes: A, B, C, D..."
+4. **What each change does**: "Change A: adds new API endpoints, Change B: fixes memory leak, etc."
+5. **Which crates are affected**: "Changes span crates: A, B, C"
+6. **What this means for users**: "Users can now do X, bug Y is fixed, etc."
+7. **How changes group together**: "Changes A and B work together to implement feature W"
+8. **Cross-crate dependencies**: "Crate A's new feature requires the new API in crate B"
+9. **Any unclear changes**: "I cannot determine the purpose of change X from the diff"
+10. **FINAL CHECK**: "I will now document all X functional changes without listing files"
+
+**MANDATORY**: The changelog should describe FUNCTIONALITY, not files. But you must capture ALL functionality changes.
+
+**Only after this analysis, write the changelog entry.**
+
+---
+
+## EXAMPLES
+
+### ❌ BAD (Vague, incomplete):
+```markdown
+### New Features
+- Enhanced blockchain processing capabilities
+- Improved error handling and various other updates
+```
+
+### ✅ GOOD (Specific, complete, grouped by crate, with source links):
+```markdown
+### New Features
+#### `brk-core`
+- Added `TransactionAnalyzer` struct with fee calculation and coinbase detection methods ([source](https://github.com/bitcoinresearchkit/brk/blob/v0.0.108/crates/brk-core/src/analyzer.rs))
+- Implemented in-memory caching layer for blockchain queries using HashMap storage
+
+#### `brk-api`
+- Added three new API endpoints: `/api/blocks/{hash}`, `/api/transactions/search`, and `/api/stats/network` ([source](https://github.com/bitcoinresearchkit/brk/blob/v0.0.108/crates/brk-api/src/routes.rs))
+- Implemented standardized error responses with error codes and descriptions
+
+### Bug Fixes
+#### `brk-core`
+- Fixed panic when processing blocks with zero transactions by adding explicit empty block validation ([source](https://github.com/bitcoinresearchkit/brk/blob/v0.0.108/crates/brk-core/src/block.rs))
+
+#### `brk-api`
+- Resolved memory leak in connection pool by implementing proper cleanup in Drop trait
+
+[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.107...v0.0.108)
+```
+
+---
+
+## SUCCESS CRITERIA
+
+✅ **COUNT files modified for internal verification only**
+✅ **Identify EVERY distinct functional change**
+✅ **Document ALL functionality changes (no "other changes")**
+✅ **Changelog describes WHAT users can do, not which files changed**
+✅ **Never mention Cargo.lock or line counts**
+✅ **Use specific, functional descriptions**
+✅ **Complete analysis before writing**
+✅ **Stop and ask for confirmation after each release**
+
+**FAILURE INDICATORS - If you do any of these, you FAILED:**
+❌ "and other changes"
+❌ "various updates"
+❌ "among other improvements"
+❌ **Diff shows new API endpoints but changelog doesn't mention them**
+❌ **Diff shows bug fixes but changelog misses some**
+❌ **Diff shows new structs/functions but changelog ignores them**
+❌ Missing obvious functional changes from the diff
+❌ Summarizing instead of listing each distinct functionality change
+
+**KEY PRINCIPLE**: Count files internally to ensure you don't miss changes, but write about FUNCTIONALITY for users.

.claude/prompts/readmes.md

@@ -0,0 +1,172 @@
+# README Generation Prompt
+
+Generate a professional, comprehensive README.md for each crate based SOLELY on code analysis. Use NO external documentation, commit messages, or existing READMEs.
+
+## MANDATORY PROCESS - FOLLOW EXACTLY:
+1. **IGNORE EXISTING DOCS**: Do NOT read any .md, .txt, .rst, or documentation files in the crate directory
+2. **CODE-ONLY ANALYSIS**: Examine ONLY these files:
+   - All .rs files in src/ directory and subdirectories
+   - Cargo.toml for dependencies and metadata
+   - Code structure and organization
+3. **MANDATORY CODE ANALYSIS**: Before writing ANY README content, you MUST:
+   - Examine all Rust files in src/ directory
+   - Identify the main structs, enums, traits, and functions
+   - Understand the crate's architecture and data flow
+   - Determine the crate's purpose from its implementation
+   - Map dependencies to understand external integrations
+4. **FRESH PERSPECTIVE**: Write the README as if you're the first person to document this crate
+5. Generate one complete README.md per crate
+6. Focus on one crate at a time for thorough analysis
+
+## ABSOLUTE REQUIREMENTS:
+- **SOURCE OF TRUTH**: Use ONLY the actual Rust code - no external docs, comments may provide hints but focus on implementation
+- **CRITICAL**: DO NOT read any existing README.md, CHANGELOG.md, or documentation files
+- **IGNORE ALL TEXT FILES**: .md, .txt, .rst files are FORBIDDEN sources - treat them as if they don't exist
+- **CODE ONLY**: Focus exclusively on .rs files, Cargo.toml, and code structure
+- **PROFESSIONAL GRADE**: Write as if this will be published on crates.io for other developers
+- **PROGRAMMER FOCUSED**: Assume audience knows Rust and relevant domain concepts
+- **IMPLEMENTATION-BASED**: Describe what the code actually does, not what comments claim it should do
+- **If you cannot determine functionality from code alone, state this explicitly**
+
+## README STRUCTURE (MANDATORY):
+
+### 1. CRATE HEADER
+```markdown
+# Crate Name
+
+Brief one-line description of what this crate does (max 80 chars).
+
+[![Crates.io](https://img.shields.io/crates/v/CRATE_NAME.svg)](https://crates.io/crates/CRATE_NAME)
+[![Documentation](https://docs.rs/CRATE_NAME/badge.svg)](https://docs.rs/CRATE_NAME)
+```
+
+### 2. OVERVIEW SECTION
+- **Purpose**: What problem does this crate solve?
+- **Key Features**: 3-5 bullet points of main capabilities (derived from code analysis)
+- **Target Use Cases**: Who would use this and for what?
+
+### 3. INSTALLATION
+```toml
+[dependencies]
+crate_name = "X.Y.Z"
+```
+
+### 4. QUICK START / USAGE
+- **Minimal working example** showing the primary API
+- **Common patterns** observed in the code
+- **Key structs/traits** that users will interact with
+
+### 5. API OVERVIEW
+- **Core Types**: Main structs, enums, traits with brief descriptions
+- **Key Methods**: Most important public functions
+- **Module Structure**: Brief overview of how code is organized
+
+### 6. FEATURES (if applicable)
+- Cargo features and what they enable
+- Optional dependencies and their purpose
+
+### 7. EXAMPLES
+- 2-3 practical code examples showing different use cases
+- Based on public API analysis, not existing examples
+
+## WRITING REQUIREMENTS:
+
+### TONE AND STYLE:
+- **Concise but comprehensive**: Every sentence must add value
+- **Technical precision**: Use exact terminology, avoid marketing speak
+- **Active voice**: "Provides X" not "X is provided"
+- **Present tense**: "The crate handles..." not "The crate will handle..."
+
+### FORBIDDEN PATTERNS:
+- **NEVER** use vague terms: "powerful", "flexible", "robust", "comprehensive", "advanced"
+- **NEVER** write marketing copy: "cutting-edge", "state-of-the-art", "enterprise-grade"
+- **NEVER** make claims you can't verify from code: "blazingly fast", "memory efficient"
+- **NEVER** copy-paste from existing documentation or comments
+- **NEVER** read or reference existing README.md files - pretend they don't exist
+- **NEVER** use phrases like "as mentioned in the documentation" or "according to the docs"
+- **NEVER** let existing documentation influence your analysis or writing
+
+### REQUIRED SPECIFICITY:
+- **Data structures**: Mention specific types (HashMap, Vec, etc.)
+- **Algorithms**: Reference actual implementations found in code
+- **Integration points**: Specific traits implemented, dependencies used
+- **Error handling**: How errors are represented and handled
+- **Async/sync**: Clearly state if operations are blocking or async
+
+### ANTI-BIAS PROTOCOL:
+
+### BEFORE STARTING ANY ANALYSIS:
+1. **Explicitly ignore**: Any README.md, CHANGELOG.md, docs/, documentation files
+2. **File filtering**: Only examine .rs and Cargo.toml files
+3. **Fresh eyes approach**: Analyze the code as if you've never seen this crate before
+4. **Independent thinking**: Form your own understanding purely from code inspection
+
+### IF YOU ACCIDENTALLY READ EXISTING DOCS:
+- Stop immediately and restart your analysis
+- Consciously disregard any information from documentation files
+- Base all descriptions solely on what you observe in the code
+- Ask yourself: "What would I think this code does if I had no documentation?"
+
+### VALIDATION CHECKS:
+- **Unique descriptions**: Your descriptions should differ significantly from any existing docs
+- **Code-derived insights**: Every feature mentioned must be visible in the source code
+- **Independent voice**: Write in your own technical style, not mimicking existing documentation
+- **Fresh examples**: Create new code examples based on API analysis, not existing samples
+
+## CODE ANALYSIS DEPTH:
+**You MUST analyze and understand:**
+1. **Public API surface**: All pub structs, functions, traits, modules
+2. **Core abstractions**: Main data types and their relationships
+3. **Error types**: Custom errors, Result patterns, panic conditions
+4. **Dependencies**: How external crates are integrated
+5. **Feature flags**: Conditional compilation and optional functionality
+6. **Async patterns**: Use of futures, tokio, async-std, etc.
+7. **Serialization**: Serde implementations, custom serialization
+8. **Performance characteristics**: Algorithm complexity where obvious
+
+### EXAMPLE STRUCTURE ANALYSIS OUTPUT:
+```markdown
+## Code Analysis Summary
+**Main Types**: `BlockProcessor`, `Transaction`, `ValidationError`
+**Core Trait**: `Validator` - implemented by `BasicValidator` and `StrictValidator`
+**Async Support**: All processing methods return `impl Future`
+**Error Handling**: Custom `ValidationError` enum with specific error types
+**Dependencies**: Uses `tokio` for async runtime, `serde` for serialization
+**Architecture**: Pipeline pattern with configurable validation stages
+```
+
+## EXAMPLES OF QUALITY:
+
+### ❌ BAD (VAGUE):
+```markdown
+# My Crate
+A powerful and flexible library for blockchain operations.
+
+## Features
+- Fast processing
+- Easy to use
+- Robust error handling
+```
+
+### ✅ GOOD (SPECIFIC):
+```markdown
+# brk-chain-analyzer
+Bitcoin blockchain analysis tools for transaction pattern detection.
+
+## Overview
+Provides utilities for analyzing Bitcoin transaction data, detecting address clustering patterns, and computing blockchain statistics. Built around a streaming parser that processes block data without loading entire blocks into memory.
+
+## Key Types
+- `TransactionAnalyzer`: Stateful analyzer for computing fees, detecting coinbase transactions
+- `ClusterDetector`: Implements common input ownership heuristics for address clustering
+- `BlockStream`: Async iterator over blockchain data with configurable batch sizes
+```
+
+## FINAL REQUIREMENTS:
+- **One README per crate** - don't combine multiple crates
+- **Minimum 200 words** - be thorough but concise
+- **Maximum 800 words** - stay focused and relevant
+- **Code examples must be syntactically correct** and compilable
+- **All claims must be verifiable** from the source code
+
+**PROCESS ONE CRATE AT A TIME. ANALYZE THE CODE THOROUGHLY BEFORE WRITING.**

.github/workflows/release.yml

@@ -64,7 +64,7 @@ jobs:
         # we specify bash to get pipefail; it guards against the `curl` command
         # failing. otherwise `sh` won't catch that `curl` returned non-0
         shell: bash
-        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.29.0/cargo-dist-installer.sh | sh"
+        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.30.0/cargo-dist-installer.sh | sh"
       - name: Cache dist
         uses: actions/upload-artifact@v4
         with:

.gitignore

@@ -4,7 +4,8 @@
 # Builds
 target
 websites/dist
-vecid-to-indexes.js
+bridge/
+/ids.txt
 
 # Copies
 *\ copy*
@@ -12,10 +13,6 @@ vecid-to-indexes.js
 # Ignored
 _*
 
-# Editors
-.vscode
-.zed
-
 # Logs
 .log
 
@@ -28,6 +25,4 @@ flamegraph.svg
 *.trace
 
 # AI
-.claude
+.claude/settings*
-CLAUDE.md
-CLAUDE*.md

.zed/settings.json

@@ -0,0 +1,22 @@
+{
+  "file_scan_exclusions": [
+    // default
+    "**/.git",
+    "**/.svn",
+    "**/.hg",
+    "**/.jj",
+    "**/CVS",
+    "**/.DS_Store",
+    "**/Thumbs.db",
+    "**/.classpath",
+    "**/.settings",
+    // custom
+    "**/lean-qr/*/index.mjs",
+    "**/modern-screenshot/*/index.mjs",
+    "**/solidjs-signals/*/dist/prod.js",
+    "uFuzzy.mjs",
+    "lightweight-charts.standalone.production.mjs"
+    // "scripts/packages",
+    // "dist"
+  ]
+}

Cargo.toml

@@ -4,16 +4,24 @@ members = ["crates/*"]
 package.description = "The Bitcoin Research Kit is a suite of tools designed to extract, compute and display data stored on a Bitcoin Core node"
 package.license = "MIT"
 package.edition = "2024"
-package.version = "0.0.89"
+package.version = "0.0.111"
 package.homepage = "https://bitcoinresearchkit.org"
 package.repository = "https://github.com/bitcoinresearchkit/brk"
 package.readme = "README.md"
 package.rust-version = "1.89"
 
+[profile.dev]
+lto = "thin"
+codegen-units = 16
+opt-level = 2
+split-debuginfo = "unpacked"
+
 [profile.release]
 lto = "fat"
 codegen-units = 1
 panic = "abort"
+strip = true
+overflow-checks = false
 
 [profile.profiling]
 inherits = "release"
@@ -22,40 +30,55 @@ debug = true
 [profile.dist]
 inherits = "release"
 
+[profile.clippy]
+inherits = "dev"
+lto = "off"
+codegen-units = 256
+opt-level = 0
+debug = false
+overflow-checks = false
+panic = "abort"
+debug-assertions = false
+
 [workspace.dependencies]
-axum = "0.8.4"
+allocative = { version = "0.3.4", features = ["parking_lot"] }
+axum = "0.8.6"
 bitcoin = { version = "0.32.7", features = ["serde"] }
 bitcoincore-rpc = "0.19.0"
-brk_bundler = { version = "0.0.89", path = "crates/brk_bundler" }
+brk_binder = { version = "0.0.111", path = "crates/brk_binder" }
-brk_cli = { version = "0.0.89", path = "crates/brk_cli" }
+brk_bundler = { version = "0.0.111", path = "crates/brk_bundler" }
-brk_computer = { version = "0.0.89", path = "crates/brk_computer" }
+brk_cli = { version = "0.0.111", path = "crates/brk_cli" }
-brk_error = { version = "0.0.89", path = "crates/brk_error" }
+brk_computer = { version = "0.0.111", path = "crates/brk_computer" }
-brk_fetcher = { version = "0.0.89", path = "crates/brk_fetcher" }
+brk_error = { version = "0.0.111", path = "crates/brk_error" }
-brk_indexer = { version = "0.0.89", path = "crates/brk_indexer" }
+brk_fetcher = { version = "0.0.111", path = "crates/brk_fetcher" }
-brk_interface = { version = "0.0.89", path = "crates/brk_interface" }
+brk_indexer = { version = "0.0.111", path = "crates/brk_indexer" }
-brk_logger = { version = "0.0.89", path = "crates/brk_logger" }
+brk_interface = { version = "0.0.111", path = "crates/brk_interface" }
-brk_mcp = { version = "0.0.89", path = "crates/brk_mcp" }
+brk_logger = { version = "0.0.111", path = "crates/brk_logger" }
-brk_parser = { version = "0.0.89", path = "crates/brk_parser" }
+brk_mcp = { version = "0.0.111", path = "crates/brk_mcp" }
-brk_server = { version = "0.0.89", path = "crates/brk_server" }
+brk_parser = { version = "0.0.111", path = "crates/brk_parser" }
-brk_store = { version = "0.0.89", path = "crates/brk_store" }
+brk_server = { version = "0.0.111", path = "crates/brk_server" }
-brk_structs = { version = "0.0.89", path = "crates/brk_structs" }
+brk_store = { version = "0.0.111", path = "crates/brk_store" }
+brk_structs = { version = "0.0.111", path = "crates/brk_structs" }
 byteview = "=0.6.1"
 derive_deref = "1.1.1"
 fjall = "2.11.2"
 jiff = "0.2.15"
-log = "0.4.27"
+log = "0.4.28"
-minreq = { version = "2.14.0", features = ["https", "serde_json"] }
+minreq = { version = "2.14.1", features = ["https", "serde_json"] }
 parking_lot = "0.12.4"
+quick_cache = "0.6.16"
 rayon = "1.11.0"
-serde = "1.0.219"
+schemars = "1.0.4"
-serde_bytes = "0.11.17"
+serde = "1.0.228"
-serde_derive = "1.0.219"
+serde_bytes = "0.11.19"
-serde_json = { version = "1.0.143", features = ["float_roundtrip"] }
+serde_derive = "1.0.228"
+serde_json = { version = "1.0.145", features = ["float_roundtrip"] }
+sonic-rs = "0.5.5"
 tokio = { version = "1.47.1", features = ["rt-multi-thread"] }
 # vecdb = { path = "../seqdb/crates/vecdb", features = ["derive"]}
-vecdb = { version = "0.2.4", features = ["derive"]}
+vecdb = { version = "0.2.16", features = ["derive"]}
-zerocopy = "0.8.26"
+zerocopy = "0.8.27"
-zerocopy-derive = "0.8.26"
+zerocopy-derive = "0.8.27"
 
 [workspace.metadata.release]
 shared-version = true
@@ -64,8 +87,8 @@ pre-release-commit-message = "release: v{{version}}"
 tag-message = "release: v{{version}}"
 
 [workspace.metadata.dist]
-cargo-dist-version = "0.29.0"
+cargo-dist-version = "0.30.0"
 ci = "github"
+allow-dirty = ["ci"]
 installers = []
 targets = ["aarch64-apple-darwin", "aarch64-unknown-linux-gnu", "x86_64-unknown-linux-gnu"]
-rust-toolchain-version = "1.89"

README.md

@@ -1,99 +0,0 @@
-# Bitcoin Research Kit
-
-<p align="left">
-  <a href="https://github.com/bitcoinresearchkit/brk">
-    <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
-  </a>
-  <a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
-    <img src="https://img.shields.io/crates/l/brk" alt="License" />
-  </a>
-  <a href="https://crates.io/crates/brk">
-    <img src="https://img.shields.io/crates/v/brk" alt="Version" />
-  </a>
-  <a href="https://docs.rs/brk">
-    <img src="https://img.shields.io/docsrs/brk" alt="Documentation" />
-  </a>
-  <img src="https://img.shields.io/crates/size/brk" alt="Size" />
-  <a href="https://deps.rs/crate/brk">
-    <img src="https://deps.rs/crate/brk/latest/status.svg" alt="Dependency status">
-  </a>
-  <a href="https://discord.gg/HaR3wpH3nr">
-    <img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
-  </a>
-  <a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
-    <img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
-  </a>
-</p>
-
-The Bitcoin Research Kit is a high-performance toolchain designed to parse, index, compute, serve and visualize data from a Bitcoin node, enabling users to gain deeper insights into the Bitcoin network.
-
-In other words it's an alternative to [Glassnode](https://glassnode.com), [mempool.space](https://mempool.space/) (soon) and [electrs](https://github.com/romanz/electrs) (soon) all in one package with a particular focus on simplicity and ease of use.
-
-The toolkit can be used in various ways to accommodate as many needs as possible:
-
-- **[Website](https://bitcoinresearchkit.org)** \
-  Everyone is welcome to visit the official instance and showcase of the suite's capabilities. \
-  It has a wide range of functionalities including charts, tables and simulations which you can visit for free and without the need for an account. \
-  Also available at: [brekit.org](https://brekit.org) // [kibo.money](https://kibo.money) // [satonomics.xyz](https://satonomics.xyz)
-- **[API](https://github.com/bitcoinresearchkit/brk/tree/main/crates/brk_server#brk-server)** \
-  Researchers and developers are free to use BRK's public API with ![Datasets variant count](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fbitcoinresearchkit.org%2Fapi%2Fvecs%2Fvec-count&query=%24&style=flat&label=%20&color=white) dataset variants at their disposal. \
-  Just like the website, it's entirely free, with no authentication or rate-limiting.
-- **[AI](https://github.com/bitcoinresearchkit/brk/blob/main/crates/brk_mcp/README.md#brk-mcp)** \
-  LLMs have to possibility to connect to BRK's backend through a [MCP](https://modelcontextprotocol.io/introduction). \
-  It will give them access to the same tools as the API, with no restrictions, and allow you to have your very own data analysts.
-- **[CLI](https://crates.io/crates/brk_cli)** \
-  Node runners are strongly encouraged to try out and self-host their own instance using BRK's command line interface. \
-  The CLI has multiple cogs available for users to tweak to adapt to all situations with even the possibility for web developers to create their own custom website which could later on be added as an alternative front-end.
-- **[Crates](https://crates.io/crates/brk)** \
-  Rust developers have access to a wide range crates, each built upon one another with its own specific purpose, enabling independent use and offering great flexibility.
-  PRs are welcome, especially if their goal is to introduce additional datasets.
-
-The primary goal of this project is to be fully-featured and accessible for everyone, regardless of their background or financial situation - whether that person is an enthusiast, researcher, miner, analyst, or simply curious.
-
-In contrast, existing alternatives tend to be either [very costly](https://studio.glassnode.com/pricing) or missing essential features, with the vast majority being closed-source and unverifiable, which fundamentally undermines the principles of Bitcoin.
-
-## Hosting as a service
-
-If you'd like to have your own instance hosted for you please contact [hosting@bitcoinresearchkit.org](mailto:hosting@bitcoinresearchkit.org).
-
-- 2 separate dedicated servers (1 GB/s each) with different ISPs and Cloudflare integration for enhanced performance and optimal availability
-- 99.99% SLA
-- Configured for speed
-- Updates delivered at your convenience
-- Direct communication for feature requests and support
-- Bitcoin Core or Knots with desired version
-- Optional subdomains: `*.bitcoinresearchkit.org`, `*.brekit.org`, `*.kibo.money` and `*.satonomics.xyz`
-- Logo featured in the Readme if desired
-
-Pricing: `0.01 BTC / month` *or* `0.1 BTC / year`
-
-## Acknowledgments
-
-Deepest gratitude to the [Open Sats](https://opensats.org/) public charity. Their grant — from December 2024 to the present — has been critical in sustaining this project.
-
-Heartfelt thanks go out to every donor on [Nostr](https://primal.net/p/npub1jagmm3x39lmwfnrtvxcs9ac7g300y3dusv9lgzhk2e4x5frpxlrqa73v44) and [Geyser.fund](https://geyser.fund/project/brk) whose support has ensured the availability of the [bitcoinresearchkit.org](https://bitcoinresearchkit.org) public instance.
-
-## Crates
-
-- [`brk`](https://crates.io/crates/brk): A wrapper around all other `brk-*` crates
-- [`brk_bundler`](https://crates.io/crates/brk_bundler): A thin wrapper around [`rolldown`](https://rolldown.rs/)
-- [`brk_cli`](https://crates.io/crates/brk_cli): A command line interface to run a BRK instance
-- [`brk_computer`](https://crates.io/crates/brk_computer): A Bitcoin dataset computer built on top of [`brk_indexer`](https://crates.io/crates/brk_indexer)
-- [`brk_error`](https://crates.io/crates/brk_error): Errors used throughout BRK
-- [`brk_fetcher`](https://crates.io/crates/brk_fetcher): A Bitcoin price fetcher
-- [`brk_indexer`](https://crates.io/crates/brk_indexer): A Bitcoin indexer built on top of [`brk_parser`](https://crates.io/crates/brk_parser)
-- [`brk_interface`](https://crates.io/crates/brk_interface): An interface to find and format data from BRK
-- [`brk_logger`](https://crates.io/crates/brk_logger): A thin wrapper around [`env_logger`](https://crates.io/crates/env_logger)
-- [`brk_mcp`](https://crates.io/crates/brk_mcp): A bridge for LLMs to access BRK
-- [`brk_parser`](https://crates.io/crates/brk_parser): A very fast Bitcoin block parser and iterator built on top of [`bitcoin-rust`](https://crates.io/crates/bitcoin)
-- [`brk_server`](https://crates.io/crates/brk_server): A server with an API for anything from BRK
-- [`brk_store`](https://crates.io/crates/brk_store): A thin wrapper around [`fjall`](https://crates.io/crates/fjall)
-- [`brk_structs`](https://crates.io/crates/brk_structs): Structs used throughout BRK
-
-## Donate
-
-[`bc1q09 8zsm89 m7kgyz e338vf ejhpdt 92ua9p 3peuve`](bitcoin:bc1q098zsm89m7kgyze338vfejhpdt92ua9p3peuve)
-
-[`lnurl1dp68gurn8ghj7ampd3kx2ar0veekzar0wd5xjtnrdakj7tnhv4kxctttdehhwm30d3h82unvwqhkxmmww3jkuar8d35kgetj8yuq363hv4`](lightning:lnurl1dp68gurn8ghj7ampd3kx2ar0veekzar0wd5xjtnrdakj7tnhv4kxctttdehhwm30d3h82unvwqhkxmmww3jkuar8d35kgetj8yuq363hv4)
-
-[Geyser Fund](https://geyser.fund/project/brk)

TODO.md

@@ -1,99 +0,0 @@
-# TODO
-
-- __crates__
-  - _cli_
-    - check disk space on first launch
-    - add custom path support for config.toml
-    - maybe add bitcoind download and launch support
-      - via: https://github.com/rust-bitcoin/corepc/blob/master/node
-    - test read/write speed, add warning if too low (<2gb/s)
-    - pull latest version and notify is out of date
-  - _computer_
-    - **add rollback of states (in stateful)**
-    - remove configurable format (raw/compressed) and chose sane ones instead
-      - linear reads: compressed (height/date/... + txindex_to_height + txindex_to_version + ...)
-      - random reads: raw (outputindex_to_value + ...)
-    - add prices paid by percentile (percentile cost basis) back
-    - add support for per index computation
-    - fix min feerate which is always ZERO due to coinbase transaction
-    - before computing multiple sources check their length, panic if not equal
-    - add oracle price dataset (https://utxo.live/oracle/UTXOracle.py)
-    - add address counts relative to all datasets
-    - make decade, quarter, year datasets `computed` instead of `eager`
-    - add 6 months (semester) interval datasets to builder
-    - some datasets in `indexes` can probably be removed
-    - add revived/sent supply datasets
-    - add `in-sats` version of all price datasets (average and co)
-    - add `p2pk` group (sum of `p2pk33` and `p2pk65`)
-    - add chopiness datasets
-    - add utxo count, address count, supply data for  by reused addresses in groups by address type
-    - add more date ranges (3-6 months and more)
-    - add puell multiple dataset
-    - add pi cycle dataset
-    - add emas of price
-    - add 7d and 30d ema to sell side risk ratio and sopr
-    - don't compute everything for all cohorts as some datasets combinations are irrelevant
-      - addresses/utxos by amount don't need mvrvz for example
-    - add all possible charts from:
-      - https://mainnet.observer
-      - https://glassnode.com
-      - https://checkonchain.com
-      - https://researchbitcoin.net/exciting-update-coming-to-the-bitcoin-lab/
-      - https://mempool.space/research
-  - _indexer_
-    - parse only the needed block number
-      - maybe using https://developer.bitcoin.org/reference/rpc/getblockhash.html
-  - _interface_
-    - create pagination enum
-      - from to
-      - from option<count>
-      - to option<count>
-      - page + option<per page> default 1000 max 1000
-    - from/to/count params don’t cap all combinations
-      - example: from -10,000 count 10, won’t work if underlying vec isn’t 10k or more long
-  - _parser_
-    - save `vec` file instead of `json`
-    - support lock file, process in read only if already opened in write mode
-    - if less than X (10 maybe ?) get block using rpc instead of parsing the block files
-  - _server_
-    - api
-      - add extensions support (.json .csv …)
-      - if format instead of extension then don't download file
-    - add support for https (rustls)
-- __docs__
-  - _README_
-    - add a comparison table with alternatives
-    - add contribution section where help is needed
-      - documentation/mcp/datasets/different front ends
-    - add faq
-- __websites__
-  - _default_
-    - explorer
-      - blocks
-      - transactions
-      - addresses
-      - miners
-      - maybe xpubs
-    - charts
-      - improve some names and colors
-      - remove `sum` series when it's a duplicate of the `base` (in subsidy for example)
-      - selected unit sometimes changes when going back end forth
-      - add support for custom charts
-      - separate z-score charts from "realized price" (with their own prices), have 4y, 2y and 1y
-      - price scale format depends on unit, hide digits for sats for example (if/when possible)
-    - table
-      - pagination
-      - exports (.json, .csv,…)
-    - search
-      - datasets add legend, and keywords ?
-      - height/address/txid
-    - api
-      - add api page with interactivity
-    - global
-      - **fix navigation/history**
-      - move share button to footer ?
-      - Use `ichart.createPane()` in wrapper
-      - improve behavior when local storage is unavailable
-        - by having a global state
-- __global__
-  - check `TODO`s in codebase

crates/brk/Cargo.toml

@@ -2,7 +2,6 @@
 name = "brk"
 description.workspace = true
 license.workspace = true
-readme.workspace = true
 homepage.workspace = true
 repository.workspace = true
 edition.workspace = true
@@ -12,6 +11,7 @@ build = "build.rs"
 
 [features]
 full = [
+    "binder",
     "bundler",
     "computer",
     "error",
@@ -25,6 +25,7 @@ full = [
     "store",
     "structs",
 ]
+binder = ["brk_binder"]
 bundler = ["brk_bundler"]
 computer = ["brk_computer"]
 error = ["brk_error"]
@@ -39,6 +40,7 @@ store = ["brk_store"]
 structs = ["brk_structs"]
 
 [dependencies]
+brk_binder = { workspace = true, optional = true }
 brk_bundler = { workspace = true, optional = true }
 brk_cli = { workspace = true }
 brk_computer = { workspace = true, optional = true }

crates/brk/README.md

@@ -1,197 +1,273 @@
 # brk
 
-**Main wrapper crate for the Bitcoin Research Kit (BRK)**
+Unified Bitcoin Research Kit crate providing optional feature-gated access to all BRK components.
 
-The `brk` crate serves as the primary entry point for the Bitcoin Research Kit, providing a unified interface to all BRK components through feature flags. It enables developers to selectively include only the components they need while maintaining a consistent API.
+[![Crates.io](https://img.shields.io/crates/v/brk.svg)](https://crates.io/crates/brk)
+[![Documentation](https://docs.rs/brk/badge.svg)](https://docs.rs/brk)
 
-## What it provides
+## Overview
 
-- **Unified Access**: Single crate providing access to the entire BRK ecosystem
+This crate serves as a unified entry point to the Bitcoin Research Kit ecosystem, providing feature-gated re-exports of all BRK components. It allows users to selectively include only the functionality they need while maintaining a single dependency declaration, with the `brk_cli` component always available for command-line interface access.
-- **Feature-based Selection**: Choose only the components you need
-- **Consistent Versioning**: All components versioned together for compatibility
-- **Simplified Dependencies**: Single dependency instead of multiple individual crates
 
-## Available Components
+**Key Features:**
 
-### Core Data Pipeline
+- Feature-gated modular access to 12 specialized BRK components
-- **`parser`** ([brk_parser](../brk_parser/)) - High-performance Bitcoin block parser
+- Single dependency entry point with selective compilation
-- **`indexer`** ([brk_indexer](../brk_indexer/)) - Blockchain data indexer with dual storage
+- Always-available CLI component for command-line operations
-- **`computer`** ([brk_computer](../brk_computer/)) - Analytics engine for computed datasets
+- Comprehensive documentation aggregation with inline re-exports
-- **`interface`** ([brk_interface](../brk_interface/)) - Unified data query and formatting API
+- `full` feature for complete BRK functionality inclusion
+- Optimized build configuration with docs.rs integration
 
-### Infrastructure
+**Target Use Cases:**
-- **`structs`** ([brk_structs](../brk_structs/)) - Bitcoin-aware type system and data structures
-- **`error`** ([brk_error](../brk_error/)) - Centralized error handling
-- **`store`** ([brk_store](../brk_store/)) - Blockchain-aware key-value storage
 
-### External Integration
+- Applications requiring selective BRK functionality to minimize dependencies
-- **`fetcher`** ([brk_fetcher](../brk_fetcher/)) - Bitcoin price data fetcher with multi-source fallback
+- Library development where only specific Bitcoin analysis components are needed
-- **`server`** ([brk_server](../brk_server/)) - HTTP server with REST API
+- Prototyping and experimentation with different BRK component combinations
-- **`mcp`** ([brk_mcp](../brk_mcp/)) - Model Context Protocol for LLM integration
+- Educational use cases demonstrating modular blockchain analytics architecture
 
-### Utilities
+## Installation
-- **`cli`** ([brk_cli](../brk_cli/)) - Command line interface (always enabled)
-- **`logger`** ([brk_logger](../brk_logger/)) - Logging utilities
-- **`bundler`** ([brk_bundler](../brk_bundler/)) - Asset bundling for web interfaces
-
-## Usage
-
-### Full Installation
 
 ```toml
 [dependencies]
-brk = { version = "0.0.88", features = ["full"] }
+# Minimal installation with CLI only
+brk = "0.0.107"
+
+# Full functionality
+brk = { version = "0.0.107", features = ["full"] }
+
+# Selective features
+brk = { version = "0.0.107", features = ["indexer", "computer", "server"] }
 ```
 
+## Quick Start
+
 ```rust
-use brk::*;
+// CLI is always available
+use brk::cli;
 
-// Access all components
+// Feature-gated components
-let config = cli::Config::load()?;
+#[cfg(feature = "indexer")]
-let parser = parser::Parser::new(/* ... */);
+use brk::indexer::Indexer;
-let indexer = indexer::Indexer::forced_import("./data")?;
+
-let computer = computer::Computer::forced_import("./data", &indexer, None)?;
+#[cfg(feature = "computer")]
+use brk::computer::Computer;
+
+#[cfg(feature = "server")]
+use brk::server::Server;
+
+// Build complete pipeline with selected features
+#[cfg(all(feature = "indexer", feature = "computer", feature = "server"))]
+fn build_pipeline() -> Result<(), Box<dyn std::error::Error>> {
+    let indexer = Indexer::build("./data")?;
+    let computer = Computer::build("./analytics", &indexer)?;
+    let interface = brk::interface::Interface::build(&indexer, &computer);
+    let server = Server::new(interface, None);
+    Ok(())
+}
 ```
 
-### Selective Components
+## API Overview
 
-```toml
+### Feature Organization
-[dependencies]
+
-brk = { version = "0.0.88", features = ["parser", "indexer", "computer"] }
+The crate provides feature-gated access to BRK components organized by functionality:
-```
+
+**Core Data Processing:**
+- `structs` - Bitcoin-aware data structures and type system
+- `error` - Centralized error handling across components
+- `store` - Transactional key-value storage wrapper
+
+**Blockchain Processing:**
+- `parser` - Multi-threaded Bitcoin block parsing
+- `indexer` - Blockchain data indexing with columnar storage
+- `computer` - Analytics computation engine
+
+**Data Access:**
+- `interface` - Unified query interface with fuzzy search
+- `fetcher` - Multi-source price data aggregation
+
+**Service Layer:**
+- `server` - HTTP API server with caching and compression
+- `mcp` - Model Context Protocol bridge for LLM integration
+- `logger` - Enhanced logging with colored output
+
+**Web Infrastructure:**
+- `bundler` - Web asset bundling using Rolldown
+
+### Always Available
+
+**`cli`**: Command-line interface module (no feature gate required)
+Provides access to the complete BRK command-line interface for running full instances.
+
+## Examples
+
+### Minimal Bitcoin Parser
 
 ```rust
-use brk::{parser, indexer, computer};
+use brk::parser::Parser;
 
-// Core data pipeline only
+fn parse_blocks() -> Result<(), Box<dyn std::error::Error>> {
-let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
+    #[cfg(feature = "parser")]
-let mut indexer = indexer::Indexer::forced_import(output_dir)?;
+    {
-let mut computer = computer::Computer::forced_import(output_dir, &indexer, None)?;
+        let parser = Parser::new("/path/to/blocks", None, rpc_client);
+        // Parse blockchain data
+        Ok(())
+    }
+    #[cfg(not(feature = "parser"))]
+    {
+        Err("Parser feature not enabled".into())
+    }
+}
 ```
 
-### Minimal Setup
+### Analytics Pipeline
-
-```toml
-[dependencies]
-brk = { version = "0.0.88", features = ["structs", "parser"] }
-```
-
-```rust
-use brk::{structs, parser};
-
-// Just parsing and types
-let height = structs::Height::new(800_000);
-let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
-```
-
-## Feature Flags
-
-| Feature | Description | Dependencies |
-|---------|-------------|--------------|
-| `full` | Enable all components | All crates |
-| `cli` | Command line interface | Always enabled |
-| `structs` | Core type system | Foundation for other crates |
-| `error` | Error handling | Used by most crates |
-| `parser` | Block parsing | `structs`, `error` |
-| `store` | Key-value storage | `structs`, `error` |
-| `indexer` | Blockchain indexing | `parser`, `store` |
-| `computer` | Analytics computation | `indexer`, `fetcher` (optional) |
-| `fetcher` | Price data fetching | `structs`, `error` |
-| `interface` | Data query API | `indexer`, `computer` |
-| `server` | HTTP server | `interface`, `mcp` |
-| `mcp` | LLM integration | `interface` |
-| `logger` | Logging utilities | Standalone |
-| `bundler` | Asset bundling | Standalone |
-
-## Common Usage Patterns
-
-### Complete BRK Instance
-
-```rust
-use brk::*;
-
-// Full data pipeline setup
-let config = cli::Config::load()?;
-let rpc = /* Bitcoin Core RPC client */;
-let parser = parser::Parser::new(config.blocks_dir, config.output_dir, rpc);
-let mut indexer = indexer::Indexer::forced_import(&config.output_dir)?;
-let mut computer = computer::Computer::forced_import(&config.output_dir, &indexer, None)?;
-let interface = interface::Interface::build(&indexer, &computer);
-let server = server::Server::new(interface, config.website_path);
-
-// Start server
-server.serve(true).await?;
-```
-
-### Data Analysis
 
 ```rust
 use brk::{indexer, computer, interface};
 
-// Analysis-focused setup
+#[cfg(all(feature = "indexer", feature = "computer", feature = "interface"))]
-let indexer = indexer::Indexer::forced_import("./brk_data")?;
+fn analytics_pipeline() -> Result<(), Box<dyn std::error::Error>> {
-let computer = computer::Computer::forced_import("./brk_data", &indexer, None)?;
+    // Initialize indexer
-let interface = interface::Interface::build(&indexer, &computer);
+    let indexer = indexer::Indexer::build("./blockchain_data")?;
 
-// Query data
+    // Compute analytics
-let params = interface::Params {
+    let computer = computer::Computer::build("./analytics", &indexer)?;
-    index: interface::Index::Height,
-    ids: vec!["price_usd", "difficulty"].into(),
-    rest: interface::ParamsOpt::default()
-        .set_from(-100)
-        .set_format(interface::Format::CSV),
-};
 
-let csv_data = interface.search_and_format(params)?;
+    // Create query interface
-```
+    let interface = interface::Interface::build(&indexer, &computer);
 
-### Custom Integration
+    // Query latest price
+    let params = interface::Params {
+        index: "date".to_string(),
+        ids: vec!["price-close".to_string()].into(),
+        from: Some(-1),
+        ..Default::default()
+    };
 
-```rust
+    let result = interface.search_and_format(params)?;
-use brk::{structs, parser, error};
+    println!("Latest Bitcoin price: {:?}", result);
-
-// Custom application with BRK components
-fn analyze_blocks() -> error::Result<()> {
-    let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
-    
-    parser.parse(None, None)
-        .iter()
-        .take(1000)  // First 1000 blocks
-        .for_each(|(height, block, hash)| {
-            println!("Block {}: {} transactions", height, block.txdata.len());
-        });
 
     Ok(())
 }
 ```
 
-## Version Compatibility
+### Web Server Setup
 
-All BRK crates are released together with synchronized versions. When using the `brk` wrapper crate, you're guaranteed compatibility between all components.
+```rust
+use brk::{server, interface, indexer, computer};
 
-- **Current version**: 0.0.88
+#[cfg(all(feature = "server", feature = "interface", feature = "indexer", feature = "computer"))]
-- **Rust MSRV**: 1.89+
+async fn web_server() -> Result<(), Box<dyn std::error::Error>> {
-- **Bitcoin Core**: v25.0 - v29.0
+    let indexer = indexer::Indexer::build("./data")?;
+    let computer = computer::Computer::build("./analytics", &indexer)?;
+    let interface = interface::Interface::build(&indexer, &computer);
 
-## Performance Characteristics
+    let server = server::Server::new(interface, None);
+    server.serve(true).await?;
 
-The `brk` crate itself adds no runtime overhead - it simply re-exports the underlying crates. Performance characteristics depend on which components you use:
+    Ok(())
+}
+```
 
-- **Full pipeline**: ~13-15 hours initial sync, ~40GB storage overhead
+### MCP Integration
-- **Parser only**: ~4 minutes to parse entire blockchain
-- **Indexer only**: ~7-8 hours to index blockchain, ~5-6GB RAM usage
-- **Server**: Low latency API responses with caching and compression
 
-## Dependencies
+```rust
+use brk::{mcp, interface, indexer, computer};
 
-The `brk` crate's dependencies are determined by enabled features. Core dependencies include:
+#[cfg(all(feature = "mcp", feature = "interface"))]
+fn mcp_server(interface: &'static interface::Interface) -> mcp::MCP {
+    mcp::MCP::new(interface)
+}
+```
 
-- `brk_cli` - Always included for configuration and CLI support
+## Feature Combinations
-- Individual `brk_*` crates based on enabled features
-- Transitive dependencies from enabled components
 
-For specific dependency information, see individual crate READMEs.
+### Common Combinations
+
+**Data Processing**: `["structs", "parser", "indexer"]`
+Basic blockchain data processing and indexing.
+
+**Analytics**: `["indexer", "computer", "fetcher", "interface"]`
+Complete analytics pipeline with price data integration.
+
+**API Server**: `["interface", "server", "logger"]`
+HTTP API server with logging capabilities.
+
+**Full Stack**: `["full"]`
+All components for complete BRK functionality.
+
+### Dependency Optimization
+
+Feature selection allows for significant dependency reduction:
+
+```toml
+# Minimal parser-only dependency
+brk = { version = "0.0.107", features = ["parser"] }
+
+# Analytics without web server
+brk = { version = "0.0.107", features = ["indexer", "computer", "interface"] }
+
+# Web server without parsing
+brk = { version = "0.0.107", features = ["interface", "server"] }
+```
+
+## Architecture
+
+### Re-export Pattern
+
+The crate uses `#[doc(inline)]` re-exports to provide seamless access to component APIs:
+
+```rust
+#[cfg(feature = "component")]
+#[doc(inline)]
+pub use brk_component as component;
+```
+
+This pattern ensures:
+- Feature-gated compilation for dependency optimization
+- Inline documentation for unified API reference
+- Namespace preservation for component-specific functionality
+
+### Build Configuration
+
+- **Documentation**: `all-features = true` for complete docs.rs documentation
+- **CLI Integration**: `brk_cli` always available without feature gates
+- **Optional Dependencies**: All components except CLI are optional
+
+## Configuration
+
+### Feature Flags
+
+| Feature | Component | Description |
+|---------|-----------|-------------|
+| `bundler` | `brk_bundler` | Web asset bundling |
+| `computer` | `brk_computer` | Analytics computation |
+| `error` | `brk_error` | Error handling |
+| `fetcher` | `brk_fetcher` | Price data fetching |
+| `indexer` | `brk_indexer` | Blockchain indexing |
+| `interface` | `brk_interface` | Data query interface |
+| `logger` | `brk_logger` | Enhanced logging |
+| `mcp` | `brk_mcp` | Model Context Protocol |
+| `parser` | `brk_parser` | Block parsing |
+| `server` | `brk_server` | HTTP server |
+| `store` | `brk_store` | Key-value storage |
+| `structs` | `brk_structs` | Data structures |
+| `full` | All components | Complete functionality |
+
+### Documentation
+
+Documentation is aggregated from all components with `#![doc = include_str!("../README.md")]` ensuring comprehensive API reference across all features.
+
+## Code Analysis Summary
+
+**Main Structure**: Feature-gated re-export crate providing unified access to 12 BRK components \
+**Feature System**: Cargo features enabling selective compilation and dependency optimization \
+**CLI Integration**: Always-available `brk_cli` access without feature requirements \
+**Documentation**: Inline re-exports with comprehensive docs.rs integration \
+**Dependency Management**: Optional dependencies for all components except CLI \
+**Build Configuration**: Optimized compilation with all-features documentation \
+**Architecture**: Modular aggregation crate enabling flexible BRK ecosystem usage
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk/src/lib.rs

@@ -1,5 +1,9 @@
 #![doc = include_str!("../README.md")]
 
+#[cfg(feature = "binder")]
+#[doc(inline)]
+pub use brk_binder as binder;
+
 #[cfg(feature = "bundler")]
 #[doc(inline)]
 pub use brk_bundler as bundler;

crates/brk_binder/Cargo.toml

@@ -0,0 +1,14 @@
+[package]
+name = "brk_binder"
+description = "A generator of binding files for other languages"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+homepage.workspace = true
+repository.workspace = true
+rust-version.workspace = true
+build = "build.rs"
+
+[dependencies]
+brk_interface = { workspace = true }
+brk_structs = { workspace = true }

crates/brk_binder/README.md

@@ -0,0 +1 @@
+# brk_binder

crates/brk_binder/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    let profile = std::env::var("PROFILE").unwrap_or_default();
+
+    if profile == "release" {
+        println!("cargo:rustc-flag=-C");
+        println!("cargo:rustc-flag=target-cpu=native");
+    }
+}

crates/brk_binder/src/js.rs

@@ -0,0 +1,250 @@
+use std::{
+    collections::{BTreeMap, HashMap},
+    fs, io,
+    path::Path,
+};
+
+use brk_interface::{Index, Interface};
+use brk_structs::pools;
+
+use super::VERSION;
+
+const AUTO_GENERATED_DISCLAIMER: &str = "//
+// File auto-generated, any modifications will be overwritten
+//";
+
+#[allow(clippy::upper_case_acronyms)]
+pub trait Bridge {
+    fn generate_js_files(&self, modules_path: &Path) -> io::Result<()>;
+}
+
+impl Bridge for Interface<'static> {
+    fn generate_js_files(&self, modules_path: &Path) -> io::Result<()> {
+        let path = modules_path.join("brk-client");
+
+        if !fs::exists(&path)? {
+            return Ok(());
+        }
+
+        let path = path.join("generated");
+        fs::create_dir_all(&path)?;
+
+        generate_version_file(&path)?;
+        generate_metrics_file(self, &path)?;
+        generate_pools_file(&path)
+    }
+}
+
+fn generate_version_file(parent: &Path) -> io::Result<()> {
+    let path = parent.join(Path::new("version.js"));
+
+    let contents = format!(
+        "{AUTO_GENERATED_DISCLAIMER}
+
+export const VERSION = \"v{VERSION}\";
+"
+    );
+
+    fs::write(path, contents)
+}
+
+fn generate_pools_file(parent: &Path) -> io::Result<()> {
+    let path = parent.join(Path::new("pools.js"));
+
+    let pools = pools();
+
+    let mut contents = format!("{AUTO_GENERATED_DISCLAIMER}\n");
+
+    contents += "
+/**
+ * @typedef {typeof POOL_ID_TO_POOL_NAME} PoolIdToPoolName
+ * @typedef {keyof PoolIdToPoolName} PoolId
+ */
+
+export const POOL_ID_TO_POOL_NAME = /** @type {const} */ ({
+";
+
+    let mut sorted_pools: Vec<_> = pools.iter().collect();
+    sorted_pools.sort_by(|a, b| a.name.to_lowercase().cmp(&b.name.to_lowercase()));
+
+    contents += &sorted_pools
+        .iter()
+        .map(|pool| {
+            let id = pool.serialized_id();
+            format!("  {id}: \"{}\",", pool.name)
+        })
+        .collect::<Vec<_>>()
+        .join("\n");
+
+    contents += "\n});\n";
+
+    fs::write(path, contents)
+}
+
+fn generate_metrics_file(interface: &Interface<'static>, parent: &Path) -> io::Result<()> {
+    let path = parent.join(Path::new("metrics.js"));
+
+    let indexes = Index::all();
+
+    let mut contents = format!(
+        "{AUTO_GENERATED_DISCLAIMER}
+
+export const INDEXES = /** @type {{const}} */ ([
+{}
+]);
+
+/**
+ * @typedef {{typeof INDEXES[number]}} IndexName
+ */
+",
+        indexes
+            .iter()
+            .map(|i| format!("  \"{}\"", i.serialize_long()))
+            .collect::<Vec<_>>()
+            .join(",\n")
+    );
+
+    // contents += &indexes
+    //     .iter()
+    //     .map(|i| format!(" * @typedef {{\"{}\"}} {i}", i.serialize_long()))
+    //     .collect::<Vec<_>>()
+    //     .join("\n");
+
+    //     contents += &format!(
+    //         "
+    //  * @typedef {{{}}} Index
+    //  */
+    // ",
+    //         indexes
+    //             .iter()
+    //             .map(|i| i.to_string())
+    //             .collect::<Vec<_>>()
+    //             .join(" | ")
+    //     );
+
+    let mut unique_index_groups = BTreeMap::new();
+
+    let mut word_to_freq: BTreeMap<_, usize> = BTreeMap::new();
+    interface
+        .metric_to_index_to_vec()
+        .keys()
+        .for_each(|metric| {
+            metric.split("_").for_each(|word| {
+                *word_to_freq.entry(word).or_default() += 1;
+            });
+        });
+    let mut word_to_freq = word_to_freq.into_iter().collect::<Vec<_>>();
+    word_to_freq.sort_unstable_by(|a, b| b.1.cmp(&a.1).then(a.0.cmp(b.0)));
+    let words = word_to_freq
+        .into_iter()
+        .map(|(str, _)| str)
+        .collect::<Vec<_>>();
+
+    contents += &format!(
+        "
+export const INDEX_TO_WORD = [
+  {}
+];
+
+",
+        words
+            .iter()
+            .enumerate()
+            .map(|(index, word)| format!("\"{word}\", // {}", index_to_letters(index)))
+            .collect::<Vec<_>>()
+            .join("\n  ")
+    );
+
+    let word_to_base62 = words
+        .into_iter()
+        .enumerate()
+        .map(|(i, w)| (w, index_to_letters(i)))
+        .collect::<HashMap<_, _>>();
+
+    let mut ser_metric_to_indexes = "
+/** @type {Record<string, IndexName[]>} */
+export const COMPRESSED_METRIC_TO_INDEXES = {
+"
+    .to_string();
+
+    interface
+        .metric_to_index_to_vec()
+        .iter()
+        .for_each(|(metric, index_to_vec)| {
+            let indexes = index_to_vec
+                .keys()
+                .map(|i| format!("\"{}\"", i.serialize_long()))
+                .collect::<Vec<_>>()
+                .join(", ");
+            let indexes = format!("[{indexes}]");
+            let unique = unique_index_groups.len();
+            let index = index_to_letters(*unique_index_groups.entry(indexes).or_insert(unique));
+
+            let compressed_metric = metric.split('_').fold(String::new(), |mut acc, w| {
+                if !acc.is_empty() {
+                    acc.push('_');
+                }
+                acc.push_str(&word_to_base62[w]);
+                acc
+            });
+
+            ser_metric_to_indexes += &format!("  {compressed_metric}: {index},\n");
+        });
+
+    ser_metric_to_indexes += "};
+";
+
+    let mut sorted_groups: Vec<_> = unique_index_groups.into_iter().collect();
+    sorted_groups.sort_by_key(|(_, index)| *index);
+    sorted_groups.into_iter().for_each(|(group, index)| {
+        let index = index_to_letters(index);
+        contents += &format!("/** @type {{IndexName[]}} */\nconst {index} = {group};\n");
+    });
+
+    contents += &ser_metric_to_indexes;
+
+    fs::write(path, contents)
+}
+
+fn index_to_letters(mut index: usize) -> String {
+    if index < 52 {
+        return (index_to_char(index) as char).to_string();
+    }
+    let mut result = [0u8; 8];
+    let mut pos = 8;
+    loop {
+        pos -= 1;
+        result[pos] = index_to_char(index % 52);
+        index /= 52;
+        if index == 0 {
+            break;
+        }
+        index -= 1;
+    }
+    unsafe { String::from_utf8_unchecked(result[pos..].to_vec()) }
+}
+
+fn index_to_char(index: usize) -> u8 {
+    match index {
+        0..=25 => b'A' + index as u8,
+        26..=51 => b'a' + (index - 26) as u8,
+        _ => unreachable!(),
+    }
+}
+
+// fn letters_to_index(s: &str) -> usize {
+//     let mut result = 0;
+//     for byte in s.bytes() {
+//         let value = char_to_index(byte) as usize;
+//         result = result * 52 + value + 1;
+//     }
+//     result - 1
+// }
+
+// fn char_to_index(byte: u8) -> u8 {
+//     match byte {
+//         b'A'..=b'Z' => byte - b'A',
+//         b'a'..=b'z' => byte - b'a' + 26,
+//         _ => 255, // Invalid
+//     }
+// }

crates/brk_binder/src/lib.rs

@@ -0,0 +1,5 @@
+mod js;
+
+pub use js::Bridge;
+
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");

crates/brk_binder/src/python.rs

@@ -0,0 +1 @@
+// TODO ?

crates/brk_bundler/Cargo.toml

@@ -12,7 +12,7 @@ build = "build.rs"
 [dependencies]
 log = { workspace = true }
 notify = "8.2.0"
-brk_rolldown = "0.1.4"
+brk_rolldown = "0.2.3"
 # brk_rolldown = { path = "../../../rolldown/crates/rolldown"}
 sugar_path = "1.2.0"
 tokio = { workspace = true }

crates/brk_bundler/README.md

@@ -1,186 +1,278 @@
 # brk_bundler
 
-**Asset bundling for BRK web interfaces using Rolldown**
+Asset bundling and development server for BRK web interfaces with hot reloading and file watching.
 
-`brk_bundler` provides JavaScript/TypeScript bundling capabilities for BRK's web interfaces. It's a thin wrapper around Rolldown (Rust-based Rollup alternative) with BRK-specific configuration for building optimized web assets with file watching and automatic rebuilding.
+[![Crates.io](https://img.shields.io/crates/v/brk_bundler.svg)](https://crates.io/crates/brk_bundler)
+[![Documentation](https://docs.rs/brk_bundler/badge.svg)](https://docs.rs/brk_bundler)
 
-## What it provides
+## Overview
 
-- **JavaScript Bundling**: Modern ES modules bundling with minification
+This crate provides a thin wrapper around the Rolldown JavaScript bundler specifically designed for BRK web interface development. It handles asset bundling, file copying, template processing, and development-mode file watching with automatic rebuilds and hot reloading for efficient web development workflows.
-- **File Watching**: Automatic rebuilding on source file changes
-- **Asset Processing**: Copies and processes static assets
-- **Version Injection**: Automatic version string replacement in service workers
-- **Development Mode**: Live rebuilding for rapid development
 
-## Key Features
+**Key Features:**
 
-### Bundling Capabilities
+- JavaScript bundling with Rolldown (Rust-based bundler)
-- **ES Module Support**: Modern JavaScript bundling with tree-shaking
+- Automatic file watching and hot reloading in development mode
-- **Minification**: Automatic code minification for production builds
+- Template processing with version injection and asset hash replacement
-- **Source Maps**: Generated source maps for debugging
+- Service worker generation with version management
-- **Entry Point Processing**: Configurable entry points with hashed output names
+- Source map generation for debugging
+- Minification for production builds
+- Async/await support with Tokio integration
 
-### File System Operations
+**Target Use Cases:**
-- **Directory Copying**: Copies entire source directories to distribution
-- **Selective Processing**: Special handling for specific file types
-- **Path Resolution**: Automatic path resolution and asset linking
 
-### Development Features
+- BRK blockchain explorer web interfaces
-- **Hot Rebuilding**: Automatic rebuilds on file changes
+- Development of Bitcoin analytics dashboards
-- **Watch Mode**: Monitors source files and triggers rebuilds
+- Building responsive web applications for blockchain data visualization
-- **Version Replacement**: Injects build version into service workers
+- Hot reloading development environment for rapid iteration
 
-## Usage
+## Installation
 
-### Basic Bundling
+```bash
+cargo add brk_bundler
+```
+
+## Quick Start
 
 ```rust
 use brk_bundler::bundle;
 use std::path::Path;
 
-// Bundle without watching (production)
+#[tokio::main]
-let websites_path = Path::new("./websites");
+async fn main() -> std::io::Result<()> {
-let source_folder = "default";
+    let websites_path = Path::new("./web");
-let dist_path = bundle(websites_path, source_folder, false).await?;
+    let source_folder = "src";
+    let watch = true; // Enable hot reloading
 
-println!("Bundled to: {:?}", dist_path);
+    // Bundle assets and start development server
-```
+    let dist_path = bundle(websites_path, source_folder, watch).await?;
 
-### Development Mode with Watching
+    println!("Assets bundled to: {}", dist_path.display());
 
-```rust
+    // Keep running for file watching (in watch mode)
-// Bundle with file watching (development)
+    if watch {
-let dist_path = bundle(websites_path, "default", true).await?;
+        tokio::signal::ctrl_c().await?;
+    }
 
-// Bundler now watches for changes and rebuilds automatically
+    Ok(())
-// This will run in the background until the process exits
-```
-
-### Integration with BRK CLI
-
-```rust
-// Typically called from brk_cli when serving websites
-async fn setup_website(config: &Config) -> Result<PathBuf> {
-    let websites_path = config.websites_path();
-    let source_folder = match config.website_mode {
-        WebsiteMode::Default => "default",
-        WebsiteMode::Custom => "custom",
-        WebsiteMode::None => return Ok(PathBuf::new()),
-    };
-    
-    // Bundle the website assets
-    let dist_path = bundle(websites_path, source_folder, config.dev_mode).await?;
-    
-    Ok(dist_path)
 }
 ```
 
-## File Structure
+## API Overview
 
-The bundler expects this directory structure:
+### Core Functions
 
-```
+**`bundle(websites_path: &Path, source_folder: &str, watch: bool) -> io::Result<PathBuf>`**
-websites/
+Main bundling function that processes web assets and optionally starts file watching.
-├── default/                 # Default website source
+
-│   ├── index.html          # Main HTML file
+### Bundling Process
-│   ├── service-worker.js   # Service worker (version injected)
+
-│   ├── scripts/            # JavaScript/TypeScript source
+1. **Directory Setup**: Creates `dist/` directory and copies source files
-│   │   ├── entry.js        # Main entry point
+2. **JavaScript Bundling**: Processes `scripts/entry.js` with Rolldown bundler
-│   │   ├── main.js         # Application logic
+3. **Template Processing**: Updates `index.html` with hashed asset references
-│   │   └── ...             # Other JS modules
+4. **Service Worker**: Generates service worker with version injection
-│   └── assets/             # Static assets
+5. **File Watching**: Optionally monitors source files for changes
-└── dist/                   # Generated output directory
+
-    ├── index.html          # Processed HTML with updated script references
+### Configuration
-    ├── service-worker.js   # Service worker with version injected
+
-    ├── scripts/            # Bundled and minified JavaScript
+**Rolldown Bundler Options:**
-    │   └── main-[hash].js  # Hashed output file
+
-    └── assets/             # Copied static assets
+- **Input**: `./src/scripts/entry.js` (main JavaScript entry point)
+- **Output**: `./dist/scripts/` directory
+- **Minification**: Enabled for production builds
+- **Source Maps**: File-based source maps for debugging
+- **Asset Hashing**: Automatic hash generation for cache busting
+
+## Examples
+
+### Development Mode with Hot Reloading
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let web_root = Path::new("./websites");
+
+    // Start development server with file watching
+    let _dist_path = bundle(web_root, "explorer", true).await?;
+
+    println!("Development server started!");
+    println!("Hot reloading enabled - edit files to see changes");
+
+    // Keep server running
+    loop {
+        tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
+    }
+}
 ```
 
-## Bundling Process
+### Production Build
 
-1. **Clean**: Removes existing `dist/` directory
+```rust
-2. **Copy**: Copies all source files to `dist/`
+use brk_bundler::bundle;
-3. **Bundle JavaScript**: 
+use std::path::Path;
-   - Processes `scripts/entry.js` as entry point
-   - Generates minified bundle with source maps
-   - Creates hashed filename for cache busting
-4. **Process HTML**: Updates script references to hashed filenames
-5. **Process Service Worker**: Injects current version string
-6. **Watch** (if enabled): Monitors for file changes and rebuilds
 
-## Configuration
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let web_root = Path::new("./websites");
 
-The bundler uses Rolldown with these optimized settings:
+    // Build for production (no watching)
+    let dist_path = bundle(web_root, "dashboard", false).await?;
+
+    println!("Production build completed: {}", dist_path.display());
+
+    // Assets are minified and ready for deployment
+    Ok(())
+}
+```
+
+### Custom Web Application Structure
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+// Expected directory structure:
+// websites/
+// ├── my_app/
+// │   ├── index.html          // Main HTML template
+// │   ├── service-worker.js   // Service worker template
+// │   ├── scripts/
+// │   │   └── entry.js        // JavaScript entry point
+// │   ├── styles/
+// │   │   └── main.css        // CSS files
+// │   └── assets/
+// │       └── images/         // Static assets
+// └── dist/                   // Generated output
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let websites_path = Path::new("./websites");
+    let source_folder = "my_app";
+
+    let dist_path = bundle(websites_path, source_folder, false).await?;
+
+    // Result: dist/ contains bundled and processed files
+    // - dist/index.html (with updated script references)
+    // - dist/service-worker.js (with version injection)
+    // - dist/scripts/main.[hash].js (minified and hashed)
+    // - dist/styles/ (copied CSS files)
+    // - dist/assets/ (copied static assets)
+
+    Ok(())
+}
+```
+
+## Architecture
+
+### File Processing Pipeline
+
+1. **Source Copying**: Recursively copies all source files to dist directory
+2. **JavaScript Bundling**: Rolldown processes entry.js with dependencies
+3. **Asset Hashing**: Generates content-based hashes for cache busting
+4. **Template Updates**: Replaces placeholders in HTML templates
+5. **Version Injection**: Updates service worker with current package version
+
+### File Watching System
+
+**Development Mode Watchers:**
+
+- **Source File Watcher**: Monitors non-script files for changes
+- **Bundle Watcher**: Watches JavaScript files and triggers rebuilds
+- **Template Watcher**: Updates HTML when bundled assets change
+
+**Event Handling:**
+
+- **File Creation/Modification**: Automatic copying to dist directory
+- **Script Changes**: Triggers Rolldown rebuild and template update
+- **Template Changes**: Processes HTML and updates asset references
+
+### Template Processing
+
+**index.html Processing:**
+
+- Scans bundled JavaScript for asset hash
+- Replaces `/scripts/main.js` with `/scripts/main.[hash].js`
+- Maintains cache busting while preserving template structure
+
+**service-worker.js Processing:**
+
+- Replaces `__VERSION__` placeholder with current crate version
+- Enables version-based cache invalidation
+- Maintains service worker functionality
+
+### Async Architecture
+
+Built on Tokio async runtime:
+
+- **Non-blocking I/O**: Efficient file operations and watching
+- **Concurrent Tasks**: Parallel file watching and bundle processing
+- **Background Processing**: Development server runs in background task
+
+## Configuration Options
+
+### Rolldown Configuration
+
+The bundler uses optimized Rolldown settings:
 
 ```rust
 BundlerOptions {
-    input: Some(vec![source_entry.into()]),    // scripts/entry.js
+    input: Some(vec!["./src/scripts/entry.js".into()]),
-    dir: Some("./dist/scripts".to_string()),   // Output directory
+    dir: Some("./dist/scripts".to_string()),
-    cwd: Some(websites_path),                  // Working directory
+    minify: Some(RawMinifyOptions::Bool(true)),
-    minify: Some(RawMinifyOptions::Bool(true)), // Enable minification
+    sourcemap: Some(SourceMapType::File),
-    sourcemap: Some(SourceMapType::File),      // Generate source maps
+    // ... other default options
-    ..Default::default()
 }
 ```
 
-## File Watching
+### File Structure Requirements
 
-In watch mode, the bundler monitors:
+**Required Files:**
 
-- **Source files**: Non-script files are copied on change
+- `src/scripts/entry.js` - JavaScript entry point
-- **JavaScript files**: Trigger full rebuild via Rolldown watcher
+- `src/index.html` - HTML template
-- **HTML files**: Processed to update script references
+- `src/service-worker.js` - Service worker template
-- **Service worker**: Version injection on changes
 
-### Watch Events Handled
+**Optional Directories:**
 
-- `Create` - New files added
+- `src/styles/` - CSS stylesheets
-- `Modify` - Existing files changed
+- `src/assets/` - Static assets (images, fonts, etc.)
-- Ignores `Delete` and other events
+- `src/components/` - Additional JavaScript modules
 
-## Version Injection
+## Development Workflow
 
-Service workers get automatic version injection:
+### Setup
 
-```javascript
+1. Create web application in `websites/app_name/`
-// In source service-worker.js
+2. Add required files (index.html, entry.js, service-worker.js)
-const VERSION = '__VERSION__';
+3. Run bundler in watch mode for development
 
-// After bundling
+### Hot Reloading
-const VERSION = 'v0.0.88';
-```
 
-This enables proper cache invalidation across releases.
+- **Script Changes**: Automatic bundle rebuild and browser refresh
+- **Template Changes**: Immediate HTML update with asset hash replacement
+- **Asset Changes**: Instant copy to dist directory
+- **Style Changes**: Direct copy without bundling
 
-## Performance Features
+### Production Deployment
 
-- **Async Operations**: All bundling operations are async
+1. Run bundler without watch mode
-- **Incremental Builds**: Only rebuilds changed files in watch mode
+2. Deploy `dist/` directory contents
-- **Parallel Processing**: Uses Tokio for concurrent file operations
+3. Assets include content hashes for cache busting
-- **Efficient Copying**: Direct file system operations
+4. Service worker includes version for cache management
 
-## Error Handling
+## Code Analysis Summary
 
-- **Graceful Failures**: Logs errors but continues watching
+**Main Function**: `bundle()` async function coordinating Rolldown bundler with file processing and watching \
-- **Path Resolution**: Automatic path absolutization and validation
+**File Operations**: Recursive directory copying with `copy_dir_all()` and selective file processing \
-- **File System Errors**: Proper error propagation with context
+**Templating**: String replacement for asset hash injection and version management \
-
+**File Watching**: Multi-watcher system using `notify` crate for real-time development feedback \
-## Dependencies
+**Async Integration**: Tokio-based async architecture with background task spawning \
-
+**Bundler Integration**: Rolldown wrapper with optimized configuration for web development \
-- `brk_rolldown` - Rust-based Rollup bundler
+**Architecture**: Development-focused asset pipeline with hot reloading and production optimization
-- `notify` - File system watching
-- `tokio` - Async runtime for file operations
-- `sugar_path` - Path manipulation utilities
-- `log` - Error logging
-
-## Integration Points
-
-The bundler integrates with:
-- **brk_cli**: Called during website setup
-- **brk_server**: Serves bundled assets
-- **Development workflow**: Provides live rebuilding
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_bundler/src/lib.rs

@@ -6,7 +6,10 @@ use std::{
     sync::Arc,
 };
 
-use brk_rolldown::{Bundler, BundlerOptions, RawMinifyOptions, SourceMapType};
+use brk_rolldown::{
+    Bundler, BundlerOptions, InlineConstConfig, InlineConstMode, InlineConstOption,
+    OptimizationOption, RawMinifyOptions, SourceMapType,
+};
 use log::error;
 use notify::{EventKind, RecursiveMode, Watcher};
 use sugar_path::SugarPath;
@@ -14,47 +17,88 @@ use tokio::sync::Mutex;
 
 const VERSION: &str = env!("CARGO_PKG_VERSION");
 
-pub async fn bundle(websites_path: &Path, source_folder: &str, watch: bool) -> io::Result<PathBuf> {
+pub async fn bundle(
-    let source_path = websites_path.join(source_folder);
+    modules_path: &Path,
-    let dist_path = websites_path.join("dist");
+    websites_path: &Path,
-
+    source_folder: &str,
-    let _ = fs::remove_dir_all(&dist_path);
+    watch: bool,
-    copy_dir_all(&source_path, &dist_path)?;
+) -> io::Result<PathBuf> {
-
+    let relative_modules_path = modules_path;
-    let source_scripts = format!("./{source_folder}/scripts");
+    let relative_source_path = websites_path.join(source_folder);
-    let source_entry = format!("{source_scripts}/entry.js");
+    let relative_dist_path = websites_path.join("dist");
 
+    let absolute_modules_path = relative_modules_path.absolutize();
+    let absolute_modules_path_clone = absolute_modules_path.clone();
     let absolute_websites_path = websites_path.absolutize();
+    let absolute_websites_path_clone = absolute_websites_path.clone();
+
+    let absolute_source_path = relative_source_path.absolutize();
+    let absolute_source_index_path = absolute_source_path.join("index.html");
+    let absolute_source_index_path_clone = absolute_source_index_path.clone();
+    let absolute_source_scripts_path = absolute_source_path.join("scripts");
+    let absolute_source_scripts_modules_path = absolute_source_scripts_path.join("modules");
+    let absolute_source_sw_path = absolute_source_path.join("service-worker.js");
+    let absolute_source_sw_path_clone = absolute_source_sw_path.clone();
+
+    let absolute_dist_path = relative_dist_path.absolutize();
+    let absolute_dist_scripts_path = absolute_dist_path.join("scripts");
+    let absolute_dist_scripts_entry_path = absolute_dist_scripts_path.join("entry.js");
+    let absolute_dist_scripts_entry_path_clone = absolute_dist_scripts_entry_path.clone();
+    let absolute_dist_index_path = absolute_dist_path.join("index.html");
+    let absolute_dist_sw_path = absolute_dist_path.join("service-worker.js");
+
+    let _ = fs::remove_dir_all(&absolute_dist_path);
+    let _ = fs::remove_dir_all(&absolute_source_scripts_modules_path);
+    copy_dir_all(
+        &absolute_modules_path,
+        &absolute_source_scripts_modules_path,
+    )?;
+    copy_dir_all(&absolute_source_path, &absolute_dist_path)?;
+    fs::remove_dir_all(&absolute_dist_scripts_path)?;
+    fs::create_dir(&absolute_dist_scripts_path)?;
+
+    // dbg!(BundlerOptions::default());
 
     let mut bundler = Bundler::new(BundlerOptions {
-        input: Some(vec![source_entry.into()]),
+        input: Some(vec![format!("./{source_folder}/scripts/entry.js").into()]),
         dir: Some("./dist/scripts".to_string()),
         cwd: Some(absolute_websites_path),
         minify: Some(RawMinifyOptions::Bool(true)),
         sourcemap: Some(SourceMapType::File),
+        // advanced_chunks: Some(AdvancedChunksOptions {
+        //     // min_size: Some(1000.0),
+        //     min_share_count: Some(20),
+        //     // min_module_size: S
+        //     // include_dependencies_recursively: Some(true),
+        //     ..Default::default()
+        // }),
+        //
+        // inline_dynamic_imports
+        // experimental: Some(ExperimentalOptions {
+        //     strict_execution_order: Some(true),
+        //     ..Default::default()
+        // }),
+        optimization: Some(OptimizationOption {
+            inline_const: Some(InlineConstOption::Config(InlineConstConfig {
+                mode: Some(InlineConstMode::All),
+                ..Default::default()
+            })),
+            // Needs benchmarks
+            // pife_for_module_wrappers: Some(true),
+            ..Default::default()
+        }),
         ..Default::default()
-    });
+    })
+    .unwrap();
 
-    bundler.write().await.unwrap();
+    if let Err(error) = bundler.write().await {
+        error!("{error:?}");
+    }
 
-    let absolute_source_index_path = source_path.join("index.html").absolutize();
+    let update_dist_index = move || {
-    let absolute_source_index_path_clone = absolute_source_index_path.clone();
-    let absolute_source_path = source_path.absolutize();
-    let absolute_source_path_clone = absolute_source_path.clone();
-    let absolute_source_scripts_path = websites_path.join(source_scripts).absolutize();
-    let absolute_source_sw_path = source_path.join("service-worker.js").absolutize();
-    let absolute_source_sw_path_clone = absolute_source_sw_path.clone();
-
-    let absolute_dist_entry_path = dist_path.join("scripts/entry.js").absolutize();
-    let absolute_dist_index_path = dist_path.join("index.html").absolutize();
-    let absolute_dist_path = dist_path.absolutize();
-    let absolute_dist_path_clone = absolute_dist_path.clone();
-    let absolute_dist_sw_path = dist_path.join("service-worker.js").absolutize();
-
-    let write_index = move || {
         let mut contents = fs::read_to_string(&absolute_source_index_path).unwrap();
 
-        if let Ok(entry) = fs::read_to_string(absolute_dist_path_clone.join("scripts/entry.js"))
+        if let Ok(entry) = fs::read_to_string(&absolute_dist_scripts_entry_path_clone)
             && let Some(start) = entry.find("main")
             && let Some(end) = entry.find(".js")
         {
@@ -65,36 +109,22 @@ pub async fn bundle(websites_path: &Path, source_folder: &str, watch: bool) -> i
         let _ = fs::write(&absolute_dist_index_path, contents);
     };
 
-    let write_sw = move || {
+    let update_source_sw = move || {
         let contents = fs::read_to_string(&absolute_source_sw_path)
             .unwrap()
             .replace("__VERSION__", &format!("v{VERSION}"));
         let _ = fs::write(&absolute_dist_sw_path, contents);
     };
 
-    write_index();
+    update_dist_index();
-    write_sw();
+    update_source_sw();
 
     if !watch {
-        return Ok(dist_path);
+        return Ok(relative_dist_path);
     }
 
     tokio::spawn(async move {
-        let write_index_clone = write_index.clone();
+        let mut event_watcher = notify::recommended_watcher(
-
-        let mut entry_watcher = notify::recommended_watcher(
-            move |res: Result<notify::Event, notify::Error>| match res {
-                Ok(_) => write_index_clone(),
-                Err(e) => error!("watch error: {e:?}"),
-            },
-        )
-        .unwrap();
-
-        entry_watcher
-            .watch(&absolute_dist_entry_path, RecursiveMode::Recursive)
-            .unwrap();
-
-        let mut source_watcher = notify::recommended_watcher(
             move |res: Result<notify::Event, notify::Error>| match res {
                 Ok(event) => match event.kind {
                     EventKind::Create(_) => event.paths,
@@ -102,18 +132,30 @@ pub async fn bundle(websites_path: &Path, source_folder: &str, watch: bool) -> i
                     _ => vec![],
                 }
                 .into_iter()
-                .filter(|path| path.starts_with(&absolute_source_path))
+                .for_each(|path| {
-                .filter(|path| !path.starts_with(&absolute_source_scripts_path))
+                    let path = path.absolutize();
-                .for_each(|source_path| {
-                    let suffix = source_path.strip_prefix(&absolute_source_path).unwrap();
-                    let dist_path = absolute_dist_path.join(suffix);
 
-                    if source_path == absolute_source_index_path_clone {
+                    if path == absolute_dist_scripts_entry_path
-                        write_index();
+                        || path == absolute_source_index_path_clone
-                    } else if source_path == absolute_source_sw_path_clone {
+                    {
-                        write_sw();
+                        update_dist_index();
-                    } else {
+                    } else if path == absolute_source_sw_path_clone {
-                        let _ = fs::copy(&source_path, &dist_path);
+                        update_source_sw();
+                    } else if let Ok(suffix) = path.strip_prefix(&absolute_modules_path) {
+                        let source_modules_path = absolute_source_scripts_modules_path.join(suffix);
+                        if path.is_file() {
+                            let _ = fs::create_dir_all(path.parent().unwrap());
+                            let _ = fs::copy(&path, &source_modules_path);
+                        }
+                    } else if let Ok(suffix) = path.strip_prefix(&absolute_source_path)
+                        // scripts are handled by rolldown
+                        && !path.starts_with(&absolute_source_scripts_path)
+                    {
+                        let dist_path = absolute_dist_path.join(suffix);
+                        if path.is_file() {
+                            let _ = fs::create_dir_all(path.parent().unwrap());
+                            let _ = fs::copy(&path, &dist_path);
+                        }
                     }
                 }),
                 Err(e) => error!("watch error: {e:?}"),
@@ -121,8 +163,11 @@ pub async fn bundle(websites_path: &Path, source_folder: &str, watch: bool) -> i
         )
         .unwrap();
 
-        source_watcher
+        event_watcher
-            .watch(&absolute_source_path_clone, RecursiveMode::Recursive)
+            .watch(&absolute_websites_path_clone, RecursiveMode::Recursive)
+            .unwrap();
+        event_watcher
+            .watch(&absolute_modules_path_clone, RecursiveMode::Recursive)
             .unwrap();
 
         let watcher =
@@ -131,7 +176,7 @@ pub async fn bundle(websites_path: &Path, source_folder: &str, watch: bool) -> i
         watcher.start().await;
     });
 
-    Ok(dist_path)
+    Ok(relative_dist_path)
 }
 
 fn copy_dir_all(src: impl AsRef<Path>, dst: impl AsRef<Path>) -> io::Result<()> {

crates/brk_cli/Cargo.toml

@@ -11,8 +11,10 @@ build = "build.rs"
 
 [dependencies]
 bitcoincore-rpc = { workspace = true }
+brk_binder = { workspace = true }
 brk_bundler = { workspace = true }
 brk_computer = { workspace = true }
+brk_error = { workspace = true }
 brk_fetcher = { workspace = true }
 brk_indexer = { workspace = true }
 brk_interface = { workspace = true }
@@ -20,15 +22,14 @@ brk_logger = { workspace = true }
 brk_parser = { workspace = true }
 brk_server = { workspace = true }
 vecdb = { workspace = true }
-clap = { version = "4.5.45", features = ["string"] }
+clap = { version = "4.5.48", features = ["derive", "string"] }
-clap_derive = "4.5.45"
 color-eyre = "0.6.5"
 log = { workspace = true }
 minreq = { workspace = true }
 serde = { workspace = true }
 tokio = { workspace = true }
-toml = "0.9.5"
+toml = "0.9.7"
-zip = { version = "4.5.0", default-features = false, features = ["deflate"] }
+zip = { version = "5.1.1", default-features = false, features = ["deflate"] }
 
 [[bin]]
 name = "brk"

crates/brk_cli/README.md

@@ -1,189 +1,232 @@
 # brk_cli
 
-**Command line interface for running complete BRK instances**
+Command-line interface orchestrating complete Bitcoin Research Kit instances with automatic configuration and continuous blockchain processing.
 
-`brk_cli` provides the main command-line interface for operating the Bitcoin Research Kit. It orchestrates the complete data pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with automatic configuration management and graceful operation.
+[![Crates.io](https://img.shields.io/crates/v/brk_cli.svg)](https://crates.io/crates/brk_cli)
+[![Documentation](https://docs.rs/brk_cli/badge.svg)](https://docs.rs/brk_cli)
 
-## What it provides
+## Overview
 
-- **Complete Pipeline Orchestration**: Coordinates parser, indexer, computer, and server components
+This crate provides the primary command-line interface for running Bitcoin Research Kit instances. It orchestrates the entire data processing pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with persistent configuration management, automatic error recovery, and continuous blockchain synchronization.
-- **Automatic Configuration**: Saves settings to `~/.brk/config.toml` for consistent operation
-- **Continuous Operation**: Handles blockchain updates and incremental processing
-- **Web Interface Options**: Configurable website serving (none, default, custom)
-- **Graceful Shutdown**: Ctrl+C handling with proper cleanup
 
-## Key Features
+**Key Features:**
 
-### Pipeline Management
+- Complete BRK pipeline orchestration with parser, indexer, computer, and server coordination
-- **Automatic dependency handling**: Ensures Bitcoin Core sync before processing
+- Persistent configuration system with TOML-based auto-save functionality
-- **Incremental updates**: Only processes new blocks since last run
+- Continuous blockchain processing with new block detection and incremental updates
-- **Error recovery**: Automatic retry logic and graceful error handling
+- Flexible Bitcoin Core RPC authentication with cookie file and user/password support
-- **Resource management**: Optimized memory usage and disk I/O
+- Configurable web interface options including auto-downloading from GitHub releases
+- Large stack allocation (512MB) for handling complex blockchain processing workloads
+- Graceful shutdown handling with proper cleanup and state preservation
 
-### Configuration System
+**Target Use Cases:**
-- **Auto-save configuration**: All CLI options saved to persistent config
-- **Flexible paths**: Configurable Bitcoin directory, blocks directory, and output directory
-- **RPC authentication**: Cookie file or username/password authentication
-- **Data source options**: Configurable price fetching and exchange APIs
 
-### Operation Modes
+- Production Bitcoin analytics deployments requiring full pipeline operation
-- **Initial sync**: Full blockchain processing from genesis
+- Development environments for Bitcoin research and analysis
-- **Continuous operation**: Real-time processing of new blocks
+- Continuous blockchain monitoring with real-time data updates
-- **Update mode**: Resume from last processed block
+- Academic research requiring comprehensive historical blockchain datasets
-- **Server mode**: HTTP API with optional web interface
 
 ## Installation
 
-### Binary Release
 ```bash
-# Download from GitHub releases
+cargo install brk # or cargo install brk_cli
-# https://github.com/bitcoinresearchkit/brk/releases/latest
 ```
 
-### Via Cargo
+## Quick Start
-```bash
-cargo install brk --locked
-```
-
-### From Source
-```bash
-git clone https://github.com/bitcoinresearchkit/brk.git
-cd brk && cargo build --release
-```
-
-## Usage
-
-### First Run (Configuration Setup)
 
 ```bash
-# Basic setup with default options
+# First run - configure and start processing
-brk --brkdir ./my_brk_data
+brk --brkdir ./data --bitcoindir ~/.bitcoin --fetch true
 
-# Full configuration
+# Subsequent runs use saved configuration
-brk --bitcoindir ~/.bitcoin \
-    --brkdir ./brk_data \
-    --fetch true \
-    --exchanges true \
-    --website default
-```
-
-### Subsequent Runs
-
-```bash
-# Uses saved configuration from ~/.brk/config.toml
 brk
 
 # Override specific options
 brk --website none --fetch false
 ```
 
-### Command Line Options
+## API Overview
+
+### Core Structure
+
+- **`Config`**: Persistent configuration with clap-based CLI parsing and TOML serialization
+- **`Bridge`**: Interface trait for generating JavaScript bridge files for web interfaces
+- **`Website`**: Enum for web interface options (None, Bitview, Custom)
+- **Path Functions**: Cross-platform default path resolution for Bitcoin and BRK directories
+
+### Main Operations
+
+**`main() -> color_eyre::Result<()>`**
+Entry point with error handling setup, directory creation, logging initialization, and high-stack thread spawning.
+
+**`run() -> color_eyre::Result<()>`**
+Core processing loop handling configuration, RPC connection, component initialization, and continuous blockchain monitoring.
+
+### Configuration Management
+
+**Persistent Settings:**
+
+- All CLI arguments automatically saved to `~/.brk/config.toml`
+- Argument overrides update saved configuration on each run
+- Cross-platform path resolution with tilde and $HOME expansion
+- Validation of Bitcoin directory, blocks directory, and RPC authentication
+
+**CLI Parameters:**
+
+- `--bitcoindir`, `--blocksdir`, `--brkdir`: Directory configuration
+- `--fetch`, `--exchanges`: Data source configuration
+- `--website`: Web interface selection
+- `--rpcconnect`, `--rpcport`, `--rpccookiefile`, `--rpcuser`, `--rpcpassword`: RPC settings
+
+## Examples
+
+### Basic Usage
 
 ```bash
-brk --help
+# Initialize with custom directories
+brk --bitcoindir /data/bitcoin --brkdir /data/brk
+
+# Enable all features with custom RPC
+brk --fetch true --exchanges true --website bitview \
+    --rpcuser myuser --rpcpassword mypass
+
+# Minimal setup with API only
+brk --website none --fetch false
 ```
 
-## Configuration Reference
+### Configuration File Example
 
-All options are automatically saved to `~/.brk/config.toml`:
+After first run, settings are saved to `~/.brk/config.toml`:
 
-### Core Paths
-- `--bitcoindir <PATH>` - Bitcoin Core directory (default: `~/.bitcoin`)
-- `--blocksdir <PATH>` - Block files directory (default: `bitcoindir/blocks`)
-- `--brkdir <PATH>` - BRK output directory (default: `~/.brk`)
-
-### Data Sources
-- `--fetch <BOOL>` - Enable price data fetching (default: `true`)
-- `--exchanges <BOOL>` - Use exchange APIs for prices (default: `true`)
-
-### Web Interface
-- `--website <OPTION>` - Web interface mode:
-  - `none` - API only, no web interface
-  - `default` - Built-in web interface from `websites/default/`
-  - `custom` - Serve custom website from `websites/custom/`
-
-### Bitcoin Core RPC
-- `--rpcconnect <IP>` - RPC host (default: `localhost`)
-- `--rpcport <PORT>` - RPC port (default: `8332`)
-- `--rpccookiefile <PATH>` - Cookie authentication file
-- `--rpcuser <USERNAME>` - Username authentication
-- `--rpcpassword <PASSWORD>` - Password authentication
-
-## Operation Flow
-
-1. **Configuration Loading**: Loads saved config from `~/.brk/config.toml`
-2. **Bitcoin Core Connection**: Establishes RPC connection and waits for sync
-3. **Data Pipeline Initialization**: Sets up parser, indexer, computer, and interface
-4. **Processing Loop**:
-   - Index new blocks from Bitcoin Core
-   - Compute analytics on new data
-   - Update cached data
-5. **Server Startup**: Launches HTTP API with optional web interface
-6. **Continuous Operation**: Monitors for new blocks and processes incrementally
-
-## System Requirements
-
-- **Bitcoin Core**: Fully synced node with RPC enabled
-- **Storage**: ~32% of blockchain size (~233GB as of 2025)
-- **Memory**:
-  - Peak: ~7-8GB during initial indexing
-  - Steady state: ~4-5GB during operation
-- **OS**: macOS or Linux
-  - Ubuntu: `sudo apt install libssl-dev pkg-config`
-
-## Performance Characteristics
-
-### Initial Sync
-- **Full blockchain processing**: ~13-15 hours total
-- **Parser phase**: ~4 minutes for block parsing
-- **Indexer phase**: ~7-8 hours for data indexing
-- **Computer phase**: ~6-7 hours for analytics computation
-
-### Continuous Operation
-- **New block processing**: 3-5 seconds per block
-- **API response times**: Typically <100ms with caching
-- **Memory usage**: Stable ~4-5GB during normal operation
-
-## Configuration File
-
-Example `~/.brk/config.toml`:
 ```toml
-bitcoindir = "/Users/username/.bitcoin"
+bitcoindir = "/home/user/.bitcoin"
-blocksdir = "/Users/username/.bitcoin/blocks"
+blocksdir = "/home/user/.bitcoin/blocks"
-brkdir = "/Users/username/brk_data"
+brkdir = "/home/user/brk_data"
 fetch = true
 exchanges = true
-website = "default"
+website = "bitview"
 rpcconnect = "localhost"
 rpcport = 8332
-rpccookiefile = "/Users/username/.bitcoin/.cookie"
+rpccookiefile = "/home/user/.bitcoin/.cookie"
 ```
 
-## Error Handling
+### Web Interface Configuration
 
-- **Bitcoin Core sync**: Waits for node sync before processing
+```bash
-- **RPC connection**: Automatic retry logic for connection issues
+# Use built-in Bitview interface
-- **Processing errors**: Graceful error handling with detailed logging
+brk --website bitview
-- **Graceful shutdown**: Ctrl+C handling with proper cleanup and state saving
 
-## Logging
+# Use custom web interface
+brk --website custom
 
-Logs are written to `~/.brk/brk.log` with colored console output:
+# API only, no web interface
-- Request/response logging with timing
+brk --website none
-- Processing progress indicators
+```
-- Error reporting and debugging information
 
-## Dependencies
+### Development Mode
 
-- `brk_parser` - Bitcoin block parsing
+```bash
-- `brk_indexer` - Blockchain data indexing
+# Development with local website directory
-- `brk_computer` - Analytics computation
+# Looks for ../../websites directory first
-- `brk_interface` - Data query interface
+brk --website bitview
-- `brk_server` - HTTP API server
+
-- `brk_logger` - Logging utilities
+# Production with auto-download from GitHub
-- `bitcoincore_rpc` - Bitcoin Core RPC client
+# Downloads websites from release artifacts
-- `color_eyre` - Enhanced error reporting
+brk --website bitview
+```
+
+## Architecture
+
+### Startup Sequence
+
+1. **Environment Setup**: Color eyre error handling, directory creation, logging initialization
+2. **High-Stack Thread**: 512MB stack for complex blockchain processing operations
+3. **Configuration Loading**: CLI parsing, TOML reading, argument merging, validation
+4. **Component Initialization**: Parser, indexer, computer, interface creation with proper dependencies
+
+### Processing Pipeline
+
+**Continuous Operation Loop:**
+
+1. **Bitcoin Core Sync Wait**: Monitors `headers == blocks` for full node synchronization
+2. **Block Count Detection**: Compares current and previous block counts for new block detection
+3. **Indexing Phase**: Processes new blocks through parser with collision detection option
+4. **Computing Phase**: Runs analytics computations on newly indexed data
+5. **Server Operation**: Serves HTTP API with optional web interface throughout processing
+
+### Web Interface Integration
+
+**Website Handling:**
+
+- **Development Mode**: Uses local `../../websites` directory if available
+- **Production Mode**: Downloads release artifacts from GitHub using semantic versioning
+- **Bundle Generation**: Creates optimized JavaScript bundles using `brk_bundler`
+- **Bridge Files**: Generates JavaScript bridge files for vector IDs and pool data
+
+**Download and Bundle Process:**
+
+```rust
+// Automatic website download and bundling
+let url = format!("https://github.com/bitcoinresearchkit/brk/archive/refs/tags/v{VERSION}.zip");
+let response = minreq::get(url).send()?;
+zip::ZipArchive::new(cursor).extract(downloads_path)?;
+bundle(&websites_path, website.to_folder_name(), true).await?
+```
+
+### RPC Authentication
+
+**Flexible Authentication Methods:**
+
+- **Cookie File**: Automatic detection at `--bitcoindir/.cookie`
+- **User/Password**: Manual configuration with `--rpcuser` and `--rpcpassword`
+- **Connection Validation**: Startup checks ensure proper Bitcoin Core connectivity
+
+### Configuration System
+
+**TOML Persistence:**
+
+- Automatic serialization/deserialization with `serde` and `toml`
+- Error-tolerant parsing with `default_on_error` deserializer
+- Argument consumption validation ensuring all CLI options are processed
+- Path expansion supporting `~` and `$HOME` environment variables
+
+## Configuration
+
+### Default Paths
+
+**Cross-Platform Path Resolution:**
+
+- **Linux**: `~/.bitcoin` for Bitcoin Core, `~/.brk` for BRK data
+- **macOS**: `~/Library/Application Support/Bitcoin` for Bitcoin Core
+- **Logs**: `~/.brk/log` for application logging
+- **Downloads**: `~/.brk/downloads` for temporary website artifacts
+
+### Performance Settings
+
+**Memory Management:**
+
+- 512MB stack size for main processing thread
+- Multi-threaded tokio runtime with all features enabled
+- Persistent configuration caching to minimize I/O operations
+
+### Error Handling
+
+**Comprehensive Validation:**
+
+- Directory existence checks with user-friendly error messages
+- RPC authentication verification before processing begins
+- Graceful exit with help suggestions for configuration issues
+
+## Code Analysis Summary
+
+**Main Structure**: `Config` struct with clap-derived CLI parsing and persistent TOML configuration management \
+**Processing Loop**: Continuous Bitcoin Core monitoring with sync detection and incremental block processing \
+**Web Integration**: Automatic website download from GitHub releases with JavaScript bundle generation \
+**Component Orchestration**: Coordination of parser, indexer, computer, and server with proper dependency management \
+**Error Handling**: `color_eyre` integration with comprehensive validation and user-friendly error messages \
+**Threading**: High-stack thread allocation (512MB) with tokio multi-threaded runtime for complex operations \
+**Architecture**: Complete BRK pipeline orchestration with persistent configuration and continuous blockchain synchronization
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_cli/src/bridge.rs

@@ -1,111 +0,0 @@
-use std::{fs, io, path::Path};
-
-use brk_interface::{Index, Interface};
-use brk_server::VERSION;
-
-use crate::website::Website;
-
-const SCRIPTS: &str = "scripts";
-
-#[allow(clippy::upper_case_acronyms)]
-pub trait Bridge {
-    fn generate_bridge_file(&self, website: Website, websites_path: &Path) -> io::Result<()>;
-}
-
-impl Bridge for Interface<'static> {
-    fn generate_bridge_file(&self, website: Website, websites_path: &Path) -> io::Result<()> {
-        if website.is_none() {
-            return Ok(());
-        }
-
-        let path = websites_path.join(website.to_folder_name());
-
-        if !fs::exists(&path)? {
-            return Ok(());
-        }
-
-        let path = path.join(SCRIPTS);
-
-        fs::create_dir_all(&path)?;
-
-        let path = path.join(Path::new("vecid-to-indexes.js"));
-
-        let indexes = Index::all();
-
-        let mut contents = format!(
-            "//
-// File auto-generated, any modifications will be overwritten
-//
-
-export const VERSION = \"v{VERSION}\";
-
-"
-        );
-
-        contents += &indexes
-            .iter()
-            .enumerate()
-            .map(|(i_of_i, i)| {
-                // let lowered = i.to_string().to_lowercase();
-                format!("/** @typedef {{{i_of_i}}} {i} */",)
-            })
-            .collect::<Vec<_>>()
-            .join("\n");
-
-        contents += &format!(
-            "\n\n/** @typedef {{{}}} Index */\n",
-            indexes
-                .iter()
-                .map(|i| i.to_string())
-                .collect::<Vec<_>>()
-                .join(" | ")
-        );
-
-        contents += "
-/** @typedef {ReturnType<typeof createIndexes>} Indexes */
-
-export function createIndexes() {
-  return {
-";
-
-        contents += &indexes
-            .iter()
-            .enumerate()
-            .map(|(i_of_i, i)| {
-                let lowered = i.to_string().to_lowercase();
-                format!("    {lowered}: /** @satisfies {{{i}}} */ ({i_of_i}),",)
-            })
-            .collect::<Vec<_>>()
-            .join("\n");
-
-        contents += "  };\n}\n";
-
-        contents += "
-/** @typedef {ReturnType<typeof createVecIdToIndexes>} VecIdToIndexes
-/** @typedef {keyof VecIdToIndexes} VecId */
-
-/**
- * @returns {Record<any, number[]>}
- */
-export function createVecIdToIndexes() {
-  return {
-";
-
-        self.id_to_index_to_vec()
-            .iter()
-            .for_each(|(id, index_to_vec)| {
-                let indexes = index_to_vec
-                    .keys()
-                    .map(|i| (*i as u8).to_string())
-                    // .map(|i| i.to_string())
-                    .collect::<Vec<_>>()
-                    .join(", ");
-
-                contents += &format!("    \"{id}\": [{indexes}],\n");
-            });
-
-        contents += "  };\n}\n";
-
-        fs::write(path, contents)
-    }
-}

crates/brk_cli/src/config.rs

@@ -6,7 +6,6 @@ use std::{
 use bitcoincore_rpc::{self, Auth, Client};
 use brk_fetcher::Fetcher;
 use clap::Parser;
-use clap_derive::Parser;
 use color_eyre::eyre::eyre;
 use serde::{Deserialize, Deserializer, Serialize};
 
@@ -284,7 +283,7 @@ Finally, you can run the program with '-h' for help."
     }
 
     pub fn website(&self) -> Website {
-        self.website.unwrap_or(Website::Default)
+        self.website.unwrap_or(Website::Bitview)
     }
 
     pub fn fetch(&self) -> bool {

crates/brk_cli/src/lib.rs

@@ -9,20 +9,22 @@ use std::{
 };
 
 use bitcoincore_rpc::{self, RpcApi};
+use brk_binder::Bridge;
 use brk_bundler::bundle;
 use brk_computer::Computer;
+use brk_error::Result;
 use brk_indexer::Indexer;
 use brk_interface::Interface;
+use brk_parser::Parser;
 use brk_server::{Server, VERSION};
 use log::info;
 use vecdb::Exit;
 
-mod bridge;
 mod config;
 mod paths;
 mod website;
 
-use crate::{bridge::Bridge, config::Config, paths::*};
+use crate::{config::Config, paths::*};
 
 pub fn main() -> color_eyre::Result<()> {
     color_eyre::install()?;
@@ -32,7 +34,7 @@ pub fn main() -> color_eyre::Result<()> {
     brk_logger::init(Some(&dot_brk_log_path()))?;
 
     thread::Builder::new()
-        .stack_size(256 * 1024 * 1024)
+        .stack_size(512 * 1024 * 1024)
         .spawn(run)?
         .join()
         .unwrap()
@@ -46,101 +48,120 @@ pub fn run() -> color_eyre::Result<()> {
     let exit = Exit::new();
     exit.set_ctrlc_handler();
 
-    let parser = brk_parser::Parser::new(config.blocksdir(), config.brkdir(), rpc);
+    let parser = Parser::new(config.blocksdir(), rpc);
 
     let mut indexer = Indexer::forced_import(&config.brkdir())?;
 
-    let wait_for_synced_node = |rpc_client: &bitcoincore_rpc::Client| -> color_eyre::Result<()> {
-        let is_synced = || -> color_eyre::Result<bool> {
-            let info = rpc_client.get_blockchain_info()?;
-            Ok(info.headers == info.blocks)
-        };
-
-        if !is_synced()? {
-            info!("Waiting for node to be synced...");
-            while !is_synced()? {
-                sleep(Duration::from_secs(1))
-            }
-        }
-
-        Ok(())
-    };
-
     let mut computer = Computer::forced_import(&config.brkdir(), &indexer, config.fetcher())?;
 
-    tokio::runtime::Builder::new_multi_thread()
+    let interface = Interface::build(&parser, &indexer, &computer);
-        .enable_all()
-        .build()?
-        .block_on(async {
-            let interface = Interface::build(&indexer, &computer);
 
-            let website = config.website();
+    let website = config.website();
 
-            let downloads_path = config.downloads_dir();
+    let downloads_path = config.downloads_dir();
 
-            let bundle_path = if website.is_some() {
+    let future = async move {
-                let websites_dev_path = Path::new("../../websites");
+        let bundle_path = if website.is_some() {
+            let websites_dev_path = Path::new("../../websites");
+            let modules_dev_path = Path::new("../../modules");
 
-                let websites_path = if fs::exists(websites_dev_path)? {
+            let websites_path;
-                    websites_dev_path.to_path_buf()
+            let modules_path;
-                } else {
-                    let downloaded_websites_path =
-                        downloads_path.join(format!("brk-{VERSION}")).join("websites");
 
-                    if !fs::exists(&downloaded_websites_path)? {
+            if fs::exists(websites_dev_path)? && fs::exists(modules_dev_path)? {
-                        info!("Downloading websites from Github...");
+                websites_path = websites_dev_path.to_path_buf();
-
+                modules_path = modules_dev_path.to_path_buf();
-                        let url = format!(
-                            "https://github.com/bitcoinresearchkit/brk/archive/refs/tags/v{VERSION}.zip",
-                        );
-
-                        let response = minreq::get(url).send()?;
-                        let bytes = response.as_bytes();
-                        let cursor = Cursor::new(bytes);
-
-                        let mut zip = zip::ZipArchive::new(cursor).unwrap();
-
-                        zip.extract(downloads_path).unwrap();
-                    }
-
-                    downloaded_websites_path
-                };
-
-                interface.generate_bridge_file(website, websites_path.as_path())?;
-
-                Some(bundle(&websites_path, website.to_folder_name(), true).await?)
             } else {
-                None
+                let downloaded_brk_path = downloads_path.join(format!("brk-{VERSION}"));
-            };
 
-            let server = Server::new(
+                let downloaded_websites_path = downloaded_brk_path.join("websites");
-                interface,
+                let downloaded_modules_path = downloaded_brk_path.join("modules");
-                bundle_path,
-            );
 
-            tokio::spawn(async move {
+                if !fs::exists(&downloaded_websites_path)? {
-                server.serve(true).await.unwrap();
+                    info!("Downloading source from Github...");
-            });
 
-            sleep(Duration::from_secs(1));
+                    let url = format!(
+                        "https://github.com/bitcoinresearchkit/brk/archive/refs/tags/v{VERSION}.zip",
+                    );
 
-            loop {
+                    let response = minreq::get(url).send()?;
-                wait_for_synced_node(rpc)?;
+                    let bytes = response.as_bytes();
+                    let cursor = Cursor::new(bytes);
 
-                let block_count = rpc.get_block_count()?;
+                    let mut zip = zip::ZipArchive::new(cursor).unwrap();
 
-                info!("{} blocks found.", block_count + 1);
+                    zip.extract(downloads_path).unwrap();
-
-                let starting_indexes =
-                    indexer.index(&parser, rpc, &exit, config.check_collisions()).unwrap();
-
-                computer.compute(&indexer, starting_indexes, &exit).unwrap();
-
-                info!("Waiting for new blocks...");
-
-                while block_count == rpc.get_block_count()? {
-                    sleep(Duration::from_secs(1))
                 }
+
+                websites_path = downloaded_websites_path;
+                modules_path = downloaded_modules_path;
             }
-        })
+
+            interface.generate_js_files(&modules_path)?;
+
+            Some(
+                bundle(
+                    &modules_path,
+                    &websites_path,
+                    website.to_folder_name(),
+                    true,
+                )
+                .await?,
+            )
+        } else {
+            None
+        };
+
+        let server = Server::new(interface, bundle_path);
+
+        tokio::spawn(async move {
+            server.serve(true).await.unwrap();
+        });
+
+        Ok(()) as Result<()>
+    };
+
+    let runtime = tokio::runtime::Builder::new_multi_thread()
+        .enable_all()
+        .build()?;
+
+    let _handle = runtime.spawn(future);
+
+    loop {
+        wait_for_synced_node(rpc)?;
+
+        let block_count = rpc.get_block_count()?;
+
+        info!("{} blocks found.", block_count + 1);
+
+        let starting_indexes = indexer
+            .index(&parser, rpc, &exit, config.check_collisions())
+            .unwrap();
+
+        computer
+            .compute(&indexer, starting_indexes, &parser, &exit)
+            .unwrap();
+
+        info!("Waiting for new blocks...");
+
+        while block_count == rpc.get_block_count()? {
+            sleep(Duration::from_secs(1))
+        }
+    }
+}
+
+fn wait_for_synced_node(rpc_client: &bitcoincore_rpc::Client) -> color_eyre::Result<()> {
+    let is_synced = || -> color_eyre::Result<bool> {
+        let info = rpc_client.get_blockchain_info()?;
+        Ok(info.headers == info.blocks)
+    };
+
+    if !is_synced()? {
+        info!("Waiting for node to sync...");
+        while !is_synced()? {
+            sleep(Duration::from_secs(1))
+        }
+    }
+
+    Ok(())
 }

crates/brk_cli/src/website.rs

@@ -1,10 +1,11 @@
-use clap_derive::ValueEnum;
+use clap::ValueEnum;
 use serde::{Deserialize, Serialize};
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize, ValueEnum)]
+#[serde(rename_all = "lowercase")]
 pub enum Website {
     None,
-    Default,
+    Bitview,
     Custom,
 }
 
@@ -20,7 +21,7 @@ impl Website {
     pub fn to_folder_name(self) -> &'static str {
         match self {
             Self::Custom => "custom",
-            Self::Default => "default",
+            Self::Bitview => "bitview",
             Self::None => unreachable!(),
         }
     }

crates/brk_computer/Cargo.toml

@@ -10,6 +10,7 @@ rust-version.workspace = true
 build = "build.rs"
 
 [dependencies]
+allocative = { workspace = true }
 bitcoin = { workspace = true }
 bitcoincore-rpc = { workspace = true }
 brk_structs = { workspace = true }
@@ -17,6 +18,7 @@ brk_error = { workspace = true }
 brk_fetcher = { workspace = true }
 brk_indexer = { workspace = true }
 brk_logger = { workspace = true }
+brk_store = { workspace = true }
 brk_parser = { workspace = true }
 vecdb = { workspace = true }
 derive_deref = { workspace = true }
@@ -26,6 +28,3 @@ rayon = { workspace = true }
 serde = { workspace = true }
 zerocopy = { workspace = true }
 zerocopy-derive = { workspace = true }
-
-[package.metadata.cargo-machete]
-ignored = ["zerocopy"]

crates/brk_computer/README.md

@@ -1,200 +1,303 @@
 # brk_computer
 
-**Bitcoin analytics engine that transforms indexed blockchain data into comprehensive metrics**
+Advanced Bitcoin analytics engine that transforms indexed blockchain data into comprehensive metrics and financial analytics.
 
-`brk_computer` is the computational layer of BRK that processes indexed blockchain data to generate analytics across multiple specialized domains. It provides comprehensive Bitcoin metrics with efficient storage and lazy computation for optimal performance.
+[![Crates.io](https://img.shields.io/crates/v/brk_computer.svg)](https://crates.io/crates/brk_computer)
+[![Documentation](https://docs.rs/brk_computer/badge.svg)](https://docs.rs/brk_computer)
 
-## What it provides
+## Overview
 
-- **Comprehensive Analytics**: 9 specialized domains covering all aspects of Bitcoin analysis
+This crate provides a sophisticated analytics engine that processes indexed Bitcoin blockchain data to compute comprehensive metrics, financial analytics, and statistical aggregations. Built on top of `brk_indexer`, it transforms raw blockchain data into actionable insights through state tracking, cohort analysis, market metrics, and advanced Bitcoin-specific calculations.
-- **Lazy Computation**: On-demand calculation with dependency tracking and caching
-- **Incremental Updates**: Only processes new data since last computation
-- **Memory Efficiency**: ~100MB operation footprint via compressed storage and memory mapping
-- **Multi-timeframe Analysis**: Daily, weekly, monthly, quarterly, yearly perspectives
 
-## Nine Analytics Domains
+**Key Features:**
 
-The computer processes data through a fixed dependency chain:
+- Comprehensive Bitcoin analytics pipeline with 6 major computation modules
+- UTXO and address cohort analysis with lifecycle tracking
+- Market metrics integration with price data and financial calculations
+- Cointime economics and realized/unrealized profit/loss analysis
+- Supply dynamics and monetary policy metrics
+- Pool analysis for centralization and mining statistics
+- Memory allocation tracking and performance optimization
+- Parallel computation with multi-threaded processing
 
-1. **indexes** - Time-based indexing (date/height mappings, epoch calculations)
+**Target Use Cases:**
-2. **constants** - Baseline values and reference metrics
-3. **blocks** - Block analytics (sizes, intervals, transaction counts, weight)
-4. **mining** - Mining economics (hashrate, difficulty, rewards, epochs)
-5. **fetched** - External price data integration (optional)
-6. **price** - OHLC data across multiple timeframes (optional, requires fetched)
-7. **transactions** - Transaction analysis (fees, sizes, patterns, RBF detection)
-8. **market** - Price correlations and market metrics (optional, requires price)
-9. **stateful** - UTXO tracking and accumulated state computations
-10. **cointime** - Coin age and time-based value analysis
 
-## Key Features
+- Bitcoin market analysis and research platforms
+- On-chain analytics for investment and trading decisions
+- Academic research requiring comprehensive blockchain metrics
+- Financial applications needing Bitcoin exposure and risk metrics
 
-### Computation Strategy
+## Installation
-- **Fixed dependency chain**: Ensures data consistency across all domains
-- **Parallel processing**: Uses Rayon for performance optimization
-- **State management**: Rollback capabilities for error recovery
-- **Incremental updates**: Only computes new data since last run
 
-### Analytics Capabilities
+```bash
-- **Multi-timeframe analysis**: Daily, weekly, monthly, quarterly, yearly aggregations
+cargo add brk_computer
-- **Chain-based metrics**: Height, difficulty epoch, halving epoch indexing
+```
-- **Price correlation**: Both dollar and satoshi denominated metrics
-- **DCA analysis**: Dollar Cost Averaging with configurable periods
-- **Supply analysis**: Circulating, realized, unrealized supply metrics
-- **Address cohort tracking**: Analysis across different Bitcoin address types
-- **UTXO cohort analysis**: Realized/unrealized gains tracking
-- **Coin time analysis**: Understanding Bitcoin velocity and dormancy
 
-### Storage Optimization
+## Quick Start
-- **Compressed vectors**: Efficient disk storage with lazy computation
-- **Memory mapping**: Minimal RAM usage during operation
-- **Version management**: Automatic invalidation on schema changes
-- **Dependency tracking**: Smart recomputation based on data changes
-
-## Usage
-
-### Basic Setup (No Price Data)
 
 ```rust
 use brk_computer::Computer;
 use brk_indexer::Indexer;
-use vecdb::Exit;
-
-// Setup without external price data
-let indexer = Indexer::forced_import("./brk_data")?;
-let mut computer = Computer::forced_import("./brk_data", &indexer, None)?;
-
-// Setup exit handler
-let exit = Exit::new();
-exit.set_ctrlc_handler();
-
-// Compute all analytics
-let starting_indexes = indexer.get_starting_indexes();
-computer.compute(&indexer, starting_indexes, &exit)?;
-```
-
-### Advanced Setup (With Price Data)
-
-```rust
 use brk_fetcher::Fetcher;
+use vecdb::Exit;
+use std::path::Path;
 
-// Setup with external price data for market analytics
+// Initialize dependencies
+let outputs_path = Path::new("./analytics_data");
+let indexer = Indexer::forced_import(outputs_path)?;
 let fetcher = Some(Fetcher::import(true, None)?);
-let mut computer = Computer::forced_import("./brk_data", &indexer, fetcher)?;
 
-// Compute all analytics including price/market domains
+// Create computer with price data support
+let mut computer = Computer::forced_import(outputs_path, &indexer, fetcher)?;
+
+// Compute analytics from indexer state
+let exit = Exit::default();
+let starting_indexes = brk_indexer::Indexes::default();
 computer.compute(&indexer, starting_indexes, &exit)?;
+
+println!("Analytics computation completed!");
 ```
 
-### Accessing Computed Data
+## API Overview
+
+### Core Structure
+
+The Computer is organized into 7 specialized computation modules:
+
+- **`indexes`**: Fundamental blockchain index computations
+- **`constants`**: Network constants and protocol parameters
+- **`market`**: Price-based financial metrics and market analysis
+- **`pools`**: Mining pool analysis and centralization metrics
+- **`chain`**: Core blockchain metrics (difficulty, hashrate, fees)
+- **`stateful`**: Advanced state tracking (UTXO lifecycles, address behaviors)
+- **`cointime`**: Cointime economics and value-time calculations
+
+### Key Methods
+
+**`Computer::forced_import(outputs_path, indexer, fetcher) -> Result<Self>`**
+Creates computer instance with optional price data integration.
+
+**`compute(&mut self, indexer: &Indexer, starting_indexes: Indexes, exit: &Exit) -> Result<()>`**
+Main computation pipeline processing all analytics modules.
+
+### Analytics Categories
+
+**Market Analytics:**
+
+- Price-based metrics (market cap, realized cap, MVRV)
+- Trading volume analysis and liquidity metrics
+- Return calculations and volatility measurements
+- Dollar-cost averaging and investment strategy metrics
+
+**On-Chain Analytics:**
+
+- Transaction count and size statistics
+- Fee analysis and block space utilization
+- Address activity and entity clustering
+- UTXO age distributions and spending patterns
+
+**Monetary Analytics:**
+
+- Circulating supply and issuance tracking
+- Realized vs. unrealized gains/losses
+- Cointime destruction and accumulation
+- Velocity and economic activity indicators
+
+## Examples
+
+### Basic Analytics Computation
 
 ```rust
-// Access all computed vectors
+use brk_computer::Computer;
-let all_vecs = computer.vecs(); // Returns Vec<&dyn AnyCollectableVec>
 
-// Access specific domain data
+// Initialize with indexer and optional price data
-let block_metrics = &computer.blocks;
+let computer = Computer::forced_import(
-let mining_data = &computer.mining;
+    "./analytics_output",
-let transaction_stats = &computer.transactions;
+    &indexer,
+    Some(price_fetcher)
+)?;
 
-// Access price data (if available)
+// Compute all analytics modules
-if let Some(price_data) = &computer.price {
+let exit = vecdb::Exit::default();
-    // Use OHLC data
+computer.compute(&indexer, starting_indexes, &exit)?;
-}
+
+// Access computed metrics
+println!("Market cap vectors computed: {}", computer.market.len());
+println!("Chain metrics computed: {}", computer.chain.len());
+println!("Stateful analysis completed: {}", computer.stateful.len());
 ```
 
-### Incremental Updates
+### Market Analysis
 
 ```rust
-// Continuous computation loop
+use brk_computer::Computer;
-loop {
+use brk_structs::{DateIndex, Height};
-    // Get latest indexes from indexer
-    let current_indexes = indexer.get_current_indexes();
 
-    // Compute only new data
+let computer = Computer::forced_import(/* ... */)?;
-    computer.compute(&indexer, current_indexes, &exit)?;
 
-    // Check for exit signal
+// Access market metrics after computation
-    if exit.is_signaled() {
+if let Some(market) = &computer.market {
-        break;
+    // Daily market cap analysis
+    let date_index = DateIndex::from_days_since_genesis(5000);
+    if let Some(market_cap) = market.dateindex_to_market_cap.get(date_index)? {
+        println!("Market cap on day {}: ${}", date_index, market_cap.to_dollars());
     }
 
-    // Wait before next update
+    // MVRV (Market Value to Realized Value) ratio
-    sleep(Duration::from_secs(60));
+    if let Some(mvrv) = market.dateindex_to_mvrv.get(date_index)? {
+        println!("MVRV ratio: {:.2}", mvrv);
+    }
+}
+
+// Chain-level metrics
+let height = Height::new(800000);
+if let Some(difficulty) = computer.chain.height_to_difficulty.get(height)? {
+    println!("Network difficulty at height {}: {}", height, difficulty);
 }
 ```
 
-## Core Computer Structure
+### Cohort Analysis
 
 ```rust
-pub struct Computer {
+use brk_computer::Computer;
-    pub indexes: indexes::Vecs,           // Time indexing
+use brk_structs::{DateIndex, CohortId};
-    pub constants: constants::Vecs,       // Baseline values
+
-    pub blocks: blocks::Vecs,            // Block analytics
+let computer = Computer::forced_import(/* ... */)?;
-    pub mining: mining::Vecs,            // Mining economics
+
-    pub market: market::Vecs,            // Market metrics (optional)
+// Address cohort analysis
-    pub price: Option<price::Vecs>,      // OHLC price data (optional)
+let cohort_date = DateIndex::from_days_since_genesis(4000);
-    pub transactions: transactions::Vecs, // Transaction analysis
+
-    pub stateful: stateful::Vecs,        // UTXO tracking
+// Analyze address behavior patterns
-    pub fetched: Option<fetched::Vecs>,  // External data (optional)
+if let Some(address_cohorts) = &computer.stateful.address_cohorts {
-    pub cointime: cointime::Vecs,        // Coin age analysis
+    for cohort_id in address_cohorts.get_cohort_ids_for_date(cohort_date)? {
+        let cohort_data = address_cohorts.get_cohort(cohort_id)?;
+
+        println!("Cohort {}: {} addresses created",
+                 cohort_id, cohort_data.addresses.len());
+        println!("Average holding period: {} days",
+                 cohort_data.avg_holding_period.as_days());
+    }
+}
+
+// UTXO cohort lifecycle analysis
+if let Some(utxo_cohorts) = &computer.stateful.utxo_cohorts {
+    let active_utxos = utxo_cohorts.get_active_utxos_for_date(cohort_date)?;
+    println!("Active UTXOs from cohort: {}", active_utxos.len());
 }
 ```
 
-## Performance Characteristics
+### Supply and Monetary Analysis
 
-**Benchmarked on MacBook Pro M3 Pro:**
+```rust
-- **Initial computation**: ~6-7 hours for complete Bitcoin blockchain
+use brk_computer::Computer;
-- **Storage efficiency**: All computed datasets total ~40GB
+use brk_structs::{Height, DateIndex};
-- **Incremental updates**: 3-5 seconds per new block
-- **Memory footprint**: Peak ~7-8GB during computation, ~100MB during operation
-- **Dependencies**: Price data domains optional (fetched, price, market)
 
-## Domain-Specific Analytics
+let computer = Computer::forced_import(/* ... */)?;
 
-### Block Analytics
+// Supply dynamics
-- Block sizes, weights, transaction counts
+let height = Height::new(750000);
-- Block intervals and mining statistics
+if let Some(supply) = computer.chain.height_to_circulating_supply.get(height)? {
-- Fee analysis per block
+    println!("Circulating supply: {} BTC", supply.to_btc());
+}
 
-### Mining Economics
+// Realized vs unrealized analysis
-- Hashrate estimation and difficulty tracking
+let date = DateIndex::from_days_since_genesis(5000);
-- Mining reward analysis
+if let Some(realized_cap) = computer.market.dateindex_to_realized_cap.get(date)? {
-- Epoch-based calculations
+    if let Some(market_cap) = computer.market.dateindex_to_market_cap.get(date)? {
+        let unrealized_pnl = market_cap - realized_cap;
+        println!("Unrealized P&L: ${:.2}B", unrealized_pnl.to_dollars() / 1e9);
+    }
+}
+```
 
-### Transaction Analysis
+## Architecture
-- Fee rate distributions
-- RBF (Replace-By-Fee) detection
-- Output type analysis
-- Transaction size patterns
 
-### Market Metrics (Optional)
+### Computation Pipeline
-- Price correlations with on-chain metrics
-- Market cap calculations
-- DCA analysis across timeframes
 
-### Stateful Analysis
+The computer implements a sophisticated multi-stage pipeline:
-- UTXO set tracking
-- Address cohort analysis
-- Realized/unrealized gains
-- Supply distribution metrics
 
-## Requirements
+1. **Index Computation**: Fundamental blockchain metrics and time-based indexes
+2. **Constants Computation**: Network parameters and protocol constants
+3. **Price Integration**: Optional price data fetching and processing
+4. **Parallel Computation**: Chain, market, pools, stateful, and cointime analytics
+5. **Cross-Dependencies**: Advanced metrics requiring multiple data sources
 
-- **Indexed data**: Requires completed `brk_indexer` output
+### Memory Management
-- **Storage space**: Additional ~40GB for computed datasets
-- **Memory**: 8GB+ RAM recommended for initial computation
-- **CPU**: Multi-core recommended for parallel processing
-- **Price data**: Optional external price feeds for market analytics
 
-## Dependencies
+**Allocation Tracking:**
 
-- `brk_indexer` - Source of indexed blockchain data
+- `allocative` integration for memory usage analysis
-- `brk_fetcher` - External price data (optional)
+- Efficient vector storage with compression options
-- `vecdb` - Vector database with lazy computation
+- Strategic lazy vs. eager evaluation for memory optimization
-- `rayon` - Parallel processing framework
+
-- `brk_structs` - Bitcoin-aware type system
+**Performance Optimization:**
+
+- `rayon` parallel processing for CPU-intensive calculations
+- Vectorized operations for time-series computations
+- Memory-mapped storage for large datasets
+
+### State Management
+
+**Stateful Analytics:**
+
+- UTXO lifecycle tracking with creation/destruction events
+- Address cohort analysis with behavioral clustering
+- Transaction pattern recognition and anomaly detection
+- Economic cycle analysis with market phase detection
+
+**Cointime Economics:**
+
+- Bitcoin days destroyed and accumulated calculations
+- Velocity measurements and economic activity indicators
+- Age-weighted value transfer analysis
+- Long-term holder vs. active trader segmentation
+
+### Modular Design
+
+Each computation module operates independently:
+
+- **Chain Module**: Basic blockchain metrics (fees, difficulty, hashrate)
+- **Market Module**: Price-dependent financial calculations
+- **Pools Module**: Mining centralization and pool analysis
+- **Stateful Module**: Advanced lifecycle and behavior tracking
+- **Cointime Module**: Economic time-value calculations
+
+### Data Dependencies
+
+**Required Dependencies:**
+
+- `brk_indexer`: Raw blockchain data access
+- `brk_structs`: Type definitions and conversions
+
+**Optional Dependencies:**
+
+- `brk_fetcher`: Price data for financial metrics
+- Market analysis requires price integration
+
+### Computation Orchestration
+
+**Sequential Stages:**
+
+1. Indexes → Constants (foundational metrics)
+2. Fetched → Price (price data processing)
+3. Parallel: Chain, Market, Pools, Stateful, Cointime
+
+**Exit Handling:**
+
+- Graceful shutdown with consistent state preservation
+- Checkpoint-based recovery for long-running computations
+- Multi-threaded coordination with exit signaling
+
+## Code Analysis Summary
+
+**Main Structure**: `Computer` struct coordinating 7 specialized analytics modules (indexes, constants, market, pools, chain, stateful, cointime) \
+**Computation Pipeline**: Multi-stage analytics processing with parallel execution and dependency management \
+**State Tracking**: Advanced UTXO and address lifecycle analysis with cohort-based behavioral clustering \
+**Financial Analytics**: Comprehensive market metrics including realized/unrealized analysis and cointime economics \
+**Memory Optimization**: `allocative` tracking with lazy/eager evaluation strategies and compressed vector storage \
+**Parallel Processing**: `rayon` integration for CPU-intensive calculations with coordinated exit handling \
+**Architecture**: Modular analytics engine transforming indexed blockchain data into actionable financial and economic insights
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_computer/examples/computer.rs

@@ -34,7 +34,7 @@ pub fn main() -> Result<()> {
             let outputs_dir = Path::new(&std::env::var("HOME").unwrap()).join(".brk");
             // let outputs_dir = Path::new("../../_outputs");
 
-            let parser = Parser::new(bitcoin_dir.join("blocks"), outputs_dir.to_path_buf(), rpc);
+            let parser = Parser::new(bitcoin_dir.join("blocks"), rpc);
 
             let mut indexer = Indexer::forced_import(&outputs_dir)?;
 
@@ -45,7 +45,7 @@ pub fn main() -> Result<()> {
             loop {
                 let i = Instant::now();
                 let starting_indexes = indexer.index(&parser, rpc, &exit, true)?;
-                computer.compute(&indexer, starting_indexes, &exit)?;
+                computer.compute(&indexer, starting_indexes, &parser, &exit)?;
                 dbg!(i.elapsed());
                 sleep(Duration::from_secs(10));
             }

crates/brk_computer/examples/pools.rs

@@ -0,0 +1,125 @@
+use std::{collections::BTreeMap, path::Path, thread};
+
+use brk_computer::Computer;
+use brk_error::Result;
+use brk_fetcher::Fetcher;
+use brk_indexer::Indexer;
+use brk_structs::{AddressBytes, OutputIndex, OutputType, pools};
+use vecdb::{AnyIterableVec, Exit, VecIterator};
+
+fn main() -> Result<()> {
+    brk_logger::init(Some(Path::new(".log")))?;
+
+    let exit = Exit::new();
+    exit.set_ctrlc_handler();
+
+    thread::Builder::new()
+        .stack_size(256 * 1024 * 1024)
+        .spawn(move || -> Result<()> {
+            let outputs_dir = Path::new(&std::env::var("HOME").unwrap()).join(".brk");
+
+            let indexer = Indexer::forced_import(&outputs_dir)?;
+
+            let fetcher = Fetcher::import(true, None)?;
+
+            let computer = Computer::forced_import(&outputs_dir, &indexer, Some(fetcher))?;
+
+            let pools = pools();
+
+            let mut res: BTreeMap<&'static str, usize> = BTreeMap::default();
+
+            let vecs = indexer.vecs;
+            let stores = indexer.stores;
+
+            let mut height_to_first_txindex_iter = vecs.height_to_first_txindex.iter();
+            let mut txindex_to_first_outputindex_iter = vecs.txindex_to_first_outputindex.iter();
+            let mut txindex_to_output_count_iter = computer.indexes.txindex_to_output_count.iter();
+            let mut outputindex_to_outputtype_iter = vecs.outputindex_to_outputtype.iter();
+            let mut outputindex_to_typeindex_iter = vecs.outputindex_to_typeindex.iter();
+            let mut p2pk65addressindex_to_p2pk65bytes_iter =
+                vecs.p2pk65addressindex_to_p2pk65bytes.iter();
+            let mut p2pk33addressindex_to_p2pk33bytes_iter =
+                vecs.p2pk33addressindex_to_p2pk33bytes.iter();
+            let mut p2pkhaddressindex_to_p2pkhbytes_iter =
+                vecs.p2pkhaddressindex_to_p2pkhbytes.iter();
+            let mut p2shaddressindex_to_p2shbytes_iter = vecs.p2shaddressindex_to_p2shbytes.iter();
+            let mut p2wpkhaddressindex_to_p2wpkhbytes_iter =
+                vecs.p2wpkhaddressindex_to_p2wpkhbytes.iter();
+            let mut p2wshaddressindex_to_p2wshbytes_iter =
+                vecs.p2wshaddressindex_to_p2wshbytes.iter();
+            let mut p2traddressindex_to_p2trbytes_iter = vecs.p2traddressindex_to_p2trbytes.iter();
+            let mut p2aaddressindex_to_p2abytes_iter = vecs.p2aaddressindex_to_p2abytes.iter();
+
+            let unknown = pools.get_unknown();
+
+            stores
+                .height_to_coinbase_tag
+                .iter()
+                .for_each(|(height, coinbase_tag)| {
+                    let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+                    let outputindex = txindex_to_first_outputindex_iter.unwrap_get_inner(txindex);
+                    let outputcount = txindex_to_output_count_iter.unwrap_get_inner(txindex);
+
+                    let pool = (*outputindex..(*outputindex + *outputcount))
+                        .map(OutputIndex::from)
+                        .find_map(|outputindex| {
+                            let outputtype =
+                                outputindex_to_outputtype_iter.unwrap_get_inner(outputindex);
+                            let typeindex =
+                                outputindex_to_typeindex_iter.unwrap_get_inner(outputindex);
+
+                            let address = match outputtype {
+                                OutputType::P2PK65 => Some(AddressBytes::from(
+                                    p2pk65addressindex_to_p2pk65bytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2PK33 => Some(AddressBytes::from(
+                                    p2pk33addressindex_to_p2pk33bytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2PKH => Some(AddressBytes::from(
+                                    p2pkhaddressindex_to_p2pkhbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2SH => Some(AddressBytes::from(
+                                    p2shaddressindex_to_p2shbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2WPKH => Some(AddressBytes::from(
+                                    p2wpkhaddressindex_to_p2wpkhbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2WSH => Some(AddressBytes::from(
+                                    p2wshaddressindex_to_p2wshbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2TR => Some(AddressBytes::from(
+                                    p2traddressindex_to_p2trbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2A => Some(AddressBytes::from(
+                                    p2aaddressindex_to_p2abytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                _ => None,
+                            };
+
+                            address
+                                .and_then(|address| pools.find_from_address(&address.to_string()))
+                        })
+                        .or_else(|| pools.find_from_coinbase_tag(&coinbase_tag))
+                        .unwrap_or(unknown);
+
+                    *res.entry(pool.name).or_default() += 1;
+                });
+
+            let mut v = res.into_iter().map(|(k, v)| (v, k)).collect::<Vec<_>>();
+            v.sort_unstable();
+            println!("{:#?}", v);
+            println!("{:#?}", v.len());
+
+            Ok(())
+        })?
+        .join()
+        .unwrap()
+}

crates/brk_computer/src/blks.rs

@@ -0,0 +1,141 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_indexer::Indexer;
+use brk_parser::Parser;
+use brk_structs::{BlkPosition, Height, TxIndex, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, CompressedVec, Database, Exit,
+    GenericStoredVec, PAGE_SIZE, VecIterator,
+};
+
+use super::{Indexes, indexes};
+
+#[derive(Clone)]
+pub struct Vecs {
+    db: Database,
+
+    pub height_to_position: CompressedVec<Height, BlkPosition>,
+    pub txindex_to_position: CompressedVec<TxIndex, BlkPosition>,
+}
+
+impl Vecs {
+    pub fn forced_import(parent_path: &Path, parent_version: Version) -> Result<Self> {
+        let db = Database::open(&parent_path.join("blks"))?;
+        db.set_min_len(PAGE_SIZE * 1_000_000)?;
+
+        let version = parent_version + Version::ZERO;
+
+        let this = Self {
+            height_to_position: CompressedVec::forced_import(
+                &db,
+                "position",
+                version + Version::TWO,
+            )?,
+            txindex_to_position: CompressedVec::forced_import(
+                &db,
+                "position",
+                version + Version::TWO,
+            )?,
+
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        parser: &Parser,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexer, indexes, starting_indexes, parser, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        parser: &Parser,
+        exit: &Exit,
+    ) -> Result<()> {
+        let min_txindex =
+            TxIndex::from(self.txindex_to_position.len()).min(starting_indexes.txindex);
+
+        let Some(min_height) = indexes
+            .txindex_to_height
+            .iter()
+            .get_inner(min_txindex)
+            .map(|h| h.min(starting_indexes.height))
+        else {
+            return Ok(());
+        };
+
+        let mut height_to_first_txindex_iter = indexer.vecs.height_to_first_txindex.iter();
+
+        parser
+            .parse(
+                Some(min_height),
+                Some((indexer.vecs.height_to_first_txindex.len() - 1).into()),
+            )
+            .iter()
+            .try_for_each(|block| -> Result<()> {
+                let height = block.height();
+
+                self.height_to_position.forced_push_at(
+                    height,
+                    block.metadata().position(),
+                    exit,
+                )?;
+
+                let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+
+                block.tx_metadata().iter().enumerate().try_for_each(
+                    |(index, metadata)| -> Result<()> {
+                        let txindex = txindex + index;
+                        self.txindex_to_position.forced_push_at(
+                            txindex,
+                            metadata.position(),
+                            exit,
+                        )?;
+                        Ok(())
+                    },
+                )?;
+
+                if *height % 1_000 == 0 {
+                    let _lock = exit.lock();
+                    self.height_to_position.flush()?;
+                    self.txindex_to_position.flush()?;
+                }
+                Ok(())
+            })?;
+
+        let _lock = exit.lock();
+        self.height_to_position.flush()?;
+        self.txindex_to_position.flush()?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        Box::new(
+            [
+                &self.height_to_position as &dyn AnyCollectableVec,
+                &self.txindex_to_position,
+            ]
+            .into_iter(),
+        )
+    }
+}

crates/brk_computer/src/blocks.rs

@@ -1,264 +0,0 @@
-use std::path::Path;
-
-use brk_error::Result;
-use brk_indexer::Indexer;
-use brk_structs::{
-    CheckedSub, DifficultyEpoch, HalvingEpoch, Height, StoredU32, StoredU64, Timestamp, Version,
-    Weight,
-};
-use vecdb::{AnyCollectableVec, Database, EagerVec, Exit, PAGE_SIZE, VecIterator};
-
-use crate::grouped::Source;
-
-use super::{
-    Indexes,
-    grouped::{ComputedVecsFromDateIndex, ComputedVecsFromHeight, VecBuilderOptions},
-    indexes,
-};
-
-const VERSION: Version = Version::ZERO;
-
-#[derive(Clone)]
-pub struct Vecs {
-    db: Database,
-
-    pub height_to_interval: EagerVec<Height, Timestamp>,
-    pub height_to_vbytes: EagerVec<Height, StoredU64>,
-    pub difficultyepoch_to_timestamp: EagerVec<DifficultyEpoch, Timestamp>,
-    pub halvingepoch_to_timestamp: EagerVec<HalvingEpoch, Timestamp>,
-    pub timeindexes_to_timestamp: ComputedVecsFromDateIndex<Timestamp>,
-    pub indexes_to_block_count: ComputedVecsFromHeight<StoredU32>,
-    pub indexes_to_block_interval: ComputedVecsFromHeight<Timestamp>,
-    pub indexes_to_block_size: ComputedVecsFromHeight<StoredU64>,
-    pub indexes_to_block_vbytes: ComputedVecsFromHeight<StoredU64>,
-    pub indexes_to_block_weight: ComputedVecsFromHeight<Weight>,
-}
-
-impl Vecs {
-    pub fn forced_import(parent: &Path, version: Version, indexes: &indexes::Vecs) -> Result<Self> {
-        let db = Database::open(&parent.join("blocks"))?;
-        db.set_min_len(PAGE_SIZE * 1_000_000)?;
-
-        Ok(Self {
-            height_to_interval: EagerVec::forced_import_compressed(
-                &db,
-                "interval",
-                version + VERSION + Version::ZERO,
-            )?,
-            timeindexes_to_timestamp: ComputedVecsFromDateIndex::forced_import(
-                &db,
-                "timestamp",
-                Source::Compute,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_first(),
-            )?,
-            indexes_to_block_interval: ComputedVecsFromHeight::forced_import(
-                &db,
-                "block_interval",
-                Source::None,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default()
-                    .add_percentiles()
-                    .add_minmax()
-                    .add_average(),
-            )?,
-            indexes_to_block_count: ComputedVecsFromHeight::forced_import(
-                &db,
-                "block_count",
-                Source::Compute,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_sum().add_cumulative(),
-            )?,
-            indexes_to_block_weight: ComputedVecsFromHeight::forced_import(
-                &db,
-                "block_weight",
-                Source::None,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_sum().add_cumulative(),
-            )?,
-            indexes_to_block_size: ComputedVecsFromHeight::forced_import(
-                &db,
-                "block_size",
-                Source::None,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_sum().add_cumulative(),
-            )?,
-            height_to_vbytes: EagerVec::forced_import_compressed(
-                &db,
-                "vbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            indexes_to_block_vbytes: ComputedVecsFromHeight::forced_import(
-                &db,
-                "block_vbytes",
-                Source::None,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_sum().add_cumulative(),
-            )?,
-            difficultyepoch_to_timestamp: EagerVec::forced_import_compressed(
-                &db,
-                "timestamp",
-                version + VERSION + Version::ZERO,
-            )?,
-            halvingepoch_to_timestamp: EagerVec::forced_import_compressed(
-                &db,
-                "timestamp",
-                version + VERSION + Version::ZERO,
-            )?,
-
-            db,
-        })
-    }
-
-    pub fn compute(
-        &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        starting_indexes: &Indexes,
-        exit: &Exit,
-    ) -> Result<()> {
-        self.compute_(indexer, indexes, starting_indexes, exit)?;
-        self.db.flush_then_punch()?;
-        Ok(())
-    }
-
-    fn compute_(
-        &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        starting_indexes: &Indexes,
-        exit: &Exit,
-    ) -> Result<()> {
-        self.timeindexes_to_timestamp.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_transform(
-                    starting_indexes.dateindex,
-                    &indexes.dateindex_to_date,
-                    |(di, d, ..)| (di, Timestamp::from(d)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        self.indexes_to_block_count.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |v, indexer, _, starting_indexes, exit| {
-                v.compute_range(
-                    starting_indexes.height,
-                    &indexer.vecs.height_to_weight,
-                    |h| (h, StoredU32::from(1_u32)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        let mut height_to_timestamp_iter = indexer.vecs.height_to_timestamp.iter();
-        self.height_to_interval.compute_transform(
-            starting_indexes.height,
-            &indexer.vecs.height_to_timestamp,
-            |(height, timestamp, ..)| {
-                let interval = height.decremented().map_or(Timestamp::ZERO, |prev_h| {
-                    let prev_timestamp = height_to_timestamp_iter.unwrap_get_inner(prev_h);
-                    timestamp
-                        .checked_sub(prev_timestamp)
-                        .unwrap_or(Timestamp::ZERO)
-                });
-                (height, interval)
-            },
-            exit,
-        )?;
-
-        self.indexes_to_block_interval.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&self.height_to_interval),
-        )?;
-
-        self.indexes_to_block_weight.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&indexer.vecs.height_to_weight),
-        )?;
-
-        self.indexes_to_block_size.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&indexer.vecs.height_to_total_size),
-        )?;
-
-        self.height_to_vbytes.compute_transform(
-            starting_indexes.height,
-            &indexer.vecs.height_to_weight,
-            |(h, w, ..)| {
-                (
-                    h,
-                    StoredU64::from(bitcoin::Weight::from(w).to_vbytes_floor()),
-                )
-            },
-            exit,
-        )?;
-
-        self.indexes_to_block_vbytes.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&self.height_to_vbytes),
-        )?;
-
-        let mut height_to_timestamp_iter = indexer.vecs.height_to_timestamp.iter();
-
-        self.difficultyepoch_to_timestamp.compute_transform(
-            starting_indexes.difficultyepoch,
-            &indexes.difficultyepoch_to_first_height,
-            |(i, h, ..)| (i, height_to_timestamp_iter.unwrap_get_inner(h)),
-            exit,
-        )?;
-
-        self.halvingepoch_to_timestamp.compute_transform(
-            starting_indexes.halvingepoch,
-            &indexes.halvingepoch_to_first_height,
-            |(i, h, ..)| (i, height_to_timestamp_iter.unwrap_get_inner(h)),
-            exit,
-        )?;
-
-        Ok(())
-    }
-
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            vec![
-                &self.height_to_interval as &dyn AnyCollectableVec,
-                &self.height_to_vbytes,
-                &self.difficultyepoch_to_timestamp,
-                &self.halvingepoch_to_timestamp,
-            ],
-            self.timeindexes_to_timestamp.vecs(),
-            self.indexes_to_block_count.vecs(),
-            self.indexes_to_block_interval.vecs(),
-            self.indexes_to_block_size.vecs(),
-            self.indexes_to_block_vbytes.vecs(),
-            self.indexes_to_block_weight.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
-    }
-}

crates/brk_computer/src/cointime.rs

@@ -1,21 +1,20 @@
 use std::path::Path;
 
 use brk_error::Result;
-use brk_indexer::Indexer;
+use brk_structs::{Bitcoin, CheckedSub, Dollars, StoredF32, StoredF64, Version};
-use brk_structs::{Bitcoin, CheckedSub, Dollars, StoredF64, Version};
 use vecdb::{AnyCollectableVec, Database, Exit, PAGE_SIZE, VecIterator};
 
+use crate::grouped::ComputedVecsFromDateIndex;
+
 use super::{
-    Indexes,
+    Indexes, chain,
     grouped::{
         ComputedRatioVecsFromDateIndex, ComputedValueVecsFromHeight, ComputedVecsFromHeight,
         Source, VecBuilderOptions,
     },
-    indexes, price, stateful, transactions,
+    indexes, price, stateful,
 };
 
-const VERSION: Version = Version::ZERO;
-
 #[derive(Clone)]
 pub struct Vecs {
     db: Database,
@@ -43,27 +42,32 @@ pub struct Vecs {
     pub indexes_to_cointime_price: ComputedVecsFromHeight<Dollars>,
     pub indexes_to_cointime_cap: ComputedVecsFromHeight<Dollars>,
     pub indexes_to_cointime_price_ratio: ComputedRatioVecsFromDateIndex,
-    // pub indexes_to_thermo_cap_relative_to_investor_cap: ComputedValueVecsFromHeight,
+    pub indexes_to_cointime_adj_inflation_rate: ComputedVecsFromDateIndex<StoredF32>,
+    pub indexes_to_cointime_adj_tx_btc_velocity: ComputedVecsFromDateIndex<StoredF64>,
+    pub indexes_to_cointime_adj_tx_usd_velocity: ComputedVecsFromDateIndex<StoredF64>,
+    // pub indexes_to_thermo_cap_rel_to_investor_cap: ComputedValueVecsFromHeight,
 }
 
 impl Vecs {
     pub fn forced_import(
-        parent: &Path,
+        parent_path: &Path,
-        version: Version,
+        parent_version: Version,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
     ) -> Result<Self> {
-        let db = Database::open(&parent.join("cointime"))?;
+        let db = Database::open(&parent_path.join("cointime"))?;
         db.set_min_len(PAGE_SIZE * 1_000_000)?;
 
         let compute_dollars = price.is_some();
 
-        Ok(Self {
+        let version = parent_version + Version::ZERO;
+
+        let this = Self {
             indexes_to_coinblocks_created: ComputedVecsFromHeight::forced_import(
                 &db,
                 "coinblocks_created",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_sum().add_cumulative(),
             )?,
@@ -71,7 +75,7 @@ impl Vecs {
                 &db,
                 "coinblocks_stored",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_sum().add_cumulative(),
             )?,
@@ -79,7 +83,7 @@ impl Vecs {
                 &db,
                 "liveliness",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -87,7 +91,7 @@ impl Vecs {
                 &db,
                 "vaultedness",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -95,7 +99,7 @@ impl Vecs {
                 &db,
                 "activity_to_vaultedness_ratio",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -103,7 +107,7 @@ impl Vecs {
                 &db,
                 "vaulted_supply",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 VecBuilderOptions::default().add_last(),
                 compute_dollars,
                 indexes,
@@ -112,7 +116,7 @@ impl Vecs {
                 &db,
                 "active_supply",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 VecBuilderOptions::default().add_last(),
                 compute_dollars,
                 indexes,
@@ -121,7 +125,7 @@ impl Vecs {
                 &db,
                 "thermo_cap",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -129,7 +133,7 @@ impl Vecs {
                 &db,
                 "investor_cap",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -137,7 +141,7 @@ impl Vecs {
                 &db,
                 "vaulted_cap",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -145,7 +149,7 @@ impl Vecs {
                 &db,
                 "active_cap",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -153,7 +157,7 @@ impl Vecs {
                 &db,
                 "vaulted_price",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -161,7 +165,7 @@ impl Vecs {
                 &db,
                 "vaulted_price",
                 Source::None,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 true,
             )?,
@@ -169,7 +173,7 @@ impl Vecs {
                 &db,
                 "active_price",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -177,7 +181,7 @@ impl Vecs {
                 &db,
                 "active_price",
                 Source::None,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 true,
             )?,
@@ -185,7 +189,7 @@ impl Vecs {
                 &db,
                 "true_market_mean",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -193,7 +197,7 @@ impl Vecs {
                 &db,
                 "true_market_mean",
                 Source::None,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 true,
             )?,
@@ -201,7 +205,7 @@ impl Vecs {
                 &db,
                 "cointime_value_destroyed",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_sum().add_cumulative(),
             )?,
@@ -209,7 +213,7 @@ impl Vecs {
                 &db,
                 "cointime_value_created",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_sum().add_cumulative(),
             )?,
@@ -217,7 +221,7 @@ impl Vecs {
                 &db,
                 "cointime_value_stored",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_sum().add_cumulative(),
             )?,
@@ -225,7 +229,7 @@ impl Vecs {
                 &db,
                 "cointime_price",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -233,7 +237,7 @@ impl Vecs {
                 &db,
                 "cointime_cap",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -241,35 +245,58 @@ impl Vecs {
                 &db,
                 "cointime_price",
                 Source::None,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 true,
             )?,
+            indexes_to_cointime_adj_inflation_rate: ComputedVecsFromDateIndex::forced_import(
+                &db,
+                "cointime_adj_inflation_rate",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_cointime_adj_tx_btc_velocity: ComputedVecsFromDateIndex::forced_import(
+                &db,
+                "cointime_adj_tx_btc_velocity",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_cointime_adj_tx_usd_velocity: ComputedVecsFromDateIndex::forced_import(
+                &db,
+                "cointime_adj_tx_usd_velocity",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
 
             db,
-        })
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
     }
 
     #[allow(clippy::too_many_arguments)]
     pub fn compute(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         price: Option<&price::Vecs>,
-        transactions: &transactions::Vecs,
+        chain: &chain::Vecs,
         stateful: &stateful::Vecs,
         exit: &Exit,
     ) -> Result<()> {
-        self.compute_(
+        self.compute_(indexes, starting_indexes, price, chain, stateful, exit)?;
-            indexer,
-            indexes,
-            starting_indexes,
-            price,
-            transactions,
-            stateful,
-            exit,
-        )?;
         self.db.flush_then_punch()?;
         Ok(())
     }
@@ -277,22 +304,17 @@ impl Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         price: Option<&price::Vecs>,
-        transactions: &transactions::Vecs,
+        chain: &chain::Vecs,
         stateful: &stateful::Vecs,
         exit: &Exit,
     ) -> Result<()> {
         let circulating_supply = &stateful.utxo_cohorts.all.1.height_to_supply;
 
-        self.indexes_to_coinblocks_created.compute_all(
+        self.indexes_to_coinblocks_created
-            indexer,
+            .compute_all(indexes, starting_indexes, exit, |vec| {
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, _, starting_indexes, exit| {
                 vec.compute_transform(
                     starting_indexes.height,
                     circulating_supply,
@@ -300,18 +322,13 @@ impl Vecs {
                     exit,
                 )?;
                 Ok(())
-            },
+            })?;
-        )?;
 
         let indexes_to_coinblocks_destroyed =
             &stateful.utxo_cohorts.all.1.indexes_to_coinblocks_destroyed;
 
-        self.indexes_to_coinblocks_stored.compute_all(
+        self.indexes_to_coinblocks_stored
-            indexer,
+            .compute_all(indexes, starting_indexes, exit, |vec| {
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, _, starting_indexes, exit| {
                 let mut coinblocks_destroyed_iter = indexes_to_coinblocks_destroyed
                     .height
                     .as_ref()
@@ -327,15 +344,10 @@ impl Vecs {
                     exit,
                 )?;
                 Ok(())
-            },
+            })?;
-        )?;
 
-        self.indexes_to_liveliness.compute_all(
+        self.indexes_to_liveliness
-            indexer,
+            .compute_all(indexes, starting_indexes, exit, |vec| {
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, _, starting_indexes, exit| {
                 vec.compute_divide(
                     starting_indexes.height,
                     indexes_to_coinblocks_destroyed
@@ -347,16 +359,11 @@ impl Vecs {
                     exit,
                 )?;
                 Ok(())
-            },
+            })?;
-        )?;
         let liveliness = &self.indexes_to_liveliness;
 
-        self.indexes_to_vaultedness.compute_all(
+        self.indexes_to_vaultedness
-            indexer,
+            .compute_all(indexes, starting_indexes, exit, |vec| {
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, _, starting_indexes, exit| {
                 vec.compute_transform(
                     starting_indexes.height,
                     liveliness.height.as_ref().unwrap(),
@@ -364,16 +371,14 @@ impl Vecs {
                     exit,
                 )?;
                 Ok(())
-            },
+            })?;
-        )?;
         let vaultedness = &self.indexes_to_vaultedness;
 
         self.indexes_to_activity_to_vaultedness_ratio.compute_all(
-            indexer,
             indexes,
             starting_indexes,
             exit,
-            |vec, _, _, starting_indexes, exit| {
+            |vec| {
                 vec.compute_divide(
                     starting_indexes.height,
                     liveliness.height.as_ref().unwrap(),
@@ -385,12 +390,11 @@ impl Vecs {
         )?;
 
         self.indexes_to_vaulted_supply.compute_all(
-            indexer,
             indexes,
             price,
             starting_indexes,
             exit,
-            |vec, _, _, starting_indexes, exit| {
+            |vec| {
                 vec.compute_multiply(
                     starting_indexes.height,
                     circulating_supply,
@@ -402,12 +406,11 @@ impl Vecs {
         )?;
 
         self.indexes_to_active_supply.compute_all(
-            indexer,
             indexes,
             price,
             starting_indexes,
             exit,
-            |vec, _, _, starting_indexes, exit| {
+            |vec| {
                 vec.compute_multiply(
                     starting_indexes.height,
                     circulating_supply,
@@ -418,6 +421,32 @@ impl Vecs {
             },
         )?;
 
+        self.indexes_to_cointime_adj_inflation_rate
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_multiply(
+                    starting_indexes.dateindex,
+                    self.indexes_to_activity_to_vaultedness_ratio
+                        .dateindex
+                        .unwrap_last(),
+                    chain.indexes_to_inflation_rate.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_cointime_adj_tx_btc_velocity
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_multiply(
+                    starting_indexes.dateindex,
+                    self.indexes_to_activity_to_vaultedness_ratio
+                        .dateindex
+                        .unwrap_last(),
+                    chain.indexes_to_tx_btc_velocity.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
         if let Some(price) = price {
             let realized_cap = stateful
                 .utxo_cohorts
@@ -438,15 +467,11 @@ impl Vecs {
                 .as_ref()
                 .unwrap();
 
-            self.indexes_to_thermo_cap.compute_all(
+            self.indexes_to_thermo_cap
-                indexer,
+                .compute_all(indexes, starting_indexes, exit, |vec| {
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
                     vec.compute_transform(
                         starting_indexes.height,
-                        transactions
+                        chain
                             .indexes_to_subsidy
                             .dollars
                             .as_ref()
@@ -457,15 +482,10 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
+                })?;
-            )?;
 
-            self.indexes_to_investor_cap.compute_all(
+            self.indexes_to_investor_cap
-                indexer,
+                .compute_all(indexes, starting_indexes, exit, |vec| {
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
                     vec.compute_subtract(
                         starting_indexes.height,
                         realized_cap,
@@ -473,15 +493,10 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
+                })?;
-            )?;
 
-            self.indexes_to_vaulted_cap.compute_all(
+            self.indexes_to_vaulted_cap
-                indexer,
+                .compute_all(indexes, starting_indexes, exit, |vec| {
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
                     vec.compute_divide(
                         starting_indexes.height,
                         realized_cap,
@@ -489,15 +504,10 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
+                })?;
-            )?;
 
-            self.indexes_to_active_cap.compute_all(
+            self.indexes_to_active_cap
-                indexer,
+                .compute_all(indexes, starting_indexes, exit, |vec| {
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
                     vec.compute_multiply(
                         starting_indexes.height,
                         realized_cap,
@@ -505,15 +515,10 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
+                })?;
-            )?;
 
-            self.indexes_to_vaulted_price.compute_all(
+            self.indexes_to_vaulted_price
-                indexer,
+                .compute_all(indexes, starting_indexes, exit, |vec| {
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
                     vec.compute_divide(
                         starting_indexes.height,
                         realized_price,
@@ -521,24 +526,17 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
+                })?;
-            )?;
 
             self.indexes_to_vaulted_price_ratio.compute_rest(
-                indexer,
-                indexes,
                 price,
                 starting_indexes,
                 exit,
                 Some(self.indexes_to_vaulted_price.dateindex.unwrap_last()),
             )?;
 
-            self.indexes_to_active_price.compute_all(
+            self.indexes_to_active_price
-                indexer,
+                .compute_all(indexes, starting_indexes, exit, |vec| {
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
                     vec.compute_multiply(
                         starting_indexes.height,
                         realized_price,
@@ -546,12 +544,9 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
+                })?;
-            )?;
 
             self.indexes_to_active_price_ratio.compute_rest(
-                indexer,
-                indexes,
                 price,
                 starting_indexes,
                 exit,
@@ -559,11 +554,10 @@ impl Vecs {
             )?;
 
             self.indexes_to_true_market_mean.compute_all(
-                indexer,
                 indexes,
                 starting_indexes,
                 exit,
-                |vec, _, _, starting_indexes, exit| {
+                |vec| {
                     vec.compute_divide(
                         starting_indexes.height,
                         self.indexes_to_investor_cap.height.as_ref().unwrap(),
@@ -579,8 +573,6 @@ impl Vecs {
             )?;
 
             self.indexes_to_true_market_mean_ratio.compute_rest(
-                indexer,
-                indexes,
                 price,
                 starting_indexes,
                 exit,
@@ -588,16 +580,15 @@ impl Vecs {
             )?;
 
             self.indexes_to_cointime_value_destroyed.compute_all(
-                indexer,
                 indexes,
                 starting_indexes,
                 exit,
-                |vec, _, _, starting_indexes, exit| {
+                |vec| {
                     // TODO: Another example when the callback should be applied to each index, instead of to base then merging from more granular to less
                     // The price taken won't be correct for time based indexes
                     vec.compute_multiply(
                         starting_indexes.height,
-                        &price.chainindexes_to_close.height,
+                        &price.chainindexes_to_price_close.height,
                         indexes_to_coinblocks_destroyed.height.as_ref().unwrap(),
                         exit,
                     )?;
@@ -606,14 +597,13 @@ impl Vecs {
             )?;
 
             self.indexes_to_cointime_value_created.compute_all(
-                indexer,
                 indexes,
                 starting_indexes,
                 exit,
-                |vec, _, _, starting_indexes, exit| {
+                |vec| {
                     vec.compute_multiply(
                         starting_indexes.height,
-                        &price.chainindexes_to_close.height,
+                        &price.chainindexes_to_price_close.height,
                         self.indexes_to_coinblocks_created.height.as_ref().unwrap(),
                         exit,
                     )?;
@@ -622,14 +612,13 @@ impl Vecs {
             )?;
 
             self.indexes_to_cointime_value_stored.compute_all(
-                indexer,
                 indexes,
                 starting_indexes,
                 exit,
-                |vec, _, _, starting_indexes, exit| {
+                |vec| {
                     vec.compute_multiply(
                         starting_indexes.height,
-                        &price.chainindexes_to_close.height,
+                        &price.chainindexes_to_price_close.height,
                         self.indexes_to_coinblocks_stored.height.as_ref().unwrap(),
                         exit,
                     )?;
@@ -637,12 +626,8 @@ impl Vecs {
                 },
             )?;
 
-            self.indexes_to_cointime_price.compute_all(
+            self.indexes_to_cointime_price
-                indexer,
+                .compute_all(indexes, starting_indexes, exit, |vec| {
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
                     vec.compute_divide(
                         starting_indexes.height,
                         self.indexes_to_cointime_value_destroyed
@@ -654,15 +639,10 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
+                })?;
-            )?;
 
-            self.indexes_to_cointime_cap.compute_all(
+            self.indexes_to_cointime_cap
-                indexer,
+                .compute_all(indexes, starting_indexes, exit, |vec| {
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
                     vec.compute_multiply(
                         starting_indexes.height,
                         self.indexes_to_cointime_price.height.as_ref().unwrap(),
@@ -670,50 +650,100 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
+                })?;
-            )?;
 
             self.indexes_to_cointime_price_ratio.compute_rest(
-                indexer,
-                indexes,
                 price,
                 starting_indexes,
                 exit,
                 Some(self.indexes_to_cointime_price.dateindex.unwrap_last()),
             )?;
+
+            self.indexes_to_cointime_adj_tx_usd_velocity.compute_all(
+                starting_indexes,
+                exit,
+                |v| {
+                    v.compute_multiply(
+                        starting_indexes.dateindex,
+                        self.indexes_to_activity_to_vaultedness_ratio
+                            .dateindex
+                            .unwrap_last(),
+                        chain.indexes_to_tx_usd_velocity.dateindex.as_ref().unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                },
+            )?;
         }
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
-            self.indexes_to_coinblocks_created.vecs(),
+            Box::new(std::iter::empty());
-            self.indexes_to_coinblocks_stored.vecs(),
+
-            self.indexes_to_liveliness.vecs(),
+        iter = Box::new(
-            self.indexes_to_vaultedness.vecs(),
+            iter.chain(
-            self.indexes_to_activity_to_vaultedness_ratio.vecs(),
+                self.indexes_to_cointime_adj_inflation_rate
-            self.indexes_to_vaulted_supply.vecs(),
+                    .iter_any_collectable(),
-            self.indexes_to_active_supply.vecs(),
+            ),
-            self.indexes_to_thermo_cap.vecs(),
+        );
-            self.indexes_to_investor_cap.vecs(),
+        iter = Box::new(
-            self.indexes_to_vaulted_cap.vecs(),
+            iter.chain(
-            self.indexes_to_active_cap.vecs(),
+                self.indexes_to_cointime_adj_tx_btc_velocity
-            self.indexes_to_vaulted_price.vecs(),
+                    .iter_any_collectable(),
-            self.indexes_to_vaulted_price_ratio.vecs(),
+            ),
-            self.indexes_to_active_price.vecs(),
+        );
-            self.indexes_to_active_price_ratio.vecs(),
+        iter = Box::new(
-            self.indexes_to_true_market_mean.vecs(),
+            iter.chain(
-            self.indexes_to_true_market_mean_ratio.vecs(),
+                self.indexes_to_cointime_adj_tx_usd_velocity
-            self.indexes_to_cointime_price.vecs(),
+                    .iter_any_collectable(),
-            self.indexes_to_cointime_cap.vecs(),
+            ),
-            self.indexes_to_cointime_price_ratio.vecs(),
+        );
-            self.indexes_to_cointime_value_destroyed.vecs(),
+        iter = Box::new(iter.chain(self.indexes_to_coinblocks_created.iter_any_collectable()));
-            self.indexes_to_cointime_value_created.vecs(),
+        iter = Box::new(iter.chain(self.indexes_to_coinblocks_stored.iter_any_collectable()));
-            self.indexes_to_cointime_value_stored.vecs(),
+        iter = Box::new(iter.chain(self.indexes_to_liveliness.iter_any_collectable()));
-        ]
+        iter = Box::new(iter.chain(self.indexes_to_vaultedness.iter_any_collectable()));
-        .into_iter()
+        iter = Box::new(
-        .flatten()
+            iter.chain(
-        .collect::<Vec<_>>()
+                self.indexes_to_activity_to_vaultedness_ratio
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_supply.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_supply.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_thermo_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_investor_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_price.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_price_ratio.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_price.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_price_ratio.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_true_market_mean.iter_any_collectable()));
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_true_market_mean_ratio
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(iter.chain(self.indexes_to_cointime_price.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_cointime_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_cointime_price_ratio.iter_any_collectable()));
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_value_destroyed
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_value_created
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(iter.chain(self.indexes_to_cointime_value_stored.iter_any_collectable()));
+        iter
     }
 }

crates/brk_computer/src/constants.rs

@@ -1,8 +1,7 @@
 use std::path::Path;
 
 use brk_error::Result;
-use brk_indexer::Indexer;
+use brk_structs::{StoredF32, StoredI16, StoredU16, Version};
-use brk_structs::{StoredI16, StoredU16, Version};
 use vecdb::{AnyCollectableVec, AnyVec, Database, Exit};
 
 use crate::grouped::Source;
@@ -13,8 +12,6 @@ use super::{
     indexes,
 };
 
-const VERSION: Version = Version::ZERO;
-
 #[derive(Clone)]
 pub struct Vecs {
     db: Database,
@@ -24,8 +21,11 @@ pub struct Vecs {
     pub constant_2: ComputedVecsFromHeight<StoredU16>,
     pub constant_3: ComputedVecsFromHeight<StoredU16>,
     pub constant_4: ComputedVecsFromHeight<StoredU16>,
+    pub constant_38_2: ComputedVecsFromHeight<StoredF32>,
     pub constant_50: ComputedVecsFromHeight<StoredU16>,
+    pub constant_61_8: ComputedVecsFromHeight<StoredF32>,
     pub constant_100: ComputedVecsFromHeight<StoredU16>,
+    pub constant_600: ComputedVecsFromHeight<StoredU16>,
     pub constant_minus_1: ComputedVecsFromHeight<StoredI16>,
     pub constant_minus_2: ComputedVecsFromHeight<StoredI16>,
     pub constant_minus_3: ComputedVecsFromHeight<StoredI16>,
@@ -33,15 +33,21 @@ pub struct Vecs {
 }
 
 impl Vecs {
-    pub fn forced_import(parent: &Path, version: Version, indexes: &indexes::Vecs) -> Result<Self> {
+    pub fn forced_import(
-        let db = Database::open(&parent.join("constants"))?;
+        parent_path: &Path,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+    ) -> Result<Self> {
+        let db = Database::open(&parent_path.join("constants"))?;
 
-        Ok(Self {
+        let version = parent_version + Version::ZERO;
+
+        let this = Self {
             constant_0: ComputedVecsFromHeight::forced_import(
                 &db,
                 "constant_0",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -49,7 +55,7 @@ impl Vecs {
                 &db,
                 "constant_1",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -57,7 +63,7 @@ impl Vecs {
                 &db,
                 "constant_2",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -65,7 +71,7 @@ impl Vecs {
                 &db,
                 "constant_3",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -73,7 +79,15 @@ impl Vecs {
                 &db,
                 "constant_4",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_38_2: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_38_2",
+                Source::Compute,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -81,7 +95,15 @@ impl Vecs {
                 &db,
                 "constant_50",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_61_8: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_61_8",
+                Source::Compute,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -89,7 +111,15 @@ impl Vecs {
                 &db,
                 "constant_100",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_600: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_600",
+                Source::Compute,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -97,7 +127,7 @@ impl Vecs {
                 &db,
                 "constant_minus_1",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -105,7 +135,7 @@ impl Vecs {
                 &db,
                 "constant_minus_2",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -113,7 +143,7 @@ impl Vecs {
                 &db,
                 "constant_minus_3",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -121,240 +151,124 @@ impl Vecs {
                 &db,
                 "constant_minus_4",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
 
             db,
-        })
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
     }
 
     pub fn compute(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        self.compute_(indexer, indexes, starting_indexes, exit)?;
+        self.compute_(indexes, starting_indexes, exit)?;
         self.db.flush_then_punch()?;
         Ok(())
     }
 
     fn compute_(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        self.constant_0.compute_all(
+        [
-            indexer,
+            (&mut self.constant_0, 0),
-            indexes,
+            (&mut self.constant_1, 1),
-            starting_indexes,
+            (&mut self.constant_2, 2),
-            exit,
+            (&mut self.constant_3, 3),
-            |vec, _, indexes, starting_indexes, exit| {
+            (&mut self.constant_4, 4),
+            (&mut self.constant_50, 50),
+            (&mut self.constant_100, 100),
+            (&mut self.constant_600, 600),
+        ]
+        .into_iter()
+        .try_for_each(|(vec, value)| {
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
                 vec.compute_to(
                     starting_indexes.height,
                     indexes.height_to_date.len(),
                     indexes.height_to_date.version(),
-                    |i| (i, StoredU16::new(0)),
+                    |i| (i, StoredU16::new(value)),
                     exit,
                 )?;
                 Ok(())
-            },
+            })
-        )?;
+        })?;
 
-        self.constant_1.compute_all(
+        [
-            indexer,
+            (&mut self.constant_minus_1, -1),
-            indexes,
+            (&mut self.constant_minus_2, -2),
-            starting_indexes,
+            (&mut self.constant_minus_3, 3),
-            exit,
+            (&mut self.constant_minus_4, 4),
-            |vec, _, indexes, starting_indexes, exit| {
+        ]
+        .into_iter()
+        .try_for_each(|(vec, value)| {
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
                 vec.compute_to(
                     starting_indexes.height,
                     indexes.height_to_date.len(),
                     indexes.height_to_date.version(),
-                    |i| (i, StoredU16::new(1)),
+                    |i| (i, StoredI16::new(value)),
                     exit,
                 )?;
                 Ok(())
-            },
+            })
-        )?;
+        })?;
 
-        self.constant_2.compute_all(
+        [
-            indexer,
+            (&mut self.constant_38_2, 38.2),
-            indexes,
+            (&mut self.constant_61_8, 61.8),
-            starting_indexes,
+        ]
-            exit,
+        .into_iter()
-            |vec, _, indexes, starting_indexes, exit| {
+        .try_for_each(|(vec, value)| {
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
                 vec.compute_to(
                     starting_indexes.height,
                     indexes.height_to_date.len(),
                     indexes.height_to_date.version(),
-                    |i| (i, StoredU16::new(2)),
+                    |i| (i, StoredF32::from(value)),
                     exit,
                 )?;
                 Ok(())
-            },
+            })
-        )?;
+        })?;
-
-        self.constant_3.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredU16::new(3)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        self.constant_4.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredU16::new(4)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        self.constant_50.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredU16::new(50)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        self.constant_100.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredU16::new(100)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        self.constant_minus_1.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredI16::new(-1)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        self.constant_minus_2.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredI16::new(-2)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        self.constant_minus_3.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredI16::new(-3)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        self.constant_minus_4.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredI16::new(-4)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
-            self.constant_0.vecs(),
+            Box::new(std::iter::empty());
-            self.constant_1.vecs(),
+
-            self.constant_2.vecs(),
+        iter = Box::new(iter.chain(self.constant_0.iter_any_collectable()));
-            self.constant_3.vecs(),
+        iter = Box::new(iter.chain(self.constant_1.iter_any_collectable()));
-            self.constant_4.vecs(),
+        iter = Box::new(iter.chain(self.constant_2.iter_any_collectable()));
-            self.constant_50.vecs(),
+        iter = Box::new(iter.chain(self.constant_3.iter_any_collectable()));
-            self.constant_100.vecs(),
+        iter = Box::new(iter.chain(self.constant_4.iter_any_collectable()));
-            self.constant_minus_1.vecs(),
+        iter = Box::new(iter.chain(self.constant_38_2.iter_any_collectable()));
-            self.constant_minus_2.vecs(),
+        iter = Box::new(iter.chain(self.constant_50.iter_any_collectable()));
-            self.constant_minus_3.vecs(),
+        iter = Box::new(iter.chain(self.constant_61_8.iter_any_collectable()));
-            self.constant_minus_4.vecs(),
+        iter = Box::new(iter.chain(self.constant_100.iter_any_collectable()));
-        ]
+        iter = Box::new(iter.chain(self.constant_600.iter_any_collectable()));
-        .into_iter()
+        iter = Box::new(iter.chain(self.constant_minus_1.iter_any_collectable()));
-        .flatten()
+        iter = Box::new(iter.chain(self.constant_minus_2.iter_any_collectable()));
-        .collect::<Vec<_>>()
+        iter = Box::new(iter.chain(self.constant_minus_3.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_4.iter_any_collectable()));
+
+        iter
     }
 }

crates/brk_computer/src/fetched.rs

@@ -16,30 +16,38 @@ pub struct Vecs {
     db: Database,
     fetcher: Fetcher,
 
-    pub dateindex_to_ohlc_in_cents: RawVec<DateIndex, OHLCCents>,
+    pub dateindex_to_price_ohlc_in_cents: RawVec<DateIndex, OHLCCents>,
-    pub height_to_ohlc_in_cents: RawVec<Height, OHLCCents>,
+    pub height_to_price_ohlc_in_cents: RawVec<Height, OHLCCents>,
 }
 
 impl Vecs {
     pub fn forced_import(parent: &Path, fetcher: Fetcher, version: Version) -> Result<Self> {
         let db = Database::open(&parent.join("fetched"))?;
 
-        Ok(Self {
+        let this = Self {
             fetcher,
 
-            dateindex_to_ohlc_in_cents: RawVec::forced_import(
+            dateindex_to_price_ohlc_in_cents: RawVec::forced_import(
                 &db,
-                "ohlc_in_cents",
+                "price_ohlc_in_cents",
                 version + Version::ZERO,
             )?,
-            height_to_ohlc_in_cents: RawVec::forced_import(
+            height_to_price_ohlc_in_cents: RawVec::forced_import(
                 &db,
-                "ohlc_in_cents",
+                "price_ohlc_in_cents",
                 version + Version::ZERO,
             )?,
 
             db,
-        })
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
     }
 
     pub fn compute(
@@ -64,12 +72,12 @@ impl Vecs {
         let height_to_timestamp = &indexer.vecs.height_to_timestamp;
         let index = starting_indexes
             .height
-            .min(Height::from(self.height_to_ohlc_in_cents.len()));
+            .min(Height::from(self.height_to_price_ohlc_in_cents.len()));
         height_to_timestamp
             .iter_at(index)
             .try_for_each(|(i, v)| -> Result<()> {
                 let v = v.into_owned();
-                self.height_to_ohlc_in_cents.forced_push_at(
+                self.height_to_price_ohlc_in_cents.forced_push_at(
                     i,
                     self.fetcher
                         .get_height(
@@ -84,11 +92,11 @@ impl Vecs {
                 )?;
                 Ok(())
             })?;
-        self.height_to_ohlc_in_cents.safe_flush(exit)?;
+        self.height_to_price_ohlc_in_cents.safe_flush(exit)?;
 
         let index = starting_indexes
             .dateindex
-            .min(DateIndex::from(self.dateindex_to_ohlc_in_cents.len()));
+            .min(DateIndex::from(self.dateindex_to_price_ohlc_in_cents.len()));
         let mut prev = None;
         indexes
             .dateindex_to_date
@@ -98,7 +106,7 @@ impl Vecs {
                 if prev.is_none() {
                     let i = i.unwrap_to_usize();
                     prev.replace(if i > 0 {
-                        self.dateindex_to_ohlc_in_cents
+                        self.dateindex_to_price_ohlc_in_cents
                             .into_iter()
                             .unwrap_get_inner_(i - 1)
                     } else {
@@ -106,7 +114,8 @@ impl Vecs {
                     });
                 }
 
-                let ohlc = if i.unwrap_to_usize() + 100 >= self.dateindex_to_ohlc_in_cents.len()
+                let ohlc = if i.unwrap_to_usize() + 100
+                    >= self.dateindex_to_price_ohlc_in_cents.len()
                     && let Ok(mut ohlc) = self.fetcher.get_date(d)
                 {
                     let prev_open = *prev.as_ref().unwrap().close;
@@ -120,20 +129,21 @@ impl Vecs {
 
                 prev.replace(ohlc.clone());
 
-                self.dateindex_to_ohlc_in_cents
+                self.dateindex_to_price_ohlc_in_cents
                     .forced_push_at(i, ohlc, exit)?;
 
                 Ok(())
             })?;
-        self.dateindex_to_ohlc_in_cents.safe_flush(exit)?;
+        self.dateindex_to_price_ohlc_in_cents.safe_flush(exit)?;
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        vec![
+        [
-            &self.dateindex_to_ohlc_in_cents as &dyn AnyCollectableVec,
+            &self.dateindex_to_price_ohlc_in_cents as &dyn AnyCollectableVec,
-            &self.height_to_ohlc_in_cents,
+            &self.height_to_price_ohlc_in_cents,
         ]
+        .into_iter()
     }
 }

crates/brk_computer/src/grouped/builder_computed.rs

@@ -1,473 +0,0 @@
-use brk_error::Result;
-
-use brk_structs::Version;
-use vecdb::{
-    AnyBoxedIterableVec, AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Computation,
-    ComputedVec, ComputedVecFrom2, Database, Exit, Format, FromCoarserIndex, StoredIndex,
-};
-
-use crate::grouped::{EagerVecBuilder, VecBuilderOptions};
-
-use super::ComputedType;
-
-#[allow(clippy::type_complexity)]
-#[derive(Clone)]
-pub struct ComputedVecBuilder<I, T, S1I, S2T>
-where
-    I: StoredIndex,
-    T: ComputedType,
-    S2T: ComputedType,
-{
-    pub first: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub average: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub sum: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub max: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub min: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub last: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub cumulative: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-}
-
-const VERSION: Version = Version::ZERO;
-
-impl<I, T, S1I, S2T> ComputedVecBuilder<I, T, S1I, S2T>
-where
-    I: StoredIndex,
-    T: ComputedType + 'static,
-    S1I: StoredIndex + 'static + FromCoarserIndex<I>,
-    S2T: ComputedType,
-{
-    #[allow(clippy::too_many_arguments)]
-    pub fn forced_import(
-        db: &Database,
-        name: &str,
-        version: Version,
-        format: Format,
-        computation: Computation,
-        source: Option<AnyBoxedIterableVec<S1I, T>>,
-        source_extra: &EagerVecBuilder<S1I, T>,
-        len_source: AnyBoxedIterableVec<I, S2T>,
-        options: ComputedVecBuilderOptions,
-    ) -> Result<Self> {
-        let only_one_active = options.is_only_one_active();
-
-        let suffix = |s: &str| format!("{name}_{s}");
-
-        let maybe_suffix = |s: &str| {
-            if only_one_active {
-                name.to_string()
-            } else {
-                suffix(s)
-            }
-        };
-
-        Ok(Self {
-            first: options.first.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &maybe_suffix("first"),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra
-                            .first
-                            .as_ref()
-                            .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            source
-                                .next_at(S1I::min_from(i))
-                                .map(|(_, cow)| cow.into_owned())
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            last: options.last.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        name,
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra.last.as_ref().map_or_else(
-                            || {
-                                source
-                                    .as_ref()
-                                    .unwrap_or_else(|| {
-                                        dbg!(db, name, I::to_string());
-                                        panic!()
-                                    })
-                                    .clone()
-                            },
-                            |v| v.clone(),
-                        ),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            source
-                                .next_at(S1I::max_from(i, source.len()))
-                                .map(|(_, cow)| cow.into_owned())
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            min: options.min.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &maybe_suffix("min"),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra
-                            .min
-                            .as_ref()
-                            .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            S1I::inclusive_range_from(i, source.len())
-                                .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
-                                .min()
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            max: options.max.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &maybe_suffix("max"),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra
-                            .max
-                            .as_ref()
-                            .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            S1I::inclusive_range_from(i, source.len())
-                                .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
-                                .max()
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            average: options.average.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &maybe_suffix("average"),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra
-                            .average
-                            .as_ref()
-                            .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            let vec = S1I::inclusive_range_from(i, source.len())
-                                .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
-                                .collect::<Vec<_>>();
-                            if vec.is_empty() {
-                                return None;
-                            }
-                            let mut sum = T::from(0);
-                            let len = vec.len();
-                            vec.into_iter().for_each(|v| sum += v);
-                            Some(sum / len)
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            sum: options.sum.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &(if !options.last && !options.average && !options.min && !options.max {
-                            name.to_string()
-                        } else {
-                            maybe_suffix("sum")
-                        }),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra
-                            .sum
-                            .as_ref()
-                            .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            let vec = S1I::inclusive_range_from(i, source.len())
-                                .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
-                                .collect::<Vec<_>>();
-                            if vec.is_empty() {
-                                return None;
-                            }
-                            let mut sum = T::from(0);
-                            vec.into_iter().for_each(|v| sum += v);
-                            Some(sum)
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            cumulative: options.cumulative.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &suffix("cumulative"),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra.cumulative.as_ref().unwrap().boxed_clone(),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            source
-                                .next_at(S1I::max_from(i, source.len()))
-                                .map(|(_, cow)| cow.into_owned())
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-        })
-    }
-
-    pub fn compute_if_necessary<T2>(
-        &mut self,
-        max_from: I,
-        len_source: &impl AnyIterableVec<I, T2>,
-        exit: &Exit,
-    ) -> Result<()> {
-        if let Some(first) = self.first.as_mut() {
-            first.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(last) = self.last.as_mut() {
-            last.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(min) = self.min.as_mut() {
-            min.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(max) = self.max.as_mut() {
-            max.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(average) = self.average.as_mut() {
-            average.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(sum) = self.sum.as_mut() {
-            sum.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(cumulative) = self.cumulative.as_mut() {
-            cumulative.compute_if_necessary(max_from, len_source, exit)?;
-        }
-
-        Ok(())
-    }
-
-    pub fn starting_index(&self, max_from: I) -> I {
-        max_from.min(I::from(
-            self.vecs().into_iter().map(|v| v.len()).min().unwrap(),
-        ))
-    }
-
-    pub fn unwrap_first(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.first.as_ref().unwrap()
-    }
-    #[allow(unused)]
-    pub fn unwrap_average(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.average.as_ref().unwrap()
-    }
-    pub fn unwrap_sum(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.sum.as_ref().unwrap()
-    }
-    pub fn unwrap_max(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.max.as_ref().unwrap()
-    }
-    pub fn unwrap_min(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.min.as_ref().unwrap()
-    }
-    pub fn unwrap_last(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.last.as_ref().unwrap()
-    }
-    #[allow(unused)]
-    pub fn unwrap_cumulative(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.cumulative.as_ref().unwrap()
-    }
-
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        let mut v: Vec<&dyn AnyCollectableVec> = vec![];
-
-        if let Some(first) = self.first.as_ref() {
-            v.push(first.as_ref());
-        }
-        if let Some(last) = self.last.as_ref() {
-            v.push(last.as_ref());
-        }
-        if let Some(min) = self.min.as_ref() {
-            v.push(min.as_ref());
-        }
-        if let Some(max) = self.max.as_ref() {
-            v.push(max.as_ref());
-        }
-        if let Some(average) = self.average.as_ref() {
-            v.push(average.as_ref());
-        }
-        if let Some(sum) = self.sum.as_ref() {
-            v.push(sum.as_ref());
-        }
-        if let Some(cumulative) = self.cumulative.as_ref() {
-            v.push(cumulative.as_ref());
-        }
-
-        v
-    }
-}
-
-#[derive(Default, Clone, Copy)]
-pub struct ComputedVecBuilderOptions {
-    average: bool,
-    sum: bool,
-    max: bool,
-    min: bool,
-    first: bool,
-    last: bool,
-    cumulative: bool,
-}
-
-impl From<VecBuilderOptions> for ComputedVecBuilderOptions {
-    fn from(value: VecBuilderOptions) -> Self {
-        Self {
-            average: value.average(),
-            sum: value.sum(),
-            max: value.max(),
-            min: value.min(),
-            first: value.first(),
-            last: value.last(),
-            cumulative: value.cumulative(),
-        }
-    }
-}
-
-impl ComputedVecBuilderOptions {
-    pub fn add_first(mut self) -> Self {
-        self.first = true;
-        self
-    }
-
-    pub fn add_last(mut self) -> Self {
-        self.last = true;
-        self
-    }
-
-    pub fn add_min(mut self) -> Self {
-        self.min = true;
-        self
-    }
-
-    pub fn add_max(mut self) -> Self {
-        self.max = true;
-        self
-    }
-
-    pub fn add_average(mut self) -> Self {
-        self.average = true;
-        self
-    }
-
-    pub fn add_sum(mut self) -> Self {
-        self.sum = true;
-        self
-    }
-
-    pub fn add_cumulative(mut self) -> Self {
-        self.cumulative = true;
-        self
-    }
-
-    #[allow(unused)]
-    pub fn rm_min(mut self) -> Self {
-        self.min = false;
-        self
-    }
-
-    #[allow(unused)]
-    pub fn rm_max(mut self) -> Self {
-        self.max = false;
-        self
-    }
-
-    #[allow(unused)]
-    pub fn rm_average(mut self) -> Self {
-        self.average = false;
-        self
-    }
-
-    #[allow(unused)]
-    pub fn rm_sum(mut self) -> Self {
-        self.sum = false;
-        self
-    }
-
-    #[allow(unused)]
-    pub fn rm_cumulative(mut self) -> Self {
-        self.cumulative = false;
-        self
-    }
-
-    pub fn add_minmax(mut self) -> Self {
-        self.min = true;
-        self.max = true;
-        self
-    }
-
-    pub fn is_only_one_active(&self) -> bool {
-        [
-            self.average,
-            self.sum,
-            self.max,
-            self.min,
-            self.first,
-            self.last,
-            self.cumulative,
-        ]
-        .iter()
-        .filter(|b| **b)
-        .count()
-            == 1
-    }
-
-    pub fn copy_self_extra(&self) -> Self {
-        Self {
-            cumulative: self.cumulative,
-            ..Self::default()
-        }
-    }
-}

crates/brk_computer/src/grouped/builder_eager.rs

@@ -1,3 +1,4 @@
+use allocative::Allocative;
 use brk_error::{Error, Result};
 use brk_structs::{CheckedSub, StoredU64, Version};
 use vecdb::{
@@ -9,7 +10,7 @@ use crate::utils::get_percentile;
 
 use super::ComputedType;
 
-#[derive(Clone, Debug)]
+#[derive(Clone, Debug, Allocative)]
 pub struct EagerVecBuilder<I, T>
 where
     I: StoredIndex,
@@ -19,11 +20,11 @@ where
     pub average: Option<Box<EagerVec<I, T>>>,
     pub sum: Option<Box<EagerVec<I, T>>>,
     pub max: Option<Box<EagerVec<I, T>>>,
-    pub _90p: Option<Box<EagerVec<I, T>>>,
+    pub pct90: Option<Box<EagerVec<I, T>>>,
-    pub _75p: Option<Box<EagerVec<I, T>>>,
+    pub pct75: Option<Box<EagerVec<I, T>>>,
     pub median: Option<Box<EagerVec<I, T>>>,
-    pub _25p: Option<Box<EagerVec<I, T>>>,
+    pub pct25: Option<Box<EagerVec<I, T>>>,
-    pub _10p: Option<Box<EagerVec<I, T>>>,
+    pub pct10: Option<Box<EagerVec<I, T>>>,
     pub min: Option<Box<EagerVec<I, T>>>,
     pub last: Option<Box<EagerVec<I, T>>>,
     pub cumulative: Option<Box<EagerVec<I, T>>>,
@@ -118,7 +119,7 @@ where
                 Box::new(
                     EagerVec::forced_import(
                         db,
-                        &maybe_suffix("average"),
+                        &maybe_suffix("avg"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
@@ -151,44 +152,44 @@ where
                     .unwrap(),
                 )
             }),
-            _90p: options._90p.then(|| {
+            pct90: options.pct90.then(|| {
                 Box::new(
                     EagerVec::forced_import(
                         db,
-                        &maybe_suffix("90p"),
+                        &maybe_suffix("pct90"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _75p: options._75p.then(|| {
+            pct75: options.pct75.then(|| {
                 Box::new(
                     EagerVec::forced_import(
                         db,
-                        &maybe_suffix("75p"),
+                        &maybe_suffix("pct75"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _25p: options._25p.then(|| {
+            pct25: options.pct25.then(|| {
                 Box::new(
                     EagerVec::forced_import(
                         db,
-                        &maybe_suffix("25p"),
+                        &maybe_suffix("pct25"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _10p: options._10p.then(|| {
+            pct10: options.pct10.then(|| {
                 Box::new(
                     EagerVec::forced_import(
                         db,
-                        &maybe_suffix("10p"),
+                        &maybe_suffix("pct10"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
@@ -292,11 +293,11 @@ where
                 let needs_average_sum_or_cumulative =
                     needs_sum_or_cumulative || self.average.is_some();
                 let needs_sorted = self.max.is_some()
-                    || self._90p.is_some()
+                    || self.pct90.is_some()
-                    || self._75p.is_some()
+                    || self.pct75.is_some()
                     || self.median.is_some()
-                    || self._25p.is_some()
+                    || self.pct25.is_some()
-                    || self._10p.is_some()
+                    || self.pct10.is_some()
                     || self.min.is_some();
                 let needs_values = needs_sorted || needs_average_sum_or_cumulative;
 
@@ -333,24 +334,24 @@ where
                             )?;
                         }
 
-                        if let Some(_90p) = self._90p.as_mut() {
+                        if let Some(pct90) = self.pct90.as_mut() {
-                            _90p.forced_push_at(index, get_percentile(&values, 0.90), exit)?;
+                            pct90.forced_push_at(index, get_percentile(&values, 0.90), exit)?;
                         }
 
-                        if let Some(_75p) = self._75p.as_mut() {
+                        if let Some(pct75) = self.pct75.as_mut() {
-                            _75p.forced_push_at(index, get_percentile(&values, 0.75), exit)?;
+                            pct75.forced_push_at(index, get_percentile(&values, 0.75), exit)?;
                         }
 
                         if let Some(median) = self.median.as_mut() {
                             median.forced_push_at(index, get_percentile(&values, 0.50), exit)?;
                         }
 
-                        if let Some(_25p) = self._25p.as_mut() {
+                        if let Some(pct25) = self.pct25.as_mut() {
-                            _25p.forced_push_at(index, get_percentile(&values, 0.25), exit)?;
+                            pct25.forced_push_at(index, get_percentile(&values, 0.25), exit)?;
                         }
 
-                        if let Some(_10p) = self._10p.as_mut() {
+                        if let Some(pct10) = self.pct10.as_mut() {
-                            _10p.forced_push_at(index, get_percentile(&values, 0.10), exit)?;
+                            pct10.forced_push_at(index, get_percentile(&values, 0.10), exit)?;
                         }
 
                         if let Some(min) = self.min.as_mut() {
@@ -401,11 +402,11 @@ where
     where
         I2: StoredIndex + StoredRaw + CheckedSub<I2>,
     {
-        if self._90p.is_some()
+        if self.pct90.is_some()
-            || self._75p.is_some()
+            || self.pct75.is_some()
             || self.median.is_some()
-            || self._25p.is_some()
+            || self.pct25.is_some()
-            || self._10p.is_some()
+            || self.pct10.is_some()
         {
             panic!("unsupported");
         }
@@ -540,7 +541,7 @@ where
 
     pub fn starting_index(&self, max_from: I) -> I {
         max_from.min(I::from(
-            self.vecs().into_iter().map(|v| v.len()).min().unwrap(),
+            self.iter_any_collectable().map(|v| v.len()).min().unwrap(),
         ))
     }
 
@@ -558,24 +559,24 @@ where
         self.max.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_90p(&self) -> &EagerVec<I, T> {
+    pub fn unwrap_pct90(&self) -> &EagerVec<I, T> {
-        self._90p.as_ref().unwrap()
+        self.pct90.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_75p(&self) -> &EagerVec<I, T> {
+    pub fn unwrap_pct75(&self) -> &EagerVec<I, T> {
-        self._75p.as_ref().unwrap()
+        self.pct75.as_ref().unwrap()
     }
     #[allow(unused)]
     pub fn unwrap_median(&self) -> &EagerVec<I, T> {
         self.median.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_25p(&self) -> &EagerVec<I, T> {
+    pub fn unwrap_pct25(&self) -> &EagerVec<I, T> {
-        self._25p.as_ref().unwrap()
+        self.pct25.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_10p(&self) -> &EagerVec<I, T> {
+    pub fn unwrap_pct10(&self) -> &EagerVec<I, T> {
-        self._10p.as_ref().unwrap()
+        self.pct10.as_ref().unwrap()
     }
     pub fn unwrap_min(&self) -> &EagerVec<I, T> {
         self.min.as_ref().unwrap()
@@ -588,47 +589,96 @@ where
         self.cumulative.as_ref().unwrap()
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        let mut v: Vec<&dyn AnyCollectableVec> = vec![];
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
 
-        if let Some(first) = self.first.as_ref() {
+        iter = Box::new(
-            v.push(first.as_ref());
+            iter.chain(
-        }
+                self.first
-        if let Some(last) = self.last.as_ref() {
+                    .as_ref()
-            v.push(last.as_ref());
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
-        }
+            ),
-        if let Some(min) = self.min.as_ref() {
+        );
-            v.push(min.as_ref());
+        iter = Box::new(
-        }
+            iter.chain(
-        if let Some(max) = self.max.as_ref() {
+                self.last
-            v.push(max.as_ref());
+                    .as_ref()
-        }
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
-        if let Some(median) = self.median.as_ref() {
+            ),
-            v.push(median.as_ref());
+        );
-        }
+        iter = Box::new(
-        if let Some(average) = self.average.as_ref() {
+            iter.chain(
-            v.push(average.as_ref());
+                self.min
-        }
+                    .as_ref()
-        if let Some(sum) = self.sum.as_ref() {
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
-            v.push(sum.as_ref());
+            ),
-        }
+        );
-        if let Some(cumulative) = self.cumulative.as_ref() {
+        iter = Box::new(
-            v.push(cumulative.as_ref());
+            iter.chain(
-        }
+                self.max
-        if let Some(_90p) = self._90p.as_ref() {
+                    .as_ref()
-            v.push(_90p.as_ref());
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
-        }
+            ),
-        if let Some(_75p) = self._75p.as_ref() {
+        );
-            v.push(_75p.as_ref());
+        iter = Box::new(
-        }
+            iter.chain(
-        if let Some(_25p) = self._25p.as_ref() {
+                self.median
-            v.push(_25p.as_ref());
+                    .as_ref()
-        }
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
-        if let Some(_10p) = self._10p.as_ref() {
+            ),
-            v.push(_10p.as_ref());
+        );
-        }
+        iter = Box::new(
+            iter.chain(
+                self.average
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.sum
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.cumulative
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct90
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct75
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct25
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct10
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
 
-        v
+        iter
     }
 
     pub fn safe_flush(&mut self, exit: &Exit) -> Result<()> {
@@ -656,17 +706,17 @@ where
         if let Some(cumulative) = self.cumulative.as_mut() {
             cumulative.safe_flush(exit)?;
         }
-        if let Some(_90p) = self._90p.as_mut() {
+        if let Some(pct90) = self.pct90.as_mut() {
-            _90p.safe_flush(exit)?;
+            pct90.safe_flush(exit)?;
         }
-        if let Some(_75p) = self._75p.as_mut() {
+        if let Some(pct75) = self.pct75.as_mut() {
-            _75p.safe_flush(exit)?;
+            pct75.safe_flush(exit)?;
         }
-        if let Some(_25p) = self._25p.as_mut() {
+        if let Some(pct25) = self.pct25.as_mut() {
-            _25p.safe_flush(exit)?;
+            pct25.safe_flush(exit)?;
         }
-        if let Some(_10p) = self._10p.as_mut() {
+        if let Some(pct10) = self.pct10.as_mut() {
-            _10p.safe_flush(exit)?;
+            pct10.safe_flush(exit)?;
         }
 
         Ok(())
@@ -697,17 +747,17 @@ where
         if let Some(cumulative) = self.cumulative.as_mut() {
             cumulative.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_90p) = self._90p.as_mut() {
+        if let Some(pct90) = self.pct90.as_mut() {
-            _90p.validate_computed_version_or_reset(Version::ZERO + version)?;
+            pct90.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_75p) = self._75p.as_mut() {
+        if let Some(pct75) = self.pct75.as_mut() {
-            _75p.validate_computed_version_or_reset(Version::ZERO + version)?;
+            pct75.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_25p) = self._25p.as_mut() {
+        if let Some(pct25) = self.pct25.as_mut() {
-            _25p.validate_computed_version_or_reset(Version::ZERO + version)?;
+            pct25.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_10p) = self._10p.as_mut() {
+        if let Some(pct10) = self.pct10.as_mut() {
-            _10p.validate_computed_version_or_reset(Version::ZERO + version)?;
+            pct10.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
 
         Ok(())
@@ -719,11 +769,11 @@ pub struct VecBuilderOptions {
     average: bool,
     sum: bool,
     max: bool,
-    _90p: bool,
+    pct90: bool,
-    _75p: bool,
+    pct75: bool,
     median: bool,
-    _25p: bool,
+    pct25: bool,
-    _10p: bool,
+    pct10: bool,
     min: bool,
     first: bool,
     last: bool,
@@ -743,24 +793,24 @@ impl VecBuilderOptions {
         self.max
     }
 
-    pub fn _90p(&self) -> bool {
+    pub fn pct90(&self) -> bool {
-        self._90p
+        self.pct90
     }
 
-    pub fn _75p(&self) -> bool {
+    pub fn pct75(&self) -> bool {
-        self._75p
+        self.pct75
     }
 
     pub fn median(&self) -> bool {
         self.median
     }
 
-    pub fn _25p(&self) -> bool {
+    pub fn pct25(&self) -> bool {
-        self._25p
+        self.pct25
     }
 
-    pub fn _10p(&self) -> bool {
+    pub fn pct10(&self) -> bool {
-        self._10p
+        self.pct10
     }
 
     pub fn min(&self) -> bool {
@@ -816,26 +866,26 @@ impl VecBuilderOptions {
     }
 
     #[allow(unused)]
-    pub fn add_90p(mut self) -> Self {
+    pub fn add_pct90(mut self) -> Self {
-        self._90p = true;
+        self.pct90 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_75p(mut self) -> Self {
+    pub fn add_pct75(mut self) -> Self {
-        self._75p = true;
+        self.pct75 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_25p(mut self) -> Self {
+    pub fn add_pct25(mut self) -> Self {
-        self._25p = true;
+        self.pct25 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_10p(mut self) -> Self {
+    pub fn add_pct10(mut self) -> Self {
-        self._10p = true;
+        self.pct10 = true;
         self
     }
 
@@ -875,26 +925,26 @@ impl VecBuilderOptions {
     }
 
     #[allow(unused)]
-    pub fn rm_90p(mut self) -> Self {
+    pub fn rm_pct90(mut self) -> Self {
-        self._90p = false;
+        self.pct90 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_75p(mut self) -> Self {
+    pub fn rm_pct75(mut self) -> Self {
-        self._75p = false;
+        self.pct75 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_25p(mut self) -> Self {
+    pub fn rm_pct25(mut self) -> Self {
-        self._25p = false;
+        self.pct25 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_10p(mut self) -> Self {
+    pub fn rm_pct10(mut self) -> Self {
-        self._10p = false;
+        self.pct10 = false;
         self
     }
 
@@ -911,20 +961,20 @@ impl VecBuilderOptions {
     }
 
     pub fn add_percentiles(mut self) -> Self {
-        self._90p = true;
+        self.pct90 = true;
-        self._75p = true;
+        self.pct75 = true;
         self.median = true;
-        self._25p = true;
+        self.pct25 = true;
-        self._10p = true;
+        self.pct10 = true;
         self
     }
 
     pub fn remove_percentiles(mut self) -> Self {
-        self._90p = false;
+        self.pct90 = false;
-        self._75p = false;
+        self.pct75 = false;
         self.median = false;
-        self._25p = false;
+        self.pct25 = false;
-        self._10p = false;
+        self.pct10 = false;
         self
     }
 
@@ -933,11 +983,11 @@ impl VecBuilderOptions {
             self.average,
             self.sum,
             self.max,
-            self._90p,
+            self.pct90,
-            self._75p,
+            self.pct75,
             self.median,
-            self._25p,
+            self.pct25,
-            self._10p,
+            self.pct10,
             self.min,
             self.first,
             self.last,

crates/brk_computer/src/grouped/builder_lazy.rs

@@ -1,3 +1,4 @@
+use allocative::Allocative;
 use brk_structs::Version;
 use vecdb::{
     AnyBoxedIterableVec, AnyCloneableIterableVec, AnyCollectableVec, FromCoarserIndex,
@@ -9,7 +10,7 @@ use crate::grouped::{EagerVecBuilder, VecBuilderOptions};
 use super::ComputedType;
 
 #[allow(clippy::type_complexity)]
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct LazyVecBuilder<I, T, S1I, S2T>
 where
     I: StoredIndex,
@@ -142,7 +143,7 @@ where
             }),
             average: options.average.then(|| {
                 Box::new(LazyVecFrom2::init(
-                    &maybe_suffix("average"),
+                    &maybe_suffix("avg"),
                     version + VERSION + Version::ZERO,
                     source_extra
                         .average
@@ -216,7 +217,7 @@ where
 
     pub fn starting_index(&self, max_from: I) -> I {
         max_from.min(I::from(
-            self.vecs().into_iter().map(|v| v.len()).min().unwrap(),
+            self.iter_any_collectable().map(|v| v.len()).min().unwrap(),
         ))
     }
 
@@ -244,32 +245,61 @@ where
         self.cumulative.as_ref().unwrap()
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        let mut v: Vec<&dyn AnyCollectableVec> = vec![];
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
 
-        if let Some(first) = self.first.as_ref() {
+        iter = Box::new(
-            v.push(first.as_ref());
+            iter.chain(
-        }
+                self.first
-        if let Some(last) = self.last.as_ref() {
+                    .as_ref()
-            v.push(last.as_ref());
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
-        }
+            ),
-        if let Some(min) = self.min.as_ref() {
+        );
-            v.push(min.as_ref());
+        iter = Box::new(
-        }
+            iter.chain(
-        if let Some(max) = self.max.as_ref() {
+                self.last
-            v.push(max.as_ref());
+                    .as_ref()
-        }
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
-        if let Some(average) = self.average.as_ref() {
+            ),
-            v.push(average.as_ref());
+        );
-        }
+        iter = Box::new(
-        if let Some(sum) = self.sum.as_ref() {
+            iter.chain(
-            v.push(sum.as_ref());
+                self.min
-        }
+                    .as_ref()
-        if let Some(cumulative) = self.cumulative.as_ref() {
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
-            v.push(cumulative.as_ref());
+            ),
-        }
+        );
+        iter = Box::new(
+            iter.chain(
+                self.max
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.average
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.sum
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.cumulative
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
 
-        v
+        iter
     }
 }
 

crates/brk_computer/src/grouped/from_dateindex.rs

@@ -1,6 +1,6 @@
+use allocative::Allocative;
 use brk_error::Result;
 
-use brk_indexer::Indexer;
 use brk_structs::{
     DateIndex, DecadeIndex, MonthIndex, QuarterIndex, SemesterIndex, Version, WeekIndex, YearIndex,
 };
@@ -10,7 +10,7 @@ use crate::{Indexes, grouped::LazyVecBuilder, indexes};
 
 use super::{ComputedType, EagerVecBuilder, Source, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedVecsFromDateIndex<T>
 where
     T: ComputedType + PartialOrd,
@@ -111,28 +111,14 @@ where
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(
+        F: FnMut(&mut EagerVec<DateIndex, T>) -> Result<()>,
-            &mut EagerVec<DateIndex, T>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
     {
-        compute(
+        compute(self.dateindex.as_mut().unwrap())?;
-            self.dateindex.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
 
         let dateindex: Option<&EagerVec<DateIndex, T>> = None;
         self.compute_rest(starting_indexes, exit, dateindex)
@@ -157,21 +143,22 @@ where
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
             self.dateindex
                 .as_ref()
-                .map_or(vec![], |v| vec![v as &dyn AnyCollectableVec]),
+                .map(|x| x as &dyn AnyCollectableVec)
-            self.dateindex_extra.vecs(),
+                .into_iter(),
-            self.weekindex.vecs(),
+        );
-            self.monthindex.vecs(),
+
-            self.quarterindex.vecs(),
+        iter = Box::new(iter.chain(self.dateindex_extra.iter_any_collectable()));
-            self.semesterindex.vecs(),
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
-            self.yearindex.vecs(),
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
-            self.decadeindex.vecs(),
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
-        ]
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
-        .into_iter()
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
-        .flatten()
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
-        .collect::<Vec<_>>()
+
+        iter
     }
 }

crates/brk_computer/src/grouped/from_height.rs

@@ -1,6 +1,6 @@
+use allocative::Allocative;
 use brk_error::Result;
 
-use brk_indexer::Indexer;
 use brk_structs::{
     DateIndex, DecadeIndex, DifficultyEpoch, Height, MonthIndex, QuarterIndex, SemesterIndex,
     Version, WeekIndex, YearIndex,
@@ -15,7 +15,7 @@ use crate::{
 
 use super::{ComputedType, EagerVecBuilder, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedVecsFromHeight<T>
 where
     T: ComputedType + PartialOrd,
@@ -133,22 +133,15 @@ where
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(&mut EagerVec<Height, T>, &Indexer, &indexes::Vecs, &Indexes, &Exit) -> Result<()>,
+        F: FnMut(&mut EagerVec<Height, T>) -> Result<()>,
     {
-        compute(
+        compute(self.height.as_mut().unwrap())?;
-            self.height.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
 
         let height: Option<&EagerVec<Height, T>> = None;
         self.compute_rest(indexes, starting_indexes, exit, height)
@@ -206,24 +199,24 @@ where
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
             self.height
                 .as_ref()
-                .map_or(vec![], |v| vec![v as &dyn AnyCollectableVec]),
+                .map(|x| x as &dyn AnyCollectableVec)
-            self.height_extra.vecs(),
+                .into_iter(),
-            self.dateindex.vecs(),
+        );
-            self.weekindex.vecs(),
+
-            self.difficultyepoch.vecs(),
+        iter = Box::new(iter.chain(self.height_extra.iter_any_collectable()));
-            self.monthindex.vecs(),
+        iter = Box::new(iter.chain(self.dateindex.iter_any_collectable()));
-            self.quarterindex.vecs(),
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
-            self.semesterindex.vecs(),
+        iter = Box::new(iter.chain(self.difficultyepoch.iter_any_collectable()));
-            self.yearindex.vecs(),
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
-            // self.halvingepoch.vecs(),
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
-            self.decadeindex.vecs(),
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
-        ]
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
-        .into_iter()
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
-        .flatten()
+
-        .collect::<Vec<_>>()
+        iter
     }
 }

crates/brk_computer/src/grouped/from_height_strict.rs

@@ -1,6 +1,5 @@
 use brk_error::Result;
 
-use brk_indexer::Indexer;
 use brk_structs::{DifficultyEpoch, Height, Version};
 use vecdb::{AnyCollectableVec, Database, EagerVec, Exit};
 
@@ -59,16 +58,15 @@ where
 
     pub fn compute<F>(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(&mut EagerVec<Height, T>, &Indexer, &indexes::Vecs, &Indexes, &Exit) -> Result<()>,
+        F: FnMut(&mut EagerVec<Height, T>) -> Result<()>,
     {
-        compute(&mut self.height, indexer, indexes, starting_indexes, exit)?;
+        compute(&mut self.height)?;
 
         self.height_extra
             .extend(starting_indexes.height, &self.height, exit)?;
@@ -84,15 +82,10 @@ where
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        [&self.height as &dyn AnyCollectableVec]
-            vec![&self.height as &dyn AnyCollectableVec],
+            .into_iter()
-            self.height_extra.vecs(),
+            .chain(self.height_extra.iter_any_collectable())
-            self.difficultyepoch.vecs(),
+            .chain(self.difficultyepoch.iter_any_collectable())
-            // self.halvingepoch.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
     }
 }

crates/brk_computer/src/grouped/from_txindex.rs

@@ -1,3 +1,4 @@
+use allocative::Allocative;
 use brk_error::Result;
 use brk_indexer::Indexer;
 use brk_structs::{
@@ -17,7 +18,7 @@ use crate::{
 
 use super::{ComputedType, EagerVecBuilder, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedVecsFromTxindex<T>
 where
     T: ComputedType + PartialOrd,
@@ -225,25 +226,25 @@ where
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
             self.txindex
                 .as_ref()
-                .map_or(vec![], |v| vec![v.as_ref() as &dyn AnyCollectableVec]),
+                .map(|x| x.as_ref() as &dyn AnyCollectableVec)
-            self.height.vecs(),
+                .into_iter(),
-            self.dateindex.vecs(),
+        );
-            self.weekindex.vecs(),
+
-            self.difficultyepoch.vecs(),
+        iter = Box::new(iter.chain(self.height.iter_any_collectable()));
-            self.monthindex.vecs(),
+        iter = Box::new(iter.chain(self.dateindex.iter_any_collectable()));
-            self.quarterindex.vecs(),
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
-            self.semesterindex.vecs(),
+        iter = Box::new(iter.chain(self.difficultyepoch.iter_any_collectable()));
-            self.yearindex.vecs(),
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
-            // self.halvingepoch.vecs(),
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
-            self.decadeindex.vecs(),
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
-        ]
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
-        .into_iter()
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
-        .flatten()
+
-        .collect::<Vec<_>>()
+        iter
     }
 }
 
@@ -319,24 +320,24 @@ impl ComputedVecsFromTxindex<Bitcoin> {
                         exit,
                     )?;
                 }
-                if let Some(_90p) = self.height._90p.as_mut() {
+                if let Some(pct90) = self.height.pct90.as_mut() {
-                    _90p.forced_push_at(
+                    pct90.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_90p()
+                                .unwrap_pct90()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
                         exit,
                     )?;
                 }
-                if let Some(_75p) = self.height._75p.as_mut() {
+                if let Some(pct75) = self.height.pct75.as_mut() {
-                    _75p.forced_push_at(
+                    pct75.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_75p()
+                                .unwrap_pct75()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
@@ -355,24 +356,24 @@ impl ComputedVecsFromTxindex<Bitcoin> {
                         exit,
                     )?;
                 }
-                if let Some(_25p) = self.height._25p.as_mut() {
+                if let Some(pct25) = self.height.pct25.as_mut() {
-                    _25p.forced_push_at(
+                    pct25.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_25p()
+                                .unwrap_pct25()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
                         exit,
                     )?;
                 }
-                if let Some(_10p) = self.height._10p.as_mut() {
+                if let Some(pct10) = self.height.pct10.as_mut() {
-                    _10p.forced_push_at(
+                    pct10.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_10p()
+                                .unwrap_pct10()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
@@ -447,7 +448,7 @@ impl ComputedVecsFromTxindex<Dollars> {
 
         let starting_index = self.height.starting_index(starting_indexes.height);
 
-        let mut close_iter = price.chainindexes_to_close.height.into_iter();
+        let mut close_iter = price.chainindexes_to_price_close.height.into_iter();
 
         (starting_index.unwrap_to_usize()..indexer.vecs.height_to_weight.len())
             .map(Height::from)
@@ -502,25 +503,25 @@ impl ComputedVecsFromTxindex<Dollars> {
                         exit,
                     )?;
                 }
-                if let Some(_90p) = self.height._90p.as_mut() {
+                if let Some(pct90) = self.height.pct90.as_mut() {
-                    _90p.forced_push_at(
+                    pct90.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_90p()
+                                .unwrap_pct90()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
                     )?;
                 }
-                if let Some(_75p) = self.height._75p.as_mut() {
+                if let Some(pct75) = self.height.pct75.as_mut() {
-                    _75p.forced_push_at(
+                    pct75.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_75p()
+                                .unwrap_pct75()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
@@ -538,25 +539,25 @@ impl ComputedVecsFromTxindex<Dollars> {
                         exit,
                     )?;
                 }
-                if let Some(_25p) = self.height._25p.as_mut() {
+                if let Some(pct25) = self.height.pct25.as_mut() {
-                    _25p.forced_push_at(
+                    pct25.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_25p()
+                                .unwrap_pct25()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
                     )?;
                 }
-                if let Some(_10p) = self.height._10p.as_mut() {
+                if let Some(pct10) = self.height.pct10.as_mut() {
-                    _10p.forced_push_at(
+                    pct10.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_10p()
+                                .unwrap_pct10()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,

crates/brk_computer/src/grouped/ratio_from_dateindex.rs

@@ -1,5 +1,4 @@
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Date, DateIndex, Dollars, StoredF32, Version};
 use vecdb::{
     AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, CollectableVec, Database, EagerVec,
@@ -8,7 +7,9 @@ use vecdb::{
 
 use crate::{
     Indexes,
-    grouped::{ComputedStandardDeviationVecsFromDateIndex, source::Source},
+    grouped::{
+        ComputedStandardDeviationVecsFromDateIndex, StandardDeviationVecsOptions, source::Source,
+    },
     indexes, price,
     utils::get_percentile,
 };
@@ -22,18 +23,18 @@ pub struct ComputedRatioVecsFromDateIndex {
     pub ratio: ComputedVecsFromDateIndex<StoredF32>,
     pub ratio_1w_sma: Option<ComputedVecsFromDateIndex<StoredF32>>,
     pub ratio_1m_sma: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p99: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct99: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p98: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct98: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p95: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct95: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p5: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct5: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p2: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct2: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p1: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct1: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p99_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct99_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
-    pub ratio_p98_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct98_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
-    pub ratio_p95_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct95_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
-    pub ratio_p5_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct5_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
-    pub ratio_p2_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct2_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
-    pub ratio_p1_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct1_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
 
     pub ratio_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
     pub ratio_4y_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
@@ -105,6 +106,7 @@ impl ComputedRatioVecsFromDateIndex {
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
                 )
                 .unwrap()
             }),
@@ -116,6 +118,7 @@ impl ComputedRatioVecsFromDateIndex {
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
                 )
                 .unwrap()
             }),
@@ -127,6 +130,7 @@ impl ComputedRatioVecsFromDateIndex {
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
                 )
                 .unwrap()
             }),
@@ -138,13 +142,14 @@ impl ComputedRatioVecsFromDateIndex {
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
                 )
                 .unwrap()
             }),
-            ratio_p99: extended.then(|| {
+            ratio_pct99: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p99"),
+                    &format!("{name}_ratio_pct99"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -152,10 +157,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p98: extended.then(|| {
+            ratio_pct98: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p98"),
+                    &format!("{name}_ratio_pct98"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -163,10 +168,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p95: extended.then(|| {
+            ratio_pct95: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p95"),
+                    &format!("{name}_ratio_pct95"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -174,10 +179,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p5: extended.then(|| {
+            ratio_pct5: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p5"),
+                    &format!("{name}_ratio_pct5"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -185,10 +190,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p2: extended.then(|| {
+            ratio_pct2: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p2"),
+                    &format!("{name}_ratio_pct2"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -196,10 +201,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p1: extended.then(|| {
+            ratio_pct1: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p1"),
+                    &format!("{name}_ratio_pct1"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -207,10 +212,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p99_as_price: extended.then(|| {
+            ratio_pct99_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p99_as_price"),
+                    &format!("{name}_ratio_pct99_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -218,10 +223,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p98_as_price: extended.then(|| {
+            ratio_pct98_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p98_as_price"),
+                    &format!("{name}_ratio_pct98_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -229,10 +234,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p95_as_price: extended.then(|| {
+            ratio_pct95_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p95_as_price"),
+                    &format!("{name}_ratio_pct95_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -240,10 +245,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p5_as_price: extended.then(|| {
+            ratio_pct5_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p5_as_price"),
+                    &format!("{name}_ratio_pct5_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -251,10 +256,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p2_as_price: extended.then(|| {
+            ratio_pct2_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p2_as_price"),
+                    &format!("{name}_ratio_pct2_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -262,10 +267,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p1_as_price: extended.then(|| {
+            ratio_pct1_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p1_as_price"),
+                    &format!("{name}_ratio_pct1_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -278,79 +283,52 @@ impl ComputedRatioVecsFromDateIndex {
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         price: &price::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         compute: F,
     ) -> Result<()>
     where
-        F: FnMut(
+        F: FnMut(&mut EagerVec<DateIndex, Dollars>) -> Result<()>,
-            &mut EagerVec<DateIndex, Dollars>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
     {
-        self.price.as_mut().unwrap().compute_all(
+        self.price
-            indexer,
+            .as_mut()
-            indexes,
+            .unwrap()
-            starting_indexes,
+            .compute_all(starting_indexes, exit, compute)?;
-            exit,
-            compute,
-        )?;
 
         let date_to_price_opt: Option<&EagerVec<DateIndex, Dollars>> = None;
-        self.compute_rest(
+        self.compute_rest(price, starting_indexes, exit, date_to_price_opt)
-            indexer,
-            indexes,
-            price,
-            starting_indexes,
-            exit,
-            date_to_price_opt,
-        )
     }
 
     pub fn compute_rest(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         price: &price::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
-        date_to_price_opt: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+        price_opt: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
     ) -> Result<()> {
-        let date_to_price = date_to_price_opt.unwrap_or_else(|| unsafe {
+        let closes = price.timeindexes_to_price_close.dateindex.as_ref().unwrap();
+
+        let price = price_opt.unwrap_or_else(|| unsafe {
             std::mem::transmute(&self.price.as_ref().unwrap().dateindex)
         });
 
-        let closes = price.timeindexes_to_close.dateindex.as_ref().unwrap();
+        self.ratio.compute_all(starting_indexes, exit, |v| {
-
+            v.compute_transform2(
-        self.ratio.compute_all(
+                starting_indexes.dateindex,
-            indexer,
+                closes,
-            indexes,
+                price,
-            starting_indexes,
+                |(i, close, price, ..)| {
-            exit,
+                    if price == Dollars::ZERO {
-            |v, _, _, starting_indexes, exit| {
+                        (i, StoredF32::from(1.0))
-                let mut price = date_to_price.iter();
+                    } else {
-                v.compute_transform(
+                        (i, StoredF32::from(*close / price))
-                    starting_indexes.dateindex,
+                    }
-                    closes,
+                },
-                    |(i, close, ..)| {
+                exit,
-                        let price = price.unwrap_get_inner(i);
+            )?;
-                        if price == Dollars::ZERO {
+            Ok(())
-                            (i, StoredF32::from(1.0))
+        })?;
-                        } else {
-                            (i, StoredF32::from(*close / price))
-                        }
-                    },
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
 
         if self.ratio_1w_sma.is_none() {
             return Ok(());
@@ -358,12 +336,10 @@ impl ComputedRatioVecsFromDateIndex {
 
         let min_ratio_date = DateIndex::try_from(Date::MIN_RATIO).unwrap();
 
-        self.ratio_1w_sma.as_mut().unwrap().compute_all(
+        self.ratio_1w_sma
-            indexer,
+            .as_mut()
-            indexes,
+            .unwrap()
-            starting_indexes,
+            .compute_all(starting_indexes, exit, |v| {
-            exit,
-            |v, _, _, starting_indexes, exit| {
                 v.compute_sma_(
                     starting_indexes.dateindex,
                     self.ratio.dateindex.as_ref().unwrap(),
@@ -372,15 +348,12 @@ impl ComputedRatioVecsFromDateIndex {
                     Some(min_ratio_date),
                 )?;
                 Ok(())
-            },
+            })?;
-        )?;
 
-        self.ratio_1m_sma.as_mut().unwrap().compute_all(
+        self.ratio_1m_sma
-            indexer,
+            .as_mut()
-            indexes,
+            .unwrap()
-            starting_indexes,
+            .compute_all(starting_indexes, exit, |v| {
-            exit,
-            |v, _, _, starting_indexes, exit| {
                 v.compute_sma_(
                     starting_indexes.dateindex,
                     self.ratio.dateindex.as_ref().unwrap(),
@@ -389,8 +362,7 @@ impl ComputedRatioVecsFromDateIndex {
                     Some(min_ratio_date),
                 )?;
                 Ok(())
-            },
+            })?;
-        )?;
 
         let ratio_version = self.ratio.dateindex.as_ref().unwrap().version();
         self.mut_ratio_vecs()
@@ -424,42 +396,42 @@ impl ComputedRatioVecsFromDateIndex {
             .iter_at(starting_dateindex)
             .try_for_each(|(index, ratio)| -> Result<()> {
                 if index < min_ratio_date {
-                    self.ratio_p5
+                    self.ratio_pct5
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, StoredF32::NAN, exit)?;
-                    self.ratio_p2
+                    self.ratio_pct2
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, StoredF32::NAN, exit)?;
-                    self.ratio_p1
+                    self.ratio_pct1
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, StoredF32::NAN, exit)?;
-                    self.ratio_p95
+                    self.ratio_pct95
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, StoredF32::NAN, exit)?;
-                    self.ratio_p98
+                    self.ratio_pct98
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, StoredF32::NAN, exit)?;
-                    self.ratio_p99
+                    self.ratio_pct99
                         .as_mut()
                         .unwrap()
                         .dateindex
@@ -471,42 +443,42 @@ impl ComputedRatioVecsFromDateIndex {
                     let pos = sorted.binary_search(&ratio).unwrap_or_else(|pos| pos);
                     sorted.insert(pos, ratio);
 
-                    self.ratio_p1
+                    self.ratio_pct1
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, get_percentile(&sorted, 0.01), exit)?;
-                    self.ratio_p2
+                    self.ratio_pct2
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, get_percentile(&sorted, 0.02), exit)?;
-                    self.ratio_p5
+                    self.ratio_pct5
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, get_percentile(&sorted, 0.05), exit)?;
-                    self.ratio_p95
+                    self.ratio_pct95
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, get_percentile(&sorted, 0.95), exit)?;
-                    self.ratio_p98
+                    self.ratio_pct98
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, get_percentile(&sorted, 0.98), exit)?;
-                    self.ratio_p99
+                    self.ratio_pct99
                         .as_mut()
                         .unwrap()
                         .dateindex
@@ -522,49 +494,47 @@ impl ComputedRatioVecsFromDateIndex {
             .into_iter()
             .try_for_each(|v| v.safe_flush(exit))?;
 
-        self.ratio_p1.as_mut().unwrap().compute_rest(
+        self.ratio_pct1.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
         )?;
-        self.ratio_p2.as_mut().unwrap().compute_rest(
+        self.ratio_pct2.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
         )?;
-        self.ratio_p5.as_mut().unwrap().compute_rest(
+        self.ratio_pct5.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
         )?;
-        self.ratio_p95.as_mut().unwrap().compute_rest(
+        self.ratio_pct95.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
         )?;
-        self.ratio_p98.as_mut().unwrap().compute_rest(
+        self.ratio_pct98.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
         )?;
-        self.ratio_p99.as_mut().unwrap().compute_rest(
+        self.ratio_pct99.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
         )?;
 
-        let date_to_price = date_to_price_opt.unwrap_or_else(|| unsafe {
+        let date_to_price = price_opt.unwrap_or_else(|| unsafe {
             std::mem::transmute(&self.price.as_ref().unwrap().dateindex)
         });
 
-        self.ratio_p99_as_price.as_mut().unwrap().compute_all(
+        self.ratio_pct99_usd
-            indexer,
+            .as_mut()
-            indexes,
+            .unwrap()
-            starting_indexes,
+            .compute_all(starting_indexes, exit, |vec| {
-            exit,
-            |vec, _, _, starting_indexes, exit| {
                 let mut iter = self
-                    .ratio_p99
+                    .ratio_pct99
                     .as_ref()
                     .unwrap()
                     .dateindex
@@ -581,67 +551,52 @@ impl ComputedRatioVecsFromDateIndex {
                     exit,
                 )?;
                 Ok(())
-            },
+            })?;
-        )?;
 
-        let compute_as_price =
+        let compute_usd =
-            |as_price: Option<&mut ComputedVecsFromDateIndex<Dollars>>,
+            |usd: Option<&mut ComputedVecsFromDateIndex<Dollars>>,
              source: Option<&ComputedVecsFromDateIndex<StoredF32>>| {
-                as_price.unwrap().compute_all(
+                usd.unwrap().compute_all(starting_indexes, exit, |vec| {
-                    indexer,
+                    let mut iter = source.unwrap().dateindex.as_ref().unwrap().into_iter();
-                    indexes,
+                    vec.compute_transform(
-                    starting_indexes,
+                        starting_indexes.dateindex,
-                    exit,
+                        date_to_price,
-                    |vec, _, _, starting_indexes, exit| {
+                        |(i, price, ..)| {
-                        let mut iter = source.unwrap().dateindex.as_ref().unwrap().into_iter();
+                            let multiplier = iter.unwrap_get_inner(i);
-                        vec.compute_transform(
+                            (i, price * multiplier)
-                            starting_indexes.dateindex,
+                        },
-                            date_to_price,
+                        exit,
-                            |(i, price, ..)| {
+                    )?;
-                                let multiplier = iter.unwrap_get_inner(i);
+                    Ok(())
-                                (i, price * multiplier)
+                })
-                            },
-                            exit,
-                        )?;
-                        Ok(())
-                    },
-                )
             };
 
-        compute_as_price(self.ratio_p1_as_price.as_mut(), self.ratio_p1.as_ref())?;
+        compute_usd(self.ratio_pct1_usd.as_mut(), self.ratio_pct1.as_ref())?;
-        compute_as_price(self.ratio_p2_as_price.as_mut(), self.ratio_p2.as_ref())?;
+        compute_usd(self.ratio_pct2_usd.as_mut(), self.ratio_pct2.as_ref())?;
-        compute_as_price(self.ratio_p5_as_price.as_mut(), self.ratio_p5.as_ref())?;
+        compute_usd(self.ratio_pct5_usd.as_mut(), self.ratio_pct5.as_ref())?;
-        compute_as_price(self.ratio_p95_as_price.as_mut(), self.ratio_p95.as_ref())?;
+        compute_usd(self.ratio_pct95_usd.as_mut(), self.ratio_pct95.as_ref())?;
-        compute_as_price(self.ratio_p98_as_price.as_mut(), self.ratio_p98.as_ref())?;
+        compute_usd(self.ratio_pct98_usd.as_mut(), self.ratio_pct98.as_ref())?;
-        compute_as_price(self.ratio_p99_as_price.as_mut(), self.ratio_p99.as_ref())?;
+        compute_usd(self.ratio_pct99_usd.as_mut(), self.ratio_pct99.as_ref())?;
 
         self.ratio_sd.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
             starting_indexes,
             exit,
             self.ratio.dateindex.as_ref().unwrap(),
             Some(date_to_price),
         )?;
         self.ratio_4y_sd.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
             starting_indexes,
             exit,
             self.ratio.dateindex.as_ref().unwrap(),
             Some(date_to_price),
         )?;
         self.ratio_2y_sd.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
             starting_indexes,
             exit,
             self.ratio.dateindex.as_ref().unwrap(),
             Some(date_to_price),
         )?;
         self.ratio_1y_sd.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
             starting_indexes,
             exit,
             self.ratio.dateindex.as_ref().unwrap(),
@@ -653,22 +608,22 @@ impl ComputedRatioVecsFromDateIndex {
 
     fn mut_ratio_vecs(&mut self) -> Vec<&mut EagerVec<DateIndex, StoredF32>> {
         [
-            self.ratio_p1
+            self.ratio_pct1
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
-            self.ratio_p2
+            self.ratio_pct2
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
-            self.ratio_p5
+            self.ratio_pct5
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
-            self.ratio_p95
+            self.ratio_pct95
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
-            self.ratio_p98
+            self.ratio_pct98
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
-            self.ratio_p99
+            self.ratio_pct99
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
         ]
@@ -677,37 +632,132 @@ impl ComputedRatioVecsFromDateIndex {
         .collect::<Vec<_>>()
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
-            self.price.as_ref().map_or(vec![], |v| v.vecs()),
+            Box::new(self.price.iter().flat_map(|v| v.iter_any_collectable()));
-            self.ratio.vecs(),
+
-            self.ratio_1w_sma.as_ref().map_or(vec![], |v| v.vecs()),
+        iter = Box::new(iter.chain(self.ratio.iter_any_collectable()));
-            self.ratio_1m_sma.as_ref().map_or(vec![], |v| v.vecs()),
+        iter = Box::new(
-            self.ratio_sd.as_ref().map_or(vec![], |v| v.vecs()),
+            iter.chain(
-            self.ratio_1y_sd.as_ref().map_or(vec![], |v| v.vecs()),
+                self.ratio_1w_sma
-            self.ratio_2y_sd.as_ref().map_or(vec![], |v| v.vecs()),
+                    .iter()
-            self.ratio_4y_sd.as_ref().map_or(vec![], |v| v.vecs()),
+                    .flat_map(|v| v.iter_any_collectable()),
-            self.ratio_p1.as_ref().map_or(vec![], |v| v.vecs()),
+            ),
-            self.ratio_p2.as_ref().map_or(vec![], |v| v.vecs()),
+        );
-            self.ratio_p5.as_ref().map_or(vec![], |v| v.vecs()),
+        iter = Box::new(
-            self.ratio_p95.as_ref().map_or(vec![], |v| v.vecs()),
+            iter.chain(
-            self.ratio_p98.as_ref().map_or(vec![], |v| v.vecs()),
+                self.ratio_1m_sma
-            self.ratio_p99.as_ref().map_or(vec![], |v| v.vecs()),
+                    .iter()
-            self.ratio_p1_as_price.as_ref().map_or(vec![], |v| v.vecs()),
+                    .flat_map(|v| v.iter_any_collectable()),
-            self.ratio_p2_as_price.as_ref().map_or(vec![], |v| v.vecs()),
+            ),
-            self.ratio_p5_as_price.as_ref().map_or(vec![], |v| v.vecs()),
+        );
-            self.ratio_p95_as_price
+        iter = Box::new(iter.chain(self.ratio_sd.iter().flat_map(|v| v.iter_any_collectable())));
-                .as_ref()
+        iter = Box::new(
-                .map_or(vec![], |v| v.vecs()),
+            iter.chain(
-            self.ratio_p98_as_price
+                self.ratio_1y_sd
-                .as_ref()
+                    .iter()
-                .map_or(vec![], |v| v.vecs()),
+                    .flat_map(|v| v.iter_any_collectable()),
-            self.ratio_p99_as_price
+            ),
-                .as_ref()
+        );
-                .map_or(vec![], |v| v.vecs()),
+        iter = Box::new(
-        ]
+            iter.chain(
-        .into_iter()
+                self.ratio_2y_sd
-        .flatten()
+                    .iter()
-        .collect::<Vec<_>>()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_4y_sd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct1
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct2
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct5
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct95
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct98
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct99
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct1_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct2_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct5_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct95_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct98_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct99_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+
+        iter
     }
 }

crates/brk_computer/src/grouped/value_from_dateindex.rs

@@ -1,5 +1,4 @@
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, DateIndex, Dollars, Sats, Version};
 use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, StoredVec};
 
@@ -43,7 +42,7 @@ impl ComputedValueVecsFromDateIndex {
             )?,
             bitcoin: ComputedVecsFromDateIndex::forced_import(
                 db,
-                &format!("{name}_in_btc"),
+                &format!("{name}_btc"),
                 Source::Compute,
                 version + VERSION,
                 indexes,
@@ -52,7 +51,7 @@ impl ComputedValueVecsFromDateIndex {
             dollars: compute_dollars.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_in_usd"),
+                    &format!("{name}_usd"),
                     Source::Compute,
                     version + VERSION,
                     indexes,
@@ -65,40 +64,24 @@ impl ComputedValueVecsFromDateIndex {
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(
+        F: FnMut(&mut EagerVec<DateIndex, Sats>) -> Result<()>,
-            &mut EagerVec<DateIndex, Sats>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
     {
-        compute(
+        compute(self.sats.dateindex.as_mut().unwrap())?;
-            self.sats.dateindex.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
 
         let dateindex: Option<&StoredVec<DateIndex, Sats>> = None;
-        self.compute_rest(indexer, indexes, price, starting_indexes, exit, dateindex)?;
+        self.compute_rest(price, starting_indexes, exit, dateindex)?;
 
         Ok(())
     }
 
     pub fn compute_rest(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
@@ -108,72 +91,50 @@ impl ComputedValueVecsFromDateIndex {
             self.sats
                 .compute_rest(starting_indexes, exit, Some(dateindex))?;
 
-            self.bitcoin.compute_all(
+            self.bitcoin.compute_all(starting_indexes, exit, |v| {
-                indexer,
+                v.compute_from_sats(starting_indexes.dateindex, dateindex, exit)
-                indexes,
+            })?;
-                starting_indexes,
-                exit,
-                |v, _, _, starting_indexes, exit| {
-                    v.compute_from_sats(starting_indexes.dateindex, dateindex, exit)
-                },
-            )?;
         } else {
             let dateindex: Option<&StoredVec<DateIndex, Sats>> = None;
 
             self.sats.compute_rest(starting_indexes, exit, dateindex)?;
 
-            self.bitcoin.compute_all(
+            self.bitcoin.compute_all(starting_indexes, exit, |v| {
-                indexer,
+                v.compute_from_sats(
-                indexes,
+                    starting_indexes.dateindex,
-                starting_indexes,
+                    self.sats.dateindex.as_ref().unwrap(),
-                exit,
+                    exit,
-                |v, _, _, starting_indexes, exit| {
+                )
-                    v.compute_from_sats(
+            })?;
-                        starting_indexes.dateindex,
-                        self.sats.dateindex.as_ref().unwrap(),
-                        exit,
-                    )
-                },
-            )?;
         }
 
         let dateindex_to_bitcoin = self.bitcoin.dateindex.as_ref().unwrap();
-        let dateindex_to_close = price
+        let dateindex_to_price_close = price
             .as_ref()
             .unwrap()
-            .timeindexes_to_close
+            .timeindexes_to_price_close
             .dateindex
             .as_ref()
             .unwrap();
 
         if let Some(dollars) = self.dollars.as_mut() {
-            dollars.compute_all(
+            dollars.compute_all(starting_indexes, exit, |v| {
-                indexer,
+                v.compute_from_bitcoin(
-                indexes,
+                    starting_indexes.dateindex,
-                starting_indexes,
+                    dateindex_to_bitcoin,
-                exit,
+                    dateindex_to_price_close,
-                |v, _, _, starting_indexes, exit| {
+                    exit,
-                    v.compute_from_bitcoin(
+                )
-                        starting_indexes.dateindex,
+            })?;
-                        dateindex_to_bitcoin,
-                        dateindex_to_close,
-                        exit,
-                    )
-                },
-            )?;
         }
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        std::iter::empty()
-            self.sats.vecs(),
+            .chain(self.sats.iter_any_collectable())
-            self.bitcoin.vecs(),
+            .chain(self.bitcoin.iter_any_collectable())
-            self.dollars.as_ref().map_or(vec![], |v| v.vecs()),
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
     }
 }

crates/brk_computer/src/grouped/value_from_height.rs

@@ -1,5 +1,5 @@
+use allocative::Allocative;
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, Dollars, Height, Sats, Version};
 use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, StoredVec};
 
@@ -12,7 +12,7 @@ use crate::{
 
 use super::{ComputedVecsFromHeight, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedValueVecsFromHeight {
     pub sats: ComputedVecsFromHeight<Sats>,
     pub bitcoin: ComputedVecsFromHeight<Bitcoin>,
@@ -43,7 +43,7 @@ impl ComputedValueVecsFromHeight {
             )?,
             bitcoin: ComputedVecsFromHeight::forced_import(
                 db,
-                &format!("{name}_in_btc"),
+                &format!("{name}_btc"),
                 Source::Compute,
                 version + VERSION,
                 indexes,
@@ -52,7 +52,7 @@ impl ComputedValueVecsFromHeight {
             dollars: compute_dollars.then(|| {
                 ComputedVecsFromHeight::forced_import(
                     db,
-                    &format!("{name}_in_usd"),
+                    &format!("{name}_usd"),
                     Source::Compute,
                     version + VERSION,
                     indexes,
@@ -65,7 +65,6 @@ impl ComputedValueVecsFromHeight {
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
@@ -73,31 +72,18 @@ impl ComputedValueVecsFromHeight {
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(
+        F: FnMut(&mut EagerVec<Height, Sats>) -> Result<()>,
-            &mut EagerVec<Height, Sats>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
     {
-        compute(
+        compute(self.sats.height.as_mut().unwrap())?;
-            self.sats.height.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
 
         let height: Option<&StoredVec<Height, Sats>> = None;
-        self.compute_rest(indexer, indexes, price, starting_indexes, exit, height)?;
+        self.compute_rest(indexes, price, starting_indexes, exit, height)?;
 
         Ok(())
     }
 
     pub fn compute_rest(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
@@ -108,67 +94,47 @@ impl ComputedValueVecsFromHeight {
             self.sats
                 .compute_rest(indexes, starting_indexes, exit, Some(height))?;
 
-            self.bitcoin.compute_all(
+            self.bitcoin
-                indexer,
+                .compute_all(indexes, starting_indexes, exit, |v| {
-                indexes,
-                starting_indexes,
-                exit,
-                |v, _, _, starting_indexes, exit| {
                     v.compute_from_sats(starting_indexes.height, height, exit)
-                },
+                })?;
-            )?;
         } else {
             let height: Option<&StoredVec<Height, Sats>> = None;
 
             self.sats
                 .compute_rest(indexes, starting_indexes, exit, height)?;
 
-            self.bitcoin.compute_all(
+            self.bitcoin
-                indexer,
+                .compute_all(indexes, starting_indexes, exit, |v| {
-                indexes,
-                starting_indexes,
-                exit,
-                |v, _, _, starting_indexes, exit| {
                     v.compute_from_sats(
                         starting_indexes.height,
                         self.sats.height.as_ref().unwrap(),
                         exit,
                     )
-                },
+                })?;
-            )?;
         }
 
         let height_to_bitcoin = self.bitcoin.height.as_ref().unwrap();
-        let height_to_close = &price.as_ref().unwrap().chainindexes_to_close.height;
+        let height_to_price_close = &price.as_ref().unwrap().chainindexes_to_price_close.height;
 
         if let Some(dollars) = self.dollars.as_mut() {
-            dollars.compute_all(
+            dollars.compute_all(indexes, starting_indexes, exit, |v| {
-                indexer,
+                v.compute_from_bitcoin(
-                indexes,
+                    starting_indexes.height,
-                starting_indexes,
+                    height_to_bitcoin,
-                exit,
+                    height_to_price_close,
-                |v, _, _, starting_indexes, exit| {
+                    exit,
-                    v.compute_from_bitcoin(
+                )
-                        starting_indexes.height,
+            })?;
-                        height_to_bitcoin,
-                        height_to_close,
-                        exit,
-                    )
-                },
-            )?;
         }
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        std::iter::empty()
-            self.sats.vecs(),
+            .chain(self.sats.iter_any_collectable())
-            self.bitcoin.vecs(),
+            .chain(self.bitcoin.iter_any_collectable())
-            self.dollars.as_ref().map_or(vec![], |v| v.vecs()),
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
     }
 }

crates/brk_computer/src/grouped/value_from_txindex.rs

@@ -1,3 +1,4 @@
+use allocative::Allocative;
 use brk_error::Result;
 use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, Close, Dollars, Height, Sats, TxIndex, Version};
@@ -10,7 +11,7 @@ use crate::{Indexes, grouped::Source, indexes, price};
 
 use super::{ComputedVecsFromTxindex, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedValueVecsFromTxindex {
     pub sats: ComputedVecsFromTxindex<Sats>,
     pub bitcoin_txindex: LazyVecFrom1<TxIndex, Bitcoin, TxIndex, Sats>,
@@ -37,8 +38,8 @@ impl ComputedValueVecsFromTxindex {
     ) -> Result<Self> {
         let compute_dollars = price.is_some();
 
-        let name_in_btc = format!("{name}_in_btc");
+        let name_btc = format!("{name}_btc");
-        let name_in_usd = format!("{name}_in_usd");
+        let name_usd = format!("{name}_usd");
 
         let sats = ComputedVecsFromTxindex::forced_import(
             db,
@@ -52,7 +53,7 @@ impl ComputedValueVecsFromTxindex {
         let source_vec = source.vec();
 
         let bitcoin_txindex = LazyVecFrom1::init(
-            &name_in_btc,
+            &name_btc,
             version + VERSION,
             source_vec.map_or_else(|| sats.txindex.as_ref().unwrap().boxed_clone(), |s| s),
             |txindex: TxIndex, iter| {
@@ -65,7 +66,7 @@ impl ComputedValueVecsFromTxindex {
 
         let bitcoin = ComputedVecsFromTxindex::forced_import(
             db,
-            &name_in_btc,
+            &name_btc,
             Source::None,
             version + VERSION,
             indexes,
@@ -74,15 +75,15 @@ impl ComputedValueVecsFromTxindex {
 
         let dollars_txindex = price.map(|price| {
             LazyVecFrom3::init(
-                &name_in_usd,
+                &name_usd,
                 version + VERSION,
                 bitcoin_txindex.boxed_clone(),
                 indexes.txindex_to_height.boxed_clone(),
-                price.chainindexes_to_close.height.boxed_clone(),
+                price.chainindexes_to_price_close.height.boxed_clone(),
                 |txindex: TxIndex,
                  txindex_to_btc_iter,
                  txindex_to_height_iter,
-                 height_to_close_iter| {
+                 height_to_price_close_iter| {
                     let txindex = txindex.unwrap_to_usize();
                     txindex_to_btc_iter.next_at(txindex).and_then(|(_, value)| {
                         let btc = value.into_owned();
@@ -90,7 +91,7 @@ impl ComputedValueVecsFromTxindex {
                             .next_at(txindex)
                             .and_then(|(_, value)| {
                                 let height = value.into_owned();
-                                height_to_close_iter
+                                height_to_price_close_iter
                                     .next_at(height.unwrap_to_usize())
                                     .map(|(_, close)| *close.into_owned() * btc)
                             })
@@ -107,7 +108,7 @@ impl ComputedValueVecsFromTxindex {
             dollars: compute_dollars.then(|| {
                 ComputedVecsFromTxindex::forced_import(
                     db,
-                    &name_in_usd,
+                    &name_usd,
                     Source::None,
                     version + VERSION,
                     indexes,
@@ -201,18 +202,16 @@ impl ComputedValueVecsFromTxindex {
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        [&self.bitcoin_txindex as &dyn AnyCollectableVec]
-            self.sats.vecs(),
+            .into_iter()
-            vec![&self.bitcoin_txindex as &dyn AnyCollectableVec],
+            .chain(self.sats.iter_any_collectable())
-            self.bitcoin.vecs(),
+            .chain(self.bitcoin.iter_any_collectable())
-            self.dollars_txindex
+            .chain(
-                .as_ref()
+                self.dollars_txindex
-                .map_or(vec![], |v| vec![v as &dyn AnyCollectableVec]),
+                    .iter()
-            self.dollars.as_ref().map_or(vec![], |v| v.vecs()),
+                    .map(|v| v as &dyn AnyCollectableVec),
-        ]
+            )
-        .into_iter()
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
-        .flatten()
-        .collect::<Vec<_>>()
     }
 }

crates/brk_computer/src/grouped/value_height.rs

@@ -1,12 +1,11 @@
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, Dollars, Height, Sats, Version};
 use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, Format, StoredVec};
 
 use crate::{
     Indexes,
     grouped::Source,
-    indexes, price,
+    price,
     traits::{ComputeFromBitcoin, ComputeFromSats},
 };
 
@@ -35,14 +34,14 @@ impl ComputedHeightValueVecs {
             }),
             bitcoin: EagerVec::forced_import(
                 db,
-                &format!("{name}_in_btc"),
+                &format!("{name}_btc"),
                 version + VERSION + Version::ZERO,
                 format,
             )?,
             dollars: compute_dollars.then(|| {
                 EagerVec::forced_import(
                     db,
-                    &format!("{name}_in_usd"),
+                    &format!("{name}_usd"),
                     version + VERSION + Version::ZERO,
                     format,
                 )
@@ -53,29 +52,15 @@ impl ComputedHeightValueVecs {
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(
+        F: FnMut(&mut EagerVec<Height, Sats>) -> Result<()>,
-            &mut EagerVec<Height, Sats>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
     {
-        compute(
+        compute(self.sats.as_mut().unwrap())?;
-            self.sats.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
 
         let height: Option<&StoredVec<Height, Sats>> = None;
         self.compute_rest(price, starting_indexes, exit, height)?;
@@ -102,13 +87,13 @@ impl ComputedHeightValueVecs {
         }
 
         let height_to_bitcoin = &self.bitcoin;
-        let height_to_close = &price.as_ref().unwrap().chainindexes_to_close.height;
+        let height_to_price_close = &price.as_ref().unwrap().chainindexes_to_price_close.height;
 
         if let Some(dollars) = self.dollars.as_mut() {
             dollars.compute_from_bitcoin(
                 starting_indexes.height,
                 height_to_bitcoin,
-                height_to_close,
+                height_to_price_close,
                 exit,
             )?;
         }
@@ -116,14 +101,10 @@ impl ComputedHeightValueVecs {
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        [&self.bitcoin as &dyn AnyCollectableVec]
-            vec![&self.bitcoin as &dyn AnyCollectableVec],
+            .into_iter()
-            self.sats.as_ref().map_or(vec![], |v| vec![v]),
+            .chain(self.sats.iter().map(|v| v as &dyn AnyCollectableVec))
-            self.dollars.as_ref().map_or(vec![], |v| vec![v]),
+            .chain(self.dollars.iter().map(|v| v as &dyn AnyCollectableVec))
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
     }
 }

crates/brk_computer/src/indexes.rs

@@ -236,7 +236,7 @@ impl Vecs {
             |index, _| Some(index),
         );
 
-        Ok(Self {
+        let this = Self {
             emptyoutputindex_to_emptyoutputindex,
             inputindex_to_inputindex,
             opreturnindex_to_opreturnindex,
@@ -472,7 +472,15 @@ impl Vecs {
             )?,
 
             db,
-        })
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
     }
 
     pub fn compute(
@@ -925,9 +933,9 @@ impl Vecs {
         })
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        vec![
+        [
-            &self.dateindex_to_date,
+            &self.dateindex_to_date as &dyn AnyCollectableVec,
             &self.dateindex_to_dateindex,
             &self.dateindex_to_first_height,
             &self.dateindex_to_height_count,
@@ -988,6 +996,7 @@ impl Vecs {
             &self.yearindex_to_yearindex,
             &self.outputindex_to_txindex,
         ]
+        .into_iter()
     }
 }
 

crates/brk_computer/src/lib.rs

@@ -5,45 +5,47 @@ use std::path::Path;
 use brk_error::Result;
 use brk_fetcher::Fetcher;
 use brk_indexer::Indexer;
+use brk_parser::Parser;
 use brk_structs::Version;
 use log::info;
 use vecdb::{AnyCollectableVec, Exit, Format};
 
-mod blocks;
+mod blks;
+mod chain;
 mod cointime;
 mod constants;
 mod fetched;
 mod grouped;
 mod indexes;
 mod market;
-mod mining;
+mod pools;
 mod price;
 mod stateful;
 mod states;
 mod traits;
-mod transactions;
 mod utils;
 
 use indexes::Indexes;
 
+pub use pools::*;
 pub use states::PriceToAmount;
 use states::*;
 
 #[derive(Clone)]
 pub struct Computer {
-    pub indexes: indexes::Vecs,
+    pub chain: chain::Vecs,
-    pub constants: constants::Vecs,
-    pub blocks: blocks::Vecs,
-    pub mining: mining::Vecs,
-    pub market: market::Vecs,
-    pub price: Option<price::Vecs>,
-    pub transactions: transactions::Vecs,
-    pub stateful: stateful::Vecs,
-    pub fetched: Option<fetched::Vecs>,
     pub cointime: cointime::Vecs,
+    pub constants: constants::Vecs,
+    pub fetched: Option<fetched::Vecs>,
+    pub indexes: indexes::Vecs,
+    pub market: market::Vecs,
+    pub pools: pools::Vecs,
+    pub blks: blks::Vecs,
+    pub price: Option<price::Vecs>,
+    pub stateful: stateful::Vecs,
 }
 
-const VERSION: Version = Version::TWO;
+const VERSION: Version = Version::new(4);
 
 impl Computer {
     /// Do NOT import multiple times or things will break !!!
@@ -52,6 +54,8 @@ impl Computer {
         indexer: &Indexer,
         fetcher: Option<Fetcher>,
     ) -> Result<Self> {
+        info!("Importing computer...");
+
         let computed_path = outputs_path.join("computed");
 
         let indexes =
@@ -66,8 +70,6 @@ impl Computer {
         });
 
         Ok(Self {
-            blocks: blocks::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes)?,
-            mining: mining::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes)?,
             constants: constants::Vecs::forced_import(
                 &computed_path,
                 VERSION + Version::ZERO,
@@ -81,13 +83,20 @@ impl Computer {
                 &indexes,
                 price.as_ref(),
             )?,
-            transactions: transactions::Vecs::forced_import(
+            chain: chain::Vecs::forced_import(
                 &computed_path,
                 VERSION + Version::ZERO,
                 indexer,
                 &indexes,
                 price.as_ref(),
             )?,
+            blks: blks::Vecs::forced_import(&computed_path, VERSION + Version::ZERO)?,
+            pools: pools::Vecs::forced_import(
+                &computed_path,
+                VERSION + Version::ZERO,
+                &indexes,
+                price.as_ref(),
+            )?,
             cointime: cointime::Vecs::forced_import(
                 &computed_path,
                 VERSION + Version::ZERO,
@@ -104,30 +113,18 @@ impl Computer {
         &mut self,
         indexer: &Indexer,
         starting_indexes: brk_indexer::Indexes,
+        parser: &Parser,
         exit: &Exit,
     ) -> Result<()> {
         info!("Computing indexes...");
         let mut starting_indexes = self.indexes.compute(indexer, starting_indexes, exit)?;
 
-        info!("Computing constants...");
-        self.constants
-            .compute(indexer, &self.indexes, &starting_indexes, exit)?;
-
-        info!("Computing blocks...");
-        self.blocks
-            .compute(indexer, &self.indexes, &starting_indexes, exit)?;
-
-        info!("Computing mining...");
-        self.mining
-            .compute(indexer, &self.indexes, &starting_indexes, exit)?;
-
         if let Some(fetched) = self.fetched.as_mut() {
             info!("Computing fetched...");
             fetched.compute(indexer, &self.indexes, &starting_indexes, exit)?;
 
             info!("Computing prices...");
             self.price.as_mut().unwrap().compute(
-                indexer,
                 &self.indexes,
                 &starting_indexes,
                 fetched,
@@ -135,44 +132,73 @@ impl Computer {
             )?;
         }
 
-        info!("Computing transactions...");
+        info!("Computing BLKs metadata...");
-        self.transactions.compute(
+        self.blks
+            .compute(indexer, &self.indexes, &starting_indexes, parser, exit)?;
+
+        std::thread::scope(|scope| -> Result<()> {
+            let constants = scope.spawn(|| -> Result<()> {
+                info!("Computing constants...");
+                self.constants
+                    .compute(&self.indexes, &starting_indexes, exit)?;
+                Ok(())
+            });
+
+            // let blks = scope.spawn(|| -> Result<()> {
+            //     info!("Computing blks...");
+            //     self.blks
+            //         .compute(indexer, &self.indexes, &starting_indexes, parser, exit)?;
+            //     Ok(())
+            // });
+
+            let chain = scope.spawn(|| -> Result<()> {
+                info!("Computing chain...");
+                self.chain.compute(
+                    indexer,
+                    &self.indexes,
+                    &starting_indexes,
+                    self.price.as_ref(),
+                    exit,
+                )?;
+                Ok(())
+            });
+
+            if let Some(price) = self.price.as_ref() {
+                info!("Computing market...");
+                self.market.compute(price, &starting_indexes, exit)?;
+            }
+
+            constants.join().unwrap()?;
+            // blks.join().unwrap()?;
+            chain.join().unwrap()?;
+            Ok(())
+        })?;
+
+        self.pools.compute(
             indexer,
             &self.indexes,
             &starting_indexes,
+            &self.chain,
             self.price.as_ref(),
             exit,
         )?;
 
-        if let Some(price) = self.price.as_ref() {
-            info!("Computing market...");
-            self.market.compute(
-                indexer,
-                &self.indexes,
-                price,
-                &mut self.transactions,
-                &starting_indexes,
-                exit,
-            )?;
-        }
-
         info!("Computing stateful...");
         self.stateful.compute(
             indexer,
             &self.indexes,
-            &self.transactions,
+            &self.chain,
             self.price.as_ref(),
-            &self.market,
             &mut starting_indexes,
             exit,
         )?;
 
+        info!("Computing cointime...");
         self.cointime.compute(
-            indexer,
             &self.indexes,
             &starting_indexes,
             self.price.as_ref(),
-            &self.transactions,
+            &self.chain,
             &self.stateful,
             exit,
         )?;
@@ -180,25 +206,57 @@ impl Computer {
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
-        [
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
-            self.constants.vecs(),
+            Box::new(self.fetched.iter().flat_map(|v| v.iter_any_collectable()));
-            self.indexes.vecs(),
+
-            self.blocks.vecs(),
+        iter = Box::new(iter.chain(self.chain.iter_any_collectable()));
-            self.mining.vecs(),
+        iter = Box::new(iter.chain(self.cointime.iter_any_collectable()));
-            self.market.vecs(),
+        iter = Box::new(iter.chain(self.constants.iter_any_collectable()));
-            self.transactions.vecs(),
+        iter = Box::new(iter.chain(self.indexes.iter_any_collectable()));
-            self.stateful.vecs(),
+        iter = Box::new(iter.chain(self.market.iter_any_collectable()));
-            self.cointime.vecs(),
+        iter = Box::new(iter.chain(self.pools.iter_any_collectable()));
-            self.fetched.as_ref().map_or(vec![], |v| v.vecs()),
+        iter = Box::new(iter.chain(self.blks.iter_any_collectable()));
-            self.price.as_ref().map_or(vec![], |v| v.vecs()),
+        iter = Box::new(iter.chain(self.price.iter().flat_map(|v| v.iter_any_collectable())));
-        ]
+        iter = Box::new(iter.chain(self.stateful.iter_any_collectable()));
-        .into_iter()
+
-        .flatten()
+        iter
-        .collect::<Vec<_>>()
     }
 
     pub fn static_clone(&self) -> &'static Self {
         Box::leak(Box::new(self.clone()))
     }
 }
+
+// pub fn generate_allocation_files(monitored: &pools::Vecs) -> Result<()> {
+//     info!("Generating Allocative files...");
+
+//     let mut flamegraph = allocative::FlameGraphBuilder::default();
+//     flamegraph.visit_root(monitored);
+//     let output = flamegraph.finish();
+
+//     let folder = format!(
+//         "at-{}",
+//         jiff::Timestamp::now().strftime("%Y-%m-%d_%Hh%Mm%Ss"),
+//     );
+
+//     let path = std::path::PathBuf::from(&format!("./target/flamegraph/{folder}"));
+//     std::fs::create_dir_all(&path)?;
+
+//     // fs::write(path.join("flamegraph.src"), &output.flamegraph())?;
+
+//     let mut fg_svg = Vec::new();
+//     inferno::flamegraph::from_reader(
+//         &mut inferno::flamegraph::Options::default(),
+//         output.flamegraph().write().as_bytes(),
+//         &mut fg_svg,
+//     )?;
+
+//     std::fs::write(path.join("flamegraph.svg"), &fg_svg)?;
+
+//     std::fs::write(path.join("warnings.txt"), output.warnings())?;
+
+//     info!("Successfully generated Allocative files");
+
+//     Ok(())
+// }

crates/brk_computer/src/mining.rs

@@ -1,151 +0,0 @@
-use std::path::Path;
-
-use brk_error::Result;
-use brk_indexer::Indexer;
-use brk_structs::{DifficultyEpoch, HalvingEpoch, StoredF64, Version};
-use vecdb::{AnyCollectableVec, Database, Exit, PAGE_SIZE, VecIterator};
-
-use crate::grouped::Source;
-
-use super::{
-    Indexes,
-    grouped::{ComputedVecsFromDateIndex, ComputedVecsFromHeight, VecBuilderOptions},
-    indexes,
-};
-
-const VERSION: Version = Version::ZERO;
-
-#[derive(Clone)]
-pub struct Vecs {
-    db: Database,
-
-    pub indexes_to_difficulty: ComputedVecsFromHeight<StoredF64>,
-    pub indexes_to_difficultyepoch: ComputedVecsFromDateIndex<DifficultyEpoch>,
-    pub indexes_to_halvingepoch: ComputedVecsFromDateIndex<HalvingEpoch>,
-}
-
-impl Vecs {
-    pub fn forced_import(parent: &Path, version: Version, indexes: &indexes::Vecs) -> Result<Self> {
-        let db = Database::open(&parent.join("mining"))?;
-        db.set_min_len(PAGE_SIZE * 1_000_000)?;
-
-        Ok(Self {
-            indexes_to_difficulty: ComputedVecsFromHeight::forced_import(
-                &db,
-                "difficulty",
-                Source::None,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_last(),
-            )?,
-            indexes_to_difficultyepoch: ComputedVecsFromDateIndex::forced_import(
-                &db,
-                "difficultyepoch",
-                Source::Compute,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_last(),
-            )?,
-            indexes_to_halvingepoch: ComputedVecsFromDateIndex::forced_import(
-                &db,
-                "halvingepoch",
-                Source::Compute,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_last(),
-            )?,
-
-            db,
-        })
-    }
-
-    pub fn compute(
-        &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        starting_indexes: &Indexes,
-        exit: &Exit,
-    ) -> Result<()> {
-        self.compute_(indexer, indexes, starting_indexes, exit)?;
-        self.db.flush_then_punch()?;
-        Ok(())
-    }
-
-    fn compute_(
-        &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        starting_indexes: &Indexes,
-        exit: &Exit,
-    ) -> Result<()> {
-        let mut height_to_difficultyepoch_iter = indexes.height_to_difficultyepoch.into_iter();
-        self.indexes_to_difficultyepoch.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                let mut height_count_iter = indexes.dateindex_to_height_count.into_iter();
-                vec.compute_transform(
-                    starting_indexes.dateindex,
-                    &indexes.dateindex_to_first_height,
-                    |(di, height, ..)| {
-                        (
-                            di,
-                            height_to_difficultyepoch_iter.unwrap_get_inner(
-                                height + (*height_count_iter.unwrap_get_inner(di) - 1),
-                            ),
-                        )
-                    },
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        let mut height_to_halvingepoch_iter = indexes.height_to_halvingepoch.into_iter();
-        self.indexes_to_halvingepoch.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                let mut height_count_iter = indexes.dateindex_to_height_count.into_iter();
-                vec.compute_transform(
-                    starting_indexes.dateindex,
-                    &indexes.dateindex_to_first_height,
-                    |(di, height, ..)| {
-                        (
-                            di,
-                            height_to_halvingepoch_iter.unwrap_get_inner(
-                                height + (*height_count_iter.unwrap_get_inner(di) - 1),
-                            ),
-                        )
-                    },
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        self.indexes_to_difficulty.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&indexer.vecs.height_to_difficulty),
-        )?;
-
-        Ok(())
-    }
-
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.indexes_to_difficulty.vecs(),
-            self.indexes_to_difficultyepoch.vecs(),
-            self.indexes_to_halvingepoch.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
-    }
-}

crates/brk_computer/src/pools/mod.rs

@@ -0,0 +1,227 @@
+use std::{collections::BTreeMap, path::Path};
+
+use allocative::Allocative;
+use brk_error::Result;
+use brk_indexer::Indexer;
+use brk_store::AnyStore;
+use brk_structs::{AddressBytes, Height, OutputIndex, OutputType, PoolId, Pools, pools};
+use rayon::prelude::*;
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, Exit, GenericStoredVec,
+    PAGE_SIZE, RawVec, StoredIndex, VecIterator, Version,
+};
+
+mod vecs;
+
+use crate::{
+    chain,
+    indexes::{self, Indexes},
+    price,
+};
+
+#[derive(Clone, Allocative)]
+pub struct Vecs {
+    db: Database,
+    pools: &'static Pools,
+    height_to_pool: RawVec<Height, PoolId>,
+
+    vecs: BTreeMap<PoolId, vecs::Vecs>,
+}
+
+impl Vecs {
+    pub fn forced_import(
+        parent_path: &Path,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+    ) -> Result<Self> {
+        let db = Database::open(&parent_path.join("pools"))?;
+        db.set_min_len(PAGE_SIZE * 1_000_000)?;
+        let pools = pools();
+
+        let version = parent_version + Version::new(3) + Version::new(pools.len() as u64);
+
+        let this = Self {
+            height_to_pool: RawVec::forced_import(&db, "pool", version + Version::ZERO)?,
+            vecs: pools
+                .iter()
+                .map(|pool| {
+                    vecs::Vecs::forced_import(
+                        &db,
+                        pool.id,
+                        pools,
+                        version + Version::ZERO,
+                        indexes,
+                        price,
+                    )
+                    .map(|vecs| (pool.id, vecs))
+                })
+                .collect::<Result<BTreeMap<_, _>>>()?,
+            pools,
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexer, indexes, starting_indexes, chain, price, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_height_to_pool(indexer, indexes, starting_indexes, exit)?;
+
+        self.vecs.par_iter_mut().try_for_each(|(_, vecs)| {
+            vecs.compute(
+                indexes,
+                starting_indexes,
+                &self.height_to_pool,
+                chain,
+                price,
+                exit,
+            )
+        })?;
+
+        Ok(())
+    }
+
+    fn compute_height_to_pool(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.height_to_pool.validate_computed_version_or_reset(
+            self.height_to_pool.version() + indexer.stores.height_to_coinbase_tag.version(),
+        )?;
+
+        let mut height_to_first_txindex_iter = indexer.vecs.height_to_first_txindex.iter();
+        let mut txindex_to_first_outputindex_iter =
+            indexer.vecs.txindex_to_first_outputindex.iter();
+        let mut txindex_to_output_count_iter = indexes.txindex_to_output_count.iter();
+        let mut outputindex_to_outputtype_iter = indexer.vecs.outputindex_to_outputtype.iter();
+        let mut outputindex_to_typeindex_iter = indexer.vecs.outputindex_to_typeindex.iter();
+        let mut p2pk65addressindex_to_p2pk65bytes_iter =
+            indexer.vecs.p2pk65addressindex_to_p2pk65bytes.iter();
+        let mut p2pk33addressindex_to_p2pk33bytes_iter =
+            indexer.vecs.p2pk33addressindex_to_p2pk33bytes.iter();
+        let mut p2pkhaddressindex_to_p2pkhbytes_iter =
+            indexer.vecs.p2pkhaddressindex_to_p2pkhbytes.iter();
+        let mut p2shaddressindex_to_p2shbytes_iter =
+            indexer.vecs.p2shaddressindex_to_p2shbytes.iter();
+        let mut p2wpkhaddressindex_to_p2wpkhbytes_iter =
+            indexer.vecs.p2wpkhaddressindex_to_p2wpkhbytes.iter();
+        let mut p2wshaddressindex_to_p2wshbytes_iter =
+            indexer.vecs.p2wshaddressindex_to_p2wshbytes.iter();
+        let mut p2traddressindex_to_p2trbytes_iter =
+            indexer.vecs.p2traddressindex_to_p2trbytes.iter();
+        let mut p2aaddressindex_to_p2abytes_iter = indexer.vecs.p2aaddressindex_to_p2abytes.iter();
+
+        let unknown = self.pools.get_unknown();
+
+        let min = starting_indexes
+            .height
+            .unwrap_to_usize()
+            .min(self.height_to_pool.len());
+
+        indexer
+            .stores
+            .height_to_coinbase_tag
+            .iter()
+            .skip(min)
+            .try_for_each(|(height, coinbase_tag)| -> Result<()> {
+                let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+                let outputindex = txindex_to_first_outputindex_iter.unwrap_get_inner(txindex);
+                let outputcount = txindex_to_output_count_iter.unwrap_get_inner(txindex);
+
+                let pool = (*outputindex..(*outputindex + *outputcount))
+                    .map(OutputIndex::from)
+                    .find_map(|outputindex| {
+                        let outputtype =
+                            outputindex_to_outputtype_iter.unwrap_get_inner(outputindex);
+                        let typeindex = outputindex_to_typeindex_iter.unwrap_get_inner(outputindex);
+
+                        let address = match outputtype {
+                            OutputType::P2PK65 => Some(AddressBytes::from(
+                                p2pk65addressindex_to_p2pk65bytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2PK33 => Some(AddressBytes::from(
+                                p2pk33addressindex_to_p2pk33bytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2PKH => Some(AddressBytes::from(
+                                p2pkhaddressindex_to_p2pkhbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2SH => Some(AddressBytes::from(
+                                p2shaddressindex_to_p2shbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2WPKH => Some(AddressBytes::from(
+                                p2wpkhaddressindex_to_p2wpkhbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2WSH => Some(AddressBytes::from(
+                                p2wshaddressindex_to_p2wshbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2TR => Some(AddressBytes::from(
+                                p2traddressindex_to_p2trbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2A => Some(AddressBytes::from(
+                                p2aaddressindex_to_p2abytes_iter.unwrap_get_inner(typeindex.into()),
+                            )),
+                            _ => None,
+                        };
+
+                        address
+                            .and_then(|address| self.pools.find_from_address(&address.to_string()))
+                    })
+                    .or_else(|| self.pools.find_from_coinbase_tag(&coinbase_tag))
+                    .unwrap_or(unknown);
+
+                self.height_to_pool.push_if_needed(height, pool.id)?;
+                Ok(())
+            })?;
+
+        self.height_to_pool.safe_flush(exit)?;
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.height_to_pool as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(
+                self.vecs
+                    .iter()
+                    .flat_map(|(_, vecs)| vecs.iter_any_collectable()),
+            )
+    }
+}

crates/brk_computer/src/pools/vecs.rs

@@ -0,0 +1,397 @@
+use allocative::Allocative;
+use brk_error::Result;
+use brk_structs::{Height, PoolId, Pools, Sats, StoredF32, StoredU16, StoredU32};
+use vecdb::{AnyCollectableVec, AnyIterableVec, Database, Exit, StoredIndex, VecIterator, Version};
+
+use crate::{
+    chain,
+    grouped::{
+        ComputedValueVecsFromHeight, ComputedVecsFromDateIndex, ComputedVecsFromHeight, Source,
+        VecBuilderOptions,
+    },
+    indexes::{self, Indexes},
+    price,
+};
+
+#[derive(Clone, Allocative)]
+pub struct Vecs {
+    id: PoolId,
+
+    indexes_to_blocks_mined: ComputedVecsFromHeight<StoredU32>,
+    indexes_to_1w_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_1m_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_1y_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_subsidy: ComputedValueVecsFromHeight,
+    indexes_to_fee: ComputedValueVecsFromHeight,
+    indexes_to_coinbase: ComputedValueVecsFromHeight,
+    indexes_to_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1d_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1w_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1m_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1y_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_days_since_block: ComputedVecsFromDateIndex<StoredU16>,
+}
+
+impl Vecs {
+    pub fn forced_import(
+        db: &Database,
+        id: PoolId,
+        pools: &Pools,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+    ) -> Result<Self> {
+        let pool = pools.get(id);
+        let name = pool.serialized_id();
+        let suffix = |s: &str| format!("{name}_{s}");
+        let compute_dollars = price.is_some();
+        let version = parent_version + Version::ZERO;
+
+        Ok(Self {
+            id,
+            indexes_to_blocks_mined: ComputedVecsFromHeight::forced_import(
+                db,
+                &suffix("blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+            )?,
+            indexes_to_1w_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1w_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1m_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1m_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1y_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1y_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_subsidy: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("subsidy"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_fee: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("fee"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_coinbase: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("coinbase"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1d_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1d_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1w_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1w_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1m_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1m_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1y_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1y_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_days_since_block: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("days_since_block"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+        })
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn compute(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        height_to_pool: &impl AnyIterableVec<Height, PoolId>,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.indexes_to_blocks_mined
+            .compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_transform(
+                    starting_indexes.height,
+                    height_to_pool,
+                    |(h, id, ..)| {
+                        (
+                            h,
+                            if id == self.id {
+                                StoredU32::ONE
+                            } else {
+                                StoredU32::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1w_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    7,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1m_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    30,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1y_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    365,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        let height_to_blocks_mined = self.indexes_to_blocks_mined.height.as_ref().unwrap();
+
+        self.indexes_to_subsidy
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_subsidy.sats.height.as_ref().unwrap(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_fee
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_fee.sats.height.unwrap_sum(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_coinbase
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_coinbase.sats.height.as_ref().unwrap(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_cumulative(),
+                    chain.indexes_to_block_count.dateindex.unwrap_cumulative(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1d_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    chain.indexes_to_block_count.dateindex.unwrap_sum(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1w_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1w_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1w_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1m_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1m_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1m_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1y_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1y_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1y_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_days_since_block
+            .compute_all(starting_indexes, exit, |v| {
+                let mut prev = None;
+                v.compute_transform2(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    self.indexes_to_blocks_mined.dateindex.unwrap_cumulative(),
+                    |(i, sum, cumulative, slf)| {
+                        if prev.is_none() {
+                            let i = i.unwrap_to_usize();
+                            prev.replace(if i > 0 {
+                                slf.into_iter().unwrap_get_inner_(i - 1)
+                            } else {
+                                StoredU16::ZERO
+                            });
+                        }
+                        let days = if !cumulative.is_zero() && sum.is_zero() {
+                            prev.unwrap() + StoredU16::ONE
+                        } else {
+                            StoredU16::ZERO
+                        };
+                        prev.replace(days);
+                        (i, days)
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
+
+        iter = Box::new(iter.chain(self.indexes_to_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1w_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1m_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1y_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_subsidy.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_fee.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_coinbase.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1d_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1w_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1m_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1y_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_days_since_block.iter_any_collectable()));
+
+        iter
+    }
+}

crates/brk_computer/src/stateful/address_cohort.rs

@@ -1,7 +1,6 @@
-use std::{ops::Deref, path::Path};
+use std::path::Path;
 
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, DateIndex, Dollars, Height, StoredU64, Version};
 use vecdb::{
     AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, EagerVec, Exit, Format,
@@ -11,7 +10,7 @@ use vecdb::{
 use crate::{
     Indexes,
     grouped::{ComputedVecsFromHeight, Source, VecBuilderOptions},
-    indexes, market, price,
+    indexes, price,
     stateful::{
         common,
         r#trait::{CohortVecs, DynCohortVecs},
@@ -29,8 +28,8 @@ pub struct Vecs {
 
     pub inner: common::Vecs,
 
-    pub height_to_address_count: EagerVec<Height, StoredU64>,
+    pub height_to_addr_count: EagerVec<Height, StoredU64>,
-    pub indexes_to_address_count: ComputedVecsFromHeight<StoredU64>,
+    pub indexes_to_addr_count: ComputedVecsFromHeight<StoredU64>,
 }
 
 impl Vecs {
@@ -43,7 +42,7 @@ impl Vecs {
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         states_path: Option<&Path>,
-        compute_relative_to_all: bool,
+        compute_rel_to_all: bool,
     ) -> Result<Self> {
         let compute_dollars = price.is_some();
 
@@ -58,15 +57,15 @@ impl Vecs {
                     compute_dollars,
                 )
             }),
-            height_to_address_count: EagerVec::forced_import(
+            height_to_addr_count: EagerVec::forced_import(
                 db,
-                &suffix("address_count"),
+                &suffix("addr_count"),
                 version + VERSION + Version::ZERO,
                 format,
             )?,
-            indexes_to_address_count: ComputedVecsFromHeight::forced_import(
+            indexes_to_addr_count: ComputedVecsFromHeight::forced_import(
                 db,
-                &suffix("address_count"),
+                &suffix("addr_count"),
                 Source::None,
                 version + VERSION + Version::ZERO,
                 indexes,
@@ -79,23 +78,27 @@ impl Vecs {
                 version,
                 indexes,
                 price,
-                compute_relative_to_all,
                 false,
+                compute_rel_to_all,
                 false,
             )?,
         })
     }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        self.inner
+            .iter_any_collectable()
+            .chain([&self.height_to_addr_count as &dyn AnyCollectableVec])
+            .chain(self.indexes_to_addr_count.iter_any_collectable())
+    }
 }
 
 impl DynCohortVecs for Vecs {
     fn min_height_vecs_len(&self) -> usize {
-        [
+        std::cmp::min(
-            self.height_to_address_count.len(),
+            self.height_to_addr_count.len(),
             self.inner.min_height_vecs_len(),
-        ]
+        )
-        .into_iter()
-        .min()
-        .unwrap()
     }
 
     fn reset_state_starting_height(&mut self) {
@@ -110,8 +113,8 @@ impl DynCohortVecs for Vecs {
         self.starting_height = Some(starting_height);
 
         if let Some(prev_height) = starting_height.decremented() {
-            self.state.as_mut().unwrap().address_count = *self
+            self.state.as_mut().unwrap().addr_count = *self
-                .height_to_address_count
+                .height_to_addr_count
                 .into_iter()
                 .unwrap_get_inner(prev_height);
         }
@@ -120,9 +123,9 @@ impl DynCohortVecs for Vecs {
     }
 
     fn validate_computed_versions(&mut self, base_version: Version) -> Result<()> {
-        self.height_to_address_count
+        self.height_to_addr_count
             .validate_computed_version_or_reset(
-                base_version + self.height_to_address_count.inner_version(),
+                base_version + self.height_to_addr_count.inner_version(),
             )?;
 
         self.inner.validate_computed_versions(base_version)
@@ -133,9 +136,9 @@ impl DynCohortVecs for Vecs {
             return Ok(());
         }
 
-        self.height_to_address_count.forced_push_at(
+        self.height_to_addr_count.forced_push_at(
             height,
-            self.state.as_ref().unwrap().address_count.into(),
+            self.state.as_ref().unwrap().addr_count.into(),
             exit,
         )?;
 
@@ -162,7 +165,7 @@ impl DynCohortVecs for Vecs {
     }
 
     fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
-        self.height_to_address_count.safe_flush(exit)?;
+        self.height_to_addr_count.safe_flush(exit)?;
 
         self.inner
             .safe_flush_stateful_vecs(height, exit, &mut self.state.as_mut().unwrap().inner)
@@ -171,30 +174,20 @@ impl DynCohortVecs for Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        self.indexes_to_address_count.compute_rest(
+        self.indexes_to_addr_count.compute_rest(
             indexes,
             starting_indexes,
             exit,
-            Some(&self.height_to_address_count),
+            Some(&self.height_to_addr_count),
         )?;
 
         self.inner
-            .compute_rest_part1(indexer, indexes, price, starting_indexes, exit)
+            .compute_rest_part1(indexes, price, starting_indexes, exit)
-    }
-
-    fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.inner.vecs(),
-            self.indexes_to_address_count.vecs(),
-            vec![&self.height_to_address_count],
-        ]
-        .concat()
     }
 }
 
@@ -205,11 +198,11 @@ impl CohortVecs for Vecs {
         others: &[&Self],
         exit: &Exit,
     ) -> Result<()> {
-        self.height_to_address_count.compute_sum_of_others(
+        self.height_to_addr_count.compute_sum_of_others(
             starting_indexes.height,
             others
                 .iter()
-                .map(|v| &v.height_to_address_count)
+                .map(|v| &v.height_to_addr_count)
                 .collect::<Vec<_>>()
                 .as_slice(),
             exit,
@@ -224,35 +217,28 @@ impl CohortVecs for Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,
     ) -> Result<()> {
         self.inner.compute_rest_part2(
-            indexer,
             indexes,
             price,
             starting_indexes,
-            market,
             height_to_supply,
             dateindex_to_supply,
+            height_to_market_cap,
+            dateindex_to_market_cap,
             height_to_realized_cap,
             dateindex_to_realized_cap,
             exit,
         )
     }
 }
-
-impl Deref for Vecs {
-    type Target = common::Vecs;
-    fn deref(&self) -> &Self::Target {
-        &self.inner
-    }
-}

crates/brk_computer/src/stateful/address_cohorts.rs

@@ -1,7 +1,6 @@
 use std::path::Path;
 
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{
     AddressGroups, Bitcoin, ByAmountRange, ByGreatEqualAmount, ByLowerThanAmount, DateIndex,
     Dollars, GroupFilter, Height, Version,
@@ -10,7 +9,7 @@ use derive_deref::{Deref, DerefMut};
 use vecdb::{AnyIterableVec, Database, Exit, Format};
 
 use crate::{
-    Indexes, indexes, market, price,
+    Indexes, indexes, price,
     stateful::{
         address_cohort,
         r#trait::{CohortVecs, DynCohortVecs},
@@ -459,18 +458,17 @@ impl Vecs {
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        let by_size_range = self.0.amount_range.as_vec();
+        let by_size_range = &self.0.amount_range;
 
         [
             self.0
                 .ge_amount
-                .as_mut_vec()
+                .iter_mut()
-                .into_iter()
                 .map(|(filter, vecs)| {
                     (
                         vecs,
                         by_size_range
-                            .into_iter()
+                            .iter()
                             .filter(|(other, _)| filter.includes(other))
                             .map(|(_, v)| v)
                             .collect::<Vec<_>>(),
@@ -479,13 +477,12 @@ impl Vecs {
                 .collect::<Vec<_>>(),
             self.0
                 .lt_amount
-                .as_mut_vec()
+                .iter_mut()
-                .into_iter()
                 .map(|(filter, vecs)| {
                     (
                         vecs,
                         by_size_range
-                            .into_iter()
+                            .iter()
                             .filter(|(other, _)| filter.includes(other))
                             .map(|(_, v)| v)
                             .collect::<Vec<_>>(),
@@ -502,51 +499,48 @@ impl Vecs {
 
     pub fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        self.as_mut_vecs().into_iter().try_for_each(|(_, v)| {
+        self.iter_mut()
-            v.compute_rest_part1(indexer, indexes, price, starting_indexes, exit)
+            .into_iter()
-        })
+            .try_for_each(|(_, v)| v.compute_rest_part1(indexes, price, starting_indexes, exit))
     }
 
     #[allow(clippy::too_many_arguments)]
     pub fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,
     ) -> Result<()> {
-        self.0.as_boxed_mut_vecs().into_iter().try_for_each(|v| {
+        self.0.iter_mut().try_for_each(|(_, v)| {
-            v.into_iter().try_for_each(|(_, v)| {
+            v.compute_rest_part2(
-                v.compute_rest_part2(
+                indexes,
-                    indexer,
+                price,
-                    indexes,
+                starting_indexes,
-                    price,
+                height_to_supply,
-                    starting_indexes,
+                dateindex_to_supply,
-                    market,
+                height_to_market_cap,
-                    height_to_supply,
+                dateindex_to_market_cap,
-                    dateindex_to_supply,
+                height_to_realized_cap,
-                    height_to_realized_cap,
+                dateindex_to_realized_cap,
-                    dateindex_to_realized_cap,
+                exit,
-                    exit,
+            )
-                )
-            })
         })
     }
 
     pub fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
-        self.as_mut_separate_vecs()
+        self.iter_separate_mut()
             .into_iter()
             .try_for_each(|(_, v)| v.safe_flush_stateful_vecs(height, exit))
     }

crates/brk_computer/src/stateful/addresstype/indexes_to_addresscount.rs

@@ -75,11 +75,9 @@ impl AddressTypeToIndexesToAddressCount {
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
         self.0
-            .as_typed_vec()
+            .iter_typed()
-            .into_iter()
+            .flat_map(|(_, v)| v.iter_any_collectable())
-            .flat_map(|(_, v)| v.vecs())
-            .collect::<Vec<_>>()
     }
 }

crates/brk_computer/src/stateful/addresstype/typeindex_tree.rs

@@ -27,6 +27,10 @@ impl<T> AddressTypeToTypeIndexTree<T> {
             mem::swap(own, other);
         }
     }
+
+    pub fn unwrap(self) -> ByAddressType<BTreeMap<TypeIndex, T>> {
+        self.0
+    }
 }
 
 impl<T> Default for AddressTypeToTypeIndexTree<T> {

crates/brk_computer/src/stateful/addresstype/vec.rs

@@ -38,6 +38,10 @@ impl<T> AddressTypeToVec<T> {
             mem::swap(own, other);
         }
     }
+
+    pub fn unwrap(self) -> ByAddressType<Vec<T>> {
+        self.0
+    }
 }
 
 impl<T> Default for AddressTypeToVec<T> {

crates/brk_computer/src/stateful/trait.rs

@@ -1,9 +1,8 @@
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, DateIndex, Dollars, Height, Version};
-use vecdb::{AnyCollectableVec, AnyIterableVec, Exit};
+use vecdb::{AnyIterableVec, Exit};
 
-use crate::{Indexes, indexes, market, price};
+use crate::{Indexes, indexes, price};
 
 pub trait DynCohortVecs: Send + Sync {
     fn min_height_vecs_len(&self) -> usize;
@@ -29,14 +28,11 @@ pub trait DynCohortVecs: Send + Sync {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()>;
-
-    fn vecs(&self) -> Vec<&dyn AnyCollectableVec>;
 }
 
 pub trait CohortVecs: DynCohortVecs {
@@ -50,13 +46,13 @@ pub trait CohortVecs: DynCohortVecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,

crates/brk_computer/src/stateful/utxo_cohort.rs

@@ -1,12 +1,11 @@
 use std::{ops::Deref, path::Path};
 
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, DateIndex, Dollars, Height, Version};
-use vecdb::{AnyCollectableVec, AnyIterableVec, Database, Exit, Format};
+use vecdb::{AnyIterableVec, Database, Exit, Format};
 
 use crate::{
-    Indexes, UTXOCohortState, indexes, market, price,
+    Indexes, UTXOCohortState, indexes, price,
     stateful::{
         common,
         r#trait::{CohortVecs, DynCohortVecs},
@@ -32,8 +31,8 @@ impl Vecs {
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         states_path: Option<&Path>,
-        compute_relative_to_all: bool,
+        extended: bool,
-        ratio_extended: bool,
+        compute_rel_to_all: bool,
         compute_adjusted: bool,
     ) -> Result<Self> {
         let compute_dollars = price.is_some();
@@ -56,8 +55,8 @@ impl Vecs {
                 version,
                 indexes,
                 price,
-                compute_relative_to_all,
+                extended,
-                ratio_extended,
+                compute_rel_to_all,
                 compute_adjusted,
             )?,
         })
@@ -122,18 +121,13 @@ impl DynCohortVecs for Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
         self.inner
-            .compute_rest_part1(indexer, indexes, price, starting_indexes, exit)
+            .compute_rest_part1(indexes, price, starting_indexes, exit)
-    }
-
-    fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        self.inner.vecs()
     }
 }
 
@@ -154,25 +148,25 @@ impl CohortVecs for Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,
     ) -> Result<()> {
         self.inner.compute_rest_part2(
-            indexer,
             indexes,
             price,
             starting_indexes,
-            market,
             height_to_supply,
             dateindex_to_supply,
+            height_to_market_cap,
+            dateindex_to_market_cap,
             height_to_realized_cap,
             dateindex_to_realized_cap,
             exit,

crates/brk_computer/src/stateful/utxo_cohorts.rs

@@ -1,7 +1,6 @@
 use std::{collections::BTreeMap, ops::ControlFlow, path::Path};
 
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{
     Bitcoin, ByAgeRange, ByAmountRange, ByEpoch, ByGreatEqualAmount, ByLowerThanAmount, ByMaxAge,
     ByMinAge, BySpendableType, ByTerm, CheckedSub, DateIndex, Dollars, GroupFilter, HalvingEpoch,
@@ -11,7 +10,7 @@ use derive_deref::{Deref, DerefMut};
 use vecdb::{AnyIterableVec, Database, Exit, Format, StoredIndex};
 
 use crate::{
-    Indexes, indexes, market, price,
+    Indexes, indexes, price,
     stateful::r#trait::DynCohortVecs,
     states::{BlockState, Transacted},
 };
@@ -38,18 +37,18 @@ impl Vecs {
                     db,
                     None,
                     format,
-                    version + VERSION + Version::ZERO,
+                    version + VERSION + Version::ONE,
                     indexes,
                     price,
                     None,
-                    false,
                     true,
+                    false,
                     true,
                 )?,
                 term: ByTerm {
                     short: utxo_cohort::Vecs::forced_import(
                         db,
-                        Some("short_term_holders"),
+                        Some("sth"),
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
@@ -61,7 +60,7 @@ impl Vecs {
                     )?,
                     long: utxo_cohort::Vecs::forced_import(
                         db,
-                        Some("long_term_holders"),
+                        Some("lth"),
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
@@ -143,8 +142,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2pk33: utxo_cohort::Vecs::forced_import(
@@ -155,8 +154,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2pkh: utxo_cohort::Vecs::forced_import(
@@ -167,8 +166,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2sh: utxo_cohort::Vecs::forced_import(
@@ -179,8 +178,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2wpkh: utxo_cohort::Vecs::forced_import(
@@ -191,8 +190,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2wsh: utxo_cohort::Vecs::forced_import(
@@ -203,8 +202,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2tr: utxo_cohort::Vecs::forced_import(
@@ -215,8 +214,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2a: utxo_cohort::Vecs::forced_import(
@@ -227,8 +226,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2ms: utxo_cohort::Vecs::forced_import(
@@ -239,8 +238,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     empty: utxo_cohort::Vecs::forced_import(
@@ -251,8 +250,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     unknown: utxo_cohort::Vecs::forced_import(
@@ -263,8 +262,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                 },
@@ -955,8 +954,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1sat_to_10sats: utxo_cohort::Vecs::forced_import(
@@ -967,8 +966,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10sats_to_100sats: utxo_cohort::Vecs::forced_import(
@@ -979,8 +978,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100sats_to_1k_sats: utxo_cohort::Vecs::forced_import(
@@ -991,8 +990,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_sats_to_10k_sats: utxo_cohort::Vecs::forced_import(
@@ -1003,8 +1002,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_sats_to_100k_sats: utxo_cohort::Vecs::forced_import(
@@ -1015,8 +1014,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100k_sats_to_1m_sats: utxo_cohort::Vecs::forced_import(
@@ -1027,8 +1026,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1m_sats_to_10m_sats: utxo_cohort::Vecs::forced_import(
@@ -1039,8 +1038,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10m_sats_to_1btc: utxo_cohort::Vecs::forced_import(
@@ -1051,8 +1050,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1btc_to_10btc: utxo_cohort::Vecs::forced_import(
@@ -1063,8 +1062,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10btc_to_100btc: utxo_cohort::Vecs::forced_import(
@@ -1075,8 +1074,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100btc_to_1k_btc: utxo_cohort::Vecs::forced_import(
@@ -1087,8 +1086,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_btc_to_10k_btc: utxo_cohort::Vecs::forced_import(
@@ -1099,8 +1098,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_btc_to_100k_btc: utxo_cohort::Vecs::forced_import(
@@ -1111,8 +1110,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100k_btc_or_more: utxo_cohort::Vecs::forced_import(
@@ -1123,8 +1122,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                 },
@@ -1137,8 +1136,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100sats: utxo_cohort::Vecs::forced_import(
@@ -1149,8 +1148,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_sats: utxo_cohort::Vecs::forced_import(
@@ -1161,8 +1160,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_sats: utxo_cohort::Vecs::forced_import(
@@ -1173,8 +1172,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100k_sats: utxo_cohort::Vecs::forced_import(
@@ -1185,8 +1184,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1m_sats: utxo_cohort::Vecs::forced_import(
@@ -1197,8 +1196,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10m_sats: utxo_cohort::Vecs::forced_import(
@@ -1209,8 +1208,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1btc: utxo_cohort::Vecs::forced_import(
@@ -1221,8 +1220,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10btc: utxo_cohort::Vecs::forced_import(
@@ -1233,8 +1232,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100btc: utxo_cohort::Vecs::forced_import(
@@ -1245,8 +1244,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_btc: utxo_cohort::Vecs::forced_import(
@@ -1257,8 +1256,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_btc: utxo_cohort::Vecs::forced_import(
@@ -1269,8 +1268,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100k_btc: utxo_cohort::Vecs::forced_import(
@@ -1281,8 +1280,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                 },
@@ -1295,8 +1294,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10sats: utxo_cohort::Vecs::forced_import(
@@ -1307,8 +1306,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100sats: utxo_cohort::Vecs::forced_import(
@@ -1319,8 +1318,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_sats: utxo_cohort::Vecs::forced_import(
@@ -1331,8 +1330,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_sats: utxo_cohort::Vecs::forced_import(
@@ -1343,8 +1342,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100k_sats: utxo_cohort::Vecs::forced_import(
@@ -1355,8 +1354,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1m_sats: utxo_cohort::Vecs::forced_import(
@@ -1367,8 +1366,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10m_sats: utxo_cohort::Vecs::forced_import(
@@ -1379,8 +1378,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1btc: utxo_cohort::Vecs::forced_import(
@@ -1391,8 +1390,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10btc: utxo_cohort::Vecs::forced_import(
@@ -1403,8 +1402,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100btc: utxo_cohort::Vecs::forced_import(
@@ -1415,8 +1414,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_btc: utxo_cohort::Vecs::forced_import(
@@ -1427,8 +1426,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_btc: utxo_cohort::Vecs::forced_import(
@@ -1439,8 +1438,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                 },
@@ -1458,8 +1457,7 @@ impl Vecs {
 
         let mut vecs = self
             .age_range
-            .as_mut_vec()
+            .iter_mut()
-            .into_iter()
             .map(|(filter, v)| (filter, &mut v.state))
             .collect::<Vec<_>>();
 
@@ -1503,9 +1501,8 @@ impl Vecs {
         let mut time_based_vecs = self
             .0
             .age_range
-            .as_mut_vec()
+            .iter_mut()
-            .into_iter()
+            .chain(self.0.epoch.iter_mut())
-            .chain(self.0.epoch.as_mut_vec())
             .collect::<Vec<_>>();
 
         let last_timestamp = chain_state.last().unwrap().timestamp;
@@ -1551,8 +1548,10 @@ impl Vecs {
                     );
                 });
 
-            sent.by_type.spendable.as_typed_vec().into_iter().for_each(
+            sent.by_type
-                |(output_type, supply_state)| {
+                .spendable
+                .iter_typed()
+                .for_each(|(output_type, supply_state)| {
                     self.0
                         ._type
                         .get_mut(output_type)
@@ -1568,12 +1567,10 @@ impl Vecs {
                             days_old_float,
                             older_than_hour,
                         )
-                },
+                });
-            );
 
             sent.by_size_group
-                .as_typed_vec()
+                .iter_typed()
-                .into_iter()
                 .for_each(|(group, supply_state)| {
                     self.0
                         .amount_range
@@ -1606,24 +1603,20 @@ impl Vecs {
             v.state.as_mut().unwrap().receive(&supply_state, price);
         });
 
-        self._type
+        self._type.iter_mut().for_each(|(filter, vecs)| {
-            .as_mut_vec()
+            let output_type = match filter {
-            .into_iter()
+                GroupFilter::Type(output_type) => *output_type,
-            .for_each(|(filter, vecs)| {
+                _ => unreachable!(),
-                let output_type = match filter {
+            };
-                    GroupFilter::Type(output_type) => *output_type,
+            vecs.state
-                    _ => unreachable!(),
+                .as_mut()
-                };
+                .unwrap()
-                vecs.state
+                .receive(received.by_type.get(output_type), price)
-                    .as_mut()
+        });
-                    .unwrap()
-                    .receive(received.by_type.get(output_type), price)
-            });
 
         received
             .by_size_group
-            .as_typed_vec()
+            .iter_typed()
-            .into_iter()
             .for_each(|(group, supply_state)| {
                 self.amount_range
                     .get_mut(group)
@@ -1640,89 +1633,64 @@ impl Vecs {
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        let by_date_range = self.0.age_range.as_vec();
+        let by_date_range = &self.0.age_range;
-        let by_size_range = self.0.amount_range.as_vec();
+        let by_size_range = &self.0.amount_range;
 
-        [
+        [(
-            vec![(&mut self.0.all.1, self.0.epoch.vecs().to_vec())],
+            &mut self.0.all.1,
-            self.0
+            by_date_range.iter().map(|(_, v)| v).collect::<Vec<_>>(),
-                .min_age
+        )]
-                .as_mut_vec()
-                .into_iter()
-                .map(|(filter, vecs)| {
-                    (
-                        vecs,
-                        by_date_range
-                            .into_iter()
-                            .filter(|(other, _)| filter.includes(other))
-                            .map(|(_, v)| v)
-                            .collect::<Vec<_>>(),
-                    )
-                })
-                .collect::<Vec<_>>(),
-            self.0
-                .max_age
-                .as_mut_vec()
-                .into_iter()
-                .map(|(filter, vecs)| {
-                    (
-                        vecs,
-                        by_date_range
-                            .into_iter()
-                            .filter(|(other, _)| filter.includes(other))
-                            .map(|(_, v)| v)
-                            .collect::<Vec<_>>(),
-                    )
-                })
-                .collect::<Vec<_>>(),
-            self.0
-                .term
-                .as_mut_vec()
-                .into_iter()
-                .map(|(filter, vecs)| {
-                    (
-                        vecs,
-                        by_date_range
-                            .into_iter()
-                            .filter(|(other, _)| filter.includes(other))
-                            .map(|(_, v)| v)
-                            .collect::<Vec<_>>(),
-                    )
-                })
-                .collect::<Vec<_>>(),
-            self.0
-                .ge_amount
-                .as_mut_vec()
-                .into_iter()
-                .map(|(filter, vecs)| {
-                    (
-                        vecs,
-                        by_size_range
-                            .into_iter()
-                            .filter(|(other, _)| filter.includes(other))
-                            .map(|(_, v)| v)
-                            .collect::<Vec<_>>(),
-                    )
-                })
-                .collect::<Vec<_>>(),
-            self.0
-                .lt_amount
-                .as_mut_vec()
-                .into_iter()
-                .map(|(filter, vecs)| {
-                    (
-                        vecs,
-                        by_size_range
-                            .into_iter()
-                            .filter(|(other, _)| filter.includes(other))
-                            .map(|(_, v)| v)
-                            .collect::<Vec<_>>(),
-                    )
-                })
-                .collect::<Vec<_>>(),
-        ]
         .into_iter()
-        .flatten()
+        .chain(self.0.min_age.iter_mut().map(|(filter, vecs)| {
+            (
+                vecs,
+                by_date_range
+                    .iter()
+                    .filter(|(other, _)| filter.includes(other))
+                    .map(|(_, v)| v)
+                    .collect::<Vec<_>>(),
+            )
+        }))
+        .chain(self.0.max_age.iter_mut().map(|(filter, vecs)| {
+            (
+                vecs,
+                by_date_range
+                    .iter()
+                    .filter(|(other, _)| filter.includes(other))
+                    .map(|(_, v)| v)
+                    .collect::<Vec<_>>(),
+            )
+        }))
+        .chain(self.0.term.iter_mut().map(|(filter, vecs)| {
+            (
+                vecs,
+                by_date_range
+                    .iter()
+                    .filter(|(other, _)| filter.includes(other))
+                    .map(|(_, v)| v)
+                    .collect::<Vec<_>>(),
+            )
+        }))
+        .chain(self.0.ge_amount.iter_mut().map(|(filter, vecs)| {
+            (
+                vecs,
+                by_size_range
+                    .iter()
+                    .filter(|(other, _)| filter.includes(other))
+                    .map(|(_, v)| v)
+                    .collect::<Vec<_>>(),
+            )
+        }))
+        .chain(self.0.lt_amount.iter_mut().map(|(filter, vecs)| {
+            (
+                vecs,
+                by_size_range
+                    .iter()
+                    .filter(|(other, _)| filter.includes(other))
+                    .map(|(_, v)| v)
+                    .collect::<Vec<_>>(),
+            )
+        }))
         .try_for_each(|(vecs, stateful)| {
             vecs.compute_from_stateful(starting_indexes, &stateful, exit)
         })
@@ -1730,52 +1698,48 @@ impl Vecs {
 
     pub fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        self.as_mut_vecs().into_iter().try_for_each(|(_, v)| {
+        self.iter_mut()
-            v.compute_rest_part1(indexer, indexes, price, starting_indexes, exit)
+            .into_iter()
-        })
+            .try_for_each(|(_, v)| v.compute_rest_part1(indexes, price, starting_indexes, exit))
     }
 
     #[allow(clippy::too_many_arguments)]
     pub fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,
     ) -> Result<()> {
-        self.0.as_boxed_mut_vecs().into_iter().try_for_each(|v| {
+        self.iter_mut().into_iter().try_for_each(|(_, v)| {
-            v.into_iter().try_for_each(|(_, v)| {
+            v.compute_rest_part2(
-                v.compute_rest_part2(
+                indexes,
-                    indexer,
+                price,
-                    indexes,
+                starting_indexes,
-                    price,
+                height_to_supply,
-                    starting_indexes,
+                dateindex_to_supply,
-                    market,
+                height_to_market_cap,
-                    height_to_supply,
+                dateindex_to_market_cap,
-                    dateindex_to_supply,
+                height_to_realized_cap,
-                    height_to_realized_cap,
+                dateindex_to_realized_cap,
-                    dateindex_to_realized_cap,
+                exit,
-                    exit,
+            )
-                )
-            })
         })
     }
 
     pub fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
-        self.as_mut_separate_vecs()
+        self.iter_separate_mut()
-            .into_iter()
             .try_for_each(|(_, v)| v.safe_flush_stateful_vecs(height, exit))
     }
 }

crates/brk_computer/src/states/cohorts/address.rs

@@ -9,14 +9,14 @@ use super::CohortState;
 
 #[derive(Clone)]
 pub struct AddressCohortState {
-    pub address_count: u64,
+    pub addr_count: u64,
     pub inner: CohortState,
 }
 
 impl AddressCohortState {
     pub fn new(path: &Path, name: &str, compute_dollars: bool) -> Self {
         Self {
-            address_count: 0,
+            addr_count: 0,
             inner: CohortState::new(path, name, compute_dollars),
         }
     }
@@ -44,14 +44,14 @@ impl AddressCohortState {
 
         let prev_realized_price = compute_price.then(|| addressdata.realized_price());
         let prev_supply_state = SupplyState {
-            utxos: addressdata.outputs_len as u64,
+            utxos: addressdata.utxos as u64,
             value: addressdata.amount(),
         };
 
         addressdata.send(value, prev_price)?;
 
         let supply_state = SupplyState {
-            utxos: addressdata.outputs_len as u64,
+            utxos: addressdata.utxos as u64,
             value: addressdata.amount(),
         };
 
@@ -79,14 +79,14 @@ impl AddressCohortState {
 
         let prev_realized_price = compute_price.then(|| address_data.realized_price());
         let prev_supply_state = SupplyState {
-            utxos: address_data.outputs_len as u64,
+            utxos: address_data.utxos as u64,
             value: address_data.amount(),
         };
 
         address_data.receive(value, price);
 
         let supply_state = SupplyState {
-            utxos: address_data.outputs_len as u64,
+            utxos: address_data.utxos as u64,
             value: address_data.amount(),
         };
 
@@ -99,7 +99,7 @@ impl AddressCohortState {
     }
 
     pub fn add(&mut self, addressdata: &LoadedAddressData) {
-        self.address_count += 1;
+        self.addr_count += 1;
         self.inner.increment_(
             &addressdata.into(),
             addressdata.realized_cap,
@@ -108,7 +108,7 @@ impl AddressCohortState {
     }
 
     pub fn subtract(&mut self, addressdata: &LoadedAddressData) {
-        self.address_count = self.address_count.checked_sub(1).unwrap();
+        self.addr_count = self.addr_count.checked_sub(1).unwrap();
         self.inner.decrement_(
             &addressdata.into(),
             addressdata.realized_cap,

crates/brk_computer/src/states/cohorts/common.rs

@@ -257,10 +257,11 @@ impl CohortState {
 
         let update_state =
             |price: Dollars, current_price: Dollars, sats: Sats, state: &mut UnrealizedState| {
-                match price.cmp(&current_price) {
+                let cmp = price.cmp(&current_price);
-                    Ordering::Less => {
+                match cmp {
+                    Ordering::Equal | Ordering::Less => {
                         state.supply_in_profit += sats;
-                        if price > Dollars::ZERO && current_price > Dollars::ZERO {
+                        if !cmp.is_eq() && price > Dollars::ZERO && current_price > Dollars::ZERO {
                             let diff = current_price.checked_sub(price).unwrap();
                             // Add back once in a while to verify, but generally not needed
                             // if diff <= Dollars::ZERO {
@@ -282,9 +283,6 @@ impl CohortState {
                             state.unrealized_loss += diff * sats;
                         }
                     }
-                    Ordering::Equal => {
-                        state.supply_even += sats;
-                    }
                 }
             };
 

crates/brk_computer/src/states/supply.rs

@@ -43,8 +43,14 @@ impl SubAssign<&SupplyState> for SupplyState {
 impl From<&LoadedAddressData> for SupplyState {
     fn from(value: &LoadedAddressData) -> Self {
         Self {
-            utxos: value.outputs_len as u64,
+            utxos: value.utxos as u64,
             value: value.amount(),
         }
     }
 }
+
+impl std::fmt::Display for SupplyState {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "utxos: {}, value: {}", self.utxos, self.value)
+    }
+}

crates/brk_computer/src/states/unrealized.rs

@@ -3,7 +3,6 @@ use brk_structs::{Dollars, Sats};
 #[derive(Debug, Default, Clone)]
 pub struct UnrealizedState {
     pub supply_in_profit: Sats,
-    pub supply_even: Sats,
     pub supply_in_loss: Sats,
     pub unrealized_profit: Dollars,
     pub unrealized_loss: Dollars,
@@ -12,7 +11,6 @@ pub struct UnrealizedState {
 impl UnrealizedState {
     pub const NAN: Self = Self {
         supply_in_profit: Sats::ZERO,
-        supply_even: Sats::ZERO,
         supply_in_loss: Sats::ZERO,
         unrealized_profit: Dollars::NAN,
         unrealized_loss: Dollars::NAN,
@@ -20,7 +18,6 @@ impl UnrealizedState {
 
     pub const ZERO: Self = Self {
         supply_in_profit: Sats::ZERO,
-        supply_even: Sats::ZERO,
         supply_in_loss: Sats::ZERO,
         unrealized_profit: Dollars::ZERO,
         unrealized_loss: Dollars::ZERO,

crates/brk_error/Cargo.toml

@@ -10,10 +10,11 @@ rust-version.workspace = true
 build = "build.rs"
 
 [dependencies]
-vecdb = { workspace = true }
+bitcoin = { workspace = true }
 bitcoincore-rpc = { workspace = true }
 fjall = { workspace = true }
 jiff = { workspace = true }
 minreq = { workspace = true }
-serde_json = { workspace = true }
+sonic-rs = { workspace = true }
+vecdb = { workspace = true }
 zerocopy = { workspace = true }

crates/brk_error/README.md

@@ -1,140 +1,143 @@
 # brk_error
 
-**Centralized error handling for the Bitcoin Research Kit**
+Centralized error handling for Bitcoin-related operations and database interactions.
 
-`brk_error` provides a unified error type and result system used throughout the BRK ecosystem. It consolidates error handling from multiple external dependencies and adds Bitcoin-specific error variants.
+[![Crates.io](https://img.shields.io/crates/v/brk_error.svg)](https://crates.io/crates/brk_error)
+[![Documentation](https://docs.rs/brk_error/badge.svg)](https://docs.rs/brk_error)
 
-## What it provides
+## Overview
 
-- **Unified Error Type**: Single `Error` enum that covers all error cases across BRK crates
+This crate provides a unified error type that consolidates error handling across Bitcoin blockchain analysis tools. It wraps errors from multiple external libraries including Bitcoin Core RPC, database operations, HTTP requests, and serialization operations into a single `Error` enum.
-- **Convenient Result Type**: Pre-configured `Result<T, E = Error>` for consistent error handling
-- **External Error Integration**: Automatic conversions from common library errors
-- **Bitcoin-Specific Errors**: Domain-specific error variants for blockchain data processing
 
-## Key Features
+**Key Features:**
 
-### Centralized Error Management
+- Unified error type covering 11+ different error sources
-- Single error type for the entire BRK ecosystem
+- Automatic conversions from external library errors
-- Consistent error handling patterns across all crates
+- Bitcoin-specific error variants for blockchain operations
-- Reduced error type complexity in public APIs
+- Database error handling for both Fjall and VecDB storage backends
+- Custom error types for domain-specific validation failures
 
-### External Library Integration
+**Target Use Cases:**
-Automatic `From` implementations for errors from:
-- **I/O Operations**: `std::io::Error`
-- **Bitcoin Core RPC**: `bitcoincore_rpc::Error`
-- **Database Operations**: `fjall::Error`, `vecdb::Error`
-- **Serialization**: `serde_json::Error`
-- **Time Operations**: `jiff::Error`, `SystemTimeError`
-- **HTTP Requests**: `minreq::Error`
-- **Zero-Copy Operations**: `zerocopy` conversion errors
 
-### Bitcoin-Specific Error Variants
+- Applications processing Bitcoin blockchain data
-- `WrongAddressType` - Invalid address type for operation
+- Systems requiring unified error handling across multiple storage backends
-- `UnindexableDate` - Date before Bitcoin genesis (2009-01-03)
+- Tools integrating Bitcoin Core RPC with local databases
-- `WrongLength` - Invalid data length for Bitcoin structures
-- `QuickCacheError` - Cache operation failures
 
-## Usage
+## Installation
 
-### Basic Error Handling
+```bash
+cargo add brk_error
+```
+
+## Quick Start
 
 ```rust
 use brk_error::{Error, Result};
 
-fn process_block() -> Result<Block> {
+fn process_transaction() -> Result<()> {
-    let rpc_client = get_rpc_client()?;  // bitcoincore_rpc::Error -> Error
+    // Function automatically converts various error types
-    let block_data = rpc_client.get_block_info(&hash)?;
+    let data = std::fs::read("transaction.json")?; // IO error auto-converted
+    let parsed: serde_json::Value = serde_json::from_slice(&data)?; // JSON error auto-converted
 
-    // Custom Bitcoin-specific validation
+    // Custom domain errors
-    if block_data.height < 0 {
+    if data.len() < 32 {
-        return Err(Error::Str("Invalid block height"));
+        return Err(Error::WrongLength);
     }
 
-    Ok(block_data)
+    Ok(())
 }
 ```
 
-### Working with External Libraries
+## API Overview
+
+### Core Types
+
+- **`Error`**: Main error enum consolidating all error types
+- **`Result<T, E = Error>`**: Type alias for `std::result::Result` with default `Error` type
+
+### Error Categories
+
+**External Library Errors:**
+
+- `BitcoinRPC`: Bitcoin Core RPC client errors
+- `BitcoinConsensusEncode`: Bitcoin consensus encoding failures
+- `Fjall`/`VecDB`/`SeqDB`: Database operation errors
+- `Minreq`: HTTP request errors
+- `SerdeJson`: JSON serialization errors
+- `Jiff`: Date/time handling errors
+- `ZeroCopyError`: Zero-copy conversion failures
+
+**Domain-Specific Errors:**
+
+- `WrongLength`: Invalid data length for Bitcoin operations
+- `WrongAddressType`: Unsupported Bitcoin address format
+- `UnindexableDate`: Date outside valid blockchain range (before 2009-01-03)
+- `QuickCacheError`: Cache operation failures
+
+**Generic Errors:**
+
+- `Str(&'static str)`: Static string errors
+- `String(String)`: Dynamic string errors
+
+### Key Methods
+
+All external error types automatically convert to `Error` via `From` trait implementations. The error type implements `std::error::Error`, `Debug`, and `Display` traits for comprehensive error reporting.
+
+## Examples
+
+### Bitcoin RPC Integration
+
+```rust
+use brk_error::Result;
+use bitcoincore_rpc::{Client, Auth};
+
+fn get_block_count(client: &Client) -> Result<u64> {
+    let count = client.get_block_count()?; // Auto-converts bitcoincore_rpc::Error
+    Ok(count)
+}
+```
+
+### Database Operations
 
 ```rust
 use brk_error::Result;
 
-fn save_data(data: &[u8]) -> Result<()> {
+fn store_transaction_data(db: &fjall::Keyspace, data: &[u8]) -> Result<()> {
-    // I/O error automatically converted
+    if data.len() != 32 {
-    std::fs::write("data.bin", data)?;
+        return Err(brk_error::Error::WrongLength);
-    
+    }
-    // JSON serialization error automatically converted  
-    let json = serde_json::to_string(&data)?;
-    
-    // Database error automatically converted
-    database.insert("key", &json)?;
 
+    db.insert(b"tx_hash", data)?; // Auto-converts fjall::Error
     Ok(())
 }
 ```
 
-### Bitcoin-Specific Validation
+### Date Validation
 
 ```rust
 use brk_error::{Error, Result};
+use jiff::civil::Date;
 
-fn validate_date(date: &Date) -> Result<()> {
+fn validate_blockchain_date(date: Date) -> Result<()> {
-    if *date < Date::new(2009, 1, 3) {
+    let genesis_date = Date::constant(2009, 1, 3);
+    let earliest_valid = Date::constant(2009, 1, 9);
+
+    if date < genesis_date || (date > genesis_date && date < earliest_valid) {
         return Err(Error::UnindexableDate);
     }
-    Ok(())
-}
 
-fn validate_address_type(output_type: OutputType) -> Result<()> {
-    if !output_type.is_address() {
-        return Err(Error::WrongAddressType);
-    }
     Ok(())
 }
 ```
 
-### String Errors
+## Code Analysis Summary
 
-```rust
+**Main Type**: `Error` enum with 24 variants covering external libraries and domain-specific cases \
-// Static string errors (zero allocation)
+**Conversion Traits**: Implements `From` for 10+ external error types enabling automatic error propagation \
-Err(Error::Str("Invalid configuration"))
+**Error Handling**: Standard Rust error handling with `std::error::Error` trait implementation \
-
+**Dependencies**: Integrates errors from `bitcoin`, `bitcoincore-rpc`, `fjall`, `vecdb`, `jiff`, `minreq`, `serde_json`, and `zerocopy` crates \
-// Dynamic string errors
+**Architecture**: Centralized error aggregation pattern with automatic conversions and custom domain errors
-Err(Error::String(format!("Block {} not found", height)))
-```
-
-## Result Type
-
-The crate provides a convenient `Result` type alias:
-
-```rust
-pub type Result<T, E = Error> = std::result::Result<T, E>;
-```
-
-This allows for clean function signatures throughout BRK:
-
-```rust
-fn parse_block() -> Result<Block> { /* ... */ }
-fn index_transactions() -> Result<Vec<Transaction>> { /* ... */ }
-```
-
-## Error Display
-
-All errors implement `Display` and `std::error::Error`, providing:
-- Formatted error messages for debugging
-- Error chain support for nested errors
-- Integration with error handling libraries like `anyhow`
-
-## Dependencies
-
-- `vecdb` - Vector database error types
-- `bitcoincore-rpc` - Bitcoin Core RPC client errors
-- `fjall` - Key-value store errors
-- `jiff` - Date/time operation errors
-- `minreq` - HTTP request errors
-- `serde_json` - JSON serialization errors
-- `zerocopy` - Zero-copy conversion errors
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_error/src/lib.rs

@@ -17,7 +17,9 @@ pub enum Error {
     SeqDB(vecdb::SeqDBError),
     Minreq(minreq::Error),
     SystemTimeError(time::SystemTimeError),
-    SerdeJson(serde_json::Error),
+    BitcoinConsensusEncode(bitcoin::consensus::encode::Error),
+    BitcoinBip34Error(bitcoin::block::Bip34Error),
+    SonicRS(sonic_rs::Error),
     ZeroCopyError,
     Vecs(vecdb::Error),
 
@@ -29,15 +31,27 @@ pub enum Error {
     String(String),
 }
 
+impl From<bitcoin::block::Bip34Error> for Error {
+    fn from(value: bitcoin::block::Bip34Error) -> Self {
+        Self::BitcoinBip34Error(value)
+    }
+}
+
+impl From<bitcoin::consensus::encode::Error> for Error {
+    fn from(value: bitcoin::consensus::encode::Error) -> Self {
+        Self::BitcoinConsensusEncode(value)
+    }
+}
+
 impl From<time::SystemTimeError> for Error {
     fn from(value: time::SystemTimeError) -> Self {
         Self::SystemTimeError(value)
     }
 }
 
-impl From<serde_json::Error> for Error {
+impl From<sonic_rs::Error> for Error {
-    fn from(error: serde_json::Error) -> Self {
+    fn from(error: sonic_rs::Error) -> Self {
-        Self::SerdeJson(error)
+        Self::SonicRS(error)
     }
 }
 
@@ -83,6 +97,12 @@ impl From<fjall::Error> for Error {
     }
 }
 
+impl From<&'static str> for Error {
+    fn from(value: &'static str) -> Self {
+        Self::Str(value)
+    }
+}
+
 impl<A, B, C> From<zerocopy::error::ConvertError<A, B, C>> for Error {
     fn from(_: zerocopy::error::ConvertError<A, B, C>) -> Self {
         Self::ZeroCopyError
@@ -98,16 +118,18 @@ impl<A, B> From<zerocopy::error::SizeError<A, B>> for Error {
 impl fmt::Display for Error {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match self {
-            Error::IO(error) => Display::fmt(&error, f),
+            Error::BitcoinConsensusEncode(error) => Display::fmt(&error, f),
-            Error::Minreq(error) => Display::fmt(&error, f),
+            Error::BitcoinBip34Error(error) => Display::fmt(&error, f),
-            Error::SerdeJson(error) => Display::fmt(&error, f),
-            Error::VecDB(error) => Display::fmt(&error, f),
-            Error::SeqDB(error) => Display::fmt(&error, f),
-            Error::Vecs(error) => Display::fmt(&error, f),
             Error::BitcoinRPC(error) => Display::fmt(&error, f),
-            Error::SystemTimeError(error) => Display::fmt(&error, f),
-            Error::Jiff(error) => Display::fmt(&error, f),
             Error::Fjall(error) => Display::fmt(&error, f),
+            Error::IO(error) => Display::fmt(&error, f),
+            Error::Jiff(error) => Display::fmt(&error, f),
+            Error::Minreq(error) => Display::fmt(&error, f),
+            Error::SeqDB(error) => Display::fmt(&error, f),
+            Error::SonicRS(error) => Display::fmt(&error, f),
+            Error::SystemTimeError(error) => Display::fmt(&error, f),
+            Error::VecDB(error) => Display::fmt(&error, f),
+            Error::Vecs(error) => Display::fmt(&error, f),
             Error::ZeroCopyError => write!(f, "ZeroCopy error"),
 
             Error::WrongLength => write!(f, "Wrong length"),

crates/brk_fetcher/Cargo.toml

@@ -15,4 +15,4 @@ brk_logger = { workspace = true }
 brk_structs = { workspace = true }
 log = { workspace = true }
 minreq = { workspace = true }
-serde_json = { workspace = true }
+sonic-rs = { workspace = true }

crates/brk_fetcher/README.md

@@ -1,39 +1,36 @@
 # brk_fetcher
 
-**Bitcoin price data fetcher with multi-source fallback and retry logic**
+Multi-source Bitcoin price data aggregator with automatic fallback between exchanges.
 
-`brk_fetcher` provides reliable Bitcoin price data retrieval from multiple sources including Binance, Kraken, and BRK instances. It offers both date-based and block height-based price queries with automatic fallback and retry mechanisms for robust data collection.
+[![Crates.io](https://img.shields.io/crates/v/brk_fetcher.svg)](https://crates.io/crates/brk_fetcher)
+[![Documentation](https://docs.rs/brk_fetcher/badge.svg)](https://docs.rs/brk_fetcher)
 
-## What it provides
+## Overview
 
-- **Multi-source fallback**: Automatic fallback between Kraken → Binance → BRK
+This crate provides a unified interface for fetching Bitcoin price data from multiple sources including Binance, Kraken, and a custom BRK API. It implements automatic failover between data sources, retry mechanisms, and supports both real-time and historical price queries using blockchain height or date-based lookups.
-- **Flexible querying**: Fetch prices by date or block height with timestamps
-- **Retry logic**: Built-in retry mechanism with exponential backoff
-- **Multiple timeframes**: 1-minute and 1-day interval support
-- **HAR file import**: Import historical Binance chart data from browser
 
-## Key Features
+**Key Features:**
 
-### Data Sources
+- Multi-source price aggregation (Binance, Kraken, BRK API)
-- **Kraken API**: Primary source for OHLC data (1-day and 1-minute intervals)
+- Automatic fallback hierarchy with intelligent retry logic
-- **Binance API**: Secondary source with additional historical data
+- Historical price queries by blockchain height or date
-- **BRK instance**: Fallback source for previously cached price data
+- Support for both 1-minute and daily OHLC data
-- **HAR import**: Manual historical data import from browser sessions
+- HAR file import for extended historical data coverage
+- Built-in caching with BTreeMap storage for performance
 
-### Query Methods
+**Target Use Cases:**
-- **Date-based queries**: Get OHLC data for specific calendar dates
-- **Height-based queries**: Get OHLC data for specific block heights with timestamps
-- **Automatic aggregation**: Combines minute-level data for block intervals
 
-### Reliability Features
+- Bitcoin blockchain analyzers requiring accurate historical pricing
-- **Automatic fallback**: Tries sources in order until successful
+- Applications needing resilient price data with multiple fallbacks
-- **Retry mechanism**: Up to 12 hours of retries with 60-second intervals
+- Tools processing large datasets requiring efficient price lookups
-- **Cache clearing**: Automatic cache refresh on failures
-- **Error handling**: Graceful degradation with detailed error messages
 
-## Usage
+## Installation
 
-### Basic Setup
+```bash
+cargo add brk_fetcher
+```
+
+## Quick Start
 
 ```rust
 use brk_fetcher::Fetcher;
@@ -42,145 +39,132 @@ use brk_structs::{Date, Height, Timestamp};
 // Initialize fetcher with exchange APIs enabled
 let mut fetcher = Fetcher::import(true, None)?;
 
-// Initialize without exchange APIs (BRK-only mode)
+// Fetch price by date
-let mut fetcher = Fetcher::import(false, None)?;
+let date = Date::from_ymd(2023, 6, 15)?;
+let daily_price = fetcher.get_date(date)?;
 
-// Initialize with HAR file for historical Binance data
+// Fetch price by blockchain height
-let har_path = Path::new("./binance.har");
+let height = Height::new(800000);
-let mut fetcher = Fetcher::import(true, Some(har_path))?;
+let timestamp = Timestamp::from(1684771200u32);
+let block_price = fetcher.get_height(height, timestamp, None)?;
+
+println!("Daily OHLC: {:?}", daily_price);
+println!("Block OHLC: {:?}", block_price);
 ```
 
-### Date-based Price Queries
+## API Overview
+
+### Core Types
+
+- **`Fetcher`**: Main aggregator managing multiple price data sources
+- **`Binance`**: Binance exchange API client with HAR file support
+- **`Kraken`**: Kraken exchange API client for OHLC data
+- **`BRK`**: Custom API client for blockchain-indexed price data
+
+### Key Methods
+
+**`Fetcher::import(exchanges: bool, hars_path: Option<&Path>) -> Result<Self>`**
+Creates a new fetcher instance with configurable data sources.
+
+**`get_date(&mut self, date: Date) -> Result<OHLCCents>`**
+Retrieves daily OHLC data for the specified date with automatic source fallback.
+
+**`get_height(&mut self, height: Height, timestamp: Timestamp, previous_timestamp: Option<Timestamp>) -> Result<OHLCCents>`**
+Fetches price data for a specific blockchain height using minute-level precision.
+
+### Data Source Hierarchy
+
+1. **Kraken API** - Primary source for both 1-minute and daily data
+2. **Binance API** - Secondary source with extended HAR file support
+3. **BRK API** - Fallback source using blockchain-indexed pricing data
+
+### Error Handling
+
+The fetcher implements aggressive retry logic with exponential backoff, attempting each source up to 12 hours (720 retries) before failing. Failed requests trigger cache clearing and source rotation.
+
+## Examples
+
+### Basic Price Fetching
 
 ```rust
+use brk_fetcher::Fetcher;
 use brk_structs::Date;
 
-// Fetch OHLC data for a specific date
+let mut fetcher = Fetcher::import(true, None)?;
-let date = Date::new(2024, 12, 25);
-let ohlc = fetcher.get_date(date)?;
 
-println!("Bitcoin price on {}: ${:.2}", date, ohlc.close.dollars());
+// Fetch Bitcoin price for a specific date
-println!("Daily high: ${:.2}", ohlc.high.dollars());
+let date = Date::from_ymd(2021, 1, 1)?;
-println!("Daily low: ${:.2}", ohlc.low.dollars());
-```
-
-### Block Height-based Price Queries
-
-```rust
-use brk_structs::{Height, Timestamp};
-
-// Fetch price at specific block height
-let height = Height::new(900_000);
-let timestamp = Timestamp::from_block_height(height);
-let previous_timestamp = Some(Timestamp::from_block_height(Height::new(899_999)));
-
-let ohlc = fetcher.get_height(height, timestamp, previous_timestamp)?;
-println!("Bitcoin price at block {}: ${:.2}", height, ohlc.close.dollars());
-```
-
-### Working with OHLC Data
-
-```rust
-use brk_structs::OHLCCents;
-
-// OHLC data is returned in cents for precision
-let ohlc: OHLCCents = fetcher.get_date(date)?;
-
-// Convert to dollars for display
-println!("Open: ${:.2}", ohlc.open.dollars());
-println!("High: ${:.2}", ohlc.high.dollars());
-println!("Low: ${:.2}", ohlc.low.dollars());
-println!("Close: ${:.2}", ohlc.close.dollars());
-
-// Access raw cent values
-println!("Close in cents: {}", ohlc.close.0);
-```
-
-### Using Individual Sources
-
-```rust
-use brk_fetcher::{Binance, Kraken, BRK};
-
-// Use specific exchanges directly
-let binance = Binance::init(None);
-let kraken = Kraken::default();
-let brk = BRK::default();
-
-// Fetch from specific source
-let binance_data = binance.get_from_1d(&date)?;
-let kraken_data = kraken.get_from_1mn(timestamp, previous_timestamp)?;
-let brk_data = brk.get_from_height(height)?;
-```
-
-### Error Handling and Retries
-
-```rust
-// The fetcher automatically retries on failures
 match fetcher.get_date(date) {
-    Ok(ohlc) => println!("Successfully fetched: ${:.2}", ohlc.close.dollars()),
+    Ok(ohlc) => println!("BTC price on {}: ${}", date, ohlc.close.to_dollars()),
-    Err(e) => {
+    Err(e) => eprintln!("Failed to fetch price: {}", e),
-        // After all retries and sources exhausted
+}
-        eprintln!("Failed to fetch price data: {}", e);
+```
+
+### Historical Data with HAR Files
+
+```rust
+use brk_fetcher::Fetcher;
+use std::path::Path;
+
+// Initialize with HAR file path for extended historical coverage
+let har_path = Path::new("./import_data");
+let mut fetcher = Fetcher::import(true, Some(har_path))?;
+
+// Fetch minute-level data using HAR file fallback
+let height = Height::new(650000);
+let timestamp = Timestamp::from(1598918400u32); // August 2020
+let price_data = fetcher.get_height(height, timestamp, None)?;
+```
+
+### Batch Processing with Caching
+
+```rust
+use brk_fetcher::Fetcher;
+
+let mut fetcher = Fetcher::import(true, None)?;
+
+// Process multiple heights - caching improves performance
+for height in 800000..800100 {
+    let timestamp = Timestamp::from(1684771200u32 + (height - 800000) * 600);
+
+    match fetcher.get_height(Height::new(height), timestamp, None) {
+        Ok(ohlc) => println!("Height {}: ${:.2}", height, ohlc.close.to_dollars()),
+        Err(e) => eprintln!("Error at height {}: {}", height, e),
     }
 }
 
-// Clear cache to force fresh data
+// Clear caches when done
 fetcher.clear();
 ```
 
-## Data Sources and Limitations
+## Architecture
 
-### Kraken API
+### Retry Mechanism
-- **1-day data**: Historical daily OHLC data
-- **1-minute data**: Limited to last ~10 hours
-- **Rate limits**: Subject to Kraken API restrictions
 
-### Binance API  
+The crate implements a sophisticated retry system with:
-- **1-day data**: Historical daily OHLC data
-- **1-minute data**: Limited to last ~16 hours
-- **HAR import**: Can extend historical coverage via browser data
 
-### BRK Instance
+- **Default retry count**: 6 attempts with 5-second delays
-- **Cached data**: Previously fetched price data
+- **Extended retry**: Up to 720 attempts (12 hours) for critical operations
-- **Offline capability**: Works without internet when data is cached
+- **Cache invalidation**: Automatic cache clearing between retry attempts
-- **Height-based**: Optimized for block height queries
+- **Exponential backoff**: 60-second delays for extended retries
 
-## HAR File Import
+### Data Aggregation
 
-For historical data beyond API limits:
+Price data is aggregated using OHLC (Open, High, Low, Close) calculations spanning timestamp ranges. The `find_height_ohlc` function computes accurate OHLC values by scanning time series data between block timestamps.
 
-1. Visit [Binance BTC/USDT chart](https://www.binance.com/en/trade/BTC_USDT?type=spot)
+### HAR File Processing
-2. Set chart to 1-minute interval
-3. Open browser dev tools, go to Network tab
-4. Filter by 'uiKlines'
-5. Scroll chart to desired historical period
-6. Export network requests as HAR file
-7. Initialize fetcher with HAR path
 
-## Fallback Strategy
+Binance integration supports HTTP Archive (HAR) files for extended historical data coverage, parsing browser network captures to extract additional pricing data beyond API limitations.
 
-The fetcher tries sources in this order:
+## Code Analysis Summary
-1. **Kraken** - Primary source for most queries
-2. **Binance** - Secondary source with extended coverage
-3. **BRK** - Fallback for cached/computed prices
 
-If all sources fail, it retries up to 12 hours with 60-second intervals.
+**Main Types**: `Fetcher` aggregator with `Binance`, `Kraken`, and `BRK` source implementations \
-
+**Caching**: BTreeMap-based caching for both timestamp and date-indexed price data \
-## Performance and Reliability
+**Network Layer**: Built on `minreq` HTTP client with automatic JSON parsing \
-
+**Error Handling**: Comprehensive retry logic with source rotation and cache management \
-- **Automatic retries**: Up to 720 attempts (12 hours) with 60-second delays
+**Dependencies**: Integrates `brk_structs` for type definitions and `brk_error` for unified error handling \
-- **Cache management**: Clears cache on failures to force fresh data
+**Architecture**: Multi-source aggregation pattern with hierarchical fallback and intelligent caching
-- **Error logging**: Detailed failure reporting with recovery instructions
-- **Graceful degradation**: Falls back through sources until successful
-
-## Dependencies
-
-- `brk_structs` - Bitcoin-aware type system (Date, Height, OHLC types)
-- `brk_error` - Unified error handling
-- `minreq` - HTTP client for API requests
-- `serde_json` - JSON parsing for API responses
-- `log` - Logging for retry and error reporting
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_fetcher/src/binance.rs

@@ -3,13 +3,12 @@ use std::{
     fs::{self, File},
     io::BufReader,
     path::{Path, PathBuf},
-    str::FromStr,
 };
 
 use brk_error::{Error, Result};
 use brk_structs::{Cents, OHLCCents, Timestamp};
 use log::info;
-use serde_json::Value;
+use sonic_rs::{JsonContainerTrait, JsonValueTrait, Value};
 
 use crate::{Close, Date, Dollars, Fetcher, High, Low, Open, default_retry};
 
@@ -17,7 +16,7 @@ use crate::{Close, Date, Dollars, Fetcher, High, Low, Open, default_retry};
 pub struct Binance {
     path: Option<PathBuf>,
     _1mn: Option<BTreeMap<Timestamp, OHLCCents>>,
-    pub _1d: Option<BTreeMap<Date, OHLCCents>>,
+    _1d: Option<BTreeMap<Date, OHLCCents>>,
     har: Option<BTreeMap<Timestamp, OHLCCents>>,
 }
 
@@ -69,11 +68,11 @@ impl Binance {
         info!("Fetching 1mn prices from Binance...");
 
         default_retry(|_| {
-            Self::json_to_timestamp_to_ohlc(
+            Self::json_to_timestamp_to_ohlc(&sonic_rs::from_str(
-                &minreq::get(Self::url("interval=1m&limit=1000"))
+                minreq::get(Self::url("interval=1m&limit=1000"))
                     .send()?
-                    .json()?,
+                    .as_str()?,
-            )
+            )?)
         })
     }
 
@@ -94,7 +93,9 @@ impl Binance {
         info!("Fetching daily prices from Binance...");
 
         default_retry(|_| {
-            Self::json_to_date_to_ohlc(&minreq::get(Self::url("interval=1d")).send()?.json()?)
+            Self::json_to_date_to_ohlc(&sonic_rs::from_str(
+                minreq::get(Self::url("interval=1d")).send()?.as_str()?,
+            )?)
         })
     }
 
@@ -119,7 +120,7 @@ impl Binance {
 
         let reader = BufReader::new(file);
 
-        let json: BTreeMap<String, Value> = if let Ok(json) = serde_json::from_reader(reader) {
+        let json: BTreeMap<String, Value> = if let Ok(json) = sonic_rs::from_reader(reader) {
             json
         } else {
             return Ok(Default::default());
@@ -129,7 +130,7 @@ impl Binance {
             .ok_or(Error::Str("Expect object to have log attribute"))?
             .as_object()
             .ok_or(Error::Str("Expect to be an object"))?
-            .get("entries")
+            .get(&"entries")
             .ok_or(Error::Str("Expect object to have entries"))?
             .as_array()
             .ok_or(Error::Str("Expect to be an array"))?
@@ -138,11 +139,11 @@ impl Binance {
                 entry
                     .as_object()
                     .unwrap()
-                    .get("request")
+                    .get(&"request")
                     .unwrap()
                     .as_object()
                     .unwrap()
-                    .get("url")
+                    .get(&"url")
                     .unwrap()
                     .as_str()
                     .unwrap()
@@ -152,14 +153,14 @@ impl Binance {
                 let response = entry
                     .as_object()
                     .unwrap()
-                    .get("response")
+                    .get(&"response")
                     .unwrap()
                     .as_object()
                     .unwrap();
 
-                let content = response.get("content").unwrap().as_object().unwrap();
+                let content = response.get(&"content").unwrap().as_object().unwrap();
 
-                let text = content.get("text");
+                let text = content.get(&"text");
 
                 if text.is_none() {
                     return Ok(BTreeMap::new());
@@ -167,7 +168,7 @@ impl Binance {
 
                 let text = text.unwrap().as_str().unwrap();
 
-                Self::json_to_timestamp_to_ohlc(&serde_json::Value::from_str(text).unwrap())
+                Self::json_to_timestamp_to_ohlc(&sonic_rs::from_str(text).unwrap())
             })
             .try_fold(BTreeMap::default(), |mut all, res| {
                 all.append(&mut res?);

crates/brk_fetcher/src/brk.rs

@@ -3,7 +3,7 @@ use std::collections::BTreeMap;
 use brk_error::{Error, Result};
 use brk_structs::{Cents, CheckedSub, Date, DateIndex, Height, OHLCCents};
 use log::info;
-use serde_json::Value;
+use sonic_rs::{JsonContainerTrait, JsonValueTrait, Value};
 
 use crate::{Close, Dollars, High, Low, Open, default_retry};
 
@@ -14,7 +14,7 @@ pub struct BRK {
     dateindex_to_ohlc: BTreeMap<DateIndex, Vec<OHLCCents>>,
 }
 
-const API_URL: &str = "https://bitcoinresearchkit.org/api/vecs";
+const API_URL: &str = "https://bitview.space/api/vecs";
 const CHUNK_SIZE: usize = 10_000;
 
 impl BRK {
@@ -46,12 +46,12 @@ impl BRK {
 
         default_retry(|_| {
             let url = format!(
-                "{API_URL}/height-to-ohlc?from={}&to={}",
+                "{API_URL}/height-to-price-ohlc?from={}&to={}",
                 height,
                 height + CHUNK_SIZE
             );
 
-            let body: Value = minreq::get(url).send()?.json()?;
+            let body: Value = sonic_rs::from_str(minreq::get(url).send()?.as_str()?)?;
 
             body.as_array()
                 .ok_or(Error::Str("Expect to be an array"))?
@@ -91,12 +91,12 @@ impl BRK {
 
         default_retry(|_| {
             let url = format!(
-                "{API_URL}/dateindex-to-ohlc?from={}&to={}",
+                "{API_URL}/dateindex-to-price-ohlc?from={}&to={}",
                 dateindex,
                 dateindex + CHUNK_SIZE
             );
 
-            let body: Value = minreq::get(url).send()?.json()?;
+            let body: Value = sonic_rs::from_str(minreq::get(url).send()?.as_str()?)?;
 
             body.as_array()
                 .ok_or(Error::Str("Expect to be an array"))?

crates/brk_fetcher/src/kraken.rs

@@ -3,7 +3,7 @@ use std::collections::BTreeMap;
 use brk_error::{Error, Result};
 use brk_structs::{Cents, Close, Date, Dollars, High, Low, OHLCCents, Open, Timestamp};
 use log::info;
-use serde_json::Value;
+use sonic_rs::{JsonContainerTrait, JsonValueTrait, Value};
 
 use crate::{Fetcher, default_retry};
 
@@ -36,7 +36,9 @@ impl Kraken {
         info!("Fetching 1mn prices from Kraken...");
 
         default_retry(|_| {
-            Self::json_to_timestamp_to_ohlc(&minreq::get(Self::url(1)).send()?.json()?)
+            Self::json_to_timestamp_to_ohlc(&sonic_rs::from_str(
+                minreq::get(Self::url(1)).send()?.as_str()?,
+            )?)
         })
     }
 
@@ -55,7 +57,11 @@ impl Kraken {
     pub fn fetch_1d() -> Result<BTreeMap<Date, OHLCCents>> {
         info!("Fetching daily prices from Kraken...");
 
-        default_retry(|_| Self::json_to_date_to_ohlc(&minreq::get(Self::url(1440)).send()?.json()?))
+        default_retry(|_| {
+            Self::json_to_date_to_ohlc(&sonic_rs::from_str(
+                minreq::get(Self::url(1440)).send()?.as_str()?,
+            )?)
+        })
     }
 
     fn json_to_timestamp_to_ohlc(json: &Value) -> Result<BTreeMap<Timestamp, OHLCCents>> {
@@ -73,11 +79,11 @@ impl Kraken {
     {
         json.as_object()
             .ok_or(Error::Str("Expect to be an object"))?
-            .get("result")
+            .get(&"result")
             .ok_or(Error::Str("Expect object to have result"))?
             .as_object()
             .ok_or(Error::Str("Expect to be an object"))?
-            .get("XXBTZUSD")
+            .get(&"XXBTZUSD")
             .ok_or(Error::Str("Expect to have XXBTZUSD"))?
             .as_array()
             .ok_or(Error::Str("Expect to be an array"))?

crates/brk_fetcher/src/retry.rs

@@ -22,6 +22,7 @@ where
         if i == retries || res.is_ok() {
             return res;
         } else {
+            let _ = dbg!(res);
             info!("Failed, waiting {sleep_in_s} seconds...");
             sleep(Duration::from_secs(sleep_in_s));
         }

crates/brk_indexer/README.md

@@ -1,190 +1,277 @@
 # brk_indexer
 
-**High-performance Bitcoin blockchain indexer with dual storage architecture**
+High-performance Bitcoin blockchain indexer with parallel processing and dual storage architecture.
 
-`brk_indexer` processes raw Bitcoin Core block data and creates efficient storage structures using both vectors (time-series) and key-value stores (lookups). It serves as the foundation of BRK's data pipeline, organizing all blockchain data into optimized formats for fast retrieval and analysis.
+[![Crates.io](https://img.shields.io/crates/v/brk_indexer.svg)](https://crates.io/crates/brk_indexer)
+[![Documentation](https://docs.rs/brk_indexer/badge.svg)](https://docs.rs/brk_indexer)
 
-## What it provides
+## Overview
 
-- **Dual Storage Architecture**: Vectors for time-series data, key-value stores for lookups
+This crate provides a comprehensive Bitcoin blockchain indexer built on top of `brk_parser`. It processes raw Bitcoin blocks in parallel, extracting and indexing transactions, addresses, inputs, outputs, and metadata into optimized storage structures. The indexer maintains two complementary storage systems: columnar vectors for analytics and key-value stores for fast lookups.
-- **Memory Efficiency**: ~5-6GB peak RAM usage during full blockchain indexing
-- **Incremental Processing**: Resume from last indexed height with rollback protection
-- **Data Integrity**: Collision detection and validation during indexing
-- **All Bitcoin Data Types**: Complete support for blocks, transactions, inputs, outputs, and addresses
 
-## Key Features
+**Key Features:**
 
-### Storage Strategy
+- Parallel block processing with multi-threaded transaction analysis
+- Dual storage architecture: columnar vectors + key-value stores
+- Address type classification and indexing for all Bitcoin script types
+- Collision detection and validation for address hashes and transaction IDs
+- Incremental processing with automatic rollback and recovery
+- Height-based synchronization with Bitcoin Core RPC validation
+- Optimized batch operations with configurable snapshot intervals
 
-**Vector Storage (time-series data):**
+**Target Use Cases:**
-- Block metadata (height, timestamp, hash, difficulty, size)
-- Transaction data (version, locktime, RBF flag, indices)
-- Input/Output mappings and values
-- Address bytes for all output types
-- Efficient for range queries and analytics
 
-**Key-Value Storage (lookups):**
+- Bitcoin blockchain analysis requiring full transaction history
-- Block hash prefixes → heights
+- Address clustering and UTXO set analysis
-- Transaction ID prefixes → transaction indices  
+- Blockchain explorers needing fast address/transaction lookups
-- Address byte hashes → type indices
+- Research applications requiring structured access to blockchain data
-- Fast point queries by hash or address
 
-### Performance Features
+## Installation
-- **Parallel Processing**: Concurrent transaction and output processing using Rayon
-- **Batch Operations**: Periodic commits every 1,000 blocks for optimal I/O
-- **Memory Efficiency**: Optimized data structures minimize RAM usage
-- **Incremental Updates**: Handles blockchain reorganizations automatically
 
-### Address Type Support
+```bash
-Complete support for all Bitcoin address types:
+cargo add brk_indexer
-- P2PK (65-byte and 33-byte), P2PKH, P2SH
+```
-- P2WPKH, P2WSH, P2TR, P2A
-- P2MS (multisig), OpReturn, Empty, Unknown
 
-## Usage
+## Quick Start
-
-### Basic Indexing
 
 ```rust
 use brk_indexer::Indexer;
 use brk_parser::Parser;
-use bitcoincore_rpc::{Auth, Client};
+use bitcoincore_rpc::{Client, Auth};
 use vecdb::Exit;
+use std::path::Path;
 
-// Setup Bitcoin Core RPC connection
+// Initialize Bitcoin Core RPC client
-let rpc = Box::leak(Box::new(Client::new(
+let rpc = Client::new("http://localhost:8332", Auth::None)?;
-    "http://localhost:8332",
+let rpc = Box::leak(Box::new(rpc));
-    Auth::CookieFile(Path::new("~/.bitcoin/.cookie")),
-)?));
 
-// Create parser for Bitcoin Core block files
+// Create parser for raw block data
-let parser = Parser::new(
+let blocks_dir = Path::new("/path/to/bitcoin/blocks");
-    Path::new("~/.bitcoin/blocks").to_path_buf(),
+let parser = Parser::new(blocks_dir, None, rpc);
-    Path::new("./brk_data").to_path_buf(),
-    rpc
-);
 
-// Create indexer with forced import (resets if needed)
+// Initialize indexer with output directory
-let mut indexer = Indexer::forced_import(Path::new("./brk_data"))?;
+let outputs_dir = Path::new("./indexed_data");
+let mut indexer = Indexer::forced_import(outputs_dir)?;
 
-// Setup graceful shutdown handler
+// Index blockchain data
-let exit = Exit::new();
+let exit = Exit::default();
-exit.set_ctrlc_handler();
+let starting_indexes = indexer.index(&parser, rpc, &exit, true)?;
 
-// Index the blockchain
+println!("Indexed up to height: {}", starting_indexes.height);
-let indexes = indexer.index(&parser, rpc, &exit, true)?;
-println!("Indexed up to height: {}", indexes.height);
 ```
 
-### Continuous Indexing
+## API Overview
+
+### Core Types
+
+- **`Indexer`**: Main coordinator managing vectors and stores
+- **`Vecs`**: Columnar storage for blockchain data analytics
+- **`Stores`**: Key-value storage for fast hash-based lookups
+- **`Indexes`**: Current indexing state tracking progress across data types
+
+### Key Methods
+
+**`Indexer::forced_import(outputs_dir: &Path) -> Result<Self>`**
+Creates or opens indexer instance with automatic version management.
+
+**`index(&mut self, parser: &Parser, rpc: &'static Client, exit: &Exit, check_collisions: bool) -> Result<Indexes>`**
+Main indexing function processing blocks from parser with collision detection.
+
+### Storage Architecture
+
+**Columnar Vectors (Vecs):**
+
+- `height_to_*`: Block-level data (hash, timestamp, difficulty, size, weight)
+- `txindex_to_*`: Transaction data (ID, version, locktime, size, RBF flag)
+- `outputindex_to_*`: Output data (value, type, address mapping)
+- `inputindex_to_outputindex`: Input-to-output relationship mapping
+
+**Key-Value Stores:**
+
+- `addressbyteshash_to_typeindex`: Address hash to internal index mapping
+- `blockhashprefix_to_height`: Block hash prefix to height lookup
+- `txidprefix_to_txindex`: Transaction ID prefix to internal index
+- `addresstype_to_typeindex_with_outputindex`: Address type to output mappings
+
+### Address Type Support
+
+Complete coverage of Bitcoin script types:
+
+- **P2PK**: Pay-to-Public-Key (33-byte and 65-byte variants)
+- **P2PKH**: Pay-to-Public-Key-Hash
+- **P2SH**: Pay-to-Script-Hash
+- **P2WPKH**: Pay-to-Witness-Public-Key-Hash
+- **P2WSH**: Pay-to-Witness-Script-Hash
+- **P2TR**: Pay-to-Taproot
+- **P2MS**: Pay-to-Multisig
+- **P2A**: Pay-to-Address (custom type)
+- **OpReturn**: OP_RETURN data outputs
+- **Empty/Unknown**: Non-standard script types
+
+## Examples
+
+### Basic Indexing Operation
 
 ```rust
-use std::time::{Duration, Instant};
+use brk_indexer::Indexer;
-use std::thread::sleep;
+use brk_parser::Parser;
+use std::path::Path;
 
-// Continuous indexing loop for real-time updates
+// Initialize components
-loop {
+let outputs_dir = Path::new("./blockchain_index");
-    let start_time = Instant::now();
+let mut indexer = Indexer::forced_import(outputs_dir)?;
 
-    // Index new blocks
+let blocks_dir = Path::new("/Users/satoshi/.bitcoin/blocks");
-    let indexes = indexer.index(&parser, rpc, &exit, true)?;
+let parser = Parser::new(blocks_dir, None, rpc);
 
-    println!("Indexed to height {} in {:?}", 
+// Index with collision checking enabled
-             indexes.height, start_time.elapsed());
+let exit = vecdb::Exit::default();
+let final_indexes = indexer.index(&parser, rpc, &exit, true)?;
 
-    // Check for exit signal
+println!("Final height: {}", final_indexes.height);
-    if exit.is_signaled() {
+println!("Total transactions: {}", final_indexes.txindex);
-        println!("Graceful shutdown requested");
+println!("Total addresses: {}", final_indexes.total_address_count());
-        break;
+```
-    }
 
-    // Wait before next update cycle
+### Querying Indexed Data
-    sleep(Duration::from_secs(5 * 60));
+
+```rust
+use brk_indexer::Indexer;
+use brk_structs::{Height, TxidPrefix, AddressBytesHash};
+
+let indexer = Indexer::forced_import("./blockchain_index")?;
+
+// Look up block hash by height
+let height = Height::new(750000);
+if let Some(block_hash) = indexer.vecs.height_to_blockhash.get(height)? {
+    println!("Block 750000 hash: {}", block_hash);
+}
+
+// Look up transaction by ID prefix
+let txid_prefix = TxidPrefix::from_str("abcdef123456")?;
+if let Some(tx_index) = indexer.stores.txidprefix_to_txindex.get(&txid_prefix)? {
+    println!("Transaction index: {}", tx_index);
+}
+
+// Query address information
+let address_hash = AddressBytesHash::from(/* address bytes */);
+if let Some(type_index) = indexer.stores.addressbyteshash_to_typeindex.get(&address_hash)? {
+    println!("Address type index: {}", type_index);
 }
 ```
 
-### Accessing Indexed Data
+### Incremental Processing
 
 ```rust
-// Access the underlying storage structures
+use brk_indexer::Indexer;
-let vecs = &indexer.vecs;
-let stores = &indexer.stores;
 
-// Get block hash at specific height
+// Indexer automatically resumes from last processed height
-let block_hash = vecs.height_to_blockhash.get(Height::new(800_000))?;
+let mut indexer = Indexer::forced_import("./blockchain_index")?;
 
-// Look up transaction by prefix
+let current_indexes = indexer.vecs.current_indexes(&indexer.stores, rpc)?;
-let tx_prefix = TxidPrefix::from(&txid);
+println!("Resuming from height: {}", current_indexes.height);
-let tx_index = stores.txidprefix_to_txindex.get(&tx_prefix)?;
 
-// Get address data
+// Process new blocks incrementally
-let address_hash = AddressBytesHash::from(&address_bytes);
+let exit = vecdb::Exit::default();
-let type_index = stores.addressbyteshash_to_anyaddressindex.get(&address_hash)?;
+let updated_indexes = indexer.index(&parser, rpc, &exit, true)?;
+
+println!("Processed {} new blocks",
+         updated_indexes.height.as_u32() - current_indexes.height.as_u32());
 ```
 
+### Address Type Analysis
+
+```rust
+use brk_indexer::Indexer;
+use brk_structs::OutputType;
+
+let indexer = Indexer::forced_import("./blockchain_index")?;
+
+// Analyze address distribution by type
+for output_type in OutputType::as_vec() {
+    let count = indexer.vecs.outputindex_to_outputtype
+        .iter()
+        .filter(|&ot| ot == output_type)
+        .count();
+
+    println!("{:?}: {} outputs", output_type, count);
+}
+
+// Query specific address type data
+let p2pkh_store = &indexer.stores.addresstype_to_typeindex_with_outputindex
+    .p2pkh;
+
+println!("P2PKH addresses: {}", p2pkh_store.len());
+```
+
+## Architecture
+
+### Parallel Processing
+
+The indexer uses sophisticated parallel processing:
+
+- **Block-Level Parallelism**: Concurrent processing of transactions within blocks
+- **Transaction Analysis**: Parallel input/output processing with `rayon`
+- **Address Resolution**: Multi-threaded address type classification and indexing
+- **Collision Detection**: Parallel validation of hash collisions across address types
+
+### Storage Optimization
+
+**Columnar Storage (vecdb):**
+
+- Compressed vectors for space-efficient analytics queries
+- Raw vectors for frequently accessed data (heights, hashes)
+- Page-aligned storage for memory mapping efficiency
+
+**Key-Value Storage (Fjall):**
+
+- LSM-tree architecture for write-heavy indexing workloads
+- Bloom filters for fast negative lookups
+- Transactional consistency with rollback support
+
+### Memory Management
+
+- **Batch Processing**: 1000-block snapshots to balance memory and I/O
+- **Reader Management**: Static readers for consistent data access during processing
+- **Collision Tracking**: BTreeMap-based collision detection with memory cleanup
+- **Exit Handling**: Graceful shutdown with consistent state preservation
+
+### Version Management
+
+- **Schema Versioning**: Automatic migration on version changes (currently v21)
+- **Rollback Support**: Automatic recovery from incomplete processing
+- **State Tracking**: Height-based synchronization across all storage components
+
 ## Performance Characteristics
 
-**Benchmarked on MacBook Pro M3 Pro (36GB RAM):**
+### Processing Speed
-- **Full blockchain sync** (to ~892k blocks): 7-8 hours
-- **Peak memory usage**: 5-6GB
-- **Storage overhead**: ~27% of Bitcoin Core block size
-- **Incremental updates**: Very fast, efficient resume from last height
 
-## Data Organization
+- **Parallel Transaction Processing**: Multi-core utilization for CPU-intensive operations
+- **Optimized I/O**: Batch operations reduce disk overhead
+- **Memory Efficiency**: Streaming processing without loading entire blockchain
 
-The indexer creates this storage structure:
+### Storage Requirements
-```
-brk_data/
-├── indexed/
-│   ├── vecs/              # Vector storage
-│   │   ├── height_to_*    # Height-indexed data
-│   │   ├── txindex_to_*   # Transaction-indexed data
-│   │   └── outputindex_to_* # Output-indexed data
-│   └── stores/            # Key-value stores
-│       ├── hash_lookups/  # Block/TX hash mappings
-│       └── address_maps/  # Address type mappings
-└── metadata/              # Versioning and state
-```
 
-## Indexes Tracking
+- **Columnar Compression**: Significant space savings for repetitive blockchain data
+- **Index Optimization**: Bloom filters reduce lookup overhead
+- **Incremental Growth**: Storage scales linearly with blockchain size
 
-The indexer maintains current indices during processing:
+### Scalability
 
-```rust
+- **Height-Based Partitioning**: Enables distributed processing strategies
-pub struct Indexes {
+- **Modular Architecture**: Separate vector and store systems for flexible deployment
-    pub height: Height,                      // Current block height
+- **Resource Configuration**: Configurable batch sizes and memory limits
-    pub txindex: TxIndex,                    // Current transaction index
-    pub inputindex: InputIndex,              // Current input index
-    pub outputindex: OutputIndex,            // Current output index
-    pub p2pkhaddressindex: P2PKHAddressIndex, // P2PKH address index
-    // ... indices for all address types
-}
-```
 
-## Requirements
+## Code Analysis Summary
 
-- **Bitcoin Core node** with RPC enabled
+**Main Structure**: `Indexer` coordinating `Vecs` (columnar analytics) and `Stores` (key-value lookups) \
-- **Block file access** to `~/.bitcoin/blocks/`
+**Processing Pipeline**: Multi-threaded block analysis with parallel transaction/address processing \
-- **Storage space**: Minimum 500GB (scales with blockchain growth)
+**Storage Architecture**: Dual system using vecdb for analytics and Fjall for lookups \
-- **Memory**: 8GB+ RAM recommended
+**Address Indexing**: Complete Bitcoin script type coverage with collision detection \
-- **CPU**: Multi-core recommended for parallel processing
+**Synchronization**: Height-based coordination with Bitcoin Core RPC validation \
-
+**Parallel Processing**: rayon-based parallelism for transaction analysis and address resolution \
-## Rollback and Recovery
+**Architecture**: High-performance blockchain indexer with ACID guarantees and incremental processing
-
-- **Automatic rollback** on interruption or blockchain reorgs
-- **State persistence** for efficient restart
-- **Version management** for storage format compatibility
-- **Graceful shutdown** with Ctrl+C handling
-
-## Dependencies
-
-- `brk_parser` - Bitcoin block parsing and sequential access
-- `brk_store` - Key-value storage wrapper (fjall-based)
-- `vecdb` - Vector database for time-series storage
-- `bitcoin` - Bitcoin protocol types and parsing
-- `rayon` - Parallel processing framework
-- `bitcoincore_rpc` - Bitcoin Core RPC client
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_indexer/examples/indexer.rs

@@ -21,8 +21,8 @@ fn main() -> Result<()> {
 
     let blocks_dir = bitcoin_dir.join("blocks");
 
-    let outputs_dir = Path::new("../../_outputs");
+    let outputs_dir = Path::new(&std::env::var("HOME").unwrap()).join(".brk");
-    fs::create_dir_all(outputs_dir)?;
+    fs::create_dir_all(&outputs_dir)?;
     // let outputs_dir = Path::new("/Volumes/WD_BLACK1/brk");
 
     let rpc = Box::leak(Box::new(bitcoincore_rpc::Client::new(
@@ -33,11 +33,11 @@ fn main() -> Result<()> {
     let exit = Exit::new();
     exit.set_ctrlc_handler();
 
-    let parser = Parser::new(blocks_dir, outputs_dir.to_path_buf(), rpc);
+    let parser = Parser::new(blocks_dir, rpc);
 
-    fs::create_dir_all(outputs_dir)?;
+    fs::create_dir_all(&outputs_dir)?;
 
-    let mut indexer = Indexer::forced_import(outputs_dir)?;
+    let mut indexer = Indexer::forced_import(&outputs_dir)?;
 
     loop {
         let i = Instant::now();

crates/brk_indexer/src/indexes.rs

@@ -1,6 +1,5 @@
 use bitcoincore_rpc::Client;
 use brk_error::{Error, Result};
-use brk_parser::NUMBER_OF_UNSAFE_BLOCKS;
 use brk_structs::{
     BlockHash, CheckedSub, EmptyOutputIndex, Height, InputIndex, OpReturnIndex, OutputIndex,
     OutputType, P2AAddressIndex, P2MSOutputIndex, P2PK33AddressIndex, P2PK65AddressIndex,
@@ -13,6 +12,8 @@ use vecdb::{
 
 use crate::{Stores, Vecs};
 
+const NUMBER_OF_UNSAFE_BLOCKS: usize = 100;
+
 #[derive(Debug, Default, Clone)]
 pub struct Indexes {
     pub emptyoutputindex: EmptyOutputIndex,
@@ -34,7 +35,7 @@ pub struct Indexes {
 }
 
 impl Indexes {
-    pub fn typeindex(&self, outputtype: OutputType) -> TypeIndex {
+    pub fn to_typeindex(&self, outputtype: OutputType) -> TypeIndex {
         match outputtype {
             OutputType::Empty => *self.emptyoutputindex,
             OutputType::OpReturn => *self.opreturnindex,

crates/brk_indexer/src/lib.rs

@@ -1,20 +1,19 @@
 #![doc = include_str!("../README.md")]
 
-use std::{collections::BTreeMap, path::Path, str::FromStr, thread, time::Instant};
+use std::{collections::BTreeMap, path::Path, str::FromStr, time::Instant};
 
 use bitcoin::{Transaction, TxIn, TxOut};
 use brk_error::{Error, Result};
-
 use brk_parser::Parser;
 use brk_store::AnyStore;
 use brk_structs::{
-    AddressBytes, AddressBytesHash, BlockHash, BlockHashPrefix, Height, InputIndex, OutputIndex,
+    AddressBytes, AddressBytesHash, BlockHashPrefix, Height, InputIndex, OutputIndex, OutputType,
-    OutputType, Sats, StoredBool, Timestamp, TxIndex, Txid, TxidPrefix, TypeIndex,
+    Sats, StoredBool, Timestamp, TxIndex, Txid, TxidPrefix, TypeIndex, TypeIndexWithOutputindex,
-    TypeIndexWithOutputindex, Unit, Version, Vin, Vout,
+    Unit, Version, Vin, Vout,
 };
 use log::{error, info};
 use rayon::prelude::*;
-use vecdb::{AnyVec, Database, Exit, GenericStoredVec, PAGE_SIZE, Reader, VecIterator};
+use vecdb::{AnyVec, Exit, GenericStoredVec, Reader, VecIterator};
 mod indexes;
 mod stores;
 mod vecs;
@@ -23,33 +22,31 @@ pub use indexes::*;
 pub use stores::*;
 pub use vecs::*;
 
+// One version for all data sources
+// Increment on **change _OR_ addition**
+const VERSION: Version = Version::new(21);
 const SNAPSHOT_BLOCK_RANGE: usize = 1_000;
 const COLLISIONS_CHECKED_UP_TO: Height = Height::new(909_150);
-const VERSION: Version = Version::ONE;
 
 #[derive(Clone)]
 pub struct Indexer {
-    pub db: Database,
     pub vecs: Vecs,
     pub stores: Stores,
 }
 
 impl Indexer {
     pub fn forced_import(outputs_dir: &Path) -> Result<Self> {
-        let db = Database::open(&outputs_dir.join("indexed/vecs"))?;
+        info!("Importing indexer...");
 
-        let vecs = Vecs::forced_import(&db, VERSION + Version::ZERO)?;
+        let path = outputs_dir.join("indexed");
 
-        db.set_min_len(PAGE_SIZE * 50_000_000)?;
+        let vecs = Vecs::forced_import(&path, VERSION)?;
+        info!("Imported vecs");
 
-        Ok(Self {
+        let stores = Stores::forced_import(&path, VERSION)?;
-            vecs,
+        info!("Imported stores");
-            stores: Stores::forced_import(
+
-                &outputs_dir.join("indexed/stores"),
+        Ok(Self { vecs, stores })
-                VERSION + Version::ZERO,
-            )?,
-            db,
-        })
     }
 
     pub fn index(
@@ -59,16 +56,9 @@ impl Indexer {
         exit: &Exit,
         check_collisions: bool,
     ) -> Result<Indexes> {
-        let db = self.db.clone();
-
-        // dbg!(self.db.regions().id_to_index());
-        // dbg!(self.db.layout());
-
         let starting_indexes = Indexes::try_from((&mut self.vecs, &self.stores, rpc))
             .unwrap_or_else(|_report| Indexes::default());
 
-        // dbg!(&starting_indexes);
-
         let lock = exit.lock();
         self.stores
             .rollback_if_needed(&mut self.vecs, &starting_indexes)?;
@@ -100,6 +90,7 @@ impl Indexer {
         let export =
             |stores: &mut Stores, vecs: &mut Vecs, height: Height, exit: &Exit| -> Result<()> {
                 info!("Exporting...");
+                // std::process::exit(0);
                 let _lock = exit.lock();
                 let i = Instant::now();
                 stores.commit(height).unwrap();
@@ -108,7 +99,6 @@ impl Indexer {
                 vecs.flush(height)?;
                 info!("Flushed vecs in {}s", i.elapsed().as_secs());
                 let i = Instant::now();
-                db.flush()?;
                 info!("Flushed db in {}s", i.elapsed().as_secs());
                 Ok(())
             };
@@ -174,7 +164,10 @@ impl Indexer {
         );
 
         parser.parse(start, end).iter().try_for_each(
-            |(height, block, blockhash)| -> Result<()> {
+            |block| -> Result<()> {
+                let height = block.height();
+                let blockhash = block.hash();
+
                 info!("Indexing block {height}...");
 
                 idxs.height = height;
@@ -190,10 +183,9 @@ impl Indexer {
                 let p2aaddressindex_to_p2abytes_reader = p2aaddressindex_to_p2abytes_reader_opt.as_ref().unwrap();
 
                 // Used to check rapidhash collisions
-                let check_collisions = check_collisions && height > COLLISIONS_CHECKED_UP_TO ;
+                let check_collisions = check_collisions && height > COLLISIONS_CHECKED_UP_TO;
 
-                let blockhash = BlockHash::from(blockhash);
+                let blockhash_prefix = BlockHashPrefix::from(blockhash);
-                let blockhash_prefix = BlockHashPrefix::from(&blockhash);
 
                 if stores
                     .blockhashprefix_to_height
@@ -210,7 +202,11 @@ impl Indexer {
                     .blockhashprefix_to_height
                     .insert_if_needed(blockhash_prefix, height, height);
 
-                vecs.height_to_blockhash.push_if_needed(height, blockhash)?;
+                stores
+                    .height_to_coinbase_tag
+                    .insert_if_needed( height, block.coinbase_tag().into(), height);
+
+                vecs.height_to_blockhash.push_if_needed(height, blockhash.clone())?;
                 vecs.height_to_difficulty
                     .push_if_needed(height, block.header.difficulty_float().into())?;
                 vecs.height_to_timestamp
@@ -218,275 +214,241 @@ impl Indexer {
                 vecs.height_to_total_size.push_if_needed(height, block.total_size().into())?;
                 vecs.height_to_weight.push_if_needed(height, block.weight().into())?;
 
-                let (
+                let txid_prefix_to_txid_and_block_txindex_and_prev_txindex = block
-                    txid_prefix_to_txid_and_block_txindex_and_prev_txindex_join_handle,
+                    .txdata
-                    input_source_vec_handle,
+                    .par_iter()
-                    outputindex_to_txout_outputtype_addressbytes_res_addressindex_opt_handle,
+                    .enumerate()
-                ) = thread::scope(|scope| {
+                    .map(|(index, tx)| {
-                    let txid_prefix_to_txid_and_block_txindex_and_prev_txindex_handle =
+                        let txid = Txid::from(tx.compute_txid());
-                        scope.spawn(|| -> Result<_> {
-                            block
-                                .txdata
-                                .iter()
-                                .enumerate()
-                                .map(|(index, tx)| {
-                                    let txid = Txid::from(tx.compute_txid());
 
-                                    let txid_prefix = TxidPrefix::from(&txid);
+                        let txid_prefix = TxidPrefix::from(&txid);
 
-                                    let prev_txindex_opt =
+                        let prev_txindex_opt =
-                                        if check_collisions && stores.txidprefix_to_txindex.needs(height) {
+                            if check_collisions && stores.txidprefix_to_txindex.needs(height) {
-                                            // Should only find collisions for two txids (duplicates), see below
+                                // Should only find collisions for two txids (duplicates), see below
-                                            stores.txidprefix_to_txindex.get(&txid_prefix)?.map(|v| *v)
+                                stores.txidprefix_to_txindex.get(&txid_prefix)?.map(|v| *v)
-                                        } else {
+                            } else {
-                                            None
+                                None
-                                        };
+                            };
 
-                                    Ok((txid_prefix, (tx, txid, TxIndex::from(index), prev_txindex_opt)))
+                        Ok((txid_prefix, (tx, txid, TxIndex::from(index), prev_txindex_opt)))
-                                })
+                    })
-                                .collect::<Result<BTreeMap<_, _>>>()
+                    .collect::<Result<BTreeMap<_, _>>>()?;
-                        });
 
-                    let input_source_vec_handle = scope.spawn(|| {
+                let inputs = block
-                        let inputs = block
+                    .txdata
-                            .txdata
+                    .iter()
+                    .enumerate()
+                    .flat_map(|(index, tx)| {
+                        tx.input
                             .iter()
                             .enumerate()
-                            .flat_map(|(index, tx)| {
+                            .map(move |(vin, txin)| (TxIndex::from(index), Vin::from(vin), txin, tx))
-                                tx.input
+                    })
-                                    .iter()
+                    .collect::<Vec<_>>();
-                                    .enumerate()
-                                    .map(move |(vin, txin)| (TxIndex::from(index), Vin::from(vin), txin, tx))
-                            })
-                            .collect::<Vec<_>>();
 
-                        inputs
+                let input_source_vec = inputs
-                            .into_par_iter()
+                    .into_par_iter()
+                    .enumerate()
+                    .map(|(block_inputindex, (block_txindex, vin, txin, tx))| -> Result<(InputIndex, InputSource)> {
+                        let txindex = idxs.txindex + block_txindex;
+                        let inputindex = idxs.inputindex + InputIndex::from(block_inputindex);
+
+                        let outpoint = txin.previous_output;
+                        let txid = Txid::from(outpoint.txid);
+
+                        if tx.is_coinbase() {
+                            return Ok((inputindex, InputSource::SameBlock((tx, txindex, txin, vin))));
+                        }
+
+                        let prev_txindex = if let Some(txindex) = stores
+                            .txidprefix_to_txindex
+                            .get(&TxidPrefix::from(&txid))?
+                            .map(|v| *v)
+                            .and_then(|txindex| {
+                                // Checking if not finding txindex from the future
+                                (txindex < idxs.txindex).then_some(txindex)
+                            }) {
+                            txindex
+                        } else {
+                            // dbg!(indexes.txindex + block_txindex, txindex, txin, vin);
+                            return Ok((inputindex, InputSource::SameBlock((tx, txindex, txin, vin))));
+                        };
+
+                        let vout = Vout::from(outpoint.vout);
+
+                        let outputindex = vecs.txindex_to_first_outputindex.get_or_read(prev_txindex, txindex_to_first_outputindex_reader)?
+                            .ok_or(Error::Str("Expect outputindex to not be none"))
+                            .inspect_err(|_| {
+                                dbg!(outpoint.txid, prev_txindex, vout);
+                            })?.into_owned()
+                            + vout;
+
+                        Ok((inputindex, InputSource::PreviousBlock((
+                            vin,
+                            txindex,
+                            outputindex,
+                        ))))
+                    })
+                    .try_fold(BTreeMap::new, |mut map, tuple| -> Result<_> {
+                        let (key, value) = tuple?;
+                        map.insert(key, value);
+                        Ok(map)
+                    })
+                    .try_reduce(BTreeMap::new, |mut map, mut map2| {
+                        if map.len() > map2.len() {
+                            map.append(&mut map2);
+                            Ok(map)
+                        } else {
+                            map2.append(&mut map);
+                            Ok(map2)
+                        }
+                    })?;
+
+                let outputs = block
+                    .txdata
+                    .iter()
+                    .enumerate()
+                    .flat_map(|(index, tx)| {
+                        tx.output
+                            .iter()
                             .enumerate()
-                            .map(|(block_inputindex, (block_txindex, vin, txin, tx))| -> Result<(InputIndex, InputSource)> {
+                            .map(move |(vout, txout)| (TxIndex::from(index), Vout::from(vout), txout, tx))
-                                let txindex = idxs.txindex + block_txindex;
+                    }).collect::<Vec<_>>();
-                                let inputindex = idxs.inputindex + InputIndex::from(block_inputindex);
 
-                                let outpoint = txin.previous_output;
+                let outputindex_to_txout_outputtype_addressbytes_res_addressindex_opt = outputs.into_par_iter()
-                                let txid = Txid::from(outpoint.txid);
+                    .enumerate()
+                    .map(
+                        #[allow(clippy::type_complexity)]
+                        |(block_outputindex, (block_txindex, vout, txout, tx))| -> Result<(
+                            OutputIndex,
+                            (
+                                &TxOut,
+                                TxIndex,
+                                Vout,
+                                OutputType,
+                                Result<AddressBytes>,
+                                Option<TypeIndex>,
+                                &Transaction,
+                            ),
+                        )> {
+                            let txindex = idxs.txindex + block_txindex;
+                            let outputindex = idxs.outputindex + OutputIndex::from(block_outputindex);
 
-                                if tx.is_coinbase() {
+                            let script = &txout.script_pubkey;
-                                    return Ok((inputindex, InputSource::SameBlock((tx, txindex, txin, vin))));
-                                }
 
-                                let prev_txindex = if let Some(txindex) = stores
+                            let outputtype = OutputType::from(script);
-                                    .txidprefix_to_txindex
-                                    .get(&TxidPrefix::from(&txid))?
-                                    .map(|v| *v)
-                                    .and_then(|txindex| {
-                                        // Checking if not finding txindex from the future
-                                        (txindex < idxs.txindex).then_some(txindex)
-                                    }) {
-                                    txindex
-                                } else {
-                                    // dbg!(indexes.txindex + block_txindex, txindex, txin, vin);
-                                    return Ok((inputindex, InputSource::SameBlock((tx, txindex, txin, vin))));
-                                };
 
-                                let vout = Vout::from(outpoint.vout);
+                            let address_bytes_res =
-
+                                AddressBytes::try_from((script, outputtype)).inspect_err(|_| {
-                                let outputindex = vecs.txindex_to_first_outputindex.get_or_read(prev_txindex, txindex_to_first_outputindex_reader)?
+                                    // dbg!(&txout, height, txi, &tx.compute_txid());
-                                    .ok_or(Error::Str("Expect outputindex to not be none"))
-                                    .inspect_err(|_| {
-                                        dbg!(outpoint.txid, prev_txindex, vout);
-                                    })?.into_owned()
-                                    + vout;
-
-                                Ok((inputindex, InputSource::PreviousBlock((
-                                    vin,
-                                    txindex,
-                                    outputindex,
-                                ))))
-                            })
-                            .try_fold(BTreeMap::new, |mut map, tuple| -> Result<_> {
-                                let (key, value) = tuple?;
-                                map.insert(key, value);
-                                Ok(map)
-                            })
-                            .try_reduce(BTreeMap::new, |mut map, mut map2| {
-                                if map.len() > map2.len() {
-                                    map.append(&mut map2);
-                                    Ok(map)
-                                } else {
-                                    map2.append(&mut map);
-                                    Ok(map2)
-                                }
-                            })
-                    });
-
-                    let outputs = block
-                        .txdata
-                        .iter()
-                        .enumerate()
-                        .flat_map(|(index, tx)| {
-                            tx.output
-                                .iter()
-                                .enumerate()
-                                .map(move |(vout, txout)| (TxIndex::from(index), Vout::from(vout), txout, tx))
-                        }).collect::<Vec<_>>();
-
-                    let outputindex_to_txout_outputtype_addressbytes_res_addressindex = outputs.into_par_iter()
-                        .enumerate()
-                        .map(
-                            #[allow(clippy::type_complexity)]
-                            |(block_outputindex, (block_txindex, vout, txout, tx))| -> Result<(
-                                OutputIndex,
-                                (
-                                    &TxOut,
-                                    TxIndex,
-                                    Vout,
-                                    OutputType,
-                                    Result<AddressBytes>,
-                                    Option<TypeIndex>,
-                                    &Transaction,
-                                ),
-                            )> {
-                                let txindex = idxs.txindex + block_txindex;
-                                let outputindex = idxs.outputindex + OutputIndex::from(block_outputindex);
-
-                                let script = &txout.script_pubkey;
-
-                                let outputtype = OutputType::from(script);
-
-                                let address_bytes_res =
-                                    AddressBytes::try_from((script, outputtype)).inspect_err(|_| {
-                                        // dbg!(&txout, height, txi, &tx.compute_txid());
-                                    });
-
-                                let typeindex_opt = address_bytes_res.as_ref().ok().and_then(|addressbytes| {
-                                    stores
-                                        .addressbyteshash_to_typeindex
-                                        .get(&AddressBytesHash::from((addressbytes, outputtype)))
-                                        .unwrap()
-                                        .map(|v| *v)
-                                        // Checking if not in the future
-                                        .and_then(|typeindex_local| {
-                                            (typeindex_local < idxs.typeindex(outputtype)).then_some(typeindex_local)
-                                        })
                                 });
 
-                                if let Some(Some(typeindex)) = check_collisions.then_some(typeindex_opt) {
+                            let typeindex_opt = address_bytes_res.as_ref().ok().and_then(|addressbytes| {
-                                    let addressbytes = address_bytes_res.as_ref().unwrap();
+                                stores
+                                    .addressbyteshash_to_typeindex
+                                    .get(&AddressBytesHash::from((addressbytes, outputtype)))
+                                    .unwrap()
+                                    .map(|v| *v)
+                                    // Checking if not in the future
+                                    .and_then(|typeindex_local| {
+                                        (typeindex_local < idxs.to_typeindex(outputtype)).then_some(typeindex_local)
+                                    })
+                            });
 
-                                    let prev_addressbytes_opt = match outputtype {
+                            if let Some(Some(typeindex)) = check_collisions.then_some(typeindex_opt) {
-                                        OutputType::P2PK65 => vecs
+                                let addressbytes = address_bytes_res.as_ref().unwrap();
-                                            .p2pk65addressindex_to_p2pk65bytes
-                                            .get_or_read(typeindex.into(), p2pk65addressindex_to_p2pk65bytes_reader)?
-                                            .map(|v| AddressBytes::from(v.into_owned())),
-                                        OutputType::P2PK33 => vecs
-                                            .p2pk33addressindex_to_p2pk33bytes
-                                            .get_or_read(typeindex.into(), p2pk33addressindex_to_p2pk33bytes_reader)?
-                                            .map(|v| AddressBytes::from(v.into_owned())),
-                                        OutputType::P2PKH => vecs
-                                            .p2pkhaddressindex_to_p2pkhbytes
-                                            .get_or_read(typeindex.into(), p2pkhaddressindex_to_p2pkhbytes_reader)?
-                                            .map(|v| AddressBytes::from(v.into_owned())),
-                                        OutputType::P2SH => vecs
-                                            .p2shaddressindex_to_p2shbytes
-                                            .get_or_read(typeindex.into(), p2shaddressindex_to_p2shbytes_reader)?
-                                            .map(|v| AddressBytes::from(v.into_owned())),
-                                        OutputType::P2WPKH => vecs
-                                            .p2wpkhaddressindex_to_p2wpkhbytes
-                                            .get_or_read(typeindex.into(), p2wpkhaddressindex_to_p2wpkhbytes_reader)?
-                                            .map(|v| AddressBytes::from(v.into_owned())),
-                                        OutputType::P2WSH => vecs
-                                            .p2wshaddressindex_to_p2wshbytes
-                                            .get_or_read(typeindex.into(), p2wshaddressindex_to_p2wshbytes_reader)?
-                                            .map(|v| AddressBytes::from(v.into_owned())),
-                                        OutputType::P2TR => vecs
-                                            .p2traddressindex_to_p2trbytes
-                                            .get_or_read(typeindex.into(), p2traddressindex_to_p2trbytes_reader)?
-                                            .map(|v| AddressBytes::from(v.into_owned())),
-                                        OutputType::P2A => vecs
-                                            .p2aaddressindex_to_p2abytes
-                                            .get_or_read(typeindex.into(), p2aaddressindex_to_p2abytes_reader)?
-                                            .map(|v| AddressBytes::from(v.into_owned())),
-                                        _ => {
-                                            unreachable!()
-                                        }
-                                    };
-                                    let prev_addressbytes =
-                                        prev_addressbytes_opt.as_ref().ok_or(Error::Str("Expect to have addressbytes"))?;
 
-                                    if stores.addressbyteshash_to_typeindex.needs(height)
+                                let prev_addressbytes_opt = match outputtype {
-                                            && prev_addressbytes != addressbytes
+                                    OutputType::P2PK65 => vecs
-                                    {
+                                        .p2pk65addressindex_to_p2pk65bytes
-                                        let txid = tx.compute_txid();
+                                        .get_or_read(typeindex.into(), p2pk65addressindex_to_p2pk65bytes_reader)?
-                                        dbg!(
+                                        .map(|v| AddressBytes::from(v.into_owned())),
-                                            height,
+                                    OutputType::P2PK33 => vecs
-                                            txid,
+                                        .p2pk33addressindex_to_p2pk33bytes
-                                            vout,
+                                        .get_or_read(typeindex.into(), p2pk33addressindex_to_p2pk33bytes_reader)?
-                                            block_txindex,
+                                        .map(|v| AddressBytes::from(v.into_owned())),
-                                            outputtype,
+                                    OutputType::P2PKH => vecs
-                                            prev_addressbytes,
+                                        .p2pkhaddressindex_to_p2pkhbytes
-                                            addressbytes,
+                                        .get_or_read(typeindex.into(), p2pkhaddressindex_to_p2pkhbytes_reader)?
-                                            &idxs,
+                                        .map(|v| AddressBytes::from(v.into_owned())),
-                                            typeindex,
+                                    OutputType::P2SH => vecs
-                                            typeindex,
+                                        .p2shaddressindex_to_p2shbytes
-                                            txout,
+                                        .get_or_read(typeindex.into(), p2shaddressindex_to_p2shbytes_reader)?
-                                            AddressBytesHash::from((addressbytes, outputtype)),
+                                        .map(|v| AddressBytes::from(v.into_owned())),
-                                        );
+                                    OutputType::P2WPKH => vecs
-                                        panic!()
+                                        .p2wpkhaddressindex_to_p2wpkhbytes
+                                        .get_or_read(typeindex.into(), p2wpkhaddressindex_to_p2wpkhbytes_reader)?
+                                        .map(|v| AddressBytes::from(v.into_owned())),
+                                    OutputType::P2WSH => vecs
+                                        .p2wshaddressindex_to_p2wshbytes
+                                        .get_or_read(typeindex.into(), p2wshaddressindex_to_p2wshbytes_reader)?
+                                        .map(|v| AddressBytes::from(v.into_owned())),
+                                    OutputType::P2TR => vecs
+                                        .p2traddressindex_to_p2trbytes
+                                        .get_or_read(typeindex.into(), p2traddressindex_to_p2trbytes_reader)?
+                                        .map(|v| AddressBytes::from(v.into_owned())),
+                                    OutputType::P2A => vecs
+                                        .p2aaddressindex_to_p2abytes
+                                        .get_or_read(typeindex.into(), p2aaddressindex_to_p2abytes_reader)?
+                                        .map(|v| AddressBytes::from(v.into_owned())),
+                                    _ => {
+                                        unreachable!()
                                     }
-                                }
+                                };
+                                let prev_addressbytes =
+                                    prev_addressbytes_opt.as_ref().ok_or(Error::Str("Expect to have addressbytes"))?;
 
-                                Ok((
+                                if stores.addressbyteshash_to_typeindex.needs(height)
-                                    outputindex,
+                                        && prev_addressbytes != addressbytes
-                                    (
+                                {
-                                        txout,
+                                    let txid = tx.compute_txid();
-                                        txindex,
+                                    dbg!(
+                                        height,
+                                        txid,
                                         vout,
+                                        block_txindex,
                                         outputtype,
-                                        address_bytes_res,
+                                        prev_addressbytes,
-                                        typeindex_opt,
+                                        addressbytes,
-                                        tx,
+                                        &idxs,
-                                    ),
+                                        typeindex,
-                                ))
+                                        typeindex,
-                            },
+                                        txout,
-                        )
+                                        AddressBytesHash::from((addressbytes, outputtype)),
-                        .try_fold(BTreeMap::new, |mut map, tuple| -> Result<_> {
+                                    );
-                            let (key, value) = tuple?;
+                                    panic!()
-                            map.insert(key, value);
+                                }
-                            Ok(map)
-                        })
-                        .try_reduce(BTreeMap::new, |mut map, mut map2| {
-                            if map.len() > map2.len() {
-                                map.append(&mut map2);
-                                Ok(map)
-                            } else {
-                                map2.append(&mut map);
-                                Ok(map2)
                             }
-                        });
 
-                    (
+                            Ok((
-                        txid_prefix_to_txid_and_block_txindex_and_prev_txindex_handle.join(),
+                                outputindex,
-                        input_source_vec_handle.join(),
+                                (
-                        outputindex_to_txout_outputtype_addressbytes_res_addressindex,
+                                    txout,
+                                    txindex,
+                                    vout,
+                                    outputtype,
+                                    address_bytes_res,
+                                    typeindex_opt,
+                                    tx,
+                                ),
+                            ))
+                        },
                     )
-                });
+                    .try_fold(BTreeMap::new, |mut map, tuple| -> Result<_> {
-
+                        let (key, value) = tuple?;
-                let txid_prefix_to_txid_and_block_txindex_and_prev_txindex =
+                        map.insert(key, value);
-                    txid_prefix_to_txid_and_block_txindex_and_prev_txindex_join_handle
+                        Ok(map)
-                        .map_err(|_|
+                    })
-                            Error::Str("Expect txid_prefix_to_txid_and_block_txindex_and_prev_txindex_join_handle to join")
+                    .try_reduce(BTreeMap::new, |mut map, mut map2| {
-                        )??;
+                        if map.len() > map2.len() {
-
+                            map.append(&mut map2);
-                let input_source_vec = input_source_vec_handle
+                            Ok(map)
-                    .map_err(|_|
+                        } else {
-                        Error::Str("Export input_source_vec_handle to join")
+                            map2.append(&mut map);
-                    )??;
+                            Ok(map2)
-
+                        }
-                let outputindex_to_txout_outputtype_addressbytes_res_addressindex_opt =
+                    })?;
-                    outputindex_to_txout_outputtype_addressbytes_res_addressindex_opt_handle
-                        .map_err(|_|
-                            Error::Str("Expect outputindex_to_txout_outputtype_addressbytes_res_addressindex_opt_handle to join")
-                        )?;
 
                 let outputs_len = outputindex_to_txout_outputtype_addressbytes_res_addressindex_opt.len();
                 let inputs_len = input_source_vec.len();
@@ -750,7 +712,6 @@ impl Indexer {
                 idxs.inputindex += InputIndex::from(inputs_len);
                 idxs.outputindex += OutputIndex::from(outputs_len);
 
-
                 if should_export(height, false) {
                     txindex_to_first_outputindex_reader_opt.take();
                     p2pk65addressindex_to_p2pk65bytes_reader_opt.take();
@@ -797,7 +758,7 @@ impl Indexer {
         }
 
         let i = Instant::now();
-        db.punch_holes()?;
+        self.vecs.punch_holes()?;
         info!("Punched holes in db in {}s", i.elapsed().as_secs());
 
         Ok(starting_indexes)

crates/brk_indexer/src/stores.rs

@@ -4,11 +4,12 @@ use brk_error::Result;
 use brk_store::{AnyStore, Store};
 use brk_structs::{
     AddressBytes, AddressBytesHash, BlockHashPrefix, ByAddressType, Height, OutputIndex,
-    OutputType, TxIndex, TxidPrefix, TypeIndex, TypeIndexWithOutputindex, Unit, Version,
+    OutputType, StoredString, TxIndex, TxidPrefix, TypeIndex, TypeIndexWithOutputindex, Unit,
+    Version,
 };
 use fjall::{PersistMode, TransactionalKeyspace};
 use rayon::prelude::*;
-use vecdb::VecIterator;
+use vecdb::{AnyVec, StoredIndex, VecIterator};
 
 use crate::Indexes;
 
@@ -20,22 +21,23 @@ pub struct Stores {
 
     pub addressbyteshash_to_typeindex: Store<AddressBytesHash, TypeIndex>,
     pub blockhashprefix_to_height: Store<BlockHashPrefix, Height>,
+    pub height_to_coinbase_tag: Store<Height, StoredString>,
     pub txidprefix_to_txindex: Store<TxidPrefix, TxIndex>,
     pub addresstype_to_typeindex_with_outputindex:
         ByAddressType<Store<TypeIndexWithOutputindex, Unit>>,
 }
 
-const VERSION: Version = Version::ZERO;
-
 impl Stores {
-    pub fn forced_import(path: &Path, version: Version) -> Result<Self> {
+    pub fn forced_import(parent: &Path, version: Version) -> Result<Self> {
-        fs::create_dir_all(path)?;
+        let path = parent.join("stores");
 
-        let keyspace = match brk_store::open_keyspace(path) {
+        fs::create_dir_all(&path)?;
+
+        let keyspace = match brk_store::open_keyspace(&path) {
             Ok(keyspace) => keyspace,
             Err(_) => {
-                fs::remove_dir_all(path)?;
+                fs::remove_dir_all(&path)?;
-                return Self::forced_import(path, version);
+                return Self::forced_import(&path, version);
             }
         };
 
@@ -43,106 +45,97 @@ impl Stores {
             let addressbyteshash_to_typeindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "addressbyteshash_to_typeindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     None,
                 )
             });
             let blockhashprefix_to_height = scope.spawn(|| {
-                Store::import(
+                Store::import(&keyspace, &path, "blockhashprefix_to_height", version, None)
-                    &keyspace,
-                    path,
-                    "blockhashprefix_to_height",
-                    version + VERSION + Version::ZERO,
-                    None,
-                )
-            });
-            let txidprefix_to_txindex = scope.spawn(|| {
-                Store::import(
-                    &keyspace,
-                    path,
-                    "txidprefix_to_txindex",
-                    version + VERSION + Version::ZERO,
-                    None,
-                )
             });
+            let txidprefix_to_txindex = scope
+                .spawn(|| Store::import(&keyspace, &path, "txidprefix_to_txindex", version, None));
             let p2aaddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2aaddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2pk33addressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2pk33addressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2pk65addressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2pk65addressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2pkhaddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2pkhaddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2shaddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2shaddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2traddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2traddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2wpkhaddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2wpkhaddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2wshaddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2wshaddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
 
+            let height_to_coinbase_tag =
+                Store::import(&keyspace, &path, "height_to_coinbase_tag", version, None)?;
+
             Ok(Self {
                 keyspace: keyspace.clone(),
 
+                height_to_coinbase_tag,
                 addressbyteshash_to_typeindex: addressbyteshash_to_typeindex.join().unwrap()?,
                 blockhashprefix_to_height: blockhashprefix_to_height.join().unwrap()?,
                 txidprefix_to_txindex: txidprefix_to_txindex.join().unwrap()?,
@@ -182,11 +175,9 @@ impl Stores {
             .map_err(|e| e.into())
     }
 
-    fn as_slice(&self) -> [&(dyn AnyStore + Send + Sync); 11] {
+    fn as_slice(&self) -> [&(dyn AnyStore + Send + Sync); 12] {
         [
             &self.addressbyteshash_to_typeindex,
-            &self.blockhashprefix_to_height,
-            &self.txidprefix_to_txindex,
             &self.addresstype_to_typeindex_with_outputindex.p2a,
             &self.addresstype_to_typeindex_with_outputindex.p2pk33,
             &self.addresstype_to_typeindex_with_outputindex.p2pk65,
@@ -195,14 +186,15 @@ impl Stores {
             &self.addresstype_to_typeindex_with_outputindex.p2tr,
             &self.addresstype_to_typeindex_with_outputindex.p2wpkh,
             &self.addresstype_to_typeindex_with_outputindex.p2wsh,
+            &self.blockhashprefix_to_height,
+            &self.height_to_coinbase_tag,
+            &self.txidprefix_to_txindex,
         ]
     }
 
-    fn as_mut_slice(&mut self) -> [&mut (dyn AnyStore + Send + Sync); 11] {
+    fn as_mut_slice(&mut self) -> [&mut (dyn AnyStore + Send + Sync); 12] {
         [
             &mut self.addressbyteshash_to_typeindex,
-            &mut self.blockhashprefix_to_height,
-            &mut self.txidprefix_to_txindex,
             &mut self.addresstype_to_typeindex_with_outputindex.p2a,
             &mut self.addresstype_to_typeindex_with_outputindex.p2pk33,
             &mut self.addresstype_to_typeindex_with_outputindex.p2pk65,
@@ -211,6 +203,9 @@ impl Stores {
             &mut self.addresstype_to_typeindex_with_outputindex.p2tr,
             &mut self.addresstype_to_typeindex_with_outputindex.p2wpkh,
             &mut self.addresstype_to_typeindex_with_outputindex.p2wsh,
+            &mut self.blockhashprefix_to_height,
+            &mut self.height_to_coinbase_tag,
+            &mut self.txidprefix_to_txindex,
         ]
     }
 
@@ -222,6 +217,7 @@ impl Stores {
         if self.addressbyteshash_to_typeindex.is_empty()?
             && self.blockhashprefix_to_height.is_empty()?
             && self.txidprefix_to_txindex.is_empty()?
+            && self.height_to_coinbase_tag.is_empty()?
             && self
                 .addresstype_to_typeindex_with_outputindex
                 .p2a
@@ -266,6 +262,12 @@ impl Stores {
                     self.blockhashprefix_to_height.remove(blockhashprefix);
                 });
 
+            (starting_indexes.height.unwrap_to_usize()..vecs.height_to_blockhash.len())
+                .map(Height::from)
+                .for_each(|h| {
+                    self.height_to_coinbase_tag.remove(h);
+                });
+
             if let Some(mut index) = vecs
                 .height_to_first_p2pk65addressindex
                 .iter()

crates/brk_indexer/src/vecs.rs

@@ -1,3 +1,5 @@
+use std::path::Path;
+
 use brk_error::Result;
 use brk_structs::{
     AddressBytes, BlockHash, EmptyOutputIndex, Height, InputIndex, OpReturnIndex, OutputIndex,
@@ -9,15 +11,15 @@ use brk_structs::{
 };
 use rayon::prelude::*;
 use vecdb::{
-    AnyCollectableVec, AnyStoredVec, CompressedVec, Database, GenericStoredVec, RawVec, Stamp,
+    AnyCollectableVec, AnyStoredVec, CompressedVec, Database, GenericStoredVec, PAGE_SIZE, RawVec,
+    Stamp,
 };
 
 use crate::Indexes;
 
-const VERSION: Version = Version::ZERO;
-
 #[derive(Clone)]
 pub struct Vecs {
+    db: Database,
     pub emptyoutputindex_to_txindex: CompressedVec<EmptyOutputIndex, TxIndex>,
     pub height_to_blockhash: RawVec<Height, BlockHash>,
     pub height_to_difficulty: CompressedVec<Height, StoredF64>,
@@ -36,7 +38,7 @@ pub struct Vecs {
     pub height_to_first_p2wshaddressindex: CompressedVec<Height, P2WSHAddressIndex>,
     pub height_to_first_txindex: CompressedVec<Height, TxIndex>,
     pub height_to_first_unknownoutputindex: CompressedVec<Height, UnknownOutputIndex>,
-    /// Doesn't guarantee continuity due to possible reorgs
+    /// Doesn't guarantee continuity due to possible reorgs and more generally the nature of mining
     pub height_to_timestamp: CompressedVec<Height, Timestamp>,
     pub height_to_total_size: CompressedVec<Height, StoredU64>,
     pub height_to_weight: CompressedVec<Height, Weight>,
@@ -67,225 +69,134 @@ pub struct Vecs {
 }
 
 impl Vecs {
-    pub fn forced_import(db: &Database, version: Version) -> Result<Self> {
+    pub fn forced_import(parent: &Path, version: Version) -> Result<Self> {
-        Ok(Self {
+        let db = Database::open(&parent.join("vecs"))?;
-            emptyoutputindex_to_txindex: CompressedVec::forced_import(
+        db.set_min_len(PAGE_SIZE * 50_000_000)?;
-                db,
+
-                "txindex",
+        let db_positions = Database::open(&parent.join("vecs/positions"))?;
-                version + VERSION + Version::ZERO,
+        db_positions.set_min_len(PAGE_SIZE * 1_000_000)?;
-            )?,
+
-            height_to_blockhash: RawVec::forced_import(
+        let this = Self {
-                db,
+            emptyoutputindex_to_txindex: CompressedVec::forced_import(&db, "txindex", version)?,
-                "blockhash",
+            height_to_blockhash: RawVec::forced_import(&db, "blockhash", version)?,
-                version + VERSION + Version::ZERO,
+            height_to_difficulty: CompressedVec::forced_import(&db, "difficulty", version)?,
-            )?,
-            height_to_difficulty: CompressedVec::forced_import(
-                db,
-                "difficulty",
-                version + VERSION + Version::ZERO,
-            )?,
             height_to_first_emptyoutputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_emptyoutputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_inputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_inputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_opreturnindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_opreturnindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_outputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_outputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2aaddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2aaddressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2msoutputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2msoutputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2pk33addressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2pk33addressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2pk65addressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2pk65addressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2pkhaddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2pkhaddressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2shaddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2shaddressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2traddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2traddressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2wpkhaddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2wpkhaddressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2wshaddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2wshaddressindex",
-                version + VERSION + Version::ZERO,
+                version,
-            )?,
-            height_to_first_txindex: CompressedVec::forced_import(
-                db,
-                "first_txindex",
-                version + VERSION + Version::ZERO,
             )?,
+            height_to_first_txindex: CompressedVec::forced_import(&db, "first_txindex", version)?,
             height_to_first_unknownoutputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_unknownoutputindex",
-                version + VERSION + Version::ZERO,
+                version,
-            )?,
-            height_to_timestamp: CompressedVec::forced_import(
-                db,
-                "timestamp",
-                version + VERSION + Version::ZERO,
-            )?,
-            height_to_total_size: CompressedVec::forced_import(
-                db,
-                "total_size",
-                version + VERSION + Version::ZERO,
-            )?,
-            height_to_weight: CompressedVec::forced_import(
-                db,
-                "weight",
-                version + VERSION + Version::ZERO,
-            )?,
-            inputindex_to_outputindex: RawVec::forced_import(
-                db,
-                "outputindex",
-                version + VERSION + Version::ZERO,
-            )?,
-            opreturnindex_to_txindex: CompressedVec::forced_import(
-                db,
-                "txindex",
-                version + VERSION + Version::ZERO,
-            )?,
-            outputindex_to_outputtype: RawVec::forced_import(
-                db,
-                "outputtype",
-                version + VERSION + Version::ZERO,
-            )?,
-            outputindex_to_typeindex: RawVec::forced_import(
-                db,
-                "typeindex",
-                version + VERSION + Version::ZERO,
-            )?,
-            outputindex_to_value: RawVec::forced_import(
-                db,
-                "value",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2aaddressindex_to_p2abytes: RawVec::forced_import(
-                db,
-                "p2abytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2msoutputindex_to_txindex: CompressedVec::forced_import(
-                db,
-                "txindex",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2pk33addressindex_to_p2pk33bytes: RawVec::forced_import(
-                db,
-                "p2pk33bytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2pk65addressindex_to_p2pk65bytes: RawVec::forced_import(
-                db,
-                "p2pk65bytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2pkhaddressindex_to_p2pkhbytes: RawVec::forced_import(
-                db,
-                "p2pkhbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2shaddressindex_to_p2shbytes: RawVec::forced_import(
-                db,
-                "p2shbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2traddressindex_to_p2trbytes: RawVec::forced_import(
-                db,
-                "p2trbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2wpkhaddressindex_to_p2wpkhbytes: RawVec::forced_import(
-                db,
-                "p2wpkhbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2wshaddressindex_to_p2wshbytes: RawVec::forced_import(
-                db,
-                "p2wshbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            txindex_to_base_size: CompressedVec::forced_import(
-                db,
-                "base_size",
-                version + VERSION + Version::ZERO,
             )?,
+            height_to_timestamp: CompressedVec::forced_import(&db, "timestamp", version)?,
+            height_to_total_size: CompressedVec::forced_import(&db, "total_size", version)?,
+            height_to_weight: CompressedVec::forced_import(&db, "weight", version)?,
+            inputindex_to_outputindex: RawVec::forced_import(&db, "outputindex", version)?,
+            opreturnindex_to_txindex: CompressedVec::forced_import(&db, "txindex", version)?,
+            outputindex_to_outputtype: RawVec::forced_import(&db, "outputtype", version)?,
+            outputindex_to_typeindex: RawVec::forced_import(&db, "typeindex", version)?,
+            outputindex_to_value: RawVec::forced_import(&db, "value", version)?,
+            p2aaddressindex_to_p2abytes: RawVec::forced_import(&db, "p2abytes", version)?,
+            p2msoutputindex_to_txindex: CompressedVec::forced_import(&db, "txindex", version)?,
+            p2pk33addressindex_to_p2pk33bytes: RawVec::forced_import(&db, "p2pk33bytes", version)?,
+            p2pk65addressindex_to_p2pk65bytes: RawVec::forced_import(&db, "p2pk65bytes", version)?,
+            p2pkhaddressindex_to_p2pkhbytes: RawVec::forced_import(&db, "p2pkhbytes", version)?,
+            p2shaddressindex_to_p2shbytes: RawVec::forced_import(&db, "p2shbytes", version)?,
+            p2traddressindex_to_p2trbytes: RawVec::forced_import(&db, "p2trbytes", version)?,
+            p2wpkhaddressindex_to_p2wpkhbytes: RawVec::forced_import(&db, "p2wpkhbytes", version)?,
+            p2wshaddressindex_to_p2wshbytes: RawVec::forced_import(&db, "p2wshbytes", version)?,
+            txindex_to_base_size: CompressedVec::forced_import(&db, "base_size", version)?,
             txindex_to_first_inputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_inputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             txindex_to_first_outputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_outputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             txindex_to_is_explicitly_rbf: CompressedVec::forced_import(
-                db,
+                &db,
                 "is_explicitly_rbf",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
-            txindex_to_rawlocktime: CompressedVec::forced_import(
+            txindex_to_rawlocktime: CompressedVec::forced_import(&db, "rawlocktime", version)?,
-                db,
+            txindex_to_total_size: CompressedVec::forced_import(&db, "total_size", version)?,
-                "rawlocktime",
+            txindex_to_txid: RawVec::forced_import(&db, "txid", version)?,
-                version + VERSION + Version::ZERO,
+            txindex_to_txversion: CompressedVec::forced_import(&db, "txversion", version)?,
-            )?,
+            unknownoutputindex_to_txindex: CompressedVec::forced_import(&db, "txindex", version)?,
-            txindex_to_total_size: CompressedVec::forced_import(
+
-                db,
+            db,
-                "total_size",
+        };
-                version + VERSION + Version::ZERO,
+
-            )?,
+        this.db
-            txindex_to_txid: RawVec::forced_import(db, "txid", version + VERSION + Version::ZERO)?,
+            .retain_regions(this.iter().flat_map(|v| v.region_names()).collect())?;
-            txindex_to_txversion: CompressedVec::forced_import(
+
-                db,
+        Ok(this)
-                "txversion",
-                version + VERSION + Version::ZERO,
-            )?,
-            unknownoutputindex_to_txindex: CompressedVec::forced_import(
-                db,
-                "txindex",
-                version + VERSION + Version::ZERO,
-            )?,
-        })
     }
 
     pub fn rollback_if_needed(&mut self, starting_indexes: &Indexes) -> Result<()> {
@@ -435,15 +346,15 @@ impl Vecs {
     }
 
     pub fn flush(&mut self, height: Height) -> Result<()> {
-        self.mut_vecs()
+        self.iter_mut()
-            .into_par_iter()
+            .par_bridge()
             .try_for_each(|vec| vec.stamped_flush(Stamp::from(height)))?;
+        self.db.flush()?;
         Ok(())
     }
 
     pub fn starting_height(&mut self) -> Height {
-        self.mut_vecs()
+        self.iter_mut()
-            .into_iter()
             .map(|vec| {
                 let h = Height::from(vec.stamp());
                 if h > Height::ZERO { h.incremented() } else { h }
@@ -452,9 +363,14 @@ impl Vecs {
             .unwrap()
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn punch_holes(&self) -> Result<()> {
-        vec![
+        self.db.punch_holes()?;
-            &self.emptyoutputindex_to_txindex,
+        Ok(())
+    }
+
+    pub fn iter(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [
+            &self.emptyoutputindex_to_txindex as &dyn AnyCollectableVec,
             &self.height_to_blockhash,
             &self.height_to_difficulty,
             &self.height_to_first_emptyoutputindex,
@@ -499,11 +415,12 @@ impl Vecs {
             &self.txindex_to_txversion,
             &self.unknownoutputindex_to_txindex,
         ]
+        .into_iter()
     }
 
-    fn mut_vecs(&mut self) -> Vec<&mut dyn AnyStoredVec> {
+    fn iter_mut(&mut self) -> impl Iterator<Item = &mut dyn AnyStoredVec> {
-        vec![
+        [
-            &mut self.emptyoutputindex_to_txindex,
+            &mut self.emptyoutputindex_to_txindex as &mut dyn AnyStoredVec,
             &mut self.height_to_blockhash,
             &mut self.height_to_difficulty,
             &mut self.height_to_first_emptyoutputindex,
@@ -548,5 +465,6 @@ impl Vecs {
             &mut self.txindex_to_txversion,
             &mut self.unknownoutputindex_to_txindex,
         ]
+        .into_iter()
     }
 }