release: v0.0.109

.claude/prompts/changelog.md

@@ -0,0 +1,137 @@
+# Changelog Generation Prompt
+
+Update docs/CHANGELOG.md for ALL latest releases not present in the file. Use ONLY git commands - no other sources.
+
+## MANDATORY PROCESS - FOLLOW EXACTLY - NO EXCEPTIONS:
+1. Run `git tag --list --sort=version:refname` to get releases in order
+2. Process EXACTLY ONE release at a time
+3. For EACH SINGLE release: run `git diff [previous-tag]..[current-tag]`
+4. **MANDATORY ANALYSIS STEP**: Before writing ANY changelog entry, you MUST:
+   - Analyze each file change and explain what the code is doing
+   - Identify the purpose and impact of each modification
+   - Group related changes together logically
+   - State clearly what functionality is being added, removed, or modified
+   - If you cannot understand what a change does from the diff, explicitly say so
+5. Only AFTER completing the analysis, write the detailed changelog entry
+6. Update the CHANGELOG.md file with that ONE entry
+7. STOP. Ask me if you should continue to the next release.
+
+## CRITICAL CONSTRAINTS:
+- **NEVER EVER** process multiple releases in one go, even if there are many
+- **NEVER** say "let me continue more efficiently by processing multiple releases"
+- **NEVER** batch releases together for any reason
+- If you feel tempted to process multiple releases, **STOP** and process only one
+- Context window concerns do **NOT** justify batching - process one release only
+
+## ABSOLUTE REQUIREMENTS:
+- **NEVER** read commit messages, PR descriptions, existing changelog, or any text documentation
+- Use **ONLY** the actual code changes shown in git diff output
+- Process releases **ONE BY ONE** - I don't care if there are 100 releases
+- **MANDATORY**: Before writing changelog entries, demonstrate understanding by analyzing what each code change accomplishes
+- Be **HIGHLY DESCRIPTIVE** about what each code change does and why it matters
+- Don't be conservative - write detailed explanations of the impact and purpose of changes
+- **If you don't understand a change from the code diff alone, DO NOT GUESS - say so explicitly**
+
+## SOURCE OF TRUTH:
+- `git diff` output is the **ONLY** source of truth
+- If you can't determine what a change does from the code diff alone, say so explicitly
+- Ignore **ALL** text/documentation - focus purely on code additions, deletions, and modifications
+
+## CHANGELOG FILE REQUIREMENTS:
+- Add a header at the top of the CHANGELOG.md file: `<!-- This changelog was generated by Claude Code -->`
+- Ensure this header appears before any changelog entries
+
+## CHANGELOG WRITING RULES:
+
+### RELEASE TITLE FORMAT:
+**MUST** use this exact format: `## [vX.Y.Z](https://github.com/bitcoinresearchkit/brk/releases/tag/vX.Y.Z) - YYYY-MM-DD`
+Use the actual release date from git tag information
+
+### ABSOLUTELY FORBIDDEN PATTERNS:
+- **NEVER** mention line counts (e.g., "with 138 lines", "1,290 lines removed")
+- **NEVER** use vague action words: "Enhanced", "Improved", "Updated", "Expanded", "Restructured", "Refactored", "Modified", "Adjusted"
+- **NEVER** write sections about Cargo.lock or dependency updates unless they represent major functional changes
+- **NEVER** use the format "Action: File with vague description"
+- **NEVER** mention version bumps of local crates (e.g., "Updated all crate versions from 0.0.61 to 0.0.62") - this is implied by the release version
+- **NEVER** mention dependency version changes in external crates unless they enable new functionality visible in the code
+- **NEVER** write entries like "Updated dependencies" or "Cargo.lock maintenance"
+
+### REQUIRED WRITING STYLE:
+- Write what the code **actually DOES**, not that it was "enhanced" or "improved"
+- Be **specific about functionality**: "Added transaction validation logic", "Implemented caching for API responses"
+- Focus on **business/functional impact**: "Enables users to...", "Fixes issue where...", "Adds support for..."
+- **Mandatory structure**: Group by: Breaking Changes, New Features, Bug Fixes, Internal Changes
+- Include GitHub file links for major changes (max 5 per entry)
+- **Skip entirely**: minor dependency bumps, Cargo.lock changes, and local crate version bumps
+
+### MANDATORY ANALYSIS WORKFLOW:
+**BEFORE writing any changelog entry, you MUST:**
+
+1. **Code Comprehension Check**: Go through each modified file and explain:
+   - What specific functionality is being added/removed/changed
+   - What the new/modified functions/structs/methods do
+   - How the changes affect the overall system behavior
+
+2. **Impact Assessment**: For each change, determine:
+   - Is this a new feature, bug fix, breaking change, or internal improvement?
+   - What user-facing or system behavior changes result from this code?
+   - What problem does this change solve?
+
+3. **Logical Grouping**: Organize related changes together:
+   - Group files that work together to implement a single feature
+   - Separate breaking changes from additions
+   - Distinguish user-facing changes from internal refactoring
+
+4. **Understanding Verification**: Before writing changelog text, state:
+   - "I understand this change does X because the code shows Y"
+   - If unclear: "I cannot determine the purpose of this change from the diff alone"
+
+**ONLY AFTER completing this analysis should you write the changelog entry.**
+
+### WHAT TO FOCUS ON (IN ORDER OF PRIORITY):
+1. **New functionality** - What can users now do that they couldn't before?
+2. **Breaking changes** - What existing functionality changed or was removed?
+3. **Bug fixes** - What specific problems were resolved?
+4. **Internal changes** - New modules, significant refactoring, architecture changes
+5. **Skip completely** - Dependency updates, version bumps, Cargo.lock changes
+
+### VERBOSITY REQUIREMENTS:
+- **Minimum 3-4 bullet points per section** when changes exist
+- **Each bullet point should be 1-2 sentences** explaining both what changed and why it matters
+- **For new features**: Explain what the feature does and what problem it solves
+- **For bug fixes**: Describe the problem that was fixed (inferred from the code changes)
+- **For internal changes**: Explain the architectural or structural improvement
+
+### EXAMPLES OF GOOD vs BAD:
+
+#### ❌ BAD EXAMPLES:
+- "Enhanced: Chain analysis with sophisticated blockchain processing capabilities"
+- "Updated: brk_rolldown from 0.0.1 to 0.1.0 with comprehensive bundling improvements"
+- "Version Bump: Updated all crate versions from 0.0.61 to 0.0.62"
+- "Improved error handling"
+- "Refactored codebase"
+- "Updated dependencies"
+
+#### ✅ GOOD EXAMPLES WITH ANALYSIS:
+
+**Analysis**: "Looking at the diff, I see a new `TransactionAnalyzer` struct was added with methods `calculate_fee()` and `is_coinbase()`. The struct takes transaction data and provides analysis methods. This enables users to programmatically analyze transaction properties."
+
+**Changelog**: "Added new `TransactionAnalyzer` struct that provides methods for computing transaction fees and detecting coinbase transactions"
+
+**Analysis**: "The diff shows error handling was added around block parsing where previously there was an unwrap(). Now it returns a Result and handles the empty block case explicitly. This prevents panics when processing malformed blocks."
+
+**Changelog**: "Fixed panic when processing blocks with zero transactions by adding explicit empty block handling and proper error propagation"
+
+**Analysis**: "I see a new caching layer was implemented with a HashMap storing block hashes as keys and block data as values. The API endpoints now check this cache before making network requests. This should improve performance for repeated queries."
+
+**Changelog**: "Implemented new caching layer for blockchain queries, reducing API response time by storing frequently accessed block data in memory"
+
+#### ❌ BAD EXAMPLES (NO UNDERSTANDING):
+- "Enhanced error handling" (What specific errors? How were they enhanced?)
+- "Improved performance" (What was improved? How?)
+- "Updated transaction logic" (What specific logic? What changed?)
+
+## FINAL REMINDER:
+**PROCESS ONLY ONE RELEASE. THEN STOP AND WAIT FOR MY CONFIRMATION.**
+
+You must be thorough and verbose - if there are code changes, there should be substantial changelog content explaining what those changes accomplish.

.claude/prompts/readmes.md

@@ -0,0 +1,172 @@
+# README Generation Prompt
+
+Generate a professional, comprehensive README.md for each crate based SOLELY on code analysis. Use NO external documentation, commit messages, or existing READMEs.
+
+## MANDATORY PROCESS - FOLLOW EXACTLY:
+1. **IGNORE EXISTING DOCS**: Do NOT read any .md, .txt, .rst, or documentation files in the crate directory
+2. **CODE-ONLY ANALYSIS**: Examine ONLY these files:
+   - All .rs files in src/ directory and subdirectories
+   - Cargo.toml for dependencies and metadata
+   - Code structure and organization
+3. **MANDATORY CODE ANALYSIS**: Before writing ANY README content, you MUST:
+   - Examine all Rust files in src/ directory
+   - Identify the main structs, enums, traits, and functions
+   - Understand the crate's architecture and data flow
+   - Determine the crate's purpose from its implementation
+   - Map dependencies to understand external integrations
+4. **FRESH PERSPECTIVE**: Write the README as if you're the first person to document this crate
+5. Generate one complete README.md per crate
+6. Focus on one crate at a time for thorough analysis
+
+## ABSOLUTE REQUIREMENTS:
+- **SOURCE OF TRUTH**: Use ONLY the actual Rust code - no external docs, comments may provide hints but focus on implementation
+- **CRITICAL**: DO NOT read any existing README.md, CHANGELOG.md, or documentation files
+- **IGNORE ALL TEXT FILES**: .md, .txt, .rst files are FORBIDDEN sources - treat them as if they don't exist
+- **CODE ONLY**: Focus exclusively on .rs files, Cargo.toml, and code structure
+- **PROFESSIONAL GRADE**: Write as if this will be published on crates.io for other developers
+- **PROGRAMMER FOCUSED**: Assume audience knows Rust and relevant domain concepts
+- **IMPLEMENTATION-BASED**: Describe what the code actually does, not what comments claim it should do
+- **If you cannot determine functionality from code alone, state this explicitly**
+
+## README STRUCTURE (MANDATORY):
+
+### 1. CRATE HEADER
+```markdown
+# Crate Name
+
+Brief one-line description of what this crate does (max 80 chars).
+
+[![Crates.io](https://img.shields.io/crates/v/CRATE_NAME.svg)](https://crates.io/crates/CRATE_NAME)
+[![Documentation](https://docs.rs/CRATE_NAME/badge.svg)](https://docs.rs/CRATE_NAME)
+```
+
+### 2. OVERVIEW SECTION
+- **Purpose**: What problem does this crate solve?
+- **Key Features**: 3-5 bullet points of main capabilities (derived from code analysis)
+- **Target Use Cases**: Who would use this and for what?
+
+### 3. INSTALLATION
+```toml
+[dependencies]
+crate_name = "X.Y.Z"
+```
+
+### 4. QUICK START / USAGE
+- **Minimal working example** showing the primary API
+- **Common patterns** observed in the code
+- **Key structs/traits** that users will interact with
+
+### 5. API OVERVIEW
+- **Core Types**: Main structs, enums, traits with brief descriptions
+- **Key Methods**: Most important public functions
+- **Module Structure**: Brief overview of how code is organized
+
+### 6. FEATURES (if applicable)
+- Cargo features and what they enable
+- Optional dependencies and their purpose
+
+### 7. EXAMPLES
+- 2-3 practical code examples showing different use cases
+- Based on public API analysis, not existing examples
+
+## WRITING REQUIREMENTS:
+
+### TONE AND STYLE:
+- **Concise but comprehensive**: Every sentence must add value
+- **Technical precision**: Use exact terminology, avoid marketing speak
+- **Active voice**: "Provides X" not "X is provided"
+- **Present tense**: "The crate handles..." not "The crate will handle..."
+
+### FORBIDDEN PATTERNS:
+- **NEVER** use vague terms: "powerful", "flexible", "robust", "comprehensive", "advanced"
+- **NEVER** write marketing copy: "cutting-edge", "state-of-the-art", "enterprise-grade"
+- **NEVER** make claims you can't verify from code: "blazingly fast", "memory efficient"
+- **NEVER** copy-paste from existing documentation or comments
+- **NEVER** read or reference existing README.md files - pretend they don't exist
+- **NEVER** use phrases like "as mentioned in the documentation" or "according to the docs"
+- **NEVER** let existing documentation influence your analysis or writing
+
+### REQUIRED SPECIFICITY:
+- **Data structures**: Mention specific types (HashMap, Vec, etc.)
+- **Algorithms**: Reference actual implementations found in code
+- **Integration points**: Specific traits implemented, dependencies used
+- **Error handling**: How errors are represented and handled
+- **Async/sync**: Clearly state if operations are blocking or async
+
+### ANTI-BIAS PROTOCOL:
+
+### BEFORE STARTING ANY ANALYSIS:
+1. **Explicitly ignore**: Any README.md, CHANGELOG.md, docs/, documentation files
+2. **File filtering**: Only examine .rs and Cargo.toml files
+3. **Fresh eyes approach**: Analyze the code as if you've never seen this crate before
+4. **Independent thinking**: Form your own understanding purely from code inspection
+
+### IF YOU ACCIDENTALLY READ EXISTING DOCS:
+- Stop immediately and restart your analysis
+- Consciously disregard any information from documentation files
+- Base all descriptions solely on what you observe in the code
+- Ask yourself: "What would I think this code does if I had no documentation?"
+
+### VALIDATION CHECKS:
+- **Unique descriptions**: Your descriptions should differ significantly from any existing docs
+- **Code-derived insights**: Every feature mentioned must be visible in the source code
+- **Independent voice**: Write in your own technical style, not mimicking existing documentation
+- **Fresh examples**: Create new code examples based on API analysis, not existing samples
+
+## CODE ANALYSIS DEPTH:
+**You MUST analyze and understand:**
+1. **Public API surface**: All pub structs, functions, traits, modules
+2. **Core abstractions**: Main data types and their relationships
+3. **Error types**: Custom errors, Result patterns, panic conditions
+4. **Dependencies**: How external crates are integrated
+5. **Feature flags**: Conditional compilation and optional functionality
+6. **Async patterns**: Use of futures, tokio, async-std, etc.
+7. **Serialization**: Serde implementations, custom serialization
+8. **Performance characteristics**: Algorithm complexity where obvious
+
+### EXAMPLE STRUCTURE ANALYSIS OUTPUT:
+```markdown
+## Code Analysis Summary
+**Main Types**: `BlockProcessor`, `Transaction`, `ValidationError`
+**Core Trait**: `Validator` - implemented by `BasicValidator` and `StrictValidator`
+**Async Support**: All processing methods return `impl Future`
+**Error Handling**: Custom `ValidationError` enum with specific error types
+**Dependencies**: Uses `tokio` for async runtime, `serde` for serialization
+**Architecture**: Pipeline pattern with configurable validation stages
+```
+
+## EXAMPLES OF QUALITY:
+
+### ❌ BAD (VAGUE):
+```markdown
+# My Crate
+A powerful and flexible library for blockchain operations.
+
+## Features
+- Fast processing
+- Easy to use
+- Robust error handling
+```
+
+### ✅ GOOD (SPECIFIC):
+```markdown
+# brk-chain-analyzer
+Bitcoin blockchain analysis tools for transaction pattern detection.
+
+## Overview
+Provides utilities for analyzing Bitcoin transaction data, detecting address clustering patterns, and computing blockchain statistics. Built around a streaming parser that processes block data without loading entire blocks into memory.
+
+## Key Types
+- `TransactionAnalyzer`: Stateful analyzer for computing fees, detecting coinbase transactions
+- `ClusterDetector`: Implements common input ownership heuristics for address clustering
+- `BlockStream`: Async iterator over blockchain data with configurable batch sizes
+```
+
+## FINAL REQUIREMENTS:
+- **One README per crate** - don't combine multiple crates
+- **Minimum 200 words** - be thorough but concise
+- **Maximum 800 words** - stay focused and relevant
+- **Code examples must be syntactically correct** and compilable
+- **All claims must be verifiable** from the source code
+
+**PROCESS ONE CRATE AT A TIME. ANALYZE THE CODE THOROUGHLY BEFORE WRITING.**

.github/workflows/release.yml

@@ -64,7 +64,7 @@ jobs:
         # we specify bash to get pipefail; it guards against the `curl` command
         # failing. otherwise `sh` won't catch that `curl` returned non-0
         shell: bash
-        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.29.0/cargo-dist-installer.sh | sh"
+        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.30.0/cargo-dist-installer.sh | sh"
       - name: Cache dist
         uses: actions/upload-artifact@v4
         with:

.gitignore

@@ -4,7 +4,8 @@
 # Builds
 target
 websites/dist
-vecid-to-indexes.js
+bridge/
+/ids.txt
 
 # Copies
 *\ copy*
@@ -12,10 +13,6 @@ vecid-to-indexes.js
 # Ignored
 _*
 
-# Editors
-.vscode
-.zed
-
 # Logs
 .log
 
@@ -28,6 +25,4 @@ flamegraph.svg
 *.trace
 
 # AI
-.claude
-CLAUDE.md
-CLAUDE*.md
+.claude/settings*

.zed/settings.json

@@ -0,0 +1,20 @@
+{
+  "file_scan_exclusions": [
+    // default
+    "**/.git",
+    "**/.svn",
+    "**/.hg",
+    "**/.jj",
+    "**/CVS",
+    "**/.DS_Store",
+    "**/Thumbs.db",
+    "**/.classpath",
+    "**/.settings",
+    // custom
+    "**/lean-qr/*/index.mjs",
+    "uFuzzy.mjs",
+    "lightweight-charts.standalone.production.mjs",
+    "**/modern-screenshot/*/index.mjs",
+    "**/solidjs-signals/*/dist/prod.js"
+  ]
+}

Cargo.toml

@@ -4,16 +4,24 @@ members = ["crates/*"]
 package.description = "The Bitcoin Research Kit is a suite of tools designed to extract, compute and display data stored on a Bitcoin Core node"
 package.license = "MIT"
 package.edition = "2024"
-package.version = "0.0.91"
+package.version = "0.0.109"
 package.homepage = "https://bitcoinresearchkit.org"
 package.repository = "https://github.com/bitcoinresearchkit/brk"
 package.readme = "README.md"
 package.rust-version = "1.89"
 
+[profile.dev]
+lto = "thin"
+codegen-units = 16
+opt-level = 2
+split-debuginfo = "unpacked"
+
 [profile.release]
 lto = "fat"
 codegen-units = 1
 panic = "abort"
+strip = true
+overflow-checks = false
 
 [profile.profiling]
 inherits = "release"
@@ -22,40 +30,53 @@ debug = true
 [profile.dist]
 inherits = "release"
 
+[profile.clippy]
+inherits = "dev"
+lto = "off"
+codegen-units = 256
+opt-level = 0
+debug = false
+overflow-checks = false
+panic = "abort"
+debug-assertions = false
+
 [workspace.dependencies]
+allocative = { version = "0.3.4", features = ["parking_lot"] }
 axum = "0.8.4"
 bitcoin = { version = "0.32.7", features = ["serde"] }
 bitcoincore-rpc = "0.19.0"
-brk_bundler = { version = "0.0.91", path = "crates/brk_bundler" }
-brk_cli = { version = "0.0.91", path = "crates/brk_cli" }
-brk_computer = { version = "0.0.91", path = "crates/brk_computer" }
-brk_error = { version = "0.0.91", path = "crates/brk_error" }
-brk_fetcher = { version = "0.0.91", path = "crates/brk_fetcher" }
-brk_indexer = { version = "0.0.91", path = "crates/brk_indexer" }
-brk_interface = { version = "0.0.91", path = "crates/brk_interface" }
-brk_logger = { version = "0.0.91", path = "crates/brk_logger" }
-brk_mcp = { version = "0.0.91", path = "crates/brk_mcp" }
-brk_parser = { version = "0.0.91", path = "crates/brk_parser" }
-brk_server = { version = "0.0.91", path = "crates/brk_server" }
-brk_store = { version = "0.0.91", path = "crates/brk_store" }
-brk_structs = { version = "0.0.91", path = "crates/brk_structs" }
+brk_bundler = { version = "0.0.109", path = "crates/brk_bundler" }
+brk_cli = { version = "0.0.109", path = "crates/brk_cli" }
+brk_computer = { version = "0.0.109", path = "crates/brk_computer" }
+brk_error = { version = "0.0.109", path = "crates/brk_error" }
+brk_fetcher = { version = "0.0.109", path = "crates/brk_fetcher" }
+brk_indexer = { version = "0.0.109", path = "crates/brk_indexer" }
+brk_interface = { version = "0.0.109", path = "crates/brk_interface" }
+brk_logger = { version = "0.0.109", path = "crates/brk_logger" }
+brk_mcp = { version = "0.0.109", path = "crates/brk_mcp" }
+brk_parser = { version = "0.0.109", path = "crates/brk_parser" }
+brk_server = { version = "0.0.109", path = "crates/brk_server" }
+brk_store = { version = "0.0.109", path = "crates/brk_store" }
+brk_structs = { version = "0.0.109", path = "crates/brk_structs" }
 byteview = "=0.6.1"
 derive_deref = "1.1.1"
 fjall = "2.11.2"
 jiff = "0.2.15"
-log = "0.4.27"
-minreq = { version = "2.14.0", features = ["https", "serde_json"] }
+log = "0.4.28"
+minreq = { version = "2.14.1", features = ["https", "serde_json"] }
 parking_lot = "0.12.4"
+quick_cache = "0.6.16"
 rayon = "1.11.0"
-serde = "1.0.219"
-serde_bytes = "0.11.17"
-serde_derive = "1.0.219"
-serde_json = { version = "1.0.143", features = ["float_roundtrip"] }
+serde = "1.0.225"
+serde_bytes = "0.11.19"
+serde_derive = "1.0.225"
+serde_json = { version = "1.0.145", features = ["float_roundtrip"] }
+sonic-rs = "0.5.4"
 tokio = { version = "1.47.1", features = ["rt-multi-thread"] }
 # vecdb = { path = "../seqdb/crates/vecdb", features = ["derive"]}
-vecdb = { version = "0.2.5", features = ["derive"]}
-zerocopy = "0.8.26"
-zerocopy-derive = "0.8.26"
+vecdb = { version = "0.2.16", features = ["derive"]}
+zerocopy = "0.8.27"
+zerocopy-derive = "0.8.27"
 
 [workspace.metadata.release]
 shared-version = true
@@ -64,7 +85,7 @@ pre-release-commit-message = "release: v{{version}}"
 tag-message = "release: v{{version}}"
 
 [workspace.metadata.dist]
-cargo-dist-version = "0.29.0"
+cargo-dist-version = "0.30.0"
 ci = "github"
 allow-dirty = ["ci"]
 installers = []

README.md

@@ -1,99 +0,0 @@
-# Bitcoin Research Kit
-
-<p align="left">
-  <a href="https://github.com/bitcoinresearchkit/brk">
-    <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
-  </a>
-  <a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
-    <img src="https://img.shields.io/crates/l/brk" alt="License" />
-  </a>
-  <a href="https://crates.io/crates/brk">
-    <img src="https://img.shields.io/crates/v/brk" alt="Version" />
-  </a>
-  <a href="https://docs.rs/brk">
-    <img src="https://img.shields.io/docsrs/brk" alt="Documentation" />
-  </a>
-  <img src="https://img.shields.io/crates/size/brk" alt="Size" />
-  <a href="https://deps.rs/crate/brk">
-    <img src="https://deps.rs/crate/brk/latest/status.svg" alt="Dependency status">
-  </a>
-  <a href="https://discord.gg/HaR3wpH3nr">
-    <img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
-  </a>
-  <a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
-    <img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
-  </a>
-</p>
-
-The Bitcoin Research Kit is a high-performance toolchain designed to parse, index, compute, serve and visualize data from a Bitcoin node, enabling users to gain deeper insights into the Bitcoin network.
-
-In other words it's an alternative to [Glassnode](https://glassnode.com), [mempool.space](https://mempool.space/) (soon) and [electrs](https://github.com/romanz/electrs) (soon) all in one package with a particular focus on simplicity and ease of use.
-
-The toolkit can be used in various ways to accommodate as many needs as possible:
-
-- **[Website](https://bitcoinresearchkit.org)** \
-  Everyone is welcome to visit the official instance and showcase of the suite's capabilities. \
-  It has a wide range of functionalities including charts, tables and simulations which you can visit for free and without the need for an account. \
-  Also available at: [brekit.org](https://brekit.org) // [kibo.money](https://kibo.money) // [satonomics.xyz](https://satonomics.xyz)
-- **[API](https://github.com/bitcoinresearchkit/brk/tree/main/crates/brk_server#brk-server)** \
-  Researchers and developers are free to use BRK's public API with ![Datasets variant count](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fbitcoinresearchkit.org%2Fapi%2Fvecs%2Fvec-count&query=%24&style=flat&label=%20&color=white) dataset variants at their disposal. \
-  Just like the website, it's entirely free, with no authentication or rate-limiting.
-- **[AI](https://github.com/bitcoinresearchkit/brk/blob/main/crates/brk_mcp/README.md#brk-mcp)** \
-  LLMs have to possibility to connect to BRK's backend through a [MCP](https://modelcontextprotocol.io/introduction). \
-  It will give them access to the same tools as the API, with no restrictions, and allow you to have your very own data analysts.
-- **[CLI](https://crates.io/crates/brk_cli)** \
-  Node runners are strongly encouraged to try out and self-host their own instance using BRK's command line interface. \
-  The CLI has multiple cogs available for users to tweak to adapt to all situations with even the possibility for web developers to create their own custom website which could later on be added as an alternative front-end.
-- **[Crates](https://crates.io/crates/brk)** \
-  Rust developers have access to a wide range crates, each built upon one another with its own specific purpose, enabling independent use and offering great flexibility.
-  PRs are welcome, especially if their goal is to introduce additional datasets.
-
-The primary goal of this project is to be fully-featured and accessible for everyone, regardless of their background or financial situation - whether that person is an enthusiast, researcher, miner, analyst, or simply curious.
-
-In contrast, existing alternatives tend to be either [very costly](https://studio.glassnode.com/pricing) or missing essential features, with the vast majority being closed-source and unverifiable, which fundamentally undermines the principles of Bitcoin.
-
-## Hosting as a service
-
-If you'd like to have your own instance hosted for you please contact [hosting@bitcoinresearchkit.org](mailto:hosting@bitcoinresearchkit.org).
-
-- 2 separate dedicated servers (1 GB/s each) with different ISPs and Cloudflare integration for enhanced performance and optimal availability
-- 99.99% SLA
-- Configured for speed
-- Updates delivered at your convenience
-- Direct communication for feature requests and support
-- Bitcoin Core or Knots with desired version
-- Optional subdomains: `*.bitcoinresearchkit.org`, `*.brekit.org`, `*.kibo.money` and `*.satonomics.xyz`
-- Logo featured in the Readme if desired
-
-Pricing: `0.01 BTC / month` *or* `0.1 BTC / year`
-
-## Acknowledgments
-
-Deepest gratitude to the [Open Sats](https://opensats.org/) public charity. Their grant — from December 2024 to the present — has been critical in sustaining this project.
-
-Heartfelt thanks go out to every donor on [Nostr](https://primal.net/p/npub1jagmm3x39lmwfnrtvxcs9ac7g300y3dusv9lgzhk2e4x5frpxlrqa73v44) and [Geyser.fund](https://geyser.fund/project/brk) whose support has ensured the availability of the [bitcoinresearchkit.org](https://bitcoinresearchkit.org) public instance.
-
-## Crates
-
-- [`brk`](https://crates.io/crates/brk): A wrapper around all other `brk-*` crates
-- [`brk_bundler`](https://crates.io/crates/brk_bundler): A thin wrapper around [`rolldown`](https://rolldown.rs/)
-- [`brk_cli`](https://crates.io/crates/brk_cli): A command line interface to run a BRK instance
-- [`brk_computer`](https://crates.io/crates/brk_computer): A Bitcoin dataset computer built on top of [`brk_indexer`](https://crates.io/crates/brk_indexer)
-- [`brk_error`](https://crates.io/crates/brk_error): Errors used throughout BRK
-- [`brk_fetcher`](https://crates.io/crates/brk_fetcher): A Bitcoin price fetcher
-- [`brk_indexer`](https://crates.io/crates/brk_indexer): A Bitcoin indexer built on top of [`brk_parser`](https://crates.io/crates/brk_parser)
-- [`brk_interface`](https://crates.io/crates/brk_interface): An interface to find and format data from BRK
-- [`brk_logger`](https://crates.io/crates/brk_logger): A thin wrapper around [`env_logger`](https://crates.io/crates/env_logger)
-- [`brk_mcp`](https://crates.io/crates/brk_mcp): A bridge for LLMs to access BRK
-- [`brk_parser`](https://crates.io/crates/brk_parser): A very fast Bitcoin block parser and iterator built on top of [`bitcoin-rust`](https://crates.io/crates/bitcoin)
-- [`brk_server`](https://crates.io/crates/brk_server): A server with an API for anything from BRK
-- [`brk_store`](https://crates.io/crates/brk_store): A thin wrapper around [`fjall`](https://crates.io/crates/fjall)
-- [`brk_structs`](https://crates.io/crates/brk_structs): Structs used throughout BRK
-
-## Donate
-
-[`bc1q09 8zsm89 m7kgyz e338vf ejhpdt 92ua9p 3peuve`](bitcoin:bc1q098zsm89m7kgyze338vfejhpdt92ua9p3peuve)
-
-[`lnurl1dp68gurn8ghj7ampd3kx2ar0veekzar0wd5xjtnrdakj7tnhv4kxctttdehhwm30d3h82unvwqhkxmmww3jkuar8d35kgetj8yuq363hv4`](lightning:lnurl1dp68gurn8ghj7ampd3kx2ar0veekzar0wd5xjtnrdakj7tnhv4kxctttdehhwm30d3h82unvwqhkxmmww3jkuar8d35kgetj8yuq363hv4)
-
-[Geyser Fund](https://geyser.fund/project/brk)

TODO.md

@@ -1,99 +0,0 @@
-# TODO
-
-- __crates__
-  - _cli_
-    - check disk space on first launch
-    - add custom path support for config.toml
-    - maybe add bitcoind download and launch support
-      - via: https://github.com/rust-bitcoin/corepc/blob/master/node
-    - test read/write speed, add warning if too low (<2gb/s)
-    - pull latest version and notify is out of date
-  - _computer_
-    - **add rollback of states (in stateful)**
-    - remove configurable format (raw/compressed) and chose sane ones instead
-      - linear reads: compressed (height/date/... + txindex_to_height + txindex_to_version + ...)
-      - random reads: raw (outputindex_to_value + ...)
-    - add costs basis by percentile (percentile cost basis) back
-    - add support for per index computation
-    - fix min feerate which is always ZERO due to coinbase transaction
-    - before computing multiple sources check their length, panic if not equal
-    - add oracle price dataset (https://utxo.live/oracle/UTXOracle.py)
-    - add address counts relative to all datasets
-    - make decade, quarter, year datasets `computed` instead of `eager`
-    - add 6 months (semester) interval datasets to builder
-    - some datasets in `indexes` can probably be removed
-    - add revived/sent supply datasets
-    - add `in-sats` version of all price datasets (average and co)
-    - add `p2pk` group (sum of `p2pk33` and `p2pk65`)
-    - add chopiness datasets
-    - add utxo count, address count, supply data for  by reused addresses in groups by address type
-    - add more date ranges (3-6 months and more)
-    - add puell multiple dataset
-    - add pi cycle dataset
-    - add emas of price
-    - add 7d and 30d ema to sell side risk ratio and sopr
-    - don't compute everything for all cohorts as some datasets combinations are irrelevant
-      - addresses/utxos by amount don't need mvrvz for example
-    - add all possible charts from:
-      - https://mainnet.observer
-      - https://glassnode.com
-      - https://checkonchain.com
-      - https://researchbitcoin.net/exciting-update-coming-to-the-bitcoin-lab/
-      - https://mempool.space/research
-  - _indexer_
-    - parse only the needed block number
-      - maybe using https://developer.bitcoin.org/reference/rpc/getblockhash.html
-  - _interface_
-    - create pagination enum
-      - from to
-      - from option<count>
-      - to option<count>
-      - page + option<per page> default 1000 max 1000
-    - from/to/count params don’t cap all combinations
-      - example: from -10,000 count 10, won’t work if underlying vec isn’t 10k or more long
-  - _parser_
-    - save `vec` file instead of `json`
-    - support lock file, process in read only if already opened in write mode
-    - if less than X (10 maybe ?) get block using rpc instead of parsing the block files
-  - _server_
-    - api
-      - add extensions support (.json .csv …)
-      - if format instead of extension then don't download file
-    - add support for https (rustls)
-- __docs__
-  - _README_
-    - add a comparison table with alternatives
-    - add contribution section where help is needed
-      - documentation/mcp/datasets/different front ends
-    - add faq
-- __websites__
-  - _default_
-    - explorer
-      - blocks
-      - transactions
-      - addresses
-      - miners
-      - maybe xpubs
-    - charts
-      - improve some names and colors
-      - remove `sum` series when it's a duplicate of the `base` (in subsidy for example)
-      - selected unit sometimes changes when going back end forth
-      - add support for custom charts
-      - separate z-score charts from "realized price" (with their own prices), have 4y, 2y and 1y
-      - price scale format depends on unit, hide digits for sats for example (if/when possible)
-    - table
-      - pagination
-      - exports (.json, .csv,…)
-    - search
-      - datasets add legend, and keywords ?
-      - height/address/txid
-    - api
-      - add api page with interactivity
-    - global
-      - **fix navigation/history**
-      - move share button to footer ?
-      - Use `ichart.createPane()` in wrapper
-      - improve behavior when local storage is unavailable
-        - by having a global state
-- __global__
-  - check `TODO`s in codebase

crates/brk/Cargo.toml

@@ -2,7 +2,6 @@
 name = "brk"
 description.workspace = true
 license.workspace = true
-readme.workspace = true
 homepage.workspace = true
 repository.workspace = true
 edition.workspace = true

crates/brk/README.md

@@ -1,197 +1,273 @@
 # brk
 
-**Main wrapper crate for the Bitcoin Research Kit (BRK)**
+Unified Bitcoin Research Kit crate providing optional feature-gated access to all BRK components.
 
-The `brk` crate serves as the primary entry point for the Bitcoin Research Kit, providing a unified interface to all BRK components through feature flags. It enables developers to selectively include only the components they need while maintaining a consistent API.
+[![Crates.io](https://img.shields.io/crates/v/brk.svg)](https://crates.io/crates/brk)
+[![Documentation](https://docs.rs/brk/badge.svg)](https://docs.rs/brk)
 
-## What it provides
+## Overview
 
-- **Unified Access**: Single crate providing access to the entire BRK ecosystem
-- **Feature-based Selection**: Choose only the components you need
-- **Consistent Versioning**: All components versioned together for compatibility
-- **Simplified Dependencies**: Single dependency instead of multiple individual crates
+This crate serves as a unified entry point to the Bitcoin Research Kit ecosystem, providing feature-gated re-exports of all BRK components. It allows users to selectively include only the functionality they need while maintaining a single dependency declaration, with the `brk_cli` component always available for command-line interface access.
 
-## Available Components
+**Key Features:**
 
-### Core Data Pipeline
-- **`parser`** ([brk_parser](../brk_parser/)) - High-performance Bitcoin block parser
-- **`indexer`** ([brk_indexer](../brk_indexer/)) - Blockchain data indexer with dual storage
-- **`computer`** ([brk_computer](../brk_computer/)) - Analytics engine for computed datasets
-- **`interface`** ([brk_interface](../brk_interface/)) - Unified data query and formatting API
+- Feature-gated modular access to 12 specialized BRK components
+- Single dependency entry point with selective compilation
+- Always-available CLI component for command-line operations
+- Comprehensive documentation aggregation with inline re-exports
+- `full` feature for complete BRK functionality inclusion
+- Optimized build configuration with docs.rs integration
 
-### Infrastructure
-- **`structs`** ([brk_structs](../brk_structs/)) - Bitcoin-aware type system and data structures
-- **`error`** ([brk_error](../brk_error/)) - Centralized error handling
-- **`store`** ([brk_store](../brk_store/)) - Blockchain-aware key-value storage
+**Target Use Cases:**
 
-### External Integration
-- **`fetcher`** ([brk_fetcher](../brk_fetcher/)) - Bitcoin price data fetcher with multi-source fallback
-- **`server`** ([brk_server](../brk_server/)) - HTTP server with REST API
-- **`mcp`** ([brk_mcp](../brk_mcp/)) - Model Context Protocol for LLM integration
+- Applications requiring selective BRK functionality to minimize dependencies
+- Library development where only specific Bitcoin analysis components are needed
+- Prototyping and experimentation with different BRK component combinations
+- Educational use cases demonstrating modular blockchain analytics architecture
 
-### Utilities
-- **`cli`** ([brk_cli](../brk_cli/)) - Command line interface (always enabled)
-- **`logger`** ([brk_logger](../brk_logger/)) - Logging utilities
-- **`bundler`** ([brk_bundler](../brk_bundler/)) - Asset bundling for web interfaces
-
-## Usage
-
-### Full Installation
+## Installation
 
 ```toml
 [dependencies]
-brk = { version = "0.0.88", features = ["full"] }
+# Minimal installation with CLI only
+brk = "0.0.107"
+
+# Full functionality
+brk = { version = "0.0.107", features = ["full"] }
+
+# Selective features
+brk = { version = "0.0.107", features = ["indexer", "computer", "server"] }
 ```
 
+## Quick Start
+
 ```rust
-use brk::*;
+// CLI is always available
+use brk::cli;
 
-// Access all components
-let config = cli::Config::load()?;
-let parser = parser::Parser::new(/* ... */);
-let indexer = indexer::Indexer::forced_import("./data")?;
-let computer = computer::Computer::forced_import("./data", &indexer, None)?;
-```
+// Feature-gated components
+#[cfg(feature = "indexer")]
+use brk::indexer::Indexer;
 
-### Selective Components
+#[cfg(feature = "computer")]
+use brk::computer::Computer;
 
-```toml
-[dependencies]
-brk = { version = "0.0.88", features = ["parser", "indexer", "computer"] }
-```
+#[cfg(feature = "server")]
+use brk::server::Server;
 
-```rust
-use brk::{parser, indexer, computer};
-
-// Core data pipeline only
-let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
-let mut indexer = indexer::Indexer::forced_import(output_dir)?;
-let mut computer = computer::Computer::forced_import(output_dir, &indexer, None)?;
-```
-
-### Minimal Setup
-
-```toml
-[dependencies]
-brk = { version = "0.0.88", features = ["structs", "parser"] }
-```
-
-```rust
-use brk::{structs, parser};
-
-// Just parsing and types
-let height = structs::Height::new(800_000);
-let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
-```
-
-## Feature Flags
-
-| Feature | Description | Dependencies |
-|---------|-------------|--------------|
-| `full` | Enable all components | All crates |
-| `cli` | Command line interface | Always enabled |
-| `structs` | Core type system | Foundation for other crates |
-| `error` | Error handling | Used by most crates |
-| `parser` | Block parsing | `structs`, `error` |
-| `store` | Key-value storage | `structs`, `error` |
-| `indexer` | Blockchain indexing | `parser`, `store` |
-| `computer` | Analytics computation | `indexer`, `fetcher` (optional) |
-| `fetcher` | Price data fetching | `structs`, `error` |
-| `interface` | Data query API | `indexer`, `computer` |
-| `server` | HTTP server | `interface`, `mcp` |
-| `mcp` | LLM integration | `interface` |
-| `logger` | Logging utilities | Standalone |
-| `bundler` | Asset bundling | Standalone |
-
-## Common Usage Patterns
-
-### Complete BRK Instance
-
-```rust
-use brk::*;
-
-// Full data pipeline setup
-let config = cli::Config::load()?;
-let rpc = /* Bitcoin Core RPC client */;
-let parser = parser::Parser::new(config.blocks_dir, config.output_dir, rpc);
-let mut indexer = indexer::Indexer::forced_import(&config.output_dir)?;
-let mut computer = computer::Computer::forced_import(&config.output_dir, &indexer, None)?;
-let interface = interface::Interface::build(&indexer, &computer);
-let server = server::Server::new(interface, config.website_path);
-
-// Start server
-server.serve(true).await?;
-```
-
-### Data Analysis
-
-```rust
-use brk::{indexer, computer, interface};
-
-// Analysis-focused setup
-let indexer = indexer::Indexer::forced_import("./brk_data")?;
-let computer = computer::Computer::forced_import("./brk_data", &indexer, None)?;
-let interface = interface::Interface::build(&indexer, &computer);
-
-// Query data
-let params = interface::Params {
-    index: interface::Index::Height,
-    ids: vec!["price_usd", "difficulty"].into(),
-    rest: interface::ParamsOpt::default()
-        .set_from(-100)
-        .set_format(interface::Format::CSV),
-};
-
-let csv_data = interface.search_and_format(params)?;
-```
-
-### Custom Integration
-
-```rust
-use brk::{structs, parser, error};
-
-// Custom application with BRK components
-fn analyze_blocks() -> error::Result<()> {
-    let parser = parser::Parser::new(blocks_dir, output_dir, rpc);
-    
-    parser.parse(None, None)
-        .iter()
-        .take(1000)  // First 1000 blocks
-        .for_each(|(height, block, hash)| {
-            println!("Block {}: {} transactions", height, block.txdata.len());
-        });
-    
+// Build complete pipeline with selected features
+#[cfg(all(feature = "indexer", feature = "computer", feature = "server"))]
+fn build_pipeline() -> Result<(), Box<dyn std::error::Error>> {
+    let indexer = Indexer::build("./data")?;
+    let computer = Computer::build("./analytics", &indexer)?;
+    let interface = brk::interface::Interface::build(&indexer, &computer);
+    let server = Server::new(interface, None);
     Ok(())
 }
 ```
 
-## Version Compatibility
+## API Overview
 
-All BRK crates are released together with synchronized versions. When using the `brk` wrapper crate, you're guaranteed compatibility between all components.
+### Feature Organization
 
-- **Current version**: 0.0.88
-- **Rust MSRV**: 1.89+
-- **Bitcoin Core**: v25.0 - v29.0
+The crate provides feature-gated access to BRK components organized by functionality:
 
-## Performance Characteristics
+**Core Data Processing:**
+- `structs` - Bitcoin-aware data structures and type system
+- `error` - Centralized error handling across components
+- `store` - Transactional key-value storage wrapper
 
-The `brk` crate itself adds no runtime overhead - it simply re-exports the underlying crates. Performance characteristics depend on which components you use:
+**Blockchain Processing:**
+- `parser` - Multi-threaded Bitcoin block parsing
+- `indexer` - Blockchain data indexing with columnar storage
+- `computer` - Analytics computation engine
 
-- **Full pipeline**: ~13-15 hours initial sync, ~40GB storage overhead
-- **Parser only**: ~4 minutes to parse entire blockchain
-- **Indexer only**: ~7-8 hours to index blockchain, ~5-6GB RAM usage
-- **Server**: Low latency API responses with caching and compression
+**Data Access:**
+- `interface` - Unified query interface with fuzzy search
+- `fetcher` - Multi-source price data aggregation
 
-## Dependencies
+**Service Layer:**
+- `server` - HTTP API server with caching and compression
+- `mcp` - Model Context Protocol bridge for LLM integration
+- `logger` - Enhanced logging with colored output
 
-The `brk` crate's dependencies are determined by enabled features. Core dependencies include:
+**Web Infrastructure:**
+- `bundler` - Web asset bundling using Rolldown
 
-- `brk_cli` - Always included for configuration and CLI support
-- Individual `brk_*` crates based on enabled features
-- Transitive dependencies from enabled components
+### Always Available
 
-For specific dependency information, see individual crate READMEs.
+**`cli`**: Command-line interface module (no feature gate required)
+Provides access to the complete BRK command-line interface for running full instances.
+
+## Examples
+
+### Minimal Bitcoin Parser
+
+```rust
+use brk::parser::Parser;
+
+fn parse_blocks() -> Result<(), Box<dyn std::error::Error>> {
+    #[cfg(feature = "parser")]
+    {
+        let parser = Parser::new("/path/to/blocks", None, rpc_client);
+        // Parse blockchain data
+        Ok(())
+    }
+    #[cfg(not(feature = "parser"))]
+    {
+        Err("Parser feature not enabled".into())
+    }
+}
+```
+
+### Analytics Pipeline
+
+```rust
+use brk::{indexer, computer, interface};
+
+#[cfg(all(feature = "indexer", feature = "computer", feature = "interface"))]
+fn analytics_pipeline() -> Result<(), Box<dyn std::error::Error>> {
+    // Initialize indexer
+    let indexer = indexer::Indexer::build("./blockchain_data")?;
+
+    // Compute analytics
+    let computer = computer::Computer::build("./analytics", &indexer)?;
+
+    // Create query interface
+    let interface = interface::Interface::build(&indexer, &computer);
+
+    // Query latest price
+    let params = interface::Params {
+        index: "date".to_string(),
+        ids: vec!["price-close".to_string()].into(),
+        from: Some(-1),
+        ..Default::default()
+    };
+
+    let result = interface.search_and_format(params)?;
+    println!("Latest Bitcoin price: {:?}", result);
+
+    Ok(())
+}
+```
+
+### Web Server Setup
+
+```rust
+use brk::{server, interface, indexer, computer};
+
+#[cfg(all(feature = "server", feature = "interface", feature = "indexer", feature = "computer"))]
+async fn web_server() -> Result<(), Box<dyn std::error::Error>> {
+    let indexer = indexer::Indexer::build("./data")?;
+    let computer = computer::Computer::build("./analytics", &indexer)?;
+    let interface = interface::Interface::build(&indexer, &computer);
+
+    let server = server::Server::new(interface, None);
+    server.serve(true).await?;
+
+    Ok(())
+}
+```
+
+### MCP Integration
+
+```rust
+use brk::{mcp, interface, indexer, computer};
+
+#[cfg(all(feature = "mcp", feature = "interface"))]
+fn mcp_server(interface: &'static interface::Interface) -> mcp::MCP {
+    mcp::MCP::new(interface)
+}
+```
+
+## Feature Combinations
+
+### Common Combinations
+
+**Data Processing**: `["structs", "parser", "indexer"]`
+Basic blockchain data processing and indexing.
+
+**Analytics**: `["indexer", "computer", "fetcher", "interface"]`
+Complete analytics pipeline with price data integration.
+
+**API Server**: `["interface", "server", "logger"]`
+HTTP API server with logging capabilities.
+
+**Full Stack**: `["full"]`
+All components for complete BRK functionality.
+
+### Dependency Optimization
+
+Feature selection allows for significant dependency reduction:
+
+```toml
+# Minimal parser-only dependency
+brk = { version = "0.0.107", features = ["parser"] }
+
+# Analytics without web server
+brk = { version = "0.0.107", features = ["indexer", "computer", "interface"] }
+
+# Web server without parsing
+brk = { version = "0.0.107", features = ["interface", "server"] }
+```
+
+## Architecture
+
+### Re-export Pattern
+
+The crate uses `#[doc(inline)]` re-exports to provide seamless access to component APIs:
+
+```rust
+#[cfg(feature = "component")]
+#[doc(inline)]
+pub use brk_component as component;
+```
+
+This pattern ensures:
+- Feature-gated compilation for dependency optimization
+- Inline documentation for unified API reference
+- Namespace preservation for component-specific functionality
+
+### Build Configuration
+
+- **Documentation**: `all-features = true` for complete docs.rs documentation
+- **CLI Integration**: `brk_cli` always available without feature gates
+- **Optional Dependencies**: All components except CLI are optional
+
+## Configuration
+
+### Feature Flags
+
+| Feature | Component | Description |
+|---------|-----------|-------------|
+| `bundler` | `brk_bundler` | Web asset bundling |
+| `computer` | `brk_computer` | Analytics computation |
+| `error` | `brk_error` | Error handling |
+| `fetcher` | `brk_fetcher` | Price data fetching |
+| `indexer` | `brk_indexer` | Blockchain indexing |
+| `interface` | `brk_interface` | Data query interface |
+| `logger` | `brk_logger` | Enhanced logging |
+| `mcp` | `brk_mcp` | Model Context Protocol |
+| `parser` | `brk_parser` | Block parsing |
+| `server` | `brk_server` | HTTP server |
+| `store` | `brk_store` | Key-value storage |
+| `structs` | `brk_structs` | Data structures |
+| `full` | All components | Complete functionality |
+
+### Documentation
+
+Documentation is aggregated from all components with `#![doc = include_str!("../README.md")]` ensuring comprehensive API reference across all features.
+
+## Code Analysis Summary
+
+**Main Structure**: Feature-gated re-export crate providing unified access to 12 BRK components \
+**Feature System**: Cargo features enabling selective compilation and dependency optimization \
+**CLI Integration**: Always-available `brk_cli` access without feature requirements \
+**Documentation**: Inline re-exports with comprehensive docs.rs integration \
+**Dependency Management**: Optional dependencies for all components except CLI \
+**Build Configuration**: Optimized compilation with all-features documentation \
+**Architecture**: Modular aggregation crate enabling flexible BRK ecosystem usage
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_bundler/Cargo.toml

@@ -12,7 +12,7 @@ build = "build.rs"
 [dependencies]
 log = { workspace = true }
 notify = "8.2.0"
-brk_rolldown = "0.1.4"
+brk_rolldown = "0.1.5"
 # brk_rolldown = { path = "../../../rolldown/crates/rolldown"}
 sugar_path = "1.2.0"
 tokio = { workspace = true }

crates/brk_bundler/README.md

@@ -1,186 +1,278 @@
 # brk_bundler
 
-**Asset bundling for BRK web interfaces using Rolldown**
+Asset bundling and development server for BRK web interfaces with hot reloading and file watching.
 
-`brk_bundler` provides JavaScript/TypeScript bundling capabilities for BRK's web interfaces. It's a thin wrapper around Rolldown (Rust-based Rollup alternative) with BRK-specific configuration for building optimized web assets with file watching and automatic rebuilding.
+[![Crates.io](https://img.shields.io/crates/v/brk_bundler.svg)](https://crates.io/crates/brk_bundler)
+[![Documentation](https://docs.rs/brk_bundler/badge.svg)](https://docs.rs/brk_bundler)
 
-## What it provides
+## Overview
 
-- **JavaScript Bundling**: Modern ES modules bundling with minification
-- **File Watching**: Automatic rebuilding on source file changes
-- **Asset Processing**: Copies and processes static assets
-- **Version Injection**: Automatic version string replacement in service workers
-- **Development Mode**: Live rebuilding for rapid development
+This crate provides a thin wrapper around the Rolldown JavaScript bundler specifically designed for BRK web interface development. It handles asset bundling, file copying, template processing, and development-mode file watching with automatic rebuilds and hot reloading for efficient web development workflows.
 
-## Key Features
+**Key Features:**
 
-### Bundling Capabilities
-- **ES Module Support**: Modern JavaScript bundling with tree-shaking
-- **Minification**: Automatic code minification for production builds
-- **Source Maps**: Generated source maps for debugging
-- **Entry Point Processing**: Configurable entry points with hashed output names
+- JavaScript bundling with Rolldown (Rust-based bundler)
+- Automatic file watching and hot reloading in development mode
+- Template processing with version injection and asset hash replacement
+- Service worker generation with version management
+- Source map generation for debugging
+- Minification for production builds
+- Async/await support with Tokio integration
 
-### File System Operations
-- **Directory Copying**: Copies entire source directories to distribution
-- **Selective Processing**: Special handling for specific file types
-- **Path Resolution**: Automatic path resolution and asset linking
+**Target Use Cases:**
 
-### Development Features
-- **Hot Rebuilding**: Automatic rebuilds on file changes
-- **Watch Mode**: Monitors source files and triggers rebuilds
-- **Version Replacement**: Injects build version into service workers
+- BRK blockchain explorer web interfaces
+- Development of Bitcoin analytics dashboards
+- Building responsive web applications for blockchain data visualization
+- Hot reloading development environment for rapid iteration
 
-## Usage
+## Installation
 
-### Basic Bundling
+```bash
+cargo add brk_bundler
+```
+
+## Quick Start
 
 ```rust
 use brk_bundler::bundle;
 use std::path::Path;
 
-// Bundle without watching (production)
-let websites_path = Path::new("./websites");
-let source_folder = "default";
-let dist_path = bundle(websites_path, source_folder, false).await?;
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let websites_path = Path::new("./web");
+    let source_folder = "src";
+    let watch = true; // Enable hot reloading
 
-println!("Bundled to: {:?}", dist_path);
-```
+    // Bundle assets and start development server
+    let dist_path = bundle(websites_path, source_folder, watch).await?;
 
-### Development Mode with Watching
+    println!("Assets bundled to: {}", dist_path.display());
 
-```rust
-// Bundle with file watching (development)
-let dist_path = bundle(websites_path, "default", true).await?;
+    // Keep running for file watching (in watch mode)
+    if watch {
+        tokio::signal::ctrl_c().await?;
+    }
 
-// Bundler now watches for changes and rebuilds automatically
-// This will run in the background until the process exits
-```
-
-### Integration with BRK CLI
-
-```rust
-// Typically called from brk_cli when serving websites
-async fn setup_website(config: &Config) -> Result<PathBuf> {
-    let websites_path = config.websites_path();
-    let source_folder = match config.website_mode {
-        WebsiteMode::Default => "default",
-        WebsiteMode::Custom => "custom",
-        WebsiteMode::None => return Ok(PathBuf::new()),
-    };
-    
-    // Bundle the website assets
-    let dist_path = bundle(websites_path, source_folder, config.dev_mode).await?;
-    
-    Ok(dist_path)
+    Ok(())
 }
 ```
 
-## File Structure
+## API Overview
 
-The bundler expects this directory structure:
+### Core Functions
 
-```
-websites/
-├── default/                 # Default website source
-│   ├── index.html          # Main HTML file
-│   ├── service-worker.js   # Service worker (version injected)
-│   ├── scripts/            # JavaScript/TypeScript source
-│   │   ├── entry.js        # Main entry point
-│   │   ├── main.js         # Application logic
-│   │   └── ...             # Other JS modules
-│   └── assets/             # Static assets
-└── dist/                   # Generated output directory
-    ├── index.html          # Processed HTML with updated script references
-    ├── service-worker.js   # Service worker with version injected
-    ├── scripts/            # Bundled and minified JavaScript
-    │   └── main-[hash].js  # Hashed output file
-    └── assets/             # Copied static assets
+**`bundle(websites_path: &Path, source_folder: &str, watch: bool) -> io::Result<PathBuf>`**
+Main bundling function that processes web assets and optionally starts file watching.
+
+### Bundling Process
+
+1. **Directory Setup**: Creates `dist/` directory and copies source files
+2. **JavaScript Bundling**: Processes `scripts/entry.js` with Rolldown bundler
+3. **Template Processing**: Updates `index.html` with hashed asset references
+4. **Service Worker**: Generates service worker with version injection
+5. **File Watching**: Optionally monitors source files for changes
+
+### Configuration
+
+**Rolldown Bundler Options:**
+
+- **Input**: `./src/scripts/entry.js` (main JavaScript entry point)
+- **Output**: `./dist/scripts/` directory
+- **Minification**: Enabled for production builds
+- **Source Maps**: File-based source maps for debugging
+- **Asset Hashing**: Automatic hash generation for cache busting
+
+## Examples
+
+### Development Mode with Hot Reloading
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let web_root = Path::new("./websites");
+
+    // Start development server with file watching
+    let _dist_path = bundle(web_root, "explorer", true).await?;
+
+    println!("Development server started!");
+    println!("Hot reloading enabled - edit files to see changes");
+
+    // Keep server running
+    loop {
+        tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
+    }
+}
 ```
 
-## Bundling Process
+### Production Build
 
-1. **Clean**: Removes existing `dist/` directory
-2. **Copy**: Copies all source files to `dist/`
-3. **Bundle JavaScript**: 
-   - Processes `scripts/entry.js` as entry point
-   - Generates minified bundle with source maps
-   - Creates hashed filename for cache busting
-4. **Process HTML**: Updates script references to hashed filenames
-5. **Process Service Worker**: Injects current version string
-6. **Watch** (if enabled): Monitors for file changes and rebuilds
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
 
-## Configuration
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let web_root = Path::new("./websites");
 
-The bundler uses Rolldown with these optimized settings:
+    // Build for production (no watching)
+    let dist_path = bundle(web_root, "dashboard", false).await?;
+
+    println!("Production build completed: {}", dist_path.display());
+
+    // Assets are minified and ready for deployment
+    Ok(())
+}
+```
+
+### Custom Web Application Structure
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+// Expected directory structure:
+// websites/
+// ├── my_app/
+// │   ├── index.html          // Main HTML template
+// │   ├── service-worker.js   // Service worker template
+// │   ├── scripts/
+// │   │   └── entry.js        // JavaScript entry point
+// │   ├── styles/
+// │   │   └── main.css        // CSS files
+// │   └── assets/
+// │       └── images/         // Static assets
+// └── dist/                   // Generated output
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let websites_path = Path::new("./websites");
+    let source_folder = "my_app";
+
+    let dist_path = bundle(websites_path, source_folder, false).await?;
+
+    // Result: dist/ contains bundled and processed files
+    // - dist/index.html (with updated script references)
+    // - dist/service-worker.js (with version injection)
+    // - dist/scripts/main.[hash].js (minified and hashed)
+    // - dist/styles/ (copied CSS files)
+    // - dist/assets/ (copied static assets)
+
+    Ok(())
+}
+```
+
+## Architecture
+
+### File Processing Pipeline
+
+1. **Source Copying**: Recursively copies all source files to dist directory
+2. **JavaScript Bundling**: Rolldown processes entry.js with dependencies
+3. **Asset Hashing**: Generates content-based hashes for cache busting
+4. **Template Updates**: Replaces placeholders in HTML templates
+5. **Version Injection**: Updates service worker with current package version
+
+### File Watching System
+
+**Development Mode Watchers:**
+
+- **Source File Watcher**: Monitors non-script files for changes
+- **Bundle Watcher**: Watches JavaScript files and triggers rebuilds
+- **Template Watcher**: Updates HTML when bundled assets change
+
+**Event Handling:**
+
+- **File Creation/Modification**: Automatic copying to dist directory
+- **Script Changes**: Triggers Rolldown rebuild and template update
+- **Template Changes**: Processes HTML and updates asset references
+
+### Template Processing
+
+**index.html Processing:**
+
+- Scans bundled JavaScript for asset hash
+- Replaces `/scripts/main.js` with `/scripts/main.[hash].js`
+- Maintains cache busting while preserving template structure
+
+**service-worker.js Processing:**
+
+- Replaces `__VERSION__` placeholder with current crate version
+- Enables version-based cache invalidation
+- Maintains service worker functionality
+
+### Async Architecture
+
+Built on Tokio async runtime:
+
+- **Non-blocking I/O**: Efficient file operations and watching
+- **Concurrent Tasks**: Parallel file watching and bundle processing
+- **Background Processing**: Development server runs in background task
+
+## Configuration Options
+
+### Rolldown Configuration
+
+The bundler uses optimized Rolldown settings:
 
 ```rust
 BundlerOptions {
-    input: Some(vec![source_entry.into()]),    // scripts/entry.js
-    dir: Some("./dist/scripts".to_string()),   // Output directory
-    cwd: Some(websites_path),                  // Working directory
-    minify: Some(RawMinifyOptions::Bool(true)), // Enable minification
-    sourcemap: Some(SourceMapType::File),      // Generate source maps
-    ..Default::default()
+    input: Some(vec!["./src/scripts/entry.js".into()]),
+    dir: Some("./dist/scripts".to_string()),
+    minify: Some(RawMinifyOptions::Bool(true)),
+    sourcemap: Some(SourceMapType::File),
+    // ... other default options
 }
 ```
 
-## File Watching
+### File Structure Requirements
 
-In watch mode, the bundler monitors:
+**Required Files:**
 
-- **Source files**: Non-script files are copied on change
-- **JavaScript files**: Trigger full rebuild via Rolldown watcher
-- **HTML files**: Processed to update script references
-- **Service worker**: Version injection on changes
+- `src/scripts/entry.js` - JavaScript entry point
+- `src/index.html` - HTML template
+- `src/service-worker.js` - Service worker template
 
-### Watch Events Handled
+**Optional Directories:**
 
-- `Create` - New files added
-- `Modify` - Existing files changed
-- Ignores `Delete` and other events
+- `src/styles/` - CSS stylesheets
+- `src/assets/` - Static assets (images, fonts, etc.)
+- `src/components/` - Additional JavaScript modules
 
-## Version Injection
+## Development Workflow
 
-Service workers get automatic version injection:
+### Setup
 
-```javascript
-// In source service-worker.js
-const VERSION = '__VERSION__';
+1. Create web application in `websites/app_name/`
+2. Add required files (index.html, entry.js, service-worker.js)
+3. Run bundler in watch mode for development
 
-// After bundling
-const VERSION = 'v0.0.88';
-```
+### Hot Reloading
 
-This enables proper cache invalidation across releases.
+- **Script Changes**: Automatic bundle rebuild and browser refresh
+- **Template Changes**: Immediate HTML update with asset hash replacement
+- **Asset Changes**: Instant copy to dist directory
+- **Style Changes**: Direct copy without bundling
 
-## Performance Features
+### Production Deployment
 
-- **Async Operations**: All bundling operations are async
-- **Incremental Builds**: Only rebuilds changed files in watch mode
-- **Parallel Processing**: Uses Tokio for concurrent file operations
-- **Efficient Copying**: Direct file system operations
+1. Run bundler without watch mode
+2. Deploy `dist/` directory contents
+3. Assets include content hashes for cache busting
+4. Service worker includes version for cache management
 
-## Error Handling
+## Code Analysis Summary
 
-- **Graceful Failures**: Logs errors but continues watching
-- **Path Resolution**: Automatic path absolutization and validation
-- **File System Errors**: Proper error propagation with context
-
-## Dependencies
-
-- `brk_rolldown` - Rust-based Rollup bundler
-- `notify` - File system watching
-- `tokio` - Async runtime for file operations
-- `sugar_path` - Path manipulation utilities
-- `log` - Error logging
-
-## Integration Points
-
-The bundler integrates with:
-- **brk_cli**: Called during website setup
-- **brk_server**: Serves bundled assets
-- **Development workflow**: Provides live rebuilding
+**Main Function**: `bundle()` async function coordinating Rolldown bundler with file processing and watching \
+**File Operations**: Recursive directory copying with `copy_dir_all()` and selective file processing \
+**Templating**: String replacement for asset hash injection and version management \
+**File Watching**: Multi-watcher system using `notify` crate for real-time development feedback \
+**Async Integration**: Tokio-based async architecture with background task spawning \
+**Bundler Integration**: Rolldown wrapper with optimized configuration for web development \
+**Architecture**: Development-focused asset pipeline with hot reloading and production optimization
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_bundler/src/lib.rs

@@ -35,7 +35,9 @@ pub async fn bundle(websites_path: &Path, source_folder: &str, watch: bool) -> i
         ..Default::default()
     });
 
-    bundler.write().await.unwrap();
+    if let Err(error) = bundler.write().await {
+        error!("{error:?}");
+    }
 
     let absolute_source_index_path = source_path.join("index.html").absolutize();
     let absolute_source_index_path_clone = absolute_source_index_path.clone();

crates/brk_cli/Cargo.toml

@@ -19,16 +19,16 @@ brk_interface = { workspace = true }
 brk_logger = { workspace = true }
 brk_parser = { workspace = true }
 brk_server = { workspace = true }
+brk_structs = { workspace = true }
 vecdb = { workspace = true }
-clap = { version = "4.5.45", features = ["string"] }
-clap_derive = "4.5.45"
+clap = { version = "4.5.48", features = ["derive", "string"] }
 color-eyre = "0.6.5"
 log = { workspace = true }
 minreq = { workspace = true }
 serde = { workspace = true }
 tokio = { workspace = true }
-toml = "0.9.5"
-zip = { version = "4.5.0", default-features = false, features = ["deflate"] }
+toml = "0.9.7"
+zip = { version = "5.1.1", default-features = false, features = ["deflate"] }
 
 [[bin]]
 name = "brk"

crates/brk_cli/README.md

@@ -1,189 +1,232 @@
 # brk_cli
 
-**Command line interface for running complete BRK instances**
+Command-line interface orchestrating complete Bitcoin Research Kit instances with automatic configuration and continuous blockchain processing.
 
-`brk_cli` provides the main command-line interface for operating the Bitcoin Research Kit. It orchestrates the complete data pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with automatic configuration management and graceful operation.
+[![Crates.io](https://img.shields.io/crates/v/brk_cli.svg)](https://crates.io/crates/brk_cli)
+[![Documentation](https://docs.rs/brk_cli/badge.svg)](https://docs.rs/brk_cli)
 
-## What it provides
+## Overview
 
-- **Complete Pipeline Orchestration**: Coordinates parser, indexer, computer, and server components
-- **Automatic Configuration**: Saves settings to `~/.brk/config.toml` for consistent operation
-- **Continuous Operation**: Handles blockchain updates and incremental processing
-- **Web Interface Options**: Configurable website serving (none, default, custom)
-- **Graceful Shutdown**: Ctrl+C handling with proper cleanup
+This crate provides the primary command-line interface for running Bitcoin Research Kit instances. It orchestrates the entire data processing pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with persistent configuration management, automatic error recovery, and continuous blockchain synchronization.
 
-## Key Features
+**Key Features:**
 
-### Pipeline Management
-- **Automatic dependency handling**: Ensures Bitcoin Core sync before processing
-- **Incremental updates**: Only processes new blocks since last run
-- **Error recovery**: Automatic retry logic and graceful error handling
-- **Resource management**: Optimized memory usage and disk I/O
+- Complete BRK pipeline orchestration with parser, indexer, computer, and server coordination
+- Persistent configuration system with TOML-based auto-save functionality
+- Continuous blockchain processing with new block detection and incremental updates
+- Flexible Bitcoin Core RPC authentication with cookie file and user/password support
+- Configurable web interface options including auto-downloading from GitHub releases
+- Large stack allocation (512MB) for handling complex blockchain processing workloads
+- Graceful shutdown handling with proper cleanup and state preservation
 
-### Configuration System
-- **Auto-save configuration**: All CLI options saved to persistent config
-- **Flexible paths**: Configurable Bitcoin directory, blocks directory, and output directory
-- **RPC authentication**: Cookie file or username/password authentication
-- **Data source options**: Configurable price fetching and exchange APIs
+**Target Use Cases:**
 
-### Operation Modes
-- **Initial sync**: Full blockchain processing from genesis
-- **Continuous operation**: Real-time processing of new blocks
-- **Update mode**: Resume from last processed block
-- **Server mode**: HTTP API with optional web interface
+- Production Bitcoin analytics deployments requiring full pipeline operation
+- Development environments for Bitcoin research and analysis
+- Continuous blockchain monitoring with real-time data updates
+- Academic research requiring comprehensive historical blockchain datasets
 
 ## Installation
 
-### Binary Release
 ```bash
-# Download from GitHub releases
-# https://github.com/bitcoinresearchkit/brk/releases/latest
+cargo install brk # or cargo install brk_cli
 ```
 
-### Via Cargo
-```bash
-cargo install brk --locked
-```
-
-### From Source
-```bash
-git clone https://github.com/bitcoinresearchkit/brk.git
-cd brk && cargo build --release
-```
-
-## Usage
-
-### First Run (Configuration Setup)
+## Quick Start
 
 ```bash
-# Basic setup with default options
-brk --brkdir ./my_brk_data
+# First run - configure and start processing
+brk --brkdir ./data --bitcoindir ~/.bitcoin --fetch true
 
-# Full configuration
-brk --bitcoindir ~/.bitcoin \
-    --brkdir ./brk_data \
-    --fetch true \
-    --exchanges true \
-    --website default
-```
-
-### Subsequent Runs
-
-```bash
-# Uses saved configuration from ~/.brk/config.toml
+# Subsequent runs use saved configuration
 brk
 
 # Override specific options
 brk --website none --fetch false
 ```
 
-### Command Line Options
+## API Overview
+
+### Core Structure
+
+- **`Config`**: Persistent configuration with clap-based CLI parsing and TOML serialization
+- **`Bridge`**: Interface trait for generating JavaScript bridge files for web interfaces
+- **`Website`**: Enum for web interface options (None, Bitview, Custom)
+- **Path Functions**: Cross-platform default path resolution for Bitcoin and BRK directories
+
+### Main Operations
+
+**`main() -> color_eyre::Result<()>`**
+Entry point with error handling setup, directory creation, logging initialization, and high-stack thread spawning.
+
+**`run() -> color_eyre::Result<()>`**
+Core processing loop handling configuration, RPC connection, component initialization, and continuous blockchain monitoring.
+
+### Configuration Management
+
+**Persistent Settings:**
+
+- All CLI arguments automatically saved to `~/.brk/config.toml`
+- Argument overrides update saved configuration on each run
+- Cross-platform path resolution with tilde and $HOME expansion
+- Validation of Bitcoin directory, blocks directory, and RPC authentication
+
+**CLI Parameters:**
+
+- `--bitcoindir`, `--blocksdir`, `--brkdir`: Directory configuration
+- `--fetch`, `--exchanges`: Data source configuration
+- `--website`: Web interface selection
+- `--rpcconnect`, `--rpcport`, `--rpccookiefile`, `--rpcuser`, `--rpcpassword`: RPC settings
+
+## Examples
+
+### Basic Usage
 
 ```bash
-brk --help
+# Initialize with custom directories
+brk --bitcoindir /data/bitcoin --brkdir /data/brk
+
+# Enable all features with custom RPC
+brk --fetch true --exchanges true --website bitview \
+    --rpcuser myuser --rpcpassword mypass
+
+# Minimal setup with API only
+brk --website none --fetch false
 ```
 
-## Configuration Reference
+### Configuration File Example
 
-All options are automatically saved to `~/.brk/config.toml`:
+After first run, settings are saved to `~/.brk/config.toml`:
 
-### Core Paths
-- `--bitcoindir <PATH>` - Bitcoin Core directory (default: `~/.bitcoin`)
-- `--blocksdir <PATH>` - Block files directory (default: `bitcoindir/blocks`)
-- `--brkdir <PATH>` - BRK output directory (default: `~/.brk`)
-
-### Data Sources
-- `--fetch <BOOL>` - Enable price data fetching (default: `true`)
-- `--exchanges <BOOL>` - Use exchange APIs for prices (default: `true`)
-
-### Web Interface
-- `--website <OPTION>` - Web interface mode:
-  - `none` - API only, no web interface
-  - `default` - Built-in web interface from `websites/default/`
-  - `custom` - Serve custom website from `websites/custom/`
-
-### Bitcoin Core RPC
-- `--rpcconnect <IP>` - RPC host (default: `localhost`)
-- `--rpcport <PORT>` - RPC port (default: `8332`)
-- `--rpccookiefile <PATH>` - Cookie authentication file
-- `--rpcuser <USERNAME>` - Username authentication
-- `--rpcpassword <PASSWORD>` - Password authentication
-
-## Operation Flow
-
-1. **Configuration Loading**: Loads saved config from `~/.brk/config.toml`
-2. **Bitcoin Core Connection**: Establishes RPC connection and waits for sync
-3. **Data Pipeline Initialization**: Sets up parser, indexer, computer, and interface
-4. **Processing Loop**:
-   - Index new blocks from Bitcoin Core
-   - Compute analytics on new data
-   - Update cached data
-5. **Server Startup**: Launches HTTP API with optional web interface
-6. **Continuous Operation**: Monitors for new blocks and processes incrementally
-
-## System Requirements
-
-- **Bitcoin Core**: Fully synced node with RPC enabled
-- **Storage**: ~32% of blockchain size (~233GB as of 2025)
-- **Memory**:
-  - Peak: ~7-8GB during initial indexing
-  - Steady state: ~4-5GB during operation
-- **OS**: macOS or Linux
-  - Ubuntu: `sudo apt install libssl-dev pkg-config`
-
-## Performance Characteristics
-
-### Initial Sync
-- **Full blockchain processing**: ~13-15 hours total
-- **Parser phase**: ~4 minutes for block parsing
-- **Indexer phase**: ~7-8 hours for data indexing
-- **Computer phase**: ~6-7 hours for analytics computation
-
-### Continuous Operation
-- **New block processing**: 3-5 seconds per block
-- **API response times**: Typically <100ms with caching
-- **Memory usage**: Stable ~4-5GB during normal operation
-
-## Configuration File
-
-Example `~/.brk/config.toml`:
 ```toml
-bitcoindir = "/Users/username/.bitcoin"
-blocksdir = "/Users/username/.bitcoin/blocks"
-brkdir = "/Users/username/brk_data"
+bitcoindir = "/home/user/.bitcoin"
+blocksdir = "/home/user/.bitcoin/blocks"
+brkdir = "/home/user/brk_data"
 fetch = true
 exchanges = true
-website = "default"
+website = "bitview"
 rpcconnect = "localhost"
 rpcport = 8332
-rpccookiefile = "/Users/username/.bitcoin/.cookie"
+rpccookiefile = "/home/user/.bitcoin/.cookie"
 ```
 
-## Error Handling
+### Web Interface Configuration
 
-- **Bitcoin Core sync**: Waits for node sync before processing
-- **RPC connection**: Automatic retry logic for connection issues
-- **Processing errors**: Graceful error handling with detailed logging
-- **Graceful shutdown**: Ctrl+C handling with proper cleanup and state saving
+```bash
+# Use built-in Bitview interface
+brk --website bitview
 
-## Logging
+# Use custom web interface
+brk --website custom
 
-Logs are written to `~/.brk/brk.log` with colored console output:
-- Request/response logging with timing
-- Processing progress indicators
-- Error reporting and debugging information
+# API only, no web interface
+brk --website none
+```
 
-## Dependencies
+### Development Mode
 
-- `brk_parser` - Bitcoin block parsing
-- `brk_indexer` - Blockchain data indexing
-- `brk_computer` - Analytics computation
-- `brk_interface` - Data query interface
-- `brk_server` - HTTP API server
-- `brk_logger` - Logging utilities
-- `bitcoincore_rpc` - Bitcoin Core RPC client
-- `color_eyre` - Enhanced error reporting
+```bash
+# Development with local website directory
+# Looks for ../../websites directory first
+brk --website bitview
+
+# Production with auto-download from GitHub
+# Downloads websites from release artifacts
+brk --website bitview
+```
+
+## Architecture
+
+### Startup Sequence
+
+1. **Environment Setup**: Color eyre error handling, directory creation, logging initialization
+2. **High-Stack Thread**: 512MB stack for complex blockchain processing operations
+3. **Configuration Loading**: CLI parsing, TOML reading, argument merging, validation
+4. **Component Initialization**: Parser, indexer, computer, interface creation with proper dependencies
+
+### Processing Pipeline
+
+**Continuous Operation Loop:**
+
+1. **Bitcoin Core Sync Wait**: Monitors `headers == blocks` for full node synchronization
+2. **Block Count Detection**: Compares current and previous block counts for new block detection
+3. **Indexing Phase**: Processes new blocks through parser with collision detection option
+4. **Computing Phase**: Runs analytics computations on newly indexed data
+5. **Server Operation**: Serves HTTP API with optional web interface throughout processing
+
+### Web Interface Integration
+
+**Website Handling:**
+
+- **Development Mode**: Uses local `../../websites` directory if available
+- **Production Mode**: Downloads release artifacts from GitHub using semantic versioning
+- **Bundle Generation**: Creates optimized JavaScript bundles using `brk_bundler`
+- **Bridge Files**: Generates JavaScript bridge files for vector IDs and pool data
+
+**Download and Bundle Process:**
+
+```rust
+// Automatic website download and bundling
+let url = format!("https://github.com/bitcoinresearchkit/brk/archive/refs/tags/v{VERSION}.zip");
+let response = minreq::get(url).send()?;
+zip::ZipArchive::new(cursor).extract(downloads_path)?;
+bundle(&websites_path, website.to_folder_name(), true).await?
+```
+
+### RPC Authentication
+
+**Flexible Authentication Methods:**
+
+- **Cookie File**: Automatic detection at `--bitcoindir/.cookie`
+- **User/Password**: Manual configuration with `--rpcuser` and `--rpcpassword`
+- **Connection Validation**: Startup checks ensure proper Bitcoin Core connectivity
+
+### Configuration System
+
+**TOML Persistence:**
+
+- Automatic serialization/deserialization with `serde` and `toml`
+- Error-tolerant parsing with `default_on_error` deserializer
+- Argument consumption validation ensuring all CLI options are processed
+- Path expansion supporting `~` and `$HOME` environment variables
+
+## Configuration
+
+### Default Paths
+
+**Cross-Platform Path Resolution:**
+
+- **Linux**: `~/.bitcoin` for Bitcoin Core, `~/.brk` for BRK data
+- **macOS**: `~/Library/Application Support/Bitcoin` for Bitcoin Core
+- **Logs**: `~/.brk/log` for application logging
+- **Downloads**: `~/.brk/downloads` for temporary website artifacts
+
+### Performance Settings
+
+**Memory Management:**
+
+- 512MB stack size for main processing thread
+- Multi-threaded tokio runtime with all features enabled
+- Persistent configuration caching to minimize I/O operations
+
+### Error Handling
+
+**Comprehensive Validation:**
+
+- Directory existence checks with user-friendly error messages
+- RPC authentication verification before processing begins
+- Graceful exit with help suggestions for configuration issues
+
+## Code Analysis Summary
+
+**Main Structure**: `Config` struct with clap-derived CLI parsing and persistent TOML configuration management \
+**Processing Loop**: Continuous Bitcoin Core monitoring with sync detection and incremental block processing \
+**Web Integration**: Automatic website download from GitHub releases with JavaScript bundle generation \
+**Component Orchestration**: Coordination of parser, indexer, computer, and server with proper dependency management \
+**Error Handling**: `color_eyre` integration with comprehensive validation and user-friendly error messages \
+**Threading**: High-stack thread allocation (512MB) with tokio multi-threaded runtime for complex operations \
+**Architecture**: Complete BRK pipeline orchestration with persistent configuration and continuous blockchain synchronization
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_cli/src/bridge.rs

@@ -2,18 +2,19 @@ use std::{fs, io, path::Path};
 
 use brk_interface::{Index, Interface};
 use brk_server::VERSION;
+use brk_structs::pools;
 
 use crate::website::Website;
 
-const SCRIPTS: &str = "scripts";
+const BRIDGE_PATH: &str = "scripts/bridge";
 
 #[allow(clippy::upper_case_acronyms)]
 pub trait Bridge {
-    fn generate_bridge_file(&self, website: Website, websites_path: &Path) -> io::Result<()>;
+    fn generate_bridge_files(&self, website: Website, websites_path: &Path) -> io::Result<()>;
 }
 
 impl Bridge for Interface<'static> {
-    fn generate_bridge_file(&self, website: Website, websites_path: &Path) -> io::Result<()> {
+    fn generate_bridge_files(&self, website: Website, websites_path: &Path) -> io::Result<()> {
         if website.is_none() {
             return Ok(());
         }
@@ -24,88 +25,130 @@ impl Bridge for Interface<'static> {
             return Ok(());
         }
 
-        let path = path.join(SCRIPTS);
+        let path = path.join(BRIDGE_PATH);
 
         fs::create_dir_all(&path)?;
 
-        let path = path.join(Path::new("vecid-to-indexes.js"));
+        generate_vecs_file(self, &path)?;
+        generate_pools_file(&path)
+    }
+}
 
-        let indexes = Index::all();
+fn generate_pools_file(parent: &Path) -> io::Result<()> {
+    let path = parent.join(Path::new("pools.js"));
 
-        let mut contents = format!(
-            "//
+    let pools = pools();
+
+    let mut contents = "//
+// File auto-generated, any modifications will be overwritten
+//
+"
+    .to_string();
+
+    contents += "
+/** @typedef {ReturnType<typeof createPools>} Pools */
+/** @typedef {keyof Pools} Pool */
+
+export function createPools() {
+return /** @type {const} */ ({
+";
+
+    let mut sorted_pools: Vec<_> = pools.iter().collect();
+    sorted_pools.sort_by(|a, b| a.name.to_lowercase().cmp(&b.name.to_lowercase()));
+
+    contents += &sorted_pools
+        .iter()
+        .map(|pool| {
+            let id = pool.serialized_id();
+            format!("    {id}: \"{}\",", pool.name)
+        })
+        .collect::<Vec<_>>()
+        .join("\n");
+
+    contents += "\n  });\n}\n";
+
+    fs::write(path, contents)
+}
+
+fn generate_vecs_file(interface: &Interface<'static>, parent: &Path) -> io::Result<()> {
+    let path = parent.join(Path::new("vecs.js"));
+
+    let indexes = Index::all();
+
+    let mut contents = format!(
+        "//
 // File auto-generated, any modifications will be overwritten
 //
 
 export const VERSION = \"v{VERSION}\";
 
 "
-        );
+    );
 
-        contents += &indexes
+    contents += &indexes
+        .iter()
+        .enumerate()
+        .map(|(i_of_i, i)| {
+            // let lowered = i.to_string().to_lowercase();
+            format!("/** @typedef {{{i_of_i}}} {i} */",)
+        })
+        .collect::<Vec<_>>()
+        .join("\n");
+
+    contents += &format!(
+        "\n\n/** @typedef {{{}}} Index */\n",
+        indexes
             .iter()
-            .enumerate()
-            .map(|(i_of_i, i)| {
-                // let lowered = i.to_string().to_lowercase();
-                format!("/** @typedef {{{i_of_i}}} {i} */",)
-            })
+            .map(|i| i.to_string())
             .collect::<Vec<_>>()
-            .join("\n");
+            .join(" | ")
+    );
 
-        contents += &format!(
-            "\n\n/** @typedef {{{}}} Index */\n",
-            indexes
-                .iter()
-                .map(|i| i.to_string())
-                .collect::<Vec<_>>()
-                .join(" | ")
-        );
-
-        contents += "
+    contents += "
 /** @typedef {ReturnType<typeof createIndexes>} Indexes */
 
 export function createIndexes() {
-  return {
+return {
 ";
 
-        contents += &indexes
-            .iter()
-            .enumerate()
-            .map(|(i_of_i, i)| {
-                let lowered = i.to_string().to_lowercase();
-                format!("    {lowered}: /** @satisfies {{{i}}} */ ({i_of_i}),",)
-            })
-            .collect::<Vec<_>>()
-            .join("\n");
+    contents += &indexes
+        .iter()
+        .enumerate()
+        .map(|(i_of_i, i)| {
+            let lowered = i.to_string().to_lowercase();
+            format!("    {lowered}: /** @satisfies {{{i}}} */ ({i_of_i}),",)
+        })
+        .collect::<Vec<_>>()
+        .join("\n");
 
-        contents += "  };\n}\n";
+    contents += "  };\n}\n";
 
-        contents += "
+    contents += "
 /** @typedef {ReturnType<typeof createVecIdToIndexes>} VecIdToIndexes
 /** @typedef {keyof VecIdToIndexes} VecId */
 
 /**
- * @returns {Record<any, number[]>}
- */
+* @returns {Record<any, number[]>}
+*/
 export function createVecIdToIndexes() {
-  return {
+return {
 ";
 
-        self.id_to_index_to_vec()
-            .iter()
-            .for_each(|(id, index_to_vec)| {
-                let indexes = index_to_vec
-                    .keys()
-                    .map(|i| (*i as u8).to_string())
-                    // .map(|i| i.to_string())
-                    .collect::<Vec<_>>()
-                    .join(", ");
+    interface
+        .id_to_index_to_vec()
+        .iter()
+        .for_each(|(id, index_to_vec)| {
+            let indexes = index_to_vec
+                .keys()
+                .map(|i| (*i as u8).to_string())
+                // .map(|i| i.to_string())
+                .collect::<Vec<_>>()
+                .join(", ");
 
-                contents += &format!("    \"{id}\": [{indexes}],\n");
-            });
+            contents += &format!("    \"{id}\": [{indexes}],\n");
+        });
 
-        contents += "  };\n}\n";
+    contents += "  };\n}\n";
 
-        fs::write(path, contents)
-    }
+    fs::write(path, contents)
 }

crates/brk_cli/src/config.rs

@@ -6,7 +6,6 @@ use std::{
 use bitcoincore_rpc::{self, Auth, Client};
 use brk_fetcher::Fetcher;
 use clap::Parser;
-use clap_derive::Parser;
 use color_eyre::eyre::eyre;
 use serde::{Deserialize, Deserializer, Serialize};
 
@@ -284,7 +283,7 @@ Finally, you can run the program with '-h' for help."
     }
 
     pub fn website(&self) -> Website {
-        self.website.unwrap_or(Website::Default)
+        self.website.unwrap_or(Website::Bitview)
     }
 
     pub fn fetch(&self) -> bool {

crates/brk_cli/src/lib.rs

@@ -13,6 +13,7 @@ use brk_bundler::bundle;
 use brk_computer::Computer;
 use brk_indexer::Indexer;
 use brk_interface::Interface;
+use brk_parser::Parser;
 use brk_server::{Server, VERSION};
 use log::info;
 use vecdb::Exit;
@@ -32,7 +33,7 @@ pub fn main() -> color_eyre::Result<()> {
     brk_logger::init(Some(&dot_brk_log_path()))?;
 
     thread::Builder::new()
-        .stack_size(256 * 1024 * 1024)
+        .stack_size(512 * 1024 * 1024)
         .spawn(run)?
         .join()
         .unwrap()
@@ -46,7 +47,7 @@ pub fn run() -> color_eyre::Result<()> {
     let exit = Exit::new();
     exit.set_ctrlc_handler();
 
-    let parser = brk_parser::Parser::new(config.blocksdir(), config.brkdir(), rpc);
+    let parser = Parser::new(config.blocksdir(), rpc);
 
     let mut indexer = Indexer::forced_import(&config.brkdir())?;
 
@@ -72,7 +73,7 @@ pub fn run() -> color_eyre::Result<()> {
         .enable_all()
         .build()?
         .block_on(async {
-            let interface = Interface::build(&indexer, &computer);
+            let interface = Interface::build(&parser, &indexer, &computer);
 
             let website = config.website();
 
@@ -106,7 +107,7 @@ pub fn run() -> color_eyre::Result<()> {
                     downloaded_websites_path
                 };
 
-                interface.generate_bridge_file(website, websites_path.as_path())?;
+                interface.generate_bridge_files(website, websites_path.as_path())?;
 
                 Some(bundle(&websites_path, website.to_folder_name(), true).await?)
             } else {
@@ -134,7 +135,7 @@ pub fn run() -> color_eyre::Result<()> {
                 let starting_indexes =
                     indexer.index(&parser, rpc, &exit, config.check_collisions()).unwrap();
 
-                computer.compute(&indexer, starting_indexes, &exit).unwrap();
+                computer.compute(&indexer, starting_indexes, &parser, &exit).unwrap();
 
                 info!("Waiting for new blocks...");
 

crates/brk_cli/src/website.rs

@@ -1,10 +1,11 @@
-use clap_derive::ValueEnum;
+use clap::ValueEnum;
 use serde::{Deserialize, Serialize};
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize, ValueEnum)]
+#[serde(rename_all = "lowercase")]
 pub enum Website {
     None,
-    Default,
+    Bitview,
     Custom,
 }
 
@@ -20,7 +21,7 @@ impl Website {
     pub fn to_folder_name(self) -> &'static str {
         match self {
             Self::Custom => "custom",
-            Self::Default => "default",
+            Self::Bitview => "bitview",
             Self::None => unreachable!(),
         }
     }

crates/brk_computer/Cargo.toml

@@ -10,6 +10,7 @@ rust-version.workspace = true
 build = "build.rs"
 
 [dependencies]
+allocative = { workspace = true }
 bitcoin = { workspace = true }
 bitcoincore-rpc = { workspace = true }
 brk_structs = { workspace = true }
@@ -17,6 +18,7 @@ brk_error = { workspace = true }
 brk_fetcher = { workspace = true }
 brk_indexer = { workspace = true }
 brk_logger = { workspace = true }
+brk_store = { workspace = true }
 brk_parser = { workspace = true }
 vecdb = { workspace = true }
 derive_deref = { workspace = true }
@@ -26,6 +28,3 @@ rayon = { workspace = true }
 serde = { workspace = true }
 zerocopy = { workspace = true }
 zerocopy-derive = { workspace = true }
-
-[package.metadata.cargo-machete]
-ignored = ["zerocopy"]

crates/brk_computer/README.md

@@ -1,200 +1,303 @@
 # brk_computer
 
-**Bitcoin analytics engine that transforms indexed blockchain data into comprehensive metrics**
+Advanced Bitcoin analytics engine that transforms indexed blockchain data into comprehensive metrics and financial analytics.
 
-`brk_computer` is the computational layer of BRK that processes indexed blockchain data to generate analytics across multiple specialized domains. It provides comprehensive Bitcoin metrics with efficient storage and lazy computation for optimal performance.
+[![Crates.io](https://img.shields.io/crates/v/brk_computer.svg)](https://crates.io/crates/brk_computer)
+[![Documentation](https://docs.rs/brk_computer/badge.svg)](https://docs.rs/brk_computer)
 
-## What it provides
+## Overview
 
-- **Comprehensive Analytics**: 9 specialized domains covering all aspects of Bitcoin analysis
-- **Lazy Computation**: On-demand calculation with dependency tracking and caching
-- **Incremental Updates**: Only processes new data since last computation
-- **Memory Efficiency**: ~100MB operation footprint via compressed storage and memory mapping
-- **Multi-timeframe Analysis**: Daily, weekly, monthly, quarterly, yearly perspectives
+This crate provides a sophisticated analytics engine that processes indexed Bitcoin blockchain data to compute comprehensive metrics, financial analytics, and statistical aggregations. Built on top of `brk_indexer`, it transforms raw blockchain data into actionable insights through state tracking, cohort analysis, market metrics, and advanced Bitcoin-specific calculations.
 
-## Nine Analytics Domains
+**Key Features:**
 
-The computer processes data through a fixed dependency chain:
+- Comprehensive Bitcoin analytics pipeline with 6 major computation modules
+- UTXO and address cohort analysis with lifecycle tracking
+- Market metrics integration with price data and financial calculations
+- Cointime economics and realized/unrealized profit/loss analysis
+- Supply dynamics and monetary policy metrics
+- Pool analysis for centralization and mining statistics
+- Memory allocation tracking and performance optimization
+- Parallel computation with multi-threaded processing
 
-1. **indexes** - Time-based indexing (date/height mappings, epoch calculations)
-2. **constants** - Baseline values and reference metrics
-3. **blocks** - Block analytics (sizes, intervals, transaction counts, weight)
-4. **mining** - Mining economics (hashrate, difficulty, rewards, epochs)
-5. **fetched** - External price data integration (optional)
-6. **price** - OHLC data across multiple timeframes (optional, requires fetched)
-7. **transactions** - Transaction analysis (fees, sizes, patterns, RBF detection)
-8. **market** - Price correlations and market metrics (optional, requires price)
-9. **stateful** - UTXO tracking and accumulated state computations
-10. **cointime** - Coin age and time-based value analysis
+**Target Use Cases:**
 
-## Key Features
+- Bitcoin market analysis and research platforms
+- On-chain analytics for investment and trading decisions
+- Academic research requiring comprehensive blockchain metrics
+- Financial applications needing Bitcoin exposure and risk metrics
 
-### Computation Strategy
-- **Fixed dependency chain**: Ensures data consistency across all domains
-- **Parallel processing**: Uses Rayon for performance optimization
-- **State management**: Rollback capabilities for error recovery
-- **Incremental updates**: Only computes new data since last run
+## Installation
 
-### Analytics Capabilities
-- **Multi-timeframe analysis**: Daily, weekly, monthly, quarterly, yearly aggregations
-- **Chain-based metrics**: Height, difficulty epoch, halving epoch indexing
-- **Price correlation**: Both dollar and satoshi denominated metrics
-- **DCA analysis**: Dollar Cost Averaging with configurable periods
-- **Supply analysis**: Circulating, realized, unrealized supply metrics
-- **Address cohort tracking**: Analysis across different Bitcoin address types
-- **UTXO cohort analysis**: Realized/unrealized gains tracking
-- **Coin time analysis**: Understanding Bitcoin velocity and dormancy
+```bash
+cargo add brk_computer
+```
 
-### Storage Optimization
-- **Compressed vectors**: Efficient disk storage with lazy computation
-- **Memory mapping**: Minimal RAM usage during operation
-- **Version management**: Automatic invalidation on schema changes
-- **Dependency tracking**: Smart recomputation based on data changes
-
-## Usage
-
-### Basic Setup (No Price Data)
+## Quick Start
 
 ```rust
 use brk_computer::Computer;
 use brk_indexer::Indexer;
-use vecdb::Exit;
-
-// Setup without external price data
-let indexer = Indexer::forced_import("./brk_data")?;
-let mut computer = Computer::forced_import("./brk_data", &indexer, None)?;
-
-// Setup exit handler
-let exit = Exit::new();
-exit.set_ctrlc_handler();
-
-// Compute all analytics
-let starting_indexes = indexer.get_starting_indexes();
-computer.compute(&indexer, starting_indexes, &exit)?;
-```
-
-### Advanced Setup (With Price Data)
-
-```rust
 use brk_fetcher::Fetcher;
+use vecdb::Exit;
+use std::path::Path;
 
-// Setup with external price data for market analytics
+// Initialize dependencies
+let outputs_path = Path::new("./analytics_data");
+let indexer = Indexer::forced_import(outputs_path)?;
 let fetcher = Some(Fetcher::import(true, None)?);
-let mut computer = Computer::forced_import("./brk_data", &indexer, fetcher)?;
 
-// Compute all analytics including price/market domains
+// Create computer with price data support
+let mut computer = Computer::forced_import(outputs_path, &indexer, fetcher)?;
+
+// Compute analytics from indexer state
+let exit = Exit::default();
+let starting_indexes = brk_indexer::Indexes::default();
 computer.compute(&indexer, starting_indexes, &exit)?;
+
+println!("Analytics computation completed!");
 ```
 
-### Accessing Computed Data
+## API Overview
+
+### Core Structure
+
+The Computer is organized into 7 specialized computation modules:
+
+- **`indexes`**: Fundamental blockchain index computations
+- **`constants`**: Network constants and protocol parameters
+- **`market`**: Price-based financial metrics and market analysis
+- **`pools`**: Mining pool analysis and centralization metrics
+- **`chain`**: Core blockchain metrics (difficulty, hashrate, fees)
+- **`stateful`**: Advanced state tracking (UTXO lifecycles, address behaviors)
+- **`cointime`**: Cointime economics and value-time calculations
+
+### Key Methods
+
+**`Computer::forced_import(outputs_path, indexer, fetcher) -> Result<Self>`**
+Creates computer instance with optional price data integration.
+
+**`compute(&mut self, indexer: &Indexer, starting_indexes: Indexes, exit: &Exit) -> Result<()>`**
+Main computation pipeline processing all analytics modules.
+
+### Analytics Categories
+
+**Market Analytics:**
+
+- Price-based metrics (market cap, realized cap, MVRV)
+- Trading volume analysis and liquidity metrics
+- Return calculations and volatility measurements
+- Dollar-cost averaging and investment strategy metrics
+
+**On-Chain Analytics:**
+
+- Transaction count and size statistics
+- Fee analysis and block space utilization
+- Address activity and entity clustering
+- UTXO age distributions and spending patterns
+
+**Monetary Analytics:**
+
+- Circulating supply and issuance tracking
+- Realized vs. unrealized gains/losses
+- Cointime destruction and accumulation
+- Velocity and economic activity indicators
+
+## Examples
+
+### Basic Analytics Computation
 
 ```rust
-// Access all computed vectors
-let all_vecs = computer.vecs(); // Returns Vec<&dyn AnyCollectableVec>
+use brk_computer::Computer;
 
-// Access specific domain data
-let block_metrics = &computer.blocks;
-let mining_data = &computer.mining;
-let transaction_stats = &computer.transactions;
+// Initialize with indexer and optional price data
+let computer = Computer::forced_import(
+    "./analytics_output",
+    &indexer,
+    Some(price_fetcher)
+)?;
 
-// Access price data (if available)
-if let Some(price_data) = &computer.price {
-    // Use OHLC data
-}
+// Compute all analytics modules
+let exit = vecdb::Exit::default();
+computer.compute(&indexer, starting_indexes, &exit)?;
+
+// Access computed metrics
+println!("Market cap vectors computed: {}", computer.market.len());
+println!("Chain metrics computed: {}", computer.chain.len());
+println!("Stateful analysis completed: {}", computer.stateful.len());
 ```
 
-### Incremental Updates
+### Market Analysis
 
 ```rust
-// Continuous computation loop
-loop {
-    // Get latest indexes from indexer
-    let current_indexes = indexer.get_current_indexes();
-    
-    // Compute only new data
-    computer.compute(&indexer, current_indexes, &exit)?;
-    
-    // Check for exit signal
-    if exit.is_signaled() {
-        break;
+use brk_computer::Computer;
+use brk_structs::{DateIndex, Height};
+
+let computer = Computer::forced_import(/* ... */)?;
+
+// Access market metrics after computation
+if let Some(market) = &computer.market {
+    // Daily market cap analysis
+    let date_index = DateIndex::from_days_since_genesis(5000);
+    if let Some(market_cap) = market.dateindex_to_market_cap.get(date_index)? {
+        println!("Market cap on day {}: ${}", date_index, market_cap.to_dollars());
     }
-    
-    // Wait before next update
-    sleep(Duration::from_secs(60));
+
+    // MVRV (Market Value to Realized Value) ratio
+    if let Some(mvrv) = market.dateindex_to_mvrv.get(date_index)? {
+        println!("MVRV ratio: {:.2}", mvrv);
+    }
+}
+
+// Chain-level metrics
+let height = Height::new(800000);
+if let Some(difficulty) = computer.chain.height_to_difficulty.get(height)? {
+    println!("Network difficulty at height {}: {}", height, difficulty);
 }
 ```
 
-## Core Computer Structure
+### Cohort Analysis
 
 ```rust
-pub struct Computer {
-    pub indexes: indexes::Vecs,           // Time indexing
-    pub constants: constants::Vecs,       // Baseline values
-    pub blocks: blocks::Vecs,            // Block analytics
-    pub mining: mining::Vecs,            // Mining economics
-    pub market: market::Vecs,            // Market metrics (optional)
-    pub price: Option<price::Vecs>,      // OHLC price data (optional)
-    pub transactions: transactions::Vecs, // Transaction analysis
-    pub stateful: stateful::Vecs,        // UTXO tracking
-    pub fetched: Option<fetched::Vecs>,  // External data (optional)
-    pub cointime: cointime::Vecs,        // Coin age analysis
+use brk_computer::Computer;
+use brk_structs::{DateIndex, CohortId};
+
+let computer = Computer::forced_import(/* ... */)?;
+
+// Address cohort analysis
+let cohort_date = DateIndex::from_days_since_genesis(4000);
+
+// Analyze address behavior patterns
+if let Some(address_cohorts) = &computer.stateful.address_cohorts {
+    for cohort_id in address_cohorts.get_cohort_ids_for_date(cohort_date)? {
+        let cohort_data = address_cohorts.get_cohort(cohort_id)?;
+
+        println!("Cohort {}: {} addresses created",
+                 cohort_id, cohort_data.addresses.len());
+        println!("Average holding period: {} days",
+                 cohort_data.avg_holding_period.as_days());
+    }
+}
+
+// UTXO cohort lifecycle analysis
+if let Some(utxo_cohorts) = &computer.stateful.utxo_cohorts {
+    let active_utxos = utxo_cohorts.get_active_utxos_for_date(cohort_date)?;
+    println!("Active UTXOs from cohort: {}", active_utxos.len());
 }
 ```
 
-## Performance Characteristics
+### Supply and Monetary Analysis
 
-**Benchmarked on MacBook Pro M3 Pro:**
-- **Initial computation**: ~6-7 hours for complete Bitcoin blockchain
-- **Storage efficiency**: All computed datasets total ~40GB
-- **Incremental updates**: 3-5 seconds per new block
-- **Memory footprint**: Peak ~7-8GB during computation, ~100MB during operation
-- **Dependencies**: Price data domains optional (fetched, price, market)
+```rust
+use brk_computer::Computer;
+use brk_structs::{Height, DateIndex};
 
-## Domain-Specific Analytics
+let computer = Computer::forced_import(/* ... */)?;
 
-### Block Analytics
-- Block sizes, weights, transaction counts
-- Block intervals and mining statistics
-- Fee analysis per block
+// Supply dynamics
+let height = Height::new(750000);
+if let Some(supply) = computer.chain.height_to_circulating_supply.get(height)? {
+    println!("Circulating supply: {} BTC", supply.to_btc());
+}
 
-### Mining Economics
-- Hashrate estimation and difficulty tracking
-- Mining reward analysis
-- Epoch-based calculations
+// Realized vs unrealized analysis
+let date = DateIndex::from_days_since_genesis(5000);
+if let Some(realized_cap) = computer.market.dateindex_to_realized_cap.get(date)? {
+    if let Some(market_cap) = computer.market.dateindex_to_market_cap.get(date)? {
+        let unrealized_pnl = market_cap - realized_cap;
+        println!("Unrealized P&L: ${:.2}B", unrealized_pnl.to_dollars() / 1e9);
+    }
+}
+```
 
-### Transaction Analysis
-- Fee rate distributions
-- RBF (Replace-By-Fee) detection
-- Output type analysis
-- Transaction size patterns
+## Architecture
 
-### Market Metrics (Optional)
-- Price correlations with on-chain metrics
-- Market cap calculations
-- DCA analysis across timeframes
+### Computation Pipeline
 
-### Stateful Analysis
-- UTXO set tracking
-- Address cohort analysis
-- Realized/unrealized gains
-- Supply distribution metrics
+The computer implements a sophisticated multi-stage pipeline:
 
-## Requirements
+1. **Index Computation**: Fundamental blockchain metrics and time-based indexes
+2. **Constants Computation**: Network parameters and protocol constants
+3. **Price Integration**: Optional price data fetching and processing
+4. **Parallel Computation**: Chain, market, pools, stateful, and cointime analytics
+5. **Cross-Dependencies**: Advanced metrics requiring multiple data sources
 
-- **Indexed data**: Requires completed `brk_indexer` output
-- **Storage space**: Additional ~40GB for computed datasets
-- **Memory**: 8GB+ RAM recommended for initial computation
-- **CPU**: Multi-core recommended for parallel processing
-- **Price data**: Optional external price feeds for market analytics
+### Memory Management
 
-## Dependencies
+**Allocation Tracking:**
 
-- `brk_indexer` - Source of indexed blockchain data
-- `brk_fetcher` - External price data (optional)
-- `vecdb` - Vector database with lazy computation
-- `rayon` - Parallel processing framework
-- `brk_structs` - Bitcoin-aware type system
+- `allocative` integration for memory usage analysis
+- Efficient vector storage with compression options
+- Strategic lazy vs. eager evaluation for memory optimization
+
+**Performance Optimization:**
+
+- `rayon` parallel processing for CPU-intensive calculations
+- Vectorized operations for time-series computations
+- Memory-mapped storage for large datasets
+
+### State Management
+
+**Stateful Analytics:**
+
+- UTXO lifecycle tracking with creation/destruction events
+- Address cohort analysis with behavioral clustering
+- Transaction pattern recognition and anomaly detection
+- Economic cycle analysis with market phase detection
+
+**Cointime Economics:**
+
+- Bitcoin days destroyed and accumulated calculations
+- Velocity measurements and economic activity indicators
+- Age-weighted value transfer analysis
+- Long-term holder vs. active trader segmentation
+
+### Modular Design
+
+Each computation module operates independently:
+
+- **Chain Module**: Basic blockchain metrics (fees, difficulty, hashrate)
+- **Market Module**: Price-dependent financial calculations
+- **Pools Module**: Mining centralization and pool analysis
+- **Stateful Module**: Advanced lifecycle and behavior tracking
+- **Cointime Module**: Economic time-value calculations
+
+### Data Dependencies
+
+**Required Dependencies:**
+
+- `brk_indexer`: Raw blockchain data access
+- `brk_structs`: Type definitions and conversions
+
+**Optional Dependencies:**
+
+- `brk_fetcher`: Price data for financial metrics
+- Market analysis requires price integration
+
+### Computation Orchestration
+
+**Sequential Stages:**
+
+1. Indexes → Constants (foundational metrics)
+2. Fetched → Price (price data processing)
+3. Parallel: Chain, Market, Pools, Stateful, Cointime
+
+**Exit Handling:**
+
+- Graceful shutdown with consistent state preservation
+- Checkpoint-based recovery for long-running computations
+- Multi-threaded coordination with exit signaling
+
+## Code Analysis Summary
+
+**Main Structure**: `Computer` struct coordinating 7 specialized analytics modules (indexes, constants, market, pools, chain, stateful, cointime) \
+**Computation Pipeline**: Multi-stage analytics processing with parallel execution and dependency management \
+**State Tracking**: Advanced UTXO and address lifecycle analysis with cohort-based behavioral clustering \
+**Financial Analytics**: Comprehensive market metrics including realized/unrealized analysis and cointime economics \
+**Memory Optimization**: `allocative` tracking with lazy/eager evaluation strategies and compressed vector storage \
+**Parallel Processing**: `rayon` integration for CPU-intensive calculations with coordinated exit handling \
+**Architecture**: Modular analytics engine transforming indexed blockchain data into actionable financial and economic insights
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_computer/examples/computer.rs

@@ -34,7 +34,7 @@ pub fn main() -> Result<()> {
             let outputs_dir = Path::new(&std::env::var("HOME").unwrap()).join(".brk");
             // let outputs_dir = Path::new("../../_outputs");
 
-            let parser = Parser::new(bitcoin_dir.join("blocks"), outputs_dir.to_path_buf(), rpc);
+            let parser = Parser::new(bitcoin_dir.join("blocks"), rpc);
 
             let mut indexer = Indexer::forced_import(&outputs_dir)?;
 
@@ -45,7 +45,7 @@ pub fn main() -> Result<()> {
             loop {
                 let i = Instant::now();
                 let starting_indexes = indexer.index(&parser, rpc, &exit, true)?;
-                computer.compute(&indexer, starting_indexes, &exit)?;
+                computer.compute(&indexer, starting_indexes, &parser, &exit)?;
                 dbg!(i.elapsed());
                 sleep(Duration::from_secs(10));
             }

crates/brk_computer/examples/pools.rs

@@ -0,0 +1,125 @@
+use std::{collections::BTreeMap, path::Path, thread};
+
+use brk_computer::Computer;
+use brk_error::Result;
+use brk_fetcher::Fetcher;
+use brk_indexer::Indexer;
+use brk_structs::{AddressBytes, OutputIndex, OutputType, pools};
+use vecdb::{AnyIterableVec, Exit, VecIterator};
+
+fn main() -> Result<()> {
+    brk_logger::init(Some(Path::new(".log")))?;
+
+    let exit = Exit::new();
+    exit.set_ctrlc_handler();
+
+    thread::Builder::new()
+        .stack_size(256 * 1024 * 1024)
+        .spawn(move || -> Result<()> {
+            let outputs_dir = Path::new(&std::env::var("HOME").unwrap()).join(".brk");
+
+            let indexer = Indexer::forced_import(&outputs_dir)?;
+
+            let fetcher = Fetcher::import(true, None)?;
+
+            let computer = Computer::forced_import(&outputs_dir, &indexer, Some(fetcher))?;
+
+            let pools = pools();
+
+            let mut res: BTreeMap<&'static str, usize> = BTreeMap::default();
+
+            let vecs = indexer.vecs;
+            let stores = indexer.stores;
+
+            let mut height_to_first_txindex_iter = vecs.height_to_first_txindex.iter();
+            let mut txindex_to_first_outputindex_iter = vecs.txindex_to_first_outputindex.iter();
+            let mut txindex_to_output_count_iter = computer.indexes.txindex_to_output_count.iter();
+            let mut outputindex_to_outputtype_iter = vecs.outputindex_to_outputtype.iter();
+            let mut outputindex_to_typeindex_iter = vecs.outputindex_to_typeindex.iter();
+            let mut p2pk65addressindex_to_p2pk65bytes_iter =
+                vecs.p2pk65addressindex_to_p2pk65bytes.iter();
+            let mut p2pk33addressindex_to_p2pk33bytes_iter =
+                vecs.p2pk33addressindex_to_p2pk33bytes.iter();
+            let mut p2pkhaddressindex_to_p2pkhbytes_iter =
+                vecs.p2pkhaddressindex_to_p2pkhbytes.iter();
+            let mut p2shaddressindex_to_p2shbytes_iter = vecs.p2shaddressindex_to_p2shbytes.iter();
+            let mut p2wpkhaddressindex_to_p2wpkhbytes_iter =
+                vecs.p2wpkhaddressindex_to_p2wpkhbytes.iter();
+            let mut p2wshaddressindex_to_p2wshbytes_iter =
+                vecs.p2wshaddressindex_to_p2wshbytes.iter();
+            let mut p2traddressindex_to_p2trbytes_iter = vecs.p2traddressindex_to_p2trbytes.iter();
+            let mut p2aaddressindex_to_p2abytes_iter = vecs.p2aaddressindex_to_p2abytes.iter();
+
+            let unknown = pools.get_unknown();
+
+            stores
+                .height_to_coinbase_tag
+                .iter()
+                .for_each(|(height, coinbase_tag)| {
+                    let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+                    let outputindex = txindex_to_first_outputindex_iter.unwrap_get_inner(txindex);
+                    let outputcount = txindex_to_output_count_iter.unwrap_get_inner(txindex);
+
+                    let pool = (*outputindex..(*outputindex + *outputcount))
+                        .map(OutputIndex::from)
+                        .find_map(|outputindex| {
+                            let outputtype =
+                                outputindex_to_outputtype_iter.unwrap_get_inner(outputindex);
+                            let typeindex =
+                                outputindex_to_typeindex_iter.unwrap_get_inner(outputindex);
+
+                            let address = match outputtype {
+                                OutputType::P2PK65 => Some(AddressBytes::from(
+                                    p2pk65addressindex_to_p2pk65bytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2PK33 => Some(AddressBytes::from(
+                                    p2pk33addressindex_to_p2pk33bytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2PKH => Some(AddressBytes::from(
+                                    p2pkhaddressindex_to_p2pkhbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2SH => Some(AddressBytes::from(
+                                    p2shaddressindex_to_p2shbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2WPKH => Some(AddressBytes::from(
+                                    p2wpkhaddressindex_to_p2wpkhbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2WSH => Some(AddressBytes::from(
+                                    p2wshaddressindex_to_p2wshbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2TR => Some(AddressBytes::from(
+                                    p2traddressindex_to_p2trbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2A => Some(AddressBytes::from(
+                                    p2aaddressindex_to_p2abytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                _ => None,
+                            };
+
+                            address
+                                .and_then(|address| pools.find_from_address(&address.to_string()))
+                        })
+                        .or_else(|| pools.find_from_coinbase_tag(&coinbase_tag))
+                        .unwrap_or(unknown);
+
+                    *res.entry(pool.name).or_default() += 1;
+                });
+
+            let mut v = res.into_iter().map(|(k, v)| (v, k)).collect::<Vec<_>>();
+            v.sort_unstable();
+            println!("{:#?}", v);
+            println!("{:#?}", v.len());
+
+            Ok(())
+        })?
+        .join()
+        .unwrap()
+}

crates/brk_computer/src/blks.rs

@@ -0,0 +1,156 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_indexer::Indexer;
+use brk_parser::Parser;
+use brk_structs::{BlkPosition, Height, StoredU32, TxIndex, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, CompressedVec, Database, Exit,
+    GenericStoredVec, PAGE_SIZE, VecIterator,
+};
+
+use super::{Indexes, indexes};
+
+#[derive(Clone)]
+pub struct Vecs {
+    db: Database,
+
+    pub height_to_position: CompressedVec<Height, BlkPosition>,
+    pub height_to_len: CompressedVec<Height, StoredU32>,
+    pub txindex_to_position: CompressedVec<TxIndex, BlkPosition>,
+    pub txindex_to_len: CompressedVec<TxIndex, StoredU32>,
+}
+
+impl Vecs {
+    pub fn forced_import(parent_path: &Path, parent_version: Version) -> Result<Self> {
+        let db = Database::open(&parent_path.join("blks"))?;
+        db.set_min_len(PAGE_SIZE * 1_000_000)?;
+
+        let version = parent_version + Version::ZERO;
+
+        let this = Self {
+            height_to_position: CompressedVec::forced_import(
+                &db,
+                "position",
+                version + Version::TWO,
+            )?,
+            height_to_len: CompressedVec::forced_import(&db, "len", version + Version::TWO)?,
+            txindex_to_position: CompressedVec::forced_import(
+                &db,
+                "position",
+                version + Version::TWO,
+            )?,
+            txindex_to_len: CompressedVec::forced_import(&db, "len", version + Version::TWO)?,
+
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        parser: &Parser,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexer, indexes, starting_indexes, parser, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        parser: &Parser,
+        exit: &Exit,
+    ) -> Result<()> {
+        let min_txindex =
+            TxIndex::from(self.txindex_to_position.len()).min(starting_indexes.txindex);
+
+        let Some(min_height) = indexes
+            .txindex_to_height
+            .iter()
+            .get_inner(min_txindex)
+            .map(|h| h.min(starting_indexes.height))
+        else {
+            return Ok(());
+        };
+
+        let mut height_to_first_txindex_iter = indexer.vecs.height_to_first_txindex.iter();
+
+        parser
+            .parse(
+                Some(min_height),
+                Some((indexer.vecs.height_to_first_txindex.len() - 1).into()),
+            )
+            .iter()
+            .try_for_each(|block| -> Result<()> {
+                let height = block.height();
+
+                self.height_to_position.forced_push_at(
+                    height,
+                    block.metadata().position(),
+                    exit,
+                )?;
+
+                self.height_to_len
+                    .forced_push_at(height, block.metadata().len().into(), exit)?;
+
+                let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+
+                block.tx_metadata().iter().enumerate().try_for_each(
+                    |(index, metadata)| -> Result<()> {
+                        let txindex = txindex + index;
+                        self.txindex_to_position.forced_push_at(
+                            txindex,
+                            metadata.position(),
+                            exit,
+                        )?;
+                        self.txindex_to_len
+                            .forced_push_at(txindex, metadata.len().into(), exit)?;
+                        Ok(())
+                    },
+                )?;
+
+                if *height % 1_000 == 0 {
+                    let _lock = exit.lock();
+                    self.height_to_position.flush()?;
+                    self.height_to_len.flush()?;
+                    self.txindex_to_position.flush()?;
+                    self.txindex_to_len.flush()?;
+                }
+                Ok(())
+            })?;
+
+        let _lock = exit.lock();
+        self.height_to_position.flush()?;
+        self.height_to_len.flush()?;
+        self.txindex_to_position.flush()?;
+        self.txindex_to_len.flush()?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        Box::new(
+            [
+                &self.height_to_position as &dyn AnyCollectableVec,
+                &self.height_to_len,
+                &self.txindex_to_position,
+                &self.txindex_to_len,
+            ]
+            .into_iter(),
+        )
+    }
+}

crates/brk_computer/src/blocks.rs

@@ -1,264 +0,0 @@
-use std::path::Path;
-
-use brk_error::Result;
-use brk_indexer::Indexer;
-use brk_structs::{
-    CheckedSub, DifficultyEpoch, HalvingEpoch, Height, StoredU32, StoredU64, Timestamp, Version,
-    Weight,
-};
-use vecdb::{AnyCollectableVec, Database, EagerVec, Exit, PAGE_SIZE, VecIterator};
-
-use crate::grouped::Source;
-
-use super::{
-    Indexes,
-    grouped::{ComputedVecsFromDateIndex, ComputedVecsFromHeight, VecBuilderOptions},
-    indexes,
-};
-
-const VERSION: Version = Version::ZERO;
-
-#[derive(Clone)]
-pub struct Vecs {
-    db: Database,
-
-    pub height_to_interval: EagerVec<Height, Timestamp>,
-    pub height_to_vbytes: EagerVec<Height, StoredU64>,
-    pub difficultyepoch_to_timestamp: EagerVec<DifficultyEpoch, Timestamp>,
-    pub halvingepoch_to_timestamp: EagerVec<HalvingEpoch, Timestamp>,
-    pub timeindexes_to_timestamp: ComputedVecsFromDateIndex<Timestamp>,
-    pub indexes_to_block_count: ComputedVecsFromHeight<StoredU32>,
-    pub indexes_to_block_interval: ComputedVecsFromHeight<Timestamp>,
-    pub indexes_to_block_size: ComputedVecsFromHeight<StoredU64>,
-    pub indexes_to_block_vbytes: ComputedVecsFromHeight<StoredU64>,
-    pub indexes_to_block_weight: ComputedVecsFromHeight<Weight>,
-}
-
-impl Vecs {
-    pub fn forced_import(parent: &Path, version: Version, indexes: &indexes::Vecs) -> Result<Self> {
-        let db = Database::open(&parent.join("blocks"))?;
-        db.set_min_len(PAGE_SIZE * 1_000_000)?;
-
-        Ok(Self {
-            height_to_interval: EagerVec::forced_import_compressed(
-                &db,
-                "interval",
-                version + VERSION + Version::ZERO,
-            )?,
-            timeindexes_to_timestamp: ComputedVecsFromDateIndex::forced_import(
-                &db,
-                "timestamp",
-                Source::Compute,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_first(),
-            )?,
-            indexes_to_block_interval: ComputedVecsFromHeight::forced_import(
-                &db,
-                "block_interval",
-                Source::None,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default()
-                    .add_percentiles()
-                    .add_minmax()
-                    .add_average(),
-            )?,
-            indexes_to_block_count: ComputedVecsFromHeight::forced_import(
-                &db,
-                "block_count",
-                Source::Compute,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_sum().add_cumulative(),
-            )?,
-            indexes_to_block_weight: ComputedVecsFromHeight::forced_import(
-                &db,
-                "block_weight",
-                Source::None,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_sum().add_cumulative(),
-            )?,
-            indexes_to_block_size: ComputedVecsFromHeight::forced_import(
-                &db,
-                "block_size",
-                Source::None,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_sum().add_cumulative(),
-            )?,
-            height_to_vbytes: EagerVec::forced_import_compressed(
-                &db,
-                "vbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            indexes_to_block_vbytes: ComputedVecsFromHeight::forced_import(
-                &db,
-                "block_vbytes",
-                Source::None,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_sum().add_cumulative(),
-            )?,
-            difficultyepoch_to_timestamp: EagerVec::forced_import_compressed(
-                &db,
-                "timestamp",
-                version + VERSION + Version::ZERO,
-            )?,
-            halvingepoch_to_timestamp: EagerVec::forced_import_compressed(
-                &db,
-                "timestamp",
-                version + VERSION + Version::ZERO,
-            )?,
-
-            db,
-        })
-    }
-
-    pub fn compute(
-        &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        starting_indexes: &Indexes,
-        exit: &Exit,
-    ) -> Result<()> {
-        self.compute_(indexer, indexes, starting_indexes, exit)?;
-        self.db.flush_then_punch()?;
-        Ok(())
-    }
-
-    fn compute_(
-        &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        starting_indexes: &Indexes,
-        exit: &Exit,
-    ) -> Result<()> {
-        self.timeindexes_to_timestamp.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_transform(
-                    starting_indexes.dateindex,
-                    &indexes.dateindex_to_date,
-                    |(di, d, ..)| (di, Timestamp::from(d)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        self.indexes_to_block_count.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |v, indexer, _, starting_indexes, exit| {
-                v.compute_range(
-                    starting_indexes.height,
-                    &indexer.vecs.height_to_weight,
-                    |h| (h, StoredU32::from(1_u32)),
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        let mut height_to_timestamp_iter = indexer.vecs.height_to_timestamp.iter();
-        self.height_to_interval.compute_transform(
-            starting_indexes.height,
-            &indexer.vecs.height_to_timestamp,
-            |(height, timestamp, ..)| {
-                let interval = height.decremented().map_or(Timestamp::ZERO, |prev_h| {
-                    let prev_timestamp = height_to_timestamp_iter.unwrap_get_inner(prev_h);
-                    timestamp
-                        .checked_sub(prev_timestamp)
-                        .unwrap_or(Timestamp::ZERO)
-                });
-                (height, interval)
-            },
-            exit,
-        )?;
-
-        self.indexes_to_block_interval.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&self.height_to_interval),
-        )?;
-
-        self.indexes_to_block_weight.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&indexer.vecs.height_to_weight),
-        )?;
-
-        self.indexes_to_block_size.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&indexer.vecs.height_to_total_size),
-        )?;
-
-        self.height_to_vbytes.compute_transform(
-            starting_indexes.height,
-            &indexer.vecs.height_to_weight,
-            |(h, w, ..)| {
-                (
-                    h,
-                    StoredU64::from(bitcoin::Weight::from(w).to_vbytes_floor()),
-                )
-            },
-            exit,
-        )?;
-
-        self.indexes_to_block_vbytes.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&self.height_to_vbytes),
-        )?;
-
-        let mut height_to_timestamp_iter = indexer.vecs.height_to_timestamp.iter();
-
-        self.difficultyepoch_to_timestamp.compute_transform(
-            starting_indexes.difficultyepoch,
-            &indexes.difficultyepoch_to_first_height,
-            |(i, h, ..)| (i, height_to_timestamp_iter.unwrap_get_inner(h)),
-            exit,
-        )?;
-
-        self.halvingepoch_to_timestamp.compute_transform(
-            starting_indexes.halvingepoch,
-            &indexes.halvingepoch_to_first_height,
-            |(i, h, ..)| (i, height_to_timestamp_iter.unwrap_get_inner(h)),
-            exit,
-        )?;
-
-        Ok(())
-    }
-
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            vec![
-                &self.height_to_interval as &dyn AnyCollectableVec,
-                &self.height_to_vbytes,
-                &self.difficultyepoch_to_timestamp,
-                &self.halvingepoch_to_timestamp,
-            ],
-            self.timeindexes_to_timestamp.vecs(),
-            self.indexes_to_block_count.vecs(),
-            self.indexes_to_block_interval.vecs(),
-            self.indexes_to_block_size.vecs(),
-            self.indexes_to_block_vbytes.vecs(),
-            self.indexes_to_block_weight.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
-    }
-}

crates/brk_computer/src/cointime.rs

@@ -1,21 +1,20 @@
 use std::path::Path;
 
 use brk_error::Result;
-use brk_indexer::Indexer;
-use brk_structs::{Bitcoin, CheckedSub, Dollars, StoredF64, Version};
+use brk_structs::{Bitcoin, CheckedSub, Dollars, StoredF32, StoredF64, Version};
 use vecdb::{AnyCollectableVec, Database, Exit, PAGE_SIZE, VecIterator};
 
+use crate::grouped::ComputedVecsFromDateIndex;
+
 use super::{
-    Indexes,
+    Indexes, chain,
     grouped::{
         ComputedRatioVecsFromDateIndex, ComputedValueVecsFromHeight, ComputedVecsFromHeight,
         Source, VecBuilderOptions,
     },
-    indexes, price, stateful, transactions,
+    indexes, price, stateful,
 };
 
-const VERSION: Version = Version::ZERO;
-
 #[derive(Clone)]
 pub struct Vecs {
     db: Database,
@@ -43,27 +42,32 @@ pub struct Vecs {
     pub indexes_to_cointime_price: ComputedVecsFromHeight<Dollars>,
     pub indexes_to_cointime_cap: ComputedVecsFromHeight<Dollars>,
     pub indexes_to_cointime_price_ratio: ComputedRatioVecsFromDateIndex,
-    // pub indexes_to_thermo_cap_relative_to_investor_cap: ComputedValueVecsFromHeight,
+    pub indexes_to_cointime_adj_inflation_rate: ComputedVecsFromDateIndex<StoredF32>,
+    pub indexes_to_cointime_adj_tx_btc_velocity: ComputedVecsFromDateIndex<StoredF64>,
+    pub indexes_to_cointime_adj_tx_usd_velocity: ComputedVecsFromDateIndex<StoredF64>,
+    // pub indexes_to_thermo_cap_rel_to_investor_cap: ComputedValueVecsFromHeight,
 }
 
 impl Vecs {
     pub fn forced_import(
-        parent: &Path,
-        version: Version,
+        parent_path: &Path,
+        parent_version: Version,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
     ) -> Result<Self> {
-        let db = Database::open(&parent.join("cointime"))?;
+        let db = Database::open(&parent_path.join("cointime"))?;
         db.set_min_len(PAGE_SIZE * 1_000_000)?;
 
         let compute_dollars = price.is_some();
 
-        Ok(Self {
+        let version = parent_version + Version::ZERO;
+
+        let this = Self {
             indexes_to_coinblocks_created: ComputedVecsFromHeight::forced_import(
                 &db,
                 "coinblocks_created",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_sum().add_cumulative(),
             )?,
@@ -71,7 +75,7 @@ impl Vecs {
                 &db,
                 "coinblocks_stored",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_sum().add_cumulative(),
             )?,
@@ -79,7 +83,7 @@ impl Vecs {
                 &db,
                 "liveliness",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -87,7 +91,7 @@ impl Vecs {
                 &db,
                 "vaultedness",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -95,7 +99,7 @@ impl Vecs {
                 &db,
                 "activity_to_vaultedness_ratio",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -103,7 +107,7 @@ impl Vecs {
                 &db,
                 "vaulted_supply",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 VecBuilderOptions::default().add_last(),
                 compute_dollars,
                 indexes,
@@ -112,7 +116,7 @@ impl Vecs {
                 &db,
                 "active_supply",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 VecBuilderOptions::default().add_last(),
                 compute_dollars,
                 indexes,
@@ -121,7 +125,7 @@ impl Vecs {
                 &db,
                 "thermo_cap",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -129,7 +133,7 @@ impl Vecs {
                 &db,
                 "investor_cap",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -137,7 +141,7 @@ impl Vecs {
                 &db,
                 "vaulted_cap",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -145,7 +149,7 @@ impl Vecs {
                 &db,
                 "active_cap",
                 Source::Compute,
-                version + VERSION + Version::ONE,
+                version + Version::ONE,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -153,7 +157,7 @@ impl Vecs {
                 &db,
                 "vaulted_price",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -161,7 +165,7 @@ impl Vecs {
                 &db,
                 "vaulted_price",
                 Source::None,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 true,
             )?,
@@ -169,7 +173,7 @@ impl Vecs {
                 &db,
                 "active_price",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -177,7 +181,7 @@ impl Vecs {
                 &db,
                 "active_price",
                 Source::None,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 true,
             )?,
@@ -185,7 +189,7 @@ impl Vecs {
                 &db,
                 "true_market_mean",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -193,7 +197,7 @@ impl Vecs {
                 &db,
                 "true_market_mean",
                 Source::None,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 true,
             )?,
@@ -201,7 +205,7 @@ impl Vecs {
                 &db,
                 "cointime_value_destroyed",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_sum().add_cumulative(),
             )?,
@@ -209,7 +213,7 @@ impl Vecs {
                 &db,
                 "cointime_value_created",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_sum().add_cumulative(),
             )?,
@@ -217,7 +221,7 @@ impl Vecs {
                 &db,
                 "cointime_value_stored",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_sum().add_cumulative(),
             )?,
@@ -225,7 +229,7 @@ impl Vecs {
                 &db,
                 "cointime_price",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -233,7 +237,7 @@ impl Vecs {
                 &db,
                 "cointime_cap",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -241,35 +245,58 @@ impl Vecs {
                 &db,
                 "cointime_price",
                 Source::None,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 true,
             )?,
+            indexes_to_cointime_adj_inflation_rate: ComputedVecsFromDateIndex::forced_import(
+                &db,
+                "cointime_adj_inflation_rate",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_cointime_adj_tx_btc_velocity: ComputedVecsFromDateIndex::forced_import(
+                &db,
+                "cointime_adj_tx_btc_velocity",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_cointime_adj_tx_usd_velocity: ComputedVecsFromDateIndex::forced_import(
+                &db,
+                "cointime_adj_tx_usd_velocity",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
 
             db,
-        })
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
     }
 
     #[allow(clippy::too_many_arguments)]
     pub fn compute(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         price: Option<&price::Vecs>,
-        transactions: &transactions::Vecs,
+        chain: &chain::Vecs,
         stateful: &stateful::Vecs,
         exit: &Exit,
     ) -> Result<()> {
-        self.compute_(
-            indexer,
-            indexes,
-            starting_indexes,
-            price,
-            transactions,
-            stateful,
-            exit,
-        )?;
+        self.compute_(indexes, starting_indexes, price, chain, stateful, exit)?;
         self.db.flush_then_punch()?;
         Ok(())
     }
@@ -277,22 +304,17 @@ impl Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         price: Option<&price::Vecs>,
-        transactions: &transactions::Vecs,
+        chain: &chain::Vecs,
         stateful: &stateful::Vecs,
         exit: &Exit,
     ) -> Result<()> {
         let circulating_supply = &stateful.utxo_cohorts.all.1.height_to_supply;
 
-        self.indexes_to_coinblocks_created.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, _, starting_indexes, exit| {
+        self.indexes_to_coinblocks_created
+            .compute_all(indexes, starting_indexes, exit, |vec| {
                 vec.compute_transform(
                     starting_indexes.height,
                     circulating_supply,
@@ -300,18 +322,13 @@ impl Vecs {
                     exit,
                 )?;
                 Ok(())
-            },
-        )?;
+            })?;
 
         let indexes_to_coinblocks_destroyed =
             &stateful.utxo_cohorts.all.1.indexes_to_coinblocks_destroyed;
 
-        self.indexes_to_coinblocks_stored.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, _, starting_indexes, exit| {
+        self.indexes_to_coinblocks_stored
+            .compute_all(indexes, starting_indexes, exit, |vec| {
                 let mut coinblocks_destroyed_iter = indexes_to_coinblocks_destroyed
                     .height
                     .as_ref()
@@ -327,15 +344,10 @@ impl Vecs {
                     exit,
                 )?;
                 Ok(())
-            },
-        )?;
+            })?;
 
-        self.indexes_to_liveliness.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, _, starting_indexes, exit| {
+        self.indexes_to_liveliness
+            .compute_all(indexes, starting_indexes, exit, |vec| {
                 vec.compute_divide(
                     starting_indexes.height,
                     indexes_to_coinblocks_destroyed
@@ -347,16 +359,11 @@ impl Vecs {
                     exit,
                 )?;
                 Ok(())
-            },
-        )?;
+            })?;
         let liveliness = &self.indexes_to_liveliness;
 
-        self.indexes_to_vaultedness.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, _, starting_indexes, exit| {
+        self.indexes_to_vaultedness
+            .compute_all(indexes, starting_indexes, exit, |vec| {
                 vec.compute_transform(
                     starting_indexes.height,
                     liveliness.height.as_ref().unwrap(),
@@ -364,16 +371,14 @@ impl Vecs {
                     exit,
                 )?;
                 Ok(())
-            },
-        )?;
+            })?;
         let vaultedness = &self.indexes_to_vaultedness;
 
         self.indexes_to_activity_to_vaultedness_ratio.compute_all(
-            indexer,
             indexes,
             starting_indexes,
             exit,
-            |vec, _, _, starting_indexes, exit| {
+            |vec| {
                 vec.compute_divide(
                     starting_indexes.height,
                     liveliness.height.as_ref().unwrap(),
@@ -385,12 +390,11 @@ impl Vecs {
         )?;
 
         self.indexes_to_vaulted_supply.compute_all(
-            indexer,
             indexes,
             price,
             starting_indexes,
             exit,
-            |vec, _, _, starting_indexes, exit| {
+            |vec| {
                 vec.compute_multiply(
                     starting_indexes.height,
                     circulating_supply,
@@ -402,12 +406,11 @@ impl Vecs {
         )?;
 
         self.indexes_to_active_supply.compute_all(
-            indexer,
             indexes,
             price,
             starting_indexes,
             exit,
-            |vec, _, _, starting_indexes, exit| {
+            |vec| {
                 vec.compute_multiply(
                     starting_indexes.height,
                     circulating_supply,
@@ -418,6 +421,32 @@ impl Vecs {
             },
         )?;
 
+        self.indexes_to_cointime_adj_inflation_rate
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_multiply(
+                    starting_indexes.dateindex,
+                    self.indexes_to_activity_to_vaultedness_ratio
+                        .dateindex
+                        .unwrap_last(),
+                    chain.indexes_to_inflation_rate.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_cointime_adj_tx_btc_velocity
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_multiply(
+                    starting_indexes.dateindex,
+                    self.indexes_to_activity_to_vaultedness_ratio
+                        .dateindex
+                        .unwrap_last(),
+                    chain.indexes_to_tx_btc_velocity.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
         if let Some(price) = price {
             let realized_cap = stateful
                 .utxo_cohorts
@@ -438,15 +467,11 @@ impl Vecs {
                 .as_ref()
                 .unwrap();
 
-            self.indexes_to_thermo_cap.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
+            self.indexes_to_thermo_cap
+                .compute_all(indexes, starting_indexes, exit, |vec| {
                     vec.compute_transform(
                         starting_indexes.height,
-                        transactions
+                        chain
                             .indexes_to_subsidy
                             .dollars
                             .as_ref()
@@ -457,15 +482,10 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
-            )?;
+                })?;
 
-            self.indexes_to_investor_cap.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
+            self.indexes_to_investor_cap
+                .compute_all(indexes, starting_indexes, exit, |vec| {
                     vec.compute_subtract(
                         starting_indexes.height,
                         realized_cap,
@@ -473,15 +493,10 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
-            )?;
+                })?;
 
-            self.indexes_to_vaulted_cap.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
+            self.indexes_to_vaulted_cap
+                .compute_all(indexes, starting_indexes, exit, |vec| {
                     vec.compute_divide(
                         starting_indexes.height,
                         realized_cap,
@@ -489,15 +504,10 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
-            )?;
+                })?;
 
-            self.indexes_to_active_cap.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
+            self.indexes_to_active_cap
+                .compute_all(indexes, starting_indexes, exit, |vec| {
                     vec.compute_multiply(
                         starting_indexes.height,
                         realized_cap,
@@ -505,15 +515,10 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
-            )?;
+                })?;
 
-            self.indexes_to_vaulted_price.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
+            self.indexes_to_vaulted_price
+                .compute_all(indexes, starting_indexes, exit, |vec| {
                     vec.compute_divide(
                         starting_indexes.height,
                         realized_price,
@@ -521,24 +526,17 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
-            )?;
+                })?;
 
             self.indexes_to_vaulted_price_ratio.compute_rest(
-                indexer,
-                indexes,
                 price,
                 starting_indexes,
                 exit,
                 Some(self.indexes_to_vaulted_price.dateindex.unwrap_last()),
             )?;
 
-            self.indexes_to_active_price.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
+            self.indexes_to_active_price
+                .compute_all(indexes, starting_indexes, exit, |vec| {
                     vec.compute_multiply(
                         starting_indexes.height,
                         realized_price,
@@ -546,12 +544,9 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
-            )?;
+                })?;
 
             self.indexes_to_active_price_ratio.compute_rest(
-                indexer,
-                indexes,
                 price,
                 starting_indexes,
                 exit,
@@ -559,11 +554,10 @@ impl Vecs {
             )?;
 
             self.indexes_to_true_market_mean.compute_all(
-                indexer,
                 indexes,
                 starting_indexes,
                 exit,
-                |vec, _, _, starting_indexes, exit| {
+                |vec| {
                     vec.compute_divide(
                         starting_indexes.height,
                         self.indexes_to_investor_cap.height.as_ref().unwrap(),
@@ -579,8 +573,6 @@ impl Vecs {
             )?;
 
             self.indexes_to_true_market_mean_ratio.compute_rest(
-                indexer,
-                indexes,
                 price,
                 starting_indexes,
                 exit,
@@ -588,16 +580,15 @@ impl Vecs {
             )?;
 
             self.indexes_to_cointime_value_destroyed.compute_all(
-                indexer,
                 indexes,
                 starting_indexes,
                 exit,
-                |vec, _, _, starting_indexes, exit| {
+                |vec| {
                     // TODO: Another example when the callback should be applied to each index, instead of to base then merging from more granular to less
                     // The price taken won't be correct for time based indexes
                     vec.compute_multiply(
                         starting_indexes.height,
-                        &price.chainindexes_to_close.height,
+                        &price.chainindexes_to_price_close.height,
                         indexes_to_coinblocks_destroyed.height.as_ref().unwrap(),
                         exit,
                     )?;
@@ -606,14 +597,13 @@ impl Vecs {
             )?;
 
             self.indexes_to_cointime_value_created.compute_all(
-                indexer,
                 indexes,
                 starting_indexes,
                 exit,
-                |vec, _, _, starting_indexes, exit| {
+                |vec| {
                     vec.compute_multiply(
                         starting_indexes.height,
-                        &price.chainindexes_to_close.height,
+                        &price.chainindexes_to_price_close.height,
                         self.indexes_to_coinblocks_created.height.as_ref().unwrap(),
                         exit,
                     )?;
@@ -622,14 +612,13 @@ impl Vecs {
             )?;
 
             self.indexes_to_cointime_value_stored.compute_all(
-                indexer,
                 indexes,
                 starting_indexes,
                 exit,
-                |vec, _, _, starting_indexes, exit| {
+                |vec| {
                     vec.compute_multiply(
                         starting_indexes.height,
-                        &price.chainindexes_to_close.height,
+                        &price.chainindexes_to_price_close.height,
                         self.indexes_to_coinblocks_stored.height.as_ref().unwrap(),
                         exit,
                     )?;
@@ -637,12 +626,8 @@ impl Vecs {
                 },
             )?;
 
-            self.indexes_to_cointime_price.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
+            self.indexes_to_cointime_price
+                .compute_all(indexes, starting_indexes, exit, |vec| {
                     vec.compute_divide(
                         starting_indexes.height,
                         self.indexes_to_cointime_value_destroyed
@@ -654,15 +639,10 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
-            )?;
+                })?;
 
-            self.indexes_to_cointime_cap.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, _, starting_indexes, exit| {
+            self.indexes_to_cointime_cap
+                .compute_all(indexes, starting_indexes, exit, |vec| {
                     vec.compute_multiply(
                         starting_indexes.height,
                         self.indexes_to_cointime_price.height.as_ref().unwrap(),
@@ -670,50 +650,100 @@ impl Vecs {
                         exit,
                     )?;
                     Ok(())
-                },
-            )?;
+                })?;
 
             self.indexes_to_cointime_price_ratio.compute_rest(
-                indexer,
-                indexes,
                 price,
                 starting_indexes,
                 exit,
                 Some(self.indexes_to_cointime_price.dateindex.unwrap_last()),
             )?;
+
+            self.indexes_to_cointime_adj_tx_usd_velocity.compute_all(
+                starting_indexes,
+                exit,
+                |v| {
+                    v.compute_multiply(
+                        starting_indexes.dateindex,
+                        self.indexes_to_activity_to_vaultedness_ratio
+                            .dateindex
+                            .unwrap_last(),
+                        chain.indexes_to_tx_usd_velocity.dateindex.as_ref().unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                },
+            )?;
         }
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.indexes_to_coinblocks_created.vecs(),
-            self.indexes_to_coinblocks_stored.vecs(),
-            self.indexes_to_liveliness.vecs(),
-            self.indexes_to_vaultedness.vecs(),
-            self.indexes_to_activity_to_vaultedness_ratio.vecs(),
-            self.indexes_to_vaulted_supply.vecs(),
-            self.indexes_to_active_supply.vecs(),
-            self.indexes_to_thermo_cap.vecs(),
-            self.indexes_to_investor_cap.vecs(),
-            self.indexes_to_vaulted_cap.vecs(),
-            self.indexes_to_active_cap.vecs(),
-            self.indexes_to_vaulted_price.vecs(),
-            self.indexes_to_vaulted_price_ratio.vecs(),
-            self.indexes_to_active_price.vecs(),
-            self.indexes_to_active_price_ratio.vecs(),
-            self.indexes_to_true_market_mean.vecs(),
-            self.indexes_to_true_market_mean_ratio.vecs(),
-            self.indexes_to_cointime_price.vecs(),
-            self.indexes_to_cointime_cap.vecs(),
-            self.indexes_to_cointime_price_ratio.vecs(),
-            self.indexes_to_cointime_value_destroyed.vecs(),
-            self.indexes_to_cointime_value_created.vecs(),
-            self.indexes_to_cointime_value_stored.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
+
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_adj_inflation_rate
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_adj_tx_btc_velocity
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_adj_tx_usd_velocity
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(iter.chain(self.indexes_to_coinblocks_created.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_coinblocks_stored.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_liveliness.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_vaultedness.iter_any_collectable()));
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_activity_to_vaultedness_ratio
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_supply.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_supply.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_thermo_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_investor_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_price.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_price_ratio.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_price.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_price_ratio.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_true_market_mean.iter_any_collectable()));
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_true_market_mean_ratio
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(iter.chain(self.indexes_to_cointime_price.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_cointime_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_cointime_price_ratio.iter_any_collectable()));
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_value_destroyed
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_value_created
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(iter.chain(self.indexes_to_cointime_value_stored.iter_any_collectable()));
+        iter
     }
 }

crates/brk_computer/src/constants.rs

@@ -1,8 +1,7 @@
 use std::path::Path;
 
 use brk_error::Result;
-use brk_indexer::Indexer;
-use brk_structs::{StoredI16, StoredU16, Version};
+use brk_structs::{StoredF32, StoredI16, StoredU16, Version};
 use vecdb::{AnyCollectableVec, AnyVec, Database, Exit};
 
 use crate::grouped::Source;
@@ -13,8 +12,6 @@ use super::{
     indexes,
 };
 
-const VERSION: Version = Version::ZERO;
-
 #[derive(Clone)]
 pub struct Vecs {
     db: Database,
@@ -24,9 +21,10 @@ pub struct Vecs {
     pub constant_2: ComputedVecsFromHeight<StoredU16>,
     pub constant_3: ComputedVecsFromHeight<StoredU16>,
     pub constant_4: ComputedVecsFromHeight<StoredU16>,
+    pub constant_38_2: ComputedVecsFromHeight<StoredF32>,
     pub constant_50: ComputedVecsFromHeight<StoredU16>,
+    pub constant_61_8: ComputedVecsFromHeight<StoredF32>,
     pub constant_100: ComputedVecsFromHeight<StoredU16>,
-    pub constant_144: ComputedVecsFromHeight<StoredU16>,
     pub constant_600: ComputedVecsFromHeight<StoredU16>,
     pub constant_minus_1: ComputedVecsFromHeight<StoredI16>,
     pub constant_minus_2: ComputedVecsFromHeight<StoredI16>,
@@ -35,15 +33,21 @@ pub struct Vecs {
 }
 
 impl Vecs {
-    pub fn forced_import(parent: &Path, version: Version, indexes: &indexes::Vecs) -> Result<Self> {
-        let db = Database::open(&parent.join("constants"))?;
+    pub fn forced_import(
+        parent_path: &Path,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+    ) -> Result<Self> {
+        let db = Database::open(&parent_path.join("constants"))?;
 
-        Ok(Self {
+        let version = parent_version + Version::ZERO;
+
+        let this = Self {
             constant_0: ComputedVecsFromHeight::forced_import(
                 &db,
                 "constant_0",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -51,7 +55,7 @@ impl Vecs {
                 &db,
                 "constant_1",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -59,7 +63,7 @@ impl Vecs {
                 &db,
                 "constant_2",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -67,7 +71,7 @@ impl Vecs {
                 &db,
                 "constant_3",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -75,7 +79,15 @@ impl Vecs {
                 &db,
                 "constant_4",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_38_2: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_38_2",
+                Source::Compute,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -83,7 +95,15 @@ impl Vecs {
                 &db,
                 "constant_50",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_61_8: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_61_8",
+                Source::Compute,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -91,15 +111,7 @@ impl Vecs {
                 &db,
                 "constant_100",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_last(),
-            )?,
-            constant_144: ComputedVecsFromHeight::forced_import(
-                &db,
-                "constant_144",
-                Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -107,7 +119,7 @@ impl Vecs {
                 &db,
                 "constant_600",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -115,7 +127,7 @@ impl Vecs {
                 &db,
                 "constant_minus_1",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -123,7 +135,7 @@ impl Vecs {
                 &db,
                 "constant_minus_2",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -131,7 +143,7 @@ impl Vecs {
                 &db,
                 "constant_minus_3",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
@@ -139,30 +151,36 @@ impl Vecs {
                 &db,
                 "constant_minus_4",
                 Source::Compute,
-                version + VERSION + Version::ZERO,
+                version + Version::ZERO,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
 
             db,
-        })
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
     }
 
     pub fn compute(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        self.compute_(indexer, indexes, starting_indexes, exit)?;
+        self.compute_(indexes, starting_indexes, exit)?;
         self.db.flush_then_punch()?;
         Ok(())
     }
 
     fn compute_(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
@@ -175,27 +193,20 @@ impl Vecs {
             (&mut self.constant_4, 4),
             (&mut self.constant_50, 50),
             (&mut self.constant_100, 100),
-            (&mut self.constant_144, 144),
             (&mut self.constant_600, 600),
         ]
         .into_iter()
         .try_for_each(|(vec, value)| {
-            vec.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, indexes, starting_indexes, exit| {
-                    vec.compute_to(
-                        starting_indexes.height,
-                        indexes.height_to_date.len(),
-                        indexes.height_to_date.version(),
-                        |i| (i, StoredU16::new(value)),
-                        exit,
-                    )?;
-                    Ok(())
-                },
-            )
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_to(
+                    starting_indexes.height,
+                    indexes.height_to_date.len(),
+                    indexes.height_to_date.version(),
+                    |i| (i, StoredU16::new(value)),
+                    exit,
+                )?;
+                Ok(())
+            })
         })?;
 
         [
@@ -206,45 +217,58 @@ impl Vecs {
         ]
         .into_iter()
         .try_for_each(|(vec, value)| {
-            vec.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |vec, _, indexes, starting_indexes, exit| {
-                    vec.compute_to(
-                        starting_indexes.height,
-                        indexes.height_to_date.len(),
-                        indexes.height_to_date.version(),
-                        |i| (i, StoredI16::new(value)),
-                        exit,
-                    )?;
-                    Ok(())
-                },
-            )
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_to(
+                    starting_indexes.height,
+                    indexes.height_to_date.len(),
+                    indexes.height_to_date.version(),
+                    |i| (i, StoredI16::new(value)),
+                    exit,
+                )?;
+                Ok(())
+            })
+        })?;
+
+        [
+            (&mut self.constant_38_2, 38.2),
+            (&mut self.constant_61_8, 61.8),
+        ]
+        .into_iter()
+        .try_for_each(|(vec, value)| {
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_to(
+                    starting_indexes.height,
+                    indexes.height_to_date.len(),
+                    indexes.height_to_date.version(),
+                    |i| (i, StoredF32::from(value)),
+                    exit,
+                )?;
+                Ok(())
+            })
         })?;
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.constant_0.vecs(),
-            self.constant_1.vecs(),
-            self.constant_2.vecs(),
-            self.constant_3.vecs(),
-            self.constant_4.vecs(),
-            self.constant_50.vecs(),
-            self.constant_100.vecs(),
-            self.constant_144.vecs(),
-            self.constant_600.vecs(),
-            self.constant_minus_1.vecs(),
-            self.constant_minus_2.vecs(),
-            self.constant_minus_3.vecs(),
-            self.constant_minus_4.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
+
+        iter = Box::new(iter.chain(self.constant_0.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_1.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_2.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_3.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_4.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_38_2.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_50.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_61_8.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_100.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_600.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_1.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_2.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_3.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_4.iter_any_collectable()));
+
+        iter
     }
 }

crates/brk_computer/src/fetched.rs

@@ -16,30 +16,38 @@ pub struct Vecs {
     db: Database,
     fetcher: Fetcher,
 
-    pub dateindex_to_ohlc_in_cents: RawVec<DateIndex, OHLCCents>,
-    pub height_to_ohlc_in_cents: RawVec<Height, OHLCCents>,
+    pub dateindex_to_price_ohlc_in_cents: RawVec<DateIndex, OHLCCents>,
+    pub height_to_price_ohlc_in_cents: RawVec<Height, OHLCCents>,
 }
 
 impl Vecs {
     pub fn forced_import(parent: &Path, fetcher: Fetcher, version: Version) -> Result<Self> {
         let db = Database::open(&parent.join("fetched"))?;
 
-        Ok(Self {
+        let this = Self {
             fetcher,
 
-            dateindex_to_ohlc_in_cents: RawVec::forced_import(
+            dateindex_to_price_ohlc_in_cents: RawVec::forced_import(
                 &db,
-                "ohlc_in_cents",
+                "price_ohlc_in_cents",
                 version + Version::ZERO,
             )?,
-            height_to_ohlc_in_cents: RawVec::forced_import(
+            height_to_price_ohlc_in_cents: RawVec::forced_import(
                 &db,
-                "ohlc_in_cents",
+                "price_ohlc_in_cents",
                 version + Version::ZERO,
             )?,
 
             db,
-        })
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
     }
 
     pub fn compute(
@@ -64,12 +72,12 @@ impl Vecs {
         let height_to_timestamp = &indexer.vecs.height_to_timestamp;
         let index = starting_indexes
             .height
-            .min(Height::from(self.height_to_ohlc_in_cents.len()));
+            .min(Height::from(self.height_to_price_ohlc_in_cents.len()));
         height_to_timestamp
             .iter_at(index)
             .try_for_each(|(i, v)| -> Result<()> {
                 let v = v.into_owned();
-                self.height_to_ohlc_in_cents.forced_push_at(
+                self.height_to_price_ohlc_in_cents.forced_push_at(
                     i,
                     self.fetcher
                         .get_height(
@@ -84,11 +92,11 @@ impl Vecs {
                 )?;
                 Ok(())
             })?;
-        self.height_to_ohlc_in_cents.safe_flush(exit)?;
+        self.height_to_price_ohlc_in_cents.safe_flush(exit)?;
 
         let index = starting_indexes
             .dateindex
-            .min(DateIndex::from(self.dateindex_to_ohlc_in_cents.len()));
+            .min(DateIndex::from(self.dateindex_to_price_ohlc_in_cents.len()));
         let mut prev = None;
         indexes
             .dateindex_to_date
@@ -98,7 +106,7 @@ impl Vecs {
                 if prev.is_none() {
                     let i = i.unwrap_to_usize();
                     prev.replace(if i > 0 {
-                        self.dateindex_to_ohlc_in_cents
+                        self.dateindex_to_price_ohlc_in_cents
                             .into_iter()
                             .unwrap_get_inner_(i - 1)
                     } else {
@@ -106,7 +114,8 @@ impl Vecs {
                     });
                 }
 
-                let ohlc = if i.unwrap_to_usize() + 100 >= self.dateindex_to_ohlc_in_cents.len()
+                let ohlc = if i.unwrap_to_usize() + 100
+                    >= self.dateindex_to_price_ohlc_in_cents.len()
                     && let Ok(mut ohlc) = self.fetcher.get_date(d)
                 {
                     let prev_open = *prev.as_ref().unwrap().close;
@@ -120,20 +129,21 @@ impl Vecs {
 
                 prev.replace(ohlc.clone());
 
-                self.dateindex_to_ohlc_in_cents
+                self.dateindex_to_price_ohlc_in_cents
                     .forced_push_at(i, ohlc, exit)?;
 
                 Ok(())
             })?;
-        self.dateindex_to_ohlc_in_cents.safe_flush(exit)?;
+        self.dateindex_to_price_ohlc_in_cents.safe_flush(exit)?;
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        vec![
-            &self.dateindex_to_ohlc_in_cents as &dyn AnyCollectableVec,
-            &self.height_to_ohlc_in_cents,
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [
+            &self.dateindex_to_price_ohlc_in_cents as &dyn AnyCollectableVec,
+            &self.height_to_price_ohlc_in_cents,
         ]
+        .into_iter()
     }
 }

crates/brk_computer/src/grouped/builder_computed.rs

@@ -1,473 +0,0 @@
-use brk_error::Result;
-
-use brk_structs::Version;
-use vecdb::{
-    AnyBoxedIterableVec, AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Computation,
-    ComputedVec, ComputedVecFrom2, Database, Exit, Format, FromCoarserIndex, StoredIndex,
-};
-
-use crate::grouped::{EagerVecBuilder, VecBuilderOptions};
-
-use super::ComputedType;
-
-#[allow(clippy::type_complexity)]
-#[derive(Clone)]
-pub struct ComputedVecBuilder<I, T, S1I, S2T>
-where
-    I: StoredIndex,
-    T: ComputedType,
-    S2T: ComputedType,
-{
-    pub first: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub average: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub sum: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub max: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub min: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub last: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-    pub cumulative: Option<Box<ComputedVecFrom2<I, T, S1I, T, I, S2T>>>,
-}
-
-const VERSION: Version = Version::ZERO;
-
-impl<I, T, S1I, S2T> ComputedVecBuilder<I, T, S1I, S2T>
-where
-    I: StoredIndex,
-    T: ComputedType + 'static,
-    S1I: StoredIndex + 'static + FromCoarserIndex<I>,
-    S2T: ComputedType,
-{
-    #[allow(clippy::too_many_arguments)]
-    pub fn forced_import(
-        db: &Database,
-        name: &str,
-        version: Version,
-        format: Format,
-        computation: Computation,
-        source: Option<AnyBoxedIterableVec<S1I, T>>,
-        source_extra: &EagerVecBuilder<S1I, T>,
-        len_source: AnyBoxedIterableVec<I, S2T>,
-        options: ComputedVecBuilderOptions,
-    ) -> Result<Self> {
-        let only_one_active = options.is_only_one_active();
-
-        let suffix = |s: &str| format!("{name}_{s}");
-
-        let maybe_suffix = |s: &str| {
-            if only_one_active {
-                name.to_string()
-            } else {
-                suffix(s)
-            }
-        };
-
-        Ok(Self {
-            first: options.first.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &maybe_suffix("first"),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra
-                            .first
-                            .as_ref()
-                            .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            source
-                                .next_at(S1I::min_from(i))
-                                .map(|(_, cow)| cow.into_owned())
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            last: options.last.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        name,
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra.last.as_ref().map_or_else(
-                            || {
-                                source
-                                    .as_ref()
-                                    .unwrap_or_else(|| {
-                                        dbg!(db, name, I::to_string());
-                                        panic!()
-                                    })
-                                    .clone()
-                            },
-                            |v| v.clone(),
-                        ),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            source
-                                .next_at(S1I::max_from(i, source.len()))
-                                .map(|(_, cow)| cow.into_owned())
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            min: options.min.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &maybe_suffix("min"),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra
-                            .min
-                            .as_ref()
-                            .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            S1I::inclusive_range_from(i, source.len())
-                                .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
-                                .min()
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            max: options.max.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &maybe_suffix("max"),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra
-                            .max
-                            .as_ref()
-                            .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            S1I::inclusive_range_from(i, source.len())
-                                .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
-                                .max()
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            average: options.average.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &maybe_suffix("average"),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra
-                            .average
-                            .as_ref()
-                            .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            let vec = S1I::inclusive_range_from(i, source.len())
-                                .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
-                                .collect::<Vec<_>>();
-                            if vec.is_empty() {
-                                return None;
-                            }
-                            let mut sum = T::from(0);
-                            let len = vec.len();
-                            vec.into_iter().for_each(|v| sum += v);
-                            Some(sum / len)
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            sum: options.sum.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &(if !options.last && !options.average && !options.min && !options.max {
-                            name.to_string()
-                        } else {
-                            maybe_suffix("sum")
-                        }),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra
-                            .sum
-                            .as_ref()
-                            .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            let vec = S1I::inclusive_range_from(i, source.len())
-                                .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
-                                .collect::<Vec<_>>();
-                            if vec.is_empty() {
-                                return None;
-                            }
-                            let mut sum = T::from(0);
-                            vec.into_iter().for_each(|v| sum += v);
-                            Some(sum)
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-            cumulative: options.cumulative.then(|| {
-                Box::new(
-                    ComputedVec::forced_import_or_init_from_2(
-                        computation,
-                        db,
-                        &suffix("cumulative"),
-                        version + VERSION + Version::ZERO,
-                        format,
-                        source_extra.cumulative.as_ref().unwrap().boxed_clone(),
-                        len_source.clone(),
-                        |i: I, source, len_source| {
-                            if i.unwrap_to_usize() >= len_source.len() {
-                                return None;
-                            }
-                            source
-                                .next_at(S1I::max_from(i, source.len()))
-                                .map(|(_, cow)| cow.into_owned())
-                        },
-                    )
-                    .unwrap(),
-                )
-            }),
-        })
-    }
-
-    pub fn compute_if_necessary<T2>(
-        &mut self,
-        max_from: I,
-        len_source: &impl AnyIterableVec<I, T2>,
-        exit: &Exit,
-    ) -> Result<()> {
-        if let Some(first) = self.first.as_mut() {
-            first.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(last) = self.last.as_mut() {
-            last.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(min) = self.min.as_mut() {
-            min.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(max) = self.max.as_mut() {
-            max.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(average) = self.average.as_mut() {
-            average.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(sum) = self.sum.as_mut() {
-            sum.compute_if_necessary(max_from, len_source, exit)?;
-        }
-        if let Some(cumulative) = self.cumulative.as_mut() {
-            cumulative.compute_if_necessary(max_from, len_source, exit)?;
-        }
-
-        Ok(())
-    }
-
-    pub fn starting_index(&self, max_from: I) -> I {
-        max_from.min(I::from(
-            self.vecs().into_iter().map(|v| v.len()).min().unwrap(),
-        ))
-    }
-
-    pub fn unwrap_first(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.first.as_ref().unwrap()
-    }
-    #[allow(unused)]
-    pub fn unwrap_average(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.average.as_ref().unwrap()
-    }
-    pub fn unwrap_sum(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.sum.as_ref().unwrap()
-    }
-    pub fn unwrap_max(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.max.as_ref().unwrap()
-    }
-    pub fn unwrap_min(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.min.as_ref().unwrap()
-    }
-    pub fn unwrap_last(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.last.as_ref().unwrap()
-    }
-    #[allow(unused)]
-    pub fn unwrap_cumulative(&self) -> &ComputedVecFrom2<I, T, S1I, T, I, S2T> {
-        self.cumulative.as_ref().unwrap()
-    }
-
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        let mut v: Vec<&dyn AnyCollectableVec> = vec![];
-
-        if let Some(first) = self.first.as_ref() {
-            v.push(first.as_ref());
-        }
-        if let Some(last) = self.last.as_ref() {
-            v.push(last.as_ref());
-        }
-        if let Some(min) = self.min.as_ref() {
-            v.push(min.as_ref());
-        }
-        if let Some(max) = self.max.as_ref() {
-            v.push(max.as_ref());
-        }
-        if let Some(average) = self.average.as_ref() {
-            v.push(average.as_ref());
-        }
-        if let Some(sum) = self.sum.as_ref() {
-            v.push(sum.as_ref());
-        }
-        if let Some(cumulative) = self.cumulative.as_ref() {
-            v.push(cumulative.as_ref());
-        }
-
-        v
-    }
-}
-
-#[derive(Default, Clone, Copy)]
-pub struct ComputedVecBuilderOptions {
-    average: bool,
-    sum: bool,
-    max: bool,
-    min: bool,
-    first: bool,
-    last: bool,
-    cumulative: bool,
-}
-
-impl From<VecBuilderOptions> for ComputedVecBuilderOptions {
-    fn from(value: VecBuilderOptions) -> Self {
-        Self {
-            average: value.average(),
-            sum: value.sum(),
-            max: value.max(),
-            min: value.min(),
-            first: value.first(),
-            last: value.last(),
-            cumulative: value.cumulative(),
-        }
-    }
-}
-
-impl ComputedVecBuilderOptions {
-    pub fn add_first(mut self) -> Self {
-        self.first = true;
-        self
-    }
-
-    pub fn add_last(mut self) -> Self {
-        self.last = true;
-        self
-    }
-
-    pub fn add_min(mut self) -> Self {
-        self.min = true;
-        self
-    }
-
-    pub fn add_max(mut self) -> Self {
-        self.max = true;
-        self
-    }
-
-    pub fn add_average(mut self) -> Self {
-        self.average = true;
-        self
-    }
-
-    pub fn add_sum(mut self) -> Self {
-        self.sum = true;
-        self
-    }
-
-    pub fn add_cumulative(mut self) -> Self {
-        self.cumulative = true;
-        self
-    }
-
-    #[allow(unused)]
-    pub fn rm_min(mut self) -> Self {
-        self.min = false;
-        self
-    }
-
-    #[allow(unused)]
-    pub fn rm_max(mut self) -> Self {
-        self.max = false;
-        self
-    }
-
-    #[allow(unused)]
-    pub fn rm_average(mut self) -> Self {
-        self.average = false;
-        self
-    }
-
-    #[allow(unused)]
-    pub fn rm_sum(mut self) -> Self {
-        self.sum = false;
-        self
-    }
-
-    #[allow(unused)]
-    pub fn rm_cumulative(mut self) -> Self {
-        self.cumulative = false;
-        self
-    }
-
-    pub fn add_minmax(mut self) -> Self {
-        self.min = true;
-        self.max = true;
-        self
-    }
-
-    pub fn is_only_one_active(&self) -> bool {
-        [
-            self.average,
-            self.sum,
-            self.max,
-            self.min,
-            self.first,
-            self.last,
-            self.cumulative,
-        ]
-        .iter()
-        .filter(|b| **b)
-        .count()
-            == 1
-    }
-
-    pub fn copy_self_extra(&self) -> Self {
-        Self {
-            cumulative: self.cumulative,
-            ..Self::default()
-        }
-    }
-}

crates/brk_computer/src/grouped/builder_eager.rs

@@ -1,3 +1,4 @@
+use allocative::Allocative;
 use brk_error::{Error, Result};
 use brk_structs::{CheckedSub, StoredU64, Version};
 use vecdb::{
@@ -9,7 +10,7 @@ use crate::utils::get_percentile;
 
 use super::ComputedType;
 
-#[derive(Clone, Debug)]
+#[derive(Clone, Debug, Allocative)]
 pub struct EagerVecBuilder<I, T>
 where
     I: StoredIndex,
@@ -19,11 +20,11 @@ where
     pub average: Option<Box<EagerVec<I, T>>>,
     pub sum: Option<Box<EagerVec<I, T>>>,
     pub max: Option<Box<EagerVec<I, T>>>,
-    pub _90p: Option<Box<EagerVec<I, T>>>,
-    pub _75p: Option<Box<EagerVec<I, T>>>,
+    pub pct90: Option<Box<EagerVec<I, T>>>,
+    pub pct75: Option<Box<EagerVec<I, T>>>,
     pub median: Option<Box<EagerVec<I, T>>>,
-    pub _25p: Option<Box<EagerVec<I, T>>>,
-    pub _10p: Option<Box<EagerVec<I, T>>>,
+    pub pct25: Option<Box<EagerVec<I, T>>>,
+    pub pct10: Option<Box<EagerVec<I, T>>>,
     pub min: Option<Box<EagerVec<I, T>>>,
     pub last: Option<Box<EagerVec<I, T>>>,
     pub cumulative: Option<Box<EagerVec<I, T>>>,
@@ -118,7 +119,7 @@ where
                 Box::new(
                     EagerVec::forced_import(
                         db,
-                        &maybe_suffix("average"),
+                        &maybe_suffix("avg"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
@@ -151,44 +152,44 @@ where
                     .unwrap(),
                 )
             }),
-            _90p: options._90p.then(|| {
+            pct90: options.pct90.then(|| {
                 Box::new(
                     EagerVec::forced_import(
                         db,
-                        &maybe_suffix("90p"),
+                        &maybe_suffix("pct90"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _75p: options._75p.then(|| {
+            pct75: options.pct75.then(|| {
                 Box::new(
                     EagerVec::forced_import(
                         db,
-                        &maybe_suffix("75p"),
+                        &maybe_suffix("pct75"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _25p: options._25p.then(|| {
+            pct25: options.pct25.then(|| {
                 Box::new(
                     EagerVec::forced_import(
                         db,
-                        &maybe_suffix("25p"),
+                        &maybe_suffix("pct25"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _10p: options._10p.then(|| {
+            pct10: options.pct10.then(|| {
                 Box::new(
                     EagerVec::forced_import(
                         db,
-                        &maybe_suffix("10p"),
+                        &maybe_suffix("pct10"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
@@ -292,11 +293,11 @@ where
                 let needs_average_sum_or_cumulative =
                     needs_sum_or_cumulative || self.average.is_some();
                 let needs_sorted = self.max.is_some()
-                    || self._90p.is_some()
-                    || self._75p.is_some()
+                    || self.pct90.is_some()
+                    || self.pct75.is_some()
                     || self.median.is_some()
-                    || self._25p.is_some()
-                    || self._10p.is_some()
+                    || self.pct25.is_some()
+                    || self.pct10.is_some()
                     || self.min.is_some();
                 let needs_values = needs_sorted || needs_average_sum_or_cumulative;
 
@@ -333,24 +334,24 @@ where
                             )?;
                         }
 
-                        if let Some(_90p) = self._90p.as_mut() {
-                            _90p.forced_push_at(index, get_percentile(&values, 0.90), exit)?;
+                        if let Some(pct90) = self.pct90.as_mut() {
+                            pct90.forced_push_at(index, get_percentile(&values, 0.90), exit)?;
                         }
 
-                        if let Some(_75p) = self._75p.as_mut() {
-                            _75p.forced_push_at(index, get_percentile(&values, 0.75), exit)?;
+                        if let Some(pct75) = self.pct75.as_mut() {
+                            pct75.forced_push_at(index, get_percentile(&values, 0.75), exit)?;
                         }
 
                         if let Some(median) = self.median.as_mut() {
                             median.forced_push_at(index, get_percentile(&values, 0.50), exit)?;
                         }
 
-                        if let Some(_25p) = self._25p.as_mut() {
-                            _25p.forced_push_at(index, get_percentile(&values, 0.25), exit)?;
+                        if let Some(pct25) = self.pct25.as_mut() {
+                            pct25.forced_push_at(index, get_percentile(&values, 0.25), exit)?;
                         }
 
-                        if let Some(_10p) = self._10p.as_mut() {
-                            _10p.forced_push_at(index, get_percentile(&values, 0.10), exit)?;
+                        if let Some(pct10) = self.pct10.as_mut() {
+                            pct10.forced_push_at(index, get_percentile(&values, 0.10), exit)?;
                         }
 
                         if let Some(min) = self.min.as_mut() {
@@ -401,11 +402,11 @@ where
     where
         I2: StoredIndex + StoredRaw + CheckedSub<I2>,
     {
-        if self._90p.is_some()
-            || self._75p.is_some()
+        if self.pct90.is_some()
+            || self.pct75.is_some()
             || self.median.is_some()
-            || self._25p.is_some()
-            || self._10p.is_some()
+            || self.pct25.is_some()
+            || self.pct10.is_some()
         {
             panic!("unsupported");
         }
@@ -540,7 +541,7 @@ where
 
     pub fn starting_index(&self, max_from: I) -> I {
         max_from.min(I::from(
-            self.vecs().into_iter().map(|v| v.len()).min().unwrap(),
+            self.iter_any_collectable().map(|v| v.len()).min().unwrap(),
         ))
     }
 
@@ -558,24 +559,24 @@ where
         self.max.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_90p(&self) -> &EagerVec<I, T> {
-        self._90p.as_ref().unwrap()
+    pub fn unwrap_pct90(&self) -> &EagerVec<I, T> {
+        self.pct90.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_75p(&self) -> &EagerVec<I, T> {
-        self._75p.as_ref().unwrap()
+    pub fn unwrap_pct75(&self) -> &EagerVec<I, T> {
+        self.pct75.as_ref().unwrap()
     }
     #[allow(unused)]
     pub fn unwrap_median(&self) -> &EagerVec<I, T> {
         self.median.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_25p(&self) -> &EagerVec<I, T> {
-        self._25p.as_ref().unwrap()
+    pub fn unwrap_pct25(&self) -> &EagerVec<I, T> {
+        self.pct25.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_10p(&self) -> &EagerVec<I, T> {
-        self._10p.as_ref().unwrap()
+    pub fn unwrap_pct10(&self) -> &EagerVec<I, T> {
+        self.pct10.as_ref().unwrap()
     }
     pub fn unwrap_min(&self) -> &EagerVec<I, T> {
         self.min.as_ref().unwrap()
@@ -588,47 +589,96 @@ where
         self.cumulative.as_ref().unwrap()
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        let mut v: Vec<&dyn AnyCollectableVec> = vec![];
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
 
-        if let Some(first) = self.first.as_ref() {
-            v.push(first.as_ref());
-        }
-        if let Some(last) = self.last.as_ref() {
-            v.push(last.as_ref());
-        }
-        if let Some(min) = self.min.as_ref() {
-            v.push(min.as_ref());
-        }
-        if let Some(max) = self.max.as_ref() {
-            v.push(max.as_ref());
-        }
-        if let Some(median) = self.median.as_ref() {
-            v.push(median.as_ref());
-        }
-        if let Some(average) = self.average.as_ref() {
-            v.push(average.as_ref());
-        }
-        if let Some(sum) = self.sum.as_ref() {
-            v.push(sum.as_ref());
-        }
-        if let Some(cumulative) = self.cumulative.as_ref() {
-            v.push(cumulative.as_ref());
-        }
-        if let Some(_90p) = self._90p.as_ref() {
-            v.push(_90p.as_ref());
-        }
-        if let Some(_75p) = self._75p.as_ref() {
-            v.push(_75p.as_ref());
-        }
-        if let Some(_25p) = self._25p.as_ref() {
-            v.push(_25p.as_ref());
-        }
-        if let Some(_10p) = self._10p.as_ref() {
-            v.push(_10p.as_ref());
-        }
+        iter = Box::new(
+            iter.chain(
+                self.first
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.last
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.min
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.max
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.median
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.average
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.sum
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.cumulative
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct90
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct75
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct25
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct10
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
 
-        v
+        iter
     }
 
     pub fn safe_flush(&mut self, exit: &Exit) -> Result<()> {
@@ -656,17 +706,17 @@ where
         if let Some(cumulative) = self.cumulative.as_mut() {
             cumulative.safe_flush(exit)?;
         }
-        if let Some(_90p) = self._90p.as_mut() {
-            _90p.safe_flush(exit)?;
+        if let Some(pct90) = self.pct90.as_mut() {
+            pct90.safe_flush(exit)?;
         }
-        if let Some(_75p) = self._75p.as_mut() {
-            _75p.safe_flush(exit)?;
+        if let Some(pct75) = self.pct75.as_mut() {
+            pct75.safe_flush(exit)?;
         }
-        if let Some(_25p) = self._25p.as_mut() {
-            _25p.safe_flush(exit)?;
+        if let Some(pct25) = self.pct25.as_mut() {
+            pct25.safe_flush(exit)?;
         }
-        if let Some(_10p) = self._10p.as_mut() {
-            _10p.safe_flush(exit)?;
+        if let Some(pct10) = self.pct10.as_mut() {
+            pct10.safe_flush(exit)?;
         }
 
         Ok(())
@@ -697,17 +747,17 @@ where
         if let Some(cumulative) = self.cumulative.as_mut() {
             cumulative.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_90p) = self._90p.as_mut() {
-            _90p.validate_computed_version_or_reset(Version::ZERO + version)?;
+        if let Some(pct90) = self.pct90.as_mut() {
+            pct90.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_75p) = self._75p.as_mut() {
-            _75p.validate_computed_version_or_reset(Version::ZERO + version)?;
+        if let Some(pct75) = self.pct75.as_mut() {
+            pct75.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_25p) = self._25p.as_mut() {
-            _25p.validate_computed_version_or_reset(Version::ZERO + version)?;
+        if let Some(pct25) = self.pct25.as_mut() {
+            pct25.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_10p) = self._10p.as_mut() {
-            _10p.validate_computed_version_or_reset(Version::ZERO + version)?;
+        if let Some(pct10) = self.pct10.as_mut() {
+            pct10.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
 
         Ok(())
@@ -719,11 +769,11 @@ pub struct VecBuilderOptions {
     average: bool,
     sum: bool,
     max: bool,
-    _90p: bool,
-    _75p: bool,
+    pct90: bool,
+    pct75: bool,
     median: bool,
-    _25p: bool,
-    _10p: bool,
+    pct25: bool,
+    pct10: bool,
     min: bool,
     first: bool,
     last: bool,
@@ -743,24 +793,24 @@ impl VecBuilderOptions {
         self.max
     }
 
-    pub fn _90p(&self) -> bool {
-        self._90p
+    pub fn pct90(&self) -> bool {
+        self.pct90
     }
 
-    pub fn _75p(&self) -> bool {
-        self._75p
+    pub fn pct75(&self) -> bool {
+        self.pct75
     }
 
     pub fn median(&self) -> bool {
         self.median
     }
 
-    pub fn _25p(&self) -> bool {
-        self._25p
+    pub fn pct25(&self) -> bool {
+        self.pct25
     }
 
-    pub fn _10p(&self) -> bool {
-        self._10p
+    pub fn pct10(&self) -> bool {
+        self.pct10
     }
 
     pub fn min(&self) -> bool {
@@ -816,26 +866,26 @@ impl VecBuilderOptions {
     }
 
     #[allow(unused)]
-    pub fn add_90p(mut self) -> Self {
-        self._90p = true;
+    pub fn add_pct90(mut self) -> Self {
+        self.pct90 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_75p(mut self) -> Self {
-        self._75p = true;
+    pub fn add_pct75(mut self) -> Self {
+        self.pct75 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_25p(mut self) -> Self {
-        self._25p = true;
+    pub fn add_pct25(mut self) -> Self {
+        self.pct25 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_10p(mut self) -> Self {
-        self._10p = true;
+    pub fn add_pct10(mut self) -> Self {
+        self.pct10 = true;
         self
     }
 
@@ -875,26 +925,26 @@ impl VecBuilderOptions {
     }
 
     #[allow(unused)]
-    pub fn rm_90p(mut self) -> Self {
-        self._90p = false;
+    pub fn rm_pct90(mut self) -> Self {
+        self.pct90 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_75p(mut self) -> Self {
-        self._75p = false;
+    pub fn rm_pct75(mut self) -> Self {
+        self.pct75 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_25p(mut self) -> Self {
-        self._25p = false;
+    pub fn rm_pct25(mut self) -> Self {
+        self.pct25 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_10p(mut self) -> Self {
-        self._10p = false;
+    pub fn rm_pct10(mut self) -> Self {
+        self.pct10 = false;
         self
     }
 
@@ -911,20 +961,20 @@ impl VecBuilderOptions {
     }
 
     pub fn add_percentiles(mut self) -> Self {
-        self._90p = true;
-        self._75p = true;
+        self.pct90 = true;
+        self.pct75 = true;
         self.median = true;
-        self._25p = true;
-        self._10p = true;
+        self.pct25 = true;
+        self.pct10 = true;
         self
     }
 
     pub fn remove_percentiles(mut self) -> Self {
-        self._90p = false;
-        self._75p = false;
+        self.pct90 = false;
+        self.pct75 = false;
         self.median = false;
-        self._25p = false;
-        self._10p = false;
+        self.pct25 = false;
+        self.pct10 = false;
         self
     }
 
@@ -933,11 +983,11 @@ impl VecBuilderOptions {
             self.average,
             self.sum,
             self.max,
-            self._90p,
-            self._75p,
+            self.pct90,
+            self.pct75,
             self.median,
-            self._25p,
-            self._10p,
+            self.pct25,
+            self.pct10,
             self.min,
             self.first,
             self.last,

crates/brk_computer/src/grouped/builder_lazy.rs

@@ -1,3 +1,4 @@
+use allocative::Allocative;
 use brk_structs::Version;
 use vecdb::{
     AnyBoxedIterableVec, AnyCloneableIterableVec, AnyCollectableVec, FromCoarserIndex,
@@ -9,7 +10,7 @@ use crate::grouped::{EagerVecBuilder, VecBuilderOptions};
 use super::ComputedType;
 
 #[allow(clippy::type_complexity)]
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct LazyVecBuilder<I, T, S1I, S2T>
 where
     I: StoredIndex,
@@ -142,7 +143,7 @@ where
             }),
             average: options.average.then(|| {
                 Box::new(LazyVecFrom2::init(
-                    &maybe_suffix("average"),
+                    &maybe_suffix("avg"),
                     version + VERSION + Version::ZERO,
                     source_extra
                         .average
@@ -216,7 +217,7 @@ where
 
     pub fn starting_index(&self, max_from: I) -> I {
         max_from.min(I::from(
-            self.vecs().into_iter().map(|v| v.len()).min().unwrap(),
+            self.iter_any_collectable().map(|v| v.len()).min().unwrap(),
         ))
     }
 
@@ -244,32 +245,61 @@ where
         self.cumulative.as_ref().unwrap()
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        let mut v: Vec<&dyn AnyCollectableVec> = vec![];
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
 
-        if let Some(first) = self.first.as_ref() {
-            v.push(first.as_ref());
-        }
-        if let Some(last) = self.last.as_ref() {
-            v.push(last.as_ref());
-        }
-        if let Some(min) = self.min.as_ref() {
-            v.push(min.as_ref());
-        }
-        if let Some(max) = self.max.as_ref() {
-            v.push(max.as_ref());
-        }
-        if let Some(average) = self.average.as_ref() {
-            v.push(average.as_ref());
-        }
-        if let Some(sum) = self.sum.as_ref() {
-            v.push(sum.as_ref());
-        }
-        if let Some(cumulative) = self.cumulative.as_ref() {
-            v.push(cumulative.as_ref());
-        }
+        iter = Box::new(
+            iter.chain(
+                self.first
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.last
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.min
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.max
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.average
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.sum
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.cumulative
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
 
-        v
+        iter
     }
 }
 

crates/brk_computer/src/grouped/from_dateindex.rs

@@ -1,6 +1,6 @@
+use allocative::Allocative;
 use brk_error::Result;
 
-use brk_indexer::Indexer;
 use brk_structs::{
     DateIndex, DecadeIndex, MonthIndex, QuarterIndex, SemesterIndex, Version, WeekIndex, YearIndex,
 };
@@ -10,7 +10,7 @@ use crate::{Indexes, grouped::LazyVecBuilder, indexes};
 
 use super::{ComputedType, EagerVecBuilder, Source, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedVecsFromDateIndex<T>
 where
     T: ComputedType + PartialOrd,
@@ -111,28 +111,14 @@ where
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(
-            &mut EagerVec<DateIndex, T>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
+        F: FnMut(&mut EagerVec<DateIndex, T>) -> Result<()>,
     {
-        compute(
-            self.dateindex.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
+        compute(self.dateindex.as_mut().unwrap())?;
 
         let dateindex: Option<&EagerVec<DateIndex, T>> = None;
         self.compute_rest(starting_indexes, exit, dateindex)
@@ -157,21 +143,22 @@ where
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
             self.dateindex
                 .as_ref()
-                .map_or(vec![], |v| vec![v as &dyn AnyCollectableVec]),
-            self.dateindex_extra.vecs(),
-            self.weekindex.vecs(),
-            self.monthindex.vecs(),
-            self.quarterindex.vecs(),
-            self.semesterindex.vecs(),
-            self.yearindex.vecs(),
-            self.decadeindex.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+                .map(|x| x as &dyn AnyCollectableVec)
+                .into_iter(),
+        );
+
+        iter = Box::new(iter.chain(self.dateindex_extra.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
+
+        iter
     }
 }

crates/brk_computer/src/grouped/from_height.rs

@@ -1,6 +1,6 @@
+use allocative::Allocative;
 use brk_error::Result;
 
-use brk_indexer::Indexer;
 use brk_structs::{
     DateIndex, DecadeIndex, DifficultyEpoch, Height, MonthIndex, QuarterIndex, SemesterIndex,
     Version, WeekIndex, YearIndex,
@@ -15,7 +15,7 @@ use crate::{
 
 use super::{ComputedType, EagerVecBuilder, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedVecsFromHeight<T>
 where
     T: ComputedType + PartialOrd,
@@ -133,22 +133,15 @@ where
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(&mut EagerVec<Height, T>, &Indexer, &indexes::Vecs, &Indexes, &Exit) -> Result<()>,
+        F: FnMut(&mut EagerVec<Height, T>) -> Result<()>,
     {
-        compute(
-            self.height.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
+        compute(self.height.as_mut().unwrap())?;
 
         let height: Option<&EagerVec<Height, T>> = None;
         self.compute_rest(indexes, starting_indexes, exit, height)
@@ -206,24 +199,24 @@ where
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
             self.height
                 .as_ref()
-                .map_or(vec![], |v| vec![v as &dyn AnyCollectableVec]),
-            self.height_extra.vecs(),
-            self.dateindex.vecs(),
-            self.weekindex.vecs(),
-            self.difficultyepoch.vecs(),
-            self.monthindex.vecs(),
-            self.quarterindex.vecs(),
-            self.semesterindex.vecs(),
-            self.yearindex.vecs(),
-            // self.halvingepoch.vecs(),
-            self.decadeindex.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+                .map(|x| x as &dyn AnyCollectableVec)
+                .into_iter(),
+        );
+
+        iter = Box::new(iter.chain(self.height_extra.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.dateindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.difficultyepoch.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
+
+        iter
     }
 }

crates/brk_computer/src/grouped/from_height_strict.rs

@@ -1,6 +1,5 @@
 use brk_error::Result;
 
-use brk_indexer::Indexer;
 use brk_structs::{DifficultyEpoch, Height, Version};
 use vecdb::{AnyCollectableVec, Database, EagerVec, Exit};
 
@@ -59,16 +58,15 @@ where
 
     pub fn compute<F>(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(&mut EagerVec<Height, T>, &Indexer, &indexes::Vecs, &Indexes, &Exit) -> Result<()>,
+        F: FnMut(&mut EagerVec<Height, T>) -> Result<()>,
     {
-        compute(&mut self.height, indexer, indexes, starting_indexes, exit)?;
+        compute(&mut self.height)?;
 
         self.height_extra
             .extend(starting_indexes.height, &self.height, exit)?;
@@ -84,15 +82,10 @@ where
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            vec![&self.height as &dyn AnyCollectableVec],
-            self.height_extra.vecs(),
-            self.difficultyepoch.vecs(),
-            // self.halvingepoch.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.height as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(self.height_extra.iter_any_collectable())
+            .chain(self.difficultyepoch.iter_any_collectable())
     }
 }

crates/brk_computer/src/grouped/from_txindex.rs

@@ -1,3 +1,4 @@
+use allocative::Allocative;
 use brk_error::Result;
 use brk_indexer::Indexer;
 use brk_structs::{
@@ -17,7 +18,7 @@ use crate::{
 
 use super::{ComputedType, EagerVecBuilder, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedVecsFromTxindex<T>
 where
     T: ComputedType + PartialOrd,
@@ -225,25 +226,25 @@ where
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
             self.txindex
                 .as_ref()
-                .map_or(vec![], |v| vec![v.as_ref() as &dyn AnyCollectableVec]),
-            self.height.vecs(),
-            self.dateindex.vecs(),
-            self.weekindex.vecs(),
-            self.difficultyepoch.vecs(),
-            self.monthindex.vecs(),
-            self.quarterindex.vecs(),
-            self.semesterindex.vecs(),
-            self.yearindex.vecs(),
-            // self.halvingepoch.vecs(),
-            self.decadeindex.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+                .map(|x| x.as_ref() as &dyn AnyCollectableVec)
+                .into_iter(),
+        );
+
+        iter = Box::new(iter.chain(self.height.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.dateindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.difficultyepoch.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
+
+        iter
     }
 }
 
@@ -319,24 +320,24 @@ impl ComputedVecsFromTxindex<Bitcoin> {
                         exit,
                     )?;
                 }
-                if let Some(_90p) = self.height._90p.as_mut() {
-                    _90p.forced_push_at(
+                if let Some(pct90) = self.height.pct90.as_mut() {
+                    pct90.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_90p()
+                                .unwrap_pct90()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
                         exit,
                     )?;
                 }
-                if let Some(_75p) = self.height._75p.as_mut() {
-                    _75p.forced_push_at(
+                if let Some(pct75) = self.height.pct75.as_mut() {
+                    pct75.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_75p()
+                                .unwrap_pct75()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
@@ -355,24 +356,24 @@ impl ComputedVecsFromTxindex<Bitcoin> {
                         exit,
                     )?;
                 }
-                if let Some(_25p) = self.height._25p.as_mut() {
-                    _25p.forced_push_at(
+                if let Some(pct25) = self.height.pct25.as_mut() {
+                    pct25.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_25p()
+                                .unwrap_pct25()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
                         exit,
                     )?;
                 }
-                if let Some(_10p) = self.height._10p.as_mut() {
-                    _10p.forced_push_at(
+                if let Some(pct10) = self.height.pct10.as_mut() {
+                    pct10.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_10p()
+                                .unwrap_pct10()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
@@ -447,7 +448,7 @@ impl ComputedVecsFromTxindex<Dollars> {
 
         let starting_index = self.height.starting_index(starting_indexes.height);
 
-        let mut close_iter = price.chainindexes_to_close.height.into_iter();
+        let mut close_iter = price.chainindexes_to_price_close.height.into_iter();
 
         (starting_index.unwrap_to_usize()..indexer.vecs.height_to_weight.len())
             .map(Height::from)
@@ -502,25 +503,25 @@ impl ComputedVecsFromTxindex<Dollars> {
                         exit,
                     )?;
                 }
-                if let Some(_90p) = self.height._90p.as_mut() {
-                    _90p.forced_push_at(
+                if let Some(pct90) = self.height.pct90.as_mut() {
+                    pct90.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_90p()
+                                .unwrap_pct90()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
                     )?;
                 }
-                if let Some(_75p) = self.height._75p.as_mut() {
-                    _75p.forced_push_at(
+                if let Some(pct75) = self.height.pct75.as_mut() {
+                    pct75.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_75p()
+                                .unwrap_pct75()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
@@ -538,25 +539,25 @@ impl ComputedVecsFromTxindex<Dollars> {
                         exit,
                     )?;
                 }
-                if let Some(_25p) = self.height._25p.as_mut() {
-                    _25p.forced_push_at(
+                if let Some(pct25) = self.height.pct25.as_mut() {
+                    pct25.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_25p()
+                                .unwrap_pct25()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
                     )?;
                 }
-                if let Some(_10p) = self.height._10p.as_mut() {
-                    _10p.forced_push_at(
+                if let Some(pct10) = self.height.pct10.as_mut() {
+                    pct10.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_10p()
+                                .unwrap_pct10()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,

crates/brk_computer/src/grouped/ratio_from_dateindex.rs

@@ -1,5 +1,4 @@
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Date, DateIndex, Dollars, StoredF32, Version};
 use vecdb::{
     AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, CollectableVec, Database, EagerVec,
@@ -8,7 +7,9 @@ use vecdb::{
 
 use crate::{
     Indexes,
-    grouped::{ComputedStandardDeviationVecsFromDateIndex, source::Source},
+    grouped::{
+        ComputedStandardDeviationVecsFromDateIndex, StandardDeviationVecsOptions, source::Source,
+    },
     indexes, price,
     utils::get_percentile,
 };
@@ -22,18 +23,18 @@ pub struct ComputedRatioVecsFromDateIndex {
     pub ratio: ComputedVecsFromDateIndex<StoredF32>,
     pub ratio_1w_sma: Option<ComputedVecsFromDateIndex<StoredF32>>,
     pub ratio_1m_sma: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p99: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p98: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p95: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p5: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p2: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p1: Option<ComputedVecsFromDateIndex<StoredF32>>,
-    pub ratio_p99_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
-    pub ratio_p98_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
-    pub ratio_p95_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
-    pub ratio_p5_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
-    pub ratio_p2_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
-    pub ratio_p1_as_price: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct99: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct98: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct95: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct5: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct2: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct1: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct99_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct98_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct95_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct5_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct2_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct1_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
 
     pub ratio_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
     pub ratio_4y_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
@@ -105,6 +106,7 @@ impl ComputedRatioVecsFromDateIndex {
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
                 )
                 .unwrap()
             }),
@@ -116,6 +118,7 @@ impl ComputedRatioVecsFromDateIndex {
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
                 )
                 .unwrap()
             }),
@@ -127,6 +130,7 @@ impl ComputedRatioVecsFromDateIndex {
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
                 )
                 .unwrap()
             }),
@@ -138,13 +142,14 @@ impl ComputedRatioVecsFromDateIndex {
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
                 )
                 .unwrap()
             }),
-            ratio_p99: extended.then(|| {
+            ratio_pct99: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p99"),
+                    &format!("{name}_ratio_pct99"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -152,10 +157,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p98: extended.then(|| {
+            ratio_pct98: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p98"),
+                    &format!("{name}_ratio_pct98"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -163,10 +168,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p95: extended.then(|| {
+            ratio_pct95: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p95"),
+                    &format!("{name}_ratio_pct95"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -174,10 +179,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p5: extended.then(|| {
+            ratio_pct5: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p5"),
+                    &format!("{name}_ratio_pct5"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -185,10 +190,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p2: extended.then(|| {
+            ratio_pct2: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p2"),
+                    &format!("{name}_ratio_pct2"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -196,10 +201,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p1: extended.then(|| {
+            ratio_pct1: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p1"),
+                    &format!("{name}_ratio_pct1"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -207,10 +212,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p99_as_price: extended.then(|| {
+            ratio_pct99_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p99_as_price"),
+                    &format!("{name}_ratio_pct99_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -218,10 +223,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p98_as_price: extended.then(|| {
+            ratio_pct98_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p98_as_price"),
+                    &format!("{name}_ratio_pct98_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -229,10 +234,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p95_as_price: extended.then(|| {
+            ratio_pct95_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p95_as_price"),
+                    &format!("{name}_ratio_pct95_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -240,10 +245,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p5_as_price: extended.then(|| {
+            ratio_pct5_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p5_as_price"),
+                    &format!("{name}_ratio_pct5_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -251,10 +256,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p2_as_price: extended.then(|| {
+            ratio_pct2_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p2_as_price"),
+                    &format!("{name}_ratio_pct2_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -262,10 +267,10 @@ impl ComputedRatioVecsFromDateIndex {
                 )
                 .unwrap()
             }),
-            ratio_p1_as_price: extended.then(|| {
+            ratio_pct1_usd: extended.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_ratio_p1_as_price"),
+                    &format!("{name}_ratio_pct1_usd"),
                     Source::Compute,
                     version + VERSION + Version::ZERO,
                     indexes,
@@ -278,78 +283,52 @@ impl ComputedRatioVecsFromDateIndex {
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         price: &price::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         compute: F,
     ) -> Result<()>
     where
-        F: FnMut(
-            &mut EagerVec<DateIndex, Dollars>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
+        F: FnMut(&mut EagerVec<DateIndex, Dollars>) -> Result<()>,
     {
-        self.price.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            compute,
-        )?;
+        self.price
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, compute)?;
 
         let date_to_price_opt: Option<&EagerVec<DateIndex, Dollars>> = None;
-        self.compute_rest(
-            indexer,
-            indexes,
-            price,
-            starting_indexes,
-            exit,
-            date_to_price_opt,
-        )
+        self.compute_rest(price, starting_indexes, exit, date_to_price_opt)
     }
 
     pub fn compute_rest(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         price: &price::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         price_opt: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
     ) -> Result<()> {
-        let closes = price.timeindexes_to_close.dateindex.as_ref().unwrap();
+        let closes = price.timeindexes_to_price_close.dateindex.as_ref().unwrap();
 
         let price = price_opt.unwrap_or_else(|| unsafe {
             std::mem::transmute(&self.price.as_ref().unwrap().dateindex)
         });
 
-        self.ratio.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |v, _, _, starting_indexes, exit| {
-                v.compute_transform2(
-                    starting_indexes.dateindex,
-                    closes,
-                    price,
-                    |(i, close, price, ..)| {
-                        if price == Dollars::ZERO {
-                            (i, StoredF32::from(1.0))
-                        } else {
-                            (i, StoredF32::from(*close / price))
-                        }
-                    },
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
+        self.ratio.compute_all(starting_indexes, exit, |v| {
+            v.compute_transform2(
+                starting_indexes.dateindex,
+                closes,
+                price,
+                |(i, close, price, ..)| {
+                    if price == Dollars::ZERO {
+                        (i, StoredF32::from(1.0))
+                    } else {
+                        (i, StoredF32::from(*close / price))
+                    }
+                },
+                exit,
+            )?;
+            Ok(())
+        })?;
 
         if self.ratio_1w_sma.is_none() {
             return Ok(());
@@ -357,12 +336,10 @@ impl ComputedRatioVecsFromDateIndex {
 
         let min_ratio_date = DateIndex::try_from(Date::MIN_RATIO).unwrap();
 
-        self.ratio_1w_sma.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |v, _, _, starting_indexes, exit| {
+        self.ratio_1w_sma
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, |v| {
                 v.compute_sma_(
                     starting_indexes.dateindex,
                     self.ratio.dateindex.as_ref().unwrap(),
@@ -371,15 +348,12 @@ impl ComputedRatioVecsFromDateIndex {
                     Some(min_ratio_date),
                 )?;
                 Ok(())
-            },
-        )?;
+            })?;
 
-        self.ratio_1m_sma.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |v, _, _, starting_indexes, exit| {
+        self.ratio_1m_sma
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, |v| {
                 v.compute_sma_(
                     starting_indexes.dateindex,
                     self.ratio.dateindex.as_ref().unwrap(),
@@ -388,8 +362,7 @@ impl ComputedRatioVecsFromDateIndex {
                     Some(min_ratio_date),
                 )?;
                 Ok(())
-            },
-        )?;
+            })?;
 
         let ratio_version = self.ratio.dateindex.as_ref().unwrap().version();
         self.mut_ratio_vecs()
@@ -423,42 +396,42 @@ impl ComputedRatioVecsFromDateIndex {
             .iter_at(starting_dateindex)
             .try_for_each(|(index, ratio)| -> Result<()> {
                 if index < min_ratio_date {
-                    self.ratio_p5
+                    self.ratio_pct5
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, StoredF32::NAN, exit)?;
-                    self.ratio_p2
+                    self.ratio_pct2
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, StoredF32::NAN, exit)?;
-                    self.ratio_p1
+                    self.ratio_pct1
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, StoredF32::NAN, exit)?;
-                    self.ratio_p95
+                    self.ratio_pct95
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, StoredF32::NAN, exit)?;
-                    self.ratio_p98
+                    self.ratio_pct98
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, StoredF32::NAN, exit)?;
-                    self.ratio_p99
+                    self.ratio_pct99
                         .as_mut()
                         .unwrap()
                         .dateindex
@@ -470,42 +443,42 @@ impl ComputedRatioVecsFromDateIndex {
                     let pos = sorted.binary_search(&ratio).unwrap_or_else(|pos| pos);
                     sorted.insert(pos, ratio);
 
-                    self.ratio_p1
+                    self.ratio_pct1
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, get_percentile(&sorted, 0.01), exit)?;
-                    self.ratio_p2
+                    self.ratio_pct2
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, get_percentile(&sorted, 0.02), exit)?;
-                    self.ratio_p5
+                    self.ratio_pct5
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, get_percentile(&sorted, 0.05), exit)?;
-                    self.ratio_p95
+                    self.ratio_pct95
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, get_percentile(&sorted, 0.95), exit)?;
-                    self.ratio_p98
+                    self.ratio_pct98
                         .as_mut()
                         .unwrap()
                         .dateindex
                         .as_mut()
                         .unwrap()
                         .forced_push_at(index, get_percentile(&sorted, 0.98), exit)?;
-                    self.ratio_p99
+                    self.ratio_pct99
                         .as_mut()
                         .unwrap()
                         .dateindex
@@ -521,32 +494,32 @@ impl ComputedRatioVecsFromDateIndex {
             .into_iter()
             .try_for_each(|v| v.safe_flush(exit))?;
 
-        self.ratio_p1.as_mut().unwrap().compute_rest(
+        self.ratio_pct1.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
         )?;
-        self.ratio_p2.as_mut().unwrap().compute_rest(
+        self.ratio_pct2.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
         )?;
-        self.ratio_p5.as_mut().unwrap().compute_rest(
+        self.ratio_pct5.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
         )?;
-        self.ratio_p95.as_mut().unwrap().compute_rest(
+        self.ratio_pct95.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
         )?;
-        self.ratio_p98.as_mut().unwrap().compute_rest(
+        self.ratio_pct98.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
         )?;
-        self.ratio_p99.as_mut().unwrap().compute_rest(
+        self.ratio_pct99.as_mut().unwrap().compute_rest(
             starting_indexes,
             exit,
             None as Option<&EagerVec<_, _>>,
@@ -556,14 +529,12 @@ impl ComputedRatioVecsFromDateIndex {
             std::mem::transmute(&self.price.as_ref().unwrap().dateindex)
         });
 
-        self.ratio_p99_as_price.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, _, starting_indexes, exit| {
+        self.ratio_pct99_usd
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, |vec| {
                 let mut iter = self
-                    .ratio_p99
+                    .ratio_pct99
                     .as_ref()
                     .unwrap()
                     .dateindex
@@ -580,67 +551,52 @@ impl ComputedRatioVecsFromDateIndex {
                     exit,
                 )?;
                 Ok(())
-            },
-        )?;
+            })?;
 
-        let compute_as_price =
-            |as_price: Option<&mut ComputedVecsFromDateIndex<Dollars>>,
+        let compute_usd =
+            |usd: Option<&mut ComputedVecsFromDateIndex<Dollars>>,
              source: Option<&ComputedVecsFromDateIndex<StoredF32>>| {
-                as_price.unwrap().compute_all(
-                    indexer,
-                    indexes,
-                    starting_indexes,
-                    exit,
-                    |vec, _, _, starting_indexes, exit| {
-                        let mut iter = source.unwrap().dateindex.as_ref().unwrap().into_iter();
-                        vec.compute_transform(
-                            starting_indexes.dateindex,
-                            date_to_price,
-                            |(i, price, ..)| {
-                                let multiplier = iter.unwrap_get_inner(i);
-                                (i, price * multiplier)
-                            },
-                            exit,
-                        )?;
-                        Ok(())
-                    },
-                )
+                usd.unwrap().compute_all(starting_indexes, exit, |vec| {
+                    let mut iter = source.unwrap().dateindex.as_ref().unwrap().into_iter();
+                    vec.compute_transform(
+                        starting_indexes.dateindex,
+                        date_to_price,
+                        |(i, price, ..)| {
+                            let multiplier = iter.unwrap_get_inner(i);
+                            (i, price * multiplier)
+                        },
+                        exit,
+                    )?;
+                    Ok(())
+                })
             };
 
-        compute_as_price(self.ratio_p1_as_price.as_mut(), self.ratio_p1.as_ref())?;
-        compute_as_price(self.ratio_p2_as_price.as_mut(), self.ratio_p2.as_ref())?;
-        compute_as_price(self.ratio_p5_as_price.as_mut(), self.ratio_p5.as_ref())?;
-        compute_as_price(self.ratio_p95_as_price.as_mut(), self.ratio_p95.as_ref())?;
-        compute_as_price(self.ratio_p98_as_price.as_mut(), self.ratio_p98.as_ref())?;
-        compute_as_price(self.ratio_p99_as_price.as_mut(), self.ratio_p99.as_ref())?;
+        compute_usd(self.ratio_pct1_usd.as_mut(), self.ratio_pct1.as_ref())?;
+        compute_usd(self.ratio_pct2_usd.as_mut(), self.ratio_pct2.as_ref())?;
+        compute_usd(self.ratio_pct5_usd.as_mut(), self.ratio_pct5.as_ref())?;
+        compute_usd(self.ratio_pct95_usd.as_mut(), self.ratio_pct95.as_ref())?;
+        compute_usd(self.ratio_pct98_usd.as_mut(), self.ratio_pct98.as_ref())?;
+        compute_usd(self.ratio_pct99_usd.as_mut(), self.ratio_pct99.as_ref())?;
 
         self.ratio_sd.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
             starting_indexes,
             exit,
             self.ratio.dateindex.as_ref().unwrap(),
             Some(date_to_price),
         )?;
         self.ratio_4y_sd.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
             starting_indexes,
             exit,
             self.ratio.dateindex.as_ref().unwrap(),
             Some(date_to_price),
         )?;
         self.ratio_2y_sd.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
             starting_indexes,
             exit,
             self.ratio.dateindex.as_ref().unwrap(),
             Some(date_to_price),
         )?;
         self.ratio_1y_sd.as_mut().unwrap().compute_all(
-            indexer,
-            indexes,
             starting_indexes,
             exit,
             self.ratio.dateindex.as_ref().unwrap(),
@@ -652,22 +608,22 @@ impl ComputedRatioVecsFromDateIndex {
 
     fn mut_ratio_vecs(&mut self) -> Vec<&mut EagerVec<DateIndex, StoredF32>> {
         [
-            self.ratio_p1
+            self.ratio_pct1
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
-            self.ratio_p2
+            self.ratio_pct2
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
-            self.ratio_p5
+            self.ratio_pct5
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
-            self.ratio_p95
+            self.ratio_pct95
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
-            self.ratio_p98
+            self.ratio_pct98
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
-            self.ratio_p99
+            self.ratio_pct99
                 .as_mut()
                 .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
         ]
@@ -676,37 +632,132 @@ impl ComputedRatioVecsFromDateIndex {
         .collect::<Vec<_>>()
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.price.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio.vecs(),
-            self.ratio_1w_sma.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_1m_sma.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_sd.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_1y_sd.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_2y_sd.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_4y_sd.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_p1.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_p2.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_p5.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_p95.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_p98.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_p99.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_p1_as_price.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_p2_as_price.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_p5_as_price.as_ref().map_or(vec![], |v| v.vecs()),
-            self.ratio_p95_as_price
-                .as_ref()
-                .map_or(vec![], |v| v.vecs()),
-            self.ratio_p98_as_price
-                .as_ref()
-                .map_or(vec![], |v| v.vecs()),
-            self.ratio_p99_as_price
-                .as_ref()
-                .map_or(vec![], |v| v.vecs()),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(self.price.iter().flat_map(|v| v.iter_any_collectable()));
+
+        iter = Box::new(iter.chain(self.ratio.iter_any_collectable()));
+        iter = Box::new(
+            iter.chain(
+                self.ratio_1w_sma
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_1m_sma
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.ratio_sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.ratio_1y_sd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_2y_sd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_4y_sd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct1
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct2
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct5
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct95
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct98
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct99
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct1_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct2_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct5_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct95_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct98_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct99_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+
+        iter
     }
 }

crates/brk_computer/src/grouped/value_from_dateindex.rs

@@ -1,5 +1,4 @@
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, DateIndex, Dollars, Sats, Version};
 use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, StoredVec};
 
@@ -43,7 +42,7 @@ impl ComputedValueVecsFromDateIndex {
             )?,
             bitcoin: ComputedVecsFromDateIndex::forced_import(
                 db,
-                &format!("{name}_in_btc"),
+                &format!("{name}_btc"),
                 Source::Compute,
                 version + VERSION,
                 indexes,
@@ -52,7 +51,7 @@ impl ComputedValueVecsFromDateIndex {
             dollars: compute_dollars.then(|| {
                 ComputedVecsFromDateIndex::forced_import(
                     db,
-                    &format!("{name}_in_usd"),
+                    &format!("{name}_usd"),
                     Source::Compute,
                     version + VERSION,
                     indexes,
@@ -65,40 +64,24 @@ impl ComputedValueVecsFromDateIndex {
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(
-            &mut EagerVec<DateIndex, Sats>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
+        F: FnMut(&mut EagerVec<DateIndex, Sats>) -> Result<()>,
     {
-        compute(
-            self.sats.dateindex.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
+        compute(self.sats.dateindex.as_mut().unwrap())?;
 
         let dateindex: Option<&StoredVec<DateIndex, Sats>> = None;
-        self.compute_rest(indexer, indexes, price, starting_indexes, exit, dateindex)?;
+        self.compute_rest(price, starting_indexes, exit, dateindex)?;
 
         Ok(())
     }
 
     pub fn compute_rest(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
@@ -108,72 +91,50 @@ impl ComputedValueVecsFromDateIndex {
             self.sats
                 .compute_rest(starting_indexes, exit, Some(dateindex))?;
 
-            self.bitcoin.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |v, _, _, starting_indexes, exit| {
-                    v.compute_from_sats(starting_indexes.dateindex, dateindex, exit)
-                },
-            )?;
+            self.bitcoin.compute_all(starting_indexes, exit, |v| {
+                v.compute_from_sats(starting_indexes.dateindex, dateindex, exit)
+            })?;
         } else {
             let dateindex: Option<&StoredVec<DateIndex, Sats>> = None;
 
             self.sats.compute_rest(starting_indexes, exit, dateindex)?;
 
-            self.bitcoin.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |v, _, _, starting_indexes, exit| {
-                    v.compute_from_sats(
-                        starting_indexes.dateindex,
-                        self.sats.dateindex.as_ref().unwrap(),
-                        exit,
-                    )
-                },
-            )?;
+            self.bitcoin.compute_all(starting_indexes, exit, |v| {
+                v.compute_from_sats(
+                    starting_indexes.dateindex,
+                    self.sats.dateindex.as_ref().unwrap(),
+                    exit,
+                )
+            })?;
         }
 
         let dateindex_to_bitcoin = self.bitcoin.dateindex.as_ref().unwrap();
-        let dateindex_to_close = price
+        let dateindex_to_price_close = price
             .as_ref()
             .unwrap()
-            .timeindexes_to_close
+            .timeindexes_to_price_close
             .dateindex
             .as_ref()
             .unwrap();
 
         if let Some(dollars) = self.dollars.as_mut() {
-            dollars.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |v, _, _, starting_indexes, exit| {
-                    v.compute_from_bitcoin(
-                        starting_indexes.dateindex,
-                        dateindex_to_bitcoin,
-                        dateindex_to_close,
-                        exit,
-                    )
-                },
-            )?;
+            dollars.compute_all(starting_indexes, exit, |v| {
+                v.compute_from_bitcoin(
+                    starting_indexes.dateindex,
+                    dateindex_to_bitcoin,
+                    dateindex_to_price_close,
+                    exit,
+                )
+            })?;
         }
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.sats.vecs(),
-            self.bitcoin.vecs(),
-            self.dollars.as_ref().map_or(vec![], |v| v.vecs()),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        std::iter::empty()
+            .chain(self.sats.iter_any_collectable())
+            .chain(self.bitcoin.iter_any_collectable())
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
     }
 }

crates/brk_computer/src/grouped/value_from_height.rs

@@ -1,5 +1,5 @@
+use allocative::Allocative;
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, Dollars, Height, Sats, Version};
 use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, StoredVec};
 
@@ -12,7 +12,7 @@ use crate::{
 
 use super::{ComputedVecsFromHeight, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedValueVecsFromHeight {
     pub sats: ComputedVecsFromHeight<Sats>,
     pub bitcoin: ComputedVecsFromHeight<Bitcoin>,
@@ -43,7 +43,7 @@ impl ComputedValueVecsFromHeight {
             )?,
             bitcoin: ComputedVecsFromHeight::forced_import(
                 db,
-                &format!("{name}_in_btc"),
+                &format!("{name}_btc"),
                 Source::Compute,
                 version + VERSION,
                 indexes,
@@ -52,7 +52,7 @@ impl ComputedValueVecsFromHeight {
             dollars: compute_dollars.then(|| {
                 ComputedVecsFromHeight::forced_import(
                     db,
-                    &format!("{name}_in_usd"),
+                    &format!("{name}_usd"),
                     Source::Compute,
                     version + VERSION,
                     indexes,
@@ -65,7 +65,6 @@ impl ComputedValueVecsFromHeight {
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
@@ -73,31 +72,18 @@ impl ComputedValueVecsFromHeight {
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(
-            &mut EagerVec<Height, Sats>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
+        F: FnMut(&mut EagerVec<Height, Sats>) -> Result<()>,
     {
-        compute(
-            self.sats.height.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
+        compute(self.sats.height.as_mut().unwrap())?;
 
         let height: Option<&StoredVec<Height, Sats>> = None;
-        self.compute_rest(indexer, indexes, price, starting_indexes, exit, height)?;
+        self.compute_rest(indexes, price, starting_indexes, exit, height)?;
 
         Ok(())
     }
 
     pub fn compute_rest(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
@@ -108,67 +94,47 @@ impl ComputedValueVecsFromHeight {
             self.sats
                 .compute_rest(indexes, starting_indexes, exit, Some(height))?;
 
-            self.bitcoin.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |v, _, _, starting_indexes, exit| {
+            self.bitcoin
+                .compute_all(indexes, starting_indexes, exit, |v| {
                     v.compute_from_sats(starting_indexes.height, height, exit)
-                },
-            )?;
+                })?;
         } else {
             let height: Option<&StoredVec<Height, Sats>> = None;
 
             self.sats
                 .compute_rest(indexes, starting_indexes, exit, height)?;
 
-            self.bitcoin.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |v, _, _, starting_indexes, exit| {
+            self.bitcoin
+                .compute_all(indexes, starting_indexes, exit, |v| {
                     v.compute_from_sats(
                         starting_indexes.height,
                         self.sats.height.as_ref().unwrap(),
                         exit,
                     )
-                },
-            )?;
+                })?;
         }
 
         let height_to_bitcoin = self.bitcoin.height.as_ref().unwrap();
-        let height_to_close = &price.as_ref().unwrap().chainindexes_to_close.height;
+        let height_to_price_close = &price.as_ref().unwrap().chainindexes_to_price_close.height;
 
         if let Some(dollars) = self.dollars.as_mut() {
-            dollars.compute_all(
-                indexer,
-                indexes,
-                starting_indexes,
-                exit,
-                |v, _, _, starting_indexes, exit| {
-                    v.compute_from_bitcoin(
-                        starting_indexes.height,
-                        height_to_bitcoin,
-                        height_to_close,
-                        exit,
-                    )
-                },
-            )?;
+            dollars.compute_all(indexes, starting_indexes, exit, |v| {
+                v.compute_from_bitcoin(
+                    starting_indexes.height,
+                    height_to_bitcoin,
+                    height_to_price_close,
+                    exit,
+                )
+            })?;
         }
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.sats.vecs(),
-            self.bitcoin.vecs(),
-            self.dollars.as_ref().map_or(vec![], |v| v.vecs()),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        std::iter::empty()
+            .chain(self.sats.iter_any_collectable())
+            .chain(self.bitcoin.iter_any_collectable())
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
     }
 }

crates/brk_computer/src/grouped/value_from_txindex.rs

@@ -1,3 +1,4 @@
+use allocative::Allocative;
 use brk_error::Result;
 use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, Close, Dollars, Height, Sats, TxIndex, Version};
@@ -10,7 +11,7 @@ use crate::{Indexes, grouped::Source, indexes, price};
 
 use super::{ComputedVecsFromTxindex, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedValueVecsFromTxindex {
     pub sats: ComputedVecsFromTxindex<Sats>,
     pub bitcoin_txindex: LazyVecFrom1<TxIndex, Bitcoin, TxIndex, Sats>,
@@ -37,8 +38,8 @@ impl ComputedValueVecsFromTxindex {
     ) -> Result<Self> {
         let compute_dollars = price.is_some();
 
-        let name_in_btc = format!("{name}_in_btc");
-        let name_in_usd = format!("{name}_in_usd");
+        let name_btc = format!("{name}_btc");
+        let name_usd = format!("{name}_usd");
 
         let sats = ComputedVecsFromTxindex::forced_import(
             db,
@@ -52,7 +53,7 @@ impl ComputedValueVecsFromTxindex {
         let source_vec = source.vec();
 
         let bitcoin_txindex = LazyVecFrom1::init(
-            &name_in_btc,
+            &name_btc,
             version + VERSION,
             source_vec.map_or_else(|| sats.txindex.as_ref().unwrap().boxed_clone(), |s| s),
             |txindex: TxIndex, iter| {
@@ -65,7 +66,7 @@ impl ComputedValueVecsFromTxindex {
 
         let bitcoin = ComputedVecsFromTxindex::forced_import(
             db,
-            &name_in_btc,
+            &name_btc,
             Source::None,
             version + VERSION,
             indexes,
@@ -74,15 +75,15 @@ impl ComputedValueVecsFromTxindex {
 
         let dollars_txindex = price.map(|price| {
             LazyVecFrom3::init(
-                &name_in_usd,
+                &name_usd,
                 version + VERSION,
                 bitcoin_txindex.boxed_clone(),
                 indexes.txindex_to_height.boxed_clone(),
-                price.chainindexes_to_close.height.boxed_clone(),
+                price.chainindexes_to_price_close.height.boxed_clone(),
                 |txindex: TxIndex,
                  txindex_to_btc_iter,
                  txindex_to_height_iter,
-                 height_to_close_iter| {
+                 height_to_price_close_iter| {
                     let txindex = txindex.unwrap_to_usize();
                     txindex_to_btc_iter.next_at(txindex).and_then(|(_, value)| {
                         let btc = value.into_owned();
@@ -90,7 +91,7 @@ impl ComputedValueVecsFromTxindex {
                             .next_at(txindex)
                             .and_then(|(_, value)| {
                                 let height = value.into_owned();
-                                height_to_close_iter
+                                height_to_price_close_iter
                                     .next_at(height.unwrap_to_usize())
                                     .map(|(_, close)| *close.into_owned() * btc)
                             })
@@ -107,7 +108,7 @@ impl ComputedValueVecsFromTxindex {
             dollars: compute_dollars.then(|| {
                 ComputedVecsFromTxindex::forced_import(
                     db,
-                    &name_in_usd,
+                    &name_usd,
                     Source::None,
                     version + VERSION,
                     indexes,
@@ -201,18 +202,16 @@ impl ComputedValueVecsFromTxindex {
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.sats.vecs(),
-            vec![&self.bitcoin_txindex as &dyn AnyCollectableVec],
-            self.bitcoin.vecs(),
-            self.dollars_txindex
-                .as_ref()
-                .map_or(vec![], |v| vec![v as &dyn AnyCollectableVec]),
-            self.dollars.as_ref().map_or(vec![], |v| v.vecs()),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.bitcoin_txindex as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(self.sats.iter_any_collectable())
+            .chain(self.bitcoin.iter_any_collectable())
+            .chain(
+                self.dollars_txindex
+                    .iter()
+                    .map(|v| v as &dyn AnyCollectableVec),
+            )
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
     }
 }

crates/brk_computer/src/grouped/value_height.rs

@@ -1,12 +1,11 @@
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, Dollars, Height, Sats, Version};
 use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, Format, StoredVec};
 
 use crate::{
     Indexes,
     grouped::Source,
-    indexes, price,
+    price,
     traits::{ComputeFromBitcoin, ComputeFromSats},
 };
 
@@ -35,14 +34,14 @@ impl ComputedHeightValueVecs {
             }),
             bitcoin: EagerVec::forced_import(
                 db,
-                &format!("{name}_in_btc"),
+                &format!("{name}_btc"),
                 version + VERSION + Version::ZERO,
                 format,
             )?,
             dollars: compute_dollars.then(|| {
                 EagerVec::forced_import(
                     db,
-                    &format!("{name}_in_usd"),
+                    &format!("{name}_usd"),
                     version + VERSION + Version::ZERO,
                     format,
                 )
@@ -53,29 +52,15 @@ impl ComputedHeightValueVecs {
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
     ) -> Result<()>
     where
-        F: FnMut(
-            &mut EagerVec<Height, Sats>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
+        F: FnMut(&mut EagerVec<Height, Sats>) -> Result<()>,
     {
-        compute(
-            self.sats.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
+        compute(self.sats.as_mut().unwrap())?;
 
         let height: Option<&StoredVec<Height, Sats>> = None;
         self.compute_rest(price, starting_indexes, exit, height)?;
@@ -102,13 +87,13 @@ impl ComputedHeightValueVecs {
         }
 
         let height_to_bitcoin = &self.bitcoin;
-        let height_to_close = &price.as_ref().unwrap().chainindexes_to_close.height;
+        let height_to_price_close = &price.as_ref().unwrap().chainindexes_to_price_close.height;
 
         if let Some(dollars) = self.dollars.as_mut() {
             dollars.compute_from_bitcoin(
                 starting_indexes.height,
                 height_to_bitcoin,
-                height_to_close,
+                height_to_price_close,
                 exit,
             )?;
         }
@@ -116,14 +101,10 @@ impl ComputedHeightValueVecs {
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            vec![&self.bitcoin as &dyn AnyCollectableVec],
-            self.sats.as_ref().map_or(vec![], |v| vec![v]),
-            self.dollars.as_ref().map_or(vec![], |v| vec![v]),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.bitcoin as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(self.sats.iter().map(|v| v as &dyn AnyCollectableVec))
+            .chain(self.dollars.iter().map(|v| v as &dyn AnyCollectableVec))
     }
 }

crates/brk_computer/src/indexes.rs

@@ -236,7 +236,7 @@ impl Vecs {
             |index, _| Some(index),
         );
 
-        Ok(Self {
+        let this = Self {
             emptyoutputindex_to_emptyoutputindex,
             inputindex_to_inputindex,
             opreturnindex_to_opreturnindex,
@@ -472,7 +472,15 @@ impl Vecs {
             )?,
 
             db,
-        })
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
     }
 
     pub fn compute(
@@ -925,9 +933,9 @@ impl Vecs {
         })
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        vec![
-            &self.dateindex_to_date,
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [
+            &self.dateindex_to_date as &dyn AnyCollectableVec,
             &self.dateindex_to_dateindex,
             &self.dateindex_to_first_height,
             &self.dateindex_to_height_count,
@@ -988,6 +996,7 @@ impl Vecs {
             &self.yearindex_to_yearindex,
             &self.outputindex_to_txindex,
         ]
+        .into_iter()
     }
 }
 

crates/brk_computer/src/lib.rs

@@ -5,45 +5,47 @@ use std::path::Path;
 use brk_error::Result;
 use brk_fetcher::Fetcher;
 use brk_indexer::Indexer;
+use brk_parser::Parser;
 use brk_structs::Version;
 use log::info;
 use vecdb::{AnyCollectableVec, Exit, Format};
 
-mod blocks;
+mod blks;
+mod chain;
 mod cointime;
 mod constants;
 mod fetched;
 mod grouped;
 mod indexes;
 mod market;
-mod mining;
+mod pools;
 mod price;
 mod stateful;
 mod states;
 mod traits;
-mod transactions;
 mod utils;
 
 use indexes::Indexes;
 
+pub use pools::*;
 pub use states::PriceToAmount;
 use states::*;
 
 #[derive(Clone)]
 pub struct Computer {
-    pub indexes: indexes::Vecs,
-    pub constants: constants::Vecs,
-    pub blocks: blocks::Vecs,
-    pub mining: mining::Vecs,
-    pub market: market::Vecs,
-    pub price: Option<price::Vecs>,
-    pub transactions: transactions::Vecs,
-    pub stateful: stateful::Vecs,
-    pub fetched: Option<fetched::Vecs>,
+    pub chain: chain::Vecs,
     pub cointime: cointime::Vecs,
+    pub constants: constants::Vecs,
+    pub fetched: Option<fetched::Vecs>,
+    pub indexes: indexes::Vecs,
+    pub market: market::Vecs,
+    pub pools: pools::Vecs,
+    pub blks: blks::Vecs,
+    pub price: Option<price::Vecs>,
+    pub stateful: stateful::Vecs,
 }
 
-const VERSION: Version = Version::TWO;
+const VERSION: Version = Version::new(4);
 
 impl Computer {
     /// Do NOT import multiple times or things will break !!!
@@ -68,8 +70,6 @@ impl Computer {
         });
 
         Ok(Self {
-            blocks: blocks::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes)?,
-            mining: mining::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes)?,
             constants: constants::Vecs::forced_import(
                 &computed_path,
                 VERSION + Version::ZERO,
@@ -83,13 +83,20 @@ impl Computer {
                 &indexes,
                 price.as_ref(),
             )?,
-            transactions: transactions::Vecs::forced_import(
+            chain: chain::Vecs::forced_import(
                 &computed_path,
                 VERSION + Version::ZERO,
                 indexer,
                 &indexes,
                 price.as_ref(),
             )?,
+            blks: blks::Vecs::forced_import(&computed_path, VERSION + Version::ZERO)?,
+            pools: pools::Vecs::forced_import(
+                &computed_path,
+                VERSION + Version::ZERO,
+                &indexes,
+                price.as_ref(),
+            )?,
             cointime: cointime::Vecs::forced_import(
                 &computed_path,
                 VERSION + Version::ZERO,
@@ -106,30 +113,18 @@ impl Computer {
         &mut self,
         indexer: &Indexer,
         starting_indexes: brk_indexer::Indexes,
+        parser: &Parser,
         exit: &Exit,
     ) -> Result<()> {
         info!("Computing indexes...");
         let mut starting_indexes = self.indexes.compute(indexer, starting_indexes, exit)?;
 
-        info!("Computing constants...");
-        self.constants
-            .compute(indexer, &self.indexes, &starting_indexes, exit)?;
-
-        info!("Computing blocks...");
-        self.blocks
-            .compute(indexer, &self.indexes, &starting_indexes, exit)?;
-
-        info!("Computing mining...");
-        self.mining
-            .compute(indexer, &self.indexes, &starting_indexes, exit)?;
-
         if let Some(fetched) = self.fetched.as_mut() {
             info!("Computing fetched...");
             fetched.compute(indexer, &self.indexes, &starting_indexes, exit)?;
 
             info!("Computing prices...");
             self.price.as_mut().unwrap().compute(
-                indexer,
                 &self.indexes,
                 &starting_indexes,
                 fetched,
@@ -137,44 +132,73 @@ impl Computer {
             )?;
         }
 
-        info!("Computing transactions...");
-        self.transactions.compute(
+        info!("Computing BLKs metadata...");
+        self.blks
+            .compute(indexer, &self.indexes, &starting_indexes, parser, exit)?;
+
+        std::thread::scope(|scope| -> Result<()> {
+            let constants = scope.spawn(|| -> Result<()> {
+                info!("Computing constants...");
+                self.constants
+                    .compute(&self.indexes, &starting_indexes, exit)?;
+                Ok(())
+            });
+
+            // let blks = scope.spawn(|| -> Result<()> {
+            //     info!("Computing blks...");
+            //     self.blks
+            //         .compute(indexer, &self.indexes, &starting_indexes, parser, exit)?;
+            //     Ok(())
+            // });
+
+            let chain = scope.spawn(|| -> Result<()> {
+                info!("Computing chain...");
+                self.chain.compute(
+                    indexer,
+                    &self.indexes,
+                    &starting_indexes,
+                    self.price.as_ref(),
+                    exit,
+                )?;
+                Ok(())
+            });
+
+            if let Some(price) = self.price.as_ref() {
+                info!("Computing market...");
+                self.market.compute(price, &starting_indexes, exit)?;
+            }
+
+            constants.join().unwrap()?;
+            // blks.join().unwrap()?;
+            chain.join().unwrap()?;
+            Ok(())
+        })?;
+
+        self.pools.compute(
             indexer,
             &self.indexes,
             &starting_indexes,
+            &self.chain,
             self.price.as_ref(),
             exit,
         )?;
 
-        if let Some(price) = self.price.as_ref() {
-            info!("Computing market...");
-            self.market.compute(
-                indexer,
-                &self.indexes,
-                price,
-                &mut self.transactions,
-                &starting_indexes,
-                exit,
-            )?;
-        }
-
         info!("Computing stateful...");
         self.stateful.compute(
             indexer,
             &self.indexes,
-            &self.transactions,
+            &self.chain,
             self.price.as_ref(),
-            &self.market,
             &mut starting_indexes,
             exit,
         )?;
 
+        info!("Computing cointime...");
         self.cointime.compute(
-            indexer,
             &self.indexes,
             &starting_indexes,
             self.price.as_ref(),
-            &self.transactions,
+            &self.chain,
             &self.stateful,
             exit,
         )?;
@@ -182,25 +206,57 @@ impl Computer {
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.constants.vecs(),
-            self.indexes.vecs(),
-            self.blocks.vecs(),
-            self.mining.vecs(),
-            self.market.vecs(),
-            self.transactions.vecs(),
-            self.stateful.vecs(),
-            self.cointime.vecs(),
-            self.fetched.as_ref().map_or(vec![], |v| v.vecs()),
-            self.price.as_ref().map_or(vec![], |v| v.vecs()),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(self.fetched.iter().flat_map(|v| v.iter_any_collectable()));
+
+        iter = Box::new(iter.chain(self.chain.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.cointime.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constants.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.market.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.pools.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.blks.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.price.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.stateful.iter_any_collectable()));
+
+        iter
     }
 
     pub fn static_clone(&self) -> &'static Self {
         Box::leak(Box::new(self.clone()))
     }
 }
+
+// pub fn generate_allocation_files(monitored: &pools::Vecs) -> Result<()> {
+//     info!("Generating Allocative files...");
+
+//     let mut flamegraph = allocative::FlameGraphBuilder::default();
+//     flamegraph.visit_root(monitored);
+//     let output = flamegraph.finish();
+
+//     let folder = format!(
+//         "at-{}",
+//         jiff::Timestamp::now().strftime("%Y-%m-%d_%Hh%Mm%Ss"),
+//     );
+
+//     let path = std::path::PathBuf::from(&format!("./target/flamegraph/{folder}"));
+//     std::fs::create_dir_all(&path)?;
+
+//     // fs::write(path.join("flamegraph.src"), &output.flamegraph())?;
+
+//     let mut fg_svg = Vec::new();
+//     inferno::flamegraph::from_reader(
+//         &mut inferno::flamegraph::Options::default(),
+//         output.flamegraph().write().as_bytes(),
+//         &mut fg_svg,
+//     )?;
+
+//     std::fs::write(path.join("flamegraph.svg"), &fg_svg)?;
+
+//     std::fs::write(path.join("warnings.txt"), output.warnings())?;
+
+//     info!("Successfully generated Allocative files");
+
+//     Ok(())
+// }

crates/brk_computer/src/mining.rs

@@ -1,151 +0,0 @@
-use std::path::Path;
-
-use brk_error::Result;
-use brk_indexer::Indexer;
-use brk_structs::{DifficultyEpoch, HalvingEpoch, StoredF64, Version};
-use vecdb::{AnyCollectableVec, Database, Exit, PAGE_SIZE, VecIterator};
-
-use crate::grouped::Source;
-
-use super::{
-    Indexes,
-    grouped::{ComputedVecsFromDateIndex, ComputedVecsFromHeight, VecBuilderOptions},
-    indexes,
-};
-
-const VERSION: Version = Version::ZERO;
-
-#[derive(Clone)]
-pub struct Vecs {
-    db: Database,
-
-    pub indexes_to_difficulty: ComputedVecsFromHeight<StoredF64>,
-    pub indexes_to_difficultyepoch: ComputedVecsFromDateIndex<DifficultyEpoch>,
-    pub indexes_to_halvingepoch: ComputedVecsFromDateIndex<HalvingEpoch>,
-}
-
-impl Vecs {
-    pub fn forced_import(parent: &Path, version: Version, indexes: &indexes::Vecs) -> Result<Self> {
-        let db = Database::open(&parent.join("mining"))?;
-        db.set_min_len(PAGE_SIZE * 1_000_000)?;
-
-        Ok(Self {
-            indexes_to_difficulty: ComputedVecsFromHeight::forced_import(
-                &db,
-                "difficulty",
-                Source::None,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_last(),
-            )?,
-            indexes_to_difficultyepoch: ComputedVecsFromDateIndex::forced_import(
-                &db,
-                "difficultyepoch",
-                Source::Compute,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_last(),
-            )?,
-            indexes_to_halvingepoch: ComputedVecsFromDateIndex::forced_import(
-                &db,
-                "halvingepoch",
-                Source::Compute,
-                version + VERSION + Version::ZERO,
-                indexes,
-                VecBuilderOptions::default().add_last(),
-            )?,
-
-            db,
-        })
-    }
-
-    pub fn compute(
-        &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        starting_indexes: &Indexes,
-        exit: &Exit,
-    ) -> Result<()> {
-        self.compute_(indexer, indexes, starting_indexes, exit)?;
-        self.db.flush_then_punch()?;
-        Ok(())
-    }
-
-    fn compute_(
-        &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        starting_indexes: &Indexes,
-        exit: &Exit,
-    ) -> Result<()> {
-        let mut height_to_difficultyepoch_iter = indexes.height_to_difficultyepoch.into_iter();
-        self.indexes_to_difficultyepoch.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                let mut height_count_iter = indexes.dateindex_to_height_count.into_iter();
-                vec.compute_transform(
-                    starting_indexes.dateindex,
-                    &indexes.dateindex_to_first_height,
-                    |(di, height, ..)| {
-                        (
-                            di,
-                            height_to_difficultyepoch_iter.unwrap_get_inner(
-                                height + (*height_count_iter.unwrap_get_inner(di) - 1),
-                            ),
-                        )
-                    },
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        let mut height_to_halvingepoch_iter = indexes.height_to_halvingepoch.into_iter();
-        self.indexes_to_halvingepoch.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                let mut height_count_iter = indexes.dateindex_to_height_count.into_iter();
-                vec.compute_transform(
-                    starting_indexes.dateindex,
-                    &indexes.dateindex_to_first_height,
-                    |(di, height, ..)| {
-                        (
-                            di,
-                            height_to_halvingepoch_iter.unwrap_get_inner(
-                                height + (*height_count_iter.unwrap_get_inner(di) - 1),
-                            ),
-                        )
-                    },
-                    exit,
-                )?;
-                Ok(())
-            },
-        )?;
-
-        self.indexes_to_difficulty.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&indexer.vecs.height_to_difficulty),
-        )?;
-
-        Ok(())
-    }
-
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.indexes_to_difficulty.vecs(),
-            self.indexes_to_difficultyepoch.vecs(),
-            self.indexes_to_halvingepoch.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
-    }
-}

crates/brk_computer/src/pools/mod.rs

@@ -0,0 +1,227 @@
+use std::{collections::BTreeMap, path::Path};
+
+use allocative::Allocative;
+use brk_error::Result;
+use brk_indexer::Indexer;
+use brk_store::AnyStore;
+use brk_structs::{AddressBytes, Height, OutputIndex, OutputType, PoolId, Pools, pools};
+use rayon::prelude::*;
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, Exit, GenericStoredVec,
+    PAGE_SIZE, RawVec, StoredIndex, VecIterator, Version,
+};
+
+mod vecs;
+
+use crate::{
+    chain,
+    indexes::{self, Indexes},
+    price,
+};
+
+#[derive(Clone, Allocative)]
+pub struct Vecs {
+    db: Database,
+    pools: &'static Pools,
+    height_to_pool: RawVec<Height, PoolId>,
+
+    vecs: BTreeMap<PoolId, vecs::Vecs>,
+}
+
+impl Vecs {
+    pub fn forced_import(
+        parent_path: &Path,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+    ) -> Result<Self> {
+        let db = Database::open(&parent_path.join("pools"))?;
+        db.set_min_len(PAGE_SIZE * 1_000_000)?;
+        let pools = pools();
+
+        let version = parent_version + Version::new(3) + Version::new(pools.len() as u64);
+
+        let this = Self {
+            height_to_pool: RawVec::forced_import(&db, "pool", version + Version::ZERO)?,
+            vecs: pools
+                .iter()
+                .map(|pool| {
+                    vecs::Vecs::forced_import(
+                        &db,
+                        pool.id,
+                        pools,
+                        version + Version::ZERO,
+                        indexes,
+                        price,
+                    )
+                    .map(|vecs| (pool.id, vecs))
+                })
+                .collect::<Result<BTreeMap<_, _>>>()?,
+            pools,
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexer, indexes, starting_indexes, chain, price, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_height_to_pool(indexer, indexes, starting_indexes, exit)?;
+
+        self.vecs.par_iter_mut().try_for_each(|(_, vecs)| {
+            vecs.compute(
+                indexes,
+                starting_indexes,
+                &self.height_to_pool,
+                chain,
+                price,
+                exit,
+            )
+        })?;
+
+        Ok(())
+    }
+
+    fn compute_height_to_pool(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.height_to_pool.validate_computed_version_or_reset(
+            self.height_to_pool.version() + indexer.stores.height_to_coinbase_tag.version(),
+        )?;
+
+        let mut height_to_first_txindex_iter = indexer.vecs.height_to_first_txindex.iter();
+        let mut txindex_to_first_outputindex_iter =
+            indexer.vecs.txindex_to_first_outputindex.iter();
+        let mut txindex_to_output_count_iter = indexes.txindex_to_output_count.iter();
+        let mut outputindex_to_outputtype_iter = indexer.vecs.outputindex_to_outputtype.iter();
+        let mut outputindex_to_typeindex_iter = indexer.vecs.outputindex_to_typeindex.iter();
+        let mut p2pk65addressindex_to_p2pk65bytes_iter =
+            indexer.vecs.p2pk65addressindex_to_p2pk65bytes.iter();
+        let mut p2pk33addressindex_to_p2pk33bytes_iter =
+            indexer.vecs.p2pk33addressindex_to_p2pk33bytes.iter();
+        let mut p2pkhaddressindex_to_p2pkhbytes_iter =
+            indexer.vecs.p2pkhaddressindex_to_p2pkhbytes.iter();
+        let mut p2shaddressindex_to_p2shbytes_iter =
+            indexer.vecs.p2shaddressindex_to_p2shbytes.iter();
+        let mut p2wpkhaddressindex_to_p2wpkhbytes_iter =
+            indexer.vecs.p2wpkhaddressindex_to_p2wpkhbytes.iter();
+        let mut p2wshaddressindex_to_p2wshbytes_iter =
+            indexer.vecs.p2wshaddressindex_to_p2wshbytes.iter();
+        let mut p2traddressindex_to_p2trbytes_iter =
+            indexer.vecs.p2traddressindex_to_p2trbytes.iter();
+        let mut p2aaddressindex_to_p2abytes_iter = indexer.vecs.p2aaddressindex_to_p2abytes.iter();
+
+        let unknown = self.pools.get_unknown();
+
+        let min = starting_indexes
+            .height
+            .unwrap_to_usize()
+            .min(self.height_to_pool.len());
+
+        indexer
+            .stores
+            .height_to_coinbase_tag
+            .iter()
+            .skip(min)
+            .try_for_each(|(height, coinbase_tag)| -> Result<()> {
+                let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+                let outputindex = txindex_to_first_outputindex_iter.unwrap_get_inner(txindex);
+                let outputcount = txindex_to_output_count_iter.unwrap_get_inner(txindex);
+
+                let pool = (*outputindex..(*outputindex + *outputcount))
+                    .map(OutputIndex::from)
+                    .find_map(|outputindex| {
+                        let outputtype =
+                            outputindex_to_outputtype_iter.unwrap_get_inner(outputindex);
+                        let typeindex = outputindex_to_typeindex_iter.unwrap_get_inner(outputindex);
+
+                        let address = match outputtype {
+                            OutputType::P2PK65 => Some(AddressBytes::from(
+                                p2pk65addressindex_to_p2pk65bytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2PK33 => Some(AddressBytes::from(
+                                p2pk33addressindex_to_p2pk33bytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2PKH => Some(AddressBytes::from(
+                                p2pkhaddressindex_to_p2pkhbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2SH => Some(AddressBytes::from(
+                                p2shaddressindex_to_p2shbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2WPKH => Some(AddressBytes::from(
+                                p2wpkhaddressindex_to_p2wpkhbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2WSH => Some(AddressBytes::from(
+                                p2wshaddressindex_to_p2wshbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2TR => Some(AddressBytes::from(
+                                p2traddressindex_to_p2trbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2A => Some(AddressBytes::from(
+                                p2aaddressindex_to_p2abytes_iter.unwrap_get_inner(typeindex.into()),
+                            )),
+                            _ => None,
+                        };
+
+                        address
+                            .and_then(|address| self.pools.find_from_address(&address.to_string()))
+                    })
+                    .or_else(|| self.pools.find_from_coinbase_tag(&coinbase_tag))
+                    .unwrap_or(unknown);
+
+                self.height_to_pool.push_if_needed(height, pool.id)?;
+                Ok(())
+            })?;
+
+        self.height_to_pool.safe_flush(exit)?;
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.height_to_pool as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(
+                self.vecs
+                    .iter()
+                    .flat_map(|(_, vecs)| vecs.iter_any_collectable()),
+            )
+    }
+}

crates/brk_computer/src/pools/vecs.rs

@@ -0,0 +1,397 @@
+use allocative::Allocative;
+use brk_error::Result;
+use brk_structs::{Height, PoolId, Pools, Sats, StoredF32, StoredU16, StoredU32};
+use vecdb::{AnyCollectableVec, AnyIterableVec, Database, Exit, StoredIndex, VecIterator, Version};
+
+use crate::{
+    chain,
+    grouped::{
+        ComputedValueVecsFromHeight, ComputedVecsFromDateIndex, ComputedVecsFromHeight, Source,
+        VecBuilderOptions,
+    },
+    indexes::{self, Indexes},
+    price,
+};
+
+#[derive(Clone, Allocative)]
+pub struct Vecs {
+    id: PoolId,
+
+    indexes_to_blocks_mined: ComputedVecsFromHeight<StoredU32>,
+    indexes_to_1w_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_1m_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_1y_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_subsidy: ComputedValueVecsFromHeight,
+    indexes_to_fee: ComputedValueVecsFromHeight,
+    indexes_to_coinbase: ComputedValueVecsFromHeight,
+    indexes_to_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1d_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1w_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1m_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1y_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_days_since_block: ComputedVecsFromDateIndex<StoredU16>,
+}
+
+impl Vecs {
+    pub fn forced_import(
+        db: &Database,
+        id: PoolId,
+        pools: &Pools,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+    ) -> Result<Self> {
+        let pool = pools.get(id);
+        let name = pool.serialized_id();
+        let suffix = |s: &str| format!("{name}_{s}");
+        let compute_dollars = price.is_some();
+        let version = parent_version + Version::ZERO;
+
+        Ok(Self {
+            id,
+            indexes_to_blocks_mined: ComputedVecsFromHeight::forced_import(
+                db,
+                &suffix("blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+            )?,
+            indexes_to_1w_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1w_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1m_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1m_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1y_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1y_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_subsidy: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("subsidy"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_fee: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("fee"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_coinbase: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("coinbase"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1d_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1d_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1w_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1w_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1m_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1m_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1y_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1y_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_days_since_block: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("days_since_block"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+        })
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn compute(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        height_to_pool: &impl AnyIterableVec<Height, PoolId>,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.indexes_to_blocks_mined
+            .compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_transform(
+                    starting_indexes.height,
+                    height_to_pool,
+                    |(h, id, ..)| {
+                        (
+                            h,
+                            if id == self.id {
+                                StoredU32::ONE
+                            } else {
+                                StoredU32::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1w_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    7,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1m_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    30,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1y_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    365,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        let height_to_blocks_mined = self.indexes_to_blocks_mined.height.as_ref().unwrap();
+
+        self.indexes_to_subsidy
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_subsidy.sats.height.as_ref().unwrap(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_fee
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_fee.sats.height.unwrap_sum(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_coinbase
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_coinbase.sats.height.as_ref().unwrap(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_cumulative(),
+                    chain.indexes_to_block_count.dateindex.unwrap_cumulative(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1d_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    chain.indexes_to_block_count.dateindex.unwrap_sum(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1w_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1w_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1w_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1m_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1m_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1m_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1y_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1y_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1y_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_days_since_block
+            .compute_all(starting_indexes, exit, |v| {
+                let mut prev = None;
+                v.compute_transform2(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    self.indexes_to_blocks_mined.dateindex.unwrap_cumulative(),
+                    |(i, sum, cumulative, slf)| {
+                        if prev.is_none() {
+                            let i = i.unwrap_to_usize();
+                            prev.replace(if i > 0 {
+                                slf.into_iter().unwrap_get_inner_(i - 1)
+                            } else {
+                                StoredU16::ZERO
+                            });
+                        }
+                        let days = if !cumulative.is_zero() && sum.is_zero() {
+                            prev.unwrap() + StoredU16::ONE
+                        } else {
+                            StoredU16::ZERO
+                        };
+                        prev.replace(days);
+                        (i, days)
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
+
+        iter = Box::new(iter.chain(self.indexes_to_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1w_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1m_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1y_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_subsidy.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_fee.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_coinbase.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1d_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1w_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1m_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1y_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_days_since_block.iter_any_collectable()));
+
+        iter
+    }
+}

crates/brk_computer/src/stateful/address_cohort.rs

@@ -1,7 +1,6 @@
-use std::{ops::Deref, path::Path};
+use std::path::Path;
 
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, DateIndex, Dollars, Height, StoredU64, Version};
 use vecdb::{
     AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, EagerVec, Exit, Format,
@@ -11,7 +10,7 @@ use vecdb::{
 use crate::{
     Indexes,
     grouped::{ComputedVecsFromHeight, Source, VecBuilderOptions},
-    indexes, market, price,
+    indexes, price,
     stateful::{
         common,
         r#trait::{CohortVecs, DynCohortVecs},
@@ -29,8 +28,8 @@ pub struct Vecs {
 
     pub inner: common::Vecs,
 
-    pub height_to_address_count: EagerVec<Height, StoredU64>,
-    pub indexes_to_address_count: ComputedVecsFromHeight<StoredU64>,
+    pub height_to_addr_count: EagerVec<Height, StoredU64>,
+    pub indexes_to_addr_count: ComputedVecsFromHeight<StoredU64>,
 }
 
 impl Vecs {
@@ -43,7 +42,7 @@ impl Vecs {
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         states_path: Option<&Path>,
-        compute_relative_to_all: bool,
+        compute_rel_to_all: bool,
     ) -> Result<Self> {
         let compute_dollars = price.is_some();
 
@@ -58,15 +57,15 @@ impl Vecs {
                     compute_dollars,
                 )
             }),
-            height_to_address_count: EagerVec::forced_import(
+            height_to_addr_count: EagerVec::forced_import(
                 db,
-                &suffix("address_count"),
+                &suffix("addr_count"),
                 version + VERSION + Version::ZERO,
                 format,
             )?,
-            indexes_to_address_count: ComputedVecsFromHeight::forced_import(
+            indexes_to_addr_count: ComputedVecsFromHeight::forced_import(
                 db,
-                &suffix("address_count"),
+                &suffix("addr_count"),
                 Source::None,
                 version + VERSION + Version::ZERO,
                 indexes,
@@ -79,23 +78,27 @@ impl Vecs {
                 version,
                 indexes,
                 price,
-                compute_relative_to_all,
                 false,
+                compute_rel_to_all,
                 false,
             )?,
         })
     }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        self.inner
+            .iter_any_collectable()
+            .chain([&self.height_to_addr_count as &dyn AnyCollectableVec])
+            .chain(self.indexes_to_addr_count.iter_any_collectable())
+    }
 }
 
 impl DynCohortVecs for Vecs {
     fn min_height_vecs_len(&self) -> usize {
-        [
-            self.height_to_address_count.len(),
+        std::cmp::min(
+            self.height_to_addr_count.len(),
             self.inner.min_height_vecs_len(),
-        ]
-        .into_iter()
-        .min()
-        .unwrap()
+        )
     }
 
     fn reset_state_starting_height(&mut self) {
@@ -110,8 +113,8 @@ impl DynCohortVecs for Vecs {
         self.starting_height = Some(starting_height);
 
         if let Some(prev_height) = starting_height.decremented() {
-            self.state.as_mut().unwrap().address_count = *self
-                .height_to_address_count
+            self.state.as_mut().unwrap().addr_count = *self
+                .height_to_addr_count
                 .into_iter()
                 .unwrap_get_inner(prev_height);
         }
@@ -120,9 +123,9 @@ impl DynCohortVecs for Vecs {
     }
 
     fn validate_computed_versions(&mut self, base_version: Version) -> Result<()> {
-        self.height_to_address_count
+        self.height_to_addr_count
             .validate_computed_version_or_reset(
-                base_version + self.height_to_address_count.inner_version(),
+                base_version + self.height_to_addr_count.inner_version(),
             )?;
 
         self.inner.validate_computed_versions(base_version)
@@ -133,9 +136,9 @@ impl DynCohortVecs for Vecs {
             return Ok(());
         }
 
-        self.height_to_address_count.forced_push_at(
+        self.height_to_addr_count.forced_push_at(
             height,
-            self.state.as_ref().unwrap().address_count.into(),
+            self.state.as_ref().unwrap().addr_count.into(),
             exit,
         )?;
 
@@ -162,7 +165,7 @@ impl DynCohortVecs for Vecs {
     }
 
     fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
-        self.height_to_address_count.safe_flush(exit)?;
+        self.height_to_addr_count.safe_flush(exit)?;
 
         self.inner
             .safe_flush_stateful_vecs(height, exit, &mut self.state.as_mut().unwrap().inner)
@@ -171,30 +174,20 @@ impl DynCohortVecs for Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        self.indexes_to_address_count.compute_rest(
+        self.indexes_to_addr_count.compute_rest(
             indexes,
             starting_indexes,
             exit,
-            Some(&self.height_to_address_count),
+            Some(&self.height_to_addr_count),
         )?;
 
         self.inner
-            .compute_rest_part1(indexer, indexes, price, starting_indexes, exit)
-    }
-
-    fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.inner.vecs(),
-            self.indexes_to_address_count.vecs(),
-            vec![&self.height_to_address_count],
-        ]
-        .concat()
+            .compute_rest_part1(indexes, price, starting_indexes, exit)
     }
 }
 
@@ -205,11 +198,11 @@ impl CohortVecs for Vecs {
         others: &[&Self],
         exit: &Exit,
     ) -> Result<()> {
-        self.height_to_address_count.compute_sum_of_others(
+        self.height_to_addr_count.compute_sum_of_others(
             starting_indexes.height,
             others
                 .iter()
-                .map(|v| &v.height_to_address_count)
+                .map(|v| &v.height_to_addr_count)
                 .collect::<Vec<_>>()
                 .as_slice(),
             exit,
@@ -224,35 +217,28 @@ impl CohortVecs for Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,
     ) -> Result<()> {
         self.inner.compute_rest_part2(
-            indexer,
             indexes,
             price,
             starting_indexes,
-            market,
             height_to_supply,
             dateindex_to_supply,
+            height_to_market_cap,
+            dateindex_to_market_cap,
             height_to_realized_cap,
             dateindex_to_realized_cap,
             exit,
         )
     }
 }
-
-impl Deref for Vecs {
-    type Target = common::Vecs;
-    fn deref(&self) -> &Self::Target {
-        &self.inner
-    }
-}

crates/brk_computer/src/stateful/address_cohorts.rs

@@ -1,7 +1,6 @@
 use std::path::Path;
 
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{
     AddressGroups, Bitcoin, ByAmountRange, ByGreatEqualAmount, ByLowerThanAmount, DateIndex,
     Dollars, GroupFilter, Height, Version,
@@ -10,7 +9,7 @@ use derive_deref::{Deref, DerefMut};
 use vecdb::{AnyIterableVec, Database, Exit, Format};
 
 use crate::{
-    Indexes, indexes, market, price,
+    Indexes, indexes, price,
     stateful::{
         address_cohort,
         r#trait::{CohortVecs, DynCohortVecs},
@@ -459,18 +458,17 @@ impl Vecs {
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        let by_size_range = self.0.amount_range.as_vec();
+        let by_size_range = &self.0.amount_range;
 
         [
             self.0
                 .ge_amount
-                .as_mut_vec()
-                .into_iter()
+                .iter_mut()
                 .map(|(filter, vecs)| {
                     (
                         vecs,
                         by_size_range
-                            .into_iter()
+                            .iter()
                             .filter(|(other, _)| filter.includes(other))
                             .map(|(_, v)| v)
                             .collect::<Vec<_>>(),
@@ -479,13 +477,12 @@ impl Vecs {
                 .collect::<Vec<_>>(),
             self.0
                 .lt_amount
-                .as_mut_vec()
-                .into_iter()
+                .iter_mut()
                 .map(|(filter, vecs)| {
                     (
                         vecs,
                         by_size_range
-                            .into_iter()
+                            .iter()
                             .filter(|(other, _)| filter.includes(other))
                             .map(|(_, v)| v)
                             .collect::<Vec<_>>(),
@@ -502,51 +499,48 @@ impl Vecs {
 
     pub fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        self.as_mut_vecs().into_iter().try_for_each(|(_, v)| {
-            v.compute_rest_part1(indexer, indexes, price, starting_indexes, exit)
-        })
+        self.iter_mut()
+            .into_iter()
+            .try_for_each(|(_, v)| v.compute_rest_part1(indexes, price, starting_indexes, exit))
     }
 
     #[allow(clippy::too_many_arguments)]
     pub fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,
     ) -> Result<()> {
-        self.0.as_boxed_mut_vecs().into_iter().try_for_each(|v| {
-            v.into_iter().try_for_each(|(_, v)| {
-                v.compute_rest_part2(
-                    indexer,
-                    indexes,
-                    price,
-                    starting_indexes,
-                    market,
-                    height_to_supply,
-                    dateindex_to_supply,
-                    height_to_realized_cap,
-                    dateindex_to_realized_cap,
-                    exit,
-                )
-            })
+        self.0.iter_mut().try_for_each(|(_, v)| {
+            v.compute_rest_part2(
+                indexes,
+                price,
+                starting_indexes,
+                height_to_supply,
+                dateindex_to_supply,
+                height_to_market_cap,
+                dateindex_to_market_cap,
+                height_to_realized_cap,
+                dateindex_to_realized_cap,
+                exit,
+            )
         })
     }
 
     pub fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
-        self.as_mut_separate_vecs()
+        self.iter_separate_mut()
             .into_iter()
             .try_for_each(|(_, v)| v.safe_flush_stateful_vecs(height, exit))
     }

crates/brk_computer/src/stateful/addresstype/indexes_to_addresscount.rs

@@ -75,11 +75,9 @@ impl AddressTypeToIndexesToAddressCount {
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
         self.0
-            .as_typed_vec()
-            .into_iter()
-            .flat_map(|(_, v)| v.vecs())
-            .collect::<Vec<_>>()
+            .iter_typed()
+            .flat_map(|(_, v)| v.iter_any_collectable())
     }
 }

crates/brk_computer/src/stateful/addresstype/typeindex_tree.rs

@@ -27,6 +27,10 @@ impl<T> AddressTypeToTypeIndexTree<T> {
             mem::swap(own, other);
         }
     }
+
+    pub fn unwrap(self) -> ByAddressType<BTreeMap<TypeIndex, T>> {
+        self.0
+    }
 }
 
 impl<T> Default for AddressTypeToTypeIndexTree<T> {

crates/brk_computer/src/stateful/addresstype/vec.rs

@@ -38,6 +38,10 @@ impl<T> AddressTypeToVec<T> {
             mem::swap(own, other);
         }
     }
+
+    pub fn unwrap(self) -> ByAddressType<Vec<T>> {
+        self.0
+    }
 }
 
 impl<T> Default for AddressTypeToVec<T> {

crates/brk_computer/src/stateful/trait.rs

@@ -1,9 +1,8 @@
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, DateIndex, Dollars, Height, Version};
-use vecdb::{AnyCollectableVec, AnyIterableVec, Exit};
+use vecdb::{AnyIterableVec, Exit};
 
-use crate::{Indexes, indexes, market, price};
+use crate::{Indexes, indexes, price};
 
 pub trait DynCohortVecs: Send + Sync {
     fn min_height_vecs_len(&self) -> usize;
@@ -29,14 +28,11 @@ pub trait DynCohortVecs: Send + Sync {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()>;
-
-    fn vecs(&self) -> Vec<&dyn AnyCollectableVec>;
 }
 
 pub trait CohortVecs: DynCohortVecs {
@@ -50,13 +46,13 @@ pub trait CohortVecs: DynCohortVecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,

crates/brk_computer/src/stateful/utxo_cohort.rs

@@ -1,12 +1,11 @@
 use std::{ops::Deref, path::Path};
 
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{Bitcoin, DateIndex, Dollars, Height, Version};
-use vecdb::{AnyCollectableVec, AnyIterableVec, Database, Exit, Format};
+use vecdb::{AnyIterableVec, Database, Exit, Format};
 
 use crate::{
-    Indexes, UTXOCohortState, indexes, market, price,
+    Indexes, UTXOCohortState, indexes, price,
     stateful::{
         common,
         r#trait::{CohortVecs, DynCohortVecs},
@@ -32,8 +31,8 @@ impl Vecs {
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         states_path: Option<&Path>,
-        compute_relative_to_all: bool,
-        ratio_extended: bool,
+        extended: bool,
+        compute_rel_to_all: bool,
         compute_adjusted: bool,
     ) -> Result<Self> {
         let compute_dollars = price.is_some();
@@ -56,8 +55,8 @@ impl Vecs {
                 version,
                 indexes,
                 price,
-                compute_relative_to_all,
-                ratio_extended,
+                extended,
+                compute_rel_to_all,
                 compute_adjusted,
             )?,
         })
@@ -122,18 +121,13 @@ impl DynCohortVecs for Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
         self.inner
-            .compute_rest_part1(indexer, indexes, price, starting_indexes, exit)
-    }
-
-    fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        self.inner.vecs()
+            .compute_rest_part1(indexes, price, starting_indexes, exit)
     }
 }
 
@@ -154,25 +148,25 @@ impl CohortVecs for Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,
     ) -> Result<()> {
         self.inner.compute_rest_part2(
-            indexer,
             indexes,
             price,
             starting_indexes,
-            market,
             height_to_supply,
             dateindex_to_supply,
+            height_to_market_cap,
+            dateindex_to_market_cap,
             height_to_realized_cap,
             dateindex_to_realized_cap,
             exit,

crates/brk_computer/src/stateful/utxo_cohorts.rs

@@ -1,7 +1,6 @@
 use std::{collections::BTreeMap, ops::ControlFlow, path::Path};
 
 use brk_error::Result;
-use brk_indexer::Indexer;
 use brk_structs::{
     Bitcoin, ByAgeRange, ByAmountRange, ByEpoch, ByGreatEqualAmount, ByLowerThanAmount, ByMaxAge,
     ByMinAge, BySpendableType, ByTerm, CheckedSub, DateIndex, Dollars, GroupFilter, HalvingEpoch,
@@ -11,7 +10,7 @@ use derive_deref::{Deref, DerefMut};
 use vecdb::{AnyIterableVec, Database, Exit, Format, StoredIndex};
 
 use crate::{
-    Indexes, indexes, market, price,
+    Indexes, indexes, price,
     stateful::r#trait::DynCohortVecs,
     states::{BlockState, Transacted},
 };
@@ -42,14 +41,14 @@ impl Vecs {
                     indexes,
                     price,
                     None,
-                    false,
                     true,
+                    false,
                     true,
                 )?,
                 term: ByTerm {
                     short: utxo_cohort::Vecs::forced_import(
                         db,
-                        Some("short_term_holders"),
+                        Some("sth"),
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
@@ -61,7 +60,7 @@ impl Vecs {
                     )?,
                     long: utxo_cohort::Vecs::forced_import(
                         db,
-                        Some("long_term_holders"),
+                        Some("lth"),
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
@@ -143,8 +142,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2pk33: utxo_cohort::Vecs::forced_import(
@@ -155,8 +154,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2pkh: utxo_cohort::Vecs::forced_import(
@@ -167,8 +166,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2sh: utxo_cohort::Vecs::forced_import(
@@ -179,8 +178,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2wpkh: utxo_cohort::Vecs::forced_import(
@@ -191,8 +190,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2wsh: utxo_cohort::Vecs::forced_import(
@@ -203,8 +202,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2tr: utxo_cohort::Vecs::forced_import(
@@ -215,8 +214,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2a: utxo_cohort::Vecs::forced_import(
@@ -227,8 +226,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     p2ms: utxo_cohort::Vecs::forced_import(
@@ -239,8 +238,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     empty: utxo_cohort::Vecs::forced_import(
@@ -251,8 +250,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     unknown: utxo_cohort::Vecs::forced_import(
@@ -263,8 +262,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                 },
@@ -955,8 +954,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1sat_to_10sats: utxo_cohort::Vecs::forced_import(
@@ -967,8 +966,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10sats_to_100sats: utxo_cohort::Vecs::forced_import(
@@ -979,8 +978,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100sats_to_1k_sats: utxo_cohort::Vecs::forced_import(
@@ -991,8 +990,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_sats_to_10k_sats: utxo_cohort::Vecs::forced_import(
@@ -1003,8 +1002,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_sats_to_100k_sats: utxo_cohort::Vecs::forced_import(
@@ -1015,8 +1014,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100k_sats_to_1m_sats: utxo_cohort::Vecs::forced_import(
@@ -1027,8 +1026,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1m_sats_to_10m_sats: utxo_cohort::Vecs::forced_import(
@@ -1039,8 +1038,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10m_sats_to_1btc: utxo_cohort::Vecs::forced_import(
@@ -1051,8 +1050,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1btc_to_10btc: utxo_cohort::Vecs::forced_import(
@@ -1063,8 +1062,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10btc_to_100btc: utxo_cohort::Vecs::forced_import(
@@ -1075,8 +1074,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100btc_to_1k_btc: utxo_cohort::Vecs::forced_import(
@@ -1087,8 +1086,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_btc_to_10k_btc: utxo_cohort::Vecs::forced_import(
@@ -1099,8 +1098,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_btc_to_100k_btc: utxo_cohort::Vecs::forced_import(
@@ -1111,8 +1110,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100k_btc_or_more: utxo_cohort::Vecs::forced_import(
@@ -1123,8 +1122,8 @@ impl Vecs {
                         indexes,
                         price,
                         Some(states_path),
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                 },
@@ -1137,8 +1136,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100sats: utxo_cohort::Vecs::forced_import(
@@ -1149,8 +1148,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_sats: utxo_cohort::Vecs::forced_import(
@@ -1161,8 +1160,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_sats: utxo_cohort::Vecs::forced_import(
@@ -1173,8 +1172,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100k_sats: utxo_cohort::Vecs::forced_import(
@@ -1185,8 +1184,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1m_sats: utxo_cohort::Vecs::forced_import(
@@ -1197,8 +1196,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10m_sats: utxo_cohort::Vecs::forced_import(
@@ -1209,8 +1208,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1btc: utxo_cohort::Vecs::forced_import(
@@ -1221,8 +1220,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10btc: utxo_cohort::Vecs::forced_import(
@@ -1233,8 +1232,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100btc: utxo_cohort::Vecs::forced_import(
@@ -1245,8 +1244,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_btc: utxo_cohort::Vecs::forced_import(
@@ -1257,8 +1256,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_btc: utxo_cohort::Vecs::forced_import(
@@ -1269,8 +1268,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100k_btc: utxo_cohort::Vecs::forced_import(
@@ -1281,8 +1280,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                 },
@@ -1295,8 +1294,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10sats: utxo_cohort::Vecs::forced_import(
@@ -1307,8 +1306,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100sats: utxo_cohort::Vecs::forced_import(
@@ -1319,8 +1318,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_sats: utxo_cohort::Vecs::forced_import(
@@ -1331,8 +1330,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_sats: utxo_cohort::Vecs::forced_import(
@@ -1343,8 +1342,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100k_sats: utxo_cohort::Vecs::forced_import(
@@ -1355,8 +1354,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1m_sats: utxo_cohort::Vecs::forced_import(
@@ -1367,8 +1366,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10m_sats: utxo_cohort::Vecs::forced_import(
@@ -1379,8 +1378,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1btc: utxo_cohort::Vecs::forced_import(
@@ -1391,8 +1390,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10btc: utxo_cohort::Vecs::forced_import(
@@ -1403,8 +1402,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _100btc: utxo_cohort::Vecs::forced_import(
@@ -1415,8 +1414,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _1k_btc: utxo_cohort::Vecs::forced_import(
@@ -1427,8 +1426,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                     _10k_btc: utxo_cohort::Vecs::forced_import(
@@ -1439,8 +1438,8 @@ impl Vecs {
                         indexes,
                         price,
                         None,
-                        true,
                         false,
+                        true,
                         false,
                     )?,
                 },
@@ -1458,8 +1457,7 @@ impl Vecs {
 
         let mut vecs = self
             .age_range
-            .as_mut_vec()
-            .into_iter()
+            .iter_mut()
             .map(|(filter, v)| (filter, &mut v.state))
             .collect::<Vec<_>>();
 
@@ -1503,9 +1501,8 @@ impl Vecs {
         let mut time_based_vecs = self
             .0
             .age_range
-            .as_mut_vec()
-            .into_iter()
-            .chain(self.0.epoch.as_mut_vec())
+            .iter_mut()
+            .chain(self.0.epoch.iter_mut())
             .collect::<Vec<_>>();
 
         let last_timestamp = chain_state.last().unwrap().timestamp;
@@ -1551,8 +1548,10 @@ impl Vecs {
                     );
                 });
 
-            sent.by_type.spendable.as_typed_vec().into_iter().for_each(
-                |(output_type, supply_state)| {
+            sent.by_type
+                .spendable
+                .iter_typed()
+                .for_each(|(output_type, supply_state)| {
                     self.0
                         ._type
                         .get_mut(output_type)
@@ -1568,12 +1567,10 @@ impl Vecs {
                             days_old_float,
                             older_than_hour,
                         )
-                },
-            );
+                });
 
             sent.by_size_group
-                .as_typed_vec()
-                .into_iter()
+                .iter_typed()
                 .for_each(|(group, supply_state)| {
                     self.0
                         .amount_range
@@ -1606,24 +1603,20 @@ impl Vecs {
             v.state.as_mut().unwrap().receive(&supply_state, price);
         });
 
-        self._type
-            .as_mut_vec()
-            .into_iter()
-            .for_each(|(filter, vecs)| {
-                let output_type = match filter {
-                    GroupFilter::Type(output_type) => *output_type,
-                    _ => unreachable!(),
-                };
-                vecs.state
-                    .as_mut()
-                    .unwrap()
-                    .receive(received.by_type.get(output_type), price)
-            });
+        self._type.iter_mut().for_each(|(filter, vecs)| {
+            let output_type = match filter {
+                GroupFilter::Type(output_type) => *output_type,
+                _ => unreachable!(),
+            };
+            vecs.state
+                .as_mut()
+                .unwrap()
+                .receive(received.by_type.get(output_type), price)
+        });
 
         received
             .by_size_group
-            .as_typed_vec()
-            .into_iter()
+            .iter_typed()
             .for_each(|(group, supply_state)| {
                 self.amount_range
                     .get_mut(group)
@@ -1640,95 +1633,64 @@ impl Vecs {
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        let by_date_range = self.0.age_range.as_vec();
-        let by_size_range = self.0.amount_range.as_vec();
+        let by_date_range = &self.0.age_range;
+        let by_size_range = &self.0.amount_range;
 
-        [
-            vec![(
-                &mut self.0.all.1,
+        [(
+            &mut self.0.all.1,
+            by_date_range.iter().map(|(_, v)| v).collect::<Vec<_>>(),
+        )]
+        .into_iter()
+        .chain(self.0.min_age.iter_mut().map(|(filter, vecs)| {
+            (
+                vecs,
                 by_date_range
-                    .into_iter()
+                    .iter()
+                    .filter(|(other, _)| filter.includes(other))
                     .map(|(_, v)| v)
                     .collect::<Vec<_>>(),
-            )],
-            self.0
-                .min_age
-                .as_mut_vec()
-                .into_iter()
-                .map(|(filter, vecs)| {
-                    (
-                        vecs,
-                        by_date_range
-                            .into_iter()
-                            .filter(|(other, _)| filter.includes(other))
-                            .map(|(_, v)| v)
-                            .collect::<Vec<_>>(),
-                    )
-                })
-                .collect::<Vec<_>>(),
-            self.0
-                .max_age
-                .as_mut_vec()
-                .into_iter()
-                .map(|(filter, vecs)| {
-                    (
-                        vecs,
-                        by_date_range
-                            .into_iter()
-                            .filter(|(other, _)| filter.includes(other))
-                            .map(|(_, v)| v)
-                            .collect::<Vec<_>>(),
-                    )
-                })
-                .collect::<Vec<_>>(),
-            self.0
-                .term
-                .as_mut_vec()
-                .into_iter()
-                .map(|(filter, vecs)| {
-                    (
-                        vecs,
-                        by_date_range
-                            .into_iter()
-                            .filter(|(other, _)| filter.includes(other))
-                            .map(|(_, v)| v)
-                            .collect::<Vec<_>>(),
-                    )
-                })
-                .collect::<Vec<_>>(),
-            self.0
-                .ge_amount
-                .as_mut_vec()
-                .into_iter()
-                .map(|(filter, vecs)| {
-                    (
-                        vecs,
-                        by_size_range
-                            .into_iter()
-                            .filter(|(other, _)| filter.includes(other))
-                            .map(|(_, v)| v)
-                            .collect::<Vec<_>>(),
-                    )
-                })
-                .collect::<Vec<_>>(),
-            self.0
-                .lt_amount
-                .as_mut_vec()
-                .into_iter()
-                .map(|(filter, vecs)| {
-                    (
-                        vecs,
-                        by_size_range
-                            .into_iter()
-                            .filter(|(other, _)| filter.includes(other))
-                            .map(|(_, v)| v)
-                            .collect::<Vec<_>>(),
-                    )
-                })
-                .collect::<Vec<_>>(),
-        ]
-        .into_iter()
-        .flatten()
+            )
+        }))
+        .chain(self.0.max_age.iter_mut().map(|(filter, vecs)| {
+            (
+                vecs,
+                by_date_range
+                    .iter()
+                    .filter(|(other, _)| filter.includes(other))
+                    .map(|(_, v)| v)
+                    .collect::<Vec<_>>(),
+            )
+        }))
+        .chain(self.0.term.iter_mut().map(|(filter, vecs)| {
+            (
+                vecs,
+                by_date_range
+                    .iter()
+                    .filter(|(other, _)| filter.includes(other))
+                    .map(|(_, v)| v)
+                    .collect::<Vec<_>>(),
+            )
+        }))
+        .chain(self.0.ge_amount.iter_mut().map(|(filter, vecs)| {
+            (
+                vecs,
+                by_size_range
+                    .iter()
+                    .filter(|(other, _)| filter.includes(other))
+                    .map(|(_, v)| v)
+                    .collect::<Vec<_>>(),
+            )
+        }))
+        .chain(self.0.lt_amount.iter_mut().map(|(filter, vecs)| {
+            (
+                vecs,
+                by_size_range
+                    .iter()
+                    .filter(|(other, _)| filter.includes(other))
+                    .map(|(_, v)| v)
+                    .collect::<Vec<_>>(),
+            )
+        }))
         .try_for_each(|(vecs, stateful)| {
             vecs.compute_from_stateful(starting_indexes, &stateful, exit)
         })
@@ -1736,52 +1698,48 @@ impl Vecs {
 
     pub fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        self.as_mut_vecs().into_iter().try_for_each(|(_, v)| {
-            v.compute_rest_part1(indexer, indexes, price, starting_indexes, exit)
-        })
+        self.iter_mut()
+            .into_iter()
+            .try_for_each(|(_, v)| v.compute_rest_part1(indexes, price, starting_indexes, exit))
     }
 
     #[allow(clippy::too_many_arguments)]
     pub fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,
     ) -> Result<()> {
-        self.0.as_boxed_mut_vecs().into_iter().try_for_each(|v| {
-            v.into_iter().try_for_each(|(_, v)| {
-                v.compute_rest_part2(
-                    indexer,
-                    indexes,
-                    price,
-                    starting_indexes,
-                    market,
-                    height_to_supply,
-                    dateindex_to_supply,
-                    height_to_realized_cap,
-                    dateindex_to_realized_cap,
-                    exit,
-                )
-            })
+        self.iter_mut().into_iter().try_for_each(|(_, v)| {
+            v.compute_rest_part2(
+                indexes,
+                price,
+                starting_indexes,
+                height_to_supply,
+                dateindex_to_supply,
+                height_to_market_cap,
+                dateindex_to_market_cap,
+                height_to_realized_cap,
+                dateindex_to_realized_cap,
+                exit,
+            )
         })
     }
 
     pub fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
-        self.as_mut_separate_vecs()
-            .into_iter()
+        self.iter_separate_mut()
             .try_for_each(|(_, v)| v.safe_flush_stateful_vecs(height, exit))
     }
 }

crates/brk_computer/src/states/cohorts/address.rs

@@ -9,14 +9,14 @@ use super::CohortState;
 
 #[derive(Clone)]
 pub struct AddressCohortState {
-    pub address_count: u64,
+    pub addr_count: u64,
     pub inner: CohortState,
 }
 
 impl AddressCohortState {
     pub fn new(path: &Path, name: &str, compute_dollars: bool) -> Self {
         Self {
-            address_count: 0,
+            addr_count: 0,
             inner: CohortState::new(path, name, compute_dollars),
         }
     }
@@ -44,14 +44,14 @@ impl AddressCohortState {
 
         let prev_realized_price = compute_price.then(|| addressdata.realized_price());
         let prev_supply_state = SupplyState {
-            utxos: addressdata.outputs_len as u64,
+            utxos: addressdata.utxos as u64,
             value: addressdata.amount(),
         };
 
         addressdata.send(value, prev_price)?;
 
         let supply_state = SupplyState {
-            utxos: addressdata.outputs_len as u64,
+            utxos: addressdata.utxos as u64,
             value: addressdata.amount(),
         };
 
@@ -79,14 +79,14 @@ impl AddressCohortState {
 
         let prev_realized_price = compute_price.then(|| address_data.realized_price());
         let prev_supply_state = SupplyState {
-            utxos: address_data.outputs_len as u64,
+            utxos: address_data.utxos as u64,
             value: address_data.amount(),
         };
 
         address_data.receive(value, price);
 
         let supply_state = SupplyState {
-            utxos: address_data.outputs_len as u64,
+            utxos: address_data.utxos as u64,
             value: address_data.amount(),
         };
 
@@ -99,7 +99,7 @@ impl AddressCohortState {
     }
 
     pub fn add(&mut self, addressdata: &LoadedAddressData) {
-        self.address_count += 1;
+        self.addr_count += 1;
         self.inner.increment_(
             &addressdata.into(),
             addressdata.realized_cap,
@@ -108,7 +108,7 @@ impl AddressCohortState {
     }
 
     pub fn subtract(&mut self, addressdata: &LoadedAddressData) {
-        self.address_count = self.address_count.checked_sub(1).unwrap();
+        self.addr_count = self.addr_count.checked_sub(1).unwrap();
         self.inner.decrement_(
             &addressdata.into(),
             addressdata.realized_cap,

crates/brk_computer/src/states/cohorts/common.rs

@@ -283,7 +283,7 @@ impl CohortState {
                         }
                     }
                     Ordering::Equal => {
-                        state.supply_even += sats;
+                        state.supply_breakeven += sats;
                     }
                 }
             };

crates/brk_computer/src/states/supply.rs

@@ -43,8 +43,14 @@ impl SubAssign<&SupplyState> for SupplyState {
 impl From<&LoadedAddressData> for SupplyState {
     fn from(value: &LoadedAddressData) -> Self {
         Self {
-            utxos: value.outputs_len as u64,
+            utxos: value.utxos as u64,
             value: value.amount(),
         }
     }
 }
+
+impl std::fmt::Display for SupplyState {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "utxos: {}, value: {}", self.utxos, self.value)
+    }
+}

crates/brk_computer/src/states/unrealized.rs

@@ -3,7 +3,7 @@ use brk_structs::{Dollars, Sats};
 #[derive(Debug, Default, Clone)]
 pub struct UnrealizedState {
     pub supply_in_profit: Sats,
-    pub supply_even: Sats,
+    pub supply_breakeven: Sats,
     pub supply_in_loss: Sats,
     pub unrealized_profit: Dollars,
     pub unrealized_loss: Dollars,
@@ -12,7 +12,7 @@ pub struct UnrealizedState {
 impl UnrealizedState {
     pub const NAN: Self = Self {
         supply_in_profit: Sats::ZERO,
-        supply_even: Sats::ZERO,
+        supply_breakeven: Sats::ZERO,
         supply_in_loss: Sats::ZERO,
         unrealized_profit: Dollars::NAN,
         unrealized_loss: Dollars::NAN,
@@ -20,7 +20,7 @@ impl UnrealizedState {
 
     pub const ZERO: Self = Self {
         supply_in_profit: Sats::ZERO,
-        supply_even: Sats::ZERO,
+        supply_breakeven: Sats::ZERO,
         supply_in_loss: Sats::ZERO,
         unrealized_profit: Dollars::ZERO,
         unrealized_loss: Dollars::ZERO,

crates/brk_error/Cargo.toml

@@ -10,10 +10,11 @@ rust-version.workspace = true
 build = "build.rs"
 
 [dependencies]
-vecdb = { workspace = true }
+bitcoin = { workspace = true }
 bitcoincore-rpc = { workspace = true }
 fjall = { workspace = true }
 jiff = { workspace = true }
 minreq = { workspace = true }
 serde_json = { workspace = true }
+vecdb = { workspace = true }
 zerocopy = { workspace = true }

crates/brk_error/README.md

@@ -1,140 +1,143 @@
 # brk_error
 
-**Centralized error handling for the Bitcoin Research Kit**
+Centralized error handling for Bitcoin-related operations and database interactions.
 
-`brk_error` provides a unified error type and result system used throughout the BRK ecosystem. It consolidates error handling from multiple external dependencies and adds Bitcoin-specific error variants.
+[![Crates.io](https://img.shields.io/crates/v/brk_error.svg)](https://crates.io/crates/brk_error)
+[![Documentation](https://docs.rs/brk_error/badge.svg)](https://docs.rs/brk_error)
 
-## What it provides
+## Overview
 
-- **Unified Error Type**: Single `Error` enum that covers all error cases across BRK crates
-- **Convenient Result Type**: Pre-configured `Result<T, E = Error>` for consistent error handling
-- **External Error Integration**: Automatic conversions from common library errors
-- **Bitcoin-Specific Errors**: Domain-specific error variants for blockchain data processing
+This crate provides a unified error type that consolidates error handling across Bitcoin blockchain analysis tools. It wraps errors from multiple external libraries including Bitcoin Core RPC, database operations, HTTP requests, and serialization operations into a single `Error` enum.
 
-## Key Features
+**Key Features:**
 
-### Centralized Error Management
-- Single error type for the entire BRK ecosystem
-- Consistent error handling patterns across all crates
-- Reduced error type complexity in public APIs
+- Unified error type covering 11+ different error sources
+- Automatic conversions from external library errors
+- Bitcoin-specific error variants for blockchain operations
+- Database error handling for both Fjall and VecDB storage backends
+- Custom error types for domain-specific validation failures
 
-### External Library Integration
-Automatic `From` implementations for errors from:
-- **I/O Operations**: `std::io::Error`
-- **Bitcoin Core RPC**: `bitcoincore_rpc::Error`
-- **Database Operations**: `fjall::Error`, `vecdb::Error`
-- **Serialization**: `serde_json::Error`
-- **Time Operations**: `jiff::Error`, `SystemTimeError`
-- **HTTP Requests**: `minreq::Error`
-- **Zero-Copy Operations**: `zerocopy` conversion errors
+**Target Use Cases:**
 
-### Bitcoin-Specific Error Variants
-- `WrongAddressType` - Invalid address type for operation
-- `UnindexableDate` - Date before Bitcoin genesis (2009-01-03)
-- `WrongLength` - Invalid data length for Bitcoin structures
-- `QuickCacheError` - Cache operation failures
+- Applications processing Bitcoin blockchain data
+- Systems requiring unified error handling across multiple storage backends
+- Tools integrating Bitcoin Core RPC with local databases
 
-## Usage
+## Installation
 
-### Basic Error Handling
+```bash
+cargo add brk_error
+```
+
+## Quick Start
 
 ```rust
 use brk_error::{Error, Result};
 
-fn process_block() -> Result<Block> {
-    let rpc_client = get_rpc_client()?;  // bitcoincore_rpc::Error -> Error
-    let block_data = rpc_client.get_block_info(&hash)?;
-    
-    // Custom Bitcoin-specific validation
-    if block_data.height < 0 {
-        return Err(Error::Str("Invalid block height"));
+fn process_transaction() -> Result<()> {
+    // Function automatically converts various error types
+    let data = std::fs::read("transaction.json")?; // IO error auto-converted
+    let parsed: serde_json::Value = serde_json::from_slice(&data)?; // JSON error auto-converted
+
+    // Custom domain errors
+    if data.len() < 32 {
+        return Err(Error::WrongLength);
     }
-    
-    Ok(block_data)
+
+    Ok(())
 }
 ```
 
-### Working with External Libraries
+## API Overview
+
+### Core Types
+
+- **`Error`**: Main error enum consolidating all error types
+- **`Result<T, E = Error>`**: Type alias for `std::result::Result` with default `Error` type
+
+### Error Categories
+
+**External Library Errors:**
+
+- `BitcoinRPC`: Bitcoin Core RPC client errors
+- `BitcoinConsensusEncode`: Bitcoin consensus encoding failures
+- `Fjall`/`VecDB`/`SeqDB`: Database operation errors
+- `Minreq`: HTTP request errors
+- `SerdeJson`: JSON serialization errors
+- `Jiff`: Date/time handling errors
+- `ZeroCopyError`: Zero-copy conversion failures
+
+**Domain-Specific Errors:**
+
+- `WrongLength`: Invalid data length for Bitcoin operations
+- `WrongAddressType`: Unsupported Bitcoin address format
+- `UnindexableDate`: Date outside valid blockchain range (before 2009-01-03)
+- `QuickCacheError`: Cache operation failures
+
+**Generic Errors:**
+
+- `Str(&'static str)`: Static string errors
+- `String(String)`: Dynamic string errors
+
+### Key Methods
+
+All external error types automatically convert to `Error` via `From` trait implementations. The error type implements `std::error::Error`, `Debug`, and `Display` traits for comprehensive error reporting.
+
+## Examples
+
+### Bitcoin RPC Integration
+
+```rust
+use brk_error::Result;
+use bitcoincore_rpc::{Client, Auth};
+
+fn get_block_count(client: &Client) -> Result<u64> {
+    let count = client.get_block_count()?; // Auto-converts bitcoincore_rpc::Error
+    Ok(count)
+}
+```
+
+### Database Operations
 
 ```rust
 use brk_error::Result;
 
-fn save_data(data: &[u8]) -> Result<()> {
-    // I/O error automatically converted
-    std::fs::write("data.bin", data)?;
-    
-    // JSON serialization error automatically converted  
-    let json = serde_json::to_string(&data)?;
-    
-    // Database error automatically converted
-    database.insert("key", &json)?;
-    
+fn store_transaction_data(db: &fjall::Keyspace, data: &[u8]) -> Result<()> {
+    if data.len() != 32 {
+        return Err(brk_error::Error::WrongLength);
+    }
+
+    db.insert(b"tx_hash", data)?; // Auto-converts fjall::Error
     Ok(())
 }
 ```
 
-### Bitcoin-Specific Validation
+### Date Validation
 
 ```rust
 use brk_error::{Error, Result};
+use jiff::civil::Date;
 
-fn validate_date(date: &Date) -> Result<()> {
-    if *date < Date::new(2009, 1, 3) {
+fn validate_blockchain_date(date: Date) -> Result<()> {
+    let genesis_date = Date::constant(2009, 1, 3);
+    let earliest_valid = Date::constant(2009, 1, 9);
+
+    if date < genesis_date || (date > genesis_date && date < earliest_valid) {
         return Err(Error::UnindexableDate);
     }
-    Ok(())
-}
 
-fn validate_address_type(output_type: OutputType) -> Result<()> {
-    if !output_type.is_address() {
-        return Err(Error::WrongAddressType);
-    }
     Ok(())
 }
 ```
 
-### String Errors
+## Code Analysis Summary
 
-```rust
-// Static string errors (zero allocation)
-Err(Error::Str("Invalid configuration"))
-
-// Dynamic string errors
-Err(Error::String(format!("Block {} not found", height)))
-```
-
-## Result Type
-
-The crate provides a convenient `Result` type alias:
-
-```rust
-pub type Result<T, E = Error> = std::result::Result<T, E>;
-```
-
-This allows for clean function signatures throughout BRK:
-
-```rust
-fn parse_block() -> Result<Block> { /* ... */ }
-fn index_transactions() -> Result<Vec<Transaction>> { /* ... */ }
-```
-
-## Error Display
-
-All errors implement `Display` and `std::error::Error`, providing:
-- Formatted error messages for debugging
-- Error chain support for nested errors
-- Integration with error handling libraries like `anyhow`
-
-## Dependencies
-
-- `vecdb` - Vector database error types
-- `bitcoincore-rpc` - Bitcoin Core RPC client errors
-- `fjall` - Key-value store errors
-- `jiff` - Date/time operation errors
-- `minreq` - HTTP request errors
-- `serde_json` - JSON serialization errors
-- `zerocopy` - Zero-copy conversion errors
+**Main Type**: `Error` enum with 24 variants covering external libraries and domain-specific cases \
+**Conversion Traits**: Implements `From` for 10+ external error types enabling automatic error propagation \
+**Error Handling**: Standard Rust error handling with `std::error::Error` trait implementation \
+**Dependencies**: Integrates errors from `bitcoin`, `bitcoincore-rpc`, `fjall`, `vecdb`, `jiff`, `minreq`, `serde_json`, and `zerocopy` crates \
+**Architecture**: Centralized error aggregation pattern with automatic conversions and custom domain errors
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_error/src/lib.rs

@@ -17,6 +17,8 @@ pub enum Error {
     SeqDB(vecdb::SeqDBError),
     Minreq(minreq::Error),
     SystemTimeError(time::SystemTimeError),
+    BitcoinConsensusEncode(bitcoin::consensus::encode::Error),
+    BitcoinBip34Error(bitcoin::block::Bip34Error),
     SerdeJson(serde_json::Error),
     ZeroCopyError,
     Vecs(vecdb::Error),
@@ -29,6 +31,18 @@ pub enum Error {
     String(String),
 }
 
+impl From<bitcoin::block::Bip34Error> for Error {
+    fn from(value: bitcoin::block::Bip34Error) -> Self {
+        Self::BitcoinBip34Error(value)
+    }
+}
+
+impl From<bitcoin::consensus::encode::Error> for Error {
+    fn from(value: bitcoin::consensus::encode::Error) -> Self {
+        Self::BitcoinConsensusEncode(value)
+    }
+}
+
 impl From<time::SystemTimeError> for Error {
     fn from(value: time::SystemTimeError) -> Self {
         Self::SystemTimeError(value)
@@ -83,6 +97,12 @@ impl From<fjall::Error> for Error {
     }
 }
 
+impl From<&'static str> for Error {
+    fn from(value: &'static str) -> Self {
+        Self::Str(value)
+    }
+}
+
 impl<A, B, C> From<zerocopy::error::ConvertError<A, B, C>> for Error {
     fn from(_: zerocopy::error::ConvertError<A, B, C>) -> Self {
         Self::ZeroCopyError
@@ -98,16 +118,18 @@ impl<A, B> From<zerocopy::error::SizeError<A, B>> for Error {
 impl fmt::Display for Error {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match self {
-            Error::IO(error) => Display::fmt(&error, f),
-            Error::Minreq(error) => Display::fmt(&error, f),
-            Error::SerdeJson(error) => Display::fmt(&error, f),
-            Error::VecDB(error) => Display::fmt(&error, f),
-            Error::SeqDB(error) => Display::fmt(&error, f),
-            Error::Vecs(error) => Display::fmt(&error, f),
+            Error::BitcoinConsensusEncode(error) => Display::fmt(&error, f),
+            Error::BitcoinBip34Error(error) => Display::fmt(&error, f),
             Error::BitcoinRPC(error) => Display::fmt(&error, f),
-            Error::SystemTimeError(error) => Display::fmt(&error, f),
-            Error::Jiff(error) => Display::fmt(&error, f),
             Error::Fjall(error) => Display::fmt(&error, f),
+            Error::IO(error) => Display::fmt(&error, f),
+            Error::Jiff(error) => Display::fmt(&error, f),
+            Error::Minreq(error) => Display::fmt(&error, f),
+            Error::SeqDB(error) => Display::fmt(&error, f),
+            Error::SerdeJson(error) => Display::fmt(&error, f),
+            Error::SystemTimeError(error) => Display::fmt(&error, f),
+            Error::VecDB(error) => Display::fmt(&error, f),
+            Error::Vecs(error) => Display::fmt(&error, f),
             Error::ZeroCopyError => write!(f, "ZeroCopy error"),
 
             Error::WrongLength => write!(f, "Wrong length"),

crates/brk_fetcher/README.md

@@ -1,39 +1,36 @@
 # brk_fetcher
 
-**Bitcoin price data fetcher with multi-source fallback and retry logic**
+Multi-source Bitcoin price data aggregator with automatic fallback between exchanges.
 
-`brk_fetcher` provides reliable Bitcoin price data retrieval from multiple sources including Binance, Kraken, and BRK instances. It offers both date-based and block height-based price queries with automatic fallback and retry mechanisms for robust data collection.
+[![Crates.io](https://img.shields.io/crates/v/brk_fetcher.svg)](https://crates.io/crates/brk_fetcher)
+[![Documentation](https://docs.rs/brk_fetcher/badge.svg)](https://docs.rs/brk_fetcher)
 
-## What it provides
+## Overview
 
-- **Multi-source fallback**: Automatic fallback between Kraken → Binance → BRK
-- **Flexible querying**: Fetch prices by date or block height with timestamps
-- **Retry logic**: Built-in retry mechanism with exponential backoff
-- **Multiple timeframes**: 1-minute and 1-day interval support
-- **HAR file import**: Import historical Binance chart data from browser
+This crate provides a unified interface for fetching Bitcoin price data from multiple sources including Binance, Kraken, and a custom BRK API. It implements automatic failover between data sources, retry mechanisms, and supports both real-time and historical price queries using blockchain height or date-based lookups.
 
-## Key Features
+**Key Features:**
 
-### Data Sources
-- **Kraken API**: Primary source for OHLC data (1-day and 1-minute intervals)
-- **Binance API**: Secondary source with additional historical data
-- **BRK instance**: Fallback source for previously cached price data
-- **HAR import**: Manual historical data import from browser sessions
+- Multi-source price aggregation (Binance, Kraken, BRK API)
+- Automatic fallback hierarchy with intelligent retry logic
+- Historical price queries by blockchain height or date
+- Support for both 1-minute and daily OHLC data
+- HAR file import for extended historical data coverage
+- Built-in caching with BTreeMap storage for performance
 
-### Query Methods
-- **Date-based queries**: Get OHLC data for specific calendar dates
-- **Height-based queries**: Get OHLC data for specific block heights with timestamps
-- **Automatic aggregation**: Combines minute-level data for block intervals
+**Target Use Cases:**
 
-### Reliability Features
-- **Automatic fallback**: Tries sources in order until successful
-- **Retry mechanism**: Up to 12 hours of retries with 60-second intervals
-- **Cache clearing**: Automatic cache refresh on failures
-- **Error handling**: Graceful degradation with detailed error messages
+- Bitcoin blockchain analyzers requiring accurate historical pricing
+- Applications needing resilient price data with multiple fallbacks
+- Tools processing large datasets requiring efficient price lookups
 
-## Usage
+## Installation
 
-### Basic Setup
+```bash
+cargo add brk_fetcher
+```
+
+## Quick Start
 
 ```rust
 use brk_fetcher::Fetcher;
@@ -42,145 +39,132 @@ use brk_structs::{Date, Height, Timestamp};
 // Initialize fetcher with exchange APIs enabled
 let mut fetcher = Fetcher::import(true, None)?;
 
-// Initialize without exchange APIs (BRK-only mode)
-let mut fetcher = Fetcher::import(false, None)?;
+// Fetch price by date
+let date = Date::from_ymd(2023, 6, 15)?;
+let daily_price = fetcher.get_date(date)?;
 
-// Initialize with HAR file for historical Binance data
-let har_path = Path::new("./binance.har");
-let mut fetcher = Fetcher::import(true, Some(har_path))?;
+// Fetch price by blockchain height
+let height = Height::new(800000);
+let timestamp = Timestamp::from(1684771200u32);
+let block_price = fetcher.get_height(height, timestamp, None)?;
+
+println!("Daily OHLC: {:?}", daily_price);
+println!("Block OHLC: {:?}", block_price);
 ```
 
-### Date-based Price Queries
+## API Overview
+
+### Core Types
+
+- **`Fetcher`**: Main aggregator managing multiple price data sources
+- **`Binance`**: Binance exchange API client with HAR file support
+- **`Kraken`**: Kraken exchange API client for OHLC data
+- **`BRK`**: Custom API client for blockchain-indexed price data
+
+### Key Methods
+
+**`Fetcher::import(exchanges: bool, hars_path: Option<&Path>) -> Result<Self>`**
+Creates a new fetcher instance with configurable data sources.
+
+**`get_date(&mut self, date: Date) -> Result<OHLCCents>`**
+Retrieves daily OHLC data for the specified date with automatic source fallback.
+
+**`get_height(&mut self, height: Height, timestamp: Timestamp, previous_timestamp: Option<Timestamp>) -> Result<OHLCCents>`**
+Fetches price data for a specific blockchain height using minute-level precision.
+
+### Data Source Hierarchy
+
+1. **Kraken API** - Primary source for both 1-minute and daily data
+2. **Binance API** - Secondary source with extended HAR file support
+3. **BRK API** - Fallback source using blockchain-indexed pricing data
+
+### Error Handling
+
+The fetcher implements aggressive retry logic with exponential backoff, attempting each source up to 12 hours (720 retries) before failing. Failed requests trigger cache clearing and source rotation.
+
+## Examples
+
+### Basic Price Fetching
 
 ```rust
+use brk_fetcher::Fetcher;
 use brk_structs::Date;
 
-// Fetch OHLC data for a specific date
-let date = Date::new(2024, 12, 25);
-let ohlc = fetcher.get_date(date)?;
+let mut fetcher = Fetcher::import(true, None)?;
 
-println!("Bitcoin price on {}: ${:.2}", date, ohlc.close.dollars());
-println!("Daily high: ${:.2}", ohlc.high.dollars());
-println!("Daily low: ${:.2}", ohlc.low.dollars());
-```
-
-### Block Height-based Price Queries
-
-```rust
-use brk_structs::{Height, Timestamp};
-
-// Fetch price at specific block height
-let height = Height::new(900_000);
-let timestamp = Timestamp::from_block_height(height);
-let previous_timestamp = Some(Timestamp::from_block_height(Height::new(899_999)));
-
-let ohlc = fetcher.get_height(height, timestamp, previous_timestamp)?;
-println!("Bitcoin price at block {}: ${:.2}", height, ohlc.close.dollars());
-```
-
-### Working with OHLC Data
-
-```rust
-use brk_structs::OHLCCents;
-
-// OHLC data is returned in cents for precision
-let ohlc: OHLCCents = fetcher.get_date(date)?;
-
-// Convert to dollars for display
-println!("Open: ${:.2}", ohlc.open.dollars());
-println!("High: ${:.2}", ohlc.high.dollars());
-println!("Low: ${:.2}", ohlc.low.dollars());
-println!("Close: ${:.2}", ohlc.close.dollars());
-
-// Access raw cent values
-println!("Close in cents: {}", ohlc.close.0);
-```
-
-### Using Individual Sources
-
-```rust
-use brk_fetcher::{Binance, Kraken, BRK};
-
-// Use specific exchanges directly
-let binance = Binance::init(None);
-let kraken = Kraken::default();
-let brk = BRK::default();
-
-// Fetch from specific source
-let binance_data = binance.get_from_1d(&date)?;
-let kraken_data = kraken.get_from_1mn(timestamp, previous_timestamp)?;
-let brk_data = brk.get_from_height(height)?;
-```
-
-### Error Handling and Retries
-
-```rust
-// The fetcher automatically retries on failures
+// Fetch Bitcoin price for a specific date
+let date = Date::from_ymd(2021, 1, 1)?;
 match fetcher.get_date(date) {
-    Ok(ohlc) => println!("Successfully fetched: ${:.2}", ohlc.close.dollars()),
-    Err(e) => {
-        // After all retries and sources exhausted
-        eprintln!("Failed to fetch price data: {}", e);
+    Ok(ohlc) => println!("BTC price on {}: ${}", date, ohlc.close.to_dollars()),
+    Err(e) => eprintln!("Failed to fetch price: {}", e),
+}
+```
+
+### Historical Data with HAR Files
+
+```rust
+use brk_fetcher::Fetcher;
+use std::path::Path;
+
+// Initialize with HAR file path for extended historical coverage
+let har_path = Path::new("./import_data");
+let mut fetcher = Fetcher::import(true, Some(har_path))?;
+
+// Fetch minute-level data using HAR file fallback
+let height = Height::new(650000);
+let timestamp = Timestamp::from(1598918400u32); // August 2020
+let price_data = fetcher.get_height(height, timestamp, None)?;
+```
+
+### Batch Processing with Caching
+
+```rust
+use brk_fetcher::Fetcher;
+
+let mut fetcher = Fetcher::import(true, None)?;
+
+// Process multiple heights - caching improves performance
+for height in 800000..800100 {
+    let timestamp = Timestamp::from(1684771200u32 + (height - 800000) * 600);
+
+    match fetcher.get_height(Height::new(height), timestamp, None) {
+        Ok(ohlc) => println!("Height {}: ${:.2}", height, ohlc.close.to_dollars()),
+        Err(e) => eprintln!("Error at height {}: {}", height, e),
     }
 }
 
-// Clear cache to force fresh data
+// Clear caches when done
 fetcher.clear();
 ```
 
-## Data Sources and Limitations
+## Architecture
 
-### Kraken API
-- **1-day data**: Historical daily OHLC data
-- **1-minute data**: Limited to last ~10 hours
-- **Rate limits**: Subject to Kraken API restrictions
+### Retry Mechanism
 
-### Binance API  
-- **1-day data**: Historical daily OHLC data
-- **1-minute data**: Limited to last ~16 hours
-- **HAR import**: Can extend historical coverage via browser data
+The crate implements a sophisticated retry system with:
 
-### BRK Instance
-- **Cached data**: Previously fetched price data
-- **Offline capability**: Works without internet when data is cached
-- **Height-based**: Optimized for block height queries
+- **Default retry count**: 6 attempts with 5-second delays
+- **Extended retry**: Up to 720 attempts (12 hours) for critical operations
+- **Cache invalidation**: Automatic cache clearing between retry attempts
+- **Exponential backoff**: 60-second delays for extended retries
 
-## HAR File Import
+### Data Aggregation
 
-For historical data beyond API limits:
+Price data is aggregated using OHLC (Open, High, Low, Close) calculations spanning timestamp ranges. The `find_height_ohlc` function computes accurate OHLC values by scanning time series data between block timestamps.
 
-1. Visit [Binance BTC/USDT chart](https://www.binance.com/en/trade/BTC_USDT?type=spot)
-2. Set chart to 1-minute interval
-3. Open browser dev tools, go to Network tab
-4. Filter by 'uiKlines'
-5. Scroll chart to desired historical period
-6. Export network requests as HAR file
-7. Initialize fetcher with HAR path
+### HAR File Processing
 
-## Fallback Strategy
+Binance integration supports HTTP Archive (HAR) files for extended historical data coverage, parsing browser network captures to extract additional pricing data beyond API limitations.
 
-The fetcher tries sources in this order:
-1. **Kraken** - Primary source for most queries
-2. **Binance** - Secondary source with extended coverage
-3. **BRK** - Fallback for cached/computed prices
+## Code Analysis Summary
 
-If all sources fail, it retries up to 12 hours with 60-second intervals.
-
-## Performance and Reliability
-
-- **Automatic retries**: Up to 720 attempts (12 hours) with 60-second delays
-- **Cache management**: Clears cache on failures to force fresh data
-- **Error logging**: Detailed failure reporting with recovery instructions
-- **Graceful degradation**: Falls back through sources until successful
-
-## Dependencies
-
-- `brk_structs` - Bitcoin-aware type system (Date, Height, OHLC types)
-- `brk_error` - Unified error handling
-- `minreq` - HTTP client for API requests
-- `serde_json` - JSON parsing for API responses
-- `log` - Logging for retry and error reporting
+**Main Types**: `Fetcher` aggregator with `Binance`, `Kraken`, and `BRK` source implementations \
+**Caching**: BTreeMap-based caching for both timestamp and date-indexed price data \
+**Network Layer**: Built on `minreq` HTTP client with automatic JSON parsing \
+**Error Handling**: Comprehensive retry logic with source rotation and cache management \
+**Dependencies**: Integrates `brk_structs` for type definitions and `brk_error` for unified error handling \
+**Architecture**: Multi-source aggregation pattern with hierarchical fallback and intelligent caching
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_fetcher/src/brk.rs

@@ -14,7 +14,7 @@ pub struct BRK {
     dateindex_to_ohlc: BTreeMap<DateIndex, Vec<OHLCCents>>,
 }
 
-const API_URL: &str = "https://bitcoinresearchkit.org/api/vecs";
+const API_URL: &str = "https://bitview.space/api/vecs";
 const CHUNK_SIZE: usize = 10_000;
 
 impl BRK {
@@ -46,7 +46,7 @@ impl BRK {
 
         default_retry(|_| {
             let url = format!(
-                "{API_URL}/height-to-ohlc?from={}&to={}",
+                "{API_URL}/height-to-price-ohlc?from={}&to={}",
                 height,
                 height + CHUNK_SIZE
             );
@@ -91,7 +91,7 @@ impl BRK {
 
         default_retry(|_| {
             let url = format!(
-                "{API_URL}/dateindex-to-ohlc?from={}&to={}",
+                "{API_URL}/dateindex-to-price-ohlc?from={}&to={}",
                 dateindex,
                 dateindex + CHUNK_SIZE
             );

crates/brk_indexer/README.md

@@ -1,190 +1,277 @@
 # brk_indexer
 
-**High-performance Bitcoin blockchain indexer with dual storage architecture**
+High-performance Bitcoin blockchain indexer with parallel processing and dual storage architecture.
 
-`brk_indexer` processes raw Bitcoin Core block data and creates efficient storage structures using both vectors (time-series) and key-value stores (lookups). It serves as the foundation of BRK's data pipeline, organizing all blockchain data into optimized formats for fast retrieval and analysis.
+[![Crates.io](https://img.shields.io/crates/v/brk_indexer.svg)](https://crates.io/crates/brk_indexer)
+[![Documentation](https://docs.rs/brk_indexer/badge.svg)](https://docs.rs/brk_indexer)
 
-## What it provides
+## Overview
 
-- **Dual Storage Architecture**: Vectors for time-series data, key-value stores for lookups
-- **Memory Efficiency**: ~5-6GB peak RAM usage during full blockchain indexing
-- **Incremental Processing**: Resume from last indexed height with rollback protection
-- **Data Integrity**: Collision detection and validation during indexing
-- **All Bitcoin Data Types**: Complete support for blocks, transactions, inputs, outputs, and addresses
+This crate provides a comprehensive Bitcoin blockchain indexer built on top of `brk_parser`. It processes raw Bitcoin blocks in parallel, extracting and indexing transactions, addresses, inputs, outputs, and metadata into optimized storage structures. The indexer maintains two complementary storage systems: columnar vectors for analytics and key-value stores for fast lookups.
 
-## Key Features
+**Key Features:**
 
-### Storage Strategy
+- Parallel block processing with multi-threaded transaction analysis
+- Dual storage architecture: columnar vectors + key-value stores
+- Address type classification and indexing for all Bitcoin script types
+- Collision detection and validation for address hashes and transaction IDs
+- Incremental processing with automatic rollback and recovery
+- Height-based synchronization with Bitcoin Core RPC validation
+- Optimized batch operations with configurable snapshot intervals
 
-**Vector Storage (time-series data):**
-- Block metadata (height, timestamp, hash, difficulty, size)
-- Transaction data (version, locktime, RBF flag, indices)
-- Input/Output mappings and values
-- Address bytes for all output types
-- Efficient for range queries and analytics
+**Target Use Cases:**
 
-**Key-Value Storage (lookups):**
-- Block hash prefixes → heights
-- Transaction ID prefixes → transaction indices  
-- Address byte hashes → type indices
-- Fast point queries by hash or address
+- Bitcoin blockchain analysis requiring full transaction history
+- Address clustering and UTXO set analysis
+- Blockchain explorers needing fast address/transaction lookups
+- Research applications requiring structured access to blockchain data
 
-### Performance Features
-- **Parallel Processing**: Concurrent transaction and output processing using Rayon
-- **Batch Operations**: Periodic commits every 1,000 blocks for optimal I/O
-- **Memory Efficiency**: Optimized data structures minimize RAM usage
-- **Incremental Updates**: Handles blockchain reorganizations automatically
+## Installation
 
-### Address Type Support
-Complete support for all Bitcoin address types:
-- P2PK (65-byte and 33-byte), P2PKH, P2SH
-- P2WPKH, P2WSH, P2TR, P2A
-- P2MS (multisig), OpReturn, Empty, Unknown
+```bash
+cargo add brk_indexer
+```
 
-## Usage
-
-### Basic Indexing
+## Quick Start
 
 ```rust
 use brk_indexer::Indexer;
 use brk_parser::Parser;
-use bitcoincore_rpc::{Auth, Client};
+use bitcoincore_rpc::{Client, Auth};
 use vecdb::Exit;
+use std::path::Path;
 
-// Setup Bitcoin Core RPC connection
-let rpc = Box::leak(Box::new(Client::new(
-    "http://localhost:8332",
-    Auth::CookieFile(Path::new("~/.bitcoin/.cookie")),
-)?));
+// Initialize Bitcoin Core RPC client
+let rpc = Client::new("http://localhost:8332", Auth::None)?;
+let rpc = Box::leak(Box::new(rpc));
 
-// Create parser for Bitcoin Core block files
-let parser = Parser::new(
-    Path::new("~/.bitcoin/blocks").to_path_buf(),
-    Path::new("./brk_data").to_path_buf(),
-    rpc
-);
+// Create parser for raw block data
+let blocks_dir = Path::new("/path/to/bitcoin/blocks");
+let parser = Parser::new(blocks_dir, None, rpc);
 
-// Create indexer with forced import (resets if needed)
-let mut indexer = Indexer::forced_import(Path::new("./brk_data"))?;
+// Initialize indexer with output directory
+let outputs_dir = Path::new("./indexed_data");
+let mut indexer = Indexer::forced_import(outputs_dir)?;
 
-// Setup graceful shutdown handler
-let exit = Exit::new();
-exit.set_ctrlc_handler();
+// Index blockchain data
+let exit = Exit::default();
+let starting_indexes = indexer.index(&parser, rpc, &exit, true)?;
 
-// Index the blockchain
-let indexes = indexer.index(&parser, rpc, &exit, true)?;
-println!("Indexed up to height: {}", indexes.height);
+println!("Indexed up to height: {}", starting_indexes.height);
 ```
 
-### Continuous Indexing
+## API Overview
+
+### Core Types
+
+- **`Indexer`**: Main coordinator managing vectors and stores
+- **`Vecs`**: Columnar storage for blockchain data analytics
+- **`Stores`**: Key-value storage for fast hash-based lookups
+- **`Indexes`**: Current indexing state tracking progress across data types
+
+### Key Methods
+
+**`Indexer::forced_import(outputs_dir: &Path) -> Result<Self>`**
+Creates or opens indexer instance with automatic version management.
+
+**`index(&mut self, parser: &Parser, rpc: &'static Client, exit: &Exit, check_collisions: bool) -> Result<Indexes>`**
+Main indexing function processing blocks from parser with collision detection.
+
+### Storage Architecture
+
+**Columnar Vectors (Vecs):**
+
+- `height_to_*`: Block-level data (hash, timestamp, difficulty, size, weight)
+- `txindex_to_*`: Transaction data (ID, version, locktime, size, RBF flag)
+- `outputindex_to_*`: Output data (value, type, address mapping)
+- `inputindex_to_outputindex`: Input-to-output relationship mapping
+
+**Key-Value Stores:**
+
+- `addressbyteshash_to_typeindex`: Address hash to internal index mapping
+- `blockhashprefix_to_height`: Block hash prefix to height lookup
+- `txidprefix_to_txindex`: Transaction ID prefix to internal index
+- `addresstype_to_typeindex_with_outputindex`: Address type to output mappings
+
+### Address Type Support
+
+Complete coverage of Bitcoin script types:
+
+- **P2PK**: Pay-to-Public-Key (33-byte and 65-byte variants)
+- **P2PKH**: Pay-to-Public-Key-Hash
+- **P2SH**: Pay-to-Script-Hash
+- **P2WPKH**: Pay-to-Witness-Public-Key-Hash
+- **P2WSH**: Pay-to-Witness-Script-Hash
+- **P2TR**: Pay-to-Taproot
+- **P2MS**: Pay-to-Multisig
+- **P2A**: Pay-to-Address (custom type)
+- **OpReturn**: OP_RETURN data outputs
+- **Empty/Unknown**: Non-standard script types
+
+## Examples
+
+### Basic Indexing Operation
 
 ```rust
-use std::time::{Duration, Instant};
-use std::thread::sleep;
+use brk_indexer::Indexer;
+use brk_parser::Parser;
+use std::path::Path;
 
-// Continuous indexing loop for real-time updates
-loop {
-    let start_time = Instant::now();
-    
-    // Index new blocks
-    let indexes = indexer.index(&parser, rpc, &exit, true)?;
-    
-    println!("Indexed to height {} in {:?}", 
-             indexes.height, start_time.elapsed());
-    
-    // Check for exit signal
-    if exit.is_signaled() {
-        println!("Graceful shutdown requested");
-        break;
-    }
-    
-    // Wait before next update cycle
-    sleep(Duration::from_secs(5 * 60));
+// Initialize components
+let outputs_dir = Path::new("./blockchain_index");
+let mut indexer = Indexer::forced_import(outputs_dir)?;
+
+let blocks_dir = Path::new("/Users/satoshi/.bitcoin/blocks");
+let parser = Parser::new(blocks_dir, None, rpc);
+
+// Index with collision checking enabled
+let exit = vecdb::Exit::default();
+let final_indexes = indexer.index(&parser, rpc, &exit, true)?;
+
+println!("Final height: {}", final_indexes.height);
+println!("Total transactions: {}", final_indexes.txindex);
+println!("Total addresses: {}", final_indexes.total_address_count());
+```
+
+### Querying Indexed Data
+
+```rust
+use brk_indexer::Indexer;
+use brk_structs::{Height, TxidPrefix, AddressBytesHash};
+
+let indexer = Indexer::forced_import("./blockchain_index")?;
+
+// Look up block hash by height
+let height = Height::new(750000);
+if let Some(block_hash) = indexer.vecs.height_to_blockhash.get(height)? {
+    println!("Block 750000 hash: {}", block_hash);
+}
+
+// Look up transaction by ID prefix
+let txid_prefix = TxidPrefix::from_str("abcdef123456")?;
+if let Some(tx_index) = indexer.stores.txidprefix_to_txindex.get(&txid_prefix)? {
+    println!("Transaction index: {}", tx_index);
+}
+
+// Query address information
+let address_hash = AddressBytesHash::from(/* address bytes */);
+if let Some(type_index) = indexer.stores.addressbyteshash_to_typeindex.get(&address_hash)? {
+    println!("Address type index: {}", type_index);
 }
 ```
 
-### Accessing Indexed Data
+### Incremental Processing
 
 ```rust
-// Access the underlying storage structures
-let vecs = &indexer.vecs;
-let stores = &indexer.stores;
+use brk_indexer::Indexer;
 
-// Get block hash at specific height
-let block_hash = vecs.height_to_blockhash.get(Height::new(800_000))?;
+// Indexer automatically resumes from last processed height
+let mut indexer = Indexer::forced_import("./blockchain_index")?;
 
-// Look up transaction by prefix
-let tx_prefix = TxidPrefix::from(&txid);
-let tx_index = stores.txidprefix_to_txindex.get(&tx_prefix)?;
+let current_indexes = indexer.vecs.current_indexes(&indexer.stores, rpc)?;
+println!("Resuming from height: {}", current_indexes.height);
 
-// Get address data
-let address_hash = AddressBytesHash::from(&address_bytes);
-let type_index = stores.addressbyteshash_to_anyaddressindex.get(&address_hash)?;
+// Process new blocks incrementally
+let exit = vecdb::Exit::default();
+let updated_indexes = indexer.index(&parser, rpc, &exit, true)?;
+
+println!("Processed {} new blocks",
+         updated_indexes.height.as_u32() - current_indexes.height.as_u32());
 ```
 
+### Address Type Analysis
+
+```rust
+use brk_indexer::Indexer;
+use brk_structs::OutputType;
+
+let indexer = Indexer::forced_import("./blockchain_index")?;
+
+// Analyze address distribution by type
+for output_type in OutputType::as_vec() {
+    let count = indexer.vecs.outputindex_to_outputtype
+        .iter()
+        .filter(|&ot| ot == output_type)
+        .count();
+
+    println!("{:?}: {} outputs", output_type, count);
+}
+
+// Query specific address type data
+let p2pkh_store = &indexer.stores.addresstype_to_typeindex_with_outputindex
+    .p2pkh;
+
+println!("P2PKH addresses: {}", p2pkh_store.len());
+```
+
+## Architecture
+
+### Parallel Processing
+
+The indexer uses sophisticated parallel processing:
+
+- **Block-Level Parallelism**: Concurrent processing of transactions within blocks
+- **Transaction Analysis**: Parallel input/output processing with `rayon`
+- **Address Resolution**: Multi-threaded address type classification and indexing
+- **Collision Detection**: Parallel validation of hash collisions across address types
+
+### Storage Optimization
+
+**Columnar Storage (vecdb):**
+
+- Compressed vectors for space-efficient analytics queries
+- Raw vectors for frequently accessed data (heights, hashes)
+- Page-aligned storage for memory mapping efficiency
+
+**Key-Value Storage (Fjall):**
+
+- LSM-tree architecture for write-heavy indexing workloads
+- Bloom filters for fast negative lookups
+- Transactional consistency with rollback support
+
+### Memory Management
+
+- **Batch Processing**: 1000-block snapshots to balance memory and I/O
+- **Reader Management**: Static readers for consistent data access during processing
+- **Collision Tracking**: BTreeMap-based collision detection with memory cleanup
+- **Exit Handling**: Graceful shutdown with consistent state preservation
+
+### Version Management
+
+- **Schema Versioning**: Automatic migration on version changes (currently v21)
+- **Rollback Support**: Automatic recovery from incomplete processing
+- **State Tracking**: Height-based synchronization across all storage components
+
 ## Performance Characteristics
 
-**Benchmarked on MacBook Pro M3 Pro (36GB RAM):**
-- **Full blockchain sync** (to ~892k blocks): 7-8 hours
-- **Peak memory usage**: 5-6GB
-- **Storage overhead**: ~27% of Bitcoin Core block size
-- **Incremental updates**: Very fast, efficient resume from last height
+### Processing Speed
 
-## Data Organization
+- **Parallel Transaction Processing**: Multi-core utilization for CPU-intensive operations
+- **Optimized I/O**: Batch operations reduce disk overhead
+- **Memory Efficiency**: Streaming processing without loading entire blockchain
 
-The indexer creates this storage structure:
-```
-brk_data/
-├── indexed/
-│   ├── vecs/              # Vector storage
-│   │   ├── height_to_*    # Height-indexed data
-│   │   ├── txindex_to_*   # Transaction-indexed data
-│   │   └── outputindex_to_* # Output-indexed data
-│   └── stores/            # Key-value stores
-│       ├── hash_lookups/  # Block/TX hash mappings
-│       └── address_maps/  # Address type mappings
-└── metadata/              # Versioning and state
-```
+### Storage Requirements
 
-## Indexes Tracking
+- **Columnar Compression**: Significant space savings for repetitive blockchain data
+- **Index Optimization**: Bloom filters reduce lookup overhead
+- **Incremental Growth**: Storage scales linearly with blockchain size
 
-The indexer maintains current indices during processing:
+### Scalability
 
-```rust
-pub struct Indexes {
-    pub height: Height,                      // Current block height
-    pub txindex: TxIndex,                    // Current transaction index
-    pub inputindex: InputIndex,              // Current input index
-    pub outputindex: OutputIndex,            // Current output index
-    pub p2pkhaddressindex: P2PKHAddressIndex, // P2PKH address index
-    // ... indices for all address types
-}
-```
+- **Height-Based Partitioning**: Enables distributed processing strategies
+- **Modular Architecture**: Separate vector and store systems for flexible deployment
+- **Resource Configuration**: Configurable batch sizes and memory limits
 
-## Requirements
+## Code Analysis Summary
 
-- **Bitcoin Core node** with RPC enabled
-- **Block file access** to `~/.bitcoin/blocks/`
-- **Storage space**: Minimum 500GB (scales with blockchain growth)
-- **Memory**: 8GB+ RAM recommended
-- **CPU**: Multi-core recommended for parallel processing
-
-## Rollback and Recovery
-
-- **Automatic rollback** on interruption or blockchain reorgs
-- **State persistence** for efficient restart
-- **Version management** for storage format compatibility
-- **Graceful shutdown** with Ctrl+C handling
-
-## Dependencies
-
-- `brk_parser` - Bitcoin block parsing and sequential access
-- `brk_store` - Key-value storage wrapper (fjall-based)
-- `vecdb` - Vector database for time-series storage
-- `bitcoin` - Bitcoin protocol types and parsing
-- `rayon` - Parallel processing framework
-- `bitcoincore_rpc` - Bitcoin Core RPC client
+**Main Structure**: `Indexer` coordinating `Vecs` (columnar analytics) and `Stores` (key-value lookups) \
+**Processing Pipeline**: Multi-threaded block analysis with parallel transaction/address processing \
+**Storage Architecture**: Dual system using vecdb for analytics and Fjall for lookups \
+**Address Indexing**: Complete Bitcoin script type coverage with collision detection \
+**Synchronization**: Height-based coordination with Bitcoin Core RPC validation \
+**Parallel Processing**: rayon-based parallelism for transaction analysis and address resolution \
+**Architecture**: High-performance blockchain indexer with ACID guarantees and incremental processing
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_indexer/examples/indexer.rs

@@ -21,8 +21,8 @@ fn main() -> Result<()> {
 
     let blocks_dir = bitcoin_dir.join("blocks");
 
-    let outputs_dir = Path::new("../../_outputs");
-    fs::create_dir_all(outputs_dir)?;
+    let outputs_dir = Path::new(&std::env::var("HOME").unwrap()).join(".brk");
+    fs::create_dir_all(&outputs_dir)?;
     // let outputs_dir = Path::new("/Volumes/WD_BLACK1/brk");
 
     let rpc = Box::leak(Box::new(bitcoincore_rpc::Client::new(
@@ -33,11 +33,11 @@ fn main() -> Result<()> {
     let exit = Exit::new();
     exit.set_ctrlc_handler();
 
-    let parser = Parser::new(blocks_dir, outputs_dir.to_path_buf(), rpc);
+    let parser = Parser::new(blocks_dir, rpc);
 
-    fs::create_dir_all(outputs_dir)?;
+    fs::create_dir_all(&outputs_dir)?;
 
-    let mut indexer = Indexer::forced_import(outputs_dir)?;
+    let mut indexer = Indexer::forced_import(&outputs_dir)?;
 
     loop {
         let i = Instant::now();

crates/brk_indexer/src/lib.rs

@@ -8,13 +8,13 @@ use brk_error::{Error, Result};
 use brk_parser::Parser;
 use brk_store::AnyStore;
 use brk_structs::{
-    AddressBytes, AddressBytesHash, BlockHash, BlockHashPrefix, Height, InputIndex, OutputIndex,
-    OutputType, Sats, StoredBool, Timestamp, TxIndex, Txid, TxidPrefix, TypeIndex,
-    TypeIndexWithOutputindex, Unit, Version, Vin, Vout,
+    AddressBytes, AddressBytesHash, BlockHashPrefix, Height, InputIndex, OutputIndex, OutputType,
+    Sats, StoredBool, Timestamp, TxIndex, Txid, TxidPrefix, TypeIndex, TypeIndexWithOutputindex,
+    Unit, Version, Vin, Vout,
 };
 use log::{error, info};
 use rayon::prelude::*;
-use vecdb::{AnyVec, Database, Exit, GenericStoredVec, PAGE_SIZE, Reader, VecIterator};
+use vecdb::{AnyVec, Exit, GenericStoredVec, Reader, VecIterator};
 mod indexes;
 mod stores;
 mod vecs;
@@ -23,13 +23,15 @@ pub use indexes::*;
 pub use stores::*;
 pub use vecs::*;
 
+// One version for all data sources
+// Increment on change OR addition
+const VERSION: Version = Version::new(21);
+
 const SNAPSHOT_BLOCK_RANGE: usize = 1_000;
 const COLLISIONS_CHECKED_UP_TO: Height = Height::new(909_150);
-const VERSION: Version = Version::ONE;
 
 #[derive(Clone)]
 pub struct Indexer {
-    pub db: Database,
     pub vecs: Vecs,
     pub stores: Stores,
 }
@@ -38,18 +40,15 @@ impl Indexer {
     pub fn forced_import(outputs_dir: &Path) -> Result<Self> {
         info!("Importing indexer...");
 
-        let db = Database::open(&outputs_dir.join("indexed/vecs"))?;
-        db.set_min_len(PAGE_SIZE * 50_000_000)?;
-        info!("Opened database");
+        let path = outputs_dir.join("indexed");
 
-        let vecs = Vecs::forced_import(&db, VERSION + Version::ZERO)?;
+        let vecs = Vecs::forced_import(&path, VERSION)?;
         info!("Imported vecs");
 
-        let stores =
-            Stores::forced_import(&outputs_dir.join("indexed/stores"), VERSION + Version::ZERO)?;
+        let stores = Stores::forced_import(&path, VERSION)?;
         info!("Imported stores");
 
-        Ok(Self { vecs, stores, db })
+        Ok(Self { vecs, stores })
     }
 
     pub fn index(
@@ -59,16 +58,9 @@ impl Indexer {
         exit: &Exit,
         check_collisions: bool,
     ) -> Result<Indexes> {
-        let db = self.db.clone();
-
-        // dbg!(self.db.regions().id_to_index());
-        // dbg!(self.db.layout());
-
         let starting_indexes = Indexes::try_from((&mut self.vecs, &self.stores, rpc))
             .unwrap_or_else(|_report| Indexes::default());
 
-        // dbg!(&starting_indexes);
-
         let lock = exit.lock();
         self.stores
             .rollback_if_needed(&mut self.vecs, &starting_indexes)?;
@@ -108,7 +100,6 @@ impl Indexer {
                 vecs.flush(height)?;
                 info!("Flushed vecs in {}s", i.elapsed().as_secs());
                 let i = Instant::now();
-                db.flush()?;
                 info!("Flushed db in {}s", i.elapsed().as_secs());
                 Ok(())
             };
@@ -174,7 +165,10 @@ impl Indexer {
         );
 
         parser.parse(start, end).iter().try_for_each(
-            |(height, block, blockhash)| -> Result<()> {
+            |block| -> Result<()> {
+                let height = block.height();
+                let blockhash = block.hash();
+
                 info!("Indexing block {height}...");
 
                 idxs.height = height;
@@ -192,8 +186,7 @@ impl Indexer {
                 // Used to check rapidhash collisions
                 let check_collisions = check_collisions && height > COLLISIONS_CHECKED_UP_TO ;
 
-                let blockhash = BlockHash::from(blockhash);
-                let blockhash_prefix = BlockHashPrefix::from(&blockhash);
+                let blockhash_prefix = BlockHashPrefix::from(blockhash);
 
                 if stores
                     .blockhashprefix_to_height
@@ -210,7 +203,11 @@ impl Indexer {
                     .blockhashprefix_to_height
                     .insert_if_needed(blockhash_prefix, height, height);
 
-                vecs.height_to_blockhash.push_if_needed(height, blockhash)?;
+                stores
+                    .height_to_coinbase_tag
+                    .insert_if_needed( height, block.coinbase_tag().into(), height);
+
+                vecs.height_to_blockhash.push_if_needed(height, blockhash.clone())?;
                 vecs.height_to_difficulty
                     .push_if_needed(height, block.header.difficulty_float().into())?;
                 vecs.height_to_timestamp
@@ -797,7 +794,7 @@ impl Indexer {
         }
 
         let i = Instant::now();
-        db.punch_holes()?;
+        self.vecs.punch_holes()?;
         info!("Punched holes in db in {}s", i.elapsed().as_secs());
 
         Ok(starting_indexes)

crates/brk_indexer/src/stores.rs

@@ -4,11 +4,12 @@ use brk_error::Result;
 use brk_store::{AnyStore, Store};
 use brk_structs::{
     AddressBytes, AddressBytesHash, BlockHashPrefix, ByAddressType, Height, OutputIndex,
-    OutputType, TxIndex, TxidPrefix, TypeIndex, TypeIndexWithOutputindex, Unit, Version,
+    OutputType, StoredString, TxIndex, TxidPrefix, TypeIndex, TypeIndexWithOutputindex, Unit,
+    Version,
 };
 use fjall::{PersistMode, TransactionalKeyspace};
 use rayon::prelude::*;
-use vecdb::VecIterator;
+use vecdb::{AnyVec, StoredIndex, VecIterator};
 
 use crate::Indexes;
 
@@ -20,22 +21,23 @@ pub struct Stores {
 
     pub addressbyteshash_to_typeindex: Store<AddressBytesHash, TypeIndex>,
     pub blockhashprefix_to_height: Store<BlockHashPrefix, Height>,
+    pub height_to_coinbase_tag: Store<Height, StoredString>,
     pub txidprefix_to_txindex: Store<TxidPrefix, TxIndex>,
     pub addresstype_to_typeindex_with_outputindex:
         ByAddressType<Store<TypeIndexWithOutputindex, Unit>>,
 }
 
-const VERSION: Version = Version::ZERO;
-
 impl Stores {
-    pub fn forced_import(path: &Path, version: Version) -> Result<Self> {
-        fs::create_dir_all(path)?;
+    pub fn forced_import(parent: &Path, version: Version) -> Result<Self> {
+        let path = parent.join("stores");
 
-        let keyspace = match brk_store::open_keyspace(path) {
+        fs::create_dir_all(&path)?;
+
+        let keyspace = match brk_store::open_keyspace(&path) {
             Ok(keyspace) => keyspace,
             Err(_) => {
-                fs::remove_dir_all(path)?;
-                return Self::forced_import(path, version);
+                fs::remove_dir_all(&path)?;
+                return Self::forced_import(&path, version);
             }
         };
 
@@ -43,106 +45,97 @@ impl Stores {
             let addressbyteshash_to_typeindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "addressbyteshash_to_typeindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     None,
                 )
             });
             let blockhashprefix_to_height = scope.spawn(|| {
-                Store::import(
-                    &keyspace,
-                    path,
-                    "blockhashprefix_to_height",
-                    version + VERSION + Version::ZERO,
-                    None,
-                )
-            });
-            let txidprefix_to_txindex = scope.spawn(|| {
-                Store::import(
-                    &keyspace,
-                    path,
-                    "txidprefix_to_txindex",
-                    version + VERSION + Version::ZERO,
-                    None,
-                )
+                Store::import(&keyspace, &path, "blockhashprefix_to_height", version, None)
             });
+            let txidprefix_to_txindex = scope
+                .spawn(|| Store::import(&keyspace, &path, "txidprefix_to_txindex", version, None));
             let p2aaddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2aaddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2pk33addressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2pk33addressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2pk65addressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2pk65addressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2pkhaddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2pkhaddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2shaddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2shaddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2traddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2traddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2wpkhaddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2wpkhaddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
             let p2wshaddressindex_with_outputindex = scope.spawn(|| {
                 Store::import(
                     &keyspace,
-                    path,
+                    &path,
                     "p2wshaddressindex_with_outputindex",
-                    version + VERSION + Version::ZERO,
+                    version,
                     Some(false),
                 )
             });
 
+            let height_to_coinbase_tag =
+                Store::import(&keyspace, &path, "height_to_coinbase_tag", version, None)?;
+
             Ok(Self {
                 keyspace: keyspace.clone(),
 
+                height_to_coinbase_tag,
                 addressbyteshash_to_typeindex: addressbyteshash_to_typeindex.join().unwrap()?,
                 blockhashprefix_to_height: blockhashprefix_to_height.join().unwrap()?,
                 txidprefix_to_txindex: txidprefix_to_txindex.join().unwrap()?,
@@ -182,11 +175,9 @@ impl Stores {
             .map_err(|e| e.into())
     }
 
-    fn as_slice(&self) -> [&(dyn AnyStore + Send + Sync); 11] {
+    fn as_slice(&self) -> [&(dyn AnyStore + Send + Sync); 12] {
         [
             &self.addressbyteshash_to_typeindex,
-            &self.blockhashprefix_to_height,
-            &self.txidprefix_to_txindex,
             &self.addresstype_to_typeindex_with_outputindex.p2a,
             &self.addresstype_to_typeindex_with_outputindex.p2pk33,
             &self.addresstype_to_typeindex_with_outputindex.p2pk65,
@@ -195,14 +186,15 @@ impl Stores {
             &self.addresstype_to_typeindex_with_outputindex.p2tr,
             &self.addresstype_to_typeindex_with_outputindex.p2wpkh,
             &self.addresstype_to_typeindex_with_outputindex.p2wsh,
+            &self.blockhashprefix_to_height,
+            &self.height_to_coinbase_tag,
+            &self.txidprefix_to_txindex,
         ]
     }
 
-    fn as_mut_slice(&mut self) -> [&mut (dyn AnyStore + Send + Sync); 11] {
+    fn as_mut_slice(&mut self) -> [&mut (dyn AnyStore + Send + Sync); 12] {
         [
             &mut self.addressbyteshash_to_typeindex,
-            &mut self.blockhashprefix_to_height,
-            &mut self.txidprefix_to_txindex,
             &mut self.addresstype_to_typeindex_with_outputindex.p2a,
             &mut self.addresstype_to_typeindex_with_outputindex.p2pk33,
             &mut self.addresstype_to_typeindex_with_outputindex.p2pk65,
@@ -211,6 +203,9 @@ impl Stores {
             &mut self.addresstype_to_typeindex_with_outputindex.p2tr,
             &mut self.addresstype_to_typeindex_with_outputindex.p2wpkh,
             &mut self.addresstype_to_typeindex_with_outputindex.p2wsh,
+            &mut self.blockhashprefix_to_height,
+            &mut self.height_to_coinbase_tag,
+            &mut self.txidprefix_to_txindex,
         ]
     }
 
@@ -222,6 +217,7 @@ impl Stores {
         if self.addressbyteshash_to_typeindex.is_empty()?
             && self.blockhashprefix_to_height.is_empty()?
             && self.txidprefix_to_txindex.is_empty()?
+            && self.height_to_coinbase_tag.is_empty()?
             && self
                 .addresstype_to_typeindex_with_outputindex
                 .p2a
@@ -266,6 +262,12 @@ impl Stores {
                     self.blockhashprefix_to_height.remove(blockhashprefix);
                 });
 
+            (starting_indexes.height.unwrap_to_usize()..vecs.height_to_blockhash.len())
+                .map(Height::from)
+                .for_each(|h| {
+                    self.height_to_coinbase_tag.remove(h);
+                });
+
             if let Some(mut index) = vecs
                 .height_to_first_p2pk65addressindex
                 .iter()

crates/brk_indexer/src/vecs.rs

@@ -1,3 +1,5 @@
+use std::path::Path;
+
 use brk_error::Result;
 use brk_structs::{
     AddressBytes, BlockHash, EmptyOutputIndex, Height, InputIndex, OpReturnIndex, OutputIndex,
@@ -9,15 +11,15 @@ use brk_structs::{
 };
 use rayon::prelude::*;
 use vecdb::{
-    AnyCollectableVec, AnyStoredVec, CompressedVec, Database, GenericStoredVec, RawVec, Stamp,
+    AnyCollectableVec, AnyStoredVec, CompressedVec, Database, GenericStoredVec, PAGE_SIZE, RawVec,
+    Stamp,
 };
 
 use crate::Indexes;
 
-const VERSION: Version = Version::ZERO;
-
 #[derive(Clone)]
 pub struct Vecs {
+    db: Database,
     pub emptyoutputindex_to_txindex: CompressedVec<EmptyOutputIndex, TxIndex>,
     pub height_to_blockhash: RawVec<Height, BlockHash>,
     pub height_to_difficulty: CompressedVec<Height, StoredF64>,
@@ -36,7 +38,7 @@ pub struct Vecs {
     pub height_to_first_p2wshaddressindex: CompressedVec<Height, P2WSHAddressIndex>,
     pub height_to_first_txindex: CompressedVec<Height, TxIndex>,
     pub height_to_first_unknownoutputindex: CompressedVec<Height, UnknownOutputIndex>,
-    /// Doesn't guarantee continuity due to possible reorgs
+    /// Doesn't guarantee continuity due to possible reorgs and more generally the nature of mining
     pub height_to_timestamp: CompressedVec<Height, Timestamp>,
     pub height_to_total_size: CompressedVec<Height, StoredU64>,
     pub height_to_weight: CompressedVec<Height, Weight>,
@@ -67,225 +69,134 @@ pub struct Vecs {
 }
 
 impl Vecs {
-    pub fn forced_import(db: &Database, version: Version) -> Result<Self> {
-        Ok(Self {
-            emptyoutputindex_to_txindex: CompressedVec::forced_import(
-                db,
-                "txindex",
-                version + VERSION + Version::ZERO,
-            )?,
-            height_to_blockhash: RawVec::forced_import(
-                db,
-                "blockhash",
-                version + VERSION + Version::ZERO,
-            )?,
-            height_to_difficulty: CompressedVec::forced_import(
-                db,
-                "difficulty",
-                version + VERSION + Version::ZERO,
-            )?,
+    pub fn forced_import(parent: &Path, version: Version) -> Result<Self> {
+        let db = Database::open(&parent.join("vecs"))?;
+        db.set_min_len(PAGE_SIZE * 50_000_000)?;
+
+        let db_positions = Database::open(&parent.join("vecs/positions"))?;
+        db_positions.set_min_len(PAGE_SIZE * 1_000_000)?;
+
+        let this = Self {
+            emptyoutputindex_to_txindex: CompressedVec::forced_import(&db, "txindex", version)?,
+            height_to_blockhash: RawVec::forced_import(&db, "blockhash", version)?,
+            height_to_difficulty: CompressedVec::forced_import(&db, "difficulty", version)?,
             height_to_first_emptyoutputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_emptyoutputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_inputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_inputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_opreturnindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_opreturnindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_outputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_outputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2aaddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2aaddressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2msoutputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2msoutputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2pk33addressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2pk33addressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2pk65addressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2pk65addressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2pkhaddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2pkhaddressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2shaddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2shaddressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2traddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2traddressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2wpkhaddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2wpkhaddressindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             height_to_first_p2wshaddressindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_p2wshaddressindex",
-                version + VERSION + Version::ZERO,
-            )?,
-            height_to_first_txindex: CompressedVec::forced_import(
-                db,
-                "first_txindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
+            height_to_first_txindex: CompressedVec::forced_import(&db, "first_txindex", version)?,
             height_to_first_unknownoutputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_unknownoutputindex",
-                version + VERSION + Version::ZERO,
-            )?,
-            height_to_timestamp: CompressedVec::forced_import(
-                db,
-                "timestamp",
-                version + VERSION + Version::ZERO,
-            )?,
-            height_to_total_size: CompressedVec::forced_import(
-                db,
-                "total_size",
-                version + VERSION + Version::ZERO,
-            )?,
-            height_to_weight: CompressedVec::forced_import(
-                db,
-                "weight",
-                version + VERSION + Version::ZERO,
-            )?,
-            inputindex_to_outputindex: RawVec::forced_import(
-                db,
-                "outputindex",
-                version + VERSION + Version::ZERO,
-            )?,
-            opreturnindex_to_txindex: CompressedVec::forced_import(
-                db,
-                "txindex",
-                version + VERSION + Version::ZERO,
-            )?,
-            outputindex_to_outputtype: RawVec::forced_import(
-                db,
-                "outputtype",
-                version + VERSION + Version::ZERO,
-            )?,
-            outputindex_to_typeindex: RawVec::forced_import(
-                db,
-                "typeindex",
-                version + VERSION + Version::ZERO,
-            )?,
-            outputindex_to_value: RawVec::forced_import(
-                db,
-                "value",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2aaddressindex_to_p2abytes: RawVec::forced_import(
-                db,
-                "p2abytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2msoutputindex_to_txindex: CompressedVec::forced_import(
-                db,
-                "txindex",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2pk33addressindex_to_p2pk33bytes: RawVec::forced_import(
-                db,
-                "p2pk33bytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2pk65addressindex_to_p2pk65bytes: RawVec::forced_import(
-                db,
-                "p2pk65bytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2pkhaddressindex_to_p2pkhbytes: RawVec::forced_import(
-                db,
-                "p2pkhbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2shaddressindex_to_p2shbytes: RawVec::forced_import(
-                db,
-                "p2shbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2traddressindex_to_p2trbytes: RawVec::forced_import(
-                db,
-                "p2trbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2wpkhaddressindex_to_p2wpkhbytes: RawVec::forced_import(
-                db,
-                "p2wpkhbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            p2wshaddressindex_to_p2wshbytes: RawVec::forced_import(
-                db,
-                "p2wshbytes",
-                version + VERSION + Version::ZERO,
-            )?,
-            txindex_to_base_size: CompressedVec::forced_import(
-                db,
-                "base_size",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
+            height_to_timestamp: CompressedVec::forced_import(&db, "timestamp", version)?,
+            height_to_total_size: CompressedVec::forced_import(&db, "total_size", version)?,
+            height_to_weight: CompressedVec::forced_import(&db, "weight", version)?,
+            inputindex_to_outputindex: RawVec::forced_import(&db, "outputindex", version)?,
+            opreturnindex_to_txindex: CompressedVec::forced_import(&db, "txindex", version)?,
+            outputindex_to_outputtype: RawVec::forced_import(&db, "outputtype", version)?,
+            outputindex_to_typeindex: RawVec::forced_import(&db, "typeindex", version)?,
+            outputindex_to_value: RawVec::forced_import(&db, "value", version)?,
+            p2aaddressindex_to_p2abytes: RawVec::forced_import(&db, "p2abytes", version)?,
+            p2msoutputindex_to_txindex: CompressedVec::forced_import(&db, "txindex", version)?,
+            p2pk33addressindex_to_p2pk33bytes: RawVec::forced_import(&db, "p2pk33bytes", version)?,
+            p2pk65addressindex_to_p2pk65bytes: RawVec::forced_import(&db, "p2pk65bytes", version)?,
+            p2pkhaddressindex_to_p2pkhbytes: RawVec::forced_import(&db, "p2pkhbytes", version)?,
+            p2shaddressindex_to_p2shbytes: RawVec::forced_import(&db, "p2shbytes", version)?,
+            p2traddressindex_to_p2trbytes: RawVec::forced_import(&db, "p2trbytes", version)?,
+            p2wpkhaddressindex_to_p2wpkhbytes: RawVec::forced_import(&db, "p2wpkhbytes", version)?,
+            p2wshaddressindex_to_p2wshbytes: RawVec::forced_import(&db, "p2wshbytes", version)?,
+            txindex_to_base_size: CompressedVec::forced_import(&db, "base_size", version)?,
             txindex_to_first_inputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_inputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             txindex_to_first_outputindex: CompressedVec::forced_import(
-                db,
+                &db,
                 "first_outputindex",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
             txindex_to_is_explicitly_rbf: CompressedVec::forced_import(
-                db,
+                &db,
                 "is_explicitly_rbf",
-                version + VERSION + Version::ZERO,
+                version,
             )?,
-            txindex_to_rawlocktime: CompressedVec::forced_import(
-                db,
-                "rawlocktime",
-                version + VERSION + Version::ZERO,
-            )?,
-            txindex_to_total_size: CompressedVec::forced_import(
-                db,
-                "total_size",
-                version + VERSION + Version::ZERO,
-            )?,
-            txindex_to_txid: RawVec::forced_import(db, "txid", version + VERSION + Version::ZERO)?,
-            txindex_to_txversion: CompressedVec::forced_import(
-                db,
-                "txversion",
-                version + VERSION + Version::ZERO,
-            )?,
-            unknownoutputindex_to_txindex: CompressedVec::forced_import(
-                db,
-                "txindex",
-                version + VERSION + Version::ZERO,
-            )?,
-        })
+            txindex_to_rawlocktime: CompressedVec::forced_import(&db, "rawlocktime", version)?,
+            txindex_to_total_size: CompressedVec::forced_import(&db, "total_size", version)?,
+            txindex_to_txid: RawVec::forced_import(&db, "txid", version)?,
+            txindex_to_txversion: CompressedVec::forced_import(&db, "txversion", version)?,
+            unknownoutputindex_to_txindex: CompressedVec::forced_import(&db, "txindex", version)?,
+
+            db,
+        };
+
+        this.db
+            .retain_regions(this.iter().flat_map(|v| v.region_names()).collect())?;
+
+        Ok(this)
     }
 
     pub fn rollback_if_needed(&mut self, starting_indexes: &Indexes) -> Result<()> {
@@ -435,15 +346,15 @@ impl Vecs {
     }
 
     pub fn flush(&mut self, height: Height) -> Result<()> {
-        self.mut_vecs()
-            .into_par_iter()
+        self.iter_mut()
+            .par_bridge()
             .try_for_each(|vec| vec.stamped_flush(Stamp::from(height)))?;
+        self.db.flush()?;
         Ok(())
     }
 
     pub fn starting_height(&mut self) -> Height {
-        self.mut_vecs()
-            .into_iter()
+        self.iter_mut()
             .map(|vec| {
                 let h = Height::from(vec.stamp());
                 if h > Height::ZERO { h.incremented() } else { h }
@@ -452,9 +363,14 @@ impl Vecs {
             .unwrap()
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        vec![
-            &self.emptyoutputindex_to_txindex,
+    pub fn punch_holes(&self) -> Result<()> {
+        self.db.punch_holes()?;
+        Ok(())
+    }
+
+    pub fn iter(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [
+            &self.emptyoutputindex_to_txindex as &dyn AnyCollectableVec,
             &self.height_to_blockhash,
             &self.height_to_difficulty,
             &self.height_to_first_emptyoutputindex,
@@ -499,11 +415,12 @@ impl Vecs {
             &self.txindex_to_txversion,
             &self.unknownoutputindex_to_txindex,
         ]
+        .into_iter()
     }
 
-    fn mut_vecs(&mut self) -> Vec<&mut dyn AnyStoredVec> {
-        vec![
-            &mut self.emptyoutputindex_to_txindex,
+    fn iter_mut(&mut self) -> impl Iterator<Item = &mut dyn AnyStoredVec> {
+        [
+            &mut self.emptyoutputindex_to_txindex as &mut dyn AnyStoredVec,
             &mut self.height_to_blockhash,
             &mut self.height_to_difficulty,
             &mut self.height_to_first_emptyoutputindex,
@@ -548,5 +465,6 @@ impl Vecs {
             &mut self.txindex_to_txversion,
             &mut self.unknownoutputindex_to_txindex,
         ]
+        .into_iter()
     }
 }

crates/brk_interface/Cargo.toml

@@ -10,14 +10,17 @@ rust-version.workspace = true
 build = "build.rs"
 
 [dependencies]
+bitcoincore-rpc = { workspace = true }
 brk_computer = { workspace = true }
 brk_error = { workspace = true }
 brk_indexer = { workspace = true }
+brk_parser = { workspace = true }
 brk_structs = { workspace = true }
 vecdb = { workspace = true }
 derive_deref = { workspace = true }
+quick_cache = { workspace = true }
 schemars = "1.0.4"
 serde = { workspace = true }
 serde_json = { workspace = true }
-serde_with = "3.14.0"
-tabled = "0.20.0"
+serde_with = "3.14.1"
+nucleo-matcher = "0.3.1"

crates/brk_interface/README.md

@@ -1,227 +1,237 @@
 # brk_interface
 
-**Unified data query and formatting interface for Bitcoin datasets**
+Unified data query and formatting interface for Bitcoin datasets with intelligent search and multi-format output.
 
-`brk_interface` provides a clean, unified API for accessing Bitcoin datasets from both indexer and computer components. It serves as the primary data access layer powering BRK's web API and MCP endpoints, offering flexible querying, pagination, and multiple output formats.
+[![Crates.io](https://img.shields.io/crates/v/brk_interface.svg)](https://crates.io/crates/brk_interface)
+[![Documentation](https://docs.rs/brk_interface/badge.svg)](https://docs.rs/brk_interface)
 
-## What it provides
+## Overview
 
-- **Unified Data Access**: Single interface to query both indexed blockchain data and computed analytics
-- **Multiple Output Formats**: JSON, CSV, TSV, and Markdown table formatting
-- **Flexible Pagination**: Range queries with positive/negative indexing and automatic pagination
-- **Multi-dataset Queries**: Retrieve multiple datasets with the same time index in one call
-- **Dynamic Search**: Intelligent ID matching with automatic fallbacks and normalization
+This crate provides a high-level interface for querying and formatting data from BRK's indexer and computer components. It offers intelligent vector search with fuzzy matching, parameter validation, range queries, and multi-format output (JSON, CSV) with efficient caching and pagination support.
 
-## Key Features
+**Key Features:**
 
-### Query Interface
-- **25 Time Indices**: From granular (Height, DateIndex) to aggregate (YearIndex, DecadeIndex)
-- **Bitcoin-Specific Indices**: Address types (P2PKH, P2SH, P2TR), output types, epochs
-- **Multi-dataset support**: Query multiple related datasets simultaneously
-- **Intelligent ID resolution**: Flexible matching with automatic fallbacks
+- Unified query interface across indexer and computer data sources
+- Intelligent search with fuzzy matching and helpful error messages
+- Multi-format output: JSON, CSV with proper formatting
+- Range-based data queries with flexible from/to parameters
+- Comprehensive pagination support for large datasets
+- Schema validation with JSON Schema generation for API documentation
+- Efficient caching system for error messages and repeated queries
 
-### Pagination and Ranges
-- **Signed indexing**: Negative indices count from end (`-1` = latest, `-10` = last 10)
-- **Range queries**: `from`/`to` parameters with optional `count` limit
-- **Efficient pagination**: 1,000 items per page with proper start/end calculation
+**Target Use Cases:**
 
-### Output Formatting
-- **JSON**: Single values, arrays, or matrices with automatic structure detection
-- **CSV/TSV**: Delimiter-separated values with headers for spreadsheet use
-- **Markdown**: Tables formatted for documentation and display
-- **Schema Support**: JSON Schema generation for API documentation
+- REST API backends requiring flexible data queries
+- Data export tools supporting multiple output formats
+- Interactive applications with user-friendly error messaging
+- Research platforms requiring structured data access
 
-## Usage
+## Installation
 
-### Basic Query Setup
+```bash
+cargo add brk_interface
+```
+
+## Quick Start
 
 ```rust
-use brk_interface::{Interface, Params, ParamsOpt, Index, Format};
+use brk_interface::{Interface, Params, Index};
 use brk_indexer::Indexer;
 use brk_computer::Computer;
 
-// Load data sources
-let indexer = Indexer::forced_import("./brk_data")?;
-let computer = Computer::forced_import("./brk_data", &indexer, None)?;
-
-// Create unified interface
+// Initialize with indexer and computer
+let indexer = Indexer::build(/* config */)?;
+let computer = Computer::build(/* config */)?;
 let interface = Interface::build(&indexer, &computer);
-```
 
-### Single Dataset Queries
-
-```rust
-// Get latest block data
+// Query data with parameters
 let params = Params {
     index: Index::Height,
-    ids: vec!["date", "timestamp", "difficulty"].into(),
-    rest: ParamsOpt::default()
-        .set_from(-1)  // Latest block
-        .set_format(Format::JSON),
+    ids: vec!["height-to-blockhash".to_string()].into(),
+    from: Some(800000),
+    to: Some(800100),
+    format: Some(Format::JSON),
+    ..Default::default()
 };
 
-let result = interface.search_and_format(params)?;
-println!("{}", result);
+// Search and format results
+let output = interface.search_and_format(params)?;
+println!("{}", output);
 ```
 
-### Range Queries
+## API Overview
+
+### Core Types
+
+- **`Interface<'a>`**: Main query interface coordinating indexer and computer access
+- **`Params`**: Query parameters including index, IDs, range, and formatting options
+- **`Index`**: Enumeration of available data indexes (Height, Date, Address, etc.)
+- **`Format`**: Output format specification (JSON, CSV)
+- **`Output`**: Formatted query results with multiple value types
+
+### Key Methods
+
+**`Interface::build(indexer: &Indexer, computer: &Computer) -> Self`**
+Creates interface instance with references to data sources.
+
+**`search(&self, params: &Params) -> Result<Vec<(String, &&dyn AnyCollectableVec)>>`**
+Searches for vectors matching the query parameters with intelligent error handling.
+
+**`format(&self, vecs: Vec<...>, params: &ParamsOpt) -> Result<Output>`**
+Formats search results according to specified output format and range parameters.
+
+**`search_and_format(&self, params: Params) -> Result<Output>`**
+Combined search and formatting operation for single-call data retrieval.
+
+### Query Parameters
+
+**Core Parameters:**
+
+- `index`: Data index to query (height, date, address, etc.)
+- `ids`: Vector IDs to retrieve from the specified index
+- `from`/`to`: Optional range filtering (inclusive start, exclusive end)
+- `format`: Output format (defaults to JSON)
+
+**Pagination Parameters:**
+
+- `offset`: Number of entries to skip
+- `limit`: Maximum entries to return
+
+## Examples
+
+### Basic Data Query
 
 ```rust
-// Get price data for last 30 days
-let params = Params {
-    index: Index::DateIndex,
-    ids: vec!["price_usd", "price_usd_high", "price_usd_low"].into(),
-    rest: ParamsOpt::default()
-        .set_from(-30)  // Last 30 days
-        .set_count(30)
-        .set_format(Format::CSV),
-};
+use brk_interface::{Interface, Params, Index, Format};
 
-let csv_data = interface.search_and_format(params)?;
-```
+let interface = Interface::build(&indexer, &computer);
 
-### Multiple Datasets
-
-```rust
-// Get comprehensive block statistics
+// Query block heights to hashes
 let params = Params {
     index: Index::Height,
-    ids: vec!["size", "weight", "tx_count", "fee_total"].into(),
-    rest: ParamsOpt::default()
-        .set_from(800_000)  // Starting from block 800,000
-        .set_to(800_100)    // Up to block 800,100
-        .set_format(Format::TSV),
+    ids: vec!["height-to-blockhash".to_string()].into(),
+    from: Some(750000),
+    to: Some(750010),
+    format: Some(Format::JSON),
+    ..Default::default()
 };
 
-let tsv_data = interface.search_and_format(params)?;
-```
-
-### Flexible ID Specification
-
-```rust
-// Different ways to specify dataset IDs
-let params = Params {
-    index: Index::DateIndex,
-    ids: vec!["price_usd,volume_usd,market_cap"].into(),  // Comma-separated
-    // OR: ids: vec!["price_usd volume_usd market_cap"].into(),  // Space-separated
-    // OR: ids: vec!["price_usd", "volume_usd", "market_cap"].into(),  // Array
-    rest: ParamsOpt::default()
-        .set_format(Format::JSON),
-};
-```
-
-### Advanced Queries with Pagination
-
-```rust
-// Query with custom pagination
-let params = Params {
-    index: Index::MonthIndex,
-    ids: vec!["supply_total", "supply_active"].into(),
-    rest: ParamsOpt::default()
-        .set_from(0)        // From beginning
-        .set_count(50)      // Max 50 results
-        .set_format(Format::Markdown),
-};
-
-let markdown_table = interface.search_and_format(params)?;
-```
-
-## API Methods
-
-### Core Query Methods
-
-```rust
-// Combined search and format
-let result = interface.search_and_format(params)?;
-
-// Separate search and format
-let vecs = interface.search(params)?;
-let formatted = interface.format(vecs, params.rest.format.unwrap_or_default())?;
-
-// Get metadata
-let current_height = interface.get_height();
-let available_datasets = interface.get_vecids(None)?;  // Paginated
-let available_indices = interface.get_indexes();
-```
-
-### Working with Results
-
-```rust
-use brk_interface::{Value, Output};
-
-// Handle different output types
 match interface.search_and_format(params)? {
-    Output::Json(Value::Single(val)) => println!("Single value: {}", val),
-    Output::Json(Value::List(arr)) => println!("Array with {} items", arr.len()),
-    Output::Json(Value::Matrix(matrix)) => println!("Matrix: {}x{}", matrix.len(), matrix[0].len()),
-    Output::Delimited(csv_data) => println!("CSV/TSV data:\n{}", csv_data),
-    Output::Markdown(table) => println!("Markdown table:\n{}", table),
+    Output::Json(value) => println!("{}", serde_json::to_string_pretty(&value)?),
+    _ => unreachable!(),
 }
 ```
 
-## Available Indices
-
-### Time-based Indices
-- `Height` - Block height (0, 1, 2, ...)
-- `DateIndex` - Days since Bitcoin genesis
-- `WeekIndex`, `MonthIndex`, `QuarterIndex`, `YearIndex`, `DecadeIndex`
-- `HalvingEpoch`, `DifficultyEpoch` - Bitcoin-specific epochs
-
-### Bitcoin-specific Indices
-- Address types: `P2PKHAddressIndex`, `P2SHAddressIndex`, `P2WPKHAddressIndex`, etc.
-- Output types: `OutputType` classifications
-- Transaction types: `TxIndex`, `InputIndex`, `OutputIndex`
-
-## Parameter Reference
+### CSV Export with Range Query
 
 ```rust
-pub struct Params {
-    pub index: Index,           // Time dimension for query
-    pub ids: MaybeIds,         // Dataset identifiers
-    pub rest: ParamsOpt,       // Optional parameters
-}
+use brk_interface::{Interface, Params, Index, Format};
 
-pub struct ParamsOpt {
-    pub from: Option<i64>,     // Starting index (negative = from end)
-    pub to: Option<i64>,       // Ending index (exclusive)
-    pub count: Option<usize>,  // Maximum results
-    pub format: Option<Format>, // Output format
+// Export price data as CSV
+let params = Params {
+    index: Index::Date,
+    ids: vec!["dateindex-to-price-close".to_string()].into(),
+    from: Some(0),      // From genesis
+    to: Some(5000),     // First ~13 years
+    format: Some(Format::CSV),
+    ..Default::default()
+};
+
+match interface.search_and_format(params)? {
+    Output::CSV(csv_text) => {
+        std::fs::write("bitcoin_prices.csv", csv_text)?;
+        println!("Price data exported to bitcoin_prices.csv");
+    },
+    _ => unreachable!(),
 }
 ```
 
-## Output Formats
-
-- **JSON**: Structured data (single values, arrays, matrices)
-- **CSV**: Comma-separated values with headers
-- **TSV**: Tab-separated values with headers
-- **Markdown**: Formatted tables for documentation
-
-## Performance Features
-
-- **Zero-copy operations**: Uses references and lifetimes to avoid cloning
-- **Memory mapping**: Leverages vecdb's memory-mapped storage
-- **Lazy evaluation**: Only processes data when formatting is requested
-- **Efficient pagination**: Smart bounds calculation and range queries
-
-## Schema Support
-
-The interface provides JSON Schema support for API documentation:
+### Intelligent Error Handling
 
 ```rust
-use schemars::schema_for;
+use brk_interface::{Interface, Params, Index};
 
-let schema = schema_for!(Params);
-// Use schema for API documentation
+// Query with typo in vector ID
+let params = Params {
+    index: Index::Height,
+    ids: vec!["height-to-blockhas".to_string()].into(), // Typo: "blockhas"
+    ..Default::default()
+};
+
+// Interface provides helpful error with suggestions
+match interface.search(&params) {
+    Err(error) => {
+        println!("{}", error);
+        // Output: No vec named "height-to-blockhas" indexed by "height" found.
+        //         Maybe you meant one of the following: ["height-to-blockhash"] ?
+    },
+    Ok(_) => unreachable!(),
+}
 ```
 
-## Dependencies
+## Architecture
 
-- `brk_indexer` - Indexed blockchain data source
-- `brk_computer` - Computed analytics data source
-- `vecdb` - Vector database with collection traits
-- `serde` - Serialization/deserialization support
-- `schemars` - JSON Schema generation
+### Data Source Integration
+
+The interface acts as a bridge between:
+
+- **Indexer**: Raw blockchain data vectors (blocks, transactions, addresses)
+- **Computer**: Computed analytics vectors (prices, statistics, aggregations)
+- **Unified Access**: Single query interface for both data sources
+
+### Search Implementation
+
+1. **Parameter Validation**: Validates index existence and parameter consistency
+2. **Vector Resolution**: Maps vector IDs to actual data structures
+3. **Fuzzy Matching**: Provides suggestions for mistyped vector names
+4. **Error Caching**: Caches error messages to avoid repeated expensive operations
+
+### Output Formatting
+
+**JSON Output:**
+
+- Single value: Direct value serialization
+- List: Array of values
+- Matrix: Array of arrays for multi-vector queries
+
+**CSV Output:**
+
+- Column headers from vector IDs
+- Row-wise data iteration with proper escaping
+
+### Caching Strategy
+
+- **Error Message Caching**: 1000-entry LRU cache for error messages
+- **Search Result Caching**: Upstream caching in server/client layers
+- **Static Data Caching**: Index and vector metadata cached during initialization
+
+## Configuration
+
+### Index Types
+
+Available indexes include:
+
+- `Height`: Block height-based indexing
+- `Date`: Calendar date indexing
+- `Address`: Bitcoin address indexing
+- `Transaction`: Transaction hash indexing
+- Custom indexes from computer analytics
+
+### Format Options
+
+- **JSON**: Structured data with nested objects/arrays
+- **CSV**: Comma-separated values with proper escaping
+
+## Code Analysis Summary
+
+**Main Structure**: `Interface` struct coordinating between `Indexer` and `Computer` data sources \
+**Query System**: Parameter-driven search with `Params` struct supporting range queries and formatting options \
+**Error Handling**: Intelligent fuzzy matching with cached error messages and helpful suggestions \
+**Output Formats**: Multi-format support (JSON, CSV) with proper data serialization \
+**Caching**: `quick_cache` integration for error messages and expensive operations \
+**Search Logic**: `nucleo-matcher` fuzzy search for user-friendly vector name resolution \
+**Architecture**: Abstraction layer providing unified access to heterogeneous Bitcoin data sources
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_interface/examples/main.rs

@@ -1,18 +1,40 @@
-use std::path::Path;
+use std::{fs, path::Path};
 
 use brk_computer::Computer;
 use brk_error::Result;
 use brk_indexer::Indexer;
 use brk_interface::{Index, Interface, Params, ParamsOpt};
+use brk_parser::Parser;
+use vecdb::Exit;
 
 pub fn main() -> Result<()> {
-    let outputs_dir = Path::new("../../_outputs");
+    let bitcoin_dir = Path::new(&std::env::var("HOME").unwrap())
+        .join("Library")
+        .join("Application Support")
+        .join("Bitcoin");
+    // let bitcoin_dir = Path::new("/Volumes/WD_BLACK1/bitcoin");
 
-    let indexer = Indexer::forced_import(outputs_dir)?;
+    let blocks_dir = bitcoin_dir.join("blocks");
 
-    let computer = Computer::forced_import(outputs_dir, &indexer, None)?;
+    let outputs_dir = Path::new(&std::env::var("HOME").unwrap()).join(".brk");
+    fs::create_dir_all(&outputs_dir)?;
+    // let outputs_dir = Path::new("/Volumes/WD_BLACK1/brk");
 
-    let interface = Interface::build(&indexer, &computer);
+    let rpc = Box::leak(Box::new(bitcoincore_rpc::Client::new(
+        "http://localhost:8332",
+        bitcoincore_rpc::Auth::CookieFile(bitcoin_dir.join(".cookie")),
+    )?));
+
+    let exit = Exit::new();
+    exit.set_ctrlc_handler();
+
+    let parser = Parser::new(blocks_dir, rpc);
+
+    let indexer = Indexer::forced_import(&outputs_dir)?;
+
+    let computer = Computer::forced_import(&outputs_dir, &indexer, None)?;
+
+    let interface = Interface::build(&parser, &indexer, &computer);
 
     dbg!(interface.search_and_format(Params {
         index: Index::Height,

crates/brk_interface/src/format.rs

@@ -1,39 +1,26 @@
-use brk_error::Error;
 use schemars::JsonSchema;
 use serde::Deserialize;
 
 #[allow(clippy::upper_case_acronyms)]
-#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize, JsonSchema)]
+#[derive(Default, Debug, Clone, Copy, PartialEq, Eq, Deserialize, JsonSchema)]
+#[serde(rename_all = "lowercase")]
 pub enum Format {
+    #[default]
     #[serde(alias = "json")]
     JSON,
     #[serde(alias = "csv")]
     CSV,
-    #[serde(alias = "tsv")]
-    TSV,
-    #[serde(alias = "md", alias = "markdown")]
-    MD,
 }
 
-impl TryFrom<Option<String>> for Format {
-    type Error = Error;
-    fn try_from(value: Option<String>) -> Result<Self, Self::Error> {
+impl From<Option<String>> for Format {
+    fn from(value: Option<String>) -> Self {
         if let Some(value) = value {
             let value = value.to_lowercase();
             let value = value.as_str();
-            if value == "md" || value == "markdown" {
-                Ok(Self::MD)
-            } else if value == "csv" {
-                Ok(Self::CSV)
-            } else if value == "tsv" {
-                Ok(Self::TSV)
-            } else if value == "json" {
-                Ok(Self::JSON)
-            } else {
-                Err(Error::Str("Fail"))
+            if value == "csv" {
+                return Self::CSV;
             }
-        } else {
-            Err(Error::Str("Fail"))
         }
+        Self::JSON
     }
 }

crates/brk_interface/src/ids.rs

@@ -0,0 +1,82 @@
+use std::fmt;
+
+use derive_deref::Deref;
+use schemars::JsonSchema;
+use serde::Deserialize;
+
+#[derive(Debug, Deref, JsonSchema)]
+pub struct MaybeIds(Vec<String>);
+
+const MAX_VECS: usize = 32;
+const MAX_STRING_SIZE: usize = 64 * MAX_VECS;
+
+impl From<String> for MaybeIds {
+    fn from(value: String) -> Self {
+        Self(vec![value])
+    }
+}
+
+impl<'a> From<Vec<&'a str>> for MaybeIds {
+    fn from(value: Vec<&'a str>) -> Self {
+        Self(value.iter().map(|s| s.to_string()).collect::<Vec<_>>())
+    }
+}
+
+impl<'de> Deserialize<'de> for MaybeIds {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        match serde_json::Value::deserialize(deserializer)? {
+            serde_json::Value::String(str) => {
+                if str.len() <= MAX_STRING_SIZE {
+                    Ok(MaybeIds(sanitize_ids(
+                        str.split(",").map(|s| s.to_string()),
+                    )))
+                } else {
+                    Err(serde::de::Error::custom("Given parameter is too long"))
+                }
+            }
+            serde_json::Value::Array(vec) => {
+                if vec.len() <= MAX_VECS {
+                    Ok(MaybeIds(sanitize_ids(
+                        vec.into_iter().map(|s| s.as_str().unwrap().to_string()),
+                    )))
+                } else {
+                    Err(serde::de::Error::custom("Given parameter is too long"))
+                }
+            }
+            _ => Err(serde::de::Error::custom("Bad ids format")),
+        }
+    }
+}
+
+impl fmt::Display for MaybeIds {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        let s = self.0.join(",");
+        write!(f, "{s}")
+    }
+}
+
+fn sanitize_ids(raw_ids: impl Iterator<Item = String>) -> Vec<String> {
+    let mut results = Vec::new();
+    raw_ids.for_each(|s| {
+        let mut current = String::new();
+        for c in s.to_lowercase().chars() {
+            match c {
+                ' ' | ',' | '+' => {
+                    if !current.is_empty() {
+                        results.push(std::mem::take(&mut current));
+                    }
+                }
+                '-' => current.push('_'),
+                c if c.is_alphanumeric() || c == '_' => current.push(c),
+                _ => {}
+            }
+        }
+        if !current.is_empty() {
+            results.push(current);
+        }
+    });
+    results
+}

crates/brk_interface/src/index.rs

@@ -2,16 +2,17 @@ use std::fmt::{self, Debug};
 
 use brk_error::Error;
 use brk_structs::{
-    DateIndex, DecadeIndex, DifficultyEpoch, EmptyOutputIndex, HalvingEpoch, Height, InputIndex,
-    MonthIndex, OpReturnIndex, OutputIndex, P2AAddressIndex, P2MSOutputIndex, P2PK33AddressIndex,
-    P2PK65AddressIndex, P2PKHAddressIndex, P2SHAddressIndex, P2TRAddressIndex, P2WPKHAddressIndex,
-    P2WSHAddressIndex, Printable, QuarterIndex, SemesterIndex, TxIndex, UnknownOutputIndex,
-    WeekIndex, YearIndex,
+    DateIndex, DecadeIndex, DifficultyEpoch, EmptyAddressIndex, EmptyOutputIndex, HalvingEpoch,
+    Height, InputIndex, LoadedAddressIndex, MonthIndex, OpReturnIndex, OutputIndex,
+    P2AAddressIndex, P2MSOutputIndex, P2PK33AddressIndex, P2PK65AddressIndex, P2PKHAddressIndex,
+    P2SHAddressIndex, P2TRAddressIndex, P2WPKHAddressIndex, P2WSHAddressIndex, PrintableIndex,
+    QuarterIndex, SemesterIndex, TxIndex, UnknownOutputIndex, WeekIndex, YearIndex,
 };
 use schemars::JsonSchema;
 use serde::Deserialize;
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, JsonSchema)]
+#[serde(rename_all = "lowercase")]
 pub enum Index {
     #[schemars(description = "Date/day index")]
     DateIndex,
@@ -63,10 +64,14 @@ pub enum Index {
     WeekIndex,
     #[schemars(description = "Year index")]
     YearIndex,
+    #[schemars(description = "Loaded Address Index")]
+    LoadedAddressIndex,
+    #[schemars(description = "Empty Address Index")]
+    EmptyAddressIndex,
 }
 
 impl Index {
-    pub fn all() -> [Self; 25] {
+    pub fn all() -> [Self; 27] {
         [
             Self::DateIndex,
             Self::DecadeIndex,
@@ -93,6 +98,8 @@ impl Index {
             Self::UnknownOutputIndex,
             Self::WeekIndex,
             Self::YearIndex,
+            Self::LoadedAddressIndex,
+            Self::EmptyAddressIndex,
         ]
     }
 
@@ -123,6 +130,8 @@ impl Index {
             Self::UnknownOutputIndex => UnknownOutputIndex::to_possible_strings(),
             Self::WeekIndex => WeekIndex::to_possible_strings(),
             Self::YearIndex => YearIndex::to_possible_strings(),
+            Self::LoadedAddressIndex => LoadedAddressIndex::to_possible_strings(),
+            Self::EmptyAddressIndex => EmptyAddressIndex::to_possible_strings(),
         }
     }
 
@@ -186,6 +195,12 @@ impl TryFrom<&str> for Index {
             v if (Self::UnknownOutputIndex).possible_values().contains(&v) => {
                 Self::UnknownOutputIndex
             }
+            v if (Self::LoadedAddressIndex).possible_values().contains(&v) => {
+                Self::LoadedAddressIndex
+            }
+            v if (Self::EmptyAddressIndex).possible_values().contains(&v) => {
+                Self::EmptyAddressIndex
+            }
             _ => return Err(Error::Str("Bad index")),
         })
     }

crates/brk_interface/src/lib.rs

@@ -1,22 +1,26 @@
 #![doc = include_str!("../README.md")]
 
-use std::collections::BTreeMap;
+use std::{collections::BTreeMap, sync::OnceLock};
 
 use brk_computer::Computer;
-use brk_error::Result;
+use brk_error::{Error, Result};
 use brk_indexer::Indexer;
+use brk_parser::Parser;
 use brk_structs::Height;
-use tabled::settings::Style;
+use nucleo_matcher::{
+    Config, Matcher,
+    pattern::{AtomKind, CaseMatching, Normalization, Pattern},
+};
+use quick_cache::sync::Cache;
 use vecdb::{AnyCollectableVec, AnyStoredVec};
 
 mod deser;
 mod format;
+mod ids;
 mod index;
-mod maybe_ids;
 mod output;
 mod pagination;
 mod params;
-mod table;
 mod vecs;
 
 pub use format::Format;
@@ -24,26 +28,33 @@ pub use index::Index;
 pub use output::{Output, Value};
 pub use pagination::{PaginatedIndexParam, PaginationParam};
 pub use params::{IdParam, Params, ParamsOpt};
-pub use table::Tabled;
 use vecs::Vecs;
 
 use crate::vecs::{IdToVec, IndexToVec};
 
+pub fn cached_errors() -> &'static Cache<String, String> {
+    static CACHE: OnceLock<Cache<String, String>> = OnceLock::new();
+    CACHE.get_or_init(|| Cache::new(1000))
+}
+
 #[allow(dead_code)]
 pub struct Interface<'a> {
     vecs: Vecs<'a>,
+    parser: &'a Parser,
     indexer: &'a Indexer,
     computer: &'a Computer,
 }
 
 impl<'a> Interface<'a> {
-    pub fn build(indexer: &Indexer, computer: &Computer) -> Self {
+    pub fn build(parser: &Parser, indexer: &Indexer, computer: &Computer) -> Self {
+        let parser = parser.static_clone();
         let indexer = indexer.static_clone();
         let computer = computer.static_clone();
         let vecs = Vecs::build(indexer, computer);
 
         Self {
             vecs,
+            parser,
             indexer,
             computer,
         }
@@ -53,38 +64,64 @@ impl<'a> Interface<'a> {
         Height::from(self.indexer.vecs.height_to_blockhash.stamp())
     }
 
-    pub fn search(&self, params: &Params) -> Vec<(String, &&dyn AnyCollectableVec)> {
-        let tuples = params
-            .ids
-            .iter()
-            .flat_map(|s| {
-                s.to_lowercase()
-                    .replace("-", "_")
-                    .split_whitespace()
-                    .flat_map(|s| {
-                        s.split(',')
-                            .flat_map(|s| s.split('+').map(|s| s.to_string()))
-                    })
-                    .collect::<Vec<_>>()
-            })
-            .map(|mut id| {
-                let mut res = self.vecs.id_to_index_to_vec.get(id.as_str());
-                if res.is_none()
-                    && let Ok(index) = Index::try_from(id.as_str())
-                {
-                    id = index.possible_values().last().unwrap().to_string();
-                    res = self.vecs.id_to_index_to_vec.get(id.as_str())
-                }
-                (id, res)
-            })
-            .filter(|(_, opt)| opt.is_some())
-            .map(|(id, vec)| (id, vec.unwrap()))
-            .collect::<Vec<_>>();
+    pub fn search(&self, params: &Params) -> Result<Vec<(String, &&dyn AnyCollectableVec)>> {
+        let ids = &params.ids;
+        let index = params.index;
 
-        tuples
-            .iter()
-            .flat_map(|(str, i_to_v)| i_to_v.get(&params.index).map(|vec| (str.to_owned(), vec)))
-            .collect::<Vec<_>>()
+        let ids_to_vec = self
+            .vecs
+            .index_to_id_to_vec
+            .get(&index)
+            .ok_or(Error::String(format!(
+                "Index \"{}\" isn't a valid index",
+                index
+            )))?;
+
+        ids.iter()
+            .map(|id| {
+                let vec = ids_to_vec.get(id.as_str()).ok_or_else(|| {
+                    let cached_errors = cached_errors();
+
+                    if let Some(message) = cached_errors.get(id) {
+                        return Error::String(message)
+                    }
+
+                    let mut message = format!(
+                        "No vec named \"{}\" indexed by \"{}\" found.\n",
+                        id,
+                        index
+                    );
+
+                    let mut matcher = Matcher::new(Config::DEFAULT);
+
+                    let matches = Pattern::new(
+                        id.as_str(),
+                        CaseMatching::Ignore,
+                        Normalization::Smart,
+                        AtomKind::Fuzzy,
+                    )
+                    .match_list(ids_to_vec.keys(), &mut matcher)
+                    .into_iter()
+                    .take(10)
+                    .map(|(s, _)| s)
+                    .collect::<Vec<_>>();
+
+                    if !matches.is_empty() {
+                        message +=
+                            &format!("\nMaybe you meant one of the following: {matches:#?} ?\n");
+                    }
+
+                    if let Some(index_to_vec) = self.id_to_index_to_vec().get(id.as_str()) {
+                        message += &format!("\nBut there is a vec named {id} which supports the following indexes: {:#?}\n", index_to_vec.keys());
+                    }
+
+                    cached_errors.insert(id.clone(), message.clone());
+
+                    Error::String(message)
+                });
+                vec.map(|vec| (id.clone(), vec))
+            })
+            .collect::<Result<Vec<_>>>()
     }
 
     pub fn format(
@@ -106,75 +143,66 @@ impl<'a> Interface<'a> {
                 .unwrap_or_default()
         });
 
-        let mut values = vecs
-            .iter()
-            .map(|(_, vec)| -> Result<Vec<serde_json::Value>> {
-                Ok(vec.collect_range_serde_json(from, to)?)
-            })
-            .collect::<Result<Vec<_>>>()?;
-
         let format = params.format();
 
-        if values.is_empty() {
-            return Ok(Output::default(format));
-        }
-
-        let ids_last_i = vecs.len() - 1;
-
         Ok(match format {
-            Some(Format::CSV) | Some(Format::TSV) => {
-                let delimiter = if format == Some(Format::CSV) {
-                    ','
-                } else {
-                    '\t'
-                };
-
-                let mut text = vecs
+            Format::CSV => {
+                let headers = vecs.iter().map(|(id, _)| id.as_str()).collect::<Vec<_>>();
+                let mut values = vecs
                     .iter()
-                    .map(|(id, _)| id.to_owned())
-                    .collect::<Vec<_>>()
-                    .join(&delimiter.to_string());
+                    .map(|(_, vec)| Ok(vec.collect_range_string(from, to)?))
+                    .collect::<Result<Vec<_>>>()?;
 
-                text.push('\n');
-
-                let values_len = values.first().unwrap().len();
-
-                (0..values_len).for_each(|i| {
-                    let mut line = "".to_string();
-                    values.iter().enumerate().for_each(|(id_i, v)| {
-                        line += &v.get(i).unwrap().to_string();
-                        if id_i == ids_last_i {
-                            line.push('\n');
-                        } else {
-                            line.push(delimiter);
-                        }
-                    });
-                    text += &line;
-                });
-
-                if format == Some(Format::CSV) {
-                    Output::CSV(text)
-                } else {
-                    Output::TSV(text)
+                if values.is_empty() {
+                    return Ok(Output::CSV(headers.join(",")));
                 }
-            }
-            Some(Format::MD) => {
-                let mut table =
-                    values.to_table(vecs.iter().map(|(s, _)| s.to_owned()).collect::<Vec<_>>());
 
-                table.with(Style::markdown());
+                let first_len = values[0].len();
+                let estimated_size = (headers.len() + values.len() * first_len) * 15;
+                let mut csv = String::with_capacity(estimated_size);
 
-                Output::MD(table.to_string())
-            }
-            Some(Format::JSON) | None => {
-                if values.len() == 1 {
-                    let mut values = values.pop().unwrap();
-                    if values.len() == 1 {
-                        let value = values.pop().unwrap();
-                        Output::Json(Value::Single(value))
-                    } else {
-                        Output::Json(Value::List(values))
+                csv.push_str(&headers.join(","));
+                csv.push('\n');
+
+                for col_index in 0..first_len {
+                    let mut first = true;
+                    for vec in &mut values {
+                        if col_index < vec.len() {
+                            if !first {
+                                csv.push(',');
+                            }
+                            first = false;
+
+                            let field = std::mem::take(&mut vec[col_index]);
+
+                            if field.contains(',') {
+                                csv.push('"');
+                                csv.push_str(&field);
+                                csv.push('"');
+                            } else {
+                                csv.push_str(&field);
+                            }
+                        }
                     }
+                    csv.push('\n');
+                }
+
+                Output::CSV(csv)
+            }
+            Format::JSON => {
+                let mut values = vecs
+                    .iter()
+                    .map(|(_, vec)| -> Result<Vec<u8>> {
+                        Ok(vec.collect_range_json_bytes(from, to)?)
+                    })
+                    .collect::<Result<Vec<_>>>()?;
+
+                if values.is_empty() {
+                    return Ok(Output::default(format));
+                }
+
+                if values.len() == 1 {
+                    Output::Json(Value::List(values.pop().unwrap()))
                 } else {
                     Output::Json(Value::Matrix(values))
                 }
@@ -183,7 +211,7 @@ impl<'a> Interface<'a> {
     }
 
     pub fn search_and_format(&self, params: Params) -> Result<Output> {
-        self.format(self.search(&params), &params.rest)
+        self.format(self.search(&params)?, &params.rest)
     }
 
     pub fn id_to_index_to_vec(&self) -> &BTreeMap<&str, IndexToVec<'_>> {
@@ -225,4 +253,16 @@ impl<'a> Interface<'a> {
     pub fn get_vecid_to_indexes(&self, id: String) -> Option<&Vec<&'static str>> {
         self.vecs.id_to_indexes(id)
     }
+
+    pub fn parser(&self) -> &Parser {
+        self.parser
+    }
+
+    pub fn indexer(&self) -> &Indexer {
+        self.indexer
+    }
+
+    pub fn computer(&self) -> &Computer {
+        self.computer
+    }
 }

crates/brk_interface/src/maybe_ids.rs

@@ -1,38 +0,0 @@
-use derive_deref::Deref;
-use schemars::JsonSchema;
-use serde::Deserialize;
-
-#[derive(Debug, Deref, JsonSchema)]
-pub struct MaybeIds(Vec<String>);
-
-impl From<String> for MaybeIds {
-    fn from(value: String) -> Self {
-        Self(vec![value])
-    }
-}
-
-impl<'a> From<Vec<&'a str>> for MaybeIds {
-    fn from(value: Vec<&'a str>) -> Self {
-        Self(value.iter().map(|s| s.to_string()).collect::<Vec<_>>())
-    }
-}
-
-impl<'de> Deserialize<'de> for MaybeIds {
-    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
-    where
-        D: serde::Deserializer<'de>,
-    {
-        let maybe_ids = match serde_json::Value::deserialize(deserializer)? {
-            serde_json::Value::String(str) => {
-                str.split(",").map(|s| s.to_string()).collect::<Vec<_>>()
-            }
-            serde_json::Value::Array(vec) => vec
-                .into_iter()
-                .map(|s| s.as_str().unwrap().to_string())
-                .collect::<Vec<_>>(),
-            _ => return Err(serde::de::Error::custom("Bad ids format")),
-        };
-        // dbg!(&maybe_ids);
-        Ok(MaybeIds(maybe_ids))
-    }
-}

crates/brk_interface/src/output.rs

@@ -1,44 +1,54 @@
-use std::fmt;
-
-use serde::Serialize;
-use tabled::Tabled as TabledTabled;
-
 use crate::Format;
 
-#[derive(Debug, Serialize)]
-#[serde(untagged)]
+#[derive(Debug)]
 pub enum Output {
     Json(Value),
     CSV(String),
-    TSV(String),
-    MD(String),
-}
-
-#[derive(Debug, Serialize, TabledTabled)]
-#[serde(untagged)]
-pub enum Value {
-    Matrix(Vec<Vec<serde_json::Value>>),
-    List(Vec<serde_json::Value>),
-    Single(serde_json::Value),
 }
 
 impl Output {
-    pub fn default(format: Option<Format>) -> Self {
-        match format {
-            Some(Format::CSV) => Output::CSV("".to_string()),
-            Some(Format::TSV) => Output::TSV("".to_string()),
-            _ => Output::Json(Value::Single(serde_json::Value::Null)),
+    #[allow(clippy::inherent_to_string)]
+    pub fn to_string(self) -> String {
+        match self {
+            Output::CSV(s) => s,
+            Output::Json(v) => unsafe { String::from_utf8_unchecked(v.to_vec()) },
         }
     }
 }
 
-impl fmt::Display for Output {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+#[derive(Debug)]
+pub enum Value {
+    Matrix(Vec<Vec<u8>>),
+    List(Vec<u8>),
+}
+
+impl Value {
+    pub fn to_vec(self) -> Vec<u8> {
         match self {
-            Self::Json(value) => write!(f, "{}", serde_json::to_string_pretty(value).unwrap()),
-            Self::CSV(string) => write!(f, "{string}"),
-            Self::TSV(string) => write!(f, "{string}"),
-            Self::MD(string) => write!(f, "{string}"),
+            Value::List(v) => v,
+            Self::Matrix(m) => {
+                let total_size = m.iter().map(|v| v.len()).sum::<usize>() + m.len() - 1 + 2;
+                let mut matrix = Vec::with_capacity(total_size);
+                matrix.push(b'[');
+
+                for (i, vec) in m.into_iter().enumerate() {
+                    if i > 0 {
+                        matrix.push(b',');
+                    }
+                    matrix.extend(vec);
+                }
+                matrix.push(b']');
+                matrix
+            }
+        }
+    }
+}
+
+impl Output {
+    pub fn default(format: Format) -> Self {
+        match format {
+            Format::CSV => Output::CSV("".to_string()),
+            Format::JSON => Output::Json(Value::List(b"[]".to_vec())),
         }
     }
 }

crates/brk_interface/src/params.rs

@@ -6,7 +6,7 @@ use serde::Deserialize;
 use crate::{
     Format, Index,
     deser::{de_unquote_i64, de_unquote_usize},
-    maybe_ids::MaybeIds,
+    ids::MaybeIds,
 };
 
 #[derive(Debug, Deserialize, JsonSchema)]
@@ -63,7 +63,7 @@ pub struct ParamsOpt {
     #[serde(default)]
     /// Format of the output
     #[schemars(description = "Format of the output")]
-    format: Option<Format>,
+    format: Format,
 }
 
 impl ParamsOpt {
@@ -82,11 +82,6 @@ impl ParamsOpt {
         self
     }
 
-    pub fn set_format(mut self, format: Format) -> Self {
-        self.format.replace(format);
-        self
-    }
-
     pub fn from(&self) -> Option<i64> {
         self.from
     }
@@ -107,7 +102,7 @@ impl ParamsOpt {
         self.to
     }
 
-    pub fn format(&self) -> Option<Format> {
+    pub fn format(&self) -> Format {
         self.format
     }
 }

crates/brk_interface/src/table.rs

@@ -1,26 +0,0 @@
-use tabled::{Table, builder::Builder};
-
-pub trait Tabled {
-    fn to_table(&self, ids: Vec<String>) -> Table;
-}
-
-impl Tabled for Vec<Vec<serde_json::Value>> {
-    fn to_table(&self, ids: Vec<String>) -> Table {
-        let mut builder = Builder::default();
-
-        builder.push_record(ids);
-
-        if let Some(first) = self.first() {
-            let len = first.len();
-
-            (0..len).for_each(|index| {
-                builder.push_record(
-                    self.iter()
-                        .map(|vec| vec.get(index).unwrap().to_string().replace("\"", "")),
-                );
-            });
-        }
-
-        builder.build()
-    }
-}

crates/brk_interface/src/vecs.rs

@@ -2,8 +2,8 @@ use std::collections::BTreeMap;
 
 use brk_computer::Computer;
 use brk_indexer::Indexer;
-use vecdb::AnyCollectableVec;
 use derive_deref::{Deref, DerefMut};
+use vecdb::AnyCollectableVec;
 
 use crate::pagination::{PaginatedIndexParam, PaginationParam};
 
@@ -27,13 +27,11 @@ impl<'a> Vecs<'a> {
     pub fn build(indexer: &'a Indexer, computer: &'a Computer) -> Self {
         let mut this = Vecs::default();
 
-        indexer
-            .vecs
-            .vecs()
-            .into_iter()
-            .for_each(|vec| this.insert(vec));
+        indexer.vecs.iter().for_each(|vec| this.insert(vec));
 
-        computer.vecs().into_iter().for_each(|vec| this.insert(vec));
+        computer
+            .iter_any_collectable()
+            .for_each(|vec| this.insert(vec));
 
         let mut ids = this.id_to_index_to_vec.keys().cloned().collect::<Vec<_>>();
 

crates/brk_logger/README.md

@@ -1,41 +1,36 @@
 # brk_logger
 
-**Logging utilities with colored console output and file logging**
+Colored console logging with optional file output for Bitcoin Research Kit applications.
 
-`brk_logger` provides a thin wrapper around `env_logger` with BRK-specific defaults, colored console output, and optional file logging. It's designed to provide clear, readable logs for Bitcoin data processing operations.
+[![Crates.io](https://img.shields.io/crates/v/brk_logger.svg)](https://crates.io/crates/brk_logger)
+[![Documentation](https://docs.rs/brk_logger/badge.svg)](https://docs.rs/brk_logger)
 
-## What it provides
+## Overview
 
-- **Colored Console Output**: Level-based color coding for easy visual parsing
-- **File Logging**: Optional log file output with automatic rotation
-- **Sensible Defaults**: Pre-configured log levels for Bitcoin and dependency crates
-- **Timestamp Formatting**: Human-readable timestamps with system timezone
+This crate provides a thin wrapper around `env_logger` with enhanced formatting, colored output, and optional file logging. Designed specifically for BRK applications, it offers structured logging with timestamps, level-based coloring, and automatic filtering of noisy dependencies.
 
-## Key Features
+**Key Features:**
 
-### Console Logging
-- **Color-coded levels**: Error (red), Warn (yellow), Info (green), Debug (blue), Trace (cyan)
-- **Formatted timestamps**: `YYYY-MM-DD HH:MM:SS` format with dimmed styling
-- **Clean output**: Minimal formatting focused on readability
+- Level-based color coding for console output (error=red, warn=yellow, info=green, etc.)
+- Dual output: colored console logs and optional plain-text file logging
+- System timezone-aware timestamps with customizable formatting
+- Automatic filtering of verbose dependencies (bitcoin, fjall, tracing, etc.)
+- Environment variable configuration support via `RUST_LOG`
 
-### File Logging
-- **Optional file output**: Writes to specified file path
-- **Automatic cleanup**: Removes existing log file on initialization
-- **Append mode**: New log entries appended to file
-- **Plain text format**: No colors in file output for better compatibility
+**Target Use Cases:**
 
-### Dependency Filtering
-Pre-configured to suppress noisy logs from common dependencies:
-- `bitcoin=off` - Bitcoin protocol library
-- `bitcoincore-rpc=off` - RPC client
-- `fjall=off` - Key-value store
-- `lsm_tree=off` - LSM tree implementation
-- `rolldown=off` - Bundler
-- `tracing=off` - Tracing framework
+- Bitcoin blockchain analysis applications requiring structured logging
+- Development and debugging of BRK components
+- Production deployments with file-based log archival
+- Applications needing both human-readable and machine-parseable logs
 
-## Usage
+## Installation
 
-### Basic Setup (Console Only)
+```bash
+cargo add brk_logger
+```
+
+## Quick Start
 
 ```rust
 use brk_logger;
@@ -43,150 +38,175 @@ use brk_logger;
 // Initialize with console output only
 brk_logger::init(None)?;
 
-// Now use standard logging macros
-log::info!("BRK starting up");
-log::warn!("Bitcoin Core not fully synced");
-log::error!("Failed to connect to RPC");
-```
-
-### With File Logging
-
-```rust
-use std::path::Path;
-
-// Initialize with both console and file output
-let log_path = Path::new("~/.brk/brk.log");
+// Initialize with file logging
+let log_path = std::path::Path::new("application.log");
 brk_logger::init(Some(log_path))?;
 
-log::info!("Logs will appear in console and file");
+// Use standard log macros
+log::info!("Application started");
+log::warn!("Configuration issue detected");
+log::error!("Failed to process block");
 ```
 
-### Environment Variable Control
+## API Overview
+
+### Core Functions
+
+**`init(path: Option<&Path>) -> io::Result<()>`**
+Initializes the logging system with optional file output. Console logging is always enabled with color formatting.
+
+**Re-exported Types:**
+
+- **`OwoColorize`**: Color formatting trait for custom colored output
+
+### Log Formatting
+
+**Console Output Format:**
+
+```
+2024-09-16 14:23:45 - INFO  Application started successfully
+2024-09-16 14:23:46 - WARN  Bitcoin RPC connection slow
+2024-09-16 14:23:47 - ERROR Failed to parse block data
+```
+
+**File Output Format (Plain Text):**
+
+```
+2024-09-16 14:23:45 - info  Application started successfully
+2024-09-16 14:23:46 - warn  Bitcoin RPC connection slow
+2024-09-16 14:23:47 - error Failed to parse block data
+```
+
+### Default Filtering
+
+The logger automatically filters out verbose logs from common dependencies:
+
+- `bitcoin=off` - Bitcoin library internals
+- `bitcoincore-rpc=off` - RPC client details
+- `fjall=off` - Database engine logs
+- `lsm_tree=off` - LSM tree operations
+- `tracing=off` - Tracing framework internals
+
+## Examples
+
+### Basic Logging Setup
+
+```rust
+use brk_logger;
+use log::{info, warn, error};
+
+fn main() -> std::io::Result<()> {
+    // Initialize logging to console only
+    brk_logger::init(None)?;
+
+    info!("Starting Bitcoin analysis");
+    warn!("Price feed temporarily unavailable");
+    error!("Block parsing failed at height 750000");
+
+    Ok(())
+}
+```
+
+### File Logging with Custom Path
+
+```rust
+use brk_logger;
+use std::path::Path;
+
+fn main() -> std::io::Result<()> {
+    let log_file = Path::new("/var/log/brk/application.log");
+
+    // Initialize with dual output: console + file
+    brk_logger::init(Some(log_file))?;
+
+    log::info!("Application configured with file logging");
+
+    // Both console (colored) and file (plain) will receive this log
+    log::error!("Critical system error occurred");
+
+    Ok(())
+}
+```
+
+### Environment Variable Configuration
 
 ```bash
-# Set log level via environment variable
-export RUST_LOG=debug
-export RUST_LOG=info,brk_parser=debug  # Override for specific crates
+# Set custom log level and targets
+export RUST_LOG="debug,brk_indexer=trace,bitcoin=warn"
 
-# Run with custom log level
-RUST_LOG=trace brk
+# Run application - logger respects RUST_LOG
+./my_brk_app
 ```
 
-### Using Color Utilities
+```rust
+use brk_logger;
 
-The crate re-exports `OwoColorize` for consistent coloring:
+fn main() -> std::io::Result<()> {
+    // Respects RUST_LOG environment variable
+    brk_logger::init(None)?;
+
+    log::debug!("Debug information (only shown if RUST_LOG=debug)");
+    log::trace!("Trace information (only for specific modules)");
+
+    Ok(())
+}
+```
+
+### Colored Output Usage
 
 ```rust
 use brk_logger::OwoColorize;
 
-println!("Success: {}", "Operation completed".green());
-println!("Warning: {}", "Low disk space".yellow());
-println!("Error: {}", "Connection failed".red());
-println!("Info: {}", "Processing block 800000".bright_black());
-```
-
-## Log Format
-
-### Console Output
-```
-2024-12-25 10:30:15 - info  Starting BRK indexer
-2024-12-25 10:30:16 - warn  Bitcoin Core still syncing (99.8% complete)
-2024-12-25 10:30:45 - info  Indexed block 900000 (1.2M transactions)
-2024-12-25 10:30:46 - error Connection to RPC failed, retrying...
-```
-
-### File Output
-```
-2024-12-25 10:30:15 - info  Starting BRK indexer
-2024-12-25 10:30:16 - warn  Bitcoin Core still syncing (99.8% complete)
-2024-12-25 10:30:45 - info  Indexed block 900000 (1.2M transactions)
-2024-12-25 10:30:46 - error Connection to RPC failed, retrying...
-```
-
-## Default Log Levels
-
-The logger uses these default settings:
-
-- **Default level**: `info` - Shows important operational information
-- **Suppressed crates**: Dependencies that produce excessive output are set to `off`
-- **Override capability**: Can be overridden via `RUST_LOG` environment variable
-
-### Common Log Level Settings
-
-```bash
-# Minimal output (errors and warnings only)
-RUST_LOG=warn
-
-# Standard output (recommended)
-RUST_LOG=info
-
-# Verbose output (for debugging)
-RUST_LOG=debug
-
-# Maximum output (for development)
-RUST_LOG=trace
-
-# Mixed levels (info by default, debug for specific crates)
-RUST_LOG=info,brk_indexer=debug,brk_computer=trace
-```
-
-## Integration Examples
-
-### In BRK CLI
-
-```rust
-use brk_logger;
-use log::info;
-
-fn main() -> Result<()> {
-    // Initialize logging early in main
-    brk_logger::init(Some(Path::new("~/.brk/brk.log")))?;
-    
-    info!("BRK CLI starting");
-    
-    // ... rest of application
-    Ok(())
+fn print_status() {
+    println!("Status: {}", "Connected".green());
+    println!("Height: {}", "800000".bright_blue());
+    println!("Error:  {}", "Connection failed".red());
 }
 ```
 
-### In Custom Applications
+## Architecture
 
-```rust
-use brk_logger::{self, OwoColorize};
-use log::{info, warn, error};
+### Logging Pipeline
 
-fn setup_logging() -> std::io::Result<()> {
-    // Console only for development
-    brk_logger::init(None)?;
-    
-    info!("Application initialized");
-    Ok(())
-}
+1. **Initialization**: `init()` configures `env_logger` with custom formatter
+2. **Environment**: Reads `RUST_LOG` or uses default filter configuration
+3. **Formatting**: Applies timestamp, level formatting, and color coding
+4. **Dual Output**: Writes colored logs to console, plain logs to file (if configured)
 
-fn process_data() {
-    info!("Processing Bitcoin data...");
-    
-    // Use color utilities for progress
-    println!("Progress: {}", "50%".green());
-    
-    warn!("Large memory usage detected");
-    error!("Critical error: {}", "Database connection lost".red());
-}
-```
+### Color Scheme
 
-## Performance Considerations
+- **ERROR**: Red text for critical failures
+- **WARN**: Yellow text for warnings and issues
+- **INFO**: Green text for normal operations
+- **DEBUG**: Blue text for debugging information
+- **TRACE**: Cyan text for detailed tracing
+- **Timestamps**: Bright black (gray) for visual hierarchy
 
-- **Minimal overhead**: Lightweight wrapper around `env_logger`
-- **Lazy evaluation**: Log messages only formatted when level is enabled
-- **File I/O**: Asynchronous file writing doesn't block main thread
-- **Memory usage**: No buffering, logs written immediately
+### File Handling
 
-## Dependencies
+- **File Creation**: Creates log file if it doesn't exist
+- **Append Mode**: Appends to existing files without truncation
+- **Error Resilience**: Console logging continues even if file write fails
+- **File Rotation**: Manual - application responsible for log rotation
 
-- `env_logger` - Core logging implementation
-- `owo_colors` - Terminal color support  
-- `jiff` - Modern date/time handling for timestamps
+### Filtering Strategy
+
+Default filter prioritizes BRK application logs while suppressing:
+
+- Verbose dependency logging that clutters output
+- Internal library debug information not relevant to users
+- Framework-level tracing that doesn't aid in debugging BRK logic
+
+## Code Analysis Summary
+
+**Main Function**: `init()` function that configures `env_logger` with custom formatting and dual output \
+**Dependencies**: Built on `env_logger` for filtering, `jiff` for timestamps, `owo-colors` for terminal colors \
+**Output Streams**: Simultaneous console (colored) and file (plain text) logging with different formatting \
+**Timestamp Format**: System timezone-aware formatting using `%Y-%m-%d %H:%M:%S` pattern \
+**Color Implementation**: Level-based color mapping with `owo-colors` for terminal output \
+**Filter Configuration**: Predefined filter string that suppresses verbose dependencies while enabling BRK logs \
+**Architecture**: Thin wrapper pattern that enhances `env_logger` with BRK-specific formatting and behavior
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_mcp/README.md

@@ -1,221 +1,283 @@
 # brk_mcp
 
-**Model Context Protocol (MCP) bridge for LLM integration with BRK**
+Model Context Protocol bridge enabling LLM access to Bitcoin Research Kit data.
 
-`brk_mcp` provides a Model Context Protocol server that enables Large Language Models (LLMs) to access Bitcoin blockchain data through BRK's interface layer. It implements the MCP specification to expose BRK's analytics capabilities as tools that LLMs can call.
+[![Crates.io](https://img.shields.io/crates/v/brk_mcp.svg)](https://crates.io/crates/brk_mcp)
+[![Documentation](https://docs.rs/brk_mcp/badge.svg)](https://docs.rs/brk_mcp)
 
-## What it provides
+## Overview
 
-- **MCP Server Implementation**: Standards-compliant Model Context Protocol server
-- **Bitcoin Data Tools**: LLM-accessible tools for querying blockchain analytics
-- **Type-Safe Parameters**: Structured tool parameters with validation
-- **Multi-Format Output**: JSON responses for LLM consumption
-- **BRK Integration**: Direct access to all indexed and computed datasets
+This crate provides a Model Context Protocol (MCP) server implementation that exposes Bitcoin blockchain analytics as tools for Large Language Models. Built on the `brk_rmcp` library, it creates a standards-compliant bridge allowing LLMs to query Bitcoin data through structured tool calls with type-safe parameters and JSON responses.
 
-## Available MCP Tools
+**Key Features:**
 
-Based on the actual implementation, the following tools are available:
+- Model Context Protocol server with tools capability
+- 10 specialized tools for Bitcoin data discovery and querying
+- Type-safe parameter validation with structured error handling
+- Stateless operation for scalability and load balancing
+- Direct integration with BRK interface layer for real-time data access
+- Paginated responses for large datasets (up to 1,000 entries per page)
+- Multi-format output support through underlying interface layer
 
-### Metadata Tools
+**Target Use Cases:**
 
-#### `get_index_count`
-Get the count of all existing indexes.
+- LLM-powered Bitcoin analytics and research applications
+- Conversational interfaces for blockchain data exploration
+- AI-driven financial analysis requiring Bitcoin metrics
+- Automated research tools with natural language queries
 
-**Parameters:** None
-**Returns:** Number of available time indices
+## Claude
 
-#### `get_vecid_count`
-Get the count of all existing vector IDs.
+### Quick Start
 
-**Parameters:** None
-**Returns:** Number of available dataset identifiers
+Add a `bitview` custom connector with the following URL:
 
-#### `get_vec_count`
-Get the count of all existing vectors (sum of supported indexes for each vector ID).
+`https://bitview.space/mcp`
 
-**Parameters:** None
-**Returns:** Total number of vector/index combinations
+## Rust
 
-#### `get_indexes`
-Get the list of all existing indexes.
+### Installation
 
-**Parameters:** None
-**Returns:** Array of available index names (height, date, week, month, etc.)
-
-#### `get_accepted_indexes`
-Get an object with all existing indexes as keys and their accepted variants as values.
-
-**Parameters:** None
-**Returns:** Object mapping indexes to their accepted variant names
-
-### Discovery Tools
-
-#### `get_vecids`
-Get a paginated list of all existing vector IDs.
-
-**Parameters:**
-- `page` (optional number): Page number (default: 0, up to 1,000 results per page)
-
-**Returns:** Array of dataset identifiers
-
-#### `get_index_to_vecids`
-Get a paginated list of all vector IDs which support a given index.
-
-**Parameters:**
-- `index` (string): Index name to query
-- `page` (optional number): Page number (default: 0)
-
-**Returns:** Array of vector IDs that support the specified index
-
-#### `get_vecid_to_indexes`
-Get a list of all indexes supported by a given vector ID.
-
-**Parameters:**
-- `id` (string): Vector ID to query
-
-**Returns:** Array of indexes supported by the vector ID (empty if ID doesn't exist)
-
-### Data Query Tool
-
-#### `get_vecs`
-Get one or multiple vectors depending on given parameters.
-
-**Parameters:**
-- `index` (string): Time dimension (height, date, week, month, etc.)
-- `ids` (string): Dataset identifiers (comma or space separated)
-- `from` (optional i64): Start index (negative = from end)
-- `to` (optional i64): End index (exclusive)
-- `count` (optional usize): Maximum results
-- `format` (optional string): Output format (json, csv, tsv, md)
-
-**Response format depends on parameters:**
-- **Single value**: One vector, one result (e.g., `from=-1`)
-- **Array**: One vector, multiple results (e.g., `from=-100&count=100`)
-- **Matrix**: Multiple vectors (always matrix, even for single results)
-
-### System Tool
-
-#### `get_version`
-Get the running version of the Bitcoin Research Kit.
-
-**Parameters:** None
-**Returns:** Version string (e.g., "v0.0.88")
-
-## Usage Examples
-
-### Discovery Workflow
-
-```
-1. Call get_indexes to see available time dimensions
-2. Call get_vecids to see available datasets (paginated)
-3. Call get_index_to_vecids to find datasets for specific timeframes
-4. Call get_vecs to query actual data
+```bash
+cargo add brk_mcp
 ```
 
-### Basic Data Query
-
-```json
-// Get latest Bitcoin price
-{
-  "tool": "get_vecs",
-  "parameters": {
-    "index": "date",
-    "ids": "close",
-    "from": -1
-  }
-}
-```
-
-### Multi-Dataset Query
-
-```json
-// Get last 30 days of OHLC data
-{
-  "tool": "get_vecs",
-  "parameters": {
-    "index": "date",
-    "ids": "open,high,low,close",
-    "from": -30,
-    "format": "csv"
-  }
-}
-```
-
-### Metadata Exploration
-
-```json
-// Discover what's available
-{
-  "tool": "get_accepted_indexes"
-}
-
-// Find datasets for weekly analysis
-{
-  "tool": "get_index_to_vecids",
-  "parameters": {
-    "index": "week"
-  }
-}
-```
-
-## Server Configuration
-
-The MCP server is stateless and integrates with BRK's HTTP server:
+### Quick Start
 
 ```rust
-// In brk_server routes
-router.add_mcp_routes(interface, true)  // Enable MCP at /mcp endpoint
+use brk_mcp::MCP;
+use brk_interface::Interface;
+use brk_rmcp::{ServerHandler, RoleServer};
+
+// Initialize with static interface reference
+let interface: &'static Interface = /* your interface */;
+let mcp = MCP::new(interface);
+
+// Use as MCP server handler
+let server_info = mcp.get_info();
+println!("MCP server ready with {} tools", server_info.capabilities.tools.unwrap().len());
+
+// Individual tool usage
+let index_count = mcp.get_index_count().await?;
+let vec_count = mcp.get_vec_count().await?;
+let version = mcp.get_version().await?;
 ```
 
-**Server Info:**
-- Protocol version: Latest MCP specification
-- Capabilities: Tools enabled
-- Instructions: Provides context about Bitcoin data access
-- Stateless mode: Compatible with load balancers
+## API Overview
 
-## Error Handling
+### Core Structure
 
-The MCP server provides structured error responses following MCP specification:
-- Invalid parameters result in proper MCP error responses
-- Tool failures are handled gracefully
-- Parameter validation ensures type safety
+- **`MCP`**: Main server struct implementing ServerHandler trait
+- **Tool Router**: Automatic tool discovery and routing with `#[tool_router]` macro
+- **Parameters**: Type-safe tool parameters with validation
+- **CallToolResult**: Structured tool responses with success/error handling
 
-## Integration
+### Available Tools
 
-### With BRK Server
-MCP is exposed at the `/mcp` endpoint when enabled:
+**Discovery Tools:**
+
+- `get_index_count()` - Total number of available indexes
+- `get_vecid_count()` - Total number of vector identifiers
+- `get_vec_count()` - Total vector/index combinations
+- `get_indexes()` - List of all available indexes
+- `get_accepted_indexes()` - Mapping of indexes to their variants
+
+**Data Access Tools:**
+
+- `get_vecids(pagination)` - Paginated list of vector IDs
+- `get_index_to_vecids(index, pagination)` - Vector IDs supporting specific index
+- `get_vecid_to_indexes(id)` - Indexes supported by vector ID
+- `get_vecs(params)` - Main data query tool with flexible parameters
+
+**System Tools:**
+
+- `get_version()` - Bitcoin Research Kit version information
+
+### Tool Parameters
+
+**Core Query Parameters:**
+
+- `index`: Time dimension (height, date, week, month, etc.)
+- `ids`: Dataset identifiers (comma or space separated)
+- `from`/`to`: Range filtering (supports negative indexing from end)
+- `format`: Output format (json, csv)
+
+**Pagination Parameters:**
+
+- `page`: Page number for paginated results (up to 1,000 entries)
+
+## Examples
+
+### Basic Tool Usage
+
+```rust
+use brk_mcp::MCP;
+use brk_interface::{Params, PaginationParam};
+
+let mcp = MCP::new(interface);
+
+// Get system information
+let version = mcp.get_version().await?;
+println!("BRK version: {:?}", version);
+
+// Discover available data
+let indexes = mcp.get_indexes().await?;
+let vec_count = mcp.get_vec_count().await?;
+
+// Get paginated vector IDs
+let pagination = PaginationParam { page: Some(0) };
+let vecids = mcp.get_vecids(pagination).await?;
 ```
-POST /mcp
-Content-Type: application/json
 
-{
-  "jsonrpc": "2.0",
-  "method": "tools/call",
-  "params": {
-    "name": "get_vecs",
-    "arguments": {
-      "index": "height",
-      "ids": "timestamp,size",
-      "from": -1
-    }
-  }
+### Data Querying
+
+```rust
+use brk_mcp::MCP;
+use brk_interface::Params;
+
+// Query latest Bitcoin price
+let params = Params {
+    index: "date".to_string(),
+    ids: vec!["dateindex-to-price-close".to_string()].into(),
+    from: Some(-1),
+    ..Default::default()
+};
+
+let result = mcp.get_vecs(params).await?;
+match result {
+    CallToolResult::Success { content, .. } => {
+        println!("Latest price: {:?}", content);
+    },
+    _ => println!("Query failed"),
+}
+```
+
+### Multi-Vector Analysis
+
+```rust
+use brk_interface::Params;
+
+// Get OHLC data for last 30 days
+let params = Params {
+    index: "date".to_string(),
+    ids: vec![
+        "dateindex-to-price-open".to_string(),
+        "dateindex-to-price-high".to_string(),
+        "dateindex-to-price-low".to_string(),
+        "dateindex-to-price-close".to_string(),
+    ].into(),
+    from: Some(-30),
+    format: Some("csv".to_string()),
+    ..Default::default()
+};
+
+let ohlc_data = mcp.get_vecs(params).await?;
+```
+
+### Vector Discovery Workflow
+
+```rust
+use brk_interface::{PaginatedIndexParam, IdParam};
+
+// 1. Get available indexes
+let indexes = mcp.get_indexes().await?;
+
+// 2. Find vectors for specific index
+let paginated_index = PaginatedIndexParam {
+    index: "height".to_string(),
+    pagination: PaginationParam { page: Some(0) },
+};
+let height_vectors = mcp.get_index_to_vecids(paginated_index).await?;
+
+// 3. Check what indexes a vector supports
+let id_param = IdParam { id: "height-to-blockhash".to_string() };
+let supported_indexes = mcp.get_vecid_to_indexes(id_param).await?;
+```
+
+## Architecture
+
+### Server Implementation
+
+**MCP Protocol Compliance:**
+
+- Implements `ServerHandler` trait for MCP compatibility
+- Provides `ServerInfo` with tools capability enabled
+- Uses latest protocol version with proper initialization
+- Includes instructional context for LLM understanding
+
+**Tool Registration:**
+
+- `#[tool_router]` macro for automatic tool discovery
+- `#[tool]` attribute for individual tool definitions
+- Structured parameter types with automatic validation
+- Consistent error handling with `McpError` responses
+
+### HTTP Integration
+
+**Axum Router Extension:**
+
+- `MCPRoutes` trait for router integration
+- Conditional MCP endpoint mounting based on configuration
+- Stateless HTTP service with `LocalSessionManager`
+- Nested service mounting at `/mcp` path
+
+**Transport Layer:**
+
+- `StreamableHttpService` for MCP over HTTP
+- Configurable server options with stateless mode
+- Session management for concurrent connections
+- Request context handling for proper MCP operation
+
+### Data Integration
+
+**Interface Layer Access:**
+
+- Direct access to `brk_interface::Interface` for data queries
+- Static lifetime requirements for server operation
+- Unified access to indexer and computer data sources
+- Consistent parameter types across tool boundaries
+
+**Response Formatting:**
+
+- JSON content serialization for all tool responses
+- Success/error response wrapping with proper MCP structure
+- Logging integration for request tracking and debugging
+- Content type handling for different response formats
+
+## Configuration
+
+### Server Information
+
+The MCP server provides comprehensive information to LLMs:
+
+```rust
+ServerInfo {
+    protocol_version: ProtocolVersion::LATEST,
+    capabilities: ServerCapabilities::builder().enable_tools().build(),
+    server_info: Implementation::from_build_env(),
+    instructions: "Context about Bitcoin data access and terminology",
 }
 ```
 
 ### LLM Instructions
-The server provides context to LLMs:
-- Explains Bitcoin data terminology (vectors, indexes, vecids)
-- Clarifies that vectors/vecs/arrays/datasets are interchangeable terms
-- Describes indexes as timeframes and vecids as dataset names
-- Encourages exploration before web browsing
 
-## Dependencies
+Built-in instructions explain Bitcoin data concepts:
 
-- `brk_interface` - Data access and formatting layer
-- `brk_rmcp` - Rust MCP implementation
-- `axum` - HTTP router integration
-- `log` - Request logging
+- Vectors/vecs/arrays/datasets are interchangeable terms
+- Indexes represent timeframes for data organization
+- VecIds are dataset names describing their content
+- Tool-based exploration before web browsing is encouraged
+
+## Code Analysis Summary
+
+**Main Structure**: `MCP` struct implementing `ServerHandler` with embedded `ToolRouter` for automatic tool discovery \
+**Tool Implementation**: 10 specialized tools using `#[tool]` attribute with structured parameters and JSON responses \
+**HTTP Integration**: `MCPRoutes` trait extending Axum routers with conditional MCP endpoint mounting \
+**Parameter Types**: Type-safe parameter structs from `brk_interface` with automatic validation \
+**Error Handling**: Consistent `McpError` responses with proper MCP protocol compliance \
+**Transport Layer**: `StreamableHttpService` with stateless configuration for scalable deployment \
+**Architecture**: Standards-compliant MCP bridge providing LLM access to comprehensive Bitcoin analytics
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_mcp/src/lib.rs

@@ -133,9 +133,12 @@ The response's format will depend on the given parameters, it will be:
 ")]
     fn get_vecs(&self, Parameters(params): Parameters<Params>) -> Result<CallToolResult, McpError> {
         info!("mcp: get_vecs");
-        Ok(CallToolResult::success(vec![
-            Content::json(self.interface.search_and_format(params).unwrap()).unwrap(),
-        ]))
+        Ok(CallToolResult::success(vec![Content::text(
+            match self.interface.search_and_format(params) {
+                Ok(output) => output.to_string(),
+                Err(e) => format!("Error:\n{e}"),
+            },
+        )]))
     }
 
     #[tool(description = "

crates/brk_parser/Cargo.toml

@@ -14,10 +14,9 @@ build = "build.rs"
 [dependencies]
 bitcoin = { workspace = true }
 bitcoincore-rpc = { workspace = true }
+brk_error = { workspace = true }
 brk_structs = { workspace = true }
 crossbeam = { version = "0.8.4", features = ["crossbeam-channel"] }
 derive_deref = { workspace = true }
+parking_lot = { workspace = true }
 rayon = { workspace = true }
-serde = { workspace = true }
-serde_json = { workspace = true }
-zerocopy = { workspace = true }

crates/brk_parser/README.md

@@ -1,153 +1,199 @@
 # brk_parser
 
-**High-performance Bitcoin block parser for raw Bitcoin Core block files**
+High-performance Bitcoin block parser for raw Bitcoin Core block files with XOR encryption support.
 
-`brk_parser` provides efficient sequential access to Bitcoin Core's raw block files (`blkXXXXX.dat`), delivering blocks in height order with automatic fork filtering and XOR encryption support. Built for blockchain analysis and indexing applications that need complete Bitcoin data access.
+[![Crates.io](https://img.shields.io/crates/v/brk_parser.svg)](https://crates.io/crates/brk_parser)
+[![Documentation](https://docs.rs/brk_parser/badge.svg)](https://docs.rs/brk_parser)
 
-## What it provides
+## Overview
 
-- **Sequential block access**: Blocks delivered in height order (0, 1, 2, ...) regardless of physical file storage
-- **Fork filtering**: Automatically excludes orphaned blocks using Bitcoin Core RPC verification
-- **XOR encryption support**: Transparently handles XOR-encrypted block files 
-- **High performance**: Multi-threaded parsing with ~500MB peak memory usage
-- **State persistence**: Caches parsing state for fast restarts
+This crate provides a multi-threaded Bitcoin block parser that processes raw Bitcoin Core `.dat` files from the blockchain directory. It supports XOR-encoded block data, parallel processing with `rayon`, and maintains chronological ordering through crossbeam channels. The parser integrates with Bitcoin Core RPC to validate block confirmations and handles file metadata tracking for incremental processing.
 
-## Key Features
+**Key Features:**
 
-### Performance Optimization
-- **Multi-threaded pipeline**: 3-stage processing (file reading, decoding, ordering)
-- **Parallel decoding**: Uses rayon for concurrent block deserialization
-- **Memory efficient**: Bounded channels prevent memory bloat
-- **State caching**: Saves parsing state to avoid re-scanning unchanged files
+- Multi-threaded pipeline architecture with crossbeam channels
+- XOR decryption support for encrypted block files
+- Parallel block decoding with rayon thread pools
+- Chronological block ordering with height-based validation
+- Bitcoin Core RPC integration for confirmation checking
+- File metadata tracking and incremental processing
+- Magic byte detection for block boundary identification
 
-### Bitcoin Integration
-- **RPC verification**: Uses Bitcoin Core RPC to filter orphaned blocks
-- **Confirmation checks**: Only processes blocks with positive confirmations
-- **Height ordering**: Ensures sequential delivery regardless of storage order
+**Target Use Cases:**
 
-### XOR Encryption Support
-- **Transparent decryption**: Automatically handles XOR-encrypted block files
-- **Streaming processing**: Applies XOR decryption on-the-fly during parsing
+- Bitcoin blockchain analysis tools requiring raw block access
+- Historical data processing applications
+- Block explorers and analytics platforms
+- Research tools needing ordered block iteration
 
-## Usage
+## Installation
 
-### Basic Block Parsing
+```bash
+cargo add brk_parser
+```
+
+## Quick Start
 
 ```rust
 use brk_parser::Parser;
+use bitcoincore_rpc::{Client, Auth, RpcApi};
 use brk_structs::Height;
-use bitcoincore_rpc::{Auth, Client};
+use std::path::PathBuf;
 
-// Setup RPC client (must have static lifetime)
+// Initialize Bitcoin Core RPC client
 let rpc = Box::leak(Box::new(Client::new(
     "http://localhost:8332",
-    Auth::CookieFile(Path::new("~/.bitcoin/.cookie")),
-)?));
+    Auth::None
+).unwrap()));
 
-// Create parser
-let parser = Parser::new(
-    Path::new("~/.bitcoin/blocks").to_path_buf(),
-    Path::new("./output").to_path_buf(),
-    rpc,
-);
+// Create parser with blocks directory
+let blocks_dir = PathBuf::from("/path/to/bitcoin/blocks");
+let outputs_dir = Some(PathBuf::from("./parser_output"));
+let parser = Parser::new(blocks_dir, outputs_dir, rpc);
 
-// Parse all blocks sequentially
-parser.parse(None, None)
-    .iter()
-    .for_each(|(height, block, hash)| {
-        println!("Block {}: {} ({} txs)", height, hash, block.txdata.len());
-    });
-```
+// Parse blocks in height range
+let start_height = Some(Height::new(700000));
+let end_height = Some(Height::new(700100));
+let receiver = parser.parse(start_height, end_height);
 
-### Range Parsing
-
-```rust
-// Parse specific height range
-let start = Some(Height::new(800_000));
-let end = Some(Height::new(800_100));
-
-parser.parse(start, end)
-    .iter()
-    .for_each(|(height, block, hash)| {
-        // Process blocks 800,000 to 800,100
-    });
-```
-
-### Single Block Access
-
-```rust
-// Get single block by height
-let genesis = parser.get(Height::new(0));
-println!("Genesis has {} transactions", genesis.txdata.len());
-```
-
-### Real-world Usage Example
-
-```rust
-use brk_parser::Parser;
-use bitcoin::Block;
-
-fn analyze_blockchain(parser: &Parser) {
-    let mut total_transactions = 0;
-    let mut total_outputs = 0;
-    
-    parser.parse(None, None)
-        .iter()
-        .for_each(|(height, block, _hash)| {
-            total_transactions += block.txdata.len();
-            total_outputs += block.txdata.iter()
-                .map(|tx| tx.output.len())
-                .sum::<usize>();
-                
-            if height.0 % 10000 == 0 {
-                println!("Processed {} blocks", height);
-            }
-        });
-        
-    println!("Total transactions: {}", total_transactions);
-    println!("Total outputs: {}", total_outputs);
+// Process blocks as they arrive
+for (height, block, block_hash) in receiver.iter() {
+    println!("Block {}: {} transactions", height, block.txdata.len());
+    println!("Block hash: {}", block_hash);
 }
 ```
 
-## Output Format
+## API Overview
 
-The parser returns tuples for each block:
-- `Height`: Block height (sequential: 0, 1, 2, ...)
-- `Block`: Complete block data from the `bitcoin` crate
-- `BlockHash`: Block's cryptographic hash
+### Core Types
 
-## Performance Characteristics
+- **`Parser`**: Main parser coordinating multi-threaded block processing
+- **`AnyBlock`**: Enum representing different block states (Raw, Decoded, Skipped)
+- **`XORBytes`**: XOR key bytes for decrypting block data
+- **`XORIndex`**: Circular index for XOR byte application
+- **`BlkMetadata`**: Block file metadata including index and modification time
 
-Benchmarked on MacBook Pro M3 Pro:
-- **Full blockchain** (0 to 855,000): ~4 minutes
-- **Recent blocks** (800,000 to 855,000): ~52 seconds
-- **Peak memory usage**: ~500MB
-- **Restart performance**: Subsequent runs much faster due to state caching
+### Key Methods
 
-## Requirements
+**`Parser::new(blocks_dir: PathBuf, outputs_dir: Option<PathBuf>, rpc: &'static Client) -> Self`**
+Creates a new parser instance with blockchain directory and RPC client.
 
-- Running Bitcoin Core node with RPC enabled
-- Access to Bitcoin Core's `blocks/` directory
-- Bitcoin Core versions v25.0 through v29.0 supported
-- RPC authentication (cookie file or username/password)
+**`parse(&self, start: Option<Height>, end: Option<Height>) -> Receiver<(Height, Block, BlockHash)>`**
+Returns a channel receiver that yields blocks in chronological order for the specified height range.
 
-## State Management
+### Processing Pipeline
 
-The parser saves parsing state in `{output_dir}/blk_index_to_blk_recap.json` containing:
-- Block file indices and maximum heights
-- File modification times for change detection
-- Restart optimization metadata
+The parser implements a three-stage pipeline:
 
-**Note**: Only one parser instance should run at a time as the state file doesn't support concurrent access.
+1. **File Reading Stage**: Scans `.dat` files, identifies magic bytes, extracts raw block data
+2. **Decoding Stage**: Parallel XOR decryption and Bitcoin block deserialization
+3. **Ordering Stage**: RPC validation and chronological ordering by block height
 
-## Dependencies
+## Examples
 
-- `bitcoin` - Bitcoin protocol types and block parsing
-- `bitcoincore_rpc` - RPC communication with Bitcoin Core
-- `crossbeam` - Multi-producer, multi-consumer channels
-- `rayon` - Data parallelism for block decoding
-- `serde` - State serialization and persistence
+### Basic Block Iteration
+
+```rust
+use brk_parser::Parser;
+
+let parser = Parser::new(blocks_dir, Some(output_dir), rpc);
+
+// Parse all blocks from height 650000 onwards
+let receiver = parser.parse(Some(Height::new(650000)), None);
+
+for (height, block, hash) in receiver.iter() {
+    println!("Processing block {} with {} transactions",
+             height, block.txdata.len());
+
+    // Process block transactions
+    for (idx, tx) in block.txdata.iter().enumerate() {
+        println!("  Tx {}: {}", idx, tx.txid());
+    }
+}
+```
+
+### Range-Based Processing
+
+```rust
+use brk_parser::Parser;
+
+let parser = Parser::new(blocks_dir, Some(output_dir), rpc);
+
+// Process specific block range
+let start = Height::new(600000);
+let end = Height::new(600999);
+let receiver = parser.parse(Some(start), Some(end));
+
+let mut total_tx_count = 0;
+for (height, block, _hash) in receiver.iter() {
+    total_tx_count += block.txdata.len();
+
+    if height == end {
+        break; // End of range reached
+    }
+}
+
+println!("Processed 1000 blocks with {} total transactions", total_tx_count);
+```
+
+### Incremental Processing with Metadata
+
+```rust
+use brk_parser::Parser;
+
+let parser = Parser::new(blocks_dir, Some(output_dir), rpc);
+
+// Parser automatically handles file metadata tracking
+// Only processes blocks that have been modified since last run
+let receiver = parser.parse(None, None); // Process all available blocks
+
+for (height, block, hash) in receiver.iter() {
+    // Parser ensures blocks are delivered in chronological order
+    // even when processing multiple .dat files in parallel
+
+    if height.as_u32() % 10000 == 0 {
+        println!("Reached block height {}", height);
+    }
+}
+```
+
+## Architecture
+
+### Multi-Threading Design
+
+The parser uses a sophisticated multi-threaded architecture:
+
+- **File Scanner Thread**: Reads raw bytes from `.dat` files and identifies block boundaries
+- **Decoder Thread Pool**: Parallel XOR decryption and block deserialization using rayon
+- **Ordering Thread**: RPC validation and chronological ordering with future block buffering
+
+### XOR Encryption Support
+
+Bitcoin Core optionally XOR-encrypts block files using an 8-byte key stored in `xor.dat`. The parser:
+
+- Automatically detects XOR encryption presence
+- Implements circular XOR index for efficient decryption
+- Supports both encrypted and unencrypted block files
+
+### Block File Management
+
+The parser handles Bitcoin Core's block file structure:
+
+- Scans directory for `blk*.dat` files
+- Tracks file modification times for incremental processing
+- Maintains block height mappings with RPC validation
+- Exports processing metadata for resumable operations
+
+## Code Analysis Summary
+
+**Main Type**: `Parser` struct coordinating multi-threaded block processing pipeline \
+**Threading**: Three-stage pipeline using crossbeam channels with bounded capacity (50) \
+**Parallelization**: rayon-based parallel block decoding with configurable batch sizes \
+**XOR Handling**: Custom XORBytes and XORIndex types for efficient encryption/decryption \
+**RPC Integration**: Bitcoin Core RPC validation for block confirmation and height mapping \
+**File Processing**: Automatic `.dat` file discovery and magic byte boundary detection \
+**Architecture**: Producer-consumer pattern with ordered delivery despite parallel processing
 
 ---
 
-*This README was generated by Claude Code*
+_This README was generated by Claude Code_

crates/brk_parser/examples/main.rs

@@ -1,60 +1,51 @@
-use std::path::Path;
+use std::{collections::BTreeMap, path::Path};
 
-use bitcoincore_rpc::{Auth, Client, Result};
+use bitcoincore_rpc::{Auth, Client};
+use brk_error::Result;
 use brk_parser::Parser;
-use brk_structs::Height;
 
 #[allow(clippy::needless_doctest_main)]
 fn main() -> Result<()> {
     let i = std::time::Instant::now();
 
-    let bitcoin_dir = Path::new("").join("");
-    let brk_dir = Path::new("").join("");
+    let bitcoin_dir = Path::new(&std::env::var("HOME").unwrap())
+        .join("Library")
+        .join("Application Support")
+        .join("Bitcoin");
 
     let rpc = Box::leak(Box::new(Client::new(
         "http://localhost:8332",
         Auth::CookieFile(bitcoin_dir.join(".cookie")),
     )?));
 
-    let start = None;
+    let parser = Parser::new(bitcoin_dir.join("blocks"), rpc);
+
+    let start = Some(915_138_u32.into());
     let end = None;
+    let mut blk_index = 0;
+    let mut diff = BTreeMap::new();
+    parser.parse(start, end).iter().for_each(|block| {
+        println!("{}: {}", block.height(), block.hash());
+        let new_blk_index = block.metadata().blk_index();
+        if new_blk_index < blk_index {
+            diff.insert(blk_index - new_blk_index, block.height());
+        }
+        blk_index = new_blk_index;
+    });
 
-    let parser = Parser::new(bitcoin_dir.join("blocks"), brk_dir, rpc);
+    // let v = diff.iter().rev().take(10).collect::<Vec<_>>();
 
-    parser
-        .parse(start, end)
-        .iter()
-        .for_each(|(height, _block, hash)| {
-            println!("{height}: {hash}");
-        });
+    // let block_0 = parser.get(Height::new(0))?;
+    // dbg!("{}", block_0.coinbase_tag());
 
-    let block_0 = parser.get(Height::new(0));
+    // let block_158251 = parser.get(Height::new(158251))?;
+    // dbg!("{}", block_158251.coinbase_tag());
 
-    println!(
-        "{}",
-        block_0
-            .txdata
-            .first()
-            .unwrap()
-            .output
-            .first()
-            .unwrap()
-            .script_pubkey
-    );
+    // let block_173195 = parser.get(Height::new(173195))?;
+    // dbg!("{}", block_173195.coinbase_tag());
 
-    let block_840_000 = parser.get(Height::new(840_000));
-
-    println!(
-        "{}",
-        block_840_000
-            .txdata
-            .first()
-            .unwrap()
-            .output
-            .first()
-            .unwrap()
-            .value
-    );
+    // let block_840_000 = parser.get(Height::new(840_004))?;
+    // dbg!("{}", block_840_000.coinbase_tag());
 
     dbg!(i.elapsed());
 

crates/brk_parser/examples/p2a.rs

@@ -8,7 +8,6 @@ fn main() {
     let i = std::time::Instant::now();
 
     let bitcoin_dir = Path::new("").join("");
-    let brk_dir = Path::new("").join("");
 
     let rpc = Box::leak(Box::new(
         Client::new(
@@ -21,7 +20,7 @@ fn main() {
     // let start = None;
     // let end = None;
 
-    let parser = Parser::new(bitcoin_dir.join("blocks"), brk_dir, rpc);
+    let parser = Parser::new(bitcoin_dir.join("blocks"), rpc);
 
     // parser
     //     .parse(start, end)
@@ -43,7 +42,7 @@ fn main() {
     //         .script_pubkey
     // );
 
-    let block_850_000 = parser.get(Height::new(850_000));
+    let block_850_000 = parser.get(Height::new(850_000)).unwrap();
 
     let tx = block_850_000.txdata.iter().find(|tx| {
         tx.compute_txid().to_string()

crates/brk_parser/src/any_block.rs

@@ -0,0 +1,74 @@
+use bitcoin::{Transaction, VarInt, block::Header, consensus::Decodable, io::Cursor};
+use bitcoincore_rpc::RpcApi;
+use brk_error::Result;
+use brk_structs::{BlkMetadata, Block, Height, ParsedBlock};
+
+use crate::{XORBytes, XORIndex};
+
+pub enum AnyBlock {
+    Raw(Vec<u8>),
+    Decoded(ParsedBlock),
+    Skipped,
+}
+
+impl AnyBlock {
+    pub fn decode(
+        self,
+        metadata: BlkMetadata,
+        rpc: &'static bitcoincore_rpc::Client,
+        mut xor_i: XORIndex,
+        xor_bytes: XORBytes,
+        start: Option<Height>,
+        end: Option<Height>,
+    ) -> Result<Self> {
+        let mut bytes = match self {
+            AnyBlock::Raw(bytes) => bytes,
+            _ => unreachable!(),
+        };
+
+        xor_i.bytes(bytes.as_mut_slice(), xor_bytes);
+
+        let mut cursor = Cursor::new(bytes);
+
+        let header = Header::consensus_decode(&mut cursor)?;
+
+        let hash = header.block_hash();
+
+        let tx_count = VarInt::consensus_decode(&mut cursor)?.0;
+
+        let Ok(block_header_result) = rpc.get_block_header_info(&hash) else {
+            return Ok(Self::Skipped);
+        };
+
+        let height = Height::from(block_header_result.height);
+
+        if let Some(start) = start
+            && start > height
+        {
+            return Ok(Self::Skipped);
+        } else if let Some(end) = end
+            && end < height
+        {
+            return Ok(Self::Skipped);
+        } else if block_header_result.confirmations <= 0 {
+            return Ok(Self::Skipped);
+        }
+
+        let mut txdata = Vec::with_capacity(tx_count as usize);
+        let mut tx_metadata = Vec::with_capacity(tx_count as usize);
+        for _ in 0..tx_count {
+            let offset = cursor.position() as u32;
+            let position = metadata.position() + offset;
+            let tx = Transaction::consensus_decode(&mut cursor)?;
+            txdata.push(tx);
+            let len = cursor.position() as u32 - offset;
+            tx_metadata.push(BlkMetadata::new(position, len));
+        }
+
+        let block = bitcoin::Block { header, txdata };
+        let block = Block::from((height, hash, block));
+        let block = ParsedBlock::from((block, metadata, tx_metadata));
+
+        Ok(Self::Decoded(block))
+    }
+}

crates/brk_parser/src/blk_index_to_blk_path.rs

@@ -7,9 +7,9 @@ use std::{
 use derive_deref::{Deref, DerefMut};
 
 const BLK: &str = "blk";
-const DAT: &str = ".dat";
+const DOT_DAT: &str = ".dat";
 
-#[derive(Debug, Deref, DerefMut)]
+#[derive(Debug, Clone, Deref, DerefMut)]
 pub struct BlkIndexToBlkPath(BTreeMap<u16, PathBuf>);
 
 impl BlkIndexToBlkPath {
@@ -24,7 +24,7 @@ impl BlkIndexToBlkPath {
                     if is_file {
                         let file_name = path.file_name().unwrap().to_str().unwrap();
 
-                        file_name.starts_with(BLK) && file_name.ends_with(DAT)
+                        file_name.starts_with(BLK) && file_name.ends_with(DOT_DAT)
                     } else {
                         false
                     }
@@ -32,7 +32,7 @@ impl BlkIndexToBlkPath {
                 .map(|path| {
                     let file_name = path.file_name().unwrap().to_str().unwrap();
 
-                    let blk_index = file_name[BLK.len()..(file_name.len() - DAT.len())]
+                    let blk_index = file_name[BLK.len()..(file_name.len() - DOT_DAT.len())]
                         .parse::<u16>()
                         .unwrap();
 

crates/brk_parser/src/blk_index_to_blk_recap.rs

@@ -1,107 +0,0 @@
-use std::{
-    collections::{BTreeMap, BTreeSet},
-    fs::File,
-    io::{BufReader, BufWriter},
-    path::{Path, PathBuf},
-};
-
-use crate::{BlkIndexToBlkPath, Height, blk_recap::BlkRecap};
-
-#[derive(Debug)]
-pub struct BlkIndexToBlkRecap {
-    pub path: PathBuf,
-    pub tree: BTreeMap<u16, BlkRecap>,
-}
-
-impl BlkIndexToBlkRecap {
-    pub fn import(
-        outputs_dir: &Path,
-        blk_index_to_blk_path: &BlkIndexToBlkPath,
-        start: Option<Height>,
-    ) -> (Self, u16) {
-        let path = outputs_dir.join("blk_index_to_blk_recap.json");
-
-        let tree = {
-            if let Ok(file) = File::open(&path) {
-                let reader = BufReader::new(file);
-                serde_json::from_reader(reader).unwrap_or_default()
-            } else {
-                BTreeMap::default()
-            }
-        };
-
-        let mut slf = Self { path, tree };
-
-        let min_removed = slf.clean_outdated(blk_index_to_blk_path);
-
-        let blk_index = slf.get_start_recap(min_removed, start);
-
-        (slf, blk_index)
-    }
-
-    fn clean_outdated(&mut self, blk_index_to_blk_path: &BlkIndexToBlkPath) -> Option<u16> {
-        let mut min_removed_blk_index: Option<u16> = None;
-
-        let mut unprocessed_keys = self.tree.keys().copied().collect::<BTreeSet<_>>();
-
-        blk_index_to_blk_path
-            .iter()
-            .for_each(|(blk_index, blk_path)| {
-                unprocessed_keys.remove(blk_index);
-                if let Some(blk_recap) = self.tree.get(blk_index)
-                    && blk_recap.has_different_modified_time(blk_path)
-                {
-                    self.tree.remove(blk_index).unwrap();
-                    if min_removed_blk_index.is_none_or(|_blk_index| *blk_index < _blk_index) {
-                        min_removed_blk_index.replace(*blk_index);
-                    }
-                }
-            });
-
-        unprocessed_keys.into_iter().for_each(|blk_index| {
-            self.tree.remove(&blk_index).unwrap();
-            if min_removed_blk_index.is_none_or(|_blk_index| blk_index < _blk_index) {
-                min_removed_blk_index.replace(blk_index);
-            }
-        });
-
-        min_removed_blk_index
-    }
-
-    pub fn get_start_recap(&mut self, min_removed: Option<u16>, start: Option<Height>) -> u16 {
-        if start.is_none() {
-            return 0;
-        }
-
-        let height = start.unwrap();
-
-        let mut start = None;
-
-        if let Some(found) = self
-            .tree
-            .iter()
-            .find(|(_, recap)| recap.max_height >= height)
-        {
-            start = Some(*found.0);
-        }
-
-        if let Some(min_removed) = min_removed
-            && start.is_none_or(|start| start > min_removed)
-        {
-            start = Some(min_removed);
-        }
-
-        // Should only be none if asking for a too high start
-        start.unwrap_or_else(|| self.tree.last_key_value().map_or(0, |(i, _)| *i))
-    }
-
-    pub fn export(&self) {
-        let file = File::create(&self.path).unwrap_or_else(|e| {
-            dbg!(e);
-            dbg!(&self.path);
-            panic!("Cannot write file");
-        });
-
-        serde_json::to_writer(&mut BufWriter::new(file), &self.tree).unwrap();
-    }
-}

crates/brk_parser/src/blk_metadata.rs

@@ -1,18 +0,0 @@
-use std::path::Path;
-
-use crate::path_to_modified_time;
-
-#[derive(Debug, Clone, Copy)]
-pub struct BlkMetadata {
-    pub index: u16,
-    pub modified_time: u64,
-}
-
-impl BlkMetadata {
-    pub fn new(index: u16, path: &Path) -> Self {
-        Self {
-            index,
-            modified_time: path_to_modified_time(path),
-        }
-    }
-}

crates/brk_parser/src/blk_recap.rs

@@ -1,19 +0,0 @@
-use std::path::Path;
-
-use brk_structs::Height;
-use serde::{Deserialize, Serialize};
-
-use crate::path_to_modified_time;
-
-#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
-#[repr(C)]
-pub struct BlkRecap {
-    pub max_height: Height,
-    pub modified_time: u64,
-}
-
-impl BlkRecap {
-    pub fn has_different_modified_time(&self, blk_path: &Path) -> bool {
-        self.modified_time != path_to_modified_time(blk_path)
-    }
-}