release: v0.0.110

Add Docker support

.claude/prompts/changelog.md

@@ -0,0 +1,221 @@
+# Changelog Generation for Claude Code
+
+**TASK**: Update docs/CHANGELOG.md for ALL releases after and including v0.0.107.
+
+## ⚠️ CRITICAL FAILURE MODE TO AVOID ⚠️
+**THE #1 FAILURE**: Ignoring most changes and only documenting a few
+**ABSOLUTELY FORBIDDEN**: Skipping changes, summarizing with "and other updates", or being incomplete
+**YOU MUST DOCUMENT EVERY SINGLE MEANINGFUL CHANGE - NO EXCEPTIONS**
+
+## CORE WORKFLOW - EXECUTE EXACTLY:
+
+### Step 1: Get Release Information
+```bash
+git tag --list --sort=version:refname
+```
+
+### Step 2: Process ONE Release at a Time
+For each missing release, execute these commands to get complete information:
+
+**First, get the file list:**
+```bash
+git diff --name-only [previous-tag]..[current-tag]
+```
+
+**Then, get changes excluding Cargo.lock:**
+```bash
+git diff [previous-tag]..[current-tag] -- . ':(exclude)Cargo.lock'
+```
+
+**If output is too large, examine files individually:**
+```bash
+git diff [previous-tag]..[current-tag] -- path/to/specific/file.rs
+```
+
+**For summary of changes per file (if needed):**
+```bash
+git diff --stat [previous-tag]..[current-tag]
+```
+
+### Step 3: Analyze Before Writing
+**MANDATORY**: Before writing ANY changelog entry, analyze the diff output and explain:
+- **Use `git diff --name-only` to see ALL changed files** - this prevents truncation issues
+- **Use `git diff -- . ':(exclude)Cargo.lock'` to see actual changes** without Cargo.lock noise
+- **If output is large, examine key files individually** with `git diff [tags] -- path/to/file`
+- **Identify every functional change** - what new capabilities, fixes, or modifications were made
+- What each code change accomplishes functionally
+- The user-facing or system impact of modifications
+- **Which crate each change belongs to** (based on file paths)
+- How changes group together logically within and across crates
+- What functionality is added/removed/modified per crate
+
+**COMPLETENESS CHECK**: State "I have analyzed X files and identified Y distinct functional changes to document"
+
+### Step 4: Write Changelog Entry
+Only after analysis, update CHANGELOG.md with ONE release entry.
+
+**REQUIRED**: End each release entry with a comparison link:
+```markdown
+[View changes](https://github.com/bitcoinresearchkit/brk/compare/vPREVIOUS...vCURRENT)
+```
+Where PREVIOUS is the previous release tag and CURRENT is the current release tag.
+
+### Step 5: Stop and Confirm
+**CRITICAL**: Process only ONE release, then ask for confirmation to continue.
+
+---
+
+## STRICT RULES
+
+### FORBIDDEN - NEVER MENTION:
+- **Cargo.lock** (ignore completely)
+- **Line counts** ("added 50 lines", "removed 200 lines")
+- **Dependency version bumps** (unless enabling major new features)
+- **Local crate version changes** (0.0.61 → 0.0.62)
+- **Vague words**: "Enhanced", "Improved", "Updated", "Refactored"
+
+### FORBIDDEN PHRASES:
+- "and other changes"
+- "various updates"
+- "along with minor improvements"
+- "among other enhancements"
+- **"along with other modifications"**
+- **"plus additional changes"**
+- **"including other updates"**
+- **"and more"**
+- **ANY phrase that suggests you're skipping changes**
+
+### REQUIRED - MUST INCLUDE:
+- **Every meaningful change** in the diff (no shortcuts)
+- **Specific functionality** descriptions
+- **Business impact** of each change
+- **Complete coverage** - if 20 changes exist, document all 20
+- **Source links** for significant changes (new features, breaking changes, major bug fixes)
+
+## WORKSPACE-SPECIFIC RULES
+
+### Crate Identification:
+- **Determine crate from file paths** in the diff (e.g., `crates/brk-core/src/lib.rs` → `brk-core`)
+- **Group all changes by their crate** before writing changelog entries
+- **Use crate name as subheading** under each change type section
+- **For root-level files**: Use `workspace` as the crate name
+
+### Cross-Crate Changes:
+- **When changes span multiple crates** for one feature, mention the relationship
+- **Example**: "Added new transaction API in `brk-core` with corresponding HTTP endpoints in `brk-api`"
+
+### Crate Naming:
+- **Use backticks** around crate names: `brk-core`, `brk-api`
+- **Use workspace structure** as shown in file paths, not display names
+
+### File Header (if missing):
+```markdown
+<!-- This changelog was generated by Claude Code -->
+```
+
+### Release Entry Format:
+```markdown
+## [vX.Y.Z](https://github.com/bitcoinresearchkit/brk/releases/tag/vX.Y.Z) - YYYY-MM-DD
+
+### Breaking Changes
+#### `crate-name`
+- Specific change with functional impact explanation ([source](https://github.com/bitcoinresearchkit/brk/blob/vX.Y.Z/path/to/file.rs))
+
+### New Features
+#### `crate-name`
+- Feature description with user benefit ([source](https://github.com/bitcoinresearchkit/brk/blob/vX.Y.Z/path/to/main/file.rs))
+- Another feature with implementation details
+
+#### `another-crate`
+- Feature specific to this crate
+
+### Bug Fixes
+#### `crate-name`
+- Specific problem that was resolved ([source](https://github.com/bitcoinresearchkit/brk/blob/vX.Y.Z/path/to/file.rs))
+- Another bug fix with impact description
+
+### Internal Changes
+#### `crate-name`
+- Architectural improvement with purpose
+- Code organization change with benefit
+
+[View changes](https://github.com/bitcoinresearchkit/brk/compare/vPREVIOUS...vCURRENT)
+```
+
+---
+
+## ANALYSIS REQUIREMENTS
+
+**Before writing changelog text, you MUST state:**
+
+1. **File discovery**: "Using git diff --name-only, I see X files were modified"
+2. **Cargo.lock check**: "After excluding Cargo.lock, Y files contain actual changes"
+3. **Functional changes identified**: "I identified these distinct changes: A, B, C, D..."
+4. **What each change does**: "Change A: adds new API endpoints, Change B: fixes memory leak, etc."
+5. **Which crates are affected**: "Changes span crates: A, B, C"
+6. **What this means for users**: "Users can now do X, bug Y is fixed, etc."
+7. **How changes group together**: "Changes A and B work together to implement feature W"
+8. **Cross-crate dependencies**: "Crate A's new feature requires the new API in crate B"
+9. **Any unclear changes**: "I cannot determine the purpose of change X from the diff"
+10. **FINAL CHECK**: "I will now document all X functional changes without listing files"
+
+**MANDATORY**: The changelog should describe FUNCTIONALITY, not files. But you must capture ALL functionality changes.
+
+**Only after this analysis, write the changelog entry.**
+
+---
+
+## EXAMPLES
+
+### ❌ BAD (Vague, incomplete):
+```markdown
+### New Features
+- Enhanced blockchain processing capabilities
+- Improved error handling and various other updates
+```
+
+### ✅ GOOD (Specific, complete, grouped by crate, with source links):
+```markdown
+### New Features
+#### `brk-core`
+- Added `TransactionAnalyzer` struct with fee calculation and coinbase detection methods ([source](https://github.com/bitcoinresearchkit/brk/blob/v0.0.108/crates/brk-core/src/analyzer.rs))
+- Implemented in-memory caching layer for blockchain queries using HashMap storage
+
+#### `brk-api`
+- Added three new API endpoints: `/api/blocks/{hash}`, `/api/transactions/search`, and `/api/stats/network` ([source](https://github.com/bitcoinresearchkit/brk/blob/v0.0.108/crates/brk-api/src/routes.rs))
+- Implemented standardized error responses with error codes and descriptions
+
+### Bug Fixes
+#### `brk-core`
+- Fixed panic when processing blocks with zero transactions by adding explicit empty block validation ([source](https://github.com/bitcoinresearchkit/brk/blob/v0.0.108/crates/brk-core/src/block.rs))
+
+#### `brk-api`
+- Resolved memory leak in connection pool by implementing proper cleanup in Drop trait
+
+[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.107...v0.0.108)
+```
+
+---
+
+## SUCCESS CRITERIA
+
+✅ **COUNT files modified for internal verification only**
+✅ **Identify EVERY distinct functional change**
+✅ **Document ALL functionality changes (no "other changes")**
+✅ **Changelog describes WHAT users can do, not which files changed**
+✅ **Never mention Cargo.lock or line counts**
+✅ **Use specific, functional descriptions**
+✅ **Complete analysis before writing**
+✅ **Stop and ask for confirmation after each release**
+
+**FAILURE INDICATORS - If you do any of these, you FAILED:**
+❌ "and other changes"
+❌ "various updates"
+❌ "among other improvements"
+❌ **Diff shows new API endpoints but changelog doesn't mention them**
+❌ **Diff shows bug fixes but changelog misses some**
+❌ **Diff shows new structs/functions but changelog ignores them**
+❌ Missing obvious functional changes from the diff
+❌ Summarizing instead of listing each distinct functionality change
+
+**KEY PRINCIPLE**: Count files internally to ensure you don't miss changes, but write about FUNCTIONALITY for users.

.claude/prompts/readmes.md

@@ -0,0 +1,172 @@
+# README Generation Prompt
+
+Generate a professional, comprehensive README.md for each crate based SOLELY on code analysis. Use NO external documentation, commit messages, or existing READMEs.
+
+## MANDATORY PROCESS - FOLLOW EXACTLY:
+1. **IGNORE EXISTING DOCS**: Do NOT read any .md, .txt, .rst, or documentation files in the crate directory
+2. **CODE-ONLY ANALYSIS**: Examine ONLY these files:
+   - All .rs files in src/ directory and subdirectories
+   - Cargo.toml for dependencies and metadata
+   - Code structure and organization
+3. **MANDATORY CODE ANALYSIS**: Before writing ANY README content, you MUST:
+   - Examine all Rust files in src/ directory
+   - Identify the main structs, enums, traits, and functions
+   - Understand the crate's architecture and data flow
+   - Determine the crate's purpose from its implementation
+   - Map dependencies to understand external integrations
+4. **FRESH PERSPECTIVE**: Write the README as if you're the first person to document this crate
+5. Generate one complete README.md per crate
+6. Focus on one crate at a time for thorough analysis
+
+## ABSOLUTE REQUIREMENTS:
+- **SOURCE OF TRUTH**: Use ONLY the actual Rust code - no external docs, comments may provide hints but focus on implementation
+- **CRITICAL**: DO NOT read any existing README.md, CHANGELOG.md, or documentation files
+- **IGNORE ALL TEXT FILES**: .md, .txt, .rst files are FORBIDDEN sources - treat them as if they don't exist
+- **CODE ONLY**: Focus exclusively on .rs files, Cargo.toml, and code structure
+- **PROFESSIONAL GRADE**: Write as if this will be published on crates.io for other developers
+- **PROGRAMMER FOCUSED**: Assume audience knows Rust and relevant domain concepts
+- **IMPLEMENTATION-BASED**: Describe what the code actually does, not what comments claim it should do
+- **If you cannot determine functionality from code alone, state this explicitly**
+
+## README STRUCTURE (MANDATORY):
+
+### 1. CRATE HEADER
+```markdown
+# Crate Name
+
+Brief one-line description of what this crate does (max 80 chars).
+
+[![Crates.io](https://img.shields.io/crates/v/CRATE_NAME.svg)](https://crates.io/crates/CRATE_NAME)
+[![Documentation](https://docs.rs/CRATE_NAME/badge.svg)](https://docs.rs/CRATE_NAME)
+```
+
+### 2. OVERVIEW SECTION
+- **Purpose**: What problem does this crate solve?
+- **Key Features**: 3-5 bullet points of main capabilities (derived from code analysis)
+- **Target Use Cases**: Who would use this and for what?
+
+### 3. INSTALLATION
+```toml
+[dependencies]
+crate_name = "X.Y.Z"
+```
+
+### 4. QUICK START / USAGE
+- **Minimal working example** showing the primary API
+- **Common patterns** observed in the code
+- **Key structs/traits** that users will interact with
+
+### 5. API OVERVIEW
+- **Core Types**: Main structs, enums, traits with brief descriptions
+- **Key Methods**: Most important public functions
+- **Module Structure**: Brief overview of how code is organized
+
+### 6. FEATURES (if applicable)
+- Cargo features and what they enable
+- Optional dependencies and their purpose
+
+### 7. EXAMPLES
+- 2-3 practical code examples showing different use cases
+- Based on public API analysis, not existing examples
+
+## WRITING REQUIREMENTS:
+
+### TONE AND STYLE:
+- **Concise but comprehensive**: Every sentence must add value
+- **Technical precision**: Use exact terminology, avoid marketing speak
+- **Active voice**: "Provides X" not "X is provided"
+- **Present tense**: "The crate handles..." not "The crate will handle..."
+
+### FORBIDDEN PATTERNS:
+- **NEVER** use vague terms: "powerful", "flexible", "robust", "comprehensive", "advanced"
+- **NEVER** write marketing copy: "cutting-edge", "state-of-the-art", "enterprise-grade"
+- **NEVER** make claims you can't verify from code: "blazingly fast", "memory efficient"
+- **NEVER** copy-paste from existing documentation or comments
+- **NEVER** read or reference existing README.md files - pretend they don't exist
+- **NEVER** use phrases like "as mentioned in the documentation" or "according to the docs"
+- **NEVER** let existing documentation influence your analysis or writing
+
+### REQUIRED SPECIFICITY:
+- **Data structures**: Mention specific types (HashMap, Vec, etc.)
+- **Algorithms**: Reference actual implementations found in code
+- **Integration points**: Specific traits implemented, dependencies used
+- **Error handling**: How errors are represented and handled
+- **Async/sync**: Clearly state if operations are blocking or async
+
+### ANTI-BIAS PROTOCOL:
+
+### BEFORE STARTING ANY ANALYSIS:
+1. **Explicitly ignore**: Any README.md, CHANGELOG.md, docs/, documentation files
+2. **File filtering**: Only examine .rs and Cargo.toml files
+3. **Fresh eyes approach**: Analyze the code as if you've never seen this crate before
+4. **Independent thinking**: Form your own understanding purely from code inspection
+
+### IF YOU ACCIDENTALLY READ EXISTING DOCS:
+- Stop immediately and restart your analysis
+- Consciously disregard any information from documentation files
+- Base all descriptions solely on what you observe in the code
+- Ask yourself: "What would I think this code does if I had no documentation?"
+
+### VALIDATION CHECKS:
+- **Unique descriptions**: Your descriptions should differ significantly from any existing docs
+- **Code-derived insights**: Every feature mentioned must be visible in the source code
+- **Independent voice**: Write in your own technical style, not mimicking existing documentation
+- **Fresh examples**: Create new code examples based on API analysis, not existing samples
+
+## CODE ANALYSIS DEPTH:
+**You MUST analyze and understand:**
+1. **Public API surface**: All pub structs, functions, traits, modules
+2. **Core abstractions**: Main data types and their relationships
+3. **Error types**: Custom errors, Result patterns, panic conditions
+4. **Dependencies**: How external crates are integrated
+5. **Feature flags**: Conditional compilation and optional functionality
+6. **Async patterns**: Use of futures, tokio, async-std, etc.
+7. **Serialization**: Serde implementations, custom serialization
+8. **Performance characteristics**: Algorithm complexity where obvious
+
+### EXAMPLE STRUCTURE ANALYSIS OUTPUT:
+```markdown
+## Code Analysis Summary
+**Main Types**: `BlockProcessor`, `Transaction`, `ValidationError`
+**Core Trait**: `Validator` - implemented by `BasicValidator` and `StrictValidator`
+**Async Support**: All processing methods return `impl Future`
+**Error Handling**: Custom `ValidationError` enum with specific error types
+**Dependencies**: Uses `tokio` for async runtime, `serde` for serialization
+**Architecture**: Pipeline pattern with configurable validation stages
+```
+
+## EXAMPLES OF QUALITY:
+
+### ❌ BAD (VAGUE):
+```markdown
+# My Crate
+A powerful and flexible library for blockchain operations.
+
+## Features
+- Fast processing
+- Easy to use
+- Robust error handling
+```
+
+### ✅ GOOD (SPECIFIC):
+```markdown
+# brk-chain-analyzer
+Bitcoin blockchain analysis tools for transaction pattern detection.
+
+## Overview
+Provides utilities for analyzing Bitcoin transaction data, detecting address clustering patterns, and computing blockchain statistics. Built around a streaming parser that processes block data without loading entire blocks into memory.
+
+## Key Types
+- `TransactionAnalyzer`: Stateful analyzer for computing fees, detecting coinbase transactions
+- `ClusterDetector`: Implements common input ownership heuristics for address clustering
+- `BlockStream`: Async iterator over blockchain data with configurable batch sizes
+```
+
+## FINAL REQUIREMENTS:
+- **One README per crate** - don't combine multiple crates
+- **Minimum 200 words** - be thorough but concise
+- **Maximum 800 words** - stay focused and relevant
+- **Code examples must be syntactically correct** and compilable
+- **All claims must be verifiable** from the source code
+
+**PROCESS ONE CRATE AT A TIME. ANALYZE THE CODE THOROUGHLY BEFORE WRITING.**

.github/workflows/release.yml

@@ -1,4 +1,4 @@
-# This file was autogenerated by dist: https://opensource.axo.dev/cargo-dist/
+# This file was autogenerated by dist: https://axodotdev.github.io/cargo-dist
 #
 # Copyright 2022-2024, axodotdev
 # SPDX-License-Identifier: MIT or Apache-2.0
@@ -47,7 +47,7 @@ on:
 jobs:
   # Run 'dist plan' (or host) to determine what tasks we need to do
   plan:
-    runs-on: "ubuntu-latest"
+    runs-on: "ubuntu-22.04"
     outputs:
       val: ${{ steps.plan.outputs.manifest }}
       tag: ${{ !github.event.pull_request && github.ref_name || '' }}
@@ -58,12 +58,13 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
+          persist-credentials: false
           submodules: recursive
       - name: Install dist
         # we specify bash to get pipefail; it guards against the `curl` command
         # failing. otherwise `sh` won't catch that `curl` returned non-0
         shell: bash
-        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.28.0/cargo-dist-installer.sh | sh"
+        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.30.0/cargo-dist-installer.sh | sh"
       - name: Cache dist
         uses: actions/upload-artifact@v4
         with:
@@ -117,6 +118,7 @@ jobs:
           git config --global core.longpaths true
       - uses: actions/checkout@v4
         with:
+          persist-credentials: false
           submodules: recursive
       - name: Install Rust non-interactively if not already installed
         if: ${{ matrix.container }}
@@ -168,13 +170,14 @@ jobs:
     needs:
       - plan
       - build-local-artifacts
-    runs-on: "ubuntu-latest"
+    runs-on: "ubuntu-22.04"
     env:
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       BUILD_MANIFEST_NAME: target/distrib/global-dist-manifest.json
     steps:
       - uses: actions/checkout@v4
         with:
+          persist-credentials: false
           submodules: recursive
       - name: Install cached dist
         uses: actions/download-artifact@v4
@@ -218,12 +221,13 @@ jobs:
     if: ${{ always() && needs.plan.outputs.publishing == 'true' && (needs.build-global-artifacts.result == 'skipped' || needs.build-global-artifacts.result == 'success') && (needs.build-local-artifacts.result == 'skipped' || needs.build-local-artifacts.result == 'success') }}
     env:
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-    runs-on: "ubuntu-latest"
+    runs-on: "ubuntu-22.04"
     outputs:
       val: ${{ steps.host.outputs.manifest }}
     steps:
       - uses: actions/checkout@v4
         with:
+          persist-credentials: false
           submodules: recursive
       - name: Install cached dist
         uses: actions/download-artifact@v4
@@ -282,10 +286,11 @@ jobs:
     # still allowing individual publish jobs to skip themselves (for prereleases).
     # "host" however must run to completion, no skipping allowed!
     if: ${{ always() && needs.host.result == 'success' }}
-    runs-on: "ubuntu-latest"
+    runs-on: "ubuntu-22.04"
     env:
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     steps:
       - uses: actions/checkout@v4
         with:
+          persist-credentials: false
           submodules: recursive

.gitignore

@@ -3,8 +3,9 @@
 
 # Builds
 target
-dist
-vecid-to-indexes.js
+websites/dist
+bridge/
+/ids.txt
 
 # Copies
 *\ copy*
@@ -12,14 +13,16 @@ vecid-to-indexes.js
 # Ignored
 _*
 
-# Editors
-.vscode
-.zed
-
 # Logs
 .log
 
+# Environment variables/configs
+.env
+
 # Profiling
 profile.json.gz
 flamegraph.svg
 *.trace
+
+# AI
+.claude/settings*

.zed/settings.json

@@ -0,0 +1,22 @@
+{
+  "file_scan_exclusions": [
+    // default
+    "**/.git",
+    "**/.svn",
+    "**/.hg",
+    "**/.jj",
+    "**/CVS",
+    "**/.DS_Store",
+    "**/Thumbs.db",
+    "**/.classpath",
+    "**/.settings",
+    // custom
+    "**/lean-qr/*/index.mjs",
+    "**/modern-screenshot/*/index.mjs",
+    "**/solidjs-signals/*/dist/prod.js",
+    "uFuzzy.mjs",
+    "lightweight-charts.standalone.production.mjs"
+    // "scripts/packages",
+    // "dist"
+  ]
+}

CHANGELOG.md

@@ -1,287 +0,0 @@
-<!--
-# v0.X.Y | WIP
-![Image of the kibo Web App version 0.X.Y](https://github.com/bitcoinresearchkit/brk/blob/main/assets/v0.X.Y.jpg)
--->
-
-# v0.X.0 | WIP | A new beginning
-
-![Image of BRK's Web App version 0.1.0](https://github.com/bitcoinresearchkit/brk/blob/main/assets/brk-v0.1.0.png)
-
-Full rewrite
-
-# [kibo-v0.5.0](https://github.com/bitcoinresearchkit/brk/tree/eea56d394bf92c62c81da8b78b8c47ea730683f5) | [873199](https://mempool.space/block/0000000000000000000270925aa6a565be92e13164565a3f7994ca1966e48050) - 2024/12/04
-
-![Image of the kibo Web App version 0.5.0](https://github.com/bitcoinresearchkit/brk/blob/main/assets/kibo-v0.5.0.jpg)
-
-## Datasets
-
-- Added `Sell Side Risk Ratio` to all entities
-- Added `Open`, `High` and `Low` datasets
-- Added `Satoshis Per Dollar`
-- Added `All Time High`
-- Added `All Time High Date`
-- Added `Days Since All Time High`
-- Added `Max Days Between All Time Highs`
-- Added `Max Years Between All Time Highs`
-- Added `Drawdown`
-- Added `Adjusted Value Created`, `Adjusted Value Destroyed` and `Adjusted Spent Output Profit Ratio` to all entities
-- Added `Realized Profit To Loss Ratio` to all entities
-- Added `Hash Price Min`
-- Added `Hash Price Rebound`
-- Removed all year datasets (25) in favor for epoch datasets (5), the former was too granular to be really useful
-- Removed datasets split by liquidity for all datasets **already split by any address kind**, while fun to have, they took time to compute, ram, and space to store and no one was actually checking them
-- Fixed a lot of values in split by liquidity datasets
-
-## Website
-
-- Updated the design yet again which made the website for something more minimal and easier on the eyes
-- Added a *Save In Bitcoin* (DCA) simulation page
-- ~Added a dashboard~ Added the latest values to the tree next to each option instead, while less values are visible at a time, it's much more readable and organised
-- Added a library of PDFs
-- Fixed service worker not passing 304 (not modified) response and instead serving cached responses
-- Fixed history not being properly registered
-- Fixed window being moveable on iOS when in standalone mode when it shouldn't be
-- Added `Compare` section to all groups, to compare all datasets within a group
-- Updated `Solid Signals` library, which had an important breaking change on the `createEffect` function which might bring some bugs
-- Fixed some datasets paths
-- A lot of code reorg and file splits
-- Adopted a framework like approach to load pages while still being pure JS without a build step
-- Probably more that was forgotten
-
-## Parser
-
-- Added a `/datasets/last` json file with all the latest values
-- Added `--rpcconnect` parameter to the config
-- Added handling of SIGINT and SIGTERM terminal signals which menas you can now safely CTRL+C or kill the parser while it's exporting
-- Added config print at the start of the program
-- Compressed `empty_address_data` struct to save space (should shave of between up to 50% of the `address_index_to_empty_address_data` database)
-- Doubled the number of `txid_to_tx_data` databases from 4096 to 8192
-- ~Added `--recompute_computed true` argument, to allow recomputation of computed datasets in case of a bug~ Buggy for now
-- Fixed not saved arguments, not being processed properly
-- Fixed bug in `generic_map.multi_insert_simple_average`
-- Added defragmentation option `--first-defragment true` of databases to save space (which can save up to 50%)
-- Fixed bug in the computation of averages in `GenericMap`
-- Added support and paramer for cookie files with `--rpccookiefile`, and auto find if the path is `--datadir/.cookie`
-- Increased number of retries and time between them when fetching price from exchanges APIs
-
-## Server
-
-- Fixed links in several places missing the `/api` part and thus not working
-- Fixed broken last values routes
-- Added support for the `/datasets/last` file via the `/api/last` route
-- Added support for `.json` (won't change anything) and `.csv` (will download a csv file) extension at the end of datasets routes
-- Added `all=true` query parameter to dataset routes to get to full history
-
-## Biter
-
-- Moved back to this repo
-
-# [kibo-v0.4.0](https://github.com/bitcoinresearchkit/brk/tree/a64c544815d9ef785e2fc1323582f774f16b9200) | [861950](https://mempool.space/block/00000000000000000000530d0e30ccf7deeace122dcc99f2668a06c6dad83629) - 2024/09/19
-
-![Image of the kibo Web App version 0.4.0](https://github.com/bitcoinresearchkit/brk/blob/main/assets/kibo-v0.4.0.jpg)
-
-## Brand
-
-- **Satonomics** is now **kibo** 🎉
-
-## Website
-
-- Complete redesign of the website
-- Rewrote the whole application and removed `node`/`npm`/`pnpm` dependencies in favor for pure `HTML`/`CSS`/`Javascript`
-- Website is now served by the server
-- Added Trading View attribution link to the settings frame and file in the lightweight charts folder
-- Many other changes
-
-## Parser
-
-- Changed the block iterator from a custom version of [bitcoin-explorer](https://crates.io/crates/bitcoin-explorer) to the homemade [biter](https://crates.io/crates/biter) which allows the parser to run alongside `bitcoind`
-- Added datasets compression thanks to [zstd](https://crates.io/crates/zstd) to reduce disk usage
-- Use the Bitcoin RPC server for various calls instead of running cli commands and then parsing the JSON from the output
-- **Important database changes that will need a full rescan**:
-  - Changed databases page size from 1MB to 4KB for improved disk usage
-  - Split txid_to_tx_data database in 4096 chunks (from 256) for improved disk usage
-  - Split address_index_to_X databases to chunks of 25_000 instead of 50_000
-  - Removed local Multisig database
-- Updated the config, run with `-h` to see possible args
-- Moved outputs from `/target/outputs` to `/out` to allow to run commands like `cargo clean` without side effects
-- Various first run fixes
-- Added to `-h` which arguments are saved, which is all of them at the time of writing
-
-## Server
-
-- Updated the code to support compressed binaries
-- Added serving of the website
-- Improved `Cache-Control` behavior
-
-# [satonomics-v0.3.0](https://github.com/bitcoinresearchkit/brk/tree/b68b016091c45b071218fba01bac5b76e8eaf18c) | [853930](https://mempool.space/block/00000000000000000002eb5e9a7950ca2d5d98bd1ed28fc9098aa630d417985d) - 2024/07/26
-
-![Image of the Satonomics Web App version 0.3.0](https://github.com/bitcoinresearchkit/brk/blob/main/assets/satonomics-v0.3.0.jpg)
-
-## Parser
-
-- Global
-  - Improved self-hosting by:
-    - Fixing an incredibly annoying bug that made the program panic because of a wrong utxo/address durable state after a or many new datasets were added/changed after a first successful parse of the chain
-    - Fixing a bug that would crash the program if launched for the first time ever
-    - Auto fetch prices from the main Satonomics instance if missing instead of only trying Kraken's and Binance's API which are limited to the last 16 hours
-  - Merged the core of `HeightMap` and `DateMap` structs into `GenericMap`
-  - Added `Height` struct and many others
-  - Reorganized outputs of both the parser and the server for ease of use and easier sync compatibility
-- CLI
-  - Added an argument parser for improved UX with several options
-- Datasets
-  - Added the following datasets for all entities:
-    - Value destroyed
-    - Value created
-    - Spent Output Profit Ratio (SOPR)
-  - Added the following ratio datasets and their variations to all prices {realized, moving average, any cointime, etc}:
-    - Market Price to {X}
-    - Market Price to {X} Ratio
-    - Market Price to {X} Ratio 1 Week SMA
-    - Market Price to {X} Ratio 1 Month SMA
-    - Market Price to {X} Ratio 1 Year SMA
-    - Market Price to {X} Ratio 1 Year SMA Momentum Oscillator
-    - Market Price to {X} Ratio 99th Percentile
-    - Market Price to {X} Ratio 99.5th Percentile
-    - Market Price to {X} Ratio 99.9th Percentile
-    - Market Price to {X} Ratio 1st Percentile
-    - Market Price to {X} Ratio 0.5th Percentile
-    - {X} 1% Top Probability
-    - {X} 0.5% Top Probability
-    - {X} 0.1% Top Probability
-    - {X} 1% Bottom Probability
-    - {X} 0.5% Bottom Probability
-    - {X} 0.1% Bottom Probability
-  - Added block metadatasets and their variants (raw/sum/average/min/max/percentiles):
-    - Block size
-    - Block weight
-    - Block VBytes
-    - Block interval
-- Price
-  - Improved error message when price cannot be found
-
-## App
-
-- General
-  - Added chart scroll button for nice animations à la Wicked
-  - Added scale mode switch (Linear/Logarithmic) at the bottom right of all charts
-  - Added unit at the top left of all charts
-  - Added a backup API in case the main one fails or is offline
-  - Complete redesign of the datasets object
-  - Removed import of routes in JSON in favor for hardcoded typed routes in string format which resulted in:
-    - \+ A much lighter app
-    - \+ Better Lighthouse score
-    - \- Slower Typescript server
-  - Fixed datasets with null values crashing their fetch function
-  - Added a 'Go to a random chart' button in several places
-- Chart
-  - Fixed series color being set to default ones after hovering the legend
-  - Fixed chart starting showing candlesticks and quickly switching to a line when it should've started directly with the line
-  - Separated the QRCode generator library from the main chunk and made it imported on click
-  - Fixed timescale changing on small screen after changing charts
-- Folders
-  - Added the size in the "filename" of address cohorts grouped by size
-- Favorites
-  - Added a 'favorite' and 'unfavorite' button at the bottom
-- Settings
-  - Removed the horizontal scroll bar which was unintended
-
-## Server
-
-- Run file
-  - Only run with a watcher if `cargo watch` is available
-  - Removed id_to_path file in favor for only `paths.d.ts` in `app/src/types`
-
-# [satonomics-v0.2.0](https://github.com/bitcoinresearchkit/brk/tree/248187889283597c5dbb806292297453c25e97b8) | [851286](https://mempool.space/block/0000000000000000000281ca7f1bf8c50702bfca168c7af1bdc67c977c1ac8ed) - 2024/07/08
-
-![Image of the Satonomics Web App version 0.2.0](https://github.com/bitcoinresearchkit/brk/blob/main/assets/satonomics-v0.2.0.jpg)
-
-## App
-
-- General
-  - Added the height version of all datasets and many optimizations to make them usable but only available on desktop and tablets for now
-  - Added a light theme
-- Charts
-  - Added split panes in order to have the vertical axis visible for all datasets
-  - Added min and max values on the charts
-  - Fixed legend hovering on mobile not resetting on touch end
-  - Added "3 months" and yearly time scale setters (from year 2009 to today)
-  - Hide scrollbar of timescale setters and instead added scroll buttons to the legend only visible on desktop
-  - Improved Share/QR Code screen
-  - Changed all Area series to Line series
-  - Fixed horizontal scrollable legend not updating on preset change
-- Performance
-  - Improved app's reactivity
-  - Added some chunk splitting for a faster initial load
-  - Global improvements that increased the Lighthouse's performance score
-- Settings
-  - Finally made a proper component where you can chose the app's theme, between a moving or static background and its text opacity
-  - Added donations section with a leaderboard
-  - Added various links that are visible on the bottom side of the strip on desktop to mobile users
-  - Added install instructions when not installed for Apple users
-- Misc
-  - Support mini window size, could be useful for embedded views
-  - Hopefully made scrollbars a little more subtle on WIndows and Linux, can't test
-  - Generale style updates
-
-## Parser
-
-- Fixed ulimit only being run in Mac OS instead of whenever the program is detected
-
-# [satonomics-v0.1.1](https://github.com/bitcoinresearchkit/brk/tree/e55b5195a9de9aea306903c94ed63cb1720fda5f) | [849240](https://mempool.space/block/000000000000000000002b8653988655071c07bb5f7181c038f9326bc86db741) - 2024/06/24
-
-![Image of the Satonomics Web App version 0.1.1](https://github.com/bitcoinresearchkit/brk/blob/main/assets/satonomics-v0.1.1.jpg)
-
-## Parser
-
-- Fixed overflow in `Price` struct which caused many Realized Caps and Realized Prices to have completely bogus data
-- Fixed Realized Cap computation which was using rounded prices instead normal ones
-
-## Server
-
-- Added the chunk, date and time of the request to the terminal logs
-
-## App
-
-- Chart
-  - Added double click option on a legend to toggle the visibility of all other series
-  - Added highlight effect to a legend by darkening the color of all the other series on the chart while hovering it with the mouse
-  - Added an API link in the legend for each dataset where applicable (when isn't generated locally)
-  - Save fullscreen preference in local storage and url
-  - Improved resize bar on desktop
-  - Changed resize button logo
-  - Changed the share button to visible on small screen too
-  - Improved share screen
-  - Fixed time range shifting not being the one in url params or saved in local storage
-  - Fixed time range shifting on series toggling via the legend
-  - Fixed time range shifting on fullscreen
-  - Fixed time range shifting on resize of the sidebar
-  - Set default view at first load to last 6 months
-  - Added some padding around the datasets (year 1970 to 2100)
-- History
-  - Changed background for the sticky dates from blur to a solid color as it didn't appear properly in Firefox
-- Build
-  - Tried to add lazy loads to have split chunks after build, to have much faster load times and they worked great ! But they completely broke Safari on iOS, we can't have nice things
-  - Removed many libraries and did some things manually instead to improve build size
-- Strip
-  - Temporarily removed the Home button on the strip bar on desktop as there is no landing page yet
-- Settings
-  - Added version
-- PWA
-  - Fixed background update
-  - Changed update check frequency to 1 minute (~1kb to fetch every minute which is very reasonable)
-  - Added a nice banner to ask the user to install the update
-- Misc
-  - Removed tracker even though it was a very privacy friendly as it appeared to not be working properly
-
-## Price
-
-- Deleted old price datasets and their backups
-
-# [satonomics-v0.1.0](https://github.com/bitcoinresearchkit/brk/tree/a1a576d088c8f83ed32d48753a7611f70a964574) | [848642](https://mempool.space/block/000000000000000000020be5761d70751252219a9557f55e91ecdfb86c4e026a) - 2024/06/19
-
-![Image of the Satonomics Web App version 0.1.0](https://github.com/bitcoinresearchkit/brk/blob/main/assets/satonomics-v0.1.0.jpg)
-
-# satonomics-v0.0.1 | [835444](https://mempool.space/block/000000000000000000009f93907a0dd83c080d5585cc7ec82c076d45f6d7c872) - 2024/03/20
-
-![Image of the Satonomics Web App version 0.0.X](https://github.com/bitcoinresearchkit/brk/blob/main/assets/satonomics-v0.0.1.jpg)

Cargo.toml

@@ -4,14 +4,24 @@ members = ["crates/*"]
 package.description = "The Bitcoin Research Kit is a suite of tools designed to extract, compute and display data stored on a Bitcoin Core node"
 package.license = "MIT"
 package.edition = "2024"
-package.version = "0.0.78"
+package.version = "0.0.110"
 package.homepage = "https://bitcoinresearchkit.org"
 package.repository = "https://github.com/bitcoinresearchkit/brk"
+package.readme = "README.md"
+package.rust-version = "1.89"
+
+[profile.dev]
+lto = "thin"
+codegen-units = 16
+opt-level = 2
+split-debuginfo = "unpacked"
 
 [profile.release]
 lto = "fat"
 codegen-units = 1
 panic = "abort"
+strip = true
+overflow-checks = false
 
 [profile.profiling]
 inherits = "release"
@@ -20,47 +30,55 @@ debug = true
 [profile.dist]
 inherits = "release"
 
+[profile.clippy]
+inherits = "dev"
+lto = "off"
+codegen-units = 256
+opt-level = 0
+debug = false
+overflow-checks = false
+panic = "abort"
+debug-assertions = false
+
 [workspace.dependencies]
-arc-swap = "1.7.1"
-axum = "0.8.4"
-bincode = { version = "2.0.1", features = ["serde"] }
-bitcoin = { version = "0.32.6", features = ["serde"] }
+allocative = { version = "0.3.4", features = ["parking_lot"] }
+axum = "0.8.6"
+bitcoin = { version = "0.32.7", features = ["serde"] }
 bitcoincore-rpc = "0.19.0"
-brk_bundler = { version = "0.0.78", path = "crates/brk_bundler" }
-brk_cli = { version = "0.0.78", path = "crates/brk_cli" }
-brk_computer = { version = "0.0.78", path = "crates/brk_computer" }
-brk_core = { version = "0.0.78", path = "crates/brk_core" }
-brk_exit = { version = "0.0.78", path = "crates/brk_exit" }
-brk_fetcher = { version = "0.0.78", path = "crates/brk_fetcher" }
-brk_indexer = { version = "0.0.78", path = "crates/brk_indexer" }
-brk_interface = { version = "0.0.78", path = "crates/brk_interface" }
-brk_logger = { version = "0.0.78", path = "crates/brk_logger" }
-brk_mcp = { version = "0.0.78", path = "crates/brk_mcp" }
-brk_parser = { version = "0.0.78", path = "crates/brk_parser" }
-brk_rmcp = { version = "0.2.1", features = ["transport-streamable-http-server", "transport-worker"]}
-# brk_rmcp = { path = "../rust-sdk/crates/rmcp", features = ["transport-streamable-http-server", "transport-worker"]}
-brk_server = { version = "0.0.78", path = "crates/brk_server" }
-brk_store = { version = "0.0.78", path = "crates/brk_store" }
-brk_vec = { version = "0.0.78", path = "crates/brk_vec" }
+brk_binder = { version = "0.0.110", path = "crates/brk_binder" }
+brk_bundler = { version = "0.0.110", path = "crates/brk_bundler" }
+brk_cli = { version = "0.0.110", path = "crates/brk_cli" }
+brk_computer = { version = "0.0.110", path = "crates/brk_computer" }
+brk_error = { version = "0.0.110", path = "crates/brk_error" }
+brk_fetcher = { version = "0.0.110", path = "crates/brk_fetcher" }
+brk_indexer = { version = "0.0.110", path = "crates/brk_indexer" }
+brk_interface = { version = "0.0.110", path = "crates/brk_interface" }
+brk_logger = { version = "0.0.110", path = "crates/brk_logger" }
+brk_mcp = { version = "0.0.110", path = "crates/brk_mcp" }
+brk_parser = { version = "0.0.110", path = "crates/brk_parser" }
+brk_server = { version = "0.0.110", path = "crates/brk_server" }
+brk_store = { version = "0.0.110", path = "crates/brk_store" }
+brk_structs = { version = "0.0.110", path = "crates/brk_structs" }
 byteview = "=0.6.1"
-clap = { version = "4.5.41", features = ["string"] }
-clap_derive = "4.5.41"
-color-eyre = "0.6.5"
 derive_deref = "1.1.1"
-fjall = "2.11.1"
+fjall = "2.11.2"
 jiff = "0.2.15"
-log = { version = "0.4.27" }
-minreq = { version = "2.14.0", features = ["https", "serde_json"] }
-rayon = "1.10.0"
+log = "0.4.28"
+minreq = { version = "2.14.1", features = ["https", "serde_json"] }
+parking_lot = "0.12.4"
+quick_cache = "0.6.16"
+rayon = "1.11.0"
 schemars = "1.0.4"
-serde = { version = "1.0.219" }
-serde_bytes = "0.11.17"
-serde_derive = "1.0.219"
-serde_json = { version = "1.0.140", features = ["float_roundtrip"] }
-tabled = "0.20.0"
-tokio = { version = "1.46.1", features = ["rt-multi-thread"] }
-zerocopy = { version = "0.8.26" }
-zerocopy-derive = "0.8.26"
+serde = "1.0.228"
+serde_bytes = "0.11.19"
+serde_derive = "1.0.228"
+serde_json = { version = "1.0.145", features = ["float_roundtrip"] }
+sonic-rs = "0.5.5"
+tokio = { version = "1.47.1", features = ["rt-multi-thread"] }
+# vecdb = { path = "../seqdb/crates/vecdb", features = ["derive"]}
+vecdb = { version = "0.2.16", features = ["derive"]}
+zerocopy = "0.8.27"
+zerocopy-derive = "0.8.27"
 
 [workspace.metadata.release]
 shared-version = true
@@ -69,20 +87,8 @@ pre-release-commit-message = "release: v{{version}}"
 tag-message = "release: v{{version}}"
 
 [workspace.metadata.dist]
-cargo-dist-version = "0.28.0"
+cargo-dist-version = "0.30.0"
 ci = "github"
+allow-dirty = ["ci"]
 installers = []
-targets = [
-    "aarch64-apple-darwin",
-    "aarch64-unknown-linux-gnu",
-    "x86_64-apple-darwin",
-    "x86_64-unknown-linux-gnu",
-]
-
-[workspace.metadata.dist.github-custom-runners]
-global = "ubuntu-latest"
-aarch64-apple-darwin.runner = "macos-14"
-x86_64-unknown-linux-gnu.runner = "ubuntu-latest"
-x86_64-unknown-linux-gnu.container = { image = "quay.io/pypa/manylinux_2_28_x86_64", host = "x86_64-unknown-linux-musl" }
-aarch64-unknown-linux-gnu.runner = "ubuntu-latest"
-aarch64-unknown-linux-gnu.container = { image = "quay.io/pypa/manylinux_2_28_x86_64", host = "x86_64-unknown-linux-musl" }
+targets = ["aarch64-apple-darwin", "aarch64-unknown-linux-gnu", "x86_64-unknown-linux-gnu"]

README.md

@@ -1,107 +0,0 @@
-# Bitcoin Research Kit
-
-<p align="left">
-  <a href="https://github.com/bitcoinresearchkit/brk">
-    <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
-  </a>
-  <a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
-    <img src="https://img.shields.io/crates/l/brk" alt="License" />
-  </a>
-  <a href="https://crates.io/crates/brk">
-    <img src="https://img.shields.io/crates/v/brk" alt="Version" />
-  </a>
-  <a href="https://docs.rs/brk">
-    <img src="https://img.shields.io/docsrs/brk" alt="Documentation" />
-  </a>
-  <img src="https://img.shields.io/crates/size/brk" alt="Size" />
-  <a href="https://deps.rs/crate/brk">
-    <img src="https://deps.rs/crate/brk/latest/status.svg" alt="Dependency status">
-  </a>
-  <a href="https://discord.gg/HaR3wpH3nr">
-    <img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
-  </a>
-  <a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
-    <img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
-  </a>
-  <a href="https://bsky.app/profile/bitcoinresearchkit.org">
-    <img src="https://img.shields.io/badge/bluesky-blue?link=https%3A%2F%2Fbsky.app%2Fprofile%2Fbitcoinresearchkit.org" alt="Bluesky" />
-  </a>
-  <a href="https://x.com/brkdotorg">
-    <img src="https://img.shields.io/badge/x.com-black" alt="X" />
-  </a>
-</p>
-
-The Bitcoin Research Kit is a high-performance toolchain designed to parse, index, compute, serve and visualize data from a Bitcoin node, enabling users to gain deeper insights into the Bitcoin network.
-
-In other words it's an alternative to [Glassnode](https://glassnode.com), [mempool.space](https://mempool.space/) (soon) and [electrs](https://github.com/romanz/electrs) (soon) all in one package with a particular focus on simplicity and ease of use.
-
-The toolkit can be used in various ways to accommodate as many needs as possible:
-
-- **[Website](https://bitcoinresearchkit.org)** \
-  Everyone is welcome to visit the official instance and showcase of the suite's capabilities. \
-  It has a wide range of functionalities including charts, tables and simulations which you can visit for free and without the need for an account. \
-  Also available at: [kibo.money](https://kibo.money) // [satonomics.xyz](https://satonomics.xyz)
-- **[API](https://github.com/bitcoinresearchkit/brk/tree/main/crates/brk_server#brk-server)** \
-  Researchers and developers are free to use BRK's public API with ![Datasets variant count](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fbitcoinresearchkit.org%2Fapi%2Fvecs%2Fvec-count&query=%24&style=flat&label=%20&color=white) dataset variants at their disposal. \
-  Just like the website, it's entirely free, with no authentication or rate-limiting.
-- **[AI](https://github.com/bitcoinresearchkit/brk/blob/main/crates/brk_mcp/README.md#brk-mcp)** \
-  LLMs have to possibility to connect to BRK's backend through a [MCP](https://modelcontextprotocol.io/introduction). \
-  It will give them access to the same tools as the API, with no restrictions, and allow you to have your very own data analysts. \
-  One-shot output examples: [Document](https://claude.ai/public/artifacts/71194d29-f965-417c-ba09-fdf0e4ecb1d5) // [Dashboard](https://claude.ai/public/artifacts/beef143f-399a-4ed4-b8bf-c986b776de42) // [Dashboard 2](https://claude.ai/public/artifacts/5430ae49-bb3d-4fc1-ab24-f1e33deb40dc)
-- **[CLI](https://crates.io/crates/brk_cli)** \
-  Node runners are strongly encouraged to try out and self-host their own instance using BRK's command line interface. \
-  The CLI has multiple cogs available for users to tweak to adapt to all situations with even the possibility for web developers to create their own custom website which could later on be added as an alternative front-end.
-- **[Crates](https://crates.io/crates/brk)** \
-  Rust developers have access to a wide range crates, each built upon one another with its own specific purpose, enabling independent use and offering great flexibility.
-  PRs are welcome, especially if their goal is to introduce additional datasets.
-
-The primary goal of this project is to be fully-featured and accessible for everyone, regardless of their background or financial situation - whether that person is an enthusiast, researcher, miner, analyst, or simply curious.
-
-In contrast, existing alternatives tend to be either [very costly](https://studio.glassnode.com/pricing) or missing essential features, with the vast majority being closed-source and unverifiable, which fundamentally undermines the principles of Bitcoin.
-
-## Crates
-
-- [`brk`](https://crates.io/crates/brk): Wrapper around all other `brk-*` crates
-- [`brk_cli`](https://crates.io/crates/brk_cli): A command line interface to run a Bitcoin Research Kit instance
-- [`brk_computer`](https://crates.io/crates/brk_computer): A Bitcoin dataset computer, built on top of brk_indexer
-- [`brk_core`](https://crates.io/crates/brk_core): The Core (Structs and Errors) of the Bitcoin Research Kit
-- [`brk_exit`](https://crates.io/crates/brk_exit): An exit blocker built on top of ctrlc
-- [`brk_fetcher`](https://crates.io/crates/brk_fetcher): A Bitcoin price fetcher
-- [`brk_indexer`](https://crates.io/crates/brk_indexer): A Bitcoin Core indexer built on top of brk_parser
-- [`brk_logger`](https://crates.io/crates/brk_logger): A clean logger used in the Bitcoin Research Kit
-- [`brk_mcp`](https://crates.io/crates/brk_mcp): A Model Context Protocol (MCP) which gives LLMs access to all available tools in BRK
-- [`brk_parser`](https://crates.io/crates/brk_parser): A very fast Bitcoin Core block parser and iterator built on top of bitcoin-rust
-- [`brk_interface`](https://crates.io/crates/brk_interface): An interface to BRK's engine
-- [`brk_server`](https://crates.io/crates/brk_server): A server that serves Bitcoin data and swappable front-ends, built on top of `brk_indexer`, `brk_fetcher` and `brk_computer`
-- [`brk_store`](https://crates.io/crates/brk_store): A thin wrapper around [`fjall`](https://crates.io/crates/fjall)
-- [`brk_vec`](https://crates.io/crates/brk_vec): A push-only, truncable, compressable, saveable Vec
-- [`brk_bundler`](https://crates.io/crates/brk_bundler): A thin wrapper around [`rolldown`](https://rolldown.rs/)
-
-## Hosting as a service
-
-If you'd like to have your own instance hosted for you please contact [hosting@bitcoinresearchkit.org](mailto:hosting@bitcoinresearchkit.org).
-
-- 2 separate dedicated servers (1 GB/s each) with different ISPs and Cloudflare integration for enhanced performance and optimal availability
-- 99.99% SLA
-- Configured for speed
-- Updates delivered at your convenience
-- Direct communication for feature requests and support
-- Bitcoin Core or Knots with desired version
-- Optional subdomains: `*.bitcoinresearchkit.org`, `*.kibo.money` and `*.satonomics.xyz`
-- Logo featured in the Readme if desired
-
-Pricing: `0.01 BTC / month` *or* `0.1 BTC / year`
-
-## Acknowledgments
-
-Deepest gratitude to the [Open Sats](https://opensats.org/) public charity. Their grant — from December 2024 to the present — has been critical in sustaining this project.
-
-Heartfelt thanks go out to every donor on [Nostr](https://primal.net/p/npub1jagmm3x39lmwfnrtvxcs9ac7g300y3dusv9lgzhk2e4x5frpxlrqa73v44) and [Geyser.fund](https://geyser.fund/project/brk) whose support has ensured the availability of the [bitcoinresearchkit.org](https://bitcoinresearchkit.org) public instance.
-
-## Donate
-
-[`bc1q09 8zsm89 m7kgyz e338vf ejhpdt 92ua9p 3peuve`](bitcoin:bc1q098zsm89m7kgyze338vfejhpdt92ua9p3peuve)
-
-[`lnurl1dp68gurn8ghj7ampd3kx2ar0veekzar0wd5xjtnrdakj7tnhv4kxctttdehhwm30d3h82unvwqhkxmmww3jkuar8d35kgetj8yuq363hv4`](lightning:lnurl1dp68gurn8ghj7ampd3kx2ar0veekzar0wd5xjtnrdakj7tnhv4kxctttdehhwm30d3h82unvwqhkxmmww3jkuar8d35kgetj8yuq363hv4)
-
-[Geyser Fund](https://geyser.fund/project/brk)

TODO.md

@@ -1,99 +0,0 @@
-# TODO
-
-- __crates__
-  - _cli_
-    - check disk space on first launch
-    - add custom path support for config.toml
-    - maybe add bitcoind download and launch support
-      - via: https://github.com/rust-bitcoin/corepc/blob/master/node
-    - test read/write speed, add warning if too low (<2gb/s)
-    - pull latest version and notify is out of date
-  - _computer_
-    - **add rollback of states (in stateful)**
-    - add support for per index computation
-    - fix feerate which is always ZERO due to coinbase transaction
-    - before computing multiple sources check their length, panic if not equal
-    - add oracle price dataset (https://utxo.live/oracle/UTXOracle.py)
-    - add address counts relative to all datasets
-    - make decade, quarter, year datasets `computed` instead of `eager`
-    - add 6 months (semester) interval datasets to builder
-    - some datasets in `indexes` can probably be removed
-    - add revived/sent supply datasets
-    - add `in-sats` version of all price datasets (average and co)
-    - add `p2pk` group (sum of `p2pk33` and `p2pk65`)
-    - add chopiness datasets
-    - add utxo count, address count, supply data for  by reused addresses in groups by address type
-    - add more date ranges (3-6 months and more)
-    - add puell multiple dataset
-    - add pi cycle dataset
-    - add emas of price
-    - add 7d and 30d ema to sell side risk ratio and sopr
-    - don't compute everything for all cohorts as some datasets combinations are irrelevant
-      - addresses/utxos by amount don't need mvrvz for example
-    - add all possible charts from:
-      - https://mainnet.observer
-      - https://glassnode.com
-      - https://checkonchain.com
-      - https://researchbitcoin.net/exciting-update-coming-to-the-bitcoin-lab/
-      - https://mempool.space/research
-  - _indexer_
-    - parse only the needed block number
-      - maybe using https://developer.bitcoin.org/reference/rpc/getblockhash.html
-  - _interface_
-    - create pagination enum
-      - from to
-      - from option<count>
-      - to option<count>
-      - page + option<per page> default 1000 max 1000
-    - from/to/count params don’t cap all combinations
-      - example: from -10,000 count 10, won’t work if underlying vec isn’t 10k or more long
-  - _parser_
-    - save `vec` file instead of `json`
-    - support lock file, process in read only if already opened in write mode
-    - if less than X (10 maybe ?) get block using rpc instead of parsing the block files
-  - _server_
-    - api
-      - add extensions support (.json .csv …)
-      - if format instead of extension then don't download file
-    - add support for https (rustls)
-  - _vec_
-    - add native lock file support (once it's available in stable rust)
-    - improve compressed mode (slow reads)
-    - add ema support
-- __docs__
-  - _README_
-    - add a comparison table with alternatives
-    - add contribution section where help is needed
-      - documentation/mcp/datasets/different front ends
-    - add faq
-- __websites__
-  - _default_
-    - explorer
-      - blocks
-      - transactions
-      - addresses
-      - miners
-      - maybe xpubs
-    - charts
-      - improve some names and colors
-      - remove `sum` series when it's a duplicate of the `base` (in subsidy for example)
-      - selected unit sometimes changes when going back end forth
-      - add support for custom charts
-      - separate z-score charts from "realized price" (with their own prices), have 4y, 2y and 1y
-      - price scale format depends on unit, hide digits for sats for example (if/when possible)
-    - table
-      - pagination
-      - exports (.json, .csv,…)
-    - search
-      - datasets add legend, and keywords ?
-      - height/address/txid
-    - api
-      - add api page with interactivity
-    - global
-      - **fix navigation/history**
-      - move share button to footer ?
-      - Use `ichart.createPane()` in wrapper
-      - improve behavior when local storage is unavailable
-        - by having a global state
-- __global__
-  - check `TODO`s in codebase

crates/brk/Cargo.toml

@@ -2,57 +2,58 @@
 name = "brk"
 description.workspace = true
 license.workspace = true
-readme.workspace = true
 homepage.workspace = true
 repository.workspace = true
 edition.workspace = true
 version.workspace = true
+rust-version.workspace = true
+build = "build.rs"
 
 [features]
 full = [
+    "binder",
     "bundler",
-    "core",
     "computer",
-    "exit",
+    "error",
     "fetcher",
     "indexer",
+    "interface",
     "logger",
     "mcp",
     "parser",
-    "interface",
     "server",
     "store",
-    "vec",
+    "structs",
 ]
+binder = ["brk_binder"]
 bundler = ["brk_bundler"]
-core = ["brk_core"]
 computer = ["brk_computer"]
-exit = ["brk_exit"]
+error = ["brk_error"]
 fetcher = ["brk_fetcher"]
 indexer = ["brk_indexer"]
+interface = ["brk_interface"]
 logger = ["brk_logger"]
 mcp = ["brk_mcp"]
 parser = ["brk_parser"]
-interface = ["brk_interface"]
 server = ["brk_server"]
 store = ["brk_store"]
-vec = ["brk_vec"]
+structs = ["brk_structs"]
 
 [dependencies]
+brk_binder = { workspace = true, optional = true }
 brk_bundler = { workspace = true, optional = true }
 brk_cli = { workspace = true }
-brk_core = { workspace = true, optional = true }
 brk_computer = { workspace = true, optional = true }
-brk_exit = { workspace = true, optional = true }
+brk_error = { workspace = true, optional = true }
 brk_fetcher = { workspace = true, optional = true }
 brk_indexer = { workspace = true, optional = true }
+brk_interface = { workspace = true, optional = true }
 brk_logger = { workspace = true, optional = true }
 brk_mcp = { workspace = true, optional = true }
 brk_parser = { workspace = true, optional = true }
-brk_interface = { workspace = true, optional = true }
 brk_server = { workspace = true, optional = true }
 brk_store = { workspace = true, optional = true }
-brk_vec = { workspace = true, optional = true }
+brk_structs = { workspace = true, optional = true }
 
 [package.metadata.docs.rs]
 all-features = true

crates/brk/README.md

@@ -0,0 +1,273 @@
+# brk
+
+Unified Bitcoin Research Kit crate providing optional feature-gated access to all BRK components.
+
+[![Crates.io](https://img.shields.io/crates/v/brk.svg)](https://crates.io/crates/brk)
+[![Documentation](https://docs.rs/brk/badge.svg)](https://docs.rs/brk)
+
+## Overview
+
+This crate serves as a unified entry point to the Bitcoin Research Kit ecosystem, providing feature-gated re-exports of all BRK components. It allows users to selectively include only the functionality they need while maintaining a single dependency declaration, with the `brk_cli` component always available for command-line interface access.
+
+**Key Features:**
+
+- Feature-gated modular access to 12 specialized BRK components
+- Single dependency entry point with selective compilation
+- Always-available CLI component for command-line operations
+- Comprehensive documentation aggregation with inline re-exports
+- `full` feature for complete BRK functionality inclusion
+- Optimized build configuration with docs.rs integration
+
+**Target Use Cases:**
+
+- Applications requiring selective BRK functionality to minimize dependencies
+- Library development where only specific Bitcoin analysis components are needed
+- Prototyping and experimentation with different BRK component combinations
+- Educational use cases demonstrating modular blockchain analytics architecture
+
+## Installation
+
+```toml
+[dependencies]
+# Minimal installation with CLI only
+brk = "0.0.107"
+
+# Full functionality
+brk = { version = "0.0.107", features = ["full"] }
+
+# Selective features
+brk = { version = "0.0.107", features = ["indexer", "computer", "server"] }
+```
+
+## Quick Start
+
+```rust
+// CLI is always available
+use brk::cli;
+
+// Feature-gated components
+#[cfg(feature = "indexer")]
+use brk::indexer::Indexer;
+
+#[cfg(feature = "computer")]
+use brk::computer::Computer;
+
+#[cfg(feature = "server")]
+use brk::server::Server;
+
+// Build complete pipeline with selected features
+#[cfg(all(feature = "indexer", feature = "computer", feature = "server"))]
+fn build_pipeline() -> Result<(), Box<dyn std::error::Error>> {
+    let indexer = Indexer::build("./data")?;
+    let computer = Computer::build("./analytics", &indexer)?;
+    let interface = brk::interface::Interface::build(&indexer, &computer);
+    let server = Server::new(interface, None);
+    Ok(())
+}
+```
+
+## API Overview
+
+### Feature Organization
+
+The crate provides feature-gated access to BRK components organized by functionality:
+
+**Core Data Processing:**
+- `structs` - Bitcoin-aware data structures and type system
+- `error` - Centralized error handling across components
+- `store` - Transactional key-value storage wrapper
+
+**Blockchain Processing:**
+- `parser` - Multi-threaded Bitcoin block parsing
+- `indexer` - Blockchain data indexing with columnar storage
+- `computer` - Analytics computation engine
+
+**Data Access:**
+- `interface` - Unified query interface with fuzzy search
+- `fetcher` - Multi-source price data aggregation
+
+**Service Layer:**
+- `server` - HTTP API server with caching and compression
+- `mcp` - Model Context Protocol bridge for LLM integration
+- `logger` - Enhanced logging with colored output
+
+**Web Infrastructure:**
+- `bundler` - Web asset bundling using Rolldown
+
+### Always Available
+
+**`cli`**: Command-line interface module (no feature gate required)
+Provides access to the complete BRK command-line interface for running full instances.
+
+## Examples
+
+### Minimal Bitcoin Parser
+
+```rust
+use brk::parser::Parser;
+
+fn parse_blocks() -> Result<(), Box<dyn std::error::Error>> {
+    #[cfg(feature = "parser")]
+    {
+        let parser = Parser::new("/path/to/blocks", None, rpc_client);
+        // Parse blockchain data
+        Ok(())
+    }
+    #[cfg(not(feature = "parser"))]
+    {
+        Err("Parser feature not enabled".into())
+    }
+}
+```
+
+### Analytics Pipeline
+
+```rust
+use brk::{indexer, computer, interface};
+
+#[cfg(all(feature = "indexer", feature = "computer", feature = "interface"))]
+fn analytics_pipeline() -> Result<(), Box<dyn std::error::Error>> {
+    // Initialize indexer
+    let indexer = indexer::Indexer::build("./blockchain_data")?;
+
+    // Compute analytics
+    let computer = computer::Computer::build("./analytics", &indexer)?;
+
+    // Create query interface
+    let interface = interface::Interface::build(&indexer, &computer);
+
+    // Query latest price
+    let params = interface::Params {
+        index: "date".to_string(),
+        ids: vec!["price-close".to_string()].into(),
+        from: Some(-1),
+        ..Default::default()
+    };
+
+    let result = interface.search_and_format(params)?;
+    println!("Latest Bitcoin price: {:?}", result);
+
+    Ok(())
+}
+```
+
+### Web Server Setup
+
+```rust
+use brk::{server, interface, indexer, computer};
+
+#[cfg(all(feature = "server", feature = "interface", feature = "indexer", feature = "computer"))]
+async fn web_server() -> Result<(), Box<dyn std::error::Error>> {
+    let indexer = indexer::Indexer::build("./data")?;
+    let computer = computer::Computer::build("./analytics", &indexer)?;
+    let interface = interface::Interface::build(&indexer, &computer);
+
+    let server = server::Server::new(interface, None);
+    server.serve(true).await?;
+
+    Ok(())
+}
+```
+
+### MCP Integration
+
+```rust
+use brk::{mcp, interface, indexer, computer};
+
+#[cfg(all(feature = "mcp", feature = "interface"))]
+fn mcp_server(interface: &'static interface::Interface) -> mcp::MCP {
+    mcp::MCP::new(interface)
+}
+```
+
+## Feature Combinations
+
+### Common Combinations
+
+**Data Processing**: `["structs", "parser", "indexer"]`
+Basic blockchain data processing and indexing.
+
+**Analytics**: `["indexer", "computer", "fetcher", "interface"]`
+Complete analytics pipeline with price data integration.
+
+**API Server**: `["interface", "server", "logger"]`
+HTTP API server with logging capabilities.
+
+**Full Stack**: `["full"]`
+All components for complete BRK functionality.
+
+### Dependency Optimization
+
+Feature selection allows for significant dependency reduction:
+
+```toml
+# Minimal parser-only dependency
+brk = { version = "0.0.107", features = ["parser"] }
+
+# Analytics without web server
+brk = { version = "0.0.107", features = ["indexer", "computer", "interface"] }
+
+# Web server without parsing
+brk = { version = "0.0.107", features = ["interface", "server"] }
+```
+
+## Architecture
+
+### Re-export Pattern
+
+The crate uses `#[doc(inline)]` re-exports to provide seamless access to component APIs:
+
+```rust
+#[cfg(feature = "component")]
+#[doc(inline)]
+pub use brk_component as component;
+```
+
+This pattern ensures:
+- Feature-gated compilation for dependency optimization
+- Inline documentation for unified API reference
+- Namespace preservation for component-specific functionality
+
+### Build Configuration
+
+- **Documentation**: `all-features = true` for complete docs.rs documentation
+- **CLI Integration**: `brk_cli` always available without feature gates
+- **Optional Dependencies**: All components except CLI are optional
+
+## Configuration
+
+### Feature Flags
+
+| Feature | Component | Description |
+|---------|-----------|-------------|
+| `bundler` | `brk_bundler` | Web asset bundling |
+| `computer` | `brk_computer` | Analytics computation |
+| `error` | `brk_error` | Error handling |
+| `fetcher` | `brk_fetcher` | Price data fetching |
+| `indexer` | `brk_indexer` | Blockchain indexing |
+| `interface` | `brk_interface` | Data query interface |
+| `logger` | `brk_logger` | Enhanced logging |
+| `mcp` | `brk_mcp` | Model Context Protocol |
+| `parser` | `brk_parser` | Block parsing |
+| `server` | `brk_server` | HTTP server |
+| `store` | `brk_store` | Key-value storage |
+| `structs` | `brk_structs` | Data structures |
+| `full` | All components | Complete functionality |
+
+### Documentation
+
+Documentation is aggregated from all components with `#![doc = include_str!("../README.md")]` ensuring comprehensive API reference across all features.
+
+## Code Analysis Summary
+
+**Main Structure**: Feature-gated re-export crate providing unified access to 12 BRK components \
+**Feature System**: Cargo features enabling selective compilation and dependency optimization \
+**CLI Integration**: Always-available `brk_cli` access without feature requirements \
+**Documentation**: Inline re-exports with comprehensive docs.rs integration \
+**Dependency Management**: Optional dependencies for all components except CLI \
+**Build Configuration**: Optimized compilation with all-features documentation \
+**Architecture**: Modular aggregation crate enabling flexible BRK ecosystem usage
+
+---
+
+_This README was generated by Claude Code_

crates/brk/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    let profile = std::env::var("PROFILE").unwrap_or_default();
+
+    if profile == "release" {
+        println!("cargo:rustc-flag=-C");
+        println!("cargo:rustc-flag=target-cpu=native");
+    }
+}

crates/brk/examples/main.rs

@@ -1 +0,0 @@
-fn main() {}

crates/brk/flamegraph.sh

@@ -1,2 +1 @@
-cargo build --profile profiling
-flamegraph -- ../../target/profiling/brk
+sudo cargo flamegraph --profile profiling --root

crates/brk/src/lib.rs

@@ -1,4 +1,8 @@
-#![doc = include_str!(concat!("../", env!("CARGO_PKG_README")))]
+#![doc = include_str!("../README.md")]
+
+#[cfg(feature = "binder")]
+#[doc(inline)]
+pub use brk_binder as binder;
 
 #[cfg(feature = "bundler")]
 #[doc(inline)]
@@ -7,17 +11,17 @@ pub use brk_bundler as bundler;
 #[doc(inline)]
 pub use brk_cli as cli;
 
-#[cfg(feature = "core")]
+#[cfg(feature = "structs")]
 #[doc(inline)]
-pub use brk_core as core;
+pub use brk_structs as structs;
 
 #[cfg(feature = "computer")]
 #[doc(inline)]
 pub use brk_computer as computer;
 
-#[cfg(feature = "exit")]
+#[cfg(feature = "error")]
 #[doc(inline)]
-pub use brk_exit as exit;
+pub use brk_error as error;
 
 #[cfg(feature = "fetcher")]
 #[doc(inline)]
@@ -50,7 +54,3 @@ pub use brk_server as server;
 #[cfg(feature = "store")]
 #[doc(inline)]
 pub use brk_store as store;
-
-#[cfg(feature = "vec")]
-#[doc(inline)]
-pub use brk_vec as vec;

crates/brk_binder/Cargo.toml

@@ -0,0 +1,14 @@
+[package]
+name = "brk_binder"
+description = "A generator of binding files for other languages"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+homepage.workspace = true
+repository.workspace = true
+rust-version.workspace = true
+build = "build.rs"
+
+[dependencies]
+brk_interface = { workspace = true }
+brk_structs = { workspace = true }

crates/brk_binder/README.md

@@ -0,0 +1 @@
+# brk_binder

crates/brk_binder/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    let profile = std::env::var("PROFILE").unwrap_or_default();
+
+    if profile == "release" {
+        println!("cargo:rustc-flag=-C");
+        println!("cargo:rustc-flag=target-cpu=native");
+    }
+}

crates/brk_binder/src/js.rs

@@ -0,0 +1,250 @@
+use std::{
+    collections::{BTreeMap, HashMap},
+    fs, io,
+    path::Path,
+};
+
+use brk_interface::{Index, Interface};
+use brk_structs::pools;
+
+use super::VERSION;
+
+const AUTO_GENERATED_DISCLAIMER: &str = "//
+// File auto-generated, any modifications will be overwritten
+//";
+
+#[allow(clippy::upper_case_acronyms)]
+pub trait Bridge {
+    fn generate_js_files(&self, modules_path: &Path) -> io::Result<()>;
+}
+
+impl Bridge for Interface<'static> {
+    fn generate_js_files(&self, modules_path: &Path) -> io::Result<()> {
+        let path = modules_path.join("brk-client");
+
+        if !fs::exists(&path)? {
+            return Ok(());
+        }
+
+        let path = path.join("generated");
+        fs::create_dir_all(&path)?;
+
+        generate_version_file(&path)?;
+        generate_metrics_file(self, &path)?;
+        generate_pools_file(&path)
+    }
+}
+
+fn generate_version_file(parent: &Path) -> io::Result<()> {
+    let path = parent.join(Path::new("version.js"));
+
+    let contents = format!(
+        "{AUTO_GENERATED_DISCLAIMER}
+
+export const VERSION = \"v{VERSION}\";
+"
+    );
+
+    fs::write(path, contents)
+}
+
+fn generate_pools_file(parent: &Path) -> io::Result<()> {
+    let path = parent.join(Path::new("pools.js"));
+
+    let pools = pools();
+
+    let mut contents = format!("{AUTO_GENERATED_DISCLAIMER}\n");
+
+    contents += "
+/**
+ * @typedef {typeof POOL_ID_TO_POOL_NAME} PoolIdToPoolName
+ * @typedef {keyof PoolIdToPoolName} PoolId
+ */
+
+export const POOL_ID_TO_POOL_NAME = /** @type {const} */ ({
+";
+
+    let mut sorted_pools: Vec<_> = pools.iter().collect();
+    sorted_pools.sort_by(|a, b| a.name.to_lowercase().cmp(&b.name.to_lowercase()));
+
+    contents += &sorted_pools
+        .iter()
+        .map(|pool| {
+            let id = pool.serialized_id();
+            format!("  {id}: \"{}\",", pool.name)
+        })
+        .collect::<Vec<_>>()
+        .join("\n");
+
+    contents += "\n});\n";
+
+    fs::write(path, contents)
+}
+
+fn generate_metrics_file(interface: &Interface<'static>, parent: &Path) -> io::Result<()> {
+    let path = parent.join(Path::new("metrics.js"));
+
+    let indexes = Index::all();
+
+    let mut contents = format!(
+        "{AUTO_GENERATED_DISCLAIMER}
+
+export const INDEXES = /** @type {{const}} */ ([
+{}
+]);
+
+/**
+ * @typedef {{typeof INDEXES[number]}} IndexName
+ */
+",
+        indexes
+            .iter()
+            .map(|i| format!("  \"{}\"", i.serialize_long()))
+            .collect::<Vec<_>>()
+            .join(",\n")
+    );
+
+    // contents += &indexes
+    //     .iter()
+    //     .map(|i| format!(" * @typedef {{\"{}\"}} {i}", i.serialize_long()))
+    //     .collect::<Vec<_>>()
+    //     .join("\n");
+
+    //     contents += &format!(
+    //         "
+    //  * @typedef {{{}}} Index
+    //  */
+    // ",
+    //         indexes
+    //             .iter()
+    //             .map(|i| i.to_string())
+    //             .collect::<Vec<_>>()
+    //             .join(" | ")
+    //     );
+
+    let mut unique_index_groups = BTreeMap::new();
+
+    let mut word_to_freq: BTreeMap<_, usize> = BTreeMap::new();
+    interface
+        .metric_to_index_to_vec()
+        .keys()
+        .for_each(|metric| {
+            metric.split("_").for_each(|word| {
+                *word_to_freq.entry(word).or_default() += 1;
+            });
+        });
+    let mut word_to_freq = word_to_freq.into_iter().collect::<Vec<_>>();
+    word_to_freq.sort_unstable_by(|a, b| b.1.cmp(&a.1).then(a.0.cmp(b.0)));
+    let words = word_to_freq
+        .into_iter()
+        .map(|(str, _)| str)
+        .collect::<Vec<_>>();
+
+    contents += &format!(
+        "
+export const INDEX_TO_WORD = [
+  {}
+];
+
+",
+        words
+            .iter()
+            .enumerate()
+            .map(|(index, word)| format!("\"{word}\", // {}", index_to_letters(index)))
+            .collect::<Vec<_>>()
+            .join("\n  ")
+    );
+
+    let word_to_base62 = words
+        .into_iter()
+        .enumerate()
+        .map(|(i, w)| (w, index_to_letters(i)))
+        .collect::<HashMap<_, _>>();
+
+    let mut ser_metric_to_indexes = "
+/** @type {Record<string, IndexName[]>} */
+export const COMPRESSED_METRIC_TO_INDEXES = {
+"
+    .to_string();
+
+    interface
+        .metric_to_index_to_vec()
+        .iter()
+        .for_each(|(metric, index_to_vec)| {
+            let indexes = index_to_vec
+                .keys()
+                .map(|i| format!("\"{}\"", i.serialize_long()))
+                .collect::<Vec<_>>()
+                .join(", ");
+            let indexes = format!("[{indexes}]");
+            let unique = unique_index_groups.len();
+            let index = index_to_letters(*unique_index_groups.entry(indexes).or_insert(unique));
+
+            let compressed_metric = metric.split('_').fold(String::new(), |mut acc, w| {
+                if !acc.is_empty() {
+                    acc.push('_');
+                }
+                acc.push_str(&word_to_base62[w]);
+                acc
+            });
+
+            ser_metric_to_indexes += &format!("  {compressed_metric}: {index},\n");
+        });
+
+    ser_metric_to_indexes += "};
+";
+
+    let mut sorted_groups: Vec<_> = unique_index_groups.into_iter().collect();
+    sorted_groups.sort_by_key(|(_, index)| *index);
+    sorted_groups.into_iter().for_each(|(group, index)| {
+        let index = index_to_letters(index);
+        contents += &format!("/** @type {{IndexName[]}} */\nconst {index} = {group};\n");
+    });
+
+    contents += &ser_metric_to_indexes;
+
+    fs::write(path, contents)
+}
+
+fn index_to_letters(mut index: usize) -> String {
+    if index < 52 {
+        return (index_to_char(index) as char).to_string();
+    }
+    let mut result = [0u8; 8];
+    let mut pos = 8;
+    loop {
+        pos -= 1;
+        result[pos] = index_to_char(index % 52);
+        index /= 52;
+        if index == 0 {
+            break;
+        }
+        index -= 1;
+    }
+    unsafe { String::from_utf8_unchecked(result[pos..].to_vec()) }
+}
+
+fn index_to_char(index: usize) -> u8 {
+    match index {
+        0..=25 => b'A' + index as u8,
+        26..=51 => b'a' + (index - 26) as u8,
+        _ => unreachable!(),
+    }
+}
+
+// fn letters_to_index(s: &str) -> usize {
+//     let mut result = 0;
+//     for byte in s.bytes() {
+//         let value = char_to_index(byte) as usize;
+//         result = result * 52 + value + 1;
+//     }
+//     result - 1
+// }
+
+// fn char_to_index(byte: u8) -> u8 {
+//     match byte {
+//         b'A'..=b'Z' => byte - b'A',
+//         b'a'..=b'z' => byte - b'a' + 26,
+//         _ => 255, // Invalid
+//     }
+// }

crates/brk_binder/src/lib.rs

@@ -0,0 +1,5 @@
+mod js;
+
+pub use js::Bridge;
+
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");

crates/brk_binder/src/python.rs

@@ -0,0 +1 @@
+// TODO ?

crates/brk_bundler/Cargo.toml

@@ -6,11 +6,13 @@ edition.workspace = true
 license.workspace = true
 homepage.workspace = true
 repository.workspace = true
+rust-version.workspace = true
+build = "build.rs"
 
 [dependencies]
 log = { workspace = true }
-notify = "8.1.0"
-brk_rolldown = "0.1.1"
+notify = "8.2.0"
+brk_rolldown = "0.2.3"
 # brk_rolldown = { path = "../../../rolldown/crates/rolldown"}
 sugar_path = "1.2.0"
 tokio = { workspace = true }

crates/brk_bundler/README.md

@@ -0,0 +1,278 @@
+# brk_bundler
+
+Asset bundling and development server for BRK web interfaces with hot reloading and file watching.
+
+[![Crates.io](https://img.shields.io/crates/v/brk_bundler.svg)](https://crates.io/crates/brk_bundler)
+[![Documentation](https://docs.rs/brk_bundler/badge.svg)](https://docs.rs/brk_bundler)
+
+## Overview
+
+This crate provides a thin wrapper around the Rolldown JavaScript bundler specifically designed for BRK web interface development. It handles asset bundling, file copying, template processing, and development-mode file watching with automatic rebuilds and hot reloading for efficient web development workflows.
+
+**Key Features:**
+
+- JavaScript bundling with Rolldown (Rust-based bundler)
+- Automatic file watching and hot reloading in development mode
+- Template processing with version injection and asset hash replacement
+- Service worker generation with version management
+- Source map generation for debugging
+- Minification for production builds
+- Async/await support with Tokio integration
+
+**Target Use Cases:**
+
+- BRK blockchain explorer web interfaces
+- Development of Bitcoin analytics dashboards
+- Building responsive web applications for blockchain data visualization
+- Hot reloading development environment for rapid iteration
+
+## Installation
+
+```bash
+cargo add brk_bundler
+```
+
+## Quick Start
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let websites_path = Path::new("./web");
+    let source_folder = "src";
+    let watch = true; // Enable hot reloading
+
+    // Bundle assets and start development server
+    let dist_path = bundle(websites_path, source_folder, watch).await?;
+
+    println!("Assets bundled to: {}", dist_path.display());
+
+    // Keep running for file watching (in watch mode)
+    if watch {
+        tokio::signal::ctrl_c().await?;
+    }
+
+    Ok(())
+}
+```
+
+## API Overview
+
+### Core Functions
+
+**`bundle(websites_path: &Path, source_folder: &str, watch: bool) -> io::Result<PathBuf>`**
+Main bundling function that processes web assets and optionally starts file watching.
+
+### Bundling Process
+
+1. **Directory Setup**: Creates `dist/` directory and copies source files
+2. **JavaScript Bundling**: Processes `scripts/entry.js` with Rolldown bundler
+3. **Template Processing**: Updates `index.html` with hashed asset references
+4. **Service Worker**: Generates service worker with version injection
+5. **File Watching**: Optionally monitors source files for changes
+
+### Configuration
+
+**Rolldown Bundler Options:**
+
+- **Input**: `./src/scripts/entry.js` (main JavaScript entry point)
+- **Output**: `./dist/scripts/` directory
+- **Minification**: Enabled for production builds
+- **Source Maps**: File-based source maps for debugging
+- **Asset Hashing**: Automatic hash generation for cache busting
+
+## Examples
+
+### Development Mode with Hot Reloading
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let web_root = Path::new("./websites");
+
+    // Start development server with file watching
+    let _dist_path = bundle(web_root, "explorer", true).await?;
+
+    println!("Development server started!");
+    println!("Hot reloading enabled - edit files to see changes");
+
+    // Keep server running
+    loop {
+        tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
+    }
+}
+```
+
+### Production Build
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let web_root = Path::new("./websites");
+
+    // Build for production (no watching)
+    let dist_path = bundle(web_root, "dashboard", false).await?;
+
+    println!("Production build completed: {}", dist_path.display());
+
+    // Assets are minified and ready for deployment
+    Ok(())
+}
+```
+
+### Custom Web Application Structure
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+// Expected directory structure:
+// websites/
+// ├── my_app/
+// │   ├── index.html          // Main HTML template
+// │   ├── service-worker.js   // Service worker template
+// │   ├── scripts/
+// │   │   └── entry.js        // JavaScript entry point
+// │   ├── styles/
+// │   │   └── main.css        // CSS files
+// │   └── assets/
+// │       └── images/         // Static assets
+// └── dist/                   // Generated output
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let websites_path = Path::new("./websites");
+    let source_folder = "my_app";
+
+    let dist_path = bundle(websites_path, source_folder, false).await?;
+
+    // Result: dist/ contains bundled and processed files
+    // - dist/index.html (with updated script references)
+    // - dist/service-worker.js (with version injection)
+    // - dist/scripts/main.[hash].js (minified and hashed)
+    // - dist/styles/ (copied CSS files)
+    // - dist/assets/ (copied static assets)
+
+    Ok(())
+}
+```
+
+## Architecture
+
+### File Processing Pipeline
+
+1. **Source Copying**: Recursively copies all source files to dist directory
+2. **JavaScript Bundling**: Rolldown processes entry.js with dependencies
+3. **Asset Hashing**: Generates content-based hashes for cache busting
+4. **Template Updates**: Replaces placeholders in HTML templates
+5. **Version Injection**: Updates service worker with current package version
+
+### File Watching System
+
+**Development Mode Watchers:**
+
+- **Source File Watcher**: Monitors non-script files for changes
+- **Bundle Watcher**: Watches JavaScript files and triggers rebuilds
+- **Template Watcher**: Updates HTML when bundled assets change
+
+**Event Handling:**
+
+- **File Creation/Modification**: Automatic copying to dist directory
+- **Script Changes**: Triggers Rolldown rebuild and template update
+- **Template Changes**: Processes HTML and updates asset references
+
+### Template Processing
+
+**index.html Processing:**
+
+- Scans bundled JavaScript for asset hash
+- Replaces `/scripts/main.js` with `/scripts/main.[hash].js`
+- Maintains cache busting while preserving template structure
+
+**service-worker.js Processing:**
+
+- Replaces `__VERSION__` placeholder with current crate version
+- Enables version-based cache invalidation
+- Maintains service worker functionality
+
+### Async Architecture
+
+Built on Tokio async runtime:
+
+- **Non-blocking I/O**: Efficient file operations and watching
+- **Concurrent Tasks**: Parallel file watching and bundle processing
+- **Background Processing**: Development server runs in background task
+
+## Configuration Options
+
+### Rolldown Configuration
+
+The bundler uses optimized Rolldown settings:
+
+```rust
+BundlerOptions {
+    input: Some(vec!["./src/scripts/entry.js".into()]),
+    dir: Some("./dist/scripts".to_string()),
+    minify: Some(RawMinifyOptions::Bool(true)),
+    sourcemap: Some(SourceMapType::File),
+    // ... other default options
+}
+```
+
+### File Structure Requirements
+
+**Required Files:**
+
+- `src/scripts/entry.js` - JavaScript entry point
+- `src/index.html` - HTML template
+- `src/service-worker.js` - Service worker template
+
+**Optional Directories:**
+
+- `src/styles/` - CSS stylesheets
+- `src/assets/` - Static assets (images, fonts, etc.)
+- `src/components/` - Additional JavaScript modules
+
+## Development Workflow
+
+### Setup
+
+1. Create web application in `websites/app_name/`
+2. Add required files (index.html, entry.js, service-worker.js)
+3. Run bundler in watch mode for development
+
+### Hot Reloading
+
+- **Script Changes**: Automatic bundle rebuild and browser refresh
+- **Template Changes**: Immediate HTML update with asset hash replacement
+- **Asset Changes**: Instant copy to dist directory
+- **Style Changes**: Direct copy without bundling
+
+### Production Deployment
+
+1. Run bundler without watch mode
+2. Deploy `dist/` directory contents
+3. Assets include content hashes for cache busting
+4. Service worker includes version for cache management
+
+## Code Analysis Summary
+
+**Main Function**: `bundle()` async function coordinating Rolldown bundler with file processing and watching \
+**File Operations**: Recursive directory copying with `copy_dir_all()` and selective file processing \
+**Templating**: String replacement for asset hash injection and version management \
+**File Watching**: Multi-watcher system using `notify` crate for real-time development feedback \
+**Async Integration**: Tokio-based async architecture with background task spawning \
+**Bundler Integration**: Rolldown wrapper with optimized configuration for web development \
+**Architecture**: Development-focused asset pipeline with hot reloading and production optimization
+
+---
+
+_This README was generated by Claude Code_

crates/brk_bundler/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    let profile = std::env::var("PROFILE").unwrap_or_default();
+
+    if profile == "release" {
+        println!("cargo:rustc-flag=-C");
+        println!("cargo:rustc-flag=target-cpu=native");
+    }
+}

crates/brk_bundler/src/lib.rs

@@ -1,6 +1,15 @@
-use std::{fs, io, path::Path, sync::Arc};
+#![doc = include_str!("../README.md")]
 
-use brk_rolldown::{Bundler, BundlerOptions, RawMinifyOptions, SourceMapType};
+use std::{
+    fs, io,
+    path::{Path, PathBuf},
+    sync::Arc,
+};
+
+use brk_rolldown::{
+    Bundler, BundlerOptions, InlineConstConfig, InlineConstMode, InlineConstOption,
+    OptimizationOption, RawMinifyOptions, SourceMapType,
+};
 use log::error;
 use notify::{EventKind, RecursiveMode, Watcher};
 use sugar_path::SugarPath;
@@ -8,89 +17,114 @@ use tokio::sync::Mutex;
 
 const VERSION: &str = env!("CARGO_PKG_VERSION");
 
-pub async fn bundle(websites_path: &Path, source_folder: &str, watch: bool) -> io::Result<()> {
-    let source_path = websites_path.join(source_folder);
-    let dist_path = websites_path.join("dist");
-
-    let _ = fs::remove_dir_all(&dist_path);
-    copy_dir_all(&source_path, &dist_path)?;
-
-    let source_scripts = format!("./{source_folder}/scripts");
-    let source_entry = format!("{source_scripts}/entry.js");
+pub async fn bundle(
+    modules_path: &Path,
+    websites_path: &Path,
+    source_folder: &str,
+    watch: bool,
+) -> io::Result<PathBuf> {
+    let relative_modules_path = modules_path;
+    let relative_source_path = websites_path.join(source_folder);
+    let relative_dist_path = websites_path.join("dist");
 
+    let absolute_modules_path = relative_modules_path.absolutize();
+    let absolute_modules_path_clone = absolute_modules_path.clone();
     let absolute_websites_path = websites_path.absolutize();
+    let absolute_websites_path_clone = absolute_websites_path.clone();
+
+    let absolute_source_path = relative_source_path.absolutize();
+    let absolute_source_index_path = absolute_source_path.join("index.html");
+    let absolute_source_index_path_clone = absolute_source_index_path.clone();
+    let absolute_source_scripts_path = absolute_source_path.join("scripts");
+    let absolute_source_scripts_modules_path = absolute_source_scripts_path.join("modules");
+    let absolute_source_sw_path = absolute_source_path.join("service-worker.js");
+    let absolute_source_sw_path_clone = absolute_source_sw_path.clone();
+
+    let absolute_dist_path = relative_dist_path.absolutize();
+    let absolute_dist_scripts_path = absolute_dist_path.join("scripts");
+    let absolute_dist_scripts_entry_path = absolute_dist_scripts_path.join("entry.js");
+    let absolute_dist_scripts_entry_path_clone = absolute_dist_scripts_entry_path.clone();
+    let absolute_dist_index_path = absolute_dist_path.join("index.html");
+    let absolute_dist_sw_path = absolute_dist_path.join("service-worker.js");
+
+    let _ = fs::remove_dir_all(&absolute_dist_path);
+    let _ = fs::remove_dir_all(&absolute_source_scripts_modules_path);
+    copy_dir_all(
+        &absolute_modules_path,
+        &absolute_source_scripts_modules_path,
+    )?;
+    copy_dir_all(&absolute_source_path, &absolute_dist_path)?;
+    fs::remove_dir_all(&absolute_dist_scripts_path)?;
+    fs::create_dir(&absolute_dist_scripts_path)?;
+
+    // dbg!(BundlerOptions::default());
 
     let mut bundler = Bundler::new(BundlerOptions {
-        input: Some(vec![source_entry.into()]),
+        input: Some(vec![format!("./{source_folder}/scripts/entry.js").into()]),
         dir: Some("./dist/scripts".to_string()),
         cwd: Some(absolute_websites_path),
         minify: Some(RawMinifyOptions::Bool(true)),
         sourcemap: Some(SourceMapType::File),
+        // advanced_chunks: Some(AdvancedChunksOptions {
+        //     // min_size: Some(1000.0),
+        //     min_share_count: Some(20),
+        //     // min_module_size: S
+        //     // include_dependencies_recursively: Some(true),
+        //     ..Default::default()
+        // }),
+        //
+        // inline_dynamic_imports
+        // experimental: Some(ExperimentalOptions {
+        //     strict_execution_order: Some(true),
+        //     ..Default::default()
+        // }),
+        optimization: Some(OptimizationOption {
+            inline_const: Some(InlineConstOption::Config(InlineConstConfig {
+                mode: Some(InlineConstMode::All),
+                ..Default::default()
+            })),
+            // Needs benchmarks
+            // pife_for_module_wrappers: Some(true),
+            ..Default::default()
+        }),
         ..Default::default()
-    });
+    })
+    .unwrap();
 
-    bundler.write().await.unwrap();
+    if let Err(error) = bundler.write().await {
+        error!("{error:?}");
+    }
 
-    let absolute_source_index_path = source_path.join("index.html").absolutize();
-    let absolute_source_index_path_clone = absolute_source_index_path.clone();
-    let absolute_source_path = source_path.absolutize();
-    let absolute_source_path_clone = absolute_source_path.clone();
-    let absolute_source_scripts_path = websites_path.join(source_scripts).absolutize();
-    let absolute_source_sw_path = source_path.join("service-worker.js").absolutize();
-    let absolute_source_sw_path_clone = absolute_source_sw_path.clone();
-
-    let absolute_dist_entry_path = dist_path.join("scripts/entry.js").absolutize();
-    let absolute_dist_index_path = dist_path.join("index.html").absolutize();
-    let absolute_dist_path = dist_path.absolutize();
-    let absolute_dist_path_clone = absolute_dist_path.clone();
-    let absolute_dist_sw_path = dist_path.join("service-worker.js").absolutize();
-
-    let write_index = move || {
+    let update_dist_index = move || {
         let mut contents = fs::read_to_string(&absolute_source_index_path).unwrap();
 
-        if let Ok(entry) = fs::read_to_string(absolute_dist_path_clone.join("scripts/entry.js")) {
-            if let Some(start) = entry.find("main") {
-                if let Some(end) = entry.find(".js") {
-                    let main_hashed = &entry[start..end];
-                    contents =
-                        contents.replace("/scripts/main.js", &format!("/scripts/{main_hashed}.js"));
-                }
-            }
+        if let Ok(entry) = fs::read_to_string(&absolute_dist_scripts_entry_path_clone)
+            && let Some(start) = entry.find("main")
+            && let Some(end) = entry.find(".js")
+        {
+            let main_hashed = &entry[start..end];
+            contents = contents.replace("/scripts/main.js", &format!("/scripts/{main_hashed}.js"));
         }
 
         let _ = fs::write(&absolute_dist_index_path, contents);
     };
 
-    let write_sw = move || {
+    let update_source_sw = move || {
         let contents = fs::read_to_string(&absolute_source_sw_path)
             .unwrap()
             .replace("__VERSION__", &format!("v{VERSION}"));
         let _ = fs::write(&absolute_dist_sw_path, contents);
     };
 
-    write_index();
-    write_sw();
+    update_dist_index();
+    update_source_sw();
 
     if !watch {
-        return Ok(());
+        return Ok(relative_dist_path);
     }
 
     tokio::spawn(async move {
-        let write_index_clone = write_index.clone();
-
-        let mut entry_watcher = notify::recommended_watcher(
-            move |res: Result<notify::Event, notify::Error>| match res {
-                Ok(_) => write_index_clone(),
-                Err(e) => error!("watch error: {e:?}"),
-            },
-        )
-        .unwrap();
-
-        entry_watcher
-            .watch(&absolute_dist_entry_path, RecursiveMode::Recursive)
-            .unwrap();
-
-        let mut source_watcher = notify::recommended_watcher(
+        let mut event_watcher = notify::recommended_watcher(
             move |res: Result<notify::Event, notify::Error>| match res {
                 Ok(event) => match event.kind {
                     EventKind::Create(_) => event.paths,
@@ -98,18 +132,30 @@ pub async fn bundle(websites_path: &Path, source_folder: &str, watch: bool) -> i
                     _ => vec![],
                 }
                 .into_iter()
-                .filter(|path| path.starts_with(&absolute_source_path))
-                .filter(|path| !path.starts_with(&absolute_source_scripts_path))
-                .for_each(|source_path| {
-                    let suffix = source_path.strip_prefix(&absolute_source_path).unwrap();
-                    let dist_path = absolute_dist_path.join(suffix);
+                .for_each(|path| {
+                    let path = path.absolutize();
 
-                    if source_path == absolute_source_index_path_clone {
-                        write_index();
-                    } else if source_path == absolute_source_sw_path_clone {
-                        write_sw();
-                    } else {
-                        let _ = fs::copy(&source_path, &dist_path);
+                    if path == absolute_dist_scripts_entry_path
+                        || path == absolute_source_index_path_clone
+                    {
+                        update_dist_index();
+                    } else if path == absolute_source_sw_path_clone {
+                        update_source_sw();
+                    } else if let Ok(suffix) = path.strip_prefix(&absolute_modules_path) {
+                        let source_modules_path = absolute_source_scripts_modules_path.join(suffix);
+                        if path.is_file() {
+                            let _ = fs::create_dir_all(path.parent().unwrap());
+                            let _ = fs::copy(&path, &source_modules_path);
+                        }
+                    } else if let Ok(suffix) = path.strip_prefix(&absolute_source_path)
+                        // scripts are handled by rolldown
+                        && !path.starts_with(&absolute_source_scripts_path)
+                    {
+                        let dist_path = absolute_dist_path.join(suffix);
+                        if path.is_file() {
+                            let _ = fs::create_dir_all(path.parent().unwrap());
+                            let _ = fs::copy(&path, &dist_path);
+                        }
                     }
                 }),
                 Err(e) => error!("watch error: {e:?}"),
@@ -117,8 +163,11 @@ pub async fn bundle(websites_path: &Path, source_folder: &str, watch: bool) -> i
         )
         .unwrap();
 
-        source_watcher
-            .watch(&absolute_source_path_clone, RecursiveMode::Recursive)
+        event_watcher
+            .watch(&absolute_websites_path_clone, RecursiveMode::Recursive)
+            .unwrap();
+        event_watcher
+            .watch(&absolute_modules_path_clone, RecursiveMode::Recursive)
             .unwrap();
 
         let watcher =
@@ -127,7 +176,7 @@ pub async fn bundle(websites_path: &Path, source_folder: &str, watch: bool) -> i
         watcher.start().await;
     });
 
-    Ok(())
+    Ok(relative_dist_path)
 }
 
 fn copy_dir_all(src: impl AsRef<Path>, dst: impl AsRef<Path>) -> io::Result<()> {

crates/brk_cli/Cargo.toml

@@ -1,30 +1,35 @@
 [package]
 name = "brk_cli"
-description = "A command line interface to run a Bitcoin Research Kit instance"
+description = "A command line interface to run a BRK instance"
 version.workspace = true
 edition.workspace = true
 license.workspace = true
 homepage.workspace = true
 repository.workspace = true
+rust-version.workspace = true
+build = "build.rs"
 
 [dependencies]
 bitcoincore-rpc = { workspace = true }
+brk_binder = { workspace = true }
+brk_bundler = { workspace = true }
 brk_computer = { workspace = true }
-brk_core = { workspace = true }
-brk_exit = { workspace = true }
+brk_error = { workspace = true }
 brk_fetcher = { workspace = true }
 brk_indexer = { workspace = true }
+brk_interface = { workspace = true }
 brk_logger = { workspace = true }
 brk_parser = { workspace = true }
 brk_server = { workspace = true }
-brk_vec = { workspace = true }
-clap = { workspace = true }
-clap_derive = { workspace = true }
-color-eyre = { workspace = true }
+vecdb = { workspace = true }
+clap = { version = "4.5.48", features = ["derive", "string"] }
+color-eyre = "0.6.5"
 log = { workspace = true }
+minreq = { workspace = true }
 serde = { workspace = true }
 tokio = { workspace = true }
-toml = "0.9.1"
+toml = "0.9.7"
+zip = { version = "5.1.1", default-features = false, features = ["deflate"] }
 
 [[bin]]
 name = "brk"

crates/brk_cli/README.md

@@ -1,96 +1,232 @@
-# BRK CLI
+# brk_cli
 
-<p align="left">
-  <a href="https://github.com/bitcoinresearchkit/brk">
-    <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
-  </a>
-  <a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
-    <img src="https://img.shields.io/crates/l/brk" alt="License" />
-  </a>
-  <a href="https://crates.io/crates/brk_cli">
-    <img src="https://img.shields.io/crates/v/brk_cli" alt="Version" />
-  </a>
-  <a href="https://docs.rs/brk_cli">
-    <img src="https://img.shields.io/docsrs/brk_cli" alt="Documentation" />
-  </a>
-  <img src="https://img.shields.io/crates/size/brk_cli" alt="Size" />
-  <a href="https://deps.rs/crate/brk_cli">
-    <img src="https://deps.rs/crate/brk_cli/latest/status.svg" alt="Dependency status">
-  </a>
-  <a href="https://discord.gg/HaR3wpH3nr">
-    <img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
-  </a>
-  <a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
-    <img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
-  </a>
-  <a href="https://bsky.app/profile/bitcoinresearchkit.org">
-    <img src="https://img.shields.io/badge/bluesky-blue?link=https%3A%2F%2Fbsky.app%2Fprofile%2Fbitcoinresearchkit.org" alt="Bluesky" />
-  </a>
-  <a href="https://x.com/brkdotorg">
-    <img src="https://img.shields.io/badge/x.com-black" alt="X" />
-  </a>
-</p>
+Command-line interface orchestrating complete Bitcoin Research Kit instances with automatic configuration and continuous blockchain processing.
 
-A command line interface to run a Bitcoin Research Kit instance.
+[![Crates.io](https://img.shields.io/crates/v/brk_cli.svg)](https://crates.io/crates/brk_cli)
+[![Documentation](https://docs.rs/brk_cli/badge.svg)](https://docs.rs/brk_cli)
 
-It's very customizable with all parameters from the underlying tools (crates) used inside.
+## Overview
 
-Run `brk -h` for more information.
+This crate provides the primary command-line interface for running Bitcoin Research Kit instances. It orchestrates the entire data processing pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with persistent configuration management, automatic error recovery, and continuous blockchain synchronization.
 
-## Requirements
+**Key Features:**
 
-### Hardware
+- Complete BRK pipeline orchestration with parser, indexer, computer, and server coordination
+- Persistent configuration system with TOML-based auto-save functionality
+- Continuous blockchain processing with new block detection and incremental updates
+- Flexible Bitcoin Core RPC authentication with cookie file and user/password support
+- Configurable web interface options including auto-downloading from GitHub releases
+- Large stack allocation (512MB) for handling complex blockchain processing workloads
+- Graceful shutdown handling with proper cleanup and state preservation
 
-#### Recommended
+**Target Use Cases:**
 
-- [Latest base model Mac mini](https://www.apple.com/mac-mini/)
-- [Thunderbolt 4 SSD enclosure](https://satechi.net/products/usb4-nvme-ssd-pro-enclosure/Z2lkOi8vc2hvcGlmeS9Qcm9kdWN0VmFyaWFudC80MDE4ODQ3MDA2NzI4OA==?queryID=7961465089021ee203a60db7e62e90d2)
-- [2 TB NVMe SSD](https://shop.sandisk.com/products/ssd/internal-ssd/wd-black-sn850x-nvme-ssd?sku=WDS200T2X0E-00BCA0)
+- Production Bitcoin analytics deployments requiring full pipeline operation
+- Development environments for Bitcoin research and analysis
+- Continuous blockchain monitoring with real-time data updates
+- Academic research requiring comprehensive historical blockchain datasets
 
-#### Minimum
-
-To be determined
-
-### Software
-
-- [Bitcoin](https://bitcoin.org/en/full-node)
-- [Rust](https://www.rust-lang.org/tools/install)
-- Unix based operating system (Mac OS or Linux)
-
-> [!IMPORTANT]
-> Ubuntu users need to install `open-ssl` via `sudo apt install libssl-dev pkg-config`
-
-## Download
-
-### Binaries
-
-You can find a pre-built binary for your operating system in the [releases page](https://github.com/bitcoinresearchkit/brk/releases/latest).
-
-### Cargo
+## Installation
 
 ```bash
-# Install
-cargo install brk # or `cargo install brk_cli`, the result is the same
-
-# Update
-cargo install brk # or `cargo install-update -a` if you have `cargo-update` installed
+cargo install brk # or cargo install brk_cli
 ```
 
-### Source
+## Quick Start
 
 ```bash
-git clone https://github.com/bitcoinresearchkit/brk.git
-cd brk/crates/brk
-cargo run -r
+# First run - configure and start processing
+brk --brkdir ./data --bitcoindir ~/.bitcoin --fetch true
+
+# Subsequent runs use saved configuration
+brk
+
+# Override specific options
+brk --website none --fetch false
 ```
 
-## Usage
+## API Overview
 
-Run `brk -h` to view each available parameter and their respective description.
+### Core Structure
 
-> [!TIP]
-> Every parameter set will be saved at `~/.brk/config.toml`, which allows you to simply run `brk` next time.
+- **`Config`**: Persistent configuration with clap-based CLI parsing and TOML serialization
+- **`Bridge`**: Interface trait for generating JavaScript bridge files for web interfaces
+- **`Website`**: Enum for web interface options (None, Bitview, Custom)
+- **Path Functions**: Cross-platform default path resolution for Bitcoin and BRK directories
 
-## Tunnel
+### Main Operations
 
-The easiest way to let others access your server is to use `cloudflared` which will also cache requests. For more information see [Cloudflare Tunnel](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/) documentation.
+**`main() -> color_eyre::Result<()>`**
+Entry point with error handling setup, directory creation, logging initialization, and high-stack thread spawning.
+
+**`run() -> color_eyre::Result<()>`**
+Core processing loop handling configuration, RPC connection, component initialization, and continuous blockchain monitoring.
+
+### Configuration Management
+
+**Persistent Settings:**
+
+- All CLI arguments automatically saved to `~/.brk/config.toml`
+- Argument overrides update saved configuration on each run
+- Cross-platform path resolution with tilde and $HOME expansion
+- Validation of Bitcoin directory, blocks directory, and RPC authentication
+
+**CLI Parameters:**
+
+- `--bitcoindir`, `--blocksdir`, `--brkdir`: Directory configuration
+- `--fetch`, `--exchanges`: Data source configuration
+- `--website`: Web interface selection
+- `--rpcconnect`, `--rpcport`, `--rpccookiefile`, `--rpcuser`, `--rpcpassword`: RPC settings
+
+## Examples
+
+### Basic Usage
+
+```bash
+# Initialize with custom directories
+brk --bitcoindir /data/bitcoin --brkdir /data/brk
+
+# Enable all features with custom RPC
+brk --fetch true --exchanges true --website bitview \
+    --rpcuser myuser --rpcpassword mypass
+
+# Minimal setup with API only
+brk --website none --fetch false
+```
+
+### Configuration File Example
+
+After first run, settings are saved to `~/.brk/config.toml`:
+
+```toml
+bitcoindir = "/home/user/.bitcoin"
+blocksdir = "/home/user/.bitcoin/blocks"
+brkdir = "/home/user/brk_data"
+fetch = true
+exchanges = true
+website = "bitview"
+rpcconnect = "localhost"
+rpcport = 8332
+rpccookiefile = "/home/user/.bitcoin/.cookie"
+```
+
+### Web Interface Configuration
+
+```bash
+# Use built-in Bitview interface
+brk --website bitview
+
+# Use custom web interface
+brk --website custom
+
+# API only, no web interface
+brk --website none
+```
+
+### Development Mode
+
+```bash
+# Development with local website directory
+# Looks for ../../websites directory first
+brk --website bitview
+
+# Production with auto-download from GitHub
+# Downloads websites from release artifacts
+brk --website bitview
+```
+
+## Architecture
+
+### Startup Sequence
+
+1. **Environment Setup**: Color eyre error handling, directory creation, logging initialization
+2. **High-Stack Thread**: 512MB stack for complex blockchain processing operations
+3. **Configuration Loading**: CLI parsing, TOML reading, argument merging, validation
+4. **Component Initialization**: Parser, indexer, computer, interface creation with proper dependencies
+
+### Processing Pipeline
+
+**Continuous Operation Loop:**
+
+1. **Bitcoin Core Sync Wait**: Monitors `headers == blocks` for full node synchronization
+2. **Block Count Detection**: Compares current and previous block counts for new block detection
+3. **Indexing Phase**: Processes new blocks through parser with collision detection option
+4. **Computing Phase**: Runs analytics computations on newly indexed data
+5. **Server Operation**: Serves HTTP API with optional web interface throughout processing
+
+### Web Interface Integration
+
+**Website Handling:**
+
+- **Development Mode**: Uses local `../../websites` directory if available
+- **Production Mode**: Downloads release artifacts from GitHub using semantic versioning
+- **Bundle Generation**: Creates optimized JavaScript bundles using `brk_bundler`
+- **Bridge Files**: Generates JavaScript bridge files for vector IDs and pool data
+
+**Download and Bundle Process:**
+
+```rust
+// Automatic website download and bundling
+let url = format!("https://github.com/bitcoinresearchkit/brk/archive/refs/tags/v{VERSION}.zip");
+let response = minreq::get(url).send()?;
+zip::ZipArchive::new(cursor).extract(downloads_path)?;
+bundle(&websites_path, website.to_folder_name(), true).await?
+```
+
+### RPC Authentication
+
+**Flexible Authentication Methods:**
+
+- **Cookie File**: Automatic detection at `--bitcoindir/.cookie`
+- **User/Password**: Manual configuration with `--rpcuser` and `--rpcpassword`
+- **Connection Validation**: Startup checks ensure proper Bitcoin Core connectivity
+
+### Configuration System
+
+**TOML Persistence:**
+
+- Automatic serialization/deserialization with `serde` and `toml`
+- Error-tolerant parsing with `default_on_error` deserializer
+- Argument consumption validation ensuring all CLI options are processed
+- Path expansion supporting `~` and `$HOME` environment variables
+
+## Configuration
+
+### Default Paths
+
+**Cross-Platform Path Resolution:**
+
+- **Linux**: `~/.bitcoin` for Bitcoin Core, `~/.brk` for BRK data
+- **macOS**: `~/Library/Application Support/Bitcoin` for Bitcoin Core
+- **Logs**: `~/.brk/log` for application logging
+- **Downloads**: `~/.brk/downloads` for temporary website artifacts
+
+### Performance Settings
+
+**Memory Management:**
+
+- 512MB stack size for main processing thread
+- Multi-threaded tokio runtime with all features enabled
+- Persistent configuration caching to minimize I/O operations
+
+### Error Handling
+
+**Comprehensive Validation:**
+
+- Directory existence checks with user-friendly error messages
+- RPC authentication verification before processing begins
+- Graceful exit with help suggestions for configuration issues
+
+## Code Analysis Summary
+
+**Main Structure**: `Config` struct with clap-derived CLI parsing and persistent TOML configuration management \
+**Processing Loop**: Continuous Bitcoin Core monitoring with sync detection and incremental block processing \
+**Web Integration**: Automatic website download from GitHub releases with JavaScript bundle generation \
+**Component Orchestration**: Coordination of parser, indexer, computer, and server with proper dependency management \
+**Error Handling**: `color_eyre` integration with comprehensive validation and user-friendly error messages \
+**Threading**: High-stack thread allocation (512MB) with tokio multi-threaded runtime for complex operations \
+**Architecture**: Complete BRK pipeline orchestration with persistent configuration and continuous blockchain synchronization
+
+---
+
+_This README was generated by Claude Code_

crates/brk_cli/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    let profile = std::env::var("PROFILE").unwrap_or_default();
+
+    if profile == "release" {
+        println!("cargo:rustc-flag=-C");
+        println!("cargo:rustc-flag=target-cpu=native");
+    }
+}

crates/brk_cli/src/config.rs

@@ -4,16 +4,14 @@ use std::{
 };
 
 use bitcoincore_rpc::{self, Auth, Client};
-use brk_core::{default_bitcoin_path, default_brk_path, default_on_error, dot_brk_path};
 use brk_fetcher::Fetcher;
-use brk_server::Website;
-use brk_vec::{Computation, Format};
 use clap::Parser;
-use clap_derive::Parser;
 use color_eyre::eyre::eyre;
-use serde::{Deserialize, Serialize};
+use serde::{Deserialize, Deserializer, Serialize};
 
-use crate::services::Services;
+use crate::{default_bitcoin_path, default_brk_path, dot_brk_path, website::Website};
+
+const DOWNLOADS: &str = "downloads";
 
 #[derive(Parser, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize)]
 #[command(version, about)]
@@ -33,27 +31,17 @@ pub struct Config {
     #[arg(long, value_name = "PATH")]
     brkdir: Option<String>,
 
-    /// Activated services, default: all, saved
-    #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(short, long)]
-    services: Option<Services>,
-
-    /// Computation of computed datasets, `lazy` computes data whenever requested without saving it, `eager` computes the data once and saves it to disk, default: `lazy`, saved
-    #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(short, long)]
-    computation: Option<Computation>,
-
-    /// Format of computed datasets, `compressed` to save disk space (experimental), `raw` to prioritize speed, default: `raw`, saved
-    #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(short, long)]
-    format: Option<Format>,
-
-    /// Activate fetching prices from exchanges APIs and the computation of all related datasets, default: true, saved
+    /// Activate fetching prices from BRK's API and the computation of all price related datasets, default: true, saved
     #[serde(default, deserialize_with = "default_on_error")]
     #[arg(short = 'F', long, value_name = "BOOL")]
     fetch: Option<bool>,
 
-    /// Website served by the server (if active), default: default, saved
+    /// Activate fetching prices from exchanges APIs if `fetch` is also set to `true`, default: true, saved
+    #[serde(default, deserialize_with = "default_on_error")]
+    #[arg(long, value_name = "BOOL")]
+    exchanges: Option<bool>,
+
+    /// Website served by the server, default: default, saved
     #[serde(default, deserialize_with = "default_on_error")]
     #[arg(short, long)]
     website: Option<Website>,
@@ -83,24 +71,9 @@ pub struct Config {
     #[arg(long, value_name = "PASSWORD")]
     rpcpassword: Option<String>,
 
-    /// Delay between runs, default: 0, saved
-    #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(long, value_name = "SECONDS")]
-    delay: Option<u64>,
-
-    /// Activate the Model Context Protocol (MCP) endpoint to give LLMs access to BRK (experimental), default: true, saved
-    #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(long, value_name = "BOOL")]
-    mcp: Option<bool>,
-
-    /// DEV: Activate watching the selected website's folder for changes, default: false, saved
-    #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(long, value_name = "BOOL")]
-    watch: Option<bool>,
-
     /// DEV: Activate checking address hashes for collisions when indexing, default: false, saved
     #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(long, value_name = "BOOL")]
+    #[arg(skip)]
     check_collisions: Option<bool>,
 }
 
@@ -129,20 +102,12 @@ impl Config {
                 config_saved.brkdir = Some(brkdir);
             }
 
-            if let Some(services) = config_args.services.take() {
-                config_saved.services = Some(services);
-            }
-
-            if let Some(computation) = config_args.computation.take() {
-                config_saved.computation = Some(computation);
-            }
-
             if let Some(fetch) = config_args.fetch.take() {
                 config_saved.fetch = Some(fetch);
             }
 
-            if let Some(format) = config_args.format.take() {
-                config_saved.format = Some(format);
+            if let Some(exchanges) = config_args.exchanges.take() {
+                config_saved.exchanges = Some(exchanges);
             }
 
             if let Some(website) = config_args.website.take() {
@@ -169,22 +134,10 @@ impl Config {
                 config_saved.rpcpassword = Some(rpcpassword);
             }
 
-            if let Some(delay) = config_args.delay.take() {
-                config_saved.delay = Some(delay);
-            }
-
             if let Some(check_collisions) = config_args.check_collisions.take() {
                 config_saved.check_collisions = Some(check_collisions);
             }
 
-            if let Some(mcp) = config_args.mcp.take() {
-                config_saved.mcp = Some(mcp);
-            }
-
-            if let Some(watch) = config_args.watch.take() {
-                config_saved.watch = Some(watch);
-            }
-
             if config_args != Config::default() {
                 dbg!(config_args);
                 panic!("Didn't consume the full config")
@@ -224,7 +177,9 @@ impl Config {
 
         if self.rpc_auth().is_err() {
             println!(
-                "No way found to authenticate the RPC client, please either set --rpccookiefile or --rpcuser and --rpcpassword.\nRun the program with '-h' for help."
+                "Unsuccessful authentication with the RPC client.
+First make sure that `bitcoind` is running. If it is then please either set --rpccookiefile or --rpcuser and --rpcpassword as the default values seemed to have failed.
+Finally, you can run the program with '-h' for help."
             );
             std::process::exit(1);
         }
@@ -275,10 +230,6 @@ impl Config {
         self.rpcport
     }
 
-    pub fn delay(&self) -> Option<u64> {
-        self.delay
-    }
-
     pub fn bitcoindir(&self) -> PathBuf {
         self.bitcoindir
             .as_ref()
@@ -298,22 +249,12 @@ impl Config {
             .map_or_else(default_brk_path, |s| Self::fix_user_path(s.as_ref()))
     }
 
-    pub fn outputsdir(&self) -> PathBuf {
-        self.brkdir().join("outputs")
-    }
-
     pub fn harsdir(&self) -> PathBuf {
-        self.outputsdir().join("hars")
+        self.brkdir().join("hars")
     }
 
-    pub fn process(&self) -> bool {
-        self.services
-            .is_none_or(|m| m == Services::All || m == Services::Processor)
-    }
-
-    pub fn serve(&self) -> bool {
-        self.services
-            .is_none_or(|m| m == Services::All || m == Services::Server)
+    pub fn downloads_dir(&self) -> PathBuf {
+        dot_brk_path().join(DOWNLOADS)
     }
 
     fn path_cookiefile(&self) -> PathBuf {
@@ -342,35 +283,34 @@ impl Config {
     }
 
     pub fn website(&self) -> Website {
-        self.website.unwrap_or(Website::Default)
+        self.website.unwrap_or(Website::Bitview)
     }
 
     pub fn fetch(&self) -> bool {
         self.fetch.is_none_or(|b| b)
     }
 
+    pub fn exchanges(&self) -> bool {
+        self.exchanges.is_none_or(|b| b)
+    }
+
     pub fn fetcher(&self) -> Option<Fetcher> {
         self.fetch()
-            .then(|| Fetcher::import(Some(self.harsdir().as_path())).unwrap())
-    }
-
-    pub fn computation(&self) -> Computation {
-        self.computation.unwrap_or_default()
-    }
-
-    pub fn format(&self) -> Format {
-        self.format.unwrap_or_default()
+            .then(|| Fetcher::import(self.exchanges(), Some(self.harsdir().as_path())).unwrap())
     }
 
     pub fn check_collisions(&self) -> bool {
         self.check_collisions.is_some_and(|b| b)
     }
+}
 
-    pub fn mcp(&self) -> bool {
-        self.mcp.is_none_or(|b| b)
-    }
-
-    pub fn watch(&self) -> bool {
-        self.watch.is_some_and(|b| b)
+fn default_on_error<'de, D, T>(deserializer: D) -> Result<T, D::Error>
+where
+    D: Deserializer<'de>,
+    T: Deserialize<'de> + Default,
+{
+    match T::deserialize(deserializer) {
+        Ok(v) => Ok(v),
+        Err(_) => Ok(T::default()),
     }
 }

crates/brk_cli/src/lib.rs

@@ -1,23 +1,167 @@
-use std::{fs, thread};
+#![doc = include_str!("../README.md")]
 
-use brk_core::{dot_brk_log_path, dot_brk_path};
+use std::{
+    fs,
+    io::Cursor,
+    path::Path,
+    thread::{self, sleep},
+    time::Duration,
+};
+
+use bitcoincore_rpc::{self, RpcApi};
+use brk_binder::Bridge;
+use brk_bundler::bundle;
+use brk_computer::Computer;
+use brk_error::Result;
+use brk_indexer::Indexer;
+use brk_interface::Interface;
+use brk_parser::Parser;
+use brk_server::{Server, VERSION};
+use log::info;
+use vecdb::Exit;
 
 mod config;
-mod run;
-mod services;
+mod paths;
+mod website;
 
-use run::*;
+use crate::{config::Config, paths::*};
 
 pub fn main() -> color_eyre::Result<()> {
     color_eyre::install()?;
 
     fs::create_dir_all(dot_brk_path())?;
 
-    brk_logger::init(Some(&dot_brk_log_path()));
+    brk_logger::init(Some(&dot_brk_log_path()))?;
 
     thread::Builder::new()
-        .stack_size(256 * 1024 * 1024)
+        .stack_size(512 * 1024 * 1024)
         .spawn(run)?
         .join()
         .unwrap()
 }
+
+pub fn run() -> color_eyre::Result<()> {
+    let config = Config::import()?;
+
+    let rpc = config.rpc()?;
+
+    let exit = Exit::new();
+    exit.set_ctrlc_handler();
+
+    let parser = Parser::new(config.blocksdir(), rpc);
+
+    let mut indexer = Indexer::forced_import(&config.brkdir())?;
+
+    let mut computer = Computer::forced_import(&config.brkdir(), &indexer, config.fetcher())?;
+
+    let interface = Interface::build(&parser, &indexer, &computer);
+
+    let website = config.website();
+
+    let downloads_path = config.downloads_dir();
+
+    let future = async move {
+        let bundle_path = if website.is_some() {
+            let websites_dev_path = Path::new("../../websites");
+            let modules_dev_path = Path::new("../../modules");
+
+            let websites_path;
+            let modules_path;
+
+            if fs::exists(websites_dev_path)? && fs::exists(modules_dev_path)? {
+                websites_path = websites_dev_path.to_path_buf();
+                modules_path = modules_dev_path.to_path_buf();
+            } else {
+                let downloaded_brk_path = downloads_path.join(format!("brk-{VERSION}"));
+
+                let downloaded_websites_path = downloaded_brk_path.join("websites");
+                let downloaded_modules_path = downloaded_brk_path.join("modules");
+
+                if !fs::exists(&downloaded_websites_path)? {
+                    info!("Downloading source from Github...");
+
+                    let url = format!(
+                        "https://github.com/bitcoinresearchkit/brk/archive/refs/tags/v{VERSION}.zip",
+                    );
+
+                    let response = minreq::get(url).send()?;
+                    let bytes = response.as_bytes();
+                    let cursor = Cursor::new(bytes);
+
+                    let mut zip = zip::ZipArchive::new(cursor).unwrap();
+
+                    zip.extract(downloads_path).unwrap();
+                }
+
+                websites_path = downloaded_websites_path;
+                modules_path = downloaded_modules_path;
+            }
+
+            interface.generate_js_files(&modules_path)?;
+
+            Some(
+                bundle(
+                    &modules_path,
+                    &websites_path,
+                    website.to_folder_name(),
+                    true,
+                )
+                .await?,
+            )
+        } else {
+            None
+        };
+
+        let server = Server::new(interface, bundle_path);
+
+        tokio::spawn(async move {
+            server.serve(true).await.unwrap();
+        });
+
+        Ok(()) as Result<()>
+    };
+
+    let runtime = tokio::runtime::Builder::new_multi_thread()
+        .enable_all()
+        .build()?;
+
+    let _handle = runtime.spawn(future);
+
+    loop {
+        wait_for_synced_node(rpc)?;
+
+        let block_count = rpc.get_block_count()?;
+
+        info!("{} blocks found.", block_count + 1);
+
+        let starting_indexes = indexer
+            .index(&parser, rpc, &exit, config.check_collisions())
+            .unwrap();
+
+        computer
+            .compute(&indexer, starting_indexes, &parser, &exit)
+            .unwrap();
+
+        info!("Waiting for new blocks...");
+
+        while block_count == rpc.get_block_count()? {
+            sleep(Duration::from_secs(1))
+        }
+    }
+}
+
+fn wait_for_synced_node(rpc_client: &bitcoincore_rpc::Client) -> color_eyre::Result<()> {
+    let is_synced = || -> color_eyre::Result<bool> {
+        let info = rpc_client.get_blockchain_info()?;
+        Ok(info.headers == info.blocks)
+    };
+
+    if !is_synced()? {
+        info!("Waiting for node to sync...");
+        while !is_synced()? {
+            sleep(Duration::from_secs(1))
+        }
+    }
+
+    Ok(())
+}

crates/brk_cli/src/run.rs

@@ -1,103 +0,0 @@
-use std::{thread::sleep, time::Duration};
-
-use bitcoincore_rpc::{self, RpcApi};
-use brk_computer::Computer;
-use brk_exit::Exit;
-use brk_indexer::Indexer;
-use brk_server::Server;
-use log::info;
-
-use crate::config::Config;
-
-pub fn run() -> color_eyre::Result<()> {
-    let config = Config::import()?;
-
-    let rpc = config.rpc()?;
-
-    let exit = Exit::new();
-
-    let parser = brk_parser::Parser::new(config.blocksdir(), rpc);
-
-    let format = config.format();
-
-    let mut indexer = Indexer::forced_import(&config.outputsdir())?;
-
-    let wait_for_synced_node = || -> color_eyre::Result<()> {
-        let is_synced = || -> color_eyre::Result<bool> {
-            let info = rpc.get_blockchain_info()?;
-            Ok(info.headers == info.blocks)
-        };
-
-        if !is_synced()? {
-            info!("Waiting for node to be synced...");
-            while !is_synced()? {
-                sleep(Duration::from_secs(1))
-            }
-        }
-
-        Ok(())
-    };
-
-    let mut computer = Computer::forced_import(
-        &config.outputsdir(),
-        &indexer,
-        config.computation(),
-        config.fetcher(),
-        format,
-    )?;
-
-    tokio::runtime::Builder::new_multi_thread()
-        .enable_all()
-        .build()?
-        .block_on(async {
-            let server = if config.serve() {
-                let served_indexer = indexer.clone();
-                let served_computer = computer.clone();
-
-                let server = Server::new(served_indexer, served_computer, config.website())?;
-
-                let watch = config.watch();
-                let mcp = config.mcp();
-                let opt = Some(tokio::spawn(async move {
-                    server.serve(watch, mcp).await.unwrap();
-                }));
-
-                sleep(Duration::from_secs(1));
-
-                opt
-            } else {
-                None
-            };
-
-            if config.process() {
-                loop {
-                    wait_for_synced_node()?;
-
-                    let block_count = rpc.get_block_count()?;
-
-                    info!("{} blocks found.", block_count + 1);
-
-                    let starting_indexes =
-                        indexer.index(&parser, rpc, &exit, config.check_collisions())?;
-
-                    computer.compute(&mut indexer, starting_indexes, &exit)?;
-
-                    if let Some(delay) = config.delay() {
-                        sleep(Duration::from_secs(delay))
-                    }
-
-                    info!("Waiting for new blocks...");
-
-                    while block_count == rpc.get_block_count()? {
-                        sleep(Duration::from_secs(1))
-                    }
-                }
-            }
-
-            if let Some(handle) = server {
-                handle.await.unwrap();
-            }
-
-            Ok(())
-        })
-}

crates/brk_cli/src/services.rs

@@ -1,23 +0,0 @@
-use clap_derive::{Parser, ValueEnum};
-use serde::{Deserialize, Serialize};
-
-#[derive(
-    Default,
-    Debug,
-    Clone,
-    Copy,
-    Parser,
-    ValueEnum,
-    Serialize,
-    Deserialize,
-    PartialEq,
-    Eq,
-    PartialOrd,
-    Ord,
-)]
-pub enum Services {
-    #[default]
-    All,
-    Processor,
-    Server,
-}

crates/brk_cli/src/website.rs

@@ -1,10 +1,11 @@
-use clap_derive::ValueEnum;
+use clap::ValueEnum;
 use serde::{Deserialize, Serialize};
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize, ValueEnum)]
+#[serde(rename_all = "lowercase")]
 pub enum Website {
     None,
-    Default,
+    Bitview,
     Custom,
 }
 
@@ -17,10 +18,10 @@ impl Website {
         !self.is_none()
     }
 
-    pub fn to_folder_name(&self) -> &str {
+    pub fn to_folder_name(self) -> &'static str {
         match self {
             Self::Custom => "custom",
-            Self::Default => "default",
+            Self::Bitview => "bitview",
             Self::None => unreachable!(),
         }
     }

crates/brk_computer/Cargo.toml

@@ -1,34 +1,30 @@
 [package]
 name = "brk_computer"
-description = "A Bitcoin dataset computer, built on top of brk_indexer"
+description = "A Bitcoin dataset computer built on top of brk_indexer"
 version.workspace = true
 edition.workspace = true
 license.workspace = true
 homepage.workspace = true
 repository.workspace = true
+rust-version.workspace = true
+build = "build.rs"
 
 [dependencies]
-bincode = { workspace = true }
+allocative = { workspace = true }
 bitcoin = { workspace = true }
 bitcoincore-rpc = { workspace = true }
-brk_core = { workspace = true }
-brk_exit = { workspace = true }
+brk_structs = { workspace = true }
+brk_error = { workspace = true }
 brk_fetcher = { workspace = true }
 brk_indexer = { workspace = true }
 brk_logger = { workspace = true }
-brk_parser = { workspace = true }
 brk_store = { workspace = true }
-brk_vec = { workspace = true }
-color-eyre = { workspace = true }
+brk_parser = { workspace = true }
+vecdb = { workspace = true }
 derive_deref = { workspace = true }
-either = "1.15.0"
-fjall = { workspace = true }
-jiff = { workspace = true }
 log = { workspace = true }
+pco = "0.4.6"
 rayon = { workspace = true }
 serde = { workspace = true }
 zerocopy = { workspace = true }
 zerocopy-derive = { workspace = true }
-
-[package.metadata.cargo-machete]
-ignored = ["zerocopy"]

crates/brk_computer/README.md

@@ -1,34 +1,303 @@
-# BRK Computer
+# brk_computer
 
-<p align="left">
-  <a href="https://github.com/bitcoinresearchkit/brk">
-    <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
-  </a>
-  <a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
-    <img src="https://img.shields.io/crates/l/brk" alt="License" />
-  </a>
-  <a href="https://crates.io/crates/brk_computer">
-    <img src="https://img.shields.io/crates/v/brk_computer" alt="Version" />
-  </a>
-  <a href="https://docs.rs/brk_computer">
-    <img src="https://img.shields.io/docsrs/brk_computer" alt="Documentation" />
-  </a>
-  <img src="https://img.shields.io/crates/size/brk_computer" alt="Size" />
-  <a href="https://deps.rs/crate/brk_computer">
-    <img src="https://deps.rs/crate/brk_computer/latest/status.svg" alt="Dependency status">
-  </a>
-  <a href="https://discord.gg/HaR3wpH3nr">
-    <img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
-  </a>
-  <a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
-    <img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
-  </a>
-  <a href="https://bsky.app/profile/bitcoinresearchkit.org">
-    <img src="https://img.shields.io/badge/bluesky-blue?link=https%3A%2F%2Fbsky.app%2Fprofile%2Fbitcoinresearchkit.org" alt="Bluesky" />
-  </a>
-  <a href="https://x.com/brkdotorg">
-    <img src="https://img.shields.io/badge/x.com-black" alt="X" />
-  </a>
-</p>
+Advanced Bitcoin analytics engine that transforms indexed blockchain data into comprehensive metrics and financial analytics.
 
-A dataset computer, built on top of `brk_indexer` and `brk_fetcher`. It computes any dataset you can think of and if it doesn't feel free to create an issue.
+[![Crates.io](https://img.shields.io/crates/v/brk_computer.svg)](https://crates.io/crates/brk_computer)
+[![Documentation](https://docs.rs/brk_computer/badge.svg)](https://docs.rs/brk_computer)
+
+## Overview
+
+This crate provides a sophisticated analytics engine that processes indexed Bitcoin blockchain data to compute comprehensive metrics, financial analytics, and statistical aggregations. Built on top of `brk_indexer`, it transforms raw blockchain data into actionable insights through state tracking, cohort analysis, market metrics, and advanced Bitcoin-specific calculations.
+
+**Key Features:**
+
+- Comprehensive Bitcoin analytics pipeline with 6 major computation modules
+- UTXO and address cohort analysis with lifecycle tracking
+- Market metrics integration with price data and financial calculations
+- Cointime economics and realized/unrealized profit/loss analysis
+- Supply dynamics and monetary policy metrics
+- Pool analysis for centralization and mining statistics
+- Memory allocation tracking and performance optimization
+- Parallel computation with multi-threaded processing
+
+**Target Use Cases:**
+
+- Bitcoin market analysis and research platforms
+- On-chain analytics for investment and trading decisions
+- Academic research requiring comprehensive blockchain metrics
+- Financial applications needing Bitcoin exposure and risk metrics
+
+## Installation
+
+```bash
+cargo add brk_computer
+```
+
+## Quick Start
+
+```rust
+use brk_computer::Computer;
+use brk_indexer::Indexer;
+use brk_fetcher::Fetcher;
+use vecdb::Exit;
+use std::path::Path;
+
+// Initialize dependencies
+let outputs_path = Path::new("./analytics_data");
+let indexer = Indexer::forced_import(outputs_path)?;
+let fetcher = Some(Fetcher::import(true, None)?);
+
+// Create computer with price data support
+let mut computer = Computer::forced_import(outputs_path, &indexer, fetcher)?;
+
+// Compute analytics from indexer state
+let exit = Exit::default();
+let starting_indexes = brk_indexer::Indexes::default();
+computer.compute(&indexer, starting_indexes, &exit)?;
+
+println!("Analytics computation completed!");
+```
+
+## API Overview
+
+### Core Structure
+
+The Computer is organized into 7 specialized computation modules:
+
+- **`indexes`**: Fundamental blockchain index computations
+- **`constants`**: Network constants and protocol parameters
+- **`market`**: Price-based financial metrics and market analysis
+- **`pools`**: Mining pool analysis and centralization metrics
+- **`chain`**: Core blockchain metrics (difficulty, hashrate, fees)
+- **`stateful`**: Advanced state tracking (UTXO lifecycles, address behaviors)
+- **`cointime`**: Cointime economics and value-time calculations
+
+### Key Methods
+
+**`Computer::forced_import(outputs_path, indexer, fetcher) -> Result<Self>`**
+Creates computer instance with optional price data integration.
+
+**`compute(&mut self, indexer: &Indexer, starting_indexes: Indexes, exit: &Exit) -> Result<()>`**
+Main computation pipeline processing all analytics modules.
+
+### Analytics Categories
+
+**Market Analytics:**
+
+- Price-based metrics (market cap, realized cap, MVRV)
+- Trading volume analysis and liquidity metrics
+- Return calculations and volatility measurements
+- Dollar-cost averaging and investment strategy metrics
+
+**On-Chain Analytics:**
+
+- Transaction count and size statistics
+- Fee analysis and block space utilization
+- Address activity and entity clustering
+- UTXO age distributions and spending patterns
+
+**Monetary Analytics:**
+
+- Circulating supply and issuance tracking
+- Realized vs. unrealized gains/losses
+- Cointime destruction and accumulation
+- Velocity and economic activity indicators
+
+## Examples
+
+### Basic Analytics Computation
+
+```rust
+use brk_computer::Computer;
+
+// Initialize with indexer and optional price data
+let computer = Computer::forced_import(
+    "./analytics_output",
+    &indexer,
+    Some(price_fetcher)
+)?;
+
+// Compute all analytics modules
+let exit = vecdb::Exit::default();
+computer.compute(&indexer, starting_indexes, &exit)?;
+
+// Access computed metrics
+println!("Market cap vectors computed: {}", computer.market.len());
+println!("Chain metrics computed: {}", computer.chain.len());
+println!("Stateful analysis completed: {}", computer.stateful.len());
+```
+
+### Market Analysis
+
+```rust
+use brk_computer::Computer;
+use brk_structs::{DateIndex, Height};
+
+let computer = Computer::forced_import(/* ... */)?;
+
+// Access market metrics after computation
+if let Some(market) = &computer.market {
+    // Daily market cap analysis
+    let date_index = DateIndex::from_days_since_genesis(5000);
+    if let Some(market_cap) = market.dateindex_to_market_cap.get(date_index)? {
+        println!("Market cap on day {}: ${}", date_index, market_cap.to_dollars());
+    }
+
+    // MVRV (Market Value to Realized Value) ratio
+    if let Some(mvrv) = market.dateindex_to_mvrv.get(date_index)? {
+        println!("MVRV ratio: {:.2}", mvrv);
+    }
+}
+
+// Chain-level metrics
+let height = Height::new(800000);
+if let Some(difficulty) = computer.chain.height_to_difficulty.get(height)? {
+    println!("Network difficulty at height {}: {}", height, difficulty);
+}
+```
+
+### Cohort Analysis
+
+```rust
+use brk_computer::Computer;
+use brk_structs::{DateIndex, CohortId};
+
+let computer = Computer::forced_import(/* ... */)?;
+
+// Address cohort analysis
+let cohort_date = DateIndex::from_days_since_genesis(4000);
+
+// Analyze address behavior patterns
+if let Some(address_cohorts) = &computer.stateful.address_cohorts {
+    for cohort_id in address_cohorts.get_cohort_ids_for_date(cohort_date)? {
+        let cohort_data = address_cohorts.get_cohort(cohort_id)?;
+
+        println!("Cohort {}: {} addresses created",
+                 cohort_id, cohort_data.addresses.len());
+        println!("Average holding period: {} days",
+                 cohort_data.avg_holding_period.as_days());
+    }
+}
+
+// UTXO cohort lifecycle analysis
+if let Some(utxo_cohorts) = &computer.stateful.utxo_cohorts {
+    let active_utxos = utxo_cohorts.get_active_utxos_for_date(cohort_date)?;
+    println!("Active UTXOs from cohort: {}", active_utxos.len());
+}
+```
+
+### Supply and Monetary Analysis
+
+```rust
+use brk_computer::Computer;
+use brk_structs::{Height, DateIndex};
+
+let computer = Computer::forced_import(/* ... */)?;
+
+// Supply dynamics
+let height = Height::new(750000);
+if let Some(supply) = computer.chain.height_to_circulating_supply.get(height)? {
+    println!("Circulating supply: {} BTC", supply.to_btc());
+}
+
+// Realized vs unrealized analysis
+let date = DateIndex::from_days_since_genesis(5000);
+if let Some(realized_cap) = computer.market.dateindex_to_realized_cap.get(date)? {
+    if let Some(market_cap) = computer.market.dateindex_to_market_cap.get(date)? {
+        let unrealized_pnl = market_cap - realized_cap;
+        println!("Unrealized P&L: ${:.2}B", unrealized_pnl.to_dollars() / 1e9);
+    }
+}
+```
+
+## Architecture
+
+### Computation Pipeline
+
+The computer implements a sophisticated multi-stage pipeline:
+
+1. **Index Computation**: Fundamental blockchain metrics and time-based indexes
+2. **Constants Computation**: Network parameters and protocol constants
+3. **Price Integration**: Optional price data fetching and processing
+4. **Parallel Computation**: Chain, market, pools, stateful, and cointime analytics
+5. **Cross-Dependencies**: Advanced metrics requiring multiple data sources
+
+### Memory Management
+
+**Allocation Tracking:**
+
+- `allocative` integration for memory usage analysis
+- Efficient vector storage with compression options
+- Strategic lazy vs. eager evaluation for memory optimization
+
+**Performance Optimization:**
+
+- `rayon` parallel processing for CPU-intensive calculations
+- Vectorized operations for time-series computations
+- Memory-mapped storage for large datasets
+
+### State Management
+
+**Stateful Analytics:**
+
+- UTXO lifecycle tracking with creation/destruction events
+- Address cohort analysis with behavioral clustering
+- Transaction pattern recognition and anomaly detection
+- Economic cycle analysis with market phase detection
+
+**Cointime Economics:**
+
+- Bitcoin days destroyed and accumulated calculations
+- Velocity measurements and economic activity indicators
+- Age-weighted value transfer analysis
+- Long-term holder vs. active trader segmentation
+
+### Modular Design
+
+Each computation module operates independently:
+
+- **Chain Module**: Basic blockchain metrics (fees, difficulty, hashrate)
+- **Market Module**: Price-dependent financial calculations
+- **Pools Module**: Mining centralization and pool analysis
+- **Stateful Module**: Advanced lifecycle and behavior tracking
+- **Cointime Module**: Economic time-value calculations
+
+### Data Dependencies
+
+**Required Dependencies:**
+
+- `brk_indexer`: Raw blockchain data access
+- `brk_structs`: Type definitions and conversions
+
+**Optional Dependencies:**
+
+- `brk_fetcher`: Price data for financial metrics
+- Market analysis requires price integration
+
+### Computation Orchestration
+
+**Sequential Stages:**
+
+1. Indexes → Constants (foundational metrics)
+2. Fetched → Price (price data processing)
+3. Parallel: Chain, Market, Pools, Stateful, Cointime
+
+**Exit Handling:**
+
+- Graceful shutdown with consistent state preservation
+- Checkpoint-based recovery for long-running computations
+- Multi-threaded coordination with exit signaling
+
+## Code Analysis Summary
+
+**Main Structure**: `Computer` struct coordinating 7 specialized analytics modules (indexes, constants, market, pools, chain, stateful, cointime) \
+**Computation Pipeline**: Multi-stage analytics processing with parallel execution and dependency management \
+**State Tracking**: Advanced UTXO and address lifecycle analysis with cohort-based behavioral clustering \
+**Financial Analytics**: Comprehensive market metrics including realized/unrealized analysis and cointime economics \
+**Memory Optimization**: `allocative` tracking with lazy/eager evaluation strategies and compressed vector storage \
+**Parallel Processing**: `rayon` integration for CPU-intensive calculations with coordinated exit handling \
+**Architecture**: Modular analytics engine transforming indexed blockchain data into actionable financial and economic insights
+
+---
+
+_This README was generated by Claude Code_

crates/brk_computer/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    let profile = std::env::var("PROFILE").unwrap_or_default();
+
+    if profile == "release" {
+        println!("cargo:rustc-flag=-C");
+        println!("cargo:rustc-flag=target-cpu=native");
+    }
+}

crates/brk_computer/examples/computer.rs

@@ -0,0 +1,55 @@
+use std::{
+    path::Path,
+    thread::{self, sleep},
+    time::{Duration, Instant},
+};
+
+use brk_computer::Computer;
+use brk_error::Result;
+use brk_fetcher::Fetcher;
+use brk_indexer::Indexer;
+use brk_parser::Parser;
+use vecdb::Exit;
+
+pub fn main() -> Result<()> {
+    brk_logger::init(Some(Path::new(".log")))?;
+
+    let bitcoin_dir = Path::new(&std::env::var("HOME").unwrap())
+        .join("Library")
+        .join("Application Support")
+        .join("Bitcoin");
+    // let bitcoin_dir = Path::new("/Volumes/WD_BLACK/bitcoin");
+
+    let rpc = Box::leak(Box::new(bitcoincore_rpc::Client::new(
+        "http://localhost:8332",
+        bitcoincore_rpc::Auth::CookieFile(bitcoin_dir.join(".cookie")),
+    )?));
+    let exit = Exit::new();
+    exit.set_ctrlc_handler();
+
+    // Can't increase main thread's stack size, thus we need to use another thread
+    thread::Builder::new()
+        .stack_size(256 * 1024 * 1024)
+        .spawn(move || -> Result<()> {
+            let outputs_dir = Path::new(&std::env::var("HOME").unwrap()).join(".brk");
+            // let outputs_dir = Path::new("../../_outputs");
+
+            let parser = Parser::new(bitcoin_dir.join("blocks"), rpc);
+
+            let mut indexer = Indexer::forced_import(&outputs_dir)?;
+
+            let fetcher = Fetcher::import(true, None)?;
+
+            let mut computer = Computer::forced_import(&outputs_dir, &indexer, Some(fetcher))?;
+
+            loop {
+                let i = Instant::now();
+                let starting_indexes = indexer.index(&parser, rpc, &exit, true)?;
+                computer.compute(&indexer, starting_indexes, &parser, &exit)?;
+                dbg!(i.elapsed());
+                sleep(Duration::from_secs(10));
+            }
+        })?
+        .join()
+        .unwrap()
+}

crates/brk_computer/examples/main.rs

@@ -1,57 +0,0 @@
-use std::{path::Path, thread};
-
-use brk_computer::Computer;
-use brk_core::{default_bitcoin_path, default_brk_path};
-use brk_exit::Exit;
-use brk_fetcher::Fetcher;
-use brk_indexer::Indexer;
-use brk_parser::Parser;
-use brk_vec::{Computation, Format};
-
-pub fn main() -> color_eyre::Result<()> {
-    color_eyre::install()?;
-
-    brk_logger::init(Some(Path::new(".log")));
-
-    // let bitcoin_dir = default_bitcoin_path();
-    let bitcoin_dir = Path::new("/Volumes/WD_BLACK/bitcoin");
-
-    let rpc = Box::leak(Box::new(bitcoincore_rpc::Client::new(
-        "http://localhost:8332",
-        bitcoincore_rpc::Auth::CookieFile(bitcoin_dir.join(".cookie")),
-    )?));
-    let exit = Exit::new();
-
-    // Can't increase main thread's stack programatically, thus we need to use another thread
-    thread::Builder::new()
-        .stack_size(256 * 1024 * 1024)
-        .spawn(move || -> color_eyre::Result<()> {
-            let parser = Parser::new(bitcoin_dir.join("blocks"), rpc);
-
-            let _outputs_dir = Path::new("/Volumes/WD_BLACK/brk").join("outputs");
-            let outputs_dir = _outputs_dir.as_path();
-            // let outputs_dir = Path::new("../../_outputs");
-
-            let format = Format::Raw;
-
-            let mut indexer = Indexer::forced_import(outputs_dir)?;
-
-            let fetcher = Fetcher::import(None)?;
-
-            let mut computer = Computer::forced_import(
-                outputs_dir,
-                &indexer,
-                Computation::Lazy,
-                Some(fetcher),
-                format,
-            )?;
-
-            let starting_indexes = indexer.index(&parser, rpc, &exit, true)?;
-
-            computer.compute(&mut indexer, starting_indexes, &exit)?;
-
-            Ok(())
-        })?
-        .join()
-        .unwrap()
-}

crates/brk_computer/examples/pools.rs

@@ -0,0 +1,125 @@
+use std::{collections::BTreeMap, path::Path, thread};
+
+use brk_computer::Computer;
+use brk_error::Result;
+use brk_fetcher::Fetcher;
+use brk_indexer::Indexer;
+use brk_structs::{AddressBytes, OutputIndex, OutputType, pools};
+use vecdb::{AnyIterableVec, Exit, VecIterator};
+
+fn main() -> Result<()> {
+    brk_logger::init(Some(Path::new(".log")))?;
+
+    let exit = Exit::new();
+    exit.set_ctrlc_handler();
+
+    thread::Builder::new()
+        .stack_size(256 * 1024 * 1024)
+        .spawn(move || -> Result<()> {
+            let outputs_dir = Path::new(&std::env::var("HOME").unwrap()).join(".brk");
+
+            let indexer = Indexer::forced_import(&outputs_dir)?;
+
+            let fetcher = Fetcher::import(true, None)?;
+
+            let computer = Computer::forced_import(&outputs_dir, &indexer, Some(fetcher))?;
+
+            let pools = pools();
+
+            let mut res: BTreeMap<&'static str, usize> = BTreeMap::default();
+
+            let vecs = indexer.vecs;
+            let stores = indexer.stores;
+
+            let mut height_to_first_txindex_iter = vecs.height_to_first_txindex.iter();
+            let mut txindex_to_first_outputindex_iter = vecs.txindex_to_first_outputindex.iter();
+            let mut txindex_to_output_count_iter = computer.indexes.txindex_to_output_count.iter();
+            let mut outputindex_to_outputtype_iter = vecs.outputindex_to_outputtype.iter();
+            let mut outputindex_to_typeindex_iter = vecs.outputindex_to_typeindex.iter();
+            let mut p2pk65addressindex_to_p2pk65bytes_iter =
+                vecs.p2pk65addressindex_to_p2pk65bytes.iter();
+            let mut p2pk33addressindex_to_p2pk33bytes_iter =
+                vecs.p2pk33addressindex_to_p2pk33bytes.iter();
+            let mut p2pkhaddressindex_to_p2pkhbytes_iter =
+                vecs.p2pkhaddressindex_to_p2pkhbytes.iter();
+            let mut p2shaddressindex_to_p2shbytes_iter = vecs.p2shaddressindex_to_p2shbytes.iter();
+            let mut p2wpkhaddressindex_to_p2wpkhbytes_iter =
+                vecs.p2wpkhaddressindex_to_p2wpkhbytes.iter();
+            let mut p2wshaddressindex_to_p2wshbytes_iter =
+                vecs.p2wshaddressindex_to_p2wshbytes.iter();
+            let mut p2traddressindex_to_p2trbytes_iter = vecs.p2traddressindex_to_p2trbytes.iter();
+            let mut p2aaddressindex_to_p2abytes_iter = vecs.p2aaddressindex_to_p2abytes.iter();
+
+            let unknown = pools.get_unknown();
+
+            stores
+                .height_to_coinbase_tag
+                .iter()
+                .for_each(|(height, coinbase_tag)| {
+                    let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+                    let outputindex = txindex_to_first_outputindex_iter.unwrap_get_inner(txindex);
+                    let outputcount = txindex_to_output_count_iter.unwrap_get_inner(txindex);
+
+                    let pool = (*outputindex..(*outputindex + *outputcount))
+                        .map(OutputIndex::from)
+                        .find_map(|outputindex| {
+                            let outputtype =
+                                outputindex_to_outputtype_iter.unwrap_get_inner(outputindex);
+                            let typeindex =
+                                outputindex_to_typeindex_iter.unwrap_get_inner(outputindex);
+
+                            let address = match outputtype {
+                                OutputType::P2PK65 => Some(AddressBytes::from(
+                                    p2pk65addressindex_to_p2pk65bytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2PK33 => Some(AddressBytes::from(
+                                    p2pk33addressindex_to_p2pk33bytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2PKH => Some(AddressBytes::from(
+                                    p2pkhaddressindex_to_p2pkhbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2SH => Some(AddressBytes::from(
+                                    p2shaddressindex_to_p2shbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2WPKH => Some(AddressBytes::from(
+                                    p2wpkhaddressindex_to_p2wpkhbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2WSH => Some(AddressBytes::from(
+                                    p2wshaddressindex_to_p2wshbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2TR => Some(AddressBytes::from(
+                                    p2traddressindex_to_p2trbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2A => Some(AddressBytes::from(
+                                    p2aaddressindex_to_p2abytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                _ => None,
+                            };
+
+                            address
+                                .and_then(|address| pools.find_from_address(&address.to_string()))
+                        })
+                        .or_else(|| pools.find_from_coinbase_tag(&coinbase_tag))
+                        .unwrap_or(unknown);
+
+                    *res.entry(pool.name).or_default() += 1;
+                });
+
+            let mut v = res.into_iter().map(|(k, v)| (v, k)).collect::<Vec<_>>();
+            v.sort_unstable();
+            println!("{:#?}", v);
+            println!("{:#?}", v.len());
+
+            Ok(())
+        })?
+        .join()
+        .unwrap()
+}

crates/brk_computer/examples/price_to_amount.rs

@@ -0,0 +1,15 @@
+use std::path::Path;
+
+use brk_computer::PriceToAmount;
+use brk_error::Result;
+use brk_structs::Height;
+
+pub fn main() -> Result<()> {
+    let path = Path::new(&std::env::var("HOME").unwrap())
+        .join(".brk")
+        .join("computed/stateful/states");
+    let mut price_to_amount = PriceToAmount::create(&path, "addrs_above_1btc_under_10btc");
+    dbg!(price_to_amount.import_at_or_before(Height::new(890000))?);
+    dbg!(price_to_amount);
+    Ok(())
+}

crates/brk_computer/src/blks.rs

@@ -0,0 +1,141 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_indexer::Indexer;
+use brk_parser::Parser;
+use brk_structs::{BlkPosition, Height, TxIndex, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, CompressedVec, Database, Exit,
+    GenericStoredVec, PAGE_SIZE, VecIterator,
+};
+
+use super::{Indexes, indexes};
+
+#[derive(Clone)]
+pub struct Vecs {
+    db: Database,
+
+    pub height_to_position: CompressedVec<Height, BlkPosition>,
+    pub txindex_to_position: CompressedVec<TxIndex, BlkPosition>,
+}
+
+impl Vecs {
+    pub fn forced_import(parent_path: &Path, parent_version: Version) -> Result<Self> {
+        let db = Database::open(&parent_path.join("blks"))?;
+        db.set_min_len(PAGE_SIZE * 1_000_000)?;
+
+        let version = parent_version + Version::ZERO;
+
+        let this = Self {
+            height_to_position: CompressedVec::forced_import(
+                &db,
+                "position",
+                version + Version::TWO,
+            )?,
+            txindex_to_position: CompressedVec::forced_import(
+                &db,
+                "position",
+                version + Version::TWO,
+            )?,
+
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        parser: &Parser,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexer, indexes, starting_indexes, parser, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        parser: &Parser,
+        exit: &Exit,
+    ) -> Result<()> {
+        let min_txindex =
+            TxIndex::from(self.txindex_to_position.len()).min(starting_indexes.txindex);
+
+        let Some(min_height) = indexes
+            .txindex_to_height
+            .iter()
+            .get_inner(min_txindex)
+            .map(|h| h.min(starting_indexes.height))
+        else {
+            return Ok(());
+        };
+
+        let mut height_to_first_txindex_iter = indexer.vecs.height_to_first_txindex.iter();
+
+        parser
+            .parse(
+                Some(min_height),
+                Some((indexer.vecs.height_to_first_txindex.len() - 1).into()),
+            )
+            .iter()
+            .try_for_each(|block| -> Result<()> {
+                let height = block.height();
+
+                self.height_to_position.forced_push_at(
+                    height,
+                    block.metadata().position(),
+                    exit,
+                )?;
+
+                let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+
+                block.tx_metadata().iter().enumerate().try_for_each(
+                    |(index, metadata)| -> Result<()> {
+                        let txindex = txindex + index;
+                        self.txindex_to_position.forced_push_at(
+                            txindex,
+                            metadata.position(),
+                            exit,
+                        )?;
+                        Ok(())
+                    },
+                )?;
+
+                if *height % 1_000 == 0 {
+                    let _lock = exit.lock();
+                    self.height_to_position.flush()?;
+                    self.txindex_to_position.flush()?;
+                }
+                Ok(())
+            })?;
+
+        let _lock = exit.lock();
+        self.height_to_position.flush()?;
+        self.txindex_to_position.flush()?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        Box::new(
+            [
+                &self.height_to_position as &dyn AnyCollectableVec,
+                &self.txindex_to_position,
+            ]
+            .into_iter(),
+        )
+    }
+}

crates/brk_computer/src/constants.rs

@@ -0,0 +1,274 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_structs::{StoredF32, StoredI16, StoredU16, Version};
+use vecdb::{AnyCollectableVec, AnyVec, Database, Exit};
+
+use crate::grouped::Source;
+
+use super::{
+    Indexes,
+    grouped::{ComputedVecsFromHeight, VecBuilderOptions},
+    indexes,
+};
+
+#[derive(Clone)]
+pub struct Vecs {
+    db: Database,
+
+    pub constant_0: ComputedVecsFromHeight<StoredU16>,
+    pub constant_1: ComputedVecsFromHeight<StoredU16>,
+    pub constant_2: ComputedVecsFromHeight<StoredU16>,
+    pub constant_3: ComputedVecsFromHeight<StoredU16>,
+    pub constant_4: ComputedVecsFromHeight<StoredU16>,
+    pub constant_38_2: ComputedVecsFromHeight<StoredF32>,
+    pub constant_50: ComputedVecsFromHeight<StoredU16>,
+    pub constant_61_8: ComputedVecsFromHeight<StoredF32>,
+    pub constant_100: ComputedVecsFromHeight<StoredU16>,
+    pub constant_600: ComputedVecsFromHeight<StoredU16>,
+    pub constant_minus_1: ComputedVecsFromHeight<StoredI16>,
+    pub constant_minus_2: ComputedVecsFromHeight<StoredI16>,
+    pub constant_minus_3: ComputedVecsFromHeight<StoredI16>,
+    pub constant_minus_4: ComputedVecsFromHeight<StoredI16>,
+}
+
+impl Vecs {
+    pub fn forced_import(
+        parent_path: &Path,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+    ) -> Result<Self> {
+        let db = Database::open(&parent_path.join("constants"))?;
+
+        let version = parent_version + Version::ZERO;
+
+        let this = Self {
+            constant_0: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_0",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_1: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_1",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_2: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_2",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_3: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_3",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_4: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_4",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_38_2: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_38_2",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_50: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_50",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_61_8: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_61_8",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_100: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_100",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_600: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_600",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_minus_1: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_minus_1",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_minus_2: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_minus_2",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_minus_3: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_minus_3",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_minus_4: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_minus_4",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexes, starting_indexes, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        [
+            (&mut self.constant_0, 0),
+            (&mut self.constant_1, 1),
+            (&mut self.constant_2, 2),
+            (&mut self.constant_3, 3),
+            (&mut self.constant_4, 4),
+            (&mut self.constant_50, 50),
+            (&mut self.constant_100, 100),
+            (&mut self.constant_600, 600),
+        ]
+        .into_iter()
+        .try_for_each(|(vec, value)| {
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_to(
+                    starting_indexes.height,
+                    indexes.height_to_date.len(),
+                    indexes.height_to_date.version(),
+                    |i| (i, StoredU16::new(value)),
+                    exit,
+                )?;
+                Ok(())
+            })
+        })?;
+
+        [
+            (&mut self.constant_minus_1, -1),
+            (&mut self.constant_minus_2, -2),
+            (&mut self.constant_minus_3, 3),
+            (&mut self.constant_minus_4, 4),
+        ]
+        .into_iter()
+        .try_for_each(|(vec, value)| {
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_to(
+                    starting_indexes.height,
+                    indexes.height_to_date.len(),
+                    indexes.height_to_date.version(),
+                    |i| (i, StoredI16::new(value)),
+                    exit,
+                )?;
+                Ok(())
+            })
+        })?;
+
+        [
+            (&mut self.constant_38_2, 38.2),
+            (&mut self.constant_61_8, 61.8),
+        ]
+        .into_iter()
+        .try_for_each(|(vec, value)| {
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_to(
+                    starting_indexes.height,
+                    indexes.height_to_date.len(),
+                    indexes.height_to_date.version(),
+                    |i| (i, StoredF32::from(value)),
+                    exit,
+                )?;
+                Ok(())
+            })
+        })?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
+
+        iter = Box::new(iter.chain(self.constant_0.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_1.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_2.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_3.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_4.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_38_2.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_50.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_61_8.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_100.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_600.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_1.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_2.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_3.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_4.iter_any_collectable()));
+
+        iter
+    }
+}

crates/brk_computer/src/fetched.rs

@@ -0,0 +1,149 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_fetcher::Fetcher;
+use brk_indexer::Indexer;
+use brk_structs::{DateIndex, Height, OHLCCents, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, Exit, GenericStoredVec,
+    RawVec, StoredIndex, VecIterator,
+};
+
+use super::{Indexes, indexes};
+
+#[derive(Clone)]
+pub struct Vecs {
+    db: Database,
+    fetcher: Fetcher,
+
+    pub dateindex_to_price_ohlc_in_cents: RawVec<DateIndex, OHLCCents>,
+    pub height_to_price_ohlc_in_cents: RawVec<Height, OHLCCents>,
+}
+
+impl Vecs {
+    pub fn forced_import(parent: &Path, fetcher: Fetcher, version: Version) -> Result<Self> {
+        let db = Database::open(&parent.join("fetched"))?;
+
+        let this = Self {
+            fetcher,
+
+            dateindex_to_price_ohlc_in_cents: RawVec::forced_import(
+                &db,
+                "price_ohlc_in_cents",
+                version + Version::ZERO,
+            )?,
+            height_to_price_ohlc_in_cents: RawVec::forced_import(
+                &db,
+                "price_ohlc_in_cents",
+                version + Version::ZERO,
+            )?,
+
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexer, indexes, starting_indexes, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        let height_to_timestamp = &indexer.vecs.height_to_timestamp;
+        let index = starting_indexes
+            .height
+            .min(Height::from(self.height_to_price_ohlc_in_cents.len()));
+        height_to_timestamp
+            .iter_at(index)
+            .try_for_each(|(i, v)| -> Result<()> {
+                let v = v.into_owned();
+                self.height_to_price_ohlc_in_cents.forced_push_at(
+                    i,
+                    self.fetcher
+                        .get_height(
+                            i,
+                            v,
+                            i.decremented().map(|prev_i| {
+                                height_to_timestamp.into_iter().unwrap_get_inner(prev_i)
+                            }),
+                        )
+                        .unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+        self.height_to_price_ohlc_in_cents.safe_flush(exit)?;
+
+        let index = starting_indexes
+            .dateindex
+            .min(DateIndex::from(self.dateindex_to_price_ohlc_in_cents.len()));
+        let mut prev = None;
+        indexes
+            .dateindex_to_date
+            .iter_at(index)
+            .try_for_each(|(i, v)| -> Result<()> {
+                let d = v.into_owned();
+                if prev.is_none() {
+                    let i = i.unwrap_to_usize();
+                    prev.replace(if i > 0 {
+                        self.dateindex_to_price_ohlc_in_cents
+                            .into_iter()
+                            .unwrap_get_inner_(i - 1)
+                    } else {
+                        OHLCCents::default()
+                    });
+                }
+
+                let ohlc = if i.unwrap_to_usize() + 100
+                    >= self.dateindex_to_price_ohlc_in_cents.len()
+                    && let Ok(mut ohlc) = self.fetcher.get_date(d)
+                {
+                    let prev_open = *prev.as_ref().unwrap().close;
+                    *ohlc.open = prev_open;
+                    *ohlc.high = (*ohlc.high).max(prev_open);
+                    *ohlc.low = (*ohlc.low).min(prev_open);
+                    ohlc
+                } else {
+                    prev.clone().unwrap()
+                };
+
+                prev.replace(ohlc.clone());
+
+                self.dateindex_to_price_ohlc_in_cents
+                    .forced_push_at(i, ohlc, exit)?;
+
+                Ok(())
+            })?;
+        self.dateindex_to_price_ohlc_in_cents.safe_flush(exit)?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [
+            &self.dateindex_to_price_ohlc_in_cents as &dyn AnyCollectableVec,
+            &self.height_to_price_ohlc_in_cents,
+        ]
+        .into_iter()
+    }
+}

crates/brk_computer/src/grouped/builder_eager.rs

@@ -1,15 +1,16 @@
-use std::path::Path;
-
-use brk_core::{CheckedSub, Result, StoredUsize, Version};
-use brk_exit::Exit;
-use brk_vec::{AnyCollectableVec, AnyIterableVec, EagerVec, Format, StoredIndex, StoredType};
-use color_eyre::eyre::ContextCompat;
+use allocative::Allocative;
+use brk_error::{Error, Result};
+use brk_structs::{CheckedSub, StoredU64, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, EagerVec, Exit, Format,
+    GenericStoredVec, StoredIndex, StoredRaw,
+};
 
 use crate::utils::get_percentile;
 
 use super::ComputedType;
 
-#[derive(Clone, Debug)]
+#[derive(Clone, Debug, Allocative)]
 pub struct EagerVecBuilder<I, T>
 where
     I: StoredIndex,
@@ -19,11 +20,11 @@ where
     pub average: Option<Box<EagerVec<I, T>>>,
     pub sum: Option<Box<EagerVec<I, T>>>,
     pub max: Option<Box<EagerVec<I, T>>>,
-    pub _90p: Option<Box<EagerVec<I, T>>>,
-    pub _75p: Option<Box<EagerVec<I, T>>>,
+    pub pct90: Option<Box<EagerVec<I, T>>>,
+    pub pct75: Option<Box<EagerVec<I, T>>>,
     pub median: Option<Box<EagerVec<I, T>>>,
-    pub _25p: Option<Box<EagerVec<I, T>>>,
-    pub _10p: Option<Box<EagerVec<I, T>>>,
+    pub pct25: Option<Box<EagerVec<I, T>>>,
+    pub pct10: Option<Box<EagerVec<I, T>>>,
     pub min: Option<Box<EagerVec<I, T>>>,
     pub last: Option<Box<EagerVec<I, T>>>,
     pub cumulative: Option<Box<EagerVec<I, T>>>,
@@ -36,13 +37,22 @@ where
     I: StoredIndex,
     T: ComputedType,
 {
+    pub fn forced_import_compressed(
+        db: &Database,
+        name: &str,
+        version: Version,
+        options: VecBuilderOptions,
+    ) -> Result<Self> {
+        Self::forced_import(db, name, version, Format::Compressed, options)
+    }
+
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         name: &str,
         version: Version,
         format: Format,
         options: VecBuilderOptions,
-    ) -> color_eyre::Result<Self> {
+    ) -> Result<Self> {
         let only_one_active = options.is_only_one_active();
 
         let suffix = |s: &str| format!("{name}_{s}");
@@ -59,7 +69,7 @@ where
             first: options.first.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
+                        db,
                         &maybe_suffix("first"),
                         version + VERSION + Version::ZERO,
                         format,
@@ -69,13 +79,13 @@ where
             }),
             last: options.last.then(|| {
                 Box::new(
-                    EagerVec::forced_import(path, name, version + Version::ZERO, format).unwrap(),
+                    EagerVec::forced_import(db, name, version + Version::ZERO, format).unwrap(),
                 )
             }),
             min: options.min.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
+                        db,
                         &maybe_suffix("min"),
                         version + VERSION + Version::ZERO,
                         format,
@@ -86,7 +96,7 @@ where
             max: options.max.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
+                        db,
                         &maybe_suffix("max"),
                         version + VERSION + Version::ZERO,
                         format,
@@ -97,7 +107,7 @@ where
             median: options.median.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
+                        db,
                         &maybe_suffix("median"),
                         version + VERSION + Version::ZERO,
                         format,
@@ -108,8 +118,8 @@ where
             average: options.average.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_suffix("average"),
+                        db,
+                        &maybe_suffix("avg"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
@@ -119,7 +129,7 @@ where
             sum: options.sum.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
+                        db,
                         &(if !options.last && !options.average && !options.min && !options.max {
                             name.to_string()
                         } else {
@@ -134,7 +144,7 @@ where
             cumulative: options.cumulative.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
+                        db,
                         &suffix("cumulative"),
                         version + VERSION + Version::ZERO,
                         format,
@@ -142,44 +152,44 @@ where
                     .unwrap(),
                 )
             }),
-            _90p: options._90p.then(|| {
+            pct90: options.pct90.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_suffix("90p"),
+                        db,
+                        &maybe_suffix("pct90"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _75p: options._75p.then(|| {
+            pct75: options.pct75.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_suffix("75p"),
+                        db,
+                        &maybe_suffix("pct75"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _25p: options._25p.then(|| {
+            pct25: options.pct25.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_suffix("25p"),
+                        db,
+                        &maybe_suffix("pct25"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _10p: options._10p.then(|| {
+            pct10: options.pct10.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_suffix("10p"),
+                        db,
+                        &maybe_suffix("pct10"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
@@ -201,7 +211,7 @@ where
             return Ok(());
         };
 
-        self.validate_computed_version_or_reset_file(source.version())?;
+        self.validate_computed_version_or_reset(source.version())?;
 
         let index = self.starting_index(max_from);
 
@@ -211,8 +221,9 @@ where
             cumulative_vec.iter().unwrap_get_inner(index)
         });
         source.iter_at(index).try_for_each(|(i, v)| -> Result<()> {
-            cumulative = cumulative.clone() + v.into_owned();
-            cumulative_vec.forced_push_at(i, cumulative.clone(), exit)
+            cumulative += v.into_owned();
+            cumulative_vec.forced_push_at(i, cumulative, exit)?;
+            Ok(())
         })?;
 
         self.safe_flush(exit)?;
@@ -225,13 +236,13 @@ where
         max_from: I,
         source: &impl AnyIterableVec<I2, T>,
         first_indexes: &impl AnyIterableVec<I, I2>,
-        count_indexes: &impl AnyIterableVec<I, StoredUsize>,
+        count_indexes: &impl AnyIterableVec<I, StoredU64>,
         exit: &Exit,
     ) -> Result<()>
     where
-        I2: StoredIndex + StoredType + CheckedSub<I2>,
+        I2: StoredIndex + StoredRaw + CheckedSub<I2>,
     {
-        self.validate_computed_version_or_reset_file(
+        self.validate_computed_version_or_reset(
             source.version() + first_indexes.version() + count_indexes.version(),
         )?;
 
@@ -250,10 +261,10 @@ where
 
         first_indexes
             .iter_at(index)
-            .try_for_each(|(i, first_index)| -> Result<()> {
+            .try_for_each(|(index, first_index)| -> Result<()> {
                 let first_index = first_index.into_owned();
 
-                let count_index = count_indexes_iter.unwrap_get_inner(i);
+                let count_index = count_indexes_iter.unwrap_get_inner(index);
 
                 if let Some(first) = self.first.as_mut() {
                     let f = source_iter
@@ -263,7 +274,7 @@ where
                 }
 
                 if let Some(last) = self.last.as_mut() {
-                    let count_index = *count_index;
+                    let count_index = *count_index as usize;
                     if count_index == 0 {
                         panic!("should compute last if count can be 0")
                     }
@@ -282,18 +293,18 @@ where
                 let needs_average_sum_or_cumulative =
                     needs_sum_or_cumulative || self.average.is_some();
                 let needs_sorted = self.max.is_some()
-                    || self._90p.is_some()
-                    || self._75p.is_some()
+                    || self.pct90.is_some()
+                    || self.pct75.is_some()
                     || self.median.is_some()
-                    || self._25p.is_some()
-                    || self._10p.is_some()
+                    || self.pct25.is_some()
+                    || self.pct10.is_some()
                     || self.min.is_some();
                 let needs_values = needs_sorted || needs_average_sum_or_cumulative;
 
                 if needs_values {
                     source_iter.set(first_index);
                     let mut values = (&mut source_iter)
-                        .take(*count_index)
+                        .take(*count_index as usize)
                         .map(|(_, v)| v.into_owned())
                         .collect::<Vec<_>>();
 
@@ -302,14 +313,14 @@ where
 
                         if let Some(max) = self.max.as_mut() {
                             max.forced_push_at(
-                                i,
-                                values
+                                index,
+                                *values
                                     .last()
-                                    .context("expect some")
+                                    .ok_or(Error::Str("expect some"))
                                     .inspect_err(|_| {
                                         dbg!(
                                             &values,
-                                            max.path(),
+                                            max.name(),
                                             first_indexes.name(),
                                             first_index,
                                             count_indexes.name(),
@@ -318,34 +329,33 @@ where
                                             source.name()
                                         );
                                     })
-                                    .unwrap()
-                                    .clone(),
+                                    .unwrap(),
                                 exit,
                             )?;
                         }
 
-                        if let Some(_90p) = self._90p.as_mut() {
-                            _90p.forced_push_at(i, get_percentile(&values, 0.90), exit)?;
+                        if let Some(pct90) = self.pct90.as_mut() {
+                            pct90.forced_push_at(index, get_percentile(&values, 0.90), exit)?;
                         }
 
-                        if let Some(_75p) = self._75p.as_mut() {
-                            _75p.forced_push_at(i, get_percentile(&values, 0.75), exit)?;
+                        if let Some(pct75) = self.pct75.as_mut() {
+                            pct75.forced_push_at(index, get_percentile(&values, 0.75), exit)?;
                         }
 
                         if let Some(median) = self.median.as_mut() {
-                            median.forced_push_at(i, get_percentile(&values, 0.50), exit)?;
+                            median.forced_push_at(index, get_percentile(&values, 0.50), exit)?;
                         }
 
-                        if let Some(_25p) = self._25p.as_mut() {
-                            _25p.forced_push_at(i, get_percentile(&values, 0.25), exit)?;
+                        if let Some(pct25) = self.pct25.as_mut() {
+                            pct25.forced_push_at(index, get_percentile(&values, 0.25), exit)?;
                         }
 
-                        if let Some(_10p) = self._10p.as_mut() {
-                            _10p.forced_push_at(i, get_percentile(&values, 0.10), exit)?;
+                        if let Some(pct10) = self.pct10.as_mut() {
+                            pct10.forced_push_at(index, get_percentile(&values, 0.10), exit)?;
                         }
 
                         if let Some(min) = self.min.as_mut() {
-                            min.forced_push_at(i, values.first().unwrap().clone(), exit)?;
+                            min.forced_push_at(index, *values.first().unwrap(), exit)?;
                         }
                     }
 
@@ -354,19 +364,19 @@ where
                         let sum = values.into_iter().fold(T::from(0), |a, b| a + b);
 
                         if let Some(average) = self.average.as_mut() {
-                            let avg = sum.clone() / len;
-                            average.forced_push_at(i, avg, exit)?;
+                            let avg = sum / len;
+                            average.forced_push_at(index, avg, exit)?;
                         }
 
                         if needs_sum_or_cumulative {
                             if let Some(sum_vec) = self.sum.as_mut() {
-                                sum_vec.forced_push_at(i, sum.clone(), exit)?;
+                                sum_vec.forced_push_at(index, sum, exit)?;
                             }
 
                             if let Some(cumulative_vec) = self.cumulative.as_mut() {
-                                let t = cumulative.as_ref().unwrap().clone() + sum;
-                                cumulative.replace(t.clone());
-                                cumulative_vec.forced_push_at(i, t, exit)?;
+                                let t = cumulative.unwrap() + sum;
+                                cumulative.replace(t);
+                                cumulative_vec.forced_push_at(index, t, exit)?;
                             }
                         }
                     }
@@ -386,22 +396,22 @@ where
         max_from: I,
         source: &EagerVecBuilder<I2, T>,
         first_indexes: &impl AnyIterableVec<I, I2>,
-        count_indexes: &impl AnyIterableVec<I, StoredUsize>,
+        count_indexes: &impl AnyIterableVec<I, StoredU64>,
         exit: &Exit,
     ) -> Result<()>
     where
-        I2: StoredIndex + StoredType + CheckedSub<I2>,
+        I2: StoredIndex + StoredRaw + CheckedSub<I2>,
     {
-        if self._90p.is_some()
-            || self._75p.is_some()
+        if self.pct90.is_some()
+            || self.pct75.is_some()
             || self.median.is_some()
-            || self._25p.is_some()
-            || self._10p.is_some()
+            || self.pct25.is_some()
+            || self.pct10.is_some()
         {
             panic!("unsupported");
         }
 
-        self.validate_computed_version_or_reset_file(
+        self.validate_computed_version_or_reset(
             VERSION + first_indexes.version() + count_indexes.version(),
         )?;
 
@@ -424,10 +434,10 @@ where
 
         first_indexes
             .iter_at(index)
-            .try_for_each(|(i, first_index, ..)| -> Result<()> {
+            .try_for_each(|(index, first_index, ..)| -> Result<()> {
                 let first_index = first_index.into_owned();
 
-                let count_index = count_indexes_iter.unwrap_get_inner(i);
+                let count_index = count_indexes_iter.unwrap_get_inner(index);
 
                 if let Some(first) = self.first.as_mut() {
                     let v = source_first_iter
@@ -438,7 +448,7 @@ where
                 }
 
                 if let Some(last) = self.last.as_mut() {
-                    let count_index = *count_index;
+                    let count_index = *count_index as usize;
                     if count_index == 0 {
                         panic!("should compute last if count can be 0")
                     }
@@ -462,22 +472,22 @@ where
                             let source_max_iter = source_max_iter.as_mut().unwrap();
                             source_max_iter.set(first_index);
                             let mut values = source_max_iter
-                                .take(*count_index)
+                                .take(*count_index as usize)
                                 .map(|(_, v)| v.into_owned())
                                 .collect::<Vec<_>>();
                             values.sort_unstable();
-                            max.forced_push_at(i, values.last().unwrap().clone(), exit)?;
+                            max.forced_push_at(index, *values.last().unwrap(), exit)?;
                         }
 
                         if let Some(min) = self.min.as_mut() {
                             let source_min_iter = source_min_iter.as_mut().unwrap();
                             source_min_iter.set(first_index);
                             let mut values = source_min_iter
-                                .take(*count_index)
+                                .take(*count_index as usize)
                                 .map(|(_, v)| v.into_owned())
                                 .collect::<Vec<_>>();
                             values.sort_unstable();
-                            min.forced_push_at(i, values.first().unwrap().clone(), exit)?;
+                            min.forced_push_at(index, *values.first().unwrap(), exit)?;
                         }
                     }
 
@@ -486,7 +496,7 @@ where
                             let source_average_iter = source_average_iter.as_mut().unwrap();
                             source_average_iter.set(first_index);
                             let values = source_average_iter
-                                .take(*count_index)
+                                .take(*count_index as usize)
                                 .map(|(_, v)| v.into_owned())
                                 .collect::<Vec<_>>();
 
@@ -495,27 +505,27 @@ where
                             // TODO: Multiply by count then divide by cumulative
                             // Right now it's not 100% accurate as there could be more or less elements in the lower timeframe (28 days vs 31 days in a month for example)
                             let avg = cumulative / len;
-                            average.forced_push_at(i, avg, exit)?;
+                            average.forced_push_at(index, avg, exit)?;
                         }
 
                         if needs_sum_or_cumulative {
                             let source_sum_iter = source_sum_iter.as_mut().unwrap();
                             source_sum_iter.set(first_index);
                             let values = source_sum_iter
-                                .take(*count_index)
+                                .take(*count_index as usize)
                                 .map(|(_, v)| v.into_owned())
                                 .collect::<Vec<_>>();
 
                             let sum = values.into_iter().fold(T::from(0), |a, b| a + b);
 
                             if let Some(sum_vec) = self.sum.as_mut() {
-                                sum_vec.forced_push_at(i, sum.clone(), exit)?;
+                                sum_vec.forced_push_at(index, sum, exit)?;
                             }
 
                             if let Some(cumulative_vec) = self.cumulative.as_mut() {
-                                let t = cumulative.as_ref().unwrap().clone() + sum;
-                                cumulative.replace(t.clone());
-                                cumulative_vec.forced_push_at(i, t, exit)?;
+                                let t = cumulative.unwrap() + sum;
+                                cumulative.replace(t);
+                                cumulative_vec.forced_push_at(index, t, exit)?;
                             }
                         }
                     }
@@ -531,7 +541,7 @@ where
 
     pub fn starting_index(&self, max_from: I) -> I {
         max_from.min(I::from(
-            self.vecs().into_iter().map(|v| v.len()).min().unwrap(),
+            self.iter_any_collectable().map(|v| v.len()).min().unwrap(),
         ))
     }
 
@@ -549,24 +559,24 @@ where
         self.max.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_90p(&self) -> &EagerVec<I, T> {
-        self._90p.as_ref().unwrap()
+    pub fn unwrap_pct90(&self) -> &EagerVec<I, T> {
+        self.pct90.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_75p(&self) -> &EagerVec<I, T> {
-        self._75p.as_ref().unwrap()
+    pub fn unwrap_pct75(&self) -> &EagerVec<I, T> {
+        self.pct75.as_ref().unwrap()
     }
     #[allow(unused)]
     pub fn unwrap_median(&self) -> &EagerVec<I, T> {
         self.median.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_25p(&self) -> &EagerVec<I, T> {
-        self._25p.as_ref().unwrap()
+    pub fn unwrap_pct25(&self) -> &EagerVec<I, T> {
+        self.pct25.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_10p(&self) -> &EagerVec<I, T> {
-        self._10p.as_ref().unwrap()
+    pub fn unwrap_pct10(&self) -> &EagerVec<I, T> {
+        self.pct10.as_ref().unwrap()
     }
     pub fn unwrap_min(&self) -> &EagerVec<I, T> {
         self.min.as_ref().unwrap()
@@ -579,47 +589,96 @@ where
         self.cumulative.as_ref().unwrap()
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        let mut v: Vec<&dyn AnyCollectableVec> = vec![];
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
 
-        if let Some(first) = self.first.as_ref() {
-            v.push(first.as_ref());
-        }
-        if let Some(last) = self.last.as_ref() {
-            v.push(last.as_ref());
-        }
-        if let Some(min) = self.min.as_ref() {
-            v.push(min.as_ref());
-        }
-        if let Some(max) = self.max.as_ref() {
-            v.push(max.as_ref());
-        }
-        if let Some(median) = self.median.as_ref() {
-            v.push(median.as_ref());
-        }
-        if let Some(average) = self.average.as_ref() {
-            v.push(average.as_ref());
-        }
-        if let Some(sum) = self.sum.as_ref() {
-            v.push(sum.as_ref());
-        }
-        if let Some(cumulative) = self.cumulative.as_ref() {
-            v.push(cumulative.as_ref());
-        }
-        if let Some(_90p) = self._90p.as_ref() {
-            v.push(_90p.as_ref());
-        }
-        if let Some(_75p) = self._75p.as_ref() {
-            v.push(_75p.as_ref());
-        }
-        if let Some(_25p) = self._25p.as_ref() {
-            v.push(_25p.as_ref());
-        }
-        if let Some(_10p) = self._10p.as_ref() {
-            v.push(_10p.as_ref());
-        }
+        iter = Box::new(
+            iter.chain(
+                self.first
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.last
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.min
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.max
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.median
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.average
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.sum
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.cumulative
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct90
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct75
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct25
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct10
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
 
-        v
+        iter
     }
 
     pub fn safe_flush(&mut self, exit: &Exit) -> Result<()> {
@@ -647,58 +706,58 @@ where
         if let Some(cumulative) = self.cumulative.as_mut() {
             cumulative.safe_flush(exit)?;
         }
-        if let Some(_90p) = self._90p.as_mut() {
-            _90p.safe_flush(exit)?;
+        if let Some(pct90) = self.pct90.as_mut() {
+            pct90.safe_flush(exit)?;
         }
-        if let Some(_75p) = self._75p.as_mut() {
-            _75p.safe_flush(exit)?;
+        if let Some(pct75) = self.pct75.as_mut() {
+            pct75.safe_flush(exit)?;
         }
-        if let Some(_25p) = self._25p.as_mut() {
-            _25p.safe_flush(exit)?;
+        if let Some(pct25) = self.pct25.as_mut() {
+            pct25.safe_flush(exit)?;
         }
-        if let Some(_10p) = self._10p.as_mut() {
-            _10p.safe_flush(exit)?;
+        if let Some(pct10) = self.pct10.as_mut() {
+            pct10.safe_flush(exit)?;
         }
 
         Ok(())
     }
 
-    pub fn validate_computed_version_or_reset_file(&mut self, version: Version) -> Result<()> {
+    pub fn validate_computed_version_or_reset(&mut self, version: Version) -> Result<()> {
         if let Some(first) = self.first.as_mut() {
-            first.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            first.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(last) = self.last.as_mut() {
-            last.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            last.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(min) = self.min.as_mut() {
-            min.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            min.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(max) = self.max.as_mut() {
-            max.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            max.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(median) = self.median.as_mut() {
-            median.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            median.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(average) = self.average.as_mut() {
-            average.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            average.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(sum) = self.sum.as_mut() {
-            sum.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            sum.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(cumulative) = self.cumulative.as_mut() {
-            cumulative.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            cumulative.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_90p) = self._90p.as_mut() {
-            _90p.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+        if let Some(pct90) = self.pct90.as_mut() {
+            pct90.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_75p) = self._75p.as_mut() {
-            _75p.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+        if let Some(pct75) = self.pct75.as_mut() {
+            pct75.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_25p) = self._25p.as_mut() {
-            _25p.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+        if let Some(pct25) = self.pct25.as_mut() {
+            pct25.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_10p) = self._10p.as_mut() {
-            _10p.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+        if let Some(pct10) = self.pct10.as_mut() {
+            pct10.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
 
         Ok(())
@@ -710,11 +769,11 @@ pub struct VecBuilderOptions {
     average: bool,
     sum: bool,
     max: bool,
-    _90p: bool,
-    _75p: bool,
+    pct90: bool,
+    pct75: bool,
     median: bool,
-    _25p: bool,
-    _10p: bool,
+    pct25: bool,
+    pct10: bool,
     min: bool,
     first: bool,
     last: bool,
@@ -734,24 +793,24 @@ impl VecBuilderOptions {
         self.max
     }
 
-    pub fn _90p(&self) -> bool {
-        self._90p
+    pub fn pct90(&self) -> bool {
+        self.pct90
     }
 
-    pub fn _75p(&self) -> bool {
-        self._75p
+    pub fn pct75(&self) -> bool {
+        self.pct75
     }
 
     pub fn median(&self) -> bool {
         self.median
     }
 
-    pub fn _25p(&self) -> bool {
-        self._25p
+    pub fn pct25(&self) -> bool {
+        self.pct25
     }
 
-    pub fn _10p(&self) -> bool {
-        self._10p
+    pub fn pct10(&self) -> bool {
+        self.pct10
     }
 
     pub fn min(&self) -> bool {
@@ -807,26 +866,26 @@ impl VecBuilderOptions {
     }
 
     #[allow(unused)]
-    pub fn add_90p(mut self) -> Self {
-        self._90p = true;
+    pub fn add_pct90(mut self) -> Self {
+        self.pct90 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_75p(mut self) -> Self {
-        self._75p = true;
+    pub fn add_pct75(mut self) -> Self {
+        self.pct75 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_25p(mut self) -> Self {
-        self._25p = true;
+    pub fn add_pct25(mut self) -> Self {
+        self.pct25 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_10p(mut self) -> Self {
-        self._10p = true;
+    pub fn add_pct10(mut self) -> Self {
+        self.pct10 = true;
         self
     }
 
@@ -866,26 +925,26 @@ impl VecBuilderOptions {
     }
 
     #[allow(unused)]
-    pub fn rm_90p(mut self) -> Self {
-        self._90p = false;
+    pub fn rm_pct90(mut self) -> Self {
+        self.pct90 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_75p(mut self) -> Self {
-        self._75p = false;
+    pub fn rm_pct75(mut self) -> Self {
+        self.pct75 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_25p(mut self) -> Self {
-        self._25p = false;
+    pub fn rm_pct25(mut self) -> Self {
+        self.pct25 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_10p(mut self) -> Self {
-        self._10p = false;
+    pub fn rm_pct10(mut self) -> Self {
+        self.pct10 = false;
         self
     }
 
@@ -902,20 +961,20 @@ impl VecBuilderOptions {
     }
 
     pub fn add_percentiles(mut self) -> Self {
-        self._90p = true;
-        self._75p = true;
+        self.pct90 = true;
+        self.pct75 = true;
         self.median = true;
-        self._25p = true;
-        self._10p = true;
+        self.pct25 = true;
+        self.pct10 = true;
         self
     }
 
     pub fn remove_percentiles(mut self) -> Self {
-        self._90p = false;
-        self._75p = false;
+        self.pct90 = false;
+        self.pct75 = false;
         self.median = false;
-        self._25p = false;
-        self._10p = false;
+        self.pct25 = false;
+        self.pct10 = false;
         self
     }
 
@@ -924,11 +983,11 @@ impl VecBuilderOptions {
             self.average,
             self.sum,
             self.max,
-            self._90p,
-            self._75p,
+            self.pct90,
+            self.pct75,
             self.median,
-            self._25p,
-            self._10p,
+            self.pct25,
+            self.pct10,
             self.min,
             self.first,
             self.last,

crates/brk_computer/src/grouped/builder_lazy.rs

@@ -0,0 +1,425 @@
+use allocative::Allocative;
+use brk_structs::Version;
+use vecdb::{
+    AnyBoxedIterableVec, AnyCloneableIterableVec, AnyCollectableVec, FromCoarserIndex,
+    LazyVecFrom2, StoredIndex,
+};
+
+use crate::grouped::{EagerVecBuilder, VecBuilderOptions};
+
+use super::ComputedType;
+
+#[allow(clippy::type_complexity)]
+#[derive(Clone, Allocative)]
+pub struct LazyVecBuilder<I, T, S1I, S2T>
+where
+    I: StoredIndex,
+    T: ComputedType,
+    S2T: ComputedType,
+{
+    pub first: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub average: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub sum: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub max: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub min: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub last: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub cumulative: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+}
+
+const VERSION: Version = Version::ZERO;
+
+impl<I, T, S1I, S2T> LazyVecBuilder<I, T, S1I, S2T>
+where
+    I: StoredIndex,
+    T: ComputedType + 'static,
+    S1I: StoredIndex + 'static + FromCoarserIndex<I>,
+    S2T: ComputedType,
+{
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        name: &str,
+        version: Version,
+        source: Option<AnyBoxedIterableVec<S1I, T>>,
+        source_extra: &EagerVecBuilder<S1I, T>,
+        len_source: AnyBoxedIterableVec<I, S2T>,
+        options: LazyVecBuilderOptions,
+    ) -> Self {
+        let only_one_active = options.is_only_one_active();
+
+        let suffix = |s: &str| format!("{name}_{s}");
+
+        let maybe_suffix = |s: &str| {
+            if only_one_active {
+                name.to_string()
+            } else {
+                suffix(s)
+            }
+        };
+
+        Self {
+            first: options.first.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &maybe_suffix("first"),
+                    version + VERSION + Version::ZERO,
+                    source_extra
+                        .first
+                        .as_ref()
+                        .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        source
+                            .next_at(S1I::min_from(i))
+                            .map(|(_, cow)| cow.into_owned())
+                    },
+                ))
+            }),
+            last: options.last.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    name,
+                    version + VERSION + Version::ZERO,
+                    source_extra.last.as_ref().map_or_else(
+                        || {
+                            source
+                                .as_ref()
+                                .unwrap_or_else(|| {
+                                    dbg!(name, I::to_string());
+                                    panic!()
+                                })
+                                .clone()
+                        },
+                        |v| v.clone(),
+                    ),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        source
+                            .next_at(S1I::max_from(i, source.len()))
+                            .map(|(_, cow)| cow.into_owned())
+                    },
+                ))
+            }),
+            min: options.min.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &maybe_suffix("min"),
+                    version + VERSION + Version::ZERO,
+                    source_extra
+                        .min
+                        .as_ref()
+                        .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        S1I::inclusive_range_from(i, source.len())
+                            .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
+                            .min()
+                    },
+                ))
+            }),
+            max: options.max.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &maybe_suffix("max"),
+                    version + VERSION + Version::ZERO,
+                    source_extra
+                        .max
+                        .as_ref()
+                        .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        S1I::inclusive_range_from(i, source.len())
+                            .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
+                            .max()
+                    },
+                ))
+            }),
+            average: options.average.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &maybe_suffix("avg"),
+                    version + VERSION + Version::ZERO,
+                    source_extra
+                        .average
+                        .as_ref()
+                        .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        let vec = S1I::inclusive_range_from(i, source.len())
+                            .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
+                            .collect::<Vec<_>>();
+                        if vec.is_empty() {
+                            return None;
+                        }
+                        let mut sum = T::from(0);
+                        let len = vec.len();
+                        vec.into_iter().for_each(|v| sum += v);
+                        Some(sum / len)
+                    },
+                ))
+            }),
+            sum: options.sum.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &(if !options.last && !options.average && !options.min && !options.max {
+                        name.to_string()
+                    } else {
+                        maybe_suffix("sum")
+                    }),
+                    version + VERSION + Version::ZERO,
+                    source_extra
+                        .sum
+                        .as_ref()
+                        .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        let vec = S1I::inclusive_range_from(i, source.len())
+                            .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
+                            .collect::<Vec<_>>();
+                        if vec.is_empty() {
+                            return None;
+                        }
+                        let mut sum = T::from(0);
+                        vec.into_iter().for_each(|v| sum += v);
+                        Some(sum)
+                    },
+                ))
+            }),
+            cumulative: options.cumulative.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &suffix("cumulative"),
+                    version + VERSION + Version::ZERO,
+                    source_extra.cumulative.as_ref().unwrap().boxed_clone(),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        source
+                            .next_at(S1I::max_from(i, source.len()))
+                            .map(|(_, cow)| cow.into_owned())
+                    },
+                ))
+            }),
+        }
+    }
+
+    pub fn starting_index(&self, max_from: I) -> I {
+        max_from.min(I::from(
+            self.iter_any_collectable().map(|v| v.len()).min().unwrap(),
+        ))
+    }
+
+    pub fn unwrap_first(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.first.as_ref().unwrap()
+    }
+    #[allow(unused)]
+    pub fn unwrap_average(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.average.as_ref().unwrap()
+    }
+    pub fn unwrap_sum(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.sum.as_ref().unwrap()
+    }
+    pub fn unwrap_max(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.max.as_ref().unwrap()
+    }
+    pub fn unwrap_min(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.min.as_ref().unwrap()
+    }
+    pub fn unwrap_last(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.last.as_ref().unwrap()
+    }
+    #[allow(unused)]
+    pub fn unwrap_cumulative(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.cumulative.as_ref().unwrap()
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
+
+        iter = Box::new(
+            iter.chain(
+                self.first
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.last
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.min
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.max
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.average
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.sum
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.cumulative
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+
+        iter
+    }
+}
+
+#[derive(Default, Clone, Copy)]
+pub struct LazyVecBuilderOptions {
+    average: bool,
+    sum: bool,
+    max: bool,
+    min: bool,
+    first: bool,
+    last: bool,
+    cumulative: bool,
+}
+
+impl From<VecBuilderOptions> for LazyVecBuilderOptions {
+    fn from(value: VecBuilderOptions) -> Self {
+        Self {
+            average: value.average(),
+            sum: value.sum(),
+            max: value.max(),
+            min: value.min(),
+            first: value.first(),
+            last: value.last(),
+            cumulative: value.cumulative(),
+        }
+    }
+}
+
+impl LazyVecBuilderOptions {
+    pub fn add_first(mut self) -> Self {
+        self.first = true;
+        self
+    }
+
+    pub fn add_last(mut self) -> Self {
+        self.last = true;
+        self
+    }
+
+    pub fn add_min(mut self) -> Self {
+        self.min = true;
+        self
+    }
+
+    pub fn add_max(mut self) -> Self {
+        self.max = true;
+        self
+    }
+
+    pub fn add_average(mut self) -> Self {
+        self.average = true;
+        self
+    }
+
+    pub fn add_sum(mut self) -> Self {
+        self.sum = true;
+        self
+    }
+
+    pub fn add_cumulative(mut self) -> Self {
+        self.cumulative = true;
+        self
+    }
+
+    #[allow(unused)]
+    pub fn rm_min(mut self) -> Self {
+        self.min = false;
+        self
+    }
+
+    #[allow(unused)]
+    pub fn rm_max(mut self) -> Self {
+        self.max = false;
+        self
+    }
+
+    #[allow(unused)]
+    pub fn rm_average(mut self) -> Self {
+        self.average = false;
+        self
+    }
+
+    #[allow(unused)]
+    pub fn rm_sum(mut self) -> Self {
+        self.sum = false;
+        self
+    }
+
+    #[allow(unused)]
+    pub fn rm_cumulative(mut self) -> Self {
+        self.cumulative = false;
+        self
+    }
+
+    pub fn add_minmax(mut self) -> Self {
+        self.min = true;
+        self.max = true;
+        self
+    }
+
+    pub fn is_only_one_active(&self) -> bool {
+        [
+            self.average,
+            self.sum,
+            self.max,
+            self.min,
+            self.first,
+            self.last,
+            self.cumulative,
+        ]
+        .iter()
+        .filter(|b| **b)
+        .count()
+            == 1
+    }
+
+    pub fn copy_self_extra(&self) -> Self {
+        Self {
+            cumulative: self.cumulative,
+            ..Self::default()
+        }
+    }
+}

crates/brk_computer/src/grouped/computed.rs

@@ -0,0 +1,23 @@
+use std::ops::{Add, AddAssign, Div};
+
+use vecdb::StoredCompressed;
+
+pub trait ComputedType
+where
+    Self: StoredCompressed
+        + From<usize>
+        + Div<usize, Output = Self>
+        + Add<Output = Self>
+        + AddAssign
+        + Ord,
+{
+}
+impl<T> ComputedType for T where
+    T: StoredCompressed
+        + From<usize>
+        + Div<usize, Output = Self>
+        + Add<Output = Self>
+        + AddAssign
+        + Ord
+{
+}

crates/brk_computer/src/grouped/from_dateindex.rs

@@ -0,0 +1,164 @@
+use allocative::Allocative;
+use brk_error::Result;
+
+use brk_structs::{
+    DateIndex, DecadeIndex, MonthIndex, QuarterIndex, SemesterIndex, Version, WeekIndex, YearIndex,
+};
+use vecdb::{AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Database, EagerVec, Exit};
+
+use crate::{Indexes, grouped::LazyVecBuilder, indexes};
+
+use super::{ComputedType, EagerVecBuilder, Source, VecBuilderOptions};
+
+#[derive(Clone, Allocative)]
+pub struct ComputedVecsFromDateIndex<T>
+where
+    T: ComputedType + PartialOrd,
+{
+    pub dateindex: Option<EagerVec<DateIndex, T>>,
+    pub dateindex_extra: EagerVecBuilder<DateIndex, T>,
+    pub weekindex: LazyVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
+    pub monthindex: LazyVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
+    pub quarterindex: LazyVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
+    pub semesterindex: LazyVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
+    pub yearindex: LazyVecBuilder<YearIndex, T, DateIndex, YearIndex>,
+    pub decadeindex: LazyVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
+}
+
+const VERSION: Version = Version::ZERO;
+
+impl<T> ComputedVecsFromDateIndex<T>
+where
+    T: ComputedType + 'static,
+{
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        source: Source<DateIndex, T>,
+        version: Version,
+        indexes: &indexes::Vecs,
+        options: VecBuilderOptions,
+    ) -> Result<Self> {
+        let dateindex = source.is_compute().then(|| {
+            EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO).unwrap()
+        });
+
+        let dateindex_extra = EagerVecBuilder::forced_import_compressed(
+            db,
+            name,
+            version + VERSION + Version::ZERO,
+            options.copy_self_extra(),
+        )?;
+
+        let options = options.remove_percentiles();
+
+        let dateindex_source = source.vec().or(dateindex.as_ref().map(|v| v.boxed_clone()));
+
+        Ok(Self {
+            weekindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.weekindex_to_weekindex.boxed_clone(),
+                options.into(),
+            ),
+            monthindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.monthindex_to_monthindex.boxed_clone(),
+                options.into(),
+            ),
+            quarterindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.quarterindex_to_quarterindex.boxed_clone(),
+                options.into(),
+            ),
+            semesterindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.semesterindex_to_semesterindex.boxed_clone(),
+                options.into(),
+            ),
+            yearindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.yearindex_to_yearindex.boxed_clone(),
+                options.into(),
+            ),
+            decadeindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.decadeindex_to_decadeindex.boxed_clone(),
+                options.into(),
+            ),
+            dateindex,
+            dateindex_extra,
+        })
+    }
+
+    pub fn compute_all<F>(
+        &mut self,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        mut compute: F,
+    ) -> Result<()>
+    where
+        F: FnMut(&mut EagerVec<DateIndex, T>) -> Result<()>,
+    {
+        compute(self.dateindex.as_mut().unwrap())?;
+
+        let dateindex: Option<&EagerVec<DateIndex, T>> = None;
+        self.compute_rest(starting_indexes, exit, dateindex)
+    }
+
+    pub fn compute_rest(
+        &mut self,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        dateindex: Option<&impl AnyIterableVec<DateIndex, T>>,
+    ) -> Result<()> {
+        if let Some(dateindex) = dateindex {
+            self.dateindex_extra
+                .extend(starting_indexes.dateindex, dateindex, exit)?;
+        } else {
+            let dateindex = self.dateindex.as_ref().unwrap();
+
+            self.dateindex_extra
+                .extend(starting_indexes.dateindex, dateindex, exit)?;
+        }
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
+            self.dateindex
+                .as_ref()
+                .map(|x| x as &dyn AnyCollectableVec)
+                .into_iter(),
+        );
+
+        iter = Box::new(iter.chain(self.dateindex_extra.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
+
+        iter
+    }
+}

crates/brk_computer/src/grouped/from_height.rs

@@ -1,24 +1,21 @@
-use std::path::Path;
+use allocative::Allocative;
+use brk_error::Result;
 
-use brk_core::{
-    DateIndex, DecadeIndex, DifficultyEpoch, Height, MonthIndex, QuarterIndex, Result,
-    SemesterIndex, Version, WeekIndex, YearIndex,
-};
-use brk_exit::Exit;
-use brk_indexer::Indexer;
-use brk_vec::{
-    AnyCollectableVec, AnyIterableVec, CloneableAnyIterableVec, Computation, EagerVec, Format,
+use brk_structs::{
+    DateIndex, DecadeIndex, DifficultyEpoch, Height, MonthIndex, QuarterIndex, SemesterIndex,
+    Version, WeekIndex, YearIndex,
 };
+use vecdb::{AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Database, EagerVec, Exit};
 
-use crate::vecs::{
+use crate::{
     Indexes,
-    grouped::{ComputedVecBuilder, Source},
+    grouped::{LazyVecBuilder, Source},
     indexes,
 };
 
 use super::{ComputedType, EagerVecBuilder, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedVecsFromHeight<T>
 where
     T: ComputedType + PartialOrd,
@@ -26,14 +23,14 @@ where
     pub height: Option<EagerVec<Height, T>>,
     pub height_extra: EagerVecBuilder<Height, T>,
     pub dateindex: EagerVecBuilder<DateIndex, T>,
-    pub weekindex: ComputedVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
+    pub weekindex: LazyVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
     pub difficultyepoch: EagerVecBuilder<DifficultyEpoch, T>,
-    pub monthindex: ComputedVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
-    pub quarterindex: ComputedVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
-    pub semesterindex: ComputedVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
-    pub yearindex: ComputedVecBuilder<YearIndex, T, DateIndex, YearIndex>,
+    pub monthindex: LazyVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
+    pub quarterindex: LazyVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
+    pub semesterindex: LazyVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
+    pub yearindex: LazyVecBuilder<YearIndex, T, DateIndex, YearIndex>,
     // TODO: pub halvingepoch: StorableVecGeneator<Halvingepoch, T>,
-    pub decadeindex: ComputedVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
+    pub decadeindex: LazyVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
 }
 
 const VERSION: Version = Version::ZERO;
@@ -45,113 +42,90 @@ where
 {
     #[allow(clippy::too_many_arguments)]
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         name: &str,
         source: Source<Height, T>,
         version: Version,
-        format: Format,
-        computation: Computation,
         indexes: &indexes::Vecs,
         options: VecBuilderOptions,
-    ) -> color_eyre::Result<Self> {
+    ) -> Result<Self> {
         let height = source.is_compute().then(|| {
-            EagerVec::forced_import(path, name, version + VERSION + Version::ZERO, format).unwrap()
+            EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO).unwrap()
         });
 
-        let height_extra = EagerVecBuilder::forced_import(
-            path,
+        let height_extra = EagerVecBuilder::forced_import_compressed(
+            db,
             name,
             version + VERSION + Version::ZERO,
-            format,
             options.copy_self_extra(),
         )?;
 
-        let dateindex = EagerVecBuilder::forced_import(
-            path,
+        let dateindex = EagerVecBuilder::forced_import_compressed(
+            db,
             name,
             version + VERSION + Version::ZERO,
-            format,
             options,
         )?;
 
         let options = options.remove_percentiles();
 
         Ok(Self {
-            weekindex: ComputedVecBuilder::forced_import(
-                path,
+            weekindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                computation,
                 None,
                 &dateindex,
                 indexes.weekindex_to_weekindex.boxed_clone(),
                 options.into(),
-            )?,
-            monthindex: ComputedVecBuilder::forced_import(
-                path,
+            ),
+            monthindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                Computation::Lazy,
                 None,
                 &dateindex,
                 indexes.monthindex_to_monthindex.boxed_clone(),
                 options.into(),
-            )?,
-            quarterindex: ComputedVecBuilder::forced_import(
-                path,
+            ),
+            quarterindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                Computation::Lazy,
                 None,
                 &dateindex,
                 indexes.quarterindex_to_quarterindex.boxed_clone(),
                 options.into(),
-            )?,
-            semesterindex: ComputedVecBuilder::forced_import(
-                path,
+            ),
+            semesterindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                Computation::Lazy,
                 None,
                 &dateindex,
                 indexes.semesterindex_to_semesterindex.boxed_clone(),
                 options.into(),
-            )?,
-            yearindex: ComputedVecBuilder::forced_import(
-                path,
+            ),
+            yearindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                Computation::Lazy,
                 None,
                 &dateindex,
                 indexes.yearindex_to_yearindex.boxed_clone(),
                 options.into(),
-            )?,
-            decadeindex: ComputedVecBuilder::forced_import(
-                path,
+            ),
+            decadeindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                Computation::Lazy,
                 None,
                 &dateindex,
                 indexes.decadeindex_to_decadeindex.boxed_clone(),
                 options.into(),
-            )?,
-            // halvingepoch: StorableVecGeneator::forced_import(path, name, version + VERSION + Version::ZERO, format, options)?,
+            ),
+            // halvingepoch: StorableVecGeneator::forced_import(db, name, version + VERSION + Version::ZERO, format, options)?,
             height,
             height_extra,
             dateindex,
-            difficultyepoch: EagerVecBuilder::forced_import(
-                path,
+            difficultyepoch: EagerVecBuilder::forced_import_compressed(
+                db,
                 name,
                 version + VERSION + Version::ZERO,
-                format,
                 options,
             )?,
         })
@@ -159,22 +133,15 @@ where
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
-    ) -> color_eyre::Result<()>
+    ) -> Result<()>
     where
-        F: FnMut(&mut EagerVec<Height, T>, &Indexer, &indexes::Vecs, &Indexes, &Exit) -> Result<()>,
+        F: FnMut(&mut EagerVec<Height, T>) -> Result<()>,
     {
-        compute(
-            self.height.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
+        compute(self.height.as_mut().unwrap())?;
 
         let height: Option<&EagerVec<Height, T>> = None;
         self.compute_rest(indexes, starting_indexes, exit, height)
@@ -186,7 +153,7 @@ where
         starting_indexes: &Indexes,
         exit: &Exit,
         height_vec: Option<&impl AnyIterableVec<Height, T>>,
-    ) -> color_eyre::Result<()> {
+    ) -> Result<()> {
         if let Some(height) = height_vec {
             self.height_extra
                 .extend(starting_indexes.height, height, exit)?;
@@ -229,63 +196,27 @@ where
             )?;
         }
 
-        self.weekindex.compute_if_necessary(
-            starting_indexes.weekindex,
-            &indexes.weekindex_to_dateindex_count,
-            exit,
-        )?;
-
-        self.monthindex.compute_if_necessary(
-            starting_indexes.monthindex,
-            &indexes.monthindex_to_dateindex_count,
-            exit,
-        )?;
-
-        self.quarterindex.compute_if_necessary(
-            starting_indexes.quarterindex,
-            &indexes.quarterindex_to_monthindex_count,
-            exit,
-        )?;
-
-        self.semesterindex.compute_if_necessary(
-            starting_indexes.semesterindex,
-            &indexes.semesterindex_to_monthindex_count,
-            exit,
-        )?;
-
-        self.yearindex.compute_if_necessary(
-            starting_indexes.yearindex,
-            &indexes.yearindex_to_monthindex_count,
-            exit,
-        )?;
-
-        self.decadeindex.compute_if_necessary(
-            starting_indexes.decadeindex,
-            &indexes.decadeindex_to_yearindex_count,
-            exit,
-        )?;
-
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
             self.height
                 .as_ref()
-                .map_or(vec![], |v| vec![v as &dyn AnyCollectableVec]),
-            self.height_extra.vecs(),
-            self.dateindex.vecs(),
-            self.weekindex.vecs(),
-            self.difficultyepoch.vecs(),
-            self.monthindex.vecs(),
-            self.quarterindex.vecs(),
-            self.semesterindex.vecs(),
-            self.yearindex.vecs(),
-            // self.halvingepoch.vecs(),
-            self.decadeindex.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+                .map(|x| x as &dyn AnyCollectableVec)
+                .into_iter(),
+        );
+
+        iter = Box::new(iter.chain(self.height_extra.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.dateindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.difficultyepoch.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
+
+        iter
     }
 }

crates/brk_computer/src/grouped/from_height_strict.rs

@@ -1,11 +1,9 @@
-use std::path::Path;
+use brk_error::Result;
 
-use brk_core::{DifficultyEpoch, Height, Result, Version};
-use brk_exit::Exit;
-use brk_indexer::Indexer;
-use brk_vec::{AnyCollectableVec, EagerVec, Format};
+use brk_structs::{DifficultyEpoch, Height, Version};
+use vecdb::{AnyCollectableVec, Database, EagerVec, Exit};
 
-use crate::vecs::{Indexes, indexes};
+use crate::{Indexes, indexes};
 
 use super::{ComputedType, EagerVecBuilder, VecBuilderOptions};
 
@@ -28,20 +26,18 @@ where
     f64: From<T>,
 {
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         name: &str,
         version: Version,
-        format: Format,
         options: VecBuilderOptions,
-    ) -> color_eyre::Result<Self> {
+    ) -> Result<Self> {
         let height =
-            EagerVec::forced_import(path, name, version + VERSION + Version::ZERO, format)?;
+            EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO)?;
 
-        let height_extra = EagerVecBuilder::forced_import(
-            path,
+        let height_extra = EagerVecBuilder::forced_import_compressed(
+            db,
             name,
             version + VERSION + Version::ZERO,
-            format,
             options.copy_self_extra(),
         )?;
 
@@ -50,29 +46,27 @@ where
         Ok(Self {
             height,
             height_extra,
-            difficultyepoch: EagerVecBuilder::forced_import(
-                path,
+            difficultyepoch: EagerVecBuilder::forced_import_compressed(
+                db,
                 name,
                 version + VERSION + Version::ZERO,
-                format,
                 options,
             )?,
-            // halvingepoch: StorableVecGeneator::forced_import(path, name, version + VERSION + Version::ZERO, format, options)?,
+            // halvingepoch: StorableVecGeneator::forced_import(db, name, version + VERSION + Version::ZERO, format, options)?,
         })
     }
 
     pub fn compute<F>(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
-    ) -> color_eyre::Result<()>
+    ) -> Result<()>
     where
-        F: FnMut(&mut EagerVec<Height, T>, &Indexer, &indexes::Vecs, &Indexes, &Exit) -> Result<()>,
+        F: FnMut(&mut EagerVec<Height, T>) -> Result<()>,
     {
-        compute(&mut self.height, indexer, indexes, starting_indexes, exit)?;
+        compute(&mut self.height)?;
 
         self.height_extra
             .extend(starting_indexes.height, &self.height, exit)?;
@@ -88,15 +82,10 @@ where
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            vec![&self.height as &dyn AnyCollectableVec],
-            self.height_extra.vecs(),
-            self.difficultyepoch.vecs(),
-            // self.halvingepoch.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.height as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(self.height_extra.iter_any_collectable())
+            .chain(self.difficultyepoch.iter_any_collectable())
     }
 }

crates/brk_computer/src/grouped/from_txindex.rs

@@ -1,25 +1,24 @@
-use std::path::Path;
-
-use brk_core::{
-    Bitcoin, DateIndex, DecadeIndex, DifficultyEpoch, Dollars, Height, MonthIndex, QuarterIndex,
-    Result, Sats, SemesterIndex, TxIndex, Version, WeekIndex, YearIndex,
-};
-use brk_exit::Exit;
+use allocative::Allocative;
+use brk_error::Result;
 use brk_indexer::Indexer;
-use brk_vec::{
-    AnyCollectableVec, AnyVec, CloneableAnyIterableVec, CollectableVec, Computation, EagerVec,
-    Format, StoredIndex, VecIterator,
+use brk_structs::{
+    Bitcoin, DateIndex, DecadeIndex, DifficultyEpoch, Dollars, Height, MonthIndex, QuarterIndex,
+    Sats, SemesterIndex, TxIndex, Version, WeekIndex, YearIndex,
+};
+use vecdb::{
+    AnyCloneableIterableVec, AnyCollectableVec, AnyVec, CollectableVec, Database, EagerVec, Exit,
+    GenericStoredVec, StoredIndex, VecIterator,
 };
 
-use crate::vecs::{
-    Indexes, fetched,
-    grouped::{ComputedVecBuilder, Source},
-    indexes,
+use crate::{
+    Indexes,
+    grouped::{LazyVecBuilder, Source},
+    indexes, price,
 };
 
 use super::{ComputedType, EagerVecBuilder, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedVecsFromTxindex<T>
 where
     T: ComputedType + PartialOrd,
@@ -27,14 +26,14 @@ where
     pub txindex: Option<Box<EagerVec<TxIndex, T>>>,
     pub height: EagerVecBuilder<Height, T>,
     pub dateindex: EagerVecBuilder<DateIndex, T>,
-    pub weekindex: ComputedVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
+    pub weekindex: LazyVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
     pub difficultyepoch: EagerVecBuilder<DifficultyEpoch, T>,
-    pub monthindex: ComputedVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
-    pub quarterindex: ComputedVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
-    pub semesterindex: ComputedVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
-    pub yearindex: ComputedVecBuilder<YearIndex, T, DateIndex, YearIndex>,
+    pub monthindex: LazyVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
+    pub quarterindex: LazyVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
+    pub semesterindex: LazyVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
+    pub yearindex: LazyVecBuilder<YearIndex, T, DateIndex, YearIndex>,
     // TODO: pub halvingepoch: StorableVecGeneator<Halvingepoch, T>,
-    pub decadeindex: ComputedVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
+    pub decadeindex: LazyVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
 }
 
 const VERSION: Version = Version::ZERO;
@@ -46,119 +45,96 @@ where
 {
     #[allow(clippy::too_many_arguments)]
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         name: &str,
         source: Source<TxIndex, T>,
         version: Version,
-        format: Format,
-        computation: Computation,
         indexes: &indexes::Vecs,
         options: VecBuilderOptions,
-    ) -> color_eyre::Result<Self> {
+    ) -> Result<Self> {
         let txindex = source.is_compute().then(|| {
             Box::new(
-                EagerVec::forced_import(path, name, version + VERSION + Version::ZERO, format)
+                EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO)
                     .unwrap(),
             )
         });
 
-        let height = EagerVecBuilder::forced_import(
-            path,
+        let height = EagerVecBuilder::forced_import_compressed(
+            db,
             name,
             version + VERSION + Version::ZERO,
-            format,
             options,
         )?;
 
         let options = options.remove_percentiles();
 
-        let dateindex = EagerVecBuilder::forced_import(
-            path,
+        let dateindex = EagerVecBuilder::forced_import_compressed(
+            db,
             name,
             version + VERSION + Version::ZERO,
-            format,
             options,
         )?;
 
         Ok(Self {
-            weekindex: ComputedVecBuilder::forced_import(
-                path,
+            weekindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                computation,
                 None,
                 &dateindex,
                 indexes.weekindex_to_weekindex.boxed_clone(),
                 options.into(),
-            )?,
-            monthindex: ComputedVecBuilder::forced_import(
-                path,
+            ),
+            monthindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                Computation::Lazy,
                 None,
                 &dateindex,
                 indexes.monthindex_to_monthindex.boxed_clone(),
                 options.into(),
-            )?,
-            quarterindex: ComputedVecBuilder::forced_import(
-                path,
+            ),
+            quarterindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                Computation::Lazy,
                 None,
                 &dateindex,
                 indexes.quarterindex_to_quarterindex.boxed_clone(),
                 options.into(),
-            )?,
-            semesterindex: ComputedVecBuilder::forced_import(
-                path,
+            ),
+            semesterindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                Computation::Lazy,
                 None,
                 &dateindex,
                 indexes.semesterindex_to_semesterindex.boxed_clone(),
                 options.into(),
-            )?,
-            yearindex: ComputedVecBuilder::forced_import(
-                path,
+            ),
+            yearindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                Computation::Lazy,
                 None,
                 &dateindex,
                 indexes.yearindex_to_yearindex.boxed_clone(),
                 options.into(),
-            )?,
-            decadeindex: ComputedVecBuilder::forced_import(
-                path,
+            ),
+            decadeindex: LazyVecBuilder::forced_import(
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                Computation::Lazy,
                 None,
                 &dateindex,
                 indexes.decadeindex_to_decadeindex.boxed_clone(),
                 options.into(),
-            )?,
+            ),
 
             txindex,
             height,
             dateindex,
-            difficultyepoch: EagerVecBuilder::forced_import(
-                path,
+            difficultyepoch: EagerVecBuilder::forced_import_compressed(
+                db,
                 name,
                 version + VERSION + Version::ZERO,
-                format,
                 options,
             )?,
-            // halvingepoch: StorableVecGeneator::forced_import(path, name, version + VERSION + Version::ZERO, format, options)?,
+            // halvingepoch: StorableVecGeneator::forced_import(db, name, version + VERSION + Version::ZERO, format, options)?,
         })
     }
 
@@ -170,7 +146,7 @@ where
     //     starting_indexes: &Indexes,
     //     exit: &Exit,
     //     mut compute: F,
-    // ) -> color_eyre::Result<()>
+    // ) -> Result<()>
     // where
     //     F: FnMut(
     //         &mut EagerVec<TxIndex, T>,
@@ -239,42 +215,6 @@ where
             exit,
         )?;
 
-        self.weekindex.compute_if_necessary(
-            starting_indexes.weekindex,
-            &indexes.weekindex_to_first_dateindex,
-            exit,
-        )?;
-
-        self.monthindex.compute_if_necessary(
-            starting_indexes.monthindex,
-            &indexes.monthindex_to_dateindex_count,
-            exit,
-        )?;
-
-        self.quarterindex.compute_if_necessary(
-            starting_indexes.quarterindex,
-            &indexes.quarterindex_to_monthindex_count,
-            exit,
-        )?;
-
-        self.semesterindex.compute_if_necessary(
-            starting_indexes.semesterindex,
-            &indexes.semesterindex_to_monthindex_count,
-            exit,
-        )?;
-
-        self.yearindex.compute_if_necessary(
-            starting_indexes.yearindex,
-            &indexes.yearindex_to_monthindex_count,
-            exit,
-        )?;
-
-        self.decadeindex.compute_if_necessary(
-            starting_indexes.decadeindex,
-            &indexes.decadeindex_to_yearindex_count,
-            exit,
-        )?;
-
         self.difficultyepoch.from_aligned(
             starting_indexes.difficultyepoch,
             &self.height,
@@ -286,25 +226,25 @@ where
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
             self.txindex
                 .as_ref()
-                .map_or(vec![], |v| vec![v.as_ref() as &dyn AnyCollectableVec]),
-            self.height.vecs(),
-            self.dateindex.vecs(),
-            self.weekindex.vecs(),
-            self.difficultyepoch.vecs(),
-            self.monthindex.vecs(),
-            self.quarterindex.vecs(),
-            self.semesterindex.vecs(),
-            self.yearindex.vecs(),
-            // self.halvingepoch.vecs(),
-            self.decadeindex.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+                .map(|x| x.as_ref() as &dyn AnyCollectableVec)
+                .into_iter(),
+        );
+
+        iter = Box::new(iter.chain(self.height.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.dateindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.difficultyepoch.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
+
+        iter
     }
 }
 
@@ -325,7 +265,7 @@ impl ComputedVecsFromTxindex<Bitcoin> {
         };
 
         self.height
-            .validate_computed_version_or_reset_file(txindex_version)?;
+            .validate_computed_version_or_reset(txindex_version)?;
 
         let starting_index = self.height.starting_index(starting_indexes.height);
 
@@ -380,24 +320,24 @@ impl ComputedVecsFromTxindex<Bitcoin> {
                         exit,
                     )?;
                 }
-                if let Some(_90p) = self.height._90p.as_mut() {
-                    _90p.forced_push_at(
+                if let Some(pct90) = self.height.pct90.as_mut() {
+                    pct90.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_90p()
+                                .unwrap_pct90()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
                         exit,
                     )?;
                 }
-                if let Some(_75p) = self.height._75p.as_mut() {
-                    _75p.forced_push_at(
+                if let Some(pct75) = self.height.pct75.as_mut() {
+                    pct75.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_75p()
+                                .unwrap_pct75()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
@@ -416,24 +356,24 @@ impl ComputedVecsFromTxindex<Bitcoin> {
                         exit,
                     )?;
                 }
-                if let Some(_25p) = self.height._25p.as_mut() {
-                    _25p.forced_push_at(
+                if let Some(pct25) = self.height.pct25.as_mut() {
+                    pct25.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_25p()
+                                .unwrap_pct25()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
                         exit,
                     )?;
                 }
-                if let Some(_10p) = self.height._10p.as_mut() {
-                    _10p.forced_push_at(
+                if let Some(pct10) = self.height.pct10.as_mut() {
+                    pct10.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_10p()
+                                .unwrap_pct10()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
@@ -495,7 +435,7 @@ impl ComputedVecsFromTxindex<Dollars> {
         exit: &Exit,
         bitcoin: &ComputedVecsFromTxindex<Bitcoin>,
         txindex: Option<&impl CollectableVec<TxIndex, Dollars>>,
-        fetched: &fetched::Vecs,
+        price: &price::Vecs,
     ) -> Result<()> {
         let txindex_version = if let Some(txindex) = txindex {
             txindex.version()
@@ -504,11 +444,11 @@ impl ComputedVecsFromTxindex<Dollars> {
         };
 
         self.height
-            .validate_computed_version_or_reset_file(txindex_version)?;
+            .validate_computed_version_or_reset(txindex_version)?;
 
         let starting_index = self.height.starting_index(starting_indexes.height);
 
-        let mut close_iter = fetched.chainindexes_to_close.height.into_iter();
+        let mut close_iter = price.chainindexes_to_price_close.height.into_iter();
 
         (starting_index.unwrap_to_usize()..indexer.vecs.height_to_weight.len())
             .map(Height::from)
@@ -563,25 +503,25 @@ impl ComputedVecsFromTxindex<Dollars> {
                         exit,
                     )?;
                 }
-                if let Some(_90p) = self.height._90p.as_mut() {
-                    _90p.forced_push_at(
+                if let Some(pct90) = self.height.pct90.as_mut() {
+                    pct90.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_90p()
+                                .unwrap_pct90()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
                     )?;
                 }
-                if let Some(_75p) = self.height._75p.as_mut() {
-                    _75p.forced_push_at(
+                if let Some(pct75) = self.height.pct75.as_mut() {
+                    pct75.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_75p()
+                                .unwrap_pct75()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
@@ -599,25 +539,25 @@ impl ComputedVecsFromTxindex<Dollars> {
                         exit,
                     )?;
                 }
-                if let Some(_25p) = self.height._25p.as_mut() {
-                    _25p.forced_push_at(
+                if let Some(pct25) = self.height.pct25.as_mut() {
+                    pct25.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_25p()
+                                .unwrap_pct25()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
                     )?;
                 }
-                if let Some(_10p) = self.height._10p.as_mut() {
-                    _10p.forced_push_at(
+                if let Some(pct10) = self.height.pct10.as_mut() {
+                    pct10.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_10p()
+                                .unwrap_pct10()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,

crates/brk_computer/src/grouped/mod.rs

@@ -1,26 +1,28 @@
-mod builder_computed;
 mod builder_eager;
+mod builder_lazy;
+mod computed;
 mod from_dateindex;
 mod from_height;
 mod from_height_strict;
 mod from_txindex;
 mod ratio_from_dateindex;
+mod sd_from_dateindex;
 mod source;
-mod r#type;
 mod value_from_dateindex;
 mod value_from_height;
 mod value_from_txindex;
 mod value_height;
 
-pub use builder_computed::*;
 pub use builder_eager::*;
+pub use builder_lazy::*;
+use computed::*;
 pub use from_dateindex::*;
 pub use from_height::*;
 pub use from_height_strict::*;
 pub use from_txindex::*;
 pub use ratio_from_dateindex::*;
+pub use sd_from_dateindex::*;
 pub use source::*;
-use r#type::*;
 pub use value_from_dateindex::*;
 pub use value_from_height::*;
 pub use value_from_txindex::*;

crates/brk_computer/src/grouped/ratio_from_dateindex.rs

@@ -0,0 +1,763 @@
+use brk_error::Result;
+use brk_structs::{Date, DateIndex, Dollars, StoredF32, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, CollectableVec, Database, EagerVec,
+    Exit, GenericStoredVec, StoredIndex, VecIterator,
+};
+
+use crate::{
+    Indexes,
+    grouped::{
+        ComputedStandardDeviationVecsFromDateIndex, StandardDeviationVecsOptions, source::Source,
+    },
+    indexes, price,
+    utils::get_percentile,
+};
+
+use super::{ComputedVecsFromDateIndex, VecBuilderOptions};
+
+#[derive(Clone)]
+pub struct ComputedRatioVecsFromDateIndex {
+    pub price: Option<ComputedVecsFromDateIndex<Dollars>>,
+
+    pub ratio: ComputedVecsFromDateIndex<StoredF32>,
+    pub ratio_1w_sma: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_1m_sma: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct99: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct98: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct95: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct5: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct2: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct1: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct99_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct98_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct95_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct5_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct2_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct1_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+
+    pub ratio_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
+    pub ratio_4y_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
+    pub ratio_2y_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
+    pub ratio_1y_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
+}
+
+const VERSION: Version = Version::TWO;
+
+impl ComputedRatioVecsFromDateIndex {
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        source: Source<DateIndex, Dollars>,
+        version: Version,
+        indexes: &indexes::Vecs,
+        extended: bool,
+    ) -> Result<Self> {
+        let options = VecBuilderOptions::default().add_last();
+
+        Ok(Self {
+            price: source.is_compute().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    name,
+                    Source::Compute,
+                    version + VERSION,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &format!("{name}_ratio"),
+                Source::Compute,
+                version + VERSION + Version::ZERO,
+                indexes,
+                options,
+            )?,
+            ratio_1w_sma: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_1w_sma"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_1m_sma: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_1m_sma"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_sd: extended.then(|| {
+                ComputedStandardDeviationVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio"),
+                    usize::MAX,
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
+                )
+                .unwrap()
+            }),
+            ratio_1y_sd: extended.then(|| {
+                ComputedStandardDeviationVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_1y"),
+                    365,
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
+                )
+                .unwrap()
+            }),
+            ratio_2y_sd: extended.then(|| {
+                ComputedStandardDeviationVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_2y"),
+                    2 * 365,
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
+                )
+                .unwrap()
+            }),
+            ratio_4y_sd: extended.then(|| {
+                ComputedStandardDeviationVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_4y"),
+                    4 * 365,
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
+                )
+                .unwrap()
+            }),
+            ratio_pct99: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct99"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct98: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct98"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct95: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct95"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct5: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct5"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct2: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct2"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct1: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct1"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct99_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct99_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct98_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct98_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct95_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct95_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct5_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct5_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct2_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct2_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct1_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct1_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+        })
+    }
+
+    pub fn compute_all<F>(
+        &mut self,
+        price: &price::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        compute: F,
+    ) -> Result<()>
+    where
+        F: FnMut(&mut EagerVec<DateIndex, Dollars>) -> Result<()>,
+    {
+        self.price
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, compute)?;
+
+        let date_to_price_opt: Option<&EagerVec<DateIndex, Dollars>> = None;
+        self.compute_rest(price, starting_indexes, exit, date_to_price_opt)
+    }
+
+    pub fn compute_rest(
+        &mut self,
+        price: &price::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        price_opt: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+    ) -> Result<()> {
+        let closes = price.timeindexes_to_price_close.dateindex.as_ref().unwrap();
+
+        let price = price_opt.unwrap_or_else(|| unsafe {
+            std::mem::transmute(&self.price.as_ref().unwrap().dateindex)
+        });
+
+        self.ratio.compute_all(starting_indexes, exit, |v| {
+            v.compute_transform2(
+                starting_indexes.dateindex,
+                closes,
+                price,
+                |(i, close, price, ..)| {
+                    if price == Dollars::ZERO {
+                        (i, StoredF32::from(1.0))
+                    } else {
+                        (i, StoredF32::from(*close / price))
+                    }
+                },
+                exit,
+            )?;
+            Ok(())
+        })?;
+
+        if self.ratio_1w_sma.is_none() {
+            return Ok(());
+        }
+
+        let min_ratio_date = DateIndex::try_from(Date::MIN_RATIO).unwrap();
+
+        self.ratio_1w_sma
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sma_(
+                    starting_indexes.dateindex,
+                    self.ratio.dateindex.as_ref().unwrap(),
+                    7,
+                    exit,
+                    Some(min_ratio_date),
+                )?;
+                Ok(())
+            })?;
+
+        self.ratio_1m_sma
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sma_(
+                    starting_indexes.dateindex,
+                    self.ratio.dateindex.as_ref().unwrap(),
+                    30,
+                    exit,
+                    Some(min_ratio_date),
+                )?;
+                Ok(())
+            })?;
+
+        let ratio_version = self.ratio.dateindex.as_ref().unwrap().version();
+        self.mut_ratio_vecs()
+            .iter_mut()
+            .try_for_each(|v| -> Result<()> {
+                v.validate_computed_version_or_reset(
+                    Version::ZERO + v.inner_version() + ratio_version,
+                )?;
+                Ok(())
+            })?;
+
+        let starting_dateindex = self
+            .mut_ratio_vecs()
+            .iter()
+            .map(|v| DateIndex::from(v.len()))
+            .min()
+            .unwrap()
+            .min(starting_indexes.dateindex);
+
+        let mut sorted = self.ratio.dateindex.as_ref().unwrap().collect_range(
+            Some(min_ratio_date.unwrap_to_usize()),
+            Some(starting_dateindex.unwrap_to_usize()),
+        )?;
+
+        sorted.sort_unstable();
+
+        self.ratio
+            .dateindex
+            .as_ref()
+            .unwrap()
+            .iter_at(starting_dateindex)
+            .try_for_each(|(index, ratio)| -> Result<()> {
+                if index < min_ratio_date {
+                    self.ratio_pct5
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                    self.ratio_pct2
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                    self.ratio_pct1
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                    self.ratio_pct95
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                    self.ratio_pct98
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                    self.ratio_pct99
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                } else {
+                    let ratio = ratio.into_owned();
+                    let pos = sorted.binary_search(&ratio).unwrap_or_else(|pos| pos);
+                    sorted.insert(pos, ratio);
+
+                    self.ratio_pct1
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.01), exit)?;
+                    self.ratio_pct2
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.02), exit)?;
+                    self.ratio_pct5
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.05), exit)?;
+                    self.ratio_pct95
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.95), exit)?;
+                    self.ratio_pct98
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.98), exit)?;
+                    self.ratio_pct99
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.99), exit)?;
+                }
+
+                Ok(())
+            })?;
+
+        self.mut_ratio_vecs()
+            .into_iter()
+            .try_for_each(|v| v.safe_flush(exit))?;
+
+        self.ratio_pct1.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+        self.ratio_pct2.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+        self.ratio_pct5.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+        self.ratio_pct95.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+        self.ratio_pct98.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+        self.ratio_pct99.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+
+        let date_to_price = price_opt.unwrap_or_else(|| unsafe {
+            std::mem::transmute(&self.price.as_ref().unwrap().dateindex)
+        });
+
+        self.ratio_pct99_usd
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, |vec| {
+                let mut iter = self
+                    .ratio_pct99
+                    .as_ref()
+                    .unwrap()
+                    .dateindex
+                    .as_ref()
+                    .unwrap()
+                    .into_iter();
+                vec.compute_transform(
+                    starting_indexes.dateindex,
+                    date_to_price,
+                    |(i, price, ..)| {
+                        let multiplier = iter.unwrap_get_inner(i);
+                        (i, price * multiplier)
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        let compute_usd =
+            |usd: Option<&mut ComputedVecsFromDateIndex<Dollars>>,
+             source: Option<&ComputedVecsFromDateIndex<StoredF32>>| {
+                usd.unwrap().compute_all(starting_indexes, exit, |vec| {
+                    let mut iter = source.unwrap().dateindex.as_ref().unwrap().into_iter();
+                    vec.compute_transform(
+                        starting_indexes.dateindex,
+                        date_to_price,
+                        |(i, price, ..)| {
+                            let multiplier = iter.unwrap_get_inner(i);
+                            (i, price * multiplier)
+                        },
+                        exit,
+                    )?;
+                    Ok(())
+                })
+            };
+
+        compute_usd(self.ratio_pct1_usd.as_mut(), self.ratio_pct1.as_ref())?;
+        compute_usd(self.ratio_pct2_usd.as_mut(), self.ratio_pct2.as_ref())?;
+        compute_usd(self.ratio_pct5_usd.as_mut(), self.ratio_pct5.as_ref())?;
+        compute_usd(self.ratio_pct95_usd.as_mut(), self.ratio_pct95.as_ref())?;
+        compute_usd(self.ratio_pct98_usd.as_mut(), self.ratio_pct98.as_ref())?;
+        compute_usd(self.ratio_pct99_usd.as_mut(), self.ratio_pct99.as_ref())?;
+
+        self.ratio_sd.as_mut().unwrap().compute_all(
+            starting_indexes,
+            exit,
+            self.ratio.dateindex.as_ref().unwrap(),
+            Some(date_to_price),
+        )?;
+        self.ratio_4y_sd.as_mut().unwrap().compute_all(
+            starting_indexes,
+            exit,
+            self.ratio.dateindex.as_ref().unwrap(),
+            Some(date_to_price),
+        )?;
+        self.ratio_2y_sd.as_mut().unwrap().compute_all(
+            starting_indexes,
+            exit,
+            self.ratio.dateindex.as_ref().unwrap(),
+            Some(date_to_price),
+        )?;
+        self.ratio_1y_sd.as_mut().unwrap().compute_all(
+            starting_indexes,
+            exit,
+            self.ratio.dateindex.as_ref().unwrap(),
+            Some(date_to_price),
+        )?;
+
+        Ok(())
+    }
+
+    fn mut_ratio_vecs(&mut self) -> Vec<&mut EagerVec<DateIndex, StoredF32>> {
+        [
+            self.ratio_pct1
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+            self.ratio_pct2
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+            self.ratio_pct5
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+            self.ratio_pct95
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+            self.ratio_pct98
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+            self.ratio_pct99
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+        ]
+        .into_iter()
+        .flatten()
+        .collect::<Vec<_>>()
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(self.price.iter().flat_map(|v| v.iter_any_collectable()));
+
+        iter = Box::new(iter.chain(self.ratio.iter_any_collectable()));
+        iter = Box::new(
+            iter.chain(
+                self.ratio_1w_sma
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_1m_sma
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.ratio_sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.ratio_1y_sd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_2y_sd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_4y_sd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct1
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct2
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct5
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct95
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct98
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct99
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct1_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct2_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct5_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct95_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct98_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct99_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+
+        iter
+    }
+}

crates/brk_computer/src/grouped/sd_from_dateindex.rs

@@ -0,0 +1,878 @@
+use brk_error::Result;
+use brk_structs::{CheckedSub, Date, DateIndex, Dollars, StoredF32, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, BoxedVecIterator, CollectableVec,
+    Database, EagerVec, Exit, GenericStoredVec, StoredIndex,
+};
+
+use crate::{Indexes, grouped::source::Source, indexes};
+
+use super::{ComputedVecsFromDateIndex, VecBuilderOptions};
+
+#[derive(Clone)]
+pub struct ComputedStandardDeviationVecsFromDateIndex {
+    days: usize,
+
+    pub sma: Option<ComputedVecsFromDateIndex<StoredF32>>,
+
+    pub sd: ComputedVecsFromDateIndex<StoredF32>,
+
+    pub zscore: Option<ComputedVecsFromDateIndex<StoredF32>>,
+
+    pub p0_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub p1sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub p1_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub p2sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub p2_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub p3sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m0_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m1sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m1_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m2sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m2_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m3sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+
+    pub _0sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p0_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p1sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p1_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p2sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p2_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p3sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m0_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m1sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m1_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m2sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m2_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m3sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+}
+
+#[derive(Debug, Default)]
+pub struct StandardDeviationVecsOptions {
+    zscore: bool,
+    bands: bool,
+    price_bands: bool,
+}
+
+impl StandardDeviationVecsOptions {
+    pub fn add_all(mut self) -> Self {
+        self.zscore = true;
+        self.bands = true;
+        self.price_bands = true;
+        self
+    }
+
+    pub fn add_zscore(mut self) -> Self {
+        self.zscore = true;
+        self
+    }
+
+    pub fn add_bands(mut self) -> Self {
+        self.bands = true;
+        self
+    }
+
+    pub fn add_price_bands(mut self) -> Self {
+        self.bands = true;
+        self.price_bands = true;
+        self
+    }
+
+    pub fn zscore(&self) -> bool {
+        self.zscore
+    }
+
+    pub fn bands(&self) -> bool {
+        self.bands
+    }
+
+    pub fn price_bands(&self) -> bool {
+        self.price_bands
+    }
+}
+
+impl ComputedStandardDeviationVecsFromDateIndex {
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        days: usize,
+        sma: Source<DateIndex, StoredF32>,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+        options: StandardDeviationVecsOptions,
+    ) -> Result<Self> {
+        let builder_options = VecBuilderOptions::default().add_last();
+
+        let version = parent_version + Version::ONE;
+
+        Ok(Self {
+            days,
+            sma: sma.is_compute().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_sma"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            sd: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &format!("{name}_sd"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                builder_options,
+            )?,
+            p0_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p0_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p1sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p1sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p1_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p1_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p2sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p2sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p2_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p2_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p3sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p3sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m0_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m0_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m1sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m1sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m1_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m1_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m2sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m2sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m2_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m2_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m3sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m3sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            _0sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_0sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p0_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p0_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p1sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p1sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p1_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p1_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p2sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p2sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p2_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p2_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p3sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p3sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m0_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m0_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m1sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m1sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m1_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m1_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m2sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m2sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m2_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m2_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m3sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m3sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            zscore: options.zscore().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_zscore"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+        })
+    }
+
+    pub fn compute_all(
+        &mut self,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        source: &impl CollectableVec<DateIndex, StoredF32>,
+        price_opt: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+    ) -> Result<()> {
+        let min_date = DateIndex::try_from(Date::MIN_RATIO).unwrap();
+
+        self.sma
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sma_(
+                    starting_indexes.dateindex,
+                    source,
+                    self.days,
+                    exit,
+                    Some(min_date),
+                )?;
+                Ok(())
+            })?;
+
+        let sma_opt: Option<&EagerVec<DateIndex, StoredF32>> = None;
+        self.compute_rest(starting_indexes, exit, sma_opt, source, price_opt)
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn compute_rest(
+        &mut self,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        sma_opt: Option<&impl AnyIterableVec<DateIndex, StoredF32>>,
+        source: &impl CollectableVec<DateIndex, StoredF32>,
+        price_opt: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+    ) -> Result<()> {
+        let sma = sma_opt.unwrap_or_else(|| unsafe {
+            std::mem::transmute(&self.sma.as_ref().unwrap().dateindex)
+        });
+
+        let min_date = DateIndex::try_from(Date::MIN_RATIO).unwrap();
+
+        let source_version = source.version();
+
+        self.mut_stateful_date_vecs()
+            .try_for_each(|v| -> Result<()> {
+                v.validate_computed_version_or_reset(
+                    Version::ZERO + v.inner_version() + source_version,
+                )?;
+                Ok(())
+            })?;
+
+        let starting_dateindex = self
+            .mut_stateful_date_vecs()
+            .map(|v| DateIndex::from(v.len()))
+            .min()
+            .unwrap()
+            .min(starting_indexes.dateindex);
+
+        let mut sorted = source.collect_range(
+            Some(min_date.unwrap_to_usize()),
+            Some(starting_dateindex.unwrap_to_usize()),
+        )?;
+
+        sorted.sort_unstable();
+
+        let mut sma_iter = sma.iter();
+
+        let mut p0_5sd = self.p0_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut p1sd = self.p1sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut p1_5sd = self.p1_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut p2sd = self.p2sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut p2_5sd = self.p2_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut p3sd = self.p3sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m0_5sd = self.m0_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m1sd = self.m1sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m1_5sd = self.m1_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m2sd = self.m2sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m2_5sd = self.m2_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m3sd = self.m3sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+
+        source
+            .iter_at(starting_dateindex)
+            .try_for_each(|(index, ratio)| -> Result<()> {
+                if index < min_date {
+                    self.sd.dateindex.as_mut().unwrap().forced_push_at(
+                        index,
+                        StoredF32::NAN,
+                        exit,
+                    )?;
+
+                    if let Some(v) = p0_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = p1sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = p1_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = p2sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = p2_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = p3sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m0_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m1sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m1_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m2sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m2_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m3sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                } else {
+                    let ratio = ratio.into_owned();
+                    let pos = sorted.binary_search(&ratio).unwrap_or_else(|pos| pos);
+                    sorted.insert(pos, ratio);
+
+                    let avg = sma_iter.unwrap_get_inner(index);
+
+                    let population =
+                        index.checked_sub(min_date).unwrap().unwrap_to_usize() as f32 + 1.0;
+
+                    let sd = StoredF32::from(
+                        (sorted.iter().map(|v| (**v - *avg).powi(2)).sum::<f32>() / population)
+                            .sqrt(),
+                    );
+
+                    self.sd
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, sd, exit)?;
+                    if let Some(v) = p0_5sd.as_mut() {
+                        v.forced_push_at(index, avg + StoredF32::from(0.5 * *sd), exit)?
+                    }
+                    if let Some(v) = p1sd.as_mut() {
+                        v.forced_push_at(index, avg + sd, exit)?
+                    }
+                    if let Some(v) = p1_5sd.as_mut() {
+                        v.forced_push_at(index, avg + StoredF32::from(1.5 * *sd), exit)?
+                    }
+                    if let Some(v) = p2sd.as_mut() {
+                        v.forced_push_at(index, avg + 2 * sd, exit)?
+                    }
+                    if let Some(v) = p2_5sd.as_mut() {
+                        v.forced_push_at(index, avg + StoredF32::from(2.5 * *sd), exit)?
+                    }
+                    if let Some(v) = p3sd.as_mut() {
+                        v.forced_push_at(index, avg + 3 * sd, exit)?
+                    }
+                    if let Some(v) = m0_5sd.as_mut() {
+                        v.forced_push_at(index, avg - StoredF32::from(0.5 * *sd), exit)?
+                    }
+                    if let Some(v) = m1sd.as_mut() {
+                        v.forced_push_at(index, avg - sd, exit)?
+                    }
+                    if let Some(v) = m1_5sd.as_mut() {
+                        v.forced_push_at(index, avg - StoredF32::from(1.5 * *sd), exit)?
+                    }
+                    if let Some(v) = m2sd.as_mut() {
+                        v.forced_push_at(index, avg - 2 * sd, exit)?
+                    }
+                    if let Some(v) = m2_5sd.as_mut() {
+                        v.forced_push_at(index, avg - StoredF32::from(2.5 * *sd), exit)?
+                    }
+                    if let Some(v) = m3sd.as_mut() {
+                        v.forced_push_at(index, avg - 3 * sd, exit)?
+                    }
+                }
+
+                Ok(())
+            })?;
+
+        drop(sma_iter);
+
+        self.mut_stateful_date_vecs()
+            .try_for_each(|v| v.safe_flush(exit))?;
+
+        self.mut_stateful_computed().try_for_each(|v| {
+            v.compute_rest(starting_indexes, exit, None as Option<&EagerVec<_, _>>)
+        })?;
+
+        if let Some(zscore) = self.zscore.as_mut() {
+            zscore.compute_all(starting_indexes, exit, |vec| {
+                vec.compute_zscore(
+                    starting_indexes.dateindex,
+                    source,
+                    sma,
+                    self.sd.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+        }
+
+        let Some(price) = price_opt else {
+            return Ok(());
+        };
+
+        let compute_usd =
+            |usd: &mut ComputedVecsFromDateIndex<Dollars>,
+             mut iter: BoxedVecIterator<DateIndex, StoredF32>| {
+                usd.compute_all(starting_indexes, exit, |vec| {
+                    vec.compute_transform(
+                        starting_indexes.dateindex,
+                        price,
+                        |(i, price, ..)| {
+                            let multiplier = iter.unwrap_get_inner(i);
+                            (i, price * multiplier)
+                        },
+                        exit,
+                    )?;
+                    Ok(())
+                })
+            };
+
+        if self._0sd_usd.is_none() {
+            return Ok(());
+        }
+
+        compute_usd(self._0sd_usd.as_mut().unwrap(), sma.iter())?;
+        compute_usd(
+            self.p0_5sd_usd.as_mut().unwrap(),
+            self.p0_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.p1sd_usd.as_mut().unwrap(),
+            self.p1sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.p1_5sd_usd.as_mut().unwrap(),
+            self.p1_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.p2sd_usd.as_mut().unwrap(),
+            self.p2sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.p2_5sd_usd.as_mut().unwrap(),
+            self.p2_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.p3sd_usd.as_mut().unwrap(),
+            self.p3sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m0_5sd_usd.as_mut().unwrap(),
+            self.m0_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m1sd_usd.as_mut().unwrap(),
+            self.m1sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m1_5sd_usd.as_mut().unwrap(),
+            self.m1_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m2sd_usd.as_mut().unwrap(),
+            self.m2sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m2_5sd_usd.as_mut().unwrap(),
+            self.m2_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m3sd_usd.as_mut().unwrap(),
+            self.m3sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+
+        Ok(())
+    }
+
+    fn mut_stateful_computed(
+        &mut self,
+    ) -> impl Iterator<Item = &mut ComputedVecsFromDateIndex<StoredF32>> {
+        [
+            Some(&mut self.sd),
+            self.p0_5sd.as_mut(),
+            self.p1sd.as_mut(),
+            self.p1_5sd.as_mut(),
+            self.p2sd.as_mut(),
+            self.p2_5sd.as_mut(),
+            self.p3sd.as_mut(),
+            self.m0_5sd.as_mut(),
+            self.m1sd.as_mut(),
+            self.m1_5sd.as_mut(),
+            self.m2sd.as_mut(),
+            self.m2_5sd.as_mut(),
+            self.m3sd.as_mut(),
+        ]
+        .into_iter()
+        .flatten()
+    }
+
+    fn mut_stateful_date_vecs(
+        &mut self,
+    ) -> impl Iterator<Item = &mut EagerVec<DateIndex, StoredF32>> {
+        self.mut_stateful_computed()
+            .map(|c| c.dateindex.as_mut().unwrap())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(self.sma.iter().flat_map(|v| v.iter_any_collectable()));
+
+        iter = Box::new(iter.chain(self.sd.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.p0_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.p1sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.p1_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.p2sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.p2_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.p3sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m0_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m1sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m1_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m2sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m2_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m3sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self._0sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.p0_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.p1sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.p1_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.p2sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.p2_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.p3sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.m0_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.m1sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.m1_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.m2sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.m2_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.m3sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.zscore.iter().flat_map(|v| v.iter_any_collectable())));
+
+        iter
+    }
+}

crates/brk_computer/src/grouped/source.rs

@@ -1,10 +1,10 @@
-use brk_vec::BoxedAnyIterableVec;
+use vecdb::AnyBoxedIterableVec;
 
 #[derive(Clone)]
 pub enum Source<I, T> {
     Compute,
     None,
-    Vec(BoxedAnyIterableVec<I, T>),
+    Vec(AnyBoxedIterableVec<I, T>),
 }
 
 impl<I, T> Source<I, T> {
@@ -20,7 +20,7 @@ impl<I, T> Source<I, T> {
         matches!(self, Self::Vec(_))
     }
 
-    pub fn vec(self) -> Option<BoxedAnyIterableVec<I, T>> {
+    pub fn vec(self) -> Option<AnyBoxedIterableVec<I, T>> {
         match self {
             Self::Vec(v) => Some(v),
             _ => None,
@@ -34,14 +34,14 @@ impl<I, T> From<bool> for Source<I, T> {
     }
 }
 
-impl<I, T> From<BoxedAnyIterableVec<I, T>> for Source<I, T> {
-    fn from(value: BoxedAnyIterableVec<I, T>) -> Self {
+impl<I, T> From<AnyBoxedIterableVec<I, T>> for Source<I, T> {
+    fn from(value: AnyBoxedIterableVec<I, T>) -> Self {
         Self::Vec(value)
     }
 }
 
-impl<I, T> From<Option<BoxedAnyIterableVec<I, T>>> for Source<I, T> {
-    fn from(value: Option<BoxedAnyIterableVec<I, T>>) -> Self {
+impl<I, T> From<Option<AnyBoxedIterableVec<I, T>>> for Source<I, T> {
+    fn from(value: Option<AnyBoxedIterableVec<I, T>>) -> Self {
         if let Some(v) = value {
             Self::Vec(v)
         } else {

crates/brk_computer/src/grouped/value_from_dateindex.rs

@@ -0,0 +1,140 @@
+use brk_error::Result;
+use brk_structs::{Bitcoin, DateIndex, Dollars, Sats, Version};
+use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, StoredVec};
+
+use crate::{
+    Indexes,
+    grouped::ComputedVecsFromDateIndex,
+    indexes, price,
+    traits::{ComputeFromBitcoin, ComputeFromSats},
+};
+
+use super::{Source, VecBuilderOptions};
+
+#[derive(Clone)]
+pub struct ComputedValueVecsFromDateIndex {
+    pub sats: ComputedVecsFromDateIndex<Sats>,
+    pub bitcoin: ComputedVecsFromDateIndex<Bitcoin>,
+    pub dollars: Option<ComputedVecsFromDateIndex<Dollars>>,
+}
+
+const VERSION: Version = Version::ZERO;
+
+impl ComputedValueVecsFromDateIndex {
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        source: Source<DateIndex, Sats>,
+        version: Version,
+        options: VecBuilderOptions,
+        compute_dollars: bool,
+        indexes: &indexes::Vecs,
+    ) -> Result<Self> {
+        Ok(Self {
+            sats: ComputedVecsFromDateIndex::forced_import(
+                db,
+                name,
+                source,
+                version + VERSION,
+                indexes,
+                options,
+            )?,
+            bitcoin: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &format!("{name}_btc"),
+                Source::Compute,
+                version + VERSION,
+                indexes,
+                options,
+            )?,
+            dollars: compute_dollars.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_usd"),
+                    Source::Compute,
+                    version + VERSION,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+        })
+    }
+
+    pub fn compute_all<F>(
+        &mut self,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        mut compute: F,
+    ) -> Result<()>
+    where
+        F: FnMut(&mut EagerVec<DateIndex, Sats>) -> Result<()>,
+    {
+        compute(self.sats.dateindex.as_mut().unwrap())?;
+
+        let dateindex: Option<&StoredVec<DateIndex, Sats>> = None;
+        self.compute_rest(price, starting_indexes, exit, dateindex)?;
+
+        Ok(())
+    }
+
+    pub fn compute_rest(
+        &mut self,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        dateindex: Option<&impl CollectableVec<DateIndex, Sats>>,
+    ) -> Result<()> {
+        if let Some(dateindex) = dateindex {
+            self.sats
+                .compute_rest(starting_indexes, exit, Some(dateindex))?;
+
+            self.bitcoin.compute_all(starting_indexes, exit, |v| {
+                v.compute_from_sats(starting_indexes.dateindex, dateindex, exit)
+            })?;
+        } else {
+            let dateindex: Option<&StoredVec<DateIndex, Sats>> = None;
+
+            self.sats.compute_rest(starting_indexes, exit, dateindex)?;
+
+            self.bitcoin.compute_all(starting_indexes, exit, |v| {
+                v.compute_from_sats(
+                    starting_indexes.dateindex,
+                    self.sats.dateindex.as_ref().unwrap(),
+                    exit,
+                )
+            })?;
+        }
+
+        let dateindex_to_bitcoin = self.bitcoin.dateindex.as_ref().unwrap();
+        let dateindex_to_price_close = price
+            .as_ref()
+            .unwrap()
+            .timeindexes_to_price_close
+            .dateindex
+            .as_ref()
+            .unwrap();
+
+        if let Some(dollars) = self.dollars.as_mut() {
+            dollars.compute_all(starting_indexes, exit, |v| {
+                v.compute_from_bitcoin(
+                    starting_indexes.dateindex,
+                    dateindex_to_bitcoin,
+                    dateindex_to_price_close,
+                    exit,
+                )
+            })?;
+        }
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        std::iter::empty()
+            .chain(self.sats.iter_any_collectable())
+            .chain(self.bitcoin.iter_any_collectable())
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
+    }
+}

crates/brk_computer/src/grouped/value_from_height.rs

@@ -0,0 +1,140 @@
+use allocative::Allocative;
+use brk_error::Result;
+use brk_structs::{Bitcoin, Dollars, Height, Sats, Version};
+use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, StoredVec};
+
+use crate::{
+    Indexes,
+    grouped::Source,
+    indexes, price,
+    traits::{ComputeFromBitcoin, ComputeFromSats},
+};
+
+use super::{ComputedVecsFromHeight, VecBuilderOptions};
+
+#[derive(Clone, Allocative)]
+pub struct ComputedValueVecsFromHeight {
+    pub sats: ComputedVecsFromHeight<Sats>,
+    pub bitcoin: ComputedVecsFromHeight<Bitcoin>,
+    pub dollars: Option<ComputedVecsFromHeight<Dollars>>,
+}
+
+const VERSION: Version = Version::ZERO;
+
+impl ComputedValueVecsFromHeight {
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        source: Source<Height, Sats>,
+        version: Version,
+        options: VecBuilderOptions,
+        compute_dollars: bool,
+        indexes: &indexes::Vecs,
+    ) -> Result<Self> {
+        Ok(Self {
+            sats: ComputedVecsFromHeight::forced_import(
+                db,
+                name,
+                source,
+                version + VERSION,
+                indexes,
+                options,
+            )?,
+            bitcoin: ComputedVecsFromHeight::forced_import(
+                db,
+                &format!("{name}_btc"),
+                Source::Compute,
+                version + VERSION,
+                indexes,
+                options,
+            )?,
+            dollars: compute_dollars.then(|| {
+                ComputedVecsFromHeight::forced_import(
+                    db,
+                    &format!("{name}_usd"),
+                    Source::Compute,
+                    version + VERSION,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+        })
+    }
+
+    pub fn compute_all<F>(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        mut compute: F,
+    ) -> Result<()>
+    where
+        F: FnMut(&mut EagerVec<Height, Sats>) -> Result<()>,
+    {
+        compute(self.sats.height.as_mut().unwrap())?;
+
+        let height: Option<&StoredVec<Height, Sats>> = None;
+        self.compute_rest(indexes, price, starting_indexes, exit, height)?;
+
+        Ok(())
+    }
+
+    pub fn compute_rest(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        height: Option<&impl CollectableVec<Height, Sats>>,
+    ) -> Result<()> {
+        if let Some(height) = height {
+            self.sats
+                .compute_rest(indexes, starting_indexes, exit, Some(height))?;
+
+            self.bitcoin
+                .compute_all(indexes, starting_indexes, exit, |v| {
+                    v.compute_from_sats(starting_indexes.height, height, exit)
+                })?;
+        } else {
+            let height: Option<&StoredVec<Height, Sats>> = None;
+
+            self.sats
+                .compute_rest(indexes, starting_indexes, exit, height)?;
+
+            self.bitcoin
+                .compute_all(indexes, starting_indexes, exit, |v| {
+                    v.compute_from_sats(
+                        starting_indexes.height,
+                        self.sats.height.as_ref().unwrap(),
+                        exit,
+                    )
+                })?;
+        }
+
+        let height_to_bitcoin = self.bitcoin.height.as_ref().unwrap();
+        let height_to_price_close = &price.as_ref().unwrap().chainindexes_to_price_close.height;
+
+        if let Some(dollars) = self.dollars.as_mut() {
+            dollars.compute_all(indexes, starting_indexes, exit, |v| {
+                v.compute_from_bitcoin(
+                    starting_indexes.height,
+                    height_to_bitcoin,
+                    height_to_price_close,
+                    exit,
+                )
+            })?;
+        }
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        std::iter::empty()
+            .chain(self.sats.iter_any_collectable())
+            .chain(self.bitcoin.iter_any_collectable())
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
+    }
+}

crates/brk_computer/src/grouped/value_from_txindex.rs

@@ -1,34 +1,24 @@
-use std::path::Path;
-
-use brk_core::{Bitcoin, Close, Dollars, Height, Sats, TxIndex, Version};
-use brk_exit::Exit;
+use allocative::Allocative;
+use brk_error::Result;
 use brk_indexer::Indexer;
-use brk_vec::{
-    AnyCollectableVec, CloneableAnyIterableVec, CollectableVec, Computation, ComputedVecFrom3,
-    Format, LazyVecFrom1, StoredIndex, StoredVec,
+use brk_structs::{Bitcoin, Close, Dollars, Height, Sats, TxIndex, Version};
+use vecdb::{
+    AnyCloneableIterableVec, AnyCollectableVec, CollectableVec, Database, Exit, LazyVecFrom1,
+    LazyVecFrom3, StoredIndex, StoredVec,
 };
 
-use crate::vecs::{Indexes, fetched, grouped::Source, indexes};
+use crate::{Indexes, grouped::Source, indexes, price};
 
 use super::{ComputedVecsFromTxindex, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedValueVecsFromTxindex {
     pub sats: ComputedVecsFromTxindex<Sats>,
     pub bitcoin_txindex: LazyVecFrom1<TxIndex, Bitcoin, TxIndex, Sats>,
     pub bitcoin: ComputedVecsFromTxindex<Bitcoin>,
     #[allow(clippy::type_complexity)]
     pub dollars_txindex: Option<
-        ComputedVecFrom3<
-            TxIndex,
-            Dollars,
-            TxIndex,
-            Bitcoin,
-            TxIndex,
-            Height,
-            Height,
-            Close<Dollars>,
-        >,
+        LazyVecFrom3<TxIndex, Dollars, TxIndex, Bitcoin, TxIndex, Height, Height, Close<Dollars>>,
     >,
     pub dollars: Option<ComputedVecsFromTxindex<Dollars>>,
 }
@@ -38,28 +28,24 @@ const VERSION: Version = Version::ZERO;
 impl ComputedValueVecsFromTxindex {
     #[allow(clippy::too_many_arguments)]
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         name: &str,
         indexes: &indexes::Vecs,
         source: Source<TxIndex, Sats>,
         version: Version,
-        computation: Computation,
-        format: Format,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         options: VecBuilderOptions,
-    ) -> color_eyre::Result<Self> {
-        let compute_dollars = fetched.is_some();
+    ) -> Result<Self> {
+        let compute_dollars = price.is_some();
 
-        let name_in_btc = format!("{name}_in_btc");
-        let name_in_usd = format!("{name}_in_usd");
+        let name_btc = format!("{name}_btc");
+        let name_usd = format!("{name}_usd");
 
         let sats = ComputedVecsFromTxindex::forced_import(
-            path,
+            db,
             name,
             source.clone(),
             version + VERSION,
-            format,
-            computation,
             indexes,
             options,
         )?;
@@ -67,7 +53,7 @@ impl ComputedValueVecsFromTxindex {
         let source_vec = source.vec();
 
         let bitcoin_txindex = LazyVecFrom1::init(
-            &name_in_btc,
+            &name_btc,
             version + VERSION,
             source_vec.map_or_else(|| sats.txindex.as_ref().unwrap().boxed_clone(), |s| s),
             |txindex: TxIndex, iter| {
@@ -79,30 +65,25 @@ impl ComputedValueVecsFromTxindex {
         );
 
         let bitcoin = ComputedVecsFromTxindex::forced_import(
-            path,
-            &name_in_btc,
+            db,
+            &name_btc,
             Source::None,
             version + VERSION,
-            format,
-            computation,
             indexes,
             options,
         )?;
 
-        let dollars_txindex = fetched.map(|fetched| {
-            ComputedVecFrom3::forced_import_or_init_from_3(
-                computation,
-                path,
-                &name_in_usd,
+        let dollars_txindex = price.map(|price| {
+            LazyVecFrom3::init(
+                &name_usd,
                 version + VERSION,
-                format,
                 bitcoin_txindex.boxed_clone(),
                 indexes.txindex_to_height.boxed_clone(),
-                fetched.chainindexes_to_close.height.boxed_clone(),
+                price.chainindexes_to_price_close.height.boxed_clone(),
                 |txindex: TxIndex,
                  txindex_to_btc_iter,
                  txindex_to_height_iter,
-                 height_to_close_iter| {
+                 height_to_price_close_iter| {
                     let txindex = txindex.unwrap_to_usize();
                     txindex_to_btc_iter.next_at(txindex).and_then(|(_, value)| {
                         let btc = value.into_owned();
@@ -110,14 +91,13 @@ impl ComputedValueVecsFromTxindex {
                             .next_at(txindex)
                             .and_then(|(_, value)| {
                                 let height = value.into_owned();
-                                height_to_close_iter
+                                height_to_price_close_iter
                                     .next_at(height.unwrap_to_usize())
                                     .map(|(_, close)| *close.into_owned() * btc)
                             })
                     })
                 },
             )
-            .unwrap()
         });
 
         Ok(Self {
@@ -127,12 +107,10 @@ impl ComputedValueVecsFromTxindex {
             dollars_txindex,
             dollars: compute_dollars.then(|| {
                 ComputedVecsFromTxindex::forced_import(
-                    path,
-                    &name_in_usd,
+                    db,
+                    &name_usd,
                     Source::None,
                     version + VERSION,
-                    format,
-                    computation,
                     indexes,
                     options,
                 )
@@ -145,11 +123,11 @@ impl ComputedValueVecsFromTxindex {
     //     &mut self,
     //     indexer: &Indexer,
     //     indexes: &indexes::Vecs,
-    //     fetched: Option<&marketprice::Vecs>,
+    //     price: Option<&marketprice::Vecs>,
     //     starting_indexes: &Indexes,
     //     exit: &Exit,
     //     mut compute: F,
-    // ) -> color_eyre::Result<()>
+    // ) -> Result<()>
     // where
     //     F: FnMut(
     //         &mut EagerVec<TxIndex, Sats>,
@@ -187,8 +165,8 @@ impl ComputedValueVecsFromTxindex {
         starting_indexes: &Indexes,
         exit: &Exit,
         txindex: Option<&impl CollectableVec<TxIndex, Sats>>,
-        fetched: Option<&fetched::Vecs>,
-    ) -> color_eyre::Result<()> {
+        price: Option<&price::Vecs>,
+    ) -> Result<()> {
         if let Some(txindex) = txindex {
             self.sats
                 .compute_rest(indexer, indexes, starting_indexes, exit, Some(txindex))?;
@@ -210,12 +188,6 @@ impl ComputedValueVecsFromTxindex {
         if let Some(dollars) = self.dollars.as_mut() {
             let dollars_txindex = self.dollars_txindex.as_mut().unwrap();
 
-            dollars_txindex.compute_if_necessary(
-                starting_indexes.txindex,
-                &indexer.vecs.txindex_to_txid,
-                exit,
-            )?;
-
             dollars.compute_rest_from_bitcoin(
                 indexer,
                 indexes,
@@ -223,25 +195,23 @@ impl ComputedValueVecsFromTxindex {
                 exit,
                 &self.bitcoin,
                 Some(dollars_txindex),
-                fetched.as_ref().unwrap(),
+                price.as_ref().unwrap(),
             )?;
         }
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.sats.vecs(),
-            vec![&self.bitcoin_txindex as &dyn AnyCollectableVec],
-            self.bitcoin.vecs(),
-            self.dollars_txindex
-                .as_ref()
-                .map_or(vec![], |v| vec![v as &dyn AnyCollectableVec]),
-            self.dollars.as_ref().map_or(vec![], |v| v.vecs()),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.bitcoin_txindex as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(self.sats.iter_any_collectable())
+            .chain(self.bitcoin.iter_any_collectable())
+            .chain(
+                self.dollars_txindex
+                    .iter()
+                    .map(|v| v as &dyn AnyCollectableVec),
+            )
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
     }
 }

crates/brk_computer/src/grouped/value_height.rs

@@ -1,11 +1,13 @@
-use std::path::Path;
+use brk_error::Result;
+use brk_structs::{Bitcoin, Dollars, Height, Sats, Version};
+use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, Format, StoredVec};
 
-use brk_core::{Bitcoin, Dollars, Height, Result, Sats, Version};
-use brk_exit::Exit;
-use brk_indexer::Indexer;
-use brk_vec::{AnyCollectableVec, CollectableVec, EagerVec, Format, StoredVec};
-
-use crate::vecs::{Indexes, fetched, grouped::Source, indexes};
+use crate::{
+    Indexes,
+    grouped::Source,
+    price,
+    traits::{ComputeFromBitcoin, ComputeFromSats},
+};
 
 #[derive(Clone)]
 pub struct ComputedHeightValueVecs {
@@ -18,28 +20,28 @@ const VERSION: Version = Version::ZERO;
 
 impl ComputedHeightValueVecs {
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         name: &str,
         source: Source<Height, Sats>,
         version: Version,
         format: Format,
         compute_dollars: bool,
-    ) -> color_eyre::Result<Self> {
+    ) -> Result<Self> {
         Ok(Self {
             sats: source.is_compute().then(|| {
-                EagerVec::forced_import(path, name, version + VERSION + Version::ZERO, format)
+                EagerVec::forced_import(db, name, version + VERSION + Version::ZERO, format)
                     .unwrap()
             }),
             bitcoin: EagerVec::forced_import(
-                path,
-                &format!("{name}_in_btc"),
+                db,
+                &format!("{name}_btc"),
                 version + VERSION + Version::ZERO,
                 format,
             )?,
             dollars: compute_dollars.then(|| {
                 EagerVec::forced_import(
-                    path,
-                    &format!("{name}_in_usd"),
+                    db,
+                    &format!("{name}_usd"),
                     version + VERSION + Version::ZERO,
                     format,
                 )
@@ -50,43 +52,29 @@ impl ComputedHeightValueVecs {
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
-    ) -> color_eyre::Result<()>
+    ) -> Result<()>
     where
-        F: FnMut(
-            &mut EagerVec<Height, Sats>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
+        F: FnMut(&mut EagerVec<Height, Sats>) -> Result<()>,
     {
-        compute(
-            self.sats.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
+        compute(self.sats.as_mut().unwrap())?;
 
         let height: Option<&StoredVec<Height, Sats>> = None;
-        self.compute_rest(fetched, starting_indexes, exit, height)?;
+        self.compute_rest(price, starting_indexes, exit, height)?;
 
         Ok(())
     }
 
     pub fn compute_rest(
         &mut self,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
         height: Option<&impl CollectableVec<Height, Sats>>,
-    ) -> color_eyre::Result<()> {
+    ) -> Result<()> {
         if let Some(height) = height {
             self.bitcoin
                 .compute_from_sats(starting_indexes.height, height, exit)?;
@@ -99,13 +87,13 @@ impl ComputedHeightValueVecs {
         }
 
         let height_to_bitcoin = &self.bitcoin;
-        let height_to_close = &fetched.as_ref().unwrap().chainindexes_to_close.height;
+        let height_to_price_close = &price.as_ref().unwrap().chainindexes_to_price_close.height;
 
         if let Some(dollars) = self.dollars.as_mut() {
             dollars.compute_from_bitcoin(
                 starting_indexes.height,
                 height_to_bitcoin,
-                height_to_close,
+                height_to_price_close,
                 exit,
             )?;
         }
@@ -113,14 +101,10 @@ impl ComputedHeightValueVecs {
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            vec![&self.bitcoin as &dyn AnyCollectableVec],
-            self.sats.as_ref().map_or(vec![], |v| vec![v]),
-            self.dollars.as_ref().map_or(vec![], |v| vec![v]),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.bitcoin as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(self.sats.iter().map(|v| v as &dyn AnyCollectableVec))
+            .chain(self.dollars.iter().map(|v| v as &dyn AnyCollectableVec))
     }
 }

crates/brk_computer/src/indexes.rs

@@ -1,40 +1,40 @@
 use std::{ops::Deref, path::Path};
 
-use brk_core::{
+use brk_error::Result;
+use brk_indexer::Indexer;
+use brk_structs::{
     Date, DateIndex, DecadeIndex, DifficultyEpoch, EmptyOutputIndex, HalvingEpoch, Height,
     InputIndex, MonthIndex, OpReturnIndex, OutputIndex, P2AAddressIndex, P2ABytes, P2MSOutputIndex,
     P2PK33AddressIndex, P2PK33Bytes, P2PK65AddressIndex, P2PK65Bytes, P2PKHAddressIndex,
     P2PKHBytes, P2SHAddressIndex, P2SHBytes, P2TRAddressIndex, P2TRBytes, P2WPKHAddressIndex,
-    P2WPKHBytes, P2WSHAddressIndex, P2WSHBytes, QuarterIndex, Sats, SemesterIndex, StoredUsize,
+    P2WPKHBytes, P2WSHAddressIndex, P2WSHBytes, QuarterIndex, Sats, SemesterIndex, StoredU64,
     Timestamp, TxIndex, Txid, UnknownOutputIndex, Version, WeekIndex, YearIndex,
 };
-use brk_exit::Exit;
-use brk_indexer::Indexer;
-use brk_vec::{
-    AnyCollectableVec, CloneableAnyIterableVec, Computation, ComputedVec, ComputedVecFrom1,
-    ComputedVecFrom2, EagerVec, Format, StoredIndex, VecIterator,
+use vecdb::{
+    AnyCloneableIterableVec, AnyCollectableVec, Database, EagerVec, Exit, LazyVecFrom1,
+    LazyVecFrom2, PAGE_SIZE, StoredIndex, VecIterator,
 };
 
-use crate::vecs::indexes;
-
 const VERSION: Version = Version::ZERO;
 
 #[derive(Clone)]
 pub struct Vecs {
+    db: Database,
+
     pub dateindex_to_date: EagerVec<DateIndex, Date>,
     pub dateindex_to_dateindex: EagerVec<DateIndex, DateIndex>,
     pub dateindex_to_first_height: EagerVec<DateIndex, Height>,
-    pub dateindex_to_height_count: EagerVec<DateIndex, StoredUsize>,
+    pub dateindex_to_height_count: EagerVec<DateIndex, StoredU64>,
     pub dateindex_to_monthindex: EagerVec<DateIndex, MonthIndex>,
     pub dateindex_to_weekindex: EagerVec<DateIndex, WeekIndex>,
     pub decadeindex_to_decadeindex: EagerVec<DecadeIndex, DecadeIndex>,
     pub decadeindex_to_first_yearindex: EagerVec<DecadeIndex, YearIndex>,
-    pub decadeindex_to_yearindex_count: EagerVec<DecadeIndex, StoredUsize>,
+    pub decadeindex_to_yearindex_count: EagerVec<DecadeIndex, StoredU64>,
     pub difficultyepoch_to_difficultyepoch: EagerVec<DifficultyEpoch, DifficultyEpoch>,
     pub difficultyepoch_to_first_height: EagerVec<DifficultyEpoch, Height>,
-    pub difficultyepoch_to_height_count: EagerVec<DifficultyEpoch, StoredUsize>,
+    pub difficultyepoch_to_height_count: EagerVec<DifficultyEpoch, StoredU64>,
     pub emptyoutputindex_to_emptyoutputindex:
-        ComputedVecFrom1<EmptyOutputIndex, EmptyOutputIndex, EmptyOutputIndex, TxIndex>,
+        LazyVecFrom1<EmptyOutputIndex, EmptyOutputIndex, EmptyOutputIndex, TxIndex>,
     pub halvingepoch_to_first_height: EagerVec<HalvingEpoch, Height>,
     pub halvingepoch_to_halvingepoch: EagerVec<HalvingEpoch, HalvingEpoch>,
     pub height_to_date: EagerVec<Height, Date>,
@@ -44,103 +44,88 @@ pub struct Vecs {
     pub height_to_halvingepoch: EagerVec<Height, HalvingEpoch>,
     pub height_to_height: EagerVec<Height, Height>,
     pub height_to_timestamp_fixed: EagerVec<Height, Timestamp>,
-    pub height_to_txindex_count: EagerVec<Height, StoredUsize>,
-    pub inputindex_to_inputindex: ComputedVecFrom1<InputIndex, InputIndex, InputIndex, OutputIndex>,
-    pub monthindex_to_dateindex_count: EagerVec<MonthIndex, StoredUsize>,
+    pub height_to_txindex_count: EagerVec<Height, StoredU64>,
+    pub inputindex_to_inputindex: LazyVecFrom1<InputIndex, InputIndex, InputIndex, OutputIndex>,
+    pub monthindex_to_dateindex_count: EagerVec<MonthIndex, StoredU64>,
     pub monthindex_to_first_dateindex: EagerVec<MonthIndex, DateIndex>,
     pub monthindex_to_monthindex: EagerVec<MonthIndex, MonthIndex>,
     pub monthindex_to_quarterindex: EagerVec<MonthIndex, QuarterIndex>,
     pub monthindex_to_semesterindex: EagerVec<MonthIndex, SemesterIndex>,
     pub monthindex_to_yearindex: EagerVec<MonthIndex, YearIndex>,
     pub opreturnindex_to_opreturnindex:
-        ComputedVecFrom1<OpReturnIndex, OpReturnIndex, OpReturnIndex, TxIndex>,
-    pub outputindex_to_outputindex: ComputedVecFrom1<OutputIndex, OutputIndex, OutputIndex, Sats>,
+        LazyVecFrom1<OpReturnIndex, OpReturnIndex, OpReturnIndex, TxIndex>,
+    pub outputindex_to_outputindex: LazyVecFrom1<OutputIndex, OutputIndex, OutputIndex, Sats>,
     pub outputindex_to_txindex: EagerVec<OutputIndex, TxIndex>,
     pub p2aaddressindex_to_p2aaddressindex:
-        ComputedVecFrom1<P2AAddressIndex, P2AAddressIndex, P2AAddressIndex, P2ABytes>,
+        LazyVecFrom1<P2AAddressIndex, P2AAddressIndex, P2AAddressIndex, P2ABytes>,
     pub p2msoutputindex_to_p2msoutputindex:
-        ComputedVecFrom1<P2MSOutputIndex, P2MSOutputIndex, P2MSOutputIndex, TxIndex>,
+        LazyVecFrom1<P2MSOutputIndex, P2MSOutputIndex, P2MSOutputIndex, TxIndex>,
     pub p2pk33addressindex_to_p2pk33addressindex:
-        ComputedVecFrom1<P2PK33AddressIndex, P2PK33AddressIndex, P2PK33AddressIndex, P2PK33Bytes>,
+        LazyVecFrom1<P2PK33AddressIndex, P2PK33AddressIndex, P2PK33AddressIndex, P2PK33Bytes>,
     pub p2pk65addressindex_to_p2pk65addressindex:
-        ComputedVecFrom1<P2PK65AddressIndex, P2PK65AddressIndex, P2PK65AddressIndex, P2PK65Bytes>,
+        LazyVecFrom1<P2PK65AddressIndex, P2PK65AddressIndex, P2PK65AddressIndex, P2PK65Bytes>,
     pub p2pkhaddressindex_to_p2pkhaddressindex:
-        ComputedVecFrom1<P2PKHAddressIndex, P2PKHAddressIndex, P2PKHAddressIndex, P2PKHBytes>,
+        LazyVecFrom1<P2PKHAddressIndex, P2PKHAddressIndex, P2PKHAddressIndex, P2PKHBytes>,
     pub p2shaddressindex_to_p2shaddressindex:
-        ComputedVecFrom1<P2SHAddressIndex, P2SHAddressIndex, P2SHAddressIndex, P2SHBytes>,
+        LazyVecFrom1<P2SHAddressIndex, P2SHAddressIndex, P2SHAddressIndex, P2SHBytes>,
     pub p2traddressindex_to_p2traddressindex:
-        ComputedVecFrom1<P2TRAddressIndex, P2TRAddressIndex, P2TRAddressIndex, P2TRBytes>,
+        LazyVecFrom1<P2TRAddressIndex, P2TRAddressIndex, P2TRAddressIndex, P2TRBytes>,
     pub p2wpkhaddressindex_to_p2wpkhaddressindex:
-        ComputedVecFrom1<P2WPKHAddressIndex, P2WPKHAddressIndex, P2WPKHAddressIndex, P2WPKHBytes>,
+        LazyVecFrom1<P2WPKHAddressIndex, P2WPKHAddressIndex, P2WPKHAddressIndex, P2WPKHBytes>,
     pub p2wshaddressindex_to_p2wshaddressindex:
-        ComputedVecFrom1<P2WSHAddressIndex, P2WSHAddressIndex, P2WSHAddressIndex, P2WSHBytes>,
+        LazyVecFrom1<P2WSHAddressIndex, P2WSHAddressIndex, P2WSHAddressIndex, P2WSHBytes>,
     pub quarterindex_to_first_monthindex: EagerVec<QuarterIndex, MonthIndex>,
-    pub quarterindex_to_monthindex_count: EagerVec<QuarterIndex, StoredUsize>,
+    pub quarterindex_to_monthindex_count: EagerVec<QuarterIndex, StoredU64>,
     pub quarterindex_to_quarterindex: EagerVec<QuarterIndex, QuarterIndex>,
     pub semesterindex_to_first_monthindex: EagerVec<SemesterIndex, MonthIndex>,
-    pub semesterindex_to_monthindex_count: EagerVec<SemesterIndex, StoredUsize>,
+    pub semesterindex_to_monthindex_count: EagerVec<SemesterIndex, StoredU64>,
     pub semesterindex_to_semesterindex: EagerVec<SemesterIndex, SemesterIndex>,
     pub txindex_to_height: EagerVec<TxIndex, Height>,
     pub txindex_to_input_count:
-        ComputedVecFrom2<TxIndex, StoredUsize, TxIndex, InputIndex, InputIndex, OutputIndex>,
+        LazyVecFrom2<TxIndex, StoredU64, TxIndex, InputIndex, InputIndex, OutputIndex>,
     pub txindex_to_output_count:
-        ComputedVecFrom2<TxIndex, StoredUsize, TxIndex, OutputIndex, OutputIndex, Sats>,
-    pub txindex_to_txindex: ComputedVecFrom1<TxIndex, TxIndex, TxIndex, Txid>,
+        LazyVecFrom2<TxIndex, StoredU64, TxIndex, OutputIndex, OutputIndex, Sats>,
+    pub txindex_to_txindex: LazyVecFrom1<TxIndex, TxIndex, TxIndex, Txid>,
     pub unknownoutputindex_to_unknownoutputindex:
-        ComputedVecFrom1<UnknownOutputIndex, UnknownOutputIndex, UnknownOutputIndex, TxIndex>,
-    pub weekindex_to_dateindex_count: EagerVec<WeekIndex, StoredUsize>,
+        LazyVecFrom1<UnknownOutputIndex, UnknownOutputIndex, UnknownOutputIndex, TxIndex>,
+    pub weekindex_to_dateindex_count: EagerVec<WeekIndex, StoredU64>,
     pub weekindex_to_first_dateindex: EagerVec<WeekIndex, DateIndex>,
     pub weekindex_to_weekindex: EagerVec<WeekIndex, WeekIndex>,
     pub yearindex_to_decadeindex: EagerVec<YearIndex, DecadeIndex>,
     pub yearindex_to_first_monthindex: EagerVec<YearIndex, MonthIndex>,
-    pub yearindex_to_monthindex_count: EagerVec<YearIndex, StoredUsize>,
+    pub yearindex_to_monthindex_count: EagerVec<YearIndex, StoredU64>,
     pub yearindex_to_yearindex: EagerVec<YearIndex, YearIndex>,
 }
 
 impl Vecs {
-    pub fn forced_import(
-        path: &Path,
-        version: Version,
-        indexer: &Indexer,
-        computation: Computation,
-        format: Format,
-    ) -> color_eyre::Result<Self> {
-        let outputindex_to_outputindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+    pub fn forced_import(parent: &Path, version: Version, indexer: &Indexer) -> Result<Self> {
+        let db = Database::open(&parent.join("indexes"))?;
+        db.set_min_len(PAGE_SIZE * 10_000_000)?;
+
+        let outputindex_to_outputindex = LazyVecFrom1::init(
             "outputindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.outputindex_to_value.boxed_clone(),
             |index, _| Some(index),
-        )?;
+        );
 
-        let inputindex_to_inputindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        let inputindex_to_inputindex = LazyVecFrom1::init(
             "inputindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.inputindex_to_outputindex.boxed_clone(),
             |index, _| Some(index),
-        )?;
+        );
 
-        let txindex_to_txindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        let txindex_to_txindex = LazyVecFrom1::init(
             "txindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.txindex_to_txid.boxed_clone(),
             |index, _| Some(index),
-        )?;
+        );
 
-        let txindex_to_input_count = ComputedVec::forced_import_or_init_from_2(
-            computation,
-            path,
+        let txindex_to_input_count = LazyVecFrom2::init(
             "input_count",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.txindex_to_first_inputindex.boxed_clone(),
             indexer.vecs.inputindex_to_outputindex.boxed_clone(),
             |index: TxIndex, txindex_to_first_inputindex_iter, inputindex_to_outputindex_iter| {
@@ -153,17 +138,14 @@ impl Vecs {
                             .next_at(txindex + 1)
                             .map(|(_, v)| usize::from(v.into_owned()))
                             .unwrap_or_else(|| inputindex_to_outputindex_iter.len());
-                        StoredUsize::from((start..end).count())
+                        StoredU64::from((start..end).count())
                     })
             },
-        )?;
+        );
 
-        let txindex_to_output_count = ComputedVec::forced_import_or_init_from_2(
-            computation,
-            path,
+        let txindex_to_output_count = LazyVecFrom2::init(
             "output_count",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.txindex_to_first_outputindex.boxed_clone(),
             indexer.vecs.outputindex_to_value.boxed_clone(),
             |index: TxIndex, txindex_to_first_outputindex_iter, outputindex_to_value_iter| {
@@ -176,121 +158,85 @@ impl Vecs {
                             .next_at(txindex + 1)
                             .map(|(_, v)| usize::from(v.into_owned()))
                             .unwrap_or_else(|| outputindex_to_value_iter.len());
-                        StoredUsize::from((start..end).count())
+                        StoredU64::from((start..end).count())
                     })
             },
-        )?;
+        );
 
-        let p2pk33addressindex_to_p2pk33addressindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        let p2pk33addressindex_to_p2pk33addressindex = LazyVecFrom1::init(
             "p2pk33addressindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.p2pk33addressindex_to_p2pk33bytes.boxed_clone(),
             |index, _| Some(index),
-        )?;
-        let p2pk65addressindex_to_p2pk65addressindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        );
+        let p2pk65addressindex_to_p2pk65addressindex = LazyVecFrom1::init(
             "p2pk65addressindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.p2pk65addressindex_to_p2pk65bytes.boxed_clone(),
             |index, _| Some(index),
-        )?;
-        let p2pkhaddressindex_to_p2pkhaddressindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        );
+        let p2pkhaddressindex_to_p2pkhaddressindex = LazyVecFrom1::init(
             "p2pkhaddressindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.p2pkhaddressindex_to_p2pkhbytes.boxed_clone(),
             |index, _| Some(index),
-        )?;
-        let p2shaddressindex_to_p2shaddressindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        );
+        let p2shaddressindex_to_p2shaddressindex = LazyVecFrom1::init(
             "p2shaddressindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.p2shaddressindex_to_p2shbytes.boxed_clone(),
             |index, _| Some(index),
-        )?;
-        let p2traddressindex_to_p2traddressindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        );
+        let p2traddressindex_to_p2traddressindex = LazyVecFrom1::init(
             "p2traddressindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.p2traddressindex_to_p2trbytes.boxed_clone(),
             |index, _| Some(index),
-        )?;
-        let p2wpkhaddressindex_to_p2wpkhaddressindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        );
+        let p2wpkhaddressindex_to_p2wpkhaddressindex = LazyVecFrom1::init(
             "p2wpkhaddressindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.p2wpkhaddressindex_to_p2wpkhbytes.boxed_clone(),
             |index, _| Some(index),
-        )?;
-        let p2wshaddressindex_to_p2wshaddressindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        );
+        let p2wshaddressindex_to_p2wshaddressindex = LazyVecFrom1::init(
             "p2wshaddressindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.p2wshaddressindex_to_p2wshbytes.boxed_clone(),
             |index, _| Some(index),
-        )?;
-        let p2aaddressindex_to_p2aaddressindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        );
+        let p2aaddressindex_to_p2aaddressindex = LazyVecFrom1::init(
             "p2aaddressindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.p2aaddressindex_to_p2abytes.boxed_clone(),
             |index, _| Some(index),
-        )?;
-        let p2msoutputindex_to_p2msoutputindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        );
+        let p2msoutputindex_to_p2msoutputindex = LazyVecFrom1::init(
             "p2msoutputindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.p2msoutputindex_to_txindex.boxed_clone(),
             |index, _| Some(index),
-        )?;
-        let emptyoutputindex_to_emptyoutputindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        );
+        let emptyoutputindex_to_emptyoutputindex = LazyVecFrom1::init(
             "emptyoutputindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.emptyoutputindex_to_txindex.boxed_clone(),
             |index, _| Some(index),
-        )?;
-        let unknownoutputindex_to_unknownoutputindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        );
+        let unknownoutputindex_to_unknownoutputindex = LazyVecFrom1::init(
             "unknownoutputindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.unknownoutputindex_to_txindex.boxed_clone(),
             |index, _| Some(index),
-        )?;
-        let opreturnindex_to_opreturnindex = ComputedVec::forced_import_or_init_from_1(
-            computation,
-            path,
+        );
+        let opreturnindex_to_opreturnindex = LazyVecFrom1::init(
             "opreturnindex",
             version + VERSION + Version::ZERO,
-            format,
             indexer.vecs.opreturnindex_to_txindex.boxed_clone(),
             |index, _| Some(index),
-        )?;
+        );
 
-        Ok(Self {
+        let this = Self {
             emptyoutputindex_to_emptyoutputindex,
             inputindex_to_inputindex,
             opreturnindex_to_opreturnindex,
@@ -309,265 +255,232 @@ impl Vecs {
             txindex_to_txindex,
             unknownoutputindex_to_unknownoutputindex,
 
-            dateindex_to_date: EagerVec::forced_import(
-                path,
+            dateindex_to_date: EagerVec::forced_import_compressed(
+                &db,
                 "date",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            dateindex_to_dateindex: EagerVec::forced_import(
-                path,
+            dateindex_to_dateindex: EagerVec::forced_import_compressed(
+                &db,
                 "dateindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            dateindex_to_first_height: EagerVec::forced_import(
-                path,
+            dateindex_to_first_height: EagerVec::forced_import_compressed(
+                &db,
                 "first_height",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            dateindex_to_monthindex: EagerVec::forced_import(
-                path,
+            dateindex_to_monthindex: EagerVec::forced_import_compressed(
+                &db,
                 "monthindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            dateindex_to_weekindex: EagerVec::forced_import(
-                path,
+            dateindex_to_weekindex: EagerVec::forced_import_compressed(
+                &db,
                 "weekindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            decadeindex_to_decadeindex: EagerVec::forced_import(
-                path,
+            decadeindex_to_decadeindex: EagerVec::forced_import_compressed(
+                &db,
                 "decadeindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            decadeindex_to_first_yearindex: EagerVec::forced_import(
-                path,
+            decadeindex_to_first_yearindex: EagerVec::forced_import_compressed(
+                &db,
                 "first_yearindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            difficultyepoch_to_difficultyepoch: EagerVec::forced_import(
-                path,
+            difficultyepoch_to_difficultyepoch: EagerVec::forced_import_compressed(
+                &db,
                 "difficultyepoch",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            difficultyepoch_to_first_height: EagerVec::forced_import(
-                path,
+            difficultyepoch_to_first_height: EagerVec::forced_import_compressed(
+                &db,
                 "first_height",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            halvingepoch_to_first_height: EagerVec::forced_import(
-                path,
+            halvingepoch_to_first_height: EagerVec::forced_import_compressed(
+                &db,
                 "first_height",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            halvingepoch_to_halvingepoch: EagerVec::forced_import(
-                path,
+            halvingepoch_to_halvingepoch: EagerVec::forced_import_compressed(
+                &db,
                 "halvingepoch",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            height_to_date: EagerVec::forced_import(
-                path,
+            height_to_date: EagerVec::forced_import_compressed(
+                &db,
                 "date",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            height_to_difficultyepoch: EagerVec::forced_import(
-                path,
+            height_to_difficultyepoch: EagerVec::forced_import_compressed(
+                &db,
                 "difficultyepoch",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            height_to_halvingepoch: EagerVec::forced_import(
-                path,
+            height_to_halvingepoch: EagerVec::forced_import_compressed(
+                &db,
                 "halvingepoch",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            height_to_height: EagerVec::forced_import(
-                path,
+            height_to_height: EagerVec::forced_import_compressed(
+                &db,
                 "height",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            monthindex_to_first_dateindex: EagerVec::forced_import(
-                path,
+            monthindex_to_first_dateindex: EagerVec::forced_import_compressed(
+                &db,
                 "first_dateindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            monthindex_to_monthindex: EagerVec::forced_import(
-                path,
+            monthindex_to_monthindex: EagerVec::forced_import_compressed(
+                &db,
                 "monthindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            monthindex_to_quarterindex: EagerVec::forced_import(
-                path,
+            monthindex_to_quarterindex: EagerVec::forced_import_compressed(
+                &db,
                 "quarterindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            monthindex_to_semesterindex: EagerVec::forced_import(
-                path,
+            monthindex_to_semesterindex: EagerVec::forced_import_compressed(
+                &db,
                 "semesterindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            monthindex_to_yearindex: EagerVec::forced_import(
-                path,
+            monthindex_to_yearindex: EagerVec::forced_import_compressed(
+                &db,
                 "yearindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            quarterindex_to_first_monthindex: EagerVec::forced_import(
-                path,
+            quarterindex_to_first_monthindex: EagerVec::forced_import_compressed(
+                &db,
                 "first_monthindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            semesterindex_to_first_monthindex: EagerVec::forced_import(
-                path,
+            semesterindex_to_first_monthindex: EagerVec::forced_import_compressed(
+                &db,
                 "first_monthindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            weekindex_to_first_dateindex: EagerVec::forced_import(
-                path,
+            weekindex_to_first_dateindex: EagerVec::forced_import_compressed(
+                &db,
                 "first_dateindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            yearindex_to_first_monthindex: EagerVec::forced_import(
-                path,
+            yearindex_to_first_monthindex: EagerVec::forced_import_compressed(
+                &db,
                 "first_monthindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            quarterindex_to_quarterindex: EagerVec::forced_import(
-                path,
+            quarterindex_to_quarterindex: EagerVec::forced_import_compressed(
+                &db,
                 "quarterindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            semesterindex_to_semesterindex: EagerVec::forced_import(
-                path,
+            semesterindex_to_semesterindex: EagerVec::forced_import_compressed(
+                &db,
                 "semesterindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            weekindex_to_weekindex: EagerVec::forced_import(
-                path,
+            weekindex_to_weekindex: EagerVec::forced_import_compressed(
+                &db,
                 "weekindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            yearindex_to_decadeindex: EagerVec::forced_import(
-                path,
+            yearindex_to_decadeindex: EagerVec::forced_import_compressed(
+                &db,
                 "decadeindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            yearindex_to_yearindex: EagerVec::forced_import(
-                path,
+            yearindex_to_yearindex: EagerVec::forced_import_compressed(
+                &db,
                 "yearindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            height_to_date_fixed: EagerVec::forced_import(
-                path,
+            height_to_date_fixed: EagerVec::forced_import_compressed(
+                &db,
                 "date_fixed",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            height_to_dateindex: EagerVec::forced_import(
-                path,
+            height_to_dateindex: EagerVec::forced_import_compressed(
+                &db,
                 "dateindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            txindex_to_height: EagerVec::forced_import(
-                path,
+            txindex_to_height: EagerVec::forced_import_compressed(
+                &db,
                 "height",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            height_to_timestamp_fixed: EagerVec::forced_import(
-                path,
+            height_to_timestamp_fixed: EagerVec::forced_import_compressed(
+                &db,
                 "timestamp_fixed",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            height_to_txindex_count: EagerVec::forced_import(
-                path,
+            height_to_txindex_count: EagerVec::forced_import_compressed(
+                &db,
                 "txindex_count",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            dateindex_to_height_count: EagerVec::forced_import(
-                path,
+            dateindex_to_height_count: EagerVec::forced_import_compressed(
+                &db,
                 "height_count",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            weekindex_to_dateindex_count: EagerVec::forced_import(
-                path,
+            weekindex_to_dateindex_count: EagerVec::forced_import_compressed(
+                &db,
                 "dateindex_count",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            difficultyepoch_to_height_count: EagerVec::forced_import(
-                path,
+            difficultyepoch_to_height_count: EagerVec::forced_import_compressed(
+                &db,
                 "height_count",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            monthindex_to_dateindex_count: EagerVec::forced_import(
-                path,
+            monthindex_to_dateindex_count: EagerVec::forced_import_compressed(
+                &db,
                 "dateindex_count",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            quarterindex_to_monthindex_count: EagerVec::forced_import(
-                path,
+            quarterindex_to_monthindex_count: EagerVec::forced_import_compressed(
+                &db,
                 "monthindex_count",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            semesterindex_to_monthindex_count: EagerVec::forced_import(
-                path,
+            semesterindex_to_monthindex_count: EagerVec::forced_import_compressed(
+                &db,
                 "monthindex_count",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            yearindex_to_monthindex_count: EagerVec::forced_import(
-                path,
+            yearindex_to_monthindex_count: EagerVec::forced_import_compressed(
+                &db,
                 "monthindex_count",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            decadeindex_to_yearindex_count: EagerVec::forced_import(
-                path,
+            decadeindex_to_yearindex_count: EagerVec::forced_import_compressed(
+                &db,
                 "yearindex_count",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-            outputindex_to_txindex: EagerVec::forced_import(
-                path,
+            outputindex_to_txindex: EagerVec::forced_import_compressed(
+                &db,
                 "txindex",
                 version + VERSION + Version::ZERO,
-                format,
             )?,
-        })
+
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
     }
 
     pub fn compute(
@@ -575,29 +488,22 @@ impl Vecs {
         indexer: &Indexer,
         starting_indexes: brk_indexer::Indexes,
         exit: &Exit,
-    ) -> color_eyre::Result<Indexes> {
+    ) -> Result<Indexes> {
+        let idxs = self.compute_(indexer, starting_indexes, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(idxs)
+    }
+
+    fn compute_(
+        &mut self,
+        indexer: &Indexer,
+        starting_indexes: brk_indexer::Indexes,
+        exit: &Exit,
+    ) -> Result<Indexes> {
         // ---
         // OutputIndex
         // ---
 
-        self.outputindex_to_outputindex.compute_if_necessary(
-            starting_indexes.outputindex,
-            &indexer.vecs.outputindex_to_value,
-            exit,
-        )?;
-
-        self.txindex_to_output_count.compute_if_necessary(
-            starting_indexes.txindex,
-            &indexer.vecs.txindex_to_txid,
-            exit,
-        )?;
-
-        self.txindex_to_input_count.compute_if_necessary(
-            starting_indexes.txindex,
-            &indexer.vecs.txindex_to_txid,
-            exit,
-        )?;
-
         self.outputindex_to_txindex.compute_inverse_less_to_more(
             starting_indexes.txindex,
             &indexer.vecs.txindex_to_first_outputindex,
@@ -605,109 +511,10 @@ impl Vecs {
             exit,
         )?;
 
-        self.p2pk33addressindex_to_p2pk33addressindex
-            .compute_if_necessary(
-                starting_indexes.p2pk33addressindex,
-                &indexer.vecs.p2pk33addressindex_to_p2pk33bytes,
-                exit,
-            )?;
-
-        self.p2pk65addressindex_to_p2pk65addressindex
-            .compute_if_necessary(
-                starting_indexes.p2pk65addressindex,
-                &indexer.vecs.p2pk65addressindex_to_p2pk65bytes,
-                exit,
-            )?;
-
-        self.p2pkhaddressindex_to_p2pkhaddressindex
-            .compute_if_necessary(
-                starting_indexes.p2pkhaddressindex,
-                &indexer.vecs.p2pkhaddressindex_to_p2pkhbytes,
-                exit,
-            )?;
-
-        self.p2shaddressindex_to_p2shaddressindex
-            .compute_if_necessary(
-                starting_indexes.p2shaddressindex,
-                &indexer.vecs.p2shaddressindex_to_p2shbytes,
-                exit,
-            )?;
-
-        self.p2traddressindex_to_p2traddressindex
-            .compute_if_necessary(
-                starting_indexes.p2traddressindex,
-                &indexer.vecs.p2traddressindex_to_p2trbytes,
-                exit,
-            )?;
-
-        self.p2wpkhaddressindex_to_p2wpkhaddressindex
-            .compute_if_necessary(
-                starting_indexes.p2wpkhaddressindex,
-                &indexer.vecs.p2wpkhaddressindex_to_p2wpkhbytes,
-                exit,
-            )?;
-
-        self.p2wshaddressindex_to_p2wshaddressindex
-            .compute_if_necessary(
-                starting_indexes.p2wshaddressindex,
-                &indexer.vecs.p2wshaddressindex_to_p2wshbytes,
-                exit,
-            )?;
-
-        self.emptyoutputindex_to_emptyoutputindex
-            .compute_if_necessary(
-                starting_indexes.emptyoutputindex,
-                &indexer.vecs.emptyoutputindex_to_txindex,
-                exit,
-            )?;
-
-        self.p2msoutputindex_to_p2msoutputindex
-            .compute_if_necessary(
-                starting_indexes.p2msoutputindex,
-                &indexer.vecs.p2msoutputindex_to_txindex,
-                exit,
-            )?;
-
-        self.opreturnindex_to_opreturnindex.compute_if_necessary(
-            starting_indexes.opreturnindex,
-            &indexer.vecs.opreturnindex_to_txindex,
-            exit,
-        )?;
-
-        self.p2aaddressindex_to_p2aaddressindex
-            .compute_if_necessary(
-                starting_indexes.p2aaddressindex,
-                &indexer.vecs.p2aaddressindex_to_p2abytes,
-                exit,
-            )?;
-
-        self.unknownoutputindex_to_unknownoutputindex
-            .compute_if_necessary(
-                starting_indexes.unknownoutputindex,
-                &indexer.vecs.unknownoutputindex_to_txindex,
-                exit,
-            )?;
-
-        // ---
-        // InputIndex
-        // ---
-
-        self.inputindex_to_inputindex.compute_if_necessary(
-            starting_indexes.inputindex,
-            &indexer.vecs.inputindex_to_outputindex,
-            exit,
-        )?;
-
         // ---
         // TxIndex
         // ---
 
-        self.txindex_to_txindex.compute_if_necessary(
-            starting_indexes.txindex,
-            &indexer.vecs.txindex_to_txid,
-            exit,
-        )?;
-
         self.height_to_txindex_count.compute_count_from_indexes(
             starting_indexes.height,
             &indexer.vecs.height_to_first_txindex,
@@ -744,14 +551,14 @@ impl Vecs {
             starting_indexes.height,
             &indexer.vecs.height_to_timestamp,
             |(h, timestamp, height_to_timestamp_fixed_iter)| {
-                if prev_timestamp_fixed.is_none() {
-                    if let Some(prev_h) = h.decremented() {
-                        prev_timestamp_fixed.replace(
-                            height_to_timestamp_fixed_iter
-                                .into_iter()
-                                .unwrap_get_inner(prev_h),
-                        );
-                    }
+                if prev_timestamp_fixed.is_none()
+                    && let Some(prev_h) = h.decremented()
+                {
+                    prev_timestamp_fixed.replace(
+                        height_to_timestamp_fixed_iter
+                            .into_iter()
+                            .unwrap_get_inner(prev_h),
+                    );
                 }
                 let timestamp_fixed =
                     prev_timestamp_fixed.map_or(timestamp, |prev_d| prev_d.max(timestamp));
@@ -1126,9 +933,9 @@ impl Vecs {
         })
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        vec![
-            &self.dateindex_to_date,
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [
+            &self.dateindex_to_date as &dyn AnyCollectableVec,
             &self.dateindex_to_dateindex,
             &self.dateindex_to_first_height,
             &self.dateindex_to_height_count,
@@ -1189,6 +996,7 @@ impl Vecs {
             &self.yearindex_to_yearindex,
             &self.outputindex_to_txindex,
         ]
+        .into_iter()
     }
 }
 
@@ -1206,7 +1014,7 @@ pub struct Indexes {
 }
 
 impl Indexes {
-    pub fn update_from_height(&mut self, height: Height, indexes: &indexes::Vecs) {
+    pub fn update_from_height(&mut self, height: Height, indexes: &Vecs) {
         self.indexes.height = height;
         self.dateindex = DateIndex::try_from(
             indexes

crates/brk_computer/src/lib.rs

@@ -1,79 +1,262 @@
 #![doc = include_str!("../README.md")]
-#![doc = "\n## Example\n\n```rust"]
-#![doc = include_str!("../examples/main.rs")]
-#![doc = "```"]
 
 use std::path::Path;
 
-use brk_core::Version;
-use brk_exit::Exit;
+use brk_error::Result;
 use brk_fetcher::Fetcher;
 use brk_indexer::Indexer;
-use brk_vec::{Computation, Format};
+use brk_parser::Parser;
+use brk_structs::Version;
 use log::info;
+use vecdb::{AnyCollectableVec, Exit, Format};
 
+mod blks;
+mod chain;
+mod cointime;
+mod constants;
+mod fetched;
+mod grouped;
+mod indexes;
+mod market;
+mod pools;
+mod price;
+mod stateful;
 mod states;
-mod stores;
+mod traits;
 mod utils;
-mod vecs;
 
+use indexes::Indexes;
+
+pub use pools::*;
+pub use states::PriceToAmount;
 use states::*;
-use stores::Stores;
-use vecs::Vecs;
 
 #[derive(Clone)]
 pub struct Computer {
-    fetcher: Option<Fetcher>,
-    pub vecs: Vecs,
-    pub stores: Stores,
+    pub chain: chain::Vecs,
+    pub cointime: cointime::Vecs,
+    pub constants: constants::Vecs,
+    pub fetched: Option<fetched::Vecs>,
+    pub indexes: indexes::Vecs,
+    pub market: market::Vecs,
+    pub pools: pools::Vecs,
+    pub blks: blks::Vecs,
+    pub price: Option<price::Vecs>,
+    pub stateful: stateful::Vecs,
 }
 
-const VERSION: Version = Version::ONE;
+const VERSION: Version = Version::new(4);
 
 impl Computer {
     /// Do NOT import multiple times or things will break !!!
     pub fn forced_import(
-        outputs_dir: &Path,
+        outputs_path: &Path,
         indexer: &Indexer,
-        computation: Computation,
         fetcher: Option<Fetcher>,
-        format: Format,
-    ) -> color_eyre::Result<Self> {
+    ) -> Result<Self> {
+        info!("Importing computer...");
+
+        let computed_path = outputs_path.join("computed");
+
+        let indexes =
+            indexes::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, indexer)?;
+
+        let fetched = fetcher.map(|fetcher| {
+            fetched::Vecs::forced_import(outputs_path, fetcher, VERSION + Version::ZERO).unwrap()
+        });
+
+        let price = fetched.is_some().then(|| {
+            price::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes).unwrap()
+        });
+
         Ok(Self {
-            vecs: Vecs::import(
-                // TODO: Give self.path, join inside import
-                &outputs_dir.join("vecs/computed"),
+            constants: constants::Vecs::forced_import(
+                &computed_path,
+                VERSION + Version::ZERO,
+                &indexes,
+            )?,
+            market: market::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes)?,
+            stateful: stateful::Vecs::forced_import(
+                &computed_path,
+                VERSION + Version::ZERO,
+                Format::Compressed,
+                &indexes,
+                price.as_ref(),
+            )?,
+            chain: chain::Vecs::forced_import(
+                &computed_path,
                 VERSION + Version::ZERO,
                 indexer,
-                fetcher.is_some(),
-                computation,
-                format,
+                &indexes,
+                price.as_ref(),
             )?,
-            stores: Stores::import(
-                // TODO: Give self.path, join inside import
-                &outputs_dir.join("stores"),
+            blks: blks::Vecs::forced_import(&computed_path, VERSION + Version::ZERO)?,
+            pools: pools::Vecs::forced_import(
+                &computed_path,
                 VERSION + Version::ZERO,
-                &indexer.stores.keyspace,
+                &indexes,
+                price.as_ref(),
             )?,
-            fetcher,
+            cointime: cointime::Vecs::forced_import(
+                &computed_path,
+                VERSION + Version::ZERO,
+                &indexes,
+                price.as_ref(),
+            )?,
+            indexes,
+            fetched,
+            price,
         })
     }
+
+    pub fn compute(
+        &mut self,
+        indexer: &Indexer,
+        starting_indexes: brk_indexer::Indexes,
+        parser: &Parser,
+        exit: &Exit,
+    ) -> Result<()> {
+        info!("Computing indexes...");
+        let mut starting_indexes = self.indexes.compute(indexer, starting_indexes, exit)?;
+
+        if let Some(fetched) = self.fetched.as_mut() {
+            info!("Computing fetched...");
+            fetched.compute(indexer, &self.indexes, &starting_indexes, exit)?;
+
+            info!("Computing prices...");
+            self.price.as_mut().unwrap().compute(
+                &self.indexes,
+                &starting_indexes,
+                fetched,
+                exit,
+            )?;
+        }
+
+        info!("Computing BLKs metadata...");
+        self.blks
+            .compute(indexer, &self.indexes, &starting_indexes, parser, exit)?;
+
+        std::thread::scope(|scope| -> Result<()> {
+            let constants = scope.spawn(|| -> Result<()> {
+                info!("Computing constants...");
+                self.constants
+                    .compute(&self.indexes, &starting_indexes, exit)?;
+                Ok(())
+            });
+
+            // let blks = scope.spawn(|| -> Result<()> {
+            //     info!("Computing blks...");
+            //     self.blks
+            //         .compute(indexer, &self.indexes, &starting_indexes, parser, exit)?;
+            //     Ok(())
+            // });
+
+            let chain = scope.spawn(|| -> Result<()> {
+                info!("Computing chain...");
+                self.chain.compute(
+                    indexer,
+                    &self.indexes,
+                    &starting_indexes,
+                    self.price.as_ref(),
+                    exit,
+                )?;
+                Ok(())
+            });
+
+            if let Some(price) = self.price.as_ref() {
+                info!("Computing market...");
+                self.market.compute(price, &starting_indexes, exit)?;
+            }
+
+            constants.join().unwrap()?;
+            // blks.join().unwrap()?;
+            chain.join().unwrap()?;
+            Ok(())
+        })?;
+
+        self.pools.compute(
+            indexer,
+            &self.indexes,
+            &starting_indexes,
+            &self.chain,
+            self.price.as_ref(),
+            exit,
+        )?;
+
+        info!("Computing stateful...");
+        self.stateful.compute(
+            indexer,
+            &self.indexes,
+            &self.chain,
+            self.price.as_ref(),
+            &mut starting_indexes,
+            exit,
+        )?;
+
+        info!("Computing cointime...");
+        self.cointime.compute(
+            &self.indexes,
+            &starting_indexes,
+            self.price.as_ref(),
+            &self.chain,
+            &self.stateful,
+            exit,
+        )?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(self.fetched.iter().flat_map(|v| v.iter_any_collectable()));
+
+        iter = Box::new(iter.chain(self.chain.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.cointime.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constants.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.market.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.pools.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.blks.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.price.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.stateful.iter_any_collectable()));
+
+        iter
+    }
+
+    pub fn static_clone(&self) -> &'static Self {
+        Box::leak(Box::new(self.clone()))
+    }
 }
 
-impl Computer {
-    pub fn compute(
-        &mut self,
-        indexer: &mut Indexer,
-        starting_indexes: brk_indexer::Indexes,
-        exit: &Exit,
-    ) -> color_eyre::Result<()> {
-        info!("Computing...");
-        self.vecs.compute(
-            indexer,
-            starting_indexes,
-            self.fetcher.as_mut(),
-            exit,
-            &mut self.stores,
-        )
-    }
-}
+// pub fn generate_allocation_files(monitored: &pools::Vecs) -> Result<()> {
+//     info!("Generating Allocative files...");
+
+//     let mut flamegraph = allocative::FlameGraphBuilder::default();
+//     flamegraph.visit_root(monitored);
+//     let output = flamegraph.finish();
+
+//     let folder = format!(
+//         "at-{}",
+//         jiff::Timestamp::now().strftime("%Y-%m-%d_%Hh%Mm%Ss"),
+//     );
+
+//     let path = std::path::PathBuf::from(&format!("./target/flamegraph/{folder}"));
+//     std::fs::create_dir_all(&path)?;
+
+//     // fs::write(path.join("flamegraph.src"), &output.flamegraph())?;
+
+//     let mut fg_svg = Vec::new();
+//     inferno::flamegraph::from_reader(
+//         &mut inferno::flamegraph::Options::default(),
+//         output.flamegraph().write().as_bytes(),
+//         &mut fg_svg,
+//     )?;
+
+//     std::fs::write(path.join("flamegraph.svg"), &fg_svg)?;
+
+//     std::fs::write(path.join("warnings.txt"), output.warnings())?;
+
+//     info!("Successfully generated Allocative files");
+
+//     Ok(())
+// }

crates/brk_computer/src/pools/mod.rs

@@ -0,0 +1,227 @@
+use std::{collections::BTreeMap, path::Path};
+
+use allocative::Allocative;
+use brk_error::Result;
+use brk_indexer::Indexer;
+use brk_store::AnyStore;
+use brk_structs::{AddressBytes, Height, OutputIndex, OutputType, PoolId, Pools, pools};
+use rayon::prelude::*;
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, Exit, GenericStoredVec,
+    PAGE_SIZE, RawVec, StoredIndex, VecIterator, Version,
+};
+
+mod vecs;
+
+use crate::{
+    chain,
+    indexes::{self, Indexes},
+    price,
+};
+
+#[derive(Clone, Allocative)]
+pub struct Vecs {
+    db: Database,
+    pools: &'static Pools,
+    height_to_pool: RawVec<Height, PoolId>,
+
+    vecs: BTreeMap<PoolId, vecs::Vecs>,
+}
+
+impl Vecs {
+    pub fn forced_import(
+        parent_path: &Path,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+    ) -> Result<Self> {
+        let db = Database::open(&parent_path.join("pools"))?;
+        db.set_min_len(PAGE_SIZE * 1_000_000)?;
+        let pools = pools();
+
+        let version = parent_version + Version::new(3) + Version::new(pools.len() as u64);
+
+        let this = Self {
+            height_to_pool: RawVec::forced_import(&db, "pool", version + Version::ZERO)?,
+            vecs: pools
+                .iter()
+                .map(|pool| {
+                    vecs::Vecs::forced_import(
+                        &db,
+                        pool.id,
+                        pools,
+                        version + Version::ZERO,
+                        indexes,
+                        price,
+                    )
+                    .map(|vecs| (pool.id, vecs))
+                })
+                .collect::<Result<BTreeMap<_, _>>>()?,
+            pools,
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexer, indexes, starting_indexes, chain, price, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_height_to_pool(indexer, indexes, starting_indexes, exit)?;
+
+        self.vecs.par_iter_mut().try_for_each(|(_, vecs)| {
+            vecs.compute(
+                indexes,
+                starting_indexes,
+                &self.height_to_pool,
+                chain,
+                price,
+                exit,
+            )
+        })?;
+
+        Ok(())
+    }
+
+    fn compute_height_to_pool(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.height_to_pool.validate_computed_version_or_reset(
+            self.height_to_pool.version() + indexer.stores.height_to_coinbase_tag.version(),
+        )?;
+
+        let mut height_to_first_txindex_iter = indexer.vecs.height_to_first_txindex.iter();
+        let mut txindex_to_first_outputindex_iter =
+            indexer.vecs.txindex_to_first_outputindex.iter();
+        let mut txindex_to_output_count_iter = indexes.txindex_to_output_count.iter();
+        let mut outputindex_to_outputtype_iter = indexer.vecs.outputindex_to_outputtype.iter();
+        let mut outputindex_to_typeindex_iter = indexer.vecs.outputindex_to_typeindex.iter();
+        let mut p2pk65addressindex_to_p2pk65bytes_iter =
+            indexer.vecs.p2pk65addressindex_to_p2pk65bytes.iter();
+        let mut p2pk33addressindex_to_p2pk33bytes_iter =
+            indexer.vecs.p2pk33addressindex_to_p2pk33bytes.iter();
+        let mut p2pkhaddressindex_to_p2pkhbytes_iter =
+            indexer.vecs.p2pkhaddressindex_to_p2pkhbytes.iter();
+        let mut p2shaddressindex_to_p2shbytes_iter =
+            indexer.vecs.p2shaddressindex_to_p2shbytes.iter();
+        let mut p2wpkhaddressindex_to_p2wpkhbytes_iter =
+            indexer.vecs.p2wpkhaddressindex_to_p2wpkhbytes.iter();
+        let mut p2wshaddressindex_to_p2wshbytes_iter =
+            indexer.vecs.p2wshaddressindex_to_p2wshbytes.iter();
+        let mut p2traddressindex_to_p2trbytes_iter =
+            indexer.vecs.p2traddressindex_to_p2trbytes.iter();
+        let mut p2aaddressindex_to_p2abytes_iter = indexer.vecs.p2aaddressindex_to_p2abytes.iter();
+
+        let unknown = self.pools.get_unknown();
+
+        let min = starting_indexes
+            .height
+            .unwrap_to_usize()
+            .min(self.height_to_pool.len());
+
+        indexer
+            .stores
+            .height_to_coinbase_tag
+            .iter()
+            .skip(min)
+            .try_for_each(|(height, coinbase_tag)| -> Result<()> {
+                let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+                let outputindex = txindex_to_first_outputindex_iter.unwrap_get_inner(txindex);
+                let outputcount = txindex_to_output_count_iter.unwrap_get_inner(txindex);
+
+                let pool = (*outputindex..(*outputindex + *outputcount))
+                    .map(OutputIndex::from)
+                    .find_map(|outputindex| {
+                        let outputtype =
+                            outputindex_to_outputtype_iter.unwrap_get_inner(outputindex);
+                        let typeindex = outputindex_to_typeindex_iter.unwrap_get_inner(outputindex);
+
+                        let address = match outputtype {
+                            OutputType::P2PK65 => Some(AddressBytes::from(
+                                p2pk65addressindex_to_p2pk65bytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2PK33 => Some(AddressBytes::from(
+                                p2pk33addressindex_to_p2pk33bytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2PKH => Some(AddressBytes::from(
+                                p2pkhaddressindex_to_p2pkhbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2SH => Some(AddressBytes::from(
+                                p2shaddressindex_to_p2shbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2WPKH => Some(AddressBytes::from(
+                                p2wpkhaddressindex_to_p2wpkhbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2WSH => Some(AddressBytes::from(
+                                p2wshaddressindex_to_p2wshbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2TR => Some(AddressBytes::from(
+                                p2traddressindex_to_p2trbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2A => Some(AddressBytes::from(
+                                p2aaddressindex_to_p2abytes_iter.unwrap_get_inner(typeindex.into()),
+                            )),
+                            _ => None,
+                        };
+
+                        address
+                            .and_then(|address| self.pools.find_from_address(&address.to_string()))
+                    })
+                    .or_else(|| self.pools.find_from_coinbase_tag(&coinbase_tag))
+                    .unwrap_or(unknown);
+
+                self.height_to_pool.push_if_needed(height, pool.id)?;
+                Ok(())
+            })?;
+
+        self.height_to_pool.safe_flush(exit)?;
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.height_to_pool as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(
+                self.vecs
+                    .iter()
+                    .flat_map(|(_, vecs)| vecs.iter_any_collectable()),
+            )
+    }
+}

crates/brk_computer/src/pools/vecs.rs

@@ -0,0 +1,397 @@
+use allocative::Allocative;
+use brk_error::Result;
+use brk_structs::{Height, PoolId, Pools, Sats, StoredF32, StoredU16, StoredU32};
+use vecdb::{AnyCollectableVec, AnyIterableVec, Database, Exit, StoredIndex, VecIterator, Version};
+
+use crate::{
+    chain,
+    grouped::{
+        ComputedValueVecsFromHeight, ComputedVecsFromDateIndex, ComputedVecsFromHeight, Source,
+        VecBuilderOptions,
+    },
+    indexes::{self, Indexes},
+    price,
+};
+
+#[derive(Clone, Allocative)]
+pub struct Vecs {
+    id: PoolId,
+
+    indexes_to_blocks_mined: ComputedVecsFromHeight<StoredU32>,
+    indexes_to_1w_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_1m_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_1y_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_subsidy: ComputedValueVecsFromHeight,
+    indexes_to_fee: ComputedValueVecsFromHeight,
+    indexes_to_coinbase: ComputedValueVecsFromHeight,
+    indexes_to_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1d_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1w_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1m_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1y_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_days_since_block: ComputedVecsFromDateIndex<StoredU16>,
+}
+
+impl Vecs {
+    pub fn forced_import(
+        db: &Database,
+        id: PoolId,
+        pools: &Pools,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+    ) -> Result<Self> {
+        let pool = pools.get(id);
+        let name = pool.serialized_id();
+        let suffix = |s: &str| format!("{name}_{s}");
+        let compute_dollars = price.is_some();
+        let version = parent_version + Version::ZERO;
+
+        Ok(Self {
+            id,
+            indexes_to_blocks_mined: ComputedVecsFromHeight::forced_import(
+                db,
+                &suffix("blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+            )?,
+            indexes_to_1w_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1w_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1m_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1m_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1y_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1y_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_subsidy: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("subsidy"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_fee: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("fee"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_coinbase: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("coinbase"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1d_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1d_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1w_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1w_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1m_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1m_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1y_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1y_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_days_since_block: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("days_since_block"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+        })
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn compute(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        height_to_pool: &impl AnyIterableVec<Height, PoolId>,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.indexes_to_blocks_mined
+            .compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_transform(
+                    starting_indexes.height,
+                    height_to_pool,
+                    |(h, id, ..)| {
+                        (
+                            h,
+                            if id == self.id {
+                                StoredU32::ONE
+                            } else {
+                                StoredU32::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1w_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    7,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1m_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    30,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1y_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    365,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        let height_to_blocks_mined = self.indexes_to_blocks_mined.height.as_ref().unwrap();
+
+        self.indexes_to_subsidy
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_subsidy.sats.height.as_ref().unwrap(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_fee
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_fee.sats.height.unwrap_sum(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_coinbase
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_coinbase.sats.height.as_ref().unwrap(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_cumulative(),
+                    chain.indexes_to_block_count.dateindex.unwrap_cumulative(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1d_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    chain.indexes_to_block_count.dateindex.unwrap_sum(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1w_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1w_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1w_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1m_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1m_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1m_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1y_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1y_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1y_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_days_since_block
+            .compute_all(starting_indexes, exit, |v| {
+                let mut prev = None;
+                v.compute_transform2(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    self.indexes_to_blocks_mined.dateindex.unwrap_cumulative(),
+                    |(i, sum, cumulative, slf)| {
+                        if prev.is_none() {
+                            let i = i.unwrap_to_usize();
+                            prev.replace(if i > 0 {
+                                slf.into_iter().unwrap_get_inner_(i - 1)
+                            } else {
+                                StoredU16::ZERO
+                            });
+                        }
+                        let days = if !cumulative.is_zero() && sum.is_zero() {
+                            prev.unwrap() + StoredU16::ONE
+                        } else {
+                            StoredU16::ZERO
+                        };
+                        prev.replace(days);
+                        (i, days)
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
+
+        iter = Box::new(iter.chain(self.indexes_to_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1w_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1m_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1y_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_subsidy.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_fee.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_coinbase.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1d_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1w_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1m_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1y_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_days_since_block.iter_any_collectable()));
+
+        iter
+    }
+}

crates/brk_computer/src/stateful/address_cohort.rs

@@ -1,145 +1,149 @@
-use std::{ops::Deref, path::Path};
+use std::path::Path;
 
-use brk_core::{Bitcoin, DateIndex, Dollars, Height, Result, StoredUsize, Version};
-use brk_exit::Exit;
-use brk_indexer::Indexer;
-use brk_vec::{
-    AnyCollectableVec, AnyIterableVec, AnyVec, Computation, EagerVec, Format, VecIterator,
+use brk_error::Result;
+use brk_structs::{Bitcoin, DateIndex, Dollars, Height, StoredU64, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, EagerVec, Exit, Format,
+    GenericStoredVec, VecIterator,
 };
 
 use crate::{
-    states::AddressCohortState,
-    vecs::{
-        Indexes, fetched,
-        grouped::{ComputedVecsFromHeight, Source, VecBuilderOptions},
-        indexes, market,
-        stateful::{
-            common,
-            r#trait::{CohortVecs, DynCohortVecs},
-        },
+    Indexes,
+    grouped::{ComputedVecsFromHeight, Source, VecBuilderOptions},
+    indexes, price,
+    stateful::{
+        common,
+        r#trait::{CohortVecs, DynCohortVecs},
     },
+    states::AddressCohortState,
 };
 
 const VERSION: Version = Version::ZERO;
 
 #[derive(Clone)]
 pub struct Vecs {
-    starting_height: Height,
+    starting_height: Option<Height>,
 
-    pub state: AddressCohortState,
+    pub state: Option<AddressCohortState>,
 
     pub inner: common::Vecs,
 
-    pub height_to_address_count: EagerVec<Height, StoredUsize>,
-    pub indexes_to_address_count: ComputedVecsFromHeight<StoredUsize>,
+    pub height_to_addr_count: EagerVec<Height, StoredU64>,
+    pub indexes_to_addr_count: ComputedVecsFromHeight<StoredU64>,
 }
 
 impl Vecs {
     #[allow(clippy::too_many_arguments)]
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         cohort_name: Option<&str>,
-        computation: Computation,
         format: Format,
         version: Version,
         indexes: &indexes::Vecs,
-        fetched: Option<&fetched::Vecs>,
-        states_path: &Path,
-        compute_relative_to_all: bool,
-    ) -> color_eyre::Result<Self> {
-        let compute_dollars = fetched.is_some();
+        price: Option<&price::Vecs>,
+        states_path: Option<&Path>,
+        compute_rel_to_all: bool,
+    ) -> Result<Self> {
+        let compute_dollars = price.is_some();
 
         let suffix = |s: &str| cohort_name.map_or(s.to_string(), |name| format!("{name}_{s}"));
 
         Ok(Self {
-            starting_height: Height::ZERO,
-            state: AddressCohortState::default_and_import(
-                states_path,
-                cohort_name.unwrap_or_default(),
-                compute_dollars,
-            )?,
-            height_to_address_count: EagerVec::forced_import(
-                path,
-                &suffix("address_count"),
+            starting_height: None,
+            state: states_path.map(|states_path| {
+                AddressCohortState::new(
+                    states_path,
+                    cohort_name.unwrap_or_default(),
+                    compute_dollars,
+                )
+            }),
+            height_to_addr_count: EagerVec::forced_import(
+                db,
+                &suffix("addr_count"),
                 version + VERSION + Version::ZERO,
                 format,
             )?,
-            indexes_to_address_count: ComputedVecsFromHeight::forced_import(
-                path,
-                &suffix("address_count"),
+            indexes_to_addr_count: ComputedVecsFromHeight::forced_import(
+                db,
+                &suffix("addr_count"),
                 Source::None,
                 version + VERSION + Version::ZERO,
-                format,
-                computation,
                 indexes,
                 VecBuilderOptions::default().add_last(),
             )?,
             inner: common::Vecs::forced_import(
-                path,
+                db,
                 cohort_name,
-                computation,
                 format,
                 version,
                 indexes,
-                fetched,
-                compute_relative_to_all,
+                price,
+                false,
+                compute_rel_to_all,
                 false,
             )?,
         })
     }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        self.inner
+            .iter_any_collectable()
+            .chain([&self.height_to_addr_count as &dyn AnyCollectableVec])
+            .chain(self.indexes_to_addr_count.iter_any_collectable())
+    }
 }
 
 impl DynCohortVecs for Vecs {
-    fn starting_height(&self) -> Height {
-        [
-            self.state.height().map_or(Height::MAX, |h| h.incremented()),
-            self.height_to_address_count.len().into(),
-            self.inner.starting_height(),
-        ]
-        .into_iter()
-        .min()
-        .unwrap()
+    fn min_height_vecs_len(&self) -> usize {
+        std::cmp::min(
+            self.height_to_addr_count.len(),
+            self.inner.min_height_vecs_len(),
+        )
     }
 
-    fn init(&mut self, starting_height: Height) {
-        if starting_height > self.starting_height() {
-            unreachable!()
-        }
+    fn reset_state_starting_height(&mut self) {
+        self.starting_height = Some(Height::ZERO);
+    }
 
-        self.starting_height = starting_height;
+    fn import_state(&mut self, starting_height: Height) -> Result<Height> {
+        let starting_height = self
+            .inner
+            .import_state(starting_height, &mut self.state.as_mut().unwrap().inner)?;
+
+        self.starting_height = Some(starting_height);
 
         if let Some(prev_height) = starting_height.decremented() {
-            self.state.address_count = *self
-                .height_to_address_count
+            self.state.as_mut().unwrap().addr_count = *self
+                .height_to_addr_count
                 .into_iter()
                 .unwrap_get_inner(prev_height);
         }
 
-        self.inner
-            .init(&mut self.starting_height, &mut self.state.inner);
+        Ok(starting_height)
     }
 
     fn validate_computed_versions(&mut self, base_version: Version) -> Result<()> {
-        self.height_to_address_count
-            .validate_computed_version_or_reset_file(
-                base_version + self.height_to_address_count.inner_version(),
+        self.height_to_addr_count
+            .validate_computed_version_or_reset(
+                base_version + self.height_to_addr_count.inner_version(),
             )?;
 
         self.inner.validate_computed_versions(base_version)
     }
 
     fn forced_pushed_at(&mut self, height: Height, exit: &Exit) -> Result<()> {
-        if self.starting_height > height {
+        if self.starting_height.unwrap() > height {
             return Ok(());
         }
 
-        self.height_to_address_count.forced_push_at(
+        self.height_to_addr_count.forced_push_at(
             height,
-            self.state.address_count.into(),
+            self.state.as_ref().unwrap().addr_count.into(),
             exit,
         )?;
 
-        self.inner.forced_pushed_at(height, exit, &self.state.inner)
+        self.inner
+            .forced_pushed_at(height, exit, &self.state.as_ref().unwrap().inner)
     }
 
     fn compute_then_force_push_unrealized_states(
@@ -156,44 +160,34 @@ impl DynCohortVecs for Vecs {
             dateindex,
             date_price,
             exit,
-            &self.state.inner,
+            &self.state.as_ref().unwrap().inner,
         )
     }
 
     fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
-        self.height_to_address_count.safe_flush(exit)?;
+        self.height_to_addr_count.safe_flush(exit)?;
 
         self.inner
-            .safe_flush_stateful_vecs(height, exit, &mut self.state.inner)
+            .safe_flush_stateful_vecs(height, exit, &mut self.state.as_mut().unwrap().inner)
     }
 
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
-    ) -> color_eyre::Result<()> {
-        self.indexes_to_address_count.compute_rest(
+    ) -> Result<()> {
+        self.indexes_to_addr_count.compute_rest(
             indexes,
             starting_indexes,
             exit,
-            Some(&self.height_to_address_count),
+            Some(&self.height_to_addr_count),
         )?;
 
         self.inner
-            .compute_rest_part1(indexer, indexes, fetched, starting_indexes, exit)
-    }
-
-    fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.inner.vecs(),
-            self.indexes_to_address_count.vecs(),
-            vec![&self.height_to_address_count],
-        ]
-        .concat()
+            .compute_rest_part1(indexes, price, starting_indexes, exit)
     }
 }
 
@@ -204,11 +198,11 @@ impl CohortVecs for Vecs {
         others: &[&Self],
         exit: &Exit,
     ) -> Result<()> {
-        self.height_to_address_count.compute_sum_of_others(
+        self.height_to_addr_count.compute_sum_of_others(
             starting_indexes.height,
             others
                 .iter()
-                .map(|v| &v.height_to_address_count)
+                .map(|v| &v.height_to_addr_count)
                 .collect::<Vec<_>>()
                 .as_slice(),
             exit,
@@ -223,35 +217,28 @@ impl CohortVecs for Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,
-    ) -> color_eyre::Result<()> {
+    ) -> Result<()> {
         self.inner.compute_rest_part2(
-            indexer,
             indexes,
-            fetched,
+            price,
             starting_indexes,
-            market,
             height_to_supply,
             dateindex_to_supply,
+            height_to_market_cap,
+            dateindex_to_market_cap,
             height_to_realized_cap,
             dateindex_to_realized_cap,
             exit,
         )
     }
 }
-
-impl Deref for Vecs {
-    type Target = common::Vecs;
-    fn deref(&self) -> &Self::Target {
-        &self.inner
-    }
-}

crates/brk_computer/src/stateful/address_cohorts.rs

@@ -1,16 +1,15 @@
 use std::path::Path;
 
-use brk_core::{
-    AddressGroups, ByAmountRange, ByGreatEqualAmount, ByLowerThanAmount, GroupFilter, Height,
-    Result, Version,
+use brk_error::Result;
+use brk_structs::{
+    AddressGroups, Bitcoin, ByAmountRange, ByGreatEqualAmount, ByLowerThanAmount, DateIndex,
+    Dollars, GroupFilter, Height, Version,
 };
-use brk_exit::Exit;
-use brk_vec::{Computation, Format};
 use derive_deref::{Deref, DerefMut};
-use rayon::prelude::*;
+use vecdb::{AnyIterableVec, Database, Exit, Format};
 
-use crate::vecs::{
-    Indexes, fetched, indexes,
+use crate::{
+    Indexes, indexes, price,
     stateful::{
         address_cohort,
         r#trait::{CohortVecs, DynCohortVecs},
@@ -24,470 +23,428 @@ pub struct Vecs(AddressGroups<(GroupFilter, address_cohort::Vecs)>);
 
 impl Vecs {
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         version: Version,
-        _computation: Computation,
         format: Format,
         indexes: &indexes::Vecs,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         states_path: &Path,
-    ) -> color_eyre::Result<Self> {
+    ) -> Result<Self> {
         Ok(Self(
             AddressGroups {
                 amount_range: ByAmountRange {
                     _0sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_with_0sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _1sat_to_10sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_1sat_under_10sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _10sats_to_100sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_10sats_under_100sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _100sats_to_1k_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_100sats_under_1k_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _1k_sats_to_10k_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_1k_sats_under_10k_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _10k_sats_to_100k_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_10k_sats_under_100k_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _100k_sats_to_1m_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_100k_sats_under_1m_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _1m_sats_to_10m_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_1m_sats_under_10m_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _10m_sats_to_1btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_10m_sats_under_1btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _1btc_to_10btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_1btc_under_10btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _10btc_to_100btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_10btc_under_100btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _100btc_to_1k_btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_100btc_under_1k_btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _1k_btc_to_10k_btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_1k_btc_under_10k_btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _10k_btc_to_100k_btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_10k_btc_under_100k_btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                     _100k_btc_or_more: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_100k_btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        Some(states_path),
                         true,
                     )?,
                 },
                 lt_amount: ByLowerThanAmount {
                     _10sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_10sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _100sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_100sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _1k_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_1k_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _10k_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_10k_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _100k_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_100k_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _1m_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_1m_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _10m_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_10m_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _1btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_1btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _10btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_10btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _100btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_100btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _1k_btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_1k_btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _10k_btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_10k_btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _100k_btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_under_100k_btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                 },
                 ge_amount: ByGreatEqualAmount {
                     _1sat: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_1sat"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _10sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_10sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _100sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_100sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _1k_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_1k_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _10k_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_10k_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _100k_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_100k_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _1m_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_1m_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _10m_sats: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_10m_sats"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _1btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_1btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _10btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_10btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _100btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_100btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _1k_btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_1k_btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                     _10k_btc: address_cohort::Vecs::forced_import(
-                        path,
+                        db,
                         Some("addrs_above_10k_btc"),
-                        _computation,
                         format,
                         version + VERSION + Version::ZERO,
                         indexes,
-                        fetched,
-                        states_path,
+                        price,
+                        None,
                         true,
                     )?,
                 },
@@ -501,18 +458,17 @@ impl Vecs {
         starting_indexes: &Indexes,
         exit: &Exit,
     ) -> Result<()> {
-        let by_size_range = self.0.amount_range.as_vec();
+        let by_size_range = &self.0.amount_range;
 
         [
             self.0
                 .ge_amount
-                .as_mut_vec()
-                .into_iter()
+                .iter_mut()
                 .map(|(filter, vecs)| {
                     (
                         vecs,
                         by_size_range
-                            .into_iter()
+                            .iter()
                             .filter(|(other, _)| filter.includes(other))
                             .map(|(_, v)| v)
                             .collect::<Vec<_>>(),
@@ -521,13 +477,12 @@ impl Vecs {
                 .collect::<Vec<_>>(),
             self.0
                 .lt_amount
-                .as_mut_vec()
-                .into_iter()
+                .iter_mut()
                 .map(|(filter, vecs)| {
                     (
                         vecs,
                         by_size_range
-                            .into_iter()
+                            .iter()
                             .filter(|(other, _)| filter.includes(other))
                             .map(|(_, v)| v)
                             .collect::<Vec<_>>(),
@@ -535,16 +490,58 @@ impl Vecs {
                 })
                 .collect::<Vec<_>>(),
         ]
-        .into_par_iter()
+        .into_iter()
         .flatten()
         .try_for_each(|(vecs, stateful)| {
             vecs.compute_from_stateful(starting_indexes, &stateful, exit)
         })
     }
 
+    pub fn compute_rest_part1(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.iter_mut()
+            .into_iter()
+            .try_for_each(|(_, v)| v.compute_rest_part1(indexes, price, starting_indexes, exit))
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn compute_rest_part2(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
+        dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+        height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.0.iter_mut().try_for_each(|(_, v)| {
+            v.compute_rest_part2(
+                indexes,
+                price,
+                starting_indexes,
+                height_to_supply,
+                dateindex_to_supply,
+                height_to_market_cap,
+                dateindex_to_market_cap,
+                height_to_realized_cap,
+                dateindex_to_realized_cap,
+                exit,
+            )
+        })
+    }
+
     pub fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
-        self.as_mut_separate_vecs()
-            .par_iter_mut()
+        self.iter_separate_mut()
+            .into_iter()
             .try_for_each(|(_, v)| v.safe_flush_stateful_vecs(height, exit))
     }
 }

crates/brk_computer/src/stateful/addresstype/addresscount.rs

@@ -1,11 +1,11 @@
-use brk_core::{ByAddressType, Height};
-use brk_vec::VecIterator;
+use brk_structs::{ByAddressType, Height};
 use derive_deref::{Deref, DerefMut};
+use vecdb::VecIterator;
 
-use crate::vecs::stateful::addresstype_to_height_to_addresscount::AddressTypeToHeightToAddressCount;
+use super::AddressTypeToHeightToAddressCount;
 
 #[derive(Debug, Default, Deref, DerefMut)]
-pub struct AddressTypeToAddressCount(ByAddressType<usize>);
+pub struct AddressTypeToAddressCount(ByAddressType<u64>);
 
 impl From<(&AddressTypeToHeightToAddressCount, Height)> for AddressTypeToAddressCount {
     fn from((groups, starting_height): (&AddressTypeToHeightToAddressCount, Height)) -> Self {

crates/brk_computer/src/stateful/addresstype/height_to_addresscount.rs

@@ -1,15 +1,15 @@
-use brk_core::{ByAddressType, Height, Result, StoredUsize};
-use brk_exit::Exit;
-use brk_vec::EagerVec;
+use brk_error::Result;
+use brk_structs::{ByAddressType, Height, StoredU64};
 use derive_deref::{Deref, DerefMut};
+use vecdb::{EagerVec, Exit, GenericStoredVec};
 
-use crate::vecs::stateful::addresstype_to_addresscount::AddressTypeToAddressCount;
+use super::AddressTypeToAddressCount;
 
 #[derive(Debug, Clone, Deref, DerefMut)]
-pub struct AddressTypeToHeightToAddressCount(ByAddressType<EagerVec<Height, StoredUsize>>);
+pub struct AddressTypeToHeightToAddressCount(ByAddressType<EagerVec<Height, StoredU64>>);
 
-impl From<ByAddressType<EagerVec<Height, StoredUsize>>> for AddressTypeToHeightToAddressCount {
-    fn from(value: ByAddressType<EagerVec<Height, StoredUsize>>) -> Self {
+impl From<ByAddressType<EagerVec<Height, StoredU64>>> for AddressTypeToHeightToAddressCount {
+    fn from(value: ByAddressType<EagerVec<Height, StoredU64>>) -> Self {
         Self(value)
     }
 }

crates/brk_computer/src/stateful/addresstype/height_to_vec.rs

@@ -1,9 +1,9 @@
 use std::collections::BTreeMap;
 
-use brk_core::Height;
+use brk_structs::Height;
 use derive_deref::{Deref, DerefMut};
 
-use crate::vecs::stateful::AddressTypeToVec;
+use crate::stateful::AddressTypeToVec;
 
 #[derive(Debug, Default, Deref, DerefMut)]
 pub struct HeightToAddressTypeToVec<T>(pub BTreeMap<Height, AddressTypeToVec<T>>);

crates/brk_computer/src/stateful/addresstype/indexes_to_addresscount.rs

@@ -1,20 +1,17 @@
-use brk_core::{ByAddressType, StoredUsize};
-use brk_exit::Exit;
-use brk_vec::AnyCollectableVec;
+use brk_error::Result;
+use brk_structs::{ByAddressType, StoredU64};
 use derive_deref::{Deref, DerefMut};
+use vecdb::{AnyCollectableVec, Exit};
 
-use crate::vecs::{
-    Indexes, grouped::ComputedVecsFromHeight, indexes,
-    stateful::addresstype_to_height_to_addresscount::AddressTypeToHeightToAddressCount,
-};
+use crate::{Indexes, grouped::ComputedVecsFromHeight, indexes};
+
+use super::AddressTypeToHeightToAddressCount;
 
 #[derive(Clone, Deref, DerefMut)]
-pub struct AddressTypeToIndexesToAddressCount(ByAddressType<ComputedVecsFromHeight<StoredUsize>>);
+pub struct AddressTypeToIndexesToAddressCount(ByAddressType<ComputedVecsFromHeight<StoredU64>>);
 
-impl From<ByAddressType<ComputedVecsFromHeight<StoredUsize>>>
-    for AddressTypeToIndexesToAddressCount
-{
-    fn from(value: ByAddressType<ComputedVecsFromHeight<StoredUsize>>) -> Self {
+impl From<ByAddressType<ComputedVecsFromHeight<StoredU64>>> for AddressTypeToIndexesToAddressCount {
+    fn from(value: ByAddressType<ComputedVecsFromHeight<StoredU64>>) -> Self {
         Self(value)
     }
 }
@@ -22,12 +19,11 @@ impl From<ByAddressType<ComputedVecsFromHeight<StoredUsize>>>
 impl AddressTypeToIndexesToAddressCount {
     pub fn compute(
         &mut self,
-        // height: Height,
         indexes: &indexes::Vecs,
         starting_indexes: &Indexes,
         exit: &Exit,
         addresstype_to_height_to_addresscount: &AddressTypeToHeightToAddressCount,
-    ) -> color_eyre::Result<()> {
+    ) -> Result<()> {
         self.p2pk65.compute_rest(
             indexes,
             starting_indexes,
@@ -78,12 +74,10 @@ impl AddressTypeToIndexesToAddressCount {
         )?;
         Ok(())
     }
-    //
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
         self.0
-            .as_typed_vec()
-            .into_iter()
-            .flat_map(|(_, v)| v.vecs())
-            .collect::<Vec<_>>()
+            .iter_typed()
+            .flat_map(|(_, v)| v.iter_any_collectable())
     }
 }

crates/brk_computer/src/stateful/addresstype/mod.rs

@@ -0,0 +1,13 @@
+mod addresscount;
+mod height_to_addresscount;
+mod height_to_vec;
+mod indexes_to_addresscount;
+mod typeindex_tree;
+mod vec;
+
+pub use addresscount::*;
+pub use height_to_addresscount::*;
+pub use height_to_vec::*;
+pub use indexes_to_addresscount::*;
+pub use typeindex_tree::*;
+pub use vec::*;

crates/brk_computer/src/stateful/addresstype/typeindex_tree.rs

@@ -1,10 +1,8 @@
 use std::{collections::BTreeMap, mem};
 
-use brk_core::TypeIndex;
+use brk_structs::{ByAddressType, TypeIndex};
 use derive_deref::{Deref, DerefMut};
 
-use super::ByAddressType;
-
 #[derive(Debug, Deref, DerefMut)]
 pub struct AddressTypeToTypeIndexTree<T>(ByAddressType<BTreeMap<TypeIndex, T>>);
 

crates/brk_computer/src/stateful/addresstype/vec.rs

@@ -0,0 +1,60 @@
+use std::mem;
+
+use brk_structs::ByAddressType;
+use derive_deref::{Deref, DerefMut};
+
+#[derive(Debug, Deref, DerefMut)]
+pub struct AddressTypeToVec<T>(ByAddressType<Vec<T>>);
+
+impl<T> AddressTypeToVec<T> {
+    pub fn merge(mut self, mut other: Self) -> Self {
+        Self::merge_(&mut self.p2pk65, &mut other.p2pk65);
+        Self::merge_(&mut self.p2pk33, &mut other.p2pk33);
+        Self::merge_(&mut self.p2pkh, &mut other.p2pkh);
+        Self::merge_(&mut self.p2sh, &mut other.p2sh);
+        Self::merge_(&mut self.p2wpkh, &mut other.p2wpkh);
+        Self::merge_(&mut self.p2wsh, &mut other.p2wsh);
+        Self::merge_(&mut self.p2tr, &mut other.p2tr);
+        Self::merge_(&mut self.p2a, &mut other.p2a);
+        self
+    }
+
+    pub fn merge_mut(&mut self, mut other: Self) {
+        Self::merge_(&mut self.p2pk65, &mut other.p2pk65);
+        Self::merge_(&mut self.p2pk33, &mut other.p2pk33);
+        Self::merge_(&mut self.p2pkh, &mut other.p2pkh);
+        Self::merge_(&mut self.p2sh, &mut other.p2sh);
+        Self::merge_(&mut self.p2wpkh, &mut other.p2wpkh);
+        Self::merge_(&mut self.p2wsh, &mut other.p2wsh);
+        Self::merge_(&mut self.p2tr, &mut other.p2tr);
+        Self::merge_(&mut self.p2a, &mut other.p2a);
+    }
+
+    fn merge_(own: &mut Vec<T>, other: &mut Vec<T>) {
+        if own.len() >= other.len() {
+            own.append(other);
+        } else {
+            other.append(own);
+            mem::swap(own, other);
+        }
+    }
+
+    pub fn unwrap(self) -> ByAddressType<Vec<T>> {
+        self.0
+    }
+}
+
+impl<T> Default for AddressTypeToVec<T> {
+    fn default() -> Self {
+        Self(ByAddressType {
+            p2pk65: vec![],
+            p2pk33: vec![],
+            p2pkh: vec![],
+            p2sh: vec![],
+            p2wpkh: vec![],
+            p2wsh: vec![],
+            p2tr: vec![],
+            p2a: vec![],
+        })
+    }
+}

crates/brk_computer/src/stateful/range_map.rs

@@ -1,6 +1,6 @@
 use std::collections::BTreeMap;
 
-use brk_vec::{IndexedVec, StoredIndex, StoredType};
+use vecdb::{CompressedVec, RawVec, StoredCompressed, StoredIndex, StoredRaw};
 
 #[derive(Debug)]
 pub struct RangeMap<I, T>(BTreeMap<I, T>);
@@ -20,12 +20,26 @@ where
     }
 }
 
-impl<I, T> From<&IndexedVec<I, T>> for RangeMap<T, I>
+impl<I, T> From<&RawVec<I, T>> for RangeMap<T, I>
 where
     I: StoredIndex,
-    T: StoredIndex + StoredType,
+    T: StoredIndex + StoredRaw,
 {
-    fn from(vec: &IndexedVec<I, T>) -> Self {
+    fn from(vec: &RawVec<I, T>) -> Self {
+        Self(
+            vec.into_iter()
+                .map(|(i, v)| (v.into_owned(), i))
+                .collect::<BTreeMap<_, _>>(),
+        )
+    }
+}
+
+impl<I, T> From<&CompressedVec<I, T>> for RangeMap<T, I>
+where
+    I: StoredIndex,
+    T: StoredIndex + StoredCompressed,
+{
+    fn from(vec: &CompressedVec<I, T>) -> Self {
         Self(
             vec.into_iter()
                 .map(|(i, v)| (v.into_owned(), i))

crates/brk_computer/src/stateful/trait.rs

@@ -1,14 +1,14 @@
-use brk_core::{Bitcoin, DateIndex, Dollars, Height, Result, Version};
-use brk_exit::Exit;
-use brk_indexer::Indexer;
-use brk_vec::{AnyCollectableVec, AnyIterableVec};
+use brk_error::Result;
+use brk_structs::{Bitcoin, DateIndex, Dollars, Height, Version};
+use vecdb::{AnyIterableVec, Exit};
 
-use crate::vecs::{Indexes, fetched, indexes, market};
+use crate::{Indexes, indexes, price};
 
 pub trait DynCohortVecs: Send + Sync {
-    fn starting_height(&self) -> Height;
+    fn min_height_vecs_len(&self) -> usize;
+    fn reset_state_starting_height(&mut self);
 
-    fn init(&mut self, starting_height: Height);
+    fn import_state(&mut self, starting_height: Height) -> Result<Height>;
 
     fn validate_computed_versions(&mut self, base_version: Version) -> Result<()>;
 
@@ -28,14 +28,11 @@ pub trait DynCohortVecs: Send + Sync {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
-    ) -> color_eyre::Result<()>;
-
-    fn vecs(&self) -> Vec<&dyn AnyCollectableVec>;
+    ) -> Result<()>;
 }
 
 pub trait CohortVecs: DynCohortVecs {
@@ -49,15 +46,15 @@ pub trait CohortVecs: DynCohortVecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,
-    ) -> color_eyre::Result<()>;
+    ) -> Result<()>;
 }

crates/brk_computer/src/stateful/utxo_cohort.rs

@@ -1,26 +1,22 @@
 use std::{ops::Deref, path::Path};
 
-use brk_core::{Bitcoin, DateIndex, Dollars, Height, Result, Version};
-use brk_exit::Exit;
-use brk_indexer::Indexer;
-use brk_vec::{AnyCollectableVec, AnyIterableVec, Computation, Format};
+use brk_error::Result;
+use brk_structs::{Bitcoin, DateIndex, Dollars, Height, Version};
+use vecdb::{AnyIterableVec, Database, Exit, Format};
 
 use crate::{
-    UTXOCohortState,
-    vecs::{
-        Indexes, fetched, indexes, market,
-        stateful::{
-            common,
-            r#trait::{CohortVecs, DynCohortVecs},
-        },
+    Indexes, UTXOCohortState, indexes, price,
+    stateful::{
+        common,
+        r#trait::{CohortVecs, DynCohortVecs},
     },
 };
 
 #[derive(Clone)]
 pub struct Vecs {
-    starting_height: Height,
+    state_starting_height: Option<Height>,
 
-    pub state: UTXOCohortState,
+    pub state: Option<UTXOCohortState>,
 
     inner: common::Vecs,
 }
@@ -28,62 +24,62 @@ pub struct Vecs {
 impl Vecs {
     #[allow(clippy::too_many_arguments)]
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         cohort_name: Option<&str>,
-        computation: Computation,
         format: Format,
         version: Version,
         indexes: &indexes::Vecs,
-        fetched: Option<&fetched::Vecs>,
-        states_path: &Path,
-        compute_relative_to_all: bool,
-        ratio_extended: bool,
-    ) -> color_eyre::Result<Self> {
-        let compute_dollars = fetched.is_some();
+        price: Option<&price::Vecs>,
+        states_path: Option<&Path>,
+        extended: bool,
+        compute_rel_to_all: bool,
+        compute_adjusted: bool,
+    ) -> Result<Self> {
+        let compute_dollars = price.is_some();
 
         Ok(Self {
-            starting_height: Height::ZERO,
+            state_starting_height: None,
 
-            state: UTXOCohortState::default_and_import(
-                states_path,
-                cohort_name.unwrap_or_default(),
-                compute_dollars,
-            )?,
+            state: states_path.map(|states_path| {
+                UTXOCohortState::new(
+                    states_path,
+                    cohort_name.unwrap_or_default(),
+                    compute_dollars,
+                )
+            }),
 
             inner: common::Vecs::forced_import(
-                path,
+                db,
                 cohort_name,
-                computation,
                 format,
                 version,
                 indexes,
-                fetched,
-                compute_relative_to_all,
-                ratio_extended,
+                price,
+                extended,
+                compute_rel_to_all,
+                compute_adjusted,
             )?,
         })
     }
 }
 
 impl DynCohortVecs for Vecs {
-    fn starting_height(&self) -> Height {
-        [
-            self.state.height().map_or(Height::MAX, |h| h.incremented()),
-            self.inner.starting_height(),
-        ]
-        .into_iter()
-        .min()
-        .unwrap()
+    fn min_height_vecs_len(&self) -> usize {
+        self.inner.min_height_vecs_len()
     }
 
-    fn init(&mut self, starting_height: Height) {
-        if starting_height > self.starting_height() {
-            unreachable!()
-        }
+    fn reset_state_starting_height(&mut self) {
+        self.state_starting_height = Some(Height::ZERO);
+    }
 
-        self.starting_height = starting_height;
+    fn import_state(&mut self, starting_height: Height) -> Result<Height> {
+        let starting_height = self
+            .inner
+            .import_state(starting_height, self.state.as_mut().unwrap())?;
 
-        self.inner.init(&mut self.starting_height, &mut self.state);
+        self.state_starting_height = Some(starting_height);
+
+        Ok(starting_height)
     }
 
     fn validate_computed_versions(&mut self, base_version: Version) -> Result<()> {
@@ -91,11 +87,12 @@ impl DynCohortVecs for Vecs {
     }
 
     fn forced_pushed_at(&mut self, height: Height, exit: &Exit) -> Result<()> {
-        if self.starting_height > height {
+        if self.state_starting_height.unwrap() > height {
             return Ok(());
         }
 
-        self.inner.forced_pushed_at(height, exit, &self.state)
+        self.inner
+            .forced_pushed_at(height, exit, self.state.as_ref().unwrap())
     }
 
     fn compute_then_force_push_unrealized_states(
@@ -112,30 +109,25 @@ impl DynCohortVecs for Vecs {
             dateindex,
             date_price,
             exit,
-            &self.state,
+            self.state.as_mut().unwrap(),
         )
     }
 
     fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
         self.inner
-            .safe_flush_stateful_vecs(height, exit, &mut self.state)
+            .safe_flush_stateful_vecs(height, exit, self.state.as_mut().unwrap())
     }
 
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part1(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
-    ) -> color_eyre::Result<()> {
+    ) -> Result<()> {
         self.inner
-            .compute_rest_part1(indexer, indexes, fetched, starting_indexes, exit)
-    }
-
-    fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        self.inner.vecs()
+            .compute_rest_part1(indexes, price, starting_indexes, exit)
     }
 }
 
@@ -156,25 +148,25 @@ impl CohortVecs for Vecs {
     #[allow(clippy::too_many_arguments)]
     fn compute_rest_part2(
         &mut self,
-        indexer: &Indexer,
         indexes: &indexes::Vecs,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
-        market: &market::Vecs,
         height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
         dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
         dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
         exit: &Exit,
-    ) -> color_eyre::Result<()> {
+    ) -> Result<()> {
         self.inner.compute_rest_part2(
-            indexer,
             indexes,
-            fetched,
+            price,
             starting_indexes,
-            market,
             height_to_supply,
             dateindex_to_supply,
+            height_to_market_cap,
+            dateindex_to_market_cap,
             height_to_realized_cap,
             dateindex_to_realized_cap,
             exit,

crates/brk_computer/src/stateful/withaddressdatasource.rs

@@ -0,0 +1,54 @@
+use brk_structs::{EmptyAddressData, EmptyAddressIndex, LoadedAddressData, LoadedAddressIndex};
+
+#[derive(Debug)]
+pub enum WithAddressDataSource<T> {
+    New(T),
+    FromLoadedAddressDataVec((LoadedAddressIndex, T)),
+    FromEmptyAddressDataVec((EmptyAddressIndex, T)),
+}
+
+impl<T> WithAddressDataSource<T> {
+    pub fn is_new(&self) -> bool {
+        matches!(self, Self::New(_))
+    }
+
+    pub fn is_from_emptyaddressdata(&self) -> bool {
+        matches!(self, Self::FromEmptyAddressDataVec(_))
+    }
+
+    pub fn deref_mut(&mut self) -> &mut T {
+        match self {
+            Self::New(v) => v,
+            Self::FromLoadedAddressDataVec((_, v)) => v,
+            Self::FromEmptyAddressDataVec((_, v)) => v,
+        }
+    }
+}
+
+impl From<WithAddressDataSource<EmptyAddressData>> for WithAddressDataSource<LoadedAddressData> {
+    fn from(value: WithAddressDataSource<EmptyAddressData>) -> Self {
+        match value {
+            WithAddressDataSource::New(v) => Self::New(v.into()),
+            WithAddressDataSource::FromLoadedAddressDataVec((i, v)) => {
+                Self::FromLoadedAddressDataVec((i, v.into()))
+            }
+            WithAddressDataSource::FromEmptyAddressDataVec((i, v)) => {
+                Self::FromEmptyAddressDataVec((i, v.into()))
+            }
+        }
+    }
+}
+
+impl From<WithAddressDataSource<LoadedAddressData>> for WithAddressDataSource<EmptyAddressData> {
+    fn from(value: WithAddressDataSource<LoadedAddressData>) -> Self {
+        match value {
+            WithAddressDataSource::New(v) => Self::New(v.into()),
+            WithAddressDataSource::FromLoadedAddressDataVec((i, v)) => {
+                Self::FromLoadedAddressDataVec((i, v.into()))
+            }
+            WithAddressDataSource::FromEmptyAddressDataVec((i, v)) => {
+                Self::FromEmptyAddressDataVec((i, v.into()))
+            }
+        }
+    }
+}

crates/brk_computer/src/states/block.rs

@@ -1,13 +1,17 @@
 use std::ops::{Add, AddAssign, SubAssign};
 
-use brk_core::{Dollars, Timestamp};
+use brk_structs::{Dollars, Timestamp};
+use serde::Serialize;
 
 use super::SupplyState;
 
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, Serialize)]
 pub struct BlockState {
+    #[serde(flatten)]
     pub supply: SupplyState,
+    #[serde(skip)]
     pub price: Option<Dollars>,
+    #[serde(skip)]
     pub timestamp: Timestamp,
 }
 

crates/brk_computer/src/states/cohorts/address.rs

@@ -1,6 +1,7 @@
 use std::path::Path;
 
-use brk_core::{AddressData, Dollars, Height, Result, Sats};
+use brk_error::Result;
+use brk_structs::{Dollars, Height, LoadedAddressData, Sats};
 
 use crate::SupplyState;
 
@@ -8,24 +9,20 @@ use super::CohortState;
 
 #[derive(Clone)]
 pub struct AddressCohortState {
-    pub address_count: usize,
+    pub addr_count: u64,
     pub inner: CohortState,
 }
 
 impl AddressCohortState {
-    pub fn default_and_import(path: &Path, name: &str, compute_dollars: bool) -> Result<Self> {
-        Ok(Self {
-            address_count: 0,
-            inner: CohortState::default_and_import(path, name, compute_dollars)?,
-        })
+    pub fn new(path: &Path, name: &str, compute_dollars: bool) -> Self {
+        Self {
+            addr_count: 0,
+            inner: CohortState::new(path, name, compute_dollars),
+        }
     }
 
-    pub fn height(&self) -> Option<Height> {
-        self.inner.height()
-    }
-
-    pub fn reset_price_to_amount(&mut self) -> Result<()> {
-        self.inner.reset_price_to_amount()
+    pub fn reset_price_to_amount_if_needed(&mut self) -> Result<()> {
+        self.inner.reset_price_to_amount_if_needed()
     }
 
     pub fn reset_single_iteration_values(&mut self) {
@@ -35,7 +32,7 @@ impl AddressCohortState {
     #[allow(clippy::too_many_arguments)]
     pub fn send(
         &mut self,
-        addressdata: &mut AddressData,
+        addressdata: &mut LoadedAddressData,
         value: Sats,
         current_price: Option<Dollars>,
         prev_price: Option<Dollars>,
@@ -47,14 +44,14 @@ impl AddressCohortState {
 
         let prev_realized_price = compute_price.then(|| addressdata.realized_price());
         let prev_supply_state = SupplyState {
-            utxos: addressdata.outputs_len as usize,
+            utxos: addressdata.utxos as u64,
             value: addressdata.amount(),
         };
 
         addressdata.send(value, prev_price)?;
 
         let supply_state = SupplyState {
-            utxos: addressdata.outputs_len as usize,
+            utxos: addressdata.utxos as u64,
             value: addressdata.amount(),
         };
 
@@ -72,19 +69,24 @@ impl AddressCohortState {
         Ok(())
     }
 
-    pub fn receive(&mut self, address_data: &mut AddressData, value: Sats, price: Option<Dollars>) {
+    pub fn receive(
+        &mut self,
+        address_data: &mut LoadedAddressData,
+        value: Sats,
+        price: Option<Dollars>,
+    ) {
         let compute_price = price.is_some();
 
         let prev_realized_price = compute_price.then(|| address_data.realized_price());
         let prev_supply_state = SupplyState {
-            utxos: address_data.outputs_len as usize,
+            utxos: address_data.utxos as u64,
             value: address_data.amount(),
         };
 
         address_data.receive(value, price);
 
         let supply_state = SupplyState {
-            utxos: address_data.outputs_len as usize,
+            utxos: address_data.utxos as u64,
             value: address_data.amount(),
         };
 
@@ -96,8 +98,8 @@ impl AddressCohortState {
         );
     }
 
-    pub fn add(&mut self, addressdata: &AddressData) {
-        self.address_count += 1;
+    pub fn add(&mut self, addressdata: &LoadedAddressData) {
+        self.addr_count += 1;
         self.inner.increment_(
             &addressdata.into(),
             addressdata.realized_cap,
@@ -105,8 +107,8 @@ impl AddressCohortState {
         );
     }
 
-    pub fn subtract(&mut self, addressdata: &AddressData) {
-        self.address_count = self.address_count.checked_sub(1).unwrap();
+    pub fn subtract(&mut self, addressdata: &LoadedAddressData) {
+        self.addr_count = self.addr_count.checked_sub(1).unwrap();
         self.inner.decrement_(
             &addressdata.into(),
             addressdata.realized_cap,

crates/brk_computer/src/states/cohorts/common.rs

@@ -1,44 +1,54 @@
 use std::{cmp::Ordering, path::Path};
 
-use brk_core::{CheckedSub, Dollars, Height, Result, Sats};
+use brk_error::Result;
+use brk_structs::{CheckedSub, Dollars, Height, Sats};
 
 use crate::{PriceToAmount, RealizedState, SupplyState, UnrealizedState};
 
 #[derive(Clone)]
 pub struct CohortState {
     pub supply: SupplyState,
+
     pub realized: Option<RealizedState>,
     pub satblocks_destroyed: Sats,
     pub satdays_destroyed: Sats,
 
-    price_to_amount: PriceToAmount,
+    price_to_amount: Option<PriceToAmount>,
 }
 
 impl CohortState {
-    pub fn default_and_import(path: &Path, name: &str, compute_dollars: bool) -> Result<Self> {
-        Ok(Self {
+    pub fn new(path: &Path, name: &str, compute_dollars: bool) -> Self {
+        Self {
             supply: SupplyState::default(),
             realized: compute_dollars.then_some(RealizedState::NAN),
             satblocks_destroyed: Sats::ZERO,
             satdays_destroyed: Sats::ZERO,
-            price_to_amount: PriceToAmount::forced_import(path, name),
-        })
+            price_to_amount: compute_dollars.then_some(PriceToAmount::create(path, name)),
+        }
     }
 
-    pub fn height(&self) -> Option<Height> {
-        self.price_to_amount.height()
+    pub fn import_at_or_before(&mut self, height: Height) -> Result<Height> {
+        if let Some(price_to_amount) = self.price_to_amount.as_mut() {
+            price_to_amount.import_at_or_before(height)
+        } else {
+            Ok(height)
+        }
     }
 
-    pub fn reset_price_to_amount(&mut self) -> Result<()> {
-        self.price_to_amount.reset()
+    pub fn reset_price_to_amount_if_needed(&mut self) -> Result<()> {
+        if let Some(price_to_amount) = self.price_to_amount.as_mut() {
+            price_to_amount.clean()?;
+            price_to_amount.init();
+        }
+        Ok(())
     }
 
     pub fn price_to_amount_first_key_value(&self) -> Option<(&Dollars, &Sats)> {
-        self.price_to_amount.first_key_value()
+        self.price_to_amount.as_ref().unwrap().first_key_value()
     }
 
     pub fn price_to_amount_last_key_value(&self) -> Option<(&Dollars, &Sats)> {
-        self.price_to_amount.last_key_value()
+        self.price_to_amount.as_ref().unwrap().last_key_value()
     }
 
     pub fn reset_single_iteration_values(&mut self) {
@@ -52,12 +62,15 @@ impl CohortState {
     pub fn increment(&mut self, supply_state: &SupplyState, price: Option<Dollars>) {
         self.supply += supply_state;
 
-        if supply_state.value > Sats::ZERO {
-            if let Some(realized) = self.realized.as_mut() {
-                let price = price.unwrap();
-                realized.increment(supply_state, price);
-                self.price_to_amount.increment(price, supply_state);
-            }
+        if supply_state.value > Sats::ZERO
+            && let Some(realized) = self.realized.as_mut()
+        {
+            let price = price.unwrap();
+            realized.increment(supply_state, price);
+            self.price_to_amount
+                .as_mut()
+                .unwrap()
+                .increment(price, supply_state);
         }
     }
 
@@ -69,23 +82,29 @@ impl CohortState {
     ) {
         self.supply += supply_state;
 
-        if supply_state.value > Sats::ZERO {
-            if let Some(realized) = self.realized.as_mut() {
-                realized.increment_(realized_cap);
-                self.price_to_amount.increment(realized_price, supply_state);
-            }
+        if supply_state.value > Sats::ZERO
+            && let Some(realized) = self.realized.as_mut()
+        {
+            realized.increment_(realized_cap);
+            self.price_to_amount
+                .as_mut()
+                .unwrap()
+                .increment(realized_price, supply_state);
         }
     }
 
     pub fn decrement(&mut self, supply_state: &SupplyState, price: Option<Dollars>) {
         self.supply -= supply_state;
 
-        if supply_state.value > Sats::ZERO {
-            if let Some(realized) = self.realized.as_mut() {
-                let price = price.unwrap();
-                realized.decrement(supply_state, price);
-                self.price_to_amount.decrement(price, supply_state);
-            }
+        if supply_state.value > Sats::ZERO
+            && let Some(realized) = self.realized.as_mut()
+        {
+            let price = price.unwrap();
+            realized.decrement(supply_state, price);
+            self.price_to_amount
+                .as_mut()
+                .unwrap()
+                .decrement(price, supply_state);
         }
     }
 
@@ -97,11 +116,14 @@ impl CohortState {
     ) {
         self.supply -= supply_state;
 
-        if supply_state.value > Sats::ZERO {
-            if let Some(realized) = self.realized.as_mut() {
-                realized.decrement_(realized_cap);
-                self.price_to_amount.decrement(realized_price, supply_state);
-            }
+        if supply_state.value > Sats::ZERO
+            && let Some(realized) = self.realized.as_mut()
+        {
+            realized.decrement_(realized_cap);
+            self.price_to_amount
+                .as_mut()
+                .unwrap()
+                .decrement(realized_price, supply_state);
         }
     }
 
@@ -123,21 +145,28 @@ impl CohortState {
     ) {
         self.supply += supply_state;
 
-        if supply_state.value > Sats::ZERO {
-            if let Some(realized) = self.realized.as_mut() {
-                let price = price.unwrap();
-                realized.receive(supply_state, price);
+        if supply_state.value > Sats::ZERO
+            && let Some(realized) = self.realized.as_mut()
+        {
+            let price = price.unwrap();
+            realized.receive(supply_state, price);
 
-                if let Some((price, supply)) = price_to_amount_increment
-                    && supply.value.is_not_zero()
-                {
-                    self.price_to_amount.increment(price, supply);
-                }
-                if let Some((price, supply)) = price_to_amount_decrement
-                    && supply.value.is_not_zero()
-                {
-                    self.price_to_amount.decrement(price, supply);
-                }
+            if let Some((price, supply)) = price_to_amount_increment
+                && supply.value.is_not_zero()
+            {
+                self.price_to_amount
+                    .as_mut()
+                    .unwrap()
+                    .increment(price, supply);
+            }
+
+            if let Some((price, supply)) = price_to_amount_decrement
+                && supply.value.is_not_zero()
+            {
+                self.price_to_amount
+                    .as_mut()
+                    .unwrap()
+                    .decrement(price, supply);
             }
         }
     }
@@ -194,12 +223,18 @@ impl CohortState {
                 if let Some((price, supply)) = price_to_amount_increment
                     && supply.value.is_not_zero()
                 {
-                    self.price_to_amount.increment(price, supply);
+                    self.price_to_amount
+                        .as_mut()
+                        .unwrap()
+                        .increment(price, supply);
                 }
                 if let Some((price, supply)) = price_to_amount_decrement
                     && supply.value.is_not_zero()
                 {
-                    self.price_to_amount.decrement(price, supply);
+                    self.price_to_amount
+                        .as_mut()
+                        .unwrap()
+                        .decrement(price, supply);
                 }
             }
         }
@@ -210,7 +245,7 @@ impl CohortState {
         height_price: Dollars,
         date_price: Option<Dollars>,
     ) -> (UnrealizedState, Option<UnrealizedState>) {
-        if self.price_to_amount.is_empty() {
+        if self.price_to_amount.as_ref().unwrap().is_empty() {
             return (
                 UnrealizedState::NAN,
                 date_price.map(|_| UnrealizedState::NAN),
@@ -222,18 +257,17 @@ impl CohortState {
 
         let update_state =
             |price: Dollars, current_price: Dollars, sats: Sats, state: &mut UnrealizedState| {
-                match price.cmp(&current_price) {
-                    Ordering::Equal => {
-                        state.supply_even += sats;
-                    }
-                    Ordering::Less => {
+                let cmp = price.cmp(&current_price);
+                match cmp {
+                    Ordering::Equal | Ordering::Less => {
                         state.supply_in_profit += sats;
-                        if price > Dollars::ZERO && current_price > Dollars::ZERO {
+                        if !cmp.is_eq() && price > Dollars::ZERO && current_price > Dollars::ZERO {
                             let diff = current_price.checked_sub(price).unwrap();
-                            if diff <= Dollars::ZERO {
-                                dbg!(price, current_price, diff, sats);
-                                panic!();
-                            }
+                            // Add back once in a while to verify, but generally not needed
+                            // if diff <= Dollars::ZERO {
+                            //     dbg!(price, current_price, diff, sats);
+                            //     panic!();
+                            // }
                             state.unrealized_profit += diff * sats;
                         }
                     }
@@ -241,33 +275,41 @@ impl CohortState {
                         state.supply_in_loss += sats;
                         if price > Dollars::ZERO && current_price > Dollars::ZERO {
                             let diff = price.checked_sub(current_price).unwrap();
-                            if diff <= Dollars::ZERO {
-                                dbg!(price, current_price, diff, sats);
-                                panic!();
-                            }
+                            // Add back once in a while to verify, but generally not needed
+                            // if diff <= Dollars::ZERO {
+                            //     dbg!(price, current_price, diff, sats);
+                            //     panic!();
+                            // }
                             state.unrealized_loss += diff * sats;
                         }
                     }
                 }
             };
 
-        self.price_to_amount.iter().for_each(|(&price, &sats)| {
-            update_state(price, height_price, sats, &mut height_unrealized_state);
+        self.price_to_amount
+            .as_ref()
+            .unwrap()
+            .iter()
+            .for_each(|(&price, &sats)| {
+                update_state(price, height_price, sats, &mut height_unrealized_state);
 
-            if let Some(date_price) = date_price {
-                update_state(
-                    price,
-                    date_price,
-                    sats,
-                    date_unrealized_state.as_mut().unwrap(),
-                )
-            }
-        });
+                if let Some(date_price) = date_price {
+                    update_state(
+                        price,
+                        date_price,
+                        sats,
+                        date_unrealized_state.as_mut().unwrap(),
+                    )
+                }
+            });
 
         (height_unrealized_state, date_unrealized_state)
     }
 
     pub fn commit(&mut self, height: Height) -> Result<()> {
-        self.price_to_amount.flush(height)
+        if let Some(price_to_amount) = self.price_to_amount.as_mut() {
+            price_to_amount.flush(height)?;
+        }
+        Ok(())
     }
 }

crates/brk_computer/src/states/cohorts/utxo.rs

@@ -1,6 +1,6 @@
 use std::path::Path;
 
-use brk_core::{Height, Result};
+use brk_error::Result;
 use derive_deref::{Deref, DerefMut};
 
 use super::CohortState;
@@ -9,19 +9,11 @@ use super::CohortState;
 pub struct UTXOCohortState(CohortState);
 
 impl UTXOCohortState {
-    pub fn default_and_import(path: &Path, name: &str, compute_dollars: bool) -> Result<Self> {
-        Ok(Self(CohortState::default_and_import(
-            path,
-            name,
-            compute_dollars,
-        )?))
+    pub fn new(path: &Path, name: &str, compute_dollars: bool) -> Self {
+        Self(CohortState::new(path, name, compute_dollars))
     }
 
-    pub fn height(&self) -> Option<Height> {
-        self.0.height()
-    }
-
-    pub fn reset_price_to_amount(&mut self) -> Result<()> {
-        self.0.reset_price_to_amount()
+    pub fn reset_price_to_amount_if_needed(&mut self) -> Result<()> {
+        self.0.reset_price_to_amount_if_needed()
     }
 }