release: v0.0.110

Add Docker support

.claude/prompts/changelog.md

@@ -0,0 +1,221 @@
+# Changelog Generation for Claude Code
+
+**TASK**: Update docs/CHANGELOG.md for ALL releases after and including v0.0.107.
+
+## ⚠️ CRITICAL FAILURE MODE TO AVOID ⚠️
+**THE #1 FAILURE**: Ignoring most changes and only documenting a few
+**ABSOLUTELY FORBIDDEN**: Skipping changes, summarizing with "and other updates", or being incomplete
+**YOU MUST DOCUMENT EVERY SINGLE MEANINGFUL CHANGE - NO EXCEPTIONS**
+
+## CORE WORKFLOW - EXECUTE EXACTLY:
+
+### Step 1: Get Release Information
+```bash
+git tag --list --sort=version:refname
+```
+
+### Step 2: Process ONE Release at a Time
+For each missing release, execute these commands to get complete information:
+
+**First, get the file list:**
+```bash
+git diff --name-only [previous-tag]..[current-tag]
+```
+
+**Then, get changes excluding Cargo.lock:**
+```bash
+git diff [previous-tag]..[current-tag] -- . ':(exclude)Cargo.lock'
+```
+
+**If output is too large, examine files individually:**
+```bash
+git diff [previous-tag]..[current-tag] -- path/to/specific/file.rs
+```
+
+**For summary of changes per file (if needed):**
+```bash
+git diff --stat [previous-tag]..[current-tag]
+```
+
+### Step 3: Analyze Before Writing
+**MANDATORY**: Before writing ANY changelog entry, analyze the diff output and explain:
+- **Use `git diff --name-only` to see ALL changed files** - this prevents truncation issues
+- **Use `git diff -- . ':(exclude)Cargo.lock'` to see actual changes** without Cargo.lock noise
+- **If output is large, examine key files individually** with `git diff [tags] -- path/to/file`
+- **Identify every functional change** - what new capabilities, fixes, or modifications were made
+- What each code change accomplishes functionally
+- The user-facing or system impact of modifications
+- **Which crate each change belongs to** (based on file paths)
+- How changes group together logically within and across crates
+- What functionality is added/removed/modified per crate
+
+**COMPLETENESS CHECK**: State "I have analyzed X files and identified Y distinct functional changes to document"
+
+### Step 4: Write Changelog Entry
+Only after analysis, update CHANGELOG.md with ONE release entry.
+
+**REQUIRED**: End each release entry with a comparison link:
+```markdown
+[View changes](https://github.com/bitcoinresearchkit/brk/compare/vPREVIOUS...vCURRENT)
+```
+Where PREVIOUS is the previous release tag and CURRENT is the current release tag.
+
+### Step 5: Stop and Confirm
+**CRITICAL**: Process only ONE release, then ask for confirmation to continue.
+
+---
+
+## STRICT RULES
+
+### FORBIDDEN - NEVER MENTION:
+- **Cargo.lock** (ignore completely)
+- **Line counts** ("added 50 lines", "removed 200 lines")
+- **Dependency version bumps** (unless enabling major new features)
+- **Local crate version changes** (0.0.61 → 0.0.62)
+- **Vague words**: "Enhanced", "Improved", "Updated", "Refactored"
+
+### FORBIDDEN PHRASES:
+- "and other changes"
+- "various updates"
+- "along with minor improvements"
+- "among other enhancements"
+- **"along with other modifications"**
+- **"plus additional changes"**
+- **"including other updates"**
+- **"and more"**
+- **ANY phrase that suggests you're skipping changes**
+
+### REQUIRED - MUST INCLUDE:
+- **Every meaningful change** in the diff (no shortcuts)
+- **Specific functionality** descriptions
+- **Business impact** of each change
+- **Complete coverage** - if 20 changes exist, document all 20
+- **Source links** for significant changes (new features, breaking changes, major bug fixes)
+
+## WORKSPACE-SPECIFIC RULES
+
+### Crate Identification:
+- **Determine crate from file paths** in the diff (e.g., `crates/brk-core/src/lib.rs` → `brk-core`)
+- **Group all changes by their crate** before writing changelog entries
+- **Use crate name as subheading** under each change type section
+- **For root-level files**: Use `workspace` as the crate name
+
+### Cross-Crate Changes:
+- **When changes span multiple crates** for one feature, mention the relationship
+- **Example**: "Added new transaction API in `brk-core` with corresponding HTTP endpoints in `brk-api`"
+
+### Crate Naming:
+- **Use backticks** around crate names: `brk-core`, `brk-api`
+- **Use workspace structure** as shown in file paths, not display names
+
+### File Header (if missing):
+```markdown
+<!-- This changelog was generated by Claude Code -->
+```
+
+### Release Entry Format:
+```markdown
+## [vX.Y.Z](https://github.com/bitcoinresearchkit/brk/releases/tag/vX.Y.Z) - YYYY-MM-DD
+
+### Breaking Changes
+#### `crate-name`
+- Specific change with functional impact explanation ([source](https://github.com/bitcoinresearchkit/brk/blob/vX.Y.Z/path/to/file.rs))
+
+### New Features
+#### `crate-name`
+- Feature description with user benefit ([source](https://github.com/bitcoinresearchkit/brk/blob/vX.Y.Z/path/to/main/file.rs))
+- Another feature with implementation details
+
+#### `another-crate`
+- Feature specific to this crate
+
+### Bug Fixes
+#### `crate-name`
+- Specific problem that was resolved ([source](https://github.com/bitcoinresearchkit/brk/blob/vX.Y.Z/path/to/file.rs))
+- Another bug fix with impact description
+
+### Internal Changes
+#### `crate-name`
+- Architectural improvement with purpose
+- Code organization change with benefit
+
+[View changes](https://github.com/bitcoinresearchkit/brk/compare/vPREVIOUS...vCURRENT)
+```
+
+---
+
+## ANALYSIS REQUIREMENTS
+
+**Before writing changelog text, you MUST state:**
+
+1. **File discovery**: "Using git diff --name-only, I see X files were modified"
+2. **Cargo.lock check**: "After excluding Cargo.lock, Y files contain actual changes"
+3. **Functional changes identified**: "I identified these distinct changes: A, B, C, D..."
+4. **What each change does**: "Change A: adds new API endpoints, Change B: fixes memory leak, etc."
+5. **Which crates are affected**: "Changes span crates: A, B, C"
+6. **What this means for users**: "Users can now do X, bug Y is fixed, etc."
+7. **How changes group together**: "Changes A and B work together to implement feature W"
+8. **Cross-crate dependencies**: "Crate A's new feature requires the new API in crate B"
+9. **Any unclear changes**: "I cannot determine the purpose of change X from the diff"
+10. **FINAL CHECK**: "I will now document all X functional changes without listing files"
+
+**MANDATORY**: The changelog should describe FUNCTIONALITY, not files. But you must capture ALL functionality changes.
+
+**Only after this analysis, write the changelog entry.**
+
+---
+
+## EXAMPLES
+
+### ❌ BAD (Vague, incomplete):
+```markdown
+### New Features
+- Enhanced blockchain processing capabilities
+- Improved error handling and various other updates
+```
+
+### ✅ GOOD (Specific, complete, grouped by crate, with source links):
+```markdown
+### New Features
+#### `brk-core`
+- Added `TransactionAnalyzer` struct with fee calculation and coinbase detection methods ([source](https://github.com/bitcoinresearchkit/brk/blob/v0.0.108/crates/brk-core/src/analyzer.rs))
+- Implemented in-memory caching layer for blockchain queries using HashMap storage
+
+#### `brk-api`
+- Added three new API endpoints: `/api/blocks/{hash}`, `/api/transactions/search`, and `/api/stats/network` ([source](https://github.com/bitcoinresearchkit/brk/blob/v0.0.108/crates/brk-api/src/routes.rs))
+- Implemented standardized error responses with error codes and descriptions
+
+### Bug Fixes
+#### `brk-core`
+- Fixed panic when processing blocks with zero transactions by adding explicit empty block validation ([source](https://github.com/bitcoinresearchkit/brk/blob/v0.0.108/crates/brk-core/src/block.rs))
+
+#### `brk-api`
+- Resolved memory leak in connection pool by implementing proper cleanup in Drop trait
+
+[View changes](https://github.com/bitcoinresearchkit/brk/compare/v0.0.107...v0.0.108)
+```
+
+---
+
+## SUCCESS CRITERIA
+
+✅ **COUNT files modified for internal verification only**
+✅ **Identify EVERY distinct functional change**
+✅ **Document ALL functionality changes (no "other changes")**
+✅ **Changelog describes WHAT users can do, not which files changed**
+✅ **Never mention Cargo.lock or line counts**
+✅ **Use specific, functional descriptions**
+✅ **Complete analysis before writing**
+✅ **Stop and ask for confirmation after each release**
+
+**FAILURE INDICATORS - If you do any of these, you FAILED:**
+❌ "and other changes"
+❌ "various updates"
+❌ "among other improvements"
+❌ **Diff shows new API endpoints but changelog doesn't mention them**
+❌ **Diff shows bug fixes but changelog misses some**
+❌ **Diff shows new structs/functions but changelog ignores them**
+❌ Missing obvious functional changes from the diff
+❌ Summarizing instead of listing each distinct functionality change
+
+**KEY PRINCIPLE**: Count files internally to ensure you don't miss changes, but write about FUNCTIONALITY for users.

.claude/prompts/readmes.md

@@ -0,0 +1,172 @@
+# README Generation Prompt
+
+Generate a professional, comprehensive README.md for each crate based SOLELY on code analysis. Use NO external documentation, commit messages, or existing READMEs.
+
+## MANDATORY PROCESS - FOLLOW EXACTLY:
+1. **IGNORE EXISTING DOCS**: Do NOT read any .md, .txt, .rst, or documentation files in the crate directory
+2. **CODE-ONLY ANALYSIS**: Examine ONLY these files:
+   - All .rs files in src/ directory and subdirectories
+   - Cargo.toml for dependencies and metadata
+   - Code structure and organization
+3. **MANDATORY CODE ANALYSIS**: Before writing ANY README content, you MUST:
+   - Examine all Rust files in src/ directory
+   - Identify the main structs, enums, traits, and functions
+   - Understand the crate's architecture and data flow
+   - Determine the crate's purpose from its implementation
+   - Map dependencies to understand external integrations
+4. **FRESH PERSPECTIVE**: Write the README as if you're the first person to document this crate
+5. Generate one complete README.md per crate
+6. Focus on one crate at a time for thorough analysis
+
+## ABSOLUTE REQUIREMENTS:
+- **SOURCE OF TRUTH**: Use ONLY the actual Rust code - no external docs, comments may provide hints but focus on implementation
+- **CRITICAL**: DO NOT read any existing README.md, CHANGELOG.md, or documentation files
+- **IGNORE ALL TEXT FILES**: .md, .txt, .rst files are FORBIDDEN sources - treat them as if they don't exist
+- **CODE ONLY**: Focus exclusively on .rs files, Cargo.toml, and code structure
+- **PROFESSIONAL GRADE**: Write as if this will be published on crates.io for other developers
+- **PROGRAMMER FOCUSED**: Assume audience knows Rust and relevant domain concepts
+- **IMPLEMENTATION-BASED**: Describe what the code actually does, not what comments claim it should do
+- **If you cannot determine functionality from code alone, state this explicitly**
+
+## README STRUCTURE (MANDATORY):
+
+### 1. CRATE HEADER
+```markdown
+# Crate Name
+
+Brief one-line description of what this crate does (max 80 chars).
+
+[![Crates.io](https://img.shields.io/crates/v/CRATE_NAME.svg)](https://crates.io/crates/CRATE_NAME)
+[![Documentation](https://docs.rs/CRATE_NAME/badge.svg)](https://docs.rs/CRATE_NAME)
+```
+
+### 2. OVERVIEW SECTION
+- **Purpose**: What problem does this crate solve?
+- **Key Features**: 3-5 bullet points of main capabilities (derived from code analysis)
+- **Target Use Cases**: Who would use this and for what?
+
+### 3. INSTALLATION
+```toml
+[dependencies]
+crate_name = "X.Y.Z"
+```
+
+### 4. QUICK START / USAGE
+- **Minimal working example** showing the primary API
+- **Common patterns** observed in the code
+- **Key structs/traits** that users will interact with
+
+### 5. API OVERVIEW
+- **Core Types**: Main structs, enums, traits with brief descriptions
+- **Key Methods**: Most important public functions
+- **Module Structure**: Brief overview of how code is organized
+
+### 6. FEATURES (if applicable)
+- Cargo features and what they enable
+- Optional dependencies and their purpose
+
+### 7. EXAMPLES
+- 2-3 practical code examples showing different use cases
+- Based on public API analysis, not existing examples
+
+## WRITING REQUIREMENTS:
+
+### TONE AND STYLE:
+- **Concise but comprehensive**: Every sentence must add value
+- **Technical precision**: Use exact terminology, avoid marketing speak
+- **Active voice**: "Provides X" not "X is provided"
+- **Present tense**: "The crate handles..." not "The crate will handle..."
+
+### FORBIDDEN PATTERNS:
+- **NEVER** use vague terms: "powerful", "flexible", "robust", "comprehensive", "advanced"
+- **NEVER** write marketing copy: "cutting-edge", "state-of-the-art", "enterprise-grade"
+- **NEVER** make claims you can't verify from code: "blazingly fast", "memory efficient"
+- **NEVER** copy-paste from existing documentation or comments
+- **NEVER** read or reference existing README.md files - pretend they don't exist
+- **NEVER** use phrases like "as mentioned in the documentation" or "according to the docs"
+- **NEVER** let existing documentation influence your analysis or writing
+
+### REQUIRED SPECIFICITY:
+- **Data structures**: Mention specific types (HashMap, Vec, etc.)
+- **Algorithms**: Reference actual implementations found in code
+- **Integration points**: Specific traits implemented, dependencies used
+- **Error handling**: How errors are represented and handled
+- **Async/sync**: Clearly state if operations are blocking or async
+
+### ANTI-BIAS PROTOCOL:
+
+### BEFORE STARTING ANY ANALYSIS:
+1. **Explicitly ignore**: Any README.md, CHANGELOG.md, docs/, documentation files
+2. **File filtering**: Only examine .rs and Cargo.toml files
+3. **Fresh eyes approach**: Analyze the code as if you've never seen this crate before
+4. **Independent thinking**: Form your own understanding purely from code inspection
+
+### IF YOU ACCIDENTALLY READ EXISTING DOCS:
+- Stop immediately and restart your analysis
+- Consciously disregard any information from documentation files
+- Base all descriptions solely on what you observe in the code
+- Ask yourself: "What would I think this code does if I had no documentation?"
+
+### VALIDATION CHECKS:
+- **Unique descriptions**: Your descriptions should differ significantly from any existing docs
+- **Code-derived insights**: Every feature mentioned must be visible in the source code
+- **Independent voice**: Write in your own technical style, not mimicking existing documentation
+- **Fresh examples**: Create new code examples based on API analysis, not existing samples
+
+## CODE ANALYSIS DEPTH:
+**You MUST analyze and understand:**
+1. **Public API surface**: All pub structs, functions, traits, modules
+2. **Core abstractions**: Main data types and their relationships
+3. **Error types**: Custom errors, Result patterns, panic conditions
+4. **Dependencies**: How external crates are integrated
+5. **Feature flags**: Conditional compilation and optional functionality
+6. **Async patterns**: Use of futures, tokio, async-std, etc.
+7. **Serialization**: Serde implementations, custom serialization
+8. **Performance characteristics**: Algorithm complexity where obvious
+
+### EXAMPLE STRUCTURE ANALYSIS OUTPUT:
+```markdown
+## Code Analysis Summary
+**Main Types**: `BlockProcessor`, `Transaction`, `ValidationError`
+**Core Trait**: `Validator` - implemented by `BasicValidator` and `StrictValidator`
+**Async Support**: All processing methods return `impl Future`
+**Error Handling**: Custom `ValidationError` enum with specific error types
+**Dependencies**: Uses `tokio` for async runtime, `serde` for serialization
+**Architecture**: Pipeline pattern with configurable validation stages
+```
+
+## EXAMPLES OF QUALITY:
+
+### ❌ BAD (VAGUE):
+```markdown
+# My Crate
+A powerful and flexible library for blockchain operations.
+
+## Features
+- Fast processing
+- Easy to use
+- Robust error handling
+```
+
+### ✅ GOOD (SPECIFIC):
+```markdown
+# brk-chain-analyzer
+Bitcoin blockchain analysis tools for transaction pattern detection.
+
+## Overview
+Provides utilities for analyzing Bitcoin transaction data, detecting address clustering patterns, and computing blockchain statistics. Built around a streaming parser that processes block data without loading entire blocks into memory.
+
+## Key Types
+- `TransactionAnalyzer`: Stateful analyzer for computing fees, detecting coinbase transactions
+- `ClusterDetector`: Implements common input ownership heuristics for address clustering
+- `BlockStream`: Async iterator over blockchain data with configurable batch sizes
+```
+
+## FINAL REQUIREMENTS:
+- **One README per crate** - don't combine multiple crates
+- **Minimum 200 words** - be thorough but concise
+- **Maximum 800 words** - stay focused and relevant
+- **Code examples must be syntactically correct** and compilable
+- **All claims must be verifiable** from the source code
+
+**PROCESS ONE CRATE AT A TIME. ANALYZE THE CODE THOROUGHLY BEFORE WRITING.**

.github/workflows/release.yml

@@ -1,4 +1,4 @@
-# This file was autogenerated by dist: https://opensource.axo.dev/cargo-dist/
+# This file was autogenerated by dist: https://axodotdev.github.io/cargo-dist
 #
 # Copyright 2022-2024, axodotdev
 # SPDX-License-Identifier: MIT or Apache-2.0
@@ -47,7 +47,7 @@ on:
 jobs:
   # Run 'dist plan' (or host) to determine what tasks we need to do
   plan:
-    runs-on: "ubuntu-latest"
+    runs-on: "ubuntu-22.04"
     outputs:
       val: ${{ steps.plan.outputs.manifest }}
       tag: ${{ !github.event.pull_request && github.ref_name || '' }}
@@ -58,12 +58,13 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
+          persist-credentials: false
           submodules: recursive
       - name: Install dist
         # we specify bash to get pipefail; it guards against the `curl` command
         # failing. otherwise `sh` won't catch that `curl` returned non-0
         shell: bash
-        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.28.0/cargo-dist-installer.sh | sh"
+        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.30.0/cargo-dist-installer.sh | sh"
       - name: Cache dist
         uses: actions/upload-artifact@v4
         with:
@@ -117,6 +118,7 @@ jobs:
           git config --global core.longpaths true
       - uses: actions/checkout@v4
         with:
+          persist-credentials: false
           submodules: recursive
       - name: Install Rust non-interactively if not already installed
         if: ${{ matrix.container }}
@@ -168,13 +170,14 @@ jobs:
     needs:
       - plan
       - build-local-artifacts
-    runs-on: "ubuntu-latest"
+    runs-on: "ubuntu-22.04"
     env:
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       BUILD_MANIFEST_NAME: target/distrib/global-dist-manifest.json
     steps:
       - uses: actions/checkout@v4
         with:
+          persist-credentials: false
           submodules: recursive
       - name: Install cached dist
         uses: actions/download-artifact@v4
@@ -218,12 +221,13 @@ jobs:
     if: ${{ always() && needs.plan.outputs.publishing == 'true' && (needs.build-global-artifacts.result == 'skipped' || needs.build-global-artifacts.result == 'success') && (needs.build-local-artifacts.result == 'skipped' || needs.build-local-artifacts.result == 'success') }}
     env:
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-    runs-on: "ubuntu-latest"
+    runs-on: "ubuntu-22.04"
     outputs:
       val: ${{ steps.host.outputs.manifest }}
     steps:
       - uses: actions/checkout@v4
         with:
+          persist-credentials: false
           submodules: recursive
       - name: Install cached dist
         uses: actions/download-artifact@v4
@@ -282,10 +286,11 @@ jobs:
     # still allowing individual publish jobs to skip themselves (for prereleases).
     # "host" however must run to completion, no skipping allowed!
     if: ${{ always() && needs.host.result == 'success' }}
-    runs-on: "ubuntu-latest"
+    runs-on: "ubuntu-22.04"
     env:
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     steps:
       - uses: actions/checkout@v4
         with:
+          persist-credentials: false
           submodules: recursive

.gitignore

@@ -3,6 +3,9 @@
 
 # Builds
 target
+websites/dist
+bridge/
+/ids.txt
 
 # Copies
 *\ copy*
@@ -10,9 +13,16 @@ target
 # Ignored
 _*
 
-# Editors
-.vscode
-.zed
-
 # Logs
 .log
+
+# Environment variables/configs
+.env
+
+# Profiling
+profile.json.gz
+flamegraph.svg
+*.trace
+
+# AI
+.claude/settings*

.zed/settings.json

@@ -0,0 +1,22 @@
+{
+  "file_scan_exclusions": [
+    // default
+    "**/.git",
+    "**/.svn",
+    "**/.hg",
+    "**/.jj",
+    "**/CVS",
+    "**/.DS_Store",
+    "**/Thumbs.db",
+    "**/.classpath",
+    "**/.settings",
+    // custom
+    "**/lean-qr/*/index.mjs",
+    "**/modern-screenshot/*/index.mjs",
+    "**/solidjs-signals/*/dist/prod.js",
+    "uFuzzy.mjs",
+    "lightweight-charts.standalone.production.mjs"
+    // "scripts/packages",
+    // "dist"
+  ]
+}

CHANGELOG.md

@@ -1,285 +0,0 @@
-<!--
-# v0.X.Y | WIP
-![Image of the kibo Web App version 0.X.Y](https://github.com/kibo-money/kibo/blob/main/_assets/v0.X.Y.jpg)
--->
-
-# v0.X.0 | WIP | A new beginning
-
-Full rewrite
-
-# [kibo-v0.5.0](https://github.com/kibo-money/kibo/tree/eea56d394bf92c62c81da8b78b8c47ea730683f5) | [873199](https://mempool.space/block/0000000000000000000270925aa6a565be92e13164565a3f7994ca1966e48050) - 2024/12/04
-
-![Image of the kibo Web App version 0.5.0](https://github.com/kibo-money/kibo/blob/main/_assets/v0.5.0.jpg)
-
-## Datasets
-
-- Added `Sell Side Risk Ratio` to all entities
-- Added `Open`, `High` and `Low` datasets
-- Added `Satoshis Per Dollar`
-- Added `All Time High`
-- Added `All Time High Date`
-- Added `Days Since All Time High`
-- Added `Max Days Between All Time Highs`
-- Added `Max Years Between All Time Highs`
-- Added `Drawdown`
-- Added `Adjusted Value Created`, `Adjusted Value Destroyed` and `Adjusted Spent Output Profit Ratio` to all entities
-- Added `Realized Profit To Loss Ratio` to all entities
-- Added `Hash Price Min`
-- Added `Hash Price Rebound`
-- Removed all year datasets (25) in favor for epoch datasets (5), the former was too granular to be really useful
-- Removed datasets split by liquidity for all datasets **already split by any address kind**, while fun to have, they took time to compute, ram, and space to store and no one was actually checking them
-- Fixed a lot of values in split by liquidity datasets
-
-## Website
-
-- Updated the design yet again which made the website for something more minimal and easier on the eyes
-- Added a *Save In Bitcoin* (DCA) simulation page
-- ~Added a dashboard~ Added the latest values to the tree next to each option instead, while less values are visible at a time, it's much more readable and organised
-- Added a library of PDFs
-- Fixed service worker not passing 304 (not modified) response and instead serving cached responses
-- Fixed history not being properly registered
-- Fixed window being moveable on iOS when in standalone mode when it shouldn't be
-- Added `Compare` section to all groups, to compare all datasets within a group
-- Updated `Solid Signals` library, which had an important breaking change on the `createEffect` function which might bring some bugs
-- Fixed some datasets paths
-- A lot of code reorg and file splits
-- Adopted a framework like approach to load pages while still being pure JS without a build step
-- Probably more that was forgotten
-
-## Parser
-
-- Added a `/datasets/last` json file with all the latest values
-- Added `--rpcconnect` parameter to the config
-- Added handling of SIGINT and SIGTERM terminal signals which menas you can now safely CTRL+C or kill the parser while it's exporting
-- Added config print at the start of the program
-- Compressed `empty_address_data` struct to save space (should shave of between up to 50% of the `address_index_to_empty_address_data` database)
-- Doubled the number of `txid_to_tx_data` databases from 4096 to 8192
-- ~Added `--recompute_computed true` argument, to allow recomputation of computed datasets in case of a bug~ Buggy for now
-- Fixed not saved arguments, not being processed properly
-- Fixed bug in `generic_map.multi_insert_simple_average`
-- Added defragmentation option `--first-defragment true` of databases to save space (which can save up to 50%)
-- Fixed bug in the computation of averages in `GenericMap`
-- Added support and paramer for cookie files with `--rpccookiefile`, and auto find if the path is `--datadir/.cookie`
-- Increased number of retries and time between them when fetching price from exchanges APIs
-
-## Server
-
-- Fixed links in several places missing the `/api` part and thus not working
-- Fixed broken last values routes
-- Added support for the `/datasets/last` file via the `/api/last` route
-- Added support for `.json` (won't change anything) and `.csv` (will download a csv file) extension at the end of datasets routes
-- Added `all=true` query parameter to dataset routes to get to full history
-
-## Biter
-
-- Moved back to this repo
-
-# [kibo-v0.4.0](https://github.com/kibo-money/kibo/tree/a64c544815d9ef785e2fc1323582f774f16b9200) | [861950](https://mempool.space/block/00000000000000000000530d0e30ccf7deeace122dcc99f2668a06c6dad83629) - 2024/09/19
-
-![Image of the kibo Web App version 0.4.0](https://github.com/kibo-money/kibo/blob/main/_assets/v0.4.0.jpg)
-
-## Brand
-
-- **Satonomics** is now **kibo** 🎉
-
-## Website
-
-- Complete redesign of the website
-- Rewrote the whole application and removed `node`/`npm`/`pnpm` dependencies in favor for pure `HTML`/`CSS`/`Javascript`
-- Website is now served by the server
-- Added Trading View attribution link to the settings frame and file in the lightweight charts folder
-- Many other changes
-
-## Parser
-
-- Changed the block iterator from a custom version of [bitcoin-explorer](https://crates.io/crates/bitcoin-explorer) to the homemade [biter](https://crates.io/crates/biter) which allows the parser to run alongside `bitcoind`
-- Added datasets compression thanks to [zstd](https://crates.io/crates/zstd) to reduce disk usage
-- Use the Bitcoin RPC server for various calls instead of running cli commands and then parsing the JSON from the output
-- **Important database changes that will need a full rescan**:
-  - Changed databases page size from 1MB to 4KB for improved disk usage
-  - Split txid_to_tx_data database in 4096 chunks (from 256) for improved disk usage
-  - Split address_index_to_X databases to chunks of 25_000 instead of 50_000
-  - Removed local Multisig database
-- Updated the config, run with `-h` to see possible args
-- Moved outputs from `/target/outputs` to `/out` to allow to run commands like `cargo clean` without side effects
-- Various first run fixes
-- Added to `-h` which arguments are saved, which is all of them at the time of writing
-
-## Server
-
-- Updated the code to support compressed binaries
-- Added serving of the website
-- Improved `Cache-Control` behavior
-
-# [kibo-v0.3.0](https://github.com/kibo-money/kibo/tree/b68b016091c45b071218fba01bac5b76e8eaf18c) | [853930](https://mempool.space/block/00000000000000000002eb5e9a7950ca2d5d98bd1ed28fc9098aa630d417985d) - 2024/07/26
-
-![Image of the Satonomics Web App version 0.3.0](https://github.com/kibo-money/kibo/blob/main/_assets/v0.3.0.jpg)
-
-## Parser
-
-- Global
-  - Improved self-hosting by:
-    - Fixing an incredibly annoying bug that made the program panic because of a wrong utxo/address durable state after a or many new datasets were added/changed after a first successful parse of the chain
-    - Fixing a bug that would crash the program if launched for the first time ever
-    - Auto fetch prices from the main Satonomics instance if missing instead of only trying Kraken's and Binance's API which are limited to the last 16 hours
-  - Merged the core of `HeightMap` and `DateMap` structs into `GenericMap`
-  - Added `Height` struct and many others
-  - Reorganized outputs of both the parser and the server for ease of use and easier sync compatibility
-- CLI
-  - Added an argument parser for improved UX with several options
-- Datasets
-  - Added the following datasets for all entities:
-    - Value destroyed
-    - Value created
-    - Spent Output Profit Ratio (SOPR)
-  - Added the following ratio datasets and their variations to all prices {realized, moving average, any cointime, etc}:
-    - Market Price to {X}
-    - Market Price to {X} Ratio
-    - Market Price to {X} Ratio 1 Week SMA
-    - Market Price to {X} Ratio 1 Month SMA
-    - Market Price to {X} Ratio 1 Year SMA
-    - Market Price to {X} Ratio 1 Year SMA Momentum Oscillator
-    - Market Price to {X} Ratio 99th Percentile
-    - Market Price to {X} Ratio 99.5th Percentile
-    - Market Price to {X} Ratio 99.9th Percentile
-    - Market Price to {X} Ratio 1st Percentile
-    - Market Price to {X} Ratio 0.5th Percentile
-    - {X} 1% Top Probability
-    - {X} 0.5% Top Probability
-    - {X} 0.1% Top Probability
-    - {X} 1% Bottom Probability
-    - {X} 0.5% Bottom Probability
-    - {X} 0.1% Bottom Probability
-  - Added block metadatasets and their variants (raw/sum/average/min/max/percentiles):
-    - Block size
-    - Block weight
-    - Block VBytes
-    - Block interval
-- Price
-  - Improved error message when price cannot be found
-
-## App
-
-- General
-  - Added chart scroll button for nice animations à la Wicked
-  - Added scale mode switch (Linear/Logarithmic) at the bottom right of all charts
-  - Added unit at the top left of all charts
-  - Added a backup API in case the main one fails or is offline
-  - Complete redesign of the datasets object
-  - Removed import of routes in JSON in favor for hardcoded typed routes in string format which resulted in:
-    - \+ A much lighter app
-    - \+ Better Lighthouse score
-    - \- Slower Typescript server
-  - Fixed datasets with null values crashing their fetch function
-  - Added a 'Go to a random chart' button in several places
-- Chart
-  - Fixed series color being set to default ones after hovering the legend
-  - Fixed chart starting showing candlesticks and quickly switching to a line when it should've started directly with the line
-  - Separated the QRCode generator library from the main chunk and made it imported on click
-  - Fixed timescale changing on small screen after changing charts
-- Folders
-  - Added the size in the "filename" of address cohorts grouped by size
-- Favorites
-  - Added a 'favorite' and 'unfavorite' button at the bottom
-- Settings
-  - Removed the horizontal scroll bar which was unintended
-
-## Server
-
-- Run file
-  - Only run with a watcher if `cargo watch` is available
-  - Removed id_to_path file in favor for only `paths.d.ts` in `app/src/types`
-
-# [kibo-v0.2.0](https://github.com/kibo-money/kibo/tree/248187889283597c5dbb806292297453c25e97b8) | [851286](https://mempool.space/block/0000000000000000000281ca7f1bf8c50702bfca168c7af1bdc67c977c1ac8ed) - 2024/07/08
-
-![Image of the Satonomics Web App version 0.2.0](https://github.com/kibo-money/kibo/blob/main/_assets/v0.2.0.jpg)
-
-## App
-
-- General
-  - Added the height version of all datasets and many optimizations to make them usable but only available on desktop and tablets for now
-  - Added a light theme
-- Charts
-  - Added split panes in order to have the vertical axis visible for all datasets
-  - Added min and max values on the charts
-  - Fixed legend hovering on mobile not resetting on touch end
-  - Added "3 months" and yearly time scale setters (from year 2009 to today)
-  - Hide scrollbar of timescale setters and instead added scroll buttons to the legend only visible on desktop
-  - Improved Share/QR Code screen
-  - Changed all Area series to Line series
-  - Fixed horizontal scrollable legend not updating on preset change
-- Performance
-  - Improved app's reactivity
-  - Added some chunk splitting for a faster initial load
-  - Global improvements that increased the Lighthouse's performance score
-- Settings
-  - Finally made a proper component where you can chose the app's theme, between a moving or static background and its text opacity
-  - Added donations section with a leaderboard
-  - Added various links that are visible on the bottom side of the strip on desktop to mobile users
-  - Added install instructions when not installed for Apple users
-- Misc
-  - Support mini window size, could be useful for embedded views
-  - Hopefully made scrollbars a little more subtle on WIndows and Linux, can't test
-  - Generale style updates
-
-## Parser
-
-- Fixed ulimit only being run in Mac OS instead of whenever the program is detected
-
-# [kibo-v0.1.1](https://github.com/kibo-money/kibo/tree/e55b5195a9de9aea306903c94ed63cb1720fda5f) | [849240](https://mempool.space/block/000000000000000000002b8653988655071c07bb5f7181c038f9326bc86db741) - 2024/06/24
-
-![Image of the Satonomics Web App version 0.1.1](https://github.com/kibo-money/kibo/blob/main/_assets/v0.1.1.jpg)
-
-## Parser
-
-- Fixed overflow in `Price` struct which caused many Realized Caps and Realized Prices to have completely bogus data
-- Fixed Realized Cap computation which was using rounded prices instead normal ones
-
-## Server
-
-- Added the chunk, date and time of the request to the terminal logs
-
-## App
-
-- Chart
-  - Added double click option on a legend to toggle the visibility of all other series
-  - Added highlight effect to a legend by darkening the color of all the other series on the chart while hovering it with the mouse
-  - Added an API link in the legend for each dataset where applicable (when isn't generated locally)
-  - Save fullscreen preference in local storage and url
-  - Improved resize bar on desktop
-  - Changed resize button logo
-  - Changed the share button to visible on small screen too
-  - Improved share screen
-  - Fixed time range shifting not being the one in url params or saved in local storage
-  - Fixed time range shifting on series toggling via the legend
-  - Fixed time range shifting on fullscreen
-  - Fixed time range shifting on resize of the sidebar
-  - Set default view at first load to last 6 months
-  - Added some padding around the datasets (year 1970 to 2100)
-- History
-  - Changed background for the sticky dates from blur to a solid color as it didn't appear properly in Firefox
-- Build
-  - Tried to add lazy loads to have split chunks after build, to have much faster load times and they worked great ! But they completely broke Safari on iOS, we can't have nice things
-  - Removed many libraries and did some things manually instead to improve build size
-- Strip
-  - Temporarily removed the Home button on the strip bar on desktop as there is no landing page yet
-- Settings
-  - Added version
-- PWA
-  - Fixed background update
-  - Changed update check frequency to 1 minute (~1kb to fetch every minute which is very reasonable)
-  - Added a nice banner to ask the user to install the update
-- Misc
-  - Removed tracker even though it was a very privacy friendly as it appeared to not be working properly
-
-## Price
-
-- Deleted old price datasets and their backups
-
-# [kibo-v0.1.0](https://github.com/kibo-money/kibo/tree/a1a576d088c8f83ed32d48753a7611f70a964574) | [848642](https://mempool.space/block/000000000000000000020be5761d70751252219a9557f55e91ecdfb86c4e026a) - 2024/06/19
-
-![Image of the Satonomics Web App version 0.1.0](https://github.com/kibo-money/kibo/blob/main/_assets/v0.1.0.jpg)
-
-# kibo-v0.0.1 | [835444](https://mempool.space/block/000000000000000000009f93907a0dd83c080d5585cc7ec82c076d45f6d7c872) - 2024/03/20
-
-![Image of the Satonomics Web App version 0.0.X](https://github.com/kibo-money/kibo/blob/main/_assets/v0.0.X.jpg)

Cargo.toml

@@ -4,55 +4,81 @@ members = ["crates/*"]
 package.description = "The Bitcoin Research Kit is a suite of tools designed to extract, compute and display data stored on a Bitcoin Core node"
 package.license = "MIT"
 package.edition = "2024"
-package.version = "0.0.56"
+package.version = "0.0.110"
 package.homepage = "https://bitcoinresearchkit.org"
 package.repository = "https://github.com/bitcoinresearchkit/brk"
+package.readme = "README.md"
+package.rust-version = "1.89"
+
+[profile.dev]
+lto = "thin"
+codegen-units = 16
+opt-level = 2
+split-debuginfo = "unpacked"
 
 [profile.release]
 lto = "fat"
 codegen-units = 1
 panic = "abort"
+strip = true
+overflow-checks = false
+
+[profile.profiling]
+inherits = "release"
+debug = true
 
 [profile.dist]
 inherits = "release"
 
+[profile.clippy]
+inherits = "dev"
+lto = "off"
+codegen-units = 256
+opt-level = 0
+debug = false
+overflow-checks = false
+panic = "abort"
+debug-assertions = false
+
 [workspace.dependencies]
-arc-swap = "1.7.1"
-axum = "0.8.4"
-bincode = { version = "2.0.1", features = ["serde"] }
-bitcoin = { version = "0.32.6", features = ["serde"] }
+allocative = { version = "0.3.4", features = ["parking_lot"] }
+axum = "0.8.6"
+bitcoin = { version = "0.32.7", features = ["serde"] }
 bitcoincore-rpc = "0.19.0"
-brk_cli = { version = "0.0.56", path = "crates/brk_cli" }
-brk_computer = { version = "0.0.56", path = "crates/brk_computer" }
-brk_core = { version = "0.0.56", path = "crates/brk_core" }
-brk_exit = { version = "0.0.56", path = "crates/brk_exit" }
-brk_fetcher = { version = "0.0.56", path = "crates/brk_fetcher" }
-brk_indexer = { version = "0.0.56", path = "crates/brk_indexer" }
-brk_logger = { version = "0.0.56", path = "crates/brk_logger" }
-brk_parser = { version = "0.0.56", path = "crates/brk_parser" }
-brk_query = { version = "0.0.56", path = "crates/brk_query" }
-brk_server = { version = "0.0.56", path = "crates/brk_server" }
-brk_state = { version = "0.0.56", path = "crates/brk_state" }
-brk_store = { version = "0.0.56", path = "crates/brk_store" }
-brk_vec = { version = "0.0.56", path = "crates/brk_vec" }
+brk_binder = { version = "0.0.110", path = "crates/brk_binder" }
+brk_bundler = { version = "0.0.110", path = "crates/brk_bundler" }
+brk_cli = { version = "0.0.110", path = "crates/brk_cli" }
+brk_computer = { version = "0.0.110", path = "crates/brk_computer" }
+brk_error = { version = "0.0.110", path = "crates/brk_error" }
+brk_fetcher = { version = "0.0.110", path = "crates/brk_fetcher" }
+brk_indexer = { version = "0.0.110", path = "crates/brk_indexer" }
+brk_interface = { version = "0.0.110", path = "crates/brk_interface" }
+brk_logger = { version = "0.0.110", path = "crates/brk_logger" }
+brk_mcp = { version = "0.0.110", path = "crates/brk_mcp" }
+brk_parser = { version = "0.0.110", path = "crates/brk_parser" }
+brk_server = { version = "0.0.110", path = "crates/brk_server" }
+brk_store = { version = "0.0.110", path = "crates/brk_store" }
+brk_structs = { version = "0.0.110", path = "crates/brk_structs" }
 byteview = "=0.6.1"
-clap = { version = "4.5.40", features = ["string"] }
-clap_derive = "4.5.40"
-color-eyre = "0.6.5"
 derive_deref = "1.1.1"
-fjall = "2.11.0"
-jiff = "0.2.14"
-log = { version = "0.4.27" }
-minreq = { version = "2.13.4", features = ["https", "serde_json"] }
-rayon = "1.10.0"
-serde = { version = "1.0.219" }
-serde_bytes = "0.11.17"
-serde_derive = "1.0.219"
-serde_json = { version = "1.0.140", features = ["float_roundtrip"] }
-tabled = "0.20.0"
-tokio = { version = "1.45.1", features = ["rt-multi-thread"] }
-zerocopy = { version = "0.8.25" }
-zerocopy-derive = "0.8.25"
+fjall = "2.11.2"
+jiff = "0.2.15"
+log = "0.4.28"
+minreq = { version = "2.14.1", features = ["https", "serde_json"] }
+parking_lot = "0.12.4"
+quick_cache = "0.6.16"
+rayon = "1.11.0"
+schemars = "1.0.4"
+serde = "1.0.228"
+serde_bytes = "0.11.19"
+serde_derive = "1.0.228"
+serde_json = { version = "1.0.145", features = ["float_roundtrip"] }
+sonic-rs = "0.5.5"
+tokio = { version = "1.47.1", features = ["rt-multi-thread"] }
+# vecdb = { path = "../seqdb/crates/vecdb", features = ["derive"]}
+vecdb = { version = "0.2.16", features = ["derive"]}
+zerocopy = "0.8.27"
+zerocopy-derive = "0.8.27"
 
 [workspace.metadata.release]
 shared-version = true
@@ -61,20 +87,8 @@ pre-release-commit-message = "release: v{{version}}"
 tag-message = "release: v{{version}}"
 
 [workspace.metadata.dist]
-cargo-dist-version = "0.28.0"
+cargo-dist-version = "0.30.0"
 ci = "github"
+allow-dirty = ["ci"]
 installers = []
-targets = [
-    "aarch64-apple-darwin",
-    "aarch64-unknown-linux-gnu",
-    "x86_64-apple-darwin",
-    "x86_64-unknown-linux-gnu",
-]
-
-[workspace.metadata.dist.github-custom-runners]
-global = "ubuntu-latest"
-aarch64-apple-darwin.runner = "macos-14"
-x86_64-unknown-linux-gnu.runner = "ubuntu-latest"
-x86_64-unknown-linux-gnu.container = { image = "quay.io/pypa/manylinux_2_28_x86_64", host = "x86_64-unknown-linux-musl" }
-aarch64-unknown-linux-gnu.runner = "ubuntu-latest"
-aarch64-unknown-linux-gnu.container = { image = "quay.io/pypa/manylinux_2_28_x86_64", host = "x86_64-unknown-linux-musl" }
+targets = ["aarch64-apple-darwin", "aarch64-unknown-linux-gnu", "x86_64-unknown-linux-gnu"]

README.md

@@ -1,102 +0,0 @@
-# Bitcoin Research Kit
-
-<p align="left">
-  <a href="https://github.com/bitcoinresearchkit/brk">
-    <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
-  </a>
-  <a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
-    <img src="https://img.shields.io/crates/l/brk" alt="License" />
-  </a>
-  <a href="https://crates.io/crates/brk">
-    <img src="https://img.shields.io/crates/v/brk" alt="Version" />
-  </a>
-  <a href="https://docs.rs/brk">
-    <img src="https://img.shields.io/docsrs/brk" alt="Documentation" />
-  </a>
-  <img src="https://img.shields.io/crates/size/brk" alt="Size" />
-  <a href="https://deps.rs/crate/brk">
-    <img src="https://deps.rs/crate/brk/latest/status.svg" alt="Dependency status">
-  </a>
-  <a href="https://discord.gg/HaR3wpH3nr">
-    <img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
-  </a>
-  <a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
-    <img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
-  </a>
-  <a href="https://bsky.app/profile/bitcoinresearchkit.org">
-    <img src="https://img.shields.io/badge/bluesky-blue?link=https%3A%2F%2Fbsky.app%2Fprofile%2Fbitcoinresearchkit.org" alt="Bluesky" />
-  </a>
-  <a href="https://x.com/brkdotorg">
-    <img src="https://img.shields.io/badge/x.com-black" alt="X" />
-  </a>
-</p>
-
-The Bitcoin Research Kit is a high-performance toolchain designed to parse, index, compute, serve and visualize data from a Bitcoin node, enabling users to gain deeper insights into the Bitcoin network.
-
-In other words it's an alternative to [Glassnode](https://glassnode.com), [mempool.space](https://mempool.space/) (soon) and [electrs](https://github.com/romanz/electrs) (soon) all in one package with a particular focus on simplicity and ease of use.
-
-The toolkit can be used in various ways to accommodate as many needs as possible:
-
-- **[Website](https://bitcoinresearchkit.org)** \
-  Everyone is welcome to visit the official instance and showcase of the suite's capabilities. \
-  It has a wide range of functionalities including charts, tables and simulations which you can visit for free and without the need for an account. \
-  Also available at: [kibo.money](https://kibo.money) // [satonomics.xyz](https://satonomics.xyz)
-- **[API](https://github.com/bitcoinresearchkit/brk/tree/main/crates/brk_server#endpoints)** \
-  Researchers and developers are free to use BRK's public API with ![Datasets variant count](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fbitcoinresearchkit.org%2Fapi%2Fvecs%2Fvariant-count&query=%24&style=flat&label=%20&color=white) dataset variants at your disposal. \
-  Just like the website, it's entirely free, with no authentication or rate-limiting.
-- **[CLI](https://crates.io/crates/brk_cli)** \
-  Node runners are strongly encouraged to try out and self-host their own instance using BRK's command line interface. \
-  The CLI has multiple cogs available for users to tweak to adapt to all situations with even the possibility for web developers to create their own custom website which could later on be added as an alternative front-end.
-- **[Crates](https://crates.io/crates/brk)** \
-  Rust developers have access to a wide range crates, each built upon one another with its own specific purpose, enabling independent use and offering great flexibility.
-  PRs are welcome, especially if their goal is to introduce additional datasets.
-
-The primary goal of this project is to be fully-featured and accessible for everyone, regardless of their background or financial situation - whether that person is an enthusiast, researcher, miner, analyst, or simply curious.
-
-In contrast, existing alternatives tend to be either [very costly](https://studio.glassnode.com/pricing) or missing essential features, with the vast majority being closed-source and unverifiable, which fundamentally undermines the principles of Bitcoin.
-
-## Crates
-
-- [`brk`](https://crates.io/crates/brk): Wrapper around all other `brk-*` crates
-- [`brk_cli`](https://crates.io/crates/brk_cli): A standalone command line interface to interact with the Bitcoin Research Kit
-- [`brk_computer`](https://crates.io/crates/brk_computer): A Bitcoin dataset computer, built on top of brk_indexer
-- [`brk_core`](https://crates.io/crates/brk_core): The Core (Structs and Errors) of the Bitcoin Research Kit
-- [`brk_exit`](https://crates.io/crates/brk_exit): An exit blocker built on top of ctrlc
-- [`brk_fetcher`](https://crates.io/crates/brk_fetcher): A Bitcoin price fetcher
-- [`brk_indexer`](https://crates.io/crates/brk_indexer): A Bitcoin Core indexer built on top of brk_parser
-- [`brk_logger`](https://crates.io/crates/brk_logger): A clean logger used in the Bitcoin Research Kit.
-- [`brk_parser`](https://crates.io/crates/brk_parser): A very fast Bitcoin Core block parser and iterator built on top of bitcoin-rust
-- [`brk_query`](https://crates.io/crates/brk_query): A library that finds requested datasets.
-- [`brk_server`](https://crates.io/crates/brk_server): A server that serves Bitcoin data and swappable front-ends, built on top of `brk_indexer`, `brk_fetcher` and `brk_computer`
-- [`brk_state`](https://crates.io/crates/brk_state): Various states used mainly by the computer
-- [`brk_store`](https://crates.io/crates/brk_store): A thin wrapper around [`fjall`](https://crates.io/crates/fjall)
-- [`brk_vec`](https://crates.io/crates/brk_vec): A push-only, truncable, compressable, saveable Vec
-
-## Hosting as a service
-
-If you'd like to have your own instance hosted for you please contact [hosting@bitcoinresearchkit.org](mailto:hosting@bitcoinresearchkit.org).
-
-- 2 separate dedicated servers (1 GB/s each) with different ISPs and Cloudflare integration for enhanced performance and optimal availability
-- 99.99% SLA
-- Configured for speed
-- Updates delivered at your convenience
-- Direct communication for feature requests and support
-- Bitcoin Core or Knots with desired version
-- Optional subdomains: `*.bitcoinresearchkit.org`, `*.kibo.money` and `*.satonomics.xyz`
-- Logo featured in the Readme if desired
-
-Pricing: `0.01 BTC / month` *or* `0.1 BTC / year`
-
-## Acknowledgments
-
-Deepest gratitude to the [Open Sats](https://opensats.org/) public charity. Their grant — from December 2024 to the present — has been critical in sustaining this project.
-
-Heartfelt thanks go out to every donor on [Nostr](https://primal.net/p/npub1jagmm3x39lmwfnrtvxcs9ac7g300y3dusv9lgzhk2e4x5frpxlrqa73v44) and [Geyser.fund](https://geyser.fund/project/brk) whose support has ensured the availability of the [kibo.money](https://kibo.money) public instance.
-
-## Donate
-
-[`bc1q09 8zsm89 m7kgyz e338vf ejhpdt 92ua9p 3peuve`](bitcoin:bc1q098zsm89m7kgyze338vfejhpdt92ua9p3peuve)
-
-[`lnurl1dp68gurn8ghj7ampd3kx2ar0veekzar0wd5xjtnrdakj7tnhv4kxctttdehhwm30d3h82unvwqhkxmmww3jkuar8d35kgetj8yuq363hv4`](lightning:lnurl1dp68gurn8ghj7ampd3kx2ar0veekzar0wd5xjtnrdakj7tnhv4kxctttdehhwm30d3h82unvwqhkxmmww3jkuar8d35kgetj8yuq363hv4)
-
-[Geyser Fund](https://geyser.fund/project/brk)

crates/brk/Cargo.toml

@@ -2,54 +2,58 @@
 name = "brk"
 description.workspace = true
 license.workspace = true
-readme.workspace = true
 homepage.workspace = true
 repository.workspace = true
 edition.workspace = true
 version.workspace = true
+rust-version.workspace = true
+build = "build.rs"
 
 [features]
 full = [
-    "core",
+    "binder",
+    "bundler",
     "computer",
-    "exit",
+    "error",
     "fetcher",
     "indexer",
+    "interface",
     "logger",
+    "mcp",
     "parser",
-    "query",
     "server",
-    "state",
     "store",
-    "vec",
+    "structs",
 ]
-core = ["brk_core"]
+binder = ["brk_binder"]
+bundler = ["brk_bundler"]
 computer = ["brk_computer"]
-exit = ["brk_exit"]
+error = ["brk_error"]
 fetcher = ["brk_fetcher"]
 indexer = ["brk_indexer"]
+interface = ["brk_interface"]
 logger = ["brk_logger"]
+mcp = ["brk_mcp"]
 parser = ["brk_parser"]
-query = ["brk_query"]
 server = ["brk_server"]
-state = ["brk_state"]
 store = ["brk_store"]
-vec = ["brk_vec"]
+structs = ["brk_structs"]
 
 [dependencies]
+brk_binder = { workspace = true, optional = true }
+brk_bundler = { workspace = true, optional = true }
 brk_cli = { workspace = true }
-brk_core = { workspace = true, optional = true }
 brk_computer = { workspace = true, optional = true }
-brk_exit = { workspace = true, optional = true }
+brk_error = { workspace = true, optional = true }
 brk_fetcher = { workspace = true, optional = true }
 brk_indexer = { workspace = true, optional = true }
+brk_interface = { workspace = true, optional = true }
 brk_logger = { workspace = true, optional = true }
+brk_mcp = { workspace = true, optional = true }
 brk_parser = { workspace = true, optional = true }
-brk_query = { workspace = true, optional = true }
 brk_server = { workspace = true, optional = true }
-brk_state = { workspace = true, optional = true }
 brk_store = { workspace = true, optional = true }
-brk_vec = { workspace = true, optional = true }
+brk_structs = { workspace = true, optional = true }
 
 [package.metadata.docs.rs]
 all-features = true

crates/brk/README.md

@@ -0,0 +1,273 @@
+# brk
+
+Unified Bitcoin Research Kit crate providing optional feature-gated access to all BRK components.
+
+[![Crates.io](https://img.shields.io/crates/v/brk.svg)](https://crates.io/crates/brk)
+[![Documentation](https://docs.rs/brk/badge.svg)](https://docs.rs/brk)
+
+## Overview
+
+This crate serves as a unified entry point to the Bitcoin Research Kit ecosystem, providing feature-gated re-exports of all BRK components. It allows users to selectively include only the functionality they need while maintaining a single dependency declaration, with the `brk_cli` component always available for command-line interface access.
+
+**Key Features:**
+
+- Feature-gated modular access to 12 specialized BRK components
+- Single dependency entry point with selective compilation
+- Always-available CLI component for command-line operations
+- Comprehensive documentation aggregation with inline re-exports
+- `full` feature for complete BRK functionality inclusion
+- Optimized build configuration with docs.rs integration
+
+**Target Use Cases:**
+
+- Applications requiring selective BRK functionality to minimize dependencies
+- Library development where only specific Bitcoin analysis components are needed
+- Prototyping and experimentation with different BRK component combinations
+- Educational use cases demonstrating modular blockchain analytics architecture
+
+## Installation
+
+```toml
+[dependencies]
+# Minimal installation with CLI only
+brk = "0.0.107"
+
+# Full functionality
+brk = { version = "0.0.107", features = ["full"] }
+
+# Selective features
+brk = { version = "0.0.107", features = ["indexer", "computer", "server"] }
+```
+
+## Quick Start
+
+```rust
+// CLI is always available
+use brk::cli;
+
+// Feature-gated components
+#[cfg(feature = "indexer")]
+use brk::indexer::Indexer;
+
+#[cfg(feature = "computer")]
+use brk::computer::Computer;
+
+#[cfg(feature = "server")]
+use brk::server::Server;
+
+// Build complete pipeline with selected features
+#[cfg(all(feature = "indexer", feature = "computer", feature = "server"))]
+fn build_pipeline() -> Result<(), Box<dyn std::error::Error>> {
+    let indexer = Indexer::build("./data")?;
+    let computer = Computer::build("./analytics", &indexer)?;
+    let interface = brk::interface::Interface::build(&indexer, &computer);
+    let server = Server::new(interface, None);
+    Ok(())
+}
+```
+
+## API Overview
+
+### Feature Organization
+
+The crate provides feature-gated access to BRK components organized by functionality:
+
+**Core Data Processing:**
+- `structs` - Bitcoin-aware data structures and type system
+- `error` - Centralized error handling across components
+- `store` - Transactional key-value storage wrapper
+
+**Blockchain Processing:**
+- `parser` - Multi-threaded Bitcoin block parsing
+- `indexer` - Blockchain data indexing with columnar storage
+- `computer` - Analytics computation engine
+
+**Data Access:**
+- `interface` - Unified query interface with fuzzy search
+- `fetcher` - Multi-source price data aggregation
+
+**Service Layer:**
+- `server` - HTTP API server with caching and compression
+- `mcp` - Model Context Protocol bridge for LLM integration
+- `logger` - Enhanced logging with colored output
+
+**Web Infrastructure:**
+- `bundler` - Web asset bundling using Rolldown
+
+### Always Available
+
+**`cli`**: Command-line interface module (no feature gate required)
+Provides access to the complete BRK command-line interface for running full instances.
+
+## Examples
+
+### Minimal Bitcoin Parser
+
+```rust
+use brk::parser::Parser;
+
+fn parse_blocks() -> Result<(), Box<dyn std::error::Error>> {
+    #[cfg(feature = "parser")]
+    {
+        let parser = Parser::new("/path/to/blocks", None, rpc_client);
+        // Parse blockchain data
+        Ok(())
+    }
+    #[cfg(not(feature = "parser"))]
+    {
+        Err("Parser feature not enabled".into())
+    }
+}
+```
+
+### Analytics Pipeline
+
+```rust
+use brk::{indexer, computer, interface};
+
+#[cfg(all(feature = "indexer", feature = "computer", feature = "interface"))]
+fn analytics_pipeline() -> Result<(), Box<dyn std::error::Error>> {
+    // Initialize indexer
+    let indexer = indexer::Indexer::build("./blockchain_data")?;
+
+    // Compute analytics
+    let computer = computer::Computer::build("./analytics", &indexer)?;
+
+    // Create query interface
+    let interface = interface::Interface::build(&indexer, &computer);
+
+    // Query latest price
+    let params = interface::Params {
+        index: "date".to_string(),
+        ids: vec!["price-close".to_string()].into(),
+        from: Some(-1),
+        ..Default::default()
+    };
+
+    let result = interface.search_and_format(params)?;
+    println!("Latest Bitcoin price: {:?}", result);
+
+    Ok(())
+}
+```
+
+### Web Server Setup
+
+```rust
+use brk::{server, interface, indexer, computer};
+
+#[cfg(all(feature = "server", feature = "interface", feature = "indexer", feature = "computer"))]
+async fn web_server() -> Result<(), Box<dyn std::error::Error>> {
+    let indexer = indexer::Indexer::build("./data")?;
+    let computer = computer::Computer::build("./analytics", &indexer)?;
+    let interface = interface::Interface::build(&indexer, &computer);
+
+    let server = server::Server::new(interface, None);
+    server.serve(true).await?;
+
+    Ok(())
+}
+```
+
+### MCP Integration
+
+```rust
+use brk::{mcp, interface, indexer, computer};
+
+#[cfg(all(feature = "mcp", feature = "interface"))]
+fn mcp_server(interface: &'static interface::Interface) -> mcp::MCP {
+    mcp::MCP::new(interface)
+}
+```
+
+## Feature Combinations
+
+### Common Combinations
+
+**Data Processing**: `["structs", "parser", "indexer"]`
+Basic blockchain data processing and indexing.
+
+**Analytics**: `["indexer", "computer", "fetcher", "interface"]`
+Complete analytics pipeline with price data integration.
+
+**API Server**: `["interface", "server", "logger"]`
+HTTP API server with logging capabilities.
+
+**Full Stack**: `["full"]`
+All components for complete BRK functionality.
+
+### Dependency Optimization
+
+Feature selection allows for significant dependency reduction:
+
+```toml
+# Minimal parser-only dependency
+brk = { version = "0.0.107", features = ["parser"] }
+
+# Analytics without web server
+brk = { version = "0.0.107", features = ["indexer", "computer", "interface"] }
+
+# Web server without parsing
+brk = { version = "0.0.107", features = ["interface", "server"] }
+```
+
+## Architecture
+
+### Re-export Pattern
+
+The crate uses `#[doc(inline)]` re-exports to provide seamless access to component APIs:
+
+```rust
+#[cfg(feature = "component")]
+#[doc(inline)]
+pub use brk_component as component;
+```
+
+This pattern ensures:
+- Feature-gated compilation for dependency optimization
+- Inline documentation for unified API reference
+- Namespace preservation for component-specific functionality
+
+### Build Configuration
+
+- **Documentation**: `all-features = true` for complete docs.rs documentation
+- **CLI Integration**: `brk_cli` always available without feature gates
+- **Optional Dependencies**: All components except CLI are optional
+
+## Configuration
+
+### Feature Flags
+
+| Feature | Component | Description |
+|---------|-----------|-------------|
+| `bundler` | `brk_bundler` | Web asset bundling |
+| `computer` | `brk_computer` | Analytics computation |
+| `error` | `brk_error` | Error handling |
+| `fetcher` | `brk_fetcher` | Price data fetching |
+| `indexer` | `brk_indexer` | Blockchain indexing |
+| `interface` | `brk_interface` | Data query interface |
+| `logger` | `brk_logger` | Enhanced logging |
+| `mcp` | `brk_mcp` | Model Context Protocol |
+| `parser` | `brk_parser` | Block parsing |
+| `server` | `brk_server` | HTTP server |
+| `store` | `brk_store` | Key-value storage |
+| `structs` | `brk_structs` | Data structures |
+| `full` | All components | Complete functionality |
+
+### Documentation
+
+Documentation is aggregated from all components with `#![doc = include_str!("../README.md")]` ensuring comprehensive API reference across all features.
+
+## Code Analysis Summary
+
+**Main Structure**: Feature-gated re-export crate providing unified access to 12 BRK components \
+**Feature System**: Cargo features enabling selective compilation and dependency optimization \
+**CLI Integration**: Always-available `brk_cli` access without feature requirements \
+**Documentation**: Inline re-exports with comprehensive docs.rs integration \
+**Dependency Management**: Optional dependencies for all components except CLI \
+**Build Configuration**: Optimized compilation with all-features documentation \
+**Architecture**: Modular aggregation crate enabling flexible BRK ecosystem usage
+
+---
+
+_This README was generated by Claude Code_

crates/brk/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    let profile = std::env::var("PROFILE").unwrap_or_default();
+
+    if profile == "release" {
+        println!("cargo:rustc-flag=-C");
+        println!("cargo:rustc-flag=target-cpu=native");
+    }
+}

crates/brk/flamegraph.sh

@@ -0,0 +1 @@
+sudo cargo flamegraph --profile profiling --root

crates/brk/samply.sh

@@ -0,0 +1,2 @@
+cargo build --profile profiling
+samply record ../../target/profiling/brk

crates/brk/src/lib.rs

@@ -1,16 +1,27 @@
-#![doc = include_str!(concat!("../", env!("CARGO_PKG_README")))]
+#![doc = include_str!("../README.md")]
 
-#[cfg(feature = "core")]
+#[cfg(feature = "binder")]
 #[doc(inline)]
-pub use brk_core as core;
+pub use brk_binder as binder;
+
+#[cfg(feature = "bundler")]
+#[doc(inline)]
+pub use brk_bundler as bundler;
+
+#[doc(inline)]
+pub use brk_cli as cli;
+
+#[cfg(feature = "structs")]
+#[doc(inline)]
+pub use brk_structs as structs;
 
 #[cfg(feature = "computer")]
 #[doc(inline)]
 pub use brk_computer as computer;
 
-#[cfg(feature = "exit")]
+#[cfg(feature = "error")]
 #[doc(inline)]
-pub use brk_exit as exit;
+pub use brk_error as error;
 
 #[cfg(feature = "fetcher")]
 #[doc(inline)]
@@ -24,26 +35,22 @@ pub use brk_indexer as indexer;
 #[doc(inline)]
 pub use brk_logger as logger;
 
+#[cfg(feature = "mcp")]
+#[doc(inline)]
+pub use brk_mcp as mcp;
+
 #[cfg(feature = "parser")]
 #[doc(inline)]
 pub use brk_parser as parser;
 
-#[cfg(feature = "query")]
+#[cfg(feature = "interface")]
 #[doc(inline)]
-pub use brk_query as query;
+pub use brk_interface as interface;
 
 #[cfg(feature = "server")]
 #[doc(inline)]
 pub use brk_server as server;
 
-#[cfg(feature = "state")]
-#[doc(inline)]
-pub use brk_state as state;
-
 #[cfg(feature = "store")]
 #[doc(inline)]
 pub use brk_store as store;
-
-#[cfg(feature = "vec")]
-#[doc(inline)]
-pub use brk_vec as vec;

crates/brk_binder/Cargo.toml

@@ -0,0 +1,14 @@
+[package]
+name = "brk_binder"
+description = "A generator of binding files for other languages"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+homepage.workspace = true
+repository.workspace = true
+rust-version.workspace = true
+build = "build.rs"
+
+[dependencies]
+brk_interface = { workspace = true }
+brk_structs = { workspace = true }

crates/brk_binder/README.md

@@ -0,0 +1 @@
+# brk_binder

crates/brk_binder/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    let profile = std::env::var("PROFILE").unwrap_or_default();
+
+    if profile == "release" {
+        println!("cargo:rustc-flag=-C");
+        println!("cargo:rustc-flag=target-cpu=native");
+    }
+}

crates/brk_binder/src/js.rs

@@ -0,0 +1,250 @@
+use std::{
+    collections::{BTreeMap, HashMap},
+    fs, io,
+    path::Path,
+};
+
+use brk_interface::{Index, Interface};
+use brk_structs::pools;
+
+use super::VERSION;
+
+const AUTO_GENERATED_DISCLAIMER: &str = "//
+// File auto-generated, any modifications will be overwritten
+//";
+
+#[allow(clippy::upper_case_acronyms)]
+pub trait Bridge {
+    fn generate_js_files(&self, modules_path: &Path) -> io::Result<()>;
+}
+
+impl Bridge for Interface<'static> {
+    fn generate_js_files(&self, modules_path: &Path) -> io::Result<()> {
+        let path = modules_path.join("brk-client");
+
+        if !fs::exists(&path)? {
+            return Ok(());
+        }
+
+        let path = path.join("generated");
+        fs::create_dir_all(&path)?;
+
+        generate_version_file(&path)?;
+        generate_metrics_file(self, &path)?;
+        generate_pools_file(&path)
+    }
+}
+
+fn generate_version_file(parent: &Path) -> io::Result<()> {
+    let path = parent.join(Path::new("version.js"));
+
+    let contents = format!(
+        "{AUTO_GENERATED_DISCLAIMER}
+
+export const VERSION = \"v{VERSION}\";
+"
+    );
+
+    fs::write(path, contents)
+}
+
+fn generate_pools_file(parent: &Path) -> io::Result<()> {
+    let path = parent.join(Path::new("pools.js"));
+
+    let pools = pools();
+
+    let mut contents = format!("{AUTO_GENERATED_DISCLAIMER}\n");
+
+    contents += "
+/**
+ * @typedef {typeof POOL_ID_TO_POOL_NAME} PoolIdToPoolName
+ * @typedef {keyof PoolIdToPoolName} PoolId
+ */
+
+export const POOL_ID_TO_POOL_NAME = /** @type {const} */ ({
+";
+
+    let mut sorted_pools: Vec<_> = pools.iter().collect();
+    sorted_pools.sort_by(|a, b| a.name.to_lowercase().cmp(&b.name.to_lowercase()));
+
+    contents += &sorted_pools
+        .iter()
+        .map(|pool| {
+            let id = pool.serialized_id();
+            format!("  {id}: \"{}\",", pool.name)
+        })
+        .collect::<Vec<_>>()
+        .join("\n");
+
+    contents += "\n});\n";
+
+    fs::write(path, contents)
+}
+
+fn generate_metrics_file(interface: &Interface<'static>, parent: &Path) -> io::Result<()> {
+    let path = parent.join(Path::new("metrics.js"));
+
+    let indexes = Index::all();
+
+    let mut contents = format!(
+        "{AUTO_GENERATED_DISCLAIMER}
+
+export const INDEXES = /** @type {{const}} */ ([
+{}
+]);
+
+/**
+ * @typedef {{typeof INDEXES[number]}} IndexName
+ */
+",
+        indexes
+            .iter()
+            .map(|i| format!("  \"{}\"", i.serialize_long()))
+            .collect::<Vec<_>>()
+            .join(",\n")
+    );
+
+    // contents += &indexes
+    //     .iter()
+    //     .map(|i| format!(" * @typedef {{\"{}\"}} {i}", i.serialize_long()))
+    //     .collect::<Vec<_>>()
+    //     .join("\n");
+
+    //     contents += &format!(
+    //         "
+    //  * @typedef {{{}}} Index
+    //  */
+    // ",
+    //         indexes
+    //             .iter()
+    //             .map(|i| i.to_string())
+    //             .collect::<Vec<_>>()
+    //             .join(" | ")
+    //     );
+
+    let mut unique_index_groups = BTreeMap::new();
+
+    let mut word_to_freq: BTreeMap<_, usize> = BTreeMap::new();
+    interface
+        .metric_to_index_to_vec()
+        .keys()
+        .for_each(|metric| {
+            metric.split("_").for_each(|word| {
+                *word_to_freq.entry(word).or_default() += 1;
+            });
+        });
+    let mut word_to_freq = word_to_freq.into_iter().collect::<Vec<_>>();
+    word_to_freq.sort_unstable_by(|a, b| b.1.cmp(&a.1).then(a.0.cmp(b.0)));
+    let words = word_to_freq
+        .into_iter()
+        .map(|(str, _)| str)
+        .collect::<Vec<_>>();
+
+    contents += &format!(
+        "
+export const INDEX_TO_WORD = [
+  {}
+];
+
+",
+        words
+            .iter()
+            .enumerate()
+            .map(|(index, word)| format!("\"{word}\", // {}", index_to_letters(index)))
+            .collect::<Vec<_>>()
+            .join("\n  ")
+    );
+
+    let word_to_base62 = words
+        .into_iter()
+        .enumerate()
+        .map(|(i, w)| (w, index_to_letters(i)))
+        .collect::<HashMap<_, _>>();
+
+    let mut ser_metric_to_indexes = "
+/** @type {Record<string, IndexName[]>} */
+export const COMPRESSED_METRIC_TO_INDEXES = {
+"
+    .to_string();
+
+    interface
+        .metric_to_index_to_vec()
+        .iter()
+        .for_each(|(metric, index_to_vec)| {
+            let indexes = index_to_vec
+                .keys()
+                .map(|i| format!("\"{}\"", i.serialize_long()))
+                .collect::<Vec<_>>()
+                .join(", ");
+            let indexes = format!("[{indexes}]");
+            let unique = unique_index_groups.len();
+            let index = index_to_letters(*unique_index_groups.entry(indexes).or_insert(unique));
+
+            let compressed_metric = metric.split('_').fold(String::new(), |mut acc, w| {
+                if !acc.is_empty() {
+                    acc.push('_');
+                }
+                acc.push_str(&word_to_base62[w]);
+                acc
+            });
+
+            ser_metric_to_indexes += &format!("  {compressed_metric}: {index},\n");
+        });
+
+    ser_metric_to_indexes += "};
+";
+
+    let mut sorted_groups: Vec<_> = unique_index_groups.into_iter().collect();
+    sorted_groups.sort_by_key(|(_, index)| *index);
+    sorted_groups.into_iter().for_each(|(group, index)| {
+        let index = index_to_letters(index);
+        contents += &format!("/** @type {{IndexName[]}} */\nconst {index} = {group};\n");
+    });
+
+    contents += &ser_metric_to_indexes;
+
+    fs::write(path, contents)
+}
+
+fn index_to_letters(mut index: usize) -> String {
+    if index < 52 {
+        return (index_to_char(index) as char).to_string();
+    }
+    let mut result = [0u8; 8];
+    let mut pos = 8;
+    loop {
+        pos -= 1;
+        result[pos] = index_to_char(index % 52);
+        index /= 52;
+        if index == 0 {
+            break;
+        }
+        index -= 1;
+    }
+    unsafe { String::from_utf8_unchecked(result[pos..].to_vec()) }
+}
+
+fn index_to_char(index: usize) -> u8 {
+    match index {
+        0..=25 => b'A' + index as u8,
+        26..=51 => b'a' + (index - 26) as u8,
+        _ => unreachable!(),
+    }
+}
+
+// fn letters_to_index(s: &str) -> usize {
+//     let mut result = 0;
+//     for byte in s.bytes() {
+//         let value = char_to_index(byte) as usize;
+//         result = result * 52 + value + 1;
+//     }
+//     result - 1
+// }
+
+// fn char_to_index(byte: u8) -> u8 {
+//     match byte {
+//         b'A'..=b'Z' => byte - b'A',
+//         b'a'..=b'z' => byte - b'a' + 26,
+//         _ => 255, // Invalid
+//     }
+// }

crates/brk_binder/src/lib.rs

@@ -0,0 +1,5 @@
+mod js;
+
+pub use js::Bridge;
+
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");

crates/brk_binder/src/python.rs

@@ -0,0 +1 @@
+// TODO ?

crates/brk_bundler/Cargo.toml

@@ -0,0 +1,18 @@
+[package]
+name = "brk_bundler"
+description = "A thin wrapper around rolldown"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+homepage.workspace = true
+repository.workspace = true
+rust-version.workspace = true
+build = "build.rs"
+
+[dependencies]
+log = { workspace = true }
+notify = "8.2.0"
+brk_rolldown = "0.2.3"
+# brk_rolldown = { path = "../../../rolldown/crates/rolldown"}
+sugar_path = "1.2.0"
+tokio = { workspace = true }

crates/brk_bundler/README.md

@@ -0,0 +1,278 @@
+# brk_bundler
+
+Asset bundling and development server for BRK web interfaces with hot reloading and file watching.
+
+[![Crates.io](https://img.shields.io/crates/v/brk_bundler.svg)](https://crates.io/crates/brk_bundler)
+[![Documentation](https://docs.rs/brk_bundler/badge.svg)](https://docs.rs/brk_bundler)
+
+## Overview
+
+This crate provides a thin wrapper around the Rolldown JavaScript bundler specifically designed for BRK web interface development. It handles asset bundling, file copying, template processing, and development-mode file watching with automatic rebuilds and hot reloading for efficient web development workflows.
+
+**Key Features:**
+
+- JavaScript bundling with Rolldown (Rust-based bundler)
+- Automatic file watching and hot reloading in development mode
+- Template processing with version injection and asset hash replacement
+- Service worker generation with version management
+- Source map generation for debugging
+- Minification for production builds
+- Async/await support with Tokio integration
+
+**Target Use Cases:**
+
+- BRK blockchain explorer web interfaces
+- Development of Bitcoin analytics dashboards
+- Building responsive web applications for blockchain data visualization
+- Hot reloading development environment for rapid iteration
+
+## Installation
+
+```bash
+cargo add brk_bundler
+```
+
+## Quick Start
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let websites_path = Path::new("./web");
+    let source_folder = "src";
+    let watch = true; // Enable hot reloading
+
+    // Bundle assets and start development server
+    let dist_path = bundle(websites_path, source_folder, watch).await?;
+
+    println!("Assets bundled to: {}", dist_path.display());
+
+    // Keep running for file watching (in watch mode)
+    if watch {
+        tokio::signal::ctrl_c().await?;
+    }
+
+    Ok(())
+}
+```
+
+## API Overview
+
+### Core Functions
+
+**`bundle(websites_path: &Path, source_folder: &str, watch: bool) -> io::Result<PathBuf>`**
+Main bundling function that processes web assets and optionally starts file watching.
+
+### Bundling Process
+
+1. **Directory Setup**: Creates `dist/` directory and copies source files
+2. **JavaScript Bundling**: Processes `scripts/entry.js` with Rolldown bundler
+3. **Template Processing**: Updates `index.html` with hashed asset references
+4. **Service Worker**: Generates service worker with version injection
+5. **File Watching**: Optionally monitors source files for changes
+
+### Configuration
+
+**Rolldown Bundler Options:**
+
+- **Input**: `./src/scripts/entry.js` (main JavaScript entry point)
+- **Output**: `./dist/scripts/` directory
+- **Minification**: Enabled for production builds
+- **Source Maps**: File-based source maps for debugging
+- **Asset Hashing**: Automatic hash generation for cache busting
+
+## Examples
+
+### Development Mode with Hot Reloading
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let web_root = Path::new("./websites");
+
+    // Start development server with file watching
+    let _dist_path = bundle(web_root, "explorer", true).await?;
+
+    println!("Development server started!");
+    println!("Hot reloading enabled - edit files to see changes");
+
+    // Keep server running
+    loop {
+        tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
+    }
+}
+```
+
+### Production Build
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let web_root = Path::new("./websites");
+
+    // Build for production (no watching)
+    let dist_path = bundle(web_root, "dashboard", false).await?;
+
+    println!("Production build completed: {}", dist_path.display());
+
+    // Assets are minified and ready for deployment
+    Ok(())
+}
+```
+
+### Custom Web Application Structure
+
+```rust
+use brk_bundler::bundle;
+use std::path::Path;
+
+// Expected directory structure:
+// websites/
+// ├── my_app/
+// │   ├── index.html          // Main HTML template
+// │   ├── service-worker.js   // Service worker template
+// │   ├── scripts/
+// │   │   └── entry.js        // JavaScript entry point
+// │   ├── styles/
+// │   │   └── main.css        // CSS files
+// │   └── assets/
+// │       └── images/         // Static assets
+// └── dist/                   // Generated output
+
+#[tokio::main]
+async fn main() -> std::io::Result<()> {
+    let websites_path = Path::new("./websites");
+    let source_folder = "my_app";
+
+    let dist_path = bundle(websites_path, source_folder, false).await?;
+
+    // Result: dist/ contains bundled and processed files
+    // - dist/index.html (with updated script references)
+    // - dist/service-worker.js (with version injection)
+    // - dist/scripts/main.[hash].js (minified and hashed)
+    // - dist/styles/ (copied CSS files)
+    // - dist/assets/ (copied static assets)
+
+    Ok(())
+}
+```
+
+## Architecture
+
+### File Processing Pipeline
+
+1. **Source Copying**: Recursively copies all source files to dist directory
+2. **JavaScript Bundling**: Rolldown processes entry.js with dependencies
+3. **Asset Hashing**: Generates content-based hashes for cache busting
+4. **Template Updates**: Replaces placeholders in HTML templates
+5. **Version Injection**: Updates service worker with current package version
+
+### File Watching System
+
+**Development Mode Watchers:**
+
+- **Source File Watcher**: Monitors non-script files for changes
+- **Bundle Watcher**: Watches JavaScript files and triggers rebuilds
+- **Template Watcher**: Updates HTML when bundled assets change
+
+**Event Handling:**
+
+- **File Creation/Modification**: Automatic copying to dist directory
+- **Script Changes**: Triggers Rolldown rebuild and template update
+- **Template Changes**: Processes HTML and updates asset references
+
+### Template Processing
+
+**index.html Processing:**
+
+- Scans bundled JavaScript for asset hash
+- Replaces `/scripts/main.js` with `/scripts/main.[hash].js`
+- Maintains cache busting while preserving template structure
+
+**service-worker.js Processing:**
+
+- Replaces `__VERSION__` placeholder with current crate version
+- Enables version-based cache invalidation
+- Maintains service worker functionality
+
+### Async Architecture
+
+Built on Tokio async runtime:
+
+- **Non-blocking I/O**: Efficient file operations and watching
+- **Concurrent Tasks**: Parallel file watching and bundle processing
+- **Background Processing**: Development server runs in background task
+
+## Configuration Options
+
+### Rolldown Configuration
+
+The bundler uses optimized Rolldown settings:
+
+```rust
+BundlerOptions {
+    input: Some(vec!["./src/scripts/entry.js".into()]),
+    dir: Some("./dist/scripts".to_string()),
+    minify: Some(RawMinifyOptions::Bool(true)),
+    sourcemap: Some(SourceMapType::File),
+    // ... other default options
+}
+```
+
+### File Structure Requirements
+
+**Required Files:**
+
+- `src/scripts/entry.js` - JavaScript entry point
+- `src/index.html` - HTML template
+- `src/service-worker.js` - Service worker template
+
+**Optional Directories:**
+
+- `src/styles/` - CSS stylesheets
+- `src/assets/` - Static assets (images, fonts, etc.)
+- `src/components/` - Additional JavaScript modules
+
+## Development Workflow
+
+### Setup
+
+1. Create web application in `websites/app_name/`
+2. Add required files (index.html, entry.js, service-worker.js)
+3. Run bundler in watch mode for development
+
+### Hot Reloading
+
+- **Script Changes**: Automatic bundle rebuild and browser refresh
+- **Template Changes**: Immediate HTML update with asset hash replacement
+- **Asset Changes**: Instant copy to dist directory
+- **Style Changes**: Direct copy without bundling
+
+### Production Deployment
+
+1. Run bundler without watch mode
+2. Deploy `dist/` directory contents
+3. Assets include content hashes for cache busting
+4. Service worker includes version for cache management
+
+## Code Analysis Summary
+
+**Main Function**: `bundle()` async function coordinating Rolldown bundler with file processing and watching \
+**File Operations**: Recursive directory copying with `copy_dir_all()` and selective file processing \
+**Templating**: String replacement for asset hash injection and version management \
+**File Watching**: Multi-watcher system using `notify` crate for real-time development feedback \
+**Async Integration**: Tokio-based async architecture with background task spawning \
+**Bundler Integration**: Rolldown wrapper with optimized configuration for web development \
+**Architecture**: Development-focused asset pipeline with hot reloading and production optimization
+
+---
+
+_This README was generated by Claude Code_

crates/brk_bundler/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    let profile = std::env::var("PROFILE").unwrap_or_default();
+
+    if profile == "release" {
+        println!("cargo:rustc-flag=-C");
+        println!("cargo:rustc-flag=target-cpu=native");
+    }
+}

crates/brk_bundler/src/lib.rs

@@ -0,0 +1,194 @@
+#![doc = include_str!("../README.md")]
+
+use std::{
+    fs, io,
+    path::{Path, PathBuf},
+    sync::Arc,
+};
+
+use brk_rolldown::{
+    Bundler, BundlerOptions, InlineConstConfig, InlineConstMode, InlineConstOption,
+    OptimizationOption, RawMinifyOptions, SourceMapType,
+};
+use log::error;
+use notify::{EventKind, RecursiveMode, Watcher};
+use sugar_path::SugarPath;
+use tokio::sync::Mutex;
+
+const VERSION: &str = env!("CARGO_PKG_VERSION");
+
+pub async fn bundle(
+    modules_path: &Path,
+    websites_path: &Path,
+    source_folder: &str,
+    watch: bool,
+) -> io::Result<PathBuf> {
+    let relative_modules_path = modules_path;
+    let relative_source_path = websites_path.join(source_folder);
+    let relative_dist_path = websites_path.join("dist");
+
+    let absolute_modules_path = relative_modules_path.absolutize();
+    let absolute_modules_path_clone = absolute_modules_path.clone();
+    let absolute_websites_path = websites_path.absolutize();
+    let absolute_websites_path_clone = absolute_websites_path.clone();
+
+    let absolute_source_path = relative_source_path.absolutize();
+    let absolute_source_index_path = absolute_source_path.join("index.html");
+    let absolute_source_index_path_clone = absolute_source_index_path.clone();
+    let absolute_source_scripts_path = absolute_source_path.join("scripts");
+    let absolute_source_scripts_modules_path = absolute_source_scripts_path.join("modules");
+    let absolute_source_sw_path = absolute_source_path.join("service-worker.js");
+    let absolute_source_sw_path_clone = absolute_source_sw_path.clone();
+
+    let absolute_dist_path = relative_dist_path.absolutize();
+    let absolute_dist_scripts_path = absolute_dist_path.join("scripts");
+    let absolute_dist_scripts_entry_path = absolute_dist_scripts_path.join("entry.js");
+    let absolute_dist_scripts_entry_path_clone = absolute_dist_scripts_entry_path.clone();
+    let absolute_dist_index_path = absolute_dist_path.join("index.html");
+    let absolute_dist_sw_path = absolute_dist_path.join("service-worker.js");
+
+    let _ = fs::remove_dir_all(&absolute_dist_path);
+    let _ = fs::remove_dir_all(&absolute_source_scripts_modules_path);
+    copy_dir_all(
+        &absolute_modules_path,
+        &absolute_source_scripts_modules_path,
+    )?;
+    copy_dir_all(&absolute_source_path, &absolute_dist_path)?;
+    fs::remove_dir_all(&absolute_dist_scripts_path)?;
+    fs::create_dir(&absolute_dist_scripts_path)?;
+
+    // dbg!(BundlerOptions::default());
+
+    let mut bundler = Bundler::new(BundlerOptions {
+        input: Some(vec![format!("./{source_folder}/scripts/entry.js").into()]),
+        dir: Some("./dist/scripts".to_string()),
+        cwd: Some(absolute_websites_path),
+        minify: Some(RawMinifyOptions::Bool(true)),
+        sourcemap: Some(SourceMapType::File),
+        // advanced_chunks: Some(AdvancedChunksOptions {
+        //     // min_size: Some(1000.0),
+        //     min_share_count: Some(20),
+        //     // min_module_size: S
+        //     // include_dependencies_recursively: Some(true),
+        //     ..Default::default()
+        // }),
+        //
+        // inline_dynamic_imports
+        // experimental: Some(ExperimentalOptions {
+        //     strict_execution_order: Some(true),
+        //     ..Default::default()
+        // }),
+        optimization: Some(OptimizationOption {
+            inline_const: Some(InlineConstOption::Config(InlineConstConfig {
+                mode: Some(InlineConstMode::All),
+                ..Default::default()
+            })),
+            // Needs benchmarks
+            // pife_for_module_wrappers: Some(true),
+            ..Default::default()
+        }),
+        ..Default::default()
+    })
+    .unwrap();
+
+    if let Err(error) = bundler.write().await {
+        error!("{error:?}");
+    }
+
+    let update_dist_index = move || {
+        let mut contents = fs::read_to_string(&absolute_source_index_path).unwrap();
+
+        if let Ok(entry) = fs::read_to_string(&absolute_dist_scripts_entry_path_clone)
+            && let Some(start) = entry.find("main")
+            && let Some(end) = entry.find(".js")
+        {
+            let main_hashed = &entry[start..end];
+            contents = contents.replace("/scripts/main.js", &format!("/scripts/{main_hashed}.js"));
+        }
+
+        let _ = fs::write(&absolute_dist_index_path, contents);
+    };
+
+    let update_source_sw = move || {
+        let contents = fs::read_to_string(&absolute_source_sw_path)
+            .unwrap()
+            .replace("__VERSION__", &format!("v{VERSION}"));
+        let _ = fs::write(&absolute_dist_sw_path, contents);
+    };
+
+    update_dist_index();
+    update_source_sw();
+
+    if !watch {
+        return Ok(relative_dist_path);
+    }
+
+    tokio::spawn(async move {
+        let mut event_watcher = notify::recommended_watcher(
+            move |res: Result<notify::Event, notify::Error>| match res {
+                Ok(event) => match event.kind {
+                    EventKind::Create(_) => event.paths,
+                    EventKind::Modify(_) => event.paths,
+                    _ => vec![],
+                }
+                .into_iter()
+                .for_each(|path| {
+                    let path = path.absolutize();
+
+                    if path == absolute_dist_scripts_entry_path
+                        || path == absolute_source_index_path_clone
+                    {
+                        update_dist_index();
+                    } else if path == absolute_source_sw_path_clone {
+                        update_source_sw();
+                    } else if let Ok(suffix) = path.strip_prefix(&absolute_modules_path) {
+                        let source_modules_path = absolute_source_scripts_modules_path.join(suffix);
+                        if path.is_file() {
+                            let _ = fs::create_dir_all(path.parent().unwrap());
+                            let _ = fs::copy(&path, &source_modules_path);
+                        }
+                    } else if let Ok(suffix) = path.strip_prefix(&absolute_source_path)
+                        // scripts are handled by rolldown
+                        && !path.starts_with(&absolute_source_scripts_path)
+                    {
+                        let dist_path = absolute_dist_path.join(suffix);
+                        if path.is_file() {
+                            let _ = fs::create_dir_all(path.parent().unwrap());
+                            let _ = fs::copy(&path, &dist_path);
+                        }
+                    }
+                }),
+                Err(e) => error!("watch error: {e:?}"),
+            },
+        )
+        .unwrap();
+
+        event_watcher
+            .watch(&absolute_websites_path_clone, RecursiveMode::Recursive)
+            .unwrap();
+        event_watcher
+            .watch(&absolute_modules_path_clone, RecursiveMode::Recursive)
+            .unwrap();
+
+        let watcher =
+            brk_rolldown::Watcher::new(vec![Arc::new(Mutex::new(bundler))], None).unwrap();
+
+        watcher.start().await;
+    });
+
+    Ok(relative_dist_path)
+}
+
+fn copy_dir_all(src: impl AsRef<Path>, dst: impl AsRef<Path>) -> io::Result<()> {
+    fs::create_dir_all(&dst)?;
+    for entry in fs::read_dir(src)? {
+        let entry = entry?;
+        let ty = entry.file_type()?;
+        if ty.is_dir() {
+            copy_dir_all(entry.path(), dst.as_ref().join(entry.file_name()))?;
+        } else {
+            fs::copy(entry.path(), dst.as_ref().join(entry.file_name()))?;
+        }
+    }
+    Ok(())
+}

crates/brk_cli/Cargo.toml

@@ -1,32 +1,35 @@
 [package]
 name = "brk_cli"
-description = "A command line interface to interact with the full Bitcoin Research Kit"
+description = "A command line interface to run a BRK instance"
 version.workspace = true
 edition.workspace = true
 license.workspace = true
 homepage.workspace = true
 repository.workspace = true
+rust-version.workspace = true
+build = "build.rs"
 
 [dependencies]
 bitcoincore-rpc = { workspace = true }
+brk_binder = { workspace = true }
+brk_bundler = { workspace = true }
 brk_computer = { workspace = true }
-brk_core = { workspace = true }
-brk_exit = { workspace = true }
+brk_error = { workspace = true }
 brk_fetcher = { workspace = true }
 brk_indexer = { workspace = true }
+brk_interface = { workspace = true }
 brk_logger = { workspace = true }
 brk_parser = { workspace = true }
-brk_query = { workspace = true }
 brk_server = { workspace = true }
-brk_vec = { workspace = true }
-clap = { workspace = true }
-clap_derive = { workspace = true }
-color-eyre = { workspace = true }
+vecdb = { workspace = true }
+clap = { version = "4.5.48", features = ["derive", "string"] }
+color-eyre = "0.6.5"
 log = { workspace = true }
+minreq = { workspace = true }
 serde = { workspace = true }
-tabled = { workspace = true }
 tokio = { workspace = true }
-toml = "0.8.23"
+toml = "0.9.7"
+zip = { version = "5.1.1", default-features = false, features = ["deflate"] }
 
 [[bin]]
 name = "brk"

crates/brk_cli/README.md

@@ -1,91 +1,232 @@
-# BRK Cli
+# brk_cli
 
-<p align="left">
-  <a href="https://github.com/bitcoinresearchkit/brk">
-    <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
-  </a>
-  <a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
-    <img src="https://img.shields.io/crates/l/brk" alt="License" />
-  </a>
-  <a href="https://crates.io/crates/brk_cli">
-    <img src="https://img.shields.io/crates/v/brk_cli" alt="Version" />
-  </a>
-  <a href="https://docs.rs/brk_cli">
-    <img src="https://img.shields.io/docsrs/brk_cli" alt="Documentation" />
-  </a>
-  <img src="https://img.shields.io/crates/size/brk_cli" alt="Size" />
-  <a href="https://deps.rs/crate/brk_cli">
-    <img src="https://deps.rs/crate/brk_cli/latest/status.svg" alt="Dependency status">
-  </a>
-  <a href="https://discord.gg/HaR3wpH3nr">
-    <img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
-  </a>
-  <a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
-    <img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
-  </a>
-  <a href="https://bsky.app/profile/bitcoinresearchkit.org">
-    <img src="https://img.shields.io/badge/bluesky-blue?link=https%3A%2F%2Fbsky.app%2Fprofile%2Fbitcoinresearchkit.org" alt="Bluesky" />
-  </a>
-  <a href="https://x.com/brkdotorg">
-    <img src="https://img.shields.io/badge/x.com-black" alt="X" />
-  </a>
-</p>
+Command-line interface orchestrating complete Bitcoin Research Kit instances with automatic configuration and continuous blockchain processing.
 
-A command line interface to interact with the full Bitcoin Research Kit. It's built on top of every other create and gives the possility to use BRK using the terminal instead of Rust.
+[![Crates.io](https://img.shields.io/crates/v/brk_cli.svg)](https://crates.io/crates/brk_cli)
+[![Documentation](https://docs.rs/brk_cli/badge.svg)](https://docs.rs/brk_cli)
 
-It has 2 commandes for now (other than `help` and `version`) which are `run` and `query`. The former is used to run the processing (indexer + computer) and/or the server. The latter uses `brk_query` as its backend just like to server to be able to get datasets via the terminal instead of the API. Both commands are very costumizable by having all the parameters of their Rust counterparts ([`run`](https://github.com/bitcoinresearchkit/brk/blob/main/crates/brk_cli/src/run.rs#L91-L147), [`query`](https://github.com/bitcoinresearchkit/brk/blob/main/crates/brk_query/src/params.rs)).
+## Overview
 
-## Requirements
+This crate provides the primary command-line interface for running Bitcoin Research Kit instances. It orchestrates the entire data processing pipeline from Bitcoin Core block parsing through analytics computation to HTTP API serving, with persistent configuration management, automatic error recovery, and continuous blockchain synchronization.
 
-### Hardware
+**Key Features:**
 
-#### Recommended
+- Complete BRK pipeline orchestration with parser, indexer, computer, and server coordination
+- Persistent configuration system with TOML-based auto-save functionality
+- Continuous blockchain processing with new block detection and incremental updates
+- Flexible Bitcoin Core RPC authentication with cookie file and user/password support
+- Configurable web interface options including auto-downloading from GitHub releases
+- Large stack allocation (512MB) for handling complex blockchain processing workloads
+- Graceful shutdown handling with proper cleanup and state preservation
 
-- [Latest base model Mac mini](https://www.apple.com/mac-mini/)
-- [Thunderbolt 4 SSD enclosure](https://satechi.net/products/usb4-nvme-ssd-pro-enclosure/Z2lkOi8vc2hvcGlmeS9Qcm9kdWN0VmFyaWFudC80MDE4ODQ3MDA2NzI4OA==?queryID=7961465089021ee203a60db7e62e90d2)
-- [2 TB NVMe SSD](https://shop.sandisk.com/products/ssd/internal-ssd/wd-black-sn850x-nvme-ssd?sku=WDS200T2X0E-00BCA0)
+**Target Use Cases:**
 
-#### Minimum
+- Production Bitcoin analytics deployments requiring full pipeline operation
+- Development environments for Bitcoin research and analysis
+- Continuous blockchain monitoring with real-time data updates
+- Academic research requiring comprehensive historical blockchain datasets
 
-To be determined
-
-### Software
-
-- [Bitcoin](https://bitcoin.org/en/full-node)
-- [Rust](https://www.rust-lang.org/tools/install)
-- Unix based operating system (Mac OS or Linux)
-  - Ubuntu users need to install `open-ssl` via `sudo apt install libssl-dev pkg-config`
-
-## Download
-
-### Binaries
-
-You can find a pre-built binary for your operating system on the releases page ([link](https://github.com/bitcoinresearchkit/brk/releases/latest)).
-
-### Cargo
+## Installation
 
 ```bash
-# Install
-cargo install brk # or `cargo install brk_cli`, the result is the same
-
-# Update
-cargo install brk # or `cargo install-update -a` if you have `cargo-update` installed
+cargo install brk # or cargo install brk_cli
 ```
 
-### Source
+## Quick Start
 
 ```bash
-git clone https://github.com/bitcoinresearchkit/brk.git
-cd brk/crates/brk
-cargo run -r
+# First run - configure and start processing
+brk --brkdir ./data --bitcoindir ~/.bitcoin --fetch true
+
+# Subsequent runs use saved configuration
+brk
+
+# Override specific options
+brk --website none --fetch false
 ```
 
-## Usage
+## API Overview
 
-Run `brk -h` to view each available command and their respective description.
+### Core Structure
 
-`-h` works also for commands, which mean that `brk run -h` will explain all the parameters of `brk run` for example.
+- **`Config`**: Persistent configuration with clap-based CLI parsing and TOML serialization
+- **`Bridge`**: Interface trait for generating JavaScript bridge files for web interfaces
+- **`Website`**: Enum for web interface options (None, Bitview, Custom)
+- **Path Functions**: Cross-platform default path resolution for Bitcoin and BRK directories
 
-Every parameter set for `brk run` will be saved at `~/.brk/config.toml`, which will allow you to simply run `brk run` next time.
+### Main Operations
 
-Then the easiest to let others access your server is to use `cloudflared` which will also cache requests. For more information go to: https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/
+**`main() -> color_eyre::Result<()>`**
+Entry point with error handling setup, directory creation, logging initialization, and high-stack thread spawning.
+
+**`run() -> color_eyre::Result<()>`**
+Core processing loop handling configuration, RPC connection, component initialization, and continuous blockchain monitoring.
+
+### Configuration Management
+
+**Persistent Settings:**
+
+- All CLI arguments automatically saved to `~/.brk/config.toml`
+- Argument overrides update saved configuration on each run
+- Cross-platform path resolution with tilde and $HOME expansion
+- Validation of Bitcoin directory, blocks directory, and RPC authentication
+
+**CLI Parameters:**
+
+- `--bitcoindir`, `--blocksdir`, `--brkdir`: Directory configuration
+- `--fetch`, `--exchanges`: Data source configuration
+- `--website`: Web interface selection
+- `--rpcconnect`, `--rpcport`, `--rpccookiefile`, `--rpcuser`, `--rpcpassword`: RPC settings
+
+## Examples
+
+### Basic Usage
+
+```bash
+# Initialize with custom directories
+brk --bitcoindir /data/bitcoin --brkdir /data/brk
+
+# Enable all features with custom RPC
+brk --fetch true --exchanges true --website bitview \
+    --rpcuser myuser --rpcpassword mypass
+
+# Minimal setup with API only
+brk --website none --fetch false
+```
+
+### Configuration File Example
+
+After first run, settings are saved to `~/.brk/config.toml`:
+
+```toml
+bitcoindir = "/home/user/.bitcoin"
+blocksdir = "/home/user/.bitcoin/blocks"
+brkdir = "/home/user/brk_data"
+fetch = true
+exchanges = true
+website = "bitview"
+rpcconnect = "localhost"
+rpcport = 8332
+rpccookiefile = "/home/user/.bitcoin/.cookie"
+```
+
+### Web Interface Configuration
+
+```bash
+# Use built-in Bitview interface
+brk --website bitview
+
+# Use custom web interface
+brk --website custom
+
+# API only, no web interface
+brk --website none
+```
+
+### Development Mode
+
+```bash
+# Development with local website directory
+# Looks for ../../websites directory first
+brk --website bitview
+
+# Production with auto-download from GitHub
+# Downloads websites from release artifacts
+brk --website bitview
+```
+
+## Architecture
+
+### Startup Sequence
+
+1. **Environment Setup**: Color eyre error handling, directory creation, logging initialization
+2. **High-Stack Thread**: 512MB stack for complex blockchain processing operations
+3. **Configuration Loading**: CLI parsing, TOML reading, argument merging, validation
+4. **Component Initialization**: Parser, indexer, computer, interface creation with proper dependencies
+
+### Processing Pipeline
+
+**Continuous Operation Loop:**
+
+1. **Bitcoin Core Sync Wait**: Monitors `headers == blocks` for full node synchronization
+2. **Block Count Detection**: Compares current and previous block counts for new block detection
+3. **Indexing Phase**: Processes new blocks through parser with collision detection option
+4. **Computing Phase**: Runs analytics computations on newly indexed data
+5. **Server Operation**: Serves HTTP API with optional web interface throughout processing
+
+### Web Interface Integration
+
+**Website Handling:**
+
+- **Development Mode**: Uses local `../../websites` directory if available
+- **Production Mode**: Downloads release artifacts from GitHub using semantic versioning
+- **Bundle Generation**: Creates optimized JavaScript bundles using `brk_bundler`
+- **Bridge Files**: Generates JavaScript bridge files for vector IDs and pool data
+
+**Download and Bundle Process:**
+
+```rust
+// Automatic website download and bundling
+let url = format!("https://github.com/bitcoinresearchkit/brk/archive/refs/tags/v{VERSION}.zip");
+let response = minreq::get(url).send()?;
+zip::ZipArchive::new(cursor).extract(downloads_path)?;
+bundle(&websites_path, website.to_folder_name(), true).await?
+```
+
+### RPC Authentication
+
+**Flexible Authentication Methods:**
+
+- **Cookie File**: Automatic detection at `--bitcoindir/.cookie`
+- **User/Password**: Manual configuration with `--rpcuser` and `--rpcpassword`
+- **Connection Validation**: Startup checks ensure proper Bitcoin Core connectivity
+
+### Configuration System
+
+**TOML Persistence:**
+
+- Automatic serialization/deserialization with `serde` and `toml`
+- Error-tolerant parsing with `default_on_error` deserializer
+- Argument consumption validation ensuring all CLI options are processed
+- Path expansion supporting `~` and `$HOME` environment variables
+
+## Configuration
+
+### Default Paths
+
+**Cross-Platform Path Resolution:**
+
+- **Linux**: `~/.bitcoin` for Bitcoin Core, `~/.brk` for BRK data
+- **macOS**: `~/Library/Application Support/Bitcoin` for Bitcoin Core
+- **Logs**: `~/.brk/log` for application logging
+- **Downloads**: `~/.brk/downloads` for temporary website artifacts
+
+### Performance Settings
+
+**Memory Management:**
+
+- 512MB stack size for main processing thread
+- Multi-threaded tokio runtime with all features enabled
+- Persistent configuration caching to minimize I/O operations
+
+### Error Handling
+
+**Comprehensive Validation:**
+
+- Directory existence checks with user-friendly error messages
+- RPC authentication verification before processing begins
+- Graceful exit with help suggestions for configuration issues
+
+## Code Analysis Summary
+
+**Main Structure**: `Config` struct with clap-derived CLI parsing and persistent TOML configuration management \
+**Processing Loop**: Continuous Bitcoin Core monitoring with sync detection and incremental block processing \
+**Web Integration**: Automatic website download from GitHub releases with JavaScript bundle generation \
+**Component Orchestration**: Coordination of parser, indexer, computer, and server with proper dependency management \
+**Error Handling**: `color_eyre` integration with comprehensive validation and user-friendly error messages \
+**Threading**: High-stack thread allocation (512MB) with tokio multi-threaded runtime for complex operations \
+**Architecture**: Complete BRK pipeline orchestration with persistent configuration and continuous blockchain synchronization
+
+---
+
+_This README was generated by Claude Code_

crates/brk_cli/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    let profile = std::env::var("PROFILE").unwrap_or_default();
+
+    if profile == "release" {
+        println!("cargo:rustc-flag=-C");
+        println!("cargo:rustc-flag=target-cpu=native");
+    }
+}

crates/brk_cli/src/config.rs

@@ -1,113 +1,21 @@
 use std::{
     fs,
     path::{Path, PathBuf},
-    thread::sleep,
-    time::Duration,
 };
 
-use bitcoincore_rpc::{self, Auth, Client, RpcApi};
-use brk_computer::Computer;
-use brk_core::{default_bitcoin_path, default_brk_path, default_on_error, dot_brk_path};
-use brk_exit::Exit;
+use bitcoincore_rpc::{self, Auth, Client};
 use brk_fetcher::Fetcher;
-use brk_indexer::Indexer;
-use brk_server::{Server, Website};
-use brk_vec::{Computation, Format};
-use clap_derive::{Parser, ValueEnum};
+use clap::Parser;
 use color_eyre::eyre::eyre;
-use log::info;
-use serde::{Deserialize, Serialize};
+use serde::{Deserialize, Deserializer, Serialize};
 
-pub fn run(config: RunConfig) -> color_eyre::Result<()> {
-    let config = RunConfig::import(Some(config))?;
+use crate::{default_bitcoin_path, default_brk_path, dot_brk_path, website::Website};
 
-    let rpc = config.rpc()?;
-
-    let exit = Exit::new();
-
-    let parser = brk_parser::Parser::new(config.blocksdir(), rpc);
-
-    let format = config.format();
-
-    let mut indexer = Indexer::new(&config.outputsdir(), config.check_collisions())?;
-    indexer.import_stores()?;
-    indexer.import_vecs()?;
-
-    let wait_for_synced_node = || -> color_eyre::Result<()> {
-        let is_synced = || -> color_eyre::Result<bool> {
-            let info = rpc.get_blockchain_info()?;
-            Ok(info.headers == info.blocks)
-        };
-
-        if !is_synced()? {
-            info!("Waiting for node to be synced...");
-            while !is_synced()? {
-                sleep(Duration::from_secs(1))
-            }
-        }
-
-        Ok(())
-    };
-
-    let mut computer = Computer::new(&config.outputsdir(), config.fetcher(), format);
-    computer.import_stores(&indexer)?;
-    computer.import_vecs(&indexer, config.computation())?;
-
-    tokio::runtime::Builder::new_multi_thread()
-        .enable_all()
-        .build()?
-        .block_on(async {
-            let server = if config.serve() {
-                let served_indexer = indexer.clone();
-                let served_computer = computer.clone();
-
-                let server = Server::new(served_indexer, served_computer, config.website())?;
-
-                let opt = Some(tokio::spawn(async move {
-                    server.serve().await.unwrap();
-                }));
-
-                sleep(Duration::from_secs(1));
-
-                opt
-            } else {
-                None
-            };
-
-            if config.process() {
-                loop {
-                    wait_for_synced_node()?;
-
-                    let block_count = rpc.get_block_count()?;
-
-                    info!("{} blocks found.", block_count + 1);
-
-                    let starting_indexes = indexer.index(&parser, rpc, &exit)?;
-
-                    computer.compute(&mut indexer, starting_indexes, &exit)?;
-
-                    if let Some(delay) = config.delay() {
-                        sleep(Duration::from_secs(delay))
-                    }
-
-                    info!("Waiting for new blocks...");
-
-                    while block_count == rpc.get_block_count()? {
-                        sleep(Duration::from_secs(1))
-                    }
-                }
-            }
-
-            if let Some(handle) = server {
-                handle.await.unwrap();
-            }
-
-            Ok(())
-        })
-}
+const DOWNLOADS: &str = "downloads";
 
 #[derive(Parser, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize)]
-pub struct RunConfig {
+#[command(version, about)]
+pub struct Config {
     /// Bitcoin main directory path, defaults: ~/.bitcoin, ~/Library/Application\ Support/Bitcoin, saved
     #[serde(default, deserialize_with = "default_on_error")]
     #[arg(long, value_name = "PATH")]
@@ -123,27 +31,17 @@ pub struct RunConfig {
     #[arg(long, value_name = "PATH")]
     brkdir: Option<String>,
 
-    /// Activated services, default: all, saved
-    #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(short, long)]
-    services: Option<Services>,
-
-    /// Computation of computed datasets, `lazy` computes data whenever requested without saving it, `eager` computes the data once and saves it to disk, default: `lazy`, saved
-    #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(short, long)]
-    computation: Option<Computation>,
-
-    /// Format of computed datasets, `compressed` to save disk space (experimental), `raw` to prioritize speed, default: `raw`, saved
-    #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(short, long)]
-    format: Option<Format>,
-
-    /// Activate fetching prices from exchanges APIs and the computation of all related datasets, default: true, saved
+    /// Activate fetching prices from BRK's API and the computation of all price related datasets, default: true, saved
     #[serde(default, deserialize_with = "default_on_error")]
     #[arg(short = 'F', long, value_name = "BOOL")]
     fetch: Option<bool>,
 
-    /// Website served by the server (if active), default: default, saved
+    /// Activate fetching prices from exchanges APIs if `fetch` is also set to `true`, default: true, saved
+    #[serde(default, deserialize_with = "default_on_error")]
+    #[arg(long, value_name = "BOOL")]
+    exchanges: Option<bool>,
+
+    /// Website served by the server, default: default, saved
     #[serde(default, deserialize_with = "default_on_error")]
     #[arg(short, long)]
     website: Option<Website>,
@@ -173,19 +71,16 @@ pub struct RunConfig {
     #[arg(long, value_name = "PASSWORD")]
     rpcpassword: Option<String>,
 
-    /// Delay between runs, default: 0, saved
-    #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(long, value_name = "SECONDS")]
-    delay: Option<u64>,
-
     /// DEV: Activate checking address hashes for collisions when indexing, default: false, saved
     #[serde(default, deserialize_with = "default_on_error")]
-    #[arg(long, value_name = "BOOL")]
+    #[arg(skip)]
     check_collisions: Option<bool>,
 }
 
-impl RunConfig {
-    pub fn import(config_args: Option<RunConfig>) -> color_eyre::Result<Self> {
+impl Config {
+    pub fn import() -> color_eyre::Result<Self> {
+        let config_args = Some(Config::parse());
+
         let path = dot_brk_path();
 
         let _ = fs::create_dir_all(&path);
@@ -207,20 +102,12 @@ impl RunConfig {
                 config_saved.brkdir = Some(brkdir);
             }
 
-            if let Some(services) = config_args.services.take() {
-                config_saved.services = Some(services);
-            }
-
-            if let Some(computation) = config_args.computation.take() {
-                config_saved.computation = Some(computation);
-            }
-
             if let Some(fetch) = config_args.fetch.take() {
                 config_saved.fetch = Some(fetch);
             }
 
-            if let Some(format) = config_args.format.take() {
-                config_saved.format = Some(format);
+            if let Some(exchanges) = config_args.exchanges.take() {
+                config_saved.exchanges = Some(exchanges);
             }
 
             if let Some(website) = config_args.website.take() {
@@ -247,15 +134,11 @@ impl RunConfig {
                 config_saved.rpcpassword = Some(rpcpassword);
             }
 
-            if let Some(delay) = config_args.delay.take() {
-                config_saved.delay = Some(delay);
-            }
-
             if let Some(check_collisions) = config_args.check_collisions.take() {
                 config_saved.check_collisions = Some(check_collisions);
             }
 
-            if config_args != RunConfig::default() {
+            if config_args != Config::default() {
                 dbg!(config_args);
                 panic!("Didn't consume the full config")
             }
@@ -267,19 +150,6 @@ impl RunConfig {
 
         config.write(&path)?;
 
-        // info!("Configuration {{");
-        // info!("  bitcoindir: {:?}", config.bitcoindir);
-        // info!("  brkdir: {:?}", config.brkdir);
-        // info!("  services: {:?}", config.services);
-        // info!("  website: {:?}", config.website);
-        // info!("  rpcconnect: {:?}", config.rpcconnect);
-        // info!("  rpcport: {:?}", config.rpcport);
-        // info!("  rpccookiefile: {:?}", config.rpccookiefile);
-        // info!("  rpcuser: {:?}", config.rpcuser);
-        // info!("  rpcpassword: {:?}", config.rpcpassword);
-        // info!("  delay: {:?}", config.delay);
-        // info!("}}");
-
         Ok(config)
     }
 
@@ -307,7 +177,9 @@ impl RunConfig {
 
         if self.rpc_auth().is_err() {
             println!(
-                "No way found to authenticate the RPC client, please either set --rpccookiefile or --rpcuser and --rpcpassword.\nRun the program with '-h' for help."
+                "Unsuccessful authentication with the RPC client.
+First make sure that `bitcoind` is running. If it is then please either set --rpccookiefile or --rpcuser and --rpcpassword as the default values seemed to have failed.
+Finally, you can run the program with '-h' for help."
             );
             std::process::exit(1);
         }
@@ -315,7 +187,7 @@ impl RunConfig {
 
     fn read(path: &Path) -> Self {
         fs::read_to_string(path).map_or_else(
-            |_| RunConfig::default(),
+            |_| Config::default(),
             |contents| toml::from_str(&contents).unwrap_or_default(),
         )
     }
@@ -358,10 +230,6 @@ impl RunConfig {
         self.rpcport
     }
 
-    pub fn delay(&self) -> Option<u64> {
-        self.delay
-    }
-
     pub fn bitcoindir(&self) -> PathBuf {
         self.bitcoindir
             .as_ref()
@@ -381,22 +249,12 @@ impl RunConfig {
             .map_or_else(default_brk_path, |s| Self::fix_user_path(s.as_ref()))
     }
 
-    pub fn outputsdir(&self) -> PathBuf {
-        self.brkdir().join("outputs")
-    }
-
     pub fn harsdir(&self) -> PathBuf {
-        self.outputsdir().join("hars")
+        self.brkdir().join("hars")
     }
 
-    pub fn process(&self) -> bool {
-        self.services
-            .is_none_or(|m| m == Services::All || m == Services::Processor)
-    }
-
-    pub fn serve(&self) -> bool {
-        self.services
-            .is_none_or(|m| m == Services::All || m == Services::Server)
+    pub fn downloads_dir(&self) -> PathBuf {
+        dot_brk_path().join(DOWNLOADS)
     }
 
     fn path_cookiefile(&self) -> PathBuf {
@@ -425,24 +283,20 @@ impl RunConfig {
     }
 
     pub fn website(&self) -> Website {
-        self.website.unwrap_or(Website::Default)
+        self.website.unwrap_or(Website::Bitview)
     }
 
     pub fn fetch(&self) -> bool {
         self.fetch.is_none_or(|b| b)
     }
 
+    pub fn exchanges(&self) -> bool {
+        self.exchanges.is_none_or(|b| b)
+    }
+
     pub fn fetcher(&self) -> Option<Fetcher> {
         self.fetch()
-            .then(|| Fetcher::import(Some(self.harsdir().as_path())).unwrap())
-    }
-
-    pub fn computation(&self) -> Computation {
-        self.computation.unwrap_or_default()
-    }
-
-    pub fn format(&self) -> Format {
-        self.format.unwrap_or_default()
+            .then(|| Fetcher::import(self.exchanges(), Some(self.harsdir().as_path())).unwrap())
     }
 
     pub fn check_collisions(&self) -> bool {
@@ -450,23 +304,13 @@ impl RunConfig {
     }
 }
 
-#[derive(
-    Default,
-    Debug,
-    Clone,
-    Copy,
-    Parser,
-    ValueEnum,
-    Serialize,
-    Deserialize,
-    PartialEq,
-    Eq,
-    PartialOrd,
-    Ord,
-)]
-pub enum Services {
-    #[default]
-    All,
-    Processor,
-    Server,
+fn default_on_error<'de, D, T>(deserializer: D) -> Result<T, D::Error>
+where
+    D: Deserializer<'de>,
+    T: Deserialize<'de> + Default,
+{
+    match T::deserialize(deserializer) {
+        Ok(v) => Ok(v),
+        Err(_) => Ok(T::default()),
+    }
 }

crates/brk_cli/src/lib.rs

@@ -1,46 +1,167 @@
-use std::{fs, thread};
+#![doc = include_str!("../README.md")]
 
-use brk_core::{dot_brk_log_path, dot_brk_path};
-use brk_query::Params as QueryArgs;
-use clap::Parser;
-use clap_derive::{Parser, Subcommand};
-use query::query;
-use run::{RunConfig, run};
+use std::{
+    fs,
+    io::Cursor,
+    path::Path,
+    thread::{self, sleep},
+    time::Duration,
+};
 
-mod query;
-mod run;
+use bitcoincore_rpc::{self, RpcApi};
+use brk_binder::Bridge;
+use brk_bundler::bundle;
+use brk_computer::Computer;
+use brk_error::Result;
+use brk_indexer::Indexer;
+use brk_interface::Interface;
+use brk_parser::Parser;
+use brk_server::{Server, VERSION};
+use log::info;
+use vecdb::Exit;
 
-#[derive(Parser)]
-#[command(version, about)]
-#[command(propagate_version = true)]
-struct Cli {
-    #[command(subcommand)]
-    command: Commands,
-}
+mod config;
+mod paths;
+mod website;
 
-#[derive(Subcommand, Debug)]
-enum Commands {
-    /// Run the indexer, computer and server, use `run -h` for more information
-    Run(RunConfig),
-    /// Query generated datasets via the `run` command in a similar fashion as the server's API, use `query -h` for more information
-    Query(QueryArgs),
-}
+use crate::{config::Config, paths::*};
 
 pub fn main() -> color_eyre::Result<()> {
     color_eyre::install()?;
 
     fs::create_dir_all(dot_brk_path())?;
 
-    brk_logger::init(Some(&dot_brk_log_path()));
-
-    let cli = Cli::parse();
+    brk_logger::init(Some(&dot_brk_log_path()))?;
 
     thread::Builder::new()
-        .stack_size(128 * 1024 * 1024)
-        .spawn(|| match cli.command {
-            Commands::Run(args) => run(args),
-            Commands::Query(args) => query(args),
-        })?
+        .stack_size(512 * 1024 * 1024)
+        .spawn(run)?
         .join()
         .unwrap()
 }
+
+pub fn run() -> color_eyre::Result<()> {
+    let config = Config::import()?;
+
+    let rpc = config.rpc()?;
+
+    let exit = Exit::new();
+    exit.set_ctrlc_handler();
+
+    let parser = Parser::new(config.blocksdir(), rpc);
+
+    let mut indexer = Indexer::forced_import(&config.brkdir())?;
+
+    let mut computer = Computer::forced_import(&config.brkdir(), &indexer, config.fetcher())?;
+
+    let interface = Interface::build(&parser, &indexer, &computer);
+
+    let website = config.website();
+
+    let downloads_path = config.downloads_dir();
+
+    let future = async move {
+        let bundle_path = if website.is_some() {
+            let websites_dev_path = Path::new("../../websites");
+            let modules_dev_path = Path::new("../../modules");
+
+            let websites_path;
+            let modules_path;
+
+            if fs::exists(websites_dev_path)? && fs::exists(modules_dev_path)? {
+                websites_path = websites_dev_path.to_path_buf();
+                modules_path = modules_dev_path.to_path_buf();
+            } else {
+                let downloaded_brk_path = downloads_path.join(format!("brk-{VERSION}"));
+
+                let downloaded_websites_path = downloaded_brk_path.join("websites");
+                let downloaded_modules_path = downloaded_brk_path.join("modules");
+
+                if !fs::exists(&downloaded_websites_path)? {
+                    info!("Downloading source from Github...");
+
+                    let url = format!(
+                        "https://github.com/bitcoinresearchkit/brk/archive/refs/tags/v{VERSION}.zip",
+                    );
+
+                    let response = minreq::get(url).send()?;
+                    let bytes = response.as_bytes();
+                    let cursor = Cursor::new(bytes);
+
+                    let mut zip = zip::ZipArchive::new(cursor).unwrap();
+
+                    zip.extract(downloads_path).unwrap();
+                }
+
+                websites_path = downloaded_websites_path;
+                modules_path = downloaded_modules_path;
+            }
+
+            interface.generate_js_files(&modules_path)?;
+
+            Some(
+                bundle(
+                    &modules_path,
+                    &websites_path,
+                    website.to_folder_name(),
+                    true,
+                )
+                .await?,
+            )
+        } else {
+            None
+        };
+
+        let server = Server::new(interface, bundle_path);
+
+        tokio::spawn(async move {
+            server.serve(true).await.unwrap();
+        });
+
+        Ok(()) as Result<()>
+    };
+
+    let runtime = tokio::runtime::Builder::new_multi_thread()
+        .enable_all()
+        .build()?;
+
+    let _handle = runtime.spawn(future);
+
+    loop {
+        wait_for_synced_node(rpc)?;
+
+        let block_count = rpc.get_block_count()?;
+
+        info!("{} blocks found.", block_count + 1);
+
+        let starting_indexes = indexer
+            .index(&parser, rpc, &exit, config.check_collisions())
+            .unwrap();
+
+        computer
+            .compute(&indexer, starting_indexes, &parser, &exit)
+            .unwrap();
+
+        info!("Waiting for new blocks...");
+
+        while block_count == rpc.get_block_count()? {
+            sleep(Duration::from_secs(1))
+        }
+    }
+}
+
+fn wait_for_synced_node(rpc_client: &bitcoincore_rpc::Client) -> color_eyre::Result<()> {
+    let is_synced = || -> color_eyre::Result<bool> {
+        let info = rpc_client.get_blockchain_info()?;
+        Ok(info.headers == info.blocks)
+    };
+
+    if !is_synced()? {
+        info!("Waiting for node to sync...");
+        while !is_synced()? {
+            sleep(Duration::from_secs(1))
+        }
+    }
+
+    Ok(())
+}

crates/brk_cli/src/query.rs

@@ -1,55 +0,0 @@
-use brk_computer::Computer;
-use brk_indexer::Indexer;
-use brk_query::{Index, Output, Params as QueryParams, Query, Tabled, Value};
-use tabled::settings::Style;
-
-use crate::run::RunConfig;
-
-pub fn query(params: QueryParams) -> color_eyre::Result<()> {
-    let config = RunConfig::import(None)?;
-
-    let format = config.format();
-
-    let mut indexer = Indexer::new(&config.outputsdir(), config.check_collisions())?;
-    indexer.import_vecs()?;
-
-    let mut computer = Computer::new(&config.outputsdir(), config.fetcher(), format);
-    computer.import_vecs(&indexer, config.computation())?;
-
-    let query = Query::build(&indexer, &computer);
-
-    let index = Index::try_from(params.index.as_str())?;
-    let ids = params.values.iter().map(|s| s.as_str()).collect::<Vec<_>>();
-    let from = params.from();
-    let to = params.to();
-    let format = params.format();
-
-    let res = query.search_and_format(index, &ids, from, to, format)?;
-
-    if format.is_some() {
-        println!("{}", res);
-    } else {
-        println!(
-            "{}",
-            match res {
-                Output::Json(v) => match v {
-                    Value::Single(v) => v.to_string().replace("\"", ""),
-                    v => {
-                        let v = match v {
-                            Value::Single(_) => unreachable!("Already processed"),
-                            Value::List(v) => vec![v],
-                            Value::Matrix(v) => v,
-                        };
-                        let mut table =
-                            v.to_table(ids.iter().map(|id| id.to_string()).collect::<Vec<_>>());
-                        table.with(Style::psql());
-                        table.to_string()
-                    }
-                },
-                _ => unreachable!(),
-            }
-        );
-    }
-
-    Ok(())
-}

crates/brk_cli/src/website.rs

@@ -1,10 +1,11 @@
-use clap_derive::ValueEnum;
+use clap::ValueEnum;
 use serde::{Deserialize, Serialize};
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize, ValueEnum)]
+#[serde(rename_all = "lowercase")]
 pub enum Website {
     None,
-    Default,
+    Bitview,
     Custom,
 }
 
@@ -17,10 +18,10 @@ impl Website {
         !self.is_none()
     }
 
-    pub fn to_folder_name(&self) -> &str {
+    pub fn to_folder_name(self) -> &'static str {
         match self {
             Self::Custom => "custom",
-            Self::Default => "default",
+            Self::Bitview => "bitview",
             Self::None => unreachable!(),
         }
     }

crates/brk_computer/Cargo.toml

@@ -1,25 +1,30 @@
 [package]
 name = "brk_computer"
-description = "A Bitcoin dataset computer, built on top of brk_indexer"
+description = "A Bitcoin dataset computer built on top of brk_indexer"
 version.workspace = true
 edition.workspace = true
 license.workspace = true
 homepage.workspace = true
 repository.workspace = true
+rust-version.workspace = true
+build = "build.rs"
 
 [dependencies]
+allocative = { workspace = true }
 bitcoin = { workspace = true }
 bitcoincore-rpc = { workspace = true }
-brk_core = { workspace = true }
-brk_exit = { workspace = true }
+brk_structs = { workspace = true }
+brk_error = { workspace = true }
 brk_fetcher = { workspace = true }
 brk_indexer = { workspace = true }
 brk_logger = { workspace = true }
+brk_store = { workspace = true }
 brk_parser = { workspace = true }
-brk_state = { workspace = true }
-brk_vec = { workspace = true }
-color-eyre = { workspace = true }
-fjall = { workspace = true }
-jiff = { workspace = true }
+vecdb = { workspace = true }
+derive_deref = { workspace = true }
 log = { workspace = true }
+pco = "0.4.6"
 rayon = { workspace = true }
+serde = { workspace = true }
+zerocopy = { workspace = true }
+zerocopy-derive = { workspace = true }

crates/brk_computer/README.md

@@ -1,34 +1,303 @@
-# BRK Computer
+# brk_computer
 
-<p align="left">
-  <a href="https://github.com/bitcoinresearchkit/brk">
-    <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/bitcoinresearchkit/brk?style=social">
-  </a>
-  <a href="https://github.com/bitcoinresearchkit/brk/blob/main/LICENSE.md">
-    <img src="https://img.shields.io/crates/l/brk" alt="License" />
-  </a>
-  <a href="https://crates.io/crates/brk_computer">
-    <img src="https://img.shields.io/crates/v/brk_computer" alt="Version" />
-  </a>
-  <a href="https://docs.rs/brk_computer">
-    <img src="https://img.shields.io/docsrs/brk_computer" alt="Documentation" />
-  </a>
-  <img src="https://img.shields.io/crates/size/brk_computer" alt="Size" />
-  <a href="https://deps.rs/crate/brk_computer">
-    <img src="https://deps.rs/crate/brk_computer/latest/status.svg" alt="Dependency status">
-  </a>
-  <a href="https://discord.gg/HaR3wpH3nr">
-    <img src="https://img.shields.io/discord/1350431684562124850?label=discord" alt="Discord" />
-  </a>
-  <a href="https://primal.net/p/nprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6">
-    <img src="https://img.shields.io/badge/nostr-purple?link=https%3A%2F%2Fprimal.net%2Fp%2Fnprofile1qqsfw5dacngjlahye34krvgz7u0yghhjgk7gxzl5ptm9v6n2y3sn03sqxu2e6" alt="Nostr" />
-  </a>
-  <a href="https://bsky.app/profile/bitcoinresearchkit.org">
-    <img src="https://img.shields.io/badge/bluesky-blue?link=https%3A%2F%2Fbsky.app%2Fprofile%2Fbitcoinresearchkit.org" alt="Bluesky" />
-  </a>
-  <a href="https://x.com/brkdotorg">
-    <img src="https://img.shields.io/badge/x.com-black" alt="X" />
-  </a>
-</p>
+Advanced Bitcoin analytics engine that transforms indexed blockchain data into comprehensive metrics and financial analytics.
 
-A dataset computer, built on top of `brk_indexer` and `brk_fetcher`. It computes any dataset you can think of and if it doesn't feel free to create an issue.
+[![Crates.io](https://img.shields.io/crates/v/brk_computer.svg)](https://crates.io/crates/brk_computer)
+[![Documentation](https://docs.rs/brk_computer/badge.svg)](https://docs.rs/brk_computer)
+
+## Overview
+
+This crate provides a sophisticated analytics engine that processes indexed Bitcoin blockchain data to compute comprehensive metrics, financial analytics, and statistical aggregations. Built on top of `brk_indexer`, it transforms raw blockchain data into actionable insights through state tracking, cohort analysis, market metrics, and advanced Bitcoin-specific calculations.
+
+**Key Features:**
+
+- Comprehensive Bitcoin analytics pipeline with 6 major computation modules
+- UTXO and address cohort analysis with lifecycle tracking
+- Market metrics integration with price data and financial calculations
+- Cointime economics and realized/unrealized profit/loss analysis
+- Supply dynamics and monetary policy metrics
+- Pool analysis for centralization and mining statistics
+- Memory allocation tracking and performance optimization
+- Parallel computation with multi-threaded processing
+
+**Target Use Cases:**
+
+- Bitcoin market analysis and research platforms
+- On-chain analytics for investment and trading decisions
+- Academic research requiring comprehensive blockchain metrics
+- Financial applications needing Bitcoin exposure and risk metrics
+
+## Installation
+
+```bash
+cargo add brk_computer
+```
+
+## Quick Start
+
+```rust
+use brk_computer::Computer;
+use brk_indexer::Indexer;
+use brk_fetcher::Fetcher;
+use vecdb::Exit;
+use std::path::Path;
+
+// Initialize dependencies
+let outputs_path = Path::new("./analytics_data");
+let indexer = Indexer::forced_import(outputs_path)?;
+let fetcher = Some(Fetcher::import(true, None)?);
+
+// Create computer with price data support
+let mut computer = Computer::forced_import(outputs_path, &indexer, fetcher)?;
+
+// Compute analytics from indexer state
+let exit = Exit::default();
+let starting_indexes = brk_indexer::Indexes::default();
+computer.compute(&indexer, starting_indexes, &exit)?;
+
+println!("Analytics computation completed!");
+```
+
+## API Overview
+
+### Core Structure
+
+The Computer is organized into 7 specialized computation modules:
+
+- **`indexes`**: Fundamental blockchain index computations
+- **`constants`**: Network constants and protocol parameters
+- **`market`**: Price-based financial metrics and market analysis
+- **`pools`**: Mining pool analysis and centralization metrics
+- **`chain`**: Core blockchain metrics (difficulty, hashrate, fees)
+- **`stateful`**: Advanced state tracking (UTXO lifecycles, address behaviors)
+- **`cointime`**: Cointime economics and value-time calculations
+
+### Key Methods
+
+**`Computer::forced_import(outputs_path, indexer, fetcher) -> Result<Self>`**
+Creates computer instance with optional price data integration.
+
+**`compute(&mut self, indexer: &Indexer, starting_indexes: Indexes, exit: &Exit) -> Result<()>`**
+Main computation pipeline processing all analytics modules.
+
+### Analytics Categories
+
+**Market Analytics:**
+
+- Price-based metrics (market cap, realized cap, MVRV)
+- Trading volume analysis and liquidity metrics
+- Return calculations and volatility measurements
+- Dollar-cost averaging and investment strategy metrics
+
+**On-Chain Analytics:**
+
+- Transaction count and size statistics
+- Fee analysis and block space utilization
+- Address activity and entity clustering
+- UTXO age distributions and spending patterns
+
+**Monetary Analytics:**
+
+- Circulating supply and issuance tracking
+- Realized vs. unrealized gains/losses
+- Cointime destruction and accumulation
+- Velocity and economic activity indicators
+
+## Examples
+
+### Basic Analytics Computation
+
+```rust
+use brk_computer::Computer;
+
+// Initialize with indexer and optional price data
+let computer = Computer::forced_import(
+    "./analytics_output",
+    &indexer,
+    Some(price_fetcher)
+)?;
+
+// Compute all analytics modules
+let exit = vecdb::Exit::default();
+computer.compute(&indexer, starting_indexes, &exit)?;
+
+// Access computed metrics
+println!("Market cap vectors computed: {}", computer.market.len());
+println!("Chain metrics computed: {}", computer.chain.len());
+println!("Stateful analysis completed: {}", computer.stateful.len());
+```
+
+### Market Analysis
+
+```rust
+use brk_computer::Computer;
+use brk_structs::{DateIndex, Height};
+
+let computer = Computer::forced_import(/* ... */)?;
+
+// Access market metrics after computation
+if let Some(market) = &computer.market {
+    // Daily market cap analysis
+    let date_index = DateIndex::from_days_since_genesis(5000);
+    if let Some(market_cap) = market.dateindex_to_market_cap.get(date_index)? {
+        println!("Market cap on day {}: ${}", date_index, market_cap.to_dollars());
+    }
+
+    // MVRV (Market Value to Realized Value) ratio
+    if let Some(mvrv) = market.dateindex_to_mvrv.get(date_index)? {
+        println!("MVRV ratio: {:.2}", mvrv);
+    }
+}
+
+// Chain-level metrics
+let height = Height::new(800000);
+if let Some(difficulty) = computer.chain.height_to_difficulty.get(height)? {
+    println!("Network difficulty at height {}: {}", height, difficulty);
+}
+```
+
+### Cohort Analysis
+
+```rust
+use brk_computer::Computer;
+use brk_structs::{DateIndex, CohortId};
+
+let computer = Computer::forced_import(/* ... */)?;
+
+// Address cohort analysis
+let cohort_date = DateIndex::from_days_since_genesis(4000);
+
+// Analyze address behavior patterns
+if let Some(address_cohorts) = &computer.stateful.address_cohorts {
+    for cohort_id in address_cohorts.get_cohort_ids_for_date(cohort_date)? {
+        let cohort_data = address_cohorts.get_cohort(cohort_id)?;
+
+        println!("Cohort {}: {} addresses created",
+                 cohort_id, cohort_data.addresses.len());
+        println!("Average holding period: {} days",
+                 cohort_data.avg_holding_period.as_days());
+    }
+}
+
+// UTXO cohort lifecycle analysis
+if let Some(utxo_cohorts) = &computer.stateful.utxo_cohorts {
+    let active_utxos = utxo_cohorts.get_active_utxos_for_date(cohort_date)?;
+    println!("Active UTXOs from cohort: {}", active_utxos.len());
+}
+```
+
+### Supply and Monetary Analysis
+
+```rust
+use brk_computer::Computer;
+use brk_structs::{Height, DateIndex};
+
+let computer = Computer::forced_import(/* ... */)?;
+
+// Supply dynamics
+let height = Height::new(750000);
+if let Some(supply) = computer.chain.height_to_circulating_supply.get(height)? {
+    println!("Circulating supply: {} BTC", supply.to_btc());
+}
+
+// Realized vs unrealized analysis
+let date = DateIndex::from_days_since_genesis(5000);
+if let Some(realized_cap) = computer.market.dateindex_to_realized_cap.get(date)? {
+    if let Some(market_cap) = computer.market.dateindex_to_market_cap.get(date)? {
+        let unrealized_pnl = market_cap - realized_cap;
+        println!("Unrealized P&L: ${:.2}B", unrealized_pnl.to_dollars() / 1e9);
+    }
+}
+```
+
+## Architecture
+
+### Computation Pipeline
+
+The computer implements a sophisticated multi-stage pipeline:
+
+1. **Index Computation**: Fundamental blockchain metrics and time-based indexes
+2. **Constants Computation**: Network parameters and protocol constants
+3. **Price Integration**: Optional price data fetching and processing
+4. **Parallel Computation**: Chain, market, pools, stateful, and cointime analytics
+5. **Cross-Dependencies**: Advanced metrics requiring multiple data sources
+
+### Memory Management
+
+**Allocation Tracking:**
+
+- `allocative` integration for memory usage analysis
+- Efficient vector storage with compression options
+- Strategic lazy vs. eager evaluation for memory optimization
+
+**Performance Optimization:**
+
+- `rayon` parallel processing for CPU-intensive calculations
+- Vectorized operations for time-series computations
+- Memory-mapped storage for large datasets
+
+### State Management
+
+**Stateful Analytics:**
+
+- UTXO lifecycle tracking with creation/destruction events
+- Address cohort analysis with behavioral clustering
+- Transaction pattern recognition and anomaly detection
+- Economic cycle analysis with market phase detection
+
+**Cointime Economics:**
+
+- Bitcoin days destroyed and accumulated calculations
+- Velocity measurements and economic activity indicators
+- Age-weighted value transfer analysis
+- Long-term holder vs. active trader segmentation
+
+### Modular Design
+
+Each computation module operates independently:
+
+- **Chain Module**: Basic blockchain metrics (fees, difficulty, hashrate)
+- **Market Module**: Price-dependent financial calculations
+- **Pools Module**: Mining centralization and pool analysis
+- **Stateful Module**: Advanced lifecycle and behavior tracking
+- **Cointime Module**: Economic time-value calculations
+
+### Data Dependencies
+
+**Required Dependencies:**
+
+- `brk_indexer`: Raw blockchain data access
+- `brk_structs`: Type definitions and conversions
+
+**Optional Dependencies:**
+
+- `brk_fetcher`: Price data for financial metrics
+- Market analysis requires price integration
+
+### Computation Orchestration
+
+**Sequential Stages:**
+
+1. Indexes → Constants (foundational metrics)
+2. Fetched → Price (price data processing)
+3. Parallel: Chain, Market, Pools, Stateful, Cointime
+
+**Exit Handling:**
+
+- Graceful shutdown with consistent state preservation
+- Checkpoint-based recovery for long-running computations
+- Multi-threaded coordination with exit signaling
+
+## Code Analysis Summary
+
+**Main Structure**: `Computer` struct coordinating 7 specialized analytics modules (indexes, constants, market, pools, chain, stateful, cointime) \
+**Computation Pipeline**: Multi-stage analytics processing with parallel execution and dependency management \
+**State Tracking**: Advanced UTXO and address lifecycle analysis with cohort-based behavioral clustering \
+**Financial Analytics**: Comprehensive market metrics including realized/unrealized analysis and cointime economics \
+**Memory Optimization**: `allocative` tracking with lazy/eager evaluation strategies and compressed vector storage \
+**Parallel Processing**: `rayon` integration for CPU-intensive calculations with coordinated exit handling \
+**Architecture**: Modular analytics engine transforming indexed blockchain data into actionable financial and economic insights
+
+---
+
+_This README was generated by Claude Code_

crates/brk_computer/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    let profile = std::env::var("PROFILE").unwrap_or_default();
+
+    if profile == "release" {
+        println!("cargo:rustc-flag=-C");
+        println!("cargo:rustc-flag=target-cpu=native");
+    }
+}

crates/brk_computer/examples/computer.rs

@@ -0,0 +1,55 @@
+use std::{
+    path::Path,
+    thread::{self, sleep},
+    time::{Duration, Instant},
+};
+
+use brk_computer::Computer;
+use brk_error::Result;
+use brk_fetcher::Fetcher;
+use brk_indexer::Indexer;
+use brk_parser::Parser;
+use vecdb::Exit;
+
+pub fn main() -> Result<()> {
+    brk_logger::init(Some(Path::new(".log")))?;
+
+    let bitcoin_dir = Path::new(&std::env::var("HOME").unwrap())
+        .join("Library")
+        .join("Application Support")
+        .join("Bitcoin");
+    // let bitcoin_dir = Path::new("/Volumes/WD_BLACK/bitcoin");
+
+    let rpc = Box::leak(Box::new(bitcoincore_rpc::Client::new(
+        "http://localhost:8332",
+        bitcoincore_rpc::Auth::CookieFile(bitcoin_dir.join(".cookie")),
+    )?));
+    let exit = Exit::new();
+    exit.set_ctrlc_handler();
+
+    // Can't increase main thread's stack size, thus we need to use another thread
+    thread::Builder::new()
+        .stack_size(256 * 1024 * 1024)
+        .spawn(move || -> Result<()> {
+            let outputs_dir = Path::new(&std::env::var("HOME").unwrap()).join(".brk");
+            // let outputs_dir = Path::new("../../_outputs");
+
+            let parser = Parser::new(bitcoin_dir.join("blocks"), rpc);
+
+            let mut indexer = Indexer::forced_import(&outputs_dir)?;
+
+            let fetcher = Fetcher::import(true, None)?;
+
+            let mut computer = Computer::forced_import(&outputs_dir, &indexer, Some(fetcher))?;
+
+            loop {
+                let i = Instant::now();
+                let starting_indexes = indexer.index(&parser, rpc, &exit, true)?;
+                computer.compute(&indexer, starting_indexes, &parser, &exit)?;
+                dbg!(i.elapsed());
+                sleep(Duration::from_secs(10));
+            }
+        })?
+        .join()
+        .unwrap()
+}

crates/brk_computer/examples/main.rs

@@ -1,54 +0,0 @@
-use std::{path::Path, thread};
-
-use brk_computer::Computer;
-use brk_core::{default_bitcoin_path, default_brk_path};
-use brk_exit::Exit;
-use brk_fetcher::Fetcher;
-use brk_indexer::Indexer;
-use brk_parser::Parser;
-use brk_vec::{Computation, Format};
-
-pub fn main() -> color_eyre::Result<()> {
-    color_eyre::install()?;
-
-    brk_logger::init(Some(Path::new(".log")));
-
-    let bitcoin_dir = default_bitcoin_path();
-
-    let rpc = Box::leak(Box::new(bitcoincore_rpc::Client::new(
-        "http://localhost:8332",
-        bitcoincore_rpc::Auth::CookieFile(bitcoin_dir.join(".cookie")),
-    )?));
-    let exit = Exit::new();
-
-    // Can't increase main thread's stack programatically, thus we need to use another thread
-    thread::Builder::new()
-        .stack_size(32 * 1024 * 1024)
-        .spawn(move || -> color_eyre::Result<()> {
-            let parser = Parser::new(bitcoin_dir.join("blocks"), rpc);
-
-            let _outputs_dir = default_brk_path().join("outputs");
-            let outputs_dir = _outputs_dir.as_path();
-            // let outputs_dir = Path::new("../../_outputs");
-
-            let format = Format::Raw;
-
-            let mut indexer = Indexer::new(outputs_dir, true)?;
-            indexer.import_stores()?;
-            indexer.import_vecs()?;
-
-            let fetcher = Fetcher::import(None)?;
-
-            let mut computer = Computer::new(outputs_dir, Some(fetcher), format);
-            computer.import_stores(&indexer)?;
-            computer.import_vecs(&indexer, Computation::Lazy)?;
-
-            let starting_indexes = indexer.index(&parser, rpc, &exit)?;
-
-            computer.compute(&mut indexer, starting_indexes, &exit)?;
-
-            Ok(())
-        })?
-        .join()
-        .unwrap()
-}

crates/brk_computer/examples/pools.rs

@@ -0,0 +1,125 @@
+use std::{collections::BTreeMap, path::Path, thread};
+
+use brk_computer::Computer;
+use brk_error::Result;
+use brk_fetcher::Fetcher;
+use brk_indexer::Indexer;
+use brk_structs::{AddressBytes, OutputIndex, OutputType, pools};
+use vecdb::{AnyIterableVec, Exit, VecIterator};
+
+fn main() -> Result<()> {
+    brk_logger::init(Some(Path::new(".log")))?;
+
+    let exit = Exit::new();
+    exit.set_ctrlc_handler();
+
+    thread::Builder::new()
+        .stack_size(256 * 1024 * 1024)
+        .spawn(move || -> Result<()> {
+            let outputs_dir = Path::new(&std::env::var("HOME").unwrap()).join(".brk");
+
+            let indexer = Indexer::forced_import(&outputs_dir)?;
+
+            let fetcher = Fetcher::import(true, None)?;
+
+            let computer = Computer::forced_import(&outputs_dir, &indexer, Some(fetcher))?;
+
+            let pools = pools();
+
+            let mut res: BTreeMap<&'static str, usize> = BTreeMap::default();
+
+            let vecs = indexer.vecs;
+            let stores = indexer.stores;
+
+            let mut height_to_first_txindex_iter = vecs.height_to_first_txindex.iter();
+            let mut txindex_to_first_outputindex_iter = vecs.txindex_to_first_outputindex.iter();
+            let mut txindex_to_output_count_iter = computer.indexes.txindex_to_output_count.iter();
+            let mut outputindex_to_outputtype_iter = vecs.outputindex_to_outputtype.iter();
+            let mut outputindex_to_typeindex_iter = vecs.outputindex_to_typeindex.iter();
+            let mut p2pk65addressindex_to_p2pk65bytes_iter =
+                vecs.p2pk65addressindex_to_p2pk65bytes.iter();
+            let mut p2pk33addressindex_to_p2pk33bytes_iter =
+                vecs.p2pk33addressindex_to_p2pk33bytes.iter();
+            let mut p2pkhaddressindex_to_p2pkhbytes_iter =
+                vecs.p2pkhaddressindex_to_p2pkhbytes.iter();
+            let mut p2shaddressindex_to_p2shbytes_iter = vecs.p2shaddressindex_to_p2shbytes.iter();
+            let mut p2wpkhaddressindex_to_p2wpkhbytes_iter =
+                vecs.p2wpkhaddressindex_to_p2wpkhbytes.iter();
+            let mut p2wshaddressindex_to_p2wshbytes_iter =
+                vecs.p2wshaddressindex_to_p2wshbytes.iter();
+            let mut p2traddressindex_to_p2trbytes_iter = vecs.p2traddressindex_to_p2trbytes.iter();
+            let mut p2aaddressindex_to_p2abytes_iter = vecs.p2aaddressindex_to_p2abytes.iter();
+
+            let unknown = pools.get_unknown();
+
+            stores
+                .height_to_coinbase_tag
+                .iter()
+                .for_each(|(height, coinbase_tag)| {
+                    let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+                    let outputindex = txindex_to_first_outputindex_iter.unwrap_get_inner(txindex);
+                    let outputcount = txindex_to_output_count_iter.unwrap_get_inner(txindex);
+
+                    let pool = (*outputindex..(*outputindex + *outputcount))
+                        .map(OutputIndex::from)
+                        .find_map(|outputindex| {
+                            let outputtype =
+                                outputindex_to_outputtype_iter.unwrap_get_inner(outputindex);
+                            let typeindex =
+                                outputindex_to_typeindex_iter.unwrap_get_inner(outputindex);
+
+                            let address = match outputtype {
+                                OutputType::P2PK65 => Some(AddressBytes::from(
+                                    p2pk65addressindex_to_p2pk65bytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2PK33 => Some(AddressBytes::from(
+                                    p2pk33addressindex_to_p2pk33bytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2PKH => Some(AddressBytes::from(
+                                    p2pkhaddressindex_to_p2pkhbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2SH => Some(AddressBytes::from(
+                                    p2shaddressindex_to_p2shbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2WPKH => Some(AddressBytes::from(
+                                    p2wpkhaddressindex_to_p2wpkhbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2WSH => Some(AddressBytes::from(
+                                    p2wshaddressindex_to_p2wshbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2TR => Some(AddressBytes::from(
+                                    p2traddressindex_to_p2trbytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                OutputType::P2A => Some(AddressBytes::from(
+                                    p2aaddressindex_to_p2abytes_iter
+                                        .unwrap_get_inner(typeindex.into()),
+                                )),
+                                _ => None,
+                            };
+
+                            address
+                                .and_then(|address| pools.find_from_address(&address.to_string()))
+                        })
+                        .or_else(|| pools.find_from_coinbase_tag(&coinbase_tag))
+                        .unwrap_or(unknown);
+
+                    *res.entry(pool.name).or_default() += 1;
+                });
+
+            let mut v = res.into_iter().map(|(k, v)| (v, k)).collect::<Vec<_>>();
+            v.sort_unstable();
+            println!("{:#?}", v);
+            println!("{:#?}", v.len());
+
+            Ok(())
+        })?
+        .join()
+        .unwrap()
+}

crates/brk_computer/examples/price_to_amount.rs

@@ -0,0 +1,15 @@
+use std::path::Path;
+
+use brk_computer::PriceToAmount;
+use brk_error::Result;
+use brk_structs::Height;
+
+pub fn main() -> Result<()> {
+    let path = Path::new(&std::env::var("HOME").unwrap())
+        .join(".brk")
+        .join("computed/stateful/states");
+    let mut price_to_amount = PriceToAmount::create(&path, "addrs_above_1btc_under_10btc");
+    dbg!(price_to_amount.import_at_or_before(Height::new(890000))?);
+    dbg!(price_to_amount);
+    Ok(())
+}

crates/brk_computer/flamegraph.sh

@@ -0,0 +1,2 @@
+cargo build --profile profiling
+flamegraph -- ../../target/profiling/examples/main

crates/brk_computer/samply.sh

@@ -0,0 +1,2 @@
+cargo build --example main --profile profiling
+samply record ../../target/profiling/examples/main

crates/brk_computer/src/blks.rs

@@ -0,0 +1,141 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_indexer::Indexer;
+use brk_parser::Parser;
+use brk_structs::{BlkPosition, Height, TxIndex, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, CompressedVec, Database, Exit,
+    GenericStoredVec, PAGE_SIZE, VecIterator,
+};
+
+use super::{Indexes, indexes};
+
+#[derive(Clone)]
+pub struct Vecs {
+    db: Database,
+
+    pub height_to_position: CompressedVec<Height, BlkPosition>,
+    pub txindex_to_position: CompressedVec<TxIndex, BlkPosition>,
+}
+
+impl Vecs {
+    pub fn forced_import(parent_path: &Path, parent_version: Version) -> Result<Self> {
+        let db = Database::open(&parent_path.join("blks"))?;
+        db.set_min_len(PAGE_SIZE * 1_000_000)?;
+
+        let version = parent_version + Version::ZERO;
+
+        let this = Self {
+            height_to_position: CompressedVec::forced_import(
+                &db,
+                "position",
+                version + Version::TWO,
+            )?,
+            txindex_to_position: CompressedVec::forced_import(
+                &db,
+                "position",
+                version + Version::TWO,
+            )?,
+
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        parser: &Parser,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexer, indexes, starting_indexes, parser, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        parser: &Parser,
+        exit: &Exit,
+    ) -> Result<()> {
+        let min_txindex =
+            TxIndex::from(self.txindex_to_position.len()).min(starting_indexes.txindex);
+
+        let Some(min_height) = indexes
+            .txindex_to_height
+            .iter()
+            .get_inner(min_txindex)
+            .map(|h| h.min(starting_indexes.height))
+        else {
+            return Ok(());
+        };
+
+        let mut height_to_first_txindex_iter = indexer.vecs.height_to_first_txindex.iter();
+
+        parser
+            .parse(
+                Some(min_height),
+                Some((indexer.vecs.height_to_first_txindex.len() - 1).into()),
+            )
+            .iter()
+            .try_for_each(|block| -> Result<()> {
+                let height = block.height();
+
+                self.height_to_position.forced_push_at(
+                    height,
+                    block.metadata().position(),
+                    exit,
+                )?;
+
+                let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+
+                block.tx_metadata().iter().enumerate().try_for_each(
+                    |(index, metadata)| -> Result<()> {
+                        let txindex = txindex + index;
+                        self.txindex_to_position.forced_push_at(
+                            txindex,
+                            metadata.position(),
+                            exit,
+                        )?;
+                        Ok(())
+                    },
+                )?;
+
+                if *height % 1_000 == 0 {
+                    let _lock = exit.lock();
+                    self.height_to_position.flush()?;
+                    self.txindex_to_position.flush()?;
+                }
+                Ok(())
+            })?;
+
+        let _lock = exit.lock();
+        self.height_to_position.flush()?;
+        self.txindex_to_position.flush()?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        Box::new(
+            [
+                &self.height_to_position as &dyn AnyCollectableVec,
+                &self.txindex_to_position,
+            ]
+            .into_iter(),
+        )
+    }
+}

crates/brk_computer/src/cointime.rs

@@ -0,0 +1,749 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_structs::{Bitcoin, CheckedSub, Dollars, StoredF32, StoredF64, Version};
+use vecdb::{AnyCollectableVec, Database, Exit, PAGE_SIZE, VecIterator};
+
+use crate::grouped::ComputedVecsFromDateIndex;
+
+use super::{
+    Indexes, chain,
+    grouped::{
+        ComputedRatioVecsFromDateIndex, ComputedValueVecsFromHeight, ComputedVecsFromHeight,
+        Source, VecBuilderOptions,
+    },
+    indexes, price, stateful,
+};
+
+#[derive(Clone)]
+pub struct Vecs {
+    db: Database,
+
+    pub indexes_to_coinblocks_created: ComputedVecsFromHeight<StoredF64>,
+    pub indexes_to_coinblocks_stored: ComputedVecsFromHeight<StoredF64>,
+    pub indexes_to_liveliness: ComputedVecsFromHeight<StoredF64>,
+    pub indexes_to_vaultedness: ComputedVecsFromHeight<StoredF64>,
+    pub indexes_to_activity_to_vaultedness_ratio: ComputedVecsFromHeight<StoredF64>,
+    pub indexes_to_vaulted_supply: ComputedValueVecsFromHeight,
+    pub indexes_to_active_supply: ComputedValueVecsFromHeight,
+    pub indexes_to_thermo_cap: ComputedVecsFromHeight<Dollars>,
+    pub indexes_to_investor_cap: ComputedVecsFromHeight<Dollars>,
+    pub indexes_to_vaulted_cap: ComputedVecsFromHeight<Dollars>,
+    pub indexes_to_active_cap: ComputedVecsFromHeight<Dollars>,
+    pub indexes_to_vaulted_price: ComputedVecsFromHeight<Dollars>,
+    pub indexes_to_vaulted_price_ratio: ComputedRatioVecsFromDateIndex,
+    pub indexes_to_active_price: ComputedVecsFromHeight<Dollars>,
+    pub indexes_to_active_price_ratio: ComputedRatioVecsFromDateIndex,
+    pub indexes_to_true_market_mean: ComputedVecsFromHeight<Dollars>,
+    pub indexes_to_true_market_mean_ratio: ComputedRatioVecsFromDateIndex,
+    pub indexes_to_cointime_value_destroyed: ComputedVecsFromHeight<StoredF64>,
+    pub indexes_to_cointime_value_created: ComputedVecsFromHeight<StoredF64>,
+    pub indexes_to_cointime_value_stored: ComputedVecsFromHeight<StoredF64>,
+    pub indexes_to_cointime_price: ComputedVecsFromHeight<Dollars>,
+    pub indexes_to_cointime_cap: ComputedVecsFromHeight<Dollars>,
+    pub indexes_to_cointime_price_ratio: ComputedRatioVecsFromDateIndex,
+    pub indexes_to_cointime_adj_inflation_rate: ComputedVecsFromDateIndex<StoredF32>,
+    pub indexes_to_cointime_adj_tx_btc_velocity: ComputedVecsFromDateIndex<StoredF64>,
+    pub indexes_to_cointime_adj_tx_usd_velocity: ComputedVecsFromDateIndex<StoredF64>,
+    // pub indexes_to_thermo_cap_rel_to_investor_cap: ComputedValueVecsFromHeight,
+}
+
+impl Vecs {
+    pub fn forced_import(
+        parent_path: &Path,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+    ) -> Result<Self> {
+        let db = Database::open(&parent_path.join("cointime"))?;
+        db.set_min_len(PAGE_SIZE * 1_000_000)?;
+
+        let compute_dollars = price.is_some();
+
+        let version = parent_version + Version::ZERO;
+
+        let this = Self {
+            indexes_to_coinblocks_created: ComputedVecsFromHeight::forced_import(
+                &db,
+                "coinblocks_created",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+            )?,
+            indexes_to_coinblocks_stored: ComputedVecsFromHeight::forced_import(
+                &db,
+                "coinblocks_stored",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+            )?,
+            indexes_to_liveliness: ComputedVecsFromHeight::forced_import(
+                &db,
+                "liveliness",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_vaultedness: ComputedVecsFromHeight::forced_import(
+                &db,
+                "vaultedness",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_activity_to_vaultedness_ratio: ComputedVecsFromHeight::forced_import(
+                &db,
+                "activity_to_vaultedness_ratio",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_vaulted_supply: ComputedValueVecsFromHeight::forced_import(
+                &db,
+                "vaulted_supply",
+                Source::Compute,
+                version + Version::ONE,
+                VecBuilderOptions::default().add_last(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_active_supply: ComputedValueVecsFromHeight::forced_import(
+                &db,
+                "active_supply",
+                Source::Compute,
+                version + Version::ONE,
+                VecBuilderOptions::default().add_last(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_thermo_cap: ComputedVecsFromHeight::forced_import(
+                &db,
+                "thermo_cap",
+                Source::Compute,
+                version + Version::ONE,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_investor_cap: ComputedVecsFromHeight::forced_import(
+                &db,
+                "investor_cap",
+                Source::Compute,
+                version + Version::ONE,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_vaulted_cap: ComputedVecsFromHeight::forced_import(
+                &db,
+                "vaulted_cap",
+                Source::Compute,
+                version + Version::ONE,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_active_cap: ComputedVecsFromHeight::forced_import(
+                &db,
+                "active_cap",
+                Source::Compute,
+                version + Version::ONE,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_vaulted_price: ComputedVecsFromHeight::forced_import(
+                &db,
+                "vaulted_price",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_vaulted_price_ratio: ComputedRatioVecsFromDateIndex::forced_import(
+                &db,
+                "vaulted_price",
+                Source::None,
+                version + Version::ZERO,
+                indexes,
+                true,
+            )?,
+            indexes_to_active_price: ComputedVecsFromHeight::forced_import(
+                &db,
+                "active_price",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_active_price_ratio: ComputedRatioVecsFromDateIndex::forced_import(
+                &db,
+                "active_price",
+                Source::None,
+                version + Version::ZERO,
+                indexes,
+                true,
+            )?,
+            indexes_to_true_market_mean: ComputedVecsFromHeight::forced_import(
+                &db,
+                "true_market_mean",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_true_market_mean_ratio: ComputedRatioVecsFromDateIndex::forced_import(
+                &db,
+                "true_market_mean",
+                Source::None,
+                version + Version::ZERO,
+                indexes,
+                true,
+            )?,
+            indexes_to_cointime_value_destroyed: ComputedVecsFromHeight::forced_import(
+                &db,
+                "cointime_value_destroyed",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+            )?,
+            indexes_to_cointime_value_created: ComputedVecsFromHeight::forced_import(
+                &db,
+                "cointime_value_created",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+            )?,
+            indexes_to_cointime_value_stored: ComputedVecsFromHeight::forced_import(
+                &db,
+                "cointime_value_stored",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+            )?,
+            indexes_to_cointime_price: ComputedVecsFromHeight::forced_import(
+                &db,
+                "cointime_price",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_cointime_cap: ComputedVecsFromHeight::forced_import(
+                &db,
+                "cointime_cap",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_cointime_price_ratio: ComputedRatioVecsFromDateIndex::forced_import(
+                &db,
+                "cointime_price",
+                Source::None,
+                version + Version::ZERO,
+                indexes,
+                true,
+            )?,
+            indexes_to_cointime_adj_inflation_rate: ComputedVecsFromDateIndex::forced_import(
+                &db,
+                "cointime_adj_inflation_rate",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_cointime_adj_tx_btc_velocity: ComputedVecsFromDateIndex::forced_import(
+                &db,
+                "cointime_adj_tx_btc_velocity",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_cointime_adj_tx_usd_velocity: ComputedVecsFromDateIndex::forced_import(
+                &db,
+                "cointime_adj_tx_usd_velocity",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn compute(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        price: Option<&price::Vecs>,
+        chain: &chain::Vecs,
+        stateful: &stateful::Vecs,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexes, starting_indexes, price, chain, stateful, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    fn compute_(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        price: Option<&price::Vecs>,
+        chain: &chain::Vecs,
+        stateful: &stateful::Vecs,
+        exit: &Exit,
+    ) -> Result<()> {
+        let circulating_supply = &stateful.utxo_cohorts.all.1.height_to_supply;
+
+        self.indexes_to_coinblocks_created
+            .compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_transform(
+                    starting_indexes.height,
+                    circulating_supply,
+                    |(i, v, ..)| (i, StoredF64::from(Bitcoin::from(v))),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        let indexes_to_coinblocks_destroyed =
+            &stateful.utxo_cohorts.all.1.indexes_to_coinblocks_destroyed;
+
+        self.indexes_to_coinblocks_stored
+            .compute_all(indexes, starting_indexes, exit, |vec| {
+                let mut coinblocks_destroyed_iter = indexes_to_coinblocks_destroyed
+                    .height
+                    .as_ref()
+                    .unwrap()
+                    .into_iter();
+                vec.compute_transform(
+                    starting_indexes.height,
+                    self.indexes_to_coinblocks_created.height.as_ref().unwrap(),
+                    |(i, created, ..)| {
+                        let destroyed = coinblocks_destroyed_iter.unwrap_get_inner(i);
+                        (i, created.checked_sub(destroyed).unwrap())
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_liveliness
+            .compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_divide(
+                    starting_indexes.height,
+                    indexes_to_coinblocks_destroyed
+                        .height_extra
+                        .unwrap_cumulative(),
+                    self.indexes_to_coinblocks_created
+                        .height_extra
+                        .unwrap_cumulative(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+        let liveliness = &self.indexes_to_liveliness;
+
+        self.indexes_to_vaultedness
+            .compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_transform(
+                    starting_indexes.height,
+                    liveliness.height.as_ref().unwrap(),
+                    |(i, v, ..)| (i, StoredF64::from(1.0).checked_sub(v).unwrap()),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+        let vaultedness = &self.indexes_to_vaultedness;
+
+        self.indexes_to_activity_to_vaultedness_ratio.compute_all(
+            indexes,
+            starting_indexes,
+            exit,
+            |vec| {
+                vec.compute_divide(
+                    starting_indexes.height,
+                    liveliness.height.as_ref().unwrap(),
+                    vaultedness.height.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            },
+        )?;
+
+        self.indexes_to_vaulted_supply.compute_all(
+            indexes,
+            price,
+            starting_indexes,
+            exit,
+            |vec| {
+                vec.compute_multiply(
+                    starting_indexes.height,
+                    circulating_supply,
+                    vaultedness.height.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            },
+        )?;
+
+        self.indexes_to_active_supply.compute_all(
+            indexes,
+            price,
+            starting_indexes,
+            exit,
+            |vec| {
+                vec.compute_multiply(
+                    starting_indexes.height,
+                    circulating_supply,
+                    liveliness.height.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            },
+        )?;
+
+        self.indexes_to_cointime_adj_inflation_rate
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_multiply(
+                    starting_indexes.dateindex,
+                    self.indexes_to_activity_to_vaultedness_ratio
+                        .dateindex
+                        .unwrap_last(),
+                    chain.indexes_to_inflation_rate.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_cointime_adj_tx_btc_velocity
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_multiply(
+                    starting_indexes.dateindex,
+                    self.indexes_to_activity_to_vaultedness_ratio
+                        .dateindex
+                        .unwrap_last(),
+                    chain.indexes_to_tx_btc_velocity.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        if let Some(price) = price {
+            let realized_cap = stateful
+                .utxo_cohorts
+                .all
+                .1
+                .height_to_realized_cap
+                .as_ref()
+                .unwrap();
+
+            let realized_price = stateful
+                .utxo_cohorts
+                .all
+                .1
+                .indexes_to_realized_price
+                .as_ref()
+                .unwrap()
+                .height
+                .as_ref()
+                .unwrap();
+
+            self.indexes_to_thermo_cap
+                .compute_all(indexes, starting_indexes, exit, |vec| {
+                    vec.compute_transform(
+                        starting_indexes.height,
+                        chain
+                            .indexes_to_subsidy
+                            .dollars
+                            .as_ref()
+                            .unwrap()
+                            .height_extra
+                            .unwrap_cumulative(),
+                        |(i, v, ..)| (i, v),
+                        exit,
+                    )?;
+                    Ok(())
+                })?;
+
+            self.indexes_to_investor_cap
+                .compute_all(indexes, starting_indexes, exit, |vec| {
+                    vec.compute_subtract(
+                        starting_indexes.height,
+                        realized_cap,
+                        self.indexes_to_thermo_cap.height.as_ref().unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                })?;
+
+            self.indexes_to_vaulted_cap
+                .compute_all(indexes, starting_indexes, exit, |vec| {
+                    vec.compute_divide(
+                        starting_indexes.height,
+                        realized_cap,
+                        self.indexes_to_vaultedness.height.as_ref().unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                })?;
+
+            self.indexes_to_active_cap
+                .compute_all(indexes, starting_indexes, exit, |vec| {
+                    vec.compute_multiply(
+                        starting_indexes.height,
+                        realized_cap,
+                        self.indexes_to_liveliness.height.as_ref().unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                })?;
+
+            self.indexes_to_vaulted_price
+                .compute_all(indexes, starting_indexes, exit, |vec| {
+                    vec.compute_divide(
+                        starting_indexes.height,
+                        realized_price,
+                        self.indexes_to_vaultedness.height.as_ref().unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                })?;
+
+            self.indexes_to_vaulted_price_ratio.compute_rest(
+                price,
+                starting_indexes,
+                exit,
+                Some(self.indexes_to_vaulted_price.dateindex.unwrap_last()),
+            )?;
+
+            self.indexes_to_active_price
+                .compute_all(indexes, starting_indexes, exit, |vec| {
+                    vec.compute_multiply(
+                        starting_indexes.height,
+                        realized_price,
+                        self.indexes_to_liveliness.height.as_ref().unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                })?;
+
+            self.indexes_to_active_price_ratio.compute_rest(
+                price,
+                starting_indexes,
+                exit,
+                Some(self.indexes_to_active_price.dateindex.unwrap_last()),
+            )?;
+
+            self.indexes_to_true_market_mean.compute_all(
+                indexes,
+                starting_indexes,
+                exit,
+                |vec| {
+                    vec.compute_divide(
+                        starting_indexes.height,
+                        self.indexes_to_investor_cap.height.as_ref().unwrap(),
+                        self.indexes_to_active_supply
+                            .bitcoin
+                            .height
+                            .as_ref()
+                            .unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                },
+            )?;
+
+            self.indexes_to_true_market_mean_ratio.compute_rest(
+                price,
+                starting_indexes,
+                exit,
+                Some(self.indexes_to_true_market_mean.dateindex.unwrap_last()),
+            )?;
+
+            self.indexes_to_cointime_value_destroyed.compute_all(
+                indexes,
+                starting_indexes,
+                exit,
+                |vec| {
+                    // TODO: Another example when the callback should be applied to each index, instead of to base then merging from more granular to less
+                    // The price taken won't be correct for time based indexes
+                    vec.compute_multiply(
+                        starting_indexes.height,
+                        &price.chainindexes_to_price_close.height,
+                        indexes_to_coinblocks_destroyed.height.as_ref().unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                },
+            )?;
+
+            self.indexes_to_cointime_value_created.compute_all(
+                indexes,
+                starting_indexes,
+                exit,
+                |vec| {
+                    vec.compute_multiply(
+                        starting_indexes.height,
+                        &price.chainindexes_to_price_close.height,
+                        self.indexes_to_coinblocks_created.height.as_ref().unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                },
+            )?;
+
+            self.indexes_to_cointime_value_stored.compute_all(
+                indexes,
+                starting_indexes,
+                exit,
+                |vec| {
+                    vec.compute_multiply(
+                        starting_indexes.height,
+                        &price.chainindexes_to_price_close.height,
+                        self.indexes_to_coinblocks_stored.height.as_ref().unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                },
+            )?;
+
+            self.indexes_to_cointime_price
+                .compute_all(indexes, starting_indexes, exit, |vec| {
+                    vec.compute_divide(
+                        starting_indexes.height,
+                        self.indexes_to_cointime_value_destroyed
+                            .height_extra
+                            .unwrap_cumulative(),
+                        self.indexes_to_coinblocks_stored
+                            .height_extra
+                            .unwrap_cumulative(),
+                        exit,
+                    )?;
+                    Ok(())
+                })?;
+
+            self.indexes_to_cointime_cap
+                .compute_all(indexes, starting_indexes, exit, |vec| {
+                    vec.compute_multiply(
+                        starting_indexes.height,
+                        self.indexes_to_cointime_price.height.as_ref().unwrap(),
+                        circulating_supply,
+                        exit,
+                    )?;
+                    Ok(())
+                })?;
+
+            self.indexes_to_cointime_price_ratio.compute_rest(
+                price,
+                starting_indexes,
+                exit,
+                Some(self.indexes_to_cointime_price.dateindex.unwrap_last()),
+            )?;
+
+            self.indexes_to_cointime_adj_tx_usd_velocity.compute_all(
+                starting_indexes,
+                exit,
+                |v| {
+                    v.compute_multiply(
+                        starting_indexes.dateindex,
+                        self.indexes_to_activity_to_vaultedness_ratio
+                            .dateindex
+                            .unwrap_last(),
+                        chain.indexes_to_tx_usd_velocity.dateindex.as_ref().unwrap(),
+                        exit,
+                    )?;
+                    Ok(())
+                },
+            )?;
+        }
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
+
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_adj_inflation_rate
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_adj_tx_btc_velocity
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_adj_tx_usd_velocity
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(iter.chain(self.indexes_to_coinblocks_created.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_coinblocks_stored.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_liveliness.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_vaultedness.iter_any_collectable()));
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_activity_to_vaultedness_ratio
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_supply.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_supply.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_thermo_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_investor_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_price.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_vaulted_price_ratio.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_price.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_active_price_ratio.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_true_market_mean.iter_any_collectable()));
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_true_market_mean_ratio
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(iter.chain(self.indexes_to_cointime_price.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_cointime_cap.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_cointime_price_ratio.iter_any_collectable()));
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_value_destroyed
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.indexes_to_cointime_value_created
+                    .iter_any_collectable(),
+            ),
+        );
+        iter = Box::new(iter.chain(self.indexes_to_cointime_value_stored.iter_any_collectable()));
+        iter
+    }
+}

crates/brk_computer/src/constants.rs

@@ -0,0 +1,274 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_structs::{StoredF32, StoredI16, StoredU16, Version};
+use vecdb::{AnyCollectableVec, AnyVec, Database, Exit};
+
+use crate::grouped::Source;
+
+use super::{
+    Indexes,
+    grouped::{ComputedVecsFromHeight, VecBuilderOptions},
+    indexes,
+};
+
+#[derive(Clone)]
+pub struct Vecs {
+    db: Database,
+
+    pub constant_0: ComputedVecsFromHeight<StoredU16>,
+    pub constant_1: ComputedVecsFromHeight<StoredU16>,
+    pub constant_2: ComputedVecsFromHeight<StoredU16>,
+    pub constant_3: ComputedVecsFromHeight<StoredU16>,
+    pub constant_4: ComputedVecsFromHeight<StoredU16>,
+    pub constant_38_2: ComputedVecsFromHeight<StoredF32>,
+    pub constant_50: ComputedVecsFromHeight<StoredU16>,
+    pub constant_61_8: ComputedVecsFromHeight<StoredF32>,
+    pub constant_100: ComputedVecsFromHeight<StoredU16>,
+    pub constant_600: ComputedVecsFromHeight<StoredU16>,
+    pub constant_minus_1: ComputedVecsFromHeight<StoredI16>,
+    pub constant_minus_2: ComputedVecsFromHeight<StoredI16>,
+    pub constant_minus_3: ComputedVecsFromHeight<StoredI16>,
+    pub constant_minus_4: ComputedVecsFromHeight<StoredI16>,
+}
+
+impl Vecs {
+    pub fn forced_import(
+        parent_path: &Path,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+    ) -> Result<Self> {
+        let db = Database::open(&parent_path.join("constants"))?;
+
+        let version = parent_version + Version::ZERO;
+
+        let this = Self {
+            constant_0: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_0",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_1: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_1",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_2: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_2",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_3: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_3",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_4: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_4",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_38_2: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_38_2",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_50: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_50",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_61_8: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_61_8",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_100: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_100",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_600: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_600",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_minus_1: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_minus_1",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_minus_2: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_minus_2",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_minus_3: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_minus_3",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            constant_minus_4: ComputedVecsFromHeight::forced_import(
+                &db,
+                "constant_minus_4",
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexes, starting_indexes, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        [
+            (&mut self.constant_0, 0),
+            (&mut self.constant_1, 1),
+            (&mut self.constant_2, 2),
+            (&mut self.constant_3, 3),
+            (&mut self.constant_4, 4),
+            (&mut self.constant_50, 50),
+            (&mut self.constant_100, 100),
+            (&mut self.constant_600, 600),
+        ]
+        .into_iter()
+        .try_for_each(|(vec, value)| {
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_to(
+                    starting_indexes.height,
+                    indexes.height_to_date.len(),
+                    indexes.height_to_date.version(),
+                    |i| (i, StoredU16::new(value)),
+                    exit,
+                )?;
+                Ok(())
+            })
+        })?;
+
+        [
+            (&mut self.constant_minus_1, -1),
+            (&mut self.constant_minus_2, -2),
+            (&mut self.constant_minus_3, 3),
+            (&mut self.constant_minus_4, 4),
+        ]
+        .into_iter()
+        .try_for_each(|(vec, value)| {
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_to(
+                    starting_indexes.height,
+                    indexes.height_to_date.len(),
+                    indexes.height_to_date.version(),
+                    |i| (i, StoredI16::new(value)),
+                    exit,
+                )?;
+                Ok(())
+            })
+        })?;
+
+        [
+            (&mut self.constant_38_2, 38.2),
+            (&mut self.constant_61_8, 61.8),
+        ]
+        .into_iter()
+        .try_for_each(|(vec, value)| {
+            vec.compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_to(
+                    starting_indexes.height,
+                    indexes.height_to_date.len(),
+                    indexes.height_to_date.version(),
+                    |i| (i, StoredF32::from(value)),
+                    exit,
+                )?;
+                Ok(())
+            })
+        })?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
+
+        iter = Box::new(iter.chain(self.constant_0.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_1.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_2.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_3.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_4.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_38_2.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_50.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_61_8.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_100.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_600.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_1.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_2.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_3.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constant_minus_4.iter_any_collectable()));
+
+        iter
+    }
+}

crates/brk_computer/src/fetched.rs

@@ -0,0 +1,149 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_fetcher::Fetcher;
+use brk_indexer::Indexer;
+use brk_structs::{DateIndex, Height, OHLCCents, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, Exit, GenericStoredVec,
+    RawVec, StoredIndex, VecIterator,
+};
+
+use super::{Indexes, indexes};
+
+#[derive(Clone)]
+pub struct Vecs {
+    db: Database,
+    fetcher: Fetcher,
+
+    pub dateindex_to_price_ohlc_in_cents: RawVec<DateIndex, OHLCCents>,
+    pub height_to_price_ohlc_in_cents: RawVec<Height, OHLCCents>,
+}
+
+impl Vecs {
+    pub fn forced_import(parent: &Path, fetcher: Fetcher, version: Version) -> Result<Self> {
+        let db = Database::open(&parent.join("fetched"))?;
+
+        let this = Self {
+            fetcher,
+
+            dateindex_to_price_ohlc_in_cents: RawVec::forced_import(
+                &db,
+                "price_ohlc_in_cents",
+                version + Version::ZERO,
+            )?,
+            height_to_price_ohlc_in_cents: RawVec::forced_import(
+                &db,
+                "price_ohlc_in_cents",
+                version + Version::ZERO,
+            )?,
+
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexer, indexes, starting_indexes, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        let height_to_timestamp = &indexer.vecs.height_to_timestamp;
+        let index = starting_indexes
+            .height
+            .min(Height::from(self.height_to_price_ohlc_in_cents.len()));
+        height_to_timestamp
+            .iter_at(index)
+            .try_for_each(|(i, v)| -> Result<()> {
+                let v = v.into_owned();
+                self.height_to_price_ohlc_in_cents.forced_push_at(
+                    i,
+                    self.fetcher
+                        .get_height(
+                            i,
+                            v,
+                            i.decremented().map(|prev_i| {
+                                height_to_timestamp.into_iter().unwrap_get_inner(prev_i)
+                            }),
+                        )
+                        .unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+        self.height_to_price_ohlc_in_cents.safe_flush(exit)?;
+
+        let index = starting_indexes
+            .dateindex
+            .min(DateIndex::from(self.dateindex_to_price_ohlc_in_cents.len()));
+        let mut prev = None;
+        indexes
+            .dateindex_to_date
+            .iter_at(index)
+            .try_for_each(|(i, v)| -> Result<()> {
+                let d = v.into_owned();
+                if prev.is_none() {
+                    let i = i.unwrap_to_usize();
+                    prev.replace(if i > 0 {
+                        self.dateindex_to_price_ohlc_in_cents
+                            .into_iter()
+                            .unwrap_get_inner_(i - 1)
+                    } else {
+                        OHLCCents::default()
+                    });
+                }
+
+                let ohlc = if i.unwrap_to_usize() + 100
+                    >= self.dateindex_to_price_ohlc_in_cents.len()
+                    && let Ok(mut ohlc) = self.fetcher.get_date(d)
+                {
+                    let prev_open = *prev.as_ref().unwrap().close;
+                    *ohlc.open = prev_open;
+                    *ohlc.high = (*ohlc.high).max(prev_open);
+                    *ohlc.low = (*ohlc.low).min(prev_open);
+                    ohlc
+                } else {
+                    prev.clone().unwrap()
+                };
+
+                prev.replace(ohlc.clone());
+
+                self.dateindex_to_price_ohlc_in_cents
+                    .forced_push_at(i, ohlc, exit)?;
+
+                Ok(())
+            })?;
+        self.dateindex_to_price_ohlc_in_cents.safe_flush(exit)?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [
+            &self.dateindex_to_price_ohlc_in_cents as &dyn AnyCollectableVec,
+            &self.height_to_price_ohlc_in_cents,
+        ]
+        .into_iter()
+    }
+}

crates/brk_computer/src/grouped/builder_eager.rs

@@ -1,16 +1,17 @@
-use std::path::Path;
-
-use brk_core::{CheckedSub, Result, StoredUsize, Version};
-use brk_exit::Exit;
-use brk_vec::{AnyCollectableVec, AnyIterableVec, EagerVec, Format, StoredIndex, StoredType};
-use color_eyre::eyre::ContextCompat;
+use allocative::Allocative;
+use brk_error::{Error, Result};
+use brk_structs::{CheckedSub, StoredU64, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, EagerVec, Exit, Format,
+    GenericStoredVec, StoredIndex, StoredRaw,
+};
 
 use crate::utils::get_percentile;
 
 use super::ComputedType;
 
-#[derive(Clone, Debug)]
-pub struct ComputedVecBuilder<I, T>
+#[derive(Clone, Debug, Allocative)]
+pub struct EagerVecBuilder<I, T>
 where
     I: StoredIndex,
     T: ComputedType,
@@ -19,11 +20,11 @@ where
     pub average: Option<Box<EagerVec<I, T>>>,
     pub sum: Option<Box<EagerVec<I, T>>>,
     pub max: Option<Box<EagerVec<I, T>>>,
-    pub _90p: Option<Box<EagerVec<I, T>>>,
-    pub _75p: Option<Box<EagerVec<I, T>>>,
+    pub pct90: Option<Box<EagerVec<I, T>>>,
+    pub pct75: Option<Box<EagerVec<I, T>>>,
     pub median: Option<Box<EagerVec<I, T>>>,
-    pub _25p: Option<Box<EagerVec<I, T>>>,
-    pub _10p: Option<Box<EagerVec<I, T>>>,
+    pub pct25: Option<Box<EagerVec<I, T>>>,
+    pub pct10: Option<Box<EagerVec<I, T>>>,
     pub min: Option<Box<EagerVec<I, T>>>,
     pub last: Option<Box<EagerVec<I, T>>>,
     pub cumulative: Option<Box<EagerVec<I, T>>>,
@@ -31,30 +32,29 @@ where
 
 const VERSION: Version = Version::ZERO;
 
-impl<I, T> ComputedVecBuilder<I, T>
+impl<I, T> EagerVecBuilder<I, T>
 where
     I: StoredIndex,
     T: ComputedType,
 {
+    pub fn forced_import_compressed(
+        db: &Database,
+        name: &str,
+        version: Version,
+        options: VecBuilderOptions,
+    ) -> Result<Self> {
+        Self::forced_import(db, name, version, Format::Compressed, options)
+    }
+
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         name: &str,
         version: Version,
         format: Format,
-        options: StorableVecGeneatorOptions,
-    ) -> color_eyre::Result<Self> {
+        options: VecBuilderOptions,
+    ) -> Result<Self> {
         let only_one_active = options.is_only_one_active();
 
-        let prefix = |s: &str| format!("{s}_{name}");
-
-        let maybe_prefix = |s: &str| {
-            if only_one_active {
-                name.to_string()
-            } else {
-                prefix(s)
-            }
-        };
-
         let suffix = |s: &str| format!("{name}_{s}");
 
         let maybe_suffix = |s: &str| {
@@ -69,8 +69,8 @@ where
             first: options.first.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_prefix("first"),
+                        db,
+                        &maybe_suffix("first"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
@@ -79,13 +79,13 @@ where
             }),
             last: options.last.then(|| {
                 Box::new(
-                    EagerVec::forced_import(path, name, version + Version::ZERO, format).unwrap(),
+                    EagerVec::forced_import(db, name, version + Version::ZERO, format).unwrap(),
                 )
             }),
             min: options.min.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
+                        db,
                         &maybe_suffix("min"),
                         version + VERSION + Version::ZERO,
                         format,
@@ -96,7 +96,7 @@ where
             max: options.max.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
+                        db,
                         &maybe_suffix("max"),
                         version + VERSION + Version::ZERO,
                         format,
@@ -107,7 +107,7 @@ where
             median: options.median.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
+                        db,
                         &maybe_suffix("median"),
                         version + VERSION + Version::ZERO,
                         format,
@@ -118,8 +118,8 @@ where
             average: options.average.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_suffix("average"),
+                        db,
+                        &maybe_suffix("avg"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
@@ -129,8 +129,12 @@ where
             sum: options.sum.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_suffix("sum"),
+                        db,
+                        &(if !options.last && !options.average && !options.min && !options.max {
+                            name.to_string()
+                        } else {
+                            maybe_suffix("sum")
+                        }),
                         version + VERSION + Version::ZERO,
                         format,
                     )
@@ -140,52 +144,52 @@ where
             cumulative: options.cumulative.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &prefix("cumulative"),
+                        db,
+                        &suffix("cumulative"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _90p: options._90p.then(|| {
+            pct90: options.pct90.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_suffix("90p"),
+                        db,
+                        &maybe_suffix("pct90"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _75p: options._75p.then(|| {
+            pct75: options.pct75.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_suffix("75p"),
+                        db,
+                        &maybe_suffix("pct75"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _25p: options._25p.then(|| {
+            pct25: options.pct25.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_suffix("25p"),
+                        db,
+                        &maybe_suffix("pct25"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
                     .unwrap(),
                 )
             }),
-            _10p: options._10p.then(|| {
+            pct10: options.pct10.then(|| {
                 Box::new(
                     EagerVec::forced_import(
-                        path,
-                        &maybe_suffix("10p"),
+                        db,
+                        &maybe_suffix("pct10"),
                         version + VERSION + Version::ZERO,
                         format,
                     )
@@ -207,7 +211,7 @@ where
             return Ok(());
         };
 
-        self.validate_computed_version_or_reset_file(source.version())?;
+        self.validate_computed_version_or_reset(source.version())?;
 
         let index = self.starting_index(max_from);
 
@@ -217,8 +221,9 @@ where
             cumulative_vec.iter().unwrap_get_inner(index)
         });
         source.iter_at(index).try_for_each(|(i, v)| -> Result<()> {
-            cumulative = cumulative.clone() + v.into_inner();
-            cumulative_vec.forced_push_at(i, cumulative.clone(), exit)
+            cumulative += v.into_owned();
+            cumulative_vec.forced_push_at(i, cumulative, exit)?;
+            Ok(())
         })?;
 
         self.safe_flush(exit)?;
@@ -231,13 +236,13 @@ where
         max_from: I,
         source: &impl AnyIterableVec<I2, T>,
         first_indexes: &impl AnyIterableVec<I, I2>,
-        count_indexes: &impl AnyIterableVec<I, StoredUsize>,
+        count_indexes: &impl AnyIterableVec<I, StoredU64>,
         exit: &Exit,
     ) -> Result<()>
     where
-        I2: StoredIndex + StoredType + CheckedSub<I2>,
+        I2: StoredIndex + StoredRaw + CheckedSub<I2>,
     {
-        self.validate_computed_version_or_reset_file(
+        self.validate_computed_version_or_reset(
             source.version() + first_indexes.version() + count_indexes.version(),
         )?;
 
@@ -256,10 +261,10 @@ where
 
         first_indexes
             .iter_at(index)
-            .try_for_each(|(i, first_index)| -> Result<()> {
-                let first_index = first_index.into_inner();
+            .try_for_each(|(index, first_index)| -> Result<()> {
+                let first_index = first_index.into_owned();
 
-                let count_index = count_indexes_iter.unwrap_get_inner(i);
+                let count_index = count_indexes_iter.unwrap_get_inner(index);
 
                 if let Some(first) = self.first.as_mut() {
                     let f = source_iter
@@ -269,7 +274,7 @@ where
                 }
 
                 if let Some(last) = self.last.as_mut() {
-                    let count_index = *count_index;
+                    let count_index = *count_index as usize;
                     if count_index == 0 {
                         panic!("should compute last if count can be 0")
                     }
@@ -280,7 +285,7 @@ where
                     //     dbg!(first_index, count_index, last_index);
                     // })
                     // .unwrap()
-                    // .into_inner();
+                    // .into_owned();
                     last.forced_push_at(index, v, exit)?;
                 }
 
@@ -288,19 +293,19 @@ where
                 let needs_average_sum_or_cumulative =
                     needs_sum_or_cumulative || self.average.is_some();
                 let needs_sorted = self.max.is_some()
-                    || self._90p.is_some()
-                    || self._75p.is_some()
+                    || self.pct90.is_some()
+                    || self.pct75.is_some()
                     || self.median.is_some()
-                    || self._25p.is_some()
-                    || self._10p.is_some()
+                    || self.pct25.is_some()
+                    || self.pct10.is_some()
                     || self.min.is_some();
                 let needs_values = needs_sorted || needs_average_sum_or_cumulative;
 
                 if needs_values {
                     source_iter.set(first_index);
                     let mut values = (&mut source_iter)
-                        .take(*count_index)
-                        .map(|(_, v)| v.into_inner())
+                        .take(*count_index as usize)
+                        .map(|(_, v)| v.into_owned())
                         .collect::<Vec<_>>();
 
                     if needs_sorted {
@@ -308,14 +313,14 @@ where
 
                         if let Some(max) = self.max.as_mut() {
                             max.forced_push_at(
-                                i,
-                                values
+                                index,
+                                *values
                                     .last()
-                                    .context("expect some")
+                                    .ok_or(Error::Str("expect some"))
                                     .inspect_err(|_| {
                                         dbg!(
                                             &values,
-                                            max.path(),
+                                            max.name(),
                                             first_indexes.name(),
                                             first_index,
                                             count_indexes.name(),
@@ -324,34 +329,33 @@ where
                                             source.name()
                                         );
                                     })
-                                    .unwrap()
-                                    .clone(),
+                                    .unwrap(),
                                 exit,
                             )?;
                         }
 
-                        if let Some(_90p) = self._90p.as_mut() {
-                            _90p.forced_push_at(i, get_percentile(&values, 0.90), exit)?;
+                        if let Some(pct90) = self.pct90.as_mut() {
+                            pct90.forced_push_at(index, get_percentile(&values, 0.90), exit)?;
                         }
 
-                        if let Some(_75p) = self._75p.as_mut() {
-                            _75p.forced_push_at(i, get_percentile(&values, 0.75), exit)?;
+                        if let Some(pct75) = self.pct75.as_mut() {
+                            pct75.forced_push_at(index, get_percentile(&values, 0.75), exit)?;
                         }
 
                         if let Some(median) = self.median.as_mut() {
-                            median.forced_push_at(i, get_percentile(&values, 0.50), exit)?;
+                            median.forced_push_at(index, get_percentile(&values, 0.50), exit)?;
                         }
 
-                        if let Some(_25p) = self._25p.as_mut() {
-                            _25p.forced_push_at(i, get_percentile(&values, 0.25), exit)?;
+                        if let Some(pct25) = self.pct25.as_mut() {
+                            pct25.forced_push_at(index, get_percentile(&values, 0.25), exit)?;
                         }
 
-                        if let Some(_10p) = self._10p.as_mut() {
-                            _10p.forced_push_at(i, get_percentile(&values, 0.10), exit)?;
+                        if let Some(pct10) = self.pct10.as_mut() {
+                            pct10.forced_push_at(index, get_percentile(&values, 0.10), exit)?;
                         }
 
                         if let Some(min) = self.min.as_mut() {
-                            min.forced_push_at(i, values.first().unwrap().clone(), exit)?;
+                            min.forced_push_at(index, *values.first().unwrap(), exit)?;
                         }
                     }
 
@@ -360,19 +364,19 @@ where
                         let sum = values.into_iter().fold(T::from(0), |a, b| a + b);
 
                         if let Some(average) = self.average.as_mut() {
-                            let avg = sum.clone() / len;
-                            average.forced_push_at(i, avg, exit)?;
+                            let avg = sum / len;
+                            average.forced_push_at(index, avg, exit)?;
                         }
 
                         if needs_sum_or_cumulative {
                             if let Some(sum_vec) = self.sum.as_mut() {
-                                sum_vec.forced_push_at(i, sum.clone(), exit)?;
+                                sum_vec.forced_push_at(index, sum, exit)?;
                             }
 
                             if let Some(cumulative_vec) = self.cumulative.as_mut() {
-                                let t = cumulative.as_ref().unwrap().clone() + sum;
-                                cumulative.replace(t.clone());
-                                cumulative_vec.forced_push_at(i, t, exit)?;
+                                let t = cumulative.unwrap() + sum;
+                                cumulative.replace(t);
+                                cumulative_vec.forced_push_at(index, t, exit)?;
                             }
                         }
                     }
@@ -390,24 +394,24 @@ where
     pub fn from_aligned<I2>(
         &mut self,
         max_from: I,
-        source: &ComputedVecBuilder<I2, T>,
+        source: &EagerVecBuilder<I2, T>,
         first_indexes: &impl AnyIterableVec<I, I2>,
-        count_indexes: &impl AnyIterableVec<I, StoredUsize>,
+        count_indexes: &impl AnyIterableVec<I, StoredU64>,
         exit: &Exit,
     ) -> Result<()>
     where
-        I2: StoredIndex + StoredType + CheckedSub<I2>,
+        I2: StoredIndex + StoredRaw + CheckedSub<I2>,
     {
-        if self._90p.is_some()
-            || self._75p.is_some()
+        if self.pct90.is_some()
+            || self.pct75.is_some()
             || self.median.is_some()
-            || self._25p.is_some()
-            || self._10p.is_some()
+            || self.pct25.is_some()
+            || self.pct10.is_some()
         {
             panic!("unsupported");
         }
 
-        self.validate_computed_version_or_reset_file(
+        self.validate_computed_version_or_reset(
             VERSION + first_indexes.version() + count_indexes.version(),
         )?;
 
@@ -430,10 +434,10 @@ where
 
         first_indexes
             .iter_at(index)
-            .try_for_each(|(i, first_index, ..)| -> Result<()> {
-                let first_index = first_index.into_inner();
+            .try_for_each(|(index, first_index, ..)| -> Result<()> {
+                let first_index = first_index.into_owned();
 
-                let count_index = count_indexes_iter.unwrap_get_inner(i);
+                let count_index = count_indexes_iter.unwrap_get_inner(index);
 
                 if let Some(first) = self.first.as_mut() {
                     let v = source_first_iter
@@ -444,7 +448,7 @@ where
                 }
 
                 if let Some(last) = self.last.as_mut() {
-                    let count_index = *count_index;
+                    let count_index = *count_index as usize;
                     if count_index == 0 {
                         panic!("should compute last if count can be 0")
                     }
@@ -468,22 +472,22 @@ where
                             let source_max_iter = source_max_iter.as_mut().unwrap();
                             source_max_iter.set(first_index);
                             let mut values = source_max_iter
-                                .take(*count_index)
-                                .map(|(_, v)| v.into_inner())
+                                .take(*count_index as usize)
+                                .map(|(_, v)| v.into_owned())
                                 .collect::<Vec<_>>();
                             values.sort_unstable();
-                            max.forced_push_at(i, values.last().unwrap().clone(), exit)?;
+                            max.forced_push_at(index, *values.last().unwrap(), exit)?;
                         }
 
                         if let Some(min) = self.min.as_mut() {
                             let source_min_iter = source_min_iter.as_mut().unwrap();
                             source_min_iter.set(first_index);
                             let mut values = source_min_iter
-                                .take(*count_index)
-                                .map(|(_, v)| v.into_inner())
+                                .take(*count_index as usize)
+                                .map(|(_, v)| v.into_owned())
                                 .collect::<Vec<_>>();
                             values.sort_unstable();
-                            min.forced_push_at(i, values.first().unwrap().clone(), exit)?;
+                            min.forced_push_at(index, *values.first().unwrap(), exit)?;
                         }
                     }
 
@@ -492,8 +496,8 @@ where
                             let source_average_iter = source_average_iter.as_mut().unwrap();
                             source_average_iter.set(first_index);
                             let values = source_average_iter
-                                .take(*count_index)
-                                .map(|(_, v)| v.into_inner())
+                                .take(*count_index as usize)
+                                .map(|(_, v)| v.into_owned())
                                 .collect::<Vec<_>>();
 
                             let len = values.len();
@@ -501,27 +505,27 @@ where
                             // TODO: Multiply by count then divide by cumulative
                             // Right now it's not 100% accurate as there could be more or less elements in the lower timeframe (28 days vs 31 days in a month for example)
                             let avg = cumulative / len;
-                            average.forced_push_at(i, avg, exit)?;
+                            average.forced_push_at(index, avg, exit)?;
                         }
 
                         if needs_sum_or_cumulative {
                             let source_sum_iter = source_sum_iter.as_mut().unwrap();
                             source_sum_iter.set(first_index);
                             let values = source_sum_iter
-                                .take(*count_index)
-                                .map(|(_, v)| v.into_inner())
+                                .take(*count_index as usize)
+                                .map(|(_, v)| v.into_owned())
                                 .collect::<Vec<_>>();
 
                             let sum = values.into_iter().fold(T::from(0), |a, b| a + b);
 
                             if let Some(sum_vec) = self.sum.as_mut() {
-                                sum_vec.forced_push_at(i, sum.clone(), exit)?;
+                                sum_vec.forced_push_at(index, sum, exit)?;
                             }
 
                             if let Some(cumulative_vec) = self.cumulative.as_mut() {
-                                let t = cumulative.as_ref().unwrap().clone() + sum;
-                                cumulative.replace(t.clone());
-                                cumulative_vec.forced_push_at(i, t, exit)?;
+                                let t = cumulative.unwrap() + sum;
+                                cumulative.replace(t);
+                                cumulative_vec.forced_push_at(index, t, exit)?;
                             }
                         }
                     }
@@ -537,7 +541,7 @@ where
 
     pub fn starting_index(&self, max_from: I) -> I {
         max_from.min(I::from(
-            self.vecs().into_iter().map(|v| v.len()).min().unwrap(),
+            self.iter_any_collectable().map(|v| v.len()).min().unwrap(),
         ))
     }
 
@@ -555,24 +559,24 @@ where
         self.max.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_90p(&self) -> &EagerVec<I, T> {
-        self._90p.as_ref().unwrap()
+    pub fn unwrap_pct90(&self) -> &EagerVec<I, T> {
+        self.pct90.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_75p(&self) -> &EagerVec<I, T> {
-        self._75p.as_ref().unwrap()
+    pub fn unwrap_pct75(&self) -> &EagerVec<I, T> {
+        self.pct75.as_ref().unwrap()
     }
     #[allow(unused)]
     pub fn unwrap_median(&self) -> &EagerVec<I, T> {
         self.median.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_25p(&self) -> &EagerVec<I, T> {
-        self._25p.as_ref().unwrap()
+    pub fn unwrap_pct25(&self) -> &EagerVec<I, T> {
+        self.pct25.as_ref().unwrap()
     }
     #[allow(unused)]
-    pub fn unwrap_10p(&self) -> &EagerVec<I, T> {
-        self._10p.as_ref().unwrap()
+    pub fn unwrap_pct10(&self) -> &EagerVec<I, T> {
+        self.pct10.as_ref().unwrap()
     }
     pub fn unwrap_min(&self) -> &EagerVec<I, T> {
         self.min.as_ref().unwrap()
@@ -585,47 +589,96 @@ where
         self.cumulative.as_ref().unwrap()
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        let mut v: Vec<&dyn AnyCollectableVec> = vec![];
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
 
-        if let Some(first) = self.first.as_ref() {
-            v.push(first.as_ref());
-        }
-        if let Some(last) = self.last.as_ref() {
-            v.push(last.as_ref());
-        }
-        if let Some(min) = self.min.as_ref() {
-            v.push(min.as_ref());
-        }
-        if let Some(max) = self.max.as_ref() {
-            v.push(max.as_ref());
-        }
-        if let Some(median) = self.median.as_ref() {
-            v.push(median.as_ref());
-        }
-        if let Some(average) = self.average.as_ref() {
-            v.push(average.as_ref());
-        }
-        if let Some(sum) = self.sum.as_ref() {
-            v.push(sum.as_ref());
-        }
-        if let Some(cumulative) = self.cumulative.as_ref() {
-            v.push(cumulative.as_ref());
-        }
-        if let Some(_90p) = self._90p.as_ref() {
-            v.push(_90p.as_ref());
-        }
-        if let Some(_75p) = self._75p.as_ref() {
-            v.push(_75p.as_ref());
-        }
-        if let Some(_25p) = self._25p.as_ref() {
-            v.push(_25p.as_ref());
-        }
-        if let Some(_10p) = self._10p.as_ref() {
-            v.push(_10p.as_ref());
-        }
+        iter = Box::new(
+            iter.chain(
+                self.first
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.last
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.min
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.max
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.median
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.average
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.sum
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.cumulative
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct90
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct75
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct25
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.pct10
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
 
-        v
+        iter
     }
 
     pub fn safe_flush(&mut self, exit: &Exit) -> Result<()> {
@@ -653,58 +706,58 @@ where
         if let Some(cumulative) = self.cumulative.as_mut() {
             cumulative.safe_flush(exit)?;
         }
-        if let Some(_90p) = self._90p.as_mut() {
-            _90p.safe_flush(exit)?;
+        if let Some(pct90) = self.pct90.as_mut() {
+            pct90.safe_flush(exit)?;
         }
-        if let Some(_75p) = self._75p.as_mut() {
-            _75p.safe_flush(exit)?;
+        if let Some(pct75) = self.pct75.as_mut() {
+            pct75.safe_flush(exit)?;
         }
-        if let Some(_25p) = self._25p.as_mut() {
-            _25p.safe_flush(exit)?;
+        if let Some(pct25) = self.pct25.as_mut() {
+            pct25.safe_flush(exit)?;
         }
-        if let Some(_10p) = self._10p.as_mut() {
-            _10p.safe_flush(exit)?;
+        if let Some(pct10) = self.pct10.as_mut() {
+            pct10.safe_flush(exit)?;
         }
 
         Ok(())
     }
 
-    pub fn validate_computed_version_or_reset_file(&mut self, version: Version) -> Result<()> {
+    pub fn validate_computed_version_or_reset(&mut self, version: Version) -> Result<()> {
         if let Some(first) = self.first.as_mut() {
-            first.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            first.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(last) = self.last.as_mut() {
-            last.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            last.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(min) = self.min.as_mut() {
-            min.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            min.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(max) = self.max.as_mut() {
-            max.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            max.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(median) = self.median.as_mut() {
-            median.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            median.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(average) = self.average.as_mut() {
-            average.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            average.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(sum) = self.sum.as_mut() {
-            sum.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            sum.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
         if let Some(cumulative) = self.cumulative.as_mut() {
-            cumulative.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+            cumulative.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_90p) = self._90p.as_mut() {
-            _90p.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+        if let Some(pct90) = self.pct90.as_mut() {
+            pct90.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_75p) = self._75p.as_mut() {
-            _75p.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+        if let Some(pct75) = self.pct75.as_mut() {
+            pct75.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_25p) = self._25p.as_mut() {
-            _25p.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+        if let Some(pct25) = self.pct25.as_mut() {
+            pct25.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
-        if let Some(_10p) = self._10p.as_mut() {
-            _10p.validate_computed_version_or_reset_file(Version::ZERO + version)?;
+        if let Some(pct10) = self.pct10.as_mut() {
+            pct10.validate_computed_version_or_reset(Version::ZERO + version)?;
         }
 
         Ok(())
@@ -712,22 +765,70 @@ where
 }
 
 #[derive(Default, Clone, Copy)]
-pub struct StorableVecGeneatorOptions {
+pub struct VecBuilderOptions {
     average: bool,
     sum: bool,
     max: bool,
-    _90p: bool,
-    _75p: bool,
+    pct90: bool,
+    pct75: bool,
     median: bool,
-    _25p: bool,
-    _10p: bool,
+    pct25: bool,
+    pct10: bool,
     min: bool,
     first: bool,
     last: bool,
     cumulative: bool,
 }
 
-impl StorableVecGeneatorOptions {
+impl VecBuilderOptions {
+    pub fn average(&self) -> bool {
+        self.average
+    }
+
+    pub fn sum(&self) -> bool {
+        self.sum
+    }
+
+    pub fn max(&self) -> bool {
+        self.max
+    }
+
+    pub fn pct90(&self) -> bool {
+        self.pct90
+    }
+
+    pub fn pct75(&self) -> bool {
+        self.pct75
+    }
+
+    pub fn median(&self) -> bool {
+        self.median
+    }
+
+    pub fn pct25(&self) -> bool {
+        self.pct25
+    }
+
+    pub fn pct10(&self) -> bool {
+        self.pct10
+    }
+
+    pub fn min(&self) -> bool {
+        self.min
+    }
+
+    pub fn first(&self) -> bool {
+        self.first
+    }
+
+    pub fn last(&self) -> bool {
+        self.last
+    }
+
+    pub fn cumulative(&self) -> bool {
+        self.cumulative
+    }
+
     pub fn add_first(mut self) -> Self {
         self.first = true;
         self
@@ -765,26 +866,26 @@ impl StorableVecGeneatorOptions {
     }
 
     #[allow(unused)]
-    pub fn add_90p(mut self) -> Self {
-        self._90p = true;
+    pub fn add_pct90(mut self) -> Self {
+        self.pct90 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_75p(mut self) -> Self {
-        self._75p = true;
+    pub fn add_pct75(mut self) -> Self {
+        self.pct75 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_25p(mut self) -> Self {
-        self._25p = true;
+    pub fn add_pct25(mut self) -> Self {
+        self.pct25 = true;
         self
     }
 
     #[allow(unused)]
-    pub fn add_10p(mut self) -> Self {
-        self._10p = true;
+    pub fn add_pct10(mut self) -> Self {
+        self.pct10 = true;
         self
     }
 
@@ -824,26 +925,26 @@ impl StorableVecGeneatorOptions {
     }
 
     #[allow(unused)]
-    pub fn rm_90p(mut self) -> Self {
-        self._90p = false;
+    pub fn rm_pct90(mut self) -> Self {
+        self.pct90 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_75p(mut self) -> Self {
-        self._75p = false;
+    pub fn rm_pct75(mut self) -> Self {
+        self.pct75 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_25p(mut self) -> Self {
-        self._25p = false;
+    pub fn rm_pct25(mut self) -> Self {
+        self.pct25 = false;
         self
     }
 
     #[allow(unused)]
-    pub fn rm_10p(mut self) -> Self {
-        self._10p = false;
+    pub fn rm_pct10(mut self) -> Self {
+        self.pct10 = false;
         self
     }
 
@@ -860,20 +961,20 @@ impl StorableVecGeneatorOptions {
     }
 
     pub fn add_percentiles(mut self) -> Self {
-        self._90p = true;
-        self._75p = true;
+        self.pct90 = true;
+        self.pct75 = true;
         self.median = true;
-        self._25p = true;
-        self._10p = true;
+        self.pct25 = true;
+        self.pct10 = true;
         self
     }
 
     pub fn remove_percentiles(mut self) -> Self {
-        self._90p = false;
-        self._75p = false;
+        self.pct90 = false;
+        self.pct75 = false;
         self.median = false;
-        self._25p = false;
-        self._10p = false;
+        self.pct25 = false;
+        self.pct10 = false;
         self
     }
 
@@ -882,11 +983,11 @@ impl StorableVecGeneatorOptions {
             self.average,
             self.sum,
             self.max,
-            self._90p,
-            self._75p,
+            self.pct90,
+            self.pct75,
             self.median,
-            self._25p,
-            self._10p,
+            self.pct25,
+            self.pct10,
             self.min,
             self.first,
             self.last,

crates/brk_computer/src/grouped/builder_lazy.rs

@@ -0,0 +1,425 @@
+use allocative::Allocative;
+use brk_structs::Version;
+use vecdb::{
+    AnyBoxedIterableVec, AnyCloneableIterableVec, AnyCollectableVec, FromCoarserIndex,
+    LazyVecFrom2, StoredIndex,
+};
+
+use crate::grouped::{EagerVecBuilder, VecBuilderOptions};
+
+use super::ComputedType;
+
+#[allow(clippy::type_complexity)]
+#[derive(Clone, Allocative)]
+pub struct LazyVecBuilder<I, T, S1I, S2T>
+where
+    I: StoredIndex,
+    T: ComputedType,
+    S2T: ComputedType,
+{
+    pub first: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub average: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub sum: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub max: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub min: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub last: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+    pub cumulative: Option<Box<LazyVecFrom2<I, T, S1I, T, I, S2T>>>,
+}
+
+const VERSION: Version = Version::ZERO;
+
+impl<I, T, S1I, S2T> LazyVecBuilder<I, T, S1I, S2T>
+where
+    I: StoredIndex,
+    T: ComputedType + 'static,
+    S1I: StoredIndex + 'static + FromCoarserIndex<I>,
+    S2T: ComputedType,
+{
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        name: &str,
+        version: Version,
+        source: Option<AnyBoxedIterableVec<S1I, T>>,
+        source_extra: &EagerVecBuilder<S1I, T>,
+        len_source: AnyBoxedIterableVec<I, S2T>,
+        options: LazyVecBuilderOptions,
+    ) -> Self {
+        let only_one_active = options.is_only_one_active();
+
+        let suffix = |s: &str| format!("{name}_{s}");
+
+        let maybe_suffix = |s: &str| {
+            if only_one_active {
+                name.to_string()
+            } else {
+                suffix(s)
+            }
+        };
+
+        Self {
+            first: options.first.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &maybe_suffix("first"),
+                    version + VERSION + Version::ZERO,
+                    source_extra
+                        .first
+                        .as_ref()
+                        .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        source
+                            .next_at(S1I::min_from(i))
+                            .map(|(_, cow)| cow.into_owned())
+                    },
+                ))
+            }),
+            last: options.last.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    name,
+                    version + VERSION + Version::ZERO,
+                    source_extra.last.as_ref().map_or_else(
+                        || {
+                            source
+                                .as_ref()
+                                .unwrap_or_else(|| {
+                                    dbg!(name, I::to_string());
+                                    panic!()
+                                })
+                                .clone()
+                        },
+                        |v| v.clone(),
+                    ),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        source
+                            .next_at(S1I::max_from(i, source.len()))
+                            .map(|(_, cow)| cow.into_owned())
+                    },
+                ))
+            }),
+            min: options.min.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &maybe_suffix("min"),
+                    version + VERSION + Version::ZERO,
+                    source_extra
+                        .min
+                        .as_ref()
+                        .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        S1I::inclusive_range_from(i, source.len())
+                            .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
+                            .min()
+                    },
+                ))
+            }),
+            max: options.max.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &maybe_suffix("max"),
+                    version + VERSION + Version::ZERO,
+                    source_extra
+                        .max
+                        .as_ref()
+                        .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        S1I::inclusive_range_from(i, source.len())
+                            .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
+                            .max()
+                    },
+                ))
+            }),
+            average: options.average.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &maybe_suffix("avg"),
+                    version + VERSION + Version::ZERO,
+                    source_extra
+                        .average
+                        .as_ref()
+                        .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        let vec = S1I::inclusive_range_from(i, source.len())
+                            .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
+                            .collect::<Vec<_>>();
+                        if vec.is_empty() {
+                            return None;
+                        }
+                        let mut sum = T::from(0);
+                        let len = vec.len();
+                        vec.into_iter().for_each(|v| sum += v);
+                        Some(sum / len)
+                    },
+                ))
+            }),
+            sum: options.sum.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &(if !options.last && !options.average && !options.min && !options.max {
+                        name.to_string()
+                    } else {
+                        maybe_suffix("sum")
+                    }),
+                    version + VERSION + Version::ZERO,
+                    source_extra
+                        .sum
+                        .as_ref()
+                        .map_or_else(|| source.as_ref().unwrap().clone(), |v| v.clone()),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        let vec = S1I::inclusive_range_from(i, source.len())
+                            .flat_map(|i| source.next_at(i).map(|(_, cow)| cow.into_owned()))
+                            .collect::<Vec<_>>();
+                        if vec.is_empty() {
+                            return None;
+                        }
+                        let mut sum = T::from(0);
+                        vec.into_iter().for_each(|v| sum += v);
+                        Some(sum)
+                    },
+                ))
+            }),
+            cumulative: options.cumulative.then(|| {
+                Box::new(LazyVecFrom2::init(
+                    &suffix("cumulative"),
+                    version + VERSION + Version::ZERO,
+                    source_extra.cumulative.as_ref().unwrap().boxed_clone(),
+                    len_source.clone(),
+                    |i: I, source, len_source| {
+                        if i.unwrap_to_usize() >= len_source.len() {
+                            return None;
+                        }
+                        source
+                            .next_at(S1I::max_from(i, source.len()))
+                            .map(|(_, cow)| cow.into_owned())
+                    },
+                ))
+            }),
+        }
+    }
+
+    pub fn starting_index(&self, max_from: I) -> I {
+        max_from.min(I::from(
+            self.iter_any_collectable().map(|v| v.len()).min().unwrap(),
+        ))
+    }
+
+    pub fn unwrap_first(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.first.as_ref().unwrap()
+    }
+    #[allow(unused)]
+    pub fn unwrap_average(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.average.as_ref().unwrap()
+    }
+    pub fn unwrap_sum(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.sum.as_ref().unwrap()
+    }
+    pub fn unwrap_max(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.max.as_ref().unwrap()
+    }
+    pub fn unwrap_min(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.min.as_ref().unwrap()
+    }
+    pub fn unwrap_last(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.last.as_ref().unwrap()
+    }
+    #[allow(unused)]
+    pub fn unwrap_cumulative(&self) -> &LazyVecFrom2<I, T, S1I, T, I, S2T> {
+        self.cumulative.as_ref().unwrap()
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
+
+        iter = Box::new(
+            iter.chain(
+                self.first
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.last
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.min
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.max
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.average
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.sum
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.cumulative
+                    .as_ref()
+                    .map(|x| x.as_ref() as &dyn AnyCollectableVec),
+            ),
+        );
+
+        iter
+    }
+}
+
+#[derive(Default, Clone, Copy)]
+pub struct LazyVecBuilderOptions {
+    average: bool,
+    sum: bool,
+    max: bool,
+    min: bool,
+    first: bool,
+    last: bool,
+    cumulative: bool,
+}
+
+impl From<VecBuilderOptions> for LazyVecBuilderOptions {
+    fn from(value: VecBuilderOptions) -> Self {
+        Self {
+            average: value.average(),
+            sum: value.sum(),
+            max: value.max(),
+            min: value.min(),
+            first: value.first(),
+            last: value.last(),
+            cumulative: value.cumulative(),
+        }
+    }
+}
+
+impl LazyVecBuilderOptions {
+    pub fn add_first(mut self) -> Self {
+        self.first = true;
+        self
+    }
+
+    pub fn add_last(mut self) -> Self {
+        self.last = true;
+        self
+    }
+
+    pub fn add_min(mut self) -> Self {
+        self.min = true;
+        self
+    }
+
+    pub fn add_max(mut self) -> Self {
+        self.max = true;
+        self
+    }
+
+    pub fn add_average(mut self) -> Self {
+        self.average = true;
+        self
+    }
+
+    pub fn add_sum(mut self) -> Self {
+        self.sum = true;
+        self
+    }
+
+    pub fn add_cumulative(mut self) -> Self {
+        self.cumulative = true;
+        self
+    }
+
+    #[allow(unused)]
+    pub fn rm_min(mut self) -> Self {
+        self.min = false;
+        self
+    }
+
+    #[allow(unused)]
+    pub fn rm_max(mut self) -> Self {
+        self.max = false;
+        self
+    }
+
+    #[allow(unused)]
+    pub fn rm_average(mut self) -> Self {
+        self.average = false;
+        self
+    }
+
+    #[allow(unused)]
+    pub fn rm_sum(mut self) -> Self {
+        self.sum = false;
+        self
+    }
+
+    #[allow(unused)]
+    pub fn rm_cumulative(mut self) -> Self {
+        self.cumulative = false;
+        self
+    }
+
+    pub fn add_minmax(mut self) -> Self {
+        self.min = true;
+        self.max = true;
+        self
+    }
+
+    pub fn is_only_one_active(&self) -> bool {
+        [
+            self.average,
+            self.sum,
+            self.max,
+            self.min,
+            self.first,
+            self.last,
+            self.cumulative,
+        ]
+        .iter()
+        .filter(|b| **b)
+        .count()
+            == 1
+    }
+
+    pub fn copy_self_extra(&self) -> Self {
+        Self {
+            cumulative: self.cumulative,
+            ..Self::default()
+        }
+    }
+}

crates/brk_computer/src/grouped/computed.rs

@@ -0,0 +1,23 @@
+use std::ops::{Add, AddAssign, Div};
+
+use vecdb::StoredCompressed;
+
+pub trait ComputedType
+where
+    Self: StoredCompressed
+        + From<usize>
+        + Div<usize, Output = Self>
+        + Add<Output = Self>
+        + AddAssign
+        + Ord,
+{
+}
+impl<T> ComputedType for T where
+    T: StoredCompressed
+        + From<usize>
+        + Div<usize, Output = Self>
+        + Add<Output = Self>
+        + AddAssign
+        + Ord
+{
+}

crates/brk_computer/src/grouped/from_dateindex.rs

@@ -0,0 +1,164 @@
+use allocative::Allocative;
+use brk_error::Result;
+
+use brk_structs::{
+    DateIndex, DecadeIndex, MonthIndex, QuarterIndex, SemesterIndex, Version, WeekIndex, YearIndex,
+};
+use vecdb::{AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Database, EagerVec, Exit};
+
+use crate::{Indexes, grouped::LazyVecBuilder, indexes};
+
+use super::{ComputedType, EagerVecBuilder, Source, VecBuilderOptions};
+
+#[derive(Clone, Allocative)]
+pub struct ComputedVecsFromDateIndex<T>
+where
+    T: ComputedType + PartialOrd,
+{
+    pub dateindex: Option<EagerVec<DateIndex, T>>,
+    pub dateindex_extra: EagerVecBuilder<DateIndex, T>,
+    pub weekindex: LazyVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
+    pub monthindex: LazyVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
+    pub quarterindex: LazyVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
+    pub semesterindex: LazyVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
+    pub yearindex: LazyVecBuilder<YearIndex, T, DateIndex, YearIndex>,
+    pub decadeindex: LazyVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
+}
+
+const VERSION: Version = Version::ZERO;
+
+impl<T> ComputedVecsFromDateIndex<T>
+where
+    T: ComputedType + 'static,
+{
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        source: Source<DateIndex, T>,
+        version: Version,
+        indexes: &indexes::Vecs,
+        options: VecBuilderOptions,
+    ) -> Result<Self> {
+        let dateindex = source.is_compute().then(|| {
+            EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO).unwrap()
+        });
+
+        let dateindex_extra = EagerVecBuilder::forced_import_compressed(
+            db,
+            name,
+            version + VERSION + Version::ZERO,
+            options.copy_self_extra(),
+        )?;
+
+        let options = options.remove_percentiles();
+
+        let dateindex_source = source.vec().or(dateindex.as_ref().map(|v| v.boxed_clone()));
+
+        Ok(Self {
+            weekindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.weekindex_to_weekindex.boxed_clone(),
+                options.into(),
+            ),
+            monthindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.monthindex_to_monthindex.boxed_clone(),
+                options.into(),
+            ),
+            quarterindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.quarterindex_to_quarterindex.boxed_clone(),
+                options.into(),
+            ),
+            semesterindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.semesterindex_to_semesterindex.boxed_clone(),
+                options.into(),
+            ),
+            yearindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.yearindex_to_yearindex.boxed_clone(),
+                options.into(),
+            ),
+            decadeindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                dateindex_source.clone(),
+                &dateindex_extra,
+                indexes.decadeindex_to_decadeindex.boxed_clone(),
+                options.into(),
+            ),
+            dateindex,
+            dateindex_extra,
+        })
+    }
+
+    pub fn compute_all<F>(
+        &mut self,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        mut compute: F,
+    ) -> Result<()>
+    where
+        F: FnMut(&mut EagerVec<DateIndex, T>) -> Result<()>,
+    {
+        compute(self.dateindex.as_mut().unwrap())?;
+
+        let dateindex: Option<&EagerVec<DateIndex, T>> = None;
+        self.compute_rest(starting_indexes, exit, dateindex)
+    }
+
+    pub fn compute_rest(
+        &mut self,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        dateindex: Option<&impl AnyIterableVec<DateIndex, T>>,
+    ) -> Result<()> {
+        if let Some(dateindex) = dateindex {
+            self.dateindex_extra
+                .extend(starting_indexes.dateindex, dateindex, exit)?;
+        } else {
+            let dateindex = self.dateindex.as_ref().unwrap();
+
+            self.dateindex_extra
+                .extend(starting_indexes.dateindex, dateindex, exit)?;
+        }
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
+            self.dateindex
+                .as_ref()
+                .map(|x| x as &dyn AnyCollectableVec)
+                .into_iter(),
+        );
+
+        iter = Box::new(iter.chain(self.dateindex_extra.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
+
+        iter
+    }
+}

crates/brk_computer/src/grouped/from_height.rs

@@ -0,0 +1,222 @@
+use allocative::Allocative;
+use brk_error::Result;
+
+use brk_structs::{
+    DateIndex, DecadeIndex, DifficultyEpoch, Height, MonthIndex, QuarterIndex, SemesterIndex,
+    Version, WeekIndex, YearIndex,
+};
+use vecdb::{AnyCloneableIterableVec, AnyCollectableVec, AnyIterableVec, Database, EagerVec, Exit};
+
+use crate::{
+    Indexes,
+    grouped::{LazyVecBuilder, Source},
+    indexes,
+};
+
+use super::{ComputedType, EagerVecBuilder, VecBuilderOptions};
+
+#[derive(Clone, Allocative)]
+pub struct ComputedVecsFromHeight<T>
+where
+    T: ComputedType + PartialOrd,
+{
+    pub height: Option<EagerVec<Height, T>>,
+    pub height_extra: EagerVecBuilder<Height, T>,
+    pub dateindex: EagerVecBuilder<DateIndex, T>,
+    pub weekindex: LazyVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
+    pub difficultyepoch: EagerVecBuilder<DifficultyEpoch, T>,
+    pub monthindex: LazyVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
+    pub quarterindex: LazyVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
+    pub semesterindex: LazyVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
+    pub yearindex: LazyVecBuilder<YearIndex, T, DateIndex, YearIndex>,
+    // TODO: pub halvingepoch: StorableVecGeneator<Halvingepoch, T>,
+    pub decadeindex: LazyVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
+}
+
+const VERSION: Version = Version::ZERO;
+
+impl<T> ComputedVecsFromHeight<T>
+where
+    T: ComputedType + Ord + From<f64> + 'static,
+    f64: From<T>,
+{
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        source: Source<Height, T>,
+        version: Version,
+        indexes: &indexes::Vecs,
+        options: VecBuilderOptions,
+    ) -> Result<Self> {
+        let height = source.is_compute().then(|| {
+            EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO).unwrap()
+        });
+
+        let height_extra = EagerVecBuilder::forced_import_compressed(
+            db,
+            name,
+            version + VERSION + Version::ZERO,
+            options.copy_self_extra(),
+        )?;
+
+        let dateindex = EagerVecBuilder::forced_import_compressed(
+            db,
+            name,
+            version + VERSION + Version::ZERO,
+            options,
+        )?;
+
+        let options = options.remove_percentiles();
+
+        Ok(Self {
+            weekindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.weekindex_to_weekindex.boxed_clone(),
+                options.into(),
+            ),
+            monthindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.monthindex_to_monthindex.boxed_clone(),
+                options.into(),
+            ),
+            quarterindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.quarterindex_to_quarterindex.boxed_clone(),
+                options.into(),
+            ),
+            semesterindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.semesterindex_to_semesterindex.boxed_clone(),
+                options.into(),
+            ),
+            yearindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.yearindex_to_yearindex.boxed_clone(),
+                options.into(),
+            ),
+            decadeindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.decadeindex_to_decadeindex.boxed_clone(),
+                options.into(),
+            ),
+            // halvingepoch: StorableVecGeneator::forced_import(db, name, version + VERSION + Version::ZERO, format, options)?,
+            height,
+            height_extra,
+            dateindex,
+            difficultyepoch: EagerVecBuilder::forced_import_compressed(
+                db,
+                name,
+                version + VERSION + Version::ZERO,
+                options,
+            )?,
+        })
+    }
+
+    pub fn compute_all<F>(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        mut compute: F,
+    ) -> Result<()>
+    where
+        F: FnMut(&mut EagerVec<Height, T>) -> Result<()>,
+    {
+        compute(self.height.as_mut().unwrap())?;
+
+        let height: Option<&EagerVec<Height, T>> = None;
+        self.compute_rest(indexes, starting_indexes, exit, height)
+    }
+
+    pub fn compute_rest(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        height_vec: Option<&impl AnyIterableVec<Height, T>>,
+    ) -> Result<()> {
+        if let Some(height) = height_vec {
+            self.height_extra
+                .extend(starting_indexes.height, height, exit)?;
+
+            self.dateindex.compute(
+                starting_indexes.dateindex,
+                height,
+                &indexes.dateindex_to_first_height,
+                &indexes.dateindex_to_height_count,
+                exit,
+            )?;
+
+            self.difficultyepoch.compute(
+                starting_indexes.difficultyepoch,
+                height,
+                &indexes.difficultyepoch_to_first_height,
+                &indexes.difficultyepoch_to_height_count,
+                exit,
+            )?;
+        } else {
+            let height = self.height.as_ref().unwrap();
+
+            self.height_extra
+                .extend(starting_indexes.height, height, exit)?;
+
+            self.dateindex.compute(
+                starting_indexes.dateindex,
+                height,
+                &indexes.dateindex_to_first_height,
+                &indexes.dateindex_to_height_count,
+                exit,
+            )?;
+
+            self.difficultyepoch.compute(
+                starting_indexes.difficultyepoch,
+                height,
+                &indexes.difficultyepoch_to_first_height,
+                &indexes.difficultyepoch_to_height_count,
+                exit,
+            )?;
+        }
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
+            self.height
+                .as_ref()
+                .map(|x| x as &dyn AnyCollectableVec)
+                .into_iter(),
+        );
+
+        iter = Box::new(iter.chain(self.height_extra.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.dateindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.difficultyepoch.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
+
+        iter
+    }
+}

crates/brk_computer/src/grouped/from_height_strict.rs

@@ -0,0 +1,91 @@
+use brk_error::Result;
+
+use brk_structs::{DifficultyEpoch, Height, Version};
+use vecdb::{AnyCollectableVec, Database, EagerVec, Exit};
+
+use crate::{Indexes, indexes};
+
+use super::{ComputedType, EagerVecBuilder, VecBuilderOptions};
+
+#[derive(Clone)]
+pub struct ComputedVecsFromHeightStrict<T>
+where
+    T: ComputedType + PartialOrd,
+{
+    pub height: EagerVec<Height, T>,
+    pub height_extra: EagerVecBuilder<Height, T>,
+    pub difficultyepoch: EagerVecBuilder<DifficultyEpoch, T>,
+    // TODO: pub halvingepoch: StorableVecGeneator<Halvingepoch, T>,
+}
+
+const VERSION: Version = Version::ZERO;
+
+impl<T> ComputedVecsFromHeightStrict<T>
+where
+    T: ComputedType + Ord + From<f64>,
+    f64: From<T>,
+{
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        version: Version,
+        options: VecBuilderOptions,
+    ) -> Result<Self> {
+        let height =
+            EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO)?;
+
+        let height_extra = EagerVecBuilder::forced_import_compressed(
+            db,
+            name,
+            version + VERSION + Version::ZERO,
+            options.copy_self_extra(),
+        )?;
+
+        let options = options.remove_percentiles();
+
+        Ok(Self {
+            height,
+            height_extra,
+            difficultyepoch: EagerVecBuilder::forced_import_compressed(
+                db,
+                name,
+                version + VERSION + Version::ZERO,
+                options,
+            )?,
+            // halvingepoch: StorableVecGeneator::forced_import(db, name, version + VERSION + Version::ZERO, format, options)?,
+        })
+    }
+
+    pub fn compute<F>(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        mut compute: F,
+    ) -> Result<()>
+    where
+        F: FnMut(&mut EagerVec<Height, T>) -> Result<()>,
+    {
+        compute(&mut self.height)?;
+
+        self.height_extra
+            .extend(starting_indexes.height, &self.height, exit)?;
+
+        self.difficultyepoch.compute(
+            starting_indexes.difficultyepoch,
+            &self.height,
+            &indexes.difficultyepoch_to_first_height,
+            &indexes.difficultyepoch_to_height_count,
+            exit,
+        )?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.height as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(self.height_extra.iter_any_collectable())
+            .chain(self.difficultyepoch.iter_any_collectable())
+    }
+}

crates/brk_computer/src/grouped/from_txindex.rs

@@ -1,121 +1,140 @@
-use std::path::Path;
-
-use brk_core::{
-    Bitcoin, DateIndex, DecadeIndex, DifficultyEpoch, Dollars, Height, MonthIndex, QuarterIndex,
-    Result, Sats, TxIndex, Version, WeekIndex, YearIndex,
-};
-use brk_exit::Exit;
+use allocative::Allocative;
+use brk_error::Result;
 use brk_indexer::Indexer;
-use brk_vec::{
-    AnyCollectableVec, AnyVec, CollectableVec, EagerVec, Format, StoredIndex, VecIterator,
+use brk_structs::{
+    Bitcoin, DateIndex, DecadeIndex, DifficultyEpoch, Dollars, Height, MonthIndex, QuarterIndex,
+    Sats, SemesterIndex, TxIndex, Version, WeekIndex, YearIndex,
+};
+use vecdb::{
+    AnyCloneableIterableVec, AnyCollectableVec, AnyVec, CollectableVec, Database, EagerVec, Exit,
+    GenericStoredVec, StoredIndex, VecIterator,
 };
 
-use crate::vecs::{Indexes, fetched, indexes};
+use crate::{
+    Indexes,
+    grouped::{LazyVecBuilder, Source},
+    indexes, price,
+};
 
-use super::{ComputedType, ComputedVecBuilder, StorableVecGeneatorOptions};
+use super::{ComputedType, EagerVecBuilder, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedVecsFromTxindex<T>
 where
     T: ComputedType + PartialOrd,
 {
     pub txindex: Option<Box<EagerVec<TxIndex, T>>>,
-    pub height: ComputedVecBuilder<Height, T>,
-    pub dateindex: ComputedVecBuilder<DateIndex, T>,
-    pub weekindex: ComputedVecBuilder<WeekIndex, T>,
-    pub difficultyepoch: ComputedVecBuilder<DifficultyEpoch, T>,
-    pub monthindex: ComputedVecBuilder<MonthIndex, T>,
-    pub quarterindex: ComputedVecBuilder<QuarterIndex, T>,
-    pub yearindex: ComputedVecBuilder<YearIndex, T>,
+    pub height: EagerVecBuilder<Height, T>,
+    pub dateindex: EagerVecBuilder<DateIndex, T>,
+    pub weekindex: LazyVecBuilder<WeekIndex, T, DateIndex, WeekIndex>,
+    pub difficultyepoch: EagerVecBuilder<DifficultyEpoch, T>,
+    pub monthindex: LazyVecBuilder<MonthIndex, T, DateIndex, MonthIndex>,
+    pub quarterindex: LazyVecBuilder<QuarterIndex, T, DateIndex, QuarterIndex>,
+    pub semesterindex: LazyVecBuilder<SemesterIndex, T, DateIndex, SemesterIndex>,
+    pub yearindex: LazyVecBuilder<YearIndex, T, DateIndex, YearIndex>,
     // TODO: pub halvingepoch: StorableVecGeneator<Halvingepoch, T>,
-    pub decadeindex: ComputedVecBuilder<DecadeIndex, T>,
+    pub decadeindex: LazyVecBuilder<DecadeIndex, T, DateIndex, DecadeIndex>,
 }
 
 const VERSION: Version = Version::ZERO;
 
 impl<T> ComputedVecsFromTxindex<T>
 where
-    T: ComputedType + Ord + From<f64>,
+    T: ComputedType + Ord + From<f64> + 'static,
     f64: From<T>,
 {
+    #[allow(clippy::too_many_arguments)]
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         name: &str,
-        compute_source: bool,
+        source: Source<TxIndex, T>,
         version: Version,
-        format: Format,
-        options: StorableVecGeneatorOptions,
-    ) -> color_eyre::Result<Self> {
-        let txindex = compute_source.then(|| {
+        indexes: &indexes::Vecs,
+        options: VecBuilderOptions,
+    ) -> Result<Self> {
+        let txindex = source.is_compute().then(|| {
             Box::new(
-                EagerVec::forced_import(path, name, version + VERSION + Version::ZERO, format)
+                EagerVec::forced_import_compressed(db, name, version + VERSION + Version::ZERO)
                     .unwrap(),
             )
         });
 
-        let height = ComputedVecBuilder::forced_import(
-            path,
+        let height = EagerVecBuilder::forced_import_compressed(
+            db,
             name,
             version + VERSION + Version::ZERO,
-            format,
             options,
         )?;
 
         let options = options.remove_percentiles();
 
+        let dateindex = EagerVecBuilder::forced_import_compressed(
+            db,
+            name,
+            version + VERSION + Version::ZERO,
+            options,
+        )?;
+
         Ok(Self {
+            weekindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.weekindex_to_weekindex.boxed_clone(),
+                options.into(),
+            ),
+            monthindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.monthindex_to_monthindex.boxed_clone(),
+                options.into(),
+            ),
+            quarterindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.quarterindex_to_quarterindex.boxed_clone(),
+                options.into(),
+            ),
+            semesterindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.semesterindex_to_semesterindex.boxed_clone(),
+                options.into(),
+            ),
+            yearindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.yearindex_to_yearindex.boxed_clone(),
+                options.into(),
+            ),
+            decadeindex: LazyVecBuilder::forced_import(
+                name,
+                version + VERSION + Version::ZERO,
+                None,
+                &dateindex,
+                indexes.decadeindex_to_decadeindex.boxed_clone(),
+                options.into(),
+            ),
+
             txindex,
             height,
-            dateindex: ComputedVecBuilder::forced_import(
-                path,
+            dateindex,
+            difficultyepoch: EagerVecBuilder::forced_import_compressed(
+                db,
                 name,
                 version + VERSION + Version::ZERO,
-                format,
-                options,
-            )?,
-            weekindex: ComputedVecBuilder::forced_import(
-                path,
-                name,
-                version + VERSION + Version::ZERO,
-                format,
-                options,
-            )?,
-            difficultyepoch: ComputedVecBuilder::forced_import(
-                path,
-                name,
-                version + VERSION + Version::ZERO,
-                format,
-                options,
-            )?,
-            monthindex: ComputedVecBuilder::forced_import(
-                path,
-                name,
-                version + VERSION + Version::ZERO,
-                format,
-                options,
-            )?,
-            quarterindex: ComputedVecBuilder::forced_import(
-                path,
-                name,
-                version + VERSION + Version::ZERO,
-                format,
-                options,
-            )?,
-            yearindex: ComputedVecBuilder::forced_import(
-                path,
-                name,
-                version + VERSION + Version::ZERO,
-                format,
-                options,
-            )?,
-            // halvingepoch: StorableVecGeneator::forced_import(path, name, version + VERSION + Version::ZERO, format, options)?,
-            decadeindex: ComputedVecBuilder::forced_import(
-                path,
-                name,
-                version + VERSION + Version::ZERO,
-                format,
                 options,
             )?,
+            // halvingepoch: StorableVecGeneator::forced_import(db, name, version + VERSION + Version::ZERO, format, options)?,
         })
     }
 
@@ -127,7 +146,7 @@ where
     //     starting_indexes: &Indexes,
     //     exit: &Exit,
     //     mut compute: F,
-    // ) -> color_eyre::Result<()>
+    // ) -> Result<()>
     // where
     //     F: FnMut(
     //         &mut EagerVec<TxIndex, T>,
@@ -163,7 +182,7 @@ where
             self.height.compute(
                 starting_indexes.height,
                 txindex,
-                &indexer.vecs().height_to_first_txindex,
+                &indexer.vecs.height_to_first_txindex,
                 &indexes.height_to_txindex_count,
                 exit,
             )?;
@@ -173,7 +192,7 @@ where
             self.height.compute(
                 starting_indexes.height,
                 txindex,
-                &indexer.vecs().height_to_first_txindex,
+                &indexer.vecs.height_to_first_txindex,
                 &indexes.height_to_txindex_count,
                 exit,
             )?;
@@ -196,46 +215,6 @@ where
             exit,
         )?;
 
-        self.weekindex.from_aligned(
-            starting_indexes.weekindex,
-            &self.dateindex,
-            &indexes.weekindex_to_first_dateindex,
-            &indexes.weekindex_to_dateindex_count,
-            exit,
-        )?;
-
-        self.monthindex.from_aligned(
-            starting_indexes.monthindex,
-            &self.dateindex,
-            &indexes.monthindex_to_first_dateindex,
-            &indexes.monthindex_to_dateindex_count,
-            exit,
-        )?;
-
-        self.quarterindex.from_aligned(
-            starting_indexes.quarterindex,
-            &self.monthindex,
-            &indexes.quarterindex_to_first_monthindex,
-            &indexes.quarterindex_to_monthindex_count,
-            exit,
-        )?;
-
-        self.yearindex.from_aligned(
-            starting_indexes.yearindex,
-            &self.monthindex,
-            &indexes.yearindex_to_first_monthindex,
-            &indexes.yearindex_to_monthindex_count,
-            exit,
-        )?;
-
-        self.decadeindex.from_aligned(
-            starting_indexes.decadeindex,
-            &self.yearindex,
-            &indexes.decadeindex_to_first_yearindex,
-            &indexes.decadeindex_to_yearindex_count,
-            exit,
-        )?;
-
         self.difficultyepoch.from_aligned(
             starting_indexes.difficultyepoch,
             &self.height,
@@ -247,24 +226,25 @@ where
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> = Box::new(
             self.txindex
                 .as_ref()
-                .map_or(vec![], |v| vec![v.as_ref() as &dyn AnyCollectableVec]),
-            self.height.vecs(),
-            self.dateindex.vecs(),
-            self.weekindex.vecs(),
-            self.difficultyepoch.vecs(),
-            self.monthindex.vecs(),
-            self.quarterindex.vecs(),
-            self.yearindex.vecs(),
-            // self.halvingepoch.vecs(),
-            self.decadeindex.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+                .map(|x| x.as_ref() as &dyn AnyCollectableVec)
+                .into_iter(),
+        );
+
+        iter = Box::new(iter.chain(self.height.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.dateindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.weekindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.difficultyepoch.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.monthindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.quarterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.semesterindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.yearindex.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.decadeindex.iter_any_collectable()));
+
+        iter
     }
 }
 
@@ -285,11 +265,11 @@ impl ComputedVecsFromTxindex<Bitcoin> {
         };
 
         self.height
-            .validate_computed_version_or_reset_file(txindex_version)?;
+            .validate_computed_version_or_reset(txindex_version)?;
 
         let starting_index = self.height.starting_index(starting_indexes.height);
 
-        (starting_index.unwrap_to_usize()..indexer.vecs().height_to_weight.len())
+        (starting_index.unwrap_to_usize()..indexer.vecs.height_to_weight.len())
             .map(Height::from)
             .try_for_each(|height| -> Result<()> {
                 if let Some(first) = self.height.first.as_mut() {
@@ -340,24 +320,24 @@ impl ComputedVecsFromTxindex<Bitcoin> {
                         exit,
                     )?;
                 }
-                if let Some(_90p) = self.height._90p.as_mut() {
-                    _90p.forced_push_at(
+                if let Some(pct90) = self.height.pct90.as_mut() {
+                    pct90.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_90p()
+                                .unwrap_pct90()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
                         exit,
                     )?;
                 }
-                if let Some(_75p) = self.height._75p.as_mut() {
-                    _75p.forced_push_at(
+                if let Some(pct75) = self.height.pct75.as_mut() {
+                    pct75.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_75p()
+                                .unwrap_pct75()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
@@ -376,24 +356,24 @@ impl ComputedVecsFromTxindex<Bitcoin> {
                         exit,
                     )?;
                 }
-                if let Some(_25p) = self.height._25p.as_mut() {
-                    _25p.forced_push_at(
+                if let Some(pct25) = self.height.pct25.as_mut() {
+                    pct25.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_25p()
+                                .unwrap_pct25()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
                         exit,
                     )?;
                 }
-                if let Some(_10p) = self.height._10p.as_mut() {
-                    _10p.forced_push_at(
+                if let Some(pct10) = self.height.pct10.as_mut() {
+                    pct10.forced_push_at(
                         height,
                         Bitcoin::from(
                             sats.height
-                                .unwrap_10p()
+                                .unwrap_pct10()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         ),
@@ -455,7 +435,7 @@ impl ComputedVecsFromTxindex<Dollars> {
         exit: &Exit,
         bitcoin: &ComputedVecsFromTxindex<Bitcoin>,
         txindex: Option<&impl CollectableVec<TxIndex, Dollars>>,
-        fetched: &fetched::Vecs,
+        price: &price::Vecs,
     ) -> Result<()> {
         let txindex_version = if let Some(txindex) = txindex {
             txindex.version()
@@ -464,13 +444,13 @@ impl ComputedVecsFromTxindex<Dollars> {
         };
 
         self.height
-            .validate_computed_version_or_reset_file(txindex_version)?;
+            .validate_computed_version_or_reset(txindex_version)?;
 
         let starting_index = self.height.starting_index(starting_indexes.height);
 
-        let mut close_iter = fetched.chainindexes_to_close.height.into_iter();
+        let mut close_iter = price.chainindexes_to_price_close.height.into_iter();
 
-        (starting_index.unwrap_to_usize()..indexer.vecs().height_to_weight.len())
+        (starting_index.unwrap_to_usize()..indexer.vecs.height_to_weight.len())
             .map(Height::from)
             .try_for_each(|height| -> Result<()> {
                 let price = *close_iter.unwrap_get_inner(height);
@@ -523,25 +503,25 @@ impl ComputedVecsFromTxindex<Dollars> {
                         exit,
                     )?;
                 }
-                if let Some(_90p) = self.height._90p.as_mut() {
-                    _90p.forced_push_at(
+                if let Some(pct90) = self.height.pct90.as_mut() {
+                    pct90.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_90p()
+                                .unwrap_pct90()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
                     )?;
                 }
-                if let Some(_75p) = self.height._75p.as_mut() {
-                    _75p.forced_push_at(
+                if let Some(pct75) = self.height.pct75.as_mut() {
+                    pct75.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_75p()
+                                .unwrap_pct75()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
@@ -559,25 +539,25 @@ impl ComputedVecsFromTxindex<Dollars> {
                         exit,
                     )?;
                 }
-                if let Some(_25p) = self.height._25p.as_mut() {
-                    _25p.forced_push_at(
+                if let Some(pct25) = self.height.pct25.as_mut() {
+                    pct25.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_25p()
+                                .unwrap_pct25()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,
                     )?;
                 }
-                if let Some(_10p) = self.height._10p.as_mut() {
-                    _10p.forced_push_at(
+                if let Some(pct10) = self.height.pct10.as_mut() {
+                    pct10.forced_push_at(
                         height,
                         price
                             * bitcoin
                                 .height
-                                .unwrap_10p()
+                                .unwrap_pct10()
                                 .into_iter()
                                 .unwrap_get_inner(height),
                         exit,

crates/brk_computer/src/grouped/mod.rs

@@ -1,22 +1,28 @@
-mod builder;
+mod builder_eager;
+mod builder_lazy;
+mod computed;
 mod from_dateindex;
 mod from_height;
 mod from_height_strict;
 mod from_txindex;
 mod ratio_from_dateindex;
-mod r#type;
+mod sd_from_dateindex;
+mod source;
 mod value_from_dateindex;
 mod value_from_height;
 mod value_from_txindex;
 mod value_height;
 
-pub use builder::*;
+pub use builder_eager::*;
+pub use builder_lazy::*;
+use computed::*;
 pub use from_dateindex::*;
 pub use from_height::*;
 pub use from_height_strict::*;
 pub use from_txindex::*;
 pub use ratio_from_dateindex::*;
-use r#type::*;
+pub use sd_from_dateindex::*;
+pub use source::*;
 pub use value_from_dateindex::*;
 pub use value_from_height::*;
 pub use value_from_txindex::*;

crates/brk_computer/src/grouped/ratio_from_dateindex.rs

@@ -0,0 +1,763 @@
+use brk_error::Result;
+use brk_structs::{Date, DateIndex, Dollars, StoredF32, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, CollectableVec, Database, EagerVec,
+    Exit, GenericStoredVec, StoredIndex, VecIterator,
+};
+
+use crate::{
+    Indexes,
+    grouped::{
+        ComputedStandardDeviationVecsFromDateIndex, StandardDeviationVecsOptions, source::Source,
+    },
+    indexes, price,
+    utils::get_percentile,
+};
+
+use super::{ComputedVecsFromDateIndex, VecBuilderOptions};
+
+#[derive(Clone)]
+pub struct ComputedRatioVecsFromDateIndex {
+    pub price: Option<ComputedVecsFromDateIndex<Dollars>>,
+
+    pub ratio: ComputedVecsFromDateIndex<StoredF32>,
+    pub ratio_1w_sma: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_1m_sma: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct99: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct98: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct95: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct5: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct2: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct1: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub ratio_pct99_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct98_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct95_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct5_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct2_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub ratio_pct1_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+
+    pub ratio_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
+    pub ratio_4y_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
+    pub ratio_2y_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
+    pub ratio_1y_sd: Option<ComputedStandardDeviationVecsFromDateIndex>,
+}
+
+const VERSION: Version = Version::TWO;
+
+impl ComputedRatioVecsFromDateIndex {
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        source: Source<DateIndex, Dollars>,
+        version: Version,
+        indexes: &indexes::Vecs,
+        extended: bool,
+    ) -> Result<Self> {
+        let options = VecBuilderOptions::default().add_last();
+
+        Ok(Self {
+            price: source.is_compute().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    name,
+                    Source::Compute,
+                    version + VERSION,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &format!("{name}_ratio"),
+                Source::Compute,
+                version + VERSION + Version::ZERO,
+                indexes,
+                options,
+            )?,
+            ratio_1w_sma: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_1w_sma"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_1m_sma: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_1m_sma"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_sd: extended.then(|| {
+                ComputedStandardDeviationVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio"),
+                    usize::MAX,
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
+                )
+                .unwrap()
+            }),
+            ratio_1y_sd: extended.then(|| {
+                ComputedStandardDeviationVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_1y"),
+                    365,
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
+                )
+                .unwrap()
+            }),
+            ratio_2y_sd: extended.then(|| {
+                ComputedStandardDeviationVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_2y"),
+                    2 * 365,
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
+                )
+                .unwrap()
+            }),
+            ratio_4y_sd: extended.then(|| {
+                ComputedStandardDeviationVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_4y"),
+                    4 * 365,
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    StandardDeviationVecsOptions::default().add_all(),
+                )
+                .unwrap()
+            }),
+            ratio_pct99: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct99"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct98: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct98"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct95: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct95"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct5: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct5"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct2: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct2"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct1: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct1"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct99_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct99_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct98_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct98_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct95_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct95_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct5_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct5_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct2_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct2_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+            ratio_pct1_usd: extended.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_ratio_pct1_usd"),
+                    Source::Compute,
+                    version + VERSION + Version::ZERO,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+        })
+    }
+
+    pub fn compute_all<F>(
+        &mut self,
+        price: &price::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        compute: F,
+    ) -> Result<()>
+    where
+        F: FnMut(&mut EagerVec<DateIndex, Dollars>) -> Result<()>,
+    {
+        self.price
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, compute)?;
+
+        let date_to_price_opt: Option<&EagerVec<DateIndex, Dollars>> = None;
+        self.compute_rest(price, starting_indexes, exit, date_to_price_opt)
+    }
+
+    pub fn compute_rest(
+        &mut self,
+        price: &price::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        price_opt: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+    ) -> Result<()> {
+        let closes = price.timeindexes_to_price_close.dateindex.as_ref().unwrap();
+
+        let price = price_opt.unwrap_or_else(|| unsafe {
+            std::mem::transmute(&self.price.as_ref().unwrap().dateindex)
+        });
+
+        self.ratio.compute_all(starting_indexes, exit, |v| {
+            v.compute_transform2(
+                starting_indexes.dateindex,
+                closes,
+                price,
+                |(i, close, price, ..)| {
+                    if price == Dollars::ZERO {
+                        (i, StoredF32::from(1.0))
+                    } else {
+                        (i, StoredF32::from(*close / price))
+                    }
+                },
+                exit,
+            )?;
+            Ok(())
+        })?;
+
+        if self.ratio_1w_sma.is_none() {
+            return Ok(());
+        }
+
+        let min_ratio_date = DateIndex::try_from(Date::MIN_RATIO).unwrap();
+
+        self.ratio_1w_sma
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sma_(
+                    starting_indexes.dateindex,
+                    self.ratio.dateindex.as_ref().unwrap(),
+                    7,
+                    exit,
+                    Some(min_ratio_date),
+                )?;
+                Ok(())
+            })?;
+
+        self.ratio_1m_sma
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sma_(
+                    starting_indexes.dateindex,
+                    self.ratio.dateindex.as_ref().unwrap(),
+                    30,
+                    exit,
+                    Some(min_ratio_date),
+                )?;
+                Ok(())
+            })?;
+
+        let ratio_version = self.ratio.dateindex.as_ref().unwrap().version();
+        self.mut_ratio_vecs()
+            .iter_mut()
+            .try_for_each(|v| -> Result<()> {
+                v.validate_computed_version_or_reset(
+                    Version::ZERO + v.inner_version() + ratio_version,
+                )?;
+                Ok(())
+            })?;
+
+        let starting_dateindex = self
+            .mut_ratio_vecs()
+            .iter()
+            .map(|v| DateIndex::from(v.len()))
+            .min()
+            .unwrap()
+            .min(starting_indexes.dateindex);
+
+        let mut sorted = self.ratio.dateindex.as_ref().unwrap().collect_range(
+            Some(min_ratio_date.unwrap_to_usize()),
+            Some(starting_dateindex.unwrap_to_usize()),
+        )?;
+
+        sorted.sort_unstable();
+
+        self.ratio
+            .dateindex
+            .as_ref()
+            .unwrap()
+            .iter_at(starting_dateindex)
+            .try_for_each(|(index, ratio)| -> Result<()> {
+                if index < min_ratio_date {
+                    self.ratio_pct5
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                    self.ratio_pct2
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                    self.ratio_pct1
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                    self.ratio_pct95
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                    self.ratio_pct98
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                    self.ratio_pct99
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, StoredF32::NAN, exit)?;
+                } else {
+                    let ratio = ratio.into_owned();
+                    let pos = sorted.binary_search(&ratio).unwrap_or_else(|pos| pos);
+                    sorted.insert(pos, ratio);
+
+                    self.ratio_pct1
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.01), exit)?;
+                    self.ratio_pct2
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.02), exit)?;
+                    self.ratio_pct5
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.05), exit)?;
+                    self.ratio_pct95
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.95), exit)?;
+                    self.ratio_pct98
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.98), exit)?;
+                    self.ratio_pct99
+                        .as_mut()
+                        .unwrap()
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, get_percentile(&sorted, 0.99), exit)?;
+                }
+
+                Ok(())
+            })?;
+
+        self.mut_ratio_vecs()
+            .into_iter()
+            .try_for_each(|v| v.safe_flush(exit))?;
+
+        self.ratio_pct1.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+        self.ratio_pct2.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+        self.ratio_pct5.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+        self.ratio_pct95.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+        self.ratio_pct98.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+        self.ratio_pct99.as_mut().unwrap().compute_rest(
+            starting_indexes,
+            exit,
+            None as Option<&EagerVec<_, _>>,
+        )?;
+
+        let date_to_price = price_opt.unwrap_or_else(|| unsafe {
+            std::mem::transmute(&self.price.as_ref().unwrap().dateindex)
+        });
+
+        self.ratio_pct99_usd
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, |vec| {
+                let mut iter = self
+                    .ratio_pct99
+                    .as_ref()
+                    .unwrap()
+                    .dateindex
+                    .as_ref()
+                    .unwrap()
+                    .into_iter();
+                vec.compute_transform(
+                    starting_indexes.dateindex,
+                    date_to_price,
+                    |(i, price, ..)| {
+                        let multiplier = iter.unwrap_get_inner(i);
+                        (i, price * multiplier)
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        let compute_usd =
+            |usd: Option<&mut ComputedVecsFromDateIndex<Dollars>>,
+             source: Option<&ComputedVecsFromDateIndex<StoredF32>>| {
+                usd.unwrap().compute_all(starting_indexes, exit, |vec| {
+                    let mut iter = source.unwrap().dateindex.as_ref().unwrap().into_iter();
+                    vec.compute_transform(
+                        starting_indexes.dateindex,
+                        date_to_price,
+                        |(i, price, ..)| {
+                            let multiplier = iter.unwrap_get_inner(i);
+                            (i, price * multiplier)
+                        },
+                        exit,
+                    )?;
+                    Ok(())
+                })
+            };
+
+        compute_usd(self.ratio_pct1_usd.as_mut(), self.ratio_pct1.as_ref())?;
+        compute_usd(self.ratio_pct2_usd.as_mut(), self.ratio_pct2.as_ref())?;
+        compute_usd(self.ratio_pct5_usd.as_mut(), self.ratio_pct5.as_ref())?;
+        compute_usd(self.ratio_pct95_usd.as_mut(), self.ratio_pct95.as_ref())?;
+        compute_usd(self.ratio_pct98_usd.as_mut(), self.ratio_pct98.as_ref())?;
+        compute_usd(self.ratio_pct99_usd.as_mut(), self.ratio_pct99.as_ref())?;
+
+        self.ratio_sd.as_mut().unwrap().compute_all(
+            starting_indexes,
+            exit,
+            self.ratio.dateindex.as_ref().unwrap(),
+            Some(date_to_price),
+        )?;
+        self.ratio_4y_sd.as_mut().unwrap().compute_all(
+            starting_indexes,
+            exit,
+            self.ratio.dateindex.as_ref().unwrap(),
+            Some(date_to_price),
+        )?;
+        self.ratio_2y_sd.as_mut().unwrap().compute_all(
+            starting_indexes,
+            exit,
+            self.ratio.dateindex.as_ref().unwrap(),
+            Some(date_to_price),
+        )?;
+        self.ratio_1y_sd.as_mut().unwrap().compute_all(
+            starting_indexes,
+            exit,
+            self.ratio.dateindex.as_ref().unwrap(),
+            Some(date_to_price),
+        )?;
+
+        Ok(())
+    }
+
+    fn mut_ratio_vecs(&mut self) -> Vec<&mut EagerVec<DateIndex, StoredF32>> {
+        [
+            self.ratio_pct1
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+            self.ratio_pct2
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+            self.ratio_pct5
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+            self.ratio_pct95
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+            self.ratio_pct98
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+            self.ratio_pct99
+                .as_mut()
+                .map_or(vec![], |v| vec![v.dateindex.as_mut().unwrap()]),
+        ]
+        .into_iter()
+        .flatten()
+        .collect::<Vec<_>>()
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(self.price.iter().flat_map(|v| v.iter_any_collectable()));
+
+        iter = Box::new(iter.chain(self.ratio.iter_any_collectable()));
+        iter = Box::new(
+            iter.chain(
+                self.ratio_1w_sma
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_1m_sma
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.ratio_sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.ratio_1y_sd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_2y_sd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_4y_sd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct1
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct2
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct5
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct95
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct98
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct99
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct1_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct2_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct5_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct95_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct98_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(
+            iter.chain(
+                self.ratio_pct99_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+
+        iter
+    }
+}

crates/brk_computer/src/grouped/sd_from_dateindex.rs

@@ -0,0 +1,878 @@
+use brk_error::Result;
+use brk_structs::{CheckedSub, Date, DateIndex, Dollars, StoredF32, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, BoxedVecIterator, CollectableVec,
+    Database, EagerVec, Exit, GenericStoredVec, StoredIndex,
+};
+
+use crate::{Indexes, grouped::source::Source, indexes};
+
+use super::{ComputedVecsFromDateIndex, VecBuilderOptions};
+
+#[derive(Clone)]
+pub struct ComputedStandardDeviationVecsFromDateIndex {
+    days: usize,
+
+    pub sma: Option<ComputedVecsFromDateIndex<StoredF32>>,
+
+    pub sd: ComputedVecsFromDateIndex<StoredF32>,
+
+    pub zscore: Option<ComputedVecsFromDateIndex<StoredF32>>,
+
+    pub p0_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub p1sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub p1_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub p2sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub p2_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub p3sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m0_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m1sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m1_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m2sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m2_5sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+    pub m3sd: Option<ComputedVecsFromDateIndex<StoredF32>>,
+
+    pub _0sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p0_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p1sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p1_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p2sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p2_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub p3sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m0_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m1sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m1_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m2sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m2_5sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+    pub m3sd_usd: Option<ComputedVecsFromDateIndex<Dollars>>,
+}
+
+#[derive(Debug, Default)]
+pub struct StandardDeviationVecsOptions {
+    zscore: bool,
+    bands: bool,
+    price_bands: bool,
+}
+
+impl StandardDeviationVecsOptions {
+    pub fn add_all(mut self) -> Self {
+        self.zscore = true;
+        self.bands = true;
+        self.price_bands = true;
+        self
+    }
+
+    pub fn add_zscore(mut self) -> Self {
+        self.zscore = true;
+        self
+    }
+
+    pub fn add_bands(mut self) -> Self {
+        self.bands = true;
+        self
+    }
+
+    pub fn add_price_bands(mut self) -> Self {
+        self.bands = true;
+        self.price_bands = true;
+        self
+    }
+
+    pub fn zscore(&self) -> bool {
+        self.zscore
+    }
+
+    pub fn bands(&self) -> bool {
+        self.bands
+    }
+
+    pub fn price_bands(&self) -> bool {
+        self.price_bands
+    }
+}
+
+impl ComputedStandardDeviationVecsFromDateIndex {
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        days: usize,
+        sma: Source<DateIndex, StoredF32>,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+        options: StandardDeviationVecsOptions,
+    ) -> Result<Self> {
+        let builder_options = VecBuilderOptions::default().add_last();
+
+        let version = parent_version + Version::ONE;
+
+        Ok(Self {
+            days,
+            sma: sma.is_compute().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_sma"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            sd: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &format!("{name}_sd"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                builder_options,
+            )?,
+            p0_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p0_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p1sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p1sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p1_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p1_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p2sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p2sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p2_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p2_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p3sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p3sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m0_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m0_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m1sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m1sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m1_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m1_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m2sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m2sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m2_5sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m2_5sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m3sd: options.bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m3sd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            _0sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_0sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p0_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p0_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p1sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p1sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p1_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p1_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p2sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p2sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p2_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p2_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            p3sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_p3sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m0_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m0_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m1sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m1sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m1_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m1_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m2sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m2sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m2_5sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m2_5sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            m3sd_usd: options.price_bands().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_m3sd_usd"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+            zscore: options.zscore().then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_zscore"),
+                    Source::Compute,
+                    version + Version::ZERO,
+                    indexes,
+                    builder_options,
+                )
+                .unwrap()
+            }),
+        })
+    }
+
+    pub fn compute_all(
+        &mut self,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        source: &impl CollectableVec<DateIndex, StoredF32>,
+        price_opt: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+    ) -> Result<()> {
+        let min_date = DateIndex::try_from(Date::MIN_RATIO).unwrap();
+
+        self.sma
+            .as_mut()
+            .unwrap()
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sma_(
+                    starting_indexes.dateindex,
+                    source,
+                    self.days,
+                    exit,
+                    Some(min_date),
+                )?;
+                Ok(())
+            })?;
+
+        let sma_opt: Option<&EagerVec<DateIndex, StoredF32>> = None;
+        self.compute_rest(starting_indexes, exit, sma_opt, source, price_opt)
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn compute_rest(
+        &mut self,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        sma_opt: Option<&impl AnyIterableVec<DateIndex, StoredF32>>,
+        source: &impl CollectableVec<DateIndex, StoredF32>,
+        price_opt: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+    ) -> Result<()> {
+        let sma = sma_opt.unwrap_or_else(|| unsafe {
+            std::mem::transmute(&self.sma.as_ref().unwrap().dateindex)
+        });
+
+        let min_date = DateIndex::try_from(Date::MIN_RATIO).unwrap();
+
+        let source_version = source.version();
+
+        self.mut_stateful_date_vecs()
+            .try_for_each(|v| -> Result<()> {
+                v.validate_computed_version_or_reset(
+                    Version::ZERO + v.inner_version() + source_version,
+                )?;
+                Ok(())
+            })?;
+
+        let starting_dateindex = self
+            .mut_stateful_date_vecs()
+            .map(|v| DateIndex::from(v.len()))
+            .min()
+            .unwrap()
+            .min(starting_indexes.dateindex);
+
+        let mut sorted = source.collect_range(
+            Some(min_date.unwrap_to_usize()),
+            Some(starting_dateindex.unwrap_to_usize()),
+        )?;
+
+        sorted.sort_unstable();
+
+        let mut sma_iter = sma.iter();
+
+        let mut p0_5sd = self.p0_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut p1sd = self.p1sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut p1_5sd = self.p1_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut p2sd = self.p2sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut p2_5sd = self.p2_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut p3sd = self.p3sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m0_5sd = self.m0_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m1sd = self.m1sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m1_5sd = self.m1_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m2sd = self.m2sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m2_5sd = self.m2_5sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+        let mut m3sd = self.m3sd.as_mut().map(|c| c.dateindex.as_mut().unwrap());
+
+        source
+            .iter_at(starting_dateindex)
+            .try_for_each(|(index, ratio)| -> Result<()> {
+                if index < min_date {
+                    self.sd.dateindex.as_mut().unwrap().forced_push_at(
+                        index,
+                        StoredF32::NAN,
+                        exit,
+                    )?;
+
+                    if let Some(v) = p0_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = p1sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = p1_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = p2sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = p2_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = p3sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m0_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m1sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m1_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m2sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m2_5sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                    if let Some(v) = m3sd.as_mut() {
+                        v.forced_push_at(index, StoredF32::NAN, exit)?
+                    }
+                } else {
+                    let ratio = ratio.into_owned();
+                    let pos = sorted.binary_search(&ratio).unwrap_or_else(|pos| pos);
+                    sorted.insert(pos, ratio);
+
+                    let avg = sma_iter.unwrap_get_inner(index);
+
+                    let population =
+                        index.checked_sub(min_date).unwrap().unwrap_to_usize() as f32 + 1.0;
+
+                    let sd = StoredF32::from(
+                        (sorted.iter().map(|v| (**v - *avg).powi(2)).sum::<f32>() / population)
+                            .sqrt(),
+                    );
+
+                    self.sd
+                        .dateindex
+                        .as_mut()
+                        .unwrap()
+                        .forced_push_at(index, sd, exit)?;
+                    if let Some(v) = p0_5sd.as_mut() {
+                        v.forced_push_at(index, avg + StoredF32::from(0.5 * *sd), exit)?
+                    }
+                    if let Some(v) = p1sd.as_mut() {
+                        v.forced_push_at(index, avg + sd, exit)?
+                    }
+                    if let Some(v) = p1_5sd.as_mut() {
+                        v.forced_push_at(index, avg + StoredF32::from(1.5 * *sd), exit)?
+                    }
+                    if let Some(v) = p2sd.as_mut() {
+                        v.forced_push_at(index, avg + 2 * sd, exit)?
+                    }
+                    if let Some(v) = p2_5sd.as_mut() {
+                        v.forced_push_at(index, avg + StoredF32::from(2.5 * *sd), exit)?
+                    }
+                    if let Some(v) = p3sd.as_mut() {
+                        v.forced_push_at(index, avg + 3 * sd, exit)?
+                    }
+                    if let Some(v) = m0_5sd.as_mut() {
+                        v.forced_push_at(index, avg - StoredF32::from(0.5 * *sd), exit)?
+                    }
+                    if let Some(v) = m1sd.as_mut() {
+                        v.forced_push_at(index, avg - sd, exit)?
+                    }
+                    if let Some(v) = m1_5sd.as_mut() {
+                        v.forced_push_at(index, avg - StoredF32::from(1.5 * *sd), exit)?
+                    }
+                    if let Some(v) = m2sd.as_mut() {
+                        v.forced_push_at(index, avg - 2 * sd, exit)?
+                    }
+                    if let Some(v) = m2_5sd.as_mut() {
+                        v.forced_push_at(index, avg - StoredF32::from(2.5 * *sd), exit)?
+                    }
+                    if let Some(v) = m3sd.as_mut() {
+                        v.forced_push_at(index, avg - 3 * sd, exit)?
+                    }
+                }
+
+                Ok(())
+            })?;
+
+        drop(sma_iter);
+
+        self.mut_stateful_date_vecs()
+            .try_for_each(|v| v.safe_flush(exit))?;
+
+        self.mut_stateful_computed().try_for_each(|v| {
+            v.compute_rest(starting_indexes, exit, None as Option<&EagerVec<_, _>>)
+        })?;
+
+        if let Some(zscore) = self.zscore.as_mut() {
+            zscore.compute_all(starting_indexes, exit, |vec| {
+                vec.compute_zscore(
+                    starting_indexes.dateindex,
+                    source,
+                    sma,
+                    self.sd.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+        }
+
+        let Some(price) = price_opt else {
+            return Ok(());
+        };
+
+        let compute_usd =
+            |usd: &mut ComputedVecsFromDateIndex<Dollars>,
+             mut iter: BoxedVecIterator<DateIndex, StoredF32>| {
+                usd.compute_all(starting_indexes, exit, |vec| {
+                    vec.compute_transform(
+                        starting_indexes.dateindex,
+                        price,
+                        |(i, price, ..)| {
+                            let multiplier = iter.unwrap_get_inner(i);
+                            (i, price * multiplier)
+                        },
+                        exit,
+                    )?;
+                    Ok(())
+                })
+            };
+
+        if self._0sd_usd.is_none() {
+            return Ok(());
+        }
+
+        compute_usd(self._0sd_usd.as_mut().unwrap(), sma.iter())?;
+        compute_usd(
+            self.p0_5sd_usd.as_mut().unwrap(),
+            self.p0_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.p1sd_usd.as_mut().unwrap(),
+            self.p1sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.p1_5sd_usd.as_mut().unwrap(),
+            self.p1_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.p2sd_usd.as_mut().unwrap(),
+            self.p2sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.p2_5sd_usd.as_mut().unwrap(),
+            self.p2_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.p3sd_usd.as_mut().unwrap(),
+            self.p3sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m0_5sd_usd.as_mut().unwrap(),
+            self.m0_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m1sd_usd.as_mut().unwrap(),
+            self.m1sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m1_5sd_usd.as_mut().unwrap(),
+            self.m1_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m2sd_usd.as_mut().unwrap(),
+            self.m2sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m2_5sd_usd.as_mut().unwrap(),
+            self.m2_5sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+        compute_usd(
+            self.m3sd_usd.as_mut().unwrap(),
+            self.m3sd
+                .as_ref()
+                .unwrap()
+                .dateindex
+                .as_ref()
+                .unwrap()
+                .iter(),
+        )?;
+
+        Ok(())
+    }
+
+    fn mut_stateful_computed(
+        &mut self,
+    ) -> impl Iterator<Item = &mut ComputedVecsFromDateIndex<StoredF32>> {
+        [
+            Some(&mut self.sd),
+            self.p0_5sd.as_mut(),
+            self.p1sd.as_mut(),
+            self.p1_5sd.as_mut(),
+            self.p2sd.as_mut(),
+            self.p2_5sd.as_mut(),
+            self.p3sd.as_mut(),
+            self.m0_5sd.as_mut(),
+            self.m1sd.as_mut(),
+            self.m1_5sd.as_mut(),
+            self.m2sd.as_mut(),
+            self.m2_5sd.as_mut(),
+            self.m3sd.as_mut(),
+        ]
+        .into_iter()
+        .flatten()
+    }
+
+    fn mut_stateful_date_vecs(
+        &mut self,
+    ) -> impl Iterator<Item = &mut EagerVec<DateIndex, StoredF32>> {
+        self.mut_stateful_computed()
+            .map(|c| c.dateindex.as_mut().unwrap())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(self.sma.iter().flat_map(|v| v.iter_any_collectable()));
+
+        iter = Box::new(iter.chain(self.sd.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.p0_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.p1sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.p1_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.p2sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.p2_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.p3sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m0_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m1sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m1_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m2sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m2_5sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.m3sd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self._0sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.p0_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.p1sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.p1_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.p2sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.p2_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.p3sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.m0_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.m1sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.m1_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.m2sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(
+            iter.chain(
+                self.m2_5sd_usd
+                    .iter()
+                    .flat_map(|v| v.iter_any_collectable()),
+            ),
+        );
+        iter = Box::new(iter.chain(self.m3sd_usd.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.zscore.iter().flat_map(|v| v.iter_any_collectable())));
+
+        iter
+    }
+}

crates/brk_computer/src/grouped/source.rs

@@ -0,0 +1,51 @@
+use vecdb::AnyBoxedIterableVec;
+
+#[derive(Clone)]
+pub enum Source<I, T> {
+    Compute,
+    None,
+    Vec(AnyBoxedIterableVec<I, T>),
+}
+
+impl<I, T> Source<I, T> {
+    pub fn is_compute(&self) -> bool {
+        matches!(self, Self::Compute)
+    }
+
+    pub fn is_none(&self) -> bool {
+        matches!(self, Self::None)
+    }
+
+    pub fn is_vec(&self) -> bool {
+        matches!(self, Self::Vec(_))
+    }
+
+    pub fn vec(self) -> Option<AnyBoxedIterableVec<I, T>> {
+        match self {
+            Self::Vec(v) => Some(v),
+            _ => None,
+        }
+    }
+}
+
+impl<I, T> From<bool> for Source<I, T> {
+    fn from(value: bool) -> Self {
+        if value { Self::Compute } else { Self::None }
+    }
+}
+
+impl<I, T> From<AnyBoxedIterableVec<I, T>> for Source<I, T> {
+    fn from(value: AnyBoxedIterableVec<I, T>) -> Self {
+        Self::Vec(value)
+    }
+}
+
+impl<I, T> From<Option<AnyBoxedIterableVec<I, T>>> for Source<I, T> {
+    fn from(value: Option<AnyBoxedIterableVec<I, T>>) -> Self {
+        if let Some(v) = value {
+            Self::Vec(v)
+        } else {
+            Self::None
+        }
+    }
+}

crates/brk_computer/src/grouped/value_from_dateindex.rs

@@ -0,0 +1,140 @@
+use brk_error::Result;
+use brk_structs::{Bitcoin, DateIndex, Dollars, Sats, Version};
+use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, StoredVec};
+
+use crate::{
+    Indexes,
+    grouped::ComputedVecsFromDateIndex,
+    indexes, price,
+    traits::{ComputeFromBitcoin, ComputeFromSats},
+};
+
+use super::{Source, VecBuilderOptions};
+
+#[derive(Clone)]
+pub struct ComputedValueVecsFromDateIndex {
+    pub sats: ComputedVecsFromDateIndex<Sats>,
+    pub bitcoin: ComputedVecsFromDateIndex<Bitcoin>,
+    pub dollars: Option<ComputedVecsFromDateIndex<Dollars>>,
+}
+
+const VERSION: Version = Version::ZERO;
+
+impl ComputedValueVecsFromDateIndex {
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        source: Source<DateIndex, Sats>,
+        version: Version,
+        options: VecBuilderOptions,
+        compute_dollars: bool,
+        indexes: &indexes::Vecs,
+    ) -> Result<Self> {
+        Ok(Self {
+            sats: ComputedVecsFromDateIndex::forced_import(
+                db,
+                name,
+                source,
+                version + VERSION,
+                indexes,
+                options,
+            )?,
+            bitcoin: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &format!("{name}_btc"),
+                Source::Compute,
+                version + VERSION,
+                indexes,
+                options,
+            )?,
+            dollars: compute_dollars.then(|| {
+                ComputedVecsFromDateIndex::forced_import(
+                    db,
+                    &format!("{name}_usd"),
+                    Source::Compute,
+                    version + VERSION,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+        })
+    }
+
+    pub fn compute_all<F>(
+        &mut self,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        mut compute: F,
+    ) -> Result<()>
+    where
+        F: FnMut(&mut EagerVec<DateIndex, Sats>) -> Result<()>,
+    {
+        compute(self.sats.dateindex.as_mut().unwrap())?;
+
+        let dateindex: Option<&StoredVec<DateIndex, Sats>> = None;
+        self.compute_rest(price, starting_indexes, exit, dateindex)?;
+
+        Ok(())
+    }
+
+    pub fn compute_rest(
+        &mut self,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        dateindex: Option<&impl CollectableVec<DateIndex, Sats>>,
+    ) -> Result<()> {
+        if let Some(dateindex) = dateindex {
+            self.sats
+                .compute_rest(starting_indexes, exit, Some(dateindex))?;
+
+            self.bitcoin.compute_all(starting_indexes, exit, |v| {
+                v.compute_from_sats(starting_indexes.dateindex, dateindex, exit)
+            })?;
+        } else {
+            let dateindex: Option<&StoredVec<DateIndex, Sats>> = None;
+
+            self.sats.compute_rest(starting_indexes, exit, dateindex)?;
+
+            self.bitcoin.compute_all(starting_indexes, exit, |v| {
+                v.compute_from_sats(
+                    starting_indexes.dateindex,
+                    self.sats.dateindex.as_ref().unwrap(),
+                    exit,
+                )
+            })?;
+        }
+
+        let dateindex_to_bitcoin = self.bitcoin.dateindex.as_ref().unwrap();
+        let dateindex_to_price_close = price
+            .as_ref()
+            .unwrap()
+            .timeindexes_to_price_close
+            .dateindex
+            .as_ref()
+            .unwrap();
+
+        if let Some(dollars) = self.dollars.as_mut() {
+            dollars.compute_all(starting_indexes, exit, |v| {
+                v.compute_from_bitcoin(
+                    starting_indexes.dateindex,
+                    dateindex_to_bitcoin,
+                    dateindex_to_price_close,
+                    exit,
+                )
+            })?;
+        }
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        std::iter::empty()
+            .chain(self.sats.iter_any_collectable())
+            .chain(self.bitcoin.iter_any_collectable())
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
+    }
+}

crates/brk_computer/src/grouped/value_from_height.rs

@@ -0,0 +1,140 @@
+use allocative::Allocative;
+use brk_error::Result;
+use brk_structs::{Bitcoin, Dollars, Height, Sats, Version};
+use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, StoredVec};
+
+use crate::{
+    Indexes,
+    grouped::Source,
+    indexes, price,
+    traits::{ComputeFromBitcoin, ComputeFromSats},
+};
+
+use super::{ComputedVecsFromHeight, VecBuilderOptions};
+
+#[derive(Clone, Allocative)]
+pub struct ComputedValueVecsFromHeight {
+    pub sats: ComputedVecsFromHeight<Sats>,
+    pub bitcoin: ComputedVecsFromHeight<Bitcoin>,
+    pub dollars: Option<ComputedVecsFromHeight<Dollars>>,
+}
+
+const VERSION: Version = Version::ZERO;
+
+impl ComputedValueVecsFromHeight {
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        name: &str,
+        source: Source<Height, Sats>,
+        version: Version,
+        options: VecBuilderOptions,
+        compute_dollars: bool,
+        indexes: &indexes::Vecs,
+    ) -> Result<Self> {
+        Ok(Self {
+            sats: ComputedVecsFromHeight::forced_import(
+                db,
+                name,
+                source,
+                version + VERSION,
+                indexes,
+                options,
+            )?,
+            bitcoin: ComputedVecsFromHeight::forced_import(
+                db,
+                &format!("{name}_btc"),
+                Source::Compute,
+                version + VERSION,
+                indexes,
+                options,
+            )?,
+            dollars: compute_dollars.then(|| {
+                ComputedVecsFromHeight::forced_import(
+                    db,
+                    &format!("{name}_usd"),
+                    Source::Compute,
+                    version + VERSION,
+                    indexes,
+                    options,
+                )
+                .unwrap()
+            }),
+        })
+    }
+
+    pub fn compute_all<F>(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        mut compute: F,
+    ) -> Result<()>
+    where
+        F: FnMut(&mut EagerVec<Height, Sats>) -> Result<()>,
+    {
+        compute(self.sats.height.as_mut().unwrap())?;
+
+        let height: Option<&StoredVec<Height, Sats>> = None;
+        self.compute_rest(indexes, price, starting_indexes, exit, height)?;
+
+        Ok(())
+    }
+
+    pub fn compute_rest(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        height: Option<&impl CollectableVec<Height, Sats>>,
+    ) -> Result<()> {
+        if let Some(height) = height {
+            self.sats
+                .compute_rest(indexes, starting_indexes, exit, Some(height))?;
+
+            self.bitcoin
+                .compute_all(indexes, starting_indexes, exit, |v| {
+                    v.compute_from_sats(starting_indexes.height, height, exit)
+                })?;
+        } else {
+            let height: Option<&StoredVec<Height, Sats>> = None;
+
+            self.sats
+                .compute_rest(indexes, starting_indexes, exit, height)?;
+
+            self.bitcoin
+                .compute_all(indexes, starting_indexes, exit, |v| {
+                    v.compute_from_sats(
+                        starting_indexes.height,
+                        self.sats.height.as_ref().unwrap(),
+                        exit,
+                    )
+                })?;
+        }
+
+        let height_to_bitcoin = self.bitcoin.height.as_ref().unwrap();
+        let height_to_price_close = &price.as_ref().unwrap().chainindexes_to_price_close.height;
+
+        if let Some(dollars) = self.dollars.as_mut() {
+            dollars.compute_all(indexes, starting_indexes, exit, |v| {
+                v.compute_from_bitcoin(
+                    starting_indexes.height,
+                    height_to_bitcoin,
+                    height_to_price_close,
+                    exit,
+                )
+            })?;
+        }
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        std::iter::empty()
+            .chain(self.sats.iter_any_collectable())
+            .chain(self.bitcoin.iter_any_collectable())
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
+    }
+}

crates/brk_computer/src/grouped/value_from_txindex.rs

@@ -1,34 +1,24 @@
-use std::path::Path;
-
-use brk_core::{Bitcoin, Close, Dollars, Height, Sats, TxIndex, Version};
-use brk_exit::Exit;
+use allocative::Allocative;
+use brk_error::Result;
 use brk_indexer::Indexer;
-use brk_vec::{
-    AnyCollectableVec, BoxedAnyIterableVec, CloneableAnyIterableVec, CollectableVec, Computation,
-    ComputedVecFrom3, Format, LazyVecFrom1, StoredIndex, StoredVec,
+use brk_structs::{Bitcoin, Close, Dollars, Height, Sats, TxIndex, Version};
+use vecdb::{
+    AnyCloneableIterableVec, AnyCollectableVec, CollectableVec, Database, Exit, LazyVecFrom1,
+    LazyVecFrom3, StoredIndex, StoredVec,
 };
 
-use crate::vecs::{Indexes, fetched, indexes};
+use crate::{Indexes, grouped::Source, indexes, price};
 
-use super::{ComputedVecsFromTxindex, StorableVecGeneatorOptions};
+use super::{ComputedVecsFromTxindex, VecBuilderOptions};
 
-#[derive(Clone)]
+#[derive(Clone, Allocative)]
 pub struct ComputedValueVecsFromTxindex {
     pub sats: ComputedVecsFromTxindex<Sats>,
     pub bitcoin_txindex: LazyVecFrom1<TxIndex, Bitcoin, TxIndex, Sats>,
     pub bitcoin: ComputedVecsFromTxindex<Bitcoin>,
     #[allow(clippy::type_complexity)]
     pub dollars_txindex: Option<
-        ComputedVecFrom3<
-            TxIndex,
-            Dollars,
-            TxIndex,
-            Bitcoin,
-            TxIndex,
-            Height,
-            Height,
-            Close<Dollars>,
-        >,
+        LazyVecFrom3<TxIndex, Dollars, TxIndex, Bitcoin, TxIndex, Height, Height, Close<Dollars>>,
     >,
     pub dollars: Option<ComputedVecsFromTxindex<Dollars>>,
 }
@@ -38,81 +28,76 @@ const VERSION: Version = Version::ZERO;
 impl ComputedValueVecsFromTxindex {
     #[allow(clippy::too_many_arguments)]
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         name: &str,
         indexes: &indexes::Vecs,
-        source: Option<BoxedAnyIterableVec<TxIndex, Sats>>,
+        source: Source<TxIndex, Sats>,
         version: Version,
-        computation: Computation,
-        format: Format,
-        fetched: Option<&fetched::Vecs>,
-        options: StorableVecGeneatorOptions,
-    ) -> color_eyre::Result<Self> {
-        let compute_source = source.is_none();
-        let compute_dollars = fetched.is_some();
+        price: Option<&price::Vecs>,
+        options: VecBuilderOptions,
+    ) -> Result<Self> {
+        let compute_dollars = price.is_some();
 
-        let name_in_btc = format!("{name}_in_btc");
-        let name_in_usd = format!("{name}_in_usd");
+        let name_btc = format!("{name}_btc");
+        let name_usd = format!("{name}_usd");
 
         let sats = ComputedVecsFromTxindex::forced_import(
-            path,
+            db,
             name,
-            compute_source,
+            source.clone(),
             version + VERSION,
-            format,
+            indexes,
             options,
         )?;
 
+        let source_vec = source.vec();
+
         let bitcoin_txindex = LazyVecFrom1::init(
-            &name_in_btc,
+            &name_btc,
             version + VERSION,
-            source.map_or_else(|| sats.txindex.as_ref().unwrap().boxed_clone(), |s| s),
+            source_vec.map_or_else(|| sats.txindex.as_ref().unwrap().boxed_clone(), |s| s),
             |txindex: TxIndex, iter| {
                 iter.next_at(txindex.unwrap_to_usize()).map(|(_, value)| {
-                    let sats = value.into_inner();
+                    let sats = value.into_owned();
                     Bitcoin::from(sats)
                 })
             },
         );
 
         let bitcoin = ComputedVecsFromTxindex::forced_import(
-            path,
-            &name_in_btc,
-            false,
+            db,
+            &name_btc,
+            Source::None,
             version + VERSION,
-            format,
+            indexes,
             options,
         )?;
 
-        let dollars_txindex = fetched.map(|fetched| {
-            ComputedVecFrom3::forced_import_or_init_from_3(
-                computation,
-                path,
-                &name_in_usd,
+        let dollars_txindex = price.map(|price| {
+            LazyVecFrom3::init(
+                &name_usd,
                 version + VERSION,
-                format,
                 bitcoin_txindex.boxed_clone(),
                 indexes.txindex_to_height.boxed_clone(),
-                fetched.chainindexes_to_close.height.boxed_clone(),
+                price.chainindexes_to_price_close.height.boxed_clone(),
                 |txindex: TxIndex,
                  txindex_to_btc_iter,
                  txindex_to_height_iter,
-                 height_to_close_iter| {
+                 height_to_price_close_iter| {
                     let txindex = txindex.unwrap_to_usize();
                     txindex_to_btc_iter.next_at(txindex).and_then(|(_, value)| {
-                        let btc = value.into_inner();
+                        let btc = value.into_owned();
                         txindex_to_height_iter
                             .next_at(txindex)
                             .and_then(|(_, value)| {
-                                let height = value.into_inner();
-                                height_to_close_iter
+                                let height = value.into_owned();
+                                height_to_price_close_iter
                                     .next_at(height.unwrap_to_usize())
-                                    .map(|(_, close)| *close.into_inner() * btc)
+                                    .map(|(_, close)| *close.into_owned() * btc)
                             })
                     })
                 },
             )
-            .unwrap()
         });
 
         Ok(Self {
@@ -122,11 +107,11 @@ impl ComputedValueVecsFromTxindex {
             dollars_txindex,
             dollars: compute_dollars.then(|| {
                 ComputedVecsFromTxindex::forced_import(
-                    path,
-                    &name_in_usd,
-                    false,
+                    db,
+                    &name_usd,
+                    Source::None,
                     version + VERSION,
-                    format,
+                    indexes,
                     options,
                 )
                 .unwrap()
@@ -138,11 +123,11 @@ impl ComputedValueVecsFromTxindex {
     //     &mut self,
     //     indexer: &Indexer,
     //     indexes: &indexes::Vecs,
-    //     fetched: Option<&marketprice::Vecs>,
+    //     price: Option<&marketprice::Vecs>,
     //     starting_indexes: &Indexes,
     //     exit: &Exit,
     //     mut compute: F,
-    // ) -> color_eyre::Result<()>
+    // ) -> Result<()>
     // where
     //     F: FnMut(
     //         &mut EagerVec<TxIndex, Sats>,
@@ -180,8 +165,8 @@ impl ComputedValueVecsFromTxindex {
         starting_indexes: &Indexes,
         exit: &Exit,
         txindex: Option<&impl CollectableVec<TxIndex, Sats>>,
-        fetched: Option<&fetched::Vecs>,
-    ) -> color_eyre::Result<()> {
+        price: Option<&price::Vecs>,
+    ) -> Result<()> {
         if let Some(txindex) = txindex {
             self.sats
                 .compute_rest(indexer, indexes, starting_indexes, exit, Some(txindex))?;
@@ -203,12 +188,6 @@ impl ComputedValueVecsFromTxindex {
         if let Some(dollars) = self.dollars.as_mut() {
             let dollars_txindex = self.dollars_txindex.as_mut().unwrap();
 
-            dollars_txindex.compute_if_necessary(
-                starting_indexes.txindex,
-                &indexer.vecs().txindex_to_txid,
-                exit,
-            )?;
-
             dollars.compute_rest_from_bitcoin(
                 indexer,
                 indexes,
@@ -216,25 +195,23 @@ impl ComputedValueVecsFromTxindex {
                 exit,
                 &self.bitcoin,
                 Some(dollars_txindex),
-                fetched.as_ref().unwrap(),
+                price.as_ref().unwrap(),
             )?;
         }
 
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self.sats.vecs(),
-            vec![&self.bitcoin_txindex as &dyn AnyCollectableVec],
-            self.bitcoin.vecs(),
-            self.dollars_txindex
-                .as_ref()
-                .map_or(vec![], |v| vec![v as &dyn AnyCollectableVec]),
-            self.dollars.as_ref().map_or(vec![], |v| v.vecs()),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.bitcoin_txindex as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(self.sats.iter_any_collectable())
+            .chain(self.bitcoin.iter_any_collectable())
+            .chain(
+                self.dollars_txindex
+                    .iter()
+                    .map(|v| v as &dyn AnyCollectableVec),
+            )
+            .chain(self.dollars.iter().flat_map(|v| v.iter_any_collectable()))
     }
 }

crates/brk_computer/src/grouped/value_height.rs

@@ -1,11 +1,13 @@
-use std::path::Path;
+use brk_error::Result;
+use brk_structs::{Bitcoin, Dollars, Height, Sats, Version};
+use vecdb::{AnyCollectableVec, CollectableVec, Database, EagerVec, Exit, Format, StoredVec};
 
-use brk_core::{Bitcoin, Dollars, Height, Result, Sats, Version};
-use brk_exit::Exit;
-use brk_indexer::Indexer;
-use brk_vec::{AnyCollectableVec, CollectableVec, EagerVec, Format, StoredVec};
-
-use crate::vecs::{Indexes, fetched, indexes};
+use crate::{
+    Indexes,
+    grouped::Source,
+    price,
+    traits::{ComputeFromBitcoin, ComputeFromSats},
+};
 
 #[derive(Clone)]
 pub struct ComputedHeightValueVecs {
@@ -18,28 +20,28 @@ const VERSION: Version = Version::ZERO;
 
 impl ComputedHeightValueVecs {
     pub fn forced_import(
-        path: &Path,
+        db: &Database,
         name: &str,
-        compute_source: bool,
+        source: Source<Height, Sats>,
         version: Version,
         format: Format,
         compute_dollars: bool,
-    ) -> color_eyre::Result<Self> {
+    ) -> Result<Self> {
         Ok(Self {
-            sats: compute_source.then(|| {
-                EagerVec::forced_import(path, name, version + VERSION + Version::ZERO, format)
+            sats: source.is_compute().then(|| {
+                EagerVec::forced_import(db, name, version + VERSION + Version::ZERO, format)
                     .unwrap()
             }),
             bitcoin: EagerVec::forced_import(
-                path,
-                &format!("{name}_in_btc"),
+                db,
+                &format!("{name}_btc"),
                 version + VERSION + Version::ZERO,
                 format,
             )?,
             dollars: compute_dollars.then(|| {
                 EagerVec::forced_import(
-                    path,
-                    &format!("{name}_in_usd"),
+                    db,
+                    &format!("{name}_usd"),
                     version + VERSION + Version::ZERO,
                     format,
                 )
@@ -50,43 +52,29 @@ impl ComputedHeightValueVecs {
 
     pub fn compute_all<F>(
         &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
         mut compute: F,
-    ) -> color_eyre::Result<()>
+    ) -> Result<()>
     where
-        F: FnMut(
-            &mut EagerVec<Height, Sats>,
-            &Indexer,
-            &indexes::Vecs,
-            &Indexes,
-            &Exit,
-        ) -> Result<()>,
+        F: FnMut(&mut EagerVec<Height, Sats>) -> Result<()>,
     {
-        compute(
-            self.sats.as_mut().unwrap(),
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-        )?;
+        compute(self.sats.as_mut().unwrap())?;
 
         let height: Option<&StoredVec<Height, Sats>> = None;
-        self.compute_rest(fetched, starting_indexes, exit, height)?;
+        self.compute_rest(price, starting_indexes, exit, height)?;
 
         Ok(())
     }
 
     pub fn compute_rest(
         &mut self,
-        fetched: Option<&fetched::Vecs>,
+        price: Option<&price::Vecs>,
         starting_indexes: &Indexes,
         exit: &Exit,
         height: Option<&impl CollectableVec<Height, Sats>>,
-    ) -> color_eyre::Result<()> {
+    ) -> Result<()> {
         if let Some(height) = height {
             self.bitcoin
                 .compute_from_sats(starting_indexes.height, height, exit)?;
@@ -99,13 +87,13 @@ impl ComputedHeightValueVecs {
         }
 
         let height_to_bitcoin = &self.bitcoin;
-        let height_to_close = &fetched.as_ref().unwrap().chainindexes_to_close.height;
+        let height_to_price_close = &price.as_ref().unwrap().chainindexes_to_price_close.height;
 
         if let Some(dollars) = self.dollars.as_mut() {
             dollars.compute_from_bitcoin(
                 starting_indexes.height,
                 height_to_bitcoin,
-                height_to_close,
+                height_to_price_close,
                 exit,
             )?;
         }
@@ -113,14 +101,10 @@ impl ComputedHeightValueVecs {
         Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            vec![&self.bitcoin as &dyn AnyCollectableVec],
-            self.sats.as_ref().map_or(vec![], |v| vec![v]),
-            self.dollars.as_ref().map_or(vec![], |v| vec![v]),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.bitcoin as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(self.sats.iter().map(|v| v as &dyn AnyCollectableVec))
+            .chain(self.dollars.iter().map(|v| v as &dyn AnyCollectableVec))
     }
 }

crates/brk_computer/src/lib.rs

@@ -1,104 +1,262 @@
 #![doc = include_str!("../README.md")]
-#![doc = "\n## Example\n\n```rust"]
-#![doc = include_str!("../examples/main.rs")]
-#![doc = "```"]
 
-use std::path::{Path, PathBuf};
+use std::path::Path;
 
-use brk_core::Version;
-use brk_exit::Exit;
+use brk_error::Result;
 use brk_fetcher::Fetcher;
 use brk_indexer::Indexer;
-use brk_vec::{AnyCollectableVec, Computation, Format};
-
-mod stores;
-mod utils;
-mod vecs;
-
+use brk_parser::Parser;
+use brk_structs::Version;
 use log::info;
-use stores::Stores;
-use vecs::Vecs;
+use vecdb::{AnyCollectableVec, Exit, Format};
+
+mod blks;
+mod chain;
+mod cointime;
+mod constants;
+mod fetched;
+mod grouped;
+mod indexes;
+mod market;
+mod pools;
+mod price;
+mod stateful;
+mod states;
+mod traits;
+mod utils;
+
+use indexes::Indexes;
+
+pub use pools::*;
+pub use states::PriceToAmount;
+use states::*;
 
 #[derive(Clone)]
 pub struct Computer {
-    path: PathBuf,
-    fetcher: Option<Fetcher>,
-    vecs: Option<Vecs>,
-    stores: Option<Stores>,
-    format: Format,
+    pub chain: chain::Vecs,
+    pub cointime: cointime::Vecs,
+    pub constants: constants::Vecs,
+    pub fetched: Option<fetched::Vecs>,
+    pub indexes: indexes::Vecs,
+    pub market: market::Vecs,
+    pub pools: pools::Vecs,
+    pub blks: blks::Vecs,
+    pub price: Option<price::Vecs>,
+    pub stateful: stateful::Vecs,
 }
 
-const VERSION: Version = Version::ONE;
+const VERSION: Version = Version::new(4);
 
 impl Computer {
-    pub fn new(outputs_dir: &Path, fetcher: Option<Fetcher>, format: Format) -> Self {
-        Self {
-            path: outputs_dir.to_owned(),
-            fetcher,
-            vecs: None,
-            stores: None,
-            format,
-        }
-    }
-
-    pub fn import_vecs(
-        &mut self,
-        indexer: &Indexer,
-        computation: Computation,
-    ) -> color_eyre::Result<()> {
-        self.vecs = Some(Vecs::import(
-            // TODO: Give self.path, join inside import
-            &self.path.join("vecs/computed"),
-            VERSION + Version::ZERO,
-            indexer,
-            self.fetcher.is_some(),
-            computation,
-            self.format,
-        )?);
-        Ok(())
-    }
-
     /// Do NOT import multiple times or things will break !!!
-    /// Clone struct instead
-    pub fn import_stores(&mut self, indexer: &Indexer) -> color_eyre::Result<()> {
-        self.stores = Some(Stores::import(
-            // TODO: Give self.path, join inside import
-            &self.path.join("stores"),
-            VERSION + Version::ZERO,
-            indexer.keyspace(),
-        )?);
-        Ok(())
-    }
-}
+    pub fn forced_import(
+        outputs_path: &Path,
+        indexer: &Indexer,
+        fetcher: Option<Fetcher>,
+    ) -> Result<Self> {
+        info!("Importing computer...");
+
+        let computed_path = outputs_path.join("computed");
+
+        let indexes =
+            indexes::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, indexer)?;
+
+        let fetched = fetcher.map(|fetcher| {
+            fetched::Vecs::forced_import(outputs_path, fetcher, VERSION + Version::ZERO).unwrap()
+        });
+
+        let price = fetched.is_some().then(|| {
+            price::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes).unwrap()
+        });
+
+        Ok(Self {
+            constants: constants::Vecs::forced_import(
+                &computed_path,
+                VERSION + Version::ZERO,
+                &indexes,
+            )?,
+            market: market::Vecs::forced_import(&computed_path, VERSION + Version::ZERO, &indexes)?,
+            stateful: stateful::Vecs::forced_import(
+                &computed_path,
+                VERSION + Version::ZERO,
+                Format::Compressed,
+                &indexes,
+                price.as_ref(),
+            )?,
+            chain: chain::Vecs::forced_import(
+                &computed_path,
+                VERSION + Version::ZERO,
+                indexer,
+                &indexes,
+                price.as_ref(),
+            )?,
+            blks: blks::Vecs::forced_import(&computed_path, VERSION + Version::ZERO)?,
+            pools: pools::Vecs::forced_import(
+                &computed_path,
+                VERSION + Version::ZERO,
+                &indexes,
+                price.as_ref(),
+            )?,
+            cointime: cointime::Vecs::forced_import(
+                &computed_path,
+                VERSION + Version::ZERO,
+                &indexes,
+                price.as_ref(),
+            )?,
+            indexes,
+            fetched,
+            price,
+        })
+    }
 
-impl Computer {
     pub fn compute(
         &mut self,
-        indexer: &mut Indexer,
+        indexer: &Indexer,
         starting_indexes: brk_indexer::Indexes,
+        parser: &Parser,
         exit: &Exit,
-    ) -> color_eyre::Result<()> {
-        info!("Computing...");
-        self.vecs
-            .as_mut()
-            .unwrap()
-            .compute(indexer, starting_indexes, self.fetcher.as_mut(), exit)
+    ) -> Result<()> {
+        info!("Computing indexes...");
+        let mut starting_indexes = self.indexes.compute(indexer, starting_indexes, exit)?;
+
+        if let Some(fetched) = self.fetched.as_mut() {
+            info!("Computing fetched...");
+            fetched.compute(indexer, &self.indexes, &starting_indexes, exit)?;
+
+            info!("Computing prices...");
+            self.price.as_mut().unwrap().compute(
+                &self.indexes,
+                &starting_indexes,
+                fetched,
+                exit,
+            )?;
+        }
+
+        info!("Computing BLKs metadata...");
+        self.blks
+            .compute(indexer, &self.indexes, &starting_indexes, parser, exit)?;
+
+        std::thread::scope(|scope| -> Result<()> {
+            let constants = scope.spawn(|| -> Result<()> {
+                info!("Computing constants...");
+                self.constants
+                    .compute(&self.indexes, &starting_indexes, exit)?;
+                Ok(())
+            });
+
+            // let blks = scope.spawn(|| -> Result<()> {
+            //     info!("Computing blks...");
+            //     self.blks
+            //         .compute(indexer, &self.indexes, &starting_indexes, parser, exit)?;
+            //     Ok(())
+            // });
+
+            let chain = scope.spawn(|| -> Result<()> {
+                info!("Computing chain...");
+                self.chain.compute(
+                    indexer,
+                    &self.indexes,
+                    &starting_indexes,
+                    self.price.as_ref(),
+                    exit,
+                )?;
+                Ok(())
+            });
+
+            if let Some(price) = self.price.as_ref() {
+                info!("Computing market...");
+                self.market.compute(price, &starting_indexes, exit)?;
+            }
+
+            constants.join().unwrap()?;
+            // blks.join().unwrap()?;
+            chain.join().unwrap()?;
+            Ok(())
+        })?;
+
+        self.pools.compute(
+            indexer,
+            &self.indexes,
+            &starting_indexes,
+            &self.chain,
+            self.price.as_ref(),
+            exit,
+        )?;
+
+        info!("Computing stateful...");
+        self.stateful.compute(
+            indexer,
+            &self.indexes,
+            &self.chain,
+            self.price.as_ref(),
+            &mut starting_indexes,
+            exit,
+        )?;
+
+        info!("Computing cointime...");
+        self.cointime.compute(
+            &self.indexes,
+            &starting_indexes,
+            self.price.as_ref(),
+            &self.chain,
+            &self.stateful,
+            exit,
+        )?;
+
+        Ok(())
     }
 
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        // pub fn vecs(&self) -> &Vecs {
-        self.vecs.as_ref().unwrap().vecs()
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(self.fetched.iter().flat_map(|v| v.iter_any_collectable()));
+
+        iter = Box::new(iter.chain(self.chain.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.cointime.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.constants.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.market.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.pools.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.blks.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.price.iter().flat_map(|v| v.iter_any_collectable())));
+        iter = Box::new(iter.chain(self.stateful.iter_any_collectable()));
+
+        iter
     }
 
-    // pub fn mut_vecs(&mut self) -> &mut Vecs {
-    //     self.vecs.as_mut().unwrap()
-    // }
-
-    pub fn stores(&self) -> &Stores {
-        self.stores.as_ref().unwrap()
-    }
-
-    pub fn mut_stores(&mut self) -> &mut Stores {
-        self.stores.as_mut().unwrap()
+    pub fn static_clone(&self) -> &'static Self {
+        Box::leak(Box::new(self.clone()))
     }
 }
+
+// pub fn generate_allocation_files(monitored: &pools::Vecs) -> Result<()> {
+//     info!("Generating Allocative files...");
+
+//     let mut flamegraph = allocative::FlameGraphBuilder::default();
+//     flamegraph.visit_root(monitored);
+//     let output = flamegraph.finish();
+
+//     let folder = format!(
+//         "at-{}",
+//         jiff::Timestamp::now().strftime("%Y-%m-%d_%Hh%Mm%Ss"),
+//     );
+
+//     let path = std::path::PathBuf::from(&format!("./target/flamegraph/{folder}"));
+//     std::fs::create_dir_all(&path)?;
+
+//     // fs::write(path.join("flamegraph.src"), &output.flamegraph())?;
+
+//     let mut fg_svg = Vec::new();
+//     inferno::flamegraph::from_reader(
+//         &mut inferno::flamegraph::Options::default(),
+//         output.flamegraph().write().as_bytes(),
+//         &mut fg_svg,
+//     )?;
+
+//     std::fs::write(path.join("flamegraph.svg"), &fg_svg)?;
+
+//     std::fs::write(path.join("warnings.txt"), output.warnings())?;
+
+//     info!("Successfully generated Allocative files");
+
+//     Ok(())
+// }

crates/brk_computer/src/pools/mod.rs

@@ -0,0 +1,227 @@
+use std::{collections::BTreeMap, path::Path};
+
+use allocative::Allocative;
+use brk_error::Result;
+use brk_indexer::Indexer;
+use brk_store::AnyStore;
+use brk_structs::{AddressBytes, Height, OutputIndex, OutputType, PoolId, Pools, pools};
+use rayon::prelude::*;
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, Exit, GenericStoredVec,
+    PAGE_SIZE, RawVec, StoredIndex, VecIterator, Version,
+};
+
+mod vecs;
+
+use crate::{
+    chain,
+    indexes::{self, Indexes},
+    price,
+};
+
+#[derive(Clone, Allocative)]
+pub struct Vecs {
+    db: Database,
+    pools: &'static Pools,
+    height_to_pool: RawVec<Height, PoolId>,
+
+    vecs: BTreeMap<PoolId, vecs::Vecs>,
+}
+
+impl Vecs {
+    pub fn forced_import(
+        parent_path: &Path,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+    ) -> Result<Self> {
+        let db = Database::open(&parent_path.join("pools"))?;
+        db.set_min_len(PAGE_SIZE * 1_000_000)?;
+        let pools = pools();
+
+        let version = parent_version + Version::new(3) + Version::new(pools.len() as u64);
+
+        let this = Self {
+            height_to_pool: RawVec::forced_import(&db, "pool", version + Version::ZERO)?,
+            vecs: pools
+                .iter()
+                .map(|pool| {
+                    vecs::Vecs::forced_import(
+                        &db,
+                        pool.id,
+                        pools,
+                        version + Version::ZERO,
+                        indexes,
+                        price,
+                    )
+                    .map(|vecs| (pool.id, vecs))
+                })
+                .collect::<Result<BTreeMap<_, _>>>()?,
+            pools,
+            db,
+        };
+
+        this.db.retain_regions(
+            this.iter_any_collectable()
+                .flat_map(|v| v.region_names())
+                .collect(),
+        )?;
+
+        Ok(this)
+    }
+
+    pub fn compute(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_(indexer, indexes, starting_indexes, chain, price, exit)?;
+        self.db.flush_then_punch()?;
+        Ok(())
+    }
+
+    fn compute_(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.compute_height_to_pool(indexer, indexes, starting_indexes, exit)?;
+
+        self.vecs.par_iter_mut().try_for_each(|(_, vecs)| {
+            vecs.compute(
+                indexes,
+                starting_indexes,
+                &self.height_to_pool,
+                chain,
+                price,
+                exit,
+            )
+        })?;
+
+        Ok(())
+    }
+
+    fn compute_height_to_pool(
+        &mut self,
+        indexer: &Indexer,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.height_to_pool.validate_computed_version_or_reset(
+            self.height_to_pool.version() + indexer.stores.height_to_coinbase_tag.version(),
+        )?;
+
+        let mut height_to_first_txindex_iter = indexer.vecs.height_to_first_txindex.iter();
+        let mut txindex_to_first_outputindex_iter =
+            indexer.vecs.txindex_to_first_outputindex.iter();
+        let mut txindex_to_output_count_iter = indexes.txindex_to_output_count.iter();
+        let mut outputindex_to_outputtype_iter = indexer.vecs.outputindex_to_outputtype.iter();
+        let mut outputindex_to_typeindex_iter = indexer.vecs.outputindex_to_typeindex.iter();
+        let mut p2pk65addressindex_to_p2pk65bytes_iter =
+            indexer.vecs.p2pk65addressindex_to_p2pk65bytes.iter();
+        let mut p2pk33addressindex_to_p2pk33bytes_iter =
+            indexer.vecs.p2pk33addressindex_to_p2pk33bytes.iter();
+        let mut p2pkhaddressindex_to_p2pkhbytes_iter =
+            indexer.vecs.p2pkhaddressindex_to_p2pkhbytes.iter();
+        let mut p2shaddressindex_to_p2shbytes_iter =
+            indexer.vecs.p2shaddressindex_to_p2shbytes.iter();
+        let mut p2wpkhaddressindex_to_p2wpkhbytes_iter =
+            indexer.vecs.p2wpkhaddressindex_to_p2wpkhbytes.iter();
+        let mut p2wshaddressindex_to_p2wshbytes_iter =
+            indexer.vecs.p2wshaddressindex_to_p2wshbytes.iter();
+        let mut p2traddressindex_to_p2trbytes_iter =
+            indexer.vecs.p2traddressindex_to_p2trbytes.iter();
+        let mut p2aaddressindex_to_p2abytes_iter = indexer.vecs.p2aaddressindex_to_p2abytes.iter();
+
+        let unknown = self.pools.get_unknown();
+
+        let min = starting_indexes
+            .height
+            .unwrap_to_usize()
+            .min(self.height_to_pool.len());
+
+        indexer
+            .stores
+            .height_to_coinbase_tag
+            .iter()
+            .skip(min)
+            .try_for_each(|(height, coinbase_tag)| -> Result<()> {
+                let txindex = height_to_first_txindex_iter.unwrap_get_inner(height);
+                let outputindex = txindex_to_first_outputindex_iter.unwrap_get_inner(txindex);
+                let outputcount = txindex_to_output_count_iter.unwrap_get_inner(txindex);
+
+                let pool = (*outputindex..(*outputindex + *outputcount))
+                    .map(OutputIndex::from)
+                    .find_map(|outputindex| {
+                        let outputtype =
+                            outputindex_to_outputtype_iter.unwrap_get_inner(outputindex);
+                        let typeindex = outputindex_to_typeindex_iter.unwrap_get_inner(outputindex);
+
+                        let address = match outputtype {
+                            OutputType::P2PK65 => Some(AddressBytes::from(
+                                p2pk65addressindex_to_p2pk65bytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2PK33 => Some(AddressBytes::from(
+                                p2pk33addressindex_to_p2pk33bytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2PKH => Some(AddressBytes::from(
+                                p2pkhaddressindex_to_p2pkhbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2SH => Some(AddressBytes::from(
+                                p2shaddressindex_to_p2shbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2WPKH => Some(AddressBytes::from(
+                                p2wpkhaddressindex_to_p2wpkhbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2WSH => Some(AddressBytes::from(
+                                p2wshaddressindex_to_p2wshbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2TR => Some(AddressBytes::from(
+                                p2traddressindex_to_p2trbytes_iter
+                                    .unwrap_get_inner(typeindex.into()),
+                            )),
+                            OutputType::P2A => Some(AddressBytes::from(
+                                p2aaddressindex_to_p2abytes_iter.unwrap_get_inner(typeindex.into()),
+                            )),
+                            _ => None,
+                        };
+
+                        address
+                            .and_then(|address| self.pools.find_from_address(&address.to_string()))
+                    })
+                    .or_else(|| self.pools.find_from_coinbase_tag(&coinbase_tag))
+                    .unwrap_or(unknown);
+
+                self.height_to_pool.push_if_needed(height, pool.id)?;
+                Ok(())
+            })?;
+
+        self.height_to_pool.safe_flush(exit)?;
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        [&self.height_to_pool as &dyn AnyCollectableVec]
+            .into_iter()
+            .chain(
+                self.vecs
+                    .iter()
+                    .flat_map(|(_, vecs)| vecs.iter_any_collectable()),
+            )
+    }
+}

crates/brk_computer/src/pools/vecs.rs

@@ -0,0 +1,397 @@
+use allocative::Allocative;
+use brk_error::Result;
+use brk_structs::{Height, PoolId, Pools, Sats, StoredF32, StoredU16, StoredU32};
+use vecdb::{AnyCollectableVec, AnyIterableVec, Database, Exit, StoredIndex, VecIterator, Version};
+
+use crate::{
+    chain,
+    grouped::{
+        ComputedValueVecsFromHeight, ComputedVecsFromDateIndex, ComputedVecsFromHeight, Source,
+        VecBuilderOptions,
+    },
+    indexes::{self, Indexes},
+    price,
+};
+
+#[derive(Clone, Allocative)]
+pub struct Vecs {
+    id: PoolId,
+
+    indexes_to_blocks_mined: ComputedVecsFromHeight<StoredU32>,
+    indexes_to_1w_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_1m_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_1y_blocks_mined: ComputedVecsFromDateIndex<StoredU32>,
+    indexes_to_subsidy: ComputedValueVecsFromHeight,
+    indexes_to_fee: ComputedValueVecsFromHeight,
+    indexes_to_coinbase: ComputedValueVecsFromHeight,
+    indexes_to_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1d_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1w_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1m_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_1y_dominance: ComputedVecsFromDateIndex<StoredF32>,
+    indexes_to_days_since_block: ComputedVecsFromDateIndex<StoredU16>,
+}
+
+impl Vecs {
+    pub fn forced_import(
+        db: &Database,
+        id: PoolId,
+        pools: &Pools,
+        parent_version: Version,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+    ) -> Result<Self> {
+        let pool = pools.get(id);
+        let name = pool.serialized_id();
+        let suffix = |s: &str| format!("{name}_{s}");
+        let compute_dollars = price.is_some();
+        let version = parent_version + Version::ZERO;
+
+        Ok(Self {
+            id,
+            indexes_to_blocks_mined: ComputedVecsFromHeight::forced_import(
+                db,
+                &suffix("blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+            )?,
+            indexes_to_1w_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1w_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1m_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1m_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1y_blocks_mined: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1y_blocks_mined"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_subsidy: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("subsidy"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_fee: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("fee"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_coinbase: ComputedValueVecsFromHeight::forced_import(
+                db,
+                &suffix("coinbase"),
+                Source::Compute,
+                version + Version::ZERO,
+                VecBuilderOptions::default().add_sum().add_cumulative(),
+                compute_dollars,
+                indexes,
+            )?,
+            indexes_to_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1d_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1d_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1w_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1w_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1m_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1m_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_1y_dominance: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("1y_dominance"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            indexes_to_days_since_block: ComputedVecsFromDateIndex::forced_import(
+                db,
+                &suffix("days_since_block"),
+                Source::Compute,
+                version + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+        })
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn compute(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        height_to_pool: &impl AnyIterableVec<Height, PoolId>,
+        chain: &chain::Vecs,
+        price: Option<&price::Vecs>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.indexes_to_blocks_mined
+            .compute_all(indexes, starting_indexes, exit, |vec| {
+                vec.compute_transform(
+                    starting_indexes.height,
+                    height_to_pool,
+                    |(h, id, ..)| {
+                        (
+                            h,
+                            if id == self.id {
+                                StoredU32::ONE
+                            } else {
+                                StoredU32::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1w_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    7,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1m_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    30,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1y_blocks_mined
+            .compute_all(starting_indexes, exit, |v| {
+                v.compute_sum(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    365,
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        let height_to_blocks_mined = self.indexes_to_blocks_mined.height.as_ref().unwrap();
+
+        self.indexes_to_subsidy
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_subsidy.sats.height.as_ref().unwrap(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_fee
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_fee.sats.height.unwrap_sum(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_coinbase
+            .compute_all(indexes, price, starting_indexes, exit, |vec| {
+                vec.compute_transform2(
+                    starting_indexes.height,
+                    height_to_blocks_mined,
+                    chain.indexes_to_coinbase.sats.height.as_ref().unwrap(),
+                    |(h, mined, sats, ..)| {
+                        (
+                            h,
+                            if mined == StoredU32::ONE {
+                                sats
+                            } else {
+                                Sats::ZERO
+                            },
+                        )
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_cumulative(),
+                    chain.indexes_to_block_count.dateindex.unwrap_cumulative(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1d_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    chain.indexes_to_block_count.dateindex.unwrap_sum(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1w_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1w_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1w_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1m_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1m_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1m_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_1y_dominance
+            .compute_all(starting_indexes, exit, |vec| {
+                vec.compute_percentage(
+                    starting_indexes.dateindex,
+                    self.indexes_to_1y_blocks_mined.dateindex.as_ref().unwrap(),
+                    chain.indexes_to_1y_block_count.dateindex.as_ref().unwrap(),
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        self.indexes_to_days_since_block
+            .compute_all(starting_indexes, exit, |v| {
+                let mut prev = None;
+                v.compute_transform2(
+                    starting_indexes.dateindex,
+                    self.indexes_to_blocks_mined.dateindex.unwrap_sum(),
+                    self.indexes_to_blocks_mined.dateindex.unwrap_cumulative(),
+                    |(i, sum, cumulative, slf)| {
+                        if prev.is_none() {
+                            let i = i.unwrap_to_usize();
+                            prev.replace(if i > 0 {
+                                slf.into_iter().unwrap_get_inner_(i - 1)
+                            } else {
+                                StoredU16::ZERO
+                            });
+                        }
+                        let days = if !cumulative.is_zero() && sum.is_zero() {
+                            prev.unwrap() + StoredU16::ONE
+                        } else {
+                            StoredU16::ZERO
+                        };
+                        prev.replace(days);
+                        (i, days)
+                    },
+                    exit,
+                )?;
+                Ok(())
+            })?;
+
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        let mut iter: Box<dyn Iterator<Item = &dyn AnyCollectableVec>> =
+            Box::new(std::iter::empty());
+
+        iter = Box::new(iter.chain(self.indexes_to_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1w_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1m_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1y_blocks_mined.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_subsidy.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_fee.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_coinbase.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1d_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1w_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1m_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_1y_dominance.iter_any_collectable()));
+        iter = Box::new(iter.chain(self.indexes_to_days_since_block.iter_any_collectable()));
+
+        iter
+    }
+}

crates/brk_computer/src/stateful/address_cohort.rs

@@ -0,0 +1,244 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_structs::{Bitcoin, DateIndex, Dollars, Height, StoredU64, Version};
+use vecdb::{
+    AnyCollectableVec, AnyIterableVec, AnyStoredVec, AnyVec, Database, EagerVec, Exit, Format,
+    GenericStoredVec, VecIterator,
+};
+
+use crate::{
+    Indexes,
+    grouped::{ComputedVecsFromHeight, Source, VecBuilderOptions},
+    indexes, price,
+    stateful::{
+        common,
+        r#trait::{CohortVecs, DynCohortVecs},
+    },
+    states::AddressCohortState,
+};
+
+const VERSION: Version = Version::ZERO;
+
+#[derive(Clone)]
+pub struct Vecs {
+    starting_height: Option<Height>,
+
+    pub state: Option<AddressCohortState>,
+
+    pub inner: common::Vecs,
+
+    pub height_to_addr_count: EagerVec<Height, StoredU64>,
+    pub indexes_to_addr_count: ComputedVecsFromHeight<StoredU64>,
+}
+
+impl Vecs {
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        cohort_name: Option<&str>,
+        format: Format,
+        version: Version,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        states_path: Option<&Path>,
+        compute_rel_to_all: bool,
+    ) -> Result<Self> {
+        let compute_dollars = price.is_some();
+
+        let suffix = |s: &str| cohort_name.map_or(s.to_string(), |name| format!("{name}_{s}"));
+
+        Ok(Self {
+            starting_height: None,
+            state: states_path.map(|states_path| {
+                AddressCohortState::new(
+                    states_path,
+                    cohort_name.unwrap_or_default(),
+                    compute_dollars,
+                )
+            }),
+            height_to_addr_count: EagerVec::forced_import(
+                db,
+                &suffix("addr_count"),
+                version + VERSION + Version::ZERO,
+                format,
+            )?,
+            indexes_to_addr_count: ComputedVecsFromHeight::forced_import(
+                db,
+                &suffix("addr_count"),
+                Source::None,
+                version + VERSION + Version::ZERO,
+                indexes,
+                VecBuilderOptions::default().add_last(),
+            )?,
+            inner: common::Vecs::forced_import(
+                db,
+                cohort_name,
+                format,
+                version,
+                indexes,
+                price,
+                false,
+                compute_rel_to_all,
+                false,
+            )?,
+        })
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        self.inner
+            .iter_any_collectable()
+            .chain([&self.height_to_addr_count as &dyn AnyCollectableVec])
+            .chain(self.indexes_to_addr_count.iter_any_collectable())
+    }
+}
+
+impl DynCohortVecs for Vecs {
+    fn min_height_vecs_len(&self) -> usize {
+        std::cmp::min(
+            self.height_to_addr_count.len(),
+            self.inner.min_height_vecs_len(),
+        )
+    }
+
+    fn reset_state_starting_height(&mut self) {
+        self.starting_height = Some(Height::ZERO);
+    }
+
+    fn import_state(&mut self, starting_height: Height) -> Result<Height> {
+        let starting_height = self
+            .inner
+            .import_state(starting_height, &mut self.state.as_mut().unwrap().inner)?;
+
+        self.starting_height = Some(starting_height);
+
+        if let Some(prev_height) = starting_height.decremented() {
+            self.state.as_mut().unwrap().addr_count = *self
+                .height_to_addr_count
+                .into_iter()
+                .unwrap_get_inner(prev_height);
+        }
+
+        Ok(starting_height)
+    }
+
+    fn validate_computed_versions(&mut self, base_version: Version) -> Result<()> {
+        self.height_to_addr_count
+            .validate_computed_version_or_reset(
+                base_version + self.height_to_addr_count.inner_version(),
+            )?;
+
+        self.inner.validate_computed_versions(base_version)
+    }
+
+    fn forced_pushed_at(&mut self, height: Height, exit: &Exit) -> Result<()> {
+        if self.starting_height.unwrap() > height {
+            return Ok(());
+        }
+
+        self.height_to_addr_count.forced_push_at(
+            height,
+            self.state.as_ref().unwrap().addr_count.into(),
+            exit,
+        )?;
+
+        self.inner
+            .forced_pushed_at(height, exit, &self.state.as_ref().unwrap().inner)
+    }
+
+    fn compute_then_force_push_unrealized_states(
+        &mut self,
+        height: Height,
+        height_price: Option<Dollars>,
+        dateindex: Option<DateIndex>,
+        date_price: Option<Option<Dollars>>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.inner.compute_then_force_push_unrealized_states(
+            height,
+            height_price,
+            dateindex,
+            date_price,
+            exit,
+            &self.state.as_ref().unwrap().inner,
+        )
+    }
+
+    fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
+        self.height_to_addr_count.safe_flush(exit)?;
+
+        self.inner
+            .safe_flush_stateful_vecs(height, exit, &mut self.state.as_mut().unwrap().inner)
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    fn compute_rest_part1(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.indexes_to_addr_count.compute_rest(
+            indexes,
+            starting_indexes,
+            exit,
+            Some(&self.height_to_addr_count),
+        )?;
+
+        self.inner
+            .compute_rest_part1(indexes, price, starting_indexes, exit)
+    }
+}
+
+impl CohortVecs for Vecs {
+    fn compute_from_stateful(
+        &mut self,
+        starting_indexes: &Indexes,
+        others: &[&Self],
+        exit: &Exit,
+    ) -> Result<()> {
+        self.height_to_addr_count.compute_sum_of_others(
+            starting_indexes.height,
+            others
+                .iter()
+                .map(|v| &v.height_to_addr_count)
+                .collect::<Vec<_>>()
+                .as_slice(),
+            exit,
+        )?;
+        self.inner.compute_from_stateful(
+            starting_indexes,
+            &others.iter().map(|v| &v.inner).collect::<Vec<_>>(),
+            exit,
+        )
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    fn compute_rest_part2(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
+        dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+        height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.inner.compute_rest_part2(
+            indexes,
+            price,
+            starting_indexes,
+            height_to_supply,
+            dateindex_to_supply,
+            height_to_market_cap,
+            dateindex_to_market_cap,
+            height_to_realized_cap,
+            dateindex_to_realized_cap,
+            exit,
+        )
+    }
+}

crates/brk_computer/src/stateful/address_cohorts.rs

@@ -0,0 +1,547 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_structs::{
+    AddressGroups, Bitcoin, ByAmountRange, ByGreatEqualAmount, ByLowerThanAmount, DateIndex,
+    Dollars, GroupFilter, Height, Version,
+};
+use derive_deref::{Deref, DerefMut};
+use vecdb::{AnyIterableVec, Database, Exit, Format};
+
+use crate::{
+    Indexes, indexes, price,
+    stateful::{
+        address_cohort,
+        r#trait::{CohortVecs, DynCohortVecs},
+    },
+};
+
+const VERSION: Version = Version::new(0);
+
+#[derive(Clone, Deref, DerefMut)]
+pub struct Vecs(AddressGroups<(GroupFilter, address_cohort::Vecs)>);
+
+impl Vecs {
+    pub fn forced_import(
+        db: &Database,
+        version: Version,
+        format: Format,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        states_path: &Path,
+    ) -> Result<Self> {
+        Ok(Self(
+            AddressGroups {
+                amount_range: ByAmountRange {
+                    _0sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_with_0sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _1sat_to_10sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_1sat_under_10sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _10sats_to_100sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_10sats_under_100sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _100sats_to_1k_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_100sats_under_1k_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _1k_sats_to_10k_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_1k_sats_under_10k_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _10k_sats_to_100k_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_10k_sats_under_100k_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _100k_sats_to_1m_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_100k_sats_under_1m_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _1m_sats_to_10m_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_1m_sats_under_10m_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _10m_sats_to_1btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_10m_sats_under_1btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _1btc_to_10btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_1btc_under_10btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _10btc_to_100btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_10btc_under_100btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _100btc_to_1k_btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_100btc_under_1k_btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _1k_btc_to_10k_btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_1k_btc_under_10k_btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _10k_btc_to_100k_btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_10k_btc_under_100k_btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                    _100k_btc_or_more: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_100k_btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        Some(states_path),
+                        true,
+                    )?,
+                },
+                lt_amount: ByLowerThanAmount {
+                    _10sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_10sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _100sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_100sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _1k_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_1k_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _10k_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_10k_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _100k_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_100k_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _1m_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_1m_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _10m_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_10m_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _1btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_1btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _10btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_10btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _100btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_100btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _1k_btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_1k_btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _10k_btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_10k_btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _100k_btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_under_100k_btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                },
+                ge_amount: ByGreatEqualAmount {
+                    _1sat: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_1sat"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _10sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_10sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _100sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_100sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _1k_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_1k_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _10k_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_10k_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _100k_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_100k_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _1m_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_1m_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _10m_sats: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_10m_sats"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _1btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_1btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _10btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_10btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _100btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_100btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _1k_btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_1k_btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                    _10k_btc: address_cohort::Vecs::forced_import(
+                        db,
+                        Some("addrs_above_10k_btc"),
+                        format,
+                        version + VERSION + Version::ZERO,
+                        indexes,
+                        price,
+                        None,
+                        true,
+                    )?,
+                },
+            }
+            .into(),
+        ))
+    }
+
+    pub fn compute_overlapping_vecs(
+        &mut self,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        let by_size_range = &self.0.amount_range;
+
+        [
+            self.0
+                .ge_amount
+                .iter_mut()
+                .map(|(filter, vecs)| {
+                    (
+                        vecs,
+                        by_size_range
+                            .iter()
+                            .filter(|(other, _)| filter.includes(other))
+                            .map(|(_, v)| v)
+                            .collect::<Vec<_>>(),
+                    )
+                })
+                .collect::<Vec<_>>(),
+            self.0
+                .lt_amount
+                .iter_mut()
+                .map(|(filter, vecs)| {
+                    (
+                        vecs,
+                        by_size_range
+                            .iter()
+                            .filter(|(other, _)| filter.includes(other))
+                            .map(|(_, v)| v)
+                            .collect::<Vec<_>>(),
+                    )
+                })
+                .collect::<Vec<_>>(),
+        ]
+        .into_iter()
+        .flatten()
+        .try_for_each(|(vecs, stateful)| {
+            vecs.compute_from_stateful(starting_indexes, &stateful, exit)
+        })
+    }
+
+    pub fn compute_rest_part1(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.iter_mut()
+            .into_iter()
+            .try_for_each(|(_, v)| v.compute_rest_part1(indexes, price, starting_indexes, exit))
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn compute_rest_part2(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
+        dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+        height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.0.iter_mut().try_for_each(|(_, v)| {
+            v.compute_rest_part2(
+                indexes,
+                price,
+                starting_indexes,
+                height_to_supply,
+                dateindex_to_supply,
+                height_to_market_cap,
+                dateindex_to_market_cap,
+                height_to_realized_cap,
+                dateindex_to_realized_cap,
+                exit,
+            )
+        })
+    }
+
+    pub fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
+        self.iter_separate_mut()
+            .into_iter()
+            .try_for_each(|(_, v)| v.safe_flush_stateful_vecs(height, exit))
+    }
+}

crates/brk_computer/src/stateful/addresstype/addresscount.rs

@@ -0,0 +1,47 @@
+use brk_structs::{ByAddressType, Height};
+use derive_deref::{Deref, DerefMut};
+use vecdb::VecIterator;
+
+use super::AddressTypeToHeightToAddressCount;
+
+#[derive(Debug, Default, Deref, DerefMut)]
+pub struct AddressTypeToAddressCount(ByAddressType<u64>);
+
+impl From<(&AddressTypeToHeightToAddressCount, Height)> for AddressTypeToAddressCount {
+    fn from((groups, starting_height): (&AddressTypeToHeightToAddressCount, Height)) -> Self {
+        if let Some(prev_height) = starting_height.decremented() {
+            Self(ByAddressType {
+                p2pk65: groups
+                    .p2pk65
+                    .into_iter()
+                    .unwrap_get_inner(prev_height)
+                    .into(),
+                p2pk33: groups
+                    .p2pk33
+                    .into_iter()
+                    .unwrap_get_inner(prev_height)
+                    .into(),
+                p2pkh: groups
+                    .p2pkh
+                    .into_iter()
+                    .unwrap_get_inner(prev_height)
+                    .into(),
+                p2sh: groups.p2sh.into_iter().unwrap_get_inner(prev_height).into(),
+                p2wpkh: groups
+                    .p2wpkh
+                    .into_iter()
+                    .unwrap_get_inner(prev_height)
+                    .into(),
+                p2wsh: groups
+                    .p2wsh
+                    .into_iter()
+                    .unwrap_get_inner(prev_height)
+                    .into(),
+                p2tr: groups.p2tr.into_iter().unwrap_get_inner(prev_height).into(),
+                p2a: groups.p2a.into_iter().unwrap_get_inner(prev_height).into(),
+            })
+        } else {
+            Default::default()
+        }
+    }
+}

crates/brk_computer/src/stateful/addresstype/height_to_addresscount.rs

@@ -0,0 +1,43 @@
+use brk_error::Result;
+use brk_structs::{ByAddressType, Height, StoredU64};
+use derive_deref::{Deref, DerefMut};
+use vecdb::{EagerVec, Exit, GenericStoredVec};
+
+use super::AddressTypeToAddressCount;
+
+#[derive(Debug, Clone, Deref, DerefMut)]
+pub struct AddressTypeToHeightToAddressCount(ByAddressType<EagerVec<Height, StoredU64>>);
+
+impl From<ByAddressType<EagerVec<Height, StoredU64>>> for AddressTypeToHeightToAddressCount {
+    fn from(value: ByAddressType<EagerVec<Height, StoredU64>>) -> Self {
+        Self(value)
+    }
+}
+
+impl AddressTypeToHeightToAddressCount {
+    pub fn forced_push_at(
+        &mut self,
+        height: Height,
+        addresstype_to_usize: &AddressTypeToAddressCount,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.p2pk65
+            .forced_push_at(height, addresstype_to_usize.p2pk65.into(), exit)?;
+        self.p2pk33
+            .forced_push_at(height, addresstype_to_usize.p2pk33.into(), exit)?;
+        self.p2pkh
+            .forced_push_at(height, addresstype_to_usize.p2pkh.into(), exit)?;
+        self.p2sh
+            .forced_push_at(height, addresstype_to_usize.p2sh.into(), exit)?;
+        self.p2wpkh
+            .forced_push_at(height, addresstype_to_usize.p2wpkh.into(), exit)?;
+        self.p2wsh
+            .forced_push_at(height, addresstype_to_usize.p2wsh.into(), exit)?;
+        self.p2tr
+            .forced_push_at(height, addresstype_to_usize.p2tr.into(), exit)?;
+        self.p2a
+            .forced_push_at(height, addresstype_to_usize.p2a.into(), exit)?;
+
+        Ok(())
+    }
+}

crates/brk_computer/src/stateful/addresstype/height_to_vec.rs

@@ -0,0 +1,9 @@
+use std::collections::BTreeMap;
+
+use brk_structs::Height;
+use derive_deref::{Deref, DerefMut};
+
+use crate::stateful::AddressTypeToVec;
+
+#[derive(Debug, Default, Deref, DerefMut)]
+pub struct HeightToAddressTypeToVec<T>(pub BTreeMap<Height, AddressTypeToVec<T>>);

crates/brk_computer/src/stateful/addresstype/indexes_to_addresscount.rs

@@ -0,0 +1,83 @@
+use brk_error::Result;
+use brk_structs::{ByAddressType, StoredU64};
+use derive_deref::{Deref, DerefMut};
+use vecdb::{AnyCollectableVec, Exit};
+
+use crate::{Indexes, grouped::ComputedVecsFromHeight, indexes};
+
+use super::AddressTypeToHeightToAddressCount;
+
+#[derive(Clone, Deref, DerefMut)]
+pub struct AddressTypeToIndexesToAddressCount(ByAddressType<ComputedVecsFromHeight<StoredU64>>);
+
+impl From<ByAddressType<ComputedVecsFromHeight<StoredU64>>> for AddressTypeToIndexesToAddressCount {
+    fn from(value: ByAddressType<ComputedVecsFromHeight<StoredU64>>) -> Self {
+        Self(value)
+    }
+}
+
+impl AddressTypeToIndexesToAddressCount {
+    pub fn compute(
+        &mut self,
+        indexes: &indexes::Vecs,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+        addresstype_to_height_to_addresscount: &AddressTypeToHeightToAddressCount,
+    ) -> Result<()> {
+        self.p2pk65.compute_rest(
+            indexes,
+            starting_indexes,
+            exit,
+            Some(&addresstype_to_height_to_addresscount.p2pk65),
+        )?;
+        self.p2pk33.compute_rest(
+            indexes,
+            starting_indexes,
+            exit,
+            Some(&addresstype_to_height_to_addresscount.p2pk33),
+        )?;
+        self.p2pkh.compute_rest(
+            indexes,
+            starting_indexes,
+            exit,
+            Some(&addresstype_to_height_to_addresscount.p2pkh),
+        )?;
+        self.p2sh.compute_rest(
+            indexes,
+            starting_indexes,
+            exit,
+            Some(&addresstype_to_height_to_addresscount.p2sh),
+        )?;
+        self.p2wpkh.compute_rest(
+            indexes,
+            starting_indexes,
+            exit,
+            Some(&addresstype_to_height_to_addresscount.p2wpkh),
+        )?;
+        self.p2wsh.compute_rest(
+            indexes,
+            starting_indexes,
+            exit,
+            Some(&addresstype_to_height_to_addresscount.p2wsh),
+        )?;
+        self.p2tr.compute_rest(
+            indexes,
+            starting_indexes,
+            exit,
+            Some(&addresstype_to_height_to_addresscount.p2tr),
+        )?;
+        self.p2a.compute_rest(
+            indexes,
+            starting_indexes,
+            exit,
+            Some(&addresstype_to_height_to_addresscount.p2a),
+        )?;
+        Ok(())
+    }
+
+    pub fn iter_any_collectable(&self) -> impl Iterator<Item = &dyn AnyCollectableVec> {
+        self.0
+            .iter_typed()
+            .flat_map(|(_, v)| v.iter_any_collectable())
+    }
+}

crates/brk_computer/src/stateful/addresstype/mod.rs

@@ -0,0 +1,13 @@
+mod addresscount;
+mod height_to_addresscount;
+mod height_to_vec;
+mod indexes_to_addresscount;
+mod typeindex_tree;
+mod vec;
+
+pub use addresscount::*;
+pub use height_to_addresscount::*;
+pub use height_to_vec::*;
+pub use indexes_to_addresscount::*;
+pub use typeindex_tree::*;
+pub use vec::*;

crates/brk_computer/src/stateful/addresstype/typeindex_tree.rs

@@ -0,0 +1,49 @@
+use std::{collections::BTreeMap, mem};
+
+use brk_structs::{ByAddressType, TypeIndex};
+use derive_deref::{Deref, DerefMut};
+
+#[derive(Debug, Deref, DerefMut)]
+pub struct AddressTypeToTypeIndexTree<T>(ByAddressType<BTreeMap<TypeIndex, T>>);
+
+impl<T> AddressTypeToTypeIndexTree<T> {
+    pub fn merge(mut self, mut other: Self) -> Self {
+        Self::merge_(&mut self.p2pk65, &mut other.p2pk65);
+        Self::merge_(&mut self.p2pk33, &mut other.p2pk33);
+        Self::merge_(&mut self.p2pkh, &mut other.p2pkh);
+        Self::merge_(&mut self.p2sh, &mut other.p2sh);
+        Self::merge_(&mut self.p2wpkh, &mut other.p2wpkh);
+        Self::merge_(&mut self.p2wsh, &mut other.p2wsh);
+        Self::merge_(&mut self.p2tr, &mut other.p2tr);
+        Self::merge_(&mut self.p2a, &mut other.p2a);
+        self
+    }
+
+    fn merge_(own: &mut BTreeMap<TypeIndex, T>, other: &mut BTreeMap<TypeIndex, T>) {
+        if own.len() >= other.len() {
+            own.append(other);
+        } else {
+            other.append(own);
+            mem::swap(own, other);
+        }
+    }
+
+    pub fn unwrap(self) -> ByAddressType<BTreeMap<TypeIndex, T>> {
+        self.0
+    }
+}
+
+impl<T> Default for AddressTypeToTypeIndexTree<T> {
+    fn default() -> Self {
+        Self(ByAddressType {
+            p2pk65: BTreeMap::default(),
+            p2pk33: BTreeMap::default(),
+            p2pkh: BTreeMap::default(),
+            p2sh: BTreeMap::default(),
+            p2wpkh: BTreeMap::default(),
+            p2wsh: BTreeMap::default(),
+            p2tr: BTreeMap::default(),
+            p2a: BTreeMap::default(),
+        })
+    }
+}

crates/brk_computer/src/stateful/addresstype/vec.rs

@@ -0,0 +1,60 @@
+use std::mem;
+
+use brk_structs::ByAddressType;
+use derive_deref::{Deref, DerefMut};
+
+#[derive(Debug, Deref, DerefMut)]
+pub struct AddressTypeToVec<T>(ByAddressType<Vec<T>>);
+
+impl<T> AddressTypeToVec<T> {
+    pub fn merge(mut self, mut other: Self) -> Self {
+        Self::merge_(&mut self.p2pk65, &mut other.p2pk65);
+        Self::merge_(&mut self.p2pk33, &mut other.p2pk33);
+        Self::merge_(&mut self.p2pkh, &mut other.p2pkh);
+        Self::merge_(&mut self.p2sh, &mut other.p2sh);
+        Self::merge_(&mut self.p2wpkh, &mut other.p2wpkh);
+        Self::merge_(&mut self.p2wsh, &mut other.p2wsh);
+        Self::merge_(&mut self.p2tr, &mut other.p2tr);
+        Self::merge_(&mut self.p2a, &mut other.p2a);
+        self
+    }
+
+    pub fn merge_mut(&mut self, mut other: Self) {
+        Self::merge_(&mut self.p2pk65, &mut other.p2pk65);
+        Self::merge_(&mut self.p2pk33, &mut other.p2pk33);
+        Self::merge_(&mut self.p2pkh, &mut other.p2pkh);
+        Self::merge_(&mut self.p2sh, &mut other.p2sh);
+        Self::merge_(&mut self.p2wpkh, &mut other.p2wpkh);
+        Self::merge_(&mut self.p2wsh, &mut other.p2wsh);
+        Self::merge_(&mut self.p2tr, &mut other.p2tr);
+        Self::merge_(&mut self.p2a, &mut other.p2a);
+    }
+
+    fn merge_(own: &mut Vec<T>, other: &mut Vec<T>) {
+        if own.len() >= other.len() {
+            own.append(other);
+        } else {
+            other.append(own);
+            mem::swap(own, other);
+        }
+    }
+
+    pub fn unwrap(self) -> ByAddressType<Vec<T>> {
+        self.0
+    }
+}
+
+impl<T> Default for AddressTypeToVec<T> {
+    fn default() -> Self {
+        Self(ByAddressType {
+            p2pk65: vec![],
+            p2pk33: vec![],
+            p2pkh: vec![],
+            p2sh: vec![],
+            p2wpkh: vec![],
+            p2wsh: vec![],
+            p2tr: vec![],
+            p2a: vec![],
+        })
+    }
+}

crates/brk_computer/src/stateful/range_map.rs

@@ -0,0 +1,49 @@
+use std::collections::BTreeMap;
+
+use vecdb::{CompressedVec, RawVec, StoredCompressed, StoredIndex, StoredRaw};
+
+#[derive(Debug)]
+pub struct RangeMap<I, T>(BTreeMap<I, T>);
+
+impl<I, T> RangeMap<I, T>
+where
+    I: StoredIndex,
+    T: StoredIndex,
+{
+    pub fn get(&self, key: I) -> Option<&T> {
+        self.0.range(..=key).next_back().map(|(&min, value)| {
+            if min > key {
+                unreachable!()
+            }
+            value
+        })
+    }
+}
+
+impl<I, T> From<&RawVec<I, T>> for RangeMap<T, I>
+where
+    I: StoredIndex,
+    T: StoredIndex + StoredRaw,
+{
+    fn from(vec: &RawVec<I, T>) -> Self {
+        Self(
+            vec.into_iter()
+                .map(|(i, v)| (v.into_owned(), i))
+                .collect::<BTreeMap<_, _>>(),
+        )
+    }
+}
+
+impl<I, T> From<&CompressedVec<I, T>> for RangeMap<T, I>
+where
+    I: StoredIndex,
+    T: StoredIndex + StoredCompressed,
+{
+    fn from(vec: &CompressedVec<I, T>) -> Self {
+        Self(
+            vec.into_iter()
+                .map(|(i, v)| (v.into_owned(), i))
+                .collect::<BTreeMap<_, _>>(),
+        )
+    }
+}

crates/brk_computer/src/stateful/trait.rs

@@ -0,0 +1,60 @@
+use brk_error::Result;
+use brk_structs::{Bitcoin, DateIndex, Dollars, Height, Version};
+use vecdb::{AnyIterableVec, Exit};
+
+use crate::{Indexes, indexes, price};
+
+pub trait DynCohortVecs: Send + Sync {
+    fn min_height_vecs_len(&self) -> usize;
+    fn reset_state_starting_height(&mut self);
+
+    fn import_state(&mut self, starting_height: Height) -> Result<Height>;
+
+    fn validate_computed_versions(&mut self, base_version: Version) -> Result<()>;
+
+    fn forced_pushed_at(&mut self, height: Height, exit: &Exit) -> Result<()>;
+
+    fn compute_then_force_push_unrealized_states(
+        &mut self,
+        height: Height,
+        height_price: Option<Dollars>,
+        dateindex: Option<DateIndex>,
+        date_price: Option<Option<Dollars>>,
+        exit: &Exit,
+    ) -> Result<()>;
+
+    fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()>;
+
+    #[allow(clippy::too_many_arguments)]
+    fn compute_rest_part1(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()>;
+}
+
+pub trait CohortVecs: DynCohortVecs {
+    fn compute_from_stateful(
+        &mut self,
+        starting_indexes: &Indexes,
+        others: &[&Self],
+        exit: &Exit,
+    ) -> Result<()>;
+
+    #[allow(clippy::too_many_arguments)]
+    fn compute_rest_part2(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
+        dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+        height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+        exit: &Exit,
+    ) -> Result<()>;
+}

crates/brk_computer/src/stateful/utxo_cohort.rs

@@ -0,0 +1,182 @@
+use std::{ops::Deref, path::Path};
+
+use brk_error::Result;
+use brk_structs::{Bitcoin, DateIndex, Dollars, Height, Version};
+use vecdb::{AnyIterableVec, Database, Exit, Format};
+
+use crate::{
+    Indexes, UTXOCohortState, indexes, price,
+    stateful::{
+        common,
+        r#trait::{CohortVecs, DynCohortVecs},
+    },
+};
+
+#[derive(Clone)]
+pub struct Vecs {
+    state_starting_height: Option<Height>,
+
+    pub state: Option<UTXOCohortState>,
+
+    inner: common::Vecs,
+}
+
+impl Vecs {
+    #[allow(clippy::too_many_arguments)]
+    pub fn forced_import(
+        db: &Database,
+        cohort_name: Option<&str>,
+        format: Format,
+        version: Version,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        states_path: Option<&Path>,
+        extended: bool,
+        compute_rel_to_all: bool,
+        compute_adjusted: bool,
+    ) -> Result<Self> {
+        let compute_dollars = price.is_some();
+
+        Ok(Self {
+            state_starting_height: None,
+
+            state: states_path.map(|states_path| {
+                UTXOCohortState::new(
+                    states_path,
+                    cohort_name.unwrap_or_default(),
+                    compute_dollars,
+                )
+            }),
+
+            inner: common::Vecs::forced_import(
+                db,
+                cohort_name,
+                format,
+                version,
+                indexes,
+                price,
+                extended,
+                compute_rel_to_all,
+                compute_adjusted,
+            )?,
+        })
+    }
+}
+
+impl DynCohortVecs for Vecs {
+    fn min_height_vecs_len(&self) -> usize {
+        self.inner.min_height_vecs_len()
+    }
+
+    fn reset_state_starting_height(&mut self) {
+        self.state_starting_height = Some(Height::ZERO);
+    }
+
+    fn import_state(&mut self, starting_height: Height) -> Result<Height> {
+        let starting_height = self
+            .inner
+            .import_state(starting_height, self.state.as_mut().unwrap())?;
+
+        self.state_starting_height = Some(starting_height);
+
+        Ok(starting_height)
+    }
+
+    fn validate_computed_versions(&mut self, base_version: Version) -> Result<()> {
+        self.inner.validate_computed_versions(base_version)
+    }
+
+    fn forced_pushed_at(&mut self, height: Height, exit: &Exit) -> Result<()> {
+        if self.state_starting_height.unwrap() > height {
+            return Ok(());
+        }
+
+        self.inner
+            .forced_pushed_at(height, exit, self.state.as_ref().unwrap())
+    }
+
+    fn compute_then_force_push_unrealized_states(
+        &mut self,
+        height: Height,
+        height_price: Option<Dollars>,
+        dateindex: Option<DateIndex>,
+        date_price: Option<Option<Dollars>>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.inner.compute_then_force_push_unrealized_states(
+            height,
+            height_price,
+            dateindex,
+            date_price,
+            exit,
+            self.state.as_mut().unwrap(),
+        )
+    }
+
+    fn safe_flush_stateful_vecs(&mut self, height: Height, exit: &Exit) -> Result<()> {
+        self.inner
+            .safe_flush_stateful_vecs(height, exit, self.state.as_mut().unwrap())
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    fn compute_rest_part1(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.inner
+            .compute_rest_part1(indexes, price, starting_indexes, exit)
+    }
+}
+
+impl CohortVecs for Vecs {
+    fn compute_from_stateful(
+        &mut self,
+        starting_indexes: &Indexes,
+        others: &[&Self],
+        exit: &Exit,
+    ) -> Result<()> {
+        self.inner.compute_from_stateful(
+            starting_indexes,
+            &others.iter().map(|v| &v.inner).collect::<Vec<_>>(),
+            exit,
+        )
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    fn compute_rest_part2(
+        &mut self,
+        indexes: &indexes::Vecs,
+        price: Option<&price::Vecs>,
+        starting_indexes: &Indexes,
+        height_to_supply: &impl AnyIterableVec<Height, Bitcoin>,
+        dateindex_to_supply: &impl AnyIterableVec<DateIndex, Bitcoin>,
+        height_to_market_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_market_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+        height_to_realized_cap: Option<&impl AnyIterableVec<Height, Dollars>>,
+        dateindex_to_realized_cap: Option<&impl AnyIterableVec<DateIndex, Dollars>>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.inner.compute_rest_part2(
+            indexes,
+            price,
+            starting_indexes,
+            height_to_supply,
+            dateindex_to_supply,
+            height_to_market_cap,
+            dateindex_to_market_cap,
+            height_to_realized_cap,
+            dateindex_to_realized_cap,
+            exit,
+        )
+    }
+}
+
+impl Deref for Vecs {
+    type Target = common::Vecs;
+    fn deref(&self) -> &Self::Target {
+        &self.inner
+    }
+}

crates/brk_computer/src/stateful/withaddressdatasource.rs

@@ -0,0 +1,54 @@
+use brk_structs::{EmptyAddressData, EmptyAddressIndex, LoadedAddressData, LoadedAddressIndex};
+
+#[derive(Debug)]
+pub enum WithAddressDataSource<T> {
+    New(T),
+    FromLoadedAddressDataVec((LoadedAddressIndex, T)),
+    FromEmptyAddressDataVec((EmptyAddressIndex, T)),
+}
+
+impl<T> WithAddressDataSource<T> {
+    pub fn is_new(&self) -> bool {
+        matches!(self, Self::New(_))
+    }
+
+    pub fn is_from_emptyaddressdata(&self) -> bool {
+        matches!(self, Self::FromEmptyAddressDataVec(_))
+    }
+
+    pub fn deref_mut(&mut self) -> &mut T {
+        match self {
+            Self::New(v) => v,
+            Self::FromLoadedAddressDataVec((_, v)) => v,
+            Self::FromEmptyAddressDataVec((_, v)) => v,
+        }
+    }
+}
+
+impl From<WithAddressDataSource<EmptyAddressData>> for WithAddressDataSource<LoadedAddressData> {
+    fn from(value: WithAddressDataSource<EmptyAddressData>) -> Self {
+        match value {
+            WithAddressDataSource::New(v) => Self::New(v.into()),
+            WithAddressDataSource::FromLoadedAddressDataVec((i, v)) => {
+                Self::FromLoadedAddressDataVec((i, v.into()))
+            }
+            WithAddressDataSource::FromEmptyAddressDataVec((i, v)) => {
+                Self::FromEmptyAddressDataVec((i, v.into()))
+            }
+        }
+    }
+}
+
+impl From<WithAddressDataSource<LoadedAddressData>> for WithAddressDataSource<EmptyAddressData> {
+    fn from(value: WithAddressDataSource<LoadedAddressData>) -> Self {
+        match value {
+            WithAddressDataSource::New(v) => Self::New(v.into()),
+            WithAddressDataSource::FromLoadedAddressDataVec((i, v)) => {
+                Self::FromLoadedAddressDataVec((i, v.into()))
+            }
+            WithAddressDataSource::FromEmptyAddressDataVec((i, v)) => {
+                Self::FromEmptyAddressDataVec((i, v.into()))
+            }
+        }
+    }
+}

crates/brk_computer/src/states/block.rs

@@ -1,13 +1,17 @@
 use std::ops::{Add, AddAssign, SubAssign};
 
-use brk_core::{Dollars, Timestamp};
+use brk_structs::{Dollars, Timestamp};
+use serde::Serialize;
 
 use super::SupplyState;
 
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, Serialize)]
 pub struct BlockState {
+    #[serde(flatten)]
     pub supply: SupplyState,
+    #[serde(skip)]
     pub price: Option<Dollars>,
+    #[serde(skip)]
     pub timestamp: Timestamp,
 }
 

crates/brk_computer/src/states/cohorts/address.rs

@@ -0,0 +1,122 @@
+use std::path::Path;
+
+use brk_error::Result;
+use brk_structs::{Dollars, Height, LoadedAddressData, Sats};
+
+use crate::SupplyState;
+
+use super::CohortState;
+
+#[derive(Clone)]
+pub struct AddressCohortState {
+    pub addr_count: u64,
+    pub inner: CohortState,
+}
+
+impl AddressCohortState {
+    pub fn new(path: &Path, name: &str, compute_dollars: bool) -> Self {
+        Self {
+            addr_count: 0,
+            inner: CohortState::new(path, name, compute_dollars),
+        }
+    }
+
+    pub fn reset_price_to_amount_if_needed(&mut self) -> Result<()> {
+        self.inner.reset_price_to_amount_if_needed()
+    }
+
+    pub fn reset_single_iteration_values(&mut self) {
+        self.inner.reset_single_iteration_values();
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn send(
+        &mut self,
+        addressdata: &mut LoadedAddressData,
+        value: Sats,
+        current_price: Option<Dollars>,
+        prev_price: Option<Dollars>,
+        blocks_old: usize,
+        days_old: f64,
+        older_than_hour: bool,
+    ) -> Result<()> {
+        let compute_price = current_price.is_some();
+
+        let prev_realized_price = compute_price.then(|| addressdata.realized_price());
+        let prev_supply_state = SupplyState {
+            utxos: addressdata.utxos as u64,
+            value: addressdata.amount(),
+        };
+
+        addressdata.send(value, prev_price)?;
+
+        let supply_state = SupplyState {
+            utxos: addressdata.utxos as u64,
+            value: addressdata.amount(),
+        };
+
+        self.inner.send_(
+            &SupplyState { utxos: 1, value },
+            current_price,
+            prev_price,
+            blocks_old,
+            days_old,
+            older_than_hour,
+            compute_price.then(|| (addressdata.realized_price(), &supply_state)),
+            prev_realized_price.map(|prev_price| (prev_price, &prev_supply_state)),
+        );
+
+        Ok(())
+    }
+
+    pub fn receive(
+        &mut self,
+        address_data: &mut LoadedAddressData,
+        value: Sats,
+        price: Option<Dollars>,
+    ) {
+        let compute_price = price.is_some();
+
+        let prev_realized_price = compute_price.then(|| address_data.realized_price());
+        let prev_supply_state = SupplyState {
+            utxos: address_data.utxos as u64,
+            value: address_data.amount(),
+        };
+
+        address_data.receive(value, price);
+
+        let supply_state = SupplyState {
+            utxos: address_data.utxos as u64,
+            value: address_data.amount(),
+        };
+
+        self.inner.receive_(
+            &SupplyState { utxos: 1, value },
+            price,
+            compute_price.then(|| (address_data.realized_price(), &supply_state)),
+            prev_realized_price.map(|prev_price| (prev_price, &prev_supply_state)),
+        );
+    }
+
+    pub fn add(&mut self, addressdata: &LoadedAddressData) {
+        self.addr_count += 1;
+        self.inner.increment_(
+            &addressdata.into(),
+            addressdata.realized_cap,
+            addressdata.realized_price(),
+        );
+    }
+
+    pub fn subtract(&mut self, addressdata: &LoadedAddressData) {
+        self.addr_count = self.addr_count.checked_sub(1).unwrap();
+        self.inner.decrement_(
+            &addressdata.into(),
+            addressdata.realized_cap,
+            addressdata.realized_price(),
+        );
+    }
+
+    pub fn commit(&mut self, height: Height) -> Result<()> {
+        self.inner.commit(height)
+    }
+}

crates/brk_computer/src/states/cohorts/common.rs

@@ -0,0 +1,315 @@
+use std::{cmp::Ordering, path::Path};
+
+use brk_error::Result;
+use brk_structs::{CheckedSub, Dollars, Height, Sats};
+
+use crate::{PriceToAmount, RealizedState, SupplyState, UnrealizedState};
+
+#[derive(Clone)]
+pub struct CohortState {
+    pub supply: SupplyState,
+
+    pub realized: Option<RealizedState>,
+    pub satblocks_destroyed: Sats,
+    pub satdays_destroyed: Sats,
+
+    price_to_amount: Option<PriceToAmount>,
+}
+
+impl CohortState {
+    pub fn new(path: &Path, name: &str, compute_dollars: bool) -> Self {
+        Self {
+            supply: SupplyState::default(),
+            realized: compute_dollars.then_some(RealizedState::NAN),
+            satblocks_destroyed: Sats::ZERO,
+            satdays_destroyed: Sats::ZERO,
+            price_to_amount: compute_dollars.then_some(PriceToAmount::create(path, name)),
+        }
+    }
+
+    pub fn import_at_or_before(&mut self, height: Height) -> Result<Height> {
+        if let Some(price_to_amount) = self.price_to_amount.as_mut() {
+            price_to_amount.import_at_or_before(height)
+        } else {
+            Ok(height)
+        }
+    }
+
+    pub fn reset_price_to_amount_if_needed(&mut self) -> Result<()> {
+        if let Some(price_to_amount) = self.price_to_amount.as_mut() {
+            price_to_amount.clean()?;
+            price_to_amount.init();
+        }
+        Ok(())
+    }
+
+    pub fn price_to_amount_first_key_value(&self) -> Option<(&Dollars, &Sats)> {
+        self.price_to_amount.as_ref().unwrap().first_key_value()
+    }
+
+    pub fn price_to_amount_last_key_value(&self) -> Option<(&Dollars, &Sats)> {
+        self.price_to_amount.as_ref().unwrap().last_key_value()
+    }
+
+    pub fn reset_single_iteration_values(&mut self) {
+        self.satdays_destroyed = Sats::ZERO;
+        self.satblocks_destroyed = Sats::ZERO;
+        if let Some(realized) = self.realized.as_mut() {
+            realized.reset_single_iteration_values();
+        }
+    }
+
+    pub fn increment(&mut self, supply_state: &SupplyState, price: Option<Dollars>) {
+        self.supply += supply_state;
+
+        if supply_state.value > Sats::ZERO
+            && let Some(realized) = self.realized.as_mut()
+        {
+            let price = price.unwrap();
+            realized.increment(supply_state, price);
+            self.price_to_amount
+                .as_mut()
+                .unwrap()
+                .increment(price, supply_state);
+        }
+    }
+
+    pub fn increment_(
+        &mut self,
+        supply_state: &SupplyState,
+        realized_cap: Dollars,
+        realized_price: Dollars,
+    ) {
+        self.supply += supply_state;
+
+        if supply_state.value > Sats::ZERO
+            && let Some(realized) = self.realized.as_mut()
+        {
+            realized.increment_(realized_cap);
+            self.price_to_amount
+                .as_mut()
+                .unwrap()
+                .increment(realized_price, supply_state);
+        }
+    }
+
+    pub fn decrement(&mut self, supply_state: &SupplyState, price: Option<Dollars>) {
+        self.supply -= supply_state;
+
+        if supply_state.value > Sats::ZERO
+            && let Some(realized) = self.realized.as_mut()
+        {
+            let price = price.unwrap();
+            realized.decrement(supply_state, price);
+            self.price_to_amount
+                .as_mut()
+                .unwrap()
+                .decrement(price, supply_state);
+        }
+    }
+
+    pub fn decrement_(
+        &mut self,
+        supply_state: &SupplyState,
+        realized_cap: Dollars,
+        realized_price: Dollars,
+    ) {
+        self.supply -= supply_state;
+
+        if supply_state.value > Sats::ZERO
+            && let Some(realized) = self.realized.as_mut()
+        {
+            realized.decrement_(realized_cap);
+            self.price_to_amount
+                .as_mut()
+                .unwrap()
+                .decrement(realized_price, supply_state);
+        }
+    }
+
+    pub fn receive(&mut self, supply_state: &SupplyState, price: Option<Dollars>) {
+        self.receive_(
+            supply_state,
+            price,
+            price.map(|price| (price, supply_state)),
+            None,
+        );
+    }
+
+    pub fn receive_(
+        &mut self,
+        supply_state: &SupplyState,
+        price: Option<Dollars>,
+        price_to_amount_increment: Option<(Dollars, &SupplyState)>,
+        price_to_amount_decrement: Option<(Dollars, &SupplyState)>,
+    ) {
+        self.supply += supply_state;
+
+        if supply_state.value > Sats::ZERO
+            && let Some(realized) = self.realized.as_mut()
+        {
+            let price = price.unwrap();
+            realized.receive(supply_state, price);
+
+            if let Some((price, supply)) = price_to_amount_increment
+                && supply.value.is_not_zero()
+            {
+                self.price_to_amount
+                    .as_mut()
+                    .unwrap()
+                    .increment(price, supply);
+            }
+
+            if let Some((price, supply)) = price_to_amount_decrement
+                && supply.value.is_not_zero()
+            {
+                self.price_to_amount
+                    .as_mut()
+                    .unwrap()
+                    .decrement(price, supply);
+            }
+        }
+    }
+
+    pub fn send(
+        &mut self,
+        supply_state: &SupplyState,
+        current_price: Option<Dollars>,
+        prev_price: Option<Dollars>,
+        blocks_old: usize,
+        days_old: f64,
+        older_than_hour: bool,
+    ) {
+        self.send_(
+            supply_state,
+            current_price,
+            prev_price,
+            blocks_old,
+            days_old,
+            older_than_hour,
+            None,
+            prev_price.map(|prev_price| (prev_price, supply_state)),
+        );
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn send_(
+        &mut self,
+        supply_state: &SupplyState,
+        current_price: Option<Dollars>,
+        prev_price: Option<Dollars>,
+        blocks_old: usize,
+        days_old: f64,
+        older_than_hour: bool,
+        price_to_amount_increment: Option<(Dollars, &SupplyState)>,
+        price_to_amount_decrement: Option<(Dollars, &SupplyState)>,
+    ) {
+        if supply_state.utxos == 0 {
+            return;
+        }
+
+        self.supply -= supply_state;
+
+        if supply_state.value > Sats::ZERO {
+            self.satblocks_destroyed += supply_state.value * blocks_old;
+
+            self.satdays_destroyed +=
+                Sats::from((u64::from(supply_state.value) as f64 * days_old).floor() as u64);
+
+            if let Some(realized) = self.realized.as_mut() {
+                let current_price = current_price.unwrap();
+                let prev_price = prev_price.unwrap();
+                realized.send(supply_state, current_price, prev_price, older_than_hour);
+                if let Some((price, supply)) = price_to_amount_increment
+                    && supply.value.is_not_zero()
+                {
+                    self.price_to_amount
+                        .as_mut()
+                        .unwrap()
+                        .increment(price, supply);
+                }
+                if let Some((price, supply)) = price_to_amount_decrement
+                    && supply.value.is_not_zero()
+                {
+                    self.price_to_amount
+                        .as_mut()
+                        .unwrap()
+                        .decrement(price, supply);
+                }
+            }
+        }
+    }
+
+    pub fn compute_unrealized_states(
+        &self,
+        height_price: Dollars,
+        date_price: Option<Dollars>,
+    ) -> (UnrealizedState, Option<UnrealizedState>) {
+        if self.price_to_amount.as_ref().unwrap().is_empty() {
+            return (
+                UnrealizedState::NAN,
+                date_price.map(|_| UnrealizedState::NAN),
+            );
+        }
+
+        let mut height_unrealized_state = UnrealizedState::ZERO;
+        let mut date_unrealized_state = date_price.map(|_| UnrealizedState::ZERO);
+
+        let update_state =
+            |price: Dollars, current_price: Dollars, sats: Sats, state: &mut UnrealizedState| {
+                let cmp = price.cmp(&current_price);
+                match cmp {
+                    Ordering::Equal | Ordering::Less => {
+                        state.supply_in_profit += sats;
+                        if !cmp.is_eq() && price > Dollars::ZERO && current_price > Dollars::ZERO {
+                            let diff = current_price.checked_sub(price).unwrap();
+                            // Add back once in a while to verify, but generally not needed
+                            // if diff <= Dollars::ZERO {
+                            //     dbg!(price, current_price, diff, sats);
+                            //     panic!();
+                            // }
+                            state.unrealized_profit += diff * sats;
+                        }
+                    }
+                    Ordering::Greater => {
+                        state.supply_in_loss += sats;
+                        if price > Dollars::ZERO && current_price > Dollars::ZERO {
+                            let diff = price.checked_sub(current_price).unwrap();
+                            // Add back once in a while to verify, but generally not needed
+                            // if diff <= Dollars::ZERO {
+                            //     dbg!(price, current_price, diff, sats);
+                            //     panic!();
+                            // }
+                            state.unrealized_loss += diff * sats;
+                        }
+                    }
+                }
+            };
+
+        self.price_to_amount
+            .as_ref()
+            .unwrap()
+            .iter()
+            .for_each(|(&price, &sats)| {
+                update_state(price, height_price, sats, &mut height_unrealized_state);
+
+                if let Some(date_price) = date_price {
+                    update_state(
+                        price,
+                        date_price,
+                        sats,
+                        date_unrealized_state.as_mut().unwrap(),
+                    )
+                }
+            });
+
+        (height_unrealized_state, date_unrealized_state)
+    }
+
+    pub fn commit(&mut self, height: Height) -> Result<()> {
+        if let Some(price_to_amount) = self.price_to_amount.as_mut() {
+            price_to_amount.flush(height)?;
+        }
+        Ok(())
+    }
+}

crates/brk_computer/src/states/cohorts/mod.rs

@@ -0,0 +1,7 @@
+mod address;
+mod common;
+mod utxo;
+
+pub use address::*;
+pub use common::*;
+pub use utxo::*;

crates/brk_computer/src/states/cohorts/utxo.rs

@@ -0,0 +1,19 @@
+use std::path::Path;
+
+use brk_error::Result;
+use derive_deref::{Deref, DerefMut};
+
+use super::CohortState;
+
+#[derive(Clone, Deref, DerefMut)]
+pub struct UTXOCohortState(CohortState);
+
+impl UTXOCohortState {
+    pub fn new(path: &Path, name: &str, compute_dollars: bool) -> Self {
+        Self(CohortState::new(path, name, compute_dollars))
+    }
+
+    pub fn reset_price_to_amount_if_needed(&mut self) -> Result<()> {
+        self.0.reset_price_to_amount_if_needed()
+    }
+}

crates/brk_computer/src/states/mod.rs

@@ -1,19 +1,15 @@
 mod block;
-mod cohort;
-mod outputs;
-mod realized;
-// mod hot;
+mod cohorts;
 mod price_to_amount;
+mod realized;
 mod supply;
 mod transacted;
 mod unrealized;
 
 pub use block::*;
-pub use cohort::*;
-pub use outputs::*;
-pub use realized::*;
-pub use unrealized::*;
-// pub use hot::*;
+pub use cohorts::*;
 pub use price_to_amount::*;
+pub use realized::*;
 pub use supply::*;
 pub use transacted::*;
+pub use unrealized::*;

crates/brk_computer/src/states/price_to_amount.rs

@@ -0,0 +1,171 @@
+use std::{
+    collections::BTreeMap,
+    fs,
+    path::{Path, PathBuf},
+};
+
+use brk_error::{Error, Result};
+use brk_structs::{Dollars, Height, Sats};
+use derive_deref::{Deref, DerefMut};
+use pco::standalone::{simple_decompress, simpler_compress};
+use serde::{Deserialize, Serialize};
+use zerocopy::{FromBytes, IntoBytes};
+
+use crate::states::SupplyState;
+
+#[derive(Clone, Debug)]
+pub struct PriceToAmount {
+    pathbuf: PathBuf,
+    state: Option<State>,
+}
+
+const STATE_AT_: &str = "state_at_";
+const STATE_TO_KEEP: usize = 10;
+
+impl PriceToAmount {
+    pub fn create(path: &Path, name: &str) -> Self {
+        Self {
+            pathbuf: path.join(format!("{name}_price_to_amount")),
+            state: None,
+        }
+    }
+
+    pub fn import_at_or_before(&mut self, height: Height) -> Result<Height> {
+        let files = self.read_dir(None)?;
+        let (&height, path) = files
+            .range(..=height)
+            .next_back()
+            .ok_or(Error::Str("Not found"))?;
+        self.state = Some(State::deserialize(&fs::read(path)?)?);
+        Ok(height)
+    }
+
+    pub fn iter(&self) -> impl Iterator<Item = (&Dollars, &Sats)> {
+        self.state.as_ref().unwrap().iter()
+    }
+
+    pub fn is_empty(&self) -> bool {
+        self.state.as_ref().unwrap().is_empty()
+    }
+
+    pub fn first_key_value(&self) -> Option<(&Dollars, &Sats)> {
+        self.state.as_ref().unwrap().first_key_value()
+    }
+
+    pub fn last_key_value(&self) -> Option<(&Dollars, &Sats)> {
+        self.state.as_ref().unwrap().last_key_value()
+    }
+
+    pub fn increment(&mut self, price: Dollars, supply_state: &SupplyState) {
+        *self.state.as_mut().unwrap().entry(price).or_default() += supply_state.value;
+    }
+
+    pub fn decrement(&mut self, price: Dollars, supply_state: &SupplyState) {
+        if let Some(amount) = self.state.as_mut().unwrap().get_mut(&price) {
+            *amount -= supply_state.value;
+            if *amount == Sats::ZERO {
+                self.state.as_mut().unwrap().remove(&price);
+            }
+        } else {
+            dbg!(price, &self.pathbuf);
+            unreachable!();
+        }
+    }
+
+    pub fn init(&mut self) {
+        self.state.replace(State::default());
+    }
+
+    pub fn clean(&mut self) -> Result<()> {
+        let _ = fs::remove_dir_all(&self.pathbuf);
+        fs::create_dir_all(&self.pathbuf)?;
+        Ok(())
+    }
+
+    fn read_dir(&self, keep_only_before: Option<Height>) -> Result<BTreeMap<Height, PathBuf>> {
+        Ok(fs::read_dir(&self.pathbuf)?
+            .filter_map(|entry| {
+                let path = entry.ok()?.path();
+                let name = path.file_name()?.to_str()?;
+                let height_str = name.strip_prefix(STATE_AT_).unwrap_or(name);
+                if let Ok(h) = height_str.parse::<u32>().map(Height::from) {
+                    if keep_only_before.is_none_or(|height| h < height) {
+                        Some((h, path))
+                    } else {
+                        let _ = fs::remove_file(path);
+                        None
+                    }
+                } else {
+                    None
+                }
+            })
+            .collect::<BTreeMap<Height, PathBuf>>())
+    }
+
+    pub fn flush(&mut self, height: Height) -> Result<()> {
+        let files = self.read_dir(Some(height))?;
+
+        for (_, path) in files
+            .iter()
+            .take(files.len().saturating_sub(STATE_TO_KEEP - 1))
+        {
+            fs::remove_file(path)?;
+        }
+
+        fs::write(
+            self.path_state(height),
+            self.state.as_ref().unwrap().serialize()?,
+        )?;
+
+        Ok(())
+    }
+
+    fn path_state(&self, height: Height) -> PathBuf {
+        Self::path_state_(&self.pathbuf, height)
+    }
+    fn path_state_(path: &Path, height: Height) -> PathBuf {
+        path.join(u32::from(height).to_string())
+    }
+}
+
+#[derive(Clone, Default, Debug, Deref, DerefMut, Serialize, Deserialize)]
+struct State(BTreeMap<Dollars, Sats>);
+
+const COMPRESSION_LEVEL: usize = 4;
+
+impl State {
+    fn serialize(&self) -> vecdb::Result<Vec<u8>> {
+        let keys: Vec<f64> = self.keys().cloned().map(f64::from).collect();
+
+        let values: Vec<u64> = self.values().cloned().map(u64::from).collect();
+
+        let compressed_keys = simpler_compress(&keys, COMPRESSION_LEVEL)?;
+        let compressed_values = simpler_compress(&values, COMPRESSION_LEVEL)?;
+
+        let mut buffer = Vec::new();
+        buffer.extend(keys.len().as_bytes());
+        buffer.extend(compressed_keys.len().as_bytes());
+        buffer.extend(compressed_keys);
+        buffer.extend(compressed_values);
+
+        Ok(buffer)
+    }
+
+    fn deserialize(data: &[u8]) -> vecdb::Result<Self> {
+        let entry_count = usize::read_from_bytes(&data[0..8])?;
+        let keys_len = usize::read_from_bytes(&data[8..16])?;
+
+        let keys: Vec<f64> = simple_decompress(&data[16..16 + keys_len])?;
+        let values: Vec<u64> = simple_decompress(&data[16 + keys_len..])?;
+
+        let map: BTreeMap<Dollars, Sats> = keys
+            .into_iter()
+            .zip(values)
+            .map(|(k, v)| (Dollars::from(k), Sats::from(v)))
+            .collect();
+
+        assert_eq!(map.len(), entry_count);
+
+        Ok(Self(map))
+    }
+}

crates/brk_computer/src/states/realized.rs

@@ -1,6 +1,6 @@
 use std::cmp::Ordering;
 
-use brk_core::{CheckedSub, Dollars};
+use brk_structs::{CheckedSub, Dollars};
 
 use super::SupplyState;
 
@@ -42,6 +42,10 @@ impl RealizedState {
             return;
         }
 
+        self.increment_(price * supply_state.value)
+    }
+
+    pub fn increment_(&mut self, realized_cap: Dollars) {
         if self.cap == Dollars::NAN {
             self.cap = Dollars::ZERO;
             self.profit = Dollars::ZERO;
@@ -52,13 +56,15 @@ impl RealizedState {
             self.adj_value_destroyed = Dollars::ZERO;
         }
 
-        let value = price * supply_state.value;
-        self.cap += value;
+        self.cap += realized_cap;
     }
 
     pub fn decrement(&mut self, supply_state: &SupplyState, price: Dollars) {
-        let value = price * supply_state.value;
-        self.cap = self.cap.checked_sub(value).unwrap();
+        self.decrement_(price * supply_state.value);
+    }
+
+    pub fn decrement_(&mut self, realized_cap: Dollars) {
+        self.cap = self.cap.checked_sub(realized_cap).unwrap();
     }
 
     pub fn receive(&mut self, supply_state: &SupplyState, current_price: Dollars) {

crates/brk_computer/src/states/supply.rs

@@ -1,12 +1,12 @@
 use std::ops::{Add, AddAssign, SubAssign};
 
-use brk_core::{CheckedSub, Sats};
+use brk_structs::{CheckedSub, LoadedAddressData, Sats};
 use serde::Serialize;
 use zerocopy_derive::{FromBytes, Immutable, IntoBytes, KnownLayout};
 
 #[derive(Debug, Default, Clone, FromBytes, Immutable, IntoBytes, KnownLayout, Serialize)]
 pub struct SupplyState {
-    pub utxos: usize,
+    pub utxos: u64,
     pub value: Sats,
 }
 
@@ -39,3 +39,18 @@ impl SubAssign<&SupplyState> for SupplyState {
         self.value = self.value.checked_sub(rhs.value).unwrap();
     }
 }
+
+impl From<&LoadedAddressData> for SupplyState {
+    fn from(value: &LoadedAddressData) -> Self {
+        Self {
+            utxos: value.utxos as u64,
+            value: value.amount(),
+        }
+    }
+}
+
+impl std::fmt::Display for SupplyState {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "utxos: {}, value: {}", self.utxos, self.value)
+    }
+}

crates/brk_computer/src/states/transacted.rs

@@ -0,0 +1,48 @@
+use std::ops::{Add, AddAssign};
+
+use brk_structs::{ByAmountRange, GroupedByType, OutputType, Sats};
+
+use super::SupplyState;
+
+#[derive(Default, Debug)]
+pub struct Transacted {
+    pub spendable_supply: SupplyState,
+    pub by_type: GroupedByType<SupplyState>,
+    pub by_size_group: ByAmountRange<SupplyState>,
+}
+
+impl Transacted {
+    #[allow(clippy::inconsistent_digit_grouping)]
+    pub fn iterate(&mut self, value: Sats, _type: OutputType) {
+        let supply = SupplyState { utxos: 1, value };
+
+        *self.by_type.get_mut(_type) += &supply;
+
+        if _type.is_unspendable() {
+            return;
+        }
+
+        self.spendable_supply += &supply;
+
+        *self.by_size_group.get_mut(value) += &supply;
+    }
+}
+
+impl Add for Transacted {
+    type Output = Self;
+    fn add(self, rhs: Self) -> Self::Output {
+        Self {
+            spendable_supply: self.spendable_supply + rhs.spendable_supply,
+            by_type: self.by_type + rhs.by_type,
+            by_size_group: self.by_size_group + rhs.by_size_group,
+        }
+    }
+}
+
+impl AddAssign for Transacted {
+    fn add_assign(&mut self, rhs: Self) {
+        self.by_size_group += rhs.by_size_group;
+        self.spendable_supply += &rhs.spendable_supply;
+        self.by_type += rhs.by_type;
+    }
+}

crates/brk_computer/src/states/unrealized.rs

@@ -1,9 +1,8 @@
-use brk_core::{Dollars, Sats};
+use brk_structs::{Dollars, Sats};
 
 #[derive(Debug, Default, Clone)]
 pub struct UnrealizedState {
     pub supply_in_profit: Sats,
-    pub supply_even: Sats,
     pub supply_in_loss: Sats,
     pub unrealized_profit: Dollars,
     pub unrealized_loss: Dollars,
@@ -12,7 +11,6 @@ pub struct UnrealizedState {
 impl UnrealizedState {
     pub const NAN: Self = Self {
         supply_in_profit: Sats::ZERO,
-        supply_even: Sats::ZERO,
         supply_in_loss: Sats::ZERO,
         unrealized_profit: Dollars::NAN,
         unrealized_loss: Dollars::NAN,
@@ -20,7 +18,6 @@ impl UnrealizedState {
 
     pub const ZERO: Self = Self {
         supply_in_profit: Sats::ZERO,
-        supply_even: Sats::ZERO,
         supply_in_loss: Sats::ZERO,
         unrealized_profit: Dollars::ZERO,
         unrealized_loss: Dollars::ZERO,

crates/brk_computer/src/stores.rs

@@ -1,34 +0,0 @@
-use std::path::Path;
-
-use brk_core::Version;
-use fjall::TransactionalKeyspace;
-
-const _VERSION: Version = Version::ZERO;
-
-#[derive(Clone)]
-pub struct Stores {
-    // pub address_to_utxos_received: Store<AddressIndexOutputIndex, Unit>,
-    // pub address_to_utxos_spent: Store<AddressIndexOutputIndex, Unit>,
-}
-
-impl Stores {
-    pub fn import(_: &Path, _: Version, _: &TransactionalKeyspace) -> color_eyre::Result<Self> {
-        // let address_to_utxos_received = Store::import(
-        //     keyspace.clone(),
-        //     path,
-        //     "address_to_utxos_received",
-        //     version + VERSION + Version::ZERO,
-        // )?;
-        // let address_to_utxos_spent = Store::import(
-        //     keyspace.clone(),
-        //     path,
-        //     "address_to_utxos_spent",
-        //     version + VERSION + Version::ZERO,
-        // )?;
-
-        Ok(Self {
-            // address_to_utxos_received,
-            // address_to_utxos_spent,
-        })
-    }
-}

crates/brk_computer/src/traits.rs

@@ -0,0 +1,316 @@
+use brk_error::Result;
+use brk_structs::{Bitcoin, CheckedSub, Close, Date, DateIndex, Dollars, Sats, StoredF32};
+use vecdb::{
+    AnyIterableVec, AnyStoredVec, AnyVec, EagerVec, Exit, GenericStoredVec, StoredIndex,
+    VecIterator, Version,
+};
+
+const DCA_AMOUNT: Dollars = Dollars::mint(100.0);
+
+pub trait ComputeDCAStackViaLen {
+    fn compute_dca_stack_via_len(
+        &mut self,
+        max_from: DateIndex,
+        closes: &impl AnyIterableVec<DateIndex, Close<Dollars>>,
+        len: usize,
+        exit: &Exit,
+    ) -> Result<()>;
+
+    fn compute_dca_stack_via_from(
+        &mut self,
+        max_from: DateIndex,
+        closes: &impl AnyIterableVec<DateIndex, Close<Dollars>>,
+        from: DateIndex,
+        exit: &Exit,
+    ) -> Result<()>;
+}
+impl ComputeDCAStackViaLen for EagerVec<DateIndex, Sats> {
+    fn compute_dca_stack_via_len(
+        &mut self,
+        max_from: DateIndex,
+        closes: &impl AnyIterableVec<DateIndex, Close<Dollars>>,
+        len: usize,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.validate_computed_version_or_reset(
+            Version::ZERO + self.inner_version() + closes.version(),
+        )?;
+
+        let mut other_iter = closes.iter();
+        let mut prev = None;
+
+        let index = max_from.min(DateIndex::from(self.len()));
+        closes.iter_at(index).try_for_each(|(i, closes)| {
+            let price = *closes.into_owned();
+            let i_usize = i.unwrap_to_usize();
+            if prev.is_none() {
+                if i_usize == 0 {
+                    prev.replace(Sats::ZERO);
+                } else {
+                    prev.replace(self.into_iter().unwrap_get_inner_(i_usize - 1));
+                }
+            }
+
+            let mut stack = Sats::ZERO;
+
+            if price != Dollars::ZERO {
+                stack = prev.unwrap() + Sats::from(Bitcoin::from(DCA_AMOUNT / price));
+
+                if i_usize >= len {
+                    let prev_price = *other_iter.unwrap_get_inner_(i_usize - len);
+                    if prev_price != Dollars::ZERO {
+                        stack = stack
+                            .checked_sub(Sats::from(Bitcoin::from(DCA_AMOUNT / prev_price)))
+                            .unwrap();
+                    }
+                }
+            }
+
+            prev.replace(stack);
+
+            self.forced_push_at(i, stack, exit)
+        })?;
+
+        self.safe_flush(exit)?;
+
+        Ok(())
+    }
+
+    fn compute_dca_stack_via_from(
+        &mut self,
+        max_from: DateIndex,
+        closes: &impl AnyIterableVec<DateIndex, Close<Dollars>>,
+        from: DateIndex,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.validate_computed_version_or_reset(
+            Version::ZERO + self.inner_version() + closes.version(),
+        )?;
+
+        let mut prev = None;
+
+        let index = max_from.min(DateIndex::from(self.len()));
+        closes.iter_at(index).try_for_each(|(i, closes)| {
+            let price = *closes.into_owned();
+            let i_usize = i.unwrap_to_usize();
+            if prev.is_none() {
+                if i_usize == 0 {
+                    prev.replace(Sats::ZERO);
+                } else {
+                    prev.replace(self.into_iter().unwrap_get_inner_(i_usize - 1));
+                }
+            }
+
+            let mut stack = Sats::ZERO;
+
+            if price != Dollars::ZERO && i >= from {
+                stack = prev.unwrap() + Sats::from(Bitcoin::from(DCA_AMOUNT / price));
+            }
+
+            prev.replace(stack);
+
+            self.forced_push_at(i, stack, exit)
+        })?;
+
+        self.safe_flush(exit)?;
+
+        Ok(())
+    }
+}
+
+pub trait ComputeDCAAveragePriceViaLen {
+    fn compute_dca_avg_price_via_len(
+        &mut self,
+        max_from: DateIndex,
+        stacks: &impl AnyIterableVec<DateIndex, Sats>,
+        len: usize,
+        exit: &Exit,
+    ) -> Result<()>;
+
+    fn compute_dca_avg_price_via_from(
+        &mut self,
+        max_from: DateIndex,
+        stacks: &impl AnyIterableVec<DateIndex, Sats>,
+        from: DateIndex,
+        exit: &Exit,
+    ) -> Result<()>;
+}
+impl ComputeDCAAveragePriceViaLen for EagerVec<DateIndex, Dollars> {
+    fn compute_dca_avg_price_via_len(
+        &mut self,
+        max_from: DateIndex,
+        stacks: &impl AnyIterableVec<DateIndex, Sats>,
+        len: usize,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.validate_computed_version_or_reset(
+            Version::ONE + self.inner_version() + stacks.version(),
+        )?;
+
+        let index = max_from.min(DateIndex::from(self.len()));
+
+        let first_price_date = DateIndex::try_from(Date::new(2010, 7, 12)).unwrap();
+
+        stacks.iter_at(index).try_for_each(|(i, stack)| {
+            let stack = stack.into_owned();
+            let mut avg_price = Dollars::from(f64::NAN);
+            if i > first_price_date {
+                avg_price = DCA_AMOUNT
+                    * len
+                        .min(i.unwrap_to_usize() + 1)
+                        .min(i.checked_sub(first_price_date).unwrap().unwrap_to_usize() + 1)
+                    / Bitcoin::from(stack);
+            }
+            self.forced_push_at(i, avg_price, exit)
+        })?;
+
+        self.safe_flush(exit)?;
+
+        Ok(())
+    }
+
+    fn compute_dca_avg_price_via_from(
+        &mut self,
+        max_from: DateIndex,
+        stacks: &impl AnyIterableVec<DateIndex, Sats>,
+        from: DateIndex,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.validate_computed_version_or_reset(
+            Version::ZERO + self.inner_version() + stacks.version(),
+        )?;
+
+        let index = max_from.min(DateIndex::from(self.len()));
+
+        let from_usize = from.unwrap_to_usize();
+
+        stacks.iter_at(index).try_for_each(|(i, stack)| {
+            let stack = stack.into_owned();
+            let mut avg_price = Dollars::from(f64::NAN);
+            if i >= from {
+                avg_price =
+                    DCA_AMOUNT * (i.unwrap_to_usize() + 1 - from_usize) / Bitcoin::from(stack);
+            }
+            self.forced_push_at(i, avg_price, exit)
+        })?;
+
+        self.safe_flush(exit)?;
+
+        Ok(())
+    }
+}
+
+pub trait ComputeFromSats<I> {
+    fn compute_from_sats(
+        &mut self,
+        max_from: I,
+        sats: &impl AnyIterableVec<I, Sats>,
+        exit: &Exit,
+    ) -> Result<()>;
+}
+impl<I> ComputeFromSats<I> for EagerVec<I, Bitcoin>
+where
+    I: StoredIndex,
+{
+    fn compute_from_sats(
+        &mut self,
+        max_from: I,
+        sats: &impl AnyIterableVec<I, Sats>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.validate_computed_version_or_reset(
+            Version::ZERO + self.inner_version() + sats.version(),
+        )?;
+
+        let index = max_from.min(I::from(self.len()));
+        sats.iter_at(index).try_for_each(|(i, sats)| {
+            let (i, v) = (i, Bitcoin::from(sats.into_owned()));
+            self.forced_push_at(i, v, exit)
+        })?;
+
+        self.safe_flush(exit)?;
+
+        Ok(())
+    }
+}
+
+pub trait ComputeFromBitcoin<I> {
+    fn compute_from_bitcoin(
+        &mut self,
+        max_from: I,
+        bitcoin: &impl AnyIterableVec<I, Bitcoin>,
+        price: &impl AnyIterableVec<I, Close<Dollars>>,
+        exit: &Exit,
+    ) -> Result<()>;
+}
+impl<I> ComputeFromBitcoin<I> for EagerVec<I, Dollars>
+where
+    I: StoredIndex,
+{
+    fn compute_from_bitcoin(
+        &mut self,
+        max_from: I,
+        bitcoin: &impl AnyIterableVec<I, Bitcoin>,
+        price: &impl AnyIterableVec<I, Close<Dollars>>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.validate_computed_version_or_reset(
+            Version::ZERO + self.inner_version() + bitcoin.version(),
+        )?;
+
+        let mut price_iter = price.iter();
+        let index = max_from.min(I::from(self.len()));
+        bitcoin.iter_at(index).try_for_each(|(i, bitcoin)| {
+            let dollars = price_iter.unwrap_get_inner(i);
+            let (i, v) = (i, *dollars * bitcoin.into_owned());
+            self.forced_push_at(i, v, exit)
+        })?;
+
+        self.safe_flush(exit)?;
+
+        Ok(())
+    }
+}
+
+pub trait ComputeDrawdown<I> {
+    fn compute_drawdown(
+        &mut self,
+        max_from: I,
+        close: &impl AnyIterableVec<I, Close<Dollars>>,
+        ath: &impl AnyIterableVec<I, Dollars>,
+        exit: &Exit,
+    ) -> Result<()>;
+}
+impl<I> ComputeDrawdown<I> for EagerVec<I, StoredF32>
+where
+    I: StoredIndex,
+{
+    fn compute_drawdown(
+        &mut self,
+        max_from: I,
+        close: &impl AnyIterableVec<I, Close<Dollars>>,
+        ath: &impl AnyIterableVec<I, Dollars>,
+        exit: &Exit,
+    ) -> Result<()> {
+        self.validate_computed_version_or_reset(
+            Version::ZERO + self.inner_version() + ath.version() + close.version(),
+        )?;
+
+        let index = max_from.min(I::from(self.len()));
+        let mut close_iter = close.iter();
+        ath.iter_at(index).try_for_each(|(i, ath)| {
+            let ath = ath.into_owned();
+            if ath == Dollars::ZERO {
+                self.forced_push_at(i, StoredF32::default(), exit)
+            } else {
+                let close = *close_iter.unwrap_get_inner(i);
+                let drawdown = StoredF32::from((*ath - *close) / *ath * -100.0);
+                self.forced_push_at(i, drawdown, exit)
+            }
+        })?;
+
+        self.safe_flush(exit)?;
+
+        Ok(())
+    }
+}

crates/brk_computer/src/vecs/blocks.rs

@@ -1,264 +0,0 @@
-use std::{fs, path::Path};
-
-use brk_core::{
-    CheckedSub, DifficultyEpoch, HalvingEpoch, Height, StoredU32, StoredU64, StoredUsize,
-    Timestamp, Version, Weight,
-};
-use brk_exit::Exit;
-use brk_indexer::Indexer;
-use brk_vec::{AnyCollectableVec, AnyIterableVec, Computation, EagerVec, Format};
-
-use super::{
-    Indexes,
-    grouped::{ComputedVecsFromDateIndex, ComputedVecsFromHeight, StorableVecGeneatorOptions},
-    indexes,
-};
-
-const VERSION: Version = Version::ZERO;
-
-#[derive(Clone)]
-pub struct Vecs {
-    pub height_to_interval: EagerVec<Height, Timestamp>,
-    pub height_to_vbytes: EagerVec<Height, StoredU64>,
-    pub difficultyepoch_to_timestamp: EagerVec<DifficultyEpoch, Timestamp>,
-    pub halvingepoch_to_timestamp: EagerVec<HalvingEpoch, Timestamp>,
-    pub timeindexes_to_timestamp: ComputedVecsFromDateIndex<Timestamp>,
-    pub indexes_to_block_count: ComputedVecsFromHeight<StoredU32>,
-    pub indexes_to_block_interval: ComputedVecsFromHeight<Timestamp>,
-    pub indexes_to_block_size: ComputedVecsFromHeight<StoredUsize>,
-    pub indexes_to_block_vbytes: ComputedVecsFromHeight<StoredU64>,
-    pub indexes_to_block_weight: ComputedVecsFromHeight<Weight>,
-}
-
-impl Vecs {
-    pub fn forced_import(
-        path: &Path,
-        version: Version,
-        _computation: Computation,
-        format: Format,
-    ) -> color_eyre::Result<Self> {
-        fs::create_dir_all(path)?;
-
-        Ok(Self {
-            height_to_interval: EagerVec::forced_import(
-                path,
-                "interval",
-                version + VERSION + Version::ZERO,
-                format,
-            )?,
-            timeindexes_to_timestamp: ComputedVecsFromDateIndex::forced_import(
-                path,
-                "timestamp",
-                true,
-                version + VERSION + Version::ZERO,
-                format,
-                StorableVecGeneatorOptions::default().add_first(),
-            )?,
-            indexes_to_block_interval: ComputedVecsFromHeight::forced_import(
-                path,
-                "block_interval",
-                false,
-                version + VERSION + Version::ZERO,
-                format,
-                StorableVecGeneatorOptions::default()
-                    .add_percentiles()
-                    .add_minmax()
-                    .add_average(),
-            )?,
-            indexes_to_block_count: ComputedVecsFromHeight::forced_import(
-                path,
-                "block_count",
-                true,
-                version + VERSION + Version::ZERO,
-                format,
-                StorableVecGeneatorOptions::default()
-                    .add_sum()
-                    .add_cumulative(),
-            )?,
-            indexes_to_block_weight: ComputedVecsFromHeight::forced_import(
-                path,
-                "block_weight",
-                false,
-                version + VERSION + Version::ZERO,
-                format,
-                StorableVecGeneatorOptions::default()
-                    .add_sum()
-                    .add_cumulative(),
-            )?,
-            indexes_to_block_size: ComputedVecsFromHeight::forced_import(
-                path,
-                "block_size",
-                false,
-                version + VERSION + Version::ZERO,
-                format,
-                StorableVecGeneatorOptions::default()
-                    .add_sum()
-                    .add_cumulative(),
-            )?,
-            height_to_vbytes: EagerVec::forced_import(
-                path,
-                "vbytes",
-                version + VERSION + Version::ZERO,
-                format,
-            )?,
-            indexes_to_block_vbytes: ComputedVecsFromHeight::forced_import(
-                path,
-                "block_vbytes",
-                false,
-                version + VERSION + Version::ZERO,
-                format,
-                StorableVecGeneatorOptions::default()
-                    .add_sum()
-                    .add_cumulative(),
-            )?,
-            difficultyepoch_to_timestamp: EagerVec::forced_import(
-                path,
-                "timestamp",
-                version + VERSION + Version::ZERO,
-                format,
-            )?,
-            halvingepoch_to_timestamp: EagerVec::forced_import(
-                path,
-                "timestamp",
-                version + VERSION + Version::ZERO,
-                format,
-            )?,
-        })
-    }
-
-    pub fn compute(
-        &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        starting_indexes: &Indexes,
-        exit: &Exit,
-    ) -> color_eyre::Result<()> {
-        self.timeindexes_to_timestamp.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_transform(
-                    starting_indexes.dateindex,
-                    &indexes.dateindex_to_date,
-                    |(di, d, ..)| (di, Timestamp::from(d)),
-                    exit,
-                )
-            },
-        )?;
-
-        self.indexes_to_block_count.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |v, indexer, _, starting_indexes, exit| {
-                let indexer_vecs = indexer.vecs();
-
-                v.compute_range(
-                    starting_indexes.height,
-                    &indexer_vecs.height_to_weight,
-                    |h| (h, StoredU32::from(1_u32)),
-                    exit,
-                )
-            },
-        )?;
-
-        let indexer_vecs = indexer.vecs();
-
-        let mut height_to_timestamp_iter = indexer_vecs.height_to_timestamp.iter();
-        self.height_to_interval.compute_transform(
-            starting_indexes.height,
-            &indexer_vecs.height_to_timestamp,
-            |(height, timestamp, ..)| {
-                let interval = height.decremented().map_or(Timestamp::ZERO, |prev_h| {
-                    let prev_timestamp = height_to_timestamp_iter.unwrap_get_inner(prev_h);
-                    timestamp
-                        .checked_sub(prev_timestamp)
-                        .unwrap_or(Timestamp::ZERO)
-                });
-                (height, interval)
-            },
-            exit,
-        )?;
-
-        self.indexes_to_block_interval.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&self.height_to_interval),
-        )?;
-
-        self.indexes_to_block_weight.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&indexer_vecs.height_to_weight),
-        )?;
-
-        self.indexes_to_block_size.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&indexer_vecs.height_to_total_size),
-        )?;
-
-        self.height_to_vbytes.compute_transform(
-            starting_indexes.height,
-            &indexer_vecs.height_to_weight,
-            |(h, w, ..)| {
-                (
-                    h,
-                    StoredU64::from(bitcoin::Weight::from(w).to_vbytes_floor()),
-                )
-            },
-            exit,
-        )?;
-
-        self.indexes_to_block_vbytes.compute_rest(
-            indexes,
-            starting_indexes,
-            exit,
-            Some(&self.height_to_vbytes),
-        )?;
-
-        let mut height_to_timestamp_iter = indexer_vecs.height_to_timestamp.iter();
-
-        self.difficultyepoch_to_timestamp.compute_transform(
-            starting_indexes.difficultyepoch,
-            &indexes.difficultyepoch_to_first_height,
-            |(i, h, ..)| (i, height_to_timestamp_iter.unwrap_get_inner(h)),
-            exit,
-        )?;
-
-        self.halvingepoch_to_timestamp.compute_transform(
-            starting_indexes.halvingepoch,
-            &indexes.halvingepoch_to_first_height,
-            |(i, h, ..)| (i, height_to_timestamp_iter.unwrap_get_inner(h)),
-            exit,
-        )?;
-
-        Ok(())
-    }
-
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            vec![
-                &self.height_to_interval as &dyn AnyCollectableVec,
-                &self.height_to_vbytes,
-                &self.difficultyepoch_to_timestamp,
-                &self.halvingepoch_to_timestamp,
-            ],
-            self.timeindexes_to_timestamp.vecs(),
-            self.indexes_to_block_count.vecs(),
-            self.indexes_to_block_interval.vecs(),
-            self.indexes_to_block_size.vecs(),
-            self.indexes_to_block_vbytes.vecs(),
-            self.indexes_to_block_weight.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
-    }
-}

crates/brk_computer/src/vecs/constants.rs

@@ -1,154 +0,0 @@
-use std::{fs, path::Path};
-
-use brk_core::{StoredU8, Version};
-use brk_exit::Exit;
-use brk_indexer::Indexer;
-use brk_vec::{AnyCollectableVec, AnyVec, Computation, Format};
-
-use super::{
-    Indexes,
-    grouped::{ComputedVecsFromHeight, StorableVecGeneatorOptions},
-    indexes,
-};
-
-const VERSION: Version = Version::ZERO;
-
-#[derive(Clone)]
-pub struct Vecs {
-    pub _0: ComputedVecsFromHeight<StoredU8>,
-    pub _1: ComputedVecsFromHeight<StoredU8>,
-    pub _50: ComputedVecsFromHeight<StoredU8>,
-    pub _100: ComputedVecsFromHeight<StoredU8>,
-}
-
-impl Vecs {
-    pub fn forced_import(
-        path: &Path,
-        version: Version,
-        _computation: Computation,
-        format: Format,
-    ) -> color_eyre::Result<Self> {
-        fs::create_dir_all(path)?;
-
-        Ok(Self {
-            _0: ComputedVecsFromHeight::forced_import(
-                path,
-                "0",
-                true,
-                version + VERSION + Version::ZERO,
-                format,
-                StorableVecGeneatorOptions::default().add_last(),
-            )?,
-            _1: ComputedVecsFromHeight::forced_import(
-                path,
-                "1",
-                true,
-                version + VERSION + Version::ZERO,
-                format,
-                StorableVecGeneatorOptions::default().add_last(),
-            )?,
-            _50: ComputedVecsFromHeight::forced_import(
-                path,
-                "50",
-                true,
-                version + VERSION + Version::ZERO,
-                format,
-                StorableVecGeneatorOptions::default().add_last(),
-            )?,
-            _100: ComputedVecsFromHeight::forced_import(
-                path,
-                "100",
-                true,
-                version + VERSION + Version::ZERO,
-                format,
-                StorableVecGeneatorOptions::default().add_last(),
-            )?,
-        })
-    }
-
-    pub fn compute(
-        &mut self,
-        indexer: &Indexer,
-        indexes: &indexes::Vecs,
-        starting_indexes: &Indexes,
-        exit: &Exit,
-    ) -> color_eyre::Result<()> {
-        self._0.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredU8::new(0)),
-                    exit,
-                )
-            },
-        )?;
-
-        self._1.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredU8::new(1)),
-                    exit,
-                )
-            },
-        )?;
-
-        self._50.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredU8::new(50)),
-                    exit,
-                )
-            },
-        )?;
-
-        self._100.compute_all(
-            indexer,
-            indexes,
-            starting_indexes,
-            exit,
-            |vec, _, indexes, starting_indexes, exit| {
-                vec.compute_to(
-                    starting_indexes.height,
-                    indexes.height_to_date.len(),
-                    indexes.height_to_date.version(),
-                    |i| (i, StoredU8::new(100)),
-                    exit,
-                )
-            },
-        )?;
-
-        Ok(())
-    }
-
-    pub fn vecs(&self) -> Vec<&dyn AnyCollectableVec> {
-        [
-            self._0.vecs(),
-            self._1.vecs(),
-            self._50.vecs(),
-            self._100.vecs(),
-        ]
-        .into_iter()
-        .flatten()
-        .collect::<Vec<_>>()
-    }
-}