mirror of
https://github.com/bitcoinresearchkit/brk.git
synced 2026-07-27 18:58:10 -07:00
readmes: regenerated
This commit is contained in:
nym21
+214
-176
@@ -1,227 +1,265 @@
|
||||
# brk_interface
|
||||
|
||||
**Unified data query and formatting interface for Bitcoin datasets**
|
||||
Unified data query and formatting interface for Bitcoin datasets with intelligent search and multi-format output.
|
||||
|
||||
`brk_interface` provides a clean, unified API for accessing Bitcoin datasets from both indexer and computer components. It serves as the primary data access layer powering BRK's web API and MCP endpoints, offering flexible querying, pagination, and multiple output formats.
|
||||
[](https://crates.io/crates/brk_interface)
|
||||
[](https://docs.rs/brk_interface)
|
||||
|
||||
## What it provides
|
||||
## Overview
|
||||
|
||||
- **Unified Data Access**: Single interface to query both indexed blockchain data and computed analytics
|
||||
- **Multiple Output Formats**: JSON, CSV, TSV, and Markdown table formatting
|
||||
- **Flexible Pagination**: Range queries with positive/negative indexing and automatic pagination
|
||||
- **Multi-dataset Queries**: Retrieve multiple datasets with the same time index in one call
|
||||
- **Dynamic Search**: Intelligent ID matching with automatic fallbacks and normalization
|
||||
This crate provides a high-level interface for querying and formatting data from BRK's indexer and computer components. It offers intelligent vector search with fuzzy matching, parameter validation, range queries, and multi-format output (JSON, CSV, TSV, Markdown) with efficient caching and pagination support.
|
||||
|
||||
## Key Features
|
||||
**Key Features:**
|
||||
|
||||
### Query Interface
|
||||
- **25 Time Indices**: From granular (Height, DateIndex) to aggregate (YearIndex, DecadeIndex)
|
||||
- **Bitcoin-Specific Indices**: Address types (P2PKH, P2SH, P2TR), output types, epochs
|
||||
- **Multi-dataset support**: Query multiple related datasets simultaneously
|
||||
- **Intelligent ID resolution**: Flexible matching with automatic fallbacks
|
||||
- Unified query interface across indexer and computer data sources
|
||||
- Intelligent search with fuzzy matching and helpful error messages
|
||||
- Multi-format output: JSON, CSV, TSV, Markdown with proper formatting
|
||||
- Range-based data queries with flexible from/to parameters
|
||||
- Comprehensive pagination support for large datasets
|
||||
- Schema validation with JSON Schema generation for API documentation
|
||||
- Efficient caching system for error messages and repeated queries
|
||||
|
||||
### Pagination and Ranges
|
||||
- **Signed indexing**: Negative indices count from end (`-1` = latest, `-10` = last 10)
|
||||
- **Range queries**: `from`/`to` parameters with optional `count` limit
|
||||
- **Efficient pagination**: 1,000 items per page with proper start/end calculation
|
||||
**Target Use Cases:**
|
||||
|
||||
### Output Formatting
|
||||
- **JSON**: Single values, arrays, or matrices with automatic structure detection
|
||||
- **CSV/TSV**: Delimiter-separated values with headers for spreadsheet use
|
||||
- **Markdown**: Tables formatted for documentation and display
|
||||
- **Schema Support**: JSON Schema generation for API documentation
|
||||
- REST API backends requiring flexible data queries
|
||||
- Data export tools supporting multiple output formats
|
||||
- Interactive applications with user-friendly error messaging
|
||||
- Research platforms requiring structured data access
|
||||
|
||||
## Usage
|
||||
## Installation
|
||||
|
||||
### Basic Query Setup
|
||||
```toml
|
||||
cargo add brk_interface
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
```rust
|
||||
use brk_interface::{Interface, Params, ParamsOpt, Index, Format};
|
||||
use brk_interface::{Interface, Params, Index};
|
||||
use brk_indexer::Indexer;
|
||||
use brk_computer::Computer;
|
||||
|
||||
// Load data sources
|
||||
let indexer = Indexer::forced_import("./brk_data")?;
|
||||
let computer = Computer::forced_import("./brk_data", &indexer, None)?;
|
||||
|
||||
// Create unified interface
|
||||
// Initialize with indexer and computer
|
||||
let indexer = Indexer::build(/* config */)?;
|
||||
let computer = Computer::build(/* config */)?;
|
||||
let interface = Interface::build(&indexer, &computer);
|
||||
```
|
||||
|
||||
### Single Dataset Queries
|
||||
|
||||
```rust
|
||||
// Get latest block data
|
||||
// Query data with parameters
|
||||
let params = Params {
|
||||
index: Index::Height,
|
||||
ids: vec!["date", "timestamp", "difficulty"].into(),
|
||||
rest: ParamsOpt::default()
|
||||
.set_from(-1) // Latest block
|
||||
.set_format(Format::JSON),
|
||||
ids: vec!["height-to-blockhash".to_string()].into(),
|
||||
from: Some(800000),
|
||||
to: Some(800100),
|
||||
format: Some(Format::JSON),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let result = interface.search_and_format(params)?;
|
||||
println!("{}", result);
|
||||
// Search and format results
|
||||
let output = interface.search_and_format(params)?;
|
||||
println!("{}", output);
|
||||
```
|
||||
|
||||
### Range Queries
|
||||
## API Overview
|
||||
|
||||
### Core Types
|
||||
|
||||
- **`Interface<'a>`**: Main query interface coordinating indexer and computer access
|
||||
- **`Params`**: Query parameters including index, IDs, range, and formatting options
|
||||
- **`Index`**: Enumeration of available data indexes (Height, Date, Address, etc.)
|
||||
- **`Format`**: Output format specification (JSON, CSV, TSV, MD)
|
||||
- **`Output`**: Formatted query results with multiple value types
|
||||
|
||||
### Key Methods
|
||||
|
||||
**`Interface::build(indexer: &Indexer, computer: &Computer) -> Self`**
|
||||
Creates interface instance with references to data sources.
|
||||
|
||||
**`search(&self, params: &Params) -> Result<Vec<(String, &&dyn AnyCollectableVec)>>`**
|
||||
Searches for vectors matching the query parameters with intelligent error handling.
|
||||
|
||||
**`format(&self, vecs: Vec<...>, params: &ParamsOpt) -> Result<Output>`**
|
||||
Formats search results according to specified output format and range parameters.
|
||||
|
||||
**`search_and_format(&self, params: Params) -> Result<Output>`**
|
||||
Combined search and formatting operation for single-call data retrieval.
|
||||
|
||||
### Query Parameters
|
||||
|
||||
**Core Parameters:**
|
||||
|
||||
- `index`: Data index to query (height, date, address, etc.)
|
||||
- `ids`: Vector IDs to retrieve from the specified index
|
||||
- `from`/`to`: Optional range filtering (inclusive start, exclusive end)
|
||||
- `format`: Output format (defaults to JSON)
|
||||
|
||||
**Pagination Parameters:**
|
||||
|
||||
- `offset`: Number of entries to skip
|
||||
- `limit`: Maximum entries to return
|
||||
|
||||
## Examples
|
||||
|
||||
### Basic Data Query
|
||||
|
||||
```rust
|
||||
// Get price data for last 30 days
|
||||
let params = Params {
|
||||
index: Index::DateIndex,
|
||||
ids: vec!["price_usd", "price_usd_high", "price_usd_low"].into(),
|
||||
rest: ParamsOpt::default()
|
||||
.set_from(-30) // Last 30 days
|
||||
.set_count(30)
|
||||
.set_format(Format::CSV),
|
||||
};
|
||||
use brk_interface::{Interface, Params, Index, Format};
|
||||
|
||||
let csv_data = interface.search_and_format(params)?;
|
||||
```
|
||||
let interface = Interface::build(&indexer, &computer);
|
||||
|
||||
### Multiple Datasets
|
||||
|
||||
```rust
|
||||
// Get comprehensive block statistics
|
||||
// Query block heights to hashes
|
||||
let params = Params {
|
||||
index: Index::Height,
|
||||
ids: vec!["size", "weight", "tx_count", "fee_total"].into(),
|
||||
rest: ParamsOpt::default()
|
||||
.set_from(800_000) // Starting from block 800,000
|
||||
.set_to(800_100) // Up to block 800,100
|
||||
.set_format(Format::TSV),
|
||||
ids: vec!["height-to-blockhash".to_string()].into(),
|
||||
from: Some(750000),
|
||||
to: Some(750010),
|
||||
format: Some(Format::JSON),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let tsv_data = interface.search_and_format(params)?;
|
||||
```
|
||||
|
||||
### Flexible ID Specification
|
||||
|
||||
```rust
|
||||
// Different ways to specify dataset IDs
|
||||
let params = Params {
|
||||
index: Index::DateIndex,
|
||||
ids: vec!["price_usd,volume_usd,market_cap"].into(), // Comma-separated
|
||||
// OR: ids: vec!["price_usd volume_usd market_cap"].into(), // Space-separated
|
||||
// OR: ids: vec!["price_usd", "volume_usd", "market_cap"].into(), // Array
|
||||
rest: ParamsOpt::default()
|
||||
.set_format(Format::JSON),
|
||||
};
|
||||
```
|
||||
|
||||
### Advanced Queries with Pagination
|
||||
|
||||
```rust
|
||||
// Query with custom pagination
|
||||
let params = Params {
|
||||
index: Index::MonthIndex,
|
||||
ids: vec!["supply_total", "supply_active"].into(),
|
||||
rest: ParamsOpt::default()
|
||||
.set_from(0) // From beginning
|
||||
.set_count(50) // Max 50 results
|
||||
.set_format(Format::Markdown),
|
||||
};
|
||||
|
||||
let markdown_table = interface.search_and_format(params)?;
|
||||
```
|
||||
|
||||
## API Methods
|
||||
|
||||
### Core Query Methods
|
||||
|
||||
```rust
|
||||
// Combined search and format
|
||||
let result = interface.search_and_format(params)?;
|
||||
|
||||
// Separate search and format
|
||||
let vecs = interface.search(params)?;
|
||||
let formatted = interface.format(vecs, params.rest.format.unwrap_or_default())?;
|
||||
|
||||
// Get metadata
|
||||
let current_height = interface.get_height();
|
||||
let available_datasets = interface.get_vecids(None)?; // Paginated
|
||||
let available_indices = interface.get_indexes();
|
||||
```
|
||||
|
||||
### Working with Results
|
||||
|
||||
```rust
|
||||
use brk_interface::{Value, Output};
|
||||
|
||||
// Handle different output types
|
||||
match interface.search_and_format(params)? {
|
||||
Output::Json(Value::Single(val)) => println!("Single value: {}", val),
|
||||
Output::Json(Value::List(arr)) => println!("Array with {} items", arr.len()),
|
||||
Output::Json(Value::Matrix(matrix)) => println!("Matrix: {}x{}", matrix.len(), matrix[0].len()),
|
||||
Output::Delimited(csv_data) => println!("CSV/TSV data:\n{}", csv_data),
|
||||
Output::Markdown(table) => println!("Markdown table:\n{}", table),
|
||||
Output::Json(value) => println!("{}", serde_json::to_string_pretty(&value)?),
|
||||
_ => unreachable!(),
|
||||
}
|
||||
```
|
||||
|
||||
## Available Indices
|
||||
|
||||
### Time-based Indices
|
||||
- `Height` - Block height (0, 1, 2, ...)
|
||||
- `DateIndex` - Days since Bitcoin genesis
|
||||
- `WeekIndex`, `MonthIndex`, `QuarterIndex`, `YearIndex`, `DecadeIndex`
|
||||
- `HalvingEpoch`, `DifficultyEpoch` - Bitcoin-specific epochs
|
||||
|
||||
### Bitcoin-specific Indices
|
||||
- Address types: `P2PKHAddressIndex`, `P2SHAddressIndex`, `P2WPKHAddressIndex`, etc.
|
||||
- Output types: `OutputType` classifications
|
||||
- Transaction types: `TxIndex`, `InputIndex`, `OutputIndex`
|
||||
|
||||
## Parameter Reference
|
||||
### CSV Export with Range Query
|
||||
|
||||
```rust
|
||||
pub struct Params {
|
||||
pub index: Index, // Time dimension for query
|
||||
pub ids: MaybeIds, // Dataset identifiers
|
||||
pub rest: ParamsOpt, // Optional parameters
|
||||
}
|
||||
use brk_interface::{Interface, Params, Index, Format};
|
||||
|
||||
pub struct ParamsOpt {
|
||||
pub from: Option<i64>, // Starting index (negative = from end)
|
||||
pub to: Option<i64>, // Ending index (exclusive)
|
||||
pub count: Option<usize>, // Maximum results
|
||||
pub format: Option<Format>, // Output format
|
||||
// Export price data as CSV
|
||||
let params = Params {
|
||||
index: Index::Date,
|
||||
ids: vec!["dateindex-to-price-close".to_string()].into(),
|
||||
from: Some(0), // From genesis
|
||||
to: Some(5000), // First ~13 years
|
||||
format: Some(Format::CSV),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
match interface.search_and_format(params)? {
|
||||
Output::CSV(csv_text) => {
|
||||
std::fs::write("bitcoin_prices.csv", csv_text)?;
|
||||
println!("Price data exported to bitcoin_prices.csv");
|
||||
},
|
||||
_ => unreachable!(),
|
||||
}
|
||||
```
|
||||
|
||||
## Output Formats
|
||||
|
||||
- **JSON**: Structured data (single values, arrays, matrices)
|
||||
- **CSV**: Comma-separated values with headers
|
||||
- **TSV**: Tab-separated values with headers
|
||||
- **Markdown**: Formatted tables for documentation
|
||||
|
||||
## Performance Features
|
||||
|
||||
- **Zero-copy operations**: Uses references and lifetimes to avoid cloning
|
||||
- **Memory mapping**: Leverages vecdb's memory-mapped storage
|
||||
- **Lazy evaluation**: Only processes data when formatting is requested
|
||||
- **Efficient pagination**: Smart bounds calculation and range queries
|
||||
|
||||
## Schema Support
|
||||
|
||||
The interface provides JSON Schema support for API documentation:
|
||||
### Multi-Vector Query with Markdown Table
|
||||
|
||||
```rust
|
||||
use schemars::schema_for;
|
||||
use brk_interface::{Interface, Params, Index, Format};
|
||||
|
||||
let schema = schema_for!(Params);
|
||||
// Use schema for API documentation
|
||||
// Query multiple vectors and format as table
|
||||
let params = Params {
|
||||
index: Index::Height,
|
||||
ids: vec![
|
||||
"height-to-blockhash".to_string(),
|
||||
"height-to-timestamp".to_string(),
|
||||
"height-to-difficulty".to_string()
|
||||
].into(),
|
||||
from: Some(800000),
|
||||
to: Some(800005),
|
||||
format: Some(Format::MD),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
match interface.search_and_format(params)? {
|
||||
Output::MD(table) => println!("{}", table),
|
||||
_ => unreachable!(),
|
||||
}
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
### Intelligent Error Handling
|
||||
|
||||
- `brk_indexer` - Indexed blockchain data source
|
||||
- `brk_computer` - Computed analytics data source
|
||||
- `vecdb` - Vector database with collection traits
|
||||
- `serde` - Serialization/deserialization support
|
||||
- `schemars` - JSON Schema generation
|
||||
```rust
|
||||
use brk_interface::{Interface, Params, Index};
|
||||
|
||||
// Query with typo in vector ID
|
||||
let params = Params {
|
||||
index: Index::Height,
|
||||
ids: vec!["height-to-blockhas".to_string()].into(), // Typo: "blockhas"
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
// Interface provides helpful error with suggestions
|
||||
match interface.search(¶ms) {
|
||||
Err(error) => {
|
||||
println!("{}", error);
|
||||
// Output: No vec named "height-to-blockhas" indexed by "height" found.
|
||||
// Maybe you meant one of the following: ["height-to-blockhash"] ?
|
||||
},
|
||||
Ok(_) => unreachable!(),
|
||||
}
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
### Data Source Integration
|
||||
|
||||
The interface acts as a bridge between:
|
||||
|
||||
- **Indexer**: Raw blockchain data vectors (blocks, transactions, addresses)
|
||||
- **Computer**: Computed analytics vectors (prices, statistics, aggregations)
|
||||
- **Unified Access**: Single query interface for both data sources
|
||||
|
||||
### Search Implementation
|
||||
|
||||
1. **Parameter Validation**: Validates index existence and parameter consistency
|
||||
2. **Vector Resolution**: Maps vector IDs to actual data structures
|
||||
3. **Fuzzy Matching**: Provides suggestions for mistyped vector names
|
||||
4. **Error Caching**: Caches error messages to avoid repeated expensive operations
|
||||
|
||||
### Output Formatting
|
||||
|
||||
**JSON Output:**
|
||||
|
||||
- Single value: Direct value serialization
|
||||
- List: Array of values
|
||||
- Matrix: Array of arrays for multi-vector queries
|
||||
|
||||
**Tabular Output (CSV/TSV/MD):**
|
||||
|
||||
- Column headers from vector IDs
|
||||
- Row-wise data iteration with proper escaping
|
||||
- Markdown tables use `tabled` crate formatting
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
- **Error Message Caching**: 1000-entry LRU cache for error messages
|
||||
- **Search Result Caching**: Upstream caching in server/client layers
|
||||
- **Static Data Caching**: Index and vector metadata cached during initialization
|
||||
|
||||
## Configuration
|
||||
|
||||
### Index Types
|
||||
|
||||
Available indexes include:
|
||||
|
||||
- `Height`: Block height-based indexing
|
||||
- `Date`: Calendar date indexing
|
||||
- `Address`: Bitcoin address indexing
|
||||
- `Transaction`: Transaction hash indexing
|
||||
- Custom indexes from computer analytics
|
||||
|
||||
### Format Options
|
||||
|
||||
- **JSON**: Structured data with nested objects/arrays
|
||||
- **CSV**: Comma-separated values with proper escaping
|
||||
- **TSV**: Tab-separated values for import compatibility
|
||||
- **MD**: Markdown tables for documentation and reports
|
||||
|
||||
## Code Analysis Summary
|
||||
|
||||
**Main Structure**: `Interface` struct coordinating between `Indexer` and `Computer` data sources \
|
||||
**Query System**: Parameter-driven search with `Params` struct supporting range queries and formatting options \
|
||||
**Error Handling**: Intelligent fuzzy matching with cached error messages and helpful suggestions \
|
||||
**Output Formats**: Multi-format support (JSON, CSV, TSV, Markdown) with proper data serialization \
|
||||
**Caching**: `quick_cache` integration for error messages and expensive operations \
|
||||
**Search Logic**: `nucleo-matcher` fuzzy search for user-friendly vector name resolution \
|
||||
**Architecture**: Abstraction layer providing unified access to heterogeneous Bitcoin data sources
|
||||
|
||||
---
|
||||
|
||||
*This README was generated by Claude Code*
|
||||
_This README was generated by Claude Code_
|
||||
|
||||
Reference in New Issue
Block a user