diff --git a/.gitignore b/.gitignore index af98ef201..da713f859 100644 --- a/.gitignore +++ b/.gitignore @@ -1,25 +1,9 @@ # Mac OS .DS_Store -# To do -/charts -TODO.md - # Builds -dist target -# I/O -in -out -.log -/datasets -/datasets2 -/price -*..* -/txout_* -/db - # Sync .stfolder @@ -29,16 +13,10 @@ out # Ignored ignore -# Scripts -/start-node.sh - # Editors .vscode .zed -# Configs -config.toml - # Flamegraph flamegraph/ flamegraph.svg @@ -53,7 +31,7 @@ snapshots*/ docker/kibo # Types -website/scripts/types/paths.d.ts +paths.d.ts -# Misc -OPENSATS.md +# Git +.git* diff --git a/CHANGELOG.md b/CHANGELOG.md index 3f703c65c..97b4a75e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,13 +1,20 @@ -# Changelog - -## v. 0.5.0 | [873199](https://mempool.space/block/0000000000000000000270925aa6a565be92e13164565a3f7994ca1966e48050) - 2024/12/04 +# v0.6.0 | WIP -![Image of the kibō Web App version 0.5.0](./assets/v0.5.0.jpg) +- Merged parser and server crates into a single project (and thus executable) +- Started using `log` and `env_logger` crates +- Improved logs +- Added `--server BOOL` and `--parser BOOL` parameters (both are true by default) +- Automated databases defragmention (and removed parameter) +- Fixed input being unfocused right after being focused in Brave browser + +# [v0.5.0](https://github.com/kibo-money/kibo/tree/eea56d394bf92c62c81da8b78b8c47ea730683f5) | [873199](https://mempool.space/block/0000000000000000000270925aa6a565be92e13164565a3f7994ca1966e48050) - 2024/12/04 + +![Image of the kibō Web App version 0.5.0](https://github.com/kibo-money/kibo/blob/main/assets/v0.5.0.jpg) ## Datasets @@ -72,15 +79,15 @@ - Moved back to this repo -## v. 0.4.0 | [861950](https://mempool.space/block/00000000000000000000530d0e30ccf7deeace122dcc99f2668a06c6dad83629) - 2024/09/19 +# [v0.4.0](https://github.com/kibo-money/kibo/tree/a64c544815d9ef785e2fc1323582f774f16b9200) | [861950](https://mempool.space/block/00000000000000000000530d0e30ccf7deeace122dcc99f2668a06c6dad83629) - 2024/09/19 -![Image of the kibō Web App version 0.4.0](./assets/v0.4.0.jpg) +![Image of the kibō Web App version 0.4.0](https://github.com/kibo-money/kibo/blob/main/assets/v0.4.0.jpg) -### Brand +## Brand - **Satonomics** is now **kibō** 🎉 -### Website +## Website - Complete redesign of the website - Rewrote the whole application and removed `node`/`npm`/`pnpm` dependencies in favor for pure `HTML`/`CSS`/`Javascript` @@ -88,7 +95,7 @@ - Added Trading View attribution link to the settings frame and file in the lightweight charts folder - Many other changes -### Parser +## Parser - Changed the block iterator from a custom version of [bitcoin-explorer](https://crates.io/crates/bitcoin-explorer) to the homemade [biter](https://crates.io/crates/biter) which allows the parser to run alongside `bitcoind` - Added datasets compression thanks to [zstd](https://crates.io/crates/zstd) to reduce disk usage @@ -103,17 +110,17 @@ - Various first run fixes - Added to `-h` which arguments are saved, which is all of them at the time of writing -### Server +## Server - Updated the code to support compressed binaries - Added serving of the website - Improved `Cache-Control` behavior -## v. 0.3.0 | [853930](https://mempool.space/block/00000000000000000002eb5e9a7950ca2d5d98bd1ed28fc9098aa630d417985d) - 2024/07/26 +# [v0.3.0](https://github.com/kibo-money/kibo/tree/b68b016091c45b071218fba01bac5b76e8eaf18c) | [853930](https://mempool.space/block/00000000000000000002eb5e9a7950ca2d5d98bd1ed28fc9098aa630d417985d) - 2024/07/26 -![Image of the Satonomics Web App version 0.3.0](./assets/v0.3.0.jpg) +![Image of the Satonomics Web App version 0.3.0](https://github.com/kibo-money/kibo/blob/main/assets/v0.3.0.jpg) -### Parser +## Parser - Global - Improved self-hosting by: @@ -156,7 +163,7 @@ - Price - Improved error message when price cannot be found -### App +## App - General - Added chart scroll button for nice animations à la Wicked @@ -182,17 +189,17 @@ - Settings - Removed the horizontal scroll bar which was unintended -### Server +## Server - Run file - Only run with a watcher if `cargo watch` is available - Removed id_to_path file in favor for only `paths.d.ts` in `app/src/types` -## v. 0.2.0 | [851286](https://mempool.space/block/0000000000000000000281ca7f1bf8c50702bfca168c7af1bdc67c977c1ac8ed) - 2024/07/08 +# [v0.2.0](https://github.com/kibo-money/kibo/tree/248187889283597c5dbb806292297453c25e97b8) | [851286](https://mempool.space/block/0000000000000000000281ca7f1bf8c50702bfca168c7af1bdc67c977c1ac8ed) - 2024/07/08 -![Image of the Satonomics Web App version 0.2.0](./assets/v0.2.0.jpg) +![Image of the Satonomics Web App version 0.2.0](https://github.com/kibo-money/kibo/blob/main/assets/v0.2.0.jpg) -### App +## App - General - Added the height version of all datasets and many optimizations to make them usable but only available on desktop and tablets for now @@ -220,24 +227,24 @@ - Hopefully made scrollbars a little more subtle on WIndows and Linux, can't test - Generale style updates -### Parser +## Parser - Fixed ulimit only being run in Mac OS instead of whenever the program is detected -## v. 0.1.1 | [849240](https://mempool.space/block/000000000000000000002b8653988655071c07bb5f7181c038f9326bc86db741) - 2024/06/24 +# [v0.1.1](https://github.com/kibo-money/kibo/tree/e55b5195a9de9aea306903c94ed63cb1720fda5f) | [849240](https://mempool.space/block/000000000000000000002b8653988655071c07bb5f7181c038f9326bc86db741) - 2024/06/24 -![Image of the Satonomics Web App version 0.1.1](./assets/v0.1.1.jpg) +![Image of the Satonomics Web App version 0.1.1](https://github.com/kibo-money/kibo/blob/main/assets/v0.1.1.jpg) -### Parser +## Parser - Fixed overflow in `Price` struct which caused many Realized Caps and Realized Prices to have completely bogus data - Fixed Realized Cap computation which was using rounded prices instead normal ones -### Server +## Server - Added the chunk, date and time of the request to the terminal logs -### App +## App - Chart - Added double click option on a legend to toggle the visibility of all other series @@ -270,14 +277,14 @@ - Misc - Removed tracker even though it was a very privacy friendly as it appeared to not be working properly -### Price +## Price - Deleted old price datasets and their backups -## v. 0.1.0 | [848642](https://mempool.space/block/000000000000000000020be5761d70751252219a9557f55e91ecdfb86c4e026a) - 2024/06/19 +# [v0.1.0](https://github.com/kibo-money/kibo/tree/a1a576d088c8f83ed32d48753a7611f70a964574) | [848642](https://mempool.space/block/000000000000000000020be5761d70751252219a9557f55e91ecdfb86c4e026a) - 2024/06/19 -![Image of the Satonomics Web App version 0.1.0](./assets/v0.1.0.jpg) +![Image of the Satonomics Web App version 0.1.0](https://github.com/kibo-money/kibo/blob/main/assets/v0.1.0.jpg) -## v. 0.0.X | [835444](https://mempool.space/block/000000000000000000009f93907a0dd83c080d5585cc7ec82c076d45f6d7c872) - 2024/03/20 +# v0.0.1 | [835444](https://mempool.space/block/000000000000000000009f93907a0dd83c080d5585cc7ec82c076d45f6d7c872) - 2024/03/20 -![Image of the Satonomics Web App version 0.0.X](./assets/v0.0.X.jpg) +![Image of the Satonomics Web App version 0.0.X](https://github.com/kibo-money/kibo/blob/main/assets/v0.0.X.jpg) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a6c0043c3..e23616908 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -3,6 +3,6 @@ ## Parser - Avoid floats as much as possible - - Use structs like `WAmount` and `Price` for calculations - - **Only** use `WAmount.to_btc()` when inserting or computing inside a dataset. It is **very** expensive. + - Use structs like `Amount` and `Price` for calculations + - **Only** use `Amount.to_btc()` when inserting or computing inside a dataset. It is **very** expensive. - No `Arc`, `Rc`, `Mutex` even from third party libraries, they're slower diff --git a/server/Cargo.lock b/Cargo.lock similarity index 90% rename from server/Cargo.lock rename to Cargo.lock index 540a98d4f..cd0f6c81f 100644 --- a/server/Cargo.lock +++ b/Cargo.lock @@ -1,6 +1,6 @@ # This file is automatically @generated by Cargo. # It is not intended for manual editing. -version = 3 +version = 4 [[package]] name = "Inflector" @@ -94,9 +94,9 @@ dependencies = [ [[package]] name = "allocator-api2" -version = "0.2.18" +version = "0.2.21" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c6cb57a04249c6480766f7f7cef5467412af1490f8d1e243141daddada3264f" +checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923" [[package]] name = "android-tzdata" @@ -164,9 +164,9 @@ dependencies = [ [[package]] name = "anyhow" -version = "1.0.89" +version = "1.0.94" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "86fdf8605db99b54d3cd748a44c6d04df638eb5dafb219b135d0149bd0db01f6" +checksum = "c1fd03a028ef38ba2276dce7e33fcd6369c158a1bca17946c4b1b701891c1ff7" [[package]] name = "arrayvec" @@ -176,9 +176,9 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "ast_node" -version = "1.0.0" +version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f4d23a6d1d5f18bdbc06d9aa908880e5f49205156ba804751af731c51f5cf81a" +checksum = "91fb5864e2f5bf9fd9797b94b2dfd1554d4c3092b535008b27d7e15c86675a2f" dependencies = [ "proc-macro2", "quote", @@ -188,9 +188,9 @@ dependencies = [ [[package]] name = "async-compression" -version = "0.4.13" +version = "0.4.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e614738943d3f68c628ae3dbce7c3daffb196665f82f8c8ea6b65de73c79429" +checksum = "df895a515f70646414f4b45c0b79082783b80552b373a68283012928df56f522" dependencies = [ "brotli", "flate2", @@ -238,9 +238,9 @@ checksum = "ace50bade8e6234aa140d9a2f552bbee1db4d353f69b8217bc503490fc1a9f26" [[package]] name = "axum" -version = "0.7.7" +version = "0.7.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "504e3947307ac8326a5437504c517c4b56716c9d98fac0028c2acc7ca47d70ae" +checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f" dependencies = [ "async-trait", "axum-core", @@ -312,8 +312,8 @@ version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2c8d66485a3a2ea485c1913c4572ce0256067a5377ac8c75c4960e1cda98605f" dependencies = [ - "bitcoin-internals", - "bitcoin_hashes", + "bitcoin-internals 0.3.0", + "bitcoin_hashes 0.14.0", ] [[package]] @@ -351,9 +351,9 @@ checksum = "d965446196e3b7decd44aa7ee49e31d630118f90ef12f97900f262eb915c951d" [[package]] name = "better_scoped_tls" -version = "0.1.1" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "794edcc9b3fb07bb4aecaa11f093fd45663b4feadb782d68303a2268bc2701de" +checksum = "50fd297a11c709be8348aec039c8b91de16075d2b2bdaee1bd562c0875993664" dependencies = [ "scoped-tls", ] @@ -378,17 +378,17 @@ dependencies = [ [[package]] name = "bitcoin" -version = "0.32.3" +version = "0.32.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0032b0e8ead7074cda7fc4f034409607e3f03a6f71d66ade8a307f79b4d99e73" +checksum = "ce6bc65742dea50536e35ad42492b234c27904a27f0abdcbce605015cb4ea026" dependencies = [ "base58ck", "bech32", - "bitcoin-internals", - "bitcoin-io", + "bitcoin-internals 0.3.0", + "bitcoin-io 0.1.2", "bitcoin-units", - "bitcoin_hashes", - "hex-conservative", + "bitcoin_hashes 0.14.0", + "hex-conservative 0.2.1", "hex_lit", "secp256k1", "serde", @@ -403,19 +403,34 @@ dependencies = [ "serde", ] +[[package]] +name = "bitcoin-internals" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b854212e29b96c8f0fe04cab11d57586c8f3257de0d146c76cb3b42b3eb9118" + [[package]] name = "bitcoin-io" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "340e09e8399c7bd8912f495af6aa58bea0c9214773417ffaa8f6460f93aaee56" +[[package]] +name = "bitcoin-io" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26792cd2bf245069a1c5acb06aa7ad7abe1de69b507c90b490bca81e0665d0ee" +dependencies = [ + "bitcoin-internals 0.4.0", +] + [[package]] name = "bitcoin-units" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5285c8bcaa25876d07f37e3d30c303f2609179716e11d688f51e8f1fe70063e2" dependencies = [ - "bitcoin-internals", + "bitcoin-internals 0.3.0", "serde", ] @@ -425,11 +440,21 @@ version = "0.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bb18c03d0db0247e147a21a6faafd5a7eb851c743db062de72018b6b7e8e4d16" dependencies = [ - "bitcoin-io", - "hex-conservative", + "bitcoin-io 0.1.2", + "hex-conservative 0.2.1", "serde", ] +[[package]] +name = "bitcoin_hashes" +version = "0.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e0982261c82a50d89d1a411602afee0498b3e0debe3d36693f0c661352809639" +dependencies = [ + "bitcoin-io 0.2.0", + "hex-conservative 0.3.0", +] + [[package]] name = "bitcoincore-rpc" version = "0.19.0" @@ -456,7 +481,7 @@ dependencies = [ [[package]] name = "biter" -version = "0.1.1" +version = "0.2.1" dependencies = [ "bitcoin", "bitcoincore-rpc", @@ -591,9 +616,9 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" [[package]] name = "chrono" -version = "0.4.38" +version = "0.4.39" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a21f936df1771bf62b77f047b726c4625ff2e8aa607c01ec06e5a05bd8463401" +checksum = "7e36cc9d416881d2e24f9a963be5fb1cd90966419ac844274161d10488b3e825" dependencies = [ "android-tzdata", "iana-time-zone", @@ -606,9 +631,9 @@ dependencies = [ [[package]] name = "clap" -version = "4.5.20" +version = "4.5.23" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b97f376d85a664d5837dbae44bf546e6477a679ff6610010f17276f686d867e8" +checksum = "3135e7ec2ef7b10c6ed8950f0f792ed96ee093fa088608f1c76e569722700c84" dependencies = [ "clap_builder", "clap_derive", @@ -616,9 +641,9 @@ dependencies = [ [[package]] name = "clap_builder" -version = "4.5.20" +version = "4.5.23" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19bc80abd44e4bed93ca373a0704ccbd1b710dc5749406201bb018272808dc54" +checksum = "30582fc632330df2bd26877bde0c1f4470d57c582bbc070376afcd04d8cb4838" dependencies = [ "anstream", "anstyle", @@ -640,9 +665,9 @@ dependencies = [ [[package]] name = "clap_lex" -version = "0.7.2" +version = "0.7.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1462739cb27611015575c0c11df5df7601141071f07518d56fcc1be504cbec97" +checksum = "f46ad14479a25103f283c0f10005961cf086d8dc42205bb44c46ac563475dca6" [[package]] name = "color-eyre" @@ -695,9 +720,9 @@ checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" [[package]] name = "cpufeatures" -version = "0.2.14" +version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "608697df725056feaccfa42cffdaeeec3fccc4ffc38358ecd19b243e716a78e0" +checksum = "16b80225097f2e5ae4e7179dd2266824648f3e2f49d9134d584b76389d31c4c3" dependencies = [ "libc", ] @@ -894,6 +919,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4f2c92ceda6ceec50f43169f9ee8424fe2db276791afde7b2cd8bc084cb376ab" dependencies = [ "log", + "regex", ] [[package]] @@ -902,7 +928,10 @@ version = "0.11.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e13fa619b91fb2381732789fc5de83b45675e882f66623b7d8cb4f643017018d" dependencies = [ + "anstream", + "anstyle", "env_filter", + "humantime", "log", ] @@ -946,9 +975,9 @@ checksum = "0ce7134b9999ecaf8bcd65542e436736ef32ddca1b3e06094cb6ec5755203b80" [[package]] name = "flate2" -version = "1.0.34" +version = "1.0.35" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1b589b4dc103969ad3cf85c950899926ec64300a1a46d76c03a6072957036f0" +checksum = "c936bfdafb507ebbf50b8074c54fa31c5be9a1e7e5f467dd659697041407d07c" dependencies = [ "crc32fast", "miniz_oxide 0.8.0", @@ -986,9 +1015,9 @@ dependencies = [ [[package]] name = "from_variant" -version = "1.0.0" +version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e8d8947525c49c73130b5a7187b55b027b6b78fe60268d9f4c283ed690698cb1" +checksum = "8d7ccf961415e7aa17ef93dcb6c2441faaa8e768abe09e659b908089546f74c5" dependencies = [ "proc-macro2", "swc_macros_common", @@ -1144,12 +1173,6 @@ version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d231dfb89cfffdbc30e7fc41579ed6066ad03abda9e567ccafae602b97ec5024" -[[package]] -name = "hermit-abi" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fbf6a919d6cf397374f7dfeeea91d974c7c0a7221d0d0f4f20d859d329e53fcc" - [[package]] name = "hex-conservative" version = "0.2.1" @@ -1159,6 +1182,15 @@ dependencies = [ "arrayvec", ] +[[package]] +name = "hex-conservative" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4afe881d0527571892c4034822e59bb10c6c991cce6abe8199b6f5cf10766f55" +dependencies = [ + "arrayvec", +] + [[package]] name = "hex_lit" version = "0.1.1" @@ -1225,6 +1257,12 @@ version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" +[[package]] +name = "humantime" +version = "2.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a3a5bfb195931eeb336b2a7b4d761daec841b97f947d34394601737a7bba5e4" + [[package]] name = "hyper" version = "1.4.1" @@ -1356,9 +1394,9 @@ dependencies = [ [[package]] name = "inferno" -version = "0.11.21" +version = "0.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "232929e1d75fe899576a3d5c7416ad0d88dbfbb3c3d6aa00873a7408a50ddb88" +checksum = "75a5d75fee4d36809e6b021e4b96b686e763d365ffdb03af2bd00786353f84fe" dependencies = [ "ahash", "clap", @@ -1367,7 +1405,6 @@ dependencies = [ "dashmap 6.1.0", "env_logger", "indexmap", - "is-terminal", "itoa", "log", "num-format", @@ -1394,27 +1431,16 @@ checksum = "ddc24109865250148c2e0f3d25d4f0f479571723792d3802153c60922a4fb708" [[package]] name = "is-macro" -version = "0.3.6" +version = "0.3.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2069faacbe981460232f880d26bf3c7634e322d49053aa48c27e3ae642f728f1" +checksum = "1d57a3e447e24c22647738e4607f1df1e0ec6f72e16182c4cd199f647cdfb0e4" dependencies = [ - "Inflector", + "heck", "proc-macro2", "quote", "syn 2.0.85", ] -[[package]] -name = "is-terminal" -version = "0.4.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "261f68e344040fbd0edea105bef17c66edf46f984ddb1115b775ce31be948f4b" -dependencies = [ - "hermit-abi 0.4.0", - "libc", - "windows-sys 0.52.0", -] - [[package]] name = "is_terminal_polyfill" version = "1.70.1" @@ -1475,6 +1501,41 @@ dependencies = [ "serde_json", ] +[[package]] +name = "kibo_money" +version = "0.6.0" +dependencies = [ + "allocative", + "axum", + "bincode", + "bitcoin_hashes 0.15.0", + "biter", + "chrono", + "clap", + "color-eyre", + "ctrlc", + "derive_deref", + "env_logger", + "inferno", + "itertools", + "log", + "ordered-float", + "rayon", + "regex", + "reqwest", + "rlimit", + "sanakirja", + "serde", + "serde_json", + "struct_iterable", + "swc", + "swc_common", + "tokio", + "toml", + "tower-http", + "zstd", +] + [[package]] name = "lazy_static" version = "1.5.0" @@ -1508,6 +1569,9 @@ name = "log" version = "0.4.22" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a7a70ba024b9dc04c27ea2f0c0548feb474ec5c54bba33a7f72f873a39d07b24" +dependencies = [ + "serde", +] [[package]] name = "lru" @@ -1541,9 +1605,9 @@ dependencies = [ [[package]] name = "miette" -version = "7.2.0" +version = "7.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4edc8853320c2a0dab800fbda86253c8938f6ea88510dc92c5f1ed20e794afc1" +checksum = "317f146e2eb7021892722af37cf1b971f0a70c8406f487e24952667616192c64" dependencies = [ "cfg-if", "miette-derive", @@ -1555,9 +1619,9 @@ dependencies = [ [[package]] name = "miette-derive" -version = "7.2.0" +version = "7.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dcf09caffaac8068c346b6df2a7fc27a177fd20b39421a39ce0a211bde679a6c" +checksum = "23c9b935fbe1d6cbd1dac857b54a688145e2d93f48db36010514d0f612d0ad67" dependencies = [ "proc-macro2", "quote", @@ -1611,7 +1675,7 @@ version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "80e04d1dcff3aae0704555fe5fee3bcfaf3d1fdf8a7e521d5b9d2b42acb52cec" dependencies = [ - "hermit-abi 0.3.9", + "hermit-abi", "libc", "wasi", "windows-sys 0.52.0", @@ -1716,7 +1780,7 @@ version = "1.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4161fcb6d602d4d2081af7c3a45852d875a03dd337a6bfdd6e06407b61342a43" dependencies = [ - "hermit-abi 0.3.9", + "hermit-abi", "libc", ] @@ -1781,9 +1845,9 @@ dependencies = [ [[package]] name = "ordered-float" -version = "4.4.0" +version = "4.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83e7ccb95e240b7c9506a3d544f10d935e142cc90b0a1d56954fb44d89ad6b97" +checksum = "c65ee1f9701bf938026630b455d5315f490640234259037edb259798b3bcf85e" dependencies = [ "num-traits", ] @@ -1854,32 +1918,6 @@ dependencies = [ "windows-targets", ] -[[package]] -name = "parser" -version = "0.5.0" -dependencies = [ - "allocative", - "bincode", - "bitcoin_hashes", - "biter", - "chrono", - "clap", - "color-eyre", - "ctrlc", - "derive_deref", - "inferno", - "itertools", - "ordered-float", - "rayon", - "reqwest", - "sanakirja", - "serde", - "serde_json", - "struct_iterable", - "toml", - "zstd", -] - [[package]] name = "path-clean" version = "0.1.0" @@ -1894,9 +1932,9 @@ checksum = "17359afc20d7ab31fdb42bb844c8b3bb1dabd7dcf7e68428492da7f16966fcef" [[package]] name = "pathdiff" -version = "0.2.1" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8835116a5c179084a830efb3adc117ab007512b535bc1a21c991d3b32a6b44dd" +checksum = "df94ce210e5bc13cb6651479fa48d14f601d9858cfe0467f43ae157023b938d3" [[package]] name = "percent-encoding" @@ -1985,9 +2023,9 @@ dependencies = [ [[package]] name = "preset_env_base" -version = "0.6.0" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1aaeaf4147a07c6615d43fce02d8139e1f835e6ee53e69db2a4810823c138c26" +checksum = "7c8a797e42c09d55157424ac6e9b6e9e5843fc68b887691b280b055e8c3ca5e4" dependencies = [ "ahash", "anyhow", @@ -2012,38 +2050,38 @@ dependencies = [ [[package]] name = "psm" -version = "0.1.23" +version = "0.1.24" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aa37f80ca58604976033fae9515a8a2989fc13797d953f7c04fb8fa36a11f205" +checksum = "200b9ff220857e53e184257720a14553b2f4aa02577d2ed9842d45d4b9654810" dependencies = [ "cc", ] [[package]] name = "ptr_meta" -version = "0.1.4" +version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0738ccf7ea06b608c10564b31debd4f5bc5e197fc8bfe088f68ae5ce81e7a4f1" +checksum = "fe9e76f66d3f9606f44e45598d155cb13ecf09f4a28199e48daf8c8fc937ea90" dependencies = [ "ptr_meta_derive", ] [[package]] name = "ptr_meta_derive" -version = "0.1.4" +version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "16b845dbfca988fa33db069c0e230574d15a3088f147a87b64c7589eb662c9ac" +checksum = "ca414edb151b4c8d125c12566ab0d74dc9cdba36fb80eb7b848c15f495fd32d1" dependencies = [ "proc-macro2", "quote", - "syn 1.0.109", + "syn 2.0.85", ] [[package]] name = "quick-xml" -version = "0.26.0" +version = "0.37.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f50b1c63b38611e7d4d7f68b82d3ad0cc71a2ad2e7f61fc10f1328d917c93cd" +checksum = "f22f29bdff3987b4d8632ef95fd6424ec7e4e0a57e2f4fc63e489e75357f6a03" dependencies = [ "memchr", ] @@ -2139,9 +2177,9 @@ dependencies = [ [[package]] name = "regex" -version = "1.11.0" +version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "38200e5ee88914975b69f657f0801b6f6dccafd44fd9326302a4aaeecfacb1d8" +checksum = "b544ef1b4eac5dc2db33ea63606ae9ffcfac26c1416a2806ae0bf5f56b201191" dependencies = [ "aho-corasick", "memchr", @@ -2151,9 +2189,9 @@ dependencies = [ [[package]] name = "regex-automata" -version = "0.4.8" +version = "0.4.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "368758f23274712b504848e9d5a6f010445cc8b87a7cdb4d7cbee666c1288da3" +checksum = "809e8dc61f6de73b46c85f4c96486310fe304c434cfa43669d7b40f711150908" dependencies = [ "aho-corasick", "memchr", @@ -2234,6 +2272,15 @@ dependencies = [ "windows-sys 0.52.0", ] +[[package]] +name = "rlimit" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7043b63bd0cd1aaa628e476b80e6d4023a3b50eb32789f2728908107bd0c793a" +dependencies = [ + "libc", +] + [[package]] name = "rustc-demangle" version = "0.1.24" @@ -2309,9 +2356,9 @@ dependencies = [ [[package]] name = "rustversion" -version = "1.0.17" +version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "955d28af4278de8121b7ebeb796b6a45735dc01436d898801014aced2773a3d6" +checksum = "0e819f2bc632f285be6d7cd36e25940d45b2391dd6d9b939e79de557f7014248" [[package]] name = "ryu" @@ -2373,7 +2420,7 @@ version = "0.29.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9465315bc9d4566e1724f0fffcbcc446268cb522e60f9a27bcded6b19c108113" dependencies = [ - "bitcoin_hashes", + "bitcoin_hashes 0.14.0", "rand", "secp256k1-sys", "serde", @@ -2437,18 +2484,18 @@ checksum = "388a1df253eca08550bef6c72392cfe7c30914bf41df5269b68cbd6ff8f570a3" [[package]] name = "serde" -version = "1.0.214" +version = "1.0.216" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f55c3193aca71c12ad7890f1785d2b73e1b9f63a0bbc353c08ef26fe03fc56b5" +checksum = "0b9781016e935a97e8beecf0c933758c97a5520d32930e460142b4cd80c6338e" dependencies = [ "serde_derive", ] [[package]] name = "serde_derive" -version = "1.0.214" +version = "1.0.216" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de523f781f095e28fa605cdce0f8307e451cc0fd14e2eb4cd2e98a355b147766" +checksum = "46f859dbbf73865c6627ed570e78961cd3ac92407a2d117204c49232485da55e" dependencies = [ "proc-macro2", "quote", @@ -2457,9 +2504,9 @@ dependencies = [ [[package]] name = "serde_json" -version = "1.0.132" +version = "1.0.133" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d726bfaff4b320266d395898905d0eba0345aae23b54aee3a737e260fd46db03" +checksum = "c7fceb2473b9166b2294ef05efcb65a3db80803f0b03ef86a5fc88a2b85ee377" dependencies = [ "itoa", "memchr", @@ -2498,27 +2545,6 @@ dependencies = [ "serde", ] -[[package]] -name = "server" -version = "0.5.0" -dependencies = [ - "axum", - "bincode", - "chrono", - "color-eyre", - "derive_deref", - "itertools", - "parser", - "regex", - "reqwest", - "serde", - "serde_json", - "swc", - "swc_common", - "tokio", - "tower-http", -] - [[package]] name = "sha1" version = "0.10.6" @@ -2595,12 +2621,6 @@ dependencies = [ "version_check", ] -[[package]] -name = "smawk" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b7c388c1b5e93756d0c740965c41e8822f866621d41acbdf6336a6a168f8840c" - [[package]] name = "socket2" version = "0.5.7" @@ -2613,9 +2633,9 @@ dependencies = [ [[package]] name = "sourcemap" -version = "9.0.0" +version = "9.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dab08a862c70980b8e23698b507e272317ae52a608a164a844111f5372374f1f" +checksum = "27c4ea7042fd1a155ad95335b5d505ab00d5124ea0332a06c8390d200bb1a76a" dependencies = [ "base64-simd", "bitvec", @@ -2690,9 +2710,9 @@ checksum = "9091b6114800a5f2141aee1d1b9d6ca3592ac062dc5decb3764ec5895a47b4eb" [[package]] name = "string_enum" -version = "0.4.4" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05e383308aebc257e7d7920224fa055c632478d92744eca77f99be8fa1545b90" +checksum = "c9fe66b8ee349846ce2f9557a26b8f1e74843c4a13fb381f9a3d73617a5f956a" dependencies = [ "proc-macro2", "quote", @@ -2736,9 +2756,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" [[package]] name = "swc" -version = "0.289.1" +version = "9.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "65d51f69ac919b22a08e8654f63def43578a174132bda87309dea8fddd41a5d9" +checksum = "a8ded6be21206da7fb30e245d72373346d72fe095b75354a413b1d2b6bfda0b7" dependencies = [ "anyhow", "base64 0.21.7", @@ -2786,9 +2806,9 @@ dependencies = [ [[package]] name = "swc_allocator" -version = "0.1.9" +version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7016ee7a5186d6a80e381aa1926e0f3c7b06eaf444745ff7af3632e978eb8dc5" +checksum = "117d5d3289663f53022ebf157df8a42b3872d7ac759e63abf96b5987b85d4af3" dependencies = [ "bumpalo", "hashbrown 0.14.5", @@ -2799,9 +2819,9 @@ dependencies = [ [[package]] name = "swc_atoms" -version = "1.0.2" +version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b25ff0f3fd48ab1a95d86fd0505fdd1ac904f84d0350dc8222bbc824e9d4fdf6" +checksum = "151a6feb82b989a087433baca7f6a6eb4fcf83f828c479eecd039c9312d60e10" dependencies = [ "hstr", "once_cell", @@ -2811,9 +2831,9 @@ dependencies = [ [[package]] name = "swc_cached" -version = "0.3.20" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83406221c501860fce9c27444f44125eafe9e598b8b81be7563d7036784cd05c" +checksum = "96b6a5ef4cfec51d3fa30b73600f206453a37fc30cf1141e4644a57b1ed88616" dependencies = [ "ahash", "anyhow", @@ -2825,9 +2845,9 @@ dependencies = [ [[package]] name = "swc_common" -version = "0.40.1" +version = "5.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f355465eaed1104244ce918b7ffb77ceb109aabeb74b04b98acae85683b0215b" +checksum = "a521e8120dc0401580864a643b5bffa035c29fc3fc41697c972743d4f008ed22" dependencies = [ "ahash", "ast_node", @@ -2854,9 +2874,9 @@ dependencies = [ [[package]] name = "swc_compiler_base" -version = "0.23.0" +version = "7.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20b82e7c3cc5954a5b2e9c545920232a1d37a70215719f5021f090eda9fcf88a" +checksum = "91e6b81e3504db35cb4931ac448cc88a80e89d04de00412b74102b9dc86ba131" dependencies = [ "anyhow", "base64 0.21.7", @@ -2880,9 +2900,9 @@ dependencies = [ [[package]] name = "swc_config" -version = "0.1.15" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4740e53eaf68b101203c1df0937d5161a29f3c13bceed0836ddfe245b72dd000" +checksum = "4aa30931f9b26af8edcb4cce605909d15dcfd7577220b22c50a2988f2a53c4c1" dependencies = [ "anyhow", "indexmap", @@ -2895,9 +2915,9 @@ dependencies = [ [[package]] name = "swc_config_macro" -version = "0.1.4" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c5f56139042c1a95b54f5ca48baa0e0172d369bcc9d3d473dad1de36bae8399" +checksum = "7f2ebd37ef52a8555c8c9be78b694d64adcb5e3bc16c928f030d82f1d65fac57" dependencies = [ "proc-macro2", "quote", @@ -2907,9 +2927,9 @@ dependencies = [ [[package]] name = "swc_ecma_ast" -version = "0.121.1" +version = "5.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db7e1b73c85ff968404867505646b3c6f26e4661e4fc831593b9e182fa59ddd4" +checksum = "94cf86f17358b93fcfe2876a9f0f7a7ebbff94cd6eaab4c809c7a0da1f4b892e" dependencies = [ "bitflags 2.6.0", "is-macro", @@ -2920,14 +2940,15 @@ dependencies = [ "string_enum", "swc_atoms", "swc_common", + "swc_visit", "unicode-id-start", ] [[package]] name = "swc_ecma_codegen" -version = "0.158.1" +version = "5.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1953878c2f44d796ff4ee8bde727890d24fe3bed9a86a23be4c4ef9ad79b6a6c" +checksum = "fb17e77270860f2a975c546c4609e9fa7ae8dbcf85260497e31af19890645800" dependencies = [ "memchr", "num-bigint", @@ -2945,9 +2966,9 @@ dependencies = [ [[package]] name = "swc_ecma_codegen_macros" -version = "0.7.7" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "859fabde36db38634f3fad548dd5e3410c1aebba1b67a3c63e67018fa57a0bca" +checksum = "5f9a42f479a6475647e248fa9750982c87cd985e19d1016a1fc18a70682305d1" dependencies = [ "proc-macro2", "quote", @@ -2957,9 +2978,9 @@ dependencies = [ [[package]] name = "swc_ecma_compat_bugfixes" -version = "0.16.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fff0f4ede38d4110f8c639a4c07c8d2b677d3eac968ab33da312bbaf6ad5adf0" +checksum = "448864548ea7c1866d140e36829196e5d7b35a55b4a1563945a8ce39322b8e7a" dependencies = [ "swc_atoms", "swc_common", @@ -2974,9 +2995,9 @@ dependencies = [ [[package]] name = "swc_ecma_compat_common" -version = "0.14.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "66a479b2061a3b2c939a9abd34cfd3e9a6983a8bf12f7d3da89ac7ad6a64d9a4" +checksum = "9a04016559d293d2ac507d931e46ecfd92eb5e96ccf5452bc81c18c47806b858" dependencies = [ "swc_common", "swc_ecma_ast", @@ -2987,9 +3008,9 @@ dependencies = [ [[package]] name = "swc_ecma_compat_es2015" -version = "0.16.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "129fa363e3ea01f9e2d2dbd06675bfd515689b261c28b30296d89c91b549916a" +checksum = "aceb1b432819541aa1dd129b4a226f41cf881a53040efc711a625ee5f23e9afd" dependencies = [ "arrayvec", "indexmap", @@ -3014,9 +3035,9 @@ dependencies = [ [[package]] name = "swc_ecma_compat_es2016" -version = "0.16.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2a0f1dd466100cd41e85be398d1f4d973f8d5760ec52376294918def88e620ad" +checksum = "fdc0d69f6541b781e4e95c1954e8ab916095b7369cdf69e14bf83be17b2dd943" dependencies = [ "swc_atoms", "swc_common", @@ -3031,9 +3052,9 @@ dependencies = [ [[package]] name = "swc_ecma_compat_es2017" -version = "0.16.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "33a3fcc3ae30628fc752d04b44c885ea6202ecfb187311d8827cc851ba7d906d" +checksum = "1ac80866f541899e4c59a954c0a37a4fd21ea1455dc44400c088b72299c1d2cc" dependencies = [ "serde", "swc_atoms", @@ -3049,9 +3070,9 @@ dependencies = [ [[package]] name = "swc_ecma_compat_es2018" -version = "0.16.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "31160d66339dda14faefd3a2816a0436f0a1e4f33a2057d8f3663a1a567eaf2e" +checksum = "cdc28efd763ad5fea7acc2f1fc73ade5b334201369b88aff472053b64b304a76" dependencies = [ "serde", "swc_atoms", @@ -3068,9 +3089,9 @@ dependencies = [ [[package]] name = "swc_ecma_compat_es2019" -version = "0.16.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f0164ad462c81c421de2af75f408ca3a96324a3c5e81d999a90a58e6e40dbef" +checksum = "f2439138f8b8c7cf1401395e0bbb6d3e35d20277268f8e8aa41fa64841af992e" dependencies = [ "swc_atoms", "swc_common", @@ -3084,9 +3105,9 @@ dependencies = [ [[package]] name = "swc_ecma_compat_es2020" -version = "0.16.0" +version = "7.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da49ae2565352c179657b47e87b4879b8c1128b5447e8fbfaaa020702f740636" +checksum = "b7a37218dcb1ad82943b9ebfbfbc283ba1ba542ed3c60aacea0b3a06e7f38809" dependencies = [ "serde", "swc_atoms", @@ -3102,9 +3123,9 @@ dependencies = [ [[package]] name = "swc_ecma_compat_es2021" -version = "0.16.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d9fed665a9f5b2b3c55fc04f36b12c8d2148fd25fb216a4d9ae9fac331a5374" +checksum = "9c4c9d40dcf5094e863ceac27c92053ef970991e46f190ddad678b8e3ca3ecde" dependencies = [ "swc_atoms", "swc_common", @@ -3118,9 +3139,9 @@ dependencies = [ [[package]] name = "swc_ecma_compat_es2022" -version = "0.16.0" +version = "7.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "241bf311a26447690bbfc36ec76886dac04343f82e3fe638fc0e0cdbaeae534b" +checksum = "93edab9540513a7622addbf2494fa61f487daf98d2c1ed819958b2a3d271835e" dependencies = [ "swc_atoms", "swc_common", @@ -3137,9 +3158,9 @@ dependencies = [ [[package]] name = "swc_ecma_compat_es3" -version = "0.16.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2a0c7867bba0129c8844cf274c9a0e3f1926d7225d6dce36343a4b1626399992" +checksum = "0f158dcb284c676bfb425c42adf307a3259d71c999c0970370ed3e09886d83c9" dependencies = [ "swc_common", "swc_ecma_ast", @@ -3152,9 +3173,9 @@ dependencies = [ [[package]] name = "swc_ecma_ext_transforms" -version = "0.123.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "55ea0bdb96b8d24305cf23614db4c734425c44329bd6e1734e6ebd068a88b536" +checksum = "d3cecd84290edc5c27d0cb6df79fcd82fe500772cd2526642955227e50359675" dependencies = [ "phf", "swc_atoms", @@ -3166,9 +3187,9 @@ dependencies = [ [[package]] name = "swc_ecma_lints" -version = "0.104.1" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4afb24beda205684867ba555f59e4a470afbe67d9deaacc29b1a5e3a4a1fd58d" +checksum = "808120ab5d77e0c9484f4e1b248d5f9dfecb103160340eab5710f068d69964b1" dependencies = [ "auto_impl", "dashmap 5.5.3", @@ -3186,9 +3207,9 @@ dependencies = [ [[package]] name = "swc_ecma_loader" -version = "0.52.1" +version = "5.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f521eea3476abc7ae7adcbfa3479881087e5ae8fe358d46376d761e83fdb3120" +checksum = "1a19b132079bfcd19d6fdabce7e55ece93a30787f3b8684c8646ddaf2237812d" dependencies = [ "anyhow", "dashmap 5.5.3", @@ -3208,9 +3229,9 @@ dependencies = [ [[package]] name = "swc_ecma_minifier" -version = "0.208.1" +version = "6.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67d0fec4e2a4a00c1605f6baa2845cc05b4059c1bf9a6e45377c8ae67f04207d" +checksum = "5368ebcf923fb8e94c94f5fe0caa230c7920b72e5d79ed4cc32592c03213b43d" dependencies = [ "arrayvec", "indexmap", @@ -3243,9 +3264,9 @@ dependencies = [ [[package]] name = "swc_ecma_parser" -version = "0.152.1" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a4cfd7930abe18c6829d4adedb5249e1b9fa68e8e786c6636250637992ba0466" +checksum = "c2c361b4153905dc088a6bacfaa944b582305cf94fbfcaa9b3aa61a7dd3adbf9" dependencies = [ "either", "new_debug_unreachable", @@ -3265,9 +3286,9 @@ dependencies = [ [[package]] name = "swc_ecma_preset_env" -version = "0.221.0" +version = "8.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6b6b5a7222fe85be1732faa7a2c9812fc516d934ba320662c90c066264cd9eec" +checksum = "39ac4bd1620798c38f767417f9b364b87fd2505a9e748d6b1d7549f155a0aa96" dependencies = [ "anyhow", "dashmap 5.5.3", @@ -3290,9 +3311,9 @@ dependencies = [ [[package]] name = "swc_ecma_transforms" -version = "0.243.0" +version = "8.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b22f584075dfbd349c107def9c52de654979f0a1e8cdc4255dd4e7ec5b39800" +checksum = "dccedb5a1e52795e132615a3ca01e2adb8fc9214ee75c258f8a3124a9b42c47e" dependencies = [ "swc_atoms", "swc_common", @@ -3310,9 +3331,9 @@ dependencies = [ [[package]] name = "swc_ecma_transforms_base" -version = "0.149.0" +version = "6.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2194d20416a03ae02058934f8387691809f86466a2f2e7a65c56dcb001bd46b" +checksum = "f31557485025a2fd1f833d63601c53010857e45633f44bcc87510f3578bde0c5" dependencies = [ "better_scoped_tls", "bitflags 2.6.0", @@ -3333,9 +3354,9 @@ dependencies = [ [[package]] name = "swc_ecma_transforms_classes" -version = "0.138.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "50e93ef88e45587864ef645e29c33f04f75360e47194c084f931855af3aac246" +checksum = "3cd3a9eacaf2aa2ec7a8139e3c3883ddf4f8e948a08ee378725a4ad658beb3d9" dependencies = [ "swc_atoms", "swc_common", @@ -3347,9 +3368,9 @@ dependencies = [ [[package]] name = "swc_ecma_transforms_compat" -version = "0.175.0" +version = "7.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35dd174f6c954a4fe1bff88a64de777ef65041411f96b60daf4a78864ad47789" +checksum = "e245e2cb96ff093c0c3170dbe4c063a0b937ce4527d37f52f467529d4977634b" dependencies = [ "arrayvec", "indexmap", @@ -3383,9 +3404,9 @@ dependencies = [ [[package]] name = "swc_ecma_transforms_macros" -version = "0.5.5" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "500a1dadad1e0e41e417d633b3d6d5de677c9e0d3159b94ba3348436cdb15aab" +checksum = "6845dfb88569f3e8cd05901505916a8ebe98be3922f94769ca49f84e8ccec8f7" dependencies = [ "proc-macro2", "quote", @@ -3395,9 +3416,9 @@ dependencies = [ [[package]] name = "swc_ecma_transforms_module" -version = "0.194.1" +version = "7.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4317b5ce1f5fc00b7d72b712c822e1aa66cdb9dc155799ee2278e00b2172f918" +checksum = "5480053dfa9b7ae34c6d5cdaa33bee3d7dfd33a259cabe9122806f36625912c1" dependencies = [ "Inflector", "anyhow", @@ -3422,9 +3443,9 @@ dependencies = [ [[package]] name = "swc_ecma_transforms_optimization" -version = "0.212.1" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b2722095aef7b6de79b170c8c4e3c885576c5394e7a20328fdc51e880d6257b" +checksum = "350a4965abfada7d5b23b3140896652acc11e110ac042a160bcea5bf8b08d367" dependencies = [ "dashmap 5.5.3", "indexmap", @@ -3446,9 +3467,9 @@ dependencies = [ [[package]] name = "swc_ecma_transforms_proposal" -version = "0.183.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "718f1e24dd96cfb0b7ba8f8a4e61c98338cdac7a3f5f9f4a83951d776ac398bf" +checksum = "cba55188e66696c43333d19492f69a6f4a7785f9486ebc1b3cf7c5f743927505" dependencies = [ "either", "rustc-hash", @@ -3466,9 +3487,9 @@ dependencies = [ [[package]] name = "swc_ecma_transforms_react" -version = "0.195.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06ddcb8943976319a4f8b899862f094f6984511490e4bfa9b4c8fbd334539b62" +checksum = "4cabf9375cfb71fc0e3d98e07e6fca39a18daa23d4878d8d2daa4c2b6c07b379" dependencies = [ "base64 0.21.7", "dashmap 5.5.3", @@ -3491,9 +3512,9 @@ dependencies = [ [[package]] name = "swc_ecma_transforms_typescript" -version = "0.202.1" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "31acb773a2d82e5524d021aa77a38ae8bc708171b0005fd420b43356d8dfba25" +checksum = "77346c37397fb238f991d6dccc027881caca539628e9a6c629299c7b94bdb08a" dependencies = [ "ryu-js", "serde", @@ -3508,9 +3529,9 @@ dependencies = [ [[package]] name = "swc_ecma_usage_analyzer" -version = "0.33.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "55a1d8b627b6adc706ccd2a4a30a5413e9df91a9cff6569cb9d3b9f41c1bc8de" +checksum = "6f58c91cfddea5723e22dbb8b1738642714d0acd1463d26d3bc7518b83468414" dependencies = [ "indexmap", "rustc-hash", @@ -3525,9 +3546,9 @@ dependencies = [ [[package]] name = "swc_ecma_utils" -version = "0.137.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "939f21c75eff61ad3e485fc54d38988f2f9744ceda24a3feb8228dd072171d94" +checksum = "527fad9bdb16883782d55291fd3330925b3572f512ef89b3d92a29e2f713fe4f" dependencies = [ "indexmap", "num_cpus", @@ -3544,9 +3565,9 @@ dependencies = [ [[package]] name = "swc_ecma_visit" -version = "0.107.0" +version = "5.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "15bd17e03251272ee04d1155036be5288055ab43d40ab2d9fd63ff815d326dad" +checksum = "b04c06c1805bda18c27165560f1617a57453feb9fb0638d90839053641af42d4" dependencies = [ "new_debug_unreachable", "num-bigint", @@ -3559,9 +3580,9 @@ dependencies = [ [[package]] name = "swc_eq_ignore_macros" -version = "0.1.4" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63db0adcff29d220c3d151c5b25c0eabe7e32dd936212b84cdaa1392e3130497" +checksum = "e96e15288bf385ab85eb83cff7f9e2d834348da58d0a31b33bdb572e66ee413e" dependencies = [ "proc-macro2", "quote", @@ -3570,9 +3591,9 @@ dependencies = [ [[package]] name = "swc_error_reporters" -version = "1.0.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83f9286183c9be40aafcbbe8c397403fb32af812a3dffe93eee9067aee4fbedb" +checksum = "4f741b530b2df577a287e193c4a111182de01b43361617af228ec9e6e6222fa4" dependencies = [ "anyhow", "miette", @@ -3583,9 +3604,9 @@ dependencies = [ [[package]] name = "swc_fast_graph" -version = "1.0.0" +version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3daff8d4379be2a99ab4b146e4dd631ef2415965dc4f1d33e988a737c5ccc39a" +checksum = "c22e0a0478b1b06610453a97c8371cafa742e371a79aff860ccfbabe1ab160a7" dependencies = [ "indexmap", "petgraph", @@ -3595,9 +3616,9 @@ dependencies = [ [[package]] name = "swc_macros_common" -version = "0.3.13" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f486687bfb7b5c560868f69ed2d458b880cebc9babebcb67e49f31b55c5bf847" +checksum = "a509f56fca05b39ba6c15f3e58636c3924c78347d63853632ed2ffcb6f5a0ac7" dependencies = [ "proc-macro2", "quote", @@ -3606,9 +3627,9 @@ dependencies = [ [[package]] name = "swc_node_comments" -version = "0.27.0" +version = "5.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "31c325461f5e78e7749241f917945164adccfadcc2138a9a2e5ac9d186e1c459" +checksum = "1b56d29b30a2b3f407cc8a64e01414a4150d10cc5dd72d9c2d34734d8c0af951" dependencies = [ "dashmap 5.5.3", "swc_atoms", @@ -3617,18 +3638,18 @@ dependencies = [ [[package]] name = "swc_timer" -version = "0.27.0" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f53899e4ab3f9ce3db83b4c845f8a27f37bf700b233febbc9a6da749651c2fbb" +checksum = "4db06b46cc832f7cf83c2ce21905fc465d01443a2bdccf63644383e1f5847532" dependencies = [ "tracing", ] [[package]] name = "swc_trace_macro" -version = "1.0.0" +version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69aa25f667e4d74ab10a17a266edeb8b354273817b20b91e60471f1c860a221b" +checksum = "4c78717a841565df57f811376a3d19c9156091c55175e12d378f3a522de70cef" dependencies = [ "proc-macro2", "quote", @@ -3637,9 +3658,9 @@ dependencies = [ [[package]] name = "swc_transform_common" -version = "0.1.1" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eda3e80e1ad638d3575bc07745a914af13dcb02215098659f864731078271f2c" +checksum = "f23ade45bb0d8b5299022dc0f674c2125512412f5b26f42cfaffa16dcc00d56b" dependencies = [ "better_scoped_tls", "once_cell", @@ -3650,21 +3671,25 @@ dependencies = [ [[package]] name = "swc_typescript" -version = "0.8.0" +version = "5.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b61f86dce2eeabd85d934a89cd481709918c751c44dfe433b5e4f1f40075fb5a" +checksum = "fe778782270111e7152a99af340bf763f12cdc60df775cd0ce51eaa74dcc0249" dependencies = [ + "petgraph", + "rustc-hash", "swc_atoms", "swc_common", "swc_ecma_ast", + "swc_ecma_utils", + "swc_ecma_visit", "thiserror", ] [[package]] name = "swc_visit" -version = "0.6.2" +version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ceb044142ba2719ef9eb3b6b454fce61ab849eb696c34d190f04651955c613d" +checksum = "9138b6a36bbe76dd6753c4c0794f7e26480ea757bee499738bedbbb3ae3ec5f3" dependencies = [ "either", "new_debug_unreachable", @@ -3753,7 +3778,6 @@ version = "0.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "23d434d3f8967a09480fb04132ebe0a3e088c173e6d0ee7897abbdf4eab0f8b9" dependencies = [ - "smawk", "unicode-linebreak", "unicode-width", ] @@ -3805,9 +3829,9 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" [[package]] name = "tokio" -version = "1.40.0" +version = "1.42.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2b070231665d27ad9ec9b8df639893f46727666c6767db40317fbe920a5d998" +checksum = "5cec9b21b0450273377fc97bd4c33a8acffc8c996c987a7c5b319a0083707551" dependencies = [ "backtrace", "bytes", @@ -3918,9 +3942,9 @@ dependencies = [ [[package]] name = "tower-http" -version = "0.6.1" +version = "0.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8437150ab6bbc8c5f0f519e3d5ed4aa883a83dd4cdd3d1b21f9482936046cb97" +checksum = "403fa3b783d4b626a8ad51d766ab03cb6d2dbfc46b1c5d4448395e6628dc9697" dependencies = [ "async-compression", "bitflags 2.6.0", @@ -3961,9 +3985,9 @@ dependencies = [ [[package]] name = "tracing-attributes" -version = "0.1.27" +version = "0.1.28" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "34704c8d6ebcbc939824180af020566b01a7c01f80641264eba0999f6c2b6be7" +checksum = "395ae124c09f9e6918a2310af6038fba074bcf474ac352496d5910dd59a2226d" dependencies = [ "proc-macro2", "quote", @@ -4043,9 +4067,9 @@ checksum = "10103c57044730945224467c09f71a4db0071c123a0648cc3e818913bde6b561" [[package]] name = "unicode-id-start" -version = "1.3.0" +version = "1.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97e2a3c5fc9de285c0e805d98eba666adb4b2d9e1049ce44821ff7707cc34e91" +checksum = "2f322b60f6b9736017344fa0635d64be2f458fbc04eef65f6be22976dd1ffd5b" [[package]] name = "unicode-ident" @@ -4105,9 +4129,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.10.0" +version = "1.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "81dfa00651efa65069b0b6b651f4aaa31ba9e3c3ce0137aaad053604ee7e0314" +checksum = "f8c5f0a0af699448548ad1a2fbf920fb4bee257eae39953ba95cb84891a0446a" [[package]] name = "valuable" diff --git a/Cargo.toml b/Cargo.toml new file mode 100644 index 000000000..bd7c6bde3 --- /dev/null +++ b/Cargo.toml @@ -0,0 +1,39 @@ +[package] +name = "kibo_money" +version = "0.6.0" +edition = "2021" + +# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html + +[dependencies] +allocative = "0.3.3" +axum = "0.7.9" +bincode = { git = "https://github.com/bincode-org/bincode.git", features = [ + "serde", +] } +bitcoin_hashes = { version = "0.15.0" } +biter = { path = "./crates/biter" } +chrono = { version = "0.4.39", features = ["serde"] } +clap = { version = "4.5.23", features = ["derive"] } +color-eyre = "0.6.3" +ctrlc = { version = "3.4.5", features = ["termination"] } +derive_deref = "1.1.1" +env_logger = "0.11.5" +inferno = "0.12.0" +itertools = "0.13.0" +log = { version = "0.4.22", features = ["std", "serde"] } +ordered-float = "4.5.0" +rayon = "1.10.0" +regex = "1.11.1" +reqwest = { version = "0.12.9", features = ["blocking", "json"] } +rlimit = "0.10.2" +sanakirja = "1.4.3" +serde = { version = "1.0.216", features = ["derive"] } +serde_json = "1.0.133" +struct_iterable = { path = "./crates/iterable" } +swc = "9.0.0" +swc_common = "5.0.0" +tokio = { version = "1.42.0", features = ["full"] } +toml = "0.8.19" +tower-http = { version = "0.6.2", features = ["compression-full"] } +zstd = "0.13.2" diff --git a/README.md b/README.md index c2e0ce2cb..d602d55d8 100644 --- a/README.md +++ b/README.md @@ -122,6 +122,7 @@ Now we can finally start by running the parser, you need to use the `./run.sh` s For the first launch, the parser will need several information such as: - `--datadir`: which is bitcoin data directory path, prefer `$HOME` to `~` as the latter might not work +- `--outdir`: where all outputs will be saved, prefer `$HOME` to `~` as the latter might not work Optionally you can also specify: @@ -136,7 +137,7 @@ Everything will be saved in a `config.toml` file, which will allow you to simply Here's an example ```bash -./run.sh --datadir=$HOME/Developer/bitcoin +./run.sh --datadir=$HOME/Developer/bitcoin --outdir=$HOME/.kibo/out ``` In a **new** terminal, go to the `server`'s folder of the repository diff --git a/crates/biter/CHANGELOG.md b/crates/biter/CHANGELOG.md new file mode 100644 index 000000000..9b9c5b195 --- /dev/null +++ b/crates/biter/CHANGELOG.md @@ -0,0 +1,12 @@ +# v0.2.1 + +- Clean `.json` if necessary +- Only save `.json` if needed +- Updated benchmarks +- Updated packages + +# v0.2.0 + +- Removed the need for an output directory path +- Changed the location of the saved json file from the previously needed output directory path to the Bitcoin data directory +- Added a save of the json file every 144 * 30 blocks instead of only at the end diff --git a/biter/Cargo.lock b/crates/biter/Cargo.lock similarity index 85% rename from biter/Cargo.lock rename to crates/biter/Cargo.lock index e178bb13f..b71079ba4 100644 --- a/biter/Cargo.lock +++ b/crates/biter/Cargo.lock @@ -1,12 +1,12 @@ # This file is automatically @generated by Cargo. # It is not intended for manual editing. -version = 3 +version = 4 [[package]] name = "arrayvec" -version = "0.7.4" +version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "96d30a06541fbafbc7f82ed10c06164cfbd2c401138f6addd8404629c4b16711" +checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "base58ck" @@ -32,9 +32,9 @@ checksum = "d965446196e3b7decd44aa7ee49e31d630118f90ef12f97900f262eb915c951d" [[package]] name = "bitcoin" -version = "0.32.2" +version = "0.32.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea507acc1cd80fc084ace38544bbcf7ced7c2aa65b653b102de0ce718df668f6" +checksum = "ce6bc65742dea50536e35ad42492b234c27904a27f0abdcbce605015cb4ea026" dependencies = [ "base58ck", "bech32", @@ -110,7 +110,7 @@ dependencies = [ [[package]] name = "biter" -version = "0.1.1" +version = "0.2.1" dependencies = [ "bitcoin", "bitcoincore-rpc", @@ -129,9 +129,12 @@ checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" [[package]] name = "cc" -version = "1.1.6" +version = "1.1.31" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2aba8f4e9906c7ce3c73463f62a7f0c65183ada1a2d47e397cc8810827f9694f" +checksum = "c2e7962b54006dcfcc61cb72735f4d89bb97061dd6a7ed882ec6b8ee53714c6f" +dependencies = [ + "shlex", +] [[package]] name = "cfg-if" @@ -258,9 +261,9 @@ dependencies = [ [[package]] name = "libc" -version = "0.2.155" +version = "0.2.161" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97b3888a4aecf77e811145cadf6eef5901f4782c53886191b2f693f24761847c" +checksum = "8e9489c2807c139ffd9c1794f4af0ebe86a828db53ecdc7fea2111d0fed085d1" [[package]] name = "log" @@ -287,27 +290,27 @@ dependencies = [ [[package]] name = "ppv-lite86" -version = "0.2.18" +version = "0.2.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dee4364d9f3b902ef14fab8a1ddffb783a1cb6b4bba3bfc1fa3922732c7de97f" +checksum = "77957b295656769bb8ad2b6a6b09d897d94f05c41b069aede1fcdaa675eaea04" dependencies = [ "zerocopy", ] [[package]] name = "proc-macro2" -version = "1.0.86" +version = "1.0.92" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e719e8df665df0d1c8fbfd238015744736151d4445ec0836b8e628aae103b77" +checksum = "37d3544b3f2748c54e147655edb5025752e2303145b5aefb3c3ea2c78b973bb0" dependencies = [ "unicode-ident", ] [[package]] name = "quote" -version = "1.0.36" +version = "1.0.37" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0fa76aaf39101c457836aec0ce2316dbdc3ab723cdda1c6bd4e6ad4208acaca7" +checksum = "b5b9d34b8991d19d98081b46eacdd8eb58c6f2b201139f7c5f643cc155a633af" dependencies = [ "proc-macro2", ] @@ -370,9 +373,9 @@ checksum = "f3cb5ba0dc43242ce17de99c180e96db90b235b8a9fdc9543c96d2209116bd9f" [[package]] name = "secp256k1" -version = "0.29.0" +version = "0.29.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e0cc0f1cf93f4969faf3ea1c7d8a9faed25918d96affa959720823dfe86d4f3" +checksum = "9465315bc9d4566e1724f0fffcbcc446268cb522e60f9a27bcded6b19c108113" dependencies = [ "bitcoin_hashes", "rand", @@ -382,27 +385,27 @@ dependencies = [ [[package]] name = "secp256k1-sys" -version = "0.10.0" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1433bd67156263443f14d603720b082dd3121779323fce20cba2aa07b874bc1b" +checksum = "d4387882333d3aa8cb20530a17c69a3752e97837832f34f6dccc760e715001d9" dependencies = [ "cc", ] [[package]] name = "serde" -version = "1.0.204" +version = "1.0.216" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc76f558e0cbb2a839d37354c575f1dc3fdc6546b5be373ba43d95f231bf7c12" +checksum = "0b9781016e935a97e8beecf0c933758c97a5520d32930e460142b4cd80c6338e" dependencies = [ "serde_derive", ] [[package]] name = "serde_derive" -version = "1.0.204" +version = "1.0.216" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0cd7e117be63d3c3678776753929474f3b04a43a080c744d6b0ae2a8c28e222" +checksum = "46f859dbbf73865c6627ed570e78961cd3ac92407a2d117204c49232485da55e" dependencies = [ "proc-macro2", "quote", @@ -411,9 +414,9 @@ dependencies = [ [[package]] name = "serde_json" -version = "1.0.122" +version = "1.0.133" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "784b6203951c57ff748476b126ccb5e8e2959a5c19e5c617ab1956be3dbc68da" +checksum = "c7fceb2473b9166b2294ef05efcb65a3db80803f0b03ef86a5fc88a2b85ee377" dependencies = [ "itoa", "memchr", @@ -422,10 +425,16 @@ dependencies = [ ] [[package]] -name = "syn" -version = "2.0.72" +name = "shlex" +version = "1.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc4b9b9bf2add8093d3f2c0204471e951b2285580335de42f9d2534f3ae7a8af" +checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" + +[[package]] +name = "syn" +version = "2.0.90" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "919d3b74a5dd0ccd15aeb8f93e7006bd9e14c295087c9896a110f490752bcf31" dependencies = [ "proc-macro2", "quote", @@ -434,9 +443,9 @@ dependencies = [ [[package]] name = "unicode-ident" -version = "1.0.12" +version = "1.0.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3354b9ac3fae1ff6755cb6db53683adb661634f67557942dea4facebec0fee4b" +checksum = "adb9e6ca4f869e1180728b7950e35922a7fc6397f7b641499e8f3ef06e50dc83" [[package]] name = "wasi" @@ -446,9 +455,9 @@ checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423" [[package]] name = "zerocopy" -version = "0.6.6" +version = "0.7.35" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "854e949ac82d619ee9a14c66a1b674ac730422372ccb759ce0c39cabcf2bf8e6" +checksum = "1b9b4fd18abc82b8136838da5d50bae7bdea537c574d8dc1a34ed098d6c166f0" dependencies = [ "byteorder", "zerocopy-derive", @@ -456,9 +465,9 @@ dependencies = [ [[package]] name = "zerocopy-derive" -version = "0.6.6" +version = "0.7.35" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "125139de3f6b9d625c39e2efdd73d41bdac468ccd556556440e322be0e1bbd91" +checksum = "fa4f8080344d4671fb4e831a13ad1e68092748387dfc4f55e356242fae12ce3e" dependencies = [ "proc-macro2", "quote", diff --git a/biter/Cargo.toml b/crates/biter/Cargo.toml similarity index 65% rename from biter/Cargo.toml rename to crates/biter/Cargo.toml index f01e3608d..fb39c739d 100644 --- a/biter/Cargo.toml +++ b/crates/biter/Cargo.toml @@ -1,19 +1,19 @@ [package] name = "biter" description = "A very fast Bitcoin block iterator" -version = "0.1.1" +version = "0.2.1" license = "MIT" -repository = "https://github.com/kibo-money/kibo/tree/main/biter" +repository = "https://github.com/kibo-money/kibo/tree/main/crates/biter" keywords = ["bitcoin", "block", "iterator"] categories = ["cryptography::cryptocurrencies", "encoding"] edition = "2021" [dependencies] -bitcoin = { version = "0.32.2", features = ["serde"] } +bitcoin = { version = "0.32.5", features = ["serde"] } rayon = "1.10.0" crossbeam = { version = "0.8.4", features = ["crossbeam-channel"] } -serde = { version = "1.0.204", features = ["derive"] } -serde_json = "1.0.122" +serde = { version = "1.0.216", features = ["derive"] } +serde_json = "1.0.133" derived-deref = "2.1.0" bitcoincore-rpc = "0.19.0" # tokio = { version = "1.39.2", features = ["rt-multi-thread"] } diff --git a/biter/LICENSE.md b/crates/biter/LICENSE.md similarity index 100% rename from biter/LICENSE.md rename to crates/biter/LICENSE.md diff --git a/biter/README.md b/crates/biter/README.md similarity index 71% rename from biter/README.md rename to crates/biter/README.md index 932108dac..13c3097e6 100644 --- a/biter/README.md +++ b/crates/biter/README.md @@ -10,6 +10,8 @@ The element returned by the iterator is a tuple which includes the: ## Example ```rust +use std::path::Path; + use bitcoincore_rpc::{Auth, Client}; fn main() { @@ -18,9 +20,6 @@ fn main() { // Path to the Bitcoin data directory let data_dir = "../../bitcoin"; - // Path to the export directory where a mini blk indexer will be exported - let export_dir = "./target"; - // Inclusive starting height of the blocks received, `None` for 0 let start = Some(850_000); @@ -29,11 +28,15 @@ fn main() { // RPC client to filter out forks let url = "http://localhost:8332"; - let auth = Auth::UserPass("satoshi".to_string(), "nakamoto".to_string()); + let cookie = Path::new(data_dir).join(".cookie"); + let auth = Auth::CookieFile(cookie); let rpc = Client::new(url, auth).unwrap(); + if cookie.is_file() { + Ok() + // Create channel receiver then iterate over the blocks - biter::new(data_dir, export_dir, start, end, rpc) + biter::new(data_dir, start, end, rpc) .iter() .for_each(|(height, _block, hash)| { println!("{height}: {hash}"); @@ -52,11 +55,11 @@ Peak memory should be around 500MB. ## Comparaison -| | [biter](https://crates.io/crates/biter) | [bitcoin-explorer](https://crates.io/crates/bitcoin-explorer) | [blocks_iterator](https://crates.io/crates/blocks_iterator) | +| | [biter](https://crates.io/crates/biter) | [bitcoin-explorer (depreciated)](https://crates.io/crates/bitcoin-explorer) | [blocks_iterator](https://crates.io/crates/blocks_iterator) | | --- | --- | --- | --- | -| Run **with** `bitcoind` | Yes ✅ | No ❌ | Yes ✅ | -| Run **without** `bitcoind` | No ❌ | Yes ✅ | Yes ✅ | -| `0..=855_000` | 16mn40s | 17mn 46s | > 2h | -| `800_000..=855_000` | 2mn 53s (16mn40s if first run) | 3mn 2s | > 2h | +| Runs **with** `bitcoind` | Yes ✅ | No ❌ | Yes ✅ | +| Runs **without** `bitcoind` | No ❌ | Yes ✅ | Yes ✅ | +| `0..=855_000` | 4mn 10s | 4mn 45s | > 2h | +| `800_000..=855_000` | 0mn 52s (4mn 10s if first run) | 0mn 55s | > 2h | *Benchmarked on a Macbook Pro M3 Pro* diff --git a/biter/src/blk_index_to_blk_recap.rs b/crates/biter/src/blk_index_to_blk_recap.rs similarity index 73% rename from biter/src/blk_index_to_blk_recap.rs rename to crates/biter/src/blk_index_to_blk_recap.rs index 9c9d53c53..c3823447c 100644 --- a/biter/src/blk_index_to_blk_recap.rs +++ b/crates/biter/src/blk_index_to_blk_recap.rs @@ -1,28 +1,31 @@ use std::{ cmp::Ordering, - collections::BTreeMap, + collections::{BTreeMap, BTreeSet}, fs::{self, File}, io::{BufReader, BufWriter}, - path::PathBuf, + path::{Path, PathBuf}, }; use derived_deref::{Deref, DerefMut}; use crate::{blk_recap::BlkRecap, BlkMetadataAndBlock}; +const TARGET_BLOCKS_PER_MONTH: usize = 144 * 30; + #[derive(Deref, DerefMut, Debug)] pub struct BlkIndexToBlkRecap { - path: String, + path: PathBuf, #[target] tree: BTreeMap, + last_safe_recap: Option, } impl BlkIndexToBlkRecap { - pub fn import(blocks_dir: &BTreeMap, export_dir: &str) -> Self { - let path = format!("{export_dir}/blk_index_to_blk_recap.json"); + pub fn import(blocks_dir: &BTreeMap, data_dir: &Path) -> Self { + let path = data_dir.join("blk_index_to_blk_recap.json"); let tree = { - fs::create_dir_all(export_dir).unwrap(); + fs::create_dir_all(data_dir).unwrap(); if let Ok(file) = File::open(&path) { let reader = BufReader::new(file); @@ -32,7 +35,11 @@ impl BlkIndexToBlkRecap { } }; - let mut this = Self { path, tree }; + let mut this = Self { + path, + tree, + last_safe_recap: None, + }; this.clean_outdated(blocks_dir); @@ -40,13 +47,22 @@ impl BlkIndexToBlkRecap { } pub fn clean_outdated(&mut self, blocks_dir: &BTreeMap) { + let mut unprocessed_keys = self.keys().copied().collect::>(); + blocks_dir.iter().for_each(|(blk_index, blk_path)| { + unprocessed_keys.remove(blk_index); if let Some(blk_recap) = self.get(blk_index) { if blk_recap.has_different_modified_time(blk_path) { self.remove(blk_index); } } }); + + unprocessed_keys.iter().for_each(|blk_index| { + self.remove(blk_index); + }); + + self.last_safe_recap = self.last_entry().map(|e| e.get().clone()); } pub fn get_start_recap(&self, start: Option) -> Option<(usize, BlkRecap)> { @@ -73,7 +89,6 @@ impl BlkIndexToBlkRecap { let blk_index = blk_metadata_and_block.blk_metadata.index; if let Some(last_entry) = self.last_entry() { - // if last_entry.get().is_older_than(height) { match last_entry.key().cmp(&blk_index) { Ordering::Greater => { last_entry.remove_entry(); @@ -83,7 +98,6 @@ impl BlkIndexToBlkRecap { } Ordering::Equal => {} }; - // } } else { if blk_index != 0 || height != 0 { // dbg!(blk_index, height); @@ -92,6 +106,14 @@ impl BlkIndexToBlkRecap { self.insert(blk_index, BlkRecap::first(blk_metadata_and_block)); } + + if self + .last_safe_recap + .map_or(true, |recap| recap.height() >= height) + && (height % TARGET_BLOCKS_PER_MONTH) == 0 + { + self.export(); + } } pub fn export(&self) { diff --git a/biter/src/blk_metadata.rs b/crates/biter/src/blk_metadata.rs similarity index 100% rename from biter/src/blk_metadata.rs rename to crates/biter/src/blk_metadata.rs diff --git a/biter/src/blk_metadata_and_block.rs b/crates/biter/src/blk_metadata_and_block.rs similarity index 100% rename from biter/src/blk_metadata_and_block.rs rename to crates/biter/src/blk_metadata_and_block.rs diff --git a/biter/src/blk_recap.rs b/crates/biter/src/blk_recap.rs similarity index 100% rename from biter/src/blk_recap.rs rename to crates/biter/src/blk_recap.rs diff --git a/biter/src/lib.rs b/crates/biter/src/lib.rs similarity index 96% rename from biter/src/lib.rs rename to crates/biter/src/lib.rs index a064fa1cb..ed80f1882 100644 --- a/biter/src/lib.rs +++ b/crates/biter/src/lib.rs @@ -2,6 +2,7 @@ use std::{ collections::{BTreeMap, BTreeSet, VecDeque}, fs::{self}, ops::ControlFlow, + path::Path, thread, }; @@ -44,7 +45,6 @@ enum BlockState { /// # Arguments /// /// * `data_dir` - Path to the Bitcoin data directory -/// * `export_dir` - Path to the export directory where a mini blk indexer will be exported /// * `start` - Inclusive starting height of the blocks received, `None` for 0 /// * `end` - Inclusive ending height of the blocks received, `None` for the last one /// * `rpc` - RPC client to filter out forks @@ -52,21 +52,23 @@ enum BlockState { /// # Example /// /// ```rust +/// use std::path::Path; +/// /// use bitcoincore_rpc::{Auth, Client}; /// /// fn main() { /// let i = std::time::Instant::now(); /// +/// let data_dir = Path::new("../../bitcoin"); /// let url = "http://localhost:8332"; -/// let auth = Auth::UserPass("satoshi".to_string(), "nakamoto".to_string()); +/// let cookie = Path::new(data_dir).join(".cookie"); +/// let auth = Auth::CookieFile(cookie); /// let rpc = Client::new(url, auth).unwrap(); /// -/// let data_dir = "../../bitcoin"; -/// let export_dir = "./target"; /// let start = Some(850_000); /// let end = None; /// -/// biter::new(data_dir, export_dir, start, end, rpc) +/// biter::new(data_dir, start, end, rpc) /// .iter() /// .for_each(|(height, _block, hash)| { /// println!("{height}: {hash}"); @@ -77,8 +79,7 @@ enum BlockState { /// ``` /// pub fn new( - data_dir: &str, - export_dir: &str, + data_dir: &Path, start: Option, end: Option, rpc: bitcoincore_rpc::Client, @@ -87,15 +88,15 @@ pub fn new( let (send_block, recv_block) = bounded(BOUND_CAP); let (send_height_block_hash, recv_height_block_hash) = bounded(BOUND_CAP); - let blocks_dir = scan_blocks_dir(data_dir); + let blk_index_to_blk_path = scan_blocks_dir(data_dir); - let mut blk_index_to_blk_recap = BlkIndexToBlkRecap::import(&blocks_dir, export_dir); + let mut blk_index_to_blk_recap = BlkIndexToBlkRecap::import(&blk_index_to_blk_path, data_dir); let start_recap = blk_index_to_blk_recap.get_start_recap(start); let starting_blk_index = start_recap.as_ref().map_or(0, |(index, _)| *index); thread::spawn(move || { - blocks_dir + blk_index_to_blk_path .into_iter() .filter(|(blk_index, _)| blk_index >= &starting_blk_index) .try_for_each(move |(blk_index, blk_path)| { diff --git a/biter/src/main.rs b/crates/biter/src/main.rs similarity index 54% rename from biter/src/main.rs rename to crates/biter/src/main.rs index 57ac1f020..ed1da58cb 100644 --- a/biter/src/main.rs +++ b/crates/biter/src/main.rs @@ -1,18 +1,20 @@ +use std::path::Path; + use bitcoincore_rpc::{Auth, Client}; fn main() { let i = std::time::Instant::now(); + let data_dir = Path::new("../../../bitcoin"); let url = "http://localhost:8332"; - let auth = Auth::UserPass("satoshi".to_string(), "nakamoto".to_string()); + let cookie = Path::new(data_dir).join(".cookie"); + let auth = Auth::CookieFile(cookie); let rpc = Client::new(url, auth).unwrap(); - let data_dir = "../bitcoin"; - let export_dir = "./target"; - let start = None; - let end = None; + let start = Some(800_000); + let end = Some(855_000); - biter::new(data_dir, export_dir, start, end, rpc) + biter::new(data_dir, start, end, rpc) .iter() .for_each(|(height, _block, hash)| { println!("{height}: {hash}"); diff --git a/biter/src/utils.rs b/crates/biter/src/utils.rs similarity index 79% rename from biter/src/utils.rs rename to crates/biter/src/utils.rs index 9b1bdb945..78819179e 100644 --- a/biter/src/utils.rs +++ b/crates/biter/src/utils.rs @@ -1,12 +1,17 @@ -use std::{collections::BTreeMap, fs, path::PathBuf, time::UNIX_EPOCH}; +use std::{ + collections::BTreeMap, + fs, + path::{Path, PathBuf}, + time::UNIX_EPOCH, +}; const BLK: &str = "blk"; const DAT: &str = ".dat"; -pub fn scan_blocks_dir(data_dir_path: &str) -> BTreeMap { - let blocks_dir_path = &format!("{data_dir_path}/blocks"); +pub fn scan_blocks_dir(data_dir: &Path) -> BTreeMap { + let blocks_dir = data_dir.join("blocks"); - fs::read_dir(blocks_dir_path) + fs::read_dir(blocks_dir) .unwrap() .map(|entry| entry.unwrap().path()) .filter(|path| { diff --git a/iterable/Cargo.lock b/crates/iterable/Cargo.lock similarity index 100% rename from iterable/Cargo.lock rename to crates/iterable/Cargo.lock diff --git a/iterable/Cargo.toml b/crates/iterable/Cargo.toml similarity index 100% rename from iterable/Cargo.toml rename to crates/iterable/Cargo.toml diff --git a/iterable/README.md b/crates/iterable/README.md similarity index 100% rename from iterable/README.md rename to crates/iterable/README.md diff --git a/iterable/src/lib.rs b/crates/iterable/src/lib.rs similarity index 100% rename from iterable/src/lib.rs rename to crates/iterable/src/lib.rs diff --git a/iterable/struct_iterable_derive/Cargo.toml b/crates/iterable/struct_iterable_derive/Cargo.toml similarity index 100% rename from iterable/struct_iterable_derive/Cargo.toml rename to crates/iterable/struct_iterable_derive/Cargo.toml diff --git a/iterable/struct_iterable_derive/README.md b/crates/iterable/struct_iterable_derive/README.md similarity index 100% rename from iterable/struct_iterable_derive/README.md rename to crates/iterable/struct_iterable_derive/README.md diff --git a/iterable/struct_iterable_derive/src/lib.rs b/crates/iterable/struct_iterable_derive/src/lib.rs similarity index 100% rename from iterable/struct_iterable_derive/src/lib.rs rename to crates/iterable/struct_iterable_derive/src/lib.rs diff --git a/iterable/struct_iterable_internal/Cargo.toml b/crates/iterable/struct_iterable_internal/Cargo.toml similarity index 100% rename from iterable/struct_iterable_internal/Cargo.toml rename to crates/iterable/struct_iterable_internal/Cargo.toml diff --git a/iterable/struct_iterable_internal/README.md b/crates/iterable/struct_iterable_internal/README.md similarity index 100% rename from iterable/struct_iterable_internal/README.md rename to crates/iterable/struct_iterable_internal/README.md diff --git a/iterable/struct_iterable_internal/src/lib.rs b/crates/iterable/struct_iterable_internal/src/lib.rs similarity index 100% rename from iterable/struct_iterable_internal/src/lib.rs rename to crates/iterable/struct_iterable_internal/src/lib.rs diff --git a/parser/Cargo.lock b/parser/Cargo.lock deleted file mode 100644 index 91514253c..000000000 --- a/parser/Cargo.lock +++ /dev/null @@ -1,2430 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 3 - -[[package]] -name = "addr2line" -version = "0.21.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a30b2e23b9e17a9f90641c7ab1549cd9b44f296d3ccbf309d2863cfe398a0cb" -dependencies = [ - "gimli", -] - -[[package]] -name = "adler" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f26201604c87b1e01bd3d98f8d5d9a8fcbb815e8cedb41ffccbeb4bf593a35fe" - -[[package]] -name = "ahash" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e89da841a80418a9b391ebaea17f5c112ffaaa96f621d2c285b5174da76b9011" -dependencies = [ - "cfg-if", - "getrandom", - "once_cell", - "version_check", - "zerocopy", -] - -[[package]] -name = "allocative" -version = "0.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "082af274fd02beef17b7f0725a49ecafe6c075ef56cac9d6363eb3916a9817ae" -dependencies = [ - "allocative_derive", - "ctor", -] - -[[package]] -name = "allocative_derive" -version = "0.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fe233a377643e0fc1a56421d7c90acdec45c291b30345eb9f08e8d0ddce5a4ab" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.85", -] - -[[package]] -name = "android-tzdata" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e999941b234f3131b00bc13c22d06e8c5ff726d1b6318ac7eb276997bbb4fef0" - -[[package]] -name = "android_system_properties" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" -dependencies = [ - "libc", -] - -[[package]] -name = "anstream" -version = "0.6.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64e15c1ab1f89faffbf04a634d5e1962e9074f2741eef6d97f3c4e322426d526" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bec1de6f59aedf83baf9ff929c98f2ad654b97c9510f4e70cf6f661d49fd5b1" - -[[package]] -name = "anstyle-parse" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eb47de1e80c2b463c735db5b217a0ddc39d612e7ac9e2e96a5aed1f57616c1cb" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d36fc52c7f6c869915e99412912f22093507da8d9e942ceaf66fe4b7c14422a" -dependencies = [ - "windows-sys 0.52.0", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5bf74e1b6e971609db8ca7a9ce79fd5768ab6ae46441c572e46cf596f59e57f8" -dependencies = [ - "anstyle", - "windows-sys 0.52.0", -] - -[[package]] -name = "arrayvec" -version = "0.7.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" - -[[package]] -name = "atomic-waker" -version = "1.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" - -[[package]] -name = "autocfg" -version = "1.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ace50bade8e6234aa140d9a2f552bbee1db4d353f69b8217bc503490fc1a9f26" - -[[package]] -name = "backtrace" -version = "0.3.71" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26b05800d2e817c8b3b4b54abd461726265fa9789ae34330622f2db9ee696f9d" -dependencies = [ - "addr2line", - "cc", - "cfg-if", - "libc", - "miniz_oxide", - "object", - "rustc-demangle", -] - -[[package]] -name = "base58ck" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2c8d66485a3a2ea485c1913c4572ce0256067a5377ac8c75c4960e1cda98605f" -dependencies = [ - "bitcoin-internals", - "bitcoin_hashes", -] - -[[package]] -name = "base64" -version = "0.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8" - -[[package]] -name = "base64" -version = "0.22.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" - -[[package]] -name = "bech32" -version = "0.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d965446196e3b7decd44aa7ee49e31d630118f90ef12f97900f262eb915c951d" - -[[package]] -name = "bincode" -version = "2.0.0-rc.3" -source = "git+https://github.com/bincode-org/bincode.git#5a91c1210168b968b957a14b743cf134f2d20719" -dependencies = [ - "bincode_derive", - "serde", - "unty", -] - -[[package]] -name = "bincode_derive" -version = "2.0.0-rc.3" -source = "git+https://github.com/bincode-org/bincode.git#5a91c1210168b968b957a14b743cf134f2d20719" -dependencies = [ - "virtue", -] - -[[package]] -name = "bitcoin" -version = "0.32.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0032b0e8ead7074cda7fc4f034409607e3f03a6f71d66ade8a307f79b4d99e73" -dependencies = [ - "base58ck", - "bech32", - "bitcoin-internals", - "bitcoin-io", - "bitcoin-units", - "bitcoin_hashes", - "hex-conservative", - "hex_lit", - "secp256k1", - "serde", -] - -[[package]] -name = "bitcoin-internals" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "30bdbe14aa07b06e6cfeffc529a1f099e5fbe249524f8125358604df99a4bed2" -dependencies = [ - "serde", -] - -[[package]] -name = "bitcoin-io" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "340e09e8399c7bd8912f495af6aa58bea0c9214773417ffaa8f6460f93aaee56" - -[[package]] -name = "bitcoin-units" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5285c8bcaa25876d07f37e3d30c303f2609179716e11d688f51e8f1fe70063e2" -dependencies = [ - "bitcoin-internals", - "serde", -] - -[[package]] -name = "bitcoin_hashes" -version = "0.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb18c03d0db0247e147a21a6faafd5a7eb851c743db062de72018b6b7e8e4d16" -dependencies = [ - "bitcoin-io", - "hex-conservative", - "serde", -] - -[[package]] -name = "bitcoincore-rpc" -version = "0.19.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aedd23ae0fd321affb4bbbc36126c6f49a32818dc6b979395d24da8c9d4e80ee" -dependencies = [ - "bitcoincore-rpc-json", - "jsonrpc", - "log", - "serde", - "serde_json", -] - -[[package]] -name = "bitcoincore-rpc-json" -version = "0.19.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d8909583c5fab98508e80ef73e5592a651c954993dc6b7739963257d19f0e71a" -dependencies = [ - "bitcoin", - "serde", - "serde_json", -] - -[[package]] -name = "biter" -version = "0.1.1" -dependencies = [ - "bitcoin", - "bitcoincore-rpc", - "crossbeam", - "derived-deref", - "rayon", - "serde", - "serde_json", -] - -[[package]] -name = "bitflags" -version = "1.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" - -[[package]] -name = "bitflags" -version = "2.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b048fb63fd8b5923fc5aa7b340d8e156aec7ec02f0c78fa8a6ddc2613f6f71de" - -[[package]] -name = "bumpalo" -version = "3.16.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "79296716171880943b8470b5f8d03aa55eb2e645a4874bdbb28adb49162e012c" - -[[package]] -name = "bytemuck" -version = "1.18.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94bbb0ad554ad961ddc5da507a12a29b14e4ae5bda06b19f575a3e6079d2e2ae" - -[[package]] -name = "byteorder" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" - -[[package]] -name = "bytes" -version = "1.7.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "428d9aa8fbc0670b7b8d6030a7fadd0f86151cae55e4dbbece15f3780a3dfaf3" - -[[package]] -name = "cc" -version = "1.1.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e80e3b6a3ab07840e1cae9b0666a63970dc28e8ed5ffbcdacbfc760c281bfc1" -dependencies = [ - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd" - -[[package]] -name = "cfg_aliases" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" - -[[package]] -name = "chrono" -version = "0.4.38" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a21f936df1771bf62b77f047b726c4625ff2e8aa607c01ec06e5a05bd8463401" -dependencies = [ - "android-tzdata", - "iana-time-zone", - "js-sys", - "num-traits", - "serde", - "wasm-bindgen", - "windows-targets", -] - -[[package]] -name = "clap" -version = "4.5.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b97f376d85a664d5837dbae44bf546e6477a679ff6610010f17276f686d867e8" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.5.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19bc80abd44e4bed93ca373a0704ccbd1b710dc5749406201bb018272808dc54" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_derive" -version = "4.5.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ac6a0c7b1a9e9a5186361f67dfa1b88213572f427fb9ab038efb2bd8c582dab" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 2.0.85", -] - -[[package]] -name = "clap_lex" -version = "0.7.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1462739cb27611015575c0c11df5df7601141071f07518d56fcc1be504cbec97" - -[[package]] -name = "color-eyre" -version = "0.6.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "55146f5e46f237f7423d74111267d4597b59b0dad0ffaf7303bce9945d843ad5" -dependencies = [ - "backtrace", - "color-spantrace", - "eyre", - "indenter", - "once_cell", - "owo-colors", - "tracing-error", -] - -[[package]] -name = "color-spantrace" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd6be1b2a7e382e2b98b43b2adcca6bb0e465af0bdd38123873ae61eb17a72c2" -dependencies = [ - "once_cell", - "owo-colors", - "tracing-core", - "tracing-error", -] - -[[package]] -name = "colorchoice" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3fd119d74b830634cea2a0f58bbd0d54540518a14397557951e79340abc28c0" - -[[package]] -name = "core-foundation" -version = "0.9.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f" -dependencies = [ - "core-foundation-sys", - "libc", -] - -[[package]] -name = "core-foundation-sys" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" - -[[package]] -name = "crossbeam" -version = "0.8.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1137cd7e7fc0fb5d3c5a8678be38ec56e819125d8d7907411fe24ccb943faca8" -dependencies = [ - "crossbeam-channel", - "crossbeam-deque", - "crossbeam-epoch", - "crossbeam-queue", - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-channel" -version = "0.5.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "33480d6946193aa8033910124896ca395333cae7e2d1113d1fef6c3272217df2" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-deque" -version = "0.8.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "613f8cc01fe9cf1a3eb3d7f488fd2fa8388403e97039e2f73692932e291a770d" -dependencies = [ - "crossbeam-epoch", - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-epoch" -version = "0.9.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-queue" -version = "0.3.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df0346b5d5e76ac2fe4e327c5fd1118d6be7c51dfb18f9b7922923f287471e35" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-utils" -version = "0.8.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22ec99545bb0ed0ea7bb9b8e1e9122ea386ff8a48c0922e43f36d45ab09e0e80" - -[[package]] -name = "ctor" -version = "0.1.26" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d2301688392eb071b0bf1a37be05c469d3cc4dbbd95df672fe28ab021e6a096" -dependencies = [ - "quote", - "syn 1.0.109", -] - -[[package]] -name = "ctrlc" -version = "3.4.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90eeab0aa92f3f9b4e87f258c72b139c207d251f9cbc1080a0086b86a8870dd3" -dependencies = [ - "nix", - "windows-sys 0.59.0", -] - -[[package]] -name = "dashmap" -version = "6.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf" -dependencies = [ - "cfg-if", - "crossbeam-utils", - "hashbrown 0.14.5", - "lock_api", - "once_cell", - "parking_lot_core 0.9.10", -] - -[[package]] -name = "derive_deref" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dcdbcee2d9941369faba772587a565f4f534e42cb8d17e5295871de730163b2b" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "derived-deref" -version = "2.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "805ef2023ccd65425743a91ecd11fc020979a0b01921db3104fb606d18a7b43e" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.85", -] - -[[package]] -name = "either" -version = "1.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60b1af1c220855b6ceac025d3f6ecdd2b7c4894bfe9cd9bda4fbb4bc7c0d4cf0" - -[[package]] -name = "encoding_rs" -version = "0.8.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b45de904aa0b010bce2ab45264d0631681847fa7b6f2eaa7dab7619943bc4f59" -dependencies = [ - "cfg-if", -] - -[[package]] -name = "env_filter" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4f2c92ceda6ceec50f43169f9ee8424fe2db276791afde7b2cd8bc084cb376ab" -dependencies = [ - "log", -] - -[[package]] -name = "env_logger" -version = "0.11.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e13fa619b91fb2381732789fc5de83b45675e882f66623b7d8cb4f643017018d" -dependencies = [ - "env_filter", - "log", -] - -[[package]] -name = "equivalent" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5443807d6dff69373d433ab9ef5378ad8df50ca6298caf15de6e52e24aaf54d5" - -[[package]] -name = "errno" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "534c5cf6194dfab3db3242765c03bbe257cf92f22b38f6bc0c58d59108a820ba" -dependencies = [ - "libc", - "windows-sys 0.52.0", -] - -[[package]] -name = "eyre" -version = "0.6.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7cd915d99f24784cdc19fd37ef22b97e3ff0ae756c7e492e9fbfe897d61e2aec" -dependencies = [ - "indenter", - "once_cell", -] - -[[package]] -name = "fastrand" -version = "2.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e8c02a5121d4ea3eb16a80748c74f5549a5665e4c21333c6098f283870fbdea6" - -[[package]] -name = "fnv" -version = "1.0.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" - -[[package]] -name = "foreign-types" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1" -dependencies = [ - "foreign-types-shared", -] - -[[package]] -name = "foreign-types-shared" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b" - -[[package]] -name = "form_urlencoded" -version = "1.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e13624c2627564efccf4934284bdd98cbaa14e79b0b5a141218e507b3a823456" -dependencies = [ - "percent-encoding", -] - -[[package]] -name = "fs2" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213" -dependencies = [ - "libc", - "winapi", -] - -[[package]] -name = "futures-channel" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2dff15bf788c671c1934e366d07e30c1814a8ef514e1af724a602e8a2fbe1b10" -dependencies = [ - "futures-core", - "futures-sink", -] - -[[package]] -name = "futures-core" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05f29059c0c2090612e8d742178b0580d2dc940c837851ad723096f87af6663e" - -[[package]] -name = "futures-io" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e5c1b78ca4aae1ac06c48a526a655760685149f0d465d21f37abfe57ce075c6" - -[[package]] -name = "futures-sink" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e575fab7d1e0dcb8d0c7bcf9a63ee213816ab51902e6d244a95819acacf1d4f7" - -[[package]] -name = "futures-task" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988" - -[[package]] -name = "futures-util" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9fa08315bb612088cc391249efdc3bc77536f16c91f6cf495e6fbe85b20a4a81" -dependencies = [ - "futures-core", - "futures-io", - "futures-sink", - "futures-task", - "memchr", - "pin-project-lite", - "pin-utils", - "slab", -] - -[[package]] -name = "getrandom" -version = "0.2.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4567c8db10ae91089c99af84c68c38da3ec2f087c3f82960bcdbf3656b6f4d7" -dependencies = [ - "cfg-if", - "libc", - "wasi", -] - -[[package]] -name = "gimli" -version = "0.28.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4271d37baee1b8c7e4b708028c57d816cf9d2434acb33a549475f78c181f6253" - -[[package]] -name = "h2" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "524e8ac6999421f49a846c2d4411f337e53497d8ec55d67753beffa43c5d9205" -dependencies = [ - "atomic-waker", - "bytes", - "fnv", - "futures-core", - "futures-sink", - "http", - "indexmap", - "slab", - "tokio", - "tokio-util", - "tracing", -] - -[[package]] -name = "hashbrown" -version = "0.14.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" - -[[package]] -name = "hashbrown" -version = "0.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e087f84d4f86bf4b218b927129862374b72199ae7d8657835f1e89000eea4fb" - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "hermit-abi" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d231dfb89cfffdbc30e7fc41579ed6066ad03abda9e567ccafae602b97ec5024" - -[[package]] -name = "hermit-abi" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fbf6a919d6cf397374f7dfeeea91d974c7c0a7221d0d0f4f20d859d329e53fcc" - -[[package]] -name = "hex-conservative" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5313b072ce3c597065a808dbf612c4c8e8590bdbf8b579508bf7a762c5eae6cd" -dependencies = [ - "arrayvec", -] - -[[package]] -name = "hex_lit" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3011d1213f159867b13cfd6ac92d2cd5f1345762c63be3554e84092d85a50bbd" - -[[package]] -name = "http" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "21b9ddb458710bc376481b842f5da65cdf31522de232c1ca8146abce2a358258" -dependencies = [ - "bytes", - "fnv", - "itoa", -] - -[[package]] -name = "http-body" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184" -dependencies = [ - "bytes", - "http", -] - -[[package]] -name = "http-body-util" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "793429d76616a256bcb62c2a2ec2bed781c8307e797e2598c50010f2bee2544f" -dependencies = [ - "bytes", - "futures-util", - "http", - "http-body", - "pin-project-lite", -] - -[[package]] -name = "httparse" -version = "1.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d71d3574edd2771538b901e6549113b4006ece66150fb69c0fb6d9a2adae946" - -[[package]] -name = "hyper" -version = "1.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "50dfd22e0e76d0f662d429a5f80fcaf3855009297eab6a0a9f8543834744ba05" -dependencies = [ - "bytes", - "futures-channel", - "futures-util", - "h2", - "http", - "http-body", - "httparse", - "itoa", - "pin-project-lite", - "smallvec", - "tokio", - "want", -] - -[[package]] -name = "hyper-rustls" -version = "0.27.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08afdbb5c31130e3034af566421053ab03787c640246a446327f550d11bcb333" -dependencies = [ - "futures-util", - "http", - "hyper", - "hyper-util", - "rustls", - "rustls-pki-types", - "tokio", - "tokio-rustls", - "tower-service", -] - -[[package]] -name = "hyper-tls" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70206fc6890eaca9fde8a0bf71caa2ddfc9fe045ac9e5c70df101a7dbde866e0" -dependencies = [ - "bytes", - "http-body-util", - "hyper", - "hyper-util", - "native-tls", - "tokio", - "tokio-native-tls", - "tower-service", -] - -[[package]] -name = "hyper-util" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41296eb09f183ac68eec06e03cdbea2e759633d4067b2f6552fc2e009bcad08b" -dependencies = [ - "bytes", - "futures-channel", - "futures-util", - "http", - "http-body", - "hyper", - "pin-project-lite", - "socket2", - "tokio", - "tower-service", - "tracing", -] - -[[package]] -name = "iana-time-zone" -version = "0.1.61" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "235e081f3925a06703c2d0117ea8b91f042756fd6e7a6e5d901e8ca1a996b220" -dependencies = [ - "android_system_properties", - "core-foundation-sys", - "iana-time-zone-haiku", - "js-sys", - "wasm-bindgen", - "windows-core", -] - -[[package]] -name = "iana-time-zone-haiku" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" -dependencies = [ - "cc", -] - -[[package]] -name = "idna" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "634d9b1461af396cad843f47fdba5597a4f9e6ddd4bfb6ff5d85028c25cb12f6" -dependencies = [ - "unicode-bidi", - "unicode-normalization", -] - -[[package]] -name = "indenter" -version = "0.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce23b50ad8242c51a442f3ff322d56b02f08852c77e4c0b4d3fd684abc89c683" - -[[package]] -name = "indexmap" -version = "2.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "707907fe3c25f5424cce2cb7e1cbcafee6bdbe735ca90ef77c29e84591e5b9da" -dependencies = [ - "equivalent", - "hashbrown 0.15.0", -] - -[[package]] -name = "inferno" -version = "0.11.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "232929e1d75fe899576a3d5c7416ad0d88dbfbb3c3d6aa00873a7408a50ddb88" -dependencies = [ - "ahash", - "clap", - "crossbeam-channel", - "crossbeam-utils", - "dashmap", - "env_logger", - "indexmap", - "is-terminal", - "itoa", - "log", - "num-format", - "once_cell", - "quick-xml", - "rgb", - "str_stack", -] - -[[package]] -name = "instant" -version = "0.1.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0242819d153cba4b4b05a5a8f2a7e9bbf97b6055b2a002b395c96b5ff3c0222" -dependencies = [ - "cfg-if", -] - -[[package]] -name = "ipnet" -version = "2.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddc24109865250148c2e0f3d25d4f0f479571723792d3802153c60922a4fb708" - -[[package]] -name = "is-terminal" -version = "0.4.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "261f68e344040fbd0edea105bef17c66edf46f984ddb1115b775ce31be948f4b" -dependencies = [ - "hermit-abi 0.4.0", - "libc", - "windows-sys 0.52.0", -] - -[[package]] -name = "is_terminal_polyfill" -version = "1.70.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7943c866cc5cd64cbc25b2e01621d07fa8eb2a1a23160ee81ce38704e97b8ecf" - -[[package]] -name = "itertools" -version = "0.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186" -dependencies = [ - "either", -] - -[[package]] -name = "itoa" -version = "1.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49f1f14873335454500d59611f1cf4a4b0f786f9ac11f4312a78e4cf2566695b" - -[[package]] -name = "jobserver" -version = "0.1.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48d1dbcbbeb6a7fec7e059840aa538bd62aaccf972c7346c4d9d2059312853d0" -dependencies = [ - "libc", -] - -[[package]] -name = "js-sys" -version = "0.3.70" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1868808506b929d7b0cfa8f75951347aa71bb21144b7791bae35d9bccfcfe37a" -dependencies = [ - "wasm-bindgen", -] - -[[package]] -name = "jsonrpc" -version = "0.18.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3662a38d341d77efecb73caf01420cfa5aa63c0253fd7bc05289ef9f6616e1bf" -dependencies = [ - "base64 0.13.1", - "minreq", - "serde", - "serde_json", -] - -[[package]] -name = "lazy_static" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" - -[[package]] -name = "libc" -version = "0.2.159" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "561d97a539a36e26a9a5fad1ea11a3039a67714694aaa379433e580854bc3dc5" - -[[package]] -name = "linux-raw-sys" -version = "0.4.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78b3ae25bc7c8c38cec158d1f2757ee79e9b3740fbc7ccf0e59e4b08d793fa89" - -[[package]] -name = "lock_api" -version = "0.4.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07af8b9cdd281b7915f413fa73f29ebd5d55d0d3f0155584dade1ff18cea1b17" -dependencies = [ - "autocfg", - "scopeguard", -] - -[[package]] -name = "log" -version = "0.4.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7a70ba024b9dc04c27ea2f0c0548feb474ec5c54bba33a7f72f873a39d07b24" - -[[package]] -name = "memchr" -version = "2.7.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78ca9ab1a0babb1e7d5695e3530886289c18cf2f87ec19a575a0abdce112e3a3" - -[[package]] -name = "memmap2" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fd3f7eed9d3848f8b98834af67102b720745c4ec028fcd0aa0239277e7de374f" -dependencies = [ - "libc", -] - -[[package]] -name = "mime" -version = "0.3.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" - -[[package]] -name = "miniz_oxide" -version = "0.7.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8a240ddb74feaf34a79a7add65a741f3167852fba007066dcac1ca548d89c08" -dependencies = [ - "adler", -] - -[[package]] -name = "minreq" -version = "2.12.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "763d142cdff44aaadd9268bebddb156ef6c65a0e13486bb81673cf2d8739f9b0" -dependencies = [ - "log", - "serde", - "serde_json", -] - -[[package]] -name = "mio" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "80e04d1dcff3aae0704555fe5fee3bcfaf3d1fdf8a7e521d5b9d2b42acb52cec" -dependencies = [ - "hermit-abi 0.3.9", - "libc", - "wasi", - "windows-sys 0.52.0", -] - -[[package]] -name = "native-tls" -version = "0.2.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a8614eb2c83d59d1c8cc974dd3f920198647674a0a035e1af1fa58707e317466" -dependencies = [ - "libc", - "log", - "openssl", - "openssl-probe", - "openssl-sys", - "schannel", - "security-framework", - "security-framework-sys", - "tempfile", -] - -[[package]] -name = "nix" -version = "0.29.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" -dependencies = [ - "bitflags 2.6.0", - "cfg-if", - "cfg_aliases", - "libc", -] - -[[package]] -name = "num-format" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a652d9771a63711fd3c3deb670acfbe5c30a4072e664d7a3bf5a9e1056ac72c3" -dependencies = [ - "arrayvec", - "itoa", -] - -[[package]] -name = "num-traits" -version = "0.2.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" -dependencies = [ - "autocfg", -] - -[[package]] -name = "object" -version = "0.32.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6a622008b6e321afc04970976f62ee297fdbaa6f95318ca343e3eebb9648441" -dependencies = [ - "memchr", -] - -[[package]] -name = "once_cell" -version = "1.20.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1261fe7e33c73b354eab43b1273a57c8f967d0391e80353e51f764ac02cf6775" - -[[package]] -name = "openssl" -version = "0.10.66" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9529f4786b70a3e8c61e11179af17ab6188ad8d0ded78c5529441ed39d4bd9c1" -dependencies = [ - "bitflags 2.6.0", - "cfg-if", - "foreign-types", - "libc", - "once_cell", - "openssl-macros", - "openssl-sys", -] - -[[package]] -name = "openssl-macros" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.85", -] - -[[package]] -name = "openssl-probe" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff011a302c396a5197692431fc1948019154afc178baf7d8e37367442a4601cf" - -[[package]] -name = "openssl-sys" -version = "0.9.103" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f9e8deee91df40a943c71b917e5874b951d32a802526c85721ce3b776c929d6" -dependencies = [ - "cc", - "libc", - "pkg-config", - "vcpkg", -] - -[[package]] -name = "ordered-float" -version = "4.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83e7ccb95e240b7c9506a3d544f10d935e142cc90b0a1d56954fb44d89ad6b97" -dependencies = [ - "num-traits", -] - -[[package]] -name = "owo-colors" -version = "3.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1b04fb49957986fdce4d6ee7a65027d55d4b6d2265e5848bbb507b58ccfdb6f" - -[[package]] -name = "parking_lot" -version = "0.11.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d17b78036a60663b797adeaee46f5c9dfebb86948d1255007a1d6be0271ff99" -dependencies = [ - "instant", - "lock_api", - "parking_lot_core 0.8.6", -] - -[[package]] -name = "parking_lot_core" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60a2cfe6f0ad2bfc16aefa463b497d5c7a5ecd44a23efa72aa342d90177356dc" -dependencies = [ - "cfg-if", - "instant", - "libc", - "redox_syscall 0.2.16", - "smallvec", - "winapi", -] - -[[package]] -name = "parking_lot_core" -version = "0.9.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e401f977ab385c9e4e3ab30627d6f26d00e2c73eef317493c4ec6d468726cf8" -dependencies = [ - "cfg-if", - "libc", - "redox_syscall 0.5.7", - "smallvec", - "windows-targets", -] - -[[package]] -name = "parser" -version = "0.5.0" -dependencies = [ - "allocative", - "bincode", - "bitcoin_hashes", - "biter", - "chrono", - "clap", - "color-eyre", - "ctrlc", - "derive_deref", - "inferno", - "itertools", - "ordered-float", - "rayon", - "reqwest", - "sanakirja", - "serde", - "serde_json", - "struct_iterable", - "toml", - "zstd", -] - -[[package]] -name = "percent-encoding" -version = "2.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3148f5046208a5d56bcfc03053e3ca6334e51da8dfb19b6cdc8b306fae3283e" - -[[package]] -name = "pin-project-lite" -version = "0.2.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bda66fc9667c18cb2758a2ac84d1167245054bcf85d5d1aaa6923f45801bdd02" - -[[package]] -name = "pin-utils" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" - -[[package]] -name = "pkg-config" -version = "0.3.31" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "953ec861398dccce10c670dfeaf3ec4911ca479e9c02154b3a215178c5f566f2" - -[[package]] -name = "ppv-lite86" -version = "0.2.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77957b295656769bb8ad2b6a6b09d897d94f05c41b069aede1fcdaa675eaea04" -dependencies = [ - "zerocopy", -] - -[[package]] -name = "proc-macro2" -version = "1.0.89" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f139b0662de085916d1fb67d2b4169d1addddda1919e696f3252b740b629986e" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "quick-xml" -version = "0.26.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f50b1c63b38611e7d4d7f68b82d3ad0cc71a2ad2e7f61fc10f1328d917c93cd" -dependencies = [ - "memchr", -] - -[[package]] -name = "quote" -version = "1.0.37" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5b9d34b8991d19d98081b46eacdd8eb58c6f2b201139f7c5f643cc155a633af" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "rand" -version = "0.8.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404" -dependencies = [ - "libc", - "rand_chacha", - "rand_core", -] - -[[package]] -name = "rand_chacha" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" -dependencies = [ - "ppv-lite86", - "rand_core", -] - -[[package]] -name = "rand_core" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" -dependencies = [ - "getrandom", -] - -[[package]] -name = "rayon" -version = "1.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b418a60154510ca1a002a752ca9714984e21e4241e804d32555251faf8b78ffa" -dependencies = [ - "either", - "rayon-core", -] - -[[package]] -name = "rayon-core" -version = "1.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1465873a3dfdaa8ae7cb14b4383657caab0b3e8a0aa9ae8e04b044854c8dfce2" -dependencies = [ - "crossbeam-deque", - "crossbeam-utils", -] - -[[package]] -name = "redox_syscall" -version = "0.2.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fb5a58c1855b4b6819d59012155603f0b22ad30cad752600aadfcb695265519a" -dependencies = [ - "bitflags 1.3.2", -] - -[[package]] -name = "redox_syscall" -version = "0.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b6dfecf2c74bce2466cabf93f6664d6998a69eb21e39f4207930065b27b771f" -dependencies = [ - "bitflags 2.6.0", -] - -[[package]] -name = "reqwest" -version = "0.12.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a77c62af46e79de0a562e1a9849205ffcb7fc1238876e9bd743357570e04046f" -dependencies = [ - "base64 0.22.1", - "bytes", - "encoding_rs", - "futures-channel", - "futures-core", - "futures-util", - "h2", - "http", - "http-body", - "http-body-util", - "hyper", - "hyper-rustls", - "hyper-tls", - "hyper-util", - "ipnet", - "js-sys", - "log", - "mime", - "native-tls", - "once_cell", - "percent-encoding", - "pin-project-lite", - "rustls-pemfile", - "serde", - "serde_json", - "serde_urlencoded", - "sync_wrapper", - "system-configuration", - "tokio", - "tokio-native-tls", - "tower-service", - "url", - "wasm-bindgen", - "wasm-bindgen-futures", - "web-sys", - "windows-registry", -] - -[[package]] -name = "rgb" -version = "0.8.50" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "57397d16646700483b67d2dd6511d79318f9d057fdbd21a4066aeac8b41d310a" -dependencies = [ - "bytemuck", -] - -[[package]] -name = "ring" -version = "0.17.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c17fa4cb658e3583423e915b9f3acc01cceaee1860e33d59ebae66adc3a2dc0d" -dependencies = [ - "cc", - "cfg-if", - "getrandom", - "libc", - "spin", - "untrusted", - "windows-sys 0.52.0", -] - -[[package]] -name = "rustc-demangle" -version = "0.1.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "719b953e2095829ee67db738b3bfa9fa368c94900df327b3f07fe6e794d2fe1f" - -[[package]] -name = "rustix" -version = "0.38.37" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8acb788b847c24f28525660c4d7758620a7210875711f79e7f663cc152726811" -dependencies = [ - "bitflags 2.6.0", - "errno", - "libc", - "linux-raw-sys", - "windows-sys 0.52.0", -] - -[[package]] -name = "rustls" -version = "0.23.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "415d9944693cb90382053259f89fbb077ea730ad7273047ec63b19bc9b160ba8" -dependencies = [ - "once_cell", - "rustls-pki-types", - "rustls-webpki", - "subtle", - "zeroize", -] - -[[package]] -name = "rustls-pemfile" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dce314e5fee3f39953d46bb63bb8a46d40c2f8fb7cc5a3b6cab2bde9721d6e50" -dependencies = [ - "rustls-pki-types", -] - -[[package]] -name = "rustls-pki-types" -version = "1.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e696e35370c65c9c541198af4543ccd580cf17fc25d8e05c5a242b202488c55" - -[[package]] -name = "rustls-webpki" -version = "0.102.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64ca1bc8749bd4cf37b5ce386cc146580777b4e8572c7b97baf22c83f444bee9" -dependencies = [ - "ring", - "rustls-pki-types", - "untrusted", -] - -[[package]] -name = "ryu" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3cb5ba0dc43242ce17de99c180e96db90b235b8a9fdc9543c96d2209116bd9f" - -[[package]] -name = "sanakirja" -version = "1.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "81aaf70d064e2122209f04d01fd91e8908e7a327b516236e1cbc0c3f34ac6d11" -dependencies = [ - "fs2", - "log", - "memmap2", - "parking_lot", - "sanakirja-core", - "serde", - "thiserror", -] - -[[package]] -name = "sanakirja-core" -version = "1.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8376db34ae3eac6e7bd91168bc638450073b708ce9fb46940de676f552238bf5" - -[[package]] -name = "schannel" -version = "0.1.26" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "01227be5826fa0690321a2ba6c5cd57a19cf3f6a09e76973b58e61de6ab9d1c1" -dependencies = [ - "windows-sys 0.59.0", -] - -[[package]] -name = "scopeguard" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" - -[[package]] -name = "secp256k1" -version = "0.29.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9465315bc9d4566e1724f0fffcbcc446268cb522e60f9a27bcded6b19c108113" -dependencies = [ - "bitcoin_hashes", - "rand", - "secp256k1-sys", - "serde", -] - -[[package]] -name = "secp256k1-sys" -version = "0.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4387882333d3aa8cb20530a17c69a3752e97837832f34f6dccc760e715001d9" -dependencies = [ - "cc", -] - -[[package]] -name = "security-framework" -version = "2.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "897b2245f0b511c87893af39b033e5ca9cce68824c4d7e7630b5a1d339658d02" -dependencies = [ - "bitflags 2.6.0", - "core-foundation", - "core-foundation-sys", - "libc", - "security-framework-sys", -] - -[[package]] -name = "security-framework-sys" -version = "2.12.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea4a292869320c0272d7bc55a5a6aafaff59b4f63404a003887b679a2e05b4b6" -dependencies = [ - "core-foundation-sys", - "libc", -] - -[[package]] -name = "serde" -version = "1.0.214" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f55c3193aca71c12ad7890f1785d2b73e1b9f63a0bbc353c08ef26fe03fc56b5" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.214" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de523f781f095e28fa605cdce0f8307e451cc0fd14e2eb4cd2e98a355b147766" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.85", -] - -[[package]] -name = "serde_json" -version = "1.0.132" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d726bfaff4b320266d395898905d0eba0345aae23b54aee3a737e260fd46db03" -dependencies = [ - "itoa", - "memchr", - "ryu", - "serde", -] - -[[package]] -name = "serde_spanned" -version = "0.6.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "87607cb1398ed59d48732e575a4c28a7a8ebf2454b964fe3f224f2afc07909e1" -dependencies = [ - "serde", -] - -[[package]] -name = "serde_urlencoded" -version = "0.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" -dependencies = [ - "form_urlencoded", - "itoa", - "ryu", - "serde", -] - -[[package]] -name = "sharded-slab" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" -dependencies = [ - "lazy_static", -] - -[[package]] -name = "shlex" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" - -[[package]] -name = "slab" -version = "0.4.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f92a496fb766b417c996b9c5e57daf2f7ad3b0bebe1ccfca4856390e3d3bb67" -dependencies = [ - "autocfg", -] - -[[package]] -name = "smallvec" -version = "1.13.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c5e1a9a646d36c3599cd173a41282daf47c44583ad367b8e6837255952e5c67" - -[[package]] -name = "socket2" -version = "0.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce305eb0b4296696835b71df73eb912e0f1ffd2556a501fcede6e0c50349191c" -dependencies = [ - "libc", - "windows-sys 0.52.0", -] - -[[package]] -name = "spin" -version = "0.9.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" - -[[package]] -name = "str_stack" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9091b6114800a5f2141aee1d1b9d6ca3592ac062dc5decb3764ec5895a47b4eb" - -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "struct_iterable" -version = "0.1.2" -dependencies = [ - "struct_iterable_derive", - "struct_iterable_internal", -] - -[[package]] -name = "struct_iterable_derive" -version = "0.1.0" -dependencies = [ - "proc-macro2", - "quote", - "struct_iterable_internal", - "syn 2.0.85", -] - -[[package]] -name = "struct_iterable_internal" -version = "0.1.1" - -[[package]] -name = "subtle" -version = "2.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" - -[[package]] -name = "syn" -version = "1.0.109" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "2.0.85" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5023162dfcd14ef8f32034d8bcd4cc5ddc61ef7a247c024a33e24e1f24d21b56" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "sync_wrapper" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7065abeca94b6a8a577f9bd45aa0867a2238b74e8eb67cf10d492bc39351394" -dependencies = [ - "futures-core", -] - -[[package]] -name = "system-configuration" -version = "0.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c879d448e9d986b661742763247d3693ed13609438cf3d006f51f5368a5ba6b" -dependencies = [ - "bitflags 2.6.0", - "core-foundation", - "system-configuration-sys", -] - -[[package]] -name = "system-configuration-sys" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e1d1b10ced5ca923a1fcb8d03e96b8d3268065d724548c0211415ff6ac6bac4" -dependencies = [ - "core-foundation-sys", - "libc", -] - -[[package]] -name = "tempfile" -version = "3.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0f2c9fc62d0beef6951ccffd757e241266a2c833136efbe35af6cd2567dca5b" -dependencies = [ - "cfg-if", - "fastrand", - "once_cell", - "rustix", - "windows-sys 0.59.0", -] - -[[package]] -name = "thiserror" -version = "1.0.64" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d50af8abc119fb8bb6dbabcfa89656f46f84aa0ac7688088608076ad2b459a84" -dependencies = [ - "thiserror-impl", -] - -[[package]] -name = "thiserror-impl" -version = "1.0.64" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08904e7672f5eb876eaaf87e0ce17857500934f4981c4a0ab2b4aa98baac7fc3" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.85", -] - -[[package]] -name = "thread_local" -version = "1.1.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b9ef9bad013ada3808854ceac7b46812a6465ba368859a37e2100283d2d719c" -dependencies = [ - "cfg-if", - "once_cell", -] - -[[package]] -name = "tinyvec" -version = "1.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "445e881f4f6d382d5f27c034e25eb92edd7c784ceab92a0937db7f2e9471b938" -dependencies = [ - "tinyvec_macros", -] - -[[package]] -name = "tinyvec_macros" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" - -[[package]] -name = "tokio" -version = "1.40.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2b070231665d27ad9ec9b8df639893f46727666c6767db40317fbe920a5d998" -dependencies = [ - "backtrace", - "bytes", - "libc", - "mio", - "pin-project-lite", - "socket2", - "windows-sys 0.52.0", -] - -[[package]] -name = "tokio-native-tls" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bbae76ab933c85776efabc971569dd6119c580d8f5d448769dec1764bf796ef2" -dependencies = [ - "native-tls", - "tokio", -] - -[[package]] -name = "tokio-rustls" -version = "0.26.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c7bc40d0e5a97695bb96e27995cd3a08538541b0a846f65bba7a359f36700d4" -dependencies = [ - "rustls", - "rustls-pki-types", - "tokio", -] - -[[package]] -name = "tokio-util" -version = "0.7.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61e7c3654c13bcd040d4a03abee2c75b1d14a37b423cf5a813ceae1cc903ec6a" -dependencies = [ - "bytes", - "futures-core", - "futures-sink", - "pin-project-lite", - "tokio", -] - -[[package]] -name = "toml" -version = "0.8.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1ed1f98e3fdc28d6d910e6737ae6ab1a93bf1985935a1193e68f93eeb68d24e" -dependencies = [ - "serde", - "serde_spanned", - "toml_datetime", - "toml_edit", -] - -[[package]] -name = "toml_datetime" -version = "0.6.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0dd7358ecb8fc2f8d014bf86f6f638ce72ba252a2c3a2572f2a795f1d23efb41" -dependencies = [ - "serde", -] - -[[package]] -name = "toml_edit" -version = "0.22.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ae48d6208a266e853d946088ed816055e556cc6028c5e8e2b84d9fa5dd7c7f5" -dependencies = [ - "indexmap", - "serde", - "serde_spanned", - "toml_datetime", - "winnow", -] - -[[package]] -name = "tower-service" -version = "0.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" - -[[package]] -name = "tracing" -version = "0.1.40" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c3523ab5a71916ccf420eebdf5521fcef02141234bbc0b8a49f2fdc4544364ef" -dependencies = [ - "pin-project-lite", - "tracing-core", -] - -[[package]] -name = "tracing-core" -version = "0.1.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c06d3da6113f116aaee68e4d601191614c9053067f9ab7f6edbcb161237daa54" -dependencies = [ - "once_cell", - "valuable", -] - -[[package]] -name = "tracing-error" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d686ec1c0f384b1277f097b2f279a2ecc11afe8c133c1aabf036a27cb4cd206e" -dependencies = [ - "tracing", - "tracing-subscriber", -] - -[[package]] -name = "tracing-subscriber" -version = "0.3.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad0f048c97dbd9faa9b7df56362b8ebcaa52adb06b498c050d2f4e32f90a7a8b" -dependencies = [ - "sharded-slab", - "thread_local", - "tracing-core", -] - -[[package]] -name = "try-lock" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" - -[[package]] -name = "unicode-bidi" -version = "0.3.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ab17db44d7388991a428b2ee655ce0c212e862eff1768a455c58f9aad6e7893" - -[[package]] -name = "unicode-ident" -version = "1.0.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e91b56cd4cadaeb79bbf1a5645f6b4f8dc5bde8834ad5894a8db35fda9efa1fe" - -[[package]] -name = "unicode-normalization" -version = "0.1.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5033c97c4262335cded6d6fc3e5c18ab755e1a3dc96376350f3d8e9f009ad956" -dependencies = [ - "tinyvec", -] - -[[package]] -name = "untrusted" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" - -[[package]] -name = "unty" -version = "0.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1a88342087869553c259588a3ec9ca73ce9b2d538b7051ba5789ff236b6c129" - -[[package]] -name = "url" -version = "2.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22784dbdf76fdde8af1aeda5622b546b422b6fc585325248a2bf9f5e41e94d6c" -dependencies = [ - "form_urlencoded", - "idna", - "percent-encoding", -] - -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - -[[package]] -name = "valuable" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "830b7e5d4d90034032940e4ace0d9a9a057e7a45cd94e6c007832e39edb82f6d" - -[[package]] -name = "vcpkg" -version = "0.2.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" - -[[package]] -name = "version_check" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" - -[[package]] -name = "virtue" -version = "0.0.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7302ac74a033bf17b6e609ceec0f891ca9200d502d31f02dc7908d3d98767c9d" - -[[package]] -name = "want" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" -dependencies = [ - "try-lock", -] - -[[package]] -name = "wasi" -version = "0.11.0+wasi-snapshot-preview1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423" - -[[package]] -name = "wasm-bindgen" -version = "0.2.93" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a82edfc16a6c469f5f44dc7b571814045d60404b55a0ee849f9bcfa2e63dd9b5" -dependencies = [ - "cfg-if", - "once_cell", - "wasm-bindgen-macro", -] - -[[package]] -name = "wasm-bindgen-backend" -version = "0.2.93" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9de396da306523044d3302746f1208fa71d7532227f15e347e2d93e4145dd77b" -dependencies = [ - "bumpalo", - "log", - "once_cell", - "proc-macro2", - "quote", - "syn 2.0.85", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-futures" -version = "0.4.43" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61e9300f63a621e96ed275155c108eb6f843b6a26d053f122ab69724559dc8ed" -dependencies = [ - "cfg-if", - "js-sys", - "wasm-bindgen", - "web-sys", -] - -[[package]] -name = "wasm-bindgen-macro" -version = "0.2.93" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "585c4c91a46b072c92e908d99cb1dcdf95c5218eeb6f3bf1efa991ee7a68cccf" -dependencies = [ - "quote", - "wasm-bindgen-macro-support", -] - -[[package]] -name = "wasm-bindgen-macro-support" -version = "0.2.93" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "afc340c74d9005395cf9dd098506f7f44e38f2b4a21c6aaacf9a105ea5e1e836" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.85", - "wasm-bindgen-backend", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-shared" -version = "0.2.93" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c62a0a307cb4a311d3a07867860911ca130c3494e8c2719593806c08bc5d0484" - -[[package]] -name = "web-sys" -version = "0.3.70" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26fdeaafd9bd129f65e7c031593c24d62186301e0c72c8978fa1678be7d532c0" -dependencies = [ - "js-sys", - "wasm-bindgen", -] - -[[package]] -name = "winapi" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" -dependencies = [ - "winapi-i686-pc-windows-gnu", - "winapi-x86_64-pc-windows-gnu", -] - -[[package]] -name = "winapi-i686-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" - -[[package]] -name = "winapi-x86_64-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" - -[[package]] -name = "windows-core" -version = "0.52.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "33ab640c8d7e35bf8ba19b884ba838ceb4fba93a4e8c65a9059d08afcfc683d9" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-registry" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e400001bb720a623c1c69032f8e3e4cf09984deec740f007dd2b03ec864804b0" -dependencies = [ - "windows-result", - "windows-strings", - "windows-targets", -] - -[[package]] -name = "windows-result" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d1043d8214f791817bab27572aaa8af63732e11bf84aa21a45a78d6c317ae0e" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-strings" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4cd9b125c486025df0eabcb585e62173c6c9eddcec5d117d3b6e8c30e2ee4d10" -dependencies = [ - "windows-result", - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.52.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-sys" -version = "0.59.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" -dependencies = [ - "windows-targets", -] - -[[package]] -name = "windows-targets" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" -dependencies = [ - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" - -[[package]] -name = "windows_i686_gnu" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" - -[[package]] -name = "windows_i686_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" - -[[package]] -name = "winnow" -version = "0.6.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "36c1fec1a2bb5866f07c25f68c26e565c4c200aebb96d7e55710c19d3e8ac49b" -dependencies = [ - "memchr", -] - -[[package]] -name = "zerocopy" -version = "0.7.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1b9b4fd18abc82b8136838da5d50bae7bdea537c574d8dc1a34ed098d6c166f0" -dependencies = [ - "byteorder", - "zerocopy-derive", -] - -[[package]] -name = "zerocopy-derive" -version = "0.7.35" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa4f8080344d4671fb4e831a13ad1e68092748387dfc4f55e356242fae12ce3e" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.85", -] - -[[package]] -name = "zeroize" -version = "1.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ced3678a2879b30306d323f4542626697a464a97c0a07c9aebf7ebca65cd4dde" - -[[package]] -name = "zstd" -version = "0.13.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fcf2b778a664581e31e389454a7072dab1647606d44f7feea22cd5abb9c9f3f9" -dependencies = [ - "zstd-safe", -] - -[[package]] -name = "zstd-safe" -version = "7.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "54a3ab4db68cea366acc5c897c7b4d4d1b8994a9cd6e6f841f8964566a419059" -dependencies = [ - "zstd-sys", -] - -[[package]] -name = "zstd-sys" -version = "2.0.13+zstd.1.5.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "38ff0f21cfee8f97d94cef41359e0c89aa6113028ab0291aa8ca0038995a95aa" -dependencies = [ - "cc", - "pkg-config", -] diff --git a/parser/Cargo.toml b/parser/Cargo.toml deleted file mode 100644 index bf12d64de..000000000 --- a/parser/Cargo.toml +++ /dev/null @@ -1,32 +0,0 @@ -[package] -name = "parser" -version = "0.5.0" -edition = "2021" - -# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html - -[dependencies] -allocative = "0.3.3" -bincode = { git = "https://github.com/bincode-org/bincode.git", features = [ - "serde", -] } -bitcoin_hashes = { version = "0.14.0" } -biter = { path = "../biter" } -chrono = { version = "0.4.38", features = ["serde"] } -clap = { version = "4.5.20", features = ["derive"] } -color-eyre = "0.6.3" -ctrlc = { version = "3.4.5", features = ["termination"] } -derive_deref = "1.1.1" -inferno = "0.11.21" -itertools = "0.13.0" -ordered-float = "4.4.0" -rayon = "1.10.0" -reqwest = { version = "0.12.9", features = ["blocking", "json"] } -sanakirja = "1.4.3" -serde = { version = "1.0.214", features = ["derive"] } -serde_json = "1.0.132" -struct_iterable = { path = "../iterable" } -toml = "0.8.19" -zstd = "0.13.2" -# memory-stats = "1.2.0" -# sysinfo = "0.32.0" diff --git a/parser/reset.sh b/parser/reset.sh deleted file mode 100755 index d7a64be38..000000000 --- a/parser/reset.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash - -echo "Deleting datasets..." -rm -r ../datasets -echo "Deleting states and databases..." -rm -r ./out -echo "Done." diff --git a/parser/samply.sh b/parser/samply.sh deleted file mode 100755 index 6376c139a..000000000 --- a/parser/samply.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/usr/bin/env bash - -echo "Increasing limit of opened files..." -ulimit -n 1000000 - -# Needed because the datasets tree is too big lol -echo "Increasing stack size..." -ulimit -s $(ulimit -Hs) - -cargo build --profile profiling && samply record ./target/profiling/parser "$HOME/Developer/bitcoin" diff --git a/parser/src/databases/_trait.rs b/parser/src/databases/_trait.rs deleted file mode 100644 index acf040d0b..000000000 --- a/parser/src/databases/_trait.rs +++ /dev/null @@ -1,52 +0,0 @@ -use std::{fs, io, path::PathBuf}; - -use crate::{ - io::OUTPUTS_FOLDER_PATH, - structs::{Date, Height}, - utils::log, -}; - -use super::AnyDatabase; - -pub trait AnyDatabaseGroup -where - Self: Sized, -{ - fn init() -> Self { - let s = Self::import(); - s.create_dir_all().unwrap(); - s - } - - fn import() -> Self; - - fn folder<'a>() -> &'a str; - - fn drain_to_vec(&mut self) -> Vec>; - fn open_all(&mut self); - - fn export_metadata(&mut self, height: Height, date: Date) -> color_eyre::Result<()>; - - fn create_dir_all(&self) -> color_eyre::Result<(), io::Error>; - - fn remove_dir_all(&self) -> color_eyre::Result<(), io::Error> { - fs::remove_dir_all(Self::root()) - } - - fn reset(&mut self) -> color_eyre::Result<(), io::Error> { - log(&format!("Reset {}", Self::folder())); - - self.reset_metadata(); - self.remove_dir_all()?; - self.create_dir_all()?; - - Ok(()) - } - - fn reset_metadata(&mut self); - - fn root() -> PathBuf { - let folder = Self::folder(); - PathBuf::from(format!("{OUTPUTS_FOLDER_PATH}/databases/{folder}")) - } -} diff --git a/parser/src/io/consts.rs b/parser/src/io/consts.rs deleted file mode 100644 index 750f7edb0..000000000 --- a/parser/src/io/consts.rs +++ /dev/null @@ -1,2 +0,0 @@ -pub const INPUTS_FOLDER_PATH: &str = "./in"; -pub const OUTPUTS_FOLDER_PATH: &str = "./out"; diff --git a/parser/src/lib.rs b/parser/src/lib.rs deleted file mode 100644 index c0b5efd29..000000000 --- a/parser/src/lib.rs +++ /dev/null @@ -1,19 +0,0 @@ -mod actions; -mod databases; -mod datasets; -mod io; -mod price; -mod states; -mod structs; -mod utils; - -pub use crate::{ - actions::iter_blocks, - databases::{AnyDatabase, Database}, - io::{Binary, Json, Serialization, COMPRESSED_BIN_EXTENSION, JSON_EXTENSION}, - structs::{ - Amount, Config, Date, DateMap, Exit, Height, HeightMap, MapChunkId, MapValue, - SerializedBTreeMap, SerializedVec, TxoutIndex, HEIGHT_MAP_CHUNK_SIZE, OHLC, - }, - utils::{create_rpc, log, reset_logs}, -}; diff --git a/parser/src/main.rs b/parser/src/main.rs deleted file mode 100644 index a95380569..000000000 --- a/parser/src/main.rs +++ /dev/null @@ -1,36 +0,0 @@ -use std::{thread::sleep, time::Duration}; - -use biter::bitcoincore_rpc::RpcApi; -use parser::{create_rpc, iter_blocks, log, reset_logs, Config, Exit}; - -fn main() -> color_eyre::Result<()> { - color_eyre::install()?; - - reset_logs(); - - let mut config = Config::import()?; - - let rpc = create_rpc(&config).unwrap(); - - let exit = Exit::new(); - - loop { - let block_count = rpc.get_blockchain_info().unwrap().blocks as usize; - - log(&format!("{block_count} blocks found.")); - - iter_blocks(&mut config, &rpc, block_count, exit.clone())?; - - if let Some(delay) = config.delay { - sleep(Duration::from_secs(delay)) - } - - log("Waiting for a new block...\n"); - - while block_count == rpc.get_blockchain_info().unwrap().blocks as usize { - sleep(Duration::from_secs(1)) - } - } - - // Ok(()) -} diff --git a/parser/src/states/_trait.rs b/parser/src/states/_trait.rs deleted file mode 100644 index 2b82fbd94..000000000 --- a/parser/src/states/_trait.rs +++ /dev/null @@ -1,48 +0,0 @@ -use std::{fmt::Debug, fs, io, path::Path}; - -use bincode::{Decode, Encode}; -use serde::{de::DeserializeOwned, Serialize}; - -use crate::{io::OUTPUTS_FOLDER_PATH, Serialization}; - -// https://github.com/djkoloski/rust_serialization_benchmark -pub trait AnyState -where - Self: Debug + Encode + Decode + Serialize + DeserializeOwned, -{ - fn name<'a>() -> &'a str; - - fn create_dir_all() -> color_eyre::Result<(), io::Error> { - fs::create_dir_all(Self::folder_path()) - } - - fn folder_path() -> String { - format!("{OUTPUTS_FOLDER_PATH}/states") - } - - fn full_path() -> String { - let name = Self::name(); - - let folder_path = Self::folder_path(); - - format!("{folder_path}/{name}") - } - - fn reset(&mut self) -> color_eyre::Result<(), io::Error> { - self.clear(); - - fs::remove_file(Self::full_path()) - } - - fn import() -> color_eyre::Result { - Self::create_dir_all()?; - - Serialization::Binary.import(Path::new(&Self::full_path())) - } - - fn export(&self) -> color_eyre::Result<()> { - Serialization::Binary.export(Path::new(&Self::full_path()), self) - } - - fn clear(&mut self); -} diff --git a/parser/src/utils/log.rs b/parser/src/utils/log.rs deleted file mode 100644 index d6593f75f..000000000 --- a/parser/src/utils/log.rs +++ /dev/null @@ -1,34 +0,0 @@ -use std::{ - fs::{self, OpenOptions}, - io::Write, -}; - -use chrono::Local; -use color_eyre::owo_colors::OwoColorize; - -const LOG_PATH: &str = "./.log"; - -pub fn reset_logs() { - let _ = fs::remove_file(LOG_PATH); -} - -#[inline(always)] -pub fn log(str: &str) { - let date_time = format!("{}", Local::now().format("%Y-%m-%d %H:%M:%S -")); - - str.lines() - .filter(|line| !line.is_empty()) - .for_each(|line| { - let mut file = OpenOptions::new() - .create(true) - .append(true) - .open(LOG_PATH) - .unwrap(); - - if let Err(e) = writeln!(file, "{} {}", date_time, line) { - eprintln!("Couldn't write to file: {}", e); - } - - println!("{} {}", date_time.bright_black(), line); - }); -} diff --git a/parser/run.sh b/run.sh similarity index 89% rename from parser/run.sh rename to run.sh index b82977f49..24531425a 100755 --- a/parser/run.sh +++ b/run.sh @@ -4,7 +4,7 @@ if command -v ulimit &> /dev/null; then echo "Increasing limit of opened files..." - ulimit -n 1000000 # Can't be $(ulimit -Hn), bitcoind needs some too ! + # ulimit -n 1000000 # Can't be $(ulimit -Hn), bitcoind needs some too ! # Needed because the datasets tree is too big lol echo "Increasing stack size..." diff --git a/server/Cargo.toml b/server/Cargo.toml deleted file mode 100644 index 92c9921f3..000000000 --- a/server/Cargo.toml +++ /dev/null @@ -1,22 +0,0 @@ -[package] -name = "server" -version = "0.5.0" -edition = "2021" - -[dependencies] -axum = "0.7.7" -bincode = { git = "https://github.com/bincode-org/bincode.git" } -chrono = "0.4.38" -color-eyre = "0.6.3" -derive_deref = "1.1.1" -itertools = "0.13.0" -parser = { path = "../parser" } -regex = "1.11.0" -reqwest = { version = "0.12.8", features = ["json"] } -serde = { version = "1.0.210", features = ["derive"] } -serde_json = { version = "1.0.128" } -swc = "0.289.1" -swc_common = "0.40.1" -tokio = { version = "1.40.0", features = ["full"] } -tower-http = { version = "0.6.1", features = ["compression-full"] } -# oxc = { version = "0.34.0", features = ["codegen", "minifier"] } diff --git a/server/restart-cloudflared.sh b/server/restart-cloudflared.sh deleted file mode 100755 index fa1695f7a..000000000 --- a/server/restart-cloudflared.sh +++ /dev/null @@ -1,4 +0,0 @@ -sudo launchctl stop com.cloudflare.cloudflared -sudo launchctl unload /Library/LaunchDaemons/com.cloudflare.cloudflared.plist -sudo launchctl load /Library/LaunchDaemons/com.cloudflare.cloudflared.plist -sudo launchctl start com.cloudflare.cloudflared diff --git a/server/run.sh b/server/run.sh deleted file mode 100755 index 77a22baac..000000000 --- a/server/run.sh +++ /dev/null @@ -1,14 +0,0 @@ -#!/usr/bin/env bash - -if cargo watch --help &> /dev/null; then - TRIGGER="./in/datasets_len.txt" - - if [ ! -f "$TRIGGER" ]; then - mkdir "./in" - echo "0" > $TRIGGER - fi - - cargo watch --no-vcs-ignores -w "./src" -w "$TRIGGER" -x "run -r" -else - cargo run -r -fi diff --git a/server/src/api/structs/routes.rs b/server/src/api/structs/routes.rs deleted file mode 100644 index fc33fbf5d..000000000 --- a/server/src/api/structs/routes.rs +++ /dev/null @@ -1,157 +0,0 @@ -use std::{ - collections::{BTreeMap, HashMap}, - fs, - path::{Path, PathBuf}, -}; - -use derive_deref::{Deref, DerefMut}; -use itertools::Itertools; -use parser::{Json, Serialization}; - -use crate::Grouped; - -use super::Paths; - -#[derive(Clone, Debug)] -pub struct Route { - pub url_path: String, - pub file_path: PathBuf, - pub values_type: String, - pub serialization: Serialization, -} - -#[derive(Clone, Default, Deref, DerefMut)] -pub struct Routes(pub Grouped>); - -const INPUTS_PATH: &str = "./in"; -const WEBSITE_TYPES_PATH: &str = "../website/scripts/types"; - -impl Routes { - pub fn build() -> Self { - let path_to_type: BTreeMap = - Json::import(Path::new(&format!("{INPUTS_PATH}/disk_path_to_type.json"))).unwrap(); - - let mut routes = Routes::default(); - - path_to_type.into_iter().for_each(|(key, value)| { - let mut split_key = key.split('/').collect_vec(); - let last = split_key.pop().unwrap().to_owned(); - - let mut skip = 2; - - let mut serialization = Serialization::Binary; - - if *split_key.get(1).unwrap() == "price" { - skip = 1; - serialization = Serialization::Json; - } - - let mut split_key = split_key.iter().skip(skip).collect_vec(); - - // Use case for: "../datasets/last": "Value", - if split_key.is_empty() { - split_key.push(&"last"); - } - - let map_key = split_key.iter().join("_"); - - let url_path = split_key.iter().join("-"); - - let file_path = PathBuf::from(key.to_owned()); - let values_type = value.to_owned(); - - if last == "date" { - routes.date.insert( - map_key, - Route { - url_path: format!("date-to-{url_path}"), - file_path, - values_type, - serialization, - }, - ); - } else if last == "height" { - routes.height.insert( - map_key, - Route { - url_path: format!("height-to-{url_path}"), - file_path, - values_type, - serialization, - }, - ); - } else if last == "last" { - routes.last.insert( - map_key, - Route { - url_path, - file_path, - values_type, - serialization, - }, - ); - } else { - dbg!(&key, value, &last); - panic!("") - } - }); - - routes - } - - pub fn generate_dts_file(&self) { - let map_to_type = |name: &str, map: &HashMap| -> String { - let paths = map - .values() - .map(|route| format!("\"{}\"", route.url_path)) - .join(" | "); - - format!("export type {}Path = {};\n", name, paths) - }; - - let date_type = map_to_type("Date", &self.date); - - let height_type = map_to_type("Height", &self.height); - - let last_type = map_to_type("Last", &self.last); - - fs::write( - format!("{WEBSITE_TYPES_PATH}/paths.d.ts"), - format!("// This file is auto generated by the server\n// Manual changes are forbidden\n\n{date_type}\n{height_type}\n{last_type}"), - ) - .unwrap(); - } - - pub fn to_full_paths(&self, host: String) -> Paths { - let url = { - let scheme = if host.contains("0.0.0.0") || host.contains("localhost") { - "http" - } else { - "https" - }; - - format!("{scheme}://{host}") - }; - - let transform = |map: &HashMap| -> BTreeMap { - map.iter() - .map(|(key, route)| { - ( - key.to_owned(), - format!("{url}/api/{}", route.url_path.to_owned()), - ) - }) - .collect() - }; - - let date_paths = transform(&self.date); - let height_paths = transform(&self.height); - let last_paths = transform(&self.last); - - Paths(Grouped { - date: date_paths, - height: height_paths, - last: last_paths, - }) - } -} diff --git a/parser/src/io/binary.rs b/src/io/binary.rs similarity index 100% rename from parser/src/io/binary.rs rename to src/io/binary.rs diff --git a/parser/src/io/json.rs b/src/io/json.rs similarity index 100% rename from parser/src/io/json.rs rename to src/io/json.rs diff --git a/parser/src/io/mod.rs b/src/io/mod.rs similarity index 77% rename from parser/src/io/mod.rs rename to src/io/mod.rs index 674784567..a0540ad92 100644 --- a/parser/src/io/mod.rs +++ b/src/io/mod.rs @@ -1,9 +1,7 @@ mod binary; -mod consts; mod json; mod serialization; pub use binary::*; -pub use consts::*; pub use json::*; pub use serialization::*; diff --git a/parser/src/io/serialization.rs b/src/io/serialization.rs similarity index 89% rename from parser/src/io/serialization.rs rename to src/io/serialization.rs index e9cd8137d..9dea77275 100644 --- a/parser/src/io/serialization.rs +++ b/src/io/serialization.rs @@ -1,4 +1,8 @@ -use std::{fmt::Debug, fs, path::Path}; +use std::{ + fmt::Debug, + fs, + path::{Path, PathBuf}, +}; use allocative::Allocative; use bincode::{Decode, Encode}; @@ -27,17 +31,6 @@ impl Serialization { } } - pub fn from_path(path: &Path) -> Self { - let path = path.to_str().unwrap(); - if path.ends_with(BIN_EXTENSION) || path.ends_with(COMPRESSED_BIN_EXTENSION) { - Self::Binary - } else if path.ends_with(JSON_EXTENSION) || path.ends_with(HAR_EXTENSION) { - Self::Json - } else { - panic!("Extension \"{path}\" isn't supported") - } - } - pub fn import(&self, path: &Path) -> color_eyre::Result where T: Debug + DeserializeOwned + Decode, @@ -119,3 +112,17 @@ impl Serialization { } } } + +impl TryFrom<&PathBuf> for Serialization { + type Error = (); + + fn try_from(path: &PathBuf) -> Result { + let extension = path.extension().ok_or(())?.to_str().unwrap(); + + match extension { + BIN_EXTENSION | COMPRESSED_BIN_EXTENSION => Ok(Self::Binary), + JSON_EXTENSION => Ok(Self::Json), + _ => Err(()), + } + } +} diff --git a/src/main.rs b/src/main.rs new file mode 100644 index 000000000..97bdef558 --- /dev/null +++ b/src/main.rs @@ -0,0 +1,70 @@ +use biter::bitcoincore_rpc::Client; +use log::info; +use rlimit::{getrlimit, setrlimit, Resource}; + +mod io; +mod parser; +mod server; +mod structs; +mod utils; + +use parser::{Databases, Datasets}; +use structs::{Config, Exit}; +use utils::init_log; + +fn main() -> color_eyre::Result<()> { + color_eyre::install()?; + + init_log(); + + let (_, nofile_limit) = getrlimit(Resource::NOFILE).unwrap(); + setrlimit(Resource::NOFILE, 138_240, nofile_limit)?; + + std::thread::Builder::new() + .stack_size(getrlimit(Resource::STACK).unwrap().1 as usize) + .spawn(|| -> color_eyre::Result<()> { + let exit = Exit::new(); + + let config = Config::import()?; + + info!("Starting..."); + + let rpc = Client::from(&config); + + let databases = Databases::import(&config); + + let datasets = Datasets::import(&config)?; + + let paths_to_type = datasets.get_paths_to_type(&config); + + tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .unwrap() + .block_on(async { + let config_clone = config.clone(); + let run_parser = config.parser(); + let run_server = config.server(); + + let handle = tokio::spawn(async move { + if run_server { + server::main(paths_to_type, &config_clone).await.unwrap(); + } else { + info!("Skipping server"); + } + }); + + if run_parser { + parser::main(&config, &rpc, &exit, databases, datasets)?; + } else { + info!("Skipping parser"); + } + + handle.await?; + + Ok(()) + }) + })? + .join() + .unwrap() +} diff --git a/parser/src/actions/export.rs b/src/parser/actions/export.rs similarity index 52% rename from parser/src/actions/export.rs rename to src/parser/actions/export.rs index 88af14a76..f43d8e3be 100644 --- a/parser/src/actions/export.rs +++ b/src/parser/actions/export.rs @@ -1,52 +1,62 @@ use std::thread::{self}; +use log::info; + use crate::{ - databases::Databases, - datasets::AllDatasets, - states::States, - structs::{Date, Height}, - utils::{log, time}, - Exit, + parser::{databases::Databases, datasets::Datasets, states::States}, + structs::{Config, Date, Exit, Height}, + utils::time, }; pub struct ExportedData<'a> { + pub config: &'a Config, pub databases: Option<&'a mut Databases>, - pub datasets: &'a mut AllDatasets, + pub datasets: &'a mut Datasets, pub date: Date, + pub defragment: bool, + pub exit: Exit, pub height: Height, pub states: Option<&'a States>, - pub exit: Exit, } pub fn export( ExportedData { + config, databases, datasets, - states, - height, date, + defragment, exit, + height, + states, }: ExportedData, ) -> color_eyre::Result<()> { if exit.active() { - log("Exit in progress, skipping export"); + info!("Exit in progress, skipping export"); return Ok(()); } exit.block(); - log("Exporting..."); + info!("Exporting..."); + if defragment { + info!("Will also defragment databases, please be patient it might take a while") + } time("Total save time", || -> color_eyre::Result<()> { - time("Datasets saved", || datasets.export())?; + time("Datasets saved", || datasets.export(config))?; thread::scope(|s| { if let Some(databases) = databases { - s.spawn(|| time("Databases saved", || databases.export(height, date))); + s.spawn(|| { + time("Databases saved", || { + databases.export(height, date, defragment) + }) + }); } if let Some(states) = states { - s.spawn(|| time("States saved", || states.export())); + s.spawn(|| time("States saved", || states.export(config))); } }); diff --git a/parser/src/actions/iter_blocks.rs b/src/parser/actions/iter_blocks.rs similarity index 78% rename from parser/src/actions/iter_blocks.rs rename to src/parser/actions/iter_blocks.rs index aaf46e35e..84e5a4af4 100644 --- a/parser/src/actions/iter_blocks.rs +++ b/src/parser/actions/iter_blocks.rs @@ -1,63 +1,52 @@ use std::{collections::BTreeSet, time::Instant}; +use biter::bitcoincore_rpc::Client; +use chrono::Datelike; use export::ExportedData; use itertools::Itertools; +use log::info; use parse::ParseData; use crate::{ - actions::{export, find_first_inserted_unsafe_height, parse}, - create_rpc, - databases::Databases, - datasets::{AllDatasets, ComputeData}, - io::OUTPUTS_FOLDER_PATH, - states::{AddressCohortsDurableStates, States, UTXOCohortsDurableStates}, - structs::{DateData, MapKey, Timestamp}, - utils::{generate_allocation_files, log, time}, - Config, Exit, Height, + parser::{ + actions::{export, find_first_inserted_unsafe_height, parse}, + databases::Databases, + datasets::{ComputeData, Datasets}, + states::{AddressCohortsDurableStates, States, UTXOCohortsDurableStates}, + }, + structs::{Config, DateData, Exit, Height, MapKey, Timestamp}, + utils::{generate_allocation_files, time}, }; pub fn iter_blocks( - config: &mut Config, + config: &Config, rpc: &biter::bitcoincore_rpc::Client, approx_block_count: usize, exit: Exit, + databases: &mut Databases, + datasets: &mut Datasets, ) -> color_eyre::Result<()> { - log("Starting..."); + let mut states = States::import(config).unwrap_or_default(); - let mut datasets = AllDatasets::import(config)?; - - log("Imported datasets"); - - let mut databases = Databases::import(); - - if config.first_defragment() { - databases.defragment(&exit); - config.disable_defragment(); - } - - log("Imported databases"); - - let mut states = States::import().unwrap_or_default(); - - log("Imported states"); + info!("Imported states"); let first_unsafe_heights = - find_first_inserted_unsafe_height(&mut states, &mut databases, &mut datasets); + find_first_inserted_unsafe_height(&mut states, databases, datasets, config); let mut height = first_unsafe_heights.min(); - log(&format!("Starting parsing at height: {height}")); + info!("Starting parsing at height: {height}"); let mut next_block_opt = None; let mut blocks_loop_date = None; + let mut next_date_opt; let block_receiver = biter::new( - config.datadir.as_ref().unwrap(), - OUTPUTS_FOLDER_PATH, + &config.path_bitcoindir(), Some(height.to_usize()), None, - create_rpc(config).unwrap(), + Client::from(config), ); let mut block_iter = block_receiver.iter(); @@ -93,15 +82,13 @@ pub fn iter_blocks( panic!() } - let next_block_date = next_block_opt.as_ref().map(|(_, next_block, _)| { + next_date_opt = next_block_opt.as_ref().map(|(_, next_block, _)| { Timestamp::wrap(next_block.header.time).to_date() }); // Always run for the first block of the loop if blocks_loop_date.is_none() { - log(&format!( - "Processing {current_block_date} (height: {height})..." - )); + info!("Processing {current_block_date} (height: {height})..."); blocks_loop_date.replace(current_block_date); @@ -125,7 +112,7 @@ pub fn iter_blocks( panic!("current block should always have the same date as the current blocks loop"); } - let is_date_last_block = next_block_date + let is_date_last_block = next_date_opt // Do NOT change `blocks_loop_date` to `current_block_date` !!! .map_or(true, |next_block_date| blocks_loop_date < next_block_date); @@ -159,16 +146,17 @@ pub fn iter_blocks( } parse(ParseData { - rpc, block: current_block, block_index: blocks_loop_i, compute_addresses, - databases: &mut databases, - datasets: &mut datasets, + config, + databases, + datasets, date: blocks_loop_date, first_date_height: height, height: current_block_height, is_date_last_block, + rpc, states: &mut states, }); } @@ -178,7 +166,7 @@ pub fn iter_blocks( if is_date_last_block { height += blocks_loop_i; - let is_check_point = next_block_date + let is_check_point = next_date_opt .as_ref() .map_or(true, |date| date.is_first_of_month()); @@ -201,13 +189,13 @@ pub fn iter_blocks( // Don't remember why -1 let last_height = height - 1_u32; - log(&format!( - "Parsing group took {} seconds (last height: {last_height})\n", + info!( + "Parsing group took {} seconds (last height: {last_height})", instant.elapsed().as_secs_f32(), - )); + ); if first_unsafe_heights.computed <= last_height { - log("Computing datasets..."); + info!("Computing datasets..."); time("Computing datasets", || { let dates = processed_dates.into_iter().collect_vec(); @@ -223,10 +211,17 @@ pub fn iter_blocks( if !config.dry_run() { let is_safe = height.is_safe(approx_block_count); + let defragment = is_safe + && next_date_opt.is_some_and(|date| { + date.year() >= 2020 && date.is_january() && date.is_first_of_month() + }); + export(ExportedData { - databases: is_safe.then_some(&mut databases), - datasets: &mut datasets, + config, + databases: is_safe.then_some(databases), + datasets, date: blocks_loop_date.unwrap(), + defragment, height: last_height, states: is_safe.then_some(&states), exit: exit.clone(), @@ -234,11 +229,11 @@ pub fn iter_blocks( if config.record_ram_usage() { time("Exporing allocation files", || { - generate_allocation_files(&datasets, &databases, &states, last_height) + generate_allocation_files(datasets, databases, &states, last_height) })?; } } else { - log("Skipping export"); + info!("Skipping export"); } println!(); diff --git a/parser/src/actions/min_height.rs b/src/parser/actions/min_height.rs similarity index 84% rename from parser/src/actions/min_height.rs rename to src/parser/actions/min_height.rs index 953c1ddf1..eb767c08b 100644 --- a/parser/src/actions/min_height.rs +++ b/src/parser/actions/min_height.rs @@ -1,9 +1,12 @@ +use log::info; + use crate::{ - databases::Databases, - datasets::{AllDatasets, AnyDatasets}, - states::States, - structs::Height, - utils::log, + parser::{ + databases::Databases, + datasets::{AnyDatasets, Datasets}, + states::States, + }, + structs::{Config, Height}, }; #[derive(Default, Debug)] @@ -21,7 +24,8 @@ impl Heights { pub fn find_first_inserted_unsafe_height( states: &mut States, databases: &mut Databases, - datasets: &mut AllDatasets, + datasets: &mut Datasets, + config: &Config, ) -> Heights { let min_initial_inserted_last_address_height = datasets .address @@ -51,7 +55,7 @@ pub fn find_first_inserted_unsafe_height( .map(|date_data| date_data.date) .and_then(|last_safe_date| { if !usable_databases { - log("Unusable databases"); + info!("Unusable databases"); return None; } @@ -61,8 +65,8 @@ pub fn find_first_inserted_unsafe_height( let min_datasets_inserted_last_height = datasets_min_initial_states.inserted.last_height; let min_datasets_inserted_last_date = datasets_min_initial_states.inserted.last_date; - log(&format!("min_datasets_inserted_last_height: {:?}", min_datasets_inserted_last_height)); - log(&format!("min_datasets_inserted_last_date: {:?}", min_datasets_inserted_last_date)); + info!("min_datasets_inserted_last_height: {:?}", min_datasets_inserted_last_height); + info!("min_datasets_inserted_last_date: {:?}", min_datasets_inserted_last_date); let inserted_last_date_is_older_than_saved_state = min_datasets_inserted_last_date.map_or(true, |min_datasets_last_date| min_datasets_last_date < last_safe_date); @@ -80,7 +84,7 @@ pub fn find_first_inserted_unsafe_height( let inserted_heights_and_dates_are_out_of_sync = min_datasets_inserted_last_height.map_or(true, |min_datasets_inserted_last_height| min_datasets_inserted_last_height < last_safe_height); if inserted_heights_and_dates_are_out_of_sync { - log(&format!("last_safe_height ({last_safe_height}) > min_datasets_height ({min_datasets_inserted_last_height:?})")); + info!("last_safe_height ({last_safe_height}) > min_datasets_height ({min_datasets_inserted_last_height:?})"); None } else { @@ -108,7 +112,7 @@ pub fn find_first_inserted_unsafe_height( ) }) .unwrap_or_else(|| { - log("Starting over..."); + info!("Starting over..."); let include_addresses = !usable_databases || min_initial_inserted_last_address_date.is_none() @@ -119,7 +123,7 @@ pub fn find_first_inserted_unsafe_height( // panic!(""); // } - states.reset(include_addresses); + states.reset(config, include_addresses); databases.reset(include_addresses); diff --git a/parser/src/actions/mod.rs b/src/parser/actions/mod.rs similarity index 100% rename from parser/src/actions/mod.rs rename to src/parser/actions/mod.rs diff --git a/parser/src/actions/parse.rs b/src/parser/actions/parse.rs similarity index 98% rename from parser/src/actions/parse.rs rename to src/parser/actions/parse.rs index 2199a5c3b..648bfb53e 100644 --- a/parser/src/actions/parse.rs +++ b/src/parser/actions/parse.rs @@ -9,18 +9,21 @@ use itertools::Itertools; use rayon::prelude::*; use crate::{ - databases::{ - AddressIndexToAddressData, AddressIndexToEmptyAddressData, AddressToAddressIndex, - Databases, TxidToTxData, TxoutIndexToAddressIndex, TxoutIndexToAmount, - }, - datasets::{AllDatasets, InsertData}, - states::{ - AddressCohortsInputStates, AddressCohortsOutputStates, AddressCohortsRealizedStates, - States, UTXOCohortsOneShotStates, UTXOCohortsSentStates, + parser::{ + databases::{ + AddressIndexToAddressData, AddressIndexToEmptyAddressData, AddressToAddressIndex, + Databases, TxidToTxData, TxoutIndexToAddressIndex, TxoutIndexToAmount, + }, + datasets::{Datasets, InsertData}, + states::{ + AddressCohortsInputStates, AddressCohortsOutputStates, AddressCohortsRealizedStates, + States, UTXOCohortsOneShotStates, UTXOCohortsSentStates, + }, }, structs::{ - Address, AddressData, AddressRealizedData, Amount, BlockData, BlockPath, Counter, Date, - EmptyAddressData, Height, PartialTxoutData, Price, SentData, Timestamp, TxData, TxoutIndex, + Address, AddressData, AddressRealizedData, Amount, BlockData, BlockPath, Config, Counter, + Date, EmptyAddressData, Height, PartialTxoutData, Price, SentData, Timestamp, TxData, + TxoutIndex, }, }; @@ -28,9 +31,10 @@ pub struct ParseData<'a> { // pub bitcoin_cli: &'a BitcoinCli, pub block: Block, pub block_index: usize, + pub config: &'a Config, pub compute_addresses: bool, pub databases: &'a mut Databases, - pub datasets: &'a mut AllDatasets, + pub datasets: &'a mut Datasets, pub date: Date, pub first_date_height: Height, pub height: Height, @@ -43,6 +47,7 @@ pub fn parse( ParseData { block, block_index, + config, compute_addresses, databases, datasets, @@ -72,7 +77,7 @@ pub fn parse( let block_price = Price::from_dollar( datasets .price - .get_height_ohlc(height, timestamp, previous_timestamp) + .get_height_ohlc(height, timestamp, previous_timestamp, config) .unwrap_or_else(|_| panic!("Expect {height} to have a price")) .close as f64, ); diff --git a/parser/src/databases/_database.rs b/src/parser/databases/_database.rs similarity index 76% rename from parser/src/databases/_database.rs rename to src/parser/databases/_database.rs index abc695f56..bd61f1078 100644 --- a/parser/src/databases/_database.rs +++ b/src/parser/databases/_database.rs @@ -1,35 +1,35 @@ +// https://docs.rs/sanakirja/latest/sanakirja/index.html +// https://pijul.org/posts/2021-02-06-rethinking-sanakirja/ + use std::{ collections::{BTreeMap, BTreeSet}, fmt::Debug, - fs, mem, + fs, io, mem, path::PathBuf, }; use allocative::Allocative; -// https://docs.rs/sanakirja/latest/sanakirja/index.html -// https://pijul.org/posts/2021-02-06-rethinking-sanakirja/ -// -// Seems indeed much faster than ReDB and LMDB (heed) -// But a lot has changed code wise between them so a retest wouldn't hurt -// -// Possible compression: https://pijul.org/posts/sanakirja-zstd/ use sanakirja::{ btree::{self, page, Db_, Iter}, Commit, Env, Error, MutTxn, RootDb, Storable, }; +/// +/// Simple wrapper around Sanakirja Database with cached puts and dels for safe use outside exports. +/// +/// There is no `cached_gets` since it's much cheaper and faster to do a parallel search first using `unsafe_get` than caching "gets" along the way. +/// #[derive(Allocative)] #[allocative(bound = "Key: Allocative, Value: Allocative")] -/// There is no `cached_gets` since it's much cheaper and faster to do a parallel search first using `unsafe_get` than caching gets along the way. pub struct Database where Key: Ord + Clone + Debug + Storable, Value: Storable + PartialEq, { - pub cached_puts: BTreeMap, - pub cached_dels: BTreeSet, path: PathBuf, + cached_puts: BTreeMap, + cached_dels: BTreeSet, #[allocative(skip)] db: Db_>, #[allocative(skip)] @@ -62,11 +62,12 @@ where }) } + #[inline] pub fn iter(&self) -> Iter<'_, MutTxn, Key, Value, page::Page> { btree::iter(&self.txn, &self.db, None).unwrap() } - pub fn iter_collect(&self) -> BTreeMap + pub fn collect(&self) -> BTreeMap where Value: Clone, { @@ -76,6 +77,7 @@ where .collect::<_>() } + #[inline] pub fn get(&self, key: &Key) -> Option<&Value> { if let Some(cached_put) = self.get_from_puts(key) { return Some(cached_put); @@ -84,6 +86,7 @@ where self.db_get(key) } + #[inline] pub fn db_get(&self, key: &Key) -> Option<&Value> { let option = btree::get(&self.txn, &self.db, key, None).unwrap(); @@ -96,17 +99,17 @@ where None } - #[inline(always)] + #[inline] pub fn get_from_puts(&self, key: &Key) -> Option<&Value> { self.cached_puts.get(key) } - #[inline(always)] + #[inline] pub fn get_mut_from_puts(&mut self, key: &Key) -> Option<&mut Value> { self.cached_puts.get_mut(key) } - #[inline(always)] + #[inline] pub fn remove(&mut self, key: &Key) -> Option { self.remove_from_puts(key).or_else(|| { self.db_remove(key); @@ -139,7 +142,6 @@ where #[inline] pub fn insert(&mut self, key: Key, value: Value) -> Option { self.cached_dels.remove(&key); - self.unsafe_insert(key, value) } @@ -165,12 +167,11 @@ where } pub trait AnyDatabase { - fn export(self) -> color_eyre::Result<(), Error>; - fn boxed_export(self: Box) -> color_eyre::Result<(), Error>; #[allow(unused)] - fn defragment(self); - fn boxed_defragment(self: Box); - fn destroy(self); + fn export(self, defragment: bool) -> color_eyre::Result<(), Error>; + fn boxed_export(self: Box, defragment: bool) -> color_eyre::Result<(), Error>; + #[allow(unused)] + fn destroy(self) -> io::Result<()>; } impl AnyDatabase for Database @@ -178,11 +179,31 @@ where Key: Ord + Clone + Debug + Storable, Value: Storable + PartialEq + Clone, { - fn export(self) -> color_eyre::Result<(), Error> { - Box::new(self).boxed_export() + fn export(self, defragment: bool) -> color_eyre::Result<(), Error> { + Box::new(self).boxed_export(defragment) } - fn boxed_export(mut self: Box) -> color_eyre::Result<(), Error> { + fn boxed_export(mut self: Box, defragment: bool) -> color_eyre::Result<(), Error> { + if defragment { + let mut btree = self.as_ref().collect(); + + let path = self.path.to_owned(); + self.cached_dels.iter().for_each(|key| { + btree.remove(key); + }); + btree.append(&mut self.cached_puts); + + self.destroy()?; + + *self = Self::open(path).unwrap(); + + if !self.is_empty() { + panic!() + } + + self.cached_puts = btree; + } + if self.cached_dels.is_empty() && self.cached_puts.is_empty() { return Ok(()); } @@ -198,35 +219,11 @@ where self.txn.commit() } - fn defragment(self) { - Box::new(self).boxed_defragment() - } - - fn boxed_defragment(self: Box) { - let btree = self.iter_collect(); - - let path = self.path.to_owned(); - - self.destroy(); - - let mut db = Self::open(path).unwrap(); - - if !db.is_empty() { - panic!() - } - - db.cached_puts = btree; - db.export().unwrap(); - } - - fn destroy(self) { + fn destroy(self) -> io::Result<()> { let path = self.path.to_owned(); drop(self); - fs::remove_file(&path).unwrap_or_else(|_| { - dbg!(path); - panic!("Error"); - }); + fs::remove_file(&path) } } diff --git a/src/parser/databases/_trait.rs b/src/parser/databases/_trait.rs new file mode 100644 index 000000000..f238c82b1 --- /dev/null +++ b/src/parser/databases/_trait.rs @@ -0,0 +1,59 @@ +use std::{fs, io, path::Path}; + +use log::info; + +use crate::structs::{Config, Date, Height}; + +use super::{AnyDatabase, Metadata}; + +pub trait AnyDatabaseGroup +where + Self: Sized, +{ + fn init(config: &Config) -> Self { + let s = Self::import(config); + s.create_dir_all().unwrap(); + s + } + + fn import(config: &Config) -> Self; + + fn drain_to_vec(&mut self) -> Vec>; + fn open_all(&mut self); + + fn metadata(&mut self) -> &mut Metadata; + fn export_metadata(&mut self, height: Height, date: Date) -> color_eyre::Result<()> { + self.metadata().export(height, date) + } + + fn create_dir_all(&self) -> color_eyre::Result<(), std::io::Error> { + fs::create_dir_all(self.path()) + } + + fn remove_dir_all(&self) -> color_eyre::Result<(), io::Error> { + fs::remove_dir_all(self.path()) + } + + fn reset(&mut self) -> color_eyre::Result<(), io::Error> { + info!( + "Reset {}", + self.path() + .components() + .last() + .unwrap() + .as_os_str() + .to_str() + .unwrap() + ); + + self.reset_metadata(); + self.remove_dir_all()?; + self.create_dir_all()?; + + Ok(()) + } + + fn reset_metadata(&mut self); + + fn path(&self) -> &Path; +} diff --git a/parser/src/databases/address_index_to_address_data.rs b/src/parser/databases/address_index_to_address_data.rs similarity index 86% rename from parser/src/databases/address_index_to_address_data.rs rename to src/parser/databases/address_index_to_address_data.rs index e6ed7fc80..82c90acf0 100644 --- a/parser/src/databases/address_index_to_address_data.rs +++ b/src/parser/databases/address_index_to_address_data.rs @@ -2,6 +2,7 @@ use std::{ collections::BTreeMap, fs, mem, ops::{Deref, DerefMut}, + path::{Path, PathBuf}, }; use allocative::Allocative; @@ -9,8 +10,8 @@ use itertools::Itertools; use rayon::prelude::*; use crate::{ - states::AddressCohortsDurableStates, - structs::{AddressData, Date, Height}, + parser::states::AddressCohortsDurableStates, + structs::{AddressData, Config}, utils::time, }; @@ -22,8 +23,8 @@ type Database = _Database; #[derive(Allocative)] pub struct AddressIndexToAddressData { + path: PathBuf, pub metadata: Metadata, - pub map: BTreeMap, } @@ -72,6 +73,7 @@ impl AddressIndexToAddressData { pub fn open_db(&mut self, key: &Key) -> &mut Database { let db_index = Self::db_index(key); + let path = self.path().to_owned(); self.entry(db_index).or_insert_with(|| { let db_name = format!( @@ -80,7 +82,7 @@ impl AddressIndexToAddressData { (db_index + 1) * ADDRESS_INDEX_DB_MAX_SIZE ); - let path = Self::root().join(db_name); + let path = path.join(db_name); Database::open(path).unwrap() }) @@ -113,30 +115,23 @@ impl AddressIndexToAddressData { } impl AnyDatabaseGroup for AddressIndexToAddressData { - fn import() -> Self { + fn import(config: &Config) -> Self { + let path = config + .path_databases() + .join("address_index_to_address_data"); Self { - metadata: Metadata::import(Self::root(), 1), - + metadata: Metadata::import(&path, 1), + path, map: BTreeMap::default(), } } - fn create_dir_all(&self) -> color_eyre::Result<(), std::io::Error> { - fs::create_dir_all(Self::root()) - } - fn reset_metadata(&mut self) { self.metadata.reset(); } - fn folder<'a>() -> &'a str { - "address_index_to_address_data" - } - fn open_all(&mut self) { - let path = Self::root(); - - let folder = fs::read_dir(path); + let folder = fs::read_dir(&self.path); if folder.is_err() { return; @@ -167,7 +162,11 @@ impl AnyDatabaseGroup for AddressIndexToAddressData { .collect_vec() } - fn export_metadata(&mut self, height: Height, date: Date) -> color_eyre::Result<()> { - self.metadata.export(height, date) + fn metadata(&mut self) -> &mut Metadata { + &mut self.metadata + } + + fn path(&self) -> &Path { + &self.path } } diff --git a/parser/src/databases/address_index_to_empty_address_data.rs b/src/parser/databases/address_index_to_empty_address_data.rs similarity index 84% rename from parser/src/databases/address_index_to_empty_address_data.rs rename to src/parser/databases/address_index_to_empty_address_data.rs index cdb0127fc..923ae4850 100644 --- a/parser/src/databases/address_index_to_empty_address_data.rs +++ b/src/parser/databases/address_index_to_empty_address_data.rs @@ -2,12 +2,13 @@ use std::{ collections::BTreeMap, fs, mem, ops::{Deref, DerefMut}, + path::{Path, PathBuf}, }; use allocative::Allocative; use itertools::Itertools; -use crate::structs::{Date, EmptyAddressData, Height}; +use crate::structs::{Config, EmptyAddressData}; use super::{ AnyDatabase, AnyDatabaseGroup, Database as _Database, Metadata, ADDRESS_INDEX_DB_MAX_SIZE, @@ -19,8 +20,8 @@ type Database = _Database; #[derive(Allocative)] pub struct AddressIndexToEmptyAddressData { + path: PathBuf, pub metadata: Metadata, - map: BTreeMap, } @@ -72,6 +73,7 @@ impl AddressIndexToEmptyAddressData { pub fn open_db(&mut self, key: &Key) -> &mut Database { let db_index = Self::db_index(key); + let path = self.path.to_owned(); self.entry(db_index).or_insert_with(|| { let db_name = format!( @@ -80,7 +82,7 @@ impl AddressIndexToEmptyAddressData { (db_index + 1) * ADDRESS_INDEX_DB_MAX_SIZE ); - let path = Self::root().join(db_name); + let path = path.join(db_name); Database::open(path).unwrap() }) @@ -92,30 +94,23 @@ impl AddressIndexToEmptyAddressData { } impl AnyDatabaseGroup for AddressIndexToEmptyAddressData { - fn import() -> Self { + fn import(config: &Config) -> Self { + let path = config + .path_databases() + .join("address_index_to_empty_address_data"); Self { - metadata: Metadata::import(Self::root(), 1), - + metadata: Metadata::import(&path, 1), + path, map: BTreeMap::default(), } } - fn create_dir_all(&self) -> color_eyre::Result<(), std::io::Error> { - fs::create_dir_all(Self::root()) - } - fn reset_metadata(&mut self) { self.metadata.reset(); } - fn folder<'a>() -> &'a str { - "address_index_to_empty_address_data" - } - fn open_all(&mut self) { - let path = Self::root(); - - let folder = fs::read_dir(path); + let folder = fs::read_dir(&self.path); if folder.is_err() { return; @@ -146,7 +141,11 @@ impl AnyDatabaseGroup for AddressIndexToEmptyAddressData { .collect_vec() } - fn export_metadata(&mut self, height: Height, date: Date) -> color_eyre::Result<()> { - self.metadata.export(height, date) + fn metadata(&mut self) -> &mut Metadata { + &mut self.metadata + } + + fn path(&self) -> &Path { + &self.path } } diff --git a/parser/src/databases/address_to_address_index.rs b/src/parser/databases/address_to_address_index.rs similarity index 86% rename from parser/src/databases/address_to_address_index.rs rename to src/parser/databases/address_to_address_index.rs index 34b7e8d5b..35d9b7e30 100644 --- a/parser/src/databases/address_to_address_index.rs +++ b/src/parser/databases/address_to_address_index.rs @@ -7,7 +7,7 @@ use std::{ use allocative::Allocative; use itertools::Itertools; -use crate::structs::{Address, Date, Height, U8x19, U8x31}; +use crate::structs::{Address, Config, U8x19, U8x31}; use super::{AnyDatabase, AnyDatabaseGroup, Database, Metadata}; @@ -30,6 +30,7 @@ type MultisigDatabase = U32Database; #[derive(Allocative)] pub struct AddressToAddressIndex { + path: PathBuf, pub metadata: Metadata, p2pk: BTreeMap, @@ -160,12 +161,12 @@ impl AddressToAddressIndex { .collect_vec() } - fn path_p2pk() -> PathBuf { - Self::root().join("p2pk") + fn path_p2pk(&self) -> PathBuf { + self.path().join("p2pk") } pub fn open_p2pk(&mut self, prefix: u16) -> &mut P2PKDatabase { - let path = Self::path_p2pk(); + let path = self.path_p2pk(); self.p2pk.entry(prefix).or_insert_with(|| { let path = path.join(prefix.to_string()); Database::open(path).unwrap() @@ -173,7 +174,7 @@ impl AddressToAddressIndex { } fn open_all_p2pk(&mut self) { - let path = Self::path_p2pk(); + let path = self.path_p2pk(); Self::path_to_group_prefixes(&path) .into_iter() .for_each(|prefix| { @@ -184,12 +185,12 @@ impl AddressToAddressIndex { }); } - fn path_p2pkh() -> PathBuf { - Self::root().join("p2pkh") + fn path_p2pkh(&self) -> PathBuf { + self.path().join("p2pkh") } pub fn open_p2pkh(&mut self, prefix: u16) -> &mut P2PKHDatabase { - let path = Self::path_p2pkh(); + let path = self.path_p2pkh(); self.p2pkh.entry(prefix).or_insert_with(|| { let path = path.join(prefix.to_string()); @@ -198,7 +199,7 @@ impl AddressToAddressIndex { } fn open_all_p2pkh(&mut self) { - let path = Self::path_p2pkh(); + let path = self.path_p2pkh(); Self::path_to_group_prefixes(&path) .into_iter() .for_each(|prefix| { @@ -209,12 +210,12 @@ impl AddressToAddressIndex { }); } - fn path_p2sh() -> PathBuf { - Self::root().join("p2sh") + fn path_p2sh(&self) -> PathBuf { + self.path().join("p2sh") } pub fn open_p2sh(&mut self, prefix: u16) -> &mut P2SHDatabase { - let path = Self::path_p2sh(); + let path = self.path_p2sh(); self.p2sh.entry(prefix).or_insert_with(|| { let path = path.join(prefix.to_string()); @@ -223,7 +224,7 @@ impl AddressToAddressIndex { } fn open_all_p2sh(&mut self) { - let path = Self::path_p2sh(); + let path = self.path_p2sh(); Self::path_to_group_prefixes(&path) .into_iter() .for_each(|prefix| { @@ -234,12 +235,12 @@ impl AddressToAddressIndex { }); } - fn path_p2wpkh() -> PathBuf { - Self::root().join("p2wpkh") + fn path_p2wpkh(&self) -> PathBuf { + self.path().join("p2wpkh") } pub fn open_p2wpkh(&mut self, prefix: u16) -> &mut P2WPKHDatabase { - let path = Self::path_p2wpkh(); + let path = self.path_p2wpkh(); self.p2wpkh.entry(prefix).or_insert_with(|| { let path = path.join(prefix.to_string()); @@ -248,7 +249,7 @@ impl AddressToAddressIndex { } fn open_all_p2wpkh(&mut self) { - let path = Self::path_p2wpkh(); + let path = self.path_p2wpkh(); Self::path_to_group_prefixes(&path) .into_iter() .for_each(|prefix| { @@ -259,12 +260,12 @@ impl AddressToAddressIndex { }); } - fn path_p2wsh() -> PathBuf { - Self::root().join("p2wsh") + fn path_p2wsh(&self) -> PathBuf { + self.path().join("p2wsh") } pub fn open_p2wsh(&mut self, prefix: u16) -> &mut P2WSHDatabase { - let path = Self::path_p2wsh(); + let path = self.path_p2wsh(); self.p2wsh.entry(prefix).or_insert_with(|| { let path = path.join(prefix.to_string()); @@ -273,7 +274,7 @@ impl AddressToAddressIndex { } fn open_all_p2wsh(&mut self) { - let path = Self::path_p2wsh(); + let path = self.path_p2wsh(); Self::path_to_group_prefixes(&path) .into_iter() .for_each(|prefix| { @@ -284,12 +285,12 @@ impl AddressToAddressIndex { }); } - fn path_p2tr() -> PathBuf { - Self::root().join("p2tr") + fn path_p2tr(&self) -> PathBuf { + self.path().join("p2tr") } pub fn open_p2tr(&mut self, prefix: u16) -> &mut P2TRDatabase { - let path = Self::path_p2tr(); + let path = self.path_p2tr(); self.p2tr.entry(prefix).or_insert_with(|| { let path = path.join(prefix.to_string()); @@ -298,7 +299,7 @@ impl AddressToAddressIndex { } fn open_all_p2tr(&mut self) { - let path = Self::path_p2tr(); + let path = self.path_p2tr(); Self::path_to_group_prefixes(&path) .into_iter() .for_each(|prefix| { @@ -311,34 +312,36 @@ impl AddressToAddressIndex { pub fn open_unknown(&mut self) -> &mut UnknownDatabase { self.unknown - .get_or_insert_with(|| Database::open(Self::root().join("unknown")).unwrap()) + .get_or_insert_with(|| Database::open(self.path.join("unknown")).unwrap()) } pub fn open_op_return(&mut self) -> &mut UnknownDatabase { self.op_return - .get_or_insert_with(|| Database::open(Self::root().join("op_return")).unwrap()) + .get_or_insert_with(|| Database::open(self.path.join("op_return")).unwrap()) } pub fn open_push_only(&mut self) -> &mut UnknownDatabase { self.push_only - .get_or_insert_with(|| Database::open(Self::root().join("push_only")).unwrap()) + .get_or_insert_with(|| Database::open(self.path.join("push_only")).unwrap()) } pub fn open_empty(&mut self) -> &mut UnknownDatabase { self.empty - .get_or_insert_with(|| Database::open(Self::root().join("empty")).unwrap()) + .get_or_insert_with(|| Database::open(self.path.join("empty")).unwrap()) } pub fn open_multisig(&mut self) -> &mut MultisigDatabase { self.multisig - .get_or_insert_with(|| Database::open(Self::root().join("multisig")).unwrap()) + .get_or_insert_with(|| Database::open(self.path.join("multisig")).unwrap()) } } impl AnyDatabaseGroup for AddressToAddressIndex { - fn import() -> Self { + fn import(config: &Config) -> Self { + let path = config.path_databases().join("address_to_address_index"); Self { - metadata: Metadata::import(Self::root(), 1), + metadata: Metadata::import(&path, 1), + path, p2pk: BTreeMap::default(), p2pkh: BTreeMap::default(), @@ -355,22 +358,18 @@ impl AnyDatabaseGroup for AddressToAddressIndex { } fn create_dir_all(&self) -> color_eyre::Result<(), std::io::Error> { - fs::create_dir_all(Self::path_p2pk()).unwrap(); - fs::create_dir_all(Self::path_p2pkh()).unwrap(); - fs::create_dir_all(Self::path_p2sh()).unwrap(); - fs::create_dir_all(Self::path_p2wpkh()).unwrap(); - fs::create_dir_all(Self::path_p2wsh()).unwrap(); - fs::create_dir_all(Self::path_p2tr()) + fs::create_dir_all(self.path_p2pk()).unwrap(); + fs::create_dir_all(self.path_p2pkh()).unwrap(); + fs::create_dir_all(self.path_p2sh()).unwrap(); + fs::create_dir_all(self.path_p2wpkh()).unwrap(); + fs::create_dir_all(self.path_p2wsh()).unwrap(); + fs::create_dir_all(self.path_p2tr()) } fn reset_metadata(&mut self) { self.metadata.reset() } - fn folder<'a>() -> &'a str { - "address_to_address_index" - } - fn drain_to_vec(&mut self) -> Vec> { mem::take(&mut self.p2pk) .into_values() @@ -424,7 +423,11 @@ impl AnyDatabaseGroup for AddressToAddressIndex { self.open_all_p2tr(); } - fn export_metadata(&mut self, height: Height, date: Date) -> color_eyre::Result<()> { - self.metadata.export(height, date) + fn metadata(&mut self) -> &mut Metadata { + &mut self.metadata + } + + fn path(&self) -> &Path { + &self.path } } diff --git a/parser/src/databases/metadata.rs b/src/parser/databases/metadata.rs similarity index 94% rename from parser/src/databases/metadata.rs rename to src/parser/databases/metadata.rs index eb0046c5d..9d69c4817 100644 --- a/parser/src/databases/metadata.rs +++ b/src/parser/databases/metadata.rs @@ -10,8 +10,8 @@ use std::{ }; use crate::{ + io::Serialization, structs::{Counter, Date, Height}, - Serialization, }; #[derive(Default, Debug, Encode, Decode, Allocative)] @@ -35,10 +35,10 @@ impl DerefMut for Metadata { } impl Metadata { - pub fn import(path: PathBuf, version: u16) -> Self { + pub fn import(path: &Path, version: u16) -> Self { Self { - data: MetadataData::import(&path, version), - path, + data: MetadataData::import(path, version), + path: path.to_owned(), } } diff --git a/parser/src/databases/mod.rs b/src/parser/databases/mod.rs similarity index 78% rename from parser/src/databases/mod.rs rename to src/parser/databases/mod.rs index 9ff6bc752..9b32a982c 100644 --- a/parser/src/databases/mod.rs +++ b/src/parser/databases/mod.rs @@ -1,5 +1,3 @@ -use std::thread::{self}; - use allocative::Allocative; mod _database; @@ -18,18 +16,14 @@ pub use address_index_to_address_data::*; pub use address_index_to_empty_address_data::*; pub use address_to_address_index::*; use itertools::Itertools; +use log::info; use metadata::*; use rayon::iter::{IntoParallelIterator, ParallelIterator}; pub use txid_to_tx_data::*; pub use txout_index_to_address_index::*; pub use txout_index_to_amount::*; -use crate::{ - log, - structs::{Date, Height}, - utils::time, - Exit, -}; +use crate::structs::{Config, Date, Height}; #[derive(Allocative)] pub struct Databases { @@ -42,18 +36,20 @@ pub struct Databases { } impl Databases { - pub fn import() -> Self { - let address_index_to_address_data = AddressIndexToAddressData::init(); + pub fn import(config: &Config) -> Self { + let address_index_to_address_data = AddressIndexToAddressData::init(config); - let address_index_to_empty_address_data = AddressIndexToEmptyAddressData::init(); + let address_index_to_empty_address_data = AddressIndexToEmptyAddressData::init(config); - let address_to_address_index = AddressToAddressIndex::init(); + let address_to_address_index = AddressToAddressIndex::init(config); - let txid_to_tx_data = TxidToTxData::init(); + let txid_to_tx_data = TxidToTxData::init(config); - let txout_index_to_address_index = TxoutIndexToAddressIndex::init(); + let txout_index_to_address_index = TxoutIndexToAddressIndex::init(config); - let txout_index_to_amount = TxoutIndexToAmount::init(); + let txout_index_to_amount = TxoutIndexToAmount::init(config); + + info!("Imported databases"); Self { address_index_to_address_data, @@ -91,62 +87,21 @@ impl Databases { Ok(()) } - pub fn export(&mut self, height: Height, date: Date) -> color_eyre::Result<()> { + pub fn export( + &mut self, + height: Height, + date: Date, + defragment: bool, + ) -> color_eyre::Result<()> { self.export_metadata(height, date)?; self.drain_to_vec() .into_par_iter() - .try_for_each(AnyDatabase::boxed_export)?; + .try_for_each(|s| AnyDatabase::boxed_export(s, defragment))?; Ok(()) } - fn open_all(&mut self) { - thread::scope(|s| { - s.spawn(|| { - self.address_index_to_address_data.open_all(); - }); - - s.spawn(|| { - self.address_index_to_empty_address_data.open_all(); - }); - - s.spawn(|| { - self.address_to_address_index.open_all(); - }); - - s.spawn(|| { - self.txid_to_tx_data.open_all(); - }); - - s.spawn(|| { - self.txout_index_to_address_index.open_all(); - }); - - s.spawn(|| { - self.txout_index_to_amount.open_all(); - }); - }); - } - - pub fn defragment(&mut self, exit: &Exit) { - exit.block(); - - log("Databases defragmentation"); - - time("Defragmenting databases", || { - time("Opened all databases", || self.open_all()); - - log("Defragmenting..."); - - self.drain_to_vec() - .into_par_iter() - .for_each(AnyDatabase::boxed_defragment); - }); - - exit.unblock(); - } - pub fn reset(&mut self, include_addresses: bool) { if include_addresses { let _ = self.address_index_to_address_data.reset(); diff --git a/parser/src/databases/txid_to_tx_data.rs b/src/parser/databases/txid_to_tx_data.rs similarity index 87% rename from parser/src/databases/txid_to_tx_data.rs rename to src/parser/databases/txid_to_tx_data.rs index 59a706249..14ee58555 100644 --- a/parser/src/databases/txid_to_tx_data.rs +++ b/src/parser/databases/txid_to_tx_data.rs @@ -2,13 +2,14 @@ use std::{ collections::BTreeMap, fs, mem, ops::{Deref, DerefMut}, + path::{Path, PathBuf}, }; use allocative::Allocative; use biter::bitcoin::Txid; use itertools::Itertools; -use crate::structs::{Date, Height, TxData, U8x31}; +use crate::structs::{Config, TxData, U8x31}; use super::{AnyDatabase, AnyDatabaseGroup, Database as _Database, Metadata}; @@ -18,8 +19,8 @@ type Database = _Database; #[derive(Allocative)] pub struct TxidToTxData { + path: PathBuf, pub metadata: Metadata, - map: BTreeMap, } @@ -109,8 +110,9 @@ impl TxidToTxData { #[inline(always)] pub fn _open_db(&mut self, db_index: u16) -> &mut Database { + let path = self.path.to_owned(); self.entry(db_index).or_insert_with(|| { - let path = Self::root().join(db_index.to_string()); + let path = path.join(db_index.to_string()); Database::open(path).unwrap() }) } @@ -125,31 +127,23 @@ impl TxidToTxData { } impl AnyDatabaseGroup for TxidToTxData { - fn import() -> Self { - let metadata = Metadata::import(Self::root(), 2); + fn import(config: &Config) -> Self { + let path = config.path_databases().join("txid_to_tx_data"); + let metadata = Metadata::import(&path, 2); Self { + path, metadata, map: BTreeMap::default(), } } - fn create_dir_all(&self) -> color_eyre::Result<(), std::io::Error> { - fs::create_dir_all(Self::root()) - } - fn reset_metadata(&mut self) { self.metadata.reset(); } - fn folder<'a>() -> &'a str { - "txid_to_tx_data" - } - fn open_all(&mut self) { - let path = Self::root(); - - let folder = fs::read_dir(path); + let folder = fs::read_dir(&self.path); if folder.is_err() { return; @@ -180,7 +174,11 @@ impl AnyDatabaseGroup for TxidToTxData { .collect_vec() } - fn export_metadata(&mut self, height: Height, date: Date) -> color_eyre::Result<()> { - self.metadata.export(height, date) + fn metadata(&mut self) -> &mut Metadata { + &mut self.metadata + } + + fn path(&self) -> &Path { + &self.path } } diff --git a/parser/src/databases/txout_index_to_address_index.rs b/src/parser/databases/txout_index_to_address_index.rs similarity index 85% rename from parser/src/databases/txout_index_to_address_index.rs rename to src/parser/databases/txout_index_to_address_index.rs index 79baeac10..5db9f4484 100644 --- a/parser/src/databases/txout_index_to_address_index.rs +++ b/src/parser/databases/txout_index_to_address_index.rs @@ -2,12 +2,13 @@ use std::{ collections::BTreeMap, fs, mem, ops::{Deref, DerefMut}, + path::{Path, PathBuf}, }; use allocative::Allocative; use itertools::Itertools; -use crate::structs::{Date, Height, TxoutIndex}; +use crate::structs::{Config, TxoutIndex}; use super::{AnyDatabase, AnyDatabaseGroup, Database as _Database, Metadata}; @@ -17,8 +18,8 @@ type Database = _Database; #[derive(Allocative)] pub struct TxoutIndexToAddressIndex { + path: PathBuf, pub metadata: Metadata, - map: BTreeMap, } @@ -69,6 +70,7 @@ impl TxoutIndexToAddressIndex { pub fn open_db(&mut self, key: &Key) -> &mut Database { let db_index = Self::db_index(key); + let path = self.path.to_owned(); self.entry(db_index).or_insert_with(|| { let db_name = format!( @@ -77,7 +79,7 @@ impl TxoutIndexToAddressIndex { (db_index + 1) * DB_MAX_SIZE ); - let path = Self::root().join(db_name); + let path = path.join(db_name); Database::open(path).unwrap() }) @@ -89,30 +91,21 @@ impl TxoutIndexToAddressIndex { } impl AnyDatabaseGroup for TxoutIndexToAddressIndex { - fn import() -> Self { + fn import(config: &Config) -> Self { + let path = config.path_databases().join("txout_index_to_address_index"); Self { - metadata: Metadata::import(Self::root(), 1), - + metadata: Metadata::import(&path, 1), + path, map: BTreeMap::default(), } } - fn create_dir_all(&self) -> color_eyre::Result<(), std::io::Error> { - fs::create_dir_all(Self::root()) - } - fn reset_metadata(&mut self) { self.metadata.reset(); } - fn folder<'a>() -> &'a str { - "txout_index_to_address_index" - } - fn open_all(&mut self) { - let path = Self::root(); - - let folder = fs::read_dir(path); + let folder = fs::read_dir(&self.path); if folder.is_err() { return; @@ -151,7 +144,11 @@ impl AnyDatabaseGroup for TxoutIndexToAddressIndex { .collect_vec() } - fn export_metadata(&mut self, height: Height, date: Date) -> color_eyre::Result<()> { - self.metadata.export(height, date) + fn metadata(&mut self) -> &mut Metadata { + &mut self.metadata + } + + fn path(&self) -> &Path { + &self.path } } diff --git a/parser/src/databases/txout_index_to_amount.rs b/src/parser/databases/txout_index_to_amount.rs similarity index 84% rename from parser/src/databases/txout_index_to_amount.rs rename to src/parser/databases/txout_index_to_amount.rs index 6774dfdb4..0464b04c7 100644 --- a/parser/src/databases/txout_index_to_amount.rs +++ b/src/parser/databases/txout_index_to_amount.rs @@ -2,12 +2,13 @@ use std::{ collections::BTreeMap, fs, mem, ops::{Deref, DerefMut}, + path::{Path, PathBuf}, }; use allocative::Allocative; use itertools::Itertools; -use crate::structs::{Amount, Date, Height, TxoutIndex}; +use crate::structs::{Amount, Config, TxoutIndex}; use super::{AnyDatabase, AnyDatabaseGroup, Database as _Database, Metadata}; @@ -17,8 +18,8 @@ type Database = _Database; #[derive(Allocative)] pub struct TxoutIndexToAmount { + path: PathBuf, pub metadata: Metadata, - map: BTreeMap, } @@ -69,6 +70,7 @@ impl TxoutIndexToAmount { pub fn open_db(&mut self, key: &Key) -> &mut Database { let db_index = Self::db_index(key); + let path = self.path.to_owned(); self.entry(db_index).or_insert_with(|| { let db_name = format!( @@ -77,7 +79,7 @@ impl TxoutIndexToAmount { (db_index + 1) * DB_MAX_SIZE ); - let path = Self::root().join(db_name); + let path = path.join(db_name); Database::open(path).unwrap() }) @@ -89,30 +91,21 @@ impl TxoutIndexToAmount { } impl AnyDatabaseGroup for TxoutIndexToAmount { - fn import() -> Self { + fn import(config: &Config) -> Self { + let path = config.path_databases().join("txout_index_to_amount"); Self { - metadata: Metadata::import(Self::root(), 1), - + metadata: Metadata::import(&path, 1), + path, map: BTreeMap::default(), } } - fn create_dir_all(&self) -> color_eyre::Result<(), std::io::Error> { - fs::create_dir_all(Self::root()) - } - fn reset_metadata(&mut self) { self.metadata.reset(); } - fn folder<'a>() -> &'a str { - "txout_index_to_amount" - } - fn open_all(&mut self) { - let path = Self::root(); - - let folder = fs::read_dir(path); + let folder = fs::read_dir(&self.path); if folder.is_err() { return; @@ -151,7 +144,11 @@ impl AnyDatabaseGroup for TxoutIndexToAmount { .collect_vec() } - fn export_metadata(&mut self, height: Height, date: Date) -> color_eyre::Result<()> { - self.metadata.export(height, date) + fn metadata(&mut self) -> &mut Metadata { + &mut self.metadata + } + + fn path(&self) -> &Path { + &self.path } } diff --git a/parser/src/datasets/_traits/any_dataset.rs b/src/parser/datasets/_traits/any_dataset.rs similarity index 99% rename from parser/src/datasets/_traits/any_dataset.rs rename to src/parser/datasets/_traits/any_dataset.rs index 6a5091efc..4f5eba00b 100644 --- a/parser/src/datasets/_traits/any_dataset.rs +++ b/src/parser/datasets/_traits/any_dataset.rs @@ -3,14 +3,14 @@ use rayon::prelude::*; use struct_iterable::Iterable; use crate::{ - datasets::{ + parser::datasets::{ cohort_metadata::AddressCohortMetadataDataset, ComputeData, DateRecapDataset, RatioDataset, SubDataset, }, structs::{ - AnyBiMap, AnyDateMap, AnyHeightMap, AnyMap, BiMap, Date, Height, MapKind, Timestamp, OHLC, + AnyBiMap, AnyDateMap, AnyHeightMap, AnyMap, BiMap, Date, DateMap, Height, HeightMap, + MapKind, Timestamp, OHLC, }, - DateMap, HeightMap, }; use super::{AnyDatasetGroup, MinInitialStates}; diff --git a/parser/src/datasets/_traits/any_dataset_group.rs b/src/parser/datasets/_traits/any_dataset_group.rs similarity index 100% rename from parser/src/datasets/_traits/any_dataset_group.rs rename to src/parser/datasets/_traits/any_dataset_group.rs diff --git a/parser/src/datasets/_traits/any_datasets.rs b/src/parser/datasets/_traits/any_datasets.rs similarity index 100% rename from parser/src/datasets/_traits/any_datasets.rs rename to src/parser/datasets/_traits/any_datasets.rs diff --git a/parser/src/datasets/_traits/min_initial_state.rs b/src/parser/datasets/_traits/min_initial_state.rs similarity index 100% rename from parser/src/datasets/_traits/min_initial_state.rs rename to src/parser/datasets/_traits/min_initial_state.rs diff --git a/parser/src/datasets/_traits/mod.rs b/src/parser/datasets/_traits/mod.rs similarity index 100% rename from parser/src/datasets/_traits/mod.rs rename to src/parser/datasets/_traits/mod.rs diff --git a/parser/src/datasets/address/all_metadata.rs b/src/parser/datasets/address/all_metadata.rs similarity index 88% rename from parser/src/datasets/address/all_metadata.rs rename to src/parser/datasets/address/all_metadata.rs index 124c27e6d..1087d62c6 100644 --- a/parser/src/datasets/address/all_metadata.rs +++ b/src/parser/datasets/address/all_metadata.rs @@ -2,8 +2,8 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates}, - structs::{BiMap, Config, MapKind}, + parser::datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates}, + structs::{BiMap, Config, MapKind, MapPath}, }; #[derive(Allocative, Iterable)] @@ -15,8 +15,8 @@ pub struct AllAddressesMetadataDataset { } impl AllAddressesMetadataDataset { - pub fn import(parent_path: &str, config: &Config) -> color_eyre::Result { - let f = |s: &str| format!("{parent_path}/{s}"); + pub fn import(path: &MapPath, config: &Config) -> color_eyre::Result { + let f = |s: &str| path.join(s); let mut s = Self { min_initial_states: MinInitialStates::default(), diff --git a/parser/src/datasets/address/cohort.rs b/src/parser/datasets/address/cohort.rs similarity index 96% rename from parser/src/datasets/address/cohort.rs rename to src/parser/datasets/address/cohort.rs index c7d24ac82..f7676ad83 100644 --- a/parser/src/datasets/address/cohort.rs +++ b/src/parser/datasets/address/cohort.rs @@ -3,9 +3,11 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates, SubDataset}, - states::{AddressCohortId, DurableStates}, - structs::{AddressSplit, BiMap, Config, Date, Height}, + parser::{ + datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates, SubDataset}, + states::{AddressCohortId, DurableStates}, + }, + structs::{AddressSplit, BiMap, Config, Date, Height, MapPath}, }; use super::cohort_metadata::AddressCohortMetadataDataset; @@ -23,7 +25,7 @@ pub struct CohortDataset { impl CohortDataset { pub fn import( - parent_path: &str, + path: &MapPath, id: AddressCohortId, config: &Config, ) -> color_eyre::Result { @@ -33,8 +35,8 @@ impl CohortDataset { let mut s = Self { min_initial_states: MinInitialStates::default(), split, - metadata: AddressCohortMetadataDataset::import(parent_path, &name, config)?, - subs: SubDataset::import(parent_path, &name, config)?, + metadata: AddressCohortMetadataDataset::import(path, &name, config)?, + subs: SubDataset::import(path, &name, config)?, }; s.min_initial_states diff --git a/parser/src/datasets/address/cohort_metadata.rs b/src/parser/datasets/address/cohort_metadata.rs similarity index 87% rename from parser/src/datasets/address/cohort_metadata.rs rename to src/parser/datasets/address/cohort_metadata.rs index 185cda2de..860596d0c 100644 --- a/parser/src/datasets/address/cohort_metadata.rs +++ b/src/parser/datasets/address/cohort_metadata.rs @@ -2,8 +2,8 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, InsertData, MinInitialStates}, - structs::{BiMap, Config, MapKind}, + parser::datasets::{AnyDataset, InsertData, MinInitialStates}, + structs::{BiMap, Config, MapKind, MapPath}, }; #[derive(Allocative, Iterable)] @@ -19,15 +19,15 @@ pub struct AddressCohortMetadataDataset { impl AddressCohortMetadataDataset { pub fn import( - parent_path: &str, + path: &MapPath, name: &Option, config: &Config, ) -> color_eyre::Result { let f = |s: &str| { if let Some(name) = name { - format!("{parent_path}/{name}/{s}") + path.join(&format!("{name}/{s}")) } else { - format!("{parent_path}/{s}") + path.join(s) } }; diff --git a/parser/src/datasets/address/mod.rs b/src/parser/datasets/address/mod.rs similarity index 93% rename from parser/src/datasets/address/mod.rs rename to src/parser/datasets/address/mod.rs index 392bf8bd6..a8760c860 100644 --- a/parser/src/datasets/address/mod.rs +++ b/src/parser/datasets/address/mod.rs @@ -7,9 +7,8 @@ use itertools::Itertools; use rayon::prelude::*; use crate::{ - states::SplitByAddressCohort, - structs::{BiMap, Config, Height}, - Date, + parser::states::SplitByAddressCohort, + structs::{BiMap, Config, Date, Height}, }; use self::{all_metadata::AllAddressesMetadataDataset, cohort::CohortDataset}; @@ -26,13 +25,15 @@ pub struct AddressDatasets { } impl AddressDatasets { - pub fn import(parent_path: &str, config: &Config) -> color_eyre::Result { + pub fn import(config: &Config) -> color_eyre::Result { let mut cohorts = SplitByAddressCohort::>::default(); + let path_dataset = config.path_datasets(); + cohorts .as_vec() .into_par_iter() - .map(|(_, id)| (id, CohortDataset::import(parent_path, id, config))) + .map(|(_, id)| (id, CohortDataset::import(&path_dataset, id, config))) .collect::>() .into_iter() .try_for_each(|(id, dataset)| -> color_eyre::Result<()> { @@ -43,7 +44,7 @@ impl AddressDatasets { let mut s = Self { min_initial_states: MinInitialStates::default(), - metadata: AllAddressesMetadataDataset::import(parent_path, config)?, + metadata: AllAddressesMetadataDataset::import(&path_dataset, config)?, cohorts: cohorts.unwrap(), }; diff --git a/parser/src/datasets/block_metadata.rs b/src/parser/datasets/block_metadata.rs similarity index 87% rename from parser/src/datasets/block_metadata.rs rename to src/parser/datasets/block_metadata.rs index aaa6ddfe9..1bfd17b0a 100644 --- a/parser/src/datasets/block_metadata.rs +++ b/src/parser/datasets/block_metadata.rs @@ -2,7 +2,7 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::AnyDataset, + parser::datasets::AnyDataset, structs::{Config, Date, HeightMap, MapKind, Timestamp}, }; @@ -16,8 +16,8 @@ pub struct BlockMetadataDataset { } impl BlockMetadataDataset { - pub fn import(parent_path: &str, config: &Config) -> color_eyre::Result { - let f = |s: &str| format!("{parent_path}/{s}"); + pub fn import(config: &Config) -> color_eyre::Result { + let f = |s: &str| config.path_datasets().join(s); let mut s = Self { min_initial_states: MinInitialStates::default(), diff --git a/parser/src/datasets/coindays.rs b/src/parser/datasets/coindays.rs similarity index 87% rename from parser/src/datasets/coindays.rs rename to src/parser/datasets/coindays.rs index aae63dfc1..c6eca0151 100644 --- a/parser/src/datasets/coindays.rs +++ b/src/parser/datasets/coindays.rs @@ -2,9 +2,8 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::AnyDataset, - structs::{Config, MapKind}, - DateMap, HeightMap, + parser::datasets::AnyDataset, + structs::{Config, DateMap, HeightMap, MapKind}, }; use super::{InsertData, MinInitialStates}; @@ -17,8 +16,8 @@ pub struct CoindaysDataset { } impl CoindaysDataset { - pub fn import(parent_path: &str, config: &Config) -> color_eyre::Result { - let f = |s: &str| format!("{parent_path}/{s}"); + pub fn import(config: &Config) -> color_eyre::Result { + let f = |s: &str| config.path_datasets().join(s); let mut s = Self { min_initial_states: MinInitialStates::default(), diff --git a/parser/src/datasets/cointime.rs b/src/parser/datasets/cointime.rs similarity index 97% rename from parser/src/datasets/cointime.rs rename to src/parser/datasets/cointime.rs index 0cd6d6773..2c4b79271 100644 --- a/parser/src/datasets/cointime.rs +++ b/src/parser/datasets/cointime.rs @@ -2,9 +2,8 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - structs::{BiMap, Config, DateMap, Height, MapKind}, + structs::{BiMap, Config, DateMap, Height, HeightMap, MapKind}, utils::{ONE_DAY_IN_DAYS, ONE_YEAR_IN_DAYS, THREE_MONTHS_IN_DAYS, TWO_WEEK_IN_DAYS}, - HeightMap, }; use super::{AnyDataset, ComputeData, InsertData, MinInitialStates, RatioDataset}; @@ -72,8 +71,9 @@ pub struct CointimeDataset { } impl CointimeDataset { - pub fn import(parent_path: &str, config: &Config) -> color_eyre::Result { - let f = |s: &str| format!("{parent_path}/{s}"); + pub fn import(config: &Config) -> color_eyre::Result { + let path_dataset = config.path_datasets(); + let f = |s: &str| path_dataset.join(s); let mut s = Self { min_initial_states: MinInitialStates::default(), @@ -93,7 +93,7 @@ impl CointimeDataset { // Computed active_cap: BiMap::new_bin(1, MapKind::Computed, &f("active_cap")), active_price: BiMap::new_bin(1, MapKind::Computed, &f("active_price")), - active_price_ratio: RatioDataset::import(parent_path, "active_price", config)?, + active_price_ratio: RatioDataset::import(&path_dataset, "active_price", config)?, active_supply: BiMap::new_bin(1, MapKind::Computed, &f("active_supply")), active_supply_3m_net_change: BiMap::new_bin( 1, @@ -140,7 +140,7 @@ impl CointimeDataset { ), cointime_cap: BiMap::new_bin(1, MapKind::Computed, &f("cointime_cap")), cointime_price: BiMap::new_bin(1, MapKind::Computed, &f("cointime_price")), - cointime_price_ratio: RatioDataset::import(parent_path, "cointime_price", config)?, + cointime_price_ratio: RatioDataset::import(&path_dataset, "cointime_price", config)?, cointime_value_created: HeightMap::new_bin( 1, MapKind::Computed, @@ -237,7 +237,11 @@ impl CointimeDataset { &f("true_market_deviation"), ), true_market_mean: BiMap::new_bin(1, MapKind::Computed, &f("true_market_mean")), - true_market_mean_ratio: RatioDataset::import(parent_path, "true_market_mean", config)?, + true_market_mean_ratio: RatioDataset::import( + &path_dataset, + "true_market_mean", + config, + )?, true_market_net_unrealized_profit_and_loss: BiMap::new_bin( 1, MapKind::Computed, @@ -245,7 +249,7 @@ impl CointimeDataset { ), vaulted_cap: BiMap::new_bin(1, MapKind::Computed, &f("vaulted_cap")), vaulted_price: BiMap::new_bin(1, MapKind::Computed, &f("vaulted_price")), - vaulted_price_ratio: RatioDataset::import(parent_path, "vaulted_price", config)?, + vaulted_price_ratio: RatioDataset::import(&path_dataset, "vaulted_price", config)?, vaulted_supply: BiMap::new_bin(1, MapKind::Computed, &f("vaulted_supply")), vaulted_supply_3m_net_change: BiMap::new_bin( 1, diff --git a/parser/src/datasets/constant.rs b/src/parser/datasets/constant.rs similarity index 90% rename from parser/src/datasets/constant.rs rename to src/parser/datasets/constant.rs index ae94965ae..4f1dce63f 100644 --- a/parser/src/datasets/constant.rs +++ b/src/parser/datasets/constant.rs @@ -16,8 +16,8 @@ pub struct ConstantDataset { } impl ConstantDataset { - pub fn import(parent_path: &str, config: &Config) -> color_eyre::Result { - let f = |s: &str| format!("{parent_path}/{s}"); + pub fn import(config: &Config) -> color_eyre::Result { + let f = |s: &str| config.path_datasets().join(s); let mut s = Self { min_initial_states: MinInitialStates::default(), diff --git a/parser/src/datasets/date_metadata.rs b/src/parser/datasets/date_metadata.rs similarity index 88% rename from parser/src/datasets/date_metadata.rs rename to src/parser/datasets/date_metadata.rs index 4f69ec24f..80f8fa9d6 100644 --- a/parser/src/datasets/date_metadata.rs +++ b/src/parser/datasets/date_metadata.rs @@ -2,7 +2,7 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::AnyDataset, + parser::datasets::AnyDataset, structs::{Config, DateMap, Height, MapKind}, }; @@ -17,8 +17,8 @@ pub struct DateMetadataDataset { } impl DateMetadataDataset { - pub fn import(parent_path: &str, config: &Config) -> color_eyre::Result { - let f = |s: &str| format!("{parent_path}/{s}"); + pub fn import(config: &Config) -> color_eyre::Result { + let f = |s: &str| config.path_datasets().join(s); let mut s = Self { min_initial_states: MinInitialStates::default(), diff --git a/parser/src/datasets/mining.rs b/src/parser/datasets/mining.rs similarity index 99% rename from parser/src/datasets/mining.rs rename to src/parser/datasets/mining.rs index 759fcf35b..667b0a8a7 100644 --- a/parser/src/datasets/mining.rs +++ b/src/parser/datasets/mining.rs @@ -4,7 +4,7 @@ use ordered_float::OrderedFloat; use struct_iterable::Iterable; use crate::{ - datasets::AnyDataset, + parser::datasets::AnyDataset, structs::{Amount, BiMap, Config, DateMap, Height, HeightMap, MapKey, MapKind}, utils::{ BYTES_IN_MB, ONE_DAY_IN_DAYS, ONE_MONTH_IN_DAYS, ONE_WEEK_IN_DAYS, ONE_YEAR_IN_DAYS, @@ -123,8 +123,8 @@ pub struct MiningDataset { } impl MiningDataset { - pub fn import(parent_path: &str, config: &Config) -> color_eyre::Result { - let f = |s: &str| format!("{parent_path}/{s}"); + pub fn import(config: &Config) -> color_eyre::Result { + let f = |s: &str| config.path_datasets().join(s); let mut s = Self { min_initial_states: MinInitialStates::default(), diff --git a/parser/src/datasets/mod.rs b/src/parser/datasets/mod.rs similarity index 75% rename from parser/src/datasets/mod.rs rename to src/parser/datasets/mod.rs index 43a8c94c1..bc4edc899 100644 --- a/parser/src/datasets/mod.rs +++ b/src/parser/datasets/mod.rs @@ -1,9 +1,10 @@ -use std::{collections::BTreeMap, fs, ops::RangeInclusive, path::Path}; +use std::{collections::BTreeMap, ops::RangeInclusive, path::PathBuf}; use allocative::Allocative; use itertools::Itertools; +use log::info; use rayon::prelude::*; mod _traits; @@ -34,16 +35,18 @@ pub use transaction::*; pub use utxo::*; use crate::{ - databases::Databases, - io::{Json, JSON_EXTENSION}, - states::{ - AddressCohortsInputStates, - AddressCohortsOneShotStates, - AddressCohortsRealizedStates, - States, - UTXOCohortsOneShotStates, - // UTXOCohortsReceivedStates, - UTXOCohortsSentStates, + io::Json, + parser::{ + databases::Databases, + states::{ + AddressCohortsInputStates, + AddressCohortsOneShotStates, + AddressCohortsRealizedStates, + States, + UTXOCohortsOneShotStates, + // UTXOCohortsReceivedStates, + UTXOCohortsSentStates, + }, }, structs::{Amount, Config, Date, Height, Price, Timestamp}, }; @@ -84,7 +87,7 @@ pub struct ComputeData<'a> { } #[derive(Allocative)] -pub struct AllDatasets { +pub struct Datasets { min_initial_states: MinInitialStates, pub constant: ConstantDataset, @@ -99,31 +102,27 @@ pub struct AllDatasets { pub utxo: UTXODatasets, } -const DATASETS_PATH: &str = "../datasets"; - -impl AllDatasets { +impl Datasets { pub fn import(config: &Config) -> color_eyre::Result { - let path = DATASETS_PATH; + let price = PriceDatasets::import(config)?; - let price = PriceDatasets::import(path, config)?; + let constant = ConstantDataset::import(config)?; - let constant = ConstantDataset::import(path, config)?; + let date_metadata = DateMetadataDataset::import(config)?; - let date_metadata = DateMetadataDataset::import(path, config)?; + let cointime = CointimeDataset::import(config)?; - let cointime = CointimeDataset::import(path, config)?; + let coindays = CoindaysDataset::import(config)?; - let coindays = CoindaysDataset::import(path, config)?; + let mining = MiningDataset::import(config)?; - let mining = MiningDataset::import(path, config)?; + let block_metadata = BlockMetadataDataset::import(config)?; - let block_metadata = BlockMetadataDataset::import(path, config)?; + let transaction = TransactionDataset::import(config)?; - let transaction = TransactionDataset::import(path, config)?; + let address = AddressDatasets::import(config)?; - let address = AddressDatasets::import(path, config)?; - - let utxo = UTXODatasets::import(path, config)?; + let utxo = UTXODatasets::import(config)?; let mut s = Self { min_initial_states: MinInitialStates::default(), @@ -143,7 +142,7 @@ impl AllDatasets { s.min_initial_states .consume(MinInitialStates::compute_from_datasets(&s, config)); - s.export_meta_files()?; + info!("Imported datasets"); Ok(s) } @@ -269,43 +268,23 @@ impl AllDatasets { } } - pub fn export_meta_files(&self) -> color_eyre::Result<()> { - let mut path_to_type: BTreeMap<&Path, &str> = self + pub fn get_paths_to_type(&self, config: &Config) -> BTreeMap { + let mut path_to_type: BTreeMap = self .to_any_dataset_vec() .into_iter() .flat_map(|dataset| dataset.to_all_map_vec()) - .flat_map(|map| map.exported_path_with_t_name()) + .flat_map(|map| map.get_paths_to_type()) .collect(); - path_to_type.insert(Path::new("../datasets/last"), "Value"); + path_to_type.insert( + config.path_datasets_last_values().unwrap().to_owned(), + "Value".to_string(), + ); - let datasets_len = path_to_type.len(); - - let server_inputs_path = "../server/in"; - - fs::create_dir_all(server_inputs_path)?; - - Json::export( - Path::new(&format!("{server_inputs_path}/disk_path_to_type.json")), - &path_to_type, - )?; - - let datasets_len_path = format!("{server_inputs_path}/datasets_len.txt"); - - if let Ok(len) = fs::read_to_string(&datasets_len_path) { - if let Ok(len) = len.parse::() { - if datasets_len == len { - return Ok(()); - } - } - } - - fs::write(datasets_len_path, datasets_len.to_string())?; - - Ok(()) + path_to_type } - pub fn export(&mut self) -> color_eyre::Result<()> { + pub fn export(&mut self, config: &Config) -> color_eyre::Result<()> { self.to_mut_any_dataset_vec() .into_iter() .for_each(|dataset| dataset.pre_export()); @@ -316,6 +295,11 @@ impl AllDatasets { let mut path_to_last: BTreeMap = BTreeMap::default(); + let path_dataset = config.path_datasets(); + let path_dataset_ser = path_dataset.to_str().unwrap(); + let path_price = config.path_price(); + let path_price_ser = path_price.to_str().unwrap(); + self.to_mut_any_dataset_vec() .into_iter() .for_each(|dataset| { @@ -326,34 +310,28 @@ impl AllDatasets { if let Some(last_value) = map.last_value() { let mut last_path = last_path.clone(); last_path.pop(); + let key = last_path + .to_str() + .unwrap() + .replace(path_dataset_ser, "") + .replace(path_price_ser, "") + .split("/") + .join("-") + .to_string(); - let last_path = last_path.to_str().unwrap(); - - let skip = if last_path.starts_with(DATASETS_PATH) { - 2 - } else { - 1 - }; - - path_to_last.insert( - last_path.split('/').skip(skip).join("-").to_string(), - last_value, - ); + path_to_last.insert(key, last_value); } } }); }); - Json::export( - Path::new(&format!("{DATASETS_PATH}/last.{JSON_EXTENSION}")), - &path_to_last, - )?; + Json::export(&config.path_datasets_last_values(), &path_to_last)?; Ok(()) } } -impl AnyDatasets for AllDatasets { +impl AnyDatasets for Datasets { fn get_min_initial_states(&self) -> &MinInitialStates { &self.min_initial_states } diff --git a/parser/src/datasets/price.rs b/src/parser/datasets/price.rs similarity index 93% rename from parser/src/datasets/price.rs rename to src/parser/datasets/price.rs index d2735c4ee..ca79bd945 100644 --- a/parser/src/datasets/price.rs +++ b/src/parser/datasets/price.rs @@ -7,7 +7,7 @@ use color_eyre::eyre::Error; use struct_iterable::Iterable; use crate::{ - price::{Binance, Kibo, Kraken}, + parser::price::{Binance, Kibo, Kraken}, structs::{ Amount, BiMap, Config, Date, DateMap, DateMapChunkId, Height, HeightMapChunkId, MapKey, MapKind, Timestamp, OHLC, @@ -85,10 +85,9 @@ pub struct PriceDatasets { } impl PriceDatasets { - pub fn import(datasets_path: &str, config: &Config) -> color_eyre::Result { - let price_path = "../price"; - - let f = |s: &str| format!("{datasets_path}/{s}"); + pub fn import(config: &Config) -> color_eyre::Result { + let path_dataset = config.path_datasets(); + let f = |s: &str| path_dataset.join(s); let mut s = Self { min_initial_states: MinInitialStates::default(), @@ -104,7 +103,7 @@ impl PriceDatasets { // --- // Inserted // --- - ohlc: BiMap::new_json(1, MapKind::Inserted, price_path), + ohlc: BiMap::new_json(1, MapKind::Inserted, &config.path_price()), // --- // Computed @@ -115,31 +114,31 @@ impl PriceDatasets { close: BiMap::new_bin(1, MapKind::Computed, &f("close")), market_cap: BiMap::new_bin(1, MapKind::Computed, &f("market_cap")), price_1w_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_1w_sma")), - price_1w_sma_ratio: RatioDataset::import(datasets_path, "price_1w_sma", config)?, + price_1w_sma_ratio: RatioDataset::import(&path_dataset, "price_1w_sma", config)?, price_1m_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_1m_sma")), - price_1m_sma_ratio: RatioDataset::import(datasets_path, "price_1m_sma", config)?, + price_1m_sma_ratio: RatioDataset::import(&path_dataset, "price_1m_sma", config)?, price_1y_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_1y_sma")), - price_1y_sma_ratio: RatioDataset::import(datasets_path, "price_1y_sma", config)?, + price_1y_sma_ratio: RatioDataset::import(&path_dataset, "price_1y_sma", config)?, price_2y_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_2y_sma")), - price_2y_sma_ratio: RatioDataset::import(datasets_path, "price_2y_sma", config)?, + price_2y_sma_ratio: RatioDataset::import(&path_dataset, "price_2y_sma", config)?, price_4y_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_4y_sma")), - price_4y_sma_ratio: RatioDataset::import(datasets_path, "price_4y_sma", config)?, + price_4y_sma_ratio: RatioDataset::import(&path_dataset, "price_4y_sma", config)?, price_8d_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_8d_sma")), - price_8d_sma_ratio: RatioDataset::import(datasets_path, "price_8d_sma", config)?, + price_8d_sma_ratio: RatioDataset::import(&path_dataset, "price_8d_sma", config)?, price_13d_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_13d_sma")), - price_13d_sma_ratio: RatioDataset::import(datasets_path, "price_13d_sma", config)?, + price_13d_sma_ratio: RatioDataset::import(&path_dataset, "price_13d_sma", config)?, price_21d_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_21d_sma")), - price_21d_sma_ratio: RatioDataset::import(datasets_path, "price_21d_sma", config)?, + price_21d_sma_ratio: RatioDataset::import(&path_dataset, "price_21d_sma", config)?, price_34d_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_34d_sma")), - price_34d_sma_ratio: RatioDataset::import(datasets_path, "price_34d_sma", config)?, + price_34d_sma_ratio: RatioDataset::import(&path_dataset, "price_34d_sma", config)?, price_55d_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_55d_sma")), - price_55d_sma_ratio: RatioDataset::import(datasets_path, "price_55d_sma", config)?, + price_55d_sma_ratio: RatioDataset::import(&path_dataset, "price_55d_sma", config)?, price_89d_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_89d_sma")), - price_89d_sma_ratio: RatioDataset::import(datasets_path, "price_89d_sma", config)?, + price_89d_sma_ratio: RatioDataset::import(&path_dataset, "price_89d_sma", config)?, price_144d_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_144d_sma")), - price_144d_sma_ratio: RatioDataset::import(datasets_path, "price_144d_sma", config)?, + price_144d_sma_ratio: RatioDataset::import(&path_dataset, "price_144d_sma", config)?, price_200w_sma: BiMap::new_bin(1, MapKind::Computed, &f("price_200w_sma")), - price_200w_sma_ratio: RatioDataset::import(datasets_path, "price_200w_sma", config)?, + price_200w_sma_ratio: RatioDataset::import(&path_dataset, "price_200w_sma", config)?, price_1d_total_return: DateMap::new_bin( 1, MapKind::Computed, @@ -540,6 +539,7 @@ impl PriceDatasets { height: Height, timestamp: Timestamp, previous_timestamp: Option, + config: &Config, ) -> color_eyre::Result { if let Some(ohlc) = self.ohlc.height.get_or_import(&height) { return Ok(ohlc); @@ -558,7 +558,7 @@ impl PriceDatasets { .unwrap_or_else(|_| { self.get_from_1mn_binance(timestamp, previous_timestamp) .unwrap_or_else(|_| { - self.get_from_har_binance(timestamp, previous_timestamp) + self.get_from_har_binance(timestamp, previous_timestamp, config) .unwrap_or_else(|_| { self.get_from_height_kibo(&height).unwrap_or_else(|_| { let date = timestamp.to_date(); @@ -659,10 +659,11 @@ How to fix this: &mut self, timestamp: Timestamp, previous_timestamp: Option, + config: &Config, ) -> color_eyre::Result { if self.binance_har.is_none() { self.binance_har - .replace(Binance::read_har_file().unwrap_or_default()); + .replace(Binance::read_har_file(config).unwrap_or_default()); } Self::find_height_ohlc( diff --git a/parser/src/datasets/subs/capitalization.rs b/src/parser/datasets/subs/capitalization.rs similarity index 90% rename from parser/src/datasets/subs/capitalization.rs rename to src/parser/datasets/subs/capitalization.rs index 20a792662..834f19a3e 100644 --- a/parser/src/datasets/subs/capitalization.rs +++ b/src/parser/datasets/subs/capitalization.rs @@ -2,9 +2,11 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates}, - states::CapitalizationState, - structs::{BiMap, Config, MapKind}, + parser::{ + datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates}, + states::CapitalizationState, + }, + structs::{BiMap, Config, MapKind, MapPath}, utils::ONE_MONTH_IN_DAYS, }; @@ -22,15 +24,15 @@ pub struct CapitalizationDataset { impl CapitalizationDataset { pub fn import( - parent_path: &str, + path: &MapPath, name: &Option, config: &Config, ) -> color_eyre::Result { let f = |s: &str| { if let Some(name) = name { - format!("{parent_path}/{name}/{s}") + path.join(&format!("{name}/{s}")) } else { - format!("{parent_path}/{s}") + path.join(s) } }; @@ -52,7 +54,7 @@ impl CapitalizationDataset { ), realized_price: BiMap::new_bin(1, MapKind::Computed, &f("realized_price")), realized_price_ratio: RatioDataset::import( - parent_path, + path, &format!( "{}realized_price", name.as_ref().map_or("".to_owned(), |n| format!("{n}-")) diff --git a/parser/src/datasets/subs/input.rs b/src/parser/datasets/subs/input.rs similarity index 87% rename from parser/src/datasets/subs/input.rs rename to src/parser/datasets/subs/input.rs index b9091205e..f7522cc03 100644 --- a/parser/src/datasets/subs/input.rs +++ b/src/parser/datasets/subs/input.rs @@ -2,10 +2,11 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, InsertData, MinInitialStates}, - states::InputState, - structs::{BiMap, Config, MapKind}, - DateMap, HeightMap, + parser::{ + datasets::{AnyDataset, InsertData, MinInitialStates}, + states::InputState, + }, + structs::{BiMap, Config, DateMap, HeightMap, MapKind, MapPath}, }; #[derive(Allocative, Iterable)] @@ -22,15 +23,15 @@ pub struct InputSubDataset { impl InputSubDataset { pub fn import( - parent_path: &str, + path: &MapPath, name: &Option, config: &Config, ) -> color_eyre::Result { let f = |s: &str| { if let Some(name) = name { - format!("{parent_path}/{name}/{s}") + path.join(&format!("{name}/{s}")) } else { - format!("{parent_path}/{s}") + path.join(s) } }; diff --git a/parser/src/datasets/subs/mod.rs b/src/parser/datasets/subs/mod.rs similarity index 95% rename from parser/src/datasets/subs/mod.rs rename to src/parser/datasets/subs/mod.rs index c34be68cd..566cd8388 100644 --- a/parser/src/datasets/subs/mod.rs +++ b/src/parser/datasets/subs/mod.rs @@ -21,7 +21,10 @@ pub use supply::*; pub use unrealized::*; pub use utxo::*; -use crate::{datasets::AnyDataset, structs::Config}; +use crate::{ + parser::datasets::AnyDataset, + structs::{Config, MapPath}, +}; use super::AnyDatasetGroup; @@ -39,7 +42,7 @@ pub struct SubDataset { impl SubDataset { pub fn import( - parent_path: &str, + parent_path: &MapPath, name: &Option, config: &Config, ) -> color_eyre::Result { diff --git a/parser/src/datasets/subs/output.rs b/src/parser/datasets/subs/output.rs similarity index 100% rename from parser/src/datasets/subs/output.rs rename to src/parser/datasets/subs/output.rs diff --git a/parser/src/datasets/subs/price_paid.rs b/src/parser/datasets/subs/price_paid.rs similarity index 96% rename from parser/src/datasets/subs/price_paid.rs rename to src/parser/datasets/subs/price_paid.rs index ca06ef2b5..7f1bba43d 100644 --- a/parser/src/datasets/subs/price_paid.rs +++ b/src/parser/datasets/subs/price_paid.rs @@ -2,9 +2,11 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, InsertData, MinInitialStates}, - states::PricePaidState, - structs::{BiMap, Config, Date, Height, MapKind}, + parser::{ + datasets::{AnyDataset, InsertData, MinInitialStates}, + states::PricePaidState, + }, + structs::{BiMap, Config, Date, Height, MapKind, MapPath}, }; #[derive(Allocative, Iterable)] @@ -34,15 +36,15 @@ pub struct PricePaidSubDataset { impl PricePaidSubDataset { pub fn import( - parent_path: &str, + path: &MapPath, name: &Option, config: &Config, ) -> color_eyre::Result { let f = |s: &str| { if let Some(name) = name { - format!("{parent_path}/{name}/{s}") + path.join(&format!("{name}/{s}")) } else { - format!("{parent_path}/{s}") + path.join(s) } }; diff --git a/parser/src/datasets/subs/ratio.rs b/src/parser/datasets/subs/ratio.rs similarity index 93% rename from parser/src/datasets/subs/ratio.rs rename to src/parser/datasets/subs/ratio.rs index d7ca03c3e..270f9dde0 100644 --- a/parser/src/datasets/subs/ratio.rs +++ b/src/parser/datasets/subs/ratio.rs @@ -2,8 +2,8 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, ComputeData, MinInitialStates}, - structs::{BiMap, Config, MapKind}, + parser::datasets::{AnyDataset, ComputeData, MinInitialStates}, + structs::{BiMap, Config, MapKind, MapPath}, utils::{ONE_MONTH_IN_DAYS, ONE_WEEK_IN_DAYS, ONE_YEAR_IN_DAYS}, }; @@ -31,9 +31,9 @@ pub struct RatioDataset { } impl RatioDataset { - pub fn import(parent_path: &str, name: &str, config: &Config) -> color_eyre::Result { - let f_ratio = |s: &str| format!("{parent_path}/market_price_to_{name}_{s}"); - let f_price = |s: &str| format!("{parent_path}/{name}_{s}"); + pub fn import(path: &MapPath, name: &str, config: &Config) -> color_eyre::Result { + let f_ratio = |s: &str| path.join(&format!("market_price_to_{name}_{s}")); + let f_price = |s: &str| path.join(&format!("{name}_{s}")); let mut s = Self { min_initial_states: MinInitialStates::default(), diff --git a/parser/src/datasets/subs/realized.rs b/src/parser/datasets/subs/realized.rs similarity index 97% rename from parser/src/datasets/subs/realized.rs rename to src/parser/datasets/subs/realized.rs index 005d4b389..c5c4ae98b 100644 --- a/parser/src/datasets/subs/realized.rs +++ b/src/parser/datasets/subs/realized.rs @@ -2,11 +2,12 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates}, - states::RealizedState, - structs::{BiMap, Config, MapKind, Price}, + parser::{ + datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates}, + states::RealizedState, + }, + structs::{BiMap, Config, DateMap, HeightMap, MapKind, MapPath, Price}, utils::ONE_MONTH_IN_DAYS, - DateMap, HeightMap, }; #[derive(Allocative, Iterable)] @@ -45,18 +46,17 @@ pub struct RealizedSubDataset { impl RealizedSubDataset { pub fn import( - parent_path: &str, + path: &MapPath, name: &Option, config: &Config, ) -> color_eyre::Result { let f = |s: &str| { if let Some(name) = name { - format!("{parent_path}/{name}/{s}") + path.join(&format!("{name}/{s}")) } else { - format!("{parent_path}/{s}") + path.join(s) } }; - let mut s = Self { min_initial_states: MinInitialStates::default(), diff --git a/parser/src/datasets/subs/recap.rs b/src/parser/datasets/subs/recap.rs similarity index 96% rename from parser/src/datasets/subs/recap.rs rename to src/parser/datasets/subs/recap.rs index e5661fa53..e02277c1e 100644 --- a/parser/src/datasets/subs/recap.rs +++ b/src/parser/datasets/subs/recap.rs @@ -3,9 +3,11 @@ use std::{iter::Sum, ops::Add}; use allocative::Allocative; use crate::{ - structs::{DateMapChunkId, GenericMap, MapKey, MapKind, MapSerialized, MapValue}, + structs::{ + Date, DateMapChunkId, GenericMap, MapChunkId, MapKey, MapKind, MapPath, MapSerialized, + MapValue, SerializedBTreeMap, + }, utils::{get_percentile, LossyFrom}, - Date, MapChunkId, SerializedBTreeMap, }; pub type DateRecapDataset = RecapDataset>; @@ -91,8 +93,8 @@ where Key: MapKey, Serialized: MapSerialized, { - pub fn import(parent_path: &str, options: RecapOptions) -> color_eyre::Result { - let f = |s: &str| format!("{parent_path}/{s}"); + pub fn import(path: &MapPath, options: RecapOptions) -> color_eyre::Result { + let f = |s: &str| path.join(s); let s = Self { // --- diff --git a/parser/src/datasets/subs/supply.rs b/src/parser/datasets/subs/supply.rs similarity index 90% rename from parser/src/datasets/subs/supply.rs rename to src/parser/datasets/subs/supply.rs index 316ae0904..7fe41efec 100644 --- a/parser/src/datasets/subs/supply.rs +++ b/src/parser/datasets/subs/supply.rs @@ -2,9 +2,11 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates}, - states::SupplyState, - structs::{BiMap, Config, MapKind}, + parser::{ + datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates}, + states::SupplyState, + }, + structs::{BiMap, Config, MapKind, MapPath}, }; #[derive(Allocative, Iterable)] @@ -19,15 +21,15 @@ pub struct SupplySubDataset { impl SupplySubDataset { pub fn import( - parent_path: &str, + path: &MapPath, name: &Option, config: &Config, ) -> color_eyre::Result { let f = |s: &str| { if let Some(name) = name { - format!("{parent_path}/{name}/{s}") + path.join(&format!("{name}/{s}")) } else { - format!("{parent_path}/{s}") + path.join(s) } }; diff --git a/parser/src/datasets/subs/unrealized.rs b/src/parser/datasets/subs/unrealized.rs similarity index 95% rename from parser/src/datasets/subs/unrealized.rs rename to src/parser/datasets/subs/unrealized.rs index 7801cf243..6290f589b 100644 --- a/parser/src/datasets/subs/unrealized.rs +++ b/src/parser/datasets/subs/unrealized.rs @@ -2,9 +2,11 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates}, - states::UnrealizedState, - structs::{BiMap, Config, MapKind}, + parser::{ + datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates}, + states::UnrealizedState, + }, + structs::{BiMap, Config, MapKind, MapPath}, }; #[derive(Allocative, Iterable)] @@ -26,15 +28,15 @@ pub struct UnrealizedSubDataset { impl UnrealizedSubDataset { pub fn import( - parent_path: &str, + path: &MapPath, name: &Option, config: &Config, ) -> color_eyre::Result { let f = |s: &str| { if let Some(name) = name { - format!("{parent_path}/{name}/{s}") + path.join(&format!("{name}/{s}")) } else { - format!("{parent_path}/{s}") + path.join(s) } }; diff --git a/parser/src/datasets/subs/utxo.rs b/src/parser/datasets/subs/utxo.rs similarity index 83% rename from parser/src/datasets/subs/utxo.rs rename to src/parser/datasets/subs/utxo.rs index 61ecf9f90..4f5a071ae 100644 --- a/parser/src/datasets/subs/utxo.rs +++ b/src/parser/datasets/subs/utxo.rs @@ -2,9 +2,11 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, InsertData, MinInitialStates}, - states::UTXOState, - structs::{BiMap, Config, MapKind}, + parser::{ + datasets::{AnyDataset, InsertData, MinInitialStates}, + states::UTXOState, + }, + structs::{BiMap, Config, MapKind, MapPath}, }; #[derive(Allocative, Iterable)] @@ -16,15 +18,15 @@ pub struct UTXOSubDataset { impl UTXOSubDataset { pub fn import( - parent_path: &str, + path: &MapPath, name: &Option, config: &Config, ) -> color_eyre::Result { let f = |s: &str| { if let Some(name) = name { - format!("{parent_path}/{name}/{s}") + path.join(&format!("{name}/{s}")) } else { - format!("{parent_path}/{s}") + path.join(s) } }; diff --git a/parser/src/datasets/transaction.rs b/src/parser/datasets/transaction.rs similarity index 97% rename from parser/src/datasets/transaction.rs rename to src/parser/datasets/transaction.rs index 7ab3ae300..94704de63 100644 --- a/parser/src/datasets/transaction.rs +++ b/src/parser/datasets/transaction.rs @@ -2,12 +2,11 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::InsertData, - structs::{BiMap, Config, HeightMap, MapKind}, + parser::datasets::InsertData, + structs::{BiMap, Config, DateMap, HeightMap, MapKind}, utils::{ ONE_DAY_IN_S, ONE_MONTH_IN_DAYS, ONE_WEEK_IN_DAYS, ONE_YEAR_IN_DAYS, TARGET_BLOCKS_PER_DAY, }, - DateMap, }; use super::{AnyDataset, ComputeData, MinInitialStates}; @@ -52,8 +51,8 @@ pub struct TransactionDataset { } impl TransactionDataset { - pub fn import(parent_path: &str, config: &Config) -> color_eyre::Result { - let f = |s: &str| format!("{parent_path}/{s}"); + pub fn import(config: &Config) -> color_eyre::Result { + let f = |s: &str| config.path_datasets().join(s); let mut s = Self { min_initial_states: MinInitialStates::default(), diff --git a/parser/src/datasets/utxo/dataset.rs b/src/parser/datasets/utxo/dataset.rs similarity index 96% rename from parser/src/datasets/utxo/dataset.rs rename to src/parser/datasets/utxo/dataset.rs index 96b878ddf..01087dbb5 100644 --- a/parser/src/datasets/utxo/dataset.rs +++ b/src/parser/datasets/utxo/dataset.rs @@ -2,9 +2,11 @@ use allocative::Allocative; use struct_iterable::Iterable; use crate::{ - datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates, SubDataset}, - states::UTXOCohortId, - structs::{BiMap, Config, Date, Height}, + parser::{ + datasets::{AnyDataset, ComputeData, InsertData, MinInitialStates, SubDataset}, + states::UTXOCohortId, + }, + structs::{BiMap, Config, Date, Height, MapPath}, }; #[derive(Allocative, Iterable)] @@ -18,7 +20,7 @@ pub struct UTXODataset { impl UTXODataset { pub fn import( - parent_path: &str, + parent_path: &MapPath, id: UTXOCohortId, config: &Config, ) -> color_eyre::Result { diff --git a/parser/src/datasets/utxo/mod.rs b/src/parser/datasets/utxo/mod.rs similarity index 94% rename from parser/src/datasets/utxo/mod.rs rename to src/parser/datasets/utxo/mod.rs index d9bc811bd..c9d0390b4 100644 --- a/parser/src/datasets/utxo/mod.rs +++ b/src/parser/datasets/utxo/mod.rs @@ -7,8 +7,8 @@ use rayon::prelude::*; use itertools::Itertools; use crate::{ - datasets::AnyDatasets, - states::{SplitByUTXOCohort, UTXOCohortId}, + parser::datasets::AnyDatasets, + parser::states::{SplitByUTXOCohort, UTXOCohortId}, structs::{BiMap, Config, Date, Height}, }; @@ -22,13 +22,15 @@ pub struct UTXODatasets { } impl UTXODatasets { - pub fn import(parent_path: &str, config: &Config) -> color_eyre::Result { + pub fn import(config: &Config) -> color_eyre::Result { let mut cohorts = SplitByUTXOCohort::>::default(); + let path_dataset = config.path_datasets(); + cohorts .as_vec() .into_par_iter() - .map(|(_, id)| (id, UTXODataset::import(parent_path, id, config))) + .map(|(_, id)| (id, UTXODataset::import(&path_dataset, id, config))) .collect::>() .into_iter() .try_for_each(|(id, dataset)| -> color_eyre::Result<()> { diff --git a/src/parser/mod.rs b/src/parser/mod.rs new file mode 100644 index 000000000..21acaa3ce --- /dev/null +++ b/src/parser/mod.rs @@ -0,0 +1,52 @@ +use std::{thread::sleep, time::Duration}; + +use biter::bitcoincore_rpc::{Client, RpcApi}; + +mod actions; +mod databases; +mod datasets; +mod price; +mod states; + +pub use actions::*; +pub use databases::*; +pub use datasets::*; +use log::info; +pub use states::*; + +use crate::structs::{Config, Exit}; + +pub fn main( + config: &Config, + rpc: &Client, + exit: &Exit, + mut databases: Databases, + mut datasets: Datasets, +) -> color_eyre::Result<()> { + loop { + let block_count = rpc.get_blockchain_info().unwrap().blocks as usize; + + info!("{block_count} blocks found."); + + iter_blocks( + config, + rpc, + block_count, + exit.clone(), + &mut databases, + &mut datasets, + )?; + + if let Some(delay) = config.delay { + sleep(Duration::from_secs(delay)) + } + + info!("Waiting for a new block..."); + + while block_count == rpc.get_blockchain_info().unwrap().blocks as usize { + sleep(Duration::from_secs(1)) + } + } + + // Ok(()) +} diff --git a/parser/src/price/binance.rs b/src/parser/price/binance.rs similarity index 92% rename from parser/src/price/binance.rs rename to src/parser/price/binance.rs index bbd988b13..3075f010d 100644 --- a/parser/src/price/binance.rs +++ b/src/parser/price/binance.rs @@ -1,26 +1,29 @@ #![allow(dead_code)] -use std::{collections::BTreeMap, fs, path::Path}; +use std::{collections::BTreeMap, fs}; use color_eyre::eyre::ContextCompat; use itertools::Itertools; +use log::info; use serde_json::Value; use crate::{ - io::{Json, INPUTS_FOLDER_PATH}, - structs::{Date, Timestamp, OHLC}, - utils::{log, retry}, + io::Json, + structs::{Config, Date, Timestamp, OHLC}, + utils::retry, }; pub struct Binance; impl Binance { - pub fn read_har_file() -> color_eyre::Result> { - log("binance: read har file"); + pub fn read_har_file(config: &Config) -> color_eyre::Result> { + info!("binance: read har file"); - fs::create_dir_all(INPUTS_FOLDER_PATH)?; + let path = config.path_inputs(); - let path_binance_har = Path::new(INPUTS_FOLDER_PATH).join("binance.har"); + fs::create_dir_all(&path)?; + + let path_binance_har = path.join("binance.har"); let json: BTreeMap = Json::import(&path_binance_har).unwrap_or_default(); @@ -104,7 +107,7 @@ impl Binance { } pub fn fetch_1mn_prices() -> color_eyre::Result> { - log("binance: fetch 1mn"); + info!("binance: fetch 1mn"); retry( |_| { @@ -151,7 +154,7 @@ impl Binance { } pub fn fetch_daily_prices() -> color_eyre::Result> { - log("binance: fetch 1d"); + info!("binance: fetch 1d"); retry( |_| { diff --git a/parser/src/price/kibo.rs b/src/parser/price/kibo.rs similarity index 94% rename from parser/src/price/kibo.rs rename to src/parser/price/kibo.rs index 51a48adc0..031fcf399 100644 --- a/parser/src/price/kibo.rs +++ b/src/parser/price/kibo.rs @@ -3,12 +3,12 @@ use std::{collections::BTreeMap, str::FromStr}; use chrono::NaiveDate; use color_eyre::eyre::ContextCompat; use itertools::Itertools; +use log::info; use serde_json::Value; use crate::{ - structs::{Date, DateMapChunkId, HeightMapChunkId, OHLC}, - utils::{log, retry}, - MapChunkId, + structs::{Date, DateMapChunkId, HeightMapChunkId, MapChunkId, OHLC}, + utils::retry, }; pub struct Kibo; @@ -28,7 +28,7 @@ impl Kibo { } pub fn fetch_height_prices(chunk_id: HeightMapChunkId) -> color_eyre::Result> { - log("kibo: fetch height prices"); + info!("kibo: fetch height prices"); retry( |try_index| { @@ -61,7 +61,7 @@ impl Kibo { } pub fn fetch_date_prices(chunk_id: DateMapChunkId) -> color_eyre::Result> { - log("kibo: fetch date prices"); + info!("kibo: fetch date prices"); retry( |try_index| { diff --git a/parser/src/price/kraken.rs b/src/parser/price/kraken.rs similarity index 97% rename from parser/src/price/kraken.rs rename to src/parser/price/kraken.rs index 7567964b1..dc7e403f4 100644 --- a/parser/src/price/kraken.rs +++ b/src/parser/price/kraken.rs @@ -1,18 +1,19 @@ use std::collections::BTreeMap; use color_eyre::eyre::ContextCompat; +use log::info; use serde_json::Value; use crate::{ structs::{Date, Timestamp, OHLC}, - utils::{log, retry}, + utils::retry, }; pub struct Kraken; impl Kraken { pub fn fetch_1mn_prices() -> color_eyre::Result> { - log("kraken: fetch 1mn"); + info!("kraken: fetch 1mn"); retry( |_| { @@ -66,7 +67,7 @@ impl Kraken { } pub fn fetch_daily_prices() -> color_eyre::Result> { - log("fetch kraken daily"); + info!("fetch kraken daily"); retry( |_| { diff --git a/parser/src/price/mod.rs b/src/parser/price/mod.rs similarity index 100% rename from parser/src/price/mod.rs rename to src/parser/price/mod.rs diff --git a/src/parser/states/_trait.rs b/src/parser/states/_trait.rs new file mode 100644 index 000000000..b8b400cc3 --- /dev/null +++ b/src/parser/states/_trait.rs @@ -0,0 +1,39 @@ +use std::{ + fmt::Debug, + fs, io, + path::{Path, PathBuf}, +}; + +use bincode::{Decode, Encode}; +use serde::{de::DeserializeOwned, Serialize}; + +use crate::{io::Serialization, structs::Config}; + +// https://github.com/djkoloski/rust_serialization_benchmark +pub trait AnyState +where + Self: Debug + Encode + Decode + Serialize + DeserializeOwned, +{ + fn name<'a>() -> &'a str; + + fn path(config: &Config) -> PathBuf { + config.path_states().join(Self::name()) + } + + fn reset(&mut self, config: &Config) -> color_eyre::Result<(), io::Error> { + self.clear(); + fs::remove_file(Self::path(config)) + } + + fn import(config: &Config) -> color_eyre::Result { + let path = Self::path(config); + fs::create_dir_all(&path)?; + Serialization::Binary.import(&path) + } + + fn export(&self, config: &Config) -> color_eyre::Result<()> { + Serialization::Binary.export(Path::new(&Self::path(config)), self) + } + + fn clear(&mut self); +} diff --git a/parser/src/states/cohorts_states/address/cohort_durable_states.rs b/src/parser/states/cohorts_states/address/cohort_durable_states.rs similarity index 97% rename from parser/src/states/cohorts_states/address/cohort_durable_states.rs rename to src/parser/states/cohorts_states/address/cohort_durable_states.rs index 55c367ce8..8bcbb201b 100644 --- a/parser/src/states/cohorts_states/address/cohort_durable_states.rs +++ b/src/parser/states/cohorts_states/address/cohort_durable_states.rs @@ -3,7 +3,7 @@ use std::ops::AddAssign; use allocative::Allocative; use crate::{ - states::{DurableStates, IsZero, OneShotStates, PriceToValue, UnrealizedState}, + parser::states::{DurableStates, IsZero, OneShotStates, PriceToValue, UnrealizedState}, structs::{Amount, Price}, }; diff --git a/parser/src/states/cohorts_states/address/cohort_id.rs b/src/parser/states/cohorts_states/address/cohort_id.rs similarity index 100% rename from parser/src/states/cohorts_states/address/cohort_id.rs rename to src/parser/states/cohorts_states/address/cohort_id.rs diff --git a/parser/src/states/cohorts_states/address/cohorts_durable_states.rs b/src/parser/states/cohorts_states/address/cohorts_durable_states.rs similarity index 99% rename from parser/src/states/cohorts_states/address/cohorts_durable_states.rs rename to src/parser/states/cohorts_states/address/cohorts_durable_states.rs index 932143461..0dece4f9d 100644 --- a/parser/src/states/cohorts_states/address/cohorts_durable_states.rs +++ b/src/parser/states/cohorts_states/address/cohorts_durable_states.rs @@ -5,7 +5,7 @@ use derive_deref::{Deref, DerefMut}; use rayon::prelude::*; use crate::{ - databases::AddressIndexToAddressData, + parser::databases::AddressIndexToAddressData, structs::{AddressData, AddressRealizedData, Amount, Price}, }; diff --git a/parser/src/states/cohorts_states/address/cohorts_input_states.rs b/src/parser/states/cohorts_states/address/cohorts_input_states.rs similarity index 97% rename from parser/src/states/cohorts_states/address/cohorts_input_states.rs rename to src/parser/states/cohorts_states/address/cohorts_input_states.rs index 5f6ef9afa..0da4376c5 100644 --- a/parser/src/states/cohorts_states/address/cohorts_input_states.rs +++ b/src/parser/states/cohorts_states/address/cohorts_input_states.rs @@ -1,7 +1,7 @@ use derive_deref::{Deref, DerefMut}; use crate::{ - states::InputState, + parser::states::InputState, structs::{AddressRealizedData, Amount, LiquidityClassification}, }; diff --git a/parser/src/states/cohorts_states/address/cohorts_one_shot_states.rs b/src/parser/states/cohorts_states/address/cohorts_one_shot_states.rs similarity index 81% rename from parser/src/states/cohorts_states/address/cohorts_one_shot_states.rs rename to src/parser/states/cohorts_states/address/cohorts_one_shot_states.rs index 042e44ea4..bda560fa6 100644 --- a/parser/src/states/cohorts_states/address/cohorts_one_shot_states.rs +++ b/src/parser/states/cohorts_states/address/cohorts_one_shot_states.rs @@ -1,6 +1,6 @@ use derive_deref::{Deref, DerefMut}; -use crate::states::OneShotStates; +use crate::parser::states::OneShotStates; use super::SplitByAddressCohort; diff --git a/parser/src/states/cohorts_states/address/cohorts_output_states.rs b/src/parser/states/cohorts_states/address/cohorts_output_states.rs similarity index 97% rename from parser/src/states/cohorts_states/address/cohorts_output_states.rs rename to src/parser/states/cohorts_states/address/cohorts_output_states.rs index d0a682129..155ef98d2 100644 --- a/parser/src/states/cohorts_states/address/cohorts_output_states.rs +++ b/src/parser/states/cohorts_states/address/cohorts_output_states.rs @@ -1,7 +1,7 @@ use derive_deref::{Deref, DerefMut}; use crate::{ - states::OutputState, + parser::states::OutputState, structs::{AddressRealizedData, Amount, LiquidityClassification}, }; diff --git a/parser/src/states/cohorts_states/address/cohorts_realized_states.rs b/src/parser/states/cohorts_states/address/cohorts_realized_states.rs similarity index 98% rename from parser/src/states/cohorts_states/address/cohorts_realized_states.rs rename to src/parser/states/cohorts_states/address/cohorts_realized_states.rs index 524cc31f6..22d2a228b 100644 --- a/parser/src/states/cohorts_states/address/cohorts_realized_states.rs +++ b/src/parser/states/cohorts_states/address/cohorts_realized_states.rs @@ -1,7 +1,7 @@ use derive_deref::{Deref, DerefMut}; use crate::{ - states::RealizedState, + parser::states::RealizedState, structs::{AddressRealizedData, LiquidityClassification, Price}, }; diff --git a/parser/src/states/cohorts_states/address/mod.rs b/src/parser/states/cohorts_states/address/mod.rs similarity index 100% rename from parser/src/states/cohorts_states/address/mod.rs rename to src/parser/states/cohorts_states/address/mod.rs diff --git a/parser/src/states/cohorts_states/address/split_by_address_cohort.rs b/src/parser/states/cohorts_states/address/split_by_address_cohort.rs similarity index 100% rename from parser/src/states/cohorts_states/address/split_by_address_cohort.rs rename to src/parser/states/cohorts_states/address/split_by_address_cohort.rs diff --git a/parser/src/states/cohorts_states/any/capitalization_state.rs b/src/parser/states/cohorts_states/any/capitalization_state.rs similarity index 100% rename from parser/src/states/cohorts_states/any/capitalization_state.rs rename to src/parser/states/cohorts_states/any/capitalization_state.rs diff --git a/parser/src/states/cohorts_states/any/durable_states.rs b/src/parser/states/cohorts_states/any/durable_states.rs similarity index 100% rename from parser/src/states/cohorts_states/any/durable_states.rs rename to src/parser/states/cohorts_states/any/durable_states.rs diff --git a/parser/src/states/cohorts_states/any/input_state.rs b/src/parser/states/cohorts_states/any/input_state.rs similarity index 100% rename from parser/src/states/cohorts_states/any/input_state.rs rename to src/parser/states/cohorts_states/any/input_state.rs diff --git a/parser/src/states/cohorts_states/any/mod.rs b/src/parser/states/cohorts_states/any/mod.rs similarity index 100% rename from parser/src/states/cohorts_states/any/mod.rs rename to src/parser/states/cohorts_states/any/mod.rs diff --git a/parser/src/states/cohorts_states/any/one_shot_states.rs b/src/parser/states/cohorts_states/any/one_shot_states.rs similarity index 100% rename from parser/src/states/cohorts_states/any/one_shot_states.rs rename to src/parser/states/cohorts_states/any/one_shot_states.rs diff --git a/parser/src/states/cohorts_states/any/output_state.rs b/src/parser/states/cohorts_states/any/output_state.rs similarity index 100% rename from parser/src/states/cohorts_states/any/output_state.rs rename to src/parser/states/cohorts_states/any/output_state.rs diff --git a/parser/src/states/cohorts_states/any/price_paid_state.rs b/src/parser/states/cohorts_states/any/price_paid_state.rs similarity index 100% rename from parser/src/states/cohorts_states/any/price_paid_state.rs rename to src/parser/states/cohorts_states/any/price_paid_state.rs diff --git a/parser/src/states/cohorts_states/any/price_to_value.rs b/src/parser/states/cohorts_states/any/price_to_value.rs similarity index 100% rename from parser/src/states/cohorts_states/any/price_to_value.rs rename to src/parser/states/cohorts_states/any/price_to_value.rs diff --git a/parser/src/states/cohorts_states/any/realized_state.rs b/src/parser/states/cohorts_states/any/realized_state.rs similarity index 100% rename from parser/src/states/cohorts_states/any/realized_state.rs rename to src/parser/states/cohorts_states/any/realized_state.rs diff --git a/parser/src/states/cohorts_states/any/supply_state.rs b/src/parser/states/cohorts_states/any/supply_state.rs similarity index 100% rename from parser/src/states/cohorts_states/any/supply_state.rs rename to src/parser/states/cohorts_states/any/supply_state.rs diff --git a/parser/src/states/cohorts_states/any/unrealized_state.rs b/src/parser/states/cohorts_states/any/unrealized_state.rs similarity index 100% rename from parser/src/states/cohorts_states/any/unrealized_state.rs rename to src/parser/states/cohorts_states/any/unrealized_state.rs diff --git a/parser/src/states/cohorts_states/any/utxo_state.rs b/src/parser/states/cohorts_states/any/utxo_state.rs similarity index 100% rename from parser/src/states/cohorts_states/any/utxo_state.rs rename to src/parser/states/cohorts_states/any/utxo_state.rs diff --git a/parser/src/states/cohorts_states/mod.rs b/src/parser/states/cohorts_states/mod.rs similarity index 100% rename from parser/src/states/cohorts_states/mod.rs rename to src/parser/states/cohorts_states/mod.rs diff --git a/parser/src/states/cohorts_states/utxo/cohort_durable_states.rs b/src/parser/states/cohorts_states/utxo/cohort_durable_states.rs similarity index 97% rename from parser/src/states/cohorts_states/utxo/cohort_durable_states.rs rename to src/parser/states/cohorts_states/utxo/cohort_durable_states.rs index af47446e4..13ccce07a 100644 --- a/parser/src/states/cohorts_states/utxo/cohort_durable_states.rs +++ b/src/parser/states/cohorts_states/utxo/cohort_durable_states.rs @@ -1,7 +1,7 @@ use allocative::Allocative; use crate::{ - states::{DurableStates, OneShotStates, PriceToValue, UnrealizedState}, + parser::states::{DurableStates, OneShotStates, PriceToValue, UnrealizedState}, structs::{Amount, Price}, }; diff --git a/parser/src/states/cohorts_states/utxo/cohort_filter.rs b/src/parser/states/cohorts_states/utxo/cohort_filter.rs similarity index 100% rename from parser/src/states/cohorts_states/utxo/cohort_filter.rs rename to src/parser/states/cohorts_states/utxo/cohort_filter.rs diff --git a/parser/src/states/cohorts_states/utxo/cohort_filters.rs b/src/parser/states/cohorts_states/utxo/cohort_filters.rs similarity index 100% rename from parser/src/states/cohorts_states/utxo/cohort_filters.rs rename to src/parser/states/cohorts_states/utxo/cohort_filters.rs diff --git a/parser/src/states/cohorts_states/utxo/cohort_id.rs b/src/parser/states/cohorts_states/utxo/cohort_id.rs similarity index 100% rename from parser/src/states/cohorts_states/utxo/cohort_id.rs rename to src/parser/states/cohorts_states/utxo/cohort_id.rs diff --git a/parser/src/states/cohorts_states/utxo/cohorts_durable_states.rs b/src/parser/states/cohorts_states/utxo/cohorts_durable_states.rs similarity index 99% rename from parser/src/states/cohorts_states/utxo/cohorts_durable_states.rs rename to src/parser/states/cohorts_states/utxo/cohorts_durable_states.rs index 16cafddcf..2ba92dbf2 100644 --- a/parser/src/states/cohorts_states/utxo/cohorts_durable_states.rs +++ b/src/parser/states/cohorts_states/utxo/cohorts_durable_states.rs @@ -3,7 +3,7 @@ use derive_deref::{Deref, DerefMut}; use rayon::prelude::*; use crate::{ - states::DateDataVec, + parser::states::DateDataVec, structs::{Amount, BlockData, Price, SentData, Timestamp}, }; diff --git a/parser/src/states/cohorts_states/utxo/cohorts_one_shot_states.rs b/src/parser/states/cohorts_states/utxo/cohorts_one_shot_states.rs similarity index 81% rename from parser/src/states/cohorts_states/utxo/cohorts_one_shot_states.rs rename to src/parser/states/cohorts_states/utxo/cohorts_one_shot_states.rs index efd4db6a5..328fddf7a 100644 --- a/parser/src/states/cohorts_states/utxo/cohorts_one_shot_states.rs +++ b/src/parser/states/cohorts_states/utxo/cohorts_one_shot_states.rs @@ -1,6 +1,6 @@ use derive_deref::{Deref, DerefMut}; -use crate::states::OneShotStates; +use crate::parser::states::OneShotStates; use super::SplitByUTXOCohort; diff --git a/parser/src/states/cohorts_states/utxo/cohorts_sent_states.rs b/src/parser/states/cohorts_states/utxo/cohorts_sent_states.rs similarity index 98% rename from parser/src/states/cohorts_states/utxo/cohorts_sent_states.rs rename to src/parser/states/cohorts_states/utxo/cohorts_sent_states.rs index 9b0da8a05..f09d908f7 100644 --- a/parser/src/states/cohorts_states/utxo/cohorts_sent_states.rs +++ b/src/parser/states/cohorts_states/utxo/cohorts_sent_states.rs @@ -3,7 +3,7 @@ use std::{cmp::Ordering, collections::BTreeMap}; use derive_deref::{Deref, DerefMut}; use crate::{ - states::{DateDataVec, InputState, RealizedState}, + parser::states::{DateDataVec, InputState, RealizedState}, structs::{BlockPath, Price, SentData, Timestamp}, }; diff --git a/parser/src/states/cohorts_states/utxo/mod.rs b/src/parser/states/cohorts_states/utxo/mod.rs similarity index 100% rename from parser/src/states/cohorts_states/utxo/mod.rs rename to src/parser/states/cohorts_states/utxo/mod.rs diff --git a/parser/src/states/cohorts_states/utxo/split_by_utxo_cohort.rs b/src/parser/states/cohorts_states/utxo/split_by_utxo_cohort.rs similarity index 100% rename from parser/src/states/cohorts_states/utxo/split_by_utxo_cohort.rs rename to src/parser/states/cohorts_states/utxo/split_by_utxo_cohort.rs diff --git a/parser/src/states/counters.rs b/src/parser/states/counters.rs similarity index 100% rename from parser/src/states/counters.rs rename to src/parser/states/counters.rs diff --git a/parser/src/states/date_data_vec.rs b/src/parser/states/date_data_vec.rs similarity index 100% rename from parser/src/states/date_data_vec.rs rename to src/parser/states/date_data_vec.rs diff --git a/parser/src/states/mod.rs b/src/parser/states/mod.rs similarity index 58% rename from parser/src/states/mod.rs rename to src/parser/states/mod.rs index 878b6d672..f67cec892 100644 --- a/parser/src/states/mod.rs +++ b/src/parser/states/mod.rs @@ -11,8 +11,9 @@ use allocative::Allocative; pub use cohorts_states::*; use counters::*; use date_data_vec::*; +use log::info; -use crate::utils::log; +use crate::structs::Config; #[derive(Default, Allocative)] pub struct States { @@ -23,12 +24,10 @@ pub struct States { } impl States { - pub fn import() -> color_eyre::Result { - let date_data_vec_handle = thread::spawn(DateDataVec::import); + pub fn import(config: &Config) -> color_eyre::Result { + let date_data_vec = DateDataVec::import(config)?; - let address_counters = Counters::import()?; - - let date_data_vec = date_data_vec_handle.join().unwrap()?; + let address_counters = Counters::import(config)?; Ok(Self { address_cohorts_durable_states: None, @@ -38,24 +37,24 @@ impl States { }) } - pub fn reset(&mut self, include_addresses: bool) { - log("Reseting all states..."); + pub fn reset(&mut self, config: &Config, include_addresses: bool) { + info!("Reseting all states..."); - let _ = self.date_data_vec.reset(); + let _ = self.date_data_vec.reset(config); self.utxo_cohorts_durable_states = None; if include_addresses { - let _ = self.address_counters.reset(); + let _ = self.address_counters.reset(config); self.address_cohorts_durable_states = None; } } - pub fn export(&self) -> color_eyre::Result<()> { + pub fn export(&self, config: &Config) -> color_eyre::Result<()> { thread::scope(|s| { - s.spawn(|| self.address_counters.export().unwrap()); - s.spawn(|| self.date_data_vec.export().unwrap()); + s.spawn(|| self.address_counters.export(config).unwrap()); + s.spawn(|| self.date_data_vec.export(config).unwrap()); }); Ok(()) diff --git a/server/src/api/handlers/dataset.rs b/src/server/api/handlers/dataset.rs similarity index 80% rename from server/src/api/handlers/dataset.rs rename to src/server/api/handlers/dataset.rs index b0ab851f8..c1fa63cec 100644 --- a/server/src/api/handlers/dataset.rs +++ b/src/server/api/handlers/dataset.rs @@ -9,15 +9,17 @@ use color_eyre::{eyre::eyre, owo_colors::OwoColorize}; use reqwest::StatusCode; use serde::Deserialize; -use parser::{ - log, Date, DateMap, Height, HeightMap, Json, MapChunkId, COMPRESSED_BIN_EXTENSION, - HEIGHT_MAP_CHUNK_SIZE, JSON_EXTENSION, OHLC, -}; - use crate::{ - api::structs::{Chunk, Kind, Route}, - header_map::HeaderMapUtils, - AppState, + io::{Json, COMPRESSED_BIN_EXTENSION, JSON_EXTENSION}, + server::{ + api::{ + structs::{Chunk, Kind, Route}, + API_URL_PREFIX, + }, + header_map::HeaderMapUtils, + log_result, AppState, + }, + structs::{Date, DateMap, Height, HeightMap, MapChunkId, HEIGHT_MAP_CHUNK_SIZE, OHLC}, }; use super::{ @@ -37,11 +39,32 @@ pub async fn dataset_handler( query: Query, State(app_state): State, ) -> Response { - match _dataset_handler(headers, path, query, app_state) { - Ok(response) => response, + match _dataset_handler(headers, &path, &query, app_state) { + Ok(response) => { + log_result( + response.status(), + &format!( + "{API_URL_PREFIX}/{}{}{}", + path.0, + query.chunk.map_or("".to_string(), |chunk| format!( + "{}{chunk}", + "?chunk=".bright_black() + )), + query.all.map_or("".to_string(), |all| format!( + "{}{all}", + "?all=".bright_black() + )) + ), + ); + + response + } Err(error) => { - let mut response = - (StatusCode::INTERNAL_SERVER_ERROR, error.to_string()).into_response(); + let code = StatusCode::INTERNAL_SERVER_ERROR; + + log_result(code, &format!("{API_URL_PREFIX}/{}", path.0)); + + let mut response = (code, error.to_string()).into_response(); response.headers_mut().insert_cors(); @@ -55,27 +78,14 @@ const HEIGHT_PREFIX: &str = "height-to-"; fn _dataset_handler( headers: HeaderMap, - Path(path): Path, - query: Query, + Path(path): &Path, + query: &Query, AppState { routes }: AppState, ) -> color_eyre::Result { if query.chunk.is_some() && query.all.is_some() { return Err(eyre!("chunk and all are exclusive")); } - log(&format!( - "{}{}{}", - path, - query.chunk.map_or("".to_string(), |chunk| format!( - "{}{chunk}", - "?chunk=".bright_black() - )), - query.all.map_or("".to_string(), |all| format!( - "{}{all}", - "?all=".bright_black() - )) - )); - let (kind, id, route) = if path.starts_with(DATE_PREFIX) { let id = convert_path_to_id(path.strip_prefix(DATE_PREFIX).unwrap()); let route = routes.date.get(&id); @@ -85,7 +95,7 @@ fn _dataset_handler( let route = routes.height.get(&id); (Kind::Height, id, route) } else { - let id = convert_path_to_id(&path); + let id = convert_path_to_id(path); let route = routes.last.get(&id); (Kind::Last, id, route) }; @@ -121,7 +131,7 @@ fn _dataset_handler( } } - let (date, response) = headers.check_if_modified_since(&route.file_path).unwrap(); + let (date, response) = headers.check_if_modified_since(&route.file_path)?; if let Some(response) = response { return Ok(response); @@ -163,7 +173,7 @@ fn process_datasets( kind: Kind, chunk: &mut Option, route: &mut Route, - query: Query, + query: &Query, datasets: BTreeMap, ) -> color_eyre::Result<()> where diff --git a/server/src/api/handlers/extension.rs b/src/server/api/handlers/extension.rs similarity index 100% rename from server/src/api/handlers/extension.rs rename to src/server/api/handlers/extension.rs diff --git a/server/src/api/handlers/fallback.rs b/src/server/api/handlers/fallback.rs similarity index 94% rename from server/src/api/handlers/fallback.rs rename to src/server/api/handlers/fallback.rs index 158261d07..3ce9c7618 100644 --- a/server/src/api/handlers/fallback.rs +++ b/src/server/api/handlers/fallback.rs @@ -1,7 +1,7 @@ use axum::{extract::State, http::HeaderMap, response::Response}; use reqwest::header::HOST; -use crate::AppState; +use crate::server::AppState; use super::response::{generic_to_reponse, update_reponse_headers}; diff --git a/server/src/api/handlers/mod.rs b/src/server/api/handlers/mod.rs similarity index 100% rename from server/src/api/handlers/mod.rs rename to src/server/api/handlers/mod.rs diff --git a/server/src/api/handlers/response.rs b/src/server/api/handlers/response.rs similarity index 96% rename from server/src/api/handlers/response.rs rename to src/server/api/handlers/response.rs index edac89415..449c73dd1 100644 --- a/server/src/api/handlers/response.rs +++ b/src/server/api/handlers/response.rs @@ -2,13 +2,15 @@ use std::fmt::Debug; use axum::response::{IntoResponse, Json, Response}; use bincode::Decode; -use parser::{Date, MapValue, SerializedBTreeMap, SerializedVec}; use serde::de::DeserializeOwned; use serde::Serialize; use crate::{ - api::structs::{Chunk, Kind, Route}, - header_map::HeaderMapUtils, + server::{ + api::structs::{Chunk, Kind, Route}, + header_map::HeaderMapUtils, + }, + structs::{Date, MapValue, SerializedBTreeMap, SerializedVec}, }; use super::extension::Extension; diff --git a/server/src/api/mod.rs b/src/server/api/mod.rs similarity index 50% rename from server/src/api/mod.rs rename to src/server/api/mod.rs index f1690fc30..5dfdc8871 100644 --- a/server/src/api/mod.rs +++ b/src/server/api/mod.rs @@ -1,19 +1,21 @@ use axum::{routing::get, Router}; use handlers::{dataset_handler, fallback}; -use crate::AppState; +use super::AppState; mod handlers; pub mod structs; +pub const API_URL_PREFIX: &str = "/api"; + pub trait ApiRoutes { fn add_api_routes(self) -> Self; } impl ApiRoutes for Router { fn add_api_routes(self) -> Self { - self.route("/api/*path", get(dataset_handler)) - .route("/api/", get(fallback)) - .route("/api", get(fallback)) + self.route(&format!("{API_URL_PREFIX}/*path"), get(dataset_handler)) + .route(&format!("{API_URL_PREFIX}/"), get(fallback)) + .route(API_URL_PREFIX, get(fallback)) } } diff --git a/server/src/api/structs/chunk.rs b/src/server/api/structs/chunk.rs similarity index 100% rename from server/src/api/structs/chunk.rs rename to src/server/api/structs/chunk.rs diff --git a/server/src/api/structs/kind.rs b/src/server/api/structs/kind.rs similarity index 100% rename from server/src/api/structs/kind.rs rename to src/server/api/structs/kind.rs diff --git a/server/src/api/structs/mod.rs b/src/server/api/structs/mod.rs similarity index 100% rename from server/src/api/structs/mod.rs rename to src/server/api/structs/mod.rs diff --git a/server/src/api/structs/paths.rs b/src/server/api/structs/paths.rs similarity index 88% rename from server/src/api/structs/paths.rs rename to src/server/api/structs/paths.rs index 1461e0138..9e707278f 100644 --- a/server/src/api/structs/paths.rs +++ b/src/server/api/structs/paths.rs @@ -3,7 +3,7 @@ use std::collections::BTreeMap; use derive_deref::{Deref, DerefMut}; use serde::Serialize; -use crate::Grouped; +use crate::server::Grouped; #[derive(Clone, Default, Deref, DerefMut, Debug, Serialize)] pub struct Paths(pub Grouped>); diff --git a/src/server/api/structs/routes.rs b/src/server/api/structs/routes.rs new file mode 100644 index 000000000..8a5caf4bc --- /dev/null +++ b/src/server/api/structs/routes.rs @@ -0,0 +1,159 @@ +use std::{ + collections::{BTreeMap, HashMap}, + path::PathBuf, +}; + +use derive_deref::{Deref, DerefMut}; +use itertools::Itertools; + +use crate::{ + io::Serialization, + server::{api::API_URL_PREFIX, Grouped}, + structs::Config, +}; + +use super::Paths; + +#[derive(Clone, Debug)] +pub struct Route { + pub url_path: String, + pub file_path: PathBuf, + pub values_type: String, + pub serialization: Serialization, +} + +#[derive(Clone, Default, Deref, DerefMut)] +pub struct Routes(pub Grouped>); + +const WEBSITE_TYPES_PATH: &str = "../website/scripts/types"; + +impl Routes { + pub fn build(paths_to_type: BTreeMap, config: &Config) -> Self { + let mut routes = Routes::default(); + + paths_to_type.into_iter().for_each(|(file_path, value)| { + let serialization = + Serialization::try_from(&file_path).unwrap_or(Serialization::Binary); + + let file_path_ser = file_path.to_str().unwrap().to_owned(); + let split_key = file_path_ser.replace( + &format!("{}/", config.path_datasets().to_str().unwrap()), + "", + ); + let split_key = + split_key.replace(&format!("{}/", config.path_price().to_str().unwrap()), ""); + let mut split_key = split_key.split('/').collect_vec(); + let last = split_key.pop().unwrap().to_owned(); + let last = last.split('.').next().unwrap(); + + // Use case for: "../datasets/last": "Value", + if split_key.is_empty() { + split_key.push("last"); + } + + let map_key = split_key.iter().join("_"); + + let url_path = split_key.iter().join("-"); + + let values_type = value.to_owned(); + + match last { + "date" => { + routes.date.insert( + map_key, + Route { + url_path: format!("date-to-{url_path}"), + file_path, + values_type, + serialization, + }, + ); + } + "height" => { + routes.height.insert( + map_key, + Route { + url_path: format!("height-to-{url_path}"), + file_path, + values_type, + serialization, + }, + ); + } + "last" => { + routes.last.insert( + map_key, + Route { + url_path, + file_path, + values_type, + serialization, + }, + ); + } + _ => { + dbg!(&file_path, value, &last, &split_key); + panic!("") + } + } + }); + + routes + } + + pub fn generate_dts_file(&self) { + let map_to_type = |name: &str, map: &HashMap| -> String { + let paths = map + .values() + .map(|route| format!("\"{}\"", route.url_path)) + .join(" | "); + + format!("export type {}Path = {};\n", name, paths) + }; + + let date_type = map_to_type("Date", &self.date); + + let height_type = map_to_type("Height", &self.height); + + let last_type = map_to_type("Last", &self.last); + + // fs::write( + // format!("{WEBSITE_TYPES_PATH}/paths.d.ts"), + // format!("// This file is auto generated by the server\n// Manual changes are forbidden\n\n{date_type}\n{height_type}\n{last_type}"), + // ) + // .unwrap(); + } + + pub fn to_full_paths(&self, host: String) -> Paths { + let url = { + let scheme = if host.contains("0.0.0.0") || host.contains("localhost") { + "http" + } else { + "https" + }; + + format!("{scheme}://{host}") + }; + + let transform = |map: &HashMap| -> BTreeMap { + map.iter() + .map(|(key, route)| { + ( + key.to_owned(), + format!("{url}{API_URL_PREFIX}/{}", route.url_path.to_owned()), + ) + }) + .collect() + }; + + let date_paths = transform(&self.date); + let height_paths = transform(&self.height); + let last_paths = transform(&self.last); + + Paths(Grouped { + date: date_paths, + height: height_paths, + last: last_paths, + }) + } +} diff --git a/server/src/header_map.rs b/src/server/header_map.rs similarity index 98% rename from server/src/header_map.rs rename to src/server/header_map.rs index f1af0527b..145f42796 100644 --- a/server/src/header_map.rs +++ b/src/server/header_map.rs @@ -6,7 +6,7 @@ use axum::{ response::IntoResponse, }; use chrono::{DateTime, Timelike, Utc}; -use parser::log; +use log::info; use reqwest::{ header::{HOST, IF_MODIFIED_SINCE}, StatusCode, @@ -163,7 +163,7 @@ impl HeaderMapUtils for HeaderMap { "png" => self.insert_content_type_image_png(), "webmanifest" => self.insert_content_type_application_manifest_json(), extension => { - log(&format!("Extension unsupported: {extension}")); + info!("Extension unsupported: {extension}"); panic!() } } diff --git a/server/src/main.rs b/src/server/mod.rs similarity index 55% rename from server/src/main.rs rename to src/server/mod.rs index 3aa94b4f6..4d0de972b 100644 --- a/server/src/main.rs +++ b/src/server/mod.rs @@ -1,13 +1,17 @@ -use std::sync::Arc; +use std::{collections::BTreeMap, path::PathBuf, sync::Arc}; use api::{structs::Routes, ApiRoutes}; -use axum::{serve, Router}; -use parser::{log, reset_logs}; +use axum::{routing::get, serve, Router}; +use color_eyre::owo_colors::OwoColorize; +use log::{error, info}; +use reqwest::StatusCode; use serde::Serialize; use tokio::net::TcpListener; use tower_http::compression::CompressionLayer; use website::WebsiteRoutes; +use crate::structs::Config; + mod api; mod header_map; mod website; @@ -24,13 +28,11 @@ pub struct AppState { routes: Arc, } -#[tokio::main] -async fn main() -> color_eyre::Result<()> { - color_eyre::install()?; - - reset_logs(); - - let routes = Routes::build(); +pub async fn main( + paths_to_type: BTreeMap, + config: &Config, +) -> color_eyre::Result<()> { + let routes = Routes::build(paths_to_type, config); routes.generate_dts_file(); @@ -47,6 +49,7 @@ async fn main() -> color_eyre::Result<()> { let router = Router::new() .add_api_routes() .add_website_routes() + .route("/version", get(env!("CARGO_PKG_VERSION"))) .with_state(state) .layer(compression_layer); @@ -61,10 +64,19 @@ async fn main() -> color_eyre::Result<()> { port += 1; } - log(&format!("Starting server on port {port}...")); + info!("Starting server on port {port}..."); let listener = listener.unwrap(); serve(listener, router).await?; Ok(()) } + +pub fn log_result(code: StatusCode, path: &str) { + match code { + StatusCode::INTERNAL_SERVER_ERROR => error!("{} {}", code.as_u16().red(), path), + StatusCode::NOT_MODIFIED => info!("{} {}", code.as_u16().bright_black(), path), + StatusCode::OK => info!("{} {}", code.as_u16().green(), path), + _ => error!("{} {}", code.as_u16().red(), path), + } +} diff --git a/server/src/website/handlers/_minify.rs b/src/server/website/handlers/_minify.rs similarity index 100% rename from server/src/website/handlers/_minify.rs rename to src/server/website/handlers/_minify.rs diff --git a/server/src/website/handlers/file.rs b/src/server/website/handlers/file.rs similarity index 53% rename from server/src/website/handlers/file.rs rename to src/server/website/handlers/file.rs index 6f1e4f636..3c6159a53 100644 --- a/server/src/website/handlers/file.rs +++ b/src/server/website/handlers/file.rs @@ -9,54 +9,77 @@ use axum::{ http::HeaderMap, response::{IntoResponse, Response}, }; -use parser::log; +use log::{error, info}; use reqwest::StatusCode; -use crate::header_map::HeaderMapUtils; +use crate::server::{header_map::HeaderMapUtils, log_result}; use super::minify_js; -const WEBSITE_PATH: &str = "../website/"; +const WEBSITE_PATH: &str = "./src/website/"; pub async fn file_handler(headers: HeaderMap, path: extract::Path) -> Response { - let mut path = path.0.replace("..", "").replace("\\", ""); - - if path.ends_with("Cargo.toml") { - path = "../server/Cargo.toml".to_owned(); - } - - let mut path = str_to_path(&path); - - if !path.exists() { - if path.extension().is_some() { - let mut response: Response = ( - StatusCode::INTERNAL_SERVER_ERROR, - "File doesn't exist".to_string(), - ) - .into_response(); - - response.headers_mut().insert_cors(); - - return response; - } else { - path = str_to_path("index.html"); - } - } - - path_to_response(headers, &path) + any_handler(headers, Some(path)) } pub async fn index_handler(headers: HeaderMap) -> Response { - path_to_response(headers, &str_to_path("index.html")) + any_handler(headers, None) } -fn path_to_response(headers: HeaderMap, path: &Path) -> Response { - log(&path.to_str().unwrap().replace(WEBSITE_PATH, "")); +fn any_handler(headers: HeaderMap, path: Option>) -> Response { + let response = if let Some(path) = path.as_ref() { + let path = path.0.replace("..", "").replace("\\", ""); - let (date, response) = headers.check_if_modified_since(path).unwrap(); + let mut path = str_to_path(&path); + + if !path.exists() { + if path.extension().is_some() { + let mut response: Response = ( + StatusCode::INTERNAL_SERVER_ERROR, + "File doesn't exist".to_string(), + ) + .into_response(); + + response.headers_mut().insert_cors(); + + return response; + } else { + path = str_to_path("index.html"); + } + } + + path_to_response(&headers, &path) + } else { + path_to_response(&headers, &str_to_path("index.html")) + }; + + log_result( + response.status(), + &format!("/{}", path.map_or("".to_owned(), |p| p.0)), + ); + + response +} + +fn path_to_response(headers: &HeaderMap, path: &Path) -> Response { + match _path_to_response(headers, path) { + Ok(response) => response, + Err(error) => { + let mut response = + (StatusCode::INTERNAL_SERVER_ERROR, error.to_string()).into_response(); + + response.headers_mut().insert_cors(); + + response + } + } +} + +fn _path_to_response(headers: &HeaderMap, path: &Path) -> color_eyre::Result { + let (date, response) = headers.check_if_modified_since(path)?; if let Some(response) = response { - return response; + return Ok(response); } let mut response; @@ -74,9 +97,9 @@ fn path_to_response(headers: HeaderMap, path: &Path) -> Response { response = Response::new(content.into()); } else { let content = fs::read(path).unwrap_or_else(|error| { - log(&error.to_string()); + error!("{error}"); let path = path.to_str().unwrap(); - log(&format!("Can't read file {path}")); + info!("Can't read file {path}"); panic!("") }); @@ -108,7 +131,7 @@ fn path_to_response(headers: HeaderMap, path: &Path) -> Response { headers.insert_last_modified(date); - response + Ok(response) } fn str_to_path(path: &str) -> PathBuf { diff --git a/server/src/website/handlers/minify.rs b/src/server/website/handlers/minify.rs similarity index 100% rename from server/src/website/handlers/minify.rs rename to src/server/website/handlers/minify.rs diff --git a/server/src/website/handlers/mod.rs b/src/server/website/handlers/mod.rs similarity index 100% rename from server/src/website/handlers/mod.rs rename to src/server/website/handlers/mod.rs diff --git a/server/src/website/mod.rs b/src/server/website/mod.rs similarity index 94% rename from server/src/website/mod.rs rename to src/server/website/mod.rs index 77cd1a1c9..fcb31516e 100644 --- a/server/src/website/mod.rs +++ b/src/server/website/mod.rs @@ -4,7 +4,7 @@ mod handlers; use handlers::{file_handler, index_handler}; -use crate::AppState; +use super::AppState; pub trait WebsiteRoutes { fn add_website_routes(self) -> Self; diff --git a/parser/src/structs/address.rs b/src/structs/address.rs similarity index 97% rename from parser/src/structs/address.rs rename to src/structs/address.rs index 7fc0255d5..b0e7930a2 100644 --- a/parser/src/structs/address.rs +++ b/src/structs/address.rs @@ -1,4 +1,4 @@ -use bitcoin_hashes::{hash160, Hash}; +use bitcoin_hashes::hash160; use biter::bitcoin::TxOut; use super::{AddressType, Counter, U8x19, U8x31}; @@ -56,7 +56,7 @@ impl Address { let hash = hash160::Hash::hash(pk); - let (prefix, rest) = Self::split_slice(&hash[..]); + let (prefix, rest) = Self::split_slice(&hash.as_byte_array()[..]); Self::P2PK((prefix, rest.into())) } else if script.is_p2pkh() { diff --git a/parser/src/structs/address_data.rs b/src/structs/address_data.rs similarity index 100% rename from parser/src/structs/address_data.rs rename to src/structs/address_data.rs diff --git a/parser/src/structs/address_liquidity.rs b/src/structs/address_liquidity.rs similarity index 100% rename from parser/src/structs/address_liquidity.rs rename to src/structs/address_liquidity.rs diff --git a/parser/src/structs/address_realized_data.rs b/src/structs/address_realized_data.rs similarity index 100% rename from parser/src/structs/address_realized_data.rs rename to src/structs/address_realized_data.rs diff --git a/parser/src/structs/address_size.rs b/src/structs/address_size.rs similarity index 100% rename from parser/src/structs/address_size.rs rename to src/structs/address_size.rs diff --git a/parser/src/structs/address_split.rs b/src/structs/address_split.rs similarity index 100% rename from parser/src/structs/address_split.rs rename to src/structs/address_split.rs diff --git a/parser/src/structs/address_type.rs b/src/structs/address_type.rs similarity index 100% rename from parser/src/structs/address_type.rs rename to src/structs/address_type.rs diff --git a/parser/src/structs/amount.rs b/src/structs/amount.rs similarity index 100% rename from parser/src/structs/amount.rs rename to src/structs/amount.rs diff --git a/parser/src/structs/any_map.rs b/src/structs/any_map.rs similarity index 53% rename from parser/src/structs/any_map.rs rename to src/structs/any_map.rs index 7f816b1c3..455a5bdfc 100644 --- a/parser/src/structs/any_map.rs +++ b/src/structs/any_map.rs @@ -2,23 +2,26 @@ use std::path::{Path, PathBuf}; use serde_json::Value; -use super::MapKind; +use super::{MapKind, MapPath}; pub trait AnyMap { fn path(&self) -> &Path; - fn path_last(&self) -> &Option; + fn path_last(&self) -> &Option; fn last_value(&self) -> Option; fn t_name(&self) -> &str; - fn exported_path_with_t_name(&self) -> Vec<(&Path, &str)> { - let t_name = self.t_name(); + fn get_paths_to_type(&self) -> Vec<(PathBuf, String)> { + let t_name = self.t_name().to_string(); if let Some(path_last) = self.path_last() { - vec![(self.path(), t_name), (path_last, t_name)] + vec![ + (self.path().to_owned(), t_name.clone()), + (path_last.unwrap().to_owned(), t_name), + ] } else { - vec![(self.path(), t_name)] + vec![(self.path().to_owned(), t_name)] } } diff --git a/parser/src/structs/array.rs b/src/structs/array.rs similarity index 100% rename from parser/src/structs/array.rs rename to src/structs/array.rs diff --git a/parser/src/structs/bi_map.rs b/src/structs/bi_map.rs similarity index 98% rename from parser/src/structs/bi_map.rs rename to src/structs/bi_map.rs index dd271ec14..87030f5b9 100644 --- a/parser/src/structs/bi_map.rs +++ b/src/structs/bi_map.rs @@ -8,7 +8,7 @@ use allocative::Allocative; use crate::utils::{LossyFrom, TARGET_BLOCKS_PER_DAY}; use super::{ - AnyDateMap, AnyHeightMap, AnyMap, Date, DateMap, Height, HeightMap, MapKind, MapValue, + AnyDateMap, AnyHeightMap, AnyMap, Date, DateMap, Height, HeightMap, MapKind, MapPath, MapValue, }; #[derive(Allocative)] @@ -24,14 +24,14 @@ impl BiMap where Value: MapValue, { - pub fn new_bin(version: u32, kind: MapKind, path: &str) -> Self { + pub fn new_bin(version: u32, kind: MapKind, path: &MapPath) -> Self { Self { height: HeightMap::_new_bin(version, kind, path, true), date: DateMap::_new_bin(version, kind, path, false), } } - pub fn new_json(version: u32, kind: MapKind, path: &str) -> Self { + pub fn new_json(version: u32, kind: MapKind, path: &MapPath) -> Self { Self { height: HeightMap::new_json(version, kind, path, true), date: DateMap::new_json(version, kind, path, false), diff --git a/parser/src/structs/block_data.rs b/src/structs/block_data.rs similarity index 100% rename from parser/src/structs/block_data.rs rename to src/structs/block_data.rs diff --git a/parser/src/structs/block_path.rs b/src/structs/block_path.rs similarity index 100% rename from parser/src/structs/block_path.rs rename to src/structs/block_path.rs diff --git a/parser/src/structs/config.rs b/src/structs/config.rs similarity index 51% rename from parser/src/structs/config.rs rename to src/structs/config.rs index db7287c24..9ad22a1c0 100644 --- a/parser/src/structs/config.rs +++ b/src/structs/config.rs @@ -6,16 +6,21 @@ use std::{ use biter::bitcoincore_rpc::Auth; use clap::Parser; use color_eyre::eyre::eyre; +use log::info; use serde::{Deserialize, Serialize}; -use crate::log; +use super::MapPath; #[derive(Parser, Debug, Clone, Default, Serialize, Deserialize, PartialEq)] #[command(version, about, long_about = None)] pub struct Config { /// Bitcoin data directory path, saved #[arg(long, value_name = "DIR")] - pub datadir: Option, + bitcoindir: Option, + + /// Kibo data directory path, saved + #[arg(long, value_name = "DIR")] + kibodir: Option, /// Bitcoin RPC ip, default: localhost, saved #[arg(long, value_name = "IP")] @@ -25,7 +30,7 @@ pub struct Config { #[arg(long, value_name = "PORT")] pub rpcport: Option, - /// Bitcoin RPC cookie file, saved + /// Bitcoin RPC cookie file, default: --bitcoindir/.cookie, saved #[arg(long, value_name = "PATH")] pub rpccookiefile: Option, @@ -44,7 +49,15 @@ pub struct Config { // Maximum ram you want the program to use in GB, default: 50% of total, not saved // #[arg(long, value_name = "GB")] // pub max_ram: Option, - /// Start a dry run, default: false, not saved + /// Enable or disable the parser, default: true, not saved + #[arg(long, value_name = "BOOL")] + parser: Option, + + /// Enable or disable the server, default: true, not saved + #[arg(long, value_name = "BOOL")] + server: Option, + + /// Start a dry run, default: true, not saved #[arg(long, value_name = "BOOL")] dry_run: Option, @@ -55,25 +68,25 @@ pub struct Config { /// Recompute all computed datasets, default: false, not saved #[arg(long, value_name = "BOOL")] recompute_computed: Option, - - /// Start the program by defragmenting all databases to reduce their footprint, default: false, not saved - #[arg(long, value_name = "BOOL")] - first_defragment: Option, } impl Config { - const PATH: &'static str = "./config.toml"; - pub fn import() -> color_eyre::Result { - let mut config_saved = fs::read_to_string(Self::PATH) - .map_or(Config::default(), |contents| { - toml::from_str(&contents).unwrap_or_default() - }); + let path = Self::path_dot_kibo(); + let _ = fs::create_dir_all(&path); + + let path = path.join("config.toml"); + + let mut config_saved = Self::read(&path); let mut config_args = Config::parse(); - if let Some(datadir) = config_args.datadir.take() { - config_saved.datadir = Some(datadir); + if let Some(bitcoindir) = config_args.bitcoindir.take() { + config_saved.bitcoindir = Some(bitcoindir); + } + + if let Some(kibodir) = config_args.kibodir.take() { + config_saved.kibodir = Some(kibodir); } if let Some(rpcconnect) = config_args.rpcconnect.take() { @@ -110,31 +123,30 @@ impl Config { config.check(); - config.write()?; + config.write(&path)?; + config.parser = config_args.parser.take(); + config.server = config_args.server.take(); config.dry_run = config_args.dry_run.take(); config.record_ram_usage = config_args.record_ram_usage.take(); config.recompute_computed = config_args.recompute_computed.take(); - config.first_defragment = config_args.first_defragment.take(); - log("---"); - log("Configuration:"); - log(&format!("datadir: {:?}", config.datadir)); - log(&format!("rpcconnect: {:?}", config.rpcconnect)); - log(&format!("rpcport: {:?}", config.rpcport)); - log(&format!("rpccookiefile: {:?}", config.rpccookiefile)); - log(&format!("rpcuser: {:?}", config.rpcuser)); - log(&format!("rpcpassword: {:?}", config.rpcpassword)); - log(&format!("delay: {:?}", config.delay)); - // log(&format!("max_ram: {:?}", config.max_ram)); - log(&format!("dry_run: {:?}", config.dry_run)); - log(&format!("record_ram_usage: {:?}", config.record_ram_usage)); - log(&format!( - "recompute_computed: {:?}", - config.recompute_computed - )); - log(&format!("first_defragment: {:?}", config.first_defragment)); - log("---"); + info!("Configuration {{"); + info!(" bitcoindir: {:?}", config.bitcoindir); + info!(" kibodir: {:?}", config.kibodir); + info!(" rpcconnect: {:?}", config.rpcconnect); + info!(" rpcport: {:?}", config.rpcport); + info!(" rpccookiefile: {:?}", config.rpccookiefile); + info!(" rpcuser: {:?}", config.rpcuser); + info!(" rpcpassword: {:?}", config.rpcpassword); + info!(" delay: {:?}", config.delay); + // info!(" max_ram: {:?}", config.max_ram); + info!(" parser: {:?}", config.parser()); + info!(" server: {:?}", config.server()); + info!(" dry_run: {:?}", config.dry_run()); + info!(" record_ram_usage: {:?}", config.record_ram_usage()); + info!(" recompute_computed: {:?}", config.recompute_computed()); + info!("}}"); if config_args != Config::default() { dbg!(config_args); @@ -145,14 +157,31 @@ impl Config { } fn check(&self) { - if self.datadir.is_none() { + if self.bitcoindir.is_none() { println!( - "You need to set the --datadir parameter at least once to run the parser.\nRun the program with '-h' for help." + "You need to set the --bitcoindir parameter at least once to run the parser.\nRun the program with '-h' for help." + ); + std::process::exit(1); + } else if !self.path_bitcoindir().is_dir() { + println!( + "Given --bitcoindir parameter doesn't seem to be a valid directory path.\nRun the program with '-h' for help." ); std::process::exit(1); } - let path = Path::new(self.datadir.as_ref().unwrap()); + if self.kibodir.is_none() { + println!( + "You need to set the --kibodir parameter at least once to run the parser.\nRun the program with '-h' for help." + ); + std::process::exit(1); + } else if !self.path_kibodir().is_dir() { + println!( + "Given --kibodir parameter doesn't seem to be a valid directory path.\nRun the program with '-h' for help." + ); + std::process::exit(1); + } + + let path = Path::new(self.bitcoindir.as_ref().unwrap()); if !path.is_dir() { println!("Expect path '{:#?}' to be a directory.", path); std::process::exit(1); @@ -166,12 +195,18 @@ impl Config { } } - fn write(&self) -> std::io::Result<()> { - fs::write(Self::PATH, toml::to_string(self).unwrap()) + fn read(path: &Path) -> Self { + fs::read_to_string(path).map_or(Config::default(), |contents| { + toml::from_str(&contents).unwrap_or_default() + }) + } + + fn write(&self, path: &Path) -> std::io::Result<()> { + fs::write(path, toml::to_string(self).unwrap()) } pub fn to_rpc_auth(&self) -> color_eyre::Result { - let cookie = Path::new(self.datadir.as_ref().unwrap()).join(".cookie"); + let cookie = Path::new(self.bitcoindir.as_ref().unwrap()).join(".cookie"); if cookie.is_file() { Ok(Auth::CookieFile(cookie)) @@ -205,11 +240,52 @@ impl Config { self.recompute_computed.is_some_and(|b| b) } - pub fn first_defragment(&self) -> bool { - self.first_defragment.is_some_and(|b| b) + pub fn path_bitcoindir(&self) -> PathBuf { + PathBuf::from(self.bitcoindir.as_ref().unwrap()) } - pub fn disable_defragment(&mut self) { - self.first_defragment.take(); + fn path_kibodir(&self) -> PathBuf { + PathBuf::from(self.kibodir.as_ref().unwrap()) + } + + pub fn path_datasets(&self) -> MapPath { + MapPath::from(self.path_kibodir().join("datasets")) + } + + pub fn path_datasets_last_values(&self) -> MapPath { + self.path_datasets().join("last.json") + } + + pub fn path_price(&self) -> MapPath { + MapPath::from(self.path_kibodir().join("price")) + } + + pub fn path_databases(&self) -> PathBuf { + self.path_kibodir().join("databases") + } + + pub fn path_states(&self) -> PathBuf { + self.path_kibodir().join("states") + } + + pub fn path_inputs(&self) -> PathBuf { + self.path_kibodir().join("inputs") + } + + fn path_dot_kibo() -> PathBuf { + let home = std::env::var("HOME").unwrap(); + Path::new(&home).join(".kibo") + } + + pub fn path_log() -> PathBuf { + Self::path_dot_kibo().join("log") + } + + pub fn parser(&self) -> bool { + self.parser.is_none_or(|b| b) + } + + pub fn server(&self) -> bool { + self.server.is_none_or(|b| b) } } diff --git a/parser/src/structs/counter.rs b/src/structs/counter.rs similarity index 100% rename from parser/src/structs/counter.rs rename to src/structs/counter.rs diff --git a/parser/src/structs/date.rs b/src/structs/date.rs similarity index 92% rename from parser/src/structs/date.rs rename to src/structs/date.rs index ec36f4c52..c2d51dd51 100644 --- a/parser/src/structs/date.rs +++ b/src/structs/date.rs @@ -44,10 +44,6 @@ impl Date { Self(chrono::offset::Utc::now().date_naive()) } - pub fn yesterday() -> Self { - Self(Self::today().checked_sub_days(Days::new(1)).unwrap()) - } - pub fn difference_in_days_between(&self, older: Self) -> u32 { (**self - *older).num_days() as u32 } @@ -74,17 +70,13 @@ impl Date { } } + pub fn is_january(&self) -> bool { + self.month() == 1 + } + pub fn is_first_of_month(&self) -> bool { self.day() == 1 } - - pub fn is_first_of_january(&self) -> bool { - self.is_first_of_month() && self.month() == 1 - } - - pub fn is_first_of_june(&self) -> bool { - self.is_first_of_month() && self.month() == 6 - } } impl MapKey for Date { diff --git a/parser/src/structs/date_data.rs b/src/structs/date_data.rs similarity index 100% rename from parser/src/structs/date_data.rs rename to src/structs/date_data.rs diff --git a/parser/src/structs/date_map.rs b/src/structs/date_map.rs similarity index 94% rename from parser/src/structs/date_map.rs rename to src/structs/date_map.rs index 76e3c6bde..37e853efe 100644 --- a/parser/src/structs/date_map.rs +++ b/src/structs/date_map.rs @@ -1,8 +1,8 @@ use std::iter::Sum; -use crate::{Date, HeightMap}; - -use super::{AnyMap, DateMapChunkId, GenericMap, Height, MapValue, SerializedBTreeMap}; +use super::{ + AnyMap, Date, DateMapChunkId, GenericMap, Height, HeightMap, MapValue, SerializedBTreeMap, +}; pub type DateMap = GenericMap>; diff --git a/parser/src/structs/date_map_chunk_id.rs b/src/structs/date_map_chunk_id.rs similarity index 95% rename from parser/src/structs/date_map_chunk_id.rs rename to src/structs/date_map_chunk_id.rs index ca3fdac32..dbc8eb4fa 100644 --- a/parser/src/structs/date_map_chunk_id.rs +++ b/src/structs/date_map_chunk_id.rs @@ -3,9 +3,7 @@ use std::path::Path; use allocative::Allocative; use chrono::Datelike; -use crate::Date; - -use super::MapChunkId; +use super::{Date, MapChunkId}; #[derive(Debug, Default, PartialEq, Eq, PartialOrd, Ord, Clone, Copy, Allocative)] pub struct DateMapChunkId(i32); diff --git a/parser/src/structs/empty_address_data.rs b/src/structs/empty_address_data.rs similarity index 100% rename from parser/src/structs/empty_address_data.rs rename to src/structs/empty_address_data.rs diff --git a/parser/src/structs/epoch.rs b/src/structs/epoch.rs similarity index 100% rename from parser/src/structs/epoch.rs rename to src/structs/epoch.rs diff --git a/parser/src/structs/exit.rs b/src/structs/exit.rs similarity index 92% rename from parser/src/structs/exit.rs rename to src/structs/exit.rs index 5122ef361..00408a42e 100644 --- a/parser/src/structs/exit.rs +++ b/src/structs/exit.rs @@ -8,7 +8,7 @@ use std::{ time::Duration, }; -use super::super::log; +use log::info; #[derive(Default, Clone)] pub struct Exit { @@ -29,12 +29,12 @@ impl Exit { let blocked = move || _blocked.load(Ordering::SeqCst); ctrlc::set_handler(move || { - log("Exitting..."); + info!("Exitting..."); active.store(true, Ordering::SeqCst); if blocked() { - log("Waiting to exit safely"); + info!("Waiting to exit safely"); while blocked() { sleep(Duration::from_millis(50)); diff --git a/parser/src/structs/generic_map.rs b/src/structs/generic_map.rs similarity index 95% rename from parser/src/structs/generic_map.rs rename to src/structs/generic_map.rs index a384f6619..8c1e52244 100644 --- a/parser/src/structs/generic_map.rs +++ b/src/structs/generic_map.rs @@ -15,12 +15,11 @@ use ordered_float::OrderedFloat; use serde::{de::DeserializeOwned, Serialize}; use crate::{ - log, + io::Serialization, utils::{get_percentile, LossyFrom}, - Serialization, }; -use super::{AnyMap, MapValue}; +use super::{AnyMap, MapPath, MapValue}; #[derive(Debug, Clone, Copy, Allocative, PartialEq, Eq)] pub enum MapKind { @@ -79,8 +78,8 @@ pub struct GenericMap { version: u32, kind: MapKind, - path_all: PathBuf, - path_last: Option, + path_all: MapPath, + path_last: Option, chunks_in_memory: usize, @@ -100,15 +99,15 @@ where Key: MapKey, Serialized: MapSerialized, { - pub fn new_bin(version: u32, kind: MapKind, path: &str) -> Self { + pub fn new_bin(version: u32, kind: MapKind, path: &MapPath) -> Self { Self::new(version, kind, path, Serialization::Binary, 1, true) } - pub fn _new_bin(version: u32, kind: MapKind, path: &str, export_last: bool) -> Self { + pub fn _new_bin(version: u32, kind: MapKind, path: &MapPath, export_last: bool) -> Self { Self::new(version, kind, path, Serialization::Binary, 1, export_last) } - pub fn new_json(version: u32, kind: MapKind, path: &str, export_last: bool) -> Self { + pub fn new_json(version: u32, kind: MapKind, path: &MapPath, export_last: bool) -> Self { Self::new( version, kind, @@ -122,7 +121,7 @@ where fn new( version: u32, kind: MapKind, - path: &str, + path: &MapPath, serialization: Serialization, chunks_in_memory: usize, export_last: bool, @@ -131,15 +130,16 @@ where panic!("Should always have at least the latest chunk in memory"); } - let path = path.replace(['-', '_', ' '], "/"); + let path_all = path.join(Key::map_name()); - let path_all = PathBuf::from(format!("{path}/{}", Key::map_name())); - - fs::create_dir_all(&path_all).unwrap(); + fs::create_dir_all(&*path_all).unwrap_or_else(|_| { + dbg!(&path_all); + panic!() + }); let path_last = { if export_last { - Some(PathBuf::from(format!("{path}/last"))) + Some(path.join("last")) } else { None } @@ -187,9 +187,9 @@ where .initial_last_key .and_then(|last_key| last_key.to_first_unsafe()); - if s.initial_first_unsafe_key.is_none() { - log(&format!("Missing dataset: {path}/{}", Key::map_name())); - } + // if s.initial_first_unsafe_key.is_none() { + // log(&format!("Missing dataset: {path:?}/{}", Key::map_name())); + // } s } @@ -312,7 +312,7 @@ where &self.path_all } - fn path_last(&self) -> &Option { + fn path_last(&self) -> &Option { &self.path_last } @@ -367,15 +367,15 @@ where panic!(); }); - let path = self.path_all.join(chunk_id.to_name()); + let path = self.path_all.join(&chunk_id.to_name()); - self.serialization.export(Path::new(&path), serialized)?; + self.serialization.export(&path, serialized)?; // Export last if index == len - 1 { if let Some(path_last) = self.path_last.as_ref() { self.serialization - .export(Path::new(path_last), serialized.last().unwrap())?; + .export(path_last, serialized.last().unwrap())?; } } @@ -418,22 +418,22 @@ where Key: MapKey, Serialized: MapSerialized, { - pub fn sum_keys(&mut self, keys: &[Key]) -> Value - where - Value: Sum, - { - keys.iter() - .map(|key| self.get_or_import(key).unwrap()) - .sum::() - } + // pub fn sum_keys(&mut self, keys: &[Key]) -> Value + // where + // Value: Sum, + // { + // keys.iter() + // .map(|key| self.get_or_import(key).unwrap()) + // .sum::() + // } - pub fn average_keys(&mut self, keys: &[Key]) -> f32 - where - Value: Sum, - f32: LossyFrom, - { - f32::lossy_from(self.sum_keys(keys)) / keys.len() as f32 - } + // pub fn average_keys(&mut self, keys: &[Key]) -> f32 + // where + // Value: Sum, + // f32: LossyFrom, + // { + // f32::lossy_from(self.sum_keys(keys)) / keys.len() as f32 + // } pub fn multi_insert(&mut self, keys: &[Key], mut callback: F) where diff --git a/parser/src/structs/height.rs b/src/structs/height.rs similarity index 91% rename from parser/src/structs/height.rs rename to src/structs/height.rs index b3d811eff..17a228275 100644 --- a/parser/src/structs/height.rs +++ b/src/structs/height.rs @@ -9,9 +9,7 @@ use biter::NUMBER_OF_UNSAFE_BLOCKS; use derive_deref::{Deref, DerefMut}; use serde::{Deserialize, Serialize}; -use crate::HEIGHT_MAP_CHUNK_SIZE; - -use super::{HeightMapChunkId, MapKey}; +use super::{HeightMapChunkId, MapKey, HEIGHT_MAP_CHUNK_SIZE}; #[derive( Debug, @@ -47,11 +45,11 @@ impl Height { **self < (block_count - NUMBER_OF_UNSAFE_BLOCKS) as u32 } - pub fn iter_range_inclusive(first: Height, last: Height) -> impl Iterator { - let range = (*first)..=(*last); + // pub fn iter_range_inclusive(first: Height, last: Height) -> impl Iterator { + // let range = (*first)..=(*last); - range.into_iter().map(Height::new) - } + // range.into_iter().map(Height::new) + // } } impl PartialEq for Height { diff --git a/parser/src/structs/height_map.rs b/src/structs/height_map.rs similarity index 97% rename from parser/src/structs/height_map.rs rename to src/structs/height_map.rs index bd2f36a73..f768f96b0 100644 --- a/parser/src/structs/height_map.rs +++ b/src/structs/height_map.rs @@ -2,9 +2,7 @@ use std::{iter::Sum, ops::RangeInclusive}; use itertools::Itertools; -use crate::SerializedVec; - -use super::{AnyMap, GenericMap, Height, HeightMapChunkId, MapValue}; +use super::{AnyMap, GenericMap, Height, HeightMapChunkId, MapValue, SerializedVec}; pub const HEIGHT_MAP_CHUNK_SIZE: u32 = 10_000; diff --git a/parser/src/structs/height_map_chunk_id.rs b/src/structs/height_map_chunk_id.rs similarity index 94% rename from parser/src/structs/height_map_chunk_id.rs rename to src/structs/height_map_chunk_id.rs index 6f662529d..06d3856ce 100644 --- a/parser/src/structs/height_map_chunk_id.rs +++ b/src/structs/height_map_chunk_id.rs @@ -3,9 +3,7 @@ use std::path::Path; use allocative::Allocative; use derive_deref::{Deref, DerefMut}; -use crate::HEIGHT_MAP_CHUNK_SIZE; - -use super::{Height, MapChunkId}; +use super::{Height, MapChunkId, HEIGHT_MAP_CHUNK_SIZE}; #[derive( Debug, Default, PartialEq, Eq, PartialOrd, Ord, Clone, Copy, Allocative, Deref, DerefMut, diff --git a/parser/src/structs/liquidity.rs b/src/structs/liquidity.rs similarity index 100% rename from parser/src/structs/liquidity.rs rename to src/structs/liquidity.rs diff --git a/src/structs/map_path.rs b/src/structs/map_path.rs new file mode 100644 index 000000000..ebf056374 --- /dev/null +++ b/src/structs/map_path.rs @@ -0,0 +1,24 @@ +use std::path::PathBuf; + +use allocative::Allocative; +use derive_deref::{Deref, DerefMut}; + +#[derive(Debug, Clone, Deref, DerefMut, Allocative)] +pub struct MapPath(PathBuf); + +impl MapPath { + pub fn join(&self, path: &str) -> Self { + let path = path.replace(['-', '_', ' '], "/"); + Self(self.0.join(path)) + } + + pub fn unwrap(&self) -> &PathBuf { + &self.0 + } +} + +impl From for MapPath { + fn from(value: PathBuf) -> Self { + Self(value) + } +} diff --git a/parser/src/structs/map_value.rs b/src/structs/map_value.rs similarity index 100% rename from parser/src/structs/map_value.rs rename to src/structs/map_value.rs diff --git a/parser/src/structs/mod.rs b/src/structs/mod.rs similarity index 97% rename from parser/src/structs/mod.rs rename to src/structs/mod.rs index 1278f5844..f1974120b 100644 --- a/parser/src/structs/mod.rs +++ b/src/structs/mod.rs @@ -25,6 +25,7 @@ mod height; mod height_map; mod height_map_chunk_id; mod liquidity; +mod map_path; mod map_value; mod ohlc; mod partial_txout_data; @@ -63,6 +64,7 @@ pub use height::*; pub use height_map::*; pub use height_map_chunk_id::*; pub use liquidity::*; +pub use map_path::*; pub use map_value::*; pub use ohlc::*; pub use partial_txout_data::*; diff --git a/parser/src/structs/ohlc.rs b/src/structs/ohlc.rs similarity index 100% rename from parser/src/structs/ohlc.rs rename to src/structs/ohlc.rs diff --git a/parser/src/structs/partial_txout_data.rs b/src/structs/partial_txout_data.rs similarity index 100% rename from parser/src/structs/partial_txout_data.rs rename to src/structs/partial_txout_data.rs diff --git a/parser/src/structs/price.rs b/src/structs/price.rs similarity index 100% rename from parser/src/structs/price.rs rename to src/structs/price.rs diff --git a/parser/src/structs/ram.rs b/src/structs/ram.rs similarity index 100% rename from parser/src/structs/ram.rs rename to src/structs/ram.rs diff --git a/parser/src/structs/sent_data.rs b/src/structs/sent_data.rs similarity index 100% rename from parser/src/structs/sent_data.rs rename to src/structs/sent_data.rs diff --git a/parser/src/structs/serialized_btreemap.rs b/src/structs/serialized_btreemap.rs similarity index 98% rename from parser/src/structs/serialized_btreemap.rs rename to src/structs/serialized_btreemap.rs index 2f1723ff1..97ec47ad5 100644 --- a/parser/src/structs/serialized_btreemap.rs +++ b/src/structs/serialized_btreemap.rs @@ -4,7 +4,7 @@ use allocative::Allocative; use bincode::{Decode, Encode}; use serde::{de::DeserializeOwned, Deserialize, Serialize}; -use crate::Serialization; +use crate::io::Serialization; use super::{DateMap, MapChunkId, MapKey, MapSerialized, MapValue}; diff --git a/parser/src/structs/serialized_vec.rs b/src/structs/serialized_vec.rs similarity index 98% rename from parser/src/structs/serialized_vec.rs rename to src/structs/serialized_vec.rs index 3d61a9738..aa02f51d3 100644 --- a/parser/src/structs/serialized_vec.rs +++ b/src/structs/serialized_vec.rs @@ -4,7 +4,7 @@ use allocative::Allocative; use bincode::{Decode, Encode}; use serde::{de::DeserializeOwned, Deserialize, Serialize}; -use crate::Serialization; +use crate::io::Serialization; use super::{HeightMap, MapChunkId, MapKey, MapSerialized, MapValue}; diff --git a/parser/src/structs/timestamp.rs b/src/structs/timestamp.rs similarity index 91% rename from parser/src/structs/timestamp.rs rename to src/structs/timestamp.rs index 04ce1b1ad..6e8455a23 100644 --- a/parser/src/structs/timestamp.rs +++ b/src/structs/timestamp.rs @@ -2,7 +2,7 @@ use std::ops::Sub; use allocative::Allocative; use bincode::{Decode, Encode}; -use chrono::{Datelike, NaiveDateTime, NaiveTime, TimeZone, Timelike, Utc}; +use chrono::{NaiveDateTime, NaiveTime, TimeZone, Timelike, Utc}; use derive_deref::{Deref, DerefMut}; use serde::{Deserialize, Serialize}; @@ -48,10 +48,6 @@ impl Timestamp { ) } - pub fn to_year(self) -> u32 { - self.to_date().year() as u32 - } - pub fn to_floored_seconds(self) -> Self { let date_time = Utc.timestamp_opt(i64::from(self.0), 0).unwrap(); diff --git a/parser/src/structs/tx_data.rs b/src/structs/tx_data.rs similarity index 100% rename from parser/src/structs/tx_data.rs rename to src/structs/tx_data.rs diff --git a/parser/src/structs/txout_index.rs b/src/structs/txout_index.rs similarity index 100% rename from parser/src/structs/txout_index.rs rename to src/structs/txout_index.rs diff --git a/parser/src/utils/consts.rs b/src/utils/consts.rs similarity index 100% rename from parser/src/utils/consts.rs rename to src/utils/consts.rs diff --git a/parser/src/utils/flamegraph.rs b/src/utils/flamegraph.rs similarity index 84% rename from parser/src/utils/flamegraph.rs rename to src/utils/flamegraph.rs index 2a40ab398..2e96ce6e3 100644 --- a/parser/src/utils/flamegraph.rs +++ b/src/utils/flamegraph.rs @@ -2,10 +2,15 @@ use std::{fs, path::PathBuf}; use chrono::Local; -use crate::{databases::Databases, datasets::AllDatasets, states::States, structs::Height}; +use crate::{ + parser::{Databases, Datasets, States}, + structs::Height, +}; + +// use crate::{databases::Databases, datasets::AllDatasets, states::States, structs::Height}; pub fn generate_allocation_files( - datasets: &AllDatasets, + datasets: &Datasets, databases: &Databases, states: &States, last_height: Height, diff --git a/src/utils/log.rs b/src/utils/log.rs new file mode 100644 index 000000000..4679d3339 --- /dev/null +++ b/src/utils/log.rs @@ -0,0 +1,82 @@ +use std::{ + fmt::Display, + fs::{self, OpenOptions}, + io::Write, +}; + +use chrono::Local; +use color_eyre::owo_colors::OwoColorize; +use env_logger::Env; + +use crate::structs::Config; + +#[inline(always)] +pub fn init_log() { + let _ = fs::remove_file(Config::path_log()); + + let file = Box::new( + OpenOptions::new() + .create(true) + .append(true) + .open(Config::path_log()) + .unwrap(), + ); + + env_logger::Builder::from_env( + Env::default().default_filter_or(format!("{}=info", env!("CARGO_PKG_NAME"))), + ) + .format(move |buf, record| { + let date_time = format!("{}", Local::now().format("%Y-%m-%d %H:%M:%S")); + let level = record.level().as_str().to_lowercase(); + let level = format!("{:5}", level); + let target = record.target(); + let dash = "-"; + let args = record.args(); + + let _ = write( + file.try_clone().unwrap(), + &date_time, + target, + &level, + dash, + args, + ); + + let colored_date_time = date_time.bright_black(); + let colored_level = match level.chars().next().unwrap() { + 'e' => level.red().to_string(), + 'w' => level.yellow().to_string(), + 'i' => level.green().to_string(), + 'd' => level.blue().to_string(), + 't' => level.cyan().to_string(), + _ => panic!(), + }; + let colored_dash = dash.bright_black(); + + write( + buf, + colored_date_time, + target, + colored_level, + colored_dash, + args, + ) + }) + .init(); +} + +fn write( + mut buf: impl Write, + date_time: impl Display, + _target: impl Display, + level: impl Display, + dash: impl Display, + args: impl Display, +) -> Result<(), std::io::Error> { + writeln!(buf, "{} {} {} {}", date_time, dash, level, args) + // writeln!( + // buf, + // "{} {} {} {} {}", + // date_time, _target, level, dash, args + // ) +} diff --git a/parser/src/utils/lossy.rs b/src/utils/lossy.rs similarity index 100% rename from parser/src/utils/lossy.rs rename to src/utils/lossy.rs diff --git a/parser/src/utils/mod.rs b/src/utils/mod.rs similarity index 93% rename from parser/src/utils/mod.rs rename to src/utils/mod.rs index 18c6f8d80..0ed6755e9 100644 --- a/parser/src/utils/mod.rs +++ b/src/utils/mod.rs @@ -13,5 +13,4 @@ pub use log::*; pub use lossy::*; pub use percentile::*; pub use retry::*; -pub use rpc::*; pub use time::*; diff --git a/parser/src/utils/percentile.rs b/src/utils/percentile.rs similarity index 100% rename from parser/src/utils/percentile.rs rename to src/utils/percentile.rs diff --git a/parser/src/utils/retry.rs b/src/utils/retry.rs similarity index 100% rename from parser/src/utils/retry.rs rename to src/utils/retry.rs diff --git a/parser/src/utils/rpc.rs b/src/utils/rpc.rs similarity index 61% rename from parser/src/utils/rpc.rs rename to src/utils/rpc.rs index 4e65bd8aa..a25275d27 100644 --- a/parser/src/utils/rpc.rs +++ b/src/utils/rpc.rs @@ -1,8 +1,14 @@ use biter::bitcoincore_rpc::Client; -use crate::Config; +use crate::structs::Config; -pub fn create_rpc(config: &Config) -> color_eyre::Result { +impl From<&Config> for Client { + fn from(config: &Config) -> Self { + create_rpc(config).unwrap() + } +} + +fn create_rpc(config: &Config) -> color_eyre::Result { Ok(Client::new( &format!( "http://{}:{}", diff --git a/parser/src/utils/time.rs b/src/utils/time.rs similarity index 66% rename from parser/src/utils/time.rs rename to src/utils/time.rs index ec5c090f1..f0001216d 100644 --- a/parser/src/utils/time.rs +++ b/src/utils/time.rs @@ -1,6 +1,6 @@ use std::time::Instant; -use crate::utils::log; +use log::info; pub fn time(name: &str, function: F) -> T where @@ -10,7 +10,7 @@ where let returned = function(); - log(&format!("{name}: {} seconds", time.elapsed().as_secs_f32())); + info!("{name}: {} seconds", time.elapsed().as_secs_f32()); returned } diff --git a/website/assets/fonts/satoshi/2024-09/font.var.woff2 b/src/website/assets/fonts/satoshi/2024-09/font.var.woff2 similarity index 100% rename from website/assets/fonts/satoshi/2024-09/font.var.woff2 rename to src/website/assets/fonts/satoshi/2024-09/font.var.woff2 diff --git a/website/assets/fonts/satoshi/FFL.txt b/src/website/assets/fonts/satoshi/FFL.txt similarity index 100% rename from website/assets/fonts/satoshi/FFL.txt rename to src/website/assets/fonts/satoshi/FFL.txt diff --git a/website/assets/pdfs/block/2022-report.pdf b/src/website/assets/pdfs/block/2022-report.pdf similarity index 100% rename from website/assets/pdfs/block/2022-report.pdf rename to src/website/assets/pdfs/block/2022-report.pdf diff --git a/website/assets/pdfs/braiins/building-bitcoin-in-rust.pdf b/src/website/assets/pdfs/braiins/building-bitcoin-in-rust.pdf similarity index 100% rename from website/assets/pdfs/braiins/building-bitcoin-in-rust.pdf rename to src/website/assets/pdfs/braiins/building-bitcoin-in-rust.pdf diff --git a/website/assets/pdfs/glassnode/cointime-economics.pdf b/src/website/assets/pdfs/glassnode/cointime-economics.pdf similarity index 100% rename from website/assets/pdfs/glassnode/cointime-economics.pdf rename to src/website/assets/pdfs/glassnode/cointime-economics.pdf diff --git a/website/assets/pdfs/multi-author/bcap_v1.0.pdf b/src/website/assets/pdfs/multi-author/bcap_v1.0.pdf similarity index 100% rename from website/assets/pdfs/multi-author/bcap_v1.0.pdf rename to src/website/assets/pdfs/multi-author/bcap_v1.0.pdf diff --git a/website/assets/pdfs/nakamoto-project/understanding-bitcoin-adoption-in-the-united-states.pdf b/src/website/assets/pdfs/nakamoto-project/understanding-bitcoin-adoption-in-the-united-states.pdf similarity index 100% rename from website/assets/pdfs/nakamoto-project/understanding-bitcoin-adoption-in-the-united-states.pdf rename to src/website/assets/pdfs/nakamoto-project/understanding-bitcoin-adoption-in-the-united-states.pdf diff --git a/website/assets/pdfs/nydig/protection-under-first-amendment.pdf b/src/website/assets/pdfs/nydig/protection-under-first-amendment.pdf similarity index 100% rename from website/assets/pdfs/nydig/protection-under-first-amendment.pdf rename to src/website/assets/pdfs/nydig/protection-under-first-amendment.pdf diff --git a/website/assets/pdfs/satoshi-nakamoto/whitepaper.pdf b/src/website/assets/pdfs/satoshi-nakamoto/whitepaper.pdf similarity index 100% rename from website/assets/pdfs/satoshi-nakamoto/whitepaper.pdf rename to src/website/assets/pdfs/satoshi-nakamoto/whitepaper.pdf diff --git a/website/assets/pdfs/square/2021-bitcoin-clean-energy-initiative.pdf b/src/website/assets/pdfs/square/2021-bitcoin-clean-energy-initiative.pdf similarity index 100% rename from website/assets/pdfs/square/2021-bitcoin-clean-energy-initiative.pdf rename to src/website/assets/pdfs/square/2021-bitcoin-clean-energy-initiative.pdf diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-icon-180.png b/src/website/assets/pwa/2024-11-20_09-41-25/apple-icon-180.png similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-icon-180.png rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-icon-180.png diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1125-2436.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1125-2436.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1125-2436.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1125-2436.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1136-640.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1136-640.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1136-640.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1136-640.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1170-2532.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1170-2532.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1170-2532.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1170-2532.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1179-2556.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1179-2556.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1179-2556.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1179-2556.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1242-2208.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1242-2208.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1242-2208.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1242-2208.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1242-2688.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1242-2688.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1242-2688.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1242-2688.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1284-2778.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1284-2778.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1284-2778.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1284-2778.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1290-2796.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1290-2796.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1290-2796.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1290-2796.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1334-750.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1334-750.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1334-750.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1334-750.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1488-2266.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1488-2266.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1488-2266.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1488-2266.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1536-2048.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1536-2048.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1536-2048.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1536-2048.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1620-2160.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1620-2160.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1620-2160.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1620-2160.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1640-2360.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1640-2360.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1640-2360.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1640-2360.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1668-2224.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1668-2224.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1668-2224.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1668-2224.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1668-2388.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1668-2388.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1668-2388.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1668-2388.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1792-828.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1792-828.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-1792-828.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-1792-828.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2048-1536.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2048-1536.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2048-1536.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2048-1536.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2048-2732.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2048-2732.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2048-2732.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2048-2732.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2160-1620.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2160-1620.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2160-1620.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2160-1620.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2208-1242.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2208-1242.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2208-1242.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2208-1242.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2224-1668.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2224-1668.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2224-1668.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2224-1668.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2266-1488.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2266-1488.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2266-1488.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2266-1488.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2360-1640.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2360-1640.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2360-1640.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2360-1640.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2388-1668.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2388-1668.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2388-1668.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2388-1668.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2436-1125.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2436-1125.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2436-1125.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2436-1125.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2532-1170.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2532-1170.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2532-1170.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2532-1170.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2556-1179.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2556-1179.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2556-1179.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2556-1179.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2688-1242.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2688-1242.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2688-1242.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2688-1242.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2732-2048.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2732-2048.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2732-2048.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2732-2048.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2778-1284.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2778-1284.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2778-1284.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2778-1284.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2796-1290.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2796-1290.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-2796-1290.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-2796-1290.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-640-1136.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-640-1136.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-640-1136.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-640-1136.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-750-1334.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-750-1334.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-750-1334.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-750-1334.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/apple-splash-828-1792.jpg b/src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-828-1792.jpg similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/apple-splash-828-1792.jpg rename to src/website/assets/pwa/2024-11-20_09-41-25/apple-splash-828-1792.jpg diff --git a/website/assets/pwa/2024-11-20_09-41-25/favicon-196.png b/src/website/assets/pwa/2024-11-20_09-41-25/favicon-196.png similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/favicon-196.png rename to src/website/assets/pwa/2024-11-20_09-41-25/favicon-196.png diff --git a/website/assets/pwa/2024-11-20_09-41-25/index.html b/src/website/assets/pwa/2024-11-20_09-41-25/index.html similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/index.html rename to src/website/assets/pwa/2024-11-20_09-41-25/index.html diff --git a/website/assets/pwa/2024-11-20_09-41-25/manifest-icon-192.maskable.png b/src/website/assets/pwa/2024-11-20_09-41-25/manifest-icon-192.maskable.png similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/manifest-icon-192.maskable.png rename to src/website/assets/pwa/2024-11-20_09-41-25/manifest-icon-192.maskable.png diff --git a/website/assets/pwa/2024-11-20_09-41-25/manifest-icon-512.maskable.png b/src/website/assets/pwa/2024-11-20_09-41-25/manifest-icon-512.maskable.png similarity index 100% rename from website/assets/pwa/2024-11-20_09-41-25/manifest-icon-512.maskable.png rename to src/website/assets/pwa/2024-11-20_09-41-25/manifest-icon-512.maskable.png diff --git a/website/assets/pwa/index.html b/src/website/assets/pwa/index.html similarity index 100% rename from website/assets/pwa/index.html rename to src/website/assets/pwa/index.html diff --git a/website/generate-icons.sh b/src/website/generate-icons.sh similarity index 100% rename from website/generate-icons.sh rename to src/website/generate-icons.sh diff --git a/website/index.html b/src/website/index.html similarity index 99% rename from website/index.html rename to src/website/index.html index b649580b9..93166e466 100644 --- a/website/index.html +++ b/src/website/index.html @@ -513,6 +513,7 @@ input { border: 0; width: 100%; + text-align: left; &:focus-visible { outline: none; @@ -834,32 +835,6 @@ } } - aside { - > header { - button { - margin: -0.5rem 0; - - &:first-child { - margin-left: -0.65rem; - } - - &:last-child { - margin-right: -0.65rem; - } - - &[data-highlight] { - * { - color: var(--orange) !important; - } - } - - > svg { - margin: 0.5rem; - } - } - } - } - #share-div { padding: 1.5rem; backdrop-filter: blur(12px); diff --git a/website/jsconfig.json b/src/website/jsconfig.json similarity index 100% rename from website/jsconfig.json rename to src/website/jsconfig.json diff --git a/website/manifest.webmanifest b/src/website/manifest.webmanifest similarity index 100% rename from website/manifest.webmanifest rename to src/website/manifest.webmanifest diff --git a/website/misc/tailwindTo550.js b/src/website/misc/tailwindTo550.js similarity index 100% rename from website/misc/tailwindTo550.js rename to src/website/misc/tailwindTo550.js diff --git a/website/packages/lean-qr/README.md b/src/website/packages/lean-qr/README.md similarity index 100% rename from website/packages/lean-qr/README.md rename to src/website/packages/lean-qr/README.md diff --git a/website/packages/lean-qr/v2.3.4/script.js b/src/website/packages/lean-qr/v2.3.4/script.js similarity index 100% rename from website/packages/lean-qr/v2.3.4/script.js rename to src/website/packages/lean-qr/v2.3.4/script.js diff --git a/website/packages/lean-qr/v2.3.4/types.d.ts b/src/website/packages/lean-qr/v2.3.4/types.d.ts similarity index 100% rename from website/packages/lean-qr/v2.3.4/types.d.ts rename to src/website/packages/lean-qr/v2.3.4/types.d.ts diff --git a/website/packages/lightweight-charts/NOTICE.md b/src/website/packages/lightweight-charts/NOTICE.md similarity index 100% rename from website/packages/lightweight-charts/NOTICE.md rename to src/website/packages/lightweight-charts/NOTICE.md diff --git a/website/packages/lightweight-charts/README.md b/src/website/packages/lightweight-charts/README.md similarity index 100% rename from website/packages/lightweight-charts/README.md rename to src/website/packages/lightweight-charts/README.md diff --git a/website/packages/lightweight-charts/types.d.ts b/src/website/packages/lightweight-charts/types.d.ts similarity index 100% rename from website/packages/lightweight-charts/types.d.ts rename to src/website/packages/lightweight-charts/types.d.ts diff --git a/website/packages/lightweight-charts/v4.2.0/script.js b/src/website/packages/lightweight-charts/v4.2.0/script.js similarity index 100% rename from website/packages/lightweight-charts/v4.2.0/script.js rename to src/website/packages/lightweight-charts/v4.2.0/script.js diff --git a/website/packages/lightweight-charts/v4.2.0/types.d.ts b/src/website/packages/lightweight-charts/v4.2.0/types.d.ts similarity index 100% rename from website/packages/lightweight-charts/v4.2.0/types.d.ts rename to src/website/packages/lightweight-charts/v4.2.0/types.d.ts diff --git a/website/packages/lightweight-charts/wrapper.js b/src/website/packages/lightweight-charts/wrapper.js similarity index 100% rename from website/packages/lightweight-charts/wrapper.js rename to src/website/packages/lightweight-charts/wrapper.js diff --git a/website/packages/solid-signals/2024-11-02/script.js b/src/website/packages/solid-signals/2024-11-02/script.js similarity index 100% rename from website/packages/solid-signals/2024-11-02/script.js rename to src/website/packages/solid-signals/2024-11-02/script.js diff --git a/website/packages/solid-signals/2024-11-02/types/core/constants.d.ts b/src/website/packages/solid-signals/2024-11-02/types/core/constants.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/core/constants.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/core/constants.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/core/core.d.ts b/src/website/packages/solid-signals/2024-11-02/types/core/core.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/core/core.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/core/core.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/core/effect.d.ts b/src/website/packages/solid-signals/2024-11-02/types/core/effect.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/core/effect.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/core/effect.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/core/error.d.ts b/src/website/packages/solid-signals/2024-11-02/types/core/error.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/core/error.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/core/error.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/core/flags.d.ts b/src/website/packages/solid-signals/2024-11-02/types/core/flags.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/core/flags.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/core/flags.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/core/index.d.ts b/src/website/packages/solid-signals/2024-11-02/types/core/index.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/core/index.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/core/index.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/core/owner.d.ts b/src/website/packages/solid-signals/2024-11-02/types/core/owner.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/core/owner.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/core/owner.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/core/scheduler.d.ts b/src/website/packages/solid-signals/2024-11-02/types/core/scheduler.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/core/scheduler.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/core/scheduler.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/index.d.ts b/src/website/packages/solid-signals/2024-11-02/types/index.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/index.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/index.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/map.d.ts b/src/website/packages/solid-signals/2024-11-02/types/map.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/map.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/map.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/signals.d.ts b/src/website/packages/solid-signals/2024-11-02/types/signals.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/signals.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/signals.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/store/index.d.ts b/src/website/packages/solid-signals/2024-11-02/types/store/index.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/store/index.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/store/index.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/store/store.d.ts b/src/website/packages/solid-signals/2024-11-02/types/store/store.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/store/store.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/store/store.d.ts diff --git a/website/packages/solid-signals/2024-11-02/types/utils.d.ts b/src/website/packages/solid-signals/2024-11-02/types/utils.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-02/types/utils.d.ts rename to src/website/packages/solid-signals/2024-11-02/types/utils.d.ts diff --git a/website/packages/solid-signals/2024-11-08/script.js b/src/website/packages/solid-signals/2024-11-08/script.js similarity index 100% rename from website/packages/solid-signals/2024-11-08/script.js rename to src/website/packages/solid-signals/2024-11-08/script.js diff --git a/website/packages/solid-signals/2024-11-08/types/core/constants.d.ts b/src/website/packages/solid-signals/2024-11-08/types/core/constants.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/core/constants.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/core/constants.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/core/core.d.ts b/src/website/packages/solid-signals/2024-11-08/types/core/core.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/core/core.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/core/core.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/core/effect.d.ts b/src/website/packages/solid-signals/2024-11-08/types/core/effect.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/core/effect.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/core/effect.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/core/error.d.ts b/src/website/packages/solid-signals/2024-11-08/types/core/error.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/core/error.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/core/error.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/core/flags.d.ts b/src/website/packages/solid-signals/2024-11-08/types/core/flags.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/core/flags.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/core/flags.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/core/index.d.ts b/src/website/packages/solid-signals/2024-11-08/types/core/index.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/core/index.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/core/index.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/core/owner.d.ts b/src/website/packages/solid-signals/2024-11-08/types/core/owner.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/core/owner.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/core/owner.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/core/scheduler.d.ts b/src/website/packages/solid-signals/2024-11-08/types/core/scheduler.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/core/scheduler.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/core/scheduler.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/index.d.ts b/src/website/packages/solid-signals/2024-11-08/types/index.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/index.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/index.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/map.d.ts b/src/website/packages/solid-signals/2024-11-08/types/map.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/map.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/map.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/signals.d.ts b/src/website/packages/solid-signals/2024-11-08/types/signals.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/signals.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/signals.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/store/index.d.ts b/src/website/packages/solid-signals/2024-11-08/types/store/index.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/store/index.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/store/index.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/store/store.d.ts b/src/website/packages/solid-signals/2024-11-08/types/store/store.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/store/store.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/store/store.d.ts diff --git a/website/packages/solid-signals/2024-11-08/types/utils.d.ts b/src/website/packages/solid-signals/2024-11-08/types/utils.d.ts similarity index 100% rename from website/packages/solid-signals/2024-11-08/types/utils.d.ts rename to src/website/packages/solid-signals/2024-11-08/types/utils.d.ts diff --git a/website/packages/solid-signals/README.md b/src/website/packages/solid-signals/README.md similarity index 100% rename from website/packages/solid-signals/README.md rename to src/website/packages/solid-signals/README.md diff --git a/website/packages/solid-signals/types.d.ts b/src/website/packages/solid-signals/types.d.ts similarity index 100% rename from website/packages/solid-signals/types.d.ts rename to src/website/packages/solid-signals/types.d.ts diff --git a/website/packages/solid-signals/wrapper.js b/src/website/packages/solid-signals/wrapper.js similarity index 100% rename from website/packages/solid-signals/wrapper.js rename to src/website/packages/solid-signals/wrapper.js diff --git a/website/packages/ufuzzy/README.md b/src/website/packages/ufuzzy/README.md similarity index 100% rename from website/packages/ufuzzy/README.md rename to src/website/packages/ufuzzy/README.md diff --git a/website/packages/ufuzzy/v1.0.14/script.js b/src/website/packages/ufuzzy/v1.0.14/script.js similarity index 100% rename from website/packages/ufuzzy/v1.0.14/script.js rename to src/website/packages/ufuzzy/v1.0.14/script.js diff --git a/website/packages/ufuzzy/v1.0.14/types.d.ts b/src/website/packages/ufuzzy/v1.0.14/types.d.ts similarity index 100% rename from website/packages/ufuzzy/v1.0.14/types.d.ts rename to src/website/packages/ufuzzy/v1.0.14/types.d.ts diff --git a/website/robots.txt b/src/website/robots.txt similarity index 100% rename from website/robots.txt rename to src/website/robots.txt diff --git a/website/scripts/chart.js b/src/website/scripts/chart.js similarity index 100% rename from website/scripts/chart.js rename to src/website/scripts/chart.js diff --git a/website/scripts/live-price.js b/src/website/scripts/live-price.js similarity index 100% rename from website/scripts/live-price.js rename to src/website/scripts/live-price.js diff --git a/website/scripts/main.js b/src/website/scripts/main.js similarity index 99% rename from website/scripts/main.js rename to src/website/scripts/main.js index 17edce5b5..9e3f3d7b7 100644 --- a/website/scripts/main.js +++ b/src/website/scripts/main.js @@ -1311,10 +1311,10 @@ function initFrameSelectors() { function setAsideParent() { const { clientWidth } = window.document.documentElement; const { aside, body, main } = elements; - if (clientWidth >= consts.MEDIUM_WIDTH) { - aside.parentElement !== body && body.append(aside); - } else { - aside.parentElement !== main && main.append(aside); + if (clientWidth >= consts.MEDIUM_WIDTH && aside.parentElement !== body) { + body.append(aside); + } else if (aside.parentElement !== main) { + main.append(aside); } } diff --git a/website/scripts/moscow-time.js b/src/website/scripts/moscow-time.js similarity index 100% rename from website/scripts/moscow-time.js rename to src/website/scripts/moscow-time.js diff --git a/website/scripts/options.js b/src/website/scripts/options.js similarity index 100% rename from website/scripts/options.js rename to src/website/scripts/options.js diff --git a/website/scripts/service-worker.js b/src/website/scripts/service-worker.js similarity index 100% rename from website/scripts/service-worker.js rename to src/website/scripts/service-worker.js diff --git a/website/scripts/simulation.js b/src/website/scripts/simulation.js similarity index 100% rename from website/scripts/simulation.js rename to src/website/scripts/simulation.js diff --git a/website/scripts/types/self.d.ts b/src/website/scripts/types/self.d.ts similarity index 100% rename from website/scripts/types/self.d.ts rename to src/website/scripts/types/self.d.ts diff --git a/website/styles/chart.css b/src/website/styles/chart.css similarity index 100% rename from website/styles/chart.css rename to src/website/styles/chart.css diff --git a/website/styles/live-price.css b/src/website/styles/live-price.css similarity index 100% rename from website/styles/live-price.css rename to src/website/styles/live-price.css diff --git a/website/styles/moscow-time.css b/src/website/styles/moscow-time.css similarity index 100% rename from website/styles/moscow-time.css rename to src/website/styles/moscow-time.css diff --git a/website/styles/simulation.css b/src/website/styles/simulation.css similarity index 100% rename from website/styles/simulation.css rename to src/website/styles/simulation.css diff --git a/website/tsconfig.json b/src/website/tsconfig.json similarity index 100% rename from website/tsconfig.json rename to src/website/tsconfig.json