Cleanup

Fix RNodeMultiInterface to work with refactored interfaces

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,11 @@
+blank_issues_enabled: false
+contact_links:
+  - name: ✨ Feature Request or Idea
+    url: https://github.com/markqvist/Reticulum/discussions/new?category=ideas
+    about: Propose and discuss features and ideas
+  - name: 💬 Questions, Help & Discussion
+    about: Ask anything, or get help
+    url: https://github.com/markqvist/Reticulum/discussions/new/choose
+  - name: 📖 Read the Reticulum Manual
+    url: https://markqvist.github.io/Reticulum/manual/
+    about: The complete documentation for Reticulum

.github/ISSUE_TEMPLATE/🐛-bug-report.md

@@ -0,0 +1,39 @@
+---
+name: "\U0001F41B Bug Report"
+about: Report a reproducible bug
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Read the Contribution Guidelines**
+Before creating a bug report on this issue tracker, you **must** read the [Contribution Guidelines](https://github.com/markqvist/Reticulum/blob/master/Contributing.md). Issues that do not follow the contribution guidelines **will be deleted without comment**.
+
+- The issue tracker is used by developers of this project. **Do not use it to ask general questions, or for support requests**.
+- Ideas and feature requests can be made on the [Discussions](https://github.com/markqvist/Reticulum/discussions). **Only** feature requests accepted by maintainers and developers are tracked and included on the issue tracker. **Do not post feature requests here**.
+- After reading the [Contribution Guidelines](https://github.com/markqvist/Reticulum/blob/master/Contributing.md), delete this section from your bug report.
+
+**Describe the Bug**
+First of all: Is this really a bug? Is it reproducible?
+
+If this is a request for help because something is not working as you expected, stop right here, and go to the [discussions](https://github.com/markqvist/Reticulum/discussions) instead, where you can post your questions and get help from other users.
+
+If this really is a bug or issue with the software, remove this section of the template, and provide **a clear and concise description of what the bug is**.
+
+**To Reproduce**
+Describe in detail how to reproduce the bug.
+
+**Expected Behavior**
+A clear and concise description of what you expected to happen.
+
+**Logs & Screenshots**
+Please include any relevant log output. If applicable, also add screenshots to help explain your problem. In most cases, without any relevant log output, we will not be able to determine the cause of the bug, or reproduce it.
+
+**System Information**
+- OS and version
+- Python version
+- Program version
+
+**Additional context**
+Add any other context about the problem here.

.github/workflows/build.yml

@@ -0,0 +1,96 @@
+name: Build Reticulum
+
+on:
+  push:
+    branches: 
+      - '*'
+    tags:
+      - "[0-9]+.[0-9]+.[0-9]+*"
+  pull_request:
+    branches: 
+      - master
+    paths-ignore:
+      - .gitignore
+      - LICENSE
+
+permissions:
+  contents: write
+
+concurrency: 
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.x
+      - run: make test
+
+  package:
+    needs: test
+    if: startsWith(github.ref, 'refs/tags/')
+    runs-on: ubuntu-latest
+    environment: ${{ contains(github.ref, '-') && 'development' || 'production' }}
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.x
+      - run: |
+          python -m pip install -q build wheel setuptools
+          make remove_symlinks
+          make build_wheel
+          make build_pure_wheel
+          make create_symlinks
+      - uses: actions/upload-artifact@v4
+        with:
+          name: package
+          path: dist/*.whl
+
+  # documentation:
+  #   needs: test
+  #   if: startsWith(github.ref, 'refs/tags/')
+  #   runs-on: ubuntu-latest
+  #   environment: ${{ contains(github.ref, '-') && 'development' || 'production' }}
+  #   steps:
+  #     - uses: actions/checkout@v4
+  #     - uses: actions/setup-python@v5
+  #       with:
+  #         python-version: 3.x
+  #     - run: |
+  #         sudo apt-get -qq update && sudo apt-get -qq install latexmk texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended
+  #         python -m pip -q install sphinx sphinx-copybutton
+  #         cd docs && make latexpdf && make epub
+  #     - uses: actions/upload-artifact@v4
+  #       with:
+  #         name: documentation
+  #         path: |
+  #           docs/build/latex/*.pdf
+  #           docs/build/epub/*.epub
+
+  release:
+    needs: [package]
+    if: startsWith(github.ref, 'refs/tags/')
+    runs-on: ubuntu-latest
+    environment: ${{ contains(github.ref, '-') && 'development' || 'production' }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/download-artifact@v4
+        with:
+          path: .artifacts
+      - uses: softprops/action-gh-release@v2
+        with:
+          files: |
+#            .artifacts/package/**.whl
+#            .artifacts/documentation/latex/reticulumnetworkstack.pdf
+#            .artifacts/documentation/epub/ReticulumNetworkStack.epub
+          draft: true
+          generate_release_notes: true
+          prerelease: ${{ contains(github.ref, '-') }}
+          fail_on_unmatched_files: true

.gitignore

@@ -10,5 +10,6 @@ docs/build
 rns*.egg-info
 profile.data
 tests/rnsconfig/storage
+tests/rnsconfig/logfile*
 *.data
 *.result

Contributing.md

@@ -0,0 +1,43 @@
+# Contributing to Reticulum
+
+Welcome, and thank you for your interest in contributing to Reticulum!
+
+Apart from writing code, there are many ways in which you can contribute. Before interacting with this community, read these short and simple guidelines.
+
+## Expected Conduct
+
+First and foremost, there is one simple requirement for taking part in this community: While we primarily interact virtually, your actions matter and have real consequences. Therefore: **Act like a responsible, civilized person** - also in the face of disputes and heated disagreements. Speak your mind here, discussions are welcome. Just do so in the spirit of being face-to-face with everyone else. Thank you.
+
+## Asking Questions
+
+If you want to ask a question, **do not open an issue**. The issue tracker is used by people *working on Reticulum* to track bugs, issues and improvements.
+
+Instead, ask away on the [discussions](https://github.com/markqvist/Reticulum/discussions) or on the [Reticulum Matrix channel](https://matrix.to/#/#reticulum:matrix.org) at `#reticulum:matrix.org`
+
+## Providing Feedback & Ideas
+
+Likewise, feedback, ideas and feature requests are a very welcome way to contribute, and should also be posted on the [discussions](https://github.com/markqvist/Reticulum/discussions), or on the [Reticulum Matrix channel](https://matrix.to/#/#reticulum:matrix.org) at `#reticulum:matrix.org`.
+
+Please do not post feature requests or general ideas on the issue tracker, or in direct messages to the primary developers. You are much more likely to get a response and start a constructive discussion by posting your ideas in the public channels created for these purposes.
+
+## Reporting Issues
+
+If you have found a bug or issue in this project, please report it using the [issue tracker](https://github.com/markqvist/Reticulum/issues). If at all possible, be sure to include details on how to reproduce the bug.
+
+Anything submitted to the issue tracker that does not follow these guidelines will be closed and removed without comments or explanation.
+
+## Writing Code
+
+If you are interested in contributing code, fixing open issues or adding features, please coordinate the effort with the maintainer or one of the main developers **before** submitting a pull request. Before deciding to contribute, it is also a good idea to ensure your efforts are in alignment with the [Roadmap](./Roadmap.md) and current development focus.
+
+Pull requests have a high chance of being accepted if they are:
+
+- In alignment with the [Roadmap](./Roadmap.md) or solve an open issue or feature request
+- Sufficiently tested to work with all API functions, and pass the standard test suite
+- Functionally and conceptually complete and well-designed
+- Not simply formatting or code style changes
+- Well-documented
+
+Even new ideas and proposals that have not been approved by a maintainer, or fall outside the established roadmap, are *occasionally* accepted - if they possess the remaining of the above qualities. If not, they will be closed and removed without comments or explanation.
+
+By contributing code to this project, you agree that copyright for the code is transferred to the Reticulum maintainers and that the code is irrevocably placed under the [MIT license](./LICENSE).

Examples/Announce.py

@@ -32,7 +32,7 @@ def program_setup(configpath):
     # Destinations are endpoints in Reticulum, that can be addressed
     # and communicated with. Destinations can also announce their
     # existence, which will let the network know they are reachable
-    # and autoomatically create paths to them, from anywhere else
+    # and automatically create paths to them, from anywhere else
     # in the network.
     destination_1 = RNS.Destination(
         identity,
@@ -53,7 +53,7 @@ def program_setup(configpath):
     )
 
     # We configure the destinations to automatically prove all
-    # packets adressed to it. By doing this, RNS will automatically
+    # packets addressed to it. By doing this, RNS will automatically
     # generate a proof for each incoming packet and transmit it
     # back to the sender of that packet. This will let anyone that
     # tries to communicate with the destination know whether their
@@ -130,10 +130,11 @@ class ExampleAnnounceHandler:
             RNS.prettyhexrep(destination_hash)
         )
 
-        RNS.log(
-            "The announce contained the following app data: "+
-            app_data.decode("utf-8")
-        )
+        if app_data:
+            RNS.log(
+                "The announce contained the following app data: "+
+                app_data.decode("utf-8")
+            )
 
 ##########################################################
 #### Program Startup #####################################

Examples/Buffer.py

@@ -0,0 +1,323 @@
+##########################################################
+# This RNS example demonstrates how to set up a link to  #
+# a destination, and pass binary data over it using a    #
+# channel buffer.                                        #
+##########################################################
+from __future__ import annotations
+import os
+import sys
+import time
+import argparse
+from datetime import datetime
+
+import RNS
+from RNS.vendor import umsgpack
+
+# Let's define an app name. We'll use this for all
+# destinations we create. Since this echo example
+# is part of a range of example utilities, we'll put
+# them all within the app namespace "example_utilities"
+APP_NAME = "example_utilities"
+
+
+##########################################################
+#### Server Part #########################################
+##########################################################
+
+# A reference to the latest client link that connected
+latest_client_link = None
+
+# A reference to the latest buffer object
+latest_buffer = None
+
+# This initialisation is executed when the users chooses
+# to run as a server
+def server(configpath):
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+    
+    # Randomly create a new identity for our example
+    server_identity = RNS.Identity()
+
+    # We create a destination that clients can connect to. We
+    # want clients to create links to this destination, so we
+    # need to create a "single" destination type.
+    server_destination = RNS.Destination(
+        server_identity,
+        RNS.Destination.IN,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "bufferexample"
+    )
+
+    # We configure a function that will get called every time
+    # a new client creates a link to this destination.
+    server_destination.set_link_established_callback(client_connected)
+
+    # Everything's ready!
+    # Let's Wait for client requests or user input
+    server_loop(server_destination)
+
+def server_loop(destination):
+    # Let the user know that everything is ready
+    RNS.log(
+        "Link buffer example "+
+        RNS.prettyhexrep(destination.hash)+
+        " running, waiting for a connection."
+    )
+
+    RNS.log("Hit enter to manually send an announce (Ctrl-C to quit)")
+
+    # We enter a loop that runs until the users exits.
+    # If the user hits enter, we will announce our server
+    # destination on the network, which will let clients
+    # know how to create messages directed towards it.
+    while True:
+        entered = input()
+        destination.announce()
+        RNS.log("Sent announce from "+RNS.prettyhexrep(destination.hash))
+
+# When a client establishes a link to our server
+# destination, this function will be called with
+# a reference to the link.
+def client_connected(link):
+    global latest_client_link, latest_buffer
+    latest_client_link = link
+
+    RNS.log("Client connected")
+    link.set_link_closed_callback(client_disconnected)
+
+    # If a new connection is received, the old reader
+    # needs to be disconnected.
+    if latest_buffer:
+        latest_buffer.close()
+
+
+    # Create buffer objects.
+    #   The stream_id parameter to these functions is
+    #   a bit like a file descriptor, except that it
+    #   is unique to the *receiver*.
+    #
+    #   In this example, both the reader and the writer
+    #   use stream_id = 0, but there are actually two
+    #   separate unidirectional streams flowing in
+    #   opposite directions.
+    #
+    channel = link.get_channel()
+    latest_buffer = RNS.Buffer.create_bidirectional_buffer(0, 0, channel, server_buffer_ready)
+
+def client_disconnected(link):
+    RNS.log("Client disconnected")
+
+def server_buffer_ready(ready_bytes: int):
+    """
+    Callback from buffer when buffer has data available
+
+    :param ready_bytes: The number of bytes ready to read
+    """
+    global latest_buffer
+
+    data = latest_buffer.read(ready_bytes)
+    data = data.decode("utf-8")
+
+    RNS.log("Received data over the buffer: " + data)
+
+    reply_message = "I received \""+data+"\" over the buffer"
+    reply_message = reply_message.encode("utf-8")
+    latest_buffer.write(reply_message)
+    latest_buffer.flush()
+
+
+
+
+##########################################################
+#### Client Part #########################################
+##########################################################
+
+# A reference to the server link
+server_link = None
+
+# A reference to the buffer object, needed to share the
+# object from the link connected callback to the client
+# loop.
+buffer = None
+
+# This initialisation is executed when the users chooses
+# to run as a client
+def client(destination_hexhash, configpath):
+    # We need a binary representation of the destination
+    # hash that was entered on the command line
+    try:
+        dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+        if len(destination_hexhash) != dest_len:
+            raise ValueError(
+                "Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2)
+            )
+            
+        destination_hash = bytes.fromhex(destination_hexhash)
+    except:
+        RNS.log("Invalid destination entered. Check your input!\n")
+        exit()
+
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+
+    # Check if we know a path to the destination
+    if not RNS.Transport.has_path(destination_hash):
+        RNS.log("Destination is not yet known. Requesting path and waiting for announce to arrive...")
+        RNS.Transport.request_path(destination_hash)
+        while not RNS.Transport.has_path(destination_hash):
+            time.sleep(0.1)
+
+    # Recall the server identity
+    server_identity = RNS.Identity.recall(destination_hash)
+
+    # Inform the user that we'll begin connecting
+    RNS.log("Establishing link with server...")
+
+    # When the server identity is known, we set
+    # up a destination
+    server_destination = RNS.Destination(
+        server_identity,
+        RNS.Destination.OUT,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "bufferexample"
+    )
+
+    # And create a link
+    link = RNS.Link(server_destination)
+
+    # We'll also set up functions to inform the
+    # user when the link is established or closed
+    link.set_link_established_callback(link_established)
+    link.set_link_closed_callback(link_closed)
+
+    # Everything is set up, so let's enter a loop
+    # for the user to interact with the example
+    client_loop()
+
+def client_loop():
+    global server_link
+
+    # Wait for the link to become active
+    while not server_link:
+        time.sleep(0.1)
+
+    should_quit = False
+    while not should_quit:
+        try:
+            print("> ", end=" ")
+            text = input()
+
+            # Check if we should quit the example
+            if text == "quit" or text == "q" or text == "exit":
+                should_quit = True
+                server_link.teardown()
+            else:
+                # Otherwise, encode the text and write it to the buffer.
+                text = text.encode("utf-8")
+                buffer.write(text)
+                # Flush the buffer to force the data to be sent.
+                buffer.flush()
+
+
+        except Exception as e:
+            RNS.log("Error while sending data over the link buffer: "+str(e))
+            should_quit = True
+            server_link.teardown()
+
+# This function is called when a link
+# has been established with the server
+def link_established(link):
+    # We store a reference to the link
+    # instance for later use
+    global server_link, buffer
+    server_link = link
+
+    # Create buffer, see server_client_connected() for
+    # more detail about setting up the buffer.
+    channel = link.get_channel()
+    buffer = RNS.Buffer.create_bidirectional_buffer(0, 0, channel, client_buffer_ready)
+
+    # Inform the user that the server is
+    # connected
+    RNS.log("Link established with server, enter some text to send, or \"quit\" to quit")
+
+# When a link is closed, we'll inform the
+# user, and exit the program
+def link_closed(link):
+    if link.teardown_reason == RNS.Link.TIMEOUT:
+        RNS.log("The link timed out, exiting now")
+    elif link.teardown_reason == RNS.Link.DESTINATION_CLOSED:
+        RNS.log("The link was closed by the server, exiting now")
+    else:
+        RNS.log("Link closed, exiting now")
+    
+    RNS.Reticulum.exit_handler()
+    time.sleep(1.5)
+    os._exit(0)
+
+# When the buffer has new data, read it and write it to the terminal.
+def client_buffer_ready(ready_bytes: int):
+    global buffer
+    data = buffer.read(ready_bytes)
+    RNS.log("Received data over the link buffer: " + data.decode("utf-8"))
+    print("> ", end=" ")
+    sys.stdout.flush()
+
+
+##########################################################
+#### Program Startup #####################################
+##########################################################
+
+# This part of the program runs at startup,
+# and parses input of from the user, and then
+# starts up the desired program mode.
+if __name__ == "__main__":
+    try:
+        parser = argparse.ArgumentParser(description="Simple buffer example")
+
+        parser.add_argument(
+            "-s",
+            "--server",
+            action="store_true",
+            help="wait for incoming link requests from clients"
+        )
+
+        parser.add_argument(
+            "--config",
+            action="store",
+            default=None,
+            help="path to alternative Reticulum config directory",
+            type=str
+        )
+
+        parser.add_argument(
+            "destination",
+            nargs="?",
+            default=None,
+            help="hexadecimal hash of the server destination",
+            type=str
+        )
+
+        args = parser.parse_args()
+
+        if args.config:
+            configarg = args.config
+        else:
+            configarg = None
+
+        if args.server:
+            server(configarg)
+        else:
+            if (args.destination == None):
+                print("")
+                parser.print_help()
+                print("")
+            else:
+                client(args.destination, configarg)
+
+    except KeyboardInterrupt:
+        print("")
+        exit()

Examples/Channel.py

@@ -0,0 +1,390 @@
+##########################################################
+# This RNS example demonstrates how to set up a link to  #
+# a destination, and pass structured messages over it    #
+# using a channel.                                       #
+##########################################################
+
+import os
+import sys
+import time
+import argparse
+from datetime import datetime
+
+import RNS
+from RNS.vendor import umsgpack
+
+# Let's define an app name. We'll use this for all
+# destinations we create. Since this echo example
+# is part of a range of example utilities, we'll put
+# them all within the app namespace "example_utilities"
+APP_NAME = "example_utilities"
+
+##########################################################
+#### Shared Objects ######################################
+##########################################################
+
+# Channel data must be structured in a subclass of
+# MessageBase. This ensures that the channel will be able
+# to serialize and deserialize the object and multiplex it
+# with other objects. Both ends of a link will need the
+# same object definitions to be able to communicate over
+# a channel.
+#
+# Note: The objects we wish to use over the channel must
+# be registered with the channel, and each link has a
+# different channel instance. See the client_connected
+# and link_established functions in this example to see
+# how message types are registered.
+
+# Let's make a simple message class called StringMessage
+# that will convey a string with a timestamp.
+
+class StringMessage(RNS.MessageBase):
+    # The MSGTYPE class variable needs to be assigned a
+    # 2 byte integer value. This identifier allows the
+    # channel to look up your message's constructor when a
+    # message arrives over the channel.
+    #
+    # MSGTYPE must be unique across all message types we
+    # register with the channel. MSGTYPEs >= 0xf000 are
+    # reserved for the system.
+    MSGTYPE = 0x0101
+
+    # The constructor of our object must be callable with
+    # no arguments. We can have parameters, but they must
+    # have a default assignment.
+    #
+    # This is needed so the channel can create an empty
+    # version of our message into which the incoming
+    # message can be unpacked.
+    def __init__(self, data=None):
+        self.data = data
+        self.timestamp = datetime.now()
+
+    # Finally, our message needs to implement functions
+    # the channel can call to pack and unpack our message
+    # to/from the raw packet payload. We'll use the
+    # umsgpack package bundled with RNS. We could also use
+    # the struct package bundled with Python if we wanted
+    # more control over the structure of the packed bytes.
+    #
+    # Also note that packed message objects must fit
+    # entirely in one packet. The number of bytes
+    # available for message payloads can be queried from
+    # the channel using the Channel.MDU property. The
+    # channel MDU is slightly less than the link MDU due
+    # to encoding the message header.
+
+    # The pack function encodes the message contents into
+    # a byte stream.
+    def pack(self) -> bytes:
+        return umsgpack.packb((self.data, self.timestamp))
+
+    # And the unpack function decodes a byte stream into
+    # the message contents.
+    def unpack(self, raw):
+        self.data, self.timestamp = umsgpack.unpackb(raw)
+
+
+##########################################################
+#### Server Part #########################################
+##########################################################
+
+# A reference to the latest client link that connected
+latest_client_link = None
+
+# This initialisation is executed when the users chooses
+# to run as a server
+def server(configpath):
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+    
+    # Randomly create a new identity for our link example
+    server_identity = RNS.Identity()
+
+    # We create a destination that clients can connect to. We
+    # want clients to create links to this destination, so we
+    # need to create a "single" destination type.
+    server_destination = RNS.Destination(
+        server_identity,
+        RNS.Destination.IN,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "channelexample"
+    )
+
+    # We configure a function that will get called every time
+    # a new client creates a link to this destination.
+    server_destination.set_link_established_callback(client_connected)
+
+    # Everything's ready!
+    # Let's Wait for client requests or user input
+    server_loop(server_destination)
+
+def server_loop(destination):
+    # Let the user know that everything is ready
+    RNS.log(
+        "Link example "+
+        RNS.prettyhexrep(destination.hash)+
+        " running, waiting for a connection."
+    )
+
+    RNS.log("Hit enter to manually send an announce (Ctrl-C to quit)")
+
+    # We enter a loop that runs until the users exits.
+    # If the user hits enter, we will announce our server
+    # destination on the network, which will let clients
+    # know how to create messages directed towards it.
+    while True:
+        entered = input()
+        destination.announce()
+        RNS.log("Sent announce from "+RNS.prettyhexrep(destination.hash))
+
+# When a client establishes a link to our server
+# destination, this function will be called with
+# a reference to the link.
+def client_connected(link):
+    global latest_client_link
+    latest_client_link = link
+
+    RNS.log("Client connected")
+    link.set_link_closed_callback(client_disconnected)
+
+    # Register message types and add callback to channel
+    channel = link.get_channel()
+    channel.register_message_type(StringMessage)
+    channel.add_message_handler(server_message_received)
+
+def client_disconnected(link):
+    RNS.log("Client disconnected")
+
+def server_message_received(message):
+    """
+    A message handler
+    @param message: An instance of a subclass of MessageBase
+    @return: True if message was handled
+    """
+    global latest_client_link
+    # When a message is received over any active link,
+    # the replies will all be directed to the last client
+    # that connected.
+
+    # In a message handler, any deserializable message
+    # that arrives over the link's channel will be passed
+    # to all message handlers, unless a preceding handler indicates it
+    # has handled the message.
+    #
+    #
+    if isinstance(message, StringMessage):
+        RNS.log("Received data on the link: " + message.data + " (message created at " + str(message.timestamp) + ")")
+
+        reply_message = StringMessage("I received \""+message.data+"\" over the link")
+        latest_client_link.get_channel().send(reply_message)
+
+        # Incoming messages are sent to each message
+        # handler added to the channel, in the order they
+        # were added.
+        # If any message handler returns True, the message
+        # is considered handled and any subsequent
+        # handlers are skipped.
+        return True
+
+
+##########################################################
+#### Client Part #########################################
+##########################################################
+
+# A reference to the server link
+server_link = None
+
+# This initialisation is executed when the users chooses
+# to run as a client
+def client(destination_hexhash, configpath):
+    # We need a binary representation of the destination
+    # hash that was entered on the command line
+    try:
+        dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+        if len(destination_hexhash) != dest_len:
+            raise ValueError(
+                "Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2)
+            )
+            
+        destination_hash = bytes.fromhex(destination_hexhash)
+    except:
+        RNS.log("Invalid destination entered. Check your input!\n")
+        exit()
+
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+
+    # Check if we know a path to the destination
+    if not RNS.Transport.has_path(destination_hash):
+        RNS.log("Destination is not yet known. Requesting path and waiting for announce to arrive...")
+        RNS.Transport.request_path(destination_hash)
+        while not RNS.Transport.has_path(destination_hash):
+            time.sleep(0.1)
+
+    # Recall the server identity
+    server_identity = RNS.Identity.recall(destination_hash)
+
+    # Inform the user that we'll begin connecting
+    RNS.log("Establishing link with server...")
+
+    # When the server identity is known, we set
+    # up a destination
+    server_destination = RNS.Destination(
+        server_identity,
+        RNS.Destination.OUT,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "channelexample"
+    )
+
+    # And create a link
+    link = RNS.Link(server_destination)
+
+    # We'll also set up functions to inform the
+    # user when the link is established or closed
+    link.set_link_established_callback(link_established)
+    link.set_link_closed_callback(link_closed)
+
+    # Everything is set up, so let's enter a loop
+    # for the user to interact with the example
+    client_loop()
+
+def client_loop():
+    global server_link
+
+    # Wait for the link to become active
+    while not server_link:
+        time.sleep(0.1)
+
+    should_quit = False
+    while not should_quit:
+        try:
+            print("> ", end=" ")
+            text = input()
+
+            # Check if we should quit the example
+            if text == "quit" or text == "q" or text == "exit":
+                should_quit = True
+                server_link.teardown()
+
+            # If not, send the entered text over the link
+            if text != "":
+                message = StringMessage(text)
+                packed_size = len(message.pack())
+                channel = server_link.get_channel()
+                if channel.is_ready_to_send():
+                    if packed_size <= channel.MDU:
+                        channel.send(message)
+                    else:
+                        RNS.log(
+                            "Cannot send this packet, the data size of "+
+                            str(packed_size)+" bytes exceeds the link packet MDU of "+
+                            str(channel.MDU)+" bytes",
+                            RNS.LOG_ERROR
+                        )
+                else:
+                    RNS.log("Channel is not ready to send, please wait for " +
+                            "pending messages to complete.", RNS.LOG_ERROR)
+
+        except Exception as e:
+            RNS.log("Error while sending data over the link: "+str(e))
+            should_quit = True
+            server_link.teardown()
+
+# This function is called when a link
+# has been established with the server
+def link_established(link):
+    # We store a reference to the link
+    # instance for later use
+    global server_link
+    server_link = link
+
+    # Register messages and add handler to channel
+    channel = link.get_channel()
+    channel.register_message_type(StringMessage)
+    channel.add_message_handler(client_message_received)
+
+    # Inform the user that the server is
+    # connected
+    RNS.log("Link established with server, enter some text to send, or \"quit\" to quit")
+
+# When a link is closed, we'll inform the
+# user, and exit the program
+def link_closed(link):
+    if link.teardown_reason == RNS.Link.TIMEOUT:
+        RNS.log("The link timed out, exiting now")
+    elif link.teardown_reason == RNS.Link.DESTINATION_CLOSED:
+        RNS.log("The link was closed by the server, exiting now")
+    else:
+        RNS.log("Link closed, exiting now")
+    
+    RNS.Reticulum.exit_handler()
+    time.sleep(1.5)
+    os._exit(0)
+
+# When a packet is received over the channel, we
+# simply print out the data.
+def client_message_received(message):
+    if isinstance(message, StringMessage):
+        RNS.log("Received data on the link: " + message.data + " (message created at " + str(message.timestamp) + ")")
+        print("> ", end=" ")
+        sys.stdout.flush()
+
+
+##########################################################
+#### Program Startup #####################################
+##########################################################
+
+# This part of the program runs at startup,
+# and parses input of from the user, and then
+# starts up the desired program mode.
+if __name__ == "__main__":
+    try:
+        parser = argparse.ArgumentParser(description="Simple channel example")
+
+        parser.add_argument(
+            "-s",
+            "--server",
+            action="store_true",
+            help="wait for incoming link requests from clients"
+        )
+
+        parser.add_argument(
+            "--config",
+            action="store",
+            default=None,
+            help="path to alternative Reticulum config directory",
+            type=str
+        )
+
+        parser.add_argument(
+            "destination",
+            nargs="?",
+            default=None,
+            help="hexadecimal hash of the server destination",
+            type=str
+        )
+
+        args = parser.parse_args()
+
+        if args.config:
+            configarg = args.config
+        else:
+            configarg = None
+
+        if args.server:
+            server(configarg)
+        else:
+            if (args.destination == None):
+                print("")
+                parser.print_help()
+                print("")
+            else:
+                client(args.destination, configarg)
+
+    except KeyboardInterrupt:
+        print("")
+        exit()

Examples/Echo.py

@@ -46,7 +46,7 @@ def server(configpath):
     )
 
     # We configure the destination to automatically prove all
-    # packets adressed to it. By doing this, RNS will automatically
+    # packets addressed to it. By doing this, RNS will automatically
     # generate a proof for each incoming packet and transmit it
     # back to the sender of that packet.
     echo_destination.set_proof_strategy(RNS.Destination.PROVE_ALL)
@@ -210,6 +210,7 @@ def client(destination_hexhash, configpath, timeout=None):
             # If we do not know this destination, tell the
             # user to wait for an announce to arrive.
             RNS.log("Destination is not yet known. Requesting path...")
+            RNS.log("Hit enter to manually retry once an announce is received.")
             RNS.Transport.request_path(destination_hash)
 
 # This function is called when our reply destination

Examples/ExampleInterface.py

@@ -0,0 +1,299 @@
+# MIT License - Copyright (c) 2024 Mark Qvist / unsigned.io
+
+# This example illustrates creating a custom interface
+# definition, that can be loaded and used by Reticulum at
+# runtime. Any number of custom interfaces can be created
+# and loaded. To use the interface place it in the folder
+# ~/.reticulum/interfaces, and add an interface entry to
+# your Reticulum configuration file similar to this:
+
+#  [[Example Custom Interface]]
+#    type = ExampleInterface
+#    enabled = no
+#    mode = gateway
+#    port = /dev/ttyUSB0
+#    speed = 115200
+#    databits = 8
+#    parity = none
+#    stopbits = 1
+
+from time import sleep
+import sys
+import threading
+import time
+
+# This HDLC helper class is used by the interface
+# to delimit and packetize data over the physical
+# medium - in this case a serial connection.
+class HDLC():
+    # This example interface packetizes data using
+    # simplified HDLC framing, similar to PPP
+    FLAG     = 0x7E
+    ESC      = 0x7D
+    ESC_MASK = 0x20
+
+    @staticmethod
+    def escape(data):
+        data = data.replace(bytes([HDLC.ESC]), bytes([HDLC.ESC, HDLC.ESC^HDLC.ESC_MASK]))
+        data = data.replace(bytes([HDLC.FLAG]), bytes([HDLC.ESC, HDLC.FLAG^HDLC.ESC_MASK]))
+        return data
+
+# Let's define our custom interface class. It must
+# be a sub-class of the RNS "Interface" class.
+class ExampleInterface(Interface):
+    # All interface classes must define a default
+    # IFAC size, used in IFAC setup when the user
+    # has not specified a custom IFAC size. This
+    # option is specified in bytes.
+    DEFAULT_IFAC_SIZE = 8
+
+    # The following properties are local to this
+    # particular interface implementation.
+    owner    = None
+    port     = None
+    speed    = None
+    databits = None
+    parity   = None
+    stopbits = None
+    serial   = None
+
+    # All Reticulum interfaces must have an __init__
+    # method that takes 2 positional arguments:
+    # The owner RNS Transport instance, and a dict
+    # of configuration values.
+    def __init__(self, owner, configuration):
+
+        # The following lines demonstrate handling
+        # potential dependencies required for the
+        # interface to function correctly.
+        import importlib
+        if importlib.util.find_spec('serial') != None:
+            import serial
+        else:
+            RNS.log("Using this interface requires a serial communication module to be installed.", RNS.LOG_CRITICAL)
+            RNS.log("You can install one with the command: python3 -m pip install pyserial", RNS.LOG_CRITICAL)
+            RNS.panic()
+
+        # We start out by initialising the super-class
+        super().__init__()
+
+        # To make sure the configuration data is in the
+        # correct format, we parse it through the following
+        # method on the generic Interface class. This step
+        # is required to ensure compatibility on all the
+        # platforms that Reticulum supports.
+        ifconf    = Interface.get_config_obj(configuration)
+
+        # Read the interface name from the configuration
+        # and set it on our interface instance.
+        name      = ifconf["name"]
+        self.name = name
+
+        # We read configuration parameters from the supplied
+        # configuration data, and provide default values in
+        # case any are missing.
+        port      = ifconf["port"] if "port" in ifconf else None
+        speed     = int(ifconf["speed"]) if "speed" in ifconf else 9600
+        databits  = int(ifconf["databits"]) if "databits" in ifconf else 8
+        parity    = ifconf["parity"] if "parity" in ifconf else "N"
+        stopbits  = int(ifconf["stopbits"]) if "stopbits" in ifconf else 1
+
+        # In case no port is specified, we abort setup by
+        # raising an exception.
+        if port == None:
+            raise ValueError(f"No port specified for {self}")
+
+        # All interfaces must supply a hardware MTU value
+        # to the RNS Transport instance. This value should
+        # be the maximum data packet payload size that the
+        # underlying medium is capable of handling in all
+        # cases without any segmentation.
+        self.HW_MTU = 564
+
+        # We initially set the "online" property to false,
+        # since the interface has not actually been fully
+        # initialised and connected yet.
+        self.online   = False
+
+        # In this case, we can also set the indicated bit-
+        # rate of the interface to the serial port speed.
+        self.bitrate  = speed
+        
+        # Configure internal properties on the interface
+        # according to the supplied configuration.
+        self.pyserial = serial
+        self.serial   = None
+        self.owner    = owner
+        self.port     = port
+        self.speed    = speed
+        self.databits = databits
+        self.parity   = serial.PARITY_NONE
+        self.stopbits = stopbits
+        self.timeout  = 100
+
+        if parity.lower() == "e" or parity.lower() == "even":
+            self.parity = serial.PARITY_EVEN
+
+        if parity.lower() == "o" or parity.lower() == "odd":
+            self.parity = serial.PARITY_ODD
+
+        # Since all required parameters are now configured,
+        # we will try opening the serial port.
+        try:
+            self.open_port()
+        except Exception as e:
+            RNS.log("Could not open serial port for interface "+str(self), RNS.LOG_ERROR)
+            raise e
+
+        # If opening the port succeeded, run any post-open
+        # configuration required.
+        if self.serial.is_open:
+            self.configure_device()
+        else:
+            raise IOError("Could not open serial port")
+
+    # Open the serial port with supplied configuration
+    # parameters and store a reference to the open port.
+    def open_port(self):
+        RNS.log("Opening serial port "+self.port+"...", RNS.LOG_VERBOSE)
+        self.serial = self.pyserial.Serial(
+            port = self.port,
+            baudrate = self.speed,
+            bytesize = self.databits,
+            parity = self.parity,
+            stopbits = self.stopbits,
+            xonxoff = False,
+            rtscts = False,
+            timeout = 0,
+            inter_byte_timeout = None,
+            write_timeout = None,
+            dsrdtr = False,
+        )
+
+    # The only thing required after opening the port
+    # is to wait a small amount of time for the
+    # hardware to initialise and then start a thread
+    # that reads any incoming data from the device.
+    def configure_device(self):
+        sleep(0.5)
+        thread = threading.Thread(target=self.read_loop)
+        thread.daemon = True
+        thread.start()
+        self.online = True
+        RNS.log("Serial port "+self.port+" is now open", RNS.LOG_VERBOSE)
+
+
+    # This method will be called from our read-loop
+    # whenever a full packet has been received over
+    # the underlying medium.
+    def process_incoming(self, data):
+        # Update our received bytes counter
+        self.rxb += len(data)            
+
+        # And send the data packet to the Transport
+        # instance for processing.
+        self.owner.inbound(data, self)
+
+    # The running Reticulum Transport instance will
+    # call this method on the interface whenever the
+    # interface must transmit a packet.
+    def process_outgoing(self,data):
+        if self.online:
+            # First, escape and packetize the data
+            # according to HDLC framing.
+            data = bytes([HDLC.FLAG])+HDLC.escape(data)+bytes([HDLC.FLAG])
+
+            # Then write the framed data to the port
+            written = self.serial.write(data)
+
+            # Update the transmitted bytes counter
+            # and ensure that all data was written
+            self.txb += len(data)            
+            if written != len(data):
+                raise IOError("Serial interface only wrote "+str(written)+" bytes of "+str(len(data)))
+
+    # This read loop runs in a thread and continously
+    # receives bytes from the underlying serial port.
+    # When a full packet has been received, it will
+    # be sent to the process_incoming methed, which
+    # will in turn pass it to the Transport instance.
+    def read_loop(self):
+        try:
+            in_frame = False
+            escape = False
+            data_buffer = b""
+            last_read_ms = int(time.time()*1000)
+
+            while self.serial.is_open:
+                if self.serial.in_waiting:
+                    byte = ord(self.serial.read(1))
+                    last_read_ms = int(time.time()*1000)
+
+                    if (in_frame and byte == HDLC.FLAG):
+                        in_frame = False
+                        self.process_incoming(data_buffer)
+                    elif (byte == HDLC.FLAG):
+                        in_frame = True
+                        data_buffer = b""
+                    elif (in_frame and len(data_buffer) < self.HW_MTU):
+                        if (byte == HDLC.ESC):
+                            escape = True
+                        else:
+                            if (escape):
+                                if (byte == HDLC.FLAG ^ HDLC.ESC_MASK):
+                                    byte = HDLC.FLAG
+                                if (byte == HDLC.ESC  ^ HDLC.ESC_MASK):
+                                    byte = HDLC.ESC
+                                escape = False
+                            data_buffer = data_buffer+bytes([byte])
+                        
+                else:
+                    time_since_last = int(time.time()*1000) - last_read_ms
+                    if len(data_buffer) > 0 and time_since_last > self.timeout:
+                        data_buffer = b""
+                        in_frame = False
+                        escape = False
+                    sleep(0.08)
+                    
+        except Exception as e:
+            self.online = False
+            RNS.log("A serial port error occurred, the contained exception was: "+str(e), RNS.LOG_ERROR)
+            RNS.log("The interface "+str(self)+" experienced an unrecoverable error and is now offline.", RNS.LOG_ERROR)
+            
+            if RNS.Reticulum.panic_on_interface_error:
+                RNS.panic()
+
+            RNS.log("Reticulum will attempt to reconnect the interface periodically.", RNS.LOG_ERROR)
+
+        self.online = False
+        self.serial.close()
+        self.reconnect_port()
+
+    # This method handles serial port disconnects.
+    def reconnect_port(self):
+        while not self.online:
+            try:
+                time.sleep(5)
+                RNS.log("Attempting to reconnect serial port "+str(self.port)+" for "+str(self)+"...", RNS.LOG_VERBOSE)
+                self.open_port()
+                if self.serial.is_open:
+                    self.configure_device()
+            except Exception as e:
+                RNS.log("Error while reconnecting port, the contained exception was: "+str(e), RNS.LOG_ERROR)
+
+        RNS.log("Reconnected serial port for "+str(self))
+
+    # Signal to Reticulum that this interface should
+    # not perform any ingress limiting.
+    def should_ingress_limit(self):
+        return False
+
+    # We must provide a string representation of this
+    # interface, that is used whenever the interface
+    # is printed in logs or external programs.
+    def __str__(self):
+        return "ExampleInterface["+self.name+"]"
+
+# Finally, register the defined interface class as the
+# target class for Reticulum to use as an interface
+interface_class = ExampleInterface

Examples/Filetransfer.py

@@ -449,8 +449,7 @@ def link_established(link):
     # And set up a small job to check for
     # a potential timeout in receiving the
     # file list
-    thread = threading.Thread(target=filelist_timeout_job)
-    thread.setDaemon(True)
+    thread = threading.Thread(target=filelist_timeout_job, daemon=True)
     thread.start()
 
 # This job just sleeps for the specified

Examples/Minimal.py

@@ -25,7 +25,7 @@ def program_setup(configpath):
     # Destinations are endpoints in Reticulum, that can be addressed
     # and communicated with. Destinations can also announce their
     # existence, which will let the network know they are reachable
-    # and autoomatically create paths to them, from anywhere else
+    # and automatically create paths to them, from anywhere else
     # in the network.
     destination = RNS.Destination(
         identity,
@@ -36,7 +36,7 @@ def program_setup(configpath):
     )
 
     # We configure the destination to automatically prove all
-    # packets adressed to it. By doing this, RNS will automatically
+    # packets addressed to it. By doing this, RNS will automatically
     # generate a proof for each incoming packet and transmit it
     # back to the sender of that packet. This will let anyone that
     # tries to communicate with the destination know whether their

Examples/Ratchets.py

@@ -0,0 +1,340 @@
+##########################################################
+# This RNS example demonstrates a simple client/server   #
+# echo utility that uses ratchets to rotate encryption   #
+# keys everytime an announce is sent.                    #
+##########################################################
+
+import argparse
+import RNS
+
+# Let's define an app name. We'll use this for all
+# destinations we create. Since this echo example
+# is part of a range of example utilities, we'll put
+# them all within the app namespace "example_utilities"
+APP_NAME = "example_utilities"
+
+
+##########################################################
+#### Server Part #########################################
+##########################################################
+
+# This initialisation is executed when the users chooses
+# to run as a server
+def server(configpath):
+    global reticulum
+
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+    
+    # Randomly create a new identity for our echo server
+    server_identity = RNS.Identity()
+
+    # We create a destination that clients can query. We want
+    # to be able to verify echo replies to our clients, so we
+    # create a "single" destination that can receive encrypted
+    # messages. This way the client can send a request and be
+    # certain that no-one else than this destination was able
+    # to read it. 
+    echo_destination = RNS.Destination(
+        server_identity,
+        RNS.Destination.IN,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "ratchet",
+        "echo",
+        "request"
+    )
+
+    # Enable ratchets on the destination by providing a file
+    # path to store ratchets. In this example, we will just
+    # use a temporary file, but in real-world applications,
+    # it's extremely important to keep this file secure, since
+    # it contains encryption keys for the destination.
+    destination_hexhash = RNS.hexrep(echo_destination.hash, delimit=False)
+    echo_destination.enable_ratchets(f"/tmp/{destination_hexhash}.ratchets")
+
+    # We configure the destination to automatically prove all
+    # packets addressed to it. By doing this, RNS will automatically
+    # generate a proof for each incoming packet and transmit it
+    # back to the sender of that packet.
+    echo_destination.set_proof_strategy(RNS.Destination.PROVE_ALL)
+    
+    # Tell the destination which function in our program to
+    # run when a packet is received. We do this so we can
+    # print a log message when the server receives a request
+    echo_destination.set_packet_callback(server_callback)
+
+    # Everything's ready!
+    # Let's Wait for client requests or user input
+    announceLoop(echo_destination)
+
+
+def announceLoop(destination):
+    # Let the user know that everything is ready
+    RNS.log(
+        "Ratcheted echo server "+
+        RNS.prettyhexrep(destination.hash)+
+        " running, hit enter to manually send an announce (Ctrl-C to quit)"
+    )
+
+    # We enter a loop that runs until the users exits.
+    # If the user hits enter, we will announce our server
+    # destination on the network, which will let clients
+    # know how to create messages directed towards it.
+    while True:
+        entered = input()
+        destination.announce()
+        RNS.log("Sent announce from "+RNS.prettyhexrep(destination.hash))
+
+
+def server_callback(message, packet):
+    global reticulum
+    
+    # Tell the user that we received an echo request, and
+    # that we are going to send a reply to the requester.
+    # Sending the proof is handled automatically, since we
+    # set up the destination to prove all incoming packets.
+
+    reception_stats = ""
+    if reticulum.is_connected_to_shared_instance:
+        reception_rssi = reticulum.get_packet_rssi(packet.packet_hash)
+        reception_snr  = reticulum.get_packet_snr(packet.packet_hash)
+
+        if reception_rssi != None:
+            reception_stats += " [RSSI "+str(reception_rssi)+" dBm]"
+        
+        if reception_snr != None:
+            reception_stats += " [SNR "+str(reception_snr)+" dBm]"
+
+    else:
+        if packet.rssi != None:
+            reception_stats += " [RSSI "+str(packet.rssi)+" dBm]"
+        
+        if packet.snr != None:
+            reception_stats += " [SNR "+str(packet.snr)+" dB]"
+
+    RNS.log("Received packet from echo client, proof sent"+reception_stats)
+
+
+##########################################################
+#### Client Part #########################################
+##########################################################
+
+# This initialisation is executed when the users chooses
+# to run as a client
+def client(destination_hexhash, configpath, timeout=None):
+    global reticulum
+    
+    # We need a binary representation of the destination
+    # hash that was entered on the command line
+    try:
+        dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+        if len(destination_hexhash) != dest_len:
+            raise ValueError(
+                "Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2)
+            )
+
+        destination_hash = bytes.fromhex(destination_hexhash)
+    except Exception as e:
+        RNS.log("Invalid destination entered. Check your input!")
+        RNS.log(str(e)+"\n")
+        exit()
+
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+
+    # We override the loglevel to provide feedback when
+    # an announce is received
+    if RNS.loglevel < RNS.LOG_INFO:
+        RNS.loglevel = RNS.LOG_INFO
+
+    # Tell the user that the client is ready!
+    RNS.log(
+        "Echo client ready, hit enter to send echo request to "+
+        destination_hexhash+
+        " (Ctrl-C to quit)"
+    )
+
+    # We enter a loop that runs until the user exits.
+    # If the user hits enter, we will try to send an
+    # echo request to the destination specified on the
+    # command line.
+    while True:
+        input()
+        
+        # Let's first check if RNS knows a path to the destination.
+        # If it does, we'll load the server identity and create a packet
+        if RNS.Transport.has_path(destination_hash):
+
+            # To address the server, we need to know it's public
+            # key, so we check if Reticulum knows this destination.
+            # This is done by calling the "recall" method of the
+            # Identity module. If the destination is known, it will
+            # return an Identity instance that can be used in
+            # outgoing destinations.
+            server_identity = RNS.Identity.recall(destination_hash)
+
+            # We got the correct identity instance from the
+            # recall method, so let's create an outgoing
+            # destination. We use the naming convention:
+            # example_utilities.ratchet.echo.request
+            # This matches the naming we specified in the
+            # server part of the code.
+            request_destination = RNS.Destination(
+                server_identity,
+                RNS.Destination.OUT,
+                RNS.Destination.SINGLE,
+                APP_NAME,
+                "ratchet",
+                "echo",
+                "request"
+            )
+
+            # The destination is ready, so let's create a packet.
+            # We set the destination to the request_destination
+            # that was just created, and the only data we add
+            # is a random hash.
+            echo_request = RNS.Packet(request_destination, RNS.Identity.get_random_hash())
+
+            # Send the packet! If the packet is successfully
+            # sent, it will return a PacketReceipt instance.
+            packet_receipt = echo_request.send()
+
+            # If the user specified a timeout, we set this
+            # timeout on the packet receipt, and configure
+            # a callback function, that will get called if
+            # the packet times out.
+            if timeout != None:
+                packet_receipt.set_timeout(timeout)
+                packet_receipt.set_timeout_callback(packet_timed_out)
+
+            # We can then set a delivery callback on the receipt.
+            # This will get automatically called when a proof for
+            # this specific packet is received from the destination.
+            packet_receipt.set_delivery_callback(packet_delivered)
+
+            # Tell the user that the echo request was sent
+            RNS.log("Sent echo request to "+RNS.prettyhexrep(request_destination.hash))
+        else:
+            # If we do not know this destination, tell the
+            # user to wait for an announce to arrive.
+            RNS.log("Destination is not yet known. Requesting path...")
+            RNS.log("Hit enter to manually retry once an announce is received.")
+            RNS.Transport.request_path(destination_hash)
+
+# This function is called when our reply destination
+# receives a proof packet.
+def packet_delivered(receipt):
+    global reticulum
+
+    if receipt.status == RNS.PacketReceipt.DELIVERED:
+        rtt = receipt.get_rtt()
+        if (rtt >= 1):
+            rtt = round(rtt, 3)
+            rttstring = str(rtt)+" seconds"
+        else:
+            rtt = round(rtt*1000, 3)
+            rttstring = str(rtt)+" milliseconds"
+
+        reception_stats = ""
+        if reticulum.is_connected_to_shared_instance:
+            reception_rssi = reticulum.get_packet_rssi(receipt.proof_packet.packet_hash)
+            reception_snr  = reticulum.get_packet_snr(receipt.proof_packet.packet_hash)
+
+            if reception_rssi != None:
+                reception_stats += " [RSSI "+str(reception_rssi)+" dBm]"
+            
+            if reception_snr != None:
+                reception_stats += " [SNR "+str(reception_snr)+" dB]"
+
+        else:
+            if receipt.proof_packet != None:
+                if receipt.proof_packet.rssi != None:
+                    reception_stats += " [RSSI "+str(receipt.proof_packet.rssi)+" dBm]"
+                
+                if receipt.proof_packet.snr != None:
+                    reception_stats += " [SNR "+str(receipt.proof_packet.snr)+" dB]"
+
+        RNS.log(
+            "Valid reply received from "+
+            RNS.prettyhexrep(receipt.destination.hash)+
+            ", round-trip time is "+rttstring+
+            reception_stats
+        )
+
+# This function is called if a packet times out.
+def packet_timed_out(receipt):
+    if receipt.status == RNS.PacketReceipt.FAILED:
+        RNS.log("Packet "+RNS.prettyhexrep(receipt.hash)+" timed out")
+
+
+##########################################################
+#### Program Startup #####################################
+##########################################################
+
+# This part of the program gets run at startup,
+# and parses input from the user, and then starts
+# the desired program mode.
+if __name__ == "__main__":
+    try:
+        parser = argparse.ArgumentParser(description="Simple ratcheted echo server and client utility")
+
+        parser.add_argument(
+            "-s",
+            "--server",
+            action="store_true",
+            help="wait for incoming packets from clients"
+        )
+
+        parser.add_argument(
+            "-t",
+            "--timeout",
+            action="store",
+            metavar="s",
+            default=None,
+            help="set a reply timeout in seconds",
+            type=float
+        )
+
+        parser.add_argument("--config",
+            action="store",
+            default=None,
+            help="path to alternative Reticulum config directory",
+            type=str
+        )
+
+        parser.add_argument(
+            "destination",
+            nargs="?",
+            default=None,
+            help="hexadecimal hash of the server destination",
+            type=str
+        )
+
+        args = parser.parse_args()
+
+        if args.server:
+            configarg=None
+            if args.config:
+                configarg = args.config
+            server(configarg)
+        else:
+            if args.config:
+                configarg = args.config
+            else:
+                configarg = None
+
+            if args.timeout:
+                timeoutarg = float(args.timeout)
+            else:
+                timeoutarg = None
+
+            if (args.destination == None):
+                print("")
+                parser.print_help()
+                print("")
+            else:
+                client(args.destination, configarg, timeout=timeoutarg)
+    except KeyboardInterrupt:
+        print("")
+        exit()

Examples/Request.py

@@ -23,8 +23,8 @@ APP_NAME = "example_utilities"
 # A reference to the latest client link that connected
 latest_client_link = None
 
-def random_text_generator(path, data, request_id, remote_identity, requested_at):
-    RNS.log("Generating response to request "+RNS.prettyhexrep(request_id))
+def random_text_generator(path, data, request_id, link_id, remote_identity, requested_at):
+    RNS.log("Generating response to request "+RNS.prettyhexrep(request_id)+" on link "+RNS.prettyhexrep(link_id))
     texts = ["They looked up", "On each full moon", "Becky was upset", "I’ll stay away from it", "The pet shop stocks everything"]
     return texts[random.randint(0, len(texts)-1)]
 

Examples/Speedtest.py

@@ -246,8 +246,8 @@ def link_established(link):
 
     # Inform the user that the server is
     # connected
-    RNS.log("Link established with server,sending...")
-    rd = os.urandom(RNS.Link.MDU)
+    RNS.log("Link established with server, sending...")
+    rd = os.urandom(link.mdu)
     started = time.time()
     while link.status == RNS.Link.ACTIVE and data_sent < data_cap*1.25:
         RNS.Packet(server_link, rd, create_receipt=False).send()

FUNDING.yml

@@ -0,0 +1,2 @@
+ko_fi: markqvist
+custom: "https://unsigned.io/donate"

LICENSE

@@ -1,6 +1,6 @@
 MIT License, unless otherwise noted
 
-Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+Copyright (c) 2016-2024 Mark Qvist / unsigned.io
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

Makefile

@@ -2,7 +2,7 @@ all: release
 
 test:
 	@echo Running tests...
-	python -m tests.all
+	python3 -m tests.all
 
 clean:
 	@echo Cleaning...
@@ -47,7 +47,7 @@ documentation:
 	make -C docs html
 
 manual:
-	make -C docs latexpdf
+	make -C docs latexpdf epub
 
 release: test remove_symlinks build_wheel build_pure_wheel documentation manual create_symlinks
 

README.md

@@ -1,10 +1,10 @@
-Reticulum Network Stack β
+Reticulum Network Stack β <img align="right" src="https://static.pepy.tech/personalized-badge/rns?period=total&units=international_system&left_color=grey&right_color=blue&left_text=Installs" style="padding-left:10px"/><a href="https://github.com/markqvist/Reticulum/actions/workflows/build.yml"><img align="right" src="https://github.com/markqvist/Reticulum/actions/workflows/build.yml/badge.svg"/></a>
 ==========
 
-<p align="center"><img width="200" src="https://unsigned.io/wp-content/uploads/2022/03/reticulum_logo_512.png"></p>
+<p align="center"><img width="200" src="https://raw.githubusercontent.com/markqvist/Reticulum/master/docs/source/graphics/rns_logo_512.png"></p>
 
-Reticulum is the cryptography-based networking stack for wide-area networks
-built on readily available hardware. It can operate even with very high latency
+Reticulum is the cryptography-based networking stack for building local and wide-area
+networks with readily available hardware. It can operate even with very high latency
 and extremely low bandwidth. Reticulum allows you to build wide-area networks
 with off-the-shelf tools, and offers end-to-end encryption and connectivity,
 initiator anonymity, autoconfiguring cryptographically backed multi-hop
@@ -35,44 +35,64 @@ userland, and can run on practically any system that runs Python 3.
 ## Read The Manual
 The full documentation for Reticulum is available at [markqvist.github.io/Reticulum/manual/](https://markqvist.github.io/Reticulum/manual/).
 
-You can also [download the Reticulum manual as a PDF](https://github.com/markqvist/Reticulum/raw/master/docs/Reticulum%20Manual.pdf)
+You can also download the [Reticulum manual as a PDF](https://github.com/markqvist/Reticulum/raw/master/docs/Reticulum%20Manual.pdf) or [as an e-book in EPUB format](https://github.com/markqvist/Reticulum/raw/master/docs/Reticulum%20Manual.epub).
 
-For more info, see [unsigned.io/projects/reticulum](https://unsigned.io/projects/reticulum/)
+For more info, see [reticulum.network](https://reticulum.network/) and [the FAQ section of the wiki](https://github.com/markqvist/Reticulum/wiki/Frequently-Asked-Questions).
 
 ## Notable Features
 - Coordination-less globally unique addressing and identification
-- Fully self-configuring multi-hop routing
-- Complete initiator anonymity, communicate without revealing your identity
+- Fully self-configuring multi-hop routing over heterogeneous carriers
+- Flexible scalability over heterogeneous topologies
+  - Reticulum can carry data over any mixture of physical mediums and topologies
+  - Low-bandwidth networks can co-exist and interoperate with large, high-bandwidth networks
+- Initiator anonymity, communicate without revealing your identity
+  - Reticulum does not include source addresses on any packets
 - Asymmetric X25519 encryption and Ed25519 signatures as a basis for all communication
-- Forward Secrecy with ephemeral Elliptic Curve Diffie-Hellman keys on Curve25519
-- Reticulum uses the [Fernet](https://github.com/fernet/spec/blob/master/Spec.md) specification for on-the-wire / over-the-air encryption
-  - Keys are ephemeral and derived from an ECDH key exchange on Curve25519
+  - The foundational Reticulum Identity Keys are 512-bit Elliptic Curve keysets
+- Forward Secrecy is available for all communication types, both for single packets and over links
+- Reticulum uses the following format for encrypted tokens:
+  - Ephemeral per-packet and link keys and derived from an ECDH key exchange on Curve25519
   - AES-128 in CBC mode with PKCS7 padding
   - HMAC using SHA256 for authentication
   - IVs are generated through os.urandom()
 - Unforgeable packet delivery confirmations
-- A variety of supported interface types
+- Flexible and extensible interface system
+  - Reticulum includes a large variety of built-in interface types
+  - Ability to load and utilise custom user- or community-supplied interface types
+  - Easily create your own custom interfaces for communicating over anything
+- Authentication and virtual network segmentation on all supported interface types
 - An intuitive and easy-to-use API
+  - Simpler and easier to use than sockets APIs and simpler, but more powerful
+  - Makes building distributed and decentralised applications much simpler
 - Reliable and efficient transfer of arbitrary amounts of data
   - Reticulum can handle a few bytes of data or files of many gigabytes
-  - Sequencing, transfer coordination and checksumming are automatic
+  - Sequencing, compression, transfer coordination and checksumming are automatic
   - The API is very easy to use, and provides transfer progress
 - Lightweight, flexible and expandable Request/Response mechanism
 - Efficient link establishment
-  - Total bandwidth cost of setting up an encrypted link is 3 packets totaling 297 bytes
+  - Total cost of setting up an encrypted and verified link is only 3 packets, totalling 297 bytes
   - Low cost of keeping links open at only 0.44 bits per second
+- Reliable sequential delivery with Channel and Buffer mechanisms
+
+## Roadmap
+While Reticulum is already a fully featured and functional networking stack,
+many improvements and additions are actively being worked on, and planned for the future.
+
+To learn more about the direction and future of Reticulum, please see the [Development Roadmap](./Roadmap.md).
 
 ## Examples of Reticulum Applications
 If you want to quickly get an idea of what Reticulum can do, take a look at the
 following resources.
 
+- You can use the [rnsh](https://github.com/acehoss/rnsh) program to establish remote shell sessions over Reticulum.
 - [LXMF](https://github.com/markqvist/lxmf) is a distributed, delay and disruption tolerant message transfer protocol built on Reticulum
 - For an off-grid, encrypted and resilient mesh communications platform, see [Nomad Network](https://github.com/markqvist/NomadNet)
-- The Android, Linux and macOS app [Sideband](https://unsigned.io/sideband) has a graphical interface and focuses on ease of use.
+- The Android, Linux, macOS and Windows app [Sideband](https://github.com/markqvist/Sideband) has a graphical interface and focuses on ease of use.
+- [MeshChat](https://github.com/liamcottle/reticulum-meshchat) is a user-friendly LXMF client, that also supports voice calls.
 
 ## Where can Reticulum be used?
 Over practically any medium that can support at least a half-duplex channel
-with 500 bits per second throughput, and an MTU of 500 bytes. Data radios,
+with greater throughput than 5 bits per second, and an MTU of 500 bytes. Data radios,
 modems, LoRa radios, serial lines, AX.25 TNCs, amateur radio digital modes,
 WiFi and Ethernet devices, free-space optical links, and similar systems are
 all examples of the types of physical devices Reticulum can use.
@@ -102,20 +122,36 @@ you want to do. For full details and examples, have a look at the
 [Getting Started Fast](https://markqvist.github.io/Reticulum/manual/gettingstartedfast.html) 
 section of the [Reticulum Manual](https://markqvist.github.io/Reticulum/manual/).
 
-To simply install Reticulum and related utilities on your system, the easiest way is via pip:
+To simply install Reticulum and related utilities on your system, the easiest way is via `pip`.
+You can then start any program that uses Reticulum, or start Reticulum as a system service with
+[the rnsd utility](https://markqvist.github.io/Reticulum/manual/using.html#the-rnsd-utility).
 
 ```bash
-pip3 install rns
+pip install rns
 ```
 
-You can then start any program that uses Reticulum, or start Reticulum as a
-system service with [the rnsd utility](https://markqvist.github.io/Reticulum/manual/using.html#the-rnsd-utility).
+If you are using an operating system that blocks normal user package installation via `pip`,
+you can return `pip` to normal behaviour by editing the `~/.config/pip/pip.conf` file,
+and adding the following directive in the `[global]` section:
+
+```text
+[global]
+break-system-packages = true
+```
+
+Alternatively, you can use the `pipx` tool to install Reticulum in an isolated environment:
+
+```bash
+pipx install rns
+```
 
 When first started, Reticulum will create a default configuration file,
 providing basic connectivity to other Reticulum peers that might be locally
 reachable. The default config file contains a few examples, and references for
 creating a more complex configuration.
 
+If you have an old version of `pip` on your system, you may need to upgrade it first with `pip install pip --upgrade`. If you no not already have `pip` installed, you can install it using the package manager of your system with `sudo apt install python3-pip` or similar.
+
 For more detailed examples on how to expand communication over many mediums such 
 as packet radio or LoRa, serial ports, or over fast IP links and the Internet using 
 the UDP and TCP interfaces, take a look at the [Supported Interfaces](https://markqvist.github.io/Reticulum/manual/interfaces.html) 
@@ -131,25 +167,29 @@ section of the [Reticulum Manual](https://markqvist.github.io/Reticulum/manual/)
 - An interface status utility called `rnstatus`, that displays information about interfaces
 - The path lookup and management tool `rnpath` letting you view and modify path tables
 - A diagnostics tool called `rnprobe` for checking connectivity to destinations
-- A simple file transfer program called `rncp` making it easy to copy files to remote systems
+- A simple file transfer program called `rncp` making it easy to transfer files between systems
+- The identity management and encryption utility `rnid` let's you manage Identities and encrypt/decrypt files
 - The remote command execution program `rnx` let's you run commands and
   programs and retrieve output from remote systems
 
 All tools, including `rnx` and `rncp`, work reliably and well even over very
-low-bandwidth links like LoRa or Packet Radio.
+low-bandwidth links like LoRa or Packet Radio. For full-featured remote shells
+over Reticulum, also have a look at the [rnsh](https://github.com/acehoss/rnsh)
+program.
 
 ## Supported interface types and devices
 
 Reticulum implements a range of generalised interface types that covers most of
 the communications hardware that Reticulum can run over. If your hardware is
-not supported, it's relatively simple to implement an interface class. I will
-gratefully accept pull requests for custom interfaces if they are generally
-useful.
+not supported, it's [simple to implement a custom interface module](https://markqvist.github.io/Reticulum/manual/interfaces.html#custom-interfaces).
 
-Currently, the following interfaces are supported:
+Pull requests for custom interfaces are gratefully accepted, provided they are
+generally useful and well-tested in real-world usage.
+
+Currently, the following built-in interfaces are supported:
 
 - Any Ethernet device
-- LoRa using [RNode](https://unsigned.io/projects/rnode/)
+- LoRa using [RNode](https://unsigned.io/rnode/)
 - Packet Radio TNCs (with or without AX.25)
 - KISS-compatible hardware and software modems
 - Any device with a serial port
@@ -160,12 +200,12 @@ Currently, the following interfaces are supported:
 
 ## Performance
 Reticulum targets a *very* wide usable performance envelope, but prioritises
-functionality and performance over low-bandwidth mediums. The goal is to
+functionality and performance on low-bandwidth mediums. The goal is to
 provide a dynamic performance envelope from 250 bits per second, to 1 gigabit
 per second on normal hardware.
 
-Currently, the usable performance envelope is approximately 500 bits per second
-to 20 megabits per second, with physical mediums faster than that not being
+Currently, the usable performance envelope is approximately 150 bits per second
+to 40 megabits per second, with physical mediums faster than that not being
 saturated. Performance beyond the current level is intended for future
 upgrades, but not highly prioritised at this point in time.
 
@@ -175,39 +215,6 @@ features are implemented and functioning, but additions will probably occur as
 real-world use is explored. There will be bugs. The API and wire-format can be
 considered relatively stable at the moment, but could change if warranted.
 
-## Development Roadmap
-- Version 0.4.0
-  - Improving [the manual](https://markqvist.github.io/Reticulum/manual/) with sections specifically for beginners
-  - Performance and memory optimisations
-  - Utilities for managing identities, signing and encryption
-  - User friendly interface configuration tool
-  - Support for radio and modem interfaces on Android
-  - More interface types for even broader compatibility
-    - Plain ESP32 devices (ESP-Now, WiFi, Bluetooth, etc.)
-    - More LoRa transceivers
-    - IR Transceivers
-- Planned, but not yet scheduled
-  - OpenWRT support
-  - Metric-based path selection
-  - Distributed Destination Naming System
-  - Network-wide path balancing
-  - Globally routable multicast
-  - Bindings for other programming languages
-  - Multiple paths in path table for quick recovery on link failures
-  - A portable Reticulum implementation in C, see [#21](https://github.com/markqvist/Reticulum/discussions/21)
-  - Easy way to share interface configurations, see [#19](https://github.com/markqvist/Reticulum/discussions/19)
-  - More interface types
-    - AT-compatible modems
-    - AWDL / OWL
-    - HF Modems
-    - CAN-bus
-    - ZeroMQ
-    - MQTT
-    - XBee
-    - SPI
-    - i²c
-    - Tor
-
 ## Dependencies
 The installation of the default `rns` package requires the dependencies listed
 below. Almost all systems and distributions have readily available packages for
@@ -215,7 +222,6 @@ these dependencies, and when the `rns` package is installed with `pip`, they
 will be downloaded and installed as well.
 
 - [PyCA/cryptography](https://github.com/pyca/cryptography)
-- [netifaces](https://github.com/al45tair/netifaces)
 - [pyserial](https://github.com/pyserial/pyserial)
 
 On more unusual systems, and in some rare cases, it might not be possible to
@@ -240,10 +246,17 @@ Primitives](#cryptographic-primitives) section of this document.
 
 ## Public Testnet
 If you just want to get started experimenting without building any physical
-networks, you are welcome to join the Unsigned.io RNS Testnet. The testnet is
-just that, an informal network for testing and experimenting. It will be up
-most of the time, and anyone can join, but it also means that there's no
-guarantees for service availability.
+networks, you are welcome to join the RNS Development Testnet.
+
+The testnet is just that, an informal network for testing and experimenting.
+It will be up most of the time, and anyone can join, but it also means that
+there's no guarantees for service availability.
+
+It probably goes without saying, but *don't use the testnet entry-points as 
+hardcoded or default interfaces in any applications you ship to users*. When
+shipping applications, the best practice is to provide your own default
+connectivity solutions, if needed and applicable, or in most cases, simply
+leave it up to the user which networks to connect to, and how.
 
 The testnet runs the very latest version of Reticulum (often even a short while
 before it is publicly released). Sometimes experimental versions of Reticulum
@@ -253,31 +266,25 @@ I2P. Just add one of the following interfaces to your Reticulum configuration
 file:
 
 ```
-# TCP/IP interface to the Dublin Hub
-  [[RNS Testnet Dublin]]
+# TCP/IP interface to the RNS Amsterdam Hub
+  [[RNS Testnet Amsterdam]]
     type = TCPClientInterface
     enabled = yes
-    target_host = dublin.connect.reticulum.network
+    target_host = amsterdam.connect.reticulum.network
     target_port = 4965
 
-# TCP/IP interface to the Frankfurt Hub
-  [[RNS Testnet Frankfurt]]
+# TCP/IP interface to the BetweenTheBorders Hub (community-provided)
+  [[RNS Testnet BetweenTheBorders]]
     type = TCPClientInterface
     enabled = yes
-    target_host = frankfurt.connect.reticulum.network
-    target_port = 5377
+    target_host = reticulum.betweentheborders.com
+    target_port = 4242
 
-# Interface to I2P Hub A
-  [[RNS Testnet I2P Hub A]]
+# Interface to Testnet I2P Hub
+  [[RNS Testnet I2P Hub]]
     type = I2PInterface
     enabled = yes
-    peers = mrwqlsioq4hoo2lmeeud7dkfscnm7yxak7dmiyvsrnpfag3z5tsq.b32.i2p
-
-# Interface to I2P Hub B
-  [[RNS Testnet I2P Hub B]]
-    type = I2PInterface
-    enabled = yes
-    peers = iwoqtz22dsr73aemwpw7guocplsjjoamyl7sogj33qtcd6ds4mza.b32.i2p
+    peers = g3br23bvx3lq5uddcsjii74xgmn6y5q325ovrkq2zw2wbzbqgbuq.b32.i2p
 ```
 
 The testnet also contains a number of [Nomad Network](https://github.com/markqvist/nomadnet) nodes, and LXMF propagation nodes.
@@ -286,16 +293,16 @@ The testnet also contains a number of [Nomad Network](https://github.com/markqvi
 You can help support the continued development of open, free and private communications systems by donating via one of the following channels:
 
 - Monero:
- ```
+  ```
   84FpY1QbxHcgdseePYNmhTHcrgMX4nFfBYtz2GKYToqHVVhJp8Eaw1Z1EedRnKD19b3B8NiLCGVxzKV17UMmmeEsCrPyA5w
   ```
 - Ethereum
   ```
-  0x81F7B979fEa6134bA9FD5c701b3501A2e61E897a
+  0xFDabC71AC4c0C78C95aDDDe3B4FA19d6273c5E73
   ```
 - Bitcoin
   ```
-  3CPmacGm34qYvR6XWLVEJmi2aNe3PZqUuq
+  35G9uWVzrpJJibzUwpNUQGQNFzLirhrYAH
   ```
 - Ko-Fi: https://ko-fi.com/markqvist
 
@@ -303,17 +310,20 @@ Are certain features in the development roadmap are important to you or your
 organisation? Make them a reality quickly by sponsoring their implementation.
 
 ## Cryptographic Primitives
-Reticulum uses a simple suite of efficient, strong and modern cryptographic
+Reticulum uses a simple suite of efficient, strong and well-tested cryptographic
 primitives, with widely available implementations that can be used both on
-general-purpose CPUs and on microcontrollers. The necessary primitives are:
+general-purpose CPUs and on microcontrollers. The utilised primitives are:
 
-- Ed25519 for signatures
-- X22519 for ECDH key exchanges
+- Reticulum Identity Keys are 512-bit Curve25519 keysets
+  - A 256-bit Ed25519 key for signatures
+  - A 256-bit X22519 key for ECDH key exchanges
 - HKDF for key derivation
-- Modified Fernet for encrypted tokens
-  - AES-128 in CBC mode
-  - HMAC for message authentication
-  - No Fernet version and timestamp fields
+- Encrypted tokens are based on the [Fernet spec](https://github.com/fernet/spec/)
+  - Ephemeral keys derived from an ECDH key exchange on Curve25519
+  - AES-128 in CBC mode with PKCS7 padding
+  - HMAC using SHA256 for message authentication
+  - IVs are generated through os.urandom()
+  - No Fernet version and timestamp metadata fields
 - SHA-256
 - SHA-512
 
@@ -322,12 +332,12 @@ In the default installation configuration, the `X25519`, `Ed25519` and
 (via the [PyCA/cryptography](https://github.com/pyca/cryptography) package).
 The hashing functions `SHA-256` and `SHA-512` are provided by the standard
 Python [hashlib](https://docs.python.org/3/library/hashlib.html). The `HKDF`,
-`HMAC`, `Fernet` primitives, and the `PKCS7` padding function are always
+`HMAC`, `Token` primitives, and the `PKCS7` padding function are always
 provided by the following internal implementations:
 
 - [HKDF.py](RNS/Cryptography/HKDF.py)
 - [HMAC.py](RNS/Cryptography/HMAC.py)
-- [Fernet.py](RNS/Cryptography/Fernet.py)
+- [Token.py](RNS/Cryptography/Token.py)
 - [PKCS7.py](RNS/Cryptography/PKCS7.py)
 
 
@@ -366,8 +376,8 @@ projects:
 - [Curve25519.py](https://gist.github.com/nickovs/cc3c22d15f239a2640c185035c06f8a3#file-curve25519-py) by [Nicko van Someren](https://gist.github.com/nickovs), *Public Domain*
 - [I2Plib](https://github.com/l-n-s/i2plib) by [Viktor Villainov](https://github.com/l-n-s)
 - [PySerial](https://github.com/pyserial/pyserial) by Chris Liechti, *BSD License*
-- [Netifaces](https://github.com/al45tair/netifaces) by [Alastair Houghton](https://github.com/al45tair), *MIT License*
 - [Configobj](https://github.com/DiffSK/configobj) by Michael Foord, Nicola Larosa, Rob Dennis & Eli Courtwright, *BSD License*
 - [Six](https://github.com/benjaminp/six) by [Benjamin Peterson](https://github.com/benjaminp), *MIT License*
+- [ifaddr](https://github.com/pydron/ifaddr) by [Pydron](https://github.com/pydron), *MIT License*
 - [Umsgpack.py](https://github.com/vsergeev/u-msgpack-python) by [Ivan A. Sergeev](https://github.com/vsergeev)
 - [Python](https://www.python.org)

RNS/Buffer.py

@@ -0,0 +1,361 @@
+# MIT License
+#
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io and contributors.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from __future__ import annotations
+import bz2
+import sys
+import time
+import threading
+from threading import RLock
+import struct
+from RNS.Channel import Channel, MessageBase, SystemMessageTypes
+import RNS
+from io import RawIOBase, BufferedRWPair, BufferedReader, BufferedWriter
+from typing import Callable
+from contextlib import AbstractContextManager
+
+class StreamDataMessage(MessageBase):
+    MSGTYPE = SystemMessageTypes.SMT_STREAM_DATA
+    """
+    Message type for ``Channel``. ``StreamDataMessage``
+    uses a system-reserved message type.
+    """
+
+    STREAM_ID_MAX = 0x3fff  # 16383
+    """
+    The stream id is limited to 2 bytes - 2 bit
+    """
+
+    OVERHEAD     = 2 + 6  # 2 for stream data message header, 6 for channel envelope
+    MAX_DATA_LEN = RNS.Link.MDU - OVERHEAD
+    """
+    When the Buffer package is imported, this value is
+    calculcated based on the value of OVERHEAD
+    """
+
+    def __init__(self, stream_id: int = None, data: bytes = None, eof: bool = False, compressed: bool = False):
+        """
+        This class is used to encapsulate binary stream
+        data to be sent over a ``Channel``.
+
+        :param stream_id: id of stream relative to receiver
+        :param data: binary data
+        :param eof: set to True if signalling End of File
+        """
+        super().__init__()
+        if stream_id is not None and stream_id > self.STREAM_ID_MAX:
+            raise ValueError("stream_id must be 0-16383")
+        self.stream_id = stream_id
+        self.compressed = compressed
+        self.data = data or bytes()
+        self.eof = eof
+
+    def pack(self) -> bytes:
+        if self.stream_id is None:
+            raise ValueError("stream_id")
+
+        header_val = (0x3fff & self.stream_id) | (0x8000 if self.eof else 0x0000) | (0x4000 if self.compressed > 0 else 0x0000)
+        return bytes(struct.pack(">H", header_val) + (self.data if self.data else bytes()))
+
+    def unpack(self, raw):
+        self.stream_id = struct.unpack(">H", raw[:2])[0]
+        self.eof = (0x8000 & self.stream_id) > 0
+        self.compressed = (0x4000 & self.stream_id) > 0
+        self.stream_id = self.stream_id & 0x3fff
+        self.data = raw[2:]
+
+        if self.compressed:
+            self.data = bz2.decompress(self.data)
+
+
+class RawChannelReader(RawIOBase, AbstractContextManager):
+    """
+    An implementation of RawIOBase that receives
+    binary stream data sent over a ``Channel``.
+
+      This class generally need not be instantiated directly.
+      Use :func:`RNS.Buffer.create_reader`,
+      :func:`RNS.Buffer.create_writer`, and
+      :func:`RNS.Buffer.create_bidirectional_buffer` functions
+      to create buffered streams with optional callbacks.
+
+      For additional information on the API of this
+      object, see the Python documentation for
+      ``RawIOBase``.
+    """
+    def __init__(self, stream_id: int, channel: Channel):
+        """
+        Create a raw channel reader.
+
+        :param stream_id: local stream id to receive at
+        :param channel: ``Channel`` object to receive from
+        """
+        self._stream_id = stream_id
+        self._channel = channel
+        self._lock = RLock()
+        self._buffer = bytearray()
+        self._eof = False
+        self._channel._register_message_type(StreamDataMessage, is_system_type=True)
+        self._channel.add_message_handler(self._handle_message)
+        self._listeners: [Callable[[int], None]] = []
+
+    def add_ready_callback(self, cb: Callable[[int], None]):
+        """
+        Add a function to be called when new data is available.
+        The function should have the signature ``(ready_bytes: int) -> None``
+
+        :param cb: function to call
+        """
+        with self._lock:
+            self._listeners.append(cb)
+
+    def remove_ready_callback(self, cb: Callable[[int], None]):
+        """
+        Remove a function added with :func:`RNS.RawChannelReader.add_ready_callback()`
+
+        :param cb: function to remove
+        """
+        with self._lock:
+            self._listeners.remove(cb)
+
+    def _handle_message(self, message: MessageBase):
+        if isinstance(message, StreamDataMessage):
+            if message.stream_id == self._stream_id:
+                with self._lock:
+                    if message.data is not None:
+                        self._buffer.extend(message.data)
+                    if message.eof:
+                        self._eof = True
+                    for listener in self._listeners:
+                        try:
+                            threading.Thread(target=listener, name="Message Callback", args=[len(self._buffer)], daemon=True).start()
+                        except Exception as ex:
+                            RNS.log("Error calling RawChannelReader(" + str(self._stream_id) + ") callback: " + str(ex), RNS.LOG_ERROR)
+                    return True
+        return False
+
+    def _read(self, __size: int) -> bytes | None:
+        with self._lock:
+            result = self._buffer[:__size]
+            self._buffer = self._buffer[__size:]
+            return result if len(result) > 0 or self._eof else None
+
+    def readinto(self, __buffer: bytearray) -> int | None:
+        ready = self._read(len(__buffer))
+        if ready is not None:
+            __buffer[:len(ready)] = ready
+        return len(ready) if ready is not None else None
+
+    def writable(self) -> bool:
+        return False
+
+    def seekable(self) -> bool:
+        return False
+
+    def readable(self) -> bool:
+        return True
+
+    def close(self):
+        with self._lock:
+            self._channel.remove_message_handler(self._handle_message)
+            self._listeners.clear()
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.close()
+        return False
+
+
+class RawChannelWriter(RawIOBase, AbstractContextManager):
+    """
+    An implementation of RawIOBase that receives
+    binary stream data sent over a channel.
+
+      This class generally need not be instantiated directly.
+      Use :func:`RNS.Buffer.create_reader`,
+      :func:`RNS.Buffer.create_writer`, and
+      :func:`RNS.Buffer.create_bidirectional_buffer` functions
+      to create buffered streams with optional callbacks.
+
+      For additional information on the API of this
+      object, see the Python documentation for
+      ``RawIOBase``.
+    """
+
+    MAX_CHUNK_LEN     = 1024*16
+    COMPRESSION_TRIES = 4
+
+    def __init__(self, stream_id: int, channel: Channel):
+        """
+        Create a raw channel writer.
+
+        :param stream_id: remote stream id to sent do
+        :param channel: ``Channel`` object to send on
+        """
+        self._stream_id = stream_id
+        self._channel = channel
+        self._eof = False
+        self._mdu = channel.mdu - StreamDataMessage.OVERHEAD
+
+    def write(self, __b: bytes) -> int | None:
+        try:
+            comp_tries = RawChannelWriter.COMPRESSION_TRIES
+            comp_try = 1
+            comp_success = False
+            chunk_len = len(__b)
+            if chunk_len > RawChannelWriter.MAX_CHUNK_LEN:
+                chunk_len = RawChannelWriter.MAX_CHUNK_LEN
+                __b = __b[:RawChannelWriter.MAX_CHUNK_LEN]
+            chunk_segment = None
+            while chunk_len > 32 and comp_try < comp_tries:
+                chunk_segment_length = int(chunk_len/comp_try)
+                compressed_chunk = bz2.compress(__b[:chunk_segment_length])
+                compressed_length = len(compressed_chunk)
+                if compressed_length < StreamDataMessage.MAX_DATA_LEN and compressed_length < chunk_segment_length:
+                    comp_success = True
+                    break
+                else:
+                    comp_try += 1
+
+            if comp_success:
+                chunk = compressed_chunk
+                processed_length = chunk_segment_length
+            else:
+                chunk = bytes(__b[:StreamDataMessage.MAX_DATA_LEN])
+                processed_length = len(chunk)
+
+            message = StreamDataMessage(self._stream_id, chunk, self._eof, comp_success)
+            
+            self._channel.send(message)
+            return processed_length
+
+        except RNS.Channel.ChannelException as cex:
+            if cex.type != RNS.Channel.CEType.ME_LINK_NOT_READY:
+                raise
+        return 0
+
+    def close(self):
+        try:
+            link_rtt = self._channel._outlet.link.rtt
+            timeout = time.time() + (link_rtt * len(self._channel._tx_ring) * 1)
+        except Exception as e:
+            timeout = time.time() + 15
+
+        while time.time() < timeout and not self._channel.is_ready_to_send():
+            time.sleep(0.05)
+
+        self._eof = True
+        self.write(bytes())
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.close()
+        return False
+
+    def seekable(self) -> bool:
+        return False
+
+    def readable(self) -> bool:
+        return False
+
+    def writable(self) -> bool:
+        return True
+
+class Buffer:
+    """
+    Static functions for creating buffered streams that send
+    and receive over a ``Channel``.
+
+    These functions use ``BufferedReader``, ``BufferedWriter``,
+    and ``BufferedRWPair`` to add buffering to
+    ``RawChannelReader`` and ``RawChannelWriter``.
+    """
+    @staticmethod
+    def create_reader(stream_id: int, channel: Channel,
+                      ready_callback: Callable[[int], None] | None = None) -> BufferedReader:
+        """
+        Create a buffered reader that reads binary data sent
+        over a ``Channel``, with an optional callback when
+        new data is available.
+
+        Callback signature: ``(ready_bytes: int) -> None``
+
+        For more information on the reader-specific functions
+        of this object, see the Python documentation for
+        ``BufferedReader``
+
+        :param stream_id: the local stream id to receive from
+        :param channel: the channel to receive on
+        :param ready_callback: function to call when new data is available
+        :return: a BufferedReader object
+        """
+        reader = RawChannelReader(stream_id, channel)
+        if ready_callback:
+            reader.add_ready_callback(ready_callback)
+        return BufferedReader(reader)
+
+    @staticmethod
+    def create_writer(stream_id: int, channel: Channel) -> BufferedWriter:
+        """
+        Create a buffered writer that writes binary data over
+        a ``Channel``.
+
+        For more information on the writer-specific functions
+        of this object, see the Python documentation for
+        ``BufferedWriter``
+
+        :param stream_id: the remote stream id to send to
+        :param channel: the channel to send on
+        :return: a BufferedWriter object
+        """
+        writer = RawChannelWriter(stream_id, channel)
+        return BufferedWriter(writer)
+
+    @staticmethod
+    def create_bidirectional_buffer(receive_stream_id: int, send_stream_id: int, channel: Channel,
+                                    ready_callback: Callable[[int], None] | None = None) -> BufferedRWPair:
+        """
+        Create a buffered reader/writer pair that reads and
+        writes binary data over a ``Channel``, with an
+        optional callback when new data is available.
+
+        Callback signature: ``(ready_bytes: int) -> None``
+
+        For more information on the reader-specific functions
+        of this object, see the Python documentation for
+        ``BufferedRWPair``
+
+        :param receive_stream_id: the local stream id to receive at
+        :param send_stream_id:  the remote stream id to send to
+        :param channel: the channel to send and receive on
+        :param ready_callback: function to call when new data is available
+        :return: a BufferedRWPair object
+        """
+        reader = RawChannelReader(receive_stream_id, channel)
+        if ready_callback:
+            reader.add_ready_callback(ready_callback)
+        writer = RawChannelWriter(send_stream_id, channel)
+        return BufferedRWPair(reader, writer)

RNS/Channel.py

@@ -0,0 +1,697 @@
+# MIT License
+#
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io and contributors.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from __future__ import annotations
+import collections
+import enum
+import threading
+import time
+from types import TracebackType
+from typing import Type, Callable, TypeVar, Generic, NewType
+import abc
+import contextlib
+import struct
+import RNS
+from abc import ABC, abstractmethod
+TPacket = TypeVar("TPacket")
+
+class SystemMessageTypes(enum.IntEnum):
+    SMT_STREAM_DATA = 0xff00
+
+class ChannelOutletBase(ABC, Generic[TPacket]):
+    """
+    An abstract transport layer interface used by Channel.
+
+    DEPRECATED: This was created for testing; eventually
+    Channel will use Link or a LinkBase interface
+    directly.
+    """
+    @abstractmethod
+    def send(self, raw: bytes) -> TPacket:
+        raise NotImplemented()
+
+    @abstractmethod
+    def resend(self, packet: TPacket) -> TPacket:
+        raise NotImplemented()
+
+    @property
+    @abstractmethod
+    def mdu(self):
+        raise NotImplemented()
+
+    @property
+    @abstractmethod
+    def rtt(self):
+        raise NotImplemented()
+
+    @property
+    @abstractmethod
+    def is_usable(self):
+        raise NotImplemented()
+
+    @abstractmethod
+    def get_packet_state(self, packet: TPacket) -> MessageState:
+        raise NotImplemented()
+
+    @abstractmethod
+    def timed_out(self):
+        raise NotImplemented()
+
+    @abstractmethod
+    def __str__(self):
+        raise NotImplemented()
+
+    @abstractmethod
+    def set_packet_timeout_callback(self, packet: TPacket, callback: Callable[[TPacket], None] | None,
+                                    timeout: float | None = None):
+        raise NotImplemented()
+
+    @abstractmethod
+    def set_packet_delivered_callback(self, packet: TPacket, callback: Callable[[TPacket], None] | None):
+        raise NotImplemented()
+
+    @abstractmethod
+    def get_packet_id(self, packet: TPacket) -> any:
+        raise NotImplemented()
+
+
+class CEType(enum.IntEnum):
+    """
+    ChannelException type codes
+    """
+    ME_NO_MSG_TYPE      = 0
+    ME_INVALID_MSG_TYPE = 1
+    ME_NOT_REGISTERED   = 2
+    ME_LINK_NOT_READY   = 3
+    ME_ALREADY_SENT     = 4
+    ME_TOO_BIG          = 5
+
+
+class ChannelException(Exception):
+    """
+    An exception thrown by Channel, with a type code.
+    """
+    def __init__(self, ce_type: CEType, *args):
+        super().__init__(args)
+        self.type = ce_type
+
+
+class MessageState(enum.IntEnum):
+    """
+    Set of possible states for a Message
+    """
+    MSGSTATE_NEW       = 0
+    MSGSTATE_SENT      = 1
+    MSGSTATE_DELIVERED = 2
+    MSGSTATE_FAILED    = 3
+
+
+class MessageBase(abc.ABC):
+    """
+    Base type for any messages sent or received on a Channel.
+    Subclasses must define the two abstract methods as well as
+    the ``MSGTYPE`` class variable.
+    """
+    # MSGTYPE must be unique within all classes sent over a
+    # channel. Additionally, MSGTYPE > 0xf000 are reserved.
+    MSGTYPE = None
+    """
+    Defines a unique identifier for a message class.
+    
+    * Must be unique within all classes registered with a ``Channel``
+    * Must be less than ``0xf000``. Values greater than or equal to ``0xf000`` are reserved.
+    """
+
+    @abstractmethod
+    def pack(self) -> bytes:
+        """
+        Create and return the binary representation of the message
+
+        :return: binary representation of message
+        """
+        raise NotImplemented()
+
+    @abstractmethod
+    def unpack(self, raw: bytes):
+        """
+        Populate message from binary representation
+
+        :param raw: binary representation
+        """
+        raise NotImplemented()
+
+
+MessageCallbackType = NewType("MessageCallbackType", Callable[[MessageBase], bool])
+
+
+class Envelope:
+    """
+    Internal wrapper used to transport messages over a channel and
+    track its state within the channel framework.
+    """
+    def unpack(self, message_factories: dict[int, Type]) -> MessageBase:
+        msgtype, self.sequence, length = struct.unpack(">HHH", self.raw[:6])
+        raw = self.raw[6:]
+        ctor = message_factories.get(msgtype, None)
+        if ctor is None:
+            raise ChannelException(CEType.ME_NOT_REGISTERED, f"Unable to find constructor for Channel MSGTYPE {hex(msgtype)}")
+        message = ctor()
+        message.unpack(raw)
+        self.unpacked = True
+        self.message = message
+
+        return message
+
+    def pack(self) -> bytes:
+        if self.message.__class__.MSGTYPE is None:
+            raise ChannelException(CEType.ME_NO_MSG_TYPE, f"{self.message.__class__} lacks MSGTYPE")
+        data = self.message.pack()
+        self.raw = struct.pack(">HHH", self.message.MSGTYPE, self.sequence, len(data)) + data
+        self.packed = True
+        return self.raw
+
+    def __init__(self, outlet: ChannelOutletBase, message: MessageBase = None, raw: bytes = None, sequence: int = None):
+        self.ts = time.time()
+        self.id = id(self)
+        self.message = message
+        self.raw = raw
+        self.packet: TPacket = None
+        self.sequence = sequence
+        self.outlet = outlet
+        self.tries = 0
+        self.unpacked = False
+        self.packed = False
+        self.tracked = False
+
+
+class Channel(contextlib.AbstractContextManager):
+    """
+    Provides reliable delivery of messages over
+    a link.
+
+    ``Channel`` differs from ``Request`` and
+    ``Resource`` in some important ways:
+
+     **Continuous**
+        Messages can be sent or received as long as
+        the ``Link`` is open.
+     **Bi-directional**
+        Messages can be sent in either direction on
+        the ``Link``; neither end is the client or
+        server.
+     **Size-constrained**
+        Messages must be encoded into a single packet.
+
+    ``Channel`` is similar to ``Packet``, except that it
+    provides reliable delivery (automatic retries) as well
+    as a structure for exchanging several types of
+    messages over the ``Link``.
+
+    ``Channel`` is not instantiated directly, but rather
+    obtained from a ``Link`` with ``get_channel()``.
+    """
+
+    # The initial window size at channel setup
+    WINDOW     = 2
+
+    # Absolute minimum window size
+    WINDOW_MIN = 2
+    WINDOW_MIN_LIMIT_SLOW    = 2
+    WINDOW_MIN_LIMIT_MEDIUM  = 5
+    WINDOW_MIN_LIMIT_FAST    = 16
+
+    # The maximum window size for transfers on slow links
+    WINDOW_MAX_SLOW      = 5
+
+    # The maximum window size for transfers on mid-speed links
+    WINDOW_MAX_MEDIUM    = 12
+
+    # The maximum window size for transfers on fast links
+    WINDOW_MAX_FAST      = 48
+    
+    # For calculating maps and guard segments, this
+    # must be set to the global maximum window.
+    WINDOW_MAX           = WINDOW_MAX_FAST
+    
+    # If the fast rate is sustained for this many request
+    # rounds, the fast link window size will be allowed.
+    FAST_RATE_THRESHOLD  = 10
+
+    # If the RTT rate is higher than this value,
+    # the max window size for fast links will be used.
+    RTT_FAST            = 0.18
+    RTT_MEDIUM          = 0.75
+    RTT_SLOW            = 1.45
+
+    # The minimum allowed flexibility of the window size.
+    # The difference between window_max and window_min
+    # will never be smaller than this value.
+    WINDOW_FLEXIBILITY   = 4
+
+    SEQ_MAX     = 0xFFFF
+    SEQ_MODULUS = SEQ_MAX+1
+
+    def __init__(self, outlet: ChannelOutletBase):
+        """
+
+        @param outlet:
+        """
+        self._outlet = outlet
+        self._lock = threading.RLock()
+        self._tx_ring: collections.deque[Envelope] = collections.deque()
+        self._rx_ring: collections.deque[Envelope] = collections.deque()
+        self._message_callbacks: [MessageCallbackType] = []
+        self._next_sequence = 0
+        self._next_rx_sequence = 0
+        self._message_factories: dict[int, Type[MessageBase]] = {}
+        self._max_tries = 5
+        self.fast_rate_rounds    = 0
+        self.medium_rate_rounds  = 0
+
+        if self._outlet.rtt > Channel.RTT_SLOW:
+            self.window              = 1
+            self.window_max          = 1
+            self.window_min          = 1
+            self.window_flexibility  = 1
+        else:
+            self.window              = Channel.WINDOW
+            self.window_max          = Channel.WINDOW_MAX_SLOW
+            self.window_min          = Channel.WINDOW_MIN
+            self.window_flexibility  = Channel.WINDOW_FLEXIBILITY
+
+    def __enter__(self) -> Channel:
+        return self
+
+    def __exit__(self, __exc_type: Type[BaseException] | None, __exc_value: BaseException | None,
+                 __traceback: TracebackType | None) -> bool | None:
+        self._shutdown()
+        return False
+
+    def register_message_type(self, message_class: Type[MessageBase]):
+        """
+        Register a message class for reception over a ``Channel``.
+
+        Message classes must extend ``MessageBase``.
+
+        :param message_class: Class to register
+        """
+        self._register_message_type(message_class, is_system_type=False)
+
+    def _register_message_type(self, message_class: Type[MessageBase], *, is_system_type: bool = False):
+        with self._lock:
+            if not issubclass(message_class, MessageBase):
+                raise ChannelException(CEType.ME_INVALID_MSG_TYPE,
+                                       f"{message_class} is not a subclass of {MessageBase}.")
+            if message_class.MSGTYPE is None:
+                raise ChannelException(CEType.ME_INVALID_MSG_TYPE,
+                                       f"{message_class} has invalid MSGTYPE class attribute.")
+            if message_class.MSGTYPE >= 0xf000 and not is_system_type:
+                raise ChannelException(CEType.ME_INVALID_MSG_TYPE,
+                                       f"{message_class} has system-reserved message type.")
+            try:
+                message_class()
+            except Exception as ex:
+                raise ChannelException(CEType.ME_INVALID_MSG_TYPE,
+                                       f"{message_class} raised an exception when constructed with no arguments: {ex}")
+
+            self._message_factories[message_class.MSGTYPE] = message_class
+
+    def add_message_handler(self, callback: MessageCallbackType):
+        """
+        Add a handler for incoming messages. A handler
+        has the following signature:
+
+        ``(message: MessageBase) -> bool``
+
+        Handlers are processed in the order they are
+        added. If any handler returns True, processing
+        of the message stops; handlers after the
+        returning handler will not be called.
+
+        :param callback: Function to call
+        """
+        with self._lock:
+            if callback not in self._message_callbacks:
+                self._message_callbacks.append(callback)
+
+    def remove_message_handler(self, callback: MessageCallbackType):
+        """
+        Remove a handler added with ``add_message_handler``.
+
+        :param callback: handler to remove
+        """
+        with self._lock:
+            if callback in self._message_callbacks:
+                self._message_callbacks.remove(callback)
+
+    def _shutdown(self):
+        with self._lock:
+            self._message_callbacks.clear()
+            self._clear_rings()
+
+    def _clear_rings(self):
+        with self._lock:
+            for envelope in self._tx_ring:
+                if envelope.packet is not None:
+                    self._outlet.set_packet_timeout_callback(envelope.packet, None)
+                    self._outlet.set_packet_delivered_callback(envelope.packet, None)
+            self._tx_ring.clear()
+            self._rx_ring.clear()
+
+    def _emplace_envelope(self, envelope: Envelope, ring: collections.deque[Envelope]) -> bool:
+        with self._lock:
+            i = 0
+            
+            for existing in ring:
+
+                if envelope.sequence == existing.sequence:
+                    RNS.log(f"Envelope: Emplacement of duplicate envelope with sequence "+str(envelope.sequence), RNS.LOG_EXTREME)
+                    return False
+                
+                if envelope.sequence < existing.sequence and not (self._next_rx_sequence - envelope.sequence) > (Channel.SEQ_MAX//2):
+                    ring.insert(i, envelope)
+
+                    envelope.tracked = True
+                    return True
+                
+                i += 1
+            
+            envelope.tracked = True
+            ring.append(envelope)
+
+            return True
+
+    def _run_callbacks(self, message: MessageBase):
+        cbs = self._message_callbacks.copy()
+
+        for cb in cbs:
+            try:
+                if cb(message):
+                    return
+            except Exception as e:
+                RNS.log("Channel "+str(self)+" experienced an error while running a message callback. The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+    def _receive(self, raw: bytes):
+        try:
+            envelope = Envelope(outlet=self._outlet, raw=raw)
+            with self._lock:
+                message = envelope.unpack(self._message_factories)
+
+                if envelope.sequence < self._next_rx_sequence:
+                    window_overflow = (self._next_rx_sequence+Channel.WINDOW_MAX) % Channel.SEQ_MODULUS
+                    if window_overflow < self._next_rx_sequence:
+                        if envelope.sequence > window_overflow:
+                            RNS.log("Invalid packet sequence ("+str(envelope.sequence)+") received on channel "+str(self), RNS.LOG_EXTREME)
+                            return
+                    else:
+                        RNS.log("Invalid packet sequence ("+str(envelope.sequence)+") received on channel "+str(self), RNS.LOG_EXTREME)
+                        return
+
+                is_new = self._emplace_envelope(envelope, self._rx_ring)
+
+            if not is_new:
+                RNS.log("Duplicate message received on channel "+str(self), RNS.LOG_EXTREME)
+                return
+            else:
+                with self._lock:
+                    contigous = []
+                    for e in self._rx_ring:
+                        if e.sequence == self._next_rx_sequence:
+                            contigous.append(e)
+                            self._next_rx_sequence = (self._next_rx_sequence + 1) % Channel.SEQ_MODULUS
+                            if self._next_rx_sequence == 0:
+                                for e in self._rx_ring:
+                                    if e.sequence == self._next_rx_sequence:
+                                        contigous.append(e)
+                                        self._next_rx_sequence = (self._next_rx_sequence + 1) % Channel.SEQ_MODULUS
+
+                    for e in contigous:
+                        if not e.unpacked:
+                            m = e.unpack(self._message_factories)
+                        else:
+                            m = e.message
+                            
+                        self._rx_ring.remove(e)
+                        self._run_callbacks(m)
+
+        except Exception as e:
+            RNS.log("An error ocurred while receiving data on "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+    def is_ready_to_send(self) -> bool:
+        """
+        Check if ``Channel`` is ready to send.
+
+        :return: True if ready
+        """
+        if not self._outlet.is_usable:
+            return False
+
+        with self._lock:
+            outstanding = 0
+            for envelope in self._tx_ring:
+                if envelope.outlet == self._outlet: 
+                    if not envelope.packet or not self._outlet.get_packet_state(envelope.packet) == MessageState.MSGSTATE_DELIVERED:
+                        outstanding += 1
+
+            if outstanding >= self.window:
+                return False
+
+        return True
+
+    def _packet_tx_op(self, packet: TPacket, op: Callable[[TPacket], bool]):
+        with self._lock:
+            envelope = next(filter(lambda e: self._outlet.get_packet_id(e.packet) == self._outlet.get_packet_id(packet),
+                                   self._tx_ring), None)
+
+            if envelope and op(envelope):
+                envelope.tracked = False
+                if envelope in self._tx_ring:
+                    self._tx_ring.remove(envelope)
+
+                    if self.window < self.window_max:
+                        self.window += 1
+
+                        # TODO: Remove at some point
+                        # RNS.log("Increased "+str(self)+" window to "+str(self.window), RNS.LOG_DEBUG)
+
+                    if self._outlet.rtt != 0:
+                        if self._outlet.rtt > Channel.RTT_FAST:
+                            self.fast_rate_rounds = 0
+
+                            if self._outlet.rtt > Channel.RTT_MEDIUM:
+                                self.medium_rate_rounds = 0
+
+                            else:
+                                self.medium_rate_rounds += 1
+                                if self.window_max < Channel.WINDOW_MAX_MEDIUM and self.medium_rate_rounds == Channel.FAST_RATE_THRESHOLD:
+                                    self.window_max = Channel.WINDOW_MAX_MEDIUM
+                                    self.window_min = Channel.WINDOW_MIN_LIMIT_MEDIUM
+                                    # TODO: Remove at some point
+                                    # RNS.log("Increased "+str(self)+" max window to "+str(self.window_max), RNS.LOG_DEBUG)
+                                    # RNS.log("Increased "+str(self)+" min window to "+str(self.window_min), RNS.LOG_DEBUG)
+                            
+                        else:
+                            self.fast_rate_rounds += 1
+                            if self.window_max < Channel.WINDOW_MAX_FAST and self.fast_rate_rounds == Channel.FAST_RATE_THRESHOLD:
+                                self.window_max = Channel.WINDOW_MAX_FAST
+                                self.window_min = Channel.WINDOW_MIN_LIMIT_FAST
+                                # TODO: Remove at some point
+                                # RNS.log("Increased "+str(self)+" max window to "+str(self.window_max), RNS.LOG_DEBUG)
+                                # RNS.log("Increased "+str(self)+" min window to "+str(self.window_min), RNS.LOG_DEBUG)
+
+
+                else:
+                    RNS.log("Envelope not found in TX ring for "+str(self), RNS.LOG_EXTREME)
+        if not envelope:
+            RNS.log("Spurious message received on "+str(self), RNS.LOG_EXTREME)
+
+    def _packet_delivered(self, packet: TPacket):
+        self._packet_tx_op(packet, lambda env: True)
+
+    def _update_packet_timeouts(self):
+        for envelope in self._tx_ring:
+            updated_timeout = self._get_packet_timeout_time(envelope.tries)
+            if envelope.packet and hasattr(envelope.packet, "receipt") and envelope.packet.receipt and envelope.packet.receipt.timeout:
+                if updated_timeout > envelope.packet.receipt.timeout:
+                    envelope.packet.receipt.set_timeout(updated_timeout)
+
+    def _get_packet_timeout_time(self, tries: int) -> float:
+        to = pow(1.5, tries - 1) * max(self._outlet.rtt*2.5, 0.025) * (len(self._tx_ring)+1.5)
+        return to
+
+    def _packet_timeout(self, packet: TPacket):
+        def retry_envelope(envelope: Envelope) -> bool:
+            if envelope.tries >= self._max_tries:
+                RNS.log("Retry count exceeded on "+str(self)+", tearing down Link.", RNS.LOG_ERROR)
+                self._shutdown()  # start on separate thread?
+                self._outlet.timed_out()
+                return True
+
+            envelope.tries += 1
+            self._outlet.resend(envelope.packet)
+            self._outlet.set_packet_delivered_callback(envelope.packet, self._packet_delivered)
+            self._outlet.set_packet_timeout_callback(envelope.packet, self._packet_timeout, self._get_packet_timeout_time(envelope.tries))
+            self._update_packet_timeouts()
+
+            if self.window > self.window_min:
+                self.window -= 1
+                # TODO: Remove at some point
+                # RNS.log("Decreased "+str(self)+" window to "+str(self.window), RNS.LOG_DEBUG)
+
+                if self.window_max > (self.window_min+self.window_flexibility):
+                    self.window_max -= 1
+                    # TODO: Remove at some point
+                    # RNS.log("Decreased "+str(self)+" max window to "+str(self.window_max), RNS.LOG_DEBUG)
+
+                # TODO: Remove at some point
+                # RNS.log("Decreased "+str(self)+" window to "+str(self.window), RNS.LOG_EXTREME)
+
+            return False
+
+        if self._outlet.get_packet_state(packet) != MessageState.MSGSTATE_DELIVERED:
+            self._packet_tx_op(packet, retry_envelope)
+
+    def send(self, message: MessageBase) -> Envelope:
+        """
+        Send a message. If a message send is attempted and
+        ``Channel`` is not ready, an exception is thrown.
+
+        :param message: an instance of a ``MessageBase`` subclass
+        """
+        envelope: Envelope | None = None
+        with self._lock:
+            if not self.is_ready_to_send():
+                raise ChannelException(CEType.ME_LINK_NOT_READY, f"Link is not ready")
+        
+            envelope = Envelope(self._outlet, message=message, sequence=self._next_sequence)
+            self._next_sequence = (self._next_sequence + 1) % Channel.SEQ_MODULUS
+            self._emplace_envelope(envelope, self._tx_ring)
+
+        if envelope is None:
+            raise BlockingIOError()
+
+        envelope.pack()
+        if len(envelope.raw) > self._outlet.mdu:
+            raise ChannelException(CEType.ME_TOO_BIG, f"Packed message too big for packet: {len(envelope.raw)} > {self._outlet.mdu}")
+        
+        envelope.packet = self._outlet.send(envelope.raw)
+        envelope.tries += 1
+        self._outlet.set_packet_delivered_callback(envelope.packet, self._packet_delivered)
+        self._outlet.set_packet_timeout_callback(envelope.packet, self._packet_timeout, self._get_packet_timeout_time(envelope.tries))
+        self._update_packet_timeouts()
+
+        return envelope
+
+    @property
+    def mdu(self):
+        """
+        Maximum Data Unit: the number of bytes available
+        for a message to consume in a single send. This
+        value is adjusted from the ``Link`` MDU to accommodate
+        message header information.
+
+        :return: number of bytes available
+        """
+        mdu = self._outlet.mdu - 6  # sizeof(msgtype) + sizeof(length) + sizeof(sequence)
+        if mdu > 0xFFFF:
+            mdu = 0xFFFF
+        return mdu
+
+
+class LinkChannelOutlet(ChannelOutletBase):
+    """
+    An implementation of ChannelOutletBase for RNS.Link.
+    Allows Channel to send packets over an RNS Link with
+    Packets.
+
+    :param link: RNS Link to wrap
+    """
+    def __init__(self, link: RNS.Link):
+        self.link = link
+
+    def send(self, raw: bytes) -> RNS.Packet:
+        packet = RNS.Packet(self.link, raw, context=RNS.Packet.CHANNEL)
+        if self.link.status == RNS.Link.ACTIVE:
+            packet.send()
+        return packet
+
+    def resend(self, packet: RNS.Packet) -> RNS.Packet:
+        receipt = packet.resend()
+        if not receipt:
+            RNS.log("Failed to resend packet", RNS.LOG_ERROR)
+        return packet
+
+    @property
+    def mdu(self):
+        return self.link.mdu
+
+    @property
+    def rtt(self):
+        return self.link.rtt
+
+    @property
+    def is_usable(self):
+        return True  # had issues looking at Link.status
+
+    def get_packet_state(self, packet: TPacket) -> MessageState:
+        if packet.receipt == None:
+            return MessageState.MSGSTATE_FAILED
+
+        status = packet.receipt.get_status()
+        if status == RNS.PacketReceipt.SENT:
+            return MessageState.MSGSTATE_SENT
+        if status == RNS.PacketReceipt.DELIVERED:
+            return MessageState.MSGSTATE_DELIVERED
+        if status == RNS.PacketReceipt.FAILED:
+            return MessageState.MSGSTATE_FAILED
+        else:
+            raise Exception(f"Unexpected receipt state: {status}")
+
+    def timed_out(self):
+        self.link.teardown()
+
+    def __str__(self):
+        return f"{self.__class__.__name__}({self.link})"
+
+    def set_packet_timeout_callback(self, packet: RNS.Packet, callback: Callable[[RNS.Packet], None] | None,
+                                    timeout: float | None = None):
+        if timeout and packet.receipt:
+            packet.receipt.set_timeout(timeout)
+
+        def inner(receipt: RNS.PacketReceipt):
+            callback(packet)
+
+        if packet and packet.receipt:
+            packet.receipt.set_timeout_callback(inner if callback else None)
+
+    def set_packet_delivered_callback(self, packet: RNS.Packet, callback: Callable[[RNS.Packet], None] | None):
+        def inner(receipt: RNS.PacketReceipt):
+            callback(packet)
+
+        if packet and packet.receipt:
+            packet.receipt.set_delivery_callback(inner if callback else None)
+
+    def get_packet_id(self, packet: RNS.Packet) -> any:
+        if packet and hasattr(packet, "get_hash") and callable(packet.get_hash):
+            return packet.get_hash()
+        else:
+            return None

RNS/Cryptography/HKDF.py

@@ -33,15 +33,12 @@ def hkdf(length=None, derive_from=None, salt=None, context=None):
     if length == None or length < 1:
         raise ValueError("Invalid output key length")
 
-    if derive_from == "None" or derive_from == "":
+    if derive_from == None or derive_from == "":
         raise ValueError("Cannot derive key from empty input material")
 
     if salt == None or len(salt) == 0:
         salt = bytes([0] * hash_len)
 
-    if salt == None:
-        salt = b""
-
     if context == None:
         context = b""
 
@@ -51,7 +48,7 @@ def hkdf(length=None, derive_from=None, salt=None, context=None):
     derived = b""
 
     for i in range(ceil(length / hash_len)):
-        block = hmac_sha256(pseudorandom_key, block + context + bytes([i + 1]))
+        block = hmac_sha256(pseudorandom_key, block + context + bytes([(i + 1)%(0xFF+1)]))
         derived += block
 
-    return derived[:length]
+    return derived[:length]

RNS/Cryptography/Token.py

@@ -27,7 +27,7 @@ from RNS.Cryptography import HMAC
 from RNS.Cryptography import PKCS7
 from RNS.Cryptography.AES import AES_128_CBC
 
-class Fernet():
+class Token():
     """
     This class provides a slightly modified implementation of the Fernet spec
     found at: https://github.com/fernet/spec/blob/master/Spec.md
@@ -37,7 +37,7 @@ class Fernet():
     not relevant to Reticulum. They are therefore stripped from this
     implementation, since they incur overhead and leak initiator metadata.
     """
-    FERNET_OVERHEAD  = 48 # Bytes
+    TOKEN_OVERHEAD  = 48 # Bytes
 
     @staticmethod
     def generate_key():
@@ -45,10 +45,10 @@ class Fernet():
 
     def __init__(self, key = None):
         if key == None:
-            raise ValueError("Fernet key cannot be None")
+            raise ValueError("Token key cannot be None")
 
         if len(key) != 32:
-            raise ValueError("Fernet key must be 32 bytes, not "+str(len(key)))
+            raise ValueError("Token key must be 32 bytes, not "+str(len(key)))
             
         self._signing_key = key[:16]
         self._encryption_key = key[16:]
@@ -72,7 +72,7 @@ class Fernet():
         current_time = int(time.time())
 
         if not isinstance(data, bytes):
-            raise TypeError("Fernet token plaintext input must be bytes")
+            raise TypeError("Token plaintext input must be bytes")
 
         ciphertext = AES_128_CBC.encrypt(
             plaintext = PKCS7.pad(data),
@@ -87,10 +87,10 @@ class Fernet():
 
     def decrypt(self, token = None):
         if not isinstance(token, bytes):
-            raise TypeError("Fernet token must be bytes")
+            raise TypeError("Token must be bytes")
 
         if not self.verify_hmac(token):
-            raise ValueError("Fernet token HMAC was invalid")
+            raise ValueError("Token HMAC was invalid")
 
         iv = token[:16]
         ciphertext = token[16:-32]
@@ -107,4 +107,4 @@ class Fernet():
             return plaintext
 
         except Exception as e:
-            raise ValueError("Could not decrypt Fernet token")
+            raise ValueError("Could not decrypt token")

RNS/Cryptography/__init__.py

@@ -5,7 +5,7 @@ from .Hashes import sha256
 from .Hashes import sha512
 from .HKDF import hkdf
 from .PKCS7 import PKCS7
-from .Fernet import Fernet
+from .Token import Token
 from .Provider import backend
 
 import RNS.Cryptography.Provider as cp
@@ -20,5 +20,7 @@ elif cp.PROVIDER == cp.PROVIDER_PYCA:
     from RNS.Cryptography.Proxies import Ed25519PrivateKeyProxy as Ed25519PrivateKey
     from RNS.Cryptography.Proxies import Ed25519PublicKeyProxy as Ed25519PublicKey
 
-modules = glob.glob(os.path.dirname(__file__)+"/*.py")
-__all__ = [ os.path.basename(f)[:-3] for f in modules if not f.endswith('__init__.py')]
+py_modules  = glob.glob(os.path.dirname(__file__)+"/*.py")
+pyc_modules = glob.glob(os.path.dirname(__file__)+"/*.pyc")
+modules     = py_modules+pyc_modules
+__all__ = list(set([os.path.basename(f).replace(".pyc", "").replace(".py", "") for f in modules if not (f.endswith("__init__.py") or f.endswith("__init__.pyc"))]))

RNS/Destination.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2024 Mark Qvist / unsigned.io and contributors
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -20,11 +20,14 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
+import os
 import math
 import time
+import threading
 import RNS
 
-from RNS.Cryptography import Fernet
+from RNS.Cryptography import Token
+from .vendor import umsgpack as umsgpack
 
 class Callbacks:
     def __init__(self):
@@ -38,14 +41,14 @@ class Destination:
     instances are used both to create outgoing and incoming endpoints. The
     destination type will decide if encryption, and what type, is used in
     communication with the endpoint. A destination can also announce its
-    presence on the network, which will also distribute necessary keys for
+    presence on the network, which will distribute necessary keys for
     encrypted communication with it.
 
     :param identity: An instance of :ref:`RNS.Identity<api-identity>`. Can hold only public keys for an outgoing destination, or holding private keys for an ingoing.
     :param direction: ``RNS.Destination.IN`` or ``RNS.Destination.OUT``.
     :param type: ``RNS.Destination.SINGLE``, ``RNS.Destination.GROUP`` or ``RNS.Destination.PLAIN``.
     :param app_name: A string specifying the app name.
-    :param \*aspects: Any non-zero number of string arguments.
+    :param \\*aspects: Any non-zero number of string arguments.
     """
 
     # Constants
@@ -69,8 +72,20 @@ class Destination:
     OUT        = 0x12;
     directions = [IN, OUT]
 
+    PR_TAG_WINDOW = 30
+
+    RATCHET_COUNT    = 512
+    """
+    The default number of generated ratchet keys a destination will retain, if it has ratchets enabled.
+    """
+
+    RATCHET_INTERVAL = 30*60
+    """
+    The minimum interval between rotating ratchet keys, in seconds.
+    """
+
     @staticmethod
-    def full_name(app_name, *aspects):
+    def expand_name(identity, app_name, *aspects):
         """
         :returns: A string containing the full human-readable name of the destination, for an app_name and a number of aspects.
         """
@@ -81,20 +96,30 @@ class Destination:
         name = app_name
         for aspect in aspects:
             if "." in aspect: raise ValueError("Dots can't be used in aspects")
-            name = name + "." + aspect
+            name += "." + aspect
+
+        if identity != None:
+            name += "." + identity.hexhash
 
         return name
 
 
     @staticmethod
-    def hash(app_name, *aspects):
+    def hash(identity, app_name, *aspects):
         """
         :returns: A destination name in adressable hash form, for an app_name and a number of aspects.
         """
-        name = Destination.full_name(app_name, *aspects)
+        name_hash = RNS.Identity.full_hash(Destination.expand_name(None, app_name, *aspects).encode("utf-8"))[:(RNS.Identity.NAME_HASH_LENGTH//8)]
+        addr_hash_material = name_hash
+        if identity != None:
+            if isinstance(identity, RNS.Identity):
+                addr_hash_material += identity.hash
+            elif isinstance(identity, bytes) and len(identity) == RNS.Reticulum.TRUNCATED_HASHLENGTH//8:
+                addr_hash_material += identity
+            else:
+                raise TypeError("Invalid material supplied for destination hash calculation")
 
-        # Create a digest for the destination
-        return RNS.Identity.full_hash(name.encode("utf-8"))[:RNS.Reticulum.TRUNCATED_HASHLENGTH//8]
+        return RNS.Identity.full_hash(addr_hash_material)[:RNS.Reticulum.TRUNCATED_HASHLENGTH//8]
 
     @staticmethod
     def app_and_aspects_from_name(full_name):
@@ -110,8 +135,8 @@ class Destination:
         :returns: A destination name in adressable hash form, for a full name string and Identity instance.
         """
         app_name, aspects = Destination.app_and_aspects_from_name(full_name)
-        aspects.append(identity.hexhash)
-        return Destination.hash(app_name, *aspects)
+
+        return Destination.hash(identity, app_name, *aspects)
 
     def __init__(self, identity, direction, type, app_name, *aspects):
         # Check input values and build name string
@@ -125,27 +150,38 @@ class Destination:
         self.type = type
         self.direction = direction
         self.proof_strategy = Destination.PROVE_NONE
+        self.ratchets = None
+        self.ratchets_path = None
+        self.ratchet_interval = Destination.RATCHET_INTERVAL
+        self.ratchet_file_lock = threading.Lock()
+        self.retained_ratchets = Destination.RATCHET_COUNT
+        self.latest_ratchet_time = None
+        self.latest_ratchet_id = None
+        self.__enforce_ratchets = False
         self.mtu = 0
 
+        self.path_responses = {}
         self.links = []
 
-        if identity != None and type == Destination.SINGLE:
-            aspects = aspects+(identity.hexhash,)
-
         if identity == None and direction == Destination.IN and self.type != Destination.PLAIN:
             identity = RNS.Identity()
             aspects = aspects+(identity.hexhash,)
 
+        if identity == None and direction == Destination.OUT and self.type != Destination.PLAIN:
+            raise ValueError("Can't create outbound SINGLE destination without an identity")
+
         if identity != None and self.type == Destination.PLAIN:
             raise TypeError("Selected destination type PLAIN cannot hold an identity")
 
         self.identity = identity
+        self.name = Destination.expand_name(identity, app_name, *aspects)
 
-        self.name = Destination.full_name(app_name, *aspects)      
-        self.hash = Destination.hash(app_name, *aspects)
+        # Generate the destination address hash
+        self.hash = Destination.hash(self.identity, app_name, *aspects)
+        self.name_hash = RNS.Identity.full_hash(self.expand_name(None, app_name, *aspects).encode("utf-8"))[:(RNS.Identity.NAME_HASH_LENGTH//8)]
         self.hexhash = self.hash.hex()
-        self.default_app_data = None
 
+        self.default_app_data = None
         self.callback = None
         self.proofcallback = None
 
@@ -158,8 +194,41 @@ class Destination:
         """
         return "<"+self.name+"/"+self.hexhash+">"
 
+    def _clean_ratchets(self):
+        if self.ratchets != None:
+            if len (self.ratchets) > self.retained_ratchets:
+                self.ratchets = self.ratchets[:Destination.RATCHET_COUNT]
 
-    def announce(self, app_data=None, path_response=False):
+    def _persist_ratchets(self):
+        try:
+            with self.ratchet_file_lock:
+                packed_ratchets = umsgpack.packb(self.ratchets)
+                persisted_data = {"signature": self.sign(packed_ratchets), "ratchets": packed_ratchets}
+                ratchets_file = open(self.ratchets_path, "wb")
+                ratchets_file.write(umsgpack.packb(persisted_data))
+                ratchets_file.close()
+        except Exception as e:
+            self.ratchets = None
+            self.ratchets_path = None
+            raise OSError("Could not write ratchet file contents for "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+    def rotate_ratchets(self):
+        if self.ratchets != None:
+            now = time.time()
+            if now > self.latest_ratchet_time+self.ratchet_interval:
+                RNS.log("Rotating ratchets for "+str(self), RNS.LOG_DEBUG)
+                new_ratchet = RNS.Identity._generate_ratchet()
+                self.ratchets.insert(0, new_ratchet)
+                self.latest_ratchet_time = now
+                self._clean_ratchets()
+                self._persist_ratchets()
+            return True
+        else:
+            raise SystemError("Cannot rotate ratchet on "+str(self)+", ratchets are not enabled")
+
+        return False
+
+    def announce(self, app_data=None, path_response=False, attached_interface=None, tag=None, send=True):
         """
         Creates an announce packet for this destination and broadcasts it on all
         relevant interfaces. Application specific data can be added to the announce.
@@ -169,35 +238,78 @@ class Destination:
         """
         if self.type != Destination.SINGLE:
             raise TypeError("Only SINGLE destination types can be announced")
-            
-        destination_hash = self.hash
-        random_hash = RNS.Identity.get_random_hash()[0:5]+int(time.time()).to_bytes(5, "big")
 
-        if app_data == None and self.default_app_data != None:
-            if isinstance(self.default_app_data, bytes):
-                app_data = self.default_app_data
-            elif callable(self.default_app_data):
-                returned_app_data = self.default_app_data()
-                if isinstance(returned_app_data, bytes):
-                    app_data = returned_app_data
+        if self.direction != Destination.IN:
+            raise TypeError("Only IN destination types can be announced")
         
-        signed_data = self.hash+self.identity.get_public_key()+random_hash
-        if app_data != None:
-            signed_data += app_data
+        ratchet = b""
+        now = time.time()
+        stale_responses = []
+        for entry_tag in self.path_responses:
+            entry = self.path_responses[entry_tag]
+            if now > entry[0]+Destination.PR_TAG_WINDOW:
+                stale_responses.append(entry_tag)
 
-        signature = self.identity.sign(signed_data)
+        for entry_tag in stale_responses:
+            self.path_responses.pop(entry_tag)
 
-        announce_data = self.identity.get_public_key()+random_hash+signature
+        if (path_response == True and tag != None) and tag in self.path_responses:
+            # This code is currently not used, since Transport will block duplicate
+            # path requests based on tags. When multi-path support is implemented in
+            # Transport, this will allow Transport to detect redundant paths to the
+            # same destination, and select the best one based on chosen criteria,
+            # since it will be able to detect that a single emitted announce was
+            # received via multiple paths. The difference in reception time will
+            # potentially also be useful in determining characteristics of the
+            # multiple available paths, and to choose the best one.
+            RNS.log("Using cached announce data for answering path request with tag "+RNS.prettyhexrep(tag), RNS.LOG_EXTREME)
+            announce_data = self.path_responses[tag][1]
+        
+        else:
+            destination_hash = self.hash
+            random_hash = RNS.Identity.get_random_hash()[0:5]+int(time.time()).to_bytes(5, "big")
 
-        if app_data != None:
-            announce_data += app_data
+            if self.ratchets != None:
+                self.rotate_ratchets()
+                ratchet = RNS.Identity._ratchet_public_bytes(self.ratchets[0])
+                RNS.Identity._remember_ratchet(self.hash, ratchet)
+
+            if app_data == None and self.default_app_data != None:
+                if isinstance(self.default_app_data, bytes):
+                    app_data = self.default_app_data
+                elif callable(self.default_app_data):
+                    returned_app_data = self.default_app_data()
+                    if isinstance(returned_app_data, bytes):
+                        app_data = returned_app_data
+            
+            signed_data = self.hash+self.identity.get_public_key()+self.name_hash+random_hash+ratchet
+            if app_data != None:
+                signed_data += app_data
+
+            signature = self.identity.sign(signed_data)
+            announce_data = self.identity.get_public_key()+self.name_hash+random_hash+ratchet+signature
+
+            if app_data != None:
+                announce_data += app_data
+
+            self.path_responses[tag] = [time.time(), announce_data]
 
         if path_response:
             announce_context = RNS.Packet.PATH_RESPONSE
         else:
             announce_context = RNS.Packet.NONE
 
-        RNS.Packet(self, announce_data, RNS.Packet.ANNOUNCE, context = announce_context).send()
+        if ratchet:
+            context_flag = RNS.Packet.FLAG_SET
+        else:
+            context_flag = RNS.Packet.FLAG_UNSET
+
+        announce_packet = RNS.Packet(self, announce_data, RNS.Packet.ANNOUNCE, context = announce_context,
+                                     attached_interface = attached_interface, context_flag=context_flag)
+        if send:
+            announce_packet.send()
+        else:
+            return announce_packet
 
     def accepts_links(self, accepts = None):
         """
@@ -253,13 +365,12 @@ class Destination:
         else:
             self.proof_strategy = proof_strategy
 
-
     def register_request_handler(self, path, response_generator = None, allow = ALLOW_NONE, allowed_list = None):
         """
         Registers a request handler.
 
         :param path: The path for the request handler to be registered.
-        :param response_generator: A function or method with the signature *response_generator(path, data, request_id, remote_identity, requested_at)* to be called. Whatever this funcion returns will be sent as a response to the requester. If the function returns ``None``, no response will be sent.
+        :param response_generator: A function or method with the signature *response_generator(path, data, request_id, link_id, remote_identity, requested_at)* to be called. Whatever this funcion returns will be sent as a response to the requester. If the function returns ``None``, no response will be sent.
         :param allow: One of ``RNS.Destination.ALLOW_NONE``, ``RNS.Destination.ALLOW_ALL`` or ``RNS.Destination.ALLOW_LIST``. If ``RNS.Destination.ALLOW_LIST`` is set, the request handler will only respond to requests for identified peers in the supplied list.
         :param allowed_list: A list of *bytes-like* :ref:`RNS.Identity<api-identity>` hashes.
         :raises: ``ValueError`` if any of the supplied arguments are invalid.
@@ -275,7 +386,6 @@ class Destination:
             request_handler = [path, response_generator, allow, allowed_list]
             self.request_handlers[path_hash] = request_handler
 
-
     def deregister_request_handler(self, path):
         """
         Deregisters a request handler.
@@ -290,14 +400,13 @@ class Destination:
         else:
             return False
 
-        
-
     def receive(self, packet):
         if packet.packet_type == RNS.Packet.LINKREQUEST:
             plaintext = packet.data
             self.incoming_link_request(plaintext, packet)
         else:
             plaintext = self.decrypt(packet.data)
+            packet.ratchet_id = self.latest_ratchet_id
             if plaintext != None:
                 if packet.packet_type == RNS.Packet.DATA:
                     if self.callbacks.packet != None:
@@ -306,13 +415,115 @@ class Destination:
                         except Exception as e:
                             RNS.log("Error while executing receive callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
 
-
     def incoming_link_request(self, data, packet):
         if self.accept_link_requests:
             link = RNS.Link.validate_request(self, data, packet)
             if link != None:
                 self.links.append(link)
 
+    def _reload_ratchets(self, ratchets_path):
+        if os.path.isfile(ratchets_path):
+            with self.ratchet_file_lock:
+                def load_attempt():
+                    ratchets_file = open(ratchets_path, "rb")
+                    persisted_data = umsgpack.unpackb(ratchets_file.read())
+                    if "signature" in persisted_data and "ratchets" in persisted_data:
+                        if self.identity.validate(persisted_data["signature"], persisted_data["ratchets"]):
+                            self.ratchets = umsgpack.unpackb(persisted_data["ratchets"])
+                            self.ratchets_path = ratchets_path
+                        else:
+                            raise KeyError("Invalid ratchet file signature")
+                
+                try:
+                    try:
+                        load_attempt()
+
+                    except Exception as e:
+                        RNS.trace_exception(e)
+                        RNS.log(f"First ratchet reload attempt for {self} failed. Possible I/O conflict. Retrying in 500ms.", RNS.LOG_ERROR)
+                        time.sleep(0.5)
+                        load_attempt()
+                        RNS.log(f"Ratchet reload retry succeeded", RNS.LOG_DEBUG)
+
+                except Exception as e:
+                    self.ratchets = None
+                    self.ratchets_path = None
+                    RNS.trace_exception(e)
+                    raise OSError("Could not read ratchet file contents for "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+        else:
+            RNS.log("No existing ratchet data found, initialising new ratchet file for "+str(self), RNS.LOG_DEBUG)
+            self.ratchets = []
+            self.ratchets_path = ratchets_path
+            self._persist_ratchets()
+
+    def enable_ratchets(self, ratchets_path):
+        """
+        Enables ratchets on the destination. When ratchets are enabled, Reticulum will automatically rotate
+        the keys used to encrypt packets to this destination, and include the latest ratchet key in announces.
+
+        Enabling ratchets on a destination will provide forward secrecy for packets sent to that destination,
+        even when sent outside a ``Link``. The normal Reticulum ``Link`` establishment procedure already performs
+        its own ephemeral key exchange for each link establishment, which means that ratchets are not necessary
+        to provide forward secrecy for links.
+
+        Enabling ratchets will have a small impact on announce size, adding 32 bytes to every sent announce.
+
+        :param ratchets_path: The path to a file to store ratchet data in.
+        :returns: True if the operation succeeded, otherwise False.
+        """
+        if ratchets_path != None:
+            self.latest_ratchet_time = 0
+            self._reload_ratchets(ratchets_path)
+
+            # TODO: Remove at some point
+            RNS.log("Ratchets enabled on "+str(self), RNS.LOG_DEBUG)
+            return True
+
+        else:
+            raise ValueError("No ratchet file path specified for "+str(self))
+
+    def enforce_ratchets(self):
+        """
+        When ratchet enforcement is enabled, this destination will never accept packets that use its
+        base Identity key for encryption, but only accept packets encrypted with one of the retained
+        ratchet keys.
+        """
+        if self.ratchets != None:
+            self.__enforce_ratchets = True
+            RNS.log("Ratchets enforced on "+str(self), RNS.LOG_DEBUG)
+            return True
+        else:
+            return False
+
+    def set_retained_ratchets(self, retained_ratchets):
+        """
+        Sets the number of previously generated ratchet keys this destination will retain,
+        and try to use when decrypting incoming packets. Defaults to ``Destination.RATCHET_COUNT``.
+
+        :param retained_ratchets: The number of generated ratchets to retain.
+        :returns: True if the operation succeeded, False if not.
+        """
+        if isinstance(retained_ratchets, int) and retained_ratchets > 0:
+            self.retained_ratchets = retained_ratchets
+            self._clean_ratchets()
+            return True
+        else:
+            return False
+
+    def set_ratchet_interval(self, interval):
+        """
+        Sets the minimum interval in seconds between ratchet key rotation.
+        Defaults to ``Destination.RATCHET_INTERVAL``.
+
+        :param interval: The minimum interval in seconds.
+        :returns: True if the operation succeeded, False if not.
+        """
+        if isinstance(interval, int) and interval > 0:
+            self.ratchet_interval = interval
+            return True
+        else:
+            return False
+
     def create_keys(self):
         """
         For a ``RNS.Destination.GROUP`` type destination, creates a new symmetric key.
@@ -326,9 +537,8 @@ class Destination:
             raise TypeError("A single destination holds keys through an Identity instance")
 
         if self.type == Destination.GROUP:
-            self.prv_bytes = Fernet.generate_key()
-            self.prv = Fernet(self.prv_bytes)
-
+            self.prv_bytes = Token.generate_key()
+            self.prv = Token(self.prv_bytes)
 
     def get_private_key(self):
         """
@@ -343,7 +553,6 @@ class Destination:
         else:
             return self.prv_bytes
 
-
     def load_private_key(self, key):
         """
         For a ``RNS.Destination.GROUP`` type destination, loads a symmetric private key.
@@ -359,7 +568,7 @@ class Destination:
 
         if self.type == Destination.GROUP:
             self.prv_bytes = key
-            self.prv = Fernet(self.prv_bytes)
+            self.prv = Token(self.prv_bytes)
 
     def load_public_key(self, key):
         if self.type != Destination.SINGLE:
@@ -367,7 +576,6 @@ class Destination:
         else:
             raise TypeError("A single destination holds keys through an Identity instance")
 
-
     def encrypt(self, plaintext):
         """
         Encrypts information for ``RNS.Destination.SINGLE`` or ``RNS.Destination.GROUP`` type destination.
@@ -379,7 +587,10 @@ class Destination:
             return plaintext
 
         if self.type == Destination.SINGLE and self.identity != None:
-            return self.identity.encrypt(plaintext)
+            selected_ratchet = RNS.Identity.get_ratchet(self.hash)
+            if selected_ratchet:
+                self.latest_ratchet_id = RNS.Identity._get_ratchet_id(selected_ratchet)
+            return self.identity.encrypt(plaintext, ratchet=selected_ratchet)
 
         if self.type == Destination.GROUP:
             if hasattr(self, "prv") and self.prv != None:
@@ -391,8 +602,6 @@ class Destination:
             else:
                 raise ValueError("No private key held by GROUP destination. Did you create or load one?")
 
-
-
     def decrypt(self, ciphertext):
         """
         Decrypts information for ``RNS.Destination.SINGLE`` or ``RNS.Destination.GROUP`` type destination.
@@ -404,7 +613,28 @@ class Destination:
             return ciphertext
 
         if self.type == Destination.SINGLE and self.identity != None:
-            return self.identity.decrypt(ciphertext)
+            if self.ratchets:
+                decrypted = None
+                try:
+                    decrypted = self.identity.decrypt(ciphertext, ratchets=self.ratchets, enforce_ratchets=self.__enforce_ratchets, ratchet_id_receiver=self)
+                except:
+                    decrypted = None
+
+                if not decrypted:
+                    try:
+                        RNS.log(f"Decryption with ratchets failed on {self}, reloading ratchets from storage and retrying", RNS.LOG_ERROR)
+                        self._reload_ratchets(self.ratchets_path)
+                        decrypted = self.identity.decrypt(ciphertext, ratchets=self.ratchets, enforce_ratchets=self.__enforce_ratchets, ratchet_id_receiver=self)
+                    except Exception as e:
+                        RNS.log(f"Decryption still failing after ratchet reload. The contained exception was: {e}", RNS.LOG_ERROR)
+                        raise e
+
+                    RNS.log("Decryption succeeded after ratchet reload", RNS.LOG_NOTICE)
+
+                return decrypted
+
+            else:
+                return self.identity.decrypt(ciphertext, ratchets=None, enforce_ratchets=self.__enforce_ratchets, ratchet_id_receiver=self)
 
         if self.type == Destination.GROUP:
             if hasattr(self, "prv") and self.prv != None:
@@ -416,7 +646,6 @@ class Destination:
             else:
                 raise ValueError("No private key held by GROUP destination. Did you create or load one?")
 
-
     def sign(self, message):
         """
         Signs information for ``RNS.Destination.SINGLE`` type destination.

RNS/Identity.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2024 Mark Qvist / unsigned.io and contributors.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -26,11 +26,12 @@ import RNS
 import time
 import atexit
 import hashlib
+import threading
 
 from .vendor import umsgpack as umsgpack
 
 from RNS.Cryptography import X25519PrivateKey, X25519PublicKey, Ed25519PrivateKey, Ed25519PublicKey
-from RNS.Cryptography import Fernet
+from RNS.Cryptography import Token
 
 
 class Identity:
@@ -49,15 +50,28 @@ class Identity:
 
     KEYSIZE     = 256*2
     """
-    X25519 key size in bits. A complete key is the concatenation of a 256 bit encryption key, and a 256 bit signing key.
-    """   
+    X.25519 key size in bits. A complete key is the concatenation of a 256 bit encryption key, and a 256 bit signing key.
+    """
+
+    RATCHETSIZE = 256
+    """
+    X.25519 ratchet key size in bits.
+    """
+
+    RATCHET_EXPIRY = 60*60*24*30
+    """
+    The expiry time for received ratchets in seconds, defaults to 30 days. Reticulum will always use the most recently
+    announced ratchet, and remember it for up to ``RATCHET_EXPIRY`` since receiving it, after which it will be discarded.
+    If a newer ratchet is announced in the meantime, it will be replace the already known ratchet.
+    """
 
     # Non-configurable constants
-    FERNET_OVERHEAD           = RNS.Cryptography.Fernet.FERNET_OVERHEAD
+    TOKEN_OVERHEAD            = RNS.Cryptography.Token.TOKEN_OVERHEAD
     AES128_BLOCKSIZE          = 16          # In bytes
     HASHLENGTH                = 256         # In bits
     SIGLENGTH                 = KEYSIZE     # In bits
 
+    NAME_HASH_LENGTH     = 80
     TRUNCATED_HASHLENGTH = RNS.Reticulum.TRUNCATED_HASHLENGTH
     """
     Constant specifying the truncated hash length (in bits) used by Reticulum
@@ -66,6 +80,9 @@ class Identity:
 
     # Storage
     known_destinations = {}
+    known_ratchets = {}
+
+    ratchet_persist_lock = threading.Lock()
 
     @staticmethod
     def remember(packet_hash, destination_hash, public_key, app_data = None):
@@ -90,6 +107,13 @@ class Identity:
             identity.app_data = identity_data[3]
             return identity
         else:
+            for registered_destination in RNS.Transport.destinations:
+                if destination_hash == registered_destination.hash:
+                    identity = Identity(create_keys=False)
+                    identity.load_public_key(registered_destination.identity.get_public_key())
+                    identity.app_data = None
+                    return identity
+
             return None
 
     @staticmethod
@@ -131,20 +155,23 @@ class Identity:
             storage_known_destinations = {}
             if os.path.isfile(RNS.Reticulum.storagepath+"/known_destinations"):
                 try:
-                    file = open(RNS.Reticulum.storagepath+"/known_destinations","rb")
-                    storage_known_destinations = umsgpack.load(file)
-                    file.close()
+                    with open(RNS.Reticulum.storagepath+"/known_destinations","rb") as file:
+                        storage_known_destinations = umsgpack.load(file)
+ 
                 except:
                     pass
 
-            for destination_hash in storage_known_destinations:
-                if not destination_hash in Identity.known_destinations:
-                    Identity.known_destinations[destination_hash] = storage_known_destinations[destination_hash]
+            try:
+                for destination_hash in storage_known_destinations:
+                    if not destination_hash in Identity.known_destinations:
+                        Identity.known_destinations[destination_hash] = storage_known_destinations[destination_hash]
+            except Exception as e:
+                RNS.log("Skipped recombining known destinations from disk, since an error occurred: "+str(e), RNS.LOG_WARNING)
 
             RNS.log("Saving "+str(len(Identity.known_destinations))+" known destinations to storage...", RNS.LOG_DEBUG)
-            file = open(RNS.Reticulum.storagepath+"/known_destinations","wb")
-            umsgpack.dump(Identity.known_destinations, file)
-            file.close()
+            with open(RNS.Reticulum.storagepath+"/known_destinations","wb") as file:
+                umsgpack.dump(Identity.known_destinations, file)
+            
 
             save_time = time.time() - save_start
             if save_time < 1:
@@ -156,6 +183,7 @@ class Identity:
 
         except Exception as e:
             RNS.log("Error while saving known destinations to disk, the contained exception was: "+str(e), RNS.LOG_ERROR)
+            RNS.trace_exception(e)
 
         Identity.saving_known_destinations = False
 
@@ -163,9 +191,8 @@ class Identity:
     def load_known_destinations():
         if os.path.isfile(RNS.Reticulum.storagepath+"/known_destinations"):
             try:
-                file = open(RNS.Reticulum.storagepath+"/known_destinations","rb")
-                loaded_known_destinations = umsgpack.load(file)
-                file.close()
+                with open(RNS.Reticulum.storagepath+"/known_destinations","rb") as file:
+                    loaded_known_destinations = umsgpack.load(file)
 
                 Identity.known_destinations = {}
                 for known_destination in loaded_known_destinations:
@@ -173,7 +200,8 @@ class Identity:
                         Identity.known_destinations[known_destination] = loaded_known_destinations[known_destination]
 
                 RNS.log("Loaded "+str(len(Identity.known_destinations))+" known destination from storage", RNS.LOG_VERBOSE)
-            except:
+
+            except Exception as e:
                 RNS.log("Error loading known destinations from disk, file will be recreated on exit", RNS.LOG_ERROR)
         else:
             RNS.log("Destinations file does not exist, no known destinations loaded", RNS.LOG_VERBOSE)
@@ -184,7 +212,7 @@ class Identity:
         Get a SHA-256 hash of passed data.
 
         :param data: Data to be hashed as *bytes*.
-        :returns: SHA-256 hash as *bytes*
+        :returns: SHA-256 hash as *bytes*.
         """
         return RNS.Cryptography.sha256(data)
 
@@ -194,7 +222,7 @@ class Identity:
         Get a truncated SHA-256 hash of passed data.
 
         :param data: Data to be hashed as *bytes*.
-        :returns: Truncated SHA-256 hash as *bytes*
+        :returns: Truncated SHA-256 hash as *bytes*.
         """
         return Identity.full_hash(data)[:(Identity.TRUNCATED_HASHLENGTH//8)]
 
@@ -204,43 +232,218 @@ class Identity:
         Get a random SHA-256 hash.
 
         :param data: Data to be hashed as *bytes*.
-        :returns: Truncated SHA-256 hash of random data as *bytes*
+        :returns: Truncated SHA-256 hash of random data as *bytes*.
         """
         return Identity.truncated_hash(os.urandom(Identity.TRUNCATED_HASHLENGTH//8))
 
     @staticmethod
-    def validate_announce(packet):
+    def current_ratchet_id(destination_hash):
+        """
+        Get the ID of the currently used ratchet key for a given destination hash
+
+        :param destination_hash: A destination hash as *bytes*.
+        :returns: A ratchet ID as *bytes* or *None*.
+        """
+        ratchet = Identity.get_ratchet(destination_hash)
+        if ratchet == None:
+            return None
+        else:
+            return Identity._get_ratchet_id(ratchet)
+
+    @staticmethod
+    def _get_ratchet_id(ratchet_pub_bytes):
+        return Identity.full_hash(ratchet_pub_bytes)[:Identity.NAME_HASH_LENGTH//8]
+
+    @staticmethod
+    def _ratchet_public_bytes(ratchet):
+        return X25519PrivateKey.from_private_bytes(ratchet).public_key().public_bytes()
+
+    @staticmethod
+    def _generate_ratchet():
+        ratchet_prv = X25519PrivateKey.generate()
+        ratchet_pub = ratchet_prv.public_key()
+        return ratchet_prv.private_bytes()
+
+    @staticmethod
+    def _remember_ratchet(destination_hash, ratchet):
+        try:
+            if destination_hash in Identity.known_ratchets and Identity.known_ratchets[destination_hash] == ratchet:
+                ratchet_exists = True
+            else:
+                ratchet_exists = False
+
+            if not ratchet_exists:
+                RNS.log(f"Remembering ratchet {RNS.prettyhexrep(Identity._get_ratchet_id(ratchet))} for {RNS.prettyhexrep(destination_hash)}", RNS.LOG_EXTREME)
+                Identity.known_ratchets[destination_hash] = ratchet
+                if not RNS.Transport.owner.is_connected_to_shared_instance:
+                    def persist_job():
+                        with Identity.ratchet_persist_lock:
+                            hexhash = RNS.hexrep(destination_hash, delimit=False)
+                            ratchet_data = {"ratchet": ratchet, "received": time.time()}
+
+                            ratchetdir = RNS.Reticulum.storagepath+"/ratchets"
+                            
+                            if not os.path.isdir(ratchetdir):
+                                os.makedirs(ratchetdir)
+
+                            outpath   = f"{ratchetdir}/{hexhash}.out"
+                            finalpath = f"{ratchetdir}/{hexhash}"
+                            with open(outpath, "wb") as ratchet_file:
+                                ratchet_file.write(umsgpack.packb(ratchet_data))
+                            os.replace(outpath, finalpath)
+
+                    
+                    threading.Thread(target=persist_job, daemon=True).start()
+
+        except Exception as e:
+            RNS.log(f"Could not persist ratchet for {RNS.prettyhexrep(destination_hash)} to storage.", RNS.LOG_ERROR)
+            RNS.log(f"The contained exception was: {e}")
+            RNS.trace_exception(e)
+
+    @staticmethod
+    def _clean_ratchets():
+        RNS.log("Cleaning ratchets...", RNS.LOG_DEBUG)
+        try:
+            now = time.time()
+            ratchetdir = RNS.Reticulum.storagepath+"/ratchets"
+            if os.path.isdir(ratchetdir):
+                for filename in os.listdir(ratchetdir):
+                    try:
+                        expired = False
+                        with open(f"{ratchetdir}/{filename}", "rb") as rf:
+                            ratchet_data = umsgpack.unpackb(rf.read())
+                            if now > ratchet_data["received"]+Identity.RATCHET_EXPIRY:
+                                expired = True
+
+                        if expired:
+                            os.unlink(f"{ratchetdir}/{filename}")
+
+                    except Exception as e:
+                        RNS.log(f"An error occurred while cleaning ratchets, in the processing of {ratchetdir}/{filename}.", RNS.LOG_ERROR)
+                        RNS.log(f"The contained exception was: {e}", RNS.LOG_ERROR)
+
+        except Exception as e:
+            RNS.log(f"An error occurred while cleaning ratchets. The contained exception was: {e}", RNS.LOG_ERROR)
+
+    @staticmethod
+    def get_ratchet(destination_hash):
+        if not destination_hash in Identity.known_ratchets:
+            ratchetdir = RNS.Reticulum.storagepath+"/ratchets"
+            hexhash = RNS.hexrep(destination_hash, delimit=False)
+            ratchet_path = f"{ratchetdir}/{hexhash}"
+            if os.path.isfile(ratchet_path):
+                try:
+                    with open(ratchet_path, "rb") as ratchet_file:
+                        ratchet_data = umsgpack.unpackb(ratchet_file.read())
+                        if time.time() < ratchet_data["received"]+Identity.RATCHET_EXPIRY and len(ratchet_data["ratchet"]) == Identity.RATCHETSIZE//8:
+                            Identity.known_ratchets[destination_hash] = ratchet_data["ratchet"]
+                        else:
+                            return None
+                
+                except Exception as e:
+                    RNS.log(f"An error occurred while loading ratchet data for {RNS.prettyhexrep(destination_hash)} from storage.", RNS.LOG_ERROR)
+                    RNS.log(f"The contained exception was: {e}", RNS.LOG_ERROR)
+                    return None
+
+        if destination_hash in Identity.known_ratchets:
+            return Identity.known_ratchets[destination_hash]
+        else:
+            RNS.log(f"Could not load ratchet for {RNS.prettyhexrep(destination_hash)}", RNS.LOG_DEBUG)
+            return None
+
+    @staticmethod
+    def validate_announce(packet, only_validate_signature=False):
         try:
             if packet.packet_type == RNS.Packet.ANNOUNCE:
+                keysize       = Identity.KEYSIZE//8
+                ratchetsize   = Identity.RATCHETSIZE//8
+                name_hash_len = Identity.NAME_HASH_LENGTH//8
+                sig_len       = Identity.SIGLENGTH//8
                 destination_hash = packet.destination_hash
-                public_key = packet.data[:Identity.KEYSIZE//8]
-                random_hash = packet.data[Identity.KEYSIZE//8:Identity.KEYSIZE//8+10]
-                signature = packet.data[Identity.KEYSIZE//8+10:Identity.KEYSIZE//8+10+Identity.KEYSIZE//8]
-                app_data = b""
-                if len(packet.data) > Identity.KEYSIZE//8+10+Identity.KEYSIZE//8:
-                    app_data = packet.data[Identity.KEYSIZE//8+10+Identity.KEYSIZE//8:]
 
-                signed_data = destination_hash+public_key+random_hash+app_data
+                # Get public key bytes from announce
+                public_key = packet.data[:keysize]
 
-                if not len(packet.data) > Identity.KEYSIZE//8+10+Identity.KEYSIZE//8:
+                # If the packet context flag is set,
+                # this announce contains a new ratchet
+                if packet.context_flag == RNS.Packet.FLAG_SET:
+                    name_hash   = packet.data[keysize:keysize+name_hash_len ]
+                    random_hash = packet.data[keysize+name_hash_len:keysize+name_hash_len+10]
+                    ratchet     = packet.data[keysize+name_hash_len+10:keysize+name_hash_len+10+ratchetsize]
+                    signature   = packet.data[keysize+name_hash_len+10+ratchetsize:keysize+name_hash_len+10+ratchetsize+sig_len]
+                    app_data    = b""
+                    if len(packet.data) > keysize+name_hash_len+10+sig_len+ratchetsize:
+                        app_data = packet.data[keysize+name_hash_len+10+sig_len+ratchetsize:]
+
+                # If the packet context flag is not set,
+                # this announce does not contain a ratchet
+                else:
+                    ratchet     = b""
+                    name_hash   = packet.data[keysize:keysize+name_hash_len]
+                    random_hash = packet.data[keysize+name_hash_len:keysize+name_hash_len+10]
+                    signature   = packet.data[keysize+name_hash_len+10:keysize+name_hash_len+10+sig_len]
+                    app_data    = b""
+                    if len(packet.data) > keysize+name_hash_len+10+sig_len:
+                        app_data = packet.data[keysize+name_hash_len+10+sig_len:]
+
+                signed_data = destination_hash+public_key+name_hash+random_hash+ratchet+app_data
+
+                if not len(packet.data) > Identity.KEYSIZE//8+Identity.NAME_HASH_LENGTH//8+10+Identity.SIGLENGTH//8:
                     app_data = None
 
                 announced_identity = Identity(create_keys=False)
                 announced_identity.load_public_key(public_key)
 
                 if announced_identity.pub != None and announced_identity.validate(signature, signed_data):
-                    RNS.Identity.remember(packet.get_hash(), destination_hash, public_key, app_data)
-                    del announced_identity
+                    if only_validate_signature:
+                        del announced_identity
+                        return True
+
+                    hash_material = name_hash+announced_identity.hash
+                    expected_hash = RNS.Identity.full_hash(hash_material)[:RNS.Reticulum.TRUNCATED_HASHLENGTH//8]
+
+                    if destination_hash == expected_hash:
+                        # Check if we already have a public key for this destination
+                        # and make sure the public key is not different.
+                        if destination_hash in Identity.known_destinations:
+                            if public_key != Identity.known_destinations[destination_hash][2]:
+                                # In reality, this should never occur, but in the odd case
+                                # that someone manages a hash collision, we reject the announce.
+                                RNS.log("Received announce with valid signature and destination hash, but announced public key does not match already known public key.", RNS.LOG_CRITICAL)
+                                RNS.log("This may indicate an attempt to modify network paths, or a random hash collision. The announce was rejected.", RNS.LOG_CRITICAL)
+                                return False
+
+                        RNS.Identity.remember(packet.get_hash(), destination_hash, public_key, app_data)
+                        del announced_identity
+
+                        if packet.rssi != None or packet.snr != None:
+                            signal_str = " ["
+                            if packet.rssi != None:
+                                signal_str += "RSSI "+str(packet.rssi)+"dBm"
+                                if packet.snr != None:
+                                    signal_str += ", "
+                            if packet.snr != None:
+                                signal_str += "SNR "+str(packet.snr)+"dB"
+                            signal_str += "]"
+                        else:
+                            signal_str = ""
+
+                        if hasattr(packet, "transport_id") and packet.transport_id != None:
+                            RNS.log("Valid announce for "+RNS.prettyhexrep(destination_hash)+" "+str(packet.hops)+" hops away, received via "+RNS.prettyhexrep(packet.transport_id)+" on "+str(packet.receiving_interface)+signal_str, RNS.LOG_EXTREME)
+                        else:
+                            RNS.log("Valid announce for "+RNS.prettyhexrep(destination_hash)+" "+str(packet.hops)+" hops away, received on "+str(packet.receiving_interface)+signal_str, RNS.LOG_EXTREME)
+
+                        if ratchet:
+                            Identity._remember_ratchet(destination_hash, ratchet)
+
+                        return True
 
-                    if hasattr(packet, "transport_id") and packet.transport_id != None:
-                        RNS.log("Valid announce for "+RNS.prettyhexrep(destination_hash)+" "+str(packet.hops)+" hops away, received via "+RNS.prettyhexrep(packet.transport_id)+" on "+str(packet.receiving_interface), RNS.LOG_EXTREME)
                     else:
-                        RNS.log("Valid announce for "+RNS.prettyhexrep(destination_hash)+" "+str(packet.hops)+" hops away, received on "+str(packet.receiving_interface), RNS.LOG_EXTREME)
-
-                    return True
+                        RNS.log("Received invalid announce for "+RNS.prettyhexrep(destination_hash)+": Destination mismatch.", RNS.LOG_DEBUG)
+                        return False
 
                 else:
-                    RNS.log("Received invalid announce for "+RNS.prettyhexrep(destination_hash), RNS.LOG_DEBUG)
+                    RNS.log("Received invalid announce for "+RNS.prettyhexrep(destination_hash)+": Invalid signature.", RNS.LOG_DEBUG)
                     del announced_identity
                     return False
         
@@ -413,7 +616,7 @@ class Identity:
             return False
         except Exception as e:
             RNS.log("Error while loading identity from "+str(path), RNS.LOG_ERROR)
-            RNS.log("The contained exception was: "+str(e))
+            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
 
     def get_salt(self):
         return self.hash
@@ -421,7 +624,7 @@ class Identity:
     def get_context(self):
         return None
 
-    def encrypt(self, plaintext):
+    def encrypt(self, plaintext, ratchet=None):
         """
         Encrypts information for the identity.
 
@@ -433,7 +636,12 @@ class Identity:
             ephemeral_key = X25519PrivateKey.generate()
             ephemeral_pub_bytes = ephemeral_key.public_key().public_bytes()
 
-            shared_key = ephemeral_key.exchange(self.pub)
+            if ratchet != None:
+                target_public_key = X25519PublicKey.from_public_bytes(ratchet)
+            else:
+                target_public_key = self.pub
+
+            shared_key = ephemeral_key.exchange(target_public_key)
             
             derived_key = RNS.Cryptography.hkdf(
                 length=32,
@@ -442,8 +650,8 @@ class Identity:
                 context=self.get_context(),
             )
 
-            fernet = Fernet(derived_key)
-            ciphertext = fernet.encrypt(plaintext)
+            token = Token(derived_key)
+            ciphertext = token.encrypt(plaintext)
             token = ephemeral_pub_bytes+ciphertext
 
             return token
@@ -451,7 +659,7 @@ class Identity:
             raise KeyError("Encryption failed because identity does not hold a public key")
 
 
-    def decrypt(self, ciphertext_token):
+    def decrypt(self, ciphertext_token, ratchets=None, enforce_ratchets=False, ratchet_id_receiver=None):
         """
         Decrypts information for the identity.
 
@@ -465,22 +673,55 @@ class Identity:
                 try:
                     peer_pub_bytes = ciphertext_token[:Identity.KEYSIZE//8//2]
                     peer_pub = X25519PublicKey.from_public_bytes(peer_pub_bytes)
-
-                    shared_key = self.prv.exchange(peer_pub)
-
-                    derived_key = RNS.Cryptography.hkdf(
-                        length=32,
-                        derive_from=shared_key,
-                        salt=self.get_salt(),
-                        context=self.get_context(),
-                    )
-
-                    fernet = Fernet(derived_key)
                     ciphertext = ciphertext_token[Identity.KEYSIZE//8//2:]
-                    plaintext = fernet.decrypt(ciphertext)
+
+                    if ratchets:
+                        for ratchet in ratchets:
+                            try:
+                                ratchet_prv = X25519PrivateKey.from_private_bytes(ratchet)
+                                ratchet_id = Identity._get_ratchet_id(ratchet_prv.public_key().public_bytes())
+                                shared_key = ratchet_prv.exchange(peer_pub)
+                                derived_key = RNS.Cryptography.hkdf(
+                                    length=32,
+                                    derive_from=shared_key,
+                                    salt=self.get_salt(),
+                                    context=self.get_context(),
+                                )
+
+                                token = Token(derived_key)
+                                plaintext = token.decrypt(ciphertext)
+                                if ratchet_id_receiver:
+                                    ratchet_id_receiver.latest_ratchet_id = ratchet_id
+                                
+                                break
+                            
+                            except Exception as e:
+                                pass
+
+                    if enforce_ratchets and plaintext == None:
+                        RNS.log("Decryption with ratchet enforcement by "+RNS.prettyhexrep(self.hash)+" failed. Dropping packet.", RNS.LOG_DEBUG)
+                        if ratchet_id_receiver:
+                            ratchet_id_receiver.latest_ratchet_id = None
+                        return None
+
+                    if plaintext == None:
+                        shared_key = self.prv.exchange(peer_pub)
+                        derived_key = RNS.Cryptography.hkdf(
+                            length=32,
+                            derive_from=shared_key,
+                            salt=self.get_salt(),
+                            context=self.get_context(),
+                        )
+
+                        token = Token(derived_key)
+                        plaintext = token.decrypt(ciphertext)
+                        if ratchet_id_receiver:
+                            ratchet_id_receiver.latest_ratchet_id = None
 
                 except Exception as e:
                     RNS.log("Decryption by "+RNS.prettyhexrep(self.hash)+" failed: "+str(e), RNS.LOG_DEBUG)
+                    if ratchet_id_receiver:
+                        ratchet_id_receiver.latest_ratchet_id = None
                     
                 return plaintext;
             else:

RNS/Interfaces/AX25KISSInterface.py

@@ -20,7 +20,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from .Interface import Interface
+from RNS.Interfaces.Interface import Interface
 from time import sleep
 import sys
 import threading
@@ -59,6 +59,7 @@ class AX25():
 class AX25KISSInterface(Interface):
     MAX_CHUNK = 32768
     BITRATE_GUESS = 1200
+    DEFAULT_IFAC_SIZE = 8
 
     owner    = None
     port     = None
@@ -68,7 +69,7 @@ class AX25KISSInterface(Interface):
     stopbits = None
     serial   = None
 
-    def __init__(self, owner, name, callsign, ssid, port, speed, databits, parity, stopbits, preamble, txtail, persistence, slottime, flow_control):
+    def __init__(self, owner, configuration):
         import importlib
         if importlib.util.find_spec('serial') != None:
             import serial
@@ -77,8 +78,26 @@ class AX25KISSInterface(Interface):
             RNS.log("You can install one with the command: python3 -m pip install pyserial", RNS.LOG_CRITICAL)
             RNS.panic()
 
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
+
+        c = Interface.get_config_obj(configuration)
+        name = c["name"]
+        preamble = int(c["preamble"]) if "preamble" in c else None
+        txtail = int(c["txtail"]) if "txtail" in c else None
+        persistence = int(c["persistence"]) if "persistence" in c else None
+        slottime = int(c["slottime"]) if "slottime" in c else None
+        flow_control = c.as_bool("flow_control") if "flow_control" in c else False
+        port = c["port"] if "port" in c else None
+        speed = int(c["speed"]) if "speed" in c else 9600
+        databits = int(c["databits"]) if "databits" in c else 8
+        parity = c["parity"] if "parity" in c else "N"
+        stopbits = int(c["stopbits"]) if "stopbits" in c else 1
+
+        callsign = c["callsign"] if "callsign" in c else ""
+        ssid = int(c["ssid"]) if "ssid" in c else -1
+
+        if port == None:
+            raise ValueError("No port specified for serial interface")
 
         self.HW_MTU = 564
         
@@ -97,7 +116,7 @@ class AX25KISSInterface(Interface):
         self.stopbits = stopbits
         self.timeout  = 100
         self.online   = False
-        self.bitrate  = KISSInterface.BITRATE_GUESS
+        self.bitrate  = AX25KISSInterface.BITRATE_GUESS
 
         self.packet_queue    = []
         self.flow_control    = flow_control
@@ -226,13 +245,13 @@ class AX25KISSInterface(Interface):
                 raise IOError("Could not enable AX.25 KISS interface flow control")
 
 
-    def processIncoming(self, data):
+    def process_incoming(self, data):
         if (len(data) > AX25.HEADER_SIZE):
             self.rxb += len(data)
             self.owner.inbound(data[AX25.HEADER_SIZE:], self)
 
 
-    def processOutgoing(self,data):
+    def process_outgoing(self,data):
         datalen = len(data)
         if self.online:
             if self.interface_ready:
@@ -282,7 +301,7 @@ class AX25KISSInterface(Interface):
         if len(self.packet_queue) > 0:
             data = self.packet_queue.pop(0)
             self.interface_ready = True
-            self.processOutgoing(data)
+            self.process_outgoing(data)
         elif len(self.packet_queue) == 0:
             self.interface_ready = True
 
@@ -301,7 +320,7 @@ class AX25KISSInterface(Interface):
 
                     if (in_frame and byte == KISS.FEND and command == KISS.CMD_DATA):
                         in_frame = False
-                        self.processIncoming(data_buffer)
+                        self.process_incoming(data_buffer)
                     elif (byte == KISS.FEND):
                         in_frame = True
                         command = KISS.CMD_UNKNOWN
@@ -367,5 +386,8 @@ class AX25KISSInterface(Interface):
 
         RNS.log("Reconnected serial port for "+str(self))
 
+    def should_ingress_limit(self):
+        return False
+
     def __str__(self):
         return "AX25KISSInterface["+self.name+"]"

RNS/Interfaces/Android/KISSInterface.py

@@ -0,0 +1,431 @@
+# MIT License
+#
+# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from RNS.Interfaces.Interface import Interface
+from time import sleep
+import sys
+import threading
+import time
+import RNS
+
+class KISS():
+    FEND              = 0xC0
+    FESC              = 0xDB
+    TFEND             = 0xDC
+    TFESC             = 0xDD
+    CMD_UNKNOWN       = 0xFE
+    CMD_DATA          = 0x00
+    CMD_TXDELAY       = 0x01
+    CMD_P             = 0x02
+    CMD_SLOTTIME      = 0x03
+    CMD_TXTAIL        = 0x04
+    CMD_FULLDUPLEX    = 0x05
+    CMD_SETHARDWARE   = 0x06
+    CMD_READY         = 0x0F
+    CMD_RETURN        = 0xFF
+
+    @staticmethod
+    def escape(data):
+        data = data.replace(bytes([0xdb]), bytes([0xdb, 0xdd]))
+        data = data.replace(bytes([0xc0]), bytes([0xdb, 0xdc]))
+        return data
+
+class KISSInterface(Interface):
+    MAX_CHUNK = 32768
+    BITRATE_GUESS = 1200
+    DEFAULT_IFAC_SIZE = 8
+
+    owner    = None
+    port     = None
+    speed    = None
+    databits = None
+    parity   = None
+    stopbits = None
+    serial   = None
+
+    def __init__(self, owner, configuration):
+        import importlib
+        if RNS.vendor.platformutils.is_android():
+            self.on_android  = True
+            if importlib.util.find_spec('usbserial4a') != None:
+                if importlib.util.find_spec('jnius') == None:
+                    RNS.log("Could not load jnius API wrapper for Android, KISS interface cannot be created.", RNS.LOG_CRITICAL)
+                    RNS.log("This probably means you are trying to use an USB-based interface from within Termux or similar.", RNS.LOG_CRITICAL)
+                    RNS.log("This is currently not possible, due to this environment limiting access to the native Android APIs.", RNS.LOG_CRITICAL)
+                    RNS.panic()
+
+                from usbserial4a import serial4a as serial
+                self.parity = "N"
+            
+            else:
+                RNS.log("Could not load USB serial module for Android, KISS interface cannot be created.", RNS.LOG_CRITICAL)
+                RNS.log("You can install this module by issuing: pip install usbserial4a", RNS.LOG_CRITICAL)
+                RNS.panic()
+        else:
+            raise SystemError("Android-specific interface was used on non-Android OS")
+
+        super().__init__()
+
+        c = Interface.get_config_obj(configuration)
+        name = c["name"]
+        preamble = int(c["preamble"]) if "preamble" in c else None
+        txtail = int(c["txtail"]) if "txtail" in c else None
+        persistence = int(c["persistence"]) if "persistence" in c else None
+        slottime = int(c["slottime"]) if "slottime" in c else None
+        flow_control = c.as_bool("flow_control") if "flow_control" in c else False
+        port = c["port"] if "port" in c else None
+        speed = int(c["speed"]) if "speed" in c else 9600
+        databits = int(c["databits"]) if "databits" in c else 8
+        parity = c["parity"] if "parity" in c else "N"
+        stopbits = int(c["stopbits"]) if "stopbits" in c else 1
+        beacon_interval = int(c["beacon_interval"]) if "beacon_interval" in c and c["beacon_interval"] != None else None
+        beacon_data = c["beacon_data"] if "beacon_data" in c else None
+        
+        self.HW_MTU = 564
+        
+        if beacon_data == None:
+            beacon_data = ""
+
+        self.pyserial = serial
+        self.serial   = None
+        self.owner    = owner
+        self.name     = name
+        self.port     = port
+        self.speed    = speed
+        self.databits = databits
+        self.parity   = "N"
+        self.stopbits = stopbits
+        self.timeout  = 100
+        self.online   = False
+        self.beacon_i = beacon_interval
+        self.beacon_d = beacon_data.encode("utf-8")
+        self.first_tx = None
+        self.bitrate  = KISSInterface.BITRATE_GUESS
+
+        self.packet_queue    = []
+        self.flow_control    = flow_control
+        self.interface_ready = False
+        self.flow_control_timeout = 5
+        self.flow_control_locked  = time.time()
+
+        self.preamble    = preamble if preamble != None else 350;
+        self.txtail      = txtail if txtail != None else 20;
+        self.persistence = persistence if persistence != None else 64;
+        self.slottime    = slottime if slottime != None else 20;
+
+        if parity.lower() == "e" or parity.lower() == "even":
+            self.parity = "E"
+
+        if parity.lower() == "o" or parity.lower() == "odd":
+            self.parity = "O"
+
+        try:
+            self.open_port()
+        except Exception as e:
+            RNS.log("Could not open serial port "+self.port, RNS.LOG_ERROR)
+            raise e
+
+        if self.serial.is_open:
+            self.configure_device()
+        else:
+            raise IOError("Could not open serial port")
+
+
+    def open_port(self):
+        RNS.log("Opening serial port "+self.port+"...")
+        # Get device parameters
+        from usb4a import usb
+        device = usb.get_usb_device(self.port)
+        if device:
+            vid = device.getVendorId()
+            pid = device.getProductId()
+
+            # Driver overrides for speficic chips
+            proxy = self.pyserial.get_serial_port
+            if vid == 0x1A86 and pid == 0x55D4:
+                # Force CDC driver for Qinheng CH34x
+                RNS.log(str(self)+" using CDC driver for "+RNS.hexrep(vid)+":"+RNS.hexrep(pid), RNS.LOG_DEBUG)
+                from usbserial4a.cdcacmserial4a import CdcAcmSerial
+                proxy = CdcAcmSerial
+
+            self.serial = proxy(
+                self.port,
+                baudrate = self.speed,
+                bytesize = self.databits,
+                parity = self.parity,
+                stopbits = self.stopbits,
+                xonxoff = False,
+                rtscts = False,
+                timeout = None,
+                inter_byte_timeout = None,
+                # write_timeout = wtimeout,
+                dsrdtr = False,
+            )
+
+            if vid == 0x0403:
+                # Hardware parameters for FTDI devices @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 16 * 1024
+                self.serial.USB_READ_TIMEOUT_MILLIS = 100
+                self.serial.timeout = 0.1
+            elif vid == 0x10C4:
+                # Hardware parameters for SiLabs CP210x @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 64 
+                self.serial.USB_READ_TIMEOUT_MILLIS = 12
+                self.serial.timeout = 0.012
+            elif vid == 0x1A86 and pid == 0x55D4:
+                # Hardware parameters for Qinheng CH34x @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 64
+                self.serial.USB_READ_TIMEOUT_MILLIS = 12
+                self.serial.timeout = 0.1
+            else:
+                # Default values
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 1 * 1024
+                self.serial.USB_READ_TIMEOUT_MILLIS = 100
+                self.serial.timeout = 0.1
+
+            RNS.log(str(self)+" USB read buffer size set to "+RNS.prettysize(self.serial.DEFAULT_READ_BUFFER_SIZE), RNS.LOG_DEBUG)
+            RNS.log(str(self)+" USB read timeout set to "+str(self.serial.USB_READ_TIMEOUT_MILLIS)+"ms", RNS.LOG_DEBUG)
+            RNS.log(str(self)+" USB write timeout set to "+str(self.serial.USB_WRITE_TIMEOUT_MILLIS)+"ms", RNS.LOG_DEBUG)
+
+    def configure_device(self):
+        # Allow time for interface to initialise before config
+        sleep(2.0)
+        thread = threading.Thread(target=self.readLoop)
+        thread.daemon = True
+        thread.start()
+        self.online = True
+        RNS.log("Serial port "+self.port+" is now open")
+        RNS.log("Configuring KISS interface parameters...")
+        self.setPreamble(self.preamble)
+        self.setTxTail(self.txtail)
+        self.setPersistence(self.persistence)
+        self.setSlotTime(self.slottime)
+        self.setFlowControl(self.flow_control)
+        self.interface_ready = True
+        RNS.log("KISS interface configured")
+
+    def setPreamble(self, preamble):
+        preamble_ms = preamble
+        preamble = int(preamble_ms / 10)
+        if preamble < 0:
+            preamble = 0
+        if preamble > 255:
+            preamble = 255
+
+        kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_TXDELAY])+bytes([preamble])+bytes([KISS.FEND])
+        written = self.serial.write(kiss_command)
+        if written != len(kiss_command):
+            raise IOError("Could not configure KISS interface preamble to "+str(preamble_ms)+" (command value "+str(preamble)+")")
+
+    def setTxTail(self, txtail):
+        txtail_ms = txtail
+        txtail = int(txtail_ms / 10)
+        if txtail < 0:
+            txtail = 0
+        if txtail > 255:
+            txtail = 255
+
+        kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_TXTAIL])+bytes([txtail])+bytes([KISS.FEND])
+        written = self.serial.write(kiss_command)
+        if written != len(kiss_command):
+            raise IOError("Could not configure KISS interface TX tail to "+str(txtail_ms)+" (command value "+str(txtail)+")")
+
+    def setPersistence(self, persistence):
+        if persistence < 0:
+            persistence = 0
+        if persistence > 255:
+            persistence = 255
+
+        kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_P])+bytes([persistence])+bytes([KISS.FEND])
+        written = self.serial.write(kiss_command)
+        if written != len(kiss_command):
+            raise IOError("Could not configure KISS interface persistence to "+str(persistence))
+
+    def setSlotTime(self, slottime):
+        slottime_ms = slottime
+        slottime = int(slottime_ms / 10)
+        if slottime < 0:
+            slottime = 0
+        if slottime > 255:
+            slottime = 255
+
+        kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_SLOTTIME])+bytes([slottime])+bytes([KISS.FEND])
+        written = self.serial.write(kiss_command)
+        if written != len(kiss_command):
+            raise IOError("Could not configure KISS interface slot time to "+str(slottime_ms)+" (command value "+str(slottime)+")")
+
+    def setFlowControl(self, flow_control):
+        kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_READY])+bytes([0x01])+bytes([KISS.FEND])
+        written = self.serial.write(kiss_command)
+        if written != len(kiss_command):
+            if (flow_control):
+                raise IOError("Could not enable KISS interface flow control")
+            else:
+                raise IOError("Could not enable KISS interface flow control")
+
+
+    def process_incoming(self, data):
+        self.rxb += len(data)
+        def af():
+            self.owner.inbound(data, self)
+        threading.Thread(target=af, daemon=True).start()
+
+    def process_outgoing(self,data):
+        datalen = len(data)
+        if self.online:
+            if self.interface_ready:
+                if self.flow_control:
+                    self.interface_ready = False
+                    self.flow_control_locked = time.time()
+
+                data = data.replace(bytes([0xdb]), bytes([0xdb])+bytes([0xdd]))
+                data = data.replace(bytes([0xc0]), bytes([0xdb])+bytes([0xdc]))
+                frame = bytes([KISS.FEND])+bytes([0x00])+data+bytes([KISS.FEND])
+
+                written = self.serial.write(frame)
+                self.txb += datalen
+
+                if data == self.beacon_d:
+                    self.first_tx = None
+                else:
+                    if self.first_tx == None:
+                        self.first_tx = time.time()
+
+                if written != len(frame):
+                    raise IOError("Serial interface only wrote "+str(written)+" bytes of "+str(len(data)))
+
+            else:
+                self.queue(data)
+
+    def queue(self, data):
+        self.packet_queue.append(data)
+
+    def process_queue(self):
+        if len(self.packet_queue) > 0:
+            data = self.packet_queue.pop(0)
+            self.interface_ready = True
+            self.process_outgoing(data)
+        elif len(self.packet_queue) == 0:
+            self.interface_ready = True
+
+    def readLoop(self):
+        try:
+            in_frame = False
+            escape = False
+            command = KISS.CMD_UNKNOWN
+            data_buffer = b""
+            last_read_ms = int(time.time()*1000)
+
+            while self.serial.is_open:
+                serial_bytes = self.serial.read()
+                got = len(serial_bytes)
+
+                for byte in serial_bytes:
+                    last_read_ms = int(time.time()*1000)
+
+                    if (in_frame and byte == KISS.FEND and command == KISS.CMD_DATA):
+                        in_frame = False
+                        self.process_incoming(data_buffer)
+                    elif (byte == KISS.FEND):
+                        in_frame = True
+                        command = KISS.CMD_UNKNOWN
+                        data_buffer = b""
+                    elif (in_frame and len(data_buffer) < self.HW_MTU):
+                        if (len(data_buffer) == 0 and command == KISS.CMD_UNKNOWN):
+                            # We only support one HDLC port for now, so
+                            # strip off the port nibble
+                            byte = byte & 0x0F
+                            command = byte
+                        elif (command == KISS.CMD_DATA):
+                            if (byte == KISS.FESC):
+                                escape = True
+                            else:
+                                if (escape):
+                                    if (byte == KISS.TFEND):
+                                        byte = KISS.FEND
+                                    if (byte == KISS.TFESC):
+                                        byte = KISS.FESC
+                                    escape = False
+                                data_buffer = data_buffer+bytes([byte])
+                        elif (command == KISS.CMD_READY):
+                            self.process_queue()
+                
+                if got == 0:
+                    time_since_last = int(time.time()*1000) - last_read_ms
+                    if len(data_buffer) > 0 and time_since_last > self.timeout:
+                        data_buffer = b""
+                        in_frame = False
+                        command = KISS.CMD_UNKNOWN
+                        escape = False
+                    sleep(0.05)
+
+                    if self.flow_control:
+                        if not self.interface_ready:
+                            if time.time() > self.flow_control_locked + self.flow_control_timeout:
+                                RNS.log("Interface "+str(self)+" is unlocking flow control due to time-out. This should not happen. Your hardware might have missed a flow-control READY command, or maybe it does not support flow-control.", RNS.LOG_WARNING)
+                                self.process_queue()
+
+                    if self.beacon_i != None and self.beacon_d != None:
+                        if self.first_tx != None:
+                            if time.time() > self.first_tx + self.beacon_i:
+                                RNS.log("Interface "+str(self)+" is transmitting beacon data: "+str(self.beacon_d.decode("utf-8")), RNS.LOG_DEBUG)
+                                self.first_tx = None
+
+                                # Pad to minimum length
+                                frame = bytearray(self.beacon_d)
+                                while len(frame) < 15:
+                                    frame.append(0x00)
+
+                                self.process_outgoing(bytes(frame))
+
+        except Exception as e:
+            self.online = False
+            RNS.log("A serial port error occurred, the contained exception was: "+str(e), RNS.LOG_ERROR)
+            RNS.log("The interface "+str(self)+" experienced an unrecoverable error and is now offline.", RNS.LOG_ERROR)
+            
+            if RNS.Reticulum.panic_on_interface_error:
+                RNS.panic()
+
+            RNS.log("Reticulum will attempt to reconnect the interface periodically.", RNS.LOG_ERROR)
+
+        self.online = False
+        self.serial.close()
+        self.reconnect_port()
+
+    def reconnect_port(self):
+        while not self.online:
+            try:
+                time.sleep(5)
+                RNS.log("Attempting to reconnect serial port "+str(self.port)+" for "+str(self)+"...", RNS.LOG_VERBOSE)
+                self.open_port()
+                if self.serial.is_open:
+                    self.configure_device()
+            except Exception as e:
+                RNS.log("Error while reconnecting port, the contained exception was: "+str(e), RNS.LOG_ERROR)
+
+        RNS.log("Reconnected serial port for "+str(self))
+
+    def should_ingress_limit(self):
+        return False
+
+    def __str__(self):
+        return "KISSInterface["+self.name+"]"

RNS/Interfaces/Android/SerialInterface.py

@@ -0,0 +1,272 @@
+# MIT License
+#
+# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from RNS.Interfaces.Interface import Interface
+from time import sleep
+import sys
+import threading
+import time
+import RNS
+
+class HDLC():
+    # The Serial Interface packetizes data using
+    # simplified HDLC framing, similar to PPP
+    FLAG              = 0x7E
+    ESC               = 0x7D
+    ESC_MASK          = 0x20
+
+    @staticmethod
+    def escape(data):
+        data = data.replace(bytes([HDLC.ESC]), bytes([HDLC.ESC, HDLC.ESC^HDLC.ESC_MASK]))
+        data = data.replace(bytes([HDLC.FLAG]), bytes([HDLC.ESC, HDLC.FLAG^HDLC.ESC_MASK]))
+        return data
+
+class SerialInterface(Interface):
+    MAX_CHUNK = 32768
+    DEFAULT_IFAC_SIZE = 8
+
+    owner    = None
+    port     = None
+    speed    = None
+    databits = None
+    parity   = None
+    stopbits = None
+    serial   = None
+
+    def __init__(self, owner, configuration):
+        import importlib
+        if RNS.vendor.platformutils.is_android():
+            self.on_android  = True
+            if importlib.util.find_spec('usbserial4a') != None:
+                if importlib.util.find_spec('jnius') == None:
+                    RNS.log("Could not load jnius API wrapper for Android, Serial interface cannot be created.", RNS.LOG_CRITICAL)
+                    RNS.log("This probably means you are trying to use an USB-based interface from within Termux or similar.", RNS.LOG_CRITICAL)
+                    RNS.log("This is currently not possible, due to this environment limiting access to the native Android APIs.", RNS.LOG_CRITICAL)
+                    RNS.panic()
+
+                from usbserial4a import serial4a as serial
+                self.parity = "N"
+            
+            else:
+                RNS.log("Could not load USB serial module for Android, Serial interface cannot be created.", RNS.LOG_CRITICAL)
+                RNS.log("You can install this module by issuing: pip install usbserial4a", RNS.LOG_CRITICAL)
+                RNS.panic()
+        else:
+            raise SystemError("Android-specific interface was used on non-Android OS")
+
+        super().__init__()
+
+        c = Interface.get_config_obj(configuration)
+        name = c["name"]
+        port = c["port"] if "port" in c else None
+        speed = int(c["speed"]) if "speed" in c else 9600
+        databits = int(c["databits"]) if "databits" in c else 8
+        parity = c["parity"] if "parity" in c else "N"
+        stopbits = int(c["stopbits"]) if "stopbits" in c else 1
+
+        if port == None:
+            raise ValueError("No port specified for serial interface")
+
+        self.HW_MTU = 564
+        
+        self.pyserial = serial
+        self.serial   = None
+        self.owner    = owner
+        self.name     = name
+        self.port     = port
+        self.speed    = speed
+        self.databits = databits
+        self.parity   = "N"
+        self.stopbits = stopbits
+        self.timeout  = 100
+        self.online   = False
+        self.bitrate  = self.speed
+
+        if parity.lower() == "e" or parity.lower() == "even":
+            self.parity = "E"
+
+        if parity.lower() == "o" or parity.lower() == "odd":
+            self.parity = "O"
+
+        try:
+            self.open_port()
+        except Exception as e:
+            RNS.log("Could not open serial port for interface "+str(self), RNS.LOG_ERROR)
+            raise e
+
+        if self.serial.is_open:
+            self.configure_device()
+        else:
+            raise IOError("Could not open serial port")
+
+
+    def open_port(self):
+        RNS.log("Opening serial port "+self.port+"...")
+        # Get device parameters
+        from usb4a import usb
+        device = usb.get_usb_device(self.port)
+        if device:
+            vid = device.getVendorId()
+            pid = device.getProductId()
+
+            # Driver overrides for speficic chips
+            proxy = self.pyserial.get_serial_port
+            if vid == 0x1A86 and pid == 0x55D4:
+                # Force CDC driver for Qinheng CH34x
+                RNS.log(str(self)+" using CDC driver for "+RNS.hexrep(vid)+":"+RNS.hexrep(pid), RNS.LOG_DEBUG)
+                from usbserial4a.cdcacmserial4a import CdcAcmSerial
+                proxy = CdcAcmSerial
+
+            self.serial = proxy(
+                self.port,
+                baudrate = self.speed,
+                bytesize = self.databits,
+                parity = self.parity,
+                stopbits = self.stopbits,
+                xonxoff = False,
+                rtscts = False,
+                timeout = None,
+                inter_byte_timeout = None,
+                # write_timeout = wtimeout,
+                dsrdtr = False,
+            )
+
+            if vid == 0x0403:
+                # Hardware parameters for FTDI devices @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 16 * 1024
+                self.serial.USB_READ_TIMEOUT_MILLIS = 100
+                self.serial.timeout = 0.1
+            elif vid == 0x10C4:
+                # Hardware parameters for SiLabs CP210x @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 64 
+                self.serial.USB_READ_TIMEOUT_MILLIS = 12
+                self.serial.timeout = 0.012
+            elif vid == 0x1A86 and pid == 0x55D4:
+                # Hardware parameters for Qinheng CH34x @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 64
+                self.serial.USB_READ_TIMEOUT_MILLIS = 12
+                self.serial.timeout = 0.1
+            else:
+                # Default values
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 1 * 1024
+                self.serial.USB_READ_TIMEOUT_MILLIS = 100
+                self.serial.timeout = 0.1
+
+            RNS.log(str(self)+" USB read buffer size set to "+RNS.prettysize(self.serial.DEFAULT_READ_BUFFER_SIZE), RNS.LOG_DEBUG)
+            RNS.log(str(self)+" USB read timeout set to "+str(self.serial.USB_READ_TIMEOUT_MILLIS)+"ms", RNS.LOG_DEBUG)
+            RNS.log(str(self)+" USB write timeout set to "+str(self.serial.USB_WRITE_TIMEOUT_MILLIS)+"ms", RNS.LOG_DEBUG)
+
+    def configure_device(self):
+        sleep(0.5)
+        thread = threading.Thread(target=self.readLoop)
+        thread.daemon = True
+        thread.start()
+        self.online = True
+        RNS.log("Serial port "+self.port+" is now open", RNS.LOG_VERBOSE)
+
+
+    def process_incoming(self, data):
+        self.rxb += len(data)
+        def af():
+            self.owner.inbound(data, self)
+        threading.Thread(target=af, daemon=True).start()
+
+    def process_outgoing(self,data):
+        if self.online:
+            data = bytes([HDLC.FLAG])+HDLC.escape(data)+bytes([HDLC.FLAG])
+            written = self.serial.write(data)
+            self.txb += len(data)            
+            if written != len(data):
+                raise IOError("Serial interface only wrote "+str(written)+" bytes of "+str(len(data)))
+
+    def readLoop(self):
+        try:
+            in_frame = False
+            escape = False
+            data_buffer = b""
+            last_read_ms = int(time.time()*1000)
+
+            while self.serial.is_open:
+                serial_bytes = self.serial.read()
+                got = len(serial_bytes)
+
+                for byte in serial_bytes:
+                    last_read_ms = int(time.time()*1000)
+
+                    if (in_frame and byte == HDLC.FLAG):
+                        in_frame = False
+                        self.process_incoming(data_buffer)
+                    elif (byte == HDLC.FLAG):
+                        in_frame = True
+                        data_buffer = b""
+                    elif (in_frame and len(data_buffer) < self.HW_MTU):
+                        if (byte == HDLC.ESC):
+                            escape = True
+                        else:
+                            if (escape):
+                                if (byte == HDLC.FLAG ^ HDLC.ESC_MASK):
+                                    byte = HDLC.FLAG
+                                if (byte == HDLC.ESC  ^ HDLC.ESC_MASK):
+                                    byte = HDLC.ESC
+                                escape = False
+                            data_buffer = data_buffer+bytes([byte])
+                        
+                if got == 0:
+                    time_since_last = int(time.time()*1000) - last_read_ms
+                    if len(data_buffer) > 0 and time_since_last > self.timeout:
+                        data_buffer = b""
+                        in_frame = False
+                        escape = False
+                    # sleep(0.08)
+                    
+        except Exception as e:
+            self.online = False
+            RNS.log("A serial port error occurred, the contained exception was: "+str(e), RNS.LOG_ERROR)
+            RNS.log("The interface "+str(self)+" experienced an unrecoverable error and is now offline.", RNS.LOG_ERROR)
+            
+            if RNS.Reticulum.panic_on_interface_error:
+                RNS.panic()
+
+            RNS.log("Reticulum will attempt to reconnect the interface periodically.", RNS.LOG_ERROR)
+
+        self.online = False
+        self.serial.close()
+        self.reconnect_port()
+
+    def reconnect_port(self):
+        while not self.online:
+            try:
+                time.sleep(5)
+                RNS.log("Attempting to reconnect serial port "+str(self.port)+" for "+str(self)+"...", RNS.LOG_VERBOSE)
+                self.open_port()
+                if self.serial.is_open:
+                    self.configure_device()
+            except Exception as e:
+                RNS.log("Error while reconnecting port, the contained exception was: "+str(e), RNS.LOG_ERROR)
+
+        RNS.log("Reconnected serial port for "+str(self))
+
+    def should_ingress_limit(self):
+        return False
+
+    def __str__(self):
+        return "SerialInterface["+self.name+"]"

RNS/Interfaces/Android/__init__.py

@@ -0,0 +1,29 @@
+# MIT License
+#
+# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import os
+import glob
+
+py_modules  = glob.glob(os.path.dirname(__file__)+"/*.py")
+pyc_modules = glob.glob(os.path.dirname(__file__)+"/*.pyc")
+modules     = py_modules+pyc_modules
+__all__ = list(set([os.path.basename(f).replace(".pyc", "").replace(".py", "") for f in modules if not (f.endswith("__init__.py") or f.endswith("__init__.pyc"))]))

RNS/Interfaces/AutoInterface.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2024 Mark Qvist / unsigned.io
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -20,9 +20,11 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from .Interface import Interface
+from RNS.Interfaces.Interface import Interface
+from collections import deque
 import socketserver
 import threading
+import re
 import socket
 import struct
 import time
@@ -34,6 +36,7 @@ class AutoInterface(Interface):
     DEFAULT_DISCOVERY_PORT = 29716
     DEFAULT_DATA_PORT      = 42671
     DEFAULT_GROUP_ID       = "reticulum".encode("utf-8")
+    DEFAULT_IFAC_SIZE      = 16
 
     SCOPE_LINK         = "2"
     SCOPE_ADMIN        = "4"
@@ -41,25 +44,64 @@ class AutoInterface(Interface):
     SCOPE_ORGANISATION = "8"
     SCOPE_GLOBAL       = "e"
 
+    MULTICAST_PERMANENT_ADDRESS_TYPE = "0"
+    MULTICAST_TEMPORARY_ADDRESS_TYPE = "1"
+
     PEERING_TIMEOUT    = 7.5
 
+    ALL_IGNORE_IFS     = ["lo0"]
     DARWIN_IGNORE_IFS  = ["awdl0", "llw0", "lo0", "en5"]
     ANDROID_IGNORE_IFS = ["dummy0", "lo", "tun0"]
 
     BITRATE_GUESS      = 10*1000*1000
 
-    def __init__(self, owner, name, group_id=None, discovery_scope=None, discovery_port=None, data_port=None, allowed_interfaces=None, ignored_interfaces=None, configured_bitrate=None):
-        import importlib
-        if importlib.util.find_spec('netifaces') != None:
-            import netifaces
-        else:
-            RNS.log("Using AutoInterface requires the netifaces module.", RNS.LOG_CRITICAL)
-            RNS.log("You can install it with the command: python3 -m pip install netifaces", RNS.LOG_CRITICAL)
-            RNS.panic()
+    MULTI_IF_DEQUE_LEN = 48
+    MULTI_IF_DEQUE_TTL = 0.75
 
-        self.netifaces = netifaces
-        self.rxb = 0
-        self.txb = 0
+    def handler_factory(self, callback):
+        def create_handler(*args, **keys):
+            return AutoInterfaceHandler(callback, *args, **keys)
+        return create_handler
+
+    def descope_linklocal(self, link_local_addr):
+        # Drop scope specifier expressd as %ifname (macOS)
+        link_local_addr = link_local_addr.split("%")[0]
+        # Drop embedded scope specifier (NetBSD, OpenBSD)
+        link_local_addr = re.sub(r"fe80:[0-9a-f]*::","fe80::", link_local_addr)
+        return link_local_addr
+
+    def list_interfaces(self):
+        ifs = self.netinfo.interfaces()
+        return ifs
+
+    def list_addresses(self, ifname):
+        ifas = self.netinfo.ifaddresses(ifname)
+        return ifas
+
+    def interface_name_to_index(self, ifname):
+
+        # socket.if_nametoindex doesn't work with uuid interface names on windows, it wants the ethernet_0 style
+        # we will just get the index from netinfo instead as it seems to work
+        if RNS.vendor.platformutils.is_windows():
+            return self.netinfo.interface_names_to_indexes()[ifname]
+
+        return socket.if_nametoindex(ifname)
+
+    def __init__(self, owner, configuration):
+        c                      = Interface.get_config_obj(configuration)
+        name                   = c["name"]
+        group_id               = c["group_id"] if "group_id" in c else None
+        discovery_scope        = c["discovery_scope"] if "discovery_scope" in c else None
+        discovery_port         = int(c["discovery_port"]) if "discovery_port" in c else None
+        multicast_address_type = c["multicast_address_type"] if "multicast_address_type" in c else None
+        data_port              = int(c["data_port"]) if "data_port" in c else None
+        allowed_interfaces     = c.as_list("devices") if "devices" in c else None
+        ignored_interfaces     = c.as_list("ignored_devices") if "ignored_devices" in c else None
+        configured_bitrate     = c["configured_bitrate"] if "configured_bitrate" in c else None
+
+        from RNS.vendor.ifaddr import niwrapper
+        super().__init__()
+        self.netinfo = niwrapper
 
         self.HW_MTU = 1064
 
@@ -70,16 +112,26 @@ class AutoInterface(Interface):
         self.peers = {}
         self.link_local_addresses = []
         self.adopted_interfaces = {}
+        self.interface_servers = {}
         self.multicast_echoes = {}
         self.timed_out_interfaces = {}
+        self.mif_deque = deque(maxlen=AutoInterface.MULTI_IF_DEQUE_LEN)
+        self.mif_deque_times = deque(maxlen=AutoInterface.MULTI_IF_DEQUE_LEN)
+        self.carrier_changed = False
 
         self.outbound_udp_socket = None
 
+        self.announce_rate_target = None
         self.announce_interval = AutoInterface.PEERING_TIMEOUT/6.0
         self.peer_job_interval = AutoInterface.PEERING_TIMEOUT*1.1
         self.peering_timeout   = AutoInterface.PEERING_TIMEOUT
         self.multicast_echo_timeout = AutoInterface.PEERING_TIMEOUT/2
 
+        # Increase peering timeout on Android, due to potential
+        # low-power modes implemented on many chipsets.
+        if RNS.vendor.platformutils.is_android():
+            self.peering_timeout *= 3
+
         if allowed_interfaces == None:
             self.allowed_interfaces = []
         else:
@@ -100,6 +152,15 @@ class AutoInterface(Interface):
         else:
             self.discovery_port = discovery_port
 
+        if multicast_address_type == None:
+            self.multicast_address_type = AutoInterface.MULTICAST_TEMPORARY_ADDRESS_TYPE
+        elif str(multicast_address_type).lower() == "temporary":
+            self.multicast_address_type = AutoInterface.MULTICAST_TEMPORARY_ADDRESS_TYPE
+        elif str(multicast_address_type).lower() == "permanent":
+            self.multicast_address_type = AutoInterface.MULTICAST_PERMANENT_ADDRESS_TYPE
+        else:
+            self.multicast_address_type = AutoInterface.MULTICAST_TEMPORARY_ADDRESS_TYPE
+
         if data_port == None:
             self.data_port = AutoInterface.DEFAULT_DATA_PORT
         else:
@@ -128,92 +189,120 @@ class AutoInterface(Interface):
         gt += ":"+"{:02x}".format(g[9]+(g[8]<<8))
         gt += ":"+"{:02x}".format(g[11]+(g[10]<<8))
         gt += ":"+"{:02x}".format(g[13]+(g[12]<<8))
-        self.mcast_discovery_address = "ff1"+self.discovery_scope+":"+gt
+        self.mcast_discovery_address = "ff"+self.multicast_address_type+self.discovery_scope+":"+gt
 
         suitable_interfaces = 0
-        for ifname in self.netifaces.interfaces():
-            if RNS.vendor.platformutils.is_darwin() and ifname in AutoInterface.DARWIN_IGNORE_IFS and not ifname in self.allowed_interfaces:
-                RNS.log(str(self)+" skipping Darwin AWDL or tethering interface "+str(ifname), RNS.LOG_EXTREME)
-            elif RNS.vendor.platformutils.is_darwin() and ifname == "lo0":
-                RNS.log(str(self)+" skipping Darwin loopback interface "+str(ifname), RNS.LOG_EXTREME)
-            elif RNS.vendor.platformutils.is_android() and ifname in AutoInterface.ANDROID_IGNORE_IFS and not ifname in self.allowed_interfaces:
-                RNS.log(str(self)+" skipping Android system interface "+str(ifname), RNS.LOG_EXTREME)
-            elif ifname in self.ignored_interfaces:
-                RNS.log(str(self)+" ignoring disallowed interface "+str(ifname), RNS.LOG_EXTREME)
-            else:
-                if len(self.allowed_interfaces) > 0 and not ifname in self.allowed_interfaces:
-                    RNS.log(str(self)+" ignoring interface "+str(ifname)+" since it was not allowed", RNS.LOG_EXTREME)
+        for ifname in self.list_interfaces():
+            try:
+                if RNS.vendor.platformutils.is_darwin() and ifname in AutoInterface.DARWIN_IGNORE_IFS and not ifname in self.allowed_interfaces:
+                    RNS.log(str(self)+" skipping Darwin AWDL or tethering interface "+str(ifname), RNS.LOG_EXTREME)
+                elif RNS.vendor.platformutils.is_darwin() and ifname == "lo0":
+                    RNS.log(str(self)+" skipping Darwin loopback interface "+str(ifname), RNS.LOG_EXTREME)
+                elif RNS.vendor.platformutils.is_android() and ifname in AutoInterface.ANDROID_IGNORE_IFS and not ifname in self.allowed_interfaces:
+                    RNS.log(str(self)+" skipping Android system interface "+str(ifname), RNS.LOG_EXTREME)
+                elif ifname in self.ignored_interfaces:
+                    RNS.log(str(self)+" ignoring disallowed interface "+str(ifname), RNS.LOG_EXTREME)
+                elif ifname in AutoInterface.ALL_IGNORE_IFS:
+                    RNS.log(str(self)+" skipping interface "+str(ifname), RNS.LOG_EXTREME)
                 else:
-                    addresses = self.netifaces.ifaddresses(ifname)
-                    if self.netifaces.AF_INET6 in addresses:
-                        link_local_addr = None
-                        for address in addresses[self.netifaces.AF_INET6]:
-                            if "addr" in address:
-                                if address["addr"].startswith("fe80:"):
-                                    link_local_addr = address["addr"]
-                                    self.link_local_addresses.append(link_local_addr.split("%")[0])
-                                    self.adopted_interfaces[ifname] = link_local_addr.split("%")[0]
-                                    self.multicast_echoes[ifname] = time.time()
-                                    RNS.log(str(self)+" Selecting link-local address "+str(link_local_addr)+" for interface "+str(ifname), RNS.LOG_EXTREME)
+                    if len(self.allowed_interfaces) > 0 and not ifname in self.allowed_interfaces:
+                        RNS.log(str(self)+" ignoring interface "+str(ifname)+" since it was not allowed", RNS.LOG_EXTREME)
+                    else:
+                        addresses = self.list_addresses(ifname)
+                        if self.netinfo.AF_INET6 in addresses:
+                            link_local_addr = None
+                            for address in addresses[self.netinfo.AF_INET6]:
+                                if "addr" in address:
+                                    if address["addr"].startswith("fe80:"):
+                                        link_local_addr = self.descope_linklocal(address["addr"])
+                                        self.link_local_addresses.append(link_local_addr)
+                                        self.adopted_interfaces[ifname] = link_local_addr
+                                        self.multicast_echoes[ifname] = time.time()
+                                        nice_name = self.netinfo.interface_name_to_nice_name(ifname)
+                                        if nice_name != None and nice_name != ifname:
+                                            RNS.log(f"{self} Selecting link-local address {link_local_addr} for interface {nice_name} / {ifname}", RNS.LOG_EXTREME)
+                                        else:
+                                            RNS.log(f"{self} Selecting link-local address {link_local_addr} for interface {ifname}", RNS.LOG_EXTREME)
 
-                        if link_local_addr == None:
-                            RNS.log(str(self)+" No link-local IPv6 address configured for "+str(ifname)+", skipping interface", RNS.LOG_EXTREME)
-                        else:
-                            mcast_addr = self.mcast_discovery_address
-                            RNS.log(str(self)+" Creating multicast discovery listener on "+str(ifname)+" with address "+str(mcast_addr), RNS.LOG_EXTREME)
+                            if link_local_addr == None:
+                                RNS.log(str(self)+" No link-local IPv6 address configured for "+str(ifname)+", skipping interface", RNS.LOG_EXTREME)
+                            else:
+                                mcast_addr = self.mcast_discovery_address
+                                RNS.log(str(self)+" Creating multicast discovery listener on "+str(ifname)+" with address "+str(mcast_addr), RNS.LOG_EXTREME)
 
-                            # Struct with interface index
-                            if_struct = struct.pack("I", socket.if_nametoindex(ifname))
+                                # Struct with interface index
+                                if_struct = struct.pack("I", self.interface_name_to_index(ifname))
 
-                            # Set up multicast socket
-                            discovery_socket = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
-                            discovery_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-                            if hasattr(socket, "SO_REUSEPORT"):
-                                discovery_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
-                            discovery_socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_MULTICAST_IF, if_struct)
+                                # Set up multicast socket
+                                discovery_socket = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
+                                discovery_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+                                if hasattr(socket, "SO_REUSEPORT"):
+                                    discovery_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
+                                discovery_socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_MULTICAST_IF, if_struct)
 
-                            # Join multicast group
-                            mcast_group = socket.inet_pton(socket.AF_INET6, mcast_addr) + if_struct
-                            discovery_socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_JOIN_GROUP, mcast_group)
+                                # Join multicast group
+                                mcast_group = socket.inet_pton(socket.AF_INET6, mcast_addr) + if_struct
+                                discovery_socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_JOIN_GROUP, mcast_group)
 
-                            # Bind socket
-                            addr_info = socket.getaddrinfo(mcast_addr+"%"+ifname, self.discovery_port, socket.AF_INET6, socket.SOCK_DGRAM)
-                            discovery_socket.bind(addr_info[0][4])
+                                # Bind socket
+                                if RNS.vendor.platformutils.is_windows():
 
-                            # Set up thread for discovery packets
-                            def discovery_loop():
-                                self.discovery_handler(discovery_socket, ifname)
+                                    # window throws "[WinError 10049] The requested address is not valid in its context"
+                                    # when trying to use the multicast address as host, or when providing interface index
+                                    # passing an empty host appears to work, but probably not exactly how we want it to...
+                                    discovery_socket.bind(('', self.discovery_port))
 
-                            thread = threading.Thread(target=discovery_loop)
-                            thread.daemon = True
-                            thread.start()
+                                else:
 
-                            suitable_interfaces += 1
+                                    if self.discovery_scope == AutoInterface.SCOPE_LINK:
+                                        addr_info = socket.getaddrinfo(mcast_addr+"%"+ifname, self.discovery_port, socket.AF_INET6, socket.SOCK_DGRAM)
+                                    else:
+                                        addr_info = socket.getaddrinfo(mcast_addr, self.discovery_port, socket.AF_INET6, socket.SOCK_DGRAM)
+
+                                    discovery_socket.bind(addr_info[0][4])
+
+                                # Set up thread for discovery packets
+                                def discovery_loop():
+                                    self.discovery_handler(discovery_socket, ifname)
+
+                                thread = threading.Thread(target=discovery_loop)
+                                thread.daemon = True
+                                thread.start()
+
+                                suitable_interfaces += 1
+
+            except Exception as e:
+                nice_name = self.netinfo.interface_name_to_nice_name(ifname)
+                if nice_name != None and nice_name != ifname:
+                    RNS.log(f"Could not configure the system interface {nice_name} / {ifname} for use with {self}, skipping it. The contained exception was: {e}", RNS.LOG_ERROR)
+                else:
+                    RNS.log(f"Could not configure the system interface {ifname} for use with {self}, skipping it. The contained exception was: {e}", RNS.LOG_ERROR)
 
         if suitable_interfaces == 0:
             RNS.log(str(self)+" could not autoconfigure. This interface currently provides no connectivity.", RNS.LOG_WARNING)
         else:
             self.receives = True
 
+            if configured_bitrate != None:
+                self.bitrate = configured_bitrate
+            else:
+                self.bitrate = AutoInterface.BITRATE_GUESS
+
             peering_wait = self.announce_interval*1.2
             RNS.log(str(self)+" discovering peers for "+str(round(peering_wait, 2))+" seconds...", RNS.LOG_VERBOSE)
 
-            def handlerFactory(callback):
-                def createHandler(*args, **keys):
-                    return AutoInterfaceHandler(callback, *args, **keys)
-                return createHandler
-
             self.owner = owner
             socketserver.UDPServer.address_family = socket.AF_INET6
 
             for ifname in self.adopted_interfaces:
-                local_addr = self.adopted_interfaces[ifname]+"%"+ifname
+                local_addr = self.adopted_interfaces[ifname]+"%"+str(self.interface_name_to_index(ifname))
                 addr_info = socket.getaddrinfo(local_addr, self.data_port, socket.AF_INET6, socket.SOCK_DGRAM)
                 address = addr_info[0][4]
 
-                self.server = socketserver.UDPServer(address, handlerFactory(self.processIncoming))
+                udp_server = socketserver.UDPServer(address, self.handler_factory(self.process_incoming))
+                self.interface_servers[ifname] = udp_server
                 
-                thread = threading.Thread(target=self.server.serve_forever)
+                thread = threading.Thread(target=udp_server.serve_forever)
                 thread.daemon = True
                 thread.start()
 
@@ -223,11 +312,6 @@ class AutoInterface(Interface):
 
             time.sleep(peering_wait)
 
-            if configured_bitrate != None:
-                self.bitrate = configured_bitrate
-            else:
-                self.bitrate = AutoInterface.BITRATE_GUESS
-
             self.online = True
 
 
@@ -266,16 +350,62 @@ class AutoInterface(Interface):
                 RNS.log(str(self)+" removed peer "+str(peer_addr)+" on "+str(removed_peer[0]), RNS.LOG_DEBUG)
 
             for ifname in self.adopted_interfaces:
+                # Check that the link-local address has not changed
+                try:
+                    addresses = self.list_addresses(ifname)
+                    if self.netinfo.AF_INET6 in addresses:
+                        link_local_addr = None
+                        for address in addresses[self.netinfo.AF_INET6]:
+                            if "addr" in address:
+                                if address["addr"].startswith("fe80:"):
+                                    link_local_addr = self.descope_linklocal(address["addr"])
+                                    if link_local_addr != self.adopted_interfaces[ifname]:
+                                        old_link_local_address = self.adopted_interfaces[ifname]
+                                        RNS.log("Replacing link-local address "+str(old_link_local_address)+" for "+str(ifname)+" with "+str(link_local_addr), RNS.LOG_DEBUG)
+                                        self.adopted_interfaces[ifname] = link_local_addr
+                                        self.link_local_addresses.append(link_local_addr)
+
+                                        if old_link_local_address in self.link_local_addresses:
+                                            self.link_local_addresses.remove(old_link_local_address)
+
+                                        local_addr = link_local_addr+"%"+ifname
+                                        addr_info = socket.getaddrinfo(local_addr, self.data_port, socket.AF_INET6, socket.SOCK_DGRAM)
+                                        listen_address = addr_info[0][4]
+
+                                        if ifname in self.interface_servers:
+                                            RNS.log("Shutting down previous UDP listener for "+str(self)+" "+str(ifname), RNS.LOG_DEBUG)
+                                            previous_server = self.interface_servers[ifname]
+                                            def shutdown_server():
+                                                previous_server.shutdown()
+                                            threading.Thread(target=shutdown_server, daemon=True).start()
+
+                                        RNS.log("Starting new UDP listener for "+str(self)+" "+str(ifname), RNS.LOG_DEBUG)
+
+                                        udp_server = socketserver.UDPServer(listen_address, self.handler_factory(self.process_incoming))
+                                        self.interface_servers[ifname] = udp_server
+
+                                        thread = threading.Thread(target=udp_server.serve_forever)
+                                        thread.daemon = True
+                                        thread.start()
+
+                                        self.carrier_changed = True
+
+                except Exception as e:
+                    RNS.log("Could not get device information while updating link-local addresses for "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+                # Check multicast echo timeouts
                 last_multicast_echo = 0
                 if ifname in self.multicast_echoes:
                     last_multicast_echo = self.multicast_echoes[ifname]
 
                 if now - last_multicast_echo > self.multicast_echo_timeout:
                     if ifname in self.timed_out_interfaces and self.timed_out_interfaces[ifname] == False:
+                        self.carrier_changed = True
                         RNS.log("Multicast echo timeout for "+str(ifname)+". Carrier lost.", RNS.LOG_WARNING)
                     self.timed_out_interfaces[ifname] = True
                 else:
                     if ifname in self.timed_out_interfaces and self.timed_out_interfaces[ifname] == True:
+                        self.carrier_changed = True
                         RNS.log(str(self)+" Carrier recovered on "+str(ifname), RNS.LOG_WARNING)
                     self.timed_out_interfaces[ifname] = False
                 
@@ -292,7 +422,7 @@ class AutoInterface(Interface):
             announce_socket = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
             addr_info = socket.getaddrinfo(self.mcast_discovery_address, self.discovery_port, socket.AF_INET6, socket.SOCK_DGRAM)
 
-            ifis = struct.pack("I", socket.if_nametoindex(ifname))
+            ifis = struct.pack("I", self.interface_name_to_index(ifname))
             announce_socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_MULTICAST_IF, ifis)
             announce_socket.sendto(discovery_token, addr_info[0][4])
             announce_socket.close()
@@ -325,17 +455,30 @@ class AutoInterface(Interface):
     def refresh_peer(self, addr):
         self.peers[addr][1] = time.time()
 
-    def processIncoming(self, data):
-        self.rxb += len(data)
-        self.owner.inbound(data, self)
+    def process_incoming(self, data):
+        if self.online:
+            data_hash = RNS.Identity.full_hash(data)
+            deque_hit = False
+            if data_hash in self.mif_deque:
+                for te in self.mif_deque_times:
+                    if te[0] == data_hash and time.time() < te[1]+AutoInterface.MULTI_IF_DEQUE_TTL:
+                        deque_hit = True
+                        break
 
-    def processOutgoing(self,data):
+            if not deque_hit:
+                self.mif_deque.append(data_hash)
+                self.mif_deque_times.append([data_hash, time.time()])
+                self.rxb += len(data)
+                self.owner.inbound(data, self)
+
+    def process_outgoing(self,data):
+        if self.online:
             for peer in self.peers:
                 try:
                     if self.outbound_udp_socket == None:
                         self.outbound_udp_socket = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
-                    
-                    peer_addr = str(peer)+"%"+str(self.peers[peer][0])
+
+                    peer_addr = str(peer)+"%"+str(self.interface_name_to_index(self.peers[peer][0]))
                     addr_info = socket.getaddrinfo(peer_addr, self.data_port, socket.AF_INET6, socket.SOCK_DGRAM)
                     self.outbound_udp_socket.sendto(data, addr_info[0][4])
 
@@ -346,6 +489,14 @@ class AutoInterface(Interface):
             self.txb += len(data)
             
 
+    # Until per-device sub-interfacing is implemented,
+    # ingress limiting should be disabled on AutoInterface
+    def should_ingress_limit(self):
+        return False
+
+    def detach(self):
+        self.online = False
+
     def __str__(self):
         return "AutoInterface["+self.name+"]"
 

RNS/Interfaces/I2PInterface.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2024 Mark Qvist / unsigned.io
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -20,7 +20,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from .Interface import Interface
+from RNS.Interfaces.Interface import Interface
 import socketserver
 import threading
 import platform
@@ -161,8 +161,6 @@ class I2PController:
                                 raise tn.status["exception"]
 
                             else:
-                                self.client_tunnels[i2p_destination] = True
-                                owner.awaiting_i2p_tunnel = False
                                 if owner.socket != None:
                                     if hasattr(owner.socket, "close"):
                                         if callable(owner.socket.close):
@@ -175,6 +173,8 @@ class I2PController:
                                                 owner.socket.close()
                                             except Exception as e:
                                                 RNS.log("Error while closing socket for "+str(owner)+": "+str(e))
+                                self.client_tunnels[i2p_destination] = True
+                                owner.awaiting_i2p_tunnel = False
 
                                 RNS.log(str(owner)+" tunnel setup complete", RNS.LOG_VERBOSE)
 
@@ -383,10 +383,14 @@ class I2PInterfacePeer(Interface):
     I2P_PROBE_AFTER = 10
     I2P_PROBE_INTERVAL = 9
     I2P_PROBES = 5
+    I2P_READ_TIMEOUT = (I2P_PROBE_INTERVAL * I2P_PROBES + I2P_PROBE_AFTER)*2
+
+    TUNNEL_STATE_INIT    = 0x00
+    TUNNEL_STATE_ACTIVE  = 0x01
+    TUNNEL_STATE_STALE   = 0x02
 
     def __init__(self, parent_interface, owner, name, target_i2p_dest=None, connected_socket=None, max_reconnect_tries=None):
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
 
         self.HW_MTU = 1064
         
@@ -409,6 +413,30 @@ class I2PInterfacePeer(Interface):
         self.i2p_tunnel_ready = False
         self.mode             = RNS.Interfaces.Interface.Interface.MODE_FULL
         self.bitrate          = I2PInterface.BITRATE_GUESS
+        self.last_read        = 0
+        self.last_write       = 0
+        self.wd_reset         = False
+        self.i2p_tunnel_state = I2PInterfacePeer.TUNNEL_STATE_INIT
+
+        self.ifac_size = self.parent_interface.ifac_size
+        self.ifac_netname = self.parent_interface.ifac_netname
+        self.ifac_netkey = self.parent_interface.ifac_netkey
+        if self.ifac_netname != None or self.ifac_netkey != None:
+            ifac_origin = b""
+            if self.ifac_netname != None:
+                ifac_origin += RNS.Identity.full_hash(self.ifac_netname.encode("utf-8"))
+            if self.ifac_netkey != None:
+                ifac_origin += RNS.Identity.full_hash(self.ifac_netkey.encode("utf-8"))
+
+            ifac_origin_hash = RNS.Identity.full_hash(ifac_origin)
+            self.ifac_key = RNS.Cryptography.hkdf(
+                length=64,
+                derive_from=ifac_origin_hash,
+                salt=RNS.Reticulum.IFAC_SALT,
+                context=None
+            )
+            self.ifac_identity = RNS.Identity.from_bytes(self.ifac_key)
+            self.ifac_signature = self.ifac_identity.sign(RNS.Identity.full_hash(self.ifac_key))
 
         self.announce_rate_target  = None
         self.announce_rate_grace   = None
@@ -454,7 +482,7 @@ class I2PInterfacePeer(Interface):
                         RNS.log("Error while while configuring "+str(self)+": "+str(e), RNS.LOG_ERROR)
                         RNS.log("Check that I2P is installed and running, and that SAM is enabled. Retrying tunnel setup later.", RNS.LOG_ERROR)
 
-                    time.sleep(15)
+                    time.sleep(8)
 
             thread = threading.Thread(target=tunnel_job)
             thread.daemon = True
@@ -463,6 +491,7 @@ class I2PInterfacePeer(Interface):
             def wait_job():
                 while self.awaiting_i2p_tunnel:
                     time.sleep(0.25)
+                time.sleep(2)
                 
                 if not self.kiss_framing:
                     self.wants_tunnel = True
@@ -482,18 +511,11 @@ class I2PInterfacePeer(Interface):
 
 
     def set_timeouts_linux(self):
-        if not self.i2p_tunneled:
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_USER_TIMEOUT, int(I2PInterfacePeer.TCP_USER_TIMEOUT * 1000))
-            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, int(I2PInterfacePeer.TCP_PROBE_AFTER))
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, int(I2PInterfacePeer.TCP_PROBE_INTERVAL))
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, int(I2PInterfacePeer.TCP_PROBES))
-        else:
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_USER_TIMEOUT, int(I2PInterfacePeer.I2P_USER_TIMEOUT * 1000))
-            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, int(I2PInterfacePeer.I2P_PROBE_AFTER))
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, int(I2PInterfacePeer.I2P_PROBE_INTERVAL))
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, int(I2PInterfacePeer.I2P_PROBES))
+        self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_USER_TIMEOUT, int(I2PInterfacePeer.I2P_USER_TIMEOUT * 1000))
+        self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
+        self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, int(I2PInterfacePeer.I2P_PROBE_AFTER))
+        self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, int(I2PInterfacePeer.I2P_PROBE_INTERVAL))
+        self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, int(I2PInterfacePeer.I2P_PROBES))
 
     def set_timeouts_osx(self):
         if hasattr(socket, "TCP_KEEPALIVE"):
@@ -502,22 +524,19 @@ class I2PInterfacePeer(Interface):
             TCP_KEEPIDLE = 0x10
 
         self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
-        
-        if not self.i2p_tunneled:
-            self.socket.setsockopt(socket.IPPROTO_TCP, TCP_KEEPIDLE, int(I2PInterfacePeer.TCP_PROBE_AFTER))
-        else:
-            self.socket.setsockopt(socket.IPPROTO_TCP, TCP_KEEPIDLE, int(I2PInterfacePeer.I2P_PROBE_AFTER))
+        self.socket.setsockopt(socket.IPPROTO_TCP, TCP_KEEPIDLE, int(I2PInterfacePeer.I2P_PROBE_AFTER))
     
-    def shutdown_socket(self, socket):
-        if callable(socket.close):
-            
+    def shutdown_socket(self, target_socket):
+        if callable(target_socket.close):
             try:
-                socket.shutdown(socket.SHUT_RDWR)
+                if socket != None:
+                    target_socket.shutdown(socket.SHUT_RDWR)
             except Exception as e:
                 RNS.log("Error while shutting down socket for "+str(self)+": "+str(e))
 
             try:
-                socket.close()
+                if socket != None:
+                    target_socket.close()
             except Exception as e:
                 RNS.log("Error while closing socket for "+str(self)+": "+str(e))    
     
@@ -571,7 +590,6 @@ class I2PInterfacePeer(Interface):
 
         return True
 
-
     def reconnect(self):
         if self.initiator:
             if not self.reconnecting:
@@ -609,14 +627,14 @@ class I2PInterfacePeer(Interface):
             RNS.log("Attempt to reconnect on a non-initiator I2P interface. This should not happen.", RNS.LOG_ERROR)
             raise IOError("Attempt to reconnect on a non-initiator I2P interface")
 
-    def processIncoming(self, data):
+    def process_incoming(self, data):
         self.rxb += len(data)
         if hasattr(self, "parent_interface") and self.parent_interface != None and self.parent_count:
             self.parent_interface.rxb += len(data)
                     
         self.owner.inbound(data, self)
 
-    def processOutgoing(self, data):
+    def process_outgoing(self, data):
         if self.online:
             while self.writing:
                 time.sleep(0.001)
@@ -632,6 +650,7 @@ class I2PInterfacePeer(Interface):
                 self.socket.sendall(data)
                 self.writing = False
                 self.txb += len(data)
+                self.last_write = time.time()
                 
                 if hasattr(self, "parent_interface") and self.parent_interface != None and self.parent_count:
                     self.parent_interface.txb += len(data)
@@ -642,8 +661,59 @@ class I2PInterfacePeer(Interface):
                 self.teardown()
 
 
+    def read_watchdog(self):
+        while self.wd_reset:
+            time.sleep(0.25)
+
+        should_run = True
+        try:
+            while should_run and not self.wd_reset:
+                time.sleep(1)
+
+                if (time.time()-self.last_read > I2PInterfacePeer.I2P_PROBE_AFTER*2):
+                    if self.i2p_tunnel_state != I2PInterfacePeer.TUNNEL_STATE_STALE:
+                        RNS.log("I2P tunnel became unresponsive", RNS.LOG_DEBUG)
+
+                    self.i2p_tunnel_state = I2PInterfacePeer.TUNNEL_STATE_STALE
+                else:
+                    self.i2p_tunnel_state = I2PInterfacePeer.TUNNEL_STATE_ACTIVE
+
+                if (time.time()-self.last_write > I2PInterfacePeer.I2P_PROBE_AFTER*1):
+                    try:
+                        if self.socket != None:
+                            self.socket.sendall(bytes([HDLC.FLAG, HDLC.FLAG]))
+                    except Exception as e:
+                        RNS.log("An error ocurred while sending I2P keepalive. The contained exception was: "+str(e), RNS.LOG_ERROR)
+                        self.shutdown_socket(self.socket)
+                        should_run = False
+                
+                if (time.time()-self.last_read > I2PInterfacePeer.I2P_READ_TIMEOUT):
+                    RNS.log("I2P socket is unresponsive, restarting...", RNS.LOG_WARNING)
+                    if self.socket != None:
+                        try:
+                            self.socket.shutdown(socket.SHUT_RDWR)
+                        except Exception as e:
+                            RNS.log("Error while shutting down socket for "+str(self)+": "+str(e))
+
+                        try:
+                            self.socket.close()
+                        except Exception as e:
+                            RNS.log("Error while closing socket for "+str(self)+": "+str(e))
+
+                    should_run = False
+
+                self.wd_reset = False
+
+        finally:
+            self.wd_reset = False
+
     def read_loop(self):
         try:
+            self.last_read  = time.time()
+            self.last_write = time.time()
+
+            wd_thread = threading.Thread(target=self.read_watchdog, daemon=True).start()
+
             in_frame = False
             escape = False
             data_buffer = b""
@@ -653,6 +723,7 @@ class I2PInterfacePeer(Interface):
                 data_in = self.socket.recv(4096)
                 if len(data_in) > 0:
                     pointer = 0
+                    self.last_read = time.time()
                     while pointer < len(data_in):
                         byte = data_in[pointer]
                         pointer += 1
@@ -661,7 +732,7 @@ class I2PInterfacePeer(Interface):
                             # Read loop for KISS framing
                             if (in_frame and byte == KISS.FEND and command == KISS.CMD_DATA):
                                 in_frame = False
-                                self.processIncoming(data_buffer)
+                                self.process_incoming(data_buffer)
                             elif (byte == KISS.FEND):
                                 in_frame = True
                                 command = KISS.CMD_UNKNOWN
@@ -688,7 +759,7 @@ class I2PInterfacePeer(Interface):
                             # Read loop for HDLC framing
                             if (in_frame and byte == HDLC.FLAG):
                                 in_frame = False
-                                self.processIncoming(data_buffer)
+                                self.process_incoming(data_buffer)
                             elif (byte == HDLC.FLAG):
                                 in_frame = True
                                 data_buffer = b""
@@ -705,6 +776,11 @@ class I2PInterfacePeer(Interface):
                                     data_buffer = data_buffer+bytes([byte])
                 else:
                     self.online = False
+
+                    self.wd_reset = True
+                    time.sleep(2)
+                    self.wd_reset = False
+
                     if self.initiator and not self.detached:
                         RNS.log("Socket for "+str(self)+" was closed, attempting to reconnect...", RNS.LOG_WARNING)
                         self.reconnect()
@@ -739,8 +815,8 @@ class I2PInterfacePeer(Interface):
         self.IN = False
 
         if hasattr(self, "parent_interface") and self.parent_interface != None:
-            if self.parent_interface.clients > 0:
-                self.parent_interface.clients -= 1
+            while self in self.parent_interface.spawned_interfaces:
+                self.parent_interface.spawned_interfaces.remove(self)
 
         if self in RNS.Transport.interfaces:
             if not self.initiator:
@@ -753,15 +829,28 @@ class I2PInterfacePeer(Interface):
 
 class I2PInterface(Interface):
     BITRATE_GUESS      = 256*1000
+    DEFAULT_IFAC_SIZE  = 16
 
-    def __init__(self, owner, name, rns_storagepath, peers, connectable = False):
-        self.rxb = 0
-        self.txb = 0
+    @property
+    def clients(self):
+        return len(self.spawned_interfaces)
+
+    def __init__(self, owner, configuration):
+        super().__init__()
+
+        c = Interface.get_config_obj(configuration)
+        name = c["name"]
+        rns_storagepath = c["storagepath"]
+        peers = c.as_list("peers") if "peers" in c else None
+        connectable = c.as_bool("connectable") if "connectable" in c else False
+        ifac_size = c["ifac_size"] if "ifac_size" in c else None
+        ifac_netname = c["ifac_netname"] if "ifac_netname" in c else None
+        ifac_netkey = c["ifac_netkey"] if "ifac_netkey" in c else None
         
         self.HW_MTU = 1064
 
         self.online = False
-        self.clients = 0
+        self.spawned_interfaces = []
         self.owner = owner
         self.connectable = connectable
         self.i2p_tunneled = True
@@ -780,6 +869,9 @@ class I2PInterface(Interface):
         self.bind_port   = self.i2p.get_free_port()
         self.address = (self.bind_ip, self.bind_port)
         self.bitrate = I2PInterface.BITRATE_GUESS
+        self.ifac_size = ifac_size
+        self.ifac_netname = ifac_netname
+        self.ifac_netkey = ifac_netkey
 
         self.online = False
 
@@ -850,9 +942,27 @@ class I2PInterface(Interface):
         spawned_interface.parent_interface = self
         spawned_interface.online = True
         spawned_interface.bitrate = self.bitrate
+
         spawned_interface.ifac_size = self.ifac_size
         spawned_interface.ifac_netname = self.ifac_netname
         spawned_interface.ifac_netkey = self.ifac_netkey
+        if spawned_interface.ifac_netname != None or spawned_interface.ifac_netkey != None:
+            ifac_origin = b""
+            if spawned_interface.ifac_netname != None:
+                ifac_origin += RNS.Identity.full_hash(spawned_interface.ifac_netname.encode("utf-8"))
+            if spawned_interface.ifac_netkey != None:
+                ifac_origin += RNS.Identity.full_hash(spawned_interface.ifac_netkey.encode("utf-8"))
+
+            ifac_origin_hash = RNS.Identity.full_hash(ifac_origin)
+            spawned_interface.ifac_key = RNS.Cryptography.hkdf(
+                length=64,
+                derive_from=ifac_origin_hash,
+                salt=RNS.Reticulum.IFAC_SALT,
+                context=None
+            )
+            spawned_interface.ifac_identity = RNS.Identity.from_bytes(spawned_interface.ifac_key)
+            spawned_interface.ifac_signature = spawned_interface.ifac_identity.sign(RNS.Identity.full_hash(spawned_interface.ifac_key))
+
         spawned_interface.announce_rate_target = self.announce_rate_target
         spawned_interface.announce_rate_grace = self.announce_rate_grace
         spawned_interface.announce_rate_penalty = self.announce_rate_penalty
@@ -860,12 +970,20 @@ class I2PInterface(Interface):
         spawned_interface.HW_MTU = self.HW_MTU
         RNS.log("Spawned new I2PInterface Peer: "+str(spawned_interface), RNS.LOG_VERBOSE)
         RNS.Transport.interfaces.append(spawned_interface)
-        self.clients += 1
+        while spawned_interface in self.spawned_interfaces:
+            self.spawned_interfaces.remove(spawned_interface)
+        self.spawned_interfaces.append(spawned_interface)
         spawned_interface.read_loop()
 
-    def processOutgoing(self, data):
+    def process_outgoing(self, data):
         pass
 
+    def received_announce(self, from_spawned=False):
+        if from_spawned: self.ia_freq_deque.append(time.time())
+
+    def sent_announce(self, from_spawned=False):
+        if from_spawned: self.oa_freq_deque.append(time.time())
+
     def detach(self):
         RNS.log("Detaching "+str(self), RNS.LOG_DEBUG)
         self.i2p.stop()

RNS/Interfaces/Interface.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -23,6 +23,8 @@
 import RNS
 import time
 import threading
+from collections import deque
+from RNS.vendor.configobj import ConfigObj
 
 class Interface:
     IN  = False
@@ -39,18 +41,190 @@ class Interface:
     MODE_BOUNDARY       = 0x05
     MODE_GATEWAY        = 0x06
 
-    # Which interface modes a Transport Node
-    # should actively discover paths for.
-    DISCOVER_PATHS_FOR  = [MODE_ACCESS_POINT, MODE_GATEWAY]
+    # Which interface modes a Transport Node should
+    # actively discover paths for.
+    DISCOVER_PATHS_FOR  = [MODE_ACCESS_POINT, MODE_GATEWAY, MODE_ROAMING]
+
+    # How many samples to use for announce
+    # frequency calculations
+    IA_FREQ_SAMPLES     = 6
+    OA_FREQ_SAMPLES     = 6
+
+    # Maximum amount of ingress limited announces
+    # to hold at any given time.
+    MAX_HELD_ANNOUNCES  = 256
+
+    # How long a spawned interface will be
+    # considered to be newly created. Two
+    # hours by default.
+    IC_NEW_TIME              = 2*60*60
+    IC_BURST_FREQ_NEW        = 3.5
+    IC_BURST_FREQ            = 12
+    IC_BURST_HOLD            = 1*60
+    IC_BURST_PENALTY         = 5*60
+    IC_HELD_RELEASE_INTERVAL = 30
+
+    AUTOCONFIGURE_MTU = False
 
     def __init__(self):
-        self.rxb = 0
-        self.txb = 0
-        self.online = False
+        self.rxb      = 0
+        self.txb      = 0
+        self.created  = time.time()
+        self.detached = False
+        self.online   = False
+        self.bitrate  = 62500
+        self.HW_MTU   = None
+
+        self.ingress_control = True
+        self.ic_max_held_announces = Interface.MAX_HELD_ANNOUNCES
+        self.ic_burst_hold = Interface.IC_BURST_HOLD
+        self.ic_burst_active = False
+        self.ic_burst_activated = 0
+        self.ic_held_release = 0
+        self.ic_burst_freq_new = Interface.IC_BURST_FREQ_NEW
+        self.ic_burst_freq = Interface.IC_BURST_FREQ
+        self.ic_new_time = Interface.IC_NEW_TIME
+        self.ic_burst_penalty = Interface.IC_BURST_PENALTY
+        self.ic_held_release_interval = Interface.IC_HELD_RELEASE_INTERVAL
+        self.held_announces = {}
+
+        self.ia_freq_deque = deque(maxlen=Interface.IA_FREQ_SAMPLES)
+        self.oa_freq_deque = deque(maxlen=Interface.OA_FREQ_SAMPLES)
 
     def get_hash(self):
         return RNS.Identity.full_hash(str(self).encode("utf-8"))
 
+    # This is a generic function for determining when an interface
+    # should activate ingress limiting. Since this can vary for
+    # different interface types, this function should be overwritten
+    # in case a particular interface requires a different approach.
+    def should_ingress_limit(self):
+        if self.ingress_control:
+            freq_threshold = self.ic_burst_freq_new if self.age() < self.ic_new_time else self.ic_burst_freq
+            ia_freq = self.incoming_announce_frequency()
+
+            if self.ic_burst_active:
+                if ia_freq < freq_threshold and time.time() > self.ic_burst_activated+self.ic_burst_hold:
+                    self.ic_burst_active = False
+                    self.ic_held_release = time.time() + self.ic_burst_penalty
+                return True
+
+            else:
+                if ia_freq > freq_threshold:
+                    self.ic_burst_active = True
+                    self.ic_burst_activated = time.time()
+                    return True
+
+                else:
+                    return False
+
+        else:
+            return False
+
+    def optimise_mtu(self):
+        if self.AUTOCONFIGURE_MTU:
+            if self.bitrate   > 16_000_000:
+                self.HW_MTU = 262144
+            elif self.bitrate > 8_000_000:
+                self.HW_MTU = 131072
+            elif self.bitrate > 4_000_000:
+                self.HW_MTU = 65536
+            elif self.bitrate > 2_000_000:
+                self.HW_MTU = 32768
+            elif self.bitrate > 1_000_000:
+                self.HW_MTU = 16384
+            elif self.bitrate > 500_000:
+                self.HW_MTU = 8192
+            elif self.bitrate > 250_000:
+                self.HW_MTU = 4096
+            elif self.bitrate > 125_000:
+                self.HW_MTU = 2048
+            elif self.bitrate > 62_500:
+                self.HW_MTU = 1024
+            else:
+                self.HW_MTU = None
+
+        RNS.log(f"{self} hardware MTU set to {self.HW_MTU}", RNS.LOG_DEBUG) # TODO: Remove debug
+
+    def age(self):
+        return time.time()-self.created
+
+    def hold_announce(self, announce_packet):
+        if announce_packet.destination_hash in self.held_announces:
+            self.held_announces[announce_packet.destination_hash] = announce_packet
+        elif not len(self.held_announces) >= self.ic_max_held_announces:
+            self.held_announces[announce_packet.destination_hash] = announce_packet
+
+    def process_held_announces(self):
+        try:
+            if not self.should_ingress_limit() and len(self.held_announces) > 0 and time.time() > self.ic_held_release:
+                freq_threshold = self.ic_burst_freq_new if self.age() < self.ic_new_time else self.ic_burst_freq
+                ia_freq = self.incoming_announce_frequency()
+                if ia_freq < freq_threshold:
+                    selected_announce_packet = None
+                    min_hops = RNS.Transport.PATHFINDER_M
+                    for destination_hash in self.held_announces:
+                        announce_packet = self.held_announces[destination_hash]
+                        if announce_packet.hops < min_hops:
+                            min_hops = announce_packet.hops
+                            selected_announce_packet = announce_packet
+
+                    if selected_announce_packet != None:
+                        RNS.log("Releasing held announce packet "+str(selected_announce_packet)+" from "+str(self), RNS.LOG_EXTREME)
+                        self.ic_held_release = time.time() + self.ic_held_release_interval
+                        self.held_announces.pop(selected_announce_packet.destination_hash)
+                        def release():
+                            RNS.Transport.inbound(selected_announce_packet.raw, selected_announce_packet.receiving_interface)
+                        threading.Thread(target=release, daemon=True).start()
+        
+        except Exception as e:
+            RNS.log("An error occurred while processing held announces for "+str(self), RNS.LOG_ERROR)
+            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+    def received_announce(self):
+        self.ia_freq_deque.append(time.time())
+        if hasattr(self, "parent_interface") and self.parent_interface != None:
+            self.parent_interface.received_announce(from_spawned=True)
+
+    def sent_announce(self):
+        self.oa_freq_deque.append(time.time())
+        if hasattr(self, "parent_interface") and self.parent_interface != None:
+            self.parent_interface.sent_announce(from_spawned=True)
+
+    def incoming_announce_frequency(self):
+        if not len(self.ia_freq_deque) > 1:
+            return 0
+        else:
+            dq_len = len(self.ia_freq_deque)
+            delta_sum = 0
+            for i in range(1,dq_len):
+                delta_sum += self.ia_freq_deque[i]-self.ia_freq_deque[i-1]
+            delta_sum += time.time() - self.ia_freq_deque[dq_len-1]
+            
+            if delta_sum == 0:
+                avg = 0
+            else:
+                avg = 1/(delta_sum/(dq_len))
+
+            return avg
+
+    def outgoing_announce_frequency(self):
+        if not len(self.oa_freq_deque) > 1:
+            return 0
+        else:
+            dq_len = len(self.oa_freq_deque)
+            delta_sum = 0
+            for i in range(1,dq_len):
+                delta_sum += self.oa_freq_deque[i]-self.oa_freq_deque[i-1]
+            delta_sum += time.time() - self.oa_freq_deque[dq_len-1]
+            
+            if delta_sum == 0:
+                avg = 0
+            else:
+                avg = 1/(delta_sum/(dq_len))
+
+            return avg
+
     def process_announce_queue(self):
         if not hasattr(self, "announce_cap"):
             self.announce_cap = RNS.Reticulum.ANNOUNCE_CAP
@@ -78,7 +252,8 @@ class Interface:
                     wait_time = (tx_time / self.announce_cap)
                     self.announce_allowed_at = now + wait_time
 
-                    self.processOutgoing(selected["raw"])
+                    self.process_outgoing(selected["raw"])
+                    self.sent_announce()
 
                     if selected in self.announce_queue:
                         self.announce_queue.remove(selected)
@@ -93,4 +268,15 @@ class Interface:
                 RNS.log("The announce queue for this interface has been cleared.", RNS.LOG_ERROR)
 
     def detach(self):
-        pass
+        pass
+
+    @staticmethod
+    def get_config_obj(config_in):
+        if type(config_in) == ConfigObj:
+            return config_in
+        else:
+            try:
+                return ConfigObj(config_in)
+            except Exception as e:
+                RNS.log(f"Could not parse supplied configuration data. The contained exception was: {e}", RNS.LOG_ERROR)
+                raise SystemError("Invalid configuration data supplied")

RNS/Interfaces/KISSInterface.py

@@ -20,7 +20,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from .Interface import Interface
+from RNS.Interfaces.Interface import Interface
 from time import sleep
 import sys
 import threading
@@ -52,6 +52,7 @@ class KISS():
 class KISSInterface(Interface):
     MAX_CHUNK = 32768
     BITRATE_GUESS = 1200
+    DEFAULT_IFAC_SIZE = 8
 
     owner    = None
     port     = None
@@ -61,7 +62,7 @@ class KISSInterface(Interface):
     stopbits = None
     serial   = None
 
-    def __init__(self, owner, name, port, speed, databits, parity, stopbits, preamble, txtail, persistence, slottime, flow_control, beacon_interval, beacon_data):
+    def __init__(self, owner, configuration):
         import importlib
         if importlib.util.find_spec('serial') != None:
             import serial
@@ -70,8 +71,25 @@ class KISSInterface(Interface):
             RNS.log("You can install one with the command: python3 -m pip install pyserial", RNS.LOG_CRITICAL)
             RNS.panic()
 
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
+
+        c = Interface.get_config_obj(configuration)
+        name = c["name"]
+        preamble = int(c["preamble"]) if "preamble" in c else None
+        txtail = int(c["txtail"]) if "txtail" in c else None
+        persistence = int(c["persistence"]) if "persistence" in c else None
+        slottime = int(c["slottime"]) if "slottime" in c else None
+        flow_control = c.as_bool("flow_control") if "flow_control" in c else False
+        port = c["port"] if "port" in c else None
+        speed = int(c["speed"]) if "speed" in c else 9600
+        databits = int(c["databits"]) if "databits" in c else 8
+        parity = c["parity"] if "parity" in c else "N"
+        stopbits = int(c["stopbits"]) if "stopbits" in c else 1
+        beacon_interval = int(c["id_interval"]) if "id_interval" in c else None
+        beacon_data = c["id_callsign"] if "id_callsign" in c else None
+
+        if port == None:
+            raise ValueError("No port specified for serial interface")
         
         self.HW_MTU = 564
         
@@ -218,12 +236,12 @@ class KISSInterface(Interface):
                 raise IOError("Could not enable KISS interface flow control")
 
 
-    def processIncoming(self, data):
+    def process_incoming(self, data):
         self.rxb += len(data)  
         self.owner.inbound(data, self)
 
 
-    def processOutgoing(self,data):
+    def process_outgoing(self,data):
         datalen = len(data)
         if self.online:
             if self.interface_ready:
@@ -257,7 +275,7 @@ class KISSInterface(Interface):
         if len(self.packet_queue) > 0:
             data = self.packet_queue.pop(0)
             self.interface_ready = True
-            self.processOutgoing(data)
+            self.process_outgoing(data)
         elif len(self.packet_queue) == 0:
             self.interface_ready = True
 
@@ -276,7 +294,7 @@ class KISSInterface(Interface):
 
                     if (in_frame and byte == KISS.FEND and command == KISS.CMD_DATA):
                         in_frame = False
-                        self.processIncoming(data_buffer)
+                        self.process_incoming(data_buffer)
                     elif (byte == KISS.FEND):
                         in_frame = True
                         command = KISS.CMD_UNKNOWN
@@ -320,7 +338,13 @@ class KISSInterface(Interface):
                             if time.time() > self.first_tx + self.beacon_i:
                                 RNS.log("Interface "+str(self)+" is transmitting beacon data: "+str(self.beacon_d.decode("utf-8")), RNS.LOG_DEBUG)
                                 self.first_tx = None
-                                self.processOutgoing(self.beacon_d)
+                                
+                                # Pad to minimum length
+                                frame = bytearray(self.beacon_d)
+                                while len(frame) < 15:
+                                    frame.append(0x00)
+                                    
+                                self.process_outgoing(bytes(frame))
 
         except Exception as e:
             self.online = False
@@ -349,5 +373,8 @@ class KISSInterface(Interface):
 
         RNS.log("Reconnected serial port for "+str(self))
 
+    def should_ingress_limit(self):
+        return False
+
     def __str__(self):
         return "KISSInterface["+self.name+"]"

RNS/Interfaces/LocalInterface.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -20,7 +20,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from .Interface import Interface
+from RNS.Interfaces.Interface import Interface
 import socketserver
 import threading
 import socket
@@ -28,6 +28,7 @@ import time
 import sys
 import os
 import RNS
+from threading import Lock
 
 class HDLC():
     FLAG              = 0x7E
@@ -41,19 +42,22 @@ class HDLC():
         return data
 
 class ThreadingTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
-    pass
+    def server_bind(self):
+        if RNS.vendor.platformutils.is_windows():
+            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_EXCLUSIVEADDRUSE, 1)
+        else:
+            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+        self.socket.bind(self.server_address)
+        self.server_address = self.socket.getsockname()
 
 class LocalClientInterface(Interface):
-    RECONNECT_WAIT = 3
+    RECONNECT_WAIT = 8
+    AUTOCONFIGURE_MTU = True
 
     def __init__(self, owner, name, target_port = None, connected_socket=None):
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
 
-        # TODO: Remove at some point
-        # self.rxptime = 0
-        
-        self.HW_MTU = 1064
+        self.HW_MTU = 262144
 
         self.online  = False
         
@@ -72,6 +76,7 @@ class LocalClientInterface(Interface):
             self.target_ip   = None
             self.target_port = None
             self.socket      = connected_socket
+            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
 
             self.is_connected_to_shared_instance = False
 
@@ -82,10 +87,12 @@ class LocalClientInterface(Interface):
             self.connect()
 
         self.owner   = owner
-        self.bitrate = 1000*1000*1000
+        self.bitrate = 1_000_000_000
         self.online  = True
         self.writing = False
 
+        self._force_bitrate = False
+
         self.announce_rate_target  = None
         self.announce_rate_grace   = None
         self.announce_rate_penalty = None
@@ -95,8 +102,12 @@ class LocalClientInterface(Interface):
             thread.daemon = True
             thread.start()
 
+    def should_ingress_limit(self):
+        return False
+
     def connect(self):
         self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
         self.socket.connect((self.target_ip, self.target_port))
 
         self.online = True
@@ -129,14 +140,17 @@ class LocalClientInterface(Interface):
                 thread = threading.Thread(target=self.read_loop)
                 thread.daemon = True
                 thread.start()
-                RNS.Transport.shared_connection_reappeared()
+                def job():
+                    time.sleep(LocalClientInterface.RECONNECT_WAIT+2)
+                    RNS.Transport.shared_connection_reappeared()
+                threading.Thread(target=job, daemon=True).start()
         
         else:
             RNS.log("Attempt to reconnect on a non-initiator shared local interface. This should not happen.", RNS.LOG_ERROR)
             raise IOError("Attempt to reconnect on a non-initiator local interface")
 
 
-    def processIncoming(self, data):
+    def process_incoming(self, data):
         self.rxb += len(data)
         if hasattr(self, "parent_interface") and self.parent_interface != None:
             self.parent_interface.rxb += len(data)
@@ -150,10 +164,20 @@ class LocalClientInterface(Interface):
         # duration = time.time() - processing_start
         # self.rxptime += duration
 
-    def processOutgoing(self, data):
+    def process_outgoing(self, data):
         if self.online:
             try:
                 self.writing = True
+
+                if self._force_bitrate:
+                    if not hasattr(self, "send_lock"):
+                        self.send_lock = Lock()
+
+                    with self.send_lock:
+                        # RNS.log(f"Simulating latency of {RNS.prettytime(s)} for {len(data)} bytes", RNS.LOG_EXTREME)
+                        s = len(data) / self.bitrate * 8
+                        time.sleep(s)
+
                 data = bytes([HDLC.FLAG])+HDLC.escape(data)+bytes([HDLC.FLAG])
                 self.socket.sendall(data)
                 self.writing = False
@@ -171,32 +195,31 @@ class LocalClientInterface(Interface):
         try:
             in_frame = False
             escape = False
+            frame_buffer = b""
+            data_in = b""
             data_buffer = b""
 
             while True:
                 data_in = self.socket.recv(4096)
                 if len(data_in) > 0:
-                    pointer = 0
-                    while pointer < len(data_in):
-                        byte = data_in[pointer]
-                        pointer += 1
-                        if (in_frame and byte == HDLC.FLAG):
-                            in_frame = False
-                            self.processIncoming(data_buffer)
-                        elif (byte == HDLC.FLAG):
-                            in_frame = True
-                            data_buffer = b""
-                        elif (in_frame and len(data_buffer) < self.HW_MTU):
-                            if (byte == HDLC.ESC):
-                                escape = True
+                    frame_buffer += data_in
+                    flags_remaining = True
+                    while flags_remaining:
+                        frame_start = frame_buffer.find(HDLC.FLAG)
+                        if frame_start != -1:
+                            frame_end = frame_buffer.find(HDLC.FLAG, frame_start+1)
+                            if frame_end != -1:
+                                frame = frame_buffer[frame_start+1:frame_end]
+                                frame = frame.replace(bytes([HDLC.ESC, HDLC.FLAG ^ HDLC.ESC_MASK]), bytes([HDLC.FLAG]))
+                                frame = frame.replace(bytes([HDLC.ESC, HDLC.ESC  ^ HDLC.ESC_MASK]), bytes([HDLC.ESC]))
+                                if len(frame) > RNS.Reticulum.HEADER_MINSIZE:
+                                    self.process_incoming(frame)
+                                frame_buffer = frame_buffer[frame_end:]
                             else:
-                                if (escape):
-                                    if (byte == HDLC.FLAG ^ HDLC.ESC_MASK):
-                                        byte = HDLC.FLAG
-                                    if (byte == HDLC.ESC  ^ HDLC.ESC_MASK):
-                                        byte = HDLC.ESC
-                                    escape = False
-                                data_buffer = data_buffer+bytes([byte])
+                                flags_remaining = False
+                        else:
+                            flags_remaining = False
+
                 else:
                     self.online = False
                     if self.is_connected_to_shared_instance and not self.detached:
@@ -223,12 +246,14 @@ class LocalClientInterface(Interface):
                     self.detached = True
                     
                     try:
-                        self.socket.shutdown(socket.SHUT_RDWR)
+                        if self.socket != None:
+                            self.socket.shutdown(socket.SHUT_RDWR)
                     except Exception as e:
                         RNS.log("Error while shutting down socket for "+str(self)+": "+str(e))
 
                     try:
-                        self.socket.close()
+                        if self.socket != None:
+                            self.socket.close()
                     except Exception as e:
                         RNS.log("Error while closing socket for "+str(self)+": "+str(e))
 
@@ -246,7 +271,8 @@ class LocalClientInterface(Interface):
             RNS.Transport.local_client_interfaces.remove(self)
             if hasattr(self, "parent_interface") and self.parent_interface != None:
                 self.parent_interface.clients -= 1
-                RNS.Transport.owner._should_persist_data()
+                if hasattr(RNS.Transport, "owner") and RNS.Transport.owner != None:
+                    RNS.Transport.owner._should_persist_data()
 
         if nowarning == False:
             RNS.log("The interface "+str(self)+" experienced an unrecoverable error and is being torn down. Restart Reticulum to attempt to open this interface again.", RNS.LOG_ERROR)
@@ -265,10 +291,10 @@ class LocalClientInterface(Interface):
 
 
 class LocalServerInterface(Interface):
+    AUTOCONFIGURE_MTU = True
 
     def __init__(self, owner, bindport=None):
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
         self.online = False
         self.clients = 0
         
@@ -292,7 +318,6 @@ class LocalServerInterface(Interface):
 
             address = (self.bind_ip, self.bind_port)
 
-            ThreadingTCPServer.allow_reuse_address = True
             self.server = ThreadingTCPServer(address, handlerFactory(self.incoming_connection))
 
             thread = threading.Thread(target=self.server.serve_forever)
@@ -317,15 +342,23 @@ class LocalServerInterface(Interface):
         spawned_interface.target_port = str(handler.client_address[1])
         spawned_interface.parent_interface = self
         spawned_interface.bitrate = self.bitrate
-        RNS.log("Accepting new connection to shared instance: "+str(spawned_interface), RNS.LOG_EXTREME)
+        if hasattr(self, "_force_bitrate"):
+            spawned_interface._force_bitrate = self._force_bitrate
+        # RNS.log("Accepting new connection to shared instance: "+str(spawned_interface), RNS.LOG_EXTREME)
         RNS.Transport.interfaces.append(spawned_interface)
         RNS.Transport.local_client_interfaces.append(spawned_interface)
         self.clients += 1
         spawned_interface.read_loop()
 
-    def processOutgoing(self, data):
+    def process_outgoing(self, data):
         pass
 
+    def received_announce(self, from_spawned=False):
+        if from_spawned: self.ia_freq_deque.append(time.time())
+
+    def sent_announce(self, from_spawned=False):
+        if from_spawned: self.oa_freq_deque.append(time.time())
+
     def __str__(self):
         return "Shared Instance["+str(self.bind_port)+"]"
 

RNS/Interfaces/PipeInterface.py

@@ -20,7 +20,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from .Interface import Interface
+from RNS.Interfaces.Interface import Interface
 from time import sleep
 import sys
 import threading
@@ -46,17 +46,25 @@ class HDLC():
 class PipeInterface(Interface):
     MAX_CHUNK = 32768
     BITRATE_GUESS = 1*1000*1000
+    DEFAULT_IFAC_SIZE = 8
 
     owner    = None
     command  = None
     
-    def __init__(self, owner, name, command, respawn_delay):
+    def __init__(self, owner, configuration):
+        super().__init__()
+
+        c = Interface.get_config_obj(configuration)
+        name = c["name"]
+        command = c["command"] if "command" in c else None
+        respawn_delay = c.as_float("respawn_delay") if "respawn_delay" in c else None
+
+        if command == None:
+            raise ValueError("No command specified for PipeInterface")
+
         if respawn_delay == None:
             respawn_delay = 5
 
-        self.rxb = 0
-        self.txb = 0
-
         self.HW_MTU = 1064
         
         self.owner    = owner
@@ -102,12 +110,12 @@ class PipeInterface(Interface):
         RNS.log("Subprocess pipe for "+str(self)+" is now connected", RNS.LOG_VERBOSE)
 
 
-    def processIncoming(self, data):
+    def process_incoming(self, data):
         self.rxb += len(data)            
         self.owner.inbound(data, self)
 
 
-    def processOutgoing(self,data):
+    def process_outgoing(self,data):
         if self.online:
             data = bytes([HDLC.FLAG])+HDLC.escape(data)+bytes([HDLC.FLAG])
             written = self.process.stdin.write(data)
@@ -135,7 +143,7 @@ class PipeInterface(Interface):
 
                     if (in_frame and byte == HDLC.FLAG):
                         in_frame = False
-                        self.processIncoming(data_buffer)
+                        self.process_incoming(data_buffer)
                     elif (byte == HDLC.FLAG):
                         in_frame = True
                         data_buffer = b""

RNS/Interfaces/SerialInterface.py

@@ -20,7 +20,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from .Interface import Interface
+from RNS.Interfaces.Interface import Interface
 from time import sleep
 import sys
 import threading
@@ -42,6 +42,7 @@ class HDLC():
 
 class SerialInterface(Interface):
     MAX_CHUNK = 32768
+    DEFAULT_IFAC_SIZE = 8
 
     owner    = None
     port     = None
@@ -51,7 +52,7 @@ class SerialInterface(Interface):
     stopbits = None
     serial   = None
 
-    def __init__(self, owner, name, port, speed, databits, parity, stopbits):
+    def __init__(self, owner, configuration):
         import importlib
         if importlib.util.find_spec('serial') != None:
             import serial
@@ -60,8 +61,18 @@ class SerialInterface(Interface):
             RNS.log("You can install one with the command: python3 -m pip install pyserial", RNS.LOG_CRITICAL)
             RNS.panic()
 
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
+
+        c = Interface.get_config_obj(configuration)
+        name = c["name"]
+        port = c["port"] if "port" in c else None
+        speed = int(c["speed"]) if "speed" in c else 9600
+        databits = int(c["databits"]) if "databits" in c else 8
+        parity = c["parity"] if "parity" in c else "N"
+        stopbits = int(c["stopbits"]) if "stopbits" in c else 1
+
+        if port == None:
+            raise ValueError("No port specified for serial interface")
 
         self.HW_MTU = 564
         
@@ -122,12 +133,12 @@ class SerialInterface(Interface):
         RNS.log("Serial port "+self.port+" is now open", RNS.LOG_VERBOSE)
 
 
-    def processIncoming(self, data):
+    def process_incoming(self, data):
         self.rxb += len(data)            
         self.owner.inbound(data, self)
 
 
-    def processOutgoing(self,data):
+    def process_outgoing(self,data):
         if self.online:
             data = bytes([HDLC.FLAG])+HDLC.escape(data)+bytes([HDLC.FLAG])
             written = self.serial.write(data)
@@ -150,7 +161,7 @@ class SerialInterface(Interface):
 
                     if (in_frame and byte == HDLC.FLAG):
                         in_frame = False
-                        self.processIncoming(data_buffer)
+                        self.process_incoming(data_buffer)
                     elif (byte == HDLC.FLAG):
                         in_frame = True
                         data_buffer = b""
@@ -201,5 +212,8 @@ class SerialInterface(Interface):
 
         RNS.log("Reconnected serial port for "+str(self))
 
+    def should_ingress_limit(self):
+        return False
+
     def __str__(self):
         return "SerialInterface["+self.name+"]"

RNS/Interfaces/TCPInterface.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2024 Mark Qvist / unsigned.io
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -20,7 +20,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from .Interface import Interface
+from RNS.Interfaces.Interface import Interface
 import socketserver
 import threading
 import platform
@@ -30,6 +30,9 @@ import sys
 import os
 import RNS
 
+class TCPInterface():
+    HW_MTU            = 262144
+
 class HDLC():
     FLAG              = 0x7E
     ESC               = 0x7D
@@ -58,8 +61,13 @@ class KISS():
 class ThreadingTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
     pass
 
+class ThreadingTCP6Server(socketserver.ThreadingMixIn, socketserver.TCPServer):
+    address_family = socket.AF_INET6
+
 class TCPClientInterface(Interface):
     BITRATE_GUESS = 10*1000*1000
+    DEFAULT_IFAC_SIZE = 16
+    AUTOCONFIGURE_MTU = True
 
     RECONNECT_WAIT = 5
     RECONNECT_MAX_TRIES = None
@@ -78,12 +86,21 @@ class TCPClientInterface(Interface):
     I2P_PROBE_INTERVAL = 9
     I2P_PROBES = 5
 
-    def __init__(self, owner, name, target_ip=None, target_port=None, connected_socket=None, max_reconnect_tries=None, kiss_framing=False, i2p_tunneled = False, connect_timeout = None):
-        self.rxb = 0
-        self.txb = 0
-        
-        self.HW_MTU = 1064
+    def __init__(self, owner, configuration, connected_socket=None):
+        super().__init__()
+
+        c = Interface.get_config_obj(configuration)
+        name = c["name"]
+        target_ip = c["target_host"] if "target_host" in c and c["target_host"] != None else None
+        target_port = int(c["target_port"]) if "target_port" in c and c["target_host"] != None else None
+        kiss_framing = False
+        if "kiss_framing" in c and c.as_bool("kiss_framing") == True:
+            kiss_framing = True
+        i2p_tunneled = c.as_bool("i2p_tunneled") if "i2p_tunneled" in c else False
+        connect_timeout = c.as_int("connect_timeout") if "connect_timeout" in c else None
+        max_reconnect_tries = c.as_int("max_reconnect_tries") if "max_reconnect_tries" in c else None
         
+        self.HW_MTU           = TCPInterface.HW_MTU
         self.IN               = True
         self.OUT              = False
         self.socket           = None
@@ -117,6 +134,8 @@ class TCPClientInterface(Interface):
             elif platform.system() == "Darwin":
                 self.set_timeouts_osx()
 
+            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+
         elif target_ip != None and target_port != None:
             self.receives    = True
             self.target_ip   = target_ip
@@ -176,19 +195,21 @@ class TCPClientInterface(Interface):
             self.socket.setsockopt(socket.IPPROTO_TCP, TCP_KEEPIDLE, int(TCPClientInterface.I2P_PROBE_AFTER))
         
     def detach(self):
+        self.online = False
         if self.socket != None:
             if hasattr(self.socket, "close"):
                 if callable(self.socket.close):
-                    RNS.log("Detaching "+str(self), RNS.LOG_DEBUG)
                     self.detached = True
                     
                     try:
-                        self.socket.shutdown(socket.SHUT_RDWR)
+                        if self.socket != None:
+                            self.socket.shutdown(socket.SHUT_RDWR)
                     except Exception as e:
                         RNS.log("Error while shutting down socket for "+str(self)+": "+str(e))
 
                     try:
-                        self.socket.close()
+                        if self.socket != None:
+                            self.socket.close()
                     except Exception as e:
                         RNS.log("Error while closing socket for "+str(self)+": "+str(e))
 
@@ -199,9 +220,14 @@ class TCPClientInterface(Interface):
             if initial:
                 RNS.log("Establishing TCP connection for "+str(self)+"...", RNS.LOG_DEBUG)
 
-            self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+            address_info = socket.getaddrinfo(self.target_ip, self.target_port, proto=socket.IPPROTO_TCP)[0]
+            address_family = address_info[0]
+            target_address = address_info[4]
+
+            self.socket = socket.socket(address_family, socket.SOCK_STREAM)
             self.socket.settimeout(TCPClientInterface.INITIAL_CONNECT_TIMEOUT)
-            self.socket.connect((self.target_ip, self.target_port))
+            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+            self.socket.connect(target_address)
             self.socket.settimeout(None)
             self.online  = True
 
@@ -263,15 +289,16 @@ class TCPClientInterface(Interface):
             RNS.log("Attempt to reconnect on a non-initiator TCP interface. This should not happen.", RNS.LOG_ERROR)
             raise IOError("Attempt to reconnect on a non-initiator TCP interface")
 
-    def processIncoming(self, data):
-        self.rxb += len(data)
-        if hasattr(self, "parent_interface") and self.parent_interface != None:
-            self.parent_interface.rxb += len(data)
-                    
-        self.owner.inbound(data, self)
+    def process_incoming(self, data):
+        if self.online and not self.detached:
+            self.rxb += len(data)
+            if hasattr(self, "parent_interface") and self.parent_interface != None:
+                self.parent_interface.rxb += len(data)
+                        
+            self.owner.inbound(data, self)
 
-    def processOutgoing(self, data):
-        if self.online:
+    def process_outgoing(self, data):
+        if self.online and not self.detached:
             # while self.writing:
             #     time.sleep(0.01)
 
@@ -299,22 +326,22 @@ class TCPClientInterface(Interface):
         try:
             in_frame = False
             escape = False
+            frame_buffer = b""
+            data_in = b""
             data_buffer = b""
-            command = KISS.CMD_UNKNOWN
 
             while True:
                 data_in = self.socket.recv(4096)
                 if len(data_in) > 0:
-                    pointer = 0
-                    while pointer < len(data_in):
-                        byte = data_in[pointer]
-                        pointer += 1
-
-                        if self.kiss_framing:
-                            # Read loop for KISS framing
+                    if self.kiss_framing:
+                        # Read loop for KISS framing
+                        pointer = 0
+                        while pointer < len(data_in):
+                            byte = data_in[pointer]
+                            pointer += 1
                             if (in_frame and byte == KISS.FEND and command == KISS.CMD_DATA):
                                 in_frame = False
-                                self.processIncoming(data_buffer)
+                                self.process_incoming(data_buffer)
                             elif (byte == KISS.FEND):
                                 in_frame = True
                                 command = KISS.CMD_UNKNOWN
@@ -337,25 +364,26 @@ class TCPClientInterface(Interface):
                                             escape = False
                                         data_buffer = data_buffer+bytes([byte])
 
-                        else:
-                            # Read loop for HDLC framing
-                            if (in_frame and byte == HDLC.FLAG):
-                                in_frame = False
-                                self.processIncoming(data_buffer)
-                            elif (byte == HDLC.FLAG):
-                                in_frame = True
-                                data_buffer = b""
-                            elif (in_frame and len(data_buffer) < self.HW_MTU):
-                                if (byte == HDLC.ESC):
-                                    escape = True
+                    else:
+                        # Read loop for standard HDLC framing
+                        frame_buffer += data_in
+                        flags_remaining = True
+                        while flags_remaining:
+                            frame_start = frame_buffer.find(HDLC.FLAG)
+                            if frame_start != -1:
+                                frame_end = frame_buffer.find(HDLC.FLAG, frame_start+1)
+                                if frame_end != -1:
+                                    frame = frame_buffer[frame_start+1:frame_end]
+                                    frame = frame.replace(bytes([HDLC.ESC, HDLC.FLAG ^ HDLC.ESC_MASK]), bytes([HDLC.FLAG]))
+                                    frame = frame.replace(bytes([HDLC.ESC, HDLC.ESC  ^ HDLC.ESC_MASK]), bytes([HDLC.ESC]))
+                                    if len(frame) > RNS.Reticulum.HEADER_MINSIZE:
+                                        self.process_incoming(frame)
+                                    frame_buffer = frame_buffer[frame_end:]
                                 else:
-                                    if (escape):
-                                        if (byte == HDLC.FLAG ^ HDLC.ESC_MASK):
-                                            byte = HDLC.FLAG
-                                        if (byte == HDLC.ESC  ^ HDLC.ESC_MASK):
-                                            byte = HDLC.ESC
-                                        escape = False
-                                    data_buffer = data_buffer+bytes([byte])
+                                    flags_remaining = False
+                            else:
+                                flags_remaining = False
+
                 else:
                     self.online = False
                     if self.initiator and not self.detached:
@@ -392,7 +420,8 @@ class TCPClientInterface(Interface):
         self.IN = False
 
         if hasattr(self, "parent_interface") and self.parent_interface != None:
-            self.parent_interface.clients -= 1
+            while self in self.parent_interface.spawned_interfaces:
+                self.parent_interface.spawned_interfaces.remove(self)
 
         if self in RNS.Transport.interfaces:
             if not self.initiator:
@@ -400,42 +429,73 @@ class TCPClientInterface(Interface):
 
 
     def __str__(self):
-        return "TCPInterface["+str(self.name)+"/"+str(self.target_ip)+":"+str(self.target_port)+"]"
+        if ":" in self.target_ip:
+            ip_str = f"[{self.target_ip}]"
+        else:
+            ip_str = f"{self.target_ip}"
+
+        return "TCPInterface["+str(self.name)+"/"+ip_str+":"+str(self.target_port)+"]"
 
 
 class TCPServerInterface(Interface):
-    BITRATE_GUESS      = 10*1000*1000
+    BITRATE_GUESS     = 10_000_000
+    DEFAULT_IFAC_SIZE = 16
+    AUTOCONFIGURE_MTU = True
 
     @staticmethod
-    def get_address_for_if(name):
-        import importlib
-        if importlib.util.find_spec('netifaces') != None:
-            import netifaces
-            return netifaces.ifaddresses(name)[netifaces.AF_INET][0]['addr']
+    def get_address_for_if(name, bind_port, prefer_ipv6=False):
+        import RNS.vendor.ifaddr.niwrapper as netinfo
+        ifaddr = netinfo.ifaddresses(name)
+        if len(ifaddr) < 1:
+            raise SystemError(f"No addresses available on specified kernel interface \"{name}\" for TCPServerInterface to bind to")
+
+        if (prefer_ipv6 or not netinfo.AF_INET in ifaddr) and netinfo.AF_INET6 in ifaddr:
+            bind_ip = ifaddr[netinfo.AF_INET6][0]["addr"]
+            if bind_ip.lower().startswith("fe80::"):
+                # We'll need to add the interface as scope for link-local addresses
+                return TCPServerInterface.get_address_for_host(f"{bind_ip}%{name}", bind_port)
+            else:
+                return TCPServerInterface.get_address_for_host(bind_ip, bind_port)
+        elif netinfo.AF_INET in ifaddr:
+            bind_ip = ifaddr[netinfo.AF_INET][0]["addr"]
+            return (bind_ip, bind_port)
         else:
-            RNS.log("Getting interface addresses from device names requires the netifaces module.", RNS.LOG_CRITICAL)
-            RNS.log("You can install it with the command: python3 -m pip install netifaces", RNS.LOG_CRITICAL)
-            RNS.panic()
+            raise SystemError(f"No addresses available on specified kernel interface \"{name}\" for TCPServerInterface to bind to")
 
     @staticmethod
-    def get_broadcast_for_if(name):
-        import importlib
-        if importlib.util.find_spec('netifaces') != None:
-            import netifaces
-            return netifaces.ifaddresses(name)[netifaces.AF_INET][0]['broadcast']
+    def get_address_for_host(name, bind_port):
+        address_info = socket.getaddrinfo(name, bind_port, proto=socket.IPPROTO_TCP)[0]
+        if address_info[0] == socket.AF_INET6:
+            return (name, bind_port, address_info[4][2], address_info[4][3])
+        elif address_info[0] == socket.AF_INET:
+            return (name, bind_port)
         else:
-            RNS.log("Getting interface addresses from device names requires the netifaces module.", RNS.LOG_CRITICAL)
-            RNS.log("You can install it with the command: python3 -m pip install netifaces", RNS.LOG_CRITICAL)
-            RNS.panic()
+            raise SystemError(f"No suitable kernel interface available for address \"{name}\" for TCPServerInterface to bind to")
 
-    def __init__(self, owner, name, device=None, bindip=None, bindport=None, i2p_tunneled=False):
-        self.rxb = 0
-        self.txb = 0
 
-        self.HW_MTU = 1064
+    @property
+    def clients(self):
+        return len(self.spawned_interfaces)
+
+    def __init__(self, owner, configuration):
+        super().__init__()
+
+        c            = Interface.get_config_obj(configuration)
+        name         = c["name"]
+        device       = c["device"] if "device" in c else None
+        port         = int(c["port"]) if "port" in c else None
+        bindip       = c["listen_ip"] if "listen_ip" in c else None
+        bindport     = int(c["listen_port"]) if "listen_port" in c else None
+        i2p_tunneled = c.as_bool("i2p_tunneled") if "i2p_tunneled" in c else False
+        prefer_ipv6  = c.as_bool("prefer_ipv6") if "prefer_ipv6" in c else False
+
+        if port != None:
+            bindport = port
+
+        self.HW_MTU = TCPInterface.HW_MTU
 
         self.online = False
-        self.clients = 0
+        self.spawned_interfaces = []
         
         self.IN  = True
         self.OUT = False
@@ -445,24 +505,41 @@ class TCPServerInterface(Interface):
         self.i2p_tunneled = i2p_tunneled
         self.mode         = RNS.Interfaces.Interface.Interface.MODE_FULL
 
-        if device != None:
-            bindip = TCPServerInterface.get_address_for_if(device)
-
-        if (bindip != None and bindport != None):
-            self.receives = True
-            self.bind_ip = bindip
+        if bindport == None:
+            raise SystemError(f"No TCP port configured for interface \"{name}\"")
+        else:
             self.bind_port = bindport
 
+        bind_address = None
+        if device != None:
+            bind_address = TCPServerInterface.get_address_for_if(device, self.bind_port, prefer_ipv6)
+        else:
+            if bindip == None:
+                raise SystemError(f"No TCP bind IP configured for interface \"{name}\"")
+            bind_address = TCPServerInterface.get_address_for_host(bindip, self.bind_port)
+
+        if bind_address != None:
+            self.receives = True
+            self.bind_ip = bind_address[0]
+
             def handlerFactory(callback):
                 def createHandler(*args, **keys):
                     return TCPInterfaceHandler(callback, *args, **keys)
                 return createHandler
 
             self.owner = owner
-            address = (self.bind_ip, self.bind_port)
 
-            ThreadingTCPServer.allow_reuse_address = True
-            self.server = ThreadingTCPServer(address, handlerFactory(self.incoming_connection))
+            if len(bind_address) == 4:
+                try:
+                    ThreadingTCP6Server.allow_reuse_address = True
+                    self.server = ThreadingTCP6Server(bind_address, handlerFactory(self.incoming_connection))
+                except Exception as e:
+                    RNS.log(f"Error while binding IPv6 socket for interface, the contained exception was: {e}", RNS.LOG_ERROR)
+                    raise SystemError("Could not bind IPv6 socket for interface. Please check the specified \"listen_ip\" configuration option")
+            else:
+                ThreadingTCPServer.allow_reuse_address = True
+                self.server = ThreadingTCPServer(bind_address, handlerFactory(self.incoming_connection))
+                self.server.daemon_threads = True
 
             self.bitrate = TCPServerInterface.BITRATE_GUESS
 
@@ -472,20 +549,41 @@ class TCPServerInterface(Interface):
 
             self.online = True
 
+        else:
+            raise SystemError("Insufficient parameters to create TCP listener")
 
     def incoming_connection(self, handler):
         RNS.log("Accepting incoming TCP connection", RNS.LOG_VERBOSE)
-        interface_name = "Client on "+self.name
-        spawned_interface = TCPClientInterface(self.owner, interface_name, target_ip=None, target_port=None, connected_socket=handler.request, i2p_tunneled=self.i2p_tunneled)
+        spawned_configuration = {"name": "Client on "+self.name, "target_host": None, "target_port": None, "i2p_tunneled": self.i2p_tunneled}
+        spawned_interface = TCPClientInterface(self.owner, spawned_configuration, connected_socket=handler.request)
         spawned_interface.OUT = self.OUT
         spawned_interface.IN  = self.IN
         spawned_interface.target_ip = handler.client_address[0]
         spawned_interface.target_port = str(handler.client_address[1])
         spawned_interface.parent_interface = self
         spawned_interface.bitrate = self.bitrate
+        spawned_interface.optimise_mtu()
+        
         spawned_interface.ifac_size = self.ifac_size
         spawned_interface.ifac_netname = self.ifac_netname
         spawned_interface.ifac_netkey = self.ifac_netkey
+        if spawned_interface.ifac_netname != None or spawned_interface.ifac_netkey != None:
+            ifac_origin = b""
+            if spawned_interface.ifac_netname != None:
+                ifac_origin += RNS.Identity.full_hash(spawned_interface.ifac_netname.encode("utf-8"))
+            if spawned_interface.ifac_netkey != None:
+                ifac_origin += RNS.Identity.full_hash(spawned_interface.ifac_netkey.encode("utf-8"))
+
+            ifac_origin_hash = RNS.Identity.full_hash(ifac_origin)
+            spawned_interface.ifac_key = RNS.Cryptography.hkdf(
+                length=64,
+                derive_from=ifac_origin_hash,
+                salt=RNS.Reticulum.IFAC_SALT,
+                context=None
+            )
+            spawned_interface.ifac_identity = RNS.Identity.from_bytes(spawned_interface.ifac_key)
+            spawned_interface.ifac_signature = spawned_interface.ifac_identity.sign(RNS.Identity.full_hash(spawned_interface.ifac_key))
+
         spawned_interface.announce_rate_target = self.announce_rate_target
         spawned_interface.announce_rate_grace = self.announce_rate_grace
         spawned_interface.announce_rate_penalty = self.announce_rate_penalty
@@ -494,21 +592,30 @@ class TCPServerInterface(Interface):
         spawned_interface.online = True
         RNS.log("Spawned new TCPClient Interface: "+str(spawned_interface), RNS.LOG_VERBOSE)
         RNS.Transport.interfaces.append(spawned_interface)
-        self.clients += 1
+        while spawned_interface in self.spawned_interfaces:
+            self.spawned_interfaces.remove(spawned_interface)
+        self.spawned_interfaces.append(spawned_interface)
         spawned_interface.read_loop()
 
-    def processOutgoing(self, data):
+    def received_announce(self, from_spawned=False):
+        if from_spawned: self.ia_freq_deque.append(time.time())
+
+    def sent_announce(self, from_spawned=False):
+        if from_spawned: self.oa_freq_deque.append(time.time())
+
+    def process_outgoing(self, data):
         pass
 
-
     def detach(self):
+        self.detached = True
+        self.online = False
         if self.server != None:
             if hasattr(self.server, "shutdown"):
                 if callable(self.server.shutdown):
                     try:
                         RNS.log("Detaching "+str(self), RNS.LOG_DEBUG)
                         self.server.shutdown()
-                        self.detached = True
+                        self.server.server_close()
                         self.server = None
 
                     except Exception as e:
@@ -516,7 +623,12 @@ class TCPServerInterface(Interface):
 
 
     def __str__(self):
-        return "TCPServerInterface["+self.name+"/"+self.bind_ip+":"+str(self.bind_port)+"]"
+        if ":" in self.bind_ip:
+            ip_str = f"[{self.bind_ip}]"
+        else:
+            ip_str = f"{self.bind_ip}"
+
+        return "TCPServerInterface["+self.name+"/"+ip_str+":"+str(self.bind_port)+"]"
 
 
 class TCPInterfaceHandler(socketserver.BaseRequestHandler):

RNS/Interfaces/UDPInterface.py

@@ -20,7 +20,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from .Interface import Interface
+from RNS.Interfaces.Interface import Interface
 import socketserver
 import threading
 import socket
@@ -31,32 +31,37 @@ import RNS
 
 class UDPInterface(Interface):
     BITRATE_GUESS = 10*1000*1000
+    DEFAULT_IFAC_SIZE = 16
 
     @staticmethod
     def get_address_for_if(name):
-        import importlib
-        if importlib.util.find_spec('netifaces') != None:
-            import netifaces
-            return netifaces.ifaddresses(name)[netifaces.AF_INET][0]['addr']
-        else:
-            RNS.log("Getting interface addresses from device names requires the netifaces module.", RNS.LOG_CRITICAL)
-            RNS.log("You can install it with the command: python3 -m pip install netifaces", RNS.LOG_CRITICAL)
-            RNS.panic()
+        import RNS.vendor.ifaddr.niwrapper as netinfo
+        ifaddr = netinfo.ifaddresses(name)
+        return ifaddr[netinfo.AF_INET][0]["addr"]
 
     @staticmethod
     def get_broadcast_for_if(name):
-        import importlib
-        if importlib.util.find_spec('netifaces') != None:
-            import netifaces
-            return netifaces.ifaddresses(name)[netifaces.AF_INET][0]['broadcast']
-        else:
-            RNS.log("Getting interface addresses from device names requires the netifaces module.", RNS.LOG_CRITICAL)
-            RNS.log("You can install it with the command: python3 -m pip install netifaces", RNS.LOG_CRITICAL)
-            RNS.panic()
+        import RNS.vendor.ifaddr.niwrapper as netinfo
+        ifaddr = netinfo.ifaddresses(name)
+        return ifaddr[netinfo.AF_INET][0]["broadcast"]
 
-    def __init__(self, owner, name, device=None, bindip=None, bindport=None, forwardip=None, forwardport=None):
-        self.rxb = 0
-        self.txb = 0
+    def __init__(self, owner, configuration):
+        super().__init__()
+
+        c           = Interface.get_config_obj(configuration)
+        name        = c["name"]
+        device      = c["device"] if "device" in c else None
+        port        = int(c["port"]) if "port" in c else None
+        bindip      = c["listen_ip"] if "listen_ip" in c else None
+        bindport    = int(c["listen_port"]) if "listen_port" in c else None
+        forwardip   = c["forward_ip"] if "forward_ip" in c else None
+        forwardport = int(c["forward_port"]) if "forward_port" in c else None
+
+        if port != None:
+            if bindport == None:
+                bindport = port
+            if forwardport == None:
+                forwardport = port
 
         self.HW_MTU = 1064
 
@@ -86,7 +91,7 @@ class UDPInterface(Interface):
             self.owner = owner
             address = (self.bind_ip, self.bind_port)
             socketserver.UDPServer.address_family = socket.AF_INET
-            self.server = socketserver.UDPServer(address, handlerFactory(self.processIncoming))
+            self.server = socketserver.UDPServer(address, handlerFactory(self.process_incoming))
 
             thread = threading.Thread(target=self.server.serve_forever)
             thread.daemon = True
@@ -100,11 +105,11 @@ class UDPInterface(Interface):
             self.forward_port = forwardport
 
 
-    def processIncoming(self, data):
+    def process_incoming(self, data):
         self.rxb += len(data)
         self.owner.inbound(data, self)
 
-    def processOutgoing(self,data):
+    def process_outgoing(self,data):
         try:
             udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
             udp_socket.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)

RNS/Interfaces/__init__.py

@@ -22,6 +22,9 @@
 
 import os
 import glob
+import RNS.Interfaces.Android
 
-modules = glob.glob(os.path.dirname(__file__)+"/*.py")
-__all__ = [ os.path.basename(f)[:-3] for f in modules if not f.endswith('__init__.py')]
+py_modules  = glob.glob(os.path.dirname(__file__)+"/*.py")
+pyc_modules = glob.glob(os.path.dirname(__file__)+"/*.pyc")
+modules     = py_modules+pyc_modules
+__all__ = list(set([os.path.basename(f).replace(".pyc", "").replace(".py", "") for f in modules if not (f.endswith("__init__.py") or f.endswith("__init__.pyc"))]))

RNS/Packet.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2024 Mark Qvist / unsigned.io and contributors.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -30,7 +30,7 @@ class Packet:
     """
     The Packet class is used to create packet instances that can be sent
     over a Reticulum network. Packets will automatically be encrypted if
-    they are adressed to a ``RNS.Destination.SINGLE`` destination,
+    they are addressed to a ``RNS.Destination.SINGLE`` destination,
     ``RNS.Destination.GROUP`` destination or a :ref:`RNS.Link<api-link>`.
 
     For ``RNS.Destination.GROUP`` destinations, Reticulum will use the
@@ -58,9 +58,7 @@ class Packet:
     # Header types
     HEADER_1     = 0x00     # Normal header format
     HEADER_2     = 0x01     # Header format used for packets in transport
-    HEADER_3     = 0x02     # Reserved
-    HEADER_4     = 0x03     # Reserved
-    header_types = [HEADER_1, HEADER_2, HEADER_3, HEADER_4]
+    header_types = [HEADER_1, HEADER_2]
 
     # Packet context types
     NONE           = 0x00   # Generic data packet
@@ -77,6 +75,7 @@ class Packet:
     PATH_RESPONSE  = 0x0B   # Packet is a response to a path request
     COMMAND        = 0x0C   # Packet is a command
     COMMAND_STATUS = 0x0D   # Packet is a status of an executed command
+    CHANNEL        = 0x0E   # Packet contains link channel data
     KEEPALIVE      = 0xFA   # Packet is a keepalive packet
     LINKIDENTIFY   = 0xFB   # Packet is a link peer identification proof
     LINKCLOSE      = 0xFC   # Packet is a link close message
@@ -84,6 +83,10 @@ class Packet:
     LRRTT          = 0xFE   # Packet is a link request round-trip time measurement
     LRPROOF        = 0xFF   # Packet is a link request proof
 
+    # Context flag values
+    FLAG_SET       = 0x01
+    FLAG_UNSET     = 0x00
+
     # This is used to calculate allowable
     # payload sizes
     HEADER_MAXSIZE = RNS.Reticulum.HEADER_MAXSIZE
@@ -92,7 +95,7 @@ class Packet:
     # With an MTU of 500, the maximum of data we can
     # send in a single encrypted packet is given by
     # the below calculation; 383 bytes.
-    ENCRYPTED_MDU  = math.floor((RNS.Reticulum.MDU-RNS.Identity.FERNET_OVERHEAD-RNS.Identity.KEYSIZE//16)/RNS.Identity.AES128_BLOCKSIZE)*RNS.Identity.AES128_BLOCKSIZE - 1
+    ENCRYPTED_MDU  = math.floor((RNS.Reticulum.MDU-RNS.Identity.TOKEN_OVERHEAD-RNS.Identity.KEYSIZE//16)/RNS.Identity.AES128_BLOCKSIZE)*RNS.Identity.AES128_BLOCKSIZE - 1
     """
     The maximum size of the payload data in a single encrypted packet 
     """
@@ -103,7 +106,9 @@ class Packet:
 
     TIMEOUT_PER_HOP = RNS.Reticulum.DEFAULT_PER_HOP_TIMEOUT
 
-    def __init__(self, destination, data, packet_type = DATA, context = NONE, transport_type = RNS.Transport.BROADCAST, header_type = HEADER_1, transport_id = None, attached_interface = None, create_receipt = True):
+    def __init__(self, destination, data, packet_type = DATA, context = NONE, transport_type = RNS.Transport.BROADCAST,
+                 header_type = HEADER_1, transport_id = None, attached_interface = None, create_receipt = True, context_flag=FLAG_UNSET):
+
         if destination != None:
             if transport_type == None:
                 transport_type = RNS.Transport.BROADCAST
@@ -112,6 +117,7 @@ class Packet:
             self.packet_type    = packet_type
             self.transport_type = transport_type
             self.context        = context
+            self.context_flag   = context_flag
 
             self.hops           = 0;
             self.destination    = destination
@@ -131,20 +137,27 @@ class Packet:
             self.fromPacked     = True
             self.create_receipt = False
 
-        self.MTU         = RNS.Reticulum.MTU
+        if destination and destination.type == RNS.Destination.LINK:
+            self.MTU     = destination.mtu
+        else:
+            self.MTU     = RNS.Reticulum.MTU
+
         self.sent_at     = None
         self.packet_hash = None
+        self.ratchet_id  = None
 
         self.attached_interface = attached_interface
         self.receiving_interface = None
         self.rssi = None
         self.snr = None
+        self.q = None
 
     def get_packed_flags(self):
         if self.context == Packet.LRPROOF:
-            packed_flags = (self.header_type << 6) | (self.transport_type << 4) | RNS.Destination.LINK | self.packet_type
+            packed_flags = (self.header_type << 6) | (self.context_flag << 5) | (self.transport_type << 4) | (RNS.Destination.LINK << 2) | self.packet_type
         else:
-            packed_flags = (self.header_type << 6) | (self.transport_type << 4) | (self.destination.type << 2) | self.packet_type
+            packed_flags = (self.header_type << 6) | (self.context_flag << 5) | (self.transport_type << 4) | (self.destination.type << 2) | self.packet_type
+
         return packed_flags
 
     def pack(self):
@@ -173,8 +186,8 @@ class Packet:
                     # Packet proofs over links are not encrypted
                     self.ciphertext = self.data
                 elif self.context == Packet.RESOURCE:
-                    # A resource takes care of symmetric
-                    # encryption by itself
+                    # A resource takes care of encryption
+                    # by itself
                     self.ciphertext = self.data
                 elif self.context == Packet.KEEPALIVE:
                     # Keepalive packets contain no actual
@@ -187,6 +200,8 @@ class Packet:
                     # In all other cases, we encrypt the packet
                     # with the destination's encryption method
                     self.ciphertext = self.destination.encrypt(self.data)
+                    if hasattr(self.destination, "latest_ratchet_id"):
+                        self.ratchet_id = self.destination.latest_ratchet_id
 
             if self.header_type == Packet.HEADER_2:
                 if self.transport_id != None:
@@ -215,8 +230,9 @@ class Packet:
             self.flags = self.raw[0]
             self.hops  = self.raw[1]
 
-            self.header_type      = (self.flags & 0b11000000) >> 6
-            self.transport_type   = (self.flags & 0b00110000) >> 4
+            self.header_type      = (self.flags & 0b01000000) >> 6
+            self.context_flag     = (self.flags & 0b00100000) >> 5
+            self.transport_type   = (self.flags & 0b00010000) >> 4
             self.destination_type = (self.flags & 0b00001100) >> 2
             self.packet_type      = (self.flags & 0b00000011)
 
@@ -277,6 +293,10 @@ class Packet:
         :returns: A :ref:`RNS.PacketReceipt<api-packetreceipt>` instance if *create_receipt* was set to *True* when the packet was instantiated, if not returns *None*. If the packet could not be sent *False* is returned.
         """
         if self.sent:
+            # Re-pack the packet to obtain new ciphertext for
+            # encrypted destinations
+            self.pack()
+            
             if RNS.Transport.outbound(self):
                 return self.receipt
             else:
@@ -325,6 +345,33 @@ class Packet:
 
         return hashable_part
 
+    def get_rssi(self):
+        """
+        :returns: The physical layer *Received Signal Strength Indication* if available, otherwise ``None``.
+        """
+        if self.rssi != None:
+            return self.rssi
+        else:
+            return reticulum.get_packet_rssi(self.packet_hash)
+            
+    def get_snr(self):
+        """
+        :returns: The physical layer *Signal-to-Noise Ratio* if available, otherwise ``None``.
+        """
+        if self.snr != None:
+            return self.snr
+        else:
+            return reticulum.get_packet_snr(self.packet_hash)
+
+    def get_q(self):
+        """
+        :returns: The physical layer *Link Quality* if available, otherwise ``None``.
+        """
+        if self.q != None:
+            return self.q
+        else:
+            return reticulum.get_packet_q(self.packet_hash)
+
 class ProofDestination:
     def __init__(self, packet):
         self.hash = packet.get_hash()[:RNS.Reticulum.TRUNCATED_HASHLENGTH//8];
@@ -365,10 +412,10 @@ class PacketReceipt:
         self.proof_packet   = None
 
         if packet.destination.type == RNS.Destination.LINK:
-            self.timeout    = packet.destination.rtt * packet.destination.traffic_timeout_factor
+            self.timeout    = max(packet.destination.rtt * packet.destination.traffic_timeout_factor, RNS.Link.TRAFFIC_TIMEOUT_MIN_MS/1000)
         else:
-            self.timeout    = Packet.TIMEOUT_PER_HOP * RNS.Transport.hops_to(self.destination.hash)
-
+            self.timeout    = RNS.Reticulum.get_instance().get_first_hop_timeout(self.destination.hash)
+            self.timeout   += Packet.TIMEOUT_PER_HOP * RNS.Transport.hops_to(self.destination.hash)
 
     def get_status(self):
         """
@@ -397,9 +444,16 @@ class PacketReceipt:
                     self.proved = True
                     self.concluded_at = time.time()
                     self.proof_packet = proof_packet
+                    link.last_proof = self.concluded_at
 
                     if self.callbacks.delivery != None:
-                        self.callbacks.delivery(self)
+                        try:
+                            self.callbacks.delivery(self)
+                        except Exception as e:
+                            RNS.log("An error occurred while evaluating external delivery callback for "+str(link), RNS.LOG_ERROR)
+                            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            RNS.trace_exception(e)
+                            
                     return True
                 else:
                     return False
@@ -430,7 +484,7 @@ class PacketReceipt:
             # This is an explicit proof
             proof_hash = proof[:RNS.Identity.HASHLENGTH//8]
             signature = proof[RNS.Identity.HASHLENGTH//8:RNS.Identity.HASHLENGTH//8+RNS.Identity.SIGLENGTH//8]
-            if proof_hash == self.hash:
+            if proof_hash == self.hash and hasattr(self.destination, "identity") and self.destination.identity != None:
                 proof_valid = self.destination.identity.validate(signature, self.hash)
                 if proof_valid:
                     self.status = PacketReceipt.DELIVERED
@@ -451,6 +505,10 @@ class PacketReceipt:
                 return False
         elif len(proof) == PacketReceipt.IMPL_LENGTH:
             # This is an implicit proof
+
+            if not hasattr(self.destination, "identity"):
+                return False
+
             if self.destination.identity == None:
                 return False
 

RNS/Resolver.py

@@ -0,0 +1,27 @@
+# MIT License
+#
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io and contributors.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+class Resolver:
+  
+  @staticmethod
+  def resolve_identity(full_name):
+    pass

RNS/Resource.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io and contributors.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -25,7 +25,9 @@ import os
 import bz2
 import math
 import time
+import tempfile
 import threading
+from threading import Lock
 from .vendor import umsgpack as umsgpack
 from time import sleep
 
@@ -47,11 +49,14 @@ class Resource:
     WINDOW               = 4
 
     # Absolute minimum window size during transfer
-    WINDOW_MIN           = 1
+    WINDOW_MIN           = 2
 
     # The maximum window size for transfers on slow links
     WINDOW_MAX_SLOW      = 10
 
+    # The maximum window size for transfers on very slow links
+    WINDOW_MAX_VERY_SLOW = 4
+
     # The maximum window size for transfers on fast links
     WINDOW_MAX_FAST      = 75
     
@@ -63,12 +68,22 @@ class Resource:
     # rounds, the fast link window size will be allowed.
     FAST_RATE_THRESHOLD  = WINDOW_MAX_SLOW - WINDOW - 2
 
+    # If the very slow rate is sustained for this many request
+    # rounds, window will be capped to the very slow limit.
+    VERY_SLOW_RATE_THRESHOLD = 2
+
     # If the RTT rate is higher than this value,
     # the max window size for fast links will be used.
     # The default is 50 Kbps (the value is stored in
     # bytes per second, hence the "/ 8").
     RATE_FAST            = (50*1000) / 8
 
+    # If the RTT rate is lower than this value,
+    # the window size will be capped at .
+    # The default is 50 Kbps (the value is stored in
+    # bytes per second, hence the "/ 8").
+    RATE_VERY_SLOW       = (2*1000) / 8
+
     # The minimum allowed flexibility of the window size.
     # The difference between window_max and window_min
     # will never be smaller than this value.
@@ -103,9 +118,11 @@ class Resource:
 
     PART_TIMEOUT_FACTOR           = 4
     PART_TIMEOUT_FACTOR_AFTER_RTT = 2
-    MAX_RETRIES                   = 8
+    PROOF_TIMEOUT_FACTOR          = 3
+    MAX_RETRIES                   = 16
     MAX_ADV_RETRIES               = 4
-    SENDER_GRACE_TIME             = 10
+    SENDER_GRACE_TIME             = 10.0
+    PROCESSING_GRACE              = 1.0
     RETRY_GRACE_TIME              = 0.25
     PER_RETRY_DELAY               = 0.5
 
@@ -144,12 +161,12 @@ class Resource:
             resource.encrypted           = True if resource.flags & 0x01 else False
             resource.compressed          = True if resource.flags >> 1 & 0x01 else False
             resource.initiator           = False
-            resource.callback             = callback
+            resource.callback            = callback
             resource.__progress_callback = progress_callback
-            resource.total_parts         = int(math.ceil(resource.size/float(Resource.SDU)))
+            resource.total_parts         = int(math.ceil(resource.size/float(resource.sdu)))
             resource.received_count      = 0
             resource.outstanding_parts   = 0
-            resource.parts                 = [None] * resource.total_parts
+            resource.parts               = [None] * resource.total_parts
             resource.window              = Resource.WINDOW
             resource.window_max          = Resource.WINDOW_MAX_SLOW
             resource.window_min          = Resource.WINDOW_MIN
@@ -167,25 +184,36 @@ class Resource:
             resource.hashmap = [None] * resource.total_parts
             resource.hashmap_height = 0
             resource.waiting_for_hmu = False
-
             resource.receiving_part = False
+            resource.consecutive_completed_height = -1
 
-            resource.consecutive_completed_height = 0
+            previous_window = resource.link.get_last_resource_window()
+            previous_eifr   = resource.link.get_last_resource_eifr()
+            if previous_window:
+                resource.window = previous_window
+            if previous_eifr:
+                resource.previous_eifr = previous_eifr
             
-            resource.link.register_incoming_resource(resource)
+            if not resource.link.has_incoming_resource(resource):
+                resource.link.register_incoming_resource(resource)
 
-            RNS.log("Accepting resource advertisement for "+RNS.prettyhexrep(resource.hash), RNS.LOG_DEBUG)
-            if resource.link.callbacks.resource_started != None:
-                try:
-                    resource.link.callbacks.resource_started(resource)
-                except Exception as e:
-                    RNS.log("Error while executing resource started callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+                RNS.log(f"Accepting resource advertisement for {RNS.prettyhexrep(resource.hash)}. Transfer size is {RNS.prettysize(resource.size)} in {resource.total_parts} parts.", RNS.LOG_DEBUG)
+                if resource.link.callbacks.resource_started != None:
+                    try:
+                        resource.link.callbacks.resource_started(resource)
+                    except Exception as e:
+                        RNS.log("Error while executing resource started callback from "+str(resource)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
 
-            resource.hashmap_update(0, resource.hashmap_raw)
+                resource.hashmap_update(0, resource.hashmap_raw)
 
-            resource.watchdog_job()
+                resource.watchdog_job()
+
+                return resource
+
+            else:
+                RNS.log("Ignoring resource advertisement for "+RNS.prettyhexrep(resource.hash)+", resource already transferring", RNS.LOG_DEBUG)
+                return None
 
-            return resource
         except Exception as e:
             RNS.log("Could not decode resource advertisement, dropping resource", RNS.LOG_DEBUG)
             return None
@@ -197,11 +225,22 @@ class Resource:
         data_size = None
         resource_data = None
         self.assembly_lock = False
+        self.preparing_next_segment = False
+        self.next_segment = None
+
+        if data != None:
+            if not hasattr(data, "read") and len(data) > Resource.MAX_EFFICIENT_SIZE:
+                original_data = data
+                data_size = len(original_data)
+                data = tempfile.TemporaryFile()
+                data.write(original_data)
+                del original_data
 
         if hasattr(data, "read"):
-            data_size = os.stat(data.name).st_size
-            self.total_size  = data_size
-            self.grand_total_parts = math.ceil(data_size/Resource.SDU)
+            if data_size == None:
+                data_size = os.stat(data.name).st_size
+
+            self.total_size = data_size
 
             if data_size <= Resource.MAX_EFFICIENT_SIZE:
                 self.total_segments = 1
@@ -222,7 +261,6 @@ class Resource:
 
         elif isinstance(data, bytes):
             data_size = len(data)
-            self.grand_total_parts = math.ceil(data_size/Resource.SDU)
             self.total_size  = data_size
             
             resource_data = data
@@ -240,6 +278,10 @@ class Resource:
 
         self.status = Resource.NONE
         self.link = link
+        if self.link.mtu:
+            self.sdu = self.link.mtu - RNS.Reticulum.HEADER_MAXSIZE - RNS.Reticulum.IFAC_MIN_SIZE
+        else:
+            self.sdu = link.mdu or Resource.SDU
         self.max_retries = Resource.MAX_RETRIES
         self.max_adv_retries = Resource.MAX_ADV_RETRIES
         self.retries_left = self.max_retries
@@ -255,9 +297,14 @@ class Resource:
         self.req_sent = 0
         self.req_resp_rtt_rate = 0
         self.rtt_rxd_bytes_at_part_req = 0
+        self.req_data_rtt_rate = 0
+        self.eifr = None
+        self.previous_eifr = None
         self.fast_rate_rounds = 0
+        self.very_slow_rate_rounds = 0
         self.request_id = request_id
         self.is_response = is_response
+        self.auto_compress = auto_compress
 
         self.req_hashlist = []
         self.receiver_min_consecutive_height = 0
@@ -273,7 +320,7 @@ class Resource:
             self.uncompressed_data = data
 
             compression_began = time.time()
-            if (auto_compress and len(self.uncompressed_data) < Resource.AUTO_COMPRESS_MAX_SIZE):
+            if (auto_compress and len(self.uncompressed_data) <= Resource.AUTO_COMPRESS_MAX_SIZE):
                 RNS.log("Compressing resource data...", RNS.LOG_DEBUG)
                 self.compressed_data   = bz2.compress(self.uncompressed_data)
                 RNS.log("Compression completed in "+str(round(time.time()-compression_began, 3))+" seconds", RNS.LOG_DEBUG)
@@ -314,7 +361,8 @@ class Resource:
 
             self.size = len(self.data)
             self.sent_parts = 0
-            hashmap_entries = int(math.ceil(self.size/float(Resource.SDU)))
+            hashmap_entries = int(math.ceil(self.size/float(self.sdu)))
+            self.total_parts = hashmap_entries
                 
             hashmap_ok = False
             while not hashmap_ok:
@@ -335,7 +383,7 @@ class Resource:
                 self.hashmap = b""
                 collision_guard_list = []
                 for i in range(0,hashmap_entries):
-                    data = self.data[i*Resource.SDU:(i+1)*Resource.SDU]
+                    data = self.data[i*self.sdu:(i+1)*self.sdu]
                     map_hash = self.get_map_hash(data)
 
                     if map_hash in collision_guard_list:
@@ -360,7 +408,8 @@ class Resource:
             if advertise:
                 self.advertise()
         else:
-            pass
+            self.receive_lock = Lock()
+            
 
     def hashmap_update_packet(self, plaintext):
         if not self.status == Resource.FAILED:
@@ -392,13 +441,15 @@ class Resource:
         Advertise the resource. If the other end of the link accepts
         the resource advertisement it will begin transferring.
         """
-        thread = threading.Thread(target=self.__advertise_job)
-        thread.daemon = True
+        thread = threading.Thread(target=self.__advertise_job, daemon=True)
         thread.start()
 
+        if self.segment_index < self.total_segments:
+            prepare_thread = threading.Thread(target=self.__prepare_next_segment, daemon=True)
+            prepare_thread.start()
+
     def __advertise_job(self):
-        data = ResourceAdvertisement(self).pack()
-        self.advertisement_packet = RNS.Packet(self.link, data, context=RNS.Packet.RESOURCE_ADV)
+        self.advertisement_packet = RNS.Packet(self.link, ResourceAdvertisement(self).pack(), context=RNS.Packet.RESOURCE_ADV)
         while not self.link.ready_for_new_resource():
             self.status = Resource.QUEUED
             sleep(0.25)
@@ -419,6 +470,22 @@ class Resource:
 
         self.watchdog_job()
 
+    def update_eifr(self):
+        if self.rtt == None:
+            rtt = self.link.rtt
+        else:
+            rtt = self.rtt
+
+        if self.req_data_rtt_rate != 0:
+            expected_inflight_rate = self.req_data_rtt_rate*8
+        else:
+            if self.previous_eifr != None:
+                expected_inflight_rate = self.previous_eifr
+            else:
+                expected_inflight_rate = self.link.establishment_cost*8 / rtt
+
+        self.eifr = expected_inflight_rate
+
     def watchdog_job(self):
         thread = threading.Thread(target=self.__watchdog_job)
         thread.daemon = True
@@ -433,9 +500,8 @@ class Resource:
                 sleep(0.025)
 
             sleep_time = None
-
             if self.status == Resource.ADVERTISED:
-                sleep_time = (self.adv_sent+self.timeout)-time.time()
+                sleep_time = (self.adv_sent+self.timeout+Resource.PROCESSING_GRACE)-time.time()
                 if sleep_time < 0:
                     if self.retries_left <= 0:
                         RNS.log("Resource transfer timeout after sending advertisement", RNS.LOG_DEBUG)
@@ -445,32 +511,36 @@ class Resource:
                         try:
                             RNS.log("No part requests received, retrying resource advertisement...", RNS.LOG_DEBUG)
                             self.retries_left -= 1
-                            self.advertisement_packet.resend()
+                            self.advertisement_packet = RNS.Packet(self.link, ResourceAdvertisement(self).pack(), context=RNS.Packet.RESOURCE_ADV)
+                            self.advertisement_packet.send()
                             self.last_activity = time.time()
                             self.adv_sent = self.last_activity
                             sleep_time = 0.001
                         except Exception as e:
-                            RNS.log("Could not resend advertisement packet, cancelling resource", RNS.LOG_VERBOSE)
+                            RNS.log("Could not resend advertisement packet, cancelling resource. The contained exception was: "+str(e), RNS.LOG_VERBOSE)
                             self.cancel()
                     
 
             elif self.status == Resource.TRANSFERRING:
                 if not self.initiator:
-
-                    if self.rtt == None:
-                        rtt = self.link.rtt
-                    else:
-                        rtt = self.rtt
-
-                    window_remaining = self.outstanding_parts
-
                     retries_used = self.max_retries - self.retries_left
                     extra_wait = retries_used * Resource.PER_RETRY_DELAY
-                    sleep_time = self.last_activity + (rtt*(self.part_timeout_factor+window_remaining)) + Resource.RETRY_GRACE_TIME + extra_wait - time.time()
+
+                    self.update_eifr()
+                    expected_tof_remaining = (self.outstanding_parts*self.sdu*8)/self.eifr
+
+                    if self.req_resp_rtt_rate != 0:
+                        sleep_time = self.last_activity + self.part_timeout_factor*expected_tof_remaining + Resource.RETRY_GRACE_TIME + extra_wait - time.time()
+                    else:
+                        sleep_time = self.last_activity + self.part_timeout_factor*((3*self.sdu)/self.eifr) + Resource.RETRY_GRACE_TIME + extra_wait - time.time()
+                    
+                    # RNS.log(f"EIFR {RNS.prettyspeed(self.eifr)}, ETOF {RNS.prettyshorttime(expected_tof_remaining)} ", RNS.LOG_DEBUG, pt=True)
+                    # RNS.log(f"Resource ST {RNS.prettyshorttime(sleep_time)}, RTT {RNS.prettyshorttime(self.rtt or self.link.rtt)}, {self.outstanding_parts} left", RNS.LOG_DEBUG, pt=True)
                     
                     if sleep_time < 0:
                         if self.retries_left > 0:
-                            RNS.log("Timed out waiting for parts, requesting retry", RNS.LOG_DEBUG)
+                            ms = "" if self.outstanding_parts == 1 else "s"
+                            RNS.log("Timed out waiting for "+str(self.outstanding_parts)+" part"+ms+", requesting retry", RNS.LOG_DEBUG)
                             if self.window > self.window_min:
                                 self.window -= 1
                                 if self.window_max > self.window_min:
@@ -495,6 +565,10 @@ class Resource:
                         sleep_time = 0.001
 
             elif self.status == Resource.AWAITING_PROOF:
+                # Decrease timeout factor since proof packets are
+                # significantly smaller than full req/resp roundtrip
+                self.timeout_factor = Resource.PROOF_TIMEOUT_FACTOR
+
                 sleep_time = self.last_part_sent + (self.rtt*self.timeout_factor+self.sender_grace_time) - time.time()
                 if sleep_time < 0:
                     if self.retries_left <= 0:
@@ -512,7 +586,7 @@ class Resource:
                         sleep_time = 0.001
 
             if sleep_time == 0:
-                RNS.log("Warning! Link watchdog sleep time of 0!", RNS.LOG_WARNING)
+                RNS.log("Warning! Link watchdog sleep time of 0!", RNS.LOG_DEBUG)
             if sleep_time == None or sleep_time < 0:
                 RNS.log("Timing error, cancelling resource transfer.", RNS.LOG_ERROR)
                 self.cancel()
@@ -586,11 +660,28 @@ class Resource:
                 proof_data = self.hash+proof
                 proof_packet = RNS.Packet(self.link, proof_data, packet_type=RNS.Packet.PROOF, context=RNS.Packet.RESOURCE_PRF)
                 proof_packet.send()
+                RNS.Transport.cache(proof_packet, force_cache=True)
             except Exception as e:
                 RNS.log("Could not send proof packet, cancelling resource", RNS.LOG_DEBUG)
                 RNS.log("The contained exception was: "+str(e), RNS.LOG_DEBUG)
                 self.cancel()
 
+    def __prepare_next_segment(self):
+        # Prepare the next segment for advertisement
+        RNS.log(f"Preparing segment {self.segment_index+1} of {self.total_segments} for resource {self}", RNS.LOG_DEBUG)
+        self.preparing_next_segment = True
+        self.next_segment = Resource(
+            self.input_file, self.link,
+            callback = self.callback,
+            segment_index = self.segment_index+1,
+            original_hash=self.original_hash,
+            progress_callback = self.__progress_callback,
+            request_id = self.request_id,
+            is_response = self.is_response,
+            advertise = False,
+            auto_compress = self.auto_compress,
+        )
+
     def validate_proof(self, proof_data):
         if not self.status == Resource.FAILED:
             if len(proof_data) == RNS.Identity.HASHLENGTH//8*2:
@@ -605,10 +696,24 @@ class Resource:
                                 self.callback(self)
                             except Exception as e:
                                 RNS.log("Error while executing resource concluded callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            finally:
+                                try:
+                                    if hasattr(self, "input_file"):
+                                        if hasattr(self.input_file, "close") and callable(self.input_file.close):
+                                            self.input_file.close()
+
+                                except Exception as e:
+                                    RNS.log("Error while closing resource input file: "+str(e), RNS.LOG_ERROR)
                     else:
                         # Otherwise we'll recursively create the
                         # next segment of the resource
-                        Resource(self.input_file, self.link, callback = self.callback, segment_index = self.segment_index+1, original_hash=self.original_hash, progress_callback = self.__progress_callback)
+                        if not self.preparing_next_segment:
+                            RNS.log(f"Next segment preparation for resource {self} was not started yet, manually preparing now. This will cause transfer slowdown.", RNS.LOG_WARNING)
+                            self.__prepare_next_segment()
+
+                        while self.next_segment == None:
+                            time.sleep(0.05)
+                        self.next_segment.advertise()
                 else:
                     pass
             else:
@@ -616,98 +721,106 @@ class Resource:
 
 
     def receive_part(self, packet):
-        while self.receiving_part:
-            sleep(0.001)
+        with self.receive_lock:
 
-        self.receiving_part = True
-        self.last_activity = time.time()
-        self.retries_left = self.max_retries
+            self.receiving_part = True
+            self.last_activity = time.time()
+            self.retries_left = self.max_retries
 
-        if self.req_resp == None:
-            self.req_resp = self.last_activity
-            rtt = self.req_resp-self.req_sent
-            
-            self.part_timeout_factor = Resource.PART_TIMEOUT_FACTOR_AFTER_RTT
-            if self.rtt == None:
-                self.rtt = self.link.rtt
-                self.watchdog_job()
-            elif rtt < self.rtt:
-                self.rtt = max(self.rtt - self.rtt*0.05, rtt)
-            elif rtt > self.rtt:
-                self.rtt = min(self.rtt + self.rtt*0.05, rtt)
+            if self.req_resp == None:
+                self.req_resp = self.last_activity
+                rtt = self.req_resp-self.req_sent
+                
+                self.part_timeout_factor = Resource.PART_TIMEOUT_FACTOR_AFTER_RTT
+                if self.rtt == None:
+                    self.rtt = self.link.rtt
+                    self.watchdog_job()
+                elif rtt < self.rtt:
+                    self.rtt = max(self.rtt - self.rtt*0.05, rtt)
+                elif rtt > self.rtt:
+                    self.rtt = min(self.rtt + self.rtt*0.05, rtt)
 
-            if rtt > 0:
-                req_resp_cost = len(packet.raw)+self.req_sent_bytes
-                self.req_resp_rtt_rate = req_resp_cost / rtt
+                if rtt > 0:
+                    req_resp_cost = len(packet.raw)+self.req_sent_bytes
+                    self.req_resp_rtt_rate = req_resp_cost / rtt
 
-                if self.req_resp_rtt_rate > Resource.RATE_FAST and self.fast_rate_rounds < Resource.FAST_RATE_THRESHOLD:
-                    self.fast_rate_rounds += 1
+                    if self.req_resp_rtt_rate > Resource.RATE_FAST and self.fast_rate_rounds < Resource.FAST_RATE_THRESHOLD:
+                        self.fast_rate_rounds += 1
 
-                    if self.fast_rate_rounds == Resource.FAST_RATE_THRESHOLD:
-                        self.window_max = Resource.WINDOW_MAX_FAST
+                        if self.fast_rate_rounds == Resource.FAST_RATE_THRESHOLD:
+                            self.window_max = Resource.WINDOW_MAX_FAST
 
-        if not self.status == Resource.FAILED:
-            self.status = Resource.TRANSFERRING
-            part_data = packet.data
-            part_hash = self.get_map_hash(part_data)
+            if not self.status == Resource.FAILED:
+                self.status = Resource.TRANSFERRING
+                part_data = packet.data
+                part_hash = self.get_map_hash(part_data)
 
-            i = self.consecutive_completed_height
-            for map_hash in self.hashmap[self.consecutive_completed_height:self.consecutive_completed_height+self.window]:
-                if map_hash == part_hash:
-                    if self.parts[i] == None:
-                        # Insert data into parts list
-                        self.parts[i] = part_data
-                        self.rtt_rxd_bytes += len(part_data)
-                        self.received_count += 1
-                        self.outstanding_parts -= 1
+                consecutive_index = self.consecutive_completed_height if self.consecutive_completed_height >= 0 else 0
+                i = consecutive_index
+                for map_hash in self.hashmap[consecutive_index:consecutive_index+self.window]:
+                    if map_hash == part_hash:
+                        if self.parts[i] == None:
 
-                        # Update consecutive completed pointer
-                        if i == self.consecutive_completed_height + 1:
-                            self.consecutive_completed_height = i
-                        
-                        cp = self.consecutive_completed_height + 1
-                        while cp < len(self.parts) and self.parts[cp] != None:
-                            self.consecutive_completed_height = cp
-                            cp += 1
+                            # Insert data into parts list
+                            self.parts[i] = part_data
+                            self.rtt_rxd_bytes += len(part_data)
+                            self.received_count += 1
+                            self.outstanding_parts -= 1
 
-                        if self.__progress_callback != None:
-                            try:
-                                self.__progress_callback(self)
-                            except Exception as e:
-                                RNS.log("Error while executing progress callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            # Update consecutive completed pointer
+                            if i == self.consecutive_completed_height + 1:
+                                self.consecutive_completed_height = i
+                            
+                            cp = self.consecutive_completed_height + 1
+                            while cp < len(self.parts) and self.parts[cp] != None:
+                                self.consecutive_completed_height = cp
+                                cp += 1
 
-                i += 1
+                            if self.__progress_callback != None:
+                                try:
+                                    self.__progress_callback(self)
+                                except Exception as e:
+                                    RNS.log("Error while executing progress callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
 
-            self.receiving_part = False
+                    i += 1
 
-            if self.received_count == self.total_parts and not self.assembly_lock:
-                self.assembly_lock = True
-                self.assemble()
-            elif self.outstanding_parts == 0:
-                # TODO: Figure out if there is a mathematically
-                # optimal way to adjust windows
-                if self.window < self.window_max:
-                    self.window += 1
-                    if (self.window - self.window_min) > (self.window_flexibility-1):
-                        self.window_min += 1
+                self.receiving_part = False
 
-                if self.req_sent != 0:
-                    rtt = time.time()-self.req_sent
-                    req_transferred = self.rtt_rxd_bytes - self.rtt_rxd_bytes_at_part_req
+                if self.received_count == self.total_parts and not self.assembly_lock:
+                    self.assembly_lock = True
+                    self.assemble()
+                elif self.outstanding_parts == 0:
+                    # TODO: Figure out if there is a mathematically
+                    # optimal way to adjust windows
+                    if self.window < self.window_max:
+                        self.window += 1
+                        if (self.window - self.window_min) > (self.window_flexibility-1):
+                            self.window_min += 1
 
-                    if rtt != 0:
-                        self.req_data_rtt_rate = req_transferred/rtt
-                        self.rtt_rxd_bytes_at_part_req = self.rtt_rxd_bytes
+                    if self.req_sent != 0:
+                        rtt = time.time()-self.req_sent
+                        req_transferred = self.rtt_rxd_bytes - self.rtt_rxd_bytes_at_part_req
 
-                        if self.req_data_rtt_rate > Resource.RATE_FAST and self.fast_rate_rounds < Resource.FAST_RATE_THRESHOLD:
-                            self.fast_rate_rounds += 1
+                        if rtt != 0:
+                            self.req_data_rtt_rate = req_transferred/rtt
+                            self.update_eifr()
+                            self.rtt_rxd_bytes_at_part_req = self.rtt_rxd_bytes
 
-                            if self.fast_rate_rounds == Resource.FAST_RATE_THRESHOLD:
-                                self.window_max = Resource.WINDOW_MAX_FAST
+                            if self.req_data_rtt_rate > Resource.RATE_FAST and self.fast_rate_rounds < Resource.FAST_RATE_THRESHOLD:
+                                self.fast_rate_rounds += 1
 
-                self.request_next()
-        else:
-            self.receiving_part = False
+                                if self.fast_rate_rounds == Resource.FAST_RATE_THRESHOLD:
+                                    self.window_max = Resource.WINDOW_MAX_FAST
+
+                            if self.fast_rate_rounds == 0 and self.req_data_rtt_rate < Resource.RATE_VERY_SLOW and self.very_slow_rate_rounds < Resource.VERY_SLOW_RATE_THRESHOLD:
+                                self.very_slow_rate_rounds += 1
+
+                                if self.very_slow_rate_rounds == Resource.VERY_SLOW_RATE_THRESHOLD:
+                                    self.window_max = Resource.WINDOW_MAX_VERY_SLOW
+
+                    self.request_next()
+            else:
+                self.receiving_part = False
 
     # Called on incoming resource to send a request for more data
     def request_next(self):
@@ -720,11 +833,11 @@ class Resource:
                 hashmap_exhausted = Resource.HASHMAP_IS_NOT_EXHAUSTED
                 requested_hashes = b""
 
-                offset = (1 if self.consecutive_completed_height > 0 else 0)
-                i = 0; pn = self.consecutive_completed_height+offset
+                i = 0; pn = self.consecutive_completed_height+1
                 search_start = pn
-
-                for part in self.parts[search_start:search_start+self.window]:
+                search_size = self.window
+                
+                for part in self.parts[search_start:search_start+search_size]:
                     if part == None:
                         part_hash = self.hashmap[pn]
                         if part_hash != None:
@@ -744,7 +857,6 @@ class Resource:
                     hmu_part += last_map_hash
                     self.waiting_for_hmu = True
 
-                requested_data = b""
                 request_data = hmu_part + self.hash + requested_hashes
                 request_packet = RNS.Packet(self.link, request_data, context = RNS.Packet.RESOURCE_REQ)
 
@@ -754,6 +866,7 @@ class Resource:
                     self.req_sent = self.last_activity
                     self.req_sent_bytes = len(request_packet.raw)
                     self.req_resp = None
+
                 except Exception as e:
                     RNS.log("Could not send resource request packet, cancelling resource", RNS.LOG_DEBUG)
                     RNS.log("The contained exception was: "+str(e), RNS.LOG_DEBUG)
@@ -846,6 +959,7 @@ class Resource:
 
             if self.sent_parts == len(self.parts):
                 self.status = Resource.AWAITING_PROOF
+                self.retries_left = 3
 
             if self.__progress_callback != None:
                 try:
@@ -887,20 +1001,68 @@ class Resource:
         """
         :returns: The current progress of the resource transfer as a *float* between 0.0 and 1.0.
         """
-        if self.initiator:
-            self.processed_parts  = (self.segment_index-1)*math.ceil(Resource.MAX_EFFICIENT_SIZE/Resource.SDU)
-            self.processed_parts += self.sent_parts
-            self.progress_total_parts = float(self.grand_total_parts)
-        else:
-            self.processed_parts  = (self.segment_index-1)*math.ceil(Resource.MAX_EFFICIENT_SIZE/Resource.SDU)            
-            self.processed_parts += self.received_count
-            if self.split:
-                self.progress_total_parts = float(math.ceil(self.total_size/Resource.SDU))
-            else:
+        if self.status == RNS.Resource.COMPLETE and self.segment_index == self.total_segments:
+            return 1.0
+        
+        elif self.initiator:
+            if not self.split:
+                self.processed_parts = self.sent_parts
                 self.progress_total_parts = float(self.total_parts)
 
-        
-        progress = self.processed_parts / self.progress_total_parts
+            else:
+                is_last_segment = self.segment_index != self.total_segments
+                total_segments = self.total_segments
+                processed_segments = self.segment_index-1
+
+                current_segment_parts = self.total_parts
+                max_parts_per_segment = math.ceil(Resource.MAX_EFFICIENT_SIZE/self.sdu)
+
+                previously_processed_parts = processed_segments*max_parts_per_segment
+
+                if current_segment_parts < max_parts_per_segment:
+                    current_segment_factor = max_parts_per_segment / current_segment_parts
+                else:
+                    current_segment_factor = 1
+
+                self.processed_parts = previously_processed_parts + self.sent_parts*current_segment_factor
+                self.progress_total_parts = self.total_segments*max_parts_per_segment
+
+        else:
+            if not self.split:
+                self.processed_parts = self.received_count
+                self.progress_total_parts = float(self.total_parts)
+
+            else:
+                is_last_segment = self.segment_index != self.total_segments
+                total_segments = self.total_segments
+                processed_segments = self.segment_index-1
+
+                current_segment_parts = self.total_parts
+                max_parts_per_segment = math.ceil(Resource.MAX_EFFICIENT_SIZE/self.sdu)
+
+                previously_processed_parts = processed_segments*max_parts_per_segment
+
+                if current_segment_parts < max_parts_per_segment:
+                    current_segment_factor = max_parts_per_segment / current_segment_parts
+                else:
+                    current_segment_factor = 1
+
+                self.processed_parts = previously_processed_parts + self.received_count*current_segment_factor
+                self.progress_total_parts = self.total_segments*max_parts_per_segment
+
+
+        progress = min(1.0, self.processed_parts / self.progress_total_parts)
+        return progress
+
+    def get_segment_progress(self):
+        if self.status == RNS.Resource.COMPLETE and self.segment_index == self.total_segments:
+            return 1.0
+        elif self.initiator:
+            processed_parts = self.sent_parts
+        else:
+            processed_parts = self.received_count
+
+        progress = min(1.0, processed_parts / self.total_parts)
         return progress
 
     def get_transfer_size(self):
@@ -988,6 +1150,7 @@ class ResourceAdvertisement:
 
 
     def __init__(self, resource=None, request_id=None, is_response=False):
+        self.link = None
         if resource != None:
             self.t = resource.size              # Transfer size
             self.d = resource.total_size        # Total uncompressed data size
@@ -1034,6 +1197,9 @@ class ResourceAdvertisement:
     def is_compressed(self):
         return self.c
 
+    def get_link(self):
+        return self.link
+
     def pack(self, segment=0):
         hashmap_start = segment*ResourceAdvertisement.HASHMAP_MAX_LEN
         hashmap_end   = min((segment+1)*(ResourceAdvertisement.HASHMAP_MAX_LEN), self.n)

RNS/Utilities/__init__.py

@@ -23,5 +23,7 @@
 import os
 import glob
 
-modules = glob.glob(os.path.dirname(__file__)+"/*.py")
-__all__ = [ os.path.basename(f)[:-3] for f in modules if not f.endswith('__init__.py')]
+py_modules  = glob.glob(os.path.dirname(__file__)+"/*.py")
+pyc_modules = glob.glob(os.path.dirname(__file__)+"/*.pyc")
+modules     = py_modules+pyc_modules
+__all__ = list(set([os.path.basename(f).replace(".pyc", "").replace(".py", "") for f in modules if not (f.endswith("__init__.py") or f.endswith("__init__.pyc"))]))

RNS/Utilities/rncp.py

@@ -24,6 +24,7 @@
 
 import RNS
 import argparse
+import threading
 import time
 import sys
 import os
@@ -32,15 +33,48 @@ from RNS._version import __version__
 
 APP_NAME = "rncp"
 allow_all = False
+allow_fetch = False
+fetch_jail = None
+save_path = None
+show_phy_rates = False
 allowed_identity_hashes = []
 
-def receive(configdir, verbosity = 0, quietness = 0, allowed = [], display_identity = False, limit = None, disable_auth = None, disable_announce = False):
-    global allow_all, allowed_identity_hashes
+REQ_FETCH_NOT_ALLOWED = 0xF0
+
+es = " "
+erase_str = "\33[2K\r"
+
+def listen(configdir, verbosity = 0, quietness = 0, allowed = [], display_identity = False,
+           limit = None, disable_auth = None, fetch_allowed = False, jail = None, save = None, announce = False):
+    global allow_all, allow_fetch, allowed_identity_hashes, fetch_jail, save_path
+    from tempfile import TemporaryFile
+
+    allow_fetch = fetch_allowed
     identity = None
+    if announce < 0:
+        announce = False
 
     targetloglevel = 3+verbosity-quietness
     reticulum = RNS.Reticulum(configdir=configdir, loglevel=targetloglevel)
 
+    if jail != None:
+        fetch_jail = os.path.abspath(os.path.expanduser(jail))
+        RNS.log("Restricting fetch requests to paths under \""+fetch_jail+"\"", RNS.LOG_VERBOSE)
+
+    if save != None:
+        sp = os.path.abspath(os.path.expanduser(save))
+        if os.path.isdir(sp):
+            if os.access(sp, os.W_OK):
+                save_path = sp
+            else:
+                RNS.log("Output directory not writable", RNS.LOG_ERROR)
+                exit(4)
+        else:
+            RNS.log("Output directory not found", RNS.LOG_ERROR)
+            exit(3)
+
+        RNS.log("Saving received files in \""+save_path+"\"", RNS.LOG_VERBOSE)
+
     identity_path = RNS.Reticulum.identitypath+"/"+APP_NAME
     if os.path.isfile(identity_path):
         identity = RNS.Identity.from_file(identity_path)                
@@ -54,16 +88,48 @@ def receive(configdir, verbosity = 0, quietness = 0, allowed = [], display_ident
 
     if display_identity:
         print("Identity     : "+str(identity))
-        print("Receiving on : "+RNS.prettyhexrep(destination.hash))
+        print("Listening on : "+RNS.prettyhexrep(destination.hash))
         exit(0)
 
     if disable_auth:
         allow_all = True
     else:
+        dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+        try:
+            allowed_file_name = "allowed_identities"
+            allowed_file = None
+            if os.path.isfile(os.path.expanduser("/etc/rncp/"+allowed_file_name)):
+                allowed_file = os.path.expanduser("/etc/rncp/"+allowed_file_name)
+            elif os.path.isfile(os.path.expanduser("~/.config/rncp/"+allowed_file_name)):
+                allowed_file = os.path.expanduser("~/.config/rncp/"+allowed_file_name)
+            elif os.path.isfile(os.path.expanduser("~/.rncp/"+allowed_file_name)):
+                allowed_file = os.path.expanduser("~/.rncp/"+allowed_file_name)
+            if allowed_file != None:
+                af = open(allowed_file, "r")
+                al = af.read().replace("\r", "").split("\n")
+                ali = []
+                for a in al:
+                    if len(a) == dest_len:
+                        ali.append(a)
+
+                if len(ali) > 0:
+                    if not allowed:
+                        allowed = ali
+                    else:
+                        allowed.extend(ali)
+                if len(ali) == 1:
+                    ms = "y"
+                else:
+                    ms = "ies"
+                
+                RNS.log("Loaded "+str(len(ali))+" allowed identit"+ms+" from "+str(allowed_file), RNS.LOG_VERBOSE)
+
+        except Exception as e:
+            RNS.log("Error while parsing allowed_identities file. The contained exception was: "+str(e), RNS.LOG_ERROR)
+
         if allowed != None:
             for a in allowed:
                 try:
-                    dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
                     if len(a) != dest_len:
                         raise ValueError("Allowed destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2))
                     try:
@@ -78,16 +144,77 @@ def receive(configdir, verbosity = 0, quietness = 0, allowed = [], display_ident
     if len(allowed_identity_hashes) < 1 and not disable_auth:
         print("Warning: No allowed identities configured, rncp will not accept any files!")
 
-    destination.set_link_established_callback(receive_link_established)
-    print("rncp ready to receive on "+RNS.prettyhexrep(destination.hash))
+    def fetch_request(path, data, request_id, link_id, remote_identity, requested_at):
+        global allow_fetch, fetch_jail
+        if not allow_fetch:
+            return REQ_FETCH_NOT_ALLOWED
 
-    if not disable_announce:
-        destination.announce()
+        if fetch_jail:
+            if data.startswith(fetch_jail+"/"):
+                data = data.replace(fetch_jail+"/", "")
+            file_path = os.path.abspath(os.path.expanduser(f"{fetch_jail}/{data}"))
+            if not file_path.startswith(fetch_jail+"/"):
+                RNS.log(f"Disallowing fetch request for {file_path} outside of fetch jail {fetch_jail}", RNS.LOG_WARNING)
+                return REQ_FETCH_NOT_ALLOWED
+        else:
+            file_path = os.path.abspath(os.path.expanduser(f"{data}"))
+
+        target_link = None
+        for link in RNS.Transport.active_links:
+            if link.link_id == link_id:
+                target_link = link
+
+        if not os.path.isfile(file_path):
+            RNS.log("Client-requested file not found: "+str(file_path), RNS.LOG_VERBOSE)
+            return False
+        else:
+            if target_link != None:
+                RNS.log("Sending file "+str(file_path)+" to client", RNS.LOG_VERBOSE)
+
+                temp_file = TemporaryFile()
+                real_file = open(file_path, "rb")
+                filename_bytes = os.path.basename(file_path).encode("utf-8")
+                filename_len = len(filename_bytes)
+
+                if filename_len > 0xFFFF:
+                    print("Filename exceeds max size, cannot send")
+                    exit(1)
+
+                temp_file.write(filename_len.to_bytes(2, "big"))
+                temp_file.write(filename_bytes)
+                temp_file.write(real_file.read())
+                temp_file.seek(0)
+
+                fetch_resource = RNS.Resource(temp_file, target_link)
+                return True
+            else:
+                return None
+
+
+    destination.set_link_established_callback(client_link_established)
+    if allow_fetch:
+        if allow_all:
+            RNS.log("Allowing unauthenticated fetch requests", RNS.LOG_WARNING)
+            destination.register_request_handler("fetch_file", response_generator=fetch_request, allow=RNS.Destination.ALLOW_ALL)
+        else:
+            destination.register_request_handler("fetch_file", response_generator=fetch_request, allow=RNS.Destination.ALLOW_LIST, allowed_list=allowed_identity_hashes)
+
+    print("rncp listening on "+RNS.prettyhexrep(destination.hash))
+
+    if announce >= 0:
+        def job():
+            destination.announce()
+            if announce > 0:
+                while True:
+                    time.sleep(announce)
+                    destination.announce()
+
+        threading.Thread(target=job, daemon=True).start()
     
     while True:
         time.sleep(1)
 
-def receive_link_established(link):
+def client_link_established(link):
     RNS.log("Incoming link established", RNS.LOG_VERBOSE)
     link.set_remote_identified_callback(receive_sender_identified)
     link.set_resource_strategy(RNS.Link.ACCEPT_APP)
@@ -130,6 +257,7 @@ def receive_resource_started(resource):
     print("Starting resource transfer "+RNS.prettyhexrep(resource.hash)+id_str)
 
 def receive_resource_concluded(resource):
+    global save_path
     if resource.status == RNS.Resource.COMPLETE:
         print(str(resource)+" completed")
 
@@ -138,12 +266,20 @@ def receive_resource_concluded(resource):
             filename = resource.data.read(filename_len).decode("utf-8")
 
             counter = 0
-            saved_filename = filename
-            while os.path.isfile(saved_filename):
+            if save_path:
+                saved_filename = os.path.abspath(os.path.expanduser(save_path+"/"+filename))
+                if not saved_filename.startswith(save_path+"/"):
+                    RNS.log(f"Invalid save path {saved_filename}, ignoring", RNS.LOG_ERROR)
+                    return
+            else:
+                saved_filename = filename
+
+            full_save_path = saved_filename
+            while os.path.isfile(full_save_path):
                 counter += 1
-                saved_filename = filename+"."+str(counter)
+                full_save_path = saved_filename+"."+str(counter)
             
-            file = open(saved_filename, "wb")
+            file = open(full_save_path, "wb")
             file.write(resource.data.read())
             file.close()
 
@@ -157,34 +293,301 @@ resource_done = False
 current_resource = None
 stats = []
 speed = 0.0
+phy_speed = 0.0
+phy_got_total = 0
 def sender_progress(resource):
     stats_max = 32
-    global current_resource, stats, speed, resource_done
+    global current_resource, stats, speed, phy_speed, phy_got_total, resource_done
     current_resource = resource
+    
     now = time.time()
-    got = current_resource.get_progress()*current_resource.total_size
-    entry = [now, got]
+    got = current_resource.get_progress()*current_resource.get_data_size()
+    phy_got = current_resource.get_segment_progress()*current_resource.get_transfer_size()
+    
+    entry = [now, got, phy_got]
     stats.append(entry)
+
     while len(stats) > stats_max:
         stats.pop(0)
 
     span = now - stats[0][0]
     if span == 0:
         speed = 0
+        phy_speed = 0
+    
     else:
         diff = got - stats[0][1]
         speed = diff/span
 
+        phy_diff = phy_got - stats[0][2]
+        if phy_diff > 0:
+            phy_speed = phy_diff/span
+            # phy_got_total += phy_diff
+
     if resource.status < RNS.Resource.COMPLETE:
         resource_done = False
     else:
         resource_done = True
 
 link = None
-def send(configdir, verbosity = 0, quietness = 0, destination = None, file = None, timeout = RNS.Transport.PATH_REQUEST_TIMEOUT):
-    global current_resource, resource_done, link, speed
+def fetch(configdir, verbosity = 0, quietness = 0, destination = None, file = None, timeout = RNS.Transport.PATH_REQUEST_TIMEOUT, silent=False, phy_rates=False, save=None):
+    global current_resource, resource_done, link, speed, show_phy_rates, save_path
+    targetloglevel = 3+verbosity-quietness
+    show_phy_rates = phy_rates
+
+    if save:
+        sp = os.path.abspath(os.path.expanduser(save))
+        if os.path.isdir(sp):
+            if os.access(sp, os.W_OK):
+                save_path = sp
+            else:
+                RNS.log("Output directory not writable", RNS.LOG_ERROR)
+                exit(4)
+        else:
+            RNS.log("Output directory not found", RNS.LOG_ERROR)
+            exit(3)
+
+    try:
+        dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+        if len(destination) != dest_len:
+            raise ValueError("Allowed destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2))
+        try:
+            destination_hash = bytes.fromhex(destination)
+        except Exception as e:
+            raise ValueError("Invalid destination entered. Check your input.")
+    except Exception as e:
+        print(str(e))
+        exit(1)
+
+    reticulum = RNS.Reticulum(configdir=configdir, loglevel=targetloglevel)
+
+    identity_path = RNS.Reticulum.identitypath+"/"+APP_NAME
+    if os.path.isfile(identity_path):
+        identity = RNS.Identity.from_file(identity_path)
+        if identity == None:
+            RNS.log("Could not load identity for rncp. The identity file at \""+str(identity_path)+"\" may be corrupt or unreadable.", RNS.LOG_ERROR)
+            exit(2)
+    else:
+        identity = None
+
+    if identity == None:
+        RNS.log("No valid saved identity found, creating new...", RNS.LOG_INFO)
+        identity = RNS.Identity()
+        identity.to_file(identity_path)
+
+    if not RNS.Transport.has_path(destination_hash):
+        RNS.Transport.request_path(destination_hash)
+        if silent:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested")
+        else:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested  ", end=es)
+        sys.stdout.flush()
+
+    i = 0
+    syms = "⢄⢂⢁⡁⡈⡐⡠"
+    estab_timeout = time.time()+timeout
+    while not RNS.Transport.has_path(destination_hash) and time.time() < estab_timeout:
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
+
+    if not RNS.Transport.has_path(destination_hash):
+        if silent:
+            print("Path not found")
+        else:
+            print(f"{erase_str}Path not found")
+        exit(1)
+    else:
+        if silent:
+            print("Establishing link with "+RNS.prettyhexrep(destination_hash))
+        else:
+            print(f"{erase_str}Establishing link with "+RNS.prettyhexrep(destination_hash)+"  ", end=es)
+
+    listener_identity = RNS.Identity.recall(destination_hash)
+    listener_destination = RNS.Destination(
+        listener_identity,
+        RNS.Destination.OUT,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "receive"
+    )
+
+    link = RNS.Link(listener_destination)
+    while link.status != RNS.Link.ACTIVE and time.time() < estab_timeout:
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
+
+    if not RNS.Transport.has_path(destination_hash):
+        if silent:
+            print("Could not establish link with "+RNS.prettyhexrep(destination_hash))
+        else:
+            print(f"{erase_str}Could not establish link with "+RNS.prettyhexrep(destination_hash))
+        exit(1)
+    else:
+        if silent:
+            print("Requesting file from remote...")
+        else:
+            print(f"{erase_str}Requesting file from remote  ", end=es)
+
+    link.identify(identity)
+
+    request_resolved = False
+    request_status = "unknown"
+    resource_resolved = False
+    resource_status = "unrequested"
+    current_resource = None
+    def request_response(request_receipt):
+        nonlocal request_resolved, request_status
+        if request_receipt.response == False:
+            request_status = "not_found"
+        elif request_receipt.response == None:
+            request_status = "remote_error"
+        elif request_receipt.response == REQ_FETCH_NOT_ALLOWED:
+            request_status = "fetch_not_allowed"
+        else:
+            request_status = "found"
+
+        request_resolved = True
+
+    def request_failed(request_receipt):
+        nonlocal request_resolved, request_status
+        request_status = "unknown"
+        request_resolved = True
+
+    def fetch_resource_started(resource):
+        nonlocal resource_status
+        current_resource = resource
+        current_resource.progress_callback(sender_progress)
+        resource_status = "started"
+
+    def fetch_resource_concluded(resource):
+        nonlocal resource_resolved, resource_status
+        global save_path
+        if resource.status == RNS.Resource.COMPLETE:
+            if resource.total_size > 4:
+                filename_len = int.from_bytes(resource.data.read(2), "big")
+                filename = resource.data.read(filename_len).decode("utf-8")
+
+                counter = 0
+                if save_path:
+                    saved_filename = os.path.abspath(os.path.expanduser(save_path+"/"+filename))
+
+                else:
+                    saved_filename = filename
+
+                full_save_path = saved_filename
+                while os.path.isfile(full_save_path):
+                    counter += 1
+                    full_save_path = saved_filename+"."+str(counter)
+ 
+                file = open(full_save_path, "wb")
+                file.write(resource.data.read())
+                file.close()
+                resource_status = "completed"
+
+            else:
+                print("Invalid data received, ignoring resource")
+                resource_status = "invalid_data"
+
+        else:
+            print("Resource failed")
+            resource_status = "failed"
+
+        resource_resolved = True
+
+    link.set_resource_strategy(RNS.Link.ACCEPT_ALL)
+    link.set_resource_started_callback(fetch_resource_started)
+    link.set_resource_concluded_callback(fetch_resource_concluded)
+    link.request("fetch_file", data=file, response_callback=request_response, failed_callback=request_failed)
+
+    syms = "⢄⢂⢁⡁⡈⡐⡠"
+    while not request_resolved:
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
+
+    if request_status == "fetch_not_allowed":
+        if not silent: print(f"{erase_str}", end="")
+        print("Fetch request failed, fetching the file "+str(file)+" was not allowed by the remote")
+        link.teardown()
+        time.sleep(0.15)
+        exit(0)
+    elif request_status == "not_found":
+        if not silent: print(f"{erase_str}", end="")
+        print("Fetch request failed, the file "+str(file)+" was not found on the remote")
+        link.teardown()
+        time.sleep(0.15)
+        exit(0)
+    elif request_status == "remote_error":
+        if not silent: print(f"{erase_str}", end="")
+        print("Fetch request failed due to an error on the remote system")
+        link.teardown()
+        time.sleep(0.15)
+        exit(0)
+    elif request_status == "unknown":
+        if not silent: print(f"{erase_str}", end="")
+        print("Fetch request failed due to an unknown error (probably not authorised)")
+        link.teardown()
+        time.sleep(0.15)
+        exit(0)
+    elif request_status == "found":
+        if not silent: print(f"{erase_str}", end="")
+
+    while not resource_resolved:
+        if not silent:
+            time.sleep(0.1)
+            if current_resource:
+                prg = current_resource.get_progress()
+                percent = round(prg * 100.0, 1)
+                if show_phy_rates:
+                    pss = size_str(phy_speed, "b")
+                    phy_str = f" ({pss}ps at physical layer)"
+                else:
+                    phy_str = ""
+                ps = size_str(int(prg*current_resource.total_size))
+                ts = size_str(current_resource.total_size)
+                ss = size_str(speed, "b")
+                stat_str = f"{percent}% - {ps} of {ts} - {ss}ps{phy_str}"
+                if prg != 1.0:
+                    print(f"{erase_str}Transferring file {syms[i]} {stat_str}", end=es)
+                else:
+                    print(f"{erase_str}Transfer complete  {stat_str}", end=es)
+            else:
+                print(f"{erase_str}Waiting for transfer to start {syms[i]} ", end=es)
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
+
+    if current_resource.status != RNS.Resource.COMPLETE:
+        if silent:
+            print("The transfer failed")
+        else:
+            print(f"{erase_str}The transfer failed")
+        exit(1)
+    else:
+        if silent:
+            print(str(file)+" fetched from "+RNS.prettyhexrep(destination_hash))
+        else:
+            print("\n"+str(file)+" fetched from "+RNS.prettyhexrep(destination_hash))
+        link.teardown()
+        time.sleep(0.15)
+        exit(0)
+
+    link.teardown()
+    exit(0)
+
+
+def send(configdir, verbosity = 0, quietness = 0, destination = None, file = None, timeout = RNS.Transport.PATH_REQUEST_TIMEOUT, silent=False, phy_rates=False, no_compress=False):
+    global current_resource, resource_done, link, speed, show_phy_rates, phy_got_total, phy_speed
     from tempfile import TemporaryFile
     targetloglevel = 3+verbosity-quietness
+    show_phy_rates = phy_rates
 
     try:
         dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
@@ -213,20 +616,25 @@ def send(configdir, verbosity = 0, quietness = 0, destination = None, file = Non
         print("Filename exceeds max size, cannot send")
         exit(1)
     else:
-        print("Preparing file...", end=" ")
+        print("Preparing file...", end=es)
 
     temp_file.write(filename_len.to_bytes(2, "big"))
     temp_file.write(filename_bytes)
     temp_file.write(real_file.read())
     temp_file.seek(0)
 
-    print("\r                                                            \r", end="")
+    print(f"{erase_str}", end="")
 
     reticulum = RNS.Reticulum(configdir=configdir, loglevel=targetloglevel)
 
     identity_path = RNS.Reticulum.identitypath+"/"+APP_NAME
     if os.path.isfile(identity_path):
-        identity = RNS.Identity.from_file(identity_path)                
+        identity = RNS.Identity.from_file(identity_path)
+        if identity == None:
+            RNS.log("Could not load identity for rncp. The identity file at \""+str(identity_path)+"\" may be corrupt or unreadable.", RNS.LOG_ERROR)
+            exit(2)
+    else:
+        identity = None
 
     if identity == None:
         RNS.log("No valid saved identity found, creating new...", RNS.LOG_INFO)
@@ -235,23 +643,33 @@ def send(configdir, verbosity = 0, quietness = 0, destination = None, file = Non
 
     if not RNS.Transport.has_path(destination_hash):
         RNS.Transport.request_path(destination_hash)
-        print("Path to "+RNS.prettyhexrep(destination_hash)+" requested  ", end=" ")
+        if silent:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested")
+        else:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested  ", end=es)
         sys.stdout.flush()
 
     i = 0
     syms = "⢄⢂⢁⡁⡈⡐⡠"
     estab_timeout = time.time()+timeout
     while not RNS.Transport.has_path(destination_hash) and time.time() < estab_timeout:
-        time.sleep(0.1)
-        print(("\b\b"+syms[i]+" "), end="")
-        sys.stdout.flush()
-        i = (i+1)%len(syms)
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
 
     if not RNS.Transport.has_path(destination_hash):
-        print("\r                                                            \rPath not found")
+        if silent:
+            print("Path not found")
+        else:
+            print(f"{erase_str}Path not found")
         exit(1)
     else:
-        print("\r                                                            \rEstablishing link with "+RNS.prettyhexrep(destination_hash)+" ", end=" ")
+        if silent:
+            print("Establishing link with "+RNS.prettyhexrep(destination_hash))
+        else:
+            print(f"{erase_str}Establishing link with "+RNS.prettyhexrep(destination_hash)+" ", end=es)
 
     receiver_identity = RNS.Identity.recall(destination_hash)
     receiver_destination = RNS.Destination(
@@ -264,48 +682,103 @@ def send(configdir, verbosity = 0, quietness = 0, destination = None, file = Non
 
     link = RNS.Link(receiver_destination)
     while link.status != RNS.Link.ACTIVE and time.time() < estab_timeout:
-        time.sleep(0.1)
-        print(("\b\b"+syms[i]+" "), end="")
-        sys.stdout.flush()
-        i = (i+1)%len(syms)
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
 
-    if not RNS.Transport.has_path(destination_hash):
-        print("\r                                                            \rCould not establish link with "+RNS.prettyhexrep(destination_hash))
+    if time.time() > estab_timeout:
+        if silent:
+            print("Link establishment with "+RNS.prettyhexrep(destination_hash)+" timed out")
+        else:
+            print(f"{erase_str}Link establishment with "+RNS.prettyhexrep(destination_hash)+" timed out")
+        exit(1)
+    elif not RNS.Transport.has_path(destination_hash):
+        if silent:
+            print("No path found to "+RNS.prettyhexrep(destination_hash))
+        else:
+            print(f"{erase_str}No path found to "+RNS.prettyhexrep(destination_hash))
         exit(1)
     else:
-        print("\r                                                            \rAdvertising file resource  ", end=" ")
+        if silent:
+            print("Advertising file resource...")
+        else:
+            print(f"{erase_str}Advertising file resource  ", end=es)
 
     link.identify(identity)
-    resource = RNS.Resource(temp_file, link, callback = sender_progress, progress_callback = sender_progress)
+    auto_compress = True
+    if no_compress:
+        auto_compress = False
+    resource = RNS.Resource(temp_file, link, callback = sender_progress, progress_callback = sender_progress, auto_compress = auto_compress)
     current_resource = resource
 
     while resource.status < RNS.Resource.TRANSFERRING:
-        time.sleep(0.1)
-        print(("\b\b"+syms[i]+" "), end="")
-        sys.stdout.flush()
-        i = (i+1)%len(syms)
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
 
+    resource_started_at = time.time()
     
     if resource.status > RNS.Resource.COMPLETE:
-        print("\r                                                            \rFile was not accepted by "+RNS.prettyhexrep(destination_hash))
+        if silent:
+            print("File was not accepted by "+RNS.prettyhexrep(destination_hash))
+        else:
+            print(f"{erase_str}File was not accepted by "+RNS.prettyhexrep(destination_hash))
         exit(1)
     else:
-        print("\r                                                            \rTransferring file  ", end=" ")
+        if silent:
+            print("Transferring file...")
+        else:
+            print(f"{erase_str}Transferring file  ", end=es)
 
-    while not resource_done:
+    def progress_update(i, done=False):
         time.sleep(0.1)
         prg = current_resource.get_progress()
         percent = round(prg * 100.0, 1)
-        stat_str = str(percent)+"% - " + size_str(int(prg*current_resource.total_size)) + " of " + size_str(current_resource.total_size) + " - " +size_str(speed, "b")+"ps"
-        print("\r                                                                                  \rTransferring file "+syms[i]+" "+stat_str, end=" ")
+        if show_phy_rates and not resource_done:
+            pss = size_str(phy_speed, "b")
+            phy_str = f" ({pss}ps at physical layer)"
+        else:
+            phy_str = ""
+        es = "  "
+        cs = size_str(int(prg*current_resource.total_size))
+        ts = size_str(current_resource.total_size)
+        ss = size_str(speed, "b")
+        stat_str = f"{percent}% - {cs} of {ts} - {ss}ps{phy_str}"
+        if not done:
+            print(f"{erase_str}Transferring file "+syms[i]+" "+stat_str, end=es)
+        else:
+            print(f"{erase_str}Transfer complete  "+stat_str, end=es)
         sys.stdout.flush()
         i = (i+1)%len(syms)
+        return i
+
+    while not resource_done:
+        if not silent:
+            i = progress_update(i)
+
+    resource_concluded_at = time.time()
+    transfer_time = resource_concluded_at - resource_started_at
+    speed = current_resource.total_size/transfer_time
+    # phy_speed = phy_got_total/transfer_time
+
+    if not silent:
+        i = progress_update(i, done=True)
 
     if current_resource.status != RNS.Resource.COMPLETE:
-        print("\r                                                            \rThe transfer failed")
+        if silent:
+            print("The transfer failed")
+        else:
+            print(f"{erase_str}The transfer failed")
         exit(1)
     else:
-        print("\r                                                                                  \r"+str(file_path)+" copied to "+RNS.prettyhexrep(destination_hash))
+        if silent:
+            print(str(file_path)+" copied to "+RNS.prettyhexrep(destination_hash))
+        else:
+            print("\n"+str(file_path)+" copied to "+RNS.prettyhexrep(destination_hash))
         link.teardown()
         time.sleep(0.25)
         real_file.close()
@@ -320,29 +793,57 @@ def main():
         parser.add_argument("--config", metavar="path", action="store", default=None, help="path to alternative Reticulum config directory", type=str)
         parser.add_argument('-v', '--verbose', action='count', default=0, help="increase verbosity")
         parser.add_argument('-q', '--quiet', action='count', default=0, help="decrease verbosity")
+        parser.add_argument("-S", '--silent', action='store_true', default=False, help="disable transfer progress output")
+        parser.add_argument("-l", '--listen', action='store_true', default=False, help="listen for incoming transfer requests")
+        parser.add_argument("-C", '--no-compress', action='store_true', default=False, help="disable automatic compression")
+        parser.add_argument("-F", '--allow-fetch', action='store_true', default=False, help="allow authenticated clients to fetch files")
+        parser.add_argument("-f", '--fetch', action='store_true', default=False, help="fetch file from remote listener instead of sending")
+        parser.add_argument("-j", "--jail", metavar="path", action="store", default=None, help="restrict fetch requests to specified path", type=str)
+        parser.add_argument("-s", "--save", metavar="path", action="store", default=None, help="save received files in specified path", type=str)
+        parser.add_argument("-b", action='store', metavar="seconds", default=-1, help="announce interval, 0 to only announce at startup", type=int)
+        parser.add_argument('-a', metavar="allowed_hash", dest="allowed", action='append', help="allow this identity (or add in ~/.rncp/allowed_identities)", type=str)
+        parser.add_argument('-n', '--no-auth', action='store_true', default=False, help="accept requests from anyone")
         parser.add_argument('-p', '--print-identity', action='store_true', default=False, help="print identity and destination info and exit")
-        parser.add_argument("-r", '--receive', action='store_true', default=False, help="wait for incoming files")
-        parser.add_argument("-b", '--no-announce', action='store_true', default=False, help="don't announce at program start")
-        parser.add_argument('-a', metavar="allowed_hash", dest="allowed", action='append', help="accept from this identity", type=str)
-        parser.add_argument('-n', '--no-auth', action='store_true', default=False, help="accept files from anyone")
         parser.add_argument("-w", action="store", metavar="seconds", type=float, help="sender timeout before giving up", default=RNS.Transport.PATH_REQUEST_TIMEOUT)
+        parser.add_argument('-P', '--phy-rates', action='store_true', default=False, help="display physical layer transfer rates")
         # parser.add_argument("--limit", action="store", metavar="files", type=float, help="maximum number of files to accept", default=None)
         parser.add_argument("--version", action="version", version="rncp {version}".format(version=__version__))
         
         args = parser.parse_args()
 
-        if args.receive or args.print_identity:
-            receive(
+        if args.listen or args.print_identity:
+            listen(
                 configdir = args.config,
                 verbosity=args.verbose,
                 quietness=args.quiet,
                 allowed = args.allowed,
+                fetch_allowed = args.allow_fetch,
+                jail = args.jail,
+                save = args.save,
                 display_identity=args.print_identity,
                 # limit=args.limit,
                 disable_auth=args.no_auth,
-                disable_announce=args.no_announce,
+                announce=args.b,
             )
 
+        elif args.fetch:
+            if args.destination != None and args.file != None:
+                fetch(
+                    configdir = args.config,
+                    verbosity = args.verbose,
+                    quietness = args.quiet,
+                    destination = args.destination,
+                    file = args.file,
+                    timeout = args.w,
+                    silent = args.silent,
+                    phy_rates = args.phy_rates,
+                    save = args.save,
+                )
+            else:
+                print("")
+                parser.print_help()
+                print("")
+
         elif args.destination != None and args.file != None:
             send(
                 configdir = args.config,
@@ -351,6 +852,9 @@ def main():
                 destination = args.destination,
                 file = args.file,
                 timeout = args.w,
+                silent = args.silent,
+                phy_rates = args.phy_rates,
+                no_compress = args.no_compress,
             )
 
         else:

RNS/Utilities/rnid.py

@@ -0,0 +1,600 @@
+#!/usr/bin/env python3
+
+# MIT License
+#
+# Copyright (c) 2023 Mark Qvist / unsigned.io
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import RNS
+import argparse
+import time
+import sys
+import os
+import base64
+
+from RNS._version import __version__
+
+APP_NAME = "rnid"
+
+SIG_EXT = "rsg"
+ENCRYPT_EXT = "rfe"
+CHUNK_SIZE = 16*1024*1024
+
+def spin(until=None, msg=None, timeout=None):
+    i = 0
+    syms = "⢄⢂⢁⡁⡈⡐⡠"
+    if timeout != None:
+        timeout = time.time()+timeout
+
+    print(msg+"  ", end=" ")
+    while (timeout == None or time.time()<timeout) and not until():
+        time.sleep(0.1)
+        print(("\b\b"+syms[i]+" "), end="")
+        sys.stdout.flush()
+        i = (i+1)%len(syms)
+
+    print("\r"+" "*len(msg)+"  \r", end="")
+
+    if timeout != None and time.time() > timeout:
+        return False
+    else:
+        return True
+
+def main():
+    try:
+        parser = argparse.ArgumentParser(description="Reticulum Identity & Encryption Utility")
+        # parser.add_argument("file", nargs="?", default=None, help="input file path", type=str)
+
+        parser.add_argument("--config", metavar="path", action="store", default=None, help="path to alternative Reticulum config directory", type=str)
+        parser.add_argument("-i", "--identity", metavar="identity", action="store", default=None, help="hexadecimal Reticulum Destination hash or path to Identity file", type=str)
+        parser.add_argument("-g", "--generate", metavar="file", action="store", default=None, help="generate a new Identity")
+        parser.add_argument("-m", "--import", dest="import_str", metavar="identity_data", action="store", default=None, help="import Reticulum identity in hex, base32 or base64 format", type=str)
+        parser.add_argument("-x", "--export", action="store_true", default=None, help="export identity to hex, base32 or base64 format")
+
+        parser.add_argument("-v", "--verbose", action="count", default=0, help="increase verbosity")
+        parser.add_argument("-q", "--quiet", action="count", default=0, help="decrease verbosity")
+
+        parser.add_argument("-a", "--announce", metavar="aspects", action="store", default=None, help="announce a destination based on this Identity")
+        parser.add_argument("-H", "--hash", metavar="aspects", action="store", default=None, help="show destination hashes for other aspects for this Identity")
+        parser.add_argument("-e", "--encrypt", metavar="file", action="store", default=None, help="encrypt file")
+        parser.add_argument("-d", "--decrypt", metavar="file", action="store", default=None, help="decrypt file")
+        parser.add_argument("-s", "--sign", metavar="path", action="store", default=None, help="sign file")
+        parser.add_argument("-V", "--validate", metavar="path", action="store", default=None, help="validate signature")
+
+        parser.add_argument("-r", "--read", metavar="file", action="store", default=None, help="input file path", type=str)
+        parser.add_argument("-w", "--write", metavar="file", action="store", default=None, help="output file path", type=str)
+        parser.add_argument("-f", "--force", action="store_true", default=None, help="write output even if it overwrites existing files")
+        parser.add_argument("-I", "--stdin", action="store_true", default=False, help=argparse.SUPPRESS) # "read input from STDIN instead of file"
+        parser.add_argument("-O", "--stdout", action="store_true", default=False, help=argparse.SUPPRESS) # help="write output to STDOUT instead of file", 
+
+        parser.add_argument("-R", "--request", action="store_true", default=False, help="request unknown Identities from the network")
+        parser.add_argument("-t", action="store", metavar="seconds", type=float, help="identity request timeout before giving up", default=RNS.Transport.PATH_REQUEST_TIMEOUT)
+        parser.add_argument("-p", "--print-identity", action="store_true", default=False, help="print identity info and exit")
+        parser.add_argument("-P", "--print-private", action="store_true", default=False, help="allow displaying private keys")
+
+        parser.add_argument("-b", "--base64", action="store_true", default=False, help="Use base64-encoded input and output")
+        parser.add_argument("-B", "--base32", action="store_true", default=False, help="Use base32-encoded input and output")
+
+        parser.add_argument("--version", action="version", version="rnid {version}".format(version=__version__))
+        
+        args = parser.parse_args()
+
+        ops = 0;
+        for t in [args.encrypt, args.decrypt, args.validate, args.sign]:
+            if t:
+                ops += 1
+        
+        if ops > 1:
+            RNS.log("This utility currently only supports one of the encrypt, decrypt, sign or verify operations per invocation", RNS.LOG_ERROR)
+            exit(1)
+
+        if not args.read:
+            if args.encrypt:
+                args.read = args.encrypt
+            if args.decrypt:
+                args.read = args.decrypt
+            if args.sign:
+                args.read = args.sign
+
+        identity_str = args.identity
+        if args.import_str:
+            identity_bytes = None
+            try:
+                if args.base64:
+                    identity_bytes = base64.urlsafe_b64decode(args.import_str)
+                elif args.base32:
+                    identity_bytes = base64.b32decode(args.import_str)
+                else:
+                    identity_bytes = bytes.fromhex(args.import_str)
+            except Exception as e:
+                print("Invalid identity data specified for import: "+str(e))
+                exit(41)
+
+            try:
+                identity = RNS.Identity.from_bytes(identity_bytes)
+            except Exception as e:
+                print("Could not create Reticulum identity from specified data: "+str(e))
+                exit(42)
+
+            RNS.log("Identity imported")
+            if args.base64:
+                RNS.log("Public Key  : "+base64.urlsafe_b64encode(identity.get_public_key()).decode("utf-8"))
+            elif args.base32:
+                RNS.log("Public Key  : "+base64.b32encode(identity.get_public_key()).decode("utf-8"))
+            else:
+                RNS.log("Public Key  : "+RNS.hexrep(identity.get_public_key(), delimit=False))
+            if identity.prv:
+                if args.print_private:
+                    if args.base64:
+                        RNS.log("Private Key : "+base64.urlsafe_b64encode(identity.get_private_key()).decode("utf-8"))
+                    elif args.base32:
+                        RNS.log("Private Key : "+base64.b32encode(identity.get_private_key()).decode("utf-8"))
+                    else:
+                        RNS.log("Private Key : "+RNS.hexrep(identity.get_private_key(), delimit=False))
+                else:
+                    RNS.log("Private Key : Hidden")
+
+            if args.write:
+                try:
+                    wp = os.path.expanduser(args.write)
+                    if not os.path.isfile(wp) or args.force:
+                        identity.to_file(wp)
+                        RNS.log("Wrote imported identity to "+str(args.write))
+                    else:
+                        print("File "+str(wp)+" already exists, not overwriting")
+                        exit(43)
+
+                except Exception as e:
+                    print("Error while writing imported identity to file: "+str(e))
+                    exit(44)
+
+            exit(0)
+
+        if not args.generate and not identity_str:
+            print("\nNo identity provided, cannot continue\n")
+            parser.print_help()
+            print("")
+            exit(2)
+
+        else:
+            targetloglevel = 4
+            verbosity = args.verbose
+            quietness = args.quiet
+            if verbosity != 0 or quietness != 0:
+                targetloglevel = targetloglevel+verbosity-quietness
+            
+            # Start Reticulum
+            reticulum = RNS.Reticulum(configdir=args.config, loglevel=targetloglevel)
+            RNS.compact_log_fmt = True
+            if args.stdout:
+                RNS.loglevel = -1
+
+            if args.generate:
+                identity = RNS.Identity()
+                if not args.force and os.path.isfile(args.generate):
+                    RNS.log("Identity file "+str(args.generate)+" already exists. Not overwriting.", RNS.LOG_ERROR)
+                    exit(3)
+                else:
+                    try:
+                        identity.to_file(args.generate)
+                        RNS.log("New identity written to "+str(args.generate))
+                        exit(0)
+                    except Exception as e:
+                        RNS.log("An error ocurred while saving the generated Identity.", RNS.LOG_ERROR)
+                        RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                        exit(4)
+
+            identity = None
+            if len(identity_str) == RNS.Reticulum.TRUNCATED_HASHLENGTH//8*2 and not os.path.isfile(identity_str):
+                # Try recalling Identity from hex-encoded hash
+                try:
+                    destination_hash = bytes.fromhex(identity_str)
+                    identity = RNS.Identity.recall(destination_hash)
+
+                    if identity == None:
+                        if not args.request:
+                            RNS.log("Could not recall Identity for "+RNS.prettyhexrep(destination_hash)+".", RNS.LOG_ERROR)
+                            RNS.log("You can query the network for unknown Identities with the -R option.", RNS.LOG_ERROR)
+                            exit(5)
+                        else:
+                            RNS.Transport.request_path(destination_hash)
+                            def spincheck():
+                                return RNS.Identity.recall(destination_hash) != None
+                            spin(spincheck, "Requesting unknown Identity for "+RNS.prettyhexrep(destination_hash), args.t)
+
+                            if not spincheck():
+                                RNS.log("Identity request timed out", RNS.LOG_ERROR)
+                                exit(6)
+                            else:
+                                identity = RNS.Identity.recall(destination_hash)
+                                RNS.log("Received Identity "+str(identity)+" for destination "+RNS.prettyhexrep(destination_hash)+" from the network")
+
+                    else:
+                        RNS.log("Recalled Identity "+str(identity)+" for destination "+RNS.prettyhexrep(destination_hash))
+
+
+                except Exception as e:
+                    RNS.log("Invalid hexadecimal hash provided", RNS.LOG_ERROR)
+                    exit(7)
+
+                
+            else:
+                # Try loading Identity from file
+                if not os.path.isfile(identity_str):
+                    RNS.log("Specified Identity file not found")
+                    exit(8)
+                else:
+                    try:
+                        identity = RNS.Identity.from_file(identity_str)
+                        RNS.log("Loaded Identity "+str(identity)+" from "+str(identity_str))
+
+                    except Exception as e:
+                        RNS.log("Could not decode Identity from specified file")
+                        exit(9)
+
+            if identity != None:
+                if args.hash:
+                    try:
+                        aspects = args.hash.split(".")
+                        if not len(aspects) > 0:
+                            RNS.log("Invalid destination aspects specified", RNS.LOG_ERROR)
+                            exit(32)
+                        else:
+                            app_name = aspects[0]
+                            aspects = aspects[1:]
+                            if identity.pub != None:
+                                destination = RNS.Destination(identity, RNS.Destination.OUT, RNS.Destination.SINGLE, app_name, *aspects)
+                                RNS.log("The "+str(args.hash)+" destination for this Identity is "+RNS.prettyhexrep(destination.hash))
+                                RNS.log("The full destination specifier is "+str(destination))
+                                time.sleep(0.25)
+                                exit(0)
+                            else:
+                                raise KeyError("No public key known")
+                    except Exception as e:
+                        RNS.log("An error ocurred while attempting to send the announce.", RNS.LOG_ERROR)
+                        RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+                    exit(0)
+
+                if args.announce:
+                    try:
+                        aspects = args.announce.split(".")
+                        if not len(aspects) > 1:
+                            RNS.log("Invalid destination aspects specified", RNS.LOG_ERROR)
+                            exit(32)
+                        else:
+                            app_name = aspects[0]
+                            aspects = aspects[1:]
+                            if identity.prv != None:
+                                destination = RNS.Destination(identity, RNS.Destination.IN, RNS.Destination.SINGLE, app_name, *aspects)
+                                RNS.log("Created destination "+str(destination))
+                                RNS.log("Announcing destination "+RNS.prettyhexrep(destination.hash))
+                                destination.announce()
+                                time.sleep(0.25)
+                                exit(0)
+                            else:
+                                destination = RNS.Destination(identity, RNS.Destination.OUT, RNS.Destination.SINGLE, app_name, *aspects)
+                                RNS.log("The "+str(args.announce)+" destination for this Identity is "+RNS.prettyhexrep(destination.hash))
+                                RNS.log("The full destination specifier is "+str(destination))
+                                RNS.log("Cannot announce this destination, since the private key is not held")
+                                time.sleep(0.25)
+                                exit(33)
+                    except Exception as e:
+                        RNS.log("An error ocurred while attempting to send the announce.", RNS.LOG_ERROR)
+                        RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+                    exit(0)
+
+                if args.print_identity:
+                    if args.base64:
+                        RNS.log("Public Key  : "+base64.urlsafe_b64encode(identity.get_public_key()).decode("utf-8"))
+                    elif args.base32:
+                        RNS.log("Public Key  : "+base64.b32encode(identity.get_public_key()).decode("utf-8"))
+                    else:
+                        RNS.log("Public Key  : "+RNS.hexrep(identity.get_public_key(), delimit=False))
+                    if identity.prv:
+                        if args.print_private:
+                            if args.base64:
+                                RNS.log("Private Key : "+base64.urlsafe_b64encode(identity.get_private_key()).decode("utf-8"))
+                            elif args.base32:
+                                RNS.log("Private Key : "+base64.b32encode(identity.get_private_key()).decode("utf-8"))
+                            else:
+                                RNS.log("Private Key : "+RNS.hexrep(identity.get_private_key(), delimit=False))
+                        else:
+                            RNS.log("Private Key : Hidden")
+                    exit(0)
+
+                if args.export:
+                    if identity.prv:
+                            if args.base64:
+                                RNS.log("Exported Identity : "+base64.urlsafe_b64encode(identity.get_private_key()).decode("utf-8"))
+                            elif args.base32:
+                                RNS.log("Exported Identity : "+base64.b32encode(identity.get_private_key()).decode("utf-8"))
+                            else:
+                                RNS.log("Exported Identity : "+RNS.hexrep(identity.get_private_key(), delimit=False))
+                    else:
+                        RNS.log("Identity doesn't hold a private key, cannot export")
+                        exit(50)
+
+                    exit(0)
+
+                if args.validate:
+                    if not args.read and args.validate.lower().endswith("."+SIG_EXT):
+                        args.read = str(args.validate).replace("."+SIG_EXT, "")
+
+                    if not os.path.isfile(args.validate):
+                        RNS.log("Signature file "+str(args.read)+" not found", RNS.LOG_ERROR)
+                        exit(10)
+
+                    if not os.path.isfile(args.read):
+                        RNS.log("Input file "+str(args.read)+" not found", RNS.LOG_ERROR)
+                        exit(11)
+
+                data_input = None
+                if args.read:
+                    if not os.path.isfile(args.read):
+                        RNS.log("Input file "+str(args.read)+" not found", RNS.LOG_ERROR)
+                        exit(12)
+                    else:
+                        try:
+                            data_input = open(args.read, "rb")
+                        except Exception as e:
+                            RNS.log("Could not open input file for reading", RNS.LOG_ERROR)
+                            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            exit(13)
+
+                # TODO: Actually expand this to a good solution
+                # probably need to create a wrapper that takes
+                # into account not closing stdin when done
+                # elif args.stdin:
+                #     data_input = sys.stdin
+
+                data_output = None
+                if args.encrypt and not args.write and not args.stdout and args.read:
+                    args.write = str(args.read)+"."+ENCRYPT_EXT
+
+                if args.decrypt and not args.write and not args.stdout and args.read and args.read.lower().endswith("."+ENCRYPT_EXT):
+                    args.write = str(args.read).replace("."+ENCRYPT_EXT, "")
+
+                if args.sign and identity.prv == None:
+                    RNS.log("Specified Identity does not hold a private key. Cannot sign.", RNS.LOG_ERROR)
+                    exit(14)
+
+                if args.sign and not args.write and not args.stdout and args.read:
+                    args.write = str(args.read)+"."+SIG_EXT
+
+                if args.write:
+                    if not args.force and os.path.isfile(args.write):
+                        RNS.log("Output file "+str(args.write)+" already exists. Not overwriting.", RNS.LOG_ERROR)
+                        exit(15)
+                    else:
+                        try:
+                            data_output = open(args.write, "wb")
+                        except Exception as e:
+                            RNS.log("Could not open output file for writing", RNS.LOG_ERROR)
+                            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            exit(15)
+                
+                # TODO: Actually expand this to a good solution
+                # probably need to create a wrapper that takes
+                # into account not closing stdout when done
+                # elif args.stdout:
+                #     data_output = sys.stdout
+
+                if args.sign:
+                    if identity.prv == None:
+                        RNS.log("Specified Identity does not hold a private key. Cannot sign.", RNS.LOG_ERROR)
+                        exit(16)
+
+                    if not data_input:
+                        if not args.stdout:
+                            RNS.log("Signing requested, but no input data specified", RNS.LOG_ERROR)
+                        exit(17)
+                    else:
+                        if not data_output:
+                            if not args.stdout:
+                                RNS.log("Signing requested, but no output specified", RNS.LOG_ERROR)
+                            exit(18)
+
+                        if not args.stdout:
+                            RNS.log("Signing "+str(args.read))
+                        
+                        try:
+                            data_output.write(identity.sign(data_input.read()))
+                            data_output.close()
+                            data_input.close()
+                            
+                            if not args.stdout:
+                                if args.read:
+                                    RNS.log("File "+str(args.read)+" signed with "+str(identity)+" to "+str(args.write))
+                                    exit(0)
+
+                        except Exception as e:
+                            if not args.stdout:
+                                RNS.log("An error ocurred while encrypting data.", RNS.LOG_ERROR)
+                                RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            try:
+                                data_output.close()
+                            except:
+                                pass
+                            try:
+                                data_input.close()
+                            except:
+                                pass
+                            exit(19)
+
+                if args.validate:
+                    if not data_input:
+                        if not args.stdout:
+                            RNS.log("Signature verification requested, but no input data specified", RNS.LOG_ERROR)
+                        exit(20)
+                    else:
+                        # if not args.stdout:
+                        #     RNS.log("Verifying "+str(args.validate)+" for "+str(args.read))
+                        
+                        try:
+                            try:
+                                sig_input = open(args.validate, "rb")
+                            except Exception as e:
+                                RNS.log("An error ocurred while opening "+str(args.validate)+".", RNS.LOG_ERROR)
+                                RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                                exit(21)
+
+
+                            validated = identity.validate(sig_input.read(), data_input.read())
+                            sig_input.close()
+                            data_input.close()
+
+                            if not validated:
+                                if not args.stdout:
+                                    RNS.log("Signature "+str(args.validate)+" for file "+str(args.read)+" is invalid", RNS.LOG_ERROR)
+                                exit(22)
+                            else:
+                                if not args.stdout:
+                                    RNS.log("Signature "+str(args.validate)+" for file "+str(args.read)+" made by Identity "+str(identity)+" is valid")
+                                exit(0)
+
+                        except Exception as e:
+                            if not args.stdout:
+                                RNS.log("An error ocurred while validating signature.", RNS.LOG_ERROR)
+                                RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            try:
+                                data_output.close()
+                            except:
+                                pass
+                            try:
+                                data_input.close()
+                            except:
+                                pass
+                            exit(23)
+
+                if args.encrypt:
+                    if not data_input:
+                        if not args.stdout:
+                            RNS.log("Encryption requested, but no input data specified", RNS.LOG_ERROR)
+                        exit(24)
+                    else:
+                        if not data_output:
+                            if not args.stdout:
+                                RNS.log("Encryption requested, but no output specified", RNS.LOG_ERROR)
+                            exit(25)
+
+                        if not args.stdout:
+                            RNS.log("Encrypting "+str(args.read))
+                        
+                        try:
+                            more_data = True
+                            while more_data:
+                                chunk = data_input.read(CHUNK_SIZE)
+                                if chunk:
+                                    data_output.write(identity.encrypt(chunk))
+                                else:
+                                    more_data = False
+                            data_output.close()
+                            data_input.close()
+                            if not args.stdout:
+                                if args.read:
+                                    RNS.log("File "+str(args.read)+" encrypted for "+str(identity)+" to "+str(args.write))
+                                    exit(0)
+
+                        except Exception as e:
+                            if not args.stdout:
+                                RNS.log("An error ocurred while encrypting data.", RNS.LOG_ERROR)
+                                RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            try:
+                                data_output.close()
+                            except:
+                                pass
+                            try:
+                                data_input.close()
+                            except:
+                                pass
+                            exit(26)
+
+                if args.decrypt:
+                    if identity.prv == None:
+                        RNS.log("Specified Identity does not hold a private key. Cannot decrypt.", RNS.LOG_ERROR)
+                        exit(27)
+
+                    if not data_input:
+                        if not args.stdout:
+                            RNS.log("Decryption requested, but no input data specified", RNS.LOG_ERROR)
+                        exit(28)
+                    else:
+                        if not data_output:
+                            if not args.stdout:
+                                RNS.log("Decryption requested, but no output specified", RNS.LOG_ERROR)
+                            exit(29)
+
+                        if not args.stdout:
+                            RNS.log("Decrypting "+str(args.read)+"...")
+                        
+                        try:
+                            more_data = True
+                            while more_data:
+                                chunk = data_input.read(CHUNK_SIZE)
+                                if chunk:
+                                    plaintext = identity.decrypt(chunk)
+                                    if plaintext == None:
+                                        if not args.stdout:
+                                            RNS.log("Data could not be decrypted with the specified Identity")
+                                        exit(30)
+                                    else:
+                                        data_output.write(plaintext)
+                                else:
+                                    more_data = False
+                            data_output.close()
+                            data_input.close()
+                            if not args.stdout:
+                                if args.read:
+                                    RNS.log("File "+str(args.read)+" decrypted with "+str(identity)+" to "+str(args.write))
+                                    exit(0)
+
+                        except Exception as e:
+                            if not args.stdout:
+                                RNS.log("An error ocurred while decrypting data.", RNS.LOG_ERROR)
+                                RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            try:
+                                data_output.close()
+                            except:
+                                pass
+                            try:
+                                data_input.close()
+                            except:
+                                pass
+                            exit(31)
+
+            if True:
+                pass
+
+            elif False:
+                pass
+
+            else:
+                print("")
+                parser.print_help()
+                print("")
+
+    except KeyboardInterrupt:
+        print("")
+        exit(255)
+
+if __name__ == "__main__":
+    main()

RNS/Utilities/rnir.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+
+# MIT License
+#
+# Copyright (c) 2023 Mark Qvist / unsigned.io
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import RNS
+import argparse
+import time
+
+from RNS._version import __version__
+
+
+def program_setup(configdir, verbosity = 0, quietness = 0, service = False):
+    targetverbosity = verbosity-quietness
+
+    if service:
+        targetlogdest  = RNS.LOG_FILE
+        targetverbosity = None
+    else:
+        targetlogdest  = RNS.LOG_STDOUT
+
+    reticulum = RNS.Reticulum(configdir=configdir, verbosity=targetverbosity, logdest=targetlogdest)
+    exit(0)
+
+def main():
+    try:
+        parser = argparse.ArgumentParser(description="Reticulum Distributed Identity Resolver")
+        parser.add_argument("--config", action="store", default=None, help="path to alternative Reticulum config directory", type=str)
+        parser.add_argument('-v', '--verbose', action='count', default=0)
+        parser.add_argument('-q', '--quiet', action='count', default=0)
+        parser.add_argument("--exampleconfig", action='store_true', default=False, help="print verbose configuration example to stdout and exit")
+        parser.add_argument("--version", action="version", version="ir {version}".format(version=__version__))
+        
+        args = parser.parse_args()
+
+        if args.exampleconfig:
+            print(__example_rns_config__)
+            exit()
+
+        if args.config:
+            configarg = args.config
+        else:
+            configarg = None
+
+        program_setup(configdir = configarg, verbosity=args.verbose, quietness=args.quiet)
+
+    except KeyboardInterrupt:
+        print("")
+        exit()
+
+__example_rns_config__ = '''# This is an example Identity Resolver file.
+'''
+
+if __name__ == "__main__":
+    main()

RNS/Utilities/rnpath.py

@@ -23,14 +23,95 @@
 # SOFTWARE.
 
 import RNS
+import os
 import sys
 import time
 import argparse
 
 from RNS._version import __version__
 
+remote_link = None
+def connect_remote(destination_hash, auth_identity, timeout, no_output = False):
+    global remote_link, reticulum
+    if not RNS.Transport.has_path(destination_hash):
+        if not no_output:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested", end=" ")
+            sys.stdout.flush()
+        RNS.Transport.request_path(destination_hash)
+        pr_time = time.time()
+        while not RNS.Transport.has_path(destination_hash):
+            time.sleep(0.1)
+            if time.time() - pr_time > timeout:
+                if not no_output:
+                    print("\r                                                          \r", end="")
+                    print("Path request timed out")
+                    exit(12)
+
+    remote_identity = RNS.Identity.recall(destination_hash)
+
+    def remote_link_closed(link):
+        if link.teardown_reason == RNS.Link.TIMEOUT:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("The link timed out, exiting now")
+        elif link.teardown_reason == RNS.Link.DESTINATION_CLOSED:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("The link was closed by the server, exiting now")
+        else:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Link closed unexpectedly, exiting now")
+        exit(10)
+
+    def remote_link_established(link):
+        global remote_link
+        link.identify(auth_identity)
+        remote_link = link
+
+    if not no_output:
+        print("\r                                                          \r", end="")
+        print("Establishing link with remote transport instance...", end=" ")
+        sys.stdout.flush()
+
+    remote_destination = RNS.Destination(remote_identity, RNS.Destination.OUT, RNS.Destination.SINGLE, "rnstransport", "remote", "management")
+    link = RNS.Link(remote_destination)
+    link.set_link_established_callback(remote_link_established)
+    link.set_link_closed_callback(remote_link_closed)
+
+def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity, timeout, drop_queues,
+                  drop_via, max_hops, remote=None, management_identity=None, remote_timeout=RNS.Transport.PATH_REQUEST_TIMEOUT,
+                  no_output=False, json=False):
+    global remote_link, reticulum
+    reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
+    if remote:
+        try:
+            dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+            if len(remote) != dest_len:
+                raise ValueError("Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2))
+            try:
+                identity_hash = bytes.fromhex(remote)
+                remote_hash = RNS.Destination.hash_from_name_and_identity("rnstransport.remote.management", identity_hash)
+            except Exception as e:
+                raise ValueError("Invalid destination entered. Check your input.")
+
+            identity = RNS.Identity.from_file(os.path.expanduser(management_identity))
+            if identity == None:
+                raise ValueError("Could not load management identity from "+str(management_identity))
+
+            try:
+                connect_remote(remote_hash, identity, remote_timeout, no_output)
+            except Exception as e:
+                raise e
+
+        except Exception as e:
+            print(str(e))
+            exit(20)
+
+        while remote_link == None:
+            time.sleep(0.1)
+
 
-def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity, timeout, drop_queues):
     if table:
         destination_hash = None
         if destination_hexhash != None:
@@ -46,23 +127,50 @@ def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity,
                 print(str(e))
                 sys.exit(1)
 
-        reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
-        table = sorted(reticulum.get_path_table(), key=lambda e: (e["interface"], e["hops"]) )
+        if not remote_link:
+            table = sorted(reticulum.get_path_table(max_hops=max_hops), key=lambda e: (e["interface"], e["hops"]) )
+        else:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Sending request...", end=" ")
+                sys.stdout.flush()
+            receipt = remote_link.request("/path", data = ["table", destination_hash, max_hops])
+            while not receipt.concluded():
+                time.sleep(0.1)
+            response = receipt.get_response()
+            if response:
+                table = response
+                print("\r                                                          \r", end="")
+            else:
+                if not no_output:
+                    print("\r                                                          \r", end="")
+                    print("The remote request failed. Likely authentication failure.")
+                exit(10)
 
         displayed = 0
-        for path in table:
-            if destination_hash == None or destination_hash == path["hash"]:
-                displayed += 1
-                exp_str = RNS.timestamp_str(path["expires"])
-                if path["hops"] == 1:
-                    m_str = " "
-                else:
-                    m_str = "s"
-                print(RNS.prettyhexrep(path["hash"])+" is "+str(path["hops"])+" hop"+m_str+" away via "+RNS.prettyhexrep(path["via"])+" on "+path["interface"]+" expires "+RNS.timestamp_str(path["expires"]))
+        if json:
+            import json
+            for p in table:
+                for k in p:
+                    if isinstance(p[k], bytes):
+                        p[k] = RNS.hexrep(p[k], delimit=False)
 
-        if destination_hash != None and displayed == 0:
-            print("No path known")
-            sys.exit(1)
+            print(json.dumps(table))
+            exit()
+        else:
+            for path in table:
+                if destination_hash == None or destination_hash == path["hash"]:
+                    displayed += 1
+                    exp_str = RNS.timestamp_str(path["expires"])
+                    if path["hops"] == 1:
+                        m_str = " "
+                    else:
+                        m_str = "s"
+                    print(RNS.prettyhexrep(path["hash"])+" is "+str(path["hops"])+" hop"+m_str+" away via "+RNS.prettyhexrep(path["via"])+" on "+path["interface"]+" expires "+RNS.timestamp_str(path["expires"]))
+
+            if destination_hash != None and displayed == 0:
+                print("No path known")
+                sys.exit(1)
 
     elif rates:
         destination_hash = None
@@ -79,60 +187,99 @@ def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity,
                 print(str(e))
                 sys.exit(1)
 
-        reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
-        table = sorted(reticulum.get_rate_table(), key=lambda e: e["last"] )
-
-        if len(table) == 0:
-            print("No information available")
-
+        if not remote_link:
+            table = reticulum.get_rate_table()
         else:
-            displayed = 0
-            for entry in table:
-                if destination_hash == None or destination_hash == entry["hash"]:
-                    displayed += 1
-                    try:
-                        last_str = pretty_date(int(entry["last"]))
-                        start_ts = entry["timestamps"][0]
-                        span = max(time.time() - start_ts, 3600.0)
-                        span_hours = span/3600.0
-                        span_str = pretty_date(int(entry["timestamps"][0]))
-                        hour_rate = round(len(entry["timestamps"])/span_hours, 3)
-                        if hour_rate-int(hour_rate) == 0:
-                            hour_rate = int(hour_rate)
-                        
-                        if entry["rate_violations"] > 0:
-                            if entry["rate_violations"] == 1:
-                                s_str = ""
-                            else:
-                                s_str = "s"
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Sending request...", end=" ")
+                sys.stdout.flush()
+            receipt = remote_link.request("/path", data = ["rates", destination_hash])
+            while not receipt.concluded():
+                time.sleep(0.1)
+            response = receipt.get_response()
+            if response:
+                table = response
+                print("\r                                                          \r", end="")
+            else:
+                if not no_output:
+                    print("\r                                                          \r", end="")
+                    print("The remote request failed. Likely authentication failure.")
+                exit(10)
 
-                            rv_str = ", "+str(entry["rate_violations"])+" active rate violation"+s_str
-                        else:
-                            rv_str = ""
-                        
-                        if entry["blocked_until"] > time.time():
-                            bli = time.time()-(int(entry["blocked_until"])-time.time())
-                            bl_str = ", new announces allowed in "+pretty_date(int(bli))
-                        else:
-                            bl_str = ""
+        table = sorted(table, key=lambda e: e["last"])
+        if json:
+            import json
+            for p in table:
+                for k in p:
+                    if isinstance(p[k], bytes):
+                        p[k] = RNS.hexrep(p[k], delimit=False)
 
-        
-                        print(RNS.prettyhexrep(entry["hash"])+" last heard "+last_str+" ago, "+str(hour_rate)+" announces/hour in the last "+span_str+rv_str+bl_str)
-
-                    except Exception as e:
-                        print("Error while processing entry for "+RNS.prettyhexrep(entry["hash"]))
-                        print(str(e))
-
-            if destination_hash != None and displayed == 0:
+            print(json.dumps(table))
+            exit()
+        else:
+            if len(table) == 0:
                 print("No information available")
-                sys.exit(1)
+
+            else:
+                displayed = 0
+                for entry in table:
+                    if destination_hash == None or destination_hash == entry["hash"]:
+                        displayed += 1
+                        try:
+                            last_str = pretty_date(int(entry["last"]))
+                            start_ts = entry["timestamps"][0]
+                            span = max(time.time() - start_ts, 3600.0)
+                            span_hours = span/3600.0
+                            span_str = pretty_date(int(entry["timestamps"][0]))
+                            hour_rate = round(len(entry["timestamps"])/span_hours, 3)
+                            if hour_rate-int(hour_rate) == 0:
+                                hour_rate = int(hour_rate)
+                            
+                            if entry["rate_violations"] > 0:
+                                if entry["rate_violations"] == 1:
+                                    s_str = ""
+                                else:
+                                    s_str = "s"
+
+                                rv_str = ", "+str(entry["rate_violations"])+" active rate violation"+s_str
+                            else:
+                                rv_str = ""
+                            
+                            if entry["blocked_until"] > time.time():
+                                bli = time.time()-(int(entry["blocked_until"])-time.time())
+                                bl_str = ", new announces allowed in "+pretty_date(int(bli))
+                            else:
+                                bl_str = ""
+
+            
+                            print(RNS.prettyhexrep(entry["hash"])+" last heard "+last_str+" ago, "+str(hour_rate)+" announces/hour in the last "+span_str+rv_str+bl_str)
+
+                        except Exception as e:
+                            print("Error while processing entry for "+RNS.prettyhexrep(entry["hash"]))
+                            print(str(e))
+
+                if destination_hash != None and displayed == 0:
+                    print("No information available")
+                    sys.exit(1)
 
     elif drop_queues:
-        reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
-        RNS.log("Dropping announce queues on all interfaces...")
+        if remote_link:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Dropping announce queues on remote instances not yet implemented")
+            exit(255)
+
+        print("Dropping announce queues on all interfaces...")
         reticulum.drop_announce_queues()
     
     elif drop:
+        if remote_link:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Dropping path on remote instances not yet implemented")
+            exit(255)
+
         try:
             dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
             if len(destination_hexhash) != dest_len:
@@ -145,17 +292,44 @@ def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity,
             print(str(e))
             sys.exit(1)
 
-
-        reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
-
         if reticulum.drop_path(destination_hash):
             print("Dropped path to "+RNS.prettyhexrep(destination_hash))
         else:
             print("Unable to drop path to "+RNS.prettyhexrep(destination_hash)+". Does it exist?")
             sys.exit(1)
 
+    elif drop_via:
+        if remote_link:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Dropping all paths via specific transport instance on remote instances yet not implemented")
+            exit(255)
+
+        try:
+            dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+            if len(destination_hexhash) != dest_len:
+                raise ValueError("Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2))
+            try:
+                destination_hash = bytes.fromhex(destination_hexhash)
+            except Exception as e:
+                raise ValueError("Invalid destination entered. Check your input.")
+        except Exception as e:
+            print(str(e))
+            sys.exit(1)
+
+        if reticulum.drop_all_via(destination_hash):
+            print("Dropped all paths via "+RNS.prettyhexrep(destination_hash))
+        else:
+            print("Unable to drop paths via "+RNS.prettyhexrep(destination_hash)+". Does the transport instance exist?")
+            sys.exit(1)
 
     else:
+        if remote_link:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Requesting paths on remote instances not implemented")
+            exit(255)
+
         try:
             dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
             if len(destination_hexhash) != dest_len:
@@ -168,9 +342,6 @@ def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity,
             print(str(e))
             sys.exit(1)
             
-
-        reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
-
         if not RNS.Transport.has_path(destination_hash):
             RNS.Transport.request_path(destination_hash)
             print("Path to "+RNS.prettyhexrep(destination_hash)+" requested  ", end=" ")
@@ -187,15 +358,20 @@ def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity,
 
         if RNS.Transport.has_path(destination_hash):
             hops = RNS.Transport.hops_to(destination_hash)
-            next_hop = RNS.prettyhexrep(reticulum.get_next_hop(destination_hash))
-            next_hop_interface = reticulum.get_next_hop_if_name(destination_hash)
-
-            if hops != 1:
-                ms = "s"
+            next_hop_bytes = reticulum.get_next_hop(destination_hash)
+            if next_hop_bytes == None:
+                print("\r                                                       \rError: Invalid path data returned")
+                sys.exit(1)
             else:
-                ms = ""
+                next_hop = RNS.prettyhexrep(next_hop_bytes)
+                next_hop_interface = reticulum.get_next_hop_if_name(destination_hash)
 
-            print("\rPath found, destination "+RNS.prettyhexrep(destination_hash)+" is "+str(hops)+" hop"+ms+" away via "+next_hop+" on "+next_hop_interface)
+                if hops != 1:
+                    ms = "s"
+                else:
+                    ms = ""
+
+                print("\rPath found, destination "+RNS.prettyhexrep(destination_hash)+" is "+str(hops)+" hop"+ms+" away via "+next_hop+" on "+next_hop_interface)
         else:
             print("\r                                                       \rPath not found")
             sys.exit(1)
@@ -227,6 +403,16 @@ def main():
             default=False
         )
 
+        parser.add_argument(
+            "-m",
+            "--max",
+            action="store",
+            metavar="hops",
+            type=int,
+            help="maximum hops to filter path table by",
+            default=None
+        )
+
         parser.add_argument(
             "-r",
             "--rates",
@@ -251,6 +437,13 @@ def main():
             default=False
         )
 
+        parser.add_argument(
+            "-x", "--drop-via",
+            action="store_true",
+            help="drop all paths via specified transport instance",
+            default=False
+        )
+
         parser.add_argument(
             "-w",
             action="store",
@@ -260,6 +453,41 @@ def main():
             default=RNS.Transport.PATH_REQUEST_TIMEOUT
         )
 
+        parser.add_argument(
+            "-R",
+            action="store",
+            metavar="hash",
+            help="transport identity hash of remote instance to manage",
+            default=None,
+            type=str
+        )
+
+        parser.add_argument(
+            "-i",
+            action="store",
+            metavar="path",
+            help="path to identity used for remote management",
+            default=None,
+            type=str
+        )
+
+        parser.add_argument(
+            "-W",
+            action="store",
+            metavar="seconds",
+            type=float,
+            help="timeout before giving up on remote queries",
+            default=RNS.Transport.PATH_REQUEST_TIMEOUT
+        )
+        
+        parser.add_argument(
+            "-j",
+            "--json",
+            action="store_true",
+            help="output in JSON format",
+            default=False
+        )
+
         parser.add_argument(
             "destination",
             nargs="?",
@@ -277,7 +505,7 @@ def main():
         else:
             configarg = None
 
-        if not args.drop_announces and not args.table and not args.rates and not args.destination:
+        if not args.drop_announces and not args.table and not args.rates and not args.destination and not args.drop_via:
             print("")
             parser.print_help()
             print("")
@@ -291,6 +519,12 @@ def main():
                 verbosity = args.verbose,
                 timeout = args.w,
                 drop_queues = args.drop_announces,
+                drop_via = args.drop_via,
+                max_hops = args.max,
+                remote=args.R,
+                management_identity=args.i,
+                remote_timeout=args.W,
+                json=args.json,
             )
             sys.exit(0)
 

RNS/Utilities/rnprobe.py

@@ -31,8 +31,10 @@ import argparse
 from RNS._version import __version__
 
 DEFAULT_PROBE_SIZE = 16
+DEFAULT_TIMEOUT = 12
 
-def program_setup(configdir, destination_hexhash, size=DEFAULT_PROBE_SIZE, full_name = None, verbosity = 0):
+def program_setup(configdir, destination_hexhash, size=None, full_name = None, verbosity = 0, timeout=None, wait=0, probes=1):
+    if size == None: size = DEFAULT_PROBE_SIZE
     if full_name == None:
         print("The full destination name including application name aspects must be specified for the destination")
         exit()
@@ -71,14 +73,19 @@ def program_setup(configdir, destination_hexhash, size=DEFAULT_PROBE_SIZE, full_
         print("Path to "+RNS.prettyhexrep(destination_hash)+" requested  ", end=" ")
         sys.stdout.flush()
 
+    _timeout = time.time() + (timeout or DEFAULT_TIMEOUT+reticulum.get_first_hop_timeout(destination_hash))
     i = 0
     syms = "⢄⢂⢁⡁⡈⡐⡠"
-    while not RNS.Transport.has_path(destination_hash):
+    while not RNS.Transport.has_path(destination_hash) and not time.time() > _timeout:
         time.sleep(0.1)
         print(("\b\b"+syms[i]+" "), end="")
         sys.stdout.flush()
         i = (i+1)%len(syms)
 
+    if time.time() > _timeout:
+        print("\r                                                          \rPath request timed out")
+        exit(1)
+
     server_identity = RNS.Identity.recall(destination_hash)
 
     request_destination = RNS.Destination(
@@ -89,101 +96,120 @@ def program_setup(configdir, destination_hexhash, size=DEFAULT_PROBE_SIZE, full_
         *aspects
     )
 
-    probe = RNS.Packet(request_destination, os.urandom(size))
-    receipt = probe.send()
+    sent = 0
+    replies = 0
+    while probes:
 
-    if more_output:
-        more = " via "+RNS.prettyhexrep(reticulum.get_next_hop(destination_hash))+" on "+str(reticulum.get_next_hop_if_name(destination_hash))
-    else:
-        more = ""
+        if sent > 0:
+            time.sleep(wait)
 
-    print("\rSent "+str(size)+" byte probe to "+RNS.prettyhexrep(destination_hash)+more+"  ", end=" ")
+        try:
+            probe = RNS.Packet(request_destination, os.urandom(size))
+            probe.pack()
+        except OSError:
+            print("Error: Probe packet size of "+str(len(probe.raw))+" bytes exceed MTU of "+str(RNS.Reticulum.MTU)+" bytes")
+            exit(3)
 
-    i = 0
-    while not receipt.status == RNS.PacketReceipt.DELIVERED:
-        time.sleep(0.1)
-        print(("\b\b"+syms[i]+" "), end="")
-        sys.stdout.flush()
-        i = (i+1)%len(syms)
+        receipt = probe.send()
+        sent += 1
 
-    print("\b\b ")
-    sys.stdout.flush()
+        if more_output:
+            nhd = reticulum.get_next_hop(destination_hash)
+            via_str = " via "+RNS.prettyhexrep(nhd) if nhd != None else ""
+            if_str  = " on "+str(reticulum.get_next_hop_if_name(destination_hash)) if reticulum.get_next_hop_if_name(destination_hash) != "None" else ""
+            more = via_str+if_str
+        else:
+            more = ""
 
-    hops = RNS.Transport.hops_to(destination_hash)
-    if hops != 1:
-        ms = "s"
-    else:
-        ms = ""
+        print("\rSent probe "+str(sent)+" ("+str(size)+" bytes) to "+RNS.prettyhexrep(destination_hash)+more+"  ", end=" ")
 
-    rtt = receipt.get_rtt()
-    if (rtt >= 1):
-        rtt = round(rtt, 3)
-        rttstring = str(rtt)+" seconds"
-    else:
-        rtt = round(rtt*1000, 3)
-        rttstring = str(rtt)+" milliseconds"
+        _timeout = time.time() + (timeout or DEFAULT_TIMEOUT+reticulum.get_first_hop_timeout(destination_hash))
+        i = 0
+        while receipt.status == RNS.PacketReceipt.SENT and not time.time() > _timeout:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
 
-    reception_stats = ""
-    if reticulum.is_connected_to_shared_instance:
-        reception_rssi = reticulum.get_packet_rssi(receipt.proof_packet.packet_hash)
-        reception_snr  = reticulum.get_packet_snr(receipt.proof_packet.packet_hash)
-
-        if reception_rssi != None:
-            reception_stats += " [RSSI "+str(reception_rssi)+" dBm]"
+        if time.time() > _timeout:
+            print("\r                                                                \rProbe timed out")
         
-        if reception_snr != None:
-            reception_stats += " [SNR "+str(reception_snr)+" dB]"
+        else:
+            print("\b\b ")
+            sys.stdout.flush()
 
+            if receipt.status == RNS.PacketReceipt.DELIVERED:
+                replies += 1
+                hops = RNS.Transport.hops_to(destination_hash)
+                if hops != 1:
+                    ms = "s"
+                else:
+                    ms = ""
+
+                rtt = receipt.get_rtt()
+                if (rtt >= 1):
+                    rtt = round(rtt, 3)
+                    rttstring = str(rtt)+" seconds"
+                else:
+                    rtt = round(rtt*1000, 3)
+                    rttstring = str(rtt)+" milliseconds"
+
+                reception_stats = ""
+                if reticulum.is_connected_to_shared_instance:
+                    reception_rssi = reticulum.get_packet_rssi(receipt.proof_packet.packet_hash)
+                    reception_snr  = reticulum.get_packet_snr(receipt.proof_packet.packet_hash)
+                    reception_q    = reticulum.get_packet_q(receipt.proof_packet.packet_hash)
+
+                    if reception_rssi != None:
+                        reception_stats += " [RSSI "+str(reception_rssi)+" dBm]"
+                    
+                    if reception_snr != None:
+                        reception_stats += " [SNR "+str(reception_snr)+" dB]"
+                    
+                    if reception_q != None:
+                        reception_stats += " [Link Quality "+str(reception_q)+"%]"
+
+                else:
+                    if receipt.proof_packet != None:
+                        if receipt.proof_packet.rssi != None:
+                            reception_stats += " [RSSI "+str(receipt.proof_packet.rssi)+" dBm]"
+                        
+                        if receipt.proof_packet.snr != None:
+                            reception_stats += " [SNR "+str(receipt.proof_packet.snr)+" dB]"
+
+                print(
+                    "Valid reply from "+
+                    RNS.prettyhexrep(receipt.destination.hash)+
+                    "\nRound-trip time is "+rttstring+
+                    " over "+str(hops)+" hop"+ms+
+                    reception_stats+"\n"
+                )
+
+            else:
+                print("\r                                                          \rProbe timed out")
+
+        probes -= 1
+
+    loss = round((1-(replies/sent))*100, 2)
+    print(f"Sent {sent}, received {replies}, packet loss {loss}%")
+    if loss > 0:
+        exit(2)
     else:
-        if receipt.proof_packet != None:
-            if receipt.proof_packet.rssi != None:
-                reception_stats += " [RSSI "+str(receipt.proof_packet.rssi)+" dBm]"
-            
-            if receipt.proof_packet.snr != None:
-                reception_stats += " [SNR "+str(receipt.proof_packet.snr)+" dB]"
-
-    print(
-        "Valid reply received from "+
-        RNS.prettyhexrep(receipt.destination.hash)+
-        "\nRound-trip time is "+rttstring+
-        " over "+str(hops)+" hop"+ms+
-        reception_stats
-    )
-
+        exit(0)
     
 
 def main():
     try:
         parser = argparse.ArgumentParser(description="Reticulum Probe Utility")
 
-        parser.add_argument("--config",
-            action="store",
-            default=None,
-            help="path to alternative Reticulum config directory",
-            type=str
-        )
-
-        parser.add_argument(
-            "--version",
-            action="version",
-            version="rnprobe {version}".format(version=__version__)
-        )
-
-        parser.add_argument(
-            "full_name",
-            nargs="?",
-            default=None,
-            help="full destination name in dotted notation",
-            type=str
-        )
-
-        parser.add_argument(
-            "destination_hash",
-            nargs="?",
-            default=None,
-            help="hexadecimal hash of the destination",
-            type=str
-        )
+        parser.add_argument("--config", action="store", default=None, help="path to alternative Reticulum config directory", type=str)
+        parser.add_argument("-s", "--size", action="store", default=None, help="size of probe packet payload in bytes", type=int)
+        parser.add_argument("-n", "--probes", action="store", default=1, help="number of probes to send", type=int)
+        parser.add_argument("-t", "--timeout", metavar="seconds", action="store", default=None, help="timeout before giving up", type=float)
+        parser.add_argument("-w", "--wait", metavar="seconds", action="store", default=0, help="time between each probe", type=float)
+        parser.add_argument("--version", action="version", version="rnprobe {version}".format(version=__version__))
+        parser.add_argument("full_name", nargs="?", default=None, help="full destination name in dotted notation", type=str)
+        parser.add_argument("destination_hash", nargs="?", default=None, help="hexadecimal hash of the destination", type=str)
 
         parser.add_argument('-v', '--verbose', action='count', default=0)
 
@@ -202,8 +228,12 @@ def main():
             program_setup(
                 configdir = configarg,
                 destination_hexhash = args.destination_hash,
+                size = args.size,
                 full_name = args.full_name,
-                verbosity = args.verbose
+                verbosity = args.verbose,
+                probes = args.probes,
+                wait = args.wait,
+                timeout = args.timeout,
             )
 
     except KeyboardInterrupt:

RNS/Utilities/rnsd.py

@@ -30,18 +30,20 @@ from RNS._version import __version__
 
 
 def program_setup(configdir, verbosity = 0, quietness = 0, service = False):
-    targetloglevel = 3+verbosity-quietness
+    targetverbosity = verbosity-quietness
 
     if service:
         targetlogdest  = RNS.LOG_FILE
-        targetloglevel = None
+        targetverbosity = None
     else:
         targetlogdest  = RNS.LOG_STDOUT
 
-    reticulum = RNS.Reticulum(configdir=configdir, loglevel=targetloglevel, logdest=targetlogdest)
+    reticulum = RNS.Reticulum(configdir=configdir, verbosity=targetverbosity, logdest=targetlogdest)
     if reticulum.is_connected_to_shared_instance:
         RNS.log("Started rnsd version {version} connected to another shared local instance, this is probably NOT what you want!".format(version=__version__), RNS.LOG_WARNING)
     else:
+        if RNS.Reticulum.get_instance().shared_instance_interface:
+            RNS.Reticulum.get_instance().shared_instance_interface.server.daemon_threads = True
         RNS.log("Started rnsd version {version}".format(version=__version__), RNS.LOG_NOTICE)
 
     while True:
@@ -87,7 +89,7 @@ __example_rns_config__ = '''# This is an example Reticulum config file.
 # always-on. This directive is optional and can be removed
 # for brevity.
 
-enable_transport = False
+enable_transport = No
 
 
 # By default, the first program to launch the Reticulum
@@ -111,6 +113,30 @@ share_instance = Yes
 shared_instance_port = 37428
 instance_control_port = 37429
 
+
+# On systems where running instances may not have access
+# to the same shared Reticulum configuration directory,
+# it is still possible to allow full interactivity for
+# running instances, by manually specifying a shared RPC
+# key. In almost all cases, this option is not needed, but
+# it can be useful on operating systems such as Android.
+# The key must be specified as bytes in hexadecimal.
+
+# rpc_key = e5c032d3ec4e64a6aca9927ba8ab73336780f6d71790
+
+
+# It is possible to allow remote management of Reticulum
+# systems using the various built-in utilities, such as
+# rnstatus and rnpath. You will need to specify one or
+# more Reticulum Identity hashes for authenticating the
+# queries from client programs. For this purpose, you can
+# use existing identity files, or generate new ones with
+# the rnid utility.
+
+# enable_remote_management = yes
+# remote_management_allowed = 9fb6d773498fb3feda407ed8ef2c3229, 2d882c5586e548d79b5af27bca1776dc
+
+
 # You can configure Reticulum to panic and forcibly close
 # if an unrecoverable interface error occurs, such as the
 # hardware device for an interface disappearing. This is
@@ -120,6 +146,17 @@ instance_control_port = 37429
 panic_on_interface_error = No
 
 
+# When Transport is enabled, it is possible to allow the
+# Transport Instance to respond to probe requests from
+# the rnprobe utility. This can be a useful tool to test
+# connectivity. When this option is enabled, the probe
+# destination will be generated from the Identity of the
+# Transport Instance, and printed to the log at startup.
+# Optional, and disabled by default.
+
+respond_to_probes = No
+
+
 [logging]
 # Valid log levels are 0 through 7:
 #   0: Log only critical information
@@ -259,6 +296,23 @@ loglevel = 4
     # Serial port for the device
     port = /dev/ttyUSB0
 
+    # It is also possible to use BLE devices
+    # instead of wired serial ports. The
+    # target RNode must be paired with the
+    # host device before connecting. BLE
+    # devices can be connected by name,
+    # BLE MAC address or by any available.
+    
+    # Connect to specific device by name
+    # port = ble://RNode 3B87
+
+    # Or by BLE MAC address
+    # port = ble://F4:12:73:29:4E:89
+
+    # Or connect to the first available,
+    # paired device
+    # port = ble://
+
     # Set frequency to 867.2 MHz
     frequency = 867200000
 

RNS/Utilities/rnstatus.py

@@ -23,6 +23,9 @@
 # SOFTWARE.
 
 import RNS
+import os
+import sys
+import time
 import argparse
 
 from RNS._version import __version__
@@ -46,17 +49,202 @@ def size_str(num, suffix='B'):
 
     return "%.2f%s%s" % (num, last_unit, suffix)
 
-def program_setup(configdir, dispall=False, verbosity=0, name_filter=None):
-    reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
+request_result = None
+request_concluded = False
+def get_remote_status(destination_hash, include_lstats, identity, no_output=False, timeout=RNS.Transport.PATH_REQUEST_TIMEOUT):
+    global request_result, request_concluded
+    link_count = None
+
+    if not RNS.Transport.has_path(destination_hash):
+        if not no_output:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested", end=" ")
+            sys.stdout.flush()
+        RNS.Transport.request_path(destination_hash)
+        pr_time = time.time()
+        while not RNS.Transport.has_path(destination_hash):
+            time.sleep(0.1)
+            if time.time() - pr_time > timeout:
+                if not no_output:
+                    print("\r                                                          \r", end="")
+                    print("Path request timed out")
+                    exit(12)
+
+    remote_identity = RNS.Identity.recall(destination_hash)
+
+    def remote_link_closed(link):
+        if link.teardown_reason == RNS.Link.TIMEOUT:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("The link timed out, exiting now")
+        elif link.teardown_reason == RNS.Link.DESTINATION_CLOSED:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("The link was closed by the server, exiting now")
+        else:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Link closed unexpectedly, exiting now")
+        exit(10)
+
+    def request_failed(request_receipt):
+        global request_result, request_concluded
+        if not no_output:
+            print("\r                                                          \r", end="")
+            print("The remote status request failed. Likely authentication failure.")
+        request_concluded = True
+
+    def got_response(request_receipt):
+        global request_result, request_concluded
+        response = request_receipt.response
+        if isinstance(response, list):
+            status = response[0]
+            if len(response) > 1:
+                link_count = response[1]
+            else:
+                link_count = None
+
+            request_result = (status, link_count)
+
+        request_concluded = True
+
+    def remote_link_established(link):
+        if not no_output:
+            print("\r                                                          \r", end="")
+            print("Sending request...", end=" ")
+            sys.stdout.flush()
+        link.identify(identity)
+        link.request("/status", data = [include_lstats], response_callback = got_response, failed_callback = request_failed)
+
+    if not no_output:
+        print("\r                                                          \r", end="")
+        print("Establishing link with remote transport instance...", end=" ")
+        sys.stdout.flush()
+
+    remote_destination = RNS.Destination(remote_identity, RNS.Destination.OUT, RNS.Destination.SINGLE, "rnstransport", "remote", "management")
+    link = RNS.Link(remote_destination)
+    link.set_link_established_callback(remote_link_established)
+    link.set_link_closed_callback(remote_link_closed)
+
+    while not request_concluded:
+        time.sleep(0.1)
+
+    if request_result != None:
+        print("\r                                                          \r", end="")
+
+    return request_result
+
+def program_setup(configdir, dispall=False, verbosity=0, name_filter=None, json=False, astats=False,
+                  lstats=False, sorting=None, sort_reverse=False, remote=None, management_identity=None,
+                  remote_timeout=RNS.Transport.PATH_REQUEST_TIMEOUT, must_exit=True, rns_instance=None, traffic_totals=False):
+    
+    if remote:
+        require_shared = False
+    else:
+        require_shared = True
 
-    stats = None
     try:
-        stats = reticulum.get_interface_stats()
+        if rns_instance:
+            reticulum = rns_instance
+            must_exit = False
+        else:
+            reticulum = RNS.Reticulum(configdir=configdir, loglevel=3+verbosity, require_shared_instance=require_shared)
+
     except Exception as e:
-        pass
+        print("No shared RNS instance available to get status from")
+        if must_exit:
+            exit(1)
+        else:
+            return
+
+    link_count = None
+    stats = None
+    if remote:
+        try:
+            dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+            if len(remote) != dest_len:
+                raise ValueError("Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2))
+            try:
+                identity_hash = bytes.fromhex(remote)
+                destination_hash = RNS.Destination.hash_from_name_and_identity("rnstransport.remote.management", identity_hash)
+            except Exception as e:
+                raise ValueError("Invalid destination entered. Check your input.")
+
+            identity = RNS.Identity.from_file(os.path.expanduser(management_identity))
+            if identity == None:
+                raise ValueError("Could not load management identity from "+str(management_identity))
+
+            try:
+                remote_status = get_remote_status(destination_hash, lstats, identity, no_output=json, timeout=remote_timeout)
+                if remote_status != None:
+                    stats, link_count = remote_status
+            except Exception as e:
+                raise e
+                    
+        except Exception as e:
+            print(str(e))
+            if must_exit:
+                exit(20)
+            else:
+                return
+
+    else:
+        if lstats:
+            try:
+                link_count = reticulum.get_link_count()
+            except Exception as e:
+                pass
+
+        try:
+            stats = reticulum.get_interface_stats()
+        except Exception as e:
+            pass
 
     if stats != None:
-        for ifstat in stats["interfaces"]:
+        if json:
+            import json
+            for s in stats:
+                if isinstance(stats[s], bytes):
+                    stats[s] = RNS.hexrep(stats[s], delimit=False)
+
+                if isinstance(stats[s], dict) or isinstance(stats[s], list):
+                    for i in stats[s]:
+                        if isinstance(i, dict):
+                            for k in i:
+                                if isinstance(i[k], bytes):
+                                    i[k] = RNS.hexrep(i[k], delimit=False)
+
+            print(json.dumps(stats))
+            if must_exit:
+                exit()
+            else:
+                return
+
+        interfaces = stats["interfaces"]
+        if sorting != None and isinstance(sorting, str):
+            sorting = sorting.lower()
+            if sorting == "rate" or sorting == "bitrate":
+                interfaces.sort(key=lambda i: i["bitrate"], reverse=not sort_reverse)
+            if sorting == "rx":
+                interfaces.sort(key=lambda i: i["rxb"], reverse=not sort_reverse)
+            if sorting == "tx":
+                interfaces.sort(key=lambda i: i["txb"], reverse=not sort_reverse)
+            if sorting == "rxs":
+                interfaces.sort(key=lambda i: i["rxs"], reverse=not sort_reverse)
+            if sorting == "txs":
+                interfaces.sort(key=lambda i: i["txs"], reverse=not sort_reverse)
+            if sorting == "traffic":
+                interfaces.sort(key=lambda i: i["rxb"]+i["txb"], reverse=not sort_reverse)
+            if sorting == "announces" or sorting == "announce":
+                interfaces.sort(key=lambda i: i["incoming_announce_frequency"]+i["outgoing_announce_frequency"], reverse=not sort_reverse)
+            if sorting == "arx":
+                interfaces.sort(key=lambda i: i["incoming_announce_frequency"], reverse=not sort_reverse)
+            if sorting == "atx":
+                interfaces.sort(key=lambda i: i["outgoing_announce_frequency"], reverse=not sort_reverse)
+            if sorting == "held":
+                interfaces.sort(key=lambda i: i["held_announces"], reverse=not sort_reverse)
+
+            
+        for ifstat in interfaces:
             name = ifstat["name"]
 
             if dispall or not (
@@ -98,7 +286,7 @@ def program_setup(configdir, dispall=False, verbosity=0, name_filter=None):
                                 else:
                                     spec_str = " programs"
 
-                                clients_string = "Serving : "+str(cnum)+spec_str
+                                clients_string = "Serving   : "+str(cnum)+spec_str
                             elif name.startswith("I2PInterface["):
                                 if "i2p_connectable" in ifstat and ifstat["i2p_connectable"] == True:
                                     cnum = clients
@@ -107,11 +295,11 @@ def program_setup(configdir, dispall=False, verbosity=0, name_filter=None):
                                     else:
                                         spec_str = " connected I2P endpoints"
 
-                                    clients_string = "Peers   : "+str(cnum)+spec_str
+                                    clients_string = "Peers     : "+str(cnum)+spec_str
                                 else:
                                     clients_string = ""
                             else:
-                                clients_string = "Clients : "+str(clients)
+                                clients_string = "Clients   : "+str(clients)
 
                         else:
                             clients = None
@@ -119,47 +307,130 @@ def program_setup(configdir, dispall=False, verbosity=0, name_filter=None):
                         print(" {n}".format(n=ifstat["name"]))
 
                         if "ifac_netname" in ifstat and ifstat["ifac_netname"] != None:
-                            print("    Network : {nn}".format(nn=ifstat["ifac_netname"]))
+                            print("    Network   : {nn}".format(nn=ifstat["ifac_netname"]))
 
-                        print("    Status  : {ss}".format(ss=ss))
+                        print("    Status    : {ss}".format(ss=ss))
 
                         if clients != None and clients_string != "":
                             print("    "+clients_string)
 
                         if not (name.startswith("Shared Instance[") or name.startswith("TCPInterface[Client") or name.startswith("LocalInterface[")):
-                            print("    Mode    : {mode}".format(mode=modestr))
+                            print("    Mode      : {mode}".format(mode=modestr))
 
                         if "bitrate" in ifstat and ifstat["bitrate"] != None:
-                            print("    Rate    : {ss}".format(ss=speed_str(ifstat["bitrate"])))
+                            print("    Rate      : {ss}".format(ss=speed_str(ifstat["bitrate"])))
+
+                        if "noise_floor" in ifstat:
+                            if ifstat["noise_floor"] != None:
+                                print("    Noise Fl. : {nfl} dBm".format(nfl=str(ifstat["noise_floor"])))
+                            else:
+                                print("    Noise Fl. : Unknown")
+
+                        if "battery_percent" in ifstat and ifstat["battery_percent"] != None:
+                            try:
+                                bpi = int(ifstat["battery_percent"])
+                                bss = ifstat["battery_state"]
+                                print(f"    Battery   : {bpi}% ({bss})")
+                            except:
+                                pass
+
+                        if "airtime_short" in ifstat and "airtime_long" in ifstat:
+                            print("    Airtime   : {ats}% (15s), {atl}% (1h)".format(ats=str(ifstat["airtime_short"]),atl=str(ifstat["airtime_long"])))
                         
+                        if "channel_load_short" in ifstat and "channel_load_long" in ifstat:
+                            print("    Ch. Load  : {ats}% (15s), {atl}% (1h)".format(ats=str(ifstat["channel_load_short"]),atl=str(ifstat["channel_load_long"])))
+
                         if "peers" in ifstat and ifstat["peers"] != None:
-                            print("    Peers   : {np} reachable".format(np=ifstat["peers"]))
+                            print("    Peers     : {np} reachable".format(np=ifstat["peers"]))
+
+                        if "tunnelstate" in ifstat and ifstat["tunnelstate"] != None:
+                            print("    I2P       : {ts}".format(ts=ifstat["tunnelstate"]))
 
                         if "ifac_signature" in ifstat and ifstat["ifac_signature"] != None:
                             sigstr = "<…"+RNS.hexrep(ifstat["ifac_signature"][-5:], delimit=False)+">"
-                            print("    Access  : {nb}-bit IFAC by {sig}".format(nb=ifstat["ifac_size"]*8, sig=sigstr))
+                            print("    Access    : {nb}-bit IFAC by {sig}".format(nb=ifstat["ifac_size"]*8, sig=sigstr))
                         
                         if "i2p_b32" in ifstat and ifstat["i2p_b32"] != None:
-                            print("    I2P B32 : {ep}".format(ep=str(ifstat["i2p_b32"])))
+                            print("    I2P B32   : {ep}".format(ep=str(ifstat["i2p_b32"])))
 
-                        if "announce_queue" in ifstat and ifstat["announce_queue"] != None and ifstat["announce_queue"] > 0:
+                        if astats and "announce_queue" in ifstat and ifstat["announce_queue"] != None and ifstat["announce_queue"] > 0:
                             aqn = ifstat["announce_queue"]
                             if aqn == 1:
-                                print("    Queued  : {np} announce".format(np=aqn))
+                                print("    Queued    : {np} announce".format(np=aqn))
                             else:
-                                print("    Queued  : {np} announces".format(np=aqn))
+                                print("    Queued    : {np} announces".format(np=aqn))
                         
-                        print("    Traffic : {txb}↑\n              {rxb}↓".format(rxb=size_str(ifstat["rxb"]), txb=size_str(ifstat["txb"])))
+                        if astats and "held_announces" in ifstat and ifstat["held_announces"] != None and ifstat["held_announces"] > 0:
+                            aqn = ifstat["held_announces"]
+                            if aqn == 1:
+                                print("    Held      : {np} announce".format(np=aqn))
+                            else:
+                                print("    Held      : {np} announces".format(np=aqn))
+                        
+                        if astats and "incoming_announce_frequency" in ifstat and ifstat["incoming_announce_frequency"] != None:
+                            print("    Announces : {iaf}↑".format(iaf=RNS.prettyfrequency(ifstat["outgoing_announce_frequency"])))
+                            print("                {iaf}↓".format(iaf=RNS.prettyfrequency(ifstat["incoming_announce_frequency"])))
+
+                        rxb_str = "↓"+RNS.prettysize(ifstat["rxb"])
+                        txb_str = "↑"+RNS.prettysize(ifstat["txb"])
+                        strdiff = len(rxb_str)-len(txb_str)
+                        if strdiff > 0:
+                            txb_str += " "*strdiff
+                        elif strdiff < 0:
+                            rxb_str += " "*-strdiff
+
+                        rxstat = rxb_str
+                        txstat = txb_str
+                        if "rxs" in ifstat and "txs" in ifstat:
+                            rxstat += "  "+RNS.prettyspeed(ifstat["rxs"])
+                            txstat += "  "+RNS.prettyspeed(ifstat["txs"])
+                        
+                        print(f"    Traffic   : {txstat}\n                {rxstat}")
+
+        lstr = ""
+        if link_count != None and lstats:
+            ms = "y" if link_count == 1 else "ies"
+            if "transport_id" in stats and stats["transport_id"] != None:
+                lstr = f", {link_count} entr{ms} in link table"
+            else:
+                lstr = f" {link_count} entr{ms} in link table"
+
+        if traffic_totals:
+            rxb_str = "↓"+RNS.prettysize(stats["rxb"])
+            txb_str = "↑"+RNS.prettysize(stats["txb"])
+            strdiff = len(rxb_str)-len(txb_str)
+            if strdiff > 0:
+                txb_str += " "*strdiff
+            elif strdiff < 0:
+                rxb_str += " "*-strdiff
+
+            rxstat  = rxb_str+"  "+RNS.prettyspeed(stats["rxs"])
+            txstat  = txb_str+"  "+RNS.prettyspeed(stats["txs"])
+            print(f"\n Totals       : {txstat}\n                {rxstat}")
 
         if "transport_id" in stats and stats["transport_id"] != None:
-            print("\n Reticulum Transport Instance "+RNS.prettyhexrep(stats["transport_id"])+" is running")
+            print("\n Transport Instance "+RNS.prettyhexrep(stats["transport_id"])+" running")
+            if "probe_responder" in stats and stats["probe_responder"] != None:
+                print(" Probe responder at "+RNS.prettyhexrep(stats["probe_responder"])+ " active")
+            if "transport_uptime" in stats and stats["transport_uptime"] != None:
+                print(" Uptime is "+RNS.prettytime(stats["transport_uptime"])+lstr)
+        else:
+            if lstr != "":
+                print(f"\n{lstr}")
 
         print("")
                 
     else:
-        print("Could not get RNS status")
+        if not remote:
+            print("Could not get RNS status")
+        else:
+            print("Could not get RNS status from remote transport instance "+RNS.prettyhexrep(identity_hash))
+        if must_exit:
+            exit(2)
+        else:
+            return
 
-def main():
+def main(must_exit=True, rns_instance=None):
     try:
         parser = argparse.ArgumentParser(description="Reticulum Network Stack Status")
         parser.add_argument("--config", action="store", default=None, help="path to alternative Reticulum config directory", type=str)
@@ -172,6 +443,82 @@ def main():
             help="show all interfaces",
             default=False
         )
+        
+        parser.add_argument(
+            "-A",
+            "--announce-stats",
+            action="store_true",
+            help="show announce stats",
+            default=False
+        )
+        
+        parser.add_argument(
+            "-l",
+            "--link-stats",
+            action="store_true",
+            help="show link stats",
+            default=False,
+        )
+        
+        parser.add_argument(
+            "-t",
+            "--totals",
+            action="store_true",
+            help="display traffic totals",
+            default=False,
+        )
+        
+        parser.add_argument(
+            "-s",
+            "--sort",
+            action="store",
+            help="sort interfaces by [rate, traffic, rx, tx, rxs, txs, announces, arx, atx, held]",
+            default=None,
+            type=str
+        )
+        
+        parser.add_argument(
+            "-r",
+            "--reverse",
+            action="store_true",
+            help="reverse sorting",
+            default=False,
+        )
+        
+        parser.add_argument(
+            "-j",
+            "--json",
+            action="store_true",
+            help="output in JSON format",
+            default=False
+        )
+
+        parser.add_argument(
+            "-R",
+            action="store",
+            metavar="hash",
+            help="transport identity hash of remote instance to get status from",
+            default=None,
+            type=str
+        )
+
+        parser.add_argument(
+            "-i",
+            action="store",
+            metavar="path",
+            help="path to identity used for remote management",
+            default=None,
+            type=str
+        )
+
+        parser.add_argument(
+            "-w",
+            action="store",
+            metavar="seconds",
+            type=float,
+            help="timeout before giving up on remote queries",
+            default=RNS.Transport.PATH_REQUEST_TIMEOUT
+        )
 
         parser.add_argument('-v', '--verbose', action='count', default=0)
 
@@ -184,11 +531,30 @@ def main():
         else:
             configarg = None
 
-        program_setup(configdir = configarg, dispall = args.all, verbosity=args.verbose, name_filter=args.filter)
+        program_setup(
+            configdir = configarg,
+            dispall = args.all,
+            verbosity=args.verbose,
+            name_filter=args.filter,
+            json=args.json,
+            astats=args.announce_stats,
+            lstats=args.link_stats,
+            sorting=args.sort,
+            sort_reverse=args.reverse,
+            remote=args.R,
+            management_identity=args.i,
+            remote_timeout=args.w,
+            must_exit=must_exit,
+            rns_instance=rns_instance,
+            traffic_totals=args.totals,
+        )
 
     except KeyboardInterrupt:
         print("")
-        exit()
+        if must_exit:
+            exit()
+        else:
+            return
 
 def speed_str(num, suffix='bps'):
     units = ['','k','M','G','T','P','E','Z']

RNS/Utilities/rnx.py

@@ -28,8 +28,8 @@ import argparse
 import shlex
 import time
 import sys
-import tty
 import os
+#import tty
 
 from RNS._version import __version__
 
@@ -538,7 +538,7 @@ def main():
         parser.add_argument("-x", '--interactive', action='store_true', default=False, help="enter interactive mode")
         parser.add_argument("-b", '--no-announce', action='store_true', default=False, help="don't announce at program start")
         parser.add_argument('-a', metavar="allowed_hash", dest="allowed", action='append', help="accept from this identity", type=str)
-        parser.add_argument('-n', '--noauth', action='store_true', default=False, help="accept files from anyone")
+        parser.add_argument('-n', '--noauth', action='store_true', default=False, help="accept commands from anyone")
         parser.add_argument('-N', '--noid', action='store_true', default=False, help="don't identify to listener")
         parser.add_argument("-d", '--detailed', action='store_true', default=False, help="show detailed result output")
         parser.add_argument("-m", action='store_true', dest="mirror", default=False, help="mirror exit code of remote command")

RNS/__init__.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io and contributors
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -24,6 +24,7 @@ import os
 import sys
 import glob
 import time
+import datetime
 import random
 import threading
 
@@ -32,16 +33,21 @@ from ._version import __version__
 from .Reticulum import Reticulum
 from .Identity import Identity
 from .Link import Link, RequestReceipt
+from .Channel import MessageBase
+from .Buffer import Buffer, RawChannelReader, RawChannelWriter
 from .Transport import Transport
 from .Destination import Destination
 from .Packet import Packet
 from .Packet import PacketReceipt
+from .Resolver import Resolver
 from .Resource import Resource, ResourceAdvertisement
 from .Cryptography import HKDF
 from .Cryptography import Hashes
 
-modules = glob.glob(os.path.dirname(__file__)+"/*.py")
-__all__ = [ os.path.basename(f)[:-3] for f in modules if not f.endswith('__init__.py')]
+py_modules  = glob.glob(os.path.dirname(__file__)+"/*.py")
+pyc_modules = glob.glob(os.path.dirname(__file__)+"/*.pyc")
+modules     = py_modules+pyc_modules
+__all__ = list(set([os.path.basename(f).replace(".pyc", "").replace(".py", "") for f in modules if not (f.endswith("__init__.py") or f.endswith("__init__.pyc"))]))
 
 LOG_CRITICAL = 0
 LOG_ERROR    = 1
@@ -54,13 +60,17 @@ LOG_EXTREME  = 7
 
 LOG_STDOUT   = 0x91
 LOG_FILE     = 0x92
+LOG_CALLBACK = 0x93
 
 LOG_MAXSIZE  = 5*1024*1024
 
-loglevel     = LOG_NOTICE
-logfile      = None
-logdest      = LOG_STDOUT
-logtimefmt   = "%Y-%m-%d %H:%M:%S"
+loglevel        = LOG_NOTICE
+logfile         = None
+logdest         = LOG_STDOUT
+logcall         = None
+logtimefmt      = "%Y-%m-%d %H:%M:%S"
+logtimefmt_p    = "%H:%M:%S.%f"
+compact_log_fmt = False
 
 instance_random = random.Random()
 instance_random.seed(os.urandom(10))
@@ -100,42 +110,63 @@ def timestamp_str(time_s):
     timestamp = time.localtime(time_s)
     return time.strftime(logtimefmt, timestamp)
 
-def log(msg, level=3, _override_destination = False):
-    global _always_override_destination
-    
+def precise_timestamp_str(time_s):
+    return datetime.datetime.now().strftime(logtimefmt_p)[:-3]
+
+def log(msg, level=3, _override_destination = False, pt=False):
+    global _always_override_destination, compact_log_fmt
+    msg = str(msg)
     if loglevel >= level:
-        logstring = "["+timestamp_str(time.time())+"] ["+loglevelname(level)+"] "+msg
-        logging_lock.acquire()
+        if pt:
+            logstring = "["+precise_timestamp_str(time.time())+"] ["+loglevelname(level)+"] "+msg
+        else:
+            if not compact_log_fmt:
+                logstring = "["+timestamp_str(time.time())+"] ["+loglevelname(level)+"] "+msg
+            else:
+                logstring = "["+timestamp_str(time.time())+"] "+msg
 
-        if (logdest == LOG_STDOUT or _always_override_destination or _override_destination):
-            print(logstring)
-            logging_lock.release()
+        with logging_lock:
+            if (logdest == LOG_STDOUT or _always_override_destination or _override_destination):
+                print(logstring)
 
-        elif (logdest == LOG_FILE and logfile != None):
-            try:
-                file = open(logfile, "a")
-                file.write(logstring+"\n")
-                file.close()
-                
-                if os.path.getsize(logfile) > LOG_MAXSIZE:
-                    prevfile = logfile+".1"
-                    if os.path.isfile(prevfile):
-                        os.unlink(prevfile)
-                    os.rename(logfile, prevfile)
+            elif (logdest == LOG_FILE and logfile != None):
+                try:
+                    file = open(logfile, "a")
+                    file.write(logstring+"\n")
+                    file.close()
+                    
+                    if os.path.getsize(logfile) > LOG_MAXSIZE:
+                        prevfile = logfile+".1"
+                        if os.path.isfile(prevfile):
+                            os.unlink(prevfile)
+                        os.rename(logfile, prevfile)
 
-                logging_lock.release()
-            except Exception as e:
-                logging_lock.release()
-                _always_override_destination = True
-                log("Exception occurred while writing log message to log file: "+str(e), LOG_CRITICAL)
-                log("Dumping future log events to console!", LOG_CRITICAL)
-                log(msg, level)
+                except Exception as e:
+                    _always_override_destination = True
+                    log("Exception occurred while writing log message to log file: "+str(e), LOG_CRITICAL)
+                    log("Dumping future log events to console!", LOG_CRITICAL)
+                    log(msg, level)
+
+            elif logdest == LOG_CALLBACK:
+                try:
+                    logcall(logstring)
+                except Exception as e:
+                    _always_override_destination = True
+                    log("Exception occurred while calling external log handler: "+str(e), LOG_CRITICAL)
+                    log("Dumping future log events to console!", LOG_CRITICAL)
+                    log(msg, level)
                 
 
 def rand():
     result = instance_random.random()
     return result
 
+def trace_exception(e):
+    import traceback
+    exception_info = "".join(traceback.TracebackException.from_exception(e).format())
+    log(f"An unhandled {str(type(e))} exception occurred: {str(e)}", LOG_ERROR)
+    log(exception_info, LOG_ERROR)
+
 def hexrep(data, delimit=True):
     try:
         iter(data)
@@ -153,6 +184,9 @@ def prettyhexrep(data):
     hexrep = "<"+delimiter.join("{:02x}".format(c) for c in data)+">"
     return hexrep
 
+def prettyspeed(num, suffix="b"):
+    return prettysize(num/8, suffix=suffix)+"ps"
+
 def prettysize(num, suffix='B'):
     units = ['','K','M','G','T','P','E','Z']
     last_unit = 'Y'
@@ -172,32 +206,73 @@ def prettysize(num, suffix='B'):
 
     return "%.2f%s%s" % (num, last_unit, suffix)
 
-def prettytime(time, verbose=False):
+def prettyfrequency(hz, suffix="Hz"):
+    num = hz*1e6
+    units = ["µ", "m", "", "K","M","G","T","P","E","Z"]
+    last_unit = "Y"
+
+    for unit in units:
+        if abs(num) < 1000.0:
+            return "%.2f %s%s" % (num, unit, suffix)
+        num /= 1000.0
+
+    return "%.2f%s%s" % (num, last_unit, suffix)
+
+def prettydistance(m, suffix="m"):
+    num = m*1e6
+    units = ["µ", "m", "c", ""]
+    last_unit = "K"
+
+    for unit in units:
+        divisor = 1000.0
+        if unit == "m": divisor = 10
+        if unit == "c": divisor = 100
+
+        if abs(num) < divisor:
+            return "%.2f %s%s" % (num, unit, suffix)
+        num /= divisor
+
+    return "%.2f %s%s" % (num, last_unit, suffix)
+
+def prettytime(time, verbose=False, compact=False):
+    neg = False
+    if time < 0:
+        time = abs(time)
+        neg = True
+
     days = int(time // (24 * 3600))
     time = time % (24 * 3600)
     hours = int(time // 3600)
     time %= 3600
     minutes = int(time // 60)
     time %= 60
-    seconds = round(time, 2)
+    if compact:
+        seconds = int(time)
+    else:
+        seconds = round(time, 2)
     
     ss = "" if seconds == 1 else "s"
     sm = "" if minutes == 1 else "s"
     sh = "" if hours == 1 else "s"
     sd = "" if days == 1 else "s"
 
+    displayed = 0
     components = []
-    if days > 0:
+    if days > 0 and ((not compact) or displayed < 2):
         components.append(str(days)+" day"+sd if verbose else str(days)+"d")
+        displayed += 1
 
-    if hours > 0:
+    if hours > 0 and ((not compact) or displayed < 2):
         components.append(str(hours)+" hour"+sh if verbose else str(hours)+"h")
+        displayed += 1
 
-    if minutes > 0:
+    if minutes > 0 and ((not compact) or displayed < 2):
         components.append(str(minutes)+" minute"+sm if verbose else str(minutes)+"m")
+        displayed += 1
 
-    if seconds > 0:
+    if seconds > 0 and ((not compact) or displayed < 2):
         components.append(str(seconds)+" second"+ss if verbose else str(seconds)+"s")
+        displayed += 1
 
     i = 0
     tstr = ""
@@ -212,14 +287,74 @@ def prettytime(time, verbose=False):
 
         tstr += c
 
-    return tstr
+    if tstr == "":
+        return "0s"
+    else:
+        if not neg:
+            return tstr
+        else:
+            return f"-{tstr}"
+
+def prettyshorttime(time, verbose=False, compact=False):
+    neg = False
+    time = time*1e6
+    if time < 0:
+        time = abs(time)
+        neg = True
+    
+    seconds = int(time // 1e6); time %= 1e6
+    milliseconds = int(time // 1e3); time %= 1e3
+
+    if compact:
+        microseconds = int(time)
+    else:
+        microseconds = round(time, 2)
+    
+    ss = "" if seconds == 1 else "s"
+    sms = "" if milliseconds == 1 else "s"
+    sus = "" if microseconds == 1 else "s"
+
+    displayed = 0
+    components = []
+    if seconds > 0 and ((not compact) or displayed < 2):
+        components.append(str(seconds)+" second"+ss if verbose else str(seconds)+"s")
+        displayed += 1
+
+    if milliseconds > 0 and ((not compact) or displayed < 2):
+        components.append(str(milliseconds)+" millisecond"+sms if verbose else str(milliseconds)+"ms")
+        displayed += 1
+
+    if microseconds > 0 and ((not compact) or displayed < 2):
+        components.append(str(microseconds)+" microsecond"+sus if verbose else str(microseconds)+"µs")
+        displayed += 1
+
+    i = 0
+    tstr = ""
+    for c in components:
+        i += 1
+        if i == 1:
+            pass
+        elif i < len(components):
+            tstr += ", "
+        elif i == len(components):
+            tstr += " and "
+
+        tstr += c
+
+    if tstr == "":
+        return "0us"
+    else:
+        if not neg:
+            return tstr
+        else:
+            return f"-{tstr}"
 
 def phyparams():
     print("Required Physical Layer MTU : "+str(Reticulum.MTU)+" bytes")
     print("Plaintext Packet MDU        : "+str(Packet.PLAIN_MDU)+" bytes")
     print("Encrypted Packet MDU        : "+str(Packet.ENCRYPTED_MDU)+" bytes")
     print("Link Curve                  : "+str(Link.CURVE))
-    print("Link Packet MDU             : "+str(Packet.ENCRYPTED_MDU)+" bytes")
+    print("Link Packet MDU             : "+str(Link.MDU)+" bytes")
     print("Link Public Key Size        : "+str(Link.ECPUBSIZE*8)+" bits")
     print("Link Private Key Size       : "+str(Link.KEYSIZE*8)+" bits")
 
@@ -228,4 +363,163 @@ def panic():
 
 def exit():
     print("")
-    sys.exit(0)
+    sys.exit(0)
+
+class Profiler:
+    _ran = False
+    profilers = {}
+    tags = {}
+
+    @staticmethod
+    def get_profiler(tag=None, super_tag=None):
+        if tag in Profiler.profilers:
+            return Profiler.profilers[tag]
+        else:
+            profiler = Profiler(tag, super_tag)
+            Profiler.profilers[tag] = profiler
+            return profiler
+
+    def __init__(self, tag=None, super_tag=None):
+        self.paused = False
+        self.pause_time = 0
+        self.pause_started = None
+        self.tag = tag
+        self.super_tag = super_tag
+        if self.super_tag in Profiler.profilers:
+            self.super_profiler = Profiler.profilers[self.super_tag]
+            self.pause_super = self.super_profiler.pause
+            self.resume_super = self.super_profiler.resume
+        else:
+            def noop(self=None):
+                pass
+            self.super_profiler = None
+            self.pause_super = noop
+            self.resume_super = noop
+
+    def __enter__(self):
+        self.pause_super()
+        tag = self.tag
+        super_tag = self.super_tag
+        thread_ident = threading.get_ident()
+        if not tag in Profiler.tags:
+            Profiler.tags[tag] = {"threads": {}, "super": super_tag}
+        if not thread_ident in Profiler.tags[tag]["threads"]:
+            Profiler.tags[tag]["threads"][thread_ident] = {"current_start": None, "captures": []}
+
+        Profiler.tags[tag]["threads"][thread_ident]["current_start"] = time.perf_counter()
+        self.resume_super()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.pause_super()
+        tag = self.tag
+        super_tag = self.super_tag
+        end = time.perf_counter() - self.pause_time
+        self.pause_time = 0
+        thread_ident = threading.get_ident()
+        if tag in Profiler.tags and thread_ident in Profiler.tags[tag]["threads"]:
+            if Profiler.tags[tag]["threads"][thread_ident]["current_start"] != None:
+                begin = Profiler.tags[tag]["threads"][thread_ident]["current_start"]
+                Profiler.tags[tag]["threads"][thread_ident]["current_start"] = None
+                Profiler.tags[tag]["threads"][thread_ident]["captures"].append(end-begin)
+                if not Profiler._ran:
+                    Profiler._ran = True
+        self.resume_super()
+
+    def pause(self, pause_started=None):
+        if not self.paused:
+            self.paused = True
+            self.pause_started = pause_started or time.perf_counter()
+            self.pause_super(self.pause_started)
+
+    def resume(self):
+        if self.paused:
+            self.pause_time += time.perf_counter() - self.pause_started
+            self.paused = False
+            self.resume_super()
+
+    @staticmethod
+    def ran():
+        return Profiler._ran
+
+    @staticmethod
+    def results():
+        from statistics import mean, median, stdev
+        results = {}
+        
+        for tag in Profiler.tags:
+            tag_captures = []
+            tag_entry = Profiler.tags[tag]
+            
+            for thread_ident in tag_entry["threads"]:
+                thread_entry = tag_entry["threads"][thread_ident]
+                thread_captures = thread_entry["captures"]
+                sample_count = len(thread_captures)
+                
+                if sample_count > 1:
+                    thread_results = {
+                        "count": sample_count,
+                        "mean": mean(thread_captures),
+                        "median": median(thread_captures),
+                        "stdev": stdev(thread_captures)
+                    }
+                elif sample_count == 1:
+                    thread_results = {
+                        "count": sample_count,
+                        "mean": mean(thread_captures),
+                        "median": median(thread_captures),
+                        "stdev": None
+                    }
+
+                tag_captures.extend(thread_captures)
+
+            sample_count = len(tag_captures)
+            if sample_count > 1:
+                tag_results = {
+                    "name": tag,
+                    "super": tag_entry["super"],
+                    "count": len(tag_captures),
+                    "mean": mean(tag_captures),
+                    "median": median(tag_captures),
+                    "stdev": stdev(tag_captures)
+                }
+            elif sample_count == 1:
+                tag_results = {
+                    "name": tag,
+                    "super": tag_entry["super"],
+                    "count": len(tag_captures),
+                    "mean": mean(tag_captures),
+                    "median": median(tag_captures),
+                    "stdev": None
+                }
+
+            results[tag] = tag_results
+
+        def print_results_recursive(tag, results, level=0):
+            print_tag_results(tag, level+1)
+
+            for tag_name in results:
+                sub_tag = results[tag_name]
+                if sub_tag["super"] == tag["name"]:
+                    print_results_recursive(sub_tag, results, level=level+1)
+
+
+        def print_tag_results(tag, level):
+            ind = "  "*level
+            name = tag["name"]; count = tag["count"]
+            mean = tag["mean"]; median = tag["median"]; stdev = tag["stdev"]
+            print(    f"{ind}{name}")
+            print(    f"{ind}  Samples  : {count}")
+            if stdev != None:
+                print(f"{ind}  Mean     : {prettyshorttime(mean)}")
+                print(f"{ind}  Median   : {prettyshorttime(median)}")
+                print(f"{ind}  St.dev.  : {prettyshorttime(stdev)}")
+            print(    f"{ind}  Total    : {prettyshorttime(mean*count)}")
+            print("")
+
+        print("\nProfiler results:\n")
+        for tag_name in results:
+            tag = results[tag_name]
+            if tag["super"] == None:
+                print_results_recursive(tag, results)
+
+profile = Profiler.get_profiler

RNS/_version.py

@@ -1 +1 @@
-__version__ = "0.3.12"
+__version__ = "0.9.0"

RNS/vendor/__init__.py

@@ -1,5 +1,7 @@
 import os
 import glob
 
-modules = glob.glob(os.path.dirname(__file__)+"/*.py")
-__all__ = [ os.path.basename(f)[:-3] for f in modules if not f.endswith('__init__.py')]
+py_modules  = glob.glob(os.path.dirname(__file__)+"/*.py")
+pyc_modules = glob.glob(os.path.dirname(__file__)+"/*.pyc")
+modules     = py_modules+pyc_modules
+__all__ = list(set([os.path.basename(f).replace(".pyc", "").replace(".py", "") for f in modules if not (f.endswith("__init__.py") or f.endswith("__init__.pyc"))]))

RNS/vendor/ifaddr/__init__.py

@@ -0,0 +1,33 @@
+# Copyright (c) 2014 Stefan C. Mueller
+
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+
+import os
+
+from RNS.vendor.ifaddr._shared import Adapter, IP
+
+if os.name == "nt":
+    from RNS.vendor.ifaddr._win32 import get_adapters
+elif os.name == "posix":
+    from RNS.vendor.ifaddr._posix import get_adapters
+else:
+    raise RuntimeError("Unsupported Operating System: %s" % os.name)
+
+__all__ = ['Adapter', 'IP', 'get_adapters']

RNS/vendor/ifaddr/_posix.py

@@ -0,0 +1,93 @@
+# Copyright (c) 2014 Stefan C. Mueller
+
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+
+import os
+import ctypes.util
+import ipaddress
+import collections
+import socket
+
+from typing import Iterable, Optional
+
+import RNS.vendor.ifaddr._shared as shared
+
+class ifaddrs(ctypes.Structure):
+    pass
+
+
+ifaddrs._fields_ = [
+    ('ifa_next', ctypes.POINTER(ifaddrs)),
+    ('ifa_name', ctypes.c_char_p),
+    ('ifa_flags', ctypes.c_uint),
+    ('ifa_addr', ctypes.POINTER(shared.sockaddr)),
+    ('ifa_netmask', ctypes.POINTER(shared.sockaddr)),
+]
+
+libc = ctypes.CDLL(ctypes.util.find_library("socket" if os.uname()[0] == "SunOS" else "c"), use_errno=True)  # type: ignore
+
+
+def get_adapters(include_unconfigured: bool = False) -> Iterable[shared.Adapter]:
+
+    addr0 = addr = ctypes.POINTER(ifaddrs)()
+    retval = libc.getifaddrs(ctypes.byref(addr))
+    if retval != 0:
+        eno = ctypes.get_errno()
+        raise OSError(eno, os.strerror(eno))
+
+    ips = collections.OrderedDict()
+
+    def add_ip(adapter_name: str, ip: Optional[shared.IP]) -> None:
+        if adapter_name not in ips:
+            index = None  # type: Optional[int]
+            try:
+                # Mypy errors on this when the Windows CI runs:
+                #     error: Module has no attribute "if_nametoindex"
+                index = socket.if_nametoindex(adapter_name)  # type: ignore
+            except (OSError, AttributeError):
+                pass
+            ips[adapter_name] = shared.Adapter(adapter_name, adapter_name, [], index=index)
+        if ip is not None:
+            ips[adapter_name].ips.append(ip)
+
+    while addr:
+        name = addr[0].ifa_name.decode(encoding='UTF-8')
+        ip_addr = shared.sockaddr_to_ip(addr[0].ifa_addr)
+        if ip_addr:
+            if addr[0].ifa_netmask and not addr[0].ifa_netmask[0].sa_familiy:
+                addr[0].ifa_netmask[0].sa_familiy = addr[0].ifa_addr[0].sa_familiy
+            netmask = shared.sockaddr_to_ip(addr[0].ifa_netmask)
+            if isinstance(netmask, tuple):
+                netmaskStr = str(netmask[0])
+                prefixlen = shared.ipv6_prefixlength(ipaddress.IPv6Address(netmaskStr))
+            else:
+                assert netmask is not None, f'sockaddr_to_ip({addr[0].ifa_netmask}) returned None'
+                netmaskStr = str('0.0.0.0/' + netmask)
+                prefixlen = ipaddress.IPv4Network(netmaskStr).prefixlen
+            ip = shared.IP(ip_addr, prefixlen, name)
+            add_ip(name, ip)
+        else:
+            if include_unconfigured:
+                add_ip(name, None)
+        addr = addr[0].ifa_next
+
+    libc.freeifaddrs(addr0)
+
+    return ips.values()

RNS/vendor/ifaddr/_shared.py

@@ -0,0 +1,198 @@
+# Copyright (c) 2014 Stefan C. Mueller
+
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+
+import ctypes
+import socket
+import ipaddress
+import platform
+
+from typing import List, Optional, Tuple, Union
+
+class Adapter(object):
+    """
+    Represents a network interface device controller (NIC), such as a
+    network card. An adapter can have multiple IPs.
+
+    On Linux aliasing (multiple IPs per physical NIC) is implemented
+    by creating 'virtual' adapters, each represented by an instance
+    of this class. Each of those 'virtual' adapters can have both
+    a IPv4 and an IPv6 IP address.
+    """
+
+    def __init__(self, name: str, nice_name: str, ips: List['IP'], index: Optional[int] = None) -> None:
+
+        #: Unique name that identifies the adapter in the system.
+        #: On Linux this is of the form of `eth0` or `eth0:1`, on
+        #: Windows it is a UUID in string representation, such as
+        #: `{846EE342-7039-11DE-9D20-806E6F6E6963}`.
+        self.name = name
+
+        #: Human readable name of the adpater. On Linux this
+        #: is currently the same as :attr:`name`. On Windows
+        #: this is the name of the device.
+        self.nice_name = nice_name
+
+        #: List of :class:`ifaddr.IP` instances in the order they were
+        #: reported by the system.
+        self.ips = ips
+
+        #: Adapter index as used by some API (e.g. IPv6 multicast group join).
+        self.index = index
+
+    def __repr__(self) -> str:
+        return "Adapter(name={name}, nice_name={nice_name}, ips={ips}, index={index})".format(
+            name=repr(self.name), nice_name=repr(self.nice_name), ips=repr(self.ips), index=repr(self.index)
+        )
+
+
+# Type of an IPv4 address (a string in "xxx.xxx.xxx.xxx" format)
+_IPv4Address = str
+
+# Type of an IPv6 address (a three-tuple `(ip, flowinfo, scope_id)`)
+_IPv6Address = Tuple[str, int, int]
+
+
+class IP(object):
+    """
+    Represents an IP address of an adapter.
+    """
+
+    def __init__(self, ip: Union[_IPv4Address, _IPv6Address], network_prefix: int, nice_name: str) -> None:
+
+        #: IP address. For IPv4 addresses this is a string in
+        #: "xxx.xxx.xxx.xxx" format. For IPv6 addresses this
+        #: is a three-tuple `(ip, flowinfo, scope_id)`, where
+        #: `ip` is a string in the usual collon separated
+        #: hex format.
+        self.ip = ip
+
+        #: Number of bits of the IP that represent the
+        #: network. For a `255.255.255.0` netmask, this
+        #: number would be `24`.
+        self.network_prefix = network_prefix
+
+        #: Human readable name for this IP.
+        #: On Linux is this currently the same as the adapter name.
+        #: On Windows this is the name of the network connection
+        #: as configured in the system control panel.
+        self.nice_name = nice_name
+
+    @property
+    def is_IPv4(self) -> bool:
+        """
+        Returns `True` if this IP is an IPv4 address and `False`
+        if it is an IPv6 address.
+        """
+        return not isinstance(self.ip, tuple)
+
+    @property
+    def is_IPv6(self) -> bool:
+        """
+        Returns `True` if this IP is an IPv6 address and `False`
+        if it is an IPv4 address.
+        """
+        return isinstance(self.ip, tuple)
+
+    def __repr__(self) -> str:
+        return "IP(ip={ip}, network_prefix={network_prefix}, nice_name={nice_name})".format(
+            ip=repr(self.ip), network_prefix=repr(self.network_prefix), nice_name=repr(self.nice_name)
+        )
+
+
+if platform.system() == "Darwin" or "BSD" in platform.system():
+
+    # BSD derived systems use marginally different structures
+    # than either Linux or Windows.
+    # I still keep it in `shared` since we can use
+    # both structures equally.
+
+    class sockaddr(ctypes.Structure):
+        _fields_ = [
+            ('sa_len', ctypes.c_uint8),
+            ('sa_familiy', ctypes.c_uint8),
+            ('sa_data', ctypes.c_uint8 * 14),
+        ]
+
+    class sockaddr_in(ctypes.Structure):
+        _fields_ = [
+            ('sa_len', ctypes.c_uint8),
+            ('sa_familiy', ctypes.c_uint8),
+            ('sin_port', ctypes.c_uint16),
+            ('sin_addr', ctypes.c_uint8 * 4),
+            ('sin_zero', ctypes.c_uint8 * 8),
+        ]
+
+    class sockaddr_in6(ctypes.Structure):
+        _fields_ = [
+            ('sa_len', ctypes.c_uint8),
+            ('sa_familiy', ctypes.c_uint8),
+            ('sin6_port', ctypes.c_uint16),
+            ('sin6_flowinfo', ctypes.c_uint32),
+            ('sin6_addr', ctypes.c_uint8 * 16),
+            ('sin6_scope_id', ctypes.c_uint32),
+        ]
+
+else:
+
+    class sockaddr(ctypes.Structure):  # type: ignore
+        _fields_ = [('sa_familiy', ctypes.c_uint16), ('sa_data', ctypes.c_uint8 * 14)]
+
+    class sockaddr_in(ctypes.Structure):  # type: ignore
+        _fields_ = [
+            ('sin_familiy', ctypes.c_uint16),
+            ('sin_port', ctypes.c_uint16),
+            ('sin_addr', ctypes.c_uint8 * 4),
+            ('sin_zero', ctypes.c_uint8 * 8),
+        ]
+
+    class sockaddr_in6(ctypes.Structure):  # type: ignore
+        _fields_ = [
+            ('sin6_familiy', ctypes.c_uint16),
+            ('sin6_port', ctypes.c_uint16),
+            ('sin6_flowinfo', ctypes.c_uint32),
+            ('sin6_addr', ctypes.c_uint8 * 16),
+            ('sin6_scope_id', ctypes.c_uint32),
+        ]
+
+
+def sockaddr_to_ip(sockaddr_ptr: 'ctypes.pointer[sockaddr]') -> Optional[Union[_IPv4Address, _IPv6Address]]:
+    if sockaddr_ptr:
+        if sockaddr_ptr[0].sa_familiy == socket.AF_INET:
+            ipv4 = ctypes.cast(sockaddr_ptr, ctypes.POINTER(sockaddr_in))
+            ippacked = bytes(bytearray(ipv4[0].sin_addr))
+            ip = str(ipaddress.ip_address(ippacked))
+            return ip
+        elif sockaddr_ptr[0].sa_familiy == socket.AF_INET6:
+            ipv6 = ctypes.cast(sockaddr_ptr, ctypes.POINTER(sockaddr_in6))
+            flowinfo = ipv6[0].sin6_flowinfo
+            ippacked = bytes(bytearray(ipv6[0].sin6_addr))
+            ip = str(ipaddress.ip_address(ippacked))
+            scope_id = ipv6[0].sin6_scope_id
+            return (ip, flowinfo, scope_id)
+    return None
+
+
+def ipv6_prefixlength(address: ipaddress.IPv6Address) -> int:
+    prefix_length = 0
+    for i in range(address.max_prefixlen):
+        if int(address) >> i & 1:
+            prefix_length = prefix_length + 1
+    return prefix_length

RNS/vendor/ifaddr/_win32.py

@@ -0,0 +1,145 @@
+# Copyright (c) 2014 Stefan C. Mueller
+
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+
+import ctypes
+from ctypes import wintypes
+from typing import Iterable, List
+
+import RNS.vendor.ifaddr._shared as shared
+
+NO_ERROR = 0
+ERROR_BUFFER_OVERFLOW = 111
+MAX_ADAPTER_NAME_LENGTH = 256
+MAX_ADAPTER_DESCRIPTION_LENGTH = 128
+MAX_ADAPTER_ADDRESS_LENGTH = 8
+AF_UNSPEC = 0
+
+
+class SOCKET_ADDRESS(ctypes.Structure):
+    _fields_ = [('lpSockaddr', ctypes.POINTER(shared.sockaddr)), ('iSockaddrLength', wintypes.INT)]
+
+
+class IP_ADAPTER_UNICAST_ADDRESS(ctypes.Structure):
+    pass
+
+
+IP_ADAPTER_UNICAST_ADDRESS._fields_ = [
+    ('Length', wintypes.ULONG),
+    ('Flags', wintypes.DWORD),
+    ('Next', ctypes.POINTER(IP_ADAPTER_UNICAST_ADDRESS)),
+    ('Address', SOCKET_ADDRESS),
+    ('PrefixOrigin', ctypes.c_uint),
+    ('SuffixOrigin', ctypes.c_uint),
+    ('DadState', ctypes.c_uint),
+    ('ValidLifetime', wintypes.ULONG),
+    ('PreferredLifetime', wintypes.ULONG),
+    ('LeaseLifetime', wintypes.ULONG),
+    ('OnLinkPrefixLength', ctypes.c_uint8),
+]
+
+
+class IP_ADAPTER_ADDRESSES(ctypes.Structure):
+    pass
+
+
+IP_ADAPTER_ADDRESSES._fields_ = [
+    ('Length', wintypes.ULONG),
+    ('IfIndex', wintypes.DWORD),
+    ('Next', ctypes.POINTER(IP_ADAPTER_ADDRESSES)),
+    ('AdapterName', ctypes.c_char_p),
+    ('FirstUnicastAddress', ctypes.POINTER(IP_ADAPTER_UNICAST_ADDRESS)),
+    ('FirstAnycastAddress', ctypes.c_void_p),
+    ('FirstMulticastAddress', ctypes.c_void_p),
+    ('FirstDnsServerAddress', ctypes.c_void_p),
+    ('DnsSuffix', ctypes.c_wchar_p),
+    ('Description', ctypes.c_wchar_p),
+    ('FriendlyName', ctypes.c_wchar_p),
+]
+
+
+iphlpapi = ctypes.windll.LoadLibrary("Iphlpapi")  # type: ignore
+
+
+def enumerate_interfaces_of_adapter(
+    nice_name: str, address: IP_ADAPTER_UNICAST_ADDRESS
+) -> Iterable[shared.IP]:
+
+    # Iterate through linked list and fill list
+    addresses = []  # type: List[IP_ADAPTER_UNICAST_ADDRESS]
+    while True:
+        addresses.append(address)
+        if not address.Next:
+            break
+        address = address.Next[0]
+
+    for address in addresses:
+        ip = shared.sockaddr_to_ip(address.Address.lpSockaddr)
+        assert ip is not None, f'sockaddr_to_ip({address.Address.lpSockaddr}) returned None'
+        network_prefix = address.OnLinkPrefixLength
+        yield shared.IP(ip, network_prefix, nice_name)
+
+
+def get_adapters(include_unconfigured: bool = False) -> Iterable[shared.Adapter]:
+
+    # Call GetAdaptersAddresses() with error and buffer size handling
+
+    addressbuffersize = wintypes.ULONG(15 * 1024)
+    retval = ERROR_BUFFER_OVERFLOW
+    while retval == ERROR_BUFFER_OVERFLOW:
+        addressbuffer = ctypes.create_string_buffer(addressbuffersize.value)
+        retval = iphlpapi.GetAdaptersAddresses(
+            wintypes.ULONG(AF_UNSPEC),
+            wintypes.ULONG(0),
+            None,
+            ctypes.byref(addressbuffer),
+            ctypes.byref(addressbuffersize),
+        )
+    if retval != NO_ERROR:
+        raise ctypes.WinError()  # type: ignore
+
+    # Iterate through adapters fill array
+    address_infos = []  # type: List[IP_ADAPTER_ADDRESSES]
+    address_info = IP_ADAPTER_ADDRESSES.from_buffer(addressbuffer)
+    while True:
+        address_infos.append(address_info)
+        if not address_info.Next:
+            break
+        address_info = address_info.Next[0]
+
+    # Iterate through unicast addresses
+    result = []  # type: List[shared.Adapter]
+    for adapter_info in address_infos:
+
+        # We don't expect non-ascii characters here, so encoding shouldn't matter
+        name = adapter_info.AdapterName.decode()
+        nice_name = adapter_info.Description
+        index = adapter_info.IfIndex
+
+        if adapter_info.FirstUnicastAddress:
+            ips = enumerate_interfaces_of_adapter(
+                adapter_info.FriendlyName, adapter_info.FirstUnicastAddress[0]
+            )
+            ips = list(ips)
+            result.append(shared.Adapter(name, nice_name, ips, index=index))
+        elif include_unconfigured:
+            result.append(shared.Adapter(name, nice_name, [], index=index))
+
+    return result

RNS/vendor/ifaddr/niwrapper.py

@@ -0,0 +1,57 @@
+import ipaddress
+import RNS.vendor.ifaddr
+import socket
+
+from typing import List
+
+AF_INET6 = socket.AF_INET6.value
+AF_INET = socket.AF_INET.value
+
+def interfaces() -> List[str]:
+    adapters = RNS.vendor.ifaddr.get_adapters(include_unconfigured=True)
+    return [a.name for a in adapters]
+
+def interface_names_to_indexes() -> dict:
+    adapters = RNS.vendor.ifaddr.get_adapters(include_unconfigured=True)
+    results = {}
+    for adapter in adapters:
+        results[adapter.name] = adapter.index
+    return results
+
+def interface_name_to_nice_name(ifname) -> str:
+    try:
+        adapters = RNS.vendor.ifaddr.get_adapters(include_unconfigured=True)
+        for adapter in adapters:
+            if adapter.name == ifname:
+                if hasattr(adapter, "nice_name"):
+                    return adapter.nice_name
+    except:
+        return None
+
+    return None
+
+def ifaddresses(ifname) -> dict:
+    adapters = RNS.vendor.ifaddr.get_adapters(include_unconfigured=True)
+    ifa = {}
+    for a in adapters:
+        if a.name == ifname:
+            ipv4s = []
+            ipv6s = []
+            for ip in a.ips:
+                t = {}
+                if ip.is_IPv4:
+                    net = ipaddress.ip_network(str(ip.ip)+"/"+str(ip.network_prefix), strict=False)
+                    t["addr"] = ip.ip
+                    t["prefix"] = ip.network_prefix
+                    t["broadcast"] = str(net.broadcast_address)
+                    ipv4s.append(t)
+                if ip.is_IPv6:
+                    t["addr"] = ip.ip[0]
+                    ipv6s.append(t)
+
+            if len(ipv4s) > 0:
+                ifa[AF_INET] = ipv4s
+            if len(ipv6s) > 0:
+                ifa[AF_INET6] = ipv6s
+
+    return ifa

RNS/vendor/platformutils.py

@@ -8,6 +8,12 @@ def get_platform():
         import sys
         return sys.platform
 
+def is_linux():
+    if get_platform() == "linux":
+        return True
+    else:
+        return False
+
 def is_darwin():
     if get_platform() == "darwin":
         return True
@@ -42,4 +48,4 @@ def cryptography_old_api():
     if cryptography.__version__ == "2.8":
         return True
     else:
-        return False
+        return False

Roadmap.md

@@ -0,0 +1,109 @@
+# Reticulum Development Roadmap
+This document outlines the currently established development roadmap for Reticulum.
+
+1. [Currently Active Work Areas](#currently-active-work-areas)
+2. [Primary Efforts](#primary-efforts)
+    - [Comprehensibility](#comprehensibility)
+    - [Universality](#universality)
+    - [Functionality](#functionality)
+    - [Usability & Utility](#usability--utility)
+    - [Interfaceability](#interfaceability)
+3. [Auxillary Efforts](#auxillary-efforts)
+4. [Release History](#release-history)
+
+## Currently Active Work Areas
+For each release cycle of Reticulum, improvements and additions from the five [Primary Efforts](#primary-efforts) are selected as active work areas, and can be expected to be included in the upcoming releases within that cycle. While not entirely set in stone for each release cycle, they serve as a pointer of what to expect in the near future.
+
+- The current `0.8.x` release cycle aims at completing
+  - [ ] Hot-pluggable interface system
+  - [ ] External interface plugins
+  - [ ] Network-wide path balancing and multi-pathing
+  - [ ] Expanded hardware support
+  - [ ] Overhauling and updating the documentation
+  - [ ] Distributed Destination Naming System
+  - [ ] A standalone RNS Daemon app for Android
+  - [ ] Addding automatic retries to all use cases of the `Request` API
+  - [ ] Performance and memory optimisations of the Python reference implementation
+  - [ ] Fixing bugs discovered while operating Reticulum systems and applications
+
+## Primary Efforts
+The development path for Reticulum is currently laid out in five distinct areas: *Comprehensibility*, *Universality*, *Functionality*, *Usability & Utility* and *Interfaceability*. Conceptualising the development of Reticulum into these areas serves to advance the implementation and work towards the Foundational Goals & Values of Reticulum.
+
+### Comprehensibility
+These efforts are aimed at improving the ease of which Reticulum is understood, and lowering the barrier to entry for people who wish to start building systems on Reticulum.
+
+- Improving [the manual](https://markqvist.github.io/Reticulum/manual/) with tutorials specifically for beginners
+- Updating the documentation to reflect recent changes and improvements
+    - Update descriptions of protocol mechanics
+        - Update announce description
+        - Add in-depth explanation of the IFAC system
+    - Software
+        - Update software descriptions and screenshots
+    - Communications hardware section
+        - Add information about RNode external displays.
+        - Possibly add other relevant types here as well.
+    - Setup *Best Practices For...* / *Installation Examples* section.
+        - Home or office (example)
+        - Vehicles (example)
+        - No-grid/solar/remote sites (example)
+
+### Universality
+These efforts seek to broaden the universality of the Reticulum software and hardware ecosystem by continously diversifying platform support, and by improving the overall availability and ease of deployment of the Reticulum stack.
+
+- OpenWRT support
+- Create a standalone RNS Daemon app for Android
+- A lightweight and portable C implementation for microcontrollers, µRNS
+- A portable, high-performance Reticulum implementation in C/C++, see [#21](https://github.com/markqvist/Reticulum/discussions/21)
+- Performance and memory optimisations of the Python implementation
+- Bindings for other programming languages
+
+### Functionality
+These efforts aim to expand and improve the core functionality and reliability of Reticulum.
+
+- Add support for user-supplied external interface drivers
+- Add interface hot-plug and live up/down control to running instances
+- Add automatic retries to all use cases of the `Request` API
+- Network-wide path balancing
+- Distributed Destination Naming System
+- Globally routable multicast
+- Destination proxying
+- [Metric-based path selection and multiple paths](https://github.com/markqvist/Reticulum/discussions/86)
+
+### Usability & Utility
+These effors seek to make Reticulum easier to use and operate, and to expand the utility of the stack on deployed systems.
+
+- Easy way to share interface configurations, see [#19](https://github.com/markqvist/Reticulum/discussions/19)
+- Transit traffic display in rnstatus
+- rnsconfig utility
+
+### Interfaceability
+These efforts aim to expand the types of physical and virtual interfaces that Reticulum can natively use to transport data.
+
+- Plain ESP32 devices (ESP-Now, WiFi, Bluetooth, etc.)
+- More LoRa transceivers
+- AT-compatible modems
+- Filesystem interface
+- Direct SDR Support
+- Optical mediums
+- IR Transceivers
+- AWDL / OWL
+- HF Modems
+- GNU Radio
+- CAN-bus
+- Raw SPI
+- Raw i²c
+- MQTT
+- XBee
+- Tor
+
+## Auxillary Efforts
+The Reticulum ecosystem is enriched by several other software and hardware projects, and the support and improvement of these, in symbiosis with the core Reticulum project helps expand the reach and utility of Reticulum itself.
+
+This section lists, in no particular order, various important efforts that would be beneficial to the goals of Reticulum.
+
+- The [RNode](https://unsigned.io/rnode/) project
+    - [x] Create a WebUSB-based bootstrapping utility, and integrate this directly into the [RNode Bootstrap Console](#), both on-device, and on an Internet-reachable copy. This will make it much easier to create new RNodes for average users.
+
+## Release History
+
+Please see the [Changelog](./Changelog.md) for a complete release history and changelog of Reticulum.

docs/Makefile

@@ -30,3 +30,8 @@ help:
 		cp -r build/latex/reticulumnetworkstack.pdf ./Reticulum\ Manual.pdf; \
 		echo "PDF Manual Generated"; \
 	fi
+
+	@if [ $@ = "epub" ]; then \
+		cp -r build/epub/ReticulumNetworkStack.epub ./Reticulum\ Manual.epub; \
+		echo "EPUB Manual Generated"; \
+	fi

docs/manual/.buildinfo

@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: 08677a3d0e0fa273687289188ec1a603
+config: e0cd5c3608b12712050d5c10fe86c594
 tags: 645f666f9bcd5a90fca523b33c5a78b7

docs/manual/_sources/examples.rst.txt

@@ -92,6 +92,28 @@ The *Request* example explores sendig requests and receiving responses.
 
 This example can also be found at `<https://github.com/markqvist/Reticulum/blob/master/Examples/Request.py>`_.
 
+.. _example-channel:
+
+Channel
+=======
+
+The *Channel* example explores using a ``Channel`` to send structured
+data between peers of a ``Link``.
+
+.. literalinclude:: ../../Examples/Channel.py
+
+This example can also be found at `<https://github.com/markqvist/Reticulum/blob/master/Examples/Channel.py>`_.
+
+Buffer
+======
+
+The *Buffer* example explores using buffered readers and writers to send
+binary data between peers of a ``Link``.
+
+.. literalinclude:: ../../Examples/Buffer.py
+
+This example can also be found at `<https://github.com/markqvist/Reticulum/blob/master/Examples/Buffer.py>`_.
+
 .. _example-filetransfer:
 
 Filetransfer
@@ -103,4 +125,18 @@ interface to efficiently pass files of any size over a Reticulum :ref:`Link<api-
 
 .. literalinclude:: ../../Examples/Filetransfer.py
 
-This example can also be found at `<https://github.com/markqvist/Reticulum/blob/master/Examples/Filetransfer.py>`_.
+This example can also be found at `<https://github.com/markqvist/Reticulum/blob/master/Examples/Filetransfer.py>`_.
+
+.. _example-custominterface:
+
+Custom Interfaces
+=================
+
+The *ExampleInterface* demonstrates creating custom interfaces for Reticulum.
+Any number of custom interfaces can be loaded and utilised by Reticulum, and
+will be fully on-par with natively included interfaces, including all supported
+:ref:`interface modes<interfaces-modes>` and :ref:`common configuration options<interfaces-options>`.
+
+.. literalinclude:: ../../Examples/ExampleInterface.py
+
+This example can also be found at `<https://github.com/markqvist/Reticulum/blob/master/Examples/ExampleInterface.py>`_.

docs/manual/_sources/forhumans.rst.txt

@@ -0,0 +1,4 @@
+********************************************
+An Explanation of Reticulum for Human Beings
+********************************************
+

docs/manual/_sources/gettingstartedfast.rst.txt

@@ -7,22 +7,78 @@ you want to do. This guide will outline sensible starting paths for different
 scenarios.
 
 
+Standalone Reticulum Installation
+=============================================
+If you simply want to install Reticulum and related utilities on a system,
+the easiest way is via the ``pip`` package manager:
+
+.. code::
+
+   pip install rns
+
+If you do not already have pip installed, you can install it using the package manager
+of your system with a command like ``sudo apt install python3-pip``,
+``sudo pamac install python-pip`` or similar.
+
+You can also dowload the Reticulum release wheels from GitHub, or other release channels,
+and install them offline using ``pip``:
+
+.. code::
+
+   pip install ./rns-0.5.1-py3-none-any.whl
+
+For more detailed installation instructions, please see the
+:ref:`Platform-Specific Install Notes<install-guides>` section.
+
+After installation is complete, it might be helpful to refer to the
+:ref:`Using Reticulum on Your System<using-main>` chapter.
+
+Resolving Dependency & Installation Issues
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+On some platforms, there may not be binary packages available for all dependencies, and
+``pip`` installation may fail with an error message. In these cases, the issue can usually
+be resolved by installing the development essentials packages for your platform:
+
+.. code::
+
+    # Debian / Ubuntu / Derivatives
+    sudo apt install build-essential
+
+    # Arch / Manjaro / Derivatives
+    sudo pamac install base-devel
+
+    # Fedora
+    sudo dnf groupinstall "Development Tools" "Development Libraries"
+
+With the base development packages installed, ``pip`` should be able to compile any missing
+dependencies from source, and complete installation even on platforms that don't have pre-
+compiled packages available.
+
 Try Using a Reticulum-based Program
 =============================================
 
 If you simply want to try using a program built with Reticulum, a few different
-programs exist that allow basic communication and a range of other useful functions
-over even extremely low-bandwidth Reticulum networks.
+programs exist that allow basic communication and a range of other useful functions,
+even over extremely low-bandwidth Reticulum networks.
 
 These programs will let you get a feel for how Reticulum works. They have been designed
-to run well over networks based on LoRa or packet radio, but can also be used completely
-over local WiFi, wired Ethernet, the Internet, or any combination.
+to run well over networks based on LoRa or packet radio, but can also be used over fast
+links, such as local WiFi, wired Ethernet, the Internet, or any combination.
 
 As such, it is easy to get started experimenting, without having to set up any radio
 transceivers or infrastructure just to try it out. Launching the programs on separate
 devices connected to the same WiFi network is enough to get started, and physical
 radio interfaces can then be added later.
 
+Remote Shell
+^^^^^^^^^^^^
+
+The `rnsh <https://github.com/acehoss/rnsh>`_ program lets you establish fully interactive
+remote shell sessions over Reticulum. It also allows you to pipe any program to or from a
+remote system, and is similar to how ``ssh`` works. The ``rnsh`` is very efficient, and
+can facilitate fully interactive shell sessions, even over extremely low-bandwidth links,
+such as LoRa or packet radio.
+
 Nomad Network
 ^^^^^^^^^^^^^
 
@@ -44,29 +100,63 @@ You can install Nomad Network via pip:
 .. code::
 
    # Install ...
-   pip3 install nomadnet
+   pip install nomadnet
 
    # ... and run
    nomadnet
 
-**Please Note**: If this is the very first time you use pip to install a program
-on your system, you might need to reboot your system for your program to become
-available. If you get a "command not found" error or similar when running the
-program, reboot your system and try again.
+.. note::
+   If this is the very first time you use ``pip`` to install a program
+   on your system, you might need to reboot your system for your program to become
+   available. If you get a "command not found" error or similar when running the
+   program, reboot your system and try again. In some cases, you may even need to
+   manually add the ``pip`` install path to your ``PATH`` environment variable.
 
 Sideband
 ^^^^^^^^
 
 If you would rather use a program with a graphical user interface, you can take
 a look at `Sideband <https://unsigned.io/sideband>`_, which is available for Android,
-Linux and macOS.
+Linux, macOS and Windows.
 
-.. image:: screenshots/sideband_1.png
-    :align: center
-    :target: _images/sideband_1.png
+.. only:: html
 
-Sideband is currently in the early stages of development, but already provides basic
-communication features, and interoperates with Nomad Network, or any other LXMF client.
+  .. image:: screenshots/sideband_devices.webp
+      :align: center
+      :target: _images/sideband_devices.webp
+
+.. only:: latexpdf
+
+  .. image:: screenshots/sideband_devices.png
+      :align: center
+      :target: _images/sideband_devices.png
+
+Sideband allows you to communicate with other people or LXMF-compatible
+systems over Reticulum networks using LoRa, Packet Radio, WiFi, I2P, Encrypted QR
+Paper Messages, or anything else Reticulum supports. It also interoperates with
+the Nomad Network program.
+
+MeshChat
+^^^^^^^^
+
+The `Reticulum MeshChat <https://github.com/liamcottle/reticulum-meshchat>`_ application
+is a user-friendly LXMF client for macOS and Windows, that also includes voice call
+functionality, and a range of other interesting functions.
+
+.. only:: html
+
+  .. image:: screenshots/meshchat_1.webp
+      :align: center
+      :target: _images/meshchat_1.webp
+
+.. only:: latexpdf
+
+  .. image:: screenshots/meshchat_1.png
+      :align: center
+      :target: _images/meshchat_1.png
+
+Reticulum MeshChat is of course also compatible with Sideband and Nomad Network, or
+any other LXMF client.
 
 Using the Included Utilities
 =============================================
@@ -86,8 +176,8 @@ Creating a Network With Reticulum
 =============================================
 To create a network, you will need to specify one or more *interfaces* for
 Reticulum to use. This is done in the Reticulum configuration file, which by
-default is located at ``~/.reticulum/config``. You can edit this file by hand,
-or use the interactive ``rnsconfig`` utility. 
+default is located at ``~/.reticulum/config``. You can get an example
+configuration file with all options via ``rnsd --exampleconfig``.
 
 When Reticulum is started for the first time, it will create a default
 configuration file, with one active interface. This default interface uses
@@ -120,29 +210,29 @@ and :ref:`Interfaces<interfaces-main>` chapters of this manual.
 Connecting Reticulum Instances Over the Internet
 ================================================
 Reticulum currently offers two interfaces suitable for connecting instances over the Internet: :ref:`TCP<interfaces-tcps>`
-and :ref:`I2P<interfaces-i2p>`. Each interface offers a different set of features, and Reticulum 
-users should carefully choose the interface which best suites their needs. 
+and :ref:`I2P<interfaces-i2p>`. Each interface offers a different set of features, and Reticulum
+users should carefully choose the interface which best suites their needs.
 
 The ``TCPServerInterface`` allows users to host an instance accessible over TCP/IP. This
 method is generally faster, lower latency, and more energy efficient than using ``I2PInterface``,
 however it also leaks more data about the server host.
 
 TCP connections reveal the IP address of both your instance and the server to anyone who can
-inspect the connection. Someone could use this information to determine your location or identity. Adversaries 
+inspect the connection. Someone could use this information to determine your location or identity. Adversaries
 inspecting your packets may be able to record packet metadata like time of transmission and packet size.
 Even though Reticulum encrypts traffic, TCP does not, so an adversary may be able to use
 packet inspection to learn that a system is running Reticulum, and what other IP addresses connect to it.
 Hosting a publicly reachable instance over TCP also requires a publicly reachable IP address,
 which most Internet connections don't offer anymore.
 
-The ``I2PInterface`` routes messages through the `Invisible Internet Protocol 
+The ``I2PInterface`` routes messages through the `Invisible Internet Protocol
 (I2P) <https://geti2p.net/en/>`_. To use this interface, users must also run an I2P daemon in
-parallel to ``rnsd``. For always-on I2P nodes it is recommended to use `i2pd <https://i2pd.website/>`_. 
+parallel to ``rnsd``. For always-on I2P nodes it is recommended to use `i2pd <https://i2pd.website/>`_.
 
-By default, I2P will encrypt and mix all traffic sent over the Internet, and 
-hide both the sender and receiver Reticulum instance IP addresses. Running an I2P node 
+By default, I2P will encrypt and mix all traffic sent over the Internet, and
+hide both the sender and receiver Reticulum instance IP addresses. Running an I2P node
 will also relay other I2P user's encrypted packets, which will use extra
-bandwidth and compute power, but also makes timing attacks and other forms of 
+bandwidth and compute power, but also makes timing attacks and other forms of
 deep-packet-inspection much more difficult.
 
 I2P also allows users to host globally available Reticulum instances from non-public IP's and behind firewalls and NAT.
@@ -160,30 +250,37 @@ by adding one of the following interfaces to your ``.reticulum/config`` file:
 
 .. code::
 
-  # TCP/IP interface to the Dublin hub
-  [[RNS Testnet Dublin]]
+  # TCP/IP interface to the RNS Amsterdam Hub
+  [[RNS Testnet Amsterdam]]
     type = TCPClientInterface
     enabled = yes
-    target_host = dublin.connect.reticulum.network
+    target_host = amsterdam.connect.reticulum.network
     target_port = 4965
 
-  # TCP/IP interface to the Frankfurt hub
-  [[RNS Testnet Dublin]]
+  # TCP/IP interface to the BetweenTheBorders Hub (community-provided)
+  [[RNS Testnet BetweenTheBorders]]
     type = TCPClientInterface
     enabled = yes
-    target_host = frankfurt.connect.reticulum.network
-    target_port = 5377
+    target_host = reticulum.betweentheborders.com
+    target_port = 4242
 
-  # Interface to I2P hub A
-  [[RNS Testnet I2P Hub A]]
+  # Interface to Testnet I2P Hub
+  [[RNS Testnet I2P Hub]]
     type = I2PInterface
     enabled = yes
-    peers = uxg5kubabakh3jtnvsipingbr5574dle7bubvip7llfvwx2tgrua.b32.i2p
+    peers = g3br23bvx3lq5uddcsjii74xgmn6y5q325ovrkq2zw2wbzbqgbuq.b32.i2p
 
 Many other Reticulum instances are connecting to this testnet, and you can also join it
 via other entry points if you know them. There is absolutely no control over the network
 topography, usage or what types of instances connect. It will also occasionally be used
 to test various failure scenarios, and there are no availability or service guarantees.
+Expect weird things to happen on this network, as people experiment and try out things.
+
+It probably goes without saying, but *don't use the testnet entry-points as 
+hardcoded or default interfaces in any applications you ship to users*. When
+shipping applications, the best practice is to provide your own default
+connectivity solutions, if needed and applicable, or in most cases, simply
+leave it up to the user which networks to connect to, and how.
 
 
 Adding Radio Interfaces
@@ -205,7 +302,7 @@ chapter for a guide. If you prefer purchasing a ready-made unit, you can refer t
 refer to these additional external resources:
 
 * `How To Make Your Own RNodes <https://unsigned.io/how-to-make-your-own-rnodes/>`_
-* `Installing RNode Firmware on Compatible LoRa Devices <https://unsigned.io/installing-rnode-firmware-on-t-beam-and-lora32-devices/>`_
+* `Installing RNode Firmware on Compatible LoRa Devices <https://unsigned.io/installing-rnode-firmware-on-supported-devices/>`_
 * `Private, Secure and Uncensorable Messaging Over a LoRa Mesh <https://unsigned.io/private-messaging-over-lora/>`_
 * `RNode Firmware <https://github.com/markqvist/RNode_Firmware/>`_
 
@@ -215,6 +312,20 @@ you are welcome to head over to the `GitHub discussion pages <https://github.com
 and propose adding an interface for the hardware.
 
 
+Creating and Using Custom Interfaces
+===========================================
+
+While Reticulum includes a flexible and broad range of built-in interfaces, these
+will not cover every conceivable type of communications hardware that Reticulum
+can potentially use to communicate.
+
+It is therefore possible to easily write your own interface modules, that can be
+loaded at run-time and used on-par with any of the built-in interface types.
+
+For more information on this subject, and code examples to build on, please see
+the :ref:`Configuring Interfaces<interfaces-main>` chapter.
+
+
 Develop a Program with Reticulum
 ===========================================
 If you want to develop programs that use Reticulum, the easiest way to get
@@ -222,20 +333,14 @@ started is to install the latest release of Reticulum via pip:
 
 .. code::
 
-   pip3 install rns
+   pip install rns
 
 The above command will install Reticulum and dependencies, and you will be
 ready to import and use RNS in your own programs. The next step will most
 likely be to look at some :ref:`Example Programs<examples-main>`.
 
-For extended functionality, you can install optional dependencies:
-
-.. code::
-
-   pip3 install pyserial netifaces
-
-
-Further information can be found in the :ref:`API Reference<api-main>`.
+The entire Reticulum API is documented in the :ref:`API Reference<api-main>`
+chapter of this manual.
 
 
 Participate in Reticulum Development
@@ -247,7 +352,7 @@ don't use pip, but try this recipe:
 .. code::
 
     # Install dependencies
-    pip3 install cryptography pyserial netifaces
+    pip install cryptography pyserial
 
     # Clone repository
     git clone https://github.com/markqvist/Reticulum.git
@@ -257,48 +362,42 @@ don't use pip, but try this recipe:
     ln -s ../RNS ./Examples/
 
     # Run an example
-    python3 Examples/Echo.py -s
+    python Examples/Echo.py -s
 
     # Unless you've manually created a config file, Reticulum will do so now,
     # and immediately exit. Make any necessary changes to the file:
     nano ~/.reticulum/config
 
     # ... and launch the example again.
-    python3 Examples/Echo.py -s
+    python Examples/Echo.py -s
 
     # You can now repeat the process on another computer,
     # and run the same example with -h to get command line options.
-    python3 Examples/Echo.py -h
+    python Examples/Echo.py -h
 
     # Run the example in client mode to "ping" the server.
     # Replace the hash below with the actual destination hash of your server.
-    python3 Examples/Echo.py 3e12fc71692f8ec47bc5
+    python Examples/Echo.py 174a64852a75682259ad8b921b8bf416
 
     # Have a look at another example
-    python3 Examples/Filetransfer.py -h
+    python Examples/Filetransfer.py -h
 
 When you have experimented with the basic examples, it's time to go read the
-:ref:`Understanding Reticulum<understanding-main>` chapter.
+:ref:`Understanding Reticulum<understanding-main>` chapter. Before submitting
+your first pull request, it is probably a good idea to introduce yourself on
+the `disucssion forum on GitHub <https://github.com/markqvist/Reticulum/discussions>`_,
+or ask one of the developers or maintainers for a good place to start.
 
+.. _install-guides:
 
-Reticulum on ARM64
+Platform-Specific Install Notes
 ==============================================
-On some architectures, including ARM64, not all dependencies have precompiled
-binaries. On such systems, you will need to install ``python3-dev`` before
-installing Reticulum or programs that depend on Reticulum.
 
-.. code::
+Some platforms require a slightly different installation procedure, or have
+various quirks that are worth being aware of. These are listed here.
 
-   # Install Python and development packages
-   sudo apt update
-   sudo apt install python3 python3-pip python3-dev
-
-   # Install Reticulum
-   python3 -m pip install rns
-
-
-Reticulum on Android
-==============================================
+Android
+^^^^^^^^^^^^^^^^^^^^^^^^
 Reticulum can be used on Android in different ways. The easiest way to get
 started is using an app like `Sideband <https://unsigned.io/sideband>`_.
 
@@ -310,12 +409,31 @@ Termux is a terminal emulator and Linux environment for Android based devices,
 which includes the ability to use many different programs and libraries,
 including Reticulum.
 
-Since the Python cryptography.io module does not offer pre-built wheels for
-Android, the standard one-line install of Reticulum does not work on Android,
-and a few extra commands are required.
+To use Reticulum within the Termux environment, you will need to install
+``python`` and the ``python-cryptography`` library using ``pkg``, the package-manager
+build into Termux. After that, you can use ``pip`` to install Reticulum.
 
 From within Termux, execute the following:
 
+.. code::
+
+    # First, make sure indexes and packages are up to date.
+    pkg update
+    pkg upgrade
+
+    # Then install python and the cryptography library.
+    pkg install python python-cryptography
+
+    # Make sure pip is up to date, and install the wheel module.
+    pip install wheel pip --upgrade
+
+    # Install Reticulum
+    pip install rns
+
+If for some reason the ``python-cryptography`` package is not available for
+your platform via the Termux package manager, you can attempt to build it
+locally on your device using the following command:
+
 .. code::
 
     # First, make sure indexes and packages are up to date.
@@ -326,7 +444,7 @@ From within Termux, execute the following:
     pkg install python build-essential openssl libffi rust
 
     # Make sure pip is up to date, and install the wheel module.
-    pip3 install wheel pip --upgrade
+    pip install wheel pip --upgrade
 
     # To allow the installer to build the cryptography module,
     # we need to let it know what platform we are compiling for:
@@ -335,24 +453,290 @@ From within Termux, execute the following:
     # Start the install process for the cryptography module.
     # Depending on your device, this can take several minutes,
     # since the module must be compiled locally on your device.
-    pip3 install cryptography
+    pip install cryptography
 
     # If the above installation succeeds, you can now install
     # Reticulum and any related software
-    pip3 install rns
+    pip install rns
 
 It is also possible to include Reticulum in apps compiled and distributed as
 Android APKs. A detailed tutorial and example source code will be included
-here at a later point.
+here at a later point. Until then you can use the `Sideband source code <https://github.com/markqvist/sideband>`_ as an example and starting point.
+
+
+ARM64
+^^^^^^^^^^^^^^^^^^^^^^^^
+On some architectures, including ARM64, not all dependencies have precompiled
+binaries. On such systems, you may need to install ``python3-dev`` (or similar) before
+installing Reticulum or programs that depend on Reticulum.
+
+.. code::
+
+   # Install Python and development packages
+   sudo apt update
+   sudo apt install python3 python3-pip python3-dev
+
+   # Install Reticulum
+   python3 -m pip install rns
+
+With these packages installed, ``pip`` will be able to build any missing dependencies
+on your system locally.
+
+
+Debian Bookworm
+^^^^^^^^^^^^^^^^^^^^^^^^
+On versions of Debian released after April 2023, it is no longer possible by default
+to use ``pip`` to install packages onto your system. Unfortunately, you will need to
+use the replacement ``pipx`` command instead, which places installed packages in an
+isolated environment. This should not negatively affect Reticulum, but will not work
+for including and using Reticulum in your own scripts and programs.
+
+.. code::
+
+    # Install pipx
+    sudo apt install pipx
+
+    # Make installed programs available on the command line
+    pipx ensurepath
+
+    # Install Reticulum
+    pipx install rns
+
+Alternatively, you can restore normal behaviour to ``pip`` by creating or editing
+the configuration file located at ``~/.config/pip/pip.conf``, and adding the
+following section:
+
+.. code:: text
+
+    [global]
+    break-system-packages = true
+
+For a one-shot installation of Reticulum, without globally enabling the ``break-system-packages``
+option, you can use the following command:
+
+.. code:: text
+
+    pip install rns --break-system-packages
+
+.. note::
+   The ``--break-system-packages`` directive is a somewhat misleading choice
+   of words. Setting it will of course not break any system packages, but will simply
+   allow installing ``pip`` packages user- and system-wide. While this *could* in rare
+   cases lead to version conflicts, it does not generally pose any problems, especially
+   not in the case of installing Reticulum.
+
+
+MacOS
+^^^^^^^^^^^^^^^^^^^^^^^^^
+To install Reticulum on macOS, you will need to have Python and the ``pip`` package
+manager installed.
+
+Systems running macOS can vary quite widely in whether or not Python is pre-installed,
+and if it is, which version is installed, and whether the ``pip`` package manager is
+also installed and set up. If in doubt, you can `download and install <https://www.python.org/downloads/>`_
+Python manually.
+
+When Python and ``pip`` is available on your system, simply open a terminal window
+and use one of the following commands:
+
+.. code::
+
+   # Install Reticulum and utilities with pip:
+   pip3 install rns
+   
+   # On some versions, you may need to use the
+   # flag --break-system-packages to install:
+   pip3 install rns --break-system-packages
+
+.. note::
+   The ``--break-system-packages`` directive is a somewhat misleading choice
+   of words. Setting it will of course not break any system packages, but will simply
+   allow installing ``pip`` packages user- and system-wide. While this *could* in rare
+   cases lead to version conflicts, it does not generally pose any problems, especially
+   not in the case of installing Reticulum.
+
+Additionally, some version combinations of macOS and Python require you to
+manually add your installed ``pip`` packages directory to your `PATH` environment
+variable, before you can use installed commands in your terminal. Usually, adding
+the following line to your shell init script (for example ``~/.zshrc``) will be enough:
+
+.. code::
+
+   export PATH=$PATH:~/Library/Python/3.9/bin
+
+Adjust Python version and shell init script location according to your system.
+
+
+OpenWRT
+^^^^^^^^^^^^^^^^^^^^^^^^^
+On OpenWRT systems with sufficient storage and memory, you can install
+Reticulum and related utilities using the `opkg` package manager and `pip`.
+
+.. note::
+
+   At the time of releasing this manual, work is underway to create pre-built
+   Reticulum packages for OpenWRT, with full configuration, service
+   and ``uci`` integration. Please see the `feed-reticulum <https://github.com/gretel/feed-reticulum>`_
+   and `reticulum-openwrt <https://github.com/gretel/reticulum-openwrt>`_
+   repositories for more information.
+
+To install Reticulum on OpenWRT, first log into a command line session, and
+then use the following instructions:
+
+.. code::
+
+   # Install dependencies
+   opkg install python3 python3-pip python3-cryptography python3-pyserial
+   
+   # Install Reticulum
+   pip install rns
+
+   # Start rnsd with debug logging enabled
+   rnsd -vvv
+
+.. note::
+   
+   The above instructions have been verified and tested on OpenWRT 21.02 only.
+   It is likely that other versions may require slightly altered installation
+   commands or package names. You will also need enough free space in your
+   overlay FS, and enough free RAM to actually run Reticulum and any related
+   programs and utilities.
+
+Depending on your device configuration, you may need to adjust firewall rules
+for Reticulum connectivity to and from your device to work. Until proper
+packaging is ready, you will also need to manually create a service or startup
+script to automatically laucnh Reticulum at boot time.
+
+Please also note that the `AutoInterface` requires link-local IPv6 addresses
+to be enabled for any Ethernet and WiFi devices you intend to use. If ``ip a``
+shows an address starting with ``fe80::`` for the device in question,
+``AutoInterface`` should work for that device.
+
+Raspberry Pi
+^^^^^^^^^^^^^^^^^^^^^^^^^
+It is currently recommended to use a 64-bit version of the Raspberry Pi OS
+if you want to run Reticulum on Raspberry Pi computers, since 32-bit versions
+don't always have packages available for some dependencies. If Python and the
+`pip` package manager is not already installed, do that first, and then
+install Reticulum using `pip`.
+
+.. code::
+
+   # Install dependencies
+   sudo apt install python3 python3-pip python3-cryptography python3-pyserial
+   
+   # Install Reticulum
+   pip install rns --break-system-packages
+
+.. note::
+   The ``--break-system-packages`` directive is a somewhat misleading choice
+   of words. Setting it will of course not break any system packages, but will simply
+   allow installing ``pip`` packages user- and system-wide. While this *could* in rare
+   cases lead to version conflicts, it does not generally pose any problems, especially
+   not in the case of installing Reticulum.
+
+While it is possible to install and run Reticulum on 32-bit Rasperry Pi OSes,
+it will require manually configuring and installing required build dependencies,
+and is not detailed in this manual.
+
+
+RISC-V
+^^^^^^^^^^^^^^^^^^^^^^^^
+On some architectures, including RISC-V, not all dependencies have precompiled
+binaries. On such systems, you may need to install ``python3-dev`` (or similar) before
+installing Reticulum or programs that depend on Reticulum.
+
+.. code::
+
+   # Install Python and development packages
+   sudo apt update
+   sudo apt install python3 python3-pip python3-dev
+
+   # Install Reticulum
+   python3 -m pip install rns
+
+With these packages installed, ``pip`` will be able to build any missing dependencies
+on your system locally.
+
+
+Ubuntu Lunar
+^^^^^^^^^^^^^^^^^^^^^^^^
+On versions of Ubuntu released after April 2023, it is no longer possible by default
+to use ``pip`` to install packages onto your system. Unfortunately, you will need to
+use the replacement ``pipx`` command instead, which places installed packages in an
+isolated environment. This should not negatively affect Reticulum, but will not work
+for including and using Reticulum in your own scripts and programs.
+
+.. code::
+
+    # Install pipx
+    sudo apt install pipx
+
+    # Make installed programs available on the command line
+    pipx ensurepath
+
+    # Install Reticulum
+    pipx install rns
+
+Alternatively, you can restore normal behaviour to ``pip`` by creating or editing
+the configuration file located at ``~/.config/pip/pip.conf``, and adding the
+following section:
+
+.. code:: text
+
+    [global]
+    break-system-packages = true
+
+For a one-shot installation of Reticulum, without globally enabling the ``break-system-packages``
+option, you can use the following command:
+
+.. code:: text
+
+    pip install rns --break-system-packages
+
+.. note::
+   The ``--break-system-packages`` directive is a somewhat misleading choice
+   of words. Setting it will of course not break any system packages, but will simply
+   allow installing ``pip`` packages user- and system-wide. While this *could* in rare
+   cases lead to version conflicts, it does not generally pose any problems, especially
+   not in the case of installing Reticulum.
+
+Windows
+^^^^^^^^^^^^^^^^^^^^^^^^^
+On Windows operating systems, the easiest way to install Reticulum is by using the
+``pip`` package manager from the command line (either the command prompt or Windows
+Powershell).
+
+If you don't already have Python installed, `download and install Python <https://www.python.org/downloads/>`_.
+At the time of publication of this manual, the recommended version is `Python 3.12.7 <https://www.python.org/downloads/release/python-3127>`_.
+
+**Important!** When asked by the installer, make sure to add the Python program to
+your PATH environment variables. If you don't do this, you will not be able to
+use the ``pip`` installer, or run the included Reticulum utility programs (such as
+``rnsd`` and ``rnstatus``) from the command line.
+
+After installing Python, open the command prompt or Windows Powershell, and type:
+
+.. code::
+
+   pip install rns
+
+You can now use Reticulum and all included utility programs directly from your
+preferred command line interface.
 
 Pure-Python Reticulum
 ==============================================
-In some rare cases, and on more obscure system types, it is not possible to
-install one or more dependencies
 
-On more unusual systems, and in some rare cases, it might not be possible to
-install or even compile one or more of the above modules. In such situations,
-you can use the ``rnspure`` package instead of the ``rns`` package. The ``rnspure``
+.. warning::
+   If you use the ``rnspure`` package to run Reticulum on systems that
+   do not support `PyCA/cryptography <https://github.com/pyca/cryptography>`_, it is
+   important that you read and understand the :ref:`Cryptographic Primitives <understanding-primitives>`
+   section of this manual.
+
+In some rare cases, and on more obscure system types, it is not possible to
+install one or more dependencies. In such situations,
+you can use the ``rnspure`` package instead of the ``rns`` package, or use ``pip``
+with the ``--no-dependencies`` command-line option. The ``rnspure``
 package requires no external dependencies for installation. Please note that the
 actual contents of the ``rns`` and ``rnspure`` packages are *completely identical*.
 The only difference is that the ``rnspure`` package lists no dependencies required
@@ -363,8 +747,3 @@ only if they are *needed* and *available*. If for example you want to use Reticu
 on a system that cannot support ``pyserial``, it is perfectly possible to do so using
 the `rnspure` package, but Reticulum will not be able to use serial-based interfaces.
 All other available modules will still be loaded when needed.
-
-**Please Note!** If you use the `rnspure` package to run Reticulum on systems that
-do not support `PyCA/cryptography <https://github.com/pyca/cryptography>`_, it is
-important that you read and understand the :ref:`Cryptographic Primitives <understanding-primitives>`
-section of this manual.

docs/manual/_sources/hardware.rst.txt

@@ -24,11 +24,20 @@ starting from scratch.
 This chapter will outline a few different sensible starting paths to get
 real-world functional wireless communications up and running with minimal cost
 and effort. Two fundamental devices categories will be covered, *RNodes* and
-*WiFi-based radios*.
+*WiFi-based radios*. Additionally, other common options will be briefly described.
+
+Knowing how to employ just a few different types of hardware will make it possible
+to build a wide range of useful networks with little effort.
+
+Combining Hardware Types
+========================
+
+It is useful to combine different link and hardware types when designing and
+building a network. One useful design pattern is to employ high-capacity point-to-point
+links based on WiFi or millimeter-wave radios (with high-gain directional antennas)
+for the network backbone, and using LoRa-based RNodes for covering large areas with
+connectivity for client devices.
 
-While there are many other device categories that are useful in building Reticulum
-networks, knowing how to employ just these two will make it possible to build
-a wide range of useful networks with little effort.
 
 .. _rnode-main:
 
@@ -66,8 +75,8 @@ completely from scratch, to your exact desired specifications, this chapter
 will explain the easiest possible approach to creating RNodes: Using common
 LoRa development boards. This approach can be boiled down to two simple steps:
 
-1. Obtain one or more supported development boards
-2. Install the RNode firmware with the automated installer
+1. Obtain one or more :ref:`supported development boards<rnode-supported>`
+2. Install the RNode firmware with the :ref:`automated installer<rnode-installation>`
 
 Once the firmware has been installed and provisioned by the install script, it
 is ready to use with any software that supports RNodes, including Reticulum.
@@ -76,82 +85,156 @@ to the configuration.
 
 .. _rnode-supported:
 
-Supported Boards
-^^^^^^^^^^^^^^^^
+Supported Boards and Devices
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 To create one or more RNodes, you will need to obtain supported development
 boards. The following boards are supported by the auto-installer.
 
-LilyGO LoRa32 v2.1
-""""""""""""""""""
-.. image:: graphics/board_t3v21.png
-    :width: 46%
+------------
+
+.. image:: graphics/board_tbeam_supreme.png
+    :width: 75%
     :align: center
 
-- **Supported Firmware Lines** v1.x & v2.x
-- **Transceiver IC** Semtech SX1276
-- **Device Platform** ESP32
-- **Manufacturer** `LilyGO <https://lilygo.cn>`_
-
-
-LilyGO LoRa32 v2.0
-""""""""""""""""""
-.. image:: graphics/board_t3v20.png
-    :width: 46%
-    :align: center
-
-- **Supported Firmware Lines** v1.x & v2.x
-- **Transceiver IC** Semtech SX1276
-- **Device Platform** ESP32
-- **Manufacturer** `LilyGO <https://lilygo.cn>`_
-
-
-LilyGO T-Beam
+LilyGO T-Beam Supreme
 """""""""""""
+- **Transceiver IC** Semtech SX1262, SX1268
+- **Device Platform** ESP32
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
+
+------------
+
 .. image:: graphics/board_tbeam.png
     :width: 75%
     :align: center
 
-- **Supported Firmware Lines** v1.x & v2.x
-- **Transceiver IC** Semtech SX1276
+LilyGO T-Beam
+"""""""""""""
+- **Transceiver IC** Semtech SX1262, SX1268, SX1276 and SX1278
 - **Device Platform** ESP32
 - **Manufacturer** `LilyGO <https://lilygo.cn>`_
 
+------------
 
-Heltec LoRa32 v2.0
-""""""""""""""""""
-.. image:: graphics/board_heltec32.png
-    :width: 58%
+.. image:: graphics/board_t3s3.png
+    :width: 50%
     :align: center
 
-- **Supported Firmware Lines** v1.x & v2.x
-- **Transceiver IC** Semtech SX1276
+LilyGO T3S3
+"""""""""""
+- **Transceiver IC** Semtech SX1262, SX1268, SX1276 and SX1278
 - **Device Platform** ESP32
-- **Manufacturer** `Heltec Automation <https://heltec.org>`_
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
 
+------------
+
+.. image:: graphics/board_rak4631.png
+    :width: 45%
+    :align: center
+
+RAK4631-based Boards
+""""""""""""""""""""
+- **Transceiver IC** Semtech SX1262, SX1268
+- **Device Platform** nRF52
+- **Manufacturer** `RAK Wireless <https://www.rakwireless.com>`_
+
+------------
+
+.. image:: graphics/board_rnodev2.png
+    :width: 68%
+    :align: center
 
 Unsigned RNode v2.x
 """""""""""""""""""
-.. image:: graphics/board_rnodev2.png
-    :width: 58%
-    :align: center
-
-- **Supported Firmware Lines** v1.x & v2.x
-- **Transceiver IC** Semtech SX1276
+- **Transceiver IC** Semtech SX1276 and SX1278
 - **Device Platform** ESP32
 - **Manufacturer** `unsigned.io <https://unsigned.io>`_
 
+------------
+
+.. image:: graphics/board_t3v21.png
+    :width: 46%
+    :align: center
+
+LilyGO LoRa32 v2.1
+""""""""""""""""""
+- **Transceiver IC** Semtech SX1276 and SX1278
+- **Device Platform** ESP32
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
+
+------------
+
+.. image:: graphics/board_t3v20.png
+    :width: 46%
+    :align: center
+
+LilyGO LoRa32 v2.0
+""""""""""""""""""
+- **Transceiver IC** Semtech SX1276 and SX1278
+- **Device Platform** ESP32
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
+
+------------
+
+.. image:: graphics/board_t3v10.png
+    :width: 46%
+    :align: center
+
+LilyGO LoRa32 v1.0
+""""""""""""""""""
+- **Transceiver IC** Semtech SX1276 and SX1278
+- **Device Platform** ESP32
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
+
+------------
+
+.. image:: graphics/board_tdeck.png
+    :width: 45%
+    :align: center
+
+LilyGO T-Deck
+"""""""""""""
+- **Transceiver IC** Semtech SX1262, SX1268
+- **Device Platform** ESP32
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
+
+------------
+
+.. image:: graphics/board_heltec32v30.png
+    :width: 58%
+    :align: center
+
+Heltec LoRa32 v3.0
+""""""""""""""""""
+- **Transceiver IC** Semtech SX1262 and SX1268
+- **Device Platform** ESP32
+- **Manufacturer** `Heltec Automation <https://heltec.org>`_
+
+------------
+
+.. image:: graphics/board_heltec32v20.png
+    :width: 58%
+    :align: center
+
+Heltec LoRa32 v2.0
+""""""""""""""""""
+- **Transceiver IC** Semtech SX1276 and SX1278
+- **Device Platform** ESP32
+- **Manufacturer** `Heltec Automation <https://heltec.org>`_
+
+------------
 
-Unsigned RNode v1.x
-"""""""""""""""""""
 .. image:: graphics/board_rnode.png
     :width: 50%
     :align: center
 
-- **Supported Firmware Lines** v1.x
-- **Transceiver IC** Semtech SX1276
+Unsigned RNode v1.x
+"""""""""""""""""""
+- **Transceiver IC** Semtech SX1276 and SX1278
 - **Device Platform** AVR ATmega1284p
 - **Manufacturer** `unsigned.io <https://unsigned.io>`_
 
+------------
 
 .. _rnode-installation:
 
@@ -160,12 +243,13 @@ Installation
 
 Once you have obtained compatible boards, you can install the `RNode Firmware <https://github.com/markqvist/RNode_Firmware>`_
 using the `RNode Configuration Utility <https://github.com/markqvist/rnodeconfigutil>`_.
-Make sure that ``Python3`` and ``pip`` is installed on your system, and then install
-the config utility with ``pip``:
+If you have installed Reticulum on your system, the ``rnodeconf`` program will already be
+available. If not, make sure that ``Python3`` and ``pip`` is installed on your system, and
+then install Reticulum with with ``pip``:
 
 .. code::
 
-   pip3 install rnodeconf
+   pip install rns
 
 Once installation has completed, it is time to start installing the firmware on your
 devices. Run ``rnodeconf`` in auto-install mode like so:
@@ -176,12 +260,7 @@ devices. Run ``rnodeconf`` in auto-install mode like so:
 
 The utility will guide you through the installation process by asking a series of
 questions about your hardware. Simply follow the guide, and the utility will
-auto-install and configure your devices
-
-**Important Note!** It is currently recommended to use the v1.x line of the RNode firmware,
-even though the v2.x line is available for early testing. The v2.x line should still be
-considered an experimental pre-release. Only use the v2.x firmware line if you want to test
-out the absolutely newest version, and don't care about stability.
+auto-install and configure your devices.
 
 .. _rnode-usage:
 
@@ -189,17 +268,8 @@ Usage with Reticulum
 ^^^^^^^^^^^^^^^^^^^^
 When the devices have been installed and provisioned, you can use them with Reticulum
 by adding the :ref:`relevant interface section<interfaces-rnode>` to the configuration
-file of Reticulum. For v1.x firmwares, you will have to specify all interface parameters,
-such as serial port and on-air parameters. For v2.x firmwares, you just need to specify
-the Connection ID of the RNode, and Reticulum will automatically locate and connect to the
-RNode, using the parameters stored in the RNode itself.
-
-.. _rnode-suppliers:
-
-Suppliers
-^^^^^^^^^
-Get in touch if you want to have your RNode supplier listed here, or if you want help to
-get started with producing RNodes.
+file of Reticulum. In the configuraion you can specify all interface parameters,
+such as serial port and on-air parameters.
 
 
 WiFi-based Hardware
@@ -235,11 +305,31 @@ that is relatively cheap while providing long range and high capacity for Reticu
 networks. As in all other cases, it is also possible for Reticulum to co-exist with IP
 networks running concurrently on such devices.
 
-Combining Hardware Types
-========================
+Ethernet-based Hardware
+=======================
 
-It is useful to combine different link and hardware types when designing and
-building a network. One useful design pattern is to employ high-capacity point-to-point
-links based on WiFi or millimeter-wave radios (with high-gain directional antennas)
-for the network backbone, and using LoRa-based RNodes for covering large areas with
-connectivity for client devices.
+Reticulum can run over any kind of hardware that can provide a switched Ethernet-based
+medium. This means that anything from a plain Ethernet switch, to fiber-optic systems,
+to data radios with Ethernet interfaces can be used by Reticulum.
+
+The Ethernet medium does not need to have any IP infrastructure such as DHCP servers
+or routing set up, but in case such infrastructure does exist, Reticulum will simply
+co-exist with.
+
+To use Reticulum over Ethernet-based mediums, it is generally enough to use the included
+:ref:`AutoInterface<interfaces-auto>`. This interface also works over any kind of
+virtual networking adapter, such as ``tun`` and ``tap`` devices in Linux.
+
+Serial Lines & Devices
+======================
+
+Using Reticulum over any kind of raw serial line is also possible with the
+:ref:`SerialInterface<interfaces-serial>`. This interface type is also useful for
+using Reticulum over communications hardware that provides a serial port interface.
+
+Packet Radio Modems
+===================
+
+Any packet radio modem that provides a standard KISS interface over USB, serial or TCP
+can be used with Reticulum. This includes virtual software modems such as
+`FreeDV TNC <https://github.com/xssfox/freedv-tnc>`_ and `Dire Wolf <https://github.com/wb2osz/direwolf>`_.

docs/manual/_sources/index.rst.txt

@@ -1,11 +1,16 @@
 ******************************
 Reticulum Network Stack Manual
 ******************************
+
 This manual aims to provide you with all the information you need to
 understand Reticulum, build networks or develop programs using it, or
 to participate in the development of Reticulum itself.
 
-.. only:: html
+.. only:: builder_html
+
+   This manual is also available in `PDF <https://github.com/markqvist/Reticulum/releases/latest/download/Reticulum.Manual.pdf>`_ and `EPUB <https://github.com/markqvist/Reticulum/releases/latest/download/Reticulum.Manual.epub>`_ formats.
+
+.. only:: builder_html
 
    Table Of Contents
    =================

docs/manual/_sources/interfaces.rst.txt

@@ -1,9 +1,9 @@
 
 .. _interfaces-main:
 
-********************
-Supported Interfaces
-********************
+**********************
+Configuring Interfaces
+**********************
 
 Reticulum supports using many kinds of devices as networking interfaces, and
 allows you to mix and match them in any way you choose. The number of distinct
@@ -19,6 +19,16 @@ types, have a look at the :ref:`Building Networks<networks-main>` chapter of thi
 manual.
 
 
+.. _interfaces-custom:
+
+Custom Interfaces
+=================
+
+In addition to the built-in interface types, Reticulum is **fully extensible** with
+custom, user- or community-supplied interfaces, and creating custom interface
+modules is straightforward. Please see the :ref:`custom interface<example-custominterface>`
+example for basic interface code to build upon.
+
 .. _interfaces-auto:
 
 Auto Interface
@@ -33,9 +43,20 @@ system, which should be enabled by default in almost all OSes.
 
 .. code::
 
-  # This example demonstrates a TCP server interface.
-  # It will listen for incoming connections on the
-  # specified IP address and port number.
+  # This example demonstrates a bare-minimum setup
+  # of an Auto Interface. It will allow communica-
+  # tion with all other reachable devices on all
+  # usable physical ethernet-based devices that
+  # are available on the system.
+
+  [[Default Interface]]
+    type = AutoInterface
+    interface_enabled = True
+
+  # This example demonstrates an more specifically
+  # configured Auto Interface, that only uses spe-
+  # cific physical interfaces, and has a number of
+  # other configuration options set.
   
   [[Default Interface]]
     type = AutoInterface
@@ -47,6 +68,12 @@ system, which should be enabled by default in almost all OSes.
 
     group_id = reticulum
 
+    # You can also choose the multicast address type:
+    # temporary (default, Temporary Multicast Address)
+    # or permanent (Permanent Multicast Address)
+
+    multicast_address_type = permanent
+
     # You can also select specifically which
     # kernel networking devices to use.
 
@@ -135,11 +162,12 @@ It can take anywhere from a few seconds to a few minutes to establish
 I2P connections to the desired peers, so Reticulum handles the process
 in the background, and will output relevant events to the log.
 
-**Please Note!** While the I2P interface is the simplest way to use
-Reticulum over I2P, it is also possible to tunnel the TCP server and
-client interfaces over I2P manually. This can be useful in situations
-where more control is needed, but requires manual tunnel setup through
-the I2P daemon configuration.
+.. note::
+   While the I2P interface is the simplest way to use
+   Reticulum over I2P, it is also possible to tunnel the TCP server and
+   client interfaces over I2P manually. This can be useful in situations
+   where more control is needed, but requires manual tunnel setup through
+   the I2P daemon configuration.
 
 It is important to note that the two methods are *interchangably compatible*.
 You can use the I2PInterface to connect to a TCPServerInterface that
@@ -154,7 +182,7 @@ TCP Server Interface
 ====================
 
 The TCP Server interface is suitable for allowing other peers to connect over
-the Internet or private IP networks. When a TCP server interface has been
+the Internet or private IPv4 and IPv6 networks. When a TCP server interface has been
 configured, other Reticulum peers can connect to it with a TCP Client interface.
 
 .. code::
@@ -183,8 +211,37 @@ configured, other Reticulum peers can connect to it with a TCP Client interface.
     # device = eth0
     # port = 4242
 
-**Please Note!** The TCP interfaces support tunneling over I2P, but to do so reliably,
-you must use the i2p_tunneled option:
+If you are using the interface on a device which has both IPv4 and IPv6 addresses available,
+you can use the ``prefer_ipv6`` option to bind to the IPv6 address:
+
+.. code::
+
+  # This example demonstrates a TCP server interface.
+  # It will listen for incoming connections on the
+  # specified IP address and port number.
+  
+  [[TCP Server Interface]]
+    type = TCPServerInterface
+    interface_enabled = True
+
+    device = eth0
+    port = 4242
+    prefer_ipv6 = True
+
+To use the TCP Server Interface over `Yggdrasil <https://yggdrasil-network.github.io/>`_, you
+can simply specify the Yggdrasil ``tun`` device and a listening port, like so:
+
+.. code::
+
+  [[Yggdrasil TCP Server Interface]]
+      type = TCPServerInterface
+      interface_enabled = yes
+      device = tun0
+      listen_port = 4343
+
+.. note::
+   The TCP interfaces support tunneling over I2P, but to do so reliably,
+   you must use the i2p_tunneled option:
 
 .. code::
 
@@ -214,7 +271,7 @@ and restore connectivity after a failure, once the other end of a TCP interface
 .. code::
 
   # Here's an example of a TCP Client interface. The
-  # target_host can either be an IP address or a hostname.
+  # target_host can be a hostname or an IPv4 or IPv6 address.
 
   [[TCP Client Interface]]
     type = TCPClientInterface
@@ -222,6 +279,17 @@ and restore connectivity after a failure, once the other end of a TCP interface
     target_host = 127.0.0.1
     target_port = 4242
 
+To use the TCP Client Interface over `Yggdrasil <https://yggdrasil-network.github.io/>`_, simply
+specify the target Yggdrasil IPv6 address and port, like so:
+
+.. code::
+
+  [[Yggdrasil TCP Client Interface]]
+      type = TCPClientInterface
+      interface_enabled = yes
+      target_host = 201:5d78:af73:5caf:a4de:a79f:3278:71e5
+      target_port = 4343
+
 It is also possible to use this interface type to connect via other programs
 or hardware devices that expose a KISS interface on a TCP port, for example
 software-based soundmodems. To do this, use the ``kiss_framing`` option:
@@ -245,8 +313,9 @@ never enable ``kiss_framing``, since this will disable internal reliability and
 recovery mechanisms that greatly improves performance over unreliable and
 intermittent TCP links.
 
-**Please Note!** The TCP interfaces support tunneling over I2P, but to do so reliably,
-you must use the i2p_tunneled option:
+.. note::
+   The TCP interfaces support tunneling over I2P, but to do so reliably,
+   you must use the i2p_tunneled option:
 
 .. code::
 
@@ -268,11 +337,12 @@ private and the internet. It can also allow broadcast communication
 over IP networks, so it can provide an easy way to enable connectivity
 with all other peers on a local area network.
 
-*Please Note!* Using broadcast UDP traffic has performance implications,
-especially on WiFi. If your goal is simply to enable easy communication
-with all peers in your local Ethernet broadcast domain, the
-:ref:`Auto Interface<interfaces-auto>` performs better, and is even
-easier to use.
+.. warning::
+   Using broadcast UDP traffic has performance implications,
+   especially on WiFi. If your goal is simply to enable easy communication
+   with all peers in your local Ethernet broadcast domain, the
+   :ref:`Auto Interface<interfaces-auto>` performs better, and is even
+   easier to use.
 
 .. code::
 
@@ -327,6 +397,11 @@ RNode LoRa Interface
 To use Reticulum over LoRa, the `RNode <https://unsigned.io/rnode/>`_ interface
 can be used, and offers full control over LoRa parameters.
 
+.. warning::
+   Radio frequency spectrum is a legally controlled resource, and legislation
+   varies widely around the world. It is your responsibility to be aware of any
+   relevant regulation for your location, and to make decisions accordingly.
+
 .. code::
 
   # Here's an example of how to add a LoRa interface
@@ -341,6 +416,23 @@ can be used, and offers full control over LoRa parameters.
     # Serial port for the device
     port = /dev/ttyUSB0
 
+    # It is also possible to use BLE devices
+    # instead of wired serial ports. The
+    # target RNode must be paired with the
+    # host device before connecting. BLE
+    # devices can be connected by name,
+    # BLE MAC address or by any available.
+    
+    # Connect to specific device by name
+    # port = ble://RNode 3B87
+
+    # Or by BLE MAC address
+    # port = ble://F4:12:73:29:4E:89
+
+    # Or connect to the first available,
+    # paired device
+    # port = ble://
+
     # Set frequency to 867.2 MHz
     frequency = 867200000
 
@@ -365,6 +457,7 @@ can be used, and offers full control over LoRa parameters.
     # out identification on the channel with
     # a set interval by configuring the
     # following two parameters.
+    
     # id_callsign = MYCALL-0
     # id_interval = 600
 
@@ -372,7 +465,141 @@ can be used, and offers full control over LoRa parameters.
     # with low amounts of RAM, using packet
     # flow control can be useful. By default
     # it is disabled.
-    flow_control = False
+    
+    # flow_control = False
+
+    # It is possible to limit the airtime
+    # utilisation of an RNode by using the
+    # following two configuration options.
+    # The short-term limit is applied in a
+    # window of approximately 15 seconds,
+    # and the long-term limit is enforced
+    # over a rolling 60 minute window. Both
+    # options are specified in percent.
+    
+    # airtime_limit_long = 1.5
+    # airtime_limit_short = 33
+
+
+.. _interfaces-rnode-multi:
+
+RNode Multi Interface
+=====================
+
+For RNodes that support multiple LoRa transceivers, the RNode
+Multi interface can be used to configure sub-interfaces individually.
+
+.. warning::
+   Radio frequency spectrum is a legally controlled resource, and legislation
+   varies widely around the world. It is your responsibility to be aware of any
+   relevant regulation for your location, and to make decisions accordingly.
+
+.. code::
+
+  # Here's an example of how to add an RNode Multi interface
+  # using the RNode LoRa transceiver.
+
+  [[RNode Multi Interface]]
+  type = RNodeMultiInterface
+
+  # Enable interface if you want to use it!
+  interface_enabled = True
+
+  # Serial port for the device
+  port = /dev/ttyACM0
+
+  # You can configure the RNode to send
+  # out identification on the channel with
+  # a set interval by configuring the
+  # following two parameters.
+
+  # id_callsign = MYCALL-0
+  # id_interval = 600
+
+    # A subinterface
+    [[[High Datarate]]]
+      # Subinterfaces can be enabled and disabled in of themselves
+      interface_enabled = True
+
+      # Set frequency to 2.4GHz
+      frequency = 2400000000
+
+      # Set LoRa bandwidth to 1625 KHz
+      bandwidth = 1625000
+
+      # Set TX power to 0 dBm (0.12 mW)
+      txpower = 0
+
+      # The virtual port, only the manufacturer
+      # or the person who wrote the board config
+      # can tell you what it will be for which
+      # physical hardware interface
+      vport = 1
+
+      # Select spreading factor 5. Valid
+      # range is 5 through 12, with 5 
+      # being the fastest and 12 having
+      # the longest range.
+      spreadingfactor = 5
+
+      # Select coding rate 5. Valid range
+      # is 5 throough 8, with 5 being the
+      # fastest, and 8 the longest range.
+      codingrate = 5
+
+      # It is possible to limit the airtime
+      # utilisation of an RNode by using the
+      # following two configuration options.
+      # The short-term limit is applied in a
+      # window of approximately 15 seconds,
+      # and the long-term limit is enforced
+      # over a rolling 60 minute window. Both
+      # options are specified in percent.
+
+      # airtime_limit_long = 100
+      # airtime_limit_short = 100
+
+    [[[Low Datarate]]]
+      # Subinterfaces can be enabled and disabled in of themselves
+      interface_enabled = True
+
+      # Set frequency to 865.6 MHz
+      frequency = 865600000
+
+      # The virtual port, only the manufacturer
+      # or the person who wrote the board config
+      # can tell you what it will be for which
+      # physical hardware interface
+      vport = 0
+
+      # Set LoRa bandwidth to 125 KHz
+      bandwidth = 125000
+
+      # Set TX power to 0 dBm (0.12 mW)
+      txpower = 0
+
+      # Select spreading factor 7. Valid
+      # range is 5 through 12, with 5 
+      # being the fastest and 12 having
+      # the longest range.
+      spreadingfactor = 7
+
+      # Select coding rate 5. Valid range
+      # is 5 throough 8, with 5 being the
+      # fastest, and 8 the longest range.
+      codingrate = 5
+
+      # It is possible to limit the airtime
+      # utilisation of an RNode by using the
+      # following two configuration options.
+      # The short-term limit is applied in a
+      # window of approximately 15 seconds,
+      # and the long-term limit is enforced
+      # over a rolling 60 minute window. Both
+      # options are specified in percent.
+
+      # airtime_limit_long = 100
+      # airtime_limit_short = 100
 
 .. _interfaces-serial:
 
@@ -434,6 +661,11 @@ radio modems and TNCs, including `OpenModem <https://unsigned.io/openmodem/>`_.
 KISS interfaces can also be configured to periodically send out beacons
 for station identification purposes.
 
+.. warning::
+   Radio frequency spectrum is a legally controlled resource, and legislation
+   varies widely around the world. It is your responsibility to be aware of any
+   relevant regulation for your location, and to make decisions accordingly.
+
 .. code::
 
   [[Packet Radio KISS Interface]]
@@ -497,6 +729,11 @@ encapsulate in AX.25.
 A more efficient way is to use the plain KISS interface with the
 beaconing functionality described above.
 
+.. warning::
+   Radio frequency spectrum is a legally controlled resource, and legislation
+   varies widely around the world. It is your responsibility to be aware of any
+   relevant regulation for your location, and to make decisions accordingly.
+
 .. code::
 
   [[Packet Radio AX.25 KISS Interface]]
@@ -746,3 +983,74 @@ conserve bandwidth, while very fast networks can support applications that
 need very frequent announces. Reticulum implements these mechanisms to ensure
 that a large span of network types can seamlessly *co-exist* and interconnect.
 
+.. _interfaces-ingress-control:
+
+New Destination Rate Limiting
+=============================
+
+On public interfaces, where anyone may connect and announce new destinations,
+it can be useful to control the rate at which announces for *new*  destinations are
+processed.
+
+If a large influx of announces for newly created or previously unknown destinations
+occur within a short amount of time, Reticulum will place these announces on hold,
+so that announce traffic for known and previously established destinations can
+continue to be processed without interruptions.
+
+After the burst subsides, and an additional waiting period has passed, the held
+announces will be released at a slow rate, until the hold queue is cleared. This
+also means, that should a node decide to connect to a public interface, announce
+a large amount of bogus destinations, and then disconnect, these destination will
+never make it into path tables and waste network bandwidth on retransmitted
+announces.
+
+**It's important to note** that the ingress control works at the level of *individual
+sub-interfaces*. As an example, this means that one client on a :ref:`TCP Server Interface<interfaces-tcps>`
+cannot disrupt processing of incoming announces for other connected clients on the same
+:ref:`TCP Server Interface<interfaces-tcps>`. All other clients on the same interface will still have new announces
+processed without interruption.
+
+By default, Reticulum will handle this automatically, and ingress announce
+control will be enabled on interface where it is sensible to do so. It should
+generally not be neccessary to modify the ingress control configuration,
+but all the parameters are exposed for configuration if needed.
+
+ * | The ``ingress_control`` option tells Reticulum whether or not
+     to enable announce ingress control on the interface. Defaults to
+     ``True``.
+
+ * | The ``ic_new_time`` option configures how long (in seconds) an
+     interface is considered newly spawned. Defaults to ``2*60*60`` seconds. This
+     option is useful on publicly accessible interfaces that spawn new
+     sub-interfaces when a new client connects. 
+
+ * | The ``ic_burst_freq_new`` option sets the maximum announce ingress
+     frequency for newly spawned interfaces. Defaults to ``3.5``
+     announces per second.
+
+ * | The ``ic_burst_freq`` option sets the maximum announce ingress
+     frequency for other interfaces. Defaults to ``12`` announces
+     per second.
+
+     *If an interface exceeds its burst frequency, incoming announces
+     for unknown destinations will be temporarily held in a queue, and
+     not processed until later.*
+
+ * | The ``ic_max_held_announces`` option sets the maximum amount of
+     unique announces that will be held in the queue. Any additional
+     unique announces will be dropped. Defaults to ``256`` announces.
+
+ * | The ``ic_burst_hold`` option sets how much time (in seconds) must
+     pass after the burst frequency drops below its threshold, for the
+     announce burst to be considered cleared. Defaults to ``60``
+     seconds.
+
+ * | The ``ic_burst_penalty`` option sets how much time (in seconds) must
+     pass after the burst is considered cleared, before held announces can
+     start being released from the queue. Defaults to ``5*60``
+     seconds.
+
+ * | The ``ic_held_release_interval`` option sets how much time (in seconds)
+     must pass between releasing each held announce from the queue. Defaults
+     to ``30`` seconds.
+

docs/manual/_sources/networks.rst.txt

@@ -60,7 +60,7 @@ with Reticulum:
 
  * | Reticulum is designed to work reliably in open, trustless environments. This
      means you can use it to create open-access networks, where participants can
-     join and leave in an free and unorganised manner. This property allows an
+     join and leave in a free and unorganised manner. This property allows an
      entirely new, and so far, mostly unexplored class of networked applications,
      where networks, and the information flow within them can form and dissolve
      organically.

docs/manual/_sources/reference.rst.txt

@@ -121,6 +121,76 @@ This chapter lists and explains all classes exposed by the Reticulum Network Sta
 .. autoclass:: RNS.Resource(data, link, advertise=True, auto_compress=True, callback=None, progress_callback=None, timeout=None)
    :members:
 
+.. _api-channel:
+
+.. only:: html
+
+   |start-h3| Channel |end-h3|
+
+.. only:: latex
+
+   Channel
+   -------
+
+.. autoclass:: RNS.Channel.Channel()
+   :members:
+
+.. _api-messsagebase:
+
+.. only:: html
+
+   |start-h3| MessageBase |end-h3|
+
+.. only:: latex
+
+   MessageBase
+   -----------
+
+.. autoclass:: RNS.MessageBase()
+   :members:
+
+.. _api-buffer:
+
+.. only:: html
+
+   |start-h3| Buffer |end-h3|
+
+.. only:: latex
+
+   Buffer
+   ------
+
+.. autoclass:: RNS.Buffer
+   :members:
+
+.. _api-rawchannelreader:
+
+.. only:: html
+
+   |start-h3| RawChannelReader |end-h3|
+
+.. only:: latex
+
+   RawChannelReader
+   ----------------
+
+.. autoclass:: RNS.RawChannelReader
+   :members: __init__, add_ready_callback, remove_ready_callback
+
+.. _api-rawchannelwriter:
+
+.. only:: html
+
+   |start-h3| RawChannelWriter |end-h3|
+
+.. only:: latex
+
+   RawChannelWriter
+   ----------------
+
+.. autoclass:: RNS.RawChannelWriter
+   :members: __init__
+
 .. _api-transport:
 
 .. only:: html

docs/manual/_sources/support.rst.txt

@@ -32,7 +32,9 @@ Provide Feedback
 ================
 All feedback on the usage, functioning and potential dysfunctioning of any and
 all components of the system is very valuable to the continued development and
-improvement of Reticulum. Absolutely no automated analytics, telemetry, error
+improvement of Reticulum.
+
+Absolutely no automated analytics, telemetry, error
 reporting or statistics is collected and reported by Reticulum under any
 circumstances, so we rely on old-fashioned human feedback.
 

docs/manual/_sources/understanding.rst.txt

@@ -75,7 +75,7 @@ guide the design of Reticulum:
     it can be easily modified and replicated by anyone interested in doing so.
 * **Very low bandwidth requirements**
     Reticulum should be able to function reliably over links with a transmission capacity as low
-    as *500 bits per second*.
+    as *5 bits per second*.
 * **Encryption by default**
     Reticulum must use strong encryption by default for all communication.
 * **Initiator Anonymity**
@@ -107,13 +107,13 @@ guide the design of Reticulum:
 Introduction & Basic Functionality
 ==================================
 
-Reticulum is a networking stack suited for high-latency, low-bandwidth links. Reticulum is at it’s
+Reticulum is a networking stack suited for high-latency, low-bandwidth links. Reticulum is at its
 core a *message oriented* system. It is suited for both local point-to-point or point-to-multipoint
 scenarios where all nodes are within range of each other, as well as scenarios where packets need
 to be transported over multiple hops in a complex network to reach the recipient.
 
 Reticulum does away with the idea of addresses and ports known from IP, TCP and UDP. Instead
-Reticulum uses the singular concept of *destinations*. Any application using Reticulum as it’s
+Reticulum uses the singular concept of *destinations*. Any application using Reticulum as its
 networking stack will need to create one or more destinations to receive data, and know the
 destinations it needs to send data to.
 
@@ -134,10 +134,11 @@ be sufficient, even far into the future.
 By default Reticulum encrypts all data using elliptic curve cryptography and AES. Any packet sent to a
 destination is encrypted with a per-packet derived key. Reticulum can also set up an encrypted
 channel to a destination, called a *Link*. Both data sent over Links and single packets offer
-*Initiator Anonymity*, and links additionally offer *Forward Secrecy* by using an Elliptic Curve
-Diffie Hellman key exchange on Curve25519 to derive per-link ephemeral keys. The multi-hop transport,
-coordination, verification and reliability layers are fully autonomous and also based on elliptic
-curve cryptography.
+*Initiator Anonymity*. Links additionally offer *Forward Secrecy* by default, employing an Elliptic Curve
+Diffie Hellman key exchange on Curve25519 to derive per-link ephemeral keys. Asymmetric, link-less
+packet communication can also provide forward secrecy, with automatic key ratcheting, by enabling
+ratchets on a per-destination basis. The multi-hop transport, coordination, verification and reliability
+layers are fully autonomous and also based on elliptic curve cryptography.
 
 Reticulum also offers symmetric key encryption for group-oriented communications, as well as
 unencrypted packets for local broadcast purposes.
@@ -220,7 +221,7 @@ packet.
 
 In actual use of *single* destination naming, it is advisable not to use any uniquely identifying
 features in aspect naming. Aspect names should be general terms describing what kind of destination
-is represented. The uniquely identifying aspect is always achieved by the appending the public key,
+is represented. The uniquely identifying aspect is always achieved by appending the public key,
 which expands the destination into a uniquely identifiable one. Reticulum does this automatically.
 
 Any destination on a Reticulum network can be addressed and reached simply by knowing its
@@ -239,7 +240,7 @@ To recap, the different destination types should be used in the following situat
 * **Plain**
     When plain-text communication is desirable, for example when broadcasting information, or for local discovery purposes.
 
-To communicate with a *single* destination, you need to know it’s public key. Any method for
+To communicate with a *single* destination, you need to know its public key. Any method for
 obtaining the public key is valid, but Reticulum includes a simple mechanism for making other
 nodes aware of your destinations public key, called the *announce*. It is also possible to request
 an unknown public key from the network, as all transport instances serve as a distributed ledger
@@ -287,7 +288,7 @@ In Reticulum, destinations are allowed to move around the network at will. This
 protocols such as IP, where an address is always expected to stay within the network segment it was assigned in.
 This limitation does not exist in Reticulum, and any destination is *completely portable* over the entire topography
 of the network, and *can even be moved to other Reticulum networks* than the one it was created in, and
-still become reachable. To update it's reachability, a destination simply needs to send an announce on any
+still become reachable. To update its reachability, a destination simply needs to send an announce on any
 networks it is part of. After a short while, it will be globally reachable in the network.
 
 Seeing how *single* destinations are always tied to a private/public key pair leads us to the next topic.
@@ -368,7 +369,7 @@ If it is a *Transport Node*, it should be given the configuration directive ``en
 The Announce Mechanism in Detail
 --------------------------------
 
-When an *announce* for a destination is transmitted by from a Reticulum instance, it will be forwarded by
+When an *announce* for a destination is transmitted by a Reticulum instance, it will be forwarded by
 any transport node receiving it, but according to some specific rules:
 
 
@@ -385,7 +386,7 @@ any transport node receiving it, but according to some specific rules:
     announces is set at 2%, but can be configured on a per-interface basis.
 
 * | If any given interface does not have enough bandwidth available for retransmitting the announce,
-    the announce will be assigned a priority inversely proportional to it's hop count, and be inserted
+    the announce will be assigned a priority inversely proportional to its hop count, and be inserted
     into a queue managed by the interface.
 
 * | When the interface has bandwidth available for processing an announce, it will prioritise announces
@@ -431,7 +432,7 @@ For exchanges of small amounts of information, Reticulum offers the *Packet* API
 
 * | A packet is always created with an associated destination and some payload data. When the packet is sent
     to a *single* destination type, Reticulum will automatically create an ephemeral encryption key, perform
-    an ECDH key exchange with the destinations public key, and encrypt the information.
+    an ECDH key exchange with the destination's public key (or ratchet key, if available), and encrypt the information.
 
 * | It is important to note that this key exchange does not require any network traffic. The sender already
     knows the public key of the destination from an earlier received *announce*, and can thus perform the ECDH
@@ -447,8 +448,8 @@ For exchanges of small amounts of information, Reticulum offers the *Packet* API
 
 * | Once the packet has been received and decrypted by the addressed destination, that destination can opt
     to *prove* its receipt of the packet. It does this by calculating the SHA-256 hash of the received packet,
-    and signing this hash with it's Ed25519 signing key. Transport nodes in the network can then direct this
-    *proof* back to the packets origin, where the signature can be verified against the destinations known
+    and signing this hash with its Ed25519 signing key. Transport nodes in the network can then direct this
+    *proof* back to the packets origin, where the signature can be verified against the destination's known
     public signing key.
 
 * | In case the packet is addressed to a *group* destination type, the packet will be encrypted with the
@@ -465,7 +466,7 @@ For exchanges of larger amounts of data, or when longer sessions of bidirectiona
     forward the packet will take note of this *link request*.
 
 * | Second, if the destination accepts the *link request* , it will send back a packet that proves the
-    authenticity of it’s identity (and the receipt of the link request) to the initiating node. All
+    authenticity of its identity (and the receipt of the link request) to the initiating node. All
     nodes that initially forwarded the packet will also be able to verify this proof, and thus
     accept the validity of the *link* throughout the network.
 
@@ -595,7 +596,7 @@ or less any medium that allows you to send and receive data, which satisfies som
 minimum requirements.
 
 The communication channel must support at least half-duplex operation, and provide an average
-throughput of around 500 bits per second, and supports a physical layer MTU of 500 bytes. The
+throughput of 5 bits per second or greater, and supports a physical layer MTU of 500 bytes. The
 Reticulum stack should be able to run on more or less any hardware that can provide a Python 3.x 
 runtime environment.
 
@@ -693,7 +694,8 @@ Wire Format
     [HEADER 2 bytes] [ADDRESSES 16/32 bytes] [CONTEXT 1 byte] [DATA 0-465 bytes]
 
     * The HEADER field is 2 bytes long.
-      * Byte 1: [IFAC Flag], [Header Type], [Propagation Type], [Destination Type] and [Packet Type]
+      * Byte 1: [IFAC Flag], [Header Type], [Context Flag], [Propagation Type],
+                [Destination Type] and [Packet Type]
       * Byte 2: Number of hops
 
     * Interface Access Code field if the IFAC flag was set.
@@ -725,12 +727,16 @@ Wire Format
     type 2           1  Two byte header, two 16 byte address fields
 
 
+    Context Flag
+    -----------------
+    unset            0  The context flag is used for various types
+    set              1  of signalling, depending on packet context
+
+
     Propagation Types
     -----------------
-    broadcast       00
-    transport       01
-    reserved        10
-    reserved        11
+    broadcast        0
+    transport        1
 
 
     Destination Types
@@ -771,7 +777,7 @@ Wire Format
     |               | |               | |               | |    |
     00000000 00000111 [HASH1, 16 bytes] [CONTEXT, 1 byte] [DATA]
     || | | |    |
-    || | | |    +-- Hops             = 0
+    || | | |    +-- Hops             = 7
     || | | +------- Packet Type      = DATA
     || | +--------- Destination Type = SINGLE
     || +----------- Propagation Type = BROADCAST
@@ -786,7 +792,7 @@ Wire Format
     |               | |             | |               | |               | |    |
     10000000 00000111 [IFAC, N bytes] [HASH1, 16 bytes] [CONTEXT, 1 byte] [DATA]
     || | | |    |
-    || | | |    +-- Hops             = 0
+    || | | |    +-- Hops             = 7
     || | | +------- Packet Type      = DATA
     || | +--------- Destination Type = SINGLE
     || +----------- Propagation Type = BROADCAST
@@ -803,7 +809,7 @@ Wire Format
     but excluding any interface access codes.
 
     - Path Request    :    51  bytes
-    - Announce        :    157 bytes
+    - Announce        :    167 bytes
     - Link Request    :    83  bytes
     - Link Proof      :    115 bytes
     - Link RTT packet :    99  bytes
@@ -858,15 +864,21 @@ both on general-purpose CPUs and on microcontrollers. The necessary primitives a
 
 * Ed25519 for signatures
 
-* X22519 for ECDH key exchanges
+* X25519 for ECDH key exchanges
 
 * HKDF for key derivation
 
-* Fernet for encrypted tokens
+* Encrypted tokens are based on the Fernet spec
 
-  * AES-128 in CBC mode
+  * Ephemeral keys derived from an ECDH key exchange on Curve25519
 
-  * HMAC for message authentication
+  * AES-128 in CBC mode with PKCS7 padding
+
+  * HMAC using SHA256 for message authentication
+
+  * IVs are generated through os.urandom()
+
+  * No Fernet version and timestamp metadata fields
 
 * SHA-256
 
@@ -876,12 +888,12 @@ In the default installation configuration, the ``X25519``, ``Ed25519`` and ``AES
 primitives are provided by `OpenSSL <https://www.openssl.org/>`_ (via the `PyCA/cryptography <https://github.com/pyca/cryptography>`_
 package). The hashing functions ``SHA-256`` and ``SHA-512`` are provided by the standard
 Python `hashlib <https://docs.python.org/3/library/hashlib.html>`_. The ``HKDF``, ``HMAC``,
-``Fernet`` primitives, and the ``PKCS7`` padding function are always provided by the
+``Token`` primitives, and the ``PKCS7`` padding function are always provided by the
 following internal implementations:
 
 - ``RNS/Cryptography/HKDF.py``
 - ``RNS/Cryptography/HMAC.py``
-- ``RNS/Cryptography/Fernet.py``
+- ``RNS/Cryptography/Token.py``
 - ``RNS/Cryptography/PKCS7.py``
 
 
@@ -892,6 +904,7 @@ with the OpenSSL backend being *much* faster. The most important consequence how
 potential loss of security by using primitives that has not seen the same amount of scrutiny,
 testing and review as those from OpenSSL.
 
-If you want to use the internal pure-python primitives, it is **highly advisable** that you
-have a good understanding of the risks that this pose, and make an informed decision on whether
-those risks are acceptable to you.
+.. warning::
+   If you want to use the internal pure-python primitives, it is **highly advisable** that you
+   have a good understanding of the risks that this pose, and make an informed decision on whether
+   those risks are acceptable to you.

docs/manual/_sources/using.rst.txt

@@ -5,7 +5,9 @@ Using Reticulum on Your System
 ******************************
 
 Reticulum is not installed as a driver or kernel module, as one might expect
-of a networking stack. Instead, Reticulum is distributed as a Python module.
+of a networking stack. Instead, Reticulum is distributed as a Python module, 
+containing the networking core, and a set of utility and daemon programs.
+
 This means that no special privileges are required to install or use it. It
 is also very light-weight, and easy to transfer to, and install on new systems.
 
@@ -16,8 +18,8 @@ already running.
 In many cases, this approach is sufficient. When any program needs to use
 Reticulum, it is loaded, initialised, interfaces are brought up, and the
 program can now communicate over any Reticulum networks available. If another
-program starts up and also wants access to the same Reticulum network, the
-instance is simply shared. This works for any number of programs running
+program starts up and also wants access to the same Reticulum network, the already
+running instance is simply shared. This works for any number of programs running
 concurrently, and is very easy to use, but depending on your use case, there
 are other options.
 
@@ -97,6 +99,17 @@ configuration file is created. The default configuration looks like this:
   instance_control_port = 37429
 
 
+  # On systems where running instances may not have access
+  # to the same shared Reticulum configuration directory,
+  # it is still possible to allow full interactivity for
+  # running instances, by manually specifying a shared RPC
+  # key. In almost all cases, this option is not needed, but
+  # it can be useful on operating systems such as Android.
+  # The key must be specified as bytes in hexadecimal.
+
+  # rpc_key = e5c032d3ec4e64a6aca9927ba8ab73336780f6d71790
+
+
   # You can configure Reticulum to panic and forcibly close
   # if an unrecoverable interface error occurs, such as the
   # hardware device for an interface disappearing. This is
@@ -106,6 +119,17 @@ configuration file is created. The default configuration looks like this:
   panic_on_interface_error = No
 
 
+  # When Transport is enabled, it is possible to allow the
+  # Transport Instance to respond to probe requests from
+  # the rnprobe utility. This can be a useful tool to test
+  # connectivity. When this option is enabled, the probe
+  # destination will be generated from the Identity of the
+  # Transport Instance, and printed to the log at startup.
+  # Optional, and disabled by default.
+
+  respond_to_probes = No
+
+
   [logging]
   # Valid log levels are 0 through 7:
   #   0: Log only critical information
@@ -143,10 +167,19 @@ configuration file is created. The default configuration looks like this:
 If Reticulum infrastructure already exists locally, you probably don't need to
 change anything, and you may already be connected to a wider network. If not,
 you will probably need to add relevant *interfaces* to the configuration, in
-order to communicate with other systems. It is a good idea to read the comments
-and explanations in the above default config. It will teach you the basic
-concepts you need to understand to configure your network. Once you have done that,
-take a look at the :ref:`Interfaces<interfaces-main>` chapter of this manual.
+order to communicate with other systems.
+
+You can generate a much more verbose configuration example by running the command:
+
+``rnsd --exampleconfig``
+
+The output includes examples for most interface types supported
+by Reticulum, along with additional options and configuration parameters.
+
+It is a good idea to read the comments and explanations in the above default config.
+It will teach you the basic concepts you need to understand to configure your network.
+Once you have done that, take a look at the :ref:`Interfaces<interfaces-main>` chapter
+of this manual.
 
 Included Utility Programs
 -------------------------
@@ -168,28 +201,47 @@ When ``rnsd`` is running, it will keep all configured interfaces open, handle tr
 it is enabled, and allow any other programs to immediately utilise the
 Reticulum network it is configured for.
 
-You can even run multiple instances of rnsd with different configurations on
+You can even run multiple instances of ``rnsd`` with different configurations on
 the same system.
 
-.. code:: text
+**Usage Examples**
 
-  # Install Reticulum
-  pip3 install rns
-
-  # Run rnsd
-  rnsd
+Run ``rnsd``:
 
 .. code:: text
 
-  usage: rnsd [-h] [--config CONFIG] [-v] [-q] [--version]
+  $ rnsd
+
+  [2023-08-18 17:59:56] [Notice] Started rnsd version 0.5.8
+
+Run ``rnsd`` in service mode, ensuring all logging output is sent directly to file:
+
+.. code:: text
+
+  $ rnsd -s
+
+Generate a verbose and detailed configuration example, with explanations of all the
+various configuration options, and interface configuration examples:
+
+.. code:: text
+
+  $ rnsd --exampleconfig
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnsd.py [-h] [--config CONFIG] [-v] [-q] [-s] [--exampleconfig] [--version]
 
   Reticulum Network Stack Daemon
 
-  optional arguments:
+  options:
     -h, --help       show this help message and exit
     --config CONFIG  path to alternative Reticulum config directory
     -v, --verbose
     -q, --quiet
+    -s, --service    rnsd is running as a service and should log to file
+    --exampleconfig  print verbose configuration example to stdout and exit
     --version        show program's version number and exit
 
 You can easily add ``rnsd`` as an always-on service by :ref:`configuring a service<using-systemd>`.
@@ -200,12 +252,14 @@ The rnstatus Utility
 Using the ``rnstatus`` utility, you can view the status of configured Reticulum
 interfaces, similar to the ``ifconfig`` program.
 
+**Usage Examples**
+
+Run ``rnstatus``:
+
 .. code:: text
 
-  # Run rnstatus
-  rnstatus
+  $ rnstatus
 
-  # Example output
   Shared Instance[37428]
      Status  : Up
      Serving : 1 program
@@ -221,7 +275,7 @@ interfaces, similar to the ``ifconfig`` program.
      Traffic : 63.23 KB↑
                80.17 KB↓
 
-  TCPInterface[RNS Testnet Frankfurt/frankfurt.rns.unsigned.io:4965]
+  TCPInterface[RNS Testnet Dublin/dublin.connect.reticulum.network:4965]
      Status  : Up
      Mode    : Full
      Rate    : 10.00 Mbps
@@ -238,52 +292,195 @@ interfaces, similar to the ``ifconfig`` program.
 
   Reticulum Transport Instance <5245a8efe1788c6a1cd36144a270e13b> running
 
+Filter output to only show some interfaces:
+
 .. code:: text
 
-  usage: rnstatus [-h] [--config CONFIG] [--version] [-a] [-v]
+  $ rnstatus rnode
+
+  RNodeInterface[RNode UHF]
+     Status  : Up
+     Mode    : Access Point
+     Rate    : 1.30 kbps
+     Access  : 64-bit IFAC by <…e702c42ba8>
+     Traffic : 8.49 KB↑
+               9.23 KB↓
+
+  Reticulum Transport Instance <5245a8efe1788c6a1cd36144a270e13b> running
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnstatus [-h] [--config CONFIG] [--version] [-a] [-A]
+                  [-l] [-s SORT] [-r] [-j] [-R hash] [-i path]
+                  [-w seconds] [-v] [filter]
 
   Reticulum Network Stack Status
 
-  optional arguments:
-    -h, --help       show this help message and exit
-    --config CONFIG  path to alternative Reticulum config directory
-    --version        show program's version number and exit
-    -a, --all        show all interfaces
+  positional arguments:
+    filter                only display interfaces with names including filter
+
+  options:
+    -h, --help            show this help message and exit
+    --config CONFIG       path to alternative Reticulum config directory
+    --version             show program's version number and exit
+    -a, --all             show all interfaces
+    -A, --announce-stats  show announce stats
+    -l, --link-stats      show link stats
+    -s SORT, --sort SORT  sort interfaces by [rate, traffic, rx, tx, announces, arx, atx, held]
+    -r, --reverse         reverse sorting
+    -j, --json            output in JSON format
+    -R hash               transport identity hash of remote instance to get status from
+    -i path               path to identity used for remote management
+    -w seconds            timeout before giving up on remote queries
     -v, --verbose
 
 
+The rnid Utility
+====================
+
+With the ``rnid`` utility, you can generate, manage and view Reticulum Identities.
+The program can also calculate Destination hashes, and perform encryption and
+decryption of files.
+
+Using ``rnid``, it is possible to asymmetrically encrypt files and information for
+any Reticulum destination hash, and also to create and verify cryptographic signatures.
+
+**Usage Examples**
+
+Generate a new Identity:
+
+.. code:: text
+
+  $ rnid -g ./new_identity
+
+Display Identity key information:
+
+.. code:: text
+
+  $ rnid -i ./new_identity -p
+
+  Loaded Identity <984b74a3f768bef236af4371e6f248cd> from new_id
+  Public Key  : 0f4259fef4521ab75a3409e353fe9073eb10783b4912a6a9937c57bf44a62c1e
+  Private Key : Hidden
+
+Encrypt a file for an LXMF user:
+
+.. code:: text
+
+  $ rnid -i 8dd57a738226809646089335a6b03695 -e my_file.txt
+
+  Recalled Identity <bc7291552be7a58f361522990465165c> for destination <8dd57a738226809646089335a6b03695>
+  Encrypting my_file.txt
+  File my_file.txt encrypted for <bc7291552be7a58f361522990465165c> to my_file.txt.rfe
+
+If the Identity for the destination is not already known, you can fetch it from the network by using the ``-R`` command-line option:
+
+.. code:: text
+
+  $ rnid -R -i 30602def3b3506a28ed33db6f60cc6c9 -e my_file.txt
+
+  Requesting unknown Identity for <30602def3b3506a28ed33db6f60cc6c9>...
+  Received Identity <2b489d06eaf7c543808c76a5332a447d> for destination <30602def3b3506a28ed33db6f60cc6c9> from the network
+  Encrypting my_file.txt
+  File my_file.txt encrypted for <2b489d06eaf7c543808c76a5332a447d> to my_file.txt.rfe
+
+Decrypt a file using the Reticulum Identity it was encrypted for:
+
+.. code:: text
+
+  $ rnid -i ./my_identity -d my_file.txt.rfe
+
+  Loaded Identity <2225fdeecaf6e2db4556c3c2d7637294> from ./my_identity
+  Decrypting ./my_file.txt.rfe...
+  File ./my_file.txt.rfe decrypted with <2225fdeecaf6e2db4556c3c2d7637294> to ./my_file.txt
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnid.py [-h] [--config path] [-i identity] [-g path] [-v] [-q] [-a aspects]
+                 [-H aspects] [-e path] [-d path] [-s path] [-V path] [-r path] [-w path]
+                 [-f] [-R] [-t seconds] [-p] [-P] [--version]
+
+  Reticulum Identity & Encryption Utility
+
+  options:
+    -h, --help            show this help message and exit
+    --config path         path to alternative Reticulum config directory
+    -i identity, --identity identity
+                          hexadecimal Reticulum Destination hash or path to Identity file
+    -g path, --generate path
+                          generate a new Identity
+    -v, --verbose         increase verbosity
+    -q, --quiet           decrease verbosity
+    -a aspects, --announce aspects
+                          announce a destination based on this Identity
+    -H aspects, --hash aspects
+                          show destination hashes for other aspects for this Identity
+    -e path, --encrypt path
+                          encrypt file
+    -d path, --decrypt path
+                          decrypt file
+    -s path, --sign path  sign file
+    -V path, --validate path
+                          validate signature
+    -r path, --read path  input file path
+    -w path, --write path
+                          output file path
+    -f, --force           write output even if it overwrites existing files
+    -R, --request         request unknown Identities from the network
+    -t seconds            identity request timeout before giving up
+    -p, --print-identity  print identity info and exit
+    -P, --print-private   allow displaying private keys
+    --version             show program's version number and exit
+
+
 The rnpath Utility
 ====================
 
 With the ``rnpath`` utility, you can look up and view paths for
 destinations on the Reticulum network.
 
-.. code:: text
+**Usage Examples**
 
-  # Run rnpath
-  rnpath c89b4da064bf66d280f0e4d8abfd9806
-
-  # Example output
-  Path found, destination <c89b4da064bf66d280f0e4d8abfd9806> is 4 hops away via <f53a1c4278e0726bb73fcc623d6ce763> on TCPInterface[Testnet/frankfurt.connect.reticulu.network:4965]
+Resolve path to a destination:
 
 .. code:: text
 
-  usage: rnpath [-h] [--config CONFIG] [--version] [-t] [-r] [-d] [-D] [-w seconds] [-v] [destination]
-  
+  $ rnpath c89b4da064bf66d280f0e4d8abfd9806
+
+  Path found, destination <c89b4da064bf66d280f0e4d8abfd9806> is 4 hops away via <f53a1c4278e0726bb73fcc623d6ce763> on TCPInterface[Testnet/dublin.connect.reticulum.network:4965]
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnpath [-h] [--config CONFIG] [--version] [-t] [-m hops]
+                [-r] [-d] [-D] [-x] [-w seconds] [-R hash] [-i path]
+                [-W seconds] [-j] [-v] [destination]
+
   Reticulum Path Discovery Utility
-  
+
   positional arguments:
     destination           hexadecimal hash of the destination
-  
-  optional arguments:
+
+  options:
     -h, --help            show this help message and exit
     --config CONFIG       path to alternative Reticulum config directory
     --version             show program's version number and exit
     -t, --table           show all known paths
+    -m hops, --max hops   maximum hops to filter path table by
     -r, --rates           show announce rate info
     -d, --drop            remove the path to a destination
     -D, --drop-announces  drop all queued announces
+    -x, --drop-via        drop all paths via specified transport instance
     -w seconds            timeout before giving up
+    -R hash               transport identity hash of remote instance to manage
+    -i path               path to identity used for remote management
+    -W seconds            timeout before giving up on remote queries
+    -j, --json            output in JSON format
     -v, --verbose
 
 
@@ -293,32 +490,72 @@ The rnprobe Utility
 The ``rnprobe`` utility lets you probe a destination for connectivity, similar
 to the ``ping`` program. Please note that probes will only be answered if the
 specified destination is configured to send proofs for received packets. Many
-destinations will not have this option enabled, and will not be probable.
+destinations will not have this option enabled, so most destinations will not
+be probable.
+
+You can enable a probe-reply destination on Reticulum Transport Instances by
+setting the ``respond_to_probes`` configuration directive. Reticulum will then
+print the probe destination to the log on Transport Instance startup.
+
+**Usage Examples**
+
+Probe a destination:
 
 .. code:: text
 
-  # Run rnprobe
-  rnprobe example_utilities.echo.request 2d03725b327348980d570f739a3a5708
+  $ rnprobe rnstransport.probe 2d03725b327348980d570f739a3a5708
 
-  # Example output
   Sent 16 byte probe to <2d03725b327348980d570f739a3a5708>
   Valid reply received from <2d03725b327348980d570f739a3a5708>
   Round-trip time is 38.469 milliseconds over 2 hops
 
+Send a larger probe:
+
 .. code:: text
 
-  usage: rnprobe [-h] [--config CONFIG] [--version] [-v] [full_name] [destination_hash]
+  $ rnprobe rnstransport.probe 2d03725b327348980d570f739a3a5708 -s 256
+
+  Sent 16 byte probe to <2d03725b327348980d570f739a3a5708>
+  Valid reply received from <2d03725b327348980d570f739a3a5708>
+  Round-trip time is 38.781 milliseconds over 2 hops
+
+If the interface that receives the probe replies supports reporting radio
+parameters such as **RSSI** and **SNR**, the ``rnprobe`` utility will print
+these as part of the result as well.
+
+.. code:: text
+
+  $ rnprobe rnstransport.probe e7536ee90bd4a440e130490b87a25124
+  
+  Sent 16 byte probe to <e7536ee90bd4a440e130490b87a25124>
+  Valid reply received from <e7536ee90bd4a440e130490b87a25124>
+  Round-trip time is 1.809 seconds over 1 hop [RSSI -73 dBm] [SNR 12.0 dB]
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnprobe [-h] [--config CONFIG] [-s SIZE] [-n PROBES]
+                 [-t seconds] [-w seconds] [--version] [-v]
+                 [full_name] [destination_hash]
 
   Reticulum Probe Utility
 
   positional arguments:
-    full_name         full destination name in dotted notation
-    destination_hash  hexadecimal hash of the destination
+    full_name             full destination name in dotted notation
+    destination_hash      hexadecimal hash of the destination
 
-  optional arguments:
-    -h, --help        show this help message and exit
-    --config CONFIG   path to alternative Reticulum config directory
-    --version         show program's version number and exit
+  options:
+    -h, --help            show this help message and exit
+    --config CONFIG       path to alternative Reticulum config directory
+    -s SIZE, --size SIZE  size of probe packet payload in bytes
+    -n PROBES, --probes PROBES
+                          number of probes to send
+    -t seconds, --timeout seconds
+                          timeout before giving up
+    -w seconds, --wait seconds
+                          time between each probe
+    --version             show program's version number and exit
     -v, --verbose
 
 
@@ -328,20 +565,40 @@ The rncp Utility
 The ``rncp`` utility is a simple file transfer tool. Using it, you can transfer
 files through Reticulum.
 
+**Usage Examples**
+
+Run rncp on the receiving system, specifying which identities are allowed to send files:
+  
 .. code:: text
 
-  # Run rncp on the receiving system, specifying which identities
-  # are allowed to send files
-  rncp --receive -a 1726dbad538775b5bf9b0ea25a4079c8 -a c50cc4e4f7838b6c31f60ab9032cbc62
+  $ rncp --listen -a 1726dbad538775b5bf9b0ea25a4079c8 -a c50cc4e4f7838b6c31f60ab9032cbc62
 
-  # From another system, copy a file to the receiving system
-  rncp ~/path/to/file.tgz 73cbd378bb0286ed11a707c13447bb1e
-
-You can specify as many allowed senders as needed, or complete disable authentication.
+You can also specify allowed identity hashes (one per line) in the file ~/.rncp/allowed_identities
+and simply running the program in listener mode:
 
 .. code:: text
 
-  usage: rncp [-h] [--config path] [-v] [-q] [-p] [-r] [-b] [-a allowed_hash] [-n] [-w seconds] [--version] [file] [destination]
+  $ rncp --listen
+
+From another system, copy a file to the receiving system:
+
+.. code:: text
+
+  $ rncp ~/path/to/file.tgz 73cbd378bb0286ed11a707c13447bb1e
+
+Or fetch a file from the remote system:
+
+.. code:: text
+
+  $ rncp --fetch ~/path/to/file.tgz 73cbd378bb0286ed11a707c13447bb1e
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rncp [-h] [--config path] [-v] [-q] [-S] [-l] [-F] [-f]
+              [-j path] [-b seconds] [-a allowed_hash] [-n] [-p]
+              [-w seconds] [--version] [file] [destination]
 
   Reticulum File Transfer Utility
 
@@ -349,19 +606,22 @@ You can specify as many allowed senders as needed, or complete disable authentic
     file                  file to be transferred
     destination           hexadecimal hash of the receiver
 
-  optional arguments:
+  options:
     -h, --help            show this help message and exit
     --config path         path to alternative Reticulum config directory
     -v, --verbose         increase verbosity
     -q, --quiet           decrease verbosity
+    -S, --silent          disable transfer progress output
+    -l, --listen          listen for incoming transfer requests
+    -F, --allow-fetch     allow authenticated clients to fetch files
+    -f, --fetch           fetch file from remote listener instead of sending
+    -j path, --jail path  restrict fetch requests to specified path
+    -b seconds            announce interval, 0 to only announce at startup
+    -a allowed_hash       allow this identity
+    -n, --no-auth         accept requests from anyone
     -p, --print-identity  print identity and destination info and exit
-    -r, --receive         wait for incoming files
-    -b, --no-announce     don't announce at program start
-    -a allowed_hash       accept from this identity
-    -n, --no-auth         accept files from anyone
     -w seconds            sender timeout before giving up
     --version             show program's version number and exit
-    -v, --verbose
 
 
 The rnx Utility
@@ -369,32 +629,43 @@ The rnx Utility
 
 The ``rnx`` utility is a basic remote command execution program. It allows you to
 execute commands on remote systems over Reticulum, and to view returned command
-output.
+output. For a fully interactive remote shell solution, be sure to also take a look
+at the `rnsh <https://github.com/acehoss/rnsh>`_ program.
+
+**Usage Examples**
+
+Run rnx on the listening system, specifying which identities are allowed to execute commands:
 
 .. code:: text
 
-  # Run rnx on the listening system, specifying which identities
-  # are allowed to execute commands
-  rncp --listen -a 941bed5e228775e5a8079fc38b1ccf3f -a 1b03013c25f1c2ca068a4f080b844a10
+  $ rnx --listen -a 941bed5e228775e5a8079fc38b1ccf3f -a 1b03013c25f1c2ca068a4f080b844a10
 
-  # From another system, run a command 
-  rnx 7a55144adf826958a9529a3bcf08b149 "cat /proc/cpuinfo"
-
-  # Or enter the interactive mode pseudo-shell
-  rnx 7a55144adf826958a9529a3bcf08b149 -x
-
-  # The default identity file is stored in
-  # ~/.reticulum/identities/rnx, but you can use
-  # another one, which will be created if it does
-  # not already exist
-  rnx 7a55144adf826958a9529a3bcf08b149 -i /path/to/identity -x
-
-You can specify as many allowed senders as needed, or completely disable authentication.
+From another system, run a command on the remote:
 
 .. code:: text
 
-  usage: rnx [-h] [--config path] [-v] [-q] [-p] [-l] [-i identity] [-x] [-b] [-a allowed_hash] [-n] [-N] [-d] [-m] [-w seconds] [-W seconds] [--stdin STDIN] [--stdout STDOUT] [--stderr STDERR] [--version]
-             [destination] [command]
+  $ rnx 7a55144adf826958a9529a3bcf08b149 "cat /proc/cpuinfo"
+
+Or enter the interactive mode pseudo-shell:
+
+.. code:: text
+
+  $ rnx 7a55144adf826958a9529a3bcf08b149 -x
+
+The default identity file is stored in ``~/.reticulum/identities/rnx``, but you can use
+another one, which will be created if it does not already exist 
+
+.. code:: text
+
+  $ rnx 7a55144adf826958a9529a3bcf08b149 -i /path/to/identity -x
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnx [-h] [--config path] [-v] [-q] [-p] [-l] [-i identity] [-x] [-b] [-n] [-N]
+             [-d] [-m] [-a allowed_hash] [-w seconds] [-W seconds] [--stdin STDIN]
+             [--stdout STDOUT] [--stderr STDERR] [--version] [destination] [command]
 
   Reticulum Remote Execution Utility
 
@@ -425,6 +696,109 @@ You can specify as many allowed senders as needed, or completely disable authent
     --version             show program's version number and exit
 
 
+The rnodeconf Utility
+=====================
+
+The ``rnodeconf`` utility allows you to inspect and configure existing :ref:`RNodes<rnode-main>`, and
+to create and provision new :ref:`RNodes<rnode-main>` from any supported hardware devices.
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnodeconf [-h] [-i] [-a] [-u] [-U] [--fw-version version]
+                   [--fw-url url] [--nocheck] [-e] [-E] [-C]
+                   [--baud-flash baud_flash] [-N] [-T] [-b] [-B] [-p] [-D i]
+                   [--display-addr byte] [--freq Hz] [--bw Hz] [--txp dBm]
+                   [--sf factor] [--cr rate] [--eeprom-backup] [--eeprom-dump]
+                   [--eeprom-wipe] [-P] [--trust-key hexbytes] [--version] [-f]
+                   [-r] [-k] [-S] [-H FIRMWARE_HASH] [--platform platform]
+                   [--product product] [--model model] [--hwrev revision]
+                   [port]
+
+  RNode Configuration and firmware utility. This program allows you to change
+  various settings and startup modes of RNode. It can also install, flash and
+  update the firmware on supported devices.
+
+  positional arguments:
+    port                  serial port where RNode is attached
+
+  options:
+    -h, --help            show this help message and exit
+    -i, --info            Show device info
+    -a, --autoinstall     Automatic installation on various supported devices
+    -u, --update          Update firmware to the latest version
+    -U, --force-update    Update to specified firmware even if version matches
+                          or is older than installed version
+    --fw-version version  Use a specific firmware version for update or
+                          autoinstall
+    --fw-url url          Use an alternate firmware download URL
+    --nocheck             Don't check for firmware updates online
+    -e, --extract         Extract firmware from connected RNode for later use
+    -E, --use-extracted   Use the extracted firmware for autoinstallation or
+                          update
+    -C, --clear-cache     Clear locally cached firmware files
+    --baud-flash baud_flash
+                          Set specific baud rate when flashing device. Default
+                          is 921600
+    -N, --normal          Switch device to normal mode
+    -T, --tnc             Switch device to TNC mode
+    -b, --bluetooth-on    Turn device bluetooth on
+    -B, --bluetooth-off   Turn device bluetooth off
+    -p, --bluetooth-pair  Put device into bluetooth pairing mode
+    -D i, --display i     Set display intensity (0-255)
+    --display-addr byte   Set display address as hex byte (00 - FF)
+    --freq Hz             Frequency in Hz for TNC mode
+    --bw Hz               Bandwidth in Hz for TNC mode
+    --txp dBm             TX power in dBm for TNC mode
+    --sf factor           Spreading factor for TNC mode (7 - 12)
+    --cr rate             Coding rate for TNC mode (5 - 8)
+    --eeprom-backup       Backup EEPROM to file
+    --eeprom-dump         Dump EEPROM to console
+    --eeprom-wipe         Unlock and wipe EEPROM
+    -P, --public          Display public part of signing key
+    --trust-key hexbytes  Public key to trust for device verification
+    --version             Print program version and exit
+    -f, --flash           Flash firmware and bootstrap EEPROM
+    -r, --rom             Bootstrap EEPROM without flashing firmware
+    -k, --key             Generate a new signing key and exit
+    -S, --sign            Display public part of signing key
+    -H FIRMWARE_HASH, --firmware-hash FIRMWARE_HASH
+                          Display installed firmware hash
+    --platform platform   Platform specification for device bootstrap
+    --product product     Product specification for device bootstrap
+    --model model         Model code for device bootstrap
+    --hwrev revision      Hardware revision for device bootstrap
+
+
+For more information on how to create your own RNodes, please read the :ref:`Creating RNodes<rnode-creating>`
+section of this manual.
+
+Remote Management
+-----------------
+
+It is possible to allow remote management of Reticulum
+systems using the various built-in utilities, such as
+``rnstatus`` and ``rnpath``. To do so, you will need to set
+the ``enable_remote_management`` directive in the ``[reticulum]``
+section of the configuration file. You will also need to specify
+one or more Reticulum Identity hashes for authenticating the
+queries from client programs. For this purpose, you can use
+existing identity files, or generate new ones with the rnid utility.
+
+The following is a truncated example of enabling remote management
+in the Reticulum configuration file:
+
+.. code:: text
+  
+  [reticulum]
+  ...
+  enable_remote_management = yes
+  remote_management_allowed = 9fb6d773498fb3feda407ed8ef2c3229, 2d882c5586e548d79b5af27bca1776dc
+  ...
+
+For a complete example configuration, you can run ``rnsd --exampleconfig``.
+
 Improving System Configuration
 ------------------------------
 
@@ -473,6 +847,9 @@ Reticulum as a System Service
 Instead of starting Reticulum manually, you can install ``rnsd`` as a system
 service and have it start automatically at boot.
 
+Systemwide Service
+^^^^^^^^^^^^^^^^^^
+
 If you installed Reticulum with ``pip``, the ``rnsd`` program will most likely
 be located in a user-local installation path only, which means ``systemd`` will not
 be able to execute it. In this case, you can simply symlink the ``rnsd`` program
@@ -519,4 +896,49 @@ If you want to automatically start ``rnsd`` at boot, run:
 
 .. code:: text
 
-  sudo systemctl enable rnsd
+  sudo systemctl enable rnsd
+
+Userspace Service
+^^^^^^^^^^^^^^^^^
+
+Alternatively you can use a user systemd service instead of a system wide one. This way the whole setup can be done as a regular user.
+Create a user systemd service files ``~/.config/systemd/user/rnsd.service`` with the following content:
+
+.. code:: text
+
+  [Unit]
+  Description=Reticulum Network Stack Daemon
+  After=default.target
+
+  [Service]
+  # If you run Reticulum on WiFi devices,
+  # or other devices that need some extra
+  # time to initialise, you might want to
+  # add a short delay before Reticulum is
+  # started by systemd:
+  # ExecStartPre=/bin/sleep 10
+  Type=simple
+  Restart=always
+  RestartSec=3
+  ExecStart=RNS_BIN_DIR/rnsd --service
+
+  [Install]
+  WantedBy=default.target
+
+Replace ``RNS_BIN_DIR`` with the path to your Reticulum binary directory (eg. /home/USERNAMEHERE/rns/bin).
+
+Start user service:
+
+.. code:: text
+
+  systemctl --user daemon-reload
+  systemctl --user start rnsd.service
+
+If you want to automatically start ``rnsd`` without having to log in as the USERNAMEHERE, do:
+
+.. code:: text
+  
+  sudo loginctl enable-linger USERNAMEHERE
+  systemctl --user enable rnsd.service
+  
+

docs/manual/_sources/whatis.rst.txt

@@ -2,50 +2,71 @@
 What is Reticulum?
 ******************
 
-Reticulum is a cryptography-based networking stack for building wide-area
-networks with readily available hardware, that can continue to operate even
-with extremely low bandwidth and very high latency.
+Reticulum is a cryptography-based networking stack for building both local and
+wide-area networks with readily available hardware, that can continue to operate
+under adverse conditions, such as extremely low bandwidth and very high latency.
 
 Reticulum allows you to build wide-area networks with off-the-shelf tools, and
-offers end-to-end encryption, autoconfiguring cryptographically backed
-multi-hop transport, efficient addressing, unforgeable packet acknowledgements
-and more.
+offers end-to-end encryption, forward secrecy, autoconfiguring cryptographically
+backed multi-hop transport, efficient addressing, unforgeable packet
+acknowledgements and more.
 
-Reticulum is a complete networking stack, and does not need IP or higher
+From a users perspective, Reticulum allows the creation of applications that
+respect and empower the autonomy and sovereignty of communities and individuals.
+Reticulum enables secure digital communication that cannot be subjected to
+outside control, manipulation or censorship.
+
+Reticulum enables the construction of both small and potentially planetary-scale
+networks, without any need for hierarchical or bureaucratic structures to control
+or manage them, while ensuring individuals and communities full sovereignty
+over their own network segments.
+
+Reticulum is a **complete networking stack**, and does not need IP or higher
 layers, although it is easy to utilise IP (with TCP or UDP) as the underlying
 carrier for Reticulum. It is therefore trivial to tunnel Reticulum over the
 Internet or private IP networks. Reticulum is built directly on cryptographic
 principles, allowing resilience and stable functionality in open and trustless
 networks.
 
-No kernel modules or drivers are required. Reticulum runs completely in
-userland, and can run on practically any system that runs Python 3. Reticulum
+No kernel modules or drivers are required. Reticulum can run completely in
+userland, and will run on practically any system that runs Python 3. Reticulum
 runs well even on small single-board computers like the Pi Zero.
 
 
 Current Status
 ==============
-Reticulum should currently be considered beta software. All core protocol
+**Please know!** Reticulum should currently be considered beta software. All core protocol
 features are implemented and functioning, but additions will probably occur as
-real-world use is explored. There will be bugs. The API and wire-format can be
-considered stable at the moment, but could change if absolutely warranted.
+real-world use is explored. *There will be bugs*. The API and wire-format can be
+considered complete and stable at the moment, but could change if absolutely warranted.
 
 
 What does Reticulum Offer?
 ==========================
+
 * Coordination-less globally unique addressing and identification
 
-* Fully self-configuring multi-hop routing
+* Fully self-configuring multi-hop routing over heterogeneous carriers
 
-* Complete initiator anonymity, communicate without revealing your identity
+* Flexible scalability over heterogeneous topologies
 
-* Asymmetric encryption based on X25519, and Ed25519 signatures as a basis for all communication
+  * Reticulum can carry data over any mixture of physical mediums and topologies
 
-* Forward Secrecy by using ephemeral Elliptic Curve Diffie-Hellman keys on Curve25519
+  * Low-bandwidth networks can co-exist and interoperate with large, high-bandwidth networks
 
-* Reticulum uses the `Fernet <https://github.com/fernet/spec/blob/master/Spec.md>`_ specification for on-the-wire / over-the-air encryption
+* Initiator anonymity, communicate without revealing your identity
 
-  * All keys are ephemeral and derived from an ECDH key exchange on Curve25519
+  * Reticulum does not include source addresses on any packets
+
+* Asymmetric X25519 encryption and Ed25519 signatures as a basis for all communication
+
+  * The foundational Reticulum Identity Keys are 512-bit Elliptic Curve keysets
+
+* Forward Secrecy is available for all communication types, both for single packets and over links
+
+* Reticulum uses the following format for encrypted tokens:
+
+  * Ephemeral per-packet and link keys and derived from an ECDH key exchange on Curve25519
 
   * AES-128 in CBC mode with PKCS7 padding
 
@@ -55,33 +76,45 @@ What does Reticulum Offer?
 
 * Unforgeable packet delivery confirmations
 
-* A variety of supported interface types
+* Flexible and extensible interface system
 
-* An intuitive and developer-friendly API
+  * Reticulum includes a large variety of built-in interface types
 
-* Efficient link establishment
+  * Ability to load and utilise custom user- or community-supplied interface types
 
-  * Total bandwidth cost of setting up a link is only 3 packets, totalling 297 bytes
+  * Easily create your own custom interfaces for communicating over anything
 
-  * Low cost of keeping links open at only 0.44 bits per second
+* Authentication and virtual network segmentation on all supported interface types
+
+* An intuitive and easy-to-use API
+
+  * Simpler and easier to use than sockets APIs and simpler, but more powerful
+
+  * Makes building distributed and decentralised applications much simpler
 
 * Reliable and efficient transfer of arbitrary amounts of data
 
   * Reticulum can handle a few bytes of data or files of many gigabytes
 
-  * Sequencing, transfer coordination and checksumming is automatic
+  * Sequencing, compression, transfer coordination and checksumming are automatic
 
   * The API is very easy to use, and provides transfer progress
 
-* Authentication and virtual network segmentation on all supported interface types
+* Lightweight, flexible and expandable Request/Response mechanism
 
-* Flexible scalability allowing extremely low-bandwidth networks to co-exist and interoperate with large, high-bandwidth networks
+* Efficient link establishment
+
+  * Total cost of setting up an encrypted and verified link is only 3 packets, totalling 297 bytes
+
+  * Low cost of keeping links open at only 0.44 bits per second
+
+* Reliable sequential delivery with Channel and Buffer mechanisms
 
 
 Where can Reticulum be Used?
 ============================
 Over practically any medium that can support at least a half-duplex channel
-with 500 bits per second throughput, and an MTU of 500 bytes. Data radios,
+with greater throughput than 5 bits per second, and an MTU of 500 bytes. Data radios,
 modems, LoRa radios, serial lines, AX.25 TNCs, amateur radio digital modes,
 ad-hoc WiFi, free-space optical links and similar systems are all examples
 of the types of interfaces Reticulum was designed for.
@@ -89,7 +122,8 @@ of the types of interfaces Reticulum was designed for.
 An open-source LoRa-based interface called `RNode <https://unsigned.io/rnode>`_
 has been designed as an example transceiver that is very suitable for
 Reticulum. It is possible to build it yourself, to transform a common LoRa
-development board into one, or it can be purchased as a complete transceiver.
+development board into one, or it can be purchased as a complete transceiver
+from various vendors.
 
 Reticulum can also be encapsulated over existing IP networks, so there's
 nothing stopping you from using it over wired Ethernet or your local WiFi
@@ -105,7 +139,7 @@ network, and vice versa.
 
 Interface Types and Devices
 ===========================
-Reticulum implements a range of generalised interface types that covers the communications hardware that Reticulum can run over. If your hardware is not supported, it's relatively simple to implement an interface class. Currently, Reticulum can use the following devices and communication mediums:
+Reticulum implements a range of generalised interface types that covers the communications hardware that Reticulum can run over. If your hardware is not supported, it's simple to :ref:`implement an interface class<example-custominterface>`. Currently, Reticulum can use the following devices and communication mediums:
 
 * Any Ethernet device
 
@@ -152,7 +186,7 @@ Caveat Emptor
 ==============
 Reticulum is an experimental networking stack, and should be considered as
 such. While it has been built with cryptography best-practices very foremost in
-mind, it has not been externally security audited, and there could very well be
+mind, it has not yet been externally security audited, and there could very well be
 privacy-breaking bugs. To be considered secure, Reticulum needs a thorough
 security review by independent cryptographers and security researchers. If you
-want to help out, or help sponsor an audit, please do get in touch.
+want to help out with this, or can help sponsor an audit, please do get in touch.

docs/manual/_static/basic.css

@@ -236,16 +236,6 @@ div.body p, div.body dd, div.body li, div.body blockquote {
 a.headerlink {
     visibility: hidden;
 }
-a.brackets:before,
-span.brackets > a:before{
-    content: "[";
-}
-
-a.brackets:after,
-span.brackets > a:after {
-    content: "]";
-}
-
 
 h1:hover > a.headerlink,
 h2:hover > a.headerlink,
@@ -334,11 +324,15 @@ aside.sidebar {
 p.sidebar-title {
     font-weight: bold;
 }
+nav.contents,
+aside.topic,
 div.admonition, div.topic, blockquote {
     clear: left;
 }
 
 /* -- topics ---------------------------------------------------------------- */
+nav.contents,
+aside.topic,
 div.topic {
     border: 1px solid #ccc;
     padding: 7px;
@@ -377,6 +371,8 @@ div.body p.centered {
 
 div.sidebar > :last-child,
 aside.sidebar > :last-child,
+nav.contents > :last-child,
+aside.topic > :last-child,
 div.topic > :last-child,
 div.admonition > :last-child {
     margin-bottom: 0;
@@ -384,6 +380,8 @@ div.admonition > :last-child {
 
 div.sidebar::after,
 aside.sidebar::after,
+nav.contents::after,
+aside.topic::after,
 div.topic::after,
 div.admonition::after,
 blockquote::after {
@@ -608,19 +606,26 @@ ol.simple p,
 ul.simple p {
     margin-bottom: 0;
 }
-dl.footnote > dt,
-dl.citation > dt {
+aside.footnote > span,
+div.citation > span {
     float: left;
-    margin-right: 0.5em;
 }
-
-dl.footnote > dd,
-dl.citation > dd {
+aside.footnote > span:last-of-type,
+div.citation > span:last-of-type {
+  padding-right: 0.5em;
+}
+aside.footnote > p {
+  margin-left: 2em;
+}
+div.citation > p {
+  margin-left: 4em;
+}
+aside.footnote > p:last-of-type,
+div.citation > p:last-of-type {
     margin-bottom: 0em;
 }
-
-dl.footnote > dd:after,
-dl.citation > dd:after {
+aside.footnote > p:last-of-type:after,
+div.citation > p:last-of-type:after {
     content: "";
     clear: both;
 }
@@ -636,10 +641,6 @@ dl.field-list > dt {
     padding-left: 0.5em;
     padding-right: 5px;
 }
-dl.field-list > dt:after {
-    content: ":";
-}
-
 
 dl.field-list > dd {
     padding-left: 0.5em;

docs/manual/_static/copybutton.css

@@ -35,7 +35,8 @@ div.highlight  {
     position: relative;
 }
 
-.highlight:hover button.copybtn {
+/* Show the copybutton */
+.highlight:hover button.copybtn, button.copybtn.success {
 	opacity: 1;
 }