Updated manual

Support IPv6 for outbound TCP interface (TCPClientInterface)

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,11 @@
+blank_issues_enabled: false
+contact_links:
+  - name: ✨ Feature Request or Idea
+    url: https://github.com/markqvist/Reticulum/discussions/new?category=ideas
+    about: Propose and discuss features and ideas
+  - name: 💬 Questions, Help & Discussion
+    about: Ask anything, or get help
+    url: https://github.com/markqvist/Reticulum/discussions/new/choose
+  - name: 📖 Read the Reticulum Manual
+    url: https://markqvist.github.io/Reticulum/manual/
+    about: The complete documentation for Reticulum

.github/ISSUE_TEMPLATE/🐛-bug-report.md

@@ -0,0 +1,35 @@
+---
+name: "\U0001F41B Bug Report"
+about: Report a reproducible bug
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Read the Contribution Guidelines**
+Before creating a bug report on this issue tracker, you **must** read the [Contribution Guidelines](https://github.com/markqvist/Reticulum/blob/master/Contributing.md). Issues that do not follow the contribution guidelines **will be deleted without comment**.
+
+- The issue tracker is used by developers of this project. **Do not use it to ask general questions, or for support requests**.
+- Ideas and feature requests can be made on the [Discussions](https://github.com/markqvist/Reticulum/discussions). **Only** feature requests accepted by maintainers and developers are tracked and included on the issue tracker. **Do not post feature requests here**.
+- After reading the [Contribution Guidelines](https://github.com/markqvist/Reticulum/blob/master/Contributing.md), delete this section from your bug report.
+
+**Describe the Bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Describe in detail how to reproduce the bug.
+
+**Expected Behavior**
+A clear and concise description of what you expected to happen.
+
+**Logs & Screenshots**
+Please include any relevant log output. If applicable, also add screenshots to help explain your problem.
+
+**System Information**
+- OS and version
+- Python version
+- Program version
+
+**Additional context**
+Add any other context about the problem here.

.github/workflows/python-app.yml

@@ -0,0 +1,28 @@
+# This workflow will install Python dependencies, run tests and lint with a single version of Python
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+
+name: Test suite
+
+on:
+  push:
+    branches: [ "master" ]
+  pull_request:
+    branches: [ "master" ]
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python 3.10
+      uses: actions/setup-python@v3
+      with:
+        python-version: "3.10"
+    - name: Test
+      run: |
+        make test

.gitignore

@@ -10,5 +10,6 @@ docs/build
 rns*.egg-info
 profile.data
 tests/rnsconfig/storage
+tests/rnsconfig/logfile*
 *.data
 *.result

Contributing.md

@@ -0,0 +1,43 @@
+# Contributing to Reticulum
+
+Welcome, and thank you for your interest in contributing to Reticulum!
+
+Apart from writing code, there are many ways in which you can contribute. Before interacting with this community, read these short and simple guidelines.
+
+## Expected Conduct
+
+First and foremost, there is one simple requirement for taking part in this community: While we primarily interact virtually, your actions matter and have real consequences. Therefore: **Act like a responsible, civilized person** - also in the face of disputes and heated disagreements. Speak your mind here, discussions are welcome. Just do so in the spirit of being face-to-face with everyone else. Thank you.
+
+## Asking Questions
+
+If you want to ask a question, **do not open an issue**. The issue tracker is used by people *working on Reticulum* to track bugs, issues and improvements.
+
+Instead, ask away on the [discussions](https://github.com/markqvist/Reticulum/discussions) or on the [Reticulum Matrix channel](https://matrix.to/#/#reticulum:matrix.org) at `#reticulum:matrix.org`
+
+## Providing Feedback & Ideas
+
+Likewise, feedback, ideas and feature requests are a very welcome way to contribute, and should also be posted on the [discussions](https://github.com/markqvist/Reticulum/discussions), or on the [Reticulum Matrix channel](https://matrix.to/#/#reticulum:matrix.org) at `#reticulum:matrix.org`.
+
+Please do not post feature requests or general ideas on the issue tracker, or in direct messages to the primary developers. You are much more likely to get a response and start a constructive discussion by posting your ideas in the public channels created for these purposes.
+
+## Reporting Issues
+
+If you have found a bug or issue in this project, please report it using the [issue tracker](https://github.com/markqvist/Reticulum/issues). If at all possible, be sure to include details on how to reproduce the bug.
+
+Anything submitted to the issue tracker that does not follow these guidelines will be closed and removed without comments or explanation.
+
+## Writing Code
+
+If you are interested in contributing code, fixing open issues or adding features, please coordinate the effort with the maintainer or one of the main developers **before** submitting a pull request. Before deciding to contribute, it is also a good idea to ensure your efforts are in alignment with the [Roadmap](./Roadmap.md) and current development focus.
+
+Pull requests have a high chance of being accepted if they are:
+
+- In alignment with the [Roadmap](./Roadmap.md) or solve an open issue or feature request
+- Sufficiently tested to work with all API functions, and pass the standard test suite
+- Functionally and conceptually complete and well-designed
+- Not simply formatting or code style changes
+- Well-documented
+
+Even new ideas and proposals that have not been approved by a maintainer, or fall outside the established roadmap, are *occasionally* accepted - if they possess the remaining of the above qualities. If not, they will be closed and removed without comments or explanation.
+
+By contributing code to this project, you agree that copyright for the code is transferred to the Reticulum maintainers and that the code is irrevocably placed under the [MIT license](./LICENSE).

Examples/Announce.py

@@ -32,7 +32,7 @@ def program_setup(configpath):
     # Destinations are endpoints in Reticulum, that can be addressed
     # and communicated with. Destinations can also announce their
     # existence, which will let the network know they are reachable
-    # and autoomatically create paths to them, from anywhere else
+    # and automatically create paths to them, from anywhere else
     # in the network.
     destination_1 = RNS.Destination(
         identity,
@@ -53,7 +53,7 @@ def program_setup(configpath):
     )
 
     # We configure the destinations to automatically prove all
-    # packets adressed to it. By doing this, RNS will automatically
+    # packets addressed to it. By doing this, RNS will automatically
     # generate a proof for each incoming packet and transmit it
     # back to the sender of that packet. This will let anyone that
     # tries to communicate with the destination know whether their

Examples/Buffer.py

@@ -0,0 +1,323 @@
+##########################################################
+# This RNS example demonstrates how to set up a link to  #
+# a destination, and pass binary data over it using a    #
+# channel buffer.                                        #
+##########################################################
+from __future__ import annotations
+import os
+import sys
+import time
+import argparse
+from datetime import datetime
+
+import RNS
+from RNS.vendor import umsgpack
+
+# Let's define an app name. We'll use this for all
+# destinations we create. Since this echo example
+# is part of a range of example utilities, we'll put
+# them all within the app namespace "example_utilities"
+APP_NAME = "example_utilities"
+
+
+##########################################################
+#### Server Part #########################################
+##########################################################
+
+# A reference to the latest client link that connected
+latest_client_link = None
+
+# A reference to the latest buffer object
+latest_buffer = None
+
+# This initialisation is executed when the users chooses
+# to run as a server
+def server(configpath):
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+    
+    # Randomly create a new identity for our example
+    server_identity = RNS.Identity()
+
+    # We create a destination that clients can connect to. We
+    # want clients to create links to this destination, so we
+    # need to create a "single" destination type.
+    server_destination = RNS.Destination(
+        server_identity,
+        RNS.Destination.IN,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "bufferexample"
+    )
+
+    # We configure a function that will get called every time
+    # a new client creates a link to this destination.
+    server_destination.set_link_established_callback(client_connected)
+
+    # Everything's ready!
+    # Let's Wait for client requests or user input
+    server_loop(server_destination)
+
+def server_loop(destination):
+    # Let the user know that everything is ready
+    RNS.log(
+        "Link buffer example "+
+        RNS.prettyhexrep(destination.hash)+
+        " running, waiting for a connection."
+    )
+
+    RNS.log("Hit enter to manually send an announce (Ctrl-C to quit)")
+
+    # We enter a loop that runs until the users exits.
+    # If the user hits enter, we will announce our server
+    # destination on the network, which will let clients
+    # know how to create messages directed towards it.
+    while True:
+        entered = input()
+        destination.announce()
+        RNS.log("Sent announce from "+RNS.prettyhexrep(destination.hash))
+
+# When a client establishes a link to our server
+# destination, this function will be called with
+# a reference to the link.
+def client_connected(link):
+    global latest_client_link, latest_buffer
+    latest_client_link = link
+
+    RNS.log("Client connected")
+    link.set_link_closed_callback(client_disconnected)
+
+    # If a new connection is received, the old reader
+    # needs to be disconnected.
+    if latest_buffer:
+        latest_buffer.close()
+
+
+    # Create buffer objects.
+    #   The stream_id parameter to these functions is
+    #   a bit like a file descriptor, except that it
+    #   is unique to the *receiver*.
+    #
+    #   In this example, both the reader and the writer
+    #   use stream_id = 0, but there are actually two
+    #   separate unidirectional streams flowing in
+    #   opposite directions.
+    #
+    channel = link.get_channel()
+    latest_buffer = RNS.Buffer.create_bidirectional_buffer(0, 0, channel, server_buffer_ready)
+
+def client_disconnected(link):
+    RNS.log("Client disconnected")
+
+def server_buffer_ready(ready_bytes: int):
+    """
+    Callback from buffer when buffer has data available
+
+    :param ready_bytes: The number of bytes ready to read
+    """
+    global latest_buffer
+
+    data = latest_buffer.read(ready_bytes)
+    data = data.decode("utf-8")
+
+    RNS.log("Received data over the buffer: " + data)
+
+    reply_message = "I received \""+data+"\" over the buffer"
+    reply_message = reply_message.encode("utf-8")
+    latest_buffer.write(reply_message)
+    latest_buffer.flush()
+
+
+
+
+##########################################################
+#### Client Part #########################################
+##########################################################
+
+# A reference to the server link
+server_link = None
+
+# A reference to the buffer object, needed to share the
+# object from the link connected callback to the client
+# loop.
+buffer = None
+
+# This initialisation is executed when the users chooses
+# to run as a client
+def client(destination_hexhash, configpath):
+    # We need a binary representation of the destination
+    # hash that was entered on the command line
+    try:
+        dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+        if len(destination_hexhash) != dest_len:
+            raise ValueError(
+                "Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2)
+            )
+            
+        destination_hash = bytes.fromhex(destination_hexhash)
+    except:
+        RNS.log("Invalid destination entered. Check your input!\n")
+        exit()
+
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+
+    # Check if we know a path to the destination
+    if not RNS.Transport.has_path(destination_hash):
+        RNS.log("Destination is not yet known. Requesting path and waiting for announce to arrive...")
+        RNS.Transport.request_path(destination_hash)
+        while not RNS.Transport.has_path(destination_hash):
+            time.sleep(0.1)
+
+    # Recall the server identity
+    server_identity = RNS.Identity.recall(destination_hash)
+
+    # Inform the user that we'll begin connecting
+    RNS.log("Establishing link with server...")
+
+    # When the server identity is known, we set
+    # up a destination
+    server_destination = RNS.Destination(
+        server_identity,
+        RNS.Destination.OUT,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "bufferexample"
+    )
+
+    # And create a link
+    link = RNS.Link(server_destination)
+
+    # We'll also set up functions to inform the
+    # user when the link is established or closed
+    link.set_link_established_callback(link_established)
+    link.set_link_closed_callback(link_closed)
+
+    # Everything is set up, so let's enter a loop
+    # for the user to interact with the example
+    client_loop()
+
+def client_loop():
+    global server_link
+
+    # Wait for the link to become active
+    while not server_link:
+        time.sleep(0.1)
+
+    should_quit = False
+    while not should_quit:
+        try:
+            print("> ", end=" ")
+            text = input()
+
+            # Check if we should quit the example
+            if text == "quit" or text == "q" or text == "exit":
+                should_quit = True
+                server_link.teardown()
+            else:
+                # Otherwise, encode the text and write it to the buffer.
+                text = text.encode("utf-8")
+                buffer.write(text)
+                # Flush the buffer to force the data to be sent.
+                buffer.flush()
+
+
+        except Exception as e:
+            RNS.log("Error while sending data over the link buffer: "+str(e))
+            should_quit = True
+            server_link.teardown()
+
+# This function is called when a link
+# has been established with the server
+def link_established(link):
+    # We store a reference to the link
+    # instance for later use
+    global server_link, buffer
+    server_link = link
+
+    # Create buffer, see server_client_connected() for
+    # more detail about setting up the buffer.
+    channel = link.get_channel()
+    buffer = RNS.Buffer.create_bidirectional_buffer(0, 0, channel, client_buffer_ready)
+
+    # Inform the user that the server is
+    # connected
+    RNS.log("Link established with server, enter some text to send, or \"quit\" to quit")
+
+# When a link is closed, we'll inform the
+# user, and exit the program
+def link_closed(link):
+    if link.teardown_reason == RNS.Link.TIMEOUT:
+        RNS.log("The link timed out, exiting now")
+    elif link.teardown_reason == RNS.Link.DESTINATION_CLOSED:
+        RNS.log("The link was closed by the server, exiting now")
+    else:
+        RNS.log("Link closed, exiting now")
+    
+    RNS.Reticulum.exit_handler()
+    time.sleep(1.5)
+    os._exit(0)
+
+# When the buffer has new data, read it and write it to the terminal.
+def client_buffer_ready(ready_bytes: int):
+    global buffer
+    data = buffer.read(ready_bytes)
+    RNS.log("Received data over the link buffer: " + data.decode("utf-8"))
+    print("> ", end=" ")
+    sys.stdout.flush()
+
+
+##########################################################
+#### Program Startup #####################################
+##########################################################
+
+# This part of the program runs at startup,
+# and parses input of from the user, and then
+# starts up the desired program mode.
+if __name__ == "__main__":
+    try:
+        parser = argparse.ArgumentParser(description="Simple buffer example")
+
+        parser.add_argument(
+            "-s",
+            "--server",
+            action="store_true",
+            help="wait for incoming link requests from clients"
+        )
+
+        parser.add_argument(
+            "--config",
+            action="store",
+            default=None,
+            help="path to alternative Reticulum config directory",
+            type=str
+        )
+
+        parser.add_argument(
+            "destination",
+            nargs="?",
+            default=None,
+            help="hexadecimal hash of the server destination",
+            type=str
+        )
+
+        args = parser.parse_args()
+
+        if args.config:
+            configarg = args.config
+        else:
+            configarg = None
+
+        if args.server:
+            server(configarg)
+        else:
+            if (args.destination == None):
+                print("")
+                parser.print_help()
+                print("")
+            else:
+                client(args.destination, configarg)
+
+    except KeyboardInterrupt:
+        print("")
+        exit()

Examples/Channel.py

@@ -0,0 +1,390 @@
+##########################################################
+# This RNS example demonstrates how to set up a link to  #
+# a destination, and pass structured messages over it    #
+# using a channel.                                       #
+##########################################################
+
+import os
+import sys
+import time
+import argparse
+from datetime import datetime
+
+import RNS
+from RNS.vendor import umsgpack
+
+# Let's define an app name. We'll use this for all
+# destinations we create. Since this echo example
+# is part of a range of example utilities, we'll put
+# them all within the app namespace "example_utilities"
+APP_NAME = "example_utilities"
+
+##########################################################
+#### Shared Objects ######################################
+##########################################################
+
+# Channel data must be structured in a subclass of
+# MessageBase. This ensures that the channel will be able
+# to serialize and deserialize the object and multiplex it
+# with other objects. Both ends of a link will need the
+# same object definitions to be able to communicate over
+# a channel.
+#
+# Note: The objects we wish to use over the channel must
+# be registered with the channel, and each link has a
+# different channel instance. See the client_connected
+# and link_established functions in this example to see
+# how message types are registered.
+
+# Let's make a simple message class called StringMessage
+# that will convey a string with a timestamp.
+
+class StringMessage(RNS.MessageBase):
+    # The MSGTYPE class variable needs to be assigned a
+    # 2 byte integer value. This identifier allows the
+    # channel to look up your message's constructor when a
+    # message arrives over the channel.
+    #
+    # MSGTYPE must be unique across all message types we
+    # register with the channel. MSGTYPEs >= 0xf000 are
+    # reserved for the system.
+    MSGTYPE = 0x0101
+
+    # The constructor of our object must be callable with
+    # no arguments. We can have parameters, but they must
+    # have a default assignment.
+    #
+    # This is needed so the channel can create an empty
+    # version of our message into which the incoming
+    # message can be unpacked.
+    def __init__(self, data=None):
+        self.data = data
+        self.timestamp = datetime.now()
+
+    # Finally, our message needs to implement functions
+    # the channel can call to pack and unpack our message
+    # to/from the raw packet payload. We'll use the
+    # umsgpack package bundled with RNS. We could also use
+    # the struct package bundled with Python if we wanted
+    # more control over the structure of the packed bytes.
+    #
+    # Also note that packed message objects must fit
+    # entirely in one packet. The number of bytes
+    # available for message payloads can be queried from
+    # the channel using the Channel.MDU property. The
+    # channel MDU is slightly less than the link MDU due
+    # to encoding the message header.
+
+    # The pack function encodes the message contents into
+    # a byte stream.
+    def pack(self) -> bytes:
+        return umsgpack.packb((self.data, self.timestamp))
+
+    # And the unpack function decodes a byte stream into
+    # the message contents.
+    def unpack(self, raw):
+        self.data, self.timestamp = umsgpack.unpackb(raw)
+
+
+##########################################################
+#### Server Part #########################################
+##########################################################
+
+# A reference to the latest client link that connected
+latest_client_link = None
+
+# This initialisation is executed when the users chooses
+# to run as a server
+def server(configpath):
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+    
+    # Randomly create a new identity for our link example
+    server_identity = RNS.Identity()
+
+    # We create a destination that clients can connect to. We
+    # want clients to create links to this destination, so we
+    # need to create a "single" destination type.
+    server_destination = RNS.Destination(
+        server_identity,
+        RNS.Destination.IN,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "channelexample"
+    )
+
+    # We configure a function that will get called every time
+    # a new client creates a link to this destination.
+    server_destination.set_link_established_callback(client_connected)
+
+    # Everything's ready!
+    # Let's Wait for client requests or user input
+    server_loop(server_destination)
+
+def server_loop(destination):
+    # Let the user know that everything is ready
+    RNS.log(
+        "Link example "+
+        RNS.prettyhexrep(destination.hash)+
+        " running, waiting for a connection."
+    )
+
+    RNS.log("Hit enter to manually send an announce (Ctrl-C to quit)")
+
+    # We enter a loop that runs until the users exits.
+    # If the user hits enter, we will announce our server
+    # destination on the network, which will let clients
+    # know how to create messages directed towards it.
+    while True:
+        entered = input()
+        destination.announce()
+        RNS.log("Sent announce from "+RNS.prettyhexrep(destination.hash))
+
+# When a client establishes a link to our server
+# destination, this function will be called with
+# a reference to the link.
+def client_connected(link):
+    global latest_client_link
+    latest_client_link = link
+
+    RNS.log("Client connected")
+    link.set_link_closed_callback(client_disconnected)
+
+    # Register message types and add callback to channel
+    channel = link.get_channel()
+    channel.register_message_type(StringMessage)
+    channel.add_message_handler(server_message_received)
+
+def client_disconnected(link):
+    RNS.log("Client disconnected")
+
+def server_message_received(message):
+    """
+    A message handler
+    @param message: An instance of a subclass of MessageBase
+    @return: True if message was handled
+    """
+    global latest_client_link
+    # When a message is received over any active link,
+    # the replies will all be directed to the last client
+    # that connected.
+
+    # In a message handler, any deserializable message
+    # that arrives over the link's channel will be passed
+    # to all message handlers, unless a preceding handler indicates it
+    # has handled the message.
+    #
+    #
+    if isinstance(message, StringMessage):
+        RNS.log("Received data on the link: " + message.data + " (message created at " + str(message.timestamp) + ")")
+
+        reply_message = StringMessage("I received \""+message.data+"\" over the link")
+        latest_client_link.get_channel().send(reply_message)
+
+        # Incoming messages are sent to each message
+        # handler added to the channel, in the order they
+        # were added.
+        # If any message handler returns True, the message
+        # is considered handled and any subsequent
+        # handlers are skipped.
+        return True
+
+
+##########################################################
+#### Client Part #########################################
+##########################################################
+
+# A reference to the server link
+server_link = None
+
+# This initialisation is executed when the users chooses
+# to run as a client
+def client(destination_hexhash, configpath):
+    # We need a binary representation of the destination
+    # hash that was entered on the command line
+    try:
+        dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+        if len(destination_hexhash) != dest_len:
+            raise ValueError(
+                "Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2)
+            )
+            
+        destination_hash = bytes.fromhex(destination_hexhash)
+    except:
+        RNS.log("Invalid destination entered. Check your input!\n")
+        exit()
+
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+
+    # Check if we know a path to the destination
+    if not RNS.Transport.has_path(destination_hash):
+        RNS.log("Destination is not yet known. Requesting path and waiting for announce to arrive...")
+        RNS.Transport.request_path(destination_hash)
+        while not RNS.Transport.has_path(destination_hash):
+            time.sleep(0.1)
+
+    # Recall the server identity
+    server_identity = RNS.Identity.recall(destination_hash)
+
+    # Inform the user that we'll begin connecting
+    RNS.log("Establishing link with server...")
+
+    # When the server identity is known, we set
+    # up a destination
+    server_destination = RNS.Destination(
+        server_identity,
+        RNS.Destination.OUT,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "channelexample"
+    )
+
+    # And create a link
+    link = RNS.Link(server_destination)
+
+    # We'll also set up functions to inform the
+    # user when the link is established or closed
+    link.set_link_established_callback(link_established)
+    link.set_link_closed_callback(link_closed)
+
+    # Everything is set up, so let's enter a loop
+    # for the user to interact with the example
+    client_loop()
+
+def client_loop():
+    global server_link
+
+    # Wait for the link to become active
+    while not server_link:
+        time.sleep(0.1)
+
+    should_quit = False
+    while not should_quit:
+        try:
+            print("> ", end=" ")
+            text = input()
+
+            # Check if we should quit the example
+            if text == "quit" or text == "q" or text == "exit":
+                should_quit = True
+                server_link.teardown()
+
+            # If not, send the entered text over the link
+            if text != "":
+                message = StringMessage(text)
+                packed_size = len(message.pack())
+                channel = server_link.get_channel()
+                if channel.is_ready_to_send():
+                    if packed_size <= channel.MDU:
+                        channel.send(message)
+                    else:
+                        RNS.log(
+                            "Cannot send this packet, the data size of "+
+                            str(packed_size)+" bytes exceeds the link packet MDU of "+
+                            str(channel.MDU)+" bytes",
+                            RNS.LOG_ERROR
+                        )
+                else:
+                    RNS.log("Channel is not ready to send, please wait for " +
+                            "pending messages to complete.", RNS.LOG_ERROR)
+
+        except Exception as e:
+            RNS.log("Error while sending data over the link: "+str(e))
+            should_quit = True
+            server_link.teardown()
+
+# This function is called when a link
+# has been established with the server
+def link_established(link):
+    # We store a reference to the link
+    # instance for later use
+    global server_link
+    server_link = link
+
+    # Register messages and add handler to channel
+    channel = link.get_channel()
+    channel.register_message_type(StringMessage)
+    channel.add_message_handler(client_message_received)
+
+    # Inform the user that the server is
+    # connected
+    RNS.log("Link established with server, enter some text to send, or \"quit\" to quit")
+
+# When a link is closed, we'll inform the
+# user, and exit the program
+def link_closed(link):
+    if link.teardown_reason == RNS.Link.TIMEOUT:
+        RNS.log("The link timed out, exiting now")
+    elif link.teardown_reason == RNS.Link.DESTINATION_CLOSED:
+        RNS.log("The link was closed by the server, exiting now")
+    else:
+        RNS.log("Link closed, exiting now")
+    
+    RNS.Reticulum.exit_handler()
+    time.sleep(1.5)
+    os._exit(0)
+
+# When a packet is received over the channel, we
+# simply print out the data.
+def client_message_received(message):
+    if isinstance(message, StringMessage):
+        RNS.log("Received data on the link: " + message.data + " (message created at " + str(message.timestamp) + ")")
+        print("> ", end=" ")
+        sys.stdout.flush()
+
+
+##########################################################
+#### Program Startup #####################################
+##########################################################
+
+# This part of the program runs at startup,
+# and parses input of from the user, and then
+# starts up the desired program mode.
+if __name__ == "__main__":
+    try:
+        parser = argparse.ArgumentParser(description="Simple channel example")
+
+        parser.add_argument(
+            "-s",
+            "--server",
+            action="store_true",
+            help="wait for incoming link requests from clients"
+        )
+
+        parser.add_argument(
+            "--config",
+            action="store",
+            default=None,
+            help="path to alternative Reticulum config directory",
+            type=str
+        )
+
+        parser.add_argument(
+            "destination",
+            nargs="?",
+            default=None,
+            help="hexadecimal hash of the server destination",
+            type=str
+        )
+
+        args = parser.parse_args()
+
+        if args.config:
+            configarg = args.config
+        else:
+            configarg = None
+
+        if args.server:
+            server(configarg)
+        else:
+            if (args.destination == None):
+                print("")
+                parser.print_help()
+                print("")
+            else:
+                client(args.destination, configarg)
+
+    except KeyboardInterrupt:
+        print("")
+        exit()

Examples/Echo.py

@@ -46,7 +46,7 @@ def server(configpath):
     )
 
     # We configure the destination to automatically prove all
-    # packets adressed to it. By doing this, RNS will automatically
+    # packets addressed to it. By doing this, RNS will automatically
     # generate a proof for each incoming packet and transmit it
     # back to the sender of that packet.
     echo_destination.set_proof_strategy(RNS.Destination.PROVE_ALL)
@@ -210,6 +210,7 @@ def client(destination_hexhash, configpath, timeout=None):
             # If we do not know this destination, tell the
             # user to wait for an announce to arrive.
             RNS.log("Destination is not yet known. Requesting path...")
+            RNS.log("Hit enter to manually retry once an announce is received.")
             RNS.Transport.request_path(destination_hash)
 
 # This function is called when our reply destination

Examples/Filetransfer.py

@@ -449,8 +449,7 @@ def link_established(link):
     # And set up a small job to check for
     # a potential timeout in receiving the
     # file list
-    thread = threading.Thread(target=filelist_timeout_job)
-    thread.setDaemon(True)
+    thread = threading.Thread(target=filelist_timeout_job, daemon=True)
     thread.start()
 
 # This job just sleeps for the specified

Examples/Minimal.py

@@ -25,7 +25,7 @@ def program_setup(configpath):
     # Destinations are endpoints in Reticulum, that can be addressed
     # and communicated with. Destinations can also announce their
     # existence, which will let the network know they are reachable
-    # and autoomatically create paths to them, from anywhere else
+    # and automatically create paths to them, from anywhere else
     # in the network.
     destination = RNS.Destination(
         identity,
@@ -36,7 +36,7 @@ def program_setup(configpath):
     )
 
     # We configure the destination to automatically prove all
-    # packets adressed to it. By doing this, RNS will automatically
+    # packets addressed to it. By doing this, RNS will automatically
     # generate a proof for each incoming packet and transmit it
     # back to the sender of that packet. This will let anyone that
     # tries to communicate with the destination know whether their

Examples/Ratchets.py

@@ -0,0 +1,343 @@
+##########################################################
+# This RNS example demonstrates a simple client/server   #
+# echo utility that uses ratchets to rotate encryption   #
+# keys everytime an announce is sent.                    #
+##########################################################
+
+import argparse
+import RNS
+
+# Let's define an app name. We'll use this for all
+# destinations we create. Since this echo example
+# is part of a range of example utilities, we'll put
+# them all within the app namespace "example_utilities"
+APP_NAME = "example_utilities"
+
+
+##########################################################
+#### Server Part #########################################
+##########################################################
+
+# This initialisation is executed when the users chooses
+# to run as a server
+def server(configpath):
+    global reticulum
+
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+
+    # TODO: Remove
+    RNS.loglevel = RNS.LOG_DEBUG
+    
+    # Randomly create a new identity for our echo server
+    server_identity = RNS.Identity()
+
+    # We create a destination that clients can query. We want
+    # to be able to verify echo replies to our clients, so we
+    # create a "single" destination that can receive encrypted
+    # messages. This way the client can send a request and be
+    # certain that no-one else than this destination was able
+    # to read it. 
+    echo_destination = RNS.Destination(
+        server_identity,
+        RNS.Destination.IN,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "ratchet",
+        "echo",
+        "request"
+    )
+
+    # Enable ratchets on the destination by providing a file
+    # path to store ratchets. In this example, we will just
+    # use a temporary file, but in real-world applications,
+    # it's extremely important to keep this file secure, since
+    # it contains encryption keys for the destination.
+    destination_hexhash = RNS.hexrep(echo_destination.hash, delimit=False)
+    echo_destination.enable_ratchets(f"/tmp/{destination_hexhash}.ratchets")
+
+    # We configure the destination to automatically prove all
+    # packets addressed to it. By doing this, RNS will automatically
+    # generate a proof for each incoming packet and transmit it
+    # back to the sender of that packet.
+    echo_destination.set_proof_strategy(RNS.Destination.PROVE_ALL)
+    
+    # Tell the destination which function in our program to
+    # run when a packet is received. We do this so we can
+    # print a log message when the server receives a request
+    echo_destination.set_packet_callback(server_callback)
+
+    # Everything's ready!
+    # Let's Wait for client requests or user input
+    announceLoop(echo_destination)
+
+
+def announceLoop(destination):
+    # Let the user know that everything is ready
+    RNS.log(
+        "Ratcheted echo server "+
+        RNS.prettyhexrep(destination.hash)+
+        " running, hit enter to manually send an announce (Ctrl-C to quit)"
+    )
+
+    # We enter a loop that runs until the users exits.
+    # If the user hits enter, we will announce our server
+    # destination on the network, which will let clients
+    # know how to create messages directed towards it.
+    while True:
+        entered = input()
+        destination.announce()
+        RNS.log("Sent announce from "+RNS.prettyhexrep(destination.hash))
+
+
+def server_callback(message, packet):
+    global reticulum
+    
+    # Tell the user that we received an echo request, and
+    # that we are going to send a reply to the requester.
+    # Sending the proof is handled automatically, since we
+    # set up the destination to prove all incoming packets.
+
+    reception_stats = ""
+    if reticulum.is_connected_to_shared_instance:
+        reception_rssi = reticulum.get_packet_rssi(packet.packet_hash)
+        reception_snr  = reticulum.get_packet_snr(packet.packet_hash)
+
+        if reception_rssi != None:
+            reception_stats += " [RSSI "+str(reception_rssi)+" dBm]"
+        
+        if reception_snr != None:
+            reception_stats += " [SNR "+str(reception_snr)+" dBm]"
+
+    else:
+        if packet.rssi != None:
+            reception_stats += " [RSSI "+str(packet.rssi)+" dBm]"
+        
+        if packet.snr != None:
+            reception_stats += " [SNR "+str(packet.snr)+" dB]"
+
+    RNS.log("Received packet from echo client, proof sent"+reception_stats)
+
+
+##########################################################
+#### Client Part #########################################
+##########################################################
+
+# This initialisation is executed when the users chooses
+# to run as a client
+def client(destination_hexhash, configpath, timeout=None):
+    global reticulum
+    
+    # We need a binary representation of the destination
+    # hash that was entered on the command line
+    try:
+        dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+        if len(destination_hexhash) != dest_len:
+            raise ValueError(
+                "Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2)
+            )
+
+        destination_hash = bytes.fromhex(destination_hexhash)
+    except Exception as e:
+        RNS.log("Invalid destination entered. Check your input!")
+        RNS.log(str(e)+"\n")
+        exit()
+
+    # We must first initialise Reticulum
+    reticulum = RNS.Reticulum(configpath)
+
+    # We override the loglevel to provide feedback when
+    # an announce is received
+    if RNS.loglevel < RNS.LOG_INFO:
+        RNS.loglevel = RNS.LOG_INFO
+
+    # Tell the user that the client is ready!
+    RNS.log(
+        "Echo client ready, hit enter to send echo request to "+
+        destination_hexhash+
+        " (Ctrl-C to quit)"
+    )
+
+    # We enter a loop that runs until the user exits.
+    # If the user hits enter, we will try to send an
+    # echo request to the destination specified on the
+    # command line.
+    while True:
+        input()
+        
+        # Let's first check if RNS knows a path to the destination.
+        # If it does, we'll load the server identity and create a packet
+        if RNS.Transport.has_path(destination_hash):
+
+            # To address the server, we need to know it's public
+            # key, so we check if Reticulum knows this destination.
+            # This is done by calling the "recall" method of the
+            # Identity module. If the destination is known, it will
+            # return an Identity instance that can be used in
+            # outgoing destinations.
+            server_identity = RNS.Identity.recall(destination_hash)
+
+            # We got the correct identity instance from the
+            # recall method, so let's create an outgoing
+            # destination. We use the naming convention:
+            # example_utilities.ratchet.echo.request
+            # This matches the naming we specified in the
+            # server part of the code.
+            request_destination = RNS.Destination(
+                server_identity,
+                RNS.Destination.OUT,
+                RNS.Destination.SINGLE,
+                APP_NAME,
+                "ratchet",
+                "echo",
+                "request"
+            )
+
+            # The destination is ready, so let's create a packet.
+            # We set the destination to the request_destination
+            # that was just created, and the only data we add
+            # is a random hash.
+            echo_request = RNS.Packet(request_destination, RNS.Identity.get_random_hash())
+
+            # Send the packet! If the packet is successfully
+            # sent, it will return a PacketReceipt instance.
+            packet_receipt = echo_request.send()
+
+            # If the user specified a timeout, we set this
+            # timeout on the packet receipt, and configure
+            # a callback function, that will get called if
+            # the packet times out.
+            if timeout != None:
+                packet_receipt.set_timeout(timeout)
+                packet_receipt.set_timeout_callback(packet_timed_out)
+
+            # We can then set a delivery callback on the receipt.
+            # This will get automatically called when a proof for
+            # this specific packet is received from the destination.
+            packet_receipt.set_delivery_callback(packet_delivered)
+
+            # Tell the user that the echo request was sent
+            RNS.log("Sent echo request to "+RNS.prettyhexrep(request_destination.hash))
+        else:
+            # If we do not know this destination, tell the
+            # user to wait for an announce to arrive.
+            RNS.log("Destination is not yet known. Requesting path...")
+            RNS.log("Hit enter to manually retry once an announce is received.")
+            RNS.Transport.request_path(destination_hash)
+
+# This function is called when our reply destination
+# receives a proof packet.
+def packet_delivered(receipt):
+    global reticulum
+
+    if receipt.status == RNS.PacketReceipt.DELIVERED:
+        rtt = receipt.get_rtt()
+        if (rtt >= 1):
+            rtt = round(rtt, 3)
+            rttstring = str(rtt)+" seconds"
+        else:
+            rtt = round(rtt*1000, 3)
+            rttstring = str(rtt)+" milliseconds"
+
+        reception_stats = ""
+        if reticulum.is_connected_to_shared_instance:
+            reception_rssi = reticulum.get_packet_rssi(receipt.proof_packet.packet_hash)
+            reception_snr  = reticulum.get_packet_snr(receipt.proof_packet.packet_hash)
+
+            if reception_rssi != None:
+                reception_stats += " [RSSI "+str(reception_rssi)+" dBm]"
+            
+            if reception_snr != None:
+                reception_stats += " [SNR "+str(reception_snr)+" dB]"
+
+        else:
+            if receipt.proof_packet != None:
+                if receipt.proof_packet.rssi != None:
+                    reception_stats += " [RSSI "+str(receipt.proof_packet.rssi)+" dBm]"
+                
+                if receipt.proof_packet.snr != None:
+                    reception_stats += " [SNR "+str(receipt.proof_packet.snr)+" dB]"
+
+        RNS.log(
+            "Valid reply received from "+
+            RNS.prettyhexrep(receipt.destination.hash)+
+            ", round-trip time is "+rttstring+
+            reception_stats
+        )
+
+# This function is called if a packet times out.
+def packet_timed_out(receipt):
+    if receipt.status == RNS.PacketReceipt.FAILED:
+        RNS.log("Packet "+RNS.prettyhexrep(receipt.hash)+" timed out")
+
+
+##########################################################
+#### Program Startup #####################################
+##########################################################
+
+# This part of the program gets run at startup,
+# and parses input from the user, and then starts
+# the desired program mode.
+if __name__ == "__main__":
+    try:
+        parser = argparse.ArgumentParser(description="Simple ratcheted echo server and client utility")
+
+        parser.add_argument(
+            "-s",
+            "--server",
+            action="store_true",
+            help="wait for incoming packets from clients"
+        )
+
+        parser.add_argument(
+            "-t",
+            "--timeout",
+            action="store",
+            metavar="s",
+            default=None,
+            help="set a reply timeout in seconds",
+            type=float
+        )
+
+        parser.add_argument("--config",
+            action="store",
+            default=None,
+            help="path to alternative Reticulum config directory",
+            type=str
+        )
+
+        parser.add_argument(
+            "destination",
+            nargs="?",
+            default=None,
+            help="hexadecimal hash of the server destination",
+            type=str
+        )
+
+        args = parser.parse_args()
+
+        if args.server:
+            configarg=None
+            if args.config:
+                configarg = args.config
+            server(configarg)
+        else:
+            if args.config:
+                configarg = args.config
+            else:
+                configarg = None
+
+            if args.timeout:
+                timeoutarg = float(args.timeout)
+            else:
+                timeoutarg = None
+
+            if (args.destination == None):
+                print("")
+                parser.print_help()
+                print("")
+            else:
+                client(args.destination, configarg, timeout=timeoutarg)
+    except KeyboardInterrupt:
+        print("")
+        exit()

Examples/Request.py

@@ -23,8 +23,8 @@ APP_NAME = "example_utilities"
 # A reference to the latest client link that connected
 latest_client_link = None
 
-def random_text_generator(path, data, request_id, remote_identity, requested_at):
-    RNS.log("Generating response to request "+RNS.prettyhexrep(request_id))
+def random_text_generator(path, data, request_id, link_id, remote_identity, requested_at):
+    RNS.log("Generating response to request "+RNS.prettyhexrep(request_id)+" on link "+RNS.prettyhexrep(link_id))
     texts = ["They looked up", "On each full moon", "Becky was upset", "I’ll stay away from it", "The pet shop stocks everything"]
     return texts[random.randint(0, len(texts)-1)]
 

FUNDING.yml

@@ -0,0 +1,2 @@
+ko_fi: markqvist
+custom: "https://unsigned.io/donate"

LICENSE

@@ -1,6 +1,6 @@
 MIT License, unless otherwise noted
 
-Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+Copyright (c) 2016-2024 Mark Qvist / unsigned.io
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

Makefile

@@ -2,7 +2,7 @@ all: release
 
 test:
 	@echo Running tests...
-	python -m tests.all
+	python3 -m tests.all
 
 clean:
 	@echo Cleaning...
@@ -47,7 +47,7 @@ documentation:
 	make -C docs html
 
 manual:
-	make -C docs latexpdf
+	make -C docs latexpdf epub
 
 release: test remove_symlinks build_wheel build_pure_wheel documentation manual create_symlinks
 

README.md

@@ -1,4 +1,4 @@
-Reticulum Network Stack β <img align="right" src="https://static.pepy.tech/personalized-badge/rns?period=total&units=international_system&left_color=grey&right_color=blue&left_text=Installs"/>
+Reticulum Network Stack β <img align="right" src="https://static.pepy.tech/personalized-badge/rns?period=total&units=international_system&left_color=grey&right_color=blue&left_text=Installs" style="padding-left:10px"/><a href="https://github.com/markqvist/reticulum/actions/workflows/python-app.yml"><img align="right" src="https://github.com/markqvist/reticulum/actions/workflows/python-app.yml/badge.svg"/></a>
 ==========
 
 <p align="center"><img width="200" src="https://raw.githubusercontent.com/markqvist/Reticulum/master/docs/source/graphics/rns_logo_512.png"></p>
@@ -35,44 +35,55 @@ userland, and can run on practically any system that runs Python 3.
 ## Read The Manual
 The full documentation for Reticulum is available at [markqvist.github.io/Reticulum/manual/](https://markqvist.github.io/Reticulum/manual/).
 
-You can also [download the Reticulum manual as a PDF](https://github.com/markqvist/Reticulum/raw/master/docs/Reticulum%20Manual.pdf)
+You can also download the [Reticulum manual as a PDF](https://github.com/markqvist/Reticulum/raw/master/docs/Reticulum%20Manual.pdf) or [as an e-book in EPUB format](https://github.com/markqvist/Reticulum/raw/master/docs/Reticulum%20Manual.epub).
 
-For more info, see [unsigned.io/projects/reticulum](https://unsigned.io/projects/reticulum/)
+For more info, see [reticulum.network](https://reticulum.network/) and [the FAQ section of the wiki](https://github.com/markqvist/Reticulum/wiki/Frequently-Asked-Questions).
 
 ## Notable Features
 - Coordination-less globally unique addressing and identification
-- Fully self-configuring multi-hop routing
+- Fully self-configuring multi-hop routing over heterogeneous carriers
 - Initiator anonymity, communicate without revealing your identity
+  - Reticulum does not include source addresses on any packets
 - Asymmetric X25519 encryption and Ed25519 signatures as a basis for all communication
-- Forward Secrecy with ephemeral Elliptic Curve Diffie-Hellman keys on Curve25519
-- Reticulum uses the [Fernet](https://github.com/fernet/spec/blob/master/Spec.md) specification for on-the-wire / over-the-air encryption
-  - Keys are ephemeral and derived from an ECDH key exchange on Curve25519
+  - The foundational Reticulum Identity Keys are 512-bit Elliptic Curve keysets
+- Forward Secrecy is available for all communication types, both for single packets and over links
+- Reticulum uses the following format for encrypted tokens:
+  - Ephemeral per-packet and link keys and derived from an ECDH key exchange on Curve25519
   - AES-128 in CBC mode with PKCS7 padding
   - HMAC using SHA256 for authentication
   - IVs are generated through os.urandom()
 - Unforgeable packet delivery confirmations
-- A variety of supported interface types
+- A large variety of supported interface types
 - An intuitive and easy-to-use API
 - Reliable and efficient transfer of arbitrary amounts of data
   - Reticulum can handle a few bytes of data or files of many gigabytes
-  - Sequencing, transfer coordination and checksumming are automatic
+  - Sequencing, compression, transfer coordination and checksumming are automatic
   - The API is very easy to use, and provides transfer progress
 - Lightweight, flexible and expandable Request/Response mechanism
 - Efficient link establishment
-  - Total bandwidth cost of setting up an encrypted link is 3 packets totaling 297 bytes
+  - Total cost of setting up an encrypted and verified link is only 3 packets, totalling 297 bytes
   - Low cost of keeping links open at only 0.44 bits per second
+- Reliable sequential delivery with Channel and Buffer mechanisms
+
+## Roadmap
+While Reticulum is already a fully featured and functional networking stack,
+many improvements and additions are actively being worked on, and planned for the future.
+
+To learn more about the direction and future of Reticulum, please see the [Development Roadmap](./Roadmap.md).
 
 ## Examples of Reticulum Applications
 If you want to quickly get an idea of what Reticulum can do, take a look at the
 following resources.
 
-- For an off-grid, encrypted and resilient mesh communications platform, see [Nomad Network](https://github.com/markqvist/NomadNet)
-- The Android, Linux and macOS app [Sideband](https://github.com/markqvist/Sideband) has a graphical interface and focuses on ease of use.
+- You can use the [rnsh](https://github.com/acehoss/rnsh) program to establish remote shell sessions over Reticulum.
 - [LXMF](https://github.com/markqvist/lxmf) is a distributed, delay and disruption tolerant message transfer protocol built on Reticulum
+- For an off-grid, encrypted and resilient mesh communications platform, see [Nomad Network](https://github.com/markqvist/NomadNet)
+- The Android, Linux, macOS and Windows app [Sideband](https://github.com/markqvist/Sideband) has a graphical interface and focuses on ease of use.
+- [MeshChat](https://github.com/liamcottle/reticulum-meshchat) is a user-friendly LXMF client, that also supports voice calls.
 
 ## Where can Reticulum be used?
 Over practically any medium that can support at least a half-duplex channel
-with 500 bits per second throughput, and an MTU of 500 bytes. Data radios,
+with greater throughput than 5 bits per second, and an MTU of 500 bytes. Data radios,
 modems, LoRa radios, serial lines, AX.25 TNCs, amateur radio digital modes,
 WiFi and Ethernet devices, free-space optical links, and similar systems are
 all examples of the types of physical devices Reticulum can use.
@@ -102,20 +113,36 @@ you want to do. For full details and examples, have a look at the
 [Getting Started Fast](https://markqvist.github.io/Reticulum/manual/gettingstartedfast.html) 
 section of the [Reticulum Manual](https://markqvist.github.io/Reticulum/manual/).
 
-To simply install Reticulum and related utilities on your system, the easiest way is via pip:
+To simply install Reticulum and related utilities on your system, the easiest way is via `pip`.
+You can then start any program that uses Reticulum, or start Reticulum as a system service with
+[the rnsd utility](https://markqvist.github.io/Reticulum/manual/using.html#the-rnsd-utility).
 
 ```bash
-pip3 install rns
+pip install rns
 ```
 
-You can then start any program that uses Reticulum, or start Reticulum as a
-system service with [the rnsd utility](https://markqvist.github.io/Reticulum/manual/using.html#the-rnsd-utility).
+If you are using an operating system that blocks normal user package installation via `pip`,
+you can return `pip` to normal behaviour by editing the `~/.config/pip/pip.conf` file,
+and adding the following directive in the `[global]` section:
+
+```text
+[global]
+break-system-packages = true
+```
+
+Alternatively, you can use the `pipx` tool to install Reticulum in an isolated environment:
+
+```bash
+pipx install rns
+```
 
 When first started, Reticulum will create a default configuration file,
 providing basic connectivity to other Reticulum peers that might be locally
 reachable. The default config file contains a few examples, and references for
 creating a more complex configuration.
 
+If you have an old version of `pip` on your system, you may need to upgrade it first with `pip install pip --upgrade`. If you no not already have `pip` installed, you can install it using the package manager of your system with `sudo apt install python3-pip` or similar.
+
 For more detailed examples on how to expand communication over many mediums such 
 as packet radio or LoRa, serial ports, or over fast IP links and the Internet using 
 the UDP and TCP interfaces, take a look at the [Supported Interfaces](https://markqvist.github.io/Reticulum/manual/interfaces.html) 
@@ -131,12 +158,15 @@ section of the [Reticulum Manual](https://markqvist.github.io/Reticulum/manual/)
 - An interface status utility called `rnstatus`, that displays information about interfaces
 - The path lookup and management tool `rnpath` letting you view and modify path tables
 - A diagnostics tool called `rnprobe` for checking connectivity to destinations
-- A simple file transfer program called `rncp` making it easy to copy files to remote systems
+- A simple file transfer program called `rncp` making it easy to transfer files between systems
+- The identity management and encryption utility `rnid` let's you manage Identities and encrypt/decrypt files
 - The remote command execution program `rnx` let's you run commands and
   programs and retrieve output from remote systems
 
 All tools, including `rnx` and `rncp`, work reliably and well even over very
-low-bandwidth links like LoRa or Packet Radio.
+low-bandwidth links like LoRa or Packet Radio. For full-featured remote shells
+over Reticulum, also have a look at the [rnsh](https://github.com/acehoss/rnsh)
+program.
 
 ## Supported interface types and devices
 
@@ -149,7 +179,7 @@ useful.
 Currently, the following interfaces are supported:
 
 - Any Ethernet device
-- LoRa using [RNode](https://unsigned.io/projects/rnode/)
+- LoRa using [RNode](https://unsigned.io/rnode/)
 - Packet Radio TNCs (with or without AX.25)
 - KISS-compatible hardware and software modems
 - Any device with a serial port
@@ -160,12 +190,12 @@ Currently, the following interfaces are supported:
 
 ## Performance
 Reticulum targets a *very* wide usable performance envelope, but prioritises
-functionality and performance over low-bandwidth mediums. The goal is to
+functionality and performance on low-bandwidth mediums. The goal is to
 provide a dynamic performance envelope from 250 bits per second, to 1 gigabit
 per second on normal hardware.
 
-Currently, the usable performance envelope is approximately 500 bits per second
-to 20 megabits per second, with physical mediums faster than that not being
+Currently, the usable performance envelope is approximately 150 bits per second
+to 40 megabits per second, with physical mediums faster than that not being
 saturated. Performance beyond the current level is intended for future
 upgrades, but not highly prioritised at this point in time.
 
@@ -175,39 +205,6 @@ features are implemented and functioning, but additions will probably occur as
 real-world use is explored. There will be bugs. The API and wire-format can be
 considered relatively stable at the moment, but could change if warranted.
 
-## Development Roadmap
-- Version 0.4.0
-  - Improving [the manual](https://markqvist.github.io/Reticulum/manual/) with sections specifically for beginners
-  - Performance and memory optimisations
-  - Utilities for managing identities, signing and encryption
-  - User friendly interface configuration tool
-  - Support for radio and modem interfaces on Android
-  - More interface types for even broader compatibility
-    - Plain ESP32 devices (ESP-Now, WiFi, Bluetooth, etc.)
-    - More LoRa transceivers
-    - IR Transceivers
-- Planned, but not yet scheduled
-  - OpenWRT support
-  - Metric-based path selection
-  - Distributed Destination Naming System
-  - Network-wide path balancing
-  - Globally routable multicast
-  - Bindings for other programming languages
-  - Multiple paths in path table for quick recovery on link failures
-  - A portable Reticulum implementation in C, see [#21](https://github.com/markqvist/Reticulum/discussions/21)
-  - Easy way to share interface configurations, see [#19](https://github.com/markqvist/Reticulum/discussions/19)
-  - More interface types
-    - AT-compatible modems
-    - AWDL / OWL
-    - HF Modems
-    - CAN-bus
-    - ZeroMQ
-    - MQTT
-    - XBee
-    - SPI
-    - i²c
-    - Tor
-
 ## Dependencies
 The installation of the default `rns` package requires the dependencies listed
 below. Almost all systems and distributions have readily available packages for
@@ -215,7 +212,6 @@ these dependencies, and when the `rns` package is installed with `pip`, they
 will be downloaded and installed as well.
 
 - [PyCA/cryptography](https://github.com/pyca/cryptography)
-- [netifaces](https://github.com/al45tair/netifaces)
 - [pyserial](https://github.com/pyserial/pyserial)
 
 On more unusual systems, and in some rare cases, it might not be possible to
@@ -240,10 +236,17 @@ Primitives](#cryptographic-primitives) section of this document.
 
 ## Public Testnet
 If you just want to get started experimenting without building any physical
-networks, you are welcome to join the Unsigned.io RNS Testnet. The testnet is
-just that, an informal network for testing and experimenting. It will be up
-most of the time, and anyone can join, but it also means that there's no
-guarantees for service availability.
+networks, you are welcome to join the RNS Development Testnet.
+
+The testnet is just that, an informal network for testing and experimenting.
+It will be up most of the time, and anyone can join, but it also means that
+there's no guarantees for service availability.
+
+It probably goes without saying, but *don't use the testnet entry-points as 
+hardcoded or default interfaces in any applications you ship to users*. When
+shipping applications, the best practice is to provide your own default
+connectivity solutions, if needed and applicable, or in most cases, simply
+leave it up to the user which networks to connect to, and how.
 
 The testnet runs the very latest version of Reticulum (often even a short while
 before it is publicly released). Sometimes experimental versions of Reticulum
@@ -253,31 +256,25 @@ I2P. Just add one of the following interfaces to your Reticulum configuration
 file:
 
 ```
-# TCP/IP interface to the Dublin Hub
-  [[RNS Testnet Dublin]]
+# TCP/IP interface to the RNS Amsterdam Hub
+  [[RNS Testnet Amsterdam]]
     type = TCPClientInterface
     enabled = yes
-    target_host = dublin.connect.reticulum.network
+    target_host = amsterdam.connect.reticulum.network
     target_port = 4965
 
-# TCP/IP interface to the Frankfurt Hub
-  [[RNS Testnet Frankfurt]]
+# TCP/IP interface to the BetweenTheBorders Hub (community-provided)
+  [[RNS Testnet BetweenTheBorders]]
     type = TCPClientInterface
     enabled = yes
-    target_host = frankfurt.connect.reticulum.network
-    target_port = 5377
+    target_host = reticulum.betweentheborders.com
+    target_port = 4242
 
-# Interface to I2P Hub A
-  [[RNS Testnet I2P Hub A]]
+# Interface to Testnet I2P Hub
+  [[RNS Testnet I2P Hub]]
     type = I2PInterface
     enabled = yes
-    peers = mrwqlsioq4hoo2lmeeud7dkfscnm7yxak7dmiyvsrnpfag3z5tsq.b32.i2p
-
-# Interface to I2P Hub B
-  [[RNS Testnet I2P Hub B]]
-    type = I2PInterface
-    enabled = yes
-    peers = iwoqtz22dsr73aemwpw7guocplsjjoamyl7sogj33qtcd6ds4mza.b32.i2p
+    peers = g3br23bvx3lq5uddcsjii74xgmn6y5q325ovrkq2zw2wbzbqgbuq.b32.i2p
 ```
 
 The testnet also contains a number of [Nomad Network](https://github.com/markqvist/nomadnet) nodes, and LXMF propagation nodes.
@@ -286,16 +283,16 @@ The testnet also contains a number of [Nomad Network](https://github.com/markqvi
 You can help support the continued development of open, free and private communications systems by donating via one of the following channels:
 
 - Monero:
- ```
+  ```
   84FpY1QbxHcgdseePYNmhTHcrgMX4nFfBYtz2GKYToqHVVhJp8Eaw1Z1EedRnKD19b3B8NiLCGVxzKV17UMmmeEsCrPyA5w
   ```
 - Ethereum
   ```
-  0x81F7B979fEa6134bA9FD5c701b3501A2e61E897a
+  0xFDabC71AC4c0C78C95aDDDe3B4FA19d6273c5E73
   ```
 - Bitcoin
   ```
-  3CPmacGm34qYvR6XWLVEJmi2aNe3PZqUuq
+  35G9uWVzrpJJibzUwpNUQGQNFzLirhrYAH
   ```
 - Ko-Fi: https://ko-fi.com/markqvist
 
@@ -303,17 +300,20 @@ Are certain features in the development roadmap are important to you or your
 organisation? Make them a reality quickly by sponsoring their implementation.
 
 ## Cryptographic Primitives
-Reticulum uses a simple suite of efficient, strong and modern cryptographic
+Reticulum uses a simple suite of efficient, strong and well-tested cryptographic
 primitives, with widely available implementations that can be used both on
-general-purpose CPUs and on microcontrollers. The necessary primitives are:
+general-purpose CPUs and on microcontrollers. The utilised primitives are:
 
-- Ed25519 for signatures
-- X22519 for ECDH key exchanges
+- Reticulum Identity Keys are 512-bit Curve25519 keysets
+  - A 256-bit Ed25519 key for signatures
+  - A 256-bit X22519 key for ECDH key exchanges
 - HKDF for key derivation
-- Modified Fernet for encrypted tokens
-  - AES-128 in CBC mode
-  - HMAC for message authentication
-  - No Fernet version and timestamp fields
+- Encrypted tokens are based on the [Fernet spec](https://github.com/fernet/spec/)
+  - Ephemeral keys derived from an ECDH key exchange on Curve25519
+  - AES-128 in CBC mode with PKCS7 padding
+  - HMAC using SHA256 for message authentication
+  - IVs are generated through os.urandom()
+  - No Fernet version and timestamp metadata fields
 - SHA-256
 - SHA-512
 
@@ -366,8 +366,8 @@ projects:
 - [Curve25519.py](https://gist.github.com/nickovs/cc3c22d15f239a2640c185035c06f8a3#file-curve25519-py) by [Nicko van Someren](https://gist.github.com/nickovs), *Public Domain*
 - [I2Plib](https://github.com/l-n-s/i2plib) by [Viktor Villainov](https://github.com/l-n-s)
 - [PySerial](https://github.com/pyserial/pyserial) by Chris Liechti, *BSD License*
-- [Netifaces](https://github.com/al45tair/netifaces) by [Alastair Houghton](https://github.com/al45tair), *MIT License*
 - [Configobj](https://github.com/DiffSK/configobj) by Michael Foord, Nicola Larosa, Rob Dennis & Eli Courtwright, *BSD License*
 - [Six](https://github.com/benjaminp/six) by [Benjamin Peterson](https://github.com/benjaminp), *MIT License*
+- [ifaddr](https://github.com/pydron/ifaddr) by [Pydron](https://github.com/pydron), *MIT License*
 - [Umsgpack.py](https://github.com/vsergeev/u-msgpack-python) by [Ivan A. Sergeev](https://github.com/vsergeev)
 - [Python](https://www.python.org)

RNS/Buffer.py

@@ -0,0 +1,359 @@
+# MIT License
+#
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io and contributors.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from __future__ import annotations
+import bz2
+import sys
+import time
+import threading
+from threading import RLock
+import struct
+from RNS.Channel import Channel, MessageBase, SystemMessageTypes
+import RNS
+from io import RawIOBase, BufferedRWPair, BufferedReader, BufferedWriter
+from typing import Callable
+from contextlib import AbstractContextManager
+
+class StreamDataMessage(MessageBase):
+    MSGTYPE = SystemMessageTypes.SMT_STREAM_DATA
+    """
+    Message type for ``Channel``. ``StreamDataMessage``
+    uses a system-reserved message type.
+    """
+
+    STREAM_ID_MAX = 0x3fff  # 16383
+    """
+    The stream id is limited to 2 bytes - 2 bit
+    """
+
+    MAX_DATA_LEN = RNS.Link.MDU - 2 - 6  # 2 for stream data message header, 6 for channel envelope
+    """
+    When the Buffer package is imported, this value is
+    calculcated based on the value of OVERHEAD
+    """
+
+    def __init__(self, stream_id: int = None, data: bytes = None, eof: bool = False, compressed: bool = False):
+        """
+        This class is used to encapsulate binary stream
+        data to be sent over a ``Channel``.
+
+        :param stream_id: id of stream relative to receiver
+        :param data: binary data
+        :param eof: set to True if signalling End of File
+        """
+        super().__init__()
+        if stream_id is not None and stream_id > self.STREAM_ID_MAX:
+            raise ValueError("stream_id must be 0-16383")
+        self.stream_id = stream_id
+        self.compressed = compressed
+        self.data = data or bytes()
+        self.eof = eof
+
+    def pack(self) -> bytes:
+        if self.stream_id is None:
+            raise ValueError("stream_id")
+
+        header_val = (0x3fff & self.stream_id) | (0x8000 if self.eof else 0x0000) | (0x4000 if self.compressed > 0 else 0x0000)
+        return bytes(struct.pack(">H", header_val) + (self.data if self.data else bytes()))
+
+    def unpack(self, raw):
+        self.stream_id = struct.unpack(">H", raw[:2])[0]
+        self.eof = (0x8000 & self.stream_id) > 0
+        self.compressed = (0x4000 & self.stream_id) > 0
+        self.stream_id = self.stream_id & 0x3fff
+        self.data = raw[2:]
+
+        if self.compressed:
+            self.data = bz2.decompress(self.data)
+
+
+class RawChannelReader(RawIOBase, AbstractContextManager):
+    """
+    An implementation of RawIOBase that receives
+    binary stream data sent over a ``Channel``.
+
+      This class generally need not be instantiated directly.
+      Use :func:`RNS.Buffer.create_reader`,
+      :func:`RNS.Buffer.create_writer`, and
+      :func:`RNS.Buffer.create_bidirectional_buffer` functions
+      to create buffered streams with optional callbacks.
+
+      For additional information on the API of this
+      object, see the Python documentation for
+      ``RawIOBase``.
+    """
+    def __init__(self, stream_id: int, channel: Channel):
+        """
+        Create a raw channel reader.
+
+        :param stream_id: local stream id to receive at
+        :param channel: ``Channel`` object to receive from
+        """
+        self._stream_id = stream_id
+        self._channel = channel
+        self._lock = RLock()
+        self._buffer = bytearray()
+        self._eof = False
+        self._channel._register_message_type(StreamDataMessage, is_system_type=True)
+        self._channel.add_message_handler(self._handle_message)
+        self._listeners: [Callable[[int], None]] = []
+
+    def add_ready_callback(self, cb: Callable[[int], None]):
+        """
+        Add a function to be called when new data is available.
+        The function should have the signature ``(ready_bytes: int) -> None``
+
+        :param cb: function to call
+        """
+        with self._lock:
+            self._listeners.append(cb)
+
+    def remove_ready_callback(self, cb: Callable[[int], None]):
+        """
+        Remove a function added with :func:`RNS.RawChannelReader.add_ready_callback()`
+
+        :param cb: function to remove
+        """
+        with self._lock:
+            self._listeners.remove(cb)
+
+    def _handle_message(self, message: MessageBase):
+        if isinstance(message, StreamDataMessage):
+            if message.stream_id == self._stream_id:
+                with self._lock:
+                    if message.data is not None:
+                        self._buffer.extend(message.data)
+                    if message.eof:
+                        self._eof = True
+                    for listener in self._listeners:
+                        try:
+                            threading.Thread(target=listener, name="Message Callback", args=[len(self._buffer)], daemon=True).start()
+                        except Exception as ex:
+                            RNS.log("Error calling RawChannelReader(" + str(self._stream_id) + ") callback: " + str(ex), RNS.LOG_ERROR)
+                    return True
+        return False
+
+    def _read(self, __size: int) -> bytes | None:
+        with self._lock:
+            result = self._buffer[:__size]
+            self._buffer = self._buffer[__size:]
+            return result if len(result) > 0 or self._eof else None
+
+    def readinto(self, __buffer: bytearray) -> int | None:
+        ready = self._read(len(__buffer))
+        if ready is not None:
+            __buffer[:len(ready)] = ready
+        return len(ready) if ready is not None else None
+
+    def writable(self) -> bool:
+        return False
+
+    def seekable(self) -> bool:
+        return False
+
+    def readable(self) -> bool:
+        return True
+
+    def close(self):
+        with self._lock:
+            self._channel.remove_message_handler(self._handle_message)
+            self._listeners.clear()
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.close()
+        return False
+
+
+class RawChannelWriter(RawIOBase, AbstractContextManager):
+    """
+    An implementation of RawIOBase that receives
+    binary stream data sent over a channel.
+
+      This class generally need not be instantiated directly.
+      Use :func:`RNS.Buffer.create_reader`,
+      :func:`RNS.Buffer.create_writer`, and
+      :func:`RNS.Buffer.create_bidirectional_buffer` functions
+      to create buffered streams with optional callbacks.
+
+      For additional information on the API of this
+      object, see the Python documentation for
+      ``RawIOBase``.
+    """
+
+    MAX_CHUNK_LEN     = 1024*16
+    COMPRESSION_TRIES = 4
+
+    def __init__(self, stream_id: int, channel: Channel):
+        """
+        Create a raw channel writer.
+
+        :param stream_id: remote stream id to sent do
+        :param channel: ``Channel`` object to send on
+        """
+        self._stream_id = stream_id
+        self._channel = channel
+        self._eof = False
+
+    def write(self, __b: bytes) -> int | None:
+        try:
+            comp_tries = RawChannelWriter.COMPRESSION_TRIES
+            comp_try = 1
+            comp_success = False
+            chunk_len = len(__b)
+            if chunk_len > RawChannelWriter.MAX_CHUNK_LEN:
+                chunk_len = RawChannelWriter.MAX_CHUNK_LEN
+                __b = __b[:RawChannelWriter.MAX_CHUNK_LEN]
+            chunk_segment = None
+            while chunk_len > 32 and comp_try < comp_tries:
+                chunk_segment_length = int(chunk_len/comp_try)
+                compressed_chunk = bz2.compress(__b[:chunk_segment_length])
+                compressed_length = len(compressed_chunk)
+                if compressed_length < StreamDataMessage.MAX_DATA_LEN and compressed_length < chunk_segment_length:
+                    comp_success = True
+                    break
+                else:
+                    comp_try += 1
+
+            if comp_success:
+                chunk = compressed_chunk
+                processed_length = chunk_segment_length
+            else:
+                chunk = bytes(__b[:StreamDataMessage.MAX_DATA_LEN])
+                processed_length = len(chunk)
+
+            message = StreamDataMessage(self._stream_id, chunk, self._eof, comp_success)
+            
+            self._channel.send(message)
+            return processed_length
+
+        except RNS.Channel.ChannelException as cex:
+            if cex.type != RNS.Channel.CEType.ME_LINK_NOT_READY:
+                raise
+        return 0
+
+    def close(self):
+        try:
+            link_rtt = self._channel._outlet.link.rtt
+            timeout = time.time() + (link_rtt * len(self._channel._tx_ring) * 1)
+        except Exception as e:
+            timeout = time.time() + 15
+
+        while time.time() < timeout and not self._channel.is_ready_to_send():
+            time.sleep(0.05)
+
+        self._eof = True
+        self.write(bytes())
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.close()
+        return False
+
+    def seekable(self) -> bool:
+        return False
+
+    def readable(self) -> bool:
+        return False
+
+    def writable(self) -> bool:
+        return True
+
+class Buffer:
+    """
+    Static functions for creating buffered streams that send
+    and receive over a ``Channel``.
+
+    These functions use ``BufferedReader``, ``BufferedWriter``,
+    and ``BufferedRWPair`` to add buffering to
+    ``RawChannelReader`` and ``RawChannelWriter``.
+    """
+    @staticmethod
+    def create_reader(stream_id: int, channel: Channel,
+                      ready_callback: Callable[[int], None] | None = None) -> BufferedReader:
+        """
+        Create a buffered reader that reads binary data sent
+        over a ``Channel``, with an optional callback when
+        new data is available.
+
+        Callback signature: ``(ready_bytes: int) -> None``
+
+        For more information on the reader-specific functions
+        of this object, see the Python documentation for
+        ``BufferedReader``
+
+        :param stream_id: the local stream id to receive from
+        :param channel: the channel to receive on
+        :param ready_callback: function to call when new data is available
+        :return: a BufferedReader object
+        """
+        reader = RawChannelReader(stream_id, channel)
+        if ready_callback:
+            reader.add_ready_callback(ready_callback)
+        return BufferedReader(reader)
+
+    @staticmethod
+    def create_writer(stream_id: int, channel: Channel) -> BufferedWriter:
+        """
+        Create a buffered writer that writes binary data over
+        a ``Channel``.
+
+        For more information on the writer-specific functions
+        of this object, see the Python documentation for
+        ``BufferedWriter``
+
+        :param stream_id: the remote stream id to send to
+        :param channel: the channel to send on
+        :return: a BufferedWriter object
+        """
+        writer = RawChannelWriter(stream_id, channel)
+        return BufferedWriter(writer)
+
+    @staticmethod
+    def create_bidirectional_buffer(receive_stream_id: int, send_stream_id: int, channel: Channel,
+                                    ready_callback: Callable[[int], None] | None = None) -> BufferedRWPair:
+        """
+        Create a buffered reader/writer pair that reads and
+        writes binary data over a ``Channel``, with an
+        optional callback when new data is available.
+
+        Callback signature: ``(ready_bytes: int) -> None``
+
+        For more information on the reader-specific functions
+        of this object, see the Python documentation for
+        ``BufferedRWPair``
+
+        :param receive_stream_id: the local stream id to receive at
+        :param send_stream_id:  the remote stream id to send to
+        :param channel: the channel to send and receive on
+        :param ready_callback: function to call when new data is available
+        :return: a BufferedRWPair object
+        """
+        reader = RawChannelReader(receive_stream_id, channel)
+        if ready_callback:
+            reader.add_ready_callback(ready_callback)
+        writer = RawChannelWriter(send_stream_id, channel)
+        return BufferedRWPair(reader, writer)

RNS/Channel.py

@@ -0,0 +1,694 @@
+# MIT License
+#
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io and contributors.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from __future__ import annotations
+import collections
+import enum
+import threading
+import time
+from types import TracebackType
+from typing import Type, Callable, TypeVar, Generic, NewType
+import abc
+import contextlib
+import struct
+import RNS
+from abc import ABC, abstractmethod
+TPacket = TypeVar("TPacket")
+
+class SystemMessageTypes(enum.IntEnum):
+    SMT_STREAM_DATA = 0xff00
+
+class ChannelOutletBase(ABC, Generic[TPacket]):
+    """
+    An abstract transport layer interface used by Channel.
+
+    DEPRECATED: This was created for testing; eventually
+    Channel will use Link or a LinkBase interface
+    directly.
+    """
+    @abstractmethod
+    def send(self, raw: bytes) -> TPacket:
+        raise NotImplemented()
+
+    @abstractmethod
+    def resend(self, packet: TPacket) -> TPacket:
+        raise NotImplemented()
+
+    @property
+    @abstractmethod
+    def mdu(self):
+        raise NotImplemented()
+
+    @property
+    @abstractmethod
+    def rtt(self):
+        raise NotImplemented()
+
+    @property
+    @abstractmethod
+    def is_usable(self):
+        raise NotImplemented()
+
+    @abstractmethod
+    def get_packet_state(self, packet: TPacket) -> MessageState:
+        raise NotImplemented()
+
+    @abstractmethod
+    def timed_out(self):
+        raise NotImplemented()
+
+    @abstractmethod
+    def __str__(self):
+        raise NotImplemented()
+
+    @abstractmethod
+    def set_packet_timeout_callback(self, packet: TPacket, callback: Callable[[TPacket], None] | None,
+                                    timeout: float | None = None):
+        raise NotImplemented()
+
+    @abstractmethod
+    def set_packet_delivered_callback(self, packet: TPacket, callback: Callable[[TPacket], None] | None):
+        raise NotImplemented()
+
+    @abstractmethod
+    def get_packet_id(self, packet: TPacket) -> any:
+        raise NotImplemented()
+
+
+class CEType(enum.IntEnum):
+    """
+    ChannelException type codes
+    """
+    ME_NO_MSG_TYPE      = 0
+    ME_INVALID_MSG_TYPE = 1
+    ME_NOT_REGISTERED   = 2
+    ME_LINK_NOT_READY   = 3
+    ME_ALREADY_SENT     = 4
+    ME_TOO_BIG          = 5
+
+
+class ChannelException(Exception):
+    """
+    An exception thrown by Channel, with a type code.
+    """
+    def __init__(self, ce_type: CEType, *args):
+        super().__init__(args)
+        self.type = ce_type
+
+
+class MessageState(enum.IntEnum):
+    """
+    Set of possible states for a Message
+    """
+    MSGSTATE_NEW       = 0
+    MSGSTATE_SENT      = 1
+    MSGSTATE_DELIVERED = 2
+    MSGSTATE_FAILED    = 3
+
+
+class MessageBase(abc.ABC):
+    """
+    Base type for any messages sent or received on a Channel.
+    Subclasses must define the two abstract methods as well as
+    the ``MSGTYPE`` class variable.
+    """
+    # MSGTYPE must be unique within all classes sent over a
+    # channel. Additionally, MSGTYPE > 0xf000 are reserved.
+    MSGTYPE = None
+    """
+    Defines a unique identifier for a message class.
+    
+    * Must be unique within all classes registered with a ``Channel``
+    * Must be less than ``0xf000``. Values greater than or equal to ``0xf000`` are reserved.
+    """
+
+    @abstractmethod
+    def pack(self) -> bytes:
+        """
+        Create and return the binary representation of the message
+
+        :return: binary representation of message
+        """
+        raise NotImplemented()
+
+    @abstractmethod
+    def unpack(self, raw: bytes):
+        """
+        Populate message from binary representation
+
+        :param raw: binary representation
+        """
+        raise NotImplemented()
+
+
+MessageCallbackType = NewType("MessageCallbackType", Callable[[MessageBase], bool])
+
+
+class Envelope:
+    """
+    Internal wrapper used to transport messages over a channel and
+    track its state within the channel framework.
+    """
+    def unpack(self, message_factories: dict[int, Type]) -> MessageBase:
+        msgtype, self.sequence, length = struct.unpack(">HHH", self.raw[:6])
+        raw = self.raw[6:]
+        ctor = message_factories.get(msgtype, None)
+        if ctor is None:
+            raise ChannelException(CEType.ME_NOT_REGISTERED, f"Unable to find constructor for Channel MSGTYPE {hex(msgtype)}")
+        message = ctor()
+        message.unpack(raw)
+        self.unpacked = True
+        self.message = message
+
+        return message
+
+    def pack(self) -> bytes:
+        if self.message.__class__.MSGTYPE is None:
+            raise ChannelException(CEType.ME_NO_MSG_TYPE, f"{self.message.__class__} lacks MSGTYPE")
+        data = self.message.pack()
+        self.raw = struct.pack(">HHH", self.message.MSGTYPE, self.sequence, len(data)) + data
+        self.packed = True
+        return self.raw
+
+    def __init__(self, outlet: ChannelOutletBase, message: MessageBase = None, raw: bytes = None, sequence: int = None):
+        self.ts = time.time()
+        self.id = id(self)
+        self.message = message
+        self.raw = raw
+        self.packet: TPacket = None
+        self.sequence = sequence
+        self.outlet = outlet
+        self.tries = 0
+        self.unpacked = False
+        self.packed = False
+        self.tracked = False
+
+
+class Channel(contextlib.AbstractContextManager):
+    """
+    Provides reliable delivery of messages over
+    a link.
+
+    ``Channel`` differs from ``Request`` and
+    ``Resource`` in some important ways:
+
+     **Continuous**
+        Messages can be sent or received as long as
+        the ``Link`` is open.
+     **Bi-directional**
+        Messages can be sent in either direction on
+        the ``Link``; neither end is the client or
+        server.
+     **Size-constrained**
+        Messages must be encoded into a single packet.
+
+    ``Channel`` is similar to ``Packet``, except that it
+    provides reliable delivery (automatic retries) as well
+    as a structure for exchanging several types of
+    messages over the ``Link``.
+
+    ``Channel`` is not instantiated directly, but rather
+    obtained from a ``Link`` with ``get_channel()``.
+    """
+
+    # The initial window size at channel setup
+    WINDOW     = 2
+
+    # Absolute minimum window size
+    WINDOW_MIN = 2
+    WINDOW_MIN_LIMIT_SLOW    = 2
+    WINDOW_MIN_LIMIT_MEDIUM  = 5
+    WINDOW_MIN_LIMIT_FAST    = 16
+
+    # The maximum window size for transfers on slow links
+    WINDOW_MAX_SLOW      = 5
+
+    # The maximum window size for transfers on mid-speed links
+    WINDOW_MAX_MEDIUM    = 12
+
+    # The maximum window size for transfers on fast links
+    WINDOW_MAX_FAST      = 48
+    
+    # For calculating maps and guard segments, this
+    # must be set to the global maximum window.
+    WINDOW_MAX           = WINDOW_MAX_FAST
+    
+    # If the fast rate is sustained for this many request
+    # rounds, the fast link window size will be allowed.
+    FAST_RATE_THRESHOLD  = 10
+
+    # If the RTT rate is higher than this value,
+    # the max window size for fast links will be used.
+    RTT_FAST            = 0.18
+    RTT_MEDIUM          = 0.75
+    RTT_SLOW            = 1.45
+
+    # The minimum allowed flexibility of the window size.
+    # The difference between window_max and window_min
+    # will never be smaller than this value.
+    WINDOW_FLEXIBILITY   = 4
+
+    SEQ_MAX     = 0xFFFF
+    SEQ_MODULUS = SEQ_MAX+1
+
+    def __init__(self, outlet: ChannelOutletBase):
+        """
+
+        @param outlet:
+        """
+        self._outlet = outlet
+        self._lock = threading.RLock()
+        self._tx_ring: collections.deque[Envelope] = collections.deque()
+        self._rx_ring: collections.deque[Envelope] = collections.deque()
+        self._message_callbacks: [MessageCallbackType] = []
+        self._next_sequence = 0
+        self._next_rx_sequence = 0
+        self._message_factories: dict[int, Type[MessageBase]] = {}
+        self._max_tries = 5
+        self.fast_rate_rounds    = 0
+        self.medium_rate_rounds  = 0
+
+        if self._outlet.rtt > Channel.RTT_SLOW:
+            self.window              = 1
+            self.window_max          = 1
+            self.window_min          = 1
+            self.window_flexibility  = 1
+        else:
+            self.window              = Channel.WINDOW
+            self.window_max          = Channel.WINDOW_MAX_SLOW
+            self.window_min          = Channel.WINDOW_MIN
+            self.window_flexibility  = Channel.WINDOW_FLEXIBILITY
+
+    def __enter__(self) -> Channel:
+        return self
+
+    def __exit__(self, __exc_type: Type[BaseException] | None, __exc_value: BaseException | None,
+                 __traceback: TracebackType | None) -> bool | None:
+        self._shutdown()
+        return False
+
+    def register_message_type(self, message_class: Type[MessageBase]):
+        """
+        Register a message class for reception over a ``Channel``.
+
+        Message classes must extend ``MessageBase``.
+
+        :param message_class: Class to register
+        """
+        self._register_message_type(message_class, is_system_type=False)
+
+    def _register_message_type(self, message_class: Type[MessageBase], *, is_system_type: bool = False):
+        with self._lock:
+            if not issubclass(message_class, MessageBase):
+                raise ChannelException(CEType.ME_INVALID_MSG_TYPE,
+                                       f"{message_class} is not a subclass of {MessageBase}.")
+            if message_class.MSGTYPE is None:
+                raise ChannelException(CEType.ME_INVALID_MSG_TYPE,
+                                       f"{message_class} has invalid MSGTYPE class attribute.")
+            if message_class.MSGTYPE >= 0xf000 and not is_system_type:
+                raise ChannelException(CEType.ME_INVALID_MSG_TYPE,
+                                       f"{message_class} has system-reserved message type.")
+            try:
+                message_class()
+            except Exception as ex:
+                raise ChannelException(CEType.ME_INVALID_MSG_TYPE,
+                                       f"{message_class} raised an exception when constructed with no arguments: {ex}")
+
+            self._message_factories[message_class.MSGTYPE] = message_class
+
+    def add_message_handler(self, callback: MessageCallbackType):
+        """
+        Add a handler for incoming messages. A handler
+        has the following signature:
+
+        ``(message: MessageBase) -> bool``
+
+        Handlers are processed in the order they are
+        added. If any handler returns True, processing
+        of the message stops; handlers after the
+        returning handler will not be called.
+
+        :param callback: Function to call
+        """
+        with self._lock:
+            if callback not in self._message_callbacks:
+                self._message_callbacks.append(callback)
+
+    def remove_message_handler(self, callback: MessageCallbackType):
+        """
+        Remove a handler added with ``add_message_handler``.
+
+        :param callback: handler to remove
+        """
+        with self._lock:
+            if callback in self._message_callbacks:
+                self._message_callbacks.remove(callback)
+
+    def _shutdown(self):
+        with self._lock:
+            self._message_callbacks.clear()
+            self._clear_rings()
+
+    def _clear_rings(self):
+        with self._lock:
+            for envelope in self._tx_ring:
+                if envelope.packet is not None:
+                    self._outlet.set_packet_timeout_callback(envelope.packet, None)
+                    self._outlet.set_packet_delivered_callback(envelope.packet, None)
+            self._tx_ring.clear()
+            self._rx_ring.clear()
+
+    def _emplace_envelope(self, envelope: Envelope, ring: collections.deque[Envelope]) -> bool:
+        with self._lock:
+            i = 0
+            
+            for existing in ring:
+
+                if envelope.sequence == existing.sequence:
+                    RNS.log(f"Envelope: Emplacement of duplicate envelope with sequence "+str(envelope.sequence), RNS.LOG_EXTREME)
+                    return False
+                
+                if envelope.sequence < existing.sequence and not (self._next_rx_sequence - envelope.sequence) > (Channel.SEQ_MAX//2):
+                    ring.insert(i, envelope)
+
+                    envelope.tracked = True
+                    return True
+                
+                i += 1
+            
+            envelope.tracked = True
+            ring.append(envelope)
+
+            return True
+
+    def _run_callbacks(self, message: MessageBase):
+        cbs = self._message_callbacks.copy()
+
+        for cb in cbs:
+            try:
+                if cb(message):
+                    return
+            except Exception as e:
+                RNS.log("Channel "+str(self)+" experienced an error while running a message callback. The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+    def _receive(self, raw: bytes):
+        try:
+            envelope = Envelope(outlet=self._outlet, raw=raw)
+            with self._lock:
+                message = envelope.unpack(self._message_factories)
+
+                if envelope.sequence < self._next_rx_sequence:
+                    window_overflow = (self._next_rx_sequence+Channel.WINDOW_MAX) % Channel.SEQ_MODULUS
+                    if window_overflow < self._next_rx_sequence:
+                        if envelope.sequence > window_overflow:
+                            RNS.log("Invalid packet sequence ("+str(envelope.sequence)+") received on channel "+str(self), RNS.LOG_EXTREME)
+                            return
+                    else:
+                        RNS.log("Invalid packet sequence ("+str(envelope.sequence)+") received on channel "+str(self), RNS.LOG_EXTREME)
+                        return
+
+                is_new = self._emplace_envelope(envelope, self._rx_ring)
+
+            if not is_new:
+                RNS.log("Duplicate message received on channel "+str(self), RNS.LOG_EXTREME)
+                return
+            else:
+                with self._lock:
+                    contigous = []
+                    for e in self._rx_ring:
+                        if e.sequence == self._next_rx_sequence:
+                            contigous.append(e)
+                            self._next_rx_sequence = (self._next_rx_sequence + 1) % Channel.SEQ_MODULUS
+                            if self._next_rx_sequence == 0:
+                                for e in self._rx_ring:
+                                    if e.sequence == self._next_rx_sequence:
+                                        contigous.append(e)
+                                        self._next_rx_sequence = (self._next_rx_sequence + 1) % Channel.SEQ_MODULUS
+
+                    for e in contigous:
+                        if not e.unpacked:
+                            m = e.unpack(self._message_factories)
+                        else:
+                            m = e.message
+                            
+                        self._rx_ring.remove(e)
+                        self._run_callbacks(m)
+
+        except Exception as e:
+            RNS.log("An error ocurred while receiving data on "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+    def is_ready_to_send(self) -> bool:
+        """
+        Check if ``Channel`` is ready to send.
+
+        :return: True if ready
+        """
+        if not self._outlet.is_usable:
+            return False
+
+        with self._lock:
+            outstanding = 0
+            for envelope in self._tx_ring:
+                if envelope.outlet == self._outlet: 
+                    if not envelope.packet or not self._outlet.get_packet_state(envelope.packet) == MessageState.MSGSTATE_DELIVERED:
+                        outstanding += 1
+
+            if outstanding >= self.window:
+                return False
+
+        return True
+
+    def _packet_tx_op(self, packet: TPacket, op: Callable[[TPacket], bool]):
+        with self._lock:
+            envelope = next(filter(lambda e: self._outlet.get_packet_id(e.packet) == self._outlet.get_packet_id(packet),
+                                   self._tx_ring), None)
+
+            if envelope and op(envelope):
+                envelope.tracked = False
+                if envelope in self._tx_ring:
+                    self._tx_ring.remove(envelope)
+
+                    if self.window < self.window_max:
+                        self.window += 1
+
+                        # TODO: Remove at some point
+                        # RNS.log("Increased "+str(self)+" window to "+str(self.window), RNS.LOG_DEBUG)
+
+                    if self._outlet.rtt != 0:
+                        if self._outlet.rtt > Channel.RTT_FAST:
+                            self.fast_rate_rounds = 0
+
+                            if self._outlet.rtt > Channel.RTT_MEDIUM:
+                                self.medium_rate_rounds = 0
+
+                            else:
+                                self.medium_rate_rounds += 1
+                                if self.window_max < Channel.WINDOW_MAX_MEDIUM and self.medium_rate_rounds == Channel.FAST_RATE_THRESHOLD:
+                                    self.window_max = Channel.WINDOW_MAX_MEDIUM
+                                    self.window_min = Channel.WINDOW_MIN_LIMIT_MEDIUM
+                                    # TODO: Remove at some point
+                                    # RNS.log("Increased "+str(self)+" max window to "+str(self.window_max), RNS.LOG_DEBUG)
+                                    # RNS.log("Increased "+str(self)+" min window to "+str(self.window_min), RNS.LOG_DEBUG)
+                            
+                        else:
+                            self.fast_rate_rounds += 1
+                            if self.window_max < Channel.WINDOW_MAX_FAST and self.fast_rate_rounds == Channel.FAST_RATE_THRESHOLD:
+                                self.window_max = Channel.WINDOW_MAX_FAST
+                                self.window_min = Channel.WINDOW_MIN_LIMIT_FAST
+                                # TODO: Remove at some point
+                                # RNS.log("Increased "+str(self)+" max window to "+str(self.window_max), RNS.LOG_DEBUG)
+                                # RNS.log("Increased "+str(self)+" min window to "+str(self.window_min), RNS.LOG_DEBUG)
+
+
+                else:
+                    RNS.log("Envelope not found in TX ring for "+str(self), RNS.LOG_EXTREME)
+        if not envelope:
+            RNS.log("Spurious message received on "+str(self), RNS.LOG_EXTREME)
+
+    def _packet_delivered(self, packet: TPacket):
+        self._packet_tx_op(packet, lambda env: True)
+
+    def _update_packet_timeouts(self):
+        for envelope in self._tx_ring:
+            updated_timeout = self._get_packet_timeout_time(envelope.tries)
+            if envelope.packet and hasattr(envelope.packet, "receipt") and envelope.packet.receipt and envelope.packet.receipt.timeout:
+                if updated_timeout > envelope.packet.receipt.timeout:
+                    envelope.packet.receipt.set_timeout(updated_timeout)
+
+    def _get_packet_timeout_time(self, tries: int) -> float:
+        to = pow(1.5, tries - 1) * max(self._outlet.rtt*2.5, 0.025) * (len(self._tx_ring)+1.5)
+        return to
+
+    def _packet_timeout(self, packet: TPacket):
+        def retry_envelope(envelope: Envelope) -> bool:
+            if envelope.tries >= self._max_tries:
+                RNS.log("Retry count exceeded on "+str(self)+", tearing down Link.", RNS.LOG_ERROR)
+                self._shutdown()  # start on separate thread?
+                self._outlet.timed_out()
+                return True
+
+            envelope.tries += 1
+            self._outlet.resend(envelope.packet)
+            self._outlet.set_packet_delivered_callback(envelope.packet, self._packet_delivered)
+            self._outlet.set_packet_timeout_callback(envelope.packet, self._packet_timeout, self._get_packet_timeout_time(envelope.tries))
+            self._update_packet_timeouts()
+
+            if self.window > self.window_min:
+                self.window -= 1
+                # TODO: Remove at some point
+                # RNS.log("Decreased "+str(self)+" window to "+str(self.window), RNS.LOG_DEBUG)
+
+                if self.window_max > (self.window_min+self.window_flexibility):
+                    self.window_max -= 1
+                    # TODO: Remove at some point
+                    # RNS.log("Decreased "+str(self)+" max window to "+str(self.window_max), RNS.LOG_DEBUG)
+
+                # TODO: Remove at some point
+                # RNS.log("Decreased "+str(self)+" window to "+str(self.window), RNS.LOG_EXTREME)
+
+            return False
+
+        if self._outlet.get_packet_state(packet) != MessageState.MSGSTATE_DELIVERED:
+            self._packet_tx_op(packet, retry_envelope)
+
+    def send(self, message: MessageBase) -> Envelope:
+        """
+        Send a message. If a message send is attempted and
+        ``Channel`` is not ready, an exception is thrown.
+
+        :param message: an instance of a ``MessageBase`` subclass
+        """
+        envelope: Envelope | None = None
+        with self._lock:
+            if not self.is_ready_to_send():
+                raise ChannelException(CEType.ME_LINK_NOT_READY, f"Link is not ready")
+        
+            envelope = Envelope(self._outlet, message=message, sequence=self._next_sequence)
+            self._next_sequence = (self._next_sequence + 1) % Channel.SEQ_MODULUS
+            self._emplace_envelope(envelope, self._tx_ring)
+
+        if envelope is None:
+            raise BlockingIOError()
+
+        envelope.pack()
+        if len(envelope.raw) > self._outlet.mdu:
+            raise ChannelException(CEType.ME_TOO_BIG, f"Packed message too big for packet: {len(envelope.raw)} > {self._outlet.mdu}")
+        
+        envelope.packet = self._outlet.send(envelope.raw)
+        envelope.tries += 1
+        self._outlet.set_packet_delivered_callback(envelope.packet, self._packet_delivered)
+        self._outlet.set_packet_timeout_callback(envelope.packet, self._packet_timeout, self._get_packet_timeout_time(envelope.tries))
+        self._update_packet_timeouts()
+
+        return envelope
+
+    @property
+    def MDU(self):
+        """
+        Maximum Data Unit: the number of bytes available
+        for a message to consume in a single send. This
+        value is adjusted from the ``Link`` MDU to accommodate
+        message header information.
+
+        :return: number of bytes available
+        """
+        return self._outlet.mdu - 6  # sizeof(msgtype) + sizeof(length) + sizeof(sequence)
+
+
+class LinkChannelOutlet(ChannelOutletBase):
+    """
+    An implementation of ChannelOutletBase for RNS.Link.
+    Allows Channel to send packets over an RNS Link with
+    Packets.
+
+    :param link: RNS Link to wrap
+    """
+    def __init__(self, link: RNS.Link):
+        self.link = link
+
+    def send(self, raw: bytes) -> RNS.Packet:
+        packet = RNS.Packet(self.link, raw, context=RNS.Packet.CHANNEL)
+        if self.link.status == RNS.Link.ACTIVE:
+            packet.send()
+        return packet
+
+    def resend(self, packet: RNS.Packet) -> RNS.Packet:
+        receipt = packet.resend()
+        if not receipt:
+            RNS.log("Failed to resend packet", RNS.LOG_ERROR)
+        return packet
+
+    @property
+    def mdu(self):
+        return self.link.MDU
+
+    @property
+    def rtt(self):
+        return self.link.rtt
+
+    @property
+    def is_usable(self):
+        return True  # had issues looking at Link.status
+
+    def get_packet_state(self, packet: TPacket) -> MessageState:
+        if packet.receipt == None:
+            return MessageState.MSGSTATE_FAILED
+
+        status = packet.receipt.get_status()
+        if status == RNS.PacketReceipt.SENT:
+            return MessageState.MSGSTATE_SENT
+        if status == RNS.PacketReceipt.DELIVERED:
+            return MessageState.MSGSTATE_DELIVERED
+        if status == RNS.PacketReceipt.FAILED:
+            return MessageState.MSGSTATE_FAILED
+        else:
+            raise Exception(f"Unexpected receipt state: {status}")
+
+    def timed_out(self):
+        self.link.teardown()
+
+    def __str__(self):
+        return f"{self.__class__.__name__}({self.link})"
+
+    def set_packet_timeout_callback(self, packet: RNS.Packet, callback: Callable[[RNS.Packet], None] | None,
+                                    timeout: float | None = None):
+        if timeout and packet.receipt:
+            packet.receipt.set_timeout(timeout)
+
+        def inner(receipt: RNS.PacketReceipt):
+            callback(packet)
+
+        if packet and packet.receipt:
+            packet.receipt.set_timeout_callback(inner if callback else None)
+
+    def set_packet_delivered_callback(self, packet: RNS.Packet, callback: Callable[[RNS.Packet], None] | None):
+        def inner(receipt: RNS.PacketReceipt):
+            callback(packet)
+
+        if packet and packet.receipt:
+            packet.receipt.set_delivery_callback(inner if callback else None)
+
+    def get_packet_id(self, packet: RNS.Packet) -> any:
+        if packet and hasattr(packet, "get_hash") and callable(packet.get_hash):
+            return packet.get_hash()
+        else:
+            return None

RNS/Cryptography/Fernet.py

@@ -45,10 +45,10 @@ class Fernet():
 
     def __init__(self, key = None):
         if key == None:
-            raise ValueError("Fernet key cannot be None")
+            raise ValueError("Token key cannot be None")
 
         if len(key) != 32:
-            raise ValueError("Fernet key must be 32 bytes, not "+str(len(key)))
+            raise ValueError("Token key must be 32 bytes, not "+str(len(key)))
             
         self._signing_key = key[:16]
         self._encryption_key = key[16:]
@@ -72,7 +72,7 @@ class Fernet():
         current_time = int(time.time())
 
         if not isinstance(data, bytes):
-            raise TypeError("Fernet token plaintext input must be bytes")
+            raise TypeError("Token plaintext input must be bytes")
 
         ciphertext = AES_128_CBC.encrypt(
             plaintext = PKCS7.pad(data),
@@ -87,10 +87,10 @@ class Fernet():
 
     def decrypt(self, token = None):
         if not isinstance(token, bytes):
-            raise TypeError("Fernet token must be bytes")
+            raise TypeError("Token must be bytes")
 
         if not self.verify_hmac(token):
-            raise ValueError("Fernet token HMAC was invalid")
+            raise ValueError("Token HMAC was invalid")
 
         iv = token[:16]
         ciphertext = token[16:-32]
@@ -107,4 +107,4 @@ class Fernet():
             return plaintext
 
         except Exception as e:
-            raise ValueError("Could not decrypt Fernet token")
+            raise ValueError("Could not decrypt token")

RNS/Cryptography/HKDF.py

@@ -33,15 +33,12 @@ def hkdf(length=None, derive_from=None, salt=None, context=None):
     if length == None or length < 1:
         raise ValueError("Invalid output key length")
 
-    if derive_from == "None" or derive_from == "":
+    if derive_from == None or derive_from == "":
         raise ValueError("Cannot derive key from empty input material")
 
     if salt == None or len(salt) == 0:
         salt = bytes([0] * hash_len)
 
-    if salt == None:
-        salt = b""
-
     if context == None:
         context = b""
 
@@ -54,4 +51,4 @@ def hkdf(length=None, derive_from=None, salt=None, context=None):
         block = hmac_sha256(pseudorandom_key, block + context + bytes([i + 1]))
         derived += block
 
-    return derived[:length]
+    return derived[:length]

RNS/Destination.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2024 Mark Qvist / unsigned.io and contributors
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -20,11 +20,14 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
+import os
 import math
 import time
+import threading
 import RNS
 
 from RNS.Cryptography import Fernet
+from .vendor import umsgpack as umsgpack
 
 class Callbacks:
     def __init__(self):
@@ -38,14 +41,14 @@ class Destination:
     instances are used both to create outgoing and incoming endpoints. The
     destination type will decide if encryption, and what type, is used in
     communication with the endpoint. A destination can also announce its
-    presence on the network, which will also distribute necessary keys for
+    presence on the network, which will distribute necessary keys for
     encrypted communication with it.
 
     :param identity: An instance of :ref:`RNS.Identity<api-identity>`. Can hold only public keys for an outgoing destination, or holding private keys for an ingoing.
     :param direction: ``RNS.Destination.IN`` or ``RNS.Destination.OUT``.
     :param type: ``RNS.Destination.SINGLE``, ``RNS.Destination.GROUP`` or ``RNS.Destination.PLAIN``.
     :param app_name: A string specifying the app name.
-    :param \*aspects: Any non-zero number of string arguments.
+    :param \\*aspects: Any non-zero number of string arguments.
     """
 
     # Constants
@@ -69,6 +72,18 @@ class Destination:
     OUT        = 0x12;
     directions = [IN, OUT]
 
+    PR_TAG_WINDOW = 30
+
+    RATCHET_COUNT    = 512
+    """
+    The default number of generated ratchet keys a destination will retain, if it has ratchets enabled.
+    """
+
+    RATCHET_INTERVAL = 30*60
+    """
+    The minimum interval between rotating ratchet keys, in seconds.
+    """
+
     @staticmethod
     def expand_name(identity, app_name, *aspects):
         """
@@ -97,7 +112,12 @@ class Destination:
         name_hash = RNS.Identity.full_hash(Destination.expand_name(None, app_name, *aspects).encode("utf-8"))[:(RNS.Identity.NAME_HASH_LENGTH//8)]
         addr_hash_material = name_hash
         if identity != None:
-            addr_hash_material += identity.hash
+            if isinstance(identity, RNS.Identity):
+                addr_hash_material += identity.hash
+            elif isinstance(identity, bytes) and len(identity) == RNS.Reticulum.TRUNCATED_HASHLENGTH//8:
+                addr_hash_material += identity
+            else:
+                raise TypeError("Invalid material supplied for destination hash calculation")
 
         return RNS.Identity.full_hash(addr_hash_material)[:RNS.Reticulum.TRUNCATED_HASHLENGTH//8]
 
@@ -130,14 +150,26 @@ class Destination:
         self.type = type
         self.direction = direction
         self.proof_strategy = Destination.PROVE_NONE
+        self.ratchets = None
+        self.ratchets_path = None
+        self.ratchet_interval = Destination.RATCHET_INTERVAL
+        self.ratchet_file_lock = threading.Lock()
+        self.retained_ratchets = Destination.RATCHET_COUNT
+        self.latest_ratchet_time = None
+        self.latest_ratchet_id = None
+        self.__enforce_ratchets = False
         self.mtu = 0
 
+        self.path_responses = {}
         self.links = []
 
         if identity == None and direction == Destination.IN and self.type != Destination.PLAIN:
             identity = RNS.Identity()
             aspects = aspects+(identity.hexhash,)
 
+        if identity == None and direction == Destination.OUT and self.type != Destination.PLAIN:
+            raise ValueError("Can't create outbound SINGLE destination without an identity")
+
         if identity != None and self.type == Destination.PLAIN:
             raise TypeError("Selected destination type PLAIN cannot hold an identity")
 
@@ -162,8 +194,41 @@ class Destination:
         """
         return "<"+self.name+"/"+self.hexhash+">"
 
+    def _clean_ratchets(self):
+        if self.ratchets != None:
+            if len (self.ratchets) > self.retained_ratchets:
+                self.ratchets = self.ratchets[:Destination.RATCHET_COUNT]
 
-    def announce(self, app_data=None, path_response=False, send=True):
+    def _persist_ratchets(self):
+        try:
+            with self.ratchet_file_lock:
+                packed_ratchets = umsgpack.packb(self.ratchets)
+                persisted_data = {"signature": self.sign(packed_ratchets), "ratchets": packed_ratchets}
+                ratchets_file = open(self.ratchets_path, "wb")
+                ratchets_file.write(umsgpack.packb(persisted_data))
+                ratchets_file.close()
+        except Exception as e:
+            self.ratchets = None
+            self.ratchets_path = None
+            raise OSError("Could not write ratchet file contents for "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+    def rotate_ratchets(self):
+        if self.ratchets != None:
+            now = time.time()
+            if now > self.latest_ratchet_time+self.ratchet_interval:
+                RNS.log("Rotating ratchets for "+str(self), RNS.LOG_DEBUG)
+                new_ratchet = RNS.Identity._generate_ratchet()
+                self.ratchets.insert(0, new_ratchet)
+                self.latest_ratchet_time = now
+                self._clean_ratchets()
+                self._persist_ratchets()
+            return True
+        else:
+            raise SystemError("Cannot rotate ratchet on "+str(self)+", ratchets are not enabled")
+
+        return False
+
+    def announce(self, app_data=None, path_response=False, attached_interface=None, tag=None, send=True):
         """
         Creates an announce packet for this destination and broadcasts it on all
         relevant interfaces. Application specific data can be added to the announce.
@@ -173,36 +238,74 @@ class Destination:
         """
         if self.type != Destination.SINGLE:
             raise TypeError("Only SINGLE destination types can be announced")
-            
-        destination_hash = self.hash
-        random_hash = RNS.Identity.get_random_hash()[0:5]+int(time.time()).to_bytes(5, "big")
 
-        if app_data == None and self.default_app_data != None:
-            if isinstance(self.default_app_data, bytes):
-                app_data = self.default_app_data
-            elif callable(self.default_app_data):
-                returned_app_data = self.default_app_data()
-                if isinstance(returned_app_data, bytes):
-                    app_data = returned_app_data
+        if self.direction != Destination.IN:
+            raise TypeError("Only IN destination types can be announced")
         
-        signed_data = self.hash+self.identity.get_public_key()+self.name_hash+random_hash
-        if app_data != None:
-            signed_data += app_data
+        ratchet = b""
+        now = time.time()
+        stale_responses = []
+        for entry_tag in self.path_responses:
+            entry = self.path_responses[entry_tag]
+            if now > entry[0]+Destination.PR_TAG_WINDOW:
+                stale_responses.append(entry_tag)
 
-        signature = self.identity.sign(signed_data)
+        for entry_tag in stale_responses:
+            self.path_responses.pop(entry_tag)
 
-        announce_data = self.identity.get_public_key()+self.name_hash+random_hash+signature
+        if (path_response == True and tag != None) and tag in self.path_responses:
+            # This code is currently not used, since Transport will block duplicate
+            # path requests based on tags. When multi-path support is implemented in
+            # Transport, this will allow Transport to detect redundant paths to the
+            # same destination, and select the best one based on chosen criteria,
+            # since it will be able to detect that a single emitted announce was
+            # received via multiple paths. The difference in reception time will
+            # potentially also be useful in determining characteristics of the
+            # multiple available paths, and to choose the best one.
+            RNS.log("Using cached announce data for answering path request with tag "+RNS.prettyhexrep(tag), RNS.LOG_EXTREME)
+            announce_data = self.path_responses[tag][1]
+        
+        else:
+            destination_hash = self.hash
+            random_hash = RNS.Identity.get_random_hash()[0:5]+int(time.time()).to_bytes(5, "big")
 
-        if app_data != None:
-            announce_data += app_data
+            if self.ratchets != None:
+                self.rotate_ratchets()
+                ratchet = RNS.Identity._ratchet_public_bytes(self.ratchets[0])
+                RNS.Identity._remember_ratchet(self.hash, ratchet)
+
+            if app_data == None and self.default_app_data != None:
+                if isinstance(self.default_app_data, bytes):
+                    app_data = self.default_app_data
+                elif callable(self.default_app_data):
+                    returned_app_data = self.default_app_data()
+                    if isinstance(returned_app_data, bytes):
+                        app_data = returned_app_data
+            
+            signed_data = self.hash+self.identity.get_public_key()+self.name_hash+random_hash+ratchet
+            if app_data != None:
+                signed_data += app_data
+
+            signature = self.identity.sign(signed_data)
+            announce_data = self.identity.get_public_key()+self.name_hash+random_hash+ratchet+signature
+
+            if app_data != None:
+                announce_data += app_data
+
+            self.path_responses[tag] = [time.time(), announce_data]
 
         if path_response:
             announce_context = RNS.Packet.PATH_RESPONSE
         else:
             announce_context = RNS.Packet.NONE
 
-        announce_packet = RNS.Packet(self, announce_data, RNS.Packet.ANNOUNCE, context = announce_context)
+        if ratchet:
+            context_flag = RNS.Packet.FLAG_SET
+        else:
+            context_flag = RNS.Packet.FLAG_UNSET
 
+        announce_packet = RNS.Packet(self, announce_data, RNS.Packet.ANNOUNCE, context = announce_context,
+                                     attached_interface = attached_interface, context_flag=context_flag)
         if send:
             announce_packet.send()
         else:
@@ -262,13 +365,12 @@ class Destination:
         else:
             self.proof_strategy = proof_strategy
 
-
     def register_request_handler(self, path, response_generator = None, allow = ALLOW_NONE, allowed_list = None):
         """
         Registers a request handler.
 
         :param path: The path for the request handler to be registered.
-        :param response_generator: A function or method with the signature *response_generator(path, data, request_id, remote_identity, requested_at)* to be called. Whatever this funcion returns will be sent as a response to the requester. If the function returns ``None``, no response will be sent.
+        :param response_generator: A function or method with the signature *response_generator(path, data, request_id, link_id, remote_identity, requested_at)* to be called. Whatever this funcion returns will be sent as a response to the requester. If the function returns ``None``, no response will be sent.
         :param allow: One of ``RNS.Destination.ALLOW_NONE``, ``RNS.Destination.ALLOW_ALL`` or ``RNS.Destination.ALLOW_LIST``. If ``RNS.Destination.ALLOW_LIST`` is set, the request handler will only respond to requests for identified peers in the supplied list.
         :param allowed_list: A list of *bytes-like* :ref:`RNS.Identity<api-identity>` hashes.
         :raises: ``ValueError`` if any of the supplied arguments are invalid.
@@ -284,7 +386,6 @@ class Destination:
             request_handler = [path, response_generator, allow, allowed_list]
             self.request_handlers[path_hash] = request_handler
 
-
     def deregister_request_handler(self, path):
         """
         Deregisters a request handler.
@@ -299,14 +400,13 @@ class Destination:
         else:
             return False
 
-        
-
     def receive(self, packet):
         if packet.packet_type == RNS.Packet.LINKREQUEST:
             plaintext = packet.data
             self.incoming_link_request(plaintext, packet)
         else:
             plaintext = self.decrypt(packet.data)
+            packet.ratchet_id = self.latest_ratchet_id
             if plaintext != None:
                 if packet.packet_type == RNS.Packet.DATA:
                     if self.callbacks.packet != None:
@@ -315,13 +415,103 @@ class Destination:
                         except Exception as e:
                             RNS.log("Error while executing receive callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
 
-
     def incoming_link_request(self, data, packet):
         if self.accept_link_requests:
             link = RNS.Link.validate_request(self, data, packet)
             if link != None:
                 self.links.append(link)
 
+    def _reload_ratchets(self, ratchets_path):
+        if os.path.isfile(ratchets_path):
+            with self.ratchet_file_lock:
+                try:
+                    ratchets_file = open(ratchets_path, "rb")
+                    persisted_data = umsgpack.unpackb(ratchets_file.read())
+                    if "signature" in persisted_data and "ratchets" in persisted_data:
+                        if self.identity.validate(persisted_data["signature"], persisted_data["ratchets"]):
+                            self.ratchets = umsgpack.unpackb(persisted_data["ratchets"])
+                            self.ratchets_path = ratchets_path
+                        else:
+                            raise KeyError("Invalid ratchet file signature")
+
+                except Exception as e:
+                    self.ratchets = None
+                    self.ratchets_path = None
+                    raise OSError("Could not read ratchet file contents for "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+        else:
+            RNS.log("No existing ratchet data found, initialising new ratchet file for "+str(self), RNS.LOG_DEBUG)
+            self.ratchets = []
+            self.ratchets_path = ratchets_path
+            self._persist_ratchets()
+
+    def enable_ratchets(self, ratchets_path):
+        """
+        Enables ratchets on the destination. When ratchets are enabled, Reticulum will automatically rotate
+        the keys used to encrypt packets to this destination, and include the latest ratchet key in announces.
+
+        Enabling ratchets on a destination will provide forward secrecy for packets sent to that destination,
+        even when sent outside a ``Link``. The normal Reticulum ``Link`` establishment procedure already performs
+        its own ephemeral key exchange for each link establishment, which means that ratchets are not necessary
+        to provide forward secrecy for links.
+
+        Enabling ratchets will have a small impact on announce size, adding 32 bytes to every sent announce.
+
+        :param ratchets_path: The path to a file to store ratchet data in.
+        :returns: True if the operation succeeded, otherwise False.
+        """
+        if ratchets_path != None:
+            self.latest_ratchet_time = 0
+            self._reload_ratchets(ratchets_path)
+
+            # TODO: Remove at some point
+            RNS.log("Ratchets enabled on "+str(self), RNS.LOG_DEBUG)
+            return True
+
+        else:
+            raise ValueError("No ratchet file path specified for "+str(self))
+
+    def enforce_ratchets(self):
+        """
+        When ratchet enforcement is enabled, this destination will never accept packets that use its
+        base Identity key for encryption, but only accept packets encrypted with one of the retained
+        ratchet keys.
+        """
+        if self.ratchets != None:
+            self.__enforce_ratchets = True
+            RNS.log("Ratchets enforced on "+str(self), RNS.LOG_DEBUG)
+            return True
+        else:
+            return False
+
+    def set_retained_ratchets(self, retained_ratchets):
+        """
+        Sets the number of previously generated ratchet keys this destination will retain,
+        and try to use when decrypting incoming packets. Defaults to ``Destination.RATCHET_COUNT``.
+
+        :param retained_ratchets: The number of generated ratchets to retain.
+        :returns: True if the operation succeeded, False if not.
+        """
+        if isinstance(retained_ratchets, int) and retained_ratchets > 0:
+            self.retained_ratchets = retained_ratchets
+            self._clean_ratchets()
+            return True
+        else:
+            return False
+
+    def set_ratchet_interval(self, interval):
+        """
+        Sets the minimum interval in seconds between ratchet key rotation.
+        Defaults to ``Destination.RATCHET_INTERVAL``.
+
+        :param interval: The minimum interval in seconds.
+        :returns: True if the operation succeeded, False if not.
+        """
+        if isinstance(interval, int) and interval > 0:
+            self.ratchet_interval = interval
+            return True
+        else:
+            return False
+
     def create_keys(self):
         """
         For a ``RNS.Destination.GROUP`` type destination, creates a new symmetric key.
@@ -338,7 +528,6 @@ class Destination:
             self.prv_bytes = Fernet.generate_key()
             self.prv = Fernet(self.prv_bytes)
 
-
     def get_private_key(self):
         """
         For a ``RNS.Destination.GROUP`` type destination, returns the symmetric private key.
@@ -352,7 +541,6 @@ class Destination:
         else:
             return self.prv_bytes
 
-
     def load_private_key(self, key):
         """
         For a ``RNS.Destination.GROUP`` type destination, loads a symmetric private key.
@@ -376,7 +564,6 @@ class Destination:
         else:
             raise TypeError("A single destination holds keys through an Identity instance")
 
-
     def encrypt(self, plaintext):
         """
         Encrypts information for ``RNS.Destination.SINGLE`` or ``RNS.Destination.GROUP`` type destination.
@@ -388,7 +575,10 @@ class Destination:
             return plaintext
 
         if self.type == Destination.SINGLE and self.identity != None:
-            return self.identity.encrypt(plaintext)
+            selected_ratchet = RNS.Identity.get_ratchet(self.hash)
+            if selected_ratchet:
+                self.latest_ratchet_id = RNS.Identity._get_ratchet_id(selected_ratchet)
+            return self.identity.encrypt(plaintext, ratchet=selected_ratchet)
 
         if self.type == Destination.GROUP:
             if hasattr(self, "prv") and self.prv != None:
@@ -400,8 +590,6 @@ class Destination:
             else:
                 raise ValueError("No private key held by GROUP destination. Did you create or load one?")
 
-
-
     def decrypt(self, ciphertext):
         """
         Decrypts information for ``RNS.Destination.SINGLE`` or ``RNS.Destination.GROUP`` type destination.
@@ -413,7 +601,28 @@ class Destination:
             return ciphertext
 
         if self.type == Destination.SINGLE and self.identity != None:
-            return self.identity.decrypt(ciphertext)
+            if self.ratchets:
+                decrypted = None
+                try:
+                    decrypted = self.identity.decrypt(ciphertext, ratchets=self.ratchets, enforce_ratchets=self.__enforce_ratchets, ratchet_id_receiver=self)
+                except:
+                    decrypted = None
+
+                if not decrypted:
+                    try:
+                        RNS.log(f"Decryption with ratchets failed on {self}, reloading ratchets from storage and retrying", RNS.LOG_ERROR)
+                        self._reload_ratchets(self.ratchets_path)
+                        decrypted = self.identity.decrypt(ciphertext, ratchets=self.ratchets, enforce_ratchets=self.__enforce_ratchets, ratchet_id_receiver=self)
+                    except Exception as e:
+                        RNS.log(f"Decryption still failing after ratchet reload. The contained exception was: {e}", RNS.LOG_ERROR)
+                        raise e
+
+                    RNS.log("Decryption succeeded after ratchet reload", RNS.LOG_NOTICE)
+
+                return decrypted
+
+            else:
+                return self.identity.decrypt(ciphertext, ratchets=None, enforce_ratchets=self.__enforce_ratchets, ratchet_id_receiver=self)
 
         if self.type == Destination.GROUP:
             if hasattr(self, "prv") and self.prv != None:
@@ -425,7 +634,6 @@ class Destination:
             else:
                 raise ValueError("No private key held by GROUP destination. Did you create or load one?")
 
-
     def sign(self, message):
         """
         Signs information for ``RNS.Destination.SINGLE`` type destination.

RNS/Identity.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2024 Mark Qvist / unsigned.io and contributors.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -26,6 +26,7 @@ import RNS
 import time
 import atexit
 import hashlib
+import threading
 
 from .vendor import umsgpack as umsgpack
 
@@ -49,8 +50,20 @@ class Identity:
 
     KEYSIZE     = 256*2
     """
-    X25519 key size in bits. A complete key is the concatenation of a 256 bit encryption key, and a 256 bit signing key.
-    """   
+    X.25519 key size in bits. A complete key is the concatenation of a 256 bit encryption key, and a 256 bit signing key.
+    """
+
+    RATCHETSIZE = 256
+    """
+    X.25519 ratchet key size in bits.
+    """
+
+    RATCHET_EXPIRY = 60*60*24*30
+    """
+    The expiry time for received ratchets in seconds, defaults to 30 days. Reticulum will always use the most recently
+    announced ratchet, and remember it for up to ``RATCHET_EXPIRY`` since receiving it, after which it will be discarded.
+    If a newer ratchet is announced in the meantime, it will be replace the already known ratchet.
+    """
 
     # Non-configurable constants
     FERNET_OVERHEAD           = RNS.Cryptography.Fernet.FERNET_OVERHEAD
@@ -67,6 +80,9 @@ class Identity:
 
     # Storage
     known_destinations = {}
+    known_ratchets = {}
+
+    ratchet_persist_lock = threading.Lock()
 
     @staticmethod
     def remember(packet_hash, destination_hash, public_key, app_data = None):
@@ -91,6 +107,13 @@ class Identity:
             identity.app_data = identity_data[3]
             return identity
         else:
+            for registered_destination in RNS.Transport.destinations:
+                if destination_hash == registered_destination.hash:
+                    identity = Identity(create_keys=False)
+                    identity.load_public_key(registered_destination.identity.get_public_key())
+                    identity.app_data = None
+                    return identity
+
             return None
 
     @staticmethod
@@ -138,9 +161,12 @@ class Identity:
                 except:
                     pass
 
-            for destination_hash in storage_known_destinations:
-                if not destination_hash in Identity.known_destinations:
-                    Identity.known_destinations[destination_hash] = storage_known_destinations[destination_hash]
+            try:
+                for destination_hash in storage_known_destinations:
+                    if not destination_hash in Identity.known_destinations:
+                        Identity.known_destinations[destination_hash] = storage_known_destinations[destination_hash]
+            except Exception as e:
+                RNS.log("Skipped recombining known destinations from disk, since an error occurred: "+str(e), RNS.LOG_WARNING)
 
             RNS.log("Saving "+str(len(Identity.known_destinations))+" known destinations to storage...", RNS.LOG_DEBUG)
             file = open(RNS.Reticulum.storagepath+"/known_destinations","wb")
@@ -157,6 +183,7 @@ class Identity:
 
         except Exception as e:
             RNS.log("Error while saving known destinations to disk, the contained exception was: "+str(e), RNS.LOG_ERROR)
+            RNS.trace_exception(e)
 
         Identity.saving_known_destinations = False
 
@@ -174,7 +201,8 @@ class Identity:
                         Identity.known_destinations[known_destination] = loaded_known_destinations[known_destination]
 
                 RNS.log("Loaded "+str(len(Identity.known_destinations))+" known destination from storage", RNS.LOG_VERBOSE)
-            except:
+
+            except Exception as e:
                 RNS.log("Error loading known destinations from disk, file will be recreated on exit", RNS.LOG_ERROR)
         else:
             RNS.log("Destinations file does not exist, no known destinations loaded", RNS.LOG_VERBOSE)
@@ -185,7 +213,7 @@ class Identity:
         Get a SHA-256 hash of passed data.
 
         :param data: Data to be hashed as *bytes*.
-        :returns: SHA-256 hash as *bytes*
+        :returns: SHA-256 hash as *bytes*.
         """
         return RNS.Cryptography.sha256(data)
 
@@ -195,7 +223,7 @@ class Identity:
         Get a truncated SHA-256 hash of passed data.
 
         :param data: Data to be hashed as *bytes*.
-        :returns: Truncated SHA-256 hash as *bytes*
+        :returns: Truncated SHA-256 hash as *bytes*.
         """
         return Identity.full_hash(data)[:(Identity.TRUNCATED_HASHLENGTH//8)]
 
@@ -205,24 +233,158 @@ class Identity:
         Get a random SHA-256 hash.
 
         :param data: Data to be hashed as *bytes*.
-        :returns: Truncated SHA-256 hash of random data as *bytes*
+        :returns: Truncated SHA-256 hash of random data as *bytes*.
         """
         return Identity.truncated_hash(os.urandom(Identity.TRUNCATED_HASHLENGTH//8))
 
     @staticmethod
-    def validate_announce(packet):
+    def current_ratchet_id(destination_hash):
+        """
+        Get the ID of the currently used ratchet key for a given destination hash
+
+        :param destination_hash: A destination hash as *bytes*.
+        :returns: A ratchet ID as *bytes* or *None*.
+        """
+        ratchet = Identity.get_ratchet(destination_hash)
+        if ratchet == None:
+            return None
+        else:
+            return Identity._get_ratchet_id(ratchet)
+
+    @staticmethod
+    def _get_ratchet_id(ratchet_pub_bytes):
+        return Identity.full_hash(ratchet_pub_bytes)[:Identity.NAME_HASH_LENGTH//8]
+
+    @staticmethod
+    def _ratchet_public_bytes(ratchet):
+        return X25519PrivateKey.from_private_bytes(ratchet).public_key().public_bytes()
+
+    @staticmethod
+    def _generate_ratchet():
+        ratchet_prv = X25519PrivateKey.generate()
+        ratchet_pub = ratchet_prv.public_key()
+        return ratchet_prv.private_bytes()
+
+    @staticmethod
+    def _remember_ratchet(destination_hash, ratchet):
+        # TODO: Remove at some point, and only log new ratchets
+        RNS.log(f"Remembering ratchet {RNS.prettyhexrep(Identity._get_ratchet_id(ratchet))} for {RNS.prettyhexrep(destination_hash)}", RNS.LOG_EXTREME)
+        try:
+            Identity.known_ratchets[destination_hash] = ratchet
+
+            if not RNS.Transport.owner.is_connected_to_shared_instance:
+                def persist_job():
+                    with Identity.ratchet_persist_lock:
+                        hexhash = RNS.hexrep(destination_hash, delimit=False)
+                        ratchet_data = {"ratchet": ratchet, "received": time.time()}
+
+                        ratchetdir = RNS.Reticulum.storagepath+"/ratchets"
+                        
+                        if not os.path.isdir(ratchetdir):
+                            os.makedirs(ratchetdir)
+
+                        outpath   = f"{ratchetdir}/{hexhash}.out"
+                        finalpath = f"{ratchetdir}/{hexhash}"
+                        ratchet_file = open(outpath, "wb")
+                        ratchet_file.write(umsgpack.packb(ratchet_data))
+                        ratchet_file.close()
+                        os.replace(outpath, finalpath)
+
+                
+                threading.Thread(target=persist_job, daemon=True).start()
+
+        except Exception as e:
+            RNS.log(f"Could not persist ratchet for {RNS.prettyhexrep(destination_hash)} to storage.", RNS.LOG_ERROR)
+            RNS.log(f"The contained exception was: {e}")
+            RNS.trace_exception(e)
+
+    @staticmethod
+    def _clean_ratchets():
+        RNS.log("Cleaning ratchets...", RNS.LOG_DEBUG)
+        try:
+            now = time.time()
+            ratchetdir = RNS.Reticulum.storagepath+"/ratchets"
+            if os.path.isdir(ratchetdir):
+                for filename in os.listdir(ratchetdir):
+                    try:
+                        expired = False
+                        with open(f"{ratchetdir}/{filename}", "rb") as rf:
+                            ratchet_data = umsgpack.unpackb(rf.read())
+                            if now > ratchet_data["received"]+Identity.RATCHET_EXPIRY:
+                                expired = True
+
+                        if expired:
+                            os.unlink(f"{ratchetdir}/{filename}")
+
+                    except Exception as e:
+                        RNS.log(f"An error occurred while cleaning ratchets, in the processing of {ratchetdir}/{filename}.", RNS.LOG_ERROR)
+                        RNS.log(f"The contained exception was: {e}", RNS.LOG_ERROR)
+
+        except Exception as e:
+            RNS.log(f"An error occurred while cleaning ratchets. The contained exception was: {e}", RNS.LOG_ERROR)
+
+    @staticmethod
+    def get_ratchet(destination_hash):
+        if not destination_hash in Identity.known_ratchets:
+            ratchetdir = RNS.Reticulum.storagepath+"/ratchets"
+            hexhash = RNS.hexrep(destination_hash, delimit=False)
+            ratchet_path = f"{ratchetdir}/{hexhash}"
+            if os.path.isfile(ratchet_path):
+                try:
+                    ratchet_file = open(ratchet_path, "rb")
+                    ratchet_data = umsgpack.unpackb(ratchet_file.read())
+                    if time.time() < ratchet_data["received"]+Identity.RATCHET_EXPIRY and len(ratchet_data["ratchet"]) == Identity.RATCHETSIZE//8:
+                        Identity.known_ratchets[destination_hash] = ratchet_data["ratchet"]
+                    else:
+                        return None
+                
+                except Exception as e:
+                    RNS.log(f"An error occurred while loading ratchet data for {RNS.prettyhexrep(destination_hash)} from storage.", RNS.LOG_ERROR)
+                    RNS.log(f"The contained exception was: {e}", RNS.LOG_ERROR)
+                    return None
+
+        if destination_hash in Identity.known_ratchets:
+            return Identity.known_ratchets[destination_hash]
+        else:
+            RNS.log(f"Could not load ratchet for {RNS.prettyhexrep(destination_hash)}", RNS.LOG_DEBUG)
+            return None
+
+    @staticmethod
+    def validate_announce(packet, only_validate_signature=False):
         try:
             if packet.packet_type == RNS.Packet.ANNOUNCE:
+                keysize       = Identity.KEYSIZE//8
+                ratchetsize   = Identity.RATCHETSIZE//8
+                name_hash_len = Identity.NAME_HASH_LENGTH//8
+                sig_len       = Identity.SIGLENGTH//8
                 destination_hash = packet.destination_hash
-                public_key = packet.data[:Identity.KEYSIZE//8]
-                name_hash = packet.data[Identity.KEYSIZE//8:Identity.KEYSIZE//8+Identity.NAME_HASH_LENGTH//8]
-                random_hash = packet.data[Identity.KEYSIZE//8+Identity.NAME_HASH_LENGTH//8:Identity.KEYSIZE//8+Identity.NAME_HASH_LENGTH//8+10]
-                signature = packet.data[Identity.KEYSIZE//8+Identity.NAME_HASH_LENGTH//8+10:Identity.KEYSIZE//8+Identity.NAME_HASH_LENGTH//8+10+Identity.SIGLENGTH//8]
-                app_data = b""
-                if len(packet.data) > Identity.KEYSIZE//8+Identity.NAME_HASH_LENGTH//8+10+Identity.SIGLENGTH//8:
-                    app_data = packet.data[Identity.KEYSIZE//8+Identity.NAME_HASH_LENGTH//8+10+Identity.SIGLENGTH//8:]
 
-                signed_data = destination_hash+public_key+name_hash+random_hash+app_data
+                # Get public key bytes from announce
+                public_key = packet.data[:keysize]
+
+                # If the packet context flag is set,
+                # this announce contains a new ratchet
+                if packet.context_flag == RNS.Packet.FLAG_SET:
+                    name_hash   = packet.data[keysize:keysize+name_hash_len ]
+                    random_hash = packet.data[keysize+name_hash_len:keysize+name_hash_len+10]
+                    ratchet     = packet.data[keysize+name_hash_len+10:keysize+name_hash_len+10+ratchetsize]
+                    signature   = packet.data[keysize+name_hash_len+10+ratchetsize:keysize+name_hash_len+10+ratchetsize+sig_len]
+                    app_data    = b""
+                    if len(packet.data) > keysize+name_hash_len+10+sig_len+ratchetsize:
+                        app_data = packet.data[keysize+name_hash_len+10+sig_len+ratchetsize:]
+
+                # If the packet context flag is not set,
+                # this announce does not contain a ratchet
+                else:
+                    ratchet     = b""
+                    name_hash   = packet.data[keysize:keysize+name_hash_len]
+                    random_hash = packet.data[keysize+name_hash_len:keysize+name_hash_len+10]
+                    signature   = packet.data[keysize+name_hash_len+10:keysize+name_hash_len+10+sig_len]
+                    app_data    = b""
+                    if len(packet.data) > keysize+name_hash_len+10+sig_len:
+                        app_data = packet.data[keysize+name_hash_len+10+sig_len:]
+
+                signed_data = destination_hash+public_key+name_hash+random_hash+ratchet+app_data
 
                 if not len(packet.data) > Identity.KEYSIZE//8+Identity.NAME_HASH_LENGTH//8+10+Identity.SIGLENGTH//8:
                     app_data = None
@@ -231,6 +393,10 @@ class Identity:
                 announced_identity.load_public_key(public_key)
 
                 if announced_identity.pub != None and announced_identity.validate(signature, signed_data):
+                    if only_validate_signature:
+                        del announced_identity
+                        return True
+
                     hash_material = name_hash+announced_identity.hash
                     expected_hash = RNS.Identity.full_hash(hash_material)[:RNS.Reticulum.TRUNCATED_HASHLENGTH//8]
 
@@ -248,10 +414,25 @@ class Identity:
                         RNS.Identity.remember(packet.get_hash(), destination_hash, public_key, app_data)
                         del announced_identity
 
-                        if hasattr(packet, "transport_id") and packet.transport_id != None:
-                            RNS.log("Valid announce for "+RNS.prettyhexrep(destination_hash)+" "+str(packet.hops)+" hops away, received via "+RNS.prettyhexrep(packet.transport_id)+" on "+str(packet.receiving_interface), RNS.LOG_EXTREME)
+                        if packet.rssi != None or packet.snr != None:
+                            signal_str = " ["
+                            if packet.rssi != None:
+                                signal_str += "RSSI "+str(packet.rssi)+"dBm"
+                                if packet.snr != None:
+                                    signal_str += ", "
+                            if packet.snr != None:
+                                signal_str += "SNR "+str(packet.snr)+"dB"
+                            signal_str += "]"
                         else:
-                            RNS.log("Valid announce for "+RNS.prettyhexrep(destination_hash)+" "+str(packet.hops)+" hops away, received on "+str(packet.receiving_interface), RNS.LOG_EXTREME)
+                            signal_str = ""
+
+                        if hasattr(packet, "transport_id") and packet.transport_id != None:
+                            RNS.log("Valid announce for "+RNS.prettyhexrep(destination_hash)+" "+str(packet.hops)+" hops away, received via "+RNS.prettyhexrep(packet.transport_id)+" on "+str(packet.receiving_interface)+signal_str, RNS.LOG_EXTREME)
+                        else:
+                            RNS.log("Valid announce for "+RNS.prettyhexrep(destination_hash)+" "+str(packet.hops)+" hops away, received on "+str(packet.receiving_interface)+signal_str, RNS.LOG_EXTREME)
+
+                        if ratchet:
+                            Identity._remember_ratchet(destination_hash, ratchet)
 
                         return True
 
@@ -433,7 +614,7 @@ class Identity:
             return False
         except Exception as e:
             RNS.log("Error while loading identity from "+str(path), RNS.LOG_ERROR)
-            RNS.log("The contained exception was: "+str(e))
+            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
 
     def get_salt(self):
         return self.hash
@@ -441,7 +622,7 @@ class Identity:
     def get_context(self):
         return None
 
-    def encrypt(self, plaintext):
+    def encrypt(self, plaintext, ratchet=None):
         """
         Encrypts information for the identity.
 
@@ -453,7 +634,12 @@ class Identity:
             ephemeral_key = X25519PrivateKey.generate()
             ephemeral_pub_bytes = ephemeral_key.public_key().public_bytes()
 
-            shared_key = ephemeral_key.exchange(self.pub)
+            if ratchet != None:
+                target_public_key = X25519PublicKey.from_public_bytes(ratchet)
+            else:
+                target_public_key = self.pub
+
+            shared_key = ephemeral_key.exchange(target_public_key)
             
             derived_key = RNS.Cryptography.hkdf(
                 length=32,
@@ -471,7 +657,7 @@ class Identity:
             raise KeyError("Encryption failed because identity does not hold a public key")
 
 
-    def decrypt(self, ciphertext_token):
+    def decrypt(self, ciphertext_token, ratchets=None, enforce_ratchets=False, ratchet_id_receiver=None):
         """
         Decrypts information for the identity.
 
@@ -485,22 +671,55 @@ class Identity:
                 try:
                     peer_pub_bytes = ciphertext_token[:Identity.KEYSIZE//8//2]
                     peer_pub = X25519PublicKey.from_public_bytes(peer_pub_bytes)
-
-                    shared_key = self.prv.exchange(peer_pub)
-
-                    derived_key = RNS.Cryptography.hkdf(
-                        length=32,
-                        derive_from=shared_key,
-                        salt=self.get_salt(),
-                        context=self.get_context(),
-                    )
-
-                    fernet = Fernet(derived_key)
                     ciphertext = ciphertext_token[Identity.KEYSIZE//8//2:]
-                    plaintext = fernet.decrypt(ciphertext)
+
+                    if ratchets:
+                        for ratchet in ratchets:
+                            try:
+                                ratchet_prv = X25519PrivateKey.from_private_bytes(ratchet)
+                                ratchet_id = Identity._get_ratchet_id(ratchet_prv.public_key().public_bytes())
+                                shared_key = ratchet_prv.exchange(peer_pub)
+                                derived_key = RNS.Cryptography.hkdf(
+                                    length=32,
+                                    derive_from=shared_key,
+                                    salt=self.get_salt(),
+                                    context=self.get_context(),
+                                )
+
+                                fernet = Fernet(derived_key)
+                                plaintext = fernet.decrypt(ciphertext)
+                                if ratchet_id_receiver:
+                                    ratchet_id_receiver.latest_ratchet_id = ratchet_id
+                                
+                                break
+                            
+                            except Exception as e:
+                                pass
+
+                    if enforce_ratchets and plaintext == None:
+                        RNS.log("Decryption with ratchet enforcement by "+RNS.prettyhexrep(self.hash)+" failed. Dropping packet.", RNS.LOG_DEBUG)
+                        if ratchet_id_receiver:
+                            ratchet_id_receiver.latest_ratchet_id = None
+                        return None
+
+                    if plaintext == None:
+                        shared_key = self.prv.exchange(peer_pub)
+                        derived_key = RNS.Cryptography.hkdf(
+                            length=32,
+                            derive_from=shared_key,
+                            salt=self.get_salt(),
+                            context=self.get_context(),
+                        )
+
+                        fernet = Fernet(derived_key)
+                        plaintext = fernet.decrypt(ciphertext)
+                        if ratchet_id_receiver:
+                            ratchet_id_receiver.latest_ratchet_id = None
 
                 except Exception as e:
                     RNS.log("Decryption by "+RNS.prettyhexrep(self.hash)+" failed: "+str(e), RNS.LOG_DEBUG)
+                    if ratchet_id_receiver:
+                        ratchet_id_receiver.latest_ratchet_id = None
                     
                 return plaintext;
             else:

RNS/Interfaces/AX25KISSInterface.py

@@ -77,8 +77,7 @@ class AX25KISSInterface(Interface):
             RNS.log("You can install one with the command: python3 -m pip install pyserial", RNS.LOG_CRITICAL)
             RNS.panic()
 
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
 
         self.HW_MTU = 564
         
@@ -97,7 +96,7 @@ class AX25KISSInterface(Interface):
         self.stopbits = stopbits
         self.timeout  = 100
         self.online   = False
-        self.bitrate  = KISSInterface.BITRATE_GUESS
+        self.bitrate  = AX25KISSInterface.BITRATE_GUESS
 
         self.packet_queue    = []
         self.flow_control    = flow_control
@@ -367,5 +366,8 @@ class AX25KISSInterface(Interface):
 
         RNS.log("Reconnected serial port for "+str(self))
 
+    def should_ingress_limit(self):
+        return False
+
     def __str__(self):
         return "AX25KISSInterface["+self.name+"]"

RNS/Interfaces/Android/KISSInterface.py

@@ -0,0 +1,409 @@
+# MIT License
+#
+# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from RNS.Interfaces.Interface import Interface
+from time import sleep
+import sys
+import threading
+import time
+import RNS
+
+class KISS():
+    FEND              = 0xC0
+    FESC              = 0xDB
+    TFEND             = 0xDC
+    TFESC             = 0xDD
+    CMD_UNKNOWN       = 0xFE
+    CMD_DATA          = 0x00
+    CMD_TXDELAY       = 0x01
+    CMD_P             = 0x02
+    CMD_SLOTTIME      = 0x03
+    CMD_TXTAIL        = 0x04
+    CMD_FULLDUPLEX    = 0x05
+    CMD_SETHARDWARE   = 0x06
+    CMD_READY         = 0x0F
+    CMD_RETURN        = 0xFF
+
+    @staticmethod
+    def escape(data):
+        data = data.replace(bytes([0xdb]), bytes([0xdb, 0xdd]))
+        data = data.replace(bytes([0xc0]), bytes([0xdb, 0xdc]))
+        return data
+
+class KISSInterface(Interface):
+    MAX_CHUNK = 32768
+    BITRATE_GUESS = 1200
+
+    owner    = None
+    port     = None
+    speed    = None
+    databits = None
+    parity   = None
+    stopbits = None
+    serial   = None
+
+    def __init__(self, owner, name, port, speed, databits, parity, stopbits, preamble, txtail, persistence, slottime, flow_control, beacon_interval, beacon_data):
+        import importlib
+        if RNS.vendor.platformutils.is_android():
+            self.on_android  = True
+            if importlib.util.find_spec('usbserial4a') != None:
+                if importlib.util.find_spec('jnius') == None:
+                    RNS.log("Could not load jnius API wrapper for Android, KISS interface cannot be created.", RNS.LOG_CRITICAL)
+                    RNS.log("This probably means you are trying to use an USB-based interface from within Termux or similar.", RNS.LOG_CRITICAL)
+                    RNS.log("This is currently not possible, due to this environment limiting access to the native Android APIs.", RNS.LOG_CRITICAL)
+                    RNS.panic()
+
+                from usbserial4a import serial4a as serial
+                self.parity = "N"
+            
+            else:
+                RNS.log("Could not load USB serial module for Android, KISS interface cannot be created.", RNS.LOG_CRITICAL)
+                RNS.log("You can install this module by issuing: pip install usbserial4a", RNS.LOG_CRITICAL)
+                RNS.panic()
+        else:
+            raise SystemError("Android-specific interface was used on non-Android OS")
+
+        super().__init__()
+        
+        self.HW_MTU = 564
+        
+        if beacon_data == None:
+            beacon_data = ""
+
+        self.pyserial = serial
+        self.serial   = None
+        self.owner    = owner
+        self.name     = name
+        self.port     = port
+        self.speed    = speed
+        self.databits = databits
+        self.parity   = "N"
+        self.stopbits = stopbits
+        self.timeout  = 100
+        self.online   = False
+        self.beacon_i = beacon_interval
+        self.beacon_d = beacon_data.encode("utf-8")
+        self.first_tx = None
+        self.bitrate  = KISSInterface.BITRATE_GUESS
+
+        self.packet_queue    = []
+        self.flow_control    = flow_control
+        self.interface_ready = False
+        self.flow_control_timeout = 5
+        self.flow_control_locked  = time.time()
+
+        self.preamble    = preamble if preamble != None else 350;
+        self.txtail      = txtail if txtail != None else 20;
+        self.persistence = persistence if persistence != None else 64;
+        self.slottime    = slottime if slottime != None else 20;
+
+        if parity.lower() == "e" or parity.lower() == "even":
+            self.parity = "E"
+
+        if parity.lower() == "o" or parity.lower() == "odd":
+            self.parity = "O"
+
+        try:
+            self.open_port()
+        except Exception as e:
+            RNS.log("Could not open serial port "+self.port, RNS.LOG_ERROR)
+            raise e
+
+        if self.serial.is_open:
+            self.configure_device()
+        else:
+            raise IOError("Could not open serial port")
+
+
+    def open_port(self):
+        RNS.log("Opening serial port "+self.port+"...")
+        # Get device parameters
+        from usb4a import usb
+        device = usb.get_usb_device(self.port)
+        if device:
+            vid = device.getVendorId()
+            pid = device.getProductId()
+
+            # Driver overrides for speficic chips
+            proxy = self.pyserial.get_serial_port
+            if vid == 0x1A86 and pid == 0x55D4:
+                # Force CDC driver for Qinheng CH34x
+                RNS.log(str(self)+" using CDC driver for "+RNS.hexrep(vid)+":"+RNS.hexrep(pid), RNS.LOG_DEBUG)
+                from usbserial4a.cdcacmserial4a import CdcAcmSerial
+                proxy = CdcAcmSerial
+
+            self.serial = proxy(
+                self.port,
+                baudrate = self.speed,
+                bytesize = self.databits,
+                parity = self.parity,
+                stopbits = self.stopbits,
+                xonxoff = False,
+                rtscts = False,
+                timeout = None,
+                inter_byte_timeout = None,
+                # write_timeout = wtimeout,
+                dsrdtr = False,
+            )
+
+            if vid == 0x0403:
+                # Hardware parameters for FTDI devices @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 16 * 1024
+                self.serial.USB_READ_TIMEOUT_MILLIS = 100
+                self.serial.timeout = 0.1
+            elif vid == 0x10C4:
+                # Hardware parameters for SiLabs CP210x @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 64 
+                self.serial.USB_READ_TIMEOUT_MILLIS = 12
+                self.serial.timeout = 0.012
+            elif vid == 0x1A86 and pid == 0x55D4:
+                # Hardware parameters for Qinheng CH34x @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 64
+                self.serial.USB_READ_TIMEOUT_MILLIS = 12
+                self.serial.timeout = 0.1
+            else:
+                # Default values
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 1 * 1024
+                self.serial.USB_READ_TIMEOUT_MILLIS = 100
+                self.serial.timeout = 0.1
+
+            RNS.log(str(self)+" USB read buffer size set to "+RNS.prettysize(self.serial.DEFAULT_READ_BUFFER_SIZE), RNS.LOG_DEBUG)
+            RNS.log(str(self)+" USB read timeout set to "+str(self.serial.USB_READ_TIMEOUT_MILLIS)+"ms", RNS.LOG_DEBUG)
+            RNS.log(str(self)+" USB write timeout set to "+str(self.serial.USB_WRITE_TIMEOUT_MILLIS)+"ms", RNS.LOG_DEBUG)
+
+    def configure_device(self):
+        # Allow time for interface to initialise before config
+        sleep(2.0)
+        thread = threading.Thread(target=self.readLoop)
+        thread.daemon = True
+        thread.start()
+        self.online = True
+        RNS.log("Serial port "+self.port+" is now open")
+        RNS.log("Configuring KISS interface parameters...")
+        self.setPreamble(self.preamble)
+        self.setTxTail(self.txtail)
+        self.setPersistence(self.persistence)
+        self.setSlotTime(self.slottime)
+        self.setFlowControl(self.flow_control)
+        self.interface_ready = True
+        RNS.log("KISS interface configured")
+
+    def setPreamble(self, preamble):
+        preamble_ms = preamble
+        preamble = int(preamble_ms / 10)
+        if preamble < 0:
+            preamble = 0
+        if preamble > 255:
+            preamble = 255
+
+        kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_TXDELAY])+bytes([preamble])+bytes([KISS.FEND])
+        written = self.serial.write(kiss_command)
+        if written != len(kiss_command):
+            raise IOError("Could not configure KISS interface preamble to "+str(preamble_ms)+" (command value "+str(preamble)+")")
+
+    def setTxTail(self, txtail):
+        txtail_ms = txtail
+        txtail = int(txtail_ms / 10)
+        if txtail < 0:
+            txtail = 0
+        if txtail > 255:
+            txtail = 255
+
+        kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_TXTAIL])+bytes([txtail])+bytes([KISS.FEND])
+        written = self.serial.write(kiss_command)
+        if written != len(kiss_command):
+            raise IOError("Could not configure KISS interface TX tail to "+str(txtail_ms)+" (command value "+str(txtail)+")")
+
+    def setPersistence(self, persistence):
+        if persistence < 0:
+            persistence = 0
+        if persistence > 255:
+            persistence = 255
+
+        kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_P])+bytes([persistence])+bytes([KISS.FEND])
+        written = self.serial.write(kiss_command)
+        if written != len(kiss_command):
+            raise IOError("Could not configure KISS interface persistence to "+str(persistence))
+
+    def setSlotTime(self, slottime):
+        slottime_ms = slottime
+        slottime = int(slottime_ms / 10)
+        if slottime < 0:
+            slottime = 0
+        if slottime > 255:
+            slottime = 255
+
+        kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_SLOTTIME])+bytes([slottime])+bytes([KISS.FEND])
+        written = self.serial.write(kiss_command)
+        if written != len(kiss_command):
+            raise IOError("Could not configure KISS interface slot time to "+str(slottime_ms)+" (command value "+str(slottime)+")")
+
+    def setFlowControl(self, flow_control):
+        kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_READY])+bytes([0x01])+bytes([KISS.FEND])
+        written = self.serial.write(kiss_command)
+        if written != len(kiss_command):
+            if (flow_control):
+                raise IOError("Could not enable KISS interface flow control")
+            else:
+                raise IOError("Could not enable KISS interface flow control")
+
+
+    def processIncoming(self, data):
+        self.rxb += len(data)
+        def af():
+            self.owner.inbound(data, self)
+        threading.Thread(target=af, daemon=True).start()
+
+    def processOutgoing(self,data):
+        datalen = len(data)
+        if self.online:
+            if self.interface_ready:
+                if self.flow_control:
+                    self.interface_ready = False
+                    self.flow_control_locked = time.time()
+
+                data = data.replace(bytes([0xdb]), bytes([0xdb])+bytes([0xdd]))
+                data = data.replace(bytes([0xc0]), bytes([0xdb])+bytes([0xdc]))
+                frame = bytes([KISS.FEND])+bytes([0x00])+data+bytes([KISS.FEND])
+
+                written = self.serial.write(frame)
+                self.txb += datalen
+
+                if data == self.beacon_d:
+                    self.first_tx = None
+                else:
+                    if self.first_tx == None:
+                        self.first_tx = time.time()
+
+                if written != len(frame):
+                    raise IOError("Serial interface only wrote "+str(written)+" bytes of "+str(len(data)))
+
+            else:
+                self.queue(data)
+
+    def queue(self, data):
+        self.packet_queue.append(data)
+
+    def process_queue(self):
+        if len(self.packet_queue) > 0:
+            data = self.packet_queue.pop(0)
+            self.interface_ready = True
+            self.processOutgoing(data)
+        elif len(self.packet_queue) == 0:
+            self.interface_ready = True
+
+    def readLoop(self):
+        try:
+            in_frame = False
+            escape = False
+            command = KISS.CMD_UNKNOWN
+            data_buffer = b""
+            last_read_ms = int(time.time()*1000)
+
+            while self.serial.is_open:
+                serial_bytes = self.serial.read()
+                got = len(serial_bytes)
+
+                for byte in serial_bytes:
+                    last_read_ms = int(time.time()*1000)
+
+                    if (in_frame and byte == KISS.FEND and command == KISS.CMD_DATA):
+                        in_frame = False
+                        self.processIncoming(data_buffer)
+                    elif (byte == KISS.FEND):
+                        in_frame = True
+                        command = KISS.CMD_UNKNOWN
+                        data_buffer = b""
+                    elif (in_frame and len(data_buffer) < self.HW_MTU):
+                        if (len(data_buffer) == 0 and command == KISS.CMD_UNKNOWN):
+                            # We only support one HDLC port for now, so
+                            # strip off the port nibble
+                            byte = byte & 0x0F
+                            command = byte
+                        elif (command == KISS.CMD_DATA):
+                            if (byte == KISS.FESC):
+                                escape = True
+                            else:
+                                if (escape):
+                                    if (byte == KISS.TFEND):
+                                        byte = KISS.FEND
+                                    if (byte == KISS.TFESC):
+                                        byte = KISS.FESC
+                                    escape = False
+                                data_buffer = data_buffer+bytes([byte])
+                        elif (command == KISS.CMD_READY):
+                            self.process_queue()
+                
+                if got == 0:
+                    time_since_last = int(time.time()*1000) - last_read_ms
+                    if len(data_buffer) > 0 and time_since_last > self.timeout:
+                        data_buffer = b""
+                        in_frame = False
+                        command = KISS.CMD_UNKNOWN
+                        escape = False
+                    sleep(0.05)
+
+                    if self.flow_control:
+                        if not self.interface_ready:
+                            if time.time() > self.flow_control_locked + self.flow_control_timeout:
+                                RNS.log("Interface "+str(self)+" is unlocking flow control due to time-out. This should not happen. Your hardware might have missed a flow-control READY command, or maybe it does not support flow-control.", RNS.LOG_WARNING)
+                                self.process_queue()
+
+                    if self.beacon_i != None and self.beacon_d != None:
+                        if self.first_tx != None:
+                            if time.time() > self.first_tx + self.beacon_i:
+                                RNS.log("Interface "+str(self)+" is transmitting beacon data: "+str(self.beacon_d.decode("utf-8")), RNS.LOG_DEBUG)
+                                self.first_tx = None
+                                self.processOutgoing(self.beacon_d)
+
+        except Exception as e:
+            self.online = False
+            RNS.log("A serial port error occurred, the contained exception was: "+str(e), RNS.LOG_ERROR)
+            RNS.log("The interface "+str(self)+" experienced an unrecoverable error and is now offline.", RNS.LOG_ERROR)
+            
+            if RNS.Reticulum.panic_on_interface_error:
+                RNS.panic()
+
+            RNS.log("Reticulum will attempt to reconnect the interface periodically.", RNS.LOG_ERROR)
+
+        self.online = False
+        self.serial.close()
+        self.reconnect_port()
+
+    def reconnect_port(self):
+        while not self.online:
+            try:
+                time.sleep(5)
+                RNS.log("Attempting to reconnect serial port "+str(self.port)+" for "+str(self)+"...", RNS.LOG_VERBOSE)
+                self.open_port()
+                if self.serial.is_open:
+                    self.configure_device()
+            except Exception as e:
+                RNS.log("Error while reconnecting port, the contained exception was: "+str(e), RNS.LOG_ERROR)
+
+        RNS.log("Reconnected serial port for "+str(self))
+
+    def should_ingress_limit(self):
+        return False
+
+    def __str__(self):
+        return "KISSInterface["+self.name+"]"

RNS/Interfaces/Android/SerialInterface.py

@@ -0,0 +1,260 @@
+# MIT License
+#
+# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from RNS.Interfaces.Interface import Interface
+from time import sleep
+import sys
+import threading
+import time
+import RNS
+
+class HDLC():
+    # The Serial Interface packetizes data using
+    # simplified HDLC framing, similar to PPP
+    FLAG              = 0x7E
+    ESC               = 0x7D
+    ESC_MASK          = 0x20
+
+    @staticmethod
+    def escape(data):
+        data = data.replace(bytes([HDLC.ESC]), bytes([HDLC.ESC, HDLC.ESC^HDLC.ESC_MASK]))
+        data = data.replace(bytes([HDLC.FLAG]), bytes([HDLC.ESC, HDLC.FLAG^HDLC.ESC_MASK]))
+        return data
+
+class SerialInterface(Interface):
+    MAX_CHUNK = 32768
+
+    owner    = None
+    port     = None
+    speed    = None
+    databits = None
+    parity   = None
+    stopbits = None
+    serial   = None
+
+    def __init__(self, owner, name, port, speed, databits, parity, stopbits):
+        import importlib
+        if RNS.vendor.platformutils.is_android():
+            self.on_android  = True
+            if importlib.util.find_spec('usbserial4a') != None:
+                if importlib.util.find_spec('jnius') == None:
+                    RNS.log("Could not load jnius API wrapper for Android, Serial interface cannot be created.", RNS.LOG_CRITICAL)
+                    RNS.log("This probably means you are trying to use an USB-based interface from within Termux or similar.", RNS.LOG_CRITICAL)
+                    RNS.log("This is currently not possible, due to this environment limiting access to the native Android APIs.", RNS.LOG_CRITICAL)
+                    RNS.panic()
+
+                from usbserial4a import serial4a as serial
+                self.parity = "N"
+            
+            else:
+                RNS.log("Could not load USB serial module for Android, Serial interface cannot be created.", RNS.LOG_CRITICAL)
+                RNS.log("You can install this module by issuing: pip install usbserial4a", RNS.LOG_CRITICAL)
+                RNS.panic()
+        else:
+            raise SystemError("Android-specific interface was used on non-Android OS")
+
+        super().__init__()
+
+        self.HW_MTU = 564
+        
+        self.pyserial = serial
+        self.serial   = None
+        self.owner    = owner
+        self.name     = name
+        self.port     = port
+        self.speed    = speed
+        self.databits = databits
+        self.parity   = "N"
+        self.stopbits = stopbits
+        self.timeout  = 100
+        self.online   = False
+        self.bitrate  = self.speed
+
+        if parity.lower() == "e" or parity.lower() == "even":
+            self.parity = "E"
+
+        if parity.lower() == "o" or parity.lower() == "odd":
+            self.parity = "O"
+
+        try:
+            self.open_port()
+        except Exception as e:
+            RNS.log("Could not open serial port for interface "+str(self), RNS.LOG_ERROR)
+            raise e
+
+        if self.serial.is_open:
+            self.configure_device()
+        else:
+            raise IOError("Could not open serial port")
+
+
+    def open_port(self):
+        RNS.log("Opening serial port "+self.port+"...")
+        # Get device parameters
+        from usb4a import usb
+        device = usb.get_usb_device(self.port)
+        if device:
+            vid = device.getVendorId()
+            pid = device.getProductId()
+
+            # Driver overrides for speficic chips
+            proxy = self.pyserial.get_serial_port
+            if vid == 0x1A86 and pid == 0x55D4:
+                # Force CDC driver for Qinheng CH34x
+                RNS.log(str(self)+" using CDC driver for "+RNS.hexrep(vid)+":"+RNS.hexrep(pid), RNS.LOG_DEBUG)
+                from usbserial4a.cdcacmserial4a import CdcAcmSerial
+                proxy = CdcAcmSerial
+
+            self.serial = proxy(
+                self.port,
+                baudrate = self.speed,
+                bytesize = self.databits,
+                parity = self.parity,
+                stopbits = self.stopbits,
+                xonxoff = False,
+                rtscts = False,
+                timeout = None,
+                inter_byte_timeout = None,
+                # write_timeout = wtimeout,
+                dsrdtr = False,
+            )
+
+            if vid == 0x0403:
+                # Hardware parameters for FTDI devices @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 16 * 1024
+                self.serial.USB_READ_TIMEOUT_MILLIS = 100
+                self.serial.timeout = 0.1
+            elif vid == 0x10C4:
+                # Hardware parameters for SiLabs CP210x @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 64 
+                self.serial.USB_READ_TIMEOUT_MILLIS = 12
+                self.serial.timeout = 0.012
+            elif vid == 0x1A86 and pid == 0x55D4:
+                # Hardware parameters for Qinheng CH34x @ 115200 baud
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 64
+                self.serial.USB_READ_TIMEOUT_MILLIS = 12
+                self.serial.timeout = 0.1
+            else:
+                # Default values
+                self.serial.DEFAULT_READ_BUFFER_SIZE = 1 * 1024
+                self.serial.USB_READ_TIMEOUT_MILLIS = 100
+                self.serial.timeout = 0.1
+
+            RNS.log(str(self)+" USB read buffer size set to "+RNS.prettysize(self.serial.DEFAULT_READ_BUFFER_SIZE), RNS.LOG_DEBUG)
+            RNS.log(str(self)+" USB read timeout set to "+str(self.serial.USB_READ_TIMEOUT_MILLIS)+"ms", RNS.LOG_DEBUG)
+            RNS.log(str(self)+" USB write timeout set to "+str(self.serial.USB_WRITE_TIMEOUT_MILLIS)+"ms", RNS.LOG_DEBUG)
+
+    def configure_device(self):
+        sleep(0.5)
+        thread = threading.Thread(target=self.readLoop)
+        thread.daemon = True
+        thread.start()
+        self.online = True
+        RNS.log("Serial port "+self.port+" is now open", RNS.LOG_VERBOSE)
+
+
+    def processIncoming(self, data):
+        self.rxb += len(data)
+        def af():
+            self.owner.inbound(data, self)
+        threading.Thread(target=af, daemon=True).start()
+
+    def processOutgoing(self,data):
+        if self.online:
+            data = bytes([HDLC.FLAG])+HDLC.escape(data)+bytes([HDLC.FLAG])
+            written = self.serial.write(data)
+            self.txb += len(data)            
+            if written != len(data):
+                raise IOError("Serial interface only wrote "+str(written)+" bytes of "+str(len(data)))
+
+    def readLoop(self):
+        try:
+            in_frame = False
+            escape = False
+            data_buffer = b""
+            last_read_ms = int(time.time()*1000)
+
+            while self.serial.is_open:
+                serial_bytes = self.serial.read()
+                got = len(serial_bytes)
+
+                for byte in serial_bytes:
+                    last_read_ms = int(time.time()*1000)
+
+                    if (in_frame and byte == HDLC.FLAG):
+                        in_frame = False
+                        self.processIncoming(data_buffer)
+                    elif (byte == HDLC.FLAG):
+                        in_frame = True
+                        data_buffer = b""
+                    elif (in_frame and len(data_buffer) < self.HW_MTU):
+                        if (byte == HDLC.ESC):
+                            escape = True
+                        else:
+                            if (escape):
+                                if (byte == HDLC.FLAG ^ HDLC.ESC_MASK):
+                                    byte = HDLC.FLAG
+                                if (byte == HDLC.ESC  ^ HDLC.ESC_MASK):
+                                    byte = HDLC.ESC
+                                escape = False
+                            data_buffer = data_buffer+bytes([byte])
+                        
+                if got == 0:
+                    time_since_last = int(time.time()*1000) - last_read_ms
+                    if len(data_buffer) > 0 and time_since_last > self.timeout:
+                        data_buffer = b""
+                        in_frame = False
+                        escape = False
+                    # sleep(0.08)
+                    
+        except Exception as e:
+            self.online = False
+            RNS.log("A serial port error occurred, the contained exception was: "+str(e), RNS.LOG_ERROR)
+            RNS.log("The interface "+str(self)+" experienced an unrecoverable error and is now offline.", RNS.LOG_ERROR)
+            
+            if RNS.Reticulum.panic_on_interface_error:
+                RNS.panic()
+
+            RNS.log("Reticulum will attempt to reconnect the interface periodically.", RNS.LOG_ERROR)
+
+        self.online = False
+        self.serial.close()
+        self.reconnect_port()
+
+    def reconnect_port(self):
+        while not self.online:
+            try:
+                time.sleep(5)
+                RNS.log("Attempting to reconnect serial port "+str(self.port)+" for "+str(self)+"...", RNS.LOG_VERBOSE)
+                self.open_port()
+                if self.serial.is_open:
+                    self.configure_device()
+            except Exception as e:
+                RNS.log("Error while reconnecting port, the contained exception was: "+str(e), RNS.LOG_ERROR)
+
+        RNS.log("Reconnected serial port for "+str(self))
+
+    def should_ingress_limit(self):
+        return False
+
+    def __str__(self):
+        return "SerialInterface["+self.name+"]"

RNS/Interfaces/Android/__init__.py

@@ -0,0 +1,27 @@
+# MIT License
+#
+# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import os
+import glob
+
+modules = glob.glob(os.path.dirname(__file__)+"/*.py")
+__all__ = [ os.path.basename(f)[:-3] for f in modules if not f.endswith('__init__.py')]

RNS/Interfaces/AutoInterface.py

@@ -21,8 +21,10 @@
 # SOFTWARE.
 
 from .Interface import Interface
+from collections import deque
 import socketserver
 import threading
+import re
 import socket
 import struct
 import time
@@ -41,25 +43,53 @@ class AutoInterface(Interface):
     SCOPE_ORGANISATION = "8"
     SCOPE_GLOBAL       = "e"
 
+    MULTICAST_PERMANENT_ADDRESS_TYPE = "0"
+    MULTICAST_TEMPORARY_ADDRESS_TYPE = "1"
+
     PEERING_TIMEOUT    = 7.5
 
+    ALL_IGNORE_IFS     = ["lo0"]
     DARWIN_IGNORE_IFS  = ["awdl0", "llw0", "lo0", "en5"]
     ANDROID_IGNORE_IFS = ["dummy0", "lo", "tun0"]
 
     BITRATE_GUESS      = 10*1000*1000
 
-    def __init__(self, owner, name, group_id=None, discovery_scope=None, discovery_port=None, data_port=None, allowed_interfaces=None, ignored_interfaces=None, configured_bitrate=None):
-        import importlib
-        if importlib.util.find_spec('netifaces') != None:
-            import netifaces
-        else:
-            RNS.log("Using AutoInterface requires the netifaces module.", RNS.LOG_CRITICAL)
-            RNS.log("You can install it with the command: python3 -m pip install netifaces", RNS.LOG_CRITICAL)
-            RNS.panic()
+    MULTI_IF_DEQUE_LEN = 48
+    MULTI_IF_DEQUE_TTL = 0.75
 
-        self.netifaces = netifaces
-        self.rxb = 0
-        self.txb = 0
+    def handler_factory(self, callback):
+        def create_handler(*args, **keys):
+            return AutoInterfaceHandler(callback, *args, **keys)
+        return create_handler
+
+    def descope_linklocal(self, link_local_addr):
+        # Drop scope specifier expressd as %ifname (macOS)
+        link_local_addr = link_local_addr.split("%")[0]
+        # Drop embedded scope specifier (NetBSD, OpenBSD)
+        link_local_addr = re.sub(r"fe80:[0-9a-f]*::","fe80::", link_local_addr)
+        return link_local_addr
+
+    def list_interfaces(self):
+        ifs = self.netinfo.interfaces()
+        return ifs
+
+    def list_addresses(self, ifname):
+        ifas = self.netinfo.ifaddresses(ifname)
+        return ifas
+
+    def interface_name_to_index(self, ifname):
+
+        # socket.if_nametoindex doesn't work with uuid interface names on windows, it wants the ethernet_0 style
+        # we will just get the index from netinfo instead as it seems to work
+        if RNS.vendor.platformutils.is_windows():
+            return self.netinfo.interface_names_to_indexes()[ifname]
+
+        return socket.if_nametoindex(ifname)
+
+    def __init__(self, owner, name, group_id=None, discovery_scope=None, discovery_port=None, multicast_address_type=None, data_port=None, allowed_interfaces=None, ignored_interfaces=None, configured_bitrate=None):
+        from RNS.vendor.ifaddr import niwrapper
+        super().__init__()
+        self.netinfo = niwrapper
 
         self.HW_MTU = 1064
 
@@ -70,16 +100,26 @@ class AutoInterface(Interface):
         self.peers = {}
         self.link_local_addresses = []
         self.adopted_interfaces = {}
+        self.interface_servers = {}
         self.multicast_echoes = {}
         self.timed_out_interfaces = {}
+        self.mif_deque = deque(maxlen=AutoInterface.MULTI_IF_DEQUE_LEN)
+        self.mif_deque_times = deque(maxlen=AutoInterface.MULTI_IF_DEQUE_LEN)
+        self.carrier_changed = False
 
         self.outbound_udp_socket = None
 
+        self.announce_rate_target = None
         self.announce_interval = AutoInterface.PEERING_TIMEOUT/6.0
         self.peer_job_interval = AutoInterface.PEERING_TIMEOUT*1.1
         self.peering_timeout   = AutoInterface.PEERING_TIMEOUT
         self.multicast_echo_timeout = AutoInterface.PEERING_TIMEOUT/2
 
+        # Increase peering timeout on Android, due to potential
+        # low-power modes implemented on many chipsets.
+        if RNS.vendor.platformutils.is_android():
+            self.peering_timeout *= 3
+
         if allowed_interfaces == None:
             self.allowed_interfaces = []
         else:
@@ -100,6 +140,15 @@ class AutoInterface(Interface):
         else:
             self.discovery_port = discovery_port
 
+        if multicast_address_type == None:
+            self.multicast_address_type = AutoInterface.MULTICAST_TEMPORARY_ADDRESS_TYPE
+        elif str(multicast_address_type).lower() == "temporary":
+            self.multicast_address_type = AutoInterface.MULTICAST_TEMPORARY_ADDRESS_TYPE
+        elif str(multicast_address_type).lower() == "permanent":
+            self.multicast_address_type = AutoInterface.MULTICAST_PERMANENT_ADDRESS_TYPE
+        else:
+            self.multicast_address_type = AutoInterface.MULTICAST_TEMPORARY_ADDRESS_TYPE
+
         if data_port == None:
             self.data_port = AutoInterface.DEFAULT_DATA_PORT
         else:
@@ -128,92 +177,120 @@ class AutoInterface(Interface):
         gt += ":"+"{:02x}".format(g[9]+(g[8]<<8))
         gt += ":"+"{:02x}".format(g[11]+(g[10]<<8))
         gt += ":"+"{:02x}".format(g[13]+(g[12]<<8))
-        self.mcast_discovery_address = "ff1"+self.discovery_scope+":"+gt
+        self.mcast_discovery_address = "ff"+self.multicast_address_type+self.discovery_scope+":"+gt
 
         suitable_interfaces = 0
-        for ifname in self.netifaces.interfaces():
-            if RNS.vendor.platformutils.is_darwin() and ifname in AutoInterface.DARWIN_IGNORE_IFS and not ifname in self.allowed_interfaces:
-                RNS.log(str(self)+" skipping Darwin AWDL or tethering interface "+str(ifname), RNS.LOG_EXTREME)
-            elif RNS.vendor.platformutils.is_darwin() and ifname == "lo0":
-                RNS.log(str(self)+" skipping Darwin loopback interface "+str(ifname), RNS.LOG_EXTREME)
-            elif RNS.vendor.platformutils.is_android() and ifname in AutoInterface.ANDROID_IGNORE_IFS and not ifname in self.allowed_interfaces:
-                RNS.log(str(self)+" skipping Android system interface "+str(ifname), RNS.LOG_EXTREME)
-            elif ifname in self.ignored_interfaces:
-                RNS.log(str(self)+" ignoring disallowed interface "+str(ifname), RNS.LOG_EXTREME)
-            else:
-                if len(self.allowed_interfaces) > 0 and not ifname in self.allowed_interfaces:
-                    RNS.log(str(self)+" ignoring interface "+str(ifname)+" since it was not allowed", RNS.LOG_EXTREME)
+        for ifname in self.list_interfaces():
+            try:
+                if RNS.vendor.platformutils.is_darwin() and ifname in AutoInterface.DARWIN_IGNORE_IFS and not ifname in self.allowed_interfaces:
+                    RNS.log(str(self)+" skipping Darwin AWDL or tethering interface "+str(ifname), RNS.LOG_EXTREME)
+                elif RNS.vendor.platformutils.is_darwin() and ifname == "lo0":
+                    RNS.log(str(self)+" skipping Darwin loopback interface "+str(ifname), RNS.LOG_EXTREME)
+                elif RNS.vendor.platformutils.is_android() and ifname in AutoInterface.ANDROID_IGNORE_IFS and not ifname in self.allowed_interfaces:
+                    RNS.log(str(self)+" skipping Android system interface "+str(ifname), RNS.LOG_EXTREME)
+                elif ifname in self.ignored_interfaces:
+                    RNS.log(str(self)+" ignoring disallowed interface "+str(ifname), RNS.LOG_EXTREME)
+                elif ifname in AutoInterface.ALL_IGNORE_IFS:
+                    RNS.log(str(self)+" skipping interface "+str(ifname), RNS.LOG_EXTREME)
                 else:
-                    addresses = self.netifaces.ifaddresses(ifname)
-                    if self.netifaces.AF_INET6 in addresses:
-                        link_local_addr = None
-                        for address in addresses[self.netifaces.AF_INET6]:
-                            if "addr" in address:
-                                if address["addr"].startswith("fe80:"):
-                                    link_local_addr = address["addr"]
-                                    self.link_local_addresses.append(link_local_addr.split("%")[0])
-                                    self.adopted_interfaces[ifname] = link_local_addr.split("%")[0]
-                                    self.multicast_echoes[ifname] = time.time()
-                                    RNS.log(str(self)+" Selecting link-local address "+str(link_local_addr)+" for interface "+str(ifname), RNS.LOG_EXTREME)
+                    if len(self.allowed_interfaces) > 0 and not ifname in self.allowed_interfaces:
+                        RNS.log(str(self)+" ignoring interface "+str(ifname)+" since it was not allowed", RNS.LOG_EXTREME)
+                    else:
+                        addresses = self.list_addresses(ifname)
+                        if self.netinfo.AF_INET6 in addresses:
+                            link_local_addr = None
+                            for address in addresses[self.netinfo.AF_INET6]:
+                                if "addr" in address:
+                                    if address["addr"].startswith("fe80:"):
+                                        link_local_addr = self.descope_linklocal(address["addr"])
+                                        self.link_local_addresses.append(link_local_addr)
+                                        self.adopted_interfaces[ifname] = link_local_addr
+                                        self.multicast_echoes[ifname] = time.time()
+                                        nice_name = self.netinfo.interface_name_to_nice_name(ifname)
+                                        if nice_name != None and nice_name != ifname:
+                                            RNS.log(f"{self} Selecting link-local address {link_local_addr} for interface {nice_name} / {ifname}", RNS.LOG_EXTREME)
+                                        else:
+                                            RNS.log(f"{self} Selecting link-local address {link_local_addr} for interface {ifname}", RNS.LOG_EXTREME)
 
-                        if link_local_addr == None:
-                            RNS.log(str(self)+" No link-local IPv6 address configured for "+str(ifname)+", skipping interface", RNS.LOG_EXTREME)
-                        else:
-                            mcast_addr = self.mcast_discovery_address
-                            RNS.log(str(self)+" Creating multicast discovery listener on "+str(ifname)+" with address "+str(mcast_addr), RNS.LOG_EXTREME)
+                            if link_local_addr == None:
+                                RNS.log(str(self)+" No link-local IPv6 address configured for "+str(ifname)+", skipping interface", RNS.LOG_EXTREME)
+                            else:
+                                mcast_addr = self.mcast_discovery_address
+                                RNS.log(str(self)+" Creating multicast discovery listener on "+str(ifname)+" with address "+str(mcast_addr), RNS.LOG_EXTREME)
 
-                            # Struct with interface index
-                            if_struct = struct.pack("I", socket.if_nametoindex(ifname))
+                                # Struct with interface index
+                                if_struct = struct.pack("I", self.interface_name_to_index(ifname))
 
-                            # Set up multicast socket
-                            discovery_socket = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
-                            discovery_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-                            if hasattr(socket, "SO_REUSEPORT"):
-                                discovery_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
-                            discovery_socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_MULTICAST_IF, if_struct)
+                                # Set up multicast socket
+                                discovery_socket = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
+                                discovery_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+                                if hasattr(socket, "SO_REUSEPORT"):
+                                    discovery_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
+                                discovery_socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_MULTICAST_IF, if_struct)
 
-                            # Join multicast group
-                            mcast_group = socket.inet_pton(socket.AF_INET6, mcast_addr) + if_struct
-                            discovery_socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_JOIN_GROUP, mcast_group)
+                                # Join multicast group
+                                mcast_group = socket.inet_pton(socket.AF_INET6, mcast_addr) + if_struct
+                                discovery_socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_JOIN_GROUP, mcast_group)
 
-                            # Bind socket
-                            addr_info = socket.getaddrinfo(mcast_addr+"%"+ifname, self.discovery_port, socket.AF_INET6, socket.SOCK_DGRAM)
-                            discovery_socket.bind(addr_info[0][4])
+                                # Bind socket
+                                if RNS.vendor.platformutils.is_windows():
 
-                            # Set up thread for discovery packets
-                            def discovery_loop():
-                                self.discovery_handler(discovery_socket, ifname)
+                                    # window throws "[WinError 10049] The requested address is not valid in its context"
+                                    # when trying to use the multicast address as host, or when providing interface index
+                                    # passing an empty host appears to work, but probably not exactly how we want it to...
+                                    discovery_socket.bind(('', self.discovery_port))
 
-                            thread = threading.Thread(target=discovery_loop)
-                            thread.daemon = True
-                            thread.start()
+                                else:
 
-                            suitable_interfaces += 1
+                                    if self.discovery_scope == AutoInterface.SCOPE_LINK:
+                                        addr_info = socket.getaddrinfo(mcast_addr+"%"+ifname, self.discovery_port, socket.AF_INET6, socket.SOCK_DGRAM)
+                                    else:
+                                        addr_info = socket.getaddrinfo(mcast_addr, self.discovery_port, socket.AF_INET6, socket.SOCK_DGRAM)
+
+                                    discovery_socket.bind(addr_info[0][4])
+
+                                # Set up thread for discovery packets
+                                def discovery_loop():
+                                    self.discovery_handler(discovery_socket, ifname)
+
+                                thread = threading.Thread(target=discovery_loop)
+                                thread.daemon = True
+                                thread.start()
+
+                                suitable_interfaces += 1
+
+            except Exception as e:
+                nice_name = self.netinfo.interface_name_to_nice_name(ifname)
+                if nice_name != None and nice_name != ifname:
+                    RNS.log(f"Could not configure the system interface {nice_name} / {ifname} for use with {self}, skipping it. The contained exception was: {e}", RNS.LOG_ERROR)
+                else:
+                    RNS.log(f"Could not configure the system interface {ifname} for use with {self}, skipping it. The contained exception was: {e}", RNS.LOG_ERROR)
 
         if suitable_interfaces == 0:
             RNS.log(str(self)+" could not autoconfigure. This interface currently provides no connectivity.", RNS.LOG_WARNING)
         else:
             self.receives = True
 
+            if configured_bitrate != None:
+                self.bitrate = configured_bitrate
+            else:
+                self.bitrate = AutoInterface.BITRATE_GUESS
+
             peering_wait = self.announce_interval*1.2
             RNS.log(str(self)+" discovering peers for "+str(round(peering_wait, 2))+" seconds...", RNS.LOG_VERBOSE)
 
-            def handlerFactory(callback):
-                def createHandler(*args, **keys):
-                    return AutoInterfaceHandler(callback, *args, **keys)
-                return createHandler
-
             self.owner = owner
             socketserver.UDPServer.address_family = socket.AF_INET6
 
             for ifname in self.adopted_interfaces:
-                local_addr = self.adopted_interfaces[ifname]+"%"+ifname
+                local_addr = self.adopted_interfaces[ifname]+"%"+str(self.interface_name_to_index(ifname))
                 addr_info = socket.getaddrinfo(local_addr, self.data_port, socket.AF_INET6, socket.SOCK_DGRAM)
                 address = addr_info[0][4]
 
-                self.server = socketserver.UDPServer(address, handlerFactory(self.processIncoming))
+                udp_server = socketserver.UDPServer(address, self.handler_factory(self.processIncoming))
+                self.interface_servers[ifname] = udp_server
                 
-                thread = threading.Thread(target=self.server.serve_forever)
+                thread = threading.Thread(target=udp_server.serve_forever)
                 thread.daemon = True
                 thread.start()
 
@@ -223,11 +300,6 @@ class AutoInterface(Interface):
 
             time.sleep(peering_wait)
 
-            if configured_bitrate != None:
-                self.bitrate = configured_bitrate
-            else:
-                self.bitrate = AutoInterface.BITRATE_GUESS
-
             self.online = True
 
 
@@ -266,16 +338,62 @@ class AutoInterface(Interface):
                 RNS.log(str(self)+" removed peer "+str(peer_addr)+" on "+str(removed_peer[0]), RNS.LOG_DEBUG)
 
             for ifname in self.adopted_interfaces:
+                # Check that the link-local address has not changed
+                try:
+                    addresses = self.list_addresses(ifname)
+                    if self.netinfo.AF_INET6 in addresses:
+                        link_local_addr = None
+                        for address in addresses[self.netinfo.AF_INET6]:
+                            if "addr" in address:
+                                if address["addr"].startswith("fe80:"):
+                                    link_local_addr = self.descope_linklocal(address["addr"])
+                                    if link_local_addr != self.adopted_interfaces[ifname]:
+                                        old_link_local_address = self.adopted_interfaces[ifname]
+                                        RNS.log("Replacing link-local address "+str(old_link_local_address)+" for "+str(ifname)+" with "+str(link_local_addr), RNS.LOG_DEBUG)
+                                        self.adopted_interfaces[ifname] = link_local_addr
+                                        self.link_local_addresses.append(link_local_addr)
+
+                                        if old_link_local_address in self.link_local_addresses:
+                                            self.link_local_addresses.remove(old_link_local_address)
+
+                                        local_addr = link_local_addr+"%"+ifname
+                                        addr_info = socket.getaddrinfo(local_addr, self.data_port, socket.AF_INET6, socket.SOCK_DGRAM)
+                                        listen_address = addr_info[0][4]
+
+                                        if ifname in self.interface_servers:
+                                            RNS.log("Shutting down previous UDP listener for "+str(self)+" "+str(ifname), RNS.LOG_DEBUG)
+                                            previous_server = self.interface_servers[ifname]
+                                            def shutdown_server():
+                                                previous_server.shutdown()
+                                            threading.Thread(target=shutdown_server, daemon=True).start()
+
+                                        RNS.log("Starting new UDP listener for "+str(self)+" "+str(ifname), RNS.LOG_DEBUG)
+
+                                        udp_server = socketserver.UDPServer(listen_address, self.handler_factory(self.processIncoming))
+                                        self.interface_servers[ifname] = udp_server
+
+                                        thread = threading.Thread(target=udp_server.serve_forever)
+                                        thread.daemon = True
+                                        thread.start()
+
+                                        self.carrier_changed = True
+
+                except Exception as e:
+                    RNS.log("Could not get device information while updating link-local addresses for "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+                # Check multicast echo timeouts
                 last_multicast_echo = 0
                 if ifname in self.multicast_echoes:
                     last_multicast_echo = self.multicast_echoes[ifname]
 
                 if now - last_multicast_echo > self.multicast_echo_timeout:
                     if ifname in self.timed_out_interfaces and self.timed_out_interfaces[ifname] == False:
+                        self.carrier_changed = True
                         RNS.log("Multicast echo timeout for "+str(ifname)+". Carrier lost.", RNS.LOG_WARNING)
                     self.timed_out_interfaces[ifname] = True
                 else:
                     if ifname in self.timed_out_interfaces and self.timed_out_interfaces[ifname] == True:
+                        self.carrier_changed = True
                         RNS.log(str(self)+" Carrier recovered on "+str(ifname), RNS.LOG_WARNING)
                     self.timed_out_interfaces[ifname] = False
                 
@@ -292,7 +410,7 @@ class AutoInterface(Interface):
             announce_socket = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
             addr_info = socket.getaddrinfo(self.mcast_discovery_address, self.discovery_port, socket.AF_INET6, socket.SOCK_DGRAM)
 
-            ifis = struct.pack("I", socket.if_nametoindex(ifname))
+            ifis = struct.pack("I", self.interface_name_to_index(ifname))
             announce_socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_MULTICAST_IF, ifis)
             announce_socket.sendto(discovery_token, addr_info[0][4])
             announce_socket.close()
@@ -326,16 +444,27 @@ class AutoInterface(Interface):
         self.peers[addr][1] = time.time()
 
     def processIncoming(self, data):
-        self.rxb += len(data)
-        self.owner.inbound(data, self)
+        data_hash = RNS.Identity.full_hash(data)
+        deque_hit = False
+        if data_hash in self.mif_deque:
+            for te in self.mif_deque_times:
+                if te[0] == data_hash and time.time() < te[1]+AutoInterface.MULTI_IF_DEQUE_TTL:
+                    deque_hit = True
+                    break
+
+        if not deque_hit:
+            self.mif_deque.append(data_hash)
+            self.mif_deque_times.append([data_hash, time.time()])
+            self.rxb += len(data)
+            self.owner.inbound(data, self)
 
     def processOutgoing(self,data):
             for peer in self.peers:
                 try:
                     if self.outbound_udp_socket == None:
                         self.outbound_udp_socket = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
-                    
-                    peer_addr = str(peer)+"%"+str(self.peers[peer][0])
+
+                    peer_addr = str(peer)+"%"+str(self.interface_name_to_index(self.peers[peer][0]))
                     addr_info = socket.getaddrinfo(peer_addr, self.data_port, socket.AF_INET6, socket.SOCK_DGRAM)
                     self.outbound_udp_socket.sendto(data, addr_info[0][4])
 
@@ -346,6 +475,11 @@ class AutoInterface(Interface):
             self.txb += len(data)
             
 
+    # Until per-device sub-interfacing is implemented,
+    # ingress limiting should be disabled on AutoInterface
+    def should_ingress_limit(self):
+        return False
+
     def __str__(self):
         return "AutoInterface["+self.name+"]"
 

RNS/Interfaces/I2PInterface.py

@@ -161,8 +161,6 @@ class I2PController:
                                 raise tn.status["exception"]
 
                             else:
-                                self.client_tunnels[i2p_destination] = True
-                                owner.awaiting_i2p_tunnel = False
                                 if owner.socket != None:
                                     if hasattr(owner.socket, "close"):
                                         if callable(owner.socket.close):
@@ -175,6 +173,8 @@ class I2PController:
                                                 owner.socket.close()
                                             except Exception as e:
                                                 RNS.log("Error while closing socket for "+str(owner)+": "+str(e))
+                                self.client_tunnels[i2p_destination] = True
+                                owner.awaiting_i2p_tunnel = False
 
                                 RNS.log(str(owner)+" tunnel setup complete", RNS.LOG_VERBOSE)
 
@@ -383,10 +383,14 @@ class I2PInterfacePeer(Interface):
     I2P_PROBE_AFTER = 10
     I2P_PROBE_INTERVAL = 9
     I2P_PROBES = 5
+    I2P_READ_TIMEOUT = (I2P_PROBE_INTERVAL * I2P_PROBES + I2P_PROBE_AFTER)*2
+
+    TUNNEL_STATE_INIT    = 0x00
+    TUNNEL_STATE_ACTIVE  = 0x01
+    TUNNEL_STATE_STALE   = 0x02
 
     def __init__(self, parent_interface, owner, name, target_i2p_dest=None, connected_socket=None, max_reconnect_tries=None):
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
 
         self.HW_MTU = 1064
         
@@ -409,6 +413,30 @@ class I2PInterfacePeer(Interface):
         self.i2p_tunnel_ready = False
         self.mode             = RNS.Interfaces.Interface.Interface.MODE_FULL
         self.bitrate          = I2PInterface.BITRATE_GUESS
+        self.last_read        = 0
+        self.last_write       = 0
+        self.wd_reset         = False
+        self.i2p_tunnel_state = I2PInterfacePeer.TUNNEL_STATE_INIT
+
+        self.ifac_size = self.parent_interface.ifac_size
+        self.ifac_netname = self.parent_interface.ifac_netname
+        self.ifac_netkey = self.parent_interface.ifac_netkey
+        if self.ifac_netname != None or self.ifac_netkey != None:
+            ifac_origin = b""
+            if self.ifac_netname != None:
+                ifac_origin += RNS.Identity.full_hash(self.ifac_netname.encode("utf-8"))
+            if self.ifac_netkey != None:
+                ifac_origin += RNS.Identity.full_hash(self.ifac_netkey.encode("utf-8"))
+
+            ifac_origin_hash = RNS.Identity.full_hash(ifac_origin)
+            self.ifac_key = RNS.Cryptography.hkdf(
+                length=64,
+                derive_from=ifac_origin_hash,
+                salt=RNS.Reticulum.IFAC_SALT,
+                context=None
+            )
+            self.ifac_identity = RNS.Identity.from_bytes(self.ifac_key)
+            self.ifac_signature = self.ifac_identity.sign(RNS.Identity.full_hash(self.ifac_key))
 
         self.announce_rate_target  = None
         self.announce_rate_grace   = None
@@ -454,7 +482,7 @@ class I2PInterfacePeer(Interface):
                         RNS.log("Error while while configuring "+str(self)+": "+str(e), RNS.LOG_ERROR)
                         RNS.log("Check that I2P is installed and running, and that SAM is enabled. Retrying tunnel setup later.", RNS.LOG_ERROR)
 
-                    time.sleep(15)
+                    time.sleep(8)
 
             thread = threading.Thread(target=tunnel_job)
             thread.daemon = True
@@ -463,6 +491,7 @@ class I2PInterfacePeer(Interface):
             def wait_job():
                 while self.awaiting_i2p_tunnel:
                     time.sleep(0.25)
+                time.sleep(2)
                 
                 if not self.kiss_framing:
                     self.wants_tunnel = True
@@ -482,18 +511,11 @@ class I2PInterfacePeer(Interface):
 
 
     def set_timeouts_linux(self):
-        if not self.i2p_tunneled:
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_USER_TIMEOUT, int(I2PInterfacePeer.TCP_USER_TIMEOUT * 1000))
-            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, int(I2PInterfacePeer.TCP_PROBE_AFTER))
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, int(I2PInterfacePeer.TCP_PROBE_INTERVAL))
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, int(I2PInterfacePeer.TCP_PROBES))
-        else:
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_USER_TIMEOUT, int(I2PInterfacePeer.I2P_USER_TIMEOUT * 1000))
-            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, int(I2PInterfacePeer.I2P_PROBE_AFTER))
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, int(I2PInterfacePeer.I2P_PROBE_INTERVAL))
-            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, int(I2PInterfacePeer.I2P_PROBES))
+        self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_USER_TIMEOUT, int(I2PInterfacePeer.I2P_USER_TIMEOUT * 1000))
+        self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
+        self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, int(I2PInterfacePeer.I2P_PROBE_AFTER))
+        self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, int(I2PInterfacePeer.I2P_PROBE_INTERVAL))
+        self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, int(I2PInterfacePeer.I2P_PROBES))
 
     def set_timeouts_osx(self):
         if hasattr(socket, "TCP_KEEPALIVE"):
@@ -502,22 +524,19 @@ class I2PInterfacePeer(Interface):
             TCP_KEEPIDLE = 0x10
 
         self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
-        
-        if not self.i2p_tunneled:
-            self.socket.setsockopt(socket.IPPROTO_TCP, TCP_KEEPIDLE, int(I2PInterfacePeer.TCP_PROBE_AFTER))
-        else:
-            self.socket.setsockopt(socket.IPPROTO_TCP, TCP_KEEPIDLE, int(I2PInterfacePeer.I2P_PROBE_AFTER))
+        self.socket.setsockopt(socket.IPPROTO_TCP, TCP_KEEPIDLE, int(I2PInterfacePeer.I2P_PROBE_AFTER))
     
-    def shutdown_socket(self, socket):
-        if callable(socket.close):
-            
+    def shutdown_socket(self, target_socket):
+        if callable(target_socket.close):
             try:
-                socket.shutdown(socket.SHUT_RDWR)
+                if socket != None:
+                    target_socket.shutdown(socket.SHUT_RDWR)
             except Exception as e:
                 RNS.log("Error while shutting down socket for "+str(self)+": "+str(e))
 
             try:
-                socket.close()
+                if socket != None:
+                    target_socket.close()
             except Exception as e:
                 RNS.log("Error while closing socket for "+str(self)+": "+str(e))    
     
@@ -571,7 +590,6 @@ class I2PInterfacePeer(Interface):
 
         return True
 
-
     def reconnect(self):
         if self.initiator:
             if not self.reconnecting:
@@ -632,6 +650,7 @@ class I2PInterfacePeer(Interface):
                 self.socket.sendall(data)
                 self.writing = False
                 self.txb += len(data)
+                self.last_write = time.time()
                 
                 if hasattr(self, "parent_interface") and self.parent_interface != None and self.parent_count:
                     self.parent_interface.txb += len(data)
@@ -642,8 +661,59 @@ class I2PInterfacePeer(Interface):
                 self.teardown()
 
 
+    def read_watchdog(self):
+        while self.wd_reset:
+            time.sleep(0.25)
+
+        should_run = True
+        try:
+            while should_run and not self.wd_reset:
+                time.sleep(1)
+
+                if (time.time()-self.last_read > I2PInterfacePeer.I2P_PROBE_AFTER*2):
+                    if self.i2p_tunnel_state != I2PInterfacePeer.TUNNEL_STATE_STALE:
+                        RNS.log("I2P tunnel became unresponsive", RNS.LOG_DEBUG)
+
+                    self.i2p_tunnel_state = I2PInterfacePeer.TUNNEL_STATE_STALE
+                else:
+                    self.i2p_tunnel_state = I2PInterfacePeer.TUNNEL_STATE_ACTIVE
+
+                if (time.time()-self.last_write > I2PInterfacePeer.I2P_PROBE_AFTER*1):
+                    try:
+                        if self.socket != None:
+                            self.socket.sendall(bytes([HDLC.FLAG, HDLC.FLAG]))
+                    except Exception as e:
+                        RNS.log("An error ocurred while sending I2P keepalive. The contained exception was: "+str(e), RNS.LOG_ERROR)
+                        self.shutdown_socket(self.socket)
+                        should_run = False
+                
+                if (time.time()-self.last_read > I2PInterfacePeer.I2P_READ_TIMEOUT):
+                    RNS.log("I2P socket is unresponsive, restarting...", RNS.LOG_WARNING)
+                    if self.socket != None:
+                        try:
+                            self.socket.shutdown(socket.SHUT_RDWR)
+                        except Exception as e:
+                            RNS.log("Error while shutting down socket for "+str(self)+": "+str(e))
+
+                        try:
+                            self.socket.close()
+                        except Exception as e:
+                            RNS.log("Error while closing socket for "+str(self)+": "+str(e))
+
+                    should_run = False
+
+                self.wd_reset = False
+
+        finally:
+            self.wd_reset = False
+
     def read_loop(self):
         try:
+            self.last_read  = time.time()
+            self.last_write = time.time()
+
+            wd_thread = threading.Thread(target=self.read_watchdog, daemon=True).start()
+
             in_frame = False
             escape = False
             data_buffer = b""
@@ -653,6 +723,7 @@ class I2PInterfacePeer(Interface):
                 data_in = self.socket.recv(4096)
                 if len(data_in) > 0:
                     pointer = 0
+                    self.last_read = time.time()
                     while pointer < len(data_in):
                         byte = data_in[pointer]
                         pointer += 1
@@ -705,6 +776,11 @@ class I2PInterfacePeer(Interface):
                                     data_buffer = data_buffer+bytes([byte])
                 else:
                     self.online = False
+
+                    self.wd_reset = True
+                    time.sleep(2)
+                    self.wd_reset = False
+
                     if self.initiator and not self.detached:
                         RNS.log("Socket for "+str(self)+" was closed, attempting to reconnect...", RNS.LOG_WARNING)
                         self.reconnect()
@@ -754,9 +830,8 @@ class I2PInterfacePeer(Interface):
 class I2PInterface(Interface):
     BITRATE_GUESS      = 256*1000
 
-    def __init__(self, owner, name, rns_storagepath, peers, connectable = False):
-        self.rxb = 0
-        self.txb = 0
+    def __init__(self, owner, name, rns_storagepath, peers, connectable = False, ifac_size = 16, ifac_netname = None, ifac_netkey = None):
+        super().__init__()
         
         self.HW_MTU = 1064
 
@@ -780,6 +855,9 @@ class I2PInterface(Interface):
         self.bind_port   = self.i2p.get_free_port()
         self.address = (self.bind_ip, self.bind_port)
         self.bitrate = I2PInterface.BITRATE_GUESS
+        self.ifac_size = ifac_size
+        self.ifac_netname = ifac_netname
+        self.ifac_netkey = ifac_netkey
 
         self.online = False
 
@@ -850,9 +928,27 @@ class I2PInterface(Interface):
         spawned_interface.parent_interface = self
         spawned_interface.online = True
         spawned_interface.bitrate = self.bitrate
+
         spawned_interface.ifac_size = self.ifac_size
         spawned_interface.ifac_netname = self.ifac_netname
         spawned_interface.ifac_netkey = self.ifac_netkey
+        if spawned_interface.ifac_netname != None or spawned_interface.ifac_netkey != None:
+            ifac_origin = b""
+            if spawned_interface.ifac_netname != None:
+                ifac_origin += RNS.Identity.full_hash(spawned_interface.ifac_netname.encode("utf-8"))
+            if spawned_interface.ifac_netkey != None:
+                ifac_origin += RNS.Identity.full_hash(spawned_interface.ifac_netkey.encode("utf-8"))
+
+            ifac_origin_hash = RNS.Identity.full_hash(ifac_origin)
+            spawned_interface.ifac_key = RNS.Cryptography.hkdf(
+                length=64,
+                derive_from=ifac_origin_hash,
+                salt=RNS.Reticulum.IFAC_SALT,
+                context=None
+            )
+            spawned_interface.ifac_identity = RNS.Identity.from_bytes(spawned_interface.ifac_key)
+            spawned_interface.ifac_signature = spawned_interface.ifac_identity.sign(RNS.Identity.full_hash(spawned_interface.ifac_key))
+
         spawned_interface.announce_rate_target = self.announce_rate_target
         spawned_interface.announce_rate_grace = self.announce_rate_grace
         spawned_interface.announce_rate_penalty = self.announce_rate_penalty
@@ -866,6 +962,12 @@ class I2PInterface(Interface):
     def processOutgoing(self, data):
         pass
 
+    def received_announce(self, from_spawned=False):
+        if from_spawned: self.ia_freq_deque.append(time.time())
+
+    def sent_announce(self, from_spawned=False):
+        if from_spawned: self.oa_freq_deque.append(time.time())
+
     def detach(self):
         RNS.log("Detaching "+str(self), RNS.LOG_DEBUG)
         self.i2p.stop()

RNS/Interfaces/Interface.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -23,6 +23,7 @@
 import RNS
 import time
 import threading
+from collections import deque
 
 class Interface:
     IN  = False
@@ -39,18 +40,161 @@ class Interface:
     MODE_BOUNDARY       = 0x05
     MODE_GATEWAY        = 0x06
 
-    # Which interface modes a Transport Node
-    # should actively discover paths for.
-    DISCOVER_PATHS_FOR  = [MODE_ACCESS_POINT, MODE_GATEWAY]
+    # Which interface modes a Transport Node should
+    # actively discover paths for.
+    DISCOVER_PATHS_FOR  = [MODE_ACCESS_POINT, MODE_GATEWAY, MODE_ROAMING]
+
+    # How many samples to use for announce
+    # frequency calculations
+    IA_FREQ_SAMPLES     = 6
+    OA_FREQ_SAMPLES     = 6
+
+    # Maximum amount of ingress limited announces
+    # to hold at any given time.
+    MAX_HELD_ANNOUNCES  = 256
+
+    # How long a spawned interface will be
+    # considered to be newly created. Two
+    # hours by default.
+    IC_NEW_TIME              = 2*60*60
+    IC_BURST_FREQ_NEW        = 3.5
+    IC_BURST_FREQ            = 12
+    IC_BURST_HOLD            = 1*60
+    IC_BURST_PENALTY         = 5*60
+    IC_HELD_RELEASE_INTERVAL = 30
 
     def __init__(self):
         self.rxb = 0
         self.txb = 0
+        self.created = time.time()
         self.online = False
+        self.bitrate = 1e6
+
+        self.ingress_control = True
+        self.ic_max_held_announces = Interface.MAX_HELD_ANNOUNCES
+        self.ic_burst_hold = Interface.IC_BURST_HOLD
+        self.ic_burst_active = False
+        self.ic_burst_activated = 0
+        self.ic_held_release = 0
+        self.ic_burst_freq_new = Interface.IC_BURST_FREQ_NEW
+        self.ic_burst_freq = Interface.IC_BURST_FREQ
+        self.ic_new_time = Interface.IC_NEW_TIME
+        self.ic_burst_penalty = Interface.IC_BURST_PENALTY
+        self.ic_held_release_interval = Interface.IC_HELD_RELEASE_INTERVAL
+        self.held_announces = {}
+
+        self.ia_freq_deque = deque(maxlen=Interface.IA_FREQ_SAMPLES)
+        self.oa_freq_deque = deque(maxlen=Interface.OA_FREQ_SAMPLES)
 
     def get_hash(self):
         return RNS.Identity.full_hash(str(self).encode("utf-8"))
 
+    # This is a generic function for determining when an interface
+    # should activate ingress limiting. Since this can vary for
+    # different interface types, this function should be overwritten
+    # in case a particular interface requires a different approach.
+    def should_ingress_limit(self):
+        if self.ingress_control:
+            freq_threshold = self.ic_burst_freq_new if self.age() < self.ic_new_time else self.ic_burst_freq
+            ia_freq = self.incoming_announce_frequency()
+
+            if self.ic_burst_active:
+                if ia_freq < freq_threshold and time.time() > self.ic_burst_activated+self.ic_burst_hold:
+                    self.ic_burst_active = False
+                    self.ic_held_release = time.time() + self.ic_burst_penalty
+                return True
+
+            else:
+                if ia_freq > freq_threshold:
+                    self.ic_burst_active = True
+                    self.ic_burst_activated = time.time()
+                    return True
+
+                else:
+                    return False
+
+        else:
+            return False
+
+    def age(self):
+        return time.time()-self.created
+
+    def hold_announce(self, announce_packet):
+        if announce_packet.destination_hash in self.held_announces:
+            self.held_announces[announce_packet.destination_hash] = announce_packet
+        elif not len(self.held_announces) >= self.ic_max_held_announces:
+            self.held_announces[announce_packet.destination_hash] = announce_packet
+
+    def process_held_announces(self):
+        try:
+            if not self.should_ingress_limit() and len(self.held_announces) > 0 and time.time() > self.ic_held_release:
+                freq_threshold = self.ic_burst_freq_new if self.age() < self.ic_new_time else self.ic_burst_freq
+                ia_freq = self.incoming_announce_frequency()
+                if ia_freq < freq_threshold:
+                    selected_announce_packet = None
+                    min_hops = RNS.Transport.PATHFINDER_M
+                    for destination_hash in self.held_announces:
+                        announce_packet = self.held_announces[destination_hash]
+                        if announce_packet.hops < min_hops:
+                            min_hops = announce_packet.hops
+                            selected_announce_packet = announce_packet
+
+                    if selected_announce_packet != None:
+                        RNS.log("Releasing held announce packet "+str(selected_announce_packet)+" from "+str(self), RNS.LOG_EXTREME)
+                        self.ic_held_release = time.time() + self.ic_held_release_interval
+                        self.held_announces.pop(selected_announce_packet.destination_hash)
+                        def release():
+                            RNS.Transport.inbound(selected_announce_packet.raw, selected_announce_packet.receiving_interface)
+                        threading.Thread(target=release, daemon=True).start()
+        
+        except Exception as e:
+            RNS.log("An error occurred while processing held announces for "+str(self), RNS.LOG_ERROR)
+            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+    def received_announce(self):
+        self.ia_freq_deque.append(time.time())
+        if hasattr(self, "parent_interface") and self.parent_interface != None:
+            self.parent_interface.received_announce(from_spawned=True)
+
+    def sent_announce(self):
+        self.oa_freq_deque.append(time.time())
+        if hasattr(self, "parent_interface") and self.parent_interface != None:
+            self.parent_interface.sent_announce(from_spawned=True)
+
+    def incoming_announce_frequency(self):
+        if not len(self.ia_freq_deque) > 1:
+            return 0
+        else:
+            dq_len = len(self.ia_freq_deque)
+            delta_sum = 0
+            for i in range(1,dq_len):
+                delta_sum += self.ia_freq_deque[i]-self.ia_freq_deque[i-1]
+            delta_sum += time.time() - self.ia_freq_deque[dq_len-1]
+            
+            if delta_sum == 0:
+                avg = 0
+            else:
+                avg = 1/(delta_sum/(dq_len))
+
+            return avg
+
+    def outgoing_announce_frequency(self):
+        if not len(self.oa_freq_deque) > 1:
+            return 0
+        else:
+            dq_len = len(self.oa_freq_deque)
+            delta_sum = 0
+            for i in range(1,dq_len):
+                delta_sum += self.oa_freq_deque[i]-self.oa_freq_deque[i-1]
+            delta_sum += time.time() - self.oa_freq_deque[dq_len-1]
+            
+            if delta_sum == 0:
+                avg = 0
+            else:
+                avg = 1/(delta_sum/(dq_len))
+
+            return avg
+
     def process_announce_queue(self):
         if not hasattr(self, "announce_cap"):
             self.announce_cap = RNS.Reticulum.ANNOUNCE_CAP
@@ -79,6 +223,7 @@ class Interface:
                     self.announce_allowed_at = now + wait_time
 
                     self.processOutgoing(selected["raw"])
+                    self.sent_announce()
 
                     if selected in self.announce_queue:
                         self.announce_queue.remove(selected)

RNS/Interfaces/KISSInterface.py

@@ -70,8 +70,7 @@ class KISSInterface(Interface):
             RNS.log("You can install one with the command: python3 -m pip install pyserial", RNS.LOG_CRITICAL)
             RNS.panic()
 
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
         
         self.HW_MTU = 564
         
@@ -349,5 +348,8 @@ class KISSInterface(Interface):
 
         RNS.log("Reconnected serial port for "+str(self))
 
+    def should_ingress_limit(self):
+        return False
+
     def __str__(self):
         return "KISSInterface["+self.name+"]"

RNS/Interfaces/LocalInterface.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -28,6 +28,7 @@ import time
 import sys
 import os
 import RNS
+from threading import Lock
 
 class HDLC():
     FLAG              = 0x7E
@@ -41,14 +42,19 @@ class HDLC():
         return data
 
 class ThreadingTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
-    pass
+    def server_bind(self):
+        if RNS.vendor.platformutils.is_windows():
+            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_EXCLUSIVEADDRUSE, 1)
+        else:
+            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+        self.socket.bind(self.server_address)
+        self.server_address = self.socket.getsockname()
 
 class LocalClientInterface(Interface):
-    RECONNECT_WAIT = 3
+    RECONNECT_WAIT = 8
 
     def __init__(self, owner, name, target_port = None, connected_socket=None):
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
 
         # TODO: Remove at some point
         # self.rxptime = 0
@@ -72,6 +78,7 @@ class LocalClientInterface(Interface):
             self.target_ip   = None
             self.target_port = None
             self.socket      = connected_socket
+            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
 
             self.is_connected_to_shared_instance = False
 
@@ -86,6 +93,8 @@ class LocalClientInterface(Interface):
         self.online  = True
         self.writing = False
 
+        self._force_bitrate = False
+
         self.announce_rate_target  = None
         self.announce_rate_grace   = None
         self.announce_rate_penalty = None
@@ -95,8 +104,12 @@ class LocalClientInterface(Interface):
             thread.daemon = True
             thread.start()
 
+    def should_ingress_limit(self):
+        return False
+
     def connect(self):
         self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
         self.socket.connect((self.target_ip, self.target_port))
 
         self.online = True
@@ -129,7 +142,10 @@ class LocalClientInterface(Interface):
                 thread = threading.Thread(target=self.read_loop)
                 thread.daemon = True
                 thread.start()
-                RNS.Transport.shared_connection_reappeared()
+                def job():
+                    time.sleep(LocalClientInterface.RECONNECT_WAIT+2)
+                    RNS.Transport.shared_connection_reappeared()
+                threading.Thread(target=job, daemon=True).start()
         
         else:
             RNS.log("Attempt to reconnect on a non-initiator shared local interface. This should not happen.", RNS.LOG_ERROR)
@@ -154,6 +170,16 @@ class LocalClientInterface(Interface):
         if self.online:
             try:
                 self.writing = True
+
+                if self._force_bitrate:
+                    if not hasattr(self, "send_lock"):
+                        self.send_lock = Lock()
+
+                    with self.send_lock:
+                        s = len(data) / self.bitrate * 8
+                        RNS.log(f"Simulating latency of {RNS.prettytime(s)} for {len(data)} bytes")
+                        time.sleep(s)
+
                 data = bytes([HDLC.FLAG])+HDLC.escape(data)+bytes([HDLC.FLAG])
                 self.socket.sendall(data)
                 self.writing = False
@@ -246,7 +272,8 @@ class LocalClientInterface(Interface):
             RNS.Transport.local_client_interfaces.remove(self)
             if hasattr(self, "parent_interface") and self.parent_interface != None:
                 self.parent_interface.clients -= 1
-                RNS.Transport.owner._should_persist_data()
+                if hasattr(RNS.Transport, "owner") and RNS.Transport.owner != None:
+                    RNS.Transport.owner._should_persist_data()
 
         if nowarning == False:
             RNS.log("The interface "+str(self)+" experienced an unrecoverable error and is being torn down. Restart Reticulum to attempt to open this interface again.", RNS.LOG_ERROR)
@@ -267,8 +294,7 @@ class LocalClientInterface(Interface):
 class LocalServerInterface(Interface):
 
     def __init__(self, owner, bindport=None):
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
         self.online = False
         self.clients = 0
         
@@ -292,7 +318,6 @@ class LocalServerInterface(Interface):
 
             address = (self.bind_ip, self.bind_port)
 
-            ThreadingTCPServer.allow_reuse_address = True
             self.server = ThreadingTCPServer(address, handlerFactory(self.incoming_connection))
 
             thread = threading.Thread(target=self.server.serve_forever)
@@ -317,7 +342,9 @@ class LocalServerInterface(Interface):
         spawned_interface.target_port = str(handler.client_address[1])
         spawned_interface.parent_interface = self
         spawned_interface.bitrate = self.bitrate
-        RNS.log("Accepting new connection to shared instance: "+str(spawned_interface), RNS.LOG_EXTREME)
+        if hasattr(self, "_force_bitrate"):
+            spawned_interface._force_bitrate = self._force_bitrate
+        # RNS.log("Accepting new connection to shared instance: "+str(spawned_interface), RNS.LOG_EXTREME)
         RNS.Transport.interfaces.append(spawned_interface)
         RNS.Transport.local_client_interfaces.append(spawned_interface)
         self.clients += 1
@@ -326,6 +353,12 @@ class LocalServerInterface(Interface):
     def processOutgoing(self, data):
         pass
 
+    def received_announce(self, from_spawned=False):
+        if from_spawned: self.ia_freq_deque.append(time.time())
+
+    def sent_announce(self, from_spawned=False):
+        if from_spawned: self.oa_freq_deque.append(time.time())
+
     def __str__(self):
         return "Shared Instance["+str(self.bind_port)+"]"
 

RNS/Interfaces/PipeInterface.py

@@ -54,8 +54,7 @@ class PipeInterface(Interface):
         if respawn_delay == None:
             respawn_delay = 5
 
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
 
         self.HW_MTU = 1064
         

RNS/Interfaces/RNodeInterface.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -43,14 +43,24 @@ class KISS():
     CMD_CR          = 0x05
     CMD_RADIO_STATE = 0x06
     CMD_RADIO_LOCK  = 0x07
+    CMD_ST_ALOCK    = 0x0B
+    CMD_LT_ALOCK    = 0x0C
     CMD_DETECT      = 0x08
+    CMD_LEAVE       = 0x0A
     CMD_READY       = 0x0F
     CMD_STAT_RX     = 0x21
     CMD_STAT_TX     = 0x22
     CMD_STAT_RSSI   = 0x23
     CMD_STAT_SNR    = 0x24
+    CMD_STAT_CHTM   = 0x25
+    CMD_STAT_PHYPRM = 0x26
+    CMD_STAT_BAT    = 0x27
     CMD_BLINK       = 0x30
     CMD_RANDOM      = 0x40
+    CMD_FB_EXT      = 0x41
+    CMD_FB_READ     = 0x42
+    CMD_FB_WRITE    = 0x43
+    CMD_BT_CTRL     = 0x46
     CMD_PLATFORM    = 0x48
     CMD_MCU         = 0x49
     CMD_FW_VERSION  = 0x50
@@ -68,9 +78,12 @@ class KISS():
     ERROR_INITRADIO     = 0x01
     ERROR_TXFAILED      = 0x02
     ERROR_EEPROM_LOCKED = 0x03
+    ERROR_QUEUE_FULL    = 0x04
+    ERROR_MEMORY_LOW    = 0x05
 
     PLATFORM_AVR   = 0x90
     PLATFORM_ESP32 = 0x80
+    PLATFORM_NRF52 = 0x70
 
     @staticmethod
     def escape(data):
@@ -82,25 +95,31 @@ class KISS():
 class RNodeInterface(Interface):
     MAX_CHUNK = 32768
 
-    owner    = None
-    port     = None
-    speed    = None
-    databits = None
-    parity   = None
-    stopbits = None
-    serial   = None
-
     FREQ_MIN = 137000000
-    FREQ_MAX = 1020000000
+    FREQ_MAX = 3000000000
 
     RSSI_OFFSET = 157
 
     CALLSIGN_MAX_LEN    = 32
 
     REQUIRED_FW_VER_MAJ = 1
-    REQUIRED_FW_VER_MIN = 26
+    REQUIRED_FW_VER_MIN = 52
+
+    RECONNECT_WAIT = 5
+
+    Q_SNR_MIN_BASE = -9
+    Q_SNR_MAX      = 6
+    Q_SNR_STEP     = 2
+
+    BATTERY_STATE_UNKNOWN     = 0x00
+    BATTERY_STATE_DISCHARGING = 0x01
+    BATTERY_STATE_CHARGING    = 0x02
+    BATTERY_STATE_CHARGED     = 0x03
+
+    def __init__(self, owner, name, port, frequency = None, bandwidth = None, txpower = None, sf = None, cr = None, flow_control = False, id_interval = None, id_callsign = None, st_alock = None, lt_alock = None, ble_addr = None, ble_name = None, force_ble=False):
+        if RNS.vendor.platformutils.is_android():
+            raise SystemError("Invalid interface type. The Android-specific RNode interface must be used on Android")
 
-    def __init__(self, owner, name, port, frequency = None, bandwidth = None, txpower = None, sf = None, cr = None, flow_control = False, id_interval = None, id_callsign = None):
         import importlib
         if importlib.util.find_spec('serial') != None:
             import serial
@@ -109,8 +128,7 @@ class RNodeInterface(Interface):
             RNS.log("You can install one with the command: python3 -m pip install pyserial", RNS.LOG_CRITICAL)
             RNS.panic()
 
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
 
         self.HW_MTU = 508
         
@@ -121,10 +139,20 @@ class RNodeInterface(Interface):
         self.port        = port
         self.speed       = 115200
         self.databits    = 8
-        self.parity      = serial.PARITY_NONE
         self.stopbits    = 1
         self.timeout     = 100
         self.online      = False
+        self.detached    = False
+        self.reconnecting= False
+
+        self.use_ble     = False
+        self.ble_name    = ble_name
+        self.ble_addr    = ble_addr
+        self.ble         = None
+        self.ble_rx_lock = threading.Lock()
+        self.ble_tx_lock = threading.Lock()
+        self.ble_rx_queue= b""
+        self.ble_tx_queue= b""
 
         self.frequency   = frequency
         self.bandwidth   = bandwidth
@@ -133,7 +161,10 @@ class RNodeInterface(Interface):
         self.cr          = cr
         self.state       = KISS.RADIO_STATE_OFF
         self.bitrate     = 0
+        self.st_alock    = st_alock
+        self.lt_alock    = lt_alock
         self.platform    = None
+        self.display     = None
         self.mcu         = None
         self.detected    = False
         self.firmware_ok = False
@@ -142,6 +173,7 @@ class RNodeInterface(Interface):
 
         self.last_id     = 0
         self.first_tx    = None
+        self.reconnect_w = RNodeInterface.RECONNECT_WAIT
 
         self.r_frequency = None
         self.r_bandwidth = None
@@ -153,26 +185,43 @@ class RNodeInterface(Interface):
         self.r_stat_rx   = None
         self.r_stat_tx   = None
         self.r_stat_rssi = None
+        self.r_stat_snr  = None
+        self.r_st_alock  = None
+        self.r_lt_alock  = None
         self.r_random    = None
+        self.r_airtime_short      = 0.0
+        self.r_airtime_long       = 0.0
+        self.r_channel_load_short = 0.0
+        self.r_channel_load_long  = 0.0
+        self.r_symbol_time_ms = None
+        self.r_symbol_rate = None
+        self.r_preamble_symbols = None
+        self.r_premable_time_ms = None
+        self.r_battery_state = RNodeInterface.BATTERY_STATE_UNKNOWN
+        self.r_battery_percent = 0
 
         self.packet_queue    = []
         self.flow_control    = flow_control
         self.interface_ready = False
+        self.announce_rate_target = None
+
+        if force_ble or self.ble_addr != None or self.ble_name != None:
+            self.use_ble = True
 
         self.validcfg  = True
         if (self.frequency < RNodeInterface.FREQ_MIN or self.frequency > RNodeInterface.FREQ_MAX):
             RNS.log("Invalid frequency configured for "+str(self), RNS.LOG_ERROR)
             self.validcfg = False
 
-        if (self.txpower < 0 or self.txpower > 17):
+        if (self.txpower < 0 or self.txpower > 22):
             RNS.log("Invalid TX power configured for "+str(self), RNS.LOG_ERROR)
             self.validcfg = False
 
-        if (self.bandwidth < 7800 or self.bandwidth > 500000):
+        if (self.bandwidth < 7800 or self.bandwidth > 1625000):
             RNS.log("Invalid bandwidth configured for "+str(self), RNS.LOG_ERROR)
             self.validcfg = False
 
-        if (self.sf < 7 or self.sf > 12):
+        if (self.sf < 5 or self.sf > 12):
             RNS.log("Invalid spreading factor configured for "+str(self), RNS.LOG_ERROR)
             self.validcfg = False
 
@@ -180,6 +229,14 @@ class RNodeInterface(Interface):
             RNS.log("Invalid coding rate configured for "+str(self), RNS.LOG_ERROR)
             self.validcfg = False
 
+        if (self.st_alock and (self.st_alock < 0.0 or self.st_alock > 100.0)):
+            RNS.log("Invalid short-term airtime limit configured for "+str(self), RNS.LOG_ERROR)
+            self.validcfg = False
+
+        if (self.lt_alock and (self.lt_alock < 0.0 or self.lt_alock > 100.0)):
+            RNS.log("Invalid long-term airtime limit configured for "+str(self), RNS.LOG_ERROR)
+            self.validcfg = False
+
         if id_interval != None and id_callsign != None:
             if (len(id_callsign.encode("utf-8")) <= RNodeInterface.CALLSIGN_MAX_LEN):
                 self.should_id = True
@@ -207,58 +264,87 @@ class RNodeInterface(Interface):
             RNS.log("Could not open serial port for interface "+str(self), RNS.LOG_ERROR)
             RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
             RNS.log("Reticulum will attempt to bring up this interface periodically", RNS.LOG_ERROR)
-            thread = threading.Thread(target=self.reconnect_port)
-            thread.daemon = True
-            thread.start()
+            if not self.detached and not self.reconnecting:
+                thread = threading.Thread(target=self.reconnect_port)
+                thread.daemon = True
+                thread.start()
 
 
     def open_port(self):
-        RNS.log("Opening serial port "+self.port+"...")
-        self.serial = self.pyserial.Serial(
-            port = self.port,
-            baudrate = self.speed,
-            bytesize = self.databits,
-            parity = self.parity,
-            stopbits = self.stopbits,
-            xonxoff = False,
-            rtscts = False,
-            timeout = 0,
-            inter_byte_timeout = None,
-            write_timeout = None,
-            dsrdtr = False,
-        )
+        if not self.use_ble:
+            RNS.log("Opening serial port "+self.port+"...")
+            self.serial = self.pyserial.Serial(
+                port = self.port,
+                baudrate = self.speed,
+                bytesize = self.databits,
+                parity = self.pyserial.PARITY_NONE,
+                stopbits = self.stopbits,
+                xonxoff = False,
+                rtscts = False,
+                timeout = 0,
+                inter_byte_timeout = None,
+                write_timeout = None,
+                dsrdtr = False,
+            )
+        
+        else:
+            RNS.log(f"Opening BLE connection for {self}...")
+            if self.ble == None:
+                self.ble = BLEConnection(owner=self, target_name=self.ble_name, target_bt_addr=self.ble_addr)
+                self.serial = self.ble
+
+            open_time = time.time()
+            while not self.ble.connected and time.time() < open_time + self.ble.CONNECT_TIMEOUT:
+                time.sleep(1)
 
 
     def configure_device(self):
+        self.r_frequency = None
+        self.r_bandwidth = None
+        self.r_txpower   = None
+        self.r_sf        = None
+        self.r_cr        = None
+        self.r_state     = None
+        self.r_lock      = None
         sleep(2.0)
+
         thread = threading.Thread(target=self.readLoop)
         thread.daemon = True
         thread.start()
 
         self.detect()
-        sleep(0.1)
+        if not self.use_ble:
+            sleep(0.2)
+        else:
+            ble_detect_timeout = 5
+            detect_time = time.time()
+            while not self.detected and time.time() < detect_time + ble_detect_timeout:
+                time.sleep(0.1)
+            if self.detected:
+                detect_time = RNS.prettytime(time.time()-detect_time)
+            else:
+                RNS.log(f"RNode detect timed out over {self.port}", RNS.LOG_ERROR)
         
         if not self.detected:
-            raise IOError("Could not detect device")
+            RNS.log("Could not detect device for "+str(self), RNS.LOG_ERROR)
+            self.serial.close()
         else:
-            if self.platform == KISS.PLATFORM_ESP32:
-                RNS.log("Resetting ESP32-based device before configuration...", RNS.LOG_VERBOSE)
-                self.hard_reset()
+            if self.platform == KISS.PLATFORM_ESP32 or self.platform == KISS.PLATFORM_NRF52:
+                self.display = True
 
-        self.online = True
         RNS.log("Serial port "+self.port+" is now open")
         RNS.log("Configuring RNode interface...", RNS.LOG_VERBOSE)
         self.initRadio()
         if (self.validateRadioState()):
             self.interface_ready = True
             RNS.log(str(self)+" is configured and powered up")
-            sleep(1.0)
+            sleep(0.3)
+            self.online = True
         else:
             RNS.log("After configuring "+str(self)+", the reported radio parameters did not match your configuration.", RNS.LOG_ERROR)
             RNS.log("Make sure that your hardware actually supports the parameters specified in the configuration", RNS.LOG_ERROR)
             RNS.log("Aborting RNode startup", RNS.LOG_ERROR)
             self.serial.close()
-            raise IOError("RNode interface did not pass configuration validation")
             
 
     def initRadio(self):
@@ -267,13 +353,62 @@ class RNodeInterface(Interface):
         self.setTXPower()
         self.setSpreadingFactor()
         self.setCodingRate()
+        self.setSTALock()
+        self.setLTALock()
         self.setRadioState(KISS.RADIO_STATE_ON)
 
+        if self.use_ble:
+            time.sleep(2)
+
     def detect(self):
         kiss_command = bytes([KISS.FEND, KISS.CMD_DETECT, KISS.DETECT_REQ, KISS.FEND, KISS.CMD_FW_VERSION, 0x00, KISS.FEND, KISS.CMD_PLATFORM, 0x00, KISS.FEND, KISS.CMD_MCU, 0x00, KISS.FEND])
         written = self.serial.write(kiss_command)
         if written != len(kiss_command):
-            raise IOError("An IO error occurred while detecting hardware for "+self(str))
+            raise IOError("An IO error occurred while detecting hardware for "+str(self))
+    
+    def leave(self):
+        kiss_command = bytes([KISS.FEND, KISS.CMD_LEAVE, 0xFF, KISS.FEND])
+        written = self.serial.write(kiss_command)
+        if written != len(kiss_command):
+            raise IOError("An IO error occurred while sending host left command to device")
+    
+    def enable_external_framebuffer(self):
+        if self.display != None:
+            kiss_command = bytes([KISS.FEND, KISS.CMD_FB_EXT, 0x01, KISS.FEND])
+            written = self.serial.write(kiss_command)
+            if written != len(kiss_command):
+                raise IOError("An IO error occurred while enabling external framebuffer on device")
+
+    def disable_external_framebuffer(self):
+        if self.display != None:
+            kiss_command = bytes([KISS.FEND, KISS.CMD_FB_EXT, 0x00, KISS.FEND])
+            written = self.serial.write(kiss_command)
+            if written != len(kiss_command):
+                raise IOError("An IO error occurred while disabling external framebuffer on device")
+
+    FB_PIXEL_WIDTH     = 64
+    FB_BITS_PER_PIXEL  = 1
+    FB_PIXELS_PER_BYTE = 8//FB_BITS_PER_PIXEL
+    FB_BYTES_PER_LINE  = FB_PIXEL_WIDTH//FB_PIXELS_PER_BYTE
+    def display_image(self, imagedata):
+        if self.display != None:
+            lines = len(imagedata)//8
+            for line in range(lines):
+                line_start = line*RNodeInterface.FB_BYTES_PER_LINE
+                line_end   = line_start+RNodeInterface.FB_BYTES_PER_LINE
+                line_data = bytes(imagedata[line_start:line_end])
+                self.write_framebuffer(line, line_data)
+
+    def write_framebuffer(self, line, line_data):
+        if self.display != None:
+            line_byte = line.to_bytes(1, byteorder="big", signed=False)
+            data = line_byte+line_data
+            escaped_data = KISS.escape(data)
+            kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_FB_WRITE])+escaped_data+bytes([KISS.FEND])
+            
+            written = self.serial.write(kiss_command)
+            if written != len(kiss_command):
+                raise IOError("An IO error occurred while writing framebuffer data device")
 
     def hard_reset(self):
         kiss_command = bytes([KISS.FEND, KISS.CMD_RESET, 0xf8, KISS.FEND])
@@ -292,7 +427,7 @@ class RNodeInterface(Interface):
         kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_FREQUENCY])+data+bytes([KISS.FEND])
         written = self.serial.write(kiss_command)
         if written != len(kiss_command):
-            raise IOError("An IO error occurred while configuring frequency for "+self(str))
+            raise IOError("An IO error occurred while configuring frequency for "+str(self))
 
     def setBandwidth(self):
         c1 = self.bandwidth >> 24
@@ -304,54 +439,83 @@ class RNodeInterface(Interface):
         kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_BANDWIDTH])+data+bytes([KISS.FEND])
         written = self.serial.write(kiss_command)
         if written != len(kiss_command):
-            raise IOError("An IO error occurred while configuring bandwidth for "+self(str))
+            raise IOError("An IO error occurred while configuring bandwidth for "+str(self))
 
     def setTXPower(self):
         txp = bytes([self.txpower])
         kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_TXPOWER])+txp+bytes([KISS.FEND])
         written = self.serial.write(kiss_command)
         if written != len(kiss_command):
-            raise IOError("An IO error occurred while configuring TX power for "+self(str))
+            raise IOError("An IO error occurred while configuring TX power for "+str(self))
 
     def setSpreadingFactor(self):
         sf = bytes([self.sf])
         kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_SF])+sf+bytes([KISS.FEND])
         written = self.serial.write(kiss_command)
         if written != len(kiss_command):
-            raise IOError("An IO error occurred while configuring spreading factor for "+self(str))
+            raise IOError("An IO error occurred while configuring spreading factor for "+str(self))
 
     def setCodingRate(self):
         cr = bytes([self.cr])
         kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_CR])+cr+bytes([KISS.FEND])
         written = self.serial.write(kiss_command)
         if written != len(kiss_command):
-            raise IOError("An IO error occurred while configuring coding rate for "+self(str))
+            raise IOError("An IO error occurred while configuring coding rate for "+str(self))
+
+    def setSTALock(self):
+        if self.st_alock != None:
+            at = int(self.st_alock*100)
+            c1 = at >> 8 & 0xFF
+            c2 = at & 0xFF
+            data = KISS.escape(bytes([c1])+bytes([c2]))
+
+            kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_ST_ALOCK])+data+bytes([KISS.FEND])
+            written = self.serial.write(kiss_command)
+            if written != len(kiss_command):
+                raise IOError("An IO error occurred while configuring short-term airtime limit for "+str(self))
+
+    def setLTALock(self):
+        if self.lt_alock != None:
+            at = int(self.lt_alock*100)
+            c1 = at >> 8 & 0xFF
+            c2 = at & 0xFF
+            data = KISS.escape(bytes([c1])+bytes([c2]))
+
+            kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_LT_ALOCK])+data+bytes([KISS.FEND])
+            written = self.serial.write(kiss_command)
+            if written != len(kiss_command):
+                raise IOError("An IO error occurred while configuring long-term airtime limit for "+str(self))
 
     def setRadioState(self, state):
         self.state = state
         kiss_command = bytes([KISS.FEND])+bytes([KISS.CMD_RADIO_STATE])+bytes([state])+bytes([KISS.FEND])
         written = self.serial.write(kiss_command)
         if written != len(kiss_command):
-            raise IOError("An IO error occurred while configuring radio state for "+self(str))
+            raise IOError("An IO error occurred while configuring radio state for "+str(self))
 
     def validate_firmware(self):
-        if (self.maj_version >= RNodeInterface.REQUIRED_FW_VER_MAJ):
-            if (self.min_version >= RNodeInterface.REQUIRED_FW_VER_MIN):
-                self.firmware_ok = True
+        if (self.maj_version > RNodeInterface.REQUIRED_FW_VER_MAJ):
+            self.firmware_ok = True
+        else:
+            if (self.maj_version >= RNodeInterface.REQUIRED_FW_VER_MAJ):
+                if (self.min_version >= RNodeInterface.REQUIRED_FW_VER_MIN):
+                    self.firmware_ok = True
         
         if self.firmware_ok:
             return
 
         RNS.log("The firmware version of the connected RNode is "+str(self.maj_version)+"."+str(self.min_version), RNS.LOG_ERROR)
         RNS.log("This version of Reticulum requires at least version "+str(RNodeInterface.REQUIRED_FW_VER_MAJ)+"."+str(RNodeInterface.REQUIRED_FW_VER_MIN), RNS.LOG_ERROR)
-        RNS.log("Please update your RNode firmware with rnodeconf (https://github.com/markqvist/rnodeconfigutil/)")
+        RNS.log("Please update your RNode firmware with rnodeconf from https://github.com/markqvist/rnodeconfigutil/")
         RNS.panic()
 
 
     def validateRadioState(self):
-        RNS.log("Wating for radio configuration validation for "+str(self)+"...", RNS.LOG_VERBOSE)
+        RNS.log("Waiting for radio configuration validation for "+str(self)+"...", RNS.LOG_VERBOSE)
         sleep(0.25);
-        if (self.frequency != self.r_frequency):
+
+        self.validcfg = True
+        if (self.r_frequency != None and abs(self.frequency - int(self.r_frequency)) > 100):
             RNS.log("Frequency mismatch", RNS.LOG_ERROR)
             self.validcfg = False
         if (self.bandwidth != self.r_bandwidth):
@@ -508,7 +672,6 @@ class RNodeInterface(Interface):
                             self.r_state = byte
                             if self.r_state:
                                 pass
-                                #RNS.log(str(self)+" Radio reporting state is online", RNS.LOG_DEBUG)
                             else:
                                 RNS.log(str(self)+" Radio reporting state is offline", RNS.LOG_DEBUG)
 
@@ -562,6 +725,115 @@ class RNodeInterface(Interface):
                             self.r_stat_rssi = byte-RNodeInterface.RSSI_OFFSET
                         elif (command == KISS.CMD_STAT_SNR):
                             self.r_stat_snr = int.from_bytes(bytes([byte]), byteorder="big", signed=True) * 0.25
+                            try:
+                                sfs = self.r_sf-7
+                                snr = self.r_stat_snr
+                                q_snr_min = RNodeInterface.Q_SNR_MIN_BASE-sfs*RNodeInterface.Q_SNR_STEP
+                                q_snr_max = RNodeInterface.Q_SNR_MAX
+                                q_snr_span = q_snr_max-q_snr_min
+                                quality = round(((snr-q_snr_min)/(q_snr_span))*100,1)
+                                if quality > 100.0: quality = 100.0
+                                if quality < 0.0: quality = 0.0
+                                self.r_stat_q = quality
+                            except:
+                                pass
+                        elif (command == KISS.CMD_ST_ALOCK):
+                            if (byte == KISS.FESC):
+                                escape = True
+                            else:
+                                if (escape):
+                                    if (byte == KISS.TFEND):
+                                        byte = KISS.FEND
+                                    if (byte == KISS.TFESC):
+                                        byte = KISS.FESC
+                                    escape = False
+                                command_buffer = command_buffer+bytes([byte])
+                                if (len(command_buffer) == 2):
+                                    at = command_buffer[0] << 8 | command_buffer[1]
+                                    self.r_st_alock = at/100.0
+                                    RNS.log(str(self)+" Radio reporting short-term airtime limit is "+str(self.r_st_alock)+"%", RNS.LOG_DEBUG)
+                        elif (command == KISS.CMD_LT_ALOCK):
+                            if (byte == KISS.FESC):
+                                escape = True
+                            else:
+                                if (escape):
+                                    if (byte == KISS.TFEND):
+                                        byte = KISS.FEND
+                                    if (byte == KISS.TFESC):
+                                        byte = KISS.FESC
+                                    escape = False
+                                command_buffer = command_buffer+bytes([byte])
+                                if (len(command_buffer) == 2):
+                                    at = command_buffer[0] << 8 | command_buffer[1]
+                                    self.r_lt_alock = at/100.0
+                                    RNS.log(str(self)+" Radio reporting long-term airtime limit is "+str(self.r_lt_alock)+"%", RNS.LOG_DEBUG)
+                        elif (command == KISS.CMD_STAT_CHTM):
+                            if (byte == KISS.FESC):
+                                escape = True
+                            else:
+                                if (escape):
+                                    if (byte == KISS.TFEND):
+                                        byte = KISS.FEND
+                                    if (byte == KISS.TFESC):
+                                        byte = KISS.FESC
+                                    escape = False
+                                command_buffer = command_buffer+bytes([byte])
+                                if (len(command_buffer) == 8):
+                                    ats = command_buffer[0] << 8 | command_buffer[1]
+                                    atl = command_buffer[2] << 8 | command_buffer[3]
+                                    cus = command_buffer[4] << 8 | command_buffer[5]
+                                    cul = command_buffer[6] << 8 | command_buffer[7]
+                                    
+                                    self.r_airtime_short      = ats/100.0
+                                    self.r_airtime_long       = atl/100.0
+                                    self.r_channel_load_short = cus/100.0
+                                    self.r_channel_load_long  = cul/100.0
+                        elif (command == KISS.CMD_STAT_PHYPRM):
+                            if (byte == KISS.FESC):
+                                escape = True
+                            else:
+                                if (escape):
+                                    if (byte == KISS.TFEND):
+                                        byte = KISS.FEND
+                                    if (byte == KISS.TFESC):
+                                        byte = KISS.FESC
+                                    escape = False
+                                command_buffer = command_buffer+bytes([byte])
+                                if (len(command_buffer) == 10):
+                                    lst = (command_buffer[0] << 8 | command_buffer[1])/1000.0
+                                    lsr = command_buffer[2] << 8 | command_buffer[3]
+                                    prs = command_buffer[4] << 8 | command_buffer[5]
+                                    prt = command_buffer[6] << 8 | command_buffer[7]
+                                    cst = command_buffer[8] << 8 | command_buffer[9]
+
+                                    if lst != self.r_symbol_time_ms or lsr != self.r_symbol_rate or prs != self.r_preamble_symbols or prt != self.r_premable_time_ms or cst != self.r_csma_slot_time_ms:
+                                        self.r_symbol_time_ms    = lst
+                                        self.r_symbol_rate       = lsr
+                                        self.r_preamble_symbols  = prs
+                                        self.r_premable_time_ms  = prt
+                                        self.r_csma_slot_time_ms = cst
+                                        RNS.log(str(self)+" Radio reporting symbol time is "+str(round(self.r_symbol_time_ms,2))+"ms (at "+str(self.r_symbol_rate)+" baud)", RNS.LOG_DEBUG)
+                                        RNS.log(str(self)+" Radio reporting preamble is "+str(self.r_preamble_symbols)+" symbols ("+str(self.r_premable_time_ms)+"ms)", RNS.LOG_DEBUG)
+                                        RNS.log(str(self)+" Radio reporting CSMA slot time is "+str(self.r_csma_slot_time_ms)+"ms", RNS.LOG_DEBUG)
+                        elif (command == KISS.CMD_STAT_BAT):
+                            if (byte == KISS.FESC):
+                                escape = True
+                            else:
+                                if (escape):
+                                    if (byte == KISS.TFEND):
+                                        byte = KISS.FEND
+                                    if (byte == KISS.TFESC):
+                                        byte = KISS.FESC
+                                    escape = False
+                                command_buffer = command_buffer+bytes([byte])
+                                if (len(command_buffer) == 2):
+                                    bat_percent = command_buffer[1]
+                                    if bat_percent > 100:
+                                        bat_percent = 100
+                                    if bat_percent < 0:
+                                        bat_percent = 0
+                                    self.r_battery_state   = command_buffer[0]
+                                    self.r_battery_percent = bat_percent
                         elif (command == KISS.CMD_RANDOM):
                             self.r_random = byte
                         elif (command == KISS.CMD_PLATFORM):
@@ -572,9 +844,12 @@ class RNodeInterface(Interface):
                             if (byte == KISS.ERROR_INITRADIO):
                                 RNS.log(str(self)+" hardware initialisation error (code "+RNS.hexrep(byte)+")", RNS.LOG_ERROR)
                                 raise IOError("Radio initialisation failure")
-                            elif (byte == KISS.ERROR_INITRADIO):
+                            elif (byte == KISS.ERROR_TXFAILED):
                                 RNS.log(str(self)+" hardware TX error (code "+RNS.hexrep(byte)+")", RNS.LOG_ERROR)
                                 raise IOError("Hardware transmit failure")
+                            elif (byte == KISS.ERROR_MEMORY_LOW):
+                                RNS.log(str(self)+" hardware error (code "+RNS.hexrep(byte)+"): Memory exhausted", RNS.LOG_ERROR)
+                                self.hw_errors.append({"error": KISS.ERROR_MEMORY_LOW, "description": "Memory exhausted on connected device"})
                             else:
                                 RNS.log(str(self)+" hardware error (code "+RNS.hexrep(byte)+")", RNS.LOG_ERROR)
                                 raise IOError("Unknown hardware failure")
@@ -595,7 +870,7 @@ class RNodeInterface(Interface):
                 else:
                     time_since_last = int(time.time()*1000) - last_read_ms
                     if len(data_buffer) > 0 and time_since_last > self.timeout:
-                        RNS.log(str(self)+" serial read timeout", RNS.LOG_DEBUG)
+                        RNS.log(str(self)+" serial read timeout in command "+str(command), RNS.LOG_WARNING)
                         data_buffer = b""
                         in_frame = False
                         command = KISS.CMD_UNKNOWN
@@ -620,11 +895,17 @@ class RNodeInterface(Interface):
             RNS.log("Reticulum will attempt to reconnect the interface periodically.", RNS.LOG_ERROR)
 
         self.online = False
-        self.serial.close()
-        self.reconnect_port()
+        try:
+            self.serial.close()
+        except Exception as e:
+            pass
+
+        if not self.detached and not self.reconnecting:
+            self.reconnect_port()
 
     def reconnect_port(self):
-        while not self.online:
+        self.reconnecting = True
+        while not self.online and not self.detached:
             try:
                 time.sleep(5)
                 RNS.log("Attempting to reconnect serial port "+str(self.port)+" for "+str(self)+"...", RNS.LOG_VERBOSE)
@@ -634,8 +915,204 @@ class RNodeInterface(Interface):
             except Exception as e:
                 RNS.log("Error while reconnecting port, the contained exception was: "+str(e), RNS.LOG_ERROR)
 
-        RNS.log("Reconnected serial port for "+str(self))
+        self.reconnecting = False
+        if self.online:
+            RNS.log("Reconnected serial port for "+str(self))
+
+    def detach(self):
+        self.detached = True
+        self.disable_external_framebuffer()
+        self.setRadioState(KISS.RADIO_STATE_OFF)
+        self.leave()
+        
+        if self.use_ble:
+            self.ble.close()
+
+    def should_ingress_limit(self):
+        return False
+
+    def get_battery_state(self):
+        return self.r_battery_state
+
+    def get_battery_state_string(self):
+        if self.r_battery_state == RNodeInterface.BATTERY_STATE_CHARGED:
+            return "charged"
+        elif  self.r_battery_state == RNodeInterface.BATTERY_STATE_CHARGING:
+            return "charging"
+        elif self.r_battery_state == RNodeInterface.BATTERY_STATE_DISCHARGING:
+            return "discharging"
+        else:
+            return "unknown"
+
+    def get_battery_percent(self):
+        return self.r_battery_percent
+
+    def ble_receive(self, data):
+        with self.ble_rx_lock:
+            self.ble_rx_queue += data
+
+    def ble_waiting(self):
+        return len(self.ble_tx_queue) > 0
+
+    def get_ble_waiting(self, n):
+        with self.ble_tx_lock:
+            data = self.ble_tx_queue[:n]
+            self.ble_tx_queue = self.ble_tx_queue[n:]
+            return data
 
     def __str__(self):
         return "RNodeInterface["+str(self.name)+"]"
 
+class BLEConnection():
+    UART_SERVICE_UUID = "6E400001-B5A3-F393-E0A9-E50E24DCCA9E"
+    UART_RX_CHAR_UUID = "6E400002-B5A3-F393-E0A9-E50E24DCCA9E"
+    UART_TX_CHAR_UUID = "6E400003-B5A3-F393-E0A9-E50E24DCCA9E"
+    bleak = None
+
+    SCAN_TIMEOUT = 2.0
+    CONNECT_TIMEOUT = 5.0
+
+    @property
+    def is_open(self):
+        return self.connected
+
+    @property
+    def in_waiting(self):
+        buflen = len(self.owner.ble_rx_queue)
+        return buflen > 0
+
+    def write(self, data_bytes):
+        with self.owner.ble_tx_lock:
+            self.owner.ble_tx_queue += data_bytes
+            return len(data_bytes)
+
+    def read(self, n):
+        with self.owner.ble_rx_lock:
+            data = self.owner.ble_rx_queue[:n]
+            self.owner.ble_rx_queue = self.owner.ble_rx_queue[n:]
+            return data
+
+    def close(self):
+        if self.connected and self.ble_device:
+            RNS.log(f"Disconnecting BLE device from {self.owner}", RNS.LOG_DEBUG)
+            self.must_disconnect = True
+
+            while self.connect_job_running:
+                time.sleep(0.1)
+
+    def __init__(self, owner=None, target_name=None, target_bt_addr=None):
+        self.owner = owner
+        self.target_name = target_name
+        self.target_bt_addr = target_bt_addr
+        self.scan_timeout = BLEConnection.SCAN_TIMEOUT
+        self.ble_device = None
+        self.connected = False
+        self.running = False
+        self.should_run = False
+        self.must_disconnect = False
+        self.connect_job_running = False
+
+        import importlib
+        if BLEConnection.bleak == None:
+            if importlib.util.find_spec("bleak") != None:
+                import bleak
+                BLEConnection.bleak = bleak
+                
+                import asyncio
+                BLEConnection.asyncio = asyncio
+            else:
+                RNS.log("Using the RNode interface over BLE requires a the \"bleak\" module to be installed.", RNS.LOG_CRITICAL)
+                RNS.log("You can install one with the command: python3 -m pip install bleak", RNS.LOG_CRITICAL)
+                RNS.panic()
+
+        self.should_run = True
+        self.connection_thread = threading.Thread(target=self.connection_job, daemon=True).start()
+
+    def connection_job(self):
+        while (self.should_run):
+            if self.ble_device == None:
+                self.ble_device = self.find_target_device()
+
+            if type(self.ble_device) == self.bleak.backends.device.BLEDevice:
+                if not self.connected:
+                    self.connect_device()
+
+            time.sleep(1)
+
+    def connect_device(self):
+        if self.ble_device != None and type(self.ble_device) == self.bleak.backends.device.BLEDevice:
+            RNS.log(f"Connecting BLE device {self.ble_device} for {self.owner}...", RNS.LOG_DEBUG)
+
+            async def connect_job():
+                self.connect_job_running = True
+                async with self.bleak.BleakClient(self.ble_device, disconnected_callback=self.device_disconnected) as ble_client:
+                    def handle_rx(device, data):
+                        if self.owner != None:
+                            self.owner.ble_receive(data)
+
+                    self.connected = True
+                    self.ble_device = ble_client
+                    self.owner.port = str(f"ble://{ble_client.address}")
+
+                    loop = self.asyncio.get_running_loop()
+                    uart_service = ble_client.services.get_service(BLEConnection.UART_SERVICE_UUID)
+                    rx_characteristic = uart_service.get_characteristic(BLEConnection.UART_RX_CHAR_UUID)
+                    await ble_client.start_notify(BLEConnection.UART_TX_CHAR_UUID, handle_rx)
+
+                    while self.connected:
+                        if self.owner != None and self.owner.ble_waiting():
+                            outbound_data = self.owner.get_ble_waiting(rx_characteristic.max_write_without_response_size)
+                            await ble_client.write_gatt_char(rx_characteristic, outbound_data, response=False)
+                        elif self.must_disconnect:
+                            await ble_client.disconnect()
+                        else:
+                            await self.asyncio.sleep(0.1)
+
+
+            try:
+                self.asyncio.run(connect_job())
+            except Exception as e:
+                RNS.log(f"Could not connect BLE device {self.ble_device} for {self.owner}. Possibly missing authentication.", RNS.LOG_ERROR)
+            self.connect_job_running = False
+
+    def device_disconnected(self, device):
+        RNS.log(f"BLE device for {self.owner} disconnected", RNS.LOG_NOTICE)
+        self.connected = False
+        self.ble_device = None
+
+    def find_target_device(self):
+        RNS.log(f"Searching for attachable BLE device for {self.owner}...", RNS.LOG_EXTREME)
+        def device_filter(device: self.bleak.backends.device.BLEDevice, adv: self.bleak.backends.scanner.AdvertisementData):
+            if BLEConnection.UART_SERVICE_UUID.lower() in adv.service_uuids:
+                if self.device_bonded(device):
+                    if self.target_bt_addr == None and self.target_name == None:
+                        if device.name.startswith("RNode "):
+                            return True
+
+                    if self.target_bt_addr == None or (device.address != None and device.address == self.target_bt_addr):
+                        if self.target_name == None or (device.name != None and device.name == self.target_name):
+                            return True
+
+                else:
+                    if self.target_bt_addr != None and device.address == self.target_bt_addr:
+                        RNS.log(f"Can't connect to target device {self.target_bt_addr} over BLE, device is not bonded", RNS.LOG_ERROR)
+                    
+                    elif self.target_name != None and device.name == self.target_name:
+                        RNS.log(f"Can't connect to target device {self.target_name} over BLE, device is not bonded", RNS.LOG_ERROR)
+
+            return False
+
+        device = self.asyncio.run(self.bleak.BleakScanner.find_device_by_filter(device_filter, timeout=self.scan_timeout))
+        return device
+
+    def device_bonded(self, device):
+        try:
+            if hasattr(device, "details"):
+                if "props" in device.details and "Bonded" in device.details["props"]:
+                    if device.details["props"]["Bonded"] == True:
+                        return True
+        
+        except Exception as e:
+            RNS.log(f"Error while determining device bond status for {device}, the contained exception was: {e}", RNS.LOG_ERROR)
+
+        return False

RNS/Interfaces/SerialInterface.py

@@ -60,8 +60,7 @@ class SerialInterface(Interface):
             RNS.log("You can install one with the command: python3 -m pip install pyserial", RNS.LOG_CRITICAL)
             RNS.panic()
 
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
 
         self.HW_MTU = 564
         
@@ -201,5 +200,8 @@ class SerialInterface(Interface):
 
         RNS.log("Reconnected serial port for "+str(self))
 
+    def should_ingress_limit(self):
+        return False
+
     def __str__(self):
         return "SerialInterface["+self.name+"]"

RNS/Interfaces/TCPInterface.py

@@ -79,8 +79,7 @@ class TCPClientInterface(Interface):
     I2P_PROBES = 5
 
     def __init__(self, owner, name, target_ip=None, target_port=None, connected_socket=None, max_reconnect_tries=None, kiss_framing=False, i2p_tunneled = False, connect_timeout = None):
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
         
         self.HW_MTU = 1064
         
@@ -117,6 +116,8 @@ class TCPClientInterface(Interface):
             elif platform.system() == "Darwin":
                 self.set_timeouts_osx()
 
+            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+
         elif target_ip != None and target_port != None:
             self.receives    = True
             self.target_ip   = target_ip
@@ -201,6 +202,7 @@ class TCPClientInterface(Interface):
 
             self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
             self.socket.settimeout(TCPClientInterface.INITIAL_CONNECT_TIMEOUT)
+            self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
             self.socket.connect((self.target_ip, self.target_port))
             self.socket.settimeout(None)
             self.online  = True
@@ -408,29 +410,18 @@ class TCPServerInterface(Interface):
 
     @staticmethod
     def get_address_for_if(name):
-        import importlib
-        if importlib.util.find_spec('netifaces') != None:
-            import netifaces
-            return netifaces.ifaddresses(name)[netifaces.AF_INET][0]['addr']
-        else:
-            RNS.log("Getting interface addresses from device names requires the netifaces module.", RNS.LOG_CRITICAL)
-            RNS.log("You can install it with the command: python3 -m pip install netifaces", RNS.LOG_CRITICAL)
-            RNS.panic()
+        import RNS.vendor.ifaddr.niwrapper as netinfo
+        ifaddr = netinfo.ifaddresses(name)
+        return ifaddr[netinfo.AF_INET][0]["addr"]
 
     @staticmethod
     def get_broadcast_for_if(name):
-        import importlib
-        if importlib.util.find_spec('netifaces') != None:
-            import netifaces
-            return netifaces.ifaddresses(name)[netifaces.AF_INET][0]['broadcast']
-        else:
-            RNS.log("Getting interface addresses from device names requires the netifaces module.", RNS.LOG_CRITICAL)
-            RNS.log("You can install it with the command: python3 -m pip install netifaces", RNS.LOG_CRITICAL)
-            RNS.panic()
+        import RNS.vendor.ifaddr.niwrapper as netinfo
+        ifaddr = netinfo.ifaddresses(name)
+        return ifaddr[netinfo.AF_INET][0]["broadcast"]
 
     def __init__(self, owner, name, device=None, bindip=None, bindport=None, i2p_tunneled=False):
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
 
         self.HW_MTU = 1064
 
@@ -483,9 +474,27 @@ class TCPServerInterface(Interface):
         spawned_interface.target_port = str(handler.client_address[1])
         spawned_interface.parent_interface = self
         spawned_interface.bitrate = self.bitrate
+        
         spawned_interface.ifac_size = self.ifac_size
         spawned_interface.ifac_netname = self.ifac_netname
         spawned_interface.ifac_netkey = self.ifac_netkey
+        if spawned_interface.ifac_netname != None or spawned_interface.ifac_netkey != None:
+            ifac_origin = b""
+            if spawned_interface.ifac_netname != None:
+                ifac_origin += RNS.Identity.full_hash(spawned_interface.ifac_netname.encode("utf-8"))
+            if spawned_interface.ifac_netkey != None:
+                ifac_origin += RNS.Identity.full_hash(spawned_interface.ifac_netkey.encode("utf-8"))
+
+            ifac_origin_hash = RNS.Identity.full_hash(ifac_origin)
+            spawned_interface.ifac_key = RNS.Cryptography.hkdf(
+                length=64,
+                derive_from=ifac_origin_hash,
+                salt=RNS.Reticulum.IFAC_SALT,
+                context=None
+            )
+            spawned_interface.ifac_identity = RNS.Identity.from_bytes(spawned_interface.ifac_key)
+            spawned_interface.ifac_signature = spawned_interface.ifac_identity.sign(RNS.Identity.full_hash(spawned_interface.ifac_key))
+
         spawned_interface.announce_rate_target = self.announce_rate_target
         spawned_interface.announce_rate_grace = self.announce_rate_grace
         spawned_interface.announce_rate_penalty = self.announce_rate_penalty
@@ -497,6 +506,12 @@ class TCPServerInterface(Interface):
         self.clients += 1
         spawned_interface.read_loop()
 
+    def received_announce(self, from_spawned=False):
+        if from_spawned: self.ia_freq_deque.append(time.time())
+
+    def sent_announce(self, from_spawned=False):
+        if from_spawned: self.oa_freq_deque.append(time.time())
+
     def processOutgoing(self, data):
         pass
 

RNS/Interfaces/UDPInterface.py

@@ -34,29 +34,18 @@ class UDPInterface(Interface):
 
     @staticmethod
     def get_address_for_if(name):
-        import importlib
-        if importlib.util.find_spec('netifaces') != None:
-            import netifaces
-            return netifaces.ifaddresses(name)[netifaces.AF_INET][0]['addr']
-        else:
-            RNS.log("Getting interface addresses from device names requires the netifaces module.", RNS.LOG_CRITICAL)
-            RNS.log("You can install it with the command: python3 -m pip install netifaces", RNS.LOG_CRITICAL)
-            RNS.panic()
+        import RNS.vendor.ifaddr.niwrapper as netinfo
+        ifaddr = netinfo.ifaddresses(name)
+        return ifaddr[netinfo.AF_INET][0]["addr"]
 
     @staticmethod
     def get_broadcast_for_if(name):
-        import importlib
-        if importlib.util.find_spec('netifaces') != None:
-            import netifaces
-            return netifaces.ifaddresses(name)[netifaces.AF_INET][0]['broadcast']
-        else:
-            RNS.log("Getting interface addresses from device names requires the netifaces module.", RNS.LOG_CRITICAL)
-            RNS.log("You can install it with the command: python3 -m pip install netifaces", RNS.LOG_CRITICAL)
-            RNS.panic()
+        import RNS.vendor.ifaddr.niwrapper as netinfo
+        ifaddr = netinfo.ifaddresses(name)
+        return ifaddr[netinfo.AF_INET][0]["broadcast"]
 
     def __init__(self, owner, name, device=None, bindip=None, bindport=None, forwardip=None, forwardport=None):
-        self.rxb = 0
-        self.txb = 0
+        super().__init__()
 
         self.HW_MTU = 1064
 

RNS/Interfaces/__init__.py

@@ -22,6 +22,7 @@
 
 import os
 import glob
+import RNS.Interfaces.Android
 
 modules = glob.glob(os.path.dirname(__file__)+"/*.py")
-__all__ = [ os.path.basename(f)[:-3] for f in modules if not f.endswith('__init__.py')]
+__all__ = [ os.path.basename(f)[:-3] for f in modules if not f.endswith('__init__.py')]

RNS/Link.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2024 Mark Qvist / unsigned.io and contributors.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -22,10 +22,12 @@
 
 from RNS.Cryptography import X25519PrivateKey, X25519PublicKey, Ed25519PrivateKey, Ed25519PublicKey
 from RNS.Cryptography import Fernet
+from RNS.Channel import Channel, LinkChannelOutlet
 
 from time import sleep
 from .vendor import umsgpack as umsgpack
 import threading
+import inspect
 import math
 import time
 import RNS
@@ -66,6 +68,7 @@ class Link:
     Timeout for link establishment in seconds per hop to destination.
     """
 
+    TRAFFIC_TIMEOUT_MIN_MS = 5
     TRAFFIC_TIMEOUT_FACTOR = 6
     KEEPALIVE_TIMEOUT_FACTOR = 4
     """
@@ -110,9 +113,10 @@ class Link:
                 link = Link(owner = owner, peer_pub_bytes=data[:Link.ECPUBSIZE//2], peer_sig_pub_bytes=data[Link.ECPUBSIZE//2:Link.ECPUBSIZE])
                 link.set_link_id(packet)
                 link.destination = packet.destination
-                link.establishment_timeout = Link.ESTABLISHMENT_TIMEOUT_PER_HOP * max(1, packet.hops)
+                link.establishment_timeout = Link.ESTABLISHMENT_TIMEOUT_PER_HOP * max(1, packet.hops) + Link.KEEPALIVE
                 link.establishment_cost += len(packet.raw)
                 RNS.log("Validating link request "+RNS.prettyhexrep(link.link_id), RNS.LOG_VERBOSE)
+                RNS.log(f"Establishment timeout is {RNS.prettytime(link.establishment_timeout)} for incoming link request "+RNS.prettyhexrep(link.link_id), RNS.LOG_EXTREME)
                 link.handshake()
                 link.attached_interface = packet.receiving_interface
                 link.prove()
@@ -121,7 +125,7 @@ class Link:
                 link.last_inbound = time.time()
                 link.start_watchdog()
                 
-                RNS.log("Incoming link request "+str(link)+" accepted", RNS.LOG_VERBOSE)
+                RNS.log("Incoming link request "+str(link)+" accepted on "+str(link.attached_interface), RNS.LOG_DEBUG)
                 return link
 
             except Exception as e:
@@ -130,7 +134,7 @@ class Link:
                 return None
 
         else:
-            RNS.log("Invalid link request payload size, dropping request", RNS.LOG_VERBOSE)
+            RNS.log("Invalid link request payload size, dropping request", RNS.LOG_DEBUG)
             return None
 
 
@@ -139,6 +143,7 @@ class Link:
             raise TypeError("Links can only be established to the \"single\" destination type")
         self.rtt = None
         self.establishment_cost = 0
+        self.establishment_rate = None
         self.callbacks = LinkCallbacks()
         self.resource_strategy = Link.ACCEPT_NONE
         self.outgoing_resources = []
@@ -146,10 +151,15 @@ class Link:
         self.pending_requests   = []
         self.last_inbound = 0
         self.last_outbound = 0
+        self.last_proof = 0
+        self.last_data = 0
         self.tx = 0
         self.rx = 0
         self.txbytes = 0
         self.rxbytes = 0
+        self.rssi = None
+        self.snr = None
+        self.q = None
         self.traffic_timeout_factor = Link.TRAFFIC_TIMEOUT_FACTOR
         self.keepalive_timeout_factor = Link.KEEPALIVE_TIMEOUT_FACTOR
         self.keepalive = Link.KEEPALIVE
@@ -160,15 +170,21 @@ class Link:
         self.type = RNS.Destination.LINK
         self.owner = owner
         self.destination = destination
+        self.expected_hops = None
         self.attached_interface = None
         self.__remote_identity = None
+        self.__track_phy_stats = False
+        self._channel = None
+
         if self.destination == None:
             self.initiator = False
             self.prv     = X25519PrivateKey.generate()
             self.sig_prv = self.owner.identity.sig_prv
         else:
             self.initiator = True
-            self.establishment_timeout = Link.ESTABLISHMENT_TIMEOUT_PER_HOP * max(1, RNS.Transport.hops_to(destination.hash))
+            self.expected_hops = RNS.Transport.hops_to(self.destination.hash)
+            self.establishment_timeout  = RNS.Reticulum.get_instance().get_first_hop_timeout(destination.hash)
+            self.establishment_timeout += Link.ESTABLISHMENT_TIMEOUT_PER_HOP * max(1, RNS.Transport.hops_to(destination.hash))
             self.prv     = X25519PrivateKey.generate()
             self.sig_prv = Ed25519PrivateKey.generate()
 
@@ -204,6 +220,7 @@ class Link:
             self.packet.send()
             self.had_outbound()
             RNS.log("Link request "+RNS.prettyhexrep(self.link_id)+" sent to "+str(self.destination), RNS.LOG_DEBUG)
+            RNS.log(f"Establishment timeout is {RNS.prettytime(self.establishment_timeout)} for link request "+RNS.prettyhexrep(self.link_id), RNS.LOG_EXTREME)
 
 
     def load_peer(self, peer_pub_bytes, peer_sig_pub_bytes):
@@ -221,15 +238,18 @@ class Link:
         self.hash = self.link_id
 
     def handshake(self):
-        self.status = Link.HANDSHAKE
-        self.shared_key = self.prv.exchange(self.peer_pub)
+        if self.status == Link.PENDING and self.prv != None:
+            self.status = Link.HANDSHAKE
+            self.shared_key = self.prv.exchange(self.peer_pub)
 
-        self.derived_key = RNS.Cryptography.hkdf(
-            length=32,
-            derive_from=self.shared_key,
-            salt=self.get_salt(),
-            context=self.get_context(),
-        )
+            self.derived_key = RNS.Cryptography.hkdf(
+                length=32,
+                derive_from=self.shared_key,
+                salt=self.get_salt(),
+                context=self.get_context(),
+            )
+        else:
+            RNS.log("Handshake attempt on "+str(self)+" with invalid state "+str(self.status), RNS.LOG_ERROR)
 
 
     def prove(self):
@@ -257,36 +277,50 @@ class Link:
         self.had_outbound()
 
     def validate_proof(self, packet):
-        if self.status == Link.PENDING:
-            if self.initiator and len(packet.data) == RNS.Identity.SIGLENGTH//8+Link.ECPUBSIZE//2:
-                peer_pub_bytes = packet.data[RNS.Identity.SIGLENGTH//8:RNS.Identity.SIGLENGTH//8+Link.ECPUBSIZE//2]
-                peer_sig_pub_bytes = self.destination.identity.get_public_key()[Link.ECPUBSIZE//2:Link.ECPUBSIZE]
-                self.load_peer(peer_pub_bytes, peer_sig_pub_bytes)
-                self.handshake()
+        try:
+            if self.status == Link.PENDING:
+                if self.initiator and len(packet.data) == RNS.Identity.SIGLENGTH//8+Link.ECPUBSIZE//2:
+                    peer_pub_bytes = packet.data[RNS.Identity.SIGLENGTH//8:RNS.Identity.SIGLENGTH//8+Link.ECPUBSIZE//2]
+                    peer_sig_pub_bytes = self.destination.identity.get_public_key()[Link.ECPUBSIZE//2:Link.ECPUBSIZE]
+                    self.load_peer(peer_pub_bytes, peer_sig_pub_bytes)
+                    self.handshake()
 
-                self.establishment_cost += len(packet.raw)
-                signed_data = self.link_id+self.peer_pub_bytes+self.peer_sig_pub_bytes
-                signature = packet.data[:RNS.Identity.SIGLENGTH//8]
-                
-                if self.destination.identity.validate(signature, signed_data):
-                    self.rtt = time.time() - self.request_time
-                    self.attached_interface = packet.receiving_interface
-                    self.__remote_identity = self.destination.identity
-                    RNS.Transport.activate_link(self)
-                    RNS.log("Link "+str(self)+" established with "+str(self.destination)+", RTT is "+str(round(self.rtt, 3))+"s", RNS.LOG_VERBOSE)
-                    rtt_data = umsgpack.packb(self.rtt)
-                    rtt_packet = RNS.Packet(self, rtt_data, context=RNS.Packet.LRRTT)
-                    rtt_packet.send()
-                    self.had_outbound()
+                    self.establishment_cost += len(packet.raw)
+                    signed_data = self.link_id+self.peer_pub_bytes+self.peer_sig_pub_bytes
+                    signature = packet.data[:RNS.Identity.SIGLENGTH//8]
+                    
+                    if self.destination.identity.validate(signature, signed_data):
+                        if self.status != Link.HANDSHAKE:
+                            raise IOError("Invalid link state for proof validation: "+str(self.status))
 
-                    self.status = Link.ACTIVE
-                    self.activated_at = time.time()
-                    if self.callbacks.link_established != None:
-                        thread = threading.Thread(target=self.callbacks.link_established, args=(self,))
-                        thread.daemon = True
-                        thread.start()
-                else:
-                    RNS.log("Invalid link proof signature received by "+str(self)+". Ignoring.", RNS.LOG_DEBUG)
+                        self.rtt = time.time() - self.request_time
+                        self.attached_interface = packet.receiving_interface
+                        self.__remote_identity = self.destination.identity
+                        self.status = Link.ACTIVE
+                        self.activated_at = time.time()
+                        self.last_proof = self.activated_at
+                        RNS.Transport.activate_link(self)
+                        RNS.log("Link "+str(self)+" established with "+str(self.destination)+", RTT is "+str(round(self.rtt, 3))+"s", RNS.LOG_VERBOSE)
+                        
+                        if self.rtt != None and self.establishment_cost != None and self.rtt > 0 and self.establishment_cost > 0:
+                            self.establishment_rate = self.establishment_cost/self.rtt
+
+                        rtt_data = umsgpack.packb(self.rtt)
+                        rtt_packet = RNS.Packet(self, rtt_data, context=RNS.Packet.LRRTT)
+                        rtt_packet.send()
+                        self.had_outbound()
+
+                        if self.callbacks.link_established != None:
+                            thread = threading.Thread(target=self.callbacks.link_established, args=(self,))
+                            thread.daemon = True
+                            thread.start()
+                    else:
+                        RNS.log("Invalid link proof signature received by "+str(self)+". Ignoring.", RNS.LOG_DEBUG)
+        
+        except Exception as e:
+            self.status = Link.CLOSED
+            RNS.log("An error ocurred while validating link request proof on "+str(self)+".", RNS.LOG_ERROR)
+            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
 
 
     def identify(self, identity):
@@ -298,7 +332,7 @@ class Link:
 
         :param identity: An RNS.Identity instance to identify as.
         """
-        if self.initiator:
+        if self.initiator and self.status == Link.ACTIVE:
             signed_data = self.link_id + identity.get_public_key()
             signature = identity.sign(signed_data)
             proof_data = identity.get_public_key() + signature
@@ -324,7 +358,7 @@ class Link:
         packed_request    = umsgpack.packb(unpacked_request)
 
         if timeout == None:
-            timeout = self.rtt * self.traffic_timeout_factor + RNS.Resource.RESPONSE_MAX_GRACE_TIME/4.0
+            timeout = self.rtt * self.traffic_timeout_factor + RNS.Resource.RESPONSE_MAX_GRACE_TIME*1.125
 
         if len(packed_request) <= Link.MDU:
             request_packet   = RNS.Packet(self, packed_request, RNS.Packet.DATA, context = RNS.Packet.REQUEST)
@@ -362,34 +396,86 @@ class Link:
 
     def rtt_packet(self, packet):
         try:
-            # TODO: This is crude, we should use the delta
-            # to model a more representative per-bit round
-            # trip time, and use that to set a sensible RTT
-            # expectancy for the link. This will have to do
-            # for now though.
             measured_rtt = time.time() - self.request_time
             plaintext = self.decrypt(packet.data)
-            rtt = umsgpack.unpackb(plaintext)
-            self.rtt = max(measured_rtt, rtt)
-            self.status = Link.ACTIVE
-            self.activated_at = time.time()
+            if plaintext != None:
+                rtt = umsgpack.unpackb(plaintext)
+                self.rtt = max(measured_rtt, rtt)
+                self.status = Link.ACTIVE
+                self.activated_at = time.time()
+
+                if self.rtt != None and self.establishment_cost != None and self.rtt > 0 and self.establishment_cost > 0:
+                    self.establishment_rate = self.establishment_cost/self.rtt
+
+                try:
+                    if self.owner.callbacks.link_established != None:
+                            self.owner.callbacks.link_established(self)
+                except Exception as e:
+                    RNS.log("Error occurred in external link establishment callback. The contained exception was: "+str(e), RNS.LOG_ERROR)
 
-            
-            if self.owner.callbacks.link_established != None:
-                    self.owner.callbacks.link_established(self)
         except Exception as e:
             RNS.log("Error occurred while processing RTT packet, tearing down link. The contained exception was: "+str(e), RNS.LOG_ERROR)
             self.teardown()
 
+    def track_phy_stats(self, track):
+        """
+        You can enable physical layer statistics on a per-link basis. If this is enabled,
+        and the link is running over an interface that supports reporting physical layer
+        statistics, you will be able to retrieve stats such as *RSSI*, *SNR* and physical
+        *Link Quality* for the link.
+
+        :param track: Whether or not to keep track of physical layer statistics. Value must be ``True`` or ``False``.
+        """
+        if track:
+            self.__track_phy_stats = True
+        else:
+            self.__track_phy_stats = False
+
+    def get_rssi(self):
+        """
+        :returns: The physical layer *Received Signal Strength Indication* if available, otherwise ``None``. Physical layer statistics must be enabled on the link for this method to return a value.
+        """
+        return self.rssi
+
+    def get_snr(self):
+        """
+        :returns: The physical layer *Signal-to-Noise Ratio* if available, otherwise ``None``. Physical layer statistics must be enabled on the link for this method to return a value.
+        """
+        return self.rssi
+
+    def get_q(self):
+        """
+        :returns: The physical layer *Link Quality* if available, otherwise ``None``. Physical layer statistics must be enabled on the link for this method to return a value.
+        """
+        return self.rssi
+
+    def get_establishment_rate(self):
+        """
+        :returns: The data transfer rate at which the link establishment procedure ocurred, in bits per second.
+        """
+        if self.establishment_rate != None:
+            return self.establishment_rate*8
+        else:
+            return None
+
     def get_salt(self):
         return self.link_id
 
     def get_context(self):
         return None
 
+    def get_age(self):
+        """
+        :returns: The time in seconds since this link was established.
+        """
+        if self.activated_at:
+            return time.time() - self.activated_at
+        else:
+            return None
+
     def no_inbound_for(self):
         """
-        :returns: The time in seconds since last inbound packet on the link.
+        :returns: The time in seconds since last inbound packet on the link. This includes keepalive packets.
         """
         activated_at = self.activated_at if self.activated_at != None else 0
         last_inbound = max(self.last_inbound, activated_at)
@@ -397,13 +483,19 @@ class Link:
 
     def no_outbound_for(self):
         """
-        :returns: The time in seconds since last outbound packet on the link.
+        :returns: The time in seconds since last outbound packet on the link. This includes keepalive packets.
         """
         return time.time() - self.last_outbound
 
+    def no_data_for(self):
+        """
+        :returns: The time in seconds since payload data traversed the link. This excludes keepalive packets.
+        """
+        return time.time() - self.last_data
+
     def inactive_for(self):
         """
-        :returns: The time in seconds since activity on the link.
+        :returns: The time in seconds since activity on the link. This includes keepalive packets.
         """
         return min(self.no_inbound_for(), self.no_outbound_for())
 
@@ -413,8 +505,10 @@ class Link:
         """
         return self.__remote_identity
 
-    def had_outbound(self):
+    def had_outbound(self, is_keepalive=False):
         self.last_outbound = time.time()
+        if not is_keepalive:
+            self.last_data = self.last_outbound
 
     def teardown(self):
         """
@@ -441,6 +535,7 @@ class Link:
                     self.teardown_reason = Link.DESTINATION_CLOSED
                 else:
                     self.teardown_reason = Link.INITIATOR_CLOSED
+                self.__update_phy_stats(packet)
                 self.link_closed()
         except Exception as e:
             pass
@@ -450,6 +545,8 @@ class Link:
             resource.cancel()
         for resource in self.outgoing_resources:
             resource.cancel()
+        if self._channel:
+            self._channel._shutdown()
             
         self.prv = None
         self.pub = None
@@ -477,7 +574,11 @@ class Link:
     def __watchdog_job(self):
         while not self.status == Link.CLOSED:
             while (self.watchdog_lock):
-                sleep(max(self.rtt, 0.025))
+                rtt_wait = 0.025
+                if hasattr(self, "rtt") and self.rtt:
+                    rtt_wait = self.rtt
+
+                sleep(max(rtt_wait, 0.025))
 
             if not self.status == Link.CLOSED:
                 # Link was initiated, but no response
@@ -496,19 +597,19 @@ class Link:
                     next_check = self.request_time + self.establishment_timeout
                     sleep_time = next_check - time.time()
                     if time.time() >= self.request_time + self.establishment_timeout:
-                        if self.initiator:
-                            RNS.log("Timeout waiting for link request proof", RNS.LOG_DEBUG)
-                        else:
-                            RNS.log("Timeout waiting for RTT packet from link initiator", RNS.LOG_DEBUG)
-
                         self.status = Link.CLOSED
                         self.teardown_reason = Link.TIMEOUT
                         self.link_closed()
                         sleep_time = 0.001
 
+                        if self.initiator:
+                            RNS.log("Timeout waiting for link request proof", RNS.LOG_DEBUG)
+                        else:
+                            RNS.log("Timeout waiting for RTT packet from link initiator", RNS.LOG_DEBUG)
+
                 elif self.status == Link.ACTIVE:
                     activated_at = self.activated_at if self.activated_at != None else 0
-                    last_inbound = max(self.last_inbound, activated_at)
+                    last_inbound = max(max(self.last_inbound, self.last_proof), activated_at)
 
                     if time.time() >= last_inbound + self.keepalive:
                         if self.initiator:
@@ -540,10 +641,25 @@ class Link:
                 sleep(sleep_time)
 
 
+    def __update_phy_stats(self, packet, query_shared = True):
+        if self.__track_phy_stats:
+            if query_shared:
+                reticulum = RNS.Reticulum.get_instance()
+                if packet.rssi == None: packet.rssi = reticulum.get_packet_rssi(packet.packet_hash)
+                if packet.snr  == None: packet.snr  = reticulum.get_packet_snr(packet.packet_hash)
+                if packet.q    == None: packet.q    = reticulum.get_packet_q(packet.packet_hash)
+
+            if packet.rssi != None:
+                self.rssi = packet.rssi
+            if packet.snr != None:
+                self.snr = packet.snr
+            if packet.q != None:
+                self.q = packet.q
+    
     def send_keepalive(self):
         keepalive_packet = RNS.Packet(self, bytes([0xFF]), context=RNS.Packet.KEEPALIVE)
         keepalive_packet.send()
-        self.had_outbound()
+        self.had_outbound(is_keepalive = True)
 
     def handle_request(self, request_id, unpacked_request):
         if self.status == Link.ACTIVE:
@@ -568,7 +684,13 @@ class Link:
 
                 if allowed:
                     RNS.log("Handling request "+RNS.prettyhexrep(request_id)+" for: "+str(path), RNS.LOG_DEBUG)
-                    response = response_generator(path, request_data, request_id, self.__remote_identity, requested_at)
+                    if len(inspect.signature(response_generator).parameters) == 5:
+                        response = response_generator(path, request_data, request_id, self.__remote_identity, requested_at)
+                    elif len(inspect.signature(response_generator).parameters) == 6:
+                        response = response_generator(path, request_data, request_id, self.link_id, self.__remote_identity, requested_at)
+                    else:
+                        raise TypeError("Invalid signature for response generator callback")
+
                     if response != None:
                         packed_response = umsgpack.packb([request_id, response])
 
@@ -588,7 +710,9 @@ class Link:
                     remove = pending_request
                     try:
                         pending_request.response_size = response_size
-                        pending_request.response_transfer_size = response_transfer_size
+                        if pending_request.response_transfer_size == None:
+                            pending_request.response_transfer_size = 0
+                        pending_request.response_transfer_size += response_transfer_size
                         pending_request.response_received(response_data)
                     except Exception as e:
                         RNS.log("Error occurred while handling response. The contained exception was: "+str(e), RNS.LOG_ERROR)
@@ -624,142 +748,190 @@ class Link:
                 if pending_request.request_id == resource.request_id:
                     pending_request.request_timed_out(None)
 
+    def get_channel(self):
+        """
+        Get the ``Channel`` for this link.
+
+        :return: ``Channel`` object
+        """
+        if self._channel is None:
+            self._channel = Channel(LinkChannelOutlet(self))
+        return self._channel
+
     def receive(self, packet):
         self.watchdog_lock = True
         if not self.status == Link.CLOSED and not (self.initiator and packet.context == RNS.Packet.KEEPALIVE and packet.data == bytes([0xFF])):
             if packet.receiving_interface != self.attached_interface:
-                RNS.log("Link-associated packet received on unexpected interface! Someone might be trying to manipulate your communication!", RNS.LOG_ERROR)
+                RNS.log(f"Link-associated packet received on unexpected interface {packet.receiving_interface} instead of {self.attached_interface}! Someone might be trying to manipulate your communication!", RNS.LOG_ERROR)
             else:
                 self.last_inbound = time.time()
+                if packet.context != RNS.Packet.KEEPALIVE:
+                    self.last_data = self.last_inbound
                 self.rx += 1
                 self.rxbytes += len(packet.data)
                 if self.status == Link.STALE:
                     self.status = Link.ACTIVE
 
                 if packet.packet_type == RNS.Packet.DATA:
+                    should_query = False
                     if packet.context == RNS.Packet.NONE:
                         plaintext = self.decrypt(packet.data)
-                        if self.callbacks.packet != None:
-                            thread = threading.Thread(target=self.callbacks.packet, args=(plaintext, packet))
-                            thread.daemon = True
-                            thread.start()
-                        
-                        if self.destination.proof_strategy == RNS.Destination.PROVE_ALL:
-                            packet.prove()
+                        packet.ratchet_id = self.link_id
+                        if plaintext != None:
+                            if self.callbacks.packet != None:
+                                thread = threading.Thread(target=self.callbacks.packet, args=(plaintext, packet))
+                                thread.daemon = True
+                                thread.start()
+                            
+                            if self.destination.proof_strategy == RNS.Destination.PROVE_ALL:
+                                packet.prove()
+                                should_query = True
 
-                        elif self.destination.proof_strategy == RNS.Destination.PROVE_APP:
-                            if self.destination.callbacks.proof_requested:
-                                try:
-                                    self.destination.callbacks.proof_requested(packet)
-                                except Exception as e:
-                                    RNS.log("Error while executing proof request callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            elif self.destination.proof_strategy == RNS.Destination.PROVE_APP:
+                                if self.destination.callbacks.proof_requested:
+                                    try:
+                                        if self.destination.callbacks.proof_requested(packet):
+                                            packet.prove()
+                                            should_query = True
+                                    except Exception as e:
+                                        RNS.log("Error while executing proof request callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+                            self.__update_phy_stats(packet, query_shared=should_query)
 
                     elif packet.context == RNS.Packet.LINKIDENTIFY:
                         plaintext = self.decrypt(packet.data)
+                        if plaintext != None:
+                            if not self.initiator and len(plaintext) == RNS.Identity.KEYSIZE//8 + RNS.Identity.SIGLENGTH//8:
+                                public_key   = plaintext[:RNS.Identity.KEYSIZE//8]
+                                signed_data  = self.link_id+public_key
+                                signature    = plaintext[RNS.Identity.KEYSIZE//8:RNS.Identity.KEYSIZE//8+RNS.Identity.SIGLENGTH//8]
+                                identity     = RNS.Identity(create_keys=False)
+                                identity.load_public_key(public_key)
 
-                        if not self.initiator and len(plaintext) == RNS.Identity.KEYSIZE//8 + RNS.Identity.SIGLENGTH//8:
-                            public_key   = plaintext[:RNS.Identity.KEYSIZE//8]
-                            signed_data  = self.link_id+public_key
-                            signature    = plaintext[RNS.Identity.KEYSIZE//8:RNS.Identity.KEYSIZE//8+RNS.Identity.SIGLENGTH//8]
-                            identity     = RNS.Identity(create_keys=False)
-                            identity.load_public_key(public_key)
-
-                            if identity.validate(signature, signed_data):
-                                self.__remote_identity = identity
-                                if self.callbacks.remote_identified != None:
-                                    try:
-                                        self.callbacks.remote_identified(self, self.__remote_identity)
-                                    except Exception as e:
-                                        RNS.log("Error while executing remote identified callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+                                if identity.validate(signature, signed_data):
+                                    self.__remote_identity = identity
+                                    if self.callbacks.remote_identified != None:
+                                        try:
+                                            self.callbacks.remote_identified(self, self.__remote_identity)
+                                        except Exception as e:
+                                            RNS.log("Error while executing remote identified callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+                                
+                                    self.__update_phy_stats(packet, query_shared=True)
 
                     elif packet.context == RNS.Packet.REQUEST:
                         try:
                             request_id = packet.getTruncatedHash()
                             packed_request = self.decrypt(packet.data)
-                            unpacked_request = umsgpack.unpackb(packed_request)
-                            self.handle_request(request_id, unpacked_request)
+                            if packed_request != None:
+                                unpacked_request = umsgpack.unpackb(packed_request)
+                                self.handle_request(request_id, unpacked_request)
+                                self.__update_phy_stats(packet, query_shared=True)
                         except Exception as e:
                             RNS.log("Error occurred while handling request. The contained exception was: "+str(e), RNS.LOG_ERROR)
 
                     elif packet.context == RNS.Packet.RESPONSE:
                         try:
                             packed_response = self.decrypt(packet.data)
-                            unpacked_response = umsgpack.unpackb(packed_response)
-                            request_id = unpacked_response[0]
-                            response_data = unpacked_response[1]
-                            transfer_size = len(umsgpack.packb(response_data))-2
-                            self.handle_response(request_id, response_data, transfer_size, transfer_size)
+                            if packed_response != None:
+                                unpacked_response = umsgpack.unpackb(packed_response)
+                                request_id = unpacked_response[0]
+                                response_data = unpacked_response[1]
+                                transfer_size = len(umsgpack.packb(response_data))-2
+                                self.handle_response(request_id, response_data, transfer_size, transfer_size)
+                                self.__update_phy_stats(packet, query_shared=True)
                         except Exception as e:
                             RNS.log("Error occurred while handling response. The contained exception was: "+str(e), RNS.LOG_ERROR)
 
                     elif packet.context == RNS.Packet.LRRTT:
                         if not self.initiator:
                             self.rtt_packet(packet)
+                            self.__update_phy_stats(packet, query_shared=True)
 
                     elif packet.context == RNS.Packet.LINKCLOSE:
                         self.teardown_packet(packet)
+                        self.__update_phy_stats(packet, query_shared=True)
 
                     elif packet.context == RNS.Packet.RESOURCE_ADV:
                         packet.plaintext = self.decrypt(packet.data)
+                        if packet.plaintext != None:
+                            self.__update_phy_stats(packet, query_shared=True)
 
-                        if RNS.ResourceAdvertisement.is_request(packet):
-                            RNS.Resource.accept(packet, callback=self.request_resource_concluded)
-                        elif RNS.ResourceAdvertisement.is_response(packet):
-                            request_id = RNS.ResourceAdvertisement.read_request_id(packet)
-                            for pending_request in self.pending_requests:
-                                if pending_request.request_id == request_id:
-                                    RNS.Resource.accept(packet, callback=self.response_resource_concluded, progress_callback=pending_request.response_resource_progress, request_id = request_id)
-                                    pending_request.response_size = RNS.ResourceAdvertisement.read_size(packet)
-                                    pending_request.response_transfer_size = RNS.ResourceAdvertisement.read_transfer_size(packet)
-                                    pending_request.started_at = time.time()
-                        elif self.resource_strategy == Link.ACCEPT_NONE:
-                            pass
-                        elif self.resource_strategy == Link.ACCEPT_APP:
-                            if self.callbacks.resource != None:
-                                try:
-                                    resource_advertisement = RNS.ResourceAdvertisement.unpack(packet.plaintext)
-                                    resource_advertisement.link = self
-                                    if self.callbacks.resource(resource_advertisement):
-                                        RNS.Resource.accept(packet, self.callbacks.resource_concluded)
-                                except Exception as e:
-                                    RNS.log("Error while executing resource accept callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
-                        elif self.resource_strategy == Link.ACCEPT_ALL:
-                            RNS.Resource.accept(packet, self.callbacks.resource_concluded)
+                            if RNS.ResourceAdvertisement.is_request(packet):
+                                RNS.Resource.accept(packet, callback=self.request_resource_concluded)
+                            elif RNS.ResourceAdvertisement.is_response(packet):
+                                request_id = RNS.ResourceAdvertisement.read_request_id(packet)
+                                for pending_request in self.pending_requests:
+                                    if pending_request.request_id == request_id:
+                                        response_resource = RNS.Resource.accept(packet, callback=self.response_resource_concluded, progress_callback=pending_request.response_resource_progress, request_id = request_id)
+                                        if response_resource != None:
+                                            if pending_request.response_size == None:
+                                                pending_request.response_size = RNS.ResourceAdvertisement.read_size(packet)
+                                            if pending_request.response_transfer_size == None:
+                                                pending_request.response_transfer_size = 0
+                                            pending_request.response_transfer_size += RNS.ResourceAdvertisement.read_transfer_size(packet)
+                                            if pending_request.started_at == None:
+                                                pending_request.started_at = time.time()
+                                            pending_request.response_resource_progress(response_resource)
+
+                            elif self.resource_strategy == Link.ACCEPT_NONE:
+                                pass
+                            elif self.resource_strategy == Link.ACCEPT_APP:
+                                if self.callbacks.resource != None:
+                                    try:
+                                        resource_advertisement = RNS.ResourceAdvertisement.unpack(packet.plaintext)
+                                        resource_advertisement.link = self
+                                        if self.callbacks.resource(resource_advertisement):
+                                            RNS.Resource.accept(packet, self.callbacks.resource_concluded)
+                                    except Exception as e:
+                                        RNS.log("Error while executing resource accept callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            elif self.resource_strategy == Link.ACCEPT_ALL:
+                                RNS.Resource.accept(packet, self.callbacks.resource_concluded)
 
                     elif packet.context == RNS.Packet.RESOURCE_REQ:
                         plaintext = self.decrypt(packet.data)
-                        if ord(plaintext[:1]) == RNS.Resource.HASHMAP_IS_EXHAUSTED:
-                            resource_hash = plaintext[1+RNS.Resource.MAPHASH_LEN:RNS.Identity.HASHLENGTH//8+1+RNS.Resource.MAPHASH_LEN]
-                        else:
-                            resource_hash = plaintext[1:RNS.Identity.HASHLENGTH//8+1]
+                        if plaintext != None:
+                            self.__update_phy_stats(packet, query_shared=True)
+                            if ord(plaintext[:1]) == RNS.Resource.HASHMAP_IS_EXHAUSTED:
+                                resource_hash = plaintext[1+RNS.Resource.MAPHASH_LEN:RNS.Identity.HASHLENGTH//8+1+RNS.Resource.MAPHASH_LEN]
+                            else:
+                                resource_hash = plaintext[1:RNS.Identity.HASHLENGTH//8+1]
 
-                        for resource in self.outgoing_resources:
-                            if resource.hash == resource_hash:
-                                # We need to check that this request has not been
-                                # received before in order to avoid sequencing errors.
-                                if not packet.packet_hash in resource.req_hashlist:
-                                    resource.req_hashlist.append(packet.packet_hash)
-                                    resource.request(plaintext)
+                            for resource in self.outgoing_resources:
+                                if resource.hash == resource_hash:
+                                    # We need to check that this request has not been
+                                    # received before in order to avoid sequencing errors.
+                                    if not packet.packet_hash in resource.req_hashlist:
+                                        resource.req_hashlist.append(packet.packet_hash)
+                                        resource.request(plaintext)
+                                        
+                                        # TODO: Test and possibly enable this at some point
+                                        # def request_job():
+                                        #     resource.request(plaintext)
+                                        # threading.Thread(target=request_job, daemon=True).start()
 
                     elif packet.context == RNS.Packet.RESOURCE_HMU:
                         plaintext = self.decrypt(packet.data)
-                        resource_hash = plaintext[:RNS.Identity.HASHLENGTH//8]
-                        for resource in self.incoming_resources:
-                            if resource_hash == resource.hash:
-                                resource.hashmap_update_packet(plaintext)
+                        if plaintext != None:
+                            self.__update_phy_stats(packet, query_shared=True)
+                            resource_hash = plaintext[:RNS.Identity.HASHLENGTH//8]
+                            for resource in self.incoming_resources:
+                                if resource_hash == resource.hash:
+                                    resource.hashmap_update_packet(plaintext)
 
                     elif packet.context == RNS.Packet.RESOURCE_ICL:
                         plaintext = self.decrypt(packet.data)
-                        resource_hash = plaintext[:RNS.Identity.HASHLENGTH//8]
-                        for resource in self.incoming_resources:
-                            if resource_hash == resource.hash:
-                                resource.cancel()
+                        if plaintext != None:
+                            self.__update_phy_stats(packet)
+                            resource_hash = plaintext[:RNS.Identity.HASHLENGTH//8]
+                            for resource in self.incoming_resources:
+                                if resource_hash == resource.hash:
+                                    resource.cancel()
 
                     elif packet.context == RNS.Packet.KEEPALIVE:
                         if not self.initiator and packet.data == bytes([0xFF]):
                             keepalive_packet = RNS.Packet(self, bytes([0xFE]), context=RNS.Packet.KEEPALIVE)
                             keepalive_packet.send()
-                            self.had_outbound()
+                            self.had_outbound(is_keepalive = True)
 
 
                     # TODO: find the most efficient way to allow multiple
@@ -769,6 +941,17 @@ class Link:
                     elif packet.context == RNS.Packet.RESOURCE:
                         for resource in self.incoming_resources:
                             resource.receive_part(packet)
+                            self.__update_phy_stats(packet)
+
+                    elif packet.context == RNS.Packet.CHANNEL:
+                        if not self._channel:
+                            RNS.log(f"Channel data received without open channel", RNS.LOG_DEBUG)
+                        else:
+                            packet.prove()
+                            plaintext = self.decrypt(packet.data)
+                            if plaintext != None:
+                                self.__update_phy_stats(packet)
+                                self._channel._receive(plaintext)
 
                 elif packet.packet_type == RNS.Packet.PROOF:
                     if packet.context == RNS.Packet.RESOURCE_PRF:
@@ -776,6 +959,7 @@ class Link:
                         for resource in self.outgoing_resources:
                             if resource_hash == resource.hash:
                                 resource.validate_proof(packet.data)
+                                self.__update_phy_stats(packet, query_shared=True)
 
         self.watchdog_lock = False
 
@@ -786,7 +970,7 @@ class Link:
                 try:
                     self.fernet = Fernet(self.derived_key)
                 except Exception as e:
-                    RNS.log("Could not "+str(self)+" instantiate Fernet while performin encryption on link. The contained exception was: "+str(e), RNS.LOG_ERROR)
+                    RNS.log("Could not instantiate Fernet while performin encryption on link "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
                     raise e
 
             return self.fernet.encrypt(plaintext)
@@ -805,6 +989,7 @@ class Link:
 
         except Exception as e:
             RNS.log("Decryption failed on link "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+            return None
 
 
     def sign(self, message):
@@ -900,6 +1085,13 @@ class Link:
     def register_incoming_resource(self, resource):
         self.incoming_resources.append(resource)
 
+    def has_incoming_resource(self, resource):
+        for incoming_resource in self.incoming_resources:
+            if incoming_resource.hash == resource.hash:
+                return True
+
+        return False
+
     def cancel_outgoing_resource(self, resource):
         if resource in self.outgoing_resources:
             self.outgoing_resources.remove(resource)
@@ -978,7 +1170,8 @@ class RequestReceipt():
     def request_resource_concluded(self, resource):
         if resource.status == RNS.Resource.COMPLETE:
             RNS.log("Request "+RNS.prettyhexrep(self.request_id)+" successfully sent as resource.", RNS.LOG_DEBUG)
-            self.started_at = time.time()
+            if self.started_at == None:
+                self.started_at = time.time()
             self.status = RequestReceipt.DELIVERED
             self.__resource_response_timeout = time.time()+self.timeout
             response_timeout_thread = threading.Thread(target=self.__response_timeout_job)
@@ -1019,24 +1212,26 @@ class RequestReceipt():
 
 
     def response_resource_progress(self, resource):
-        if not self.status == RequestReceipt.FAILED:
-            self.status = RequestReceipt.RECEIVING
-            if self.packet_receipt != None:
-                self.packet_receipt.status = RNS.PacketReceipt.DELIVERED
-                self.packet_receipt.proved = True
-                self.packet_receipt.concluded_at = time.time()
-                if self.packet_receipt.callbacks.delivery != None:
-                    self.packet_receipt.callbacks.delivery(self.packet_receipt)
+        if resource != None:
+            if not self.status == RequestReceipt.FAILED:
+                self.status = RequestReceipt.RECEIVING
+                if self.packet_receipt != None:
+                    if self.packet_receipt.status != RNS.PacketReceipt.DELIVERED:
+                        self.packet_receipt.status = RNS.PacketReceipt.DELIVERED
+                        self.packet_receipt.proved = True
+                        self.packet_receipt.concluded_at = time.time()
+                        if self.packet_receipt.callbacks.delivery != None:
+                            self.packet_receipt.callbacks.delivery(self.packet_receipt)
 
-            self.progress = resource.get_progress()
-            
-            if self.callbacks.progress != None:
-                try:
-                    self.callbacks.progress(self)
-                except Exception as e:
-                    RNS.log("Error while executing response progress callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
-        else:
-            resource.cancel()
+                self.progress = resource.get_progress()
+                
+                if self.callbacks.progress != None:
+                    try:
+                        self.callbacks.progress(self)
+                    except Exception as e:
+                        RNS.log("Error while executing response progress callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+            else:
+                resource.cancel()
 
     
     def response_received(self, response):
@@ -1101,6 +1296,17 @@ class RequestReceipt():
         else:
             return None
 
+    def concluded(self):
+        """
+        :returns: True if the associated request has concluded (successfully or with a failure), otherwise False.
+        """
+        if self.status == RequestReceipt.READY:
+            return True
+        elif self.status == RequestReceipt.FAILED:
+            return True
+        else:
+            return False
+
 
 
 class RequestReceiptCallbacks:

RNS/Packet.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2024 Mark Qvist / unsigned.io and contributors.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -30,7 +30,7 @@ class Packet:
     """
     The Packet class is used to create packet instances that can be sent
     over a Reticulum network. Packets will automatically be encrypted if
-    they are adressed to a ``RNS.Destination.SINGLE`` destination,
+    they are addressed to a ``RNS.Destination.SINGLE`` destination,
     ``RNS.Destination.GROUP`` destination or a :ref:`RNS.Link<api-link>`.
 
     For ``RNS.Destination.GROUP`` destinations, Reticulum will use the
@@ -58,9 +58,7 @@ class Packet:
     # Header types
     HEADER_1     = 0x00     # Normal header format
     HEADER_2     = 0x01     # Header format used for packets in transport
-    HEADER_3     = 0x02     # Reserved
-    HEADER_4     = 0x03     # Reserved
-    header_types = [HEADER_1, HEADER_2, HEADER_3, HEADER_4]
+    header_types = [HEADER_1, HEADER_2]
 
     # Packet context types
     NONE           = 0x00   # Generic data packet
@@ -77,6 +75,7 @@ class Packet:
     PATH_RESPONSE  = 0x0B   # Packet is a response to a path request
     COMMAND        = 0x0C   # Packet is a command
     COMMAND_STATUS = 0x0D   # Packet is a status of an executed command
+    CHANNEL        = 0x0E   # Packet contains link channel data
     KEEPALIVE      = 0xFA   # Packet is a keepalive packet
     LINKIDENTIFY   = 0xFB   # Packet is a link peer identification proof
     LINKCLOSE      = 0xFC   # Packet is a link close message
@@ -84,6 +83,10 @@ class Packet:
     LRRTT          = 0xFE   # Packet is a link request round-trip time measurement
     LRPROOF        = 0xFF   # Packet is a link request proof
 
+    # Context flag values
+    FLAG_SET       = 0x01
+    FLAG_UNSET     = 0x00
+
     # This is used to calculate allowable
     # payload sizes
     HEADER_MAXSIZE = RNS.Reticulum.HEADER_MAXSIZE
@@ -103,7 +106,9 @@ class Packet:
 
     TIMEOUT_PER_HOP = RNS.Reticulum.DEFAULT_PER_HOP_TIMEOUT
 
-    def __init__(self, destination, data, packet_type = DATA, context = NONE, transport_type = RNS.Transport.BROADCAST, header_type = HEADER_1, transport_id = None, attached_interface = None, create_receipt = True):
+    def __init__(self, destination, data, packet_type = DATA, context = NONE, transport_type = RNS.Transport.BROADCAST,
+                 header_type = HEADER_1, transport_id = None, attached_interface = None, create_receipt = True, context_flag=FLAG_UNSET):
+
         if destination != None:
             if transport_type == None:
                 transport_type = RNS.Transport.BROADCAST
@@ -112,6 +117,7 @@ class Packet:
             self.packet_type    = packet_type
             self.transport_type = transport_type
             self.context        = context
+            self.context_flag   = context_flag
 
             self.hops           = 0;
             self.destination    = destination
@@ -134,17 +140,20 @@ class Packet:
         self.MTU         = RNS.Reticulum.MTU
         self.sent_at     = None
         self.packet_hash = None
+        self.ratchet_id  = None
 
         self.attached_interface = attached_interface
         self.receiving_interface = None
         self.rssi = None
         self.snr = None
+        self.q = None
 
     def get_packed_flags(self):
         if self.context == Packet.LRPROOF:
-            packed_flags = (self.header_type << 6) | (self.transport_type << 4) | RNS.Destination.LINK | self.packet_type
+            packed_flags = (self.header_type << 6) | (self.context_flag << 5) | (self.transport_type << 4) | (RNS.Destination.LINK << 2) | self.packet_type
         else:
-            packed_flags = (self.header_type << 6) | (self.transport_type << 4) | (self.destination.type << 2) | self.packet_type
+            packed_flags = (self.header_type << 6) | (self.context_flag << 5) | (self.transport_type << 4) | (self.destination.type << 2) | self.packet_type
+
         return packed_flags
 
     def pack(self):
@@ -173,8 +182,8 @@ class Packet:
                     # Packet proofs over links are not encrypted
                     self.ciphertext = self.data
                 elif self.context == Packet.RESOURCE:
-                    # A resource takes care of symmetric
-                    # encryption by itself
+                    # A resource takes care of encryption
+                    # by itself
                     self.ciphertext = self.data
                 elif self.context == Packet.KEEPALIVE:
                     # Keepalive packets contain no actual
@@ -187,6 +196,8 @@ class Packet:
                     # In all other cases, we encrypt the packet
                     # with the destination's encryption method
                     self.ciphertext = self.destination.encrypt(self.data)
+                    if hasattr(self.destination, "latest_ratchet_id"):
+                        self.ratchet_id = self.destination.latest_ratchet_id
 
             if self.header_type == Packet.HEADER_2:
                 if self.transport_id != None:
@@ -215,8 +226,9 @@ class Packet:
             self.flags = self.raw[0]
             self.hops  = self.raw[1]
 
-            self.header_type      = (self.flags & 0b11000000) >> 6
-            self.transport_type   = (self.flags & 0b00110000) >> 4
+            self.header_type      = (self.flags & 0b01000000) >> 6
+            self.context_flag     = (self.flags & 0b00100000) >> 5
+            self.transport_type   = (self.flags & 0b00010000) >> 4
             self.destination_type = (self.flags & 0b00001100) >> 2
             self.packet_type      = (self.flags & 0b00000011)
 
@@ -277,6 +289,10 @@ class Packet:
         :returns: A :ref:`RNS.PacketReceipt<api-packetreceipt>` instance if *create_receipt* was set to *True* when the packet was instantiated, if not returns *None*. If the packet could not be sent *False* is returned.
         """
         if self.sent:
+            # Re-pack the packet to obtain new ciphertext for
+            # encrypted destinations
+            self.pack()
+            
             if RNS.Transport.outbound(self):
                 return self.receipt
             else:
@@ -365,10 +381,10 @@ class PacketReceipt:
         self.proof_packet   = None
 
         if packet.destination.type == RNS.Destination.LINK:
-            self.timeout    = packet.destination.rtt * packet.destination.traffic_timeout_factor
+            self.timeout    = max(packet.destination.rtt * packet.destination.traffic_timeout_factor, RNS.Link.TRAFFIC_TIMEOUT_MIN_MS/1000)
         else:
-            self.timeout    = Packet.TIMEOUT_PER_HOP * RNS.Transport.hops_to(self.destination.hash)
-
+            self.timeout    = RNS.Reticulum.get_instance().get_first_hop_timeout(self.destination.hash)
+            self.timeout   += Packet.TIMEOUT_PER_HOP * RNS.Transport.hops_to(self.destination.hash)
 
     def get_status(self):
         """
@@ -397,9 +413,16 @@ class PacketReceipt:
                     self.proved = True
                     self.concluded_at = time.time()
                     self.proof_packet = proof_packet
+                    link.last_proof = self.concluded_at
 
                     if self.callbacks.delivery != None:
-                        self.callbacks.delivery(self)
+                        try:
+                            self.callbacks.delivery(self)
+                        except Exception as e:
+                            RNS.log("An error occurred while evaluating external delivery callback for "+str(link), RNS.LOG_ERROR)
+                            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            RNS.trace_exception(e)
+                            
                     return True
                 else:
                     return False
@@ -430,7 +453,7 @@ class PacketReceipt:
             # This is an explicit proof
             proof_hash = proof[:RNS.Identity.HASHLENGTH//8]
             signature = proof[RNS.Identity.HASHLENGTH//8:RNS.Identity.HASHLENGTH//8+RNS.Identity.SIGLENGTH//8]
-            if proof_hash == self.hash:
+            if proof_hash == self.hash and hasattr(self.destination, "identity") and self.destination.identity != None:
                 proof_valid = self.destination.identity.validate(signature, self.hash)
                 if proof_valid:
                     self.status = PacketReceipt.DELIVERED

RNS/Resolver.py

@@ -0,0 +1,27 @@
+# MIT License
+#
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io and contributors.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+class Resolver:
+  
+  @staticmethod
+  def resolve_identity(full_name):
+    pass

RNS/Resource.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io and contributors.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -25,7 +25,9 @@ import os
 import bz2
 import math
 import time
+import tempfile
 import threading
+from threading import Lock
 from .vendor import umsgpack as umsgpack
 from time import sleep
 
@@ -47,11 +49,14 @@ class Resource:
     WINDOW               = 4
 
     # Absolute minimum window size during transfer
-    WINDOW_MIN           = 1
+    WINDOW_MIN           = 2
 
     # The maximum window size for transfers on slow links
     WINDOW_MAX_SLOW      = 10
 
+    # The maximum window size for transfers on very slow links
+    WINDOW_MAX_VERY_SLOW = 4
+
     # The maximum window size for transfers on fast links
     WINDOW_MAX_FAST      = 75
     
@@ -63,12 +68,22 @@ class Resource:
     # rounds, the fast link window size will be allowed.
     FAST_RATE_THRESHOLD  = WINDOW_MAX_SLOW - WINDOW - 2
 
+    # If the very slow rate is sustained for this many request
+    # rounds, window will be capped to the very slow limit.
+    VERY_SLOW_RATE_THRESHOLD = 2
+
     # If the RTT rate is higher than this value,
     # the max window size for fast links will be used.
     # The default is 50 Kbps (the value is stored in
     # bytes per second, hence the "/ 8").
     RATE_FAST            = (50*1000) / 8
 
+    # If the RTT rate is lower than this value,
+    # the window size will be capped at .
+    # The default is 50 Kbps (the value is stored in
+    # bytes per second, hence the "/ 8").
+    RATE_VERY_SLOW       = (2*1000) / 8
+
     # The minimum allowed flexibility of the window size.
     # The difference between window_max and window_min
     # will never be smaller than this value.
@@ -103,9 +118,10 @@ class Resource:
 
     PART_TIMEOUT_FACTOR           = 4
     PART_TIMEOUT_FACTOR_AFTER_RTT = 2
-    MAX_RETRIES                   = 8
+    MAX_RETRIES                   = 16
     MAX_ADV_RETRIES               = 4
-    SENDER_GRACE_TIME             = 10
+    SENDER_GRACE_TIME             = 10.0
+    PROCESSING_GRACE              = 1.0
     RETRY_GRACE_TIME              = 0.25
     PER_RETRY_DELAY               = 0.5
 
@@ -149,7 +165,7 @@ class Resource:
             resource.total_parts         = int(math.ceil(resource.size/float(Resource.SDU)))
             resource.received_count      = 0
             resource.outstanding_parts   = 0
-            resource.parts                 = [None] * resource.total_parts
+            resource.parts               = [None] * resource.total_parts
             resource.window              = Resource.WINDOW
             resource.window_max          = Resource.WINDOW_MAX_SLOW
             resource.window_min          = Resource.WINDOW_MIN
@@ -167,25 +183,29 @@ class Resource:
             resource.hashmap = [None] * resource.total_parts
             resource.hashmap_height = 0
             resource.waiting_for_hmu = False
-
             resource.receiving_part = False
-
-            resource.consecutive_completed_height = 0
+            resource.consecutive_completed_height = -1
             
-            resource.link.register_incoming_resource(resource)
+            if not resource.link.has_incoming_resource(resource):
+                resource.link.register_incoming_resource(resource)
 
-            RNS.log("Accepting resource advertisement for "+RNS.prettyhexrep(resource.hash), RNS.LOG_DEBUG)
-            if resource.link.callbacks.resource_started != None:
-                try:
-                    resource.link.callbacks.resource_started(resource)
-                except Exception as e:
-                    RNS.log("Error while executing resource started callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+                RNS.log(f"Accepting resource advertisement for {RNS.prettyhexrep(resource.hash)}. Transfer size is {RNS.prettysize(resource.size)} in {resource.total_parts} parts.", RNS.LOG_DEBUG)
+                if resource.link.callbacks.resource_started != None:
+                    try:
+                        resource.link.callbacks.resource_started(resource)
+                    except Exception as e:
+                        RNS.log("Error while executing resource started callback from "+str(resource)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
 
-            resource.hashmap_update(0, resource.hashmap_raw)
+                resource.hashmap_update(0, resource.hashmap_raw)
 
-            resource.watchdog_job()
+                resource.watchdog_job()
+
+                return resource
+
+            else:
+                RNS.log("Ignoring resource advertisement for "+RNS.prettyhexrep(resource.hash)+", resource already transferring", RNS.LOG_DEBUG)
+                return None
 
-            return resource
         except Exception as e:
             RNS.log("Could not decode resource advertisement, dropping resource", RNS.LOG_DEBUG)
             return None
@@ -197,10 +217,22 @@ class Resource:
         data_size = None
         resource_data = None
         self.assembly_lock = False
+        self.preparing_next_segment = False
+        self.next_segment = None
+
+        if data != None:
+            if not hasattr(data, "read") and len(data) > Resource.MAX_EFFICIENT_SIZE:
+                original_data = data
+                data_size = len(original_data)
+                data = tempfile.TemporaryFile()
+                data.write(original_data)
+                del original_data
 
         if hasattr(data, "read"):
-            data_size = os.stat(data.name).st_size
-            self.total_size  = data_size
+            if data_size == None:
+                data_size = os.stat(data.name).st_size
+
+            self.total_size = data_size
             self.grand_total_parts = math.ceil(data_size/Resource.SDU)
 
             if data_size <= Resource.MAX_EFFICIENT_SIZE:
@@ -256,6 +288,7 @@ class Resource:
         self.req_resp_rtt_rate = 0
         self.rtt_rxd_bytes_at_part_req = 0
         self.fast_rate_rounds = 0
+        self.very_slow_rate_rounds = 0
         self.request_id = request_id
         self.is_response = is_response
 
@@ -273,7 +306,7 @@ class Resource:
             self.uncompressed_data = data
 
             compression_began = time.time()
-            if (auto_compress and len(self.uncompressed_data) < Resource.AUTO_COMPRESS_MAX_SIZE):
+            if (auto_compress and len(self.uncompressed_data) <= Resource.AUTO_COMPRESS_MAX_SIZE):
                 RNS.log("Compressing resource data...", RNS.LOG_DEBUG)
                 self.compressed_data   = bz2.compress(self.uncompressed_data)
                 RNS.log("Compression completed in "+str(round(time.time()-compression_began, 3))+" seconds", RNS.LOG_DEBUG)
@@ -360,7 +393,8 @@ class Resource:
             if advertise:
                 self.advertise()
         else:
-            pass
+            self.receive_lock = Lock()
+            
 
     def hashmap_update_packet(self, plaintext):
         if not self.status == Resource.FAILED:
@@ -392,13 +426,15 @@ class Resource:
         Advertise the resource. If the other end of the link accepts
         the resource advertisement it will begin transferring.
         """
-        thread = threading.Thread(target=self.__advertise_job)
-        thread.daemon = True
+        thread = threading.Thread(target=self.__advertise_job, daemon=True)
         thread.start()
 
+        if self.segment_index < self.total_segments:
+            prepare_thread = threading.Thread(target=self.__prepare_next_segment, daemon=True)
+            prepare_thread.start()
+
     def __advertise_job(self):
-        data = ResourceAdvertisement(self).pack()
-        self.advertisement_packet = RNS.Packet(self.link, data, context=RNS.Packet.RESOURCE_ADV)
+        self.advertisement_packet = RNS.Packet(self.link, ResourceAdvertisement(self).pack(), context=RNS.Packet.RESOURCE_ADV)
         while not self.link.ready_for_new_resource():
             self.status = Resource.QUEUED
             sleep(0.25)
@@ -435,7 +471,7 @@ class Resource:
             sleep_time = None
 
             if self.status == Resource.ADVERTISED:
-                sleep_time = (self.adv_sent+self.timeout)-time.time()
+                sleep_time = (self.adv_sent+self.timeout+Resource.PROCESSING_GRACE)-time.time()
                 if sleep_time < 0:
                     if self.retries_left <= 0:
                         RNS.log("Resource transfer timeout after sending advertisement", RNS.LOG_DEBUG)
@@ -445,12 +481,13 @@ class Resource:
                         try:
                             RNS.log("No part requests received, retrying resource advertisement...", RNS.LOG_DEBUG)
                             self.retries_left -= 1
-                            self.advertisement_packet.resend()
+                            self.advertisement_packet = RNS.Packet(self.link, ResourceAdvertisement(self).pack(), context=RNS.Packet.RESOURCE_ADV)
+                            self.advertisement_packet.send()
                             self.last_activity = time.time()
                             self.adv_sent = self.last_activity
                             sleep_time = 0.001
                         except Exception as e:
-                            RNS.log("Could not resend advertisement packet, cancelling resource", RNS.LOG_VERBOSE)
+                            RNS.log("Could not resend advertisement packet, cancelling resource. The contained exception was: "+str(e), RNS.LOG_VERBOSE)
                             self.cancel()
                     
 
@@ -470,7 +507,8 @@ class Resource:
                     
                     if sleep_time < 0:
                         if self.retries_left > 0:
-                            RNS.log("Timed out waiting for parts, requesting retry", RNS.LOG_DEBUG)
+                            ms = "" if self.outstanding_parts == 1 else "s"
+                            RNS.log("Timed out waiting for "+str(self.outstanding_parts)+" part"+ms+", requesting retry", RNS.LOG_DEBUG)
                             if self.window > self.window_min:
                                 self.window -= 1
                                 if self.window_max > self.window_min:
@@ -591,6 +629,21 @@ class Resource:
                 RNS.log("The contained exception was: "+str(e), RNS.LOG_DEBUG)
                 self.cancel()
 
+    def __prepare_next_segment(self):
+        # Prepare the next segment for advertisement
+        RNS.log(f"Preparing segment {self.segment_index+1} of {self.total_segments} for resource {self}", RNS.LOG_DEBUG)
+        self.preparing_next_segment = True
+        self.next_segment = Resource(
+            self.input_file, self.link,
+            callback = self.callback,
+            segment_index = self.segment_index+1,
+            original_hash=self.original_hash,
+            progress_callback = self.__progress_callback,
+            request_id = self.request_id,
+            is_response = self.is_response,
+            advertise = False,
+        )
+
     def validate_proof(self, proof_data):
         if not self.status == Resource.FAILED:
             if len(proof_data) == RNS.Identity.HASHLENGTH//8*2:
@@ -605,10 +658,24 @@ class Resource:
                                 self.callback(self)
                             except Exception as e:
                                 RNS.log("Error while executing resource concluded callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            finally:
+                                try:
+                                    if hasattr(self, "input_file"):
+                                        if hasattr(self.input_file, "close") and callable(self.input_file.close):
+                                            self.input_file.close()
+
+                                except Exception as e:
+                                    RNS.log("Error while closing resource input file: "+str(e), RNS.LOG_ERROR)
                     else:
                         # Otherwise we'll recursively create the
                         # next segment of the resource
-                        Resource(self.input_file, self.link, callback = self.callback, segment_index = self.segment_index+1, original_hash=self.original_hash, progress_callback = self.__progress_callback)
+                        if not self.preparing_next_segment:
+                            RNS.log(f"Next segment preparation for resource {self} was not started yet, manually preparing now. This will cause transfer slowdown.", RNS.LOG_WARNING)
+                            self.__prepare_next_segment()
+
+                        while self.next_segment == None:
+                            time.sleep(0.05)
+                        self.next_segment.advertise()
                 else:
                     pass
             else:
@@ -616,98 +683,105 @@ class Resource:
 
 
     def receive_part(self, packet):
-        while self.receiving_part:
-            sleep(0.001)
+        with self.receive_lock:
 
-        self.receiving_part = True
-        self.last_activity = time.time()
-        self.retries_left = self.max_retries
+            self.receiving_part = True
+            self.last_activity = time.time()
+            self.retries_left = self.max_retries
 
-        if self.req_resp == None:
-            self.req_resp = self.last_activity
-            rtt = self.req_resp-self.req_sent
-            
-            self.part_timeout_factor = Resource.PART_TIMEOUT_FACTOR_AFTER_RTT
-            if self.rtt == None:
-                self.rtt = self.link.rtt
-                self.watchdog_job()
-            elif rtt < self.rtt:
-                self.rtt = max(self.rtt - self.rtt*0.05, rtt)
-            elif rtt > self.rtt:
-                self.rtt = min(self.rtt + self.rtt*0.05, rtt)
+            if self.req_resp == None:
+                self.req_resp = self.last_activity
+                rtt = self.req_resp-self.req_sent
+                
+                self.part_timeout_factor = Resource.PART_TIMEOUT_FACTOR_AFTER_RTT
+                if self.rtt == None:
+                    self.rtt = self.link.rtt
+                    self.watchdog_job()
+                elif rtt < self.rtt:
+                    self.rtt = max(self.rtt - self.rtt*0.05, rtt)
+                elif rtt > self.rtt:
+                    self.rtt = min(self.rtt + self.rtt*0.05, rtt)
 
-            if rtt > 0:
-                req_resp_cost = len(packet.raw)+self.req_sent_bytes
-                self.req_resp_rtt_rate = req_resp_cost / rtt
+                if rtt > 0:
+                    req_resp_cost = len(packet.raw)+self.req_sent_bytes
+                    self.req_resp_rtt_rate = req_resp_cost / rtt
 
-                if self.req_resp_rtt_rate > Resource.RATE_FAST and self.fast_rate_rounds < Resource.FAST_RATE_THRESHOLD:
-                    self.fast_rate_rounds += 1
+                    if self.req_resp_rtt_rate > Resource.RATE_FAST and self.fast_rate_rounds < Resource.FAST_RATE_THRESHOLD:
+                        self.fast_rate_rounds += 1
 
-                    if self.fast_rate_rounds == Resource.FAST_RATE_THRESHOLD:
-                        self.window_max = Resource.WINDOW_MAX_FAST
+                        if self.fast_rate_rounds == Resource.FAST_RATE_THRESHOLD:
+                            self.window_max = Resource.WINDOW_MAX_FAST
 
-        if not self.status == Resource.FAILED:
-            self.status = Resource.TRANSFERRING
-            part_data = packet.data
-            part_hash = self.get_map_hash(part_data)
+            if not self.status == Resource.FAILED:
+                self.status = Resource.TRANSFERRING
+                part_data = packet.data
+                part_hash = self.get_map_hash(part_data)
 
-            i = self.consecutive_completed_height
-            for map_hash in self.hashmap[self.consecutive_completed_height:self.consecutive_completed_height+self.window]:
-                if map_hash == part_hash:
-                    if self.parts[i] == None:
-                        # Insert data into parts list
-                        self.parts[i] = part_data
-                        self.rtt_rxd_bytes += len(part_data)
-                        self.received_count += 1
-                        self.outstanding_parts -= 1
+                consecutive_index = self.consecutive_completed_height if self.consecutive_completed_height >= 0 else 0
+                i = consecutive_index
+                for map_hash in self.hashmap[consecutive_index:consecutive_index+self.window]:
+                    if map_hash == part_hash:
+                        if self.parts[i] == None:
 
-                        # Update consecutive completed pointer
-                        if i == self.consecutive_completed_height + 1:
-                            self.consecutive_completed_height = i
-                        
-                        cp = self.consecutive_completed_height + 1
-                        while cp < len(self.parts) and self.parts[cp] != None:
-                            self.consecutive_completed_height = cp
-                            cp += 1
+                            # Insert data into parts list
+                            self.parts[i] = part_data
+                            self.rtt_rxd_bytes += len(part_data)
+                            self.received_count += 1
+                            self.outstanding_parts -= 1
 
-                        if self.__progress_callback != None:
-                            try:
-                                self.__progress_callback(self)
-                            except Exception as e:
-                                RNS.log("Error while executing progress callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            # Update consecutive completed pointer
+                            if i == self.consecutive_completed_height + 1:
+                                self.consecutive_completed_height = i
+                            
+                            cp = self.consecutive_completed_height + 1
+                            while cp < len(self.parts) and self.parts[cp] != None:
+                                self.consecutive_completed_height = cp
+                                cp += 1
 
-                i += 1
+                            if self.__progress_callback != None:
+                                try:
+                                    self.__progress_callback(self)
+                                except Exception as e:
+                                    RNS.log("Error while executing progress callback from "+str(self)+". The contained exception was: "+str(e), RNS.LOG_ERROR)
 
-            self.receiving_part = False
+                    i += 1
 
-            if self.received_count == self.total_parts and not self.assembly_lock:
-                self.assembly_lock = True
-                self.assemble()
-            elif self.outstanding_parts == 0:
-                # TODO: Figure out if there is a mathematically
-                # optimal way to adjust windows
-                if self.window < self.window_max:
-                    self.window += 1
-                    if (self.window - self.window_min) > (self.window_flexibility-1):
-                        self.window_min += 1
+                self.receiving_part = False
 
-                if self.req_sent != 0:
-                    rtt = time.time()-self.req_sent
-                    req_transferred = self.rtt_rxd_bytes - self.rtt_rxd_bytes_at_part_req
+                if self.received_count == self.total_parts and not self.assembly_lock:
+                    self.assembly_lock = True
+                    self.assemble()
+                elif self.outstanding_parts == 0:
+                    # TODO: Figure out if there is a mathematically
+                    # optimal way to adjust windows
+                    if self.window < self.window_max:
+                        self.window += 1
+                        if (self.window - self.window_min) > (self.window_flexibility-1):
+                            self.window_min += 1
 
-                    if rtt != 0:
-                        self.req_data_rtt_rate = req_transferred/rtt
-                        self.rtt_rxd_bytes_at_part_req = self.rtt_rxd_bytes
+                    if self.req_sent != 0:
+                        rtt = time.time()-self.req_sent
+                        req_transferred = self.rtt_rxd_bytes - self.rtt_rxd_bytes_at_part_req
 
-                        if self.req_data_rtt_rate > Resource.RATE_FAST and self.fast_rate_rounds < Resource.FAST_RATE_THRESHOLD:
-                            self.fast_rate_rounds += 1
+                        if rtt != 0:
+                            self.req_data_rtt_rate = req_transferred/rtt
+                            self.rtt_rxd_bytes_at_part_req = self.rtt_rxd_bytes
 
-                            if self.fast_rate_rounds == Resource.FAST_RATE_THRESHOLD:
-                                self.window_max = Resource.WINDOW_MAX_FAST
+                            if self.req_data_rtt_rate > Resource.RATE_FAST and self.fast_rate_rounds < Resource.FAST_RATE_THRESHOLD:
+                                self.fast_rate_rounds += 1
 
-                self.request_next()
-        else:
-            self.receiving_part = False
+                                if self.fast_rate_rounds == Resource.FAST_RATE_THRESHOLD:
+                                    self.window_max = Resource.WINDOW_MAX_FAST
+
+                            if self.fast_rate_rounds == 0 and self.req_data_rtt_rate < Resource.RATE_VERY_SLOW and self.very_slow_rate_rounds < Resource.VERY_SLOW_RATE_THRESHOLD:
+                                self.very_slow_rate_rounds += 1
+
+                                if self.very_slow_rate_rounds == Resource.VERY_SLOW_RATE_THRESHOLD:
+                                    self.window_max = Resource.WINDOW_MAX_VERY_SLOW
+
+                    self.request_next()
+            else:
+                self.receiving_part = False
 
     # Called on incoming resource to send a request for more data
     def request_next(self):
@@ -720,11 +794,11 @@ class Resource:
                 hashmap_exhausted = Resource.HASHMAP_IS_NOT_EXHAUSTED
                 requested_hashes = b""
 
-                offset = (1 if self.consecutive_completed_height > 0 else 0)
-                i = 0; pn = self.consecutive_completed_height+offset
+                i = 0; pn = self.consecutive_completed_height+1
                 search_start = pn
-
-                for part in self.parts[search_start:search_start+self.window]:
+                search_size = self.window
+                
+                for part in self.parts[search_start:search_start+search_size]:
                     if part == None:
                         part_hash = self.hashmap[pn]
                         if part_hash != None:
@@ -744,7 +818,6 @@ class Resource:
                     hmu_part += last_map_hash
                     self.waiting_for_hmu = True
 
-                requested_data = b""
                 request_data = hmu_part + self.hash + requested_hashes
                 request_packet = RNS.Packet(self.link, request_data, context = RNS.Packet.RESOURCE_REQ)
 
@@ -754,6 +827,7 @@ class Resource:
                     self.req_sent = self.last_activity
                     self.req_sent_bytes = len(request_packet.raw)
                     self.req_resp = None
+
                 except Exception as e:
                     RNS.log("Could not send resource request packet, cancelling resource", RNS.LOG_DEBUG)
                     RNS.log("The contained exception was: "+str(e), RNS.LOG_DEBUG)
@@ -887,7 +961,9 @@ class Resource:
         """
         :returns: The current progress of the resource transfer as a *float* between 0.0 and 1.0.
         """
-        if self.initiator:
+        if self.status == RNS.Resource.COMPLETE and self.segment_index == self.total_segments:
+            return 1.0
+        elif self.initiator:
             self.processed_parts  = (self.segment_index-1)*math.ceil(Resource.MAX_EFFICIENT_SIZE/Resource.SDU)
             self.processed_parts += self.sent_parts
             self.progress_total_parts = float(self.grand_total_parts)
@@ -899,8 +975,7 @@ class Resource:
             else:
                 self.progress_total_parts = float(self.total_parts)
 
-        
-        progress = self.processed_parts / self.progress_total_parts
+        progress = min(1.0, self.processed_parts / self.progress_total_parts)
         return progress
 
     def get_transfer_size(self):
@@ -988,6 +1063,7 @@ class ResourceAdvertisement:
 
 
     def __init__(self, resource=None, request_id=None, is_response=False):
+        self.link = None
         if resource != None:
             self.t = resource.size              # Transfer size
             self.d = resource.total_size        # Total uncompressed data size
@@ -1034,6 +1110,9 @@ class ResourceAdvertisement:
     def is_compressed(self):
         return self.c
 
+    def get_link(self):
+        return self.link
+
     def pack(self, segment=0):
         hashmap_start = segment*ResourceAdvertisement.HASHMAP_MAX_LEN
         hashmap_end   = min((segment+1)*(ResourceAdvertisement.HASHMAP_MAX_LEN), self.n)

RNS/Reticulum.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2024 Mark Qvist / unsigned.io and contributors.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -29,6 +29,9 @@ if get_platform() == "android":
     from .Interfaces import TCPInterface
     from .Interfaces import UDPInterface
     from .Interfaces import I2PInterface
+    from .Interfaces.Android import RNodeInterface
+    from .Interfaces.Android import SerialInterface
+    from .Interfaces.Android import KISSInterface
 else:
     from .Interfaces import *
 
@@ -76,7 +79,7 @@ class Reticulum:
     MTU            = 500
     """
     The MTU that Reticulum adheres to, and will expect other peers to
-    adhere to. By default, the MTU is 507 bytes. In custom RNS network
+    adhere to. By default, the MTU is 500 bytes. In custom RNS network
     implementations, it is possible to change this value, but doing so will
     completely break compatibility with all other RNS networks. An identical
     MTU is a prerequisite for peers to communicate in the same network.
@@ -103,18 +106,20 @@ class Reticulum:
     it will eventually be dropped.
 
     This value will be applied by default to all created interfaces,
-    but it can be configured individually on a per-interface basis.
+    but it can be configured individually on a per-interface basis. In
+    general, the global default setting should not be changed, and any
+    alterations should be made on a per-interface basis instead.
     """
 
-    MINIMUM_BITRATE = 500
+    MINIMUM_BITRATE = 5
+    """
+    Minimum bitrate required across a medium for Reticulum to be able
+    to successfully establish links. Currently 5 bits per second.
+    """
 
-    # TODO: To reach the 300bps level without unreasonably impacting
-    # performance on faster links, we need a mechanism for setting
-    # this value more intelligently. One option could be inferring it
-    # from interface speed, but a better general approach would most
-    # probably be to let Reticulum somehow continously build a map of
-    # per-hop latencies and use this map for the timeout calculation. 
-    DEFAULT_PER_HOP_TIMEOUT = 5
+    # TODO: Let Reticulum somehow continously build a map of per-hop
+    # latencies and use this map for global timeout calculation.
+    DEFAULT_PER_HOP_TIMEOUT = 6
 
     # Length of truncated hashes in bits.
     TRUNCATED_HASHLENGTH = 128
@@ -130,6 +135,7 @@ class Reticulum:
     JOB_INTERVAL     = 5*60
     CLEAN_INTERVAL   = 15*60
     PERSIST_INTERVAL = 60*60*12
+    GRACIOUS_PERSIST_INTERVAL = 60*5
 
     router           = None
     config           = None
@@ -141,6 +147,8 @@ class Reticulum:
     configpath       = ""
     storagepath      = ""
     cachepath        = ""
+
+    __instance       = None
     
     @staticmethod
     def exit_handler():
@@ -152,6 +160,9 @@ class Reticulum:
         RNS.Transport.exit_handler()
         RNS.Identity.exit_handler()
 
+        if RNS.profiler_ran:
+            RNS.profiler_results()
+
     @staticmethod
     def sigint_handler(signal, frame):
         RNS.Transport.detach_interfaces()
@@ -164,7 +175,14 @@ class Reticulum:
         RNS.exit()
 
 
-    def __init__(self,configdir=None, loglevel=None, logdest=None):
+    @staticmethod
+    def get_instance():
+        """
+        Return the currently running Reticulum instance
+        """
+        return Reticulum.__instance
+
+    def __init__(self,configdir=None, loglevel=None, logdest=None, verbosity=None):
         """
         Initialises and starts a Reticulum instance. This must be
         done before any other operations, and Reticulum will not
@@ -173,6 +191,11 @@ class Reticulum:
         :param configdir: Full path to a Reticulum configuration directory.
         """
 
+        if Reticulum.__instance != None:
+            raise OSError("Attempt to reinitialise Reticulum, when it was already running")
+        else:
+            Reticulum.__instance = self
+
         RNS.vendor.platformutils.platform_checks()
 
         if configdir != None:
@@ -196,7 +219,9 @@ class Reticulum:
         Reticulum.identitypath  = Reticulum.configdir+"/storage/identities"
 
         Reticulum.__transport_enabled = False
+        Reticulum.__remote_management_enabled = False
         Reticulum.__use_implicit_proof = True
+        Reticulum.__allow_probes = False
 
         Reticulum.panic_on_interface_error = False
 
@@ -204,10 +229,12 @@ class Reticulum:
         self.local_control_port   = 37429
         self.share_instance       = True
         self.rpc_listener         = None
+        self.rpc_key              = None
 
         self.ifac_salt = Reticulum.IFAC_SALT
 
         self.requested_loglevel = loglevel
+        self.requested_verbosity = verbosity
         if self.requested_loglevel != None:
             if self.requested_loglevel > RNS.LOG_EXTREME:
                 self.requested_loglevel = RNS.LOG_EXTREME
@@ -256,7 +283,8 @@ class Reticulum:
         RNS.Transport.start(self)
 
         self.rpc_addr = ("127.0.0.1", self.local_control_port)
-        self.rpc_key  = RNS.Identity.full_hash(RNS.Transport.identity.get_private_key())
+        if self.rpc_key == None:
+            self.rpc_key  = RNS.Identity.full_hash(RNS.Transport.identity.get_private_key())
         
         if self.is_shared_instance:
             self.rpc_listener = multiprocessing.connection.Listener(self.rpc_addr, authkey=self.rpc_key)
@@ -270,6 +298,7 @@ class Reticulum:
 
     def __start_jobs(self):
         if self.jobs_thread == None:
+            RNS.Identity._clean_ratchets()
             self.jobs_thread = threading.Thread(target=self.__jobs)
             self.jobs_thread.daemon = True
             self.jobs_thread.start()
@@ -284,7 +313,6 @@ class Reticulum:
 
             if now > self.last_data_persist+Reticulum.PERSIST_INTERVAL:
                 self.__persist_data()
-                self.last_data_persist = time.time()
             
             time.sleep(Reticulum.JOB_INTERVAL)
 
@@ -296,6 +324,10 @@ class Reticulum:
                     self.local_interface_port
                 )
                 interface.OUT = True
+                if hasattr(Reticulum, "_force_shared_instance_bitrate"):
+                    interface.bitrate = Reticulum._force_shared_instance_bitrate
+                    interface._force_bitrate = Reticulum._force_shared_instance_bitrate
+                    RNS.log(f"Forcing shared instance bitrate of {RNS.prettyspeed(interface.bitrate)}", RNS.LOG_WARNING)
                 RNS.Transport.interfaces.append(interface)
                 
                 self.is_shared_instance = True
@@ -310,11 +342,17 @@ class Reticulum:
                         self.local_interface_port)
                     interface.target_port = self.local_interface_port
                     interface.OUT = True
+                    if hasattr(Reticulum, "_force_shared_instance_bitrate"):
+                        interface.bitrate = Reticulum._force_shared_instance_bitrate
+                        interface._force_bitrate = True
+                        RNS.log(f"Forcing shared instance bitrate of {RNS.prettyspeed(interface.bitrate)}", RNS.LOG_WARNING)
                     RNS.Transport.interfaces.append(interface)
                     self.is_shared_instance = False
                     self.is_standalone_instance = False
                     self.is_connected_to_shared_instance = True
                     Reticulum.__transport_enabled = False
+                    Reticulum.__remote_management_enabled = False
+                    Reticulum.__allow_probes = False
                     RNS.log("Connected to locally available Reticulum instance via: "+str(interface), RNS.LOG_DEBUG)
                 except Exception as e:
                     RNS.log("Local shared instance appears to be running, but it could not be connected", RNS.LOG_ERROR)
@@ -334,6 +372,8 @@ class Reticulum:
                 value = self.config["logging"][option]
                 if option == "loglevel" and self.requested_loglevel == None:
                     RNS.loglevel = int(value)
+                    if self.requested_verbosity != None:
+                        RNS.loglevel += self.requested_verbosity
                     if RNS.loglevel < 0:
                         RNS.loglevel = 0
                     if RNS.loglevel > 7:
@@ -351,10 +391,41 @@ class Reticulum:
                 if option == "instance_control_port":
                     value = int(self.config["reticulum"][option])
                     self.local_control_port = value
+                if option == "rpc_key":
+                    try:
+                        value = bytes.fromhex(self.config["reticulum"][option])
+                        self.rpc_key = value
+                    except Exception as e:
+                        RNS.log("Invalid shared instance RPC key specified, falling back to default key", RNS.LOG_ERROR)
+                        self.rpc_key = None
                 if option == "enable_transport":
                     v = self.config["reticulum"].as_bool(option)
                     if v == True:
                         Reticulum.__transport_enabled = True
+                if option == "enable_remote_management":
+                    v = self.config["reticulum"].as_bool(option)
+                    if v == True:
+                        Reticulum.__remote_management_enabled = True
+                if option == "remote_management_allowed":
+                    v = self.config["reticulum"].as_list(option)
+                    for hexhash in v:
+                        dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+                        if len(hexhash) != dest_len:
+                            raise ValueError("Identity hash length for remote management ACL "+str(hexhash)+" is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2))
+                        try:
+                            allowed_hash = bytes.fromhex(hexhash)
+                        except Exception as e:
+                            raise ValueError("Invalid identity hash for remote management ACL: "+str(hexhash))
+
+                        if not allowed_hash in RNS.Transport.remote_management_allowed:
+                            RNS.Transport.remote_management_allowed.append(allowed_hash)
+                if option == "respond_to_probes":
+                    v = self.config["reticulum"].as_bool(option)
+                    if v == True:
+                        Reticulum.__allow_probes = True
+                if option == "force_shared_instance_bitrate":
+                    v = self.config["reticulum"].as_int(option)
+                    Reticulum._force_shared_instance_bitrate = v
                 if option == "panic_on_interface_error":
                     v = self.config["reticulum"].as_bool(option)
                     if v == True:
@@ -435,6 +506,23 @@ class Reticulum:
                             if c["pass_phrase"] != "":
                                 ifac_netkey = c["pass_phrase"]
                                 
+                        ingress_control = True
+                        if "ingress_control" in c: ingress_control = c.as_bool("ingress_control")
+                        ic_max_held_announces = None
+                        if "ic_max_held_announces" in c: ic_max_held_announces = c.as_int("ic_max_held_announces")
+                        ic_burst_hold = None
+                        if "ic_burst_hold" in c: ic_burst_hold = c.as_float("ic_burst_hold")
+                        ic_burst_freq_new = None
+                        if "ic_burst_freq_new" in c: ic_burst_freq_new = c.as_float("ic_burst_freq_new")
+                        ic_burst_freq = None
+                        if "ic_burst_freq" in c: ic_burst_freq = c.as_float("ic_burst_freq")
+                        ic_new_time = None
+                        if "ic_new_time" in c: ic_new_time = c.as_float("ic_new_time")
+                        ic_burst_penalty = None
+                        if "ic_burst_penalty" in c: ic_burst_penalty = c.as_float("ic_burst_penalty")
+                        ic_held_release_interval = None
+                        if "ic_held_release_interval" in c: ic_held_release_interval = c.as_float("ic_held_release_interval")
+
                         configured_bitrate = None
                         if "bitrate" in c:
                             if c.as_int("bitrate") >= Reticulum.MINIMUM_BITRATE:
@@ -471,44 +559,40 @@ class Reticulum:
 
                             if (("interface_enabled" in c) and c.as_bool("interface_enabled") == True) or (("enabled" in c) and c.as_bool("enabled") == True):
                                 if c["type"] == "AutoInterface":
-                                    if not RNS.vendor.platformutils.is_windows():
-                                        group_id        = c["group_id"] if "group_id" in c else None
-                                        discovery_scope = c["discovery_scope"] if "discovery_scope" in c else None
-                                        discovery_port  = int(c["discovery_port"]) if "discovery_port" in c else None
-                                        data_port  = int(c["data_port"]) if "data_port" in c else None
-                                        allowed_interfaces = c.as_list("devices") if "devices" in c else None
-                                        ignored_interfaces = c.as_list("ignored_devices") if "ignored_devices" in c else None
+                                    group_id        = c["group_id"] if "group_id" in c else None
+                                    discovery_scope = c["discovery_scope"] if "discovery_scope" in c else None
+                                    discovery_port  = int(c["discovery_port"]) if "discovery_port" in c else None
+                                    multicast_address_type = c["multicast_address_type"] if "multicast_address_type" in c else None
+                                    data_port  = int(c["data_port"]) if "data_port" in c else None
+                                    allowed_interfaces = c.as_list("devices") if "devices" in c else None
+                                    ignored_interfaces = c.as_list("ignored_devices") if "ignored_devices" in c else None
 
-                                        interface = AutoInterface.AutoInterface(
-                                            RNS.Transport,
-                                            name,
-                                            group_id,
-                                            discovery_scope,
-                                            discovery_port,
-                                            data_port,
-                                            allowed_interfaces,
-                                            ignored_interfaces
-                                        )
-
-                                        if "outgoing" in c and c.as_bool("outgoing") == False:
-                                            interface.OUT = False
-                                        else:
-                                            interface.OUT = True
-
-                                        interface.mode = interface_mode
-
-                                        interface.announce_cap = announce_cap
-                                        if configured_bitrate:
-                                            interface.bitrate = configured_bitrate
-                                        if ifac_size != None:
-                                            interface.ifac_size = ifac_size
-                                        else:
-                                            interface.ifac_size = 16
+                                    interface = AutoInterface.AutoInterface(
+                                        RNS.Transport,
+                                        name,
+                                        group_id,
+                                        discovery_scope,
+                                        discovery_port,
+                                        multicast_address_type,
+                                        data_port,
+                                        allowed_interfaces,
+                                        ignored_interfaces
+                                    )
 
+                                    if "outgoing" in c and c.as_bool("outgoing") == False:
+                                        interface.OUT = False
                                     else:
-                                        RNS.log("AutoInterface is not currently supported on Windows, disabling interface.", RNS.LOG_ERROR);
-                                        RNS.log("Please remove this AutoInterface instance from your configuration file.", RNS.LOG_ERROR);
-                                        RNS.log("You will have to manually configure other interfaces for connectivity.", RNS.LOG_ERROR);
+                                        interface.OUT = True
+
+                                    interface.mode = interface_mode
+
+                                    interface.announce_cap = announce_cap
+                                    if configured_bitrate:
+                                        interface.bitrate = configured_bitrate
+                                    if ifac_size != None:
+                                        interface.ifac_size = ifac_size
+                                    else:
+                                        interface.ifac_size = 16
 
                                 if c["type"] == "UDPInterface":
                                     device       = c["device"] if "device" in c else None
@@ -628,12 +712,18 @@ class Reticulum:
                                     i2p_peers = c.as_list("peers") if "peers" in c else None
                                     connectable = c.as_bool("connectable") if "connectable" in c else False
 
+                                    if ifac_size == None:
+                                        ifac_size = 16
+
                                     interface = I2PInterface.I2PInterface(
                                         RNS.Transport,
                                         name,
                                         Reticulum.storagepath,
                                         i2p_peers,
                                         connectable = connectable,
+                                        ifac_size = ifac_size,
+                                        ifac_netname = ifac_netname,
+                                        ifac_netkey = ifac_netkey,
                                     )
 
                                     if "outgoing" in c and c.as_bool("outgoing") == False:
@@ -650,10 +740,6 @@ class Reticulum:
                                     interface.announce_cap = announce_cap
                                     if configured_bitrate:
                                         interface.bitrate = configured_bitrate
-                                    if ifac_size != None:
-                                        interface.ifac_size = ifac_size
-                                    else:
-                                        interface.ifac_size = 16
 
                                 if c["type"] == "SerialInterface":
                                     port = c["port"] if "port" in c else None
@@ -827,12 +913,31 @@ class Reticulum:
                                     flow_control = c.as_bool("flow_control") if "flow_control" in c else False
                                     id_interval = int(c["id_interval"]) if "id_interval" in c else None
                                     id_callsign = c["id_callsign"] if "id_callsign" in c else None
+                                    st_alock = float(c["airtime_limit_short"]) if "airtime_limit_short" in c else None
+                                    lt_alock = float(c["airtime_limit_long"]) if "airtime_limit_long" in c else None
+
+                                    force_ble = False
+                                    ble_name = None
+                                    ble_addr = None
 
                                     port = c["port"] if "port" in c else None
-                                    
+
                                     if port == None:
                                         raise ValueError("No port specified for RNode interface")
 
+                                    if port != None:
+                                        ble_uri_scheme = "ble://"
+                                        if port.lower().startswith(ble_uri_scheme):
+                                            force_ble = True
+                                            ble_string = port[len(ble_uri_scheme):]
+                                            port = None
+                                            if len(ble_string) == 0:
+                                                pass
+                                            elif len(ble_string.split(":")) == 6 and len(ble_string) == 17:
+                                                ble_addr = ble_string
+                                            else:
+                                                ble_name = ble_string
+                                    
                                     interface = RNodeInterface.RNodeInterface(
                                         RNS.Transport,
                                         name,
@@ -844,7 +949,12 @@ class Reticulum:
                                         cr = codingrate,
                                         flow_control = flow_control,
                                         id_interval = id_interval,
-                                        id_callsign = id_callsign
+                                        id_callsign = id_callsign,
+                                        st_alock = st_alock,
+                                        lt_alock = lt_alock,
+                                        ble_addr = ble_addr,
+                                        ble_name = ble_name,
+                                        force_ble = force_ble,
                                     )
 
                                     if "outgoing" in c and c.as_bool("outgoing") == False:
@@ -862,10 +972,110 @@ class Reticulum:
                                     else:
                                         interface.ifac_size = 8
 
+                                if c["type"] == "RNodeMultiInterface":
+                                    count = 0
+                                    enabled_count = 0
+
+                                    # Count how many interfaces are in the file
+                                    for subinterface in c:
+                                        # if the retrieved entry is not a string, it must be a dictionary, which is what we want
+                                        if not isinstance(c[subinterface], str):
+                                            count += 1
+
+                                    # Count how many interfaces are enabled to allow for appropriate matrix sizing
+                                    for subinterface in c:
+                                        # if the retrieved entry is not a string, it must be a dictionary, which is what we want
+                                        if not isinstance(c[subinterface], str):
+                                            subinterface_config = self.config["interfaces"][name][subinterface]
+                                            if (("interface_enabled" in subinterface_config) and subinterface_config.as_bool("interface_enabled") == True) or (("enabled" in c) and c.as_bool("enabled") == True):
+                                                enabled_count += 1
+
+                                    # Create an array with a row for each subinterface
+                                    subint_config = [[0 for x in range(11)] for y in range(enabled_count)]
+                                    subint_index = 0
+
+                                    for subinterface in c:
+                                        # If the retrieved entry is not a string, it must be a dictionary, which is what we want
+                                        if not isinstance(c[subinterface], str):
+                                            subinterface_config = self.config["interfaces"][name][subinterface]
+                                            if (("interface_enabled" in subinterface_config) and subinterface_config.as_bool("interface_enabled") == True) or (("enabled" in c) and c.as_bool("enabled") == True):
+                                                subint_config[subint_index][0] = subinterface
+
+                                                subint_vport = subinterface_config["vport"] if "vport" in subinterface_config else None
+                                                subint_config[subint_index][1] = subint_vport
+
+                                                frequency = int(subinterface_config["frequency"]) if "frequency" in subinterface_config else None
+                                                subint_config[subint_index][2] = frequency
+                                                bandwidth = int(subinterface_config["bandwidth"]) if "bandwidth" in subinterface_config else None
+                                                subint_config[subint_index][3] = bandwidth
+                                                txpower = int(subinterface_config["txpower"]) if "txpower" in subinterface_config else None
+                                                subint_config[subint_index][4] = txpower
+                                                spreadingfactor = int(subinterface_config["spreadingfactor"]) if "spreadingfactor" in subinterface_config else None
+                                                subint_config[subint_index][5] = spreadingfactor 
+                                                codingrate = int(subinterface_config["codingrate"]) if "codingrate" in subinterface_config else None
+                                                subint_config[subint_index][6] = codingrate
+                                                flow_control = subinterface_config.as_bool("flow_control") if "flow_control" in subinterface_config else False
+                                                subint_config[subint_index][7] = flow_control 
+                                                st_alock = float(subinterface_config["airtime_limit_short"]) if "airtime_limit_short" in subinterface_config else None
+                                                subint_config[subint_index][8] = st_alock
+                                                lt_alock = float(subinterface_config["airtime_limit_long"]) if "airtime_limit_long" in subinterface_config else None
+                                                subint_config[subint_index][9] = lt_alock
+
+                                                if "outgoing" in subinterface_config and subinterface_config.as_bool("outgoing") == False:
+                                                    subint_config[subint_index][10] = False
+                                                else:
+                                                    subint_config[subint_index][10] = True
+                                                subint_index += 1
+
+                                    # if no subinterfaces are defined
+                                    if count == 0:
+                                        raise ValueError("No subinterfaces configured for "+name)
+                                    # if no subinterfaces are enabled
+                                    elif enabled_count == 0:
+                                        raise ValueError("No subinterfaces enabled for "+name)
+
+                                    id_interval = int(c["id_interval"]) if "id_interval" in c else None
+                                    id_callsign = c["id_callsign"] if "id_callsign" in c else None
+                                    port = c["port"] if "port" in c else None
+                                    
+                                    if port == None:
+                                        raise ValueError("No port specified for "+name)
+
+                                    interface = RNodeMultiInterface.RNodeMultiInterface(
+                                        RNS.Transport,
+                                        name,
+                                        port,
+                                        subint_config,
+                                        id_interval = id_interval,
+                                        id_callsign = id_callsign
+                                    )
+
+                                    interface.IN = False
+                                    interface.OUT = False
+
+                                    interface.mode = interface_mode
+
+                                    interface.announce_cap = announce_cap
+                                    if configured_bitrate:
+                                        interface.bitrate = configured_bitrate
+                                    if ifac_size != None:
+                                        interface.ifac_size = ifac_size
+                                    else:
+                                        interface.ifac_size = 8
+
+
                                 if interface != None:
                                     interface.announce_rate_target = announce_rate_target
                                     interface.announce_rate_grace = announce_rate_grace
                                     interface.announce_rate_penalty = announce_rate_penalty
+                                    interface.ingress_control = ingress_control
+                                    if ic_max_held_announces != None: interface.ic_max_held_announces = ic_max_held_announces
+                                    if ic_burst_hold != None: interface.ic_burst_hold = ic_burst_hold
+                                    if ic_burst_freq_new != None: interface.ic_burst_freq_new = ic_burst_freq_new
+                                    if ic_burst_freq != None: interface.ic_burst_freq = ic_burst_freq
+                                    if ic_new_time != None: interface.ic_new_time = ic_new_time
+                                    if ic_burst_penalty != None: interface.ic_burst_penalty = ic_burst_penalty
+                                    if ic_held_release_interval != None: interface.ic_held_release_interval = ic_held_release_interval
 
                                     interface.ifac_netname = ifac_netname
                                     interface.ifac_netkey = ifac_netkey
@@ -892,6 +1102,9 @@ class Reticulum:
 
                                     RNS.Transport.interfaces.append(interface)
 
+                                    if isinstance(interface, RNS.Interfaces.RNodeMultiInterface.RNodeMultiInterface):
+                                        interface.start()
+
                             else:
                                 RNS.log("Skipping disabled interface \""+name+"\"", RNS.LOG_DEBUG)
 
@@ -911,7 +1124,6 @@ class Reticulum:
                 
                 if mode == None:
                     mode = Interface.Interface.MODE_FULL
-                
                 interface.mode = mode
 
                 if configured_bitrate:
@@ -922,7 +1134,7 @@ class Reticulum:
                 else:
                     interface.ifac_size = 8
 
-                interface.announce_cap = announce_cap
+                interface.announce_cap = announce_cap if announce_cap != None else Reticulum.ANNOUNCE_CAP/100.0
                 interface.announce_rate_target = announce_rate_target
                 interface.announce_rate_grace = announce_rate_grace
                 interface.announce_rate_penalty = announce_rate_penalty
@@ -953,11 +1165,13 @@ class Reticulum:
                 RNS.Transport.interfaces.append(interface)
 
     def _should_persist_data(self):
-        self.__persist_data()
+        if time.time() > self.last_data_persist+Reticulum.GRACIOUS_PERSIST_INTERVAL:
+            self.__persist_data()
 
     def __persist_data(self):
         RNS.Transport.persist_data()
         RNS.Identity.persist_data()
+        self.last_data_persist = time.time()
 
     def __clean_caches(self):
         RNS.log("Cleaning resource and packet caches...", RNS.LOG_EXTREME)
@@ -1010,7 +1224,8 @@ class Reticulum:
                         rpc_connection.send(self.get_interface_stats())
 
                     if path == "path_table":
-                        rpc_connection.send(self.get_path_table())
+                        mh = call["max_hops"]
+                        rpc_connection.send(self.get_path_table(max_hops=mh))
 
                     if path == "rate_table":
                         rpc_connection.send(self.get_rate_table())
@@ -1021,18 +1236,30 @@ class Reticulum:
                     if path == "next_hop":
                         rpc_connection.send(self.get_next_hop(call["destination_hash"]))
 
+                    if path == "first_hop_timeout":
+                        rpc_connection.send(self.get_first_hop_timeout(call["destination_hash"]))
+
+                    if path == "link_count":
+                        rpc_connection.send(self.get_link_count())
+
                     if path == "packet_rssi":
                         rpc_connection.send(self.get_packet_rssi(call["packet_hash"]))
 
                     if path == "packet_snr":
                         rpc_connection.send(self.get_packet_snr(call["packet_hash"]))
 
+                    if path == "packet_q":
+                        rpc_connection.send(self.get_packet_q(call["packet_hash"]))
+
                 if "drop" in call:
                     path = call["drop"]
 
                     if path == "path":
                         rpc_connection.send(self.drop_path(call["destination_hash"]))
 
+                    if path == "all_via":
+                        rpc_connection.send(self.drop_all_via(call["destination_hash"]))
+
                     if path == "announce_queues":
                         rpc_connection.send(self.drop_announce_queues())
 
@@ -1057,6 +1284,10 @@ class Reticulum:
                 else:
                     ifstats["clients"] = None
 
+                if hasattr(interface, "parent_interface") and interface.parent_interface != None:
+                    ifstats["parent_interface_name"] = str(interface.parent_interface)
+                    ifstats["parent_interface_hash"] = interface.parent_interface.get_hash()
+
                 if hasattr(interface, "i2p") and hasattr(interface, "connectable"):
                     if interface.connectable:
                         ifstats["i2p_connectable"] = True
@@ -1069,6 +1300,38 @@ class Reticulum:
                     else:
                         ifstats["i2p_b32"] = None
 
+                if hasattr(interface, "i2p_tunnel_state"):
+                    if interface.i2p_tunnel_state != None:
+                        state_description = "Unknown State"
+                        if interface.i2p_tunnel_state == I2PInterface.I2PInterfacePeer.TUNNEL_STATE_ACTIVE:
+                            state_description = "Tunnel Active"
+                        elif interface.i2p_tunnel_state == I2PInterface.I2PInterfacePeer.TUNNEL_STATE_INIT:
+                            state_description = "Creating Tunnel"
+                        elif interface.i2p_tunnel_state == I2PInterface.I2PInterfacePeer.TUNNEL_STATE_STALE:
+                            state_description = "Tunnel Unresponsive"
+                        ifstats["tunnelstate"] = state_description
+                    else:
+                        ifstats["tunnelstate"] = None
+
+                if hasattr(interface, "r_airtime_short"):
+                    ifstats["airtime_short"] = interface.r_airtime_short
+
+                if hasattr(interface, "r_airtime_long"):
+                    ifstats["airtime_long"] = interface.r_airtime_long
+
+                if hasattr(interface, "r_channel_load_short"):
+                    ifstats["channel_load_short"] = interface.r_channel_load_short
+
+                if hasattr(interface, "r_channel_load_long"):
+                    ifstats["channel_load_long"] = interface.r_channel_load_long
+
+                if hasattr(interface, "r_battery_state"):
+                    if interface.r_battery_state != 0x00:
+                        ifstats["battery_state"] = interface.r_battery_state
+
+                    if hasattr(interface, "r_battery_percent"):
+                        ifstats["battery_percent"] = interface.r_battery_percent
+
                 if hasattr(interface, "bitrate"):
                     if interface.bitrate != None:
                         ifstats["bitrate"] = interface.bitrate
@@ -1097,8 +1360,14 @@ class Reticulum:
                         ifstats["announce_queue"] = None
 
                 ifstats["name"] = str(interface)
+                ifstats["short_name"] = str(interface.name)
+                ifstats["hash"] = interface.get_hash()
+                ifstats["type"] = str(type(interface).__name__)
                 ifstats["rxb"] = interface.rxb
                 ifstats["txb"] = interface.txb
+                ifstats["incoming_announce_frequency"] = interface.incoming_announce_frequency()
+                ifstats["outgoing_announce_frequency"] = interface.outgoing_announce_frequency()
+                ifstats["held_announces"] = len(interface.held_announces)
                 ifstats["status"] = interface.online
                 ifstats["mode"] = interface.mode
 
@@ -1108,28 +1377,35 @@ class Reticulum:
             stats["interfaces"] = interfaces
             if Reticulum.transport_enabled():
                 stats["transport_id"] = RNS.Transport.identity.hash
+                stats["transport_uptime"] = time.time()-RNS.Transport.start_time
+                if Reticulum.probe_destination_enabled():
+                    stats["probe_responder"] = RNS.Transport.probe_destination.hash
+                else:
+                    stats["probe_responder"] = None
 
             return stats
 
-    def get_path_table(self):
+    def get_path_table(self, max_hops=None):
         if self.is_connected_to_shared_instance:
             rpc_connection = multiprocessing.connection.Client(self.rpc_addr, authkey=self.rpc_key)
-            rpc_connection.send({"get": "path_table"})
+            rpc_connection.send({"get": "path_table", "max_hops": max_hops})
             response = rpc_connection.recv()
             return response
 
         else:
             path_table = []
             for dst_hash in RNS.Transport.destination_table:
-                entry = {
-                    "hash": dst_hash,
-                    "timestamp": RNS.Transport.destination_table[dst_hash][0],
-                    "via": RNS.Transport.destination_table[dst_hash][1],
-                    "hops": RNS.Transport.destination_table[dst_hash][2],
-                    "expires": RNS.Transport.destination_table[dst_hash][3],
-                    "interface": str(RNS.Transport.destination_table[dst_hash][5]),
-                }
-                path_table.append(entry)
+                path_hops = RNS.Transport.destination_table[dst_hash][2]
+                if max_hops == None or path_hops <= max_hops:
+                    entry = {
+                        "hash": dst_hash,
+                        "timestamp": RNS.Transport.destination_table[dst_hash][0],
+                        "via": RNS.Transport.destination_table[dst_hash][1],
+                        "hops": path_hops,
+                        "expires": RNS.Transport.destination_table[dst_hash][3],
+                        "interface": str(RNS.Transport.destination_table[dst_hash][5]),
+                    }
+                    path_table.append(entry)
 
             return path_table
 
@@ -1164,6 +1440,22 @@ class Reticulum:
         else:
             return RNS.Transport.expire_path(destination)
 
+    def drop_all_via(self, transport_hash):
+        if self.is_connected_to_shared_instance:
+            rpc_connection = multiprocessing.connection.Client(self.rpc_addr, authkey=self.rpc_key)
+            rpc_connection.send({"drop": "all_via", "destination_hash": transport_hash})
+            response = rpc_connection.recv()
+            return response
+
+        else:
+            dropped_count = 0
+            for destination_hash in RNS.Transport.destination_table:
+                if RNS.Transport.destination_table[destination_hash][1] == transport_hash:
+                    RNS.Transport.expire_path(destination_hash)
+                    dropped_count += 1
+
+            return dropped_count
+
     def drop_announce_queues(self):
         if self.is_connected_to_shared_instance:
             rpc_connection = multiprocessing.connection.Client(self.rpc_addr, authkey=self.rpc_key)
@@ -1184,16 +1476,51 @@ class Reticulum:
         else:
             return str(RNS.Transport.next_hop_interface(destination))
 
+    def get_first_hop_timeout(self, destination):
+        if self.is_connected_to_shared_instance:
+            try:
+                rpc_connection = multiprocessing.connection.Client(self.rpc_addr, authkey=self.rpc_key)
+                rpc_connection.send({"get": "first_hop_timeout", "destination_hash": destination})
+                response = rpc_connection.recv()
+
+                if self.is_connected_to_shared_instance and hasattr(self, "_force_shared_instance_bitrate") and self._force_shared_instance_bitrate:
+                    simulated_latency = ((1/self._force_shared_instance_bitrate)*8)*RNS.Reticulum.MTU
+                    RNS.log("Adding simulated latency of "+RNS.prettytime(simulated_latency)+" to first hop timeout", RNS.LOG_DEBUG)
+                    response += simulated_latency
+
+                return response
+            except Exception as e:
+                RNS.log("An error occurred while getting first hop timeout from shared instance: "+str(e), RNS.LOG_ERROR)
+                return RNS.Reticulum.DEFAULT_PER_HOP_TIMEOUT
+
+        else:
+            return RNS.Transport.first_hop_timeout(destination)
+
     def get_next_hop(self, destination):
         if self.is_connected_to_shared_instance:
             rpc_connection = multiprocessing.connection.Client(self.rpc_addr, authkey=self.rpc_key)
             rpc_connection.send({"get": "next_hop", "destination_hash": destination})
             response = rpc_connection.recv()
+
+            # TODO: Remove this debugging function
+            # if not response:
+            #     response = RNS.Transport.next_hop(destination)
+
             return response
 
         else:
             return RNS.Transport.next_hop(destination)
 
+    def get_link_count(self):
+        if self.is_connected_to_shared_instance:
+            rpc_connection = multiprocessing.connection.Client(self.rpc_addr, authkey=self.rpc_key)
+            rpc_connection.send({"get": "link_count"})
+            response = rpc_connection.recv()
+            return response
+
+        else:
+            return len(RNS.Transport.link_table)
+
     def get_packet_rssi(self, packet_hash):
         if self.is_connected_to_shared_instance:
             rpc_connection = multiprocessing.connection.Client(self.rpc_addr, authkey=self.rpc_key)
@@ -1222,6 +1549,20 @@ class Reticulum:
 
             return None
 
+    def get_packet_q(self, packet_hash):
+        if self.is_connected_to_shared_instance:
+            rpc_connection = multiprocessing.connection.Client(self.rpc_addr, authkey=self.rpc_key)
+            rpc_connection.send({"get": "packet_q", "packet_hash": packet_hash})
+            response = rpc_connection.recv()
+            return response
+
+        else:
+            for entry in RNS.Transport.local_client_q_cache:
+                if entry[0] == packet_hash:
+                    return entry[1]
+
+            return None
+
 
     @staticmethod
     def should_use_implicit_proof():
@@ -1246,6 +1587,23 @@ class Reticulum:
         """
         return Reticulum.__transport_enabled
 
+    @staticmethod
+    def remote_management_enabled():
+        """
+        Returns whether remote management is enabled for the
+        running instance.
+
+        When remote management is enabled, authenticated peers
+        can remotely query and manage this instance.
+
+        :returns: True if remote management is enabled, False if not.
+        """
+        return Reticulum.__remote_management_enabled
+
+    @staticmethod
+    def probe_destination_enabled():
+        return Reticulum.__allow_probes
+
 # Default configuration file:
 __default_rns_config__ = '''# This is the default Reticulum config file.
 # You should probably edit it to include any additional,

RNS/Utilities/rncp.py

@@ -24,6 +24,7 @@
 
 import RNS
 import argparse
+import threading
 import time
 import sys
 import os
@@ -32,15 +33,29 @@ from RNS._version import __version__
 
 APP_NAME = "rncp"
 allow_all = False
+allow_fetch = False
+fetch_jail = None
 allowed_identity_hashes = []
 
-def receive(configdir, verbosity = 0, quietness = 0, allowed = [], display_identity = False, limit = None, disable_auth = None, disable_announce = False):
-    global allow_all, allowed_identity_hashes
+REQ_FETCH_NOT_ALLOWED = 0xF0
+
+def listen(configdir, verbosity = 0, quietness = 0, allowed = [], display_identity = False,
+           limit = None, disable_auth = None, fetch_allowed = False, jail = None, announce = False):
+    global allow_all, allow_fetch, allowed_identity_hashes, fetch_jail
+    from tempfile import TemporaryFile
+
+    allow_fetch = fetch_allowed
     identity = None
+    if announce < 0:
+        announce = False
 
     targetloglevel = 3+verbosity-quietness
     reticulum = RNS.Reticulum(configdir=configdir, loglevel=targetloglevel)
 
+    if jail != None:
+        fetch_jail = os.path.abspath(os.path.expanduser(jail))
+        RNS.log("Restricting fetch requests to paths under \""+fetch_jail+"\"", RNS.LOG_VERBOSE)
+
     identity_path = RNS.Reticulum.identitypath+"/"+APP_NAME
     if os.path.isfile(identity_path):
         identity = RNS.Identity.from_file(identity_path)                
@@ -54,16 +69,48 @@ def receive(configdir, verbosity = 0, quietness = 0, allowed = [], display_ident
 
     if display_identity:
         print("Identity     : "+str(identity))
-        print("Receiving on : "+RNS.prettyhexrep(destination.hash))
+        print("Listening on : "+RNS.prettyhexrep(destination.hash))
         exit(0)
 
     if disable_auth:
         allow_all = True
     else:
+        dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+        try:
+            allowed_file_name = "allowed_identities"
+            allowed_file = None
+            if os.path.isfile(os.path.expanduser("/etc/rncp/"+allowed_file_name)):
+                allowed_file = os.path.expanduser("/etc/rncp/"+allowed_file_name)
+            elif os.path.isfile(os.path.expanduser("~/.config/rncp/"+allowed_file_name)):
+                allowed_file = os.path.expanduser("~/.config/rncp/"+allowed_file_name)
+            elif os.path.isfile(os.path.expanduser("~/.rncp/"+allowed_file_name)):
+                allowed_file = os.path.expanduser("~/.rncp/"+allowed_file_name)
+            if allowed_file != None:
+                af = open(allowed_file, "r")
+                al = af.read().replace("\r", "").split("\n")
+                ali = []
+                for a in al:
+                    if len(a) == dest_len:
+                        ali.append(a)
+
+                if len(ali) > 0:
+                    if not allowed:
+                        allowed = ali
+                    else:
+                        allowed.extend(ali)
+                if len(ali) == 1:
+                    ms = "y"
+                else:
+                    ms = "ies"
+                
+                RNS.log("Loaded "+str(len(ali))+" allowed identit"+ms+" from "+str(allowed_file), RNS.LOG_VERBOSE)
+
+        except Exception as e:
+            RNS.log("Error while parsing allowed_identities file. The contained exception was: "+str(e), RNS.LOG_ERROR)
+
         if allowed != None:
             for a in allowed:
                 try:
-                    dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
                     if len(a) != dest_len:
                         raise ValueError("Allowed destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2))
                     try:
@@ -78,16 +125,66 @@ def receive(configdir, verbosity = 0, quietness = 0, allowed = [], display_ident
     if len(allowed_identity_hashes) < 1 and not disable_auth:
         print("Warning: No allowed identities configured, rncp will not accept any files!")
 
-    destination.set_link_established_callback(receive_link_established)
-    print("rncp ready to receive on "+RNS.prettyhexrep(destination.hash))
+    def fetch_request(path, data, request_id, link_id, remote_identity, requested_at):
+        global allow_fetch, fetch_jail
+        if not allow_fetch:
+            return REQ_FETCH_NOT_ALLOWED
 
-    if not disable_announce:
-        destination.announce()
+        file_path = os.path.abspath(os.path.expanduser(data))
+        if fetch_jail:
+            if not file_path.startswith(jail):
+                return REQ_FETCH_NOT_ALLOWED
+
+        target_link = None
+        for link in RNS.Transport.active_links:
+            if link.link_id == link_id:
+                target_link = link
+
+        if not os.path.isfile(file_path):
+            RNS.log("Client-requested file not found: "+str(file_path), RNS.LOG_VERBOSE)
+            return False
+        else:
+            if target_link != None:
+                RNS.log("Sending file "+str(file_path)+" to client", RNS.LOG_VERBOSE)
+
+                temp_file = TemporaryFile()
+                real_file = open(file_path, "rb")
+                filename_bytes = os.path.basename(file_path).encode("utf-8")
+                filename_len = len(filename_bytes)
+
+                if filename_len > 0xFFFF:
+                    print("Filename exceeds max size, cannot send")
+                    exit(1)
+
+                temp_file.write(filename_len.to_bytes(2, "big"))
+                temp_file.write(filename_bytes)
+                temp_file.write(real_file.read())
+                temp_file.seek(0)
+
+                fetch_resource = RNS.Resource(temp_file, target_link)
+                return True
+            else:
+                return None
+
+
+    destination.set_link_established_callback(client_link_established)
+    destination.register_request_handler("fetch_file", response_generator=fetch_request, allow=RNS.Destination.ALLOW_LIST, allowed_list=allowed_identity_hashes)
+    print("rncp listening on "+RNS.prettyhexrep(destination.hash))
+
+    if announce >= 0:
+        def job():
+            destination.announce()
+            if announce > 0:
+                while True:
+                    time.sleep(announce)
+                    destination.announce()
+
+        threading.Thread(target=job, daemon=True).start()
     
     while True:
         time.sleep(1)
 
-def receive_link_established(link):
+def client_link_established(link):
     RNS.log("Incoming link established", RNS.LOG_VERBOSE)
     link.set_remote_identified_callback(receive_sender_identified)
     link.set_resource_strategy(RNS.Link.ACCEPT_APP)
@@ -181,7 +278,233 @@ def sender_progress(resource):
         resource_done = True
 
 link = None
-def send(configdir, verbosity = 0, quietness = 0, destination = None, file = None, timeout = RNS.Transport.PATH_REQUEST_TIMEOUT):
+def fetch(configdir, verbosity = 0, quietness = 0, destination = None, file = None, timeout = RNS.Transport.PATH_REQUEST_TIMEOUT, silent=False):
+    global current_resource, resource_done, link, speed
+    targetloglevel = 3+verbosity-quietness
+
+    try:
+        dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+        if len(destination) != dest_len:
+            raise ValueError("Allowed destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2))
+        try:
+            destination_hash = bytes.fromhex(destination)
+        except Exception as e:
+            raise ValueError("Invalid destination entered. Check your input.")
+    except Exception as e:
+        print(str(e))
+        exit(1)
+
+    reticulum = RNS.Reticulum(configdir=configdir, loglevel=targetloglevel)
+
+    identity_path = RNS.Reticulum.identitypath+"/"+APP_NAME
+    if os.path.isfile(identity_path):
+        identity = RNS.Identity.from_file(identity_path)
+        if identity == None:
+            RNS.log("Could not load identity for rncp. The identity file at \""+str(identity_path)+"\" may be corrupt or unreadable.", RNS.LOG_ERROR)
+            exit(2)
+    else:
+        identity = None
+
+    if identity == None:
+        RNS.log("No valid saved identity found, creating new...", RNS.LOG_INFO)
+        identity = RNS.Identity()
+        identity.to_file(identity_path)
+
+    if not RNS.Transport.has_path(destination_hash):
+        RNS.Transport.request_path(destination_hash)
+        if silent:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested")
+        else:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested  ", end=" ")
+        sys.stdout.flush()
+
+    i = 0
+    syms = "⢄⢂⢁⡁⡈⡐⡠"
+    estab_timeout = time.time()+timeout
+    while not RNS.Transport.has_path(destination_hash) and time.time() < estab_timeout:
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
+
+    if not RNS.Transport.has_path(destination_hash):
+        if silent:
+            print("Path not found")
+        else:
+            print("\r                                                            \rPath not found")
+        exit(1)
+    else:
+        if silent:
+            print("Establishing link with "+RNS.prettyhexrep(destination_hash))
+        else:
+            print("\r                                                            \rEstablishing link with "+RNS.prettyhexrep(destination_hash)+"  ", end=" ")
+
+    listener_identity = RNS.Identity.recall(destination_hash)
+    listener_destination = RNS.Destination(
+        listener_identity,
+        RNS.Destination.OUT,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        "receive"
+    )
+
+    link = RNS.Link(listener_destination)
+    while link.status != RNS.Link.ACTIVE and time.time() < estab_timeout:
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
+
+    if not RNS.Transport.has_path(destination_hash):
+        if silent:
+            print("Could not establish link with "+RNS.prettyhexrep(destination_hash))
+        else:
+            print("\r                                                            \rCould not establish link with "+RNS.prettyhexrep(destination_hash))
+        exit(1)
+    else:
+        if silent:
+            print("Requesting file from remote...")
+        else:
+            print("\r                                                            \rRequesting file from remote  ", end=" ")
+
+    link.identify(identity)
+
+    request_resolved = False
+    request_status = "unknown"
+    resource_resolved = False
+    resource_status = "unrequested"
+    current_resource = None
+    def request_response(request_receipt):
+        nonlocal request_resolved, request_status
+        if request_receipt.response == False:
+            request_status = "not_found"
+        elif request_receipt.response == None:
+            request_status = "remote_error"
+        elif request_receipt.response == REQ_FETCH_NOT_ALLOWED:
+            request_status = "fetch_not_allowed"
+        else:
+            request_status = "found"
+
+        request_resolved = True
+
+    def request_failed(request_receipt):
+        nonlocal request_resolved, request_status
+        request_status = "unknown"
+        request_resolved = True
+
+    def fetch_resource_started(resource):
+        nonlocal resource_status
+        current_resource = resource
+        current_resource.progress_callback(sender_progress)
+        resource_status = "started"
+
+    def fetch_resource_concluded(resource):
+        nonlocal resource_resolved, resource_status
+        if resource.status == RNS.Resource.COMPLETE:
+            if resource.total_size > 4:
+                filename_len = int.from_bytes(resource.data.read(2), "big")
+                filename = resource.data.read(filename_len).decode("utf-8")
+
+                counter = 0
+                saved_filename = filename
+                while os.path.isfile(saved_filename):
+                    counter += 1
+                    saved_filename = filename+"."+str(counter)
+                
+                file = open(saved_filename, "wb")
+                file.write(resource.data.read())
+                file.close()
+                resource_status = "completed"
+
+            else:
+                print("Invalid data received, ignoring resource")
+                resource_status = "invalid_data"
+
+        else:
+            print("Resource failed")
+            resource_status = "failed"
+
+        resource_resolved = True
+
+    link.set_resource_strategy(RNS.Link.ACCEPT_ALL)
+    link.set_resource_started_callback(fetch_resource_started)
+    link.set_resource_concluded_callback(fetch_resource_concluded)
+    link.request("fetch_file", data=file, response_callback=request_response, failed_callback=request_failed)
+
+    syms = "⢄⢂⢁⡁⡈⡐⡠"
+    while not request_resolved:
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
+
+    if request_status == "fetch_not_allowed":
+        if not silent: print("\r                                                            \r", end="")
+        print("Fetch request failed, fetching the file "+str(file)+" was not allowed by the remote")
+        link.teardown()
+        time.sleep(0.15)
+        exit(0)
+    elif request_status == "not_found":
+        if not silent: print("\r                                                            \r", end="")
+        print("Fetch request failed, the file "+str(file)+" was not found on the remote")
+        link.teardown()
+        time.sleep(0.15)
+        exit(0)
+    elif request_status == "remote_error":
+        if not silent: print("\r                                                            \r", end="")
+        print("Fetch request failed due to an error on the remote system")
+        link.teardown()
+        time.sleep(0.15)
+        exit(0)
+    elif request_status == "unknown":
+        if not silent: print("\r                                                            \r", end="")
+        print("Fetch request failed due to an unknown error (probably not authorised)")
+        link.teardown()
+        time.sleep(0.15)
+        exit(0)
+    elif request_status == "found":
+        if not silent: print("\r                                                            \r", end="")
+
+    while not resource_resolved:
+        if not silent:
+            time.sleep(0.1)
+            if current_resource:
+                prg = current_resource.get_progress()
+                percent = round(prg * 100.0, 1)
+                stat_str = str(percent)+"% - " + size_str(int(prg*current_resource.total_size)) + " of " + size_str(current_resource.total_size) + " - " +size_str(speed, "b")+"ps"
+                if prg != 1.0:
+                    print("\r                                                                                  \rTransferring file "+syms[i]+" "+stat_str, end=" ")
+                else:
+                    print("\r                                                                                  \rTransfer complete  "+stat_str, end=" ")
+            else:
+                print("\r                                                                                  \rWaiting for transfer to start "+syms[i]+" ", end=" ")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
+
+    if current_resource.status != RNS.Resource.COMPLETE:
+        if silent:
+            print("The transfer failed")
+        else:
+            print("\r                                                            \rThe transfer failed")
+        exit(1)
+    else:
+        if silent:
+            print(str(file)+" fetched from "+RNS.prettyhexrep(destination_hash))
+        else:
+            #print("\r                                                                                  \r"+str(file)+" fetched from "+RNS.prettyhexrep(destination_hash))
+            print("\n"+str(file)+" fetched from "+RNS.prettyhexrep(destination_hash))
+        link.teardown()
+        time.sleep(0.15)
+        exit(0)
+
+    link.teardown()
+    exit(0)
+
+
+def send(configdir, verbosity = 0, quietness = 0, destination = None, file = None, timeout = RNS.Transport.PATH_REQUEST_TIMEOUT, silent=False):
     global current_resource, resource_done, link, speed
     from tempfile import TemporaryFile
     targetloglevel = 3+verbosity-quietness
@@ -226,7 +549,12 @@ def send(configdir, verbosity = 0, quietness = 0, destination = None, file = Non
 
     identity_path = RNS.Reticulum.identitypath+"/"+APP_NAME
     if os.path.isfile(identity_path):
-        identity = RNS.Identity.from_file(identity_path)                
+        identity = RNS.Identity.from_file(identity_path)
+        if identity == None:
+            RNS.log("Could not load identity for rncp. The identity file at \""+str(identity_path)+"\" may be corrupt or unreadable.", RNS.LOG_ERROR)
+            exit(2)
+    else:
+        identity = None
 
     if identity == None:
         RNS.log("No valid saved identity found, creating new...", RNS.LOG_INFO)
@@ -235,23 +563,33 @@ def send(configdir, verbosity = 0, quietness = 0, destination = None, file = Non
 
     if not RNS.Transport.has_path(destination_hash):
         RNS.Transport.request_path(destination_hash)
-        print("Path to "+RNS.prettyhexrep(destination_hash)+" requested  ", end=" ")
+        if silent:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested")
+        else:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested  ", end=" ")
         sys.stdout.flush()
 
     i = 0
     syms = "⢄⢂⢁⡁⡈⡐⡠"
     estab_timeout = time.time()+timeout
     while not RNS.Transport.has_path(destination_hash) and time.time() < estab_timeout:
-        time.sleep(0.1)
-        print(("\b\b"+syms[i]+" "), end="")
-        sys.stdout.flush()
-        i = (i+1)%len(syms)
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
 
     if not RNS.Transport.has_path(destination_hash):
-        print("\r                                                            \rPath not found")
+        if silent:
+            print("Path not found")
+        else:
+            print("\r                                                            \rPath not found")
         exit(1)
     else:
-        print("\r                                                            \rEstablishing link with "+RNS.prettyhexrep(destination_hash)+" ", end=" ")
+        if silent:
+            print("Establishing link with "+RNS.prettyhexrep(destination_hash))
+        else:
+            print("\r                                                            \rEstablishing link with "+RNS.prettyhexrep(destination_hash)+" ", end=" ")
 
     receiver_identity = RNS.Identity.recall(destination_hash)
     receiver_destination = RNS.Destination(
@@ -264,48 +602,86 @@ def send(configdir, verbosity = 0, quietness = 0, destination = None, file = Non
 
     link = RNS.Link(receiver_destination)
     while link.status != RNS.Link.ACTIVE and time.time() < estab_timeout:
-        time.sleep(0.1)
-        print(("\b\b"+syms[i]+" "), end="")
-        sys.stdout.flush()
-        i = (i+1)%len(syms)
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
 
-    if not RNS.Transport.has_path(destination_hash):
-        print("\r                                                            \rCould not establish link with "+RNS.prettyhexrep(destination_hash))
+    if time.time() > estab_timeout:
+        if silent:
+            print("Link establishment with "+RNS.prettyhexrep(destination_hash)+" timed out")
+        else:
+            print("\r                                                            \rLink establishment with "+RNS.prettyhexrep(destination_hash)+" timed out")
+        exit(1)
+    elif not RNS.Transport.has_path(destination_hash):
+        if silent:
+            print("No path found to "+RNS.prettyhexrep(destination_hash))
+        else:
+            print("\r                                                            \rNo path found to "+RNS.prettyhexrep(destination_hash))
         exit(1)
     else:
-        print("\r                                                            \rAdvertising file resource  ", end=" ")
+        if silent:
+            print("Advertising file resource...")
+        else:
+            print("\r                                                            \rAdvertising file resource  ", end=" ")
 
     link.identify(identity)
     resource = RNS.Resource(temp_file, link, callback = sender_progress, progress_callback = sender_progress)
     current_resource = resource
 
     while resource.status < RNS.Resource.TRANSFERRING:
-        time.sleep(0.1)
-        print(("\b\b"+syms[i]+" "), end="")
-        sys.stdout.flush()
-        i = (i+1)%len(syms)
+        if not silent:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
 
     
     if resource.status > RNS.Resource.COMPLETE:
-        print("\r                                                            \rFile was not accepted by "+RNS.prettyhexrep(destination_hash))
+        if silent:
+            print("File was not accepted by "+RNS.prettyhexrep(destination_hash))
+        else:
+            print("\r                                                            \rFile was not accepted by "+RNS.prettyhexrep(destination_hash))
         exit(1)
     else:
-        print("\r                                                            \rTransferring file  ", end=" ")
+        if silent:
+            print("Transferring file...")
+        else:
+            print("\r                                                            \rTransferring file  ", end=" ")
 
-    while not resource_done:
+    def progress_update(i, done=False):
         time.sleep(0.1)
         prg = current_resource.get_progress()
         percent = round(prg * 100.0, 1)
         stat_str = str(percent)+"% - " + size_str(int(prg*current_resource.total_size)) + " of " + size_str(current_resource.total_size) + " - " +size_str(speed, "b")+"ps"
-        print("\r                                                                                  \rTransferring file "+syms[i]+" "+stat_str, end=" ")
+        if not done:
+            print("\r                                                                                  \rTransferring file "+syms[i]+" "+stat_str, end=" ")
+        else:
+            print("\r                                                                                  \rTransfer complete  "+stat_str, end=" ")
         sys.stdout.flush()
         i = (i+1)%len(syms)
+        return i
+
+    while not resource_done:
+        if not silent:
+            i = progress_update(i)
+
+    if not silent:
+        i = progress_update(i, done=True)
 
     if current_resource.status != RNS.Resource.COMPLETE:
-        print("\r                                                            \rThe transfer failed")
+        if silent:
+            print("The transfer failed")
+        else:
+            print("\r                                                            \rThe transfer failed")
         exit(1)
     else:
-        print("\r                                                                                  \r"+str(file_path)+" copied to "+RNS.prettyhexrep(destination_hash))
+        if silent:
+            print(str(file_path)+" copied to "+RNS.prettyhexrep(destination_hash))
+        else:
+            # print("\r                                                                                  \r"+str(file_path)+" copied to "+RNS.prettyhexrep(destination_hash))
+            print("\n"+str(file_path)+" copied to "+RNS.prettyhexrep(destination_hash))
         link.teardown()
         time.sleep(0.25)
         real_file.close()
@@ -320,29 +696,51 @@ def main():
         parser.add_argument("--config", metavar="path", action="store", default=None, help="path to alternative Reticulum config directory", type=str)
         parser.add_argument('-v', '--verbose', action='count', default=0, help="increase verbosity")
         parser.add_argument('-q', '--quiet', action='count', default=0, help="decrease verbosity")
+        parser.add_argument("-S", '--silent', action='store_true', default=False, help="disable transfer progress output")
+        parser.add_argument("-l", '--listen', action='store_true', default=False, help="listen for incoming transfer requests")
+        parser.add_argument("-F", '--allow-fetch', action='store_true', default=False, help="allow authenticated clients to fetch files")
+        parser.add_argument("-f", '--fetch', action='store_true', default=False, help="fetch file from remote listener instead of sending")
+        parser.add_argument("-j", "--jail", metavar="path", action="store", default=None, help="restrict fetch requests to specified path", type=str)
+        parser.add_argument("-b", action='store', metavar="seconds", default=-1, help="announce interval, 0 to only announce at startup", type=int)
+        parser.add_argument('-a', metavar="allowed_hash", dest="allowed", action='append', help="allow this identity", type=str)
+        parser.add_argument('-n', '--no-auth', action='store_true', default=False, help="accept requests from anyone")
         parser.add_argument('-p', '--print-identity', action='store_true', default=False, help="print identity and destination info and exit")
-        parser.add_argument("-r", '--receive', action='store_true', default=False, help="wait for incoming files")
-        parser.add_argument("-b", '--no-announce', action='store_true', default=False, help="don't announce at program start")
-        parser.add_argument('-a', metavar="allowed_hash", dest="allowed", action='append', help="accept from this identity", type=str)
-        parser.add_argument('-n', '--no-auth', action='store_true', default=False, help="accept files from anyone")
         parser.add_argument("-w", action="store", metavar="seconds", type=float, help="sender timeout before giving up", default=RNS.Transport.PATH_REQUEST_TIMEOUT)
         # parser.add_argument("--limit", action="store", metavar="files", type=float, help="maximum number of files to accept", default=None)
         parser.add_argument("--version", action="version", version="rncp {version}".format(version=__version__))
         
         args = parser.parse_args()
 
-        if args.receive or args.print_identity:
-            receive(
+        if args.listen or args.print_identity:
+            listen(
                 configdir = args.config,
                 verbosity=args.verbose,
                 quietness=args.quiet,
                 allowed = args.allowed,
+                fetch_allowed = args.allow_fetch,
+                jail = args.jail,
                 display_identity=args.print_identity,
                 # limit=args.limit,
                 disable_auth=args.no_auth,
-                disable_announce=args.no_announce,
+                announce=args.b,
             )
 
+        elif args.fetch:
+            if args.destination != None and args.file != None:
+                fetch(
+                    configdir = args.config,
+                    verbosity = args.verbose,
+                    quietness = args.quiet,
+                    destination = args.destination,
+                    file = args.file,
+                    timeout = args.w,
+                    silent = args.silent,
+                )
+            else:
+                print("")
+                parser.print_help()
+                print("")
+
         elif args.destination != None and args.file != None:
             send(
                 configdir = args.config,
@@ -351,6 +749,7 @@ def main():
                 destination = args.destination,
                 file = args.file,
                 timeout = args.w,
+                silent = args.silent,
             )
 
         else:

RNS/Utilities/rnid.py

@@ -0,0 +1,600 @@
+#!/usr/bin/env python3
+
+# MIT License
+#
+# Copyright (c) 2023 Mark Qvist / unsigned.io
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import RNS
+import argparse
+import time
+import sys
+import os
+import base64
+
+from RNS._version import __version__
+
+APP_NAME = "rnid"
+
+SIG_EXT = "rsg"
+ENCRYPT_EXT = "rfe"
+CHUNK_SIZE = 16*1024*1024
+
+def spin(until=None, msg=None, timeout=None):
+    i = 0
+    syms = "⢄⢂⢁⡁⡈⡐⡠"
+    if timeout != None:
+        timeout = time.time()+timeout
+
+    print(msg+"  ", end=" ")
+    while (timeout == None or time.time()<timeout) and not until():
+        time.sleep(0.1)
+        print(("\b\b"+syms[i]+" "), end="")
+        sys.stdout.flush()
+        i = (i+1)%len(syms)
+
+    print("\r"+" "*len(msg)+"  \r", end="")
+
+    if timeout != None and time.time() > timeout:
+        return False
+    else:
+        return True
+
+def main():
+    try:
+        parser = argparse.ArgumentParser(description="Reticulum Identity & Encryption Utility")
+        # parser.add_argument("file", nargs="?", default=None, help="input file path", type=str)
+
+        parser.add_argument("--config", metavar="path", action="store", default=None, help="path to alternative Reticulum config directory", type=str)
+        parser.add_argument("-i", "--identity", metavar="identity", action="store", default=None, help="hexadecimal Reticulum Destination hash or path to Identity file", type=str)
+        parser.add_argument("-g", "--generate", metavar="file", action="store", default=None, help="generate a new Identity")
+        parser.add_argument("-m", "--import", dest="import_str", metavar="identity_data", action="store", default=None, help="import Reticulum identity in hex, base32 or base64 format", type=str)
+        parser.add_argument("-x", "--export", action="store_true", default=None, help="export identity to hex, base32 or base64 format")
+
+        parser.add_argument("-v", "--verbose", action="count", default=0, help="increase verbosity")
+        parser.add_argument("-q", "--quiet", action="count", default=0, help="decrease verbosity")
+
+        parser.add_argument("-a", "--announce", metavar="aspects", action="store", default=None, help="announce a destination based on this Identity")
+        parser.add_argument("-H", "--hash", metavar="aspects", action="store", default=None, help="show destination hashes for other aspects for this Identity")
+        parser.add_argument("-e", "--encrypt", metavar="file", action="store", default=None, help="encrypt file")
+        parser.add_argument("-d", "--decrypt", metavar="file", action="store", default=None, help="decrypt file")
+        parser.add_argument("-s", "--sign", metavar="path", action="store", default=None, help="sign file")
+        parser.add_argument("-V", "--validate", metavar="path", action="store", default=None, help="validate signature")
+
+        parser.add_argument("-r", "--read", metavar="file", action="store", default=None, help="input file path", type=str)
+        parser.add_argument("-w", "--write", metavar="file", action="store", default=None, help="output file path", type=str)
+        parser.add_argument("-f", "--force", action="store_true", default=None, help="write output even if it overwrites existing files")
+        parser.add_argument("-I", "--stdin", action="store_true", default=False, help=argparse.SUPPRESS) # "read input from STDIN instead of file"
+        parser.add_argument("-O", "--stdout", action="store_true", default=False, help=argparse.SUPPRESS) # help="write output to STDOUT instead of file", 
+
+        parser.add_argument("-R", "--request", action="store_true", default=False, help="request unknown Identities from the network")
+        parser.add_argument("-t", action="store", metavar="seconds", type=float, help="identity request timeout before giving up", default=RNS.Transport.PATH_REQUEST_TIMEOUT)
+        parser.add_argument("-p", "--print-identity", action="store_true", default=False, help="print identity info and exit")
+        parser.add_argument("-P", "--print-private", action="store_true", default=False, help="allow displaying private keys")
+
+        parser.add_argument("-b", "--base64", action="store_true", default=False, help="Use base64-encoded input and output")
+        parser.add_argument("-B", "--base32", action="store_true", default=False, help="Use base32-encoded input and output")
+
+        parser.add_argument("--version", action="version", version="rnid {version}".format(version=__version__))
+        
+        args = parser.parse_args()
+
+        ops = 0;
+        for t in [args.encrypt, args.decrypt, args.validate, args.sign]:
+            if t:
+                ops += 1
+        
+        if ops > 1:
+            RNS.log("This utility currently only supports one of the encrypt, decrypt, sign or verify operations per invocation", RNS.LOG_ERROR)
+            exit(1)
+
+        if not args.read:
+            if args.encrypt:
+                args.read = args.encrypt
+            if args.decrypt:
+                args.read = args.decrypt
+            if args.sign:
+                args.read = args.sign
+
+        identity_str = args.identity
+        if args.import_str:
+            identity_bytes = None
+            try:
+                if args.base64:
+                    identity_bytes = base64.urlsafe_b64decode(args.import_str)
+                elif args.base32:
+                    identity_bytes = base64.b32decode(args.import_str)
+                else:
+                    identity_bytes = bytes.fromhex(args.import_str)
+            except Exception as e:
+                print("Invalid identity data specified for import: "+str(e))
+                exit(41)
+
+            try:
+                identity = RNS.Identity.from_bytes(identity_bytes)
+            except Exception as e:
+                print("Could not create Reticulum identity from specified data: "+str(e))
+                exit(42)
+
+            RNS.log("Identity imported")
+            if args.base64:
+                RNS.log("Public Key  : "+base64.urlsafe_b64encode(identity.get_public_key()).decode("utf-8"))
+            elif args.base32:
+                RNS.log("Public Key  : "+base64.b32encode(identity.get_public_key()).decode("utf-8"))
+            else:
+                RNS.log("Public Key  : "+RNS.hexrep(identity.get_public_key(), delimit=False))
+            if identity.prv:
+                if args.print_private:
+                    if args.base64:
+                        RNS.log("Private Key : "+base64.urlsafe_b64encode(identity.get_private_key()).decode("utf-8"))
+                    elif args.base32:
+                        RNS.log("Private Key : "+base64.b32encode(identity.get_private_key()).decode("utf-8"))
+                    else:
+                        RNS.log("Private Key : "+RNS.hexrep(identity.get_private_key(), delimit=False))
+                else:
+                    RNS.log("Private Key : Hidden")
+
+            if args.write:
+                try:
+                    wp = os.path.expanduser(args.write)
+                    if not os.path.isfile(wp) or args.force:
+                        identity.to_file(wp)
+                        RNS.log("Wrote imported identity to "+str(args.write))
+                    else:
+                        print("File "+str(wp)+" already exists, not overwriting")
+                        exit(43)
+
+                except Exception as e:
+                    print("Error while writing imported identity to file: "+str(e))
+                    exit(44)
+
+            exit(0)
+
+        if not args.generate and not identity_str:
+            print("\nNo identity provided, cannot continue\n")
+            parser.print_help()
+            print("")
+            exit(2)
+
+        else:
+            targetloglevel = 4
+            verbosity = args.verbose
+            quietness = args.quiet
+            if verbosity != 0 or quietness != 0:
+                targetloglevel = targetloglevel+verbosity-quietness
+            
+            # Start Reticulum
+            reticulum = RNS.Reticulum(configdir=args.config, loglevel=targetloglevel)
+            RNS.compact_log_fmt = True
+            if args.stdout:
+                RNS.loglevel = -1
+
+            if args.generate:
+                identity = RNS.Identity()
+                if not args.force and os.path.isfile(args.generate):
+                    RNS.log("Identity file "+str(args.generate)+" already exists. Not overwriting.", RNS.LOG_ERROR)
+                    exit(3)
+                else:
+                    try:
+                        identity.to_file(args.generate)
+                        RNS.log("New identity written to "+str(args.generate))
+                        exit(0)
+                    except Exception as e:
+                        RNS.log("An error ocurred while saving the generated Identity.", RNS.LOG_ERROR)
+                        RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                        exit(4)
+
+            identity = None
+            if len(identity_str) == RNS.Reticulum.TRUNCATED_HASHLENGTH//8*2 and not os.path.isfile(identity_str):
+                # Try recalling Identity from hex-encoded hash
+                try:
+                    destination_hash = bytes.fromhex(identity_str)
+                    identity = RNS.Identity.recall(destination_hash)
+
+                    if identity == None:
+                        if not args.request:
+                            RNS.log("Could not recall Identity for "+RNS.prettyhexrep(destination_hash)+".", RNS.LOG_ERROR)
+                            RNS.log("You can query the network for unknown Identities with the -R option.", RNS.LOG_ERROR)
+                            exit(5)
+                        else:
+                            RNS.Transport.request_path(destination_hash)
+                            def spincheck():
+                                return RNS.Identity.recall(destination_hash) != None
+                            spin(spincheck, "Requesting unknown Identity for "+RNS.prettyhexrep(destination_hash), args.t)
+
+                            if not spincheck():
+                                RNS.log("Identity request timed out", RNS.LOG_ERROR)
+                                exit(6)
+                            else:
+                                identity = RNS.Identity.recall(destination_hash)
+                                RNS.log("Received Identity "+str(identity)+" for destination "+RNS.prettyhexrep(destination_hash)+" from the network")
+
+                    else:
+                        RNS.log("Recalled Identity "+str(identity)+" for destination "+RNS.prettyhexrep(destination_hash))
+
+
+                except Exception as e:
+                    RNS.log("Invalid hexadecimal hash provided", RNS.LOG_ERROR)
+                    exit(7)
+
+                
+            else:
+                # Try loading Identity from file
+                if not os.path.isfile(identity_str):
+                    RNS.log("Specified Identity file not found")
+                    exit(8)
+                else:
+                    try:
+                        identity = RNS.Identity.from_file(identity_str)
+                        RNS.log("Loaded Identity "+str(identity)+" from "+str(identity_str))
+
+                    except Exception as e:
+                        RNS.log("Could not decode Identity from specified file")
+                        exit(9)
+
+            if identity != None:
+                if args.hash:
+                    try:
+                        aspects = args.hash.split(".")
+                        if not len(aspects) > 0:
+                            RNS.log("Invalid destination aspects specified", RNS.LOG_ERROR)
+                            exit(32)
+                        else:
+                            app_name = aspects[0]
+                            aspects = aspects[1:]
+                            if identity.pub != None:
+                                destination = RNS.Destination(identity, RNS.Destination.OUT, RNS.Destination.SINGLE, app_name, *aspects)
+                                RNS.log("The "+str(args.hash)+" destination for this Identity is "+RNS.prettyhexrep(destination.hash))
+                                RNS.log("The full destination specifier is "+str(destination))
+                                time.sleep(0.25)
+                                exit(0)
+                            else:
+                                raise KeyError("No public key known")
+                    except Exception as e:
+                        RNS.log("An error ocurred while attempting to send the announce.", RNS.LOG_ERROR)
+                        RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+                    exit(0)
+
+                if args.announce:
+                    try:
+                        aspects = args.announce.split(".")
+                        if not len(aspects) > 1:
+                            RNS.log("Invalid destination aspects specified", RNS.LOG_ERROR)
+                            exit(32)
+                        else:
+                            app_name = aspects[0]
+                            aspects = aspects[1:]
+                            if identity.prv != None:
+                                destination = RNS.Destination(identity, RNS.Destination.IN, RNS.Destination.SINGLE, app_name, *aspects)
+                                RNS.log("Created destination "+str(destination))
+                                RNS.log("Announcing destination "+RNS.prettyhexrep(destination.hash))
+                                destination.announce()
+                                time.sleep(0.25)
+                                exit(0)
+                            else:
+                                destination = RNS.Destination(identity, RNS.Destination.OUT, RNS.Destination.SINGLE, app_name, *aspects)
+                                RNS.log("The "+str(args.announce)+" destination for this Identity is "+RNS.prettyhexrep(destination.hash))
+                                RNS.log("The full destination specifier is "+str(destination))
+                                RNS.log("Cannot announce this destination, since the private key is not held")
+                                time.sleep(0.25)
+                                exit(33)
+                    except Exception as e:
+                        RNS.log("An error ocurred while attempting to send the announce.", RNS.LOG_ERROR)
+                        RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+
+                    exit(0)
+
+                if args.print_identity:
+                    if args.base64:
+                        RNS.log("Public Key  : "+base64.urlsafe_b64encode(identity.get_public_key()).decode("utf-8"))
+                    elif args.base32:
+                        RNS.log("Public Key  : "+base64.b32encode(identity.get_public_key()).decode("utf-8"))
+                    else:
+                        RNS.log("Public Key  : "+RNS.hexrep(identity.get_public_key(), delimit=False))
+                    if identity.prv:
+                        if args.print_private:
+                            if args.base64:
+                                RNS.log("Private Key : "+base64.urlsafe_b64encode(identity.get_private_key()).decode("utf-8"))
+                            elif args.base32:
+                                RNS.log("Private Key : "+base64.b32encode(identity.get_private_key()).decode("utf-8"))
+                            else:
+                                RNS.log("Private Key : "+RNS.hexrep(identity.get_private_key(), delimit=False))
+                        else:
+                            RNS.log("Private Key : Hidden")
+                    exit(0)
+
+                if args.export:
+                    if identity.prv:
+                            if args.base64:
+                                RNS.log("Exported Identity : "+base64.urlsafe_b64encode(identity.get_private_key()).decode("utf-8"))
+                            elif args.base32:
+                                RNS.log("Exported Identity : "+base64.b32encode(identity.get_private_key()).decode("utf-8"))
+                            else:
+                                RNS.log("Exported Identity : "+RNS.hexrep(identity.get_private_key(), delimit=False))
+                    else:
+                        RNS.log("Identity doesn't hold a private key, cannot export")
+                        exit(50)
+
+                    exit(0)
+
+                if args.validate:
+                    if not args.read and args.validate.lower().endswith("."+SIG_EXT):
+                        args.read = str(args.validate).replace("."+SIG_EXT, "")
+
+                    if not os.path.isfile(args.validate):
+                        RNS.log("Signature file "+str(args.read)+" not found", RNS.LOG_ERROR)
+                        exit(10)
+
+                    if not os.path.isfile(args.read):
+                        RNS.log("Input file "+str(args.read)+" not found", RNS.LOG_ERROR)
+                        exit(11)
+
+                data_input = None
+                if args.read:
+                    if not os.path.isfile(args.read):
+                        RNS.log("Input file "+str(args.read)+" not found", RNS.LOG_ERROR)
+                        exit(12)
+                    else:
+                        try:
+                            data_input = open(args.read, "rb")
+                        except Exception as e:
+                            RNS.log("Could not open input file for reading", RNS.LOG_ERROR)
+                            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            exit(13)
+
+                # TODO: Actually expand this to a good solution
+                # probably need to create a wrapper that takes
+                # into account not closing stdin when done
+                # elif args.stdin:
+                #     data_input = sys.stdin
+
+                data_output = None
+                if args.encrypt and not args.write and not args.stdout and args.read:
+                    args.write = str(args.read)+"."+ENCRYPT_EXT
+
+                if args.decrypt and not args.write and not args.stdout and args.read and args.read.lower().endswith("."+ENCRYPT_EXT):
+                    args.write = str(args.read).replace("."+ENCRYPT_EXT, "")
+
+                if args.sign and identity.prv == None:
+                    RNS.log("Specified Identity does not hold a private key. Cannot sign.", RNS.LOG_ERROR)
+                    exit(14)
+
+                if args.sign and not args.write and not args.stdout and args.read:
+                    args.write = str(args.read)+"."+SIG_EXT
+
+                if args.write:
+                    if not args.force and os.path.isfile(args.write):
+                        RNS.log("Output file "+str(args.write)+" already exists. Not overwriting.", RNS.LOG_ERROR)
+                        exit(15)
+                    else:
+                        try:
+                            data_output = open(args.write, "wb")
+                        except Exception as e:
+                            RNS.log("Could not open output file for writing", RNS.LOG_ERROR)
+                            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            exit(15)
+                
+                # TODO: Actually expand this to a good solution
+                # probably need to create a wrapper that takes
+                # into account not closing stdout when done
+                # elif args.stdout:
+                #     data_output = sys.stdout
+
+                if args.sign:
+                    if identity.prv == None:
+                        RNS.log("Specified Identity does not hold a private key. Cannot sign.", RNS.LOG_ERROR)
+                        exit(16)
+
+                    if not data_input:
+                        if not args.stdout:
+                            RNS.log("Signing requested, but no input data specified", RNS.LOG_ERROR)
+                        exit(17)
+                    else:
+                        if not data_output:
+                            if not args.stdout:
+                                RNS.log("Signing requested, but no output specified", RNS.LOG_ERROR)
+                            exit(18)
+
+                        if not args.stdout:
+                            RNS.log("Signing "+str(args.read))
+                        
+                        try:
+                            data_output.write(identity.sign(data_input.read()))
+                            data_output.close()
+                            data_input.close()
+                            
+                            if not args.stdout:
+                                if args.read:
+                                    RNS.log("File "+str(args.read)+" signed with "+str(identity)+" to "+str(args.write))
+                                    exit(0)
+
+                        except Exception as e:
+                            if not args.stdout:
+                                RNS.log("An error ocurred while encrypting data.", RNS.LOG_ERROR)
+                                RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            try:
+                                data_output.close()
+                            except:
+                                pass
+                            try:
+                                data_input.close()
+                            except:
+                                pass
+                            exit(19)
+
+                if args.validate:
+                    if not data_input:
+                        if not args.stdout:
+                            RNS.log("Signature verification requested, but no input data specified", RNS.LOG_ERROR)
+                        exit(20)
+                    else:
+                        # if not args.stdout:
+                        #     RNS.log("Verifying "+str(args.validate)+" for "+str(args.read))
+                        
+                        try:
+                            try:
+                                sig_input = open(args.validate, "rb")
+                            except Exception as e:
+                                RNS.log("An error ocurred while opening "+str(args.validate)+".", RNS.LOG_ERROR)
+                                RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                                exit(21)
+
+
+                            validated = identity.validate(sig_input.read(), data_input.read())
+                            sig_input.close()
+                            data_input.close()
+
+                            if not validated:
+                                if not args.stdout:
+                                    RNS.log("Signature "+str(args.validate)+" for file "+str(args.read)+" is invalid", RNS.LOG_ERROR)
+                                exit(22)
+                            else:
+                                if not args.stdout:
+                                    RNS.log("Signature "+str(args.validate)+" for file "+str(args.read)+" made by Identity "+str(identity)+" is valid")
+                                exit(0)
+
+                        except Exception as e:
+                            if not args.stdout:
+                                RNS.log("An error ocurred while validating signature.", RNS.LOG_ERROR)
+                                RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            try:
+                                data_output.close()
+                            except:
+                                pass
+                            try:
+                                data_input.close()
+                            except:
+                                pass
+                            exit(23)
+
+                if args.encrypt:
+                    if not data_input:
+                        if not args.stdout:
+                            RNS.log("Encryption requested, but no input data specified", RNS.LOG_ERROR)
+                        exit(24)
+                    else:
+                        if not data_output:
+                            if not args.stdout:
+                                RNS.log("Encryption requested, but no output specified", RNS.LOG_ERROR)
+                            exit(25)
+
+                        if not args.stdout:
+                            RNS.log("Encrypting "+str(args.read))
+                        
+                        try:
+                            more_data = True
+                            while more_data:
+                                chunk = data_input.read(CHUNK_SIZE)
+                                if chunk:
+                                    data_output.write(identity.encrypt(chunk))
+                                else:
+                                    more_data = False
+                            data_output.close()
+                            data_input.close()
+                            if not args.stdout:
+                                if args.read:
+                                    RNS.log("File "+str(args.read)+" encrypted for "+str(identity)+" to "+str(args.write))
+                                    exit(0)
+
+                        except Exception as e:
+                            if not args.stdout:
+                                RNS.log("An error ocurred while encrypting data.", RNS.LOG_ERROR)
+                                RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            try:
+                                data_output.close()
+                            except:
+                                pass
+                            try:
+                                data_input.close()
+                            except:
+                                pass
+                            exit(26)
+
+                if args.decrypt:
+                    if identity.prv == None:
+                        RNS.log("Specified Identity does not hold a private key. Cannot decrypt.", RNS.LOG_ERROR)
+                        exit(27)
+
+                    if not data_input:
+                        if not args.stdout:
+                            RNS.log("Decryption requested, but no input data specified", RNS.LOG_ERROR)
+                        exit(28)
+                    else:
+                        if not data_output:
+                            if not args.stdout:
+                                RNS.log("Decryption requested, but no output specified", RNS.LOG_ERROR)
+                            exit(29)
+
+                        if not args.stdout:
+                            RNS.log("Decrypting "+str(args.read)+"...")
+                        
+                        try:
+                            more_data = True
+                            while more_data:
+                                chunk = data_input.read(CHUNK_SIZE)
+                                if chunk:
+                                    plaintext = identity.decrypt(chunk)
+                                    if plaintext == None:
+                                        if not args.stdout:
+                                            RNS.log("Data could not be decrypted with the specified Identity")
+                                        exit(30)
+                                    else:
+                                        data_output.write(plaintext)
+                                else:
+                                    more_data = False
+                            data_output.close()
+                            data_input.close()
+                            if not args.stdout:
+                                if args.read:
+                                    RNS.log("File "+str(args.read)+" decrypted with "+str(identity)+" to "+str(args.write))
+                                    exit(0)
+
+                        except Exception as e:
+                            if not args.stdout:
+                                RNS.log("An error ocurred while decrypting data.", RNS.LOG_ERROR)
+                                RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+                            try:
+                                data_output.close()
+                            except:
+                                pass
+                            try:
+                                data_input.close()
+                            except:
+                                pass
+                            exit(31)
+
+            if True:
+                pass
+
+            elif False:
+                pass
+
+            else:
+                print("")
+                parser.print_help()
+                print("")
+
+    except KeyboardInterrupt:
+        print("")
+        exit(255)
+
+if __name__ == "__main__":
+    main()

RNS/Utilities/rnir.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+
+# MIT License
+#
+# Copyright (c) 2023 Mark Qvist / unsigned.io
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import RNS
+import argparse
+import time
+
+from RNS._version import __version__
+
+
+def program_setup(configdir, verbosity = 0, quietness = 0, service = False):
+    targetverbosity = verbosity-quietness
+
+    if service:
+        targetlogdest  = RNS.LOG_FILE
+        targetverbosity = None
+    else:
+        targetlogdest  = RNS.LOG_STDOUT
+
+    reticulum = RNS.Reticulum(configdir=configdir, verbosity=targetverbosity, logdest=targetlogdest)
+    exit(0)
+
+def main():
+    try:
+        parser = argparse.ArgumentParser(description="Reticulum Distributed Identity Resolver")
+        parser.add_argument("--config", action="store", default=None, help="path to alternative Reticulum config directory", type=str)
+        parser.add_argument('-v', '--verbose', action='count', default=0)
+        parser.add_argument('-q', '--quiet', action='count', default=0)
+        parser.add_argument("--exampleconfig", action='store_true', default=False, help="print verbose configuration example to stdout and exit")
+        parser.add_argument("--version", action="version", version="ir {version}".format(version=__version__))
+        
+        args = parser.parse_args()
+
+        if args.exampleconfig:
+            print(__example_rns_config__)
+            exit()
+
+        if args.config:
+            configarg = args.config
+        else:
+            configarg = None
+
+        program_setup(configdir = configarg, verbosity=args.verbose, quietness=args.quiet)
+
+    except KeyboardInterrupt:
+        print("")
+        exit()
+
+__example_rns_config__ = '''# This is an example Identity Resolver file.
+'''
+
+if __name__ == "__main__":
+    main()

RNS/Utilities/rnpath.py

@@ -23,14 +23,95 @@
 # SOFTWARE.
 
 import RNS
+import os
 import sys
 import time
 import argparse
 
 from RNS._version import __version__
 
+remote_link = None
+def connect_remote(destination_hash, auth_identity, timeout, no_output = False):
+    global remote_link, reticulum
+    if not RNS.Transport.has_path(destination_hash):
+        if not no_output:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested", end=" ")
+            sys.stdout.flush()
+        RNS.Transport.request_path(destination_hash)
+        pr_time = time.time()
+        while not RNS.Transport.has_path(destination_hash):
+            time.sleep(0.1)
+            if time.time() - pr_time > timeout:
+                if not no_output:
+                    print("\r                                                          \r", end="")
+                    print("Path request timed out")
+                    exit(12)
+
+    remote_identity = RNS.Identity.recall(destination_hash)
+
+    def remote_link_closed(link):
+        if link.teardown_reason == RNS.Link.TIMEOUT:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("The link timed out, exiting now")
+        elif link.teardown_reason == RNS.Link.DESTINATION_CLOSED:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("The link was closed by the server, exiting now")
+        else:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Link closed unexpectedly, exiting now")
+        exit(10)
+
+    def remote_link_established(link):
+        global remote_link
+        link.identify(auth_identity)
+        remote_link = link
+
+    if not no_output:
+        print("\r                                                          \r", end="")
+        print("Establishing link with remote transport instance...", end=" ")
+        sys.stdout.flush()
+
+    remote_destination = RNS.Destination(remote_identity, RNS.Destination.OUT, RNS.Destination.SINGLE, "rnstransport", "remote", "management")
+    link = RNS.Link(remote_destination)
+    link.set_link_established_callback(remote_link_established)
+    link.set_link_closed_callback(remote_link_closed)
+
+def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity, timeout, drop_queues,
+                  drop_via, max_hops, remote=None, management_identity=None, remote_timeout=RNS.Transport.PATH_REQUEST_TIMEOUT,
+                  no_output=False, json=False):
+    global remote_link, reticulum
+    reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
+    if remote:
+        try:
+            dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+            if len(remote) != dest_len:
+                raise ValueError("Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2))
+            try:
+                identity_hash = bytes.fromhex(remote)
+                remote_hash = RNS.Destination.hash_from_name_and_identity("rnstransport.remote.management", identity_hash)
+            except Exception as e:
+                raise ValueError("Invalid destination entered. Check your input.")
+
+            identity = RNS.Identity.from_file(os.path.expanduser(management_identity))
+            if identity == None:
+                raise ValueError("Could not load management identity from "+str(management_identity))
+
+            try:
+                connect_remote(remote_hash, identity, remote_timeout, no_output)
+            except Exception as e:
+                raise e
+
+        except Exception as e:
+            print(str(e))
+            exit(20)
+
+        while remote_link == None:
+            time.sleep(0.1)
+
 
-def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity, timeout, drop_queues):
     if table:
         destination_hash = None
         if destination_hexhash != None:
@@ -46,23 +127,50 @@ def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity,
                 print(str(e))
                 sys.exit(1)
 
-        reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
-        table = sorted(reticulum.get_path_table(), key=lambda e: (e["interface"], e["hops"]) )
+        if not remote_link:
+            table = sorted(reticulum.get_path_table(max_hops=max_hops), key=lambda e: (e["interface"], e["hops"]) )
+        else:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Sending request...", end=" ")
+                sys.stdout.flush()
+            receipt = remote_link.request("/path", data = ["table", destination_hash, max_hops])
+            while not receipt.concluded():
+                time.sleep(0.1)
+            response = receipt.get_response()
+            if response:
+                table = response
+                print("\r                                                          \r", end="")
+            else:
+                if not no_output:
+                    print("\r                                                          \r", end="")
+                    print("The remote request failed. Likely authentication failure.")
+                exit(10)
 
         displayed = 0
-        for path in table:
-            if destination_hash == None or destination_hash == path["hash"]:
-                displayed += 1
-                exp_str = RNS.timestamp_str(path["expires"])
-                if path["hops"] == 1:
-                    m_str = " "
-                else:
-                    m_str = "s"
-                print(RNS.prettyhexrep(path["hash"])+" is "+str(path["hops"])+" hop"+m_str+" away via "+RNS.prettyhexrep(path["via"])+" on "+path["interface"]+" expires "+RNS.timestamp_str(path["expires"]))
+        if json:
+            import json
+            for p in table:
+                for k in p:
+                    if isinstance(p[k], bytes):
+                        p[k] = RNS.hexrep(p[k], delimit=False)
 
-        if destination_hash != None and displayed == 0:
-            print("No path known")
-            sys.exit(1)
+            print(json.dumps(table))
+            exit()
+        else:
+            for path in table:
+                if destination_hash == None or destination_hash == path["hash"]:
+                    displayed += 1
+                    exp_str = RNS.timestamp_str(path["expires"])
+                    if path["hops"] == 1:
+                        m_str = " "
+                    else:
+                        m_str = "s"
+                    print(RNS.prettyhexrep(path["hash"])+" is "+str(path["hops"])+" hop"+m_str+" away via "+RNS.prettyhexrep(path["via"])+" on "+path["interface"]+" expires "+RNS.timestamp_str(path["expires"]))
+
+            if destination_hash != None and displayed == 0:
+                print("No path known")
+                sys.exit(1)
 
     elif rates:
         destination_hash = None
@@ -79,60 +187,99 @@ def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity,
                 print(str(e))
                 sys.exit(1)
 
-        reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
-        table = sorted(reticulum.get_rate_table(), key=lambda e: e["last"] )
-
-        if len(table) == 0:
-            print("No information available")
-
+        if not remote_link:
+            table = reticulum.get_rate_table()
         else:
-            displayed = 0
-            for entry in table:
-                if destination_hash == None or destination_hash == entry["hash"]:
-                    displayed += 1
-                    try:
-                        last_str = pretty_date(int(entry["last"]))
-                        start_ts = entry["timestamps"][0]
-                        span = max(time.time() - start_ts, 3600.0)
-                        span_hours = span/3600.0
-                        span_str = pretty_date(int(entry["timestamps"][0]))
-                        hour_rate = round(len(entry["timestamps"])/span_hours, 3)
-                        if hour_rate-int(hour_rate) == 0:
-                            hour_rate = int(hour_rate)
-                        
-                        if entry["rate_violations"] > 0:
-                            if entry["rate_violations"] == 1:
-                                s_str = ""
-                            else:
-                                s_str = "s"
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Sending request...", end=" ")
+                sys.stdout.flush()
+            receipt = remote_link.request("/path", data = ["rates", destination_hash])
+            while not receipt.concluded():
+                time.sleep(0.1)
+            response = receipt.get_response()
+            if response:
+                table = response
+                print("\r                                                          \r", end="")
+            else:
+                if not no_output:
+                    print("\r                                                          \r", end="")
+                    print("The remote request failed. Likely authentication failure.")
+                exit(10)
 
-                            rv_str = ", "+str(entry["rate_violations"])+" active rate violation"+s_str
-                        else:
-                            rv_str = ""
-                        
-                        if entry["blocked_until"] > time.time():
-                            bli = time.time()-(int(entry["blocked_until"])-time.time())
-                            bl_str = ", new announces allowed in "+pretty_date(int(bli))
-                        else:
-                            bl_str = ""
+        table = sorted(table, key=lambda e: e["last"])
+        if json:
+            import json
+            for p in table:
+                for k in p:
+                    if isinstance(p[k], bytes):
+                        p[k] = RNS.hexrep(p[k], delimit=False)
 
-        
-                        print(RNS.prettyhexrep(entry["hash"])+" last heard "+last_str+" ago, "+str(hour_rate)+" announces/hour in the last "+span_str+rv_str+bl_str)
-
-                    except Exception as e:
-                        print("Error while processing entry for "+RNS.prettyhexrep(entry["hash"]))
-                        print(str(e))
-
-            if destination_hash != None and displayed == 0:
+            print(json.dumps(table))
+            exit()
+        else:
+            if len(table) == 0:
                 print("No information available")
-                sys.exit(1)
+
+            else:
+                displayed = 0
+                for entry in table:
+                    if destination_hash == None or destination_hash == entry["hash"]:
+                        displayed += 1
+                        try:
+                            last_str = pretty_date(int(entry["last"]))
+                            start_ts = entry["timestamps"][0]
+                            span = max(time.time() - start_ts, 3600.0)
+                            span_hours = span/3600.0
+                            span_str = pretty_date(int(entry["timestamps"][0]))
+                            hour_rate = round(len(entry["timestamps"])/span_hours, 3)
+                            if hour_rate-int(hour_rate) == 0:
+                                hour_rate = int(hour_rate)
+                            
+                            if entry["rate_violations"] > 0:
+                                if entry["rate_violations"] == 1:
+                                    s_str = ""
+                                else:
+                                    s_str = "s"
+
+                                rv_str = ", "+str(entry["rate_violations"])+" active rate violation"+s_str
+                            else:
+                                rv_str = ""
+                            
+                            if entry["blocked_until"] > time.time():
+                                bli = time.time()-(int(entry["blocked_until"])-time.time())
+                                bl_str = ", new announces allowed in "+pretty_date(int(bli))
+                            else:
+                                bl_str = ""
+
+            
+                            print(RNS.prettyhexrep(entry["hash"])+" last heard "+last_str+" ago, "+str(hour_rate)+" announces/hour in the last "+span_str+rv_str+bl_str)
+
+                        except Exception as e:
+                            print("Error while processing entry for "+RNS.prettyhexrep(entry["hash"]))
+                            print(str(e))
+
+                if destination_hash != None and displayed == 0:
+                    print("No information available")
+                    sys.exit(1)
 
     elif drop_queues:
-        reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
-        RNS.log("Dropping announce queues on all interfaces...")
+        if remote_link:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Dropping announce queues on remote instances not yet implemented")
+            exit(255)
+
+        print("Dropping announce queues on all interfaces...")
         reticulum.drop_announce_queues()
     
     elif drop:
+        if remote_link:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Dropping path on remote instances not yet implemented")
+            exit(255)
+
         try:
             dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
             if len(destination_hexhash) != dest_len:
@@ -145,17 +292,44 @@ def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity,
             print(str(e))
             sys.exit(1)
 
-
-        reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
-
         if reticulum.drop_path(destination_hash):
             print("Dropped path to "+RNS.prettyhexrep(destination_hash))
         else:
             print("Unable to drop path to "+RNS.prettyhexrep(destination_hash)+". Does it exist?")
             sys.exit(1)
 
+    elif drop_via:
+        if remote_link:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Dropping all paths via specific transport instance on remote instances yet not implemented")
+            exit(255)
+
+        try:
+            dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+            if len(destination_hexhash) != dest_len:
+                raise ValueError("Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2))
+            try:
+                destination_hash = bytes.fromhex(destination_hexhash)
+            except Exception as e:
+                raise ValueError("Invalid destination entered. Check your input.")
+        except Exception as e:
+            print(str(e))
+            sys.exit(1)
+
+        if reticulum.drop_all_via(destination_hash):
+            print("Dropped all paths via "+RNS.prettyhexrep(destination_hash))
+        else:
+            print("Unable to drop paths via "+RNS.prettyhexrep(destination_hash)+". Does the transport instance exist?")
+            sys.exit(1)
 
     else:
+        if remote_link:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Requesting paths on remote instances not implemented")
+            exit(255)
+
         try:
             dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
             if len(destination_hexhash) != dest_len:
@@ -168,9 +342,6 @@ def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity,
             print(str(e))
             sys.exit(1)
             
-
-        reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
-
         if not RNS.Transport.has_path(destination_hash):
             RNS.Transport.request_path(destination_hash)
             print("Path to "+RNS.prettyhexrep(destination_hash)+" requested  ", end=" ")
@@ -187,15 +358,20 @@ def program_setup(configdir, table, rates, drop, destination_hexhash, verbosity,
 
         if RNS.Transport.has_path(destination_hash):
             hops = RNS.Transport.hops_to(destination_hash)
-            next_hop = RNS.prettyhexrep(reticulum.get_next_hop(destination_hash))
-            next_hop_interface = reticulum.get_next_hop_if_name(destination_hash)
-
-            if hops != 1:
-                ms = "s"
+            next_hop_bytes = reticulum.get_next_hop(destination_hash)
+            if next_hop_bytes == None:
+                print("\r                                                       \rError: Invalid path data returned")
+                sys.exit(1)
             else:
-                ms = ""
+                next_hop = RNS.prettyhexrep(next_hop_bytes)
+                next_hop_interface = reticulum.get_next_hop_if_name(destination_hash)
 
-            print("\rPath found, destination "+RNS.prettyhexrep(destination_hash)+" is "+str(hops)+" hop"+ms+" away via "+next_hop+" on "+next_hop_interface)
+                if hops != 1:
+                    ms = "s"
+                else:
+                    ms = ""
+
+                print("\rPath found, destination "+RNS.prettyhexrep(destination_hash)+" is "+str(hops)+" hop"+ms+" away via "+next_hop+" on "+next_hop_interface)
         else:
             print("\r                                                       \rPath not found")
             sys.exit(1)
@@ -227,6 +403,16 @@ def main():
             default=False
         )
 
+        parser.add_argument(
+            "-m",
+            "--max",
+            action="store",
+            metavar="hops",
+            type=int,
+            help="maximum hops to filter path table by",
+            default=None
+        )
+
         parser.add_argument(
             "-r",
             "--rates",
@@ -251,6 +437,13 @@ def main():
             default=False
         )
 
+        parser.add_argument(
+            "-x", "--drop-via",
+            action="store_true",
+            help="drop all paths via specified transport instance",
+            default=False
+        )
+
         parser.add_argument(
             "-w",
             action="store",
@@ -260,6 +453,41 @@ def main():
             default=RNS.Transport.PATH_REQUEST_TIMEOUT
         )
 
+        parser.add_argument(
+            "-R",
+            action="store",
+            metavar="hash",
+            help="transport identity hash of remote instance to manage",
+            default=None,
+            type=str
+        )
+
+        parser.add_argument(
+            "-i",
+            action="store",
+            metavar="path",
+            help="path to identity used for remote management",
+            default=None,
+            type=str
+        )
+
+        parser.add_argument(
+            "-W",
+            action="store",
+            metavar="seconds",
+            type=float,
+            help="timeout before giving up on remote queries",
+            default=RNS.Transport.PATH_REQUEST_TIMEOUT
+        )
+        
+        parser.add_argument(
+            "-j",
+            "--json",
+            action="store_true",
+            help="output in JSON format",
+            default=False
+        )
+
         parser.add_argument(
             "destination",
             nargs="?",
@@ -277,7 +505,7 @@ def main():
         else:
             configarg = None
 
-        if not args.drop_announces and not args.table and not args.rates and not args.destination:
+        if not args.drop_announces and not args.table and not args.rates and not args.destination and not args.drop_via:
             print("")
             parser.print_help()
             print("")
@@ -291,6 +519,12 @@ def main():
                 verbosity = args.verbose,
                 timeout = args.w,
                 drop_queues = args.drop_announces,
+                drop_via = args.drop_via,
+                max_hops = args.max,
+                remote=args.R,
+                management_identity=args.i,
+                remote_timeout=args.W,
+                json=args.json,
             )
             sys.exit(0)
 

RNS/Utilities/rnprobe.py

@@ -31,8 +31,10 @@ import argparse
 from RNS._version import __version__
 
 DEFAULT_PROBE_SIZE = 16
+DEFAULT_TIMEOUT = 12
 
-def program_setup(configdir, destination_hexhash, size=DEFAULT_PROBE_SIZE, full_name = None, verbosity = 0):
+def program_setup(configdir, destination_hexhash, size=None, full_name = None, verbosity = 0, timeout=None, wait=0, probes=1):
+    if size == None: size = DEFAULT_PROBE_SIZE
     if full_name == None:
         print("The full destination name including application name aspects must be specified for the destination")
         exit()
@@ -71,14 +73,19 @@ def program_setup(configdir, destination_hexhash, size=DEFAULT_PROBE_SIZE, full_
         print("Path to "+RNS.prettyhexrep(destination_hash)+" requested  ", end=" ")
         sys.stdout.flush()
 
+    _timeout = time.time() + (timeout or DEFAULT_TIMEOUT+reticulum.get_first_hop_timeout(destination_hash))
     i = 0
     syms = "⢄⢂⢁⡁⡈⡐⡠"
-    while not RNS.Transport.has_path(destination_hash):
+    while not RNS.Transport.has_path(destination_hash) and not time.time() > _timeout:
         time.sleep(0.1)
         print(("\b\b"+syms[i]+" "), end="")
         sys.stdout.flush()
         i = (i+1)%len(syms)
 
+    if time.time() > _timeout:
+        print("\r                                                          \rPath request timed out")
+        exit(1)
+
     server_identity = RNS.Identity.recall(destination_hash)
 
     request_destination = RNS.Destination(
@@ -89,101 +96,120 @@ def program_setup(configdir, destination_hexhash, size=DEFAULT_PROBE_SIZE, full_
         *aspects
     )
 
-    probe = RNS.Packet(request_destination, os.urandom(size))
-    receipt = probe.send()
+    sent = 0
+    replies = 0
+    while probes:
 
-    if more_output:
-        more = " via "+RNS.prettyhexrep(reticulum.get_next_hop(destination_hash))+" on "+str(reticulum.get_next_hop_if_name(destination_hash))
-    else:
-        more = ""
+        if sent > 0:
+            time.sleep(wait)
 
-    print("\rSent "+str(size)+" byte probe to "+RNS.prettyhexrep(destination_hash)+more+"  ", end=" ")
+        try:
+            probe = RNS.Packet(request_destination, os.urandom(size))
+            probe.pack()
+        except OSError:
+            print("Error: Probe packet size of "+str(len(probe.raw))+" bytes exceed MTU of "+str(RNS.Reticulum.MTU)+" bytes")
+            exit(3)
 
-    i = 0
-    while not receipt.status == RNS.PacketReceipt.DELIVERED:
-        time.sleep(0.1)
-        print(("\b\b"+syms[i]+" "), end="")
-        sys.stdout.flush()
-        i = (i+1)%len(syms)
+        receipt = probe.send()
+        sent += 1
 
-    print("\b\b ")
-    sys.stdout.flush()
+        if more_output:
+            nhd = reticulum.get_next_hop(destination_hash)
+            via_str = " via "+RNS.prettyhexrep(nhd) if nhd != None else ""
+            if_str  = " on "+str(reticulum.get_next_hop_if_name(destination_hash)) if reticulum.get_next_hop_if_name(destination_hash) != "None" else ""
+            more = via_str+if_str
+        else:
+            more = ""
 
-    hops = RNS.Transport.hops_to(destination_hash)
-    if hops != 1:
-        ms = "s"
-    else:
-        ms = ""
+        print("\rSent probe "+str(sent)+" ("+str(size)+" bytes) to "+RNS.prettyhexrep(destination_hash)+more+"  ", end=" ")
 
-    rtt = receipt.get_rtt()
-    if (rtt >= 1):
-        rtt = round(rtt, 3)
-        rttstring = str(rtt)+" seconds"
-    else:
-        rtt = round(rtt*1000, 3)
-        rttstring = str(rtt)+" milliseconds"
+        _timeout = time.time() + (timeout or DEFAULT_TIMEOUT+reticulum.get_first_hop_timeout(destination_hash))
+        i = 0
+        while receipt.status == RNS.PacketReceipt.SENT and not time.time() > _timeout:
+            time.sleep(0.1)
+            print(("\b\b"+syms[i]+" "), end="")
+            sys.stdout.flush()
+            i = (i+1)%len(syms)
 
-    reception_stats = ""
-    if reticulum.is_connected_to_shared_instance:
-        reception_rssi = reticulum.get_packet_rssi(receipt.proof_packet.packet_hash)
-        reception_snr  = reticulum.get_packet_snr(receipt.proof_packet.packet_hash)
-
-        if reception_rssi != None:
-            reception_stats += " [RSSI "+str(reception_rssi)+" dBm]"
+        if time.time() > _timeout:
+            print("\r                                                                \rProbe timed out")
         
-        if reception_snr != None:
-            reception_stats += " [SNR "+str(reception_snr)+" dB]"
+        else:
+            print("\b\b ")
+            sys.stdout.flush()
 
+            if receipt.status == RNS.PacketReceipt.DELIVERED:
+                replies += 1
+                hops = RNS.Transport.hops_to(destination_hash)
+                if hops != 1:
+                    ms = "s"
+                else:
+                    ms = ""
+
+                rtt = receipt.get_rtt()
+                if (rtt >= 1):
+                    rtt = round(rtt, 3)
+                    rttstring = str(rtt)+" seconds"
+                else:
+                    rtt = round(rtt*1000, 3)
+                    rttstring = str(rtt)+" milliseconds"
+
+                reception_stats = ""
+                if reticulum.is_connected_to_shared_instance:
+                    reception_rssi = reticulum.get_packet_rssi(receipt.proof_packet.packet_hash)
+                    reception_snr  = reticulum.get_packet_snr(receipt.proof_packet.packet_hash)
+                    reception_q    = reticulum.get_packet_q(receipt.proof_packet.packet_hash)
+
+                    if reception_rssi != None:
+                        reception_stats += " [RSSI "+str(reception_rssi)+" dBm]"
+                    
+                    if reception_snr != None:
+                        reception_stats += " [SNR "+str(reception_snr)+" dB]"
+                    
+                    if reception_q != None:
+                        reception_stats += " [Link Quality "+str(reception_q)+"%]"
+
+                else:
+                    if receipt.proof_packet != None:
+                        if receipt.proof_packet.rssi != None:
+                            reception_stats += " [RSSI "+str(receipt.proof_packet.rssi)+" dBm]"
+                        
+                        if receipt.proof_packet.snr != None:
+                            reception_stats += " [SNR "+str(receipt.proof_packet.snr)+" dB]"
+
+                print(
+                    "Valid reply from "+
+                    RNS.prettyhexrep(receipt.destination.hash)+
+                    "\nRound-trip time is "+rttstring+
+                    " over "+str(hops)+" hop"+ms+
+                    reception_stats+"\n"
+                )
+
+            else:
+                print("\r                                                          \rProbe timed out")
+
+        probes -= 1
+
+    loss = round((1-(replies/sent))*100, 2)
+    print(f"Sent {sent}, received {replies}, packet loss {loss}%")
+    if loss > 0:
+        exit(2)
     else:
-        if receipt.proof_packet != None:
-            if receipt.proof_packet.rssi != None:
-                reception_stats += " [RSSI "+str(receipt.proof_packet.rssi)+" dBm]"
-            
-            if receipt.proof_packet.snr != None:
-                reception_stats += " [SNR "+str(receipt.proof_packet.snr)+" dB]"
-
-    print(
-        "Valid reply received from "+
-        RNS.prettyhexrep(receipt.destination.hash)+
-        "\nRound-trip time is "+rttstring+
-        " over "+str(hops)+" hop"+ms+
-        reception_stats
-    )
-
+        exit(0)
     
 
 def main():
     try:
         parser = argparse.ArgumentParser(description="Reticulum Probe Utility")
 
-        parser.add_argument("--config",
-            action="store",
-            default=None,
-            help="path to alternative Reticulum config directory",
-            type=str
-        )
-
-        parser.add_argument(
-            "--version",
-            action="version",
-            version="rnprobe {version}".format(version=__version__)
-        )
-
-        parser.add_argument(
-            "full_name",
-            nargs="?",
-            default=None,
-            help="full destination name in dotted notation",
-            type=str
-        )
-
-        parser.add_argument(
-            "destination_hash",
-            nargs="?",
-            default=None,
-            help="hexadecimal hash of the destination",
-            type=str
-        )
+        parser.add_argument("--config", action="store", default=None, help="path to alternative Reticulum config directory", type=str)
+        parser.add_argument("-s", "--size", action="store", default=None, help="size of probe packet payload in bytes", type=int)
+        parser.add_argument("-n", "--probes", action="store", default=1, help="number of probes to send", type=int)
+        parser.add_argument("-t", "--timeout", metavar="seconds", action="store", default=None, help="timeout before giving up", type=float)
+        parser.add_argument("-w", "--wait", metavar="seconds", action="store", default=0, help="time between each probe", type=float)
+        parser.add_argument("--version", action="version", version="rnprobe {version}".format(version=__version__))
+        parser.add_argument("full_name", nargs="?", default=None, help="full destination name in dotted notation", type=str)
+        parser.add_argument("destination_hash", nargs="?", default=None, help="hexadecimal hash of the destination", type=str)
 
         parser.add_argument('-v', '--verbose', action='count', default=0)
 
@@ -202,8 +228,12 @@ def main():
             program_setup(
                 configdir = configarg,
                 destination_hexhash = args.destination_hash,
+                size = args.size,
                 full_name = args.full_name,
-                verbosity = args.verbose
+                verbosity = args.verbose,
+                probes = args.probes,
+                wait = args.wait,
+                timeout = args.timeout,
             )
 
     except KeyboardInterrupt:

RNS/Utilities/rnsd.py

@@ -30,15 +30,15 @@ from RNS._version import __version__
 
 
 def program_setup(configdir, verbosity = 0, quietness = 0, service = False):
-    targetloglevel = 3+verbosity-quietness
+    targetverbosity = verbosity-quietness
 
     if service:
         targetlogdest  = RNS.LOG_FILE
-        targetloglevel = None
+        targetverbosity = None
     else:
         targetlogdest  = RNS.LOG_STDOUT
 
-    reticulum = RNS.Reticulum(configdir=configdir, loglevel=targetloglevel, logdest=targetlogdest)
+    reticulum = RNS.Reticulum(configdir=configdir, verbosity=targetverbosity, logdest=targetlogdest)
     if reticulum.is_connected_to_shared_instance:
         RNS.log("Started rnsd version {version} connected to another shared local instance, this is probably NOT what you want!".format(version=__version__), RNS.LOG_WARNING)
     else:
@@ -87,7 +87,7 @@ __example_rns_config__ = '''# This is an example Reticulum config file.
 # always-on. This directive is optional and can be removed
 # for brevity.
 
-enable_transport = False
+enable_transport = No
 
 
 # By default, the first program to launch the Reticulum
@@ -111,6 +111,30 @@ share_instance = Yes
 shared_instance_port = 37428
 instance_control_port = 37429
 
+
+# On systems where running instances may not have access
+# to the same shared Reticulum configuration directory,
+# it is still possible to allow full interactivity for
+# running instances, by manually specifying a shared RPC
+# key. In almost all cases, this option is not needed, but
+# it can be useful on operating systems such as Android.
+# The key must be specified as bytes in hexadecimal.
+
+# rpc_key = e5c032d3ec4e64a6aca9927ba8ab73336780f6d71790
+
+
+# It is possible to allow remote management of Reticulum
+# systems using the various built-in utilities, such as
+# rnstatus and rnpath. You will need to specify one or
+# more Reticulum Identity hashes for authenticating the
+# queries from client programs. For this purpose, you can
+# use existing identity files, or generate new ones with
+# the rnid utility.
+
+# enable_remote_management = yes
+# remote_management_allowed = 9fb6d773498fb3feda407ed8ef2c3229, 2d882c5586e548d79b5af27bca1776dc
+
+
 # You can configure Reticulum to panic and forcibly close
 # if an unrecoverable interface error occurs, such as the
 # hardware device for an interface disappearing. This is
@@ -120,6 +144,17 @@ instance_control_port = 37429
 panic_on_interface_error = No
 
 
+# When Transport is enabled, it is possible to allow the
+# Transport Instance to respond to probe requests from
+# the rnprobe utility. This can be a useful tool to test
+# connectivity. When this option is enabled, the probe
+# destination will be generated from the Identity of the
+# Transport Instance, and printed to the log at startup.
+# Optional, and disabled by default.
+
+respond_to_probes = No
+
+
 [logging]
 # Valid log levels are 0 through 7:
 #   0: Log only critical information
@@ -259,6 +294,23 @@ loglevel = 4
     # Serial port for the device
     port = /dev/ttyUSB0
 
+    # It is also possible to use BLE devices
+    # instead of wired serial ports. The
+    # target RNode must be paired with the
+    # host device before connecting. BLE
+    # devices can be connected by name,
+    # BLE MAC address or by any available.
+    
+    # Connect to specific device by name
+    # port = ble://RNode 3B87
+
+    # Or by BLE MAC address
+    # port = ble://F4:12:73:29:4E:89
+
+    # Or connect to the first available,
+    # paired device
+    # port = ble://
+
     # Set frequency to 867.2 MHz
     frequency = 867200000
 

RNS/Utilities/rnstatus.py

@@ -23,6 +23,9 @@
 # SOFTWARE.
 
 import RNS
+import os
+import sys
+import time
 import argparse
 
 from RNS._version import __version__
@@ -46,17 +49,174 @@ def size_str(num, suffix='B'):
 
     return "%.2f%s%s" % (num, last_unit, suffix)
 
-def program_setup(configdir, dispall=False, verbosity=0, name_filter=None):
+request_result = None
+request_concluded = False
+def get_remote_status(destination_hash, include_lstats, identity, no_output=False, timeout=RNS.Transport.PATH_REQUEST_TIMEOUT):
+    global request_result, request_concluded
+    link_count = None
+
+    if not RNS.Transport.has_path(destination_hash):
+        if not no_output:
+            print("Path to "+RNS.prettyhexrep(destination_hash)+" requested", end=" ")
+            sys.stdout.flush()
+        RNS.Transport.request_path(destination_hash)
+        pr_time = time.time()
+        while not RNS.Transport.has_path(destination_hash):
+            time.sleep(0.1)
+            if time.time() - pr_time > timeout:
+                if not no_output:
+                    print("\r                                                          \r", end="")
+                    print("Path request timed out")
+                    exit(12)
+
+    remote_identity = RNS.Identity.recall(destination_hash)
+
+    def remote_link_closed(link):
+        if link.teardown_reason == RNS.Link.TIMEOUT:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("The link timed out, exiting now")
+        elif link.teardown_reason == RNS.Link.DESTINATION_CLOSED:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("The link was closed by the server, exiting now")
+        else:
+            if not no_output:
+                print("\r                                                          \r", end="")
+                print("Link closed unexpectedly, exiting now")
+        exit(10)
+
+    def request_failed(request_receipt):
+        global request_result, request_concluded
+        if not no_output:
+            print("\r                                                          \r", end="")
+            print("The remote status request failed. Likely authentication failure.")
+        request_concluded = True
+
+    def got_response(request_receipt):
+        global request_result, request_concluded
+        response = request_receipt.response
+        if isinstance(response, list):
+            status = response[0]
+            if len(response) > 1:
+                link_count = response[1]
+            else:
+                link_count = None
+
+            request_result = (status, link_count)
+
+        request_concluded = True
+
+    def remote_link_established(link):
+        if not no_output:
+            print("\r                                                          \r", end="")
+            print("Sending request...", end=" ")
+            sys.stdout.flush()
+        link.identify(identity)
+        link.request("/status", data = [include_lstats], response_callback = got_response, failed_callback = request_failed)
+
+    if not no_output:
+        print("\r                                                          \r", end="")
+        print("Establishing link with remote transport instance...", end=" ")
+        sys.stdout.flush()
+
+    remote_destination = RNS.Destination(remote_identity, RNS.Destination.OUT, RNS.Destination.SINGLE, "rnstransport", "remote", "management")
+    link = RNS.Link(remote_destination)
+    link.set_link_established_callback(remote_link_established)
+    link.set_link_closed_callback(remote_link_closed)
+
+    while not request_concluded:
+        time.sleep(0.1)
+
+    if request_result != None:
+        print("\r                                                          \r", end="")
+
+    return request_result
+
+def program_setup(configdir, dispall=False, verbosity=0, name_filter=None, json=False, astats=False,
+                  lstats=False, sorting=None, sort_reverse=False, remote=None, management_identity=None,
+                  remote_timeout=RNS.Transport.PATH_REQUEST_TIMEOUT):
     reticulum = RNS.Reticulum(configdir = configdir, loglevel = 3+verbosity)
 
+    link_count = None
     stats = None
-    try:
-        stats = reticulum.get_interface_stats()
-    except Exception as e:
-        pass
+    if remote:
+        try:
+            dest_len = (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2
+            if len(remote) != dest_len:
+                raise ValueError("Destination length is invalid, must be {hex} hexadecimal characters ({byte} bytes).".format(hex=dest_len, byte=dest_len//2))
+            try:
+                identity_hash = bytes.fromhex(remote)
+                destination_hash = RNS.Destination.hash_from_name_and_identity("rnstransport.remote.management", identity_hash)
+            except Exception as e:
+                raise ValueError("Invalid destination entered. Check your input.")
+
+            identity = RNS.Identity.from_file(os.path.expanduser(management_identity))
+            if identity == None:
+                raise ValueError("Could not load management identity from "+str(management_identity))
+
+            try:
+                remote_status = get_remote_status(destination_hash, lstats, identity, no_output=json, timeout=remote_timeout)
+                if remote_status != None:
+                    stats, link_count = remote_status
+            except Exception as e:
+                raise e
+                    
+        except Exception as e:
+            print(str(e))
+            exit(20)
+
+    else:
+        if lstats:
+            try:
+                link_count = reticulum.get_link_count()
+            except Exception as e:
+                pass
+
+        try:
+            stats = reticulum.get_interface_stats()
+        except Exception as e:
+            pass
 
     if stats != None:
-        for ifstat in stats["interfaces"]:
+        if json:
+            import json
+            for s in stats:
+                if isinstance(stats[s], bytes):
+                    stats[s] = RNS.hexrep(stats[s], delimit=False)
+
+                if isinstance(stats[s], dict) or isinstance(stats[s], list):
+                    for i in stats[s]:
+                        if isinstance(i, dict):
+                            for k in i:
+                                if isinstance(i[k], bytes):
+                                    i[k] = RNS.hexrep(i[k], delimit=False)
+
+            print(json.dumps(stats))
+            exit()
+
+        interfaces = stats["interfaces"]
+        if sorting != None and isinstance(sorting, str):
+            sorting = sorting.lower()
+            if sorting == "rate" or sorting == "bitrate":
+                interfaces.sort(key=lambda i: i["bitrate"], reverse=not sort_reverse)
+            if sorting == "rx":
+                interfaces.sort(key=lambda i: i["rxb"], reverse=not sort_reverse)
+            if sorting == "tx":
+                interfaces.sort(key=lambda i: i["txb"], reverse=not sort_reverse)
+            if sorting == "traffic":
+                interfaces.sort(key=lambda i: i["rxb"]+i["txb"], reverse=not sort_reverse)
+            if sorting == "announces" or sorting == "announce":
+                interfaces.sort(key=lambda i: i["incoming_announce_frequency"]+i["outgoing_announce_frequency"], reverse=not sort_reverse)
+            if sorting == "arx":
+                interfaces.sort(key=lambda i: i["incoming_announce_frequency"], reverse=not sort_reverse)
+            if sorting == "atx":
+                interfaces.sort(key=lambda i: i["outgoing_announce_frequency"], reverse=not sort_reverse)
+            if sorting == "held":
+                interfaces.sort(key=lambda i: i["held_announces"], reverse=not sort_reverse)
+
+            
+        for ifstat in interfaces:
             name = ifstat["name"]
 
             if dispall or not (
@@ -98,7 +258,7 @@ def program_setup(configdir, dispall=False, verbosity=0, name_filter=None):
                                 else:
                                     spec_str = " programs"
 
-                                clients_string = "Serving : "+str(cnum)+spec_str
+                                clients_string = "Serving   : "+str(cnum)+spec_str
                             elif name.startswith("I2PInterface["):
                                 if "i2p_connectable" in ifstat and ifstat["i2p_connectable"] == True:
                                     cnum = clients
@@ -107,11 +267,11 @@ def program_setup(configdir, dispall=False, verbosity=0, name_filter=None):
                                     else:
                                         spec_str = " connected I2P endpoints"
 
-                                    clients_string = "Peers   : "+str(cnum)+spec_str
+                                    clients_string = "Peers     : "+str(cnum)+spec_str
                                 else:
                                     clients_string = ""
                             else:
-                                clients_string = "Clients : "+str(clients)
+                                clients_string = "Clients   : "+str(clients)
 
                         else:
                             clients = None
@@ -119,45 +279,91 @@ def program_setup(configdir, dispall=False, verbosity=0, name_filter=None):
                         print(" {n}".format(n=ifstat["name"]))
 
                         if "ifac_netname" in ifstat and ifstat["ifac_netname"] != None:
-                            print("    Network : {nn}".format(nn=ifstat["ifac_netname"]))
+                            print("    Network   : {nn}".format(nn=ifstat["ifac_netname"]))
 
-                        print("    Status  : {ss}".format(ss=ss))
+                        print("    Status    : {ss}".format(ss=ss))
 
                         if clients != None and clients_string != "":
                             print("    "+clients_string)
 
                         if not (name.startswith("Shared Instance[") or name.startswith("TCPInterface[Client") or name.startswith("LocalInterface[")):
-                            print("    Mode    : {mode}".format(mode=modestr))
+                            print("    Mode      : {mode}".format(mode=modestr))
 
                         if "bitrate" in ifstat and ifstat["bitrate"] != None:
-                            print("    Rate    : {ss}".format(ss=speed_str(ifstat["bitrate"])))
+                            print("    Rate      : {ss}".format(ss=speed_str(ifstat["bitrate"])))
+
+                        if "battery_percent" in ifstat and ifstat["battery_percent"] != None:
+                            try:
+                                bpi = int(ifstat["battery_percent"])
+                                print("    Battery   : {bp}%".format(bp=bpi))
+                            except:
+                                pass
+
+                        if "airtime_short" in ifstat and "airtime_long" in ifstat:
+                            print("    Airtime   : {ats}% (15s), {atl}% (1h)".format(ats=str(ifstat["airtime_short"]),atl=str(ifstat["airtime_long"])))
+                        
+                        if "channel_load_short" in ifstat and "channel_load_long" in ifstat:
+                            print("    Ch.Load   : {ats}% (15s), {atl}% (1h)".format(ats=str(ifstat["channel_load_short"]),atl=str(ifstat["channel_load_long"])))
                         
                         if "peers" in ifstat and ifstat["peers"] != None:
-                            print("    Peers   : {np} reachable".format(np=ifstat["peers"]))
+                            print("    Peers     : {np} reachable".format(np=ifstat["peers"]))
+
+                        if "tunnelstate" in ifstat and ifstat["tunnelstate"] != None:
+                            print("    I2P       : {ts}".format(ts=ifstat["tunnelstate"]))
 
                         if "ifac_signature" in ifstat and ifstat["ifac_signature"] != None:
                             sigstr = "<…"+RNS.hexrep(ifstat["ifac_signature"][-5:], delimit=False)+">"
-                            print("    Access  : {nb}-bit IFAC by {sig}".format(nb=ifstat["ifac_size"]*8, sig=sigstr))
+                            print("    Access    : {nb}-bit IFAC by {sig}".format(nb=ifstat["ifac_size"]*8, sig=sigstr))
                         
                         if "i2p_b32" in ifstat and ifstat["i2p_b32"] != None:
-                            print("    I2P B32 : {ep}".format(ep=str(ifstat["i2p_b32"])))
+                            print("    I2P B32   : {ep}".format(ep=str(ifstat["i2p_b32"])))
 
-                        if "announce_queue" in ifstat and ifstat["announce_queue"] != None and ifstat["announce_queue"] > 0:
+                        if astats and "announce_queue" in ifstat and ifstat["announce_queue"] != None and ifstat["announce_queue"] > 0:
                             aqn = ifstat["announce_queue"]
                             if aqn == 1:
-                                print("    Queued  : {np} announce".format(np=aqn))
+                                print("    Queued    : {np} announce".format(np=aqn))
                             else:
-                                print("    Queued  : {np} announces".format(np=aqn))
+                                print("    Queued    : {np} announces".format(np=aqn))
                         
-                        print("    Traffic : {txb}↑\n              {rxb}↓".format(rxb=size_str(ifstat["rxb"]), txb=size_str(ifstat["txb"])))
+                        if astats and "held_announces" in ifstat and ifstat["held_announces"] != None and ifstat["held_announces"] > 0:
+                            aqn = ifstat["held_announces"]
+                            if aqn == 1:
+                                print("    Held      : {np} announce".format(np=aqn))
+                            else:
+                                print("    Held      : {np} announces".format(np=aqn))
+                        
+                        if astats and "incoming_announce_frequency" in ifstat and ifstat["incoming_announce_frequency"] != None:
+                            print("    Announces : {iaf}↑".format(iaf=RNS.prettyfrequency(ifstat["outgoing_announce_frequency"])))
+                            print("                {iaf}↓".format(iaf=RNS.prettyfrequency(ifstat["incoming_announce_frequency"])))
+
+                        print("    Traffic   : {txb}↑\n                {rxb}↓".format(rxb=size_str(ifstat["rxb"]), txb=size_str(ifstat["txb"])))
+
+        lstr = ""
+        if link_count != None and lstats:
+            ms = "y" if link_count == 1 else "ies"
+            if "transport_id" in stats and stats["transport_id"] != None:
+                lstr = f", {link_count} entr{ms} in link table"
+            else:
+                lstr = f" {link_count} entr{ms} in link table"
 
         if "transport_id" in stats and stats["transport_id"] != None:
-            print("\n Reticulum Transport Instance "+RNS.prettyhexrep(stats["transport_id"])+" is running")
+            print("\n Transport Instance "+RNS.prettyhexrep(stats["transport_id"])+" running")
+            if "probe_responder" in stats and stats["probe_responder"] != None:
+                print(" Probe responder at "+RNS.prettyhexrep(stats["probe_responder"])+ " active")
+            if "transport_uptime" in stats and stats["transport_uptime"] != None:
+                print(" Uptime is "+RNS.prettytime(stats["transport_uptime"])+lstr)
+        else:
+            if lstr != "":
+                print(f"\n{lstr}")
 
         print("")
                 
     else:
-        print("Could not get RNS status")
+        if not remote:
+            print("Could not get RNS status")
+        else:
+            print("Could not get RNS status from remote transport instance "+RNS.prettyhexrep(identity_hash))
+        exit(1)
 
 def main():
     try:
@@ -172,6 +378,74 @@ def main():
             help="show all interfaces",
             default=False
         )
+        
+        parser.add_argument(
+            "-A",
+            "--announce-stats",
+            action="store_true",
+            help="show announce stats",
+            default=False
+        )
+        
+        parser.add_argument(
+            "-l",
+            "--link-stats",
+            action="store_true",
+            help="show link stats",
+            default=False,
+        )
+        
+        parser.add_argument(
+            "-s",
+            "--sort",
+            action="store",
+            help="sort interfaces by [rate, traffic, rx, tx, announces, arx, atx, held]",
+            default=None,
+            type=str
+        )
+        
+        parser.add_argument(
+            "-r",
+            "--reverse",
+            action="store_true",
+            help="reverse sorting",
+            default=False,
+        )
+        
+        parser.add_argument(
+            "-j",
+            "--json",
+            action="store_true",
+            help="output in JSON format",
+            default=False
+        )
+
+        parser.add_argument(
+            "-R",
+            action="store",
+            metavar="hash",
+            help="transport identity hash of remote instance to get status from",
+            default=None,
+            type=str
+        )
+
+        parser.add_argument(
+            "-i",
+            action="store",
+            metavar="path",
+            help="path to identity used for remote management",
+            default=None,
+            type=str
+        )
+
+        parser.add_argument(
+            "-w",
+            action="store",
+            metavar="seconds",
+            type=float,
+            help="timeout before giving up on remote queries",
+            default=RNS.Transport.PATH_REQUEST_TIMEOUT
+        )
 
         parser.add_argument('-v', '--verbose', action='count', default=0)
 
@@ -184,7 +458,20 @@ def main():
         else:
             configarg = None
 
-        program_setup(configdir = configarg, dispall = args.all, verbosity=args.verbose, name_filter=args.filter)
+        program_setup(
+            configdir = configarg,
+            dispall = args.all,
+            verbosity=args.verbose,
+            name_filter=args.filter,
+            json=args.json,
+            astats=args.announce_stats,
+            lstats=args.link_stats,
+            sorting=args.sort,
+            sort_reverse=args.reverse,
+            remote=args.R,
+            management_identity=args.i,
+            remote_timeout=args.w,
+        )
 
     except KeyboardInterrupt:
         print("")

RNS/Utilities/rnx.py

@@ -538,7 +538,7 @@ def main():
         parser.add_argument("-x", '--interactive', action='store_true', default=False, help="enter interactive mode")
         parser.add_argument("-b", '--no-announce', action='store_true', default=False, help="don't announce at program start")
         parser.add_argument('-a', metavar="allowed_hash", dest="allowed", action='append', help="accept from this identity", type=str)
-        parser.add_argument('-n', '--noauth', action='store_true', default=False, help="accept files from anyone")
+        parser.add_argument('-n', '--noauth', action='store_true', default=False, help="accept commands from anyone")
         parser.add_argument('-N', '--noid', action='store_true', default=False, help="don't identify to listener")
         parser.add_argument("-d", '--detailed', action='store_true', default=False, help="show detailed result output")
         parser.add_argument("-m", action='store_true', dest="mirror", default=False, help="mirror exit code of remote command")

RNS/__init__.py

@@ -1,6 +1,6 @@
 # MIT License
 #
-# Copyright (c) 2016-2022 Mark Qvist / unsigned.io
+# Copyright (c) 2016-2023 Mark Qvist / unsigned.io and contributors
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -32,10 +32,13 @@ from ._version import __version__
 from .Reticulum import Reticulum
 from .Identity import Identity
 from .Link import Link, RequestReceipt
+from .Channel import MessageBase
+from .Buffer import Buffer, RawChannelReader, RawChannelWriter
 from .Transport import Transport
 from .Destination import Destination
 from .Packet import Packet
 from .Packet import PacketReceipt
+from .Resolver import Resolver
 from .Resource import Resource, ResourceAdvertisement
 from .Cryptography import HKDF
 from .Cryptography import Hashes
@@ -57,10 +60,11 @@ LOG_FILE     = 0x92
 
 LOG_MAXSIZE  = 5*1024*1024
 
-loglevel     = LOG_NOTICE
-logfile      = None
-logdest      = LOG_STDOUT
-logtimefmt   = "%Y-%m-%d %H:%M:%S"
+loglevel        = LOG_NOTICE
+logfile         = None
+logdest         = LOG_STDOUT
+logtimefmt      = "%Y-%m-%d %H:%M:%S"
+compact_log_fmt = False
 
 instance_random = random.Random()
 instance_random.seed(os.urandom(10))
@@ -101,10 +105,14 @@ def timestamp_str(time_s):
     return time.strftime(logtimefmt, timestamp)
 
 def log(msg, level=3, _override_destination = False):
-    global _always_override_destination
-    
+    global _always_override_destination, compact_log_fmt
+    msg = str(msg)
     if loglevel >= level:
-        logstring = "["+timestamp_str(time.time())+"] ["+loglevelname(level)+"] "+msg
+        if not compact_log_fmt:
+            logstring = "["+timestamp_str(time.time())+"] ["+loglevelname(level)+"] "+msg
+        else:
+            logstring = "["+timestamp_str(time.time())+"] "+msg
+
         logging_lock.acquire()
 
         if (logdest == LOG_STDOUT or _always_override_destination or _override_destination):
@@ -136,6 +144,12 @@ def rand():
     result = instance_random.random()
     return result
 
+def trace_exception(e):
+    import traceback
+    exception_info = "".join(traceback.TracebackException.from_exception(e).format())
+    log(f"An unhandled {str(type(e))} exception occurred: {str(e)}", LOG_ERROR)
+    log(exception_info, LOG_ERROR)
+
 def hexrep(data, delimit=True):
     try:
         iter(data)
@@ -153,6 +167,9 @@ def prettyhexrep(data):
     hexrep = "<"+delimiter.join("{:02x}".format(c) for c in data)+">"
     return hexrep
 
+def prettyspeed(num, suffix="b"):
+    return prettysize(num/8, suffix=suffix)+"ps"
+
 def prettysize(num, suffix='B'):
     units = ['','K','M','G','T','P','E','Z']
     last_unit = 'Y'
@@ -172,32 +189,68 @@ def prettysize(num, suffix='B'):
 
     return "%.2f%s%s" % (num, last_unit, suffix)
 
-def prettytime(time, verbose=False):
+def prettyfrequency(hz, suffix="Hz"):
+    num = hz*1e6
+    units = ["µ", "m", "", "K","M","G","T","P","E","Z"]
+    last_unit = "Y"
+
+    for unit in units:
+        if abs(num) < 1000.0:
+            return "%.2f %s%s" % (num, unit, suffix)
+        num /= 1000.0
+
+    return "%.2f%s%s" % (num, last_unit, suffix)
+
+def prettydistance(m, suffix="m"):
+    num = m*1e6
+    units = ["µ", "m", "c", ""]
+    last_unit = "K"
+
+    for unit in units:
+        divisor = 1000.0
+        if unit == "m": divisor = 10
+        if unit == "c": divisor = 100
+
+        if abs(num) < divisor:
+            return "%.2f %s%s" % (num, unit, suffix)
+        num /= divisor
+
+    return "%.2f %s%s" % (num, last_unit, suffix)
+
+def prettytime(time, verbose=False, compact=False):
     days = int(time // (24 * 3600))
     time = time % (24 * 3600)
     hours = int(time // 3600)
     time %= 3600
     minutes = int(time // 60)
     time %= 60
-    seconds = round(time, 2)
+    if compact:
+        seconds = int(time)
+    else:
+        seconds = round(time, 2)
     
     ss = "" if seconds == 1 else "s"
     sm = "" if minutes == 1 else "s"
     sh = "" if hours == 1 else "s"
     sd = "" if days == 1 else "s"
 
+    displayed = 0
     components = []
-    if days > 0:
+    if days > 0 and ((not compact) or displayed < 2):
         components.append(str(days)+" day"+sd if verbose else str(days)+"d")
+        displayed += 1
 
-    if hours > 0:
+    if hours > 0 and ((not compact) or displayed < 2):
         components.append(str(hours)+" hour"+sh if verbose else str(hours)+"h")
+        displayed += 1
 
-    if minutes > 0:
+    if minutes > 0 and ((not compact) or displayed < 2):
         components.append(str(minutes)+" minute"+sm if verbose else str(minutes)+"m")
+        displayed += 1
 
-    if seconds > 0:
+    if seconds > 0 and ((not compact) or displayed < 2):
         components.append(str(seconds)+" second"+ss if verbose else str(seconds)+"s")
+        displayed += 1
 
     i = 0
     tstr = ""
@@ -212,14 +265,64 @@ def prettytime(time, verbose=False):
 
         tstr += c
 
-    return tstr
+    if tstr == "":
+        return "0s"
+    else:
+        return tstr
+
+def prettyshorttime(time, verbose=False, compact=False):
+    time = time*1e6
+    
+    seconds = int(time // 1e6); time %= 1e6
+    milliseconds = int(time // 1e3); time %= 1e3
+
+    if compact:
+        microseconds = int(time)
+    else:
+        microseconds = round(time, 2)
+    
+    ss = "" if seconds == 1 else "s"
+    sms = "" if milliseconds == 1 else "s"
+    sus = "" if microseconds == 1 else "s"
+
+    displayed = 0
+    components = []
+    if seconds > 0 and ((not compact) or displayed < 2):
+        components.append(str(seconds)+" second"+ss if verbose else str(seconds)+"s")
+        displayed += 1
+
+    if milliseconds > 0 and ((not compact) or displayed < 2):
+        components.append(str(milliseconds)+" millisecond"+sms if verbose else str(milliseconds)+"ms")
+        displayed += 1
+
+    if microseconds > 0 and ((not compact) or displayed < 2):
+        components.append(str(microseconds)+" microsecond"+sus if verbose else str(microseconds)+"µs")
+        displayed += 1
+
+    i = 0
+    tstr = ""
+    for c in components:
+        i += 1
+        if i == 1:
+            pass
+        elif i < len(components):
+            tstr += ", "
+        elif i == len(components):
+            tstr += " and "
+
+        tstr += c
+
+    if tstr == "":
+        return "0us"
+    else:
+        return tstr
 
 def phyparams():
     print("Required Physical Layer MTU : "+str(Reticulum.MTU)+" bytes")
     print("Plaintext Packet MDU        : "+str(Packet.PLAIN_MDU)+" bytes")
     print("Encrypted Packet MDU        : "+str(Packet.ENCRYPTED_MDU)+" bytes")
     print("Link Curve                  : "+str(Link.CURVE))
-    print("Link Packet MDU             : "+str(Packet.ENCRYPTED_MDU)+" bytes")
+    print("Link Packet MDU             : "+str(Link.MDU)+" bytes")
     print("Link Public Key Size        : "+str(Link.ECPUBSIZE*8)+" bits")
     print("Link Private Key Size       : "+str(Link.KEYSIZE*8)+" bits")
 
@@ -228,4 +331,95 @@ def panic():
 
 def exit():
     print("")
-    sys.exit(0)
+    sys.exit(0)
+
+
+profiler_ran = False
+profiler_tags = {}
+def profiler(tag=None, capture=False, super_tag=None):
+    global profiler_ran, profiler_tags
+    try:
+        thread_ident = threading.get_ident()
+
+        if capture:
+            end = time.perf_counter()
+            if tag in profiler_tags and thread_ident in profiler_tags[tag]["threads"]:
+                if profiler_tags[tag]["threads"][thread_ident]["current_start"] != None:
+                    begin = profiler_tags[tag]["threads"][thread_ident]["current_start"]
+                    profiler_tags[tag]["threads"][thread_ident]["current_start"] = None
+                    profiler_tags[tag]["threads"][thread_ident]["captures"].append(end-begin)
+                    if not profiler_ran:
+                        profiler_ran = True
+
+        else:
+            if not tag in profiler_tags:
+                profiler_tags[tag] = {"threads": {}, "super": super_tag}
+            if not thread_ident in profiler_tags[tag]["threads"]:
+                profiler_tags[tag]["threads"][thread_ident] = {"current_start": None, "captures": []}
+
+            profiler_tags[tag]["threads"][thread_ident]["current_start"] = time.perf_counter()
+
+    except Exception as e:
+        trace_exception(e)
+
+def profiler_results():
+    from statistics import mean, median, stdev
+    results = {}
+    
+    for tag in profiler_tags:
+        tag_captures = []
+        tag_entry = profiler_tags[tag]
+        
+        for thread_ident in tag_entry["threads"]:
+            thread_entry = tag_entry["threads"][thread_ident]
+            thread_captures = thread_entry["captures"]
+            sample_count = len(thread_captures)
+            
+            if sample_count > 2:
+                thread_results = {
+                    "count": sample_count,
+                    "mean": mean(thread_captures),
+                    "median": median(thread_captures),
+                    "stdev": stdev(thread_captures)
+                }
+
+            tag_captures.extend(thread_captures)
+
+        sample_count = len(tag_captures)
+        if sample_count > 2:
+            tag_results = {
+                "name": tag,
+                "super": tag_entry["super"],
+                "count": len(tag_captures),
+                "mean": mean(tag_captures),
+                "median": median(tag_captures),
+                "stdev": stdev(tag_captures)
+            }
+
+            results[tag] = tag_results
+
+    def print_results_recursive(tag, results, level=0):
+        print_tag_results(tag, level+1)
+
+        for tag_name in results:
+            sub_tag = results[tag_name]
+            if sub_tag["super"] == tag["name"]:
+                print_results_recursive(sub_tag, results, level=level+1)
+
+
+    def print_tag_results(tag, level):
+        ind = "  "*level
+        name = tag["name"]; count = tag["count"]
+        mean = tag["mean"]; tag["median"]; stdev = tag["stdev"]
+        print(f"{ind}{name}")
+        print(f"{ind}  Samples : {count}")
+        print(f"{ind}  Mean    : {prettyshorttime(mean)}")
+        print(f"{ind}  Median  : {prettyshorttime(median)}")
+        print(f"{ind}  St.dev. : {prettyshorttime(stdev)}")
+        print("")
+
+    print("\nProfiler results:\n")
+    for tag_name in results:
+        tag = results[tag_name]
+        if tag["super"] == None:
+            print_results_recursive(tag, results)

RNS/_version.py

@@ -1 +1 @@
-__version__ = "0.3.14"
+__version__ = "0.8.2"

RNS/vendor/ifaddr/__init__.py

@@ -0,0 +1,33 @@
+# Copyright (c) 2014 Stefan C. Mueller
+
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+
+import os
+
+from RNS.vendor.ifaddr._shared import Adapter, IP
+
+if os.name == "nt":
+    from RNS.vendor.ifaddr._win32 import get_adapters
+elif os.name == "posix":
+    from RNS.vendor.ifaddr._posix import get_adapters
+else:
+    raise RuntimeError("Unsupported Operating System: %s" % os.name)
+
+__all__ = ['Adapter', 'IP', 'get_adapters']

RNS/vendor/ifaddr/_posix.py

@@ -0,0 +1,93 @@
+# Copyright (c) 2014 Stefan C. Mueller
+
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+
+import os
+import ctypes.util
+import ipaddress
+import collections
+import socket
+
+from typing import Iterable, Optional
+
+import RNS.vendor.ifaddr._shared as shared
+
+class ifaddrs(ctypes.Structure):
+    pass
+
+
+ifaddrs._fields_ = [
+    ('ifa_next', ctypes.POINTER(ifaddrs)),
+    ('ifa_name', ctypes.c_char_p),
+    ('ifa_flags', ctypes.c_uint),
+    ('ifa_addr', ctypes.POINTER(shared.sockaddr)),
+    ('ifa_netmask', ctypes.POINTER(shared.sockaddr)),
+]
+
+libc = ctypes.CDLL(ctypes.util.find_library("socket" if os.uname()[0] == "SunOS" else "c"), use_errno=True)  # type: ignore
+
+
+def get_adapters(include_unconfigured: bool = False) -> Iterable[shared.Adapter]:
+
+    addr0 = addr = ctypes.POINTER(ifaddrs)()
+    retval = libc.getifaddrs(ctypes.byref(addr))
+    if retval != 0:
+        eno = ctypes.get_errno()
+        raise OSError(eno, os.strerror(eno))
+
+    ips = collections.OrderedDict()
+
+    def add_ip(adapter_name: str, ip: Optional[shared.IP]) -> None:
+        if adapter_name not in ips:
+            index = None  # type: Optional[int]
+            try:
+                # Mypy errors on this when the Windows CI runs:
+                #     error: Module has no attribute "if_nametoindex"
+                index = socket.if_nametoindex(adapter_name)  # type: ignore
+            except (OSError, AttributeError):
+                pass
+            ips[adapter_name] = shared.Adapter(adapter_name, adapter_name, [], index=index)
+        if ip is not None:
+            ips[adapter_name].ips.append(ip)
+
+    while addr:
+        name = addr[0].ifa_name.decode(encoding='UTF-8')
+        ip_addr = shared.sockaddr_to_ip(addr[0].ifa_addr)
+        if ip_addr:
+            if addr[0].ifa_netmask and not addr[0].ifa_netmask[0].sa_familiy:
+                addr[0].ifa_netmask[0].sa_familiy = addr[0].ifa_addr[0].sa_familiy
+            netmask = shared.sockaddr_to_ip(addr[0].ifa_netmask)
+            if isinstance(netmask, tuple):
+                netmaskStr = str(netmask[0])
+                prefixlen = shared.ipv6_prefixlength(ipaddress.IPv6Address(netmaskStr))
+            else:
+                assert netmask is not None, f'sockaddr_to_ip({addr[0].ifa_netmask}) returned None'
+                netmaskStr = str('0.0.0.0/' + netmask)
+                prefixlen = ipaddress.IPv4Network(netmaskStr).prefixlen
+            ip = shared.IP(ip_addr, prefixlen, name)
+            add_ip(name, ip)
+        else:
+            if include_unconfigured:
+                add_ip(name, None)
+        addr = addr[0].ifa_next
+
+    libc.freeifaddrs(addr0)
+
+    return ips.values()

RNS/vendor/ifaddr/_shared.py

@@ -0,0 +1,198 @@
+# Copyright (c) 2014 Stefan C. Mueller
+
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+
+import ctypes
+import socket
+import ipaddress
+import platform
+
+from typing import List, Optional, Tuple, Union
+
+class Adapter(object):
+    """
+    Represents a network interface device controller (NIC), such as a
+    network card. An adapter can have multiple IPs.
+
+    On Linux aliasing (multiple IPs per physical NIC) is implemented
+    by creating 'virtual' adapters, each represented by an instance
+    of this class. Each of those 'virtual' adapters can have both
+    a IPv4 and an IPv6 IP address.
+    """
+
+    def __init__(self, name: str, nice_name: str, ips: List['IP'], index: Optional[int] = None) -> None:
+
+        #: Unique name that identifies the adapter in the system.
+        #: On Linux this is of the form of `eth0` or `eth0:1`, on
+        #: Windows it is a UUID in string representation, such as
+        #: `{846EE342-7039-11DE-9D20-806E6F6E6963}`.
+        self.name = name
+
+        #: Human readable name of the adpater. On Linux this
+        #: is currently the same as :attr:`name`. On Windows
+        #: this is the name of the device.
+        self.nice_name = nice_name
+
+        #: List of :class:`ifaddr.IP` instances in the order they were
+        #: reported by the system.
+        self.ips = ips
+
+        #: Adapter index as used by some API (e.g. IPv6 multicast group join).
+        self.index = index
+
+    def __repr__(self) -> str:
+        return "Adapter(name={name}, nice_name={nice_name}, ips={ips}, index={index})".format(
+            name=repr(self.name), nice_name=repr(self.nice_name), ips=repr(self.ips), index=repr(self.index)
+        )
+
+
+# Type of an IPv4 address (a string in "xxx.xxx.xxx.xxx" format)
+_IPv4Address = str
+
+# Type of an IPv6 address (a three-tuple `(ip, flowinfo, scope_id)`)
+_IPv6Address = Tuple[str, int, int]
+
+
+class IP(object):
+    """
+    Represents an IP address of an adapter.
+    """
+
+    def __init__(self, ip: Union[_IPv4Address, _IPv6Address], network_prefix: int, nice_name: str) -> None:
+
+        #: IP address. For IPv4 addresses this is a string in
+        #: "xxx.xxx.xxx.xxx" format. For IPv6 addresses this
+        #: is a three-tuple `(ip, flowinfo, scope_id)`, where
+        #: `ip` is a string in the usual collon separated
+        #: hex format.
+        self.ip = ip
+
+        #: Number of bits of the IP that represent the
+        #: network. For a `255.255.255.0` netmask, this
+        #: number would be `24`.
+        self.network_prefix = network_prefix
+
+        #: Human readable name for this IP.
+        #: On Linux is this currently the same as the adapter name.
+        #: On Windows this is the name of the network connection
+        #: as configured in the system control panel.
+        self.nice_name = nice_name
+
+    @property
+    def is_IPv4(self) -> bool:
+        """
+        Returns `True` if this IP is an IPv4 address and `False`
+        if it is an IPv6 address.
+        """
+        return not isinstance(self.ip, tuple)
+
+    @property
+    def is_IPv6(self) -> bool:
+        """
+        Returns `True` if this IP is an IPv6 address and `False`
+        if it is an IPv4 address.
+        """
+        return isinstance(self.ip, tuple)
+
+    def __repr__(self) -> str:
+        return "IP(ip={ip}, network_prefix={network_prefix}, nice_name={nice_name})".format(
+            ip=repr(self.ip), network_prefix=repr(self.network_prefix), nice_name=repr(self.nice_name)
+        )
+
+
+if platform.system() == "Darwin" or "BSD" in platform.system():
+
+    # BSD derived systems use marginally different structures
+    # than either Linux or Windows.
+    # I still keep it in `shared` since we can use
+    # both structures equally.
+
+    class sockaddr(ctypes.Structure):
+        _fields_ = [
+            ('sa_len', ctypes.c_uint8),
+            ('sa_familiy', ctypes.c_uint8),
+            ('sa_data', ctypes.c_uint8 * 14),
+        ]
+
+    class sockaddr_in(ctypes.Structure):
+        _fields_ = [
+            ('sa_len', ctypes.c_uint8),
+            ('sa_familiy', ctypes.c_uint8),
+            ('sin_port', ctypes.c_uint16),
+            ('sin_addr', ctypes.c_uint8 * 4),
+            ('sin_zero', ctypes.c_uint8 * 8),
+        ]
+
+    class sockaddr_in6(ctypes.Structure):
+        _fields_ = [
+            ('sa_len', ctypes.c_uint8),
+            ('sa_familiy', ctypes.c_uint8),
+            ('sin6_port', ctypes.c_uint16),
+            ('sin6_flowinfo', ctypes.c_uint32),
+            ('sin6_addr', ctypes.c_uint8 * 16),
+            ('sin6_scope_id', ctypes.c_uint32),
+        ]
+
+else:
+
+    class sockaddr(ctypes.Structure):  # type: ignore
+        _fields_ = [('sa_familiy', ctypes.c_uint16), ('sa_data', ctypes.c_uint8 * 14)]
+
+    class sockaddr_in(ctypes.Structure):  # type: ignore
+        _fields_ = [
+            ('sin_familiy', ctypes.c_uint16),
+            ('sin_port', ctypes.c_uint16),
+            ('sin_addr', ctypes.c_uint8 * 4),
+            ('sin_zero', ctypes.c_uint8 * 8),
+        ]
+
+    class sockaddr_in6(ctypes.Structure):  # type: ignore
+        _fields_ = [
+            ('sin6_familiy', ctypes.c_uint16),
+            ('sin6_port', ctypes.c_uint16),
+            ('sin6_flowinfo', ctypes.c_uint32),
+            ('sin6_addr', ctypes.c_uint8 * 16),
+            ('sin6_scope_id', ctypes.c_uint32),
+        ]
+
+
+def sockaddr_to_ip(sockaddr_ptr: 'ctypes.pointer[sockaddr]') -> Optional[Union[_IPv4Address, _IPv6Address]]:
+    if sockaddr_ptr:
+        if sockaddr_ptr[0].sa_familiy == socket.AF_INET:
+            ipv4 = ctypes.cast(sockaddr_ptr, ctypes.POINTER(sockaddr_in))
+            ippacked = bytes(bytearray(ipv4[0].sin_addr))
+            ip = str(ipaddress.ip_address(ippacked))
+            return ip
+        elif sockaddr_ptr[0].sa_familiy == socket.AF_INET6:
+            ipv6 = ctypes.cast(sockaddr_ptr, ctypes.POINTER(sockaddr_in6))
+            flowinfo = ipv6[0].sin6_flowinfo
+            ippacked = bytes(bytearray(ipv6[0].sin6_addr))
+            ip = str(ipaddress.ip_address(ippacked))
+            scope_id = ipv6[0].sin6_scope_id
+            return (ip, flowinfo, scope_id)
+    return None
+
+
+def ipv6_prefixlength(address: ipaddress.IPv6Address) -> int:
+    prefix_length = 0
+    for i in range(address.max_prefixlen):
+        if int(address) >> i & 1:
+            prefix_length = prefix_length + 1
+    return prefix_length

RNS/vendor/ifaddr/_win32.py

@@ -0,0 +1,145 @@
+# Copyright (c) 2014 Stefan C. Mueller
+
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+
+import ctypes
+from ctypes import wintypes
+from typing import Iterable, List
+
+import RNS.vendor.ifaddr._shared as shared
+
+NO_ERROR = 0
+ERROR_BUFFER_OVERFLOW = 111
+MAX_ADAPTER_NAME_LENGTH = 256
+MAX_ADAPTER_DESCRIPTION_LENGTH = 128
+MAX_ADAPTER_ADDRESS_LENGTH = 8
+AF_UNSPEC = 0
+
+
+class SOCKET_ADDRESS(ctypes.Structure):
+    _fields_ = [('lpSockaddr', ctypes.POINTER(shared.sockaddr)), ('iSockaddrLength', wintypes.INT)]
+
+
+class IP_ADAPTER_UNICAST_ADDRESS(ctypes.Structure):
+    pass
+
+
+IP_ADAPTER_UNICAST_ADDRESS._fields_ = [
+    ('Length', wintypes.ULONG),
+    ('Flags', wintypes.DWORD),
+    ('Next', ctypes.POINTER(IP_ADAPTER_UNICAST_ADDRESS)),
+    ('Address', SOCKET_ADDRESS),
+    ('PrefixOrigin', ctypes.c_uint),
+    ('SuffixOrigin', ctypes.c_uint),
+    ('DadState', ctypes.c_uint),
+    ('ValidLifetime', wintypes.ULONG),
+    ('PreferredLifetime', wintypes.ULONG),
+    ('LeaseLifetime', wintypes.ULONG),
+    ('OnLinkPrefixLength', ctypes.c_uint8),
+]
+
+
+class IP_ADAPTER_ADDRESSES(ctypes.Structure):
+    pass
+
+
+IP_ADAPTER_ADDRESSES._fields_ = [
+    ('Length', wintypes.ULONG),
+    ('IfIndex', wintypes.DWORD),
+    ('Next', ctypes.POINTER(IP_ADAPTER_ADDRESSES)),
+    ('AdapterName', ctypes.c_char_p),
+    ('FirstUnicastAddress', ctypes.POINTER(IP_ADAPTER_UNICAST_ADDRESS)),
+    ('FirstAnycastAddress', ctypes.c_void_p),
+    ('FirstMulticastAddress', ctypes.c_void_p),
+    ('FirstDnsServerAddress', ctypes.c_void_p),
+    ('DnsSuffix', ctypes.c_wchar_p),
+    ('Description', ctypes.c_wchar_p),
+    ('FriendlyName', ctypes.c_wchar_p),
+]
+
+
+iphlpapi = ctypes.windll.LoadLibrary("Iphlpapi")  # type: ignore
+
+
+def enumerate_interfaces_of_adapter(
+    nice_name: str, address: IP_ADAPTER_UNICAST_ADDRESS
+) -> Iterable[shared.IP]:
+
+    # Iterate through linked list and fill list
+    addresses = []  # type: List[IP_ADAPTER_UNICAST_ADDRESS]
+    while True:
+        addresses.append(address)
+        if not address.Next:
+            break
+        address = address.Next[0]
+
+    for address in addresses:
+        ip = shared.sockaddr_to_ip(address.Address.lpSockaddr)
+        assert ip is not None, f'sockaddr_to_ip({address.Address.lpSockaddr}) returned None'
+        network_prefix = address.OnLinkPrefixLength
+        yield shared.IP(ip, network_prefix, nice_name)
+
+
+def get_adapters(include_unconfigured: bool = False) -> Iterable[shared.Adapter]:
+
+    # Call GetAdaptersAddresses() with error and buffer size handling
+
+    addressbuffersize = wintypes.ULONG(15 * 1024)
+    retval = ERROR_BUFFER_OVERFLOW
+    while retval == ERROR_BUFFER_OVERFLOW:
+        addressbuffer = ctypes.create_string_buffer(addressbuffersize.value)
+        retval = iphlpapi.GetAdaptersAddresses(
+            wintypes.ULONG(AF_UNSPEC),
+            wintypes.ULONG(0),
+            None,
+            ctypes.byref(addressbuffer),
+            ctypes.byref(addressbuffersize),
+        )
+    if retval != NO_ERROR:
+        raise ctypes.WinError()  # type: ignore
+
+    # Iterate through adapters fill array
+    address_infos = []  # type: List[IP_ADAPTER_ADDRESSES]
+    address_info = IP_ADAPTER_ADDRESSES.from_buffer(addressbuffer)
+    while True:
+        address_infos.append(address_info)
+        if not address_info.Next:
+            break
+        address_info = address_info.Next[0]
+
+    # Iterate through unicast addresses
+    result = []  # type: List[shared.Adapter]
+    for adapter_info in address_infos:
+
+        # We don't expect non-ascii characters here, so encoding shouldn't matter
+        name = adapter_info.AdapterName.decode()
+        nice_name = adapter_info.Description
+        index = adapter_info.IfIndex
+
+        if adapter_info.FirstUnicastAddress:
+            ips = enumerate_interfaces_of_adapter(
+                adapter_info.FriendlyName, adapter_info.FirstUnicastAddress[0]
+            )
+            ips = list(ips)
+            result.append(shared.Adapter(name, nice_name, ips, index=index))
+        elif include_unconfigured:
+            result.append(shared.Adapter(name, nice_name, [], index=index))
+
+    return result

RNS/vendor/ifaddr/niwrapper.py

@@ -0,0 +1,57 @@
+import ipaddress
+import RNS.vendor.ifaddr
+import socket
+
+from typing import List
+
+AF_INET6 = socket.AF_INET6.value
+AF_INET = socket.AF_INET.value
+
+def interfaces() -> List[str]:
+    adapters = RNS.vendor.ifaddr.get_adapters(include_unconfigured=True)
+    return [a.name for a in adapters]
+
+def interface_names_to_indexes() -> dict:
+    adapters = RNS.vendor.ifaddr.get_adapters(include_unconfigured=True)
+    results = {}
+    for adapter in adapters:
+        results[adapter.name] = adapter.index
+    return results
+
+def interface_name_to_nice_name(ifname) -> str:
+    try:
+        adapters = RNS.vendor.ifaddr.get_adapters(include_unconfigured=True)
+        for adapter in adapters:
+            if adapter.name == ifname:
+                if hasattr(adapter, "nice_name"):
+                    return adapter.nice_name
+    except:
+        return None
+
+    return None
+
+def ifaddresses(ifname) -> dict:
+    adapters = RNS.vendor.ifaddr.get_adapters(include_unconfigured=True)
+    ifa = {}
+    for a in adapters:
+        if a.name == ifname:
+            ipv4s = []
+            ipv6s = []
+            for ip in a.ips:
+                t = {}
+                if ip.is_IPv4:
+                    net = ipaddress.ip_network(str(ip.ip)+"/"+str(ip.network_prefix), strict=False)
+                    t["addr"] = ip.ip
+                    t["prefix"] = ip.network_prefix
+                    t["broadcast"] = str(net.broadcast_address)
+                    ipv4s.append(t)
+                if ip.is_IPv6:
+                    t["addr"] = ip.ip[0]
+                    ipv6s.append(t)
+
+            if len(ipv4s) > 0:
+                ifa[AF_INET] = ipv4s
+            if len(ipv6s) > 0:
+                ifa[AF_INET6] = ipv6s
+
+    return ifa

RNS/vendor/platformutils.py

@@ -8,6 +8,12 @@ def get_platform():
         import sys
         return sys.platform
 
+def is_linux():
+    if get_platform() == "linux":
+        return True
+    else:
+        return False
+
 def is_darwin():
     if get_platform() == "darwin":
         return True
@@ -42,4 +48,4 @@ def cryptography_old_api():
     if cryptography.__version__ == "2.8":
         return True
     else:
-        return False
+        return False

Roadmap.md

@@ -0,0 +1,109 @@
+# Reticulum Development Roadmap
+This document outlines the currently established development roadmap for Reticulum.
+
+1. [Currently Active Work Areas](#currently-active-work-areas)
+2. [Primary Efforts](#primary-efforts)
+    - [Comprehensibility](#comprehensibility)
+    - [Universality](#universality)
+    - [Functionality](#functionality)
+    - [Usability & Utility](#usability--utility)
+    - [Interfaceability](#interfaceability)
+3. [Auxillary Efforts](#auxillary-efforts)
+4. [Release History](#release-history)
+
+## Currently Active Work Areas
+For each release cycle of Reticulum, improvements and additions from the five [Primary Efforts](#primary-efforts) are selected as active work areas, and can be expected to be included in the upcoming releases within that cycle. While not entirely set in stone for each release cycle, they serve as a pointer of what to expect in the near future.
+
+- The current `0.8.x` release cycle aims at completing
+  - [ ] Hot-pluggable interface system
+  - [ ] External interface plugins
+  - [ ] Network-wide path balancing and multi-pathing
+  - [ ] Expanded hardware support
+  - [ ] Overhauling and updating the documentation
+  - [ ] Distributed Destination Naming System
+  - [ ] A standalone RNS Daemon app for Android
+  - [ ] Addding automatic retries to all use cases of the `Request` API
+  - [ ] Performance and memory optimisations of the Python reference implementation
+  - [ ] Fixing bugs discovered while operating Reticulum systems and applications
+
+## Primary Efforts
+The development path for Reticulum is currently laid out in five distinct areas: *Comprehensibility*, *Universality*, *Functionality*, *Usability & Utility* and *Interfaceability*. Conceptualising the development of Reticulum into these areas serves to advance the implementation and work towards the Foundational Goals & Values of Reticulum.
+
+### Comprehensibility
+These efforts are aimed at improving the ease of which Reticulum is understood, and lowering the barrier to entry for people who wish to start building systems on Reticulum.
+
+- Improving [the manual](https://markqvist.github.io/Reticulum/manual/) with tutorials specifically for beginners
+- Updating the documentation to reflect recent changes and improvements
+    - Update descriptions of protocol mechanics
+        - Update announce description
+        - Add in-depth explanation of the IFAC system
+    - Software
+        - Update software descriptions and screenshots
+    - Communications hardware section
+        - Add information about RNode external displays.
+        - Possibly add other relevant types here as well.
+    - Setup *Best Practices For...* / *Installation Examples* section.
+        - Home or office (example)
+        - Vehicles (example)
+        - No-grid/solar/remote sites (example)
+
+### Universality
+These efforts seek to broaden the universality of the Reticulum software and hardware ecosystem by continously diversifying platform support, and by improving the overall availability and ease of deployment of the Reticulum stack.
+
+- OpenWRT support
+- Create a standalone RNS Daemon app for Android
+- A lightweight and portable C implementation for microcontrollers, µRNS
+- A portable, high-performance Reticulum implementation in C/C++, see [#21](https://github.com/markqvist/Reticulum/discussions/21)
+- Performance and memory optimisations of the Python implementation
+- Bindings for other programming languages
+
+### Functionality
+These efforts aim to expand and improve the core functionality and reliability of Reticulum.
+
+- Add support for user-supplied external interface drivers
+- Add interface hot-plug and live up/down control to running instances
+- Add automatic retries to all use cases of the `Request` API
+- Network-wide path balancing
+- Distributed Destination Naming System
+- Globally routable multicast
+- Destination proxying
+- [Metric-based path selection and multiple paths](https://github.com/markqvist/Reticulum/discussions/86)
+
+### Usability & Utility
+These effors seek to make Reticulum easier to use and operate, and to expand the utility of the stack on deployed systems.
+
+- Easy way to share interface configurations, see [#19](https://github.com/markqvist/Reticulum/discussions/19)
+- Transit traffic display in rnstatus
+- rnsconfig utility
+
+### Interfaceability
+These efforts aim to expand the types of physical and virtual interfaces that Reticulum can natively use to transport data.
+
+- Plain ESP32 devices (ESP-Now, WiFi, Bluetooth, etc.)
+- More LoRa transceivers
+- AT-compatible modems
+- Filesystem interface
+- Direct SDR Support
+- Optical mediums
+- IR Transceivers
+- AWDL / OWL
+- HF Modems
+- GNU Radio
+- CAN-bus
+- Raw SPI
+- Raw i²c
+- MQTT
+- XBee
+- Tor
+
+## Auxillary Efforts
+The Reticulum ecosystem is enriched by several other software and hardware projects, and the support and improvement of these, in symbiosis with the core Reticulum project helps expand the reach and utility of Reticulum itself.
+
+This section lists, in no particular order, various important efforts that would be beneficial to the goals of Reticulum.
+
+- The [RNode](https://unsigned.io/rnode/) project
+    - [x] Create a WebUSB-based bootstrapping utility, and integrate this directly into the [RNode Bootstrap Console](#), both on-device, and on an Internet-reachable copy. This will make it much easier to create new RNodes for average users.
+
+## Release History
+
+Please see the [Changelog](./Changelog.md) for a complete release history and changelog of Reticulum.

docs/Makefile

@@ -30,3 +30,8 @@ help:
 		cp -r build/latex/reticulumnetworkstack.pdf ./Reticulum\ Manual.pdf; \
 		echo "PDF Manual Generated"; \
 	fi
+
+	@if [ $@ = "epub" ]; then \
+		cp -r build/epub/ReticulumNetworkStack.epub ./Reticulum\ Manual.epub; \
+		echo "EPUB Manual Generated"; \
+	fi

docs/manual/.buildinfo

@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: f917c18521ed6a10811c2a61e23b9822
+config: bd0a676b0f490cda5d841ab7cd7cade1
 tags: 645f666f9bcd5a90fca523b33c5a78b7

docs/manual/_sources/examples.rst.txt

@@ -92,6 +92,28 @@ The *Request* example explores sendig requests and receiving responses.
 
 This example can also be found at `<https://github.com/markqvist/Reticulum/blob/master/Examples/Request.py>`_.
 
+.. _example-channel:
+
+Channel
+=======
+
+The *Channel* example explores using a ``Channel`` to send structured
+data between peers of a ``Link``.
+
+.. literalinclude:: ../../Examples/Channel.py
+
+This example can also be found at `<https://github.com/markqvist/Reticulum/blob/master/Examples/Channel.py>`_.
+
+Buffer
+======
+
+The *Buffer* example explores using buffered readers and writers to send
+binary data between peers of a ``Link``.
+
+.. literalinclude:: ../../Examples/Buffer.py
+
+This example can also be found at `<https://github.com/markqvist/Reticulum/blob/master/Examples/Buffer.py>`_.
+
 .. _example-filetransfer:
 
 Filetransfer

docs/manual/_sources/forhumans.rst.txt

@@ -0,0 +1,4 @@
+********************************************
+An Explanation of Reticulum for Human Beings
+********************************************
+

docs/manual/_sources/gettingstartedfast.rst.txt

@@ -7,22 +7,73 @@ you want to do. This guide will outline sensible starting paths for different
 scenarios.
 
 
+Standalone Reticulum Installation
+=============================================
+If you simply want to install Reticulum and related utilities on a system,
+the easiest way is via the ``pip`` package manager:
+
+.. code::
+
+   pip install rns
+
+If you do not already have pip installed, you can install it using the package manager
+of your system with a command like ``sudo apt install python3-pip``,
+``sudo pamac install python-pip`` or similar.
+
+You can also dowload the Reticulum release wheels from GitHub, or other release channels,
+and install them offline using ``pip``:
+
+.. code::
+
+   pip install ./rns-0.5.1-py3-none-any.whl
+
+
+Resolving Dependency & Installation Issues
+=============================================
+On some platforms, there may not be binary packages available for all dependencies, and
+``pip`` installation may fail with an error message. In these cases, the issue can usually
+be resolved by installing the development essentials packages for your platform:
+
+.. code::
+
+    # Debian / Ubuntu / Derivatives
+    sudo apt install build-essential
+
+    # Arch / Manjaro / Derivatives
+    sudo pamac install base-devel
+
+    # Fedora
+    sudo dnf groupinstall "Development Tools" "Development Libraries"
+
+With the base development packages installed, ``pip`` should be able to compile any missing
+dependencies from source, and complete installation even on platforms that don't have pre-
+compiled packages available.
+
 Try Using a Reticulum-based Program
 =============================================
 
 If you simply want to try using a program built with Reticulum, a few different
-programs exist that allow basic communication and a range of other useful functions
-over even extremely low-bandwidth Reticulum networks.
+programs exist that allow basic communication and a range of other useful functions,
+even over extremely low-bandwidth Reticulum networks.
 
 These programs will let you get a feel for how Reticulum works. They have been designed
-to run well over networks based on LoRa or packet radio, but can also be used completely
-over local WiFi, wired Ethernet, the Internet, or any combination.
+to run well over networks based on LoRa or packet radio, but can also be used over fast
+links, such as local WiFi, wired Ethernet, the Internet, or any combination.
 
 As such, it is easy to get started experimenting, without having to set up any radio
 transceivers or infrastructure just to try it out. Launching the programs on separate
 devices connected to the same WiFi network is enough to get started, and physical
 radio interfaces can then be added later.
 
+Remote Shell
+^^^^^^^^^^^^
+
+The `rnsh <https://github.com/acehoss/rnsh>`_ program lets you establish fully interactive
+remote shell sessions over Reticulum. It also allows you to pipe any program to or from a
+remote system, and is similar to how ``ssh`` works. The ``rnsh`` is very efficient, and
+can facilitate fully interactive shell sessions, even over extremely low-bandwidth links,
+such as LoRa or packet radio.
+
 Nomad Network
 ^^^^^^^^^^^^^
 
@@ -44,7 +95,7 @@ You can install Nomad Network via pip:
 .. code::
 
    # Install ...
-   pip3 install nomadnet
+   pip install nomadnet
 
    # ... and run
    nomadnet
@@ -59,14 +110,46 @@ Sideband
 
 If you would rather use a program with a graphical user interface, you can take
 a look at `Sideband <https://unsigned.io/sideband>`_, which is available for Android,
-Linux and macOS.
+Linux, macOS and Windows.
 
-.. image:: screenshots/sideband_1.png
-    :align: center
-    :target: _images/sideband_1.png
+.. only:: html
 
-Sideband is currently in the early stages of development, but already provides basic
-communication features, and interoperates with Nomad Network, or any other LXMF client.
+  .. image:: screenshots/sideband_devices.webp
+      :align: center
+      :target: _images/sideband_devices.webp
+
+.. only:: latexpdf
+
+  .. image:: screenshots/sideband_devices.png
+      :align: center
+      :target: _images/sideband_devices.png
+
+Sideband allows you to communicate with other people or LXMF-compatible
+systems over Reticulum networks using LoRa, Packet Radio, WiFi, I2P, Encrypted QR
+Paper Messages, or anything else Reticulum supports. It also interoperates with
+the Nomad Network program.
+
+MeshChat
+^^^^^^^^
+
+The `Reticulum MeshChat <https://github.com/liamcottle/reticulum-meshchat>`_ application
+is a user-friendly LXMF client for macOS and Windows, that also includes voice call
+functionality, and a range of other interesting functions.
+
+.. only:: html
+
+  .. image:: screenshots/meshchat_1.webp
+      :align: center
+      :target: _images/meshchat_1.webp
+
+.. only:: latexpdf
+
+  .. image:: screenshots/meshchat_1.png
+      :align: center
+      :target: _images/meshchat_1.png
+
+Reticulum MeshChat is of course also compatible with Sideband and Nomad Network, or
+any other LXMF client.
 
 Using the Included Utilities
 =============================================
@@ -86,8 +169,8 @@ Creating a Network With Reticulum
 =============================================
 To create a network, you will need to specify one or more *interfaces* for
 Reticulum to use. This is done in the Reticulum configuration file, which by
-default is located at ``~/.reticulum/config``. You can edit this file by hand,
-or use the interactive ``rnsconfig`` utility. 
+default is located at ``~/.reticulum/config``. You can get an example
+configuration file with all options via ``rnsd --exampleconfig``.
 
 When Reticulum is started for the first time, it will create a default
 configuration file, with one active interface. This default interface uses
@@ -120,29 +203,29 @@ and :ref:`Interfaces<interfaces-main>` chapters of this manual.
 Connecting Reticulum Instances Over the Internet
 ================================================
 Reticulum currently offers two interfaces suitable for connecting instances over the Internet: :ref:`TCP<interfaces-tcps>`
-and :ref:`I2P<interfaces-i2p>`. Each interface offers a different set of features, and Reticulum 
-users should carefully choose the interface which best suites their needs. 
+and :ref:`I2P<interfaces-i2p>`. Each interface offers a different set of features, and Reticulum
+users should carefully choose the interface which best suites their needs.
 
 The ``TCPServerInterface`` allows users to host an instance accessible over TCP/IP. This
 method is generally faster, lower latency, and more energy efficient than using ``I2PInterface``,
 however it also leaks more data about the server host.
 
 TCP connections reveal the IP address of both your instance and the server to anyone who can
-inspect the connection. Someone could use this information to determine your location or identity. Adversaries 
+inspect the connection. Someone could use this information to determine your location or identity. Adversaries
 inspecting your packets may be able to record packet metadata like time of transmission and packet size.
 Even though Reticulum encrypts traffic, TCP does not, so an adversary may be able to use
 packet inspection to learn that a system is running Reticulum, and what other IP addresses connect to it.
 Hosting a publicly reachable instance over TCP also requires a publicly reachable IP address,
 which most Internet connections don't offer anymore.
 
-The ``I2PInterface`` routes messages through the `Invisible Internet Protocol 
+The ``I2PInterface`` routes messages through the `Invisible Internet Protocol
 (I2P) <https://geti2p.net/en/>`_. To use this interface, users must also run an I2P daemon in
-parallel to ``rnsd``. For always-on I2P nodes it is recommended to use `i2pd <https://i2pd.website/>`_. 
+parallel to ``rnsd``. For always-on I2P nodes it is recommended to use `i2pd <https://i2pd.website/>`_.
 
-By default, I2P will encrypt and mix all traffic sent over the Internet, and 
-hide both the sender and receiver Reticulum instance IP addresses. Running an I2P node 
+By default, I2P will encrypt and mix all traffic sent over the Internet, and
+hide both the sender and receiver Reticulum instance IP addresses. Running an I2P node
 will also relay other I2P user's encrypted packets, which will use extra
-bandwidth and compute power, but also makes timing attacks and other forms of 
+bandwidth and compute power, but also makes timing attacks and other forms of
 deep-packet-inspection much more difficult.
 
 I2P also allows users to host globally available Reticulum instances from non-public IP's and behind firewalls and NAT.
@@ -160,30 +243,37 @@ by adding one of the following interfaces to your ``.reticulum/config`` file:
 
 .. code::
 
-  # TCP/IP interface to the Dublin hub
-  [[RNS Testnet Dublin]]
+  # TCP/IP interface to the RNS Amsterdam Hub
+  [[RNS Testnet Amsterdam]]
     type = TCPClientInterface
     enabled = yes
-    target_host = dublin.connect.reticulum.network
+    target_host = amsterdam.connect.reticulum.network
     target_port = 4965
 
-  # TCP/IP interface to the Frankfurt hub
-  [[RNS Testnet Dublin]]
+  # TCP/IP interface to the BetweenTheBorders Hub (community-provided)
+  [[RNS Testnet BetweenTheBorders]]
     type = TCPClientInterface
     enabled = yes
-    target_host = frankfurt.connect.reticulum.network
-    target_port = 5377
+    target_host = reticulum.betweentheborders.com
+    target_port = 4242
 
-  # Interface to I2P hub A
-  [[RNS Testnet I2P Hub A]]
+  # Interface to Testnet I2P Hub
+  [[RNS Testnet I2P Hub]]
     type = I2PInterface
     enabled = yes
-    peers = uxg5kubabakh3jtnvsipingbr5574dle7bubvip7llfvwx2tgrua.b32.i2p
+    peers = g3br23bvx3lq5uddcsjii74xgmn6y5q325ovrkq2zw2wbzbqgbuq.b32.i2p
 
 Many other Reticulum instances are connecting to this testnet, and you can also join it
 via other entry points if you know them. There is absolutely no control over the network
 topography, usage or what types of instances connect. It will also occasionally be used
 to test various failure scenarios, and there are no availability or service guarantees.
+Expect weird things to happen on this network, as people experiment and try out things.
+
+It probably goes without saying, but *don't use the testnet entry-points as 
+hardcoded or default interfaces in any applications you ship to users*. When
+shipping applications, the best practice is to provide your own default
+connectivity solutions, if needed and applicable, or in most cases, simply
+leave it up to the user which networks to connect to, and how.
 
 
 Adding Radio Interfaces
@@ -205,7 +295,7 @@ chapter for a guide. If you prefer purchasing a ready-made unit, you can refer t
 refer to these additional external resources:
 
 * `How To Make Your Own RNodes <https://unsigned.io/how-to-make-your-own-rnodes/>`_
-* `Installing RNode Firmware on Compatible LoRa Devices <https://unsigned.io/installing-rnode-firmware-on-t-beam-and-lora32-devices/>`_
+* `Installing RNode Firmware on Compatible LoRa Devices <https://unsigned.io/installing-rnode-firmware-on-supported-devices/>`_
 * `Private, Secure and Uncensorable Messaging Over a LoRa Mesh <https://unsigned.io/private-messaging-over-lora/>`_
 * `RNode Firmware <https://github.com/markqvist/RNode_Firmware/>`_
 
@@ -222,7 +312,7 @@ started is to install the latest release of Reticulum via pip:
 
 .. code::
 
-   pip3 install rns
+   pip install rns
 
 The above command will install Reticulum and dependencies, and you will be
 ready to import and use RNS in your own programs. The next step will most
@@ -232,7 +322,7 @@ For extended functionality, you can install optional dependencies:
 
 .. code::
 
-   pip3 install pyserial netifaces
+   pip install pyserial
 
 
 Further information can be found in the :ref:`API Reference<api-main>`.
@@ -247,7 +337,7 @@ don't use pip, but try this recipe:
 .. code::
 
     # Install dependencies
-    pip3 install cryptography pyserial netifaces
+    pip install cryptography pyserial
 
     # Clone repository
     git clone https://github.com/markqvist/Reticulum.git
@@ -257,25 +347,25 @@ don't use pip, but try this recipe:
     ln -s ../RNS ./Examples/
 
     # Run an example
-    python3 Examples/Echo.py -s
+    python Examples/Echo.py -s
 
     # Unless you've manually created a config file, Reticulum will do so now,
     # and immediately exit. Make any necessary changes to the file:
     nano ~/.reticulum/config
 
     # ... and launch the example again.
-    python3 Examples/Echo.py -s
+    python Examples/Echo.py -s
 
     # You can now repeat the process on another computer,
     # and run the same example with -h to get command line options.
-    python3 Examples/Echo.py -h
+    python Examples/Echo.py -h
 
     # Run the example in client mode to "ping" the server.
     # Replace the hash below with the actual destination hash of your server.
-    python3 Examples/Echo.py 3e12fc71692f8ec47bc5
+    python Examples/Echo.py 174a64852a75682259ad8b921b8bf416
 
     # Have a look at another example
-    python3 Examples/Filetransfer.py -h
+    python Examples/Filetransfer.py -h
 
 When you have experimented with the basic examples, it's time to go read the
 :ref:`Understanding Reticulum<understanding-main>` chapter. Before submitting
@@ -284,10 +374,84 @@ the `disucssion forum on GitHub <https://github.com/markqvist/Reticulum/discussi
 or ask one of the developers or maintainers for a good place to start.
 
 
-Reticulum on ARM64
+Platform-Specific Install Notes
 ==============================================
+
+Some platforms require a slightly different installation procedure, or have
+various quirks that are worth being aware of. These are listed here.
+
+Android
+^^^^^^^^^^^^^^^^^^^^^^^^
+Reticulum can be used on Android in different ways. The easiest way to get
+started is using an app like `Sideband <https://unsigned.io/sideband>`_.
+
+For more control and features, you can use Reticulum and related programs via
+the `Termux app <https://termux.com/>`_, at the time of writing available on
+`F-droid <https://f-droid.org>`_.
+
+Termux is a terminal emulator and Linux environment for Android based devices,
+which includes the ability to use many different programs and libraries,
+including Reticulum.
+
+To use Reticulum within the Termux environment, you will need to install
+``python`` and the ``python-cryptography`` library using ``pkg``, the package-manager
+build into Termux. After that, you can use ``pip`` to install Reticulum.
+
+From within Termux, execute the following:
+
+.. code::
+
+    # First, make sure indexes and packages are up to date.
+    pkg update
+    pkg upgrade
+
+    # Then install python and the cryptography library.
+    pkg install python python-cryptography
+
+    # Make sure pip is up to date, and install the wheel module.
+    pip install wheel pip --upgrade
+
+    # Install Reticulum
+    pip install rns
+
+If for some reason the ``python-cryptography`` package is not available for
+your platform via the Termux package manager, you can attempt to build it
+locally on your device using the following command:
+
+.. code::
+
+    # First, make sure indexes and packages are up to date.
+    pkg update
+    pkg upgrade
+
+    # Then install dependencies for the cryptography library.
+    pkg install python build-essential openssl libffi rust
+
+    # Make sure pip is up to date, and install the wheel module.
+    pip install wheel pip --upgrade
+
+    # To allow the installer to build the cryptography module,
+    # we need to let it know what platform we are compiling for:
+    export CARGO_BUILD_TARGET="aarch64-linux-android"
+
+    # Start the install process for the cryptography module.
+    # Depending on your device, this can take several minutes,
+    # since the module must be compiled locally on your device.
+    pip install cryptography
+
+    # If the above installation succeeds, you can now install
+    # Reticulum and any related software
+    pip install rns
+
+It is also possible to include Reticulum in apps compiled and distributed as
+Android APKs. A detailed tutorial and example source code will be included
+here at a later point. Until then you can use the `Sideband source code <https://github.com/markqvist/sideband>`_ as an example and starting point.
+
+
+ARM64
+^^^^^^^^^^^^^^^^^^^^^^^^
 On some architectures, including ARM64, not all dependencies have precompiled
-binaries. On such systems, you will need to install ``python3-dev`` before
+binaries. On such systems, you may need to install ``python3-dev`` before
 installing Reticulum or programs that depend on Reticulum.
 
 .. code::
@@ -300,62 +464,90 @@ installing Reticulum or programs that depend on Reticulum.
    python3 -m pip install rns
 
 
-Reticulum on Android
-==============================================
-Reticulum can be used on Android in different ways. The easiest way to get
-started is using an app like `Sideband <https://unsigned.io/sideband>`_.
+Raspberry Pi
+^^^^^^^^^^^^^^^^^^^^^^^^^
+It is currently recommended to use a 64-bit version of the Raspberry Pi OS
+if you want to run Reticulum on Raspberry Pi computers, since 32-bit versions
+don't always have packages available for some dependencies.
 
-For more control and features, you can use Reticulum and related programs via
-the `Termux app <https://termux.com/>`_, at the time of writing available on
-`F-droid <https://f-droid.org>`_.
+While it is possible to install and run Reticulum on 32-bit Rasperry Pi OSes,
+it will require manually configuring and installing some packages, and is not
+detailed in this manual.
 
-Termux is a terminal emulator and Linux environment for Android based devices,
-which includes the ability to use many different programs and libraries,
-including Reticulum.
 
-Since the Python cryptography.io module does not offer pre-built wheels for
-Android, the standard one-line install of Reticulum does not work on Android,
-and a few extra commands are required.
-
-From within Termux, execute the following:
+Debian Bookworm
+^^^^^^^^^^^^^^^^^^^^^^^^
+On versions of Debian released after April 2023, it is no longer possible by default
+to use ``pip`` to install packages onto your system. Unfortunately, you will need to
+use the replacement ``pipx`` command instead, which places installed packages in an
+isolated environment. This should not negatively affect Reticulum, but will not work
+for including and using Reticulum in your own scripts and programs.
 
 .. code::
 
-    # First, make sure indexes and packages are up to date.
-    pkg update
-    pkg upgrade
+    # Install pipx
+    sudo apt install pipx
 
-    # Then install dependencies for the cryptography library.
-    pkg install python build-essential openssl libffi rust
+    # Make installed programs available on the command line
+    pipx ensurepath
 
-    # Make sure pip is up to date, and install the wheel module.
-    pip3 install wheel pip --upgrade
+    # Install Reticulum
+    pipx install rns
 
-    # To allow the installer to build the cryptography module,
-    # we need to let it know what platform we are compiling for:
-    export CARGO_BUILD_TARGET="aarch64-linux-android"
+Alternatively, you can restore normal behaviour to ``pip`` by creating or editing
+the configuration file located at ``~/.config/pip/pip.conf``, and adding the
+following section:
 
-    # Start the install process for the cryptography module.
-    # Depending on your device, this can take several minutes,
-    # since the module must be compiled locally on your device.
-    pip3 install cryptography
+.. code:: text
 
-    # If the above installation succeeds, you can now install
-    # Reticulum and any related software
-    pip3 install rns
+    [global]
+    break-system-packages = true
 
-It is also possible to include Reticulum in apps compiled and distributed as
-Android APKs. A detailed tutorial and example source code will be included
-here at a later point. Until then you can use the `Sideband source code <https://github.com/markqvist/sideband>`_ as an example and startig point.
+Please note that the "break-system-packages" directive is a somewhat misleading choice
+of words. Setting it will of course not break any system packages, but will simply
+allow installing ``pip`` packages user- and system-wide. While this *could* in rare
+cases lead to version conflicts, it does not generally pose any problems.
+
+
+Ubuntu Lunar
+^^^^^^^^^^^^^^^^^^^^^^^^
+On versions of Ubuntu released after April 2023, it is no longer possible by default
+to use ``pip`` to install packages onto your system. Unfortunately, you will need to
+use the replacement ``pipx`` command instead, which places installed packages in an
+isolated environment. This should not negatively affect Reticulum, but will not work
+for including and using Reticulum in your own scripts and programs.
+
+.. code::
+
+    # Install pipx
+    sudo apt install pipx
+
+    # Make installed programs available on the command line
+    pipx ensurepath
+
+    # Install Reticulum
+    pipx install rns
+
+Alternatively, you can restore normal behaviour to ``pip`` by creating or editing
+the configuration file located at ``~/.config/pip/pip.conf``, and adding the
+following section:
+
+.. code:: text
+
+    [global]
+    break-system-packages = true
+
+Please note that the "break-system-packages" directive is a somewhat misleading choice
+of words. Setting it will of course not break any system packages, but will simply
+allow installing ``pip`` packages user- and system-wide. While this _could_ in rare
+cases lead to version conflicts, it does not generally pose any problems.
 
 Pure-Python Reticulum
 ==============================================
 In some rare cases, and on more obscure system types, it is not possible to
-install one or more dependencies
-
-On more unusual systems, and in some rare cases, it might not be possible to
-install or even compile one or more of the above modules. In such situations,
-you can use the ``rnspure`` package instead of the ``rns`` package. The ``rnspure``
+install one or more dependencies. In such situations,
+you can use the ``rnspure`` package instead of the ``rns`` package, or use ``pip``
+with the ``--no-dependencies`` command-line option. The ``rnspure``
 package requires no external dependencies for installation. Please note that the
 actual contents of the ``rns`` and ``rnspure`` packages are *completely identical*.
 The only difference is that the ``rnspure`` package lists no dependencies required

docs/manual/_sources/hardware.rst.txt

@@ -24,11 +24,20 @@ starting from scratch.
 This chapter will outline a few different sensible starting paths to get
 real-world functional wireless communications up and running with minimal cost
 and effort. Two fundamental devices categories will be covered, *RNodes* and
-*WiFi-based radios*.
+*WiFi-based radios*. Additionally, other common options will be briefly described.
+
+Knowing how to employ just a few different types of hardware will make it possible
+to build a wide range of useful networks with little effort.
+
+Combining Hardware Types
+========================
+
+It is useful to combine different link and hardware types when designing and
+building a network. One useful design pattern is to employ high-capacity point-to-point
+links based on WiFi or millimeter-wave radios (with high-gain directional antennas)
+for the network backbone, and using LoRa-based RNodes for covering large areas with
+connectivity for client devices.
 
-While there are many other device categories that are useful in building Reticulum
-networks, knowing how to employ just these two will make it possible to build
-a wide range of useful networks with little effort.
 
 .. _rnode-main:
 
@@ -81,77 +90,151 @@ Supported Boards
 To create one or more RNodes, you will need to obtain supported development
 boards. The following boards are supported by the auto-installer.
 
-LilyGO LoRa32 v2.1
-""""""""""""""""""
-.. image:: graphics/board_t3v21.png
-    :width: 46%
+------------
+
+.. image:: graphics/board_tbeam_supreme.png
+    :width: 75%
     :align: center
 
-- **Supported Firmware Lines** v1.x & v2.x
-- **Transceiver IC** Semtech SX1276
-- **Device Platform** ESP32
-- **Manufacturer** `LilyGO <https://lilygo.cn>`_
-
-
-LilyGO LoRa32 v2.0
-""""""""""""""""""
-.. image:: graphics/board_t3v20.png
-    :width: 46%
-    :align: center
-
-- **Supported Firmware Lines** v1.x & v2.x
-- **Transceiver IC** Semtech SX1276
-- **Device Platform** ESP32
-- **Manufacturer** `LilyGO <https://lilygo.cn>`_
-
-
-LilyGO T-Beam
+LilyGO T-Beam Supreme
 """""""""""""
+- **Transceiver IC** Semtech SX1262, SX1268
+- **Device Platform** ESP32
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
+
+------------
+
 .. image:: graphics/board_tbeam.png
     :width: 75%
     :align: center
 
-- **Supported Firmware Lines** v1.x & v2.x
-- **Transceiver IC** Semtech SX1276
+LilyGO T-Beam
+"""""""""""""
+- **Transceiver IC** Semtech SX1262, SX1268, SX1276 and SX1278
 - **Device Platform** ESP32
 - **Manufacturer** `LilyGO <https://lilygo.cn>`_
 
+------------
 
-Heltec LoRa32 v2.0
-""""""""""""""""""
-.. image:: graphics/board_heltec32.png
-    :width: 58%
+.. image:: graphics/board_t3s3.png
+    :width: 50%
     :align: center
 
-- **Supported Firmware Lines** v1.x & v2.x
-- **Transceiver IC** Semtech SX1276
+LilyGO T3S3
+"""""""""""
+- **Transceiver IC** Semtech SX1262, SX1268, SX1276 and SX1278
 - **Device Platform** ESP32
-- **Manufacturer** `Heltec Automation <https://heltec.org>`_
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
 
+------------
+
+.. image:: graphics/board_rak4631.png
+    :width: 45%
+    :align: center
+
+RAK4631-based Boards
+""""""""""""""""""""
+- **Transceiver IC** Semtech SX1262, SX1268
+- **Device Platform** nRF52
+- **Manufacturer** `RAK Wireless <https://www.rakwireless.com>`_
+
+------------
+
+.. image:: graphics/board_rnodev2.png
+    :width: 68%
+    :align: center
 
 Unsigned RNode v2.x
 """""""""""""""""""
-.. image:: graphics/board_rnodev2.png
-    :width: 58%
-    :align: center
-
-- **Supported Firmware Lines** v1.x & v2.x
-- **Transceiver IC** Semtech SX1276
+- **Transceiver IC** Semtech SX1276 and SX1278
 - **Device Platform** ESP32
 - **Manufacturer** `unsigned.io <https://unsigned.io>`_
 
+------------
+
+.. image:: graphics/board_t3v21.png
+    :width: 46%
+    :align: center
+
+LilyGO LoRa32 v2.1
+""""""""""""""""""
+- **Transceiver IC** Semtech SX1276 and SX1278
+- **Device Platform** ESP32
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
+
+------------
+
+.. image:: graphics/board_t3v20.png
+    :width: 46%
+    :align: center
+
+LilyGO LoRa32 v2.0
+""""""""""""""""""
+- **Transceiver IC** Semtech SX1276 and SX1278
+- **Device Platform** ESP32
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
+
+------------
+
+.. image:: graphics/board_t3v10.png
+    :width: 46%
+    :align: center
+
+LilyGO LoRa32 v1.0
+""""""""""""""""""
+- **Transceiver IC** Semtech SX1276 and SX1278
+- **Device Platform** ESP32
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
+
+------------
+
+.. image:: graphics/board_tdeck.png
+    :width: 45%
+    :align: center
+
+LilyGO T-Deck
+"""""""""""""
+- **Transceiver IC** Semtech SX1262, SX1268
+- **Device Platform** ESP32
+- **Manufacturer** `LilyGO <https://lilygo.cn>`_
+
+------------
+
+.. image:: graphics/board_heltec32v30.png
+    :width: 58%
+    :align: center
+
+Heltec LoRa32 v3.0
+""""""""""""""""""
+- **Transceiver IC** Semtech SX1262 and SX1268
+- **Device Platform** ESP32
+- **Manufacturer** `Heltec Automation <https://heltec.org>`_
+
+------------
+
+.. image:: graphics/board_heltec32v20.png
+    :width: 58%
+    :align: center
+
+Heltec LoRa32 v2.0
+""""""""""""""""""
+- **Transceiver IC** Semtech SX1276 and SX1278
+- **Device Platform** ESP32
+- **Manufacturer** `Heltec Automation <https://heltec.org>`_
+
+------------
 
-Unsigned RNode v1.x
-"""""""""""""""""""
 .. image:: graphics/board_rnode.png
     :width: 50%
     :align: center
 
-- **Supported Firmware Lines** v1.x
-- **Transceiver IC** Semtech SX1276
+Unsigned RNode v1.x
+"""""""""""""""""""
+- **Transceiver IC** Semtech SX1276 and SX1278
 - **Device Platform** AVR ATmega1284p
 - **Manufacturer** `unsigned.io <https://unsigned.io>`_
 
+------------
 
 .. _rnode-installation:
 
@@ -160,12 +243,13 @@ Installation
 
 Once you have obtained compatible boards, you can install the `RNode Firmware <https://github.com/markqvist/RNode_Firmware>`_
 using the `RNode Configuration Utility <https://github.com/markqvist/rnodeconfigutil>`_.
-Make sure that ``Python3`` and ``pip`` is installed on your system, and then install
-the config utility with ``pip``:
+If you have installed Reticulum on your system, the ``rnodeconf`` program will already be
+available. If not, make sure that ``Python3`` and ``pip`` is installed on your system, and
+then install Reticulum with with ``pip``:
 
 .. code::
 
-   pip3 install rnodeconf
+   pip install rns
 
 Once installation has completed, it is time to start installing the firmware on your
 devices. Run ``rnodeconf`` in auto-install mode like so:
@@ -176,12 +260,7 @@ devices. Run ``rnodeconf`` in auto-install mode like so:
 
 The utility will guide you through the installation process by asking a series of
 questions about your hardware. Simply follow the guide, and the utility will
-auto-install and configure your devices
-
-**Important Note!** It is currently recommended to use the v1.x line of the RNode firmware,
-even though the v2.x line is available for early testing. The v2.x line should still be
-considered an experimental pre-release. Only use the v2.x firmware line if you want to test
-out the absolutely newest version, and don't care about stability.
+auto-install and configure your devices.
 
 .. _rnode-usage:
 
@@ -189,17 +268,8 @@ Usage with Reticulum
 ^^^^^^^^^^^^^^^^^^^^
 When the devices have been installed and provisioned, you can use them with Reticulum
 by adding the :ref:`relevant interface section<interfaces-rnode>` to the configuration
-file of Reticulum. For v1.x firmwares, you will have to specify all interface parameters,
-such as serial port and on-air parameters. For v2.x firmwares, you just need to specify
-the Connection ID of the RNode, and Reticulum will automatically locate and connect to the
-RNode, using the parameters stored in the RNode itself.
-
-.. _rnode-suppliers:
-
-Suppliers
-^^^^^^^^^
-Get in touch if you want to have your RNode supplier listed here, or if you want help to
-get started with producing RNodes.
+file of Reticulum. In the configuraion you can specify all interface parameters,
+such as serial port and on-air parameters.
 
 
 WiFi-based Hardware
@@ -235,11 +305,31 @@ that is relatively cheap while providing long range and high capacity for Reticu
 networks. As in all other cases, it is also possible for Reticulum to co-exist with IP
 networks running concurrently on such devices.
 
-Combining Hardware Types
-========================
+Ethernet-based Hardware
+=======================
 
-It is useful to combine different link and hardware types when designing and
-building a network. One useful design pattern is to employ high-capacity point-to-point
-links based on WiFi or millimeter-wave radios (with high-gain directional antennas)
-for the network backbone, and using LoRa-based RNodes for covering large areas with
-connectivity for client devices.
+Reticulum can run over any kind of hardware that can provide a switched Ethernet-based
+medium. This means that anything from a plain Ethernet switch, to fiber-optic systems,
+to data radios with Ethernet interfaces can be used by Reticulum.
+
+The Ethernet medium does not need to have any IP infrastructure such as DHCP servers
+or routing set up, but in case such infrastructure does exist, Reticulum will simply
+co-exist with.
+
+To use Reticulum over Ethernet-based mediums, it is generally enough to use the included
+:ref:`AutoInterface<interfaces-auto>`. This interface also works over any kind of
+virtual networking adapter, such as ``tun`` and ``tap`` devices in Linux.
+
+Serial Lines & Devices
+======================
+
+Using Reticulum over any kind of raw serial line is also possible with the
+:ref:`SerialInterface<interfaces-serial>`. This interface type is also useful for
+using Reticulum over communications hardware that provides a serial port interface.
+
+Packet Radio Modems
+===================
+
+Any packet radio modem that provides a standard KISS interface over USB, serial or TCP
+can be used with Reticulum. This includes virtual software modems such as
+`FreeDV TNC <https://github.com/xssfox/freedv-tnc>`_ and `Dire Wolf <https://github.com/wb2osz/direwolf>`_.

docs/manual/_sources/index.rst.txt

@@ -1,11 +1,16 @@
 ******************************
 Reticulum Network Stack Manual
 ******************************
+
 This manual aims to provide you with all the information you need to
 understand Reticulum, build networks or develop programs using it, or
 to participate in the development of Reticulum itself.
 
-.. only:: html
+.. only:: builder_html
+
+   This manual is also available in `PDF <https://github.com/markqvist/Reticulum/releases/latest/download/Reticulum.Manual.pdf>`_ and `EPUB <https://github.com/markqvist/Reticulum/releases/latest/download/Reticulum.Manual.epub>`_ formats.
+
+.. only:: builder_html
 
    Table Of Contents
    =================

docs/manual/_sources/interfaces.rst.txt

@@ -1,9 +1,9 @@
 
 .. _interfaces-main:
 
-********************
-Supported Interfaces
-********************
+**********************
+Configuring Interfaces
+**********************
 
 Reticulum supports using many kinds of devices as networking interfaces, and
 allows you to mix and match them in any way you choose. The number of distinct
@@ -33,9 +33,20 @@ system, which should be enabled by default in almost all OSes.
 
 .. code::
 
-  # This example demonstrates a TCP server interface.
-  # It will listen for incoming connections on the
-  # specified IP address and port number.
+  # This example demonstrates a bare-minimum setup
+  # of an Auto Interface. It will allow communica-
+  # tion with all other reachable devices on all
+  # usable physical ethernet-based devices that
+  # are available on the system.
+
+  [[Default Interface]]
+    type = AutoInterface
+    interface_enabled = True
+
+  # This example demonstrates an more specifically
+  # configured Auto Interface, that only uses spe-
+  # cific physical interfaces, and has a number of
+  # other configuration options set.
   
   [[Default Interface]]
     type = AutoInterface
@@ -47,6 +58,12 @@ system, which should be enabled by default in almost all OSes.
 
     group_id = reticulum
 
+    # You can also choose the multicast address type:
+    # temporary (default, Temporary Multicast Address)
+    # or permanent (Permanent Multicast Address)
+
+    multicast_address_type = permanent
+
     # You can also select specifically which
     # kernel networking devices to use.
 
@@ -341,6 +358,23 @@ can be used, and offers full control over LoRa parameters.
     # Serial port for the device
     port = /dev/ttyUSB0
 
+    # It is also possible to use BLE devices
+    # instead of wired serial ports. The
+    # target RNode must be paired with the
+    # host device before connecting. BLE
+    # devices can be connected by name,
+    # BLE MAC address or by any available.
+    
+    # Connect to specific device by name
+    # port = ble://RNode 3B87
+
+    # Or by BLE MAC address
+    # port = ble://F4:12:73:29:4E:89
+
+    # Or connect to the first available,
+    # paired device
+    # port = ble://
+
     # Set frequency to 867.2 MHz
     frequency = 867200000
 
@@ -365,6 +399,7 @@ can be used, and offers full control over LoRa parameters.
     # out identification on the channel with
     # a set interval by configuring the
     # following two parameters.
+    
     # id_callsign = MYCALL-0
     # id_interval = 600
 
@@ -372,7 +407,136 @@ can be used, and offers full control over LoRa parameters.
     # with low amounts of RAM, using packet
     # flow control can be useful. By default
     # it is disabled.
-    flow_control = False
+    
+    # flow_control = False
+
+    # It is possible to limit the airtime
+    # utilisation of an RNode by using the
+    # following two configuration options.
+    # The short-term limit is applied in a
+    # window of approximately 15 seconds,
+    # and the long-term limit is enforced
+    # over a rolling 60 minute window. Both
+    # options are specified in percent.
+    
+    # airtime_limit_long = 1.5
+    # airtime_limit_short = 33
+
+
+.. _interfaces-rnode-multi:
+
+RNode Multi Interface
+=====================
+
+For RNodes that support multiple LoRa transceivers, the RNode
+Multi interface can be used to configure sub-interfaces individually.
+
+.. code::
+
+  # Here's an example of how to add an RNode Multi interface
+  # using the RNode LoRa transceiver.
+
+  [[RNode Multi Interface]]
+  type = RNodeMultiInterface
+
+  # Enable interface if you want to use it!
+  interface_enabled = True
+
+  # Serial port for the device
+  port = /dev/ttyACM0
+
+  # You can configure the RNode to send
+  # out identification on the channel with
+  # a set interval by configuring the
+  # following two parameters.
+
+  # id_callsign = MYCALL-0
+  # id_interval = 600
+
+    # A subinterface
+    [[[HIGHDATARATE]]]
+    # Subinterfaces can be enabled and disabled in of themselves
+    interface_enabled = True
+
+    # Set frequency to 2.4GHz
+    frequency = 2400000000
+
+    # Set LoRa bandwidth to 1625 KHz
+    bandwidth = 1625000
+
+    # Set TX power to 0 dBm (0.12 mW)
+    txpower = 0
+
+    # The virtual port, only the manufacturer
+    # or the person who wrote the board config
+    # can tell you what it will be for which
+    # physical hardware interface
+    vport = 1
+
+    # Select spreading factor 5. Valid
+    # range is 5 through 12, with 5 
+    # being the fastest and 12 having
+    # the longest range.
+    spreadingfactor = 5
+
+    # Select coding rate 5. Valid range
+    # is 5 throough 8, with 5 being the
+    # fastest, and 8 the longest range.
+    codingrate = 5
+
+    # It is possible to limit the airtime
+    # utilisation of an RNode by using the
+    # following two configuration options.
+    # The short-term limit is applied in a
+    # window of approximately 15 seconds,
+    # and the long-term limit is enforced
+    # over a rolling 60 minute window. Both
+    # options are specified in percent.
+
+    # airtime_limit_long = 100
+    # airtime_limit_short = 100
+
+    [[[LOWDATARATE]]]
+    # Subinterfaces can be enabled and disabled in of themselves
+    interface_enabled = True
+
+    # Set frequency to 865.6 MHz
+    frequency = 865600000
+
+    # The virtual port, only the manufacturer
+    # or the person who wrote the board config
+    # can tell you what it will be for which
+    # physical hardware interface
+    vport = 0
+
+    # Set LoRa bandwidth to 125 KHz
+    bandwidth = 125000
+
+    # Set TX power to 0 dBm (0.12 mW)
+    txpower = 0
+
+    # Select spreading factor 7. Valid
+    # range is 5 through 12, with 5 
+    # being the fastest and 12 having
+    # the longest range.
+    spreadingfactor = 7
+
+    # Select coding rate 5. Valid range
+    # is 5 throough 8, with 5 being the
+    # fastest, and 8 the longest range.
+    codingrate = 5
+
+    # It is possible to limit the airtime
+    # utilisation of an RNode by using the
+    # following two configuration options.
+    # The short-term limit is applied in a
+    # window of approximately 15 seconds,
+    # and the long-term limit is enforced
+    # over a rolling 60 minute window. Both
+    # options are specified in percent.
+
+    # airtime_limit_long = 100
+    # airtime_limit_short = 100
 
 .. _interfaces-serial:
 
@@ -746,3 +910,74 @@ conserve bandwidth, while very fast networks can support applications that
 need very frequent announces. Reticulum implements these mechanisms to ensure
 that a large span of network types can seamlessly *co-exist* and interconnect.
 
+.. _interfaces-ingress-control:
+
+New Destination Rate Limiting
+=============================
+
+On public interfaces, where anyone may connect and announce new destinations,
+it can be useful to control the rate at which announces for *new*  destinations are
+processed.
+
+If a large influx of announces for newly created or previously unknown destinations
+occur within a short amount of time, Reticulum will place these announces on hold,
+so that announce traffic for known and previously established destinations can
+continue to be processed without interruptions.
+
+After the burst subsides, and an additional waiting period has passed, the held
+announces will be released at a slow rate, until the hold queue is cleared. This
+also means, that should a node decide to connect to a public interface, announce
+a large amount of bogus destinations, and then disconnect, these destination will
+never make it into path tables and waste network bandwidth on retransmitted
+announces.
+
+**It's important to note** that the ingress control works at the level of *individual
+sub-interfaces*. As an example, this means that one client on a :ref:`TCP Server Interface<interfaces-tcps>`
+cannot disrupt processing of incoming announces for other connected clients on the same
+:ref:`TCP Server Interface<interfaces-tcps>`. All other clients on the same interface will still have new announces
+processed without interruption.
+
+By default, Reticulum will handle this automatically, and ingress announce
+control will be enabled on interface where it is sensible to do so. It should
+generally not be neccessary to modify the ingress control configuration,
+but all the parameters are exposed for configuration if needed.
+
+ * | The ``ingress_control`` option tells Reticulum whether or not
+     to enable announce ingress control on the interface. Defaults to
+     ``True``.
+
+ * | The ``ic_new_time`` option configures how long (in seconds) an
+     interface is considered newly spawned. Defaults to ``2*60*60`` seconds. This
+     option is useful on publicly accessible interfaces that spawn new
+     sub-interfaces when a new client connects. 
+
+ * | The ``ic_burst_freq_new`` option sets the maximum announce ingress
+     frequency for newly spawned interfaces. Defaults to ``3.5``
+     announces per second.
+
+ * | The ``ic_burst_freq`` option sets the maximum announce ingress
+     frequency for other interfaces. Defaults to ``12`` announces
+     per second.
+
+     *If an interface exceeds its burst frequency, incoming announces
+     for unknown destinations will be temporarily held in a queue, and
+     not processed until later.*
+
+ * | The ``ic_max_held_announces`` option sets the maximum amount of
+     unique announces that will be held in the queue. Any additional
+     unique announces will be dropped. Defaults to ``256`` announces.
+
+ * | The ``ic_burst_hold`` option sets how much time (in seconds) must
+     pass after the burst frequency drops below its threshold, for the
+     announce burst to be considered cleared. Defaults to ``60``
+     seconds.
+
+ * | The ``ic_burst_penalty`` option sets how much time (in seconds) must
+     pass after the burst is considered cleared, before held announces can
+     start being released from the queue. Defaults to ``5*60``
+     seconds.
+
+ * | The ``ic_held_release_interval`` option sets how much time (in seconds)
+     must pass between releasing each held announce from the queue. Defaults
+     to ``30`` seconds.
+

docs/manual/_sources/networks.rst.txt

@@ -60,7 +60,7 @@ with Reticulum:
 
  * | Reticulum is designed to work reliably in open, trustless environments. This
      means you can use it to create open-access networks, where participants can
-     join and leave in an free and unorganised manner. This property allows an
+     join and leave in a free and unorganised manner. This property allows an
      entirely new, and so far, mostly unexplored class of networked applications,
      where networks, and the information flow within them can form and dissolve
      organically.

docs/manual/_sources/reference.rst.txt

@@ -121,6 +121,76 @@ This chapter lists and explains all classes exposed by the Reticulum Network Sta
 .. autoclass:: RNS.Resource(data, link, advertise=True, auto_compress=True, callback=None, progress_callback=None, timeout=None)
    :members:
 
+.. _api-channel:
+
+.. only:: html
+
+   |start-h3| Channel |end-h3|
+
+.. only:: latex
+
+   Channel
+   -------
+
+.. autoclass:: RNS.Channel.Channel()
+   :members:
+
+.. _api-messsagebase:
+
+.. only:: html
+
+   |start-h3| MessageBase |end-h3|
+
+.. only:: latex
+
+   MessageBase
+   -----------
+
+.. autoclass:: RNS.MessageBase()
+   :members:
+
+.. _api-buffer:
+
+.. only:: html
+
+   |start-h3| Buffer |end-h3|
+
+.. only:: latex
+
+   Buffer
+   ------
+
+.. autoclass:: RNS.Buffer
+   :members:
+
+.. _api-rawchannelreader:
+
+.. only:: html
+
+   |start-h3| RawChannelReader |end-h3|
+
+.. only:: latex
+
+   RawChannelReader
+   ----------------
+
+.. autoclass:: RNS.RawChannelReader
+   :members: __init__, add_ready_callback, remove_ready_callback
+
+.. _api-rawchannelwriter:
+
+.. only:: html
+
+   |start-h3| RawChannelWriter |end-h3|
+
+.. only:: latex
+
+   RawChannelWriter
+   ----------------
+
+.. autoclass:: RNS.RawChannelWriter
+   :members: __init__
+
 .. _api-transport:
 
 .. only:: html

docs/manual/_sources/understanding.rst.txt

@@ -75,7 +75,7 @@ guide the design of Reticulum:
     it can be easily modified and replicated by anyone interested in doing so.
 * **Very low bandwidth requirements**
     Reticulum should be able to function reliably over links with a transmission capacity as low
-    as *500 bits per second*.
+    as *5 bits per second*.
 * **Encryption by default**
     Reticulum must use strong encryption by default for all communication.
 * **Initiator Anonymity**
@@ -107,13 +107,13 @@ guide the design of Reticulum:
 Introduction & Basic Functionality
 ==================================
 
-Reticulum is a networking stack suited for high-latency, low-bandwidth links. Reticulum is at it’s
+Reticulum is a networking stack suited for high-latency, low-bandwidth links. Reticulum is at its
 core a *message oriented* system. It is suited for both local point-to-point or point-to-multipoint
 scenarios where all nodes are within range of each other, as well as scenarios where packets need
 to be transported over multiple hops in a complex network to reach the recipient.
 
 Reticulum does away with the idea of addresses and ports known from IP, TCP and UDP. Instead
-Reticulum uses the singular concept of *destinations*. Any application using Reticulum as it’s
+Reticulum uses the singular concept of *destinations*. Any application using Reticulum as its
 networking stack will need to create one or more destinations to receive data, and know the
 destinations it needs to send data to.
 
@@ -134,10 +134,11 @@ be sufficient, even far into the future.
 By default Reticulum encrypts all data using elliptic curve cryptography and AES. Any packet sent to a
 destination is encrypted with a per-packet derived key. Reticulum can also set up an encrypted
 channel to a destination, called a *Link*. Both data sent over Links and single packets offer
-*Initiator Anonymity*, and links additionally offer *Forward Secrecy* by using an Elliptic Curve
-Diffie Hellman key exchange on Curve25519 to derive per-link ephemeral keys. The multi-hop transport,
-coordination, verification and reliability layers are fully autonomous and also based on elliptic
-curve cryptography.
+*Initiator Anonymity*. Links additionally offer *Forward Secrecy* by default, employing an Elliptic Curve
+Diffie Hellman key exchange on Curve25519 to derive per-link ephemeral keys. Asymmetric, link-less
+packet communication can also provide forward secrecy, with automatic key ratcheting, by enabling
+ratchets on a per-destination basis. The multi-hop transport, coordination, verification and reliability
+layers are fully autonomous and also based on elliptic curve cryptography.
 
 Reticulum also offers symmetric key encryption for group-oriented communications, as well as
 unencrypted packets for local broadcast purposes.
@@ -220,7 +221,7 @@ packet.
 
 In actual use of *single* destination naming, it is advisable not to use any uniquely identifying
 features in aspect naming. Aspect names should be general terms describing what kind of destination
-is represented. The uniquely identifying aspect is always achieved by the appending the public key,
+is represented. The uniquely identifying aspect is always achieved by appending the public key,
 which expands the destination into a uniquely identifiable one. Reticulum does this automatically.
 
 Any destination on a Reticulum network can be addressed and reached simply by knowing its
@@ -239,7 +240,7 @@ To recap, the different destination types should be used in the following situat
 * **Plain**
     When plain-text communication is desirable, for example when broadcasting information, or for local discovery purposes.
 
-To communicate with a *single* destination, you need to know it’s public key. Any method for
+To communicate with a *single* destination, you need to know its public key. Any method for
 obtaining the public key is valid, but Reticulum includes a simple mechanism for making other
 nodes aware of your destinations public key, called the *announce*. It is also possible to request
 an unknown public key from the network, as all transport instances serve as a distributed ledger
@@ -287,7 +288,7 @@ In Reticulum, destinations are allowed to move around the network at will. This
 protocols such as IP, where an address is always expected to stay within the network segment it was assigned in.
 This limitation does not exist in Reticulum, and any destination is *completely portable* over the entire topography
 of the network, and *can even be moved to other Reticulum networks* than the one it was created in, and
-still become reachable. To update it's reachability, a destination simply needs to send an announce on any
+still become reachable. To update its reachability, a destination simply needs to send an announce on any
 networks it is part of. After a short while, it will be globally reachable in the network.
 
 Seeing how *single* destinations are always tied to a private/public key pair leads us to the next topic.
@@ -368,7 +369,7 @@ If it is a *Transport Node*, it should be given the configuration directive ``en
 The Announce Mechanism in Detail
 --------------------------------
 
-When an *announce* for a destination is transmitted by from a Reticulum instance, it will be forwarded by
+When an *announce* for a destination is transmitted by a Reticulum instance, it will be forwarded by
 any transport node receiving it, but according to some specific rules:
 
 
@@ -385,7 +386,7 @@ any transport node receiving it, but according to some specific rules:
     announces is set at 2%, but can be configured on a per-interface basis.
 
 * | If any given interface does not have enough bandwidth available for retransmitting the announce,
-    the announce will be assigned a priority inversely proportional to it's hop count, and be inserted
+    the announce will be assigned a priority inversely proportional to its hop count, and be inserted
     into a queue managed by the interface.
 
 * | When the interface has bandwidth available for processing an announce, it will prioritise announces
@@ -431,7 +432,7 @@ For exchanges of small amounts of information, Reticulum offers the *Packet* API
 
 * | A packet is always created with an associated destination and some payload data. When the packet is sent
     to a *single* destination type, Reticulum will automatically create an ephemeral encryption key, perform
-    an ECDH key exchange with the destinations public key, and encrypt the information.
+    an ECDH key exchange with the destination's public key (or ratchet key, if available), and encrypt the information.
 
 * | It is important to note that this key exchange does not require any network traffic. The sender already
     knows the public key of the destination from an earlier received *announce*, and can thus perform the ECDH
@@ -447,8 +448,8 @@ For exchanges of small amounts of information, Reticulum offers the *Packet* API
 
 * | Once the packet has been received and decrypted by the addressed destination, that destination can opt
     to *prove* its receipt of the packet. It does this by calculating the SHA-256 hash of the received packet,
-    and signing this hash with it's Ed25519 signing key. Transport nodes in the network can then direct this
-    *proof* back to the packets origin, where the signature can be verified against the destinations known
+    and signing this hash with its Ed25519 signing key. Transport nodes in the network can then direct this
+    *proof* back to the packets origin, where the signature can be verified against the destination's known
     public signing key.
 
 * | In case the packet is addressed to a *group* destination type, the packet will be encrypted with the
@@ -465,7 +466,7 @@ For exchanges of larger amounts of data, or when longer sessions of bidirectiona
     forward the packet will take note of this *link request*.
 
 * | Second, if the destination accepts the *link request* , it will send back a packet that proves the
-    authenticity of it’s identity (and the receipt of the link request) to the initiating node. All
+    authenticity of its identity (and the receipt of the link request) to the initiating node. All
     nodes that initially forwarded the packet will also be able to verify this proof, and thus
     accept the validity of the *link* throughout the network.
 
@@ -595,7 +596,7 @@ or less any medium that allows you to send and receive data, which satisfies som
 minimum requirements.
 
 The communication channel must support at least half-duplex operation, and provide an average
-throughput of around 500 bits per second, and supports a physical layer MTU of 500 bytes. The
+throughput of 5 bits per second or greater, and supports a physical layer MTU of 500 bytes. The
 Reticulum stack should be able to run on more or less any hardware that can provide a Python 3.x 
 runtime environment.
 
@@ -693,7 +694,8 @@ Wire Format
     [HEADER 2 bytes] [ADDRESSES 16/32 bytes] [CONTEXT 1 byte] [DATA 0-465 bytes]
 
     * The HEADER field is 2 bytes long.
-      * Byte 1: [IFAC Flag], [Header Type], [Propagation Type], [Destination Type] and [Packet Type]
+      * Byte 1: [IFAC Flag], [Header Type], [Context Flag], [Propagation Type],
+                [Destination Type] and [Packet Type]
       * Byte 2: Number of hops
 
     * Interface Access Code field if the IFAC flag was set.
@@ -725,12 +727,16 @@ Wire Format
     type 2           1  Two byte header, two 16 byte address fields
 
 
+    Context Flag
+    -----------------
+    unset            0  The context flag is used for various types
+    set              1  of signalling, depending on packet context
+
+
     Propagation Types
     -----------------
-    broadcast       00
-    transport       01
-    reserved        10
-    reserved        11
+    broadcast        0
+    transport        1
 
 
     Destination Types
@@ -771,7 +777,7 @@ Wire Format
     |               | |               | |               | |    |
     00000000 00000111 [HASH1, 16 bytes] [CONTEXT, 1 byte] [DATA]
     || | | |    |
-    || | | |    +-- Hops             = 0
+    || | | |    +-- Hops             = 7
     || | | +------- Packet Type      = DATA
     || | +--------- Destination Type = SINGLE
     || +----------- Propagation Type = BROADCAST
@@ -786,7 +792,7 @@ Wire Format
     |               | |             | |               | |               | |    |
     10000000 00000111 [IFAC, N bytes] [HASH1, 16 bytes] [CONTEXT, 1 byte] [DATA]
     || | | |    |
-    || | | |    +-- Hops             = 0
+    || | | |    +-- Hops             = 7
     || | | +------- Packet Type      = DATA
     || | +--------- Destination Type = SINGLE
     || +----------- Propagation Type = BROADCAST
@@ -858,16 +864,18 @@ both on general-purpose CPUs and on microcontrollers. The necessary primitives a
 
 * Ed25519 for signatures
 
-* X22519 for ECDH key exchanges
+* X25519 for ECDH key exchanges
 
 * HKDF for key derivation
 
-* Fernet for encrypted tokens
+* Modified Fernet for encrypted tokens
 
   * AES-128 in CBC mode
 
   * HMAC for message authentication
 
+  * No Version and Timestamp metadata included
+
 * SHA-256
 
 * SHA-512

docs/manual/_sources/using.rst.txt

@@ -5,7 +5,9 @@ Using Reticulum on Your System
 ******************************
 
 Reticulum is not installed as a driver or kernel module, as one might expect
-of a networking stack. Instead, Reticulum is distributed as a Python module.
+of a networking stack. Instead, Reticulum is distributed as a Python module, 
+containing the networking core, and a set of utility and daemon programs.
+
 This means that no special privileges are required to install or use it. It
 is also very light-weight, and easy to transfer to, and install on new systems.
 
@@ -16,8 +18,8 @@ already running.
 In many cases, this approach is sufficient. When any program needs to use
 Reticulum, it is loaded, initialised, interfaces are brought up, and the
 program can now communicate over any Reticulum networks available. If another
-program starts up and also wants access to the same Reticulum network, the
-instance is simply shared. This works for any number of programs running
+program starts up and also wants access to the same Reticulum network, the already
+running instance is simply shared. This works for any number of programs running
 concurrently, and is very easy to use, but depending on your use case, there
 are other options.
 
@@ -97,6 +99,17 @@ configuration file is created. The default configuration looks like this:
   instance_control_port = 37429
 
 
+  # On systems where running instances may not have access
+  # to the same shared Reticulum configuration directory,
+  # it is still possible to allow full interactivity for
+  # running instances, by manually specifying a shared RPC
+  # key. In almost all cases, this option is not needed, but
+  # it can be useful on operating systems such as Android.
+  # The key must be specified as bytes in hexadecimal.
+
+  # rpc_key = e5c032d3ec4e64a6aca9927ba8ab73336780f6d71790
+
+
   # You can configure Reticulum to panic and forcibly close
   # if an unrecoverable interface error occurs, such as the
   # hardware device for an interface disappearing. This is
@@ -106,6 +119,17 @@ configuration file is created. The default configuration looks like this:
   panic_on_interface_error = No
 
 
+  # When Transport is enabled, it is possible to allow the
+  # Transport Instance to respond to probe requests from
+  # the rnprobe utility. This can be a useful tool to test
+  # connectivity. When this option is enabled, the probe
+  # destination will be generated from the Identity of the
+  # Transport Instance, and printed to the log at startup.
+  # Optional, and disabled by default.
+
+  respond_to_probes = No
+
+
   [logging]
   # Valid log levels are 0 through 7:
   #   0: Log only critical information
@@ -143,10 +167,19 @@ configuration file is created. The default configuration looks like this:
 If Reticulum infrastructure already exists locally, you probably don't need to
 change anything, and you may already be connected to a wider network. If not,
 you will probably need to add relevant *interfaces* to the configuration, in
-order to communicate with other systems. It is a good idea to read the comments
-and explanations in the above default config. It will teach you the basic
-concepts you need to understand to configure your network. Once you have done that,
-take a look at the :ref:`Interfaces<interfaces-main>` chapter of this manual.
+order to communicate with other systems.
+
+You can generate a much more verbose configuration example by running the command:
+
+``rnsd --exampleconfig``
+
+The output includes examples for most interface types supported
+by Reticulum, along with additional options and configuration parameters.
+
+It is a good idea to read the comments and explanations in the above default config.
+It will teach you the basic concepts you need to understand to configure your network.
+Once you have done that, take a look at the :ref:`Interfaces<interfaces-main>` chapter
+of this manual.
 
 Included Utility Programs
 -------------------------
@@ -168,28 +201,47 @@ When ``rnsd`` is running, it will keep all configured interfaces open, handle tr
 it is enabled, and allow any other programs to immediately utilise the
 Reticulum network it is configured for.
 
-You can even run multiple instances of rnsd with different configurations on
+You can even run multiple instances of ``rnsd`` with different configurations on
 the same system.
 
-.. code:: text
+**Usage Examples**
 
-  # Install Reticulum
-  pip3 install rns
-
-  # Run rnsd
-  rnsd
+Run ``rnsd``:
 
 .. code:: text
 
-  usage: rnsd [-h] [--config CONFIG] [-v] [-q] [--version]
+  $ rnsd
+
+  [2023-08-18 17:59:56] [Notice] Started rnsd version 0.5.8
+
+Run ``rnsd`` in service mode, ensuring all logging output is sent directly to file:
+
+.. code:: text
+
+  $ rnsd -s
+
+Generate a verbose and detailed configuration example, with explanations of all the
+various configuration options, and interface configuration examples:
+
+.. code:: text
+
+  $ rnsd --exampleconfig
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnsd.py [-h] [--config CONFIG] [-v] [-q] [-s] [--exampleconfig] [--version]
 
   Reticulum Network Stack Daemon
 
-  optional arguments:
+  options:
     -h, --help       show this help message and exit
     --config CONFIG  path to alternative Reticulum config directory
     -v, --verbose
     -q, --quiet
+    -s, --service    rnsd is running as a service and should log to file
+    --exampleconfig  print verbose configuration example to stdout and exit
     --version        show program's version number and exit
 
 You can easily add ``rnsd`` as an always-on service by :ref:`configuring a service<using-systemd>`.
@@ -200,12 +252,14 @@ The rnstatus Utility
 Using the ``rnstatus`` utility, you can view the status of configured Reticulum
 interfaces, similar to the ``ifconfig`` program.
 
+**Usage Examples**
+
+Run ``rnstatus``:
+
 .. code:: text
 
-  # Run rnstatus
-  rnstatus
+  $ rnstatus
 
-  # Example output
   Shared Instance[37428]
      Status  : Up
      Serving : 1 program
@@ -221,7 +275,7 @@ interfaces, similar to the ``ifconfig`` program.
      Traffic : 63.23 KB↑
                80.17 KB↓
 
-  TCPInterface[RNS Testnet Frankfurt/frankfurt.rns.unsigned.io:4965]
+  TCPInterface[RNS Testnet Dublin/dublin.connect.reticulum.network:4965]
      Status  : Up
      Mode    : Full
      Rate    : 10.00 Mbps
@@ -238,52 +292,195 @@ interfaces, similar to the ``ifconfig`` program.
 
   Reticulum Transport Instance <5245a8efe1788c6a1cd36144a270e13b> running
 
+Filter output to only show some interfaces:
+
 .. code:: text
 
-  usage: rnstatus [-h] [--config CONFIG] [--version] [-a] [-v]
+  $ rnstatus rnode
+
+  RNodeInterface[RNode UHF]
+     Status  : Up
+     Mode    : Access Point
+     Rate    : 1.30 kbps
+     Access  : 64-bit IFAC by <…e702c42ba8>
+     Traffic : 8.49 KB↑
+               9.23 KB↓
+
+  Reticulum Transport Instance <5245a8efe1788c6a1cd36144a270e13b> running
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnstatus [-h] [--config CONFIG] [--version] [-a] [-A]
+                  [-l] [-s SORT] [-r] [-j] [-R hash] [-i path]
+                  [-w seconds] [-v] [filter]
 
   Reticulum Network Stack Status
 
-  optional arguments:
-    -h, --help       show this help message and exit
-    --config CONFIG  path to alternative Reticulum config directory
-    --version        show program's version number and exit
-    -a, --all        show all interfaces
+  positional arguments:
+    filter                only display interfaces with names including filter
+
+  options:
+    -h, --help            show this help message and exit
+    --config CONFIG       path to alternative Reticulum config directory
+    --version             show program's version number and exit
+    -a, --all             show all interfaces
+    -A, --announce-stats  show announce stats
+    -l, --link-stats      show link stats
+    -s SORT, --sort SORT  sort interfaces by [rate, traffic, rx, tx, announces, arx, atx, held]
+    -r, --reverse         reverse sorting
+    -j, --json            output in JSON format
+    -R hash               transport identity hash of remote instance to get status from
+    -i path               path to identity used for remote management
+    -w seconds            timeout before giving up on remote queries
     -v, --verbose
 
 
+The rnid Utility
+====================
+
+With the ``rnid`` utility, you can generate, manage and view Reticulum Identities.
+The program can also calculate Destination hashes, and perform encryption and
+decryption of files.
+
+Using ``rnid``, it is possible to asymmetrically encrypt files and information for
+any Reticulum destination hash, and also to create and verify cryptographic signatures.
+
+**Usage Examples**
+
+Generate a new Identity:
+
+.. code:: text
+
+  $ rnid -g ./new_identity
+
+Display Identity key information:
+
+.. code:: text
+
+  $ rnid -i ./new_identity -p
+
+  Loaded Identity <984b74a3f768bef236af4371e6f248cd> from new_id
+  Public Key  : 0f4259fef4521ab75a3409e353fe9073eb10783b4912a6a9937c57bf44a62c1e
+  Private Key : Hidden
+
+Encrypt a file for an LXMF user:
+
+.. code:: text
+
+  $ rnid -i 8dd57a738226809646089335a6b03695 -e my_file.txt
+
+  Recalled Identity <bc7291552be7a58f361522990465165c> for destination <8dd57a738226809646089335a6b03695>
+  Encrypting my_file.txt
+  File my_file.txt encrypted for <bc7291552be7a58f361522990465165c> to my_file.txt.rfe
+
+If the Identity for the destination is not already known, you can fetch it from the network by using the ``-R`` command-line option:
+
+.. code:: text
+
+  $ rnid -R -i 30602def3b3506a28ed33db6f60cc6c9 -e my_file.txt
+
+  Requesting unknown Identity for <30602def3b3506a28ed33db6f60cc6c9>...
+  Received Identity <2b489d06eaf7c543808c76a5332a447d> for destination <30602def3b3506a28ed33db6f60cc6c9> from the network
+  Encrypting my_file.txt
+  File my_file.txt encrypted for <2b489d06eaf7c543808c76a5332a447d> to my_file.txt.rfe
+
+Decrypt a file using the Reticulum Identity it was encrypted for:
+
+.. code:: text
+
+  $ rnid -i ./my_identity -d my_file.txt.rfe
+
+  Loaded Identity <2225fdeecaf6e2db4556c3c2d7637294> from ./my_identity
+  Decrypting ./my_file.txt.rfe...
+  File ./my_file.txt.rfe decrypted with <2225fdeecaf6e2db4556c3c2d7637294> to ./my_file.txt
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnid.py [-h] [--config path] [-i identity] [-g path] [-v] [-q] [-a aspects]
+                 [-H aspects] [-e path] [-d path] [-s path] [-V path] [-r path] [-w path]
+                 [-f] [-R] [-t seconds] [-p] [-P] [--version]
+
+  Reticulum Identity & Encryption Utility
+
+  options:
+    -h, --help            show this help message and exit
+    --config path         path to alternative Reticulum config directory
+    -i identity, --identity identity
+                          hexadecimal Reticulum Destination hash or path to Identity file
+    -g path, --generate path
+                          generate a new Identity
+    -v, --verbose         increase verbosity
+    -q, --quiet           decrease verbosity
+    -a aspects, --announce aspects
+                          announce a destination based on this Identity
+    -H aspects, --hash aspects
+                          show destination hashes for other aspects for this Identity
+    -e path, --encrypt path
+                          encrypt file
+    -d path, --decrypt path
+                          decrypt file
+    -s path, --sign path  sign file
+    -V path, --validate path
+                          validate signature
+    -r path, --read path  input file path
+    -w path, --write path
+                          output file path
+    -f, --force           write output even if it overwrites existing files
+    -R, --request         request unknown Identities from the network
+    -t seconds            identity request timeout before giving up
+    -p, --print-identity  print identity info and exit
+    -P, --print-private   allow displaying private keys
+    --version             show program's version number and exit
+
+
 The rnpath Utility
 ====================
 
 With the ``rnpath`` utility, you can look up and view paths for
 destinations on the Reticulum network.
 
-.. code:: text
+**Usage Examples**
 
-  # Run rnpath
-  rnpath c89b4da064bf66d280f0e4d8abfd9806
-
-  # Example output
-  Path found, destination <c89b4da064bf66d280f0e4d8abfd9806> is 4 hops away via <f53a1c4278e0726bb73fcc623d6ce763> on TCPInterface[Testnet/frankfurt.connect.reticulu.network:4965]
+Resolve path to a destination:
 
 .. code:: text
 
-  usage: rnpath [-h] [--config CONFIG] [--version] [-t] [-r] [-d] [-D] [-w seconds] [-v] [destination]
-  
+  $ rnpath c89b4da064bf66d280f0e4d8abfd9806
+
+  Path found, destination <c89b4da064bf66d280f0e4d8abfd9806> is 4 hops away via <f53a1c4278e0726bb73fcc623d6ce763> on TCPInterface[Testnet/dublin.connect.reticulum.network:4965]
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnpath [-h] [--config CONFIG] [--version] [-t] [-m hops]
+                [-r] [-d] [-D] [-x] [-w seconds] [-R hash] [-i path]
+                [-W seconds] [-j] [-v] [destination]
+
   Reticulum Path Discovery Utility
-  
+
   positional arguments:
     destination           hexadecimal hash of the destination
-  
-  optional arguments:
+
+  options:
     -h, --help            show this help message and exit
     --config CONFIG       path to alternative Reticulum config directory
     --version             show program's version number and exit
     -t, --table           show all known paths
+    -m hops, --max hops   maximum hops to filter path table by
     -r, --rates           show announce rate info
     -d, --drop            remove the path to a destination
     -D, --drop-announces  drop all queued announces
+    -x, --drop-via        drop all paths via specified transport instance
     -w seconds            timeout before giving up
+    -R hash               transport identity hash of remote instance to manage
+    -i path               path to identity used for remote management
+    -W seconds            timeout before giving up on remote queries
+    -j, --json            output in JSON format
     -v, --verbose
 
 
@@ -293,32 +490,72 @@ The rnprobe Utility
 The ``rnprobe`` utility lets you probe a destination for connectivity, similar
 to the ``ping`` program. Please note that probes will only be answered if the
 specified destination is configured to send proofs for received packets. Many
-destinations will not have this option enabled, and will not be probable.
+destinations will not have this option enabled, so most destinations will not
+be probable.
+
+You can enable a probe-reply destination on Reticulum Transport Instances by
+setting the ``respond_to_probes`` configuration directive. Reticulum will then
+print the probe destination to the log on Transport Instance startup.
+
+**Usage Examples**
+
+Probe a destination:
 
 .. code:: text
 
-  # Run rnprobe
-  rnprobe example_utilities.echo.request 2d03725b327348980d570f739a3a5708
+  $ rnprobe rnstransport.probe 2d03725b327348980d570f739a3a5708
 
-  # Example output
   Sent 16 byte probe to <2d03725b327348980d570f739a3a5708>
   Valid reply received from <2d03725b327348980d570f739a3a5708>
   Round-trip time is 38.469 milliseconds over 2 hops
 
+Send a larger probe:
+
 .. code:: text
 
-  usage: rnprobe [-h] [--config CONFIG] [--version] [-v] [full_name] [destination_hash]
+  $ rnprobe rnstransport.probe 2d03725b327348980d570f739a3a5708 -s 256
+
+  Sent 16 byte probe to <2d03725b327348980d570f739a3a5708>
+  Valid reply received from <2d03725b327348980d570f739a3a5708>
+  Round-trip time is 38.781 milliseconds over 2 hops
+
+If the interface that receives the probe replies supports reporting radio
+parameters such as **RSSI** and **SNR**, the ``rnprobe`` utility will print
+these as part of the result as well.
+
+.. code:: text
+
+  $ rnprobe rnstransport.probe e7536ee90bd4a440e130490b87a25124
+  
+  Sent 16 byte probe to <e7536ee90bd4a440e130490b87a25124>
+  Valid reply received from <e7536ee90bd4a440e130490b87a25124>
+  Round-trip time is 1.809 seconds over 1 hop [RSSI -73 dBm] [SNR 12.0 dB]
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnprobe [-h] [--config CONFIG] [-s SIZE] [-n PROBES]
+                 [-t seconds] [-w seconds] [--version] [-v]
+                 [full_name] [destination_hash]
 
   Reticulum Probe Utility
 
   positional arguments:
-    full_name         full destination name in dotted notation
-    destination_hash  hexadecimal hash of the destination
+    full_name             full destination name in dotted notation
+    destination_hash      hexadecimal hash of the destination
 
-  optional arguments:
-    -h, --help        show this help message and exit
-    --config CONFIG   path to alternative Reticulum config directory
-    --version         show program's version number and exit
+  options:
+    -h, --help            show this help message and exit
+    --config CONFIG       path to alternative Reticulum config directory
+    -s SIZE, --size SIZE  size of probe packet payload in bytes
+    -n PROBES, --probes PROBES
+                          number of probes to send
+    -t seconds, --timeout seconds
+                          timeout before giving up
+    -w seconds, --wait seconds
+                          time between each probe
+    --version             show program's version number and exit
     -v, --verbose
 
 
@@ -328,20 +565,40 @@ The rncp Utility
 The ``rncp`` utility is a simple file transfer tool. Using it, you can transfer
 files through Reticulum.
 
+**Usage Examples**
+
+Run rncp on the receiving system, specifying which identities are allowed to send files:
+  
 .. code:: text
 
-  # Run rncp on the receiving system, specifying which identities
-  # are allowed to send files
-  rncp --receive -a 1726dbad538775b5bf9b0ea25a4079c8 -a c50cc4e4f7838b6c31f60ab9032cbc62
+  $ rncp --listen -a 1726dbad538775b5bf9b0ea25a4079c8 -a c50cc4e4f7838b6c31f60ab9032cbc62
 
-  # From another system, copy a file to the receiving system
-  rncp ~/path/to/file.tgz 73cbd378bb0286ed11a707c13447bb1e
-
-You can specify as many allowed senders as needed, or complete disable authentication.
+You can also specify allowed identity hashes (one per line) in the file ~/.rncp/allowed_identities
+and simply running the program in listener mode:
 
 .. code:: text
 
-  usage: rncp [-h] [--config path] [-v] [-q] [-p] [-r] [-b] [-a allowed_hash] [-n] [-w seconds] [--version] [file] [destination]
+  $ rncp --listen
+
+From another system, copy a file to the receiving system:
+
+.. code:: text
+
+  $ rncp ~/path/to/file.tgz 73cbd378bb0286ed11a707c13447bb1e
+
+Or fetch a file from the remote system:
+
+.. code:: text
+
+  $ rncp --fetch ~/path/to/file.tgz 73cbd378bb0286ed11a707c13447bb1e
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rncp [-h] [--config path] [-v] [-q] [-S] [-l] [-F] [-f]
+              [-j path] [-b seconds] [-a allowed_hash] [-n] [-p]
+              [-w seconds] [--version] [file] [destination]
 
   Reticulum File Transfer Utility
 
@@ -349,19 +606,22 @@ You can specify as many allowed senders as needed, or complete disable authentic
     file                  file to be transferred
     destination           hexadecimal hash of the receiver
 
-  optional arguments:
+  options:
     -h, --help            show this help message and exit
     --config path         path to alternative Reticulum config directory
     -v, --verbose         increase verbosity
     -q, --quiet           decrease verbosity
+    -S, --silent          disable transfer progress output
+    -l, --listen          listen for incoming transfer requests
+    -F, --allow-fetch     allow authenticated clients to fetch files
+    -f, --fetch           fetch file from remote listener instead of sending
+    -j path, --jail path  restrict fetch requests to specified path
+    -b seconds            announce interval, 0 to only announce at startup
+    -a allowed_hash       allow this identity
+    -n, --no-auth         accept requests from anyone
     -p, --print-identity  print identity and destination info and exit
-    -r, --receive         wait for incoming files
-    -b, --no-announce     don't announce at program start
-    -a allowed_hash       accept from this identity
-    -n, --no-auth         accept files from anyone
     -w seconds            sender timeout before giving up
     --version             show program's version number and exit
-    -v, --verbose
 
 
 The rnx Utility
@@ -369,32 +629,43 @@ The rnx Utility
 
 The ``rnx`` utility is a basic remote command execution program. It allows you to
 execute commands on remote systems over Reticulum, and to view returned command
-output.
+output. For a fully interactive remote shell solution, be sure to also take a look
+at the `rnsh <https://github.com/acehoss/rnsh>`_ program.
+
+**Usage Examples**
+
+Run rnx on the listening system, specifying which identities are allowed to execute commands:
 
 .. code:: text
 
-  # Run rnx on the listening system, specifying which identities
-  # are allowed to execute commands
-  rncp --listen -a 941bed5e228775e5a8079fc38b1ccf3f -a 1b03013c25f1c2ca068a4f080b844a10
+  $ rnx --listen -a 941bed5e228775e5a8079fc38b1ccf3f -a 1b03013c25f1c2ca068a4f080b844a10
 
-  # From another system, run a command 
-  rnx 7a55144adf826958a9529a3bcf08b149 "cat /proc/cpuinfo"
-
-  # Or enter the interactive mode pseudo-shell
-  rnx 7a55144adf826958a9529a3bcf08b149 -x
-
-  # The default identity file is stored in
-  # ~/.reticulum/identities/rnx, but you can use
-  # another one, which will be created if it does
-  # not already exist
-  rnx 7a55144adf826958a9529a3bcf08b149 -i /path/to/identity -x
-
-You can specify as many allowed senders as needed, or completely disable authentication.
+From another system, run a command on the remote:
 
 .. code:: text
 
-  usage: rnx [-h] [--config path] [-v] [-q] [-p] [-l] [-i identity] [-x] [-b] [-a allowed_hash] [-n] [-N] [-d] [-m] [-w seconds] [-W seconds] [--stdin STDIN] [--stdout STDOUT] [--stderr STDERR] [--version]
-             [destination] [command]
+  $ rnx 7a55144adf826958a9529a3bcf08b149 "cat /proc/cpuinfo"
+
+Or enter the interactive mode pseudo-shell:
+
+.. code:: text
+
+  $ rnx 7a55144adf826958a9529a3bcf08b149 -x
+
+The default identity file is stored in ``~/.reticulum/identities/rnx``, but you can use
+another one, which will be created if it does not already exist 
+
+.. code:: text
+
+  $ rnx 7a55144adf826958a9529a3bcf08b149 -i /path/to/identity -x
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnx [-h] [--config path] [-v] [-q] [-p] [-l] [-i identity] [-x] [-b] [-n] [-N]
+             [-d] [-m] [-a allowed_hash] [-w seconds] [-W seconds] [--stdin STDIN]
+             [--stdout STDOUT] [--stderr STDERR] [--version] [destination] [command]
 
   Reticulum Remote Execution Utility
 
@@ -425,6 +696,109 @@ You can specify as many allowed senders as needed, or completely disable authent
     --version             show program's version number and exit
 
 
+The rnodeconf Utility
+=====================
+
+The ``rnodeconf`` utility allows you to inspect and configure existing :ref:`RNodes<rnode-main>`, and
+to create and provision new :ref:`RNodes<rnode-main>` from any supported hardware devices.
+
+**All Command-Line Options**
+
+.. code:: text
+
+  usage: rnodeconf [-h] [-i] [-a] [-u] [-U] [--fw-version version]
+                   [--fw-url url] [--nocheck] [-e] [-E] [-C]
+                   [--baud-flash baud_flash] [-N] [-T] [-b] [-B] [-p] [-D i]
+                   [--display-addr byte] [--freq Hz] [--bw Hz] [--txp dBm]
+                   [--sf factor] [--cr rate] [--eeprom-backup] [--eeprom-dump]
+                   [--eeprom-wipe] [-P] [--trust-key hexbytes] [--version] [-f]
+                   [-r] [-k] [-S] [-H FIRMWARE_HASH] [--platform platform]
+                   [--product product] [--model model] [--hwrev revision]
+                   [port]
+
+  RNode Configuration and firmware utility. This program allows you to change
+  various settings and startup modes of RNode. It can also install, flash and
+  update the firmware on supported devices.
+
+  positional arguments:
+    port                  serial port where RNode is attached
+
+  options:
+    -h, --help            show this help message and exit
+    -i, --info            Show device info
+    -a, --autoinstall     Automatic installation on various supported devices
+    -u, --update          Update firmware to the latest version
+    -U, --force-update    Update to specified firmware even if version matches
+                          or is older than installed version
+    --fw-version version  Use a specific firmware version for update or
+                          autoinstall
+    --fw-url url          Use an alternate firmware download URL
+    --nocheck             Don't check for firmware updates online
+    -e, --extract         Extract firmware from connected RNode for later use
+    -E, --use-extracted   Use the extracted firmware for autoinstallation or
+                          update
+    -C, --clear-cache     Clear locally cached firmware files
+    --baud-flash baud_flash
+                          Set specific baud rate when flashing device. Default
+                          is 921600
+    -N, --normal          Switch device to normal mode
+    -T, --tnc             Switch device to TNC mode
+    -b, --bluetooth-on    Turn device bluetooth on
+    -B, --bluetooth-off   Turn device bluetooth off
+    -p, --bluetooth-pair  Put device into bluetooth pairing mode
+    -D i, --display i     Set display intensity (0-255)
+    --display-addr byte   Set display address as hex byte (00 - FF)
+    --freq Hz             Frequency in Hz for TNC mode
+    --bw Hz               Bandwidth in Hz for TNC mode
+    --txp dBm             TX power in dBm for TNC mode
+    --sf factor           Spreading factor for TNC mode (7 - 12)
+    --cr rate             Coding rate for TNC mode (5 - 8)
+    --eeprom-backup       Backup EEPROM to file
+    --eeprom-dump         Dump EEPROM to console
+    --eeprom-wipe         Unlock and wipe EEPROM
+    -P, --public          Display public part of signing key
+    --trust-key hexbytes  Public key to trust for device verification
+    --version             Print program version and exit
+    -f, --flash           Flash firmware and bootstrap EEPROM
+    -r, --rom             Bootstrap EEPROM without flashing firmware
+    -k, --key             Generate a new signing key and exit
+    -S, --sign            Display public part of signing key
+    -H FIRMWARE_HASH, --firmware-hash FIRMWARE_HASH
+                          Display installed firmware hash
+    --platform platform   Platform specification for device bootstrap
+    --product product     Product specification for device bootstrap
+    --model model         Model code for device bootstrap
+    --hwrev revision      Hardware revision for device bootstrap
+
+
+For more information on how to create your own RNodes, please read the :ref:`Creating RNodes<rnode-creating>`
+section of this manual.
+
+Remote Management
+-----------------
+
+It is possible to allow remote management of Reticulum
+systems using the various built-in utilities, such as
+``rnstatus`` and ``rnpath``. To do so, you will need to set
+the ``enable_remote_management`` directive in the ``[reticulum]``
+section of the configuration file. You will also need to specify
+one or more Reticulum Identity hashes for authenticating the
+queries from client programs. For this purpose, you can use
+existing identity files, or generate new ones with the rnid utility.
+
+The following is a truncated example of enabling remote management
+in the Reticulum configuration file:
+
+.. code:: text
+  
+  [reticulum]
+  ...
+  enable_remote_management = yes
+  remote_management_allowed = 9fb6d773498fb3feda407ed8ef2c3229, 2d882c5586e548d79b5af27bca1776dc
+  ...
+
+For a complete example configuration, you can run ``rnsd --exampleconfig``.
+
 Improving System Configuration
 ------------------------------
 
@@ -473,6 +847,9 @@ Reticulum as a System Service
 Instead of starting Reticulum manually, you can install ``rnsd`` as a system
 service and have it start automatically at boot.
 
+Systemwide Service
+^^^^^^^^^^^^^^^^^^
+
 If you installed Reticulum with ``pip``, the ``rnsd`` program will most likely
 be located in a user-local installation path only, which means ``systemd`` will not
 be able to execute it. In this case, you can simply symlink the ``rnsd`` program
@@ -519,4 +896,49 @@ If you want to automatically start ``rnsd`` at boot, run:
 
 .. code:: text
 
-  sudo systemctl enable rnsd
+  sudo systemctl enable rnsd
+
+Userspace Service
+^^^^^^^^^^^^^^^^^
+
+Alternatively you can use a user systemd service instead of a system wide one. This way the whole setup can be done as a regular user.
+Create a user systemd service files ``~/.config/systemd/user/rnsd.service`` with the following content:
+
+.. code:: text
+
+  [Unit]
+  Description=Reticulum Network Stack Daemon
+  After=default.target
+
+  [Service]
+  # If you run Reticulum on WiFi devices,
+  # or other devices that need some extra
+  # time to initialise, you might want to
+  # add a short delay before Reticulum is
+  # started by systemd:
+  # ExecStartPre=/bin/sleep 10
+  Type=simple
+  Restart=always
+  RestartSec=3
+  ExecStart=RNS_BIN_DIR/rnsd --service
+
+  [Install]
+  WantedBy=default.target
+
+Replace ``RNS_BIN_DIR`` with the path to your Reticulum binary directory (eg. /home/USERNAMEHERE/rns/bin).
+
+Start user service:
+
+.. code:: text
+
+  systemctl --user daemon-reload
+  systemctl --user start rnsd.service
+
+If you want to automatically start ``rnsd`` without having to log in as the USERNAMEHERE, do:
+
+.. code:: text
+  
+  sudo loginctl enable-linger USERNAMEHERE
+  systemctl --user enable rnsd.service
+  
+

docs/manual/_sources/whatis.rst.txt

@@ -21,15 +21,15 @@ networks, without any need for hierarchical or beaureucratic structures to contr
 or manage them, while ensuring individuals and communities full sovereignty
 over their own network segments.
 
-Reticulum is a complete networking stack, and does not need IP or higher
+Reticulum is a **complete networking stack**, and does not need IP or higher
 layers, although it is easy to utilise IP (with TCP or UDP) as the underlying
 carrier for Reticulum. It is therefore trivial to tunnel Reticulum over the
 Internet or private IP networks. Reticulum is built directly on cryptographic
 principles, allowing resilience and stable functionality in open and trustless
 networks.
 
-No kernel modules or drivers are required. Reticulum runs completely in
-userland, and can run on practically any system that runs Python 3. Reticulum
+No kernel modules or drivers are required. Reticulum can run completely in
+userland, and will run on practically any system that runs Python 3. Reticulum
 runs well even on small single-board computers like the Pi Zero.
 
 
@@ -38,7 +38,7 @@ Current Status
 **Please know!** Reticulum should currently be considered beta software. All core protocol
 features are implemented and functioning, but additions will probably occur as
 real-world use is explored. *There will be bugs*. The API and wire-format can be
-considered stable at the moment, but could change if absolutely warranted.
+considered complete and stable at the moment, but could change if absolutely warranted.
 
 
 What does Reticulum Offer?
@@ -53,9 +53,9 @@ What does Reticulum Offer?
 
 * Forward Secrecy by using ephemeral Elliptic Curve Diffie-Hellman keys on Curve25519
 
-* Reticulum uses the `Fernet <https://github.com/fernet/spec/blob/master/Spec.md>`_ specification for on-the-wire / over-the-air encryption
+* Reticulum uses a modified version of the `Fernet <https://github.com/fernet/spec/blob/master/Spec.md>`_ specification for on-the-wire / over-the-air encryption
 
-  * All keys are ephemeral and derived from an ECDH key exchange on Curve25519
+  * Keys are ephemeral and derived from an ECDH key exchange on Curve25519
 
   * AES-128 in CBC mode with PKCS7 padding
 
@@ -63,6 +63,8 @@ What does Reticulum Offer?
 
   * IVs are generated through os.urandom()
 
+  * No Version and Timestamp metadata included
+
 * Unforgeable packet delivery confirmations
 
 * A variety of supported interface types
@@ -71,7 +73,7 @@ What does Reticulum Offer?
 
 * Efficient link establishment
 
-  * Total bandwidth cost of setting up a link is only 3 packets, totalling 297 bytes
+  * Total cost of setting up an encrypted and verified link is only 3 packets, totalling 297 bytes
 
   * Low cost of keeping links open at only 0.44 bits per second
 
@@ -91,7 +93,7 @@ What does Reticulum Offer?
 Where can Reticulum be Used?
 ============================
 Over practically any medium that can support at least a half-duplex channel
-with 500 bits per second throughput, and an MTU of 500 bytes. Data radios,
+with greater throughput than 5 bits per second, and an MTU of 500 bytes. Data radios,
 modems, LoRa radios, serial lines, AX.25 TNCs, amateur radio digital modes,
 ad-hoc WiFi, free-space optical links and similar systems are all examples
 of the types of interfaces Reticulum was designed for.
@@ -99,7 +101,8 @@ of the types of interfaces Reticulum was designed for.
 An open-source LoRa-based interface called `RNode <https://unsigned.io/rnode>`_
 has been designed as an example transceiver that is very suitable for
 Reticulum. It is possible to build it yourself, to transform a common LoRa
-development board into one, or it can be purchased as a complete transceiver.
+development board into one, or it can be purchased as a complete transceiver
+from various vendors.
 
 Reticulum can also be encapsulated over existing IP networks, so there's
 nothing stopping you from using it over wired Ethernet or your local WiFi
@@ -162,7 +165,7 @@ Caveat Emptor
 ==============
 Reticulum is an experimental networking stack, and should be considered as
 such. While it has been built with cryptography best-practices very foremost in
-mind, it has not been externally security audited, and there could very well be
+mind, it has not yet been externally security audited, and there could very well be
 privacy-breaking bugs. To be considered secure, Reticulum needs a thorough
 security review by independent cryptographers and security researchers. If you
-want to help out, or help sponsor an audit, please do get in touch.
+want to help out with this, or can help sponsor an audit, please do get in touch.

docs/manual/_static/basic.css

@@ -236,16 +236,6 @@ div.body p, div.body dd, div.body li, div.body blockquote {
 a.headerlink {
     visibility: hidden;
 }
-a.brackets:before,
-span.brackets > a:before{
-    content: "[";
-}
-
-a.brackets:after,
-span.brackets > a:after {
-    content: "]";
-}
-
 
 h1:hover > a.headerlink,
 h2:hover > a.headerlink,
@@ -334,11 +324,15 @@ aside.sidebar {
 p.sidebar-title {
     font-weight: bold;
 }
+nav.contents,
+aside.topic,
 div.admonition, div.topic, blockquote {
     clear: left;
 }
 
 /* -- topics ---------------------------------------------------------------- */
+nav.contents,
+aside.topic,
 div.topic {
     border: 1px solid #ccc;
     padding: 7px;
@@ -377,6 +371,8 @@ div.body p.centered {
 
 div.sidebar > :last-child,
 aside.sidebar > :last-child,
+nav.contents > :last-child,
+aside.topic > :last-child,
 div.topic > :last-child,
 div.admonition > :last-child {
     margin-bottom: 0;
@@ -384,6 +380,8 @@ div.admonition > :last-child {
 
 div.sidebar::after,
 aside.sidebar::after,
+nav.contents::after,
+aside.topic::after,
 div.topic::after,
 div.admonition::after,
 blockquote::after {
@@ -608,19 +606,26 @@ ol.simple p,
 ul.simple p {
     margin-bottom: 0;
 }
-dl.footnote > dt,
-dl.citation > dt {
+aside.footnote > span,
+div.citation > span {
     float: left;
-    margin-right: 0.5em;
 }
-
-dl.footnote > dd,
-dl.citation > dd {
+aside.footnote > span:last-of-type,
+div.citation > span:last-of-type {
+  padding-right: 0.5em;
+}
+aside.footnote > p {
+  margin-left: 2em;
+}
+div.citation > p {
+  margin-left: 4em;
+}
+aside.footnote > p:last-of-type,
+div.citation > p:last-of-type {
     margin-bottom: 0em;
 }
-
-dl.footnote > dd:after,
-dl.citation > dd:after {
+aside.footnote > p:last-of-type:after,
+div.citation > p:last-of-type:after {
     content: "";
     clear: both;
 }
@@ -636,10 +641,6 @@ dl.field-list > dt {
     padding-left: 0.5em;
     padding-right: 5px;
 }
-dl.field-list > dt:after {
-    content: ":";
-}
-
 
 dl.field-list > dd {
     padding-left: 0.5em;

docs/manual/_static/copybutton.css

@@ -35,7 +35,8 @@ div.highlight  {
     position: relative;
 }
 
-.highlight:hover button.copybtn {
+/* Show the copybutton */
+.highlight:hover button.copybtn, button.copybtn.success {
 	opacity: 1;
 }
 

docs/manual/_static/copybutton.js

@@ -20,7 +20,7 @@ const messages = {
   },
   'fr' : {
     'copy': 'Copier',
-    'copy_to_clipboard': 'Copié dans le presse-papier',
+    'copy_to_clipboard': 'Copier dans le presse-papier',
     'copy_success': 'Copié !',
     'copy_failure': 'Échec de la copie',
   },
@@ -102,18 +102,25 @@ const clearSelection = () => {
   }
 }
 
-// Changes tooltip text for two seconds, then changes it back
+// Changes tooltip text for a moment, then changes it back
+// We want the timeout of our `success` class to be a bit shorter than the
+// tooltip and icon change, so that we can hide the icon before changing back.
+var timeoutIcon = 2000;
+var timeoutSuccessClass = 1500;
+
 const temporarilyChangeTooltip = (el, oldText, newText) => {
   el.setAttribute('data-tooltip', newText)
   el.classList.add('success')
-  setTimeout(() => el.setAttribute('data-tooltip', oldText), 2000)
-  setTimeout(() => el.classList.remove('success'), 2000)
+  // Remove success a little bit sooner than we change the tooltip
+  // So that we can use CSS to hide the copybutton first
+  setTimeout(() => el.classList.remove('success'), timeoutSuccessClass)
+  setTimeout(() => el.setAttribute('data-tooltip', oldText), timeoutIcon)
 }
 
 // Changes the copy button icon for two seconds, then changes it back
 const temporarilyChangeIcon = (el) => {
   el.innerHTML = iconCheck;
-  setTimeout(() => {el.innerHTML = iconCopy}, 2000)
+  setTimeout(() => {el.innerHTML = iconCopy}, timeoutIcon)
 }
 
 const addCopyButtonToCodeCells = () => {
@@ -125,7 +132,8 @@ const addCopyButtonToCodeCells = () => {
   }
 
   // Add copybuttons to all of our code cells
-  const codeCells = document.querySelectorAll('div.highlight pre')
+  const COPYBUTTON_SELECTOR = 'div.highlight pre';
+  const codeCells = document.querySelectorAll(COPYBUTTON_SELECTOR)
   codeCells.forEach((codeCell, index) => {
     const id = codeCellId(index)
     codeCell.setAttribute('id', id)
@@ -141,10 +149,25 @@ function escapeRegExp(string) {
     return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
 }
 
+/**
+ * Removes excluded text from a Node.
+ *
+ * @param {Node} target Node to filter.
+ * @param {string} exclude CSS selector of nodes to exclude.
+ * @returns {DOMString} Text from `target` with text removed.
+ */
+function filterText(target, exclude) {
+    const clone = target.cloneNode(true);  // clone as to not modify the live DOM
+    if (exclude) {
+        // remove excluded nodes
+        clone.querySelectorAll(exclude).forEach(node => node.remove());
+    }
+    return clone.innerText;
+}
+
 // Callback when a copy button is clicked. Will be passed the node that was clicked
 // should then grab the text and replace pieces of text that shouldn't be used in output
 function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") {
-
     var regexp;
     var match;
 
@@ -199,7 +222,12 @@ function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onl
 
 var copyTargetText = (trigger) => {
   var target = document.querySelector(trigger.attributes['data-clipboard-target'].value);
-  return formatCopyText(target.innerText, '', false, true, true, true, '', '')
+
+  // get filtered text
+  let exclude = '.linenos';
+
+  let text = filterText(target, exclude);
+  return formatCopyText(text, '', false, true, true, true, '', '')
 }
 
   // Initialize with a callback so we can modify the text before copy

docs/manual/_static/copybutton_funcs.js

@@ -2,10 +2,25 @@ function escapeRegExp(string) {
     return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
 }
 
+/**
+ * Removes excluded text from a Node.
+ *
+ * @param {Node} target Node to filter.
+ * @param {string} exclude CSS selector of nodes to exclude.
+ * @returns {DOMString} Text from `target` with text removed.
+ */
+export function filterText(target, exclude) {
+    const clone = target.cloneNode(true);  // clone as to not modify the live DOM
+    if (exclude) {
+        // remove excluded nodes
+        clone.querySelectorAll(exclude).forEach(node => node.remove());
+    }
+    return clone.innerText;
+}
+
 // Callback when a copy button is clicked. Will be passed the node that was clicked
 // should then grab the text and replace pieces of text that shouldn't be used in output
 export function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") {
-
     var regexp;
     var match;
 

docs/manual/_static/documentation_options.js

@@ -1,6 +1,6 @@
 var DOCUMENTATION_OPTIONS = {
     URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'),
-    VERSION: '0.3.14 beta',
+    VERSION: '0.8.2 beta',
     LANGUAGE: 'en',
     COLLAPSE_INDEX: false,
     BUILDER: 'html',

docs/manual/_static/pygments.css

@@ -22,6 +22,7 @@
 .highlight .cs { color: #8f5902; font-style: italic } /* Comment.Special */
 .highlight .gd { color: #a40000 } /* Generic.Deleted */
 .highlight .ge { color: #000000; font-style: italic } /* Generic.Emph */
+.highlight .ges { color: #000000; font-weight: bold; font-style: italic } /* Generic.EmphStrong */
 .highlight .gr { color: #ef2929 } /* Generic.Error */
 .highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */
 .highlight .gi { color: #00A000 } /* Generic.Inserted */
@@ -101,20 +102,21 @@ body[data-theme="dark"] .highlight .x { color: #d0d0d0 } /* Other */
 body[data-theme="dark"] .highlight .p { color: #d0d0d0 } /* Punctuation */
 body[data-theme="dark"] .highlight .ch { color: #ababab; font-style: italic } /* Comment.Hashbang */
 body[data-theme="dark"] .highlight .cm { color: #ababab; font-style: italic } /* Comment.Multiline */
-body[data-theme="dark"] .highlight .cp { color: #cd2828; font-weight: bold } /* Comment.Preproc */
+body[data-theme="dark"] .highlight .cp { color: #ff3a3a; font-weight: bold } /* Comment.Preproc */
 body[data-theme="dark"] .highlight .cpf { color: #ababab; font-style: italic } /* Comment.PreprocFile */
 body[data-theme="dark"] .highlight .c1 { color: #ababab; font-style: italic } /* Comment.Single */
 body[data-theme="dark"] .highlight .cs { color: #e50808; font-weight: bold; background-color: #520000 } /* Comment.Special */
-body[data-theme="dark"] .highlight .gd { color: #d22323 } /* Generic.Deleted */
+body[data-theme="dark"] .highlight .gd { color: #ff3a3a } /* Generic.Deleted */
 body[data-theme="dark"] .highlight .ge { color: #d0d0d0; font-style: italic } /* Generic.Emph */
-body[data-theme="dark"] .highlight .gr { color: #d22323 } /* Generic.Error */
+body[data-theme="dark"] .highlight .ges { color: #d0d0d0; font-weight: bold; font-style: italic } /* Generic.EmphStrong */
+body[data-theme="dark"] .highlight .gr { color: #ff3a3a } /* Generic.Error */
 body[data-theme="dark"] .highlight .gh { color: #ffffff; font-weight: bold } /* Generic.Heading */
 body[data-theme="dark"] .highlight .gi { color: #589819 } /* Generic.Inserted */
 body[data-theme="dark"] .highlight .go { color: #cccccc } /* Generic.Output */
 body[data-theme="dark"] .highlight .gp { color: #aaaaaa } /* Generic.Prompt */
 body[data-theme="dark"] .highlight .gs { color: #d0d0d0; font-weight: bold } /* Generic.Strong */
 body[data-theme="dark"] .highlight .gu { color: #ffffff; text-decoration: underline } /* Generic.Subheading */
-body[data-theme="dark"] .highlight .gt { color: #d22323 } /* Generic.Traceback */
+body[data-theme="dark"] .highlight .gt { color: #ff3a3a } /* Generic.Traceback */
 body[data-theme="dark"] .highlight .kc { color: #6ebf26; font-weight: bold } /* Keyword.Constant */
 body[data-theme="dark"] .highlight .kd { color: #6ebf26; font-weight: bold } /* Keyword.Declaration */
 body[data-theme="dark"] .highlight .kn { color: #6ebf26; font-weight: bold } /* Keyword.Namespace */
@@ -186,20 +188,21 @@ body:not([data-theme="light"]) .highlight .x { color: #d0d0d0 } /* Other */
 body:not([data-theme="light"]) .highlight .p { color: #d0d0d0 } /* Punctuation */
 body:not([data-theme="light"]) .highlight .ch { color: #ababab; font-style: italic } /* Comment.Hashbang */
 body:not([data-theme="light"]) .highlight .cm { color: #ababab; font-style: italic } /* Comment.Multiline */
-body:not([data-theme="light"]) .highlight .cp { color: #cd2828; font-weight: bold } /* Comment.Preproc */
+body:not([data-theme="light"]) .highlight .cp { color: #ff3a3a; font-weight: bold } /* Comment.Preproc */
 body:not([data-theme="light"]) .highlight .cpf { color: #ababab; font-style: italic } /* Comment.PreprocFile */
 body:not([data-theme="light"]) .highlight .c1 { color: #ababab; font-style: italic } /* Comment.Single */
 body:not([data-theme="light"]) .highlight .cs { color: #e50808; font-weight: bold; background-color: #520000 } /* Comment.Special */
-body:not([data-theme="light"]) .highlight .gd { color: #d22323 } /* Generic.Deleted */
+body:not([data-theme="light"]) .highlight .gd { color: #ff3a3a } /* Generic.Deleted */
 body:not([data-theme="light"]) .highlight .ge { color: #d0d0d0; font-style: italic } /* Generic.Emph */
-body:not([data-theme="light"]) .highlight .gr { color: #d22323 } /* Generic.Error */
+body:not([data-theme="light"]) .highlight .ges { color: #d0d0d0; font-weight: bold; font-style: italic } /* Generic.EmphStrong */
+body:not([data-theme="light"]) .highlight .gr { color: #ff3a3a } /* Generic.Error */
 body:not([data-theme="light"]) .highlight .gh { color: #ffffff; font-weight: bold } /* Generic.Heading */
 body:not([data-theme="light"]) .highlight .gi { color: #589819 } /* Generic.Inserted */
 body:not([data-theme="light"]) .highlight .go { color: #cccccc } /* Generic.Output */
 body:not([data-theme="light"]) .highlight .gp { color: #aaaaaa } /* Generic.Prompt */
 body:not([data-theme="light"]) .highlight .gs { color: #d0d0d0; font-weight: bold } /* Generic.Strong */
 body:not([data-theme="light"]) .highlight .gu { color: #ffffff; text-decoration: underline } /* Generic.Subheading */
-body:not([data-theme="light"]) .highlight .gt { color: #d22323 } /* Generic.Traceback */
+body:not([data-theme="light"]) .highlight .gt { color: #ff3a3a } /* Generic.Traceback */
 body:not([data-theme="light"]) .highlight .kc { color: #6ebf26; font-weight: bold } /* Keyword.Constant */
 body:not([data-theme="light"]) .highlight .kd { color: #6ebf26; font-weight: bold } /* Keyword.Declaration */
 body:not([data-theme="light"]) .highlight .kn { color: #6ebf26; font-weight: bold } /* Keyword.Namespace */