Messaging on the Autonomi Network - Developer Feedback Needed

now we need that api to exchange p2p connection info

(and yes - doesn’t scale because everyone connects to everyone)

EDIT:

maybe for the interested reader … the used python code :slight_smile: it reads/writes to a local file rendezvous.info - inserts its own address/removes it on ctrl+c … if we now replace the file with a scratchpad on autonomi we should have the first p2p live chat solely based on autonomi …

import socket
import threading
import time
import random
import os
from datetime import datetime, timezone
from blessed import Terminal
import fire

RENDEZVOUS_FILE = "rendezvous.info"
term = Terminal()

def get_external_address(udp_socket: socket.socket, server_ip: str, server_port: int) -> tuple[str, int]:
    """
    Contacts the server to retrieve the external IP and port.
    """
    udp_socket.sendto(b"whoami", (server_ip, server_port))
    udp_socket.settimeout(2)
    try:
        response, _ = udp_socket.recvfrom(1024)
        external_ip, external_port = response.decode("utf-8").split(":")
        return external_ip, int(external_port)
    except socket.timeout:
        print("No response from the server.")
        return None, None

def update_rendezvous_file(ip: str, port: int, name: str) -> list[tuple[str, int, str]]:
    """
    Updates the rendezvous.info file with the new IP/port/name and returns the list of participants.
    """
    if not os.path.exists(RENDEZVOUS_FILE):
        open(RENDEZVOUS_FILE, "w").close()

    participants = []
    with open(RENDEZVOUS_FILE, "r") as f:
        for line in f:
            p_ip, p_port, p_name = line.strip().split(":")
            participants.append((p_ip, int(p_port), p_name))

    # Add own address if not already present
    if (ip, port, name) not in participants:
        with open(RENDEZVOUS_FILE, "a") as f:
            f.write(f"{ip}:{port}:{name}\n")

    # Remove own address from the participants list
    participants = [p for p in participants if not (p[0] == ip and p[1] == port)]

    return participants

def remove_self_from_rendezvous(ip: str, port: int, name: str):
    """
    Removes the user's contact information from rendezvous.info.
    """
    if not os.path.exists(RENDEZVOUS_FILE):
        return

    with open(RENDEZVOUS_FILE, "r") as f:
        lines = f.readlines()

    with open(RENDEZVOUS_FILE, "w") as f:
        for line in lines:
            if line.strip() != f"{ip}:{port}:{name}":
                f.write(line)

def watch_rendezvous_file(update_event):
    """
    Watches the rendezvous.info file for changes and triggers an update event.
    """
    last_modified = os.path.getmtime(RENDEZVOUS_FILE)
    while True:
        time.sleep(1)
        try:
            current_modified = os.path.getmtime(RENDEZVOUS_FILE)
            if current_modified != last_modified:
                last_modified = current_modified
                update_event.set()
        except FileNotFoundError:
            continue

def send_message(socket: socket.socket, message: str, recipient: tuple[str, int], name: str):
    """
    Sends a message with a random ID, UTC timestamp, and sender name to a recipient.
    """
    msg_id = random.randint(1000, 9999)
    utc_time = datetime.now(timezone.utc).isoformat()
    packet = f"{msg_id}|{utc_time}|{name}|{message}".encode("utf-8")
    socket.sendto(packet, recipient)

def render_chat(chat_log: list[str], input_prompt: str):
    """
    Renders the chat window and input prompt.
    """
    with term.location(0, 0):
        print(term.clear, end="")
        for line in chat_log[-(term.height - 2):]:
            print(line)
        print(term.move_xy(0, term.height - 1) + input_prompt, end="", flush=True)

def main(local_port: int, name: str, local_ip: str = None,
         discovery_ip: str = "185.198.27.221", discovery_port: int = 59595):
    """
    Starts the P2P chat client.

    Args:
        local_port (int): The local port to use.
        name (str): The name of the participant.
        local_ip (str): The local IP address. If set, no discovery server is used.
        discovery_ip (str): The IP of the discovery server.
        discovery_port (int): The port of the discovery server.
    """
    udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    udp_socket.bind(("0.0.0.0", local_port))

    if local_ip:
        external_ip, external_port = local_ip, local_port
    else:
        external_ip, external_port = get_external_address(udp_socket, discovery_ip, discovery_port)

    if not external_ip or not external_port:
        print("Error: Could not retrieve external address.")
        return

    participants = update_rendezvous_file(external_ip, external_port, name)
    print(f"Participants in the network: {participants}")

    chat_log = []
    running = True
    input_prompt = ">>> "
    input_lock = threading.Lock()
    update_event = threading.Event()

    def receive_messages():
        while running:
            try:
                data, addr = udp_socket.recvfrom(1024)
                msg_id, utc_time, sender_name, message = data.decode("utf-8").split("|", 3)

                if message == "heartbeat":
                    continue  # Ignore heartbeats

                local_time = datetime.fromisoformat(utc_time).astimezone().strftime("%Y-%m-%d %H:%M:%S")
                with input_lock:
                    chat_log.append(f"[{local_time}] {sender_name}: {message}")
            except Exception:
                continue

    def heartbeat():
        while running:
            for participant in participants:
                send_message(udp_socket, "heartbeat", (participant[0], participant[1]), name)
            time.sleep(2)

    def auto_render():
        while running:
            with input_lock:
                render_chat(chat_log, input_prompt)
            time.sleep(0.1)

    def reload_participants():
        while running:
            update_event.wait()
            if not running:
                break
            with input_lock:
                participants.clear()
                participants.extend(update_rendezvous_file(external_ip, external_port, name))
            update_event.clear()

    threading.Thread(target=receive_messages, daemon=True).start()
    threading.Thread(target=heartbeat, daemon=True).start()
    threading.Thread(target=auto_render, daemon=True).start()
    threading.Thread(target=watch_rendezvous_file, args=(update_event,), daemon=True).start()
    threading.Thread(target=reload_participants, daemon=True).start()

    try:
        with term.cbreak(), term.hidden_cursor():
            while running:
                user_input = term.inkey(timeout=0.1)

                if user_input.name == "KEY_ENTER":
                    with input_lock:
                        if input_prompt.strip() != ">>> ":
                            for participant in participants:
                                send_message(udp_socket, input_prompt[4:], (participant[0], participant[1]), name)
                            local_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
                            chat_log.append(f"[{local_time}] {name}: {input_prompt[4:]}")
                            input_prompt = ">>> "
                elif user_input.name == "KEY_BACKSPACE":
                    with input_lock:
                        input_prompt = input_prompt[:-1] if len(input_prompt) > 4 else input_prompt
                elif user_input:
                    with input_lock:
                        input_prompt += user_input

    except KeyboardInterrupt:
        print("\nExiting chat...")
        remove_self_from_rendezvous(external_ip, external_port, name)
        running = False
    finally:
        udp_socket.close()

if __name__ == "__main__":
    fire.Fire(main)

if you start it with local ip info it will use that (well … it would need to be a global ip actually xD … anyway)

python p2p_chat.py --local_port=12345 --local_ip=192.168.0.100 --name="Alice"

if you don’t pass the ip info it queries it from an external server

python p2p_chat.py --local_port=12345 --name="Alice"

it uses heartbeating to keep the hole-punched communication channel open and just fires to all other participants.

the local file rendezvous.info must have lines like:

77.12.239.17:13131:somerandomdude

that identify the other communication partners

10 Likes

Very cool, were all of the random dudes you, or were you testing with someone else?

2 Likes

Was me with 2 PCs behind my router and a cloud machine with active firewall all communicating with each other (obviously unencrypted) via the Internet

4 Likes

minor improvements:

people can now be kicked (removed from the list of peers; maybe useful if people suddenly disconnect without properly removing their connection info from the contacts file) and you can see the list of other chat participants in the upper right corner :slight_smile:

6 Likes

Hello @Helen, the topic of communication in the Autonomi network has been occupying me for a long time, and the community has already approached this topic many times :wink: I am glad that you have taken up the topic anew with the very interesting proposal of the Trystero system.

After my various thoughts and analyses of instant messaging, I focused on the SimpleX Chat solution, which I recognised in more depth last year, and as @Southside mentioned was also being tested by some of us. Last year I posted a thread in which I think I went a bit overboard :grin: :rofl: but perhaps some of the information will be useful to you:

Among the many solutions, SimpleX seems to me much more dedicated to Web3 than the others, and it is very close to Autonomi in concept, which, apart from the technological differences, could be very conducive to integration or implementation. I would love to know your opinion :slight_smile:

Trystero is very interesting for its lightness, direct communication with low latency and efficient data transfer, but the poor metadata protection and possible communication problems if the connected devices are behind NAT or a restrictive firewall seem to be major drawbacks.

If communication is to be as private and secure as data storage in Autonomi, SimpleX Chat is, in my opinion, the best candidate for today.

1 Like

Oi!!! thats my job!!!

AFAICS, its clean and simple, does what it says on the tin with no frills.

4 Likes

Thanks for your post @Profess. SimpleX has been mentioned a few times now. I am looking at it more closely, along with delta chat, mentioned by @rreive. I read that SimpleX relies on servers, but not centralized servers in the same sense as Autonomi intends to avoid. I’m not sure how that fits with Autonomi’s aim to be server-less, or if an adaptation is possible. I know @dirvine was looking at it, so maybe could weigh in on this aspect.

Riddim’s simple hole-punching chat program is nice but not encrypted, so would not serve the wished-for purpose either, though could become a nice tool for gossip etc. Trystero seemed to be geared along these lines as well, with good encryption I thought. I am not sure what you are referring to with respect to poor metadata protection, though.

I do like delta chat as a candidate for integration with Autonomi, so far, and would be interested in your opinion on that as well. I’ll also dive further into the SimpleX documentation…

4 Likes

Do you mean

Delta Chat, decentralized secure messenger - Delta Chat?

If I’m looking correctly, @rreive mentioned Jami, unless I’ve missed something :slight_smile:

Yes, the SimpleX network is made up of independent servers that are hosted in a decentralised manner, there is no central server that manages communications and data is sent through the servers but not stored or processed on them, all messages are end-to-end encrypted to ensure confidentiality of content. I think there is no problem here, the difference is that Autonomi is a decentralised network based on nodes that can perform different functions like data storage, information processing or direct communication between P2P nodes, which can be complementary to the SimpleX architecture.

What’s key about SimpleX is that it’s post-quantum cryptography just like Autonomi and doesn’t use any user IDs, not even random ones, you can use multiple profiles and connect to the network anywhere, from any device - and that’s very similar to Autonomi’s privacy rules. In addition, SimpleX is inherently private, does not collect any data and has a number of very advanced features, in that it significantly outperforms other solutions - I have bulleted all the features of the messenger in the article.

If I understand correctly, Trystero uses signalling (e.g. STUN server) to establish the connection, metadata such as connection time or IP addresses may be visible at the signalling stage, but I am not sure… :slight_smile:

Yes I did mention that, however the fit we, @Helen and @Southside discovered in Delta is that on review and actual trial between each other we found the fit better with Autonomi with Delta given its DHT and hugely proven and adopted out in the real world, build on a Signal fork… with many translations language wise…

Also Delta Chat at 168Mbytes RAM and six(6) PIDs is 40% less resource intense and lighter weight than JAMI 281Mbyte of RAM monolithic single PID which is JRE… 3X slower to load than Delta, etc…, also the UI has not progressed in that the UX is a bit abstract. That said JAMI has better fine grained user controls for video signalling and screen size control.

We are still digging into the Delta chat details, but were also waiting for the API to firm up on AN…, which now that Dave is released we are starting to see what is possible initially as an integration approach which is largely WIP in my case… my day job frittered most of my spare time this week…

Oh right, we had a separate topic about speed of network that morphed into further messaging discussions, and that’s probably where Delta Chat came up…

Sorry, I havent used Delta at all, only SimpleX.

1 Like

We all used Delta Chat the other day, @Southside

3 Likes

We did.
Can only assume I blanked it cos there was no replies in their forum

Been looking at so much other stuff recently this totally slipped my memory

There is one click on the Autonomi link I posted and that was likely me checking it went to the right url…

Somebody else say Hello and see if we can get a response.

1 Like

Well… Adding encryption would just mean to encrypt/decrypt the communicated data - that would be a pretty simple addon tbh… Just didn’t think it’s worth the 3 lines of code for now…

but probably doesn’t hurt anyway - you’re right …

import socket
import threading
import time
import random
import os
from datetime import datetime, timezone
from blessed import Terminal
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.primitives import padding
import fire

RENDEZVOUS_FILE = "rendezvous.info"
term = Terminal()

def get_aes_key_from_filename(filename: str) -> bytes:
    """
    Derives a 32-byte AES key from the filename.
    Pads the filename if necessary to make it exactly 32 bytes long.
    """
    key = filename.encode("utf-8")
    if len(key) > 32:
        key = key[:32]  # Truncate to 32 bytes
    elif len(key) < 32:
        key = key.ljust(32, b'\0')  # Pad with null bytes
    return key

def encrypt_message(key: bytes, message: str) -> bytes:
    """
    Encrypts a message using AES in CBC mode.
    """
    iv = os.urandom(16)  # Generate a random initialization vector (IV)
    cipher = Cipher(algorithms.AES(key), modes.CBC(iv))
    padder = padding.PKCS7(algorithms.AES.block_size).padder()
    padded_message = padder.update(message.encode("utf-8")) + padder.finalize()
    encryptor = cipher.encryptor()
    ciphertext = encryptor.update(padded_message) + encryptor.finalize()
    return iv + ciphertext  # Prefix the IV to the ciphertext

def decrypt_message(key: bytes, ciphertext: bytes) -> str:
    """
    Decrypts a message encrypted with AES in CBC mode.
    """
    iv = ciphertext[:16]  # Extract the IV from the beginning
    encrypted_message = ciphertext[16:]
    cipher = Cipher(algorithms.AES(key), modes.CBC(iv))
    decryptor = cipher.decryptor()
    padded_message = decryptor.update(encrypted_message) + decryptor.finalize()
    unpadder = padding.PKCS7(algorithms.AES.block_size).unpadder()
    message = unpadder.update(padded_message) + unpadder.finalize()
    return message.decode("utf-8")

def get_external_address(udp_socket: socket.socket, server_ip: str, server_port: int) -> tuple[str, int]:
    """
    Contacts the server to retrieve the external IP and port.
    """
    udp_socket.sendto(b"whoami", (server_ip, server_port))
    udp_socket.settimeout(2)
    try:
        response, _ = udp_socket.recvfrom(1024)
        external_ip, external_port = response.decode("utf-8").split(":")
        return external_ip, int(external_port)
    except socket.timeout:
        print("No response from the server.")
        return None, None

def update_rendezvous_file(ip: str, port: int, name: str) -> list[tuple[str, int, str]]:
    """
    Updates the rendezvous.info file with the new IP/port/name and returns the list of participants.
    """
    if not os.path.exists(RENDEZVOUS_FILE):
        open(RENDEZVOUS_FILE, "w").close()

    participants = []
    with open(RENDEZVOUS_FILE, "r") as f:
        for line in f:
            p_ip, p_port, p_name = line.strip().split(":")
            participants.append((p_ip, int(p_port), p_name))

    # Add own address if not already present
    if (ip, port, name) not in participants:
        with open(RENDEZVOUS_FILE, "a") as f:
            f.write(f"{ip}:{port}:{name}\n")

    # Remove own address from the participants list
    participants = [p for p in participants if not (p[0] == ip and p[1] == port)]

    return participants

def remove_self_from_rendezvous(ip: str, port: int, name: str):
    """
    Removes the user's contact information from rendezvous.info.
    """
    if not os.path.exists(RENDEZVOUS_FILE):
        return

    with open(RENDEZVOUS_FILE, "r") as f:
        lines = f.readlines()

    with open(RENDEZVOUS_FILE, "w") as f:
        for line in lines:
            if line.strip() != f"{ip}:{port}:{name}":
                f.write(line)

def send_message(socket: socket.socket, message: str, recipient: tuple[str, int], name: str, aes_key: bytes):
    """
    Encrypts and sends a message with a random ID, UTC timestamp, and sender name to a recipient.
    """
    msg_id = random.randint(1000, 9999)
    utc_time = datetime.now(timezone.utc).isoformat()
    plaintext_message = f"{msg_id}|{utc_time}|{name}|{message}"
    encrypted_message = encrypt_message(aes_key, plaintext_message)
    socket.sendto(encrypted_message, recipient)

def decrypt_and_process_message(key: bytes, data: bytes) -> tuple[str, str, str, str]:
    """
    Decrypts the received message and processes its content.
    Returns the message ID, UTC timestamp, sender name, and message.
    """
    plaintext_message = decrypt_message(key, data)
    msg_id, utc_time, sender_name, message = plaintext_message.split("|", 3)
    return msg_id, utc_time, sender_name, message

def render_chat(chat_log: list[str], input_prompt: str, participants: list[str]):
    """
    Renders the chat window with the sidebar and input prompt.
    """
    with term.location(0, 0):
        print(term.clear, end="")

        # Sidebar with participants
        sidebar_width = 20
        for i, participant in enumerate(participants[:term.height - 2]):
            print(term.move_xy(term.width - sidebar_width, i) + f"{participant}")

        # Chat log
        for i, line in enumerate(chat_log[-(term.height - 2):]):
            print(term.move_xy(0, i) + line[:term.width - sidebar_width - 2])

        # Input prompt
        print(term.move_xy(0, term.height - 1) + input_prompt, end="", flush=True)

def main(local_port: int, name: str, local_ip: str = None,
         discovery_ip: str = "185.198.27.221", discovery_port: int = 59595):
    """
    Starts the P2P chat client.

    Args:
        local_port (int): The local port to use.
        name (str): The name of the participant.
        local_ip (str): The local IP address. If set, no discovery server is used.
        discovery_ip (str): The IP of the discovery server.
        discovery_port (int): The port of the discovery server.
    """
    aes_key = get_aes_key_from_filename(RENDEZVOUS_FILE)
    udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    udp_socket.bind(("0.0.0.0", local_port))

    if local_ip:
        external_ip, external_port = local_ip, local_port
    else:
        external_ip, external_port = get_external_address(udp_socket, discovery_ip, discovery_port)

    if not external_ip or not external_port:
        print("Error: Could not retrieve external address.")
        return

    participants = update_rendezvous_file(external_ip, external_port, name)
    print(f"Participants in the network: {participants}")

    chat_log = []
    running = True
    input_prompt = ">>> "
    input_lock = threading.Lock()

    def receive_messages():
        while running:
            try:
                data, addr = udp_socket.recvfrom(1024)
                msg_id, utc_time, sender_name, message = decrypt_and_process_message(aes_key, data)

                if message == "heartbeat":
                    continue  # Ignore heartbeats

                local_time = datetime.fromisoformat(utc_time).astimezone().strftime("%Y-%m-%d %H:%M:%S")
                with input_lock:
                    chat_log.append(f"[{local_time}] {sender_name}: {message}")
            except Exception:
                continue

    def heartbeat():
        while running:
            for participant in participants:
                send_message(udp_socket, "heartbeat", (participant[0], participant[1]), name, aes_key)
            time.sleep(2)

    def auto_render():
        while running:
            with input_lock:
                render_chat(chat_log, input_prompt, [p[2] for p in participants])
            time.sleep(0.1)

    threading.Thread(target=receive_messages, daemon=True).start()
    threading.Thread(target=heartbeat, daemon=True).start()
    threading.Thread(target=auto_render, daemon=True).start()

    try:
        with term.cbreak(), term.hidden_cursor():
            while running:
                user_input = term.inkey(timeout=0.1)

                if user_input.name == "KEY_ENTER":
                    with input_lock:
                        if input_prompt.strip() != ">>> ":
                            for participant in participants:
                                send_message(udp_socket, input_prompt[4:], (participant[0], participant[1]), name, aes_key)
                            local_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
                            chat_log.append(f"[{local_time}] {name}: {input_prompt[4:]}")
                            input_prompt = ">>> "
                elif user_input.name == "KEY_BACKSPACE":
                    with input_lock:
                        input_prompt = input_prompt[:-1] if len(input_prompt) > 4 else input_prompt
                elif user_input:
                    with input_lock:
                        input_prompt += user_input

    except KeyboardInterrupt:
        print("\nExiting chat...")
        remove_self_from_rendezvous(external_ip, external_port, name)
        running = False
    finally:
        udp_socket.close()

if __name__ == "__main__":
    fire.Fire(main)

here comes the chat with AES encryption

=>

1 Like

how fast was sending or receiveing message? (Or how slow will be this on ETH network?¨)

well … it’s a p2p chat - it’s instant; just the connection info must be distributed/uploaded via autonomi

so setting up a new chatroom might take some moments; and joining a room does take some moments (all other participants must refresh the info coming from the network)

…so joining a chat may e.g. take 1 minute if all participants refresh the contact file every minute … but after that communication is as fast as it gets … putting a server in the middle would always be an additional hop so cannot be faster but must always be slower … there is no faster chat on this planet than this simple p2p chat :slight_smile:

6 Likes

Given it is p2p, presumably each peer knows who else is connected? Probably just need to resolve the chat ‘room’ for you all to reach one another?

3 Likes

exactly.

my thinking is:

step 1: “chat name” → we need a basic name resolution system to get from this to e.g. a pointer
step 2: pointer needs to point to a scratchpad
step 3: scratchpad content is the shared list of peers; scratchpad address could be the AES key too

… some gossip protocol; adding public keys to contacts, signed messages (or encrypted messages to others) would enable DMs and reduce connection count … writing a “chat_relay”; hosting it on a publicly accessible server+port would enable message retaining and enable people to join a chat if ports would get twisted very strangely by their connection (and/or hide IPs of chat participants) …

fairly easy to expand and add some features … a communication channel doesn’t need to be created in a complex way (could be easily used for a live-chat on auto-shops/exchanges/…)

6 Likes

And I am recalling now that there is also minimal server use in Delta chat.

3 Likes

yes, with Delta chat there really is a first time bootstrap server touch the first time the client is activated on your device,

then thereafter, its peer2peer between parties as you add connections,

plus you can also create your own bootstrap server to avoid using a 3rd party bootstrap sever, this in theory could also be where @riddim 's idea of a chat relay server could live , with permission storing of chat messages stored for groups,

if I am reading the documentation correctly, anyone can fire up a bootstrap server…

Well the deeper I dig Delta Chat makes use of Imap servers, so that would mean to get this to work, users would have to run FOSS dovcot or postfix IMAP server instances as a service over top of Autonomi Network, doable, but messy…

So that is what they mean with bootstrap server…

I am now less inclined to thing it will work on AN…

I regress back to JAMI which can operate peer to peer , and can make use of STUN and TURN servers,

“Jami, developed and maintained by Savoir Faire Linux, can operate without the use of STUN or TURN servers. Jami uses a decentralized network and distributed hash table technology, which allows it to distribute directory functions, authentication, and encryption across all systems connected to it. This means that Jami can establish peer-to-peer connections directly between users without needing centralized STUN or TURN servers for NAT traversal. However, for users behind certain types of network address translators (NATs), especially symmetric NATs, STUN or TURN servers can still be beneficial for ensuring reliable communication.”

2 Likes