I was trying to remember the hairpinning as I remember you mentioned it some weeks ago but couldn’t find it in my head!
So if hairpinning is supported, plus some new config params that New and improved qp2p API by lionel1704 · Pull Request #206 · maidsafe/qp2p · GitHub seems to be adding (external_ip) then it should work in a setup like this, it seems…
send a message from the external IP:port of one node from the other.
Send a message? How do I do that?
Using Safe, then make sure your genesis node IP:port in the config other nodes read is the external one (not the lan one)
Do I understand this correctly?
[folaht@Rezosur-uq node]$ ps -ef | grep sn_node
folaht 76753 1 0 15:47 pts/1 00:00:00 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 192.168.178.24 --first --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-genesis --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-genesis
folaht 76809 1 0 15:49 pts/1 00:00:00 /home/folaht/.safe/node/sn_node -vv --hard-coded-contacts ["83.163.103.119:12001"] --root-dir /home/folaht/.safe/node/local-node --log-dir /home/folaht/.safe/node/local-node
folaht 76822 1 0 15:49 pts/1 00:00:00 /home/folaht/.safe/node/sn_node -vv --hard-coded-contacts ["83.163.103.119:12001"] --root-dir /home/folaht/.safe/node/local-node --log-dir /home/folaht/.safe/node/local-node
folaht 76835 1 0 15:49 pts/1 00:00:00 /home/folaht/.safe/node/sn_node -vv --hard-coded-contacts ["83.163.103.119:12001"] --root-dir /home/folaht/.safe/node/local-node --log-dir /home/folaht/.safe/node/local-node
folaht 76848 1 0 15:49 pts/1 00:00:00 /home/folaht/.safe/node/sn_node -vv --hard-coded-contacts ["83.163.103.119:12001"] --root-dir /home/folaht/.safe/node/local-node --log-dir /home/folaht/.safe/node/local-node
folaht 76861 1 0 15:49 pts/1 00:00:00 /home/folaht/.safe/node/sn_node -vv --hard-coded-contacts ["83.163.103.119:12001"] --root-dir /home/folaht/.safe/node/local-node --log-dir /home/folaht/.safe/node/local-node
folaht 76874 1 0 15:49 pts/1 00:00:00 /home/folaht/.safe/node/sn_node -vv --hard-coded-contacts ["83.163.103.119:12001"] --root-dir /home/folaht/.safe/node/local-node --log-dir /home/folaht/.safe/node/local-node
folaht 76887 1 0 15:49 pts/1 00:00:00 /home/folaht/.safe/node/sn_node -vv --hard-coded-contacts ["83.163.103.119:12001"] --root-dir /home/folaht/.safe/node/local-node --log-dir /home/folaht/.safe/node/local-node
folaht 76900 1 0 15:49 pts/1 00:00:00 /home/folaht/.safe/node/sn_node -vv --hard-coded-contacts ["83.163.103.119:12001"] --root-dir /home/folaht/.safe/node/local-node --log-dir /home/folaht/.safe/node/local-node
folaht 76913 1 0 15:49 pts/1 00:00:00 /home/folaht/.safe/node/sn_node -vv --hard-coded-contacts ["83.163.103.119:12001"] --root-dir /home/folaht/.safe/node/local-node --log-dir /home/folaht/.safe/node/local-node
folaht 76926 1 0 15:49 pts/1 00:00:00 /home/folaht/.safe/node/sn_node -vv --hard-coded-contacts ["83.163.103.119:12001"] --root-dir /home/folaht/.safe/node/local-node --log-dir /home/folaht/.safe/node/local-node
folaht 76935 58028 0 15:49 pts/1 00:00:00 grep sn_node
I think that’s correct, but I believe we’ll need that option in sn_node/qp2p which allows us to manually set the external IP the node shall be publishing itself for others to connect to (which in this case is different from where it’s actually listening on).
Also, it seems the hairpinning is not working/supported in your env as the nodes don’t seem to be joining to the genesis node, I just tried and your genesis is giving me only its own address (and LAN IP):
[2021-02-01T14:52:55Z TRACE sn_client::connection_manager] HandshakeResponse::Join Elders: ([(e3f80c.., 192.168.178.24:12000)])
Perhaps double check the nodes logs to confirm they have trouble joining/connecting to the genesis node.
I guess you can try with any protocol, like ping or ssh/telnet/scp/http, whatever you have available.
Also, it seems the hairpinning is not working/supported in your env as the nodes don’t seem to be joining to the genesis node, I just tried and your genesis is giving me only its own address
I don’t know if that’s from right now or seconds before, when I had started the nodes before I had changed the port forwarding address and discovering I can’t do that.
Can you try again?
I can connect to 12001 port but still getting LAN IPs:
[2021-02-01T15:02:59Z TRACE sn_client::config_handler] New contact added to the hard-coded contacts list: 83.163.103.119:12001
[2021-02-01T15:02:59Z TRACE sn_client::connection_manager] Trying to bootstrap to the network with public_key: PublicKey::Ed25519(8b71b5..)
[2021-02-01T15:02:59Z TRACE sn_client::connection_manager] Bootstrapping with contacts...
[2021-02-01T15:03:00Z TRACE sn_client::connection_manager] Sending handshake request to bootstrapped node...
[2021-02-01T15:03:00Z TRACE sn_client::connection_manager] HandshakeResponse::Join Elders: ([(458d23.., 192.168.178.24:47868), (617542.., 192.168.178.24:36827), (729103.., 192.168.178.24:38763), (9d9f2d.., 192.168.178.24:12000), (c4b01f.., 192.168.178.24:33570)])
If that’s working with hairpinning already, we just need the --external-ip/--external-port arg I was mentioning above I think
I guess you can try with any protocol, like ping or ssh/telnet/scp/http, whatever you have available.
You mean like port forwarding SSH from external 22222 to 22 and then this?
[folaht@pjehrsohmehj safe]$ ssh folaht@83.163.103.119 -p 22222
Welcome to Manjaro-ARM
~~Website: https://manjaro.org
~~Forum: https://forum.manjaro.org/c/arm
~~Matrix: #manjaro-arm-public:matrix.org
Last login: Mon Feb 1 02:30:09 2021 from 192.168.178.23
[folaht@Rezosur-uq ~]$ exit
Yes…I think that’s right, it looks to me like it’s working fine, so you are on the right track I believe, we just need that feature external-ip in the nodes…
Where does one ask for this feature?
sn_node or qp2p?
Let’s wait for https://github.com/maidsafe/qp2p/pull/206 to be completed, it’s is WIP now, then for sure we’ll be having sn-node adapted to it, at that point see how we can make use of those flags from sn-node and launch-tool.
Currently nodes work fine with IPv6: I successfully created a local network using docker containers using IPv6 addresses (connected to a docker bridged IPv6 network).
A limitation is that a client cannot connect to this network (I get: sn_cli error: [Error] AuthdError - [Error] ClientError - Network bootstrap failed). I guess this is a small bug that can be easily corrected because a node can connect to it and both client and node use qp2p services.
The real problem is that a IPv4 node cannot connect to it because each node has only one IP address and so cannot listen to both IPv4 and IPv6. This means that a safe network is either entirely IPv4 or entirely IPv6.
If we ever dream of a network having billions of nodes we need IPv6 but on the other hand removing support for IPv4 might alienate a lot of people.
To have both we need to have a second IP address. This is what is added by this PR, but it defines one local address and one external address instead of 2 external addresses. So, I am afraid it won’t allow mixed nodes managing both IPv4 and IPv6 over the Internet.
I would like my fears proved wrong. Otherwise priorities should be reevaluated because adding support for IPv6 is much more important than allowing several nodes behind the same IPv4 address (and I remember that a long time ago Maidsafe was even trying to prevent this).
We need both IMO. Many nodes able to run behind a NAT device (large groups in school, libraries, small offices, families etc., plus all the bad dudes and Googles etc.) and ipv6. However as you say IPv6 gives us issues a bit like nodes that cannot be contacted by the network. By this I mean:
A node can connect to network nodes - required
Network nodes can connect back to any node - required
So this means we need to make sure nodes are connectable by all other nodes, so it must have a public address that is connectable. If we allow nodes that are not connectable then the network loses balance and many bad things happen (complexity).
So this is only IPv4, if we have ipv6 we then need to say all nodes must be connectable in IPv4 and IPv6, otherwise we get the same imbalance.
The issue is say we just consider voting nodes (Elders), they all need to speak to each other and when a node does not speak to another node it’s accused of non-responsive behaviour and if it does not speak to enough, then it’s voted out. We can imagine we can accept some nodes that only connect out, but it fails in bad ways. Clients need to connect to these nodes a well etc.
So with IPv6 it just adds some complexity, not impossible and needs done, but I fear it’s not and should not be a current priority. Stable upgradable network is priority right now, then IPv6 and more should be on the horizon.
I call them the dark days when we had the unfortunate experience of some network experts that had never designed a network, still argued they were right though 
It cannot work that way in decentralised networks. To ban IP addresses etc. you need global state and then it’s just turtles all the way down. The team now don’t worry themselves with such craziness and are more focussed on real world. So now we say when it rains don’t should stop and cry, instead just put up an umbrella. What I mean is we are as a team now saying, this is how the real world is, we must work with that and not against it. In any case, it’s so much simpler, faster and more secure now
Thanks, I understand the problem.
So currently the network is either entirely IPv4 or entirely IPv6 and adding support for mixed nodes is much more complex than simply adding a second IP address.
What about clients? Can we expect a correction that allows IPv6 clients to connect to a IPv6 network?
No worries, I knew you would see it straight away.
Yes, this is the case. It’s a bit of a PITA, but not unsolvable IMO. There are just some issues and we really need to make sure all nodes don’t need both v4 and v6, but that’s even more complex. IPv6 is great and I wish we all had it but … well you know. This graph is the worry if we force all nodes to have both (or just v6) IPv6 – Google
I feel that the sn_client lib will enable ipv6 when the network can be sure all Elders (at least) are connectable IPv6. Hopefully, this is all invisible to app devs. I am sure it will be.
There may be some clever thing we can do, but just not the bandwidth right now to handle it, typically network looks like you can do something but the side effects are always insane and you end up in a recursively more complex thing. We will defo try though when we are up and running, but hopefully, not proxies/relays etc. as we cannot trust them.
In fact no correction is needed. I found a way to make IPv6 clients connect to the network: Instead of creating ~/.safe/node/node_connection_info.config file, just create ~/.safe/client/sn_client.config file with following content:
{
"hard_coded_contacts": [
"[fd2f:9ab3:8b80:69fa::3]:5483"
],
"ip": "fd2f:9ab3:8b80:69fa::2",
"forward_port": false,
"fresh": true,
"clean": true
}
where:
This means that today an IPv6 network can be created and accessed with CLI commands from an IPv6 client and this works perfectly.
Oh cool. I’m going to do that right now.
[update]
~/.safe/client/sn_client.config ← is this server side or client side?
And is this cli or client?
[update]
Alright, I’ll assume it’s on client side client.
[update]
It’s working on LAN.
[update]
Don’t just give me hearts, test my connection!
Look at my first post again.
Sorry, I get : sn_cli error: Failed to connect: [Error] ConnectionError - Failed to connect to the SAFE Network: InsufficientElderConnections
I mentioned earlier that the network must be entirely IPv6, which means that all nodes must be IPv6. To do that you must recreate the network with the following arguments:
--ip "2001:983:8610:1:854:efb1:52e6:85a3" -p 12000 -f
--ip "2001:983:8610:1:854:efb1:52e6:85a3" -h '["[2001:983:8610:1:854:efb1:52e6:85a3]:12000"]'
I suppose that your nodes have all the same ip address and only differ on the port number which is automatically generated. If not, just specify the right IPv6 address in --ip argument.
Maybe you must also to pass a specific -p parameter for each node and open the corresponding ports.
I’m a bit confused because the other nodes are already IPv6.
[folaht@Rezosur-uq home]$ ps -ef | grep sn_node
folaht 101199 1 44 10:04 pts/1 01:09:51 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 2001:983:8610:1:854:efb1:52e6:85a3 --first --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-genesis --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-genesis
folaht 101207 1 43 10:04 pts/1 01:07:58 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 2001:983:8610:1:854:efb1:52e6:85a3 --hard-coded-contacts ["[2001:983:8610:1:854:efb1:52e6:85a3]:12000"] --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-2 --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-2
folaht 101215 1 43 10:04 pts/1 01:08:15 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 2001:983:8610:1:854:efb1:52e6:85a3 --hard-coded-contacts ["[2001:983:8610:1:854:efb1:52e6:85a3]:12000"] --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-3 --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-3
folaht 101223 1 43 10:04 pts/1 01:07:13 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 2001:983:8610:1:854:efb1:52e6:85a3 --hard-coded-contacts ["[2001:983:8610:1:854:efb1:52e6:85a3]:12000"] --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-4 --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-4
folaht 101232 1 39 10:04 pts/1 01:02:10 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 2001:983:8610:1:854:efb1:52e6:85a3 --hard-coded-contacts ["[2001:983:8610:1:854:efb1:52e6:85a3]:12000"] --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-5 --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-5
folaht 101241 1 30 10:04 pts/1 00:47:15 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 2001:983:8610:1:854:efb1:52e6:85a3 --hard-coded-contacts ["[2001:983:8610:1:854:efb1:52e6:85a3]:12000"] --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-6 --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-6
folaht 101249 1 30 10:04 pts/1 00:47:02 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 2001:983:8610:1:854:efb1:52e6:85a3 --hard-coded-contacts ["[2001:983:8610:1:854:efb1:52e6:85a3]:12000"] --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-7 --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-7
folaht 101257 1 30 10:04 pts/1 00:47:08 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 2001:983:8610:1:854:efb1:52e6:85a3 --hard-coded-contacts ["[2001:983:8610:1:854:efb1:52e6:85a3]:12000"] --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-8 --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-8
folaht 101265 1 20 10:04 pts/1 00:32:13 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 2001:983:8610:1:854:efb1:52e6:85a3 --hard-coded-contacts ["[2001:983:8610:1:854:efb1:52e6:85a3]:12000"] --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-9 --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-9
folaht 101274 1 30 10:04 pts/1 00:47:09 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 2001:983:8610:1:854:efb1:52e6:85a3 --hard-coded-contacts ["[2001:983:8610:1:854:efb1:52e6:85a3]:12000"] --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-10 --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-10
folaht 101282 1 30 10:04 pts/1 00:47:17 /home/folaht/.safe/node/sn_node -vv --idle-timeout-msec 5500 --keep-alive-interval-msec 5 --ip 2001:983:8610:1:854:efb1:52e6:85a3 --hard-coded-contacts ["[2001:983:8610:1:854:efb1:52e6:85a3]:12000"] --root-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-11 --log-dir /home/folaht/.safe/node/baby-fleming-nodes/sn-node-11
folaht 102169 58028 0 12:40 pts/1 00:00:00 grep sn_node
I’ll try ipv6 forwarding all the sn_node ports listed on netstat -tulpn
at port redirection settings on my router.
[update]
They should be opened right now.
Much better now. I get sn_cli error: [Error] ContentNotFound - No FilesContainer found at this addres. Please, change to the right safe URL in the OP and I think it will work.
Alright, try again.
[update]
Something is not going well.
It seems to hang.