D 17-03-03 11:32:48.154323 [crust::main::service service.rs:454] Network name: Some(“test_network_12_c”)
T 17-03-03 11:32:50.330570 [routing::states::bootstrapping bootstrapping.rs:131] Bootstrapping(3cddd7…) Listener started on port 5483.
E 17-03-03 11:32:54.427318 [crust::main::bootstrap mod.rs:185] Failed to Bootstrap: (FailedExternalReachability) Bootstrapee node could not establish connection to us.
I 17-03-03 11:32:54.428098 [routing::states::bootstrapping bootstrapping.rs:216] Bootstrapping(3cddd7…) Failed to bootstrap. Terminating.
D 17-03-03 11:32:54.428120 [routing::state_machine state_machine.rs:268] State::Bootstrapping(3cddd7…) Terminating state machine
ok so it seems I’m not visible from outside. I suppose I have an issue with my firewalld
Looks that way. I guess you’ve tried starting your vault a few times with the same result? I’ve seen this message myself once, but immediately after that without changing any network settings, my vault started fine. (Not sure why that happened yet).
So you didn’t change any firewall settings? I didn’t have to do any fiddling with the Linux firewall for 12b and it wasn’t a problem, worked straight away. This time I get a similar error to @nice .
Yeah, ‘consistently’ meaning I’ve tried it a handful of times and it hasn’t worked, so I’ve wandered off to do something more fun. It works fine on Windows, it’s just Linux I’m struggling with. No big deal, but I just wondered if there was something obvious I’d missed.
Yes I tried in Ubuntu in Openbox and also in Qubes which is VM-based. I’ll try again with a bare-metal version when I have a chance. Funny, though, because it worked last time.
ok before getting crazy with my firewall, I just tried to change sshd port ( which I do see from outside when its on 22 ) , to 5483.
Now I do see 5483 from outside. So I know for sure my router forwards correctly AND 5483 is open and my machine is accepting tcp connections on 5483.
So, I suppose there may be something else than my firewall. I tried setting selinux to permissive, but it doesn’t change anything. I am not on a VM btw, it is a plain native fedora workstation.
Would you be able to show that in HO (hangout) @nice ? I’ll set it up and give you a link, then you share your screen and we can see what’s happening ?
Hm, tried to start now for like 5-6 Times.
My windows firewall already has an exception added.
Does someone else have an idea/fix?
Thanks.
I 17-03-03 18:30:08.474525 [safe_vault safe_vault.rs:95]
Running safe_vault v0.13.1
D 17-03-03 18:30:08.475528 [crust::main::service service.rs:454] Network name: Some(“test_network_12_c”)
T 17-03-03 18:30:13.482180 [routing::states::bootstrapping bootstrapping.rs:131] Bootstrapping(59f7c3…) Listener started on port 5483.
E 17-03-03 18:30:17.511596 [crust::main::bootstrap mod.rs:185] Failed to Bootstrap: (FailedExternalReachability) Bootstrapee node could not establish connection to us.
I 17-03-03 18:30:17.511596 [routing::states::bootstrapping bootstrapping.rs:216] Bootstrapping(59f7c3…) Failed to bootstrap. Terminating.
D 17-03-03 18:30:17.512599 [routing::state_machine state_machine.rs:268] State::Bootstrapping(59f7c3…) Terminating state machine
First of all a big thank you for your help, allowing me to ssh into your machine to test your case and also for ur patience for a long Hangout before that.
To narrow down the case, I recompiled vaults with a few crust loggings here and there and deployed it on a standalone droplet starting it as a 1st seed node. Then i modified the crust-config on your machine to include only that droplet as the hardcoded contact. I tired with listener on both 5483 and 10000 which you had port forwarded.
With that what i saw is something really strange is happening. While you are totally reachable if i run a listener on those and connect from outside world, the reverse is sort of messing things up. Normally we use the echo service to find out our external IP in crust. So say you wanted to give your ext endpoint of your TCP listener on your forwarded port to the hard-coded contacts to bootstrap. To know your external endpoint, you would connect to any publicly reachable echo/stun servers which would tell you what they see you as (thus your external endpoint). This functionality is built into crust and is the first step. Then you try to bootstrap off the many hard-coded contacts giving them your obtained external endpoint so that they can test your external reachability.
Normally if your listener was say bound to 0.0.0.0:5483 then the 1st step of echo/stun service would let you know your external ip as 109.x.x.x:5483 (whatever public ip you have). However your router (or maybe some firewall) seems to be very nasty. An outgoing connection made via forwarded port seems not be actually mapped to the forwarded port. So the router is remapping outbound connections made from 5483 as random ephimeral port each time for e.g 54899. Now when your crust gives the droplet-crusts your external E.P, it gives 109.x.x.x:54899 which was never externally reachable and thus they are unable to connect to it and you end up failing the external reachability criteria.
If someone already knew your external e.p was 109.x.x.x:5483 somehow (say because you told them so) then there is no problem and your router allows the inbound connection to your listener. But unfortunately it’s not preserving the semantics for outbound connection, remapping it each time as if the forwarded port was nothing special.
Though i will need to think more on this and discuss with others, there are two things that immediately come to my mind:
You could check if there is some firewall/router config that must be additionally disabled/enabled for this to not happen so that your router behaves normally like others.
Something we could do in crust config for such scenarios to say include one more field which basically tells crust: do not try to include port number when you find my external E.P using echo/stun services, but instead use the one I am supplying to you because my router is a special case. So say this field is force_use_given_port: true in safe_vault.crust.config. In that case crust will use the echo mechanism to know your external E.P, discard the port mentioned there and keep only the IP, and finally fuse that IP with the port mentioned in the crust.config to form the actual E.P and now give this to the bootstrappee nodes. The default could be false because normally UPnP and other router-types would prefer this to be detected naturally in most cases.
Thank you for taking the time to look into this. I tried to find some info about this on my particular router ( TG788vn from my isp ) , but their documentation is , mhh, bad !
I have no clue why it would rewrite outbound ports , or why my firewall would do that.
I could try to run the vault from another machine, too, maybe that could tell if this one’s rules are faulty.
EDIT : my laptop behaves just the same, with standard firewall rules out of fedora install - meaning I never tinkered with that but opening the port.
So…
reading about this now