First of all a big thank you for your help, allowing me to ssh into your machine to test your case and also for ur patience for a long Hangout before that.
To narrow down the case, I recompiled vaults with a few crust loggings here and there and deployed it on a standalone droplet starting it as a 1st seed node. Then i modified the crust-config on your machine to include only that droplet as the hardcoded contact. I tired with listener on both 5483 and 10000 which you had port forwarded.
With that what i saw is something really strange is happening. While you are totally reachable if i run a listener on those and connect from outside world, the reverse is sort of messing things up. Normally we use the echo service to find out our external IP in crust. So say you wanted to give your ext endpoint of your TCP listener on your forwarded port to the hard-coded contacts to bootstrap. To know your external endpoint, you would connect to any publicly reachable echo/stun servers which would tell you what they see you as (thus your external endpoint). This functionality is built into crust and is the first step. Then you try to bootstrap off the many hard-coded contacts giving them your obtained external endpoint so that they can test your external reachability.
Normally if your listener was say bound to 0.0.0.0:5483 then the 1st step of echo/stun service would let you know your external ip as 109.x.x.x:5483 (whatever public ip you have). However your router (or maybe some firewall) seems to be very nasty. An outgoing connection made via forwarded port seems not be actually mapped to the forwarded port. So the router is remapping outbound connections made from 5483 as random ephimeral port each time for e.g 54899. Now when your crust gives the droplet-crusts your external E.P, it gives 109.x.x.x:54899 which was never externally reachable and thus they are unable to connect to it and you end up failing the external reachability criteria.
If someone already knew your external e.p was 109.x.x.x:5483 somehow (say because you told them so) then there is no problem and your router allows the inbound connection to your listener. But unfortunately it’s not preserving the semantics for outbound connection, remapping it each time as if the forwarded port was nothing special.
Though i will need to think more on this and discuss with others, there are two things that immediately come to my mind:
- You could check if there is some firewall/router config that must be additionally disabled/enabled for this to not happen so that your router behaves normally like others.
- Something we could do in crust config for such scenarios to say include one more field which basically tells crust:
do not try to include port number when you find my external E.P using echo/stun services, but instead use the one I am supplying to you because my router is a special case. So say this field isforce_use_given_port: truein safe_vault.crust.config. In that case crust will use the echo mechanism to know your external E.P, discard the port mentioned there and keep only the IP, and finally fuse that IP with the port mentioned in the crust.config to form the actual E.P and now give this to the bootstrappee nodes. The default could be false because normally UPnP and other router-types would prefer this to be detected naturally in most cases.
Phew, who knew the world was so bad !