Node Manager UX and Issues

aatonnomicc · April 10, 2024, 3:22pm

luckily I still had it in the same terminal window
also I stopped my resource gathering script during the update as i wasn’t sure if running safenode-manager status during the upgrade would bork the node registry ?

ubuntu@s13:~$ =================================================
           Upgrade Safenode Services
=================================================
Download completed: /tmp/971cc09d-ed5e-4257-bd7e-cc9fa3ae8eae/safenode
Refreshing the node registry...
Attempting to stop safenode1...
✓ Service safenode1 with PID 7471 was stopped
Attempting to start safenode1...
✓ Started safenode1 service
  - PID: 2120595
  - Bin path: /var/safenode-manager/services/safenode1/safenode
  - Data path: /var/safenode-manager/services/safenode1
  - Logs path: /var/log/safenode/safenode1
Attempting to stop safenode2...
✓ Service safenode2 with PID 7928 was stopped
Attempting to start safenode2...
✓ Started safenode2 service
  - PID: 2120756
  - Bin path: /var/safenode-manager/services/safenode2/safenode
  - Data path: /var/safenode-manager/services/safenode2
  - Logs path: /var/log/safenode/safenode2
Attempting to stop safenode3...
✓ Service safenode3 with PID 8499 was stopped
Attempting to start safenode3...
✓ Started safenode3 service
  - PID: 2120834
  - Bin path: /var/safenode-manager/services/safenode3/safenode
  - Data path: /var/safenode-manager/services/safenode3
  - Logs path: /var/log/safenode/safenode3
Attempting to stop safenode4...
✓ Service safenode4 with PID 9162 was stopped
Attempting to start safenode4...
✓ Started safenode4 service
  - PID: 2120945
  - Bin path: /var/safenode-manager/services/safenode4/safenode
  - Data path: /var/safenode-manager/services/safenode4
  - Logs path: /var/log/safenode/safenode4
Attempting to stop safenode5...
✓ Service safenode5 with PID 9928 was stopped
Attempting to start safenode5...
✓ Started safenode5 service
  - PID: 2121049
  - Bin path: /var/safenode-manager/services/safenode5/safenode
  - Data path: /var/safenode-manager/services/safenode5
  - Logs path: /var/log/safenode/safenode5
Attempting to stop safenode6...
✓ Service safenode6 with PID 10794 was stopped
Attempting to start safenode6...
✓ Started safenode6 service
  - PID: 2121127
  - Bin path: /var/safenode-manager/services/safenode6/safenode
  - Data path: /var/safenode-manager/services/safenode6
  - Logs path: /var/log/safenode/safenode6
Attempting to stop safenode7...
✓ Service safenode7 with PID 11761 was stopped
Attempting to start safenode7...
✓ Started safenode7 service
  - PID: 2121204
  - Bin path: /var/safenode-manager/services/safenode7/safenode
  - Data path: /var/safenode-manager/services/safenode7
  - Logs path: /var/log/safenode/safenode7
Attempting to stop safenode8...
✓ Service safenode8 with PID 12843 was stopped
Attempting to start safenode8...
✓ Started safenode8 service
  - PID: 2121282
  - Bin path: /var/safenode-manager/services/safenode8/safenode
  - Data path: /var/safenode-manager/services/safenode8
  - Logs path: /var/log/safenode/safenode8
Attempting to stop safenode9...
✓ Service safenode9 with PID 14011 was stopped
Attempting to start safenode9...
✓ Started safenode9 service
  - PID: 2121359
  - Bin path: /var/safenode-manager/services/safenode9/safenode
  - Data path: /var/safenode-manager/services/safenode9
  - Logs path: /var/log/safenode/safenode9
Attempting to stop safenode10...
✓ Service safenode10 with PID 15284 was stopped
Attempting to start safenode10...
✓ Started safenode10 service
  - PID: 2121437
  - Bin path: /var/safenode-manager/services/safenode10/safenode
  - Data path: /var/safenode-manager/services/safenode10
  - Logs path: /var/log/safenode/safenode10
Attempting to stop safenode11...
✓ Service safenode11 with PID 16661 was stopped
Attempting to start safenode11...
✓ Started safenode11 service
  - PID: 2121514
  - Bin path: /var/safenode-manager/services/safenode11/safenode
  - Data path: /var/safenode-manager/services/safenode11
  - Logs path: /var/log/safenode/safenode11
Attempting to stop safenode12...
✓ Service safenode12 with PID 18144 was stopped
Attempting to start safenode12...
✓ Started safenode12 service
  - PID: 2121600
  - Bin path: /var/safenode-manager/services/safenode12/safenode
  - Data path: /var/safenode-manager/services/safenode12
  - Logs path: /var/log/safenode/safenode12
Attempting to stop safenode13...
✓ Service safenode13 with PID 19739 was stopped
Attempting to start safenode13...
✓ Started safenode13 service
  - PID: 2121677
  - Bin path: /var/safenode-manager/services/safenode13/safenode
  - Data path: /var/safenode-manager/services/safenode13
  - Logs path: /var/log/safenode/safenode13
Attempting to stop safenode14...
✓ Service safenode14 with PID 22111 was stopped
Attempting to start safenode14...
✓ Started safenode14 service
  - PID: 2121755
  - Bin path: /var/safenode-manager/services/safenode14/safenode
  - Data path: /var/safenode-manager/services/safenode14
  - Logs path: /var/log/safenode/safenode14
Attempting to stop safenode15...
✓ Service safenode15 with PID 23895 was stopped
Attempting to start safenode15...
✓ Started safenode15 service
  - PID: 2121834
  - Bin path: /var/safenode-manager/services/safenode15/safenode
  - Data path: /var/safenode-manager/services/safenode15
  - Logs path: /var/log/safenode/safenode15
Attempting to stop safenode16...
✓ Service safenode16 with PID 25790 was stopped
Attempting to start safenode16...
✓ Started safenode16 service
  - PID: 2121911
  - Bin path: /var/safenode-manager/services/safenode16/safenode
  - Data path: /var/safenode-manager/services/safenode16
  - Logs path: /var/log/safenode/safenode16
Attempting to stop safenode17...
✓ Service safenode17 with PID 27779 was stopped
Attempting to start safenode17...
✓ Started safenode17 service
  - PID: 2122014
  - Bin path: /var/safenode-manager/services/safenode17/safenode
  - Data path: /var/safenode-manager/services/safenode17
  - Logs path: /var/log/safenode/safenode17
Attempting to stop safenode18...
✓ Service safenode18 with PID 29876 was stopped
Attempting to start safenode18...
✓ Started safenode18 service
  - PID: 2122091
  - Bin path: /var/safenode-manager/services/safenode18/safenode
  - Data path: /var/safenode-manager/services/safenode18
  - Logs path: /var/log/safenode/safenode18
Attempting to stop safenode19...
✓ Service safenode19 with PID 32062 was stopped
Attempting to start safenode19...
✓ Started safenode19 service
  - PID: 2122168
  - Bin path: /var/safenode-manager/services/safenode19/safenode
  - Data path: /var/safenode-manager/services/safenode19
  - Logs path: /var/log/safenode/safenode19
Attempting to stop safenode20...
✓ Service safenode20 with PID 34362 was stopped
Attempting to start safenode20...
✓ Started safenode20 service
  - PID: 2122245
  - Bin path: /var/safenode-manager/services/safenode20/safenode
  - Data path: /var/safenode-manager/services/safenode20
  - Logs path: /var/log/safenode/safenode20
Attempting to stop safenode21...
✓ Service safenode21 with PID 36761 was stopped
Attempting to start safenode21...
✓ Started safenode21 service
  - PID: 2122322
  - Bin path: /var/safenode-manager/services/safenode21/safenode
  - Data path: /var/safenode-manager/services/safenode21
  - Logs path: /var/log/safenode/safenode21
Attempting to stop safenode22...
✓ Service safenode22 with PID 39258 was stopped
Attempting to start safenode22...
✓ Started safenode22 service
  - PID: 2122399
  - Bin path: /var/safenode-manager/services/safenode22/safenode
  - Data path: /var/safenode-manager/services/safenode22
  - Logs path: /var/log/safenode/safenode22
Attempting to stop safenode23...
✓ Service safenode23 with PID 41872 was stopped
Attempting to start safenode23...
✓ Started safenode23 service
  - PID: 2122476
  - Bin path: /var/safenode-manager/services/safenode23/safenode
  - Data path: /var/safenode-manager/services/safenode23
  - Logs path: /var/log/safenode/safenode23
Attempting to stop safenode24...
✓ Service safenode24 with PID 44574 was stopped
Attempting to start safenode24...
✓ Started safenode24 service
  - PID: 2122553
  - Bin path: /var/safenode-manager/services/safenode24/safenode
  - Data path: /var/safenode-manager/services/safenode24
  - Logs path: /var/log/safenode/safenode24
Attempting to stop safenode25...
✓ Service safenode25 with PID 47375 was stopped
Attempting to start safenode25...
✓ Started safenode25 service
  - PID: 2122630
  - Bin path: /var/safenode-manager/services/safenode25/safenode
  - Data path: /var/safenode-manager/services/safenode25
  - Logs path: /var/log/safenode/safenode25
Attempting to stop safenode26...
✓ Service safenode26 with PID 50284 was stopped
Attempting to start safenode26...
Attempting to stop safenode27...
✓ Service safenode27 with PID 53291 was stopped
Attempting to start safenode27...
Attempting to stop safenode28...
✓ Service safenode28 with PID 56407 was stopped
Attempting to start safenode28...
✓ Started safenode28 service
  - PID: 2122889
  - Bin path: /var/safenode-manager/services/safenode28/safenode
  - Data path: /var/safenode-manager/services/safenode28
  - Logs path: /var/log/safenode/safenode28
Attempting to stop safenode29...
✓ Service safenode29 with PID 59622 was stopped
Attempting to start safenode29...
✓ Started safenode29 service
  - PID: 2122967
  - Bin path: /var/safenode-manager/services/safenode29/safenode
  - Data path: /var/safenode-manager/services/safenode29
  - Logs path: /var/log/safenode/safenode29
Attempting to stop safenode30...
✓ Service safenode30 with PID 62930 was stopped
Attempting to start safenode30...
Attempting to stop safenode31...
✓ Service safenode31 with PID 66345 was stopped
Attempting to start safenode31...
✓ Started safenode31 service
  - PID: 2123124
  - Bin path: /var/safenode-manager/services/safenode31/safenode
  - Data path: /var/safenode-manager/services/safenode31
  - Logs path: /var/log/safenode/safenode31
Attempting to stop safenode32...
✓ Service safenode32 with PID 69859 was stopped
Attempting to start safenode32...
Attempting to stop safenode33...
✓ Service safenode33 with PID 73488 was stopped
Attempting to start safenode33...
✓ Started safenode33 service
  - PID: 2123279
  - Bin path: /var/safenode-manager/services/safenode33/safenode
  - Data path: /var/safenode-manager/services/safenode33
  - Logs path: /var/log/safenode/safenode33
Attempting to stop safenode34...
✓ Service safenode34 with PID 77202 was stopped
Attempting to start safenode34...
Attempting to stop safenode35...
✓ Service safenode35 with PID 81042 was stopped
Attempting to start safenode35...
✓ Started safenode35 service
  - PID: 2123458
  - Bin path: /var/safenode-manager/services/safenode35/safenode
  - Data path: /var/safenode-manager/services/safenode35
  - Logs path: /var/log/safenode/safenode35
Attempting to stop safenode36...
✓ Service safenode36 with PID 84959 was stopped
Attempting to start safenode36...
Attempting to stop safenode37...
✓ Service safenode37 with PID 88985 was stopped
Attempting to start safenode37...
Attempting to stop safenode38...
✓ Service safenode38 with PID 93111 was stopped
Attempting to start safenode38...
Attempting to stop safenode39...
✓ Service safenode39 with PID 97346 was stopped
Attempting to start safenode39...
✓ Started safenode39 service
  - PID: 2123774
  - Bin path: /var/safenode-manager/services/safenode39/safenode
  - Data path: /var/safenode-manager/services/safenode39
  - Logs path: /var/log/safenode/safenode39
Attempting to stop safenode40...
✓ Service safenode40 with PID 101683 was stopped
Attempting to start safenode40...
✓ Started safenode40 service
  - PID: 2123851
  - Bin path: /var/safenode-manager/services/safenode40/safenode
  - Data path: /var/safenode-manager/services/safenode40
  - Logs path: /var/log/safenode/safenode40
Attempting to stop safenode41...
✓ Service safenode41 with PID 106110 was stopped
Attempting to start safenode41...
Attempting to stop safenode42...
✓ Service safenode42 with PID 110634 was stopped
Attempting to start safenode42...
Attempting to stop safenode43...
✓ Service safenode43 with PID 115272 was stopped
Attempting to start safenode43...
Attempting to stop safenode44...
✓ Service safenode44 with PID 120010 was stopped
Attempting to start safenode44...
Attempting to stop safenode45...
✓ Service safenode45 with PID 124834 was stopped
Attempting to start safenode45...
Attempting to stop safenode46...
✓ Service safenode46 with PID 129781 was stopped
Attempting to start safenode46...
Attempting to stop safenode47...
✓ Service safenode47 with PID 134802 was stopped
Attempting to start safenode47...
Attempting to stop safenode48...
✓ Service safenode48 with PID 139946 was stopped
Attempting to start safenode48...
✓ Started safenode48 service
  - PID: 2124523
  - Bin path: /var/safenode-manager/services/safenode48/safenode
  - Data path: /var/safenode-manager/services/safenode48
  - Logs path: /var/log/safenode/safenode48
Attempting to stop safenode49...
✓ Service safenode49 with PID 145181 was stopped
Attempting to start safenode49...
Attempting to stop safenode50...
✓ Service safenode50 with PID 150521 was stopped
Attempting to start safenode50...
Upgrade summary:
✓ safenode1 upgraded from 0.105.2 to 0.105.6
✓ safenode2 upgraded from 0.105.2 to 0.105.6
✓ safenode3 upgraded from 0.105.2 to 0.105.6
✓ safenode4 upgraded from 0.105.2 to 0.105.6
✓ safenode5 upgraded from 0.105.2 to 0.105.6
✓ safenode6 upgraded from 0.105.2 to 0.105.6
✓ safenode7 upgraded from 0.105.2 to 0.105.6
✓ safenode8 upgraded from 0.105.2 to 0.105.6
✓ safenode9 upgraded from 0.105.2 to 0.105.6
✓ safenode10 upgraded from 0.105.2 to 0.105.6
✓ safenode11 upgraded from 0.105.2 to 0.105.6
✓ safenode12 upgraded from 0.105.2 to 0.105.6
✓ safenode13 upgraded from 0.105.2 to 0.105.6
✓ safenode14 upgraded from 0.105.2 to 0.105.6
✓ safenode15 upgraded from 0.105.2 to 0.105.6
✓ safenode16 upgraded from 0.105.2 to 0.105.6
✓ safenode17 upgraded from 0.105.2 to 0.105.6
✓ safenode18 upgraded from 0.105.2 to 0.105.6
✓ safenode19 upgraded from 0.105.2 to 0.105.6
✓ safenode20 upgraded from 0.105.2 to 0.105.6
✓ safenode21 upgraded from 0.105.2 to 0.105.6
✓ safenode22 upgraded from 0.105.2 to 0.105.6
✓ safenode23 upgraded from 0.105.2 to 0.105.6
✓ safenode24 upgraded from 0.105.2 to 0.105.6
✓ safenode25 upgraded from 0.105.2 to 0.105.6
✕ safenode26 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:46563'
✕ safenode27 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:34315'
✓ safenode28 upgraded from 0.105.2 to 0.105.6
✓ safenode29 upgraded from 0.105.2 to 0.105.6
✕ safenode30 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:45593'
✓ safenode31 upgraded from 0.105.2 to 0.105.6
✕ safenode32 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:39499'
✓ safenode33 upgraded from 0.105.2 to 0.105.6
✕ safenode34 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:40425'
✓ safenode35 upgraded from 0.105.2 to 0.105.6
✕ safenode36 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:41639'
✕ safenode37 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:37865'
✕ safenode38 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:41659'
✓ safenode39 upgraded from 0.105.2 to 0.105.6
✓ safenode40 upgraded from 0.105.2 to 0.105.6
✕ safenode41 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:34799'
✕ safenode42 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:46689'
✕ safenode43 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:44065'
✕ safenode44 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:43509'
✕ safenode45 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:41283'
✕ safenode46 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:43777'
✕ safenode47 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:43419'
✓ safenode48 upgraded from 0.105.2 to 0.105.6
✕ safenode49 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:42031'
✕ safenode50 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:38275'
*/5 * * * * ubuntu /usr/bin/mkdir -p /tmp/influx-resources && /bin/bash /usr/bin/influx-resources.sh > /tmp/influx-resources/influx-resources

chriso · April 10, 2024, 3:27pm

Cool, thanks. Yeah, for the services reported as not upgraded, they have not started again. That actually doesn’t mean the binary wasn’t upgraded. Probably need to re-word that error reporting.

As for mixing status and upgrade, yeah, probably shouldn’t do that. Though the name implies that status is read only, it actually isn’t. It refreshes the state of all the nodes at the time it was ran, then writes that status back out.

aatonnomicc · April 10, 2024, 3:31pm

yea makes sense also as processor was maxed out at 100% I’m thinking a 5 min delay between node upgrades wont hurt.

other small point the machines I upgraded early doors about 11:00 today did not have this issue so not sure if its related to the current condition of the network and its putting more load on the node as they try to get connected ?

chriso · April 10, 2024, 3:34pm

5 minutes between each node, when you have over 50? Isn’t that a bit excessive? I wasn’t aware that safenode was really all that CPU intensive.

aatonnomicc · April 10, 2024, 3:38pm

its not normally but for joining large amounts of nodes I have been using 10 min and I think @shu has been using 7min after doing some analysis of what was going on with network traffic and CPU usage etc.

chriso · April 10, 2024, 3:43pm

Hmm, it would be good to get more details on this, because that just seems excessive. I’m upgrading the nodes on our machines now and so far I’ve not had any errors yet. We have delays at the level of the VMs, but not the nodes on the VM. Unless the errors aren’t coming back properly, but that shouldn’t be the case.

aatonnomicc · April 10, 2024, 3:45pm

ok if you are not seeing any errors on your machines ill try another machine now and report back how it goes

chriso · April 10, 2024, 3:46pm

Well, I mean, I think using a small delay of maybe like 2 - 5 seconds would be reasonable, but 5 minutes in between each node just seems insane. If that’s necessary, we’d need to find what’s causing that and do something about it.

aatonnomicc · April 10, 2024, 4:01pm

5 min was just the first number that I picked out my head as a starting point to work back down from.

using safenode-manager 0.7.4

on 4 vcpu 6gb ram 80 gb hard disk cloud node with 100Mbs connection

before upgrade process

ubuntu@s14:~$ safenode-manager status
=================================================
                Safenode Services
=================================================
Refreshing the node registry...
Service Name       Peer ID                                              Status  Connected Peers
safenode1          12D3KooWFcTgo26WNQAhghmSb3Lg47hU14Wn2g3ny4xo62TRczJZ RUNNING              18
safenode2          12D3KooWM7dvCKCoXzXAZGHWAFSWmaBW6P6T4BRPpEGwzPHuuGkT RUNNING              16
safenode3          12D3KooWQk626R3Tjm7Vwk7VdoYvkw3wVAzkh9jrMgKUANEhdm92 RUNNING              22
safenode4          12D3KooWBgpnQztjN7FkTaNrAdF4UFjp8UvuDUVi8xMWfCLzSKSC RUNNING              22
safenode5          12D3KooWPznwEb5zGuzcZ5BKDcYxucmFZeMTAE8AhDZE4Eg6vTm2 RUNNING              16
safenode6          12D3KooW9zQDrG5ZsYZZwPL6HRGkniKrK1EKaoCHyunmU4BQ3RvB RUNNING              18
safenode7          12D3KooWM3ECDqQD7FuLcPtNYYwa7bbSeLSuaQKitK5FR7gXutNv RUNNING              20
safenode8          12D3KooWSu6LFGpDmYvuqG1ujVwRKZQACSBM2zYDnPwNqJ7gDdf2 RUNNING              15
safenode9          12D3KooWEWQ41A8D7qfFyrUctaioVhe3qWwmjW6dbHD9fcYgN4qu RUNNING              22
safenode10         12D3KooWSZ8YZYqLQqN2iYLcrHEqkWKJucLBnosQMd5qxAAbEL6T RUNNING              21
safenode11         12D3KooWMWYV2NufjuCkGz7sUYwX9eJxv2mjCk8dNMD6ziq8toKX RUNNING              15
safenode12         12D3KooWAYryyU4Q6PKUBtRzJTo3e53hn9eLYY7YcW2hq4cJRScq RUNNING              24
safenode13         12D3KooWHDjffako1ASmfnnDuMFretjeo3MsUv2sEqCeDkfx7Sdf RUNNING             187
safenode14         12D3KooWHSxA1K5PRJpYM4YX5RsUvnVoHNcDKyS8dMS11iyetyM4 RUNNING              18
safenode15         12D3KooWHACMf1MtWRCGtSACcACWkqm8MXBvEQFm4GiEmEU9Wk78 RUNNING             206
safenode16         12D3KooWLaY5hX8yzJhLXDwgcBDoEJHHbiY3ZqzyiSWaumWtZZXZ RUNNING              16
safenode17         12D3KooWEoHxStkEJGdfommhBA1mMwYUPBReYAceWeTDgH1LiNgn RUNNING             187
safenode18         12D3KooWH8ypRARh8uiNSuixTY2yUcLMym9MuwDqwDBHUwB6hVNv RUNNING              22
safenode19         12D3KooWCFNGxvqZ3ssT21wRQntfBW4gcvhLjovH5Cv1YAAVH3tX RUNNING              76
safenode20         12D3KooWHDYCbquRMYRaKW5pV1bJT5EwUsciY46bv9667dHUYuPk RUNNING             227
safenode21         12D3KooWHKVZUM3Zj9omZhMvCFbGYYJpmRtHQPMA8QenSaX2wMXV RUNNING             145
safenode22         12D3KooWJy6KLS4YpZmjBcE2mVvtsVNMhEkFibkeGn1CDRr12H4F RUNNING              21
safenode23         12D3KooWRuLPMPBi3ko8PHdWj3ZHtLtE2nwU61BRR8a3x4Jbfy1V RUNNING              15
safenode24         12D3KooWBM9tHGGV67GGAPeheraEv6jNj3XfCjiE5mKb5fzd87wV RUNNING              20
safenode25         12D3KooWM1mrj5CRiztsU3ykvqedjb36g2w5B4XTgc7efTDNhzV6 RUNNING              17
safenode26         12D3KooWLYQYyk685QyG3Lw3HMvmSboNF88FQswVFhJB6dXvpU92 RUNNING              20
safenode27         12D3KooWT2SrwcAGCB8QwsZZn8n7g3T3s3ufPZ9rt8ztqG5J35ic RUNNING              24
safenode28         12D3KooWLmaBJoLFVN1K7DkjhbWzovkfx6hDq9c8mKMxvBQYnfUM RUNNING              20
safenode29         12D3KooWJLaid3joXxVqhVGVfU3fvTkTTNW425jAvr4d3u5gQLxH RUNNING              13
safenode30         12D3KooWT1fwRRoeWHjsW3ood8tjsyZbYoBaGKhBRPfUwkvcJMfM RUNNING              17
safenode31         12D3KooWRr83px91Ea9ui5otMfDzqrTSFkbjP5sh2cKhfvCBbe3S RUNNING             221
safenode32         12D3KooWLkgdjGWYWjzLQqYqqeeyrLhGswriqzCGxvGSLFNP7yGT RUNNING               9
safenode33         12D3KooWLvSz17YjFqJS3pd8LpeM5rnJDyB5RjRZEMuEGWsDumws RUNNING              17
safenode34         12D3KooWNdGLviVPDHTWfKtf4Zvgj2UYDYYFee6R2oQnPXUtcdeF RUNNING              24
safenode35         12D3KooWP38JR9Guat4jYhJQC3mL73o5XiEKnR7zkBpfqZUE2XxY RUNNING              19
safenode36         12D3KooWQdycUNAkhQSB6ezr63Byh5957rTqfAi91GseQRfoGEUa RUNNING              22
safenode37         12D3KooWR1vNyvVpcD5yKref1vLNWH9NdJkkFjdb8oWHCQvD1Qv4 RUNNING              17
safenode38         12D3KooWNQLxNJJt7Se1VjKTSyzVR6kt5xe7RVwkn8T694cZibWy RUNNING              18
safenode39         12D3KooWBkmJaJewarB4KvZVL9hr5udjm77QkVX9Uv9FXFrMAsnk RUNNING             206
safenode40         12D3KooWR2gaypdyXRZhZX4SR425vVnYeaJBgES8HGgRtKK8B96k RUNNING              25
safenode41         12D3KooWFJ67GVpQv7dXdJmoSmZKUtR1Tn9sxUXA6aXndu2feo9i RUNNING              22
safenode42         12D3KooWHcGXQCwjB3uE3hybLTB2nHhnqhWT1pzGjzuiJhmaNCXd RUNNING              21
safenode43         12D3KooWFHHondSBZ3zKqZwX1CreamqVBTLsWADQHVQBo9cCXkkZ RUNNING              20
safenode44         12D3KooWM7mwTtfJPkQb4eG3d8JGHvyNdS93CmV2qKn2eAT3zLp7 RUNNING             183
safenode45         12D3KooWFpFHe7B7X7fUQqmVXyycTbGRoaNSMGNenGbEByErWAsp RUNNING              80
safenode46         12D3KooWLYNGoyjSK5RkaJK2tdUPnCN1MkfAzag8FK5bV2i5MYws RUNNING              16
safenode47         12D3KooWSVy89A3AotHfzotD2xSVg78k4CaWsoKbm5L62h6gKPEU RUNNING              13
safenode48         12D3KooWCwv7jDtagdATqpidqFeapGMeLvvJoE33GZq6dcHQ8dGF RUNNING              90
safenode49         12D3KooWN3oNxwh5aJecNR1yhiS21jqX2rng4x3LETQT3eWWXLef RUNNING              16
safenode50         12D3KooWPyJp3QRWWrog1EUfzH3necP3w46FKvTkaWnj26djghEr RUNNING              72

same result again

ubuntu@s14:~$ =================================================
           Upgrade Safenode Services
=================================================
Download completed: /tmp/f7b7fe5d-d7d5-4bf3-b69c-d235782b4424/safenode
Refreshing the node registry...
Attempting to stop safenode1...
✓ Service safenode1 with PID 7248 was stopped
Attempting to start safenode1...
✓ Started safenode1 service
  - PID: 2147482
  - Bin path: /var/safenode-manager/services/safenode1/safenode
  - Data path: /var/safenode-manager/services/safenode1
  - Logs path: /var/log/safenode/safenode1
Attempting to stop safenode2...
✓ Service safenode2 with PID 7706 was stopped
Attempting to start safenode2...
✓ Started safenode2 service
  - PID: 2147588
  - Bin path: /var/safenode-manager/services/safenode2/safenode
  - Data path: /var/safenode-manager/services/safenode2
  - Logs path: /var/log/safenode/safenode2
Attempting to stop safenode3...
✓ Service safenode3 with PID 8264 was stopped
Attempting to start safenode3...
✓ Started safenode3 service
  - PID: 2147666
  - Bin path: /var/safenode-manager/services/safenode3/safenode
  - Data path: /var/safenode-manager/services/safenode3
  - Logs path: /var/log/safenode/safenode3
Attempting to stop safenode4...
✓ Service safenode4 with PID 8930 was stopped
Attempting to start safenode4...
✓ Started safenode4 service
  - PID: 2147791
  - Bin path: /var/safenode-manager/services/safenode4/safenode
  - Data path: /var/safenode-manager/services/safenode4
  - Logs path: /var/log/safenode/safenode4
Attempting to stop safenode5...
✓ Service safenode5 with PID 9694 was stopped
Attempting to start safenode5...
✓ Started safenode5 service
  - PID: 2147869
  - Bin path: /var/safenode-manager/services/safenode5/safenode
  - Data path: /var/safenode-manager/services/safenode5
  - Logs path: /var/log/safenode/safenode5
Attempting to stop safenode6...
✓ Service safenode6 with PID 10572 was stopped
Attempting to start safenode6...
✓ Started safenode6 service
  - PID: 2147960
  - Bin path: /var/safenode-manager/services/safenode6/safenode
  - Data path: /var/safenode-manager/services/safenode6
  - Logs path: /var/log/safenode/safenode6
Attempting to stop safenode7...
✓ Service safenode7 with PID 11541 was stopped
Attempting to start safenode7...
✓ Started safenode7 service
  - PID: 2148037
  - Bin path: /var/safenode-manager/services/safenode7/safenode
  - Data path: /var/safenode-manager/services/safenode7
  - Logs path: /var/log/safenode/safenode7
Attempting to stop safenode8...
✓ Service safenode8 with PID 12611 was stopped
Attempting to start safenode8...
✓ Started safenode8 service
  - PID: 2148124
  - Bin path: /var/safenode-manager/services/safenode8/safenode
  - Data path: /var/safenode-manager/services/safenode8
  - Logs path: /var/log/safenode/safenode8
Attempting to stop safenode9...
✓ Service safenode9 with PID 13784 was stopped
Attempting to start safenode9...
✓ Started safenode9 service
  - PID: 2148201
  - Bin path: /var/safenode-manager/services/safenode9/safenode
  - Data path: /var/safenode-manager/services/safenode9
  - Logs path: /var/log/safenode/safenode9
Attempting to stop safenode10...
✓ Service safenode10 with PID 15061 was stopped
Attempting to start safenode10...
✓ Started safenode10 service
  - PID: 2148278
  - Bin path: /var/safenode-manager/services/safenode10/safenode
  - Data path: /var/safenode-manager/services/safenode10
  - Logs path: /var/log/safenode/safenode10
Attempting to stop safenode11...
✓ Service safenode11 with PID 16438 was stopped
Attempting to start safenode11...
✓ Started safenode11 service
  - PID: 2148355
  - Bin path: /var/safenode-manager/services/safenode11/safenode
  - Data path: /var/safenode-manager/services/safenode11
  - Logs path: /var/log/safenode/safenode11
Attempting to stop safenode12...
✓ Service safenode12 with PID 17918 was stopped
Attempting to start safenode12...
✓ Started safenode12 service
  - PID: 2148433
  - Bin path: /var/safenode-manager/services/safenode12/safenode
  - Data path: /var/safenode-manager/services/safenode12
  - Logs path: /var/log/safenode/safenode12
Attempting to stop safenode13...
✓ Service safenode13 with PID 19494 was stopped
Attempting to start safenode13...
✓ Started safenode13 service
  - PID: 2148511
  - Bin path: /var/safenode-manager/services/safenode13/safenode
  - Data path: /var/safenode-manager/services/safenode13
  - Logs path: /var/log/safenode/safenode13
Attempting to stop safenode14...
✓ Service safenode14 with PID 21184 was stopped
Attempting to start safenode14...
✓ Started safenode14 service
  - PID: 2148588
  - Bin path: /var/safenode-manager/services/safenode14/safenode
  - Data path: /var/safenode-manager/services/safenode14
  - Logs path: /var/log/safenode/safenode14
Attempting to stop safenode15...
✓ Service safenode15 with PID 22975 was stopped
Attempting to start safenode15...
✓ Started safenode15 service
  - PID: 2148665
  - Bin path: /var/safenode-manager/services/safenode15/safenode
  - Data path: /var/safenode-manager/services/safenode15
  - Logs path: /var/log/safenode/safenode15
Attempting to stop safenode16...
✓ Service safenode16 with PID 24864 was stopped
Attempting to start safenode16...
✓ Started safenode16 service
  - PID: 2148749
  - Bin path: /var/safenode-manager/services/safenode16/safenode
  - Data path: /var/safenode-manager/services/safenode16
  - Logs path: /var/log/safenode/safenode16
Attempting to stop safenode17...
✓ Service safenode17 with PID 26851 was stopped
Attempting to start safenode17...
✓ Started safenode17 service
  - PID: 2148832
  - Bin path: /var/safenode-manager/services/safenode17/safenode
  - Data path: /var/safenode-manager/services/safenode17
  - Logs path: /var/log/safenode/safenode17
Attempting to stop safenode18...
✓ Service safenode18 with PID 28952 was stopped
Attempting to start safenode18...
✓ Started safenode18 service
  - PID: 2148922
  - Bin path: /var/safenode-manager/services/safenode18/safenode
  - Data path: /var/safenode-manager/services/safenode18
  - Logs path: /var/log/safenode/safenode18
Attempting to stop safenode19...
✓ Service safenode19 with PID 31145 was stopped
Attempting to start safenode19...
✓ Started safenode19 service
  - PID: 2148999
  - Bin path: /var/safenode-manager/services/safenode19/safenode
  - Data path: /var/safenode-manager/services/safenode19
  - Logs path: /var/log/safenode/safenode19
Attempting to stop safenode20...
✓ Service safenode20 with PID 33437 was stopped
Attempting to start safenode20...
✓ Started safenode20 service
  - PID: 2149076
  - Bin path: /var/safenode-manager/services/safenode20/safenode
  - Data path: /var/safenode-manager/services/safenode20
  - Logs path: /var/log/safenode/safenode20
Attempting to stop safenode21...
✓ Service safenode21 with PID 35847 was stopped
Attempting to start safenode21...
✓ Started safenode21 service
  - PID: 2149154
  - Bin path: /var/safenode-manager/services/safenode21/safenode
  - Data path: /var/safenode-manager/services/safenode21
  - Logs path: /var/log/safenode/safenode21
Attempting to stop safenode22...
✓ Service safenode22 with PID 38364 was stopped
Attempting to start safenode22...
✓ Started safenode22 service
  - PID: 2149231
  - Bin path: /var/safenode-manager/services/safenode22/safenode
  - Data path: /var/safenode-manager/services/safenode22
  - Logs path: /var/log/safenode/safenode22
Attempting to stop safenode23...
✓ Service safenode23 with PID 40970 was stopped
Attempting to start safenode23...
✓ Started safenode23 service
  - PID: 2149308
  - Bin path: /var/safenode-manager/services/safenode23/safenode
  - Data path: /var/safenode-manager/services/safenode23
  - Logs path: /var/log/safenode/safenode23
Attempting to stop safenode24...
✓ Service safenode24 with PID 43688 was stopped
Attempting to start safenode24...
✓ Started safenode24 service
  - PID: 2149385
  - Bin path: /var/safenode-manager/services/safenode24/safenode
  - Data path: /var/safenode-manager/services/safenode24
  - Logs path: /var/log/safenode/safenode24
Attempting to stop safenode25...
✓ Service safenode25 with PID 46503 was stopped
Attempting to start safenode25...
✓ Started safenode25 service
  - PID: 2149463
  - Bin path: /var/safenode-manager/services/safenode25/safenode
  - Data path: /var/safenode-manager/services/safenode25
  - Logs path: /var/log/safenode/safenode25
Attempting to stop safenode26...
✓ Service safenode26 with PID 49427 was stopped
Attempting to start safenode26...
✓ Started safenode26 service
  - PID: 2149541
  - Bin path: /var/safenode-manager/services/safenode26/safenode
  - Data path: /var/safenode-manager/services/safenode26
  - Logs path: /var/log/safenode/safenode26
Attempting to stop safenode27...
✓ Service safenode27 with PID 52437 was stopped
Attempting to start safenode27...
✓ Started safenode27 service
  - PID: 2149631
  - Bin path: /var/safenode-manager/services/safenode27/safenode
  - Data path: /var/safenode-manager/services/safenode27
  - Logs path: /var/log/safenode/safenode27
Attempting to stop safenode28...
✓ Service safenode28 with PID 55582 was stopped
Attempting to start safenode28...
✓ Started safenode28 service
  - PID: 2149710
  - Bin path: /var/safenode-manager/services/safenode28/safenode
  - Data path: /var/safenode-manager/services/safenode28
  - Logs path: /var/log/safenode/safenode28
Attempting to stop safenode29...
✓ Service safenode29 with PID 58813 was stopped
Attempting to start safenode29...
✓ Started safenode29 service
  - PID: 2149801
  - Bin path: /var/safenode-manager/services/safenode29/safenode
  - Data path: /var/safenode-manager/services/safenode29
  - Logs path: /var/log/safenode/safenode29
Attempting to stop safenode30...
✓ Service safenode30 with PID 62119 was stopped
Attempting to start safenode30...
✓ Started safenode30 service
  - PID: 2149878
  - Bin path: /var/safenode-manager/services/safenode30/safenode
  - Data path: /var/safenode-manager/services/safenode30
  - Logs path: /var/log/safenode/safenode30
Attempting to stop safenode31...
✓ Service safenode31 with PID 65535 was stopped
Attempting to start safenode31...
✓ Started safenode31 service
  - PID: 2149955
  - Bin path: /var/safenode-manager/services/safenode31/safenode
  - Data path: /var/safenode-manager/services/safenode31
  - Logs path: /var/log/safenode/safenode31
Attempting to stop safenode32...
✓ Service safenode32 with PID 69045 was stopped
Attempting to start safenode32...
✓ Started safenode32 service
  - PID: 2150032
  - Bin path: /var/safenode-manager/services/safenode32/safenode
  - Data path: /var/safenode-manager/services/safenode32
  - Logs path: /var/log/safenode/safenode32
Attempting to stop safenode33...
✓ Service safenode33 with PID 72705 was stopped
Attempting to start safenode33...
Attempting to stop safenode34...
✓ Service safenode34 with PID 76410 was stopped
Attempting to start safenode34...
✓ Started safenode34 service
  - PID: 2150192
  - Bin path: /var/safenode-manager/services/safenode34/safenode
  - Data path: /var/safenode-manager/services/safenode34
  - Logs path: /var/log/safenode/safenode34
Attempting to stop safenode35...
✓ Service safenode35 with PID 80273 was stopped
Attempting to start safenode35...
✓ Started safenode35 service
  - PID: 2150269
  - Bin path: /var/safenode-manager/services/safenode35/safenode
  - Data path: /var/safenode-manager/services/safenode35
  - Logs path: /var/log/safenode/safenode35
Attempting to stop safenode36...
✓ Service safenode36 with PID 84182 was stopped
Attempting to start safenode36...
✓ Started safenode36 service
  - PID: 2150359
  - Bin path: /var/safenode-manager/services/safenode36/safenode
  - Data path: /var/safenode-manager/services/safenode36
  - Logs path: /var/log/safenode/safenode36
Attempting to stop safenode37...
✓ Service safenode37 with PID 88203 was stopped
Attempting to start safenode37...
✓ Started safenode37 service
  - PID: 2150449
  - Bin path: /var/safenode-manager/services/safenode37/safenode
  - Data path: /var/safenode-manager/services/safenode37
  - Logs path: /var/log/safenode/safenode37
Attempting to stop safenode38...
✓ Service safenode38 with PID 92324 was stopped
Attempting to start safenode38...
Attempting to stop safenode39...
✓ Service safenode39 with PID 96546 was stopped
Attempting to start safenode39...
✓ Started safenode39 service
  - PID: 2150606
  - Bin path: /var/safenode-manager/services/safenode39/safenode
  - Data path: /var/safenode-manager/services/safenode39
  - Logs path: /var/log/safenode/safenode39
Attempting to stop safenode40...
✓ Service safenode40 with PID 100875 was stopped
Attempting to start safenode40...
Attempting to stop safenode41...
✓ Service safenode41 with PID 105318 was stopped
Attempting to start safenode41...
Attempting to stop safenode42...
✓ Service safenode42 with PID 110298 was stopped
Attempting to start safenode42...
Attempting to stop safenode43...
✓ Service safenode43 with PID 115968 was stopped
Attempting to start safenode43...
Attempting to stop safenode44...
✓ Service safenode44 with PID 121319 was stopped
Attempting to start safenode44...
✓ Started safenode44 service
  - PID: 2151017
  - Bin path: /var/safenode-manager/services/safenode44/safenode
  - Data path: /var/safenode-manager/services/safenode44
  - Logs path: /var/log/safenode/safenode44
Attempting to stop safenode45...
✓ Service safenode45 with PID 126158 was stopped
Attempting to start safenode45...
Attempting to stop safenode46...
✓ Service safenode46 with PID 131095 was stopped
Attempting to start safenode46...
Attempting to stop safenode47...
✓ Service safenode47 with PID 136129 was stopped
Attempting to start safenode47...
✓ Started safenode47 service
  - PID: 2151325
  - Bin path: /var/safenode-manager/services/safenode47/safenode
  - Data path: /var/safenode-manager/services/safenode47
  - Logs path: /var/log/safenode/safenode47
Attempting to stop safenode48...
✓ Service safenode48 with PID 141262 was stopped
Attempting to start safenode48...
✓ Started safenode48 service
  - PID: 2151403
  - Bin path: /var/safenode-manager/services/safenode48/safenode
  - Data path: /var/safenode-manager/services/safenode48
  - Logs path: /var/log/safenode/safenode48
Attempting to stop safenode49...
✓ Service safenode49 with PID 146513 was stopped
Attempting to start safenode49...
Attempting to stop safenode50...
✓ Service safenode50 with PID 151884 was stopped
Attempting to start safenode50...
Upgrade summary:
✓ safenode1 upgraded from 0.105.2 to 0.105.6
✓ safenode2 upgraded from 0.105.2 to 0.105.6
✓ safenode3 upgraded from 0.105.2 to 0.105.6
✓ safenode4 upgraded from 0.105.2 to 0.105.6
✓ safenode5 upgraded from 0.105.2 to 0.105.6
✓ safenode6 upgraded from 0.105.2 to 0.105.6
✓ safenode7 upgraded from 0.105.2 to 0.105.6
✓ safenode8 upgraded from 0.105.2 to 0.105.6
✓ safenode9 upgraded from 0.105.2 to 0.105.6
✓ safenode10 upgraded from 0.105.2 to 0.105.6
✓ safenode11 upgraded from 0.105.2 to 0.105.6
✓ safenode12 upgraded from 0.105.2 to 0.105.6
✓ safenode13 upgraded from 0.105.2 to 0.105.6
✓ safenode14 upgraded from 0.105.2 to 0.105.6
✓ safenode15 upgraded from 0.105.2 to 0.105.6
✓ safenode16 upgraded from 0.105.2 to 0.105.6
✓ safenode17 upgraded from 0.105.2 to 0.105.6
✓ safenode18 upgraded from 0.105.2 to 0.105.6
✓ safenode19 upgraded from 0.105.2 to 0.105.6
✓ safenode20 upgraded from 0.105.2 to 0.105.6
✓ safenode21 upgraded from 0.105.2 to 0.105.6
✓ safenode22 upgraded from 0.105.2 to 0.105.6
✓ safenode23 upgraded from 0.105.2 to 0.105.6
✓ safenode24 upgraded from 0.105.2 to 0.105.6
✓ safenode25 upgraded from 0.105.2 to 0.105.6
✓ safenode26 upgraded from 0.105.2 to 0.105.6
✓ safenode27 upgraded from 0.105.2 to 0.105.6
✓ safenode28 upgraded from 0.105.2 to 0.105.6
✓ safenode29 upgraded from 0.105.2 to 0.105.6
✓ safenode30 upgraded from 0.105.2 to 0.105.6
✓ safenode31 upgraded from 0.105.2 to 0.105.6
✓ safenode32 upgraded from 0.105.2 to 0.105.6
✕ safenode33 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:33369'
✓ safenode34 upgraded from 0.105.2 to 0.105.6
✓ safenode35 upgraded from 0.105.2 to 0.105.6
✓ safenode36 upgraded from 0.105.2 to 0.105.6
✓ safenode37 upgraded from 0.105.2 to 0.105.6
✕ safenode38 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:46547'
✓ safenode39 upgraded from 0.105.2 to 0.105.6
✕ safenode40 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:40381'
✕ safenode41 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:37859'
✕ safenode42 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:38563'
✕ safenode43 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:35561'
✓ safenode44 upgraded from 0.105.2 to 0.105.6
✕ safenode45 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:36939'
✕ safenode46 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:40957'
✓ safenode47 upgraded from 0.105.2 to 0.105.6
✓ safenode48 upgraded from 0.105.2 to 0.105.6
✕ safenode49 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:43163'
✕ safenode50 was not upgraded: Error: Could not connect to RPC endpoint 'https://127.0.0.1:43119'

and now running safenode-manager appears to be hanging and the processor is jammed on 100 %

chriso · April 10, 2024, 4:03pm

Thanks for the report.

Shu · April 10, 2024, 5:37pm

I am not using the --interval at all. I have my own logic to sleep in between node service startup sequence based on CPU % for now.

wes · April 10, 2024, 6:19pm

That’s got to be one of my next moves. Currently up to 111 nodes on this machine, which runs just fine while everything is humming along, but my last batch of 20 and the CPU didn’t quite see eye to eye. If I have to restart this machine one of them is going to get KO’d.

Running state with 110 nodes:

Would you be willing to share your startup script? I know you said it was particular to your setup, but may be able to glean some ideas from it?

chriso · April 10, 2024, 6:32pm

I could add something into the node manager based on CPU usage, with some good criteria, but not sure what that could be?

It seems a bit over the top to me, but willing to consider it if people have good ideas.

chriso · April 10, 2024, 6:35pm

We should be able to get most of these things in.

chriso · April 10, 2024, 6:37pm

I’m going to see if I can do some experimentation to reproduce these failed start ups.

What was the interval you used for this one, or what other values have you tried and not been successful with?

aatonnomicc · April 10, 2024, 6:53pm

I am just trying out 5 min delay now and its looking like it was to short across a selection of systems.
previously I have had good results with a 10 min per node time out.

currently s05 - s19 have 26 nodes running on them and i was hoping to get to 50 nodes each
spec is 4cpu 6gb ram 150gb hard drives

the idea for some logic to tell safe node manager when to start a node based on cpu usage would be amazing if some good settings can be figured out.

chriso · April 10, 2024, 6:55pm

Cool, thanks. Have you confirmed it is safenode processes that are generating that usage?

aatonnomicc · April 10, 2024, 7:10pm

id say so here is the resources being currently consumed by the safenode processes on one of the machines.

chriso · April 10, 2024, 7:12pm

Does each line on that graph represent a unique safenode process?

wes · April 10, 2024, 7:26pm

Yes. I was using that monitor script before. It does some bash voodoo to read the stats from ps

process_info=$(ps -o rss,%cpu -p "${dir_pid[$dir_name]}" | awk 'NR>1')
if [[ -n "$process_info" ]]; then
    status=TRUE
    total_nodes_running=`echo $total_nodes_running+1 | bc`
    mem_used=$(echo "$process_info" | awk '{print $1/1024}')
    cpu_usage=$(echo "$process_info" | awk '{print $2}')
else
    status=FALSE
    total_nodes_killed=`echo $total_nodes_killed+1 | bc`
    mem_used=0.0
    cpu_usage=0.0
fi

Topic		Replies	Views
Bash scripts for managing safe nodes on Linux Community	96	1529	July 7, 2024
User run network based on test 12b binaries Community	466	13933	February 24, 2017
Launch of a community safe network Community	426	15569	March 6, 2020
OFFLINE Will it be a Quicky? (run 4) Community community-test	173	3247	December 23, 2021
MaidSafe Dev Update - 14th June 2016 - TEST 4 Updates	338	19239	October 8, 2016

Node Manager UX and Issues

Related topics