chriso
May 17, 2024, 12:39pm
543
Sorry for the inconvenience, I’ll need to take a deeper look into this. Not sure when I’ll get to it.
1 Like
Josh
May 17, 2024, 12:42pm
544
Haha no need for that. The convenience far outways the inconvenience
1 Like
chriso
May 17, 2024, 12:45pm
545
I will look at it at some point, because it’s an unintentional regression.
3 Likes
I guess Chris and the team are busy today. I’m unable to test my app because PunchBowl fails on certain files and my local testnet is borked.
I’m getting lots of side-tasks done which is good, but does anyone know how to wipe everything safenode-manager does from a machine to get a clean start?
It says the faucet is running but the process isn’t there and there’s nothing listening on 8000, so I think it is wrong and I don’t know how to clear it. I can’t use ‘stop’ or ‘reset’ because it want sudo and when I do that cargo fails, which I need to run locally.
Josh
May 20, 2024, 7:27pm
547
You should check top after a reset quite often you will find many node processes still running. Or grep for them, I find top easier.
I make a habit of sudo killall safenode now.
And this.
Hey Josh,
To clean things up manually, you need to remove the services and the directories where the node manager is saving state.
To remove the services:
sudo systemctl disable --now safenodeX
sudo rm /etc/systemd/system/safenode*
sudo systemctl daemon-reload
Then delete these directories:
sudo rm -rf /var/safenode-manager
sudo rm -rf /var/log/safenode
1 Like
this works for me for a full tear down
#!/usr/bin/env bash
sudo pkill -e safe
# stop nodes
for i in {1..100}
do
# your-unix-command-here
sudo systemctl disable --now safenode$i
done
sudo rm /etc/systemd/system/safenode*
sudo systemctl daemon-reload
rm -rf $HOME/.local/share/safe/node
sudo rm -rf /var/safenode-manager
sudo rm -rf /var/log/safenode
2 Likes
I’m back in business. Thanks all of you
5 Likes
neo
May 21, 2024, 10:48am
551
@chriso So I just did a upgrade of safenode-manager and one of my scripts died, showed no data from the status and balance options
I have to now use sudo for add/start/status/balance (and maybe more). Not sure if this was mentioned before, but thought you need to know.
I know you’ll look into it so no need for a response
3 Likes
Toivo
May 22, 2024, 2:39pm
552
I think that one essential feature would be a possibility to quickly close down a selected portion of nodes, like 10%, 20%…
When people are running too many nodes and run into problems with other connections in the household, they are going to kill nodes. And if killing a subset is not made easy, they’ll kill all. Maybe a pause
could also be applied.
This happened to me today when I reached the capacity of our network, and then my spouse needed bandwidth quickly.
6 Likes
chriso
May 22, 2024, 3:33pm
553
Thanks for the useful suggestion. I’ve added it to the issue:
opened 08:46PM - 08 Nov 23 UTC
## Node Manager
- [x] Provide `start` control command
- [x] Provide `stop` con… trol command
- [x] Provide `status` command
- [x] Provide `uninstall` command
- [x] Initial release process
- [x] Extend release process to publish 'latest' version to S3
- [x] Provide `upgrade` command
- [x] Extend `add` command to provide `--url` argument for custom binaries
- [x] Make the `status` command more robust
- [ ] Extend `add` definition to use stop arguments
- [x] Provide `run` command
- [x] Provide `kill` command
- [x] Provide `run` command should support downloading binaries
- [x] Provide `join` command
- [x] Change `safe_network` CI to use the node manager for local networks
- [x] Merge `sn-node-manager` into the `safe_network` repository
- [x] Add the node manager to the `safe_network` release process
- [x] Small extensions for `run` command based on development feedback
- [x] Extend `run` command with `--build` option for building node and faucet binaries
- [x] Extend `upgrade` command with `--url` and `--force` arguments
- [x] Remove use of `custom` for binaries fetched with `--url`
- [x] Pass the `SN_LOG` value to the service definitions
- [x] Services should use the same port on restart/upgrade
- [x] Provide `faucet` subcommand to enable the faucet to be upgraded
- [x] Provide `faucet upgrade` command
- [ ] Provide `daemon upgrade` command (sn_releases needs updated for this)
- [ ] Incorporate release channels
- [ ] Rename `safenode-manager` binary to `safenodemgr`
- [ ] Rename `safenodemand` binary to `safenodemgrd`
- [x] Provide a daemon that can execute commands through RPC
- [ ] Change `--rpc-address` to use `SocketAddr` rather than `Ipv4Addr`
- [x] Build source on Alpine [non-essential]
- [x] Change the `--port` argument on the `add` command to accept a range
- [x] Change each of the three `add` commands to have uniform behaviour
- [x] Provide a `--path` argument on the `add` command
- [x] Provide a `--metrics-port` argument on the `add` command
- [x] Provide an `--rpc-port` argument on the `add` command
- [x] Change `--service-name` and `--peer-id` arguments for multiple usage
- [x] Update documentation for `--version` arguments to provide the format
- [x] Provide `--interval` argument on the `start` command
- [x] Bug: Alpine sometimes does detect the correct service manager type
- [x] Bug: `start`/`stop` commands should preclude removed services
- [x] Bug: `remove` should be able to remove services that were deleted manually
- [x] Write a new README
- [x] New service management utility crate
- [x] Provide a `reset` command
- [x] Remove all services when the `remove` command runs with no arguments
- [x] Use the `network-contacts` feature to remove the necessity to provide a peer
- [x] Move local network commands into a `local` subcommand
- [ ] ~~Provide `restart` command for the faucet~~
- [x] The `status --details` command should include the wallet balance
- [ ] Tidy the node banners on the `status --details` command
- [x] Provide a `balance` command to output the balance of all nodes
- [ ] Remove use of port ranges in favour of a starting port that increments as far as the value of the `--count` argument
- [ ] Output the current store cost in `balance` and `status --details` commands
- [ ] On Windows the `add` command should error with a suggestion if WinSW is not found
- [x] The reward balance should be in the `status --json` output
- [x] The interval on the `start` command should only apply to services not at `RUNNING` status
- [ ] The `status` command should report the connected peers from the routing table
- [ ] RPC `nodeinfo` command extensions: current store cost, wallet balance, cpu usage, memory usage.
- [x] The `upgrade` command should provide an `--interval` argument
- [x] The `upgrade` command should provide an `--path` argument
- [x] The `upgrade` command should return an error if any services do not upgrade correctly
- [ ] The `add` command should provide a flag for disabling auto restart on the service
- [x] Systemd services should optionally be defined as user-mode services
- [x] Launchd services should optionally be defined as user-mode services
- [ ] Make sure the listening ports are output in the `status --details` command
- [x] The `add` command should have a `--upnp` argument to pass to the node
- [ ] The `status` command should be a read-only operation or lock the registry file for writing
- [ ] Provide a `pause` command for stopping a portion of nodes
## Bugs/Issues
- [ ] On Fedora 39 `safenode` services will not start
- [ ] The `status` command needs Administrator access on Windows (this may not be a bug exactly and could be required)
- [x] When lots of nodes are added and started the `status` command does not correctly report all of them as `RUNNING`
- [ ] When node directories have been manually removed it forces the use of `sudo` for the `status` command
- [ ] Running `upgrade` can result in service definitions having text appended rather than being re-written [link](https://forum.autonomi.community/t/node-manager-ux-and-issues/39231/379?u=chriso)
- [ ] On `upgrade` custom ports are not retained
- [x] When `--path` is used on `add` and `upgrade` commands the binary should not be deleted
- [ ] Reports of high CPU usage [link](https://forum.autonomi.community/t/node-manager-ux-and-issues/39231/398?u=chriso)
- [ ] The `balance` command should not error when it is called before services have started [link](https://forum.autonomi.community/t/punchbowl-testnet-09-05-2024/39727/21?u=chriso)
- [ ] The `status` command now requires `sudo` after separating service and local network status [link](https://forum.autonomi.community/t/node-manager-ux-and-issues/39231/540?u=chriso)
## Node Manager Daemon/RPC
- [x] Provide an `rpc restart` client command
- [ ] Extend node manager daemon tests to run on Windows and macOS
- [ ] Support running the daemon on Windows
- [ ] Remove `retain_peer_id` from RPC `restart` command and remove new node creation
- [ ] Provide an RPC `add` command
- [ ] Provide an RPC `remove` command
- [ ] Provide an RPC `start` command
- [ ] Provide an `rpc add` client command
- [ ] Provide an `rpc remove` client command
- [ ] Provide an `rpc start` client command
- [ ] Update the nightly test runs to use new RPC mechanisms for creating a new node
## Testnet Deployment
- [x] Refactor to deploy nodes with node manager
- [x] Trigger upgrades for existing testnets
- [ ] GHA workflow for running testnet upgrades
- [x] Extend upgrade process to include the faucet
- [ ] Extend upgrade process to include `safenodemand`
- [ ] Status report for `upgrade` command
- [ ] Upgrade the `safenodemgr` binaries
## Misc
- [x] Provide an `sn_rpc_client` crate
- [x] Introduce `--first` argument on `safenode` for genesis node
- [x] Modify peer acquisition such that it doesn't combine `--peer` and `SAFE_PEERS`
- [x] Remove the `sn_testnet` crate
- [x] Update `sn-releases` to get the latest faucet version from `crates.io`
- [ ] Use signals in `safenode` for graceful shutdown
- [x] Update `safeup` to get the node manager latest release from the `safe_network` repo
- [ ] Backwards compatibility test in `safe_network` repository
- [x] Provide `node-manager` command in `safeup`
- [ ] Extend service manager crate to delete OpenRC service definitions
- [ ] The node uptime should be in the RPC `info` command
- [ ] Remove the RPC `upgrade` command
8 Likes
Josh
May 22, 2024, 10:29pm
554
I am just a simple man, this sudo and no sudo is making me do stupid things.
6 Likes
chriso
May 23, 2024, 12:52pm
555
Can you be a bit more specific here please? I would like to help, but I need to know more about what you’re finding troubling.
1 Like
neo
May 23, 2024, 12:55pm
556
When to know using sudo & when you don’t have to and the version currently requires sudo for things it did not before and if you forget it just gives output that seems genuine (like no nodes found) but is obviously wrong
1 Like
chriso
May 23, 2024, 12:58pm
557
OK, I think there might have been a regression somewhere, but what would be really helpful is if you could show me the output or refer to specific commands or scenarios where there has been a problem. Otherwise I don’t have much to go on.
1 Like
Josh
May 23, 2024, 1:15pm
558
Ahh, I thought it was all part a plan that has not yet been announced.
In short my difficulty seems to be that you can run commands with or without sudo.
Nothing really prevents you from mixing them up which ends in endless confusion.
Seems very simple but I have been getting myself into all kinds of a mess.
@aatonnomicc noted that when run without sudo everything goes to .local as it did in the past. Perhaps he can comment on that finding.
1 Like
neo
May 23, 2024, 1:16pm
559
AFAIK all commands in safenode-manager require sudo or it doesn’t work.
I just tried status and without sudo there is no output and it just exits normally
chriso
May 23, 2024, 1:24pm
560
OK. I think the confusion here is due to the introduction of the ability to run user-mode services.
Here is a snippet from the recently updated README:
The primary use case for Safenode Manager is to setup safenode
as a long-running background service, using the service infrastructure provided by the operating system.
On macOS and most distributions of Linux, user-mode services are supported. Traditionally, services are system-wide infrastructure that require elevated privileges to create and work with. However, with user-mode services, they can be defined and used without sudo. The main difference is, a user-mode service requires an active user session, whereas a system-wide service can run completely in the background, without any active session. It’s a user decision as to which is more appropriate for their use case. On Linux, some service managers, like OpenRC, used on Alpine, do not support user-mode services. Most distributions use Systemd, which does have support for them.
The commands defined in the rest of this guide will operate on the basis of a user-mode service, and so will not use sudo
. If you would like to run system-wide services, you can go through the same guide, but just prefix each command with sudo
.
Windows does not support user-mode services at all, and therefore, the node manager must always be used in an elevated, administrative session.
The reason for the introduction of user-mode services was to make the TUI usable for those less technical. Prompting for passwords is not really feasible, both from a technical and a UX point of view.
There may need to be some distinction made in the commands to indicate the use of user-mode vs. system-wide service, because clearly people are making mistakes there. I am open to suggestions as to how to handle things in terms of the usage of flags or whatever.
4 Likes
Josh
May 23, 2024, 1:27pm
561
Ahh ok. I am just happy to know that I am not entirely losing my mind. I WILL however accept that sticking to the same pattern should not require more than 3 working brain cells. I have been falling short there.
chriso
May 23, 2024, 1:30pm
562
It’s fine. I acknowledge that mistakes will inevitably be made here. Perhaps we could have system
and user
subcommands? Or use flags?
Another option could be to take user mode services completely out of the CLI and just have that supported in the TUI.
2 Likes