Possible problem with baby-fleming safe_network 0.58.5 sn_cli 0.50.4

Southside · March 14, 2022, 5:48pm

I was running a local baby-fleming with the std 11 nodes and noticed this

willie@gagarin:~$ safe files put -r /fgfs/fguk/Satnightroutes/
Error: 
   0: ClientError: Error received from the network: InsufficientAdults { prefix: Prefix(), expected: 4, found: 3 } for cmd: MsgId(78e3..8d0a)
   1: Error received from the network: InsufficientAdults { prefix: Prefix(), expected: 4, found: 3 } for cmd: MsgId(78e3..8d0a)
   2: Failed to store data. Insufficient replication count at section Prefix(). Expected 4, found 3.

Location:
   sn_cli/src/subcommands/files.rs:218

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Grepping for “insufficient” in the logs threw up another surprise - this time in Italiano …

willie@gagarin:~/projects/maidsafe$ grep -rn insufficient ~/.safe/node/baby-fleming-nodes/
/home/willie/.safe/node/baby-fleming-nodes/sn-node-9/chunkdb/0/0/0/0/0/1/0/0/1/0/1/1/0/0/1/0/0/0/0/1/hy13bn1cjceacm4f5zk8aksjtzrkfejpisf9u7zht4rf54siixzco:15:  "Not enough non-option arguments: got %s, need at least %s": "Numero insufficiente di argomenti non opzione: inseriti %s, richiesti almeno %s",
/home/willie/.safe/node/baby-fleming-nodes/sn-node-9/chunkdb/0/0/0/0/0/1/0/0/1/0/1/1/0/0/1/0/0/0/0/1/hy13bn1cjceacm4f5zk8aksjtzrkfejpisf9u7zht4rf54siixzco:33:  "Not enough arguments following: %s": "Argomenti insufficienti dopo: %s",
/home/willie/.safe/node/baby-fleming-nodes/sn-node-9/chunkdb/1/0/0/1/1/1/1/1/1/1/1/0/1/0/0/0/1/0/1/1/hu9wmoigoigdhnhaxjmub8ydg43c3y6g44dy8ca1qgs8cqabhboxo:15:  "Not enough non-option arguments: got %s, need at least %s": "Numero insufficiente di argomenti non opzione: inseriti %s, richiesti almeno %s",
/home/willie/.safe/node/baby-fleming-nodes/sn-node-9/chunkdb/1/0/0/1/1/1/1/1/1/1/1/0/1/0/0/0/1/0/1/1/hu9wmoigoigdhnhaxjmub8ydg43c3y6g44dy8ca1qgs8cqabhboxo:33:  "Not enough arguments following: %s": "Argomenti insufficienti dopo: %s",

Now where did these messages come from? A pizza with your choice of toppings (max 4) for the first person with a sane answer that does not mention Italy

Southside · March 14, 2022, 7:53pm

Can anyone replicate this?

Start a local baby fleming with the software versions mentioned above

safe node run-baby-fleming

safe files put -r <some 500Mb+ directory>

keep an eye on the logs or vdash – that’s it.

aatonnomicc · March 14, 2022, 8:06pm

I’ll try it latter on when I get a min.

Southside · March 14, 2022, 8:06pm

Thanks Neil

aatonnomicc · March 14, 2022, 11:00pm

ubuntu@safe:~/safe$ safe files put crow.mkv
Error:
   0: ClientError: Error received from the network: InsufficientAdults { prefix: Prefix(), expected: 4, found: 0 } for cmd: MsgId(7cfc..3c3a)
   1: Error received from the network: InsufficientAdults { prefix: Prefix(), expected: 4, found: 0 } for cmd: MsgId(7cfc..3c3a)
   2: Failed to store data. Insufficient replication count at section Prefix(). Expected 4, found 0.

Location:
   sn_cli/src/subcommands/files.rs:218

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

ubuntu@safe:~/safe$ grep -rn Insufficient ~/.safe/node/baby-fleming-nodes/
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24213:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24240:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24252:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24271:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24288:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24308:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24320:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24332:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24351:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24385:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24423:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24429:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24437:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24453:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24467:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24475:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24483:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24492:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24504:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24510:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24544:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24552:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24555:    ➤ InsufficientAdults for storing data reliably
/home/ubuntu/.safe/node/baby-fleming-nodes/sn-node-genesis/sn_node.log:24561:    ➤ InsufficientAdults for storing data reliably

Southside · March 14, 2022, 11:06pm

OK can you also check vdash please?

Are you seeing GETs when you expect PUTs? Or did you not get to PUT any chunks at all?
Also is vdash reporting sane values for space available and space free across all nodes?
Does the load show anything other than zero ?
I know load was working ok a week or so back but its only showing zeroes now . Im about to write this up as a GH issue for @happybeing but I’ll wait to see if you can confirm any of it.

aatonnomicc · March 14, 2022, 11:11pm

I’ll check that in a bit

I would hang fire I think it’s due to the nodes getting voted off ass the the network is set to be overly strict at the minnute. If I understood Weekley update correctly?

Southside · March 15, 2022, 12:13am

I rebooted the box cos upgrades and started agian. I got no errors about insufficient elders. Again the vault size was shown as 2.1GB and when I filled that it stopped. I cannot quickly see where this max vault size is being set.
There is no --max-capacity value set on the sn_node processes as shown by ps aux and there are no leftover env variables set from old comnet-install runs.
What vault size are you seeing on vdash?

Southside · March 15, 2022, 12:18am

THats fair enough but its all local with baby-fleming so we should expect it to handle “overly strict” with no problems.

tfa · March 15, 2022, 12:18am

I launched such a big recursive put more than one hour ago. Not with baby fleming, but compiled from latest safe_network source. Directory size to upload is 776 MB (release directory of safe_network crate).

It’s still running. The Grafana dashboard on one of my adults shows this:

But given used space is not the truth (du -sh gives 291MB instead).

I need to go to sleep now. We’ll see final result tomorrow (I hope!)

Southside · March 15, 2022, 12:34am

I compiled here a couple of days ago, run-baby-fleming just launches 11 sn_node processes with whatever is in .safe/node/sn_node - safe_network 0.58.5 for me

Josh · March 15, 2022, 1:21am

I think you are looking for line 28.

github.com/maidsafe/autonomi

sn/src/node/cfg/config_handler.rs

5176b3a72

// Copyright 2022 MaidSafe.net limited.
//
// This SAFE Network Software is licensed to you under The General Public License (GPL), version 3.
// Unless required by applicable law or agreed to in writing, the SAFE Network Software distributed
// under the GPL Licence is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. Please review the Licences for the specific language governing
// permissions and limitations relating to use of the SAFE Network Software.

use crate::node::{Error, NetworkConfig, Result};
use serde::{Deserialize, Serialize};
use std::{
    collections::BTreeSet,
    io::{self},
    net::SocketAddr,
    path::PathBuf,
    time::Duration,
};
use structopt::StructOpt;
use tokio::{
    fs::{self, File},

This file has been truncated. show original

Southside · March 15, 2022, 1:23am

Brilliant, thanks

tfa · March 15, 2022, 7:38am

The whole 776 MB directory was uploaded in 1 hour and 20 minutes.

Remarks:

one error message at the end
as mentioned above, invalid used spaced is displayed (211 MB instead of 364)

joshuef · March 15, 2022, 10:06am

Aye, @yogesh has just noticed the same working on some droplet logging. That’ll get looked into soon.

Can baby-fleming launch 15 nodes? I generally use the cargo run --release --bin testnet for launching nodes (which you can configure, but also defaults to 15 nodes).

11 nodes was throwing this error up oreviously. I think related to the membership flow. There may not be all the nodes joined as yet (due to the way node joins work atm). So with 15 this was largely avoided.

(This should be sorted w/ the upcoming membership work, but also I think 15 seems like a better default number as that’s what sections will have as a lower bound after the genesis section)

tfa · March 15, 2022, 10:35am

Well, 11 nodes worked for me as shown above.

Please note that I add a constant 4 seconds delay between each node.

I have doing this for years, but i don’t know if this is useful.

Southside · March 15, 2022, 2:25pm

I changed safe_network/sn/src/node/cfg/config_handler.rs to give a default 10GB node Update config_handler.rs · safenetwork-community/safe_network@14a14d1 · GitHub and ran baby-fleming with 20 nodes. Then ran out of memory on a box with 32GB RAM and the same in swap and saw a peak 1min CPU load of >5000 while trying to put 2.6Gb of photos. Probably best not to try this at home, folks.

However I can confirm baby-fleming will run with 15 and 20 nodes with default 1 sec wait between nodes. I also started with default 11 nodes and added nodes individually up to a total of 18. Again not for any machine that is doing anything else important at the time.
I am hopeful that we will see less resource-hogging on upcoming releases. But then again is 20-off vaults of 10GB each on a single box a sensible real-world scenario? Or just for eejits like me that want to poke?

EDIT Vdash showed 10.7GB available after I recompiled. Which is strange as it is defined as 10 * 1024 * 1024 * 1024. @happybeing

Josh · March 15, 2022, 2:59pm

This is mebibyte which is correct.

To get 10GB exactly it needs to changed to 10 * 1000 * 1000 * 1000

Southside · March 15, 2022, 3:08pm

I always get that arse-for-elbow

happybeing · March 15, 2022, 3:15pm

As a programmer I think your calculation is the preferred one. I think there’s a convention that K is 1000 and k is 1024, so perhaps similarly for M/m and G/g?

Topic		Replies	Views
NotEnoughChunks Support	5	648	February 19, 2022
Delete me Support	2	675	April 13, 2021
Support Issues with Baby Fleming Version 1 (Vaults Phase 2a - single-section network) Development	96	4036	March 12, 2020
Is it OK to have two nodes of the same age? Development	2	439	March 15, 2022
User Experience for the CLIs Features	363	6251	November 22, 2023

Possible problem with baby-fleming safe_network 0.58.5 sn_cli 0.50.4

Related topics