Support Issues with Baby Fleming Version 1 (Vaults Phase 2a - single-section network)

I can confirm this. Authenticator works for me though.

6 Likes

Hey @Southside, some answers to your qns, to the best of my knowledge:

  1. Logging verbosity for authd - I tried all the usual suspects for a way to make the logs more verbose, -v, -y, but it doesn’t appear to have been setup for authd. I tried running export RUST_LOG=debug before running ./safe auth start and it provided debug logs on screen. Probably your best bet for now, I’m not sure there’s a way to get the authd log files themselves to log more, other than forking the code & changing the default, then building locally & running.

  2. I’m really not sure about any password config file -c flag, it’s not something I’ve been asked to test and it’s not in the help or readme’s. I had a search through our internal Slack history & I can’t find any mention of -c with regards to the CLI/authd.

  3. Also not came across any mention of being able to use env variables for pwd & p’phrase in my time testing the CLI & auth, perhaps this was something tested by Calum before he left late last year. Had a search for mentions of this and a scroll through the readme’s - I assume you’re not thinking of the json-rpc requests/responses in the authd readme here.

Apologies I couldn’t be more help, I’ve only been working closely with the CLI & authd since December. Gabriel is on vacation for a week now, he’s the expert here.

11 Likes

Thanks for looking, I’ll keep trying :slight_smile:

Enjoy whats left of your weekend.

EDIT: I found it - SAFE Network Dev Update - April 25, 2019 - #20 by tfa has posts from @tfa and @bochaco that will help :slight_smile:

EDIT:
Perhaps I should have made it clear that the env variales and config file methods are horribly insecure and for testing ONLY. So no real surprise they were dropped from the README

7 Likes

I’m seeing an infinite loop issue (with 2s delay between loops) before any uploading has been done (ie just starting vaults). Here’s my steps:

  • start baby-fleming
  • kill vault 8
  • start vault 8 up again manually with trace level loggin: RUST_LOG=trace ./safe_vault --hard-coded-contacts='["127.0.0.1:41207"]' --root-dir=/home/ian/.safe/vault/baby-fleming-vaults/safe-vault-8 > /home/ian/.safe/vault/baby-fleming-vaults/safe-vault-8/trace.log 2>&1
  • same set of log messages repeats every 2 seconds, but cpu load is essentially zero and memory is unchanged, so I’m really just flagging what to me seems like unknown behaviour (which may in fact be normal gossip stuff).
TRACE 2020-03-08T16:42:12.942913749+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/approved.rs:541] Adult(6d8a9d..()) - Send MemberKnowledge { elders_version: 3, parsec_version: 3 } to [P2pNode(019f56.. at 127.0.0.1:56541), P2pNode(3b994c.. at 127.0.0.1:41207), P2pNode(5b10a1.. at 127.0.0.1:41921), P2pNode(8a28a9.. at 127.0.0.1:41252), P2pNode(bf327d.. at 127.0.0.1:35065), P2pNode(cf67db.. at 127.0.0.1:38761), P2pNode(f9828b.. at 127.0.0.1:60113)]
TRACE 2020-03-08T16:42:12.943400071+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:424] Adult(6d8a9d..()) Sending message ID 492 to [V4(127.0.0.1:56541)]
TRACE 2020-03-08T16:42:12.943755917+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:424] Adult(6d8a9d..()) Sending message ID 493 to [V4(127.0.0.1:41207)]
TRACE 2020-03-08T16:42:12.944061369+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:424] Adult(6d8a9d..()) Sending message ID 494 to [V4(127.0.0.1:41921)]
TRACE 2020-03-08T16:42:12.944367568+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:424] Adult(6d8a9d..()) Sending message ID 495 to [V4(127.0.0.1:41252)]
TRACE 2020-03-08T16:42:12.944713544+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:424] Adult(6d8a9d..()) Sending message ID 496 to [V4(127.0.0.1:35065)]
TRACE 2020-03-08T16:42:12.945235974+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:424] Adult(6d8a9d..()) Sending message ID 497 to [V4(127.0.0.1:38761)]
TRACE 2020-03-08T16:42:12.945750538+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:424] Adult(6d8a9d..()) Sending message ID 498 to [V4(127.0.0.1:60113)]
TRACE 2020-03-08T16:42:12.945906507+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:335] Adult(6d8a9d..()) Successfully sent message with ID 492 to V4(127.0.0.1:56541)
TRACE 2020-03-08T16:42:12.946006594+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:335] Adult(6d8a9d..()) Successfully sent message with ID 493 to V4(127.0.0.1:41207)
TRACE 2020-03-08T16:42:12.946086820+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:335] Adult(6d8a9d..()) Successfully sent message with ID 494 to V4(127.0.0.1:41921)
TRACE 2020-03-08T16:42:12.946164639+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:335] Adult(6d8a9d..()) Successfully sent message with ID 495 to V4(127.0.0.1:41252)
TRACE 2020-03-08T16:42:12.946240019+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:335] Adult(6d8a9d..()) Successfully sent message with ID 496 to V4(127.0.0.1:35065)
TRACE 2020-03-08T16:42:12.946315964+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:335] Adult(6d8a9d..()) Successfully sent message with ID 497 to V4(127.0.0.1:38761)
TRACE 2020-03-08T16:42:12.947142378+11:00 [/usr/local/cargo/git/checkouts/routing-616e72cf40f12bc0/76713d9/src/states/common/base.rs:335] Adult(6d8a9d..()) Successfully sent message with ID 498 to V4(127.0.0.1:60113)

This group of lines repeats endlessly with 2s delays between groups.


I did follow this up with an upload that gave 100% cpu etc but am not diving into logs until I have trace level logging on all vaults.

When uploading and getting 100% cpu my vault-8 logs with trace didn’t change compared to when no upload was happening which seemed strange.

I admit I feel like I’m shooting a bit from the hip here, maybe if there were details about what the message being sent contained it might be easier to know if it were worth reporting or not… the source code at the lines being logged is not possible to trace back to the origin of the message since the functions are (to me) generic send_message_to_initial_targets and handle_sent_message calls, with some message in the input parameters. Would be handy to know what that message is so it can be traced further back in the stack toward it’s origin. Or maybe some other way to get more detail, I’m not sure if what I suggest is best.

15 Likes

For anyone thats interested the structure of the login parameters file is dead simple and is as below

{ "passphrase":"aaaaa", "password":"bbbbbb" }

I saved this as myconfig.json and call it from my script as

safe auth create-acc --test-coins -c myconfig.json

and
safe auth login --self-auth -c myconfig.json

This is totally insecure and just speeds up testing a little bit

11 Likes

@StephenC Meanwhile, I tested the same thing on a better Windows 10 machine (more RAM, better CPU) and it works fine. I seem to have run into a performance limitation with the first laptop (which is 8 years old).
It has 4GB RAM, and an i5-3317 CPU @ 1.7 GHz. Is there a list with minimum specs?

9 Likes

@Joop77 what was the largest file size you were able to store successfully?
How long did it take? Roughly…

I see there is no direct Windows equivalent of the time command in linux. You will need to use Powershell and Measure-Command cmdlet so it seems…

linux - Windows time command equivalent - Server Fault

2 Likes

I’m sure there will be eventually but we are all feeling our way around right now to see what works and what doesn’t.

6 Likes

I have not uploaded anything yet, I did not get past creating an account. That has been solved now with a more powerful laptop. Will test a bit later next week.

5 Likes

Hopefully these kinda problems I am reading easily point to why at the base layers we need excellent logging and debugging baked in to help with early test-nets(spend a 1-2 sprints here just to add logging and think through as many areas of the code that desperately need it). Idk much about the QUIC besides that its UDP with HTTP/2 Multiplexing and security baked in by changing whats in the early packets as well to prevent some MITM stuff. Fingers crossed these newer protocols don’t bite us in terms of debugging capabilities :stuck_out_tongue: . I am too used to good old tcp dumps to expose ephemeral port exhaustion and port reuse issues and such to figure out why connections are having issues.

As far as memory utilization goes, anyone tracking over time the mem consumption of the running application(are vaults analogous in design to separate running processes, so much like NGINX webserver where I can track mem consumption by worker process?), if all these uploads write to disk(which duh its vaults so persistent eh) then mem consumption should stabilize at some point else we may be looking at memory leaks somewhere I imagine in the rust-land or transport buffers somehow don’t flush properly always when under duress/high transfer throughput.

11 Likes

@Joop77 @StephenC, this is a limitation currently in authd for Windows, I didn’t find yet the way to log from a Windows service, so I confirm the logs are not output as in other platforms. The good news is that I plan to refactor authd for Windows and launch it in the background just like we do for vaults in Baby Fleming and don’t set it up as a Windows service. This will allow us to send the logs to the same location as it’s done on Linux and Mac, and also this will remove the need of having admin perms for authd commands on Windows. I’m hoping you all Windows users agree with the approach as it’ll make it as simple as in other platforms, no admin perms needed.

11 Likes

Can you suggest specific tests we could do to help narrow this down?
It certainly looks like a memory leak, once safe decides its not going to put a file, CPU utilisation remains at 100% over all 4 cores,(BTW I got hold of an i7 8 core laptop today so I’ll see if that is similar) and memory usage rises to 16Gb and then fills up the swap - so you need to hit safe vault killall in another terminal sharpish or its a complete reboot…

5 Likes

I haven’t noticed the memory hogging - it seems to be all CPU with me (Win and Linux, Task Manager / Htop)

4 Likes

I’m getting an error with the ā€˜Auth allow/deny’ test when trying to allow from second console:

in first console I get:

auth

Authorising CLI application…

Awaiting for authorising response from authd…

Application authorisation failed: [Error] AuthdClientError - [Error] ClientError - Failed to read response bytes: read error

and then in second console in which I need to allow I get:

auth reqs
Requesting list of pending authorisation requests from authd…
There are no pending authorisation requests
auth reqs
Requesting list of pending authorisation requests from authd…
±-------------------------------±-----------------±---------±-----------------±-----------------------+
| Pending Authorisation requests | | | | |
±-------------------------------±-----------------±---------±-----------------±-----------------------+
| Request Id | App Id | Name | Vendor | Permissions requested |
±-------------------------------±-----------------±---------±-----------------±-----------------------+
| 1936091719 | net.maidsafe.cli | SAFE CLI | MaidSafe.net Ltd | Own container: No |
| | | | | Transfer coins: Yes |
| | | | | Mutations: Yes |
| | | | | Read coin balance: Yes |
| | | | | Containers: None |
±-------------------------------±-----------------±---------±-----------------±-----------------------+
auth allow 1936091719
Sending request to authd to allow an authorisation request…
[Error] AuthdError - No pending authorisation request found with id ā€˜1936091719’

My system:
macOS Catalina, 10.15.3
3.2 GHz Quad-Core Intel Core i5
16 GB 1600 MHz DDR3
Nvidia GeForce GTX 675MX 1 GB

3 Likes

@Michael_Hills, unless you are really testing those commands, I’d recommend you simple self-authorise CLI with auth login --self-auth to get you going: https://github.com/maidsafe/safe-api/tree/master/safe-cli#self-authorising-the-cli-application

4 Likes

ok, sounds good, I’ll do it that way, thanks

3 Likes

I see this quite often myself.
I am running it a few times to try and spot a pattern – seems random so far

2 Likes

This is the set of commands I use

    safe vault run-baby-fleming -y 
    safe networks add my-network
    safe networks switch my-network
    safe auth restart
    safe auth status
    safe auth create-acc --test-coins -c myconfig.json
    safe auth login --self-auth -c /myconfig.json
    safe auth status

myconfig.json simply consists of :
{ "passphrase":"bbbbb", "password":"aaaaaa" }

The safe auth status commands are optional

3 Likes

It must be the authd just taking longer sometimes to login and authorise the app, which causes CLI to just timeout (even that the error is not saying so and which could be enhanced)

3 Likes

I don’t know if I’m adding anything here but just tried a smaller 1.4MB file and it didn’t stress RAM so hard but timed out, which is perhaps odd.

[2020-03-09T20:57:52Z ERROR safe] safe-cli error: [Error] NetDataError - Failed to PUT Sequenced Append Only Data: Error(CoreError(Request has timed out - CoreError::RequestTimeout))

If I now have a log file that’s useful, let me know where.

3 Likes