Superb, this gives us something to look at. I suspect there may be some usability issues for sure and also it should expose XorName etc. types for sure. Easy fixes though cc @Anselme
1 Like
That will be great, especially if it knows what is an archive vs date file. I have to check for an error on archive download to do this atm.
2 Likes
We need to do better really and those types should all be subtypes where we serialise into whatever type you request. It’s no use having different types like this really. Bu now we have to consider backwards compatibility so a bit more tricky, but we will get there, TOGETHER 
7 Likes
Not sure we’re really already in that situation 
This is something I had some concerns about. The changes in archive broke previous stored data on the last test net.
It looks like the data types are directly serialised to the network. I know in the java world, that tends to be considered unwise, for exactly this sort of reason, i.e. changing the datastructure breaks deserialisation of old structures.
I’m not sure what best practices are in rust, but maybe worth some thought?
2 Likes
It’s a big issue we have as data has to be serialised to store or transmit. The serialisation types are a consideration json/binary json/bincode etc. but also the structure of the data types where in rust algebraic enums are the SOTA really. It’s all part of making the API correct as well though
1 Like
I don’t know if you were answering to riddim or me (or both), so I’ll post my commands:
I’m using micromamba (light version of anaconda package manager) on Ubuntu 24.04
micromamba create -n autonomi-nodes
micromamba activate autonomi-nodes
micromamba install python=3.12
pip install antnode==0.3.7rc3
Then running this simple python script:
from antnode import AntNode
node = AntNode()
2 Likes
I cannot remember the API for python antnode, but if you do help(AntNode) or doc(AntNode) it would tell you. It looks like there is no default constructor that has no parameters.
1 Like
Thanks for the feedback @riddim! Feel free to share anything regarding our Python bindings.
I’ve just noticed that we missed exporting some classes we built for these bindings. With these changes, you’d need to instantiate a DataAddress like so:
data_address = DataAddress.from_hex("a7d2fdbb975efaea25b7ebe3d38be4a0b82c1d71e9b89ac4f37bc9f8677826e0")
dog_picture = await client.data_get_public(data_address)
edit: Hope to republish this package with the fixes soon!
7 Likes
Some general feedback for archive/file APIs. They’re good for simple integrations, but a bit limited for more complex use cases. Some thoughts below:
- Add a call to calculate and return XOR addresses, rather than having to wait until an upload is completed. This prevents async upload calls, which due to upload speed, would be very useful. As these can be derived up front, they could be returned to the app and/or the user immediately, while upload is processed in the background.
- Expose lower level calls to retrieve chunks in memory, rather saving files to disk first. In many cases we want to pipe that data immediately into another routine. Having to persist the whole file to disk blocks until that operation is complete - the larger the files, the worse the pain. Exposing a finer grained function that provides data to a buffer chunk by chunk would be very useful and could be re-used by the disk saving version too.
- Like the above, but for uploads. Not sure if possible with self-encryption, but persisting a file to disk, then uploading it, may just be an overhead in some/many cases.
- Async calls could be useful too, where the call returns a handle to check status on in some way. Relates to 1 (and that would be higher in priority), but may help the dev experience if this stuff was built in.
- Sleep the autonomi client when not in use or have a mechanism to make it sleep. It is nice to have a long running instance, so we don’t need to keep bootstrapping new ones, but it would be nice to tell it to sleep after prolonged idle, to save power and bandwidth.
For AntTP, the upload/download/async use cases are fairly obvious - as a proxy, it wants to shuttle data to/from the network. It also doesn’t want to sit blocking on POSTs, as clients will timeout.
Streaming music/videos/other both uploads and downloads (latter is the priority) is key too, as you don’t want to wait for AntTP to download the file in full, to disk, then start sending the beginning of it to the client - it should do that as soon an AntTP has the first chunk.
As a proxy, AntTP may spend extended periods idle too. Allowing the client to sleep would be a help here.
Hope that helps!
3 Likes
Have you looked at the self_encryption module for this. You can select in memory functions there or the chunk store functor, so you can get some more leverage directly form that crate, bu use autonomi-client for our other non Chunk data types.
1 Like
I did something like that with an old API version, but was hesitant to do the same again on shifting API sands. Ofc, I can do a deep dive on this, but thought it may benefit the API if mroe easily exposed.
This is the output of help(AntNode):
class AntNode(object)
| Python wrapper for the Autonomi Node
|
| Methods defined here:
|
| get_all_record_addresses(self, /)
| Get all record addresses stored by the node
|
| get_data_dir(self, /)
| Get the data directory path where records are stored
|
| get_kbuckets(self, /)
| Get the node's kbuckets information
|
| get_logs_dir(self, /)
| Get the logs directory path
|
| get_record(self, /, key)
| Get a record from the node's storage
|
| get_rewards_address(self, /)
| Get the node's rewards/wallet address as a hex string
|
| get_root_dir(self, /)
| Get the current root directory path for node data
|
| get_stored_records_size(self, /)
| Get the total size of stored records
|
| peer_id(self, /)
| Get the node's PeerId as a string
|
| store_record(self, /, key, value, _data_type)
| Store a record in the node's storage
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
No information on the constructor (and some methods seem to be missing from the doc, like the start method).
I also tried to instanciate the AntNode with up to 8 arguments, I always have the same error.
2 Likes
We have likely to update antnode here. Can you check you have the latest version as well, just to be clear
2 Likes
I have the last version of antctl and antnode installed (0.12.0 and 0.3.7)
1 Like
Where exactly are the Python API docs? The website API document link just takes you to PyPi which earlier in the thread was said it is not updated. I’m just going around in circles.
2 Likes
all docs are being focused on at docs.autonomi…com @JimCollinson is pulling everything into one place. so API docs and more are all in there, let us know what’s not obvious and Jim can help clear it up. I feel as a dev it’s maybe not a clear path there, so feedback welcomed.
12 Likes
Hi @Josh, 10439365308292854940 is a “local address”. It is in fact the short hand address of a local file containing the datamap which can be used to fetch your private data. Using this local address on another computer doesn’t work indeed because it refers to a local file on the first computer.
If you upload data using
ant file upload --public not_secret.txt
You’ll get a public address accessible by anyone with the address. As we use private data by default in the CLI, one cannot get it via an “address”.
Although in your case, you might want to access that private data on another computer. There are two ways to do so:
- use the
vault syncfeature on the CLI to sync your private data between two computers you own (with the same key). This will save your local private data maps in an encrypted vault on the Network. - manually get the data map from the file at
[your data dir]/user_data/private_file_archives/[your local addr]. If you are on Linux that would be:
~/.local/share/client/user_data/private_file_archives/10439365308292854940
data dir varies depending on your OS:
- Linux
$HOME/.local/share/- macOS
$HOME/Library/Application Support/- Windows
C:\Users\Alice\AppData\Roaming\
For example:
cat ~/Library/Application\ Support/autonomi/client/user_data/private_file_archives/10654665515211232423
Gives me:
{"name":"README.md","secret_access":"81a54669727374939400dc0020cc9066cc99cce5cce40c6bccc722cce0ccadcce7601dcc927d4538073cccfe116070cc8fccaaccc8cc9dccb4cca2006bdc002055055fccbd2accd7ccce69ccbb31ccd4cca7cc8e7213632e31cca5ccb7cca849cc9742ccd620ccd1cc9d064a35ccadcc809401dc00201756cccb660171665dccd0cc956100ccb0ccc41d2e0c7333cca2cc84ccc3cca238ccdd2033ccb0cce50accb4cca4dc0020cca7ccff0e2cccd7ccbb4c142dcca6cc9e58cc984ecc9bccd5ccd4ccc7010cccc7ccfbcca2164d280ccca6510ecc96cc80cc809402dc0020cca01dccd0ccd662cca6ccb0cccc643eccd3cc8dcca9ccde327f43511c4a770b47301040ccc6cc88783acce774dc00201321cc99642426cccd29343850cc8d30cc97ccf96fcc88cc86cca42c39cceecc896accee3eccd9cceb66ccab545bcc81"}
Which I can then use on another computer to get that data (use the secret_access):
ant file download 81a54669727374939400dc0020cc9066cc99cce5cce40c6bccc722cce0ccadcce7601dcc927d4538073cccfe116070cc8fccaaccc8cc9dccb4cca2006bdc002055055fccbd2accd7ccce69ccbb31ccd4cca7cc8e7213632e31cca5ccb7cca849cc9742ccd620ccd1cc9d064a35ccadcc809401dc00201756cccb660171665dccd0cc956100ccb0ccc41d2e0c7333cca2cc84ccc3cca238ccdd2033ccb0cce50accb4cca4dc0020cca7ccff0e2cccd7ccbb4c142dcca6cc9e58cc984ecc9bccd5ccd4ccc7010cccc7ccfbcca2164d280ccca6510ecc96cc80cc809402dc0020cca01dccd0ccd662cca6ccb0cccc643eccd3cc8dcca9ccde327f43511c4a770b47301040ccc6cc88783acce774dc00201321cc99642426cccd29343850cc8d30cc97ccf96fcc88cc86cca42c39cceecc896accee3eccd9cceb66ccab545bcc81 my_readme.md
Hopefully that helps?
9 Likes
Very much so, thank you @Anselme!
5 Likes