For performance reasons, I would let each identity maintain one big list of (public) actions they performed on other identities. Each identity would periodically grab the lists of all identities in it’s own list above a certain score threshold, which is then merged with their own list (hashmap merging and lookups are very fast). When a piece of content with signature endorsements is encountered, each corresponding publicID can be looked up in the merged list, and if present and above a certain score, the endorsement is counted. I think it should be possible to track about 10.000 identities in a single MB that way.
3 Likes
I think “classic” SD will be insufficient for many of the use cases in the long term; it would be awesome to have a skinny little data type just for storing references. This would do away with the need for these clumsy workarounds.
These references wouldn’t even need to be mutable, and their address (which matters only to determine storage location) could be computed from their content. They would have a fixed size (just a few hundred bytes) so their storage and lookup could be highly optimized.
It’s impossible to search SD for content (e.g. the references they may contain) so there’s no way to say “I want the tags belonging to this document” or “documents with this tag” unless somebody aggregated them (relying on which becomes a SPOF.)
If we had a very simple data type for [owner, category, label, reference, signature] that could be searched for a combination of owner, label, reference, it would be trivial to ask questions like “stuff tagged by this person” or “stuff tagged by this label” or “tags by this person for this document.” This could be the basis of many different applications, including simple tagging, search engines, RDF triplestores, WoT entries, etc
(the “category” field above is just a way to let one owner come up with independent sets of labels)
In my semantic search feature idea I also suggested a fuzzy search for the “label” in the above scheme: there are ways to compute (or manually assign) binary tags where values with a small Hamming distance (“XOR distance”) depict semantically similar content, which would allow for “search by example” (documents about related subjects, similar images, …)
3 Likes
This is why there are reserved types and chat of a domain specific language (next step). Ultimately I would love to be able to have something as we do for publishing web sites, but semantically. So we have a reserved SD type that acts as we wish but demands tags to be filled in.
There is room for this for sure. Please keep chipping away at this one, it’s important.
7 Likes
Let me see if I get what you mean. SD is just a very generic and opaque storage type, but there will be a way to introduce type specific extensions that say “this type must validate against this specific structure, and it should respond to this specific kind of search”? Because that’s really exciting.
4 Likes
Is this an absolute? I noticed that a few forum members support BTC lower down the thread. As you mentioned it will most likely bump the price.
If you will only accept MAID when do you intend to start the sale so that I can time the purchase of additional MAID appropriately.
1 Like
More than a few, many people have asked me to accept BTC in private as well. Today I settled on a fairly easy mechanism to do this. Poloniex has an API call in which the 24 hour weighted average price of MAID can be requested of a particular time frame, so my plan is to accept BTC after all, and for every transaction the past 24 hour weighted average BTC/MAID price will be taken to convert the BTC amount to a (virtual) MAID amount, which will be used in further calculations to figure out who gets how much.
Doing it this way is convenient for MAID contributors, because it won’t matter then if the price of MAID jumps up in the latter weeks of the crowdsale. In the end, a MAID is a MAID, no matter at what price point it was contributed.
15 Likes
Seneca, not only are you brilliant but you show a great willingness to listen and adapt to the needs of many in the forum. Wisdom well beyond your years. Congrats!
9 Likes
It is variable size up to 100KB
There is the addressing info (address, tag_type, owner(s) etc) and whats left is available for data. It does not need to be the 100KB but I gather what ever size the address/owner info is plus your data up to 100KB.
Owner info is variable size since you could have multiple owners or just one.
Yea I get that. I’m not sure if it’s a legit concern, but it feels like when people think about SD, they assume a few kilobytes, maybe tens of kilobytes, worth of data.
I think the overwhelming majority of SD blocks will be just a few hundred bytes, storing metadata, tagging information, etc about the “actual” (bigger) data, which can affect the economy: should a PUT cost the same for a 100KB SD than and 250-byte SD?
rambling alert Maybe SD is not the best choice for metadata. These are just simple statements: “I say this document was made at this time”, “I add this tag to this document”, and changing them makes little sense (though revoking may do.)
I think about it this way: immutable blocks are pictures and movies, structured data store someone’s own … structured data, I guess? But there could be a 3rd type to store the “dust”: random stuff that random people can say about random things; these could be immutable but deletable.
Well, this of course could be a type of SD; apart from my above concerns, which may well be unfounded, there’s nothing about them that SD (with the proper extensions) couldn’t do.
1 Like
I assume you know SD only costs when you create it, rewrites are free of cost.
I would not store single tag info in one SD, but rather store multiple entries in each SD. Effectively address your data with the index into the SD as part of the address.
That is until SAFE comes up with a better data structure so suit your purpose.
3 Likes
Yup. But these things would be mostly static.
But I’m talking about searching. I consult Google possibly a few dozen times on an average day; I can’t live without quick and effective search.
Your approach have a problem: Multiple entries in one SD => “related in one way, unrelated in one or more ways” <= not ideal for searching.
How would you do you something like “NEO tagz video X with word CAT”? It would be easy to let people search for “stuffz tagged CAT by NEO”, because you could just compile them into one document. How about “stuffz tagged CAT by ANYBODY”?
Any kind of “real” (i.e. un-curated) search will need such tiny, simple statements floating around like dust, easy to look up.
Otherwise, a noble hero would need to arise and announce they collect CAT tags. People who believed our hero (or cared enough) would send in their tags ideas, which in turn would be complied (and maintained) into a single SD, publicized, and then whoever decided they trust this person’s honesty and competence could use that pre-compiled index to find cat videos…
Much Unprettiness 
Yea, I’m brainstorming my heart away because Project Decorum reminded me how great something like this would be. Like here.
Can solve that, not hard, but does add an extra layer. A bit more code in a layer between the program and the accessing of the SD elements. Code write, and sometimes longer data access are the problems with that.
Its amazing what you learn to do when programming on a mainframe in assembler and how to organise search data without any formal data structures. Then later with micros then OSs like cp/m, DOS, UNIXflex, etc.
Buy hey its way better if the underlying data store has the storage objects more suited to the task at hand 
[quote=“Tim87, post:134, topic:6119”]
Yea, I’m brainstorming my heart away because Project Decorum reminded me how great something like this would be.
[/quote]
Yea. I might even get back to some database programming myself. Helped with a SQL data engine write back in the 80’s for a little while.
2 Likes
Whoa that’s some history there 
From a SAFE point of view, I just can’t see how data like that could be shared using “plain” SD…
We need to know the address of the block, right? But who tells us the address? I mean, these blocks would be shared by random people… I can’t see a way to coordinate them without having to assume some sort of authority, which we don’t like, do we…
I mean, it’s all going to be solved by what David wrote above; SD will be able to handle this. I’m just considering whether these index thingies are different enough from the “usual” SD use case that it would be useful to treat them completely separately; it may not be, but I’m not 100% convinced.
That is handled in the added layer. And likely require a second layer of indexing, thus the sometimes increase in access time. But this is theory since [quote=“Tim87, post:136, topic:6119”]
I mean, it’s all going to be solved by what David wrote above;
[/quote]
Pumping back up this thread to share something cool I think we can do with this kind of tech.
Let’s call the beginning of a Decorum thread the “root”. Using the Decorum protocol we can figure out where to find the next reply in XOR space. That’s pretty awesome (seriously that’s brilliant). But we still need to know the address of the root to get to the conversation. The simplest method to get the address of the root is when the owner of a Safe page provides it directly. This allows anyone with a Safe page to easily have a conversation going. That’s staggeringly innovative.
Here’s a twist.
What happens when we derive the address of the root from something else? For example, from the XOR address of something else on Safe? Here’s what happens: you now know where to find the conversation related to any piece of data stored on Safe. Any movies, pictures, songs, blog post, everything can now be discussed directly. The only thing you need is a program that check if these conversation exist whenever you travel on Safe. If they don’t, you know how to start them and whenever someone else stumble upon it they will be able to take part of it. Pretty cool right?
Why stop there though? How about we derive it from a cleanet’s URL?
We can now have an unmoderated conversation directly linked to any piece of data found on the clearnet. Let this sink in for a moment because that’s quite empowering. Any news, any advertisements, any press release, absolutely everything is now up for debate, anonymously, and available on a global scale. All you need is a program or a plugin that check if the conversation exist on safe, if it does you just join. It’s like a 4chan that is linked to absolutely everything, which is as awesome as it is scary.
And all we need for all this is to agree on a simple standard.
Take the address of something(XOR or URL), add something else we agree on, for example: “side_talk”, hash it, and this is where you’ll find the discussion related to this piece of data on Safe. If it’s taken just hash it again until you find it. That’s it.
Pretty, freaking, awesome.
16 Likes
I think that the vulnerability here is, I can form the hash using key words, and discuss something totally off topic.
TL;DR, you can make the address to a discussion using some keywords, then if you want to find something about “those words” form the hash and see what’s being said about “Apples, www.apples.com, side_talk” hash it, search, and see what comes up. right @DavidMtl?
2 Likes
That’s a brilliant concept.
I’m thinking how cool this would be for any specific news story;
You’re reading an article on the regular internet, then you have a plugin to interface with Safe / Decorum that shows all available article-specific channels for chats, ratings, reviews, links to similar / related articles elsewhere, relevant poll results related to the article, etc etc.
These ‘channels’ could be up & down voted so that spammy / off topic nonsense doesn’t clog up the experience for regular users, and people can customise the kind of channels they want to see. Clikes could be used to promote specific discussions.
This user generated data would be open and could enable incredibly powerful search features for the web.
There may need to be an option for protecting ‘tags’, e.g. “side_talk” may be completely open, but the “yougov_polls” channel may be curated by yougov when they link their poll data to specific articles.
Interesting - there may be lots of potential for this!
11 Likes
[quote=“DavidMtl, post:139, topic:6119”]
Let this sink in for a moment because that’s quite empowering. Any news, any advertisements, any press release, absolutely everything is now up for debate, anonymously, and available on a global scale. [/quote]
Wow! This amazing idea deserves it’s own thread for sure.
3 Likes