How so?
If it was from me, you should know it was not my intent.
Your point of view is respected even if not agree by. I hope you know this. 
Of course, I do know. I took it to some degree as a reference to my prior post to @vtnerd .
I’ve gotten a bit frustrated on this because, while collaboration in putting resources in play for farming might be useful (local bandwidth or shared NAS, etc.) collaboration of nodes is antithetical to the whole philosophy the network is based upon. If pooling is possible, it’s a flaw in the core protocol and would need to be corrected at that level. Given that I’m certain this is how the core team views it (as confirmed by David Irvine), our talking about it in detail here is . . . disconnected(?). Thus my prior post to @vtnerd , to whom I also apologize if I was preachy. But it’s frustrating.
@vtnerd, I’m just speculating based on my experience with storage. I haven’t had time to examine the actual workings of MaidSAFE or reach much about it yet.
That being said, to clarify my potentially wrong assumption:
- It appears the speed of response is a factor. You mention “time takes to deliver” but as you and others noticed, it’s not sure whether this may actually be time takes to receive. Assuming it’s time to deliver, then the other node needs to contact your node, and your node needs to respond. I (another assumption) assume the vault node that’s semi-idle can respond fast and that it works asynchronously, so it will respond at the speed of the network plus some mathematical operations (hashing, and so on).
- In order to complete the transaction it’ll have to report to its peer what it’s done.
So far we have say 30ms to contact the farmer, say 15ms to process the request and 30ms to respond.
Compare this to disk IO time which may be just a small fraction of that. Considering economics of today’s storage, I think SSDs at this point aren’t interesting.
I have not added data transfer time which should be added, probably another 11 to 500 millisecond per each 1MB.
Ah, I see. Good point. I’d just store data on shared storage such as NAS :-). I’m not sure whether that’d work…
And I’m not competent in the subject you mentioned to give a valid opinion.
If it’s not doing it already, MaidSAFE may need to ensure that each chunk does have 4 copies, so some host-specific info should be used to label each chunk. Although as mentioned earlier the network shouldn’t allocate chunks to nodes that are that close to each other. I guess you’d have to be extremely lucky (same as the likelihood of concurrent data loss of 2 chunks) to get two or more copies of the same chunk on your pool (I like to call it cluster although I was reminded I shouldn’t :-)).
1 Like
It may not seem like it, but I am on your side. I want a fair system where people can share data and ideas, anonymously and securely, AND be fairly compensated for their contributions (resources, labor, etc.). I do not want a system where farmers “over-subscribe” their storage. This would clearly hurt other farmers on the network, because they would be penalized for providing storage but rarely accessing their own data. These are the best kinds of people on the network, and they should be rewarded. This is why I was pushing for information on the merkle tree design, it will help verify that a vault is storing all of the data it claims. I have terrible prose in my writing; I never differentiated between something I may do (pool resources to provide quicker response times), with something I find unethical (dropping users data because it gives me an advantage). I will try to use “the attacker” when I think something should be changed because it gives unfair advantages.
Unfortunately people have to realize that rewarding performance, total storage capabilities, and consistency is likely to give an advantage to people with more money. Some pooling may be necessary for smaller players to keep up with the network. IMO, if the network guarantees anonymity, robust secure private storage, and a fair compensation scheme, then we shouldn’t be worried about people working together. The governments of the world have us running scared so we want to trust NO ONE. David Irvine’s ideals are great, but hard economics could hit at some point. People banding together to run a better operation may not be a bad thing. For example, imagine several 20ish young adults that live together and contribute money so one of their roommates can run a farm. They are a “pool” even if no additional technology was used. Is this a bad thing? The technology will be pushed much further eventually, and lines are going to blur as to whether its beneficial to the community as a whole. Eventually just as you have to trust the technical people to do this properly, I’m going have to trust the econ fanatics that their crazy free market ideas are going to work on a wide open platform. FWIW, see @gibbons934 for my thoughts on economics, as they are aligned closely enough.
You’ve described the SAFE network that you want, not necessarily the one that has been (or will be) implemented. The reality is that spoken ideas and ideals won’t be controlling the network. People trying to maximize their earnings are going to look at the design of the system and possibly the codebase if they wish to exploit it. The codebase includes all libraries and operating system code leveraged by the system, and the design includes existing network technologies that this platform relies on. This is unfortunate, because this system scares me. And not in the usual everything-must-be-regulated-and-controlled-by-worthless-bureaucrats way either. If the technical team isn’t careful, this system could turn against some of its users to provide a great spying platform. I’m mentioning this because people have to be careful about running this software in its Beta form. Personally I’m considering dedicating a machine for testing, no personal data on it at all. I’ve also began thinking about how to put together a easy linux container to “trap” this entire system in a limited resource box that others could use too. Hopefully the financial aspects of the system will encourage ethical security researchers to find holes, and identify ways to eliminate bad players.
I’m going to assume that my ideas haven’t been far outside the boundaries of the system design since the community hasn’t completely told me to quiet down and read more material. Admittedly my past suggestions have ignored some of the network routing design (discussed with @janitor below), which does make response times somewhat unpredictable from farm to user. The trouble I have is that the documentation provides a quick sketch of the system. Some of the documentation is outdated, some concepts are not completely described, and some new ideas are unavailable to me (or are scattered on this forum). I’ll do my best at keeping my thoughts coherent, but its going to be difficult given my imperfect view. I think it would help the documentation if it linked to github folders or files that implemented the design being discussed. I realize some of them will span files, however, a starting point makes it easier than my current guessing method (which is still likely decent).
I like learning, creating, and being involved in something that could benefit lots of people excites me. So I doubt you’ll have issues with me. I’m completely crazy because I care little about money at this particular point in my life. I figure it will find me eventually if I provide enough value to the world. Probably too naive. @dyamanaka I’ll be following your posts with the goal of pushing the network from all vantage points (good and bad).
To clarify, you mean the host that requested the chunk has to report to its managers which vault provided that chunk first?
I’m not sure if this what you are getting at, but the network routing design should cause packets to “snake” throughout the world geographically as I understand it (can @dirvine confirm this?) . This will make it difficult to predict response times from farm to end-user somewhat, especially since it will change if a vault were to do a “power cycle”. Still seems likely that vaults that can respond quicker will have an advantage overall, depending on the weighting for response time with mining. If storage capacity and response time have a good balance in the total reward scheme (there is POR for storage AND coin mining in this network), then perhaps providing excellent response times will be too costly to have any return. Depends heavily on the mining algorithm, and POR algorithm which are in flux (unfortunately from my perspective).
I do have another thought on defeating the snaking design of the network, but I need to think about it more. I’m not entirely sure how to design to prevent it, or even if it should be considered bad. I feel like it had to be discussed before, so I will hunt around the forums and old mailing list too.
3 Likes
vtnerd, I’m similar to you - the docs seem a bit sketchy and strewn around (all understandable, at this point) so I’m randomly speculating what I think (which may be wrong) the system would logically do. In this particular case I thought it has to be a round trip - “farmer” should receive data (which, considering current network speeds and other parts in IO path stack, is probably the longest time segment in this communication), store it (the write can be done asynchronously, in which case RAM could usually (unless you have very little memory or a very fast network) completely compensate for lack of SSD) and respond with a proof that it’s done its work. Once client has received information it tells him that his chunk (validated by teh correct checksum) has been transferred and stored, it can “pay”.
Like I said I’m talking out of my behind and I did not verify that’s how it works, but I can’t imagine some significantly different way. (Who would want to pay before he’s sure that the work has been done?).
To answer your second question, it’s basically what I said above - I think time spent on disk IO would be a very low percentage value (2-3%?) of the total time (from the moment of sending the PUT request to received “chunk stored OK”), but then as you say if QoS (disk IO time, read latency, etc.) time is given disproportionately larger weight, it can be made significant. I agree with you, but I recognize it’s really a policy (not to say political :-)) decision.
Economically speaking maybe fastest nodes should be rewarded, but that’d encourage larger and more powerful farms and maybe turn out to be bad in the long run (like all four copies of a chunk landing in Amazon East, West, London and Tokyo, respectively). It’s very difficult to design this policy up front and expect it to be un-gamable or perfect, so we can only trust the Team that they’ll tune it so that a good balance is achieved.
IMO (and this is difficult, so I don’t event want to make a suggestion or request - maybe in the future), users (clients) could create their preferences/profiles and then all sorts of farmers would be in demand - some for cheap capacity (and slow), some for their consistently low performance (but pricey), and so on. I’m only afraid some folks already see this as inequality (“Everyone should have the fastest network performance”) and will be against this idea just because it’d work the way it should.
I agree. And to clarify my claim (I hope I can do this on your behalf as well), this isn’t to say that David’s ideas are anti-economical (they aren’t), but that even a small departure from free market economics can put some SAFEnet users at a disadvantage (and - which is actually the same thing - create opportunities for freeloading).
If some cost/reward factor is skewed by just a little, over a large user base (or many PB’s) it adds up and creates significant opportunities for competitors, so I hope the Team will consider that when they start fine-tuning the system.
Kinda, not quite though. This is where its hard to explain and document, there are many parts that depend on a lot of rules from different libraries. The toughness of explaining it all is immense. I think you are very close, but (I mean no harm or preachy stuff here) still a little traditional approach. I do this myself so do not take offense (We all do in the office, its easy).
Data will go to where its suited, so its scattered as evenly distributed as a SHA512 is (whose definition of security ensures this, even distribution across an address range for random inputs, its never that accurate though). When it ‘lands’ on a random node that node may be good enough (unlikely) and the data moved to another node (randomly). It keeps moving until it is in a place that suits it (on line time mainly). When a Get is requested, if the node is on line and responds then it may request a safecoin (if it matches the algorithm, not explaining that here, but its based on a number of things aligning in that request).
Anyway I hope this helps. I try and answer a lot but am way deep in code and you know how it is, system code and the correct algorithm are a nightmare to get right, but simple to get wrong if you are distracted.
[on the merkle tree proof, I think we have significantly better 
I know I know, we use proof of retrievability and the ability to respond to random questions about the data (no memory required such as a merkle root and no processing required, well almost none). The data Managers will send random data to all holders of a chunk, they append this to the chunk hash it and send back the result. Any node that cannot or has a bad response is flagged as corrupted and rank reduced etc. It’s very simple and not open to as much attack or forgery attacks on tree roots etc. ]
5 Likes
I agree. All I’m saying is that that is the intended design, so that if pooling is possible (in a bitcoin-type sense) then it is because of a bug at the most basic core level. I’m not capable of conversing at that level. I actually imagine that there are not too many people on the planet at this point who are, though Irvine and the team are certainly making efforts to get as many as possible educated and placed in remote pods.
Pooling of physical resources is, of course, something people will want to do to achieve more efficiency. It’s just not something that vaults/nodes should be able to do. If they can, there’s a catastrophic bug.
As I said, I apologize. I was frustrated. There are many levels to approach involvement with this project, and none of us are the same. But a scan and spot study of some of a bunch of the past topics really would us sync up more easily.
@fergish thanks for giving time to this, but I am afraid the approach you take is not really what I started this thread to engage. By this, I mean you already believe pooling is impossible, and are setting out to prove that. This is not IMO a good way to achieve what we both want: to ensure pooling is either impossible, or just not beneficial.
The method I advise is to start by assuming there are fiendish and beneficial ways to pool, and then try as hard as possible to think them up! If we don’t find any, good. But we’ve not proved they are not possible. If we do find some we can attempt to do something about them.
I admire your attempt to show it isn’t possible, but how about this fiendish plan…
A. The Wrapper Pool:
-
software is built that behaves just like a node - in fact it is built using a large part of the readily available MaidSafe codebase. The only difference is in what it does internally. This is a “pooling machine”.
-
instead of storing information on its local disk, this software communicates with other computers, and they store the data. Those computers run some simple software that links them to the “pooling node”
So to SAFE, the pooling node looks like a normal node, but is is really a collection of computers that simulate their presence on the network through the “pooling machine”.
I dont claim this is worthwhile, but it illustrates that we can think up ways of people hooking machines together and operating as a pool. One advantage here might be for people whose machines can’t be on all the time. Perhaps the pooling node is on all the time and is able to serve most GETS even if individual machines are off some of the time, by keeping multiple copies of the data on different pool machines. This starts to feel quite similar to the motivation behind bitcoin pools no?
B. The Stand-in Pool:
- A variation on the above might be in having multiple machines available to “stand-in” for each other, so the network sees one node that is on all the time, but which is in fact at different machines at different times. So long as a machine is on, it shares in the rewards won by the particular node that is appearing on the network at any time. For this scheme, quite small pools would benefit especially if machines in a pool were spread around different timezones, and modified node software could achieve this (quite small mods I would think). The benefit here is that nodes which are not on all the time, or nearly all the time, will not get much opportunity to farm (as I understand it), but could be able to farm effectively for even small periods of on-time. One can imagine a co-ordinating service allocating nodes to small geographically distributed pools, for which it “owns” the identity of the appearance nodes and allocates them to stand-in machines.
I don’t know if the current code would allow either of these schemes, but they are feasible at least in principle. The problem with both these pooling schemes for SAFE is that it tends to create centralisation risks, both for failure modes and, if a really good pool gets big enough, to enable network attacks.
These are just first thoughts after letting this simmer for a few days, and I have to say that your “disproof” helped a lot in getting me to crystalise them and share them. Together with reading some of the other posts here. So it really has been a team effort. 
I don’t think I understand the details well enough, or am fiendish enough at devising cunning pooling techniques to really do this justice though.
2 Likes
@happybeing Thanks for these examples. I can’t pretend to know with any certainty if/how these and other approaches might achieve pooling. But, as I said, if pooling is possible, it is a bug in the core because it contradicts the philosophy that the network is designed around. We seem to agree on this.
We’d have to get someone familiar with the system overall to say what might or might not exist to allow or counter such. I don’t want to disturb David Irvine on it now cause he’s knee deep in the business of making it happen.
I’m glad to provoke your thoughts.
I didn’t want to say that pooling is not possible, just antithetical to the nature of the network, and hopefully already designed against.
On your examples, I can envision problems with swapping off for each other, considering that they will have to also hand off relations to the other nodes in their group.
I’m not positive how a node’s ID is generated, but I believe it has some component of hashing certain data unique to it.
If any of these pass-offs aren’t perfect all, participants should be tagged as unreliable and eventually ostracized.
Just some ideas.
I’ve gotten a little off-topic as per the original post, so I’ll try to keep it brief.
I don’t know I why I was so stuck on the “this server owns this data chunk” concept. I kept thinking that the vault re-keying was to change its peers, and another static key dictated what chunks would be stored in a vault. I feel so dumb, the strongest feature of this design is changing the data chunks that are stored by a vault. Even if a farm were to store data chunks from the last vault key, it is unlikely to see the GET requests. And the storage locations are constantly in flux as other vaults connect and disconnect.
This makes my over-subscription hack more risky. The algorithm that decides when to generate random questions and which chunks to select will be an interesting. Also, forced vault re-key seems intriguing too, but could unfairly increase bandwidth requirements.
Its easier when the information is in one location (or a few, based on topic). Looking all over forum posts is time-consuming. I can’t fault the team for outdated or missing documentation, it happens in every software project. The code evolves faster than qualified team members have time to document.
Pooling resources across vaults doesn’t seem advantageous (I can’t think how at the moment), provided its difficult to over-subscribe. Having multiple people pool resources for a single vault still seems plausible.
I won’t say this scheme is impossible, but it will be difficult. It would be beneficial to people with poor internet connections that are frequently lost and who can only provide resources from their personal machines. This scheme also differs from MaidSafe significantly, so you’d have to make sure that each person in the pool was trustworthy.
The difficulty is the additional storage space. In my original secondary network scheme, I forgot about redundancy. Regardless of the algorithm involved, you will never want to lose a data chunk that someone tries to retrieve from you. My aggressive secondary network assumed that none of the participants went offline or lost a significant portion of data. Each participant either has to have a highly fault tolerant system or the secondary network has to be similar to MaidSafe (4 copies distributed).
Still, if you can vet some of the people coming into the pool, it might be beneficial for people in certain parts of the world. They would have to trust the pool operator to distribute the funds. Maybe local communities band together to start a pool like this; its easier to trust people you meet personally and see everyday. Everyone pays for an EC2 instance that is “always” running, and everyone in the community shares their hard-drive to reduce the EC2 storage costs (to nearly none).
I’m going to assume this is possible, but multiple people would have to be authorized to use a single account. So its back to trust again as the major problem. I’m also not sure on a scenario where people would do this. I might be lacking creativity right now.
Ah, the terminology. “People,” “account,” “vault”, “node”. But what you’re really talking about here is “computers” sharing a “vault”. I propose that this is a very different critter which the network will not take kindly to.
I think @vtnerd said correctly this could collapse the moment one of temp farmers forgets to power on his vaults and causes everyone to suffer massive losses
It perhaps can be mitigated by smart contracting (a combo that may be suitable for Ethereum, although I’m not a big fan of that platform). Overall, though, each layer of complexity adds to the cost and let’s not forget the cost of operating a redundant “virtualizer” system wouldn’t come cheap either.
Data synchronization among the nodes and depreciation costs would be prohibitive.
But those are creative ideas.
OT: In this discussion my term “clustering” was called wrong, but I still think it’s a good term. We’ll see. So far I am not persuaded that “units of work” can be economically divided among multiple nodes, so the grouping of farmer systems in clusters (e.g. to realize better utilization of bandwidth and so on) still seems viable, while pooling to divide work doesn’t.
Hey everyone, you keep missing the point of this thread - to stimulate us to think of ways that pooling might be possible, multiply and refine those ideas, so that later we can mitigate against them.
There seems to be a tendency of not wanting to think it is possible, which I think is preventing us from ensuring that it is impossible or at least not beneficial.
My suggestions were not intended as serious propositions, to be critiqued, but as example of how to think about this so that the project benefits.
Let’s here more ideas: if you think of a flaw in any of them, don’t post about the flaw, come up with a way around the flaw, and keep going!
If this is in reply to my previous comment, I wasn’t criticizing, like I said the ideas were creative and I also like the discussion. The reason why I couldn’t improve on your ideas is that I didn’t see how (but I thought I saw why it wouldn’t work). If I had an idea on how you make your proposal better I would have.
And I did submit my own idea (way above) which was to store vaults on NFS which I don’t think sound too good either but we’ll find out after we start playing with the s/w.
I said critique, not critisising (which I see differently) and it wasn’t directed at anyone. I’m responding to the general tendency to look for ways pooling is not possible, and trying to explain how I think we can best serve the project. Its like: to catch a criminal, it helps to think like one. Same for hacking etc. If you want to secure a network against hackers, you need to become a hacker to know how to defend.
I’m suggesting we focus on figuring out how to break this (i.e. figure out how to pool farm), in order to prevent anyone from actually breaking it. I’m not critisising, and I did say I’ve found the discussion helpful in my own thinking - but I started this topic because I really don’t feel up to figuring out how to pool farm on my own.
Pooling is a bad idea for this type of network. A pyramid scheme of sorts. One can write software to automate payouts from different IDs if they want it that badly.
1 Like