Thoughts on Safe App Dev + This Week In Frames

[intrz]

@edapa

Some things seem to be impossible (such as maintaining an efficient search index).

Have you looked at these threads?

Semantic search

I don’t usually suggest things that require extensions to the core, but this is one of them. The idea is about the semantic tagging of documents, and it requires: a new data type (yes, I knowww ) that can be searched in XOR space in fuzzy way The idea is based on this paper: Semantic Hashing (Salakhutdinov, Hinton). [image] What the Heck This Is About “Dimensionality reduction” is the method where we assign an unlimited number of documents to a limited number of categories, w…

Indexed Structured Data

It would be useful to have a Structured Data type that is automatically indexed and searchable. EDIT: Here’s a GIST with a bunch of questions and ideas about a possible implementation. It could be used for extremely useful things like semantic search and graph databases. Semantic hashing fits right into the already existing structure of SAFE (it uses XOR metrics for expressing similarity), and much of what we care about can be expressed as a graph database. @dirvine mentioned both deep learnin…

And this post by dirvine

Search though from a decentralised perspective can make use of lucene type indexing, with the indexes using using Immutable Data, this has to be arranged in a manner where it can append information as more links and data is found.As data can be referenced via a datamap hash then the end data will always be found (like way back machine inbuilt to the network).

So the research part if you like will be fitting this in a type of binary tree/Directed Graph where leaves can be updated to reflect new information that fits that part of the object (graph).

So very possible and a few whiteboard sessions could product a crude first attempt at this. If done correctly and without doubt in the open then I think we can have a decent, albeit crude search.

More interesting though will be the use of deep learning algorithms to provide more than search, so more like wolfram type question engines with larger data sets. I spoke briefly with Wolfram and I am sure there is a very long conversation to be had there, with the new data structures available in SAFE as well as their capabilities in domain specific languages in this field.