Safe Web Crawler

[JBishop]

@davidpbrown: It is slightly different from clear web

@happybeing: Crawling SAFE is no different to the clear web

Think so too: Either other pages already known to crawlers link to the new page or a scan of GMail messages stumbles over links to this new page. You can also submit it yourself in their webform for this purpose. All other means of initial indexing derive from this?

About submitting pages yourself, according to @Tim87’s proposal it could be done like this for pages on the Safe-network:

I don’t usually suggest things that require extensions to the core, but this is one of them.

The idea is about the semantic tagging of documents, and it requires:

- a new data type (yes, I knowww )
- that can be searched in XOR space in fuzzy way

What the Heck This Is About

“Dimensionality reduction” is the method where we assign an unlimited number of documents to a limited number of categories, with a certain weight.

The categories can be handcrafted, but they can also be automatically generated (e.g. by neural networks or support vector machines) in which case the model is not readily understood by humans, but it is more optimal for similarity search.

“Semantic hashing” is the binary variation of the same idea: age becomes “young” and “old”, and then “young” can be further subdivided into “child” and “teen”, “child” into “baby” and “no longer a baby”, etc.

The resulting semantic hash is exactly like what you think: a looooong binary number. However, unlike in the case of cryptographic hashes, if the XOR distance between two numbers is small, we can be fairly sure that the two things are also quite similar …

In the thread David seems to like this approach, leading to “answer engines instead of search engines”.