I’m no expert on proper search engines (I’m not a programmer at all) but I’m currently working on something very basic that works essentially the way Neo describes, initially just to teach myself a bit of simple programming, and in the hope that it’ll maybe fill in a gap at least during these late stages of the test nets.
I’m always short of time to work on it, but pretty close to having the first version done. Out of curiosity I’ve been wondering what a decentralised version of either my app or a proper search engine would like, and it shouldn’t be too difficult as the next stage of my app. One useful feature for this, which I’m not sure whether exists yet on the network, would be for an app to be able to meaningfully identify or certify itself, so that only that app can alter data in an otherwise public data store.
One interesting product of these sort of thoughts has been that I think decentralised apps of all kinds may settle on a similar structure to each other, whereby there is some level of centralised database (easy to imagine the practical necessity of this in the case of a search engine,) also including some kind of verification system, whilst the raw data is held in a ‘pod’ in the user’s account which can be accessed by other apps, eg. other search engines. In the case of a search engine that data could be generated and certified by a piece of scraper software, and all the user needs to do is point the software in the direction of the web address they want to index, store the resultant data, and add the necessary links to that data in the central database. Whether that database should be baked into the network is also an interesting question I think.
Like I say though, this is very much a layperson’s answer, and I think the field of databases is in itself pretty vast, let alone all the other wonderful things something like Google does, which we now take for granted.