Hmm. Yeh a crawler is definitely useful. But refining indexing and so forth is a big task. One that requires money and/or a lot of time / infrastructure (and in the end, I guess: ads / google / black box company type shenanigans).
To clarify my thinking regarding ‘spam’ here:
Right now there is a problem with marketing content, vs content creation. Or: SEO sites with not so useful copy. Or stolen copy. Scraped. Etc.
This ‘spam’ ideally would be discouraged in a system that uses user reviews of domains… (stack overflow style perhaps).
It’s a problem as noted over in the PtP threads re: piracy etc. How do we find the original creator? Who does that and why would they bother vetting these sites.
Some system of user approval, backed by a crawler as you note, might be able to achieve this… And this could ideally be reflected in search rankings.
Upfront cost for submission/rating discourages automated gaming of the system (would have to, or it’s useless).
This cost could be rerouted to the community. A split to devs/ users. If you’re amazing at rating / categorizing etc, you’d get a bigger share. If that’s reinforced by other users, then you get a bigger share (reputation system). Ideally over and above any submission cost.
The search would be weighted accordingly. Ad free. Cost free for visitors (outside of GET rewards, also routed to the community pool).
This would be manual labour. But that’s kind of the joy of it. Too much of the internet today is low quality / SEO / clickbait crap (IMO). In order to get views in order to get ads. And the ‘quality’ of that is determined by google.
This sort of setup could provide relevant, interesting content. And there’s no central unaccountable, profiteering corporation controlling what gets to the top (with open sourcing all the things).
I’d love to be involved in that sort of setup of a SAFE search engine. I’m really for something ad-free and community driven.