Tiny, decentralized search
TYPE: MUSE
Been tinkering around with SearXNG lately. Thinking about how import search engines have become to the economy and society as a whole. Despite their importance, there are very few competitors in this sector. Search engines are a critical part to the way we discover the internet, and having them owned by only a handful of big tech companies acting as gatekeepers to the internet does not sound like a good idea. I think the reason for this is that it's so difficult to start an alternative. Building a search engine is an immensely difficult endeavor, both financially and technically. Not mention the tech giants will do anything in their power to keep their monopoly. I found an attempt at a blockchain-based alternative to search that interests me: Presearch. However, it doesn't seem like the ideal solution to me. I think the best solution I can think of is a collection of small, high quality, topic focused search engines built by individuals or small organizations (a couple engines for recipe blogs, maybe one for forum posts, a handful for Github repos, etc.) I think while it is near impossible for an individual with limited time and money but good intentions to build a full scale search engine like google, a smaller scale search engine that focuses on trying to crawl a tiny part of the internet is definitely doable. Not does this allow for a more open and decentralized community, it also allows engines to focus on building unique, effective implementations specific to their area of focus. Naturally, the next question is,
If all the search engines are going to be fragmented into hundreds, or even thousands of tiny sites/APIs, how are you going to find the ones you need for each query? A search engine for search engines?!
If there are over 200 million active web pages on the internet1, and each tiny engine indexed on average 1,000 pages, that would mean indexing and searching 200,000 search engines!!! At that point its not decentralized anymore.2 Not to mention what happens if/when the web grows even further. So what's the solution? No clue. Email me if you have an idea, or just want to comment. Also if anyone makes a post as a reply to this post, email me and I'll probably add you to the bottom of my post. (This is an early test for Webmention.
source: internetlivestats.com↩
Of course my mind automatically starts thinking about search engines for search engines for search engines for search engines...↩