Yeah, this just makes it much more difficult for small, niche instances to grow. It focuses sign ups into monolith instances (since you don’t need to do discovery), which imo is pretty central to the issue reddit created in the first place.

Personally, I don’t think lemmys like lemmy.ml, or lemmy.world, or beehub.org should really exist. The community duplication and centralization of content is going to represent a real barrier to effectively replacing reddit (as a viable option). I think it makes a lot more sense to have smaller lemmys focused on niche topics (for example, lofi, or sound engineering, or gardening, or in my case, degenerate financial advise). Each lemmy might be composed of a few communities which are all on theme, but can be federated more broadly into the entire ecosystem, so that cool niche things can be found.

For this to function there has to be some kind of automated community discovery (like you mentioned with a crawler, which it sounds like what I’ll need to make for this to work.)

Already were seeing the issue that the centralization of users and content have created. All the drama lemmy.world and beehub.org have caused. Lemmy.ml crashing and struggling to serve content. Its all a result of centralization, which imo, is antithetical to the principal of the fediverse.