When querying for “legal” and “law” on lemmyverse.net, it’s starkly clear that many communities are missing from the database. I get far more results when querying on specific instances. So what’s the problem? Is it no longer crawling?

  • activistPnk@slrpnk.netOP
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 month ago

    The law communities I was expecting to find are old and on small instances. When I searched on just the domain portion of a small instance, Lemmyverse finds only like 10% of the communities on that node.

    You would expect LV to not forget a community once it is in the DB. But it must also have a removal mechanism for communities that are deleted. It’s as if the removal detection is getting false positives for removals.

    • FrostyTrichs@walledgarden.xyz
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 month ago

      I understand the confusion.

      Maybe I should’ve been clearer that lemmyverse performance has been unreliable, not just the crawler. Entire instances, communities, and mixes of the two have disappeared and reappeared between scans. It also hasn’t been reliably picking up changes to instances it already knows about.

      TLDR lemmyverse has something wrong with it.