As the reddit mods gets ready for the June 12-14 black-out, there some anticipation that an influx in user base will shift over to many of the lemmy instances as user seek out a home to post their internet memes and discuss their interests.

In anticipation of this increased volume I will be growing our current instance from

  • 16 CPU
  • 8 GB ram

to

  • 24 CPU
  • 64 GB ram

This server is currently equipped with SSDs that are configured in a raid 10 array (NVMEs will come in the next gen that get deployed)

Earlier today I also configured some monitoring that I’ll be watching closely in order to have a better understanding on how the lemmy platform does under stress (for science!)

I’ll be sharing graphs and some other insights in this thread for everyone that is interested. Feel free to ask anything you might be interested in knowing more of!

EDIT: I’ll be posting and updating the graphs in this main post periodically! Last updated: 6:21AM ET June 12th

CPU - 48 hours

Memory - 48 hours

Network - 48 hours

Load Average - 48 hours

System Disk I/O - 48 hours

  • phase_change@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    I’d love to see a few additional charts with the next update:

    • Disk space consumed
    • Subscribers
    • Communities federated

    My guess is that’s not available vie SNMP for Librenms. :)

    • TheDude@sh.itjust.worksOPM
      link
      fedilink
      English
      arrow-up
      9
      ·
      1 year ago

      I’m being a little private about the total amount of disk assigned to this instance for now. I do plan on sharing these details when I have implemented a viable solution. That being said I can tell you that with the amount of current users and activities this instance is growing by about 20GB per day in disk size.

      • LetterboxPancake@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        5
        ·
        1 year ago

        Holy shit… You tell us when you’ll accept donations, right? This doesn’t have to be your financial responsibility.

        Or do you and I just haven’t found it yet?

        • TheDude@sh.itjust.worksOPM
          link
          fedilink
          English
          arrow-up
          2
          ·
          1 year ago

          Not taking donations at the moment. Still need to figure out the best way to collect donations and haven’t had the chance to think about it! I will eventually

    • God@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Joined from recommended a bit less than an hour ago. Tried lemmy.ml, got rejected, went for this one cuz it was top recommended and I figured with the name and the user count the likelihood of rejection was smaller.

      • Acheron@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        Just joined you from that featured site, it just works. People are complaining about how hard it is to switch, I genuinely don’t know what the difficulty is

  • borari@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    I’m starting to get random 500’s, I guess that’s a sign I need to go to bed and let the server breathe lol. Can’t wait for the updated graphs!

  • Dhrun@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Yeah I’m trying Lemmy anticipating that Relay Pro may stop working soon. Searched for my sub reddit subjects, seemed to work though some of them seem a bit empty so far.

  • manifex@sh.itjust.worksM
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Thanks @TheDude. I work in enterprise network/systems/cloud operations as a network/security engineer. Would love to contribute monetarily or with time.

  • TheDude@sh.itjust.worksOPM
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    1 year ago

    A little update for all of you interested. I allocated the additional resources to the VM and will post some updated graphs once they update with the new configurations.

    For those who are like me and like looking at graphs here are some prior to the upgrade.

    CPU - 48 hours

    Memory - 48 hours

    Network - 48 hours

    • donut@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      1 year ago

      Check out Vultr. Super customizable. Have not gone down since I got mine (currently has 900 days uptime).

    • crius@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      I’ve recently migrated a forum community (vbulletin -> discourse) using a vpn by Contabo (germany hosting). Their price are incredible and while you don’t get an official SLA, it’s around 99.5%.

    • TheDude@sh.itjust.worksOPM
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      It probably doesn’t need that much but I’m curious in seeing how much It can handle until it can’t I have more resources available but it would make more sense after this size to start separating the different components of Lemmy into their own individual instances across multiple servers instead of trying to scale it vertically. Might require a little more time than I have but I’ll see what I can manage!

      • Jumuta@lemmy.ml
        link
        fedilink
        English
        arrow-up
        0
        ·
        1 year ago

        huh, interesting.

        How much room is there right now for lemmy to scale horizontally?

        • TheDude@sh.itjust.worksOPM
          link
          fedilink
          English
          arrow-up
          1
          ·
          edit-2
          1 year ago

          A couple hundred gigs worth of ram and around ~100 cores. I’m not sure I would want to bring it to that size without thinking first of a plan to properly maintain, manage and support.

          • imperator@sh.itjust.works
            link
            fedilink
            English
            arrow-up
            0
            ·
            1 year ago

            Just joined. I tried a couple of instances but couldn’t get signed up. Yours worked!

            Could Lemmy handle having the same insurance running on multiple servers with a load balancer and database syncing?

            I don’t know anything, just asking lol

            • TheDude@sh.itjust.worksOPM
              link
              fedilink
              English
              arrow-up
              1
              ·
              1 year ago

              Welcome! In short it is possible however would require some engineering to include a kubernetes deployment type. That is already more than the typical user can handle so its likely not something the lemmy developers would support themselves and would require some SREs or devop engineers to review and deploy, maintain that type of deployment themselves. That’s unfortunately more time then I have to give currently.

              • imperator@sh.itjust.works
                link
                fedilink
                English
                arrow-up
                0
                ·
                1 year ago

                Cool thanks. Was just worried about scalability (in general). As people will generally get drawn to the more popular servers. Do you have a patreon or something to allow donations for server costs?