Hello Reddthatians, Reddthians, reddies? 🤷 Everyone!

If you were lucky enough to be browsing the site at around 10am UTC you might have seen a down message!

Fear not, we are now back and faster than ever. We are speed.

Images

Using Backblaze was an unfortunate mistake but one that allowed us to migrate to another object store without any issues!
We are now using Wasabi Object Storage for USD$7/month. (Which is slightly higher than the previous budgeted USD$60/year but the benefits outweigh the costs. Instead of the 180ms responses from Australia to eastern USA, we now have 0.5ms responses from Wasabi!

“You’ve lost me Tiff, what does this all mean?”

Well my good friend, it means picture uploads and responses are so fast you should be seeing next to 0 of those pesky red JSON popups while uploading pictures. [0]

Scaling Out

We have successfully scaled out our frontend and backend applications, Lemmy-ui & Lemmy respectively. We are successfully using 2 independent applications to process traffic. This allows for Lemmy to process items concurrently. This also allows us to reduce the downtimes when pushing updates even lower. As the services will now bring 1 of the applications down, perform the updates, bring it back up and then do the same on the other ones.

Now that we have successfully tested scaling on 1 server it is time to purchase more. Which leads me into the database.

Database performance

Currently, our database performance is sub-optimal, and the reason that some of the issues with viewing content arise. Lemmy has scheduled tasks which run so it can pre-populate what is “Hot”. It has to perform this query on all the posts & comments it knows about. I’ve been seeing deadlocks where queries start “waiting” for other queries to finish. This is especially unfortunate as this results in maxing out our underlying storage. Our little server cannot run all 3 at the same time unfortunately.

Thus the need for scaling out.

Purchase plan and Endpoint Migration

Option 1:

Purchase this beefy completely dedicated instance so we can run our database, image service, as well as multiple backend & frontend services. This will be in conjunction to running more backends & frontend on our current server. This will give us a tremendous amount of processing power, and will possibly allow for video uploads as well! [1]

  • $120.00AUD Monthly
  • E3-1230v6 (4x 3.5 GHz Core)
  • 32GB RAM
  • 2x 480GB SSD
  • 10TB data

(If we wanted a NVMe option, it would be $170AUD/month)

Option 2:

Purchase another VPS with our current provider and dedicate that instance to purly database related items. This would free up all the compute power to run our apps.

  • $44.00AUD Monthly
  • 4 Core CPU
  • 16GB RAM
  • 160 GB NVMe
  • 8TB data

Either option I think is valid except it comes down to pricing. Our wonderful donators have currently provided us with … checks open collective … $557.28 AUD to work with. Which is an amazing feat! With $27 in monthly donations.

Monthly Costs:

  • Object storage: $7.00USD/m (About $10.50AUD)
  • Ram Upgrade: $4.50AUD/m Total Monthly Costs: $15.00AUD

So, a quick back of the math calculation will give us:

  • Option 1: ~5 months before we have problems
  • Option 2: ~17 months before we have problems

At this time I think a dedicated server is premature and will put a strain on a budget, knowing that in 5 months we will have a budgeting issue.
Once we have more monthly donators we can revisit purchasing a dedicated server.

I’ll be monitoring server performance with the new multi-frontend/backend system over the coming days, while everyone reads this post. Then probably on the 8th/9th of July (this weekend), I will purchase the new server for a 1 month trial and start setting it up, securing it and validating our proposed system.

Thank you all for being here again! I’m loving all the new communities, new display pictures & hilarious usernames. I cannot wait to see what our little community is going to become.

Cheers,
Tiff

Notes

[0] - Technically there is a 10 second hard limit between Lemmy and pictrs. So the video needs to be uploaded and possible converted into a viable format within 10 seconds. This is also why if your internet is slow you will get the error popups more frequently than others.

[1] - With the dedicated server, we could convert/process videos faster allowing us to possibly get under that 10second timeout. But I do not think it would still be able to be done. Lemmy will need a different way of handling video uploads. Where we tell the users that it is processing. Just like every other video upload site.

[2] - This dedicated server is a similar size to what lemmy.world is running and they have over 7x the active users we have! Probably overkill in the end.