Status update July 4th
Just wanted to let you know where we are with Lemmy.world.
Issues
As you might have noticed, things still won’t work as desired… we see several issues:
Performance
- Loading is mostly OK, but sometimes things take forever
- We (and you) see many 502 errors, resulting in empty pages etc.
- System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)
Bugs
- Replying to a DM doesn’t seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
- 2FA seems to be a problem for many people. It doesn’t always work as expected.
Troubleshooting
We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @nutomic@lemmy.ml is also helping with current issues.
So, all is not yet running smoothly as we hoped, but with all this help we’ll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!
I’m just excited to be back in the Wild West again – all of the big players had bumps, at least this one is working to fix them.
And giving updates!
I’d rather have to deal with hiccups and bumps along the way, because the community only grows more each time.
I still remember early reddit days of 502 it went through, 504 try once more.
Early? Haha, reddit still shits the bed regularly
Yee-haw, mother fuckers.
That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
Lemmy has a memory leak? Or, should I say, a “lemmory leak”?
A pretty bad one at that…
But… but… Rust…
Rust makes holes and that’s how leaks happen
Underrated comment
Rust protects you from segfaulting and trying to access deallocated memory, but doesn’t protect you from just deciding to keep everything in memory. That’s a design choice. The original developers probably didn’t expect such a deluge of users.
Leaking memory is safe
Rust’s memory safety guarantees make it difficult, but not impossible, to accidentally create memory that is never cleaned up (known as a memory leak). Preventing memory leaks entirely is not one of Rust’s guarantees in the same way that disallowing data races at compile time is, meaning memory leaks are memory safe in Rust. We can see that Rust allows memory leaks by using Rc<T> and RefCell<T>: it’s possible to create references where items refer to each other in a cycle. This creates memory leaks because the reference count of each item in the cycle will never reach 0, and the values will never be dropped.
I’m calling it - if there’s actually a memory leak in the Rust code, it’s gonna be the in memory queues because the DB’s iops can’t cope with the number of users.
I think I found what eats the memory. DB iops isn’t the cause - looks like the server doesn’t reply before all the database operations are done. The problem is the unbounded queue in the activitypub_federation crate, spawned when creating the ActivityQueue struct. The point is, this queue holds all the “activities” - events to be sent to federated instances. If, for whatever reason, the events aren’t delivered to all the federated servers, they are retried with an exponential backoff for up to 2.5 days. If even a single federated instance is unreachable, all events remain in memory. For a large instance, this will eat up the memory for every upvote/downvote, post or comment.
Lemmy needs to figure out a scalable eventual consistency algorithm. Most importantly, to store the messages in the DB, not in memory.
You should consider bringing this up at !lemmyperformance@lemmy.ml
You should take this entire comment and paste it in as a issue on the Lemmy Github Issues page.
I think the devs have been aware of the issue, theoretically, for a while. A proper solution requires some significant changes, so it was being postponed because this wasn’t considered urgent.
Lemmy has a memory leak? Or, should I say, a “lemmory leak”?
A lemmory meek, obviously!
On some of the latest release candidates I think so.
Wait isn’t lemmy written in rust how do you create a memory leak in rust? Unsafe mode?
let mut memory_leak = Vec::<u8>::new(); loop { memory_leak.push(0); }That’s not a memory leak though. That’s just hording memory. Leaked memory is inaccessible.
In the example it’s inaccessible.
It’s not. The vec is still accessible and if it goes out of scope rust automatically clears the memory.
But it will never fall out of scope because of the loop
Exactly but it’s still accessible since it’s in scope.
Exactly.
Really appreciate all the time and effort you all put in especially while Lemmy is growing so fast. Couldn’t happen without you!
I want this to succeed so badly. I truly feel like it’s going to be sink or swim and will reflect how all enshitification efforts will play out.
Band together now and people see there’s a chance. Fail and we are doomed to corporate greed in every facet of our lives.
Wow. Drama! Chill bruh. They’ll get there. If it takes too long for some, they can switch to other instances.
That’s just a lemming living up to their name!
I’m here for the drama too!
Fuck the cleanliness of the corporate web.
Thank you so much for your hard work and for fixing everything tirelessly, so that we can waste some time with posting beans and stuff lol.
Seriously, you’re doing a great job <3
Great distraction from the collapse
i just wanted to thank you for doing your best to fix lemmy.world as soon as possible.
but please, don’t feel forced to overwork yourselves. i understand you want to do it soon so more people can move from Reddit, but i wouldn’t like that Lemmy software and community developers overwork and feel miserable, as those things are some of the very motives you escaped from Reddit in first place.
in my opinion, it would be nice that we users understand this situation and, if we want lemmy so bad, we actively help with it.
this applies to all lemmy instances and communities, ofc. have a nice day you all! ^^
Plus, slow steady growth means eventual success. Burnout is very real if you never take a break.
so true, pal! slowly, with patience, no rushing, putting love into it, organizing ourselves, working smart is better than working hard and fast.
because of the federated nature of fediverses like Lemmy, it is very possible that many people are doing the very same task without even knowing they are duping each other’s efforts.
and that’s sad because if they knew, they could be teaming up, or splitting the task in two, in order to avoid wasting different efforts into dupe results.
i have learnt a thing or two about burnout, it’s better for me to make 40% planning, and 40 % self-care and so the 20 % of execution becomes piece of cake.
but this is just my opinion. anyway, please take care, pals <3
As somebody who flocked to Voat during the height of the Ellen Pao controversy and remembered the site being rendered unusable for whole days at a time from the Reddit Hug of Death, I’m remarkably surprised at how well Lemmy.world has held up. I thought the fediverse would have truly crumbled from this exodus.
I remember when Voat came out and the slight exedous that brought. I made an account and everything but it never properly took off. I checked on it two or three years later and it was just filled with alt-right/racist/transphobic garbage. Sad it never took off as a reddit alternative, reddit likely would have greatly benefited from a proper alternative, not sad it closed down after I saw what it ended up.
So far the fediverse feels really different tho, very explicitly anti that type of shit. I’m sure it will pop up, they always do, but maybe now people know how to deal with it. Block it, defederate, deplatform.
FYI, it has popped up
explodingheadsis a great example but many servers including lemmy.world became proactive in defederating from the instanceYeah, but as soon as instances defederated from there, those bigots started making alts on those instances.
Because it’s not enough for them to have their own safe space for bigotry, they want to argue that bigotry is ok so they get their rage fix from arguing online.
If Lemmy can instances can’t IP ban like reddit does, they’re just going to keep making accounts here every time they’re banned. Then go to exploding heads to circle jerk about how they got banned again.
One can also increase the cost of registration by requiring some questions and a waiting period. Adding friction makes this attack a lot less fun for the attacker.
It’s not a perfect solution, but perfect must not be the enemy of good.
At the same time, it’s also not suitable for a server the size of lemmy.world. We just have too many users, and enough new people joining that we’d need several people whose singular task was to process registrations in that case.
One of the only servers I’ve defederated my personal instance from.
Lemmy is a distributed system that can’t quite be perma-killed, kind of like ye olde usenet. Anyone can create a new instance at will.
As a game dev for bigwigs I know all too well about memory leaks, and so very much appreciate your patch notes, updates, and transparency. You’re doing great with such fast exponential growth
💙 Thanks for your hard work!
Amen. So happy to be done with Reddit.
This is the level of transparency that most companies should strive for. Ironic that in terms of fixing things, volunteer and passion projects seem to be more on top of issues compared to big companies with hundreds of employees.
You said it: passion projects. While being paid is surely a motivator, seeing your pet project take off the way Lemmy is can be so intoxicating and rewarding! I plan to donate as soon as I get paid on Friday! I want to see this succeed, even if it is just to spite Reddit, and I am willing to pay for the pleasure.
What was that? We’re going to need more and better hardware soon, and you have a Patreon and a paypal on the sidebar?
Yeah, that sounds pretty reasonable, we can work with that.
There is a sidebar?
On web browser, yep.
I’m thoroughly confused by most of what is going on. I’m using kbin on mobile. The interface isn’t terrible and it fills the random surfing void that I am used to.
Just a heads up, you’re using kbin which is a different “website” than lemmy. You are also seeing a thread that originated in lemmy. That means both kbin and lemmy can see each other’s content, but the website UI is different between them so if you read about some UI element and can’t find it in kbin that might be one of the reasons.
That being said, kbin also has a sidebar with useful information. In the mobile site you can find it at the very bottom on the page, since it doesn’t fit to the side in such a small screen.
Omg I never scrolled that far down lmao, now I can go block 196 and the other shit posting magazines
Yesss! Lemmy was difficult to use until I did that. Then eventually I had a long enough list of communities/magazines that I could browse subscribed only and that really supposed the issue. I’m busy not a fan of random porn and bigoted memes y’know?
I mean if a random nude crosses my feed I’m not bothered by it but I really just want to read some cool stuff maybe news etc for like 15 mins at a time throughout the day and this is scratching that itch for me.
If you’re on Kbin mobile, you can also tap the hamburger bar on the top left to bring up that menu directly, instead of having to scroll all the way down.
Also great for if you want to fiddle with dark mode and all of that.
Omg so thats what that goddamn button does… thank you
Every community has a sidebar. If you’re at the page for the community it should be at the top of the website. “sidebar+”
If you’re using a 3rd party app it’s different for each one in terms of how to see it.
Just signed up for the 8€/month tier, hopefully my blue checkmark will arrive soon!
It’s here!
✅
(Please do note that the blue checkmark is for our 1€ “Bean” tier subscribers.)
We’re very pleased to provide you with our premium deluxe green checkmark for your generous 8€ subscription to our “Bacon” tier!
Feel free to reach out to our Lemmy Cares team @ 1-800-FUC-SPEZ for any questions or concerns that you may have.
Could I get a discord invite? I’m an ex sysadmin with a. Lot of free time
@ruud > That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
Hmm, makes me curious if there is a Lemmy memory leak, or simply that the load wants to stabilize above of the RAM you have? I hope contributions can help you with another 32 GB RAM? Thank you for your work! 🍻
We have 128GB of RAM. It just skyrockets after a while!
@ruud Oh damn. This spontaneously sounds crazy but I’m admittedly a novice at servers on this scale.
I haven’t checked into the code yet, but I imagine you can map out what all is in memory and force more aggressive garbage collection to find some middle ground.
Rust is not a GC language.
I read down the list afterwards and found it was using Rust. I skimmed through the source and it is well organized, but would still take quite a while to get up to speed on.
I saw unit and integration tests. It might be beneficial to generate or capture some data to replay to simulate the load and add debugging. I don’t know much about the abstraction layers. I did see opentelemetry, which is a project I got frustrated with on the lack of stability (fast changes on api).
I have only dabbled with Rust to test the waters. The largest thing I’ve made was a GUI snake game, and made it portable so it could be compiled for cross platform.
Thanks for all of your effort. Even though we are on different instances, it’s important for the Fediverse community that you succeed. You are doing valuable work, and I appreciate it.
Not just that, but the code contributed to Lemmy by this debugging will make Lemmy run faster for everyone on every instance, which is makes the ecosystem that much better.
The work you’re doing is greatly appreciated! It’s like you invited half the internet into your house. I feel like I should’ve brought a cake or something
You should consider setting up a small donation to the site, if you can afford it. If you throw $5 a month at them, you’re not only covering your own hosting fees, but covering for some of the folks who can’t afford to as well. And maybe Ruud gets to buy some beans.
It appears I have no idea how to imbed an image, but good call. Subscribed.
Of course these performance issues are a bit annoying, but I gotta say that I love these updates and explanations here. Great communication, keep it up, please!
