While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.
As I am told, this was the issue:
- There is an vulnerability which was exploited
- Several people had their JWT cookies leaked, including at least one admin
- Attackers started changing site settings and posting fake announcements etc
Our mitigations:
- We removed the vulnerability
- Deleted all comments and private messages that contained the exploit
- Rotated JWT secret which invalidated all existing cookies
The vulnerability will be fixed by the Lemmy devs.
Details of the vulnerability are here
Many thanks for all that helped, and sorry for any inconvenience caused!
Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).
For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.
Very impressed by how quickly action has been taken by this and other instances to patch the issue.
Very, seems like great work.
Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.
I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.
In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)
I’m on iOS with the Memmy app. It’s a work in progress that’s officially unfinished so I’m not surprised but it has also been a bit buggy. Doesn’t seem that I can log out without deleting and reinstalling the app so hopefully this doesn’t happen too often XD
So I was actually just struggling with that myself, also in the Memmy app in case that isn’t clear
What I did was add my account (again)
There was no warning or anything, and it populated the list with two of me.
At that point, a “delete account” option appeared under both of them. So I guess in normal circumstances, it wants you to keep one account around at all times?
I deleted one of them, and the app basically reinitialized. Both were gone and it showed me the welcome screen.
I logged back in, and now everything is back to normal
I just did edit account and then saved, it seemed to trick it into logging in again (secrets on my instance were also reset).
Ah interesting. I’ve had multiple accounts from the start so it was much easier for me. Just removed my main account and added it back.
Finally I found good instructions, was about to delete and reinstall until I followed this!
Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.
Whoops, glitched double response.
I did this, but I just didn’t delete either accounts and it worked fine. Idk if it’s detrimental to have two of the same but it worked for me.
Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.
I found I didn’t actually have to log out, just go into account settings and reconfirm everything without changing it
Go into account settings, clear your password, re-enter your password, save, go to feed and pull to refresh. That’s what worked for me.
For Memmy, I went to the accounts page in the settings. Click d on my lemmy.world account then to the page where you can change the password then navigated away. That reactivated the account. Maybe we should add a ticket on Memmy’s GitHub about reactivating cookies when there’s an issue. Or at least place à poput to double check credentials or something.
I was I able to upvote anything or subscribe. Seems like uninstalling and reinstalling fixed my issue
No you can. You just remove the account from the accounts list. It’s labeled “delete this account” which is scary but it just removes it from Memmy. You can add it right back and that logs you back in. Not a great experience.
I sure hope this doesn’t happen a lot. This kind of barrier hurts site growth. I’ve managed a lot of large sites and seen a lot of bugs and when everyone gets logged out there is a measurable impact, and some folks never return. Just look at all the comments here saying “thank I didn’t know to do that.” For every one of those there are 100 people going “huh… Lemmy is down… oh well… on to something else…”
thanks for posting this, I wouldn’t have figured that out lol
Oh, I was wondering why it was showing me as logged in but wouldn’t let me upvote due to not being logged in. Your liftoff psa just cleared that right up for me, thanks!
deleted by creator
In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)
Good PSA. It took me a bit to figure it out, the app doesn’t make this obvious.
How have I never thought of comment hijacking?!
deleted by creator
Removed by mod
uh, why did you have negative one dislike?
Negative one upvotes would mean that enough people disliked me/another poster to bring my upvote total to zero. (Upvotes and likes are effectively the same thing, it’s just a naming convention). Reddit totals them up and seemingly Lemmy does as well.
huh that’s weird (yes I meant negative one downvote), I already know that the total can be either positive or negative, but shouldn’t the upvote number and downvote number be either positive or zero? (for now I’ll just accept it as a lemmy bug/ inconsistencies between instances)
Nope, just like Reddit it’s a value that ranges between negatives and positives. If I get two thousand upvotes, positive 2k. If I get two thousand downvotes, negative 1999 (because iirc you start with one by default).
Not exactly sure I understood what you meant by “either positive or zero”.
see your comment rn, it has 1 upvote (from yourself by default) and 0 dislike (so it’s not shown)
but in the screenshot I sent above you got 287 upvote and minus -1 downvote (making your total 288) which is mathematically correct but seems like an unintended behavior
for example this comment of mine normally have 9 upvote and 2 downvote (which is shown as a positive integer 2, not negative), making my total upvote 7
Just occurred to me that the app I use also shows separate counters. I fooled myself into thinking it was a single counter.
That’s interesting. Remember it’s a very new platform, minor bugs aren’t out of the ordinary.
I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.
Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.
what steps are being taken to ensure it doesn’t happen again? was any personal data compromised for users?
Good point, I’ll update the post.
Also I am curious, what’s the easiest way to currently reach the admins in case this happens again somehow? Two of them on their account have been seemingly inactive for a month and as per your own statement you rarely check your notifications and dms. Is there a discord somewhere for it?
Mail: info@lemmy.world Mastodon: @mwadmin@mastodon.world Matrix: https://matrix.to/#/#lemmy-support-general:discuss.online
Why wasn’t there an info on /lemmy-world.statuspage.io ?
I think the admins that were on it didn’t think of updating the status page…
Would it be a good idea to have a secondary email not attached to lemmy.world in case of a domain hack?
info@mastodon.world
The mail server records of a domain name do not usually point to the same server as other services like Lemmy.
Domain registrar hack could happen too
deleted by creator
So all our cookies are negated now with the JWT changed, and we just needed to login again? Can attackers have stolen our cookies in order to use our accounts to post as if it was us? I’m sure they were only interested in admin cookies, so most others were “useless” to them? I see nothing wrong with my posts so I should be safe, right?
Prior to the JWT secret being rotated, yes, they could have authenticated as you. The tokens are now all invalid and useless
Probably. I had to re-login myself.
If you think they could change your password:
YES, they could.
They could have changed the email => “Forgot PW” and with that you lost ur account.
I think I’ve lost my account, I clicked Forgot Password and nothing came into my mailbox. This account is the one I made just now.
My old account:
If you see that account post or comment on anything, please report it
Edit: Nvm, I use another email to sign up for Lemmy and forgot about it
Report it directly to Ruud or otherwise he will just delete it.
actually nevermind, I forgot that I use a different email for Lemmy, I can log back in now
It happens to all of us. Additionally, assuming that you’ve come here recently, there’s not much data on it, and it being deleted will not be that much of a big deal.
Nice work on the recovery, especially from a 0-day.
deleted by creator
They defaced it with dicks and changed the federation list to be only threads.net. I don’t think it was a state sponsored chinese hacking group. :)
I’m ok with the dicks but the threads are TOO FAR!!! shuffles off to the angry done**
Thank you all for staying on top of it.
right after the update we also had most of the serverlist cleared except threads.net (which was the last one added so i assumed it was some bug) – otherwise nothing appears to be touched on this instance tho.
IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.
That’s something a hacker would say.
🤣
First - really good summary and sounds like everyone is working hard.
Cross posting the below comment.
Under GDPR if you have had a data breach you have a legal obligation to assess whether you need to report it and you must make the report within 72 hours of discovering the breach.
There are other types of reportable breaches too, I only mention data as it sounds most likely. You may or may not be subject to PECR which may also have been breached although less likely. I don’t really have enough familiarity with the regulation to discuss that one.
If you are not sure if there has been a breach you may also need to discuss it with the relevant body or make a report.
Please can you update what action you have taken regarding this and if the incident was reportable or not and the reasons why. Edit - from that new information, it sounds like this is a reportable breach.
For a full understanding, it would be good to know if you had 2FA enabled on the compromised account particularly as it had admin privileges and if so how 2FA was circumvented with this exploit.
It would also be good to know what measures you have in place to prevent the same or other malicious attempts on your Open Collective and Patreon accounts as issues with those are potentially more serious. They may not be vulnerable to this, but it is going to be reassuring to know there is good security practice, 2FA protection etc enabled and you have robust procedures in place.
Thanks for letting us know - this is the kind of transparency that I wish the world had more of!
So what happened:
- Someone posted a post.
- The post contained some instruction to display custom emoji.
- So far so good.
- There is a bug in JavaScript (TypeScript) that runs on client’s machine (arbitrary code execution?).
- The attacker leveraged the bug to grab victim’s JWT (cookie) when the victim visited the page with that post.
- The attacker used the grabbed JWTs to log-in as victim (some of them were admins) and do bad stuff on the server.
Am I right?
I’m old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:
- User provided content (post using custom emojis) caused havoc when processing (doesn’t matter if on server or on client). This is lack of sanitization of user-provided-data.
- JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
- How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
- The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.
Am I right? Correct me if I’m wrong.
Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naïve? Maybe.
Marek.
I’m old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:
I’m a modern web developer who used to be an old-school one.
User provided content (post using custom emojis) caused havoc when processing (doesn’t matter if on server or on client). This is lack of sanitization of user-provided-data.
Yeah - pretty much, though there are some mitigating factors.
Strictly speaking, it was the alt text for the emoji. Alt text is HTML, and rather than allow arbitrary HTML they allowed another language called Markdown. Markdown is “a plain text” language with human readable syntax specifically designed to be converted into HTML.
Markdown is the right format to use for emoji alt texts, but you do need to be careful of one thing - the original purpose of Markdown was to allow HTML content to be easier to write/read and it is a superset of the HTML language. So arbitrary HTML is valid markdown.
Virtually all modern Markdown parsers disable arbitrary HTML by default, but it’s a behaviour which can be changed and that leaves potential for mistakes like this one here. Specifically the way Lemmy injected emojis with alt text into the Markdown content allowed arbitrary HTML.
This wasn’t an obvious mistake - the issue over on Lemmy’s issue tracker is titled “Possible XSS Attack” because they knew there was an XSS Attack somewhere and they weren’t immediately sure if they had found it in the emoji system. Even now reading the diff to fix the vulnerability, it still isn’t obvious to me what they did wrong.
It’s fairly complex code and complexity is the enemy of security… but sometimes you have to do complex things. Back in the “old-school” days, nobody would have even attempted to write something as complicated as a federated social network…
JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS.
Yeah - the Lemmy developers made a mistake there. There are a few things they aren’t doing right around cookies and JWT tokens.
Hopefully they fix it. I expect they will… It was already actively being discussed before this incident, and those discussions have been seen by a lot more people now.
How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
There are several levels of isolation that could have blocked this:
- Users should not be able to inject arbitrary HTML.
- A flag on the page should be set telling the browser to ignore JavaScript in the body of the page - this is a relatively new feature in the web and disabled by default for obvious backwards compatibility reasons, but it should be set especially on a high value target like Lemmy, and I expect once it’s been around a little longer browsers will enable it by default.
- A flag should have been set to block JavaScript from contacting an unknown third party domain. Again, this isolation is a relatively new web feature and currently disabled by default.
- As you say, JavaScript shouldn’t be able to access the JWT token or the cookie. That’s not a new feature in the web, it’s just one Lemmy developers didn’t take advantage of (I don’t know why)
- Even if all of those previous levels of isolation failed… there are things Lemmy should be doing to mitigate the attack. In particular instance admins have had to manually reset JWT tokens. Those tokens should have expired somehow on their own - possibly the moment the attacker tried to use them.
The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.
Yep - the modern best practice is for admins to manage the site via a completely different system. That adds considerable complexity and cost though, so it’s rarely done unfortunately. But you know, Lemmy is open source… so if someone wants to take on that work they can do it.
I’ll add one more - it should have taken less time to close the exploit… but given this is the first serious exploit I’ll forgive that.
Ultimately several of failures contributed to this attack. I expect many of those failures will be corrected in the coming weeks, and that will make Lemmy far more secure than it is right now - so that next time there’s a bug like the one in the Markdown parser it isn’t able to cause so much disruption.
The good news is no harm was done, and a lot of people are going to learn some valuable lessons as a result of this incident. Ultimately the outcome is a positive one in my opinion.
JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
JavaScript needs access to the cookies, they are the data storage for a given site.
To protect them, the browser silos them to the individual site that created them, that’s why developers haven’t been able to easily load cross domain content for years, to mitigate XSS attacks.
The security relies on the premise that the only valid source of script is the originating domain.
The flaw here was allowing clients to add arbitrary script that was displayed to others.
You’re dead right that only the way to fix this is to do away with JavaScript access to certain things, but it will require a complete refactor of how cookies work.
I haven’t done any web dev in a few years, this might even be a solved problem by now and we are just seeing an old school implementation. 🤷this might even be a solved problem by now
Yes, it is called
HttpOnlyand is decided by the server who is sending the cookie to you in HTTP response header. I believe there are also HTTPS-only cookies that when received via HTTPS, cannot be used from HTTP, but I cannot find it right now.Secureis what you’re looking for.
Oh I forgot another line of defense / basic security mitigation. If a server produces an access token (such as JWT or any other old school cookie / session ID), pair it with an IP address. So in case of cookie theft, the attacker cannot use this cookie from his computer (IP address). If the IP changes (mobile / WiFi / ADSL / whatever), the legitimate user should log-in again, now storing two auth cookies. In case of another IP change, no problemo, one of the stored cookies will work. Of course limit validity of the cookie in time (lets, say, keep it valid only for a day or for a week or so).
deleted by creator
mobile devices change IP addresses all the time
I never noticed this. Yes, switch between mobile and WiFi, but this is only two addresses. In case of IPv4 this seems not problem. In case of IPv6, use /64 or /48 (or whatever is now recommended for residential end users) prefix instead of the entire 128bits. I’m not proposing to log-out the suer after IP change, I’m proposing multiple sessions to be accessible at the same time.
Mobile will often switch ip’s on tower handoffs. If you’re driving down the road or on a train, it’s nothing to change mobile ip addresses every 2 minutes.
Not in my experience. But OK, if this is the case, don’t use exact IPv4 address, lookup the routing database and use the sub-net. Or whatever. This is belt & suspenders style of defense in depth, just another layer of security if all others fail. Not core functionality.
I work for a mobile game company. Millions of clients. We deal with this a lot. You can’t even predict that they’ll stay in the same class A. I wouldn’t be surprised if they worked out a way to hand off ipv4 to 6 and vice-versa.
Then you have ISP’s and large work networks who send everyone out under the same NAT/PAT, 10’s of thousands of users all coming from one address.
IMO Providing a public service then trying to identify individuals by network without screwing someone over is a fools errand.
If you’re dipping logs and see one jackass doing something on x IP, you always have to go back and see how many ip’s that jackass is coming from and also how much viable traffic is coming from that ip.
You’re totally right. I just looked at my old jwt cookie and was susceptible to CSRF (cross site request forgery) by virtue of not having the SameSite flag being set. This has since been fixed, but it looks like there might still be changes pending as Javascript is currently able to read the cookie value (the HttpOnly flag is currently set to false, meaning that it is able to be accessed by the browser). While this isn’t a major risk, it does increase the attack surface a bit.
“never happen in a desktop/server application” Just don’t run mystery floppys and CDs right :)?
Or rather Electron apps.
How the attacker got those JWTs? JavaScript sent them to him?
If the “special emoji” code contained a script, it could simply read the cookie value and send it to a remote URL by using a XMLHttpRequest. I don’t think it has to be decoded to be used.
The web being a PoS? Well, you’re certainly right on the money there. Not only that, but it has also become the norm for web technologies to be used in creating desktop applications too.
So we’re bringing the security nightmare to desktop applications, and on top of that, it devours RAM like there’s no tomorrow, all for the sake of faster development.
To be fair, it’s not an issue with “the web.” A bug in JavaScript is not a problem for the web. It’s a problem for people who use JavaScript.
Damn, I go to bed early and I miss everything! Thanks for the quick resolution and transparent disclosure, this place is great!
Thank you for your work 🙏
This is really good to see such transparency from admins
Love the transparency!
Good thing we all use randomly generated passwords for every account and always remember to change them every few months.
Can we get another admin to sign off on this being authentic? In other words, short of a signed GPG signature how do we trust announcements after a breach where admin accounts are compromised?
Good point. I did post about this on Mastodon @mwadmin@mastodon.world
Thanks for the reply!
Don’t fall for it. They’re also an admin on mastodon.world! :)
Now I don’t know who to believe! Is Lemmy even real?
Are we all just lemmings? Oh…
Yes.
Thanks for the work. As a heads up it appears most of the block instances are back however I believe explodingheads is still missing which you may want to confirm.
EDIT: it has been added back to the block list.
Hey how do you check on that?
As of the time of me posting this comment, exploding heads is appearing in my feed with some anti lgbt posts. Idk what’s going on because I’m pretty sure they’re supposed to be defederated currently
