While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.
As I am told, this was the issue:
- There is an vulnerability which was exploited
- Several people had their JWT cookies leaked, including at least one admin
- Attackers started changing site settings and posting fake announcements etc
Our mitigations:
- We removed the vulnerability
- Deleted all comments and private messages that contained the exploit
- Rotated JWT secret which invalidated all existing cookies
The vulnerability will be fixed by the Lemmy devs.
Details of the vulnerability are here
Many thanks for all that helped, and sorry for any inconvenience caused!
Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).
For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.
Very impressed by how quickly action has been taken by this and other instances to patch the issue.
I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.
Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.
Thank you.
In case anyone else is having trouble logging in, my password wasn’t working so I had to reset it from the website.
So now I can log in via a 3rd party app but not the website (with the new password that I reset via the website.)
I’m currently posting from the 3rd party app. Digging around to try and find 2FA settings for Lemmy.
Someone else commented that cookies need to be cleared on the website (again, for anyone else reading this and having issues.)
Further 3rd party heads up – for us nontech refugees:
If it looks like you are logged in, you may not be. I use Connect, and at your reminder, I clicked my acct and it says I was logged in. I tried to comment that Connect login was working, and my comment didn’t show up.
I tried again, only to see an ”error: not logged in” message pop up.
Signed out, signed in again manually, and all is well.
So do a double check, Lemurs. Trust in your actions, not your eyes.
Can confirm. Was like: “Memmy is fine!” — **Narrator:**nope. It was not.
Thanks for the info! Here is my test to see if I’m actually logged in.
For capable people this is a minor annoyance but whenever there’s an “everyone needs to login again” issue, we will lose mere mortal users. In this case it wasn’t even clear that was needed - I appeared to be logged in but nothing worked. Ordinary users give up over things like this. I’ve seen it happen many times on sites where I had access to the analytics. I hope we regard this as a really bad thing to be avoided at all costs and not a “no big deal, just log in again.” Easy for you, easy for me, many others will just bail.
Thanks for the heads-up.
Thank you!
deleted by creator
what steps are being taken to ensure it doesn’t happen again? was any personal data compromised for users?
IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.
First - really good summary and sounds like everyone is working hard.
Cross posting the below comment.
Under GDPR if you have had a data breach you have a legal obligation to assess whether you need to report it and you must make the report within 72 hours of discovering the breach.
There are other types of reportable breaches too, I only mention data as it sounds most likely. You may or may not be subject to PECR which may also have been breached although less likely. I don’t really have enough familiarity with the regulation to discuss that one.
If you are not sure if there has been a breach you may also need to discuss it with the relevant body or make a report.
Please can you update what action you have taken regarding this and if the incident was reportable or not and the reasons why. Edit - from that new information, it sounds like this is a reportable breach.
For a full understanding, it would be good to know if you had 2FA enabled on the compromised account particularly as it had admin privileges and if so how 2FA was circumvented with this exploit.
It would also be good to know what measures you have in place to prevent the same or other malicious attempts on your Open Collective and Patreon accounts as issues with those are potentially more serious. They may not be vulnerable to this, but it is going to be reassuring to know there is good security practice, 2FA protection etc enabled and you have robust procedures in place.
Thanks for letting us know - this is the kind of transparency that I wish the world had more of!
So what happened:
- Someone posted a post.
- The post contained some instruction to display custom emoji.
- So far so good.
- There is a bug in JavaScript (TypeScript) that runs on client’s machine (arbitrary code execution?).
- The attacker leveraged the bug to grab victim’s JWT (cookie) when the victim visited the page with that post.
- The attacker used the grabbed JWTs to log-in as victim (some of them were admins) and do bad stuff on the server.
Am I right?
I’m old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:
- User provided content (post using custom emojis) caused havoc when processing (doesn’t matter if on server or on client). This is lack of sanitization of user-provided-data.
- JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
- How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
- The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.
Am I right? Correct me if I’m wrong.
Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naïve? Maybe.
Marek.
Damn, I go to bed early and I miss everything! Thanks for the quick resolution and transparent disclosure, this place is great!
Don’t worry, all you missed was a lemonparty redirect and some weird picture of some guy’s face and a caption. It was just some basic disruption, glad the staff resolved it quickly though.
Thank you for your work 🙏
This is really good to see such transparency from admins
Ikr! This is really great to see!
Love the transparency!
We need to learn more about the exploit. Lemmy essentially runs on donations, we all need to support this site. I have a lot of faith in lemmy. I am a reddit refugee and i think it is reminiscent of the early days of reddit and not the pos that it became.
Once it’s fixed in Lemmy I’m pretty sure we’ll see in-depth explanations, at the moment though it’s being discussed briefly on GitHub if you’re interested
Good thing we all use randomly generated passwords for every account and always remember to change them every few months.
Can we get another admin to sign off on this being authentic? In other words, short of a signed GPG signature how do we trust announcements after a breach where admin accounts are compromised?
So thats why MalwareBytes gave me this message yesterday.
Wow that’s cool. How did malwarebytes know the website was compromised ?
I think it sees that the browser is trying to execute code that is suspicious (the payload of the XSS was pretty obvious).
probably looking for obvious patterns like
"onload="...in image and link tags, because an onload event handler would usually never be put in those tags otherwise so the only plausible explanation is that it’s a XSS attackI wish I knew. I tried logging into Lemmy yesterday and I was kept giving this message. I thought it might be relevant and saved this snip. I am only about to post this pic now. I did whitelist Lemmy on Malwarebytes after as well.
Interestingly, here is a log when browsing Lemmy over the last week or so.
Believe the derp.foo and .today are both federated instances. Don’t know what the other rows are.
Admin of derp.foo here. My best guess is Hetzner gave me an IP that had been used to host a botnet C&C before. As a precaution I switched to a new VPS; please contact me via matrix at @irdc:tchncs.de if the problem persists.
