Another day, another update.
More troubleshooting was done today. What did we do:
- Yesterday evening @phiresky@phiresky@lemmy.world did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
- @cetra3@lemmy.ml created a docker image containing 3PR’s: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
- We started using this image, and saw a big drop in CPU usage and disk load.
- We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a
return 404in nginx conf for/api/v3/ws. - We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
- We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set
proxy_next_upstream timeout;max_fails=5in nginx.
Currently we’re running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the proxy_next_upstream timeout;max_fails=5 workaround but for now it seems to hold with 1.
Thanks to @phiresky@lemmy.world , @cetra3@lemmy.ml , @stanford@discuss.as200950.com, @db0@lemmy.dbzer0.com , @jelloeater85@lemmy.world , @TragicNotCute@lemmy.world for their help!
And not to forget, thanks to @nutomic@lemmy.ml and @dessalines@lemmy.ml for their continuing hard work on Lemmy!
And thank you all for your patience, we’ll keep working on it!
Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.
Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that’s now started, and I noticed the proxy_next_upstream timeout setting didn’t work (or I didn’t set it properly) so I used max_fails=5 for each upstream, that does actually work.
server load is too low, everyone upvote more stuff so i can optimize more
edit: guess there is some more work to be done 😁
Upvote causes an endless spinner on Liftoff. 😁
I’m getting 504 gateway time outs when I try to upvote
It doesn’t for me actually. Maybe just on Lemmy.world?
I don’t understand your graph. It says you are measuring gigabit/sec but shouldn’t the true performance rating be gigabeans/sec for a Lemmy instance?
aye aye sir, to the upvote machine!
Double the image upload size and you will see more shitposts
Web-ui is very smooth rn… is this .world?
😅
Joke aside, the improvement is like heaven and earth. Love it!. Good work teams!I’m on another instance, but here’s some federated activity for you.
All hail @phiresky@lemmy.world! Today is your day. You have made the single most valuable contribution and you must be celebrated! Bravo! Hurrah!
I was just going to post a meme about choosing either creating activity or spare the server from overloading. Now the joke won’t stick.
deleted by creator
Test:
Upvote if you can see this comment. 👍
The change is noticeable. Good job guys.
Thanks for the updates.
I agree. Felt it immediately when I started browsing. Everything is faster and more responsive, on top of the error messages disappearing
Yup I can even post comments first try, without getting an error! Things are working well!
Really noticeable. Cool update. Thank you, guys! ❤️
This is why having a big popular instance isn’t all bad. It helps detect and fix the scaling problems and inefficiencies for all the other 1000s of instances out there!
Wow. So much smoother today.
Great work.
You dropped this 👑
Damn tootin.
You’re gonna be when you see how fast I can bean post now.
You guys had better quit it with all this amazing transparency or it’s going to completely ruin every other service for me. Seriously though amazing work and amazing communication.
I love the smell of updates in the morning.
My upvote can go through fast now
Good work
Thank you guys for your awesome work!
Also to other people: DONATE TO FOSS PROJECTS. If 50.000 people donate only 0.5€, we have 25.000€ for funding the servers, coding, motivating/ people etc. Just don’t take a cup of coffee for 1 day. We are already 2 millions in Lemmy instances. We can build a decentralized world together!!
You can pry my cup of coffee from my my cold, dead hands.
Will donate anyway, I really want this project to keep going.
euro contributed 🫡
Is there a good link? One for USD?
If there’s a thing you really need to pay in a foreign currency, look into Revolut or Wise. Since I occasionally have to pay stuff in Turkish lira, GBP and donated to USD-only Receivers I like to keep Revolut as my secondary bank account since exchanging one currency to another is completely free!
you inspired me to serve the greater good!
For example, if you speak a second language, you can even help with translation in projects. Its very easy. E.g. I translated the Jerboa (Lemmy client for Android) in Greek 2-3 days ago. I needed only 1 hour to finish and special 15-20 minutes for fixes that I missed, yesterday.
Boy does it feel good to have those reports and understand the work you guys do. It’s really inspiring! Thanks for your hard work, everything has been silk smooth! This instance is really great, Lemmy and its devs are really amazing and I feel at home in a nice, cozy community.
Am I getting this correct: the whole lemmy.world instance run in one single container on one single host?
You’d be surprised at how much performance this kind of setup can squeeze off. Often the limitation is more on the DB/storage than network handling and processing power.
This. Most of the time, the bottleneck will be the database backend.
Curious if lemmy.world uses separate reader/writer instances.
Impressive if true!
if it runs this well on a single container, considering the amount of users it has, I fear the power it’d hold with more
Lemmy is built on an async work stealing runtime and you can get very large instances from Amazon and the like.
I’m sure the instance isn’t massive but number of containers and a single “host” are not great indicators of efficiency.
I’m not sure wtf you just said, but lemmy.world feels very smooth today, so thank you for your continued hard work!
So that’s why it was so smooth today… Great work!
Certainly feels fast today!
Submitting PRs is literally the most effective response that helps everyone who uses Lemmy. Thanks to you all.
This is better optimization than most enterprise devs will see in their lifetimes.
Some managers of the devs are not that interested in significant optimizations… Depends on what incentives and company culture drives them
Some company would rather throw more hardware at the problem and make the devs work on another useless feature no one use
Yes! And that’s a very short seighted solution. And it feels so good to improve performance in code! That’s extra performance for “free” 😎
