I’ve been seeing the exact same posts, in the same order, with almost no change in up/downvotes. It’s been like this for at least 6 hours.
Did everyone agree to take an internet break and not tell me or is something not working correctly?
my apologies everyone - it was working rock solid for a few days and I got lulled into a false sense of security and took my eye off the site for a few hours :/ I am still working on some alerting to tell me when rabbitmq dies, but it’s obviously got some room for improvement.
In any event, I am aware, I’ve fixed the current problem and the queues are draining now. I’ll work on a better monitoring and response automation system tomorrow.
Literally just sorted by newest to save my sanity and found this.
Is it just fedia.io or is this happening to other mbin instances too? seems like this happens every now and then, and i also find posts from fedia aren’t appearing on lemmy instances until there is server admin intervention of some sort, then it all updates at the same time. maybe the server admin isn’t available due to the jewish new year holiday?
Per @jerry there’s some bug in federation where an incoming message is malformed or something and it causes the message queue to silently freeze. He just has to restart it and things start moving again. Unfortunately it sounds like the mbin devs haven’t got it figured out yet so we deal with the problem now and then.
My apologies - I thought I had the problem fixed, but clearly not. It’s working again, and I will get back to fixing the alerting and auto-error correcting tomorrow.
Thank you for your work!
No worries man. I appreciate you taking the time and expense running the instance.
My apologies - I was off doing other things today as I thought this problem had been fixed. Things are almost complete with catching up now and I’ll be working on the problem more tomorrow.
Yea, federation keeps breaking unfortunately. Hopefully an automated solution can be implemented.
Btw, can someone tell me how to ping a user? Admin could probably use a heads up.
I’ve tried to look up how to DM a user and I haven’t found a way yet, so I’d like to know as well.
To DM someone open their user profile by clicking their name then click the little envelope icon. To ping someone just mention their username preceeded by an ampersand like so: @dhhyfddehhfyy4673
Cool, thanks! And just a heads up, but ampersand is &, don’t think the at symbol has an equivalent one word term.
Just checking in after not having been on here for a few hours, and yes, completely.
