Almost every single one of my posts has had major issues. Even those from other instances. e.g. Rimu messaged me about this one that did not federate for 2-3 days, and consequently was seen by very few people. And here’s one from a difference instance that made it to its destination on !tenforward@lemmy.world, but from its originating server I could see none of the comments, and had to respond from a third account involved in that 3-way attempt at communication. (a post talking about such federation issues on that same server) So to be very clear, I am not saying that instances running PieFed software are having issues, but more that the issues are with Lemmy regardless of software type run.
Uh, right. Sorry, I did not notice you were from Piefed… I was talking about the times when we had the borked Lemmy updates… Did you ever debug or resolve your issues? Is there a way to tell something didn’t federate? And is this an issue specific to Piefed? Or to the whole Fediverse? I’m not sure if I’m affected. I occasionally check my posts from another account and it always seems okay. But I mean I don’t do it very often.
Yes often I’ve chased it down to some small degree. The one that Rimu messaged me was near a time of great instability on piefed.social and I got a bunch of gateway errors even so much as trying to reach it as a user (having nothing to do with posting I mean). This instance has calmed down a ton since then and works perfectly these days, but things happen from time to time. Likewise the incident with StarTrek.Website that I mentioned and provided a link describing more.
Other occurrences have still other causes - e.g. a few of them in one community seems to have been due to my posts getting “locked”. I have no idea why - perhaps the new mod was just fat-fingering the button? I did not ask. But now the vote counts vary GREATLY (192 vs. 183 vs. 98 vs. 0 etc.) depending on which instance you view it from. If you want to test for yourself, a good one to use is https://piefed.social/post/330559 - though I notice that (fairly recently created) community !tech_memes@lemmy.world does not appear at all on your instance.
The primary cause though was a limitation in how the ActivityPub protocol was implemented in the Lemmy codebase, and not having anticipated that ~80% of the entire Lemmy-based Fediverse would concentrate itself onto a single server, Lemmy.World. So how it works is that any “action” - a post, a comment, an upvote or downvote - will be federated out to all the other instances world-wide at a rate of 1 per second. However, if the ping from the other servers to Lemmy.World is itself a significant approximation of that, then the list of actions to be federated will fall behind and take longer to catch up. Eventually after more than a week it gives up entirely, but in the meantime an action can be delayed for days. Poor Aussie.Zone - geographically distant from the EU - has been really having a hard time of it ( https://aussie.zone/post/13429731 ).
Fortunately this problem has already been fixed in the Lemmy codebase by allowing multiple actions to be sent in parallel (https://github.com/LemmyNet/lemmy/pull/4623) - however, what causes the continued problems nowadays is the fact that Lemmy.World is still awaiting that upgrade to 0.19.6 to make use of that change in the codebase (release notes) (actually now 0.19.7 is already out too, having come less than a week after the former, and representing just a few bugfixes, release notes). When Lemmy.World upgrades to one of those, a good deal of these systemic issues should calm down, by a GREAT deal, if not entirely.
Afaik, there is nothing particularly special about instances running PieFed having troubles connecting to any Lemmy instances. In fact it seems rather stable compared to many (even most!) others - particularly StarTrek.Website that has poor uptime. In fact, https://piefed.fediverse.observer/list reports that piefed.social has a remarkable uptime rate of 99.89, which I very much believe, compared to the aforementioned StarTrek.Website’s rate of 98.20, although a year ago when I left it it must have been significantly poorer b/c it would be down for days sometimes, and every single action took like a minute sometimes, back then. Your own instance reports 98.60 - does that sound right?
Rather, it is Lemmy instances - particularly smaller ones (e.g. https://lemmings.world/post/14171987) - having trouble federating specifically with Lemmy.World.
And then recently there were a bunch of instances having troubles connecting to lemmy.ml too (https://lemmy.world/post/22196027) - though this one is more expected as that one is administered by the developers of the Lemmy codebase, and thus that is the place where they test out all of their new code in beta, prior to deploying it across the entire Fediverse. Sometimes that leads to some REALLY odd behaviors, such as entries disappearing from modlog files that were extremely concerning to people, but it is par for the course with that highly special instance, which is unique in its manner.
Edit: ah and I neglected to answer one of your questions: as you said, the way to tell if something federated properly or not is to check the instance - specifically the one hosting the community that you are sending it to. So e.g. to check a post to !tenforward@lemmy.world, I would visit Lemmy.World. If it is there but not on your home instance, then at least that particular message packet got sent, even if the message packet from Lemmy.World to your instance got lost or fell behind in its processing backlog somehow.
Thank you for all the detailed explanation. Seems I wasn’t aware of lots of current events. Especially regarding Lemmy. I always thought it can’t be too hard syncing posts to 45k (monthly active) users… I guess there’s still some way to go.
By the way, Piefed does not pull in remote communities by default. It only does it once a local user subscribes. And that’s why a lot of them are missing on my own instance. I skipped quite some of the meme communities when I switched. And I’m already trying to foster smaller instances. I don’t subscribe to communities on lemmy.ml and I’d like to have an alternative to lemmy.world. But you’re right. A lot of the activity here happens on lemmy.world and we can’t do without.
And last Lemmy instance I used was discuss.tchncs.de which always seemed fine. But it has a very capable admin and is located in Germany, so probably not too far out.
My uptime should be less than other servers. I’m using it for testing and development. And sometimes I break stuff and it’s down until i figure things out.
By the way, Piefed does not pull in remote communities by default. It only does it once a local user subscribes. And that’s why a lot of them are missing on my own instance.
Oh yes absolutely - and Lemmy is the same way, so this too is not something specific to PieFed. For instance, that post I mentioned previously where my vote counts are all over the place, when viewed from StarTrek.Website has zero comment associated with it, and no upvotes beyond the default - and I have another post that is the same way, though it seems like later that same day someone subscribed and from that point onwards the community starts to have comments and some of the votes seen from other instances.
Or perhaps it is that these posts were locked somehow / for some reason? Which looks to be accidental from a new, inexperienced moderator in this brand-new community, and it was reversed a couple of days later - although that fact again depends on where you look. With an account at StarTrek.Website, I look at the post moderation history and it says that it is still locked: https://startrek.website/modlog?postId=16510256. However, with an account on DiscussOnline (substitute with whatever other alt you may have - hendrik@lemmy.ml?) I see that not only was it unlocked, but that unlock event happened 7 days ago: https://discuss.online/modlog?postId=13575162.
Even so, the post when viewed from DiscussOnline shows 98 upvotes and 8 comments, but when viewed from Lemmy.World (where the !tech_memes@lemmy.world community is located) it shows 191 upvotes and 9 comments (or I think it’s 193 upvotes and 2 downvotes, but the web UI no longer shows those individually, unless you jump through many many, undocumented, hoops - e.g. I think I can see those broken down into their individual components on a mobile device in Firefox, possibly solely when viewing a individual users list of posts but not when looking at a post directly or in the standard community view, and definitely you cannot see this breakdown from either Chrome or Firefox on a desktop, etc.). And since it has been 7+ days, this is now enshrined in stone, and we can be confident that having not caught up by now, it never will. A decade from now, if e.g. DiscussOnline is still with us, it will show this post as having 98 upvotes rather than the true value of 193, and StarTrekOnline will still show the default upvotes=1 and no comments, thus providing 3 different stories for this same identical post, depending on how you try to view it - and only one of those stories being explanable by the fact that nobody on StarTrek.Website had subscribed to the community yet (MAYBE, b/c there are 2 other posts that are even older in that community, which have +1 upvote added!? so perhaps this is a complex mixture of that + the locking effect, with the unlock action having not been propagated correctly).
The above stories reveal - federation is NOTHING AT ALL LIKE EMAIL. In the latter, the message either gets passed or it does not, whereas in federation, you can see partial messages as I’ve shown. And this has not even begun to delve into the variety of defederations that further complicate any mess - especially with a unidirectional defederation where one account can talk to someone on a server that has defederated from them, though the recipient will never be alerted to that fact nor have the capability to respond. Thus it is my opinion that trying to fit the square peg into the circular hole is never going to work - the email analogy is hopelessly simplified, so much so that as soon as users begin to encounter such complexities when they make their posts, especially the content creator types that we very much want to come here, they may outright leave, and moreover be very vocal about how we are not what was promised to them. So while we could say “it’s a little bit like sending email”, I don’t think we should push too hard on that avenue, making it sound so simple, b/c it’s really not!
Lemmy.World is still awaiting that upgrade to 0.19.6 to make use of that change in the codebase (release notes) (actually now 0.19.7 is already out too, having come less than a week after the former, and representing just a few bugfixes, release notes).
And will Lemmy.World wait until that is fixed do you know? MrKaplan said perhaps a week ago that they were not ready to announce any plans, but perhaps you’ve heard something since then. I don’t know how stable 0.19.6 itself is in that case, or if they need to first wait until 0.19.8 and then wait further for the bugs in it, like 0.19.7 (and 0.19.6 too), to be discovered and patched.
Almost every single one of my posts has had major issues. Even those from other instances. e.g. Rimu messaged me about this one that did not federate for 2-3 days, and consequently was seen by very few people. And here’s one from a difference instance that made it to its destination on !tenforward@lemmy.world, but from its originating server I could see none of the comments, and had to respond from a third account involved in that 3-way attempt at communication. (a post talking about such federation issues on that same server) So to be very clear, I am not saying that instances running PieFed software are having issues, but more that the issues are with Lemmy regardless of software type run.
Uh, right. Sorry, I did not notice you were from Piefed… I was talking about the times when we had the borked Lemmy updates… Did you ever debug or resolve your issues? Is there a way to tell something didn’t federate? And is this an issue specific to Piefed? Or to the whole Fediverse? I’m not sure if I’m affected. I occasionally check my posts from another account and it always seems okay. But I mean I don’t do it very often.
Yes often I’ve chased it down to some small degree. The one that Rimu messaged me was near a time of great instability on piefed.social and I got a bunch of gateway errors even so much as trying to reach it as a user (having nothing to do with posting I mean). This instance has calmed down a ton since then and works perfectly these days, but things happen from time to time. Likewise the incident with StarTrek.Website that I mentioned and provided a link describing more.
Other occurrences have still other causes - e.g. a few of them in one community seems to have been due to my posts getting “locked”. I have no idea why - perhaps the new mod was just fat-fingering the button? I did not ask. But now the vote counts vary GREATLY (192 vs. 183 vs. 98 vs. 0 etc.) depending on which instance you view it from. If you want to test for yourself, a good one to use is https://piefed.social/post/330559 - though I notice that (fairly recently created) community !tech_memes@lemmy.world does not appear at all on your instance.
The primary cause though was a limitation in how the ActivityPub protocol was implemented in the Lemmy codebase, and not having anticipated that ~80% of the entire Lemmy-based Fediverse would concentrate itself onto a single server, Lemmy.World. So how it works is that any “action” - a post, a comment, an upvote or downvote - will be federated out to all the other instances world-wide at a rate of 1 per second. However, if the ping from the other servers to Lemmy.World is itself a significant approximation of that, then the list of actions to be federated will fall behind and take longer to catch up. Eventually after more than a week it gives up entirely, but in the meantime an action can be delayed for days. Poor Aussie.Zone - geographically distant from the EU - has been really having a hard time of it ( https://aussie.zone/post/13429731 ).
Fortunately this problem has already been fixed in the Lemmy codebase by allowing multiple actions to be sent in parallel (https://github.com/LemmyNet/lemmy/pull/4623) - however, what causes the continued problems nowadays is the fact that Lemmy.World is still awaiting that upgrade to 0.19.6 to make use of that change in the codebase (release notes) (actually now 0.19.7 is already out too, having come less than a week after the former, and representing just a few bugfixes, release notes). When Lemmy.World upgrades to one of those, a good deal of these systemic issues should calm down, by a GREAT deal, if not entirely.
Afaik, there is nothing particularly special about instances running PieFed having troubles connecting to any Lemmy instances. In fact it seems rather stable compared to many (even most!) others - particularly StarTrek.Website that has poor uptime. In fact, https://piefed.fediverse.observer/list reports that piefed.social has a remarkable uptime rate of 99.89, which I very much believe, compared to the aforementioned StarTrek.Website’s rate of 98.20, although a year ago when I left it it must have been significantly poorer b/c it would be down for days sometimes, and every single action took like a minute sometimes, back then. Your own instance reports 98.60 - does that sound right?
Rather, it is Lemmy instances - particularly smaller ones (e.g. https://lemmings.world/post/14171987) - having trouble federating specifically with Lemmy.World.
And then recently there were a bunch of instances having troubles connecting to lemmy.ml too (https://lemmy.world/post/22196027) - though this one is more expected as that one is administered by the developers of the Lemmy codebase, and thus that is the place where they test out all of their new code in beta, prior to deploying it across the entire Fediverse. Sometimes that leads to some REALLY odd behaviors, such as entries disappearing from modlog files that were extremely concerning to people, but it is par for the course with that highly special instance, which is unique in its manner.
Edit: ah and I neglected to answer one of your questions: as you said, the way to tell if something federated properly or not is to check the instance - specifically the one hosting the community that you are sending it to. So e.g. to check a post to !tenforward@lemmy.world, I would visit Lemmy.World. If it is there but not on your home instance, then at least that particular message packet got sent, even if the message packet from Lemmy.World to your instance got lost or fell behind in its processing backlog somehow.
Thank you for all the detailed explanation. Seems I wasn’t aware of lots of current events. Especially regarding Lemmy. I always thought it can’t be too hard syncing posts to 45k (monthly active) users… I guess there’s still some way to go.
By the way, Piefed does not pull in remote communities by default. It only does it once a local user subscribes. And that’s why a lot of them are missing on my own instance. I skipped quite some of the meme communities when I switched. And I’m already trying to foster smaller instances. I don’t subscribe to communities on lemmy.ml and I’d like to have an alternative to lemmy.world. But you’re right. A lot of the activity here happens on lemmy.world and we can’t do without.
And last Lemmy instance I used was discuss.tchncs.de which always seemed fine. But it has a very capable admin and is located in Germany, so probably not too far out.
My uptime should be less than other servers. I’m using it for testing and development. And sometimes I break stuff and it’s down until i figure things out.
Oh yes absolutely - and Lemmy is the same way, so this too is not something specific to PieFed. For instance, that post I mentioned previously where my vote counts are all over the place, when viewed from StarTrek.Website has zero comment associated with it, and no upvotes beyond the default - and I have another post that is the same way, though it seems like later that same day someone subscribed and from that point onwards the community starts to have comments and some of the votes seen from other instances.
Or perhaps it is that these posts were locked somehow / for some reason? Which looks to be accidental from a new, inexperienced moderator in this brand-new community, and it was reversed a couple of days later - although that fact again depends on where you look. With an account at StarTrek.Website, I look at the post moderation history and it says that it is still locked: https://startrek.website/modlog?postId=16510256. However, with an account on DiscussOnline (substitute with whatever other alt you may have - hendrik@lemmy.ml?) I see that not only was it unlocked, but that unlock event happened 7 days ago: https://discuss.online/modlog?postId=13575162.
Even so, the post when viewed from DiscussOnline shows 98 upvotes and 8 comments, but when viewed from Lemmy.World (where the !tech_memes@lemmy.world community is located) it shows 191 upvotes and 9 comments (or I think it’s 193 upvotes and 2 downvotes, but the web UI no longer shows those individually, unless you jump through many many, undocumented, hoops - e.g. I think I can see those broken down into their individual components on a mobile device in Firefox, possibly solely when viewing a individual users list of posts but not when looking at a post directly or in the standard community view, and definitely you cannot see this breakdown from either Chrome or Firefox on a desktop, etc.). And since it has been 7+ days, this is now enshrined in stone, and we can be confident that having not caught up by now, it never will. A decade from now, if e.g. DiscussOnline is still with us, it will show this post as having 98 upvotes rather than the true value of 193, and StarTrekOnline will still show the default upvotes=1 and no comments, thus providing 3 different stories for this same identical post, depending on how you try to view it - and only one of those stories being explanable by the fact that nobody on StarTrek.Website had subscribed to the community yet (MAYBE, b/c there are 2 other posts that are even older in that community, which have +1 upvote added!? so perhaps this is a complex mixture of that + the locking effect, with the unlock action having not been propagated correctly).
The above stories reveal - federation is NOTHING AT ALL LIKE EMAIL. In the latter, the message either gets passed or it does not, whereas in federation, you can see partial messages as I’ve shown. And this has not even begun to delve into the variety of defederations that further complicate any mess - especially with a unidirectional defederation where one account can talk to someone on a server that has defederated from them, though the recipient will never be alerted to that fact nor have the capability to respond. Thus it is my opinion that trying to fit the square peg into the circular hole is never going to work - the email analogy is hopelessly simplified, so much so that as soon as users begin to encounter such complexities when they make their posts, especially the content creator types that we very much want to come here, they may outright leave, and moreover be very vocal about how we are not what was promised to them. So while we could say “it’s a little bit like sending email”, I don’t think we should push too hard on that avenue, making it sound so simple, b/c it’s really not!
On the other hand, 0.19.7 still has a picture issue: https://github.com/LemmyNet/lemmy/issues/5196
Ah, so is this what happened to lemmy.cafe?
And will Lemmy.World wait until that is fixed do you know? MrKaplan said perhaps a week ago that they were not ready to announce any plans, but perhaps you’ve heard something since then. I don’t know how stable 0.19.6 itself is in that case, or if they need to first wait until 0.19.8 and then wait further for the bugs in it, like 0.19.7 (and 0.19.6 too), to be discovered and patched.
Edit: in any case, thanks for sharing!:-)