the normal bug report form does not serve this kind of issue. So far, the problem can be observed and reported, but not identified at the code or log levels.
Even after lemmy.ml upgraded hardware yesterday, this posting serves as an example of how comments are not making it to other instances of Lemmy. This is the same posting, different id on each instance:
https://lemmy.ml/post/1239920 has 19 comments (community home)
https://lemmy.pro/post/1354 has 1 comment
https://sh.itjust.works/post/74015 has 9 comments
https://lemmy.world/post/103943 has 9 comments
https://programming.dev/post/29631 has 13 comments
https://beehaw.org/post/536574 has 7 comments
https://lemmy.haigner.me/post/8 has 6 comments (posting originated with user there)
https://lemmy.wtf/post/1021 has 10 commentsOriginally posted by RocketDerp in #3101
I can federate, and see federated instances/communities, I cannot see votes or comments at all.
At least one user on your site must subscribe/join a community for messages to content to start being shared. Did you do that?
Originally posted by RocketDerp in #3101
The federation fix mentioned above is still not merged (https://github.com/LemmyNet/lemmy/pull/3379). It will be included in one of the next rcs, so you should wait a bit with further testing. Anyway pasting different comment counts is not helpful at all.
there is a big amount of lemmy.world posts and comments now visible on the lemm.ee
I’m seeing more post/comment delivery from lemmy.world too. Great news! I’m seeing over 10 delivered comments a minute, most recent 5 minute period: https://lemmyadmin.BulletinTree.com/query/comments_ap_id_host_prev?output=table&timeperiod=5
@sunaurus : lemmy.world upgraded to 0.18.1 today,
https://lemmy.world/post/920294 : “we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.”
To have to fire up several lemmy_server services against a singularity of PostgreSQL on the same underlying hardware implies that Rust isn’t scaling or is running into resource limits that aren’t being logged (in a clear manner).
Originally posted by RocketDerp in #3101
missing 0 unequal 0 0 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 1 unequal 0 1 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 2 unequal 0 2 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 24 unequal 0 25 on https://lemmy.ml/ vs. 3 on https://sh.itjust.works/
missing 0 unequal 0 0 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 10 unequal 0 14 on https://lemmy.ml/ vs. 4 on https://sh.itjust.works/
missing 3 unequal 0 4 on https://lemmy.ml/ vs. 1 on https://sh.itjust.works/
missing 4 unequal 0 4 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 6 unequal 0 7 on https://lemmy.ml/ vs. 1 on https://sh.itjust.works/
missing 1 unequal 0 1 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 3 unequal 0 3 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 0 unequal 0 0 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 0 unequal 0 0 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 9 unequal 0 11 on https://lemmy.ml/ vs. 2 on https://sh.itjust.works/
missing 0 unequal 0 0 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 0 unequal 0 0 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 1 unequal 0 1 on https://lemmy.ml/ vs. 1 on https://sh.itjust.works/
missing 1 unequal 0 1 on https://lemmy.ml/ vs. 1 on https://sh.itjust.works/
missing 1 unequal 0 0 on https://lemmy.ml/ vs. 1 on https://sh.itjust.works/
missing 1 unequal 0 3 on https://lemmy.ml/ vs. 2 on https://sh.itjust.works/
missing 5 unequal 0 8 on https://lemmy.ml/ vs. 5 on https://sh.itjust.works/
missing 3 unequal 0 10 on https://lemmy.ml/ vs. 7 on https://sh.itjust.works/
missing 6 unequal 0 7 on https://lemmy.ml/ vs. 1 on https://sh.itjust.works/
missing 5 unequal 1 9 on https://lemmy.ml/ vs. 4 on https://sh.itjust.works/
missing 2 unequal 0 11 on https://lemmy.ml/ vs. 9 on https://sh.itjust.works/
missing 0 unequal 0 0 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 8 unequal 1 16 on https://lemmy.ml/ vs. 8 on https://sh.itjust.works/
missing 1 unequal 0 3 on https://lemmy.ml/ vs. 2 on https://sh.itjust.works/
missing 2 unequal 0 6 on https://lemmy.ml/ vs. 4 on https://sh.itjust.works/
missing 1 unequal 0 1 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 3 unequal 0 4 on https://lemmy.ml/ vs. 1 on https://sh.itjust.works/
missing 0 unequal 0 2 on https://lemmy.ml/ vs. 2 on https://sh.itjust.works/
missing 5 unequal 0 10 on https://lemmy.ml/ vs. 5 on https://sh.itjust.works/Originally posted by RocketDerp in #3101
I’m experiencing a similar issue on my instance but I’m not sure if it’s a federation issue or an issue with my configuration. My setup is via Docker, and while I can federate, and see federated instances/communities, I cannot see votes or comments at all.
@Nutomic could you take a look at this then please, I believe it’s the same problem but can’t find any reason in the logs to suggest why this is happening.
https://lemmy.zip/post/15952 - our community https://lemmy.world/post/72446 - their community
To add to that, if I look at the user’s profile from our community, there is only one comment: https://lemmy.zip/u/robotsheepboy@lemmy.world
If I look from their community, lots of comments including others in that community: https://lemmy.world/u/robotsheepboy
Is this the same issue? There is no user ban, and no community ban either.
Originally posted by Hellhound1 in #3101
Fresh reports of missing comments and posts: https://lemmy.world/post/406956
I am reaching a point where I feel like the major site operators are not publicly, on Lemmy or Github, acknowledging the scaling problems that Lemmy platform is having. Issue #3203 is happening again today from remote instances trying to reach lemmy.ml - and that is after the upgrade to 0.18
Why am I seeing daily “nginx 500” errors that rapidly return from a browser refresh on Beehaw, Lemmy.ml, Lemmy.world - and none of these major operators have open issues with posting their application error logs and nginx logs for us to get our eyes on? The whole community is in scaling crisis.
Making a fool of myself: https://lemmy.ml/post/1453121
Originally posted by RocketDerp in #3101
Ok, for past hour I’ve subscribed from my remote instance to lemmy.ml communities I had never subscribed to before (at a rate of one community every 2 or 3 minutes). And I just now got my first stuck ‘pending’. SPECULATION: The restart of lemmy.ml server may have helped the problem the first couple hours after upgrade, but it seems to be coming back.
Originally posted by RocketDerp in #3101
https://github.com/LemmyNet/activitypub-federation-rust/pull/52 will help with this.
Issue #3133 is closed due to being a duplicate of the same problem, but it does offer up some more hand-linked examples of how this issue is widespread.
I think instance server operators also need to look at how many users are having ‘pending subscribe/join’ stuck problems to other federated instances. This is one place in the UI that they can see something isn’t right regarding data making it between instance servers. This PostgreSQL query will show you how many pending subscribe/join are in your database:
SELECT * FROM community_follower WHERE pending='t';
– EDIT: I have since opened an issue on this specific problem: #3203I personally am struggling with my communications on these topics, but I am trying to help. I did start to play with a NodeJS webapp to do direct SQL against Lemmy for instance administrators: https://github.com/RocketDerp/lemmy_helper
Originally posted by RocketDerp in #3101
I’m experiencing a similar issue on my instance but I’m not sure if it’s a federation issue or an issue with my configuration. My setup is via Docker, and while I can federate, and see federated instances/communities, I cannot see votes or comments at all.
I’m having the same issue, trying to figure out what I did wrong during set-up.
Edit: My fault, I was using Postgresql 11 instead of 15 - was getting errors trying to add comments to the DB. Woopsie! After upgrading Postgres and my DB, all’s good.
Likely related to this issue, my server log is showing incoming activity that has ‘Header is Expired’ on the HTTP. Lemmy federation logic is very aggressive in using a short time window in 0.17.4 - and it’s entirely possible clock differences between servers and/or retry logic is causing failures. See issue https://github.com/LemmyNet/activitypub-federation-rust/issues/46
The time window has gone from 5 minutes up to 24 hours now in the pending 0.18 release, so maybe this problem will improve.
Originally posted by RocketDerp in #3101
These problems have to do with the PostgreSQL backend and timeouts, as too with the federation HTTP design and timeouts because servers swarm each other with concurrent federation activity. It is not just issue #3379, and the lack of database caching in the lemmy_server application is one of the fundamental causes.
Originally posted by RocketDerp in #3101