I made a Lemmy instance with a custom algorithm that keeps only the top 20% most unique (=interesting?) posts. It does this by calculating a similarity score between every post on my instance and all posts that came before it. The top 80% of posts with the highest self-similarity get removed instantly.
The idea would be that this allows me to cut through the noise that’s running through the communities, similar to how xkcd-signal attempted to do 20 years ago.
The instance is mostly meant for reading, not posting. So it has a very open federation policy (for now).
If anything, this is experimental. So please let me know what you think! You can see the type of stuff that gets removed in the modlog (https://lemmy.coffee/modlog).
If there are a bunch of posts on a particular topic, shouldn’t it keep at least one of them? Otherwise it would tend to completely filter out the most significant or interesting topics.
Yeah, that’s one of the potential issues that I’m currently looking out for. So far the main thing I can tell is that memes get removed like crazy (https://lemmy.coffee/c/memes@lemmy.world?dataType=Post&sort=New) and the posts on the homepage are generally much less meme-intensive when compared to instances like lemmy.world or lemmy.ml.
Interesting. One of my instance’s guiding philosophies is “Quality over Quantity”. I’ve taken different steps toward achieving that (defederate from the Reddit repost instances, disallow pretty much all content bots, manually/locally mod duplicate posts, etc).
Do you plan to publish your algorithm/filter? Would be interested in seeing if it could be tuned and possibly reduce some of the workload for me.
Do you plan to publish your algorithm/filter?
In an ideal world sure. But I’d have to think about that some more, because in principle I don’t want people to game it :)
Lemmy’s license is AGPL, so you would need to at least publish changes to Lemmy itself 😉
(I don’t know if e.g. the code for the algorithm is separate, in order to have a closed source algorithm with an open source Lemmy fork)
I made no changes to the lemmy codebase, its all done through an auto-moderating bot that auto-removes posts that don’t meet the standard :)
Does GPL/AGPL require you to publish the code even if you are not selling the software? As in I could run a library computer with my custom Linux distro without giving anyone the source, but I wouldn’t be able to publish it or sell it only as binary blobs, right?
Selling is outside the scope of the licence, you can do whatever you want with monetisation, be it free or paid-for.
But any one person that uses your GPL if local, AGPL if local or through a remote service, has the right to request you a copy of the code and you have an obligation to comply and provide itThanks!
Oh, kinda like r9k!
That’s the idea, yes.
I was curious what would happen to the ratio of political posts, specifically Trump/Elon, to other communities, but it feels >= the amount as All on lemmy.world.
None of my superb owls look to have made it through, but I didn’t see them removed in the mod log. We’re a pretty large community, so I’d have thought some would have gotten through. I don’t recall if I saw stuff from any of the animal comms.
I’ll take a look at Superbowl@lemmy.world!
The political posts thing is interesting. So far, mainly news posts with text descriptions remain on the front page (which I personally prefer over the memes, but that’s just my preference). I am having a hard time deciding if I actually find them interesting though… ^^ I think I actually prefer it over my reddit homepage, there tends to be much of the same, usually.
For comparison, see: https://imgur.com/a/xbzMXmQ
I do look to have made it to your feed now! It deleted my 3 posts that were photos with short blurbs and kept my news article that is mostly text.
We rarely get the meme type posts, but if your setup is looking at anything like text:photo ratio, that could be doing it.
I have most meme communities block myself, so I understand how overtaking they can be to a feed. 😁
I’m interested in what you’re doing because I am interested in news and politics, but it would be nice to get rid of most of the garbage US news and the dupe posts, and let a better variety of news come in. I’ve gone to keyword blocking, but while I have “Trump” blocked, I’d still like to see things from other countries about their responses to his actions., for example.
I do very few things explicitly, I just punish self-similarity in a very specific way. I guess posts with actual text in the body are just more unique, given all previous posts on the instance.
Maybe using the filtered posts as a base in combination with some client side keyword blocking will be useful? The keyword blocking would be much more individual for each user.
I’ve been through a few pages, could only find this post about Lemmy apps from !buyfromeu@feddit.org ; https://lemmy.coffee/post/6860?scrollToComments=true , with a single comment (mine).
No posts from !buyeuropean@feddit.uk, while it is much more active. Do you know why?
On the other hand, !world@lemmy.world , !news@lemmy.world and !europe@feddit.org seem to be doing fine
I added the larger communities before starting to remove posts, so there may be historical posts still hanging around. Maybe everything from BuyFromEU was deleted?
You can see the kind of stuff that stays best via homepage ALL > Top last N hours
Seems like !buyeuropean@feddit.uk votes and comments aren’t federated to your instance: https://lemmy.coffee/c/buyeuropean@feddit.uk
They are now :)
If the posts in https://lemmy.coffee/c/buyeuropean@feddit.uk are more interesting than in https://feddit.uk/c/buyeuropean, I am having a hard time deciding that lol
Thanks!
Like this, for example: https://lemmy.coffee/?dataType=Post&listingType=All&sort=TopSixHour
Cool Idea!, Created an account to check it out.
Cool. Let me know what you think!
This is very interesting. I’ve been thinking of how a similar but different system could be implemented. The front page of any instance always seems to have two to four posts by the same person, and I’ve been following a rule that if I notice it I block them to remove their clutter of posts from my feed. Unfortunately, most of these accounts are brand new ones posting memes, so it feels like for every two I swat down, four more take their place.
I was looking for an instance that allowed me to mute all new accounts, or less likely an instance that filters out posts by the same person, but it doesn’t seem such a thing has been created.
This isn’t the same that I wanted, and if I’m not mistaken it only applies to posts inside this instance, since the algorithm is removing them? Or does it filter out all similar posts across the fediverse feed? Still, it’s close in concept.
From what I can tell, the larger instances have frontpages without much user duplication. They have a LOT more memes than I do tho :)
PieFed places an icon next to the username to help highlight such aspects of a person’s account. The one for “new account, less than 2 weeks old” is very useful. Others need tweaking such as “potential unregistered bot that posts far more often than comments”, and “contentious user with far more downvotes than upvotes”. I find it useful not to block people but to simply scroll past or to tailor my response to knowing that info.
In addition to PieFed, there are some Lemmy apps that will do this too, although I am not sure which ones (perhaps check out Sync and Connect) - the trick here is ofc to auto apply it to all accounts as you read through your feed.
I would still argue that thunder is the best app for lemmy
Really, if you don’t mind my asking, why is that? Or do you know somewhere that offers a feature comparison? I seem to hear people saying that more for Voyager than Thunder, and both are FOSS. My own only super brief look seemed to confirm that in some few ways like display of full images in the former whereas in the latter it will only show like a third and then you have to click and wait for it to load to see the rest. However, I have only tried Thunder without logging in, so I guess I’ll give it my account credentials and try it out again:-).
Maybe I misremember but didn’t voyager have ads? That was a turn off for me.
Or maybe that was boost?
That sounds like Boost. Voyager is FOSS.
I want to like Thunder, and I’m sure there’s all sorts of reasons to, I just haven’t seen any of them yet:-).
My mistake then. I have to give Voyager another try, I guess :)
