On Monday, June 26th, 2023 at 07:44:07 UTC, our esteemed host @self posted:

wanna see some code? c’mere

Federation came a few weeks later on 17 July. Our subs now have regulars from across Lemmy and Mastodon.

It’s been a wild party and we’re not stopping any time soon. Go wedgie a nerd today.

  • sue_me_please@awful.systems
    link
    fedilink
    English
    arrow-up
    5
    ·
    edit-2
    7 days ago

    Off topic, but don’t think it’s appropriate for a new thread: I’m curious about hardware and traffic requirements for running a federated Lemmy node.

    Is this instance clustered? What are the specs of the machines/instances running the node? How much traffic ingress and egress are you seeing?

    • self@awful.systemsM
      link
      fedilink
      English
      arrow-up
      3
      ·
      7 days ago

      the actual Lemmy service running awful.systems isn’t clustered yet – all of Lemmy, its Postgres database, and pict-rs run on a single Hetzner Cloud CPX31 with quite a bit of CPU and memory to spare. some services (mostly static hosting for a couple of things, plus the staging environment for upgrades and configuration changes) are offloaded to a CX21 that’s definitely overprovisioned for the usage it gets. the CPX31 hosting us is behind a LB11 load balancer for future expansion – if I need to stand up another instance of the Lemmy frontend, I can live reconfigure the LB11 to round robin onto that host without any downtime. there are downsides to using Hetzner’s LBs – they’re extremely inflexible and basically can’t be configured outside of a typical “round robin and terminate TLS” use case, though they’re very nicely automated (they’ll even manage Let’s Encrypt for you) for that use case.

      as with all federated services, a Lemmy server that’s being used in any capacity will slowly fill its available disk space with posts and associated data. currently we’re still rolling with the storage included with the CPX31, but Postgres and especially the rather inefficient image cache are gradually filling that disk. part of the plan for the deployment is to either offload the image cache to object storage (which can be extremely cheap, but definitely do the math on egress charges) or, more likely (because it helps keep us portable between cloud vendors), I’ll expand the LVM for the node’s disk onto a Hetzner volume when we get to around 75% capacity.

      if you’re looking at establishing an instance along these lines, make sure you look at rate limiting first if you run into any performance issues. before considering any upgrades, check your access logs to make sure you’re not seeing a spike due to malicious traffic. ActivityPub is a unique challenge to rate limit properly since some of your endpoints will always have a ton of repeated, automated traffic from other instances, but there are a few guides out there that have good defaults, and getting this right before you have a ton of users will save you time later.

      other than the above, we have an external email service and backups that I can provide more detail on if you need recommendations as you get closer to rolling out your deployment.

      obligatory: I don’t recommend Hetzner as a company, but David and I have yet to find a host with comparable pricing that isn’t somebody’s hobby or Oracle, a company neither of us will deal with due to personal experience and industry reputation. the above runs about $25/month at the prices I get for Hetzner’s resources (I think on Cloud they lock you into whatever rate you were at when you joined, so mine are cheaper than the ones on their main site), but you may get better value from a server auction or other host depending on your needs.

      • self@awful.systemsM
        link
        fedilink
        English
        arrow-up
        3
        ·
        7 days ago

        and reading this back, it feels like there’s so much free space on that CX21 maybe I should do a best-effort WriteFreely on there, just to justify the budget for the node

      • David Gerard@awful.systemsOP
        link
        fedilink
        English
        arrow-up
        2
        ·
        edit-2
        7 days ago

        on Hetzner - actually looked into what if I move my personal box recently, and Hetzner are still it. the only comparable service for price is OVH and at least Hetzner can probably work a computer. there’s a reason large swathes of Mastodon are hosted on Hetzner or OVH and the reason has a dollar sign on the front.

        • self@awful.systemsM
          link
          fedilink
          English
          arrow-up
          2
          ·
          7 days ago

          god, OVH was fucking terrible when I tried them a few years ago. a lot of fediverse admins swear by them now though so either something changed or the bar for usable hosting and global outages lowered

  • self@awful.systemsM
    link
    fedilink
    English
    arrow-up
    25
    ·
    4 months ago

    🎉🎉🎉🎉 fuck yeah 🎉🎉🎉🎉

    the community we’ve built here is amazing, and it’s thanks to everyone who contributes that we’ve made it to a year!

    our instance has exceeded my wildest expectations, and it keeps pushing the boundaries of the technology that we use to host it. my initial thought behind starting this instance was “huh, I wonder how hard it’d be to deploy Lemmy on NixOS. maybe David and some of the SneerClub regulars would find it useful now that Reddit’s effectively dead?” and now we’re likely about to outlive NixOS’s relevancy (@self a year ago would be horrified at this) and one day soon we’ll even have a true Lemmy fork worth running. we’ve got so much in store for this instance, so let’s keep this party going!

    • V0ldek@awful.systems
      link
      fedilink
      English
      arrow-up
      18
      ·
      4 months ago

      Thank you for weilding the banhammer with grace and ruthlesness required to keep this community awesome.

      We appreciate your sacrifice.

    • Steve@awful.systems
      link
      fedilink
      English
      arrow-up
      5
      ·
      4 months ago

      I love this place and actually have to block myself from it in weekdays because it’s the only way I can get work done. I’m working hard on something that I hope to post to the FreeAssembly channel soon, though! Thanks for making this space and maintaining it so well.

      • self@awful.systemsM
        link
        fedilink
        English
        arrow-up
        3
        ·
        4 months ago

        that’s an awesome compliment, thank you!

        I’ve got a couple of things nearing completion to post to freeasm too! it’s taken a bit longer than I’d hoped to get my code out there, but I’m really excited to see freeasm become a place where we can collaborate.

    • flizzo@awful.systems
      link
      fedilink
      English
      arrow-up
      3
      ·
      4 months ago

      are we going to have to move to guix, I get rms toejam nightmares when I close too many parentheses

      • self@awful.systemsM
        link
        fedilink
        English
        arrow-up
        5
        ·
        4 months ago

        I really hope not — I like Lisp, but a bunch of old gnu folks showed their whole ass every chance they got as things got worse in the nix community, and guix is nothing if not peak gnu. unfortunately we might have to stick with nix until either a viable replacement exists, or the packages we rely on fall so far out of maintenance it endangers the instance; at that point, guix might become the least bad option.

        I’m open to suggestions for other tools or ways forward, though I’m hoping I can keep us off of the set of devops tooling that’s documented mostly in the form of logrocket posts and other breathless SEO spam and gets actively painful to use as soon as you scale past example code, because I don’t particularly want administrating our instance to resemble my day job, where I only tolerate that horseshit because it pays. likewise, native is vastly preferred over containers.

          • self@awful.systemsM
            link
            fedilink
            English
            arrow-up
            6
            ·
            4 months ago

            that was the loudest of the shitheads, yeah! god that was wild — he’s really just some fucker who decided I wasn’t smart enough to contribute to one of his projects who then decided to start reply guying to my posts about gnu being as toxic as nix with “you don’t know what you’re talking about, gnu would never be toxic”, and then shortly after started calling you a… German Nazi who must hate Polish people, because you disagreed with his shit takes about how Free Software means you must collaborate with fascists???

            by and large that’s been my experience with the folks in charge at gnu; their opinions and decisions have been beyond parody in how bad and fucking inept they’ve been. as far as I can tell that rot’s institutionalized in how the gnu project operates, and the project’s successes happen in spite of the efforts of the folks in charge.

            and this is the bullshit power structure that people go out of their way to Do Advocacy for (which itself is just reply guying; notably, FLOSS advocacy almost never helps the person who’s being advocated at actually do anything). I’d give so much for a software commons social structure that doesn’t empower mediocre little white men to be the biggest shitheads they can without repercussions.

  • sinedpick@awful.systems
    link
    fedilink
    English
    arrow-up
    16
    ·
    4 months ago

    Much respect to @self for actually maintaining and hosting open infrastructure and the various moderators for preserving the vibe. Doing either as well as y’all are doing it is not easy. Thank you!

  • Mii@awful.systems
    link
    fedilink
    English
    arrow-up
    14
    ·
    4 months ago

    Happy birthday to the oasis of sanity for everyone working in tech. 🎉

  • Soyweiser@awful.systems
    link
    fedilink
    English
    arrow-up
    13
    ·
    4 months ago

    happy birthday server, and thanks for all the hard work mods, developers/maintaners, commenters/posters and all you lurkers.

  • BigMuffin69@awful.systems
    link
    fedilink
    English
    arrow-up
    9
    ·
    edit-2
    4 months ago

    Cheers lads :) glad that we’ve found a new non reddit home (ty mods for all your hard work ;_; ) , here’s to another year of shit posts