• tal@lemmy.todayEnglish
    0·
    3 months ago

    All of those systems in your homelab…they aren’t all pulling down their updates multiple times over your network link, right? You’re making use of a network-wide cache? For Debian-family systems, something like Apt-Cacher NG?

    Oh. You’re not. Well, that’s probably okay. I mean, not everyone can have their environment optimized to minimize network traffic.

    • [object Object]@lemmy.caEnglish
      0·
      3 months ago

      You can forgejo with a container index enabled, I don’t know if there’s a way to use that as a proxy for downloading containers though.

    • the_tab_key@lemmy.worldEnglish
      0·
      3 months ago

      I set this up years ago, but then decided it was better to just install different distros on each of my computers. Problem solved?

    • notabot@piefed.socialEnglish
      0·
      3 months ago

      Have you tested your backups recently? Having them complete is one thing, having the data you need for recovery is another. Have you backed up your vm configurations and build scripts?

      Go test your latest backup!

        • notabot@piefed.socialEnglish
          0·
          3 months ago

          Ah, that frission of excitement when you come to restore! Will it work? Does it contain that very important file? Is it up to date? How much will future you hate past you if it isn’t there?

  • tal@lemmy.todayEnglish
    0·
    3 months ago

    logging is probably down

    You do, of course have a dedicated rsyslogd server? An isolated system to which logs are sent, so that if someone compromises another one of your systems, they can’t wipe traces of that compromise from those systems?

    Oh. You don’t. Well, that’s okay. Not every lab can be complete. That Raspberry Pi over there in the corner isn’t actually doing anything, but it’s probably happy where it is. You know, being off, not doing anything.

  • Admax@lemmy.worldEnglish
    0·
    3 months ago

    Then it turns out your monitoring system failed and FUCK IT’S BEEN A MONTH SINCE THE LAST PROPER BACKUP

    • wizardbeard@lemmy.dbzer0.comEnglish
      0·
      3 months ago

      Hearbeat notifications man. “Yes I am online” email once a day or so. Yeah it’s more emails to delete but it can be a lifesaver.

        • jeffep@lemmy.worldEnglish
          0·
          3 months ago

          Couple it to your smart watch, backup every 10 seconds, and make it vibrate when successful

          • WhyJiffie@sh.itjust.worksEnglish
            0·
            3 months ago

            you are just making yourself learn to ignore that your smartwatch vibrates. It’s a bit like breathing and blinking, you are so used to it you can completely forget that its happening. if your smartwatch, or phone, or whatever, starts vibrating all the time, you will get used to it and not notice when it stops happening anymore, but also it will hide any actually meaningful notification.

      • Admax@lemmy.worldEnglish
        0·
        3 months ago

        Oh but I have them !
        Every day an email is sent out with the backup status.
        Every day I got my email in the morning with the back up logs.
        For years.
        I associated email received to backup successful, until a month or so when my vpn broke and the emails where just “could not connect”, but it took me a while to bother actually opening the message body as it had always been the same for years.

        So I’ll manage it differently, have the email subject be more explicit about a success or a failure amongst other things.
        Always learning :^)

  • tal@lemmy.todayEnglish
    0·
    3 months ago

    You have squid or some other forward http proxy set up to share a cache among all the devices on your network set up to access the Web, to minimize duplicate traffic?

    And you have a shared caching DNS server set up locally, something like BIND?

    Oh. You don’t. Well, that’s probably okay. I mean, it probably doesn’t matter that your devices are pulling duplicate copies of data down. Not everyone can have a network that minimizes latency and avoids inefficiency across devices.

    • InnerScientist@lemmy.worldEnglish
      0·
      3 months ago

      That won’t work in most cases, all https traffic isn’t cached unless you mitm https which is a bad idea and not worth it.

      Only cache updates those are worth it and most have a caching server option.

  • tal@lemmy.todayEnglish
    0·
    3 months ago

    You have an intrusion detection system set up, right? A server watching your network’s traffic, looking for signs that systems on your network have been compromised, and to warn you? Snort or something like that?

    Oh. You don’t. Well, that’s probably okay. I mean, probably nothing on your network has been compromised. And probably nothing in the future will be.

  • neidu3@sh.itjust.worksEnglish
    0·
    3 months ago

    Barring any hardware issues or external factors, will it run for 10000 years? Any logs not properly rotated? And other outputs accumulating and eventually filling up a filesystem?

  • tal@lemmy.todayEnglish
    0·
    3 months ago

    You have remote power management set up for the systems in your homelab, right? A server set up that you can reach to power-cycle other servers, so that if they wedge in some unusable state and you can’t be physically there, you can still reboot them? A managed/smart PDU or something like that? Something like one of these guys?

    Oh. You don’t. Well, that’s probably okay. I mean, nothing will probably go wrong and render a device in need of being forcibly rebooted when you’re physically away from home.

    • Fmstrat@lemmy.worldEnglish
      0·
      3 months ago

      I built an 8 outlet version of those with relays and wall outlets for… a lot less.

    • FauxLiving@lemmy.worldEnglish
      0·
      3 months ago

      Oh. You don’t. Well, that’s probably okay. I mean, nothing will probably go wrong and render a device in need of being forcibly rebooted when you’re physically away from home.

      *furiously adds a new item to the TODO list*

    • tychosmoose@lemmy.world
      English
      0·
      3 months ago

      If you do have the smart PSU and power management server you probably also went down the rabbit hole of scripting the power cycling, right? Maybe made that server hardened against power loss disk corruption so it can be run until UPS battery exhaustion.

      What if there is a power outage and NUT shuts everything down? Would be nice to have everything brought back up in an orderly way when power returns. Without manual intervention. But keeping you informed via logging and push notifications.

      • TerHu@lemmy.dbzer0.comEnglish
        0·
        3 months ago

        if you can cycle your home assistant with the shelly plug whilst your home assistant is down, yes. from experience it’s really quite annoying to have a smart plug switch off HA…

        • lemming741@lemmy.worldEnglish
          0·
          3 months ago

          HA is on the same proxmox host as the router. So yeah I can end up locked out. Hasn’t happened yet tho! The relay is on my test machine, it’s always nvidia that crashes there.

          • Fmstrat@lemmy.worldEnglish
            0·
            3 months ago

            An 8 switch relay, old Pi, and 8 hardware store outlets can be had for not much more. I did that and let PiKVM control my outlets directly.

            This is the back of my 10" rack before it was cleaned up. Lots of custom work on this that I’ll be posting a page on my site about when complete.

            @[email protected] in case you are interested

  • tal@lemmy.todayEnglish
    0·
    3 months ago

    All of your systems are set up, but are they capable of being redeployed using a configuration management software package? Ansible or something like that?

    Oh. They’re not. Well, that’s probably okay. I mean, you could probably go manually reproduce configurations, more or less.

    • Sabata@ani.socialEnglish
      0·
      3 months ago

      If you know how your setup works, then that’s a great time for another project that breaks everything.

      • cenzorrll@piefed.caEnglish
        0·
        3 months ago

        Saturday morning: “Incus and podman seem interesting. I bet I could swap everything over while the family is out this afternoon”

        Sunday evening: “Dad, when will the lights work again?”

          • tal@lemmy.todayEnglish
            0·
            3 months ago

            Sure. What that guy is using is actually not the most-interesting diagram style, IMHO, for automatic layout of network maps, if you want large-scale stuff, which is where the automatic layout gets more interesting. I have some scripts floating around somewhere that will generate very large network maps — run a bunch of traceroutes, geolocate IPs, dump the results into an sqlite database, and then generate an automatically laid-out Internet network map. I don’t want to go to the trouble of anonymizing the addresses and locations right now, but if you have a graphviz graph and want to try playing with it, I used:

            goes looking

            Ugh, it’s Python 2, a decade-and-a-half old, and never got ported to Python 3. Lemme gin up an example for the non-hierarchical graphviz stuff:

            graph.dot:

            graph foo {
    a--b
    a--d
    b--c
    d--e
    c--e
    e--f
    b--d
}

            Processed with:

            $ sfdp -Goverlap=prism -Gsep=+5 -Gesep=+4 -Gremincross -Gpack -Gsplines=true -Tpdf -o graph.pdf graph.dot

            Generates something like this:

            That’ll take a ton of graphviz edges and nicely lay them out while trying to avoid crossing edges and stuff, in a non-hierarchical map. Get more complicated maps that it can’t use direct lines on, it’ll use splines to curve lines around nodes. You can create massive network maps like this. Note that I was last looking at graphviz’s automated layout stuff about 15 years ago, so it’s possible that they have better layout algorithms now, but this can deal with enormous numbers of nodes and will do reasonable things with them.

            I just grabbed his example because it was the first graphviz network map example that came up on a Web search.