Welp. My Forgejo instance got popped with an RCE two days ago by CVE-2026-60004. Luckily, I noticed the following morning and had the day free to figure out what happened. Let’s dive in!

As a homelab enthusiast, I found this a very interesting post. Here are my take aways from the post that I’m implementing myself:

  • Miner detection. I’ve updated monitoring rules to now watch the CPU on my hosts. If the same thing happened to me I would not have been alerted at all as I’m doing simple up / down monitoring. Fixed.
  • Access logging. I turned on access logging for my homelab Caddy instances.
  • Log retention. I have increased the amount and retention of my logging. The hope is this will help me reconstruct what happened after a breach.
  • Logs offsite. The VPS access logs now ride along with the normal backup process, which runs hourly. The homelab side still only gets caught by the weekly VM backup, so that’s next.
  • Closed an open signup. My webtrees instance (genealogy) had self registration enabled, which is the same door this guy got hit through. Oops. Fixed.
  • Built a tool. log-inventory.sh, so “could I actually reconstruct what happened” is a command I run instead of a thing I assume.
  • gaylord_fartmaster@lemmy.world
    link
    fedilink
    English
    arrow-up
    5
    ·
    2 months ago

    I don’t think I will ever be convinced to leave anything on my home network open to the internet no matter how convenient it is.

    • lemmyvore@feddit.nl
      link
      fedilink
      English
      arrow-up
      4
      ·
      2 months ago

      It’s fairly safe as long as you add a strong enough form of access control. For example if you put it behind a VPN, or a SSH tunnel, or require mTLS. Even a key in a custom HTTP header or Basic HTTP auth can be good enough if the key is strong enough.

      You can further decrease the probability of drive-by bots reaching a publicly exposed service by merely scanning IPs and ports if you use a reverse proxy and hide your service FQDNs and IP.

      You can do this by using TLS certs on wildcard domains rather than explicit domains, using explicit CNAMEs for the service subdomains rather than a wildcard domain, and keeping the A/AAAA records on an obfuscated subdomain rather than the base domain. If the bots can’t figure out a FQDN they’re not getting past the reverse proxy even if they find the IP and port.

      This is obfuscation not real security but it cuts down tremendously on bot hits.

        • frongt@lemmy.zip
          link
          fedilink
          English
          arrow-up
          1
          ·
          2 months ago

          The VPN service is open to the Internet. But it’s only one service, versus however many things you are running behind it.

      • hietsu@sopuli.xyz
        link
        fedilink
        English
        arrow-up
        1
        ·
        2 months ago

        I have knockd listening on one very high port number, and only after accessing that reverse proxy whitelists the source ip. Sure some scanners might do a full port scan and try http only after that, but FQDN is required too. Plus I have geo ip list blocking all but my country. Oh and Crowdsec. So far so good…

    • leds@feddit.dk
      link
      fedilink
      English
      arrow-up
      2
      ·
      2 months ago

      I gave up , don’t have the time to maintain it anymore. Changed DNS to make my domain point to local lan IP so everything still works but only at home.

      • village604@adultswim.fan
        link
        fedilink
        English
        arrow-up
        0
        ·
        2 months ago

        I wonder if the person who downvoted you thought you pointed your public DNS record to a 192.168 network, instead of making a change to your local DNS server like I’m assuming you did.

        • jet@hackertalks.com
          link
          fedilink
          English
          arrow-up
          0
          ·
          2 months ago

          If you want https it’s gotta be a external DNS record anyway, might as well point to internal ips

          Depending on how paranoid your setup is not everything will use the DHCP provided DNS servers

          • KubeRoot@discuss.tchncs.de
            link
            fedilink
            English
            arrow-up
            0
            ·
            2 months ago

            If you want https it’s gotta be a external DNS record anyway

            Not sure what you mean by that, but off the top of my head, you can get certificates via challenges that prove DNS control (rather than checking if the DNS points to your server), and you can get wildcard certificates so you don’t even have to expose the existence of subdomains.

            And that’s ignoring the option of using your own CA, which only really works with your own devices, but for local access might be viable.

            • MadPsyentist@lemmy.nz
              link
              fedilink
              English
              arrow-up
              1
              ·
              21 days ago

              So what you do is you have an external DNS that sharing your URL. You have the DNS point “myrealcoolurl.net” at 192.168.1.123. Now when anyone goes to “myrealcoolurl.net” the dns points them at 192.168.1.123 which worse case is their own router, if they are on your network though they get pointed at your server and are loving all your home hosted applications.

              Now you can get a wildcard TLS cert issued for “myrealcoolyrl.net” and apps in your home network can use https. Anyone outside your network still gets whatever 192.168.1.123 means for them in their home network

              • KubeRoot@discuss.tchncs.de
                link
                fedilink
                English
                arrow-up
                2
                ·
                21 days ago

                I’m a bit confused by what you’re saying, because to my knowledge the DNS entry you’re mentioning won’t help with getting a certificate at all. Challenges will typically require either having the DNS entry point to a publicly reachable website (doesn’t get you wildcard) or proving the ability to modify records for the domain by adding something like a TXT record on the fly, which shouldn’t require A/AAAA records.

                So you should be able to do the second type of challenge without a global A record to get a wildcard certificate, and use local DNS for the A record?

                • MadPsyentist@lemmy.nz
                  link
                  fedilink
                  English
                  arrow-up
                  0
                  ·
                  21 days ago

                  The challenge dosent require your site to be public only the DNS to be public. So your second description works.

                  Your serve talks to LetsEncrypt. LE gives your server a challenge. Your server adds the challenge record to your public dns. LE confirms the challenge and issues a dns-01 wildcard certificate to your server. LE never had to vist your site to confirm the challenge.

                  See for yourself and, from a terminal of your choosing, run

                  nslookup madpsy.nz
                  

                  You should get back my local ip which doesnt work for anyone outside of my local network.

  • carrylex@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    2 months ago

    Not updating stuff + Public sign ups enabled (did you even read the setup guide?) + Unrestricted internet access

    Yeah I wonder what could possibly go wrong…

    But great writeup

  • irmadlad@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    2 months ago

    About the only recent even I’ve had was when I was sitting at my desk reading an article. I noticed the mouse slightly move, but I hand no hand on the mouse. I sit there watching it, not manually moving the mouse or typing on the keyboard. It really freaked me out. Turns out, my old desk pad had a slight curl towards the edge the mouse was at, and the curl would make the mouse sense movement ever so slightly. I deployed a new desk pad, and things are back to normal. Whew!

  • IanTwenty@piefed.social
    link
    fedilink
    English
    arrow-up
    1
    ·
    2 months ago

    Firewalling containers’ outbound access seems to be rare but another powerful layer of protection

    • chaospatterns@lemmy.world
      link
      fedilink
      English
      arrow-up
      0
      ·
      2 months ago

      Yeah it’s useful and I’ve been using it for my home lab, but too many of my containers want outbound connections to the Internet. Like Forgejo needs :443 to be able to mirror or remote push repositories, so either I have to identify and allow list github.com, gitlab.com, etc. or end up allowing 0.0.0.0/0:443 which is not very strong protection.

      I’d love for some kind of CNI or network plugin that filtered based on domain name.

  • chunkystyles@sopuli.xyz
    link
    fedilink
    English
    arrow-up
    1
    ·
    2 months ago

    I just recently went through a much more benign, but scary nonetheless version of this.

    I realized that my Ansible directory, that I had made public on GitHub to share as an example to some folks, had secrets committed and pushed.

    It was the direct URL and credentials of an app I developed to store non-PII customer data. Now, it wouldn’t be the end of the world if someone noticed this and scraped the data, but it wouldn’t be good, either.

    Luckily, I have Caddy access logs, and it appears no one ever accessed it.

    So I pulled the secrets out of the Ansible directory and made the repo private, I rotated the credentials, and installed Crowdsec to monitor Caddy access logs and ban bad actors.

    I only noticed the secrets because I had just setup Authelia as an OAuth2 provider for my homelab, and I was adding it to my backup scripts.

    • Appoxo@lemmy.dbzer0.com
      link
      fedilink
      English
      arrow-up
      0
      ·
      2 months ago

      Ayy, just went through the same troubles with Authelia earlier this week.
      Only 4 services I could conmect ao far but still neat to have :)

  • Wispy2891@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    2 months ago

    And detected mostly because the automated script kiddie set up a cryptominer to generate $0.0001

    There are way more valuable and destructive ways once you have RCE like:

    • Searching for committed API keys “anyway my repo is private”
    • Ransomware in case there were no backups
  • PieMePlenty@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    2 months ago

    My server got hacked once and was mining crypto! I noticed it when I heard the fans were at 100% when walking passed the room lol. Turns out, qbittorrent-nox used UPNP for its web admin endpoint enabled by default and qbittorrent can do many things like run custom scripts. My mistake was not changing its login credentials. Exposed my full system. Live and learn.

    • BorgDrone@feddit.nl
      link
      fedilink
      English
      arrow-up
      1
      ·
      2 months ago

      UPNP is a security nightmare, better to never enable it at all or if you must only for those networks where you need it.

    • ShortN0te@lemmy.ml
      link
      fedilink
      English
      arrow-up
      1
      ·
      2 months ago

      My guess, auto updates for major versions is not good practice. Installations potentially corrupt and users come screaming to the project maintainers instead of searching the blame by themselfe.

      forgejo defined the lifecycle for every version so just add a calendar entry for when the supports runs out and then update the 3 characters in the docker compose file.

  • Decronym@lemmy.decronym.xyzB
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    20 days ago

    Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

    Fewer Letters More Letters
    CA (SSL) Certificate Authority
    DHCP Dynamic Host Configuration Protocol, automates assignment of IPs when connecting to a network
    DNS Domain Name Service/System
    Git Popular version control system, primarily for code
    SSH Secure Shell for remote terminal access
    TLS Transport Layer Security, supersedes SSL
    VPN Virtual Private Network
    VPS Virtual Private Server (opposed to shared hosting)

    8 acronyms in this thread; the most compressed thread commented on today has 13 acronyms.

    [Thread #97 for this comm, first seen 7th Sep 2026, 20:40] [FAQ] [Full list] [Contact] [Source code]

  • snrkl@lemmus.org
    link
    fedilink
    English
    arrow-up
    0
    ·
    2 months ago

    I’d suggest hobbling those log links that point to active payloads etc - usually best practice when writing this stiff up, so someone doesn’t accidentally click on them…

  • mystik@lemmy.world
    link
    fedilink
    English
    arrow-up
    0
    ·
    2 months ago

    What version of forgejo were you using? CVE-2026-60004 references gitea 1.27.1

    • doeknius_gloek@discuss.tchncs.de
      link
      fedilink
      English
      arrow-up
      0
      ·
      2 months ago

      Unfortunately, Forgejo only offers versioned image tags, meaning no latest tag. I still had mine pinned to v13 which reached EOL 6 months ago in January, 2026.

  • glizzyguzzler@piefed.blahaj.zone
    link
    fedilink
    English
    arrow-up
    0
    ·
    2 months ago

    The real takeaway is to run your containers ALL as read-only. There’s no way to run executables like this as read-only. They’d download to /tmp but not have executable access and womp womp.

    Many things aren’t read-only friendly, but so far I’ve found all containers can be beat into read-only mode! (Heimdall is one of the worst, shout out to it)

    • Onomatopoeia@lemmy.cafe
      link
      fedilink
      English
      arrow-up
      0
      ·
      2 months ago

      I have a lot of stuff self-hosted.

      Just no outside access. Well, that’s not quite true, I do have a Tailscale VPN for remote management, but only certain machines are allowed to join, joins require my authorization, and access to the account is via two-factor.

      The big thing with external access is to never permit it directly through your local connection. Do it though a Virtual Private Server (VPS) - let it get hit with access attempts. And then only allow specific traffic from the VPS to a specific service in your network using a VPN of some sort.

    • webghost0101@sopuli.xyz
      link
      fedilink
      English
      arrow-up
      0
      ·
      2 months ago

      Don’t let this discourage you.

      You can self host on your local network just fine and never portforward which is what exposes it to the outside.

      I don’t need to watch tv or upload pictures to my server outside of my home. Its perfectly useable.

      When you are ready and feel more confident you can setup your own wireguard vpn and only expose that.

      Then Your device can connected to that vpn tunnels inside and can acces everything like home.

      Currently without vpn your device is at risk every time it connects to a network you don’t own. A third party vpn is not a guarantee your data is safe either. Your home vpn though means all your outside the home networking is fully encrypted and outsiders cant even detect your home adresses is running a vpn to hack. (No ping unless you have the key)

      • SusanoStyle@lemmy.ml
        link
        fedilink
        English
        arrow-up
        0
        ·
        2 months ago

        This! Good advice! I went this path last year so i can give some extra advice for everyone who wants to start:

        Before doing anything, buy a new router and put your iot devices and phones in a isolated guest network. Maintain the router updated. After that, you can work, learn and test without fear. Everything stays in your home.

        Then like @[email protected] already said, once you get confident, you can run a vpn tunnel to access your lan. This setup is pretty robust, hard to fuck up, and doable even for newbies. (If i can do it, everyone can, i assure you). Lots of guides out there.

        Last advice, which you should already be doing, but setup a backup.

  • tofu@lemmy.nocturnal.garden
    link
    fedilink
    English
    arrow-up
    0
    ·
    2 months ago

    Good write up! Since you’re using Komodo already, are your compose stacks in git? Because if they are, check the renovate bot. It will read your files with their pinned image tags, check upstream for newer tags, and creates MRs (can also be set to auto merge) to update. Can be configured per stack.

    • ExperimentalGuy@programming.dev
      link
      fedilink
      English
      arrow-up
      0
      ·
      2 months ago

      What’s komodo? I’m thinking of starting a home server so learning how to harden/automate updates (which is what it sounds like you’re saying) is what I want to learn.