Nuclear fission has been used in nuclear reactors for good, and was exceedingly efficient at that time, and has been a significant factor in the recent human history. Just as it’s been with the “evil” use in nuclear weapons.
I’m not aware of any breakthrough that has not been used for both good and evil.
This is a pure marketing stunt, OpenAI saw that it worked with Mythos and wanted to do the same.
Those instructions, according to OpenAI, called for using “complex attack paths” to test how well the AI could exploit a computer system. […] “It went off and did this hack all by itself, as far as we can tell,” said Colin Shea-Blymyer, a cybersecurity research fellow at Georgetown University’s Center for Security and Emerging Technology.
Emphasis mine. So, the frontier model that has been tasked with attacking a system, did what it’s been told to, after its safeguards that will prevent you from opening chatgpt and doing the same have been turned off.
OpenAI said the intrusion was caused by a combination of its AI models, including its newly released GPT‑5.6 Sol and an “even more capable” model that is still being tested internally.
Oh my god, an “even more capable” model? Let me buy OpenAI stock immediately. This is such an obvious bait and Antrophic move they’re pulling off, it’s amazing.
I’m not assuming he’s not competent, and I’ve looked him up - he’s by no means incompetent. But he himself said he’s not qualified to write tests for that. If you cannot write tests for whatever you’re doing, you shouldn’t be doing that. Someone with his knowledge, or at least the knowledge he should have given his CV, should know that. In this specific case he is incompetent, because what he’s doing is simply wrong on every level.
You don’t need to be an expert on what you’re doing to use LLMs efficiently. You can also have solid prompts and ideas to use a LLM to cancel out your personal lack of knowledge in a specific domain. In any case, expecting LLMs to produce correct output when you’re actively guiding it to do something wrong is simply stupid.
Any claim of actual intelligence in a LLM is simply not true. Never been, never will be. Artificial intelligence is an umbrella term for ANI, AGI and ASI, artificial narrow, general and super intelligence respectively. A narrow intelligence is not even close to human intelligence, and is hyper-specialized in a single task. All and any LLMs are and always will be ANIs, and their hyper-specialization is basically a stochastic word (well, token) completion on steroids. An AGI is mostly defined as “close to” or “approaching” human intelligence, as in general knowledge and transfer of it into unrelated fields.
This, reasoning and capabilities will help you nothing when you guide it in the wrong direction. You need to keep in mind the absolutely mind blowing amount of money involved around LLMs. The bubble is too big to fail. Any LLM is a product, and their first and foremost goal is to make you use it, so you pay for it - therefore the primary directive of the AI is to give you what you ordered, to glaze you, and to be your best, obedient buddy. You want a video of the bug, of course! Here you have a video of how that bug looks like - stochastically that’s the answer to the prompt.
Naturally, this code didn’t have tests
Codebase with no tests, check.
it was a UI interaction bug for which I’m not even really qualified to write a test for
What the hell are they doing in bugfixing an UI bug, when they are “not qualified” to write a test for it. Anyhow, not competent enough for the codebase you’re working on - check.
so I asked Codex to bisect between dates X and Y to find the commit that introduced this bug.
So, instead of asking the LLM to e.g. create a proper reproduction as a test case, asking it to bisect, which the author claimed that I wasn’t possible, for some reason. So, also adding can’t bisect on his own, and can’t prompt properly, check and check.
[Waffling about hallucinations] I then asked it to show me by making a video with the full developer end-to-end stack in the normal browser test environment. […] The video made it look like Codex had reproduced the bug, but it was an artificial browser environment that was designed to create a fake repro, not the real environment.
So, the author realized it hallucinates. The author asks for video proof (instead of a fucking test, again). The author is surprised it generated him a video of exactly what they wanted to see, only creating it in a different way than they wanted to.
This reads like “I have close to zero clue what I’m doing, I also don’t really know how to achieve what I want properly, and now I’m making a salty blog post that my magical text microwave didn’t fix my half-assed description of a problem”. Like, honestly, what the hell was the expectation here?
The controller didn’t even have a fraction of the hype that the steam machine has. The index at that time was extremely pricey when alternatives existed for a third of the price. The steam machine is hyped up as fuck, comparably cheap for a pre-built, and is one of the first systems coming with steam os out of the box. Your comparison sucks, and you sure as hell cannot beat bots when there is a serious scalping operation going on.
Personally I see LLMs as a tool like any other. You can use it to mass produce low quality slop, just as you can use it to help you produce a higher quality output.
You’re perfectly right about architecture and guardrails, that’s how it has always been with any other tool or piece of software. It depends on how you use it. Remember the no-code hype train? It’s literally the same, people have been shoving it into everything, no matter whether it made sense. It worked for some, and it made development costs explode for others.
Guardrails are especially important for LLMs because you do not have deterministic outputs and potentially exploding costs.
So analyze, measure, and think about where and how it makes sense to integrate, and build it incrementally, again, just like with any other piece of software. Start slow, keep humans in the loop, measure and analyze, and improve incrementally. When you achieve confidence, potentially start automating going into an agentic direction, when it makes sense and the risks have been considered, but always keep provenance. You do not want blind decisions by the magical AI box.
And just to repeat, because I’ve seen heads roll because of dumb decisions: keep cost under control and always have limits set, and always consider which data flows into the AI and what happens with it afterwards.
Producing a half a million bill in a month by accident or neglect or suddenly having your customer database queryable on a public model is a surefire way to drive the company or at least your career to the ground in seconds of wrong decisions.
Also, read into all the stuff built around LLMs, protocols like MCP, attacks and defenses on LLMs, get knowledge about the inner workings, experiment and learn. When you’re the head of AI, you’re supposed to be the person who knows. And when you know what it does, how it works, and how to use it, you’ll find actually good and appropriate use-cases naturally.
Since there’s zero information about what kind of company you’re working at, the following is extremely generalized.
Whatever you do, take security and data security especially into consideration first, not after:
The tech giant says the system only analyzes hand-movement points from a short video, does not record audio, and deletes the footage after verification.
It’s just a short video guys, there’s even no audio! And they pinky promise to delete it.
What a fucking shitshow that is. Like, honestly, I’m fine with regular captchas, even if they are the shitty ones. The newer (?) captchas that force you to do solve 5 bullshit “place this there” captchas are already reason enough for me to just leave the site. But if you force me to record a video of me throwing gang signs at the camera, probably several times again, because the movement was not correctly identified, I’m sure as fuck to never visit anything related to you ever again.
I also love the irony that Google fights people bots, while they are scraping the whole internet and investing into AI automation massively.
I fully agree with you: it’s NOT easy. And you must understand what you do. It’s not just deploy a container and run happy.
This is literally what you’ve called misinformation.
Again, not everyone is self-hosting only for learning and experimentation only. Making a deliberate call that mailing infra might be too hard might be too hard, have too big of a knowledge gap, or is simply not worth the effort is something I’d call more serious than hardlining on “self host everything or stay on gmail”, especially in the case of mailing, where it’s pretty much impossible to self-host on your own hardware / network.
Full instructions do not reduce any effort or resources involved or complexity of the problem. And the problem is that you’re suddenly moving from “I’m hosting a few services” to being balls deep in networking, dns, and a deceivingly easy protocol which blows up in complexity due to being federated and absolutely dominated by big providers at the same time, and all of the extensions for security.
Except for learning, self-hosting serves a purpose. You might want privacy, you might not want to be dependent on corpo infra or external services at all, you might want to host something that offers something more or better than a SaaS solution - but first of all, it needs to work. For mail, you gain none of those. Self-hosting on your own hardware (or rather network) is pretty much impossible, so you’re reliant on a hosting provider at least. There is basically zero difference in functionality between mailing servers or providers. Sure, you’ll run into problems when copy pasting instructions, but those problems will break the service. Fucking up your DNS or networking will break your whole server. At the same time, while failing silently it will costs a magnitude of effort more than most other usually self-hosted services.
Because it works for you, doesn’t mean it’s easy. If you have the experience, and done it at least once successfully, it’s “easy”. Compared to the average self-hosted configure and run a docker image and reverse proxy it’s objectively harder to run.
The issue is not running the individual components or servers, but that there’s infrastructure and to some extent crypto involved, which is just outside of the comfort zone for many. You tried to host it like any other thing on your homelab? Nope. Has your VPS been involved in spam? Enjoy the blacklist you’ll never find out about and the debugging why it doesn’t work. No experience in managing your DNS? Have fun getting DMARC/DKIM/SPF to work.
Theres just way more stuff that needs to be done, and a lot of it will fail silently.
Use at your own risk.
What an amazing conclusion, and the best part is, no matter what you’ve been waffling about before - it’s always right. Can we stop calling random things AI slop and telling to be careful bEcAuSe iTs Ai sLoP, and back to being cautious until something has been reviewed properly? Being careful with random stuff from GitHub you install and run in your private network?
Your whole comment may have been AI slop as well. “From a quick glance at the repo”, you should be careful! Thanks, Sherlock.
My interpretation was OP isn’t necessarily the target here, but a victim of some Windows hack spreading around their shared network. It’s possible the whole network was “worth” such attention.
Yeah, it might be that another system in the network was the initially compromised system, but I’m questioning whether Windows malware would be able to spread over wine to a unix machine to actually cause damage there. But that’s an attack vector I literally have zero idea about, just kinda seems suspicious.
And yeah, everything in OPs story is absolutely plausible, but it’s more of a gut feeling given the provided information that it just feels off. I might be fully in the wrong here, and they’re the unluckiest random person to ever have touched a unix machine, I don’t know. Definitely curious how this will develop though.
Something about this post is weird as fuck and some part of this story is missing for sure.
First of all, routine scans with ClamAV. Why are you routinely scanning your system, and what’s your expectation here? In most cases system compromise happens by executing something malicious or by exploiting something on your system, For the former, an active background scanner would help, but not a routine scan, and it’s easier to just not execute suspicious stuff. For the latter, your routine scanning is worthless.
Then the compromise over a WINE DLL seems something between borderline impossible on one hand, and like a very targeted and handcrafted attack on the other hand. Sure, wine is not a sandbox, but seeing this as the point of entry for a full blown persistent RAT is weirding me out massively.
Lastly, “them” setting up seemingly good persistence on your system, yet not hiding any indicators of compromise, and then nuking everything when they are seen. Why that effort? Either set yourself up for the long run and hide, or when detected just say “eh, whatever”. This also seems weird, since on one hand there’s indication for a professional, targeted attack, and other points sound more like rookie script kiddies.
Lastly, you. You seem like a pretty confident user while getting hit like that. It just feels off.
I’m not claiming you’re lying, and I couldn’t blame you for leaving information out because of opsec. But everything about this story feels off. I kinda assume that you’ve been actively targeted, and you should ask yourself why. What information or access do you have? How have you been pwned that “easily” and where did that DLL come from? How was it placed and executed?
The easiest way would be to set up caddy to use acme on the servers, and never care about certificates again. See https://caddyserver.com/docs/automatic-https.
If you insist on your centralized solution, which is perfectly fine imo, just place the certificates to a directory properly accessible to caddy, and make sure to keep the permissions minimal, so that the keys are only accessible by authorized users.
If the certificates are only for caddy, there’s no reason to mess around in system folders.
In all honesty, the constant rambling against any service provider when something goes wrong is tiring. as. fuck.
“I’m not using anything, I’m self-hosting everything and no cloudflare can take ME down!” - hot stuff buddy, let’s talk again when at some point you’ll have something interesting and get hugged to death. Or when something of your diy self hosted stack breaks or gets taken down by an attack.
“I’m not using (big company name) but (small startup name), and I’m not having any issues!” - wow, great, obviously the goal of the company is to stay as small as they are and supply your service. Let’s talk again too, when at some point your friendly startup gets sold, or grows more. Oh btw, smaller company usually also means less resources.
“That’s all because they are using centralized services, we need to federate everything to not have a single point of failure” - federation alone won’t help if the centralized service has several magnitudes of resources more. Any single cloudflare exit node can probably handle several times the load of the fediverse. We’ve seen lemmy instances go down all the same, and this will happen with any infrastructure.
I’m not supporting big companies having that much market share and the amount of control over the Internet as a whole that they have. But, have at least some respect from a technical standpoint for the things they’ve built. I’d say way over 80% here haven’t seen infrastructure, traffic and software on a scale that’s even remotely close to the big players, but are waffling about how this or that is better and how those problems should be solved and handled. Sit the fuck down.
No matter how well reasoned, allegedly fit for purpose or how much something pretends to be it, we shouldn’t be trusting those promises, especially not from people we don’t know. That does not end well neither for the free candy van nor for cybersecurity. Trust like that has been responsible for a lot of attacks over varying vectors and for projects going wrong.
On the other hand, detrimental reliance is a tort and if someone is relying on an app for a specific safety function, the app could be civilly liable if it fails it’s function in some way.
Yes, if the app would be any kind of official tool.
Imagine if you had this attitude about an insulin use tracker/calculator, that sometimes gave wildly wrong insulin dose numbers.
Yes, and that’s why regulations for those kinds of things exist, that prevent those things. There is no regulation for the ice tracker.
Maybe down the road, it’s decided that aiding and abetting ICE is a crime, and providing misinformation intentionally or unintentionally is a criminal act. App developer dude could be criminally liable if he knew or ought to have known he had vulnerabilities. You know, in your New Nuremberg trials that you are going to get sometime in the next decade or so.
If down the road a regulation would happen for, app developer dude would be forced to either comply or to stop operations.
So fucking what? He is not being paid in any kind, and anything he does on that project is volunteer work. If he was not able to do anything on that project due to regular work, vacation, personal issues, or the simple fact that he didn’t want to?
If you don’t pay for a service, you don’t get to decide what people do, deal with it
Honestly, apart from the report being potentially wrong, the researcher seems pretty entitled as well. Like good intentions and all that, but he’s given him a week to fix the issue, usual practice in responsible disclosure are 90 days. We’re not talking about a company here, it’s some single random dude providing the app.
This really sounds like some personal issue written down for public drama, while making himself ridiculous for not knowing his own shit properly.
Fomenting chaos, one fridge at a time. Because in year 4 of a full scale war there are no more valuable targets than fridges. There’s nothing else that might contribute to issues like that, like unnecessary internet-connected devices in domains without any IT knowledge, an influx of “hacking” incidents due to LLMs getting better at cyber at an alarming rate, and somehow still the most basic security not getting implemented.