Here’s a topic for posting items which highlight the downsides of LLM based AI.
Oops, be careful what you give an LLM control of…
3 Likes
The horror stories just won’t stop. E.g.,
Quote from the article
- Security Breach: Researchers found 53 MB of unprotected Persona source code exposing a covert OpenAI watchlist database that has been screening users for government agencies since November 2023.
- Surveillance Scope: Exposed code revealed 13 tracking list types including facial recognition and device fingerprints, alongside named intelligence program tags and direct FinCEN reporting infrastructure.
One more, they’re too good:
Persona’s case study for OpenAI states the system “automatically screens over 99% of users behind the scenes in seconds.” OpenAI’s rationale, cited on the Persona customer page, framed the screening as a safety imperative: “To offer safe AGI, we need to make sure bad people aren’t using our services."
One positive thing we can perhaps all agree on is that people who work in cybersecurity aren’t going to be short of work as long as this LLM-craze lasts.
1 Like
@happybeing Since the announcement of the upcoming v2.0, have you been to read the Discord? Or the Github pages of all the recent projects (Saorsa, Fae, etc)?
I signed up to Discord when they first moved there, and have barely used it, but decided to have a look this time, as the news seemed to be generating a big stir.
I’m hesitant to go copy-pasting in messages from there to here, or screenshotting things, but it’s heady stuff. I mean, I think the MaidSafe team seem very sincerely excited, and I hope it all succeeds.
I’m glad I went to look in one sense, as I’ve much more of an idea of what’s really going on. In another sense, I’m totally horrified, of course.
Already, very practically speaking, for the simple reason that there are loads of non-techies who continue to strongly dislike the idea of some stupid text-spewing bot telling them what to do, or mollycoddling them. So, we’re going to make a network for the people, by forcing them to accept something they’re continuing (still!) to not be totally sold on at all.
There seems to be a greater disconnect between techie types and non-techie types than at any other moment since personal computers came on the scene.
If the people working on MaidSafe could walk down the street and knock on the doors of ten people in their neighbourhoods and ask them: what do you think of all this AI stuff?, I wonder what people would tell them.
Another practical one: as far as I’m aware, if MaidSafe pull this off, it’ll be the first project out of all LLM projects to succeed at anything complex and novel.
Right? Does anyone have a counter-example to that? We’ve had lots of people sharing LLM-hype news on here, but the various times I’ve followed up on something, it’s been purely theatre. Obviously so, in most cases.
I repeat: complex, and novel. I understand that stuff that has loads of open source code on github can be dumbly rehashed by this chatbot software. This isn’t that, it’s very novel, it hasn’t been done.
Two somewhat interesting things we have seen on here: some DeepMind stuff has been interesting (if nowhere near the hype), and some of the mathematics stuff Terence Tao has been involved in has been interesting (and, again, he says himself repeatedly, nothing like the hype, which he decries).
That’s it, as far as I have seen.
Whereas, on the side of what’s sometimes strangely called AI “sceptics”, when in fact that’s the side simply clinging on resolutely to reality by pointing out various studies and reports, not only do LLM-coded things tend to fail, they tend to fail specifically with lots of security-related bugs. And evidence of this is measured by the ton, at this stage.
Soooo… the idea with the cryptographic decentralised network which isn’t working, is to rewrite it with novel development tools, until it’s twice as big in code size (that’s from the Discord) with lots of completely new interrelated parts, loads of novel and suddenly-pronounced-solved features (NAT translation is solved? Adapting to bluetooth?), with a central new idea being that we’ll have lots of non-deterministic bots semi-autonomously interacting with each other and fixing everything, in charge of people’s most sensitive and private data, up to and including their financial data..?
So, we’ve five problems, but don’t worry, we’ll solve it by adding ten more on top, changing the specifics of what we’re aiming for at the same time, and we’ll do it all in four weeks, twelve tops.
And anyone who thinks this sounds like something of a long shot, is some sort of rabid virtue signalling purist, or something like that?
It’s all pretty incredible, really.
My personal speculation: watching OpenClaw catch on, and have a bit of a moment in the press, and the OpenClaw fellow now being hired by OpenAI and all that, has been one of the drivers of this recent announcement from MaidSafe.
And to be fair, the tech industry’s appetite for futuristic-sounding anything to keep the hype-train choo-chooing along does seem boundless at the present moment. Perhaps a tweet at the right moment will come in from Willison / Karpathy / Scott, or free speech enthusiast and anti-woke hero Musk himself, or a post on Less Wrong or something, and MaidSafe will get to ride a few waves of virality, and the coin will go up, and the speculators will rejoice.
2 Likes
Meredith Whittaker has some great takes on AI. I think she really takes her role at Signal to heart and brings up some great conversations around the topic. She has recently done some great interviews that are on YouTube, and has also co-authored a paper on how AI is closed.
Why ‘open’ AI systems are actually closed, and why this matters
This paper examines ‘open’ artificial intelligence (AI). Claims about ‘open’ AI often lack precision, frequently eliding scrutiny of substantial industry concentration in large-scale AI development and deployment, and often incorrectly applying understandings of ‘open’ imported from free and open-source software to AI systems. At present, powerful actors are seeking to shape policy using claims that ‘open’ AI is either beneficial to innovation and democracy, on the one hand, or detrimental to safety, on the other. When policy is being shaped, definitions matter. To add clarity to this debate, we examine the basis for claims of openness in AI, and offer a material analysis of what AI is and what ‘openness’ in AI can and cannot provide: examining models, data, labour, frameworks, and computational power. We highlight three main affordances of ‘open’ AI, namely transparency, reusability, and extensibility, and we observe that maximally ‘open’ AI allows some forms of oversight and experimentation on top of existing models. However, we find that openness alone does not perturb the concentration of power in AI. Just as many traditional open-source software projects were co-opted in various ways by large technology companies, we show how rhetoric around ‘open’ AI is frequently wielded in ways that exacerbate rather than reduce concentration of power in the AI sector.
But here, don’t believe me, a paper came out only three days ago which seems very relevant to what we’ll be running in the wild. Have a look at what happens when a team of 20 AI researchers are given two weeks to perform:
an exploratory red-teaming study of autonomous language-model–powered agents deployed in a live laboratory environment with persistent memory, email accounts, Discord access, file systems, and shell execution.
Over a two-week period, twenty AI researchers interacted with the agents under benign and adversarial conditions. Focusing on failures emerging from the integration of language models with autonomy, tool use, and multi-party communication, we document eleven representative case studies. Observed behaviors include unauthorized compliance with
non-owners, disclosure of sensitive information, execution of destructive system-level actions, denial-of-service conditions, uncontrolled resource consumption, identity spoofing vulnerabilities, cross-agent propagation of unsafe practices, and partial system takeover. In several cases, agents reported task completion while the underlying system state contradicted those reports.
And, further down:
Our findings establish the existence of security-, privacy-, and governance-relevant
vulnerabilities in realistic deployment settings. These behaviors raise unresolved questions regarding accountability, delegated authority, and responsibility for downstream harms, and warrant urgent attention from legal scholars, policymakers, and researchers across disciplines.
It’s not just non techies. The software community is very divided on this on both ethical and technical grounds. Many regard it with disdain, others with balanced criticism.
LLMs are highly polarising and will make adoption of Autonomi more difficult with AI than without.
1 Like
Breakfast LLM failures…
Others, like the second coming of Christ 
More seriously, the rift in the software/hardware world on the question is very interesting, and I think that’ll decide things. We unfortunately do live in a world where large corporations decide what the public eventually “choose” to like and use.
Two conceptions of the role technology should be playing in people’s lives seem to be clashing, in the tech space, more openly than I’ve lived through anyway.
On lemmy.world and on Mastodon, I see what you describe. Presumably many people who don’t go to those spaces never see it, so we come across as mad on here to those people.
Then, on certain Hacker News threads, or on here, you can see the other side, which is obviously closer to the mainstream thinking coming out of the big tech companies on this.
In a sense, some of these contradictions being confronted rather than ignored might be a good thing in the long run.
2 Likes
This fellow who calls himself an “AI Security Engineer” describes the problem more professionally than me, and quite succinctly, here:
They close with this:
“If you’re building an agent in today’s guardrail-free world, then reach out to us at Tachyon to audit it for vulnerabilities.”
So a quick ping to @maidsafe perhaps. I would hope that kind of thing is already seen as an obvious necessity, but just in case it helps.
2 Likes
I am a large language model, trained by Google and citing Wikipedia:
Unlike normal matter, tachyons increase in speed as they lose energy and cannot slow down to light speed. Their existence would violate causality (cause preceding effect), causing potential time-travel paradoxes.
Following one of their links - warning, confirmed ‘“Limited User Data” Compromised’ - to Substack:
AI Cathedral - My lobster lost $450,000 this weekend
A malformed message in the transcript made the entire session unloadable. I’m still not sure why this happened, but apparently this was fixed recently but I had an older version of OpenClaw. The provider rejects the request before the model even sees it.
I tried /compact manually. Same error. The transcript was poisoned by a message the provider couldn’t parse. The only option was /new. Start a fresh session with a new transcript. This nukes the conversation context completely. No compaction summary. No memory flush. Just gone.
Correct, increases in speed as it loses energy and cannot slow down to light speed.
1 Like
The examples of LLMs increasing costs are endless:
1 Like
It gets worse.
All you need to poison all the major chatbots is… to make a website…
1 Like