They want to make people aware of the risks and build a case for stopping the attempt to build ASI.
Yudkowsky was a researcher trying to accelerate AI research in the early 2000s. He realised the risks and switched to how to identify and manage the risks and develop ‘friendly AI’. He now believes the risks are not manageable with our level of understanding and we won’t be able to make it friendly so we should, as a whole, just stop.
It seems they wrote the book to make that case.
But it is inevitable and they seem to know that, given what you’ve said about their book. That’s my point and what puzzles me. You can mess with evolution - in fact messing with it is another aspect of it - but you can’t stop it.
The thing that may well prevent our developing super AI is our self destruction coming first, not us developing the will to even try not building it. Just look at how we’re handling climate change.
I’ve not looked at Autonomi 2 code, but as Claude is a contributor to the repos, perhaps a developer can comment on whether the code is a badly written as Claudr’s own self written code:
Any code written by Claude is likely to only be maintainable using Claude (if at all). 
Do folk see where this is going yet?
Over the past couple of days on Mastodon I had a sensible and civil debate with Simon, a FOSS author who is using LLMs to add features to his popular hledger plain text accounting programme. He’s open about the dilemma developers have here and for now feels it’s justified. He’s explained his reasoning in an AI Policy on which I commented. You can read our short discussion here.
He’s a thoughtful intelligent chap and I’m pleased we had a discussion which I think educated us both on the questions surrounding the use of this tech.
I believe we agreed on a few things:
I see so many problems with massive effects far outside the immediate context of any application of this tech that it seems inconceivable that someone can genuinely establish an overall balance of good from its application. I look around, follow research, explore etc, and everything I see backs this up.
My case can be summed up in the maxim, first do no harm which is the opposite of move fast and break things. The latter something David once rejected in favour of move slow and fix things.
That puts the onus on those using and advovating for LLMs to make the case and to address the criticisms made against it. I think that should always be the way, but we see it is often not so, and can look at history to see the cost of “let’s wait and see if it’s safe”, and of assuming there are no serious issues until they become inescapable and the damage from them immense.
I’m not alone in this conclusion, though we each reach this position in our own way. I believe those using and supporting this tech are not honest about this, with themselves most often.
Unlike Simon, most don’t engage with the dilemma. Instead they say things like, it’s here to stay as if that’s an argument for causing harm by supporting and using it. They focus on short term localised gains and don’t think about or consider the hard questions and the inconvenient truth.
This is climate change all over again, but harder to push back against probably. People are ripe for the taking and are being taken with such ease.
Tldr:
Another pithy take:
They haven’t totally given up though and think there is still time to persuade people to change course. Hence the book. They do admit it is unlikely any party will just stop, much less all at the same time.
One of the points they make is continuing to develop AI with the goal of eventually achieving ASI is like climbing a ladder where each rung you can reach gets you 5 times as much money but the final rung means everyone dies. Even if you know or just strongly suspect that eventually you will end up on a rung that blows up the world the temptation is to keep climbing. Because someone else is going to keep climbing anyway and collect the money until then.
Another point they make is that all the people trying think that they are cleverer than the others and will be able to avoid the pitfalls and keep their ASI under control. Believing that means it would be irresponsible for them to not continue because otherwise someone else will get there and mess it up and their ASI will kill everyone.
It’s not about giving up but realising that for all that effort you can only hope to delay things a bit. But as the remainder of your reply shows, even that is futile.
So I remain puzzled.
I take comfort from not believing all the bullshit about super intelligence being imminent or close.
That seems much more likely to be just another part of the hype which has been deployed to get trillions in investment poured into this superficially impressive mediocre technology. And if I’m wrong, so what, there was nothing I could have done to change this.
Given that David is using several different models for planning, coding and testing, each checking the others work, I think this can simply be filed under snark, Mark.
Not at all. It’s a fair observation.
You’re assuming that David is able to use these models more effectively than Anthropic themselves. That’s possible but not reasonable to assume. Meanwhile I hear he’s talking them up in some pretty bonkers language.
This is what I see from LLM fans all the time. Then occasionally I find someone like Simon (see above) who’s looking seriously at the issues and willing to have a discussion about them without making personal remarks like yours.
There are plenty of people who are smarter and deeper into this than me who share my analysis.
Also LLMs will have no problem hacking LLMs.
After the leak of Claude’s source code, which was written using Claude, my feed has threads from folk who have tried to understand it, and how it does things. It’s both unfathomable and includes numerous terrible ways of doing things.
After commenting about this yesterday I took a brief look at the Autonomi 2 code and am happy to say the few files I looked at appear completely normal.
I don’t know how the developers are using Claude, and I’d be interested to know, but I suspect it’s more as a simple suggester/adjuster rather than generating whole feature implementations. And that they are checking every line before a commit.
I’ve heard the story about David using multiple agents to implement large feature sets based on a highly LLM assisted and human crafted specification, so maybe that is in fact what I’m seeing. But given how poor Claude’s output is said to be, I think it more likely they’re using it in a much more focussed bit by bit way, as I’ve just described.
I’d be very interested @chriso as one of your joint commits with Claude was something I happened to look at.
That said I only looked at a handful of files so other areas of the code may be very different.
I understand the plan is to assert Autonomi’s copyright, while in the USA and (probably soon also in the EU, and I expect UK) any auto generated code is not copyrightable. And unless this is clearly identified in the codebase, it’s inclusion makes any human written code be treated the same way, and not copyrightable.
So maybe there’s less Claude generated code there than we think after all?
If you can spare the time, please dig deeper. You do us all a favour by either validating Davids work or by exposing flaws. You have contributed a lot, far more than just about anyone on this sadly decaying forum. It would be great if you could continue by running a skeptical eye over even more of the code.
Yes - the guy is a very hard-working genius 1% inspiration 99% perspiration or something like that. Given what we know of him over the past couple of decades, I think that is a much more reasonable assumption than at first glance
“I hear” — sources?
“bonkers language” a bit pejorative, eh ?
So was “Snark, Mark” but I liked the way it sounded and its true, so there, Ya boo sucks as youse English say 
The Claude code on Github has been taken down according to @aatonnomicc
But not before I (and likely thousands of others) forked it for ourselves.
Fill yer boots
If you do not have a local copy then eventually it will be gone
Thanks @neo
git clone https://github.com/willief/claude-code.git
No necessarily. There should soon be a network, a kind of autonomous cloud service, that’s outside of their control, and has a “once uploaded, always there” property… But you may not have heard about it yet?
My remark was for Southside’s Git repo
America innovate 
Europe regulate 
China 
ate (innovating & regulating so hard that it scares others) 
Johnny has been picking his way through the Claude source code and sharing his analysis in a mega Mastodon thread. Now his ideas are forming and we can expect some revealing blog posts.
Here’s a taste, and all of it makes total sense when you consider what Claude is (and isn’t). It’s exposing the fallacy of believing the appearance of something like human thought, but patently is very different because one thing nature doesn’t do is waste. And yes, Claude is a collection of agents all correcting reach other’s mistakes in an exponential explosion of waste, and humans hard coding the worst problems away.
Here’s some food for thought for LLM believers, and his mega thread is worth dipping into too:
A snippet from Johnny… oops: