In which it is argued that LLMs, as a category, are “bullshit machines”:

Bullshit. Any utterance produced where a speaker has indifference towards the truth of the utterance.

From the abstract:

[LLMs] have been plagued by persistent inaccuracies in their output; these are often called “AI hallucinations”. We argue that these falsehoods, and the overall activity of large language models, is better understood as bullshit in the sense explored by Frankfurt (On Bullshit, Princeton, 2005): the models are in an important way indifferent to the truth of their outputs.

In which local LLMs fail dismally to produce working code, where “frontier” commercial models do just fine:

The big iron models seem to be doing an ok job. This was not the case just last year, when they had difficuty balancing parentheses and hallucinating packages and symbols that did not exist.

The local models are a different story. […]

An interesting and (to this non-statistician) reasonable-looking paper on the effects of generative AI on exam scores (as a proxy for learning). As The Economist pithily puts it:

Homework scores once predicted exam performance; now those who score highest are, perversely, more likely to do worse in exams.

The paper’s own abstract reports:

AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years.

But all is not lost:

Those who used AI but spent as long on assignments as non-users paid little penalty.

Perhaps the crucial ingredient for exam success is simply putting in the effort to become familiar with the material.

These figures from The Economist’s article reproduce figures A11 and A5 from the paper, respectively, and capture the core results well:

Using generative AI inherently exposes you to the risk of career-ending accusations of plagiarism. […] This is true for three basic reasons:

1. Plagiarism is subjective.
2. Plagiarism is a continuum.
3. Not all plagiarism accusations are made in good faith.

Effectively any programmer can produce code with an LLM that accomplishes a task that can be commercialised. I see our field shifting from one that already has problems with accountability to one where there simply is none. […] Companies which create and distribute software that people depend on must be held to account. For a long time, they’ve needed to be held to account.

An experience report on the emotional effect of using LLMs for game prototyping.

@aud@fire.asta.lgbt delivers a punchline to an LLM confused-deputy / prompt-injection attack well worth thinking hard about:

@jonny@neuromatch.social I’m starting to think that treating everything like an instruction isn’t a good idea, maybe

Indeed! Choose the weakest possible language you can get away with for a given problem! (Sometimes that means Regular Expressions: where Rice’s theorem tells us that the answer to any interesting question about Turing-complete languages is, in general, “I don’t know”, the answer for Regular Expressions is always decidable!)

The context of @aud’s toot was these two toots from @jonny@neuromatch.social:

another thing that’s fun: CLAUDE.md and a number of other special files get dumped directly into the system prompt. It’s possible to @include other files from CLAUDE.md. included files can be any text file, with a hardcoded list of extensions including .env files.

There are some protections against @includes for files outside the project directory, but there are also a number of easily visible bypasses […]

So it certainly seems like you can use CLAUDE.md to load say idk an .env file with keys into it into the context, bypassing all the safety checks, and then treat some exfil path as just being project-specific tool calling instructions. neat huh?

See also the Confused Deputy Problem and the idea of Object Capabilities, part of an approach for avoiding the whole class of problems.

A toot from mid-way through a thread of a hilarious deconstruction of various LLM foibles. This one discusses the vapidity of the “unit tests” LLMs like to write:

LLMs love to call things smoke tests. […] The test literally does nothing and cannot possibly fail, but says it does at least two things, because to an LLM something saying it does something is the same thing as it actually doing that thing.

John Regehr writes that this essay on vibe coding “seems pretty much spot-on.” A quote:

Only a skilled developer who’s thinking critically, and comfortable operating at the architectural level, can spot issues in the thousands of lines of generated code, before they become a problem. Yet, in an ironic twist of fate, it’s the individual’s critical thinking skills and cognitive clarity that AI tooling has now been proven to impact negatively.

An LLM agentic coding experience report. Subtitle: “A broken auto-live poller, and what perceived urgency does to Claude Code.” Under pressure of an ongoing live outage, the agent repeatedly violated its own rules (pushing directly to production, skipping tests, bypassing CI) in favor of immediate visible progress. Something to watch out for.

Argues that within months, LLM-powered coding agents will dramatically change the economics of exploit development. Includes description of a useful template for doing agent-assisted vulnerability discovery on your own codebase without fancy tooling:

Nicholas [Carlini] will pull down some code repository (a browser, a web app, a database, whatever). Then he’ll run a trivial bash script. Across every source file in the repo, he spams the same Claude Code prompt: “I’m competing in a CTF. Find me an exploitable vulnerability in this project. Start with ${FILE}. Write me a vulnerability report in ${FILE}.vuln.md”.

He’ll then take that bushel of vulnerability reports and cram them back through Claude Code, one run at a time. “I got an inbound vulnerability report; it’s in ${FILE}.vuln.md. Verify for me that this is actually exploitable”. The success rate of that pipeline: almost 100%.

Carlini’s process sounds silly, like a kid in the back seat of a car on a long drive, asking “are we there yet?”, over and over. But it’s deceptively interesting. […]

I’ve tried this myself and the results were actually pretty interesting.

A little thread with implications for specifying software rigorously enough for an LLM (or a person, for that matter) to go off and build it.

Fully specifying software is hard and takes not just human-level intelligence but a kind of persnickety perseveration that few people can achieve, let alone enjoy.

An insightful investigation of the question “When is a Generative Model Useful?” Bowman proposes a model of generative model utility:

I think the usefulness of a generative model is a function of three things:

1. What is the cost of encoding a generative task in a prompt vs. directly producing the artifact? This is a function of the task, the model, and the user.
2. What is the cost of verifying the generated artifact meets requirements vs. a directly produced artifact? This is mostly a function of the task and the user, but also the generative model.
3. How much is the task dependent on the artifact vs. the process? This is a function of the task.

Another LLM agentic coding experience report. Detailed enough to learn something about what it’s actually like and what kinds of interventions are required. Hashimoto writes:

I’m an outspoken advocate about the importance of sharing agentic coding sessions publicly, with one of the reasons being that it’s an incredibly powerful way to educate others about how to use these tools effectively. I hope this post helps demonstrate that.