In Which I Commission A Novel

The following text was written solely by Michiel de Mare, without assistance by artificial intelligence.[1]

It used to be a truth rarely denied that it is cheaper to buy a novel than to commission one. In the past year, this truth has quietly ceased to be true.[2]

I am not customarily engaged with the procurement of literature, or such things [3], but last Wednesday, I commissioned a novel. Considering the magnitude of the assignment, I confess I was rather lax in supplying instructions, hurriedly jotting down a few words on its subject and theme, eager to get the project underway. On this flimsy foundation, my team of mercenary writers were to build a fictional world, consisting of 12 chapters of circa 6000 words each. As the first drafts came in, I experienced a pang of regret at not having supplied more detailed instructions. These 69 words launched three thousand paragraphs:

In 1817, Jane Austen is dying from Addison's Disease. However, back in the year 2032, an international team of physicists and book lovers have found a way to save her. They have built a time machine that will transport Jane Austen to the present, where she can be cured by modern medicine. However, they cannot send her back. How will a 19th century novelist fare in the 21st century?

You may be surprised to learn that I originally did not plan to read this novel. Commissioning a novel without intending to read it may seem like the absolute summum of extravagance, but that was before software companies started buying second-hand nuclear reactors. I ventured upon the idea purely as an experiment, as I am investigating how to persuade Large Language Models (LLMs) to create a project that is much larger than what they are able to handle in a single session. These agents have a limited amount of information that they can fit inside their memory. There is no strict limit, but as the amount that agents need to hold in memory increases (which includes all text that they have produced in the current session, but also their "stream of consciousness"), speed goes down and prices go up. So we must limit the length of the sessions, and provide continuity by taking copious notes and consulting them later. In future sessions, the agent only has its notes to rely on, as if it were a forgetful psychoanalyst such as Irvin Yalom, with no recollection of the previous sessions whatsoever. [4]

And because forgetful psychoanalysts cannot read all the notes of the previous sessions in the fifteen minutes they have between two clients, it is not sufficient merely to take notes. A process is required. Not only does Irvin Yalom need to be told which notes to read, but also that he has taken notes in the past, and even that he is a psychoanalyst, and not a plumber. This is the responsibility of the process: to tell the agent what to read, what to do, who they are. Imagine a single novel being written by different 400 writers, each partly writing paragraphs, partly taking notes on the novel's architecture or a character's psychological development, and after two or three hours, all of their papers get wrapped up in a parcel and sent to the next writer, with a brief cover note about the current state of the project. That is how this novel has been produced.

For quite a while, I entertained the idea of designing a novel-writing process: when to write, when to review, when to make notes, what kind of notes to make... Novels need a plot, characters, locations, timelines, themes, a style... Some have intricate narrated structures, with nested narrators, unreliable or not. Some novels have laws of physics that differ from ours; others contain magic. I decided that creating a process capable of dealing with all these aspects would be too tightly coupled to a specific type of novel. What I was looking for was a process that was much more general than that. I wanted a process that could produce any big digital textual artifact: a novel, an epic poem on the Minneapolean Resistance, a Klingon rhyming dictionary, Bilbo Baggins' collected tax returns, Lego building instructions for St. Paul's Cathedral, and of course all kinds of software projects.

Since you can solve any software engineering problem by adding a layer of indirection, except for having too many layers of indirection (a truth that should be more widely known than it is) I decided to add a layer of indirection. I designed a process that designed a process: I created a project folder with a simple computer program. This program produced a prompt (a textual instruction) for an agent. I sent the prompt to an agent, with permissions to edit the project folder - including the program which produced the prompt! After the agent had finished its session, I stored the changes it made. Then I planned to repeat this process as many times as needed to write all twelve chapters.

The original program alternated between telling the agent to write a part of a chapter, to review the previous changes, and to improve the novel-producing process. I always pointed it to my shamefully terse 69 word synopsis. The first time I ran the process, the agents quickly proceeded to turn the program into an impenetrable mess, which, after a dozen iterations, failed with an error. When the program fails once, it keeps failing, since there is no opportunity for the agents to correct the error, and the process came to a complete halt.

Wary of falling into the trap of becoming the agents' helicopter parent, I changed the prompts in the original program to exhort the agent to "Follow good software engineering principles". That worked surprisingly well for the first 285 iterations. The agents, Claude, Cody and Gene, went to work with serious zeal. The original program grew from 90 lines (mostly the prompts) to over a thousand, with a series of automated tests. Documents with subjects like "themes", "timeline", "world building", "plot holes", "anachronisms" had been produced. Meticulous evaluations were being performed periodically, in which concepts are mentioned that I do not even begin to grasp, such as:

Add strict-mode quality gates (env-flagged) for blocking `:content` when unresolved metadata issues exceed threshold or when chapter plan beats are missing.

At iteration 285, with the novel halfway produced, an agent finally made a change which broke the program. The process halted, and I had to step in, much later than I had anticipated. I added a step to the supervising program, the one which sends the prompts to the agent, specifying that if the program producing the prompt returns with an error, it should instead give the agent a prompt with the error output, the last change to the program, and a friendly admonition to fix the error. This proved sufficient to fix the faulty change and continue the writing process until the agents considered the novel as finished.

I had been wondering what would happen after all the chapters of the novel had been completed. Would the process enter a never-ending editing phase, rewriting the entire novel multiple times until I would have to put a stop to it? Or would it look upon its novel, consider it mostly perfect, maybe altering a word here and there? Would it manage to make edits that improved the novel? As it turned out, it did not make any edits at all. It performed a number of evalations, added a step which generated the entire novel as a single file, but it never edited a single chapter. All section and chapters have been written sequentially. No earlier section has ever been revisited by an agent, reinventing the serial novel, popular in the 19th century and used by Charles Dickens in the Pickwick Papers for example..

Did I read my novel? Yes, dear reader, I read it all. If not I, who? Is it any good? To be blunt, no. But it's not the worst novel I've ever read. And writing a novel, even a bad one, is quite an achievement. For it is a novel: It has a structure, a plot, protagonists, some drama, some sharp observations, some witty dialogue, an ending. It occassionally managed to make me chuckle. It is at its weakest when it puts tired sociological clichés into Jane Austen's mouth. The emotional stuntedness of all protagonists is also off-putting. And quoting at length from a 21th century Jane Austen novel is much like a Mozart biopic having the composer create another symphony and performing it. Authors should know their weaknesses.

I resisted the temptation to send it back to the agents for editing, since I intended this as an experiment to determine what agents could produce without any human guidance whatsoever. Also, I have no desire whatsoever to read it again, even if greatly improved. I am going to commission another novel though, this time with extensive instructions, including regarding the editing process afterwards. I might even let it write a preface!

Dear reader, you do not wish to read this novel, but to satisfy your curiosity, here it is anyway. And here is the repository where you can see the agents at work. I would caution you against considering the faults of this novel as faults of Large Language Models in general. For one, the models I used have already been superceded Nor have I expended any effort into evaluating the different capabilities of the three different LLM providers that I used. My hunch is that we have not yet taught LLMs the difference between great sentences and passable ones. Maybe the way we perform training is not compatible with this skill. But it is certainly much too early to draw such a conclusion.

[1] We increasingly need some kind of pledge like this. The benefit of the doubt, that anything you read online has been written by a real person, is evaporating fast. For now, this will do, but if it catches on, LLMs will start generating similar pledges. (And of course, one can still lie.)

[2] You should read Pride and Prejudice, by Jane Austen, if you haven't already.

[3] You should also read Max Havelaar, by Multatuli, which you almost certainly haven't yet.

[4] In Yalom's last book, Hour of the Heart, he describes giving one-time therapy sessions as he is nearing 90.