On LLM Writing

I believe the current way we use AI for text correction is fundamentally flawed. Instead of attempting to predict my intended message, allow me to express myself in a disorganized manner and then make minimal edits to ensure coherence.

Try it out here. You need to put in your own OpenAI API Key. The source is available here.

This post was edited with Take1 by copy and pasting paragraphs into and from the app.

Lately when I'm writing notes, I've been trying not to go backwards, that is to minimally revise what I write. Too much time is spent editing as I go, and it defeats the entire purpose of getting my thoughts on paper.

Spelling and grammar mistakes accumulate as I go, even while I write this. It happens. What I want is to write without restraint, to make absurdly bad spelling mistakes for an entire sentence, and the "editor" automatically knows what I wanted to write.

Enter LLMs.

"Sftr enginrng ivlvs dsigng, devlpg, nd tstng sftwr"

->

"Software engineering involves designing, developing, and testing software"

This is a pathological case that should be handled. Autocorrect, such as that available in the Notes app or in iMessage, is not sufficient for this task. Correcting the above sentence requires examining it in its entirety.

LLMs are perfectly suited for this task:

Chat, however, is not the ideal workflow for note editing. It should happen on the fly, as you type. Ideally, I could make a plugin for the Notes app or edit the notes manually via its SQLite database. The former is not possible and while the latter might be, I'd have to decode how the data is stored. Apple will probably add this within 1-2 years.

Proof of concept:

• Minimal app or website.
• Disallows deletion except excess whitespace.
• Edits previously written text using gpt-3.5-turbo.
• Handles basic text without markdown.

The newest GPT4 model would be better but the idea is to do this for a cheaply as possible. According to the rough estimate at 1 token every 3-4 characters the cost of editing 5MB of text with turbo would be $1.50, around 14 books. Or if you're writing 500 words a day, editing over the course of a year would cost $0.30.

So this is exactly what Take1 is. It follows a similar UI to Notes but far more minimal. It saves notes in LocalStorage, along with settings, such as the API Key. If the LLM editor is enabled and an API Key is present, it will automatically edit the text you've written. It works pretty damn well, there are some caveats but nothing that can't be sufficiently addressed by some pre/post text processing.

Parts where the LLM struggles

I found that the LLM works very well when presented "whole thoughts", such as sentences and paragraphs. When I let it edit any selection it saw fit, except for un-editable text, it would encounter several issues.

These issues could be managed with additional pre/post text processing, but it felt quite fragile. However, when I presented the LLM with a complete thought to edit, it performed much better. So, I started sending only whole thoughts and suddenly the need for hacks disappeared. There is still some text processing involved to determine whether an additional space or newline is needed, as well as capitalization.

In my experience working with LLMs (this project and others), the initial output you receive is not always the final result you want, but it is usually around 70% to 90% there. It is worth considering if the input is optimal and experimenting with different inputs. It is unlikely that you will use the optimal input right from the start. Before resorting to additional code to process the output, ask yourself if constraining the input can make the output more consistent.

So where do I see this going? Well I believe the PoC was successful but I don't want another writing app. I want this in my current writing app. Maybe I'll delve deeper into finagling this into the Notes app.