Ben Roy explains why prompting an LLM to write an essay misunderstands what writing actually is:
People fundamentally can’t prompt good essays into existence because writing is not a top-down exercise of applying knowledge you have upfront and asking an LLM to create something. AI agents also can’t create good essays for the same reason. Even though their step-by-step reasoning is more complex and iterative than human prompting, a chain of thought is still trying to accomplish a predefined goal. By contrast, real writing is bottom up. You don’t know what you want to say in advance. It’s a process of discovery where you start with a set of half-baked ideas and work with them in non-linear ways to find out what you really think.
I will continue to argue that for general business writing LLMs are fantastic if they are given the right context and guidance, and that it can save hours of work (with high quality results). But all my experiments with using LLMs for creative writing has so far fallen flat. Maybe—likely?—that will change within the next few months. But for now, the brain work this kind of writing requires remains. Not a bad thing imo.