The magic words are fan fiction: what actually improves AI output

In early 2024, two researchers at VMware, Rick Battle and Teja Gollapudi, ran a tidy experiment on grade-school math problems. They wrote 60 combinations of “positive thinking” prompt snippets, the kind of encouraging preamble people swear by, and tested them across three open-source models. Then they handed the job to an automatic prompt optimizer and let it search for whatever actually scored highest. For Llama2-70B, the winning system message began: “Command, we need you to plot a course through this turbulence and locate the source of the anomaly.” It required every answer to open with “Captain’s Log, Stardate [insert date here].”

“Surprisingly, it appears that the model’s proficiency in mathematical reasoning can be enhanced by the expression of an affinity for Star Trek,” the authors wrote, presumably with a straight face.

No human would have composed that prompt. That is the useful lesson.

The magic-words theory of prompting holds that some secret phrase unlocks the good outputs, and it is true only in the sense that such phrases exist, are found by machines rather than written by gurus, are absurd, and don’t transfer between models or tasks. Meanwhile the things that reliably improve output are documented and boring. Free, too.

This is the first entry in Prompt Lab, and the house rule is simple: prompting claims get treated like any other tech claim. Sourced, or cut. Everything below links to a paper or a vendor’s own documentation, and everything is reproducible in a chat window.

The expert costume doesn’t work

Start with the most durable piece of folklore: “Act as a world-class expert.” In 2023, a University of Michigan-led team tested 162 personas, social roles and domain experts both, across four families of models on 2,410 factual questions. Their finding, verbatim: “adding personas in system prompts does not improve model performance across a range of questions compared to the control setting where no persona is added.” Worse for the guru economy, when they looked for patterns in which personas helped which questions, they concluded “the effect of each persona can be largely random.” Picking the best persona in advance performed about as well as picking one out of a hat.

The politeness cousin fares little better. A cross-lingual study of prompt courtesy in English, Chinese, and Japanese found that rude prompts tended to degrade output, but stacking on extra deference bought roughly nothing. Don’t insult the model; don’t grovel either.

One honest caveat: vendor docs do recommend role prompting, and they’re not wrong — for voice. Telling a model it’s writing as a copy editor changes register and format. What the evidence doesn’t support is the accuracy claim, the idea that “you are a Nobel laureate” makes the reasoning better. Costume changes the performance, not the performer.

Specificity beats vibes

The single highest-return move, per every major lab’s own guidance, is embarrassingly plain: say what you actually want. Anthropic’s prompting guide frames the model as “a brilliant but new employee who lacks context on your norms and workflows,” and offers a test anyone can run before hitting enter:

“Show your prompt to a colleague with minimal context on the task and ask them to follow it. If they’d be confused, Claude will be too.”

The same guide has a concrete before/after you can reproduce: ask “Can you suggest some changes?” and the model will sometimes offer suggestions instead of making the edits; ask it to implement specific changes and it does. Ambiguity costs more than it used to, not less. OpenAI’s GPT-5 prompting guide warns that “poorly-constructed prompts containing contradictory or vague instructions can be more damaging to GPT-5 than to other models, as it expends reasoning tokens searching for a way to reconcile the contradictions rather than picking one instruction at random.”

Try it on the most common prompt in office life. “Summarize this report” leaves the model to guess the length and the audience. “Summarize this report in five bullets for a finance director who hasn’t read it; flag any figure that contradicts the executive summary” removes the guessing. Same model, different output, and you can verify it in two minutes with any document you have lying around.

One example outranks three paragraphs of description

If specificity is the first lever, examples are the second, and here the vendors are unusually blunt. Google’s Gemini documentation: “We recommend to always include few-shot examples in your prompts. Prompts without few-shot examples are likely to be less effective.” Anthropic’s guide calls well-crafted examples one of the most reliable ways to steer output and suggests three to five, varied enough to cover edge cases.

The reproducible test: take a formatting task, like extracting names and dates from messy text into a table. Describe the format in prose and count the corrections you have to make. Then instead paste one completed input-output pair and the words “now do the same for these.” The described format drifts; the demonstrated one mostly doesn’t. Google’s guide states the reason plainly: “One of the primary objectives of adding few-shot examples in prompts is to show the model the response format.”

The technique with actual benchmark numbers

Only one prompting technique has produced results dramatic enough to reshape the field. In 2022, Google researchers showed that giving a large model a handful of worked examples with reasoning steps spelled out, a method they called chain-of-thought prompting, pushed a 540-billion-parameter model to state-of-the-art accuracy on the GSM8K math benchmark with “just eight chain of thought exemplars,” beating a fine-tuned GPT-3 with a verifier. Months later, a second team found a lazier version worked too: appending “Let’s think step by step” lifted the same benchmark’s zero-shot accuracy from 10.4 to 40.7 percent, and a simpler arithmetic set from 17.7 to 78.7 percent.

Before you tattoo the phrase somewhere: current reasoning models bake this in. They generate their own intermediate steps whether you ask or not, which is why the magic incantation now buys you little on frontier chat models. What survives is the underlying principle: decomposition. Google’s guide recommends breaking complicated work apart (“instead of having many instructions in one prompt, create one prompt per instruction”) and chaining, where the output of one prompt becomes the input of the next. Draft, then critique, then revise, as three calls instead of one. Slower. Better.

The part nobody sells

There is now a 31-author academic survey cataloguing 58 distinct text-prompting techniques, which tells you two things: the field is real, and no single trick dominates. What separates the published record from the listicles is that researchers treat prompts as testable artifacts. Battle and Gollapudi’s actual conclusion was not “use Star Trek prompts.” It was that systematic optimization beat every hand-tuned snippet they wrote, on every model they tried.

You don’t need an optimizer to borrow the discipline. Keep five or ten representative inputs with known-good outputs. When you change a prompt, run the set. Change one thing at a time. That’s it. That’s the whole secret the $47 prompt-pack PDF is standing in front of.

The strongest math prompt anyone found for that VMware benchmark was never written by a person at all. It was discovered, by a machine, and it reads like fan fiction. If magic words exist, they look like “Captain’s Log, Stardate [insert date here]” — and they stop working the moment you change the model.