I Spent 5+ Hours Pushing the Limits of an AI Writer (It Was Terrifying)

An AI didn't write this. Or did it?

I asked myself that a lot as I tested the alleged "best AI writer" on the market.

I've spent enough time covering emerging technologies like AR and VR to know not to underestimate how fast nascent tech can improve. That's why I was never skeptical of artificial intelligence's capacity for creativity. Creativity, in essence, is sifting through many data points looking for meaningful connections.

What makes an algorithm's process so different from the one I used to write this headline?

The question remained though: "Can an artificial intelligence actually perform on par with a human writer?"

That's where I had my doubts. Until I spent some time throwing every conceivable challenge I could think of at an AI:

• Writing an introduction for an article when given a real brief.
• Performing granular research about a specific topic.
• Roasting me based on my About page (I'm worried how quickly it resorted to racism).
• Crafting arguments based on unwinnable stances.
• Imitating the style of famous writers.
• Reflecting on a specific lived experience despite having never "lived".

After spending over 5 hours testing an AI writer, I can confidently say the bottom 30% of writers are probably screwed.

AI is pretty good at paint-by-numbers writing 👍🏾

I'm embarrassed to say how long I spent writing what you're reading now. I hate settling for anything less than ambitious angles and artful articulation. That inclination towards perfectionism often becomes a liability with "easy" writing tasks, like drafting an email.

So when someone told me they spent an hour stressing over an apology email to customers for a service outage, I decided to see what the AI could do when given the same task.

What originally took an hour and a great deal of emotional energy was done in seconds with a decent level of emotional intelligence.

If the relationship between the writer and the reader is purely transactional—news reporting, informational emails, simple product descriptions, press releases—I think AI can do a better job faster than most humans, making it a good companion for non-writers (or writers with friends who constantly ask, "Can you make this sound better?").

AI writers can't compete in complex contexts—even with a human editor 👎🏾

While I was testing out this AI writer, I was wrapping up an article for a client about turning Twitter into a sales channel.

I'd taken my time crafting the introduction. Those first few lines set the tone for the rest of the article.

Could an AI measure up to my own standards? Or at least give me something to work with? After all, people have said AI writers work well when paired with a human editor.

But what I got back from the AI was:

• Not just generic but bad advice for the reader.
• Filled with 3 back-to-back uses of "a great way" to introduce an idea (to be fair, I had just lectured a human writer about the same).
• Padded with unnecessary repetition, like a college stoner trying to hit a word count on a last-minute essay.
• Not something I could salvage through editing.

I write in a competitive and fast-moving niche though. AI performed much better when I tested it with low-stakes, consumer-oriented content about topics like tea or pop culture. But I wouldn't rely on it to serve an audience in a niche where expertise and primary research are table stakes.

AI can be opinionated—kind of 🤷🏾‍

I love opinions. They're at the heart of every good discussion.

They're also a common feature of strong writing. So, can an AI writer have a distinct point of view about a topic? Can it shift its perspective and rationale to adopt an opinion from another angle?

I thought, what better way to test this than "Kiss, Marry, Kill": the most juvenile of thought experiments. I gave it Beyoncé, Ed Sheeran, and Kermit the Frog as options.

What I found was:

• The AI consistently answered based on the order the choices were presented. Always choosing to kill Kermit until (at the risk of being cancelled by the "Beyhive") I put Beyoncé last.
• It was able to provide a different rationale each time it made the same choice.
• It referenced the distinct traits of each option (like Kermit being a puppet) in justifying its choices.

AI can mimic lived human experiences in its writing 👍🏾

Some of my most resonant writing, the stuff people reach out to me privately about, is first-person storytelling drawn from my own experience.

That's something I thought an AI could never do. How can it write about a lived experience when it has never lived? I was wrong.

A friend suggested testing if the AI can navigate the cultural nuances of a piece about growing up as an Asian American.

The result?

• It gave a surprisingly poignant account of what it's like growing up between two identities.
• The emotions expressed were quite relatable, given they came from an AI
• The story lacked the specifics a real human might offer.
• It could pass a Turing test (which is depressing when I routinely fail CAPTCHA tests).

AI sucks at research, but it can generate misinformation faster than Alex Jones👎🏾

It's not the "taking over our jobs" narrative that scares me. It sounds harsh, but if you're the kind of writer who can be easily replaced by an AI, you're not a Writer, you're a Typer.

What actually scares me is how easily AI can generate believable content that is so patently false. Misinformation used to be easy to spot because the work of its worst perpetrators was often riddled with errors. If nothing else, AI writers can produce grammatically correct writing with an air of objectivity, which can be dangerous when the content itself is factually incorrect.

Say what you want about click-hungry journalists, but some due diligence is better than none at all in a world where anyone can publish and distribute anything.

To test its research skills, I gave it three tasks ranging from easy to impossible:

1. Reporting on an easy-to-find, non-controversial stat.
2. Curating facts about a niche topic.
3. Justifying the stupidest conspiracy theory I could think of.

AI is capable of nuanced reasoning (unlike some people) 👍🏾

If you've ever studied formal logic, you won't be surprised to find AI is pretty good at it. Logic is at the heart of programming, after all. It's "If this, then that", not "I feel like this, so maybe that."

I wanted to know how well an AI could apply that logical prowess to make a nuanced argument that has to also consider opposing viewpoints.

I asked it the most open-ended question steeped in conflict of interest that I could think of to see if it could navigate nuance. I'm not sure whether I was surprised or not to find it was better than a lot of humans.

I'd also tested it with different viewpoints, including some that were "unwinnable". While its supporting evidence was weak or non-existent, the reasoning was usually solid.

AI is okay at emulating writing styles 👍🏾

Style is the true signature of a writer, not the name they put on the byline.

Copying another writer's style involves more intuition than logic, looking for patterns across their body of work, from preferred rhythm, tone, themes, rhetorical devices, sentence lengths and structure.  

It's this ability that separates wordsmiths from writers. If AI can do this, then that's another win over most writers.

Intuition is just unconscious pattern-matching and, like logical reasoning, AI is no stranger to it.

So I gave it the same prompt multiple times, but with the added instruction to write in the immediately recognizable styles of three famous writers:

• Rupi Kaur
• William Blake
• Dr. Seuss

You probably wouldn't be able to recognize each writer's style from these samples alone. But if I told you which three writers I'd used, you'd likely be able to match them to each poem.

AI can get writer's block. Or so I thought 😐

"Aha! Finally found your limit, stupid robot," I thought when I saw the loading icon spin for half a minute with no response.

I'd asked it to write something no algorithm could possibly understand—to elicit the human reaction so raw and pure it's the one most of us have when we enter this world and when we leave it too.

After all, how can an AI possibly know how to reach into a reader's soul and draw out the tears they hold inside?

Well, turns out it was just a temporary service outage.

When it was back online, I tried again and what it wrote made me decide it was probably time to go outside and touch some grass.

In conclusion? I'll let the robot write it 🤖

I have mixed feelings about AI-generated writing.

Among the positives, I think it can:

• Become a time-saving tool for top writers to delegate low-impact work and save them some keystrokes
• Augment the divergent thinking process of rapid trial-and-error at the heart of all creativity
• "Ghost write" for non-writers who have great ideas, allowing anyone to communicate their expertise more effectively, especially in other languages

There are clear negatives too, as it will likely:

• Encourage laziness among even great writers who put making money before professional integrity
• Be used to circumvent the research process, which without human fact-checkers can lead to inaccurate coverage of complex topics
• Enable fraud and misinformation on an unprecedented scale
• Eliminate low-level jobs, raising the barriers to entry for writing careers

That's just my take as an imperfect flesh-based intelligence. But I wonder what conclusion an AI would draw.

Well, I couldn't have said it faster. But I definitely could've said it better.

And I think that's what it will mean to be a Writer in a world where AI makes it so easy to get words on a page.

It will mean approaching writing as a craft worthy of our ambition, pride, and time through the work we do off the page too:

• Getting lost down research rabbit holes in pursuit of truth, knowing dead ends are part of the adventure  
• Consuming the world at large, and the worlds within others, in ways an AI can't—through intimate conversation and powers of observation
• Stepping away from the screen to let the work marinate because it hasn't satisfied our tastes

A rational AI will always win against us humans in a time-based competition.

But what an AI will never do by design is what the irrational artist does by default:

Forget time entirely to be immersed in their process, based on sheer faith alone that whatever's on the other side is worth it.

"It's a lot of time to spend on something that means absolutely nothing...That’s what I do. That’s what people want me to do–spend a lot of time wastefully."  —Jerry Seinfeld