Investing in Human Feedback: Why I Paid People to Test My README

Developers are notoriously poor at writing documentation. Usually, we provide a haphazard mix of outdated tips, fragmented notes, and a link to a ghost-town forum where the same questions are asked in an endless loop.

As I have previously argued, you must actually test your README. The best way to do this is to launch a clean Virtual Machine and follow your own instructions to see if they actually hold water.

Often, we suffer from "curse of knowledge" blindness:

• We forget that certain commands require sudo.
• We write -foo when we actually meant --foo.
• We assume everyone knows a reboot is required.

The ActivityBot Experiment

While applying for an NLnet grant for ActivityBot, I committed to testing the installation experience with actual users. To ensure quality feedback, I offered €25 for one hour of their time.

I recruited volunteers via Mastodon and conducted the sessions over video calls. My goal was simple: gather constructive criticism on the "first-run" experience.

The Methodology

I asked participants to share their screens and—most importantly—think out loud. I wanted to hear their confusion, their frustrations, and their moments of delight. I avoided digital tools for note-taking, preferring to write by hand to avoid "feeding the machine."

Key Discoveries

Through these sessions, I realized how many incorrect assumptions I had made. Some of the highlights included:

• Broken Links: The link to the demo tool was incorrect.
• Architectural Confusion: Users questioned whether the demo tool should be web-based or strictly local.
• Syntax Ambiguity: Why was some text quoted while other text wasn't?
• Irrelevant Detail: I had included a section that was technically fascinating to me but completely baffling to everyone else.

The Cost of Quality

In total, I spent approximately €150 to have strangers critique my work to my face.

Some might ask: Why not just use an LLM to simulate different user personas?

"The answer is simple: I want to speak to real people."

Humans bring a level of unpredictability and warmth—like a cat wandering onto a video call—that an AI cannot replicate. More importantly, they are genuinely happy to receive a €25 voucher.

Lessons from GOV.UK

This approach mirrored my time doing technical writing for GOV.UK. Back then, my "beautiful" prose was always reviewed by another human who would decorate it with adjectives eviscerate my flowery language to make the document actually readable. They caught the nuanced errors that no spell-checker ever could.

The most vital signal was the tone of their voice; hearing frustration is the clearest indicator that a mistake needs fixing.

Final Thoughts

While the ActivityBot README isn't "perfect," it is now demonstrably easier to use. If you can't afford to pay people, simply find a few volunteers who are willing to talk through their process while following your guide.

Your Documentation Checklist:

• Spin up a fresh VM
• Verify all sudo requirements
• Check all hyperlinks
• Conduct "Think-Aloud" user testing

The mathematical value of this process can be seen as: $\text{Better Docs} = (\text{Technical Accuracy} + \text{Human Friction Testing}) \times \text{Iterative Updates}$

---

Community Perspectives

The post sparked several interesting discussions:

• James Aylett suggested that grants should specifically fund documentation efforts.
• Tim Chase noted that some markup converters accidentally turn -- into em-dashes (—), breaking commands like $ foo —option.
• Tito Swineflu admitted to only reading READMEs in the terminal, finding HTML/Markdown versions irritating.
• Marcus Rohrmoser emphasized that UX testing is inevitable for anything intended for public use.

Technical Appendix: The Share Function

The original post included a snippet for sharing the content:

{ 
  try { 
    await navigator.share({url:u}); 
  } catch { 
    alert('Copied URl to clipboard'); 
  } 
}

Related Assets: