Automating Immersive Reading

Originally published on smoores.dev — Building stuff on the Internet.

The Evolution of Storyteller

What began as a modest project has blossomed into a comprehensive software ecosystem. Storyteller currently spans a wide array of platforms:

We are currently in the midst of alpha and beta testing for our v3 releases, which aim to deliver:

• Refreshed User Interfaces
• Expanded library management toolsets
• Enhanced performance optimizations

---

What is "Immersive Reading"?

At its core, Storyteller synchronizes an ebook with its corresponding audiobook. The system identifies the exact timestamp where every word in the text is spoken.

The Technical Workflow

The application leverages the Media Overlays system—a feature built directly into the EPUB specification. By embedding synchronization metadata, the reader app can highlight sentences in real-time as the narrator speaks them.

Example Scenario: Imagine reading Peter and Wendy by J. Barrie; as the narrator describes Wendy realizing she must grow up, the corresponding text glows on your screen, guiding your eyes in perfect harmony with the audio.

---

From a Simple Script to a Complex System

In its infancy, Storyteller wasn't a suite of apps; it was merely a single python script.

# Conceptual origin of Storyteller
def align_book(ebook, audiobook):
    # Process audio and text
    # Generate Media Overlay metadata
    return synchronized_epub

At the time, the ecosystem for these files was desolate. There were virtually no hardware e-readers that supported Media Overlays. I would process the files locally and transfer them to my phone, relying on the early, experimental support provided by BookFusion.

During this phase, I was an amateur in the world of forced alignment. My initial attempts were, in my own words:

"A wobbly construction of clunky, nested while loops. I felt like I was stumbling through the dark, aware that there was probably light at the end, but unable to see it."

---

The Hurdles of Forced Alignment

Aligning a book is significantly harder than aligning a simple transcript. Even specialized tools struggle with the nuances of publishing. Storyteller had to solve several specific problems:

1. Discrepancies in Chapter Order

The sequence of content often differs between formats.

• Ebook: $\text{Frontmatter} \rightarrow \text{Story} \rightarrow \text{Backmatter}$
• Audiobook: $\text{Story} \rightarrow \text{Frontmatter/Acknowledgements}$

Case Study: In Tress of the Emerald Sea by Brandon Sanderson, the acknowledgments appear at the start of the ebook but are relegated to the very end of the audio recording.

2. Missing Content (Chapter Existence)

Not every chapter exists in both versions. Audiobooks frequently omit:

• Tables of Contents
• Forewords
• Appendices

Case Study: In You Just Need to Lose Weight by Aubrey Gordon, the ebook contains a final acknowledgments chapter that is completely absent nonexistent in the audiobook.

3. Skipped Spans

Sometimes, the narrator skips a few sentences, or the audio includes descriptions of images that aren't in the text.

Case Study: In the Rosner translation of Siddhartha by Herman Hesse, the audio skips several sentences regarding the Chandogya-Upanishads, though the rest of the text remains aligned.

4. Lexical Substitutions

Narrators often change words to improve oral flow. This is common in non-fiction.

Example: You Didn't Hear This From Me by Kelsey McKinney

• Ebook: "Reading Reading this book..."
• Audiobook: "Listening this book..."

---

Summary of Alignment Difficulty

While basic forced alignment can handle word swaps (like "read" $\rightarrow$ "listen") and minor skips, the structural issues are the real "dealbreakers."

Previous tools, such as the impressive syncabook, required a manual mapping process: $\text{User Input} \rightarrow (\text{Ebook Chapter } n \approx \text{Audio Chapter } m)$

This manual overhead is tedious and difficult, which is exactly the problem Storyteller aims to automate.