Case Study: How The Philadelphia Inquirer Developed Scrape to Uncover Hyperlocal News

By Sonali Verma | September 30, 2026 A project supported by the Lenfest AI Collaborative and Fellowship Program

🚩 The Challenge: The "Noise" of Hyperlocal Reporting

The Philadelphia Inquirer produces a series of hyperlocal newsletters tailored to specific neighborhoods. However, the process of finding story leads was inefficient. The news ecosystem for these areas is highly fragmented, consisting of:

• Municipal meeting minutes
• School district calendars
• Community Facebook pages

The manual labor involved was staggering. One editor was spending approximately: $\text{Time Spent} = 12 \text{ hours/week}$ ...just on the discovery phase. This left precious little time for the actual craft of journalism—writing and developing a unique voice for the publication.

"Gathering news for these regions was hard because it comes from a lot of really small sources, and so it’s a lot to sift through," explained Kevin Hoffman, the Lenfest Institute’s AI Fellow embedded at The Inquirer.

---

🛠️ The Solution: Introducing Scrape

To combat this, the team developed Scrape, an AI-powered tool designed to increase the signal-to-noise ratio of news gathering. Rather than treating the tool as a standalone technical experiment, it was integrated directly into the existing editorial workflow.

The Development Workflow

The team didn't start from scratch; they grounded the AI in the editor's existing manual "tip sheets."

The process involved a rigorous three-to-four-month feedback loop. Every other day, the editor would annotate the AI's output, marking what was valuable and what was irrelevant.

Filtering the Noise

Through iterative prompt engineering, the team taught the AI to distinguish between routine updates and actual news.

---

🎯 Precision through Editorial Context

A pivotal realization was that geography alone is not a sufficient filter for "newsworthiness." The team began embedding editorial context directly into the system.

Instead of just saying "find news in this zip code," they used detailed newsletter briefs. These briefs defined:

1. The Audience: Who is reading?
2. Relevant Topics: Local government, retail, and restaurants.
3. Exclusions: The AI was explicitly told to ignore straight business news (e.g., stock exchange updates) or personnel changes.

Example of a conceptual prompt structure used in Scrape:

{
  "newsletter_id": "Philly_Local_01",
  "target_audience": "Residents of Lower Merion",
  "include": ["municipal_impact", "local_retail", "school_board_decisions"],
  "exclude": ["stock_market", "corporate_HR_changes"],
  "geographic_scope": "Lower Merion School District"
}

The Impact

Scrape has transitioned from a prototype to an experiment critical, load-bearing infrastructure.

• Scale: Currently supports 6 newsletters (expanding to 8).
• Reach: Utilized by nearly 20 journalists across the newsroom.
• Outcome: The Local team can now meet high audience demand without burnout.

---

📉 Lessons Learned & Course Corrections

The journey wasn't without hurdles. The team identified two primary areas for improvement:

1. Scaling Philosophy: Original Approach: Scrape everything in a region $\rightarrow$ Apply one generic "newsworthiness" filter. New Approach: Anchor "newsworthiness" in specific, editor-authored briefs that define the exact coverage boundaries.

2. Cost vs. Quality: Early on, the team struggled with the high cost of querying numerous scattered sources. They learned that prompting implicitly affects search depth.

   • Lesson: Be explicit about cost-quality trade-offs from day one.

---

📋 Blueprint for Other Newsrooms

For organizations dealing with "messy" local data, the Scrape model provides a repeatable framework:

• Audit Manual Effort: Identify the person spending the most time on manual discovery (e.g., the "12-hour-a-week" editor).
• Establish Benchmarks: Use existing manual products (tip sheets/newsletters) as the "gold standard" for AI evaluation.
• Implement Daily Feedback: Require editors to annotate AI outputs daily to refine the prompts.