Senior SWE-Bench: Assessing Agents as Senior Engineers

"We treat agents like senior engineers, so why evaluate them like junior engineers?"

Senior SWE-Bench is designed to move beyond basic coding tests, challenging AI agents to operate with the autonomy and intuition of a seasoned software engineer.

🛠️ The Three Pillars of Senior Evaluation

Evaluation Workflow

---

📖 Case Study: The BookWorm Metadata Integration

To illustrate the complexity of these tasks, consider the following requirement for the BookWorm system.

🚨 The Problem / Opportunity

Currently, BookWorm relies exclusively on Amazon and ISBNdb. This creates a failure point when metadata is missing or malformed, particularly for books only possessing ISBN-13s. This leads to:

• Failed enrichment for /api/import or promise items.
• Low-quality entries in the Open Library.
• Reduced success rates for international or rare titles.

The Goal: Integrate Google Books as a fallback source to improve data completeness and user trust.

✅ Definition of Success

• BookWorm can successfully fetch and stage metadata via ISBN-13 from Google Books.
• Automated tests verify parsing of:
  • Core fields: Title, subtitle, authors, publisher, page count, description, and publish date.
  • Edge cases: Missing fields or null authors.
  • Null results: Correctly returning no result for zero or multiple matches.

📋 Technical Proposal & Requirements

The implementation requires modifications across several core files:

1. Configuration: In openlibrary/core/imports.py, the STAGED_SOURCES tuple must be updated to include google_books.
2. API Integration: The staging URL is defined as: http://{affiliate_server_url}/isbn/{identifier}?high_priority=true&stage_import=true (Where affiliate_server_url is sourced from openlibrary/core/vendors.py and identifier is an ISBN-10, ISBN-13, or BASIN).*
3. Data Persistence:
   • In openlibrary/plugins/importapi/code.py, the function supplement_rec_with_import_item_metadata must extend the source_records field rather than overwriting it.
   • In scripts/affiliate_server.py, a new function stage_from_google_books must use Batch.add_items to persist successful API fetches.
4. Fallback Logic: The system should trigger the Google Books fallback based on the following logic: $\text{Fallback} \iff (\text{Amazon Result} = \emptyset) \land (\text{ID} = \text{ISBN-13}) \land (\text{high\_priority} = \text{true}) \land (\text{stage\_import} = \text{true})$
5. Safety Constraints: If the Google Books API returns $> 1$ result for a single ISBN, the system must log a warning and $\text{skip}$ staging to prevent data corruption.
6. Schema Requirements: Staged metadata must include: isbn_10, isbn_13, title, subtitle, authors, source_records, publishers, publish_date, number_of_pages, and description.
7. Pipeline Update: In scripts/promise_batch_imports.py, replace direct Amazon-only logic with stage_bookworm_metadata.

🔌 New Public Interfaces

The following functions are introduced in scripts/affiliate_server.py:

# Example of the expected logic flow in scripts/affiliate_server.py
def stage_from_google_books(isbn):
    data = fetch_google_book(isbn)
    if data:
        processed = process_google_book(data)
        Batch.add_items(processed)
        return True
    return False