OpenJev

SemIf is an independent research initiative designed to facilitate local decision-making directly within your web browser.

This "unsloppified" site serves as a live, local experiment where a decision model operates entirely on the client side. The core objective is to compare two distinct ways a local model can handle options:

1. Direct Readout: Extracting probabilities for permitted options without performing any decoding.
2. Token Generation: Writing out the same probability distribution one token at a time.

Users can select a model size, execute both methods on their own GPU, and analyze the resulting performance and accuracy differences.

Key Advantage: There is a waitlist no waitlist! The system is browser-only with no backend, meaning your timings are purely local.

---

00 / Setup & Model Selection

To begin, you must load a model into your browser cache. For those using mobile phones or low-spec hardware, switching to the Qwen3 0.6B model is recommended.

Available Models & Benchmarks

The following table outlines the quality of native checkpoints based on public benchmarks. Note: "Owned" columns represent balanced accuracy.

• TypeSafe Definition: This metric refers to equal-case agreement measured on a specific 102-row public subset.

Initialization Process

• Download/Cache: Triggered upon clicking "load model."
• Load: The process of downloading and preparing the weights.
• Warmup: Running compile passes for both the readout and generation methods.

Weights are sourced from Hugging Face and stored in your browser's local cache. Depending on your GPU and network speed, the initial load may take several minutes.

---

01 / The Decision Process

You can test the model with a real-world choice. For example, consider an Email Triage scenario:

State: A customer reports that while their password reset was successful, every subsequent login attempt results in an “account locked” error.

Question & Options:

• Option A: remove
• Option B: remove
• Option C: add

Both methods receive the identical decision state. The difference lies in the execution path:

02A / Direct Readout

This method involves Choice Probabilities without decoding. The system reads the model's choice logits and applies a normalization process.

$\text{Probability} = \frac{e^{z_i}}{\sum_{j \in \text{options}} e^{z_j}}$

• Metrics tracked: Total time, input time, and output time.

02B / Generation

This method utilizes JSON Probabilities generated token by token. The model is prompted to estimate the distribution for the displayed options and format the result as a JSON object.

{
  "Option A": 0.15,
  "Option B": 0.10,
  "Option C": 0.75
}

• Metrics tracked: First token latency, total time, input/output time, and the measured wall-time ratio.

---

Technical Nuances & Performance

Because the methods run sequentially on the same loaded model, they do not compete for GPU resources.

What the data indicates:

• Conditional Probabilities: Direct scores are a $\text{softmax}$ calculated only over the tokens of the options provided.
• Calibration: These are not calibrated confidence scores and do not account for every possible answer the model might have preferred.

Timing Methodology: All performance metrics—including setup, warmup, prompt prep, execution, and the interval between the first and last generated token—are measured using the performance.now() API.