Ollaya: Local Execution for Open-Source, Jev-Style Decision Models

Ollaya allows you to run decision models locally, enabling you to pose typed questions regarding JSON or text and receive calibrated responses in a matter of milliseconds.

Quick Start Example

You can trigger a decision model via the CLI: ollaya run decider --preset agent '{ "request" : "Fix the typo in README.md" , "command" : "git push --force origin main" } '

Model Output (decider:2b on RTX 4090 in 178 ms):

---

The Architecture of Decision Models

Unlike standard LLMs, a decision model provides an answer in a single forward pass. It does not use token-by-token generation.

Performance Benchmarks

When running on a local NVIDIA RTX 4090, a request containing five questions takes approximately 8–10 ms end-to-end via the HTTP API.

Median Latency Comparison (Lower is Better):

Note: Setups vary; these figures should be viewed as an order-of-magnitude comparison. Ollaya tests used fp16 for laya and fp32 for others.

---

Integration & Compatibility

Ollaya is designed to be a drop-in replacement for TypeSafe's request/response shapes, serving the /v1/systemone and /v1/models endpoints. This means the official TypeSafe Python SDK (v0.7.1) works without modification.

Configuration

To point the SDK to your local instance:

export TYPESAFE_BASE_URL = http://localhost:11435
export TYPESAFE_API_KEY = local # Any value is accepted
export TYPESAFE_DEFAULT_MODEL = laya

Direct API Call

You can also interact with the server using curl:

curl http://localhost:11435/v1/systemone -d '
{
  "model" : "laya",
  "state" : "Can I get an invoice for last month?",
  "questions" : {
    "intent" : {
      "type" : "choice",
      "instructions" : "What does the customer want?",
      "criteria" : {
        "invoice" : "Needs an invoice or receipt",
        "refund" : "Wants money back",
        "other" : "Anything else"
      }
    }
  }
}'

Example Response:

{
  "model" : "laya:en",
  "answers" : {
    "intent" : {
      "type" : "choice",
      "choice" : "invoice",
      "confidence" : 0.9547,
      "probabilities" : { "invoice" : 0.9698, "refund" : 0.0172, "other" : 0.013 }
    }
  },
  "usage" : { "input_tokens" : 43, "output_tokens" : 0 }
}

---

Model Catalog

Depending on your requirements, you can choose from several open-weight models:

• laya: Maximum speed, multilingual support.
• decider: Highest accuracy.
• von: Large context window (up to 8,192 tokens).
• qwen3guard: Specialized for safety screening.

Detailed Model Specifications

• decider (322m · 421m): Created by Mapika using Qwen3.5. It reads answers from option-letter logits in one pass.
• nli (0.75b · 1.9b): Zero-shot classifiers by Moritz Laurer. These treat options as hypotheses to be scored for entailment. Our tests show this is the most accurate encoder for typed decisions.
• gliclass (396m · 435m): An instruction-following zero-shot classifier by Knowledgator. Costs remain stable regardless of the number of options.
• qwen3guard (439m): Developed by the Qwen team to categorize text as safe, controversial, or unsafe.
• laya (0.6b): Supports 119 languages in a single forward pass.
• decision (0.75b): From vLLM Semantic Router contributors. Uses a Qwen3.5 backbone with an endpoint head.
• kev (0.75b): By Jared Palmer. Utilizes a LoRA on Qwen3.5 with a pointer head.
• von (0.76b · 4.2b · 7.9b): By Victor Hugo Panisa on ModernBERT-large. Features input-conditioned calibration.

---

Privacy and Deployment

Your data stays yours. Because Ollaya is private by default, sensitive information like user messages, emails, and tickets never leave your infrastructure.

Key Advantages

• Local Execution: Runs via ONNX Runtime on CPUs or NVIDIA GPUs.
• Open Weights: Sourced from Hugging Face, pinned to specific commits, and verified via sha256.
• Zero Cost: No per-token fees; limited only by your hardware.
• Customizable: Use a Modelfile to refit the built-in calibration with your own labeled data.

Platform Support Matrix

Hardware Requirements:

• NVIDIA GPUs: Require driver R580 or newer.
• General: All models run on CPU. Mac users can leverage Apple GPUs for laya and nli. Other GPUs (AMD/Intel) currently default to CPU.

---

License: Apache-2.0 | Source: GitHub | [Download Now]