Laya vs Ornith 1.5 9B: Speed and Accuracy on Text Classification

Can a model 21 times smaller than an LLM sort our support tickets just as well? For some jobs, yes. On support tickets, Laya beat a 9B model while answering in 12 ms. On product reviews it guessed the star rating right only 37% of the time, and the 9B model got 60%.

Laya from ConvAI Innovations is not a chat model. It never writes an answer. We give it some text and a question with a fixed set of answers, and it returns a score for every answer in one pass. People have been talking about how fast it is and how good its answers are. So we tested both claims against a normal LLM that can do the same job, Ornith 1.5 9B.

In this blog, we will learn how Laya works, how we ran both models fairly on five real labeling tasks, and what we get and lose with each one. Everything runs on one MacBook Pro M5 Max with 36GB of memory, using MLX.

What Is Laya?

Laya is a decision model. An LLM writes an answer one word at a time. Laya reads the text and the question together once, and then scores each possible answer at the same moment. In simple words, it fills in a multiple choice sheet instead of writing an essay.

Under the hood, it is a ModernBERT-large encoder of 395M parameters with a small decision head on top, 421M parameters in total.

It supports three kinds of questions:

• choice: pick one label from a list, like a department or an intent.
• score: pick a level on a scale, like 1 to 5 stars.
• noul: a yes or no question, returned as a probability.

It also ships a set of temperatures. A temperature is one number that makes the model's confidence more or less sharp, so that 80% confident means right about 80% of the time. We will come back to this, because it turned out to matter.

We used the MLX port aac6fef/laya-mlx at 16-bit, with the laya-mlx 0.2.0 runtime. Let's see how we ask it a question as below:

import laya_mlx as laya

agent = laya.load("aac6fef/laya-mlx", dtype="float16")

questions = {
    "department": {
        "type": "choice",
        "instructions": "Which support department should handle the customer's `message`?",
        "criteria": {
            "refund": "refund policy, requesting or tracking a refund",
            "order": "place, change, cancel or track an order",
            "payment": "payment methods and problems paying",
        },
    },
    "refund_request": {
        "type": "noul",
        "instructions": "Is the customer asking to get money back or checking on a refund they requested?",
    },
}

out = agent.predict({"message": "I paid twice for order 5521, please send one payment back"}, questions)
print(out["answers"])

• laya.load(...) downloads the 0.84 GB model and loads it in 0.4 seconds.
• criteria lists the allowed answers. A short description next to each label helps the model tell them apart.
• agent.predict(...) takes the input as a dictionary and answers every question in one call.

The output is as below:

{'department': {'type': 'choice', 'confidence': 0.5082, 'choice': 'payment', 'probabilities': {'refund': 0.1244, 'order': 0.0407, 'payment': 0.8349}, ...}, 'refund_request': {'type': 'noul', 'confidence': 0.7065, 'noul': 0.7065, ...}}

Here, we can see both answers came back from one call, with a probability for every label. There is no text to parse and no way for the model to answer with a label that is not on the list.

Our Test Setup

Ornith 1.5 9B is a normal LLM, built on the Qwen3.5 hybrid design. We used the 8-bit MLX build ornith-ai/Ornith-1.5-9B-MLX-8bit with thinking mode off. Let me tabulate the setup for your better understanding:

Both models saw the same records, the same question wording, and the same JSON input. We did not train or tune either model on our data. This is zero-shot for both.

We made the LLM as fair and as fast as we could. It could only pick labels from the list, because we limited its output to valid labels token by token. This is called constrained decoding. For yes or no questions and star ratings, we read the probability of its very first token, so it never wrote more than it had to. We also reused the shared part of the prompt, the question and the label list, for every record. So, its time counts only the new text of each record.

To save time, Ornith ran on the full set for ticket triage and complaint routing, and on a fixed sample of 300 records for the other three tasks. For those three, we scored Laya on exactly the same 300 records, so every number in a row below compares like with like.

The Five Tasks

We picked five jobs that teams really use a classifier for. Let me tabulate them:

The Bitext messages are short and clean. The CFPB complaints are long, messy letters written by real people. CLINC has 150 intents plus an "out of scope" label for requests that fit none of them. The log task is hard on purpose: we made sure half the normal lines also carry a FATAL or FAILURE level, so a model cannot just look for scary words.

Which Model Gives Better Answers?

Now the most important question. For label questions we report accuracy. For yes or no questions we report AUROC, which asks one thing: if we pick one real "yes" and one real "no", how often does the model give the "yes" a higher score? 1.0 is perfect and 0.5 is a coin flip.

Let me tabulate the full results:

Here, we can see a clear split. On short, clean support messages, Laya wins. On long complaints, many labels, fine grades of opinion, and messy logs, Ornith wins by a lot. Let's go through each task to see why.

Support Tickets: Laya's Best Task

Laya picked the right intent out of 27 for 99.0% of the tickets, and Ornith got 84.8%. Most of Ornith's 167 misses were close cousins. It sent delivery_period to track_order 32 times, check_invoice to get_invoice 22 times, and track_refund to get_refund 20 times. Laya kept those pairs apart.

The refund question is more mixed. Laya ranks refund tickets well, with an AUROC of 0.964. But at its default cut-off of 0.5, it flagged too many tickets as refunds. It caught 74% of real refund tickets, but only 43% of the tickets it flagged were real ones, for an F1 of 54.8. Ornith scored 83.3 on the same measure.

Complaint Routing: Long Letters Hurt Laya

On the CFPB complaints, Laya reached 53.1% against 74.9% for Ornith. Two things went wrong.

First, Laya reads at most 512 tokens, and 22.5% of the complaints were longer than that. The end of the letter, where people often say what the product was, got cut off. Ornith read up to 3,000 tokens.

Second, Laya sent far too much to debt_collection. It picked that label for 624 of the 1,800 complaints, when only 200 were really about debt collection. Many complaints about a loan or a card also mention collection calls, which may be what pulled it there. Personal loan complaints were routed right only 12% of the time.

Many-Label Intent: 151 Labels Is a Lot

CLINC asks the model to pick one of 150 intents, or say "out of scope". The first problem is that Laya has a budget of 192 tokens for the answer list, and 151 labels do not fit. So we raised its limits to 1,024 tokens for the whole input and 768 for the labels. Without that, most of the labels are silently dropped.

Even with room for every label, Laya reached 55.0% on the 300 records, against 76.7% for Ornith. The bigger gap is out of scope. Ornith said "none of these" correctly 79.3% of the time. Laya did it 6.9% of the time. It almost always forced a real intent onto a request that had none.

We also tried a common trick: first shortlist the 20 closest intents with Laya's own embeddings, then ask Laya to pick from those 20. Over the full 5,500 CLINC records, that dropped accuracy to 31.4%, because the right intent was in the shortlist only 29.8% of the time. So, for Laya, giving it every label was better.

Sentiment: Stars Are Harder Than Polarity

For positive, neutral, or negative, the two models are close: 79.0% for Laya and 83.3% for Ornith.

Star ratings are where Laya falls apart. It picked the exact star count 37.3% of the time, against 59.7% for Ornith. Even more telling, Ornith was within one star 99.0% of the time, and Laya only 65.7%. Laya picked "2 stars" for 193 of the 300 reviews. Of the 69 five-star reviews, it called 52 of them 2 stars. So, it knows roughly whether a review is good or bad, but it has not learned the steps in between.

Log Alerts: The Input Format Matters

This one surprised us. Each log line has a component, a level such as FATAL or INFO, and the message. When we gave both models all three as JSON, Ornith won: 0.910 against 0.802 AUROC.

When we gave them only the message text, Laya jumped to 0.918, and now it beat Ornith's 0.890. Why? The level field misled Laya. Remember, half the normal lines in our set are marked FATAL or FAILURE too. With the level in front of it, Laya scored 95% of the normal FATAL lines as alerts. On the FATAL lines alone, its AUROC was 0.637 with JSON and 0.844 with text only, against 0.866 and 0.905 for Ornith.

One more thing to know: at the default cut-off of 0.5, Laya flagged about three quarters of all normal lines as alerts in both formats. The AUROC shows the ranking is useful, but we must choose our own cut-off before using it.

Which Model Is Faster?

Now the speed. We timed one request at a time: one input with all its questions, the way an app would call it. We also measured how many records per second each model gets through a whole dataset.

Let me tabulate the time per request, as the middle value (p50) and the slow tail (p95):

Here, we can see Laya answers in 8 to 29 ms, and Ornith in 106 to 382 ms. That is 5 to 22 times faster at the middle. The slow tail is even more lopsided. On long Yelp reviews, Ornith's slowest 5% took over a second, and Laya's stayed under 40 ms.

CLINC is the one task where the gap shrinks to 5 times. Laya has to read all 151 labels with every record, so its input grows to about 680 tokens. Ornith reads the label list once and reuses it, so it only reads the new question each time.

For bulk jobs, Laya can also run 32 records at once, which gave it 37 to 163 records a second. Ornith handled 2 to 9 records a second, one at a time.

How Much Memory Does Each One Need?

Laya is small enough to run next to almost anything.

Laya peaked at about 2 GB on every task. Ornith peaked at 10 to 12.3 GB, depending on how long the inputs were. On a 16GB Mac, Ornith would take most of the memory. Laya would barely be noticed.

Can We Trust the Confidence Scores?

A classifier is most useful when its confidence means something. If it says 90%, it should be right about 9 times in 10. Then we can send the confident answers straight through and send the unsure ones to a person.

We measure this with ECE, expected calibration error. We group answers by how confident the model was, and check how far that confidence is from the real hit rate in each group. In simple words, it is the average gap between "how sure it said it was" and "how often it was right". Lower is better, and 0 is perfect.

Here, we can see that on the four harder tasks, Laya's confidence was far from its real hit rate. On support tickets both were already well calibrated, with ECE under 0.05 on the label questions.

There is a small detail here. Laya ships a temperature of 0.1006 for questions with 11 or more labels, but laya-mlx does not allow a value below 0.5, so it uses 0.5 instead. We checked the shipped 0.1006 as well, on the full sets. It was worse than 0.5 on all three questions it applies to: ticket department, ticket intent, and CLINC.

The good news is that this is cheap to fix. We learned a new temperature on 20% of each dataset and tested it on the other 80%. Let me tabulate the ECE before and after:

So, if we use Laya's confidence to decide anything, we fit our own temperature on a few hundred labeled examples first. It does not change which label wins, only how sure the model says it is.

Which Model Should We Use?

Let me tabulate the answer for your better understanding:

A sensible plan is to use both. Let Laya handle the easy, high-volume questions, and send only its unsure cases to the LLM. Laya's scores, once we fit a temperature, are good enough to make that split.

Limits of This Test

This is one laptop and one run per task.

The honest limits:

• For CLINC, Yelp, and logs, both models ran on the same 300 records, not the full sets. A 5-point gap on 300 records is real, but a 1 or 2-point gap could move.
• We used one question wording per task and did not tune it for either model. Better wording might help either one.
• Ornith ran with thinking off. Thinking mode might raise its accuracy, and it would make it much slower.
• Laya's MLX build is a community port. We did not compare it with the original PyTorch model.
• Laya's speed is on 16-bit weights. An 8-bit Laya would be smaller again, and we did not test it.
• Our Ornith timings reuse the shared part of the prompt. Without that reuse, Ornith would be slower than shown.

If you run Laya on your own data, please share your task, your accuracy, and your time per request in the comments.

Conclusion

This is how Laya, a 421M decision model, compares with Ornith 1.5 9B, a normal LLM, on five labeling tasks on a MacBook Pro M5 Max. We started with how Laya answers a question in a single pass without writing any text. We saw it win on short support tickets and on log text, and fall well behind on long complaints, many labels, star ratings, and "none of these" answers. We saw it run 5 to 22 times faster in about a sixth of the memory. Finally, we saw that its confidence needs a new temperature before we can trust it.

Key takeaways:

• Laya was more accurate on support tickets: 99.0% against 84.8% for intent, and 96.2% against 94.8% for department.
• Ornith won complaint routing, CLINC intent, Yelp stars, and Yelp polarity, often by more than 20 points.
• Laya almost never says "none of these". It found out of scope requests only 6.9% of the time.
• Laya answered in 8 to 29 ms and peaked at about 2 GB. Ornith took 106 to 382 ms and 10 to 12.3 GB.
• Learning a temperature on a few hundred labeled examples fixes most of Laya's confidence problem.

Next steps:

• Read the Ornith 1.5 9B vs 35B-A3B benchmark to see how the bigger Ornith compares.
• See Bonsai 2 27B vs Qwen 3.8 27B on a MacBook Pro M5 Max for another test of speed and memory on the same Mac.
• Use the local LLMs technical reference guide to match a model size to your own machine.

In short, Laya is the fast, light choice for short, clean labeling jobs, and a 9B LLM is still the safer choice when the text is long, the labels are many, or the answer might be "none of these".