Laya 421M vs Qwen3.5 0.8B vs Qwen2.5 0.5B: Small Models for Text Classification

Is a 421M classifier better than a small LLM for sorting text? We tested it. On support tickets, Laya picked the right intent 99% of the time. Qwen3.5 0.8B, a model about twice its size, got 45%.

Laya from ConvAI Innovations is not a chat model. We give it some text and a question with a fixed set of answers, and it scores every answer in one pass. So we put it against two LLMs of about the same size doing the same job: Qwen3.5 0.8B and Qwen2.5 0.5B.

In this blog, we will learn how Laya works, how the three models compare on five real labeling tasks, and what each one costs in speed and memory. Everything runs on one MacBook Pro M5 Max with 36GB of memory, using MLX.

What Is Laya?

Laya is a decision model. An LLM writes an answer one word at a time. Laya reads the text and the question once, and scores every possible answer at the same moment. In simple words, it fills in a multiple choice sheet instead of writing an essay.

It is a ModernBERT-large encoder with a small decision head on top, 421M parameters in total. It answers three kinds of questions: choice (pick one label), score (pick a level, like 1 to 5 stars), and noul (yes or no). Let's see how we ask it a question as below:

import laya_mlx as laya

agent = laya.load("aac6fef/laya-mlx", dtype="float16")

questions = {
    "department": {
        "type": "choice",
        "instructions": "Which support department should handle the customer's `message`?",
        "criteria": {
            "refund": "refund policy, requesting or tracking a refund",
            "order": "place, change, cancel or track an order",
            "payment": "payment methods and problems paying",
        },
    },
    "refund_request": {
        "type": "noul",
        "instructions": "Is the customer asking to get money back or checking on a refund they requested?",
    },
}

out = agent.predict({"message": "I paid twice for order 5521, please send one payment back"}, questions)
print(out["answers"])

• criteria lists the allowed answers, with a short description next to each label.
• agent.predict(...) takes the input as a dictionary and answers every question in one call.

The output is as below:

{'department': {'type': 'choice', 'confidence': 0.5082, 'choice': 'payment', 'probabilities': {'refund': 0.1244, 'order': 0.0407, 'payment': 0.8349}, ...}, 'refund_request': {'type': 'noul', 'confidence': 0.7065, 'noul': 0.7065, ...}}

Here, we can see both answers came back from one call, with a probability for every label. There is no text to parse, and no way to answer with a label that is not on the list.

Our Test Setup

Let me tabulate the three models for your better understanding:

Every model saw the same records, the same question wording, and the same JSON input. None was trained on our data, so this is zero-shot for all three.

We made the LLMs as strong and as fast as we could. Qwen3.5 ran with thinking off. Each LLM could only answer with a label from the list, because we limited its output token by token. This is called constrained decoding. For yes or no questions and star ratings, we read the probabilities of its very first token. We also reused the shared part of each prompt across records, so the LLM timings count only the new text.

The Five Tasks

We picked five jobs that teams really use a classifier for:

The last three use a fixed sample of 300 records, the same 300 for every model. In the log set, half the normal lines also carry a FATAL or FAILURE level, so a model cannot just look for scary words.

Which Model Gives Better Answers?

For label questions we report accuracy. For yes or no questions we report AUROC: if we pick one real "yes" and one real "no", how often does the model score the "yes" higher? 1.0 is perfect and 0.5 is a coin flip.

Let me tabulate the results:

Here, we can see Laya is about twice as accurate as Qwen3.5 0.8B on tickets, complaints, and CLINC, and further ahead of Qwen2.5 0.5B. Qwen3.5 0.8B wins on star ratings and on log alerts with JSON input. Qwen2.5 0.5B is close to a coin flip on the logs.

Why the Small LLMs Fall Behind

They keep picking the same label. Qwen2.5 0.5B sent 978 of the 1,100 tickets to contact, and 1,686 of the 1,800 complaints to credit_card. Qwen3.5 0.8B sent 1,393 complaints to debt_collection. With nine complaint labels, always saying one of them gets about 11%, and that is where Qwen2.5 landed.

Laya does not have this problem, because it scores every label against the text instead of writing one out.

Where Laya Falls Short

Laya is not better everywhere.

Let's see where it struggles:

• Long complaints. Laya reads at most 512 tokens, and 22.5% of the complaints were longer. The LLMs read up to 3,000.
• "None of these". On CLINC, Laya said "out of scope" correctly only 6.9% of the time. It almost always forced a real intent onto a request that had none.
• Star ratings. Laya called 52 of the 69 five-star reviews "2 stars". Qwen3.5 0.8B was within one star 93% of the time.
• The log level field. With the level in the JSON, Laya scored 95% of the normal FATAL lines as alerts. Given the message text only, it jumped from 0.802 to 0.918 AUROC and beat both LLMs.

Which Model Is Faster?

We timed one request at a time, one input with all its questions, the way an app would call the model. We also measured records per second over a whole dataset.

Let me tabulate the middle time (p50) per request in ms:

Here, we can see Laya was 1.2 to 6.5 times faster on every task except CLINC. The gap is biggest on long inputs like complaints, and smallest on short log lines.

On CLINC, Qwen2.5 0.5B was faster: 21.0 ms against 28.6. Laya has to read all 151 labels with every record, so its input grows to about 680 tokens. The LLMs read the label list once and reuse it.

For bulk jobs, Laya can run 32 records at once, which gave it 37 to 163 records a second. The LLMs, one at a time, did 6 to 84.

How Much Memory Does Each One Need?

We measured the weights on disk and the highest memory use across all six task runs:

All three fit in 2 to 4 GB, and Laya is the lightest on both counts. So memory is not what sets these models apart. Accuracy is.

Can We Trust the Confidence Scores?

A classifier's confidence is useful when 90% confident means right about 9 times in 10. Then we can send confident answers straight through and unsure ones to a person. We measure the gap with ECE, expected calibration error. In simple words, it is the average gap between how sure the model said it was and how often it was right. Lower is better.

Here, we can see that none of the three is well calibrated out of the box. On support tickets, Laya was already honest, with an ECE of 0.017 on department against 0.273 and 0.627 for the Qwens.

For Laya the rest is cheap to fix. A new temperature learned on 20% of each dataset cut its complaint ECE from 0.207 to 0.029, and its star rating ECE from 0.400 to 0.085. So, before we use Laya's confidence to route work, we fit our own temperature on a few hundred labeled examples.

Which Model Should We Use?

Let me tabulate the answer:

We would not use Qwen2.5 0.5B for classification. It collapsed onto one label on two tasks and was near a coin flip on the logs.

Limits of This Test

This is one laptop and one run per task:

• Three tasks use 300 records. A 5-point gap is real, but a 1 or 2-point gap could move.
• We used one question wording per task and did not tune it for any model.
• Qwen3.5 ran with thinking off. Thinking might help it, and would make it much slower.
• Laya's MLX build is a community port. We did not compare it with the original PyTorch model.
• The LLM timings reuse the shared part of the prompt. Without that, they would be slower.

If you run Laya on your own data, please share your task, your accuracy, and your time per request in the comments.

Conclusion

This is how Laya, a 421M decision model, compares with Qwen3.5 0.8B and Qwen2.5 0.5B on five labeling tasks on a MacBook Pro M5 Max. At about the same size, Laya was far more accurate on tickets, complaints, and many-label intent, because the small LLMs kept picking the same label. It was faster on five of six tasks and used the least memory. Qwen3.5 0.8B still beat it on star ratings and JSON log alerts.

Key takeaways:

• Laya beat both LLMs on tickets, complaints, and CLINC, often by 2 times or more.
• Small LLMs collapse onto one label. Qwen2.5 0.5B sent 94% of complaints to credit_card.
• Laya answered in 8.5 to 29 ms and peaked at 2.13 GB, the lowest of the three.
• Laya almost never says "none of these", and it cuts long inputs at 512 tokens.
• A temperature fitted on a few hundred labeled examples fixes most of Laya's confidence problem.

Next steps:

• See Bonsai 2 27B vs Qwen 3.8 27B on a MacBook Pro M5 Max for another speed and memory test on the same Mac.
• Try sentiment classification with DistilBERT to fine-tune a small encoder on your own labels.
• Use the local LLMs technical reference guide to match a model size to your own machine.

In short, when the model has to be small, Laya is the better pick for short, clean labeling jobs, and a small LLM only makes sense for fine grades like star ratings.