We asked Qwen 3.8 by how many basis points AWS operating margin changed from 2023 to 2024. We asked for the number only. It answered 271. The true change is about 990. Then we added one line to the prompt, "Think step by step", and the same model got it right. But here's the real mystery: the model did not learn anything new. It only got room to write.
Chain-of-thought prompting means asking the model to show its working before the final answer. In simple words, we let it do the math on paper instead of in its head. In this blog, we will test a bare answer, a step-by-step prompt and Qwen 3.8's thinking mode on 12 finance questions, and we will count the tokens and seconds each one costs.
Every basis point question failed as a bare number and passed with working shown.
Setup
We use Qwen 3.8 27B in Ollama, with the same answer on every run. This time, the helper also returns how many tokens the model wrote and how many seconds the call took. The think argument lets us turn thinking mode on later. Let's see the code as below:
import re
import ollama
MODEL = "qwen3.8:27b"
def ask(prompt, think=False):
response = ollama.chat(
model=MODEL,
messages=[{"role": "user", "content": prompt}],
think=think,
options={"temperature": 0, "seed": 7},
)
seconds = response.total_duration / 1e9
return response.message.content, response.eval_count, secondsOur Test: 12 Questions From Two 10-K Filings
We take the segment tables from the 2024 10-K filings of Amazon and Alphabet. Then we write 12 questions that need two or more steps of math: margins, changes in basis points, shares, growth rates and ratios. Python computes the true answer for each one, so we never trust the model for the answer key.
tables = """Amazon 10-K, fiscal year 2024 (in millions)
Net sales 2023 2024
North America $ 352,828 $ 387,497
International 131,200 142,906
AWS 90,757 107,556
Consolidated $ 574,785 $ 637,959
Operating income (loss)
North America $ 14,877 $ 24,967
International (2,656) 3,792
AWS 24,631 39,834
Consolidated $ 36,852 $ 68,593
Alphabet 10-K, fiscal year 2024 (in millions)
Revenues 2023 2024
YouTube ads 31,510 36,147
Google advertising 237,855 264,590
Google Services total 272,543 304,930
Google Cloud 33,088 43,229
Total revenues $ 307,394 $ 350,018
Operating income (loss)
Google Services $ 95,858 $ 121,263
Google Cloud 1,716 6,112
Total income from operations $ 84,293 $ 112,390"""
def margin(income, sales):
return income / sales * 100
def bps(i23, s23, i24, s24):
return (margin(i24, s24) - margin(i23, s23)) * 100
questions = [
("By how many basis points did AWS operating margin change from 2023 to 2024?", bps(24631, 90757, 39834, 107556), 1.5),
("By how many basis points did North America operating margin change from 2023 to 2024?", bps(14877, 352828, 24967, 387497), 1.5),
("By how many basis points did Amazon's consolidated operating margin change from 2023 to 2024?", bps(36852, 574785, 68593, 637959), 1.5),
("By how many basis points did Google Cloud operating margin change from 2023 to 2024?", bps(1716, 33088, 6112, 43229), 1.5),
("By how many basis points did Google Services operating margin change from 2023 to 2024?", bps(95858, 272543, 121263, 304930), 1.5),
("By how many basis points did Alphabet's total operating margin change from 2023 to 2024?", bps(84293, 307394, 112390, 350018), 1.5),
("What percent of Amazon's 2024 consolidated operating income came from AWS?", 39834 / 68593 * 100, 0.06),
("What was Google Cloud's operating margin in 2024, in percent?", margin(6112, 43229), 0.06),
("By what percent did Google Cloud revenue grow from 2023 to 2024?", (43229 / 33088 - 1) * 100, 0.06),
("What percent of Google advertising revenue came from YouTube ads in 2024?", 36147 / 264590 * 100, 0.06),
("How many times larger was AWS operating income than Google Cloud operating income in 2024?", 39834 / 6112, 0.02),
("By how many million dollars did Amazon International operating income change from 2023 to 2024?", 3792 - (-2656), 1),
]
for q, truth, tol in questions:
print(f"{truth:9.2f} {q}") 989.61 By how many basis points did AWS operating margin change from 2023 to 2024?
222.66 By how many basis points did North America operating margin change from 2023 to 2024?
434.05 By how many basis points did Amazon's consolidated operating margin change from 2023 to 2024?
895.25 By how many basis points did Google Cloud operating margin change from 2023 to 2024?
459.58 By how many basis points did Google Services operating margin change from 2023 to 2024?
468.80 By how many basis points did Alphabet's total operating margin change from 2023 to 2024?
58.07 What percent of Amazon's 2024 consolidated operating income came from AWS?
14.14 What was Google Cloud's operating margin in 2024, in percent?
30.65 By what percent did Google Cloud revenue grow from 2023 to 2024?
13.66 What percent of Google advertising revenue came from YouTube ads in 2024?
6.52 How many times larger was AWS operating income than Google Cloud operating income in 2024?
6448.00 By how many million dollars did Amazon International operating income change from 2023 to 2024?A basis point is one hundredth of a percentage point. So a basis point question needs four steps: two margins, a difference, and a change of units. We allow a small error for each kind of answer: 1.5 basis points, 0.06 percentage points, 0.02 for the ratio and 1 for the dollar amount.
Next, we write a scorer. It reads the number after "Final answer:", compares it with the truth, and adds up the tokens and seconds:
def number(text):
tail = text.split("Final answer:")[-1]
found = re.findall(r"-?\d[\d,]*\.?\d*", tail)
return float(found[0].replace(",", "")) if found else None
def score(make_prompt, think=False):
right, tokens, seconds, rows = 0, 0, 0.0, []
for q, truth, tol in questions:
text, n, s = ask(make_prompt(q), think=think)
value = number(text)
ok = value is not None and abs(value - truth) <= tol
right += ok
tokens += n
seconds += s
rows.append((value, n, ok))
return right, tokens, seconds, rows
def report(name, result):
right, tokens, seconds, _ = result
print(f"{name:16} right {right:2}/12 output tokens {tokens:6} time {seconds:6.1f} s")What Happens When We Ask for the Number Only?
First, the bare answer. We ask for the final number only, so the model has no room to show working. Let's test it:
only = lambda q: f"{tables}\n\n{q} Reply with the final number only."
asked_result = score(only)
report("number only", asked_result)
for (q, truth, tol), (value, n, ok) in zip(questions, asked_result[3]):
print(f" {'ok ' if ok else 'BAD'} {n:4} tokens answer {value} (true {truth:.2f})")number only right 5/12 output tokens 59 time 3.8 s
BAD 4 tokens answer 271.0 (true 989.61)
BAD 4 tokens answer 243.0 (true 222.66)
BAD 4 tokens answer 101.0 (true 434.05)
BAD 7 tokens answer 10188.0 (true 895.25)
BAD 4 tokens answer 106.0 (true 459.58)
BAD 4 tokens answer 213.0 (true 468.80)
ok 6 tokens answer 58.07 (true 58.07)
ok 6 tokens answer 14.14 (true 14.14)
BAD 5 tokens answer 24.6 (true 30.65)
ok 6 tokens answer 13.66 (true 13.66)
ok 4 tokens answer 6.5 (true 6.52)
ok 5 tokens answer 6448.0 (true 6448.00)Here, we can see a clear split:
- All six basis point questions were wrong. AWS came back as 271 instead of about 990, and Google Cloud as 10188 instead of about 895.
- Five of the six one-step questions were right. The miss was Google Cloud revenue growth: 24.6% instead of 30.65%.
Each answer took only 4 to 7 tokens. That is the problem. A model writes one token at a time, and each token gets the same fixed amount of computation. A four-step calculation squeezed into a few tokens has nowhere to happen. So the model wrote a number that looked right instead of one that was right.
What Is Chain-of-Thought Prompting?
So, here comes chain-of-thought prompting to the rescue. We add a line that asks the model to think step by step and show each calculation. Every number it writes down becomes part of the prompt for the next token, as we saw in Prompt Anatomy. In simple words, the working on the page is the model's scratch paper.
We also tell it not to round until the last step. In Prompt Chaining, early rounding cost us three basis point answers. Let's test it:
step = lambda q: (f"{tables}\n\n{q} Think step by step and show each calculation. "
"Do not round until the last step. End with 'Final answer: <number>'.")
step_result = score(step)
report("step by step", step_result)step by step right 12/12 output tokens 7441 time 45.4 sHere, we can see all 12 answers right. The cost is easy to see as well: the bare answers used 59 tokens in total, and the step-by-step answers used 7,441.
Let's read one answer to see what the model wrote:
text, n, s = ask(step(questions[1][0]))
print(text)To determine the change in North America operating margin for Amazon in basis points, we need to calculate the operating margin for both 2023 and 2024, find the difference, and convert that difference into basis points.
**Step 1: Identify the relevant data for Amazon North America**
From the provided Amazon 10-K data:
* **2023:**
* Net Sales (North America): $352,828 million
* Operating Income (North America): $14,877 million
* **2024:**
* Net Sales (North America): $387,497 million
* Operating Income (North America): $24,967 million
**Step 2: Calculate the Operating Margin for 2023**
The formula for operating margin is:
$ \text{Operating Margin} = \frac{\text{Operating Income}}{\text{Net Sales}} $
$ \text{Margin}_{2023} = \frac{14,877}{352,828} $
$ \text{Margin}_{2023} \approx 0.042164986 $
**Step 3: Calculate the Operating Margin for 2024**
$ \text{Margin}_{2024} = \frac{24,967}{387,497} $
$ \text{Margin}_{2024} \approx 0.064432166 $
**Step 4: Calculate the change in Operating Margin**
$ \text{Change} = \text{Margin}_{2024} - \text{Margin}_{2023} $
$ \text{Change} = 0.064432166 - 0.042164986 $
$ \text{Change} \approx 0.02226718 $
**Step 5: Convert the change to basis points**
One basis point (bp) is equal to 0.01% or 0.0001 in decimal form. To convert a decimal change to basis points, multiply by 10,000.
$ \text{Change in bps} = 0.02226718 \times 10,000 $
$ \text{Change in bps} = 222.6718 $
Rounding to two decimal places (standard for basis points in financial reporting contexts, though the prompt asks not to round until the last step, we will present the precise calculated value rounded reasonably for the final answer format):
$ 222.67 \text{ bps} $
Let's double-check the calculation without intermediate rounding:
$ \left( \frac{24,967}{387,497} - \frac{14,877}{352,828} \right) \times 10,000 $
$ \frac{24,967}{387,497} \approx 0.0644321663 $
$ \frac{14,877}{352,828} \approx 0.0421649861 $
$ \text{Difference} \approx 0.0222671802 $
$ \text{Basis Points} \approx 222.671802 $
Final answer: 222.67Here, we can see the four steps we described: the 2023 margin, the 2024 margin, the difference, and the change to basis points. The model even checked its result once more without rounding. Each small step is easy, and each one is written down before the next one starts.
The bare answer skipped the steps and missed by about 20 basis points.
How Is Thinking Mode Different?
Qwen 3.8 also has a thinking mode. With think=True, the model writes its reasoning in a separate thinking section before the answer. Ollama returns that section apart from the reply, so the user only sees the answer. We do not need to ask for steps in the prompt at all. Let's test it:
plain = lambda q: f"{tables}\n\n{q} End with 'Final answer: <number>'."
think_result = score(plain, think=True)
report("thinking mode", think_result)thinking mode right 12/12 output tokens 18728 time 101.8 sHere, we can see 12 right answers again. But the thinking took 18,728 tokens, about two and a half times as many as the step-by-step prompt. Thinking mode checks and rechecks its work, and on these questions that extra care did not change the score.
Let me tabulate the three runs for your better understanding:
| Approach | Right | Output tokens | Time |
|---|---|---|---|
| Number only | 5 of 12 | 59 | 3.8 s |
| Think step by step | 12 of 12 | 7,441 | 45.4 s |
| Thinking mode | 12 of 12 | 18,728 | 101.8 s |
Every right answer in the hard group came with working written down.
When Is Chain-of-Thought Worth the Cost?
Tokens are not free. On a local model they cost time, and on a paid API they cost money. So the question is not "does it help?" but "does this task need it?"
| Task | Bare answer is fine? | Why |
|---|---|---|
| Copy a number from the 10-K | Yes | No calculation to do |
| One division, like a margin | Mostly | Five of six right in our test |
| Several steps, like a change in basis points | No | Zero of six right as a bare number |
| A report that people will act on | No | Use steps, or better, do the math in code |
There is one more idea worth knowing. Self-consistency means asking the same question several times at a higher temperature and taking the most common answer. In Temperature, Top-k and Top-p, we saw that scattered answers are a sign of guessing. We did not need it here, since step by step already got every answer right, but it is a useful extra check for harder problems.
Finally, the step-by-step working is a help, not a proof. For numbers that matter, the best path is still the one from lesson 4: let the model read and write, and let Python do the arithmetic.
Recap
This is how chain-of-thought prompting works. We started with a bare-number prompt, and Qwen 3.8 missed every basis point question, because a few tokens left no room for four steps of math. Then we asked it to think step by step, and all 12 answers came out right for about 7,400 tokens. Finally, thinking mode also got 12 of 12, but it spent about two and a half times as many tokens to get there.
In the next lesson, we will look at structured output, and how to make the model return JSON that our code can always read.
