Knowledge Distillation for Compact Language Models on Mathematical Reasoning Tasks

Abstract

Large language models (LLMs) have demonstrated exceptional capabilities
in complex mathematical reasoning; however, their massive parameter scales
impose prohibitive computational costs, limiting their deployment in resource constrained or local environments. To address these challenges, this thesis
investigates advanced knowledge distillation methodologies to enhance the
mathematical problem-solving performance of lightweight student models
(0.5 billion and 1.5 billion parameters). We propose a framework that integrates large-scale reasoning data synthesis with classic knowledge distillation.
Leveraging a high-performance teacher model as the generator, we explicitly
produce detailed reasoning trajectories to expand the training corpus while
simultaneously utilizing distribution alignment to match the teacher’s soft
probability targets. Through the design of four comparative training strategies, we systematically evaluate the individual and combined effects of data
synthesis and soft-target distillation. Evaluations on standardized benchmarks (GSM8K and MATH) reveal that our approach effectively mitigates
the capacity bottlenecks inherent in small architectures. The results demonstrate that the proposed framework delivers substantial performance gains
over both the un-tuned base models and the supervised fine-tuning baselines
on targeted reasoning tasks, particularly on highly demanding competitive
mathematics. These findings contribute to the development of efficient, locally deployable mathematical AI assistants under strict resource constraints.

Popular Abstract

Title: Pocketing the Professor: Making Tiny AI Master Hard Math

Introduction:
Imagine a world where you do not need an internet connection to access a
top-tier math assistant. Instead, a tiny Artificial Intelligence, running right
on your smartphone, is ready to help you solve tough equations anytime. We
have successfully broken through the intelligence barrier of small-scale AI.
By using a clever teaching technique, we transferred the reasoning logic of a
massive, supercomputer-sized AI into lightweight devices. Our experiments
prove that these pocket-sized models can stop memorizing answers and start
understanding logic, opening the door for affordable, private, and offline AI
helpers.

Main Text:
Today’s giant artificial intelligence models solve complex math with incredible skill, but they are also digital “power-hungry monsters.” In reality, asking
them for help is like being stuck on a difficult homework problem and having
to hop on an international call to consult a brilliant math professor overseas.
While the professor knows everything, maintaining his supercomputer home
is incredibly expensive. You are also entirely trapped by your internet connection, and your private data might be exposed. Why should our personal,
everyday devices be stuck running slow and unhelpful local software?
To solve this problem, we stepped in as an “AI Coach.” Our goal was to
train a reliable “classmate” who sits right next to you—requiring zero internet—to master complex mathematics. We targeted small student models.
Conventional training methods usually force these small models to blindly
memorize the professor’s final answers word-for-word. This brute-force rote
learning quickly overloads the small model’s limited brain, causing its grades
to plummet due to formatting confusion during actual exams.
Instead, we completely changed the teaching script. First, we made the
professor write down a highly detailed, step-by-step thinking path for every
single problem. More importantly, we used a clever strategy called “mode-seeking” distribution alignment. Rather than forcing the student to mimic every
single wild idea flashing through the professor’s massive brain, we instructed
the student to focus exclusively on imitating the professor’s most stable,
clear, and high-confidence thinking habits. We taught the student how to think confidently, rather than what to memorize.
The results validated the power of this strategy. We sent our pocket-sized
student to sit for two rigorous math examinations: one featuring school-level
word problems, and another packed with challenging competitive mathematics. Compared to the basic models that completely panicked and failed under
exam conditions, our student demonstrated amazing resilience.
If we view these brutal examinations as standard 100-point tests, our specially tutored student model achieved incredible results. In the grade-school
word problem exam, our student’s score jumped by a steady 7 points compared to its un-trained self. Even more impressively, on the highly challenging competitive mathematics exam, it secured a score that was over 8 points
higher than its baseline performance! A highly intriguing detail emerged
from our data: as the math problems became increasingly difficult and complex, the advantage of teaching the student stable thinking habits over raw
memorization became exponentially more pronounced.
The real-world impact of this research means that we can now develop cheap,
efficient, and entirely private offline math assistants. These can be seamlessly
embedded into educational apps for remote schools lacking internet access,
offline scientific calculators, or daily mobile applications to democratize advanced technology.

Project Note: For technical verification, this project systematically executed
four comparative training strategies to isolate and evaluate the independent and
synergistic effects of reasoning data synthesis versus high-dimensional logit distribution alignment during knowledge distillation.

• author: Li, Haoran
• supervisor: Alexandros Sopasakis; Shuzhen Li
• organization: Mathematics (Faculty of Engineering)
• course: FMAM02 20261
• year: 2026
• type: H2 - Master's Degree (Two Years)
• subject: Technology and Engineering
• publication/series: Master's Theses in Mathematical Sciences 2026:E41
• report number: LUTFMA-3623-2026
• ISSN: 1404-6342
• other publication id: E:41
• language: English
• id: 9232626
• date added to LUP: 2026-06-12 15:12:29
• date last changed: 2026-06-12 15:12:29