Skip to main content

LUP Student Papers

LUND UNIVERSITY LIBRARIES

MobileLLM: Comparing Deep-Thin vs Wide-Shallow Architectures for Small Language Models

Cai, Yaliang LU ; Zhou, Chun LU and Zu, Haobo LU (2026) EDAN70 20252
Department of Computer Science
Abstract
Recent research, such as MobileLLM, suggests that "deep-thin" architectures are optimal for sub-billion parameter language models. This work re-examines this design philosophy at the extremely small scale of 60M parameters. We first established a reproduction pipeline using 80M models on RedPajama, and subsequently conducted a systematic comparison of 60M architectures trained on FineWeb-Edu. Contrary to prevailing scaling laws (Hoffmann et al., 2022), our empirical results demonstrate that "wide-shallow" architectures (12–16 layers) tend to outperform "deep-thin" counterparts (30–43 layers) on average at this scale, particularly in reading comprehension tasks. Additionally, we validate the
usability of these micro-models by demonstrating... (More)
Recent research, such as MobileLLM, suggests that "deep-thin" architectures are optimal for sub-billion parameter language models. This work re-examines this design philosophy at the extremely small scale of 60M parameters. We first established a reproduction pipeline using 80M models on RedPajama, and subsequently conducted a systematic comparison of 60M architectures trained on FineWeb-Edu. Contrary to prevailing scaling laws (Hoffmann et al., 2022), our empirical results demonstrate that "wide-shallow" architectures (12–16 layers) tend to outperform "deep-thin" counterparts (30–43 layers) on average at this scale, particularly in reading comprehension tasks. Additionally, we validate the
usability of these micro-models by demonstrating that they can acquire instruction-following capabilities through LoRA fine-tuning, albeit with constrained reasoning depth. (Less)
Popular Abstract (Swedish)
Stora språkmodeller kräver mycket minne och beräkningskraft, vilket gör mindre modeller intressanta för användning på resursbegränsade enheter. I detta arbete undersöker vi hur mycket små språkmodeller med cirka 60 miljoner parametrar bör utformas.
Vi jämför djupa och smala modeller med bredare och grundare modeller. Resultaten visar att en bredare modell med 12 lager gav bäst genomsnittligt resultat, medan en mycket grund modell med 6 lager presterade sämre. Detta tyder på att små modeller behöver en bra balans mellan bredd och djup.
Vi visar också att en 60-miljonersmodell kan vidaretränas för att följa enkla instruktioner och svara på grundläggande frågor, även om mer avancerat resonemang fortfarande är begränsat.
Please use this url to cite or link to this publication:
author
Cai, Yaliang LU ; Zhou, Chun LU and Zu, Haobo LU
supervisor
organization
course
EDAN70 20252
year
type
L3 - Miscellaneous, Projetcs etc.
subject
keywords
Small Language Models, Model Architecture, On-Device AI, LoRA
language
English
id
9251916
date added to LUP
2026-09-27 11:42:18
date last changed
2026-09-27 13:00:08
@misc{9251916,
  abstract     = {{Recent research, such as MobileLLM, suggests that "deep-thin" architectures are optimal for sub-billion parameter language models. This work re-examines this design philosophy at the extremely small scale of 60M parameters. We first established a reproduction pipeline using 80M models on RedPajama, and subsequently conducted a systematic comparison of 60M architectures trained on FineWeb-Edu. Contrary to prevailing scaling laws (Hoffmann et al., 2022), our empirical results demonstrate that "wide-shallow" architectures (12–16 layers) tend to outperform "deep-thin" counterparts (30–43 layers) on average at this scale, particularly in reading comprehension tasks. Additionally, we validate the
usability of these micro-models by demonstrating that they can acquire instruction-following capabilities through LoRA fine-tuning, albeit with constrained reasoning depth.}},
  author       = {{Cai, Yaliang and Zhou, Chun and Zu, Haobo}},
  language     = {{eng}},
  note         = {{Student Paper}},
  title        = {{MobileLLM: Comparing Deep-Thin vs Wide-Shallow Architectures for Small Language Models}},
  year         = {{2026}},
}