Efficient and Scalable Language Models for Training, Inference and Test Time Compute

Diplôme Thèse Efficient and Scalable Language Models for Training, Inference and Test Time Compute eng 2026-02-26 2026-05-08 author Paliotta Danièle 954918 director Fleuret François Jean 766513 Département d'informatique (DINFO) 10.13097/archive-ouverte/unige:195581 urn:nbn:ch:unige-1955814 Sc. 5998 Large Language Models (LLMs) have emerged as the central paradigm of modern AI, but their rapid growth in size and capabilities has created pressing challenges in training efficiency, inference speed, and deployment scalability. This dissertation develops a systematic study of architectural, algorithmic, and system-level techniques to make LLMs more efficient without sacrificing performance. We investigate methods that tackle bottlenecks across three critical stages. For pretraining, we introduce fast attention algorithms and novel solutions to mitigate outlier activations, a key obstacle to effective model quantization. For inference, we accelerate generation through advances in speculative decoding, cross-architecture distillation, and the development of subquadratic and hybrid models. Finally, at test time, we scale reasoning capabilities by designing distilled models that trade additional inference compute for improved accuracy. We provide new insights into the emergence and mitigation of outlier features in Transformers, novel dynamic sparse attention mechanisms with efficient GPU kernels, and a principled framework for distilling hybrid architectures that bridge Transformers and structured state space models such as Mamba. We further demonstrate that scaling inference-time compute with distilled reasoning students can rival or surpass strong Transformers at fixed compute budgets for complex reasoning tasks, pointing toward a more cost-effective trajectory for LLM development. Together, these results provide theoretical and empirical evidence that LLM efficiency can be substantially improved across the training–inference spectrum. Overall, this work advances the design of scalable, more sustainable foundation models that retain the capabilities of current state-of-the-art systems while reducing their computational bottlenecks. Dewey 025.063 32 Département d'informatique 18 false a5ffbfa7-728b-4b3a-8cdd-e4a9475e923b Thesis Public PhD_Dissertation_paliotta.pdf 21324052 application/pdf 02715901a011de04d7ce4d5b7795a479b675292c5e5b1e03a22f65d86aba1da7 f12c7351-74dd-4409-ad46-0626ae584418 Imprimatur Public imprimatur.pdf 26296 application/pdf a5b0ce6e067c6e178527abebf83c35b1b99525c3cadb01e7f6cc5f65cdcbb055 openAccess Faculté des sciences / Département d'informatique

Title	Published in	Access level	OA Policy	Year	Views	Downloads

Efficient and Scalable Language Models for Training, Inference and Test Time Compute				2026	17	134

Archive ouverte UNIGE

Paliotta, Danièle

account_balance UNIGE affiliation entities

UNIGE affiliation entities