Transformer Based Next Generation Wireless Communication Receiver

Abstract

The Integration of Machine Learning in OFDM Receivers

The integration of Machine Learning (ML) into various fields has prompted exploration into the effectiveness of advanced ML architectures within Orthogonal Frequency Division Multiplexing (OFDM) receivers. Prior work by Nvidia demonstrated the promise of Graph Neural Networks (GNNs) in replacing key components of a traditional receiver, surpassing conventional methods such as Least Squares (LS) channel estimation and Linear Minimum Mean Square Error (LMMSE) combined with K-Best detection. This thesis extends that line of inquiry by investigating the transformer architecture as an alternative base for an AI-based OFDM receiver. Various transformer configurations are designed and optimized, and their performance is compared against classical baselines including LMMSE + Maximum Likelihood Detection (MLD), perfect Channel State Information (CSI) + MLD, and the Nvidia GNN model. The study explores the following main topics:





Comparison of transformer, GNN, and baselines. Results show that the transformer achieves a gain of up to 0.3 dB at 10% BLER compared to the GNN, and a gain of approximately 1 dB at 10% BLER compared to the LMMSE + MLD baseline.



The effect of different channel estimation strategies as initial input to the system, where tested methods include LS estimation, LMMSE estimation, and a separately trained transformer model solely for channel estimation. Results show no major performance difference in BLER between methods, allowing the simplest strategy, LS channel estimation, to be used without loss of performance.



The effect of Positional Encoding (PE) as input to the system. Results show a slight gain for certain SNR values, e.g., 0.4 dB at 10% BLER for both transformer and GNN.



The impact of varied and fixed Signal-To-Noise Ratio (SNR) during training. Results show similar performance for the specified SNR, but a significant loss for lower SNR values, reaching upwards of 2 dB at 40% BLER.



The effect of different channel evaluation methods. Comparisons of Mean Square Error (MSE) loss, Squared Generalized Cosine Similarity (SGCS) loss, and no channel estimation loss showed no major difference in BLER performance. Channel estimation loss was still retained in models, however, due to stability advantages presented by Nvidia.



The impact of different Demodulation Reference Signal (DMRS) patterns. Results show that different DMRS patterns can impact the BLER performance of the models. A DMRS pattern with high temporal density gave a larger separation of the GNN and transformer models’ BLER performance than a DMRS pattern with higher frequency density, showcasing how the different models can leverage the DMRS pattern in different ways.



The potential of Iterative Channel Estimation Detection (ICED). Results show that ICED can be used to improve channel estimation, but improvement in BLER performance was not observed.



The effect of power boosting of DMRS symbols. Results show that power boosting of DMRS symbols can improve the Uncoded Bit Error Rate (UcBER) of the transformer to the point where it approaches perfect CSI + MLD. The same was, however, not observed for BLER or Channel Estimation error (CE error), where it reached saturation.



The generalization capability of the model across different channel types. Results show that the transformer generalizes well to channels of similar characteristics but can incur significant loss when channels differ greatly. This is especially evident for models trained on a channel with low Doppler and frequency selectivity, such as EPA5, and tested on a channel with high Doppler and frequency selectivity, such as ETU100. In this case, the model saw a loss of up to 3 dB at 20% BLER.

Experimental results show that the proposed transformer-based receivers outperform both the LMMSE + MLD baseline and the NVIDIA GNN in the SISO case, demonstrating the potential of attention-based models in the future of wireless signal processing.

Popular Abstract

With recent developments in artificial intelligence, specifically in fields such as text generation and large language models, a natural question was whether this technology could be applied within the field of wireless communication to improve data transmission. As such, this thesis explores this topic by implementing the transformer model - which is the leading model used by large language models - for parts of the receiver in a 5G system, replacing traditional modules, showing that it has the potential to outperform traditional methods.

Various versions of the transformer model were designed and tested against traditional methods. This showed that the transformer model has the potential to outperform certain traditional methods by a large margin. The transformer model was also compared with a previously existing implementation of an artificial intelligence based 5G receiver designed by Nvidia based on another type of model. It was shown that the transformer could outperform Nvidia's implementation, with the degree of improvement depending on the system conditions. The thesis also explores different methods to improve the performance of the transformer as well as how the model behaves in different environments, such as if the receiver is in an urban setting, if the receiver is traveling at the speed of a pedestrian, or if the receiver is traveling at the speed of a vehicle. This test showed that the transformer has the potential to perform well in environments other than those for which it was optimized, with results depending on the difference in the environment.

An interesting discovery during the optimization process, where a discrepancy between different performance metrics was found, resulted in the conclusion that artificial intelligence models, such as the transformer model, may require modified coding schemes - which is the method of which data is processed in order to perform error mitigation - due to a bias in the output of the model, stemming from how the performance of the model is evaluated when training the model.

In the end, the transformer was concluded to have the potential to outperform traditional methods, but specialized tests to evaluate the efficiency of the model as well as a further expansion into larger systems would be necessary before a physical implementation should be considered.

• author: Mo, Fan and Lendrop, Oscar
• supervisor: Aleksei Fedorov; Galina Sidorenko
• organization: Department of Electrical and Information Technology
• course: EITM02 20251
• year: 2025
• type: H2 - Master's Degree (Two Years)
• subject: Technology and Engineering
• keywords: Transformer, Machine Learning, Graphical Neural Network, Artificial Intelligence, Orthogonal Frequency Division Multiplexing, 5G, Wireless Communication
• report number: LU/LTH-EIT 2025-1067
• language: English
• id: 9198959
• date added to LUP: 2025-06-17 15:28:10
• date last changed: 2025-06-17 15:28:10