Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Towards Federated Learning with on-device Training and Communication in 8-bit Floating Point

Wang, Bokun ; Berg, Axel LU orcid ; Acar, Durmus Alp Emre and Zhou, Chuteng (2024) FedKDD: International Joint Workshop on Federated Learning for Data Mining and Graph Analytics
Abstract
Recent work has shown that 8-bit floating point (FP8) can be used for efficiently training neural networks with reduced computational overhead compared to training in FP32/FP16. In this work, we investigate the use of FP8 training in a federated learning context. This brings not only the usual benefits of FP8 which are desirable for on-device training at the edge, but also reduces client-server
communication costs due to significant weight compression. We present a novel method for combining FP8 client training while maintaining a global FP32 server model and provide convergence analysis. Experiments with various machine learning models and datasets show that our method consistently yields communication reductions of at least 2.9x... (More)
Recent work has shown that 8-bit floating point (FP8) can be used for efficiently training neural networks with reduced computational overhead compared to training in FP32/FP16. In this work, we investigate the use of FP8 training in a federated learning context. This brings not only the usual benefits of FP8 which are desirable for on-device training at the edge, but also reduces client-server
communication costs due to significant weight compression. We present a novel method for combining FP8 client training while maintaining a global FP32 server model and provide convergence analysis. Experiments with various machine learning models and datasets show that our method consistently yields communication reductions of at least 2.9x across a variety of tasks and models compared to an FP32 baseline.
(Less)
Please use this url to cite or link to this publication:
author
; ; and
organization
publishing date
type
Contribution to conference
publication status
published
subject
keywords
federated learning, quantization, fp8
conference name
FedKDD: International Joint Workshop on Federated Learning for Data Mining and Graph Analytics
conference location
Barcelona, Spain
conference dates
2024-08-26 - 2024-08-26
language
English
LU publication?
yes
id
a92e2aa8-8bcb-45ae-b2ee-ec7275da1399
alternative location
https://openreview.net/pdf?id=YfGoRu5gR6
date added to LUP
2024-09-03 07:36:40
date last changed
2024-10-03 10:34:32
@misc{a92e2aa8-8bcb-45ae-b2ee-ec7275da1399,
  abstract     = {{Recent work has shown that 8-bit floating point (FP8) can be used for efficiently training neural networks with reduced computational overhead compared to training in FP32/FP16. In this work, we investigate the use of FP8 training in a federated learning context. This brings not only the usual benefits of FP8 which are desirable for on-device training at the edge, but also reduces client-server<br/>communication costs due to significant weight compression. We present a novel method for combining FP8 client training while maintaining a global FP32 server model and provide convergence analysis. Experiments with various machine learning models and datasets show that our method consistently yields communication reductions of at least 2.9x across a variety of tasks and models compared to an FP32 baseline.<br/>}},
  author       = {{Wang, Bokun and Berg, Axel and Acar, Durmus Alp Emre and Zhou, Chuteng}},
  keywords     = {{federated learning; quantization; fp8}},
  language     = {{eng}},
  month        = {{08}},
  title        = {{Towards Federated Learning with on-device Training and Communication in 8-bit Floating Point}},
  url          = {{https://openreview.net/pdf?id=YfGoRu5gR6}},
  year         = {{2024}},
}