Federated Gradient Averaging for Multi-Site Training with Momentum-Based Optimizers

Samuel W. Remedios; John A. Butman; Bennett A. Landman; Dzung L. Pham

doi:10.1007/978-3-030-60548-3_17

Federated Gradient Averaging for Multi-Site Training with Momentum-Based Optimizers

Samuel W. Remedios^*, John A. Butman, Bennett A. Landman, Dzung L. Pham

^*Corresponding author for this work

Radiology and Bioengineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

13 Scopus citations

Abstract

Multi-site training methods for artificial neural networks are of particular interest to the medical machine learning community primarily due to the difficulty of data sharing between institutions. However, contemporary multi-site techniques such as weight averaging and cyclic weight transfer make theoretical sacrifices to simplify implementation. In this paper, we implement federated gradient averaging (FGA), a variant of federated learning without data transfer that is mathematically equivalent to single site training with centralized data. We evaluate two scenarios: a simulated multi-site dataset for handwritten digit classification with MNIST and a real multi-site dataset with head CT hemorrhage segmentation. We compare federated gradient averaging to single site training, federated weight averaging (FWA), and cyclic weight transfer. In the MNIST task, we show that training with FGA results in a weight set equivalent to centralized single site training. In the hemorrhage segmentation task, we show that FGA achieves on average superior results to both FWA and cyclic weight transfer due to its ability to leverage momentum-based optimization.

Original language	English
Title of host publication	Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning - 2nd MICCAI Workshop, DART 2020, and 1st MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Proceedings
Editors	Shadi Albarqouni, Spyridon Bakas, Konstantinos Kamnitsas, M. Jorge Cardoso, Bennett Landman, Wenqi Li, Fausto Milletari, Nicola Rieke, Holger Roth, Daguang Xu, Ziyue Xu
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	170-180
Number of pages	11
ISBN (Print)	9783030605476
DOIs	https://doi.org/10.1007/978-3-030-60548-3_17
State	Published - 2020
Event	2nd MICCAI Workshop on Domain Adaptation and Representation Transfer, DART 2020, and the 1st MICCAI Workshop on Distributed and Collaborative Learning, DCL 2020, held in conjunction with the Medical Image Computing and Computer Assisted Intervention, MICCAI 2020 - Lima, Peru Duration: 4 Oct 2020 → 8 Oct 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12444 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	2nd MICCAI Workshop on Domain Adaptation and Representation Transfer, DART 2020, and the 1st MICCAI Workshop on Distributed and Collaborative Learning, DCL 2020, held in conjunction with the Medical Image Computing and Computer Assisted Intervention, MICCAI 2020
Country/Territory	Peru
City	Lima
Period	4/10/20 → 8/10/20

Keywords

Deep learning
Federated learning
Multi-site

Access to Document

10.1007/978-3-030-60548-3_17

Cite this

Remedios, S. W., Butman, J. A., Landman, B. A., & Pham, D. L. (2020). Federated Gradient Averaging for Multi-Site Training with Momentum-Based Optimizers. In S. Albarqouni, S. Bakas, K. Kamnitsas, M. J. Cardoso, B. Landman, W. Li, F. Milletari, N. Rieke, H. Roth, D. Xu, & Z. Xu (Eds.), Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning - 2nd MICCAI Workshop, DART 2020, and 1st MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Proceedings (pp. 170-180). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12444 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-60548-3_17

Remedios, Samuel W. ; Butman, John A. ; Landman, Bennett A. et al. / Federated Gradient Averaging for Multi-Site Training with Momentum-Based Optimizers. Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning - 2nd MICCAI Workshop, DART 2020, and 1st MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Proceedings. editor / Shadi Albarqouni ; Spyridon Bakas ; Konstantinos Kamnitsas ; M. Jorge Cardoso ; Bennett Landman ; Wenqi Li ; Fausto Milletari ; Nicola Rieke ; Holger Roth ; Daguang Xu ; Ziyue Xu. Springer Science and Business Media Deutschland GmbH, 2020. pp. 170-180 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{57c884e751c54177b0f975bb1c2428fc,

title = "Federated Gradient Averaging for Multi-Site Training with Momentum-Based Optimizers",

abstract = "Multi-site training methods for artificial neural networks are of particular interest to the medical machine learning community primarily due to the difficulty of data sharing between institutions. However, contemporary multi-site techniques such as weight averaging and cyclic weight transfer make theoretical sacrifices to simplify implementation. In this paper, we implement federated gradient averaging (FGA), a variant of federated learning without data transfer that is mathematically equivalent to single site training with centralized data. We evaluate two scenarios: a simulated multi-site dataset for handwritten digit classification with MNIST and a real multi-site dataset with head CT hemorrhage segmentation. We compare federated gradient averaging to single site training, federated weight averaging (FWA), and cyclic weight transfer. In the MNIST task, we show that training with FGA results in a weight set equivalent to centralized single site training. In the hemorrhage segmentation task, we show that FGA achieves on average superior results to both FWA and cyclic weight transfer due to its ability to leverage momentum-based optimization.",

keywords = "Deep learning, Federated learning, Multi-site",

author = "Remedios, {Samuel W.} and Butman, {John A.} and Landman, {Bennett A.} and Pham, {Dzung L.}",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 2nd MICCAI Workshop on Domain Adaptation and Representation Transfer, DART 2020, and the 1st MICCAI Workshop on Distributed and Collaborative Learning, DCL 2020, held in conjunction with the Medical Image Computing and Computer Assisted Intervention, MICCAI 2020 ; Conference date: 04-10-2020 Through 08-10-2020",

year = "2020",

doi = "10.1007/978-3-030-60548-3_17",

language = "English",

isbn = "9783030605476",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "170--180",

editor = "Shadi Albarqouni and Spyridon Bakas and Konstantinos Kamnitsas and Cardoso, {M. Jorge} and Bennett Landman and Wenqi Li and Fausto Milletari and Nicola Rieke and Holger Roth and Daguang Xu and Ziyue Xu",

booktitle = "Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning - 2nd MICCAI Workshop, DART 2020, and 1st MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Proceedings",

}

Remedios, SW, Butman, JA, Landman, BA & Pham, DL 2020, Federated Gradient Averaging for Multi-Site Training with Momentum-Based Optimizers. in S Albarqouni, S Bakas, K Kamnitsas, MJ Cardoso, B Landman, W Li, F Milletari, N Rieke, H Roth, D Xu & Z Xu (eds), Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning - 2nd MICCAI Workshop, DART 2020, and 1st MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12444 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 170-180, 2nd MICCAI Workshop on Domain Adaptation and Representation Transfer, DART 2020, and the 1st MICCAI Workshop on Distributed and Collaborative Learning, DCL 2020, held in conjunction with the Medical Image Computing and Computer Assisted Intervention, MICCAI 2020, Lima, Peru, 4/10/20. https://doi.org/10.1007/978-3-030-60548-3_17

Federated Gradient Averaging for Multi-Site Training with Momentum-Based Optimizers. / Remedios, Samuel W.; Butman, John A.; Landman, Bennett A. et al.
Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning - 2nd MICCAI Workshop, DART 2020, and 1st MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Proceedings. ed. / Shadi Albarqouni; Spyridon Bakas; Konstantinos Kamnitsas; M. Jorge Cardoso; Bennett Landman; Wenqi Li; Fausto Milletari; Nicola Rieke; Holger Roth; Daguang Xu; Ziyue Xu. Springer Science and Business Media Deutschland GmbH, 2020. p. 170-180 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12444 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Federated Gradient Averaging for Multi-Site Training with Momentum-Based Optimizers

AU - Remedios, Samuel W.

AU - Butman, John A.

AU - Landman, Bennett A.

AU - Pham, Dzung L.

PY - 2020

Y1 - 2020

N2 - Multi-site training methods for artificial neural networks are of particular interest to the medical machine learning community primarily due to the difficulty of data sharing between institutions. However, contemporary multi-site techniques such as weight averaging and cyclic weight transfer make theoretical sacrifices to simplify implementation. In this paper, we implement federated gradient averaging (FGA), a variant of federated learning without data transfer that is mathematically equivalent to single site training with centralized data. We evaluate two scenarios: a simulated multi-site dataset for handwritten digit classification with MNIST and a real multi-site dataset with head CT hemorrhage segmentation. We compare federated gradient averaging to single site training, federated weight averaging (FWA), and cyclic weight transfer. In the MNIST task, we show that training with FGA results in a weight set equivalent to centralized single site training. In the hemorrhage segmentation task, we show that FGA achieves on average superior results to both FWA and cyclic weight transfer due to its ability to leverage momentum-based optimization.

AB - Multi-site training methods for artificial neural networks are of particular interest to the medical machine learning community primarily due to the difficulty of data sharing between institutions. However, contemporary multi-site techniques such as weight averaging and cyclic weight transfer make theoretical sacrifices to simplify implementation. In this paper, we implement federated gradient averaging (FGA), a variant of federated learning without data transfer that is mathematically equivalent to single site training with centralized data. We evaluate two scenarios: a simulated multi-site dataset for handwritten digit classification with MNIST and a real multi-site dataset with head CT hemorrhage segmentation. We compare federated gradient averaging to single site training, federated weight averaging (FWA), and cyclic weight transfer. In the MNIST task, we show that training with FGA results in a weight set equivalent to centralized single site training. In the hemorrhage segmentation task, we show that FGA achieves on average superior results to both FWA and cyclic weight transfer due to its ability to leverage momentum-based optimization.

KW - Deep learning

KW - Federated learning

KW - Multi-site

UR - http://www.scopus.com/inward/record.url?scp=85092175621&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-60548-3_17

DO - 10.1007/978-3-030-60548-3_17

M3 - Conference contribution

AN - SCOPUS:85092175621

SN - 9783030605476

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 170

EP - 180

BT - Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning - 2nd MICCAI Workshop, DART 2020, and 1st MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Proceedings

A2 - Albarqouni, Shadi

A2 - Bakas, Spyridon

A2 - Kamnitsas, Konstantinos

A2 - Cardoso, M. Jorge

A2 - Landman, Bennett

A2 - Li, Wenqi

A2 - Milletari, Fausto

A2 - Rieke, Nicola

A2 - Roth, Holger

A2 - Xu, Daguang

A2 - Xu, Ziyue

PB - Springer Science and Business Media Deutschland GmbH

T2 - 2nd MICCAI Workshop on Domain Adaptation and Representation Transfer, DART 2020, and the 1st MICCAI Workshop on Distributed and Collaborative Learning, DCL 2020, held in conjunction with the Medical Image Computing and Computer Assisted Intervention, MICCAI 2020

Y2 - 4 October 2020 through 8 October 2020

ER -

Remedios SW, Butman JA, Landman BA, Pham DL. Federated Gradient Averaging for Multi-Site Training with Momentum-Based Optimizers. In Albarqouni S, Bakas S, Kamnitsas K, Cardoso MJ, Landman B, Li W, Milletari F, Rieke N, Roth H, Xu D, Xu Z, editors, Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning - 2nd MICCAI Workshop, DART 2020, and 1st MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 170-180. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-60548-3_17