FLBench: A Comprehensive Experimental Evaluation of Federated Learning Frameworks

Title FLBench: A Comprehensive Experimental Evaluation of Federated Learning Frameworks
Summary Exploring Federated Learning Frameworks
Keywords Federated Learning
Prerequisites - DevOps: Good installation and configuration.   

- Programming: Pytorch, and Python. 

Supervisor Sadi Alawadi, Jens Lundström
Level Master
Status Open

The advent of distributed Machine Learning (ML) promoted sophisticated analytics at the network's edge. This decentralized and large-scale ML architecture is known as Federated Learning (FL). FL aims to enable multiple actors to build a common and robust ML model over multiple local dataset. Furthermore, the new wave of FL frameworks promotes data privacy, security, access rights, and access to heterogeneous data. However, the variety of these frameworks require an experimental evaluation of performance analysis. Therefore, this project aims to analyze, evaluate, compare and conclude the popular federated learning frameworks extensively (A similar comparison paper can be found here: https://link.springer.com/content/pdf/10.1007/s10586-021-03240-4.pdf ). The main intended tasks of this project are: 

Evaluating the following open-source federated learning frameworks.

Paddle Federated Learning Framework (https://github.com/PaddlePaddle/PaddleFL) 

PySyft Framework /pygrid (https://github.com/OpenMined/PySyft  ) 

Flower (https://github.com/adap/flower)

TensorFlow FL (https://github.com/tensorflow/federated )

FEDn (https://github.com/scaleoutsystems/fedn )

Intel FL (https://github.com/intel/openfl)

FATE (https://github.com/FederatedAI/FATE  )

Benchmarking suit (Experiment design), i.e., network architecture (the ML model, e.g., LSTM, CNN, etc.), the used datasets, benchmark tool/framework. (Mnist, Cifer10 & 100, IMDB, for IoT data CASA activity recognition).  

Theoretically comparing the federated algorithm they support (FedAVG, FedProx, etc.), cross-device and cross-silo, horizontal and vertical federated learning. Also, open-source, Diversified Computing Paradigms (Standalone simulation, Distributed computing, on-device training), ML heterogeneity (Pytorch, TensorFlow, MXnet,...etc.), development coding language, Framework's timeline.  

Use of existing and development of comparison criteria, performance (task per time, aka, throughput), resources consumption (CPU, Memory, GPU), convergence, deployment effort, Flexibility, accuracy, scalability.