Improving federated learning performance with similarity guided feature extraction and pruning.

Palihawadana, Chamath

Abstract

Federated Learning (FL) is a Machine Learning (ML) paradigm that learns from distributed clients to collaboratively train a global model in a privacy-preserved manner without sharing their private data. Traditional centralised ML approaches require aggregating data from various sources into a single location. This poses substantial risks regarding data privacy, security breaches and compliance with data protection regulations. The primary goal of FL is to ensure data privacy by keeping the raw data on clients' devices, while sharing only model parameters with a central server. Each client updates its model locally using its data and then sends these updated parameters (weights) to the server. The server aggregates these updates to create an improved global model, which is then distributed back to the clients. This aggregation process is crucial in FL, as it combines knowledge learned across a diverse range of clients, enabling them to benefit from collective insights while preserving the privacy of their data. This iterative process continues, gradually refining the global model through multiple rounds of local training and aggregation. However, the adoption of FL is not without its challenges. FL's decentralised nature introduces complexities such as the impact on model performance with aggregation methods, communication overhead and security threats. Existing aggregation methodologies often lack generalisability across different datasets and applications. Moreover, communication efficiency remains a significant bottleneck. The frequent exchange of model updates between clients and the server can be resource-intensive. Additionally, communicating model updates and the distributed nature of FL opens up more threat surfaces for attacks. In this thesis, we present several methodologies to improve FL performance, with a focus on neural architectures for classification tasks. The models considered include Multinomial Logistic Regression (MLR), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN) and Multi-layer Perceptrons (MLP) trained using Stochastic Gradient Descent (SGD). First, we introduce FedSim, a similarity-guided model aggregation algorithm designed to enhance FL accuracy by leveraging inter-client relationships. Instead of relying on client data, we extract similarity knowledge by comparing client gradients. The FedSim algorithm decomposes into cluster aggregation and global aggregation steps. Cluster aggregation considers only the updated models within the cluster and then globally aggregates them to ensure better coverage and reduce variance. FedSim prioritises gradient updates that are consistent across multiple clients to ensure that these aligned updates have a stronger influence on the global model. To evaluate the generalisability of FedSim, we conducted extensive experiments across various datasets and model architectures to demonstrate its effectiveness. Secondly, we introduce FedFT, a communication-efficient FL algorithm that leverages frequency space transformation to reduce communication overhead while maintaining model accuracy. Communicating in the frequency space enables efficient compression due to its compact representation. Its linear properties eliminate the need for multiple transformations during aggregation reducing additional computational overhead. We then address the security challenges in FL, recognising the potential risks posed by gradient inversion attacks. To mitigate these threats, we present pFGD, a defence mechanism that utilises FedFT to protect against such privacy attacks. Finally, we validated our proposed methodologies through a real-world case study in the healthcare domain. Applying FedSim and FedFT in this context demonstrated their practical applicability and generalisability, highlighting that these methods enhance FL performance. This thesis contributes to the field of FL by introducing novel methods that address critical challenges while ensuring their applicability across diverse scenarios.

Citation

PALIHAWADANA, C. 2024. Improving federated learning performance with similarity guided feature extraction and pruning. Robert Gordon University, PhD thesis. Hosted on OpenAIR [online]. Available from: https://doi.org/10.48526/rgu-wt-2801100

• Thesis Type: Thesis
• Deposit Date: Apr 22, 2025
• Publicly Available Date: Apr 22, 2025
• DOI: https://doi.org/10.48526/rgu-wt-2801100
• Keywords: Machine learning; Federated learning; Data privacy; Data security; Systems security; Health data
• Public URL: https://rgu-repository.worktribe.com/output/2801100
• Additional Information: Source code and datasets are available on GitHub. See pages iv - v of the thesis for links.
• Award Date: Nov 30, 2024