Robust Federated Reinforcement Learning under Adversarial Clients via Prototype Consistency Regularization
Keywords:
federated reinforcement learning, adversarial clients, prototype consistency, robustness, Byzantine tolerance, decentralized policy learningAbstract
Federated reinforcement learning enables multiple agents to collaboratively learn a shared policy without exchanging raw experience data, offering significant benefits for decentralized control in domains such as autonomous driving, robotics, and smart grid management. However, the distributed nature of federated learning makes it inherently vulnerable to adversarial clients that may poison model updates through data manipulation, gradient injection, or malicious policy deviations. Existing defenses often rely on statistical outlier detection or robust aggregation rules, but these methods struggle under non-identically distributed data distributions and high-dimensional policy spaces. This paper proposes a novel framework that integrates prototype consistency regularization into federated reinforcement learning to enhance robustness against adversarial clients. The approach leverages the idea that each client’s learned policy should produce consistent feature representations—prototypes—across benign participants, and any client whose prototype distribution diverges significantly from the consensus can be identified and downweighted during aggregation. We discuss the system-level architecture, trade-offs between robustness and convergence efficiency, and deployment considerations such as communication overhead and privacy preservation. Through conceptual analysis and illustrative case studies, we demonstrate that prototype consistency regularization provides a principled mechanism for adversarial client detection without requiring access to raw data or assuming independent and identically distributed client distributions. Policy implications for federated learning governance, fairness, and sustainability are also examined. The findings indicate that prototype-based regularization offers a scalable and interpretable defense suitable for large-scale federated reinforcement learning systems.
References
1. Blanchard, P., Mhamdi, E. M., Guerraoui, R., & Stainer, J. (2017). Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 30.
2. Yin, D., Chen, Y., Kannan, R., & Bartlett, P. (2018). Byzantine-robust distributed learning: Towards optimal statistical rates. Proceedings of the 35th International Conference on Machine Learning, 80, 5650–5659.
3. Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2020). Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2, 429–450.
4. Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., & Suresh, A. T. (2020). SCAFFOLD: Stochastic controlled averaging for federated learning. Proceedings of the 37th International Conference on Machine Learning, 119, 5132–5143.
5. Snell, J., Swersky, K., & Zemel, R. (2017). Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 30.
6. Ruff, L., Vandermeulen, R. A., Görnitz, N., Deecke, L., Siddiqui, S. A., Binder, A., Müller, E., & Kloft, M. (2018). Deep one-class classification. Proceedings of the 35th International Conference on Machine Learning, 80, 4393–4402.
7. McMahan, B., Moore, E., Ramage, D., Hampson, S., & y Arcas, B. A. (2017). Communication-efficient learning of deep networks from decentralized data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 54, 1273–1282.
8. Shui, Y., Jin, R., Dou, Z., & Gao, Z. (2026). ProtoGuard-SL: Prototype Consistency Based Backdoor Defense for Vertical Split Learning. arXiv preprint arXiv:2604.03595.
9. Jin, H., Peng, Y., Yang, W., Wang, S., & Zhang, Z. (2020). Federated reinforcement learning: A survey. arXiv preprint arXiv:2010.13261.
10. Zhu, H., Jin, R., & Gao, Z. (2021). Federated reinforcement learning with asynchronous agents. Proceedings of the AAAI Conference on Artificial Intelligence, 35(11), 9990–9998.
11. Zhang, J., Wang, Y., & Li, S. (2021). Adversarial attacks and defenses for federated reinforcement learning. Proceedings of the IEEE Conference on Computer Communications, 1–10.
12. Prabhudesai, M., Chaganti, S., Vemuri, V., & Gopalan, R. (2020). Consistency regularization for domain adaptation. Proceedings of the European Conference on Computer Vision, 12356, 527–543.
13. Cao, X., Lai, J., & Lv, J. (2022). Robust aggregation for federated learning with geometric median and clipping. IEEE Transactions on Information Forensics and Security, 17, 2356–2368.
14. Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, 119, 1597–1607.
15. Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3–4), 211–407.
16. Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2020). How to backdoor federated learning. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, 108, 2938–2948.
17. Baruch, G., Baruch, M., & Goldberg, Y. (2019). A little is enough: Circumventing defenses for distributed learning. Advances in Neural Information Processing Systems, 32.
18. Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., & Pedarsani, R. (2020). FedPAQ: A communication-efficient federated learning method with periodic averaging and quantization. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, 108, 2021–2031.
19. Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., & Seth, K. (2017). Practical secure aggregation for privacy-preserving machine learning. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 1175–1191.
20. Li, L., Xu, W., Chen, T., Giannakis, G. B., & Yin, W. (2021). RSA: Byzantine-robust stochastic aggregation methods for distributed learning from heterogeneous datasets. IEEE Transactions on Signal Processing, 69, 1254–1269.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Computer Science and Engineering Transactions
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
