Adaptive Q-Learning-Based Radio Resource Management Optimization in 5G and Beyond Heterogeneous-heterogeneous Networks: A Comprehensive Review
Article Sidebar
Page Numbers:
137-152
Digital Object Identifier:
10.58578/ajstea.v3i1.4578
Viewed: 465 times
Downloaded: 287 times
LYAS Publisher cordially invites qualified professionals to serve as Editors or Reviewers. Your expertise will make an important contribution to maintaining and strengthening the academic quality of our publications. Interested applicants are kindly requested to complete the application form at the following link: Editors & Reviewers
Please feel free to contact us if you need any further information about the submission process or if you have any additional questions.
Authors:
Abubakar Abdulkadir1*, Mahmoud T. Kabir2, H. A. Abdulkareem3, ZM Abdullahi4 
Copyright :
Authors retain copyright and grant the journal right of first publication.
Main Article Content
Abstract
This paper reviews advanced radio resource management (RRM) optimization techniques in 5G and beyond heterogeneous-heterogeneous networks (Het-HetNets). Key innovations include fairness-aware models for mmWave 5G, machine learning (ML)-driven traffic management, and game-theoretic approaches for interference mitigation in Massive MIMO systems. Blockchain technology emerges as a promising tool for secure spectrum sharing, while deep learning enhances handover management and resource allocation. Hybrid frameworks, such as deep reinforcement learning and non-orthogonal multiple access, address energy efficiency and quality of service (QoS) challenges for IoT, autonomous vehicles, and smart cities. Despite these advancements, challenges like scalability, computational complexity, and data privacy persist. Q-learning-based adaptive RRM frameworks demonstrate potential for optimizing energy and spectral efficiency by addressing dynamic network conditions. The integration of ML with blockchain enables secure and decentralized RRM. Critical research gaps identified include scalability, real-time deployment, and interference management in ultra-dense networks. This review highlights the importance of scalable, efficient, and adaptive solutions to advance the telecommunications system.
Downloads
Article Details
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
References
Abbass, W., Hussain, R., Frnda, J., Abbas, N., Javed, M. A., & Malik, S. A. (2022). Resource Allocation in Spectrum Access System Using Multi-Objective Optimization Methods. Sensors, 22(4), 1–24. https://doi.org/10.3390/s22041318
Agarwal, B., Togou, M. A., Ruffini, M., & Muntean, G. M. (2022a). A Comprehensive Survey on Radio Resource Management in 5G HetNets: Current Solutions, Future Trends and Open Issues. IEEE Communications Surveys and Tutorials, 24(4), 2495–2534. https://doi.org/10.1109/COMST.2022.3207967
Agarwal, B., Togou, M. A., Ruffini, M., & Muntean, G. M. (2022b). A Comprehensive Survey on Radio Resource Management in 5G HetNets: Current Solutions, Future Trends and Open Issues. IEEE Communications Surveys and Tutorials, 24(4), 2495–2534. https://doi.org/10.1109/COMST.2022.3207967
Ahmad, R., Wazirali, R., & Abu-Ain, T. (2022). Machine Learning for Wireless Sensor Networks Security: An Overview of Challenges and Issues. Sensors, 22(13). https://doi.org/10.3390/s22134730
Al Haj Hassan, H., Jaber, S., El Amine, A., Nasser, A., & Nuaymi, L. (2024). Reinforcement learning for radio resource management of hybrid energy cellular networks with battery constraints. Computer Communications, 213, 135–146. https://doi.org/10.1016/j.comcom.2023.10.025
Alwarafy, A., Ciftler, B. S., Abdallah, M., Hamdi, M., & Al-Dhahir, N. (2022). Hierarchical Multi-Agent DRL-Based Framework for Joint Multi-RAT Assignment and Dynamic Resource Allocation in Next-Generation HetNets. IEEE Transactions on Network Science and Engineering, 9(4), 2481–2494. https://doi.org/10.1109/TNSE.2022.3164648
Amani, N., Parsaeefard, S., & Yanikomeroglu, H. (2023). Multi-Objective Energy Efficient Resource Allocation in Massive Multiple Input Multiple Output-Aided Heterogeneous Cloud Radio Access Networks. IEEE Access, 11(March), 33480–33497. https://doi.org/10.1109/ACCESS.2023.3263951
Amine, A. El, Dini, P., & Nuaymi, L. (2020). Reinforcement learning for delay-constrained energy-aware small cells with multi-sleeping control. 2020 IEEE International Conference on Communications Workshops, ICC Workshops 2020 - Proceedings, May. https://doi.org/10.1109/ICCWorkshops49005.2020.9145431
analytixlabs. (2021). Types of Algorithms With Different Machine Learning Algorithm Examples. https://www.analytixlabs.co.in/blog/types-of-machine-learning/
Bartsiokas, I. A., Gkonis, P. K., Kaklamani, D. I., & Venieris, I. S. (2023). A DL-Enabled Relay Node Placement and Selection Framework in Multicellular Networks. IEEE Access, 11(July), 65153–65169. https://doi.org/10.1109/ACCESS.2023.3290482
Deepthi, J. V. N. R., Khan, A. K., & Acharjee, T. (2024). Reinforcement Learning Based Spectrum Sensing and Resource Allocation in WSN-IoT Smart Applications. Springer, 105–120. https://doi.org/10.1007/978-981-99-6690-5_8
Ghafoor, U., Khan, H. Z., Ali, M., Siddiqui, A. M., Naeem, M., & Rashid, I. (2022a). Energy Efficient Resource Allocation for H-NOMA Assisted B5G HetNets. IEEE Access, 10(September), 91699–91711. https://doi.org/10.1109/ACCESS.2022.3201527
Ghafoor, U., Khan, H. Z., Ali, M., Siddiqui, A. M., Naeem, M., & Rashid, I. (2022b). Energy Efficient Resource Allocation for H-NOMA Assisted B5G HetNets. IEEE Access, 10(July), 91699–91711. https://doi.org/10.1109/ACCESS.2022.3201527
Guo, W. (2020). Explainable Artificial Intelligence for 6G: Improving Trust between Human and Machine. IEEE Communications Magazine, 58(6), 39–45. https://doi.org/10.1109/MCOM.001.2000050
Gupta, C., Johri, I., Srinivasan, K., Hu, Y. C., Qaisar, S. M., & Huang, K. Y. (2022). A Systematic Review on Machine Learning and Deep Learning Models for Electronic Information Security in Mobile Networks. Sensors, 22(5). https://doi.org/10.3390/s22052017
Gures, E., Shayea, I., Ergen, M., Azmi, M. H., & El-Saleh, A. A. (2022). Machine Learning-Based Load Balancing Algorithms in Future Heterogeneous Networks: A Survey. IEEE Access, 10, 37689–37717. https://doi.org/10.1109/ACCESS.2022.3161511
Hossain, E., Rasti, M., & Le, L. B. (2017). Radio Resource Management in Wireless Networks. Radio Resource Management in Wireless Networks. https://doi.org/10.1017/9781316212493
Iqbal, M. U., Ansari, E. A., & Akhtar, S. (2021a). Interference mitigation in hetnets to improve the QoS using q-learning. IEEE Access, 9, 32405–32424. https://doi.org/10.1109/ACCESS.2021.3060480
Iqbal, M. U., Ansari, E. A., & Akhtar, S. (2021b). Interference mitigation in hetnets to improve the QoS using q-learning. IEEE Access, 9, 32405–32424. https://doi.org/10.1109/ACCESS.2021.3060480
Iqbal, M. U., Ansari, E. A., Akhtar, S., Farooq-I-Azam, M., Hassan, S. R., & Asif, R. (2023). Optimal Learning Paradigm and Clustering for Effective Radio Resource Management in 5G HetNets. IEEE Access. https://doi.org/10.1109/ACCESS.2023.3268543
Iqbal, M. U., Ansari, E. A., Akhtar, S., Rafiq, M. N., Farooq-i-Azam, M., & Hassan, B. (2023). Hybrid Learning based Radio Resource Management in 5G Heterogeneous Networks. Pakistan Journal of Engineering and Technology, 6(1), 92–97. https://doi.org/10.51846/vol6iss1pp92-97
Jorquera Valero, J. M., Sánchez Sánchez, P. M., Lekidis, A., Fernandez Hidalgo, J., Gil Pérez, M., Siddiqui, M. S., Huertas Celdrán, A., & Martínez Pérez, G. (2022). Design of a Security and Trust Framework for 5G Multi-domain Scenarios. Journal of Network and Systems Management, 30(1). https://doi.org/10.1007/S10922-021-09623-7
Jusoh, M. H. (2023). HetNets : Current State , Challenges , and Future Directions. Sensors, 1–36.
Kamal, M. A., Raza, H. W., Alam, M. M., Su’ud, M. M., & Sajak, A. B. A. B. (2021). Resource allocation schemes for 5G network: A systematic review. Sensors, 21(19). https://doi.org/10.3390/s21196588
Kaur, J., Khan, M. A., Iftikhar, M., Imran, M., & Emad Ul Haq, Q. (2021a). Machine Learning Techniques for 5G and beyond. IEEE Access, 9, 23472–23488. https://doi.org/10.1109/ACCESS.2021.3051557
Kaur, J., Khan, M. A., Iftikhar, M., Imran, M., & Emad Ul Haq, Q. (2021b). Machine Learning Techniques for 5G and beyond. IEEE Access, 9, 23472–23488. https://doi.org/10.1109/ACCESS.2021.3051557
Lan, Q., Wen, D., Zhang, Z., Zeng, Q., Chen, X., Popovski, P., & Huang, K. (2021). What Is Semantic Communication? A View on Conveying Meaning in the Era of Machine Intelligence. Journal of Communications and Information Networks, 6(4), 336–371. https://doi.org/10.23919/jcin.2021.9663101
Luong, N. C., Wang, P., Niyato, D., Liang, Y. C., Han, Z., & Hou, F. (2019). Applications of Economic and Pricing Models for Resource Management in 5G Wireless Networks: A Survey. IEEE Communications Surveys and Tutorials, 21(4), 3298–3339. https://doi.org/10.1109/COMST.2018.2870996
Mollel, M. S., Abubakar, A. I., Ozturk, M., Kaijage, S. F., Kisangiri, M., Hussain, S., Imran, M. A., & Abbasi, Q. H. (2021). A Survey of Machine Learning Applications to Handover Management in 5G and beyond. IEEE Access, 9, 45770–45802. https://doi.org/10.1109/ACCESS.2021.3067503
Naderializadeh, N., Sydir, J. J., Simsek, M., & Nikopour, H. (2021). Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning. IEEE Transactions on Wireless Communications, 20(6), 3507–3523. https://doi.org/10.1109/TWC.2021.3051163
Reebadiya, D., Rathod, T., Gupta, R., Tanwar, S., & Kumar, N. (2021). Blockchain-based Secure and Intelligent Sensing Scheme for Autonomous Vehicles Activity Tracking Beyond 5G Networks. Peer-to-Peer Networking and Applications, 14(5), 2757–2774. https://doi.org/10.1007/s12083-021-01073-x
Rezaei, Z., & Ghahfarokhi, B. S. (2023). Energy and spectrum efficient cell switch-off with channel and power allocation in ultra-dense networks: A deep reinforcement learning approach. Computer Networks, 234. https://doi.org/10.1016/j.comnet.2023.109912
Rony, R. I., Lopez-Aguilera, E., & Garcia-Villegas, E. (2021a). Dynamic Spectrum Allocation following Machine Learning-Based Traffic Predictions in 5G. IEEE Access, 9, 143458–143472. https://doi.org/10.1109/ACCESS.2021.3122331
Rony, R. I., Lopez-Aguilera, E., & Garcia-Villegas, E. (2021b). Dynamic Spectrum Allocation following Machine Learning-Based Traffic Predictions in 5G. IEEE Access, 9(3), 143458–143472. https://doi.org/10.1109/ACCESS.2021.3122331
rosas, ahmed. (2022). D2D communication with Radio resources scheduling in 5G Millimeter wave Heterogeneous Networks. Menoufia Journal of Electronic Engineering Research, 31(2), 94–99. https://doi.org/10.21608/mjeer.2022.136723.1055
Ryu, K., & Kim, W. (2021). Multi-objective optimization of energy saving and throughput in heterogeneous networks using deep reinforcement learning. Sensors, 21(23). https://doi.org/10.3390/s21237925
Saddoud, A., & Fourati, L. C. (2021). Joint microcell-based relay selection and RRM for LTE-A toward 5G networks. Telecommunication Systems, 78(3), 363–375. https://doi.org/10.1007/s11235-021-00817-7
Salah, I., Mabrook, M. M., Hussein, A. I., & Raliouma, K. H. (2021). Comparative study of efficiency enhancement technologies in 5G networks- A survey. Procedia Computer Science, 182, 150–158. https://doi.org/10.1016/j.procs.2021.02.020
Salahdine, F., Han, T., & Zhang, N. (2023). Security in 5G and beyond recent advances and future challenges . Security and Privacy, 6(1). https://doi.org/10.1002/spy2.271
Shabbir, A., Rizvi, S., Shirazi, M. F., Alam, M. M., & Su’ud, M. M. (2024). Maximizing energy efficiency in HetNets through centralized and distributed sleep strategies under QoS constraint. Scientific Reports, 14(1), 25839. https://doi.org/10.1038/s41598-024-70714-x
Siddiqui, M. U. A., Qamar, F., Tayyab, M., Hindia, M. H. D. N., Nguyen, Q. N., & Hassan, R. (2022). Mobility Management Issues and Solutions in 5G-and-Beyond Networks: A Comprehensive Review. Electronics (Switzerland), 11(9). https://doi.org/10.3390/electronics11091366
Sridevi, M. J., Sharma, P. K., Kumar, M. D., Loganathan, D., & Ahmed, S. (2024). Optimizing Efficiency and Performance in 5G Networks Through a Dynamic Resource Allocation Algorithmic Framework. ICTACT Journal on Communication Technology, 3112–3118. https://doi.org/10.21917/ijct.2024.0463
Tang, F., Zhou, Y., & Kato, N. (2020). Deep Reinforcement Learning for Dynamic Uplink/Downlink Resource Allocation in High Mobility 5G HetNet. IEEE Journal on Selected Areas in Communications, 38(12), 2773–2782. https://doi.org/10.1109/JSAC.2020.3005495
Tariq, F., Khandaker, M. R. A., Wong, K. K., Imran, M. A., Bennis, M., & Debbah, M. (2020). A Speculative Study on 6G. IEEE Wireless Communications, 27(4), 118–125. https://doi.org/10.1109/MWC.001.1900488
Tashan, W., Shayea, I., Aldirmaz-Colak, S., Ergen, M., Azmi, M. H., & Alhammadi, A. (2022). Mobility Robustness Optimization in Future Mobile Heterogeneous Networks: A Survey. IEEE Access, 10, 45522–45541. https://doi.org/10.1109/ACCESS.2022.3168717
Ullah, R., Ullah, S., Faisal, F., Ullah, R., Mabrouk, I. Ben, Al Hasan, M. J., & Kamal, B. (2021). A novel multi-band and multi-generation (2G, 3G,4G, and 5G) 9-elements MIMO antenna system for 5G smartphone applications. Wireless Networks, 27(7), 4825–4837. https://doi.org/10.1007/s11276-021-02772-2
Wang, C., Wu, L., Zhang, N., Wang, Z., Yang, R., Liu, Y., Xie, Z., Ge, M., & Zhu, R. (2023). Routing and Spectrum Resource Allocation for Advance Reservation Services in the Space-Terrestrial Integrated Network. Ieeexplore.Ieee.Org, 109–117. https://doi.org/10.1109/icait59485.2023.10367358
Wang, Z., Ding, H., Li, B., Bao, L., Yang, Z., & Liu, Q. (2022). Energy Efficient Cluster Based Routing Protocol for WSN Using Firefly Algorithm and Ant Colony Optimization. Wireless Personal Communications, 125(3), 2167–2200. https://doi.org/10.1007/s11277-022-09651-9
William, P., Choubey, S., Ramkumar, M., Verma, A., Vengatesan, K., & Choubey, A. (2022). Implementation of 5G Network Architecture with Interoperability in Heterogeneous Wireless Environment using Radio Spectrum. Proceedings of the International Conference on Electronics and Renewable Systems, ICEARS 2022, 786–791. https://doi.org/10.1109/ICEARS53579.2022.9752267
Yadav, I., Chaporkar, P., Jha, P., & Karandikar, A. (2021). Spectrum Allocation in IAB Networks: A Hierarchical Auction-based Approach. IEEE Vehicular Technology Conference, 2021-Septe. https://doi.org/10.1109/VTC2021-Fall52928.2021.9625522
Yasaka, T., Yoda, K., & Otsuka, H. (2022). Optimization of Pico-eNB Tx Power and the Effects of Picocell Range Expansion in Multiband HetNet. Journal of Sensor and Actuator Networks, 11(2). https://doi.org/10.3390/jsan11020027
Zhang, H., Networks, D. W.-N., & 2023, undefined. (n.d.). Deep MCANC: A deep learning approach to multi-channel active noise control. Elsevier. Retrieved March 11, 2024, from https://www.sciencedirect.com/science/article/pii/S0893608022004701
Zhihao, H., Donglai, Z., & Gang, W. (2021a). Research on Spectrum Allocation Strategy Based on Stackelberg Game in Ultra Dense Network. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST, 342, 36–46. https://doi.org/10.1007/978-3-030-66785-6_4
Zhihao, H., Donglai, Z., & Gang, W. (2021b). Research on Spectrum Allocation Strategy Based on Stackelberg Game in Ultra Dense Network. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST, 342, 36–46. https://doi.org/10.1007/978-3-030-66785-6_4