International Journal of Emerging Research in Engineering, Science, and Management
Vol. 5, Issue 3, pp. 110-128, Jul-Sep 2026.
https://doi.org/10.58482/ijeresm.v5i3.8

Received: 01 May 2026 | Revised: 28 Aug 2026 | Accepted: 11 Sep 2026 | Published: 19 Sep 2026

This work is licensed under a Creative Commons Attribution 4.0 International License.

Adaptive Fusion-Driven Multimodal Sentiment Analysis Using U-Net++, Capsule Network, and Transformer-Enhanced Extreme Learning

Full Text PDF

1J. Ramachandraiah

2M. Sirish Kumar

1Research Scholar, School of Computing, Department of CSE, Mohan Babu University, Tirupati, India.

2Associate Professor, School of Computing, Department of CSE, Mohan Babu University, Tirupati, India.

Abstract: Multimodal social media data contain textual, visual, and emoji information that can provide complementary cues for sentiment analysis. However, combining these modalities remains challenging because they differ in representation and may contain redundant or noisy information. This study presents an Adaptive Fusion-Driven Multimodal Sentiment Analysis Framework that combines U-Net++ with self-attention, a Capsule Network encoder-decoder, a Cross-Modal Transformer, and a Transformer-Enhanced Extreme Learning Machine (T-ELM). U-Net++ with self-attention extracts contextual features, while the Capsule Network captures hierarchical relationships among the extracted representations. The Cross-Modal Transformer then models interactions between the text, image, and emoji modalities before classification by T-ELM. The trimodal experiments use 9,200 samples selected from the Social Media MultiSent Dataset (SM-MSD), with positive, neutral, and negative sentiment polarity as the target classes. Using a 70% training, 15% validation, and 15% test split, the proposed framework achieves competitive sentiment classification performance across the evaluated datasets. The results show that the proposed fusion framework effectively combines complementary multimodal information while keeping the final classification stage lightweight.

Keywords: Multimodal sentiment analysis, U-Net++, capsule network, Transformer-Enhanced Extreme Learning Machine, cross-modal fusion.

References
  1. J. Che, M. Sun, Y. Wang, and Z. Xu, “Fusion-driven multimodal learning for biomedical time series in surgical care,” Frontiers in Physiology, vol. 16, p. 1605406, Sep. 2025. https://doi.org/10.3389/fphys.2025.1605406
  2. B. Subbaiah, K. Murugesan, P. Saravanan, and K. Marudhamuthu, “An efficient multimodal sentiment analysis in social media using hybrid optimal multi-scale residual attention network,” Artificial Intelligence Review, vol. 57, no. 2, Feb. 2024. https://doi.org/10.1007/s10462-023-10645-7
  3. H. Wang, C. Ren, and Z. Yu, “Multimodal sentiment analysis based on cross-instance graph neural networks,” Applied Intelligence, vol. 54, no. 4, pp. 3403–3416, Feb. 2024. https://doi.org/10.1007/s10489-024-05309-0
  4. H. Li, Y. Lu, and H. Zhu, “Multi-Modal sentiment analysis based on image and text fusion based on Cross-Attention Mechanism,” Electronics, vol. 13, no. 11, p. 2069, May 2024. https://doi.org/10.3390/electronics13112069
  5. J. Huang, J. Zhou, Z. Tang, J. Lin, and C. Y.-C. Chen, “TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis,” Knowledge-Based Systems, vol. 285, p. 111346, Dec. 2023. https://doi.org/10.1016/j.knosys.2023.111346
  6. Y. Jin et al., “Harnessing multimodal emotion features in depression detection across gender: Integrating large language model, acoustic fusion and facial expression recognition,” Journal of Affective Disorders, vol. 400, p. 121207, Jan. 2026. https://doi.org/10.1016/j.jad.2026.121207
  7. G. K. P., A. A. V. S., J. P. V., A. Paul, and A. Nayyar, “A context-sensitive multi-tier deep learning framework for multimodal sentiment analysis,” Multimedia Tools and Applications, vol. 83, no. 18, pp. 54249–54278, Dec. 2023. https://doi.org/10.1007/s11042-023-17601-1
  8. S. Lai, X. Hu, H. Xu, Z. Ren, and Z. Liu, “Multimodal sentiment analysis: A survey,” Displays, vol. 80, p. 102563, Oct. 2023. https://doi.org/10.1016/j.displa.2023.102563
  9. S. J. T. Jwalanaiah, I. J. Jacob, and A. K. Mandava, “Effective deep learning based multimodal sentiment analysis from unstructured big data,” Expert Systems, vol. 40, no. 1, Jul. 2022. https://doi.org/10.1111/exsy.13096
  10. G. Chandrasekaran, N. Antoanela, G. Andrei, C. Monica, and J. Hemanth, “Visual Sentiment Analysis Using Deep Learning Models with Social Media Data,” Applied Sciences, vol. 12, no. 3, p. 1030, Jan. 2022. https://doi.org/10.3390/app12031030
  11. A. Alsayat, “Improving sentiment analysis for social media applications using an ensemble deep learning language model,” Arabian Journal for Science and Engineering, vol. 47, no. 2, pp. 2499–2511, Oct. 2021. https://doi.org/10.1007/s13369-021-06227-w
  12. R. K. Behera, M. Jena, S. K. Rath, and S. Misra, “Co-LSTM: Convolutional LSTM model for sentiment analysis in social big data,” Information Processing & Management, vol. 58, no. 1, p. 102435, Nov. 2020. https://doi.org/10.1016/j.ipm.2020.102435
  13. G. Kaur and A. Sharma, “A deep learning-based model using hybrid feature extraction approach for consumer sentiment analysis,” Journal of Big Data, vol. 10, no. 1, p. 5, Jan. 2023. https://doi.org/10.1186/s40537-022-00680-6
  14. I. Priyadarshini and C. Cotton, “A novel LSTM–CNN–grid search-based deep neural network for sentiment analysis,” The Journal of Supercomputing, vol. 77, no. 12, pp. 13911–13932, May 2021. https://doi.org/10.1007/s11227-021-03838-w
  15. Y. Zhang, J. Fan, and B. Dong, “Deep Learning-Based Analysis of Social Media Sentiment Impact on Cryptocurrency Market Microstructure,” Academic Journal of Sociology and Management, vol. 3, no. 2, pp. 13–21, Mar. 2025. https://doi.org/10.70393/616a736d.323730
  16. S. D. Pasham, “Scalable Graph-Based algorithms for Real-Time analysis of big data in social networks,” Mathematics and Computer Science Journal, Jan. 2024. https://doi.org/10.18535/mcsj/v2024.01
  17. V. Tejaswini, K. S. Babu, and B. Sahoo, “Depression Detection from Social Media Text Analysis using Natural Language Processing Techniques and Hybrid Deep Learning Model,” ACM Transactions on Asian and Low-Resource Language Information Processing, vol. 23, no. 1, pp. 1–20, Nov. 2022. https://doi.org/10.1145/3569580
  18. A. Helmy, R. Nassar, and N. Ramdan, “Depression detection for twitter users using sentiment analysis in English and Arabic tweets,” Artificial Intelligence in Medicine, vol. 147, p. 102716, Nov. 2023. https://doi.org/10.1016/j.artmed.2023.102716
  19. R. Alfaisal, H. Hashim, and U. H. Azizan, “Convolutional Neural Network for Sentiment Analysis on Metaverse-Related Tweets: A Deep Learning approach,” SN Computer Science, vol. 5, no. 6, Aug. 2024. https://doi.org/10.1007/s42979-024-03121-8
  20. A. Kuruva and C. N. Chiluka, “Hybrid deep learning approach for sentiment analysis using text and emojis,” Network Computation in Neural Systems, vol. 36, no. 3, pp. 923–952, May 2024. https://doi.org/10.1080/0954898X.2024.2349275
  21. S. T. Kokab, S. Asghar, and S. Naz, “Transformer-based deep learning models for the sentiment analysis of social media data,” Array, vol. 14, p. 100157, Apr. 2022. https://doi.org/10.1016/j.array.2022.100157
  22. N. S., D. Paulraj, P. Ezhumalai, and M. Prakash, “A Deep Learning Modified Neural Network (DLMNN) based proficient sentiment analysis technique on Twitter data,” Journal of Experimental & Theoretical Artificial Intelligence, pp. 1–20, Jun. 2022. https://doi.org/10.1080/0952813X.2022.2093405
  23. H. Saif, M. Fernández, Y. He, and H. Alani, “Evaluation datasets for Twitter sentiment analysis: A survey and a new dataset, the STS-Gold,” Proc. ESSEM@AI*IA, 2013, pp. 9–21.
  24. M. Hodosh, P. Young, and J. Hockenmaier, “Framing image description as a ranking task: data, models and evaluation metrics,” Journal of Artificial Intelligence Research, vol. 47, pp. 853–899, Aug. 2013. https://doi.org/10.1613/jair.3994
  25. L. Vadicamo et al., “Cross-Media Learning for Image Sentiment Analysis in the Wild,” 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy, 2017, pp. 308–317. https://doi.org/10.1109/ICCVW.2017.45
  26. J. Peng et al., “A social media dataset and H-GNN-Based Contrastive learning scheme for multimodal sentiment analysis,” Applied Sciences, vol. 15, no. 2, p. 636, Jan. 2025. https://doi.org/10.3390/app15020636
  27. J. McAuley and J. Leskovec, “Hidden factors and hidden topics: Understanding rating dimensions with review text,” Proc. 7th ACM Conf. Recommender Systems (RecSys), 2013, pp. 165–172. https://doi.org/10.1145/2507157.2507163
  28. X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” Advances in Neural Information Processing Systems 28 (NeurIPS), pp. 649–657, 2015.
Download PDF

This article is part of Volume 5, Issue 3 (Jul–Sep 2026)

×

© 2026 The Author(s). Published by IJERESM. This work is licensed under the Creative Commons Attribution 4.0 International License.

Archiving: All articles are permanently archived in Zenodo IJERESM Community.