Ensemble Graph Attention Networks
DOI:
https://doi.org/10.14738/tmlai.103.12399Keywords:
Ensemble learning, Bagging, Graph Neural Network, Heterogeneous Graph Neural Network, Graph Attention NetworkAbstract
Graph neural networks have demonstrated its success in many applications on graph-structured data. Many efforts have been devoted to elaborating new network architectures and learning algorithms over the past decade. The exploration of applying ensemble learning techniques to enhance existing graph algorithms have been overlooked. In this work, we propose a simple generic bagging-based ensemble learning strategy which is applicable to any backbone graph models. We then propose two ensemble graph neural network models – Ensemble-GAT and Ensemble-HetGAT by applying the ensemble strategy to the graph attention network (GAT), and a heterogeneous graph attention network (HetGAT). We demonstrate the effectiveness of the proposed ensemble strategy on GAT and HetGAT through comprehensive experiments with four real-world homogeneous graph datasets and three real-world heterogeneous graph datasets on node classification tasks. The proposed Ensemble-GAT and Ensemble-HetGAT outperform the state-of-the-art graph neural network and heterogeneous graph neural network models on most of the benchmark datasets. The proposed ensemble strategy also alleviates the over-smoothing problem in GAT and HetGAT.
References
Zhang, C. and Y. Ma, Ensemble machine learning: methods and applications2012: Springer.
Breiman, L., Bagging predictors. Machine learning, 1996. 24(2): p. 123-140.
Freund, Y. and R.E. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 1997. 55(1): p. 119-139.
Schapire, R.E., The strength of weak learnability. Machine learning, 1990. 5(2): p. 197-227.
Wolpert, D.H., Stacked generalization. Neural networks, 1992. 5(2): p. 241-259.
Jacobs, R.A., et al., Adaptive mixtures of local experts. Neural computation, 1991. 3(1): p. 79-87.
Jordan, M.I. and R.A. Jacobs, Hierarchical mixtures of experts and the EM algorithm. Neural computation, 1994. 6(2): p. 181-214.
Breiman, L., Random forests. Machine learning, 2001. 45(1): p. 5-32. 40
Ho, T.K., The random subspace method for constructing decision forests. IEEE transactions on pattern analysis and machine intelligence, 1998. 20(8): p. 832-844.
Friedman, J.H., Greedy function approximation: a gradient boosting machine. Annals of statistics, 2001: p. 1189-1232.
Sun, K., Z. Zhu, and Z. Lin, Adagcn: Adaboosting graph convolutional networks into deep models. arXiv preprint arXiv:1908.05081, 2019.
Du, J., et al., Topology adaptive graph convolutional networks. arXiv preprint arXiv:1710.10370, 2017.
Liu, M., H. Gao, and S. Ji. Towards deeper graph neural networks. in Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 2020.
Wang, M., et al., Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.01315, 2019.
Veličković, P., et al., Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.
Klicpera, J., A. Bojchevski, and S. Günnemann, Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997, 2018.
Lv, Q., et al. Are we really making much progress? Revisiting, benchmarking and refining heterogeneous graph neural networks. in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.
Wang, X., et al. Heterogeneous graph attention network. in The world wide web conference. 2019.
Chen, M., et al. Simple and deep graph convolutional networks. in International Conference on Machine Learning. 2020. PMLR.
Wang, G., et al., Improving graph attention networks with large margin-based constraints. arXiv preprint arXiv:1910.11945, 2019.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Nan Wu, Chaofan Wang

This work is licensed under a Creative Commons Attribution 4.0 International License.