Paper "BenchTemp: A General Benchmark for Evaluating Temporal Graph Neural Networks" accepted at ICDE’24
The following paper has been accepted at the 40th IEEE Conference on Data Engineering
(external pageICDE'24) to be held in Utrecht, the Netherlands on the 13- 17 May 2024:
Title
BenchTemp: A General Benchmark for Evaluating Temporal Graph Neural Networks
Authors
Qiang Huang (Wuhan University), Xin Wang (Wuhan University), Susie Xi Rao (ETH Zürich), Zhichao Han (eBay), Zitao Zhang (eBay), Yongjun He (ETH Zürich), Quanqing Xu (OceanBase), Yang Zhao (eBay), Zhigao Zheng (Wuhan University), Jiawei Jiang (Wuhan University)
Abstract
To handle graphs in which features or connectivities are evolving over time, a series of temporal graph neural networks (TGNNs) have been proposed. Despite the success of these TGNNs, the previous TGNN evaluations reveal several limitations regarding four critical issues: 1) inconsistent datasets, 2) inconsistent evaluation pipelines, 3) lacking workload diversity, and 4) lacking efficient comparison. Overall, there lacks an empirical study that puts TGNN models onto the same ground and compares them comprehensively. To this end, we propose BenchTemp, a general benchmark for evaluating TGNN models on various workloads. BenchTemp provides a set of benchmark datasets so that different TGNN models can be fairly compared. Further, BenchTemp engineers a standard pipeline that unifies the TGNN evaluation. With BenchTemp, we extensively compare the representative TGNN models on different tasks (e.g., link prediction and node classification) and settings (transductive and inductive), w.r.t. both effectiveness and efficiency metrics. We have made BenchTemp publicly available at external pagehttps://github.com/qianghuangwhu/benchtemp and datasets at external pagehttps://zenodo.org/records/8267846.