Anticipating Technical Expertise and Capability Evolution in Research Communities Using Dynamic Graph Transformers

Horawalavithana, Yasanka Sameera; Ayton, Ellyn M.; Usenko, Anastasiya A.; Cosbey, Robin J.; Volkova, Svitlana

doi:10.1109/tcss.2024.3416837

Anticipating Technical Expertise and Capability Evolution in Research Communities Using Dynamic Graph Transformers

Journal Article · Thu Jul 11 00:00:00 EDT 2024 · IEEE transactions on computational social systems (Online)

DOI:https://doi.org/10.1109/tcss.2024.3416837· OSTI ID:2468642

^[1]; ^[1]; Usenko, Anastasiya A. ^[1]; Cosbey, Robin J. ^[1]; ^[1]

Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

The ability to anticipate global technical expertise and capability evolution trends is essential for national and global security, especially in safety-critical domains such as nuclear nonproliferation (NN) and rapidly emerging fields like artificial intelligence (AI). Here, in this work, we extend traditional statistical relational learning approaches (e.g., link prediction in collaboration networks) and formulate a problem of anticipating technical expertise and capability evolution using dynamic heterogeneous graph representations. We develop novel capabilities to forecast collaboration patterns, authorship behavior, and technical capability evolution at different granularities (e.g., scientist and institution levels) in two distinct research fields. We implement a dynamic graph transformer (DGT) neural architecture, which pushes the state-of-the-art graph neural network models by: 1) forecasting heterogeneous (rather than homogeneous) nodes and edges; and 2) relying on both discrete- and continuous-time inputs. We demonstrate that our DGT models predict collaboration, partnership, and expertise patterns with 0.26, 0.73, and 0.53 mean reciprocal rank values for AI and 0.48, 0.93, and 0.22 for NN domains. DGT model performance exceeds the best-performing static graph baseline models by 30%–80% across AI and NN domains. Our findings demonstrate that DGT models boost inductive task performance when previously unseen nodes appear in the test data for the domains with emerging collaboration patterns (e.g., AI). Specifically, models accurately predict which established scientists will collaborate with early career scientists and vice versa in the AI domain.

View Accepted Manuscript (DOE)

Research Organization:: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA), Office of Defense Nuclear Nonproliferation

Grant/Contract Number:: AC05-76RL01830

OSTI ID:: 2468642

Report Number(s):: PNNL-SA--181649

Journal Information:: IEEE transactions on computational social systems (Online), Journal Name: IEEE transactions on computational social systems (Online) Journal Issue: 5 Vol. 11; ISSN 2329-924X

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (19)

Modeling Relational Data with Graph Convolutional Networks Schlichtkrull, Michael; Kipf, Thomas N.; Bloem, Peter The Semantic Web https://doi.org/10.1007/978-3-319-93417-4_38	book	January 2018
Defining and evaluating network communities based on ground-truth Yang, Jaewon; Leskovec, Jure Knowledge and Information Systems, Vol. 42, Issue 1 https://doi.org/10.1007/s10115-013-0693-z	journal	October 2013
dyngraph2vec: Capturing network dynamics using dynamic graph representation learning Goyal, Palash; Chhetri, Sujit Rokka; Canedo, Arquimedes Knowledge-Based Systems, Vol. 187 https://doi.org/10.1016/j.knosys.2019.06.024	journal	January 2020
The Science of Science Wang, Dashun; Barabási, Albert-László https://doi.org/10.1017/9781108610834	book	January 2021
Understanding collaboration patterns on funded research projects: A network analysis Smith, Matthew; Sarabi, Yasaman; Christopoulos, Dimitris Network Science, Vol. 11, Issue 1 https://doi.org/10.1017/nws.2022.33	journal	November 2022
The importance of industrial publications No authors listed Nature Catalysis, Vol. 1, Issue 7, p. 479-480 https://doi.org/10.1038/s41929-018-0119-0	journal	July 2018
Impactful scientists have higher tendency to involve collaborators in new topics Zeng, An; Fan, Ying; Di, Zengru Proceedings of the National Academy of Sciences, Vol. 119, Issue 33 https://doi.org/10.1073/pnas.2207436119	journal	August 2022
Science of science Fortunato, Santo; Bergstrom, Carl T.; Börner, Katy Science, Vol. 359, Issue 6379 https://doi.org/10.1126/science.aao0185	journal	March 2018
Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks Kumar, Srijan; Zhang, Xikun; Leskovec, Jure Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining https://doi.org/10.1145/3292500.3330895	conference	July 2019
Transformers in Vision: A Survey Khan, Salman; Naseer, Muzammal; Hayat, Munawar ACM Computing Surveys, Vol. 54, Issue 10s https://doi.org/10.1145/3505244	journal	January 2022
Interdisciplinarity, Gender Diversity, and Network Structure Predict the Centrality of AI Organizations Vlasceanu, Madalina; Dudik, Miroslav; Momennejad, Ida 2022 ACM Conference on Fairness, Accountability, and Transparency https://doi.org/10.1145/3531146.3533069	conference	June 2022
Overview of the 2003 KDD Cup Gehrke, Johannes; Ginsparg, Paul; Kleinberg, Jon ACM SIGKDD Explorations Newsletter, Vol. 5, Issue 2 https://doi.org/10.1145/980972.980992	journal	December 2003
Microsoft Academic Graph: When experts are not enough Wang, Kuansan; Shen, Zhihong; Huang, Chiyuan Quantitative Science Studies, Vol. 1, Issue 1 https://doi.org/10.1162/qss_a_00021	journal	February 2020
EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs Pareja, Aldo; Domeniconi, Giacomo; Chen, Jie Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, Issue 04 https://doi.org/10.1609/aaai.v34i04.5984	journal	April 2020
Nuclear Proliferation and Arms Control Monitoring, Detection, and Verification No authors listed https://doi.org/10.17226/26088	report	July 2021
Recurrent Event Network: Autoregressive Structure Inferenceover Temporal Knowledge Graphs Jin, Woojeong; Qu, Meng; Jin, Xisen Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) https://doi.org/10.18653/v1/2020.emnlp-main.541	conference	January 2020
Identifying Causal Influences on Publication Trends and Behavior: A Case Study of the Computational Linguistics Community Glenski, Maria; Volkova, Svitlana Proceedings of the First Workshop on Causal Inference and NLP https://doi.org/10.18653/v1/2021.cinlp-1.7	conference	January 2021
Foundation Models of Scientific Knowledge for Chemistry: Opportunities, Challenges and Lessons Learned Horawalavithana, Sameera; Ayton, Ellyn; Sharma, Shivam Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models https://doi.org/10.18653/v1/2022.bigscience-1.12	conference	January 2022
Attention Is All You Need Vaswani, Ashish; Shazeer, Noam; Parmar, Niki arXiv https://doi.org/10.48550/ARXIV.1706.03762	preprint	January 2017