Scientific article
OA Policy
English

Transformer Performance for Chemical Reactions: Analysis of Different Predictive and Evaluation Scenarios

Published in

Journal of chemical information and modeling, vol. 63, no. 7, p. 1914-1924

Publication date

2023-04-10

First online date

2023-03-23

Abstract

The prediction of chemical reaction pathways has been accelerated by the development of novel machine learning architectures based on the deep learning paradigm. In this context, deep neural networks initially designed for language translation have been used to accurately predict a wide range of chemical reactions. Among models suited for the task of language translation, the recently introduced molecular transformer reached impressive performance in terms of forward-synthesis and retrosynthesis predictions. In this study, we first present an analysis of the performance of transformer models for product, reactant, and reagent prediction tasks under different scenarios of data availability and data augmentation. We find that the impact of data augmentation depends on the prediction task and on the metric used to evaluate the model performance. Second, we probe the contribution of different combinations of input formats, tokenization schemes, and embedding strategies to model performance. We find that less stable input settings generally lead to better performance. Lastly, we validate the superiority of round-trip accuracy over simpler evaluation metrics, such as top-k accuracy, using a committee of human experts and show a strong agreement for predictions that pass the round-trip test. This demonstrates the usefulness of more elaborate metrics in complex predictive scenarios and highlights the limitations of direct comparisons to a predefined database, which may include a limited number of chemical reaction pathways.

Keywords

  • Benchmarking
  • Databases, Factual
  • Electric Power Supplies
  • Humans
  • Machine Learning
  • Neural Networks, Computer

Citation (ISO format)

JAUME SANTERO, Fernando et al. Transformer Performance for Chemical Reactions: Analysis of Different Predictive and Evaluation Scenarios. In: Journal of chemical information and modeling, 2023, vol. 63, n° 7, p. 1914–1924. doi: 10.1021/acs.jcim.2c01407

Main files (1)

Article (Published version)

Secondary files (1)

Appendix

Identifiers

Additional URL for this publication

https://pubs.acs.org/doi/10.1021/acs.jcim.2c01407

Journal ISSN

1549-9596
119views
176downloads

Technical informations

Creation

24/03/2023 12:32:15

First validation

06/06/2023 07:52:52

Update

13/10/2025 14:25:56

Status update

06/06/2023 07:52:52

Last indexation

27/08/2026 07:19:41
All rights reserved by Archive ouverte UNIGE and the University of Geneva
OpenAIRE Indexed & Validated