Accurate, interpretable predictions of materials properties within transformer language models

Summary

Property prediction accuracy has long been a key parameter of machine learning in materials informatics. Accordingly, advanced models showing state-of-the-art performance turn into highly parameterized black boxes missing interpretability. Here, we present an elegant way to make their reasoning transparent. Human-readable text-based descriptions automatically generated within a suite of open-source tools are proposed as materials representation. Transformer language models pretrained on 2 million peer-reviewed articles take as input well-known terms such as chemical composition, crystal symmetry, and site geometry. Our approach outperforms crystal graph networks by classifying four out of five analyzed properties if one considers all available reference data. Moreover, fine-tuned text-based models show high accuracy in the ultra-small data limit. Explanations of their internal machinery are produced using local interpretability techniques and are faithful and consistent with domain expert rationales. This language-centric framework makes accurate property predictions accessible to people without artificial-intelligence expertise.

Highlights

• •

  Text descriptions are efficient in representing materials for property prediction

• •

  Pretrained language models outperform graph neural networks in most cases

• •

  Explanations provided by language models are consistent with human rationales

The bigger picture

Description, prediction, and explanation are traditionally acknowledged as central aims of science. In the field of materials informatics, the second objective receives the most attention, while the understanding of the resulting structure-property relationships is less emphasized. In this study, we reconcile large-scale language models and human-readable descriptions of crystal structure to facilitate materials design insights. The presented approach surpasses the state of the art in property prediction and provides transparency in the machinery of artificial-intelligence algorithms, thereby possibly improving the trust of materials scientists. In addition, the clarity of text-based representation and maturity of associated explainability methods make the approach appealing for educational uses.

Abstract

Black-boxed algorithms dominate in materials property prediction. Pretrained large-scale models in conjunction with interpretability techniques counteract the unfavorable tendency by providing clear explanations of suggested outputs.