T2Acmrcmr

# DIALOG-22 RuATD Generated Text Detection

Narek Maloyan
Lomonosov MSU
<EMAIL_ADDRESS>
Bulat Nutfullin
Lomonosov MSU
<EMAIL_ADDRESS>
Eugene Ilyushin
Lomonosov MSU
<EMAIL_ADDRESS>

###### Abstract

Text Generation Models (TGMs) succeed in creating text that matches human
language style reasonably well. Detectors that can distinguish between TGM-
generated text and human-written ones play an important role in preventing
abuse of TGM. In this paper, we describe our pipeline for the two DIALOG-22
RuATD tasks: detecting generated text (binary task) and classification of
which model was used to generate text (multiclass task) [Shamardina et al.,
2022a]. We achieved 1st place on the binary classification task with an
accuracy score of 0.82995 on the private test set and 4th place on the
multiclass classification task with an accuracy score of 0.62856 on the
private test set. We proposed an ensemble method of different pre-trained
models based on the attention mechanism111github.com/maloyan/ruatd.

Keywords: Generated text, text classification, ensemble methods, multi-class
classification task

###### ?abstractname?

Модели генерации текста успешно синтезируют текст, который сложно отличим от
написанного человеком текста. Детекторы, способные отличить текст, созданный
автоматически, от написанного человеком текста позволяют предотвратить
злоупотребление сгенерированными текстами. В этой статье мы описываем наше
решение для задач DIALOG-22 RuATD по обнаружению сгенерированного текста и
классификации с помощью какой модели был сгенерирован текст. Мы заняли 1-е
место в задаче бинарной классификации с оценкой точности 0,829 в частном
наборе тестов и 4-е место в задаче мультиклассовой классификации с оценкой
точности 0,628 в тестовом наборе. Наше решение является ансамблем дообученных
моделей, основанных на механизме внимания.

Ключевые слова: Сгенерированные тексты, классификация текстов, ансамблевые
методы, задача многоклассовой классификации

##  1 Introduction

As the language neural nets got better at generating texts, it’s getting
harder and harder to distinguish the human-written text from generated one. So
manual detection of these texts got impossible. For that reason, it’s
desirable to build a system that can automatically detect generated text.

The proposed system will use various features extracted from the text such as
length, punctuation, word choice etc. To determine whether the text is human-
generated or not. The accuracy of this system can be improved by using machine
learning algorithms which will learn how humans generate texts and then use
those features for detection purpose.

There are many ways to build such a system, but probably the most reliable one
is based on machine learning algorithms. These algorithms can be trained on a
large number of examples - both human-generated and computer-generated texts.
After being trained, they should be able to identify which texts are computer-
generated with high accuracy.

This approach already works in other areas, such as spam detection. Some early
experiments have shown promising results and indicate that this approach works
well for the detection of the generated text.

The DIALOG-22 shared task of RuATD 2022 had 2 tracks for binary classification
and multiclass classification. In this report, we will describe the data, how
we handled it, the models we used, and the ensembling technique.

##  2 Task Definition

For the binary classification task $F_{binary}$, we frame the generated text
detection problem as follows: given a piece of text $X$, label it as either
human-written or machine-generated $y_{binary}=\\{H,M\\}$.

$F_{binary}:X\rightarrow y_{binary}$

For the multiclass classification task $F_{multiclass}$, we set up the problem
as follows: given a piece of text $X$, label it as one of the 14 classes that
represent deep neural models $y_{multiclass}$ = {  M2M-100, Human, OPUS-MT,
M-BART50, ruGPT3-Medium, ruGPT3-Small, mT5-Large, ruGPT3-Large, ruT5-Base-
Multitask, mT5-Small, ruT5-Base, ruGPT2-Large, M-BART, ruT5-Large  }

$F_{multiclass}:X\rightarrow y_{multiclass}$

##  3 Datasets

Provided datasets offer the train and test splits. The part of the set was
annotated automatically by different generative models. Various language
models were fine-tuned on different tasks: machine translation, paraphrasing,
summarization, simplification, and unconditional text generation - are used to
generate texts. The texts written by a human were collected from open sources
from different domains. [Shamardina et al., 2022b], [Shamardina et al.,
2022c].

Text | Class
---|---
Обустройство тротуаров, мостовых (в том числе тротуарной плиткой). | H
Минстрой обозначил способы снижения энергоемкости российской экономики. | M
В конце 1873 года военный суд вынес решение по делу Франциска Ахиллы Базейн. | M
увеличение правовой грамотности и развитие правосознания граждан. | H
Table 1: Example of the binary classification data. Text | Class
---|---
Прочла автобиографию Каутского, Одесса, 1905. | Human
Вы не можете быть в печи и в муulin. | M-BART50
Вторая попытка привела к тому же результату. | OPUS-MT
Сколько учеников в вашем классе? | M2M-100
Table 2: Example of the multiclass classification data.

##  4 Related Works

In this section, we will discuss various methods for detecting machine-
generated texts.

Over the past years, many approaches appeared for generated text detection.
The latest works are usually based on using transformer-based models. Either
fine-tuning the proposed task or using probabilities distribution to make
decision-based on them [Ippolito et al., 2019]. Here we list some examples of
different methods:

  * •

First, we calculate the mean likelihood over all machine-generated sequences,
then the mean of human-written ones. If the likelihood according to some
language model is closer to the machine-generated mean likelihood, then we
consider it as generated text [Solaiman et al., 2019];

  * •

In GLTR [Gehrmann et al., 2019] described a method using a language model to
compute the probability distribution of the next word given the text sequence.
For each sequence position, we get the likelihood of the ground-truth next
word within this list. Then these ranks are displayed on a histogram. Based on
the distribution of bins, we can decide if this text is generated or not.

  * •

Bert fine-tuning on the classification task. Having a label of text if it’s
machine-generated or not, we can fine-tune the language model to predict
[Solaiman et al., 2019]

  * •

Also, there is possible to use human-machine collaboration. Real or Fake Tool
provides a game-like interface for humans to decide at what point the text
begins to be written by a computer [Dugan et al., 2020].

More approaches are described in the survey [Jawahar et al., 2020].

##  5 Experiment Setup

In this section, we present the experiment configurations we use to solve
binary and multiclass tasks.

###  5.1 Data preproccessing

We decided not to perform any preprocessing of the text itself. Regarding the
data split:

  * •

Binary classification. We concatenated the train set with the validation set.
This new concatenated dataset gave us an option to perform a 5-fold cross-
validation;

  * •

Multiclass classification. We took the data as is without changing anything.

###  5.2 Models

All models we used for fine-tuning we got from transformers library [Wolf et
al., 2020]. Here we will describe these models:

  * •

sberbank-ai/sbert_large_nlu_ru[Antuhov, 2020], sberbank-ai/ruBert-large,
DeepPavlov/rubert-base-cased are BERT models [Devlin et al., 2018]. Sber-ai
models were fine-tuned on a closed dataset collected by their research group
[Sber, 2022]. DeepPavlov’s RuBERT was fine-tuned on the Russian part of
Wikipedia and news data.

  * •

IlyaGusev/mbart_ru_sum_gazeta is a mBART model fine-tuned on the dataset for
summarization of Russian news[Gusev, 2020]. The original MBart model was
pretrained on large-scale monolingual corpora in many languages using the BART
objective. mBART is one of the first methods for pre-training a complete
sequence-to-sequence model by denoising full texts in multiple languages,
while previous approaches have focused only on the encoder, decoder, or
reconstructing parts of the text [Liu et al., 2020].

  * •

MoritzLaurer/mDeBERTa-v3-base-mnli-xnli [He et al., 2021] This multilingual
model is suitable for multilingual zero-shot classification. The original
model was pre-trained by Microsoft on the CC100 multilingual dataset [Wenzek
et al., 2020].

This model was fine-tuned on the XNLI development dataset and the MNLI train
dataset. The XNLI development set consists of translated texts for each of the
15 languages [Conneau et al., 2018].

  * •

DeepPavlov/xlm-roberta-large-en-ru-mnli is an XLM-RoBERTa model [Conneau et
al., 2019] which was fine-tuned on the english-russian part of the MNLI
[Williams et al., 2017] dataset.

###  5.3 Binary classification (with ensembling technique)

Five chosen models were used in the experiement: sberbank-
ai/sbert_large_nlu_ru, sberbank-ai/ruBert-large, IlyaGusev/mbart_ru_sum_gazeta
, MoritzLaurer/mDeBERTa-v3-base-mnli-xnli, DeepPavlov/xlm-roberta-large-en-ru-
mnli

Let’s describe steps for training each of these models:

  1. 1.

We split our training dataset into non-overlapping 5-folds and performed
cross-validation (Figure 1);

  2. 2.

for each validation fold we predicted the target and as a result, we got out-
of-fold predictions for training data;

  3. 3.

for the test set, we predicted the target from every 5 models from cross-
validation and averaged the result which is marked with red;

As a meta-model, we choose Logistic Regression, which was trained on out-of-
fold predictions. Then we predict the final results for the test set.

Figure 1: Ensembling scheme

###  5.4 Multiclass classification

For the multiclass classification we chose these models: DeepPavlov/rubert-
base-cased, DeepPavlov/xlm-roberta-large-en-ru, IlyaGusev/mbart_ru_sum_gazeta.
We fine-tune these models without cross-validation only on provided train set.

##  6 Results

Table 3 shows each model accuracy score on binary classification task. The
multiclass classification accuracy is shown in Table 4. On both tasks, the
best performing single model was DeepPavlov/xlm-roberta-large-en-ru-mnli. We
managed to ensemble models in the binary task, so the ensemble of models
showed the best accuracy.

Model name | Accuracy
---|---
sberbank-ai/sbert_large_nlu_ru | 0.79986 ±0.003
sberbank-ai/ruBert-large | 0.80154 ±0.002
IlyaGusev/mbart_ru_sum_gazeta | 0.80566 ±0.001
MoritzLaurer/mDeBERTa-v3-base-mnli-xnli | 0.80710 ±0.001
DeepPavlov/xlm-roberta-large-en-ru-mnli | 0.81708 ± 0.002
Ensemble | 0.82995222Kaggle private leaderboard link
Table 3: Results for binary classification for different models Model name | Accuracy(validation) | Accuracy(kaggle public/private)
---|---|---
IlyaGusev/mbart_ru_sum_gazeta | 0.6142 | 0.61459/0.61092
DeepPavlov/rubert-base-cased. | 0.6045 | 0.60433/0.60472
DeepPavlov/xlm-roberta-large-en-ru-mnli. | 0.6242 | 0.62856/0.62644
Table 4: Results for multiclass classification for different models

##  7 Conclusion

In this paper, we described our pipeline for the DIALOG-22 RuATD challenge.
Our solution achieved 1st place in binary classification using ensembling
techniques and 4th place for the multiclass classification task using only a
single model. However, the proposed solution requires a lot of computational
power, so it cannot be used in real-time systems to detect generated texts.
But it gives us an understanding that we still need to upgrade methods to
distinguish generated texts from human-written ones.

## References

  * [Antuhov, 2020]  Denis Antuhov.  2020\.  Natulral language model learning with bert and tensorflow link:https://habr.com/ru/company/sberdevices/blog/527576/, Nov.
  * [Conneau et al., 2018]  Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov.  2018\.  Xnli: Evaluating cross-lingual sentence representations.  arXiv preprint arXiv:1809.05053.
  * [Conneau et al., 2019]  Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  2019\.  Unsupervised cross-lingual representation learning at scale.  arXiv preprint arXiv:1911.02116.
  * [Devlin et al., 2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * [Dugan et al., 2020]  Liam Dugan, Daphne Ippolito, Arun Kirubarajan, and Chris Callison-Burch.  2020\.  Roft: A tool for evaluating human detection of machine-generated text.  arXiv preprint arXiv:2010.03070.
  * [Gehrmann et al., 2019]  Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush.  2019\.  Gltr: Statistical detection and visualization of generated text.  arXiv preprint arXiv:1906.04043.
  * [Gusev, 2020]  Ilya Gusev.  2020\.  Dataset for automatic summarization of russian news.  // Conference on Artificial Intelligence and Natural Language, P 122–134. Springer.
  * [He et al., 2021]  Pengcheng He, Jianfeng Gao, and Weizhu Chen.  2021\.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  arXiv preprint arXiv:2111.09543.
  * [Ippolito et al., 2019]  Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck.  2019\.  Automatic detection of generated text is easiest when humans are fooled.  arXiv preprint arXiv:1911.00650.
  * [Jawahar et al., 2020]  Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan.  2020\.  Automatic detection of machine generated text: A critical survey.  arXiv preprint arXiv:2011.01314.
  * [Liu et al., 2020]  Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer.  2020\.  Multilingual denoising pre-training for neural machine translation.  Transactions of the Association for Computational Linguistics, 8:726–742.
  * [Sber, 2022]  Sber.  2022\.  Sberbank-ai.  https://huggingface.co/sberbank-ai.
  * [Shamardina et al., 2022a]  Tatiana Shamardina, Vladislav Mikhailov, Daniil Chernianskii, Alena Fenogenova, Marat Saidov, Anastasiya Valeeva, Tatiana Shavrina, Ivan Smurov, Elena Tutubalina, and Ekaterina Artemova.  2022a.  Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian.  // Computational Linguistics and Intellectual Technologies: Papers from the Annual Conference "Dialogue", volume 21, P xxx–xxx.
  * [Shamardina et al., 2022b]  Tatiana Shamardina, Vladislav Mikhailov, Daniil Chernianskii, Alena Fenogenova, Marat Saidov, Anastasiya Valeeva, Tatiana Shavrina, Ivan Smurov, Elena Tutubalina, and Ekaterina Artemova.  2022b.  Ruatd 2022 (binary task).  https://www.kaggle.com/competitions/ruatd-2022-bi.  Accessed: 2022-03-28.
  * [Shamardina et al., 2022c]  Tatiana Shamardina, Vladislav Mikhailov, Daniil Chernianskii, Alena Fenogenova, Marat Saidov, Anastasiya Valeeva, Tatiana Shavrina, Ivan Smurov, Elena Tutubalina, and Ekaterina Artemova.  2022c.  Ruatd 2022 (multi task).  https://www.kaggle.com/competitions/ruatd-2022-multi-task.  Accessed: 2022-03-28.
  * [Solaiman et al., 2019]  Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al.  2019\.  Release strategies and the social impacts of language models.  arXiv preprint arXiv:1908.09203.
  * [Wenzek et al., 2020]  Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave.  2020\.  CCNet: Extracting high quality monolingual datasets from web crawl data.  // Proceedings of the 12th Language Resources and Evaluation Conference, P 4003–4012, Marseille, France, May. European Language Resources Association.
  * [Williams et al., 2017]  Adina Williams, Nikita Nangia, and Samuel R Bowman.  2017\.  A broad-coverage challenge corpus for sentence understanding through inference.  arXiv preprint arXiv:1704.05426.
  * [Wolf et al., 2020]  Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  2020\.  Transformers: State-of-the-art natural language processing.  // Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, P 38–45.


# Machine Learning Benchmarks for the Classification of Equivalent Circuit
Models from Electrochemical Impedance Spectra

Joachim Schaeffer
CCPS Laboratory
TU Darmstadt, Germany
<EMAIL_ADDRESS>
&Paul Gasper
National Renewable Energy Lab
Golden, CO, USA
<EMAIL_ADDRESS>&Esteban Garcia-Tamayo
Titan Advanced Energy Solutions
Salem, MA, USA
<EMAIL_ADDRESS>&Raymond Gasper
Kingston, MA, USA
<EMAIL_ADDRESS>&Masaki Adachi
Machine Learning Research Group
University of Oxford, UK
<EMAIL_ADDRESS>&Juan Pablo Gaviria-Cardona
Universidad Pontificia Bolivariana
Medellin, Colombia
<EMAIL_ADDRESS>&Simon Montoya-Bedoya
Verasonics SAS
Medellin, Colombia
<EMAIL_ADDRESS>&Anoushka Bhutani
Department of Mechanical Engineering
Carnegie Mellon University
Pittsburgh, PA, USA
<EMAIL_ADDRESS>&Andrew Schiek
National Renewable Energy Lab
Golden, CO, USA
<EMAIL_ADDRESS>&Rhys Goodall
Chemix.ai
Sunnyvale, CA, USA
<EMAIL_ADDRESS>& Rolf Findeisen
CCPS Laboratory
TU Darmstadt, Germany
<EMAIL_ADDRESS>
&Richard D. Braatz
Massachusetts Institute of Technology
Cambridge, MA, USA
<EMAIL_ADDRESS>&Simon Engelke
Battery Associates
Dublin, Ireland
<EMAIL_ADDRESS>
###### Abstract

Analysis of Electrochemical Impedance Spectroscopy (EIS) data for
electrochemical systems often consists of defining an Equivalent Circuit Model
(ECM) using expert knowledge and then optimizing the model parameters to
deconvolute various resistance, capacitive, inductive, or diffusion responses.
For small data sets, this procedure can be conducted manually; however, it is
not feasible to manually define a proper ECM for extensive data sets with a
wide range of EIS responses. Automatic identification of an ECM would
substantially accelerate the analysis of large sets of EIS data. We showcase
machine learning methods to classify the ECMs of 9,300 impedance spectra
provided by QuantumScape for the BatteryDEV hackathon. The best-performing
approach is a gradient-boosted tree model utilizing a library to automatically
generate features, followed by a random forest model using the raw spectral
data. A convolutional neural network using boolean images of Nyquist
representations is presented as an alternative, although it achieves a lower
accuracy. We publish the data and open source the associated code. The
approaches described in this article can serve as benchmarks for further
studies. A key remaining challenge is the identifiability of the labels,
underlined by the model performances and the comparison of misclassified
spectra.

_Keywords_ Electrochemical Impedance Spectroscopy $\cdot$ Machine Learning
$\cdot$ Equivalent Circuit Model $\cdot$ Classification $\cdot$ Lithium-Ion
Batteries $\cdot$ Hackathon $\cdot$ Open Data

##  1 Introduction

Processes inside lithium-ion batteries (LIBs) and other electrochemical
devices occur at different timescales [1]. In LIBs, lithium ions are shuttled
between positive and negative electrodes, via the electrolyte and separator,
mostly through diffusion processes. The kinetics during this transition vary
due to differences between lithium-ion diffusion coefficients in liquids
(electrolyte) and solids (positive/negative electrode active materials), which
give rise to the different timescales mentioned. One method for monitoring the
various responses of electrochemical systems over different timescales is
Electrochemical Impedance Spectroscopy (EIS), a non-invasive technique that
uses AC voltage or current signals over a spectrum of frequencies to excite
processes within the electrochemical system. These spectra can thus facilitate
the evaluation of electrochemical systems [2], such as batteries [3, 4, 5, 1],
fuel cells [6], supercapacitors [7], corrosion [8], or biological systems [9].
For batteries, particular research areas exploiting EIS are Equivalent Circuit
Model (ECM) characterization [3], blocking electrode experiments to
investigate purely ionic or electronic behaviors [10], diffusion processes
modeling [11], characterization of porous electrodes [12], electrode
characterization via transmission line modeling [13], and monitoring of cell
performance [14]. However, to analyze these spectra and to assign a specific
mechanism such as electronic resistance, charge-transfer, mass transport,
etc., electrochemists usually employ ECM to represent the different
physicochemical processes in the battery by parameterizing them in terms of
electrical circuit elements such as inductance, resistance, capacitance, or a
combination of them. Defining the structure of an ECM generally requires
expert judgment, meaning that evaluation of a very large number of EIS
measurements is a difficult process to automate.

As in many other scientific and engineering fields, Machine Learning (ML)
methods have become popular in the area of electrochemistry to accelerate data
analysis or modeling tasks, especially for large data sets. For example, ML
methods have been used successfully for predicting the remaining useful life
of batteries both in laboratory environments [15] as well as in deployed
systems [16]. Furthermore, machine learning methods gained popularity during
the last years for analyzing spectral data such as FTIR spectra [17, 18],
Raman spectra [18, 19, 20], X-ray diffraction spectra [21], and EIS data [22,
23, 24, 25, 26, 14, 27]. Recent developments include Bayesian model selection
for EIS data [54, 55] based on fast Bayesian inference using quadrature [53].
[49] Further development of software tools and ML methods for analyzing
impedance data can be accelerated by the publication of open-source software
libraries and data sets. There exists open-source software to analyze EIS data
(e.g. [32, 33]). However, there is still significant potential and need for
analysis software and machine learning approaches to be shared and published
open source. The situation is similar for data. Various data sets are
available [14, 34, 35, 27], but the total amount of open data is still small
compared to the wide range of applications and the diversity of EIS data that
arises from those. So, this article aims to contribute to the growing body of
open-source battery data and software. We focus on using ML methods to
accurately classify the latent ECM. With this article, we share a large
synthetic EIS data set with the associates ECMs and an unlabeled data set
consisting of synthetic and measured data, both provided by QuantumScape (QS)
for the international BatteryDEV hackathon to attract a variety of researchers
and source interdisciplinary solutions for the problem of ECM identification.

##### BatteryDEV Hackathon

The machine learning approaches described in this article were partially
developed during the one-week open-source BatteryDEV hackathon in March 2022.
However, additional approaches were developed afterward, and the existing work
was refined for this article. This publication makes the data and the code
publicly available. The BatteryDEV hackathons were started by its host
organization Battery Associates to foster innovation in the battery space. The
first BatteryDEV hackathon took place in January 2021, and the second
iteration in March 2022. The BatteryDEV hackathons receive support from
industry and academia, as described in the acknowledgments. Within the context
of batteries, the objectives of BatteryDEV are to (1) increase global
collaboration involving data across sectors, (2) encourage the development of
open-source solutions for analyzing data, and (3) provide an opportunity for
hands-on training to grow the pool of global talent. For BatteryDEV 2022,
there were 140 registrations, 85 people joined the hackathon, and there were
submissions from 60 participants. There were registrations from more than 20
countries, and participants included Data Analytics, ML, Battery, and Energy
Materials experts from industry and academia, many participating in
interdisciplinary teams. The need for more openly available data sets in the
battery space is widely accepted [36, 37]. Hackathons can accelerate
innovation and have been shown to yield exciting results in other fields [38].

This article is organized as follows. Section 2 describes the EIS data set
provided by QS. Section 3 defines the purpose and the challenges of the
hackathon. Section 4 reports the approaches followed by Section 5 for
discussion of other ideas and Section 6 outlines the paper conclusions.

##  2 EIS Data Set

The EIS data set, $\mathcal{Z}$, associated with this article was created by
QS. It comprises approximately 9,300 synthetic impedance spectra,
$\mathbf{z}\in\mathcal{Z}$ that are vectors of impedances,
$\mathbf{z}\in\mathbb{C}^{n}$. Furthermore, the data set includes the
generating ECM types. No noise was added to the data. The parameters of the
ECMs were drawn from independent reciprocal distributions, except for the time
constants, which were drawn from uniform distributions. Furthermore, it was
ensured that the time constants of the RC elements are significantly different
for each impedance spectrum associated with the RC-RC-RCPE-RCPE circuit, i.e.,
$\max(\\{\tau_{1}/\tau_{2},\tau_{2}/\tau_{1}\\})>10$. The parameter bounds and
frequency ranges are informed by the solid-state battery R&D of QS.
Furthermore, QS shared an unlabeled data set containing 80% synthetic and 20%
measured data, which is also made available but not analyzed in this article.
Information about how to access the data can be found in the data sharing
section.

Table 1 illustrates the nine classes of the predefined ECMs. A hyphen denotes
series connections, and a combination of two elements denotes a parallel
connection. Each parallel connection has only two branches, each consisting of
one element. The name in parenthesis is a shorthand notation. The number of
impedance spectra in parenthesis is the number of spectra prior to filtering
out spectra that met filter criteria described in the Supplementary
Information Sec. A.1. These criteria remove spectra that are either unphysical
or unlikely to represent a physical battery or both. However, for
completeness, we include results for the filtered and unfiltered data in Tab.
4. All other results correspond to the filtered data. Table 2 states the six
circuit elements, names, number of parameters, and equations for each of the
elements which form the ECMs in Table 1.

Table 1: The predefined ECM configurations, see Table 2 for detailed descriptions of each element. Name | Number of parameters | Number of spectra (without filtering)
---|---|---
L-R-RCPE | 5 | 752 (1,084)
L-R-RCPE-RCPE (L-R-2RCPE) | 8 | 1,050 (1,132)
L-R-RCPE-RCPE-RCPE (L-R-3RCPE) | 11 | 1,101 (1,114)
RC-G-G | 6 | 1,099 (1,099)
RC-RC-RCPE-RCPE | 10 | 1,146 (1,152)
RCPE-RCPE (2RCPE) | 6 | 1,060 ( 1,064)
RCPE-RCPE-RCPE (3RCPE) | 9 | 1,129 (1,140)
RCPE-RCPE-RCPE-RCPE (4RCPE) | 12 | 1,134 (1,138)
R-Ws (Rs_Ws) | 4 |  330 (404)
Table 2: The circuit elements Symbol | Name | Number of parameters | Equation
---|---|---|---
L | Inductance | 1 | $j\omega L$
R | Resistance | 1 | $R$
C | Capacitance | 1 | $\frac{1}{j\omega C}$
CPE | Constant phase element | 2 | $\frac{1}{C(j\omega)^{t}}$
G | Gerischer element | 2 | $\frac{R}{\sqrt{1+j\omega t}}$
Ws | Warburg short element | 3 | $R\frac{\text{tanh}\left((j\omega t)^{p}\right)}{(j\omega t)^{p}}$

Figure 1 shows four example spectra with different corresponding ECMs. The
Bode plot on the left of each subplot shows the magnitude in black and the
phase shift in green. The Nyquist plot shows the impedance response to each
frequency used to excite the system, with each data point in the Nyquist plot
corresponding to a distinct frequency; note that a Nyquist plot alone without
frequency labeling does not show the data fully, as all detail of the
frequency dimension is lost. The values in the Nyquist plot with a small
absolute value correspond to the higher frequencies.

Figure 1: (a)–(d) four selected electrical impedance spectra from the data
set. The Bode plot on the left of each subplot shows the magnitude in black
and the phase shift in green. The Nyquist plot on the right of each subplot
shows the impedance response to each frequency used to excite the system, with
each data point in the Nyquist plot corresponding to a distinct frequency. The
dots are experimental recordings, and the lines are from equivalent circuit
models using circuits and parameters provided by QS.

###  2.1 Data Preprocessing

The range of frequencies and the number of measured frequencies vary widely
between the spectra of each circuit class (c.f., Supplementary Information,
Figs. A.3 and A.4). The frequency ranges and ranges of the number of measured
frequencies are the same for all circuits. Choosing specific frequency ranges
is standard practice, depending on the underlying battery’s characteristics
and the scope of the EIS-based investigation [39, 1]. We interpolated the real
and imaginary impedance vectors for every EIS spectra at $n=30$
logarithmically spaced frequencies across the common frequency basis, ranging
from 101 Hz to 105 Hz to obtain consistent data for feature design and machine
learning. This limited frequency range may neglect certain features,
especially at lower frequencies. Another possible approach would be to rescale
all frequency ranges to a common vector, though this would lead to further
issues since many circuit parameter values are dependent on frequency.

###  2.2 Data Visualization

Visualization of a large number of impedance spectra is difficult. For this
data set, a particular challenge is the distribution of parameters over orders
of magnitude, resulting in spectra that also span orders of magnitude. Figure
2 visualizes the spectra associated with each circuit class by rescaling them.

$\displaystyle\operatorname{Re}(\tilde{z_{j}})$
$\displaystyle=(\operatorname{Re}(z_{j})-\min_{i}{\operatorname{Re}(z_{i})})/(\max_{i}{\operatorname{Re}(z_{i})}-\min_{i}{\operatorname{Re}(z_{i})}),\quad
j=1,...,n$ (1) $\displaystyle\operatorname{Im}(\tilde{z_{j}})$
$\displaystyle=(\operatorname{Im}(z_{j})-\min_{i}{\operatorname{Im}(z_{i})})/(\max_{i}{\operatorname{Im}(z_{i})}-\min_{i}{\operatorname{Im}(z_{i})}),\quad
j=1,...,n$ (2)

This normalization approach yields
$\operatorname{Re}(\tilde{z}_{j}),\operatorname{Im}(\tilde{z}_{j})\in[0,1]$
but leads to the loss of the information contained in offset, magnitude and
magnitude ratio $|\operatorname{Re}(z_{i})|/|\operatorname{Im}(z_{i})|$. As a
consequence, the slope associated with mainly linear spectra is lost,
resulting in many spectra that are close to the diagonal of the associated
subfigure in Fig. 2, but might look very different when plotted without
rescaling.

Figure 2: Visualization of normalized impedances of the data set for each
circuit class.

While Fig. 2 does not show any frequency information, it shows the shape of
the spectra associated with each circuit. The L-R-RCPE and Rs-Ws circuits have
shapes that differ significantly from the other classes. Furthermore, the
shapes are increasingly diverse, with an increasing number of RC and RCPE
elements. Another critical observation is that a significant overlap of shapes
can be observed with semi-circle-shaped and linearly-shaped spectra in all
classes, indicating potential classification issues.

Visualizations of the data set in lower dimensions with Uniform Manifold
Approximation and Projection (UMAP) are included in the Supplementary
Information (B). The UMAP analysis shows a significant overlap of spectra
between different classes. The amount of overlap, however, depends on the
classes that are compared. The fact that UMAP didn’t find a lower dimensional
manifold that separates the spectra associated with different ECMs well
indicates that the supervised learning problem on this data set is
challenging, further supporting the intuition gained from Fig. 2.

##  3 The Challenge

The purpose of the EIS challenge of the hackathon was to automate the
classification of appropriate ECMs based on the data set. In particular, the
challenge was to create a model that could predict the ECM class in the test
data set as accurately as possible. In this article, we use the F1-score’s
weighted average to compare the results of the classification task. We also
report the unweighted F1-score’s average and the unweighted and weighted
average of the recall.

In addition, the automation of guesses for ECM parameters was part of the
challenge. The guess of the parameters does not need to yield a perfect fit
but should be a good initial starting point for more traditional parameter
optimization. Due to the difficulty of the classification task, this article
focuses entirely on the classification. The Supplementary Information of this
article contains more information about parameter estimation and suggests a
machine learning approach.

Code for reading the data file, basic EIS modeling, and plotting of EIS models
versus the recorded data points was provided to the participants to accelerate
their efforts. The EIS model and parameter scoring code are also included in
the package so participants can easily self-assess their work. This code can
be found in the GitHub repository associated with this article. As well,
participants were given examples of how to install the required code
environment – Python, Jupyter, and the SciPy stack – using Docker, Anaconda,
or Poetry. Jupyter Notebooks containing examples of how to implement the
provided code for reading, modeling, plotting, and scoring were provided.

##  4 Classification approaches

This section presents the different classification approaches. The models and
their performance was studied using a random 80% train split. The prediction
accuracies are subsequently reported on the remaining 20% of the data that was
held out for testing.

###  4.1 Random Forest: The baseline model

Reliable baseline models are essential to quantify performance gains from more
complicated models. For example, Linear Discriminant Analysis (LDA) and
Quadratic Discriminant Analysis (QDA) are linear, static classification
methods traditionally used for analyzing chemical systems or spectral data.
However, these methods do not perform well in classifying EIS data due to the
nonlinearity of the task (cf. Fig. B.1). Therefore nonlinear approaches are
needed. Here, we present a Random Forest (RF) model that learns from the raw
spectra that were preprocessed to a uniform frequency range according to
Section A.2. Furthermore, the spectral data is arranged in a matrix format:

$\mathbf{F}_{\mathrm{Re}}=\begin{bmatrix}\mathrm{Re}(f_{1}(\omega_{1}))&\mathrm{Re}(f_{1}(\omega_{2}))&\cdots&\mathrm{Re}(f_{1}(\omega_{p}))\\\
\mathrm{Re}(f_{2}(\omega_{1}))&\mathrm{Re}(f_{2}(\omega_{2}))&\cdots&\mathrm{Re}(f_{2}(\omega_{p}))\\\
\vdots&\vdots&\ddots&\vdots\\\
\mathrm{Re}(f_{n}(\omega_{1}))&\mathrm{Re}(f_{n}(\omega_{2}))&\cdots&\mathrm{Re}(f_{n}(\omega_{p}))\\\
\end{bmatrix}$ (3)
$\mathbf{F}_{\mathrm{Im}}=\begin{bmatrix}\mathrm{Im}(f_{1}(\omega_{1}))&\mathrm{Im}(f_{1}(\omega_{2}))&\cdots&\mathrm{Im}(f_{1}(\omega_{p}))\\\
\mathrm{Im}(f_{2}(\omega_{1}))&\mathrm{Im}(f_{2}(\omega_{2}))&\cdots&\mathrm{Im}(f_{2}(\omega_{p}))\\\
\vdots&\vdots&\ddots&\vdots\\\
\mathrm{Im}(f_{n}(\omega_{1}))&\mathrm{Im}(f_{n}(\omega_{2}))&\cdots&\mathrm{Im}(f_{n}(\omega_{p}))\\\
\end{bmatrix}$ (4) $\mathbf{X}=\begin{bmatrix}\mathbf{F}_{\mathrm{Re}}\ \
\mathbf{F}_{\mathrm{Im}}\end{bmatrix}$ (5)

where $f_{i}(\omega_{j})$ denotes the impedance corresponding to the frequency
$\omega_{j}$ of a battery with the index $i$, and Im and Re denote the real
and imaginary parts of the impedance, respectively. Each spectrum was scaled
by dividing through its maximum real impedance to achieve a consistent scale
without decoupling the scale of the real impedance from the scale of the
imaginary impedance.

Figure 3: Confusion matrix of predictions on the random 20% data held out for
testing for the RF baseline classification model. Weighted F1-score: 0.38

The hyperparameter optimization of the RF was carried out with extensive
cross-validation. Further details and parameter ranges can be found in the
associated code.

The resulting confusion matrix of the test data in Fig. 3 showcases that most
classes are separated well from one another. However, the model struggles to
differentiate between the L-R-RCPE, L-R-2RCPE, and L-R-3RCPE circuits.
Similarly, the model struggles to distinguish between the 2RCPE, 3RCPE, and
4RCPE models. These errors are physically sensible, as multiple RCPE elements
are often used to fit overlapping peaks. Determining the number of RCPE
elements required to accurately fit the data without overfitting is a key
challenge when analyzing EIS data [40]. However, ECM types that should be
qualitatively much different from one another, such as L-R-nRCPE circuits and
R-Ws circuits, are rarely confused.

###  4.2 Time-series features, XGBoost: A well-performing solution

The best-performing solution treated each EIS spectrum as a multivariate time
series, using the log of frequency as a proxy for time. Classification of time
series data often uses engineered features to extract information from the raw
data, such as the linearity of the trend, the number of obvious peaks, or the
magnitude and phase of periodic fluctuations in the data. To simplify the
procedure for proposing possible features and developing methods to extract
them, the Python library tsfresh was used [41]. This library extracts hundreds
of possible features from time series data and then uses hypothesis testing to
remove irrelevant features prior to model training. The data preprocessing
documented in Sec. A.2 ensured that all impedance data were evenly spaced with
respect to the log of frequency; many of the features generated by tsfresh
assume that data points are evenly spaced in time. The model architecture used
was a gradient-boosted tree implemented by the extreme gradient boosting
(XGBoost) model architecture, chosen for its high performance in many data
science tasks often even without needing substantial hyperparameter
optimization or domain expertise [42]. We optimized the hyperparameters with
random search, yielding minor improvements over the model using the default
hyperparameters.

Figure 4 shows the confusion matrix for the predictions on the test set. As
noted in the explanation of the data set, there are clear groups of equivalent
circuits that are similar to each other, with L-R-nRCPE circuits being easily
confused between them but almost never confused with data labeled as R-Ws or
RC-G-G circuits, similar to the results of the baseline random forest model
but with better performance. The confusion matrix in Fig. 4 shows a clear
improvement over the baseline model (Fig. 3), however, qualitatively, the
confusion is similar.

Figure 4: Confusion matrix of predictions on the random 20% data held out for
testing for the tsfresh-XGBoost classification model. Weighted F1-score: 0.50

Feature importance for the XGBoost classifier model after generation of time
series features using tsfresh is investigated using SHapley Additive
exPlanations (SHAP) [43]. Figure 5 shows the sum of the average SHAP values
for all classes for the top 7 features. Of the top 7 features, the importance
to each class varies; for instance, the number of peaks in the imaginary
impedance has very high importance, on average, for predicting the L-R-RCPE
class but almost no importance for predicting the RCPE-RCPE-RCPE class.
Another example is the R-value feature of the real impedance (ranked 1st) that
has high importance for the diffusion-dominated ECM classes (RC-G-G and Rs_Ws)
but little importance to all other classes. This agrees with the physical
understanding of diffusion processes generating approximately linear segments
in the impedance spectra that are expected to be fitted well by a linear
model. Many of the top 7 features are related to the shape of the impedance
spectra. Examples are features related to the linear trend (ranked 1st and
7th) and the number of imaginary impedance peaks (ranked 2nd). A key takeaway
from the analysis of SHAP values is that there are features that tend to be
important for only one or two circuit types (e.g., features ranked 1st, 2nd,
3rd, and 7th), while other features are important for many circuits (e.g.,
features ranked 4th, 5th, and 6th). However, no single feature alone can
classify all spectra accurately. Last, it should be noted that the SHAP values
of the features are relatively close. For example, the mean absolute SHAP
value of the 7th ranked feature is almost the same as the mean absolute SHAP
value of the 4th ranked feature and the top 3 ranked features. Therefore, it
is expected that relatively small changes in the data set can lead to shuffled
rankings.

Figure 5: Average feature importance for each class calculated using SHAP on test set predictions for the tsfresh-XGBoost classification model.  Table 3: Description of feature in Fig. 5 according to the tsfresh documentation [44]. For more information, we refer to the tsfresh documentation that describes the individual functions that calculate the features. SHAP Ranking |  Feature Name |  Description, from [44]
---|---|---
1 |  zreal__agg_linear_trend__attr_"rvalue"__chunk_len_10__f_agg_"max" |  R-value of linear least-squares regression for values of the time series that were aggregated over chunks versus the sequence from 0 up to the number of chunks minus one, maximum value aggregation
2 |  zimag__number_peaks__n_1 |  Number of peaks of at least support 1 in the time series zimag
3 |  zreal__energy_ratio_by_chunks__num_segments_10__segment_focus_9 |  Sum of squares of chunk 9 out of 10 chunks expressed as a ratio with the sum of squares over the whole series zreal
4 |  zreal__ar_coefficient__coeff_1__k_10 |  First coefficient of an unconditional maximum likelihood of an autoregressive AR(k=10) process.
5 |  zreal__ar_coefficient__coeff_0__k_10 |  Constant coefficient of an unconditional maximum likelihood of an autoregressive AR(k=10) process.
6 |  zreal__minimum |  Lowest value of the time series zreal
7 |  zreal__agg_linear_trend__attr_"rvalue"__chunk_len_10__f_agg_"min |  R-value of linear least-squares regression for values of the time series that were aggregated over chunks versus the sequence from 0 up to the number of chunks minus one, minimum value aggregation

Figure 6 shows SHAP values for every observation by class for each of the top
5 features in Fig. 5. This figure helps to explain model behavior in more
detail. For instance, for the L-R-RCPE circuit, a high value for the number of
peaks in the imaginary impedance has a very large negative SHAP value. This
reflects domain knowledge, suggesting that impedance spectra with multiple
obvious peaks should have more than one RC or RCPE element and thus would not
be modeled by the L-R-RCPE circuit. Similarly, Fig. 5 outlined that the RC-G-G
and Rs-Ws circuits have a strong dependence on the linearity of the real
impedance as mentioned previously. Note that the features plotted here may not
necessarily correspond to the most important features for each class on their
own; rather, we are just plotting the top 5 average features to simplify
comparison.

Figure 6: Feature-specific SHAP values from the top 5 average features in Fig.
5 segregated by class for the tsfresh-XGBoost classification model. Each data
point corresponds to an observation from the test split. Overlapping points
are dispersed to represent the density of values. The x-axis of each plot is
the SHAP value for the feature denoted on the y-axis. The features on the
y-axis are numbered by their order in Fig. 5. Points are colored by the value
of the feature, with blue corresponding to low values and pink corresponding
to high values.

###  4.3 Convolutional Neural Network: The creative approach

Convolutional Neural Networks (CNNs) are commonly used to analyze image data
and can learn complex features and relationships. In the past, efforts using
artificial neural networks to analyze large amounts of EIS data, without
humans having to choose initial parameter values for the equivalent circuits,
have been employed. Buteau and Dahn used an inverse model parameterized with a
convolutional neural network over a data set containing 100,000 impedance
spectra [40]. Rastegarpanah et al. [45] developed a rapid neural network
starting with a single hidden layer baseline model, optimized by a Gaussian
process hyperparameter scheme, to estimate the state of health of Nissan Leaf
2011 battery modules using a data set of 106 samples.

Here we present a CNN model for classifying ECMs. This approach is motivated
by the fact that experts also look at impedance spectra visually because the
shape of a spectrum is essential to determine an appropriate ECM. Furthermore,
the SHAP feature analysis in Sec. 4.2 showed that features related to the
shape of the impedance spectra are important for ECM classification. The
preprocessed impedance spectra were visualized in a Nyquist plot. The x-axis
corresponds to the real part and the y-axis to the imaginary part of the
impedance.

Figure 7: Different EIS spectra interpolated to a common frequency range
visualized in a Nyquist plot.

Figure 7 shows a small subset of the generated images. Each spectrum can be
interpreted as a battery’s signature. While many spectra look very different,
similar patterns can be identified. There are similarities between the EIS
images and the MNIST digit classification data set for which many proven,
easy-to-use architectures exist. For a proof of concept CNN, we decided on a
simple network that yielded a high performance on the MNIST data set [46]. We
modified the kernel and stride sizes of the first layer to account for the
image resolution and structure. To keep training time low, we used a
resolution of 56$\times$56 pixels.

Figure 8: Confusion matrix of predictions on the random 20% data held out for
testing for the CNN classification model. Weighted F1-score: 0.32

The confusion matrix shown in Fig. 8 shows a similar pattern to Figs. 3 and 4.
The F1-score of 0.33 is lower than for the other models. Nevertheless, this
result is a promising proof of concept, given that the network was only
slightly adapted, and the frequency and magnitude information is not
considered. The same CNN architecture was tested using images with colored
lines encoding the frequency information. However, the F1-score improvements
were not statistically significant. This suggests that the model with the
tested architecture cannot account for the frequency information. How to
incorporate the frequency and magnitude information into the model and how to
design suitable CNN architectures for classifying ECMs remain open questions.

##### Lessons from a Transfer Learning Approach

Transfer learning in the context of (deep) neural networks refers to using a
network with a defined architecture that was trained on one data set for
another somehow related application. The idea is that the embeddings learned
by the network will also be helpful for the new task. During the BatteryDEV
hackathon, a transfer learning approach based on the MobileNetV2 architecture
was suggested, see Supplementary Information (D). However, this approach was
not (yet) successful. The computational costs to handle a deep network like
MobileNetV2 are high. Consequently, it is costly to experiment and tune the
model. Furthermore, the original training data of the MobileNetV2 were color
images of objects, and the resulting embeddings generated by the layers close
to the final layer of the network, do not work well to classify EIS spectra.
One explanation is that the statistics of (natural) images follow special
distributions [47], which are very different from the sparse nature of the EIS
data. Consequently, the embeddings learned from the training data of the
MobileNetV2 do not suit the EIS classification task. A possible solution for
this issue would be to allow for retraining the parameters of the MobileNetV2.
However, this is non-trivial given the small amount of fewer than 10k spectra
and was thus not pursued further.

###  4.4 Comparison of Classification Results

The classification results show that the highest prediction accuracy was
obtained by the tsfresh-XGBoost approach, clearly outperforming the other two
models. However, the CNN approach was only investigated as a proof of concept,
and there is still potential for further performance improvements.

Table 4: Classification results, (results for unfiltered data). The reported accuracies for the CNN are average accouracies of ten trained networks. Approach | F1-score, macro | F1-score, weighted | Recall, macro | Recall, weighted
---|---|---|---|---
RF Baseline | 0.40 (0.43) | 0.38 (0.41) | 0.41 (0.45) | 0.40 (0.42)
tsfresh-XGBoost | 0.52 (0.54) | 0.50 (0.52) | 0.52 (0.54) | 0.50 (0.52)
CNN | 0.35 (0.36) | 0.32 (0.33) | 0.36 (0.37) | 0.32 (0.34)

Furthermore, Tab. 4 shows that the models trained on the entire data set
without filtering out any spectra performed slightly better than their
counterparts that were only trained on the data that passed the filter
criteria. First, many removed spectra were associated with the L-R-RCPE and
Rs-Ws circuits (c.f. Tab.1). In addition, the filtered-out spectra are quite
different from the other spectra. The models based on the entire data thus
managed to classify them slightly better on average than the other spectra.
However, the filtered-out spectra are mostly unphysical and unlikely to be
observed from a real battery. Thus, the models based on the unlabeled data set
picked up relationships that are not expected to generalize beyond this data
set.

Figure 9: Confusion matrix of normalized impedance spectra associated with the
XGB-tsfresh model.

Although very different, the three investigated approaches share similar
patterns in their confusion matrices (compare Figs. 3, 4, and 8). All
approaches struggle to distinguish the L-R-RCPE, L-R-2RCPE, and L-R-3RCPE
circuits. Similarly, the classification methods struggle to distinguish the
2RCPE, 3RCPE, and 4RCPE circuits. Furthermore, there is high confusion between
the L-R-3RCPE and 4RCPE circuit classes. The other combinations of L-R-nRCPE
and nRCPE circuits also show higher levels of confusion relative to the
combinations not mentioned here. A potential explanation is that the above-
mentioned ECMs can generate similar spectra. Figure 9 supports this claim
because many misclassified spectra show patterns that also agree well with
other circuit types. For example, the spectra classified as L-R-RCPE but
associated with RCPE-RCPE and those classified as RCPE-RCPE but associated
with L-R-RCPE look similar to a human observer. Also, many of the
misclassified spectra have a semi-circle shape which can be generated by
different ECMs, hinting at potential identifiability issues. A key observation
is that the patterns shared by the columns of Fig. 9 (i.e., the spectra that
were classified to be associated with the same circuit) appear to be more
similar than the patterns shared by the rows (i.e., the spectra that are
associated with the same circuit). However, Fig. 9 suffers from the same
drawbacks as Fig. 2 (i.e., loss of magnitude, magnitude ratio, and frequency
information).

##  5 Discussion

While broad, this EIS data set does not represent the whole variety of
impedance measurements observed from LIBs. The fact that the underlying ECM
parameters were drawn from independent distributions results in a data set
containing spectra that are unlikely to be observed by a real battery or
unphysical. While we addressed this drawback by filtering out spectra, we
suspect spectra remain in the data set that would be unlikely to be observed
during the R&D of new battery materials. Furthermore, certain parameter
combinations of models with many parameters might generate spectra that could
likely be modeled by simpler ECMs.

The choice of ECM classes and parameter bounds was informed by the R&D of
solid-state batteries. The electrode materials generally used by solid-state
and liquid-electrolyte batteries are similar; however, solid electrolytes’
diffusivity measurements include grain-boundary and bulk diffusion effects,
while liquid electrolytes have no grain-boundary effects, which can lead to
differences in impedance spectra. Given the complexity of the systems to which
ECMs modeling is typically applied, automating the parameterization of ECMs
would be helpful to automate ECM modeling for non-experts or to process big
data of impedance spectra. However, it is uncertain how well the trained
models presented in this article would perform on different data sets, e.g.,
measured spectra from solid-state batteries or measured spectra from liquid
electrolyte batteries. Nevertheless, the proposed approaches are flexible and
can be used to learn from other EIS data sets.

A further limitation of this data set is that it limits the supervised
learning problem to predicting one of the nine possible ECM types. A more
generally useful result would be the generation of candidate ECMs in an
unsupervised learning approach, which could then learn from data sets such as
this to then propose useful ECMs on new systems. Furthermore, the model
selection and its parameter estimation are preferably performed in one go, as
the motivation of ECM analysis is to parameterize and quantify the raw
impedance spectra to comparable variables. Although a classification suggests
which model to select, it does not provide the parameter values or initial
guess for its fitting process.

To this end, the second stage of the BatteryDEV hackathon was supposed to be
the parameter estimation of ECMs. Still, the limited duration of the hackathon
hindered the complete exploration of this direction. The parameterization was
defined to estimate best-fit circuit parameters. Preliminary results of the
regression task with the tsfresh-XGBoost approach and comparison to the
parameters provided by QS are included in the Supplementary Information (C).
Furthermore, a Bayesian inference approach was conceptualized; see
Supplementary Information (E).

##  6 Conclusion

The EIS classification challenge led to the exploration and development of
novel ECM classification methods. The presented approaches cut the time
engineers spend on model selection for electrochemical impedance spectra,
allowing them to focus on modeling and making better conclusions. The best-
performing model used the tsfresh library to automatically calculate features
of the impedance spectra and an XGBoost model for the classification of EIS
spectra into the appropriate ECM class. The RF model based on the raw spectral
data performed slightly worse. The CNN approach showed that CNNs can classify
EIS spectra. However, challenges remain to apply CNNs to accurately classify
impedance spectra. Future studies can refine and build upon the techniques and
benchmarks described in this article.

With this article, we publish the analyzed data set of 9,300 impedance spectra
provided by QS. The software and data for this article are available as open
source in the corresponding GitHub repository. This work demonstrates how
companies can contribute to and leverage open-source innovation. We hope that
work will pave the way for more such collaborations.

## Data and Code Availability

The data and code are available on the corresponding GitHub repository:
https://github.com/BatteryDEV/AutoECM. QuantumScape (QS) provided the EIS data
contained in the repository. The first data set comprises approximately 9,300
synthetic spectra with the associated Equivalent Circuit Model (ECM). The
second data set contains approximately 19,000 unlabeled spectra consisting of
about $80\%$ synthetic and $20\%$ measured data. The parameter ranges for all
synthetic data are informed by the R&D of QS. The measured spectra are from a
range of different materials, with some replicate measurements at different
temperatures, and/or State-Of-Charge (SOC), and/or State-Of-Health (SOH). The
code comes with an open-source MIT license, and the data are available openly
under the terms of the CC BY license.

## Author Contributions

Joachim Schaeffer: Formal Analysis, Funding acquisition, Methodology, Project
administration, Supervision, Software, Writing; Paul Gasper: Formal Analysis,
Methodology, Software, Writing; Masaki Adachi: Funding acquisition, Project
administration, Software, Writing; Raymond Gasper: Funding acquisition,
Project Administration, Software; Esteban Garcia-Tamayo: Formal Analysis,
Methodology, Writing; Juan Pablo Gaviria-Cardona: Formal Analysis,
Methodology, Software; Simon Montoya-Bedoya: Formal Analysis, Methodology,
Writing, Software; Richard D. Braatz: Writing – review & editing; Rolf
Findeisen: Writing – review & editing; Anoushka Bhutani: Funding acquisition,
Project Administration, Software, Writing; Andrew Schiek: Software, Formal
Analysis, Writing; Rhys Goodall: Software, Formal Analysis, Writing – review &
editing; Simon Engelke: Funding acquisition, Project administration,
Supervision, Writing.

## Acknowledgments

The first BatteryDEV hackathon was supported through the 10toGO hackathon
initiative by Volkswagen and Microsoft. The second BatteryDEV hackathon had
sponsors from industries spanning consumer electronics, power tools,
utilities, HR, recycling, startups, and academia. We thank QuantumScape for
making their EIS data set available; in particular, Tim Holme for supporting
the BatteryDEV initiative and the effort to make these data publicly
available. Furthermore, we thank Ryan Lu, Krishna Ayer, and Michael Plews for
their assistance during the office hours of the hackathon. We thank the
Battery Associates team for supporting the hackathon.

Parts of the work for this paper were done during Joachim Schaeffer’s time at
the Massachusetts Institute of Technology.

Paul Gasper, Rhys Goodall, and Andrew Schiek thank the efforts and time for
the rest of ’Team Battmen’ from the BatteryDev 2022 competition: Tushar
Deshai, and Hugo Leduc.

Likewise, appreciation for the time and effort from the rest of
BatteryDev2022’s runner-up ’Team ejjn’: Juan E. Betancur, Juan P. Tamayo,
Nicolas Montoya-Escobar and Michael Guzman-De Las Salas.

## Funding

Financial support for Joachim Schaeffer’s time at the Massachusetts Institute
of Technology is acknowledged by a fellowship within the IFI program of the
German Academic Exchange Service (DAAD), funded by the Federal Ministry of
Education and Research (BMBF).

Paul Gasper and Andrew Schiek are supported by the Assistant Secretary for
Energy Efficiency and Renewable Energy, Office of Vehicle Technologies of the
U.S. Department of Energy (DOE) through the Machine Learning for Accelerated
Life Prediction & Cell Design program, technology manager Dr. Simon Thompson.
The National Renewable Energy Laboratory is operated by Alliance for
Sustainable Energy under Contract No. DE-AC36-08GO28308 for the U.S.
Department of Energy. The views expressed in the article do not necessarily
represent the views of the DOE or the U.S. Government. The U.S. Government
retains and the publisher, by accepting the article for publication,
acknowledges that the U.S. Government retains a nonexclusive, paid-up,
irrevocable, worldwide license to publish or reproduce the published form of
this work, or allow others to do so, for U.S. Government purposes.

## Competing Interests

Masaki Adachi is an employee of Toyota Motor Corporation and is the founder of
Inferable Energy OÜ and is affiliated with the University of Oxford. Rhys
Goodall is an employee of Chemix.ai. Simon Engelke is the founder of Battery
Associates. Esteban Garcia-Tamayo is an employee of Titan Advanced Energy
Solutions.

## Inclusion and Diversity

We support inclusive, diverse, and equitable conduct of research.

## References

  * [1] Ulrike Krewer, Fridolin Röder, Eranda Harinath, Richard D Braatz, Benjamin Bedürftig and Rolf Findeisen  “Dynamic models of Li-ion batteries for diagnosis and operation: A review and perspective”  In _Journal of the Electrochemical Society_ 165.16 IOP Publishing, 2018, pp. A3656
  * [2] Shangshang Wang, Jianbo Zhang, Oumaı̈ma Gharbi, Vincent Vivier, Ming Gao and Mark E Orazem  “Electrochemical impedance spectroscopy”  In _Nature Reviews Methods Primers_ 1.1 Nature Publishing Group, 2021, pp. 41  DOI: 10.1038/s43586-021-00039-w
  * [3] Woosung Choi, Heon-Cheol Shin, Ji Man Kim, Jae-Young Choi and Won-Sub Yoon  “Modeling and applications of electrochemical impedance spectroscopy (EIS) for lithium-ion batteries”  In _Journal of Electrochemical Science and Technology_ 11.1 The Korean Electrochemical Society, 2020, pp. 1–13
  * [4] D Andre, M Meiler, K Steiner, Ch Wimmer, T Soczka-Guth and DU Sauer  “Characterization of high-power lithium-ion batteries by electrochemical impedance spectroscopy. I. Experimental investigation”  In _Journal of Power Sources_ 196.12 Elsevier, 2011, pp. 5334–5341
  * [5] Uwe Westerhoff, Kerstin Kurbach, Frank Lienesch and Michael Kurrat  “Analysis of lithium-ion battery models based on electrochemical impedance spectroscopy”  In _Energy Technology_ 4.12, 2016, pp. 1620–1630  DOI: 10.1002/ente.201600154
  * [6] Seyed Mohammad Rezaei Niya and Mina Hoorfar  “Study of proton exchange membrane fuel cells using electrochemical impedance spectroscopy technique–A review”  In _Journal of Power Sources_ 240 Elsevier, 2013, pp. 281–293
  * [7] Amin Shiralizadeh Dezfuli, Mohammad Reza Ganjali, Hamid Reza Naderi and Parviz Norouzi  “A high performance supercapacitor based on a ceria/graphene nanocomposite synthesized by a facile sonochemical method”  In _RSC Advances_ 5 The Royal Society of Chemistry, 2015, pp. 46050–46058  DOI: 10.1039/C5RA02957K
  * [8] Pier Luigi Bonora, Flavio Deflorian and Lorenzo Fedrizzi  “Electrochemical impedance spectroscopy as a tool for investigating underpaint corrosion”  In _Electrochimica Acta_ 41.7-8 Elsevier, 1996, pp. 1073–1082
  * [9] Edward P. Randviir and Craig E. Banks  “Electrochemical impedance spectroscopy: An overview of bioanalytical applications”  In _Anal. Methods_ 5 The Royal Society of Chemistry, 2013, pp. 1098–1115  DOI: 10.1039/C3AY26476A
  * [10] Xinming Qian, Ningyu Gu, Zhiliang Cheng, Xiurong Yang, Erkang Wang and Shaojun Dong  “Impedance study of (PEO)10LiClO4–Al2O3 composite polymer electrolyte with blocking electrodes”  In _Electrochimica Acta_ 46.12 Elsevier, 2001, pp. 1829–1836
  * [11] Marc Oldenburger, Benjamin Beduerftig, Andreas Gruhle, Florian Grimsmann, Ernst Richter, Rolf Findeisen and Andreas Hintennach  “Investigation of the low frequency Warburg impedance of Li-ion cells by frequency domain measurements”  In _Journal of Energy Storage_ 21 Elsevier, 2019, pp. 272–280
  * [12] Nobuhiro Ogihara, Shigehiro Kawauchi, Chikaaki Okuda, Yuichi Itou, Yoji Takeuchi and Yoshio Ukyo  “Theoretical and experimental analysis of porous electrodes for lithium-ion batteries by electrochemical impedance spectroscopy using a symmetric cell”  In _Journal of The Electrochemical Society_ 159.7 IOP Publishing, 2012, pp. A1034
  * [13] D.. Abarbanel, K.. Nelson and J.. Dahn  “Exploring impedance growth in high voltage NMC/graphite Li-ion cells using a transmission line model”  In _Journal of The Electrochemical Society_ 163.3 The Electrochemical Society, 2015, pp. A522–A529  DOI: 10.1149/2.0901603jes
  * [14] Yunwei Zhang, Qiaochu Tang, Yao Zhang, Jiabin Wang, Ulrich Stimming and Alpha A. Lee  “Identifying degradation patterns of lithium ion batteries from impedance spectroscopy using machine learning”  In _Nature Communications_ 11.1, 2020, pp. 1706  DOI: 10.1038/s41467-020-15235-7
  * [15] Kristen A. Severson et al.  “Data-driven prediction of battery cycle life before capacity degradation”  In _Nature Energy_ 4.5 Nature Publishing Group, 2019, pp. 383–391
  * [16] Antti Aitio and David A. Howey  “Predicting battery end of life from solar off-grid system field data using machine learning”  In _Joule_ 5.12, 2021, pp. 3204–3220  DOI: 10.1016/j.joule.2021.11.006
  * [17] Mikael Kedzierski, Mathilde Falcou-Préfol, Marie Emmanuelle Kerros, Maryvonne Henry, Maria Luiza Pedrotti and Stephane Bruzaud  “A machine learning algorithm for high throughput identification of FTIR spectra: Application on microplastics collected in the Mediterranean Sea”  In _Chemosphere_ 234 Elsevier, 2019, pp. 242–251
  * [18] Joachim Schaeffer and Richard D. Braatz  “Latent Variable Method Demonstrator – Software for understanding multivariate data analytics algorithms”  In _Computers & Chemical Engineering_ 167, 2022, pp. 108014  DOI: 10.1016/j.compchemeng.2022.108014
  * [19] Nicole M. Ralbovsky and Igor K. Lednev  “Towards development of a novel universal medical diagnostic method: Raman spectroscopy and machine learning”  In _Chemical Society Reviews_ 49.20 Royal Society of Chemistry, 2020, pp. 7428–7453
  * [20] Félix Lussier, Vincent Thibault, Benjamin Charron, Gregory Q. Wallace and Jean-Francois Masson  “Deep learning and artificial intelligence methods for Raman and surface-enhanced Raman scattering”  In _TrAC Trends in Analytical Chemistry_ 124 Elsevier, 2020, pp. 115796
  * [21] Yuta Suzuki, Hideitsu Hino, Takafumi Hawai, Kotaro Saito, Masato Kotsugi and Kanta Ono  “Symmetry prediction and knowledge discovery from X-ray diffraction patterns using an interpretable machine learning approach”  In _Scientific Reports_ 10.1 Springer, 2020, pp. 1–11  DOI: 10.1038/s41598-020-77474-4
  * [22] Shan Zhu, Xinyang Sun, Xiaoyang Gao, Jianrong Wang, Naiqin Zhao and Junwei Sha  “Equivalent circuit model recognition of electrochemical impedance spectroscopy via machine learning”  In _Journal of Electroanalytical Chemistry_ 855 Elsevier, 2019, pp. 113627
  * [23] Zhaoyang Zhao, Yang Zou, Peng Liu, Zhaogui Lai, Lei Wen and Ying Jin  “EIS equivalent circuit model prediction using interpretable machine learning and parameter identification using global optimization algorithms”  In _Electrochimica Acta_ 418, 2022, pp. 140350  DOI: 10.1016/j.electacta.2022.140350
  * [24] Pumidech Puthongkham, Supacha Wirojsaengthong and Akkapol Suea-Ngam  “Machine learning and chemometrics for electrochemical sensors: moving forward to the future of analytical chemistry”  In _Analyst_ 146.21 Royal Society of Chemistry, 2021, pp. 6351–6364
  * [25] V. Bongiorno, S. Gibbon, E. Michailidou and M. Curioni  “Exploring the use of machine learning for interpreting electrochemical impedance spectroscopy data: Evaluation of the training dataset size”  In _Corrosion Science_ 198, 2022, pp. 110119  DOI: https://doi.org/10.1016/j.corsci.2022.110119
  * [26] Ying Xu, Yang Jiang, Chao Li, Yangzi Chen and Yong Yang  “Integration of an XGBoost model and EIS detection to determine the effect of low inhibitor concentrations on E. coli”  In _Journal of Electroanalytical Chemistry_ 877 Elsevier, 2020, pp. 114534
  * [27] Penelope K. Jones, Ulrich Stimming and Alpha A. Lee  “Impedance-based forecasting of lithium-ion battery performance amid uneven usage”  In _Nature Communications_ 13.1, 2022, pp. 4806  DOI: 10.1038/s41467-022-32422-w
  * [28] Masaki Adachi, Yannick Kuhn, Birger Horstmann, Michael A Osborne and David A Howey  “Bayesian model selection of lithium-ion battery models via bayesian quadrature” arXiv preprint, https://arxiv.org/abs/2210.17299, 2022  arXiv:2210.17299
  * [29] Masaki Adachi, Satoshi Hayakawa, Saad Hamid, Martin Jørgensen, Harald Oberhauser and Micheal A. Osborne  “SOBER: Scalable batch Bayesian optimization and quadrature using recombination constraints” arXiv preprint, https://arxiv.org/abs/2301.11832, 2023  arXiv:2301.11832
  * [30] Masaki Adachi, Satoshi Hayakawa, Martin Jørgensen, Harald Oberhauser and Michael A. Osborne  “Fast Bayesian inference with batch Bayesian quadrature via kernel recombination”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 35, 2022
  * [31] Andrew G. Howard et al.  “MobileNets: Efficient convolutional neural networks for mobile vision applications” arXiv preprint, https://arxiv.org/abs/1704.04861, 2017  arXiv:1704.04861
  * [32] Matthew D. Murbach and Daniel T. Schwartz  “Analysis of Li-Ion battery electrochemical impedance spectroscopy data: An easy-to-implement approach for physics-based parameter estimation using an open-source tool”  In _Journal of The Electrochemical Society_ 165.2 The Electrochemical Society, 2018, pp. A297–A304  DOI: 10.1149/2.1021802jes
  * [33] Jake Huang, Meagan Papac and Ryan O’Hayre  “Towards robust autonomous impedance spectroscopy analysis: A calibrated hierarchical Bayesian approach for electrochemical impedance spectroscopy (EIS) inversion”  In _Electrochimica Acta_ 367, 2021, pp. 137493  DOI: 10.1016/j.electacta.2020.137493
  * [34] Phillip Kollmeyer  “Panasonic 18650PF Li-ion Battery Data”  In _Mendeley Data_ , 2022  URL: https://data.mendeley.com/datasets/wykht8y7tg/1
  * [35] Peyman Mohtat, Suhak Lee, Jason B. Siegel and Anna G. Stefanopoulou  “Reversible and irreversible expansion of lithium-ion batteries under a wide range of stress factors”  In _Journal of The Electrochemical Society_ 168.10 IOP Publishing, 2021, pp. 100520
  * [36] P.. Attia et al.  “Closed-loop optimization of fast-charging protocols for batteries with machine learning”  In _Nature_ 578 Nature Publishing Group, 2020, pp. 397–402
  * [37] Logan Ward et al.  “Principles of the Battery Data Genome” in press  In _Joule_ , 2022  DOI: 10.1016/j.joule.2022.08.008
  * [38] Christopher Sutton et al.  “Crowd-sourcing materials-science challenges with the NOMAD 2018 Kaggle competition”  In _npj Computational Materials_ 5.1, 2019, pp. 111  DOI: 10.1038/s41524-019-0239-3
  * [39] Yoana Fernández Pulido, Cecilio Blanco, David Anseán, Víctor M. García, Francisco Ferrero and Marta Valledor  “Determination of suitable parameters for battery analysis by Electrochemical Impedance Spectroscopy”  In _Measurement_ 106, 2017, pp. 1–11  DOI: https://doi.org/10.1016/j.measurement.2017.04.022
  * [40] Sam Buteau and J.. Dahn  “Analysis of thousands of electrochemical impedance spectra of lithium-ion cells through a machine learning inverse model”  In _Journal of The Electrochemical Society_ 166.8 The Electrochemical Society, 2019, pp. A1611–A1622  DOI: 10.1149/2.1051908jes
  * [41] Maximilian Christ, Nils Braun, Julius Neuffer and Andreas W. Kempa-Liehr  “Time series feature extraction on basis of scalable hypothesis tests (tsfresh–a python package)”  In _Neurocomputing_ 307 Elsevier, 2018, pp. 72–77
  * [42] Tianqi Chen and Carlos Guestrin  “Xgboost: A scalable tree boosting system”  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2016, pp. 785–794
  * [43] Scott M Lundberg and Su-In Lee  “A Unified Approach to Interpreting Model Predictions”  In _Advances in Neural Information Processing Systems_ 30 Curran Associates, Inc., 2017, pp. 4765–4774
  * [44] “Overview on Extracted Features”, 2022  URL: https://tsfresh.readthedocs.io/en/latest/text/list_of_features.html
  * [45] Alireza Rastegarpanah, Jamie Hathaway, Mohamed Ahmeid, Simon Lambert, Allan Walton and Rustam Stolkin  “A rapid neural network–based state of health estimation scheme for screening of end of life electric vehicle batteries”  In _Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering_ 235.3, 2021, pp. 330–346  DOI: 10.1177/0959651820953254
  * [46] , 2022  URL: https://www.kaggle.com/code/elcaiseri/mnist-simple-cnn-keras-accuracy-0-99-top-1/
  * [47] Daniel L. Ruderman  “The statistics of natural images”  In _Network: Computation in Neural Systems_ 5.4 IOP Publishing, 1994, pp. 517

## References

  * [48] Leland McInnes, John Healy and James Melville  “Umap: Uniform manifold approximation and projection for dimension reduction”  In _Journal of Open Source Software_ 3.29, 2018, pp. 861  DOI: 10.21105/joss.00861
  * [49] Andrew G. Howard et al.  “MobileNets: Efficient convolutional neural networks for mobile vision applications” arXiv preprint, https://arxiv.org/abs/1704.04861, 2017  arXiv:1704.04861
  * [50] Laith Alzubaidi et al.  “Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions”  In _Journal of Big Data_ 8.1 Springer, 2021, pp. 1–74
  * [51] Carl Rasmussen and Zoubin Ghahramani  “Occam’s razor”  In _Advances in Neural Information Processing Systems_ 13 MIT Press, 2000
  * [52] John Skilling  “Nested sampling for general Bayesian computation”  In _Bayesian Analysis_ 1.4, 2006, pp. 833–859  DOI: 10.1214/06-BA127
  * [53] Masaki Adachi, Satoshi Hayakawa, Martin Jørgensen, Harald Oberhauser and Michael A. Osborne  “Fast Bayesian inference with batch Bayesian quadrature via kernel recombination”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 35, 2022
  * [54] Masaki Adachi, Yannick Kuhn, Birger Horstmann, Michael A Osborne and David A Howey  “Bayesian model selection of lithium-ion battery models via bayesian quadrature” arXiv preprint, https://arxiv.org/abs/2210.17299, 2022  arXiv:2210.17299
  * [55] Masaki Adachi, Satoshi Hayakawa, Saad Hamid, Martin Jørgensen, Harald Oberhauser and Micheal A. Osborne  “SOBER: Scalable batch Bayesian optimization and quadrature using recombination constraints” arXiv preprint, https://arxiv.org/abs/2301.11832, 2023  arXiv:2301.11832

page1

## Supplementary Information for “Machine learning benchmarks for the
classification of equivalent circuit models from electrochemical impedance
spectra”

###  A Data Preprocessing

####  A.1 Filtering

The provided synthetic data set contains spectra that are unphysical or very
unlikely to represent a physical battery or both. Such spectra are part of the
data set because the generating equivalent circuit models rely on parameters
sampled from independent reciprocal distributions as described in Sec. 2.

The labeled data set, $\mathcal{Z}$, contains 9327 individual spectra
$\mathbf{z}\in\mathcal{Z}$ that are vectors of impedances,
$\mathbf{z}\in\mathbb{C}^{n}$.

Table A.1: Filtering Criteria 1\. Remove all impedances $z_{i}$ from each
spectrum $\mathbf{z}$ with:
---
$\imaginary(z_{i})>0$ | Positive imag. impedances
2\. Remove all spectra $\mathbf{z}$ that have:
$|\set{z_{i}}{\imaginary(z_{i})<0}|<10$ | Less than 10 impedances with negative imag. part
$|\max_{i}(-z_{i})|<0.5|\min_{i}(-z_{i})|$ | Negative img. range smaller than half the positive imag. range
$\exists\,i:\real(z_{i})<0$ | Negative real impedances
$\exists\,i,j:i>j$ and $(\real(z_{i})-\real(z_{j}))>a$ | Not "mainly" decreasing real impedance with increasing frequency
$a=-0.03$ |

The filter criteria removed 422 spectra from 9327 spectra in total, thereby
removing approximately $4.5\%$ of the data. Figure A.1 shows the spectra
associated with the L-RCPE circuit that were filtered out. Most of the spectra
in Fig A.1 are mainly in the fourth quadrant of the zreal, -zimag plane or are
close to vertical lines or both.

Figure A.2 shows the filtered-out spectra associated with the other ECMs
circuit types, sorted by circuit type. The bottom seven rows contain only
spectra related to the Rs-Ws circuit. Most of these spectra are unphysical,
with impedances corresponding to neighboring frequency values changing wildly
and negative real impedances or both.

Figure A.1: Impedance spectra associated with the L-RCPE circuit class that
were filtered out based on criteria described in Tab. A.1, 150 spectra were
randomly sampled from the 332 spectra that were filtered out in total. Figure
A.2: Impedance spectra associated with the denoted ECM that were filtered out
based on criteria described in Tab. A.1. The spectra are ordered by ECM type.
L-R-RCPE-RCPE (82 spectra), L-R-RCPE-RCPE-RCPE (13), RC-RC-RCPE-RCPE: 6
spectra), RCPE-RCPE (4 spectra), RCPE-RCPE-RCPE (11 spectra), RCPE-RCPE-RCPE-
RCPE (4 spectra), Rs-Ws (74 spectra).

####  A.2 Frequency Range and Number of Data Points

Figure A.3: Frequency ranges of the various ECM classes. Figure A.4: Number of
measured frequencies of spectra associated with the various ECM classes. The
circles are scaled by the number of spectra that have the same number of
measured frequencies in the respective ECM class.

###  B Visualization of Nonlinear Dimensionality Reduction

Data visualization through dimensionality reduction helps to understand the
data’s structure which can be challenging to observe directly. Figure B.1
shows the entire data set after dimensionality reduction to just two
components using UMAP [48] to empower visualization of the entire data set on
a single axis. Because the components for each circuit type substantially
overlap, the components for each circuit type are plotted as a binned
hexagonal grid. Overall, the color-coded shape of the reduced dimensionality
space is horseshoe-like, and several trends are visible across the circuits.
Some circuits show higher density across the outside edge of the horseshoe
(e.g., L-R-RCPE-RCPE), and others show higher density on the inside edge of
the horseshoe shape (RCPE-RCPE-RCPE-RCPE). Various circuits are more obviously
separable by the side of the horseshoe, with RC-G-G, RC-RC-RCPE-RCPE, and Rs-
Ws circuits having almost no samples on the higher right side of the horseshoe
and dense clustering on the higher left and the L-R-RCPE-RCPE-RCPE circuit
having almost no samples on the upper left side of the horseshoe. The L-R-RCPE
circuit has a very high density at the upper right corner and very few spectra
associated with the rest of the horseshoe. Other circuits show similar trends
as those discussed, but overall, the clustering problem presented by this data
set appears challenging and nonlinear. The overlap of component values after
UMAP dimensionality reduction across all circuit types indicates that the EIS
measurements are not clearly differentiated by their ECM label, and so
identification of the correct ECM label in this supervised learning problem
may not be possible to achieve with high accuracy.

Figure B.1: Two-component UMAP reduction of the data set. Because clusters are
substantially overlapping, component values for each circuit type in the data
set are plotted on separate axes. To help distinguish the density of points,
each axis is plotted as a binned hexagonal grid, where bright regions
correspond to high bin counts and dark regions correspond to low bin counts.
UMAP is conducted after data frequency range preprocessing.

###  C Regression model

An additional challenge of the hackathon was parameter estimation. While it
was suggested that the finetuning of ECM parameters should be done with the
scipy optimization framework curve_fit, the reliable suggestion of initial
circuit parameters is essential.

In the following, we present machine learning results based on a supervised
approach. The results mentioned in this section are based on the original data
set without filtering out any spectra as it was provided for the hackathon. We
present it mainly to outline challenges and inspire others to develop new
approaches. A key challenge is that the values of ECM parameters used to
generate the spectra differ by several orders of magnitude because they were
mainly drawn from reciprocal distributions. This variation of parameter values
across many orders of magnitude makes model training very difficult. The
distribution of the values for all of the parameters for any given circuit
model may be distributed non-normally. This intuition was validated by
training regressors using the tsfresh-XGBoost pipeline and comparing it with a
median dummy model. Here we use the same 80%/20% train/test split as used for
the classification model. Separate models were trained for each circuit model,
and multiple parameter values for each circuit model were predicted using the
MultiOutputRegressor from sklearn. Comparing this broad set of predictions,
the dummy model often performed substantially better on the test data than the
tsfresh-XGBoost pipeline.

Several methods for normalizing the distributions of the target variables were
attempted: logarithmic transform, Yeo-Johnson power transform, and quantile
transform using ten quantiles. Only the quantile transformation performed
reliably better than the dummy model across the circuit types and parameters.
Table LABEL:tab:regression_tsfresh_xgboost_maes reports mean absolute errors
for regression models using the dummy model, the tsfresh-XGBoost model without
any target variable transform, and the tsfresh-XGBoost model with quantile
target transformation are reported for each circuit type and each parameter on
the test split using.

Furthermore, the results of the tsfresh-XGBoost models with and without
quantile transformation change with changing random seed, suggesting that the
regression task is brittle, i.e., minor changes in features and model can lead
to considerable changes in prediction accuracy. The results in Tab.
LABEL:tab:regression_tsfresh_xgboost_maes showcase thus only trends and
difficulties with this task. Another question is how well the predictions
perform as initial guesses in the subsequent ECM parameter optimization.
Further investigation and improvements or novel approaches are needed to
tackle the abovementioned challenges.

Table C.1: Mean absolute error of parameter regression models on test splits for each ECM type, compared between dummy models that use the median values for each parameter in the training split, tsfresh-XGBoost regression models without any target data transformation, and tsfresh-XGBoost regression models using a quantile transformation on the target data. Bold entries correspond to the lowest value for each row. Circuit | Parameter | Dummy | tsfresh-XGBoost | tsfresh-XGBoost-Quantile
---|---|---|---|---
L-R-RCPE | L1 | 1.61E-06 | 1.81E-06 | 9.45E-07
| R1 | 4.58E+01 | 3.62E+01 | 3.69E+01
| R2 | 3.92E+04 | 3.88E+04 | 2.70E+04
| CPE1_t | 1.17E-01 | 1.57E-02 | 1.48E-02
| CPE1_C | 4.67E-03 | 3.60E-03 | 3.59E-03
L-R-RCPE-RCPE | L1 | 1.49E-06 | 1.79E-06 | 1.11E-06
| R1 | 3.71E+01 | 4.54E+01 | 3.79E+01
| R2 | 3.93E+02 | 3.22E+02 | 3.43E+02
| CPE1_t | 1.24E-01 | 1.27E-01 | 1.29E-01
| CPE1_C | 1.87E-06 | 3.57E-06 | 2.59E-06
| R3 | 4.28E+04 | 5.36E+04 | 4.20E+04
| CPE2_t | 1.36E-01 | 2.88E-02 | 2.89E-02
| CPE2_C | 1.47E-02 | 7.69E-03 | 6.96E-03
L-R-RCPE-RCPE-RCPE | L1 | 1.41E-06 | 1.73E-06 | 1.35E-06
| R1 | 3.69E+01 | 5.20E+01 | 4.02E+01
| R2 | 1.90E+02 | 2.10E+02 | 1.73E+02
| CPE1_t | 1.32E-01 | 1.41E-01 | 1.40E-01
| CPE1_C | 1.77E-06 | 3.21E-06 | 2.20E-06
| R3 | 2.71E+03 | 1.63E+03 | 1.80E+03
| CPE2_t | 1.36E-01 | 9.93E-02 | 1.00E-01
| CPE2_C | 5.89E-04 | 7.05E-04 | 5.58E-04
| R4 | 9.87E+04 | 1.05E+05 | 9.38E+04
| CPE3_t | 1.26E-01 | 7.86E-02 | 8.38E-02
| CPE3_C | 3.19E-02 | 3.01E-02 | 1.81E-02
RC-G-G | R1 | 5.18E+01 | 4.81E+01 | 5.27E+01
| C1 | 5.38E-06 | 7.82E-06 | 7.38E-06
| R_g1 | 1.80E+00 | 4.71E+00 | 1.80E+00
| t_g1 | 6.12E-01 | 3.47E+00 | 6.13E-01
| R_g2 | 4.83E+01 | 4.77E+01 | 5.02E+01
| t_g2 | 1.67E+01 | 3.89E+01 | 1.57E+01
RC-RC-RCPE-RCPE | R1 | 2.88E+01 | 3.05E+01 | 3.33E+00
| R2 | 8.98E+00 | 1.05E+01 | 6.24E+00
| R3 | 3.50E+01 | 1.93E+01 | 4.82E+00
| R4 | 6.92E+04 | 2.68E+05 | 7.02E+04
| C2 | 8.52E-06 | 1.27E-05 | 6.04E-06
| CPE3_C | 1.28E-04 | 3.58E-04 | 5.52E-05
| CPE4_t | 1.57E-01 | 1.77E-01 | 7.77E-02
| CPE4_C | 4.31E-01 | 5.57E-01 | 1.72E-01
| C1 | 0.00E+00 | 1.65E-16 | 5.66E-17
| CPE3_t | 0.00E+00 | 8.34E-08 | 2.38E-08
RCPE-RCPE | R1 | 4.24E+01 | 2.56E+01 | 3.29E+01
| R2 | 3.71E+04 | 3.95E+04 | 2.34E+04
| CPE1_t | 1.20E-01 | 1.30E-01 | 1.27E-01
| CPE1_C | 1.16E-03 | 1.36E-03 | 1.58E-03
| CPE2_t | 1.26E-01 | 2.70E-02 | 2.91E-02
| CPE2_C | 1.23E-02 | 9.52E-03 | 4.97E-03
RCPE-RCPE-RCPE | R1 | 1.19E+01 | 1.31E+01 | 1.16E+01
| R2 | 3.29E+02 | 5.24E+02 | 3.06E+02
| R3 | 7.34E+04 | 1.04E+05 | 7.42E+04
| CPE1_t | 1.25E-01 | 1.55E-01 | 1.40E-01
| CPE1_C | 1.74E-06 | 2.60E-06 | 1.74E-06
| CPE2_t | 1.27E-01 | 1.26E-01 | 1.10E-01
| CPE2_C | 5.95E-05 | 1.47E-04 | 5.11E-05
| CPE3_t | 1.23E-01 | 1.15E-01 | 4.71E-02
| CPE3_C | 2.73E-02 | 2.10E-01 | 1.50E-02
RCPE-RCPE-RCPE-RCPE | R1 | 3.93E+01 | 3.99E+01 | 3.74E+01
| R2 | 1.37E+02 | 1.84E+02 | 1.36E+02
| R3 | 2.50E+03 | 2.76E+03 | 2.29E+03
| R4 | 9.05E+04 | 1.52E+05 | 8.22E+04
| CPE1_t | 1.30E-01 | 1.36E-01 | 1.40E-01
| CPE1_C | 4.55E-07 | 2.90E-04 | 4.56E-07
| CPE2_t | 1.31E-01 | 1.26E-01 | 1.34E-01
| CPE2_C | 8.80E-04 | 1.82E-03 | 8.82E-04
| CPE3_t | 1.29E-01 | 1.06E-01 | 1.05E-01
| CPE3_C | 3.50E-03 | 6.71E-03 | 3.62E-03
| CPE4_t | 1.30E-01 | 1.05E-01 | 1.06E-01
| CPE4_C | 2.77E-02 | 3.06E-02 | 2.04E-02
Rs_Ws | R1 | 1.00E+02 | 1.63E+01 | 2.63E+01
| W1_R | 7.83E+02 | 2.59E+02 | 2.40E+02
| W1_T | 1.89E+02 | 1.94E+02 | 2.17E+02
| W1_p | 9.69E-01 | 8.12E-02 | 8.08E-02

###  D CNN with colored images approach: MobileNetV2 architecture

During the BatteryDEV hackathon, one team employed the MobileNetV2
architecture, available as part of the TensorFlow-Slim model library.
MobileNetV2 is a neural network architecture with a lower computational cost
at deployment relative to other CNNs with similar performance. It has been
widely used for mobile and embedded vision applications. This neural network
works based on a depthwise separable convolution, which has two steps. In the
first step, a depthwise convolution is applied to the input layer, whose
output acts as an intermediate set of values that will be the input of a
second step, which consists of a pointwise convolution [49].

In this case, we used interpolated Nyquist plots obtained from the data set as
the input layer for the MobileNetV2, as seen in the first layer of Fig. D.1.

Figure D.1: Architecture of MobileNetV2. MobileNetV2 works based on a
depthwise separable convolution. The bottleneck block increases the size of
the input layer’s representation, allowing the neural network to learn more
abstract and complex relationships.

The preprocessing step of this approach was similar to that described in
Section A.2 for the XGBoost model. The spectra were interpolated to the shared
frequency range (101 Hz – 105 Hz). In addition, as the main reason to use a
deep learning CNN approach is to extract features from images, we constructed
Nyquist plot figures (with real impedance at the x-axis and imaginary
impedance at the y-axis) with an added color feature that could represent the
associated frequency (ranging from yellow tones for high-frequencies to violet
tones for low-frequencies) as can be seen in the example Nyquist plots in Fig.
D.1. Finally, all the images were created in the same size of 128$\times$128
pixels.

Some potential reasons for this CNN approach’s low and fluctuating accuracies
compared to the other methods explored in this article could be the use of a
scattered Nyquist plot instead of solid lines because it could make it harder
for the CNN to extract features of the images. In addition, another challenge
with colored images is to have all the RGB information in a unique image; some
authors have proposed an alternative approach to enhance the performance of
CNN approaches. This alternate method involves achieving augmentations in the
color space of the channels. Separating a channel of a certain color – such as
Red, Green, or Blue – is a simple way to encode information in color [50].
Furthermore, the architecture of CNNs, can be sensitive to the thickness of
the line or dots that represent the EIS spectra. Last the embeddings/features
that we hope to extract from the CNNs are very supposedly very different from
those of the original image classification task of the MobileNETV2. Last, the
amount of data that was available during this competition might not be
sufficient to learn these features reliably.

###  E Bayesian Approach

The following Bayesian approach was not implemented during the hackathon and
is presented here as an idea for future work. Future development in this
direction is also desired as it can answer the regression and classification
tasks in an unsupervised manner. Other metrics can be adopted to overcome
this, such as Occam’s razor [51], which can be computed as marginal likelihood
with the Bayesian paradigm.

##### Bayesian inference

In the context of Bayesian inference, the objective can be formulated as a
parameter estimation task with

$\displaystyle y_{\text{noise}}\sim\mathcal{N}(y;0,\sigma_{n}),$ (6)
$\displaystyle\textbf{y}=\text{ECM}(\textbf{X})+y_{\text{noise}},$ (7)
$\displaystyle\ell(x)=\mathcal{N}(y_{*};\textbf{y},\sigma_{n}\textbf{I}),$ (8)
$\displaystyle p(x|\textbf{y})=\frac{\ell(x)\pi(x)}{Z},$ (9) $\displaystyle
Z=\int\ell(x)\pi(x)\text{d}x,$ (10)

where X is the parameter of the ground truth, y is the observed EIS,
$\sigma_{n}$ is the experimental noise variance, I is the identity matrix,
$\ell(x)$ is the likelihood, $\pi(x)$ is the prior, $p(x|\textbf{y})$ is the
posterior, and $Z$ is the model evidence. Therefore, the estimation of the
parameter is the same as inferring posterior in Eq. (9). The classification
criterion is given by the model evidence in Eq. (10). Such inferences can be
simultaneously made with nested sampling [52] or Bayesian quadrature [53, 54,
55]. However, inferring the evidence from 10,000 models is computationally
expensive. Bayesian quadrature can help to perform such inference tasks.