# A Span Selection Model for Semantic Role Labeling

Hiroki Ouchi2,3 Hiroyuki Shindo1,2 Yuji Matsumoto1,2
1 Nara Institute of Science and Technology
2 RIKEN Center for Advanced Intelligence Project (AIP)
3 Tohoku University
<EMAIL_ADDRESS>{ shindo, matsu<EMAIL_ADDRESS>
###### Abstract

We present a simple and accurate span-based model for semantic role labeling
(SRL). Our model directly takes into account all possible argument spans and
scores them for each label. At decoding time, we greedily select higher
scoring labeled spans. One advantage of our model is to allow us to design and
use span-level features, that are difficult to use in token-based BIO tagging
approaches. Experimental results demonstrate that our ensemble model achieves
the state-of-the-art results, 87.4 F1 and 87.0 F1 on the CoNLL-2005 and 2012
datasets, respectively.

##  1 Introduction

Semantic Role Labeling (SRL) is a shallow semantic parsing task whose goal is
to recognize the predicate-argument structure of each predicate. Given a
sentence and a target predicate, SRL systems have to predict semantic
arguments of the predicate. Each argument is a span, a unit that consists of
one or more words. A key to the argument span prediction is how to represent
and model spans.

One popular approach to it is based on BIO tagging schemes. State-of-the-art
neural SRL models adopt this approach Zhou and Xu (2015); He et al. (2017);
Tan et al. (2018). Using features induced by neural networks, they predict a
BIO tag for each word. Words at the beginning and inside of argument spans
have the “B” and “I” tags, and words outside argument spans have the tag “O.”
While yielding high accuracies, this approach reconstructs argument spans from
the predicted BIO tags instead of directly predicting the spans.

Another approach is based on labeled span prediction Täckström et al. (2015);
FitzGerald et al. (2015). This approach scores each span with its label. One
advantage of this approach is to allow us to design and use span-level
features, that are difficult to use in BIO tagging approaches. However, the
performance has lagged behind that of the state-of-the-art BIO-based neural
models.

To fill this gap, this paper presents a simple and accurate span-based model.
Inspired by recent span-based models in syntactic parsing and coreference
resolution Stern et al. (2017); Lee et al. (2017), our model directly scores
all possible labeled spans based on span representations induced from neural
networks. At decoding time, we greedily select higher scoring labeled spans.
The model parameters are learned by optimizing log-likelihood of correct
labeled spans.

We evaluate the performance of our span-based model on the CoNLL-2005 and 2012
datasets Carreras and Màrquez (2005); Pradhan et al. (2012). Experimental
results show that the span-based model outperforms the BiLSTM-CRF model. In
addition, by using contextualized word representations, ELMo Peters et al.
(2018), our ensemble model achieves the state-of-the-art results, 87.4 F1 and
87.0 F1 on the CoNLL-2005 and 2012 datasets, respectively. Empirical analysis
on these results shows that the label prediction ability of our span-based
model is better than that of the CRF-based model. Another finding is that ELMo
improves the model performance for span boundary identification.

In summary, our main contributions include:

  * •

A simple span-based model that achieves the state-of-the-art results.

  * •

Quantitative and qualitative analysis on strengths and weaknesses of the span-
based model.

  * •

Empirical analysis on the performance gains by ELMo.

Our code and scripts are publicly
available.111https://github.com/hiroki13/span-based-srl

##  2 Model

We treat SRL as span selection, in which we select appropriate spans from a
set of possible spans for each label. This section formalizes the problem and
provides our span selection model.

###  2.1 Span Selection Problem

### Problem Setting

Given a sentence that consists of $T$ words $w_{1:T}=w_{1},\cdots,w_{T}$ and
the target predicate position index $p$, the goal is to predict a set of
labeled spans $Y=\\{\langle i,j,r\rangle_{k}\\}^{|Y|}_{k=1}$.

Input $\displaystyle\text{:}\>X=\\{w_{1:T},p\\},$ Output
$\displaystyle\text{:}\>Y=\\{\langle i,j,r\rangle_{k}\\}^{|Y|}_{k=1}\>\>.$

Each labeled span $\langle i,j,r\rangle$ consists of word indices $i$ and $j$
in the sentence ($1\leq i\leq j\leq T$) and a semantic role label
$r\in\mathcal{R}$.

One simple method to predict $Y$ is to select the highest scoring span $(i,j)$
from all possible spans $\mathcal{S}$ for each label $r$,

$\displaystyle\mathop{\rm
argmax}\limits_{(i,j)\in\mathcal{S}}\>\textsc{Score}_{r}(i,j),\>r\in\mathcal{R}\>\>.$
(1)

Function $\textsc{Score}_{r}(i,j)$ returns a real value for each span
$(i,j)\in\mathcal{S}$ (described in Section 2.2 in more detail). The number of
possible spans $\mathcal{S}$ in the input sentence $w_{1:T}$ is
$\frac{T(T+1)}{2}$, and $\mathcal{S}$ is defined as follows,

$\mathcal{S}=\\{(i,j)\>|\>i,j\in\\{1,\cdots,T\\},i\leq j\\}\>\>.$

Note that some semantic roles may not appear in the sentence. To deal with the
absence of some labels, we define the predicate position span $(p,p)$ as a
Null span and train a model to select the Null span when there is no span for
the label.222Since the predicate itself can never be an argument of its own,
we define the position as the Null span.

### Example

Consider the following sentence with the set of correct labeled spans $Y$.

She1 kept2 a3 cat4

[ A0 ] [ A1 ]

$\displaystyle Y=\\{\>$ $\displaystyle\langle 1,1,\texttt{A0}\rangle,\>\langle
3,4,\texttt{A1}\rangle,$ $\displaystyle\langle
2,2,\texttt{A2}\rangle,\cdots,\langle 2,2,\texttt{TMP}\rangle\>\\}$

The input sentence is $w_{1:4}=\text{``She kept a cat"}$, and the target
predicate position is $p=2$. The correct labeled span $\langle
1,1,\texttt{A0}\rangle$ indicates that the A0 argument is “She”, and $\langle
3,4,\texttt{A1}\rangle$ indicates that the A1 argument is “a cat”. The other
labeled spans $\langle 2,2,*\rangle$ indicate there are no arguments.

All the possible spans in this sentence are as follows,

$\displaystyle\mathcal{S}_{w_{1:4}}=\\{$
$\displaystyle(1,1),(1,2),(1,3),(1,4),(2,2),$
$\displaystyle(2,3),(2,4),(3,3),(3,4),(4,4)\\}\>\>,$

where the predicate span $(2,2)$ is treated as the Null span. Among these
candidates, we select the highest scoring span for each label. As a result, we
can obtain correct labeled spans $Y$.

###  2.2 Scoring Function

As the scoring function for each span in Eq. 1, we model normalized
distribution over all possible spans $\mathcal{S}$ for each label $r$,

$\displaystyle\textsc{Score}_{r}(i,j)$
$\displaystyle=\text{P}_{\theta}(i,j\>|\>r)$
$\displaystyle=\frac{\text{exp}(\text{F}_{\theta}(i,j,r))}{\displaystyle\sum_{(i^{\prime},j^{\prime})\in\mathcal{S}}\text{exp}(\text{F}_{\theta}(i^{\prime},j^{\prime},r))}\>\>,$
(2)

where function $\text{F}_{\theta}$ returns a real value.

We train the parameters $\theta$ of $\text{F}_{\theta}$ on a training set,

$\displaystyle\mathcal{D}$
$\displaystyle=\\{(X^{(n)},Y^{(n)})\\}_{n=1}^{|\mathcal{D}|}\>\>,$
$\displaystyle X$ $\displaystyle=\\{w_{1:T},p\\}\>\>,$ $\displaystyle Y$
$\displaystyle=\\{\langle i,j,r\rangle_{k}\\}_{k=1}^{|Y|}\>\>.$

To train the parameters $\theta$ of $\text{F}_{\theta}$, we minimize the
cross-entropy loss function,

$\displaystyle\mathcal{L}({\theta})$
$\displaystyle=\sum_{(X,Y)\in\mathcal{D}}\ell_{\theta}(X,Y)\>\>,$ (3)
$\displaystyle\ell_{\theta}(X,Y)$ $\displaystyle=\sum_{\langle i,j,r\rangle\in
Y}\text{log}\>\text{P}_{\theta}(i,j|r)\>\>,$

where function $\ell_{\theta}(X,Y)$ is a loss for each sample.

###  2.3 Function $\text{F}_{\theta}$

Function $\text{F}_{\theta}$ in Eq. 2.2 consists of three types of functions;
the base feature function $f_{base}$, the span feature function $f_{span}$ and
the labeling function $f_{label}$ as follows,

$\displaystyle{\bf h}_{1:T}$ $\displaystyle=f_{base}(w_{1:T},p)\>\>,$ (4)
$\displaystyle{\bf h}_{s}$ $\displaystyle=f_{span}({\bf h}_{1:T},s)\>\>,$ (5)
$\displaystyle\text{F}_{\theta}(i,j,r)$ $\displaystyle=f_{label}({\bf
h}_{s},r)\>\>.$ (6)

Firstly, $f_{base}$ calculates a base feature vector ${\bf h}_{t}$ for each
word $w_{t}\in w_{1:T}$. Then, from a sequence of the base feature vectors
${\bf h}_{1:T}$, $f_{span}$ calculates a span feature vector ${\bf h}_{s}$ for
a span $s=(i,j)$. Finally, using ${\bf h}_{s}$, $f_{label}$ calculates the
score for the span $s=(i,j)$ with a label $r$.

Each function in Eqs. 4, 5 and 6 can arbitrarily be defined. In Section 3, we
describe our functions used in this paper.

###  2.4 Inference

The simple argmax inference (Eq. 1) selects one span for each label. While
this argmax inference is computationally efficient, it faces the following two
problematic issues.

(a)

The argmax inference sometimes selects spans that overlap with each other.

(b)

The argmax inference cannot select multiple spans for one label.

In terms of (a), for example, when $\langle 1,3,\texttt{A0}\rangle$ and
$\langle 2,4,\texttt{A1}\rangle$ are selected, a part of these two spans
overlaps. In terms of (b), consider the following sentence.

He came to the U.S. yesterday at 5 p.m.

[A0] [ A4 ] [ TMP ] [ TMP ]

In this example, the label TMP is assigned to the two spans (“yesterday” and
“at 5 p.m.”). Semantic role labels are mainly categorized into (i) core labels
or (ii) adjunct labels. In the above example, the labels A0 and A4 are
regarded as core labels, which indicate obligatory arguments for the
predicate. In contrast, the labels like TMP are regarded as adjunct labels,
which indicate optional arguments for the predicate. As the example shows,
adjunct labels can be assigned to multiple spans.

To deal with these issues, we use a greedy search that keeps the consistency
among spans and can return multiple spans for adjunct labels. Specifically, we
greedily select higher scoring labeled spans subject to two constraints.

Overlap Constraint:

Any spans that overlap with the selected spans cannot be selected.

Number Constraint:

While multiple spans can be selected for each adjunct label, at most one span
can be selected for each core label.

As a precise description of this algorithm, we describe the pseudo code and
its explanation in Appendix A.

##  3 Network Architecture

To compute the score for each span, we have introduced three functions
($f_{base},f_{span},f_{label}$) in Section 2.3. As an instantiation of each
function, we use neural networks. This section describes our neural networks
for each function and the overall network architecture.

###  3.1 BiLSTM-Span Model

Figure 1: Overall architecture of our BiLSTM-span model.

Figure 1 illustrates the overall architecture of our model. The first
component $f_{base}$ uses bidirectional LSTMs (BiLSTMs) Schuster and Paliwal
(1997); Graves et al. (2005, 2013) to calculate the base features. From the
base features, the second component $f_{span}$ extracts span features. Based
on them, the final component $f_{label}$ calculates the score for each labeled
span. In the following, we describe these three components in detail.

### Base Feature Function

As the base feature function $f_{base}$, we use BiLSTMs,

$\displaystyle f_{base}(w_{1:T},p)=\textsc{BiLSTM}(w_{1:T},p)\>\>.$

There are some variants of BiLSTMs. Following the deep SRL models proposed by
Zhou and Xu (2015) and He et al. (2017), we stack BiLSTMs in an interleaving
fashion. The stacked BiLSTMs process an input sequence in a left-to-right
manner at odd-numbered layers and in a right-to-left manner at even-numbered
layers.

The first layer of the stacked BiLSTMs receives word embeddings ${\bf
x}^{word}\in\mathbb{R}^{d^{word}}$ and predicate mark embeddings ${\bf
x}^{mark}\in\mathbb{R}^{d^{mark}}$. As the word embeddings, we can use
existing word embeddings. The mark embeddings are created from the mark
feature which has a binary value. The value is 1 if the word is the target
predicate and 0 otherwise. For example, at the bottom part of Figure 1, the
word “bought” is the target predicate and assigned $1$ as its mark feature.

Receiving these inputs, the stacked BiLSTMs calculates the hidden states until
the top layer. We use these hidden states as the input feature vectors ${\bf
h}_{1:T}$ for the span feature function $f_{span}$ (Eq. 5). Each vector ${\bf
h}_{t}\in{\bf h}_{1:T}$ has $d^{hidden}$ dimensions. We provide a detailed
description of the stacked BiLSTMs in Appendix B.

### Span Feature Function

From the base features induced by the BiLSTMs, we create the span feature
representations,

$\displaystyle f_{span}({\bf h}_{1:T},s)=[{\bf h}_{i}+{\bf h}_{j};{\bf
h}_{i}-{\bf h}_{j}]\>\>,$ (7)

where the addition and subtraction features of the $i$-th and $j$-th hidden
states are concatenated and used as the feature for a span $s=(i,j)$. The
resulting vector ${\bf h}_{s}$ is a $2d^{hidden}$ dimensional vector.

The middle part of Figure 1 shows an example of this process. For the span
$(3,5)$, the span feature function $f_{span}$ receives the $3$rd and $5$th
features (${\bf h}_{3}$ and ${\bf h}_{5}$). Then, these two vectors are added,
and the $5$th vector is subtracted from the $3$rd vector. The resulting
vectors are concatenated and given to the labeling function $f_{label}$.

Our design of the span features is inspired by the span (or segment) features
used in syntactic parsing Wang and Chang (2016); Stern et al. (2017);
Teranishi et al. (2017). While these neural span features cannot be used in
BIO-based SRL models, they can easily be incorporated into span-based models.

### Labeling Function

Taking a span representation ${\bf h}_{s}$ as input, the labeling function
$f_{label}$ returns the score for the span $s=(i,j)$ with a label $r$.
Specifically, we use the following labeling function,

$\displaystyle f_{label}({\bf h}_{s},r)={\bf W}[r]\cdot{\bf h}_{s}\>\>,$ (8)

where ${\bf W}\in\mathbb{R}^{|\mathcal{R}|\times 2d^{hidden}}$ has a row
vector associated with each label $r$, and ${\bf W}[r]$ denotes the $r$-th row
vector. As the result of the inner product of ${\bf W}[r]$ and ${\bf h}_{s}$,
we obtain the score for a span $(i,j)$ with a label $r$.

The upper part of Figure 1 shows an example of this process. The span
representation ${\bf h}_{s}$ for the span $s=(3,5)$ is created from addition
and subtraction of ${\bf h}_{3}$ and ${\bf h}_{5}$. Then, we calculate the
inner product of ${\bf h}_{s}$ and ${\bf W}[r]$. The score for the label A0 is
$2.1$, and the score for the label A1 is $3.7$. In the same manner, by
calculating the scores for all the spans $\mathcal{S}$ and labels
$\mathcal{R}$, we can obtain the score matrix (at the top part of Figure 1).

###  3.2 Ensembling

We propose an ensemble model that uses span representations from multiple
models. Each base model trained with different random initializations has
variance in span representations. To take advantage of it, we introduce a
variant of a mixture of experts (MoE) Shazeer et al. (2017), 333One popular
ensemble model for SRL is the product of experts (PoE) model FitzGerald et al.
(2015); He et al. (2017); Tan et al. (2018). In our preliminary experiments,
we tried the PoE model but it did not improve the performance.

$\displaystyle{\bf h}^{\text{moe}}_{s}$ $\displaystyle={\bf
W}^{\text{moe}}_{s}\cdot\sum^{M}_{m=1}\alpha_{m}\>{\bf h}^{(m)}_{s}\>\>,$ (9)
$\displaystyle f^{\text{moe}}_{label}({\bf h}^{\text{moe}}_{s},r)$
$\displaystyle={\bf W}^{\text{moe}}[r]\cdot{\bf h}^{\text{moe}}_{s}\>\>.$ (10)

Firstly, we combine span representations ${\bf h}^{(m)}_{s}$ from each model
$m\in\\{1,\cdots,M\\}$. ${\bf W}^{\text{moe}}_{s}$ is a parameter matrix and
$\\{\alpha_{m}\\}^{M}_{m=1}$ are trainable, softmax-normalized parameters.
Then, using the combined span representation ${\bf h}^{\text{moe}}_{s}$, we
calculate the score in the same way as Eq. 8. We use the same greedy search
algorithm used for our base model (Section 2.4).

During training, we update only the parameters of the ensemble model, i.e.,
$\\{{\bf W}^{\text{moe}}_{s},{\bf
W}^{\text{moe}},\\{\alpha_{m}\\}^{M}_{m=1}\\}$. That is, we fix the parameters
of each trained model $m$. As the loss function, we use the cross-entropy (Eq.
3).

##  4 Experiments

###  4.1 Datasets

We use the CoNLL-2005 and 2012 datasets444We use the version of OntoNotes
downloaded at: http://cemantix.org/data/ontonotes.html.. We follow the
standard train-development-test split and use the official evaluation
script555The script can be downloaded at: http://www.lsi.upc.edu/
srlconll/soft.html from the CoNLL-2005 shared task on both datasets.

|  | Development | Test WSJ | Test Brown | Test ALL
---|---|---|---|---|---
Emb | Model | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Senna | Crf | 81.7 | 81.3 | 81.5 | 83.3 | 82.5 | 82.9 | 72.6 | 70.0 | 71.3 | 81.9 | 80.8 | 81.4
Span | 83.6 | 81.4 | 82.5 | 84.7 | 82.3 | 83.5 | 76.0 | 70.4 | 73.1 | 83.6 | 80.7 | 82.1
Span (Ensemble) | 85.6 | 82.6 | 84.1 | 86.6 | 83.6 | 85.1 | 78.2 | 71.8 | 74.8 | 85.5 | 82.0 | 83.7
ELMo | Crf | 86.6 | 86.8 | 86.7 | 87.4 | 87.3 | 87.3 | 78.5 | 78.3 | 78.4 | 86.2 | 86.1 | 86.1
Span | 87.4 | 86.3 | 86.9 | 88.2 | 87.0 | 87.6 | 79.9 | 77.5 | 78.7 | 87.1 | 85.7 | 86.4
Span (Ensemble) | 88.0 | 86.9 | 87.4 | 89.2 | 87.9 | 88.5 | 81.0 | 78.4 | 79.6 | 88.1 | 86.6 | 87.4
Table 1:  Experimental results on the CoNLL-2005 dataset, in terms of precision (P), recall (R) and F1. The bold numbers denote the highest precision, recall and F1 scores among all the models. |  | Development | Test
---|---|---|---
Emb | Model | P | R | F1 | P | R | F1
Senna | Crf | 82.8 | 81.9 | 82.4 | 82.9 | 81.9 | 82.4
Span | 84.3 | 81.5 | 82.9 | 84.4 | 81.7 | 83.0
Span (Ensemble) | 86.0 | 83.0 | 84.5 | 86.1 | 83.3 | 84.7
ELMo | Crf | 86.1 | 85.8 | 85.9 | 86.0 | 85.7 | 85.9
Span | 87.2 | 85.5 | 86.3 | 87.1 | 85.3 | 86.2
Span (Ensemble) | 88.6 | 85.7 | 87.1 | 88.5 | 85.5 | 87.0
Table 2:  Experimental results on the CoNLL-2012 dataset. | CoNLL-05 | CoNLL12
---|---|---
| WSJ | Brown | ALL |
Single Model
ELMo-Span | 87.6 | 78.7 | 86.4 | 86.2
He+ 18 | 87.4 | 80.4 | - | 85.5
Peters+ 18 | - | - | - | 84.6
Strubell+ 18 | 83.9 | 72.6 | - | -
Tan+ 18 | 84.8 | 74.1 | 83.4 | 82.7
He+ 17 | 83.1 | 72.1 | 81.6 | 81.7
Zhou+ 15 | 82.8 | 69.4 | 81.1 | 81.3
FitzGerald+ 15 | 79.4 | 71.2 | - | 79.6
Täckström+ 15 | 79.9 | 71.3 | - | 79.4
Toutanova+ 08 | 79.7 | 67.8 | - | -
Punyakanok+ 08 | 79.4 | 67.8 | 77.9 | -
Ensemble Model
ELMo-Span | 88.5 | 79.6 | 87.4 | 87.0
Tan+ 18 | 86.1 | 74.8 | 84.6 | 83.9
He+ 17 | 84.6 | 73.6 | 83.2 | 83.4
FitzGerald+ 15 | 80.3 | 72.2 | - | 80.1
Toutanova+ 08 | 80.3 | 68.8 | - | -
Punyakanok+ 08 | 79.4 | 67.8 | 77.9 | -
Table 3:  Comparison with existing models. The numbers denote F1 scores on
each test set.

###  4.2 Baseline Model

For comparison, as a model based on BIO tagging approaches, we use the BiLSTM-
CRF model proposed by Zhou and Xu (2015). The BiLSTMs for the base feature
function $f_{base}$ are the same as those used in our BiLSTM-span model.

###  4.3 Model Setup

As the base function $f_{base}$, we use 4 BiLSTM layers with 300 dimensional
hidden units. To optimize the model parameters, we use Adam Kingma and Ba
(2014). Other hyperparameters are described in Appendix C in detail.

### Word Embeddings

Word embeddings have a great influence on SRL models. To validate the model
performance, we use two types of word embeddings.

  * •

Typical word embeddings, SENNA666http://ronan.collobert.com/senna/ Collobert
et al. (2011)

  * •

Contextualized word embeddings, ELMo777http://allennlp.org/elmo Peters et al.
(2018)

SENNA and ELMo can be regarded as different types of embeddings in terms of
the context sensitivity. SENNA and other typical word embeddings always assign
an identical vector to each word regardless of the input context. In contrast,
ELMo assigns different vectors to each word depending on the input context. In
this work, we use these word embeddings that have different properties.888In
our preliminary experiments, we also used the GloVe embeddings Pennington et
al. (2014), but the performance was worse than SENNA. These embeddings are
fixed during training.

### Training

As the objective function, we use the cross-entropy $\mathcal{L}_{\theta}$ in
Eq. 3 with L2 weight decay,

$\mathcal{L}_{\theta}=\sum_{(X,Y)\in\mathcal{D}}\ell_{\theta}(X,Y)+\frac{\lambda}{2}||\theta||^{2}\>\>,$
(11)

where the hyperparameter $\lambda$ is the coefficient governing the L2 weight
decay.

###  4.4 Results

We report averaged scores across five different runs of the model training.

Tables 1 and 2 show the experimental results on the CoNLL-2005 and 2012
datasets. Overall, our span-based ensemble model using ELMo achieved the best
F1 scores, 87.4 F1 and 87.0 F1 on the CoNLL-2005 and CoNLL-2012 datasets,
respectively. In comparison with the CRF-based single model, our span-based
single model consistently yielded better F1 scores regardless of the word
embeddings, Senna and ELMo. Although the performance difference was small
between these models using ELMo, it seems natural because both models got much
better results and approached to the performance upper bound.

Table 3 shows the comparison with existing models in F1 scores. Our single and
ensemble models using ELMo achieved the best F1 scores on all the test sets
except the Brown test set.

##  5 Analysis

To better understand our span-based model, we addressed the following
questions and obtained the following findings.

### Questions

(a)

What are strengths and weaknesses of our span-based model compared with the
CRF-based model?

(b)

What aspect of SRL does ELMo improve?

### Findings

(a)

While the CRF-based model is better at span boundary identification (Section
5.1), the span-based model is better at label prediction, especially for A2
(Section 5.2).

(b)

ELMo improves the model performance for span boundary identification (Section
5.1).

In addition, we have conducted qualitative analysis on span and label
representations learned in the span-based model (Section 5.3).

###  5.1 Performance for Span Boundary Identification

|  | CoNLL-05 | CoNLL-12
---|---|---|---
Emb | Model | F1 | diff | F1 | diff
Senna | Span | 86.6 | -0.4 | 87.3 | -0.6
Crf | 87.0 | 87.9
ELMo | Span | 90.5 | -0.7 | 90.3 | -0.6
Crf | 91.2 | 90.9
Table 4:  F1 scores only for span boundary match.

We analyze the results predicted by the single models. We evaluate F1 scores
only for the span boundary match, shown by Table 4. We regard a predicted
boundary $\langle i,j,*\rangle$ as correct if it matches the gold annotation
regardless of its label.

On both datasets, the CRF-based models achieved better F1 than that of the
span-based models. Also, compared with Senna, ELMo yielded much better F1 by
over 3.0. This suggests that a factor of the overall SRL performance gain by
ELMo is the improvement of the model ability to identify span boundaries.

###  5.2 Performance for Label Prediction

|  | CoNLL-05 | CoNLL-12
---|---|---|---
Emb | Model | Acc. | diff | Acc. | diff
Senna | Span | 95.3 | +1.5 | 95.1 | +1.5
Crf | 93.8 | 93.6
ELMo | Span | 96.1 | +0.9 | 95.7 | +1.3
Crf | 95.2 | 94.4
Table 5:  Accuracies only for semantic role labels.

We analyze labels of the predicted results. For labeled spans whose boundaries
match the gold annotation, we evaluate the label accuracies. As Table 5 shows,
the span-based models outperformed the CRF-based models. Also, interestingly,
the performance gap between Senna and ELMo was not so big as that for span
boundary identification.

### Label-wise Performance

| CoNLL-2005 | CoNLL-2012
---|---|---
| Senna | ELMo | Senna | ELMo
Label | Crf | Span | Crf | Span | Crf | Span | Crf | Span
A0 | 89.9 | 90.2 | 93.0 | 93.2 | 89.9 | 90.0 | 92.5 | 92.5
A1 | 83.2 | 83.8 | 89.1 | 89.2 | 84.7 | 85.1 | 88.7 | 89.0
A2 | 70.9 | 73.1 | 80.0 | 81.2 | 78.6 | 79.4 | 83.2 | 84.2
A3 | 64.4 | 71.2 | 78.8 | 78.5 | 61.9 | 62.9 | 69.0 | 70.7
ADV | 59.3 | 61.9 | 68.1 | 67.0 | 63.2 | 63.7 | 67.5 | 67.0
DIR | 43.2 | 47.3 | 56.6 | 54.5 | 54.1 | 52.0 | 61.1 | 59.7
LOC | 58.2 | 60.5 | 68.1 | 68.3 | 65.8 | 65.0 | 72.0 | 72.0
MNR | 61.4 | 61.3 | 66.5 | 67.7 | 64.4 | 65.7 | 70.5 | 71.1
PNC | 57.3 | 60.2 | 68.8 | 67.7 | 18.5 | 13.7 | 20.2 | 16.1
TMP | 81.8 | 82.7 | 86.1 | 86.0 | 82.2 | 82.3 | 86.1 | 86.2
Overall | 81.5 | 82.5 | 86.7 | 86.9 | 82.4 | 82.9 | 85.9 | 86.3
Table 6:  F1 Scores for frequent labels on the development set of the
CoNLL-2005 and 2012 datasets.

Table 6 shows F1 scores for frequent labels on the CoNLL-2005 and 2012
datasets. For A0 and A1, the performances of the CRF-based and span-based
models were almost the same. For A2, the span-based models outperformed the
CRF-based model by about 1.0 F1 on the both datasets. 999The PNC label got low
scores on the CoNLL-2012 dataset in Table 6. Almost all the gold PNC (purpose)
labels are assigned to only the news article domain texts of the CoNLL-2012
dataset. The other 6 domain texts have no or very few PNC labels. This can
lead to the low performance.

### Label Confusion Matrix

Figure 2:  Confusion matrix for labeling errors of our span-based model using
ELMo. Each cell shows the percentage of predicted labels for each gold label.

Figure 2 shows a confusion matrix for labeling errors of the span-based model
using ELMo.101010We have observed the same tendency of labeling confusions
between the models using ELMo and SENNA. Following He et al. (2017), we only
count predicted arguments that match the gold span boundaries.

The span-based model confused A0 and A1 arguments the most. In particular, the
model confused them for ergative verbs. Consider the following two sentences:

People start their own business …
[ A0 ]
.. Congress has started to jump on …
[ A1 ]

where the constituents located at the syntactic subjective position fulfill a
different role A0 or A1 according to their semantic properties, such as
animacy. Such arguments are difficult for SRL models to correctly identify.

Another point is the confusions of A2 with DIR and LOC. As He et al. (2017)
pointed out, A2 in a lot of verb frames represents semantic relations such as
direction or location, which can cause the confusions of A2 with such
location-related adjuncts. To remedy these two problematic issues, it can be a
promising approach to incorporate frame knowledge into SRL models by using
verb frame dictionaries.

###  5.3 Qualitative Analysis on Our Model

### On Span Representations

“$\cdots$ toy makers to move [ across the border ] .”
---
GOLD:A2
PRED:DIR
Nearest neighbors of “across the border”
1 | DIR | across the Hudson
2 | DIR | outside their traditional tony circle
3 | DIR | across the floor
4 | DIR | through this congress
5 | A2 | off their foundations
6 | DIR | off its foundation
7 | DIR | off the center field wall
8 | A3 | out of bed
9 | A2 | through cottage rooftops
10 | DIR | through San Francisco
Table 7:  Example of the CoNLL-2005 development set, in which our model
misclassified the label for the span “across the border”. We collect 10
nearest neighbors of this span from the training set.

Our span-based model computes and uses span representations (Eq. 7) for label
prediction. To investigate a relation between the span representations and
predicted labels, we qualitatively analyze nearest neighbors of each span
representation with its predicted label. Specifically, for each predicted span
in the development set, we collect 10 nearest neighbor spans with their gold
labels from the training set.

Table 7 shows 10 nearest neighbors of a span “across the border” for the
predicate “move”. The label of this span was misclassified, i.e., the
predicted label is DIR but the gold is A2. Looking at its nearest neighbor
spans, they have different gold labels, such as DIR, A2 and A3. Like this
case, we have observed that spans with a misclassified label often have their
nearest neighbors with inconsistent labels.

### On Label Embeddings

Figure 3:  Label embedding distribution of our span-based model.

We analyze the label embeddings in the labeling function (Eq. 8). Figure 3
shows the distribution of the learned label embeddings. The adjunct labels are
close to each other, which are likely to be less discriminative. Also, the
core label A2 is close to the adjunct label DIR, which are often confused by
the model. To enhance the discriminative power, it is promising to apply
techniques that keep label representations far away from each other Wen et al.
(2016); Luo et al. (2017).

##  6 Related Work

###  6.1 Semantic Role Labeling Tasks

Automatic SRL has been widely studied Gildea and Jurafsky (2002). There have
been two main styles of SRL.

  * •

FrameNet-style SRL Baker et al. (1998)

  * •

PropBank-style SRL Palmer et al. (2005)

In this paper, we have tackled PropBank-style SRL.111111Detailed descriptions
on FrameNet-style and PropBank-style SRL can be found in Baker et al. (1998);
Das et al. (2014); Kingsbury and Palmer (2002); Palmer et al. (2005).

In PropBank-style SRL, there have been two main task settings.

  * •

Span-based SRL: CoNLL-2004 and 2005 shared tasks Carreras and Marquez (2004);
Carreras and Màrquez (2005)

  * •

Dependency-based SRL: CoNLL-2008 and 2009 shared tasks Surdeanu et al. (2008);
Hajič et al. (2009)

Figure 4 illustrates an example of span-based and dependency-based SRL. In
dependency-based SRL (at the upper part of Figure 4), the correct A2 argument
for the predicate “hit” is the word “with”. On one hand, in span-based SRL (at
the lower part of Figure 4), the correct A2 argument is the span “with the
bat”.

For span-based SRL, the CoNLL-2004 and 2005 shared tasks Carreras and Marquez
(2004); Carreras and Màrquez (2005) provided the task settings and datasets.
In the task settings, various SRL models, from traditional pipeline models to
recent neural ones, have been proposed and competed with each other Pradhan et
al. (2005); He et al. (2017); Tan et al. (2018). For dependency-based SRL, the
CoNLL-2008 and 2009 shared tasks Surdeanu et al. (2008); Hajič et al. (2009)
provided the task settings and datasets. As in span-based SRL, recent neural
models achieved high-performance in dependency-based SRL Marcheggiani et al.
(2017); Marcheggiani and Titov (2017); He et al. (2018b); Cai et al. (2018).
This paper focuses on span-based SRL.

###  6.2 BIO-based SRL Models

[text only label, label style=above, edge style=green!60!black,very thick]

[column sep=.2cm] He & hit & the & ball & with & the & bat

[edge unit distance=0.38cm]21A0 [edge unit distance=0.2cm]24A1 [edge unit
distance=0.3cm, edge start x offset=-0.1cm]25A2

[minimum height=0.75cm]111a0 [minimum height=0.75cm]134a1 [minimum
height=0.75cm]157a2 [minimum height=0.75cm, group style=fill=orange!30]122prd

[edge below, edge style=blue!60!black,ultra thick, label style=below,
yshift=-0.05cm]prda0A00.35cm [edge below, edge style=blue!60!black,ultra
thick, label style=below, yshift=-0.05cm]prda1A10.35cm [edge below, edge
style=blue!60!black,ultra thick, label style=below, yshift=-0.05cm, edge start
x offset=-0.1cm]prda2A20.8cm

Figure 4:  Example of dependency-based SRL (the upper part) and span-based SRL
(the lower part).

Span-based SRL can be solved as BIO sequential tagging Hacioglu et al. (2004);
Pradhan et al. (2005); Màrquez et al. (2005).

Neural models State-of-the-art SRL models use neural networks based on the BIO
tagging approach. The pioneering neural SRL model was proposed by Collobert et
al. (2011). They use convolutional neural networks (CNNs) and CRFs. Instead of
CNNs, Zhou and Xu (2015) and He et al. (2017) used stacked BiLSTMs and
achieved strong performance without syntactic inputs. Tan et al. (2018)
replaced stacked BiLSTMs with self-attention architectures. Strubell et al.
(2018) improved the self-attention SRL model by incorporating syntactic
information.

Word representations Typical word representations, such as SENNA Collobert et
al. (2011) and GloVe Pennington et al. (2014), have been used and contributed
to the performance improvement Collobert et al. (2011); Zhou and Xu (2015); He
et al. (2017). Recently, Peters et al. (2018) integrated contextualized word
representation, ELMo, into the model of He et al. (2017) and improved the
performance by 3.2 F1 score. Strubell and McCallum (2018) also integrated ELMo
into the model of Strubell et al. (2018) and reported the performance
improvement.

###  6.3 Span-based SRL Models

Another line of approaches to SRL is labeled span modeling Xue and Palmer
(2004); Koomen et al. (2005); Toutanova et al. (2005).

Typical models Typically, in this approach, models firstly identify candidate
argument spans (argument identification) and then classify each span into one
of the semantic role labels (argument classification). For inference, several
effective methods have been proposed, such as structural constraint inference
by using integer linear programming Punyakanok et al. (2008) or dynamic
programming Täckström et al. (2015); FitzGerald et al. (2015).

Recent span-based model A very recent work, He et al. (2018a), proposed a
span-based SRL model similar to our model. They also used BiLSTMs to induce
span representations in an end-to-end fashion. A main difference is that while
they model $\text{P}(r|i,j)$, we model $\text{P}(i,j|r)$. In other words,
while their model seeks to select an appropriate label for each span (label
selection), our model seeks to select appropriate spans for each label (span
selection). This point distinguishes between their model and ours.

FrameNet span-based model For FrameNet-style SRL, Swayamdipta et al. (2017)
used a segmental RNN Kong et al. (2016), combining bidirectional RNNs with
semi-Markov CRFs Sarawagi and Cohen (2004). Their model computes span
representations using BiLSTMs and learns a conditional distribution over all
possible labeled spans of an input sequence. Although we cannot compare our
results with theirs, we can regard that our model is simpler and effective for
PropBank-style SRL.

###  6.4 Span-based Models in Other NLP Tasks

In syntactic parsing, Wang and Chang (2016) proposed an LSTM-based sentence
segment embedding method named LSTM-Minus. Stern et al. (2017); Kitaev and
Klein (2018) incorporated the LSTM Minus into their parsing model and achieved
the best results in constituency parsing. In coreference resolution, Lee et
al. (2017, 2018) presented an end-to-end coreference resolution model, which
considers all spans in a document as potential mentions and learn
distributions over possible antecedents for each. Our model can be regarded as
an extension of their model.

##  7 Conclusion and Future Work

We have presented a simple and accurate span-based model. We treat SRL as span
selection and our model seeks to select appropriate spans for each label.
Experimental results have demonstrated that despite the simplicity, the model
outperforms a strong BiLSTM-CRF model. Also, our span-based ensemble model
using ELMo achieves the state-of-the-art results on the CoNLL-2005 and 2012
datasets. Through empirical analysis, we have obtained some interesting
findings. One of them is that the span-based model is better at label
prediction compared with the CRF-based model. Another one is that ELMo
improves the model performance for span boundary identification.

An interesting direction for future work concerns evaluating span
representations from our span-based model. Since the investigation on the
characteristics of the representations can lead to interesting findings, it is
worthwhile evaluating them intrinsically and extrinsically. Another promising
direction is to explore methods of incorporating frame knowledge into SRL
models. We have observed that a lot of label confusions arise due to the lack
of such knowledge. The use of frame knowledge to reduce these confusions is a
straightforward approach.

## Acknowledgments

This work was partially supported by JST CREST Grant Number JPMJCR1513 and
JSPS KAKENHI Grant Number 18K18109. We are grateful to the members of the
NAIST Computational Linguistics Laboratory, the members of Tohoku University
Inui-Suzuki Laboratory, Kentaro Inui, Jun Suzuki, Yuichiro Matsubayashi, and
the anonymous reviewers for their insightful comments.

## References

  * Baker et al. (1998) Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998.  The berkeley framenet project.  In _Proceedings of ACL-COLING_ , pages 86–90.
  * Cai et al. (2018) Jiaxun Cai, Shexia He, Zuchao Li, and Hai Zhao. 2018.  A full end-to-end semantic role labeler, syntactic-agnostic over syntactic-aware?  In _Proceedings of COLING_ , pages 2753–2765.
  * Carreras and Marquez (2004) Xavier Carreras and Lluís Marquez. 2004.  Introduction to the conll-2004 shared task: Semantic role labeling.  In _Proceedings of CoNLL_ , pages 89–97.
  * Carreras and Màrquez (2005) Xavier Carreras and Lluís Màrquez. 2005.  Introduction to the conll-2005 shared task: Semantic role labeling.  In _Proceedings of CoNLL_ , pages 152–164.
  * Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011.  Natural language processing (almost) from scratch.  In _Journal of Machine Learning Research_.
  * Das et al. (2014) Dipanjan Das, Desai Chen, André FT Martins, Nathan Schneider, and Noah A Smith. 2014.  Frame-semantic parsing.  _Computational linguistics_ , 40(1):9–56.
  * FitzGerald et al. (2015) Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015\.  Semantic role labeling with neural network factors.  In _Proceedings of EMNLP_ , pages 960–970.
  * Gildea and Jurafsky (2002) Daniel Gildea and Daniel Jurafsky. 2002.  Automatic labeling of semantic roles.  _Computational linguistics_ , 28(3):245–288.
  * Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of AISTATS_ , pages 249–256.
  * Graves et al. (2013) Alan Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013.  Hybrid speech recognition with deep bidirectional LSTM.  In _Proceedings of Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop_.
  * Graves et al. (2005) Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. 2005.  Bidirectional LSTM networks for improved phoneme classification and recognition.  In _Proceedings of International Conference on Artificial Neural Networks_ , pages 799–804.
  * Hacioglu et al. (2004) Kadri Hacioglu, Sameer Pradhan, Wayne Ward, James H Martin, and Daniel Jurafsky. 2004.  Semantic role labeling by tagging syntactic chunks.  In _Proceedings of CoNLL_.
  * Hajič et al. (2009) Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martí, Lluís Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, Pavel Straňák, Mihai Surdeanu, Nianwen Xue, and Yi Zhang. 2009.  The conll-2009 shared task: Syntactic and semantic dependencies in multiple languages.  In _Proceedings of CoNLL_ , pages 1–18.
  * He et al. (2018a) Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018a.  Jointly predicting predicates and arguments in neural semantic role labeling.  In _Proceedings of ACL_ , pages 364–369.
  * He et al. (2017) Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017.  Deep semantic role labeling: What works and what’s next.  In _Proceedings of ACL_ , pages 473–483.
  * He et al. (2018b) Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018b.  Syntax for semantic role labeling, to be, or not to be.  In _Proceedings of ACL_ , pages 2061–2071.
  * Kingma and Ba (2014) D.P. Kingma and J. Ba. 2014.  Adam: A method for stochastic optimization.  arXiv preprint arXiv: 1412.6980.
  * Kingsbury and Palmer (2002) Paul Kingsbury and Martha Palmer. 2002.  From treebank to propbank.  In _Proceedings of LREC_.
  * Kitaev and Klein (2018) Nikita Kitaev and Dan Klein. 2018.  Constituency parsing with a self-attentive encoder.  In _Proceedings of ACL_ , pages 2676–2686.
  * Kong et al. (2016) Lingpeng Kong, Chris Dyer, and Noah A Smith. 2016.  Segmental recurrent neural networks.  In _Proceedings of ICLR_.
  * Koomen et al. (2005) Peter Koomen, Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2005.  Generalized inference with multiple semantic role labeling systems.  In _Proceedings of CoNLL_ , pages 181–184.
  * Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017.  End-to-end neural coreference resolution.  In _Proceedings of EMNLP_ , pages 188–197.
  * Lee et al. (2018) Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018.  Higher-order coreference resolution with coarse-to-fine inference.  In _Proceedings of NAACL_ , pages 687–692.
  * Luo et al. (2017) Lingkun Luo, Xiaofang Wang, Shiqiang Hu, Chao Wang, Yuxing Tang, and Liming Chen. 2017.  Close yet distinctive domain adaptation.  arXiv preprint arXiv:1704.04235.
  * Marcheggiani et al. (2017) Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017.  A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling.  In _Proceedings of CoNLL_ , pages 411–420.
  * Marcheggiani and Titov (2017) Diego Marcheggiani and Ivan Titov. 2017.  Encoding sentences with graph convolutional networks for semantic role labeling.  In _Proceedings of EMNLP_ , pages 1506–1515.
  * Màrquez et al. (2005) Lluís Màrquez, Pere Comas, Jesús Giménez, and Neus Catala. 2005\.  Semantic role labeling as sequential tagging.  In _Proceedings of CoNLL_ , pages 193–196.
  * Palmer et al. (2005) Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005.  The proposition bank: An annotated corpus of semantic roles.  _Computational linguistics_ , 31(1):71–106.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of EMNLP_ , pages 1532–1543.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of NAACL-HLT_ , pages 2227–2237.
  * Pradhan et al. (2005) Sameer Pradhan, Kadri Hacioglu, Wayne Ward, James H Martin, and Daniel Jurafsky. 2005.  Semantic role chunking combining complementary syntactic views.  In _Proceedings of CoNLL_ , pages 217–220.
  * Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012.  Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes.  In _Proceedings of EMNLP-CoNLL_ , pages 1–40.
  * Punyakanok et al. (2008) Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008.  The importance of syntactic parsing and inference in semantic role labeling.  _Computational Linguistics_ , 34(2):257–287.
  * Sarawagi and Cohen (2004) Sunita Sarawagi and William W Cohen. 2004.  Semi-markov conditional random fields for information extraction.  In _Proceedings of NIPS_ , pages 1185–1192.
  * Saxe et al. (2013) Andrew M Saxe, James L McClelland, and Surya Ganguli. 2013.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  arXiv preprint arXiv:1312.6120.
  * Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997.  Bidirectional recurrent neural networks.  _IEEE Transactions on Signal Processing_ , pages 2673–2681.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  arXiv preprint arXiv:1701.06538.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: A simple way to prevent neural networks from overfitting.  _The Journal of Machine Learning Research_ , 15(1):1929–1958.
  * Stern et al. (2017) Mitchell Stern, Jacob Andreas, and Dan Klein. 2017.  A minimal span-based neural constituency parser.  In _Proceedings of ACL_ , pages 818–827.
  * Strubell and McCallum (2018) Emma Strubell and Andrew McCallum. 2018.  Syntax helps elmo understand semantics: Is syntax still relevant in a deep neural architecture for srl?  In _Proceedings of the Workshop on the Relevance of Linguistic Structure in Neural Architectures for NLP_ , pages 19–27.
  * Strubell et al. (2018) Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018\.  Linguistically-informed self-attention for semantic role labeling.  arXiv preprint arXiv:1804.08199.
  * Surdeanu et al. (2008) Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluís Màrquez, and Joakim Nivre. 2008.  The conll 2008 shared task on joint parsing of syntactic and semantic dependencies.  In _Proceedings of CoNLL_ , pages 159–177.
  * Swayamdipta et al. (2017) Swabha Swayamdipta, Sam Thomson, Chris Dyer, and Noah A Smith. 2017.  Frame-semantic parsing with softmax-margin segmental rnns and a syntactic scaffold.  arXiv preprint arXiv:1706.09528.
  * Täckström et al. (2015) Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015.  Efficient inference and structured learning for semantic role labeling.  _Transactions of ACL_ , 3:29–41.
  * Tan et al. (2018) Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018.  Deep semantic role labeling with self-attention.  In _Proceedings of AAAI_.
  * Teranishi et al. (2017) Hiroki Teranishi, Hiroyuki Shindo, and Yuji Matsumoto. 2017.  Coordination boundary identification with similarity and replaceability.  In _Proceedings of IJCNLP_ , pages 264–272.
  * Toutanova et al. (2005) Kristina Toutanova, Aria Haghighi, and Christopher D Manning. 2005.  Joint learning improves semantic role labeling.  In _Proceedings of ACL_ , pages 589–596.
  * Wang and Chang (2016) Wenhui Wang and Baobao Chang. 2016.  Graph-based dependency parsing with bidirectional lstm.  In _Proceedings of ACL_ , pages 2306–2315.
  * Wen et al. (2016) Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. 2016.  A discriminative feature learning approach for deep face recognition.  In _Proceedings of ECCV_ , pages 499–515.
  * Xue and Palmer (2004) Nianwen Xue and Martha Palmer. 2004.  Calibrating features for semantic role labeling.  In _Proceedings of EMNLP_.
  * Zhou and Xu (2015) Jie Zhou and Wei Xu. 2015.  End-to-end learning of semantic role labeling using recurrent neural networks.  In _Proceedings of ACL-IJCNLP_ , pages 1127–1137.

##  Appendix A Span-Consistent Greedy Search

Algorithm 1 Span-Consistent Greedy Search

1: Input: Score Matrix ${\bf
M}\in\mathbb{R}^{|\mathcal{R}|\times|\mathcal{S}|}$,

2: Predicate Position Index $p$

3: Core Label Set $\mathcal{R}^{(\text{core})}$

4: spans $\leftarrow\phi$

5: used_cores $\leftarrow\phi$

6: $\mathcal{U}\leftarrow\\{(i,j,r,\text{score})\in flatten(\textbf{M})\\}$

7: $\mathcal{U}\leftarrow filter(\mathcal{U},p)$

8: for $(i,j,r,\text{score})\in sort(\mathcal{U})$ do

9: if $r\notin\text{used\\_cores}$ and

10: $\text{is\\_overlap}((i,j),\text{spans})$ is False then

11: spans $\leftarrow$ spans $\cup\>\\{\langle i,j,r\rangle\\}$

12: if $r\in\mathcal{R}^{(\text{core})}$ then

13: used_cores $\leftarrow$ used_cores $\cup\>\\{r\\}$

14: return spans

Algorithm 1 describes the pseudo code of the greedy search algorithm
introduced in Section 2.4. This algorithm receives the three inputs (line
1-3). ${\bf M}$ is the score matrix illustrated at the top part of Figure 1 in
Section 3. Each cell of the matrix represents the score of each span. $p$ is a
target predicate position index. $\mathcal{R}^{\text{(core)}}$ is the set of
core labels. At line 4, the variable “spans” is initialized. This variable
stores the selected spans to be returned as the output. At line 5, the
variable “used$\\_$cores” is initialized. This variable keeps track of the
already selected core labels.

At line 6, the score matrix ${\bf M}$ is converted to tuples,
$(i,j,r,\text{score})$, by the function $flatten(\cdot)$. These tuples are
stored in the variable $\mathcal{U}$. At line 7, from $\mathcal{U}$, we remove
the tuples that fall into any one of the followings, (i) the tuples whose
boundary $(i,j)$ overlaps with the predicate position $p$ or (ii) the tuples
whose score is lower than that of the predicate span tuples. In terms of (i),
since spans whose boundary $(i,j)$ overlaps with the predicate position,
$i\leq p\leq j$, can never be a correct argument, we remove such tuples. In
terms of (ii), we remove the tuples $(*,*,r,\text{score})$ whose score is
lower than that of the predicate span tuple $(p,p,r,\text{score})$. In Section
2, we define the predicate span $(p,p)$ as the Null span, implying that we can
regard the spans whose score is lower than that of the Null span as an
inappropriate argument. Thus, we remove such tuples from the set of the
candidates $\mathcal{U}$.

The main processing starts from line 8. Based on the scores, the function
$sort(\cdot)$ sorts the tuples $(i,j,r,\text{score})$ in a descending order.
At line 9-10, there are constraints for output spans. At line 9,
“$r\notin\text{used\\_cores}$” represents the constraint that at most one span
can be selected for each core label. At line 10, the function
$\text{is\\_overlap}(\cdot)$ takes as input a span $(i,j)$ and the set of the
selected spans, and returns the boolean value (“True” or “False”) that
represents whether the span overlaps with any one of the selected spans or
not.

At line 11, the span is added to the set of the selected spans. At line 12-13,
if the label $r$ is included in the core labels $\mathcal{R}^{\text{(core)}}$,
the label is added to “used_cores”. At line 14, as the final output, the set
of the selected spans “spans” is returned.

##  Appendix B BiLSTMs

As the base feature function $f_{base}$ (Eq. 4 in Section 2.3), we use
BiLSTMs,

$\displaystyle f_{base}(w_{1:T},p)=\textsc{BiLSTM}(w_{1:T},p)\>\>.$

In particular, we use the stacked BiLSTMs in an interleaving fashion Zhou and
Xu (2015); He et al. (2017). The stacked BiLSTMs process an input sequence in
a left-to-right manner for odd-numbered layers and in a right-to-left manner
for even-numbered layers.

The stacked BiLSTMs consist of $L$ layers. The hidden state in each layer
$\ell\in\\{1,\cdots,L\\}$ is calculated as follows,

${\bf h}^{(\ell)}_{t}=\begin{cases}\text{LSTM}^{(\ell)}({\bf
x}^{(\ell)}_{t},\>{\bf h}^{(\ell)}_{t-1})&(\ell\>\mbox{=}\>\mbox{odd})\\\
\text{LSTM}^{(\ell)}({\bf x}^{(\ell)}_{t},\>{\bf
h}^{(\ell)}_{t+1})&(\ell\>\mbox{=}\>\mbox{even})\>\>.\end{cases}$

Both of the odd- and even-numbered layers receive ${\bf x}^{(\ell)}_{t}$ as
the first input of the LSTM. For the second input, odd-numbered layers receive
${\bf h}^{(\ell)}_{t-1}$, whereas even-numbered layers receive ${\bf
h}^{(\ell)}_{t+1}$.

Between the LSTM layers, we use the following connection Zhou and Xu (2015),

${\bf x}^{(\ell+1)}_{t}=\text{ReLU}({\bf W}^{(\ell)}\cdot[{\bf
x}^{(\ell)}_{t};{\bf h}^{(\ell)}_{t}])\>\>.$

Here, we firstly concatenate ${\bf x}^{(\ell)}_{t}$ and ${\bf
h}^{(\ell)}_{t}$, and then calculate the inner product of the concatenated
vector and the parameter matrix ${\bf W}^{(\ell)}$ with the rectified linear
units (ReLU). As a result, we obtain the input representation ${\bf
x}^{(\ell+1)}_{t}$ for the next ($\ell+1$-th) LSTM layer.

In the first layer, $\text{LSTM}^{(1)}$ receives an input feature vector ${\bf
x}^{(1)}_{t}$. Following He et al. (2017), we create this vector by
concatenating a word embedding and predicate mark embedding,

$\displaystyle{\bf x}^{(1)}_{t}=[{\bf x}^{word}_{t};{\bf x}^{mark}_{t}]\>\>,$

where ${\bf x}^{word}\in\mathbb{R}^{d^{word}}$ and ${\bf
x}^{mark}\in\mathbb{R}^{d^{mark}}$. The mark embedding is created from the
binary mark feature. The value is 1 if the word is the target predicate and 0
otherwise.

After the $L$-th LSTM layer runs, we obtain ${\bf x}^{(L+1)}_{1:T}={\bf
x}^{(L+1)}_{1},\cdots,{\bf x}^{(L+1)}_{T}$. We use them as the input of the
span feature function $f_{span}$ (Eq. 5 in Section 2.3), i.e., ${\bf
h}_{1:T}={\bf x}^{(L+1)}_{1:T}$. Each vector ${\bf h}_{t}\in{\bf h}_{1:T}$ has
$d^{hidden}$ dimensions.

##  Appendix C Hyperparameters

Name | Value
---|---
Word Embedding $d^{word}$ | 50-dimensional SENNA
1024-dimensional ELMo
Mark Embedding $d^{mark}$ | 50-dimensional vector
LSTM Layers $L$ | 4
LSTM Hidden Units $d^{hidden}$ | 300 dimensions
Mini-batch Size | 32
Optimization | Adam
Learning Rate | 0.001
L2 Regularization $\lambda$ | 0.0001
Dropout Ratio for BiLSTMs | 0.1
Dropout Ratio for ELMo | 0.5
Table 8:  Hyperparameters for our span-based model.

###  C.1 Span-based Model

Table 8 lists the hyperparameters used for our span-based model.

Word representation setup As word embeddings ${\bf x}^{word}$, we use two
types of embeddings, (i) SENNA Collobert et al. (2011), 50-dimensional word
vectors ($d^{word}=50$), and (ii) ELMo Peters et al. (2018), 1024-dimensional
vectors ($d^{word}=1024$). During training, we fix these word embeddings (not
update them). As predicate mark embeddings ${\bf x}^{mark}$, we use randomly
initialized 50-dimensional vectors ($d^{mark}=50$). During training, we update
them.

Network setup As the base feature function $f_{base}$, we use $4$ stacked
BiLSTMs (2 forward and 2 backward LSTMs) with 300-dimensional hidden units
($d^{hidden}=300$). Following He et al. (2017), we initialize all the
parameter matrices in BiLSTMs with random orthonormal matrices Saxe et al.
(2013). Other parameters are initialized following Glorot and Bengio (2010),
and bias parameters are initialized with zero vectors.

Regularization We set the coefficient $\lambda$ for the L2 weight decay (Eq.
11 in Section 4.3) to $0.0001$. We apply dropout Srivastava et al. (2014) to
the input vectors of each LSTM with dropout ratio of 0.1 and the ELMo
embeddings with dropout ratio of 0.5.

Training To optimize the parameters, we use Adam Kingma and Ba (2014) with
$\beta_{1}=0.9$ and $\beta_{2}=0.999$. The learning rate is initialized to
0.001. After training 50 epochs, we halve the learning rate every 25 epochs.
Parameter updates are performed in mini-batches of 32. The number of training
epochs is set to 100. We save the parameters that achieve the best F1 score on
the development set and evaluate them on the test set. Training our model on
the CoNLL-2005 training set takes about one day and on the CoNLL-2012 training
set takes about two days on a single GPU, respectively.

###  C.2 Ensemble Model

Our ensemble model uses span representations ${\bf h}^{(m)}_{s}$ from base
models $m\in\\{1,\cdots,M\\}$ (Section 3.2). We use 5 base models ($M=5$)
learned over different runs. Note that, during training, we fix the parameters
of the five base models and update only the parameters of the ensemble model.

Network setup The parameter matrix ${\bf W}^{\text{moe}}_{s}$ (Eq. 9 in
Section 3.2) is initialized with the identity matrix. The scalar parameters
$\\{\alpha_{m}\\}^{M}_{m=1}$ (Eq. 9) are initialized with $0$. Each row vector
${\bf W}^{\text{moe}}[r]$ of the parameter matrix ${\bf W}^{\text{moe}}$ (Eq.
10) is initialized with the averaged vector over the row vectors ${\bf
W}^{(m)}[r]$ of each model $m$, i.e., $\frac{1}{M}\sum_{m=1}^{M}{\bf
W}^{(m)}[r]$.

Training To optimize the parameters, we use Adam with $\beta_{1}=0.9$ and
$\beta_{2}=0.999$. The learning rate is set to 0.0001. Parameter updates are
performed in mini-batches of 8. The number of training epochs is set to 20. We
save the parameters that achieve the best F1 score on the development set and
evaluate them on the test set. Training one ensemble model on the CoNLL-2005
and 2012 training sets takes about one day on a single GPU.


# AN EFFICIENT ALGORITHM FOR MULTIUSER SUM-RATE MAXIMIZATION OF LARGE-SCALE
Active RIS-AIDED MIMO SYSTEM

###### Abstract

Active reconfigurable intelligent surface (RIS) is a new RIS architecture that
can reflect and amplify communication signals. It can provide enhanced
performance gain compared to the conventional passive RIS systems that can
only reflect the signals. On the other hand, the design problem of active RIS-
aided systems is more challenging than the passive RIS-aided systems and its
efficient algorithms are less studied. In this paper, we consider the sum rate
maximization problem in the multiuser massive multiple-input single-output
(MISO) downlink with the aid of a large-scale active RIS. Existing approaches
usually resort to general optimization solvers and can be computationally
prohibitive in the considered settings. We propose an efficient block
successive upper bound minimization (BSUM) method, of which each step has a
(semi) closed-form update. Thus, the proposed algorithm has an attractive low
per-iteration complexity. By simulation, our proposed algorithm consumes much
less computation than the existing approaches. In particular, when the MIMO
and/or RIS sizes are large, our proposed algorithm can be orders-of-magnitude
faster than existing approaches.

Index Terms—  Active reconfigurable intelligent surface, efficient algorithm,
computational complexity

##  1 Introduction

Reconfigurable intelligent surface (RIS) refers to a planar of adaptive
electromagnetic elements that can be designed to improve the wireless channel
environment [1, 2, 3]. When it was first proposed, the RIS elements can
passively reflect the signal such that system performance metrics such as
multiuser sum rate [4], energy efficiency [5], and symbol-error probability
[6, 7] are enhanced. However, it is also found that the reflected signal power
of passive RIS-aided systems is limited when the RIS size is not large enough
or when the transmission distance is long [8]. To alleviate this limitation,
recently the active RIS architecture was proposed [9, 8, 10]. Compared to the
passive RIS systems, the active RIS systems are equipped with active
amplifying components in order to reflect and amplify the signal at the same
time [9, 8]. It has been reported that under the same total power budget of
the base station (BS) and RIS, active RIS-aided systems can provide superior
system gain than passive RIS-aided systems [8]. We refer the readers to [8,
10] for a more detailed discussion of the active and passive RIS systems.

On the other hand, the design problem of active RIS systems poses a different
form from that of passive RIS systems, which calls for new efficient design
algorithms. The most widely adopted approach in the literature is to apply
general purpose optimization solvers, such as CVX [11], to solve the
subproblems in a RIS-aided system; see, e.g., sum rate maximization [12], mean
square error minimization [13], and energy efficiency maximization [14]. In
[12], Zhu et al. broke down the sum rate maximization problem by a dedicated
algorithm that composes block coordinate descent (BCD), alternating direction
method of multipliers (ADMM), CVX, and majorization-minimization (MM), which
is a three-tier iterative algorithm. In future 6G systems, the BS will be
associated with massive MIMO systems, which may contain hundreds of antennas;
the number of RIS elements can also be large in order to support high
performance. These physical aspects will contribute to a large-scale joint
design problem for beamforming in active RIS-aided systems. The aforementioned
methods can inevitably incur unaffordable computational complexity as the
computational complexity of general purpose solvers scales quickly with the
problem dimensions.

In this paper, we propose an efficient algorithm for active RIS-aided
multiuser multiple-input single-output (MISO) downlink systems. Our focus is
multiuser sum rate maximization (SRM) under the joint constraints arising from
the BS and active RIS. We apply the block successive upper bound minimization
(BSUM) method [15] to the resulting problem. The BSUM algorithm breaks down
the design problem by a recursive optimization process, and each subproblem
involves only one variable. One key feature of our algorithm is that each step
has a closed-form solution, either by an exact minimization or the proximal
distance algorithm [16]. As a result, our proposed algorithm enjoys a low per-
iteration complexity. Simulation results demonstrate that our algorithm is
much faster than existing methods when attaining the same sum rate
performance; in particular, our algorithm can be over 100 times faster when
the MIMO and RIS sizes are large.

##  2 System Model

Consider a single-cell active RIS-aided multiuser multiple-input single-output
(MISO) downlink scenario, which is shown in Fig. 1. A base station (BS), which
is associated with $M$ antennas, employs linear precoding to serve $K$ single-
antenna users with the aid of an active RIS equipping with $N$ reflecting
elements. Let $\bm{\overline{h}}_{k}\in\mathbb{C}^{M\times 1}$,
$\bm{f}_{k}\in\mathbb{C}^{N\times 1}$, and ${\bm{G}}\in\mathbb{C}^{N\times M}$
denote the downlink channels from BS to user $k$, RIS to user $k$, and BS to
RIS, respectively. Also, denote $\bm{w}_{k}\in\mathbb{C}^{M\times 1}$ as the
linear precoding vector for the user $k$ at the BS and
$\bm{\phi}\in\mathbb{C}^{N\times 1}$ as the reflecting coefficients of active
RIS elements. Then, the achievable rate of user $k$ can be characterized as
[10, 8]

$\begin{split}R_{k}({\bm{w}},\bm{\phi})={\rm{log}}_{2}\,(1+{\sf
SINR}_{k}),\enspace k=1,\ldots,K,\end{split}$ (1)

where ${\bm{w}}=[\bm{w}_{1}^{T},\bm{w}_{2}^{T},\dots,\bm{w}_{K}^{T}]^{T}$,

${\sf SINR}_{k}=\frac{|\bm{h}_{k}^{H}\bm{w}_{k}|^{2}}{\sum_{i=1,i\neq
k}^{K}|\bm{h}_{k}^{H}\bm{w}_{i}|^{2}+\|\bm{f}_{k}^{H}\mathbb{\Phi}\|_{2}^{2}\sigma_{v}^{2}+\sigma_{k}^{2}}$

is the signal-to-interference-plus-noise ratio (SINR) of user $k$;
$\bm{\Phi}=\mbox{Diag}(\bm{\phi})$ is a diagonal matrix with the diagonal
entries being $\phi$; $\sigma_{v}^{2}$ and $\sigma_{k}^{2}$ denote the noise
power at the active RIS and user sides, respectively;
$\bm{h}_{k}^{H}=\bm{\overline{h}}_{k}^{H}+\bm{f}_{k}^{H}\mathbb{\Phi}{\bm{G}}\in\mathbb{C}^{1\times
M}$.

We jointly optimize the precoder ${\bm{w}}$ and the reflecting coefficients
$\bm{\phi}$ to maximize the multiuser sum-rate under both BS power constraints
and active RIS power constraints. The problem of interest can be formulated as

$\displaystyle\min_{{\bm{w}},\bm{\phi}}$
$\displaystyle~{}f({\bm{w}},\bm{\phi})=-\sum_{k=1}^{K}R_{k}({\bm{w}},\bm{\phi})$
(2a) s.t. $\displaystyle~{}\mathcal{C}_{\sf
BS}:~{}\sum_{k=1}^{K}\|\bm{w}_{k}\|_{2}^{2}\leq P_{B},$ (2b)
$\displaystyle~{}\mathcal{C}_{\sf
RIS}:~{}|\phi_{n}|\leq\eta_{n},n=1,2,\dots,N,$ (2c)
$\displaystyle~{}\mathcal{C}_{\sf
BR}:~{}\sum_{k=1}^{K}\|\mathbb{\Phi}{\bm{G}}\bm{w}_{k}\|_{2}^{2}+\|\bm{\phi}\|_{2}^{2}\sigma_{v}^{2}\leq
P_{A}.$ (2d)

In problem (2), the constraint $\mathcal{C}_{\sf BS}$ represents the total
transmission power budget at the BS is no larger than $P_{B}$; one can instead
consider the per-antenna power constraint

$\|\bar{{\bm{w}}}_{m}\|_{2}^{2}\leq P_{B}/M,\quad m=1,\ldots,M,$ (3)

where $\bar{{\bm{w}}}_{m}$ is the $m$th row of
$(\bm{w}_{1},\bm{w}_{2},\dots,\bm{w}_{K})$; the constraint $\mathcal{C}_{\sf
RIS}$ restricts the reflecting gain at each RIS element is no larger than the
corresponding threshold $\eta_{n}$; the constraint $\mathcal{C}_{\sf BR}$ is
the reflected power budget at the active RIS side.

The sum-rate maximization (SRM) problem in (2) is a basic problem in the
active RIS system. It resembles many system designs including secrecy rate
maximization [17], energy efficiency maximization [18], and sub-connected
array architecture [14, 12]. However, solving problem (2) is difficult.
Problem (2) is a non-convex program as the objective function and the
constraint (2d) are non-convex; plus, the variables ${\bm{w}}$ and $\bm{\phi}$
are coupled in both the objective function and constraint (2d). In the
existing literature, the most popular way is to block coordinate descent (BCD)
and general purpose optimizers, such as CVX [11]; see, e.g., [8, 17, 14, 12,
18]. This can incur prohibitively high computational complexity when $M$ and
$N$ are large, which happens in massive MIMO and large-scale RIS systems. Our
goal in this paper is to custom-build an efficient algorithm to tackle problem
(2). As will be shown, our algorithm has a (semi) closed-form update in each
step, and its efficiency will be demonstrated by simulation.

Fig. 1: Illustration for single-cell active RIS-aided multiuser MIMO downlink
scenario.

##  3 Proposed Algorithm

Our method has two key ingredients: 1) reformulating the objective function,
and 2) dealing with the coupled constraint.

###  3.1 Reformulation of the Objective Function

We apply a reformulation to the objective function, which is shown in the
following fact.

###### Fact 1

Problem (2) is equivalent to the following problem

$\begin{split}\min_{{\bm{w}},\bm{\phi},\bm{{\bm{u}}},\bm{\rho}}g({\bm{w}},\bm{\phi},{\bm{u}},\bm{\rho})\quad{\rm
s.t.}~{}\mathcal{C}_{\sf BS},~{}\mathcal{C}_{\sf RIS},~{}\mathcal{C}_{\sf
BR},\end{split}$ (4)

where $\bm{\rho}\in\mathbb{R}_{++}^{K}$, ${\bm{u}}\in\mathbb{C}^{K}$, and

$\begin{split}g({\bm{w}},\bm{\phi},{\bm{u}},\bm{\rho})=&~{}\textstyle\sum_{k=1}^{K}\rho_{k}F_{k}({\bm{w}},\bm{\phi},u_{k})-\log(\rho_{k}),\\\
F_{k}({\bm{w}},\bm{\phi},u_{k})=&~{}\textstyle|u_{k}|^{2}\big{(}\sum_{i=1}^{K}|{\bm{h}}_{k}^{H}\bm{w}_{i}|^{2}+\|\bm{f}_{k}^{H}\mathbb{\Phi}\|_{2}^{2}\sigma_{v}^{2}\\\
&+\sigma_{k}^{2}\big{)}-2{\rm{Re}}\big{(}u_{k}^{*}{\bm{h}}_{k}^{H}{\bm{w}}_{k}\big{)}+1.\\\\[-3.0pt]
\end{split}$

Fact 1 can be shown in the same vein as the WMMSE method [19]. We shall omit
the proof details.

###  3.2 Precoding Algorithm

In Fact 1, with the aid of the auxiliary variables $({\bm{u}},\bm{\rho})$, the
reformulated problem (4) can be efficiently tackled by the block successive
upper bound minimization (BSUM) method [15]. To describe, let
$({\bm{w}}^{\ell},\bm{\phi}^{\ell},\bm{{\bm{u}}}^{\ell},\bm{\rho}^{\ell})$
denote the solution at current iteration $\ell$. The BSUM method takes the
following updates

$\begin{split}{\bm{u}}^{\ell+1}=&~{}\arg\min_{{\bm{u}}}g({\bm{w}}^{\ell},\bm{\phi}^{\ell},\bm{{\bm{u}}},\bm{\rho}^{\ell});\\\
\bm{\rho}^{\ell+1}=&~{}\arg\min_{\bm{\rho}}g({\bm{w}}^{\ell},\bm{\phi}^{\ell},\bm{{\bm{u}}}^{\ell+1},\bm{\rho});\\\
{\bm{w}}^{\ell+1}=&~{}\arg\min_{{\bm{w}}}g({\bm{w}},\bm{\phi}^{\ell},\bm{{\bm{u}}}^{\ell+1},\bm{\rho}^{\ell+1});\\\
\bm{\phi}^{\ell+1}=&~{}\arg\min_{\bm{\phi}}g({\bm{w}}^{\ell+1},\bm{\phi},\bm{{\bm{u}}}^{\ell+1},\bm{\rho}^{\ell+1}).\\\\[-3.0pt]
\end{split}$ (5)

The solutions to the ${\bm{u}}$ and $\bm{\rho}$ subproblems can be computed in
closed forms, which are given by

$\begin{split}u_{k}^{\ell+1}=&~{}\frac{{\bm{h}}_{k}^{H}{\bm{w}}_{k}^{\ell}}{\sum_{i=1}^{K}|{\bm{h}}_{k}^{H}{\bm{w}}_{i}^{\ell}|^{2}+\|\bm{f}_{k}^{H}\bm{\Phi}^{\ell}\|_{2}^{2}\sigma_{v}^{2}+\sigma_{k}^{2}},~{}k=1,\ldots,K,\\\
\rho_{k}^{\ell+1}=&~{}(1-(u_{k}^{\ell+1})^{*}{\bm{h}}_{k}^{H}{\bm{w}}_{k}^{\ell})^{-1},~{}k=1,\ldots,K.\end{split}$

The ${\bm{w}}$ and $\bm{\phi}$ subproblems do not have closed-form solutions.
We first consider the $\bm{\phi}$ subproblem, which takes the form

$\begin{split}\min_{\bm{\phi}}&~{}{\bm{\phi}}^{H}{\bm{Q}}\bm{\phi}-\Re\\{2\bm{\phi}^{H}{\bm{z}}\\}\\\\[-3.0pt]
\mbox{s.t.}&~{}\mathcal{C}_{\sf
RIS}:~{}|\phi_{n}|\leq\eta_{n},~{}n=1,2,\dots,N,\\\\[-3.0pt]
&~{}\mathcal{C}_{\sf BR}:\bm{\phi}^{H}\bm{\Lambda}\bm{\phi}\leq
P_{A},\end{split}$ (6)

where
${\bm{W}}^{\ell+1}=\sum_{i=1}^{K}{\bm{w}}_{i}^{\ell+1}({\bm{w}}_{i}^{\ell+1})^{H}$
and

$\begin{split}\bm{\Lambda}=&\textstyle\sum_{k=1}^{K}{\rm{Diag}}({\bm{G}}{\bm{w}}_{k}^{\ell+1}){\rm{Diag}}({\bm{G}}{\bm{w}}_{k}^{\ell+1})^{H}+\sigma_{v}^{2}{\bm{I}}_{N},\\\
{\bm{Q}}=&\textstyle\sum_{k=1}^{K}|u_{k}^{\ell+1}|^{2}\rho_{k}^{\ell+1}{\rm{Diag}}(\bm{f}_{k}^{H}){\bm{G}}{\bm{W}}^{\ell+1}{\bm{G}}^{H}{\rm{Diag}}(\bm{f}_{k})\\\
&~{}\textstyle+\sigma_{v}^{2}\sum_{k=1}^{K}|u_{k}^{\ell+1}|^{2}\rho_{k}^{\ell+1}{\rm{Diag}}(\bm{f}_{k}^{H}){\rm{Diag}}(\bm{f}_{k}),\\\
{\bm{z}}=&\textstyle\\!\sum_{k=1}^{K}{\rm{Diag}}(\bm{f}_{k}^{H}){\bm{G}}\big{(}\rho_{k}^{\ell+1}(u_{k}^{\ell+1})^{*}{\bm{w}}_{k}-\rho_{k}^{\ell+1}|u_{k}^{\ell+1}|^{2}{\bm{W}}^{\ell+1}{\bm{h}}_{k}\big{)}.\\\\[-5.0pt]
\end{split}$

We tackle the problem (6) by the proximal distance algorithm [16], which
recast the problem as

$\begin{split}\min_{\bm{\phi}}&~{}{\bm{\phi}}^{H}{\bm{Q}}\bm{\phi}-\Re\\{2\bm{\phi}^{H}{\bm{z}}\\}+\mu\cdot\mbox{dist}^{2}(\bm{\phi},\mathcal{C}_{\sf
BR}\cap\mathcal{C}_{\sf RIS}),\end{split}$ (7)

where

$\mbox{dist}({\bm{x}},\mathcal{X})=\min_{{\bm{y}}\in\mathcal{X}}\|{\bm{x}}-{\bm{y}}\|_{2}$

is the distance function from a point ${\bm{x}}$ to a set $\mathcal{X}$, and
$\mu>0$ is a given penalty parameter. Problem (7) approaches problem (6) when
$\mu\rightarrow+\infty$. Note that problem (7) takes a bilevel optimization
form as the distance function is embedded within the main problem. We majorize
the distance function by

$\mbox{dist}(\bm{\phi},\mathcal{C}_{\sf BR}\cap\mathcal{C}_{\sf
RIS})\leq\|\bm{\phi}-\Pi_{\mathcal{C}_{\sf BR}\cap\mathcal{C}_{\sf
RIS}}(\bm{\phi}^{\ell})\|_{2},$

where
$\Pi_{\mathcal{X}}({\bm{x}})=\arg\min_{{\bm{y}}\in\mathcal{X}}\|{\bm{x}}-{\bm{y}}\|_{2}$
is the projection of ${\bm{x}}$ onto the set $\mathcal{X}$. Then, problem (7)
can be majorized by

$\begin{split}\min_{\bm{\phi}}&~{}{\bm{\phi}}^{H}{\bm{Q}}\bm{\phi}-\Re\\{2\bm{\phi}^{H}{\bm{z}}\\}+\mu\|\bm{\phi}-\Pi_{\mathcal{C}_{\sf
BR}\cap\mathcal{C}_{\sf RIS}}(\bm{\phi}^{\ell})\|_{2}^{2},\end{split}$ (8)

which has a closed form solution that is given by

$\bm{\phi}^{\ell+1}=({\bm{Q}}+\mu{\bm{I}}_{N})^{-1}({\bm{z}}+\mu\Pi_{\mathcal{C}_{\sf
BR}\cap\mathcal{C}_{\sf RIS}}(\bm{\phi}^{\ell})).$

It remains to derive the projection $\Pi_{\mathcal{C}_{\sf
BR}\cap\mathcal{C}_{\sf RIS}}(\bm{\phi})$. Interestingly, we found that the
projection has a semi-closed form solution, in Fact 2. The proof of Fact 2 is
shown in the Appendix.

###### Fact 2

For a fixed ${\bm{w}}$, the projection $\hat{\bm{\phi}}=\Pi_{\mathcal{C}_{\sf
BR}\cap\mathcal{C}_{\sf RIS}}(\bm{\phi})$ is given by

$\hat{\phi}_{i}=\begin{cases}\min(|\phi_{i}|,\eta_{i})\cdot
e^{\mathfrak{j}\angle\phi_{i}},&\mbox{if
}\sum\limits_{i=1}^{N}\lambda_{i}\min(a_{i},\eta_{i})^{2}\leq P_{A}\\\
\min\big{\\{}\frac{|\phi_{i}|}{1+\gamma\lambda_{i}},\eta_{i}\big{\\}}e^{\mathfrak{j}\angle\phi_{i}},&\mbox{otherwise}.\end{cases}$

for $i=1,\ldots,N$, where $\gamma\geq 0$ is chosen such that
$\hat{\bm{\phi}}^{H}\bm{\Lambda}\hat{\bm{\phi}}=P_{A}$; $\lambda_{i}$ is the
$i$th diagonal element of $\bm{\Lambda}$.

We tackle the ${\bm{w}}$ subproblem by the same rationale. The ${\bm{w}}$
subproblem amounts to solve

$\begin{split}\min_{{\bm{w}}}&~{}\textstyle\sum_{k=1}^{K}{\bm{w}}_{k}^{H}{\bm{A}}{\bm{w}}_{k}-\sum_{k=1}^{K}\Re\\{2\bm{b}_{k}^{H}{\bm{w}}_{k}\\}\\\
\mbox{s.t.}&~{}\textstyle\mathcal{C}_{\sf
BS}:\sum_{k=1}^{K}\|{\bm{w}}_{k}\|_{2}^{2}\leq P_{B},~{}\mathcal{C}_{\sf
BR}:\sum_{k=1}^{K}{\bm{w}}_{k}^{H}\bm{\Psi}{\bm{w}}_{k}\leq P,\\\ \end{split}$
(9)

where
${\bm{A}}=\sum_{k=1}^{K}\rho_{k}^{\ell+1}|u_{k}^{\ell+1}|^{2}{\bm{h}}_{k}{\bm{h}}_{k}^{H}$,
$\bm{b}_{k}=\rho_{k}^{\ell+1}(u_{k}^{\ell+1})^{*}{\bm{h}}_{k}$,
$P=P_{A}-\|\bm{\phi}^{\ell}\|_{2}^{2}\sigma_{v}^{2}$, and
$\bm{\Psi}={\bm{G}}^{H}(\bm{\Phi}^{\ell})^{H}\bm{\Phi}^{\ell}{\bm{G}}$. The
set $\mathcal{C}_{\sf BS}\cap\mathcal{C}_{\sf BR}$ is an intersection of a
ball and an ellipsoid, whose associated projection does not have an explicit
solution. To bypass this issue, we apply distance majorization independently
for the two sets $\mathcal{C}_{\sf BS}$ and $\mathcal{C}_{\sf BR}$. To be
specific, we approximate problem (9) by

$\begin{split}\min_{{\bm{w}}}&~{}\textstyle\sum_{k=1}^{K}{\bm{w}}_{k}^{H}{\bm{A}}{\bm{w}}_{k}-\sum_{k=1}^{K}\Re\\{2\bm{b}_{k}^{H}{\bm{w}}_{k}\\}\\\
&~{}+\mu\cdot\mbox{dist}^{2}({\bm{w}},\mathcal{C}_{\sf
BS})+\mu\cdot\mbox{dist}^{2}({\bm{w}},\mathcal{C}_{\sf BR}).\end{split}$ (10)

We majorize the distance function by

$\mbox{dist}({\bm{w}},\mathcal{C}_{\sf
BS})\leq\|{\bm{w}}-\tilde{{\bm{w}}}_{\sf
BS}\|_{2},~{}\mbox{dist}({\bm{w}},\mathcal{C}_{\sf
BR})\leq\|{\bm{w}}-\tilde{{\bm{w}}}_{\sf BR}\|_{2},$ (11)

where $\tilde{{\bm{w}}}_{\sf BS}=\Pi_{\mathcal{C}_{\sf BS}}({\bm{w}})$,
$\tilde{{\bm{w}}}_{\sf BR}=\Pi_{\mathcal{C}_{\sf BR}}({\bm{w}})$ and

$\begin{split}[\tilde{{\bm{w}}}_{\sf
BS}]_{k}=&\begin{cases}{\bm{w}}_{k},&\mbox{if }\|{\bm{w}}\|_{2}^{2}\leq
P_{B}\\\
\sqrt{P_{B}}\frac{{\bm{w}}_{k}}{\|{\bm{w}}\|_{2}},&\mbox{otherwise},\end{cases}\\\
[\tilde{{\bm{w}}}_{\sf BR}]_{k}=&\begin{cases}{\bm{w}}_{k},&\mbox{if
}\sum_{k=1}^{K}{\bm{w}}_{k}^{H}\bm{\Psi}{\bm{w}}_{k}\leq P\\\
({\bm{I}}_{M}+2\nu\bm{\Psi})^{-1}{\bm{w}}_{k},&\mbox{otherwise},\end{cases}\end{split}$

for $k=1,\ldots,K$, where $\nu>0$ ensures
$\sum_{k=1}^{K}[\tilde{{\bm{w}}}_{\sf
BR}]_{k}^{H}\bm{\Psi}[\tilde{{\bm{w}}}_{\sf BR}]_{k}=P$ and can be found by
the one-dimensional bisection search.

We apply the distance majorization (11) to problem (10). Solving the resulting
problem leads to a closed-form solution

${\bm{w}}_{k}^{\ell+1}=(2\mu{\bm{I}}_{M}+{\bm{A}})^{-1}({\bm{b}}_{k}+\mu[\tilde{{\bm{w}}}_{\sf
BS}]_{k}^{\ell}+\mu[\tilde{{\bm{w}}}_{\sf BR}]_{k}^{\ell}),~{}k=1,\ldots,K.$

The whole algorithm is summarized in Algorithm 1. It is seen that each step of
Algorithm 1 has a closed form, which contributes to the efficient iteration
updates. In addition, for the penalty parameter $\mu$, we initialize it by a
small value and keep gradually increasing its value as the algorithm proceeds.
This strategy is known as homotopy optimization, which can help avoid bad
local minima and achieve good performance; see, e.g., [20, 21, 22], for
detailed discussions.

Algorithm 1 An Efficient BSUM Algorithm for Problem (4)

1: Input: initialization ${\bm{w}}$, $\bm{\phi}$ and $\mu$; $\gamma>1$;
$\ell=1$.

2: repeat

3:
$u_{k}^{\ell+1}=\frac{{\bm{h}}_{k}^{H}{\bm{w}}_{k}^{\ell}}{\sum_{i=1}^{K}|{\bm{h}}_{k}^{H}{\bm{w}}_{i}^{\ell}|^{2}+\|\bm{f}_{k}^{H}\bm{\Phi}^{\ell}\|^{2}\sigma_{v}^{2}+\sigma_{k}^{2}},~{}k=1,\ldots,K$;

4:
$\rho_{k}^{\ell+1}=(1-(u_{k}^{\ell+1})^{*}{\bm{h}}_{k}^{H}{\bm{w}}_{k}^{\ell})^{-1},~{}k=1,\ldots,K$;

5:
${\bm{w}}_{k}^{\ell+1}=(2\mu{\bm{I}}_{M}+{\bm{A}})^{-1}({\bm{b}}_{k}+\mu[\tilde{{\bm{w}}}_{\sf
BS}]_{k}^{\ell}+\mu[\tilde{{\bm{w}}}_{\sf BR}]_{k}^{\ell}),~{}k=1,\ldots,K$;

6:
$\bm{\phi}^{\ell+1}=({\bm{Q}}+\mu{\bm{I}}_{N})^{-1}({\bm{z}}+\mu\Pi_{\mathcal{C}_{\sf
BR}\cap\mathcal{C}_{\sf RIS}}(\bm{\phi}^{\ell}))$;

7: $\mu=\mu\cdot\gamma$;

8: $\ell=\ell+1$;

9: until some stopping criterion is satisfied.

We should discuss the complexity of Algorithm 1. It can be verified that the
per-iteration complexity of Algorithm 1 is
$\mathcal{O}(M^{3}+N^{3}+M^{2}N+M^{2}K+N^{2}M+N^{2}K+MNK)$. By comparison, the
per-iteration complexity of the algorithm in [8, 14] is
$\mathcal{O}(M^{4.5}+N^{4.5}+MNK)$, and the per-iteration complexity of the
algorithm in [12] is $\mathcal{O}(N^{4.5}+M^{4.5}K^{4.5}+M^{2}K+NK^{2})$.

##  4 Simulation Results and Conclusion

In this section, we provide simulation results to show the performance of our
algorithm. The simulation settings are as follows. We set $r=8m$ and
${\rm{L}}=100m$ in Fig. 1. For the small-scale fading, we employ the Rician
fading channel model for all channels involved in active RIS systems, see (37)
in [8]. The large-scale fading of BS-user channel is set as
${\rm{PL}}=41.2+28.7{\rm{log}}(d)$, and all large-scale fading of BS-RIS
channel and RIS-user channel are set as ${\rm{PL}}=37.3+22.0{\rm{log}}(d)$,
where $d$ is the distance between two devices [8]. The noise power is set as
$\sigma_{k}^{2}=\sigma_{v}^{2}=-80{\rm{dBm}}$. We set $\eta_{n}=8,\forall n$,
and $K=16$. The benchmark schemes are: 1) the algorithm in [8, 14],
implemented by its open source code[23], which applies BCD and CVX solver; and
2) the algorithm in [12], which applies BCD, alternating direction method of
multipliers (ADMM), and CVX solver. We stop all the algorithms under the same
stopping criterion
$\left|R_{k}({\bm{w}}^{\ell+1},\bm{\phi}^{\ell+1})-R_{k}({\bm{w}}^{\ell},\bm{\phi}^{\ell})\right|\leq
10^{-4}$. In addition to the total power constraint (2b), we also consider the
design problem under the per-antenna power constraint (3). All the results are
averaged over 200 independent simulation trials.

Fig. 2: Multiuser sum rate of different algorithms.

In Fig. 2, we show the sum rate of all users achieved by different algorithms
under different power budgets $P_{\max}$, where $P_{\max}$ denotes the overall
power budget of BS and active RIS. It is seen that all tested algorithms
achieve comparable performance under both total power constraint and per-
antenna power constraint. However, their runtime performance can be quite
different. Figs. 3-4 show the average runtime of the considered algorithms
under different MIMO-RIS system sizes. In particular, Fig. 3 shows the average
runtime over different number of RIS elements $N$ with fixed number of BS
antenna elements $M$, while Fig. 4 shows the average runtime over different
number of antenna elements $M$ at the BS with fixed number of RIS elements
$N$. The overall power budget is set as $P_{\max}=30$ dBm; $P_{A}$ and $P_{B}$
are set to 0.01 and 0.99 times $P_{\max}$, respectively. First, it is seen
that under all the tested settings our proposed algorithm is much faster than
the other algorithms. Second, it is seen that our proposed algorithm shows
better scalability with respect to the problem size. For instance, in Fig. 4,
when $M=512$, the average runtimes of benchmark algorithms exceed 900 seconds,
while our proposed algorithm consumes only about 0.8 seconds; in other words,
our algorithm is more than 1,000 times faster.

To conclude, we have proposed an efficient BSUM method for the SRM problem in
the active RIS system. The algorithm has simple closed-form solution in each
step. The simulation results have shown that the proposed algorithm is
significantly faster than existing algorithms.

Fig. 3: Average runtime under different $N$; $M=256$. Fig. 4: Average runtime
under different $M$; $N=128$.

##  Appendix A Proof of Fact 2

Consider the projection problem

$\begin{split}\hat{\bm{\phi}}&~{}=\arg\min_{{\bm{y}}}\|{\bm{y}}-\bm{\phi}\|_{2}^{2}\\\\[-3.0pt]
\mbox{s.t.}&~{}|y_{n}|^{2}\leq\eta_{n}^{2},~{}n=1,2,\dots,N,~{}~{}\textstyle\sum_{i=1}^{N}\lambda_{i}|y_{i}|^{2}\leq
P_{A}.\end{split}$ (12)

Here, note that $\bm{\Lambda}$ is a diagonal matrix and $\lambda_{i}$ denote
its diagonal elements. The associated Lagrangian function is given by

$\textstyle{\cal
L}_{\gamma,\bm{\beta}}({\bm{y}})=\|{\bm{y}}-\bm{\phi}\|_{2}^{2}+\gamma\sum_{i=1}^{N}\lambda_{i}|y_{i}|^{2}+\sum_{i=1}^{N}\beta_{i}|y_{i}|^{2},$
(13)

where $\gamma\geq 0$ and $\\{\beta_{i}\\}\geq 0$ are the dual variables. The
Karush–Kuhn–
Tucker (KKT) conditions are given by

$\begin{split}&\frac{\partial{\cal L}_{\gamma,\bm{\beta}}({\bm{y}})}{\partial
y_{i}}=y_{i}-\phi_{i}+\gamma\lambda_{i}y_{i}+\beta_{i}y_{i}=0,~{}\forall i,\\\
&\textstyle\sum_{i=1}^{N}\lambda_{i}|y_{i}|^{2}\leq P_{A},~{}\gamma\geq
0,~{}|y_{i}|^{2}\leq\eta_{i}^{2},~{}\beta_{i}\geq 0,\\\
&\textstyle\gamma(\sum_{i=1}^{N}\lambda_{i}|y_{i}|^{2}-P_{A})=0,~{}\beta_{i}(|y_{i}|^{2}-\eta_{i}^{2})=0.\end{split}$

From the first line of KKT condition, we get

$\textstyle y_{i}=\frac{1}{1+\gamma\lambda_{i}+\beta_{i}}\phi_{i},$ (14)

which implies that $y_{i}$ and $\phi_{i}$ have the same angle. Also, the
amplitude $|y_{i}|$ decreases as $\gamma$ and/or $\beta_{i}$ increase. It
remains to specify the amplitude $|y_{i}|$’s. By the complementary slackness
in the last line, we know that if $|y_{i}|<\eta_{i}$, then $\beta_{i}=0$ must
hold. This, together with (14), leads to

$\textstyle
y_{i}=\min\big{\\{}\frac{1}{1+\gamma\lambda_{i}}|\phi_{i}|,\eta_{i}\big{\\}}e^{\mathfrak{j}\angle\phi_{i}}.$
(15)

Again, by the complementary slackness with respect to $\gamma$, we know if
$\sum_{i=1}^{N}\lambda_{i}|y_{i}|^{2}<P_{A}$, then $\gamma=0$ must hold and
from $y_{i}=\min\\{|\phi_{i}|,\eta_{i}\\}e^{\mathfrak{j}\angle\phi_{i}}$.
Otherwise, $\gamma$ is chosen such that
$\sum_{i=1}^{N}\lambda_{i}|y_{i}|^{2}=P_{A}$. The proof is complete.

## References

  * [1] L. Zhang, X. Q. Chen, S. Liu, Q. Zhang, J. Zhao, J. Y. Dai, G. D. Bai, X. Wan, Q. Cheng, G. Castaldi, V. Galdi, and T. J. Cui,  “Space-time-coding digital metasurfaces,”  Nat. Commun., vol. 9, no. 4338, 2018.
  * [2] H. Ren,  “A light-programmable metasurface,”  Nat. Elect., vol. 3, pp. 137–138, 2020.
  * [3] S. Venkatesh, X. Lu, H. Saeidi, and K. Sengupta,  “A high-speed programmable and scalable terahertz holographic metasurface based on tiled cmos chips,”  Nat. Elect., vol. 3, pp. 785–793, 2020.
  * [4] H. Guo, Y. -C. Liang, J. Chen, and E. G. Larsson,  “Weighted sum-rate maximization for reconfigurable intelligent surface aided wireless networks,”  IEEE Trans. Wireless Commun., vol. 19, no. 5, pp. 3064–3076, 2020\.
  * [5] C. Huang, A. Zappone, G. C. Alexandropoulos, M. Debbah, and C. Yuen,  “Reconfigurable intelligent surfaces for energy efficiency in wireless communication,”  IEEE Trans. Wireless Commun., vol. 18, no. 8, pp. 4157–4170, 2019\.
  * [6] M. Shao, Q. Li, and W. -K. Ma,  “Minimum symbol-error probability symbol-level precoding with intelligent reflecting surface,”  IEEE Wireless Commun. Lett., vol. 9, no. 10, pp. 1601–1605, 2020\.
  * [7] R. Liu, M. Li, Q. Liu, A. L. Swindlehurst, and Q. Wu,  “Intelligent reflecting surface based passive information transmission: A symbol-level precoding approach,”  IEEE Trans. Veh. Technol., vol. 70, no. 7, pp. 6735–6749, 2021\.
  * [8] Z. Zhang, L. Dai, X. Chen, C. Liu, F. Yang, R. Schober, and H. V. Poor,  “Active RIS vs. passive RIS: Which will prevail in 6G?,”  IEEE Trans. Commun., vol. 71, no. 3, pp. 1707–1725, 2023.
  * [9] R. Long, Y. -C. Liang, Y. Pei, and E. G. Larsson,  “Active reconfigurable intelligent surface aided wireless communications,”  IEEE Trans. Wireless Commun, vol. 20, no. 8, pp. 4962–4975, 2021\.
  * [10] C. You and R. Zhang,  “Wireless communication aided by intelligent reflecting surface: Active or passive?,”  IEEE Wireless Commun. Lett., vol. 10, no. 12, pp. 2659–2663, 2021\.
  * [11] M. Grant and S. Boyd,  “CVX: Matlab software for disciplined convex programming, version 2.1 beta,”  2014,  [Online]. Available: http://cvxr.com/cvx.
  * [12] Q. Zhu, M. Li, R. Liu, Y. Liu, and Q. Liu,  “Joint beamforming designs for active reconfigurable intelligent surface: A sub-connected array architecture,”  IEEE Trans. Commun., vol. 70, no. 11, pp. 7628–7643, 2022.
  * [13] R. Allu, O. Taghizadeh, S. K. Singh, K. Singh, and C. -P. Li,  “Robust beamformer design in active RIS-assisted multiuser MIMO cognitive radio networks,”  IEEE Trans. Cognit. Commun. Netw., vol. 9, no. 2, pp. 398–413, 2023\.
  * [14] K. Liu, Z. Zhang, L. Dai, S. Xu, and F. Yang,  “Active reconfigurable intelligent surface: Fully-connected or sub-connected?,”  IEEE Commun. Lett., vol. 26, no. 1, pp. 167–171, 2022.
  * [15] M. Razaviyayn, M. Hong, and Z. -Q. Luo,  “A unified convergence analysis of block successive minimization methods for nonsmooth optimization,”  SIAM J. Optim., vol. 23, no. 2, pp. 1126–1153, 2013.
  * [16] K. L. Keys, H. Zhou, and K. Lange,  “Proximal distance algorithms: Theory and practice,”  J. Machine Learning Research, vol. 20, no. 1, pp. 2384–2421, 2019\.
  * [17] L. Dong, H. -M. Wang, and J. Bai,  “Active reconfigurable intelligent surface aided secure transmission,”  IEEE Trans. Veh. Technol., vol. 71, no. 2, pp. 2181–2186, 2022\.
  * [18] H. Niu, Z. Lin, K. An, J. Wang, G. Zheng, N. Al-Dhahir, and K. -K. Wong,  “Active RIS assisted rate-splitting multiple access network: Spectral and energy efficiency tradeoff,”  IEEE J. Sel. Areas Commun., vol. 41, no. 5, pp. 1452–1467, 2023\.
  * [19] Q. Shi, M. Razaviyayn, Z. -Q. Luo, and C. He,  “An iteratively weighted MMSE approach to distributed sum-utility maximization for a MIMO interfering broadcast channel,”  IEEE Trans. Signal Process., vol. 59, no. 9, pp. 4331–4340, 2011\.
  * [20] L. T. Watson and R. T. Haftka,  “Modern homotopy methods in optimization,”  Computer Methods in Applied Mechanics and Engineering, vol. 74, no. 3, pp. 289–305, 1989.
  * [21] D. M. Dunlavy and D. P. O’Leary,  “Homotopy optimization methods for global optimization,”  Report SAND2005-7495, Sandia National Laboratories, 2005.
  * [22] M. Shao and W. -K. Ma,  “Binary MIMO detection via homotopy optimization and its deep adaptation,”  IEEE Trans. Signal Process., vol. 69, pp. 781–796, 2021.
  * [23] K. Liu, Z. Zhang, L. Dai, S. Xu, and F. Yang,  “Source code of “active reconfigurable intelligent surface: Fully-connected or sub-connected?”,” http://oa.ee.tsinghua.edu.cn/dailinglong/publications/publications.html, accessed on Jun. 15, 2023.