abbr

# Convex Support Vector Regression

Zhiqiang Liao  Sheng Dai  Timo Kuosmanen

(September 2022)

###### Abstract

Nonparametric regression subject to convexity or concavity constraints is
increasingly popular in economics, finance, operations research, machine
learning, and statistics. However, the conventional convex regression based on
the least squares loss function often suffers from overfitting and outliers.
This paper proposes to address these two issues by introducing the convex
support vector regression (CSVR) method, which effectively combines the key
elements of convex regression and support vector regression. Numerical
experiments demonstrate the performance of CSVR in prediction accuracy and
robustness that compares favorably with other state-of-the-art methods.
Keywords: Robustness and sensitivity analysis, Convex regression, Support
vector regression, Overfitting, Regularization

##  1 Introduction

Convex regression (CR) is a classic approach to nonparametric regression that
builds upon global concavity or convexity of the regression function
(hildreth1954point, *hildreth1954point). Since the explicit piecewise linear
characterization of the multivariate model proposed by
kuosmanen2008representation, CR has become an active research field with an
increasing number of applications in economics, statistics, operational
research and related fields (see, e.g., guntuboyina2018nonparametric,
*guntuboyina2018nonparametric; johnson2018shape, *johnson2018shape). Recent
methodological advances in CR include extensions to quantile-based approaches
such as convex quantile regression (wang2014nonparametric,
*wang2014nonparametric; kuosmanen2015stochastic, *kuosmanen2015stochastic) and
convex expectile regression (kuosmanen2020how, *kuosmanen2020how;
kuosmanen2021shadow, *kuosmanen2021shadow). There has been significant
development in the computational tools and algorithms (see, e.g., lee2013a,
*lee2013a; mazumder2019a, *mazumder2019a; Dai2022c, *Dai2022c;
lin2022augmented, *lin2022augmented).

Overfitting is a longstanding issue in nonparametric methods, including CR.
The subgradients fitted by CR can be very large near the boundary of the
convex hull of the design points (seijo2011nonparametric,
*seijo2011nonparametric; chen2020on, *chen2020on), which can seriously hamper
the out-of-sample predictive power. To alleviate overfitting, [lim2014on] has
proposed to restrict the domain of the convex hull by imposing additional
constraints on the subgradients of the regression function. Another approach
bound subgradients is to impose regularization either in objective function or
constraints, such as the $L_{2}$-norm Lipschitz regularization (mazumder2019a,
*mazumder2019a) or the $L_{\infty}$-norm Lipschitz regularization
(balazs2015near, *balazs2015near).

In the literature on machine learning, support vector regression (SVR) is a
well-known approach firstly introduced by vapnik1999the. SVR deviates from the
linear regression in that it introduces an $\varepsilon$-insensitive loss
function instead of the commonly used $L_{2}$-norm loss function, which helps
to improve its out-of-sample performance (vapnik1999the, *vapnik1999the).
Therefore, SVR has been considered as a robust alternative against outliers
and to reduce overfitting in the context of linear regression.

Thus far, only few studies extend SVR to the context of shape-constrained
regression or frontier estimation. The pioneering work by wang2012multivariate
was the first one to consider nonparametric convexity-constrained support
vector regression (henceforth NCCSVR). In this approach, the Hessian matrix of
a nonparametric representor function is constrained to be positive
semidefinite in each observation. The authors transform the shape-constrained
SVR into a semidefinite programming problem, assuming the regression function
to be continuous and twice differentiable throughout its domain. One notable
limitation of NCCSVR is that it is not applicable for the univariate CR with a
single regressor.

Recently, valero-carreras2021support and [valero2022multi] relax continuity
and convexity assumptions, adapting SVR to the nonparametric estimation of
frontier production and cost functions that envelop all observations. A
notable limitation of their approach is the deterministic nature of the data
generating process, which assumes away any noise in data. This observation
motivates us to combine the key elements of both CR and SVR in a unified
framework.

The main objective of the present paper is to improve the out-of-sample
predictive power of CR by alleviating overfitting. To this end, we combine the
key characteristics of both CR and SVR in a new approach referred to as convex
support vector regression (CSVR). A notable difference between NCCSVR by
wang2012multivariate and the proposed CSVR approach concerns the
implementation of the convexity constraints: NCCSVR imposes constraints on the
Hessian matrix, whereas CSVR makes use of the inequality constraints known as
the Afriat inequalities (see kuosmanen2008representation,
*kuosmanen2008representation). Using Monte Carlo simulations and two real-
world examples, we show that the proposed CSVR approach yields a smaller mean
squared error (MSE) than other state-of-the-art methods, including the NCCSVR
method.

Our secondary objective is to outline how the proposed CSVR approach can be
extended to facilitate the automatic variable selection in applications with
high dimensionality. Inspired by such works as [bradley1998feature],
[zhao2009composite] and [negahban2011simultaneous], two alternative
formulations of LASSO CSVR are considered (LASSO refers to the least absolute
shrinkage and selection operator). This extension further enhances the
linkages between SVR and LASSO that originated in the machine learning
literature and CR that has emerged in econometrics and statistics.

The rest of this paper is organized as follows. Section 2 briefly reviews
classical statistics and machine learning methods for regression problems. We
then introduce the new shape-constrained SVR method, extend it to the Lasso
version, and present a graphical illustration in section 3. Section 4 presents
some evidence from Monte Carlo simulations. In the section 5 we experimentally
compare CSVR against competing methods on two real-world datasets. Section 6
presents our concluding remarks.

##  2 Preliminaries on regression

###  2.1 Convex regression

Considering a general nonparametric regression model with a set of
observations $\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{n}$ satisfying

$y_{i}=f(\mathbf{x}_{i})+\varepsilon_{i},\quad\mbox{ for }i=1,\ldots,n,$ (1)

where $\mathbf{x}\in\mathbb{R}^{d}$ is an observed vector of predictors,
$y_{i}\in\mathbb{R}$ is the response variable, and $\varepsilon_{i}$ is a
random noise with zero mean. The regression function
$f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ in Eq. (1) is unknown but satisfies
certain shape restrictions such as monotonicity, concavity, and homogeneity
(see, e.g., kuosmanen2010data, *kuosmanen2010data; yagi2020shape,
*yagi2020shape). In this paper we focus exclusively on the class $\mathcal{F}$
of concave function $f$, that is

$\mathcal{F}:=\Big{\\{}f:\mathbb{R}^{d}\rightarrow\mathbb{R}\mid\forall\mathbf{x}_{1},\mathbf{x}_{2}\in\mathbb{R}^{d},\tau
f(\mathbf{x}_{1})+(1-\tau)f(\mathbf{x}_{2})\leq
f(\tau\mathbf{x}_{1}+(1-\tau)\mathbf{x}_{2})\Big{\\}}.$

The basic idea of CR is to find the best fitting function $f$ from a family of
continuous and concave functions $\mathcal{F}$ by minimizing the sum of
squares of the residuals

$\displaystyle\min\,$
$\displaystyle\frac{1}{2}\sum_{i=1}^{n}(y_{i}-f(\mathbf{x}_{i}))^{2}$ (2) s.t.
$\displaystyle f\in\mathcal{F}$

While problem (2) is the infinite-dimensional multivariate convex regression
problem, it can be equivalently represented by a finite-dimensional quadratic
programming (QP) problem. Following kuosmanen2008representation, we consider
the following least squares estimator as the operational multivariate convex
regression model

$\displaystyle\min_{\boldsymbol{\beta},\boldsymbol{\alpha},\boldsymbol{\varepsilon}}\quad$
$\displaystyle\frac{1}{2}\sum_{i=1}^{n}\varepsilon_{i}^{2}$ (3) s.t.
$\displaystyle
y_{i}=\alpha_{i}+\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}+\varepsilon_{i}$
$\displaystyle\forall i$
$\displaystyle\alpha_{i}+\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}\leq\alpha_{h}+\boldsymbol{\beta}_{h}^{{}^{\prime}}\mathbf{x}_{i}$
$\displaystyle\forall i,h$

where $\boldsymbol{\beta}^{{}^{\prime}}$ indicates the transpose of
$\boldsymbol{\beta}$, and its subscript $h$ is any index of data point not
equal to $i$. The first constraint of (3) simply restates the regression
equation (1) in terms of a piecewise linear approximation of the true but
unknown regression function $f$, and the second constraint enforces concavity
of the piecewise linear regression function (reversing the sign of the
inequality imposes convexity). Note that additional monotonicity constraints
could be implemented by restricting the sign of $\boldsymbol{\beta}$ (e.g.,
$\boldsymbol{\beta}\geq 0$ for monotonic increasing and
$\boldsymbol{\beta}\leq 0$ for monotonic decreasing functions). Further,
imposing $\alpha=0$ imposes linear homogeneity (constant returns to scale).
See kuosmanen2015stochastic for more detailed discussion.

Given the optimal solutions ($\hat{\alpha}_{i}$,
$\boldsymbol{\hat{\beta}}_{i}$) to problem (3), we can reconstruct the
explicit representor function $\hat{f}^{CR}(\mathbf{x})$ as
(kuosmanen2008representation, *kuosmanen2008representation)

$\hat{f}^{CR}(\mathbf{x})=\min_{i=1,\ldots,n}\big{\\{}\hat{\alpha}_{i}+\boldsymbol{\hat{\beta}}_{i}^{{}^{\prime}}\mathbf{x}\big{\\}}$
(4)

However, the estimated coefficients $\hat{\boldsymbol{\beta}}_{i}$ could be
arbitrarily large, particularly near the boundary of the convex hull of the
covariate domain, due to the fact that the feasible set of problem (3) can be
unbounded (see, e.g., mazumder2019a, *mazumder2019a; chen2020on, *chen2020on).
This may lead to potential overfitting and deteriorate the out-of-sample
performance of the estimated function. Even for the univariate case, the
estimated $\hat{\beta}_{i}$ are also unbounded at the boundary
(ghosal2017univariate, *ghosal2017univariate).

To alleviate the overfitting problem in convex regression (3), [mazumder2019a]
apply the $L_{2}$-norm Lipschitz regularization on subgradients with a known
bound $L>0$ to reduce overfitting. For a prespecified $L>0$, the class of
$F_{L}$ of concave functions with Lipschitz regularization

$F_{L}:=\\{f:\mathbb{R}^{d}\rightarrow\mathbb{R}|f\text{ }\mbox{is
concave};\underset{\mathbf{x}\in\mathbb{R}^{d}}{\sup}\|\partial
f(\mathbf{x})\|\leq L\\}.$

where $\|\partial f(\mathbf{x})\|$ is the maximum of $\|\cdot\|_{2}$-norm of
vectors in $\partial f(\mathbf{x})$. The corresponding Lipschitz convex
regression (LCR) is then formulated as

$\displaystyle\min_{\boldsymbol{\beta},\boldsymbol{\alpha},\boldsymbol{\varepsilon}}\quad$
$\displaystyle\frac{1}{2}\sum_{i=1}^{n}\varepsilon_{i}^{2}$ (5) s.t.
$\displaystyle
y_{i}=\alpha_{i}+\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}+\varepsilon_{i}$
$\displaystyle\forall i$
$\displaystyle\alpha_{i}+\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}\leq\alpha_{h}+\boldsymbol{\beta}_{h}^{{}^{\prime}}\mathbf{x}_{i}$
$\displaystyle\forall i,h$ $\displaystyle\|\boldsymbol{\beta}_{i}\|_{2}\leq L$
$\displaystyle\forall i$

Note that the estimated function $\hat{f}^{LCR}$ can be obtained by inserting
the optimal $\hat{\alpha}_{i}$ and $\boldsymbol{\hat{\beta}}_{i}$ from (5) to
the equation of explicit representor function (4). We can also resort to other
Lipschitz norms (e.g., $\|\cdot\|_{\infty}$-norm) to address the overfitting
problem in convex regression (lim2014on, *lim2014on; balazs2015near,
*balazs2015near). Following [mazumder2019a], $\|\cdot\|_{2}$-norm is more
effective than $\|\cdot\|_{\infty}$-norm in avoiding overfitting. However,
when the data with high noise, the outliers and the large variance in the
error term could weaken the effectiveness of the Lipschitz norm approaches in
reducing overfitting. Such a gap also motivates this paper to propose a new
convex support vector regression approach and empirically compare the proposed
method with other related approaches in terms of finite sample performance.

###  2.2 Support vector regression

Support vector regression (SVR) belongs to the class of support vector
machines. As a regression method, it aims to estimate a function $f(x)$ that
follows the structural risk minimization principle grounded on the statistical
learning theory. It gives good generalization capacity by minimizing the upper
bound of the risk (vapnik1999the, *vapnik1999the).

SVR has excellent potential to reduce overfitting because the structural risk
minimization principle makes a trade-off between the prediction accuracy and
the complexity of the regression function. When some unwanted data exit, some
parameters in the conventional regression models may become large to
accommodate such outliers. Thus, these models may suffer from overfitting,
while the SVR has good generalization performance as its optimization object
function guarantees the flatness of the regression function. Flatness in the
regression model means that one minimizes vector $\boldsymbol{\beta}$. Taking
into account data errors, one can introduce the slack variables $\xi$ and
$\xi^{*}$ and penalty term $C$. The slack variables can be used to allow some
errors that lie on the outside of the margin $\varepsilon$ which refers to
soft margin (cortes1995support, *cortes1995support). Hence we can obtain the
regression function by solving the following optimization problem

$\displaystyle\min_{\boldsymbol{\beta},\alpha,\xi_{i},\xi_{i}^{*}}\quad$
$\displaystyle\frac{1}{2}\|\boldsymbol{\beta}\|^{2}_{2}+C\sum_{i=1}^{n}(\xi_{i}+\xi_{i}^{*})$
(6) s.t. $\displaystyle
y_{i}-\alpha-\boldsymbol{\beta}^{{}^{\prime}}\mathbf{x}_{i}\leq\varepsilon+\xi_{i}$
$\displaystyle\forall i$
$\displaystyle\alpha+\boldsymbol{\beta}^{{}^{\prime}}\mathbf{x}_{i}-y_{i}\leq\varepsilon+\xi_{i}^{*}$
$\displaystyle\forall i$ $\displaystyle\xi_{i}\geq 0,\xi_{i}^{*}\geq 0$
$\displaystyle\forall i$

where $C$ is a prespecified parameter that determines the trade-off between
the complexity of regression function $f(x)$ and the prediction accuracy
$\varepsilon$. This corresponds to the most commonly adopted
$\varepsilon$-insensitive loss function $\lvert\xi\rvert_{\varepsilon}$
described by

$|\xi|_{\varepsilon}=\begin{cases}0,&\text{if $|\xi|\leq\varepsilon$;}\\\
|\xi|-\varepsilon,&\text{otherwise.}\end{cases}$ (7)

for a user-determined nonnegative number $\varepsilon$. A potential benefit of
using $\varepsilon$-insensitive loss function is robustness to outliers
because it is less sensitive to noisy inputs. In practice, the penalty
constant $C$ and parameter $\varepsilon$ can be chosen based on the user’s
experience. It can also be determined by cross-validation, a standard model
selection technique in machine learning. Furthermore, the estimated support
vector function is simply
$\hat{f}^{SVR}(\mathbf{x})=\alpha+\boldsymbol{\beta}^{{}^{\prime}}\mathbf{x}$.

##  3 Combining CR and SVR

###  3.1 Convex support vector regression (CSVR)

To address the overfitting problem and improve the model robustness, we blend
the key elements of CR and SVR and propose the convex support vector
regression (CSVR) approach. Consider the following quadratic programming
problem

$\displaystyle\min_{\boldsymbol{\beta}_{i},\alpha,\xi,\xi^{*}}\quad$
$\displaystyle\frac{1}{2}\sum_{i=1}^{n}\|\boldsymbol{\beta}_{i}\|_{2}^{2}+C\sum_{i=1}^{n}(\xi_{i}+\xi_{i}^{*})$
(8) s.t. $\displaystyle
y_{i}-\alpha_{i}-\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}\leq\varepsilon+\xi_{i}$
$\displaystyle\forall i$
$\displaystyle\alpha_{i}+\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}-y_{i}\leq\varepsilon+\xi_{i}^{*}$
$\displaystyle\forall i$
$\displaystyle\alpha_{i}+\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}\leq\alpha_{h}+\boldsymbol{\beta}_{h}^{{}^{\prime}}\mathbf{x}_{i}$
$\displaystyle\forall i,h$ $\displaystyle\xi_{i}\geq 0,\xi_{i}^{*}\geq 0$
$\displaystyle\forall i$

where the first two constraints restrict the error terms into a specified
margin, noted as the maximum error ($\varepsilon$), and consider the possible
outliers with the deviation from the margin as $\xi_{i}$. The third
constraint, a system of Afriat inequalities, guarantees the concavity of the
unknown function $f$. Compared to problem (6), problem (8) is a shape
constrained extension by means of Afriat inequalities. Note that the
coefficients $\boldsymbol{\beta}_{i}$ represent the subgradient of the concave
function $f$ at point $\mathbf{x}_{i}$.

To further investigate the relationship between the CSVR and regularized
function estimation, we rewrite problem (8) as the following equivalent
formulation

$\displaystyle\min_{\boldsymbol{\beta}_{i},\alpha}\quad$
$\displaystyle\sum_{i=1}^{n}|y_{i}-\alpha_{i}-\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}|_{\varepsilon}+\frac{A}{2}\sum_{i=1}^{n}\|\boldsymbol{\beta}_{i}\|^{2}_{2}$
(9) s.t.
$\displaystyle\alpha_{i}+\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}\leq\alpha_{h}+\boldsymbol{\beta}_{h}^{{}^{\prime}}\mathbf{x}_{i}\quad\forall
i,h$

where $A$ is the tuning parameter, playing a similar role as $C$. The function
$|\cdot|_{\varepsilon}$ indicates the $\varepsilon$-insensitive loss function
described by equation (7). Note that the objective function has a form of
loss+penalty; hence, the parameter $A$ controls the trade-off between loss and
penalty. The standard cross-validation techniques can be used to determine the
tuning parameter.

The penalty term in problem (9) is a $L_{2}$-norm of the subgradient vector,
the same as the penalized convex regression using the norm of the subgradients
(aybat2014a, *aybat2014a; lim2014on, *lim2014on; Dai2022, *Dai2022). The ridge
penalty shrinks the subgradients towards zero, which means the regression
function will be as flat as possible. This shrinkage can control the variance
of the subgradients, thus helping to alleviate the overfitting problem via the
bias-variance trade-off, especially when many highly correlated variables
exist. We will further study the effect of regularization in the following
section.

Using the $\varepsilon$-insensitive loss function also enables a sparse set of
support vectors to be obtained. That is, the number of support vectors
increases more slowly than linearly. As an extended SVR model, the CSVR
approach retains this advantage of sparsity. Moreover, the
$\varepsilon$-insensitive loss function is less sensitive to outliers than the
quadratic loss function used in problem (5). CSVR is more robust than
conventional CR when there are outliers in the dataset, therefore, leading to
a smaller prediction error than the usual CR; see Section 4 for simulation
evidence. When $\varepsilon=0$, we can obtain a special case of $L_{1}$ loss
function in problem (9). Note that the $L_{1}$ loss function is relatively
more robust to outliers than the quadratic loss function
(alquier2019estimation, *alquier2019estimation).

To derive a similar version to problem (5), letting
$\phi_{\varepsilon}(\cdot)$ be the $\varepsilon$-insensitive loss function,
problem (9) can be rewritten as fully constrained optimization problem as
follows

$\displaystyle\min_{\boldsymbol{\beta}_{i},\alpha}\quad$
$\displaystyle\sum_{i=1}^{n}\phi_{\varepsilon}(y_{i}-f(\mathbf{x}_{i}))$ (10)
s.t.
$\displaystyle\alpha_{i}+\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}\leq\alpha_{h}+\boldsymbol{\beta}_{h}^{{}^{\prime}}\mathbf{x}_{i}\quad\forall
i,h$ $\displaystyle\sum_{i=1}^{n}\|\boldsymbol{\beta}_{i}\|_{2}\leq C$

where $\sum_{i=1}^{n}\|\boldsymbol{\beta}_{i}\|_{2}\leq C$ is a $L_{2}$-norm
penalty, and $C$ is a nonnegative tuning parameter. We note here that the
$L_{2}$-norm penalty used in this study differs from the $L_{2}$-norm
Lipschitz regularization in [mazumder2019a]. [mazumder2019a] consider a least
square estimator over a set of convex functions that are uniformly Lipschitz
with a known bound (see problem (5)), whereas problem (10) is ridge
regularized. Also, note that we use a different loss function, whose
insensitivity may contribute to the performance of CSVR.

###  3.2 Lasso CSVR

In addition to $L_{2}$-norm, we could introduce other regularization methods
such as $L_{1}$-norm to extend the present CSVR approach. The $L_{1}$-norm
support vector machine was first proposed by [bradley1998feature] for solving
classification problems. Inspired by this idea, we briefly describe an
extension of CSVR: the Lasso CSVR model for the variable selection regression
analysis. This extension provides an alternative path of extending CSVR to
select variables automatically. The first version of Lasso CSVR replaces the
$L_{2}$-norm penalty in problem (9) with $L_{1}$-norm penalty

$\displaystyle\min_{\boldsymbol{\beta}_{i},\alpha}\quad$
$\displaystyle\sum_{i=1}^{n}|y_{i}-\alpha_{i}-\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}|_{\varepsilon}+\frac{A}{2}\sum_{i=1}^{n}\|\boldsymbol{\beta}_{i}\|_{1}$
(11) s.t.
$\displaystyle\alpha_{i}+\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}\leq\alpha_{h}+\boldsymbol{\beta}_{h}^{{}^{\prime}}\mathbf{x}_{i}\quad\forall
i,h$

Similar to the $L_{2}$-norm penalty, the $L_{1}$-norm penalty can control the
variance of the estimation and improve prediction accuracy. Moreover, the
Lasso performs automatic variable selection, which is not the case for the
$L_{2}$-norm penalty. Although the performance of Lasso does not uniformly
dominate the ridge regression (tibshirani1996regression,
*tibshirani1996regression), the $L_{1}$-norm CSVR appears very promising
because the variable selection is increasingly important in modern data
science.

The variable selection aspect of the $L_{1}$-norm CSVR approach is useful for
regression analysis in the case that there exist no highly correlated
variables (see the limitations of $L_{1}$-norm penalty in
zou2005regularization, *zou2005regularization). While the $L_{1}$-norm penalty
can shrink the subgradients of the function and make the subgradients of
irrelevant variables small, it cannot reduce them to zero exactly. The reason
is that selecting variables by regularizing the subgradient
$\boldsymbol{\beta}_{i}$ in problem (11) with a group sparsity penalty is not
an effective way due to the existing of Afriat inequalities (xu2016faithful,
*xu2016faithful; Dai2022c, *Dai2022c). That is, the small changes to
$\boldsymbol{\beta}_{i}$ in each Afrait inequality
$\alpha_{i}+\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}\leq\alpha_{h}+\boldsymbol{\beta}_{h}^{{}^{\prime}}\mathbf{x}_{i}$
may not violate the concavity assumption. Therefore, the $L_{1}$-norm CSVR
approach can make certain coefficients very small but not be zero, and thus
does not necessarily make the representor function (4) sparse.

This motivates us to consider a $L_{\infty}$-regularized Lasso CSVR. As shown
in [zhao2009composite] and [negahban2011simultaneous], the $L_{\infty}$-norm
taking the maximum encourages all $d$ components of the subgradient
$\boldsymbol{\beta}_{i}$ to be zero simultaneously or to be nonzero
simultaneously. The second version of Lasso CSVR can be formulated as the
following optimization problem

$\displaystyle\min_{\boldsymbol{\beta}_{i},\alpha}\quad$
$\displaystyle\sum_{i=1}^{n}|y_{i}-\alpha_{i}-\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}|_{\varepsilon}+\frac{A}{2}\sum_{i=1}^{n}\|\boldsymbol{\beta}_{i}\|_{\infty}$
(12) s.t.
$\displaystyle\alpha_{i}+\boldsymbol{\beta}_{i}^{{}^{\prime}}\mathbf{x}_{i}\leq\alpha_{h}+\boldsymbol{\beta}_{h}^{{}^{\prime}}\mathbf{x}_{i}\quad\forall
i,h$

Compared to the $L_{1}$-norm penalty, the $L_{\infty}$-norm penalty restricts
each subgradient $\beta_{j,i}$ with a fixed bound instead of all $d$
components of the subgradient $\boldsymbol{\beta}_{i}$. Furthermore, the
$L_{\infty}$-regularized Lasso approach is a special case of the block
$L_{1}/L_{\infty}$-regularization when the number of blocks is $n$ in
[negahban2011simultaneous].

###  3.3 Illustrative example

We proceed to demonstrate how the fitted functions estimated by CSVR and
convex regression are different and illustrate the potential advantages of
CSVR in reducing overfitting via an artificial example. In so doing, we
generate 50 observations with $y_{i}=3+\ln(x_{i})+\epsilon_{i}$, where $x_{i}$
is randomly drawn from $U(1,10)$ and the error term $\epsilon_{i}$ is
generated independently from $N(0,0.7^{2})$. To search the optimal
hyperparameters, we resort to the standard fivefold cross-validation approach
(see, e.g., mazumder2019a, *mazumder2019a; Dai2022c, *Dai2022c).

Fig. 1 depicts the fitted functions estimated by CSVR and convex regression.
We observe that both shape-constrained approaches yield piecewise concave
lines and can capture the shape of data points. The fitted convex regression
function appears to be more sharper, while the fitted CSVR functions seem to
be relatively smooth, implying the difference in the loss function. When
comparing the approximation errors, CR has a smaller in-sample error (i.e.,
$MSE=0.44$) in comparison with CSVR. However, a smaller training error may
lead to a larger test error, suggesting that CR is hampered by overfitting,
and CSVR can effectively avoid this problem.

Figure 1: Illustration of the fitted functions estimated by CSVR and convex
regression.

We then illustrate the robustness of the estimated CSVR function to the choice
of the tuning parameter $C$. In Fig. 2, the hyperparameter $\varepsilon$ is
fixed at 0.1 and $C$ is tuned over 5 values from the set $\\{1,2,4,6,10\\}$.
Note that the optimal hyperparameter $C^{*}=6$ is determined by the standard
cross-validation approach. It is evidently from Fig. 2 that the parameter $C$
can reshape the estimated CSVR functions but produce very similar estimated
piecewise-linear curves.

Figure 2: Illustration of the estimated CSVR functions with different values
of $C$.

##  4 Monte Carlo study

Having illustrated the estimated CSVR function, we proceed to investigate the
finite sample performance of the CSVR, CR, SVR with radial basis function
kernel, and LCR approaches in the controlled environment of Monte Carlo
simulations. The main objective of our simulations is to examine whether the
proposed CSVR approach can better fit the true function by addressing the
overfitting problem.

###  4.1 Setup

Consider the following data generating processes (DGP) (see, e.g., valero-
carreras2021support, *valero-carreras2021support)

  1. 1)

DGP I: $y=3+x_{1}^{0.5}+\epsilon$

  2. 2)

DGP II: $y=3+x_{1}^{0.2}+x_{2}^{0.3}+\epsilon$

  3. 3)

DGP III: $y=3+x_{1}^{0.05}+x_{2}^{0.15}+x_{3}^{0.3}+\epsilon$

where $x_{1}$, $x_{2}$, and $x_{3}$ are independently and randomly sampled
from the uniform distribution $U[1,10]$ and the error term $\epsilon$ is drawn
from $N(0,\sigma^{2})$. For each DGP, we consider 12 different scenarios with
different number of observations ($n=50,100,200,500$) and the levels of noise
($\sigma=0.5,1,2$). Each scenario is replicated 50 times to calculate the in-
sample and out-of-sample performances with the mean squared error (MSE)
statistic.

Regarding the tuning parameter selection, the optimal hyperparameters (i.e.,
$C$ and $\varepsilon$) are determined in the SVR and CSVR approaches by using
the fivefold cross-validation method. Following [valero-carreras2021support],
the tuning values $C$ and $\varepsilon$ are varied from the multiplier sets
$\\{0.1,0.5,1,2,5\\}$ and $\\{0,0.001,0.01,0.1,0.2\\}$, respectively. For the
LCR approach, as in mazumder2019a, we employ the one standard error rule in
cross-validation to search the optimal Lipschitz parameter $L$.

In the following experiments, we resort to the pyStoNED package (Dai2021b,
*Dai2021b) with the standard solver Mosek (9.2.44). All computations are
performed on Aalto University’s high-performance computing cluster Triton with
Xeon @2.8 GHz processors, 1 CPU, and 8 GB RAM per task. The simulation code
and data are available at the GitHub repository
(https://github.com/ds2010/CSVR).

###  4.2 In-sample performance

We first compare the in-sample performance of CSVR with alternatives. Table 1
reports the MSE statistic of each approaches with $n\in\\{50,100,200,500\\}$,
$d\in\\{1,2,3\\}$, and $\sigma=1$. Note that the MSEs of NCCSVR in the
univariate cases are missing due to the fact that the NCCSVR approach requires
the multidimensional data space (i.e., $d\geq 2$) (wang2012multivariate,
*wang2012multivariate). The optimal hyperparameters in LCR, CSVR, and NCCSVR
are prespecified via the standard cross-validation technique.

Table 1 indicates that CSVR exhibits the lowest values of MSE in almost all
scenarios. Compared to CSVR, LCR has competitive performance in the univariate
case (i.e., DGP I), but its performance deteriorates in multivariate cases.

Compared to the regularized approaches, the traditional CR approach performs
poorly in the multivariate cases (i.e., DGP II and III), even though it is
actually quite competitive in the univariate setting. After restricting the
subgradients in CR, the performance in fitting the true function will be
improved as the optimal subgradients cannot take any values for given
feasibility. As expected, the performance of each approach deteriorates as
more input variables or smaller sample sizes are introduced.

Table 1: MSE comparison of different approaches with $\sigma=1$. DGP | $n$ | CSVR | SVR | CR | LCR | NCCSVR
---|---|---|---|---|---|---
I | 50 | 0.0644 | 0.1099 | 0.0838 | 0.0673 | –
| 100 | 0.0384 | 0.0694 | 0.0427 | 0.0377 | –
| 200 | 0.0209 | 0.0403 | 0.0252 | 0.0192 | –
| 500 | 0.0086 | 0.0152 | 0.0107 | 0.0080 | –
II | 50 | 0.0771 | 0.1125 | 0.2193 | 0.1674 | 0.1085
| 100 | 0.0386 | 0.0696 | 0.1270 | 0.0904 | 0.0596
| 200 | 0.0271 | 0.0400 | 0.0829 | 0.0536 | 0.0300
| 500 | 0.0083 | 0.0188 | 0.0341 | 0.0222 | 0.0123
III | 50 | 0.0758 | 0.0994 | 0.4189 | 0.3198 | 0.1543
| 100 | 0.0556 | 0.0788 | 0.2765 | 0.2124 | 0.0883
| 200 | 0.0365 | 0.0484 | 0.1855 | 0.1276 | 0.0453
| 500 | 0.0184 | 0.0274 | 0.0974 | 0.0599 | 0.0188

To assess the robustness of the CSVR approach, we next consider scenarios with
different levels of error variance (i.e., $\sigma$) using a fixed sample size
$n=500$ (see Table 2). As expected, we observe that the MSE values increase
for all methods as the data noise variance increases. However, the proposed
CSVR approach has the smallest MSEs in all cases and is robust to the
increasing noise. Note that those methods that use the regularization
techniques are more robust to noise than the original CR approach. Overall,
CSVR maintains its good performance in different levels of data noise and
dimensions.

Table 2: MSE comparison of different approaches with $n=500$. DGP | $\sigma$ | CSVR | SVR | CR | LCR | NCCSVR
---|---|---|---|---|---|---
I | 0.5 | 0.0030 | 0.0043 | 0.0031 | 0.0025 | –
| 1 | 0.0086 | 0.0152 | 0.0107 | 0.0080 | –
| 2 | 0.0257 | 0.0522 | 0.0392 | 0.0271 | –
II | 0.5 | 0.0038 | 0.0076 | 0.0090 | 0.0066 | 0.0043
| 1 | 0.0083 | 0.0188 | 0.0341 | 0.0222 | 0.0123
| 2 | 0.0214 | 0.0482 | 0.1341 | 0.0755 | 0.0494
III | 0.5 | 0.0077 | 0.0114 | 0.0248 | 0.0185 | 0.0070
| 1 | 0.0184 | 0.0274 | 0.0974 | 0.0599 | 0.0188
| 2 | 0.0493 | 0.0642 | 0.3874 | 0.1862 | 0.0671

Compared to other regularized approaches, CSVR is relatively more robust when
the data noise varies from 0.5 to 2 (see Table 2). While the LCR approach can
benefit from the Lipschitz regularization and avoid overfitting, it still uses
the squared L2 norm loss function and thus tends to be sensitive to outliers
and heteroscedasticity. However, the CSVR approach introduces the
$\epsilon$-insensitive loss function inherited from SVR to increase
robustness. For smaller $\sigma$ values, as shown in Table 2, it seems that
NCCSVR may outperform the proposed CSVR method (i.e. DGP III, $\sigma=0.5$).

Recall that [wang2012multivariate] have proposed a similar NCCSVR approach,
where they introduce convexity/concavity into SVR by using the Hessian matrix
and then transform it to a semidefinite programming problem. In contrast, our
CSVR approach resorts to the system of Afriat inequalities to ensure the
fitted function to be convex/concave. Moreover, the NCCSVR approach can be
applied to the multivariate cases only (i.e., $d\geq 2$) and requires one more
tuning parameter. To further understand the performance difference between
these two similar approaches, we implement the following additional experiment
to report the prediction accuracy.

In previous experiments, we have observed that the variation of the noise
$\sigma$ has a significant impact on the MSE values, especially in the cases
with small $\sigma$. Here we investigate the impact of noise variation on
these two models’ performance. Fig. 3 depicts the performance of CSVR and
NCCSVR when $n=500$, $d=\\{2,3\\}$, and $\sigma$ varies from 0.2 to 3. Fig. 3
a) indicates that the performance of CSVR dominates NCCSVR, whereas Fig. 3 b)
shows that there are tiny differences in the finite sample performances
between NCCSVR and CSVR, but increasing larger differences occur as $\sigma$
grows. This might suggest that imposing the Hessian matrix is not as efficient
as the system of Afriat inequalities in terms of overfitting reduction.

(a) $d=2$

(b) $d=3$

Figure 3: Illustration of the impacts of noise variation on MSE.

We further investigate the impact of outliers. The additional five outliers
are drawn from the uniform distribution $U[90,100]$, and the other normal
observations are also drawn from $U[1,10]$. For the sake of illustration, we
simply consider the instances with $n\in\\{50,100\\}$, $d\in\\{2,3\\}$, and
$\sigma=1$. The in-sample MSEs of each scenario are averaged in a total of 50
replications. As expected, Table 3 shows that the CSVR approach performs best
among all compared approaches in terms of prediction accuracy. The Lipschitz
norm convex regression approach described in mazumder2019a can also control
the impact of outliers and, in this case, outperforms the conventional SVR
approach, which would have better performance if there were no additional
outliers. Furthermore, compared to results reported in Table 1, the results
shown in Table 3 imply that the additional outliers can lead to worse accuracy
for all methods. We also observe that the performance of NCCSVR varies sharply
with the parameters, but this does not happen in CSVR. Moreover, the choice of
its kernel parameter is highly dependent on the values of input data, which
might lead to a limitation in practical applications. Thus it could be hurt
for practitioners to decide the range of the three parameters used in NCCSVR.

Table 3: MSE comparison of different approaches with additional five outliers. DGP | $n$ | CSVR | SVR | CR | LCR | NCCSVR
---|---|---|---|---|---|---
II | 50 | 0.0979 | 0.1900 | 0.1948 | 0.1477 | 0.1096
| 100 | 0.0563 | 0.1175 | 0.1154 | 0.0879 | 0.0595
III | 50 | 0.0996 | 0.2251 | 0.4093 | 0.1034 | 0.1556
| 100 | 0.0658 | 0.1115 | 0.2074 | 0.0897 | 0.1065

###  4.3 Out-of-sample performance

We proceed to investigate out-of-sample performance by considering six
scenarios with $d\in\\{2,3\\}$, $\sigma\in\\{0.5,1,2\\}$, and $n=500$ for the
training set and another 1000 hold-out observations for the test set. We then
replicate each scenario 50 times to obtain an empirical distribution of the
out-of-sample MSE statistic. Note that the in-sample overfitting may result in
low prediction accuracy of the out-of-sample model, and if an estimator is
likely to overfit in multidimensional data space, then it will have a smaller
in-sample MSE and a larger out-of-sample MSE.

The boxplots in Figs. 4 and 5 illustrate the distributions of out-of-sample
MSE. When comparing traditional CR with regularized alternatives, we observe
that CR has a relatively large out-of-sample MSE, which is far more than that
of the other regularized approaches. For instance, the values of out-of-sample
MSE for CR are $0.121(\pm 0.342)$, $0.454(\pm 1.192)$, and $1.705(\pm 4.300)$
respectively with $d=2$ and $\sigma\in\\{0.5,1,2\\}$. To facilitate a
comparison of the most competitive alternatives, we exclude the MSE results
for CR from Figs. 4 and 5.

Figure 4: Out-of-sample MSE of the four methods with $d=2$ and
$\sigma\in\\{0.5,1,2\\}$.

CSVR performs better than other methods in alleviating the overfitting
problem. Compared to the LCR, the three SVR-based approaches seem to perform
better in alleviating the overfitting problem. The $\epsilon$-insensitive loss
function used in SVR-based approaches is more robust to outliers and large
errors than the least squares loss function, which, in turn, helps to build a
robust prediction model as demonstrated in Fig. 5. In the experiments, we also
find that the out-sample-sample performance of LCR deteriorates rapidly as the
data noise increases.

Figure 5: Out-of-sample MSE of the four methods with $d=3$ and
$\sigma\in\\{0.5,1,2\\}$.

##  5 Experiments with real data

In this section, we apply the proposed approach to two real-world datasets:
the Boston housing data and the NBER-CES manufacturing industry data. Those
real datasets have been used extensively in various fields of economics,
econometrics, statistics, and machine learning for different purposes such as
new models test or algorithms benchmarking (see, e.g., lin2011vif,
*lin2011vif; wang2013estimating, *wang2013estimating). In both examples below,
all hyperparameters are tuned over from 50 candidate values via the fivefold
cross-validation technique, and the in-sample MSE and out-of-sample MSE for
each approach are calculated to compare the effectiveness in reducing
overfitting.

###  5.1 Boston housing data

This dataset contains housing price information in the Boston area collected
from the StatLib archive.111 StatLib–Datasets Archive:
http://lib.stat.cmu.edu/datasets/boston.  The data includes 13 variables with
506 observations.222Note that the dummy variable (i.e., CHAS) is excluded from
the dataset for the sake of simplicity. Note that CR can handle contextual
variables, but this falls beyond the scope of this paper; cf. johnson2012one
(*johnson2012one), for further details. Following the commonly used setting,
the variable _MEDV_ is taken as the response variable of interest, and others
are the explanatory variables. The descriptive statistics for all variables
are presented in Table A1.

Table 4 reports the descriptive statistics for the estimated coefficients
(i.e., $\hat{\alpha}_{i}$ and $\hat{\boldsymbol{\beta}}_{i}$) of CSVR, LCR,
and CR. We omit SVR and NCCSVR here due to the incomparable dual variable
(i.e., $\hat{\boldsymbol{\beta}}_{i}$) (see smola2004tutorial,
*smola2004tutorial). As shown in Table 4, all values of $\beta$ for CSVR and
LCR lie roughly between -0.5 and 0.5, whereas those values for CR lie in a
larger interval, which can be a symptom of overfitting. We also note that
although both CSVR and LCR have small $\beta$, LCR obtains coefficients even a
bit smaller than CSVR. Furthermore, LCR produces a more flat function $f$, but
that does not mean a better performance automatically.

To evaluate the prediction accuracy, we divide the 506 samples randomly into
405 training samples and 101 test samples and then compute the in-sample and
out-of-sample prediction error in terms of MSE. The calculated in-sample and
out-of-sample MSEs and estimated standard deviations are demonstrated in Table
5. Overall, as expected, we observe that the CSVR method achieves the best
prediction performance. The results show that the restriction on the
subgradients or regularization described in Section 3 leads to a flatter
estimated function that can overcome overfitting. Although LCR is also
developed for solving overfitting problems, our approach still outperforms the
LCR method. A possible explanation is that owing to the structure of the SVR-
based approaches, CSVR can achieve a good bias-variance trade-off. Compared to
the non-regularized method (i.e., CR in Table 5), all methods benefit from
additional regularization in terms of the out-of-sample prediction accuracy.
However, note that CR yields the lowest in-sample MSE in this empirical data.

Table 4: Descriptive statistics for estimates of all explanatory variables. | CSVR | LCR | CR
---|---|---|---
Mean | Std.Dev. | Mean | Std.Dev. | Mean | Std.Dev.
$\hat{\beta}_{CRIM}$ | 0.08 | 0.17 | -0.01 | 0.08 | 2.66 | 10.47
$\hat{\beta}_{ZN}$ | 0.12 | 0.15 | 0.07 | 0.06 | 0.80 | 9.22
$\hat{\beta}_{INDUS}$ | -0.02 | 0.22 | -0.03 | 0.11 | 2.59 | 12.29
$\hat{\beta}_{NOX}$ | 0.00 | 0.04 | 0.00 | 0.02 | 147.61 | 963.24
$\hat{\beta}_{RM}$ | 0.19 | 0.34 | 0.05 | 0.10 | 11.10 | 64.63
$\hat{\beta}_{AGE}$ | -0.02 | 0.19 | -0.01 | 0.08 | -0.39 | 2.87
$\hat{\beta}_{DIS}$ | -0.17 | 0.34 | -0.04 | 0.10 | -0.60 | 48.50
$\hat{\beta}_{RAD}$ | 0.09 | 0.26 | 0.06 | 0.10 | -5.01 | 20.08
$\hat{\beta}_{TAX}$ | -0.01 | 0.05 | -0.01 | 0.02 | 0.09 | 0.85
$\hat{\beta}_{PTRATIO}$ | -0.18 | 0.33 | -0.06 | 0.11 | 1.89 | 42.85
$\hat{\beta}_{B}$ | -0.21 | 0.26 | -0.08 | 0.10 | -0.25 | 0.88
$\hat{\beta}_{LSTAT}$ | -0.42 | 0.29 | -0.28 | 0.12 | 0.24 | 7.68
$\hat{\alpha}$ | 132.61 | 103.68 | 69.44 | 39.31 | 276.42 | 1264.22
Table 5: In-sample and out-of-sample MSEs: Boston housing data.

MSE | CSVR | SVR | CR | LCR | NCCSVR
---|---|---|---|---|---
Out-of-sample MSE | 38.44(5.04) | 64.00(14.28) | 2334.17(933.04) | 43.72(6.47) | 42.72(9.28)
In-sample MSE | 21.43(0.86) | 62.57(2.90) | 1.41(0.33) | 35.05(1.52) | 40.17(2.05)

  * Note: standard deviation in parentheses.

###  5.2 NBER-CES manufacturing industry data

The data consist of 473 manufacturing industries (the 1997 6-digit NAICS
codes) and are collected from the NBER-CES Manufacturing Industry
Database.333NBER database: https://www.nber.org/research/data/nber-ces-
manufacturing-industry-database.  In this application, we apply the present
approaches to estimate the production function, where, as in
wang2012multivariate, the input includes the capital (INVEST), labor (PAY),
and raw materials (MATCOST), and the output is the value added (VADD).
Furthermore, the training and test set are 376 and 97 observations,
respectively.

CSVR’s in-sample and out-of-sample MSEs are $0.53\times 10^{7}$ and
$0.99\times 10^{7}$ (see Table 6). While the in-sample MSE value of CSVR is
not the lowest in comparison with other shape-constrained approaches, the out-
of-sample MSE remains the lowest. This is perhaps due to the fact that there
is a good trade-off between model performance and overfitting alleviation.

Table 6: In-sample and out-of-sample MSEs ($\times 10^{7}$): NBER-CES manufacturing industry data. MSE | CSVR | SVR | CR | LCR | NCCSVR
---|---|---|---|---|---
Out-of-sample MSE | 0.99 | 3.96 | 74.78 | 1.09 | 1.83
In-sample MSE | 0.53 | 3.94 | 0.37 | 0.91 | 1.01

It is worth noting that CR has the best in-sample fit in the applications but
not in the MC simulations. It is because the MSE is measured differently. In
the simulations, we measure the deviation between the estimated function $f$
and the true function $F$. In the applications, the true function $f$ is
unknown, and we thus measure the deviation of the predictions from the
observed $y$. This is not the same MSE because $y=f(x)+\epsilon$ also includes
the noise. CR will always minimize the MSE wrt y, but due to overfitting not
wrt $f(x)$.

In conclusion, both simulations and real-world applications demonstrate that
all regularized shape-constrained methods have a superior ability to control
overfitting, but CSVR would be more appealing than other regularized shape-
constrained methods because of its simplicity, capacity for univariate
regression, and robust performance.

##  6 Conclusions

Overfitting is a commonly seen phenomenon in nonparametric regression. To
mitigate the effects of overfitting, we have introduced a new approach called
convex support vector regression, which effectively combines the key elements
of support vector regression and convex regression. The paper investigates the
finite sample performance of the developed CSVR approach in contrast to other
state-of-the-art regression methods through Monte Carlo simulations.
Additional two real-world datasets are also used to test and compare the
performance of these approaches. We hope that the proposed approach can help
to further bridge the gaps between the data-driven estimation approaches known
in econometrics and statistics, machine learning, and operations research and
management science.

The evidence from the simulations indicates that CSVR performs at least as
well as LCR and much better than SVR and traditional convex regression. Two
real-world applications also show that our approach outperforms other state-
of-the-art regression methods. The regularized convex regression model can
help to alleviate the overfitting problem, also owing to its insensitive loss
function and robustness in the presence of outliers.

In this paper, we have restricted attention to regularizations known in the
literature, but there could be more efficient ways to restrict the domain of
subgradients (e.g., weight-restricted regression). Another promising future
research direction is developing the simple and fast algorithmic framework of
convex support vector regression. The main bottleneck of CSVR is that the full
problem (8) has $n(n-1)$ constraints and thus becomes computationally
inefficient for more than a few thousand observations. Furthermore, we have
deliberately kept away from statistical inferences, and further work in this
direction, e.g., exploring the asymptotic property of CSVR, would be needed.

## Acknowledgments

We acknowledge the computational resources provided by the Aalto Science-IT
project. Zhiqiang Liao gratefully acknowledges financial support from the
Foundation for Economic Education (Liikesivistysrahasto) [grant no. 210038].
Sheng Dai gratefully acknowledges financial support from the Foundation for
Economic Education (Liikesivistysrahasto) [grants no. 180019, 190073, 210075]
and the HSE Support Foundation [grant no. 11–2290].

## References

  * [1] [Alquier et al.]Alquier, Cottet Lecué2019alquier2019estimation Alquier, P., Cottet, V. Lecué, G. 2019, Estimation bounds and sharp oracle inequalities of regularized procedures with lipschitz loss functions, The Annals of Statistics 47, 2117–2144.
  * [2] Aybat Wang2014aybat2014a Aybat, N. S. Wang, Z. 2014, A parallel method for large scale convex regression problems, Proceedings of the IEEE Conference on Decision and Control, pp. 5710–5717.
  * [3] [Balázs et al.]Balázs, György Szepesvári2015balazs2015near Balázs, G., György, A. Szepesvári, C. 2015, Near-optimal max-affine estimators for convex regression, 18th Artificial Intelligence and Statistics, PMLR, pp. 38:56–64.
  * [4] Bradley Mangasarian1998bradley1998feature Bradley, P. S. Mangasarian, O. L. 1998, Feature selection via concave minimization and support vector machines., Proceedings of the 15th International Conference on Machine Learning, pp. 98:82–90.
  * [5] [Chen et al.]Chen, Lin Sen2020chen2020on Chen, X., Lin, Q. Sen, B. 2020, On degrees of freedom of projection estimators with applications to multivariate nonparametric regression, Journal of the American Statistical Association 115, 173–186.
  * [6] Cortes Vapnik1995cortes1995support Cortes, C. Vapnik, V. 1995, Support-vector networks, Machine Learning 20, 273–297.
  * [7] Dai2022Dai2022c Dai, S. 2022, Variable selection in convex quantile regression: L1-norm or L0-norm regularization?, European Journal of Operational Research .
  * [8] [Dai et al.]Dai, Fang, Lee Kuosmanen2021Dai2021b Dai, S., Fang, Y. H., Lee, C. Y. Kuosmanen, T. 2021, pyStoNED: A Python package for convex regression and frontier estimation, arXiv preprint arXiv:2109.12962.
  * [9] [Dai et al.]Dai, Kuosmanen Zhou2022Dai2022 Dai, S., Kuosmanen, T. Zhou, X. 2022, Non-crossing convex quantile regression, arXiv preprint arXiv:2204.01371.
  * [10] Ghosal Sen2017ghosal2017univariate Ghosal, P. Sen, B. 2017, On univariate convex regression, Sankhya A 79, 215–253.
  * [11] Guntuboyina Sen2018guntuboyina2018nonparametric Guntuboyina, A. Sen, B. 2018, Nonparametric shape-restricted regression, Statistical Science 33, 568–594.
  * [12] Hildreth1954hildreth1954point Hildreth, C. 1954, Point estimates of ordinates of concave functions, Journal of the American Statistical Association 49, 598–619.
  * [13] Johnson Jiang2018johnson2018shape Johnson, A. L. Jiang, D. R. 2018, Shape constraints in economics and operations research, Statistical Science 33, 527–546.
  * [14] Johnson Kuosmanen2012johnson2012one Johnson, A. L. Kuosmanen, T. 2012, One-stage and two-stage DEA estimation of the effects of contextual variables, European Journal of Operational Research 220, 559–570.
  * [15] Kuosmanen2008kuosmanen2008representation Kuosmanen, T. 2008, Representation theorem for convex nonparametric least squares, Econometrics Journal 11, 308–325.
  * [16] Kuosmanen Johnson2010kuosmanen2010data Kuosmanen, T. Johnson, A. L. 2010, Data envelopment analysis as nonparametric least-squares regression, Operations Research 58, 149–160.
  * [17] [Kuosmanen et al.]Kuosmanen, Johnson Saastamoinen2015kuosmanen2015stochastic Kuosmanen, T., Johnson, A. L. Saastamoinen, A. 2015, Stochastic nonparametric approach to efficiency analysis: A unified framework, in J. Zhu (ed.), Data Envelopment Analysis, Springer, Boston, MA, chapter 7, pp. 191–244.
  * [18] Kuosmanen Zhou2021kuosmanen2021shadow Kuosmanen, T. Zhou, X. 2021, Shadow prices and marginal abatement costs: Convex quantile regression approach, European Journal of Operational Research 289, 666–675.
  * [19] [Kuosmanen et al.]Kuosmanen, Zhou Dai2020kuosmanen2020how Kuosmanen, T., Zhou, X. Dai, S. 2020, How much climate policy has cost for oecd countries?, World Development 125, 104681.
  * [20] [Lee et al.]Lee, Johnson, Moreno-Centeno Kuosmanen2013lee2013a Lee, C. Y., Johnson, A. L., Moreno-Centeno, E. Kuosmanen, T. 2013, A more efficient algorithm for convex nonparametric least squares, European Journal of Operational Research 227, 391–400.
  * [21] Lim2014lim2014on Lim, E. 2014, On convergence rates of convex regression in multiple dimensions, INFORMS Journal on Computing 26, 616–628.
  * [22] [Lin et al.]Lin, Foster Ungar2011lin2011vif Lin, D., Foster, D. P. Ungar, L. H. 2011, VIF regression: A fast regression algorithm for large data, Journal of the American Statistical Association 106, 232–247.
  * [23] [Lin et al.]Lin, Sun Toh2022lin2022augmented Lin, M., Sun, D. Toh, K. C. 2022, An augmented Lagrangian method with constraint generation for shape-constrained convex regression problems, Mathematical Programming Computation 14, 223–270.
  * [24] [Mazumder et al.]Mazumder, Choudhury, Iyengar Sen2019mazumder2019a Mazumder, R., Choudhury, A., Iyengar, G. Sen, B. 2019, A computational framework for multivariate convex regression and its variants, Journal of the American Statistical Association 114, 318–331.
  * [25] Negahban Wainwright2011negahban2011simultaneous Negahban, S. N. Wainwright, M. J. 2011, Simultaneous support recovery in high dimensions: Benefits and perils of block $\ell_{1}/\ell_{\infty}-$regularization, IEEE Transactions on Information Theory 57, 3841–3863.
  * [26] Seijo Sen2011seijo2011nonparametric Seijo, E. Sen, B. 2011, Nonparametric least squares estimation of a multivariate convex regression function, The Annals of Statistics 39, 1633–1657.
  * [27] Smola Schölkopf2004smola2004tutorial Smola, A. J. Schölkopf, B. 2004, A tutorial on support vector regression, Statistics and computing 14, 199–222.
  * [28] Tibshirani1996tibshirani1996regression Tibshirani, R. 1996, Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society Series B 58, 267–288.
  * [29] [Valero-Carreras et al.]Valero-Carreras, Aparicio Guerrero2021valero-carreras2021support Valero-Carreras, D., Aparicio, J. Guerrero, N. M. 2021, Support vector frontiers: A new approach for estimating production functions through support vector machines, Omega 104, 102490.
  * [30] [Valero-Carreras et al.]Valero-Carreras, Aparicio Guerrero2022valero2022multi Valero-Carreras, D., Aparicio, J. Guerrero, N. M. 2022, Multi-output support vector frontiers, Computers & Operations Research 143, 105765.
  * [31] Vapnik1999vapnik1999the Vapnik, V. 1999, The nature of statistical learning theory, Springer, New York.
  * [32] Wang Ni2012wang2012multivariate Wang, Y. Ni, H. 2012, Multivariate convex support vector regression with semidefinite programming, Knowledge-Based Systems 30, 87–94.
  * [33] Wang Wang2013wang2013estimating Wang, Y. Wang, S. 2013, Estimating $\alpha$-frontier technical efficiency with shape-restricted kernel quantile regression, Neurocomputing 101, 243–251.
  * [34] [Wang et al.]Wang, Wang, Dang Ge2014wang2014nonparametric Wang, Y., Wang, S., Dang, C. Ge, W. 2014, Nonparametric quantile frontier estimation under shape restriction, European Journal of Operational Research 232, 671–678.
  * [35] [Xu et al.]Xu, Chen Lafferty2016xu2016faithful Xu, M., Chen, M. Lafferty, J. 2016, Faithful variable screening for high-dimensional convex regression, The Annals of Statistics 44, 2624–2660.
  * [36] [Yagi et al.]Yagi, Chen, Johnson Kuosmanen2020yagi2020shape Yagi, D., Chen, Y., Johnson, A. L. Kuosmanen, T. 2020, Shape-constrained kernel-weighted least squares: Estimating production functions for Chilean manufacturing industries, Journal of Business & Economic Statistics 38, 43–54.
  * [37] [Zhao et al.]Zhao, Rocha Yu2009zhao2009composite Zhao, P., Rocha, G. Yu, B. 2009, The composite absolute penalties family for grouped and hierarchical variable selection, The Annals of Statistics 37, 3468–3497.
  * [38] Zou Hastie2005zou2005regularization Zou, H. Hastie, T. 2005, Regularization and variable selection via the elastic net, Journal of the Royal Statistical Society Series B 67, 301–320.
  * [39]

## Appendix

Table A1: Descriptive statistics of all variables: the Boston housing data. Variable | Description | Mean | Std. Dev. | Min. | Max.
---|---|---|---|---|---
CRIM | per capita crime rate by town | 3.61 | 8.60 | 0.01 | 88.98
ZN | proportion of residential land zoned | 11.36 | 23.32 | 0.00 | 100.00
INDUS | proportion of non-retail business acres per town | 11.14 | 6.86 | 0.46 | 27.74
NOX | nitric oxides concentration | 0.55 | 0.11 | 0.36 | 0.87
RM | average number of rooms per dwelling | 6.28 | 0.70 | 3.56 | 8.78
AGE | proportion of owner built prior to 1940 | 68.57 | 28.15 | 2.90 | 100.00
DIS | weighted distances to city centres | 3.80 | 2.11 | 1.13 | 12.13
RAD | index of accessibility to radial highways | 9.55 | 8.71 | 1.00 | 24.00
TAX | full-value property-tax rate per $10,000 | 408.24 | 168.54 | 187.00 | 711.00
PTRATIO | pupil-teacher ratio by town | 18.46 | 2.16 | 12.60 | 22.00
B | black proportion of population | 356.67 | 91.29 | 0.32 | 396.90
LSTAT | proportion of population that is lower status | 12.65 | 7.14 | 1.73 | 37.97
MEDV | median value of owner-occupied homes | 22.53 | 9.20 | 5.00 | 50.00


# Super-Prompting: Utilizing Model-Independent Contextual Data
to Reduce Data Annotation Required in Visual Commonsense Tasks

Navid Rezaei and Marek Z. Reformat
University of Alberta
Edmonton, T6G 1H9, Canada
<EMAIL_ADDRESS>
###### Abstract

Pre-trained language models have shown excellent results in few-shot learning
scenarios using in-context learning. Although it is impressive, the size of
language models can be prohibitive to make them usable in on-device
applications, such as sensors or smartphones. With smaller language models,
task-specific data annotation is needed to fine-tune the language model for a
specific purpose. However, data annotation can have a substantial financial
and time burden for small research groups, startups, and even companies.

In this paper, we analyze different prompt-based fine-tuning techniques to
improve results on both language and multimodal causal transformer models. To
evaluate our results, we use a dataset focusing on visual commonsense
reasoning in time.

Our results show that by simple model-agnostic prompt-based fine-tuning,
comparable results can be reached by only using 35%-40% of the fine-tuning
training dataset. The proposed approaches result in significant time and
financial savings.

As the proposed methods make minimal architectural assumptions, other
researchers can use the results in their transformer models with minimal
adaptations. We plan to release the source code freely to make it easier for
the community to use and contribute to our work.

##  1 Introduction

Human annotation is time-consuming and is also a financial burden for research
groups, startups, and companies. To put it in context, almost $240,000 has
been spent on the annotation of the Visual Commonsense Reasoning in Time
(VisualCOMET) dataset and this figure only includes the payment to crowd-
workers from Amazon Mechanical Turk [19]. The real financial burden can be
much higher when including the time value of the staff involved in the
annotation process.

Although large pre-trained language models, such as GPT-3 transformer [2], are
impressive at multi-task few-shot learning, their huge size can be prohibitive
for different scenarios, including on-device applications. Fine-tuning still
plays an important role in achieving the state of the art, even with a
relatively smaller model. As an example, the two current leading
models111https://super.gluebenchmark.com/leaderboard (better than human
baseline) on SuperGLUE task [28] are fine-tuned variants of T5 [21] and
DeBERTa [9] language models, while GPT-3 is at 14th place.

Our goal is to devise a model-independent process that could improve results
based on fine-tuning with much less annotated training data.

##  2 Related Work

Several recent works have focused on improving fine-tuning methods in language
models, such as [12], [5], [14], and [30]. The focus has been put mostly on
optimization and regularization, but not on using less data for fine-tuning.
The results from those studies are complementary to our work.

Some previous efforts have been put on prompt-based fine-tuning to improve
classification or regression tasks in natural language processing (NLP). [24]
and [25] convert textual inputs into cloze-style questions with a task
description. [6] studies smaller language models for few-shot learning
capability by using automatically-generated prompts for fine-tuning and by
incorporating demonstrations into context.

On another topic, a group of recent research studies, including [11], [20] and
[16], aim at task-dependent added parameters to adapt models to different
tasks. This way, one does not need to re-train a complete model to fine-tune
it to a specific task but only needs to re-train a fraction of parameters.

There is a recent body of work that utilizes inherent knowledge of language
models combined with fine-tuning on specialized large-scale training datasets
to infer different commonsense and causal scenarios. [1] uses generative
language models to expand on ATOMIC [23] and ConceptNet [26] commonsense
knowledge graphs. [13] introduces an updated knowledge graph similar to ATOMIC
and uses BART [15] encoder-decoder model to generate new knowledge. [17] uses
generative language models to expand on an introduced knowledge base of causal
mini-story explanations.

Given the success of prompt-based fine-tuning and in-context learning in
classification and regression tasks, we are motivated to assess similar
principles in the context of commonsense generation using generative language
models, which are fine-tuned on a commonsense knowledge graph.

##  3 Dataset

For this paper, we have selected a multi-modal commonsense knowledge graph for
fine-tuning. The Visual Commonsense Reasoning in Time (VisualCOMET) dataset
[19] consists of $1.4$ million commonsense inferences over 59,356 images and
139,377 specific events at present. The dataset has human-annotated inferences
regarding three different aspects: the intention of the person mentioned, the
possible events that could happen next, and the possible preceding events. The
inferences are made based on a single image. The annotators have access to
short clips before and after the event, which are not part of the dataset.
Each image is also annotated with event and place descriptions. There is a
total amount of 1,465,704 commonsense inferences.

The images are sourced from the VCR dataset [29]. The images usually have a
complex visual scene with multiple people and activities present. This dataset
includes automatically-detected object bounding boxes and people are annotated
with numerical tags.

##  4 Method

In this work, we focus on using generative language models and analyze how
prompt-based fine-tuning and in-context learning could help to reduce the size
of the data required for fine-tuning training.

As seen in Fig. 1, there are several scenarios where extra context could help
lead the generative language model to a correct answer, but lack of correct
understanding about the scene and the event text can result in incorrect
results. Extra human annotations, focused on these shortcomings, could improve
the results, but that comes with extra time and money expenditure.

We propose using the underutilized context already present in text and image,
then transforming them to a form that is usable by most transformer models,
which is a sequence. We analyze if this kind of addition helps the language
model achieve better results in the case of limited annotated data available.

Assuming the added context text is represented with $c$ and its tokenized
version with $\\{c\\}$, we can represent the context with
$\\{c\\}=\\{w_{1}^{c},w_{2}^{c},...w_{q}^{c}\\}$, where $w^{c}_{i}$ represents
each token created from tokenization of the context $c$. This context is
merged with tokenized versions of event and place, which are represented as:
$\\{e\\}=\\{w_{1}^{e},w_{2}^{e},...w_{n}^{e}\\}$ and
$\\{p\\}=\\{w_{1}^{p},w_{2}^{p},...w_{m}^{p}\\}$, respectively. Using the
merged versions of event and place texts with context, the updated sequence-
to-sequence loss can be written as:

$\displaystyle\mathcal{L}=$ $\displaystyle-\sum_{i=1}^{n}{\log
P(w_{i}^{e}|w_{<i}^{e},v))}-\sum_{i=1}^{m}{\log P(w_{i}^{p}|w_{<i}^{p},e,v))}$
$\displaystyle-$ $\displaystyle\sum_{i=1}^{q}{\log
P(w_{i}^{c}|w_{<i}^{c},p,e,v))}$ $\displaystyle-$
$\displaystyle\sum_{i=1}^{l}{\log P(w_{hi}^{r}|w_{h<i}^{r},c,p,e,v)}$ (1)

where $v$ represents visual features, including overall images and person-
specific boxes, $r$ represents inference prompts, which could be intent,
before and after, and $w_{<i}^{*}$ represents past tokens for each case.

(a) Event: Person-4 is sitting on the couch with her legs over the arm.
Place: In a living room
Annotated Intent Inferences:
1) be comfortable
2) get cozy
A Predicted Intent: Show boredom
Missing context: Happy facial expression

(b) Event: Person-2 is taking a tiny spoon and scooping up a heap of caviar.
Place: In a dining room
Annotated Intent Inferences:
1) live luxuriously
2) enjoy a delicious treat
A Predicted Intent: keep everything neat
Missing context: Caviar is a luxury edible.

(c) Event: Person-1 is sitting down staring at someone angrily.
Place: In a dining room
Annotated After Inferences:
1) choose not to express her anger verbally
2) look down in front of her
3) slam the table
4) walk out of the restaurant
A Predicted After: Fight
Missing context: Objects in a dining room and its location.

Figure 1: Predictions based on the fine-tuned language model introduced in
[19]. Each example shows a piece of missing contextual information that could
be utilized.

##  5 Experiments

The goal of the experiments is to see how much we could reduce the annotated
data and still achieve results comparable to a case where the full human-
annotated data is used. We tried different contextual data, which did not
require extra annotations, such as captions, facial expressions, and related
concepts.

As shown in Fig. 1, we can intuitively see that some extra context could
potentially help the language model to reach a more logical deduction of
intention, past, and future events.

For each scenario, the VisualCOMET dataset provides several human annotations
for comparison, each showing intent of a person, what could happen next, and
what happened before. The experiments are evaluated using BLEU [18], METEOR
[4] and CIDEr [27] automatic metrics to compare the generated texts for
different scenarios of before, intent and after with the human-annotated
texts.

We tried two different methods of adding relevant concepts. One method is
based on converting relevant concept graphs into a readable sentence and the
other method is based on only prepending concept words to the target sentence.
In either method, the text is scanned for concepts, and the related concepts
are extracted based on a commonsense knowledge graph such as in [26, 22].
Although sentence-based inputs perform well, they require a longer input width
that may not be available given the language model. To sort relevant concepts,
crowd-based scores or frequency scores are used based on the specific
knowledge graph used. These triples are then converted to text with some hand-
designed rules. An example of this process is shown in Fig. 2(a). Table 1
shows three of the top-performing models with added conceptual contexts. They
are compared with the original data, which does not have any added context.
Evaluation is done on a validation dataset with a size of a hundred. Concept
words added in this specific scenario are connected via HasProperty and PartOf
predicates. Concept sentences use the HasProperty predicate. Adding similar
information during inference time does not result in much improvement in this
specific case. More comparisons can be found in the Appendix.

Method | BLEU-2 | METEOR | CIDEr
---|---|---|---
GPT-2 [19] | 13.81 | 10.85 | 15.37
Concept Word (NVP) | 17.25 | 12.17 | 19.79
Concept Word (VP) | 17.17 | 12.28 | 19.34
Concept Sent. (NVP) | 14.92 | 11.25 | 16.9
Table 1: Effects of adding relevant concepts. Results are shown at the fourth
epoch using almost 25,000 (22%) of the available annotated data. NVP: No
Validation Prompt. VP: Validation Prompt.

As seen in Fig. 1(a), lack of the model’s attention to some visual cues, such
as facial expressions, could also result in errors of judgment. To fix this
issue, we trained a ResNet [8] model on FER2013 [7] dataset with almost $70\%$
accuracy. The dataset consists of human face images and emotion labels of
angry, disgust, fear, happy, neutral, sad, and surprise. Only the emotion of
people mentioned in the event text is processed. The results are then
prepended to the event text. An example of this process is shown in Fig. 2(b).
Table 2 shows effects of adding facial expressions as a context in the final
performance of the model. Contrary to relevant concepts, adding facial
expressions during inference time improves the results. Evaluation is done on
a validation data size of a hundred.

Method | BLEU-2 | METEOR | CIDEr
---|---|---|---
GPT-2 [19] | 13.81 | 10.85 | 15.37
FE (NVP) | 14.45 | 11.27 | 15.7
FE (VP) | 15.11 | 11.23 | 16.03
Table 2: Effects of adding information about facial expressions. Results are
shown at the fourth epoch using almost 25,000 (22%) of the available annotated
data. NVP: No Validation Prompt. VP: Validation Prompt. FE: Facial
Expressions.

Another type of automatically-generated context that we experimented with is
image captioning. The idea is that some of the image dynamics may have been
missed, even though image features are fed into the GPT-2 model. Adding
generated captions proves to be effective as shown in Table 3. Meshed-Memory
transformer model [3] with beam search decoding is used for image captioning.
The process of adding these captions is illustrated in Fig. 2(c).

Method | BLEU-2 | METEOR | CIDEr
---|---|---|---
GPT-2 [19] | 13.81 | 10.85 | 15.37
Caption (NVP) | 14.08 | 10.78 | 15.63
Caption (VP) | 16.49 | 11.85 | 18.8
Table 3: Effects of adding image captions. Results are shown at the fourth
epoch using almost 25,000 (22%) of the available annotated data. NVP: No
Validation Prompt. VP: Validation Prompt.

A mixture of different contextual information is shown to be more effective
than individual ones. A combination of concept words, image captions, and
facial expressions of relevant individuals in the image achieve the best
result compared to other experiments. As seen in Table 4, this combination can
achieve comparable results to full-data finetuning by only using 35%-40% of
the annotated data. This results in less human time spent doing annotations
and can potentially reduce costs and completion times of projects. Results of
other experiments are included in the Appendix.

Method | Inference Data | Data Size | BLEU-2 | METEOR | CIDEr
---|---|---|---|---|---
GPT-2 [19] | N/A | 111,796 (100%) | 18.05 | 13.21 | 22.72
CW + C + FE | C + CW + FE | 39,000 ( 35%) | 18.38 | 12.97 | 22.65
CW + C + FE | C + CW + FE | 45,000 ( 40%) | 18.58 | 13.01 | 22.97
Table 4: Analyzing the effect of combining multiple contextual data. All
models are finetuned for five epochs. Contexts are added based on the order
shown. CW: Concept Words. C: Captions. FE: Facial Expressions.

To reduce the effects of other variables in these experiments, we have limited
ourselves to only train the final models for five epochs. The decoding method
and hyperparameters are also kept constant throughout the experiments. We use
nucleus sampling [10] with $p=0.9$ to generate five sentences for each
scenario of intent, before and after. The finetuning was run on two NVIDIA RTX
GPUs with 24 GB memory each. For the case with all concept words, captions,
and facial expression contexts, the fine-tuning time is around 1.5 hours per
epoch while using mixed precision.

(a) Process of adding relevant concepts.

(b) Process of adding facial expressions.

(c) Process of adding image captions.

Figure 2: The process of extracting and adding prompts shown through examples.

##  6 Conclusion

In this work, we analyzed the effects of automatically-generated contexts in
multimodal transformer models used in a commonsensical task. These prompts can
help us reduce the human annotation needed in the task by as much as 60%-65%
and still, achieve comparable results to when the whole human-annotated
dataset is used. These findings result in time and cost savings for future
multimodal data annotation projects.

As future work, it is interesting to find a lower bound for data annotation
reduction without affecting the final result of a model. It is also useful to
find a method to automatically find and apply the best contextual data for
different tasks and models.

## References

  * [1] Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi.  COMET: Commonsense transformers for automatic knowledge graph construction.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * [3] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara.  Meshed-Memory Transformer for Image Captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * [4] Michael Denkowski and Alon Lavie.  Meteor universal: Language specific translation evaluation for any target language.  In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics.
  * [5] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  ArXiv, abs/2002.06305, 2020.
  * [6] Tianyu Gao, Adam Fisch, and Danqi Chen.  Making pre-trained language models better few-shot learners, 2020.
  * [7] Ian J. Goodfellow, D. Erhan, Pierre Luc Carrier, Aaron C. Courville, M. Mirza, Benjamin Hamner, William Cukierski, Y. Tang, D. Thaler, Dong-Hyun Lee, Yingbo Zhou, Chetan Ramaiah, Fangxiang Feng, Ruifan Li, X. Wang, Dimitris Athanasakis, J. Shawe-Taylor, Maxim Milakov, John Park, Radu Tudor Ionescu, M. Popescu, C. Grozea, J. Bergstra, Jingjing Xie, Lukasz Romaszko, Bing Xu, Chuang Zhang, and Yoshua Bengio.  Challenges in representation learning: A report on three machine learning contests.  Neural networks : the official journal of the International Neural Network Society, 64:59–63, 2015.
  * [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016.
  * [9] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  DeBERTa: Decoding-enhanced BERT with disentangled attention, 2020\.
  * [10] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration.  arXiv preprint arXiv:1904.09751, 2019.
  * [11] N. Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and S. Gelly.  Parameter-efficient transfer learning for nlp.  In ICML, 2019.
  * [12] J. Howard and Sebastian Ruder.  Universal language model fine-tuning for text classification.  In ACL, 2018.
  * [13] Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi.  COMET-ATOMIC 2020: On symbolic and neural commonsense knowledge graphs.  ArXiv, abs/2010.05953, 2020.
  * [14] Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang.  Mixout: Effective regularization to finetune large-scale pretrained language models.  ArXiv, abs/1909.11299, 2020.
  * [15] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online, July 2020. Association for Computational Linguistics.
  * [16] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation, 2021.
  * [17] Nasrin Mostafazadeh, Aditya Kalyanpur, Lori Moon, David Buchanan, Lauren Berkowitz, Or Biran, and Jennifer Chu-Carroll.  GLUCOSE: GeneraLized and COntextualized story explanations.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4569–4586, Online, Nov. 2020. Association for Computational Linguistics.
  * [18] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
  * [19] J. Park, Chandra Bhagavatula, R. Mottaghi, Ali Farhadi, and Yejin Choi.  VisualCOMET: Reasoning about the dynamic context of a still image.  In ECCV, 2020.
  * [20] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych.  Adapterhub: A framework for adapting transformers.  In EMNLP, 2020.
  * [21] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research, 21(140):1–67, 2020.
  * [22] Navid Rezaei, Marek Z Reformat, and Ronald R Yager.  Image-based world-perceiving knowledge graph (wpkg) with imprecision.  In International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, pages 415–428. Springer, 2020.
  * [23] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi.  ATOMIC: An atlas of machine commonsense for if-then reasoning.  ArXiv, abs/1811.00146, 2019.
  * [24] Timo Schick and Hinrich Schütze.  Exploiting cloze questions for few shot text classification and natural language inference, 2020.
  * [25] Timo Schick and Hinrich Schütze.  It’s not just size that matters: Small language models are also few-shot learners, 2020.
  * [26] Robyn Speer, J. Chin, and Catherine Havasi.  ConceptNet 5.5: An open multilingual graph of general knowledge.  ArXiv, abs/1612.03975, 2017.
  * [27] R. Vedantam, C. L. Zitnick, and D. Parikh.  Cider: Consensus-based image description evaluation.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, 2015.
  * [28] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  In Advances in neural information processing systems, pages 3266–3280, 2019.
  * [29] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6713–6724, 2019.
  * [30] Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi.  Revisiting few-sample bert fine-tuning.  ArXiv, abs/2006.05987, 2020.

##  Appendix A Appendix

Experimentation results from using different types of training and inference
prompts are included in this appendix. The model used in the experiments is
GPT-2 as described in [19].

The best types of prompts are chosen to be combined. The experiments show that
the order in which prompts are added can affect the final results.

The vision-based inference prompts seem to better affect the final metric
results when compared to the text-based inference prompts. This could be due
to the lack of enough visual attention paid during the decoding process.
Future work could involve developing a multimodal model that makes better use
of visual contexts not only during the training phase, but also the inference
time.

The quality of the annotated data can have an impact on the training model. We
do not hand-select the annotated data based on quality and this may result in
variability in final results when training with different data sizes. It can
be a good practice to assess the quality of the annotated data and prompts
based on the final goal of the model.

Training Prompt | Inference Prompt | Training Data Size | BLEU-2 | METEOR | CIDEr
---|---|---|---|---|---
None | None | 111,796 (100%) | 17.94 | 13.14 | 22.71
None | None | 25,000 ( 22%) | 13.81 | 10.85 | 15.37
CS (AtLocation) | None | 25,000 ( 22%) | 12.26 | 10.42 | 15.09
CS (AtLocation) + Place | None | 25,000 ( 22%) | 14.3 | 11.08 | 14.66
CS (CapableOf) | None | 25,000 ( 22%) | 12.65 | 10.6 | 15.9
CS (CapableOf) + Place | None | 25,000 ( 22%) | 14.78 | 11.16 | 15.1
CS (HasA) | None | 25,000 ( 22%) | 12.73 | 10.65 | 15.83
CS (HasA) + Place | None | 25,000 ( 22%) | 14.58 | 11.15 | 15
CS (HasProperty) | None | 25,000 ( 22%) | 12.25 | 10.5 | 15.52
CS (HasProperty) + Place | None | 25,000 ( 22%) | 15.25 | 11.38 | 16.3
CS (IsA) | None | 25,000 ( 22%) | 12.73 | 10.41 | 15.73
CS (IsA) + Place | None | 25,000 ( 22%) | 14.04 | 11.03 | 14.32
CS (PartOf) | None | 25,000 ( 22%) | 12.65 | 10.51 | 15.71
CS (PartOf) + Place | None | 25,000 ( 22%) | 14.26 | 11.19 | 14.64
FE | None | 25,000 ( 22%) | 14.45 | 11.27 | 15.7
FE | FE | 25,000 ( 22%) | 15.11 | 11.23 | 16.03
CW (PartOf + HasProperty) | None | 25,000 ( 22%) | 17.25 | 12.17 | 19.79
CW (PartOf + HasProperty) | CW (PartOf + HasProperty) | 25,000 ( 22%) | 17.17 | 12.28 | 19.34
C | None | 25,000 ( 22%) | 14.08 | 10.78 | 15.63
C | C | 25,000 ( 22%) | 16.49 | 11.85 | 18.8
C + FE | C + FE | 25,000 ( 22%) | 16.75 | 12.19 | 19.16
CW + C + FE | None | 25,000 ( 22%) | 12.6 | 10.18 | 14.57
CW + C + FE | CW + C + FE | 25,000 ( 22%) | 17.4 | 11.97 | 20.03
CW + C + FE | CW + FE | 39,000 ( 35%) | 16.57 | 12.32 | 19.68
CW + C + FE | C + FE | 39,000 ( 35%) | 16.71 | 12.27 | 19.01
CW + C + FE | CW + C + FE | 39,000 ( 35%) | 16.75 | 12.4 | 19.86
CW + C + FE | CW + C + FE + Syns | 39,000 ( 35%) | 16.7 | 12.43 | 19.94
CW + C + FE + PCW | CW + C + FE + PCW | 39,000 ( 35%) | 17.74 | 12.73 | 20.52
CW + C + FE | C + CW + FE | 39,000 ( 35%) | 17.34 | 12.45 | 20.11
CW + C + FE | CW + C + FE | 45,000 ( 40%) | 17.46 | 12.84 | 21.56
CW + C + FE | C + CW + FE | 45,000 ( 40%) | 17.66 | 12.95 | 21.41
Table 5: Experimentation results of using different training and inference
prompts. The model used is GPT-2 [19]. Results are shown at epoch four and
evaluated on validation data of size 100. Prompts are added based on the order
shown. CW: Concept Words. C: Captions. FE: Facial Expressions. CS: Concept
Sentences. Syns: Synonyms. PCW: Place Concept Words.