11institutetext: Nanyang Technological University, Singapore 22institutetext:
Sensetime Research 33institutetext: Monash University
33email: {daxuan001, asjmzheng, E180176<EMAIL_ADDRESS><EMAIL_ADDRESS>
# ExtrudeNet: Unsupervised Inverse Sketch-and-Extrude for Shape Parsing

Daxuan Ren 1122 Jianmin Zheng✉ 11 Jianfei Cai 1133
Jiatong Li 1122 Junzhe Zhang 1122

###### Abstract

Sketch-and-extrude is a common and intuitive modeling process in computer
aided design. This paper studies the problem of learning the shape given in
the form of point clouds by “inverse” sketch-and-extrude. We present
ExtrudeNet, an unsupervised end-to-end network for discovering sketch and
extrude from point clouds. Behind ExtrudeNet are two new technical components:
1) an effective representation for sketch and extrude, which can model
extrusion with freeform sketches and conventional cylinder and box primitives
as well; and 2) a numerical method for computing the signed distance field
which is used in the network learning. This is the first attempt that uses
machine learning to reverse engineer the sketch-and-extrude modeling process
of a shape in an unsupervised fashion. ExtrudeNet not only outputs a compact,
editable and interpretable representation of the shape that can be seamlessly
integrated into modern CAD software, but also aligns with the standard CAD
modeling process facilitating various editing applications, which
distinguishes our work from existing shape parsing research. Code is released
at https://github.com/kimren227/ExtrudeNet.

##  1 Introduction

Pen draws a line, paint roller sweeps a surface, and pasta maker extrudes
Fusilli from a stencil. From a point to a line, then to a surface and to a
solid shape, the process of using lower dimensional shapes to construct a
higher dimensional object seems to be a human instinct. In this paper, we
explore the inverse of this process by training a neural network to infer 2D
drawings of a point cloud and then extrude them into 3D to reconstruct the
shape.

With recent development in 3D reconstruction technologies and cheaper sensors,
point clouds can be easily obtained and become a widely adopted 3D data
representation [23, 24, 35]. However, the unordered and unstructured nature of
point clouds makes it difficult to perform high level manipulation and easy
editing of their underlying geometries. Thus in recent years, the research of
extracting shape features and generating high level shape representation from
point clouds is very active, especially in computer vision and graphics.

Figure 1: ExtrudeNet studies the problem of learning the shape, given in the
form of point clouds, by “inverse” sketch-and-extrude.

We take inspiration from the process of sketch and extrude, a popular and
intuitive approach widely used in the field of computer aided design (CAD)
where engineers usually model shapes by first “sketching” a closed free form
sketch (profile) in a 2D sketch plane and then “extruding” the sketch into 3D.
We propose ExtrudeNet, the first of its kind end-to-end unsupervised network
for learning high level (editable and interpretable) shape representation
through inverse sketch and extrude process from point clouds.

To realize ExtrudeNet, as shown in Fig. 1, we create three modular components:
1) rBézierSketch, which generates a simple closed curve (i.e. no self-
intersection); 2) Sketch2SDF, a versatile numerical method for computing
Signed-Distance-Field from parametric curves; and 3) DiffExtruder, a
differentiable method for extruding 2D Signed-Distance-Field (SDF) into a 3D
solid shape. Built upon these components, ExtrudeNet takes a point cloud as
input and outputs sketch and extrude parameters which form a compact,
interpretable and editable shape representation. ExtrudeNet’s outputs are
highly compatible with modern CAD software [3], allowing control points based
editing, which is much easier compared to directly editing triangle, polygonal
meshes or even primitive based constructive solid geometry (CSG) models [26].

There are prior works on converting point clouds to high level shape
representations. These representatives are discovering CSG in either a
supervised or unsupervised manner. Supervised approaches [32, 36, 11, 19]
suffer issues such as invalid syntax, infeasible models, and requiring large
amount of expert annotated data. Unsupervised methods [21, 33, 8, 9, 25, 37]
find the Boolean combinations of pre-defined geometric primitives such as box
and cylinder. Our ExtrudeNet goes beyond these works. First, 2D sketch can be
complex freeform curves, which allows us to model much more complex shapes
using a single extrusion. Second, “Sketch-and-Extrude” is more user-friendly
when it comes to editing and secondary-development, as editing a 2D sketch is
more intuitive than editing 3D parameters. This Sketch-and-Extrude process
happens to be a widely adopted method in CAD software for modeling 3D shapes
[3, 2, 15, 1], making our method highly compatible with industry standards.
Moreover, extensive experiments show that our ExtrudeNet can reconstruct
highly interpretable and editable representations from point clouds. We also
show through qualitative visualizations and quantitative evaluation metrics
that ExtrudeNet outputs better overall results. The main contributions of the
paper are:

  * •

We present an end-to-end network, ExtrudeNet, for unsupervised inverse sketch
and extrude for shape parsing. To the best of our knowledge, ExtrudeNet is the
first unsupervised learning method for discovering sketch and extrude from
point clouds.

  * •

We design a special rational cubic Bézier curve based representation for
sketch and extrusion learning, which can model freeform extrusion shapes, and
the common cylinder and box primitives as well.

  * •

We present a simple and general numerical method for computing the signed
distance field of 2D parametric curves and their 3D extrusions which is proven
to be suitable for gradient-based learning.

##  2 Related Work

Shape Representation. There have been different representations for 3D shapes.
Recently implicit representation [18, 20, 12], usually in the form of
Occupancy or Signed Distance Field, has drawn a lot of attention. It frees
from intricate data representation and can be used directly via neural
networks. To extract the underlying geometry, however, further processing is
required [17], which is computationally intensive. Parametric representation
describes shapes by parametric equations and is widely used in industry for
modeling shapes thanks to its strong edibility and infinite resolution.
However, generating parametric shapes from raw data like point clouds is a
non-trivial task.

High Level Shape Learning. High level shape representations are often
required, which benefit various practical applications. With the advance in
machine learning, learning high level shape representations from raw data
structure gains popularity. There have been many works for reconstructing CAD
and especially CSG from point clouds. CSG is a tree-like structure
representing shapes by starting from primitive objects and iteratively
combining geometric shapes through Boolean operations [16]. It is well adapted
in professional CAD software.

CSGNet [32] pioneers supervised CSG learning by modeling CSG as a sequence of
tokens. It processes the sequence into a valid CSG-Tree using NLP techniques.
With recent NLP technologies, DeepCAD [36] and CAD-As-Language [11] employ
more powerful language models, e.g. Transformer, and add additional
constraints to better predict CAD models. However, modeling CAD as language
gives rise to addition problems (such as producing grammatically correct but
invalid representations), which are not easy to solve.

VP [33] pioneers the unsupervised approaches by using the union of a set of
boxes to approximate shapes. SQ [21] takes a step further by using super
quadrics instead of boxes to better approximate complex shapes. BSPNet [8] and
CVXNet [9] propose to use the union of a set of convexes to represent a
complex shape, where the convexes are constructed by intersecting half-spaces.
These methods extend the modeling capability of using standard primitives, but
abundant planes are required to approximate freeform surfaces, which also
limits their edibility. UCSGNet [13] proposes CSG-Layers that iteratively
selects primitives and Boolean operations for reconstruction. CSG-StumpNet
[25] reformulates CSG-Tree with arbitrary depth into a fixed three layer
structure similar to Disjunctive Normal Form, and uses a simple network to
generate binary matrices for select fundamental products to union into the
final reconstruction. However, these methods focuses on CSG operations and use
only basic primitives (boxes, spheres, etc.), which is not efficient to
approximate complicated shapes. CAPRI-Net [37] uses quadric implicit shapes to
construct two intermediate shapes using an approach similar to BSP-Net. The
two intermediate shapes are then subtracted to form the final shape. The
construction process is interpretable, but the use of quadric implicit shapes
reduces its edibility.

Figure 2: Framework overview. The input point cloud first goes through the
encoder-decoder phase to predict shape parameters and connection matrices via
shape head and connection head. rBézierSketch is then used to generate the
profile curve from the decoded sketch parameters. Sketch2SDF computes the SDF
of the sketches. Sketch SDF, sketch plane parameters, and extrusion height are
passed into DiffExtruder to generate the occupancy of the extrusion shapes,
which together with the connection matrices are passed into CSG-Stump to
generate the final shape occupancy.

##  3 ExtrudeNet

This section presents ExtrudeNet, an end-to-end network for unsupervised
inverse sketch and extrude for compact and editable shape parsing. The input
to the network is a point cloud representing a shape to be learned. ExtrudeNet
outputs a set of 3D extrusions as the building blocks and their Boolean
operations, which together create the shape. Each of the 3D extrusions is
defined by a 2D sketch profile curve and a 3D extrusion process.

The pipeline of the entire network is illustrated in Fig. 2. The main
components are briefly described below. Note that different encoder and
assembly methods can be used to adapt for different use cases.

  1. (1)

Encoder-Decoder: ExtrudeNet first encodes the input point cloud into a latent
feature using the off-the-shelf DGCNN as a backbone encoder [35]. The latent
code is then enhanced by three fully connected layers with size 512, 1024, and
2048. After that, the latent feature is passed to the shape head and the
connection head to decode into extruded shape parameters and connection
matrices. Extruded shape parameters consist of 2D sketch parameters, sketch
plane parameters, and extrusion height. Connection matrices represent Boolean
operations among the extruded shapes. Since binary value is not
differentiable, we use the Sigmoid function to predict a soft connection
weight in $[0,1]$ for each matrix.

  2. (2)

rBézierSketch: rBézierSketch is used to convert sketch parameters into a
closed profile curve defined by a set of rational cubic Bézier curves for
extrusion. The generated 2D sketch curve is guaranteed to be free of self-
intersection and can represent free-form curves, circular arc and even polygon
in a single formulation.

  3. (3)

SDF-Generation: This consists of Sketch2SDF and DiffExtruder. Sketch2SDF is
first used to compute the Signed-Distance-Field (SDF) of the generated 2D
sketch on a plane. The computed 2D sketch SDF, the sketch plane parameters,
and extrusion height are then passed into DiffExtruder to compute the extruded
shapes’ occupancy field in 3D space.

  4. (4)

Assembly: Given the predicted extrusion shapes’ occupancy and connection
matrices, we are in a position to assemble the extrusion shapes to complete
the final reconstruction. For this purpose, we choose to directly use CSG-
Stump from [25] for its simplicity and learning friendly nature. CSG-Stump
reformulates CSG-Trees into a three layer structure similar to Disjunctive
Normal Form and use three fixed size binary matrices to generate and select
fundamental products. The first layer of CSG-Stump is a complement layer
indicating whether the complement of the input occupancy should be used for
the down steam operations. The second layer is an intersection layer which
selects and intersects complement layer’s outputs into intersected shapes. The
last layer selects intersected shapes to union into the final shape.

Below we describe rBézierSketch and SDF-Generation in more detail.

###  3.1 rBézierSketch and Extrusion

To create an extrusion shape, a profile curve should be sketched and then
extruded in 3D space. We use a network to predict the sketch parameters that
define the profile curve in the $XY-$plane (serving as its local coordinate
system) and then extrude it along the $Z-$direction to create the extrusion
shape. The shape is then transformed to the required location and orientation
predicted by the network which mimics the sketch plane transform in CAD
software.

####  3.1.1 rBézierSketch.

Considering the common modeling practice and shapes in CAD applications, there
are a few assumptions for the profile curve: (i) it is closed and has no self-
intersection, which has advantages in defining a valid solid shape; (ii) it is
piecewise smooth for creating quality shapes; and (iii) it can model freeform
curves, and circles or polygons as well. Meanwhile, we also have to balance
the capability and complexity of the representation such that it can be easily
deployed into a learning pipeline. Based on these considerations, we propose
the following model for our profile curve.

The basic mathematical model is a closed curve formed by $N$ curve segments
defined by special rational cubic Bézier curves
$C_{k}(t),t\in[0,1],k=0,1,\cdots,N-1$, which may explain the name
rBézierSketch. The equation of $C_{k}(t)$ is:

$C_{k}(t)=\frac{P^{k}_{0}B_{0}^{3}(t)+w^{k}_{1}P^{k}_{1}B_{1}^{3}(t)+w^{k}_{2}P^{k}_{2}B_{2}^{3}(t)+P^{k}_{3}B_{3}^{3}(t)}{B_{0}^{3}(t)+w^{k}_{1}B_{1}^{3}(t)+w^{k}_{2}B_{2}^{3}(t)+B_{3}^{3}(t)}$
(1)

where $P^{k}_{i}=(x_{i}^{k},y_{i}^{k})$ are the control points on the
$XY-$plane, weights $w_{1}^{k}\geq 0,w_{2}^{k}\geq 0$ are for the two inner
control points, and $B_{i}^{3}(t)={3\choose i}(1-t)^{3-i}t^{i}$ are Bernstein
polynomials. To make sure that consecutive segments are connected to form a
closed curve, the constraints $P^{k}_{3}=P^{(k+1)\text{mod}N}_{0}$ are added.

The closed curve is defined around the origin. Thus it is convenient to
express each control points
$P^{k}_{i}=(x^{k}_{i},y^{k}_{i})=(\rho_{i}^{k}\cos(\alpha_{i}^{k}),\rho^{k}_{i}\sin(\alpha^{k}_{i}))$
by the radial coordinate $\rho^{k}_{i}$ and the polar angle $\alpha_{i}^{k}$.
For simplicity, we further distribute the central angles of the segments
evenly. That is, each Bézier curve has the central angle $\frac{2\pi}{N}$.
Within each segment $C_{k}(t)$, the polar angles of control points are chosen
to be:

$\alpha^{k}_{1}=\alpha^{k}_{0}+\theta,\;\;\alpha^{k}_{3}=\alpha^{k}_{0}+\frac{2\pi}{N},\;\;\alpha^{k}_{2}=\alpha^{k}_{3}-\theta$
(2)

where
$\displaystyle\theta=\frac{2\pi}{4N}+\tan^{-1}\left(\frac{1}{3}\tan\left(\frac{2\pi}{4N}\right)\right)$.
It is worth pointing out that these polar angles are specially designed to
achieve the capability of circle recovery (see Proposition 2 below). In this
way, to specify the Bézier control points, we just need to provide the radial
coordinates. Connecting all the control points in order forms a polygon that
is homeomorphic to the origin-centered unit circle, which assures good
behavior of the generated profile curve. In summary, the network only has to
estimate the radial coordinates $\rho^{k}_{i}$ and the weights
$w^{k}_{1},w^{k}_{2}$ in order to sketch the profile curve.

Remark 1. The proposed curve model is specially designed to deliver a few nice
properties, which are outlined in the following paragraphs.

Rational cubic Bézier representation in (1) is proposed because it is a simple
form of NURBS that is the industry standard in CAD and meanwhile sufficient to
model freeform curves [10]. Besides freeform smooth shapes, this
representation is able to represent a straight line or a polygon, for example,
as long as we let all the control points $P^{k}_{i}$ lie on a line. The reason
that the rational form is chosen is that it includes polynomial curves as a
special case and has the capability of exactly representing a circle. These
properties enable the extrusion shapes to include box and cylinder primitive
shapes as special cases.

Due to the special angular set-up of rBézierSketch, the generated profile
curve is a simple closed curve, i.e., it does not self-intersect. In fact, we
have an even stronger result.

###### Proposition 1

The area bounded by the curve generated by rBézierSketch is a star-shaped set.

Moreover, the special choice of angle $\theta$ in (2) makes it possible to
exactly represent a circular arc using Eq. 1.

###### Proposition 2

Let the polar angles be given by Eq. 2. If $\rho_{0}^{k}=\rho_{3}^{k}$,
$\rho_{1}^{k}=\rho_{2}^{k}=\frac{\rho^{k}_{0}}{\cos(\theta)}$, and
$w^{k}_{1}=w^{k}_{2}=\frac{1}{3}\left(1+2\cos\left(\frac{\pi}{N}\right)\right)$,
then the rational Bézier curve of (1) defines a circular arc, as shown in Fig.
3 (left).

The proposed curve model assures $C^{0}$ continuity among the segments since
the Bézier control polygons are connected in a loop by enforcing each curve’s
last control point to be the same as the next curve’s first control point (see
Fig. 3 (middle)). Thus totally $3N$ radial coordinates and $2N$ weights need
to be predicted. In case the predicted control points and weights happen to
satisfy certain condition given by Proposition 3, $C^{1}$ continuity can be
achieved.

###### Proposition 3

If the control points and weights satisfy

$P_{0}^{k+1}=P_{3}^{k}=\frac{\rho_{2}^{k}P^{k+1}_{1}+\rho^{k+1}_{1}P^{k}_{2}}{\rho_{2}^{k}+\rho_{1}^{k+1}},\;\frac{w^{k}_{2}}{w^{k+1}_{1}}=\frac{\rho^{k+1}_{1}}{\rho^{k}_{2}},$
(3)

curve segments $C_{k+1}(t)$ and $C_{k}(t)$ meet at $P_{0}^{k+1}$ with $C^{1}$
continuity.

The proof of above propositions is provided in the supplementary material.

Figure 3: Left: a circular arc. Middle: a $C^{0}$ profile curve. Right: a
$C^{1}$ profile curve.

If for some CAD applications we already have prior knowledge that the models
should be at least $C^{1}$ continuous, then we can enforce $C^{1}$ continuity
by letting $P^{k}_{0}$ and $w_{2}^{k}$ be computed from Eq. S.1. In this case
we just predict $2N$ control points $P^{k}_{1},P^{k}_{2}$ and $N$ weights
$w_{2}^{k}$, which have fewer variables (see Fig. 3 (right)).

Remark 2. In CAD there have been some works to define single-valued curves in
the polar coordinate system. For example, Sanchez-Reyes proposed a subset of
rational Bézier curves that can be used to define single-valued curves [28]
and later extended them to splines [29]. It should be pointed out that our
proposed curves are different from those proposed by Sanchez-Reyes.
Particularly, for a Sanchez-Reyes’s curve, the control points are on radial
directions regularly spaced by a constant angle and each weight must equal the
inverse of the radial coordinate of the corresponding control point, which
leaves very few degrees of freedom (DoF) for shape modeling. Our curves have
more DoFs.

####  3.1.2 Extrusion.

Once the profile curve $C(t)=(x(t),y(t))$ on the $XY-$plane is obtained, the
extrusion shape is generated by directly extruding the curve along the
$Z-$direction. The extrusion shape is bounded by the top and bottom planes and
a side surface. The side surface has the parametric equations
$(x,y,z)=((x(t),y(t),s)$ where $s\in[0,h]$ is the second parameter and $h$ is
the extrusion height estimated by the network.

The network also estimates a quaternion that defines a rotation matrix $R$ and
a translation vector ${\bf t}=(t_{x},t_{y},t_{z})$ that defines a translation
matrix $T$. The matrix $R$ makes the $XY-$plane be in the orientation of the
target sketch plane and the matrix $T$ moves the origin to the target sketch
center on the sketch plane. Thus applying matrices $R$ and $T$ to the upright
extrusion shape gives the target extrusion shape in 3D space, as shown in Fig.
4.

Figure 4: Left: 2D sketch in the $XY-$plane; Middle: direct extrusion; Right:
the target extrusion shape.

###  3.2 SDF-Generation

####  3.2.1 Sketch2SDF.

Note that the generated sketches are in parametric forms. Unlike explicit or
implicit functions, computing a signed-distance-field of parametric curves is
not trivial. There were a few implicitization algorithms for converting a
parametric curve into an implicit representation [30, 38]. However, they
require exact arithmetic computation. Moreover, as observed in [31], the
existence of singularity in parametric representation often makes the
resulting implicit expression useless.

Figure 5: The SDF of a curve is computed by a numerical method.

In this section, we present a numerical method for computing the SDF of
parametric sketches. The method is general. While it applies to the rational
cubic Bézier curves here, it also works for other parametric curves. This
method also yields good gradients which is friendly for deep learning
applications (see Sec. 4.2). The SDF of a sketch is defined by the Distance-
Field $DF(p)$, the smallest distance from a given testing point $p$ to the
curve, multiplied by a sign $SIGN(p)$ which indicates whether the testing
point $p$ is inside or outside of the sketch.

Distance-Field. To compute the Distance-Field of a sketch, we first sample a
set of points $S$ from the curve and take the smallest distance between the
test point $p$ and sample points $S$ as the value of $DF(p)$. Specifically,
given a parametric curve $C(t)=(x(t),y(t)),t\in[0,1]$, the sampling points are
obtained by evenly sampling the parameter values in the parameter domain:

$S=\left\\{\left(x(\frac{i}{n}),y(\frac{i}{n})\right)\mid
i=0,1,\cdots,n\right\\}$ (4)

Then the distance between the testing point $p$ and the set $S$ is found by

$DF(p)=\underset{s\in S}{\min}\|s-p\|_{2}$ (5)

and its corresponding closest point’s parameter value is denoted by $CT(p)$.

Signed Distance-Field. To test if a point $p$ is inside or outside of a
sketch, we check whether if the displacement vector from the testing point
towards the closest point in $S$ has the same direction as the normal vector
of the curve at the closest point. We assume that the curves are parameterized
such that they are traced in the counter-clockwise direction. Otherwise, a
simple reparameterization can correct it. Then the normal vector of the curve
$C(t)$ can be computed by rotating its tangent vector by 90 degrees clockwise:

$N(t)=\left(\frac{dy(t)}{dt},-\frac{dx(t)}{dt}\right)$ (6)

If two consecutive curve segments of the sketch meet with $C^{0}$ continuity,
the normal vectors of the two segments at the joint point are different. To
ensure a consistent normal at the junction of two curves, the average of the
two is used. In this way, the sign function for a testing point $p$ can be
computed:

$SIGN(p)=\frac{N(CT(p)))\cdot(C(CT(p))-p)}{\|N(CT(p)))\cdot(C(CT(p))-p)\|_{2}+\epsilon}$
(7)

where a small positive number $\epsilon$ is added to prevent from zero
division. Finally, the Signed-Distance of $p$ is:

$SDF_{s}(p)=SIGN(p)\times DF(p).$ (8)

Fig. 5 gives an example of a computed SDF.

####  3.2.2 DiffExtruder.

Now we show how to compute the SDF of the extrusion shape. We first transform
the testing point $p$ back to a point $p^{\prime}$ by reversing the
transformations that transform the $XY-$plane to the target sketch plane:
$p^{\prime}=R^{-1}(T^{-1}(p))$. Then we compute the signed distance of
$p^{\prime}$ with respect to the upright extrusion shape whose base lies on
the $XY-$plane. For this purpose, we project point $p^{\prime}$ onto the
$XY-$plane by setting its z-value to 0 and denote the footprint by
$p^{\prime\prime}$, and further find its nearest point $(c_{x},c_{y},0)$ on
the profile curve. Then the projected testing point $p^{\prime\prime}$, the
nearest point $(c_{x},c_{y},0)$ and its corresponding point $(c_{x},c_{y},h)$
on the end plane of the extrusion define a vertical plane. Now, the problem of
finding the SDF of the extrusion shape is reduced into computing the 2D SDF on
the vertical plane.

If a point is inside of the extrusion shape, its signed distance to the shape
simply becomes the minimum of the 2D sketch SDF and the distances from the
point to the two planes that bound the extrusion.

$SDF_{i}(p^{\prime})=\max(\min(SDF_{s}((p^{\prime}_{x},p^{\prime}_{y})),h-p^{\prime}_{z},p^{\prime}_{z}),0)$
(9)

If the point lies outside the extruded primitive, its signed distance becomes:

$\begin{array}[]{lcl}SDF_{o}(p^{\prime})&=&-[(\min(h-p^{\prime}_{z},0))^{2}+(\min(p^{\prime}_{z},0))^{2}\\\
\mbox{}&&+(\min(SDF_{s}((p^{\prime}_{x},p^{\prime}_{y})),0))^{2}]^{\frac{1}{2}}.\end{array}$
(10)

It can be seen that for a testing point outside the extruded shape, $SDF_{i}$
becomes zero, and for a testing point inside the extrusion shape, $SDF_{o}$
becomes zero. Therefore we can simply get the overall SDF by adding the two
terms:

$SDF(p^{\prime})=SDF_{i}(p^{\prime})+SDF_{o}(p^{\prime}).$ (11)

Occupancy of Extrusion Shape. After getting the SDF of an extrusion shape, the
occupancy can be computed by a sigmoid function $\Phi$ where $\eta$ indicates
how sharp the conversion is taken place:

$O(p^{\prime})=\Phi(-\eta\times SDF(p^{\prime})).$ (12)

###  3.3 Training and Inference

We train ExtrudeNet in an end-to-end and unsupervised fashion as no ground
truth sketch and extrusion parameters are present. The supervision signals are
mainly generated by the reconstruction loss $L_{re}$ that computes the
discrepancy between the reconstructed shape’s occupancy $\hat{O}$ and the
ground truth shape’s occupancy $O^{*}$ using a set of testing points
$P\subset{R^{3}}$ sampled within the shape’s bounding box:

$L_{re}=\mathbb{E}_{p\sim P}||\hat{O}_{i}-O^{*}_{i}||_{2}^{2}.$ (13)

As suggested in [25], when a shape is too far from any testing points, the
gradient becomes extremely small due to the use of a sigmoid function. Thus a
primitive loss is introduced to “poll” the primitive shapes towards testing
points:

$L_{prim}=\frac{1}{K}\sum_{k}^{K}\min_{n}SDF_{k}^{2}(p_{n}),$ (14)

We enforce the weights to be close to one, which encourage the rational Bézier
curves to be close to Bézier curves.

Thus, the overall objective is then defined:

$L_{total}=L_{re}+\lambda_{p}\cdot
L_{prim}+\lambda_{w}\cdot\sum\limits_{k=0}^{N-1}\sum\limits_{i=1}^{2}(w^{k}_{i}-1)^{2}$
(15)

where $\lambda_{p}$ and $\lambda_{w}$ are the trade-off factors.

During inference, we discretize the connection matrices into Boolean values
for interpretable construction. In addition, we implement a CAD converter that
directly takes binary connection matrices and the network generated parameters
as input and converts the reconstructed shape into CAD compatible format. We
use OpenSCAD [15] to render the final shape into STL [27] format.

##  4 Experiments

In this section, we first evaluate our complete pipeline ExtrudeNet,
particularly its ability to extract editable and interpretable shape
representations with ablation on different settings. Then, we evaluate the key
component of rBézierSketch and Sketch2SDF on 2D toy examples.

Figure 6: Qualitative comparison between ExtrudeNet and other baselines.
ExtrudeNet generates visually more pleasing outputs with curved surfaces
compared to all the baselines.

###  4.1 Evaluations of ExtrudeNet

Dataset. We evaluate the ExtrudeNet on ShapeNet Dataset [7] with train, test,
and val split aligned with prior methods. The input point clouds are sampled
from the original mesh surface via Poisson Disk Sampling[6]. The testing
points are sampled using [5] in a grid from the mesh bounding box with 15% of
padding on each side. This padding is important to remove unwanted artifacts,
see Supplementary Material.

Implementation Details. We implement ExtrudeNet using PyTorch [22], and train
the network with Adam Optimizer [14] with a learning rate of 1e-4. We train
each class on a single NVIDIA V100 GPU with a batch size of 16. It took about
5 days to converge.

In our experiments, each sketch is constructed with 4 Bézier curves with a
sample rate of 100. We estimate 64 extruded shapes in total, and use 64 as the
number of intersection nodes in Assembly (see Fig. 2) adopted from CSG-Stump
[25].

Main Results - Comparisons with Baselines. We compare our method with related
approaches focusing on editable shape abstraction, namely, VP [33], SQ [21],
UCSG-Net [13], and CSG-Stump Net [25]. Note that BSPNet and CVXNet are
excluded as they mainly focus on the reconstruction of thousands of planes
that contain too many primitives and thus lack editability. For fair
comparison, we align the number of available primitives to 64 for VP, SQ and
UCSG. We show qualitative comparisons with the baseline methods in Fig. 6. It
can be seen that our ExtrudeNet models curved surfaces more effectively
compared to all the baseline methods, thus resulting in a more detailed
reconstruction.

Moreover, we also conduct quantitative evaluations using symmetric $L_{2}$
Chamfer Distance (CD), volumetric IoU and surface F1 scores. The comparisons
of the results are given in Table 1. We can see that ExtrudeNet achieves the
best overall reconstruction quality.

Table 1: Different Metric computed between 3D reconstruction results and the
ground truth shapes. ExtrudeNet outputs better overall results

| VP [33] | SQ [21] | UCSG [13] | Stump [25] | ExtrudeNet
---|---|---|---|---|---
CD | V-IoU | F1 | CD | V-IoU | F1 | CD | V-IoU | F1 | CD | V-IoU | F1 | CD | V-IoU | F1
Chair | 1.10 | 0.28 | 67.07 | 2.85 | 0.21 | 41.49 | 3.54 | 0.25 | 61.01 | 1.34 | 0.41 | 63.33 | 1.47 | 0.41 | 60.17
Car | 1.02 | 0.67 | 62.71 | 1.25 | 0.17 | 78.58 | 0.64 | 0.11 | 82.54 | 0.76 | 0.32 | 76.45 | 0.67 | 0.41 | 82.20
Sofa | 2.18 | 0.29 | 48.75 | 1.27 | 0.37 | 69.14 | 1.30 | 0.29 | 79.59 | 0.85 | 0.61 | 83.88 | 0.79 | 0.62 | 83.42
Plane | 5.11 | 0.29 | 37.55 | 0.58 | 0.23 | 82.94 | 0.71 | 0.10 | 87.66 | 0.70 | 0.36 | 74.57 | 0.66 | 0.33 | 77.95
Lamp | 8.41 | 0.24 | 35.57 | 1.79 | 0.16 | 63.00 | 7.02 | 0.22 | 55.66 | 3.48 | 0.24 | 57.93 | 3.77 | 0.20 | 42.86
Telephone | 2.75 | 0.55 | 48.22 | 0.46 | 0.38 | 88.29 | 0.34 | 0.69 | 93.32 | 1.60 | 0.58 | 91.24 | 0.56 | 0.62 | 89.00
Vessel | 2.84 | 0.37 | 54.21 | 0.76 | 0.27 | 78.52 | 4.54 | 0.09 | 61.65 | 1.15 | 0.44 | 72.30 | 1.63 | 0.48 | 61.06
Loudspeaker | 1.67 | 0.45 | 50.06 | 2.07 | 0.34 | 65.00 | 1.81 | 0.19 | 56.35 | 1.70 | 0.52 | 63.32 | 1.21 | 0.60 | 75.04
Cabinate | 3.16 | 0.48 | 42.90 | 1.97 | 0.31 | 40.52 | 1.09 | 0.38 | 73.56 | 0.77 | 0.56 | 79.11 | 0.73 | 0.68 | 84.94
Table | 1.62 | 0.26 | 60.05 | 2.89 | 0.17 | 60.01 | 3.64 | 0.30 | 65.09 | 1.35 | 0.35 | 74.48 | 1.06 | 0.45 | 73.49
Display | 1.25 | 0.36 | 60.52 | 0.72 | 0.33 | 80.09 | 1.13 | 0.54 | 78.18 | 1.64 | 0.50 | 75.75 | 0.96 | 0.55 | 83.66
Bench | 1.57 | 0.26 | 63.31 | 1.09 | 0.17 | 73.46 | 1.73 | 0.21 | 74.12 | 1.04 | 0.29 | 73.40 | 0.78 | 0.39 | 81.17
Rifle | 1.35 | 0.35 | 66.06 | 0.38 | 0.26 | 90.55 | 1.05 | 0.29 | 84.38 | 0.78 | 0.37 | 85.61 | 0.76 | 0.44 | 84.18
Mean | 2.61 | 0.37 | 53.61 | 1.39 | 0.25 | 70.12 | 2.19 | 0.28 | 73.31 | 1.32 | 0.42 | 74.64 | 1.15 | 0.47 | 75.32

Table 2: CD results ($\times 10^{-3}$) with shapes extruded from different numbers of sketches.  #Primitives | 8 | 16 | 32 | 64
---|---|---|---|---
Plane | 0.89 | 0.96 | 0.81 | 0.66
Chair | 2.01 | 1.81 | 1.47 | 1.47
Bench | 1.41 | 1.18 | 1.30 | 0.78

Effect of Different Numbers of Primitives. To show that our proposed extrude
shapes with freeform profile curves are highly adaptable, we train ExtrudeNet
with limited numbers of extruded shapes. Table 2 reports the results on the
three classes. It can be seen that even with only 8 available extruded shapes,
our method can still achieve reasonable reconstruction results.

Table 3: CD results ($\times 10^{-3}$) with shapes extruded from different sketches on the Airplane class.  Sketch Type | Freeform | Circle | Poly
---|---|---|---
CD | 0.66 | 0.877 | 0.838

Effect of Different Sketches. Our method can also use other profile curves
such as polygonal and circular. Table 3 gives a comparison on the CD results
under different types of profile curves. We can see that extruded shapes with
the proposed freeform profile curves outperform the results using the other
two common profile curves by a significant margin, which indicates the strong
representability of the extruded shapes with freeform profile curves.

Edibility of Sketch and Extrude. After getting the results from ExtrudeNet, a
designer can do a secondary development with ease. Using the bench example in
Fig. 7, we show that by editing the Bezier Control Points and reducing the
extrusion height, we can easily generate an armchair from the reconstruction
result.

Figure 7: As ExtrudeNet’s output is directly compatible with Industrial CAD
software, we can directly import the shape into Fusion360 and edit using its
GUI interface. We show that a bench can be edited into an armchair by simply
edit the 2D cross-section sketch by dragging the control points and adjust the
extrusion height. Left: ExtrudeNet’s output. Middle: Edit sketch using Fusion
360. Right: Result after editing.

More ablations such as sampling rate, number of curves, sketch plane placement
etc., can be found in the supplementary material.

###  4.2 Evaluation of rBézierSketch

To demonstrate the approximation ability and the ease of learning of the
proposed rBézierSketch, we conduct a fitting experiment that directly optimize
for the radial coordinate that best reconstructs a given raster emoji image
from [4]. As the emojis contains different color blocks, which is not suitable
to be represented using occupancy or SDFs, we choose to approximate the
individual boundaries instead of solid colors blocks. Given a raster emoji
image, we first compute its edges using Canny edge detection, and then we
compute the Distance Field for the edge image. We fit the sketch by minimizing
the MSE loss between the predicted and ground truth distance fields using
standard gradient decent. As shown in Fig. 8, the emoji can be reconstructed
by rBézierSketch with good quality. We also show the training plot, where a
smooth decrease in the loss term indicating that rBézierSketch and
Sketch2Field yields good gradients and it is friendly to used in a learning
setup.

Figure 8: Left: We show that using rBézierSketch alone we can reconstruct the
emojis with good qualities. Right: Our rBézierSketch and Sketch2SDF provide
low variance gradients which yields a smooth convergence.

###  4.3 Evaluation of Sketch2SDF

Here we demonstrate the versatility and accuracy of Sketch2SDF on computing
parametric sketches’ SDF by giving concrete examples.

Versatility. To show the versatility, we implemented four different kinds of
parametric curves, namely polygon, ellipse, Cubic Bezier Sketchs with $C^{1}$
and $C^{0}$ continuity, see Fig. 9(Top). Note that our method is not limited
to these curves, and can be easily adapted to new parametric curve types.
During the implementation of these curves, only the curve sampling and normal
computation need to be adapted accordingly while the rest of the code remains
untouched. This indicates that our pipeline is highly reusable and can support
new parametric curves with minimum effort.

Figure 9: Top: Sketch2SDF can be easily adapted to different parametric curve
types. Bottom: Visualization of the approximation errors by comparing
numerical and analytical SDFs of Bezier sketch. (a)$\sim$(d) indicate the
errors at sampling rate of 20, 40, 80 and 120. We can see that error is barely
noticeable with a relatively small sampling rate.

Approximation Accuracy. As Sketch2SDF is a numerical method, approximation is
introduced, where the sample rate is the most influential factor. Thus, we
study the degree of approximation under different sampling rates by comparing
our numerical results against the analytical results using Bézier Sketches. In
Fig.9, we plot the maximum error of all testing points versus sample rate and
visualize the error heat map. We can see that too few sample points will
dramatically decrease the accuracy, and higher sampling rates yield more
accurate approximation, which is however at the cost of higher computation and
memory usage. Empirically, we find that around 400 sample points per curve is
a reasonable trade-off.

##  5 Conclusion

In this paper, we have presented ExtrudeNet, which is an effective framework
for unsupervised inverse sketch-and-extrude for shape parsing from point
clouds. It not only output a compact, editable and interpretable shape
representation but also a meaningful sketch-and-extrude process, which can
benefit various applications. As demonstrated by extensive experiments,
ExtudeNet can express complex shapes with high compactness and
interpretability while achieving state-of-the-art reconstruction results both
visually and quantitatively.

Discussion. Currently, ExtrudeNet requires a relatively long training time as
sketches are more freeformed and have higher degrees of freedom. We also
notice that some artifacts have been reconstructed due to high degrees of
freedom. The current extrusion is a simple extrusion along the direction
orthogonal to the sketch plane. Generalizing the extrusion to include more
advanced processes such as sweeping and lofting warrants further
investigation, which would have more impacts in the industry.

Acknowledgements The work is partially supported by a joint WASP/NTU project
(04INS000440C130), Monash FIT Startup Grant, and SenseTime Gift Fund.

## References

  * [1] Shapr3d: The world’s most intuitive 3d design app, https://www.shapr3d.com/
  * [2] Solidworks: 3d cad design software, https://www.solidworks.com/
  * [3] Fusion 360: Cloud powered 3d cad/cam software for product design (Jun 2021), https://www.autodesk.com.sg/products/fusion-360/overview
  * [4] Color and black-and-white noto emoji fonts, and tools for working with them. (Mar 2022), https://github.com/googlefonts/noto-emoji
  * [5] Batty, C.: Sdfgen. https://github.com/christopherbatty/SDFGen (2015)
  * [6] Bridson, R.: Fast poisson disk sampling in arbitrary dimensions. SIGGRAPH sketches 10, 1 (2007)
  * [7] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al.: Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)
  * [8] Chen, Z., Tagliasacchi, A., Zhang, H.: Bsp-net: Generating compact meshes via binary space partitioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 45–54 (2020)
  * [9] Deng, B., Genova, K., Yazdani, S., Bouaziz, S., Hinton, G., Tagliasacchi, A.: Cvxnet: Learnable convex decomposition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 31–44 (2020)
  * [10] Farin, G.: Curves and Surfaces for CAGD: A Practical Guide. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 5th edn. (2001)
  * [11] Ganin, Y., Bartunov, S., Li, Y., Keller, E., Saliceti, S.: Computer-aided design as language. arXiv preprint arXiv:2105.02769 (2021)
  * [12] Hao, Z., Averbuch-Elor, H., Snavely, N., Belongie, S.: Dualsdf: Semantic shape manipulation using a two-level representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7631–7641 (2020)
  * [13] Kania, K., Zieba, M., Kajdanowicz, T.: Ucsg-net–unsupervised discovering of constructive solid geometry tree. arXiv preprint arXiv:2006.09102 (2020)
  * [14] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [15] Kintel, M., Wolf, C.: Openscad. GNU General Public License, p GNU General Public License (2014)
  * [16] Laidlaw, D.H., Trumbore, W.B., Hughes, J.F.: Constructive solid geometry for polyhedral objects. In: Proceedings of the 13th annual conference on Computer graphics and interactive techniques. pp. 161–170 (1986)
  * [17] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics 21(4), 163–169 (1987)
  * [18] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4460–4470 (2019)
  * [19] Para, W.R., Bhat, S.F., Guerrero, P., Kelly, T., Mitra, N., Guibas, L., Wonka, P.: Sketchgen: Generating constrained cad sketches. arXiv preprint arXiv:2106.02711 (2021)
  * [20] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: Deepsdf: Learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 165–174 (2019)
  * [21] Paschalidou, D., Ulusoy, A.O., Geiger, A.: Superquadrics revisited: Learning 3d shape parsing beyond cuboids. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10344–10353 (2019)
  * [22] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019), http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [23] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 652–660 (2017)
  * [24] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems 30 (2017)
  * [25] Ren, D., Zheng, J., Cai, J., Li, J., Jiang, H., Cai, Z., Zhang, J., Pan, L., Zhang, M., Zhao, H., et al.: Csg-stump: A learning friendly csg-like representation for interpretable shape parsing. arXiv preprint arXiv:2108.11305 (2021)
  * [26] Requicha, A.A., Voelcker, H.B.: Constructive solid geometry (1977)
  * [27] Roscoe, L., et al.: Stereolithography interface specification. America-3D Systems Inc 27(2020), 10 (1988)
  * [28] Sanchez-Reyes, J.: Single-valued curves in polar coordinates. Computer-Aided Design 22(1), 19–26 (1990)
  * [29] Sánchez-Reyes, J.: Single-valued spline curves in polar coordinates. Computer-aided design 24(6), 307–315 (1992)
  * [30] Sederberg, T.W., Anderson, D.C., Goldman, R.N.: Implicit representation of parametric curves and surfaces. Computer Vision, Graphics, and Image Processing 28(1), 72–84 (1984)
  * [31] Sederberg, T.W., Zheng, J., Klimaszewski, K., Dokken, T.: Approximate implicitization using monoid curves and surfaces. Graph. Model. Image Process. 61(4), 177–198 (1999), https://doi.org/10.1006/gmip.1999.0497
  * [32] Sharma, G., Goyal, R., Liu, D., Kalogerakis, E., Maji, S.: Csgnet: Neural shape parser for constructive solid geometry. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5515–5523 (2018)
  * [33] Tulsiani, S., Su, H., Guibas, L.J., Efros, A.A., Malik, J.: Learning shape abstractions by assembling volumetric primitives. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2635–2643 (2017)
  * [34] Wang, G.J.: Rational cubic circular arcs and their application in cad. Computers in Industry 16(3), 283–288 (1991). https://doi.org/https://doi.org/10.1016/0166-3615(91)90066-I, https://www.sciencedirect.com/science/article/pii/016636159190066I
  * [35] Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog) 38(5), 1–12 (2019)
  * [36] Wu, R., Xiao, C., Zheng, C.: Deepcad: A deep generative network for computer-aided design models. arXiv preprint arXiv:2105.09492 (2021)
  * [37] Yu, F., Chen, Z., Li, M., Sanghi, A., Shayani, H., Mahdavi-Amiri, A., Zhang, H.: Capri-net: Learning compact cad shapes with adaptive primitive assembly. arXiv preprint arXiv:2104.05652 (2021)
  * [38] Zheng, J., Sederberg, T.W.: A direct approach to computing the $\mathrm{\mu}$-basis of planar rational curves. J. Symb. Comput. 31(5), 619–629 (2001). https://doi.org/10.1006/jsco.2001.0437, https://doi.org/10.1006/jsco.2001.0437

ExtrudeNet: Unsupervised Inverse Sketch and Extrude for Shape Parsing
– Supplementary Materials –

##  F Propositions and Proofs

###### Proposition 4

The area bounded by the curve generated by rBézierSketch is a star-shaped set.

Proof: Consider an arbitrary ray cast from the origin. Since the polygon
generated by connecting all the control points in order is homeomorphic to the
origin-centered unit circle, the ray will intersect the polygon at least once.
On the other hand, based on the construction of the control points of the
rational Bézier curve, when we travel along the polygon in the counter-clock
direction, the polar angle is monotonically increasing, which means that the
ray will not intersect the polygon more than once. Therefore the ray
intersects the Bézier control polygon only once, so does it intersect the
profile curve, which is confirmed by the continuity of the curve and the
variation diminishing property of rational Bézier curves. The variation
diminishing property says that for an arbitrary line, the number of
intersections with the curve will not be greater than the number of
intersections with the control polygon [10]. Hence any point on the profile
curve can be visible to the origin.

###### Proposition 5

Let the polar angles be given by Eq.2. If $\rho_{0}^{k}=\rho_{3}^{k}$,
$\rho_{1}^{k}=\rho_{2}^{k}=\frac{\rho^{k}_{0}}{\cos(\theta)}$, and
$w^{k}_{1}=w^{k}_{2}=\frac{1}{3}\left(1+2\cos\left(\frac{\pi}{N}\right)\right)$,
then the rational Bézier curve of (1) defines a circular arc, as shown in
Fig.3 (left).

Proof: Consider triangle $\triangle OP_{0}^{k}P_{1}^{k}$ shown in Fig.3
(left). If $\rho_{1}^{k}=\frac{\rho^{k}_{0}}{\cos(\theta)}$, $\angle
OP_{0}^{k}P_{1}^{k}$ is a right angle. Then
$\|P_{0}^{k}P_{1}^{k}\|=\rho^{k}_{0}\tan(\theta)$. Since
$\displaystyle\theta=\frac{\pi}{2N}+\tan^{-1}\left(\frac{1}{3}\tan\left(\frac{\pi}{2N}\right)\right)$,
we thus have

$\begin{array}[]{lcl}\|P_{0}^{k}P_{1}^{k}\|&=&\displaystyle\rho_{0}^{k}\frac{\tan\left(\frac{\pi}{2N}\right)+\frac{1}{3}\tan\left(\frac{\pi}{2N}\right)}{1-\tan\left(\frac{\pi}{2N}\right)\cdot\frac{1}{3}\tan\left(\frac{\pi}{2N}\right)}\\\
\mbox{}&=&\displaystyle\rho_{0}^{k}\frac{\frac{4}{3}\frac{\sin\left(\frac{\pi}{2N}\right)}{\cos\left(\frac{\pi}{2N}\right)}}{1-\frac{1}{3}(\sec^{2}\left(\frac{\pi}{2N}\right)-1)}\\\
\mbox{}&=&\displaystyle\rho_{0}^{k}\frac{4\sin\left(\frac{\pi}{2N}\right)\cos\left(\frac{\pi}{2N}\right)}{4\cos^{2}\left(\frac{\pi}{2N}\right)-1}\\\
\mbox{}&=&\displaystyle\rho_{0}^{k}\frac{2\sin\left(\frac{\pi}{N}\right)}{1+2\cos\left(\frac{\pi}{N}\right)}.\end{array}$

Similar analysis applies to the other end of the Bézier curve. It can be
easily verified that these equations together with the chosen weights in the
proposition make the conditions given in [34] be satisfied, which ensure that
the generated rational cubic Bézier curve is an circular arc, and thus we
arrive at the conclusion.

###### Proposition 6

If the control points and weights satisfy

$P_{0}^{k+1}=P_{3}^{k}=\frac{\rho_{2}^{k}P^{k+1}_{1}+\rho^{k+1}_{1}P^{k}_{2}}{\rho_{2}^{k}+\rho_{1}^{k+1}},\;\frac{w^{k}_{2}}{w^{k+1}_{1}}=\frac{\rho^{k+1}_{1}}{\rho^{k}_{2}},$
(S.1)

curve segments $C_{k+1}(t)$ and $C_{k}(t)$ meet at $P_{0}^{k+1}$ with $C^{1}$
continuity.

Proof: Consider curve segments $C_{k}(t)$ and $C_{k+1}(t)$. First,
$P_{0}^{k+1}=P_{3}^{k}$ assures $C^{0}$ continuity of the two segments.
Second, from Eq.S.1, we have

$\rho_{2}^{k}(P_{1}^{k+1}-P_{0}^{k+1})=\rho_{1}^{k+1}(P^{k}_{3}-P^{k}_{2})$
(S.2)

or

$w_{1}^{k+1}(P_{1}^{k+1}-P_{0}^{k+1})=w_{2}^{k}(P^{k}_{3}-P^{k}_{2}).$ (S.3)

Eq.S.2 implied that $P_{0}^{k+1}$ or $P_{3}^{k}$ does lie on the bisector of
angle $\angle P_{2}^{k}OP_{1}^{k+1}$. Eq.S.3 means
$C^{\prime}_{k}(1)=C^{\prime}_{k+1}(0)$. This is because

$C^{\prime}_{k}(1)=3\left((P_{3}^{k}-w^{k}_{2}P_{2}^{k})-(1-w^{k}_{2})P^{k}_{3}\right)=3w_{2}^{k}(P_{3}^{k}-P^{k}_{2})$

and similarly,

$C^{\prime}_{k+1}(0)=3w_{1}^{k+1}(P_{1}^{k+1}-P^{k+1}_{0}).$

Therefore $C_{k+1}(t)$ and $C_{k}(t)$ meet at $P_{0}^{k+1}$ with $C^{1}$
continuity.

##  G More Ablation Studies

##### Ablation on Number of Sample Points Per Curve

As discussed in Fig.10, sample rate plays an important role on the accuracy of
the approximation. To further demonstrate that Sketch2SDF can work with a
relative small sampling rate, we vary the sampling rate and train ExtrudeNet
separately, as shown in the Tab.D, even with a small sample rate (80 per
sketch) ExtrudeNet still manages to generate comparable results.

Table D: Chamfer Distances of Airplane class under different sample rate. We can see that ExtrudeNet performs comparably even under small number of sample points. Sample Rate | 400 | 320 | 160 | 80
---|---|---|---|---
CD | 0.664 | 0.702 | 0.726 | 0.725

##### Ablation on Number of Bezier Curve Per Sketch

rBézierSketch can be configured to generate sketches with different number of
Bezier Curves. Highly flexible sketch can be achieved via using a large number
of curves, however, this increase the computation cost and also gives network
too much degree of freedom, making the network much harder to train.

Table E: Performance of ExtrudeNet under different numbers of Curves. We can see that too few Bézier curves is not flexible enough while large number of curve makes network harder to train. # Curves | 2 | 4 | 6 | 8
---|---|---|---|---
CD | 0.754 | 0.664 | 0.722 | 1.09

##### Ablation on Sketch Plane Orientation

Three-view-drawing is a popular drawing method that placing the sketch plane
on 3D coordinate planes. In this ablation, we study the effect of the
ExtrudeNet’s performance under constraint sketch plane orientation, see Tab.F.
Interestingly, the results are consistent with how human tends to draw shapes
(from top and front).

Figure J: Reconstruction results by setting constraints on the extrusion plane orientation. Left: Top Extrude. Middle: Side Extrude. Left: Front Extrude. Table F: We train ExtrudeNet with constraint sketch plane orientation, we can see that freely placed sketch plane yields the best result. Orientation | side | top | front | free
---|---|---|---|---
CD | 3.291 | 0.821 | 0.733 | 0.664
Figure K: Left: Without padding the generated mesh contains artifacts that
overshoot the gray bounding box. Right: Result with padding.

Effect of Padding. As Bézier Sketches are very flexible and locally supported,
only the control points that have testing points in their proximity are
updated. Note that we add 15% padding on each side of the mesh bounding box.
Without padding, artifacts that “over-shoot” outside the bounding box can be
generated (Fig.K). Adding padding when sampling the testing points can solve
this problem, as “over-shot” shapes in the padded region will incur a higher
loss which forces all the control points to be updated.

##  H More Visualizations

We have created a video for better visualization, see attached. The video is
composed of two sections. First section illustrates ExtrudeNet in detail with
special attention on the ”Sketch” and ”Extrude” process. Note that, to better
visualize individual extruded shapes, we using UNION as the assembly method,
thus the final reconstruction results are worse than the complete ExtrudeNet.
Second section of the video shows detailed reconstructed results. We have also
attached more rendered results in Fig.L.

Figure L: More visualization of ExtrudeNet results


# Fast DCTTS: Efficient Deep Convolutional Text-to-Speech

###### Abstract

We propose an end-to-end speech synthesizer, Fast DCTTS, that synthesizes
speech in real time on a single CPU thread. The proposed model is composed of
a carefully-tuned lightweight network designed by applying multiple network
reduction and fidelity improvement techniques. In addition, we propose a novel
group highway activation that can compromise between computational efficiency
and the regularization effect of the gating mechanism. As well, we introduce a
new metric called elastic mel-cepstral distortion (EMCD) to measure the
fidelity of the output mel-spectrogram. In experiments, we analyze the effect
of the acceleration techniques on speed and speech quality. Compared with the
baseline model, the proposed model exhibits improved MOS from 2.62 to 2.74
with only 1.76% computation and 2.75% parameters. The speed on a single CPU
thread was improved by 7.45 times, which is fast enough to produce mel-
spectrogram in real time without GPU.

Index Terms—  Text-to-speech, real-time, light-weight autoregressive model,
network reduction

##  1 Introduction

In recent few years, deep-learning based speech synthesis technology has been
rapidly improved. Tacotron[1] is the end-to-end speech synthesis model that
has demonstrated the possibility of the neural speech synthesizer. The
following work, Tacotron2[2] exhibits a fidelity comparable to that of human
speech. Tacotron model consists of the encoder-attention-decoder structure
based on recurrent neural networks (RNN), which requires a long time to learn.
To accelerate training speed, Tachibana et al. propose a TTS model, called
DCTTS[3], that is based on convolutions instead of recurrences. Leveraging
parallel computing, the training speed of DCTTS is faster than that of
Tacotron, while the quality of the output speech is comparable. Transformer-
TTS[4] is based on Transformer[5] network which has shown successful results
in many Natural Language Processing (NLP) tasks. Transformer-TTS[4] exhibits
slightly improved speech quality compared with Tacotron2[2].

In regard to synthesis speed on a high-performance GPU, the most important
bottleneck of the aforementioned models is the autoregressive decoding process
that requires sequential computations. Recent works including FastSpeech[6],
AlignTTS[7], and FastSpeech2[8] apply non-autoregressive architecture and
significantly improve synthesis speed. However, their improvements are
achieved mainly by parallelization rather than reduction in computations.

In this paper, we propose a neural TTS model that synthesizes high-fidelity
mel-spectrogram in real time without a GPU. We designed a lightweight network
by applying multiple techniques to reduce computation and improve fidelity. In
addition, we propose a novel group highway activation that can compromise
between computational efficiency and the regularization effect of the gating
mechanism. For the evaluation of speech quality, we propose a new metric
called elastic mel-cepstral distortion (EMCD) that measures the distance
between the synthesized mel-spectrogram and the ground truth considering
skipping and repeating errors.

In experiments, we evaluate the effect of the acceleration techniques
measuring their influence on fidelity using EMCD and mean opinion score (MOS).
The experimental results demonstrate that the network reduction techniques are
effective in accelerating a neural TTS model. Compared with the baseline
model, the proposed model exhibits improved MOS from 2.62 to 2.74 with only
1.76% computation and 2.75% parameters111Sound demos are available at:
https://jackson-kang.github.io/paper_works/FastDCTTS/. The synthesis speed on
a single CPU thread was improved by 7.45 times, which is fast enough to
produce mel-spectrogram in real time without a GPU.

On the other hand, our experimental results suggest that the depthwise
separable convolution, which is known effective in the image recognition
field, does not improve the speed of neural TTS in practice, although it
significantly reduces the computation of 1D convolution in theory. We have
also observed that replacing the highway activation of DCTTS with residual
connection degrades the output quality significantly.

The contributions of our work are summarized as follows$\colon$

  * •

We present a neural TTS model that generates a mel-spectrogram in real time on
a single CPU thread.

  * •

We qualitatively evaluate the effect of multiple techniques to reduce
computation and improve fidelity on a neural TTS model using EMCD.

  * •

We propose a novel group highway activation that can compromise between
computational efficiency and the regularization effect of the gating
mechanism.

  * •

We propose a new evaluation metric, EMCD, to measure the quality of a mel-
spectrogram.

††
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

##  2 Related Works

###  2.1 Baseline model

We chose DCTTS [3] as the baseline model because of two primary reasons:
first, there are many acceleration techniques applicable for CNNs[9, 10, 11,
12] most of which have been developed for image recognition. Second, DCTTS [3]
can be trained and evaluated in a short time. In practice, this is an
important advantage when searching for the best model through a large number
of experiments.

The Text2Mel network of DCTTS [3] is composed of a text encoder, an attention
module, an audio encoder, and an audio decoder. The text encoder converts the
input text into two sequences of character embeddings, $K$ and $V$. The audio
encoder converts the previously generated mel-spectrogram into a sequence of
audio embeddings $Q$. The attention module computes an attention matrix $A$
from $K$ and $Q$, which represents the alignment between the parts of the
input text and the output mel-spectrogram. The audio decoder generates the
mel-spectrogram of the next time step from $A$, $V$ and $Q$ in an auto-
regressive way.

The number of layers in the text encoder, audio encoder and audio decoder are
14, 13, and 11, respectively. All networks apply the highway activation[13].
Additionally, we applied weight normalization[14] to improve training
efficiency.

###  2.2 Building lightweight networks

Neural TTS models require a huge amount of computation. Most of the successful
results in TTS acceleration have been achieved through parallelization.
However, to improve synthesis speed on a CPU, we need to reduce the actual
amount of computation.

In the image processing field, various techniques have been developed to
reduce the computational requirement of CNN. Iandola et al. present
SqueezeNet[15], a lightweight network composed of Fire modules. MobileNet[9]
reduces the amount of computation by applying depthwise separable convolution,
width multiplier, and resolution multiplier. ShuffleNet[12] improves
computational efficiency by applying group convolution combined with channel
shuffling. MobileNet2[16] further improves MobileNet[9] by applying additional
techniques such as linear bottlenecks and inverted residuals. Another popular
technique to build a reduced network is network pruning[17] that removes less
important parts of a neural network to save computation and parameters.

##  3 Optimization Techniques for Neural TTS

###  3.1 Computational optimization techniques

####  3.1.1 Depthwise separable convolution

Depthwise separable convolution[9] in image processing fields approximates a
3D convolution ($channel\times height\times width$) with a 2D depthwise
convolution ($height\times width$) and a 1D pointwise convolution ($channel$).
In speech synthesis, depthwise separable convolution [9] decomposes 2D
convolution ($channel\times time$) into a 1D depthwise convolution and a 1D
pointwise convolution. The computational complexity of 2D convolution is
$O(D_{K}MND_{F})$, where $D_{K}$ and $D_{F}$ are the sizes of the convolution
filter and the feature map. $M$ and $N$ denote and the number of the input and
output channels, respectively. On the other hand, the complexity of depthwise
and pointwise convolutions are $O(D_{K}MD_{F})$ and $O(MD_{F}D_{F})$,
respectively. Consequently, in theory, the reduction in computation is
$\frac{1}{N}+\frac{1}{D_{K}}$.

####  3.1.2 Group highway activation

$\displaystyle y=T(x,W_{T})H(x,W_{H})+C(x,W_{C})x$ (1)

DCTTS [3] applies highway activation layers[13] as equation (1), where $x$ and
$y$ denote the input and output feature maps, respectively, and $H(x,W_{H})$
is an operator such as convolution. $T(x,W_{T})$ and $C(x,W_{C})$ are the
transformation and carry gates. $W_{T}$, $W_{C}$, and $W_{H}$ are the
parameters of the gates and operation, respectively. $C(x,W_{C})$ can be
replaced by $1-T(x,W_{T})$. The two gates help the learning of deep neural
networks. However, they increase the computational burden by 2x or 3x, as the
computational cost of each gate is the same as that of $H(x,W_{H})$.

To reduce computation, we applied two alternatives of highway activation [13].
One is residual connection[18], which is a special case of highway activation
[13] with $T(x,W_{T})=C(x,W_{C})=1$. Residual connection is also known
effective in training deep neural networks, while requiring only minimal
additional overhead. First, we replaced highway activation[13] with residual
connection[18] on all layers. The resulting model is called Residual DCTTS.
However, as displayed in Table 1, Residual DCTTS degrades speech quality
significantly. Particularly, it substantially increases skipping and repeating
errors when applied to audio encoder or audio decoder. However, it does not
degrade speech quality seriously when applied only to text encoder. We presume
that removing the learned gating mechanism increases noise in the intermediate
acoustic features, and consequently, destabilizes attention.

After observing that the learned gating mechanism is important for attention
stability, we developed the other alternative called group highway activation.
The new activation combines feature elements into groups, and the elements
within the group share gate values. The size of the gate vector is reduced by
$\frac{1}{g}$, where $g$ is group size. Replacing highway activation by group
highway activation, the computation of a convolution layer is reduced by
$(1+\frac{1}{g})/2$. By adjusting $g$, we can search for a trade-off between
computational efficiency and the regularizing effect of the gating mechanism.

####  3.1.3 Network size reduction

Reducing network size can improve synthesis speed, as well. The text encoder,
audio encoder and audio decoder networks of the baseline model consist of 14,
13, and 11 convolution layers, respectively. Each layer of the audio encoder
and audio decoder networks contain 256 channels. The layers of the text
encoder contain 512 channels. We searched for the smallest network that
maintains acceptable fidelity by reducing the number of layers and channels
monitoring the quality of the output speech.

####  3.1.4 Network pruning

Network pruning[17] is a technique to reduce the size and computation of the
neural network. After training, the pruning algorithm estimates the importance
value of each unit, and then, removes the units with low importance values.
The pruning of a CNN model[10] is performed for the convolution filters or
feature maps. In this study, we applied the pruning method of [10] with a few
modification to fit group highway activation. First, we remove less important
feature maps. Then, we remove the gate elements whose features have been
removed.

###  3.2 Fidelity improvement techniques

Positional encoding is a way to add a piece of information that reflects
absolute or relative position to the feature vector. Particularly, it improves
attention stability by helping to learn the temporal relation between feature
vectors[11]. In this research, we applied the scaled positional encoding of
Transformer-TTS[4]. Positional information is added to the feature vector as
${x^{\prime}}_{i}={x}_{i}+\alpha PE(pos,i)$, where $PE(pos,i)$ is defined as
$PE(pos,2k)=sin({pos\over base^{2k/dim}})$ for $i=2k$, and
$PE(pos,2k+1)=cos({pos\over base^{2k/dim}})$ for $i=2k+1$. $\alpha$ is a
trainable weight.

In addition to positional encoding, we applied scheduled sampling[19]. In the
training of an autoregressive decoder, each step requires the output of the
previous step, which makes it difficult to parallelize. Applying teacher
forcing, we can train the autoregressive decoder using the corresponding part
of the ground-truth mel-spectrogram instead of the output of the previous
step. However, teacher forcing cause a discrepancy that the model is optimized
for the ground truth in training, while it generates a new output from the
previous output. Scheduled sampling[19] alleviates such discrepancy.
Initially, the model learns mainly from the ground truth. As the learning
progresses, it increases portion of the previous output.

##  4 Elastic Mel Cepstral Distortion (EMCD)

To build a high-fidelity lightweight TTS model, we need to evaluate not only
speed but also speech quality. We evaluate the quality of the output mel-
spectrogram by the distance from the ground truth. However, it is not simple
to measure the distance between mel-spectrograms that are not aligned. For
example, the Euclidean metric does not work well for the mel-spectrograms when
one of them contains the skipping or repeating problem. For this reason, we
have developed a new evaluation metric, elastic mel cepstral distortion
(EMCD), that measures the difference between mel-spectrograms considering the
alignment between them.

$\displaystyle D(i,$ $\displaystyle j)=w_{m}\times MCD(x_{i},y_{j})$ (2)
$\displaystyle+min\big{\\{}D(i,j-1),D(i-1,j),D(i-1,j-1)\big{\\}}$

Mel cepstral distortion(MCD)[20] is known effective in measuring perceptual
distance between speech signals and expressed as
$MCD(i,j)=\sqrt{2\sum_{d=1}^{D}({x}_{d}[i]-{y}_{d}[j])^{2}}$, where
$i=\\{1,...,{T}_{syn}\\}$, $j=\\{1,...,{T}_{gt}\\}$, $x$ and $y$ are the
sequences of mel frequency cepstral coefficient (MFCC) converted from the
synthesized and ground truth mel-spectrograms, and $T_{syn}$ and $T_{gt}$
denote their length, respectively. We have extended MCD by combining it with
dynamic time warping (DTW)[21], to measure distance while finding the optimal
alignment via dynamic programming. Each step of the alignment process is
formulated as equation (2), where $w=[w_{hor},w_{ver},w_{diag}]$ and
$m=argmin\big{\\{}D(i,j-1),\ D(i-1,j),\ D(i-1,j-1)\big{\\}}$. The weight
vector $w$ is composed of hyper-parameters to control the penalties for the
horizontal, vertical, and diagonal transitions, each of which corresponds to
repeating, skipping, and matching, respectively.

In this research, we set $w=(1,1,\sqrt{2})$, which leads the matching
algorithm to assign a penalty weight of $w_{diag}=\sqrt{2}$ to the diagonal
matching path from $(i-1,j-1)$ to $(i,j)$ and a penalty weight of
$w_{hor}+w_{ver}=2$ to the rectangular matching trajectory between the same
points. After matching, $EMCD(T_{syn},T_{gt})$ provides EMCD between $x$ and
$y$. In evaluation, we normalize the EMCD value by $T_{gt}$ to eliminate the
effect of speech length.

The idea of EMCD is similar to that of MCD-DTW[22]. However, in contrast to
MCD-DTW[22], EMCD assigns different penalty weights to horizontal, vertical,
and diagonal transitions, and therefore, is more effective in measuring the
difference caused by skipping and repeating.

Model |  Syn. time (compared to BGPU) | EMCD
---|---|---
LJ | KSS
|  BaseGPU (HC)
---
|  1.28 sec (1.00 $\times$)
---
9.45 | 10.36
|  BaseCPU (HC)
---
|  6.85 sec (5.35 $\times$)
---
| ResDCTTS
---
|  3.85 sec (3.00 $\times$)
---
12.93 | 13.59
| GH DCTTS
---
|  6.37 sec (4.98 $\times$)
---
9.10 | 9.29
Table 1: Comparison of the baseline model, residual DCTTS, and group highway
DCTTS. HC, ResDCTTS and GH DCTTS denote highway convolution, Residual DCTTS
and Group highway DCTTS, respectively. The baseline model is denoted as Base.

##  5 Experiments

###  5.1 Experimental settings

We used two datasets for experiments: the LJSpeech[23] and the Korean Single
Speaker (KSS) speech dataset[24]. The LJSpeech dataset[23] is composed of
13,100 audio samples, and the KSS dataset[24] consists of 12,853 audio
samples. Both datasets were recorded by a single female speaker. We used 70%
of the samples for training, and reserved the remaining portions for
validation (10%) and test (20%) for a reliable evaluation. The total length of
the training samples we used are 8.4 hours (KSS) and 16.8 hours (LJSpeech).

We conducted experiments on a computer that has a E3-1240 v3 CPU (3.40 GHz)
and a GTX-1080 GPU. We implemented the TTS models using PyTorch v1.1 compiled
from the source code. We set batch size to one, and limited the maximum length
of each mel-sequence to 200. We trained on the GPU, but measured synthesis
speed on a single CPU thread.

###  5.2 Experimental results

####  5.2.1 Depthwise separable convolution

We reduced computation by replacing convolution with depthwise separable
convolution[9]. In theoretical calculation, depthwise separable convolution[9]
significantly reduces the number of operations from 275B to 100B. However,
replacing ordinary convolutions by depthwise separable convolutions[9] rather
increased synthesis time from 6.85 seconds to 18.16 seconds.

####  5.2.2 Group highway activation

We compared the synthesis time of the three types of activation described in
section 3.1.2 as well as their output quality using EMCD. In Residual DCTTS
and Group Highway DCTTS, highway activation[13] was replaced by residual
connections[18] and group highway activation (g = 2), respectively. In theory,
the amount of computation of a convolution layer with group highway activation
is 75% of that of a convolution layer with highway activation. The actual
synthesis times are presented in Table 1. Compared with the baseline model,
Residual DCTTS almost doubled the synthesis speed, but significantly increased
EMCD. Particularly, Residual DCTTS often produced mel-spectrogram containing
the skipping and repeating problems. On the other hand, Group Highway DCTTS
exhibited synthesis time reduced by 7%. The actual improvement in synthesis
speed was not as significant as the reduction in theoretical computation. In
addition, Group Highway DCTTS reduced EMCD by 3.7% from 9.45 to 9.10 on the
LJSpeech dataset[23] and by 10.3% from 10.36 to 9.29 on the KSS dataset[24].

Fig. 1: The effect of network size reduction on synthesis time and speech
quality. The acronym GH denotes group highway activation, and the postfixes La
and Cb indicate that the number of layers and channels, respectively.

####  5.2.3 Network size reduction

Figure 1 displays the experimental results on KSS dataset[24]. As expected,
the smallest model, GH_L6_C64, was the fastest. Although it exhibited a
significantly high EMCD, we chose this model for the following experiments,
and applied quality improvement techniques.

| Weight normalization
---|---
on$\rightarrow{}$on | on$\rightarrow{}$off
LJ | GH_L6_C64 | 30.59 | 15.26
| GH_L6_C64 (10% Pr.)
---
Unrecognizable | 16.24
KSS | GH_L6_C64 | 46.92 | 9.75
| GH_L6_C64 (10% Pr.)
---
Unrecognizable | 10.69
Table 2: The effect of the weight normalization trick. The models marked as
(10% Pr.) are the models in which 10% of the convolution filters were pruned.

####  5.2.4 Network pruning and weight normalization trick

We applied network pruning[10] to the fastest GH_L6_C64 model to further
improve synthesis speed using KSS dataset [24]. After pruning 10% of the
convolution filters, the synthesis time was reduced by 18.09% from 1.05 sec.
to 0.86 sec. (Note that these EMCDs and synthesis time were averaged by the
number of test samples, which is 20% of all samples.) However, the pruned
models often produced unrecognizable outputs. We hypothesized that the reason
was the lack of network capacity caused by the overly aggressive size
reduction. To alleviate this problem, we applied a trick. To improve the speed
and stability of learning, we applied weight normalization [14] at the
beginning of training. However, after pruning, we deactivated weight
normalization to supplement network capacity and fine-tuned the reduced
network. As shown in Table 2, the weight normalization trick remarkably
decreased the EMCD on both datasets.

Additionally, we applied the same trick to the baseline model. In this case,
the trick decreased EMCD by only 1.02 and 1.44 on LJSpeech and KSS,
respectively.

####  5.2.5 Positional encoding and scheduled sampling

We applied the two fidelity improvement techniques[4, 19] described in Section
3.2. Positional encoding [4] reduced EMCD by 41.19% from 16.24 to 9.55 on the
LJSpeech dataset [23] and by 12.16% from 10.69 to 9.39 on the KSS dataset
[24]. However, scheduled sampling[19] did not lead to any improvement.

| baseline model | Fast DCTTS
---|---|---
| Text
---
Encoder
| C-128-512
---
C-512-512
12$\times$HC-512-512
|  2$\times$C-128-128
---
12$\times$RC-128-128
| Audio
---
Encoder
| C-80-256
---
2$\times$C-256-256
10$\times$HC-256-256
| C-80-64
---
5$\times$GH-64-64
| Audio
---
Decoder
| C-512-256
---
6$\times$HC-256-256
3$\times$C-256-256
C-256-80
| C-128-64
---
4$\times$GH-64-64
C-64-80
| Num. of
---
Computations
275,098,419,200 |  | 4,835,728,000
---
| Num. of Params
---
23,896,064 |  | 657,728
---
|  Syn-timeCPU
---
| 6.35 sec.
---
| 0.92 sec.
---
| EMCD
---
(LJ, KSS)
9.45, 10.36 | 9.55, 9.39
| MOS
---
(LJ, KSS)
2.42, 2.62 | 2.45, 2.74
Table 3:  Comparison of DCTTS and Fast DCTTS

###  5.3 Fast DCTTS

Table 3 summarizes our best model, Fast DCTTS, compared with the baseline
model, DCTTS. The proposed model requires only 1.6% of computations and 2.75%
of parameters compared to baseline model. Its synthesis time on a single CPU
thread was only only 0.92 second, which is 7.45x faster than the baseline
model. It is even faster than the baseline model running on a GPU.

We measured the mean opinion score (MOS) of the output speech. We used
Parallel WaveGAN [25] for the vocoder. We randomly selected 10 samples from
KSS and 10 samples from LJSpeech dataset. Then, 25 subjects rated the quality
of the synthesized speeches with a score in the range of 1 (bad) to 5
(excellent). The MOS of Fast DCTTS was 2.45 on LJSpeech and 2.74 on KSS, which
were slightly higher than the MOS 2.42 on LJSpeech and 2.62 on KSS of the
baseline model.

##  6 Conclusion

We proposed a high-fidelity neural TTS, Fast DCTTS, that converts text into
mel-spectrogram in real time on a single CPU thread. Its output quality is
comparable or even slightly improved compared with the baseline model DCTTS.
The proposed methods reduce the actual amount of computation rather than
relying on GPU-based parallel computing. Additionally, we proposed a novel
group highway convolution as well as a new evaluation metric, EMCD, to measure
the quality of the synthesized mel-spectrogram containing the skipping and
repeating errors.

##  7 Acknowledgements

This work was supported by Speech AI Lab at NCSOFT Inc., and the National
Program for Excellence in Software at Handong Global University (2017-0-00130)
funded by the Ministry of Science and ICT.

## References

  * [1] Y. Wang, R. Skerry-Ryan, D. Santon, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous,  “Tacotron: Towards end-to-end speech synthesis,”  in Proc. Interspeech, 2017, pp. 4006–4010.
  * [2] J. Shen, R. Pang, R. J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, and Y. Wu,  “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,”  in IEEE International Conference on Acoustics, Speech and Signal Processing, 2018, pp. 4779–4783.
  * [3] H. Tachibana, K. Uenoyama, and S. Aihara,  “Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention,”  in IEEE International Conference on Acoustics, Speech and Signal Processing, 2018, pp. 4784–4788.
  * [4] N. Li, S. Liu, Y. Liu, S. Zhao, M. Liu, and M. Zhou,  “Neural speech synthesis with transformer network,”  in Proc. of the AAAI Conference on Artificial Intelligence, 2018, pp. 6706–6713.
  * [5] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,  “Attention is all you need,”  in Advances in Neural Information Processing Systems (NIPS), 2017, pp. 5998–6008.
  * [6] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T. Y. Liu,  “Fastspeech: Fast, robust and controllable text to speech,”  in Advances in Neural Information Processing Systems (NIPS), 2017, pp. 3171–3180.
  * [7] Z. Zeng, J. Wang, N. Cheng, T. Xia, and J. Xiao,  “Aligntts: Efficient feed-forward text-to-speech system without explicit alignment,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 6714–6718.
  * [8] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T. Y. Liu,  “Fastspeech 2: Fast and high-quality end-to-end text-to-speech,”  in arXiv:2006.04558, 2020.
  * [9] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam,  “Mobilenets: Efficient convolutional neural networks for mobile vision applications,”  in arXiv:1704.04861, 2017.
  * [10] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf,  “Pruning filters for efficient convnets,”  in arXiv:1608.08710, 2016.
  * [11] W. Ping, K. Peng, A. Gibiansky, S. O. Arik, A. Kannan, S. Narang, J. Raiman, and J. Miller,  “Deep voice 3: Scaling text-to-speech with convolutional sequence learning,”  in International Conference on Learning Representations (ICLR), 2018, vol. 79, pp. 1094–1099.
  * [12] X. Zhang, X. Zhou, M. Lin, and J. Sun,  “Shufflenet: An extremely efficient convolutional neural network for mobile devices,”  in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 6848–6856.
  * [13] R. K. Srivastava, K. Greff, and J. Schmidhuber,  “Highway networks,”  in arXiv:1505.00387, 2015.
  * [14] T. Salimans and D. P. Kingma,  “Weight normalization: A simple reparameterization to accelerate training of deep neural networks,”  in Advances in Neural Information Processing Systems (NIPS), 2016, pp. 901–909.
  * [15] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer,  “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 0.5mb model size,”  in arXiv:1602.07360, 2016.
  * [16] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. C. Chen,  “Mobilenetv2: Inverted residuals and linear bottlenecks,”  in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4510–4520.
  * [17] S. Han, J. Pool, J. Tran, and W. J. Dally,  “Learning both weights and connections for efficient neural networks,”  in Advances in Neural Information Processing systems, 2015, pp. 1135–1143.
  * [18] K. He, X. Zhang, S. Ren, and J. Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
  * [19] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer,  “Scheduled sampling for sequence prediction with recurrent neural networks,”  in Advances in Neural Information Processing Systems (NIPS), 2015, pp. 1171–1179.
  * [20] R. Kubicheck,  “Mel-cepstral distance measure for objective speech quality assessment,”  in Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, 1993, vol. 1, pp. 125–128.
  * [21] D. J. Berndt and J. Clifford,  “Using dynamic time warping to find patterns in time series,”  in AAAI Workshop on Knowledge Discovery in Databases, 1994, vol. 10(16), pp. 359–370.
  * [22] E. Battenberg, S. Mariooryad, D. Stanton, RJ Skerry-Ryan, M. Shannon, D. Kao, and T. Bagby,  “Effective use of variational embedding capacity in expressive end-to-end speech synthesis,”  in International Conference on Learning Representations (ICLR), 2020\.
  * [23] K. Ito,  “The lj speech dataset,”  in keithito.com/LJ-Speech-Dataset, 2017.
  * [24] K. Park,  “Kss dataset: Korean single speaker speech dataset,”  in kaggle.com/bryanpark/korean-single-speaker-speech-dataset, 2019\.
  * [25] R. Yamamoto, E. Song, and J.-M. Kim,  “Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,”  in International Conference on Machine Learning(ICML), 2020, pp. 6199–6203.