# BinMLM: Binary Authorship Verification with Flow-aware Mixture-of-Shared
Language Model

Qige Song∗†, Yongzheng Zhang†, Linshu Ouyang∗†, Yige Chen∗† ∗ Institute of
Information Engineering, Chinese Academy of Science, Beijing, China  † School
of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Binary authorship analysis is a significant problem in many software
engineering applications. In this paper, we formulate a binary authorship
verification task to accurately reflect the real-world working process of
software forensic experts. It aims to determine whether an anonymous binary is
developed by a specific programmer with a small set of support samples, and
the actual developer may not belong to the known candidate set but from the
wild.

We propose an effective binary authorship verification framework, BinMLM.
BinMLM trains the RNN language model on consecutive opcode traces extracted
from the control-flow-graph (CFG) to characterize the candidate developers’
programming styles. We build a mixture-of-shared architecture with multiple
shared encoders and author-specific gate layers, which can learn the
developers’ combination preferences of universal programming patterns and
alleviate the problem of low training resources. Through an optimization
pipeline of external pre-training, joint training, and fine-tuning, our
framework can eliminate additional noise and accurately distill developers’
unique styles.

Extensive experiments show that BinMLM achieves promising results on Google
Code Jam (GCJ) and Codeforces datasets with different numbers of programmers
and supporting samples. It significantly outperforms the baselines built on
the state-of-the-art feature set (4.73% to 19.46% improvement) and remains
robust in multi-author collaboration scenarios. Furthermore, BinMLM can
perform organization-level verification on a real-world APT malware dataset,
which can provide valuable auxiliary information for exploring the group
behind the APT attack.

###### Index Terms:

binary authorship verification, language model, mixture-of-shared
architecture, deep learning

##  I Introduction

###  I-A Background and Motivation

Binary authorship analysis aims at recovering the developer’s identity
information through executable binary files. It is critical in various
software engineering applications, especially security-related scenarios, such
as copyright infringement detection, cybercrime tracking, and malicious
software (malware) forensics, where source code is rarely available. In this
paper, we formulate the problem as a practical _binary authorship verification
task_. As shown in figure 1, given an anonymous binary sample, our goal is to
determine whether it is developed by a specific programmer of the candidate
set. Each candidate programmer has a small set of binaries that can
characterize his programming style. The difficulty of the task is that the
actual developer of the questioned binary is likely not to belong to the
candidate set but from the wild, which is in line with the realistic
experiences of professional binary analysts [1].

Binary authorship verification is nontrivial as it faces two main challenges:

(a) Binary samples of the same programmer may implement relevant or completely
different functions. Prior researchers manually designed programming stylistic
features to establish the unified programmer templates. It heavily relies on
expert experience and can not capture specific programming behaviors that have
not appeared in the known samples.

Deep learning can perform automatic feature extraction and has strong
generalization capabilities. It has achieved promising results in many
software engineering applications [2] [3] [4] [5]. However, the collected
samples of candidate authors are usually very limited, making it challenging
to train the deep neural network with a large number of parameters.

(b) Modern software development is usually completed by cooperation, even
including malware production, like APT attackers usually cooperate to launch
staged events. The collaborators’ code snippets may introduce noise when
identifying the major contributor, and it is more challenging to implement
organization-level authorship verification due to the mixture of different
programmers’ styles.

Figure 1: Binary authorship verification task

###  I-B Limitation of Prior Art

Most previous binary authorship analysis approaches try to solve the problem
by traditional supervised machine learning algorithms [6] [7] [8] [9] [10].
They extract the known programmers’ stylistic features from training examples
and identify the developer of the anonymous binaries in the test stage. Table
I lists the state-of-the-art approaches. Rosenblum. et al. [6] extracted
bytecode n-gram, instruction idioms, and CFG graphlets to create author-style
templates. BinAuthor [9] constructed the authors’ choices set to recognize
their programming habits. Caliskan-Islam. et al. [10] extracted hybrid
features from disassembly instructions and decompiled code to model the
author’s style survives compilation.

The above approaches require manual feature engineering, which relies on
domain knowledge and has been shown to be dataset-specific. To automatically
extract the programmer’s style characteristics, Bineye [11] proposed a deep
learning-based authorship attribution method, which uses three convolutional
neural networks (CNNs) on gray-scale images, opcode sequences, and function
invocations. Although showing performance improvement, deep neural networks
rely on sufficient collected samples with annotations to fit the large-scale
parameters, which is usually unrealistic in real-world binary forensics
scenarios.

Furthermore, one common dilemma of most existing approaches is the inability
to handle the developers who do not belong to the candidate set. Caliskan-
Islam _et al._ [10] decided to accept or reject the results according to the
classifier’s confidence score. Rosenblum _et al._ [6] calculated the stylistic
similarities between anonymous binary pairs based on a distance metric for
authorship clustering. However, their performance is not ideal when dealing
with large size of candidates and insufficient samples. In this paper, we
consider the real-life experience of professional binary analysts and
formulate the _binary authorship verification task_. Our goal is to determine
whether a binary is developed by a candidate author with limited samples based
on the preserved programming style.

TABLE I: State-of-the-art binary authorship analysis approaches Approach |  Feature engineering |  Handling unknown programmers from the wild
---|---|---
Rosenblum _et al._ [6] |  Manual |  ✓ (by stylistic similarity)
Oba2 [7] |  Manual |  ✗
Meng _et al._ [8] |  Manual |  ✗
BinAuthor [9] |  Manual |  ✗
Caliskan-Islam _et al._ [10] |  Manual |  ✓ (by confidence score)
Bineye [11] |  Automatic |  ✗

###  I-C Proposed Method

In this paper, we propose BinMLM, a binary authorship verification framework
with flow-aware mixture-of-shared language model. It addresses the above
challenges as follows:

Specific programming languages have strict syntax restrictions. Therefore,
developers’ personal habits only account for a small percentage of the
program, while most parts are overlapping general patterns. We interpret
programmers’ personal style as the selection and combination of _universal
programming patterns_ , such as the preferences for specific programming
paradigms, branch and loop forms, organization structures of user-defined
functions, and exception handling ways. These stylistic preferences are
implicit in the instruction execution sequence.

We train RNN language models with powerful sequence modeling capabilities on
the disassembly code to automatically characterize the styles of candidate
programmers, avoiding explicit feature engineering. To recover the high-level
stylistic patterns from the binaries, we extract consecutive opcode traces as
the essential units of the language model and preserve the flow information in
the basic block bi-grams of CFG.

We introduce a _mixture-of-shared_ architecture with multiple encoders shared
among all developers to model the universal programming patterns from
different views. Each programmer is assigned a gate layer to learn the
specific combination weights of shared representations, reflecting their
preferences of multiple generic patterns. Compared to training a separate
language model for each author, this layer sharing mechanism can alleviate the
insufficient data problem by effectively utilizing the small-scale training
samples.

We utilize an _optimization pipeline_ to reduce the noise of collaborators and
further isolate the small proportion of unique styles from generic programming
patterns. We first train the overall parameters jointly. Specifically, we
randomly select an author’s mini-batches each time and optimize the shared
encoders and the corresponding author-specific layers. After that, we fix the
parameters of shared layers and separately fine-tune each author-specific
decoder with specially designed regularization terms. Joint training enables
the model to learn the general programming patterns better. Author-specific
fine-tuning facilitates the model to explore how to decode personal styles
accurately, improving the robustness of major author characterization in
multi-programmer collaboration scenarios.

###  I-D Key Contributions

We summarize our major contributions as follows:

  * •

We formulate a practical _binary authorship verification_ task, which
considers that the developers of anonymous binaries may not belong to the
candidate set but from the wild. Our task setting is in line with the working
process of software forensic experts, and we provide an effective solution for
them to perform automatic analysis.

  * •

We creatively characterize the authors’ programming styles by training the RNN
language model on flow-aware instruction execution traces, which can avoid
manual feature engineering.

  * •

We design a novel _mixture-of-shared_ language model and an effective
_optimization pipeline_. With multiple shared encoders and author-specific
gate layers, we can fully utilize limited samples and model the developers’
combination preferences of general programming patterns. Furthermore, our
_optimization pipeline_ can eliminate additional noise and accurately distill
developers’ unique stylistic characteristics.

  * •

We conduct extensive experiments to evaluate BinMLM on datasets with different
numbers of developers and samples. Results show that it can surpass baselines
built on the state-of-the-art feature set by a large margin (AUC = 0.83 $\sim$
0.94, 4.73% $\sim$ 19.46% improvement). Moreover, BinMLM remains robust in
multi-programmer collaboration scenarios and can perform practical
organization-level verification on a real-world APT malware dataset.

##  II Problem Defination

In this section, we formalize the binary authorship verification task as
follows:

We first construct a candidate set $P$ = ($p_{1}$, $p_{2}$, …, $p_{n}$) of $n$
known programmers, and each programmer $p_{i}$ has several previously
collected binary samples $B_{i}$ = ($b_{i_{1}}$, $b_{i_{2}}$, … ,$b_{i_{m}}$).
Then for an authorship verification pair $\langle p_{i},b_{j}\rangle$, we aim
to determine whether the anonymous binary $b_{j}$ is developed by the
candidate programmer $p_{i}$. Note that we assume $b_{j}$ is developed by a
single programmer or has a major contributor in multi-programmer collaboration
scenarios, and the actual developer of $b_{j}$ may belong to $P$ or may come
from the wild.

Previous binary authorship analysis methods directly adopt a close-world
classification setting, which assumes the anonymous binaries to be analyzed
are all developed by known programmers from the candidate set. It is
inconsistent with the real-world software forensic scenarios, and our proposed
binary authorship verification task can handle the problem from a more
realistic and comprehensive perspective.

##  III Preliminaries

###  III-A RNN Language Model

Language models has shown strong text style characterization abilities in
previous literature [12] [13] [14]. For an input sentence $S$ = ($u_{1}$,
$u_{2}$,…, $u_{l}$), language model quantitatively describe the joint
probability $p(S)$ as:

$p(S)=\prod_{i=1}^{l}p\left(u_{t}\mid u_{1:t-1}\right)$ (1)

$u_{i}$ denotes the statistical unit of the language model. In this paper, we
train the language model on the disassembly instruction sequence to
characterize developers’ programming styles. We use recurrent neural network
(RNN) to encode $u_{1:t-1}$ [15]. For an input sequence, we first transform
each token $u_{i}$ into a vector $\mathbf{e}_{t}$ through an embedding layer,
then feed the sequence into RNN layer to model the context representation. At
each time-step $t$ of RNN, the hidden state $\mathbf{h}_{t}$ is updated as
follows:

$\mathbf{h}_{t}=RNN\left(\mathbf{h}_{t-1},\mathbf{e}_{t-1}\right)$ (2)

After that, $\mathbf{h}_{t}$ is fed into a fully-connected layer with
_softmax_ function, which acts as a decoder to estimate the probability
distribution of the next unit:

$y\left(u_{t}\right)=f\left(\mathbf{W}\cdot\mathbf{h}_{t}+\mathbf{b}\right)$
(3)

$p\left({u_{t}}\mid
u_{1:t-1}\right)=\frac{e^{\left(y\left(u_{t}\right)\right)}}{\sum_{k=1}^{|V|}e^{\left(y\left(u_{k}\right)\right)}}$
(4)

$f$ is the nonlinear activation function, $|V|$ is the vocabulary size of the
statistical units. The loss function of the language model is defined as the
cross-entropy between the predicted distribution and the ground-truth:

$L(S)=-\sum_{t=1}^{m}u_{t}\log\left(p\left({u_{t}}\mid
u_{1:t-1}\right)\right)\vspace*{-0.3\baselineskip}$ (5)

Figure 2: Mixture-of-Shared language model architecture Figure 3: Binary
authorship verification workflow of BinMLM

###  III-B Mixture-of-Shared Network Architecture

To conduct binary authorship verification formalized in section II, we expect
to train a language model for each candidate author on his binary samples.
Each author’s programming style will lead to the differences of the learned
probability $p(S)$, and a binary has a relatively high probability on its
actual developer’s language model. Since the number of candidate programmers
can be large, setting an independent network for each author will limit the
model’s scalability. In addition, the syntax rules of programming languages
are much stricter than natural language. Developers use a fixed set of
keywords and APIs, and the assembly instructions compiled from high-level
source code have more limited operations types. Therefore, developers’
personal styles mainly come from the selection and combination of universal
programming patterns.

Based on these considerations, we exploit a mixture-of-shared network to train
the language models of candidate programmers simultaneously. Figure 2 shows
the network architecture. We set up multiple RNN encoders shared among all
programmers to map the processed samples into different subspaces, modeling
the generic programming patterns from different perspectives. For programmer
$p_{i}$, the shared encoder $j$ will generate the corresponding hidden
representation $\mathbf{h}_{j}^{(i)}$.

To extract the developer’s specific preference for the combination of common
programming patterns, we build a separate gate layer for each programmer to
learn the importance weights of shared representations generated by different
RNN encoders. Specifically, the weight parameters $\mathbf{g_{i}}$ of
programmer $p_{i}$ is calculated by a feed-forward layer as follows:

$\mathbf{g}_{i}=\sigma\left(\mathbf{W}_{g_{i}}\cdot\mathbf{E}_{i}+\mathbf{b}_{g_{i}}\right)$
(6)

$\mathbf{E}_{i}$ denotes the embedded vectors of the input sequence. We
compute the context representation $\mathbf{r}_{i}$ by the element-wise
multiplication of $\mathbf{h}_{j}^{(i)}$ with their corresponding gates
weights:

$\mathbf{r}_{i}=\sum_{j=1}^{s}\mathbf{g}_{i}\cdot\mathbf{h}_{j}^{(i)}$ (7)

$s$ is the number of shared RNN encoders. $\mathbf{r}_{i}$ is finally fed into
the corresponding author-specific linear decoder to generate predictions of
subsequent units.

Our model is inspired by the text style characterization approaches proposed
by Bagnall [12] and Ouyang _et al._ [14], and the multi-task learning network
designed by Ma _et al._ [16]. Such an architecture allows all programmers’
samples to optimize the shared layers, and in turn optimize each specific
decoder, improving the generalization ability of low-resource programmers’
models. Moreover, multiple shared encoders and author-specific gates can
flexibly combine the subspace of the universal programming patterns,
accurately characterizing the programmer’s personal style.

##  IV Proposed Method

###  IV-A Overview

The general idea of our proposed binary authorship verification framework
BinMLM is to characterize the programmer’s unique style contained in binary
files by language model and simultaneously learn specific language models of
large numbers of candidate programmers. To this end, BinMLM involves four
major modules (shown in Figure 3).

  * •

Binary pre-processing module (_Module 1_): We disassemble the input binary
file and extract the CFG structure. Then we extract the instructions traces
with control flow semantics and use the opcode n-grams as the essential units
of the subsequent programming style characterization process. This module is a
fundamental component shared by the following modules.

  * •

External pre-training module (_Module 2_): We construct a large-scale external
binary code corpus without authorship annotations and pre-process it by
_Module 1_. Then we pre-train a general language model on these wild binary
files to initialize the subsequent models.

  * •

Candidate programmer style characterization module (_Module 3_): We introduce
a mixture-of-shared architecture with multiple shared encoders and author-
specific gate layers and decoders to train the language models of all
candidate programmers. We first jointly optimize the overall parameters by
binary samples of all programmers, then freeze the shared layer and
individually fine-tune the separate decoders with additional regularization
items.

  * •

Authorship verification module (_Module 4_): For a verification pair $\langle
p_{i},b_{j}\rangle$. We pre-process binary $b_{j}$ and evaluate its loss value
$l_{i}$ on the trained language model of programmer $p_{i}$. Then we use the
loss array $\mathbf{L}^{(j)}$ of $b_{j}$ on all candidate programmers’
language models to normalize $l_{i}^{(j)}$ and obtain the final verification
score.

###  IV-B Binary File Pre-processing

Figure 4: Binary file pre-processing module

Figure 4 shows the details of our binary file pre-processing module. The major
components include processing the disassembly code and building language
models on the collected binary samples.

####  IV-B1 Process disassembly code

A binary to be analyzed is represented as discrete byte streams, and we use
disassembling tool to extract its corresponding CFG structure. The nodes in
CFG, called basic blocks, act as the smallest units of sequential instruction
executions. The edges represent the control flow transfer between basic
blocks, which may be caused by _if-else_ branches, loop structures, or jumps
across blocks.

To preserve the flow transfer semantics in CFG, we first connect the endpoint
basic blocks of all edges as basic block bi-grams. Then we extract the
instruction traces within the two basic blocks sequentially as the sequence
$I$ = ($i_{1}$, $i_{2}$, …, $i_{l}$), $l$ is the number of instructions.
Assembly instruction is composed of an opcode and zero to more operands. The
opcode specifies the operation to be conducted. The operands specify
registers, immediate literals, or memory locations, which can be different
under various compilation environments. To improve the versatility of our
framework and reduce the scale of the language model’s vocabulary, we only
keep the opcode in the instruction sequence and update the sequence to $O$ =
($o_{1}$, $o_{2}$, …, $o_{l}$), $o_{i}$ represents the opcode at position $i$.

We explore how to better characterize the developer’s programming style in the
binaries by comparing the disassembly code with the original high-level source
code. Figure 5 shows a source code snippet and part of its corresponding
assembly instruction traces. We can see that a short statement in the source
code, such as $maxAns=max(maxAns,pre)$, will be transformed into a relatively
long assembly instruction sequence, and the corresponding opcode sequence is
$MOV$, …, $CALL$, $MOV$, which implements the operations of register loading,
parameter reading, and function invocations. Therefore, consecutive short
opcode traces can potentially reflect the programming patterns of source code
statements. Since the number of the assembly instruction complied from each
statement is uncertain, we slide a window fixed to size $n$ on the original
opcode sequence to obtain $G$ = ($g_{1}$, $g_{2}$, …, $g_{l}$), $g_{i}$
denotes an opcode n-gram representing a small continuous sequence of
operations composed of $n$ instructions.

Figure 5: Source code statements and counterpart assembly instructions

####  IV-B2 Build language model for binaries

We train the RNN language model on the extracted opcode n-gram sequences of
binaries to characterize the developer’s programming style. The RNN encoding
layer extracts the context representation of the previous opcode n-gram units,
and the linear decoding layer estimates the probability distribution of the
next unit. In our experiment, we found that setting the subsequent opcode
n-gram unit of multiple hops as the prediction target of the language model
can generate more distinguishable author-specific styles. Because increasing
the task difficulty encourages the model to dig out more robust programming
patterns from the training samples.

###  IV-C External Language Model Pre-training

The collected binaries of candidate programmers are insufficient for training
RNN language models with a large number of parameters. Note that traditional
text language models are usually trained on millions of tokens, but our
collected samples may only contain a few thousand opcode n-gram units. To
tackle this problem, we first pre-train a common RNN language model on the
opcode n-gram sequences extracted from a large-scale disassembled external
binary corpus. The pre-trained model, with an opcode n-gram embedding layer,
an LSTM encoding layer, and a linear decoding layer, contains the general
programming features of these wild binary files. It will be used to initialize
the parameters of candidate programmers’ language models.

###  IV-D Candidate Programmer Style Characterization

####  IV-D1 Mixture-of-shared language model

We use the collected binaries to train an author-specific language model for
each candidate programmer. As defined in section II, there are $n$ programmers
in the candidate set $P$ = ($p_{1}$, $p_{2}$, …, $p_{n}$). Each programmer
$p_{i}$ has a set of binary samples $B_{i}$ = ($b_{i_{1}}$, $b_{i_{2}}$, … ,
$b_{i_{m}}$). We process each binary in $B_{i}$ as control-flow aware opcode
n-gram sequences, and each sequence is represented as $G_{i}$ = ($g_{i_{1}}$,
$g_{i_{2}}$,…, $g_{i_{l}}$). The processed sequences of the programmers’
binary samples are grouped into input mini-batches to train the corresponding
language models.

We utilize the mixture-of-shared architecture elaborated in section III to
train the binary code language models of all candidate programmers. We first
deploy a shared opcode n-gram embedding layer to convert the input sequence
$G_{i}$ into a vector sequence $\mathbf{E_{i}}$ = ($\mathbf{e}_{i_{1}}$,
$\mathbf{e}_{i_{2}}$, …, $\mathbf{e}_{i_{l}}$). Then we use multiple shared
RNN encoders with the same structure to encode common programming patterns
from different views. The $j$-th shared RNN encodes the serialized context
information into hidden state vectors $\mathbf{H_{i}}$ =
($\mathbf{h}_{i_{1}}^{(j)}$, $\mathbf{h}_{i_{2}}^{(j)}$, …,
$\mathbf{h}_{i_{l}}^{(j)}$). Next, for each developer, we set up a separate
gate layer to produce a weighted aggregation of hidden states generated by
multiple RNNs as Equation 7. It can learn the developers’ combination
preferences of universal programming patterns and form their personal style on
this basis. Finally, we feed the mixed context representation $\mathbf{R_{i}}$
= ($\mathbf{r}_{i_{1}}$, $\mathbf{r}_{i_{2}}$, …, $\mathbf{r}_{i_{l}}$) into
$p_{i}$ ’s independent linear decoder with the _softmax_ function to estimate
the subsequent units. The author-specific decoder is trained on the
developer’s own samples.

We initialize the parameters of all shared layers and author-specific decoders
by the language model pre-trained on the external binary code corpus. The loss
$\operatorname{L}_{p_{i}}$ of programmer $p_{i}$’s language model is the
average cross-entropy losses on the mini-batches constructed by his binary
samples. The overall loss of our mixture-of-shared language model is the
weighted sum of all programmers’ losses, $w_{i}$ denotes the weight of
$\operatorname{L}_{p_{i}}$:

$\operatorname{L}_{lm}=\sum_{i=1}^{n}w_{i}\cdot\operatorname{L}_{p_{i}}$ (8)

####  IV-D2 Optimization pipeline

The optimization pipeline of the mixture-of-shared language model is divided
into a joint training phase and a fine-tuning phase to learn generic
programming patterns and author-specific stylistic features. During the joint
training phase, we randomly select a programmer’s mini-batches as the input of
each iteration, optimizing the shared layers and the corresponding specific
gate layers and linear decoders. Then in the fine-tuning phase, we freeze the
optimized shared embedding layer and LSTM encoders, as well as the gate
layers, and individually fine-tune the decoders of each programmer.

According to our analysis, the programmer’s distinguishable unique styles only
account for a small percentage of the binary code, while most parts are common
programming patterns. Our optimization pipeline can effectively exploit the
limited training samples of each candidate to distill the author-specific
stylistic features and eliminate additional noise.

####  IV-D3 Regularization of specific-decoders

Extracting opcode n-grams will cause the vocabulary of the statistical units
to increase significantly as the value of $n$ increases, thereby enlarging the
sample space of the language model. In the fine-tuning stage, each
programmer’s decoder is only optimized by its own small size of binary
samples. To limit the parameter space of the author-specific language models,
we add regularization terms between a pair of decoders to encourage their
parameters to be more similar and prevent overfitting on the inadequate
training data.

Specifically, when fine-tuning the linear decoder of programmer $p_{i}$, we
calculate the $l_{1}$ distance among its parameters $\mathbf{W}_{p_{i}}$,
$\mathbf{b}_{p_{i}}$, and the parameters of other programmer’s decoders as the
regularization loss. We average the regularization loss of all programmers as
$\operatorname{L_{reg}}$ and add it to the loss of the overall language model
$\operatorname{L_{lm}}$. $\lambda$ is the weight coefficient of
$\operatorname{L_{reg}}$:

$\operatorname{L_{reg}}=\sum_{i=1}^{n}\sum_{j=1,j\neq
i}^{n}\left|\mathbf{W}_{p_{i}}-\mathbf{W}_{p_{j}}\right|+\left|\mathbf{b}_{p_{i}}-\mathbf{b}_{p_{j}}\right|$
(9)

$\operatorname{L}_{total}=\operatorname{L}_{lm}+\lambda\cdot\operatorname{L_{reg}}\vspace*{-0.3\baselineskip}$
(10)

###  IV-E Binary Authorship Verification

In the binary authorship verification phase, for a verification pair $\langle
p_{i},b_{j}\rangle$, our goal is to determine whether the anonymous binary
sample $b_{j}$ is developed by programmer $p_{i}$. We first process $b_{j}$ in
the same way as the binary files of the training phrase, disassemble it into a
CFG structure and extract the opcode n-gram sequences of the basic block bi-
grams. Then we feed the processed sequences into the trained mixture-of-shared
language models to get the corresponding loss array $\mathbf{L}^{(j)}$ =
[${l_{1}}^{(j)}$, ${l_{2}}^{(j)}$, …, ${l_{i}}^{(j)}$, … ${l_{n}}^{(j)}$] of
$b_{j}$ on the binary code language models of all programmers in the candidate
set $P$. We jointly determine the verification result based on the loss value
${l_{i}}^{(j)}$ of $b_{j}$ on $p_{i}$’s language model and the loss array
$\mathbf{L}^{(j)}$. If ${l_{i}}^{(j)}$ is relatively small, it is more likely
that $b_{j}$ was developed by the programmer $p_{i}$.

To be more specific, we calculate the average value
$\operatorname{Avg}({\mathbf{L}^{(j)})}$ of $\mathbf{L}^{(j)}$ and its
variance $\operatorname{Var}({\mathbf{L}^{(j)})}$ to normalize ${l_{i}}^{(j)}$
and obtain the verification score $s(i,j)$ of pair $\langle
p_{i},b_{j}\rangle$:

$s(i,j)=\frac{{l_{i}}^{(j)}-\operatorname{Avg}({\mathbf{L}^{(j)})}}{\operatorname{Var}({\mathbf{L}^{(j)}})}\vspace*{-0.3\baselineskip}$
(11)

TABLE II: Main results on different datasets

Approach |  Setting | GCJ-C (N = 50, m = 5) | GCJ-C++ (N = 100, m = 5) | Codeforces (N = 100, m = 20)
---|---|---|---|---
|  |  AUC-ROC |  AP |  AUC-ROC |  AP |  AUC-ROC |  AP
Sim-Base |  with full features |  0.8448 |  0.8656 |  0.7820 |  0.7950 |  0.7154 |  0.7209
|  w/o CFG n-grams |  0.8526 |  0.8526 |  0.7497 |  0.7596 |  0.7159 |  0.7267
|  with opcode n-grams |  0.7587 |  0.7915 |  0.7718 |  0.7882 |  0.6132 |  0.6186
|  with opcode n-grams + CFG n-grams |  0.7801 |  0.7979 |  0.7801 |  0.7979 |  0.6239 |  0.6264
Con-Base |  with feature selection |  0.7702 |  0.7719 |  0.6011 |  0.6014 |  0.6184 |  0.6155
|  w/o feature selection |  0.7623 |  0.6249 |  0.6120 |  0.6153 |  0.7613 |  0.6209
BinMLM |  0.8929 |  0.9135 |  0.8414 |  0.8588 |  0.8552 |  0.8640
$\varDelta$ to the best results of baselines |  \+ 4.73% |  \+ 5.53% |  \+ 7.59% |  \+ 7.63% |  \+ 19.46% |  \+ 18.89%

  * *

GCJ is the abbreviation of Google Code Jam dataset. $N$ denotes the number of
candidate programmers, $m$ is the number of support samples. The underlines
indicate the best results of baselines. Bold indicates the overall best
results.

##  V Evaluation

In this section, we conduct extensive experiments to evaluate our proposed
binary authorship verification framework, BinMLM. First, we describe the
dataset and evaluation metrics of our experiments (section V.A). Next, we
compare BinMLM with baselines built on the state-of-the-art feature set
(section V.B.1) and evaluate BinMLM on datasets of different numbers of
candidate programmers (section V.B.2) and different numbers of collected
samples (section V.B.3). Then we evaluate how the designed core components
contribute to performance improvements (section V.C). Furthermore, we
construct synthetic datasets with different noise proportions to evaluate
BinMLM in multi-programmer collaboration scenarios (section V.D). Finally, we
explore the organization-level authorship verification ability of BinMLM on a
real-world APT malware dataset (section V.E).

Figure 6: Results under under different numbers of programmers

###  V-A Experiment Setup

####  V-A1 Datasets

We use the following data sources for evaluation: (a) Google Code Jam (GCJ):
GCJ is an annual programming competition organized by Google [17]. We collect
participants’ solutions written in C and C++ language from 2008 to 2020. (b)
Codeforces: We collect open submissions written in C++ from the competitive
programming contest hosted by the online judge website Codeforces [18]. (c)
APT-Malware: We collect malware samples of ten real-world APT groups from open
source threat intelligence reports to evaluate BinMLM in security-related
authorship verification scenarios.

For each data source, we construct the corresponding dataset according to the
following rules: We randomly select $N$ programmers to construct the candidate
set $P$. Each programmer has a small set of $m$ binaries as training samples
with known annotations. In the verification phase, we sample anonymous
binaries to construct authorship verification pairs. For a positive pair, we
randomly select an unseen binary sample of the candidate programmer. For
negative pairs, half of the test samples are developed by other $N-1$
programmers in the candidate set, and the others by randomly sampled remaining
developers who do not belong to this set. The ratio of positive and negative
verification pairs is 1:1.

####  V-A2 Evaluation metrics

We use two complementary metrics to evaluate BinMLM: AUC-ROC (Area under ROC
curve) and AP (average precision). These two indicators can evaluate the
performance of the binary authorship verification approaches without selecting
specific thresholds.

####  V-A3 Implementation Details

We implement our prototype with PyTorch framework and use dissembler $radare2$
[19] to extract the CFG and instruction sequences in the corresponding basic
blocks. The parameters of our neural network is optimized by Adam [20] with a
learning rate of 1e-2, and we truncated the gradient of the language model for
20 time-steps. The hidden dimension of the embedding layer and LSTM encoder is
set to 64. The number of the shared encoders is set to 5. The weight of the
regularization loss is set to 1e-4. Under this hyperparameter setting, BinMLM
can achieve best results on our development sets.

Figure 7: Results under different numbers of binary sapmles

###  V-B Main results on different datasets

####  V-B1 Comparision with Baselines

Among the six existing binary authorship attribution approaches listed in
table I, only [6] and [10] can deal with programmers from the wild by
stylistic similarity and classifier confidence score, respectively. We build
our baselines based on these two implementation ways and conduct evaluations
on the binary authorship verification task. We use the state-of-the-art
feature set proposed by Caliskan-Islam _et al._ [10]. Specifically, they
extracted: (a) Instruction traces of disassembly code, including instruction
uni-grams, bi-grams, and tri-grams in a single line of assembly, and 6-grams
spanning two consecutive assembly lines. (b) Uni-grams and bi-grams of basic
blocks extracted from the CFG of the binary sapmle. (c) Abstract syntax tree
(AST) node features extracted from the decompiled code.

Considering the time cost and the unstable success rate in real-world binary
decompilation, we use the first two types of feature sets above, which can be
easily constructed from disassembly code and shown to be important according
to the feature selection results of the original paper.

We construct two groups of baselines based on the feature set. The first (Sim-
Base) is built on similarity measures. For a known programmer $p_{i}$, we
construct the feature vector of each collected sample and take the average as
the programmer’s stylistic representation. For an anonymous sample $b_{j}$, we
build the feature vector in the same way and calculate the verification scores
of the pair $\langle p_{i},b_{j}\rangle$ by the cosine similarity between the
corresponding vectors. We also use our proposed flow-aware opcode n-grams to
construct variant feature sets.

The second group of baselines (Con-Base) is implemented based on the
experiment designed by Caliskan-Islam _et al._ [10] for the open-world
scenario, in which the author of the test binary sample may not belong to the
known programmers set. It exploits the confidence score of the classifier to
determine whether to accept or reject the result. For a verification pair
$\langle p_{i},b_{j}\rangle$, we take the percentage of trees in the random
forest that voted for $p_{i}$ when classifying $b_{j}$ as the confidence
score, and then we use the normalized margin of the highest and second-highest
confidences as the final verification score. Our implementation way is
consistent with the original paper.

Table II shows the binary authorship verification performance of Sim-Base,
Con-Base, and BinMLM. We evaluate Sim-Base with different features combination
ways and Con-Base with or without feature selection for comprehensive
comparisons. Overall, the performance of Sim-Base is better than Con-Base, and
BinMLM significantly surpasses them with 4.73% $\sim$ 19.46% improvement.

####  V-B2 Performance under different numbers of programmers

In this section, we evaluate the performance of BinMLM with the different
candidate set sizes. Increasing the number of candidate programmers will test
BinMLM’s ability to characterize diverse developers’ styles. Figure 6 shows
the comparison of BinMLM and the two baselines Con-Base and Sim-Base. We
gradually increase the number of candidates from 50 to 500 on GCJ-C, GCJ-C++,
and Codeforces datasets.

It can be seen that when the number of candidate programmers increases, the
performance of Con-Base implemented by the classifier confidence score
decreases significantly on GCJ-C++ and Codeforces datasets, while BinMLM and
Sim-Base remain relatively stable. When the number of programmers is in the
range of 300 to 500, the AUC-ROC of Con-Base may be lower than 0.6, indicating
a poor authorship verification ability. The AUC-ROC of Sim-Base ranges from
0.7231 to 0.8821. BinMLM significantly outperforms the two baselines under
each setting. When the candidate set size is 500, the AUC-ROC values of BinMLM
on the three datasets reach 0.9098, 0.8610, and 0.8279, respectively. This
proves that our proposed author-specific programming style characterization
method has strong versatility and can effectively perform the binary
authorship verification for large-scale candidate programmers.

####  V-B3 Performance under different numbers of binary sapmles

In real-world software forensics scenarios, especially security-related
applications, the collected binaries are often very insufficient, and
annotating the ground-truth is highly dependent on expert knowledge. Limited
samples can not adequately reflect the author’s programming behavior when
implementing different functions. We evaluate BinMLM and the baselines with
different numbers of support samples per candidate author. Similar to [21], we
merge the GCJ solutions with the same and complex participant ID from
different years to increase the sample scales for comprehensive evaluations.
Figure 7 shows the comparison results. It can be seen that with the increase
of the sample size, the performance of BinMLM has been steadily improved
because the shared encoders and author-specific layers of the mixture-of-
shared architecture can be trained better, and each developer’s style can be
extracted more accurately. However, more samples do not benefit Con-Base and
Sim-Base as much because the diversified functions of each author’s programs
may lead Con-Base’s classifier to be confused and cause the similarity between
the candidate author’s profile and the test sample features unstable.

When the number of samples is very small, BinMLM still has significant
advantages over Con-Base and Sim-Base. With only five support binaries, the
AUC-ROC of BinMLM on the three datasets are 0.8929, 0.8341, and 0.7578,
respectively. On Codeforces dataset with the more challenging setting of 100
authors and five support samples, the accuracy of BinMLM drops slightly, but
still surpasses Con-Base and Sim-Base by a large margin. It concludes that
BinMLM can accurately extract the programming styles of candidate authors with
very limited binary samples.

TABLE III: Results of different opcode n-grams and different hops. Setting |  GCJ-C |  GCJ-C++ |  Codeforces
---|---|---|---
Opcode 1-gram |  0.8216 |  0.6935 |  0.7857
Opcode 2-gram |  0.8592 |  0.7529 |  0.8290
Opcode 3-gram |  0.8671 |  0.8053 |  0.8519
Opcode 4-gram |  0.8816 |  0.8235 |  0.8561
Predict target units of 1-hop |  0.8826 |  0.8204 |  0.8429
Predict target units of 2-hops |  0.8869 |  0.8285 |  0.8514
BinMLM (5-gram, 3-hops) |  0.8929 |  0.8414 |  0.8552
TABLE IV: Effectiveness of each component in BinMLM framework Approach | GCJ-C | GCJ-C++ | Codeforces
---|---|---|---
|  AUC-ROC |  AP |  AUC-ROC |  AP |  AUC-ROC |  AP
_MoS + OPT + REG_ (BinMLM) |  0.8929 |  0.9135 |  0.8414 |  0.8588 |  0.8552 |  0.8640
_MoS + OPT_ |  0.8632 |  0.8856 |  0.7682 |  0.7670 |  0.8451 |  0.8523
_MoS_ |  0.8703 |  0.8901 |  0.7682 |  0.7666 |  0.8421 |  0.8495
_Single-encoder_ |  0.8578 |  0.8809 |  0.7528 |  0.7483 |  0.8380 |  0.8453
_Naive_ |  0.8492 |  0.8760 |  0.7357 |  0.7397 |  0.8273 |  0.8309
TABLE V: Results on multi-programmer collaboration datasets Major Proportion | GCJ-C (N = 50, m = 8) | GCJ-C++ (N = 100, m = 10) | Codeforces (N = 100, m = 30)
---|---|---|---
|  Con-Base |  Sim-Base |  BinMLM |  Con-Base |  Sim-Base |  BinMLM |  Con-Base |  Sim-Base |  BinMLM
1.0 |  0.8100 |  0.8569 |  0.9201 |  0.6416 |  0.7779 |  0.8652 |  0.6442 |  0.7360 |  0.8612
0.9 |  0.7786 |  0.8497 |  0.9020 |  0.6347 |  0.7778 |  0.8419 |  0.6241 |  0.6347 |  0.8376
0.8 |  0.7769 |  0.8621 |  0.9088 |  0.6063 |  0.7642 |  0.8344 |  0.6083 |  0.7090 |  0.8310
0.7 |  0.7677 |  0.8264 |  0.9049 |  0.6013 |  0.7628 |  0.8210 |  0.6170 |  0.6938 |  0.8324
0.6 |  0.7455 |  0.7949 |  0.8736 |  0.6051 |  0.7549 |  0.8346 |  0.5972 |  0.6873 |  0.8048

###  V-C Ablation studies of BinMLM

We design three groups of ablation studies to evaluate the effects of BinMLM’s
core components. The dataset settings of our ablation studies are the same as
section V.B.1.

The first group of ablation studies compare opcode n-grams with different
values of $n$, and results in table III show that the performance is
positively correlated with the value of $n$ in the range of 1 to 5. When the
$n$ exceeds 5, we still observe a slight improvement, but we do not increase
it further due to limited time and computing resources.

The second group of ablation studies evaluate the performance when predicting
subsequent units of different hops. As shown in table III, predicting target
units of more hops can improve the verification performance. We finally set
$n$ to 5 and the number of hops to 3.

The third group of ablation studies evaluate the contribution of each part of
our model. We set up five variant models for comparison: (a) _MoS (mixture-of-
shared architecture) + OPT + REG_ (original BinMLM): The _MoS_ architecture of
BinMLM contains multiple shared encoders to extract common programming
patterns from different views, and a separate gate layer for each programmer
to learn the combination weights of shared representations. We train _MoS_
with the optimization pipeline combining joint-training and author-specific
fine-tuning. Then we add the regularization loss in the fine-tuning stage to
encourage the parameters of specific linear decoders to be more similar and
prevent overfitting. (b) _MoS + OPT_ : Remove the regularization loss in the
fine-tuning stage. (c) _MoS_ : Remove the optimization pipeline of BinMLM. (d)
_Single-encoder_ : Set up a single RNN encoder shared among all programmers.
(e) _Naive_ architecture: Train a separate RNN language model for each
programmer.

Table IV shows the performance of BinMLM and its variant models on the three
datasets. The results prove that the mixture-of-shared architecture,
optimization pipeline, and specially designed regularization loss
significantly improve BinMLM on the binary authorship verification task.

###  V-D Robutness on multi-programmer collaboration datasets

Modern software is usually developed through cooperation. Code snippets of
collaborators will introduce noise when characterizing the major developer’s
style. We construct synthetic datasets to evaluate BinMLM in multi-programmer
collaboration scenarios. Gong _et al._ [22] conducted an empirical study on
open source software projects. They found that for most program files, about
80% of their code lines are contributed by a single programmer. So we assume
each program is completed by a major contributor and several other developers.
For a verification pair $\langle p_{i},b_{j}\rangle$, we randomly select two
other programmers as collaborators and mix their program fragments into the
sample $b_{j}$. We adjust the proportion of the collaborator’s fragments to
control the difficulty of the task, and we ensure that the proportion of the
program developed by the original author of $b_{j}$ remains above 60%.

Table V shows the results on the multi-programmer collaborative datasets. With
the increase of the proportion of collaborators’ programs, the difficulty of
modeling the style of the major developer has increased, and the performance
of the three approaches has decreased correspondingly. But under each setting,
BinMLM still significantly outperforms the baselines based on similarity
metrics and classifier confidence scores. The AUC-RO of BinMLM remains above
0.8, and the decline degree ranges from 0.0306 to 0.0564. It proves that our
designed programmer’s style characterizing method based on the mixture-of-
shared language models and specific optimization pipeline performs more robust
when dealing with the mixed programming style of collaborators.

###  V-E Case study in a real-world scenario

In this section, we provide case study to demonstrate the effectiveness of
BinMLM in real-world scenarios. Binary analysts care about the common coding
habits of authors in a specific organization, but existing work mainly
conducts manual analysis of individual cases [1] [9]. We perform systematic
experiments on the authorship verification of malware samples produced by
specific APT (Advanced Persistent Threat) attack groups. APT attacks target
particular companies or nations for political or commercial motivations. The
malware samples adopt in the attack are likely to contain the identity
information of the hacker team.

We collect malware from public threat intelligence sources and construct the
APT-Malware dataset. The malware comes from ten famous APT events includes
_Equation Group_ , _Gorgon Group_ , _DarkHotel_ , _Energetic Bear_ , _Winnti_
, _APT 10_ , _APT 21_ , _APT 28_ , _APT 29_ , and _APT 30_. Compared with the
GCJ and Codeforces datasets, the binary samples of the APT-Malware dataset are
much more complicated, and the label granularity is coarser, reflecting that
the samples were developed by multiple collaborators in specific
organizations.

To perform staged attacks, malware belonging to the same APT group may
implement very different functions, and some samples are intentionally
obfuscated to prevent detection by anti-virus products, which makes it more
difficult to locate the attacker’s identity. We randomly select five support
samples for each attack group to train the APT-specific language models.
Considering data imbalance in real-world scenarios, where the proportion of
negative verification pairs should be relatively larger, we set the ratio of
the negative pairs to positive pairs in the range of 1 to 8. Half of the test
samples in negative pairs are from the benign developers of the GCJ dataset.

Figure 8 shows the authorship verification performance of BinMLM on the APT-
Malware dataset under different negative to positive ratios. We can see that
the AUC-ROC value of BinMLM is relatively stable and remains in the range of
0.7736 to 0.8172, while as the proportion of negative verification pairs
increases, the AP value gradually decreases since it is more sensitive to the
skewed datasets. When the ratio of the positive and negative pairs is 1:5, the
AUC-ROC and AP of BinMLM are 0.8172 and 0.7617, respectively. Overall, BinMLM
can extract programming style information related to identity characteristics
from APT groups’ malware samples and achieve organization-level authorship
verification with decent performance, providing valuable auxiliary evidence
for identifying the organization behind the APT event.

Figure 8: Results on the APT-Malware dataset

##  VI Related Work

###  VI-A Source code authorship analysis

Source code contains rich style characteristics which can be very helpful in
software forensics, such as plagiarism detection and copyright investigation
[23] [24] [25] [26] [27] [28] [22] [29]. For source code de-anonymizing,
Caliskan-Islam _et al._ [25] constructed a dynamic code stylometry feature set
including lexical, layout, and syntactic features. Abuhamad _et al._ [21]
selected code n-grams with high TF-IDF scores and fed them into the RNN
network to extract the deep representations. Alsulami _et al._ [26] converted
the AST into multiple subtrees and applied hierarchical bi-LSTM to extract the
syntax features. Bogomolov _et al._ [28] designed two language-agnostic models
work with path-based code fragment representations. Some researchers performed
authorship identification on Android platform [30] [31] [29]. Wang [29]
divided packages of Android applications into modules and built the author’s
coding habit fingerprint of the main module.

###  VI-B Binary authorship analysis

Authorship analysis targeting binaries is significant, because analysts cannot
obtain source code in many real-world scenarios [1] [6] [7] [9] [11].
Rosenblum _et al._ [6] built author stylistic templates on features like
instruction idioms and CFG graphlets. Alrabaee _et al._ [9] first filtered out
irrelevant compiler functions, then characterized the author’s habits
contained in user-defined functions by programming choices. Caliskan-Islam _et
al._ [10] extracted multi-source style features from disassembly code and the
AST of decompiled code. Meng _et al._ [8] conducted empirical analysis on open
source software and concluded that binary authorship attribution should be
performed at the basic block granularity. The manual feature engineering
processes of the above approaches are dataset-dependent and rely on domain
knowledge. BinEye [11] proposed an intelligent deep learning-based binary
authorship attribution model. It applied three CNNs on binary gray-scale
images, API call sequence, and opcode sequences.

###  VI-C Text authorship analysis

Text authorship analysis has a more extended research history [12] [14] [32]
[33] [34]. Our task setting is inspired by the open-world text stylometry
problems defined by Stolerman _et al._ [35] and the shared PAN series of
competitions [36]. For text authorship verification, Bagnall [12] designed a
char-level language model with a single encoder and independent softmax groups
to model the authors’ writing styles, which won first place in the PAN-2015
competition. Ouyang _et al._ [14] found that the POS (Part of Speech)-level
language model can better characterize authors’ syntactic styles. They
designed a gated unit in the decoding stage to integrate common writing
patterns and specific styles.

##  VII Conclusion

In this paper, we formulate a practical binary authorship verification task,
which can handle binary samples from unknown programmers and accurately
reflect the real-life experiences of software forensic experts. We implement a
binary authorship verification framework, BinMLM. It trains author-specific
language models on the flow-aware opcode n-gram sequences to automatically
characterize the developer’s programming styles. We exploit a mixture-of-
shared architecture to fully use the limit training samples and model the
developer’s combination preference of multiple universal programming patterns.
Through an effective optimization pipeline, BinMLM can separate the
programmer’s unique style from a large portion of general patterns. Extensive
experiments show that BinMLM outperforms baselines built on the state-of-the-
art feature set by a large margin and remains robust in multi-author
collaboration scenarios and organization-level verification on real-world APT
malware datasets.

## Acknowledgment

The authors would like to thank the anonymous reviewers for their insightful
comments. This work was supported by the National Natural Science Foundation
of China under Grant U1736218. The corresponding author is Yongzheng Zhang.

## References

  * [1] M. Marquis-Boire, M. Marschalek, and C. Guarnieri, “Big game hunting: The peculiarities in nation-state malware research,” _Black Hat, Las Vegas, NV, USA_ , 2015.
  * [2] W. Wang, G. Li, B. Ma, X. Xia, and Z. Jin, “Detecting code clones with graph neural network and flow-augmented abstract syntax tree,” in _2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 2020, pp. 261–271.
  * [3] H. Tian, K. Liu, A. K. Kaboré, A. Koyuncu, L. Li, J. Klein, and T. F. Bissyandé, “Evaluating representation learning of code changes for predicting patch correctness in program repair,” in _2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2020, pp. 981–992.
  * [4] D. Wang, Y. Yu, S. Li, W. Dong, J. Wang, and L. Qing, “Mulcode: A multi-task learning approach for source code understanding,” in _2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 2021, pp. 48–59.
  * [5] N. D. Bui, Y. Yu, and L. Jiang, “Infercode: Self-supervised learning of code representations by predicting subtrees,” in _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 2021, pp. 1186–1197.
  * [6] N. Rosenblum, X. Zhu, and B. P. Miller, “Who wrote this code? identifying the authors of program binaries,” in _European Symposium on Research in Computer Security_. Springer, 2011, pp. 172–189.
  * [7] S. Alrabaee, N. Saleem, S. Preda, L. Wang, and M. Debbabi, “Oba2: An onion approach to binary code authorship attribution,” _Digital Investigation_ , vol. 11, pp. S94–S103, 2014.
  * [8] X. Meng, “Fine-grained binary code authorship identification,” in _Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering_ , 2016, pp. 1097–1099.
  * [9] S. Alrabaee, P. Shirani, L. Wang, M. Debbabi, and A. Hanna, “On leveraging coding habits for effective binary authorship attribution,” in _European Symposium on Research in Computer Security_. Springer, 2018, pp. 26–47.
  * [10] A. Caliskan-Islam, F. Yamaguchi, E. Dauber, R. Harang, K. Rieck, R. Greenstadt, and A. Narayanan, “When coding style survives compilation: De-anonymizing programmers from executable binaries,” in _25th Annual Network and Distributed System Security Symposium, NDSS 18_. The Internet Society, 2018.
  * [11] S. Alrabaee, E. B. Karbab, L. Wang, and M. Debbabi, “Bineye: Towards efficient binary authorship characterization using deep learning,” in _European Symposium on Research in Computer Security_. Springer, 2019, pp. 47–67.
  * [12] D. Bagnall, “Author identification using multi-headed recurrent neural networks,” _arXiv preprint arXiv:1506.04891_ , 2015.
  * [13] Z. Ge, Y. Sun, and M. Smith, “Authorship attribution using a neural network language model,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 30, no. 1, 2016.
  * [14] L. Ouyang, Y. Zhang, H. Liu, Y. Chen, and Y. Wang, “Gated pos-level language model for authorship verification.” in _IJCAI_ , 2020, pp. 4025–4031.
  * [15] T. Mikolov, M. Karafiát, L. Burget, J. Černockỳ, and S. Khudanpur, “Recurrent neural network based language model,” in _Eleventh annual conference of the international speech communication association_ , 2010.
  * [16] J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi, “Modeling task relationships in multi-task learning with multi-gate mixture-of-experts,” in _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 1930–1939.
  * [17] http://code.google.com/codejam/.
  * [18] http://codeforces.com.
  * [19] https://www.radare.org/.
  * [20] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [21] M. Abuhamad, T. AbuHmed, A. Mohaisen, and D. Nyang, “Large-scale and language-oblivious code authorship identification,” in _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ , 2018, pp. 101–114.
  * [22] S. Gong and H. Zhong, “Code authors hidden in file revision histories: An empirical study,” in _2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC)_. IEEE, 2021, pp. 71–82.
  * [23] X. Yang, G. Xu, Q. Li, Y. Guo, and M. Zhang, “Authorship attribution of source code by using back propagation neural network based on particle swarm optimization,” _PloS one_ , vol. 12, no. 11, p. e0187204, 2017.
  * [24] S. Burrows, A. L. Uitdenbogerd, and A. Turpin, “Comparing techniques for authorship attribution of source code,” _Software: Practice and Experience_ , vol. 44, no. 1, pp. 1–32, 2014.
  * [25] A. Caliskan-Islam, R. Harang, A. Liu, A. Narayanan, C. Voss, F. Yamaguchi, and R. Greenstadt, “De-anonymizing programmers via code stylometry,” in _24th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 15)_, 2015, pp. 255–270.
  * [26] B. Alsulami, E. Dauber, R. Harang, S. Mancoridis, and R. Greenstadt, “Source code authorship attribution using long short-term memory based networks,” in _European Symposium on Research in Computer Security_. Springer, 2017, pp. 65–82.
  * [27] H. J. Kang, T. F. Bissyandé, and D. Lo, “Assessing the generalizability of code2vec token embeddings,” in _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2019, pp. 1–12.
  * [28] E. Bogomolov, V. Kovalenko, Y. Rebryk, A. Bacchelli, and T. Bryksin, “Authorship attribution of source code: A language-agnostic approach and applicability in software engineering,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2021, pp. 932–944.
  * [29] W. Wang, G. Meng, H. Wang, K. Chen, W. Ge, and X. Li, “A 3 ident: A two-phased approach to identify the leading authors of android apps,” in _2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, pp. 617–628.
  * [30] V. Kalgutkar, N. Stakhanova, P. Cook, and A. Matyukhina, “Android authorship attribution through string analysis,” in _Proceedings of the 13th International Conference on Availability, Reliability and Security_ , 2018, pp. 1–10.
  * [31] H. Gonzalez, N. Stakhanova, and A. A. Ghorbani, “Authorship attribution of android apps,” in _Proceedings of the Eighth ACM Conference on Data and Application Security and Privacy_ , 2018, pp. 277–286.
  * [32] V. Kalgutkar, R. Kaur, H. Gonzalez, N. Stakhanova, and A. Matyukhina, “Code authorship attribution: Methods and challenges,” _ACM Computing Surveys (CSUR)_ , vol. 52, no. 1, pp. 1–36, 2019.
  * [33] S. H. Ding, B. C. Fung, F. Iqbal, and W. K. Cheung, “Learning stylometric representations for authorship analysis,” _IEEE transactions on cybernetics_ , vol. 49, no. 1, pp. 107–121, 2017.
  * [34] J. Bevendorff, M. Hagen, B. Stein, and M. Potthast, “Bias analysis and mitigation in the evaluation of authorship verification,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019, pp. 6301–6306.
  * [35] A. Stolerman, R. Overdorf, S. Afroz, and R. Greenstadt, “Classify, but verify: Breaking the closed-world assumption in stylometric authorship attribution,” in _IFIP Working Group_ , vol. 11, 2013, p. 64.
  * [36] https://pan.webis.de.