highlights the competitive landscape of open-source LLMs.

The lowest-tier performers include models such as Phi-3-mini-128k-instruct by
Microsoft and Stablelm-2-1_6b by Stability AI. Phi-3-mini-128k-instruct has
the lowest performance, with only 1.25% accuracy on the 80-question set and
0.88% on the 10,000-question set. Despite being from a major company like
Microsoft and licensed under MIT, this model underscores the importance of
continuous development and optimization in LLMs. Stablelm-2-1_6b scores 16.25%
on the 80-question set, decreasing to 20.09% on the 10,000-question set,
demonstrating smaller models’ difficulties in scaling up effectively.

In conclusion, the table reveals that proprietary models perform better than
open-source models, suggesting that controlled environments and dedicated
resources may significantly enhance model performance. However, larger models
do not always guarantee higher performance, as seen with some mid and lower-
tier performers. Additionally, many models show a decline in accuracy as the
number of questions increases, highlighting the challenges in maintaining
performance consistency across larger datasets. The analysis indicates that
while top-tier proprietary models lead in performance, there is significant
potential within the open-source community to develop competitive models.
Continuous improvements in model architecture, training data quality, and
optimization techniques are crucial for advancing state-of-the-art
cybersecurity knowledge within LLMs.

##  VI Code-specific LLMs

The rapid evolution of technology and software development has increased the
demand for specialized tools that aid in coding, debugging, and enhancing
software security [115, 116]. Recognizing this need, various organizations
have developed Code-specific LLMs, each offering unique features and
capabilities. These models leverage advanced machine learning techniques to
understand, generate, and manipulate code, thereby revolutionizing the field
of software development [117, 118]. This section delves into several notable
Code-specific LLMs, exploring their architectures, training methods, and
potential applications in cybersecurity and beyond [119, 120, 121, 122]. Table
IX and Table X compare Code-specific Large Language Models.

###  VI-A Prevalent LLMs

####  VI-A1 SantaCoder

As part of the BigCode project, HuggingFace and ServiceNow have proposed
SantaCoder LLM [123]. Based on the decoder-only architecture and with a 1.1B
parameter, SantaCoder was trained on 268GB of Python, Java, and JavaScript
subsets of The Stack dataset. Multiple filtering techniques were used for the
training data without much impact except for one (i.e., filtering files from
repositories with 5+ GitHub stars), significantly deteriorating the
performance on text2code benchmarks. Pre-training methods included Multi-
Query-Attention (MQA) and Fill-in-the-Middle (FIM). Although these techniques
have led to a slight drop in the model’s performance compared to Multi-Head-
Attention (MHA) and training without FIM, the model could still outperform
previous multi-lingual code models like CodeGen0Multi-2.7B and InCoder-6.7B
despite being substantially smaller. Such performance can be promising if
deployed in cybersecurity for tasks like software vulnerability and secure
code generation.

####  VI-A2 StarCoder

StarCoder is another decoder-only model developed within the BigCode project
[124]. With 15.5B parameters, StarCoder was pre-trained on 1000B tokens from
over 80 different programming languages. The pre-training utilized techniques
such as FIM and MQA and Learned Absolute Positional Embeddings. After pre-
training, the base model was fine-tuned on an additional 35B tokens of Python.
Compared to other Code LLMs, StarCoder outperformed all fine-tuning models on
Python. Moreover, the base model outperformed OpenAI code-cushman-001.
StarCoder’s exceptional performance in Python and its broad training in
multiple programming languages position it as a highly versatile tool for
various coding tasks.

####  VI-A3 StarChat-Alpha

StarChat Alpha is a variant of StarCoder fine-tuned to act as a helpful coding
assistant that accepts natural language prompting (considering that StarCoder
needs specific structured prompting) [125]. With 16B parameters, the model was
fine-tuned on a mixture of oasst1 and databricks-dolly-15k datasets. The model
has not undergone RLHF or similar methods, which would have helped align it
with human preferences. Nevertheless, the comprehensive pre-training of the
base model contributed to the model’s ability to interpret various coding
tasks and provide accurate code suggestions. This capability makes it an
invaluable programming tool, simplifying code development and problem-solving.

####  VI-A4 CodeGen-2

Developed by Salesforce AI research, CodeGen2 was proposed as a product of
extensive research in the field of LLM aimed at optimizing model architectures
and learning algorithms to enhance the efficiency and reduce the costs
associated with LLMs [126]. The final findings were examined in multiple
variants with parameters ranging from 1B to 16B, where the 16B model is
trained on 400B tokens from the Stack dataset. Causal language modeling,
cross-entropy loss, and other techniques were used for pre-training, resulting
in a robust program synthesis model. CodeGen2’s proficiency in program
synthesis makes it a valuable asset in cybersecurity applications, such as
aiding in vulnerability detection and enhancing code security analysis. Its
ability to understand and generate complex small models can be trained for
multiple epochs with specific settings, efficient security protocols, and
automated threat detection systems.

####  VI-A5 CodeGen-2.5

Another version of the CodeGen family is CodeGen 2.5 [127]. The 7B parameters
model was introduced to prove that good models don’t necessarily have to be
big, especially with the trend of scaling up LLMs and the data size
limitations. CodeGen 2.5 was trained on 1400B training tokens from
StarCoderData. A strategic selection of pre-training techniques, such as Flash
Attention, Infill Sampling, and Span Corruption, enhanced the model’s
performance. Moreover, that led to a good performance that is on par with
popular LLMs of larger size. The results indicated that small models can be
trained for multiple epochs with specific settings and achieve comparable
results to bigger models.

####  VI-A6 CodeT5+

CodeT5+ is an encoder-decoder transformer proposed by Salesforce AI Research
to address some code LLMs limitations [128]. Specifically, those related to
the architecture being either inflexible or serving as a single system and
pre-training task limitations related to a limited set of pre-training
objectives can result in a substantial degradation in performance. The
proposed model has different variants ranging from 220M to 16B parameters.
Trained on 51.5B tokens from CodeSearchNet and GitHub code datasets using
techniques like span denoising, contrastive learning, and others, the model
achieved new state-of-the-art results on various code-related tasks like code
generation, code completion, etc. A model with such capabilities can be
valuable to cybersecurity for threat intelligence and software vulnerability.

####  VI-A7 XGen-7B

Another production of Salesforce AI Research is XGen-7B LLM, a decoder-only
transformer with 7B parameters [129]. The model was developed to address the
problem of sequence length constraints in the available open-source LLMs as
many tasks require inference over an input context. XGen-7B, with up to 8K
sequence length, was trained on 1500B tokens from a mixture of text and code
data. Techniques like standard dense attention and a two-stage training
strategy were utilized for pre-training. Additionally, the model was enhanced
with instructional tuning, a technique that refines its responses to align
closely with specific user instructions. As a result, XGen-7B achieved
comparable or better results than other 7B state-of-the-art open-source LLMs.

####  VI-A8 Replit code v1

Proposed by Replit, Inc., the 2.7B parameters causal language model Replit-
code-v1-3b, with a focus on code completion, was trained on 525B tokens from a
subset of the stack Dedup v1.2 dataset [130]. The model underwent advanced
pre-training techniques such as Flash Attention for efficient computation,
AliBi positional embeddings for enhanced context interpretation, and the LionW
optimizer for improved training dynamics. The Replit code v1 model is also
available in two quantization options: 8-bit and 4-bit. The Replit-code-v1-3b
model’s capabilities in understanding and generating code make it particularly
suited for cybersecurity applications, such as automating the detection of
code vulnerabilities and generating secure coding patterns. Additionally, its
quantized versions can be utilized for edge security.

####  VI-A9 DeciCoder-1B

DeciCoder-1B is an open-source 1B parameter decoder-only transformer developed
by Deci AI with a 2048-context window [131]. Subsets of Python, Java, and
JavaScript from the StarCoderData dataset were used for training. The model
architecture was built using Automated Neural Architecture Construction
(AutoNAC) developed by the company, which is a technology designed to
automatically create and optimize deep learning models, particularly neural
networks, for specific tasks and hardware environments. Moreover, Grouped
Query Attention (GQA) and FIM were utilized to pre-train the model.
Consequently, the model has shown smaller memory usage compared to popular
code LLMs like StarCoder and outperformed SantaCoder in the languages it was
trained on with remarkable inference speed.

####  VI-A10 CodeLLAMA

Based on LLAMA 2, CodeLLAMA was introduced by Meta as a decoder-only
transformer code LLM [132]. With variants ranging from 7 to 34B parameters of
base, python specialized, and instruction-following models, all trained on
text and code from multiple datasets, CodeLLAMA emerges as a comprehensive
suite of models, adept at handling a wide array of programming-related tasks.
Causal infilling, Long-context fine-tuning, and other techniques were utilized
for pre-training and fine-tuning. CodeLLAMA models’ family achieved state-of-
the-art performance in multiple benchmarks, indicating their potential for
transformative applications in cybersecurity. Their advanced code analysis and
generation capabilities could be crucial in automating threat detection and
enhancing vulnerability assessments.

####  VI-A11 CodeQwen1.5-7B

CodeQwen1.5-7B-Chat[133] is a transformer-based decoder-only language model
trained on 3 trillion tokens of code data. It supports 92 coding languages and
has strong code-generation capabilities. The model can understand and generate
long contexts of up to 64,000 tokens and has shown excellent performance in
text-to-SQL and bug-fixing tasks. It is based on Qwen1.5, which offers eight
model sizes, including 0.5B, 1.8B, 4B, 7B, 14B, 32B, and 72B dense models, and
an MoE model of 14B with 2.7B activated.

####  VI-A12 DeepSeek Coder-33B-instruct

Deepseek Coder [134] is a series of code language models, with each model
trained from scratch on 2 trillion tokens, 87% of which are code and 13%
natural language in English and Chinese. The model comes in various sizes,
ranging from 1B to 33B, with the 33B model being fine-tuned on 2 billion
tokens of instruction data. It achieves state-of-the-art performance among
open-source code models on multiple programming languages and benchmarks.

####  VI-A13 CodeGemma-7B

CodeGemma [135] is a collection of lightweight open code models built on top
of Gemma. It is a text-to-text and text-to-code decoder-only model with 7
billion parameters, specializing in code completion and generation tasks. It
can answer questions about code fragments, generate code from natural
language, or discuss programming or technical problems. CodeGemma was trained
on 500 billion tokens of primarily English language data from publicly
available code repositories, open-source mathematics datasets and
synthetically generated code.

####  VI-A14 Granite 8B Code

IBM released a family of Granite code models [136], including the
Granite-8B-Code-Base, to make coding more accessible and efficient for
developers. Granite-8B-Code-Base is a decoder-only code model designed for
code generation, explanation, and fixing. It is trained in two phases: first
on 4 trillion tokens from 116 programming languages, then on 500 billion
tokens from a carefully designed mixture of high-quality code and natural
language data. This two-phase training strategy ensures the model can reason
and follow instructions while understanding programming languages and syntax.

####  VI-A15 DeepSeek-V2

DeepSeek-V2 [137] is a mixture-of-experts (MoE) language model with 236
billion parameters, of which 21 billion are activated for each token. It is a
significant upgrade from the previous DeepSeek model, offering stronger
performance while reducing training costs by 42.5%. The model was pre-trained
on a vast and diverse corpus of 8.1 trillion tokens, followed by supervised
fine-tuning and reinforcement learning to maximise its capabilities.
DeepSeek-V2 excels at live coding tasks and open-ended generation, supporting
both English and Chinese.

TABLE VI: Comparison of Large Language Models Model |  Architecture |  Base Model |  Para-meters |  Training Tokens |  Pre-training |  Corpus Volume |  Released By |  Applications |  Use Cases in Cybersecurity |  Training Scheme |  Key Training Techniques |  Quanti-zation |  Ref
---|---|---|---|---|---|---|---|---|---|---|---|---|---
GPT-3 |  Decoder-only |  NA |  175B |  300B |  Books, Web text, Wikipedia, Common Crawl |  +570GB |  Open AI |  Language Modeling, Text Completion, QA |  Malware Detection, Threat Intelligence, Social Engineering Detection |  Pre-training, In-context learning |  Autoregressive training, Scaled Cross Entropy Loss, Backpropagation and gradient descent, Mixed precision training. |  NA |  [96]
GPT-4 |  Decoder-only |  NA |  NA |  NA |  Web Data, Third-party licensed data |  NA |  Open AI |  Language Modeling, Text Completion, QA |  Malware Detection, Threat Intelligence, Social Engineering Detection |  Pre-training, RLHF |  Autoregressive training |  NA |  [97]
T5 |  Encoder-decoder |  NA |  11B |  1000B |  C4, Web Text, Wikipedia |  750GB |  Google |  Language Modeling, Summarization, Translation |  Malware Detection, Threat Intelligence, Social Engineering Detection |  Pre-training, Fine-tuning |  Text-to-text framework, Denotation-based pretraining |  NA |  [98]
BERT |  Encoder-only |  NA |  340M |  250B |  BooksCorpus, English Wikipedia |  126GB |  Google |  Language Modeling, Classification, QA, NER |  Malware Detection, Threat Intelligence, Intrusion Detection, Phishing Detection |  Pre-training |  Masked LM(MLM), Next-sentence prediction(NSP) |  NA |  [99]
ALBERT |  Encoder-only |  BERT |  235M |  +250B (calculated) |  BooksCorpus, English Wikipedia |  NA |  Google |  Language Modeling, Classification |  Malware Detection, Threat Intelligence, Intrusion Detection, Phishing Detection |  Pre-training |  Factorized embedding parameterization, Cross-layer parameter sharing, Inter-sentence coherence loss, Sentence order prediction (SOP) |  NA |  [100]
RoBERTa |  Encoder-only |  BERT |  355M |  2000B |  BooksCorpus, English Wikipedia |  NA |  Meta |  Language Modeling, Classification, QA, NER |  Malware Detection, Threat Intelligence, Intrusion Detection, Phishing Detection |  Pre-training |  Dynamic Masking, Full-Sentences without NSP loss, Large mini-batches, Larger byte-level BPE |  NA |  [101]
XLNet |  Encoder-only |  Transformer-XL |  340M |  +2000B (calculated) |  English Wikipedia |  158GB (calculated) |  CMU, Google |  Language Modeling, Classification, QA |  Malware Detection, Threat Intelligence, Intrusion Detection, Phishing Detection |  Pre-training |  Permutation LM(PLM), Two-stream self-attention, Segment Recurrence and Relative Encoding |  NA |  [102]
ProphetNet |  Encoder-decoder |  NA |  550M |  +260B (calculated) |  Web Data, Books |  160GB |  Microsoft Research Asia |  Language Modeling, Question Generation, Summarization |  Cybersecurity Reporting, Threat Intelligence |  Pre-training, Fine-tuning |  Masked Sequence generation, Autoregressive training, Denoising Autoencoder objective, Shared Parameters between encoder and decoder, Maximum Likelihood Estimation (MLE) |  NA |  [103]
Falcon |  Decoder-only |  NA |  7-180B |  5000B |  Web Data |  NA |  TII |  Language Modeling, Text Completion, QA |  Malware Detection, Threat Intelligence, Social Engineering Detection |  Pre-training |  Autoregressive training, FlashAttention, ALiBi Positional encoding |  NA |  [104]
Reformer |  Encoder-decoder |  NA |  Up to 6B |  +150B (calculated) |  Web Data |  NA |  Google |  Language Modeling, Classification |  Malware Detection, Threat Intelligence, Intrusion Detection, Phishing Detection |  Pre-training |  Locality-Sensitive Hashing (LSH) Attention, Chunked Processing, Shared-QK Attention Heads, Reversible layers |  NA |  [105]
TABLE VII: Continued Model |  Architecture |  Base Model |  Para-meters |  Training Tokens |  Pre-training |  Corpus Volume |  Released By |  Applications |  Use Cases in Cybersecurity |  Training Scheme |  Key Training Techniques |  Quanti-zation |  Ref
---|---|---|---|---|---|---|---|---|---|---|---|---|---
PaLM |  Decoder-only |  NA |  540B |  780B |  Webpages, books, Wikipedia, news articles, source code, social media conversations, GitHub |  2TB |  Google |  Language Modeling, QA, Translation |  Threat Intelligence, Security Policies Generation |  Pre-training |  SwiGLU Activation, Parallel Layers, Multi-Query attention (MQA), RoPE embeddings, Shared Input-Output embedding |  NA |  [106]
PaLM2 |  Decoder-only |  NA |  NA |  NA |  web documents, books, code, mathematics, conversational data |  NA |  Google |  Language Modeling, QA, Summarization |  Threat Intelligence, Security Policies Generation |  Pre-training |  Compute optimal scaling, Canary token sequences, Control tokens for inference |  NA |  [107]
LLaMA |  Decoder-only |  NA |  7-65B |  1400B |  CommonCrawl, C4, GitHub, Wikipedia, Books, arXiv, StackExchange |  177GB |  Meta |  Language Modeling, Text Completion, QA |  Threat Intelligence, Malware Detection |  Pre-training |  Pre-normalization, SwiGLU activation function, Rotary Embedding, Model and sequence parallelism |  NA |  [108]
LLaMA2 |  Decoder-only |  NA |  7-70B |  2000B |  Mix of pulically available data |  NA |  Meta |  Language Modeling, Text Completion, QA |  Threat Intelligence, Malware Detection |  Pre-training, Fine-tuning, RLHF |  Optimized autoregressive training, Grouped Query Attention (GQA) |  NA |  [109]
GShard |  MoE |  NA |  600B |  1000B |  Web Data |  NA |  Google |  Language Modeling |  Threat Intelligence, Intrusion Detection, Malware Detection |  Pre-training |  Conditional Computation, Lightweight Annotation APIs, XLA SPMD partitioning, Position-wise MoE |  NA |  [110]
ELECTRA |  Encoder-only |  NA |  335M |  +1800B (calculated) |  BooksCorpus, English Wikipedia |  158GB |  Google |  Language Modeling, Classification |  Threat Intelligence, Intrusion Detection, Malware Detection, Phishing Detection |  Pre-training, Fine-tuning |  Replaced token detection, Generator-discriminator framework, Token replacement, Weight-sharing |  NA |  [111]
MPT-30B |  Decoder-only |  NA |  30B |  1000B |  C4, mC4, CommonCrawl, Wikipedia, Books, arXiv |  NA |  MosaicML |  Language Modeling, Text Completion, QA |  Threat Intelligence, Malware Detection, Software Vulnerability |  Pre-training |  FlashAttention, ALiBi positional encoding |  NA |  [112]
Yi-34B |  NA |  NA |  34B |  3000B |  Chinese and English dataset |  NA |  01.AI |  Language Modeling, Question Answering |  Threat Intelligence, Phishing Detection, Vulnerability Assessment |  Pre-training, Fine-tuning |  NA |  GPTQ, AWQ |  [113]
Phi-3-mini |  Decoder-only |  NA |  3.8B |  3.3T |  Phi-3 datasets (Public documents, synthetic, chat formats) |  NA |  Microsoft |  Language Modeling, Text Completion, QA |  Threat Intelligence, Intrusion Detection, Malware Detection |  Pre-training, Fine-tuning |  LongRope, Query Attention (GQA) |  NA |  [138]
Mistral 7B |  Decoder-only |  NA |  7.24B |  NA |  NA |  NA |  Mistral AI |  Language Modeling, Text Completion, QA |  Threat Intelligence, Intrusion Detection, Malware Detection |  Pre-training, Fine-tuning |  Sliding Window Attention, Query Attention (GQA), Byte-fallback BPE tokenizer |  NA |  [139]
Cerebras-GPT 2.7B |  Decoder-only |  NA |  2.7B |  371B |  The Pile Dataset |  825 GB |  Cerebras |  Language Modeling, Text Completion, QA |  Threat Intelligence, Intrusion Detection, Malware Detection |  Pre-training |  standard trainable positional embeddings and GPT-2 transformer, GPT-2/3 vocabulary and tokenizer block |  NA |  [140]
ZySec-AI/ZySec 7B |  Decoder-only |  NA |  7.24B |  NA |  Trained across 30+ domains in cybersecurity |  NA |  ZySec AI |  Language Modeling, Text Completion, QA |  Expert guidance in cybersecurity issues |  Pre-training |  NA |  NA |  [141]
DeciLM 7B |  Decoder-only |  NA |  7.04 |  NA |  NA |  NA |  Deci |  Language Modeling, Text Completion, QA |  Threat Intelligence, Intrusion Detection, Malware Detection |  Pre-trained |  Grouped-Query Attention (GQA) |  NA |  [142]
TABLE VIII: Continued Model |  Architecture |  Base Model |  Para-meters |  Training Tokens |  Pre-training |  Corpus Volume |  Released By |  Applications |  Use Cases in Cybersecurity |  Training Scheme |  Key Training Techniques |  Quanti-zation |  Ref
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Zephyr 7B Beta |  Decoder-only |  Mistral 7B |  7.24B |  NA |  NA |  NA |  Hugging-Face |  Language Modeling, Text Completion, QA |  Threat Intelligence, Intrusion Detection, Malware Detection |  Fine-tuning |  Flash Attention, Direct Preference Optimization (DPO) |  NA |  [143]
Dolly v2 12B |  Decoder-only |  Pythia 12B |  12B |  3T |  The Pile Dataset |  825GiB |  Databricks |  Language Modeling, Text Completion, QA |  Threat Intelligence, Intrusion Detection, Malware Detection |  Fine-tuning |  NA |  NA |  [144]
Falcon2 11B |  Decoder-only |  NA |  11.1B |  5T |  RefinedWeb enhanced with curated corpora. |  NA |  TII |  Language Modeling, Text Completion, QA |  Malware Detection, Threat Intelligence, Social Engineering Detection |  Pre-training |  ZeRO, high-performance Triton kernels, FlashAttention-2 |  NA |  [114]
TABLE IX: Comparison of Code-specific Large Language Models Model |  Architecture |  Base Model |  Para-meters |  Training Tokens |  Pre-training |  Corpus Volume |  Released By |  Applications |  Use Cases in Cybersecurity |  Training Scheme |  Key Training Techniques |  Quanti-zation |  Ref
---|---|---|---|---|---|---|---|---|---|---|---|---|---
SantaCoder |  Decoder-only |  NA |  1.1B |  236B |  The Stack v1.1 dataset (Python, Java, and JavaScript) |  268GB |  Hugging-Face, ServiceNow |  Code Generation, Code Completion, Code Analysis, QA |  Threat Intelligence, Software Vulnerability, Source Code Generation |  Pre-training |  Multi Query Attention (MQA), Fill-in-the-Middle (FIM) |  NA |  [123]
StarCoder |  Decoder-only |  NA |  15.5B |  PT 1000B, FT 35B |  80+ programming languages, Git commits, GitHub issues, and Jupyter notebooks |  +800GB |  Hugging-Face, ServiceNow |  Code Generation, Code Completion, Code Analysis, QA |  Threat Intelligence, Software Vulnerability Detection |  Pre-training, Fine-tuning |  Fill-in-the-Middle (FIM), Multi Query Attention (MQA), Learned absolute positional embeddings |  NA |  [124]
StarChat Alpha |  Decoder-only |  StarCoder-base |  16B |  NA |  oasst1 and databricks-dolly-15k datasets |  NA |  Hugging-Face, ServiceNow |  Code Generation, Code Completion, Code Analysis, QA |  Threat Intelligence, Software Vulnerability |  Fine-tuning |  NA |  NA |  [125]
CodeGen2 |  Decoder-only (causal LM) |  NA |  1-16B |  400B |  Stack dataset v1.1 |  NA |  Salesforce |  Program Synthesis, Code Generation |  Threat Intelligence, Software Vulnerability |  Pre-training |  Causal Language Modeling, Cross-entropy Loss, File-level Span Corruption, Infilling |  NA |  [126]
CodeGen2.5 |  Decoder-only (causal LM) |  NA |  7B |  1400B |  StarCoderData |  NA |  Salesforce |  Code Generation, Code Completion, Code Analysis |  Threat Intelligence, Software Vulnerability |  Pre-training |  Flash Attention, Infill Sampling, Span Corruption |  NA |  [127]
CodeT5+ |  Encoder-decoder |  NA |  220M-16B |  51.5B |  CodeSearchNet dataset, GitHub code dataset |  NA |  Salesforce |  Code Generation and Completion, Math Programming, Text-to-code Retrieval Tasks |  Threat Intelligence, Software Vulnerability |  Pre-training |  Span Denoising, Contrastive Learning, text-code Matching, Causal Language Modeling (CLM) |  NA |  [128]
XGen-7B |  Decoder-only |  NA |  7B |  1500B |  GitHub, Several public sources, Apex code data (mixture of natural text data and code data) |  NA |  Salesforce |  Code Generation, Summarization |  Threat Intelligence, Software Vulnerability |  Pre-training, Fine-tuning |  Standard Dense Attention, Two-stage Training Strategy |  NA |  [129]
Replit Code V1 |  Decoder-only (causal LM) |  NA |  2.7B |  525B |  Stack Dedup v1.2 dataset (20 different languages) |  NA |  Replit, Inc. |  Code Completion, Code Generation |  Threat Intelligence, Software Vulnerability |  Pre-training |  Flash Attention, AliBi Positional Embeddings, LionW Optimizer |  Matrix Multiplication |  [130]
DeciCoder-1B |  Decoder-only |  NA |  1B |  446B |  StarCoderData (Python, Java, and JavaScript) |  NA |  Deci |  Code Completion, Code Generation, Code Analysis |  Threat Intelligence, Software Vulnerability |  Pre-training |  Fill-in-the-Middle training (FIM), Grouped Query Attention (GQA) |  NA |  [131]
CodeLLAMA |  Decoder-only |  LLaMA2 |  7-34B |  620B |  Text and code from multiple datasets |  NA |  Meta |  Code Completion, Code Generation, Code Analysis |  Threat Intelligence, Software Vulnerability |  Pre-training, Fine-tuning |  Causal Infilling, Autoregressive Training, Repository-level Reasoning, Long-context Fine-tuning |  NA |  [132]
CodeQwen1.5-7B |  Decoder-only |  Qwen1.5 |  7.25B |  3T |  code-related data |  NA |  Qwen |  Code Generation, Code Completion, Code Analysis |  Threat Intelligence, Software Vulnerability, Bug fixes |  Pre-training |  Flash Attention, RoPE, Grouped-Query Attention (GQA) |  NA |  [133]
DeepSeek Coder-33B-instruct |  Decoder-only |  NA |  33.3B |  2T |  Composition of code and natural language |  NA |  DeepSeek |  Code Generation, Code Completion, Code Analysis |  Threat Intelligence, Software Vulnerability |  Pre-training, Long-context pre-training, Instruction fine-tuning |  Flash Attention, RoPE, Grouped-Query Attention (GQA) |  NA |  [134]
CodeGemma-7B |  Decoder-only |  Gemma |  8.54B |  500B |  Code repositories, Mathematics datasets, Synthetic code |  NA |  Google |  Code completion, Code generation, Code chat, Instruction following |  Threat Intelligence, Software Vulnerability |  Pre-training, Fine-tuning |  Fill-in-the-middle (FIM) tasks, dependency graph-based packing, unit test-based lexical packing |  NA |  [135]
TABLE X: Continued Granite 8B Code |  Decoder-only |  NA |  8.05B |  4.05T |  Publicly Datasets (GitHub Code Clean, Starcoder data) |  NA |  IBM Granite |  Code generation, Code explanation, Code fixing, etc. |  Threat Intelligence, Intrusion Detection, Malware Detection |  Pre-trained in two phases (the second phase for high-quality data) |  RoPE embedding, Grouped-Query Attention (GQA), Context Length of 4096 Tokens |  NA |  [136]
---|---|---|---|---|---|---|---|---|---|---|---|---|---
DeepSeek-V2 |  Decoder-only |  NA |  236B |  8.1T |  Composition of code and natural language |  NA |  DeepSeek |  Code Generation, Code Completion, Code Analysis |  Threat Intelligence, Software Vulnerability |  Pre-training, SFT, RL, Long Context Extension |  Mixture-of-Experts (MoE), Multi-head Latent Attention (MLA) |  NA |  [137]
TABLE XI: Datasets Used for Pre-training Foundation Models in Coding Dataset |  Title |  Year |  Purpose |  Content |  Significance
---|---|---|---|---|---
CodeSearchNet [145] |  ”CodeSearchNet Challenge: Evaluating the State of Semantic Code Search” |  2019 |  Focuses on bridging natural language and code. |  Contains about 6 million functions from six languages and 2 million automatically generated query-like annotations. |  Advances the semantic code search field with a challenge including 99 queries and 4k expert annotations.
The Pile [146] |  ”The Pile: An 800GB Dataset of Diverse Text for Language Modeling” |  2020 |  Designed to train large-scale language models. |  Comprises 22 high-quality, diverse text subsets totaling 825 GiB. |  Improves model generalization capabilities; evaluates with GPT-2 and GPT-3.
CodeParrot 111https://huggingface.co/datasets/codeparrot/github-code |  CodeParrot Dataset |  2022 |  Facilitates model training in code understanding and generation. |  Consists of 115M code files from GitHub in 32 programming languages, totaling 1TB. |  Aids in diverse language and format model training.
The Stack [147] |  ”The Stack: 3 TB of permissively licensed source code” |  2022 |  Aimed at fostering research on AI for code. |  Features 3.1 TB of code in 30 programming languages. |  Demonstrates improved performance on text2code benchmarks; introduces data governance.
ROOTS [148] |  ”The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset” |  2023 |  Supports ethical, multilingual model research. |  Spans 59 languages and focuses on diverse, inclusive data. |  Advances large-scale language model research with an ethical approach.
The Stack v2 [149] |  ”StarCoder 2 and The Stack v2: The Next Generation” |  2024 |  Enhances foundation models for code. |  Built from sources including 619 programming languages, significantly larger than its predecessor. |  Shows improvements in code LLM benchmarks; ensures transparency in training data.

###  VI-B Datasets Development for Code-centric LLM Models

The development of large-scale datasets has played a crucial role in advancing
LLM models, especially those focused on understanding and generating code.
Table XI presents the datasets used for pre-training foundation models in
Coding. Datasets like CodeSearchNet [145] and The Pile [146] have been
instrumental in bridging the gap between natural language and code, improving
semantic search capabilities, and enhancing language model training across
diverse domains. These datasets provide a rich source of real-world code in
multiple programming languages and include expert annotations and natural
language queries that challenge and push the boundaries of LLM performance in
code-related tasks.

Over time, the focus has shifted towards increasing the size, diversity, and
ethical considerations of the data used in training AI models. Introducing
datasets such as ROOTS and The Stack v2 [149] reflects a growing emphasis on
responsible LLM development. These newer datasets encompass a broader range of
programming languages and coding scenarios, and they incorporate governance
frameworks to ensure the ethical use of the data. In addition, these datasets
are designed to address the needs of large multilingual language models and
the specific challenges of code generation and comprehension, demonstrating
the evolving landscape of LLM research driven by enhanced dataset quality and
scope.

TABLE XII: Comparative Analysis of Vulnerabilities in LLM-Generated Code Reference |  Year |  Primary Focus |  Methodology |  Key Findings
---|---|---|---|---
Schuster et al. [150] |  2021 |  Poisoning in code autocompletion |  Experimental poisoning attacks on autocompleters |  Demonstrated effective targeted and untargeted poisoning; current defenses are largely ineffective.
Asare et al. [151] |  2023 |  Security analysis of GitHub’s Copilot |  Empirical analysis comparing human and Copilot-generated code vulnerabilities |  Copilot does not consistently replicate human vulnerabilities, showing variable performance across different types.
Sandoval et al. [152] |  2023 |  Security implications of LLM code assistants in C programming |  User study with AI-assisted coding tasks |  Minimal increase in security risks from LLM assistance compared to control.
Perry et al. [153] |  2023 |  Impact of AI code assistants on security |  Large-scale user study on security task performance |  Participants using AI wrote less secure code but were overconfident in its security.
Hamer et al. [154] |  2024 |  Security vulnerabilities in LLM vs. StackOverflow code |  Empirical analysis of code snippets for security vulnerabilities |  LLM-generated code had fewer vulnerabilities than StackOverflow, highlighting differences in security risks.
Cotroneo et al. [155] |  2024 |  Security assessment tool for AI-generated code |  Development and validation of DeVAIC tool |  DeVAIC effectively identifies vulnerabilities in Python code, outperforming other tools.
Tóth et al. [156] |  2024 |  Evaluating security of LLM-generated PHP web code |  Hybrid evaluation using static and dynamic analysis |  Significant vulnerabilities found in AI-generated PHP code, emphasizing the need for thorough testing.
Tihanyi et al. [157] |  2024 |  Security of LLM-generated C code from neutral prompts |  Dataset creation and analysis using formal verification |  Over 63% of generated C programs were found vulnerable, with minor variations between different LLMs.

###  VI-C Vulnerabilities Analysis of LLM-Generated Code

The evolution of LLMs in software development has brought significant
advancements and new security challenges [158]. Table XII presents a
comparative analysis of vulnerabilities in LLM-generated code.

Schuster et al. [150] demonstrate how LLMs employed in code autocompletion are
susceptible to poisoning attacks, which can manipulate the model’s output to
suggest insecure code. This vulnerability is intensified by the ability to
target specific developers or repositories, making the attacks more effective
and difficult to detect. Despite defenses against such attacks, their
effectiveness remains limited, raising concerns over the secure deployment of
these technologies [150].

Recent studies, such as those by Asare et al. [151] and Sandoval et al. [152],
provide an empirical and comparative analysis of the security aspects of code
generated by LLMs like GitHub’s Copilot and OpenAI Codex. Asare et al. [151]
find that while Copilot occasionally replicates vulnerabilities known from
human-written code, it does not consistently do so across different
vulnerabilities. In contrast, Sandoval et al. [152] report a minimal increase
in security risks when developers use LLMs in coding, indicating that LLMs do
not necessarily degrade the security of the code more than human developers
would.

Moreover, Perry et al. [153] reveal a concerning trend where users interacting
with AI code assistants tend to write less secure code but believe otherwise.
Their findings underscore the need for heightened awareness and better design
of user interfaces to foster critical engagement with the code suggestions
provided by LLMs [153]. In a similar vein, Hamer et al. [154] emphasize the
educational gap among developers regarding the security implications of using
code snippets from AI like ChatGPT or traditional sources like StackOverflow,
highlighting that both sources can propagate insecure code.

Lastly, novel tools like DeVAIC introduced by Cotroneo et al. [155] and
comprehensive vulnerability evaluations in LLM-generated web application code
by Tóth et al. [156] and Tihanyi et al. [157] illustrate ongoing efforts to
understand better and mitigate the risks associated with AI-generated code.
DeVAIC, for instance, offers a promising approach to detecting vulnerabilities
in incomplete Python code snippets, potentially enhancing the security
assessment capabilities for AI-generated code.

##  VII CyberSecurity Datasets for LLMs

###  VII-A Cyber Security Dataset Lifecycle

Creating a cybersecurity dataset for use with LLMs involves several steps that
ensure the dataset is comprehensive, accurate, and effective for training or
evaluating the models. Figure 6 presents the cyber security dataset lifecycle
for LLM development.

####  VII-A1 Define Objectives

Defining the objectives for a cybersecurity dataset for LLMs is crucial as it
dictates its construction and application. For training purposes, the dataset
should cover various cybersecurity topics and incorporate various data types
like text, code, and logs, aiming to develop a robust and versatile LLM
capable of understanding diverse threats (e.g., Edge-IIoT dataset [159] for
Network Security and FormAI dataset [157, 160] for Software Security). For
evaluation, the focus narrows to specific areas, such as benchmarking the
LLMs’ knowledge in cybersecurity (e.g., CyberMetric [89]).

####  VII-A2 Scope and Content Gathering

For the scope and content gathering stage of building a cybersecurity dataset
aimed at training and fine-tuning LLMs, selecting a broad range of topics is
essential to ensure comprehensive coverage. Key areas include network
security, malware analysis, software security, cryptographic protocols, cloud
security, and incident response. The data should be sourced from diverse and
reliable origins, such as public and private databases such as Common Weakness
Enumeration (CWE) and Common Vulnerabilities and Exposures (CVE) [161, 162].

####  VII-A3 Data Cleaning and Preprocessing

This process involves filtering out irrelevant content to maintain a focus on
cybersecurity and standardizing formats across the dataset. For example,
processing the Starcoder 2 dataset [149] involves several meticulous steps to
refine GitHub issues collected from GHArchive. Initially, auto-generated texts
from email replies and brief messages under 200 characters are removed, along
with truncating longer comments to maintain a maximum of 100 lines while
preserving the last 20 lines. This step alone reduced the dataset volume by
17%. The dataset then undergoes further cleaning to remove comments by bots
identified through specific keywords in usernames, eliminating an additional
3% of the issues. A notable focus is placed on the interaction quality within
the issues; conversations with two or more users are prioritized, and those
with extensive text under a single user are preserved if they stay under 7,000
characters. Issues dominated by a single user with more than ten events are
excluded, recognizing them as potentially low-quality or bot-driven, resulting
in a 38% reduction of the remaining dataset. For privacy, usernames are
anonymized by replacing them with a sequential participant counter,
maintaining confidentiality while preserving the integrity of conversational
dynamics.

Figure 6: Cyber Security Dataset Lifecycle for LLM development.

####  VII-A4 Annotation and Labeling

A sophisticated hybrid approach can be adopted to ensure precision and
scalability in the annotation and labeling stage of developing a cybersecurity
dataset for LLMs. Cybersecurity experts manually annotate the dataset,
meticulously labeling complex attributes such as threat type, guaranteeing
high accuracy. Concurrently, automated tools like static analyzers (e.g.,
Clang for C/C++ and Bandit for Python), formal verification methods (e.g.,
ESBMC), and dynamic tools are employed to handle the large volume of data
efficiently. These tools initially tag the data, which human experts carefully
review and correct [163].

###  VII-B Software Cyber Security datasets

In software cyber security, datasets play a crucial role in understanding,
detecting, and mitigating vulnerabilities in software systems. This sub-
section explores several significant software cybersecurity datasets, each
offering unique insights and methodologies for vulnerability analysis in
cybersecurity. From the extensive BigVul dataset, which links vulnerabilities
in the CVE database to specific code commits, to the innovative FormAI
dataset, leveraging AI-generated C programs and advanced verification methods
for precise vulnerability classification, each dataset contributes uniquely to
the field. These datasets range from manually labeled by security experts to
those generated using state-of-the-art automated tools, providing diverse
resources for researchers and practitioners. Table XIII provides an overview
of software vulnerability datasets that can be used for fine-tuning LLMs for
software security.

####  VII-B1 Sate IV - Juliet dataset

Nist has developed the SARD Juliet dataset to assess the capabilities of
static analysis tools on C/C++ program code out of many other programming
languages. The dataset contains the source files, with each test case
containing bad functions and good functions that patch the vulnerable “bad”
code. Test cases are labeled with CWEs to indicate the type of vulnerability
exposed in the program. The dataset contains keywords to indicate precisely
where vulnerable and non-vulnerable functions exist. Thus, the dataset needs
careful sanitization /obfuscation. While the dataset has many vulnerability
types and gives concrete examples, they are still programs purposefully built
to demonstrate vulnerabilities rather than naturally occurring ones.

####  VII-B2 Draper dataset

Researchers in work [164] leveraged a new dataset for vulnerability detection
using deep representation. A custom lexer was used to create a generic
representation to capture the essential tokens while minimizing the token
count. It was curated using open-source C/C++ code from SATE IV, Github, and
Debian and labeled using three static analyzers. The dataset is substantial,
but the vulnerability percentage is low, standing at roughly 6.8%. The dataset
is multi-labelled, where more than one CWE can exist in a code sample. The
dataset focuses on four main CWEs or categories, while the rest of the
vulnerabilities are grouped into one class. The researchers mapped the static
analyzer findings to CWEs and binary labels. Furthermore, since the
researchers did the mapping, warnings, and functions flagged by static
analyzers that would not typically be exploited were not flagged as
vulnerable. In addition, a strict deduplication process was used to refine the
dataset. The authors utilize this dataset to train their model after lexing
the source code to reduce the code representation and use a limited vocabulary
size. Due to lexing the source code, the approach reduces the needed
vocabulary size compared to the original size required. However, the
vulnerable portion is minimal compared to the dataset. Moreover, the labeling
considers four categories, which are limited compared to other datasets.

####  VII-B3 Reveal dataset

Reveal [165] was proposed to provide an accurate dataset that reflects the
real world, which is why it is also reflected in the imbalance of the samples.
Their work finds that performance drops by more than 50% in real-world
prediction, highlighting the need for a dataset subjected to a realistic
setting. The authors focus on two open-source projects, Linux Debian and
Chromium, as they are popular, well-maintained, showcase important domains,
and have publicly available vulnerability reports. The data is not used as
text but as Code Property Graphs (CPG), which are then converted to graph
embeddings for training a Gated Graph Neural Network (GGNN). The authors use
an approach inspired by Zhou et. al [166] to identify the security
vulnerabilities in the project, and they remedy the class imbalance due to the
majority of non-vulnerable code through SMOTE. Such data was collected from
Bugzilla and Debian security tracker. Looking at the vulnerable portion, it
constitutes 9.16% out of the 18,169 programs. While the dataset attempts to
depict a realistic dataset, relying on two sole projects might limit how well
a model trained on the dataset would perform in a real-world prediction case.

####  VII-B4 Devign dataset

Researchers of Devign [162] required an accurate dataset to be used in several
graph forms, which they believe is better in reflecting the structural and
logical aspects of source code. The proposed approach contains a graph
embedding layer that uses Abstract Syntax Tree (AST), Control Flow Graph
(CFG), Data Flow Graph (DFG), and Natural Code Sequence (NCS) to generate a
joint graph representation. The rationale behind a joint representation is the
ability of certain graphs to portray different vulnerability types not
uncovered by others. Motivated to propose a more accurate dataset instead of
those generated using static analyzers, the researchers invested in a security
team to manually label the samples. The data is collected from large open-
source projects: Linux, Wireshark, QEMU, and FFmpeg. The dataset is manually
labeled over two rounds, with 600 hours put into labeling it. While a
significant advantage of the dataset is that it is manually labeled and
verified, the dataset is only binary labeled. Also, it is worth noting that
only 2 out of the four datasets are available.

####  VII-B5 VUDENC

The VUDENC dataset [167] is comprised of 25,040 vulnerability-fixing commits
from 14,686 different GitHub repositories. The commits were filtered only to
include those that changed the code in a limited number of places, ensuring
that the changed code was related to the commit message. The dataset covers
seven common vulnerability types: SQL injection, cross-site scripting (XSS),
command injection, cross-site request forgery (XSRF), remote code execution,
path disclosure, and open redirect. This Python-specific dataset focuses on
improving software systems’ security by identifying potentially vulnerable
code. Each vulnerability type has a dedicated dataset, with the number of
repositories ranging from 39 to 336 and the number of changed files ranging
from 80 to 650. The total lines of code across all vulnerability types exceed
200,000, demonstrating the comprehensive nature of the dataset.

####  VII-B6 BigVul dataset

BigVul [168] is a C/C++ vulnerability dataset curated from the CVE database
and its relevant open-source projects. 3,754 code vulnerabilities were
collected from 348 open-source projects spanning 91 vulnerability types. The
dataset links CVEs in the CVE database with code commits and project bug
reports. Furthermore, the dataset contains 21 features to show changes and
where the vulnerability lies. Compared to other datasets, BigVul provides many
characteristics that can be useful for thoroughly analyzing vulnerabilities
and the history of change. Moreover, the diversity of the projects and the
vulnerability types expose the models being trained on it to several patterns.
However, the dataset only contains 11,823 vulnerable functions as opposed to
the 253,096 non-vulnerable functions. While it may depict real projects, the
data is imbalanced, and more vulnerable functions are needed to train large
models.

####  VII-B7 D2A dataset

A Dataset proposed by IBM [161] is curated using differential analysis to
label issues reported by static analysis tools. Bug-fixing commit pairs are
extracted from open-source projects with a static analyzer running on them. If
issues were detected in the “before” version and disappeared in the “after”
version, then it is assumed to be a bug. Compared to other datasets, the bug
trace is included in the dataset to determine the type and exact location of
the bug. Open-source projects such as FFmpeg, OpenSSL, httpd, libtiff, libav
and NGINX constitute the curated dataset. This dataset also has a limited
number of vulnerable samples, and a manual validation experiment shows that
the results are better than those of regular differential analysis. However,
it is still not at the desired accuracy, with manual validation showing an
accuracy of 53%. The paper’s authors applied the dataset to build a classifier
to identify false alarms in static analyzers to reduce the false positive
rate.

####  VII-B8 CVEfixes

CVEfixes [169] is a dataset built using the method proposed by the authors to
curate vulnerability datasets based on CVEs. The automated tool was used to
release CVEfixes, a dataset covering CVEs up to 9 June 2021. The dataset is
organized in a relational database, which can be used to extract data with the
desired information. It contains the code changes in several levels, namely on
the repository, commit, file, and method levels. The dataset contains 5495
vulnerability fixing commits with 5365 CVE records, covering 1754 open-source
projects. The mining tool is shared, and the most recent CVE records can be
mined.

####  VII-B9 CrossVul

The CrossVul dataset [170] encompasses a diverse range of programming
languages, exceeding 40 in total, and comprises vulnerable source files. The
dataset was curated by extracting data from GitHub projects referenced by the
National Vulnerability Database (NVD), specifically focusing on files modified
through git-diff. Files preceding the commit are tagged as vulnerable, while
those following the commit are designated as non-vulnerable. Organized by
Common Weakness Enumerations (CWEs)/Common Vulnerabilities and Exposures
(CVEs), as well as language types, the dataset offers a comprehensive
classification of vulnerabilities. It encompasses 1675 GitHub projects,
spanning 5877 commits and 27,476 files, with an equal distribution of 13,738
files marked as vulnerable and non-vulnerable, respectively. A supplementary
dataset containing the commit messages for each sample is provided.

####  VII-B10 SySeVR dataset

SySeVR framework was proposed in [171], which builds on the previous work in
VulDeePecker [172]. While VulDeePecker only considers library/ API function
calls, SySeVR covers a variety of vulnerabilities. Furthermore, SySeVR
utilized a unique approach using the notions of syntax-based Vulnerability
Candidates(SyVCs) and Semantics-based Vulnerability Candidates (SeVCs) to
represent programs as vectors that accommodate syntax and semantic
information. Their approach results show a reduced false-negative rate. The
dataset is collected from the National Vulnerability Database (NVD) and
Software Assurance Reference Dataset (SARD). The NVD dataset contains 19
popular C/C++ open source products and the SARD data comprises 126
vulnerability types. There are 1,591 programs from open-source projects, of
which 874 are vulnerable. As for SARD, there are 14,000 programs, with 13,906
being vulnerable. While this dataset uses the existing datasets published by
NIST, the datasets would need further processing in most cases. For example,
many vulnerable SARD programs contain the vulnerable snippet and its patch.
Not separating them into different samples might yield unwanted results
depending on the application.

####  VII-B11 DiverseVul dataset

DiverseVul [173] is proposed as a new vulnerable source code dataset that
covers 295 than the previous datasets combined. Furthermore, the dataset is
60% bigger than previous open-source C/C++ datasets. The data is collected by
crawling security issue websites and extracting the commits. The security-
based commits are labeled vulnerable before and non-vulnerable for the version
after the commit. DiverseVul covers over 797 projects and 7,514 commits with
more than 130 CWEs. The MD5 hashes are used to de-duplicate functions,
yielding 18,495 vulnerable and 330,492 non-vulnerable functions. The authors
conduct several experiments to validate the dataset, combining their dataset
with previous datasets and showing insights and possibilities of their use.
The paper shows that using their dataset along with the previous datasets
yields the best result in their experiments, as opposed to using a single
dataset.

####  VII-B12 FormAI dataset

The FormAI dataset [160] represents a significant advancement in cybersecurity
and LLM, featuring an extensive collection of 112,000 AI-generated,
independent, and compilable C programs. This dataset is unique because it
utilizes dynamic zero-shot prompting techniques to create various programs,
ranging from complex tasks like network management and encryption to simpler
ones like string manipulation. These programs were generated using
GPT-3.5-turbo, demonstrating the ability of Large Language Models (LLMs) to
produce diverse and realistic code samples. A standout feature of the FormAI
dataset is its meticulous vulnerability classification. Each program is
thoroughly analyzed for vulnerabilities, with the type of vulnerability, the
specific line number, and the name of the vulnerable function clearly labeled.
This precise labeling is achieved using the Efficient SMT-based Bounded Model
Checker (ESBMC), an advanced formal verification method. ESBMC employs
techniques like model checking, abstract interpretation, constraint
programming, and satisfiability modulo theories to rigorously assess safety
and security properties in the programs. This approach ensures that
vulnerabilities are definitively detected, providing a formal model or
counterexample for each finding and effectively eliminating false positives.

####  VII-B13 Chrysalis-HLS

Chrysalis-HLS [75] dataset, a helpful resource for improving Large Language
Models’ performance in hardware and software design. This comprehensive
dataset targets functional verification and code debugging in High-Level
Synthesis (HLS). It offers a realistic evaluation environment with over 1,000
function-level designs and up to 45 injected bug combinations. Named
”Chrysalis” to symbolize code transformation, it includes diverse HLS
applications with various error types. Created with GPT-4 and curated prompts,
Chrysalis-HLS is a valuable resource for advancing LLM capabilities in HLS
verification and debugging, enhancing hardware engineering.

####  VII-B14 ReFormAI

The ReFormAI dataset [174] is a large-scale dataset of 60,000 independent
SystemVerilog designs with varied complexity levels, targeting different
Common Weakness Enumerations (CWEs). The dataset was generated by four
different LLMs and features a unique set of designs for each of the 10 CWEs
evaluated. The designs were labeled based on the vulnerabilities identified by
formal verification with unbounded proof. The LLMs evaluated include
GPT-3.5-Turbo, Perplexity AI, Text-Davinci-003, and LLaMA. The results
indicate that at least 60% of the samples from the 60,000 SystemVerilog
designs are vulnerable to CWEs, highlighting the need for caution when using
LLM-generated code in real-world projects.

####  VII-B15 PrimeVul

PrimeVul [80] dataset is a benchmark dataset based on existing open-source
datasets. Mainly taking into consideration BigVul [168], CrossVul [170],
CVEfixes [169] and DiverseVul [173]. The proposed pipeline consists of
merging, de-duplication, and labeling through 1) PRIMEVUL-ONEFUNC and 2)
PRIMEVUL-NVDCHECK. ONEFUNC selects only single functions that are associated
with security commits. NVDCHECK is the compartment where a commit is linked to
its CVE and checked for in the NVD database. The function is labeled
vulnerable if the description precisely mentions the function. The other case
is the description containing the file name and the function being the single
function changed by a security commit. After such a process, the yielded
dataset consists of 7k vulnerable functions and 228,800 benign functions. The
dataset spans 755 projects and contains 6,827 commits. Their work also
assesses the label quality of their dataset and other related datasets,
showing low label errors in PrimeVul.

####  VII-B16 X1

X1 [78] dataset is constructed from several open-source vulnerability
datasets: CVEFixes, a Manually-Curated Dataset, and VCMatch. The dataset
contains standalone functions labeled as either vulnerable or non-vulnerable.
The labeling process involves extracting functions from vulnerability-fixing
commits, assuming pre-change versions are vulnerable and post-change versions
are non-vulnerable. A modified dataset (X1) is created to address potential
false positives, containing only functions that were the sole change in a
commit. The final dataset consists of X1 without P3, which has 1334 samples,
and X1 with P3, which has 22945 samples. X1 without P3 is balanced, with a 1:1
ratio of positive to negative classes, while X1 with P3 is imbalanced,
reflecting the real-world distribution of vulnerable functions with a 1:34
ratio. The dataset size is relatively small, which may limit its
representativeness of the real vulnerability distribution.

TABLE XIII: Overview of Software Vulnerability Datasets that can be used for
fine-tuning LLMs for software security.

Dataset |  Year |  Lang |  Source |  Multi-class |  Type |  Samples |  Labelling Method |  Classification Method |  Challenges/Limitations
---|---|---|---|---|---|---|---|---|---
Sate IV - Juliet |  2012 |  C, C++ & Java |  SARD |  Yes |  Synthetic |  Approx 60k (C/C++) & 29k (Java) test cases |  Testcases are vulnerable by design, with corresponding patch |  CWE |  Designed to be vulnerable, might not accurately depict real-world projects.
Draper [164] |  2018 |  C |  Open-source |  Yes |  Real |  Total: 1.27M V: 82K NV: 1.19M |  Static analyzers |  CWE |  Small percentage of vulnerable samples. Limited to four categories.
Reveal [165] |  2018 |  C/C++ |  Open-source |  No |  Real |  Total: 18k V: 1.6k NV: 16k |  Vulnerability-fixing commits identified by security terms |  Binary classes |  Imbalance in sample distribution and only binary labeled. Limited to two projects.
Devign [162] |  2019 |  C |  Open-source |  No |  Real |  Total: 26K V: 12K NV: 14K |  Binary Manual labeling |  Binary classes |  Binary labeled. Partial dataset release.
VUDENC [167] |  2019 |  Python |  Open-source |  Yes |  Real |  1,009 commits from 812 repositories |  Vulnerability-fixing commits from GitHub repositories |  Vulnerability type |  Relatively small Dataset, No guarantee that the commits fixed vulnerabilities.
BigVul [168] |  2020 |  C/C++ |  Open-source |  Yes |  Real |  Total: 264k V: 11.8k NV: 253k |  Vulnerability-fixing commits from CVE database |  CVE/CWE |  Significant class imbalance. Lack of CWEs/categories for all samples.
D2A [161] |  2021 |  C/C++ |  Open-source |  Yes |  Real |  Total: 1.3M V: 18.6k NV: 1.27M |  Vunerability-fixing commits with static analyzers |  Categories based on static analyzer |  Small percentage of vulnerable samples. Manual validation shows low accuracy.
CVEfixes [169] |  2021 |  27 languages |  Open-source |  Yes |  Real |  5,495 commits, 50k methods |  Vulnerability-fixing commits from CVE database |  CVE/CWE |  Labelling accuracy needs enhancement and dataset size increased (only limited to CVE records).
CrossVul [170] |  2021 |  40+ languages |  Open-source |  Yes |  Real |  5,877 commits, 27k (13,738 V/NV) files |  Vulnerability-fixing commits from CVE database |  CVE/CWE |  Labelling accuracy needs enhancement and dataset size increased. Takes the whole file without pinpointing functions. (only limited to CVE records).
SySeVR [171] |  2022 |  C/C++ |  SARD/NVD |  Yes |  Semi-Synthetic |  Total: 15.6k V: 14.8k NV: 811 |  Extracted from existing databases NVD and SARD |  CVE/CWE |  Limited subset of SARD/NVD. SARD is synthetic, while NVD is limited in the number of labeled vulnerabilities.
DiverseVul [173] |  2023 |  C/C++ |  Open-source |  Yes |  Real |  Total: 349K V: 18.9k NV: 330K |  Vulnerability-fixing commits from security trackers |  CWE |  Labelling accuracy needs enhancement and dataset size increased (specifically vulnerable functions).
FormAI [160] |  2023 |  C |  AI-generated |  Yes |  Artificial |  Total: 112k V: 57k NV: 55K |  Formal verification Bounded Model checker |  Custom categories |  Bounded formal verification does not cover all types of vulnerabilities and depth.
Chrysalis-HLS [75] |  2024 |  C++ |  Open-source |  Yes |  Synthetic |  Over 1,000 function-level HLS designs |  Predefined errors |  Bug Type |  Addressing scalability and generalization challenges
FormAI v2 [157] |  2024 |  C |  AI-generated |  Yes |  Artificial |  Total: 265k V: 168k NV: 23k |  Formal verification Bounded Model checker |  Custom categories |  Bounded formal verification does not cover all vulnerabilities and depth.
ReFormAI [174] |  2024 |  System Verilog |  AI-generated |  Yes |  Artificial |  Total: 60k V: 60k NV: 0 |  Formal verification Bounded Model checker |  CWE |  Formal verification with an unbounded proof.
PrimeVul [80] |  2024 |  C/C++ |  Open-source |  Yes |  Real |  Total: 236k V: 7k NV: 229k |  Single function selection and extraction from NVD |  CWE |  Limited vulnerable samples due filtering existing samples and specific function selection.
X1 [78] |  2024 |  Java |  Open-source |  Yes |  Real |  Total: 22.9k V: 0.6k NV: 22.3k |  Analyzing vulnerability-fixing commits |  Binary classes |  Imbalanced, small, and may not represent the true vulnerability distribution.

V : Vulnerable , NV: Non Vulnerable

Figure 7: LLM vulnerabilities included in the OWASP project.

##  VIII LLM vulnerabilities and Mitigation

This section reviews the OWASP Top 10 for LLM Applications project [175], a
comprehensive initiative designed to increase awareness about LLM security
vulnerabilities. This project targets a wide audience, including developers,
designers, architects, managers, and organizations that deploy and manage
LLMs. Its core deliverable lists the top 10 most critical security
vulnerabilities commonly found in LLM applications. In addition, we include
other LLM vulnerabilities not included in the OWASP project, as presented in
Table XIV and 7.

###  VIII-A Prompt Injection

Integrating LLMs into various digital platforms has brought to light the
critical issue of prompt injection [176]. This cybersecurity concern involves
crafting inputs that manipulate LLMs, potentially leading to unauthorized
system exploitation or sensitive information disclosure. As LLMs become more
prevalent, understanding and countering prompt injection attacks is paramount
for safeguarding the integrity and security of these systems [177].

####  VIII-A1 Nature of Prompt Injection Attacks

Prompt injection attacks in LLMs can manifest in various forms. One common
method involves manipulating the model to retrieve private information.
Attackers may craft inputs that subtly direct the LLM to divulge confidential
data. Another technique involves embedding hidden prompts in web pages, which
can solicit sensitive information from unsuspecting users [178]. In addition,
attackers might embed specific prompts in documents, such as resumes, to alter
the LLM’s output for deceptive purposes. Finally, the risk of web plugins
being exploited through rogue website instructions leads to unauthorized
actions by the LLM [179].

####  VIII-A2 Mitigation Strategies

To combat these threats, several mitigation strategies can be employed. First,
operational restrictions are vital; limiting the LLM’s capabilities to
essential functions significantly reduces the risk of malicious exploitation.
Requiring user consent for sensitive operations is another critical measure
[180]. This approach ensures that high-risk activities or operations involving
sensitive data only occur with explicit user approval. Therefore, the
influence of untrusted or unfamiliar content on user prompts should be
minimized to prevent indirect manipulations. Establishing clear trust
boundaries within the system is also crucial. These boundaries maintain user
control and prevent unauthorized actions, safeguarding the system from
external manipulations [181].

####  VIII-A3 Potential Attack Scenarios

The scenarios for prompt injection attacks are diverse and concerning. One
scenario involves adversarial prompt injections on websites, leading to
unauthorized actions by the LLM. Another potential threat is hidden prompt
injections in documents like resumes, designed to manipulate the LLM’s output
[182]. Furthermore, there’s the risk of direct user control over the LLM
through prompt injections, where malicious users craft inputs to gain undue
influence over the model’s responses. By understanding these risks and
implementing robust prevention strategies, developers and users of LLMs can
protect against potential exploitations [183].

###  VIII-B Insecure Output Handling

This issue arises when an application or plugin blindly trusts LLM outputs,
funneling them into client-side or backend operations. Such oversight can lead
to critical security risks like Cross-Site Scripting (XSS), Cross-Site Request
Forgery (CSRF), Server-Side Request Forgery (SSRF), privilege escalation, and
remote code execution.

####  VIII-B1 Nature of Insecure Output Handling Vulnerabilities

The core of the problem lies in the unverified acceptance of LLM outputs. For
example, if LLM-generated content, such as JavaScript or Markdown, is directly
processed by a browser or a backend function, it can lead to XSS or remote
code execution. This highlights the danger of assuming LLM outputs are safe by
default, emphasizing the need for thorough validation and sanitization.

####  VIII-B2 Prevention Strategies

Preventing these vulnerabilities involves two key strategies. Firstly,
implementing stringent validation for LLM outputs before interacting with
backend functions can help identify and neutralize potential threats.
Secondly, encoding LLM outputs before they reach the end user can prevent
misinterpretation of the code, thereby reducing the risk of malicious
executions.

####  VIII-B3 Potential Attack Scenarios

The scenarios for exploitation are varied. They range from an application
inadvertently allowing LLM-generated responses to manipulate internal
functions, leading to unauthorized actions, to an LLM-powered tool capturing
and transmitting sensitive data to malicious entities. Other risks include
allowing users to generate unvetted SQL queries through an LLM, which could
result in data breaches and the potential for LLMs to create and execute
harmful XSS payloads.

TABLE XIV: Overview of LLM vulnerabilities and Mitigation Vulnerabilities |  Nature of the Vulnerability |  Examples |  Mitigation Strategies |  Potential Attack Scenarios
---|---|---|---|---
Prompt Injection |  Manipulation of LLMs through crafted inputs leading to unauthorized exploitation or sensitive information disclosure. |  • Hidden prompts in web pages • Deceptive documents • Rogue web plugin instructions |  • Operational restrictions • User consent for sensitive operations • Trust boundaries establishment |  • Adversarial injections on websites • Hidden prompts in documents • Direct user control through crafted inputs
Insecure Output Handling |  Blind trust in LLM outputs lead to security risks like XSS, CSRF, SSRF, etc. |  • Direct processing of LLM-generated JavaScript or Markdown |  • Validation of LLM outputs • Encoding outputs before reaching end-users |  • LLM responses manipulating internal functions • Generating unvetted SQL queries • Creating harmful XSS payloads
Inference Data Poisoning |  Stealthy activation of malicious responses under specific operational conditions such as token-limited output. |  • Conditions based on token-output limits in user settings • Stealthily altered outputs when cost-saving modes are enabled |  • Monitoring and anomaly detection systems specifically designed for conditional outputs • Regular audits of outputs under various token limitations |  • Manipulated responses under token limitations leading to misinformation • Triggered malicious behavior in cost-sensitive environments
Adversarial Natural Language Instructions |  Code LLMs produce functionally accurate code with hidden vulnerabilities due to adversarial instructions. |  • DeceptPrompt algorithm creating deceptive instructions |  • Advanced code validation • Training LLMs with adversarial examples • Continuous updates and security patches |  • Crafted prompts leading to code with vulnerabilities • Unauthorized access or system compromises
Automatic Adversarial Prompt Generation |  Automated methods to generate prompts that bypass LLM alignment measures. |  • Crafting specific suffixes for objectionable content generation |  • Developing advanced alignment algorithms • Real-time monitoring • Training models with new adversarial examples |  • Bypassing alignment measures leading to the generation of objectionable content
Training Data Poisoning |  Manipulation of training data to skew LLM learning, introducing biases or vulnerabilities. |  • Injecting biased or harmful data into training sets |  • Verifying data sources • Employing dedicated models • Sandboxing, input filters • Monitoring for poisoning signs |  • Misleading outputs spreading biased opinions • Injection of false data into training
Insecure Plugins |  Vulnerabilities in plugin design and interaction with external systems or data sources. |  • Inadequate input validation • Overprivileged access • Insecure API interactions |  • Rigorous input validation • Adherence to least privilege • Secure API practices • Regular security audits |  • Exploiting input handling vulnerabilities • Overprivileged plugins for privilege escalation • SQL injections
Denial of Service (DoS) Attack |  Attempts to make a system inaccessible by overwhelming it with traffic or triggering crashes. |  • Volume-based attacks • Protocol attacks • Application layer attacks |  • Rate limiting • Robust infrastructure • Continuous monitoring and rapid response |  • Overloading servers • Disrupting communication between users and services • Straining system resources

###  VIII-C Adversarial Natural Language Instructions

Wu et al. [184] proposed presented DeceptPrompt, highlighting a critical
vulnerability in Code LLMs: their susceptibility to adversarial natural
language instructions. These instructions are designed to appear benign while
leading Code LLMs to produce functionally accurate code containing hidden
vulnerabilities. The DeceptPrompt algorithm utilizes a sophisticated
evolution-based methodology with a fine-grained loss design, crafting
deceptive instructions that maintain the appearance of normal language inputs
while introducing security flaws into the generated code. This vulnerability
is exacerbated by the challenges in preserving the code’s functionality,
targeting specific vulnerabilities, and maintaining the semantics of the
natural language prompts.

####  VIII-C1 Prevention Strategies

The study suggests a set of prevention strategies to counter these threats.
This involves integrating advanced code validation mechanisms within LLMs to
identify and mitigate potential vulnerabilities in the generated code.
Enriching the training of LLMs with adversarial examples produced by
DeceptPrompt is recommended to boost their defense against security threats.
Furthermore, continuous updates and security patches, informed by the latest
cybersecurity research, are crucial for maintaining the LLMs’ defenses against
new adversarial techniques. Addressing these challenges involves preserving
the code’s functionality, targeting specific vulnerabilities, and maintaining
the semantics of the natural language prompts used in the generation process.

####  VIII-C2 Potential Attack Scenarios

The authors highlight various potential attack scenarios that could exploit
the vulnerabilities exposed by DeceptPrompt. These scenarios include attackers
using crafted natural language prompts to induce Code LLMs into generating
code with vulnerabilities, leading to data breaches, unauthorized access, or
system compromises. The effectiveness of DeceptPrompt in real-world settings
underscores the urgency for robust security measures in Code LLMs, given their
increasing use in critical systems and infrastructure. The challenges in
preserving the code’s functionality, targeting specific vulnerabilities, and
maintaining the semantics of the natural language prompts add complexity to
these potential attack scenarios, amplifying the need for enhanced security
protocols in Code LLMs.

###  VIII-D Automatic adversarial prompt generation

In the highlighted study, Zou et al. [185] address the challenge of automatic
adversarial prompt generation in aligned language models. Their method focuses
on crafting a specific suffix that maximizes the likelihood of generating
objectionable content when attached to various queries directed at an LLM.
This approach differs significantly from previous methods by employing
automated techniques, including greedy and gradient-based search strategies.
The novelty lies in the method’s ability to bypass the alignment measures
designed to prevent undesirable outputs from LLMs, effectively circumventing
the safeguards put in place. Therefore, the study’s findings underscore the
need for robust prevention strategies against these sophisticated adversarial
attacks. Enhanced security measures are required to safeguard aligned LLMs.
These could include developing more advanced alignment algorithms resistant to
adversarial manipulations and implementing real-time monitoring systems to
detect and neutralize such attacks. In addition, continuously updating and
training models with new adversarial examples can help build resilience
against these evolving threats. The focus should be on creating systems that
can quickly adapt and respond to the changing tactics of adversarial prompt
generation.

###  VIII-E Training Data Poisoning

Training Data Poisoning in LLMs represents a critical security and ethical
issue, where malicious actors manipulate the training dataset to skew the
model’s learning process. This manipulation can range from introducing biased
or incorrect data to embedding hidden, harmful instructions, compromising
model integrity and reliability. The impact is profound, as poisoned LLMs may
produce biased, offensive, or inaccurate outputs, raising significant
challenges in detection due to the vast and complex nature of training
datasets [186].

####  VIII-E1 Nature of Training Data Poisoning

Training data poisoning in LLMs occurs when an attacker deliberately
manipulates the training data or fine-tuning processes. This manipulation
introduces vulnerabilities, backdoors, or biases, significantly compromising
the model’s security, effectiveness, and ethical behavior. Examples include
intentionally including targeted, inaccurate documents, training models using
unverified data, or allowing unrestricted dataset access, leading to loss of
control. Such actions can detrimentally affect model performance, erode user
trust, and harm brand reputation [187].

####  VIII-E2 Prevention Strategies

To combat training data poisoning, several prevention strategies are
essential. Firstly, verifying the supply chain of training data and the
legitimacy of data sources is crucial. This step ensures the integrity and
quality of the data used for training models. Employing dedicated models for
specific use cases can help isolate and protect different applications from a
compromised data source [185]. Another effective strategy is implementing
sandboxing and input filters and ensuring adversarial robustness. In addition,
regularly monitoring for signs of poisoning attacks through loss measurement
and model analysis is vital in identifying and mitigating such threats.

The prevention of training data poisoning in LLMs can be significantly
bolstered by incorporating advanced strategies before and after the training
phase. The pre-training defense is a dataset-level strategy that filters
suspicious samples from the training data. This method assumes that text and
image pairs (i.e., Multimodal data) in a dataset should be relevant to each
other. The post-training defense is another crucial strategy, which involves
”sterilizing” a poisoned model by further fine-tuning it on clean data, thus
maintaining its utility. This is conducted by fine-tuning the poisoned models
on a clean dataset (e.g., the VG dataset in the study) with a specific
learning rate [186].

####  VIII-E3 Potential Attack Scenarios

Several potential attack scenarios arise from training data poisoning. These
include the generation of misleading LLM outputs that could spread biased
opinions or even incite hate crimes. Malicious users might inject false data
into training, intentionally skewing the model’s outputs [188]. Adversaries
could also manipulate a model’s training data to compromise its integrity.
Such scenarios highlight the need for stringent security measures in the
training and maintaining LLMs, as the implications of compromised models
extend beyond technical performance to societal impacts and ethical
considerations.

###  VIII-F Inference Data Poisoning

####  VIII-F1 Nature of Inference Data Poisoning

Inference data poisoning targets LLMs during their operational phase, unlike
training-time attacks that tamper with a model’s training dataset. This attack
subtly alters the input data to trigger specific, often malicious behaviors in
a model without any modifications to the model itself. The approach detailed
by He et al. [189] utilizes a novel method where the poison is activated not
by obvious, fixed triggers but by conditions related to output token
limitations. Such conditions are generally overlooked as they are a part of
normal user interactions aimed at managing computational costs, thus enhancing
the stealth and undetectability of attacks.

####  VIII-F2 Prevention Strategies

Preventing inference data poisoning requires a multi-faceted approach.
Firstly, robust anomaly detection systems can be implemented to scrutinize
input patterns and detect deviations from typical user queries. Regular audits
of model responses under various conditions can also help identify any
inconsistencies that suggest poisoning. Implementing stricter input handling
controls and limiting the impact of token limitation settings could also
reduce vulnerabilities.

####  VIII-F3 Potential Attack Scenarios

The potential scenarios for inference data poisoning are varied and context-
dependent. For example, in a cost-sensitive environment where users frequently
limit token outputs to manage expenses, an attacker could leverage this
setting to trigger harmful responses from the model. Such scenarios could
include delivering incorrect or biased information, manipulating sentiment in
text generation, or generating content that could lead to reputational damage
or legal issues. The BrieFool framework [189] effectively exploits this
vector, demonstrating high success rates in controlled experiments,
highlighting the need for heightened security measures in environments where
LLMs are deployed.

###  VIII-G Insecure Plugins

####  VIII-G1 Nature of Insecure Plugins

The nature of insecure plugins in LLMs revolves around several key
vulnerabilities that stem from how these plugins are designed, implemented,
and interact with external systems or data sources. These vulnerabilities can
compromise the security, reliability, and integrity of both the LLM and the
systems it interacts with. The primary issues associated with insecure plugins
in LLMs include inadequate input validation, overprivileged access, insecure
API interactions, SQL injection, and database vulnerabilities.

####  VIII-G2 Prevention Strategies

To counter the Insecure Plugins, a multi-faceted approach to security is
essential. Implementing rigorous input validation, including type-checking,
sanitization, and parameterization, is crucial, especially in data query
construction. Adhering to the principle of least privilege is key in plugin
design; each plugin should only access necessary resources and
functionalities. Ensuring secure API practices and avoiding direct URL
construction from user inputs is vital. Employing parameterized queries for
SQL interactions helps prevent injection attacks. In addition, regular
security audits and vulnerability assessments are necessary to identify and
address potential weaknesses proactively.

####  VIII-G3 Potential Attack Scenarios

Various attack scenarios emerge from Insecure Plugins. For instance, an
attacker could exploit input handling vulnerabilities to extract sensitive
data or gain unauthorized system access. Overprivileged plugins could be used
for privilege escalation, allowing attackers to perform restricted actions.
Manipulation of API calls can lead to redirection to malicious sites, opening
doors to further system exploits. SQL injection through plugin queries can
compromise database integrity and confidentiality, leading to significant data
breaches.

###  VIII-H Denial of Service (DoS) attack

####  VIII-H1 Nature of DoS attack

A Denial of Service (DoS) attack is a malicious attempt to disrupt the normal
functioning of a targeted system, making it inaccessible to its intended
users. The attack typically involves overwhelming the target with a flood of
internet traffic. This could be achieved through various means, such as
sending more requests than the system can handle or sending information that
triggers a crash. In the context of services like LLM, a DoS attack could
bombard the service with a high volume of complex queries, significantly
slowing down the system or causing it to fail [190].

####  VIII-H2 Potential Attack Scenarios

The DoS attacks against LLM can be divided into three categories: volume-based
attacks, protocol attacks, and application layer attacks.

  * •

Volume-based Attacks: This is the most straightforward kind of DoS attack,
where the attacker attempts to saturate the bandwidth of the targeted system.
For LLM, this could involve sending many requests simultaneously, more than
what the servers are equipped to handle, leading to service disruption [191].

  * •

Protocol Attacks: These attacks exploit weaknesses in the network protocol
stack layers to render the target inaccessible. They could involve, for
instance, manipulating the communication process between the user and the LLM
service in a way that disrupts or halts service [192].

  * •

Application Layer Attacks: These are more sophisticated and involve sending
requests that appear to be legitimate but are designed to exhaust application
resources. For LLM, this could involve complex queries requiring extensive
processing power or memory, thereby straining the system [193].

####  VIII-H3 Prevention Strategies

To combat DoS attacks in LLM services, the following prevention strategies can
be applied:

  * •

Rate Limiting: Implementing a rate-limiting strategy is crucial. This involves
limiting the number of requests a user can make within a given timeframe,
which helps prevent an overload of the system.

  * •

Robust Infrastructure: A robust and scalable server infrastructure can help
absorb the influx of traffic during an attack. This could involve using load
balancers, redundant systems, and cloud-based services that can scale
dynamically in response to increased traffic.

  * •

Monitoring and Rapid Response: Continuous traffic monitoring can help quickly
identify unusual patterns indicative of a DoS attack. Once detected, rapid
response measures, such as traffic filtering or rerouting, can be employed to
mitigate the attack.

TABLE XV: LLM Cybersecurity Insights. Aspect |  Details |  Tools/Methods |  Applications
---|---|---|---
Architecture |  Focus on model components such as tokenization, attention mechanisms, and output generation. |  • Paper: Attention Is All You Need |  Threat Detection and Analysis, Security Automation, Cyber Forensics, Penetration Testing, Security Training and Awareness, and Chatbots.
Cyber Security Dataset |  Creation of prompt-response pairs that simulate cyber threats using synthetic data. |  • OpenAI API for synthetic data • Evol-Instruct for data refinement • Regex filtering for uniqueness |  Building datasets that mirror real-world threats for training and refining LLMs.
Pre-training Models |  Training on large-scale datasets comprising billions of tokens, filtered and aligned with cybersecurity lexicon. |  • Megatron-LM for handling large datasets • gpt-neox for sequential data handling • Distributed training tools |  Preparing LLMs to understand and predict cybersecurity-specific content accurately.
Supervised Fine-Tuning |  Incorporating specialized cybersecurity datasets into pre-trained models for tailored applications. |  • LoRA for parameter-efficient adjustments • QLoRA for quantization and efficient memory management |  Enhancing LLMs to address unique cybersecurity threats and scenarios specifically.
Cyber Security Evaluation |  Setting up specialized frameworks and datasets to test LLMs against potential cyber threats. |  • Bespoke cybersecurity benchmarks • Authoritative datasets for threat detection |  Evaluating how well LLMs detect, understand, and respond to cyber threats.
Advanced LLM Techniques |  Implementing techniques like RAG and RLHF to augment LLMs with real-time data and expert-aligned feedback. |  • RAG for context retrieval from databases • RLHF with specialized preference datasets and reward models |  Improving response relevance and accuracy in cybersecurity applications.
LLM Deployments |  Adopting deployment strategies that range from local installations to large-scale server setups. |  • Platforms like Gradio and Streamlit for prototyping • Cloud services for robust deployment • Edge deployment strategies for resource-limited environments |  Deploying LLMs in various environments to ensure accessibility and responsiveness across devices.
Securing LLMs |  Addressing vulnerabilities unique to LLMs such as prompt hacking and training data leakage. |  • Security measures like prompt injection prevention • Red teaming • Continuous monitoring systems |  Preventing and mitigating security threats to maintain data integrity and model reliability in LLMs.
Optimizing LLMs |  Implementing strategies to reduce memory and computational requirements while maintaining output quality. |  • Model quantization • Use of bfloat16 data formats • Optimization of attention mechanisms |  Enabling efficient LLM operation on various hardware, making them scalable and practical for diverse applications.
Figure 8: Parameter Efficient Fine-Tuning (PEFT) provides an efficient
approach by minimizing the number of parameters needed for fine-tuning and
reducing memory consumption comparable to that of traditional fine-tuning.

##  IX LLM Cybersecurity Insights, Challenges and Limitations

###  IX-A Challenges and Limitations

####  IX-A1 Adapting to Sophisticated Phishing Techniques

The increasing sophistication of phishing attacks, especially those enhanced
by AI, presents a major challenge for LLMs in cybersecurity. These models need
to evolve to identify and counteract these threats effectively continuously.
The challenge lies in the need for regular updates and training to keep pace
with the advanced tactics of attackers, which demands substantial resources
and expertise. For example, a large company implemented an LLM-based security
system to detect phishing emails. Initially, the system was highly effective,
identifying and blocking 95% of phishing attempts. However, attackers quickly
adapted, using AI to generate more convincing phishing emails that mimicked
the company’s official communication style and included personalized
information about the customers. The company’s LLM struggled to keep up with
these advanced tactics. Phishing emails have become so sophisticated that they
can bypass traditional detection methods, significantly increasing the number
of successful attacks. Hence, evolving and adapting LLMs in cybersecurity to
combat AI-enhanced phishing threats is an open challenge.

####  IX-A2 Managing Data Overload in Enterprise Applications

With the proliferation of enterprise applications, IT teams are overwhelmed by
the sheer volume of data they need to manage and secure, often without
corresponding increases in staffing. LLMs are expected to assist in managing
this data deluge efficiently. However, ensuring these models can process vast
amounts of data accurately and identify threats amidst this complexity is
daunting, necessitating high levels of efficiency and accuracy in the LLMs.
The corporation faced a situation where the LLM failed to recognize a
sophisticated cyber-attack hidden within the massive influx of data. This
oversight occurred because the model hadn’t been trained with the latest
attack patterns, highlighting a gap in its learning. The incident underscored
the need for LLMs to process data efficiently and maintain high accuracy and
adaptability in threat detection.

####  IX-A3 Training Data Availability and Quality

A critical challenge for AI-based cyber defense is the lack of high-quality,
accessible training data, as organizations generally hesitate to share
sensitive information. The effectiveness of LLMs in cybersecurity depends
heavily on the quality and availability of training data. Overcoming this data
gap remains a significant hurdle, whether through synthetic data generation or
other means.

####  IX-A4 Developing and Training Custom Models for Unique Cybersecurity
Domains

Certain specialized areas in cybersecurity require custom models due to their
unique vocabularies or data structures, which standard LLMs might not address
adequately. Unique Vocabularies and Data Structures: Cybersecurity domains,
such as network security, malware analysis, and threat intelligence, have
their terminologies, data formats, and communication protocols. Standard LLMs,
typically trained on general datasets, might not be familiar with these
specialized terms and structures, leading to ineffective or inaccurate threat
detection and response. Customizing and training these models to handle
specific cybersecurity scenarios is complex and demands substantial resources,
presenting a significant challenge in the field.

####  IX-A5 Real-Time Information Provision by Security Copilots

Security copilots powered by LLMs need to provide accurate, up-to-date
information in real-time to be effective in the dynamic threat landscape of
cybersecurity. Ensuring the relevance and accuracy of information provided by
these models in real-time is challenging but essential for effective responses
to cybersecurity threats.

###  IX-B LLM Cybersecurity Insights

Table XV presents various facets of LLM integration into cybersecurity,
providing insights into architectural nuances, dataset creation, pre-training,
fine-tuning methodologies, evaluation metrics, advanced techniques, deployment
strategies, security measures, and optimization approaches.

####  IX-B1 LLM architecture

A cyber security scientist venturing into utilizing LLMs must understand the
architecture’s nuances (presented in Section III) to tailor these tools for
security applications effectively. Understanding the architecture of LLMs,
including their ability to process and generate language-based data, is
crucial for detecting phishing attempts, deciphering malicious scripts, or
identifying unusual patterns in network traffic that may indicate a breach.
Knowledge of how these models tokenize input data, their attention mechanisms
to weigh information, and their output generation techniques provide the
foundational skills necessary to tweak models for optimized threat detection
and response [194].

####  IX-B2 Building Cyber Security dataset

Building a robust cybersecurity dataset using LLMs involves generating and
refining intricate prompt-response pairs to mirror real-world cyber threats.
Employing synthetic data generation via the OpenAI API allows for diverse
cybersecurity scenarios, while advanced tools like Evol-Instruct [195] enhance
dataset quality by adding complexity and removing outdated threats. Techniques
such as regex filtering and removing near-duplicates ensure the data’s
uniqueness and relevance. In addition, familiarizing with various prompt
templates like Alpaca [196] is essential for structuring this data
effectively, ensuring that the LLM can be finely tuned to respond to the
nuanced landscape of cybersecurity challenges efficiently.

####  IX-B3 Pre-training models

Pre-training a model for cybersecurity tasks involves a complex and resource-
intensive process to prepare a language model to understand and predict
cybersecurity-specific content. This requires a massive dataset comprising
billions or trillions of tokens, which undergo rigorous processes like
filtering, tokenizing, and aligning with a pre-defined vocabulary to ensure
relevance and accuracy. Techniques such as causal language modeling, distinct
from masked language modeling, are employed, where the loss functions and
training methodologies, such as those used in Megatron-LM [197] or gpt-neox
[198], are optimized for handling sequential data predictively. Understanding
the scaling laws is crucial, as these laws help predict how increases in model
size, dataset breadth, and computational power can proportionally enhance
model performance [199]. While in-depth knowledge of High-Performance
Computing (HPC) isn’t necessary for using pre-trained models, it becomes
essential when building a large-scale language model for cyber security from
scratch, requiring an understanding of hardware capabilities and managing
distributed workloads effectively.

Most pre-training LLM models are trained using smdistributed libraries,
proposed by AWS SageMaker, which offer robust solutions for distributed
training machine learning models, enhancing efficiency on large-scale
deployments. The smdistributed.dataparallel library supports data parallelism,
optimizing GPU usage by partitioning the training data across multiple GPUs,
thus speeding up the learning process and minimizing communication overhead.
On the other hand, smdistributed.modelparallel is tailored for model
parallelism, allowing large models to be split across multiple GPUs when a
single model cannot fit into the memory of one GPU. These tools seamlessly
integrate with frameworks like TensorFlow, PyTorch, and MXNet, simplifying the
implementation of complex distributed training tasks.

####  IX-B4 Supervised Fine-Tuning

Supervised fine-tuning (SFT) of pre-trained Large Language Models for
cybersecurity applications enables these models to move beyond basic next-
token prediction tasks, transforming them into specialized tools tailored to
specific cybersecurity needs. This fine-tuning process allows for
incorporating proprietary or novel datasets that have not been previously
exposed to models like Falcon 180b, providing a significant edge in addressing
unique security challenges. Figure 8 outlines a comprehensive three-step
process for training a large language model specialized in cybersecurity,
beginning with unsupervised pre-training on a vast corpus of cybersecurity-
related texts, including diverse data such as malware, network security, and
dark web content. Following this, the model undergoes traditional fine-tuning
using a smaller, targeted dataset to refine its capabilities for specific
cybersecurity tasks. However, the Parameter-Efficient Fine-Tuning (PEFT) [200]
involves freezing the original model weights and fine-tuning a small set of
new parameters, enhancing the model’s adaptability and efficiency while
minimizing the risk of overfitting, thus preparing the LLM to tackle advanced
cybersecurity challenges efficiently.

Techniques such as LoRA (Low-rank Adapters) [201] offer a parameter-efficient
approach by adjusting only a subset of the model’s parameters, thus optimizing
computational resources while maintaining performance. More advanced methods
like QLoRA [202] enhance this by quantizing the model’s weights and managing
memory more efficiently, making executing these operations even on limited
platforms like Google Colab with only one GPU A100. In addition, tools like
Axolotl and DeepSpeed [203, 204] facilitate the deployment of these fine-tuned
models across various hardware setups, ensuring that the enhanced models can
be scaled efficiently for real-world cybersecurity tasks, ranging from
intrusion detection to real-time threat analysis. This strategic fine-tuning
enhances model specificity and significantly boosts their utility in practical
cybersecurity applications.

TABLE XVI: Comparison of Benchmarks for Evaluating LLMs in Cybersecurity Knowledge Benchmark |  Source |  Year |  Description |  Key Features and Metrics
---|---|---|---|---
CyberSecEval 1 |  Meta [205] |  2023 |  A benchmark tests LLMs across two critical security domains—generating insecure code and compliance with requests to assist in cyberattacks. |  It measures the frequency and conditions LLMs propose insecure code solutions under.
SecQA |  Liu et al. [206] |  2023 |  A dataset of multiple-choice questions designed to evaluate the performance of LLMs in computer security. Features two versions of varying complexity and tests LLMs in both 0-shot and 5-shot learning settings. |  Evaluates understanding and application of security principles.
CyberMetric |  Tihanyi et al. [89] |  2024 |  A dataset designed for evaluating LLMs in cybersecurity knowledge, consisting of 10,000 questions from various authoritative sources. Used to measure the spectrum of cybersecurity topics covered by LLMs. |  Direct comparison between human expertise and LLMs.
CyberSecEval 2 |  Meta [207] |  2024 |  Focuses on quantifying security risks associated with LLMs, such as prompt injection and code interpreter abuse. It highlights challenges in mitigating attack risks and introduces the False Refusal Rate (FRR) metric. |  Testing areas: prompt injection, code interpreter abuse; Metric: FRR.
WMDP-Cyber |  Li et al. [208] |  2024 |  Consists of 3,668 multiple-choice questions designed to measure LLMs’ knowledge in biosecurity, cybersecurity, and chemical security. Excludes sensitive and export-controlled information. |  Covers biosecurity, cybersecurity, and chemical security.
LLM4Vuln |  Sun et al. [209] |  2024 |  A unified evaluation framework for assessing the vulnerability reasoning capabilities of LLMs, using 75 verified high-risk smart contract vulnerabilities in 4,950 scenarios across three LLMs. |  Focuses on vulnerability reasoning in LLMs.
CyberBench |  Liu et al. [210] |  2024 |  A domain-specific, multi-task benchmark for assessing LLM performance in cybersecurity tasks. |  Includes diverse tasks such as vulnerability detection, threat analysis, and incident response.

####  IX-B5 Cyber Security Evaluation

To evaluate the code generation models, Hugging Face uses the following 7 code
generation Python tasks: DS-1000, MBPP, MBPP+, APPS, InstructHumanEval,
HumanEval+, and HumanEval [211, 212, 213, 214, 215, 216, 217].

In cybersecurity, evaluating large language models demands a specialized
framework considering such applications’ unique security and accuracy needs.
When setting up evaluation metrics for cybersecurity-focused LLMs, test cases
should closely mimic potential security scenarios to assess how well the model
detects, understands, and responds to cyber threats. This involves configuring
the LLM with tailored inputs, expected outputs, and security-specific
contextual data [26]. For instance, IBM’s D2A dataset [161] and Microsoft’s
dataset [218] aids in evaluating AI models’ capability to identify software
vulnerabilities using specific metrics such as accuracy.

Table XVI compares benchmarks for evaluating LLMs in cybersecurity knowledge.
CyberMetric [89] is a benchmark dataset designed explicitly for evaluating
large language models in knowledge cybersecurity. It consists of 10,000
questions derived from various authoritative sources within the field. The
dataset is used to measure the knowledge of LLMs across a spectrum of
cybersecurity topics, facilitating direct comparisons between human expertise
and machine capabilities. This unique dataset aids in understanding the
strengths and limitations of LLMs in cybersecurity, providing a foundation for
further development and specialized training of these models in this critical
area.

Similar to the CyberMetric benchmark, Meta proposed the CyberSecEval 2
benchmark [207] to quantify security risks associated with LLMs such as GPT-4
and Meta Llama 3 70B-Instruct. They highlight new testing areas, notably
prompt injection and code interpreter abuse, revealing that mitigating attack
risks in LLMs remains challenging, with significant rates of successful prompt
injections. The study also explores the safety-utility tradeoff, proposing the
False Refusal Rate (FRR) to measure how conditioning LLMs to avoid unsafe
prompts might reduce their overall utility by also rejecting benign requests.
Additionally, the research assesses LLMs’ capabilities in automating core
cybersecurity tasks, suggesting that models with coding abilities perform
better in exploit generation tasks. The benchmark code is open source to
facilitate further research 222https://github.com/meta-
llama/PurpleLlama/tree/main/CybersecurityBenchmarks.

Liu et al. introduced SecQA [206], a novel dataset designed to evaluate the
performance of LLMs in computer security. The dataset, generated by GPT-4,
consists of multiple-choice questions to assess LLMs’ understanding and
application of security principles. SecQA features two versions with varying
complexity to challenge LLMs across different difficulty levels. The authors
comprehensively evaluated prominent LLMs, including GPT-3.5-Turbo, GPT-4,
Llama-2, Vicuna, Mistral, and Zephyr, in both 0-shot and 5-shot learning
settings. The findings from the SecQA v1 and v2 datasets reveal diverse
capabilities and limitations of these models in handling security-related
content. Li et al. [208] introduced the Weapons of Mass Destruction Proxy
(WMDP) benchmark. This publicly available dataset consists of 3,668 multiple-
choice questions designed to measure LLMs’ knowledge in biosecurity,
cybersecurity, and chemical security, ensuring the exclusion of sensitive and
export-controlled information. Sun et al. [209] introduced LLM4Vuln, a unified
evaluation framework designed to precisely assess the vulnerability reasoning
capabilities of LLMs independent of their other functions such as information
seeking, knowledge adoption, and structured output. This framework aims to
determine how enhancing these separate capabilities could boost LLMs’
effectiveness in identifying vulnerabilities. To test the efficacy of
LLM4Vuln, controlled experiments were conducted with 75 verified high-risk
smart contract vulnerabilities sourced from Code4rena audits conducted between
August and November 2023. These vulnerabilities were tested in 4,950 scenarios
across three LLMs: GPT-4, Mixtral, and Code Llama.

TABLE XVII: Optimization Strategies for Large Language Models in Cybersecurity Optimization Strategy |  Description |  Key Benefits |  Cybersecurity Use Case Scenarios
---|---|---|---
Advanced Attention Mechanisms |  Implements techniques like Flash Attention [219] to optimize self-attention layers, reducing computation times, particularly effective for long input sequences. |  Speeds up processing saves compute resources. |  Efficient processing of long log files and network traffic data for anomaly detection.
Bitsnbytes |  Introduces k-bit quantization (notably 8-bit) using block-wise methods to maintain performance while halving memory usage. |  Halves memory usage without loss in performance. |  Efficient real-time malware analysis and intrusion detection on edge devices.
GPTQ [220] |  A novel quantization method for GPT models that reduces bit width to 3 or 4 bits per weight, enabling the operation of large models on single GPUs with minimal accuracy loss. |  Compresses model size, minimizes accuracy loss. |  Deploying large-scale threat prediction models on consumer-grade hardware.
GGUF Quantization |  Optimized for quick model loading and saving, making LLM inference more efficient. Supported by Hugging Face Hub. |  Enhances efficiency of model deployment. |  Rapid deployment of updated models to respond to emerging threats and vulnerabilities.
QLoRA [202] |  Enables training using memory-saving techniques with a small set of trainable low-rank adaptation weights. |  Preserves performance with reduced memory. |  Training complex cybersecurity models on systems with limited memory resources.
Lower-precision data Formats |  Uses formats like bfloat16 instead of float32 for training and inference to optimize resource usage without compromising performance accuracy. |  Reduces computational overhead. |  Enhancing the speed and efficiency of continuous cybersecurity monitoring systems.
FSDP-QLoRA |  Combines Fully Sharded Data Parallelism (FSDP) with 4-bit quantization and LoRA to shard model parameters, optimizer states, and gradients across GPUs. |  Scales up model training across multiple GPUs. |  Enabling the collaborative training of security models across different organizations without requiring top-tier hardware.
Half-Quadratic Quantization (HQQ) [221] |  A model quantization technique that enables the quantization of large models rapidly and accurately without the need for calibration data. |  Works efficiently with CUDA/Triton kernels and aims for seamless integration with $torch.compile$. |  HQQ can be employed in cybersecurity to protect models by reducing the precision of model weights, making it harder for attackers to reverse engineer or tamper with the models..
Multi-token Prediction [222] |  A new training approach for large language models where models predict multiple future tokens simultaneously rather than the next token only. |  Models trained with 4-token predictions can achieve up to 3x faster inference speeds, even with large batch sizes. |  Multi-token prediction can enhance the modeling of sophisticated cyber attack patterns.
Trust Region Policy Optimization (TRPO) [223] |  An advanced policy gradient method in reinforcement learning that addresses the inefficiencies of standard policy gradient methods. |  TRPO enhances training stability by using trust regions to prevent overly large updates that could destabilize the policy. |  In environments with dynamic and evolving threats, TRPO can help maintain a stable and effective response mechanism, adjusting policies incrementally to handle new types of malware.
Proximal Policy Optimization (PPO) [224] |  A reinforcement learning technique designed to improve training stability by cautiously updating policies. |  Prevents ”falling off the cliff” scenarios where a policy update is too large could irreversibly damage the policy’s effectiveness. |  By limiting the extent of policy updates, PPO helps maintain a steady adaptation to evolving cybersecurity threats, reducing the risk of overfitting to specific attack patterns.
Direct Preference Optimization (DPO) [225] |  A fine-tuning methodology for foundation models optimize policies directly using a Kullback–Leibler divergence-constrained framework, removing the need for a separate reward model. |  Requires significantly less data and compute resources than previous methods like PPO. |  Reduces the computational and data demands of continuously training cybersecurity models, allowing for more scalable solutions.
Odds Ratio Preference Optimization (ORPO) [226] |  An algorithm designed for supervised fine-tuning (SFT) of language models that optimizes preference alignment without the need for a separate reference model. |  Eliminates the need for an additional preference alignment phase, simplifying the fine-tuning process. |  Enables dynamic adaptation of security models to new and evolving cyber threats by optimizing preference alignment efficiently.

####  IX-B6 Advanced LLM techniques (RAG and RLHF)

Advanced techniques like Retrieval Augmentation Generation (RAG) can
significantly enhance Language Model performance by enabling the model to
access external databases for additional context and information, making it
highly effective in specialized fields such as cybersecurity. In cybersecurity
applications, RAG can dynamically retrieve up-to-date information from well-
known databases such as CVE (Common Vulnerabilities and Exposures), CWE
(Common Weakness Enumeration), and the NIST (National Institute of Standards
and Technology) database [227]. This capability allows the model to offer
current and specific advice regarding vulnerabilities, threat intelligence,
and compliance standards. Integrating real-time data from these authoritative
sources into the response generation process allows RAG to empower Language
Models to deliver precise and contextually relevant cybersecurity insights
without extensive retraining, thus enhancing decision-making in critical
security operations [228].

Reinforcement Learning from Human Feedback (RLHF) is an advanced method to
enhance LLMs tailored for cybersecurity applications, focusing on aligning the
model’s responses with expert expectations in the security domain. This
involves utilizing specialized preference datasets, which contain responses
ranked by cybersecurity professionals, presenting a more challenging
production process than typical instructional datasets. Techniques like
Proximal Policy Optimization (PPO) leverage a reward model to evaluate how
well text outputs align with security expert rankings, refining the model’s
training through adjustments based on KL divergence [224]. Direct Preference
Optimization (DPO) further optimizes this by framing it as a classification
challenge, using a stable reference model that avoids the complexities of
training reward models and requires minimal hyperparameter adjustments [229].
These methods are crucial for reducing biases, fine-tuning threat detection
accuracy, and enhancing the overall effectiveness of cybersecurity-focused
LLMs.

In practical cybersecurity applications, the integration of RAG can be
facilitated by orchestrators like LangChain, LlamaIndex, and FastRAG, which
connect Language Models to relevant tools, databases, and other resources.
These orchestrators ensure efficient information flow, enabling Language
Models to seamlessly access and incorporate essential cybersecurity
information [230]. Advanced techniques such as multi-query retrievers and HyDE
are used to optimize the retrieval of relevant cybersecurity documents and
adapt user queries into more effective forms for document retrieval.
Furthermore, incorporating a memory system that recalls previous interactions
allows these models to provide consistent and context-aware responses over
time. This amalgamation of advanced retrieval mechanisms and memory
enhancement through RAG significantly boosts the efficacy of Language Models
in handling complex and evolving cybersecurity challenges, making them
invaluable tools for tracking vulnerabilities, managing risks, and adhering to
industry standards in the cybersecurity domain [231].

####  IX-B7 LLM deployments

Deploying LLMs offers a range of approaches tailored to the scale and specific
needs of different applications. At one end of the spectrum, local deployment
offers enhanced privacy and control, utilizing platforms like LM Studio and
Ollama to power apps directly on users’ machines, thus capitalizing on the
open-source nature of some LLMs. For more dynamic or temporary setups,
frameworks such as Gradio and Streamlit allow developers to prototype and
share demos quickly, with hosting options like Hugging Face Spaces providing
an accessible path to broader distribution. On the industrial scale, deploying
LLMs can require robust server setups, utilizing cloud services or on-premises
infrastructure that might leverage specialized frameworks for peak performance
and efficiency. Meanwhile, edge deployment strategies bring LLM capabilities
to devices with limited resources, using advanced, lightweight frameworks to
integrate smart capabilities directly into mobile and web platforms, ensuring
responsiveness and accessibility across a broad spectrum of user environments
[232, 233].

Currently, LLMs can be deployed on Phones. Microsoft [138] propose phi-3-mini.
This highly efficient 3.8 billion parameter language model delivers robust
performance on par with much larger models such as Mixtral 8x7B and GPT-3.5,
achieving impressive scores like 69% on the MMLU and 8.38 on MT-bench.
Remarkably, the phi-3-mini’s compact size allows for deployment on mobile
devices, expanding its accessibility and utility. This performance
breakthrough is primarily attributed to an innovative approach in training
data selection—a significantly enhanced version of the dataset used for phi-2,
which integrates heavily filtered web data and synthetic data tailored for
relevance and diversity. It has been further aligned to ensure the model’s
practicality in real-world applications for enhanced robustness, safety, and
optimization for chat formats. In addition, the research extends into larger
models, phi-3-small and phi-3-medium, which are trained on 4.8 trillion tokens
with 7 billion and 14 billion parameters, respectively. These models retain
the foundational strengths of phi-3-mini and exhibit superior performance,
scoring up to 78% on MMLU and 8.9 on MT-bench, illustrating significant
enhancements in language understanding capabilities with scaling. In addition,
AirLLM 333https://pypi.org/project/airllm/ enhances memory management for
inference, enabling large language models, such as those with 70 billion
parameters (e.g., Llama3 70B), to operate on a single 4GB GPU card. This can
be achieved without requiring quantization, distillation, pruning, or any
other form of model compression that could diminish performance.

####  IX-B8 Securing LLMs

Securing LLMs is essential due to their inherent susceptibility to traditional
software vulnerabilities and unique risks stemming from their design and
operational methods. Specifically, LLMs are prone to prompt hacking, where
techniques such as prompt injection can be used to manipulate model responses,
prompt leaking that risks exposure of training data, and jailbreaking intended
to circumvent built-in safety mechanisms. These specific threats necessitate
implementing comprehensive security measures that directly address the unique
challenges LLMs pose. Additionally, inserting backdoors during training,
either by poisoning the data or embedding secret triggers, can significantly
alter a model’s behavior during inference, posing severe risks to data
integrity and model reliability.

As discussed in Section VIII, to mitigate these threats effectively,
organizations must adopt rigorous defensive strategies as recommended by the
OWASP LLM security checklist 444https://owasp.org/www-project-top-10-for-
large-language-model-applications/. This includes testing LLM applications
against known vulnerabilities using methods like red teaming and specific
tools such as garak [234] to identify and address security flaws. In addition,
deploying continuous monitoring systems like langfuse
555https://github.com/langfuse/langfuse in production environments helps
detect and rectify anomalous behaviors or potential breaches in real-time. The
OWASP checklist also emphasizes the importance of governance frameworks that
ensure data used in training is ethically sourced and handled, maintaining
transparency about data sources and model training methodologies. This
structured approach to security and governance ensures that LLMs are used
responsibly and remain secure from conventional cyber threats and those unique
to their operational nature.

####  IX-B9 Optimizing LLMs

Optimizing LLMs for production encompasses several crucial techniques to
enhance speed, reduce memory requirements, and maintain output quality. One
pivotal strategy is model quantization, which significantly reduces the
precision of model weights—often to 4-bit or 8-bit—thereby decreasing the GPU
memory requirements. Table XVII presents the optimization strategies for LLMs
that can be adopted for Cybersecurity use cases. For instance, quantizing a
model to 4-bit can bring down the VRAM requirement from 32 GB to just over 9
GB, allowing these models to run efficiently on consumer-level hardware like
the RTX 3090 GPU. Therefore, advanced attention mechanisms such as Flash
Attention reduce computation times by optimizing self-attention layers, which
are integral to transformers [219]. This optimization is especially beneficial
for handling long input sequences, where traditional self-attention mechanisms
could become prohibitively expensive regarding memory and processing power
[235, 236].

The quantization methods include Bitsnbytes, 4-bit GPTQ, 2-bit GPTQ, and GGUF
quantization. Bitsnbytes introduces a k-bit quantization approach that
significantly reduces memory consumption while maintaining performance [220].
It employs an 8-bit optimization using block-wise quantization to achieve
32-bit performance at a lower memory cost and uses LLM.Int() for 8-bit
quantization during inference, halving the required memory without performance
loss. Furthermore, QLoRA [202], or 4-bit quantization, enables the training of
LLMs using memory-saving techniques that include a small set of trainable low-
rank adaptation weights, allowing for performance preservation. In parallel,
GPTQ is a novel quantization method for GPT models, facilitating the reduction
of bit width to 3 or 4 bits per weight, enabling the operation of models as
large as 175 billion parameters on a single GPU with minimal accuracy loss.
This method provides substantial compression and speed advantages, making
high-performance LLMs more accessible and cost-effective. Additionally, the
GGUF format, supported by Hugging Face Hub and optimized for quick model
loading and saving, enhances the efficiency of LLM inference.

Another effective optimization is incorporating lower-precision data formats
such as bfloat16 for training and inference. This approach aligns with the
training precision and avoids the computational overhead associated with
float32 precision, optimizing resource usage without compromising performance
accuracy. The potential VRAM requirements for different models using bfloat16
are substantial. For example, GPT-3 might require up to 350 GB. In comparison,
smaller models like Llama-2-70b and Falcon-40b require 140 GB and 80 GB,
respectively, illustrating the scale of resources needed even with efficient
data formats
666https://huggingface.co/docs/transformers/main/en/llm_tutorial_optimization.

Recently, FSDP-QLoRA
777https://huggingface.co/docs/bitsandbytes/main/en/fsdp_qlora, a new
technique combining data parallelism, 4-bit quantization, and LoRA, was
introduced by Answer.AI in collaboration with bitsandbytes. Utilizing Fully
Sharded Data Parallelism (FSDP) to shard model parameters, optimizer states,
and gradients across GPUs, this approach enables the training of LLMs up to 70
billion parameters on dual 24GB GPU systems. FSDP-QLoRA represents a
significant step forward in making the training of large-scale LLMs.

Collectively, these techniques make it feasible to deploy powerful LLMs on a
wider range of hardware and enhance their scalability and practicality in
diverse applications, ensuring they can deliver high performance even under
hardware constraints.

##  X Conclusion

In this paper, we presented a comprehensive and in-depth review of the future
of cybersecurity through the lens of Generative AI and Large Language Models
(LLMs). Our exploration covered a wide range of LLM applications in
cybersecurity, including hardware design security, intrusion detection,
software engineering, design verification, cyber threat intelligence, malware
detection, and phishing and spam detection, illustrating the broad potential
of LLMs across various domains.

We provided a detailed examination of the evolution and current state of LLMs,
highlighting advancements in 35 specific models, such as GPT-4, GPT-3.5, BERT,
Falcon, and LLaMA. Our analysis included an in-depth look at the
vulnerabilities associated with LLMs, such as prompt injection, insecure
output handling, training and inference data poisoning, DDoS attacks, and
adversarial natural language instructions. We discussed mitigation strategies
to protect these models, offering a thorough understanding of potential attack
scenarios and prevention techniques.

Our evaluation of 40 LLM models in terms of cybersecurity knowledge and
hardware security demonstrated their varying strengths and weaknesses. We also
conducted a detailed assessment of cybersecurity datasets used for LLM
training and testing, from data creation to usage, identifying gaps and
opportunities for future research.

We addressed the challenges and limitations of employing LLMs in cybersecurity
settings, including the difficulty of defending against adversarial attacks
and ensuring model robustness. Additionally, we explored advanced techniques
like Half-Quadratic Quantization (HQQ), Reinforcement Learning with Human
Feedback (RLHF), Direct Preference Optimization (DPO), Odds Ratio Preference
Optimization (ORPO), GPT-Generated Unified Format (GGUF), Quantized Low-Rank
Adapters (QLoRA), and Retrieval-Augmented Generation (RAG) to enhance real-
time cybersecurity defenses and improve the sophistication of LLM applications
in threat detection and response.

Our findings underscore the significant potential of LLMs in transforming
cybersecurity practices. By integrating LLMs into future cybersecurity
frameworks, we can leverage their capabilities to develop more robust and
sophisticated defenses against evolving cyber threats. The strategic direction
outlined in this paper aims to guide future research and deployment,
emphasizing the importance of innovation and resilience in safeguarding
digital infrastructures.

## References

  * [1] R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio, “How to construct deep recurrent neural networks,” _arXiv preprint arXiv:1312.6026_ , 2013.
  * [2] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [3] R. Dey and F. M. Salem, “Gate-variants of gated recurrent unit (gru) neural networks,” in _2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS)_. IEEE, 2017, pp. 1597–1600.
  * [4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [5] B. Yan, K. Li, M. Xu, Y. Dong, Y. Zhang, Z. Ren, and X. Cheng, “On protecting the data privacy of large language models (llms): A survey,” _arXiv preprint arXiv:2403.05156_ , 2024.
  * [6] D. Myers, R. Mohawesh, V. I. Chellaboina, A. L. Sathvik, P. Venkatesh, Y.-H. Ho, H. Henshaw, M. Alhawawreh, D. Berdik, and Y. Jararweh, “Foundation and large language models: fundamentals, challenges, opportunities, and social impacts,” _Cluster Computing_ , vol. 27, no. 1, pp. 1–26, 2024.
  * [7] S. Tonmoy, S. Zaman, V. Jain, A. Rani, V. Rawte, A. Chadha, and A. Das, “A comprehensive survey of hallucination mitigation techniques in large language models,” _arXiv preprint arXiv:2401.01313_ , 2024.
  * [8] I. H. Sarker, H. Janicke, M. A. Ferrag, and A. Abuadbba, “Multi-aspect rule-based ai: Methods, taxonomy, challenges and directions toward automation, intelligence and transparent cybersecurity modeling for critical infrastructures,” _Internet of Things_ , p. 101110, 2024.
  * [9] Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,” _High-Confidence Computing_ , p. 100211, 2024.
  * [10] Y. Yan, Y. Zhang, and K. Huang, “Depending on yourself when you should: Mentoring llm with rl agents to become the master in cybersecurity games,” _arXiv preprint arXiv:2403.17674_ , 2024.
  * [11] M. Sladić, V. Valeros, C. Catania, and S. Garcia, “Llm in the shell: Generative honeypots,” _arXiv preprint arXiv:2309.00155_ , 2023.
  * [12] W. Tann, Y. Liu, J. H. Sim, C. M. Seah, and E.-C. Chang, “Using large language models for cybersecurity capture-the-flag challenges and certification questions,” _arXiv preprint arXiv:2308.10443_ , 2023.
  * [13] O. G. Lira, A. Marroquin, and M. A. To, “Harnessing the advanced capabilities of llm for adaptive intrusion detection systems,” in _International Conference on Advanced Information Networking and Applications_. Springer, 2024, pp. 453–464.
  * [14] C. Ebert and M. Beck, “Artificial intelligence for cybersecurity,” _IEEE Software_ , vol. 40, no. 6, pp. 27–34, 2023.
  * [15] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,” _IEEE Transactions on Software Engineering_ , 2024.
  * [16] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, É. Goffinet, D. Hesslow, J. Launay, Q. Malartic _et al._ , “The falcon series of open language models,” _arXiv preprint arXiv:2311.16867_ , 2023.
  * [17] H. Zhou, C. Hu, Y. Yuan, Y. Cui, Y. Jin, C. Chen, H. Wu, D. Yuan, L. Jiang, D. Wu, X. Liu, C. Zhang, X. Wang, and J. Liu, “Large language model (llm) for telecommunications: A comprehensive survey on principles, key techniques, and opportunities,” 2024.
  * [18] H. Lai and M. Nissim, “A survey on automatic generation of figurative language: From rule-based systems to large language models,” _ACM Computing Surveys_ , 2024.
  * [19] M. A. Ferrag, M. Ndhlovu, N. Tihanyi, L. C. Cordeiro, M. Debbah, T. Lestable, and N. S. Thandi, “Revolutionizing cyber threat detection with large language models: A privacy-preserving bert-based lightweight model for iot/iiot devices,” _IEEE Access_ , 2024.
  * [20] Z. Liu, “A review of advancements and applications of pre-trained language models in cybersecurity,” in _2024 12th International Symposium on Digital Forensics and Security (ISDFS)_ , 2024, pp. 1–10.
  * [21] S. Jamal, H. Wimmer, and I. H. Sarker, “An improved transformer-based model for detecting phishing, spam and ham emails: A large language model approach,” _Security and Privacy_ , p. e402, 2024. [Online]. Available: https://doi.org/10.1002/spy2.402
  * [22] F. R. Alzaabi and A. Mehmood, “A review of recent advances, challenges, and opportunities in malicious insider threat detection using machine learning methods,” _IEEE Access_ , vol. 12, pp. 30 907–30 927, 2024.
  * [23] R. Fang, R. Bindu, A. Gupta, and D. Kang, “Llm agents can autonomously exploit one-day vulnerabilities,” _arXiv preprint arXiv:2404.08144_ , 2024.
  * [24] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong _et al._ , “A survey of large language models,” _arXiv preprint arXiv:2303.18223_ , 2023.
  * [25] M. A. K. Raiaan, M. S. H. Mukta, K. Fatema, N. M. Fahad, S. Sakib, M. M. J. Mim, J. Ahmad, M. E. Ali, and S. Azam, “A review on large language models: Architectures, applications, taxonomies, open issues and challenges,” _IEEE Access_ , vol. 12, pp. 26 839–26 874, 2024.
  * [26] Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang _et al._ , “A survey on evaluation of large language models,” _ACM Transactions on Intelligent Systems and Technology_ , 2023.
  * [27] D. Saha, S. Tarek, K. Yahyaei, S. K. Saha, J. Zhou, M. Tehranipoor, and F. Farahmandi, “Llm for soc security: A paradigm shift,” _arXiv preprint arXiv:2310.06046_ , 2023.