# Securing an Application Layer Gateway:
An Industrial Case Study

Carmine Cesarano, Roberto Natella  Università degli Studi di Napoli Federico
II, Italy
{carmine.cesarano2<EMAIL_ADDRESS>
###### Abstract

Application Layer Gateways (ALGs) play a crucial role in securing critical
systems, including railways, industrial automation, and defense applications,
by segmenting networks at different levels of criticality. However, they
require rigorous security testing to prevent software vulnerabilities, not
only at the network level but also at the application layer (e.g., deep
traffic inspection components). This paper presents a vulnerability-driven
methodology for the comprehensive security testing of ALGs. We present the
methodology in the context of an industrial case study in the railways domain,
and a simulation-based testing environment to support the methodology.

###### Index Terms:

Application Layer Gateway, Security Testing, Virtualized Testing Environment

##  I Introduction

The interconnection of diverse applications and networks is essential for
building complex systems. Application Layer Gateways (ALGs) are responsible
for handling the communication between different networks, with different
levels of criticality, and for performing security controls. In particular,
ALGs are crucial in critical domains such as railways, avionics, and defense.
In these domains, network segments for industrial control (e.g., TDRP traffic
in the railways domains) should be protected from attacks arising from other
network segments (e.g., multimedia traffic over HTTP, FTP, or XMPP); at the
same time, communication between network segments should be allowed for
specific cases (e.g., for diagnostic purposes). The ALG performs security
controls, by analyzing the traffic at the application layer to enforce
security policies (e.g., based on the contents and on the authors of the
data).

Given the role of ALGs in such critical domains, there is a need for
methodologies for rigorous security testing, in order to prevent software
vulnerabilities that could allow attackers to escape security controls.
Security standards, such as Common Criteria [1], define security functional
requirements (SFR) catalogs for several software categories, including ALGs
[2]. The same standard also defines the security assurance requirements (SARs)
for specific types of software. Although SARs can be useful to assess software
security, they are broadly expressed and lack completeness for ALG systems
[2]. Furthermore, in the literature, no comprehensive methodologies have been
proposed for the assessment of ALG systems. Existing toolkits and techniques
focus on network layer attacks (e.g., IP and port scanning [3], IP and MAC
spoofing, or frameworks like BurpSuite [4] and ZAP [5]), leaving out the
application layer.

In this work, we present a novel methodology for assessing the security of ALG
systems. Our proposal offers a comprehensive approach that addresses not only
network-layer vulnerabilities but also delves into the application layer,
where ALGs mainly operate. For example, our tests include attack scenarios
involving techniques like fuzzing and stress testing specifically tailored to
trigger vulnerabilities at the application layer. The contributions of this
paper include:

  * •

We conduct an in-depth analysis of potential ALG system weaknesses and propose
an assessment methodology for their systematic identification.

  * •

We develop a simulation-based testing environment for performing ALG systems
evaluations, improving the test reproducibility and system observability.

  * •

We present a practical case study in the railways domain, demonstrating the
application of our methodology to assess the security of an ALG system built
on open-source components.

The proposed methodology offers systematic guidance for assessing ALG
products, covering network and application layers in a holistic testing
framework. The methodology is adaptable to different ALG systems, while the
practical case study is meant to demonstrate its applicability in real
systems.

In the following of this paper, Section II provides an overview of related
work concerning ALGs, Section III presents the proposed methodology, Section
IV introduces the case study, Section V outlines the experimental analysis,
and Section VI summarizes findings and outlines future research directions.

##  II Related work

TABLE I: CWEs and Attack Scenarios in ALGs and Firewall software CWE-ID |  Description |  Attack Scenario |  Occurred in
---|---|---|---
CWE-20 |  Improper Input Validation |  Crafted DNS, RTSP packets |  Cisco ASA, Cisco FTD
CWE-22 |  Improper Limitation of a Pathname to a Restricted Directory |  Crafted HTTP requests |  HAProxy, Pfsense
CWE-74 |  Improper Neutralization of characters (’Injection’) |  Encapsulation attack (via HTTP req) |  HAProxy
CWE-78 |  Improper Neutralization of characters (’OS Command Injection’) |  OS command injection (via HTTP req) |  Haproxy, Pfsense
CWE-79 |  Improper Neutralization of characters (’Cross-site Scripting’) |  Crafted HTTP requests |  Pfsense
CWE-91 |  XML Injection (aka Blind XPath Injection) |  Manipulation of configuration XML files |  Pfsense
CWE-120 |  Buffer Copy without Checking Size of Input (’Buffer Overflow’) |  Specific FTP transfer |  Cisco IOS
CWE-190 |  Integer Overflow or Wraparound |  HTTP Request Smuggling |  HAProxy
CWE-200 |  Exposure of Sensitive Information to an Unauthorized Actor |  Log Data Extraction |  HAProxy
CWE-281 |  Improper Preservation of Permissions |  Crafted FTP commands |  Pfsense
CWE-290 |  Authentication Bypass by Spoofing |  IP and MAC spoofing |  PfSense
CWE-307 |  Improper Restriction of Excessive Authentication Attempts |  Brute Force to Authentication |  Pfsense
CWE-358 |  Improperly Implemented Security Check for Standard |  NAT Slipstreaming |  Cisco ASA, Cisco FTD
CWE-399 |  Resource Management Errors |  Crafted SIP, H.323 packets |  Cisco IOS
CWE-400 |  Uncontrolled Resource Consumption |  Resource Exhaustion Attacks |  F5 BIP-IP AFM
CWE-401 |  Missing Release of Memory after Effective Lifetime |  Crafted SIP packets |  Junos OS
CWE-434 |  Unrestricted Upload of Dangerous File |  Specific FTP transfer |  Cisco IOS
CWE-444 |  Inconsistent Interpretation of HTTP Requests |  HTTP Request/Response Smuggling |  HAProxy
CWE-459 |  Incomplete Cleanup |  Crafted HTTP requests |  HAProxy
CWE-665 |  Improper Initialization |  Crafted SIP packets |  Cisco IOS
CWE-693 |  Protection Mechanism Failure |  Crafted H.323 packet |  Cisco IOS
CWE-754 |  Improper Check for Unusual or Exceptional Conditions |  Crafted DNS packets |  Cisco IOS
CWE-755 |  Improper Handling of Exceptional Conditions |  Crafted HTTP request |  HAProxy
CWE-787 |  Out-of-bounds Write |  Crafted HTTP requests |  HAProxy
CWE-824 |  Access of Uninitialized Pointer |  Crafted SIP packets |  Junos OS
CWE-835 |  Loop with Unreachable Exit Condition (’Infinite Loop’) |  Crafted HTTP responses |  HAProxy
CWE-908 |  Use of Uninitialized Resource |  Resource Exhaustion Attacks |  F5 BIP-IP AFM

Many commercial and open-source systems have been proposed for ALGs. Most
notably, GoThings [6] proposes an extensible architecture that facilitates
interconnectivity between different messaging protocols such as REST, CoAP,
MQTT, and XMPP. However, even if these systems address connectivity challenges
in the IoT landscape, they are not primarily oriented toward the enforcement
of security policies.

Other products, instead, are specifically designed to ensure security policies
in communication. For example, we can mention Multiple Independent Levels of
Security (MILS) kernels [7], a high assurance architecture for handling
separate execution environments with different classification levels and
controlled information flow among them. MILS security guards [8] are
specialized components in secure MILS kernel communication, and can be
considered as an ALG.

Sec-ALG [9] prioritizes secure end-to-end communication between private
networks. It relies on Deep Packet Inspection for protocol detection and
packet filtering. Despite its emphasis on security policies, Sec-ALG does not
introduce methodologies for evaluating the effectiveness of its security
measures. This underscores the need for a comprehensive testing approach,
especially in critical environments where security is a key aspect.

ALGs aim to impose stringent security policies but can be susceptible to
sophisticated attack techniques like BALG [10], which uses polymorphic and
encrypted shellcodes to bypass security measures. However, this attack is
highly specific, and evaluating ALGs requires a testing methodology with
diverse attack vectors across various layers (e.g., network, application, and
policy).

Our paper introduces a comprehensive methodology for the security testing of
ALGs deployed in critical domains. This methodology has been designed to
address the limitations inherent in existing approaches, offering a systematic
evaluation of ALG security, including policy enforcement, network layer, and
application layer vulnerabilities.

##  III Methodology

The proposed methodology for assessing ALG security includes four key steps.
First, we conduct a thorough analysis of Common Weakness Enumeration (CWEs)
relevant to ALG systems. We then define attack scenarios using a CWE-driven
approach. Subsequently, we configure a simulated testing environment. Finally,
we conclude the security evaluation by executing the tests and analyzing the
results.

###  III-A CWE-driven risk analysis

The proposed methodology is driven by an initial risk analysis phase, which
includes an examination of Common Vulnerabilities and Exposures (CVEs)
affecting software products within the same category of ALG. Subsequently, we
retrieved corresponding CWEs for each identified CVE.

Table I presents the results of this phase, using commercial and open-source
software as a sample, including ALGs, proxies, and firewalls. For example,
CVE-2021-40346 (CWE-190) reveals an integer overflow in HAProxy, enabling
attackers to bypass configured Access Control Lists (ACLs). Similarly,
CVE-2019-12655 (CWE-20) discloses a buffer overflow, allowing a remote
attacker to trigger a Denial of Service (DoS). Due to space constraints, the
detailed mapping between CVEs and CWEs are available at
111https://dessert.unina.it:8088/carmain/edcc2024; only the aggregated CWE
results are presented in the Table. The CWEs listed in the table are
representative instances of common weaknesses that may affect ALGs. However,
they could certainly be augmented with additional analysis from ongoing
research on other products or emerging vulnerabilities. This preliminary
phase, starting with CVEs and progressing through CWEs, enables the building
of a focused knowledge base on potential risks associated with ALGs.
Consequently, it guides the phase of defining attack scenarios.

###  III-B Attack scenarios definition

In the next step of our methodology, we define the security tests required to
validate the security posture of ALG. By adopting a CWE-driven approach, we
can conduct security testing more focused on real-world weaknesses to which
the system may be susceptible. Using the insights gained from the CVE
analysis, we can extract details on the exploits associated with each
vulnerability, thus facilitating the formulation of the corresponding attack
scenario. The fourth column in Table I enumerates these inferred attack
scenarios. For example, according to the CVE description, an HTTP request
smuggling attack can be used to assess CVE-2019-12655 mentioned in Section
III-A. In the case of CVE-2019-12655, a crafted FTP traffic attack can be
employed. By executing all defined attack scenarios, we effectively assess all
vulnerabilities we identified during the risk analysis phase.

For each defined attack scenario, a set of testing techniques can be selected.
For example, we can select generation fuzzing to generate the FTP traffic
mentioned above or mutation fuzzing to mutate benign traffic sniffed from the
network to assess weaknesses at the application layer. We can select packet
manipulation and crafting tools to assess weaknesses at the network layer.
Additionally, we can select stress testing to trigger DoS vulnerabilities. We
further detail the employed testing techniques in Section V.

###  III-C Testing environment setup

In industrial domains, the assessment process for ALGs typically involves
deploying the software component in a real pre-production environment. This is
desirable, as the System Under Test (SUT) may exhibit different behaviors
depending on the operational environment. The closer these conditions align
with those of the production environment, the more effective security testing
will be in identifying vulnerabilities that could be exploited when the system
is in use. This process is not always feasible and may pose complexities.
Conducting ALG testing within the real system can potentially pose risks to
costly and safety-critical components. Furthermore, it is not guaranteed that
the physical system encompasses debugging tools for observing the SUT’s state.
Additionally, this approach does not align with the security-by-design
principles, which are crucial for security-critical and safety-critical
systems, where there is a need to apply security testing from the earliest
phases of the design process. To address these challenges, this work proposes
the use of a virtual testing environment. Using network simulation
technologies, we simulate the network infrastructure within which the SUT will
operate, including topology parameters such as subnets, routing
configurations, switches, and network protocols, as well as IP addresses, and
port configurations. Beyond all this, network simulation facilitates tasks
such as monitoring traffic and resources. Furthermore, we use lightweight
virtualization and hardware virtualization to emulate the network hosts.
During testing, hosts generate and exchange network packets through the ALG.
The testing environment also supports the integration of physical machines
into the simulated network infrastructure to accommodate non-virtualizable
hardware bundled with ALG. This approach is suitable for assessing the
security of third-party ALG products, whether open-source or commercial.

###  III-D Security Testing

In the last phase of our methodology, we implement and perform the security
tests according to the defined attack scenarios. During test execution, we
capture and record critical data, including the state of the SUT, system logs,
performance metrics, and network traffic using virtualized introspection
capabilities. In the final phase of our methodology, we analyze the behavior
traces and collected metrics to assess security effectiveness, identify
vulnerabilities, and provide recommendations for improvement.

##  IV Case study

Figure 1: On-board Train Network

In order to demonstrate the practical application of the proposed methodology
we present a case study involving the integration of an ALG into a real-world
network infrastructure, specifically an On-Board Train Network. The analyzed
case study stems from a collaboration with an Italian railways company.

###  IV-A On-board Train Network

Figure 1 shows the network topology under analysis. All on-board services and
functions are segregated from each other using VLANs, as described by the EN
50159 standard [11], which governs communication, signaling, and processing in
safety-related transmission systems. This segmentation is essential to ensure
the safety and reliability of the train’s operations. The on-board VLANs can
be categorized into three main groups: 1) VLANs for the Train Control and
Monitoring System (TCMS), 2) VLANs for train control services (maintenance,
audio, on-board intranet, train integrity, fire protection), and 3) VLANs for
passenger multimedia services. In each VLAN, there are one or more switches to
which the hosts are connected (CCTV cameras, IP cams, PCN sensors,
loudspeakers, etc.).

VLANs must be isolated, but at the same time some communications are allowed.
For this reason, the ALG is critical in ensuring the security of the
communication between VPNs with different levels of criticality. For example,
the train control service VLAN can transfer information to the passenger
multimedia service VLAN, and vice versa.

TABLE II: Security Policies Categories Security Policy |  Type
---|---
Network Isolation: Block all direct traffic between the two networks unless explicitly authorized |  Firewall
IP Filtering: Allow traffic only from authorized IP addresses in the low-criticality network. |  Firewall
Ports and Protocols: Allow only traffic based on specific protocols (e.g., HTTP, HTTPS, FTP) and block unnecessary protocols. |  Firewall
User/Role Access: Allow access to sensitive resources in the high-criticality network only for authorized users based on their roles. |  Application
Content Inspection: Apply Deep Packet Inspection to identify malicious payloads or anomalous behavior. |  Application
URL Filtering: Block access to URLs or domains known to be unauthorized. |  Application
Protocol-based and service-based routing: Route traffic based on the specific protocol or type of service to its designated servers in the high-criticality network. |  Routing
Bandwidth Limitations: Limit the available bandwidth for certain traffic categories to ensure fair use of network resources. |  Routing

###  IV-B ALG security policies

The ALG deployed in the on-board network mitigates security issues, including
information disclosure, data integrity, and DoS, by controlling traffic
between VLANs of varying levels of criticality. The ALG intercepts and
inspects each packet, applying policies to enforce security between the VLANs.

The first responsibility of the ALG is to provide firewall capabilities,
operating at the network level (Layer 3). Policies for filtering packets can
be based on IP addresses, ports, specific protocols, and service types. ALG
can restrict traffic from a specific set of source IPs and allow or block
certain protocols (e.g., HTTP and FTP) or prevent access to services (e.g.,
SMTP and POP3).

The second responsibility of the ALG is to enforce the application level
(Layer 7), securing the communication between hosts with different privilege
levels. For instance, the ALG can manage requests for privileged resources
using HTTP or FTP, maintaining a whitelist of authorized URLs. Policies based
on users or roles are enforced, allowing access only for authorized users.
Deep packet inspection is employed to identify malicious payloads containing
malware.

Additionally, the ALG provides separation between different VLANs, ensuring
unawareness of each other’s network topology and composition. Therefore, the
third responsibility of the ALG is to route traffic to the correct destination
host based on the requested protocol or service type.

For each of the ALG’s responsibilities, security policies falling under
different categories can be configured. Table II provides a list of these
categories, specifying the types including firewall, application, or routing.

##  V Experimental Analysis

Figure 2: On-board train network: simulation on GNS3

In this section, we will present the simulated network setup, the deployment
and configuration of an open-source ALG, and its security assessment by using
the proposed methodology.

TABLE III: ALG Security Policies and Attack Scenarios CWE |  Security Policy |  Attack Description |  Result
---|---|---|---
CWE-20 |  ALG shall manage all communications between VLANs. Hosts cannot communicate with each other. |  Verify that all and only the configured connections are allowed using ICMP requests |  Enforced
CWE-290 |  ALG shall allow only authorized source IPs from a whitelist. |  Crafted packets with spoofed source IP as an existing host in the multimedia VLAN. |  Not Enforced
CWE-290 |  ALG shall allow only authorized source MACs from a whitelist. |  Crafted packets with spoofed source MAC as an existing host in the multimedia VLAN. |  Not Enforced
CWE-290 |  ALG shall ensure static MAC/IP bindings in the ARP table preventing any manipulation. |  Crafted ARP packets (poisoning attack) |  Enforced
CWE-281 |  ALG shall block ’MKD’ command to the FTP server 10.10.20.5 |  Attempt to execute ’MKD’ command from the FTP client 10.10.10.2 |  Enforced
CWE-434 |  ALG shall check malicious patterns in data transferred through the ’CP’ command to the FTP server 10.10.20.5 |  Crafted file sent by the FTP client 10.10.10.2 |  Not Enforced
CWE-281 |  ALG shall authorize only MPEG files from client 10.10.10.3 through port 8085 and redirect them to server 10.10.20.5 |  Crafted HTTP requests with different file types on different ports. |  Enforced
CWE-281 |  ALG shall authorize only DOC files from client 10.10.10.4 through port 8080 and redirect to server 10.10.20.3. |  Crafted HTTP requests with different file types on different ports. |  Enforced
CWE-281 |  The ALG shall authorize files included in an HTTP payload only if it is authored by a user in a whitelist. |  Crafted HTTP requests containing files with random author |  Enforced
CWE-444 |  ALG shall normalize and standardize incoming HTTP requests to prevent inconsistencies that attackers might exploit. |  Crafted HTTP requests |  Not Enforced
CWE-1333 |  ALG shall employ an efficient regex for the check of packets, to minimize resource consumption and mitigate ReDoS. |  Crafted HTTP requests |  Not Enforced
CWE-400 |  ALG shall ensure no performance degradation under normal conditions |  Stress testing |  No performance degradation within 100K req/min.
CWE-400 |  ALG shall introduce an acceptable maximum latency. |  Stress testing |  Latency of 39 ms

###  V-A Network and ALG Setup

We employ state-of-the-art GNS3 [12] software, which provides an effective
platform for network simulation [13], to build the testing environment. For
the sake of simplicity, experiments are conducted on a simplified version of
the topology shown in Section IV, which only includes the Control Network VLAN
(subnet 10.10.10.0) and the Multimedia Network VLAN (subnet 10.10.20.0).

Adhering to the on-board network project specifications, hosts within VLANs
communicate using various protocols. Open-source protocols like HTTP and FTP
are included in the simulation, while closed-source ones, such as TRDP, are
deliberately omitted. To simulate the required communications, an FTP server
and two HTTP webservers, responsible for FTP connections, MPEG data streams,
and DOC file transfers, are instantiated in the Control VLAN. Corresponding
clients reside in the multimedia VLAN. Additional hosts, namely Spoof, Fuzz,
and Jmeter, are instantiated for security testing operations. All network
hosts are instantiated as Docker containers, except for the JMeter host,
utilizing hardware virtualization for enhanced performance. Figure 2 provides
an overview of the entire simulated network and its interconnected hosts.

We simulate ALG functionalities and security policies by using a set of open-
source software components since the available ALGs are commercial products
(such as BIG-IP Advanced WAF [14] or Palo Alto Networks NGF [15]). We employ
Pfsense [16] firewall for the routing layer and HAProxy [17] load balancer for
the packet filtering at the application layer. In addition, since HAProxy does
not provide advanced packet inspection capabilities, we extended its
functionality using some Python custom plugins. The first plugin is designed
to handle HTTP requests containing textual files, such as docx documents. All
HTTP traffic containing documents is redirected to the plugin, which extracts
the document and applies security checks to its fields and content. The second
plugin extends ALG functionalities for the FTP protocol, applying security
checks to the FTP control and data packets. When properly configured, these
components simulate the behavior of a real ALG. We deploy the simulated ALG on
top of a Linux-based virtual machine connected to a third VLAN (subnet
10.10.11.0) configured with 1 vCPU and 1 GB RAM. After deploying the ALG
within the simulated network, we configured multiple security policies from
the categories described in Table II.

The first two columns of Table III provide details on all configured security
policies and their corresponding mitigated CWEs. For example, one of these
forces all traffic exchanged between the control VLAN and the multimedia VLAN
to pass through the ALG and deny direct connections between hosts.
Additionally, IP/MAC whitelisting to reject packets from unrecognized hosts,
and IP/MAC binding to prevent ARP cache poisoning are employed. Furthermore,
other policies allow the HTTP web servers to handle only specific types of
files (i.e., DOC and mpeg), and force routing based on the HTTP content-type
field. For example, only HTTP requests containing DOC files are allowed on
ALG’s 8080 port and redirected to the DOC server; requests containing mpeg
files are allowed on port 8085 and redirected to the MPEG server. It is also
required that some FTP commands be not allowed, such as ’mkd’ command.

Other configured security policies address the application layer. For example,
ALG checks the HTTP payload or header, allowing only requests containing files
created by whitelisted entities. Other ALG policies prevent inconsistencies in
the HTTP request format or enforce scanning for malicious patterns in FTP-
transferred data. In addition, efficient regex rules must be employed in these
checks to minimize resource consumption and mitigate ReDoS attacks.

Lastly, some policies are related to the availability and responsiveness
properties of the ALG. For example, ALG shall ensure no performance
degradation under normal conditions and shall guarantee an acceptable response
time.

###  V-B Experimental results

In the experimental analysis, we conducted the final phase of our methodology,
focusing on the security testing of the deployed simulated ALG. Each security
policy, configured to mitigate specific CWEs, underwent assessment for correct
enforcement through a specific attack scenario, as outlined in Section III-A.
All the performed attacks are listed in Table III along with the corresponding
outcomes, indicating whether the security policy was ’Enforced’ or ’Not
Enforced’.

We employed various security testing techniques for the implementation of
attack scenarios. We used ICMP Echo requests to test whether the ALG allows or
denies communication between hosts in separate VLANS. This test passed,
confirming that traffic can only pass through the ALG. Then, we use the Scapy
tool [18] to assess other security policies at the network layer and related
CWEs. This tool allows us to craft custom packets and design multiple attack
scenarios, which we run from the SPOOF client on the multimedia VLAN. For
instance, we checked if ALG only allows communication from recognized hosts,
determined by their IP or MAC addresses. While ALG does deny communication
from unrecognized hosts, we discovered that it is vulnerable to IP or MAC
spoofing when we send packets using spoofed IP or MAC addresses of recognized
hosts. To prevent such attacks, an ALG should incorporate authentication
mechanisms, such as the use of certificates.

Furthermore, we executed an ARP cache poisoning attack on the ALG aiming to
alter an existing IP-MAC binding in the ALG’s ARP table. This manipulation
leads the ALG to redirect traffic meant for a victim host in the VLAN to the
malicious host (SPOOF client). We crafted an ARP request for the ALG, using
the source IP of the victim host and the MAC address of the malicious host.
The test has passed because the ALG did not update its ARP table.

To assess application layer security policies, we perform fuzzing tests from
the FUZZ client in the multimedia VLAN. Generative fuzzers, such as the
BooFuzz tool [19] can be configured to automatically generate HTTP request
packets. These tools employ a generation model to define request templates and
vary packet parameters, to identify how the target application responds to
different inputs. By altering the content-type and payload in the generated
packets, we verified that the ALG correctly allows only specific preconfigured
file types (e.g., MPEG and DOC) and authorized file authors.

In addition, BooFuzz discovered two vulnerabilities (CVE-2023-25950 and
CVE-2023-25725) in Haproxy. We used a generation model to craft HTTP requests
with duplicated Content-Type or Content-Length fields. Normally, the ALG
should reject these requests, but it mistakenly processes them, leading to a
vulnerability known as HTTP Request Smuggling. ALG treats the last duplicated
field as valid, causing the generation of multiple responses to the client,
and leading to a vulnerability known as HTTP Response Splitting.

Mutation fuzzers, such as Mutiny [20], are also utilized during attack
scenarios. We intentionally introduced an inefficient regular expression for
malicious pattern recognition in the FTP parser plugin. Mutiny allows us to
use FTP sniffed traffic and mutate it to stress the configured regex. This
causes the ALG to consume excessive computational resources when processing
requests, making the ALG vulnerable to a ReDoS attack.

We conducted additional tests to assess the performance impact of the ALG when
processing packets exchanged between two VLANs. Using JMeter, we generated an
increasing number of requests from the JMeter client in the multimedia network
to the ALG. For the hardware configuration used, we did not observe
performance degradation up to 100,000 requests per minute. Beyond this value,
we observed linear degradation in terms of response time. Another test
involved measuring the latency introduced by the ALG. We again used JMeter to
generate HTTP requests and measure response times. By configuring the ALG to
forward a response following each request, we could approximate the latency
introduced by the ALG, which amounted to approximately 39 ms for the given
hardware configuration.

##  VI Conclusion and Future Work

The proposed methodology offers the advantage of being systematic and
comprehensive. It provides a clear understanding of specific vulnerabilities
affecting ALGs, which can be effectively addressed with tests defined through
a CWE-driven approach. This methodology serves as a driving framework for
testing and potentially certifying these products, streamlining assessments in
critical industrial infrastructures.

In addition, it covers both the network and the application layers within a
unified testing environment. Unlike solutions tailored to specific target
systems, our approach can be readily adapted to a wide range of systems,
making it generalizable and reusable. The provided practical case study
further demonstrates the applicability of our methodology in real-world
scenarios.

In future work, we plan to use more advanced fuzzing frameworks like AFL, or
machine learning-based testing tools to assess other security policies not
addressed in this work. We will also extend the experimental analysis, using
the proposed methodology for the assessment of other products.

## Acknowledgment

This work was partially supported by the FLEGREA project (CUP E53D23007950001)
funded by MUR and the GENIO project (CUP B69J23005770005) funded by MIMIT.

## References

  * [1] National Information Assurance Partnership, “Common Criteria for Information Technology Security Evaluation.”
  * [2] “Common Criteria - Genugate Firewall 10.0 Security Target [Online],” 2021, https://www.commoncriteriaportal.org/files/epfiles/1154b_pdf.pdf/.
  * [3] H. Anderson, “Introduction to nessus,” _Retrieved from Symantec_ , 2003.
  * [4] S. Bennetts, “Owasp zed attack proxy,” _AppSec USA_ , 2013.
  * [5] A. Mahajan, _Burp Suite Essentials_. Packt Publishing Ltd, 2014.
  * [6] W. L. de AM Macêdo, T. Da Rocha, and E. D. Moreno, “Gothings-an application-layer gateway architecture for the internet of things.” in _WEBIST_ , 2015.
  * [7] J. Alves-Foss, P. W. Oman, C. Taylor, and W. S. Harrison, “The mils architecture for high-assurance embedded systems,” _International journal of embedded systems_ , 2006.
  * [8] “MILS security guard [Online],” https://www.eetimes.com/mils-architecture-yields-high-assurance-systems-part-2/, 2023.
  * [9] M. Riaz, J.-M. Tilli, and R. Kantola, “Sec-alg: An open-source application layer gateway for secure access to private networks,” in _2020 29th International Conference on Computer Communications and Networks (ICCCN)_. IEEE, 2020.
  * [10] S. Roschke, F. Cheng, and C. Meinel, “Balg: Bypassing application layer gateways using multi-staged encrypted shellcodes,” in _12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops_. IEEE, 2011.
  * [11] “EN 50159. Railway applications - Communication, signalling and processing systems Safety-related communication in transmission systems,” 2023.
  * [12] “GNS3,” https://www.gns3.com/, 2023.
  * [13] T.-S. Chou, S. K. Baker, and M. Vega-Herrera, “A comparison of network simulation and emulation virtualization tools,” in _2016 ASEE Annual Conference & Exposition_, 2016.
  * [14] “BIG-IP Advanced WAF [Online],” https://www.f5.com/products/big-ip-services/advanced-waf, 2023.
  * [15] “Palo Alto Networks next-generation firewall [Online],” https://www.paloaltonetworks.com/, 2023.
  * [16] “pfsense Open Source Firewall [Online],” https://www.pfsense.org/, 2023.
  * [17] “HAProxy TCP/HTTP Load Balancer [Online],” https://www.haproxy.org/, 2023.
  * [18] R. Rohith, M. Moharir, G. Shobha _et al._ , “Scapy-a powerful interactive packet manipulation program,” in _2018 international conference on networking, embedded and wireless systems (ICNEWS)_. IEEE, 2018.
  * [19] “BooFuzz [Online],” https://boofuzz.readthedocs.io/, 2023.
  * [20] “Mutiny Fuzzing Framework [Online],” https://github.com/Cisco-Talos/mutiny-fuzzer/, 2023.


# HeadSculpt: Crafting 3D Head Avatars with Text

Xiao Han1,4 Yukang Cao2∗ Kai Han2 Xiatian Zhu1,5 Jiankang Deng3 Yi-Zhe Song1,4
Tao Xiang1,4† Kwan-Yee K. Wong2†
1University of Surrey 2The University of Hong Kong 3Imperial College London
4iFlyTek-Surrey Joint Research Centre on AI 5Surrey Institute for People-
Centred AI  Equal contributions † Corresponding authors ‡ Webpage:
https://brandonhan.uk/HeadSculpt

###### Abstract

Recently, text-guided 3D generative methods have made remarkable advancements
in producing high-quality textures and geometry, capitalizing on the
proliferation of large vision-language and image diffusion models. However,
existing methods still struggle to create high-fidelity 3D head avatars in two
aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model
whilst missing the necessary 3D awareness and head priors. This makes them
prone to inconsistency and geometric distortions in the generated avatars. (2)
They fall short in fine-grained editing. This is primarily due to the
inherited limitations from the pre-trained 2D image diffusion models, which
become more pronounced when it comes to 3D head avatars. In this work, we
address these challenges by introducing a versatile coarse-to-fine pipeline
dubbed HeadSculpt for crafting (i.e., generating and editing) 3D head avatars
from textual prompts. Specifically, we first equip the diffusion model with 3D
awareness by leveraging landmark-based control and a learned textual embedding
representing the back view appearance of heads, enabling 3D-consistent head
avatar generations. We further propose a novel identity-aware editing score
distillation strategy to optimize a textured mesh with a high-resolution
differentiable rendering technique. This enables identity preservation while
following the editing instruction. We showcase HeadSculpt’s superior fidelity
and editing capabilities through comprehensive experiments and comparisons
with existing methods. ‡

##  1 Introduction

|  |  |  |  |
---|---|---|---|---|---
∗Iron Man |  ∗Terracotta Army |  †Napoleon Bonaparte |  †Obama with a baseball cap |  make him like the Joker |  turn him into Pixar style
|  |  |  |  |
†Batman |  ∗Simpson in the Simpsons |  †Kratos in God of War |  †Taylor Swift |  put her a masquerade mask |  make her a sculpture
|  |  |  |  |
†Black Panther in Marvel |  ∗I am Groot |  †Vincent van Gogh |  †Audrey Hepburn |  make her color-restored |  make her a claymation
|  |  |  |  |
†Two-face in DC |  ∗Naruto Uzumaki |  †Leo Tolstoy |  †Geralt in The Witcher |  make him smiling |  turn him into Minecraft
|  |  |  |  |
†Doctor Strange |  ∗Hulk |  †a boy with facial painting |  †Caesar in Rise of the Planet of the Apes |  as a swimmer with a goggle |  make it carved out of wood
Figure 1:  Examples of generation and editing results obtained using the
proposed HeadSculpt. It enables the creation and fine-grained editing of high-
quality head avatars, featuring intricate geometry and texture, for any type
of head avatar using simple descriptions or instructions. Symbols indicate the
following prompt prefixes: $\ast$ “a head of [text]” and $\dagger$ “a DSLR
portrait of [text]”. The captions in gray are the prompt suffixes while the
blue ones are the editing instructions.

Modeling 3D head avatars underpins a wide range of emerging applications
(e.g., digital telepresence, game character creation, and AR/VR).
Historically, the creation of intricate and detailed 3D head avatars demanded
considerable time and expertise in art and engineering. With the advent of
deep learning, existing works [83, 25, 30, 68, 7, 35, 14] have shown promising
results on the reconstruction of 3D human heads from monocular images or
videos. However, these methods remain restricted to head appearance contained
in their training data which is often limited in size, resulting in the
inability to generalize to new appearance beyond the training data. This
constraint calls for the need of more flexible and generalizable methods for
3D head modeling.

Recently, vision-language models (e.g., CLIP [51]) and diffusion models (e.g.,
Stable Diffusion [65, 57, 55]) have attracted increasing interest. These
progresses have led to the emergence of text-to-3D generative models [31, 58,
41, 24] which create 3D content in a self-supervised manner. Notably,
DreamFusion [50] introduces a score distillation sampling (SDS) strategy that
leverages a pre-trained image diffusion model to compute the noise-level loss
from the textual description, unlocking the potential to optimize
differentiable 3D scenes (e.g., neural radiance field [42], tetrahedron mesh
[62], texture [54, 8], or point clouds [47]) with 2D diffusion prior only.
Subsequent research efforts [40, 6, 61, 75, 39, 71, 37, 52, 72] improve and
extend DreamFusion from various perspectives (e.g., higher resolution [36] and
better geometry [9]).

Considering the flexibility and versatility of natural languages, one might
think that these SDS-based text-to-3D generative methods would be sufficient
for generating diverse 3D avatars. However, it is noted that existing methods
have two major drawbacks (see Fig. 4): (1) Inconsistency and geometric
distortions: The 2D diffusion models used in these methods lack 3D awareness
particularly regarding camera pose; without any remedy, existing text-to-3D
methods inherited this limitation, leading to the multi-face “Janus” problem
in the generated head avatars. (2) Fine-grained editing limitations: Although
previous methods propose to edit 3D models by naively fine-tuning trained
models with modified prompts [50, 36], we find that this approach is prone to
biased outcomes, such as identity loss or inadequate editing. This problem
arises from two causes: (a) inherent bias in prompt-based editing in image
diffusion models, and (b) challenges with inconsistent gradient back-
propagation at separate iterations when using SDS calculated from a vanilla
image diffusion model.

In this paper, we introduce a new head-avatar-focused text-to-3D method,
dubbed HeadSculpt, that supports high-fidelity generation and fine-grained
editing. Our method comprises two novel components: (1) Prior-driven score
distillation: We first arm a pre-trained image diffusion model with 3D
awareness by integrating a landmark-based ControlNet [80]. Specifically, we
adopt the parametric 3D head model, FLAME [35], as a prior to obtain a 2D
landmark map [38, 28], which will serve as an additional condition for the
diffusion model, ensuring the consistency of generated head avatars across
different views. Further, to remedy the front-view bias in the pre-trained
diffusion model, we utilize an improved view-dependent prompt through textual
inversion [16], by learning a specialized <back-view> token to emphasize back
views of heads and capture their unique visual details. (2) Identity-aware
editing score distillation (IESD): To address the challenges of fine-grained
editing for head avatars, we introduce a novel method called IESD. It blends
two scores, one for editing and the other for identity preservation, both
predicted by a ControlNet-based implementation of InstructPix2Pix [5]. This
approach maintains a controlled editing direction that respects both the
original identity and the editing instructions. To further improve the
fidelity of our method, we integrate these two novel components into a coarse-
to-fine pipeline [36], utilizing NeRF [45] as the low-resolution coarse model
and DMTet [62] as the high-resolution fine model. As demonstrated in Fig. 1,
our method can generate high-fidelity human-like and non-human-like head
avatars while enabling fine-grained editing, including local changes,
shape/texture modifications, and style transfers.

##  2 Related work

Text-to-2D generation. In recent years, groundbreaking vision-language
technologies such as CLIP [51] and diffusion models [22, 12, 55, 64] have led
to significant advancements in text-to-2D content generation [57, 53, 1, 65,
66]. Trained on extensive 2D multimodal datasets [59, 60], they are empowered
with the capability to “dream” from the prompt. Follow-up works endeavor to
efficiently control the generated results [80, 81, 44], extend the diffusion
model to video sequence [63, 3], accomplish image or video editing [21, 29,
77, 5, 73, 13, 20], enhance the performance for personalized subjects [56,
16], etc. Although significant progress has been made in generating 2D content
from text, carefully crafting the prompt is crucial, and obtaining the desired
outcome often requires multiple attempts. The inherent randomness remains a
challenge, especially for editing tasks.

Text-to-3D generation. Advancements in text-to-2D generation have paved the
way for text-to-3D techniques. Early efforts [78, 24, 41, 58, 31, 26, 10]
propose to optimize the 3D neural radiance field (NeRF) or vertex-based meshes
by employing the CLIP language model. However, these models encounter
difficulties in generating expressive 3D content, primarily because of the
limitations of CLIP in comprehending natural language. Fortunately, the
development of image diffusion models [65, 1] has led to the emergence of
DreamFusion [50]. It proposes Score Distillation Sampling (SDS) based on a
pre-trained 2D diffusion prior [57], showcasing promising generation results.
Subsequent works [34] have endeavored to improve DreamFusion from various
aspects: Magic3D [36] proposes a coarse-to-fine pipeline for high-resolution
generations; Latent-NeRF [40] includes shape guidance for more robust
generation on the latent space [55]; DreamAvatar [6] leverages SMPL [4] to
generate 3D human full-body avatars under controllable shapes and poses;
Fantasia3D [9] disentangles the geometry and texture training with DMTet [62]
and PBR texture [46] as their 3D representation; 3DFuse [61] integrates depth
control and semantic code sampling to stabilize the generation process.
Despite notable progress, current text-to-3D generative models still face
challenges in producing view-consistent 3D content, especially for intricate
head avatars. This is primarily due to the absence of 3D awareness in text-
to-2D diffusion models. Additionally, to the best of our knowledge, there is
currently no approach that specifically focuses on editing the generated 3D
content, especially addressing the intricate fine-grained editing needs of
head avatars.

3D head modeling and creation. Statistical mesh-based models, such as FLAME
[35, 14], enable the reconstruction of 3D head models from images. However,
they struggle to capture fine details like hair and wrinkles. To overcome this
issue, recent approaches [7, 67, 68, 48] employ Generative Adversarial
Networks (GANs) [43, 18, 27] to train 3D-aware networks on 2D head datasets
and produce 3D-consistent images through latent code manipulation.
Furthermore, neural implicit methods [83, 15, 25, 84] introduce implicit and
subject-oriented head models based on neural rendering fields [42, 45, 2].
Recently, text-to-3D generative methods have gained traction, generating high-
quality 3D head avatars from natural language using vision-language models
[51, 65]. Typically, T2P [81] predicts bone-driven parameters of head avatars
via a game engine under the CLIP guidance [51]. Rodin [76] proposes a roll-out
diffusion network to perform 3D-aware diffusion. DreamFace [79] employs a
selection strategy in the CLIP embedding space to generate coarse geometry and
uses SDS [50] to optimize UV-texture. Despite producing promising results, all
these methods require a large amount of data for supervised training and
struggle to generalize well to non-human-like avatars. In contrast, our
approach relies solely on pre-trained text-to-2D models, generalizes well to
out-of-domain avatars, and is capable of performing fine-grained editing
tasks.

##  3 Methodology

Figure 2: Overall architecture of HeadSculpt. We craft high-resolution 3D head
avatars in a coarse-to-fine manner. (a) We optimize neural field
representations for the coarse model. (b) We refine or edit the model using
the extracted 3D mesh and apply identity-aware editing score distillation if
editing is the target. (c) The core of our pipeline is the prior-driven score
distillation, which incorporates landmark control, enhanced view-dependent
prompts, and an InstructPix2Pix branch.

HeadSculpt is a 3D-aware text-to-3D approach that utilizes a pre-trained text-
to-2D Stable Diffusion model [65, 55] to generate high-resolution head avatars
and perform fine-grained editing tasks. As illustrated in Fig. 2, the
generation pipeline has two stages: coarse generation via the neural radiance
field (NeRF) [45] and refinement/editing using tetrahedron mesh (DMTet) [62].
Next, we will first introduce the preliminaries that form the basis of our
method in Sec. 3.1. We will then discuss the key components of our approach in
Sec. 3.2 and Sec, 3.3, including (1) the prior-driven score distillation
process via landmark-based ControlNet [80] and textual inversion [16], and (2)
identity-aware editing score distillation accomplished in the fine stage using
the ControlNet-based InstructPix2Pix [5].

###  3.1 Preliminaries

Score distillation sampling. Recently, DreamFusion [50] proposed score
distillation sampling (SDS) to self-optimize a text-consistent neural radiance
field (NeRF) based a the pre-trained text-to-2D diffusion model [57]. Due to
the unavailability of the Imagen model [57] used by DreamFusion, we employ the
latent diffusion model in [55] instead. Specifically, given a latent feature
$z_{t}$ encoded from an image $x$, the SDS strategy adds random noise
$\epsilon$ to $z_{t}$ and utilizes the pre-trained denoising function
$\epsilon_{\phi}\left(z_{t};y,t\right)$ to predict the added noise. The SDS
loss is defined as the difference between predicted and added noise and its
gradient is given by

$\nabla_{\theta}\mathcal{L}_{\mathrm{SDS}}(\phi,g(\theta))=\mathbb{E}_{t,\epsilon\sim\mathcal{N}(0,1)}\left[w(t)\left(\epsilon_{\phi}\left(z_{t};y,t\right)-\epsilon\right)\frac{\partial
z}{\partial x}\frac{\partial x}{\partial\theta}\right],$ (1)

where $y$ is the text embedding, $w(t)$ weights the loss from noise level $t$.
With the expressive text-to-2D diffusion model and self-supervised SDS loss,
we can back-propagate the gradients to optimize an implicit 3D scene
$g(\theta)$, eliminating the need for an expensive 3D dataset.

3D scene optimization. HeadSculpt explores the potential of two different 3D
differentiable representations as the optimization basis for crafting 3D head
avatars. Specifically, we employ NeRF [45] in the coarse stage due to its
greater flexibility in geometry deformation, while utilizing DMTet [62] in the
fine stage for efficient high-resolution optimization.

(1) 3D prior-based NeRF. DreamAvatar [6] recently proposed a density-residual
setup to enhance the robustness of the generated 3D NeRF. Given a point
$\mathbf{x}$ inside the 3D volume, we can derive its density and color value
based on a prior-based density field $\bar{\sigma}$:

$F(\mathbf{x},\bar{\sigma})=F_{\theta}(\gamma(\mathbf{x}))+(\bar{\sigma}(\mathbf{x}),\mathbf{0})\mapsto(\sigma,\mathbf{c}),$
(2)

where $\gamma(\cdot)$ denotes a hash-grid frequency encoder [45], and $\sigma$
and $\mathbf{c}$ are the density and RGB color respectively. We can derive
$\bar{\sigma}$ from the signed distance $d(\mathbf{x})$ of a given 3D shape
prior (e.g., a canonical FLAME model [35] in our implementation):

$\bar{\sigma}(\mathbf{x})=\max\left(0,\operatorname{softplus}^{-1}(\tau(\mathbf{x}))\right),\,\tau(\mathbf{x})=\frac{1}{a}\operatorname{sigmoid}(-d(\mathbf{x})/a),\textrm{where
}a=0.005.$ (3)

To obtain a 2D RGB image from the implicit volume defined above, we employ a
volume rendering technique that involves casting a ray $\mathbf{r}$ from the
2D pixel location into the 3D scene, sampling points $\boldsymbol{\mu}_{i}$
along the ray, and calculating their density and color value using $F$ in Eq.
(2):

$C(\mathbf{r})=\sum_{i}W_{i}\mathbf{c}_{i},\quad
W_{i}=\alpha_{i}\prod_{j<i}(1-\alpha_{j}),\quad\alpha_{i}=1-e^{(-\sigma_{i}||\boldsymbol{\mu}_{i}-\boldsymbol{\mu}_{i+1}||)}.$
(4)

(2) DMTet. It discretizes a deformable tetrahedral grid ($V_{T},T$), where
$V_{T}$ denotes the vertices within grid $T$ [17, 62], to model the 3D space.
Every vertex $\mathbf{v}_{i}\in V_{T}\subset\mathbb{R}^{3}$ possesses a signed
distance value $s_{i}\in\mathbb{R}$, along with a position offset
$\Delta\mathbf{v}_{i}\in\mathbb{R}^{3}$ of the vertex relative to its initial
canonical coordinates. Subsequently, the underlying mesh can be extracted
based on $s_{i}$ with the differentiable marching tetrahedra algorithm. In
addition to the geometry, we adopt the Magic3D approach [36] to construct a
neural color field. This involves re-utilizing the MLP trained in the coarse
NeRF stage to predict the RGB color value for each 3D point. During
optimization, we render this textured surface mesh into high-resolution images
using a differentiable rasterizer [33, 46].

###  3.2 3D-Prior-driven score distillation

Existing text-to-3D methods with SDS [50] assume that maximizing the
likelihood of images rendered from various viewpoints of a scene model
$g(\cdot)$ is equivalent to maximizing the overall likelihood of $g(\cdot)$.
This assumption can result in inconsistencies and geometric distortions [50,
61]. A notable issue is the “Janus problem” characterized by multiple faces on
a single object (see Fig. 4). There are two possible causes: (1) the
randomness of the diffusion model which can cause inconsistencies among
different views, and (2) the lack of 3D awareness in controlling the
generation process, causing the model to struggle in determining the front
view, back view, etc. To address these issues in generating head avatars, we
integrate 3D head priors into the diffusion model.

Landmark-based ControlNet. In Section 3.1, we explain our adoption of FLAME
[35] as the density guidance for our NeRF. Nevertheless, this guidance by
itself is insufficient to have a direct impact on the SDS loss. What is
missing is a link between the NeRF and the diffusion model, incorporating the
same head priors. Such a link is key to improving the view consistency of the
generated head avatars. To achieve this objective, as illustrated in Fig. 2,
we propose the incorporation of 2D landmark maps as an additional condition
for the diffusion model using ControlNet [80]. Specifically, we employ a
ControlNet $\mathcal{C}$ trained on a large-scale 2D face dataset [82, 11],
using facial landmarks rendered from MediaPipe [38, 28] as ground-truth data.
When given a randomly sampled camera pose $\pi$, we first project the vertices
of the FLAME model onto the image. Following that, we select and render some
of these vertices into a landmark map $\mathcal{P}_{\pi}$ based on some
predefined vertex indexes. The landmark map will be fed into ControlNet and
its output features are added to the intermediate features within the
diffusion U-Net. The gradient of our SDS loss can be re-written as

$\nabla_{\theta}\mathcal{L}_{\mathrm{SDS}}(\phi,g(\theta))=\mathbb{E}_{t,\epsilon\sim\mathcal{N}(0,1),\pi}\left[w(t)\left(\epsilon_{\phi}\left(z_{t};y,t,\mathcal{C}(\mathcal{P}_{\pi})\right)-\epsilon\right)\frac{\partial
z}{\partial x}\frac{\partial x}{\partial\theta}\right].$ (5)

Enhanced view-dependent prompt via textual inversion. Although the landmark-
based ControlNet can inject 3D awareness into the pre-trained diffusion
models, it struggles to maintain back view head consistency. This is expected
as the 2D image dataset used for training mostly contains only front or side
face views. Consequently, when applied directly to back views, the model
introduces ambiguity as front and back 3D landmark views can appear similar,
as shown in Fig. 6. To address this issue, we propose a simple yet effective
method. Our method is inspired by previous works [50, 61, 36] which found it
beneficial to append view-dependent text (e.g., “front view”, “side view” or
“back view”) to the provided input text based on the azimuth angle of the
randomly sampled camera. We extend this idea by learning a special token
<back-view> to replace the plain text “back view” in order to emphasize the
rear appearance of heads. This is based on the assumption that a pre-trained
Stable Diffusion does has the ability to “imagine” the back view of a head -
it has seen some during training. The main problem is that a generic text
embedding of “back view” is inadequate in telling the model what appearance it
entails. A better embedding for “back view” is thus required. To this end, we
first randomly download 34 images of the back view of human heads, without
revealing any personal identities, to construct a tiny dataset $\mathcal{D}$,
and then we optimize the special token $v$ (i.e., <back-view>) to better fit
the collected images, similar to the textual inversion [16]:

$v_{*}=\underset{v}{\arg\min}\,\mathbb{E}_{t,\epsilon\sim\mathcal{N}(0,1),z\sim\mathcal{D}}\left[\left\|\epsilon-\epsilon_{\phi}\left(z_{t};v,t\right)\right\|_{2}^{2}\right],$
(6)

which is achieved by employing the same training scheme as the original
diffusion model, while keeping $\epsilon_{\phi}$ fixed. This constitutes a
reconstruction task, which we anticipate will encourage the learned embedding
to capture the fine visual details of the back views of human heads. Notably,
as we do not update the weights of $\epsilon_{\phi}$, it stays compatible with
the landmark-based ControlNet.

Reference Avatar | $\omega_{e}=0.2$ | $\omega_{e}=0.4$ | $\omega_{e}=0.6$ | $\omega_{e}=0.8$ | $\omega_{e}=1.0$
---|---|---|---|---|---
|  |  |  |  |
Saul Goodman | turn him into a clown
Figure 3: Impact of the edit scale $\omega_{e}$ in IESD. It balances the
preservation of the initial appearance and the extent of the desired editing,
making the editing process more controllable and flexible.

###  3.3 Identity-aware editing score distillation

After generating avatars, editing them to fulfill particular requirements
poses an additional challenge. Previous works [50, 36] have shown promising
editing results by fine-tuning a trained scene model with a new target prompt.
However, when applied to head avatars, these methods often suffer from
identity loss or inadequate appearance modifications (see Fig. 7). This
problem stems from the inherent constraint of the SDS loss, where the 3D
models often sacrifice prominent features to preserve view consistency.
Substituting Stable Diffusion with InstructPix2Pix [5, 19] might seem like a
simple solution, but it also faces difficulties in maintaining facial identity
during editing based only on instructions, as it lacks a well-defined anchor
point.

To this end, we propose identity-aware editing score distillation (IESD) to
regulate the editing direction by blending two predicted scores, i.e., one for
editing instruction and another for the original description. Rather than
using the original InstructPix2Pix [5], we employ a ControlNet-based
InstructPix2Pix $\mathcal{I}$ [80] trained on the same dataset, ensuring
compatibility with our landmark-based ControlNet $\mathcal{C}$ and the learned
<back-view> token. Formally, given an initial textual prompt $y$ describing
the avatar to be edited and an editing instruction $\hat{y}$, we first input
them separately into the same diffusion model equipped with two ControlNets,
$\mathcal{I}$ and $\mathcal{C}$. This allows us to obtain two predicted
noises, which are then combined using a predefined hyper-parameter
$\omega_{e}$ like classifier-free diffusion guidance (CFG) [23]:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\mathrm{IESD}}(\phi,g(\theta))=\mathbb{E}_{t,\epsilon\sim\mathcal{N}(0,1),\pi}\left[w(t)\left(\underbrace{\hat{\epsilon}_{\phi}\left(z_{t};{\color[rgb]{0.01,1,0.48}y},{\color[rgb]{1,0,0.5}\hat{y}},t,\mathcal{C}(\mathcal{P}_{\pi}),\mathcal{I}(\mathcal{M}_{\pi})\right)}-\epsilon\right)\frac{\partial
z}{\partial x}\frac{\partial x}{\partial\theta}\right],$ (7)
$\displaystyle{\color[rgb]{1,0,0.5}\omega_{e}}\epsilon_{\phi}\left(z_{t};{\color[rgb]{1,0,0.5}\hat{y}},t,\mathcal{C}(\mathcal{P}_{\pi}),\mathcal{I}(\mathcal{M}_{\pi})\right)+{\color[rgb]{0.01,1,0.48}(1-\omega_{e})}\epsilon_{\phi}\left(z_{t};{\color[rgb]{0.01,1,0.48}y},t,\mathcal{C}(\mathcal{P}_{\pi}),\mathcal{I}(\mathcal{M}_{\pi})\right)$

where $\mathcal{P}_{\pi}$ and $\mathcal{M}_{\pi}$ represent the 2D landmark
maps and the reference images rendered in the coarse stage, both being
obtained under the sampled camera pose $\pi$. The parameter $\omega_{e}$
governs a trade-off between the original appearance and the desired editing,
which defaults to $0.6$ in our experiments.

##  4 Experiments

DreamFusion* [69] | Latent-NeRF [40] | 3DFuse [61] | Fantasia3D* [70] | DreamFace$\dagger$ [79] | HeadSculpt (Ours)
---|---|---|---|---|---
|  |  |  |  |
a DSLR portrait of Salvador Dalí
|  |  |  |  |
a head of Stormtrooper
|  |  |  |  |
a DSLR portrait of a female soldier, wearing a helmet
|  |  |  |  |
a DSLR portrait of a young black lady with short hair, wearing a headphone
Figure 4: Comparison with existing text-to-3D methods. Unlike other methods
that struggle or fail to generate reasonable results, our approach
consistently achieves high-quality geometry and texture, yielding superior
results. *Non-official implementation. $\dagger$ Generated from the online
website demo.

We will now assess the efficacy of our HeadSculpt across different scenarios,
while also conducting a comparative analysis against state-of-the-art text-
to-3D generation pipelines.

Implementation details. HeadSculpt builds upon Stable-DreamFusion [69] and
Huggingface Diffusers [74, 49]. We utilize version 1.5 of Stable Diffusion
[65] and version 1.1 of ControlNet [80, 11] in our implementation. In the
coarse stage, we optimize our 3D model at $64\times 64$ grid resolution, while
using $512\times 512$ grid resolution for the fine stage (refinement or
editing). Typically, each text prompt requires approximately $7,000$
iterations for the coarse stage and $5,000$ iterations for the fine stage. It
takes around 1 hour for each stage on a single Tesla V100 GPU with a default
batch size of 4. We use Adam [32] optimizer with a fixed learning rate of
$0.001$. Additional implementation details can be found in the supplementary
material.

Baseline methods. We compare with five baselines: DreamFusion [69], Latent-
NeRF [40], 3DFuse [61] (improved version of SJC [75]), Fantasia3D [9], and
DreamFace [79]. Since official implementations for DreamFusion and Fantasia3D
are not yet available, we employ their respective open-source counterparts,
i.e., Stable-DreamFusion [69] and Fantasia3D.unofficial [70]. We do not
directly compare with DreamAvatar [6] as it involves deformation fields for
full-body-related tasks.

###  4.1 Qualitative evaluations

Head avatar generation and editing with various prompts. In Fig. 1, we show a
diverse array of 3D head avatars generated by HeadSculpt, consistently
demonstrating high-quality geometry and texture across various viewpoints. Our
method’s versatility is emphasized by its ability to create an assortment of
avatars, including humans (both celebrities and ordinary individuals) as well
as non-human characters like superheroes, comic/game characters, paintings,
and more. Additionally, HeadSculpt’s adaptability is showcased through its
ability to perform fine-grained editing, such as local changes (e.g., adding
accessories or altering expressions), shape and texture modifications, and
style transfers.

Head avatar editing with different edit scales. In Fig. 3, we demonstrate the
effectiveness of IESD with different $\omega_{e}$ values, highlighting its
ability to control editing influence on the reference identity.

Comparison with SOTA methods. We provide qualitative comparisons with existing
SOTA methods in Fig. 4. We employ the same FLAME model for Latent-NeRF [40] to
compute their sketch-guided loss and for Fantasia3D [70] as the initial
geometry.

The following observations can be made: (1) All baselines tend to be more
unstable during training than ours, often resulting in diverged training
processes; (2) Latent-NeRF occasionally produces plausible results due to its
use of the shape prior, but its textures are inferior to ours since
optimization occurs solely in the latent space; (3) Despite 3DFuse’s depth
control to mitigate the Janus problem, it still struggles to generate 3D
consistent head avatars; (4) While Fantasia3D can generate a mesh-based 3D
avatar, its geometry is heavily distorted, as its disentangled geometry
optimization might be insufficient for highly detailed head avatars; (5)
Although DreamFace generates realistic human face textures, it falls short in
generating (i) complete heads, (ii) intricate geometry, (iii) non-human-like
appearance, and (iv) composite accessories. In comparison, our method
consistently yields superior results in both geometry and texture with much
better consistency for the given prompt.

###  4.2 User study

To further assess the quality of the generated results, we conduct a user
study with the participation of 20 volunteers. The four baselines [69, 70, 61,
40] and our HeadSculpt are compared based on three dimensions: (1) consistency
with the text, (2) texture quality, and (3) geometry quality. Volunteers are
presented with 20 randomly selected generated results in the form of rendered
rotating videos and asked to assign a score from 1 (worst) to 5 (best) for
each example and criterion. The results, shown in Fig. 6, indicate that our
method achieved the highest rank in all three aspects by large margins.

Figure 5: User study.

HeadSculpt (Ours) | w/o Landmark Ctrl | HeadSculpt (Ours) | w/o Textual Inversion
---|---|---|---
|  |  |
a head of Woody in the Toy Story | a head of Walter White, wearing a bowler hat
|  |  |
a head of Bumblebee in Transformers | a head of Mario in Mario Franchise
Figure 6: Analysis of prior-driven score distillation.

###  4.3 Further analysis

Effectiveness of prior-driven score distillation. In Fig. 6, we conduct
ablation studies to examine the impact of the proposed landmark control and
textual inversion priors in our method. We demonstrate this on the coarse
stage because the refinement and editing results heavily depend on this stage.
The findings show that landmark control is essential for generating spatially
consistent head avatars. Without it, the optimized 3D avatar faces challenges
in maintaining consistent facial views, particularly for non-human-like
characters. Moreover, textual inversion is shown to be another vital component
in mitigating the Janus problem, specifically for the back view, as landmarks
cannot exert control on the rear view. Overall, the combination of both
components enables HeadSculpt to produce view-consistent avatars with high-
quality geometry.

B1: One-stage | B2: Coarse + Coarse | B3: Coarse + Fine | B4: Naive IP2P | B5: Ours w/o $\omega_{e}$ | HeadSculpt (Ours)
---|---|---|---|---|---
|  |  |  |  |
Modified description: a DSLR portrait of +[older] Saul Goodman | Instruction: make him older
|  |  |  |  |
Modified description: a DSLR portrait skull of Vincent van Gogh | Instruction: turn his face into a skull
Figure 7: Analysis of identity-aware editing score distillation.

Effectiveness of IESD. We assess IESD’s efficacy for fine-grained 3D head
avatar editing by comparing it with various alternatives since no dedicated
method exists for this: (B1) One-step optimization on the coarse stage without
initialization; (B2) Initialized from the coarse stage, followed by
optimization of another coarse stage with an altered description; (B3)
Initialized from the coarse stage, followed by optimization of a new fine
stage with an altered description; (B4) Initialized from the coarse stage,
followed by optimization of a new fine stage with an instruction based on the
vanilla InstructPix2Pix [5]; (B5) Ours without edit scale (i.e.,
$\omega_{e}=1$). Notably, B2 represents the editing method proposed in
DreamFusion [50], while B3 has a similar performance as Magic3D [36], which
employs a three-stage editing process (i.e., Coarse + Coarse + Fine).

In Fig. 7, we present two common biased editing scenarios produced by the
baseline methods: insufficient editing and loss of identity. With Stable
Diffusion, specific terms like “Saul Goodman” and “skull” exert a more
substantial influence on the text embeddings compared to other terms, such as
“older” and “Vincent van Gogh”. B1, B2, and B3, all based on vanilla Stable
Diffusion, inherit such bias in their generated 3D avatars. Although B4 does
not show such bias, it faces two other issues: (1) the Janus problem reemerges
due to incompatibility between vanilla InstructPix2Pix and the proposed prior-
driven score distillation; (2) it struggles to maintain facial identity during
editing based solely on instructions, lacking a well-defined anchor point. In
contrast, B5 employs ControlNet-based InstructPix2Pix [80] with the proposed
prior score distillation, resulting in more view-consistent editing.
Additionally, our IESD further uses the proposed edit scale to merge two
predicted scores, leading to better identity preservation and more effective
editing. This approach allows our method to overcome the limitations faced by
the alternative solutions, producing high-quality 3D avatars with improved
fine-grained editing results.

##  5 Conclusions

We have introduced HeadSculpt, a novel pipeline for generating high-resolution
3D human avatars and performing identity-aware editing tasks through text. We
proposed to utilize a prior-driven score distillation that combines a
landmark-based ControlNet and view-dependent textual inversion to address the
Janus problem. We also introduced identity-aware editing score distillation
that preserves both the original identity information and the editing
instruction. Extensive evaluations demonstrated that our HeadSculpt produces
high-fidelity results under various scenarios, outperforming state-of-the-art
methods significantly.

Limitations. Although HeadSculpt produces new SOTA results, we notice certain
limitations: (1) non-deformable results hinder further extensions and
applications in audio or video-driven problems; (2) generated textures of
human faces are less realistic than conventional supervised face
reconstruction methods; (3) some inherited biases from Stable Diffusion [65]
still remain, e.g., the generated Asian heads are highly stereotyped; and (4)
limitations inherited from InstructPix2Pix [5], e.g., the inability to perform
large spatial manipulations.

Societal impact. The advancements in geometry and texture generation for human
head avatars could be deployed in many AR/VR use cases but also raises
concerns about their potential malicious use. We encourage responsible
research and application, fostering open and transparent practices.

## References

  * [1] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324, 2022.
  * [2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In CVPR, 2022.
  * [3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In CVPR, 2023.
  * [4] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black.  Keep it smpl: Automatic estimation of 3d human pose and shape from a single image.  In ECCV, 2016.
  * [5] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In CVPR, 2023.
  * [6] Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan-Yee K Wong.  Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models.  arXiv preprint arXiv:2304.00916, 2023.
  * [7] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein.  Efficient geometry-aware 3D generative adversarial networks.  In CVPR, 2022.
  * [8] Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner.  Text2tex: Text-driven texture synthesis via diffusion models.  arXiv preprint arXiv:2303.11396, 2023.
  * [9] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  arXiv preprint arXiv:2303.13873, 2023.
  * [10] Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, and Kui Jia.  Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition.  In NeurIPS, 2023.
  * [11] CrucibleAI.  ControlNetMediaPipeFace.  https://huggingface.co/CrucibleAI/ControlNetMediaPipeFace, 2023\.
  * [12] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  In NeurIPS, 2020.
  * [13] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  arXiv preprint arXiv:2302.03011, 2023.
  * [14] Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart.  Learning an animatable detailed 3d face model from in-the-wild images.  ACM Transactions on Graphics (ToG), 2021.
  * [15] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In CVPR, 2021.
  * [16] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  In ICLR, 2023.
  * [17] Jun Gao, Wenzheng Chen, Tommy Xiang, Alec Jacobson, Morgan McGuire, and Sanja Fidler.  Learning deformable tetrahedral meshes for 3d reconstruction.  In NeurIPS, 2020.
  * [18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 2020.
  * [19] Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa.  Instruct-nerf2nerf: Editing 3d scenes with instructions.  arXiv preprint arXiv:2303.12789, 2023.
  * [20] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or.  Delta denoising score.  arXiv preprint arXiv:2304.07090, 2023.
  * [21] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [22] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In NeurIPS, 2020.
  * [23] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In NeurIPS Workshop, 2021.
  * [24] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu.  Avatarclip: zero-shot text-driven generation and animation of 3d avatars.  ACM Transactions on Graphics (TOG), 2022.
  * [25] Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang.  Headnerf: A real-time nerf-based parametric head model.  In CVPR, 2022.
  * [26] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  In CVPR, 2022.
  * [27] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [28] Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, and Matthias Grundmann.  Real-time facial surface geometry from monocular video on mobile gpus.  In CVPR workshops, 2019.
  * [29] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  arXiv preprint arXiv:2210.09276, 2022.
  * [30] Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov.  Realistic one-shot mesh-based head avatars.  In ECCV, 2022.
  * [31] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Popa Tiberiu.  Clip-mesh: Generating textured meshes from text using pretrained image-text models.  In SIGGRAPH Asia, 2022.
  * [32] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [33] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila.  Modular primitives for high-performance differentiable rendering.  ACM Transactions on Graphics (ToG), 2020.
  * [34] Chenghao Li, Chaoning Zhang, Atish Waghwase, Lik-Hang Lee, Francois Rameau, Yang Yang, Sung-Ho Bae, and Choong Seon Hong.  Generative ai meets 3d: A survey on text-to-3d in aigc era.  arXiv preprint arXiv:2305.06131, 2023.
  * [35] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Transactions on Graphics (TOG), 2017.
  * [36] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In CVPR, 2023.
  * [37] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  arXiv preprint arXiv:2303.11328, 2023.
  * [38] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong, Juhyun Lee, et al.  Mediapipe: A framework for perceiving and processing reality.  In CVPR workshops, 2019.
  * [39] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Realfusion: 360 $\\{$$\backslash$deg$\\}$ reconstruction of any object from a single image.  arXiv preprint arXiv:2302.10663, 2023.
  * [40] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  arXiv preprint arXiv:2211.07600, 2022.
  * [41] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka.  Text2mesh: Text-driven neural stylization for meshes.  In CVPR, 2022.
  * [42] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [43] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [44] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  arXiv preprint arXiv:2302.08453, 2023.
  * [45] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Transactions on Graphics (TOG), 2022.
  * [46] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler.  Extracting triangular 3d models, materials, and lighting from images.  In CVPR, 2022.
  * [47] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen.  Point-e: A system for generating 3d point clouds from complex prompts.  arXiv preprint arXiv:2212.08751, 2022.
  * [48] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher-Shlizerman.  Stylesdf: High-resolution 3d-consistent image and geometry generation.  In CVPR, 2022.
  * [49] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In NeurIPS, 2019.
  * [50] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In ICLR, 2022.
  * [51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [52] Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al.  Dreambooth3d: Subject-driven text-to-3d generation.  arXiv preprint arXiv:2303.13508, 2023.
  * [53] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [54] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or.  Texture: Text-guided texturing of 3d shapes.  arXiv preprint arXiv:2302.01721, 2023.
  * [55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [56] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In CVPR, 2023.
  * [57] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [58] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan.  Clip-forge: Towards zero-shot text-to-shape generation.  In CVPR, 2022.
  * [59] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  In NeurIPS, 2022.
  * [60] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [61] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim.  Let 2d diffusion model know 3d-consistency for robust text-to-3d generation.  arXiv preprint arXiv:2303.07937, 2023.
  * [62] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  In NeurIPS, 2021.
  * [63] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  In ICLR, 2023.
  * [64] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In ICLR, 2021.
  * [65] Stability.AI.  Stable diffusion.  https://stability.ai/blog/stable-diffusion-public-release, 2022\.
  * [66] Stability.AI.  Stability AI releases DeepFloyd IF, a powerful text-to-image model that can smartly integrate text into images.  https://stability.ai/blog/deepfloyd-if-text-to-image-model, 2023\.
  * [67] Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang, Jue Wang, and Yebin Liu.  Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis.  ACM Transactions on Graphics (ToG), 2022.
  * [68] Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, and Yebin Liu.  Next3d: Generative neural texture rasterization for 3d-aware head avatars.  In CVPR, 2023.
  * [69] Jiaxiang Tang.  Stable-dreamfusion: Text-to-3d with stable-diffusion.  https://github.com/ashawkey/stable-dreamfusion, 2022.
  * [70] Jiaxiang Tang.  Fantasia3d.unofficial.  https://github.com/ashawkey/fantasia3d.unofficial, 2023.
  * [71] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen.  Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior.  arXiv preprint arXiv:2303.14184, 2023.
  * [72] Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari.  Textmesh: Generation of realistic 3d meshes from text prompts.  arXiv preprint arXiv:2304.12439, 2023.
  * [73] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan.  Unitune: Text-driven image editing by fine tuning an image generation model on a single image.  arXiv preprint arXiv:2210.09477, 2022.
  * [74] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers, 2022.
  * [75] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  arXiv preprint arXiv:2212.00774, 2022.
  * [76] Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al.  Rodin: A generative model for sculpting 3d digital avatars using diffusion.  In CVPR, 2023.
  * [77] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  arXiv preprint arXiv:2212.11565, 2022.
  * [78] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao.  Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models.  arXiv preprint arXiv:2212.14704, 2022.
  * [79] Longwen Zhang, Qiwei Qiu, Hongyang Lin, Qixuan Zhang, Cheng Shi, Wei Yang, Ye Shi, Sibei Yang, Lan Xu, and Jingyi Yu.  Dreamface: Progressive generation of animatable 3d faces under text guidance.  arXiv preprint arXiv:2304.03117, 2023.
  * [80] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  arXiv preprint arXiv:2302.05543, 2023.
  * [81] Rui Zhao, Wei Li, Zhipeng Hu, Lincheng Li, Zhengxia Zou, Zhenwei Shi, and Changjie Fan.  Zero-shot text-to-parameter translation for game character auto-creation.  In CVPR, 2023.
  * [82] Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen.  General facial representation learning in a visual-linguistic manner.  In CVPR, 2022.
  * [83] Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, and Otmar Hilliges.  I M Avatar: Implicit morphable head avatars from videos.  In CVPR, 2022.
  * [84] Wojciech Zielonka, Timo Bolkart, and Justus Thies.  Instant volumetric head avatars.  In CVPR, 2023.

##  Appendix A Implementation details

###  A.1 Details about 3D scene models

In the coarse stage, we make use of the grid frequency encoder $\gamma(\cdot)$
from the publicly available Stable DreamFusion [69]. This encoder maps the
input $\mathbf{x}\in\mathbb{R}^{3}$ to a higher-frequency dimension, yielding
$\gamma(\mathbf{x})\in\mathbb{R}^{32}$. The MLP within our NeRF model consists
of three layers with dimensions [32, 64, 64, 3+1+3]. Here, the output channels
‘3’, ‘1’, and ‘3’ represent the predicted normals, density value, and RGB
colors, respectively. In the fine stage, we directly optimize the signed
distance value $s_{i}\in\mathbb{R}$, along with a position offset
$\Delta\mathbf{v}_{i}\in\mathbb{R}^{3}$ for each vertex $\mathbf{v}_{i}$. We
found that fitting $s_{i}$ and $\mathbf{v}_{i}$ into MLP, as done by
Fantasia3D [70], often leads to diverged training.

To ensure easy reproducibility, we have included all the hyperparameters used
in our experiments in Tab 1. The other hyper-parameters are set to be the
default of Stable-DreamFusion [69].

Table 1: Hyper-parameters of HeadSculpt.

Camera setting |  $\theta$ range | (20, 110)
---|---|---
Radius range | (1.0, 1.5)
FoV range | (30, 50)
Render setting | Resolution for coarse | (64, 64)
Resolution for fine | (512, 512)
Max num steps sampled per ray | 1024
Iter interval to update extra status | 16
Diffusion setting | Guidance scale | 100
$t$ range | (0.02, 0.98)
$\omega(t)$ | $\sqrt{\alpha_{t}}(1-\alpha_{t})$
Training setting | #Iterations for coarse | $70k$
#Iterations for fine | $50k$
Batch size | 4
LR of grid encoder | 1e-3
LR of NeRF MLP | 1e-3
LR of $s_{i}$ and $\Delta\mathbf{v}_{i}$ in DMTET | 1e-2
LR scheduler | constant
Warmup iterations | $20k$
Optimizer | Adam (0.9, 0.99)
Weight decay | 0
Precision | fp16
Hardware | GPU | 1 $\times$ Tesla V100 (32GB)
Training duration | 1h (coarse) + 1h (fine)

###  A.2 Details about textual inversion

In the main paper, we discussed the collection of a tiny dataset consisting of
34 images depicting the back view of heads. This dataset was used to train a
special token, <back-view>, to address the ambiguity associated with the back
view of landmarks. The images in the dataset were selected to encompass a
diverse range of gender, color, age, and other characteristics. A few samples
from the dataset are shown in Fig. 8. While our simple selection strategy has
proven effective in our specific case, we believe that a more refined
collection process could further enhance the controllability of the learned
<back-view> token. We use the default training recipe provided by HuggingFace
Diffusers
111https://github.com/huggingface/diffusers/blob/main/examples/textual_inversion,
which took us 1 hour on a single Tesla V100 GPU.

Figure 8: Samples of the tiny dataset collected for learning <back-view>
token.

##  Appendix B Further analysis

###  B.1 Effectiveness of textual inversion on 2D generation

To show the effectiveness of the learned <back-view> token, we conduct an
analysis of its control capabilities in the context of 2D generation results.
Specifically, we compare two generation results using Stable Diffusion [65],
with both experiments sharing the same random seed. One experiment has the
plain text prompt appended with the plain phrase “back view,” while the other
experiment utilizes the learned special token <back-view> in the prompt. We
present a selection of randomly generated results in Fig. 9. The observations
indicate that the <back-view> token effectively influences the pose of the
generated heads towards the back, resulting in a distinct appearance.
Remarkably, the <back-view> token demonstrates a notable generalization
ability, as evidenced by the Batman case, despite not having been trained
specifically on back views of Batman in the textual inversion process.

###  B.2 Inherent bias in 2D diffusion models

In our main paper, we discussed the motivation behind our proposed identity-
aware editing score distillation (IESD), which can be attributed to two key
factors. Firstly, the limitations of prompt-based editing [50, 36] are due to
the inherent bias present in Stable Diffusion (SD). Secondly, while
InstructPix2Pix (IP2P) [5] offers a solution by employing instruction-based
editing to mitigate bias, it often results in identity loss. To further
illustrate this phenomenon, we showcase the biased 2D outputs of SD and
ControlNet-based IP2P in Fig. 10. Modified descriptions and instructions are
utilized in these respective methods to facilitate the editing process and
achieve the desired results. The results provide clear evidence of the
following: (1) SD generates biased outcomes, with a tendency to underweight
the “older” aspect and overweight the “skull” aspect in the modified
description; (2) IP2P demonstrates the ability to edit the image successfully,
but it faces challenges in preserving the identity of the avatar.

The aforementioned inherent biases are amplified in the domain of 3D
generation (refer to Fig. 7 in the main paper) due to the optimization process
guided by SDS loss, which tends to prioritize view consistency at the expense
of sacrificing prominent features. To address this issue, our proposed IESD
approach combines two types of scores: one for editing and the other for
identity preservation. This allows us to strike a balance between preserving
the initial appearance and achieving the desired editing outcome.

w/ “back view” |  w/ <back-view> |  w/ “back view” |  w/ <back-view> |  w/ “back view” |  w/ <back-view>
---|---|---|---|---|---
seed: 413 | seed: 16772 | seed: 40805
|  |
a DSLR portrait of Obama
seed: 50682 | seed: 93440 | seed: 96458
|  |
a DSLR portrait of Hillary Clinton
seed: 2367 | seed: 19656 | seed: 62156
|  |
a DSLR portrait of a boy with facial painting
seed: 53236 | seed: 62424 | seed: 72649
|  |
a DSLR portrait of Batman
Figure 9: Analysis of the learned <back-view> on 2D image generation. For each pair of images, we present two 2D images generated with the same random seed, where the left image is conditioned on the plain text "back view" and the right image is conditioned on the <back-view> token. Landmark Map | Stable Diffusion | Reference Image | InstructPix2Pix
---|---|---|---
| seed: 19056 | seed: 72854 |  | seed: 50233 | seed: 64136
|  |  |  |  |
Modified description: a DSLR portrait of +[older] Saul Goodman | Instruction: make him older
| seed: 5427 | seed: 91282 |  | seed: 60104 | seed: 88141
|  |  |  |  |
Modified description: a DSLR portrait skull of Vincent van Gogh | Instruction: turn his face into a skull
Figure 10: Analysis of the inherent bias in 2D diffusion models. For each
case, we display several 2D outputs of SD and IP2P, utilizing modified
descriptions and instructions, respectively, with reference images from our
coarse-stage NeRF model to facilitate the editing process.

##  Appendix C Additional qualitative comparisons

We provide more qualitative comparisons with four baseline methods [69, 40,
61, 70] in Fig. 11 and Fig. 12. These results serve to reinforce the claims
made in Sec 4.1 of the main paper, providing further evidence of the superior
performance of our HeadSculpt in generating high-fidelity head avatars. These
results showcase the ability of our method to capture intricate details,
realistic textures, and overall visual quality, solidifying its position as a
state-of-the-art solution in this task.

Notably, to provide a more immersive and comprehensive understanding of our
results, we include multiple outcomes of our HeadSculpt in the form of
$360^{\circ}$ rotating videos. These videos can be accessed in the
accompanying HTML file, enabling viewers to observe the generated avatars from
various angles and perspectives.

DreamFusion* [69] | Latent-NeRF [40] | 3DFuse [61] | Fantasia3D* [70] | HeadSculpt (Ours)
---|---|---|---|---
|  |  |  |
a DSLR portrait of Batman
|  |  |  |
a DSLR portrait of Black Panther in Marvel
|  |  |  |
a DSLR portrait of Two-face in DC
|  |  |  |
a DSLR portrait of Doctor Strange
|  |  |  |
a head of Terracotta Army
Figure 11: Additional comparisons with existing text-to-3D methods (Part 1). *Non-official. DreamFusion* [69] | Latent-NeRF [40] | 3DFuse [61] | Fantasia3D* [70] | HeadSculpt (Ours)
---|---|---|---|---
|  |  |  |
a head of Simpson in the Simpsons
|  |  |  |
a head of Naruto Uzumaki
|  |  |  |
a DSLR portrait of Napoleon Bonaparte
|  |  |  |
a DSLR portrait of Leo Tolstoy
|  |  |  |
a DSLR portrait of Audrey Hepburn
|  |  |  |
a DSLR portrait of Obama with a baseball cap
|  |  |  |
a DSLR portrait of Taylor Swift
Figure 12: Additional comparisons with existing text-to-3D methods (Part 2).
*Non-official.