# Content-Based Image Retrieval Based on Late Fusion of Binary and Local
Descriptors

Nouman Ali1,2, Danish Ali Mazhar2, Zeshan Iqbal2, Rehan Ashraf1,Jawad Ahmed3,
Farrukh Zeeshan Khan2

<EMAIL_ADDRESS>1Department of Computer Science, National Textile
University, Faisalabad, Pakistan  2Faculty of Telecommunication & Information
Engineering, University of Engineering & Technology, Taxila, Pakistan
3Department of Basic Sciences, University of Engineering & Technology, Taxila,
Pakistan

###### Abstract

One of the challenges in Content-Based Image Retrieval (CBIR) is to reduce the
semantic gaps between low-level features and high-level semantic concepts. In
CBIR, the images are represented in the feature space and the performance of
CBIR depends on the type of selected feature representation. Late fusion also
known as visual words integration is applied to enhance the performance of
image retrieval. The recent advances in image retrieval diverted the focus of
research towards the use of binary descriptors as they are reported
computationally efficient. In this paper, we aim to investigate the late
fusion of Fast Retina Keypoint (FREAK) and Scale Invariant Feature Transform
(SIFT). The late fusion of binary and local descriptor is selected because
among binary descriptors, FREAK has shown good results in classification-based
problems while SIFT is robust to translation, scaling, rotation and small
distortions. The late fusion of FREAK and SIFT integrates the performance of
both feature descriptors for an effective image retrieval. Experimental
results and comparisons show that the proposed late fusion enhances the
performances of image retrieval.

##  I Introduction

In image classification and retrieval-based problems, the extraction of a
meaningful image descriptor is an open research problem [1]. Due to an
increase in the number of image archives, it is necessary to design an
effective system for image search [1, 2]. The conventional annotations-based
approach for image retrieval relies on text and keywords-based image search
[1, 3, 4]. Human manual efforts and difference in the visual perception make
the conventional approach less effective [1, 2]. The modern image search
systems retrieve the images on the basis of image visual contents and this is
referred as CBIR [1]. Image retrieval has vast applications in many domains
like image analysis, search of image over internet, medical image retrieval,
remote sensing and video surveillance [1]. The recent research is carried out
to enhance the performance of image retrieval by retrieving the images that
are similar to the query image [1, 5, 6]. In recent few years, different types
of image representations are proposed that are associated with different
application domains [1, 2, 7, 8, 9, 10]. In this paper, we are interested to
explore a novel image representation based on the late fusion of binary and
local features. The proposed framework can describe the visual contents in a
meaningful way and the main focus of this research is image retrieval.

In CBIR, the commonly used image representations are mainly based on global
and local features [1, 2, 11]. The global feature representations are based on
color, texture and shape [1]. Color is one of the fundamental image feature
and it is not dependent on size, direction and angle [2]. Color features lack
spatial distribution and perceptual meaning [1, 2]. Texture features are
divided into two main classes and texture is extracted to capture the spatial
attributes in the group of pixels [2, 5]. The main limitation of spatial
texture techniques is their sensitivity to noise and distortion [2, 5]. The
spectral texture techniques are dependant on shape and have shown good
performance in-case of square regions [5]. The sensitivity to noise and
computational complexity limits the use of texture features in many
application domains [2, 5]. According to Zhang et al. [12], region-based and
contour-based are the two main approaches that are commonly applied to extract
shape features. The sensitivity to scaling, translation and rotation are the
main limitations of shape-based image representations [2, 5].

The local feature descriptors like Scale Invariant Feature Transform (SIFT)
[13], Speeded-Up Robust Features (SURF) [14] and Histogram of Oriented
Gradients (HOG) [15] have shown good performance in various applications of
image retrieval [7, 16]. Local feature descriptors are robust to translations,
scaling, rotation, and small distortions and are applied in various image
matching applications [7]. According to the recent literature [7], the
classification and retrieval performance of local features is better than
global features.

The recent advances in computer vision turned the attention of research to the
use of binary descriptors [7, 17]. Binary Robust Invariant Scalable Keypoints
(BRISK) [18], Fast Retina Keypoint (FREAK) [19], Binary Robust Independent
Elementary Features (BRIEF)[20] and Features from Accelerated Segment Test
(FAST) [21] are the examples of binary descriptors. Local and binary
descriptors are used to describe the image information in a high-dimensional
feature space by using keypoints [7]. The late fusion (also known as visual
word integration) based on proper features selection improves the
effectiveness of image retrieval. [5, 22, 23]. The performance of FREAK is
reported efficient for classification-based problems [7, 17] while SIFT is
robust to translation, scaling, rotation, and small distortions [24].

Due to these facts, in this paper, we investigated the late fusion/visual word
integration of FREAK and SIFT. Binary and local features are extracted from
training images. A quanization algorithm such as k-means is applied to cluster
two codebooks (visual vocabularies), that are based on descriptors of FREAK
and SIFT, respectively. The codebooks of binary and local features are
combined to represent an image in a dimension that is twice the size of
constructed vocabulary. The main contributions of the proposed research is the
CBIR based on the late fusion of FREAK (binary descriptor) and SIFT (local
descriptor).

The remaining paper is organized as follow: Section II describes the related
work. Section III provides an overview about the proposed research
methodology. Section IV presents evaluation measures and results obtained on
three image benchmarks while Section V concludes our work and points towards
the future directions in the research.

##  II Related Work

CBIR is used to search the images that are visually similar to the query image
[1, 10, 25]. Wang et al. [26] proposed Spatial Weighting BOF (SWBOF) by
extracting the spatial information from different sub-blocks of the images.
Adjacent block distance, local entropy and variance are used for the
extraction of spatial information. The spatial weighting is achieved by
weighting the correspondence visual words with the help of texture
information. Yuan et al. [23] proposed image retrieval by using integration
SIFT and Local Binary Pattern (LBP). The combination of SIFT-LBP is selected
to enhance the performance of image retrieval in-case of noisy background and
ambiguous objects. Yu et al. [22] proposed two new image representations based
on early features fusion for an effective image retrieval. The clusters are
constructed by applying a weighted scheme to maintain a balance between two
features. The two new feature integrations are image-based SIFT-LBP and patch-
based HOG-LBP. The image-based integration of SIFT-LBP outperforms the state-
of-the-art approaches [22]. Zhang et al. [27] proposed a rotation invariant
image matching system by using combination of SIFT and LBP. The regions of LBP
descriptors are calculated by using SIFT detector. Kabbai et al. [28] proposed
a new approach to extract invariant features from the regions of interest. The
uniform pattern is applied to the LBP and Center Symmetric Local Binary
Pattern (CSLBP) for a robust image matching application.

According to [6], the spatial information can be extracted by dividing an
image into Level 1 and Level 2 triangles. Dense SIFT is used for feature
extraction and three different classifiers are applied to determine the best
retrieval performance. Zeng et al. [29] proposed an image representation that
is based on generalized histogram of quantized colors. Gaussian Mixture Models
(GMMs) is applied for quantization and Expectation-Maximization (EM) algorithm
is used for training. Bayesian Information Criterion (BIC) is applied for
determination of quantized color bins. Images are retrieved on the basis of
similarity between the respective spatiograms. Walia et al. [30] proposed a
fusion framework for color-based image retrieval. The color and texture are
extracted by applying Color Difference Histogram (CDH) and Angular Radial
Transform (ART) and modification in CDH algorithm is proposed to make it more
effective. Dubey et al. [31] proposed a rotation and scale invariant hybrid
image descriptor for an efficient image retrieval. The color features are
extracted by quantizing RGB color space while texture is extracted by
structuring the patterns that are generated from locally structured elements.
Color and textural are integrated to construct the inherently Rotation and
Scale-invariant Hybrid image Descriptor (RSHD). Montazer et al. [32] proposed
a new learning method for RBFNNs (Radial Basis Function Neural Networks).
Particle Swarm Optimization (PSO) is applied to initialize the radial basis
function units in more accurate way. The spatial information of the data and
non-linearity of the function are approximated to determine the widths of
RBFNNs. According to Wan et al. [33], the modern machine learning techniques
based on Convolutional Neural Networks (CNN) can reduce the semantic gaps. CNN
model pre-trained on large dataset can be used for feature extraction and the
optimized feature extraction techniques based on deep learning outperforms
conventional features. In our previous work [34], we proposed the visual words
integration (late fusion) of two local features. Different Weighed Averages
(WA) of local features are also calculated to sort out the second best
performance for image retrieval. The research presented in this paper is
different from our previous work [34] as in this paper, we replaced SURF with
FREAK to evaluate the late fusion (visual words integration) of SIFT (local
descriptor) with FREAK (binary descriptor). The experimental results
demonstrates the dominant effect of FREAK when used with the late fusion SIFT
descriptor.

##  III Bag of Features Representation

Fig. 1 represents the block diagram of the proposed image representation that
is based on late fusion of binary and local features. The proposed research is
based on BoVW image representation model [35]. The detail about features
extraction using FREAK, SIFT and late fusion of FREAK and SIFT is mentioned in
the following sub-sections.

Fig. 1: Proposed framework based on late fusion of FREAK and SIFT.

###  III-A Fast Retina Keypoint (FREAK)

FREAK is a binary descriptor that is computed on the basis of brightness
comparison tests around the keypoints and is inspired from the human visual
system [19]. In the first step, a circular sampling grid is applied to
generate retinal sampling pattern with higher density of points near the
center . One-bit Difference of Gaussian (DoG) is applied to compute a binary
descriptor. The feature are calculated by applying a saccadic search. In the
last step, the sum of local gradients over selected pairs are used to compute
the rotation of keypoints [19].

###  III-B Scale Invariant Feature Transform (SIFT)

There are four main steps that are involved to compute SIFT descriptors [13].
The first step involves the computation of interest points, for this purpose
the Difference of Gaussian (DoG) is used to produce several Gaussian blurred
images. The neighborhood image are compared for the calculation of DoG. The
second step involves the calculation of extrema to find out the stable
keypoints and low contrast and pints along the edges are removed by apply
Taylor series. Determinant and trace of Hessian metric are used to remove the
outliers. In the third step, to achieve rotation invariance, the principal
orientation is calculated for the keypoints. The last step involves the
computation of SIFT descriptor. For each keypoint, a set of orientations
histograms are created on 4x4 pixel neighborhoods, with 8 orientation bins
[13].

###  III-C Proposed Late Fusion Based on Binary and Local Descriptors

  1. 1.

In BoVW model, a raw image I is represented as:

$I=(a_{r,s})$ (1)

where r,s are the pixels of image.

  2. 2.

Binary and local features (FREAK and SIFT) are extracted from a set of
training images and an image I is represented as:

$I=\\{d_{1},d_{2},....,d_{T}\\}$ (2)

Where d1 to dT are the image descriptors.

  3. 3.

A quantization algorithm such as k-means is applied to construct the codebook
(visual vocabulary) consisting of Z words, represented as CB:

$CB=\\{w_{1},w_{2},....,w_{Z}\\}$ (3)

where CB is the codebook consisting of wZ visual words, separate codebooks are
constructed for FREAK and SIFT by extracting the respective features.

  4. 4.

The features are extracted from the image and quantized in the feature space.
Mapping of each visual word is done over the image areas and the nearest words
are assigned to the quantized descriptors according to the following equation:

$w(d_{m})=\underset{w\;\varepsilon\;CB}{\text{argmin}}\;Dist(w,d_{m})$ (4)

where w(dm) is representing the visual word assigned to the mth descriptor dm
while Dist(w,dm) is the distance between the descriptor dm and visual word w.
Each image is represented as a collection of patches and each patch is
represented by a visual word (visual words of FREAK are mapped on the images
by using the codebook of FREAK while visual words of SIFT are mapped on the
images by the using the codebook of SIFT).

  5. 5.

The histograms consisting of the visual words of FREAK and SIFT are combined
and an image is represented in a dimension that is twice the size of
constructed codebook. Consider Z as the number of visual words of the
codebook. Let Fi be the set of the descriptors that are mapped to the visual
word wi then the ith bin of the histogram of visual words bi, is the
cardinality of the set Di.

$b_{i}=Card(F_{i})\quad and\quad F_{i}=\\{d_{m},m\in(1,....,Z)\mid
w(d_{m})=w_{i}\\}$ (5)

##  IV Experimental Parameters and Results

The proposed late fusion is evaluated by using three image dataset [36, 37,
38]. We used the image classification parameters as mentioned in [34]. 70% of
the images from each class are selected for training and 30% for testing.
Keeping in view, the unsupervised nature of clustering using k-means, each
experiment is repeated 10 times. During every run, images are randomly
selected for training and testing. The set of selected images for training are
used for the construction of codebook and image retrieval performance is
reported by using the images from test dataset. The images are retrieved by
calculating the closeness among the classifier score values within the same
class.

The size of the codebook affects image retrieval performance [39, 40]. An
increase in the the size of the codebook increases the retrieval precision.
The larger size codebook decreases the retrieval precision due to over-fitting
[39, 40]. We constructed different sizes of codebook from randomly set of
selected training images to sort out the best retrieval performance. The
codebook is constructed by a random selection of 25 %, 50 % and 75 % of
features (both for FREAK and SIFT) per image (from the training dataset).
Additional experiments are performed to evaluate the performance of FREAK and
SIFT separately with the same experimental parameters to represent the
dominant affect of late fusion on image retrieval performance.

###  IV-A Evaluation Measures

We selected precision and recall to determine the performance of our proposed
late fusion [2]. Precision determines the number of correctly retrieved
images.

$Precision=\frac{K_{r}}{X{r}}$ (6)

where Kr represents the number of relevant images similar to the query and Xr
indicates the number of images retrieved by the system in response to the
query.

$Recall=\frac{K_{r}}{X{c}}$ (7)

where Xc is total number of images of that class in the database.

Fig. 2: Randomly selected images from each class of Corel-1000 image dataset
[36].

###  IV-B Performance Evaluation using Corel-1000

There are 10 classes in Corel-1000 image dataset and each class contains 100
images. Fig. 2 represents a sample of randomly selected images from each class
of the Corel-1000. We selected Corel-1000 for the evaluation of proposed
framework as it has been recently used to evaluate the performance CBIR
research [22, 26, 34, 32, 41]. We varied the codebook size to sort out the
best retrieval performance of proposed work. Table I presents the numerical
values of Mean Average Precision (MAP) obtained from the proposed framework
for top 20 retrievals. Fig. 3 presents a comparison of MAP as a function of
codebook size using different evaluation parameters.

TABLE I:  MAP as a function of codebook size (late fusion of FREAK and SIFT)

.  Codebook size and features % used 50 100 200 300 400 600 800 25 % 68.85
70.45 74.36 74.52 74.68 74.58 73.99 50 % 69.31 70.56 73.86 74.91 74.16 74.96
74.01 75 % 70.45 71.13 73.99 73.69 74.55 74.87 74.25 Mean 69.54 70.71 74.07
74.37 74.46 74.80 74.08

$0$$200$$400$$600$$800$$55$$60$$65$$70$$75$Size of the codebookMAPProposed
late fusionSIFTFREAK

Fig. 3: MAP as a function of codebook size (Corel-100).

The results and comparisons presented in Table I and Fig. 3 indicate that the
best MAP is obtained from the proposed last fusion on a codebook of size 600
with a value of 74.80 %. The best MAP obtained by using SIFT and FREAK on a
codebook with a size of 600 is 68.29% and 56.76%, respectively. The MAP on
codebook of each size by using the proposed late fusion is higher than that of
SIFT and FREAK. To present a sustainable performance, the best MAP obtained
from the proposed late fusion is compared with existing research [22, 26, 34,
32, 41]. Table II and Table III presents the comparison of precision and
recall for top 20 retrievals ( as the research selected for comparison is also
reported for top 20 retrievals).

TABLE II: Comparison of retrieval precision. Name of class and method |  Proposed framework |  WA SIFT - SURF [34] |  RBF-NN [32] |  Color SIFT [41] |  Spatial SIFT [26] |  SIFT-LBP [22]
---|---|---|---|---|---|---
Africa |  63.64 |  69.75 |  58.73 |  74.60 |  64 |  57
Beach |  60.99 |  54.25 |  48.94 |  37.80 |  54 |  58
Buildings |  68.21 |  63.95 |  53.74 |  53.90 |  53 |  43
Buses |  92.75 |  89.65 |  95.81 |  96.70 |  94 |  93
Dinosaurs |  100.00 |  98.7 |  98.36 |  99 |  98 |  98
Elephants |  72.64 |  48.8 |  64.14 |  65.90 |  78 |  58
Flowers |  91.54 |  92.3 |  85.64 |  91.20 |  71 |  83
Horses |  80.06 |  89.45 |  80.31 |  86.90 |  93 |  68
Mountains |  59.67 |  47.3 |  54.27 |  58.50 |  42 |  46
Food |  58.56 |  70.9 |  63.14 |  62.20 |  50 |  53
Mean |  74.80 |  70.58 |  70.31 |  72.67 |  69.70 |  65.70
TABLE III: Comparison of recall. Name of class and method |  Proposed framework |  WA SIFT - SURF [34] |  RBF-NN [32] |  Color SIFT [41] |  Spatial SIFT [26] |  SIFT-LBP [22]
---|---|---|---|---|---|---
Africa |  12.73 |  13.95 |  11.75 |  14.92 |  12.80 |  11.40
Beach |  12.20 |  10.85 |  9.79 |  7.56 |  10.80 |  11.60
Buildings |  13.64 |  12.79 |  10.75 |  10.78 |  10.60 |  8.60
Buses |  18.55 |  17.93 |  19.16 |  19.34 |  18.80 |  18.60
Dinosaurs |  20.00 |  19.74 |  19.67 |  19.80 |  19.60 |  19.60
Elephants |  14.53 |  9.76 |  12.83 |  13.18 |  15.60 |  11.60
Flowers |  18.31 |  18.46 |  17.13 |  18.24 |  14.20 |  16.60
Horses |  16.01 |  17.89 |  16.06 |  17.38 |  18.60 |  13.60
Mountains |  11.93 |  9.46 |  10.85 |  11.70 |  8.40 |  9.20
Food |  11.71 |  14.18 |  12.63 |  12.44 |  10.00 |  10.60
Mean |  14.96 |  14.12 |  14.06 |  14.53 |  13.94 |  13.14

The best values of precision and recall are mentioned as bold in Table II and
Table III. The MAP of proposed late fusion outperforms state-of-the-art
research [22, 26, 34, 32, 41]. Fig. 4 and Fig. 5 represents the image
retrieval results obtained on the basis of similarity measures for the
semantic class ”Flowers” and ”Buses”, respectively. The classifier output
label determines the class of the image while the images are retrieved on the
basis of closeness among classifier score values.

Fig. 4: Retrieval result on the basis of similarity measures for the semantic
class ”Flowers”

Fig. 5: Retrieval result on the basis of similarity measures for the semantic
class ”Buses”

Fig. 6: Randomly selected images from each class of Corel-1500 image dataset
[37].

###  IV-C Performance Evaluation Using Corel-1500

There are 15 classes in the Corel-1500 [37] image benchmark and each class
contains 100 images [37]. Fig. 6 represents a sample of randomly selected
images from each class of the Corel-1500. We evaluated the proposed framework
on Corel-1500 and compared the results with state-of-the-art CBIR methods [29,
34]. Fig. 3 presents a comparison of MAP as a function of codebook size using
different evaluation parameters.

$0$$200$$400$$600$$800$$50$$60$$70$Size of the codebookMean average
precisionProposed late fusionSIFTFREAK

Fig. 7: MAP as a function of codebook size (Corel-1500).

The comparison results results presented in Fig. 3 indicate that the best MAP
is obtained from the proposed last fusion on a codebook of size 600 with a
value of 72.59 %. The best MAP obtained by SIFT and FREAK is 66.94 % and 53.23
%, respectively. Table IV presents a comparison of precision and recall
values.

TABLE IV: Comparison of precision and recall. Image retrieval performance |  Proposed late fusion |  WA SIFT-SURF [34] |  SQ + Spatiogram [29]
---|---|---|---
Precision |  72.60 |  68.05 |  63.95
Recall |  14.52 |  13.61 |  12.79

The proposed framework based on late fusion of FREAK and SIFT provides a
better retrieval performance with higher values of precision and recall than
the existing research [29, 34].

Fig. 8: Randomly selected images from each class of OT-Scene image dataset
[38].

###  IV-D Performance Evaluation using Oliva and Torralba (OT-Scene)

There are 08 classes in OT-Scene image dataset that contains a total of 2688
images. Fig. 8 represents a sample of randomly selected images from each class
of the OT-Scene image dataset. We evaluated the proposed framework using OT-
Scene benchmark and compared the results with state-of-the-art CBIR methods
[30, 42]. Fig. 9 presents a comparison of mean precision as a function of
codebook size using different evaluation parameters. Table V represents the
comparison of MAP values.

$0$$200$$400$$600$$800$$1{,}000$$40$$45$$50$$55$$60$$65$Size of the
vocabularyMean average precisionProposed late fusionSIFTFREAK

Fig. 9: MAP as a function of codebook size (OT-Scene).

TABLE V: Mean precision comparison. Image retrieval performance |  Proposed late fusion |  Morphology based IR [42] |  Min Max Fusion [30]
---|---|---|---
MAP |  63.14 |  60.7 |  51.04

The results and comparisons presented in Fig. 9 and Table V indicate that the
proposed late fusion of FREAK and SIFT provides a better retrieval performance
than existing research [30, 42].

##  V Conclusion and Future Directions

The research scope of this paper is based on the late fusion/visual word
integration of binary and local descriptor for an effective image retrieval.
Keeping in view, the good recognition ability and efficient computation time,
we selected FREAK as the binary descriptor. SIFT is selected as the local
feature as it is robust to change in translation, scaling, rotation, and small
distortions. The image is represented in the form of late fusion of FREAK and
SIFT. The proposed research is based on the BoVW framework and classification
is performed by using SVM. Testing is performed by varying the size of
codebook, to sort out the best image retrieval precision. The results obtained
from the proposed research are compared with state-of-the-art CBIR research.
The late fusion of FREAK and SIFT outperforms several CBIR methods. In future,
we will evaluate our proposed work on real world retrieval problem by using a
pre-trained CNN model.

## Acknowledgments

The authors would like to thank Professor James Z. Wang, Pennsylvania State
University, USA for sharing Corel image benchmark that is used for the
evaluation of the proposed research.

## References

  * [1] T. Dharani and I. L. Aroquiaraj, “A survey on content based image retrieval,” in _Pattern Recognition, Informatics and Mobile Engineering (PRIME), 2013 International Conference on_. IEEE, 2013, pp. 485–490.
  * [2] A. Alzu bi, A. Amira, and N. Ramzan, “Semantic content-based image retrieval: A comprehensive study,” _Journal of Visual Communication and Image Representation_ , vol. 32, pp. 20–54, 2015.
  * [3] A. Kumar, J. Kim, W. Cai, M. Fulham, and D. Feng, “Content-based medical image retrieval: a survey of applications to multidimensional and multimodality data,” _Journal of digital imaging_ , vol. 26, no. 6, pp. 1025–1039, 2013\.
  * [4] E. Rashedi, H. Nezamabadi-Pour, and S. Saryazdi, “A simultaneous feature adaptation and feature selection method for content-based image retrieval systems,” _Knowledge-Based Systems_ , vol. 39, pp. 85–94, 2013.
  * [5] D. Zhang, M. M. Islam, and G. Lu, “A review on automatic image annotation techniques,” _Pattern Recognition_ , vol. 45, no. 1, pp. 346–362, 2012.
  * [6] N. Ali, K. B. Bajwa, R. Sablatnig, and Z. Mehmood, “Image retrieval by addition of spatial information based on histograms of triangular regions,” _Computers & Electrical Engineering_, vol. 54, pp. 539–550, 2016.
  * [7] D. Mukherjee, Q. J. Wu, and G. Wang, “A comparative experimental study of image feature detectors and descriptors,” _Machine Vision and Applications_ , vol. 26, no. 4, pp. 443–466, 2015.
  * [8] R. Ashraf, T. Mahmood, A. Irtaza, and K. Bajwa, “A novel approach for the gender classification through trained neural networks,” _J. Basic Appl. Sci. Res_ , vol. 4, pp. 136–144, 2014.
  * [9] R. Ashraf, K. Bashir, A. Irtaza, and M. T. Mahmood, “Content based image retrieval using embedded neural networks with bandletized regions,” _Entropy_ , vol. 17, no. 6, pp. 3552–3580, 2015.
  * [10] R. Ashraf, K. Bashir, and T. Mahmood, “Content-based image retrieval by exploring bandletized regions through support vector machines,” _Journal of Information Science and Engineering_ , vol. 32, no. 2, pp. 245–269, 2016.
  * [11] Z. Mehmood, S. M. Anwar, N. Ali, H. A. Habib, and M. Rashid, “A novel image retrieval based on a combination of local and global histograms of visual words,” _Mathematical Problems in Engineering_ , vol. 2016, 2016.
  * [12] D. Zhang and G. Lu, “Review of shape representation and description techniques,” _Pattern recognition_ , vol. 37, no. 1, pp. 1–19, 2004.
  * [13] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _International journal of computer vision_ , vol. 60, no. 2, pp. 91–110, 2004\.
  * [14] H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robust features,” in _Computer vision–ECCV 2006_. Springer, 2006, pp. 404–417.
  * [15] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _Computer Vision and Pattern Recognition, 2005. CVPR 2005\. IEEE Computer Society Conference on_ , vol. 1. IEEE, 2005, pp. 886–893.
  * [16] T. Krajnik, P. Cristóforis, M. Nitsche, K. Kusumam, and T. Duckett, “Image features and seasons revisited,” in _Mobile Robots (ECMR), 2015 European Conference on_. IEEE, 2015, pp. 1–7.
  * [17] J. Figat, T. Kornuta, and W. Kasprzak, “Performance evaluation of binary descriptors of local features,” in _Computer Vision and Graphics_. Springer, 2014, pp. 187–194.
  * [18] S. Leutenegger, M. Chli, and R. Y. Siegwart, “Brisk: Binary robust invariant scalable keypoints,” in _Computer Vision (ICCV), 2011 IEEE International Conference on_. IEEE, 2011, pp. 2548–2555.
  * [19] A. Alahi, R. Ortiz, and P. Vandergheynst, “Freak: Fast retina keypoint,” in _Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on_. Ieee, 2012, pp. 510–517.
  * [20] M. Calonder, V. Lepetit, C. Strecha, and P. Fua, “Brief: Binary robust independent elementary features,” _Computer Vision–ECCV 2010_ , pp. 778–792, 2010.
  * [21] D. G. Viswanathan, “Features from accelerated segment test (fast),” 2009.
  * [22] J. Yu, Z. Qin, T. Wan, and X. Zhang, “Feature integration analysis of bag-of-features model for image retrieval,” _Neurocomputing_ , vol. 120, pp. 355–364, 2013.
  * [23] X. Yuan, J. Yu, Z. Qin, and T. Wan, “A sift-lbp image retrieval model based on bag of features,” in _IEEE International Conference on Image Processing_ , 2011.
  * [24] C. Valgren and A. J. Lilienthal, “Sift, surf & seasons: Appearance-based long-term localization in outdoor environments,” _Robotics and Autonomous Systems_ , vol. 58, no. 2, pp. 149–156, 2010.
  * [25] Z. Mehmood, S. M. Anwar, M. Altaf, and N. Ali, “A novel image retrieval based on rectangular spatial histograms of visual words,” _Kuwait Journal of Science_ , 2017.
  * [26] C. Wang, B. Zhang, Z. Qin, and J. Xiong, “Spatial weighting for bag-of-features based image retrieval,” in _Integrated Uncertainty in Knowledge Modelling and Decision Making_. Springer, 2013, pp. 91–100.
  * [27] Y.-b. Zheng, X.-s. Huang, and S.-j. Feng, “An image matching algorithm based on combination of sift and the rotation invariant lbp,” _Journal of computer-aided design & computer graphics_, vol. 22, no. 2, pp. 286–292, 2010\.
  * [28] L. Kabbai, A. Azaza, M. Abdellaoui, and A. Douik, “Image matching based on lbp and sift descriptor,” in _Systems, Signals & Devices (SSD), 2015 12th International Multi-Conference on_. IEEE, 2015, pp. 1–6.
  * [29] S. Zeng, R. Huang, H. Wang, and Z. Kang, “Image retrieval using spatiograms of colors quantized by gaussian mixture models,” _Neurocomputing_ , vol. 171, pp. 673–684, 2016.
  * [30] E. Walia and A. Pal, “Fusion framework for effective color image retrieval,” _Journal of Visual Communication and Image Representation_ , vol. 25, no. 6, pp. 1335–1348, 2014.
  * [31] S. R. Dubey, S. K. Singh, and R. K. Singh, “Rotation and scale invariant hybrid image descriptor and retrieval,” _Computers & Electrical Engineering_, vol. 46, pp. 288–302, 2015.
  * [32] G. A. Montazer and D. Giveki, “An improved radial basis function neural network for object image retrieval,” _Neurocomputing_ , vol. 168, pp. 221–233, 2015.
  * [33] J. Wan, D. Wang, S. C. H. Hoi, P. Wu, J. Zhu, Y. Zhang, and J. Li, “Deep learning for content-based image retrieval: A comprehensive study,” in _Proceedings of the ACM International Conference on Multimedia_. ACM, 2014, pp. 157–166.
  * [34] N. Ali, K. B. Bajwa, R. Sablatnig, S. A. Chatzichristofis, Z. Iqbal, M. Rashid, and H. A. Habib, “A novel image retrieval based on visual words integration of sift and surf,” _PloS one_ , vol. 11, no. 6, p. e0157428, 2016.
  * [35] J. Sivic and A. Zisserman, “Video google: A text retrieval approach to object matching in videos,” in _Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on_. IEEE, 2003, pp. 1470–1477.
  * [36] J. Z. Wang, J. Li, and G. Wiederhold, “Simplicity: Semantics-sensitive integrated matching for picture libraries,” _Pattern Analysis and Machine Intelligence, IEEE Transactions on_ , vol. 23, no. 9, pp. 947–963, 2001\.
  * [37] J. Li and J. Z. Wang, “Real-time computerized annotation of pictures,” _Pattern Analysis and Machine Intelligence, IEEE Transactions on_ , vol. 30, no. 6, pp. 985–1002, 2008.
  * [38] A. Oliva and A. Torralba, “Modeling the shape of the scene: A holistic representation of the spatial envelope,” _International journal of computer vision_ , vol. 42, no. 3, pp. 145–175, 2001.
  * [39] E. Nowak, F. Jurie, and B. Triggs, “Sampling strategies for bag-of-features image classification,” in _Computer Vision–ECCV 2006_. Springer, 2006, pp. 490–503.
  * [40] G. Csurka, C. Dance, L. Fan, J. Willamowski, and C. Bray, “Visual categorization with bags of keypoints,” in _Workshop on statistical learning in computer vision, ECCV_ , vol. 1, no. 1-22. Prague, 2004, pp. 1–2.
  * [41] X. Tian, L. Jiao, X. Liu, and X. Zhang, “Feature integration of eodh and color-sift: Application to image retrieval based on codebook,” _Signal Processing: Image Communication_ , vol. 29, no. 4, pp. 530–545, 2014.
  * [42] R. Das, S. Thepade, and S. Ghosh, “Multi technique amalgamation for enhanced information identification with content based image data,” _SpringerPlus_ , vol. 4, no. 1, pp. 1–26, 2015.


$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\bigg{[}\big{(}\widetilde{\mu}_{1}^{*}(X_{i},Z_{i})-\widetilde{\mu}_{0}^{*}(X_{i},Z_{i})\big{)}+\frac{T_{i}}{\widehat{p}}\big{(}\widetilde{Y}_{i}(1)-\widetilde{\mu}_{1}^{*}(X_{i},Z_{i})\big{)}-\frac{1-T_{i}}{1-\widehat{p}}\big{(}\widetilde{Y}_{i}(0)-\widetilde{\mu}_{0}^{*}(X_{i},Z_{i})\big{)}\bigg{]}$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\Gamma_{i}^{*}+\Big{(}\frac{1}{\widehat{p}}-\frac{1}{p}\Big{)}\cdot\frac{1}{n}\sum_{i=1}^{n}T_{i}\big{(}\widetilde{Y}_{i}(1)-\widetilde{\mu}_{1}^{*}(X_{i},Z_{i})\big{)}$
$\displaystyle\qquad+\Big{(}\frac{1}{1-\widehat{p}}-\frac{1}{1-p}\Big{)}\cdot\frac{1}{n}\sum_{i=1}^{n}(1-T_{i})\cdot\big{(}\widetilde{Y}_{i}(0)-\widetilde{\mu}_{0}^{*}(X_{i},Z_{i})\big{)},$

where $\Gamma_{i}^{*}$ is defined in (A.1). Since
$\widehat{p}\stackrel{{\scriptstyle P}}{{\to}}p$ and
$\widetilde{Y}_{i}(w)-\widetilde{\mu}_{w}^{*}(X_{i},Z_{i})$ are i.i.d. with
mean zero, the last two terms in the above summation are both
$o_{P}(1/\sqrt{n})$. Therefore

$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\Gamma_{i}=\mathbb{E}\big{[}Y_{i}(1)\big{]}-\mathbb{E}\big{[}Y_{i}(0)\big{]}+\frac{1}{n}\sum_{i=1}^{n}\Gamma_{i}^{*}+o_{P}(1/\sqrt{n}).$

By a standard argument of delta method or Taylor expansion, we see that

$\displaystyle\widehat{\delta}^{\prime}$
$\displaystyle=\frac{\frac{1}{n}\sum_{i=1}^{n}\Gamma_{i}}{\frac{1}{n}\sum_{i=1}^{n}Z_{i}}=\frac{\mathbb{E}[Y_{i}(1)]-\mathbb{E}[Y_{i}(0)]+\frac{1}{n}\sum_{i=1}^{n}\Gamma_{i}^{*}+o_{P}(1/\sqrt{n})}{\mathbb{E}[Z_{i}]+\frac{1}{n}\sum_{i=1}^{n}(Z_{i}-\mathbb{E}[Z_{i}])}$
$\displaystyle=\frac{\mathbb{E}[Y_{i}(1)]-\mathbb{E}[Y_{i}(0)]}{\mathbb{E}[Z_{i}]}+\frac{1}{n}\sum_{i=1}^{n}\phi_{\delta}^{\prime}(Y_{i},Z_{i},X_{i},T_{i})+o_{P}(1/\sqrt{n}).$

This leads to the asymptotic distribution
$\sqrt{n}(\widehat{\delta}^{\prime}-\delta^{\prime})\stackrel{{\scriptstyle
d}}{{\to}}N(0,\sigma_{\delta^{\prime}}^{2})$, where

$\displaystyle\sigma_{\delta^{\prime}}^{2}=\operatorname{{\rm
Var}}\big{(}\phi_{\delta}^{\prime}(Y_{i},Z_{i},X_{i},T_{i})\big{)}.$

We now proceed to establish the asymptotic consistency of the variance
estimator. To this end, writing $1/\mathbb{E}[Z_{i}]=\theta_{1}$ and
$-\frac{\mathbb{E}[Y_{i}(1)]-\mathbb{E}[Y_{i}(0)]}{\mathbb{E}[Z_{i}]^{2}}=\theta_{2}$,
we have

$\displaystyle\phi_{\delta}^{\prime}(Y_{i},Z_{i},X_{i},T_{i})$
$\displaystyle=T_{i}\cdot\phi_{1}(Y_{i}(1),Y_{i}(0),Z_{i},X_{i})+(1-T_{i})\cdot\phi_{0}(Y_{i}(1),Y_{i}(0),Z_{i},X_{i}),$
$\displaystyle\text{where}\quad\phi_{1}(Y_{i}(1),Y_{i}(0),Z_{i},X_{i})$
$\displaystyle=\theta_{1}\widetilde{\mu}_{1}^{*}(X_{i},Z_{i})-\theta_{1}\widetilde{\mu}_{0}^{*}(X_{i},Z_{i})+\frac{\theta_{1}}{p}\big{(}\widetilde{Y}_{i}(1)-\widetilde{\mu}_{1}^{*}(X_{i},Z_{i})\big{)}+\theta_{2}\big{(}Z_{i}-\mathbb{E}[Z_{i}]\big{)},$
$\displaystyle\phi_{0}(Y_{i}(1),Y_{i}(0),Z_{i},X_{i})$
$\displaystyle=\theta_{1}\widetilde{\mu}_{1}^{*}(X_{i},Z_{i})-\theta_{1}\widetilde{\mu}_{0}^{*}(X_{i},Z_{i})-\frac{\theta_{1}}{1-p}\big{(}\widetilde{Y}_{i}(0)-\widetilde{\mu}_{0}^{*}(X_{i},Z_{i})\big{)}+\theta_{2}\big{(}Z_{i}-\mathbb{E}[Z_{i}]\big{)}.$

Since $T_{i}$’s are independent of all other random variables, we have

$\displaystyle\operatorname{{\rm
Var}}\big{(}\phi_{\delta}^{\prime}(Y_{i},Z_{i},X_{i},T_{i})\big{)}=p\cdot\operatorname{{\rm
Var}}\big{(}\phi_{1}(Y_{i}(1),Y_{i}(0),Z_{i},X_{i})\big{)}+(1-p)\cdot\operatorname{{\rm
Var}}\big{(}\phi_{0}(Y_{i}(1),Y_{i}(0),Z_{i},X_{i})\big{)}.$

Since adding constants does not change variance, the law of large numbers
implies

$\displaystyle\operatorname{{\rm
Var}}\big{(}\phi_{1}(Y_{i}(1),Y_{i}(0),Z_{i},X_{i})\big{)}$
$\displaystyle=\operatorname{{\rm
Var}}\Big{(}\theta_{1}\mu_{1}^{*}(X_{i},Z_{i})-\theta_{1}\mu_{0}^{*}(X_{i},Z_{i})+\frac{\theta_{1}}{p}\big{(}{Y}_{i}(1)-\mu_{1}^{*}(X_{i},Z_{i})\big{)}+\theta_{2}Z_{i}\Big{)}$
$\displaystyle=\frac{1}{n_{t}}\sum_{T_{i}=1}(D_{i}^{*})^{2}-\Big{(}\frac{1}{n_{t}}\sum_{T_{i}=1}D_{i}^{*}\Big{)}^{2}+o_{P}(1),$

where we write

$\displaystyle D_{i}^{*}$
$\displaystyle=\theta_{1}\mu_{1}^{*}(X_{i},Z_{i})-\theta_{1}\mu_{0}^{*}(X_{i},Z_{i})+\frac{\theta_{1}}{p}\big{(}{Y}_{i}(1)-\mu_{1}^{*}(X_{i},Z_{i})\big{)}+\theta_{2}Z_{i}$
$\displaystyle\text{and}\quad D_{1,i}$
$\displaystyle=\widehat{\theta}_{1}\widehat{\mu}_{1}(X_{i},Z_{i})-\widehat{\theta}_{1}\widehat{\mu}_{0}(X_{i},Z_{i})+\frac{\widehat{\theta}_{1}}{\widehat{p}}\big{(}{Y}_{i}(1)-\widehat{\mu}_{1}(X_{i},Z_{i})\big{)}+\widehat{\theta}_{2}Z_{i},$

with $\widehat{\theta}_{1}=1/\bar{Z}\stackrel{{\scriptstyle
P}}{{\to}}\theta_{1}$,
$\widehat{\theta}_{2}=-\frac{\bar{Y}(1)-\bar{Y}(0)}{\bar{Z}^{2}}\stackrel{{\scriptstyle
P}}{{\to}}\theta_{2}$ and $\widehat{p}=n_{t}/n\stackrel{{\scriptstyle
P}}{{\to}}p$ being consistent estimators. Now we define
$D_{1,i}=D_{2,i}+E_{2,i}$, where

$\displaystyle
D_{2,i}=\widehat{\theta}_{1}\mu_{1}^{*}(X_{i},Z_{i})-\widehat{\theta}_{1}\mu_{0}^{*}(X_{i},Z_{i})+\frac{\widehat{\theta}_{1}}{\widehat{p}}\big{(}{Y}_{i}(1)-\mu_{1}^{*}(X_{i},Z_{i})\big{)}+\widehat{\theta}_{2}Z_{i}.$

Also write
$\Delta_{w}(X_{i},Z_{i})=\widehat{\mu}_{w}(X_{i},Z_{i})-\mu_{w}^{*}(X_{i},Z_{i})$
for $w\in\\{0,1\\}$. Then by Cauchy-Schwarz inequality,

$\displaystyle\frac{1}{n_{t}}\sum_{T_{i}=1}E_{2,i}^{2}\leq\widehat{\theta}_{1}^{2}\Big{(}1-\frac{1}{\widehat{p}}\Big{)}^{2}\cdot\frac{2}{n_{t}}\sum_{T_{i}=1}\Delta_{1}(X_{i},Z_{i})^{2}+\widehat{\theta}_{1}^{2}\cdot\frac{2}{n_{t}}\sum_{T_{i}=1}\Delta_{0}(X_{i},Z_{i})^{2}.$

Using similar arguments as in the proof of Theorem 4.3, we have

$\displaystyle\frac{1}{n_{t}}\sum_{T_{i}=1}E_{2,i}^{2}=o_{P}(1),\quad\frac{1}{n_{t}}\sum_{T_{i}=1}E_{2,i}=o_{P}(1),$

and

$\displaystyle\frac{1}{n_{t}}\sum_{T_{i}=1}D_{1,i}^{2}-\Big{(}\frac{1}{n_{t}}\sum_{T_{i}=1}D_{1,i}\Big{)}^{2}$
$\displaystyle=\frac{1}{n_{t}}\sum_{T_{i}=1}D_{2,i}^{2}-\Big{(}\frac{1}{n_{t}}\sum_{T_{i}=1}D_{2,i}\Big{)}^{2}+o_{P}(1).$

Writing $D_{2,i}=D_{i}^{*}+E_{3,i}$, where

$\displaystyle E_{3,i}$
$\displaystyle=(\widehat{\theta}_{1}-\theta_{1})\mu_{1}^{*}(X_{i},Z_{i})-(\widehat{\theta}_{1}-\theta_{1})\mu_{0}^{*}(X_{i},Z_{i})+\Big{(}\frac{\widehat{\theta}_{1}}{\widehat{p}}-\frac{\theta_{1}}{p}\Big{)}\big{(}{Y}_{i}(1)-\mu_{1}^{*}(X_{i},Z_{i})\big{)}+(\widehat{\theta}_{1}-\theta_{2})Z_{i},$

similar arguments as the proof of Theorem 4.3 lead to

$\displaystyle\frac{1}{n_{t}}\sum_{T_{i}=1}E_{3,i}^{2}=o_{P}(1),\quad\frac{1}{n_{t}}\sum_{T_{i}=1}E_{3,i}=o_{P}(1),$

and (noting that $D_{1,i}=g_{1,i}$ as defined in (A.2))

$\displaystyle\operatorname{{\rm
Var}}\big{(}\phi_{1}(Y_{i}(1),Y_{i}(0),Z_{i},X_{i})\big{)}$
$\displaystyle=\frac{1}{n_{t}}\sum_{T_{i}=1}(D_{i}^{*})^{2}-\Big{(}\frac{1}{n_{t}}\sum_{T_{i}=1}D_{i}^{*}\Big{)}^{2}+o_{P}(1)$
$\displaystyle=\frac{1}{n_{t}}\sum_{T_{i}=1}D_{2,i}^{2}-\Big{(}\frac{1}{n_{t}}\sum_{T_{i}=1}D_{2,i}\Big{)}^{2}+o_{P}(1)$
$\displaystyle=\frac{1}{n_{t}}\sum_{T_{i}=1}D_{1,i}^{2}-\Big{(}\frac{1}{n_{t}}\sum_{T_{i}=1}D_{1,i}\Big{)}^{2}+o_{P}(1)=\widehat{V}_{1}+o_{P}(1)$

for $\widehat{V}_{1}$ defined in (A.2). With exactly the same arguments, we
can show

$\displaystyle\operatorname{{\rm
Var}}\big{(}\phi_{0}(Y_{i}(1),Y_{i}(0),Z_{i},X_{i})\big{)}=\widehat{V}_{0}+o_{P}(1).$

Combining the above two results and the fact that
$\widehat{p}\stackrel{{\scriptstyle P}}{{\to}}p$, we know
$\widehat{\sigma}_{\delta^{\prime}}^{2}\stackrel{{\scriptstyle
P}}{{\to}}\sigma_{\delta^{\prime}}^{2}$. By Slutsky’s theorem, we have

$\displaystyle\sqrt{n}(\widehat{\delta}^{\prime}-\delta^{\prime})/\widehat{\sigma}_{\delta^{\prime}}\stackrel{{\scriptstyle
d}}{{\to}}N(0,1),$

which establishes the validity of confidence interval. Therefore, we complete
the proof of Theorem A.3.

∎

##  Appendix D Results on semiparametric efficiency

In this section, we provide proofs for semiparametric efficiency of the
proposed procedures, which are based on variance bounds established in
Appendix E.

###  D.1 Semiparametric efficiency for count metric

###### Proof of Theorem 3.7.

We first compute the asymptotic variance of
$\widehat{\theta}_{\textrm{Debias}}$ and
$\widetilde{\theta}_{\textrm{Debias}}$ under Assumptions 3.4 and 3.6. Here
under the consistency condition, Assumption 3.3 is satisfied with
$\mu_{1}^{*}=\mu_{1}$ and $\mu_{0}^{*}=\mu_{0}$. Thus by Theorem 3.5, the
asymptotic variance is

$\displaystyle\sigma_{*}^{2}$ $\displaystyle=\frac{1}{p}\operatorname{{\rm
Var}}\Big{(}Y_{i}(1)-(1-p)\mu_{1}(X_{i})-p\mu_{0}(X_{i})\Big{)}+\frac{1}{1-p}\operatorname{{\rm
Var}}\Big{(}Y_{i}(0)-(1-p)\mu_{1}(X_{i})-p\mu_{0}(X_{i})\Big{)}.$

To establish the semiparametric efficiency, we quote the classical
semiparametric efficiency results on treatment effects, see Lemma E.6. Here
since $T_{i}$’s are independent of all other random variables, the condition
in Lemma E.6 is satisfied with $e(X_{i})\equiv p$. Recall that
$\mu_{1}(x)=\mathbb{E}[Y_{i}(1){\,|\,}X_{i}=x]$ and
$\mu_{0}(x)=\mathbb{E}[Y_{i}(0){\,|\,}X_{i}=x]$. The asymptotic variance bound
for $\tau$ is

$\displaystyle V^{*}$
$\displaystyle=\mathbb{E}\bigg{[}\frac{\operatorname{{\rm
Var}}(Y_{i}(1){\,|\,}X_{i})}{e(X_{i})}+\frac{\operatorname{{\rm
Var}}(Y_{i}(0){\,|\,}X_{i})}{1-e(X_{i})}+\big{(}\mathbb{E}[Y_{i}(1){\,|\,}X_{i}]-\mathbb{E}[Y_{i}(0){\,|\,}X_{i}]-\tau\big{)}^{2}\bigg{]}$
$\displaystyle=\frac{1}{p}\mathbb{E}\big{[}\operatorname{{\rm
Var}}(Y_{i}(1){\,|\,}X_{i})]+\frac{1}{1-p}\mathbb{E}\big{[}\operatorname{{\rm
Var}}(Y_{i}(0){\,|\,}X_{i})]+\operatorname{{\rm
Var}}\big{(}\mu_{1}(X_{i})-\mu_{0}(X_{i})\big{)}.$

Since $\mu_{1}(X_{i})=\mathbb{E}[Y_{i}(1){\,|\,}X_{i}]$, we know
$Y_{i}(1)-\mu_{1}(X_{i})$ is uncorrelated with
$\mu_{1}(X_{i})-\mu_{0}(X_{i})$. Similarly for $Y_{i}(0)-\mu_{0}(X_{i})$.
Thus,

$\displaystyle\sigma_{*}^{2}$ $\displaystyle=\frac{1}{p}\operatorname{{\rm
Var}}\Big{(}Y_{i}(1)-\mu_{1}(X_{i})+p\big{[}\mu_{1}(X_{i})-\mu_{0}(X_{i})\big{]}\Big{)}$
$\displaystyle\qquad+\frac{1}{1-p}\operatorname{{\rm
Var}}\Big{(}Y_{i}(0)-\mu_{0}(X_{i})-(1-p)\big{[}\mu_{1}(X_{i})-\mu_{0}(X_{i})\big{]}\Big{)}$
$\displaystyle=\frac{1}{p}\operatorname{{\rm
Var}}\big{(}Y_{i}(1)-\mu_{1}(X_{i})\big{)}+p\cdot\operatorname{{\rm
Var}}\big{(}\mu_{1}(X_{i})-\mu_{0}(X_{i})\big{)}$
$\displaystyle\qquad+\frac{1}{1-p}\operatorname{{\rm
Var}}\big{(}Y_{i}(0)-\mu_{0}(X_{i})\big{)}+(1-p)\cdot\operatorname{{\rm
Var}}\big{(}\mu_{1}(X_{i})-\mu_{0}(X_{i})\big{)}$
$\displaystyle=\frac{1}{p}\mathbb{E}\big{[}\operatorname{{\rm
Var}}(Y_{i}(1){\,|\,}X_{i})]+\frac{1}{1-p}\mathbb{E}\big{[}\operatorname{{\rm
Var}}(Y_{i}(0){\,|\,}X_{i})]+\operatorname{{\rm
Var}}\big{(}\mu_{1}(X_{i})-\mu_{0}(X_{i})\big{)}=V^{*}.$

Therefore, $\sigma_{*}^{2}$ is the semiparametric asymptotic variance bound
for $\tau$ in this case. ∎

###  D.2 Semiparametric efficiency for ratio metric: without SDA

###### Proof of Theorem 4.5.

We first show that $\sigma_{\delta,{\dagger}}^{2}$ is the semiparametric
asymptotic variance bound for $\delta$. Applying Theorem E.7, the condition
$T_{i}\perp\\!\\!\\!\perp(Y_{i}(1),Y_{i}(0),Z_{i}(1),Z_{i}(0)){\,|\,}X_{i}$ is
satisfied in the randomized experiment setting and the propensity score is
$e(X_{i})\equiv p$. Therefore, the efficient influence function is
$\phi_{\delta}^{*}$ in (E.2), which coincides with $\phi_{\delta,{\dagger}}$
in (4.3). So the variance bound is given by $\sigma^{2}_{\delta,{\dagger}}$.

It remains to show that
$\sqrt{n}(\widehat{\delta}-\delta)\stackrel{{\scriptstyle
d}}{{\to}}N(0,\sigma_{\delta,{\dagger}}^{2})$. By Assumption 4.4, Assumption
3.3 is satisfied with $\mu_{w,Y}^{*}=\mu_{w,Y}$ and $\mu_{w,Z}^{*}=\mu_{w,Z}$
for $w\in\\{0,1\\}$. In this case,
$\mathbb{E}[\mu_{w,Y}^{*}(X_{i})]=\mathbb{E}[Y_{i}(w)]$ and
$\mathbb{E}[\mu_{w,Z}^{*}(X_{i})]=\mathbb{E}[Z_{i}(w)]$ for $w\in\\{0,1\\}$.
Thus $A_{i}^{*}=A_{i}^{\dagger}$. Therefore, applying Theorem 4.3, we have
$\sqrt{n}(\widehat{\delta}-\delta)\stackrel{{\scriptstyle
d}}{{\to}}N(0,\sigma_{\delta,{\dagger}}^{2})$, which completes the proof. ∎

###  D.3 Semiparametric efficiency for ratio metric: with SDA

###### Proof of Theorem A.5.

We first show that $\sigma_{\delta^{\prime},{\dagger}}^{2}$ is the
semiparametric variance bound for $\delta^{\prime}$. The condition
$T_{i}\perp\\!\\!\\!\perp(Y_{i}(1),Y_{i}(0)){\,|\,}(X_{i},Z_{i})$ in Theorem
E.8 is satisfied here in the randomized experiment since $T_{i}$ is
independent of all other random variables. Thus the efficient influence
function is given by $\phi_{\delta^{\prime}}^{*}$ in (E.6), which coincides
with $\phi_{\delta^{\prime},{\dagger}}$ in (A.3) since $e(X_{i},Z_{i})\equiv
p$. Hence the variance bound is exactly
$\sigma_{\delta^{\prime},{\dagger}}^{2}$.

Then we show that
$\sqrt{n}(\widehat{\delta}^{\prime}-\delta^{\prime})\stackrel{{\scriptstyle
d}}{{\to}}N(0,\sigma_{\delta,{\dagger}}^{2})$. According to Assumption A.4,
Assumption 3.3 is satisfied with $\mu_{1}^{*}=\mu_{1}$ and
$\mu_{0}^{*}=\mu_{0}$, and $\mathbb{E}[Y(1)]=\mathbb{E}[\mu_{1}(X,Z)]$,
$\mathbb{E}[Y(0)]=\mathbb{E}[\mu_{0}(X,Z)]$. Hence
$\Gamma_{i}^{*}=\Gamma_{i}^{\dagger}$ and
$\sigma_{\delta^{\prime}}^{2}=\sigma_{\delta^{\prime},{\dagger}}^{2}$, and
applying Theorem A.3 yields
$\sqrt{n}(\widehat{\delta}^{\prime}-\delta^{\prime})\stackrel{{\scriptstyle
d}}{{\to}}N(0,\sigma_{\delta^{\prime},{\dagger}}^{2})$, hence completing the
proof of Theorem A.5. ∎

##  Appendix E Semiparametric variance bounds

In this part, we establish semiparametric variance bounds for the estimands
$\tau$, $\delta$ and $\delta^{\prime}$, which forms the basis for the
optimality of the proposed procedures.

###  E.1 Backgrounds and definitions

We first introduce basic definitions that are necessary for semiparametric
variance bounds from the literature of semiparametric efficiency. Let $\mu$ be
a fixed $\sigma$-finite measure on $(\mathcal{X},\mathcal{B})$, and let
$L_{2}(\mu)$ be the space of all $L_{2}$-integrable functions with respect to
$\mu$.

Let $\mathcal{P}$ be a collection of measures on $(\mathcal{X},\mathcal{B})$
dominated by $\mu$, which can be equivalently viewed as a subset of
$L_{2}(\mu)$ via the embedding

$\displaystyle
P~{}\to~{}p=\frac{{\,\mathrm{d}}P}{{\,\mathrm{d}}\mu}~{}\to~{}r=\sqrt{p}\in
L_{2}(\mu)$ (E.1)

for all $P\in\mathcal{P}$, where $\frac{{\,\mathrm{d}}P}{{\,\mathrm{d}}\mu}$
is the Radon–Nikodym derivative.

######  Definition E.1 (Regular parametric space).

Suppose $\mathcal{Q}$ is a collection of measures on
$(\mathcal{X},\mathcal{B})$ dominated by $\mu$, which is parametrized by
$\theta\in\Theta\subset\mathcal{R}^{k}$, so that
$\mathcal{Q}=\\{P_{\theta}:\theta\in\Theta\\}$. Denote the embedding
$r(\theta)=\sqrt{p(\theta)}\in L_{2}(\mu)$, where
$p(\theta)=\frac{{\,\mathrm{d}}P_{\theta}}{{\,\mathrm{d}}\mu}$ is the
Radon–Nikodym derivative. We say the parametrization $\theta\to p(\theta)$ is
regular if $\Theta$ is open, and for any $\theta_{0}\in\Theta$, the map
$\theta\to r(\theta)$ is Fréchet differentiable at $\theta_{0}$ with
derivative $\dot{r}(\theta_{0})\in L_{2}(\mu)^{k}$, and the matrix
$\int\dot{r}(\theta_{0})\dot{r}(\theta_{0})^{\top}{\,\mathrm{d}}\mu$ is non-
singular.

######  Definition E.2 (Regular non-parametric space).

Let $\mathcal{P}$ be a collection of measures on $(\mathcal{X},\mathcal{B})$
dominated by $\mu$. An element $P_{0}\in\mathcal{P}$ is regular if it is
contained in a regular parametric subspace of $\mathcal{P}$. We say
$\mathcal{P}$ is regular if every point $P\in\mathcal{P}$ is regular.

######  Definition E.3 (Tangent space).

Let $\mathcal{P}$ be a collection of measures on $(\mathcal{X},\mathcal{B})$
dominated by $\mu$, which can be viewed as a subset $\mathcal{R}$ of
$L_{2}(\mu)$ via the embedding in Equation (E.1). For a fixed
$P_{0}\in\mathcal{P}$, we define the tangent set at $P_{0}$ as the union of
all (one-dimensional) tangent spaces of curves $C\subset\mathcal{P}$ passing
through $P_{0}$. We define tangent space at $P_{0}$ as the closed linear span
of the tangent set at $P_{0}$.

Note that in the above definition, everything considered is in the Hilbert
space $L_{2}(\mu)$.

######  Definition E.4 (Pathwise differentiability).

Let $\mathcal{P}$ be a regular nonparametric model, and
$\beta:\mathcal{P}\to\mathcal{R}$ be a functional on $\mathcal{P}$. Under the
embedding in Equation (E.1), let $\beta$ also denote the mapping
$\beta:L_{2}(\mu)\to\mathcal{R}$ with $\beta(r(P))=\beta(P)$. For a fixed
$P_{0}\in\mathcal{P}$, let $\dot{\mathcal{P}}$ denote the tangent space of
$\mathcal{P}$ at $P_{0}$, and let $r_{0}=r(P_{0})$ be its embedding in
$L_{2}(\mu)$, with tangent space $\dot{\mathcal{R}}$ at $r_{0}$. We say
$\beta$ is pathwise differentiable on $\mathcal{P}$ at $P_{0}$ (or $r_{0}$) if
there exists a bounded linear functional
$\dot{\beta}(r_{0}):\dot{\mathcal{P}}\to\mathcal{R}$ so that

$\displaystyle{\beta}(r(\eta))=\beta(r_{0})+\eta\dot{\beta}(r_{0})(t)+o(|\eta|)$

for any curve $r(\cdot)$ in $\mathcal{R}$ passing through $r_{0}=r(0)$ and
$\dot{r}(0)=t$. We also define
$\dot{\beta}(P_{0}):\dot{\mathcal{P}}\to\mathcal{R}$ by
$\dot{\beta}(P_{0})(h)=\dot{\beta}(r_{0})(hr_{0}/2)$, so that
$\beta(P_{\eta})=\beta(P_{0})+\eta\dot{\beta}(P_{0})(h)+o(|\eta|)$, where
$P_{\eta}$ is a curve in $\mathcal{P}$ corresponding to $r(\eta)$ in
$\mathcal{R}$ and $h=2t/r_{0}$.

The following definition gives the notion of efficient influence function by
defining it as the projection of pathwise derivative on the tangent space, and
the efficient variance, or semiparametric variance bound, is defined as the
variance of efficient influence function. The efficient variance is actually a
lower bound for all asymptotically linear estimators, hence any asymptotically
linear estimator whose asymptotic variance coincide with the efficient
variance is semiparametrically efficient. More detailed results are in the
textbook Bickel et al. (1993).

######  Definition E.5 (Efficient influence function).

Let $\mathcal{P}$ be a regular nonparametric model, and
$\beta:\mathcal{P}\to\mathcal{R}$ be a functional on $\mathcal{P}$ that is
pathwise differentiable at $P_{0}$ with derivative $\dot{\beta}(P_{0})$. We
define the efficient influence function as
$\widetilde{\phi}(\cdot,P_{0}{\,|\,}\beta,\mathcal{P})=\Pi(\dot{\beta}(P_{0}){\,|\,}\dot{\mathcal{P}})$,
the projection of $\dot{\beta}(P_{0})\in L_{2}(\mu)$ onto the tangent space
$\dot{\mathcal{P}}$. We also define the efficient variance as

$\displaystyle\widetilde{V}_{\beta}=\|\widetilde{\phi}(\cdot,P_{0}{\,|\,}\beta,\mathcal{P})\|_{0}^{2}~{},$

where the norm $\|\cdot\|_{0}$ is defined as $\|f\|_{0}^{2}=\langle
f,f\rangle_{0}=\int f^{2}{\,\mathrm{d}}P_{0}$ for any $f\in L_{2}(\mu)$.

Based on these definitions, given an estimand $\beta=\beta(\mathcal{P})$, we
outline the following protocol of providing semiparametric variance bounds,
which will be applied in Appedix E.3 and E.4.

  * •

Step 1: Characterize parametric submodels
$\\{P_{\theta}\colon\theta\in\Theta\\}$ for $\mathcal{P}$.

  * •

Step 2: Find the tangent space ${\mathcal{T}}$ of any regular parametric
submodel $\\{P_{\theta}\colon\theta\in\Theta\\}$.

  * •

Step 3: Find pathwise derivative
${\,\mathrm{d}}\beta(\theta)/{\,\mathrm{d}}\theta$ of the estimand at the
parametric submodel.

  * •

Step 4: Project the pathwise derivative, as a member in the Hilbert space
$L_{2}(\mu)$, onto the tangent space. Then the projection is the efficient
influence function, whose variance is the efficient variance.

###  E.2 Semiparametric variance bound for $\tau$

The semiparametric variance bound for $\tau$ has been established in a general
setting, as in the following lemma.

######  Lemma E.6 (Theorem 1 in Hahn (1998)).

Suppose $T_{i}\perp\\!\\!\\!\perp(Y_{i}(1),Y_{i}(0){\,|\,}X_{i}$, and let
$e(x)=\mathbb{P}(T_{i}=1{\,|\,}X_{i}=x)$ be the propensity score. Then the
asymptotic variance bounds for $\tau=\mathbb{E}[Y_{i}(1)-Y_{i}(0)]$ is

$\displaystyle\mathbb{E}\bigg{[}\frac{\operatorname{{\rm
Var}}(Y_{i}(1){\,|\,}X_{i})}{e(X_{i})}+\frac{\operatorname{{\rm
Var}}(Y_{i}(0){\,|\,}X_{i})}{1-e(X_{i})}+\big{(}\mathbb{E}[Y_{i}(1){\,|\,}X_{i}]-\mathbb{E}[Y_{i}(0){\,|\,}X_{i}]-\tau\big{)}^{2}\bigg{]}.$

###  E.3 Semiparametric variance bound for $\delta$

We establish the efficient influence function and efficient variance for
$\delta$ in the general setting following the protocol in Appedix E.1. Recall
that $\mu_{w,Y}(x)=\mathbb{E}[Y_{i}(w){\,|\,}X_{i}=x]$ and
$\mu_{w,Z}(x)=\mathbb{E}[Z_{i}(w){\,|\,}X_{i}=x]$, $w\in\\{0,1\\}$ are the
true conditional mean functions.

######  Theorem E.7.

Suppose
$(X_{i},Z_{i}(0),Z_{i}(1),Y_{i}(0),Y_{i}(1))\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}\mathbb{P}$
and
$T_{i}\perp\\!\\!\\!\perp(Y_{i}(1),Y_{i}(0),Z_{i}(1),Z_{i}(0)){\,|\,}X_{i}$.
Let $e(x)=\mathbb{P}(T_{i}=1{\,|\,}X_{i}=x)$ be the propensity score. Then the
efficient influence function for
$\delta=\mathbb{E}[Y_{i}(1)]/\mathbb{E}[Z_{i}(1)]-\mathbb{E}[Y_{i}(0)]/\mathbb{E}[Z_{i}(0)]$
is $\phi_{\delta}^{*}(Y_{i},Z_{i},X_{i},T_{i})$, where

$\displaystyle\phi_{\delta}^{*}(Y_{i},Z_{i},X_{i},T_{i})$
$\displaystyle=\frac{A_{i}^{**}}{\mathbb{E}[Z_{i}(1)]}-\frac{\mathbb{E}[Y_{i}(1)]}{\mathbb{E}[Z_{i}(1)]^{2}}B_{i}^{**}-\frac{C_{i}^{**}}{\mathbb{E}[Z_{i}(0)]}+\frac{\mathbb{E}[Y_{i}(0)]}{\mathbb{E}[Z_{i}(0)]^{2}}D_{i}^{**},$
(E.2) $\displaystyle\textrm{where}\quad A_{i}^{**}$
$\displaystyle=\mu_{1,Y}(X_{i})+\frac{T_{i}}{e(X_{i})}\big{(}Y_{i}-\mu_{1,Y}(X_{i})\big{)}-\mathbb{E}[Y_{i}(1)],$
$\displaystyle B_{i}^{**}$
$\displaystyle=\mu_{1,Z}(X_{i})+\frac{T_{i}}{e(X_{i})}\big{(}Z_{i}-\mu_{1,Z}(X_{i})\big{)}-\mathbb{E}[Z_{i}(1)],$
$\displaystyle C_{i}^{**}$
$\displaystyle=\mu_{0,Y}^{*}(X_{i})+\frac{1-T_{i}}{1-e(X_{i})}\big{(}Y_{i}-\mu_{0,Y}(X_{i})\big{)}-\mathbb{E}[Y_{i}(0)],$
$\displaystyle D_{i}^{**}$
$\displaystyle=\mu_{0,Z}^{*}(X_{i})+\frac{1-T_{i}}{1-e(X_{i})}\big{(}Z_{i}-\mu_{0,Z}(X_{i})\big{)}-\mathbb{E}[Z_{i}(0)].$

The asymptotic variance bound for $\delta$ is
$V_{\delta}^{*}=\operatorname{{\rm
Var}}(\phi_{\delta}^{*}(Y_{i},Z_{i},X_{i},T_{i}))$.

###### Proof of Theorem E.7.

We follow the general protocol in Appendix E.1, which is similar to Hahn
(1998).

#### Step 1: Parametric submodel.

Under the strong ignorability
$T_{i}\perp\\!\\!\\!\perp(Y_{i}(1),Y_{i}(0),Z_{i}(1),Z_{i}(0)){\,|\,}X_{i}$,
the density of the joint distribution of
$(X_{i},Z_{i}(0),Z_{i}(1),Y_{i}(0),Y_{i}(1))\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}\mathbb{P}$
is given by

$\displaystyle\bar{q}(x,y_{0},y_{1},z_{0},z_{1},t)=f(x)\cdot
f(y_{0},y_{1},z_{0},z_{1}{\,|\,}x)\cdot e(x)^{t}(1-e(x))^{1-t},$

where $f(y_{0},y_{1},z_{0},z_{1}{\,|\,}x)$ is the conditional density
function. The joint distribution of the observed quantities
$(X_{i},Y_{i},Z_{i},T_{i})$ where $Y_{i}=Y_{i}(T_{i})$ and
$Z_{i}=Z_{i}(T_{i})$ is thus

$\displaystyle
q(x,y,z,t)=\big{[}f_{1}(y,z{\,|\,}x)e(x)\big{]}^{t}\cdot\big{[}f_{0}(y,z{\,|\,}x)(1-e(x))\big{]}^{1-t}f(x),$

where $f_{w}(y,z{\,|\,}x)$ is the conditional density function of $Y(w),Z(w)$
given $X$ for $w\in\\{0,1\\}$. Then a regular parametric submodel containing
$\mathbb{P}$ is characterized by

$\displaystyle
q(x,y,z,t;\theta)=\big{[}f_{1}(y,z{\,|\,}x,\theta)e(x,\theta)\big{]}^{t}\cdot\big{[}f_{0}(y,z{\,|\,}x,\theta)(1-e(x,\theta))\big{]}^{1-t}f(x,\theta),$

where the submodel is parametrizedby $\theta\in\Theta$, and the density equals
the true density $q(x,y,z,t)$ when $\theta=\theta_{0}$. Also, the functions
$f_{1}(y,z{\,|\,}x,\theta)$, $f_{0}(y,z{\,|\,}x,\theta)$, $f(x,\theta)$ and
$e(x,\theta)$ is differentiable w.r.t. $\theta$ at $\theta_{0}$.

#### Step 2: Tangent space.

By a straightforward calculation, the score function at $\mathbb{P}(\theta)$
with density $q(x,y,z,t;\theta)$ is

$\displaystyle
s(\theta;x,y,z,t)=t\cdot\frac{\dot{f}_{1}(y,z{\,|\,}x,\theta)}{f_{1}(y,z{\,|\,}x,\theta)}+(1-t)\cdot\frac{\dot{f}_{0}(y,z{\,|\,}x,\theta)}{f_{0}(y,z{\,|\,}x,\theta)}+\frac{t-e(x,\theta)}{e(x,\theta)(1-e(x,\theta))}\cdot\dot{e}(x,\theta)+\frac{\dot{f}(x,\theta)}{f(x,\theta)},$

where

$\displaystyle\dot{f}_{w}(y,z{\,|\,}x,\theta)=\frac{\partial}{\partial\theta}f_{w}(y,z{\,|\,}x,\theta),~{}w\in\\{0,1\\},\quad\dot{e}(x,\theta)=\frac{\partial}{\partial\theta}{e}(x,\theta),\quad\dot{f}(x,\theta)=\frac{\partial}{\partial\theta}f(x,\theta).$

Therefore, the tangent space at $\mathbb{P}$ is

$\displaystyle{\mathcal{T}}=\Big{\\{}$ $\displaystyle t\cdot
s_{1}(y,z{\,|\,}x)+(1-t)\cdot
s_{0}(y,z{\,|\,}x)+a(x)\big{(}t-e(x)\big{)}+s(x)\colon$ (E.3)
$\displaystyle\quad\int
s_{w}(y,z{\,|\,}x)f_{w}(y,z{\,|\,}x){\,\mathrm{d}}y{\,\mathrm{d}}z=0,\forall
w\in\\{0,1\\},~{}\forall x,z,$ (E.4) $\displaystyle\qquad~{}\int
s(x)f(x){\,\mathrm{d}}x=0,~{}a(x)\text{ is square integrable}\Big{\\}}.$

#### Step 3: Pathwise derivative.

The target $\delta$ can be written as

$\delta(\theta)=\frac{\int
yf_{1}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z}{\int
zf_{1}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z}-\frac{\int
yf_{0}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z}{\int
zf_{0}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z}.$

Taking derivative w.r.t. $\theta$, we have

$\displaystyle\frac{\partial\delta(\theta)}{\partial\theta}$
$\displaystyle=\frac{1}{\mathbb{E}[Z(1)]}\bigg{(}\int
y\dot{f}_{1}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z+\int
yf_{1}(y,z{\,|\,}x,\theta)\dot{f}(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z\bigg{)}$
$\displaystyle\quad-\frac{\mathbb{E}[Y(1)]}{\mathbb{E}[Z(1)]^{2}}\bigg{(}\int
z\dot{f}_{1}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z+\int
zf_{1}(y,z{\,|\,}x,\theta)\dot{f}(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z\bigg{)}$
$\displaystyle\quad-\frac{1}{\mathbb{E}[Z(0)]}\bigg{(}\int
y\dot{f}_{0}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z+\int
yf_{0}(y,z{\,|\,}x,\theta)\dot{f}(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z\bigg{)}$
$\displaystyle\quad+\frac{\mathbb{E}[Y(0)]}{\mathbb{E}[Z(0)]^{2}}\bigg{(}\int
z\dot{f}_{0}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z+\int
zf_{0}(y,z{\,|\,}x,\theta)\dot{f}(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z\bigg{)}.$
(E.5)

#### Step 4: Projection onto tangent space.

We now establish the projection of $\partial\delta(\theta)/\partial\theta$
onto the tangent space ${\mathcal{T}}$ by showing that the influence function
$\phi_{\delta}^{*}$ in (4.3) is in ${\mathcal{T}}$, and is essentially the
projection. Note that $\phi_{\delta}^{*}\in{\mathcal{T}}$ by setting

$\displaystyle s_{1}(y,z{\,|\,}x)$
$\displaystyle=\frac{1}{e(x)}\frac{1}{\mathbb{E}[Z(1)]}\big{(}y-\mu_{1,Y}(x)\big{)}-\frac{1}{e(x)}\frac{\mathbb{E}[Y(1)]}{\mathbb{E}[Z(1)]^{2}}\big{(}z-\mu_{1,z}(x)\big{)},$
$\displaystyle s_{0}(y,z{\,|\,}x)$
$\displaystyle=-\frac{1}{1-e(x)}\frac{1}{\mathbb{E}[Z(0)]}\big{(}y-\mu_{0,Y}(x)\big{)}+\frac{1}{1-e(x)}\frac{\mathbb{E}[Y(0)]}{\mathbb{E}[Z(0)]^{2}}\big{(}z-\mu_{0,z}(x)\big{)},$
$\displaystyle t(x)$
$\displaystyle=\frac{\mu_{1,Y}(x)-\mathbb{E}[Y(1)]}{\mathbb{E}[Z(1)]}-\frac{\mathbb{E}[Y(1)]}{\mathbb{E}[Z(1)]^{2}}\big{(}\mu_{1,z}(x)-\mathbb{E}[Z(1)]\big{)}$
$\displaystyle\quad-\frac{\mu_{0,Y}(x)-\mathbb{E}[Y(0)]}{\mathbb{E}[Z(0)]}+\frac{\mathbb{E}[Y(0)]}{\mathbb{E}[Z(0)]^{2}}\big{(}\mu_{0,z}(x)-\mathbb{E}[Z(0)]\big{)}.\quad
a(x)\equiv 0$

in the definition of ${\mathcal{T}}$ in (E.3). Now we show that
$\phi_{\delta}^{*}$ is the projection onto ${\mathcal{T}}$, for which it
suffices to show

$\displaystyle\frac{\partial\delta(\theta)}{\partial\theta}=\mathbb{E}_{\theta}\big{[}s(\theta;X_{i},Y_{i},Z_{i},T_{i})\cdot\phi_{\delta}^{*}(Y_{i},Z_{i},X_{i},T_{i})\big{]},$

which implies $\phi_{\delta}^{*}(\cdot)$ is the Riesz representation of
$\partial\delta(\theta)/\partial\theta$ in $L_{2}(\mu)$. In light of (4.3) and
(E.3), it suffices to show the following four relations

$\displaystyle\mathbb{E}\big{[}A_{i}^{**}\cdot
s(\theta;X_{i},Y_{i},Z_{i},T_{i})\big{]}$ $\displaystyle=\int
y\dot{f}_{1}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z+\int
yf_{1}(y,z{\,|\,}x,\theta)\dot{f}(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z,$
$\displaystyle\mathbb{E}\big{[}B_{i}^{**}\cdot
s(\theta;X_{i},Y_{i},Z_{i},T_{i})\big{]}$ $\displaystyle=\int
z\dot{f}_{1}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z+\int
zf_{1}(y,z{\,|\,}x,\theta)\dot{f}(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z,$
$\displaystyle\mathbb{E}\big{[}C_{i}^{**}\cdot
s(\theta;X_{i},Y_{i},Z_{i},T_{i})\big{]}$ $\displaystyle=\int
y\dot{f}_{0}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z+\int
yf_{0}(y,z{\,|\,}x,\theta)\dot{f}(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z$
$\displaystyle\mathbb{E}\big{[}D_{i}^{**}\cdot
s(\theta;X_{i},Y_{i},Z_{i},T_{i})\big{]}$ $\displaystyle=\int
z\dot{f}_{0}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z+\int
zf_{0}(y,z{\,|\,}x,\theta)\dot{f}(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z.$

By symmetric roles of $Y_{i},Z_{i}$ and $T_{i},1-T_{i}$, we only show the
first equation, and others follow exactly the same arguments. Note that

$\displaystyle\int
y\dot{f}_{1}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z+\int
yf_{1}(y,z{\,|\,}x,\theta)\dot{f}(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z$
$\displaystyle=\int
y\frac{\dot{f}_{1}(y,z{\,|\,}x,\theta)}{f_{1}(y,z{\,|\,}x,\theta)}\cdot
f_{1}(y,z{\,|\,}x,\theta)f(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z+\int
y\frac{\dot{f}(x,\theta)}{{f}(x,\theta)}\cdot
f_{1}(y,z{\,|\,}x,\theta){f}(x,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z$
$\displaystyle=\mathbb{E}\bigg{[}Y(1)\cdot\frac{\dot{f}_{1}(Y(1),Z(1){\,|\,}X,\theta)}{f_{1}(Y(1),Z(1){\,|\,}X,\theta)}\bigg{]}+\mathbb{E}\bigg{[}Y(1)\cdot\frac{\dot{f}(X,\theta)}{f(X,\theta)}\bigg{]}.$

On the other hand,

$\displaystyle\mathbb{E}\big{[}A_{i}^{**}\cdot
s(\theta;X_{i},Y_{i},Z_{i},T_{i})\big{]}$
$\displaystyle=\mathbb{E}\bigg{[}A_{i}^{**}\cdot
T_{i}\cdot\frac{\dot{f}_{1}(Y,Z{\,|\,}X,\theta)}{f_{1}(Y,Z{\,|\,}X,\theta)}\bigg{]}+\mathbb{E}\bigg{[}A_{i}^{**}\cdot(1-T_{i})\cdot\frac{\dot{f}_{0}(Y,Z{\,|\,}X,\theta)}{f_{0}(Y,Z{\,|\,}X,\theta)}\bigg{]}$
$\displaystyle\qquad+\mathbb{E}\bigg{[}A_{i}^{**}\cdot\frac{T_{i}-e(X_{i})}{e(X_{i})(1-e(X_{i}))}\cdot\dot{e}(X_{i},\theta)\bigg{]}+\mathbb{E}\bigg{[}A_{i}^{**}\cdot\frac{\dot{f}(X_{i},\theta)}{f(X_{i},\theta)}\bigg{]}.$

By the definition of $A_{i}^{**}$,

$\displaystyle\mathbb{E}\bigg{[}A_{i}^{**}\cdot
T_{i}\cdot\frac{\dot{f}_{1}(Y,Z{\,|\,}X,\theta)}{f_{1}(Y,Z{\,|\,}X,\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\big{(}\mu_{1,Y}(X)-\mathbb{E}[Y(1)]\big{)}\cdot
T\cdot\frac{\dot{f}_{1}(Y,Z{\,|\,}X,\theta)}{f_{1}(Y,Z{\,|\,}X,\theta)}\bigg{]}$
$\displaystyle\qquad+\mathbb{E}\bigg{[}\big{(}Y(1)-\mu_{1,Y}(X)\big{)}\cdot\frac{T}{e(X)}\cdot\frac{\dot{f}_{1}(Y,Z{\,|\,}X,\theta)}{f_{1}(Y,Z{\,|\,}X,\theta)}\bigg{]},$

where by the tower property and takeout property of conditional expectations,

$\displaystyle\mathbb{E}\bigg{[}\big{(}\mu_{1,Y}(X)-\mathbb{E}[Y(1)]\big{)}\cdot
T\cdot\frac{\dot{f}_{1}(Y,Z{\,|\,}X,\theta)}{f_{1}(Y,Z{\,|\,}X,\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\big{(}\mu_{1,Y}(X)-\mathbb{E}[Y(1)]\big{)}\cdot\mathbb{E}[T{\,|\,}X]\cdot\mathbb{E}\Big{[}\frac{\dot{f}_{1}(Y,Z{\,|\,}X,\theta)}{f_{1}(Y,Z{\,|\,}X,\theta)}\,\Big{|}\,X\Big{]}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\big{(}\mu_{1,Y}(X)-\mathbb{E}[Y(1)]\big{)}\cdot
e(X)\cdot\frac{\dot{f}_{1}(Y,Z{\,|\,}X,\theta)}{f_{1}(Y,Z{\,|\,}X,\theta)}\bigg{]}=0,$

where the last equality follows the fact that for any measurable function
$g(\cdot)$,

$\displaystyle\mathbb{E}\bigg{[}g(X)\frac{\dot{f}_{1}(Y,Z{\,|\,}X,\theta)}{f_{1}(Y,Z{\,|\,}X,\theta)}\bigg{]}=\int_{x}g(x)f(x,\theta)\int_{y,z}\dot{f}_{1}(y,z{\,|\,}x,\theta){\,\mathrm{d}}y{\,\mathrm{d}}z{\,\mathrm{d}}x=0.$

Similarly,

$\displaystyle\mathbb{E}\bigg{[}\big{(}Y(1)-\mu_{1,Y}(X)\big{)}\cdot\frac{T}{e(X)}\cdot\frac{\dot{f}_{1}(Y,Z{\,|\,}X,\theta)}{f_{1}(Y,Z{\,|\,}X,\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\big{(}Y(1)-\mu_{1,Y}(X)\big{)}\cdot\frac{\dot{f}_{1}(Y,Z{\,|\,}X,\theta)}{f_{1}(Y,Z{\,|\,}X,\theta)}\bigg{]}=\mathbb{E}\bigg{[}Y(1)\cdot\frac{\dot{f}_{1}(Y(1),Z(1){\,|\,}X,\theta)}{f_{1}(Y(1),Z(1){\,|\,}X,\theta)}\bigg{]}.$

Meanwhile, still by the conditional independence of $Y_{i}(1),Z_{i}(1)$ and
$T_{i}$ given $X_{i}$ as well as properties of conditional expectations,

$\displaystyle\mathbb{E}\bigg{[}A_{i}^{**}\cdot(1-T_{i})\cdot\frac{\dot{f}_{0}(Y,Z{\,|\,}X,\theta)}{f_{0}(Y,Z{\,|\,}X,\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\big{(}\mu_{1,Y}(X)-\mathbb{E}[Y(1)]\big{)}\cdot(1-T)\cdot\frac{\dot{f}_{0}(Y,Z{\,|\,}X,\theta)}{f_{0}(Y,Z{\,|\,}X,\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\big{(}\mu_{1,Y}(X)-\mathbb{E}[Y(1)]\big{)}\cdot\big{(}1-e(X)\big{)}\cdot\frac{\dot{f}_{0}(Y,Z{\,|\,}X,\theta)}{f_{0}(Y,Z{\,|\,}X,\theta)}\bigg{]}=0.$

For the third term,

$\displaystyle\mathbb{E}\bigg{[}A_{i}^{**}\cdot\frac{T_{i}-e(X_{i})}{e(X_{i})(1-e(X_{i}))}\cdot\dot{e}(X_{i},\theta)\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\Big{(}\mu_{1,Y}(X)-\mathbb{E}[Y(1)]\Big{)}\cdot\frac{T-e(X)}{e(X)(1-e(X))}\cdot\dot{e}(X,\theta)\bigg{]}$
$\displaystyle\qquad+\mathbb{E}\bigg{[}\frac{T}{e(X)}\big{(}Y-\mu_{1,Y}(X)\big{)}\cdot\frac{T-e(X)}{e(X)(1-e(X))}\cdot\dot{e}(X,\theta)\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\Big{(}\mu_{1,Y}(X)-\mathbb{E}[Y(1)]\Big{)}\cdot\frac{\mathbb{E}[T{\,|\,}X]-e(X)}{e(X)(1-e(X))}\cdot\dot{e}(X,\theta)\bigg{]}$
$\displaystyle\qquad+\mathbb{E}\bigg{[}\frac{T}{e(X)}\big{(}Y(1)-\mu_{1,Y}(X)\big{)}\cdot\frac{T-e(X)}{e(X)(1-e(X))}\cdot\dot{e}(X,\theta)\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\frac{T}{e(X)}\big{(}Y(1)-\mu_{1,Y}(X)\big{)}\cdot\frac{T-e(X)}{e(X)(1-e(X))}\cdot\dot{e}(X,\theta)\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\frac{T}{e(X)}\big{(}\mathbb{E}[Y(1){\,|\,}X]-\mu_{1,Y}(X)\big{)}\cdot\frac{T-e(X)}{e(X)(1-e(X))}\cdot\dot{e}(X,\theta)\bigg{]}=0.$

Here the second equality follows from the tower property and takeout property
and the fact that $TY=TY(1)$, the forth equality follows from the conditional
independence of $Y(1)$ and $T$ given $X$ as well as the tower property and
takeout property. The last term satisfies

$\displaystyle\mathbb{E}\bigg{[}A_{i}^{**}\cdot\frac{\dot{f}(X_{i},\theta)}{f(X_{i},\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\Big{(}\mu_{1,Y}(X)-\mathbb{E}[Y(1)]\Big{)}\cdot\frac{\dot{f}(X,\theta)}{f(X,\theta)}\bigg{]}+\mathbb{E}\bigg{[}\frac{T}{e(X)}\big{(}Y-\mu_{1,Y}(X)\big{)}\cdot\frac{\dot{f}(X,\theta)}{f(X,\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\frac{T}{e(X)}\big{(}Y(1)-\mu_{1,Y}(X)\big{)}\cdot\frac{\dot{f}(X,\theta)}{f(X,\theta)}\bigg{]}=\mathbb{E}\bigg{[}Y(1)\cdot\frac{\dot{f}(X,\theta)}{f(X,\theta)}\bigg{]}.$

The above derivation uses the fact that $TY=TY(1)$, $Y(1)\perp\\!\\!\\!\perp
T{\,|\,}X$ and
$\mathbb{E}\big{[}g(X)\cdot\frac{\dot{f}(X,\theta)}{f(X,\theta)}\big{]}=0$ for
all measurable function $g(\cdot)$. Combining the four terms, we arrive at the
first of the desired relations. By symmetric roles of $Y_{i},Z_{i}$ and
$T_{i},1-T_{i}$, the other four follow similarly. Thus, we complete the proof
that $\phi_{\delta}^{*}$ is the projection of
$\partial\delta(\theta)/\partial\theta$ onto ${\mathcal{T}}$. Hence,
$\phi_{\delta}^{*}$ is the efficient influence function, and its variance is
the semiparametric asymptotic variance bound for estimating $\delta$. ∎

###  E.4 Semiparametric variance bound for $\delta^{\prime}$

Similar to the case of $\delta$, we establish the efficient influence function
and efficient variance for $\delta^{\prime}$ in the general setting following
the protocol in Appedix E.1. Recall that
$\mu_{w}(x,z)=\mathbb{E}[Y_{i}(w){\,|\,}X_{i}=x,Z_{i}=z]$, $w\in\\{0,1\\}$ are
the true conditional mean functions.

######  Theorem E.8.

Suppose
$(X_{i},Z_{i},Y_{i}(0),Y_{i}(1))\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}\mathbb{P}$
and $T_{i}\perp\\!\\!\\!\perp(Y_{i}(1),Y_{i}(0)){\,|\,}(X_{i},Z_{i})$. Let
$e(x,z)=\mathbb{P}(T_{i}=1{\,|\,}X_{i}=x,Z_{i}=z)$ be the propensity score.
Then the efficient influence function for
$\delta^{\prime}=\mathbb{E}[Y_{i}(1)]/\mathbb{E}[Z_{i}]-\mathbb{E}[Y_{i}(0)]/\mathbb{E}[Z_{i}]$
is $\phi_{\delta^{\prime}}^{*}(Y_{i},Z_{i},X_{i},T_{i})$, where

$\displaystyle\phi_{\delta^{\prime}}^{*}(Y_{i},Z_{i},X_{i},T_{i})$
$\displaystyle=\frac{\Gamma_{i}^{**}}{\mathbb{E}[Z_{i}]}-\frac{\mathbb{E}[Y_{i}(1)]-\mathbb{E}[Y_{i}(0)]}{\mathbb{E}[Z_{i}]^{2}}\big{(}Z_{i}-\mathbb{E}[Z_{i}]),$
(E.6) $\displaystyle\textrm{where}\quad\Gamma_{i}^{**}$
$\displaystyle=\mu_{1}(X_{i},Z_{i})-\mu_{0}(X_{i},Z_{i})-\big{(}\mathbb{E}[Y_{i}(1)]-\mathbb{E}[Y_{i}(0)]\big{)}$
(E.7)
$\displaystyle\quad+\frac{T_{i}}{e(X_{i},Z_{i})}\big{(}Y_{i}-\mu_{1}(X_{i},Z_{i})\big{)}-\frac{1-T_{i}}{1-e(X_{i},Z_{i})}\big{(}Y_{i}-\mu_{0}(X_{i},Z_{i})\big{)}.$

The asymptotic variance bound for $\delta$ is
$V_{\delta^{\prime}}^{*}=\operatorname{{\rm
Var}}(\phi_{\delta^{\prime}}^{*}(Y_{i},Z_{i},X_{i},T_{i}))$.

###### Proof of Theorem E.8.

The proof follows the protocol in Appendix E.1.

#### Step 1: Parametric submodel.

Under the strong ignorability
$T_{i}\perp\\!\\!\\!\perp(Y_{i}(1),Y_{i}(0)){\,|\,}(X_{i},Z_{i})$, the density
of the joint distribution of
$(X_{i},Z_{i},Y_{i}(0),Y_{i}(1),T_{i})\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}\mathbb{P}$
is given by

$\displaystyle\bar{q}(x,y_{0},y_{1},z,t)=f(x,z)\cdot
f(y_{0},y_{1},{\,|\,}x,z)\cdot e(x)^{t}(1-e(x))^{1-t},$

where $f(y_{0},y_{1}{\,|\,}x,z)$ is the conditional density function. The
joint distribution of the observed quantities $(X_{i},Y_{i},Z_{i},T_{i})$
where $Y_{i}=Y_{i}(T_{i})$ is

$\displaystyle
q(x,y,z,t)=\big{[}f_{1}(y{\,|\,}x,z)e(x,z)\big{]}^{t}\cdot\big{[}f_{0}(y{\,|\,}x,z)(1-e(x,z))\big{]}^{1-t}f(x,z),$

where $f_{w}(y{\,|\,}x,z)$ is the conditional density function of $Y(w)$ given
$X,Z$ for $w\in\\{0,1\\}$. Then a regular parametric submodel containing
$\mathbb{P}$ is characterized by

$\displaystyle
q(x,y,z,t;\theta)=\big{[}f_{1}(y{\,|\,}x,z,\theta)e(x,z,\theta)\big{]}^{t}\cdot\big{[}f_{0}(y{\,|\,}x,z,\theta)(1-e(x,z,\theta))\big{]}^{1-t}f(x,z,\theta),$

where the submodel is parametrizedby $\theta\in\Theta$, and the density equals
the true density $q(x,y,z,t)$ when $\theta=\theta_{0}$. Also, the functions
$f_{1}(y,z{\,|\,}x,\theta)$, $f_{0}(y{\,|\,}x,z,\theta)$, $f(x,z,\theta)$ and
$e(x,z,\theta)$ are differentiable w.r.t. $\theta$ at $\theta=\theta_{0}$.

#### Step 2: Tangent space.

By the joint distribution of the observable, the score function at $\theta$ or
$\mathbb{P}(\theta)$ is

$\displaystyle
s(\theta;x,y,z,t)=t\cdot\frac{\dot{f}_{1}(y{\,|\,}x,z,\theta)}{f_{1}(y{\,|\,}x,z,\theta)}+(1-t)\cdot\frac{\dot{f}_{0}(y{\,|\,}x,z,\theta)}{f_{0}(y{\,|\,}x,z,\theta)}+\frac{t-e(x,z,\theta)}{e(x,z,\theta)(1-e(x,z,\theta))}\cdot\dot{e}(x,z,\theta)+\frac{\dot{f}(x,z,\theta)}{f(x,z,\theta)},$
(E.8)

where

$\displaystyle\dot{f}_{w}(y{\,|\,}x,z,\theta)=\frac{\partial}{\partial\theta}f_{w}(y{\,|\,}x,z,\theta),~{}w\in\\{0,1\\},\quad\dot{e}(x,z,\theta)=\frac{\partial}{\partial\theta}{e}(x,z,\theta),\quad\dot{f}(x,z,\theta)=\frac{\partial}{\partial\theta}f(x,z,\theta).$

Therefore, the tangent space ${\mathcal{T}}$ at $\theta$ is

$\displaystyle{\mathcal{T}}=\Big{\\{}$ $\displaystyle t\cdot
s_{1}(y{\,|\,}x,z)+(1-t)\cdot
s_{0}(y{\,|\,}x,z)+a(x,z)\big{(}t-e(x,z)\big{)}+s(x,z)\colon$ (E.9)
$\displaystyle\int
s_{w}(y{\,|\,}x,z)f_{w}(y{\,|\,}x,z){\,\mathrm{d}}y=0,\forall
w\in\\{0,1\\},~{}\forall x,z,$ (E.10) $\displaystyle\qquad~{}\int
s(x,z)f(x,z){\,\mathrm{d}}x=0,~{}a(x,z)\text{ is square integrable}\Big{\\}}.$

#### Step 3: Pathwise derivative.

The target $\delta^{\prime}$ can be parametrized as

$\displaystyle\delta^{\prime}(\theta)=\frac{\int
yf_{1}(y{\,|\,}x,z,\theta)f(x,z,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z-\int
yf_{0}(y{\,|\,}x,z,\theta)f(x,z,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z}{\int
zf(x,z,\theta){\,\mathrm{d}}x{\,\mathrm{d}}z}.$

Therefore, the pathwise derivative at $\theta$ is

$\displaystyle\frac{\partial\delta^{\prime}(\theta)}{\partial\theta}$
$\displaystyle=\frac{1}{\mathbb{E}[Z]}\bigg{(}\int
y\dot{f}_{1}(y{\,|\,}x,z,\theta)f(x,z,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z-\int
y\dot{f}_{0}(y{\,|\,}x,z,\theta)f(x,z,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z\bigg{)}$
$\displaystyle\quad+\frac{1}{\mathbb{E}[Z]}\bigg{(}\int
y{f}_{1}(y{\,|\,}x,z,\theta)\dot{f}(x,z,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z-\int
y{f}_{0}(y{\,|\,}x,z,\theta)\dot{f}(x,z,\theta){\,\mathrm{d}}x{\,\mathrm{d}}y{\,\mathrm{d}}z\bigg{)}$
$\displaystyle\quad+\frac{\mathbb{E}[Y(1)]-\mathbb{E}[Y(0)]}{\mathbb{E}[Z]^{2}}\int
z\dot{f}(x,z,\theta){\,\mathrm{d}}x{\,\mathrm{d}}z$
$\displaystyle=\frac{1}{\mathbb{E}[Z]}\bigg{(}\mathbb{E}\bigg{[}Y(1)\frac{\dot{f}_{1}(Y(1){\,|\,}X,Z,\theta)}{{f}_{1}(Y(1){\,|\,}X,Z,\theta)}\bigg{]}-\mathbb{E}\bigg{[}Y(0)\frac{\dot{f}_{0}(Y(0){\,|\,}X,Z,\theta)}{{f}_{0}(Y(0){\,|\,}X,Z,\theta)}\bigg{]}\bigg{)}$
$\displaystyle\quad+\frac{1}{\mathbb{E}[Z]}\cdot\mathbb{E}\bigg{[}\big{(}Y(1)-Y(0)\big{)}\frac{\dot{f}(X,Z,\theta)}{{f}(X,Z,\theta)}\bigg{]}+\frac{\mathbb{E}[Y(1)]-\mathbb{E}[Y(0)]}{\mathbb{E}[Z]^{2}}\cdot\mathbb{E}\bigg{[}Z\cdot\frac{\dot{f}(X,Z,\theta)}{{f}(X,Z,\theta)}\bigg{]},$

where the second equality follows similar arguments as in the proof of Theorem
E.7.

#### Step 4: Projection onto tangent space.

We show that $\phi_{\delta^{\prime}}^{*}$ is in ${\mathcal{T}}$ and is the
projection of $\frac{\partial\delta^{\prime}(\theta)}{\partial\theta}$ onto
${\mathcal{T}}$. Firstly, note that
$\phi_{\delta^{\prime}}^{*}\in{\mathcal{T}}$ by setting $a(x,z)\equiv 0$ and

$\displaystyle s_{1}(y{\,|\,}x,z)=\frac{y-\mu_{1}(x,z)}{e(x,z)},\quad
s_{0}(y{\,|\,}x,z)=\frac{y-\mu_{0}(x,z)}{1-e(x,z)},$ $\displaystyle
s(x,z)=\mu_{1}(x,z)-\mu_{0}(x,z)-\mathbb{E}[Y(1)]+\mathbb{E}[Y(0)]$

in the specification of E.9. Then to show that $\phi_{\delta^{\prime}}^{*}$ is
the projection of $\frac{\partial\delta^{\prime}(\theta)}{\partial\theta}$
onto ${\mathcal{T}}$, it suffices to show that

$\displaystyle\frac{\partial\delta^{\prime}(\theta)}{\partial\theta}=\mathbb{E}_{\theta}\big{[}s(\theta;X_{i},Y_{i},Z_{i},T_{i})\cdot\phi_{\delta^{\prime}}^{*}(Y_{i},Z_{i},X_{i},T_{i})\big{]}$
(E.11)

for the score function in (E.8). To this end, we now show that

$\displaystyle\mathbb{E}\big{[}\big{(}Z_{i}-\mathbb{E}[Z_{i}]\big{)}\cdot
s(\theta;X_{i},Y_{i},Z_{i},T_{i})]=\mathbb{E}\bigg{[}Z\cdot\frac{\dot{f}(X,Z,\theta)}{{f}(X,Z,\theta)}\bigg{]},\quad\text{and}$
$\displaystyle\mathbb{E}\big{[}\Gamma_{i}^{**}\cdot
s(\theta;X_{i},Y_{i},Z_{i},T_{i})]=\mathbb{E}\bigg{[}Y(1)\frac{\dot{f}_{1}(Y(1){\,|\,}X,Z,\theta)}{{f}_{1}(Y(1){\,|\,}X,Z,\theta)}\bigg{]}-\mathbb{E}\bigg{[}Y(0)\frac{\dot{f}_{0}(Y(0){\,|\,}X,Z,\theta)}{{f}_{0}(Y(0){\,|\,}X,Z,\theta)}\bigg{]}$
$\displaystyle\qquad\qquad+\mathbb{E}\bigg{[}\big{(}Y(1)-Y(0)\big{)}\frac{\dot{f}(X,Z,\theta)}{{f}(X,Z,\theta)}\bigg{]}.$

For the first equation, note that
$\mathbb{E}[s(\theta;X_{i},Y_{i},Z_{i},T_{i})]=0$, hence

$\displaystyle\mathbb{E}\big{[}\big{(}Z_{i}-\mathbb{E}[Z_{i}]\big{)}\cdot
s(\theta;X_{i},Y_{i},Z_{i},T_{i})]=\mathbb{E}\big{[}Z_{i}\cdot
s(\theta;X_{i},Y_{i},Z_{i},T_{i})]$ $\displaystyle=\mathbb{E}\bigg{[}T\cdot
Z\cdot\frac{\dot{f}_{1}(Y(1){\,|\,}X,Z,\theta)}{f_{1}(Y(1){\,|\,}X,Z,\theta)}+(1-T)\cdot
Z\cdot\frac{\dot{f}_{0}(Y(0){\,|\,}X,Z,\theta)}{f_{0}(Y(0){\,|\,}X,Z,\theta)}\bigg{]}$
$\displaystyle\qquad+\mathbb{E}\bigg{[}Z\cdot\frac{T-e(X,Z,\theta)}{e(X,Z,\theta)(1-e(X,Z,\theta))}\cdot\dot{e}(X,Z,\theta)+Z\cdot\frac{\dot{f}(X,Z,\theta)}{f(X,Z,\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}e(X,Z)\cdot
Z\cdot\frac{\dot{f}_{1}(Y(1){\,|\,}X,Z,\theta)}{f_{1}(Y(1){\,|\,}X,Z,\theta)}+(1-e(X,Z))\cdot
Z\cdot\frac{\dot{f}_{0}(Y(0){\,|\,}X,Z,\theta)}{f_{0}(Y(0){\,|\,}X,Z,\theta)}\bigg{]}$
$\displaystyle\qquad+\mathbb{E}\bigg{[}Z\cdot\frac{\mathbb{E}[T{\,|\,}X,Z]-e(X,Z,\theta)}{e(X,Z,\theta)(1-e(X,Z,\theta))}\cdot\dot{e}(X,Z,\theta)+Z\cdot\frac{\dot{f}(X,Z,\theta)}{f(x,Z,\theta)}\bigg{]}=\mathbb{E}\bigg{[}Z\cdot\frac{\dot{f}(X,Z,\theta)}{f(X,Z,\theta)}\bigg{]},$

where the third equality follows from the conditional independence of $T$ and
$Y(1),Y(0)$ given $X,Z$ and the last equality follows the fact that

$\displaystyle\mathbb{E}\bigg{[}g(X,Z)\frac{\dot{f}_{w}(Y(w){\,|\,}X,Z,\theta)}{f_{w}(Y(w){\,|\,}X,Z,\theta)}\bigg{]}=0,\quad\forall~{}w\in\\{0,1\\}$
(E.12)

and any measurable function $g(\cdot,\cdot)$. For the second equality, note
that

$\displaystyle\mathbb{E}\bigg{[}\Gamma_{i}^{**}\cdot
T_{i}\cdot\frac{\dot{f}_{1}(Y_{i}(1){\,|\,}X_{i},Z_{i},\theta)}{f_{1}(Y_{i}(1){\,|\,}X_{i},Z_{i},\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\Big{(}\mu_{1}(X,Z)-\mu_{0}(X,Z)-\mathbb{E}[Y(1)]+\mathbb{E}[Y(0)]\Big{)}\cdot
T\cdot\frac{\dot{f}_{1}(Y(1){\,|\,}X,Z,\theta)}{f_{1}(Y(1){\,|\,}X,Z,\theta)}\bigg{]}$
$\displaystyle\qquad+\mathbb{E}\bigg{[}\frac{T}{e(X,Z)}\big{(}Y(1)-\mu_{1}(X,Z)\big{)}\cdot\frac{\dot{f}_{1}(Y(1){\,|\,}X,Z,\theta)}{f_{1}(Y(1){\,|\,}X,Z,\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\Big{(}\mu_{1}(X,Z)-\mu_{0}(X,Z)-\mathbb{E}[Y(1)]+\mathbb{E}[Y(0)]\Big{)}\cdot
e(X,Z)\cdot\frac{\dot{f}_{1}(Y(1){\,|\,}X,Z,\theta)}{f_{1}(Y(1){\,|\,}X,Z,\theta)}\bigg{]}$
$\displaystyle\qquad+\mathbb{E}\bigg{[}\big{(}Y(1)-\mu_{1}(X,Z)\big{)}\cdot\frac{\dot{f}_{1}(Y(1){\,|\,}X,Z,\theta)}{f_{1}(Y(1){\,|\,}X,Z,\theta)}\bigg{]}=\mathbb{E}\bigg{[}Y(1)\cdot\frac{\dot{f}_{1}(Y(1){\,|\,}X,Z,\theta)}{f_{1}(Y(1){\,|\,}X,Z,\theta)}\bigg{]}.$

Here the second equality follows from the conditional independence of $T$ and
$Y(1),Y(0)$ given $X,Z$ and the last equality follows from (E.12). Following
exactly the same arguments, we have

$\displaystyle\mathbb{E}\bigg{[}\Gamma_{i}^{**}\cdot(1-T_{i})\cdot\frac{\dot{f}_{0}(Y_{i}(0){\,|\,}X_{i},Z_{i},\theta)}{f_{0}(Y_{i}(0){\,|\,}X_{i},Z_{i},\theta)}\bigg{]}=\mathbb{E}\bigg{[}Y(0)\cdot\frac{\dot{f}_{0}(Y_{i}(0){\,|\,}X_{i},Z_{i},\theta)}{f_{0}(Y_{i}(0){\,|\,}X_{i},Z_{i},\theta)}\bigg{]}.$

Meanwhile, note that

$\displaystyle\mathbb{E}\bigg{[}\Gamma_{i}^{**}\cdot\frac{\dot{f}(X_{i},Z_{i},\theta)}{f(X_{i},Z_{i},\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\Big{(}\mu_{1}(X,Z)-\mu_{0}(X,Z)-\mathbb{E}[Y(1)]+\mathbb{E}[Y(0)]\Big{)}\cdot\frac{\dot{f}(X,Z,\theta)}{f(X,Z,\theta)}\bigg{]}$
$\displaystyle\quad+\mathbb{E}\bigg{[}\frac{T}{e(X,Z)}\cdot\big{(}Y(1)-\mu_{1}(X,Z)\big{)}\cdot\frac{\dot{f}(X,Z,\theta)}{f(X,Z,\theta)}\bigg{]}+\mathbb{E}\bigg{[}\frac{1-T}{1-e(X,Z)}\cdot\big{(}Y(0)-\mu_{0}(X,Z)\big{)}\cdot\frac{\dot{f}(X,Z,\theta)}{f(X,Z,\theta)}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\Big{(}\mu_{1}(X,Z)-\mu_{0}(X,Z)\Big{)}\cdot\frac{\dot{f}(X,Z,\theta)}{f(X,Z,\theta)}\bigg{]}=\mathbb{E}\bigg{[}\big{(}Y(1)-Y(0)\big{)}\cdot\frac{\dot{f}(X,Z,\theta)}{f(X,Z,\theta)}\bigg{]},$

where the second equality follows from the conditional independence and the
fact that
$\mathbb{E}\big{[}\frac{\dot{f}(X,Z,\theta)}{f(X,Z,\theta)}\big{]}=0$, and the
last equality follows from the takeout and tower property of conditional
expectations. Also,

$\displaystyle\mathbb{E}\bigg{[}\Gamma_{i}^{**}\cdot\frac{T_{i}-e(X_{i},Z_{i})}{e(X_{i},Z_{i})(1-e(X_{i},Z_{i}))}\bigg{]}$
$\displaystyle=\mathbb{E}\bigg{[}\Big{(}\mu_{1}(X,Z)-\mu_{0}(X,Z)-\mathbb{E}[Y(1)]+\mathbb{E}[Y(0)]\Big{)}\cdot\frac{T-e(X,Z)}{e(X,Z)(1-e(X,Z))}\bigg{]}$
$\displaystyle\quad+\mathbb{E}\bigg{[}\frac{T}{e(X,Z)}\cdot\big{(}Y(1)-\mu_{1}(X,Z)\big{)}\cdot\frac{T-e(X,Z)}{e(X,Z)(1-e(X,Z))}\bigg{]}$
$\displaystyle\quad+\mathbb{E}\bigg{[}\frac{1-T}{1-e(X,Z)}\cdot\big{(}Y(0)-\mu_{0}(X,Z)\big{)}\cdot\frac{T-e(X,Z)}{e(X,Z)(1-e(X,Z))}\bigg{]}=0,$

where all the three terms in the summation equals zero because of the tower
and takeout property of conditional expectations. Putting them together, we’ve
proved (E.11). Therefore, $\phi_{\delta^{\prime}}^{*}$ is the projection of
$\frac{\partial\delta^{\prime}(\theta)}{\partial\theta}$ onto ${\mathcal{T}}$.
Hence, $\phi_{\delta^{\prime}}^{*}$ is the efficient influence function, and
its variance is the semiparametric asymptotic variance bound for estimating
$\delta^{\prime}$. ∎

##  Appendix F Technical lemmas

######  Lemma F.1.

Let $\mathcal{F}_{n}$ be a sequence of $\sigma$-algebra, and let $A_{n}\geq 0$
be a sequence of nonnegative random variables. If
$\mathbb{E}[A_{n}{\,|\,}\mathcal{F}_{n}]=o_{P}(1)$, then $A_{n}=o_{P}(1)$.

###### Proof of Lemma F.1.

By Markov’s inequality, for any $\epsilon>0$, we have

$\displaystyle
B_{n}:=\mathbb{P}(A_{n}>\epsilon{\,|\,}\mathcal{F}_{n})\leq\frac{\mathbb{E}[A_{n}{\,|\,}\mathcal{F}_{n}]}{\epsilon}=o_{P}(1),$

and $B_{n}\in[0,1]$ are bounded random variables. For any subsequence
$\\{n_{k}\\}_{k\geq 1}$ of $\mathbb{N}$, since
$B_{n_{k}}\stackrel{{\scriptstyle P}}{{\to}}0$, there exists a subsequence
$\\{n_{k_{i}}\\}_{i\geq 1}\subset\\{n_{k}\\}_{k\geq 1}$ such that
$B_{n_{k_{i}}}\stackrel{{\scriptstyle\text{a.s.}}}{{\to}}0$ as $i\to\infty$.
By the dominated convergence theorem, we have $\mathbb{E}[B_{n_{k_{i}}}]\to
0$, or equivalently, $\mathbb{P}(A_{n_{k_{i}}}>\epsilon)\to 0.$ Therefore, for
any subsequence $\\{n_{k}\\}_{k\geq 1}$ of $\mathbb{N}$, there exists a
subsequence $\\{n_{k_{i}}\\}_{i\geq 1}\subset\\{n_{k}\\}_{k\geq 1}$ such that
$A_{n_{k_{i}}}\stackrel{{\scriptstyle P}}{{\to}}0$ as $i\to\infty$. By the
arbitrariness of $\\{n_{k}\\}_{k\geq 1}$, we know
$A_{n}\stackrel{{\scriptstyle P}}{{\to}}0$ as $n\to\infty$, which completes
the proof. ∎