Universal Lymph Node Detection in T2 MRI using Neural Networks]Universal Lymph Node Detection in T2 MRI using Neural Networks

[1]Tejas Sudharshan Mathai
1]Sungwon Lee
1]Thomas C. Shen
2]Zhiyong Lu
1]Ronald M. Summers

[1]Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, Clinical Center, NIH, Bethesda, MD, USA

[2]National Center for Biotechnology Information, National Library of Medicine, NIH, Bethesda, MD, USA

Purpose: Identification of abdominal Lymph Nodes (LN) that are suspicious for metastasis in T2 Magnetic Resonance Imaging (MRI) scans is critical for staging of lymphoproliferative diseases. Prior work on LN detection has been limited to specific anatomical regions of the body (pelvis, rectum) in single MR slices. Therefore, the development of a universal approach to detect LN in full T2 MRI volumes is highly desirable.

Methods: In this study, a Computer Aided Detection (CAD) pipeline to universally identify abdominal LN in volumetric T2 MRI using neural networks is proposed. First, we trained various neural network models for detecting LN: Faster RCNN with and without Hard Negative Example Mining (HNEM), FCOS, FoveaBox, VFNet, and Detection Transformer (DETR). Next, we show that the state-of-the-art (SOTA) VFNet model with Adaptive Training Sample Selection (ATSS) outperforms Faster RCNN with HNEM. Finally, we ensembled models that surpassed a 45% mAP threshold. We found that the VFNet model and one-stage model ensemble can be interchangeably used in the CAD pipeline.

Results: Experiments on 122 test T2 MRI volumes revealed that VFNet achieved a 51.1% mAP and 78.7% recall at 4 false positives (FP) per volume, while the one-stage model ensemble achieved a mAP of 52.3% and sensitivity of 78.7% at 4FP.

Conclusion: Our contribution is a CAD pipeline that detects LN in T2 MRI volumes, resulting in a sensitivity improvement of $\sim$14 points over the current SOTA method for LN detection (sensitivity of 78.7% at 4 FP vs. 64.6% at 5 FP per volume).

§ INTRODUCTION

The lymphatic system consists of lymph nodes that fight infection in the body by ridding it of foreign matter. In lympho-proliferative diseases, patients may present with enlarged and metastatic LN, which need to be identified and distinguished from non-metastatic LN. Furthermore, cancerous LN appearing at sites in the body that do not correlate with the first site of lymphatic spread indicate distant metastasis [1]. LN are visualized using multi-parametric MRI, e.g. T2-weighted MRI, and radiologists use the AJCC guidelines [1] to measure the size of a LN through its long and short axis diameters (LAD and SAD). Nodal size predominantly defines the cancer management and therapy; nodes with a SAD of $\geq$ 10mm are considered suspicious for cancer [2]. It is often difficult for a radiologist to identify and size LN due to their inconsistent shapes, varying appearances, and diverse anatomical locations. Typically, they utilize another MRI scan, such as Diffusion Weighted Imaging (DWI), to confirm their findings. Usually, multiple scanners with different exam protocols are used in a clinical setting to image patients. Despite the complimentary sources of information, radiologists can sometimes miss suspicious LN. Due to these challenging circumstances involving imaging and radiology workflow issues, an automated CAD pipeline is needed for identification of LN in T2 MRI scans such that LN sizing can be completed.

There has been a moderate amount of prior work [3, 4, 5, 6, 7, 8] on MRI-based LN detection, but they focus on specific anatomical regions. A Mask-RCNN network was used in [3] to identify and segment LN in pelvic MRI images with 62.6% recall at 8.2 FP per volume. In [4], a GentleBoost classifier followed by a convolutional neural network (CNN) was used to achieve a 85% LN detection sensitivity at 5-10 FP per image. In [5], Faster-RCNN identified pelvic LN with an AUC of 91.2%. Universal LN detection in the abdomen has only been researched recently with [6] describing a 64.6% sensitivity at 5 FP per volume, while [7] posted a 91.6% sensitivity at 4 FP per image on 2D slices only.

In this study, we design a CAD pipeline that detects LN in T2-weighted MRI volumes. We evaluate SOTA off-the-shelf detection models, such as VFNet [9], DETR transformer [10], FoveaBox [11], Faster RCNN [12] etc., on the LN detection task. Next, we describe the superiority of the VFNet model with Adaptive Training Sample Selection (ATSS) [9] over a Faster RCNN model trained with HNEM [13]. Finally, models that exceed a 45% mAP threshold are ensembled together, and we observe that the one-stage model ensemble and the VFNet model are similarly proficient at detecting LN in T2 MRI scans: a $\geq$51% mean average precision (mAP) and $\sim$78.7% sensitivity at 4FP per volume. We do not use DWI scans so that we can compare our results against prior work (although the addition of DWI would enable reliable LN detection). Despite this, an improvement in sensitivity by $\sim$14 points (78.7% recall at 4FP vs. 64.6% recall at 5P per volume) is achieved over the most recent SOTA LN detection method [6].

(a) F-RCNN

(e) FoveaBox

(b) F-RCNN + HN

(f) DETR

(c) VFNet

(g) Ens All

(d) FCOS

(h) Ens OS

LN detection results by different neural networks on a slice from a T2 MRI volume. Green boxes: ground truth, yellow: true positives, and red: false positives. (a) and (b) show the Faster RCNN model without and with HNEM respectively. Notice the reduced FP count in (b). In (c) and (d), LN detection results by VFNet with ATSS and FCOS is shown. More FP are detected by FCOS. (e) and (f) show the results from other SOTA detectors such as FoveaBox and DETR. Finally, (g) shows the ensemble detection result using all models (Faster RCNN + HNEM, VFNet, FoveaBox and DETR), while (h) shows the result of using only the one-stage (OS) model ensemble (VFNet, FoveaBox and DETR). Note the missed LN in (g) captured in (h).

§ METHODS

Object Detectors. The following were the object detection models that were used in this paper: 1) Faster R-CNN [12], 2) VFNet [9], 3) FCOS [14], 4) FoveaBox [11], and 5) DETR [10]. Faster RCNN is a two-stage anchor-based detector where the first stage generates region proposals of objects of interest, while the second stage classifies these proposals and regresses the object's bounding box coordinates. As Faster RCNN can generate numerous false positives (FP), Hard Negative Example Mining (HNEM) [13] is a strategy that can be employed to reduce the number and thereby boost precision and recall values. This method involves sampling negative bounding boxes, which do not overlap with the ground truth and have prediction probability scores that are larger than the true positives, allowing us to iteratively train the model to reduce the incidence of FP.

FCOS [14] is different from the Faster RCNN model; it extracts features from multi-levels of the Feature Pyramid Network (FPN) [15] using shared heads at different levels to regress the bounding box coordinates, and a centerness score to eliminate false predictions that are far away from the center of the target object location. FoveaBox [11] consists of a backbone and a fovea head network, the backbone being a simple FPN yielding feature maps at multi-levels. The fovea head has two branches: the classification branch performs per-pixel classification on the backbone features, and the box regression branch predicts the coordinates of the box at each location that may be potentially covered by an object.

VFNet combines FCOS (without the centerness branch) with the Adaptive Training Sample Selection (ATSS) module [9] to accurately select the high quality detections from the pool of candidate predictions. By replacing the classification score with the IoU between the ground truth and the prediction, an IoU-aware classification score (IACS) is obtained that is merged in a novel IoU-aware Varifocal loss, which down-weights the contribution of negative detections to the loss, but up-weights the contribution of positive samples. Inside this setup, a star-shaped bounding box provides context cues to reduce the misalignment between ground truth and the predicted boxes.

The DEtection Transformer (DETR) [10] uses a transformer encoder-decoder architecture, wherein the self-attention provided by the transformer removes duplicate predictions. Spatial position encoding is added to the feature maps obtained from the ResNet50 backbone inside the encoder-decoder architecture. Prediction feed-forward network heads decode the output embeddings of object queries independently into the bounding box coordinates and labels. A bipartite set matching loss is used to train the model, however in our implementation, we replaced the original cross entropy classification loss with the focal loss [16] to overcome the class imbalance problem between positive and negative samples, and weighted it with $\lambda_{c}=2$.

Weighted Boxes Fusion. At test time, utilizing multiple object detection models or multiple epochs of a single model can yield significantly higher mAP and sensitivity metrics at the cost of a large number of predictions. The predictions consist of the bounding box coordinates of an object and the classification probability. Many strategies, such as Non-Maximal Suppression (NMS) [17, 18], exist to reduce these duplicates, but we use Weighted Boxes Fusion [18] to reduce the number of FP during test time.

§ EXPERIMENTS AND RESULTS

Data. Between January 2015 and September 2019, patients who underwent imaging for a lymphoproliferative disease were identified. Initially, 584 T2-weighted abdominal MRI scans were downloaded from the National Institutes of Health (NIH) Picture Archiving and Communication System (PACS). Radiology reports for these studies were also downloaded, and natural language processing [19] was used to pull the nodal extent and size measurements in these studies. A radiologist performed a quality check on the collected data, and removed incorrect annotations and scans containing only one LN annotation (either LAD or SAD measures). This process yielded 421 T2 scans (n = 421 patients), which were divided into training ($\sim$60%, 254 scans), validation ($\sim$10%, 45 scans), and test ($\sim$30%, 122 scans) splits. The train and validation set consisted of only 2D key slices with 1-2 LN annotated with both the LAD and SAD, while the 3D extent of all LN in the test set were fully labeled. N4 bias normalization [20] was first applied to the scans, followed by normalization to [1%, 99%] of the voxel intensity range [21], such that the contrast between bright and dark structures in the slices was boosted. The resulting scans had dimensions in the range from (256 $\sim$ 640) $\times$ (192 $\sim$ 640) $\times$ (18 $\sim$ 60) voxels.

Implementation. As radiologists read MRI scans by scrolling through the slices, we replicated their workflow by using 3-slice T2 MRI images where the center slice contained the LN annotated by the radiologist. These 3-channel images were passed to each detection model implemented with the mmDetection framework [22]. Data augmentation was performed: random flips, crops, shifts and rotations in the range of [0, 32] pixels, and [0, 10] degrees etc. ResNet-50 was the backbone for all models used in this paper (pre-trained with MS COCO weights). A grid search was run across the batch size and learning rate parameters to obtain the optimal values; batch size was kept constant at 2 for all models, while the learning rate was 1e-3 for all models (for 12 epochs) except DETR whose learning rate was 1e-5 (for 150 epochs). Each model was executed 5 times, and the best checkpoint from each run with the lowest validation loss was chosen for testing. The ensemble comprised of the lowest validation loss checkpoint for each model across all runs. All experiments were run on a workstation running Ubuntu 18.04LTS with 4 NVIDIA Tesla V100 GPUs. Evaluation was always performed at an IoU threshold of 25% to be consistent with prior work.

Results. From prior work, a clinically acceptable result for LN detection is a sensitivity of 65% at 4-6 FP per volume [6, 3]. Higher detection sensitivity is required as missing any cancerous LN can be troublesome from a screening and triage perspective, however adequate precision is also necessary as too many FP can overwhelm the interpretation of a scan. Our quantitative results are presented in Table <ref> in terms of mean average precision (mAP) and sensitivity, while qualitative results are shown in Fig. <ref>. Our first experiment was to assess the performance of Faster RCNN with and without HNEM respectively. Faster RCNN with HNEM improved LN detection mAP and sensitivity by $\sim$2.2 (48.6% vs. 46.4%) and $\sim$3 points (76.9% vs. 74.1%) at 4FP respectively over Faster RCNN without HNEM. These results are in accordance with prior work on the utility of Faster RCNN with HNEM in reducing FP [13].

In our second experiment, we commissioned the VFNet model to detect LN. VFNet with the ATSS module adequately increased the detection mAP and sensitivity by $\sim$4.7 (51.1% vs. 46.4%) and $\sim$4.6 points (78.7% vs. 74.1%) at 4FP respectively over Faster RCNN without HNEM. There was a marginal improvement of $\sim$2.5 mAP and $\sim$2 recall by VFNet over Faster RCNN with HNEM. For our third experiment, only the FCOS model in VFNet was used without the ATSS module, and the detection performance drops significantly. The detection mAP dropped by $\sim$11 points (51.1% vs. 39.6%), while the sensitivity dropped by $\sim$7 points (78.7% vs. 72%). This experiment attests to the utility of the ATSS module within the VFNet model. Note however that these results already show a significant improvement over the current SOTA in LN detection [6]; the worst performing FCOS model posts a $\sim$7.4 increase in sensitivity at 4FP per volume.

Detection performance of various detectors and our proposed model ensemble. “S” stands for Sensitivity @[0.5, 1, 2, 4, 6, 8, 16] FP.  indicates unavailable metric values.
max width=

Method                                                              mAP<EMAIL_ADDRESS>       S@1           S@2           S@4           S@6           S@8           S@16

Faster RCNN (without HNEM) [12]                     46.4          42.3          54.3          65.1          74.1          77.8          78.1          78.1

Faster RCNN (with HNEM) [12]                        48.6          39.4          53.2          66.5          76.9          79.5          79.5          79.5

VFNet [9]                                        51.1          45.7          56.8          67.9          78.7          82.6          84.9          86.2

FCOS [14]                                           39.6          35.1          47.1          60.1          72            76.9          79.5          81.3

FoveaBox [11]                                   50.2          46            57.2          68            77.4          82.1          84.1          85.6

DETR [10]                                         45.5          39.2          51            61.3          74.8          79.8          81.8          82.5

Ensemble (All)                                                      49.7          44.6          58.4          69            78.5          82.3          83.8          84.1

Ensemble (One-Stage)                                                52.3          46.5          58            68.9          78.7          82.7          85.2          86.4

Wang 2022 [6] (3D)                                  –            –            –            –            64.6          –            –            –

Zhao 2020 [3] (3D)                             64.5          –            –            –            –            –             62.6          –

Mathai 2021 [7] (2D only)                        71.7          73.8          79.7          85.7          91.6          91.6          91.6          91.6

Mathai 2021 [8] (2D only)                        65.4          65.4          76.1          88.1          91.7          91.7          91.7          91.7

Debats 2019 [4] (2D only)                         –            –            –            –            –            –             80            –

The fourth experiment incorporated other one-stage or SOTA models for LN detection. In recent works [7, 8], FoveaBox and DETR have been used to detect LN in T2 MRI scans on a slice level, and the detection metrics for these models were calculated. FoveaBox provided results that were on par with those of VFNet with a $\sim$50.2% mAP and $\sim$77.4% recall. The DETR transformer model, however, clocked in a lower detection mAP of $\sim$45.5% and $\sim$74.8% sensitivity. For our final experiment, we took all the detection models that had a mAP over 45% and ensembled them (Faster RCNN with HNEM, VFNet, FoveaBox, and DETR) to be consistent with prior literature [7] in obtaining results for comparison. Initially, we ensembled all the models together (VFNet, FoveaBox, DETR, and Faster RCNN with HNEM) and noticed that the detection performance did not change significantly; the mAP dropped to $\sim$49.7% while the sensitivity was $\sim$78.5%. Then, we ensembled only the one-stage models (VFNet, FoveaBox, DETR) together, and observed the best mAP of $\sim$52.3% and sensitivity of $\sim$78.7% at 4FP per volume.

In prior work specifically focused on LN detection [4, 3, 7, 8, 6], the models were tested on 2D slices where the LN were annotated with the LAD and SAD, and the detection metrics were computed on only these 2D slices resulting in higher mAP and recall as shown in Table <ref>. However, this is problematic because it does not reflect the true detection performance, i.e., true positives that have not been marked by the radiologist in the current 2D slice or adjoining slices will be counted as FP when in fact they were detected correctly. Since a desirable automated method should process the whole T2 MRI scan, precision and recall should be associated with the FP per volume and not per slice. Therefore, in this work, these models are tested on 122 full 3D scans, where all the LN have been annotated in the test set, and the presented results are more representative of describing the LN existence. From past literature, only two approaches [3, 6] have attempted a full 3D volumetric detection of LN in T2 MRI. In [3], a precision of 64.5% was achieved with a combination of T2 and DWI MRI images, whereas we obtain 52.3% precision with T2 MRI only. Our one-stage ensemble and VFNet models improve upon the sensitivity of [3] by $\sim$16 points (78.7% sensitivity at 4FP vs. 62.6% sensitivity at 8FP) and that of [6] by $\sim$14 points (78.7% sensitivity at 4FP vs. 64.6% sensitivity at 5FP).

§ DISCUSSION

Among the various models and ensembles that were presented above, VFNet and the one-stage ensemble (consisting of VFNet, FoveaBox, DETR) met our pre-clinical usage goals; they achieved 78.7% sensitivity at 4 FP surpassing the required 65% sensitivity threshold at 4-6 FP [3, 6]. These models generated detection results that greatly improved upon prior 3D LN detection work as evidenced by Table <ref>. From our pipeline, Weighted Boxes Fusion was used as a post-processing step, and this is distinct from the original VFNet model that did not use any post-processing step. As we use multiple models (and checkpoints), we found WBF useful in merging the distinct predictions of various models (or epochs) with the mAP and sensitivity either being maintained or improved. In this paper, DWI is not utilized and its exclusion is a limitation of our work as radiologists in current clinical practice refer to these DWI scans to verify their diagnosis. Adding these DWI scans into the detection pipeline will provide a distinct feature representation in the model layers that can improve the detection results. Furthermore, not providing segmentation supervision in this pipeline is another limitation of our work as supervision from a segmentation branch has been shown to improve detection results [13]. These are avenues that can be explored in future work.

§ CONCLUSION

Detection of lymph nodes in T2 MRI scans aids radiologists in differentiating metastatic nodes from non-metastatic nodes and is useful for staging of lymphoproliferative diseases. In this paper, a CAD pipeline is proposed to identify LN in T2 MRI scans. The VFNet model and the one-stage ensemble provide the best results in terms of detection with $\geq$51% mAP and 78.7% sensitivity at 4FP per volume. These results significantly improve upon the state-of-the-art methods [3, 6] in lymph node detection with an increase in sensitivity of 16 and 14 points respectively. Our results indicate that the proposed CAD pipeline can be used for detection of lymph nodes in a pre-clinical setting.

§ ACKNOWLEDGMENTS

Funding: This work was supported by the Intramural Research Programs of the NIH National Library of Medicine and NIH Clinical Center (project number 1Z01 CL040004). We also thank Jaclyn Burge for the helpful comments and suggestions.

Ethical approval: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and the 1964 Helsinki declaration and its later amendments or comparable ethical standards. For this study, informed consent was not required.

Conflict of Interest: RMS receives royalties from iCAD, Philips, PingAn, ScanMed, and Translation Holdings. His lab received research support from PingAn. The authors have no additional conflicts of interest to declare.

[1]
Mahul B. Amin, Frederick L. Greene, Stephen B. Edge, Carolyn C. Compton,
Jeffrey E. Gershenwald, Robert K. Brookland, Laura Meyer, Donna M. Gress,
David R. Byrd, and David P. Winchester.
The eighth edition ajcc cancer staging manual: Continuing to build a
bridge from a population-based to a more “personalized” approach to
cancer staging.
CA: A Cancer Journal for Clinicians, 67(2):93–99, 2017.

[2]
Matthais Taupitz.
Imaging of lymph nodes – mri and ct.
Springer, pages 321–329, 2007.

[3]
Xingyu Zhao, Peiyi Xie, Mengmeng Wang, Perry J. Pickhardt, Wei Xia, Fei Xiong,
Rui Zhang, Yao Xie, and Junming Jian.
Deep learning based fully automated detection and segmentation of
lymph nodes on multiparametric mri for rectal cancer: A multicentre study.
eBioMedicine, 56, 2020.

[4]
Oscar A. Debats, Geert J.S. Litjens, and Henkjan J. Huisman.
Lymph node detection in mr lymphography: false positive reduction
using multi-view convolutional neural networks.
PeerJ, 7:e8052, November 2019.

[5]
Yun Lu, Qiyue Yu, Yuanxiang Gao, Yunpeng Zhou, Guangwei Liu, Qian Dong, Jinlong
Ma, Lei Ding, Hong wei Yao, Zhongtao Zhang, Gang Xiao, Qi An, Guiying Wang,
Jinchuan Xi, Wei-Tang Yuan, Yugui Lian, Dianliang Zhang, Chun-Gang Zhao, Qin
Yao, Wei Liu, Xiaoming Zhou, Shuhao Liu, Qingyao Wu, Wenjian Xu, Jianli
Zhang, Dong sheng Wang, Zhen qing Sun, Yuanxiang Gao, Xian xiang Zhang,
Ji lin Hu, Maoshen Zhang, Guanrong Wang, Xuefeng Zheng, Lei Wang, Jie Zhao,
and Shujian Yang.
Identification of metastatic lymph nodes in mr imaging with faster
region-based convolutional neural networks.
Cancer research, 78 17:5135–5143, 2018.

[6]
Shuai Wang, Yingying Zhu, Sungwon Lee, Daniel C. Elton, Thomas C. Shen, Youbao
Tang, Yifan Peng, Zhiyong Lu, and Ronald M. Summers.
Global-local attention network with multi-task uncertainty loss for
abnormal lymph node detection in mr images.
Medical Image Analysis, page 102345, 2022.

[7]
Tejas Sudharshan Mathai, Sungwon Lee, Daniel C. Elton, Thomas C. Shen, Yifan
Peng, Zhiyong Lu, and Ronald M. Summers.
Detection of lymph nodes in t2 mri using neural network ensembles.
In Chunfeng Lian, Xiaohuan Cao, Islem Rekik, Xuanang Xu, and Pingkun
Yan, editors, Machine Learning in Medical Imaging, pages 682–691,
Cham, 2021. Springer International Publishing.

[8]
Tejas Sudharshan Mathai, Sungwon Lee, Daniel C. Elton, Thomas C. Shen, Yifan
Peng, Zhiyong Lu, and Ronald M. Summers.
Lymph node detection in t2 mri with transformers.
arXiv, 2021.

[9]
Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sünderhauf.
Varifocalnet: An iou-aware dense object detector.
arXiv, 2021.

[10]
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander
Kirillov, and Sergey Zagoruyko.
End-to-end object detection with transformers.
In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm,
editors, Computer Vision – ECCV 2020, pages 213–229, Cham, 2020.
Springer International Publishing.

[11]
Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, Lei Li, and Jianbo Shi.
Foveabox: Beyound anchor-based object detection.
IEEE Transactions on Image Processing, 29:7389–7398, 2020.

[12]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster r-cnn: Towards real-time object detection with region proposal
In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett,
editors, Advances in Neural Information Processing Systems, volume 28.
Curran Associates, Inc., 2015.

[13]
You-bao Tang, Ke Yan, Yu-Xing Tang, Jiamin Liu, Jin Xiao, and Ronald M.
Uldor: A universal lesion detector for ct scans with pseudo masks and
hard negative example mining.
In 2019 IEEE 16th International Symposium on Biomedical Imaging
(ISBI 2019), pages 833–836, 2019.

[14]
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.
Fcos: Fully convolutional one-stage object detection.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV), October 2019.

[15]
Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and
Serge Belongie.
Feature pyramid networks for object detection.
In 2017 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pages 936–944, 2017.

[16]
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.
Focal loss for dense object detection.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
42(2):318–327, 2020.

[17]
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S. Davis.
Soft-nms – improving object detection with one line of code.
In Proceedings of the IEEE International Conference on Computer
Vision (ICCV), Oct 2017.

[18]
Roman Solovyev, Weimin Wang, and Tatiana Gabruseva.
Weighted boxes fusion: Ensembling boxes from different object
detection models.
Image and Vision Computing, 107:104117, Mar 2021.

[19]
Yifan Peng, Sungwon Lee, Daniel C. Elton, Thomas Shen, Yu-xing Tang, Qingyu
Chen, Shuai Wang, Yingying Zhu, Ronald Summers, and Zhiyong Lu.
Automatic recognition of abdominal lymph nodes from clinical text.
In Proceedings of the 3rd Clinical Natural Language Processing
Workshop, pages 101–110, Online, November 2020. Association for
Computational Linguistics.

[20]
Nicholas J. Tustison, Brian B. Avants, Philip A. Cook, Yuanjie Zheng, Alexander
Egan, Paul A. Yushkevich, and James C. Gee.
N4itk: Improved n3 bias correction.
IEEE Transactions on Medical Imaging, 29(6):1310–1320, 2010.

[21]
Marcin Kociołek, Michał Strzelecki, and Rafał Obuchowicz.
Does image normalization and intensity resolution impact texture
Computerized Medical Imaging and Graphics, 81:101716, 2020.

[22]
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li,
Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng,
Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu,
Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and
Dahua Lin.
Mmdetection: Open mmlab detection toolbox and benchmark, 2019.

[1]
Mahul B. Amin, Frederick L. Greene, Stephen B. Edge, Carolyn C. Compton,
Jeffrey E. Gershenwald, Robert K. Brookland, Laura Meyer, Donna M. Gress,
David R. Byrd, and David P. Winchester.
The eighth edition ajcc cancer staging manual: Continuing to build a
bridge from a population-based to a more “personalized” approach to
cancer staging.
CA: A Cancer Journal for Clinicians, 67(2):93–99, 2017.

[2]
Matthais Taupitz.
Imaging of lymph nodes – mri and ct.
Springer, pages 321–329, 2007.

[3]
Xingyu Zhao, Peiyi Xie, Mengmeng Wang, Perry J. Pickhardt, Wei Xia, Fei Xiong,
Rui Zhang, Yao Xie, and Junming Jian.
Deep learning based fully automated detection and segmentation of
lymph nodes on multiparametric mri for rectal cancer: A multicentre study.
eBioMedicine, 56, 2020.

[4]
Oscar A. Debats, Geert J.S. Litjens, and Henkjan J. Huisman.
Lymph node detection in mr lymphography: false positive reduction
using multi-view convolutional neural networks.
PeerJ, 7:e8052, November 2019.

[5]
Yun Lu, Qiyue Yu, Yuanxiang Gao, Yunpeng Zhou, Guangwei Liu, Qian Dong, Jinlong
Ma, Lei Ding, Hong wei Yao, Zhongtao Zhang, Gang Xiao, Qi An, Guiying Wang,
Jinchuan Xi, Wei-Tang Yuan, Yugui Lian, Dianliang Zhang, Chun-Gang Zhao, Qin
Yao, Wei Liu, Xiaoming Zhou, Shuhao Liu, Qingyao Wu, Wenjian Xu, Jianli
Zhang, Dong sheng Wang, Zhen qing Sun, Yuanxiang Gao, Xian xiang Zhang,
Ji lin Hu, Maoshen Zhang, Guanrong Wang, Xuefeng Zheng, Lei Wang, Jie Zhao,
and Shujian Yang.
Identification of metastatic lymph nodes in mr imaging with faster
region-based convolutional neural networks.
Cancer research, 78 17:5135–5143, 2018.

[6]
Shuai Wang, Yingying Zhu, Sungwon Lee, Daniel C. Elton, Thomas C. Shen, Youbao
Tang, Yifan Peng, Zhiyong Lu, and Ronald M. Summers.
Global-local attention network with multi-task uncertainty loss for
abnormal lymph node detection in mr images.
Medical Image Analysis, page 102345, 2022.

[7]
Tejas Sudharshan Mathai, Sungwon Lee, Daniel C. Elton, Thomas C. Shen, Yifan
Peng, Zhiyong Lu, and Ronald M. Summers.
Detection of lymph nodes in t2 mri using neural network ensembles.
In Chunfeng Lian, Xiaohuan Cao, Islem Rekik, Xuanang Xu, and Pingkun
Yan, editors, Machine Learning in Medical Imaging, pages 682–691,
Cham, 2021. Springer International Publishing.

[8]
Tejas Sudharshan Mathai, Sungwon Lee, Daniel C. Elton, Thomas C. Shen, Yifan
Peng, Zhiyong Lu, and Ronald M. Summers.
Lymph node detection in t2 mri with transformers.
arXiv, 2021.

[9]
Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sünderhauf.
Varifocalnet: An iou-aware dense object detector.
arXiv, 2021.

[10]
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander
Kirillov, and Sergey Zagoruyko.
End-to-end object detection with transformers.
In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm,
editors, Computer Vision – ECCV 2020, pages 213–229, Cham, 2020.
Springer International Publishing.

[11]
Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, Lei Li, and Jianbo Shi.
Foveabox: Beyound anchor-based object detection.
IEEE Transactions on Image Processing, 29:7389–7398, 2020.

[12]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster r-cnn: Towards real-time object detection with region proposal
In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett,
editors, Advances in Neural Information Processing Systems, volume 28.
Curran Associates, Inc., 2015.

[13]
You-bao Tang, Ke Yan, Yu-Xing Tang, Jiamin Liu, Jin Xiao, and Ronald M.
Uldor: A universal lesion detector for ct scans with pseudo masks and
hard negative example mining.
In 2019 IEEE 16th International Symposium on Biomedical Imaging
(ISBI 2019), pages 833–836, 2019.

[14]
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.
Fcos: Fully convolutional one-stage object detection.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV), October 2019.

[15]
Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and
Serge Belongie.
Feature pyramid networks for object detection.
In 2017 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pages 936–944, 2017.

[16]
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.
Focal loss for dense object detection.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
42(2):318–327, 2020.

[17]
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S. Davis.
Soft-nms – improving object detection with one line of code.
In Proceedings of the IEEE International Conference on Computer
Vision (ICCV), Oct 2017.

[18]
Roman Solovyev, Weimin Wang, and Tatiana Gabruseva.
Weighted boxes fusion: Ensembling boxes from different object
detection models.
Image and Vision Computing, 107:104117, Mar 2021.

[19]
Yifan Peng, Sungwon Lee, Daniel C. Elton, Thomas Shen, Yu-xing Tang, Qingyu
Chen, Shuai Wang, Yingying Zhu, Ronald Summers, and Zhiyong Lu.
Automatic recognition of abdominal lymph nodes from clinical text.
In Proceedings of the 3rd Clinical Natural Language Processing
Workshop, pages 101–110, Online, November 2020. Association for
Computational Linguistics.

[20]
Nicholas J. Tustison, Brian B. Avants, Philip A. Cook, Yuanjie Zheng, Alexander
Egan, Paul A. Yushkevich, and James C. Gee.
N4itk: Improved n3 bias correction.
IEEE Transactions on Medical Imaging, 29(6):1310–1320, 2010.

[21]
Marcin Kociołek, Michał Strzelecki, and Rafał Obuchowicz.
Does image normalization and intensity resolution impact texture
Computerized Medical Imaging and Graphics, 81:101716, 2020.

[22]
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li,
Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng,
Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu,
Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and
Dahua Lin.
Mmdetection: Open mmlab detection toolbox and benchmark, 2019.

[1]
Mahul B. Amin, Frederick L. Greene, Stephen B. Edge, Carolyn C. Compton,
Jeffrey E. Gershenwald, Robert K. Brookland, Laura Meyer, Donna M. Gress,
David R. Byrd, and David P. Winchester.
The eighth edition ajcc cancer staging manual: Continuing to build a
bridge from a population-based to a more “personalized” approach to
cancer staging.
CA: A Cancer Journal for Clinicians, 67(2):93–99, 2017.

[2]
Matthais Taupitz.
Imaging of lymph nodes – mri and ct.
Springer, pages 321–329, 2007.

[3]
Xingyu Zhao, Peiyi Xie, Mengmeng Wang, Perry J. Pickhardt, Wei Xia, Fei Xiong,
Rui Zhang, Yao Xie, and Junming Jian.
Deep learning based fully automated detection and segmentation of
lymph nodes on multiparametric mri for rectal cancer: A multicentre study.
eBioMedicine, 56, 2020.

[4]
Oscar A. Debats, Geert J.S. Litjens, and Henkjan J. Huisman.
Lymph node detection in mr lymphography: false positive reduction
using multi-view convolutional neural networks.
PeerJ, 7:e8052, November 2019.

[5]
Yun Lu, Qiyue Yu, Yuanxiang Gao, Yunpeng Zhou, Guangwei Liu, Qian Dong, Jinlong
Ma, Lei Ding, Hong wei Yao, Zhongtao Zhang, Gang Xiao, Qi An, Guiying Wang,
Jinchuan Xi, Wei-Tang Yuan, Yugui Lian, Dianliang Zhang, Chun-Gang Zhao, Qin
Yao, Wei Liu, Xiaoming Zhou, Shuhao Liu, Qingyao Wu, Wenjian Xu, Jianli
Zhang, Dong sheng Wang, Zhen qing Sun, Yuanxiang Gao, Xian xiang Zhang,
Ji lin Hu, Maoshen Zhang, Guanrong Wang, Xuefeng Zheng, Lei Wang, Jie Zhao,
and Shujian Yang.
Identification of metastatic lymph nodes in mr imaging with faster
region-based convolutional neural networks.
Cancer research, 78 17:5135–5143, 2018.

[6]
Shuai Wang, Yingying Zhu, Sungwon Lee, Daniel C. Elton, Thomas C. Shen, Youbao
Tang, Yifan Peng, Zhiyong Lu, and Ronald M. Summers.
Global-local attention network with multi-task uncertainty loss for
abnormal lymph node detection in mr images.
Medical Image Analysis, page 102345, 2022.

[7]
Tejas Sudharshan Mathai, Sungwon Lee, Daniel C. Elton, Thomas C. Shen, Yifan
Peng, Zhiyong Lu, and Ronald M. Summers.
Detection of lymph nodes in t2 mri using neural network ensembles.
In Chunfeng Lian, Xiaohuan Cao, Islem Rekik, Xuanang Xu, and Pingkun
Yan, editors, Machine Learning in Medical Imaging, pages 682–691,
Cham, 2021. Springer International Publishing.

[8]
Tejas Sudharshan Mathai, Sungwon Lee, Daniel C. Elton, Thomas C. Shen, Yifan
Peng, Zhiyong Lu, and Ronald M. Summers.
Lymph node detection in t2 mri with transformers.
arXiv, 2021.

[9]
Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sünderhauf.
Varifocalnet: An iou-aware dense object detector.
arXiv, 2021.

[10]
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander
Kirillov, and Sergey Zagoruyko.
End-to-end object detection with transformers.
In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm,
editors, Computer Vision – ECCV 2020, pages 213–229, Cham, 2020.
Springer International Publishing.

[11]
Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, Lei Li, and Jianbo Shi.
Foveabox: Beyound anchor-based object detection.
IEEE Transactions on Image Processing, 29:7389–7398, 2020.

[12]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster r-cnn: Towards real-time object detection with region proposal
In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett,
editors, Advances in Neural Information Processing Systems, volume 28.
Curran Associates, Inc., 2015.

[13]
You-bao Tang, Ke Yan, Yu-Xing Tang, Jiamin Liu, Jin Xiao, and Ronald M.
Uldor: A universal lesion detector for ct scans with pseudo masks and
hard negative example mining.
In 2019 IEEE 16th International Symposium on Biomedical Imaging
(ISBI 2019), pages 833–836, 2019.

[14]
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.
Fcos: Fully convolutional one-stage object detection.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV), October 2019.

[15]
Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and
Serge Belongie.
Feature pyramid networks for object detection.
In 2017 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pages 936–944, 2017.

[16]
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.
Focal loss for dense object detection.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
42(2):318–327, 2020.

[17]
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S. Davis.
Soft-nms – improving object detection with one line of code.
In Proceedings of the IEEE International Conference on Computer
Vision (ICCV), Oct 2017.

[18]
Roman Solovyev, Weimin Wang, and Tatiana Gabruseva.
Weighted boxes fusion: Ensembling boxes from different object
detection models.
Image and Vision Computing, 107:104117, Mar 2021.

[19]
Yifan Peng, Sungwon Lee, Daniel C. Elton, Thomas Shen, Yu-xing Tang, Qingyu
Chen, Shuai Wang, Yingying Zhu, Ronald Summers, and Zhiyong Lu.
Automatic recognition of abdominal lymph nodes from clinical text.
In Proceedings of the 3rd Clinical Natural Language Processing
Workshop, pages 101–110, Online, November 2020. Association for
Computational Linguistics.

[20]
Nicholas J. Tustison, Brian B. Avants, Philip A. Cook, Yuanjie Zheng, Alexander
Egan, Paul A. Yushkevich, and James C. Gee.
N4itk: Improved n3 bias correction.
IEEE Transactions on Medical Imaging, 29(6):1310–1320, 2010.

[21]
Marcin Kociołek, Michał Strzelecki, and Rafał Obuchowicz.
Does image normalization and intensity resolution impact texture
Computerized Medical Imaging and Graphics, 81:101716, 2020.

[22]
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li,
Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng,
Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu,
Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and
Dahua Lin.
Mmdetection: Open mmlab detection toolbox and benchmark, 2019.


$\mathrm{HR}(c)=\mathbb{P}(m(\boldsymbol{X})>c|Y=1),\qquad\mathrm{FAR}(c)=\mathbb{P}(m(\boldsymbol{X})>c|Y=0).$

The ROC curve is then formally defined as the graph

$\\{(\mathrm{FAR}(c),\mathrm{HR}(c))\colon c\in[0,1]\\}\subset[0,1]^{2}.$

This definition on the population level has a clear counterpart on the sample
level.

####  6.2.3 Imbalanced classes

There is a large strand of literature about the so-called imbalanced learning
problem [37]. For binary classification, this means that one class
significantly outnumbers the other one. Typically, main interest is in the
rare class $Y=1$, called minority class, with
$\mathbb{P}(Y=1)\ll\mathbb{P}(Y=0)$, e.g., with a ratio of 1:10, 1:1000 or
even more unbalanced.434343An example are Covid-19 cases. Main interest is in
the minority group of infected persons, luckily outnumbering the group of not
infected people by a large margin.

Poor models in this setting are often a result of—any combination of—fitting
deterministic classifiers, using re-sampling or re-weighting methods to
balance class frequencies in the training data and evaluating the model with a
score such as accuracy. As mentioned above, accuracy is a poor score for model
comparison, especially so for imbalanced classes where the cost-ratio $c$, if
known, is likely far away from $1/2$. The frequently encountered practice of
over-sampling the minority class or under-sampling the majority class might
help to get better (accuracy) scores, but it does not help with the underlying
problem: If the minority class is very rare in absolute terms, no matter the
size of the training set, one has effectively a small sample problem. No re-
sampling technique will magically generate more information out of the few
cases with the rare class.444444Let us play coin toss with a probability $p$
for $Y=1=\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pthead\textquotedblright\ltxml@oqmark@close{}}$. Every toss generates a
data point adding the Shannon entropy $H=-p\log_{2}p-(1-p)\log_{2}(1-p)$. $H$
is maximised for $p=1/2$ giving $H=1$ bit of information and minimised by
$p\in\\{0,1\\}$ giving $H=0$ bits. Data points for imbalanced classes
therefore have little information content.  Finally, we generally recommend
probabilistic classifiers based on consistent scoring rules over deterministic
classifiers. For instance, a well specified logistic regression model with
logit link and intercept fulfils (9) and therefore, at least on the training
set, is guaranteed to predict the class probabilities according to their
observed frequencies. We refer to [46] for more information about logistic
regression with rare events, e.g., data collection, finite sample corrections
and correcting estimates.

[nonumber=false, tocentry=Example, reference=Example]Example: Binary
Classification of Customer Churn

As an example for binary classification, we take the Telco Customer Churn data
set454545https://www.openml.org/d/42178 . Each of the $n=7043$ rows represents
a customer of a telecommunication company and the response variable
$Y=(\texttt{Churn}=\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002ptYes\textquotedblright\ltxml@oqmark@close{}})$ indicates if the
customer left within the last month. For the sake of brevity, we refrain from
repeating the detailed steps of the regression example and instead focus on
the new aspects of this classification task.

Listing 3 shows the short preprocessing and the features used for modelling.

Listing 3: Common data preprocessing for classification example

⬇

library(tidyverse)

library(OpenML)

df_origin <- getOMLDataSet(data.id = 42178L)

df <- tibble(df_origin$data)

df[df == "No␣internet␣service" | df == "No␣phone␣service"] <- "No"

df <- df %>%

mutate(

LogTotalCharges = log(as.numeric(TotalCharges)),

Churn = (Churn == "Yes") + 0,

) %>%

replace_na(list(LogTotalCharges = median(.$LogTotalCharges, na.rm = TRUE)))
%>%

mutate_if(is.character, as.factor)

y_var <- "Churn"

x_continuous <- c("tenure", "MonthlyCharges", "LogTotalCharges")

x_discrete <- setdiff(colnames(select_if(df, is.factor)),

c("customerID", y_var, "TotalCharges"))

x_vars <- c(x_continuous, x_discrete)

For a short exploratory data analysis, we refer to Subsection A.2.

We use a stratified train–test split:

⬇

library(splitTools)

set.seed(34621L)

inds <- partition(df$Churn, p = c(train = 0.75, test = 0.25))

train <- df[inds$train, ]

test <- df[inds$test, ]

y_train <- train[[y_var]]

y_test <- test[[y_var]]

This guarantees the same average churn frequency on both sets. We obtain
$\bar{y}=$0.265$$ on the training and $\bar{y}=$0.266$$ on the test set. This
helps to better compare results across both sets.464646If those numbers
deviated too much, the i.i.d. assumption would clearly be violated.

We train four different models:

  * •

Trivial model with $m_{\text{trivial}}(\boldsymbol{x})=0.2652907$;

  * •

Logistic regression (LogReg), i.e. a GLM with binomial family and logistic
link function;

⬇

form <- reformulate(x_vars, y_var)

fit_glm <- glm(form, data = train, family = binomial())

  * •

Random forest (RF) with R package ranger;

⬇

fit_rf <- ranger(

form,

data = train,

probability = TRUE,

seed = 774,

min.node.size = 30,

oob.error = TRUE

)

  * •

XGBoost model with log loss as objective function, logit link function and
optimised hyperparameters via cross-validation similar to the regression case.

The unconditional calibration with $V(z,y)=z-y$ is summarised in Table 10.

| $\overline{V}(m;D)$
---|---
model | $D=\text{train}$ | $D=\text{test}$
Trivial | $0$ | $-0.000\,317$
LogReg | $0$ | $-0.005\,33$
RF | $0.001\,17$ | $-0.003\,29$
XGBoost | $0.000\,483$ | $-0.005\,23$
Table 10: Assessment of classifiers’ unconditional calibration in terms of
bias.

The GLM with canonical link, i.e. the logistic regression, once more shows the
balance property on the training set. On the test set, the trivial model has
quite a good calibration and, among the feature using models, the random
forest is the best unconditionally calibrated one.

As graphical tool to check auto-calibration, we show reliability diagrams on
the test set with the CORP approach of [17] in Figure 10.

Figure 10: Reliability diagrams. The $x$-axis shows predictions/forecast
values, the $y$-axis conditional event probabilities (CEP). At the bottom of
each plot, the histogram of forecast values is drawn. The values in the boxes
are the decomposition of the log loss (23) into miscalibration (MCB),
discrimination (DSC) and uncertainty (UNC), such that
$\overline{S}_{\log}=\mathrm{MCB}-\mathrm{DSC}+\mathrm{UNC}$; see also (17).

The CORP approach essentially performs the binning in an optimal way such that
the conditional event probabilities (CEP), which are binned observed
frequencies, are isotonic. The random forest seems to be the best auto-
calibrated one. This is confirmed by the score decomposition of the log loss
(23) according to the auto-calibration variant of (17) given in Figure 10. The
lowest miscalibration (MCB) is obtained by the random forest. On the other
hand, the logistic regression has the highest discriminative power (DSC).

Finally, we assess the predictive performance with the log loss and the
relative reduction in log loss w.r.t. the trivial model, $\mathrm{R}^{\star}$,
see Table 11 and Figure 11.

| mean log loss | $\mathrm{R}^{\star}$ |  | AUC
---|---|---|---|---
Model | train | test | train | test |  | train | test
Trivial | $0.579$ | $0.579$ | $0$ | $0$ |  | $0.500$ | $0.500$
LogReg | $0.415$ | $0.382$ | $0.282$ | $0.340$ |  | $0.845$ | $0.875$
RF | $0.318$ | $0.390$ | $0.450$ | $0.325$ |  | $0.935$ | $0.868$
XGB | $0.401$ | $0.390$ | $0.306$ | $0.327$ |  | $0.859$ | $0.871$
Table 11: Performance on training and validation data in terms of mean log
loss (smaller is better), its relative reduction $\mathrm{R}^{\star}$ and AUC
(for both larger is better). Figure 11: Performance on training and validation
data in terms of mean log loss (smaller is better), its relative reduction
$\mathrm{R}^{\star}$ (larger is better).

The random forest clearly shows overfitting as it has the lowest mean log loss
on the training set but also the second highest one behind the trivial model
on the test set. This is not surprising as random forests are typically built
of very deep trees. The XGBoost model performs a touch better than the random
forest on the test set. Note, however, that this time no correction factor was
required. Despite the fact that we did not put much effort in the logistic
regression, e.g., we did not include any interaction terms nor splines or
polynomials of degree two or higher, it ranks on top of all other models
regarding predictive performance. A possible explanation for this ranking is
the modest size of the data set.

We also show the AUC. As the models show different calibration, their ranking
in terms of AUC could be very different from the ranking based on log loss.
Coincidentally, AUC shows the same out-of-sample ranking as log loss. This can
be explained by the score decomposition of the log loss in Figure 10. The
discrimination components DSC are an order of magnitude larger than the
miscalibration components MCB and give the same ranking as according to the
log loss. Despite the appeal of AUC, we generally prefer to evaluate the
models with a strictly consistent scoring function, i.e. log loss in our case.

We conclude this example with the Murphy diagram in Figure 12.

Figure 12: Murphy diagram with elementary scores for $\theta$ from $0$ to $1$.
The black vertical line indicates the average response on the training data
set.

We observe the small values of the random forest on the training set for a
large range of scores. From the right-hand side of the figure, we are visually
inclined to award the logistic regression with the best out-of-sample
performance. In fact, it has the best test Brier score as can be inferred from
the score decomposition given in Figure 10. However, the differences out-of-
sample seem to be relatively small such that we refrain from a definite answer
as to what model exhibits the best predictive accuracy measured by a wide
range of consistent scoring functions.

##  7 Conclusion

In this user guide, we have elaborated on best statistical practice in model
assessment and model comparison, important tasks in Machine Learning and
actuarial science. We have argued that for both tasks, specifying the
modelling target in the form of a statistical functional is crucial. The tools
for calibration assessment and model comparison—identification functions and
scoring functions—need to be chosen in line with the modelling target at hand.
Calibration assessment via strict identification functions checks whether
there are any systematic errors in the predictive model. On the other hand,
strictly consistent scoring functions simultaneously encourage good
calibration and discrimination ability, honouring the information content of a
model. The methodology has been illustrated with running examples in
regression and in binary classification, based on two (almost) real world data
sets. Making a clear distinction between learning (model building) and model
evaluation, we have stressed the importance of thorough train–test split of
the data.

Clearly, it was not possible to elaborate on all practically relevant aspects
in this field. For instance, we have entirely omitted the situation of missing
or incomplete data. Examples from insurance are IBNR (incurred but not yet
reported) for frequency models or RBNS (reported but not settled) for claims
severity. Moreover, we have mainly focused on cross-sectional data, only
providing some comments on data with serial dependence.

While the majority of the used guide at hand is devoted to a definition of
what predictive performance is and how it can be assessed, we would like to
close it by remarking that there are sometimes other important (and sometimes
conflicting) goals in modelling such as calibration, explainability [49],
impartiality [44], or robustness of predictions [51].

## Acknowledgements

The authors are very grateful to Jürg Schelldorfer and Mario Wüthrich as well
as to Timo Dimitriadis, Daniel Mayer and Marc-Oliver Pohle for their
comprehensive reviews and their innumerable inputs which led to substantial
improvements of this work. Christian Lorentzen and Michael Mayer like to thank
their company la Mobilière for its support.

## References

  * [1] S. Arlot and A. Celisse  “A survey of cross-validation procedures for model selection”  In _Statistics Surveys_ 4 Amer. Statist. Assoc., the Bernoulli Soc., the Inst. Math. Statist.,the Statist. Soc. Canada, 2010, pp. 40 –79  DOI: 10.1214/09-ss054
  * [2] P. L. Bartlett, P. M. Long, G. Lugosi and A. Tsigler  “Benign overfitting in linear regression”  In _Proceedings of the National Academy of Sciences_ 117.48 National Academy of Sciences, 2020, pp. 30063–30070  DOI: 10.1073/pnas.1907378117
  * [3] P. L. Bartlett and S. Mendelson  “Rademacher and Gaussian Complexities: Risk Bounds and Structural Results”  In _Computational Learning Theory_ Berlin, Heidelberg: Springer Berlin Heidelberg, 2001, pp. 224–240  DOI: 10.1007/3-540-44581-1_15
  * [4] S. Bates, T. Hastie and R. Tibshirani  “Cross-validation: what does it estimate and how well does it do it?”, 2021  arXiv:2104.00673 [stat.ME]
  * [5] M. Belkin, D. Hsu, S. Ma and S. Mandal  “Reconciling modern machine-learning practice and the classical bias–variance trade-off”  In _Proceedings of the National Academy of Sciences_ 116.32 National Academy of Sciences, 2019, pp. 15849–15854  DOI: 10.1073/pnas.1903070116
  * [6] M. Belkin, D. Hsu and J. Xu  “Two Models of Double Descent for Weak Features”  In _SIAM Journal on Mathematics of Data Science_ 2.4 Society for Industrial & Applied Mathematics (SIAM), 2020, pp. 1167–1180  DOI: 10.1137/20m1336072
  * [7] R. R. Bouckaert and E. Frank  “Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms”  In _Advances in Knowledge Discovery and Data Mining_ Springer Berlin Heidelberg, 2004, pp. 3–12  DOI: 10.1007/978-3-540-24775-3_3
  * [8] J. R. Brehmer and T. Gneiting  “Scoring interval forecasts: Equal-tailed, shortest, and modal interval”  In _Bernoulli_ 27.3, 2021, pp. 1993–2010  DOI: 10.3150/20-BEJ1298
  * [9] A. Buja, W. Stuetzle and Y. Shen  “Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications”, 2005  URL: http://www-stat.wharton.upenn.edu/~buja/PAPERS/paper-proper-scoring.pdf
  * [10] S. Byrne  “A note on the use of empirical AUC for evaluating probabilistic forecasts”  In _Electronic Journal of Statistics_ 10.1 The Institute of Mathematical Statisticsthe Bernoulli Society, 2016, pp. 380–393  DOI: 10.1214/16-EJS1109
  * [11] T. Chen and C. Guestrin  “XGBoost: A Scalable Tree Boosting System”  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16  San Francisco, California, USA: Association for Computing Machinery, 2016, pp. 785–794  DOI: 10.1145/2939672.2939785
  * [12] Y. Dar, V. Muthukumar and R. G. Baraniuk  “A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning”, 2021  arXiv:2109.02355 [stat.ML]
  * [13] M. H. A. Davis  “Verification of internal risk measure estimates”  In _Statistics & Risk Modeling_ 33.3–4, 2016, pp. 67–93  DOI: 10.1515/strm-2015-0007
  * [14] A. P. Dawid and V. G. Vovk  “Prequential probability: principles and properties”  In _Bernoulli_ 5.1, 1999, pp. 125–162  DOI: bj/1173707098
  * [15] F. X. Diebold and R. S. Mariano  “Comparing Predictive Accuracy”  In _Journal of Business & Economic Statistics_ 13.1 American Statistical Association, 1995, pp. 253–265  DOI: 10.2307/1392185
  * [16] T. Dimitriadis, T. Fissler and J. F. Ziegel  “The Efficiency Gap”  In _arXiv_ , 2020  arXiv:2010.14146 [math.ST]
  * [17] T. Dimitriadis, T. Gneiting and A. I. Jordan  “Stable reliability diagrams for probabilistic classifiers”  In _Proceedings of the National Academy of Sciences_ 118.8, 2021, pp. e2016191118  DOI: 10.1073/pnas.2016191118
  * [18] W. Ehm, T. Gneiting, A. I. Jordan and F. Krüger  “Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings”  In _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ 78.3, 2016, pp. 505–562  DOI: 10.1111/rssb.12154
  * [19] C. Elkan  “The Foundations of Cost-Sensitive Learning”  In _Proceedings of the 17th International Joint Conference on Artificial Intelligence_ , IJCAI’01  San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2001, pp. 973–978  URL: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.29.514&rep=rep1&type=pdf
  * [20] T. Fawcett  “An introduction to ROC analysis”  In _ROC Analysis in Pattern Recognition_ 27.8, 2006, pp. 861–874  DOI: 10.1016/j.patrec.2005.10.010
  * [21] T. Fissler, F. Krüger and M.-O. Pohle  “Regression Diagnostics via Generalized Residuals” Mimeo, 2022
  * [22] T. Fissler, M. Merz and M. V. Wüthrich  “Deep Quantile and Deep Composite Model Regression”  In _arXiv_ , 2021  arXiv:2112.03075 [stat.ME]
  * [23] T. Fissler, R.Frongillo, J. Hlavinová and B. Rudloff  “Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals”  In _Electronic Journal of Statistics_ 15.1, 2021, pp. 1034–1084  DOI: 10.1214/21-EJS1808
  * [24] T. Fissler and J. F. Ziegel  “Higher order elicitability and Osband’s principle”  In _The Annals of Statistics_ 44.4, 2016, pp. 1680–1707  DOI: 10.1214/16-AOS1439
  * [25] T. Fissler and J. F. Ziegel  “Order-sensitivity and equivariance of scoring functions”  In _Electronic Journal of Statistics_ 13.1 Institute of Mathematical StatisticsBernoulli Society, 2019, pp. 1166–1211  DOI: 10.1214/19-EJS1552
  * [26] S. Geman, E. Bienenstock and R. Doursat  “Neural Networks and the Bias/Variance Dilemma”  In _Neural Computation_ 4.1, 1992, pp. 1–58  DOI: 10.1162/neco.1992.4.1.1
  * [27] T. Gneiting  “Making and Evaluating Point Forecasts”  In _Journal of the American Statistical Association_ 106.494, 2011, pp. 746–762  DOI: 10.1198/jasa.2011.r10138
  * [28] T. Gneiting  “When is the mode functional the Bayes classifier?”  In _Stat_ 6.1, 2017, pp. 204–206  DOI: 10.1002/sta4.148
  * [29] T. Gneiting and M. Katzfuss  “Probabilistic Forecasting”  In _Annual Review of Statistics and Its Application_ 1.1 Annual Reviews, 2014, pp. 125–151  DOI: 10.1146/annurev-statistics-062713-085831
  * [30] T. Gneiting and A. E. Raftery  “Strictly Proper Scoring Rules, Prediction, and Estimation”  In _Journal of the American Statistical Association_ 102, 2007, pp. 359–378  DOI: 10.1198/016214506000001437
  * [31] T. Gneiting and J. Resin  “Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination”  In _arXiv_ , 2021  arXiv:2108.03210 [stat.ME]
  * [32] T. Gneiting and P. Vogel  “Receiver operating characteristic (ROC) curves: equivalences, beta model, and minimum distance estimation”  In _arXiv_ , 2021  DOI: 10.1007/s10994-021-06115-2
  * [33] C. W. J. Granger and M. Hashem Pesaran  “Economic and Statistical Measures of Forecast Accuracy”  In _Journal of Forecasting_ 19, 1999, pp. 537–560  DOI: 10.1002/1099-131X(200012)19:7<537::AID-FOR769>3.0.CO;2-G
  * [34] F. E. Harrell  “Regression Modeling Strategies”, Springer Series in Statistics  Springer International Publishing Switzerland 2015, 2015  DOI: 10.1007/978-3-319-19425-7
  * [35] T. Hastie, R. Tibshirani and J. Friedman  “The Elements of Statistical Learning”, Springer Series in Statistics  New York, NY, USA: Springer New York Inc., 2001  URL: https://web.stanford.edu/~hastie/ElemStatLearn/
  * [36] Trevor Hastie, Andrea Montanari, Saharon Rosset and Ryan J. Tibshirani  “Surprises in High-Dimensional Ridgeless Least Squares Interpolation”, 2020  arXiv:1903.08560 [math.ST]
  * [37] “Imbalanced Learning: Foundations, Algorithms, and Applications”  John Wiley & Sons, Inc., 2013  DOI: 10.1002/9781118646106
  * [38] C. Heinrich-Mertsching and T. Fissler  “Is the mode elicitable relative to unimodal distributions?”  In _Biometrika (forthcoming)_ , 2021  DOI: 10.1093/biomet/asab065
  * [39] A. Henzi and J. F. Ziegel  “Valid sequential inference on probability forecast performance”  In _arXiv_ , 2021  arXiv:2103.08402 [stat.ME]
  * [40] H. Holzmann and M. Eulert  “The role of the information set for forecasting – with applications to risk management”  In _The Annals of Applied Statistics_ 8, 2014, pp. 79–83  DOI: 10.1214/13-AOAS709
  * [41] P. J. Huber  “Robust Estimation of a Location Parameter”  In _The Annals of Mathematical Statistics_ 35.1, 1964, pp. 73–101  DOI: 10.1214/aoms/1177703732
  * [42] P. J. Huber and E. M. Ronchetti  “Robust Statistics”  Hoboken, New Jersey: John Wiley & Sons, Inc., 2009
  * [43] R. J. Hyndman and G. Athanasopoulos  “Forecasting: principles and practice”  OTexts: Melbourne, Australia, 2021  URL: https://otexts.com/fpp3
  * [44] K. D. Johnson, D. P. Foster and R. A. Stine  “Impartial Predictive Modeling and the Use of Proxy Variables”, 2016  arXiv:1608.00528 [stat.ME]
  * [45] B. Jørgensen  “The Theory of Dispersion Models”  London: ChapmanHall/CRC, 1997
  * [46] G. King and L. Zeng  “Logistic Regression in Rare Events Data”  In _Political Analysis_ 9.2 Cambridge University Press, 2001, pp. 137–163  DOI: 10.1093/oxfordjournals.pan.a004868
  * [47] D.P. Kroese, Z.I. Botev, T. Taimre and R. Vaisman  “Data Science and Machine Learning: Mathematical and Statistical Methods”, Chapman & Hall/CRC machine learning & pattern recognition  Boca Raton: CRC Press, 2019  URL: https://people.smp.uq.edu.au/DirkKroese/DSML/
  * [48] Sebastian Lerch, Thordis L. Thorarinsdottir, Francesco Ravazzolo and Tilmann Gneiting  “Forecaster’s Dilemma: Extreme Events and Forecast Evaluation”  In _Statist. Sci._ 32.1 The Institute of Mathematical Statistics, 2017, pp. 106–127  DOI: 10.1214/16-STS588
  * [49] C. Lorentzen and M. Mayer  “Peeking into the Black Box: An Actuarial Case Study for Interpretable Machine Learning”  In _SSRN Manuscript ID 3595944_ , 2020  DOI: 10.2139/ssrn.3595944
  * [50] U. Luxburg and B. Schoelkopf  “Statistical Learning Theory: Models, Concepts, and Results”, 2008  arXiv:0810.4752 [stat.ML]
  * [51] M. Mayer, S. C. Bourassa, M. Hoesli and D. Scognamiglio  “Estimation and Updating Methods for Hedonic Valuation”  In _Journal of European Real Estate Research_ 12.1 Emerald Group Publishing Ltd., 2019, pp. 134–150  DOI: 10.1108/JERER-08-2018-0035
  * [52] T. M. Mitchell  “Machine Learning”  USA: McGraw-Hill, Inc., 1997
  * [53] A. H. Murphy and H. Daan  “Forecast Evaluation”  In _Probability, Statistics and Decision Making in the Atmospheric Sciences_ Westview Press, Boulder, Colorado, 1985, pp. 379–437
  * [54] A. H. Murphy and R. L. Winkler  “Probability Forecasting in Meterology”  In _Journal of the American Statistical Association_ 79.387 [American Statistical Association, Taylor & Francis, Ltd.], 1984, pp. 489–500  DOI: 10.2307/2288395
  * [55] Allan H. Murphy  “A New Vector Partition of the Probability Score”  In _Journal of Applied Meteorology and Climatology_ 12.4 Boston MA, USA: American Meteorological Society, 1973, pp. 595–600  DOI: 10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2
  * [56] C. Nadeau and Y. Bengio  “Inference for the Generalization Error”  In _Machine Learning_ 52.3 Springer ScienceBusiness Media LLC, 2003, pp. 239–281  DOI: 10.1023/a:1024068626366
  * [57] B. Neyshabur, R. Tomioka and N. Srebro  “In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning”, 2015  arXiv:1412.6614 [cs.LG]
  * [58] N. Nolde and J. F. Ziegel  “Elicitability and backtesting: Perspectives for banking regulation”  In _The Annals of Applied Statistics_ 11.4, 2017, pp. 1833–1874  DOI: 10.1214/17-AOAS1041
  * [59] K. H. Osband  “Providing Incentives for Better Cost Forecasting”, 1985  DOI: 10.5281/zenodo.4355667
  * [60] A. J. Patton  “Volatility forecast comparison using imperfect volatility proxies”  In _Journal of Econometrics_ 160.1 Elsevier BV, 2011, pp. 246–256  DOI: 10.1016/j.jeconom.2010.03.034
  * [61] M.-O. Pohle  “The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation”  In _arXiv_ , 2020  arXiv:2005.01835 [stat.ME]
  * [62] D. M. W. Powers  “Evaluation: From Precision, Recall and F-Factor to ROC, Informedness, Markedness & Correlation”, 2007  arXiv:2010.16061 [cs.LG]
  * [63] S. Raschka  “Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning”  In _arXiv_ , 2018  arXiv:1811.12808 [cs.LG]
  * [64] D. R. Roberts, V. Bahn, S. Ciuti, M. S. Boyce, J. Elith, G. Guillera-Arroita, S. Hauenstein, J. J. Lahoz-Monfort, B. Schröder, W. Thuiller, D. I. Warton, B. A. Wintle, F. Hartig and C. F. Dormann  “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure”  In _Ecography_ 40.8, 2017, pp. 913–929  DOI: 10.1111/ecog.02881
  * [65] S. J. Russell and P. Norvig  “Artificial Intelligence: A Modern Approach”  Prentice Hall, 2009
  * [66] M. Schnaubelt  “A comparison of machine learning model validation schemes for non-stationary time series data”  Nürnberg: Friedrich-Alexander-Universität Erlangen-Nürnberg, Institute for Economics, 2019  URL: http://hdl.handle.net/10419/209136
  * [67] S. Shalev-Shwartz and S. Ben-David  “Understanding Machine Learning”  Cambridge University Press, 2014  DOI: 10.1017/cbo9781107298019
  * [68] R. M. Stein  “Benchmarking default prediction models: pitfalls and remedies in model validation”  In _The Journal of Risk Model Validation_ 1.1, 2007, pp. 77–113  DOI: 10.21314/JRMV.2007.002
  * [69] I. Steinwart, C. Pasin, R. Williamson and S. Zhang  “Elicitation and Identification of Properties”  In _JMLR Workshop Conf. Proc._ 35, 2014, pp. 1–45  URL: http://proceedings.mlr.press/v35/steinwart14.html
  * [70] A. Tharwat  “Classification assessment methods”  In _New England Journal of Entrepreneurship_ 17.1 Emerald Publishing Limited, 2020, pp. 168–192  DOI: 10.1016/j.aci.2018.08.003
  * [71] M. C. K. Tweedie  “An index which distinguishes between some important exponential families” Proceedings of the Indian Statistical InstituteGolden Jubilee International Conference, Statistics: Applications and New Directions., 1984, pp. 579–604
  * [72] D. S. Wilks  “Forecast Verification”  In _Statistical Methods in the Atmospheric Sciences_ , International Geophysics Series  Elsevier, 2011, pp. 301–394  DOI: 10.1016/B978-0-12-385022-5.00008-7
  * [73] J. Witkowski, R. Freeman, J. Wortman Vaughan, D. M. Pennock and A. Krause  “Incentive-Compatible Forecasting Competitions”  In _arXiv_ , 2021  arXiv:2101.01816 [cs.GT]
  * [74] M. V. Wüthrich and M. Merz  “Statistical Foundations of Actuarial Learning and its Applications”  In _SSRN Manuscript ID 3822407_ , 2021  DOI: 10.2139/ssrn.3822407
  * [75] C. Zhang, S. Bengio, M. Hardt, B. Recht and O. Vinyals  “Understanding deep learning requires rethinking generalization”, 2017  arXiv:1611.03530 [cs.LG]

##  Appendix A Exploratory Data Analysis

###  A.1 Regression: Workers’ Compensation Data Set

See Figures 13, 14, 15 and 16.

Figure 13: Histograms of categorical features of Workers’ Compensation data
set. Figure 14: Histograms of numerical features of Workers’ Compensation data
set. Figure 15: Boxplots of response UltimateIncurredClaimCost conditional on
categorical features of Workers’ Compensation data set. Figure 16: Density
plots of response UltimateIncurredClaimCost conditional on numerical features
Workers’ Compensation data set.

###  A.2 Classification: Telco Customer Churn Data Set

See Figures 17, 18, 19 and 20.

Figure 17: Histograms of categorical features of Telco Customer Churn data
set. Figure 18: Histograms of numerical features of Telco Customer Churn data
set. Figure 19: Mean response Churn conditional on categorical features of
Telco Customer Churn data set. Figure 20: Density plots of response Churn
conditional on numerical features of Telco Customer Churn data set.

##  Appendix B Tweedie Deviance and Homogeneous Bregman Functions

In Eq. (20) of [27], originally in [60], a rich family of homogeneous Bregman
functions with parameter $b\in\mathbb{R}$ is introduced for the positive real
line $z,y\in\mathbb{R}_{+}$ as

$S_{b}(z,y)=\begin{cases}\frac{1}{b(b-1)}(y^{b}-z^{b})-\frac{1}{b-1}z^{b-1}(y-z)&b\in\mathbb{R}\setminus\\{0,1\\}\\\
y\log\frac{y}{z}-y+z&b=1\\\ \frac{y}{z}-\log\frac{y}{z}-1&b=0.\end{cases}$

Up to a multiplicative constant, this coincides with the homogeneous scores in
Table 8 for $b=a>1$.

The Tweedie distributions $Tw_{p}(\mu,\varphi)$ [71] are a subfamily of the
exponential dispersion family (EDF), see [45, 74], and are characterised by
their mean–variance relation $\operatorname{Var}[Y]=\varphi\cdot\mu^{p}$ with
mean $\mu=\operatorname{\mathbb{E}}[Y]$, dispersion parameter $\varphi>0$ and
power $p$. Their deviance is given by

$d_{p}(y,\mu)=2\cdot\begin{cases}\frac{\max(0,y^{2-p})}{(1-p)\cdot(2-p)}-\frac{y\cdot\mu^{1-p}}{1-p}+\frac{\mu^{2-p}}{2-p}&p\in\mathbb{R}\setminus(0,1]\cup\\{2\\}\\\
y\log\frac{y}{\mu}-y+\mu&p=1\\\
\frac{y}{\mu}-\log\frac{y}{\mu}-1&p=2\end{cases}$

with valid domains:

response $y$ | mean $\mu$ | power $p$
---|---|---
$y\in\mathbb{R}$ | $\mu\in\mathbb{R}_{+}$ | $p<0$
$y\in\mathbb{R}$ | $\mu\in\mathbb{R}$ | $p=0$
$y\in\mathbb{R}_{+}\cup\\{0\\}$ | $\mu\in\mathbb{R}_{+}$ | $1\leq p<2$
$y\in\mathbb{R}_{+}$ | $\mu\in\mathbb{R}_{+}$ | $p\geq 2$

Well-known members are the Normal distribution ($p=0$), the Poisson
distribution ($p=1$), the Gamma distribution ($p=2$), and the inverse Gaussian
distribution ($p=3$). Remarkably, for $p\in(0,1)$, no Tweedie distribution
exists while the Bregman function $S_{b}$ is defined for all $b\in\mathbb{R}$.

On the common domains of $y$ and $z$, it holds that

$d_{p}(y,x)=2\cdot S_{2-p}(x,y)\,.$

This indicates that Tweedie deviances for $p\in(0,1)$ may be used as
consistent scoring function for the expectation, although no distribution
exists for this parameter range. On the other hand, the domain of $z$ for a
Bregman function can be extended to allow for $z=0$ for $0<b\leq 1$.

Furthermore, Tweedie distributions are the only distributions within the EDF
that are closed under scale transformations [45]: If $Y\sim
Tw_{p}(\mu,\varphi)$ then $tY\sim Tw_{p}(t\mu,t^{2-p}\varphi)$ for any $t>0$.
We recover the degree of homogeneity of Tweedie deviances, $h=2-p$, as the law
of transformation of the dispersion parameter of Tweedie random variables
under scale transformations.

##  Appendix C Simulation study on efficiency

As outlined in Subsubsection 5.1.6, the coefficient of variation
$\operatorname{CV}(Z)=\left(\operatorname{Var}[Z]/\operatorname{\mathbb{E}}[Z]^{2}\right)^{\nicefrac{{1}}{{2}}}$,
where $Z$ is the score difference, determines the asymptotic power of a
Diebold–Mariano test. To be more precise and to take into account the sample
size $n$ of the test data set, the power is inversely linked to the
coefficient of variation of the average score difference

$\overline{Z}_{n}=\frac{1}{n}\sum_{i=1}^{n}S(m_{A}(\boldsymbol{X}_{i}),Y_{i})-S(m_{B}(\boldsymbol{X}_{i}),Y_{i})\,.$

Figure 21: Speed of convergence with increasing test sample size of several
scoring functions in terms of $\operatorname{CV}(\overline{Z}_{n})$ evaluated
over $100$ simulations. Horizontal lines in the right figure indicate the
value of $\operatorname{CV}(Z)$ based on $10^{8}$ data points. Figure 22:
Speed of convergence with increasing test sample size of several scoring
functions for 9 different simulations of test sets. Values on $y$-axis are the
relative deviation of score differences with respect to their respective
values at $n=10^{6}$.

As an empirical illustration of the convergence speed, we simulate a Gamma
distributed response, conditionally on two features
$\boldsymbol{X}=(X_{1},X_{2})$, one categorical and one numerical feature,
$Y|\boldsymbol{X}\sim\mathrm{Gamma}$.

⬇

generate_data <- function(n_samples, seed = NA) {

if (!is.na(seed) && seed%%1==0) {

set.seed(seed)

}

df <- tibble(

color = sample(c("red", "blue"), n_samples, replace = TRUE, prob = c(0.2,
0.8)),

length = runif(n_samples, min = -2, max = 2)) %>%

mutate(color = as.factor(color))

true_mean <- exp(model.matrix(~ color + length, data = df) %*% c(4, -2, 1))

dispersion <- 2 # results are highly sensitive to this parameter

# E[Y] = shape * scale = mu

# Var[Y] = shape * scale**2 = dispersion * mu**2

# shape = 1 / dispersion

# scale = mu * dispersion

df$y <- rgamma(n_samples, shape = 1 / dispersion, scale = true_mean *
dispersion)

df

}

We fit a trivial model as $m_{A}$ as well as a (correctly specified) Gamma GLM
with log link as $m_{B}$ on a training sample of $\text{size}=$1000$$. We
evaluate different scoring functions—squared error (corresponding to Gaussian
deviance), Poisson deviance, Gamma deviance and inverse Gaussian deviance,
which are Tweedie deviances with power $p=0,1,2,3$—on a sequence of nested
test sets with increasing size $n=1,\ldots,10^{6}$.

Take the following results with a grain of salt as they are very sensitive to
slight changes of the distribution of the data. The relative deviation of the
score difference $Z$ on different test sets can be seen in Figure 22. We
repeat this simulation $100$ times with different random seeds and calculate
the coefficient of variation for each test size $n$. Figure 21 visualises
$\operatorname{CV}(\overline{Z}_{n})$ as well as
$\sqrt{n}\operatorname{CV}(\overline{Z}_{n})$, taking into account the
convergence rate of $n^{-1/2}$ of $\operatorname{CV}(\overline{Z}_{n})$. It
further exhibits the theoretical quantity of $\sqrt{n}\operatorname{CV}(Z)$
(based on a Monte Carlo simulation with size $10^{8}$). It is well visible
that the Gamma deviance converges fastest with inverse Gaussian and Poisson
deviance close together as second. The squared error seems to suffer from some
data points on which it produces outliers.474747We even observed erratic
behaviour of the inverse Gaussian deviance for values of $y$ very close to
zero when experimenting with different parameters of data distributions.  This
simulation study suggests that scores derived from a quasi maximum likelihood
approach of the underlying data distribution exhibit a fast convergence. This
underpins the theoretical intuition provided in Subsubsection 5.1.6. We
emphasise that this intuition ignored the presence of features. This could be
the reason for the sensitivity of the results with respect to small changes in
the distribution of the data. We encourage further research which takes the
presence of feature information into account.