Qureshi and Zaman

The Impact of COVID-19 on Sports Betting Markets

The Impact of COVID-19 on Sports Betting Markets

Khizar Qureshi Massachusetts Institute of Technology, Cambridge, MA 02139,
<EMAIL_ADDRESS>Tauhid Zaman Yale University, New Haven, CT 06511,
<EMAIL_ADDRESS>
We investigate the impact of the COVID-19 pandemic on the betting markets of
professional and college sports. We find that during the pandemic, the
moneyline betting markets of the National Basketball Association (NBA) became
very inefficient. During this period, if one bet uniformly on underdog teams
in the NBA, one could achieve a 16.7% profit margin. It is hypothesized that
this inefficiency is due to the absence of live audiences during the NBA
games. Such inefficiencies are not seen for any other sport. Much of the
inefficiency comes from a small fraction of games with moneyline odds in the
233 to 400 range. We find that with clever strategies, one is able to achieve
a 26-fold gain on an initial investment by betting on NBA underdogs during
this time period.

probability, statistics, sports, COVID-19, moneyline bets

##  1 Introduction

Over the past three decades, there has been rapid growth in the size of
betting markets for professional and college sports. A survey by the American
Gaming Association found that between 2018 and 2020 the legal U.S. sports
betting market surged from $6.6 billion to $25.5 billion (AGA 2021). Alongside
the launch of several new sports betting markets, there was also adoption on a
state level, most recently in Illinois and Colorado. Despite this increase in
the popularity of sports betting, the general consensus has been that betting
markets have stayed efficient. There have been some studies finding specific
inefficiencies embedded within a small set of features or match-ups
(Berkowitz, et al. 2017, Borghesi 2007, Gandar et al. 2001), but nothing
systematically inefficient. It is unlikely that these markets are profitable
for amateur betters, unless there was some extreme exogenous event that
disrupted either these markets or the sports themselves.

The COVID-19 pandemic was just such an extreme event that affected nearly
every professional and college sports league in the world in 2020 and 2021.
The National Basketball Association (NBA) and National Hockey League (NHL) had
to suspend their on-going seasons when the pandemic began (NBA 2020, NHL
2020). Later, in the summer of 2020, the NBA resumed its season in an isolated
environment where players were quarantined and no fans were allowed (NBA
2020). This isolation _bubble_ was the NBA’s attempt to let the season
continue safely. While Major League Baseball (MLB) and the NHL resumed their
2020 seasons without fans, the National Football League (NFL) resumed with 13
teams allowing fans at partial capacity. The NBA’s 2021 season began in
complete isolation and then gradually fans were allowed to attend the games,
depending on the hometown city’s policies. COVID-19 affected other aspects of
the sports besides fan attendance. Game schedules were altered to minimize the
amount of travel for the teams. Players who tested positive for COVID-19 (or
had contact exposure), were excluded from participating in games. This often
resulted in games where superstars or key players were missing.

An interesting element of sports is their respective betting markets. One
popular type of bet in these markets is known as the _moneyline_. In this bet,
the odds makers give payouts for each team in a game. The team with the higher
payout is known as the _underdog_ because they have a lower implied
probability of winning, assuming the payouts are chosen to make the bet have
non-positive mean payout. In efficient markets, one cannot make a consistent
profit by betting solely on underdogs or favorites. An interesting question is
what impact COVID-19 had on the efficiency of betting markets. If COVID-19
created inefficiencies, in what sport did this occur, what was the nature of
the inefficiencies, and how much could one earn by betting in these
inefficient markets?

In this work we analyze the impact of COVID-19 on the efficiency of moneyline
betting markets for a variety of sports. Our main finding is that the NBA
experienced incredible inefficiencies in its betting markets during COVID-19,
while other major sports had markets that stayed efficient. The inefficiency
in the NBA market was so stark that simple betting strategies were profitable.
For instance, consider the strategy where each day one bets an equal amount on
the underdog in each game. Using this simple underdog strategy from when the
NBA COVID-19 bubble began until the end of the 2020 season gave a return of
16.7%. More clever strategies that reinvest the winnings gave returns as high
as 2,666%. We conduct a deeper analysis of the data to explain the nature of
the NBA inefficiencies. We also test different betting strategies to see which
ones are most profitable in this market.

This paper is organized as follows. We begin by reviewing related literature
in Section 2. Then in Section 3 we describe our dataset on moneyline odds for
various sports. In Section 4 we present a detailed analysis of the efficiency
of the moneyline betting markets for all sports across multiple seasons,
including during the COVID-19 pandemic. We study different betting strategies
given the moneyline inefficiencies in Section 5. Finally, we conclude in
Section 6.

##  2 Related Literature

Many studies have looked at the efficiency in the sports betting markets.
Gandar et al. (2001) find that the home-field advantage is efficient for the
NFL, MLB, and NBA in both the regular season and playoff games. However,
Borghesi (2007) finds that in the NFL the realized winning probability for the
home-underdog is significantly larger during the later part of a season.
Woodland, L.M., Woodland, B.M. (1994) found for college basketball and
football that there is clear evidence of the favorite long shot bias in money-
line betting markets and that betting on heavy favorites offers a near zero
return over several years, suggesting that the markets for these two sports
are efficient within transaction costs. Gray (1997) find that in select
seasons a few linear models can be used to generate significant profits in NFL
betting markets, but for most years the market is efficient.

Much of the COVID-19 related sports research focused not on betting markets,
but on the potential impacts of the virus on athletes at both the microscopic
and macroscopic level. For example, Verwoert, G.C., et al. (2020) consider a
tree-based protocol dictating the stratification of athletes from a
cardiovascular perspective. The logic branches on asymptomatic/regional/local
symptoms and hospitalization. Similarly, researchers have provided a cardio-
vascular based risk-mitigation approach for the eligibility of an athlete to
return to a sport depending on his or her state (positive test or waiting for
test) (Schellhorn, P., et al. 2020, Baggish, A. L., Levine, B.D. 2020).
Meanwhile, Wong, A.Y.Y., et al. (2020) studied the transmission risk of
COVID-19 in soccer based on the degree of contact between players and their
actions. For example, using forwards and mid-fielders, they detail the
contagion risk for a player over various time intervals of the game.

##  3 Moneyline Definitions and Datasets

###  3.1 Moneyline Bets

We begin by presenting a brief description of moneyline bets. In a game
between two teams, oddsmakers will set moneyline odds for each team. If one
team’s odds are strictly greater than the other team’s odds, we refer to the
team with the larger odds as the _underdog_ and the other team as the
_favorite_. If both teams’ odds are equal, we refer to them both as underdogs.
The moneyline bet works as follows. Consider a game where team $u$ has
moneyline odds $o_{u}$. In typical moneyline bets we either have
$o_{u}\leq-100$ or $o_{u}\geq 100$, and each case has a different payout.
First, we consider $o_{u}\geq 100$. In this case, if one bets 100 USD on $u$
and the team wins, then one receives a payout of $100+o_{u}$ USD (a net profit
of $o_{u}$ USD). If on the other hand, $u$ loses, then one incurs a loss of
100 USD. Second, we consider $o_{u}\leq-100$. In this case, if one bets
$o_{u}$ USD on $u$ and the team wins, then one receives a payout of
$100+o_{u}$ USD (a net profit of $100$ USD). If $u$ loses, then one incurs a
loss of $o_{u}$ USD.

The implied probability of a team winning is determined by the moneyline odds
and the payout of the moneyline bet. Let us define the implied probability of
winning for a team $u$ as $p_{u}$. We can calculate the implied probability by
setting the expected profit of a moneyline bet on the team equal to zero. For
$o_{u}\geq 100$, this gives $o_{u}p_{u}-100(1-p_{u})=0$ and for
$o_{u}\leq-100$, this gives $100p_{u}+o_{u}(1-p_{u})=0$. Solving for $p_{u}$
gives

$\displaystyle p_{u}$
$\displaystyle=\frac{100}{100+|o_{u}|},~{}~{}~{}o_{u}\geq 100.$ (1)
$\displaystyle p_{u}$
$\displaystyle=\frac{|o_{u}|}{100+|o_{u}|}~{}~{}~{}o_{u}\leq-100.$ (2)

###  3.2 Underdog Profit Margin

We now consider measures to quantify the efficiency of betting markets. There
are multiple ways to do this, but in this work we focus on one simple measure
which will prove useful. We assume a bettor places a 1 USD bet on the underdog
in each game in a set of games. Then the efficiency measure, which we refer to
as the _average underdog profit margin_ , is the average profit margin per
game using this underdog betting strategy. To provide an expression for this
measure, we define some terms. For a team $u$, let $p_{u}$ be the implied win
probability based on the moneyline odds, and let $q_{u}$ be the actual win
probability. Let $W_{u}$ be a Bernoulli random variable that is one if the
team wins the game. With our notation we have $\mathbf{E}[W_{u}]=q_{u}$.
Assume we bet 1 USD on the underdog in $n$ different games. Then a simple
calculation shows that the average underdog profit margin, which we define as
$\alpha$, is equal to

$\displaystyle\alpha=-1+\frac{1}{n}\sum_{u=1}^{n}\frac{W_{u}}{p_{u}}.$ (3)

To gain insights into this measure, we take the expectation over the
uncertainty in game outcome to obtain

$\mathbf{E}[\alpha]=-1+\frac{1}{n}\sum_{u=1}^{n}\frac{q_{u}}{p_{u}}.$

We see from this expression that $\alpha$ measures how much the mean of the
ratio $q_{u}/p_{u}$ deviates from one. If the oddsmakers set the odds
accurately so $p_{u}=q_{u}$, then $\alpha$ is zero. If the oddsmakers are
efficiently setting underdog moneyline odds, $\alpha$ should be negative,
implying one cannot make a consistent profit by betting on underdogs. However,
if there are many upsets and underdogs win more frequently than implied by the
odds, $\alpha$ will be positive.

###  3.3 Data

We collected the moneyline odds for thousands of games from a website that
archived this data for multiple sports across multiple seasons
(SportsBookReviewsOnline 2021). Our dataset covers professional sports leagues
such as the NFL, NBA, NHL, and MLB, along with college sports such as NCAA
Football (NCAAF) and NCAA Basketball (NCAAB). For each game, we have the
initial moneyline odds for both teams, the date of the game, and the game
outcome. In total we have moneyline data for over 130 thousand games spanning
14 years. The complete dataset can be obtained from our project repository
(Sports anomalies 2021).

Our data covers seasons as far back as 2007, but the data for MLB begins at
2010. For this reason, in our analysis we consider seasons between 2010 and
2021 for all sports. Also, in order to exclude potentially erroneous records
in our dataset, we require the following conditions for a game to be included:

  1. 1.

The favorite odds must be less than or equal to -100

  2. 2.

The underdog odds must be either greater than 100 or between -200 and -100.

There are 109,249 games which satisfy these constraints in our data set and 40
games which do not. Table 1 contains a summary of the dataset.

Table 1: The number of games for each sport in our moneyline odds dataset, along with the dates covered by each sport. | Start Date | End Date | Number of Games
---|---|---|---
MLB | 2010-04-04 | 2020-10-27 | 25,599
NBA | 2007-10-30 | 2021-03-04 | 17,196
NCAAB | 2007-11-05 | 2021-02-16 | 56,052
NCAAF | 2007-08-30 | 2020-09-26 | 10,672
NFL | 2007-09-06 | 2021-02-07 | 3,740
NHL | 2007-09-29 | 2021-03-22 | 17,080

##  4 Statistical Analysis of Betting Market Inefficiencies During COVID-19

COVID-19 disrupted certain games in our dataset, both in terms of game
outcomes and betting market efficiency. In this section we conduct a
statistical analysis comparing these properties with respect to underdogs for
COVID-19 games and normal season games. To begin this analysis we had to
determine which games to designate as COVID-19 games. The NBA and NHL
experienced mid-season pauses due to COVID-19. The COVID-19 games for these
sports were played after the resumption of the season. For the NFL and NCAAF,
the season was not paused, so their COVID-19 games started in the fall of
2020. The NCAAB only cancelled the March Madness Tournament, but all regular
season games were played before COVID-19. Table 2 contains the start dates of
the COVID-19 games and the number of COVID-19 games played for each sport,
along with the number of games in a normal season for reference. There is a
range of games due to some teams receiving different treatment based on their
win-loss records when COVID-19 began impacting game schedules.

Sport | COVID-19 Games | Number of Normal | Number of COVID-19
---|---|---|---
| Start Date | Season Games | Games
NBA | 2020-07-30 | 81 | 64-72
NFL | 2020-09-10 | 16 | 17
NHL | 2020-08-01 | 68 | 57-63
MLB | 2020-07-23 | 162 | 60
NCAAF | 2020-09-03 | 10-13 | 4-12
NCAAB | N/A | 25-35 | 25-35
Table 2: Start date of COVID-19 games, number of games per season, and number
of COVID-19 games played for each sport in our dataset.

###  4.1 Underdog Profit Margin and Win Probability

For each season we study two properties of underdogs: their actual win
probability, and the efficiency of their moneyline odds. The underdog win
probability shows how frequently upsets occurred. The underdog profit margin
provides a monetary value to these upsets. Figure 1 shows the average underdog
win probability and profit margin for the sports during normal and COVID-19
games. It can be seen that the NBA shows the largest increase in win
probability and profit margin for underdogs during COVID-19. The NBA underdog
win probability goes from approximately 0.3 to 0.4. For all other sports,
there is not such a visible difference. In fact, if we look at the underdog
profit margin in Figure 1, we see that the NBA is the only sport showing a
substantial positive mean during COVID-19.

We tested the significance of the differences in both underdog win probability
and profit margin using multiple statistical tests. Significance was assessed
using the Holm-Bonferonni correction (Holm, S. 1979) due to the testing of
hypotheses for multiple sports. We used non-parametric tests such as the
Kolomogorov-Smirnov (KS) and Mann-Whitney U (MW) test. Non-parametric tests
are more appropriate for the underdog profit margin, which exhibits multi-
modal behavior and has substantial skew.

The test results are shown in Table 3. The difference in underdog win
probability for the NBA is statistically significant for all tests at a 1%
level. The only other sport showing a significant difference in win
probability is the NCAAB, but this is only for the MW test. The NBA also shows
a significant difference in its underdog profit margin for both non-parametric
tests. The NCAAB underdog profit margin has a significant difference only for
the MW test. However, from Figure 1 we see that the mean value is negative,
which is less interesting from a betting perspective.

The average NBA underdog profit margin during COVID-19 is the only
substantially positive value among all sports. To verify that this value is
truly positive and not just a random fluctuation, we conduct a Wilcoxon
signed-rank test (Wilcoxon, Frank 1945) on the NBA COVID-19 games. This test
is a non-parametric version of the standard paired t-test without any
assumptions on normality. The null hypothesis is that the distribution of the
values is symmetric about zero. We find that according to this test, we can
reject the null hypothesis at the 1% level (p-value $\leq 10^{-5}$). This
indicates that the average underdog profit margin is positive for the NBA
during COVID-19. We also carried out the test on a season by season basis, and
found that the only two seasons for which this result holds was during the
2019-2020 and 2020-2021 COVID-19 seasons. We could not reject the null
hypothesis for the remaining seasons.

Figure 1: Plot of (left) mean underdog win probability and (right) mean underdog profit margin grouped by sport and COVID-19 status. The error bars represent 95% confidence intervals. Table 3: P-values for different statistical tests for the underdog win probability and profit margin during COVID-19 versus normal time periods segmented by sport (∗ indicates significance at a 5% level and ∗∗ indicates significance at a 1% level under the Holm-Bonferonni correction). The tests are Kolmogorov-Smirnov (KS) and Mann-Whitney U (MW). Sport | Underdog win probability | Underdog profit margin
---|---|---
KS | MW | KS | MW
NBA | $0.0010^{**}$ | $0.0000^{**}$ | $0.0010^{**}$ | $0.0000^{**}$
NFL | 1.0000 | 0.2937 | 0.9957 | 0.2693
NHL | 0.5572 | 0.0541 | 0.5572 | 0.1895
MLB | 0.8812 | 0.1173 | $0.0001^{**}$ | 0.0838
NCAAB | 0.0332 | $0.0006^{**}$ | 0.0207 | $0.0010^{**}$
NCAAF | 0.5449 | 0.0319 | 0.5449 | 0.0326

###  4.2 Underdog Profit Margin Versus Moneyline Odds for NBA COVID-19 Games

We now examine NBA COVID-19 games more closely to understand which games have
the highest underdog profit margin. The underdog moneyline odds cover a wide
range. An interesting question is which segments of this range showed a high
underdog profit margin. To answer this question, we segment the NBA games by
their underdog implied win probabilities into bins ranging from zero to one.
Table 4 shows the bin intervals and the number of games in each bin.

We plot the mean underdog win probability and profit margin versus implied
underdog probability bin in Figure 2. We see that both quantities are greater
during COVID-19 than during normal seasons for several bins. To asses the
statistical significance of these differences, we conducted multiple tests.
The results are shown in Table 5. Because we are testing multiple bins
simultaneously, we used the Holm-Bonferonni correction to assess significance.
We find that only the $(0.2,0.3]$ bin has a significant difference in both
metrics for the MW test at a 1% level.

From this analysis we find evidence that games with implied underdog win
probabilities between 0.2 and 0.3 provided much of the inefficiency in the NBA
betting markets during COVID-19. This corresponds to games where the underdog
odds are between 233 and 400. From Table 4 we see that there are 153 games in
this bin. This represents 21.7% of NBA games played during COVID-19.
Therefore, we see that a small fraction of the NBA games played during
COVID-19 are responsible for the positive underdog profit margin during this
period.

Figure 2: Plot of (left) mean underdog win probability and (right) mean underdog profit margin for the NBA grouped by underdog implied win probability (the value on the x-axis is the inclusive upper bound of the implied underdog probability bin.) The error bars represent 95% confidence intervals. Table 4: Number of NBA games in each implied probability bin for normal and COVID-19 time periods. Implied probability | Number of NBA games | Number of NBA games
---|---|---
bin | (Normal) | (COVID-19)
$(0.0,0.1]$ | 539 | 5
$(0.1,0.2]$ | 1,955 | 67
$(0.2,0.3]$ | 2,603 | 153
$(0.3,0.4]$ | 3,627 | 240
$(0.4,0.5]$ | 3,445 | 210
$(0.5,1.0]$ | 374 | 30
Table 5: P-values for different statistical tests for the NBA underdog win probability and profit margin during COVID-19 versus normal time periods segmented by underdog implied win probability (∗ indicates significance at a 5% level and ∗∗ indicates significance at a 1% level under the Holm-Bonferonni correction). The tests are Kolmogorov-Smirnov (KS) and Mann-Whitney U (MW) Underdog implied | Underdog win probability | Underdog profit margin
---|---|---
win probability | KS | MW | KS | MW
$(0.0,0.1]$ | 0.5524 | $0.0025^{*}$ | 0.5524 | 0.0025
$(0.1,0.2]$ | 0.8359 | 0.0514 | 0.8359 | 0.0537
$(0.2,0.3]$ | 0.0191 | $0.00021^{**}$ | 0.0186 | $0.0003^{**}$
$(0.3,0.4]$ | 0.8743 | 0.1118 | 0.8457 | 0.1139
$(0.4,0.5]$ | 0.6038 | 0.0652 | 0.5935 | 0.0862
$(0.5,1.0]$ | 1.0000 | 0.4333 | 0.9999 | 0.2974

###  4.3 Post-COVID-19 NBA Games

Though the COVID-19 pandemic continued throughout the NBA 2020-2021 season,
there was a point when the league began changing its policies. The first
return from COVID-19 in the 2019-2020 season was played in the isolated
bubble. The beginning of the 2020-2021 season was played in the normal arenas,
but with no or very limited fans in attendance. After the NBA All-Star Game,
many teams began allowing larger numbers of fans to attend the games (NBA
2021). We wish to understand the impact of fans on the performance of the
betting markets.

We plot the average underdog profit margin versus season for the NBA in Figure
3. We see that the average profit margin became very positive in the COVID-19
seasons, but then became negative during the post-COVID-19 season when fans
returned. We conduct a one-sided Wilcoxon signed-rank test on the two COVID-19
seasons and the post-COVID-19 season using a Bonferonni-Holm correction to
verify the sign of these profit margins. The resulting average underdog profit
margin and p-values are shown in Table 6. As can be seen, the average underdog
profit margin is negative for the post-COVID games when fans returned in
attendance, while it is positive for the COVID-19 games with no fans
(significant at a 1% level).

Figure 3: Plot of mean underdog profit margin for the NBA versus season and COVID-19 game status.  Table 6: Average underdog profit margin and one sided p-values for Wilcoxon signed-rank test for the NBA underdog profit margin during the 2019-2020 and 2020-2021 COVID-19 seasons and the 2020-2021 post-COVID-19 seasons (∗ indicates significance at a 5% level and ∗∗ indicates significance at a 1% level under the Holm-Bonferonni correction).  Season | Average Underdog
---|---
| Profit Margin (p-value)
2019-2020 COVID-19 | 0.17 ($0.001^{**}$)
2020-2021 COVID-19 | 0.10 ($0.001^{**}$)
2020-2021 Post-COVID-19 | -0.9 ($0.001^{**}$)

###  4.4 Discussion

Our analysis has shown that the NBA had large inefficiencies in their
moneyline betting markets during COVID-19, while other sports did not. If we
assume that the oddsmakers are good at incorporating known information to set
the odds for the games, then any inefficiency must come from some unaccounted
factors. These factors may arise from a combination of the structure of the
sports leagues, the nature of the gameplay, and the impact of COVID-19 on the
gameplay.

We begin with the structure of the leagues. By structure, we refer to the
tendency for underdogs to win games. In leagues with a concentrated
distribution of talent, one would expect underdogs to win more frequently. In
fact, for a league where all teams are equally skilled, we would expect the
underdog win probability to be near 0.5. From Figure 1, we see that the sports
with the lowest underdog win probabilities are the two college sports (NCAAF
and NCAAB). The spectrum of talent is quite wide in college sports, as they
are not professional. Therefore, one would expect in some games upsets to be
incredibly unlikely. While we did see a small increase in the underdog win
probability for the NCAAB, the average underdog profit margin was not positive
for both of these college sports. Therefore, it seems that COVID-19 was not
able to impact the games enough to overcome the wide talent spectrum in the
college leagues.

We next consider the nature of gameplay. The highest average underdog win
probabilities during normal season games belong to the NHL and MLB, with
values exceeding 0.4. This may be due to how these sports are played. Table 7
shows statistics for the sum of points scored by both teams in a game, which
we refer to as _total points_. We see that the NHL and MLB have the lowest
average total points and highest coefficient of variation (standard deviation
divided by mean) for total points. This suggests that scoring is rare and the
points scored in a game can fluctuate greatly. These fluctuations may be due
to random factors which cannot be incorporated into the odds. Baseball
involves hitting a small ball thrown at a high velocity with a narrow bat.
Hockey involves hitting a puck into a small net guarded by a goalie.
Randomness plays a large role in both of these sports. This is likely why the
underdog win probability is so large. We saw that the average underdog profit
margin of the MLB and NHL were both negative. Therefore, it appears that the
random factors arising from COVID-19 could not increase the overall randomness
such that the betting markets became inefficient.

The NBA’s average underdog win probability was near 0.3 during normal seasons,
and increased to 0.38 during COVID-19. From Table 7 we see that the NBA has
the lowest coefficient of variation of total points. Therefore, NBA basketball
is inherently less random than other sports. The major impact of COVID-19 was
to remove live audiences from the games. In the bubble, there was also no
travel and all teams lived in a closed environment. However, we saw in Figure
3 that when the NBA went from the bubble to their own arenas, but without
fans, the average underdog profit margin stayed positive. However, once
audiences were allowed to attend the games after the All-Star Game, the
average underdog profit margin became negative again. Therefore, it seems
likely that the absence of fans at the games was a cause of the betting market
inefficiency. It is not clear why this is the case. One hypothesis is that
when fans are removed, the home team advantage is eliminated. The NBA is a
professional league, so it is likely the skill level of the players are
concentrated. If this is the case, then when the home field advantage is
absent, the game becomes more susceptible to randomness. In fact, the average
underdog win probability increased during COVID-19 to a value comprable to
more random sports such as hockey or baseball. Our analysis shows that the
oddsmakers were not able to account for this, resulting in the inefficiency in
the markets.

Total Points per Game | NBA | NFL | NHL | MLB | NCAAB | NCAAF
---|---|---|---|---|---|---
Mean | 205.3 | 45.3 | 5.6 | 8.8 | 139.6 | 55.6
Standard deviation | 22.1 | 14.1 | 2.3 | 4.4 | 19.8 | 18.4
Coefficient of variation | 0.11 | 0.31 | 0.41 | 0.50 | 0.14 | 0.33
Table 7: Statistics of the total points per game (sum of the points scored by
each team in a game) for different sports.

##  5 Underdog Betting Strategies

The analysis in Section 4 showed that the NBA average underdog profit margin
was positive during COVID-19. This suggests that one could have made profit by
betting on NBA underdogs during this period. From Figure 1 we see that by just
betting an equal amount of money on the underdog in each game results in a
16.7% profit margin. In this section we explore how much more profit could be
achieved with more complex betting strategies.

###  5.1 Betting Scenario

We consider a scenario where we place daily bets on the underdogs in each
game. The bets for games played in a single day are placed simultaneously.
This is close to what would be done in practice as many games are played at
the same time. The bankroll on day $t$ is denoted $M_{t}$ and we begin with a
bankroll of $M_{0}$. We use the following strategy to determine the amount of
the bankroll to bet each day. We select a value $\lambda\in[0,1]$, and each
day we only reinvest a fraction $\lambda$ of the total bankroll on the games.

After designating $\lambda M_{t}$ for betting on day $t$, we must decide how
to allocate these funds across the available games. In practice we would not
know a priori which games are profitable. Therefore, we must consider
strategies which only utilize information available before the games begin.
For an underdog team $u$ in a game, the information we consider is the implied
underdog win probability $p_{u}$ which is set by the betting market. We must
choose a strategy that maps the $p_{u}$ to a bet amount. To do this, we define
$w_{u}\in[0,1]$ as the fraction of the allocated bankroll to bet on the
underdog. With this notation the wager on $u$ is given by $\lambda
M_{t}w_{u}$. To specify $w_{u}$, we select a non-negative weight function
$f(p_{u})$. Then $w_{u}$ is given by

$\displaystyle w_{u}=\frac{f(p_{u})}{\sum_{v\in G}f(p_{v})},$

where we denote the set of games played on the day as $G$.

There are many possibilities for $f(p_{u})$. If we assume that the implied
underdog win probability $p_{u}$ equals the true win probability $q_{u}$, then
the underdog bet has a mean payout of zero. In this case, underdog bets can be
distinguished by other statistics, such as their variance. For a 1 USD
moneyline bet on an underdog with implied and true win probabilities both
equal to $p_{u}$, the variance is $(1-p_{u})/p_{u}$. This means that games
with lower underdog win probabilities have a higher variance. Risk-averse
strategies would place smaller wagers on high variance bets, given that the
bets have equal means. This translates to weight functions $f(p_{u})$ that are
monotonically increasing in $p_{u}$. In contrast, risk-seeking strategies
would have monotonically decreasing weight functions. We list several
different choices for the weight functions in Table 8. We consider both risk-
averse and risk-seeking strategies. The weight functions for risk-seeking
strategies are the inverse of the implied underdog win probability, the
inverse standard deviation of a Bernoulli random variable for the underdog
winning (with $p_{u}\leq 0.5$), and the standard deviation of a 1 USD underdog
moneyline bet. The risk-averse weight functions are simply the inverse of
these functions. We also consider a uniform weight function, which has no
risk-preference.

Name | Weight Function
---|---
Uniform | $1$
Probability | ${p_{u}}$
Inverse Probability | $\frac{1}{p_{u}}$
Bernoulli | $\sqrt{p_{u}(1-p_{u})}$
Inverse Bernoulli | $\frac{1}{\sqrt{p_{u}(1-p_{u})}}$
Moneyline | $\sqrt{\frac{1-p_{u}}{p_{u}}}$
Inverse Moneyline | $\sqrt{\frac{p_{u}}{1-p_{u}}}$
Table 8: Weight functions $f(p_{u})$ used in different underdog betting
strategies.

###  5.2 Betting Performance

We test different betting strategies by varying $\lambda$ and $f(p_{u})$. For
$\lambda$ we use values between zero and one spaced 0.1 apart. We start with a
bankroll of $M_{0}=100$ USD and bets are place on all games in the COVID-19
portion of the 2019-2020 season, and all games in the 2020-2021 season before
the All-Star Game. Figure 4 shows the return for each strategy. We see that
the return is maximized for the strategy where $\lambda=1.0$ and the weight
function is inverse probability. The corresponding return is 2,666 USD, or
nearly a 26-fold gain in the initial investment. The inverse probability
weight is a risk-seeking strategy and places more bets on games with high
underdog odds. However, all weight functions give similar returns when the
entire bankroll is bet, except for the probability weight, which ends up
losing the entire bankroll. We compare the time evolution of the returns for
each weighting function with $\lambda=1.0$ in Figure 5. As can be seen, the
probability weight function loses the entire bankroll within a month. All
other weight functions show very similar growth for the return over the course
of the COVID-19 seasons.

Figure 4: Returns [USD] from an initial investment of 100 USD of NBA underdog
betting strategies during COVID-19 for different weight functions and values
of the reinvestment fraction $\lambda$. A value of zero indicates ruin.
Figure 5: Plot of the cumulative returns of NBA underdog betting strategies
during COVID-19 for different weight functions and a reinvestment fraction of
$\lambda=1.0$. The initial investment is 100 USD.

The final return is not the only metric we can use to evaluate the betting
strategies. Many times one is more concerned with the risk taken to achieve a
given return. One way to quantify this is to use the Sharpe ratio, which
equals the mean return divided by the standard deviation of the return.
Strategies with large Sharpe ratios achieve a return with very little risk.
The Sharpe ratio assumes the returns are normally distributed. However, in the
case of moneyline bets, the returns have a bimodal distribution (in fact, for
a given game the returns take two values, one negative and one positive). For
this reason, we can define a more robust version of the Sharpe ratio as

$\gamma=\frac{\text{median}(R)}{\text{MAD}(R)}$ (4)

where $R$ is the daily return, and MAD is the median absolute deviation. Both
the numerator and denominator of this modified Sharpe ration are robust to
outliers, making this measure more appropriate for the returns associated with
moneyline bets.

We show the robust and normal Sharpe ratios for different betting strategies
in Figures 6 and 7. We see that $\lambda=1.0$ does not have the highest Sharpe
ratio. Rather, the highest Sharpe ratios come from the Bernoulli weight with
$\lambda$ = 0.1. This is a risk-averse strategy as most money is not placed on
low probability games and only a small fraction of the bankroll is reinvested
each day. The return is 291.73 USD for this strategy, which is much lower than
the 26-fold gain achieved with a more risk-seeking approach.

Figure 6: Robust Sharpe ratios of NBA underdog betting strategies during
COVID-19 for different weight functions and values of the reinvestment
fraction $\lambda$.  Figure 7: Sharpe ratios of NBA underdog betting
strategies during COVID-19 for different weight functions and values of the
reinvestment fraction $\lambda$.

##  6 Conclusion

COVID-19 affected nearly all professional and college sports. However, our
analysis found that the betting markets for most sports remained efficient,
except for the NBA. Here we saw that there were many more upsets than
predicted by the odds makers. We are not able to precisely identify the
reasons for this inefficiency, and it remains an open question as to why it
occurred. However, we do have supporting evidence that it may be due to the
more frequent scoring in basketball combined with the absence of fans when the
NBA season resumed during COVID-19. For whatever reason, odds makers were not
able to adjust for the impact of these factors.

This NBA market inefficiency provided a lucrative opportunity for bettors. We
found that most of the inefficiency was concentrated around a small percentage
of the games where the underdog had odds between 233 and 400. Simply betting
an equal amount on every underdog resulted in a 16.7% return, while more
complex strategies which combine game specific allocations and reinvestment of
winnings resulted in a 26-fold gain in the initial investment. Our work shows
that sports betting markets are generally efficient, but occasionally odds
makers are not able to correctly account for the impact of extreme events.

## References

  * Baggish, A. L., Levine, B.D. (2020) Baggish, A. L., Levine, B.D. (2020) Icarus and Sports After COVID 19: Too Close to the Sun? Circulation 142:615-517.
  * Berkowitz, et al. (2017) Berkowitz, J.P., et al. (2017) A favorite-longshot bias in fixed-odds betting markets: Evidence from college basketball and college football The Quarterly Review of Economics and Finance 63:233-239.
  * Borghesi (2007) Borghesi R (2007). The Late-Season Bias: Explaining the NFL’s Home Underdog Effect Applied Economics 39\.
  * Gandar et al. (2001) Gandar JM, et al. (2001) The home field advantage revisited: a search for the bias in other sports betting markets Journal of Economics and Business 53:439-453.
  * Gray (1997) Gray, PK., Gray SF. (2001) Testing Market Efficiency: Evidence from the NFL Sports Betting Market The Journal of Finance 52:1725-1737.
  * Hausch, Donald B., Ziemba, W.T. (2008) Hausch, Donald B., Ziemba, W.T. (2008) Handbook of Sports and Lottery Markets (Elsevier).
  * Kaplan, E.H., Rich, R. (2017) Kaplan, E.H., Rich, R. (2017) Decomposing Pythagoras Journal of Quantitative Analysis in Sports.
  * Lopez-Gonzalez (2017) Lopez-Gonzalez, H., Griffiths, M.D. (2017). Understanding the convergence of online sports betting markets. International Review for the Sociology of Sport in press.
  * Ludden, I.G., et al. (2020) Ludden, I.G., et al. (2020) Models for generating NCAA men’s basketball tournament bracket pools Journal of Quantitative Analysis in Sports.
  * Moskowitz (2015) Moskowitz, T. (2015). Asset Pricing and Sports Betting. Behavioral & Experimental Finance eJournal.
  * NBA (2020) NBA (2020) Coronavirus pandemic causes NBA to suspend season after player tests positive https://www.nba.com/news/coronavirus-pandemic-causes-nba-suspend-season
  * NBA (2020) NBA (2020) It’s official: NBA Plans to Restart Season in July at Disney World Orlando https://www.nba.com/blazers/forwardcenter/its-official-nba-plans-restart-season-july-disney-world-orlando
  * NBA (2021) NBA (2021) Where NBA teams stand on in-arena attendance https://www.nba.com/news/where-nba-teams-stand-on-in-arena-attendance
  * NHL (2020) NHL (2020) NHL to pause season due to coronavirus https://www.nhl.com/news/nhl-coronavirus-to-provide-update-on-concerns/c-316131734
  * Pearson, M., et al. (2020) Pearson, M., et al. (2020) An exploration of predictive football modelling Journal of Quantitative Analysis in Sports.
  * Schellhorn, P., et al. (2020) Schellhorn, P., et al. (2020) Return to sports after COVID-19 infection Eur Heart J.
  * Sports anomalies (2021) Sports anomalies (2021) https://github.com/kai-trading-bot/sports_anomalies
  * SportsBookReviewsOnline (2021) Sports Book Reviews Online (2021) https://sportsbookreviewsonline.com/
  * Verwoert, G.C., et al. (2020) Verwoert, G.C., et al. (2020) Return to sports after COVID-19: a position paper from the Dutch Sports Cardiology Section of the Netherlands Society of Cardiology Netherlands Heart Journal 28:391-395
  * Woodland, L.M., Woodland, B.M. (1994) Woodland, L.M., Woodland, B.M. (1994) Market Efficiency and the Favorite-Longshot Bias: The Baseball Betting Market The Journal of Finance 49:260-279.
  * Wong, A.Y.Y., et al. (2020) Wong, A.Y.Y., et al. (2020) Impact of the COVID-19 pandemic on sports and exercise Asia-Pacific Journal of Sports Medicine, Arthroscopy, Rehabilitation and Technology 22:39-44.
  * Wilcoxon, Frank (1945) Wilcoxon, Frank. ”Individual comparisons by ranking methods.” _Biometrics Bulletin_. 1 (6): 80–83.
  * Highkin, S. (2020) How Much Respect Does Lakers’ NBA Bubble Title Deserve? (2020) https://bleacherreport.com/articles/2913154-how-much-respect-does-lakers-nba-bubble-title-deserve
  * Baer, J. (2020) Pat Riley on Lakers’ NBA Finals win over Heat: ’There will be always be that asterisk’ (2020) https://sports.yahoo.com/pat-riley-lakers-nba-finals-win-heat-asterisk-010021842.html
  * Leitner, W. (2021) Lakers’ 2020 Championship in ’Bubble’ Was Most Fraudulent in NBA History (2021) https://foxsportsradio.iheart.com/content/2021-06-05-lakers-2020-championship-in-bubble-was-most-fraudulent-in-nba-history/
  * Holdahl, C. (2020) IT’S OFFICIAL: NBA PLANS TO RESTART SEASON IN JULY AT DISNEY WORLD ORLANDO (2020) https://www.nba.com/blazers/forwardcenter/its-official-nba-plans-restart-season-july-disney-world-orlando
  * Feinsand, M. (2020) Play Ball: MLB announces 2020 regular season (2020) https://www.mlb.com/news/mlb-announces-2020-regular-season
  * Battista, J. (2021) How the NFL Navigated through a Pandemic and Made it to the Finish Line of a Historic Season (2021) https://www.nfl.com/news/sidelines/2020-in-review-inside-the-most-unique-season-in-nfl-history
  * NHL (2020) NHL season to start Jan. 13, play 56 games in agreement with NHLPA (2020) https://www.nhl.com/news/nhl-nhlpa-season-start-agreement/c-319839598
  * Bender, B. (2020) College football schedule 2020: Complete list of games for all 76 FBS schools, week by week (2020) https://www.sportingnews.com/us/ncaa-football/news/college-football-schedule-2020-week-by-week/1jtsaefcofke31u09wpfenuieq
  * Higgins, L. (2021) A College Basketball Season in Which One in 10 Games Vanished (2021) https://www.wsj.com/articles/college-basketball-season-ncaa-tournament-bubble-11614917053
  * Holm, S. (1979) Holm, S. (1979) A simple sequentially rejective multiple test procedure Scandinavian Journal of Statistics 6:65-70.
  * Arrow, K. J. (1965) Arrow, K. J. (1965). Aspects of the Theory of Risk Bearing Essays in the Theory of Risk Bearing 90-109
  * AGA (2021) AGA Survey of the Commercial Casino Industry https://www.americangaming.org/wp-content/uploads/2021/05/AGA-2021-State-of-the-States_FINALweb-150ppi.pdf 64-66
  * Pratt, John W. (1964) Pratt, John W. (1964) Risk Aversion in the Small and in the Large Econometrica 32:122–136
  * Wilcoxon, Frank (1945) Wilcoxon, Frank (1945). ”Individual comparisons by ranking methods” Biometrics Bulletin 1 (6): 80–83
  * Sharpe, W. F. (1994) Sharpe, William F. (1994). ”The Sharpe Ratio” The Journal of Portfolio Management 21 (1): 49–58

The authors thank Professor Toby Moskowitz for helpful discussions about this
topic.


HVSHVSHuman Visual System ARARAugmented Reality VRVRVirtual Reality
SLMSLMSpatial Light Modulator FoVFoVField Of View HOEHOEHolographic Optical
Element 3D3DThree-Dimensional CNNCNNConvolutional Neural Network PSFPSFPoint-
Spread Function MLPMLPMultilayer Perceptron CBAMCBAMConvolutional Block
Attention Module FPNFPNFeature Pyramid Network MDEMDEMonocular Depth
Estimation PSPPSPPyramid Spatial Pooling BCPBCPBin Center Predictor
CGHCGHComputer-Generated Holography SASASegment Anything HDRHDRHigh Dynamic
Range LRLRLearning Rate TVTVTotal Variation SILogSILogScale Invariant Log
MTLMTLMulti-task Learning ASMASMAngular Spectrum Method ViTViTVision
Transformer 2D2DTwo-Dimensional FPSFPSFrames Per Second DPDPDouble Phase
GMGMGradient Matching KDKDKnowledge Distillation SAMSAMSkip Attention Module

#  Configurable Learned Holography
Figure 1: Configurable Learned Holography. Our method presents a single-stage
end-to-end learned approach for predicting 3D holograms and their
corresponding laser powers from RGB-only 2D images. Our method supports
various optical configurations using existing Spatial Light Modulators (SLMs)
on the market without requiring retraining. Unique to our case in the
literature, we eliminate the need for an additional stage for input depth
estimation by simultaneously predicting depth and holograms in the same
learned method. With the help of a knowledge-distillation technique, our final
model can estimate hologram at interactive rates. We experimentally
demonstrate 3D image reconstructions using two holographic display prototypes
with different optical configurations to showcase our method in practice
(Image Source: Creative Commons [Spragg(2009)]).

Yicheng Zhan  University College London, Gower Street, London, UK  Liang Shi
Massachusetts Institute of Technology, Massachusetts, USA  Wojciech Matusik
Massachusetts Institute of Technology, Massachusetts, USA  Qi Sun  New York
University, New York, USA  Kaan Akşit  University College London, Gower
Street, London, UK

###### Abstract

In the pursuit of advancing holographic display technology, we face a unique
yet persistent roadblock: the inflexibility of learned holography in adapting
to various hardware configurations. This is due to the variances in the
complex optical components and system settings in existing holographic
displays. Although the emerging learned approaches have enabled rapid and
high-quality hologram generation, any alteration in display hardware still
requires a retraining of the model.

Our work introduces a configurable learned model that interactively computes
3D holograms from RGB-only 2D images for a variety of holographic displays.
The model can be conditioned to predefined hardware parameters of existing
holographic displays such as working wavelengths, pixel pitch, propagation
distance, and peak brightness without having to retrain. In addition, our
model accommodates various hologram types, including conventional single-color
and emerging multi-color holograms that simultaneously use multiple color
primaries in holographic displays. Notably, we enabled our hologram
computations to rely on identifying the correlation between depth estimation
and 3D hologram synthesis tasks within the learning domain for the first time
in the literature. We employ knowledge distillation via a student-teacher
learning strategy to streamline our model for interactive performance.
Achieving up to a 2x speed improvement compared to state-of-the-art models
while consistently generating high-quality 3D holograms with different
hardware configurations.

Keywords: deep learning, neural networks, depth estimation, hard-parameter
sharing, computer-generated holography

|  Input |  Speed | Hologram |  SLM
Refresh
Rate |  Stage |  3D |  Depth
Accuracy |  Learn |  Max Z
(mm) |  VD (mm) |  Con
figur
able
---|---|---|---|---|---|---|---|---|---|---|---
Type |  PD |  DP
Our Method |  RGB-only |  Fast |  M+S |  8 |  No |  60 Hz |  Single |  True |  Moderate |  Full |  ~10.0 |  ~4.0 |  Yes
HoloBeam[Akşit and Itoh(2023)] |  G-only |  Fast |  S |  8 |  Yes |  60 Hz |  Single |  True |  Inaccurate |  Full |  ~0.0 |  ~6.0 |  No
Multi-DNN [Ishii et al.(2023)] |  RGB-only |  Slow |  S |  8 |  Yes |  60 Hz |  Three |  True |  Moderate |  Full |  ~50.0 |  ~2.0 |  No
NH [Peng et al.(2020)] |  RGB-only |  Fast |  S |  8 |  No |  60 Hz |  Two |  False |  Accurate |  Full |  ~100.0 |  ~12.5 |  No
NH3D [Choi et al.(2021a)] |  RGB-D |  Slow |  S |  8 |  No |  60 Hz |  Two |  True |  Accurate |  Semi |  ~8.2 |  ~4.4 |  No
TensorV2 [Shi et al.(2022)] |  RGB-D |  Fast |  S |  8 |  Yes |  60 Hz |  Two |  True |  Accurate |  Full |  ~12.0 |  ~6.0 |  No
DGE-CNN [Liu et al.(2023a)] |  RGB-D |  Slow |  S |  8 |  No |  58 Hz |  Two |  True |  Moderate |  Semi |  ~10.0 |  ~30.0 |  No
4K-DMDNet [Liu et al.(2023b)] |  RGB-D |  Slow |  S |  8 |  No |  58 Hz |  Two |  False |  Accurate |  Full |  ~300.0 |  ~40.0 |  No
Time-multiplexed [Choi et al.(2022a)] |  RGB-D |  Slow |  S |  4 |  No |  480 Hz |  Two |  True |  Accurate |  Semi |  ~79.0 |  ~12.0 |  No
Table 1:  Comparison of hologram estimation methods. Our method can generate
both single and multi-color holograms with accurate depth, RGB-only input and
preferred optical configurations (Configurable). In Speed, Fast refers to
$>10$ FPS, and Slow refers to $<10$ FPS. In Type, M refers to multi-color
holograms and S refers to single-color holograms. In Learn, Full refers to the
full learning-based method, and Semi refers to the semi learning-based method.
The label PD refers to pixel depth. The label DP refers to double phase
encoding [Hsueh and Sawchuk(1978)]. The label Z refers to propagation
distance. The label VD refers to volume depth. SLM refers to Spatial Light
Modulator.

##  1 Introduction

The technology offers immersive 3D visuals, paving the way for future
advancements in display technologies. Despite its promising potential,
holographic displays require complex computation [Blinder et al.(2019)], and
more critically, hardware configurations such as optical component models and
spatial placement. The recently emerging deep learning approaches have been
advanced to address the computation bottleneck with real-time and high-quality
hologram generation in both 2D and 3D [Shi et al.(2022)]. However, the models
only accommodate a given preset optical configuration including working
wavelengths, pixel pitch, propagation distance, and peak brightness. If
applying the model to another end user’s display, they are not applicable and
the whole system requires heavy retraining.

Our work focuses on generating Three-Dimensional (3D) holographic images in
the variety of existing holographic displays. To achieve this goal, we
introduce an end-to-end single-stage learned model capable of efficiently
generating conventional single-color and emerging multi-color [Kavaklı et
al.(2023b), Markley et al.(2023)] holograms at interactive rates. At its
heart, our method identifies the correlation between depth estimation and
hologram estimation tasks for the first time in the literature. The model
adopts a strategy of hard-parameter sharing [Caruana(1993)] to yield high-
quality hologram predictions straight from RGB-only input by predicting depth
as a subtask. Our model can be conditioned for various optical configurations,
including working wavelengths, pixel pitch, and peak brightness conditions,
enabling a configurable approach compatible with different holographic display
design choices. Furthermore, we use knowledge distillation through student-
teacher learning strategy [Wang and Yoon(2021)] to unlock a learned model with
lesser computational expense, supporting interactive rates. Our work
introduces the following contributions:

  * •

Configurable Hologram Estimation Model. We introduce a configurable hologram
estimation model supporting a range of optical configurations from existing
holographic displays and hologram types, including single and multi-color
holograms.

  * •

Single-stage Hologram Estimation from RGB-only inputs. Unlike previous methods
with a dedicated depth estimation stage, our learned model leverages hard-
parameter sharing to convert RGB-only images to holograms in a single stage,
eliminating the need for an additional depth estimation step. This is the
first time that the correlation between depth estimation and hologram
estimation tasks has been identified in literature.

  * •

Advancing Holograms towards Interactive Rates. We use knowledge distillation
through a student-teacher learning strategy, we speed up our for interactive
rate hologram estimation.

  * •

Evaluations. We evaluate the image quality and speed of our model through an
extensive study comparing against state-of-the-art hologram estimation methods
and their potential future derivatives. We also verify our findings in two
holographic display prototypes, demonstrating comparable image quality with
the state-of-the-art in literature.

In pursuing our primary objective, we aspire for this research to serve as the
first step in the development of self-configurable holographic displays, where
future displays are envisioned to autonomously generate high-quality images,
effectively adapting to varying optical configurations without compromising
image quality.

##  2 Related Work

Our work deals with learned methods from inputs, and leverages multi-task
learning using hard-parameter sharing. Here, we review relevant literature
accordingly.

##### Learned Computer-Generated Holography.

Early works on conventional single-color hologram estimation utilize U-Net-
based structures [Siddique et al.(2021)]. These methods require RGB-D inputs
and are designed for specific optical configurations. A recent study by Shi et
al. [Shi et al.(2022)] propose a stack of residual blocks without resolution
reduction to speed up the estimation process for conventional single-color
holograms. However, these methods [Shi et al.(2022)] still rely on RGB-D
inputs. Liu et al. [Liu et al.(2023a)] propose a two-stage solution for
removing the dependency on depth information in the input (RGB-only). The
first stage involves depth estimation, while the second stage optimizes
conventional single-color holograms using RGB-only inputs. Akşit and Itoh
[Akşit and Itoh(2023)] merge these two stages into a single stage while using
RGB-only inputs. However, their estimated single-color holograms [Akşit and
Itoh(2023)] provide inaccurate and limited depth in their reconstructed
images. Ishii et al. [Liu et al.(2023a)] propose a triple-stage solution where
the first estimates depth, the second generates holograms using RGB-D input
and the third optimizes the quality of the holograms. Our review suggests that
the entirety of works in the literature are designed to work with a fixed
preset optical configurations, typically requires RGB-D inputs and designed
for single-color holograms. We provide a summary comparison of our method
against the literature in Tbl. 1. Our work offers the first learned model to
support various hologram typles, RGB-only inputs and a wide set of optical
configurations. Currently, our method does not support heavily quantized
holograms with 4-bit pixel depth [Choi et al.(2022b)], but only supports
conventional single-color and emerging multi-color holograms with 8-bit pixel
depth.

##### Multi-task Learning.

The work by Zhang and Yang [Zhang and Yang(2021)] introduces a comprehensive
review of . Our work leverages [Caruana(1993)] to solve depth inaccuracies in
previous works with RGB-only inputs [Akşit and Itoh(2023)], jointly learning
input depths, holograms, and light source intensities using shared weights and
biases. A common strategy in is to share layers among all tasks (hard-
parameter sharing) while keeping output layers or several input layers (heads)
task-specific [Ruder et al.(2019), Sarwar et al.(2019)]. Similarly, our work
follows hard-parameter sharing while keeping only output heads task-specific.
This approach is separate from soft-parameter sharing [Duong et al.(2015)],
where each task has an independent , but other tasks constrain the weights of
each . To our knowledge, our work is the first to identify that jointly
learning subtasks of depth and hologram estimation can help estimate depth-
accurate holograms when working with RGB-only inputs.

##### Knowledge Distillation.

[Hinton et al.(2015)] transfers knowledge from large models to introduce
compact and lightweight models. is a powerful technique for reducing the
computational costs of models in various tasks, including semantic
segmentation [Liu et al.(2019)] and depth estimation [Pilzer et al.(2019)]. As
mentioned in a survey by Gou et al. [Gou et al.(2021)], methods follow
teacher-student architecture. A teacher model can transfer knowledge to a
student model either by regularizing the output responses (response-based)
[Micaelli and Storkey(2019)] or layer responses (feature-based) [Chen et
al.(2020)] between the two models. In the past, has also been applied in [Li
and Bilen(2020)]. Following the findings from the work by Ignatov et al.
[Ignatov et al.(2021)], our work transfers knowledge from a teacher model
based on to a student model through pixel-wise distillation (response-based).
Our work is the first model to experiment with for accelerating the hologram
estimation process in learned holography methods.

Figure 2: The overview of the teacher model. Based on the standard U-Net
[Ronneberger et al.(2015)] structure, a [Lin et al.(2016)] is connected to
every stage of our U-Net’s decoder to leverage spatial information at varying
scales from an input RGB-only input. Our method uses [Woo et al.(2018)] to
introduce both channel and spatial attention mechanisms, while each decoder is
conditioned with optical characteristics, including peak brightness,
wavelength, volume depth, location offset and pixel pitch. A Pyramid Spatial
Pooling layer [He et al.(2014)] is integrated to enhance the model’s ability
to aggregate spatial context for depth estimation. Our light head layer
predicts required peak brightnesses. Our model uses two phase heads dedicated
to long and short location offsets. Here, the $PSP$ represents the pyramid
spatial pooling layer [He et al.(2014)], $BCP$ represents the [Agarwal and
Arora(2022)] (RGB-only input: Creative Commons [Bujar I Gashi 2012]).

##  3 Interactive 3D Holograms from 2D images

Our method estimates phase-only holograms that reconstruct scenes from RGB-
only inputs while supporting single-color and multi-color hologram types.
Here, we revisit the hologram generation formulation to introduce our method.

###  3.1 Synthesizing Single-color and Multi-color Holograms

Holographic displays play successive holograms rapidly to generate full-color
images. In this process, the color primaries of holographic displays
illuminate these holograms in accordance with their hologram type. The fuses
the light from these illuminated holograms into full-color images. Suppose
only a single-color primary illuminates a specific hologram, which is
calculated for the particular wavelength of that single-color primary. In that
case, this hologram type is single-color. In contrast, if multiple color
primaries illuminate a specific hologram, and the hologram is calculated for
all the wavelengths provided by these multiple color primaries, then this
hologram is a multi-color hologram. Regardless of the hologram type, we could
frame the hologram synthesis problem with the following minimization problem,

$\hat{u}_{t},\hat{l}_{(p,t)}\leftarrow\operatorname*{argmin}_{u_{t},l_{(p,t)}}\underbrace{\sum_{p=1}^{P}\left\lVert\left(\sum_{t=1}^{T}\left\lvert
l_{(p,t)}e^{i\frac{\lambda_{p}}{\lambda_{p_{\text{anchor}}}}u_{t}}*h_{p}\right\rvert^{2}\right)-sI_{p}\right\rVert_{2}^{2}}_{L_{\text{image}}},$
(1)

where $P$ denotes the total number of color primaries (i.e. typically three in
existing display technology), $p$ denotes the index of a color primary, $T$
denotes the total number of subframes reproducing a full-color image (i.e.
typically three in holographic displays), $t$ denotes the index of a subframe,
$l_{(t,p)}$ represents the peak brightness for the $p$-th primary at the
$t$-th subframe, $\lambda_{p}$ denotes the wavelength of the active primary,
$\lambda_{p_{\text{anchor}}}$ denotes the wavelength of the anchor primary,
for which the nominal value of the SLM phase is calibrated against, $u_{t}$
represents the phase-only hologram (a subframe), $s$ represents the brightness
scale, $h_{p}$ denotes the wavelength and location, $z$, dependent light
transport kernel [Matsushima and Shimobaba(2009), Kavaklı et al.(2022), Choi
et al.(2021b)], $I_{p}$ represents a parameter to scale the brightness of a
target image intensity, and $L_{\text{image}}$ represents the visual
difference between the intended target image for the given plane and
reconstructed image by a hologram. Note that $l_{(p,t)}$ is by default an
identity matrix in the single-color holograms, where only one color primary
illuminates a subframe ($p=t$). Recent studies [Chao et al.(2023)] introduce
improvements in brightness levels in single-color holograms. But the primary
benefit of multi-color holograms concerning single-color is their support for
a broader range of $s$. Thus, these multi-color holograms can support brighter
images in holographic displays [Kavaklı et al.(2023b), Markley et al.(2023)]
without requiring higher power light sources. We develop our learned model to
support these two hologram types and their existing and potential benefits.

###  3.2 Estimating Single-color and Multi-color Holograms

Our final learned model is a student model that we refine from a teacher model
using a knowledge distillation technique. Here, we introduce our teacher model
and our fast student model.

####  3.2.1 Teacher Model

We provide a complete layout of our teacher model in Fig. 2. Our teacher model
takes an RGB-only image, $I_{input}$, color primaries, $\lambda_{p}$, physical
pixel pitch size, $d_{x}$, brightness scale, $s$, volume depth along
propagation direction, $VD$, location offset, $Z$, as inputs in our training.
Here, $Z$ refers to the propagation distance of the plane in the middle of
$VD$. During our training, we vary some of these inputs for a given set of
values so that our final trained model can condition the estimation process
for preferred optical configurations at the test time. Specifically, our
training draws the following variables from the below set in Eq. (2),

$\begin{split}\lambda_{p}\subseteq\\{(639,515,473)\\}~{}nm,\\\
s\subseteq\\{1.0,1.4,1.8\\},VD\subseteq\\{4.0\\}~{}mm,\\\
Z\subseteq\\{2,10\\}~{}mm,d_{x}\subseteq\\{3.74\\}~{}\mu m.\end{split}$ (2)

We generate permutations of our set in Eq. (2). These pairings are unique and
don’t interchange in permutations. When training our teacher model, we train
it for $I_{input}$, for a fixed setting chosen from permutations of our set in
Eq. 2. Specifically, our configuration choice is as follows:

$\begin{split}\lambda_{p}=(639,515,473)~{}nm,s=1.0\\\
VD=4.0~{}mm,Z=10~{}mm,d_{x}=3.74~{}\mu m.\end{split}$ (3)

As our teacher model is trained for the chosen fixed setting in Eq. (3), we
further our training for $I_{input}$, using entire set of our permutations in
one additional training session. In our teacher model as provided in Fig. 2, a
U-Net structure [Ronneberger et al.(2015)] processes $I_{input}$. The encoder
of our U-Net, $Enc$, is the EfficientNet 1B111Source code adopted from
GitHub:qubvel/segmentation_modelsṗytorch#6db76a1. from the work by Tan and Le
[Tan and Le(2019)]. EfficientNet [Tan and Le(2019)] is performant and memory-
saving concerning other classical encoders (e.g. ResNet), which also guided
our choice in design. In the decoder of our U-Net, $Dec$, the input from the
previous layer goes into a transpose convolution layer, and the processed
input is then concatenated with the input from the skip connection. The later
stage in our decoder involves a [Woo et al.(2018)]222Source code adopted from
GitHub:Jongchan/attention-module#459efad. Note that is an attention-block that
processes local information such as channel and spatial information, which we
found necessary in our trials as it allowed us to estimate depth and
reconstruct 3D images with our estimated holograms more accurately. The last
stage of our decoder conditions our U-Net with $\lambda,s,d_{x}$.
Specifically, we condition the variables using a that includes two consecutive
fully connected layers. The conditioning enables our teacher model to estimate
holograms for various optical configurations. We use features harvested from
every level of our U-Net’s decoder and process it following the work by Lin et
al. [Lin et al.(2016)]. The features from various levels of our U-Net are
processed with convolutional layers with batch normalization and a non-linear
activation function of ReLU. Later, we upsample these features using bilinear
filters. Finally, we process these upsampled features further using another
set of convolutional layers with batch normalization and sum them all to form
our latent code.

We provide the latent code of our teacher model to “heads” and dedicated to
resolving three tasks, including predicting a phase-only hologram, peak
brightness, and depth map of a given input image. We regularize the output of
all tasks by using the following penalization term,

$\mathcal{L}_{train}=\alpha_{0}\mathcal{L}_{recon}+\alpha_{1}\mathcal{L}_{light}+\alpha_{2}\mathcal{L}_{depth},$
(4)

where $\alpha$s represents a set of hyperparameters
($\alpha_{0}=1,\alpha_{1}=1,\alpha_{2}=30$), $\mathcal{L}_{recon}$ represents
multiplane image reconstruction loss as introduced by Kavaklı et al. [Kavaklı
et al.(2023a)], $\mathcal{L}_{light}$ represents light power constrain as
introduced by Kavaklı et al. [Kavaklı et al.(2023b)], and
$\mathcal{L}_{depth}$ represents the depth estimation. In our implementation,
we also compare the color performance of reconstructed images using
regularizations from Andersson et al. [Andersson et al.(2020)]. This
modification to $\mathcal{L}_{recon}$ is an addition to the original work by
Kavakli et al. [Kavaklı et al.(2023a)] and helps improve color consistency in
our reconstructions. Please refer to the supplementary materials for detailed
information on specific aspects of our complete $\mathcal{L}_{train}$
implementation.

##### Phase-only hologram estimation task.

Our model has two branches for the phase-only hologram estimation task, each
dedicated to a different range of $Z$. Specifically, if $Z$ is above $4~{}mm$,
we load the long propagation weight $w\\_long$ to phase head and ASM CNN
denoted in Fig. 2, otherwise, we load the short propagation weight
$w\\_short$. The phase head consists of an identical convolutional layer. The
output of the phase head provide pseudo amplitude and phase values, which are
later converted into a complex-valued field. This complex-valued field is
propagated using band-limited method using the same location offset $Z$ and
wavelength $\lambda$ over all subframes. We later retrieve the phase from the
complex-valued field as our phase-only hologram. As the last step, the phase
will be fed into an additional ASM CNN, Note that as an extra step, when
predicting long propagation distance hologram, the phase prediction will be
fed into an additional ASM CNN, which consists of a convolution layer,
followed by three consecutive convolution layers, batch normalization, and a
nonlinear activation function ReLU. The end of the ASM CNN is a convolution
layer. The output from this layer is combined with the previous phase from the
complex field, culminating in the final phase output.

##### Light power estimation task.

The light head takes the latent code from the our teacher model as the input.
In this head, the latent code will first be processed by a convolution layer
and consecutive convolution layers with batch normalization and a sigmoid
activation function. Next, the output from the previous layer will be fed into
an adaptive pooling layer to aggregate the spatial information. Finally, the
pooled output will be processed by two linear layers followed by a sigmoid
activation function to make sure the shape of the prediction is $t$ x $p$
($3\times 3$) and the data range is between 0 and 1.

##### Depth estimation task.

Our teacher model includes a dedicated depth head for this task. We feed the
latent output of our U-Net and the bin center predictor’s output to our depth
head as shown in Fig. 2. The bin center predictor cascades two modules: [He et
al.(2014)] and [Agarwal and Arora(2022)]. 333Source code adopted from
GitHub:ashutosh1807/PixelFormer#bb0ae9f.. The bin center predictor estimates
the distribution of depth values across different bins in the depth map,
aiding precise depth prediction. We feed our U-Net’s last encoder layer output
(Enc 4) into within the bin center predictor. In the depth head, the model’s
latent code first passes through a depthwise convolution layer and a Softmax
activation function. Then, we multiply the softmax layer’s output by the bin
center prediction to generate the final depth prediction. We apply
regularization to the depth estimations from the depth head using a loss
function, $\mathcal{L}_{depth}$, the sum of three sub-loss functions without
any weights: $\mathcal{L}_{silog}$, $\mathcal{L}_{gm}$, and
$\mathcal{L}_{tv}$. The scale-invariant loss, $\mathcal{L}_{silog}$, is
introduced by eigen et al. in [Eigen et al.(2014)]. The gradient matching
loss, $\mathcal{L}_{gm}$, compares the edges of estimated depths with ground
truth depth maps. The total variant loss, $\mathcal{L}_{tv}$, smoothes the
edges of objects in the depth map. For more information on implementing
$\mathcal{L}_{depth}=\mathcal{L}_{silog}+\mathcal{L}_{gm}+\mathcal{L}_{tv}$,
refer to our supplementary materials.

####  3.2.2 Student Model.

Our teacher model can be enhanced by transforming it into a simpler student
model through the knowledge distillation technique [Wang and Yoon(2021)],
which allows for more rapid estimation of 3D phase-only holograms. Firstly, we
present a comprehensive layout of our student model in Fig. 3. Similar to the
teacher model, the student model also takes $I_{input}$, $\lambda_{p}$,
$d_{x}$, $s$, $VD$, and $Z$ as inputs. Furthermore, our student model employs
the U-Net structure to process $I_{input}$, but with a different encoder -
MobileNetV3 [Howard et al.(2019)] 444Source code adopted from
GitHub:qubvel/segmentation_modelsṗytorch#6db76a1.. MobileNetV3 serves as a
more compact alternative to the teacher model’s encoder, often used for
lightweight models due to its efficiency. In our student decoders, similar to
the teacher model, the input from the previous layer also goes into a
transpose convolution layer, and the processed input is then concatenated with
the input from the skip connection. Concatenated information is processed by a
consecutive depthwise separable convolution layer with batch normalization and
a nonlinear ReLU activation function. The last stage of our student decoder
conditions our U-Net with $\lambda,s,d_{x}$ as well. The structure of the
linear condition layers remains the same as the teacher model. The structures
of the prediction heads in the student model are also the same as the teacher
model.

Figure 3: The overview of the student model.

In training of our student model, the logit based [Hinton et al.(2015)] is
applied to balance the model performance and the inference speed. The original
knowledge distillation loss $\mathcal{L}_{KD}$, is defined as:

$\mathcal{L}_{KD}=T^{2}\cdot D_{KL}(S(y_{student}/T),S(y_{teacher}/T)).$ (5)

Here, $D_{KL}$ represents the Kullback-Leibler divergence [Kullback(1951)],
$y_{student}$ and $y_{teacher}$ are the phase and depth predictions of the
student and teacher models, and $S$ denotes the softmax function. The
temperature parameter $T$ controls the softness of probabilities. The scalar
factor $T^{2}$ is used to scale the loss back to the correct magnitude.
However, $\mathcal{L}_{KD}$ only captures the overall data distribution, the
information of the edge and the texture of the depth map will be missing. To
have a robust and accurate depth distillation, we combined the
$\mathcal{L}_{KD_{depth}}$ with the depth estimation loss
$\mathcal{L}_{depth}$. The target of the $\mathcal{L}_{depth}$ will be depth
map prediction from the teacher model instead of the ground truth depth.
Additionally, since the student model’s capacity is much smaller than the
teacher model, we found that during training, the student model finds it hard
to maintain a smooth phase prediction. To resolve this issue, we involved
Charboninner loss [Barron(2017)] as our pixel-wise distillation loss of phase
prediction between the student and teacher model. The updated distillation
loss $\mathcal{L}_{distill}$ will be defined as
$\mathcal{L}_{distill}=\mathcal{L}_{KD_{phase}}+\mathcal{L}_{charboninner}+\mathcal{L}_{KD_{depth}}+\mathcal{L}_{depth}$.
The final loss that combines the original training loss and the knowledge
distillation loss is defined as:

$\mathcal{L}=\mathcal{L}_{distill}+\mathcal{L}_{train}.$ (6)

###  3.3 Implementation

We used the dataset SA-1B, omitting the masks, in our training sessions for
teacher and student model. Furtermore, we employed the MiDaS [Lasinger et
al.(2019)] for generating ground truth depth for these images. Due to
limitations in computational resources and time, our usage was confined to a
subset of the dataset, specifically 30,000 images, which represents
approximately $0.3\%$ of the entire dataset. We implemented our method using
Pytorch [Paszke et al.(2019)]. We use Adam [Kingma and Ba(2015)] optimizer
($\beta_{1}=0.9,\beta_{2}=0.99$), with batch size 16 and CosineAnnealingLR
[Loshchilov and Hutter(2016)] as the scheduler. The teacher model and student
model were trained individually under the same training setting. Specifically,
the teacher model was trained with 30K images at the resolution of 768 x 768,
while the student model utilized a smaller set of 20,000 images at the
resolution of 1024 x 1024. Both models are trained for 20 epochs with an
initial of 0.001, which was progressively reduced following a cosine decay
function. All experiments were conducted on eight NVIDIA RTX 3090 24G GPUs,
and no data augmentation techniques were employed during the training process.
The total time of training teacher model and distilling student model roughly
takes 4.5 days. Training our teacher model for a wide variety of optical
configurations could easily exhaust computational resources. To prove the
validity of our approach, in parallel we also derive a version of our teacher
model that works with RGB-D inputs where depth prediction task is not
involved. we trained the model with another variable set containing various
volume depths, wavelengths, and pixel pitches. The result is available in
supplementary materials.

Method |  Input |  Optical Configuration |  PSNR↑ (dB) |  SSIM↑ |  LPIPS↓ |  FLIP↓ |  Para
meters |  Speed
---|---|---|---|---|---|---|---|---
Our Method (teacher) |  RGB-only |  $d_{x}$: 3.74 µm, Z: 2 mm, VD: 4 mm, x1.0 |  30.45$\pm$2.43 |  0.95$\pm$0.03 |  0.35 |  0.09 |  10.74 M |  769 ms
$d_{x}$: 3.74 µm, Z: 2 mm, VD: 4 mm, x1.4 |  24.66$\pm$1.80 |  0.90$\pm$0.04 |  0.47 |  0.26
$d_{x}$: 3.74 µm, Z: 2 mm, VD: 4 mm, x1.8 |  29.19$\pm$2.37 |  0.91$\pm$0.04 |  0.38 |  0.12
$d_{x}$: 3.74 µm, Z: 10 mm, VD: 4 mm, x1.0 |  23.72$\pm$1.37 |  0.90$\pm$0.04 |  0.60 |  0.27
$d_{x}$: 3.74 µm, Z: 10 mm, VD: 4 mm, x1.4 |  21.75$\pm$1.34 |  0.83$\pm$0.04 |  0.73 |  0.34
$d_{x}$: 3.74 µm, Z: 10 mm, VD: 4 mm, x1.8 |  21.39$\pm$1.28 |  0.84$\pm$0.03 |  0.71 |  0.36
Our Method (student) |  RGB-only |  $d_{x}$: 3.74 µm, Z: 2 mm, VD: 4 mm, x1.0 |  29.89$\pm$2.80 |  0.96$\pm$0.03 |  0.35 |  0.08 |  2.18 M |  42 ms
$d_{x}$: 3.74 µm, Z: 2 mm, VD: 4 mm, x1.4 |  30.14$\pm$2.71 |  0.96$\pm$0.04 |  0.36 |  0.09
$d_{x}$: 3.74 µm, Z: 2 mm, VD: 4 mm, x1.8 |  29.01$\pm$2.64 |  0.94$\pm$0.05 |  0.38 |  0.11
$d_{x}$: 3.74 µm, Z: 10 mm, VD: 4 mm, x1.0 |  25.87$\pm$1.60 |  0.87$\pm$0.04 |  0.46 |  0.17
$d_{x}$: 3.74 µm, Z: 10 mm, VD: 4 mm, x1.4 |  25.49$\pm$1.62 |  0.88$\pm$0.04 |  0.44 |  0.17
$d_{x}$: 3.74 µm, Z: 10 mm, VD: 4 mm, x1.8 |  24.34$\pm$1.85 |  0.86$\pm$0.04 |  0.47 |  0.18
HoloBeam |  RGB-only* |  $d_{x}$: 3.74 µm, Z: 2 mm, VD: 4 mm, x1.0 |  29.12$\pm$2.70 |  0.93$\pm$0.04 |  0.37 |  0.10 |  1.94 M |  27 ms
$d_{x}$: 3.74 µm, Z: 0 mm, VD: 6 mm, x1.0 |  28.75$\pm$2.84 |  0.93$\pm$0.04 |  0.38 |  0.11
TensorHolo V2 |  RGB-D |  $d_{x}$: 8.0 µm, Z: 0 mm, VD: 6 mm, x1.0 |  29.27$\pm$3.39 |  0.97$\pm$0.03 |  0.29 |  0.09 |  0.04 M |  75 ms
$d_{x}$: 8.0 µm, Z: 12 mm, VD: 6 mm, x1.0 |  22.78$\pm$1.45 |  0.96$\pm$0.03 |  0.31 |  0.09
modified 3D NH** |  RGB-D |  $d_{x}$: 3.74 µm, Z: 2 mm, VD: 4 mm, x1.0 |  29.01$\pm$2.45 |  0.94$\pm$0.03 |  0.40 |  0.11 |  3.87 M |  49 ms
$d_{x}$: 3.74 µm, Z: 100 mm, VD: 12.5 mm, x1.0*** |  28.92$\pm$3.25 |  0.87$\pm$0.04 |  0.47 |  0.13
$d_{x}$: 8.0 µm, Z: 2 mm, VD: 4 mm, x1.0 |  31.15$\pm$2.08 |  0.95$\pm$0.02 |  0.37 |  0.10
$d_{x}$: 8.0 µm, Z: 100 mm, VD: 12.5 mm, x1.0*** |  30.79$\pm$2.98 |  0.90$\pm$0.03 |  0.43 |  0.10
Table 2: Evaluation of various models at different holography settings. The
test resolution is at 1920 x 1080. Parameters refers to the size of the model.
Speed refers to the inference time of the model. Note that Our Method (RGB
teacher), Our Method (RGB student), HoloBeam, TensorHolo V2 and modified 3D NH
used 768 x 768, 1024 x 1024, 1024 x 1024, 384 x 384 and 1024 x 1024 resolution
to train, respectively. *We improve the model from G-only to RGB-only input.
**In our experiment, training modified 3D NH with 1024 x 1024 resolution for
long propagation distance will cause image quality to drop severely. Hence, we
used 1920 x 1080 resolution to train the long propagation version of modified
3D NH. ***Since NH only supports 2D holograms, we modified NH‘s HoloNet [Peng
et al.(2020)] to enable 3D holograms generation. For more details of modified
3D NH, please consult the supplementary materials.

##  4 Evaluation and Discussion

##### Quantitative Analysis

In Tbl. 2, we calculated the mean and standard deviation for PSNR, SSIM, LPIPS
[Zhang et al.(2018)], and FLIP [Andersson et al.(2020)]. The metrics for this
assessment were obtained using 100 images from the DIV2K dataset [Agustsson
and Timofte(2017)]. First, concerning the other models in Tbl. 2, the student
model can generate 3D holograms with different optical configurations without
compromising the image quality. The observation is evidenced by the fact that
the overall PNSR/SSIM only marginally changed $7.9\%$ and $4.4\%$. For LPIPS,
though TensorHolo V2 has the highest value, overall the student model is as
good as the modified 3D NH and HoloBeam. For FLIP, student model achieves the
highest value. Besides the numerical analysis, we also employ pair-wise ANOVA
test as the statistical analysis to evaluate the significance of these
metrics, which is readily available in supplementary. Second, in line with
existing iterative methods [Kavaklı et al.(2023b)], increasing the peak
brightness or propagation distance tend to lower image quality in the student
model. For example, 1.8 brightness induces the decrease of $3.0\%$, $2.1\%$,
$8.6\%$ and $37.5\%$ in PSNR, SSIM, LPIPS and FLIP, respectively. This trend
is consistent across different configurations and methods. Lastly, due to the
absence of feature aggregation components, the student model outperforms the
teacher model by $10.4\%$, $2.7\%$, $20.1\%$ and $36.8\%$ in PSNR, SSIM, LPIPS
and FLIP, respectively, which evidences the effectivness of the student model
design. Although these components are essential in improving depth accuracy,
the pooling and upsample operations adversely affect phase prediction.

##### Qualitative Analysis

We evaluate our method, student model, using simulation results. In addition,
we provide experimental results from two actual holographic display prototypes
in our supplementary. Fig. 4 shows the reconstruction results of student model
at short propagation distance, where reconstructions provides fine details
across various peak brightness levels. Our method demonstrated a high degree
of fidelity in the reconstruction of fine details and texture preservation in
different peak brightnesses. The model managed to retain the structural
integrity and color accuracy of the original images. Fig. 5 shows the
reconstruction results of student model at long propagation distance. Compared
with short propagation distance, the long propagation distance results reveal
a noticeable deviation in color from the ground truth. This deviation is due
to the challenges of long propagation distance tasks, which require a larger
training resolution for the model to fit the high-resolution data and the long
propagation distance. The student model is distilled using data at a
resolution of $1024\times 1024$, which is smaller than the test resolution
1920 x 1080. Due to limitation of computational resources and time, we were
unable to utilize larger images for RGB-only model training. To validate our
point, we used full HD data to train our RGB-D condition model for long
propagation distance and compared with modified 3D NH and TensorHolo V2, the
result is readily available in our supplementary materials.

##### Discussion

Our research facilitates the generation of configurable 3D holograms at
interactive rate. However, the current model can not adapt beyond pre-existing
conditions, it still requires retraining for new optical configurations. Our
ultimate goal is to make the model continuous configurable, meaning the model
can adapt to novel configurations outside of the variable set. Inspired by
Towaki et al. [Takikawa et al.(2023)], a potential approach might involve
training separate models for each single optical configuration. When a new
configuration is required, a selective combination of weights from these
models could be utilized to interpolate and adjust the model’s output to meet
the new conditions. Besides the inability of continuous configurable, our
solution has the following limitations: Extended Training Time. Although
linear condition layers can help the model to understand the difference
between different holographic settings, the number of permutations of the
condition settings increases exponentially when the size of the variables set
increases. This will lead to longer training time. The original variable set
of our paper also contains different wavelengths and pixel pitches, however
having such a large variable set takes weeks to train. The lack of scalability
of . The performance of significantly decreases with the escalation of test
image resolutions. This is due to the model’s training, which is limited to
the current or lower resolution scales, impeding its ability to generalize
patterns at higher resolutions. Consequently, this limitation affects the
accuracy of hologram defocus regions, as depth estimation result directly
influences the focus regions. Testing images at large resolution may thus
erroneously defocus some areas that are meant to be in focus at given depth
plane.

An emerging next-generation display method, holographic display technology has
shown remarkable improvements, particularly in applications akin to glasses
[Jang et al.(2022)]. These advancements require corresponding innovations in
computational methods to fully realize their potential in enhancing user
experiences and facilitating their integration into everyday use.

In conclusion, we have presented the first approach to create multi-color 3D
holograms from RGB-only inputs for various display configurations. Most
importantly, our work unearthed the connection between depth estimation and 3D
hologram generation tasks, a first in this field.

## References

  * [1]
  * [Agarwal and Arora(2022)] Ashutosh Agarwal and Chetan Arora. 2022.  Attention Attention Everywhere: Monocular Depth Prediction with Skip Attention.  arXiv:2210.09071 [cs.CV]
  * [Agustsson and Timofte(2017)] Eirikur Agustsson and Radu Timofte. 2017.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_.
  * [Akşit and Itoh(2023)] Kaan Akşit and Yuta Itoh. 2023.  HoloBeam: Paper-Thin Near-Eye Displays. In _2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)_. IEEE, 581–591.
  * [Andersson et al.(2020)] Pontus Andersson, Jim Nilsson, Tomas Akenine Moller, Magnus Oskarsson, Kalle AAstrom, and Mark D. Fairchild. 2020.  FLIP: A Difference Evaluator for Alternating Images.  _Proceedings of the ACM on Computer Graphics and Interactive Techniques_ 3, 2 (2020), 15:1–15:23.  https://doi.org/10.1145/3406183
  * [Barron(2017)] Jonathan T. Barron. 2017\.  A More General Robust Loss Function.  _CoRR_ abs/1701.03077 (2017).  arXiv:1701.03077 http://arxiv.org/abs/1701.03077
  * [Blinder et al.(2019)] David Blinder, Ayyoub Ahar, Stijn Bettens, Tobias Birnbaum, Athanasia Symeonidou, Heidi Ottevaere, Colas Schretter, and Peter Schelkens. 2019\.  Signal processing challenges for digital holographic video display systems.  _Signal Processing: Image Communication_ 70 (2019), 114–130.
  * [Caruana(1993)] R Caruana. 1993\.  Multitask learning: A knowledge-based source of inductive bias1. In _Proceedings of the Tenth International Conference on Machine Learning_. Citeseer, 41–48.
  * [Chao et al.(2023)] Brian Chao, Manu Gopakumar, Suyeon Choi, and Gordon Wetzstein. 2023.  High-brightness holographic projection.  _Optics Letters_ 48, 15 (2023), 4041–4044.
  * [Chen et al.(2020)] Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. 2020.  Learning student networks via feature embedding.  _IEEE Transactions on Neural Networks and Learning Systems_ 32, 1 (2020), 25–35.
  * [Choi et al.(2022a)] Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, Matthew O’Toole, and Gordon Wetzstein. 2022a.  Time-multiplexed Neural Holography: A Flexible Framework for Holographic Near-eye Displays with Fast Heavily-quantized Spatial Light Modulators. In _SIGGRAPH ’22: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Vancouver, BC, Canada, August 7 - 11, 2022_ , Munkhtsetseg Nandigjav, Niloy J. Mitra, and Aaron Hertzmann (Eds.). ACM, 32:1–32:9.  https://doi.org/10.1145/3528233.3530734
  * [Choi et al.(2022b)] Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, Matthew O’Toole, and Gordon Wetzstein. 2022b.  Time-multiplexed Neural Holography: A Flexible Framework for Holographic Near-eye Displays with Fast Heavily-quantized Spatial Light Modulators. In _SIGGRAPH ’22: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Vancouver, BC, Canada, August 7 - 11, 2022_ , Munkhtsetseg Nandigjav, Niloy J. Mitra, and Aaron Hertzmann (Eds.). ACM, 32:1–32:9.  https://doi.org/10.1145/3528233.3530734
  * [Choi et al.(2021a)] Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, and Gordon Wetzstein. 2021a.  Neural 3D holography: learning accurate wave propagation models for 3D holographic virtual and augmented reality displays.  _ACM Trans. Graph._ 40, 6 (2021), 240:1–240:12.  https://doi.org/10.1145/3478513.3480542
  * [Choi et al.(2021b)] Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, and Gordon Wetzstein. 2021b.  Neural 3D holography: learning accurate wave propagation models for 3D holographic virtual and augmented reality displays.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021), 1–12.
  * [Duong et al.(2015)] Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015\.  Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser. In _Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference on natural language processing (volume 2: short papers)_. 845–850.
  * [Eigen et al.(2014)] David Eigen, Christian Puhrsch, and Rob Fergus. 2014\.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network.  _CoRR_ abs/1406.2283 (2014).  arXiv:1406.2283 http://arxiv.org/abs/1406.2283
  * [Gou et al.(2021)] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021.  Knowledge distillation: A survey.  _International Journal of Computer Vision_ 129 (2021), 1789–1819.
  * [He et al.(2014)] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2014\.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition.  _CoRR_ abs/1406.4729 (2014).  arXiv:1406.4729 http://arxiv.org/abs/1406.4729
  * [Hinton et al.(2015)] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015\.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ (2015).
  * [Howard et al.(2019)] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. 2019\.  Searching for MobileNetV3.  _CoRR_ abs/1905.02244 (2019).  arXiv:1905.02244 http://arxiv.org/abs/1905.02244
  * [Hsueh and Sawchuk(1978)] Chung-Kai Hsueh and Alexander A Sawchuk. 1978.  Computer-generated double-phase holograms.  _Applied optics_ 17, 24 (1978), 3874–3883.
  * [Ignatov et al.(2021)] Andrey Ignatov, Grigory Malivenko, David Plowman, Samarth Shukla, and Radu Timofte. 2021\.  Fast and accurate single-image depth estimation on mobile devices, mobile ai 2021 challenge: Report. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2545–2557.
  * [Ishii et al.(2023)] Yoshiyuki Ishii, Fan Wang, Harutaka Shiomi, Takashi Kakue, Tomoyoshi Ito, and Tomoyoshi Shimobaba. 2023.  Multi-depth hologram generation from two-dimensional images by deep learning.  _Optics and Lasers in Engineering_ 170 (2023), 107758.  https://doi.org/10.1016/j.optlaseng.2023.107758
  * [Jang et al.(2022)] Changwon Jang, Kiseung Bang, Minseok Chae, Byoungho Lee, and Douglas Lanman. 2022.  Waveguide Holography: Towards True 3D Holographic Glasses.  _CoRR_ abs/2211.02784 (2022).  https://doi.org/10.48550/ARXIV.2211.02784 arXiv:2211.02784
  * [Kavaklı et al.(2023a)] Koray Kavaklı, Yuta Itoh, Hakan Urey, and Kaan Akşit. 2023a.  Realistic defocus blur for multiplane computer-generated holography. In _2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)_. IEEE, 418–426.
  * [Kavaklı et al.(2023b)] Koray Kavaklı, Liang Shi, Hakan Urey, Wojciech Matusik, and Kaan Akşit. 2023b.  Multi-color Holograms improve Brightness in Holographic Displays. In _ACM SIGGRAPH ASIA 2023 Conference Proceedings_. –.  https://doi.org/10.1145/3610548.3618135
  * [Kavaklı et al.(2022)] Koray Kavaklı, Hakan Urey, and Kaan Akşit. 2022\.  Learned holographic light transport.  _Applied Optics_ 61, 5 (2022), B50–B55.
  * [Kingma and Ba(2015)] Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).  http://arxiv.org/abs/1412.6980
  * [Kullback(1951)] Solomon Kullback. 1951\.  Kullback-leibler divergence.
  * [Lasinger et al.(2019)] Katrin Lasinger, René Ranftl, Konrad Schindler, and Vladlen Koltun. 2019.  Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer.  _CoRR_ abs/1907.01341 (2019).  arXiv:1907.01341 http://arxiv.org/abs/1907.01341
  * [Li and Bilen(2020)] Wei-Hong Li and Hakan Bilen. 2020.  Knowledge distillation for multi-task learning. In _Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16_. Springer, 163–176.
  * [Lin et al.(2016)] Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. 2016.  Feature Pyramid Networks for Object Detection.  _CoRR_ abs/1612.03144 (2016).  arXiv:1612.03144 http://arxiv.org/abs/1612.03144
  * [Liu et al.(2023b)] Kexuan Liu, Jiachen Wu, Zehao He, and Liangcai Cao. 2023b.  4K-DMDNet: diffraction model-driven network for 4K computer-generated holography.  _OptoElectron Adv_ 6, 5 (2023), 220135–1–220135–13.  https://doi.org/10.29026/oea.2023.220135
  * [Liu et al.(2023a)] Ninghe Liu, Zhengzhong Huang, Zehao He, and Liangcai Cao. 2023a.  DGE-CNN: 2D-to-3D holographic display based on a depth gradient extracting module and ZCNN network.  _Optics Express_ 31, 15 (2023), 23867–23876.
  * [Liu et al.(2019)] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. 2019\.  Structured Knowledge Distillation for Semantic Segmentation. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, 2604–2613.  https://doi.org/10.1109/CVPR.2019.00271
  * [Loshchilov and Hutter(2016)] Ilya Loshchilov and Frank Hutter. 2016.  SGDR: Stochastic Gradient Descent with Restarts.  _CoRR_ abs/1608.03983 (2016).  arXiv:1608.03983 http://arxiv.org/abs/1608.03983
  * [Markley et al.(2023)] Eric Markley, Nathan Matsuda, Florian Schiffers, Oliver Cossairt, and Grace Kuo. 2023\.  Simultaneous Color Computer Generated Holography. In _SIGGRAPH Asia 2023 Conference Papers, SA 2023, Sydney, NSW, Australia, December 12-15, 2023_ , June Kim, Ming C. Lin, and Bernd Bickel (Eds.). ACM, 22:1–22:11.  https://doi.org/10.1145/3610548.3618250
  * [Matsushima and Shimobaba(2009)] Kyoji Matsushima and Tomoyoshi Shimobaba. 2009.  Band-limited angular spectrum method for numerical simulation of free-space propagation in far and near fields.  _Optics express_ 17, 22 (2009), 19662–19673.
  * [Micaelli and Storkey(2019)] Paul Micaelli and Amos J Storkey. 2019.  Zero-shot knowledge transfer via adversarial belief matching.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * [Paszke et al.(2019)] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  _CoRR_ abs/1912.01703 (2019).  arXiv:1912.01703 http://arxiv.org/abs/1912.01703
  * [Peng et al.(2020)] Yifan Peng, Suyeon Choi, Nitish Padmanaban, and Gordon Wetzstein. 2020.  Neural holography with camera-in-the-loop training.  _ACM Transactions on Graphics (TOG)_ 39, 6 (2020), 1–14.
  * [Pilzer et al.(2019)] Andrea Pilzer, Stephane Lathuiliere, Nicu Sebe, and Elisa Ricci. 2019.  Refine and distill: Exploiting cycle-inconsistency and knowledge distillation for unsupervised monocular depth estimation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 9768–9777.
  * [Ronneberger et al.(2015)] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015\.  U-net: Convolutional networks for biomedical image segmentation. In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 234–241.
  * [Ruder et al.(2019)] Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2019.  Latent multi-task architecture learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 4822–4829.
  * [Sarwar et al.(2019)] Syed Shakib Sarwar, Aayush Ankit, and Kaushik Roy. 2019\.  Incremental learning in deep convolutional neural networks using partial network sharing.  _IEEE Access_ 8 (2019), 4615–4628.
  * [Shi et al.(2022)] Liang Shi, Beichen Li, and Wojciech Matusik. 2022.  End-to-end learning of 3d phase-only holograms for holographic display.  _Light: Science & Applications_ 11, 1 (2022), 247\.
  * [Siddique et al.(2021)] Nahian Siddique, Sidike Paheding, Colin P Elkin, and Vijay Devabhaktuni. 2021.  U-net and its variants for medical image segmentation: A review of theory and applications.  _Ieee Access_ 9 (2021), 82031–82057.
  * [Takikawa et al.(2023)] Towaki Takikawa, Thomas Müller, Merlin Nimier-David, Alex Evans, Sanja Fidler, Alec Jacobson, and Alexander Keller. 2023.  Compact Neural Graphics Primitives with Learned Hash Probing. In _SIGGRAPH Asia 2023 Conference Papers, SA 2023, Sydney, NSW, Australia, December 12-15, 2023_ , June Kim, Ming C. Lin, and Bernd Bickel (Eds.). ACM, 120:1–120:10.  https://doi.org/10.1145/3610548.3618167
  * [Tan and Le(2019)] Mingxing Tan and Quoc Le. 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks. In _International conference on machine learning_. PMLR, 6105–6114.
  * [Wang and Yoon(2021)] Lin Wang and Kuk-Jin Yoon. 2021.  Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks.  _IEEE transactions on pattern analysis and machine intelligence_ 44, 6 (2021), 3048–3068.
  * [Woo et al.(2018)] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018.  CBAM: Convolutional Block Attention Module.  _CoRR_ abs/1807.06521 (2018).  arXiv:1807.06521 http://arxiv.org/abs/1807.06521
  * [Zhang et al.(2018)] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.  _CoRR_ abs/1801.03924 (2018).  arXiv:1801.03924 http://arxiv.org/abs/1801.03924
  * [Zhang and Yang(2021)] Yu Zhang and Qiang Yang. 2021.  A survey on multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ 34, 12 (2021), 5586–5609.

Figure 4: The short propagation distance reconstructions comparison of student
model when peak brightness are 1.0, 1.4, and 1.8. The volume depth of the
results is 4mm and the location offset is 2mm (Image Source: Creative Commons
[Eric Hunt 2003]).  Figure 5: The long propagation distance reconstructions
comparison of student model when peak brightness are 1.0, 1.4, and 1.8. The
volume depth of the results is 4mm and the location offset is 10mm (Image
Source: Creative Commons [Eric Hunt 2003]).