figurec [firstpage,color=black!90,angle=0,scale=0.28,
xpos=0in,ypos=-5in]*correspondence<EMAIL_ADDRESS>
# A Rational Model of Dimension-reduced Human Categorization

Yifan Hong  Chen Wang

###### Abstract

Humans tend to categorize objects based on a few key features. We propose a
rational model of categorization that utilizes a mixture of probabilistic
principal component analyzers (mPPCA). This model represents each category
with reduced feature dimensions and allows local features to be shared across
categories to facilitate few-shot learning. Theoretically, we identify the
necessary and sufficient condition for dimension-reduced representation to
outperform full-dimension representation. We then show the superior
performance of mPPCA in predicting human categorization over exemplar and
prototype models in a behavioral experiment. When combined with the
convolutional neural network, the mPPCA classifier with a single principal
component dimension for each category achieves comparable performance to
ResNet with a linear classifier on the CIFAR-10H human categorization dataset.

_K_ eywords Categorization, Computational cognitive science, Bayesian
modeling, Probabilistic PCA

11footnotetext<EMAIL_ADDRESS><EMAIL_ADDRESS>
##  1 Introduction

Categorization is the behavior of grouping stimuli into discrete clusters
(Ashby et al., 2005). How do people determine if a new instance belongs to the
same category as previous observations? It involves developing consistent
expectations beyond what we have observed. Learning about categories also
shapes an individual’s internal feature system (Goldstone, 1994; Schyns &
Rodet, 1997). We assign instances with similar features to the same category,
reflecting their proximity in the psychological space (Shepard, 1987).

Humans can learn new categories with just a few instances (Lake et al., 2015)
and generalize to novel observations (Salakhutdinov et al., 2012; Tiedemann et
al., 2022). However, natural stimuli can have numerous features, and
irrelevant dimensions can harm categorization performance (Vong et al., 2019).
Recent studies suggest that a low-dimensional feature set can accurately
predict human similarity judgment and categorization (Hebart et al., 2020; Jha
et al., 2023). People tend to generalize based on a few meaningful dimensions
such as size and color (Smith, 1989). For example, a child can recognize a
giraffe with simple verbal descriptions.

We propose the mixture of probabilistic principal component analyzers (mPPCA)
model to capture human categorization behavior. The probabilistic PCA (Tipping
& Bishop, 1999) characterizes a category with fewer feature dimensions, and
the hierarchical mixture allows categories to share similar dimensions. The
mPPCA model can learn category prototypes and dimensions from scratch,
creating a parsimonious feature system that supports few-shot learning.

Theoretical analysis shows that excluding a dimension from the category
representation is beneficial if and only if the dimension provides more
information about cross-category variation than within-category variation.
Simulation analysis reveals that a hierarchical prior allows the principal
component dimensions to be shared across categories, leading to human-like
generalization patterns. Moreover, we test the model’s predictability using a
newly designed behavioral experiment and the CIFAR-10H dataset. Specifically,
mPPCA outperforms the exemplar and prototype models in predicting human
categorization in a simple visual task. When combined with the convolutional
neural network, the mPPCA classifier with a single principal component
dimension for each category achieves comparable performance to ResNet with
linear classifier on CIFAR-10H.

Section 2 summarizes previous work. Section 3 introduces the PPCA
representation for each category with theoretical analysis of when dimension-
reduction improves categorization performance, followed by the hierarchical
infinite mixture model in Section 4. We validate the model with simulations in
Section 5 and behavioral experiments and tests on CIFAR-10H in Section 6.
Section 7 presents the discussion and limitation, and section 8 concludes the
paper.

##  2 Related work

We briefly summarize human categorization and generalization theories,
focusing on the computational models.

###  2.1 Models of human category learning

Categorization groups instances based on similar features. Category
representations enable accurate predictions and consistent generalizations.
Cognitive models of categorization make various assumptions about category
representations. For example, the prototype model (Reed, 1972) assumes that
categories can be represented as abstract prototypes. People assign an
instance to the category with the most similar prototype. The exemplar model
(Nosofsky, 1986) considers a category to include all its known members.
Although these classical models have been supported by empirical evidence
(Nosofsky et al., 2018), they are limited to supervised settings and a fixed
number of categories. The rational model of categorization (RMC) (Anderson,
1991) offers a different perspective on human categorization. It postulates
that human categorization results from adapting to the optimal prediction of
features. RMC models categories as probability distributions and performs
Bayesian density estimation. Denote $x_{n}\in\mathbb{R}^{d}$ and $c_{n}$ the
new observation and its category assignment, respectively, and
$\textbf{x}_{n-1}$ and $\textbf{c}_{n-1}$ the set of previous observations and
their category memberships, respectively. The (posterior) predictive
distribution of the features for a new observation is given by

$\begin{split}P(x_{n}|\textbf{x}_{n-1},\textbf{c}_{n-1})=\sum_{k=1}^{K}P(c_{n}=k|\textbf{c}_{n-1})\cdot
P(x_{n}|c_{n}=k,\textbf{x}_{n-1},\textbf{c}_{n-1}).\end{split}$ (1)

The formulation decomposes the prediction task into two parts: the prior bias
towards a particular category and the likelihood of an observation belonging
to that category. The prior can take the form of a Chinese restaurant process
(CRP) (Blackwell & MacQueen, 1973), a sequential process that builds up the
assignments $\textbf{c}_{n-1}$, allows for infinite many categories. The
probability of assigning a sample to an existing category $k$ is proportional
to the number of existing samples $M_{k}$. Meanwhile, a new category emerges
with probability proportional to a concentration parameter $\gamma>0$:

$P(c_{n}=k|\textbf{c}_{n-1})\propto\left\\{\begin{aligned} &M_{k}&&\text{if
}M_{k}>0~{}(k\text{ is old})\\\ &\gamma&&\text{if }M_{k}=0~{}(k\text{ is
new})\\\ \end{aligned}\right.$ (2)

The CRP is the marginal distribution of category assignment corresponding to a
Dirichlet Process (DP), which governs the joint distribution of category
assignments and parameters for each category (Teh et al., 2010). DP has the
constructive process known as the stick-breaking construction (Blei & Jordan,
2006). For the prior probability measure $G$ of category parameter $\theta$
(without specifying the category), we have $G\sim DP(\gamma,H)$, and $G$ can
be constructed as follows.

$\begin{split}\beta_{k}\sim\text{Beta}(1,\gamma),&\quad\theta_{k}\sim H,\\\
\pi_{k}=\beta_{k}\prod_{l=1}^{k-1}(1-\beta_{l}),&\quad
G=\sum_{k=1}^{\infty}\pi_{k}\delta_{\theta_{k}}.\end{split}$ (3)

where $H$ is the base measure in the DP. The intuition is to sequentially
sample for each category parameter $\theta_{k}$ a proportion $\beta_{k}$ from
the remaining part of a stick (with $\sum_{k=1}^{\infty}\pi_{k}=1$).

The likelihood in Equation 1 can be a multivariate normal distribution for
continuous variables with parameters $\theta_{k}=(\mu_{k},\Sigma_{k})$ for
category $k$. The distribution specifies the mean parameters $\mu_{k}$
representing category prototypes and the covariance parameters $\Sigma_{k}$
defining dimensional variations.

RMC enjoys the flexibility to learn an indefinite number of categories. It can
also be used in supervised and unsupervised settings and allows subcategory
modeling (Griffiths et al., 2007). However, how RMC models dimensional
variations within and across categories lacks interpretability because it is
generally difficult to discern the similarity between covariances.

###  2.2 Models of human generalization patterns

Humans exhibit consistent generalization patterns in the feature space, e.g.,
isotropic or dimension-aligned (Smith, 1989). Researchers have proposed
different models to describe such patterns. For example, Shepard (1987) uses
$L_{1}$ metric to depict rule-based generalization. Nosofsky (1986) uses
$L_{2}$ metric for similarity-based generalization of exemplars. In RMC
models, the covariance matrix reflects graded generalization that rotates and
scales the feature space.

Moreover, the covariance matrix implies a direction of strong generalization
through its first principal component. Researchers have imposed a mixture
prior on the covariance matrix (Heller et al., 2009) to highlight a restricted
number of dimensions for strong generalization while enabling multiple
categories to share these dimensions. Consider a mixture of inverse Wishart
distributions with $J$ components. Denote $\Phi_{j}$ as the parameters for the
$j$-th component. The prior for the covariance matrix of category $k$ is given
by
$P(\Sigma_{k}|\Phi_{1},\dots,\Phi_{J})=\sum_{j=1}^{J}P(u_{k}=j)P(\Sigma_{k}|\Phi_{j})$
where $u_{k}$ indicates which component to take effect. This model can also
include infinitely many components, e.g., using the CRP prior (Sanborn et al.,
2021).

Notice that the covariance $\Sigma_{k}$ holds full-rank information about
rotating and scaling the feature space. However, humans tend to focus only on
a selective subset of dimensions when categorizing things (Aha & Goldstone,
1992). Besides, the covariance implicitly determines the direction of strong
generalization, making it challenging to identify subcategories. Therefore, a
model that reduces the number of differentiable dimensions for each category
may provide a better account of human behavior.

##  3 Dimension-reduced category representation

To properly characterize how humans categorize things, we need a combination
of two elements: a hierarchical structure for generalization and a flexible
local dimension-reduction representation. We start with dimension reduction
for each category and then move on to a hierarchical model that allows
categories to share feature dimensions for strong generalization.
Additionally, we discuss the implications of this model for identifying
subcategories within each category.

We propose a low-dimensional representation of categories based on
probabilistic principal component analysis (PPCA, Tipping & Bishop (1999)).
PPCA assumes that an observation $x_{n}\in\mathbb{R}^{d}$ is generated from a
low-dimensional latent variable $z_{n}\in\mathbb{R}^{q}$ with transformation

$x_{n}=Wz_{n}+\mu+\epsilon_{n}.$ (4)

The columns of the _loading_ matrix $W\in\mathbb{R}^{d\times q}$ suggest the
directions of strong generalization. The latent variable $z_{n}$ indicates
variations in these directions. For convenience, we assume normal priors for
the latent variables $z_{n}\sim N(0,I_{q})$ and noises $\epsilon_{n}\sim
N(0,\sigma^{2}I_{d})$.

Denote $\\{\theta_{c}\\}=\\{(\mu_{c},W_{c},\sigma_{c}^{2})\\}$ the parameters
of all categories $c\in C$. Adopting a DP prior, after knowing $n-1$
observations $\textbf{x}_{n-1}$ and their category assignments
$\textbf{c}_{n-1}$, we have the joint posterior distribution

$\begin{split}P(\\{\theta_{c},\beta_{c}\\}&|\textbf{c}_{n-1},\textbf{x}_{n-1})\\\
&\propto
P(\\{\theta_{c}\\})P(\\{\beta_{c}\\})P(\textbf{c}_{n-1}|\\{\beta_{c}\\})P(\textbf{x}_{n-1}|\textbf{c}_{n-1},\\{\theta_{c}\\})\end{split}$
(5)

where $\\{\beta_{c}\\}$ come from the stick-breaking process, and
$\\{\theta_{c}\\}$ are sampled from the base measure $H$ of the DP. The
marginal posterior distribution of the category parameter $\theta$ (without
specifying the category assignment) is derived as

$\begin{split}&P(\\{\theta_{c}\\}|\textbf{c}_{n-1},\textbf{x}_{n-1})\propto
P(\\{\theta_{c}\\})P(\textbf{x}_{n-1}|\textbf{c}_{n-1},\\{\theta_{c}\\}),\\\
&P(\\{\beta_{c}\\}|\textbf{c}_{n-1},\textbf{x}_{n-1})\propto
P(\\{\beta_{c}\\})P(\textbf{c}_{n-1}|\\{\beta_{c}\\}),\\\
&\theta|\textbf{c}_{n-1},\textbf{x}_{n-1}\sim G=\sum_{c\in
C}\pi_{c}\delta_{\theta_{c}}\end{split}$ (6)

To formulate $H$, we assume independent normal for $\mu$ and multivariate
normal for $W$. The prior for $W$ will be modified in Section 4 to incorporate
shared principal component dimensions across categories.

The features of an observation $x_{n}$ given its category assignment $c_{n}$
follow the multivariate normal distribution that marginalizes out the latent
variables $z_{n}$

$x_{n}|c_{n},\theta_{c_{n}}\sim
N(\mu_{c_{n}},W_{c_{n}}W_{c_{n}}^{T}+\sigma_{c_{n}}^{2}I_{d}).$ (7)

The columns of the loading matrix $W_{c_{n}}$ correspond to $q$ dimensions of
strong generalization for category $c_{n}$.

We then introduce the PPCA _classifier_ as the predictive distribution of
category assignment $c_{n}$ given observation $x_{n}$

$\begin{split}P(c_{n}|x_{n},\textbf{c}_{n-1},\textbf{x}_{n-1})=P(c_{n}|\textbf{c}_{n-1})P(x_{n}|c_{n},\textbf{c}_{n-1},\textbf{x}_{n-1}),\end{split}$
(8)

where $P(c_{n}|\textbf{c}_{n-1})$ is easy to obtain based on the CRP process,
and the latter involves simulating the posterior of $\theta$

$P(x_{n}|c_{n},\textbf{c}_{n-1},\textbf{x}_{n-1})=\int_{\theta}P(x_{n}|c_{n},\theta)dP(\theta|\textbf{c}_{n-1},\textbf{x}_{n-1}).$
(9)

Meanwhile, given category assignment $c_{n}$, the low-dimensional latent
variable $z_{n}$ has the following posterior
$z_{n}|x_{n},c_{n},\theta_{c_{n}}\sim
N\bigg{(}(W_{c_{n}}^{T}W_{c_{n}}+\sigma_{c_{n}}^{2}I_{q})^{-1}W_{c_{n}}^{T}(x_{n}-\mu_{c_{n}}),\sigma_{c_{n}}^{2}(W_{c_{n}}^{T}W_{c_{n}}+\sigma_{c_{n}}^{2}I_{q})^{-1}\bigg{)}$.
The principal components for each category span a low-dimensional local
feature system, with the latent variable $z_{n}$ explicitly capturing within-
category variations. We explain how this relates to subcategory learning in
Section 4.2.

###  3.1 Theoretical analysis of dimension reduction

When is a low-dimensional representation better than a full-rank
representation? We explore this question by considering the limiting case of
PPCA when $\sigma^{2}\rightarrow 0$ so that it reduces to PCA and by focusing
on two categories $C=\\{a,b\\}$. Observations from each category $c$ follow
$x|c\sim N(\mu_{c},\Sigma_{c}),\forall c\in C$. For simplicity, we assume
equal covariance $\Sigma_{a}=\Sigma_{b}=\Sigma$. The probability of assigning
observation $x$ to the correct category (set to be $a$ without loss of
generality) can be expressed by a sigmoid function

$p(a|x)=\frac{e^{-\tau_{q}(x,\mu_{a})}}{e^{-\tau_{q}(x,\mu_{a})}+e^{-\tau_{q}(x,\mu_{b})}}=\frac{1}{1+e^{-\\{\tau_{q}(x,\mu_{b})-\tau_{q}(x,\mu_{a})\\}}},$
(10)

where $\tau_{q}$ is the projected distance in the subspace spanned by the
first $q$ PC dimensions $(q<d)$, specified by eigenvectors $u_{i},i=1,\dots,q$
with decreasing eigenvalues $\lambda_{1}\geq...\geq\lambda_{q}$. The squared
distance between category prototypes in the full-dimension space
$r_{ab}=||\mu_{a}-\mu_{b}||^{2}$ implies the amount of total information,
while $r_{i}=||(\mu_{a}-\mu_{b})^{T}u_{i}||^{2}$ describes the proportion of
total information explained by the $i$-th PC (with
$\sum_{i=1}^{d}r_{i}=r_{ab}$). We call
$\alpha_{q}\triangleq\tau_{q}(x,\mu_{b})-\tau_{q}(x,\mu_{a})$ the sample
discrimination index for the $q$-dimensional PC subspace, reflecting how far
the observation is from the wrong category relative to the correct category.

We investigate whether the $(q+1)$-th PC dimension should be reduced, and use
the representation in the subspace spanned by the first $q$ PC dimensions.
Define the signal-to-noise ratio (SNR) of the sample discrimination index

$\text{SNR}_{q}=\frac{E_{x}[\alpha_{q}]^{2}}{Var_{x}[\alpha_{q}]},\quad
q=0,1,...,d-2.$

Proposition 3.1 presents the necessary and sufficient condition for excluding
a PC dimension for the category representation.

######  Proposition 3.1.

For given category prototypes $\mu_{a},\mu_{b}$, discarding the $(q+1)$-th PC
dimension (for $q=0,1,\dots,q-2$ from the category representation increases
the signal-to-noise ratio of $\alpha_{q}$ ($\text{SNR}_{q}<\text{SNR}_{q+1}$)
if and only if

$\lambda_{q+1}<\bigg{(}\frac{r_{q+1}}{\sum_{i=q+2}^{d}r_{i}}+2\bigg{)}\bigg{(}\frac{\sum_{i=q+2}^{d}r_{i}\lambda_{i}}{\sum_{i=q+2}^{d}r_{i}}\bigg{)}.$
(11)

The first term on the right-hand side reflects the information provided by the
$(q+1)$-th PC dimension for differentiating categories. We need to exclude a
PC dimension if it provides more information about cross-category variation
than within-category variation. Adopting the rule (11) improves performance
bound in categorization.

######  Corollary 3.2.

If condition (11) holds, dimension-reduction improves the accuracy lower bound
for the PCA classifier.

##  4 Hierarchical prior on feature dimensions

###  4.1 Hierarchical infinite mixture of PPCA

Now we present the hierarchical infinite mixture of PPCA (mPPCA), a
nonparametric Bayesian hierarchical model based on the PPCA representation,
introducing dependencies between categories by explicitly sharing local PCs.

Figure 1: Schematic illustration of the hierarchical prior over PCs.
Categories share these components for common variation patterns.

There are two infinite mixtures in constructing the prior for mPPCA. On the
lower level, mPPCA describes observations as an infinite mixture of
categories. The CRP prior (2) over category assignment allows infinitely many
categories, but materializes only a finite set given the observations. Each
category is represented by PPCA with its own parameters $\mu_{c},W_{c}$. In
this section, we assume each category has only one direction of strong
generalization and is represented by a _single_ PC dimension. On the higher
level, we introduce another infinite mixture to share PC dimensions among
categories. For each category $c$, an ownership indicator $u_{c}$ indexes a PC
component $\nu_{j}$ in the top-level mixture, modeled by the CRP prior. The
chosen dimension $W_{c}$ (with $\dim(W_{c})=1$) is called the local or
category-specific PC. Observations are drawn from the generative process
presented in the Appendix Appendix A, with main idea presented in Figure 2.

Figure 2: Graphical representation of mPPCA.

Adding a hierarchical prior changes the inference process. The posterior can
be decomposed as the product of the conditional distribution of component-
level parameters $\\{\beta_{j}^{*},\nu_{j}^{*}\\}$ and the marginal of
category-level parameters $\\{\theta_{c},\beta_{c}\\}$.

$\begin{split}p(\\{\beta_{j}^{*},\nu_{j}\\},\\{\theta_{c},\beta_{c}\\}&|\textbf{x}_{n-1},\textbf{c}_{n-1})=\\\
&p({\beta_{j}^{*},\nu_{j}}|\\{\theta_{c}\\})p(\\{\theta_{c},\beta_{c}\\}|\textbf{x}_{n-1},\textbf{c}_{n-1})\end{split}$
(12)

where the second term is Equation 5. The first term is the posterior of the
CRP mixture with concentration parameter $\gamma^{*}$ and normal base measure
with covariance $\frac{1}{\alpha_{\nu}}I$.

$\begin{split}p(\\{\beta_{j}^{*},\nu_{j}\\}|\\{\theta_{c}\\})\propto&p(\\{\beta_{j}^{*}\\}|\gamma^{*})p(\nu_{j}|\alpha_{\nu})\\\
&\cdot
p(\\{u_{c}\\}|\\{\beta_{j}^{*}\\})p(\\{w_{c}\\}|\\{u_{c}\\},\\{\nu_{j}\\}),\end{split}$
(13)

Equation 12 implies that the full posterior can be derived by the marginal
posterior of category-level parameters and the conditional probability of
component-level parameters.

Hierarchical prior in mPPCA supports consistent and flexible generalization.
The components of the top-level prior constitute an expressive set of feature
dimensions shared among categories, resembling the central repository of
features in humans (Austerweil & Griffiths, 2013). For a new category with
only a handful of observations, the prior guides generalization along these
learned directions of PCs.

###  4.2 Few-shot generalization

mPPCA supports few-shot generalization with the hierarchical mixture prior,
when the observation alone cannot support reliable parameter estimation. We
consider generalization both within and beyond an existing category.

#### Learning new categories

Consider generalizing in a certain context, with several observed categories.
Previous learning leads to finite global PCs and finite categories. Given only
one sample $x_{new,1}$ from a new category, we cannot estimate a covariance
directly. Hierarchical prior allows the new category to inherit generalization
patterns from the existing ones. mPPCA suggests a category with mean
$x_{new,1}$ and a PC $w_{new}$ sampled from the CRP posterior, with strong
generalization along directions previously observed in the context. In a
simple context where individuals learn to generalize along one certain
direction $w_{new}$, a new category is represented as
$x_{new}|w_{new},x_{new,1}\sim N(x_{new,1},\sigma^{2}I+w_{new}w_{new}^{T})$.
As a result, learning a new category can be accomplished with only one
observation, whose role is mainly to locate the new category prototype.

#### Learning sub-categories

The PPCA category representation also has the flexibility to accommodate
subcategory learning. It is equivalent to a normal mixture of subcategories
with the same covariance structure. Consider forming a subcategory within
category $c$ with prototype $\mu_{c}$ and one local PC $w_{c}$. Let the latent
variable be a probabilistic mixture of two components $z=sz_{1}+(1-s)z_{2}$,
where $s\sim\text{Bernoulli}(p_{1})$ indicates which component is realized.
Utilizing the realized latent variable $z_{sub}$ as the subcategory prototype,
the PPCA utilizes the PC $w_{c}$ as a local feature system to understand
subcategories $x_{sub}|z_{sub}\sim N(\mu_{c}+w_{c}z_{sub},\sigma^{2}I)$.
Assuming standard normal prior for each component $z_{1},z_{2}$, the marginal
distribution of latent variable $z$ remains normal. When no observation is
available, latent variables can help understand subcategories.

##  5 Simulation studies

###  5.1 Categorization with PPCA representation

We show that the optimal choice of dimensionality $q$ for category
representation depends on the distribution of category prototypes and within-
category variation.

#### Procedure

We generate 10000 samples from two categories $a,b$ in a 3-dimensional space.
Category $a$ prototype is fixed at the origin, while category $b$ prototype is
manipulated, specified with a unit vector and a scalar for the direction and
distance. For covariance, we let the principal components align with the
coordinates. Hence, the relative position of categories determines the
information structure. The variance of the $1^{\text{st}}$ and $3^{\text{rd}}$
are fixed, while the noise level of the second PC varies to control noise
distribution.

We consider three models, with $q=0,1,2$ for the category representation.
$q=0$ implies the evaluation of Euclidean distance to category prototype.
$q=1$ leads to a 1-dimensional PPCA representation. $q=2$ corresponds to a
full-rank representation. For each model, $\sigma^{2}$ is set to the MLE, the
average of variance of discarded dimensions in the representation
$\sigma^{2}=\frac{1}{d-q}(\sum_{i=q+1}^{d}\lambda_{i})$.

Figure 3: Model Accuracy with different information distribution, varying
prototype distance and noisy structure.

#### Results

When only the first dimension is informative, it should be removed from
category representation, though it captures major within-category variation
(see Figure 8 (a)111Representative sub-figures are selected and shown in
Figure 3. The complete result is presented in Figure 8 in the Appendix.). This
also holds for the other dimensions: when only the second dimension is
informative, the optimal choice is to adopt a rank-1 PPCA representation, as
shown in Figure 3.

When there are two equally informative dimensions, one of which is more noisy,
it’s better to adopt a category representation with the noisy dimension
included. This allows a better account of within-category variation during
categorization, while the other, more informative dimension supports
categorization. When the two dimensions have about the same level of noise,
the performance gap vanishes, as is shown by the second row of Figure 3: as
the noise standard deviation increases, the difference between models
decreases. One informative dimension with a small variation is sufficient for
good categorization performance. This finding is backed by the third setting
(Figure 8 (c)), where all dimensions provide the same amount of information.

The results demonstrate that a dimension-reduced representation will achieve
high accuracy when category prototypes are distributed along some dimensions
with moderate-to-low within-category variation. Removing these dimensions
helps focus on differences across categories, and thus improves categorization
performance.

###  5.2 Hierarchical learning of generalization biases

We run simulations to illustrate the context-dependent learning of dimension-
aligned generalization patterns. For mPPCA, PCs in the hierarchical mixture
prior transfer and specify directions of strong generalization.

#### Procedure

The simulation involves two stages, a learning stage and a generalization
stage. Training data is generated from a mixture distribution of axes-aligned
categories, with large variance along one of the two dimensions. We also
included a rotated version of categories (Heller et al., 2009; Sanborn et al.,
2021). In the learning stage, the model performs unsupervised learning on the
training data. In the generalization stage, a new stimulus is given, and we
visualize the generalization pattern of the models by the probability of
assigning any other new stimulus to the same category. We implement posterior
inference using pyro (Bingham et al., 2019). Details of model setup are shown
in the Appendix.

Figure 4: Equal-generalization-probability contour of different models. The
two axes correspond to dimensions in the psychological space (e.g., size and
color). After learning on axes-aligned clusters, hierarchical models (REFRESH
and mPPCA) exhibit knowledge transfer.

#### Results

The generalization patterns of models are shown in Figure 4. Because the
training data contains a balanced number of categories along each axis, we
anticipate a mixture of strong generalizations along two dimensions. mPPCA
adapts to the natural variations in the training data quite well. REFRESH, a
hierarchical model with a mixture prior over covariance, exhibits a similar
generalization pattern as mPPCA. This resembles human generalization on
separable dimensions (Sanborn et al., 2021). By contrast, the exemplar model
and DP mixture model cannot transfer knowledge from previous learning. mPPCA
also produces a narrower ellipse of equal generalization probability compared
with REFRESH, due to the low-rank representation.

##  6 Experiments

###  6.1 Behavioral experiment

We carried out two experiments to study human few-shot generalization behavior
on a new category or subcategory, and compare the predictions of different
models.

Figure 5: Procedure of few-shot generalization experiment. Categories contain
semicircles of different sizes. Category 1 or 2 has regular size but varying
color. After learning these categories in session 1, session 2 provides either
one-shot or zero-shot instructions. The new category is similar to category 1
and 2 but locate near a different size value. The subcategory is generated
from an isotropic Gaussian distribution, aligned with category 2 on the size
dimension. Generalization patterns are tested in session 3.

#### Stimuli

The experiment stimuli we choose are semicircle varying along color and size
dimensions, two commonly used separable dimensions (Smith, 1989; Heller et
al., 2009). We used dimension rating data to scale stimulus parameters based
on perceived similarity. Each category corresponds to a multivariate normal
distribution in the two-dimensional size-color space. The categories have
small variance on the size dimension, and large variance on the color
dimension. The subcategory has small variance on both dimensions. Stimuli used
in experiments are independent and identically distributed samples from the
(sub)categories.

#### Procedure

Participants go through 3 sessions for each experiment: a category learning
session containing the train and test phase, a few-shot instruction session,
and a generalization test session. In the category learning session,
participants get familiar with the categories and their variations. They
undergo training and testing phase, with 20 samples in each phase from
category 1, 2 or neither of the two. Training lasts until participants
categorize all the training samples correctly, then tested on 20 test stimuli
with no feedback. In the few-shot instruction session, participants receive
zero or one sample from a new (sub)category. In the subcategory experiment,
verbal description of a subcategory is provided, describing its category PC
(color) score. In the new category experiment, one sample from the new
category is provided. In the generalization test session, participants
categorize 20 stimuli into one of the previous categories or the new
(sub)category. Their choice is collected. The subcategory experiment includes
samples from one of the categories, some of which come from the subcategory.
The participants judge whether the test stimuli come from the subcategory. In
the new category experiment, participants classify the samples into one of the
three categories, category 1, 2 and the new one. See Figure 5 for an
illustration.

Experiment data is collected from the online platform Credamo. 200
participants were recruited for each experiment, with 172 and 186 passing
predetermined attention and accuracy tests, respectively. Each participant
goes through the 3 sessions described above. We compare predictions of mPPCA
(Section 4.2) to prototype models and exemplar models, with or without
attention mechanism. Details of setup are presented in the Appendix.

#### Results

After training, most participants effectively learned the categories. Figure
11 shows human performance in session 1 test phase. During generalization
test, mPPCA provides accurate predictions of human categorization.

(a) new category

(b) subcategory

Figure 6: Model prediction of human assignment probability.

In the new category experiment, exemplar models underestimate the probability
of assigning samples to the new category, seriously affected by unbalanced
categories. Prototype models (with or without attention) produce similar
generalization patterns as mPPCA (see Figure 6(a)), both providing good
predictions (Figure 12). Stimulus-level analysis shows that exemplar models
are accurate on category 1 and category 2 (Figure 13), as the new category is
given little weight. mPPCA performs comparably to the prototype model with
attention, and significantly beats the baselines on some stimuli.
Quantitatively, mPPCA achieves higher accuracy in predicting human category
assignments, as shown in Table 1.

In the subcategory experiment, mPPCA produces significantly better prediction
of human choice (See Figure 10 and Table 1). Both exemplar models and
prototype models, with or without attention mechanism, underestimate the
probability of the subcategory. (see Figure 6(b)). Qualitatively, mPPCA
achieves significantly better accuracy and correlation with human choice (see
Table 1), outperforming even the oracle model (a linear regression model with
human choice data). In the subcategory experiment, attention mechanism hurts
categorization performance, as it is not suited for variations within category
2. Instead, humans adopt more flexible feature weighting strategies than a
fixed set of weights, which is captured by mPPCA.

To summarize, in the cases where a new (sub)category is introduced with sparse
data, exemplar models fail to capture rapid learning of a new category,
prototype representations are predictive, but cannot generalize well with the
attention mechanism. Hierarchical mixture of covariance matrices, analogously,
lacks the flexibility to explain human generalization either within or beyond
existing categories. mPPCA provides a better account of human few-shot
categorization.

Table 1: Accuracy in predicting human few-shot generalization behavior on the subcategory and the new category | subcategory learning | new category learning
---|---|---
Model | expected accuracy | correlation | expected accuracy | correlation
Exemplar | 0.517$\pm$0.043 | -0.102$\pm$0.118 | 0.594$\pm$0.063 | 0.372$\pm$0.128
Exemplar+Attention | 0.498$\pm$0.041 | -0.102$\pm$0.118 | 0.620$\pm$0.064 | 0.407$\pm$0.117
Prototype | 0.599$\pm$0.020 | 0.351$\pm$0.091 | 0.562$\pm$0.030 | 0.607$\pm$0.051
Prototype + Attention | 0.555$\pm$0.012 | 0.351$\pm$0.091 | 0.638$\pm$0.030 | 0.688$\pm$0.044
Linear rule (Oracle) | 0.623$\pm$0.026 | 0.463$\pm$0.061 | - | -
mPPCA (Ours) | 0.662$\pm$0.033 | 0.451$\pm$0.065 | 0.705$\pm$0.028 | 0.696$\pm$0.040

###  6.2 Experiment on natural image categorization dataset

The previous experiments (and most experiments in previous research (Nosofsky,
1986)) use low-dimensional stimuli, convenient for manipulation but not
realistic. A recent trend is to embrace large-scale experiments with
realistic, high-dimensional natural stimuli (Nosofsky et al., 2018; Battleday
et al., 2020). In an effort to scale up, we explore human mental
representations of natural categories using CIFAR-10H (Peterson et al., 2019),
which records human categorization on natural images.

Our goal is to elicit human mental representations from categorization data.
We take an approach with ideas similar to that of multidimensional scaling
(MDS). Viewing stimuli as points and categories as distributions in a
multidimensional space, we numerically adjust the category representations to
approximate categorization patterns. Given a stimulus $i$ with raw features
$x_{i}$, the probability of assigning it to category $j$, $p_{ij}\propto
f_{j}(x_{i}|\theta_{j})$, where $f_{j}$ is the likelihood function
parameterized by $\theta_{j}$. We parameterize the feature mapping with a
convolutional neural network (CNN), denoted as $\phi(\cdot)$. The problem of
approximating human mental representations is formalized as the following
optimization problem:

$\min_{\phi,\theta}\sum_{i=1}^{N}\sum_{j=1}^{C}\mathcal{L}(p_{ij},f(\phi(x_{i})|\theta_{j})),$
(14)

Using a CNN feature map overcomes MDS’s limitations and makes predictions
easier. Optimization is accomplished via stochastic gradient descent, as is
typical for neural networks.

#### Network architecture: PPCA layer

The model includes a feature map and a multi-head classifier (see Figure 15),
similar to the architecture proposed by Singh et al. (2020). We adopt a
ResNet222We adapt models and pre-trained weights from
https://github.com/huyvnphan/PyTorch_CIFAR10. as our feature map $\phi$. In
the multi-head classifier, each head corresponds to a category parameterized
by $\theta_{j}$. For a rank-1 PPCA category,
$\theta_{j}=(\mu_{j},w_{j},\sigma_{j}^{2})$. The forward pass can be
implemented with basic modules. Details are shown in the Appendix Appendix F.

#### Loss function: inner product regularization

Regularization helps introduce proper inductive biases into machine learning
models. For example, L2 regularization is equivalent to a Gaussian prior on
parameters. In the previously described architecture, category representations
are independent. Each head learns its own parameters $\theta_{j}$. By
contrast, we have argued for a more concise and explainable feature system.
Therefore, we introduce a regularization term on
$\theta_{j}\cdot\theta_{j^{\prime}}(j\neq j^{\prime})$ to encourage correlated
local features. Details are presented in the Appendix (Equation 20).

#### Results

In an exploratory run with full-rank classifiers, training on CIFAR-10H leads
to cross-entropy of 0.5 and accuracy of 91%. Eigendecompostion of the
covariance shows that each category has a dominating PC (see Figure 16),
supporting the idea of low-rank representations.

Using PPCA as classifiers and training the model by minimizing the regularized
loss, our models reach cross-entropy of 0.39 on the holdout sets, comparable
to the best performance reported by Peterson et al. (2019), and accuracy on
CIFAR-10 reaches 97%. The best model exhibits larger correlations between
category PCs of animals (Figure 7(a)). The latent variable values of randomly
sampled category members are approximately normally distributed Figure 7(b).
The one-dimensional latent variable reflects the typicality of an observation
(Murphy, 2004), where more typical samples locate near the center of the
distribution.

(a) Pairwise correlation of category principal components.

(b) Distribution over latent variables within category "plane".

Figure 7: Results from the experiment on natural image categorization dataset.

##  7 Discussion and limitation

Properties of PPCA makes it possible to explain cross-categorization and
context-dependent behavior. PPCA does not impose orderings among PCs, enabling
context-dependent ordering of features, as in cross-categorization (Shafto et
al., 2011). Besides, PPCA does not assume orthogonality and can learn
correlated features. This resembles human feature learning. For example,
saturation and brightness are learned by color experts, though correlated with
other color dimensions (Austerweil & Griffiths, 2010).

Humans can also learn categories in other ways, like social learning. It’s
unrealistic that all principal components in human minds are learned through
direct observation. This assumption is made for simplification purposes. Other
aspects of human learning are out of the scope of this paper.

##  8 Conclusion

We propose a rational model of categorization with dimension-reduced category
representation. Such a low-dimensional category representation is beneficial
when certain dimensions are more important for categorization than for
characterizing within-category variation. Based on the PPCA representation,
the mPPCA model utilizes the hierarchical prior for flexible generalization.
Experiments show that mPPCA model can effectively capture human few-shot
generalization within or beyond categories, and when combined with neural
networks, can predict human categorization of natural stimuli.

## Acknowledgements

This work was supported by the National Natural Science Foundation of China
(NSFC) 72192824.

## References

  * Aha & Goldstone (1992) Aha, D. W. and Goldstone, R. L.  Concept learning and flexible weighting.  In _Proceedings of the fourteenth annual conference of the Cognitive Science Society_ , pp. 534–539. Citeseer, 1992.
  * Anderson (1991) Anderson, J. R.  The adaptive nature of human categorization.  _Psychological Review_ , 98(3):409, 1991.
  * Ashby et al. (2005) Ashby, F. G., Maddox, W. T., et al.  Human category learning.  _Annual Review of Psychology_ , 56(1):149–178, 2005.
  * Austerweil & Griffiths (2010) Austerweil, J. and Griffiths, T.  Learning hypothesis spaces and dimensions through concept learning.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , volume 32, 2010.
  * Austerweil & Griffiths (2013) Austerweil, J. L. and Griffiths, T. L.  A nonparametric bayesian framework for constructing flexible feature representations.  _Psychological Review_ , 120(4):817, 2013.
  * Battleday et al. (2020) Battleday, R. M., Peterson, J. C., and Griffiths, T. L.  Capturing human categorization of natural images by combining deep networks and cognitive models.  _Nature Communications_ , 11(1):1–14, 2020.
  * Bingham et al. (2019) Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., and Goodman, N. D.  Pyro: Deep universal probabilistic programming.  _The Journal of Machine Learning Research_ , 20(1):973–978, 2019.
  * Blackwell & MacQueen (1973) Blackwell, D. and MacQueen, J. B.  Ferguson distributions via pólya urn schemes.  _The Annals of Statistics_ , 1(2):353–355, 1973.
  * Blei & Jordan (2006) Blei, D. M. and Jordan, M. I.  Variational inference for Dirichlet process mixtures.  _Bayesian Analysis_ , pp. 121–144 , 2006.
  * Blei et al. (2017) Blei, D. M., Kucukelbir, A., and McAuliffe, J. D.  Variational inference: A review for statisticians.  _Journal of the American statistical Association_ , 112(518):859–877, 2017.
  * Goldstone (1994) Goldstone, R. L.  Influences of categorization on perceptual discrimination.  _Journal of Experimental Psychology: General_ , 123(2):178, 1994.
  * Griffiths et al. (2007) Griffiths, T., Canini, K., Sanborn, A., and Navarro, D.  Unifying rational models of categorization via the hierarchical dirichlet process.  _Proceedings of the Twenty-Ninth Annual Conference of the Cognitive Science Society_ , 2007.
  * Hebart et al. (2020) Hebart, M. N., Zheng, C. Y., Pereira, F., and Baker, C. I.  Revealing the multidimensional mental representations of natural objects underlying human similarity judgements.  _Nature Human Behaviour_ , 4(11):1173–1185, 2020.
  * Heller et al. (2009) Heller, K. A., Sanborn, A., and Chater, N.  Hierarchical learning of dimensional biases in human categorization.  _Advances in Neural Information Processing Systems_ , 22, 2009.
  * Jha et al. (2023) Jha, A., Peterson, J. C., and Griffiths, T. L.  Extracting low-dimensional psychological representations from convolutional neural networks.  _Cognitive Science_ , 47(1):e13226, 2023.
  * Kucukelbir et al. (2017) Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., and Blei, D. M.  Automatic differentiation variational inference.  _Journal of Machine Learning Research_ , 2017.
  * Lake et al. (2015) Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B.  Human-level concept learning through probabilistic program induction.  _Science_ , 350(6266):1332–1338, 2015.
  * Murphy (2004) Murphy, G.  _The big book of concepts_.  MIT press, 2004.
  * Neal (2000) Neal, R. M.  Markov chain sampling methods for dirichlet process mixture models.  _Journal of Computational and Graphical Statistics_ , 9(2):249–265, 2000.
  * Nosofsky (1986) Nosofsky, R. M.  Attention, similarity, and the identification–categorization relationship.  _Journal of Experimental Psychology: General_ , 115(1):39, 1986.
  * Nosofsky et al. (2018) Nosofsky, R. M., Sanders, C. A., and McDaniel, M. A.  Tests of an exemplar-memory model of classification learning in a high-dimensional natural-science category domain.  _Journal of Experimental Psychology: General_ , 147(3):328, 2018.
  * Peterson et al. (2019) Peterson, J. C., Battleday, R. M., Griffiths, T. L., and Russakovsky, O.  Human uncertainty makes classification more robust.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9617–9626, 2019.
  * Ranganath et al. (2014) Ranganath, R., Gerrish, S., and Blei, D.  Black box variational inference.  In _Artificial Intelligence and Statistics_ , pp. 814–822. PMLR, 2014.
  * Reed (1972) Reed, S. K.  Pattern recognition and categorization.  _Cognitive Psychology_ , 3(3):382–407, 1972.
  * Salakhutdinov et al. (2012) Salakhutdinov, R., Tenenbaum, J., and Torralba, A.  One-shot learning with a hierarchical nonparametric bayesian model.  In _Proceedings of ICML Workshop on Unsupervised and Transfer Learning_ , pp. 195–206. JMLR Workshop and Conference Proceedings, 2012.
  * Sanborn et al. (2010) Sanborn, A. N., Griffiths, T. L., and Navarro, D. J.  Rational approximations to rational models: alternative algorithms for category learning.  _Psychological Review_ , 117(4):1144, 2010.
  * Sanborn et al. (2021) Sanborn, A. N., Heller, K., Austerweil, J. L., and Chater, N.  Refresh: A new approach to modeling dimensional biases in perceptual similarity and categorization.  _Psychological Review_ , 128(6):1145, 2021.
  * Schyns & Rodet (1997) Schyns, P. G. and Rodet, L.  Categorization creates functional features.  _Journal of Experimental Psychology: Learning, Memory, and Cognition_ , 23(3):681, 1997.
  * Shafto et al. (2011) Shafto, P., Kemp, C., Mansinghka, V., and Tenenbaum, J. B.  A probabilistic model of cross-categorization.  _Cognition_ , 120(1):1–25, 2011.
  * Shepard (1987) Shepard, R. N.  Toward a universal law of generalization for psychological science.  _Science_ , 237(4820):1317–1323, 1987.
  * Singh et al. (2020) Singh, P., Peterson, J., Battleday, R., and Griffiths, T.  End-to-end deep prototype and exemplar models for predicting human behavior.  In _Proceedings of the 42nd Annual Conference of the Cognitive Science Society_ , 2020.
  * Smith (1989) Smith, L. B.  A model of perceptual classification in children and adults.  _Psychological Review_ , 96(1):125, 1989.
  * Teh et al. (2010) Teh, Y. W. et al.  Dirichlet process.  _Encyclopedia of machine learning_ , 1063:280–287, 2010.
  * Tiedemann et al. (2022) Tiedemann, H., Morgenstern, Y., Schmidt, F., and Fleming, R. W.  One-shot generalization in humans revealed through a drawing task.  _Elife_ , 11:e75485, 2022.
  * Tipping & Bishop (1999) Tipping, M. E. and Bishop, C. M.  Probabilistic principal component analysis.  _Journal of the Royal Statistical Society Series B: Statistical Methodology_ , 61(3):611–622, 1999b.
  * Vong et al. (2019) Vong, W. K., Hendrickson, A. T., Navarro, D. J., and Perfors, A.  Do additional features help or hurt category learning? the curse of dimensionality in human learners.  _Cognitive Science_ , 43(3):e12724, 2019.

##  Appendix A Generative process of mPPCA

  * (1)

For each component in the higher-level mixture,

    * (a)

Draw probabilistic PC $\nu_{j}\sim N(0,\frac{1}{\alpha_{\nu}}I_{d})$.

    * (b)

Draw stick-breaking weight
$\beta_{j}^{*}\sim\text{Beta}(1,\gamma^{*}),\,\pi_{j}^{*}=\beta_{j}^{*}\prod_{i=1}^{j-1}(1-\beta_{j})$.

  * (2)

For each category in the lower-level mixture,

    * (a)

Draw component assignment $u_{c}\sim\text{Mult}(\\{\pi_{j}^{*}\\})$.
$w_{c}=\nu_{u_{c}}+\xi_{c}$, where $\xi_{c}$ is a normal noise term.

    * (b)

Draw category prototype $\mu_{c}\sim N(0,\frac{1}{\alpha_{\mu}}I_{d})$.

    * (c)

Draw stick-breaking weight
$\beta_{c}\sim\text{Beta}(1,\gamma),\,\pi_{c}=\beta_{c}\prod_{l=1}^{c-1}(1-\beta_{c})$.

    * (d)

Draw noise variance $\sigma_{c}^{2}\sim\text{Inv-Gamma}(a_{\tau},b_{\tau})$.

  * (3)

For each sample $x_{n},\,n=1,...,N$,

    * (a)

Draw category assignment $c_{n}\sim\text{Mult}(\\{\pi_{c}\\})$

    * (b)

Draw latent variable $z_{n}\sim N(0,1)$.

    * (c)

Draw observation

$x_{n}|z_{n},c_{n}\sim N(\mu_{c_{n}}+w_{c_{n}}z_{n},\sigma_{c}^{2}I_{d})$

##  Appendix B Proofs for the theoretical analysis

#### Proof for Proposition 3.1

###### Proof.

We first formalize some useful notations. The covariance matrix of the
categories has eigen-decomposition $\Sigma_{c}=U\Lambda U^{T}$, where the
diagonal matrix $\Lambda=\text{diag}(\lambda_{1},...,\lambda_{d})$ consists of
the eigenvalues, and columns of $U$ are corresponding eigenvectors. The
truncated matrix containing first $q$ columns of $U$ is denoted as $U_{q}$,
with corresponding
$\Lambda_{q}\text{diag}(\lambda_{1},...,\lambda_{q}),\,q<d$. Linear projection
into the subspace is represented as $P=W(W^{T}W)^{-1}W^{T}=U_{q}U_{q}^{T}$.

According to the definition,
$\alpha\triangleq||(I-P)(x-\mu_{b})||^{2}-||(I-P)(x-\mu_{a})||^{2}$. For any
given category prototypes, $\mu_{a},\mu_{b}$, and projection matrix
$P=U_{q}U_{q}^{T}$, the expectation and variance of the sample distinction
index $\alpha$ can be derived as

$\mathbb{E}_{x}[\alpha]=||(I-P)(\mu_{a}-\mu_{b})||^{2}=r_{ab}-\sum_{i=1}^{q}r_{i},$
(15)
$\text{Var}_{x}[\alpha]=4(\mu_{a}-\mu_{b})^{T}\Sigma_{c}(\mu_{a}-\mu_{b})-4((\mu_{a}-\mu_{b})^{T}U_{q}\Lambda_{q}U_{q}^{T}(\mu_{a}-\mu_{b}))=4\sum_{i=q+1}^{d}\lambda_{i}r_{i}.$
(16)

When considering distance to the principal subspaces spanned by the first $q$
eigenvectors, the signal-to-noise ratio of $\alpha$

$\text{SNR}_{q}=\frac{\mathbb{E}_{x}[\alpha]^{2}}{var_{x}(\alpha)}=\frac{1}{4}\frac{(r_{ab}-\sum_{i=1}^{q}r_{i})^{2}}{\sum_{i=q+1}^{d}\lambda_{i}r_{i}}$
(17)

Hence, the decision to exclude dimension $q+1$ will increase signal-to-noise
ratio ($\text{SNR}_{q+1}>\text{SNR}_{q}$) if and only if
$\lambda_{q+1}<\frac{2r_{ab}-2\sum_{i=1}^{q+1}r_{i}+r_{q+1}}{(r_{ab}-\sum_{i=1}^{q+1}r_{i})^{2}}\sum_{i=q+2}^{d}r_{i}\lambda_{i}$,
which leads to inequality 11 with minor transformation. ∎

#### Proof for Corollary 3.2

###### Proof.

PCA corresponds to the limit of PPCA as $\sigma^{2}\rightarrow 0$. Hence, the
classifier chooses with probability 1 the category whose principal subspace is
the closest. This leads to $p(\hat{y}=a|a,b)=p(\alpha>0|a,b)$. From one-sided
Chebyshev’s inequality,

$P(\alpha>0|a,b)\geq\frac{E_{x}[\alpha]^{2}}{Var(\alpha|a,b)+E_{x}[\alpha]^{2}}=\frac{\text{SNR}}{1+\text{SNR}}$
(18)

Since it is a monotonic function of signal-to-noise ratio, we immediately
arrives at the corollary. ∎

##  Appendix C Details of simulation study: categorization with PPCA
representation

Here we present the results in the first simulation study in Figure 8,
demonstrating when will dimension-reduced category representations be helpful.
Each sub-figure corresponds to a distribution of information on the 3
dimensions. In Figure 8(a), there is only one informative dimension. Two
equally informative dimensions are present in Figure 8 (b). All three
dimensions provide the same amount of information in Figure 8 (c). For each
sub-figure, the first column on the left illustrates the relative position of
the two categories for the plots in the same row. From left to right, the
variance on the second dimension is increased, changing the distribution of
within-category variation. In each plot, x-axis represents the distance
between category prototypes (mean parameter). Y-axis presents the accuracy of
the three models, with dimensionality of category representation $q=0,1,2$.
$q=0$ implies adoption of Euclidean distance, $q=1$ leads to a 1-dimensional
PPCA representation, and $q=2$ is equivalent to a full-rank category
representation.

(a) One informative dimension

(b) Two informative dimensions

(c) Three dimensions equally informative

Figure 8: Simulation results. The left column marks the location of category
$b$ relative to category $a$. Each figure on the right compares expected
accuracy of models with different dimensionality $q$, as distances increases.
Figures on in each row demonstrates how increasing the noisiness of the second
dimension affects categorization.

##  Appendix D Details of simulation study: hierarchical learning of
generalization biases

We set the concentration parameters in the CRP $\gamma^{*}$ and $\gamma$ to 1,
reflecting moderate preference for new components. We also set vague priors
$\Gamma(1,1)$ on parameters $\alpha_{\mu}$, $\alpha_{v}$, as well as the
precision $\tau=1/\sigma^{2}$. We choose variational inference (VI) for
posterior inference, a common alternative to the Monte Carlo Markov Chain
(MCMC) techniques (Neal, 2000; Sanborn et al., 2010). VI approximates
intractable posteriors with a family of tractable distributions with
optimization (Blei et al., 2017). It enjoys better scalability compared with
MCMC. In recent years, general-purpose VI algorithms have been proposed
(Ranganath et al., 2014; Kucukelbir et al., 2017). We adopted pyro (Bingham et
al., 2019), an expressive, scalable and flexible Probabilistic Program
Languages, to implement the models. During inference, we perform greedy search
for initialization, and train each model with stochastic VI for 3000 epochs.

##  Appendix E Details of behavioral experiment Section 6.1

Figure 9: Prediction of generalization pattern in one-shot generalization of
new category. Dashed lines represent equal generalization probability, dots
are the training/generalization test exemplars. Figure 10: Prediction of
generalization pattern in one-shot generalization of subcategory. Dashed lines
represent equal generalization probability, dots are the training and
generalization test exemplars.

Here we present some detailed results that are not included in the main body
due to space constraints.

###  E.1 Model setup

To provide predictions of human categorization, our models experience the same
set of data. Given both stimuli ${x_{n}}$ and labels $c_{n}$, the higher-level
mixture is disentangled from the lower level. We obtain MLE of the lower-level
mixture, then use variational inference for the high-level mixture, i.e. the
global PCs, according to Equation 13. Due to task context, the base rate term
in Equation 8 is removed, which leads to better prediction for all models. In
the new category experiment, all stimuli in the train and test phase of
session 1 is used to get a more reliable estimate of global PCs. This will not
be necessary for the subcategory experiment, since only the local PC is
needed.

We compare our model with the exemplar model (Nosofsky, 1986), prototype model
(Reed, 1972), with and without attention mechanism. All models provide
predictions without access to human choice. Attention mechanism scales the
original space with a set of dimensional weights, optimized based on cluster
variations. Prototype model with attention mechanism generates generalization
pattern similar to that of hierarchical models (Salakhutdinov et al., 2012;
Sanborn et al., 2021), since the categories are dimension-aligned in the
experiment. Quadratic prototype model (Battleday et al., 2020) can not be
applied as it cannot estimate covariance for the new (sub)category.

###  E.2 Session 1: learning

The training and testing phases helps the participants familiarize the
stimuli, and learn the category structure in this artificial environment.
Figure 11 shows that the subjects have indeed learned the categories, with
accuracy significantly surpassing random guess.

(a) new category

(b) subcategory

Figure 11: Subject categorization test accuracy in Session 1. The majority of
participants effectively learn the categories.

###  E.3 Session 3: generalization

Test on generalization of categories is the main part of the experiment. Here
we present the model predictions on the test stimuli, given the same training
experience as human participants.

Figure 6 plots the predicted probability of assigning test stimuli to the
subcategory (Figure 6(b)) and new category (Figure 6(a)) against human
assignment probability. For the subcategory experiment, exemplar models
systematically underestimate the probability of the subcategory. Notice that
even for the quantitatively best-performing PPCA, there is some under-estimate
of assignment probability, especially on those stimuli with human assignment
probability around 0.5. We consider this may be an effect of task context.
Given specific instruction in Session 2 about the existence of a subcategory,
participants may naturally tend to choose the subcategory, when they are
actually uncertain about the category membership. Prototype models exhibit
complex nonlinear patterns. They cannot capture human generalization with
flexible switching between contexts. For the new category experiment, the
exemplar models again underestimates the probability of the new category.
Prototype models provide similar predictions, but generally deviates more from
the "ground truth".

For the new category experiment, we also use heatmap in Figure 12 to
illustrate categorization patterns. We can see that mPPCA and prototype model
(with attention) provide predictions similar to human categorization
probability. Without attention mechanism, prototype model fails to focus on
important dimensions for the current task. Exemplar models, on the other hand,
underestimates the probability of the new category.

Figure 12: Stimulus-category similarity heatmaps of models. On the right most
is human choice probability.

For more detailed analysis, we show prediction performance for each
participant, and each randomly generated stimuli. Figure 14(a) and Figure
13(a) illustrate the expected accuracy of mPPCA when predicting human choice
probability on the subcategory and new category experiment, respectively.
Figure 15b and Figure 14b show the correlation with human categorization.
mPPCA provides a good estimation in these two experiments.

Figure 14(c) and Figure 14c show the expected accuracy on for each randomly
generated stimulus in the subcategory and new category experiments. In the
subcategory experiment, mPPCA performs at least comparably with other models,
and is significantly better on some of them. In other words, mPPCA dominates
the baseline models, both in terms of accuracy and correlation. In the new
category experiment, mPPCA is outperformed by the exemplar models on simuli
from category 1 and category 2. This is caused by the bias of exemplar models
towards these categories, which have more training samples. In general, mPPCA
provides a better account of human categorization pattern.

(a) Model accuracy for each individual participant in the subcategory
experiment.

(b) Model correlation to each individual participant in the new category
experiment.

(c) Model prediction performance on each stimulus.

Figure 13: Participant and stimulus-level analysis of the new category
experiment

(a) Model accuracy for each individual participant in the subcategory
experiment.

(b) Model correlation to each individual participant in the subcategory
experiment.

(c) Model prediction performance on each stimulus.

Figure 14: Participant and stimulus-level analysis of the subcategory
experiment.

##  Appendix F Details of experiment on natural image categorization dataset
Section 6.2

Figure 15: Schematic illustration of deep PPCA model for natural image
categorization.

The deep PPCA architecture is shown in Figure 15. The marginal likelihood of
PPCA can be rewrite as follows

$\begin{split}\log f(\phi(x_{i})|\theta_{j})&=\log
N(\phi(x_{i})|\mu_{j},w_{j}w_{j}^{T}+\sigma_{j}^{2}I)\\\
&\propto-\log|w_{j}w_{j}^{T}+\sigma_{j}^{2}I|-\frac{1}{2}(\phi(x_{i})-\mu_{j})^{T}(w_{j}w_{j}^{T}+\sigma_{j}^{2}I)^{-1}(\phi(x_{i})-\mu_{j})\\\
&=-\log(w_{j}^{T}w_{j}+\sigma_{j}^{2})-\frac{1}{2\sigma_{j}^{2}}\bigg{(}||\phi(x_{i})-\mu_{j}||_{2}^{2}-\frac{1}{w_{j}^{T}w_{j}+\sigma_{j}^{2}}||w_{j}^{T}(\phi(x_{i})-\mu_{j})||_{2}^{2}\bigg{)}.\end{split}$
(19)

This expression can be implemented with linear module and norm calculation.

###  F.1 Penalty term in the loss function

The penalty is implemented by two penalty terms into the loss function:

$J(\phi,\theta)=\mathcal{L}+\lambda_{1}\sum_{j=1}^{C}||\theta_{j}||_{2}^{2}-\lambda_{2}\sum_{j=1}^{C}\sum_{k=1\atop
k\neq j}^{C}\theta_{j}\theta_{k}.$ (20)

The penalty is equivalent to a Gaussian prior on $\theta$ with marginal
precision $\alpha$ and pairwise correlation $1/\beta,\,\beta>\alpha$. The
penalty term involves some prior assumptions. Each valid pair of
$\lambda_{1},\lambda_{2}$ corresponds to some $\alpha,\beta$, with positive-
definiteness ensured by setting $\lambda_{1}>(C-1)\lambda_{2}$. In this
experiment we take $\lambda_{1}=C\lambda_{2}$, which is equivalent to setting
$\alpha=\beta/2$. In our experiment, we choose cross-entropy as our original
loss function $\mathcal{L}$.

###  F.2 Model setup

We use human categorization data $p_{ij}$ in CIFAR-10H (Peterson et al., 2019)
as soft labels for training, which includes 1000 images for each of the 10
categories. For each image, about 50 human assignments are collected. We use
cross-entropy as the loss function $\mathcal{L}$. We optimize the model using
Adam. K-fold cross-validation is adopted for stable results. We tune the
coefficient for regularization terms based on model performance on the
validation sets. During optimization, a smaller learning rate is set for the
feature map.

##  Appendix G Result of exploratory run of deep categorization model

Here we present the results of the deep categorization model, where each head
in the multi-head classifier evaluates a Mahalanobis distance. The model is
optimized with grid-search over learning rate. After eigendecomposition of the
derived covariance parameter of the category representations, we can see in
Figure 16, the first PC dominates the others, explaining the majority of
variance.

Figure 16: Variance explained by principal components of learned full-rank
categories.