GenZ: Foundational models as latent variable generators within traditional statistical models

Given an item’s semantics, such as a text description, image, or audio recording, large foundational models can answer specific questions about the content, and thus convert the item’s semantic representation into a canonical discrete representation. Such a representation is useful for traditional statistical models—such as graphical generative models, belief networks, or probabilistic circuits—which can capture statistics from external, possibly proprietary, datasets related to these items.

Report issue for preceding element

Movies are an example of such items. We can use an LLM to featurize them as binary vectors based on answers to questions as illustrated in Fig. 1. These binary vectors can help explain the patterns found in the movie-user ratings matrix. Collaborative filtering techniques reduce the observation matrix to embeddings, which capture usage statistics absent from the foundational models’ training data. The foundational models, however, can reason about the semantic content—such as the movie synopsis, critical response, credits, and other metadata—either from their training or by retrieving information from a database as part of a RAG system.

Report issue for preceding element

We describe how appropriate prompts can be learned from such item statistics by jointly modeling the semantic features and the statistical patterns in the data.

Report issue for preceding element

We study a general hybrid statistical-foundational model defined as follows. Let $s$ be the semantic representation of an item, e.g., text with a movie title or its description. Given a collection of semantic feature descriptors $\theta_{f}$, such as ’historical war film’, a foundational model, which we refer to as function $h$, makes a judgment whether the feature applies to the item (Figure 1). Let ${\bf z}$ be the true binary feature vector that allows for possible errors in either the model’s judgment or the wrongly worded feature descriptors. We model uncertainty by defining a conditional distribution $p({\bf z}|s,\theta_{f},\theta_{e})$, where ${\bf z}=[z_{1},z_{2},...,z_{n_{f}}]^{T}$ is a discrete feature vector (binary, in the example in the figure and in our experiments), $\theta_{f}$ is a collection of semantic feature descriptors, such as ’historical war film’, and $\theta_{e}=[p^{e}_{1},p^{e}_{2},...,p^{e}_{n_{f}}]$ is a set of estimated probabilities of feature classification errors. A feature vector is then linked to observation ${\bf y}$ through a classical statistical model $p({\bf y}|{\bf z},\theta_{y})$, possibly involving latent variables or some other way of modeling structure, and which has its own parameters $\theta_{y}$. By jointly optimizing for the parameters of the model $p({\bf y}|s)=\sum_{{\bf z}}p({\bf y}|{\bf z})p({\bf z}|s)$ based on a large number of pairs $({\bf y},s)$ we can discover the feature descriptions, or the right questions to ask, in order to explain the statistics in the dataset. More generally, hybrid models could have more than one category of items with a (learnable) statistical model of their relationships, for example geographic regions and native plants.

Report issue for preceding element

The idea of using interpretable intermediate representations to bridge inputs and predictions has a long history in machine learning. Concept bottleneck models (CBM) koh2020concept; kim2023probabilistic formalized this approach by training neural networks to predict pre-defined human-interpretable concepts from inputs, then using those concepts to make final predictions. Recent work has leveraged large language models (LLMs) to overcome the need for manually labeled concepts, e.g., benara2024craftinginterpretableembeddingsasking; feng2025bayesianconceptbottleneckmodels; chattopadhyay2023variationalinformationpursuitinterpretable; zhong2025explainingdatasetswordsstatistical; ludan2024interpretablebydesigntextunderstandingiteratively. These methods may or may not use the CBM formulation but share with it the goal of feature interpretability. In addition, they have a goal of feature discovery, which is also one of the goals here. They address the difficulty of modeling and jointly adjusting the semantic and statistical parts of the model—in our notation $p({\bf z}|s)$ and $p({\bf y}|{\bf z})$—in different ways.

Report issue for preceding element

In most cases, the target ${\bf y}$ is a discrete variable (the goal is a classification through intermediate concepts), and thus the classification model $p({\bf y}|{\bf z})$ is a simple logistic regressor or an exhaustive conditional table chattopadhyay2023variationalinformationpursuitinterpretable; feng2025bayesianconceptbottleneckmodels; zhong2025explainingdatasetswordsstatistical. The optimization of the featurizer $p({\bf z}|s)$ primarily relies on the LLM’s domain understanding (such as a task description), but some methods also provide examples of misclassified items, e.g., zhong2025explainingdatasetswordsstatistical; ludan2024interpretablebydesigntextunderstandingiteratively.

Report issue for preceding element

When real-valued multi-dimensional targets ${\bf y}$ are modeled, showing items with large prediction errors is much less informative because each item may have a different error vector, and thus item-driven discovery is usually abandoned. For example, in benara2024craftinginterpretableembeddingsasking multidimensional targets (fMRI recordings) are modeled, and the method relies on direct LLM prompting about potentially predictive features based on domain knowledge alone, followed by feature selection, rather than showing specific items to contrast.

Report issue for preceding element

A fundamental limitation shared by these approaches is their reliance on the LLM’s domain understanding to propose discriminative features. This works well when the LLM has been trained on relevant data and the task aligns with common patterns in its training distribution. However, as our experiments demonstrate, domain understanding alone is insufficient when dataset-specific patterns diverge from general knowledge—such as house pricing dynamics in particular markets, or user preferences captured in collaborative filtering from specific time periods or demographics. Moreover, for high-dimensional real-valued targets ${\bf y}$, individual prediction errors have multidimensional structure that cannot be easily communicated to an LLM by simply showing "incorrect" examples.

Report issue for preceding element

Our approach differs in several key aspects. First, as in some of the recent work, we avoid concept supervision by using a frozen foundational model as an oracle within $p({\bf z}|s)$, but with trainable uncertainty parameters $\theta_{e}$. Second, our feature mining is driven by contrasting groups of items identified through the statistical model’s posterior $q({\bf z})$ rather than by querying the LLM’s domain knowledge or showing individual misclassified examples (Figure 2). This group-based contrast allows the method to discover dataset-specific patterns that may not align with the LLM’s training distribution—or even in cases where the LLM has no understanding of what the target ${\bf y}$ represents. (We do, however, require the (M)LLM to reason about the semantic items $s$, either based on knowledge encoded in its weights or by retrieving information from a database as in RAG systems). Third, we support arbitrary mappings $p({\bf y}|{\bf z})$ including nonlinear functions of binary features for high-dimensional real-valued observations, shifting the burden of modeling complex feature interactions from the LLM to the statistical model. The LLM is only responsible for explaining the semantic coherence of binary splits discovered based on the model’s prediction errors, making it applicable to domains where the relationship between semantics and observations is not well understood in advance.

Report issue for preceding element

We refer to our hybrid model as GenZ (or GenAIz?) as it relies on a pre-trained generative AI model to generate candidate latents ${\bf z}$ in a prediction model $s\rightarrow{\bf z}\rightarrow{\bf y}$. We derive a generalized EM algorithm for training it, which naturally requires investigation of items $s$ when tuning parameters $\theta_{f}$ of the featurizer $p({\bf z}|s)$.

Report issue for preceding element

Before developing the full algorithm, we first describe the semantic part: the semantic parameters $\theta_{f}$, how $p({\bf z}|s)$ depends on them and on a foundational model, and how statistical signals from the model enable updating these parameters in an iterative EM-like algorithm.

Report issue for preceding element

Given an item’s semantics $s$, e.g., a movie description, and its corresponding observed vector ${\bf y}$, e.g., the movie’s embedding derived from the statistics in the movie-user matrix, the log likelihood can be bounded as

Report issue for preceding element

	$\displaystyle\log p({\bf y}|s)\geq B$	$\displaystyle=$	$\displaystyle\sum_{\bf z}q({\bf z})\log\frac{p({\bf y}|{\bf z})p({\bf z}|s)}{q({\bf z})}$		(1)
		$\displaystyle=$	$\displaystyle E_{q}[\log p({\bf y}|{\bf z})]+E_{q}[\log p({\bf z}|s)]-E_{q}[\log q({\bf z})],$

where $E_{q}$ refers to an expectation under the distribution $q$. Depending on the variational approximation of $q$ and the model $p({\bf y}|{\bf z})$ these expectations can be computed exactly, as illustrated by the example of the mean-field $q$ and linear $p({\bf y}|{\bf z})$ in Section B. Otherwise, a sampling method can be used, often with very few samples due to the iterative nature of the learning algorithm. In experiments (Section 5), we simply evaluate at the mode of $q$.

Report issue for preceding element

The bound is tight when the variational distribution $q$ is equal to the true posterior $p({\bf z}|{\bf y},s)$, otherwise $q({\bf z})$ serves as an approximate posterior. We refer to the three additive terms as the observation part $T_{1}=E_{q}[\log p({\bf y}|{\bf z})]$, the feature part $T_{2}=E_{q}[\log p({\bf z}|s)]$, and the posterior entropy $T_{3}=-E_{q}[\log q({\bf z})]$.

Report issue for preceding element

Given a dataset of pairs $\{s^{t},{\bf y}^{t}\}_{t=1}^{T}$, the optimization criterion—the bound on the log likelihood of the entire dataset—is

Report issue for preceding element

$$L(\{q^{t}\},\theta_{f},\theta_{e},\theta_{y})=\sum_{t}B_{t}=\sum_{t}E_{q^{t}}[\log p({\bf y}^{t}|{\bf z})]+E_{q^{t}}[\log p({\bf z}|s)]-E_{q^{t}}[\log q^{t}({\bf z})]$$

(2)

In general, maximizing the criterion $L$ w.r.t. all variables except semantic model parameters (feature descriptions) $\theta_{f}$ is straightforward: we can use existing packages or derive an EM algorithm specific to the choice of the model’s statistical distributions. The variational distributions $q^{t}({\bf z})$ depend only on the individual bounds for each data pair $({\bf y}^{t},s^{t})$ in (1) and can be fitted independently for given model parameters $\theta_{f},\theta_{e},\theta_{y}$.

Report issue for preceding element

The (approximate) posteriors $q_{t}(z_{i})$ provide the statistical signal discussed in Section 2 for tuning the semantic feature descriptions ${\theta_{f}}_{i}$: sampling them separates the samples $s^{t}$ into two groups, and exemplars from each are used to prompt the foundational model with the prompt in Fig. 2 for a new definition of the $i$-th feature. The optimization of $L$ then consists of steps that improve it w.r.t. the statistical model parameters and the (approximate) posteriors $q$, alternating with updates of the feature descriptions using those posteriors.

Report issue for preceding element

This general recipe can be used in a variety of ways, with different choices for the structure of the conditionals in the model $p$ and different approximations of the posterior $q$ leading to a variety of algorithms including variational learning, belief propagation and sampling frey-jojic-pami-tutorial. We now develop specific modeling choices and a learning algorithm we use in our experiments.

Report issue for preceding element

Binary feature model

Report issue for preceding element

As discussed, the feature model uses a prompt-based feature classification, where each feature is treated independently based on its description ${\theta_{f}}_{i}$ (Fig. 1)

Report issue for preceding element

$$p({\bf z}|s,\theta_{f},\theta_{e})=\prod_{i}p(z_{i}|s,{\theta_{f}}_{i},p^{e}_{i}),$$

(3)

with the uncertainty (probability of error) $p^{e}_{i}$, so that

Report issue for preceding element

$$p(z_{i}|s)=(1-p^{e}_{i})^{[h(s,{\theta_{f}}_{i})=z_{i}]}(p^{e}_{i})^{[h(s,{\theta_{f}}_{i})\neq z_{i}]}$$

(4)

where $h$ is the binary function that uses one of the prompts in Fig. 1 to detect the feature ${\theta_{f}}_{i}$ in the semantic item $s$. The expression above uses the indicator function $[]$ to express that with the probability of error $p_{e}^{i}$, the feature indicator $z_{i}$ is different from $h(s,{\theta_{f}}_{i})$. Denoting $h_{i}=h(s,{\theta_{f}}_{i})$, we write

Report issue for preceding element

	$\displaystyle\log p(z_{i}|s)=[z_{i}=1]\ell^{1}_{i}+[z_{i}=0]\ell^{0}_{i},$		(5)
	$\displaystyle\qquad\ell^{1}_{i}=[h_{i}=1]\log(1-p^{e}_{i})+[h_{i}=0]\log p^{e}_{i},$
	$\displaystyle\quad\ell^{0}_{i}=[h_{i}=1]\log p^{e}_{i}+[h_{i}=0]\log(1-p^{e}_{i})$

Using a factorized posterior $q=\prod_{i}q(z_{i})$, the feature term $T_{2}$ simplifies to

Report issue for preceding element

$\displaystyle T_{2}=E_{q}[\log p({\bf z}|s)]=\sum_{i}\sum_{z_{i}}q(z_{i})\log p(z_{i}|s)=\sum_{i}q_{i}\ell^{1}_{i}+(1-q_{i})\ell^{0}_{i},$

(6)

with $\ell^{1}_{i}$ and $\ell^{0}_{i}$ defined in (5). Finally, the entropy term becomes

Report issue for preceding element

$$T_{3}=-E_{q}[\log q({\bf z})]=-\sum_{i}q_{i}\log q_{i}+(1-q_{i})\log(1-q_{i}),\quad q_{i}=q(z_{i}=1)$$

(7)

General observation model

Report issue for preceding element

We can soften an arbitrary mapping function ${\bf y}=f({\bf z};\theta_{m})$ by defining a distribution:

Report issue for preceding element

	$\displaystyle p({\bf y}|{\bf z},\theta_{y}=\{\theta_{m},{\bf\sigma}^{2}_{y}\})$	$\displaystyle=$	$\displaystyle{\cal N}({\bf y}|f({\bf z},\theta_{m}),\Sigma=diag({\bf\sigma}^{2}_{y})),$		(8)
	$\displaystyle T_{1}=E_{q}[\log p({\bf y}|{\bf z})]$	$\displaystyle=$	$\displaystyle E_{q}\big[-\frac{1}{2}({\bf y}-f({\bf z}))^{T}\Sigma^{-1}(y-f({\bf z}))\big]-\frac{1}{2}\log 2\pi|\Sigma|,$		(9)

with $\theta_{m}$ being the mapping parameters of function $f$, e.g., regression weights in ${\bf y}={\bf w}^{T}{\bf z}$, and ${\bf\sigma}^{2}_{y}$ being a vector of variances, one for each dimension of ${\bf y}$. Different specialized inference algorithms for specific forms of function $f$ can be derived. For example, see the appendix for a procedure derived for a linear model (factor analysis). However, in our experiments, we use a generic inference technique using $f$ in plug-and-play fashion, described below.

Report issue for preceding element

Inference of $q(z_{i})$ in the fully factorized (mean field) $q$ model

Report issue for preceding element

Keeping all but one $q(z_{i})$ fixed, we derive the update for the $i$-th feature’s posterior by setting the derivative of the bound $B$ wrt parameters $q(z_{i}=1)$ and $q(z_{i}=0)$, subject to $q(z_{i}=1)+q(z_{i}=0)=1$. We can express the result as:

Report issue for preceding element

	$\displaystyle q(z_{i}=1)$	$\displaystyle\propto$	$\displaystyle e^{E_{q_{+i}}[\log p({\bf y}|{\bf z})]+E_{q_{+i}}[\log p({\bf z}|s)]-E_{q_{+i}}[\log q({\bf z})]},$
	$\displaystyle q(z_{i}=0)$	$\displaystyle\propto$	$\displaystyle e^{E_{q_{-i}}[\log p({\bf y}|{\bf z})]+E_{q_{-i}}[\log p({\bf z}|s)]-E_{q_{-i}}[\log q({\bf z})]}$		(10)

where $q_{+i}=[z_{i}=1]\prod_{j\neq i}q(z_{j})$, and $q_{+i}=[z_{i}=0]\prod_{j\neq i}q(z_{j})$, i.e the "+i" and "-i" posteriors have their $z_{i}$ value clamped to 1 and 0 respectively in the factorized $q=\prod q(z_{i})$. Upon normalization, as the entropy terms $E_{{\bf q}^{+i}}[\log q({\bf z})]$ and $E_{{\bf q}^{-i}}[\log q({\bf z})]$ cancel each other, we obtain:

Report issue for preceding element

$$q_{i}=q(z_{i}=1)=1-q(z_{i}=0)=\frac{1}{1+e^{E_{{\bf q}^{-i}}[\log p({\bf y}|{\bf z})]-E_{{\bf q}^{+i}}[\log p({\bf y}|{\bf z})]+\ell^{0}_{i}-\ell^{1}_{i}}}$$

(11)

Interpretation: Whether or not $z_{i}$ should take value 1 or 0 for particular semantic item $s$ depends on:

Report issue for preceding element

• •

  the assignment $h$ that the foundational model assigns it and the level of trust we have in that assignment (the probability of error $p^{e}_{i}$). The two are captured in prior log likelihoods $\ell_{1},\ell_{0}$.

  Report issue for preceding element
• •

  error vector ${\bf y}-f({\bf z})$, scaled by the currently estimated variances ${\bf\sigma}^{2}_{y}$ for different elements of the target ${\bf y}$, for two possible assignments $z_{i}=1$ and $z_{i}=0$, with other assignments $z_{j}$ following the current distributions over other features $q(z_{j})$

  Report issue for preceding element

While expectations $E_{q_{+i}}$ and $E_{q_{+i}}$ can potentially be better estimated through sampling, in our experiments, we simply compute $\log p({\bf y}|{\bf z})$ at the mode (assuming most likely values for other $z_{j}$ variables).

Report issue for preceding element

Optimizing numerical parameters of $p({\bf y}|{\bf z})$ and $p({\bf z}|s)$

Report issue for preceding element

Optimizing the bound $B$ wrt to parameters $\theta_{m}$ of the mapping in ${\bf y}\approx f({\bf z};\theta_{m})$ and the remaining uncertainty/variance ${\bf\sigma}^{2}_{y}$ reduces to maximizing sum of $T_{1}$ terms over the traing set, i.e. $\sum_{t}E_{q}^{t}[\log p({\bf y}^{t}|{\bf z}^{t})]$. Again, while sampling of $q$ might be helpful, in our experiments we simply use the mode and iterate:

Report issue for preceding element

	$\displaystyle\theta_{m}$	$\displaystyle=$	$\displaystyle\arg\min\sum_{t}({\bf y}^{t}-f({\bf z}^{t};\theta_{m}))^{T}\Sigma^{-1}({\bf y}^{t}-f({\bf z}^{t};\theta_{m}))$		(12)
	$\displaystyle{\bf\sigma}^{2}_{y}$	$\displaystyle=$	$\displaystyle\frac{1}{T}\sum_{t}({\bf y}^{t}-f({\bf z}^{t};\theta_{m}))\circ({\bf y}^{t}-f({\bf z}^{t};\theta_{m}))$		(13)

where ${\bf z}^{t}$ is the binary vector of assignments of features $z_{i}$, either a sample from $q_{t}$ or at the mode of the posterior $q_{t}$ for the $t$-th out of $T$ entries in the data set of pairs $s_{t},{\bf y}_{t}$. $\circ$ is a point-wise multiplication.

Report issue for preceding element

The semantic model has numeric parameters $p^{e}_{i}$, modeling the hybrid model’s uncertainty in semantic function $h$ that makes an API call to the foundational model. By optimizing the bound wrt these parameters we obtain the update:

Report issue for preceding element

$$p^{e}_{i}=\frac{1}{T}\sum_{t=1}^{T}q_{i}^{t}[h_{i}^{t}=0]+(1-q_{i}^{t})[h_{i}^{t}=1]$$

(14)

Thus, for a given set of semantic feature descriptors $\theta_{f}$, iterating equations (11) for each item $t$, and (12, 14) using all items in the training set, would fit the numerical model parameters and create the soft feature assignments $q^{t}$ that best balance those feature descriptors with the target real and possible multi-dimensional targets ${\bf y}^{t}$.

Report issue for preceding element

In section 2 we pointed out that individual $q(z_{i})$ distributions carry information about how the data should be split into two groups to create the mining prompt in Fig. 2. But as each of these distributions depend on the others, there is the question of good initialization. In our experiment we use a method akin to the variational EM view of boosting neal1998boosting, albeit in our case applied to learning to predict real-valued multi-dimensional targets, rather than just for classification.

Report issue for preceding element

Note that if we iterate (11, 12, 14) while keeping $p^{e}_{i}=0.5$ for a select feature $i$ the entire hybrid model and the posterior $q(z_{i})$ are decoupled from the foundational model’s prediction $h_{i}$ which depends on feature description ${\theta_{f}}_{i}$ to provide its guess at $z_{i}$. Therefore, the iterative process is allowed to come up with the assignment of $z^{t}_{i}$ for each item $t$ in such a way that maximizes the likelihood (and minimizes the prediction error). This provides a natural binary split of the data into two groups based on error vectors. In case of predicting house prices (one dimensional $y$, those two groups would be the lower and higher priced houses relative to current prediction, but in case of multi-dimensional observations, such as movie embeddings, the split would be into two clusters, each with their own multi-dimensional adjustment to the current prediction.

Report issue for preceding element

A simple algorithm for adding features is shown in Algorithm 1. The key idea is to fit a model with only features $j\in[1..i]$ while keeping the new feature $i$ decoupled from the LLM prediction by fixing $p^{e}_{i}=0.5$. This allows the data to determine the optimal split before discovering its semantic explanation.

Report issue for preceding element

The algorithm operates on the model $p(\{{\bf y}^{t},\{z_{j}^{t}\}_{j=1}^{i},s^{t}\}_{t=1}^{T})$, where features $j>i$ do not participate. By keeping $p^{e}_{i}=0.5$ fixed during the inner loop, the posterior $q_{i}^{t}$ is determined purely by how well $z_{i}^{t}$ helps explain ${\bf y}^{t}$, independent of any LLM prediction. This provides the statistical signal needed to discover meaningful features through the mining prompt.

Report issue for preceding element

As we show in experiments on the simple task of discovering binary representation of numbers, iteratively running Algorithm 1 with increasing $i$ adds new features in coarse-to-fine manner, from those that split the data based on large differences in target ${\bf y}$ to those that contribute to explaining increasingly smaller differences.

Report issue for preceding element

Conditional (targeted) feature addition

Report issue for preceding element

A possible problem with such coarse-to-fine approach, especially for nonlinear and easy to overfit functions $f({\bf z})$, is that items with different combinations of features can be arbitrarily split with the feature $z_{i}$ we are updating. For example, suppose we are working on the house pricing dataset and are running the Algorithm 1 for the second time, having added only one feature so far: $\theta_{f_{1}}=$’Located in Arizona desert communities’. Then as we iteratively look for a natural split $q_{2}$ it is possible that for items with high $q_{1}$, the procedure may split the data along a different semantic axis (e.g., presence of swimming pools) than for the items with low $q_{1}$. To deal with this, we alter Algorithm 1 very slightly: When we sample positive and negative examples, we take them from items that have other features identical. This is akin to using the structured variational approximation $q=q(z_{i}|\{z_{j}\}_{j\neq i})\prod_{j\neq i}q(z_{j})$, but instead of fitting this for all combinations of other features, we target only one combination at a time. The discovered feature $\theta_{f_{i}}$ is still used to assign values $h_{i}$ for all data samples, whether they were in the targeted group or not, so that it can be used in the next step.

Report issue for preceding element

To select the combination of features $\{z_{j}\}_{j=1}^{i-1}$ on which we condition the $i$-th feature discovery, we start with the cost being optimized in (12), from the observation part ($T_{1}$) of the approximate log likelihood based on a sample (or the mode) ${\bf z}^{t}$ for each item:

Report issue for preceding element

$$\sum_{t}\big({\bf y}^{t}-f({\bf z}^{t})\big)^{T}\Sigma^{-1}\big({\bf y}^{t}-f({\bf z}^{t})\big),$$

(15)

or equivalently, by partitioning over all observed combinations ${\bf c}$ of the other features:

Report issue for preceding element

$$\sum_{{\bf c}}\sum_{t:{\bf z}_{-i}^{t}={\bf c}}\big({\bf y}^{t}-f({\bf z}^{t})\big)^{T}\Sigma^{-1}\big({\bf y}^{t}-f({\bf z}^{t})\big)=\sum_{{\bf c}}T_{1}^{{\bf c}},$$

(16)

where $T_{1}^{{\bf c}}$ are the summands that show the total error due to each observed combination ${\bf c}$ of the other features ${\bf z}_{-i}=[z_{1},\ldots,z_{i-1}]^{T}$. We select the combination ${\bf c}^{*}$ with the largest total error ${\bf c}^{*}=\arg\max_{{\bf c}}T_{1}^{{\bf c}}$. Algorithm 2 shows the conditional (targeted) feature addition procedure. The key difference from Algorithm 1 is that positive and negative examples are sampled from a targeted subgroup $G^{*}$ (items sharing the same values for features $1$ through $i-1$), selected by identifying which combination has the largest total error. The discovered feature $\theta_{f_{i}}$ is then applied to all items.

Report issue for preceding element

These algorithms add a feature but can also be used to update a feature by removing it from the set and then adding a new one. While we can select a feature to update at random, we use a more efficient approach: we choose the feature whose removal affects the prediction (likelihood bound) the least, as shown in Algorithm 3. A single feature update becomes a removal of one feature (Algorithm 3) followed by adding a feature (Algorithm 1). Generally, we alternate an arbitrary number of feature adding steps with feature removal steps, as shown in Algorithm 4.

Report issue for preceding element

Especially exciting applications of hybrid models discussed above involve situations in which the foundational model is capable of recognizing patterns in the data, i.e., features that some items share and others do not, but the high-dimensional target ${\bf y}$ is not understood well by the model, e.g., in scientific discovery based on new experiments. However, there is value in analyzing situations where we (humans) understand the links so that we can better understand what the model is discovering. This is especially interesting when the data involves well understood semantic items, but also novel (or dated!), or otherwise skewed statistics between these items and the targets ${\bf y}$ which we want to discover, despite a potential domain shift w.r.t. what the foundational model was trained on.

Report issue for preceding element

We analyzed three datasets we expected to be illustrative:

Report issue for preceding element

• •

  Learning binary: A toy dataset with items $s$ being strings $"0","1",\ldots,"511"$, and the corresponding targets $y=\text{str2num}(s)$. We demonstrate that iterating Algorithm 1 recovers a 9-bit binary representation.

  Report issue for preceding element
• •

  Hedonic regression on house prices: A multimodal dataset from ahmed2016house, consisting of four listing images and metadata, which we turned into semantic items $s$ describing all information about a house and the target log price $y$ (Fig. 3).

  Report issue for preceding element
• •

  Cold start in recommendation systems: The Netflix prize dataset. Singular value decomposition of the user-movie matrix is used to make 32-dimensional embedding ${\bf y}$ for each movie, and the associated semantic item is the movie title and the year of release.

  Report issue for preceding element

Since LLMs can understand similarities between movies or houses and prices, there was a chance that the models could discover relevant features without our iterative algorithms. In fact, as discussed in Section 1, most prior work on using off-the-shelf LLMs to discover features useful in classification and, especially, regression focused on probing LLMs with the description of the task, rather than the data itself, and subsequent feature selection. Our baseline that represents such approaches, which we refer to as 0-shot, is to provide an LLM with an example list of semantic items, describe the task, and ask for around 50 yes/no characteristics that in the LLM’s judgment would be most useful for the task.

Report issue for preceding element

Except for the first experiment, whose purpose is to illustrate coarse-to-fine nature of feature discovery with a linear model, we tested two $f({\bf z})$ functions (softened in $p({\bf y}|{\bf z})$ through learned variances $\sigma^{2}_{y}$): a linear model and a neural network with two hidden layers (64 and 32 units with ReLU activations and dropout rate 0.1), trained using Adam optimizer with learning rate 0.001 for 100 epochs and batch size 32.

Report issue for preceding element

For the two more involved datasets (houses and movies), we used conditional feature adding Algorithm 2 with 10-20 cycles of adding 5 features and removing 2. During the process, when a discovered feature is active for less than 2% or more than 98% of the data, it is discarded, resulting in the feature count generally growing until it stabilizes towards the end of the runs.

Report issue for preceding element

Mining prompts are executed by GPT 5, while the extraction prompts are executed with GPT 4.

Report issue for preceding element

The differences and similarities of the learning curves relative to the baseline in both experiments indicate that components of the model jointly adjust to different semantics of the data as well as target statistics and their relationships to semantics. Even though the models used here (GPT 4/5) are familiar with the semantics of the data, their ability to predict the real-valued, possibly multi-dimensional targets benefits from the rest of the system and the modeling choices. For example, while a neural network better models interactions of features needed to predict house prices, the linear model is better suited for modeling movie embeddings. In both applications, the learning curves show classic behavior of iterative improvement until over-training. On the other hand, features generated by the LLMs alone are highly redundant preventing both effective prediction and overtraining. As evident in the discovered features (Section A), our system focuses on properties of the semantic items that fit the statistics of the links $s\rightarrow{\bf y}$, rather than ’conventional wisdom’ of a zero-shot approach: For houses, the system discovers that location and quality of build/remodel matters most in the data it was given for listings in 2016, while for movies, the system discovers the features important to the audience in the period 1998-2006, such as the difference between newer and older content, importance of franchises, specific actors/directors, and so on.

Report issue for preceding element