Predicting demand for new products is an important and challenging problem in marketing, especially for product categories where brand is a key driver of choice. In these settings, observed product attributes do not explain choice patterns well, which makes predicting sales of a new product as a function of marketing mix variables intractable. To address this problem, I develop a scalable framework that enriches structural demand models with large language models (LLMs) to predict consumer preferences for new brands. After estimating preferences for existing brands using a structural model, I use an LLM to generate predictions of these brand utilities from text descriptions of the brand and consumer. When fine-tuned on this data, I show that the tuned LLM is able to generalize to previously unseen brands that were excluded from the training sample. In contrast, conventional models based on text embeddings return predictions that are essentially uncorrelated with the actual utilities, and general-purpose LLMs are also uninformative. My fine-tuned LLM's predictions attain a correlation of 0.52 with the held out preferences and 17 times higher mutual information than embedding-based models; i.e. for the first time, informative predictions can be made for consumer preferences of new brands from text. To understand why the tuned LLM works, I modify various components of the LLM and measure the change in prediction error. Focusing on the LLM's ability to extract useful features from the text and its subjective prior on the mapping from the features to the outcome, I find that adapting both the features and the outcome mapping from the values initialized at a pretrained LLM improve performance. However the manner in which the parameters of the LLM are adapted are critical, which I support with theoretical analysis. Finally, I combine causal estimates of the price effect from instrumental variables methods with the LLM predictions to enable pricing-related counterfactuals. By integrating the powerful generalization abilities of LLMs with principled economic modeling, my framework enables informed decisions on optimizing the marketing mix of a new product. More broadly, this approach illustrates how new kinds of questions can be answered by using the capabilities of modern LLMs to systematically combine the richness of qualitative data with the precision of quantitative data.

We develop a general framework for improving human decisions in unstructured action spaces using language models and A/B tests. Given results from past A/B tests, we fine-tune a language model to convert worse-performing decisions into better-performing decisions. When deployed, the language model generates improvements to decisions proposed by a human. This design makes it unlikely for AI assistance to harm performance, which mitigates risks and eases adoption, and is applicable to generic business or organizational objectives. We confirm in a field experiment that our framework performs as intended, with AI-assisted decisions outperforming a human expert baseline. In 36 email marketing campaigns covering 283 million total impressions, subject lines created with assistance from our tuned model attain click-through rates 33% higher than an unassisted human. The precise measurement of treatment effects in the email marketing setting ensures that observed performance improvements are attributable to improvements in decision quality, thus validating the effectiveness of our framework. Additionally, assistance from a general-purpose language model with 30x the number of parameters fails to improve outcomes, i.e., fine-tuning is necessary, and small language models are sufficient. Overall, our results illustrate how significant gains can be achieved by integrating AI and human expertise in decision-making processes.

Many business scenarios require a decision maker to balance multiple competing objectives. Typical decision support algorithms for such scenarios require calibrating the relative weights on the objectives, but this is brittle and sensitive to misspecification. As a result, suggestions from learned policies demonstrate pathologies, appear unrealistic, and are rejected by decision makers. An alternative approach is to imitate the decisions of managers (i.e. the mapping from state to action) in the historical data, but this replicates the mistakes that managers made in the past. We develop a new approach that imputes state values that make the past decisions of managers optimal. Policies trained on this modified data perform well when deployed, which we demonstrate in an application to scheduling shift workers to match forecasted demand in fast food and retail stores. More broadly, this method is applicable to reinforcement learning problems with underspecified reward, where policy learning is infeasible because the mapping from the state to the optimal action is not uniquely identified. In many of these settings, including our case with multiple weighted rewards, inferring the state that would have made an action optimal is easier than explicitly learning the reward and allows us to learn a superior policy compared to imitation learning.

Merger simulation models are key inputs into antitrust enforcement decisions but are poor predictors of actual post-merger outcomes. The major sources of error are incorrect predictions of post-merger product positioning and faulty assumptions on firm conduct. We propose a new approach to modeling these supply side decisions that makes progress on addressing these issues. For mergers in the consumer packaged goods (CPG) category that were subject to review, we assemble a dataset of pre and post merger product positions and merging entities' stated intents during the review process. We train a large language model to predict post-merger product positioning from pre-merger data and stated intents, and we evaluate the accuracy of augmenting standard merger simulation models with this rich predictor of product positioning. This analysis is made possible by recent advances in modeling demand side preferences as a function of raw text descriptions of product positions. The contribution of this work is twofold: first, we collect a novel dataset for systematically evaluating product positioning in merger simulation models, and second, we develop a new approach for doing so that gives a more accurate assessment of the competitive effects of a merger.

Foundation models like large language models (LLMs) have shown promise in systematically optimizing unstructured marketing content (text, images) for engagement metrics (e.g. for ads, emails) based on outcomes from A/B tests. A first order concern when doing so is that the LLM will overfit to the engagement metric in an undesirable way, e.g. by learning to say false and inflammatory things. I quantify the tradeoff between performance and safety by repeatedly improving a given piece of content using an LLM that has been trained to morph content to improve engagement. I track both safety metrics and predicted engagement as a function of the number of "improvement" iterations. Starting from content that was actually used, I find initial gains in performance with little degradation in safety metrics, indicating that existing content decisions are suboptimal, but eventually find a tradeoff in later iterations. Additionally, I find that consistency with "brand voice" is a cost-effective proxy for safety, which is expensive to evaluate, and imposing this constraint in the generative model improves the tradeoff. In addition to being practically useful for deploying AI, this result suggests organizational rules can serve as a mechanism for preventing gaming of incentives and has implications for the design of incentive contracts.

Consumers' choice behavior is driven by measurable, objective attributes of products as well as unmeasured attributes (perceived convenience, quality, sportiness, etc.). Past approaches of modeling the effect of unmeasured attributes on decisions have entailed eliciting some components of a Fishbein multiattribute utility model from consumers and inferring the remaining components from observed choices. These approaches have either elicited perceptual ratings on each product in the choice set along some pre-specified dimensions and inferred the importance of each dimension, or have elicited ratings of the importance of each dimension and inferred the perceived level of each dimension. The former approach doesn't scale well to many products, as it requires rating every product on every dimension, and the latter approach still relies on an analyst selecting the correct dimensions to rate. Notably, it will fail if consumer perceptions are high-dimensional. I propose an alternative method that merges choice data (e.g. from conjoint) with responses to open-ended questions on consumers' needs (e.g. from interviews or focus groups). I use these stated needs to identify consumer attitudes in the Fishbein model, which implies that the estimated perceptions are the extent to which needs are satisfied. I show how to apply causal abstraction, a recently developed method from the NLP literature, to interpretably estimate the extent to which each product satisfies these needs. This result shows how to operationalize the vast literature on laddering by integrating it into models of choice in an interpretable way. It is also informative for decisions on changing product attributes to better meet consumer needs or for crafting marketing content strategies that emphasize these needs.