Overview and Demo

Conjoint surveys are commonly used to estimate consumers' willingness to pay for product attributes, like RAM or disk space in a laptop. I recently read some cool papers that used mouse-tracking data from these surveys to improve predictions and reveal mechanisms for how people make choices (e.g. comparisons within vs across products).

Since Qualtrics surveys are easy to distribute, I was curious whether it was possible to collect mouse-tracking data directly in the browser, within Qualtrics. It turns out it is! Try my demo below -- hover to unblur, and click a button to choose an option. When you unblur a product attribute, the duration of unblurring and the current time are recorded. Refresh the page to clear the tables.

How it works

Product attributes are blurred by setting their filter CSS property to blur. When the attribute is hovered over, blurring is disabled by setting filter to none using the CSS pseudoclass :hover. That's all you need to implement blurring/unblurring!

To store the data from unblurring, listeners for the mouseenter and mouseleave events are added to each product attribute using Javascript. These record the timestamp and duration of the hover and add it to the browser's sessionStorage. The event listeners are triggered at the same time as the CSS hover, so the visual appearance and recorded data are consistent.

When the user goes to the next question in Qualtrics, the choice set, hover events, final choice, and Qualtrics survey ID are sent to an external database. It was pretty straightforward to get this working with Firebase. This data can be merged with other data from Qualtrics and analyzed.

If you implement this yourself, just a heads up that there is one tricky part when recording hover durations due to mutable state. If there's demand for it, I can add more details to this post or release a Github repo, just let me know.

Applications

What can we do with this? Here are some preliminary ideas.

A mechanical question that might still be interesting: how early in the event sequence is the final choice predictable?

If this can be done, we can look for jumps in the choice probability that occur for specific reasons, e.g. attention to a particular product attribute. And more interestingly, we can measure the extent to which choices can be steered by drawing attention to that attribute (e.g. unblurring it), kind of like what was done in this paper. Another possibility is to introduce an "attention friction" by adding a delay before an attribute is unblurred.

The mouse-tracking papers I've seen estimate models for the probability Pr(Choice | Events). Given a complete event sequence, they predict which product is chosen. This is not ideal if we want to predict choice probabilities from an incomplete event sequence. We'd need to simulate the remainder of the event sequence until it terminates, which seems like it'd add a lot of noise, then predict the choice.

I think that learning Pr(Events | Choice) solves this problem. Very powerful autoregressive models can be trained from masking the event sequences. For each observed choice and event sequence, cover up some of the events and train the model to predict them. (This is the same idea as training language models to predict the next word.) Then we can predict choices from incomplete sequences via Bayes rule:

Pr(Choice|Events) ∝ Pr(Events|Choice) Pr(Choice)

Pr(Choice) can be obtained by estimating a standard choice model (e.g. logit) on the observed choices alone, without mouse-tracking events.

Takeaways

What does learning Pr(Events | Choice) buy us?

1. Ability to compute Pr(Choice | Events) for incomplete event sequences.

2. Given an incomplete event sequence where the choice hasn't been observed yet, can simulate the rest of the sequence that makes a specific option more likely.

(1) can reveal what kinds of information make choice probabilities jump. (2) might suggest interventions that can steer someone towards one of the options.

Implementation of this kind of study would run in two phases. The first phase would distribute a survey and collect a bunch of data in order to learn a model for Pr(Events|Choice). The second phase would test some implication of the model, e.g. whether choices are steerable.

Anyway, I made this more for fun than to answer a specific research question. Feel free to use it for your own projects, and I'd also be happy to talk if anyone wants to collaborate.