Mind reading machines have been a favorite of science fiction stories for years. But in recent years, these machines have started to become a reality. With recent advances in applying machine learning to brain imaging data, it is now possible decode what a person is perceiving, imagining, thinking, or remembering by looking at the vast and complex pattern of activity distributed across the entire brain. The resolution of what we can currently detect is limited. For instance, we can very confidently tell whether you are imagining a face or a place. However, trying to figure out which particular person you are imagining, or which specific location you are imagining, is not very reliable yet. Nonetheless, the introduction of this methodology is revolutionizing cognitive neuroscience. As we see continued improvements in imaging resolution and improvements in algorithms, the resolution of mind reading is destined to improve dramatically in the future.

Memories “replay” in the brain

When you look at a cat, that cat elicits a unique pattern of activity in your brain. If you then look at a shoe, there is a very different pattern in your brain. That’s how you know you are looking at a cat and not a shoe: they have different neural fingerprints. Now what happens when you remember the cat? It turns out that you replay that same pattern of activity in your visual cortex. In other words, even though you aren’t looking at the cat, your visual cortex is acting as if you were looking at the cat. Your visual cortex is able to regenerate what is no longer in front of you. This is how you are able to see the cat in your mind’s eye even when the cat is not in the room. This is one of the core mechanisms of memory recall in the brain. In what follows, I’m going toexplain how we are able to use machine learning to measure memory replay in the brain and how this can shed light on the variability we see in memory health late in the life span.

The data

First, let me tell you a little bit what our data looks like. We put people in an MRI machine and have them recall faces and places. The MRI machine gives us a 3D movie of their brain activity. Every second we get a high resolution 3D image of brain activity, with hundreds of thousands of data points. Each data point corresponds to a little 3D pixel, or voxel, in the brain. Keep the person in there for an hour or two, and you have several gigabytes of data for each person. The data itself is messy. Things like head motion have to be carefully modeled and corrected for. Once that nasty work is done, the scale and complexity of the data is well suited to machine learning.

How mind reading with machine learning works

Lets start with trying to figure out whether the person in our experiment is currently looking at a face (e.g., Marilyn Monroe) or a place (e.g., Taj Mahal). Of course, the experimenter could just look at the screen or, failing that, look at the log file recorded during the experiment. But that would not be mind reading, which is what we are trying to do here. Instead, we are going to use a machine learning algorithm to tell us what the person is looking at.

We will train our model on a series of examples. Each example corresponds to a time point in the experiment where we presented a picture. So, we might give the model 50 examples of a face brain pattern and 50 examples of a place brain pattern. The model will then be able to look at a new pattern of activity and decide whether it looks more like a face brain pattern or a place brain pattern. Once it has done that, we have a mind reading machine, albeit one with a limited repertoire.

But how does the model do that? Each example brain pattern contains hundreds of thousands of voxels, which can be thought of as features or attributes that a model can use to make its decision. For instance, I could ask you to guess whether someone is republican or a democrat based on attributes like their age, gender, income, and city of residence. If someone is 22, female, middle class, and lives in San Francisco do you think that person is a republican or a democrat? The challenge here is deciding which attributes are important and how the attribute maps onto the prediction. For instance, does age matter? If so, are people more likely to be republican if they are older or vice versa? With brain scanning, we do precisely the same thing. The level of activity in each little voxel is treated as an attribute that we can use to make our prediction. So, we have hundreds of thousands of attributes. The model learns to decide which brain regions are important and how they map onto the prediction.

A simple two-dimensional mind reading model that you can visualize

Hundreds of thousands of attributes are difficult to visualize. But if we had just two attributes that’d be much easier illustrate. We will accomplish this by finding 500 voxels that are highly active for faces and then average them to create a single attribute for “face brain regions”. We will do the same thing to create a “place brain regions” attribute. We’ve potentially lost a lot of information here, but we will have a simple model that is easy to visualize. We will then train the model using only these two attributes and then test the model on separate data that the model has never seen before. This is what the model looks like for a single representative person in our experiment.

On the x-axis we have our first attribute, activity in “face brain regions”. On the y-axis we have our second attribute, activity in “scene brain regions”.  Each blue dot represents a time point in the experiment where we showed the person a face. The green dots correspond to pictures of places. The model has to figure out if both features matter and how they map onto the decision. The model accomplishes this by finding a decision bound on the graph, which is represented by the black lines. In this case, the decision bound is basically saying: if you have have greater activity in face regions than place regions, then the person is looking at a face. Conversely, if you have greater activity in place regions that face regions, then the person is looking at a place. In this case, we are doing very well; nearly all of our examples are being classified correctly. The reason we are able to do so well is because we are making a very coarse distinction between faces and places. If we tried to do something more subtle like deciding whether the person was looking at Marilyn Monroe or Julia Roberts, we would have a lot more difficulty.

As an aside, you might be wondering why there are several lines, instead of just one. The reason is that we have several models. Our data is split up into 5 different scans. We train the model on 4 scans and then test it on the left out scan. But which scan we test it on is arbitrary, so we give each scan a turn. Thus, we’ve estimated a total of 5 models. Fortunately, they all converge on very similar decision bounds.

How we use mind reading to measure the replay of memories in visual cortex

So far, we’ve been predicting what a person is currently looking at. Our model has learned to detect a particular brain pattern for faces and a distinct brain pattern for scenes. By applying the same model to memory recall, we can see if these same patterns are present when people are recalling faces and scenes. This is what our simple 2D model looks like when applied to recall in the same person.

Here we are only looking at cases where the person correctly remembered the picture. This person remembered more faces than places, so that’s why there are more green dots. We trained a single model on all of the data from when the person was looking at the pictures and then applied it to separate recall data, so this time there is just one decision bound. You can see that we aren’t doing nearly as well as we were before: there are lots of green dots on the left side of the decision bound and lots of blue dots on the right side of the decision bound. Nonetheless, we are reliably above chance (68% correct). This suggest that, for at least most of the pictures, this person’s visual cortex was “replaying” a pattern similar to the pattern seen during perception.

If our model is really measuring neural reactivation of memories, then we might expect it to correlate with memory performance. People with really good memories should have stronger reactivation and people with poor memories should have weaker reactivation. To get at this question, we’ll correlate people’s memory for the pictures with the performance of our model. For the rest of our analyses, we’ll throw out our relatively primitive 2D model and use all of the information available in our data.

In this plot, each data point is a person in our experiment. As you can see, there’s a pretty strong correlation: people with high memory performance also have high neural reactivation, as indicated by high performance from our machine learning model.

Understanding the decline of memory performance with age

Given that we think our machine learning model is doing a reasonable job of picking up on the neural reactivation of memories, we can now use it to probe the decline of memory with age. It is common knowledge that our memories tend to decline with age. As we get older, we tend to get more forgetful. The current experiment is no exception. The next plot shows how memory for the pictures in our experiment declines with age in our sample of people 60-90 years old.

It’s clear that memory performance is declining with age, which is not much of a surprise. What’s also interesting is that there is quite a bit of variability in performance even among the people that are in their 60s. Some people in their 60s are performing as poorly as people in their 80s. How do we account for this? One possibility is that chronological age per se is not really the critical factor. What matters most is how well you are able to reactivate your memories at a neural level. For some people, that ability declines relatively early in life. Other people have brains that stay healthy for longer. If that is the case, then we should be able to use our neural reactivation measure to account for a lot of the variability in memory performance that is not already accounted for by age. In the next plot, we have statistically removed the influence of age on both measures.

As you can see, there is still a reasonably strong correlation between memory performance and neural reactivation, as measured by our model. So, regardless of a person’s age, the degree to which neural reactivation is preserved and healthy is a critical predictor of how well someone can remember things.

Our study is in progress. In addition to collecting and analyzing hundreds of more people, critical next steps will include localizing our neural reactivation measures to distinct regions known to be involved in memory and relating neural reactivation to atrophy, genetic risk for Alzheimer’s, and molecular markers of Alzheimer’s progression.

The potential for mind reading provided by machine learning algorithms is a lot more than a cool trick — it is a compelling measure of mechanisms that are critical to memory function that are otherwise difficult, and sometimes impossible, to measure without machine learning. But the application of machine learning to neuroimaging is not limited to memory. These mind reading algorithms are being applied to just about every cognitive function we study with neuroimaging. Still in its infancy, we have already seen this technique mature dramatically in the last several years. With coming improvements in imaging technology and analytic techniques, mind reading machines will be a reality in our future.

AuthorScott Guerin