You're gonna carry that weight

In my last post, I presented data that suggests that the odor code dramatically changes between the first and subsequent breaths. Later, however, I discovered a subtle mistake (to me at least) in my analysis, which slightly changed the result. Too often in science, we see the end point of research, and don't see how it evolved. Today I'm going to show how I found my mistake, what the mistake was, and how fixing that mistake modified the result.

Calculating spike distance

Previously, I argued that the odor codes evolves over multiple breaths. After presenting some example cells where the odor code shifted between the first and second breath, I turned to the population level, and showed this figure:

A. Schematic of population vector. B. Distance between breaths. Breath identity shown underneath.

On the left is the schematic for the analysis. For each cell, I binned the response to an odor over a breathing cycle into 8 bins. To add more cells to the population, I added 8 bins for each cell to the bottom of the population vector. To get different observations, I repeated this for each breath that I recorded.

It's possible to calculate the "population spike distance" by just calculating the Euclidian distance between the population vector for each breath. Yet that method is fairly noisy, as most cells are uninformative. When I tried that simple method, the result was similar to that shown above, but not as clean. To make the differences more obvious, I performed PCA on the population vector, and then calculated the distances between the first five principal components (shown above). Here, each of the five components was informative, and the distances between the breaths were much clearer.

The problem discovered


After looking at different breaths for the same odor, I next wanted to investigate the differences between odors. Rather than look at "spike distances," though, I used a prediction algorithm.

To do the prediction, I built population vectors as I did above, but instead of observing different breaths for the same odor, I observed different odors for the same breath. Once again, I transformed the data via PCA and took the first 5-10 principal components. I then created a sample population vector for an individual trial, and calculated the distance between the sample vector and the average vectors for each odor. The "predicted" odor is the one with the smallest distance from the trial. This was repeated for all trials to get the average prediction rate. When I did this, I got the prediction rates show in the top panel:

n= 105 neurons, and 6 odors, split between two sets of three odors. 10-12 trials.

Here, the predictions are between three different odors, so the chance level is 33%. The first five breaths are pre-odor breaths, while breaths 6-10 are during the odor. As you can see, the odor breaths are >95% correct, which is great. However, many of the pre-odor breaths have prediction rates >50%, which is obviously bad (I have different control breaths for each odor; the pre-odor prediction chooses among the three control breaths).

In playing with the data, I then noticed something odd: as I increased the numbers of bins or principal components that I used to make predictions, the pre-odor predictions got higher (panel B above). With 20 bins and 20 PCs, I could get pre-odor predition rates of >70% for each odor! I wasn't making predictions, but was over-fitting my model to the data so that it could never be wrong!

And this is where I realized my mistake. When you do PCA, the algorithm tells you how much each component describes the variance in the data. The first few principal components (PC) account for most of the variance, while the later components account for less. When I was doing my prediction algorithm (and my distance calculations above), I was weighing each PC equally, and over-representing principal components which didn't have much meaning.

Once I realized this, it was a simple procedure to weigh each PC according to its variance, and re-run the prediction (panel C above). Following that, the pre-odor predictions are at chance; the positive control is finally working. The downside to this correction, however, is that the odor prediction during the odor was now between 60-80%. This lower predictive ability makes more sense, though, given the trial-to-trial noise in the signal, and the relatively low number of  neurons in the population vector.

Back to breath distance


Having realized my error, I returned to my original analysis on the breath distance, and added the proper weightings. When I did this, the results were slightly different:

n=11 odors from 5 experiments with >15 cells.

The control breaths are still quite distant from the odor breaths. However, the first breath is no longer so different from the subsequent breaths. Indeed, it appears that there is an evolution in the code over the first few breaths before the code stabilizes. The stark difference between the different breaths had blurred.

I'm guessing that this is a tyro analysis mistake. I only stumbled upon it because I figured a reviewer would want to see pre-odor prediction rates to compare to those during the odor. I know that when I read a paper, I rarely delve into the detailed methods of these more complicated analyses. And if I do, they aren't always informative. Given how often people perform analysis by themself, with custom code, it's easy to forget how many simple, subtle mistakes one can make. The only way to avoid them is to gain experience, and to constantly question whether what you're doing actually makes sense, and agrees with what you've already done.

Update:


Today I found an even BIGGER problem with my odor prediction. When I was creating my "average spike population" I was including the test trial in the population. And I was once again getting pre-odor prediction rates near 100%. Excluding the test-trial from the average population made everything MUCH more sensible.

The odor code for the first breath is different from subsequent breaths

As I mentioned last post, I have been recording from the olfactory bulb of awake, head-fixed mice. In general, the responses I'm seeing are in accord with those reported by Shusterman and Rinberg: about half of cell-odor pairs respond to a given odor. In trying to quantify these responsive cell-odor pairs, I stumbled upon another finding, that the odor code for the first breath is different from the rest.

One cell's response

(Brief methods: To look at whether a given breath is responsive, I segmented the recordings into breaths, and fit each breath to a standard breath length (if a given breath was longer than the average breath length, I deleted all spikes after the end of the standard breath; if a given breath was shorter, I assumed the rest of the time included no spikes). To quantify whether breaths were "responsive," I compared a breath's tonic firing rate to the control, pre-odor breaths (using ANOVA with p<0.05, and Tukey's post-hoc testing); and I tested whether the "phase" or timing of the breath differed from the pre-firing rate (using a Kolmogorov Smirnov test; here I used p<0.02 as the threshold for significance as using p<0.05 yielded many false positives when comparing different control breaths). And when I looked at the data, it was obvious that some cells had strikingly different codes for the first breath versus later breaths.)

One example is shown below (this is the same neuron-odor pair from the previous post, albeit different trials). The top panel shows the PSTH of the cell's response to amyl acetate, with 40ms bins. You can see that before odor presentation, the neuron fired irrespective of phase. During the first sniff of the odor, there was a strong, transient burst of activity in the middle of the breathing cycle. However, in the subsequent sniffs, the cell was inhibited.

This cell is excited during the first breath before becoming inhibited. (top) PSTH of the response with 40ms bins. The odor is applied at t=0s. Blue dashed lines represent inpiration. (middle) PSTH of the response with a single bin for each breath. The cell is inhibited during breaths 2-4. (bottom) Cumulative distribution of spikes during the ctl breath (black), first breath (blue), and second breath (red).

While seeing the difference by eye is nice, I wanted to test these withous bias and quantitatively. To detect tonic changes, I averaged the firing rate for each breath, as shown in the middle panel. In this example, over the whole breath, the first sniff's firing rate does not differ from the control breaths'. However, the three subsequent breaths are all inhibited.

To look at the phasic changes, I plotted (for ten trials) the cumulative spike times for control breaths and breaths during the odor (bottom panel, above). Before the odor, the spikes occur without phase bias (black line), while during the first breath you can see that most spikes come between 150-200ms of the breathing cycle. However, on the second breath, the phasic nature of the response has begun to dissipate.

Population changes in the odor code

The transformation between the first and second breath can take many forms. The example above shows a neuron that switches from a strong, phasic, excitatory response to an inhibitory response. Many other neurons are inhibited during the first breath but not afterward. Below is a more subtle example, where the neuron does not respond to ethyl butyrate during the first sniff. However, on the subsequent sniffs, the timing of the response shifts to earlier in the breathing cycle.

This cell appears to not respond during the first breath, but has a phasic response during later breaths.

Rather than exhaustively quantify how individual cells change their code between the first and second breath, I took a different approach and looked at the population code. I have three experiments where I was able to record from at least ten cells at the same time. For these three experiments, I created a population vector of the responses to odor, and calculated the "spike distance" between the representation of each breath. Another way to think of spike distance is how dissimilar two representations are: small spike distances imply similar population representations. All distances were normalized to the average distance between control breaths.

The odor code for the first breath is as different from control as it is from the 2nd breath. A. For each breath during an odor, I created a "population" vector, where for each breath and cell, I broke the response into eight bins. To reduce the dimensionality, I performed PCA. To calculate the distance, I used the first five PCA scores for each breath. B. Normalized "spike distance" between odor code for different breaths. All distances were normalized to the average distance between control breaths (C) for an experiment-odor pair. All post-odor breaths are distant from the control breaths (white). The first breath is also different from the 2nd-4th breaths (blue). However, the 2nd-4th breaths are relatively similar to each other (red).

The population response during all of the odor breaths are distant from the control breaths (white bars). The distance here presumably encodes the presence of the odor. However, if you look at the distance between the first breath and subsequent breaths (blue bars), you can see they are also quite far apart. In fact, the first breath is almost as distant from the other breaths as it is from the control breaths. In contrast, the distance between breaths 2-4 is much lower, and almost comparable to the distance between control breaths.

When I showed this to my boss he was not impressed, and said they had already showed this in a previous paper. And indeed, buried in three panels of Fig. 4, they did show something similar (below). There are some significant differences, though. First, those experiments were in anesthetized animals, rather than awake animals. Second, I've shown that individual cells use strikingly different codes between breaths. Third, they did not create their population vector to consider a cell's firing as a whole. This could change the interpretation of the results. In any case, asking around the lab, no one seemed to remember this was even in the paper.

The velocity of the population representation is highest during the first odor and post-odor breaths. A. The population vector contains the firing of each cell in a given time-bin. B. Cross-correlation and distance for the population during a given odor. C. The velocity of the population vector (how much the distance changes) is highest at the beginning and end of odor presentation.
From Bathellier, et al, 2008.

Given this result, one needs to be careful with how one characterizes "responsive" cells in the olfactory bulb. First, when determining whether a cell-odor pair is responsive, one needs to always look at more than just the first breath. For example, the second cell shown above was responsive during the second breath, but not the first. Second, when characterizing responses, it is difficult to say whether a cell was excited or inhibited, as cells can be both excited and inhibited in a given breath, as well as be excited for one breath but not others. While it may be unsatisfying, it is probably best to just call them "responsive" cells.

These results also gave me an idea for an experiment to test whether the difference in coding is perceptually important. It is now possible to stimulate the olfactory epithelium via Channelrhodopsin while mice sniff (using an OMP-ChR2 line), which makes it possible to mask an odor response with olfactory white noise. To test how important different sniffs are to perception, you would start by establishing the detection threshold for an odor. Then you could measure the detection threshold while masking either the first or second sniff with the olfactory noise. There are a few possible results. First, the threshold might not change at all, as both the first and second sniff contain sufficient information to detect an odor. Second, the sensitivity could be equally decreased when either sniff is blocked. This would also imply their is equal information in each sniff. The third possibility is that masking the first sniff would decrease sensitivity far more than masking the second sniff (which is what I expect). It has been shown that mice and humans can detect odors in a single sniff. And in daily life, no odor is as strong as its first whiff. The difference in odor coding between the first and second sniff might be one step towards explaining why.

While this is pretty basic analysis, I had to perform this en route to doing more sophisticated comparisons while trying to measure a form of plasticity in the odor code. This is also the first complete data I've shown from this lab. I would appreciate any feedback on this, as it's always useful to get a perspective outside the insular confines of a lab. Were the figures legible? The analysis convincing? Or is this entirely un-novel?

Don't Warp Yo Breath

In my last post, I briefly reviewed a paper from the Rinberg lab where they recorded from the olfactory bulb of awake, head-fixed mice. When they were analyzing their neurons' responses, they performed a time-warping manipulation on the data that increased the precision of the responses. Today I'm going to present some counter-evidence that shows why their time warping is a bad idea.

Time Warping of Neuronal Responses

The first figure of their paper clearly explains how they time-warped their data. They recorded extracellularly from mitral cells in the olfactory bulb while presenting head-fixed mice with odorants. In their recording, as in previous studies, if they did not perform any temporal alignment, they saw very weak responses in response to odorant application (black traces, below). However, it is well known that sniffing can influence olfactory bulb activity, so they realigned all of their mitral cell activity to the first inhalation following odor onset (blue traces, below). When they did this, they found that the mitral cell responses were quite strong, and around 59% of odor-cell pairs were responsive.

Aligning responses to inhalation reveals odor responses. a. Diagram of inhalation alignment, and temporal warping. Odor presentation in yellow. c. Spike rasters to odor under three alignment paradigms: to odor (black); to inhalation onset (blue); and time-warped (red). d. Peri-stimulus time responses(?) of the responses in c.
From Shusterman, et al, 2011.

Not satisfied with the precision of their responses, they performed one more manipulation. The breathing cycle, while fairly regular, does vary; and they reasoned that duration of the breathing cycle could influence neuronal activity. To normalize this, they fit curves to both inhalation and exhalation, and then stretched time (and moved spikes) until the breaths fit a standard breathing cycle (red traces, above). When they did this, they found that both the precision and magnitude of responses were increased.

This did not sit well with me for three reasons. First, if you take the perspective of a neuron in the olfactory bulb, it means that the neuron has to somehow keep track of where it is in the breathing cycle, not in terms of time, but in terms of phase. To do this, 50ms following inhalation, a neuron has to know when the next inhalation is going to come. They're psychic! [Update: a commenter noted that the OB could receive an efference copy from brainstem respiratory centers. I am not aware of any evidence that it does, however. Which of course does not mean it does not exist.]

Second, I think that the timing of mitral cell responses is in large part dictated by the temporal dynamics of the olfactory epithelium. The olfactory epithelium, in turn, has its kinetics determined by the taus of the G-protein signaling cascade, and the concentration of odorants in the epithelium. The kinetics depend on inhalation onset and intensity, not phase.

The third reason I have a problem with time-warping is that I have counter-evidence

Mitral Cell Response Timing is Independent of Breath Length


While I wait for my mice, I have been performing head-fixed recordings from the olfactory bulb of awake mice. In general, I've been getting population responses in line with what the Rinberg lab has suggested: ~50% of the odor-cell pairs in the OB differ from baseline. Like the Rinberg lab, the Carleton lab aligns responses to inhalation onset. Yet, unlike the Rinberg lab, we have not performed any time warping.

After seeing the Shusterman paper, I took a closer look at my data, to see whether time-warping makes sense. In general, respiration is regular enough that time-warping would little effect the responses. However, I found a few cases where time warping is a bad idea.

Below is a raster plot of the firing of one neuron in response to amyl acetate at 20x dilution. I have zoomed in on the first second following odor onset. Inhalations are denoted by blue lines, while spikes are shown in black. We trigger odor delivery by waiting for an exhale, which is why the inhalation times are non-random. Following the first sniff, you can see that this neuron fires vigorously, with some delay. (I should say that this response is easily one of the highest firing rates in my data set.)

Raster plot of a neuron's response to Amyl Acetate. Amyl Acetatate began application at t=6s. Blue lines  are inhalations, black lines are spikes. Ten trials shown.

We can also align this data by moving time such that the first post-odor breaths all occur at the same time. If you do this, you can more clearly see the strong response to the odor. This response is fairly long, and has a high firing rate (>100Hz at its peak).

Same raster plot as above, except aligned to the first inhalation following  odor. Here the response is much clearer.

And now I can finally address the issue of whether time-warping is a good idea. In the example above, there are two trials with short breaths, trials 6 & 7; and two trials with long breaths, trials 8 & 10. Despite the different lengths of these breaths, both of these trials have high amplitude firing rates between 6.2 and 6.3s. If you were to time-warp these trials, you would be moving the spikes from trials 6&7 later in time, and the spikes from trials 8&10 earlier. Both manipulations would cause a decrease in precision.

This is just one extraordinary example, but it shows that time-warping can have deleterious effects on precision. In my view, if you are recording from the olfactory bulb, you should align all your responses to breath onset, and truncate your breaths to the same standard breath.

I hope you are convinced that time-warping your data is a bad idea for mitral cells in the olfactory bulb. If I've missed anything, please let me know in the comments.