Journal of Vision - Color opponent neurons in V1: A review and model reconciling results from imaging and single-unit recording, by Schluppeck & Engel

The signals in visual cortex that ultimately give rise to color perception remain poorly understood. Controversy has particularly surrounded one aspect of color's encoding in the visual system—opponent processing in primary visual cortex. Early single-unit studies suggested that V1 contains relatively few color-opponent neurons. Neuroimaging measurements, however, have suggested that such neurons might be relatively numerous. Here, we reconcile these apparently discrepant results and conclude that V1 contains relatively large numbers of color-opponent neurons. We first review results from each method and find that most neuroimaging studies provide evidence of substantial color opponency in V1, and that despite apparent controversy, most single-unit studies agree that relatively large numbers of V1 neurons show some sort of color opponency. To reconcile the results from different techniques more formally, we used electrophysiological data to predict the outcomes of neuroimaging experiments. We simulated the expected fMRI response in V1 to spatial patterns of different color, based on the neurons’ properties, as reported in Johnson, Hawken, and Shapley, (2001). The simulated responses to stimuli used in Engel, Zhang, and Wandell, (1997) agree well with the actually observed fMRI results. The model identifies several factors that led to the apparent discrepancy between techniques, and makes testable predictions about how these factors influence the magnitude of color-opponent signals. fMRI and single-unit data converge to show that large numbers of color-opponent neurons exist in V1.

Neurons in primary visual cortex (V1) are jointly tuned for a wide variety of stimulus properties, including retinotopic location, orientation, direction of motion, spatial and temporal frequency, binocular disparity, eye of origin, and wavelength. For many of these properties, details of the cortical representation have been well established. How V1 encodes information that supports the perception of color and brightness, however, remains controversial.

The precortical processing of color signals is relatively well understood (though not without its own controversies). The retina encodes spectral properties of light using three classes of cones that respond preferentially to long (L), middle (M), and short (S) wavelengths. In a second stage, information from the cones is then combined: An L-M color-opponent neuron, for example, responds to the relative amounts of long and middle wavelength light as encoded in the responses of the L and M cones. . In primate retina and LGN, at least three classes of such cells are found: (a) “red-green” neurons, that respond to differences in L and M cone inputs (e.g., +L-M), (b) “blue-yellow” neurons, that roughly compute +S -(L+M), and (c) “light-dark” or luminance neurons, that combine L and M inputs as +L+M (see, e.g., Derrington, Krauskopf, & Lennie, 1984; Reid & Shapley, 1992). The first two classes of neurons are referred to as cone- or color-opponent, since they compute differences of cone signals. We will use the term “color-opponent” here to refer to such neurons, although their relation to perceptual opponency remains unclear. An important aspect of this early processing is the conversion of inputs to an approximate contrast representation.

In cortex, the representation of color is less clearly understood. One source of controversy concerns the number of color-opponent neurons in V1. Some classical electrophysiological results suggested that the majority of neurons in V1 were luminance cells tuned for orientation. A smaller population of color-opponent neurons was found (Livingstone & Hubel, 1984). More recent results, however, suggest that the number of color-opponent neurons may have been underestimated. These reports emphasize that many neurons are color-opponent, but with unbalanced cone inputs, leading them to respond somewhat to luminance, e.g. stimuli producing equal, same sign L and M cone signals (Lennie, Krauskopf, & Sclar, 1990; Johnson et al., 2001; Thorell, De Valois & Albrecht, 1984). Adding to the controversy are a number of recent neuroimaging studies that find larger responses in V1 to stimuli that are preferred by color-opponent cells than to stimuli that are be preferred by luminance cells.

The goal of this paper is to reconcile these apparently discrepant findings in order to arrive at a general conclusion about the number of color-opponent neurons in V1. We first review the literature to determine whether there is consensus among the results of each methodology. Most imaging studies that are informative about color-opponency find strong opponent signals in V1, while most electrophysiological studies find that only a minority of cells in V1 are color-opponent. We then reconcile results from the two methodologies using a simple model of V1. Our modeling results suggest that single-unit and fMRI data are not, in fact, discordant, and that there is substantial neural color-opponency at the level of V1.

Neuroimaging techniques may be a useful tool for addressing the prevalence of opponency because they pool across large samples of cells. Many imaging studies have addressed the representation of color in the visual cortex, but most have concentrated on localizing the “color center” in the brain, and so have chosen paradigms that are suboptimal for addressing issues of opponency. We begin our review with the few studies that were explicitly designed to measure color-opponent signals in V1.

Importantly, the total cone contrast for the two stimuli was the same. Researchers who assume that early in the visual system cone signals are normalized relative to local average responses, often represent their stimuli in terms of cone contrast. Cone contrast accounts for this normalization and is defined as the difference between each cone’s response (r_e = [L, M, S] in cone excitation space) and the local mean response (r₀ = [L₀, M₀, S₀]) for that cone class, divided by the local mean response (r_c = [(L - L₀)/L₀, (M - M₀)/M₀, (S - S₀)/S₀]).

Interpretation of this experiment is not entirely straightforward, because responses to L-M stimuli are not guaranteed to arise solely in color-opponent neurons. They might reflect, for example, activity in neurons whose response is simply proportional to the stimulus L cone contrast. Such neurons, however, should never be more active to patterns containing L-M cone contrast than to patterns containing L+M (luminance) cone contrast, given that the two patterns are of equal contrast and contain both positive and negative modulation. Greater response to L-M than to luminance, given equal total cone contrast, can only occur when color-opponent neurons respond to the stimulus.

In Kleinschmidt et al. (1996), primary visual cortex was much more active for the L-M stimulus, than for the L+M stimulus. These results provide evidence for relatively large color-opponent signals in V1, at least under the particular set of stimulus conditions used in the experiment. A second region in the collateral sulcus showed a similar pattern of activity; this area corresponds roughly to the ventral color selective areas found in other studies.

A more detailed fMRI experiment (Engel et al., 1997) further characterized the color signals in early visual cortex (V1/V2), by measuring more comprehensive color tuning functions. Subjects viewed a large number of different colors at many different contrasts while the amplitude of corresponding increases in the fMRI signal was measured. The stimuli were radial checkerboards that reversed their contrast at 2Hz, 4Hz, or 10Hz. Colors were chosen to sample many points in color space, and included ones that optimally stimulate L+M and L-M neurons. From the response to several different stimulus contrasts, the researchers interpolated for each color the amount of cone contrast needed to generate an fMRI response whose size was half the maximum obtained for any color. If this set of stimuli is plotted in a coordinate system where the two axes represent cone contrast, color tuning curves (or iso-response plots) can be obtained (see Figure 5 ). If the cone contrasts needed to reach the criterion level of fMRI response are high, it implies that the responsiveness of cortex is low for that particular color. Conversely, if only small cone contrasts are needed to produce the criterion level of fMRI response, it implies high sensitivity for that color.

The fMRI results showed strong responses in V1 to L-M stimuli. The amount of L-M cone contrast needed to produce the criterion level of fMRI response in V1 was only a fraction of the required L+M contrast. This implies that the overall responsiveness of V1 is much higher to L-M stimuli than to L+M stimuli. The authors suggested that such results are difficult to reconcile with models that require only a small number of red-green color-opponent neurons.

In another study, Engel and Furmanski (2001) directly compared the responses in V1 to L-M contrast and L+M contrast. V1 responses to L+M and L-M stimuli were equal, even though the former stimulus contained roughly twice the total cone contrast of the latter. These results again suggest that for the chosen stimulus configuration, a relatively large population of neurons combined cone signals with opposite sign.

Many other studies have addressed the representation of color in the human cortex, albeit with markedly different hypotheses and methodology. We will consider each of these experiments briefly to determine whether they allow us to draw conclusions about the prevalence of color-opponent neurons in V1. In general, these studies show some evidence for color-opponency, and their results are summarized in Table 1.

The aim of experiments performed by McKeefry and Zeki (1997), Hadjikhani, Liu, Dale, Cavanagh, and Tootel (1998), and Zeki and Marini (1998) was to localize foci of activation in ventral areas of occipital and temporal cortex involved in the processing of color information—the so-called “color center.” To this end, they compared activity in the cortex when subjects viewed light-dark patterns and color patterns of the same spatial composition. The reports of these studies concentrated on the ventral cortical areas, but more interestingly, from our point of view, in all of these papers there was also evidence for stronger responsiveness of V1 to chromatic than to achromatic stimuli.

In all these studies, isoluminant patterns were used in the chromatic condition. In isoluminant patterns (also called equi-luminant) the sum of L and M cone signals are kept constant across the image. In converting these patterns to cone contrast, the constant sum of L and M signals is removed, leaving a pattern where the L cone contrast plus the M cone contrast is zero, or L=-M. Thus, isoluminant patterns contain opposing L and M cone contrast. The achromatic stimuli differed slightly between studies, but they can reasonably be expected to have mainly same sign (L+M) contrast, visible to luminance neurons. As described above, greater response to the L-M stimulus than the L+M stimulus implies the presence of color-opponent neurons.

Authors	Year	Imaging	Stimulus Details	Opponency
Studies using isoluminant stimuli
Kleinschmidt et al.	1996	2T	cone	++
Engel et al.	1997	1.5T	cone	++
McKeefry and Zeki	1997	2T	–	+
Hadjikhani et al.	1998	3/1.5T	CIE	+
Zeki and Marini	1998	2T	–	+
Engel and Furmanski	2001	3T	cone	++
Studies using chromatic stimuli that contain luminance contrast
Lueck et al.	1989	PET	–	?
Zeki et al.	1991	PET	–	?
Beauchamp et al.	1999	1.5T	CIE	?
Bartels and Zeki	2000	2T	CIE	?
Studies with insufficient detail
Gulyas and Roland	1994	PET	–	–
Sakai et al.	1995	1.5T	CIE	–
Howard et al.	1998	1.5T	–	–
Chao and Martin	1999	PET	–	–

Note. This table summarizes results from imaging of color vision. The fourth column indicates how details about the stimuli were reported: using cone contrast calculations (cone), CIE coordinates (CIE), or “–“, when no details were given. The last column indicates, whether the results from the study are consistent with relatively large numbers of color-opponent neurons in V1 (++), show likely color-opponent signals with a possible contribution by S cones (+), are inconclusive (?), or whether the report did not allow any inferences to be made about opponency in V1 (–).

The stimuli here are more complicated than those considered previously, however, and so require additional discussion before conclusions about opponency can be drawn. First, the cone contrasts of the stimuli are unknown, and the logic of comparison described above only holds if the contrast in the L+M stimulus is greater than in the L-M stimulus. This is almost certainly the case, though, because typical displays produce much higher contrast for luminance patterns than for isoluminant patterns. (Hadjikhani et al., 1998, used near maximum available contrast for both their achromatic and chromatic stimuli. McKeefrey & Zeki, 1997, and Zeki & Marini, 1998, used essentially random contrast values for both types of patterns. For all three of these studies, then, the L and M cone contrasts in the luminance patterns were likely much higher than the contrasts in isoluminant patterns.) Second, the amplitude of the S cone signals produced by the stimuli is unknown; because S cone signals are not included in the luminance calculation, their cone contrast can vary freely across the image. In addition, most displays can produce very large S cone contrasts. In these studies, then, larger responses to chromatic than achromatic stimuli indicate some unknown mixture of S cone and opponent, L-M signals.

Chromatic patterns in all three of these studies produced greater activity in V1 than the achromatic controls. McKeefry and Zeki (1997) measured cortical activity in 12 subjects viewing chromatic and achromatic geometric patterns (Mondrians). They found greater V1 activity for the chromatic patterns in 10 of their 12 subjects. Zeki and Marini (1998) include a replication of these results reported simply as reliable in a group analysis. Hadjikhani et al. (1998) showed subjects sinusoidal radial gratings (pinwheels) of 95% luminance contrast and maximum available isoluminant contrast. V1 was more active in the chromatic condition in 26/26 hemispheres (13 subjects). These results are consistent with the idea that V1 contains large numbers of color-opponent neurons. (Because S cone responses in V1 are relatively weak compared to L+M responses, Wandell et al., 1999, which are in turn weak compared to L-M responses, Engel et al., 1997, we believe that a reasonable proportion of the chromatic response can be attributed to L-M neurons.)

Results from other studies are more difficult to interpret, because of how the stimuli were constructed. The patterns used in the chromatic conditions of some of these studies were patches of essentially random colors (Lueck et al., 1989; Zeki et al., 1991; Bartels & Zeki, 2000). Because these colors were not matched for luminance, the chromatic condition contained some luminance contrast. In all these studies, the achromatic stimuli were spatially identical gray patterns where each patch contained the same amount of luminance as the chromatic patches. In another study, color patches were presented that all contained equal luminance, but on a background of lower luminance, again leaving L+M contrast in the chromatic condition (Beauchamp, Haxby, Jennings, & DeYoe, 1999). The achromatic condition contained stimuli that varied in luminance, but whose average was the same as the luminance of the chromatic stimuli.

The aim of these studies was to isolate responses to nonluminance stimuli using subtraction. The chromatic pattern can be thought of as containing both “luminance” and “color,” while the achromatic pattern represents a matched luminance condition. The difference in activity generated by the two patterns, then, cannot be attributed to responses to luminance alone. But because the stimuli are undefined in terms of their effects on the cones, such a subtraction is not informative about color-opponency. For example, the color portion of the chromatic stimulus might only contain contrast visible to the L cones. Although such signals certainly qualify as color, in the sense that they do not optimally stimulate the luminance mechanism, they are not color-opponent.

To confirm opponency in the color response, color responses must be larger than responses to luminance patterns of greater or equal contrast. Since the color response is isolated by subtracting the chromatic and luminance condition, these studies show evidence of opponency only when the difference between conditions is greater than the response to the luminance condition alone. Unfortunately, these studies were only concerned with identifying nonluminance signals, and so all fail to compare the difference between the two stimuli to the luminance stimulus; some fail to report the amplitude of the difference at all (Bartels & Zeki, 2000; Beauchamp et al., 1999). An additional concern in these studies and in the previously discussed studies using Mondrians, is that the luminance conditions were simply “gray” patterns chosen by eye rather than using photometric criteria. Thus, they may include some contrast in nonluminance color directions. This is likely to be fairly small, however, compared to the chromatic conditions.

Overall, there is only marginal support for strong color-opponent representations in V1 from this type of imaging study, though the results are not inconsistent with that conclusion. In Lueck et al. (1989) the activity increase in V1, as measured by PET, was 9% for the achromatic versus rest condition, and 15% for the chromatic versus rest condition. Thus, when color was added (in the way described above) to the luminance pattern, there was a 6% increase in activity; the luminance pattern itself produced a 9% increase in activity from a resting baseline. Although this result is suggestive, and the extra activity due to color may very well have been due to color-opponent neurons, the results have alternative explanations (e.g., neurons that respond to L cone contrast alone). Furthermore, a replication failed to show any difference between chromatic and luminance stimulation in V1 (Zeki et al., 1991). In the most recent version of the experiment the authors reported “weak activity” in V1 in their difference maps (Bartels & Zeki, 2000). Another recent study also suggests greater activity in V1 for chromatic than for luminance stimuli (Beauchamp et al., 1999); but because the amplitude of the difference between conditions is unknown, little can be concluded about color-opponency in V1.

A final class of imaging study simply does not include enough details about the stimuli or pattern of responses to allow any useful conclusions about color-opponency in V1 (Chao & Martin, 1999; Gulyas and Roland, 1994; Howard et al., 1998; Sakai et al., 1995). Table 1 lists almost all of the neuroimaging studies that addressed the representation of color in human (visual) cortex. The final column in the table indicates whether the results from the study imply strong color-opponent signals in V1.

Electrophysiology in primary visual cortex has provided seemingly mixed evidence regarding the number of color-opponent neurons. One early set of measurements found a relatively small number (less than 15% of all neurons recorded) of color-opponent neurons that were not tuned for orientation (Livingstone & Hubel, 1984). This percentage rose to over 20% within the central 2.5º of vision. A roughly comparable number of color responsive cells tuned for orientation were also found, and in total approximately 47% of neurons in the central 2.5 degrees of vision were selective for color, and likely color-opponent. A contemporary investigation, however, reported somewhat larger numbers of color responsive neurons in V1 (Thorell et al., 1984). In this sample fully 79% of the neurons gave reasonably strong responses to isoluminant color stimuli. If a slightly more conservative criterion for color-opponent cells is set—those responding more to isoluminance than to luminance—the number falls to about 60%. (Because the cone contrasts of these stimuli are unknown, the true number of color-opponent cells is uncertain.)

A later study characterized cells more completely, and the linear combination of cone signals that predicted the neurons' responses was inferred (Lennie et al., 1990). In this sample, over 70 percent of V1 neurons combined signals from L and M cones with opposing sign. Roughly one third of all neurons were not tuned for orientation, and these cells generally assigned equal weight to L and M cone signals. The remainder of the color-opponent neurons had oriented receptive fields and received mainly unbalanced inputs from the L and M cones, leading them to respond well to luminance.

The authors of a more recent study took a similar approach, paying closer attention to neurons’ spatial receptive field structure. Johnson et al. (2001) classified neurons based on their responses to an isoluminant, L-M pattern and a luminance pattern. They found that 11% of their sample responded primarily to the L-M pattern (“color” cells), 29% of neurons responded approximately equally well to either pattern (“color-luminance” cells), and 60% of neurons responded preferentially to the luminance pattern (“luminance” cells). The color cells were mainly unoriented, while the other two classes contained mainly orientation-selective neurons. Phase analysis experiments using cone-isolating gratings and color exchange (silent substitution) experiments revealed that over 75% of the color-luminance cells were color-opponent.

Two recent reports (Conway, Hubel & Livingstone, 2002, and Landisman & Ts’o, 2002b) find mainly balanced, unoriented color-opponent neurons in V1. These papers did not have an entire V1 population survey as their goals, however, and so did not quantify weights attached to cone inputs and orientation bandwidths; it remains possible that unbalanced oriented neurons were included in some other category of cells.

Thus, the preponderance of single-unit evidence indicates that color-opponent neurons are relatively common in primary visual cortex, with opponency in roughly 40% of cells (Johnson et al., 2001; Lennie et al., 1990; Livingstone & Hubel, 1984, Thorell et al., 1984). Interpretations of the results have varied widely, however. Some discussions and measurements have focused on strictly balanced L-M neurons with unoriented receptive fields (Conway, 2001; Conway et al., 2002; Landisman & T’so, 2002a; Landisman & T’so, 2002b; Livingstone & Hubel, 1984; Ts’o & Gilbert, 1988) that comprise 10-20% of foveal V1 neurons. Such a focus naturally leads to the conclusion that a small, specialized population of neurons represents color. Nevertheless, most studies also find an additional, large population of unbalanced color-opponent cells that are likely tuned for orientation. Many of these neurons prefer intermediate color directions, leading them to respond to color patterns, luminance patterns, and mixtures. Future research may provide better estimates of the precise numbers of each class of cell. Below we use a quantitative model to show that the proportions reported by Johnson et al. (2001) are reasonably consistent with results from neuroimaging.

While both electrophysiological and neuroimaging results indicate that V1 contains substantial numbers of color-opponent neurons, imaging experiments appear to find larger amounts of opponency than do the single-unit studies. It is not obvious, for example, how larger responses to L-M stimuli than to L+M stimuli can obtain in neuroimaging experiments, when single unit data indicate that only 40% of neurons respond well to L-M and over 60% of neurons respond well to L+M (with some neurons responding well to both). Data from the two methodologies are not directly comparable, however, because they were gathered under vastly different stimulus conditions.

To examine whether there is agreement about the relative number of color-opponent neurons in V1, we tested whether a simple model, based upon recent single-unit measurements, could account for the most complete fMRI results. The model estimated the pooled response of the V1 neurons in the population measured by Johnson et al. (2001) to the set of stimuli used in Engel et al. (1997). Details about the neurons’ response properties were taken with kind permission from Johnson et al. The computation was performed in two steps. We first estimated a spatial response that measured how sensitive each model cell was to the checkerboard pattern. We next calculated a color response that measured how sensitive each neuron was to a particular color. The model’s total response for each colored pattern was the sum across all neurons of the product of the neurons’ spatial and color responses. The model’s total responsiveness as a function of stimulus color can be represented as a contour that can be compared directly to plots from Engel et al. Our calculations suggest a general agreement between the predicted fMRI response based on single-unit data and observed fMRI data. Many assumptions were made in trying to calculate the predicted color tuning functions; these are discussed in the section Model Assumptions below.

Figure 1. (a) Radial checkerboard stimulus. The stimulus as used by Engel et al. (1997) subtended 20º of visual angle. The colors of the dark and light segments in the stimulus were changed to different colors and contrasts during the experiment. (b) Colors were picked in L/M cone contrast space. The total cone contrast of the stimulus is given by .

An example of the stimulus is shown in Figure 1. The radial checkerboard pattern subtended 20º of visual angle and reversed its contrast at various temporal frequencies. During different blocks in the fMRI experiment, the segments were changed to other colors and contrasts that excited the cones in known proportions. Because the electrophysiological recordings that provided the basis of our model were made at 2-5º eccentricity (Johnson et al., 2001), we restricted our analysis to those eccentricities.

For each neuron, we obtained a measure of the relative input strength from the three different cone types using L-, M- and S-cone isolating stimuli from Johnson et al. (2001), who measured them using gratings that stimulated each cone class individually. For our model, we assume that the outputs of V1 neurons are roughly a linear transformation of the signals transmitted by the cones (Lennie et al., 1990). From the relative weights, we estimated a neuron's firing rate for a given stimulus.

We obtained only group statistics on spatial tuning, specifically the median tuning parameters for each of three groups of cells; each model neurons’ tuning was set to that of the group to which it belonged. Neurons were assigned to one of three groups depending upon the ratio of their responses to L+M and L-M stimuli (Johnson et al., 2001). For every cell, Johnson et al. calculated a sensitivity index, the ratio of peak responses to L+M and L-M. Cells with an index of 1.0 responded equally well to the two stimuli. A higher index indicates a stronger response to color than to luminance gratings. Cells were classified as “luminance” if the index < 0.5, “color-luminance” if 0.5 ≤ index ≤ 2.0, or “color” if their sensitivity index > 2.0. We will adhere to this nomenclature, even though the role of these different cells in perception is unknown. Color cells are on average low-pass, color-luminance and luminance cells band-pass. For each group, we obtained median parameters of a Difference of Gaussians (DOG) function that best fit each cell's measured response to gratings of various spatial frequencies (again see Johnson et al., 2001). The three corresponding spatial frequency response curves are plotted in Figure 2.

We assumed that color-preferring cells are generally unselective for orientation (Conway, 2001; Lennie et al., 1990; Livingstone & Hubel, 1984). Most luminance-preferring cells in V1 are orientation-selective, with an average bandwidth of approximately 50º (Ringach, Bredfeldt, Shapley, & Hawken, 2002). Cells responsive to both color and luminance have been reported to be orientation-selective for luminance (Johnson et al., 2001; Thorell et al., 1984) and chromatic stimuli (Lennie et al., 1990; Thorell et al., 1984). Spatial properties of luminance and color-luminance cells have been reported to be quite similar. For example, spatial frequency tuning preferences and bandwidths for color-luminance cells are generally similar to the spatial frequency preference and bandwidths found for luminance-preferring cells (Johnson et al., 2001). Accordingly, we assigned both luminance and color-luminance cells an orientation bandwidth of 50º.

Figure 2. Population spatial frequency tuning curves—average firing rate versus spatial frequency. Color (dotted line, mean preferred sf ~0.51 cyc/º, low-pass), color luminance (dashed line, mean preferred sf 2.56 ± 1.26 cyc/º, bandwidth [fwhm] 2.05 ± 0.70 octaves), and luminance (solid line, mean preferred sf 2.09 ± 1.00 cyc/º, bandwidth 1.96 ± 0.69 octaves).

Cells that are not selective for orientation will respond regardless of the stimulus orientation in their receptive field, but only a portion of the orientation selective cells are activated by any particular oriented stimulus. We used the orientation bandwidth parameters to estimate that only 29.7% of luminance and color-luminance cells would be activated compared to 100% of color cells for any particular oriented stimulus (assuming a Gaussian profile for orientation tuning curves). We also calculated the effect of increasing this orientation bandwidth parameter by a factor of two.

The spatial response captures the effect of the stimulus pattern on each cell, separate from the effect of the pattern’s color. Because our sample of cells was relatively small, and their precise receptive field locations were unknown, we modeled the spatial response as the average responsiveness of the neuron across the stimulus. To estimate this quantity, we first calculated an isotropic, that is, nonorientation selective receptive field (kernel) by taking the Inverse Fourier Transform of the neuron’s spatial frequency tuning. The convolution of the stimulus with this kernel produces a neural image, an image of the response of the isotropic receptive field to the radial checkerboard stimulus. We masked the neural image with an annulus to exclude eccentricities smaller than 2º and larger than 5º. To account for the fact that luminance and color-luminance neurons have oriented, nonisotropic receptive fields, we scaled the images by a factor based upon the cells’ orientation bandwidths (see above). The neural images then represent the spatial pattern of activity generated by an average cell from each class—in the case of orientation selective cells, the average over many different preferred orientations. Figure 3 shows these neural images for color-preferring, color-luminance, and luminance-preferring cells. The average magnitude of each of the neural images was our measure of the cells' responsiveness to the pattern.

Figure 3. Images of the spatial response to checkerboard stimulus of one particular contrast: (a) color cells, (b) color-luminance cells and (c) luminance cells (cf. Stimulus in Panel (a) of Figure 1). These images were obtained by convolving the stimulus image with an iso-tropic (nonorientation selective) kernel and scaling with a factor that captures the average orientation tuning bandwidth of each sub-population (see text). The pixel color corresponds to average firing rates. Eccentricities smaller than 2º and larger than 5º have been masked. Note that responses have been half-wave rectified, and so represent the response rates of model simple cells.

The color composition of a stimulus was represented as a three-element vector in cone contrast space with components for L, M, and S excitation. This vector represents the signals that reach the cortex from the L, M, and S cones respectively. For every cell with given cone weights, the response to a stimulus was calculated as the dot product of the stimulus and cell cone weights. In order to obtain responses independent of the sign of contrast, e.g., +[L - M] and -[L - M], and response contours in all four quadrants of the L/M plane, we duplicated the cone weights and reflected them about the origin of the L/M plane. This doubles, in a sense, the number of neurons in the model, but forces the responses to positive and negative contrasts to be symmetric.

For each neuron in our sample, we calculated responses to many differently colored checkerboard stimuli as the product of the spatial and chromatic responses. This calculation assumes color-pattern separable receptive fields (see Model Assumptions below). As a final step, we accounted for two well-known nonlinearities in cortical neurons. The responses of simple cells were half-wave rectified, that is, negative responses were set to zero. This step captures the relatively low resting spiking rate of the neurons in the sample. The responses of complex cells were full-wave rectified, which corresponds to an absolute value calculation. This reflects the sign invariance of complex cells. The initial version of our model did not contain any final nonlinearities in response, which gave it a linear contrast-response function (Figure 4). This assumption was relaxed in later versions and is discussed below.

The neural model of V1 showed color responses that resembled the results of neuroimaging experiments. The overall model response was stronger for L-M checkerboard patterns than for L+M patterns. Most neuroimaging experiments in V1 have also found stronger responses to L-M stimuli than to luminance patterns. The entire model color tuning curve was also similar to analogous results from neuroimaging experiments. Our model result is shown superimposed on the 4Hz fMRI iso-response contours from Engel et al. (1997) in Figure 5. The iso-response contour obtained from the model, shown in red, fall close to the fMRI iso-response contours, shown as black solid lines. The general agreement between the model results and fMRI data suggest that there is no conflict between the results of neuroimaging and single-unit studies of the color tuning of V1.

Figure 5. Scaled model output is shown as an iso-response contour in the L/M cone contrast plane (red traces). fMRI iso-response contours from Engel et al. (1997) for two different observers (A, B) are shown as solid black lines. The data plotted was obtained with stimuli at a temporal frequency of 4Hz. Dashed contours are 10% and 90% confidence intervals obtained by resampling with replacement. We do not plot the 10% and 90% confidence intervals (also estimated by resampling) for the model calculations, as they are only about twice the line thickness.

First, the checkerboard pattern used by Engel et al. (1997) contained predominantly low spatial frequencies (as do the Mondrian patterns used in other imaging studies). Since the color cells in the model maintain their responses at low spatial frequencies, where the responses of other cell types are greatly attenuated, color cells’ responses were relatively strong. The model response of an optimally oriented luminance neuron to the stimulus pattern in its preferred color direction was only 64% of the model response of an average color cell to its preferred color direction. Neuroimaging studies have enhanced color responses by using low spatial frequency stimuli.

This conclusion leads to the natural prediction that stimuli that contained mainly relatively low or high spatial frequencies would lead to dramatically different results in an imaging experiment that measured iso-response curves. As an example, we have simulated results for color tuning functions obtained with a simple contrast modulated sinusoidal grating in a 2º-5º eccentricity annulus, with spatial frequencies of 0.25 cyc/º and 2 cyc/º (Figure 6). The high spatial frequency stimulus results in iso-response curves elongated along the L-M axis, whereas the low spatial frequency stimulus (0.25 cyc/º) biases the ellipse to lie along the orthogonal, L+M, axis.

Figure 6. Predicted fMRI iso-response contours in V1 based on model calculations in this paper. (a) shows expected results for color tuning functions obtained with low spatial frequency (0.25 cyc/º) gratings and (b) with higher spatial frequency (2.0 cyc/º) gratings. At low stimulus spatial frequencies, the overall response is dominated by color-opponent neurons (see text), whereas luminance-preferring neurons respond predominantly at higher spatial frequencies.

Second, differences in the orientation tuning of color, color-luminance, and luminance cells also strongly affect the model responses. We assume color cells are not tuned for orientation, and that color-luminance cells have orientation tuning similar to luminance-preferring cells. This means that for a particular oriented stimulus only a proportion of orientation tuned cells will become active, while all of the nonoriented cells will respond. We calculated the proportion of active orientation tuned cells to be 29.6%, compared to 100% of (non-oriented) color cells, based on the average tuning bandwidth of 50º for the color-luminance and luminance cells. One reason why luminance cells may be numerous in cortex is that it takes many of them to completely tile the space of image parameters for which they are specialized.

To assess the importance of orientation bandwidth on cortical response, we reran the model using an orientation tuning bandwidth for luminance and color-luminance cells of 100º (compare (a) and (b) in Figure 7). Doubling the bandwidth predictably increased the response to luminance, and dramatically changed the overall shape of the iso-response curve.

The relatively large number of cells responsive to color was also critical for obtaining agreement with fMRI results. An additional simulation examined the iso-response curve that resulted if the color-luminance cells were eliminated from the model. The results, shown in Figure 7(c), reveal a large change in predicted cortical color tuning. Hence, neural models that contain relatively small numbers of cells responsive to L-M contrast cannot account for the data observed in fMRI experiments.

While the model results are a reasonable match to the data from Engel et al. (1997) that were collected with a stimulus frequency of 4 Hz, they show less opponency than the data gathered at 1 Hz. One explanation for this discrepancy is that our calculations do not take into account differences in temporal frequency tuning between cell classes. Color vision is generally optimized for low temporal frequencies, and it seems likely that color and color-luminance cells might exhibit low-pass temporal frequency tuning. Including a temporal response factor, then, could boost the relative contribution of color and color-luminance cells for simulated stimuli at 1 Hz and perhaps even at 4 Hz.

There are several additional reasons why the model may actually underestimate the magnitude of color-opponent signals in response to stimuli used in imaging experiments. First, our model considered a smaller portion of the visual field than do imaging experiments. We limited our calculations to stimulus eccentricities between 2º and 5º, the range over which we had access to single-unit data. Imaging results, however, may emphasize even more central portions of the visual field. Due to cortical magnification, the central portions of the visual field occupy far more pixels, and thus contribute far more strongly to the results than do more peripheral regions. Since color-opponent cells are more numerous near the foveal representation than in the periphery (Livingstone & Hubel, 1984), including a larger range of eccentricities in the model may yield even stronger color-opponent responses.

Additionally, the model may overestimate the high spatial frequency content of the stimuli used in the imaging experiments. The display systems in most fMRI experiments use back-projection, which invariably blurs the stimulus somewhat. This will attenuate the contrast at high spatial frequencies, which will in turn reduce the responses of cells—predominately luminance and color-luminance neurons—that are selective for high spatial frequencies. Incorporating stimulus blur into our simulation could further boost the amount of opponency in the model results.

Figure 7. A change in orientation tuning bandwidth or proportion of color-luminance cells affects the shape of the calculated iso-response contours. (a) Iso-response curve obtained with the orientation tuning bandwidth of “luminance” and “color-luminance” cells set to 50º (as in Figure 5). The ellipse is elongated along the L+M axis. (b) An increased orientation tuning bandwidth (×2) increases the proportion of active oriented cells, which mainly prefer luminance. The ellipse is roughly circular. (c) If all “color-luminance” cells are assigned to the group of “luminance” cells, the ellipse is elongated along the L-M axis.

Perhaps most importantly, we have assumed that the group of cells for which we obtained parameters is an unbiased sample of the actual distribution found in cortex. It remains possible that certain cell types are under-represented in single-unit recording. In this context, it is important to consider that luminance stimuli are often used to initially characterize the spatial properties of neurons. Stimuli at optimal spatial parameters are then used to characterize the neurons’ color tuning. It seems likely that such a procedure could underestimate the number of cells best responding to isoluminant stimuli, particularly if the neurons are not color-pattern separable (see below). Further bias could also be introduced if there are systematic differences in the morphology or size of cells with different color tuning, and if electrodes therefore selectively sample certain groups of cells.

In modeling the responses of V1, we made many simplifying assumptions. Two of these have to do with the nature of V1 receptive fields. Most critically, we have assumed that the neurons in our sample have color-pattern separable receptive fields; that is, the color tunings of individual cells remain constant as the spatial pattern used to stimulate them changes. This assumption is violated by neurons in the lateral geniculate (Derrington et al., 1984) and almost certainly by some neurons in cortex. However, many V1 neurons show similar spatial frequency tuning for stimuli in different color directions (Johnson et al., 2001). Furthermore, the changes in color tuning are likely to be relatively small given that the stimulus used in the simulation is very heavily weighted for low spatial frequencies.

We also assume that the cells are linear with respect to the cone signals. There is some evidence that this is widely true for V1 neurons to a first approximation, with the notable exception of the rectification that we explicitly included in our model (Lennie et al., 1990). Neurons in V1 do generally show nonlinear contrast response functions, however, and so we have begun exploring the consequences of relaxing this assumption. The contrast dependence of many neurons in V1 is well-described by R∝ r₊ⁿ / (c_1/2ⁿ + r₊ⁿ), where r₊ is the rectified linear estimate of the response, the exponent n is close to 2, and c_1/2 is the contrast that produces a half-maximal response (Albrecht & Hamilton, 1982). This sigmoid nonlinearity tends to reduce the firing of neurons at low and high levels (compression) and to boost the response otherwise. In the case of our model, the responses by the luminance and color-luminance neurons to the low-spatial frequency stimulus tend to be further attenuated, unless c_1/2 is very small. The relatively enhanced response of neurons to L-M stimuli leads to an iso-response ellipse that is more elongated along the L+M axis.

Contrast normalization is a second type of nonlinearity that is not currently implemented in the model. We can reason, however, about one of its likely effects. Without such a nonlinearity (but with the basic contrast-response function described above), very high contrast stimuli would cause all model neurons to fire at asymptotic levels, regardless of their tuning. In such a case, model responses would simply reflect the overall number of neurons of each type, and because luminance cells are most numerous, we would expect larger responses to L+M than to L-M. The color-tuning function, then, would change its shape dramatically as stimulus contrast reached high levels. One effect of contrast normalization is to cause neurons to maintain their tuning even at very high stimulus contrasts (Geisler & Albrecht, 1992; Heeger, 1992). Because of this, including contrast normalization in our model would prevent iso-response contours from dramatically changing shape at high stimulus contrasts. Additional effects of contrast normalization remain to be explored.

A further assumption of the model is that different cell types contribute equally to the signals measured in neuroimaging experiments. Neuroimaging techniques generally use some measure of the blood supply as a proxy for neural activity. However, it is known, for example, that the cytochrome-oxidase blobs contain specialized vasculature (Zheng, LaMantia, & Purves, 1991), which could possibly lead neurons therein to be over-represented in neuroimaging signals. While much controversy surrounds the claim that the blobs contain relatively high densities of color cells, it is nevertheless far from clear that all cell types are weighted equally in the fMRI response.

Finally, our model assumes that the fMRI signal is proportional to the average neural firing rate within a local patch of cortex. Although large deviations from this assumption have not been found, it could be that other neural signals, such as local field potentials, are more closely related to the fMRI response (Logothetis, Pauls, Augath, Trinath, & Oeltermann, 2001). Local field potentials are thought to reflect the input and intra-cortical processing, rather than the output (spiking rate) of neurons. If this is the case, some of the fMRI measurements of activity in V1 may reflect the strongly cone-opponent input from the LGN.

Our literature review and simulation results allow several conclusions to be drawn. First, neuroimaging experiments, which often use stimuli that are optimized for color perception, generally find strong color-opponent signals in V1. Second, studies using single-unit recording find a modest percentage of neurons in V1 (~20% near the fovea) that are balanced L-M color-opponent neurons. However, a roughly equally sized population show unbalanced color-opponency. Third, our modeling results show that data from these two methodologies are consistent. For the stimuli used in imaging experiments, large signals are produced by relatively large overall populations of color-opponent neurons.

The model results underscore the importance of several factors in shaping the response of V1 to colored spatial patterns. The spatial frequency content of the stimuli is crucial. Another important factor is the difference in orientation tuning of cells responsive to L-M and L+M stimuli. Eccentricity, temporal frequency, and other factors certainly influence color responses as well, though they were not systematically explored here.

The perceptual role of color-opponent neurons in V1 has only begun to be investigated. Two studies have found reasonably good agreement between the large, pooled fMRI color response and behavior in pattern detection (Engel et al., 1997) and judgments of apparent contrast (Engel & Furmanski, 2001). These results suggest that the entire population of color-opponent neurons in V1 participate in these two perceptual tasks.

It remains possible, however, that other tasks will preferentially draw upon certain specific populations of color-opponent neurons in V1. For example, tasks in which edges play a role may be supported mainly by the unbalanced L-M neurons, which generally have oriented receptive fields. Similarly, some psychophysical results (e.g., Hurvich & Jameson, 1955; Krauskopf, Williams, & Heeley, 1982) suggest a particularly important role for balanced L-M neurons in color categorization and detection tasks. However, other results have found evidence supporting the use of unbalanced neurons in detection (Krauskopf, Williams, Mandler, & Brown, 1986), color matching (Webster & Mollon, 1994) and other color-related tasks (Krauskopf & Gegenfurtner, 1992). Thus, although some behavioral results make it tempting to label balanced neurons the “color vision system,” it seems more likely that all of the many color-opponent neurons in V1 play some role in the collection of functions that comprise color vision.

We are grateful to Elizabeth Johnson, Michael Hawken, and Robert Shapley for allowing us to use their data, as well as helpful comments on the manuscript. We would also like to thank Peter Lennie for discussion of details of his data set, and the three anonymous referees for helpful comments on an earlier version of the manuscript. This research was supported by an NIH grant (EY11862) to SAE.