Natural visual scenes contain an abundance of cues that can be used to perform visual tasks such as object detection, object discrimination or localization of one object relative to another. For example, cues such as luminance, colour, disparity, texture and motion information may all be used by the visual system to effectuate visual tasks. There is now compelling evidence for the existence of separate visual processing streams and functionally specialised cortical areas (van Essen & Maunsell, 1983; Zihl, von Cramon & Mai, 1983; Maunsell & Newsome, 1987). Each visual stream is thought to be involved in the analysis of a particular sensory cue (e.g. luminance, texture, motion or colour) which in turn contributes to a particular aspect of our everyday perceptual experience. Cumulative evidence for this modular framework to visual analysis is derived from studies which examine lesion-induced deficits, response properties of neuronal populations, and the architecture of anatomical connections within the cortex (Pearlman, Birch & Meadows, 1979; Damasio, A., et al. 1980; Zihl et al., 1983; Zeki & Shipp, 1988; Desimone & Ungerleider, 1989; De Yoe et al., 1990). Striking examples of this cortical “division of labour” result when localized areas of the human cerebral cortex suffer bilateral damage. For instance, individuals can suffer a complete loss in sensitivity for a particular visual attribute such as colour (Damasio et al., 1980) or motion (Zihl et al., 1983), yet show little or no deficit in processing other types of visual information.

In addition to the major sub-divisions outlined above, other examples of functional subdivisions within and between early cortical areas (V1 & V2) also exist. Recently, both physiological (Zhou & Baker, 1993; Mareschal & Baker, 1998), and psychophysical (Badcock & Derrington, 1985; Chubb & Sperling, 1988; Derrington, Badcock & Henning, 1993; Ledgeway & Smith, 1994; Li-Ming & Wilson, 1996; Whitaker, McGraw & Levi, 1997; McGraw, Levi & Whitaker, 1999) investigations of early visual coding have focused on how the visual system analyses both luminance-defined and contrast-defined image components using linear and non-linear processes respectively. Neurophysiological investigations have shown that the visual system contains many neurons which signal differences in average luminance between the excitatory and inhibitory sub-regions of their receptive field in a linear manner. However, such linear neurons are not ideally suited to the analysis of texture- or contrast-defined stimuli where the spatial extent of luminance variations can be small relative to the receptive field size. In this situation, linear summation of luminance increments and decrements across the extent of the receptive field may produce no net variation in luminance relative to the surround. For this reason large linear cortical neurons are unable to signal the presence of such texture-defined visual stimuli. In order for neurons to be able to detect image features such as contrast variations, the outputs of relatively small scale initial filters must be subjected to a form of non-linearity (such as rectification) before they become amenable to conventional linear processing. Contemporary models of non-linear visual processing are therefore based upon an initial linear filtering stage, followed by a non-linear step (rectification stage), and subsequent linear filtering at a relatively coarse spatial scale. This processing cascade has previously been identified in the striate cortex of cats (Zhou & Baker, 1993) and psychophysically in humans (Wilson, Ferrera & Yo, 1992; Graham, Beck & Sutter, 1992).

The present study examines how the visual system localizes stimuli that are composed of both luminance and textural information. A combination of luminance and texture was chosen for two reasons. Firstly, this combination is one of the most commonly encountered in our visual environment. Secondly, there now exists convincing physiological (Olavarria et al., 1992; Zhou & Baker, 1993: Mareschal & Baker, 1998; see also Shapley, 1994) and psychophysical evidence (Badcock & Derrington, 1985; Chubb & Sperling, 1988; Derrington, Badcock & Henning, 1993; Ledgeway & Smith, 1994; Li-Ming & Wilson, 1996; Whitaker, McGraw & Levi, 1997; McGraw, Levi & Whitaker, 1999; Badcock & Khuu, 2001) to suggest that each attribute is processed by a dedicated cortical stream. Objects defined by variations in luminance are detected by linear neurons located in the primary visual cortex (V1) (Hubel & Wiesel, 1962; Movshon, Thompson & Tolhurst, 1978). Texture, or contrast-defined objects, constructed from balanced increments and decrements in luminance are invisible to linear V1 neurons, and are recovered via a non-linear operation carried out by a dedicated neural population located in V1 and V2 (Zhou & Baker, 1993; Mareschal & Baker, 1998; Mareschal & Baker, 1999). These neurons, which differ in their response output to image features, provide the physiological framework for the luminance and texture processing streams in the human visual system. We ask how the visual system localizes objects within a framework of seemingly autonomous visual processing streams. Specifically, we investigate how the visual system assigns a unitary location to objects defined by multiple stimulus attributes, where such attributes provide conflicting positional cues. Issues related to this question have been examined previously. For example, Rivest and Cavanagh (1996) determined how the precision of visual localization changes when multiple attributes (such as luminance, colour and texture) are combined at a single location. However, in their experimental arrangement stimulus attributes always provided harmonious positional information. Similar findings were reported by Gray and Regan (1997). Landy and co-workers (Landy, 1993; Landy & Kojima, 2001) investigated vernier alignment of texture-defined edges, where the edge location could be signalled by differences in orientation, contrast or spatial frequency. The results showed that perceived edge location is determined from a weighted average of individual component estimates. In some of their conditions the edge location signalled by one cue was displaced relative to that of another in order to obtain estimates of the individual cue weights. In the current experiment, rather than varying the contrast of one cue whilst the other remains fixed, we reduce the contrast of one cue in proportion to an increase in the other in an attempt to keep the global contrast of the combined stimulus constant.

Three of the authors acted as observers, and wore their optimal refractive correction where necessary.

The stimulus elements were composed of an additive combination of luminance contrast and texture contrast components. The luminance component is

The stimulus was an additive mixture of these two patterns, having a contrast C_lum for the luminance pattern and C_text = 1 - C_lum for the texture pattern. Thus,

Generation and control of stimuli was performed using the macro capabilities of the public domain software NIH Image 1.61 (developed at the U.S. National Institutes of Health and available from http://rsb.info.nih.gov/nih-image/ or on floppy disk from the National Technical Information Service, Springfield, Virginia, part number PB95-500195GEI). Stimuli were presented on a Mitsubishi 21 inch d2 Colour Display Monitor with a mean luminance, L, of 38.3 cd m^-2 and a frame rate of 75 Hz. The non-linear luminance response of the display was linearized by using the inverse function of the luminance response as measured with a Minolta CS-100 photometer. The host computer was a Motorola Starmax 4000/200 PowerPC. Sufficient contrast resolution for the measurement of contrast detection thresholds was achieved by the use of a video summation device (Pelli & Zhang, 1991).

Observers were asked to perform a three patch Vernier alignment task in which the horizontal position of the central element had to be judged with reference to two vertically separated reference patches (Figure 1). The vertical separation between each of the elements was 3.44 deg. Two conditions were investigated. In one the Gaussian envelopes of all three elements were symmetric, whilst in the second, the central element of the stimulus was composed of asymmetric Gaussian profiles; i.e. they contained a different Gaussian standard deviation (σ) either side of the midline. However, the overall size of each component remained constant – a reduction in the standard deviation on one side was balanced by a proportionate increase on the other. Importantly, the polarity of asymmetry was opposite for luminance and texture information, resulting in the centroid for each type of information being offset in opposite directions (see Figure 1B). The technique of offsetting stimulus centroids has previously been applied to motion displacement thresholds (Morgan, Ward & Cleary, 1994), and measures of stereoscopic disparity (Harris & Morgan, 1993). The perceived offset of the central blob, for both symmetric and asymmetric stimuli, was established using a method of constant stimuli with two alternatives, right or left. Within any experimental run, perceived offset was established for two stimuli of equal but opposite asymmetry (i.e. one stimulus and its mirror image), and either of these could occur with equal probability on any one trial. Each of the stimuli could be presented at any one of seven offsets, equally spaced around an alignment position determined by an initial method of adjustment. A step size of 3.22 arcmin between each of the seven offsets produced an appropriate ambit of responses ranging from approximately 100% rightwards to 100% leftwards. Stimuli were presented within a rectangular temporal window of 500msec duration. The results of the first 20 trials were discarded to allow subjects to familiarise themselves with the task. Following these, 80 trials were presented at each of the seven offsets and the proportion of “rightward” responses was calculated for each offset. The resulting data were fitted with a logistic function of the form

The relative amplitude of modulation of the luminance and texture components was obtained by varying C_lum between 0 and 1, and the perceived location of the central element was established as a function of this parameter. In addition, luminance and noise detection thresholds were measured to examine the role of stimulus visibility. Detection thresholds were established using a two alternative forced choice method of constant stimuli. Nine levels of contrast were used each separated by 0.05 log units. The task of the subject was to decide which of the two 500 msec intervals contained the stimulus. Twenty trials were randomly presented at each of the nine contrast levels. The data were then fitted with a logistic function in order to reveal the contrast level resulting in a 75% correct response level. All procedures followed the tenets of the Declaration of Helsinki.

When both luminance and texture components are symmetrical and superimposed (Figure 2A-C, open circles), no perceived offset of the patch as a whole is observed, and, the stimulus is veridically perceived as at or near it’s centroid (Figure 2A-C, solid lines). The perceived location of asymmetric patches composed of either luminance or texture information alone were also consistent with the calculated centroid positions of their stimulus envelopes (Figure 2A-C, dashed lines). The results of the present study support the view that observers locate luminance- and texture-only patches at or very close to the centroid of their respective distributions.

When the luminance and texture components of the central patch are made asymmetric, but skewed in opposite directions, a modulation in amplitude of one type of information relative to that of the other produces a smooth change in the perceived location of the object as a whole (Figure 2A-C, filled symbols). This effect could result from two very different modes of positional analysis. The representation of the object as a whole may be subjected to a nonlinear transformation (such as rectification) early in the visual pathway, following which a single positional cue is extracted. Alternatively, the visual system may extract a positional signal from both sources of visual information which are subsequently combined in order to extract a global representation of spatial position - the smooth change in position is a consequence of variation in the relative salience of each attribute as the 1^st- and 2^nd-order contrasts are traded off prior to the combination stage.

If the visual system is asked to locate either luminance or texture in isolation (luminance contrast of either 1 or 0 in Figure 2D-F), thresholds are similar in both the symmetric and asymmetric conditions. In the symmetric condition a reduction in the contrast of one component is offset by the increase in contrast of the other, resulting in an approximately linear function. However, when observers have to make localization judgements on patches composed of conflicting luminance and textural information (the asymmetric condition), localization accuracy is compromised as shown by the peak in thresholds near a luminance and texture contrast value of 0.5. The data in Figure 2D-F have been fitted by a function that allows quantification of this threshold elevation, as described in the legend to this figure.

For the inter-element separation used in this study, reducing the contrast of each cue in isolation from 100% to 50% has little effect on alignment thresholds. Nevertheless, whilst the accuracy of individual alignment measures shows a certain degree of contrast independence, the data presented in Figure 2A-C show that co-varying the relative supra-threshold contrasts of each component produces a marked, systematic change in perceived position, indicating that the relative level of supra-threshold contrast is of critical importance. In order to examine the role of visibility we measured threshold detection for asymmetric luminance and texture patches alone. Single-cue detection thresholds for both attributes were then used to express the point of perceived alignment as a multiple of its respective single-cue detection threshold (i.e. the contrast of a single cue in the combined stimulus is expressed as a function of its single-cue detection threshold at the point where no offset is perceived). The results are presented in Table 1. A much larger multiple of luminance detection threshold is required to balance textural information in situations where the two types of visual information provide conflicting positional cues.

The question of what aspect of an object actually defines its apparent position has been of considerable interest for some time. There are a number of cues or ‘location tags’ which an observer can use to locate the relative position of objects within a visual scene. These include the peak of the object’s luminance or contrast distribution, points of inflexion or zero crossings in the luminance distribution, the position at which edges of the object reach threshold, and the weighted mean or centroid of the distribution. Previous studies have suggested that the most likely candidate is that of the centroid or ‘centre of gravity’ of the stimulus envelope for both luminance-defined and contrast-defined objects (Westheimer & McKee, 1977; Watt & Morgan, 1983; Morgan & Aiba 1985a; Morgan & Glennerster, 1991; Morgan, Ward & Cleary, 1994; Whitaker et al, 1996). The results of the present study support this assertion — the perceived location of both asymmetric luminance- and texture-defined patches was found to agree very closely with the calculated centroid position for each distribution.

When the luminance and texture components of the central patch provide conflicting positional cues, a modulation in amplitude of one type of information relative to that of the other produces a smooth change in the perceived location of the object as a whole. This indicates that a global estimate of object location is extracted after the visual system combines positional signals from both sources of visual information. In an elegant series of experiments, Rivest and Cavanagh (1996) showed that a contour defined by one visual attribute (e.g. luminance, colour, texture or motion) could influence the perceived location of that defined by another attribute. Furthermore, they showed that combining different attributes at a common location improves the accuracy of localization. The results of Rivest and Cavanagh’s study strongly suggest that information from different visual attributes are combined at a common neural site prior to the level at which a localization decision is reached. Inspection of Figure 2D-F confirms that this is likely to be the case. The accuracy of relative localization for luminance or texture in isolation is very similar for both symmetric and asymmetric distributions. However, when observers are asked to locate patches composed of conflicting luminance and texture cues, localization thresholds are elevated, reaching a maximum in threshold elevation near a luminance and texture contrast value of 0.5. What might be the reason for this threshold elevation? Internal noise is likely to affect the relative salience of the two components from one trial to the next. For the symmetric condition this will have no effect since the positional cue provided by each component is in exact spatial registration. However, in the asymmetric condition, where each individual component provides a unique positional signal, trial-to-trial fluctuations in the relative strength of each signal constitute a significant additional source of variance. The localization noise for each individual component is no worse in the asymmetric condition compared to the symmetric condition, it is simply that the noise results in an increased response variance only when components signal conflicting positional estimates. A model of positional analysis which employed an early nonlinear transformation, followed by the extraction of a single positional estimate, would not contain this additional source of variance. Furthermore, the results of both Rivest and Cavanagh (1996) and Gray and Regan (1997), suggest that the rule for combining positional signals derived from different visual attributes is consistent with probability summation between independent channels.

Alternative potential explanations exist for the elevation in localization thresholds for stimuli consisting of conflicting luminance and texture information. One possibility involves changes in the overall stimulus profile produced by combining two individual asymmetric profiles. Morgan and Aiba (1985b) have demonstrated that the precision with which the mean of a distribution can be extracted is dependent upon both the width of the distribution and its area. Our methodology ensured that asymmetric patches consisting of either luminance or texture alone differed from their symmetric counterparts in neither width nor area, since increases in the standard deviation of the patches on one side were counterbalanced by decreases on the other. It is reassuring, therefore, that asymmetric patches of either luminance or texture can be located with the same precision as their symmetric counterparts (Figure 2D-F). For stimuli consisting of an asymmetric combination of luminance and texture, however, it is important to eliminate potential changes in overall width and area as contributors to the threshold elevation in this specific region (Figure 2D-F). We therefore performed a control experiment in which alignment thresholds were measured for a combination of two asymmetric luminance profiles (each of contrast = 0.5) skewed in opposite directions, and also two asymmetric texture profiles. This allows us to directly compare performance against that for the combination of asymmetric luminance and texture profiles (shown in Figure2D-F, C_lum=0.5). Results are shown in the table below.

For each observer, localization performance for combinations of the same type of information (i.e. luminance + luminance or texture + texture) are similar, and comparable with thresholds for the symmetric conditions (Figure2D-F). Thresholds for the combination of disparate sources of information (luminance + texture) are consistently higher, indicating that this threshold elevation reflects a true cost of disparate cue combination.

It might be argued that the threshold elevation is a result of a reduction in contrast of the individual luminance and texture components, i.e. at the extremes of Figure 2D-F (luminance contrast of C_lum= 0 and 1) either the luminance or texture component is at maximum contrast, whilst in the region of greatest threshold elevation both components are present at half their maximum contrast levels. However, if threshold elevation were a result of reduced individual component contrast, one would expect the same threshold elevation for the symmetric condition. This proves not to be the case, and indicates that the elevation in localization thresholds is likely to be a direct result of combining disparate sources of visual information.

Perceived alignment for patches composed of competing luminance and textural cues was obtained when the physical contrast of the luminance component was approximately equivalent to that of the texture component. This might seem to suggest that both components play an equivalent role in dictating the perceived position of the overall patch. However, this would only be the case if the visual system were equally adept at detecting the presence of luminance and textural information. In order to examine the role of visibility we measured threshold detection for asymmetric luminance and texture patches alone. Detection thresholds for both attributes were then used to express the point of perceived alignment (i.e. where no offset is perceived) as a multiple of its respective detection threshold. The results are presented in Table 1. A much larger multiple of luminance detection threshold is required to balance textural information when the two sources provide conflicting positional cues. It follows that, if both luminance and texture components were presented at an equal multiple of their detection thresholds, then the perceived location of the entire patch should appear offset in the direction of the textural component, which is indeed the case. An asymmetry in perceptual weights might be taken to indicate that the visual system does not treat all attributes equally but rather primacy is given to textural information over luminance information. This is in contrast to previous reports suggesting that luminance information was the dominant attribute in contour localization tasks (Livingstone & Hubel, 1988; Grossberg & Mingolla, 1985). Rivest and Cavanagh (1996) reported that when different attributes are combined at a single location, each providing concordant information, localization thresholds improve by an equivalent and statistically predictable amount as each attribute is added. This implies an equal role for each visual attribute. On the other hand, evidence for the unequal weighting of visual attributes has been suggested previously (Landy, Maloney, Johnston & Young, 1995; Mather & Smith, 2000; Landy & Kojima, 2001). For example, in the localization of texture-defined edges, Landy (1993) presents a model in which separate location estimates are made for each visual attribute, the attributes themselves are then weighted, and the overall location is derived from the average of the weighted attributes. Within this framework, there is scope to assign larger weights to estimates derived from the particular visual cues that are the most robust and thus provide the most reliable estimate of the edge location. For example, in regions of a visual scene that contain little or no textural information preference might be given to more abundant visual attributes. Therefore, the final weighting of visual attributes is likely to be a product of the reliability of a particular cue and its availability. The quality of information provided by a visual attribute can vary not only from location to location but also over time, and the visual system needs to be able to accommodate such dynamic changes. However, the question of how the visual system weights different attributes remains. Landy et al. (1995) suggest that the weighting factors are derived from subsidiary cues which in isolation do not aid edge localization but do comment directly on the reliability of information provided by a particular attribute.

The results of the present study show that the visual cortex is able to effortlessly integrate disparate sources of visual information to form a global estimate of object position, although this conflation of visual attributes results in a modest loss of localization accuracy. Analogous effects have been reported in the motion domain, where the integration of luminance and chromatic information results in either enhancement or disruption of the motion percept depending on whether each attribute conflicts or concurs (Cavanagh, Arguin, von Grünau, 1989; Morgan & Ingle, 1994; Edwards & Badcock, 1996). Mis-matches between luminance and texture information are commonplace in the real world, where textured objects often vary in luminance across their surface, as a result of shadows or changes in illuminant position. The results of the present study suggest that texture information may be a more potent indicator of object position, implying that the human visual system gives more weight to visual attributes that are reliably related to the contours of objects. It is likely that visual experience plays an important role in shaping the weighting map of visual attributes, and conceivable that this weighting might be modified in different visual environments. Consider for example the mottled illumination of the forest floor. Textural difference between foliage can be small, and local luminance can change dramatically due to shadows, introducing luminance ‘noise’ to the scene. In such an environment, chromatic differences or colour cues, which are not subject to the same variability, may be particularly important. Therefore, the weighting map of visual attributes might reflect the evolutionary pressures imposed by the visual environment.

PVM is supported by a Research Career Development Fellowship from the Wellcome Trust. DRB is supported by the Australian Research Council. The authors would like to thank Mike Landy for his comments on an earlier draft of this manuscript.

Commercial Relationships: None.