Visual short-term memory (VSTM) plays a crucial role in what we colloquially call “seeing.” A dramatic demonstration of this memory limit to the conscious appreciation of visual scenes was given by Molly Potter in a series of now classic studies (Potter, 1975; Potter, 1976; Potter & Levy, 1969). By using the Rapid Serial Visual Presentation Procedure (RSVP), she showed that short-term recall of pictures seen briefly within a continuous stream could be very poor, yet a cued search for a particular item within the stream remained efficient even when the pictures were seen for only 125 ms each. Evidently, enough visual processing takes place within such a short temporal window to allow the efficient recognition of a template object defined by abstract instructions. However, the fast alternation rate of the RSVP stream exceeds the limit for encoding in a more permanent memory store, making impossible the recall of the contents of the scene during a subsequent memory recognition stage. As such, the experiments of Potter and later those of many others (Coltheart, 1999) have shifted the focus of attention to short-term memory, as opposed to early visual mechanisms, as the real bottleneck of conscious vision. Even more crucial must be the role of short-term memory in the visual interpretation of scenes with affective content, because in many conditions emotional content mandates a response from the organism. Yet, almost nothing is known of how emotions and visual short-term memory interact in guiding behavior. Only a handful of studies have looked at the influence of emotion on short-term memory, with controversial results. Some authors deny any selective effects of emotions on STM (Bianchin, Mello e Souza, Medina, & Izquierdo, 1999; Izquierdo, Medina, Vianna, Izquierdo, & Barros, 1999; Quevedo et al., 2003). However, at least two recent studies provide hints as to possible emotional effects on VSTM and/or attention (Anderson & Phelps, 2001; Kensinger & Corkin, 2003). In contrast, the influence of emotional content on encoding and retrieval is well characterized for long-term memory: Emotionally charged information is encoded and retrieved more efficiently than emotion-neutral information (Hamann, 2001), and the role of the amygdala in this context has been particularly well studied (Cahill & McGaugh, 1998; Klüver & Bucy, 1937; McGaugh, McIntyre, & Power, 2002).

Beginning with Wundt (1916), it has been recognized that the emotional experience varies along two principal, independent axes: valence, or degree of pleasantness, and arousal, or intensity of the evoked emotion (Lang, Bradley, & Cuthbert, 1990; Osgood, Suci, & Tannenbaum, 1957; Russell, 1980). Following such a model, the emotional quality of any stimulus can be classified by asking observers to rate the stimulus on appropriate scales along the two dimensions (valence and arousal) and reporting the obtained scores as the coordinates of a point in a cartesian plot (see Figures 1 and 2). In the present study, we have investigated the temporal dynamics of information accumulation in short-term memory for real-life scenes with affective content varying in valence and arousal. We propose a simple model (Memory model) of the dynamics of information accumulation in VSTM during an RSVP task and demonstrate that its independent parameters are modulated orthogonally by valence and arousal (Experiment 1). From this we argue that arousal and valence must act as largely independent factors on VSTM. Arousal modulates the time constant of the memory accumulation process for both negative and positive valence images. In contrast, the effect of valence is specific for negative images and consists of a progressive acceleration of information accrual with increasing image exposure. We provide evidence that such valence effects are unlikely to be due to trivial differences in image statistics (Image statistics) or to greater difficulties of semantic categorization at short exposures for negative valence images compared to positive ones (Experiment 2). Finally, we show that the effects observed are most probably specific to the memory mechanism, because they are not found in a recognition task where memory load is thought to be minimal (Experiment 3).

The main dependent variable in our study is the fraction of pictures recognized as function of exposure time. We take this fraction to represent the probability that a given picture will be remembered when seen for a given exposure within the RSVP stream. As will be seen in the data, cumulative recognition probability increases monotonically with exposure, presumably reflecting the accumulation of information extracted from the stimulus and stored in memory.

We assume that following the stimulus’ appearance a number of symbolic representations are formed, embodied in the neural responses to various features of the stimulus. The information represented in these various neural activities competes for access to a temporary memory store, where a cumulative representation of the stimulus at hand is being formed. We envisage such information accumulation process as a form of sampling with replacement. Due to capacity limitations, within a given temporal bin only a fraction of the total stimulus information is accessed. Because sampling is with replacement, the incremental gain of stored information per unit time will be large immediately after the stimulus onset and will subsequently diminish due to repeated re-sampling of identical bits of information. This incremental information gain is reflected in the increasing cumulative probability of a correct response with increasing exposure.

We seek to formalize in terms of a probability the act of sampling and storing in memory a limited amount of information per unit time. The concept of hazard rate is the most appropriate for this task (Luce, 1986; Ross, 2003). The hazard rate of information accumulation in memory is the conditional probability that a bit of information never sampled previously is sampled and stored in the current unit of time. In the context of the RSVP experiment, the hazard rate can be taken to indicate the probability that a picture will be correctly remembered if seen for an extra instant of time, having failed to be remembered when seen for a shorter time.

The hazard rate h(t) can be expressed as

Consider the case of a constant hazard rate:

This is the familiar exponential function, which has been used to model the results of short-term memory experiments in many previous studies (Bundesen & Harms, 1999; Lamberts, Brockdorff, & Heit, 2002; Loftus & Ruthruff, 1994; Shibuya & Bundesen, 1988).

However, there is no a priori reason to assume a constant hazard rate: Information accumulation could accelerate or decelerate over time, necessitating a model able to accommodate increasing or decreasing hazard rates. Consider then the following hazard function:

Here, the hazard rate decreases, remains constant, or increases in time with choices of β < 1, β = 1, β > 1, respectively. The corresponding distribution function

is the well-known two-parameter Weibull function. Note that when expressed as log(t), the function changes scale (shifts laterally on the x-axis) but not shape (steepness) with changes in α, whereas it changes shape but not scale with changes in β. We adopt Equation 6 as our model for the accumulation of data in short-term memory. While our choice is dictated by the necessity to allow for a range of hazard rates and by the ease with which such property is embedded in a single parameter in the case of the Weibull function, such a choice might have a deeper justification if it were possible to demonstrate that data accumulation in memory can be described as an extremal process, in which case the Weibull would be the appropriate limiting distribution (Kotz & Nadarajah, 2000).

Subjects were students at The University of Chicago, aged 18-25 years, naïve to the purpose of the experiments, without prior exposure to the image set, and with both sexes equally represented.

The International Affective Pictures System (IAPS) set (Lang, Bradley, & Cuthbert, 1999) of 384 color images was shown to each subject in each task, each image only once per subject. The images were chosen to represent a variety of combinations of all possible arousal and valence levels, clustered as shown in Figures 1 and 2 to facilitate experimental counterbalancing and subsequent data analysis. All images subtended 8 x 10 deg of visual angle (when the size and/or aspect ratio of the original IAPS images differed from such template, they were cropped and scaled as needed), and were presented surrounded by a black background on a computer monitor at a refresh rate of 75 Hz. The experiments took place in a dimly lit room. Subjects entered their responses on the computer keyboard. The presentation procedures were programmed in-house on an Apple G3 computer using C and the VisionShell routines by Raynald Comtois (Comtois, 2000-2003).

The RSVP task consisted of 24 self-paced consecutive trials during which each subject was exposed to the entire stimulus set. On each trial (see Figure 3 for an example), subjects were shown a 10-image stream, followed after a 1.5-s blank delay by a test phase in which 16 pictures where shown singly, 8 new and 8 old randomly interspersed. To eliminate primacy and recency effects, each stream began and ended by a neutral, very low arousal image, and these images were never tested for recall. Within each stream, images appeared for a set exposure (ranging from 13 ms to 4 s) and without interruption between successive frames. During the test phase, each test picture stayed on the screen until the subject indicated whether it belonged to the previously seen stream. Two practice trials (not entered into the final analysis) preceded 24 experimental trials. Each subject was tested at 4 different exposures, for a total of 96 subjects over 15 exposures.

To ensure a maximal degree of randomization, the experimental design was carefully constructed according to several counterbalancing rules. Within the limits imposed by the high dimensionality of the parameters’ space, the pseudorandom design matrix thus minimized the chances that the results were contaminated by properties of individual images, serial presentation and serial testing position, combinations of particular images, predictability of the upcoming arousal/valence category, and primacy and recency effects.

Previous pilot rating experiments had indicated that at long exposures arousal and valence rating scores obtained from the subject population of the current study were very highly correlated to the normative IAPS scores (r_arousal=.72; r_valence=.94). Subjects’ responses from the RSVP task were then parsed into eight arousal categories (arousal ratings 3-3.5, 3.5-4, 4-4.5, 4.5-5, 5-5.5, 5.5-6, 6-6.5, and > 6.5) and seven valence categories (valence ratings 1-2, 2-3, 3-4, 4-5, 5-6, 6-7, and > 7) according to the normative ratings of the IAPS set. Within each category, proportion correct recalls were calculated at each exposure. The scores were corrected for guessing by subtracting the false alarm rate from the hit rate. Scores never reached 100% correct even at the longest exposures: The errors were interpreted as lapses of attention or random errors in motor choices. As none of these factors were the focus of the experiment, they were removed by normalizing the corrected-for-guessing scores to the asymptotic value, taken as the average score at the two longest exposures (in all cases 95% or better). Scores were then fitted by nonlinear regression with a two-parameter Weibull function of the form indicated in Equation 6. All reported fits corresponded to R² > .96. To obtain non-parametric estimates of hazard rates, probability densities were calculated from the cumulative scores and then divided by the survival rates (as detailed in Results).

This experiment revealed two effects: (1) Valence (pleasantness) and arousal (intensity of the evoked emotion) independently affect memory accumulation, and (2) recognition memory for negative images demonstrates an increasing hazard rate, whereas memory for neutral and positive images follows a constant hazard rate. An illustrative summary of these findings is presented in Figure 4, which shows the measurements and associated best-fitting Weibull functions for the two extreme arousal and valence categories. In the arousal graph, subjects’ scores where averaged across all valence categories; conversely, in the valence graph, subjects’ scores were averaged across all arousal categories. In semi-logarithmic coordinates (proportion remembered vs. log time), increasing arousal shifts the curve leftward (decreases α) signifying overall improvement in performance, whereas a change from positive to negative valence increases the slope of the curve (increases β), indicating a progressive acceleration of correct response accumulation as exposure increases.

The subjects’ behavior across all arousal and valence categories can be parameterized by expressing the results in terms of the Weibull model’s α and β values, as shown in Figure 5. The α values show a linear correlation with arousal (r = −.71, n = 8, p < .047), but do not correlate with changes in valence (r = −.00004, n = 7, p < .999). Conversely, β values correlate significantly with valence (r = −.87, n = 7, p < .01), but are constant as a function of arousal (r = .04, n = 8, p < .92). Notice that there are also clear nonlinear trends in the data: For example, in Figure 5, the α/arousal graph seems to asymptote to a constant level and the β/valence graph might be interpreted as a step function. These higher order trends could be the result of uneven sampling, the nonuniform coverage of all valence and arousal categories in the image set, or more specific factors affecting each particular condition. Nonetheless, the first-order, linear approximation is an important and robust generalization, and overall the pattern of results suggests that arousal and valence operate independently.

To independently assess the differences in the responses to the valence categories, without the potential contaminating constraints introduced by the parametric modeling of the data, we also calculated hazard rates directly from the observed scores, as indicated in Equation 1. The results, shown in Figure 6, indicate that indeed the responses to neutral and positive images display a constant hazard rate, whereas the hazard for negative images accelerates over time, particularly during the first 500 ms of exposure. Thus, the nonparametric analysis confirms the results of the parametric modeling, indicating that negative images are treated differently than neutral or positive ones.

Notice also that the mean hazard rate for neutral images is lower than for positive images, corresponding to a longer time constant. The same trend is also found in the parametric estimates (see Figure 5, top-right panel), where the average α value for neutral images is higher than for positive images. This difference is likely due to a sampling problem inherent in the structure of the image set, which introduces a spurious interaction between valence and arousal: Neutral images, compared to positive ones, lack high arousal content (as can be seen in Figure 1). Indeed, such a difference disappears if the responses to low arousal positive images are compared to the neutral ones. A similar interaction is present also for negative and positive images: On average, negative images, compared to positive ones, are scored higher in arousal. However, the analysis of subsets containing only low-arousal/negative (mean arousal=4.47) and only high-arousal/positive (mean arousal=5.80) pictures indicates that negative images have a slope significantly higher than unity (β = 1.30, CI = 1.13–1.48), whereas positive images do not deviate significantly from unity (β = .96, CI = .88–1.03). This suggests that differences in slope cannot be accounted for by biases in the sampling of arousal levels.

Improved performance with increased arousal may relate to increased alertness or a better sensory quality of the stimulus. Changes in arousal may affect the availability of processing resources or the magnitude of the sensory response, thus changing the amount of information made available per unit time. The effect of arousal can then be interpreted as modulating the time constant of the accumulation process, similar to the fast changes in gain induced by increasing image contrast (Loftus & Ruthruff, 1994; Stromeyer & Martini, 2003) or brightness (Kelly, 1961). Thus arousal modulates the α parameter of the Weibull model.

What about the β parameter? The results of similar RSVP tasks have been previously interpreted within the framework of a conceptual short-term memory system (Potter, 1976). Current models of conceptual short-term memory (Bundesen & Harms, 1999; Lamberts et al., 2002; Loftus & McLean, 1999; Shibuya & Bundesen, 1988) postulate that the proportion of correct responses at any given image exposure is equivalent to total information acquired in memory. According to this class of models, information accumulates in memory at a constant rate following stimulus presentation, as evidenced by cumulative frequency of response curves that can be fitted with simple exponential functions. Our data for neutral and positive images conform to such models: A β value approaching unity effectively reduces Weibull curves to simple exponentials.

The rate of correct response accumulation shows, however, a marked difference for negative compared to positive or neutral images: It increases with the duration of image exposure, requiring a β > 1. This effect is best understood by examining the hazard rates of the responses. An increasing hazard rate implies that information accumulation in memory accelerates over time. Neutral and positive images, which elicit constant hazard rates, appear to be processed in a manner where each successive state of the system is independent of the previous states. In probability terms this is equivalent to sampling with replacement and defines a processing mode that is essentially automatic. Negative images may instead trigger a more intelligent processing mode whose current state is influenced by the previous history and thus shows signs of a more controlled activity. The corresponding hazard rate starts lower, but then increases, indicating that memory encoding of information from negative images accelerates, particularly during the first 500 ms of exposure.

We are now facing the question of what is the nature of the mechanism responsible for such processing differences between images of different affective value. We examine first the two most immediate explanations, both based on the observation that negative images are at a disadvantage at the shortest exposures. Images eliciting negative affect might have poorer physical image qualities, such as brightness, contrast, and spatial frequency content, or might require more time to be processed conceptually, a conjecture tested in Experiment 2. The specific contribution of short-term memory is examined in Experiment 3, and more elaborate accounts are further discussed in General Discussion.

An extensive examination of the contribution of low-level image properties to the categorization of emotional images will be described elsewhere (for a preliminary report, see Maljkovic, Martini, & Farid, 2004). Here we provide a brief report examining basic image properties that might differ between valence categories, thus potentially forming the bases for the differential effects of valence on memory accumulation.

The 384 images from the IAPS set were analyzed for mean luminance, contrast, and spatial frequency differences. The mean (related to mean-luminance) and SD (related to contrast) of the intensity of all pixels across each image were calculated separately for each RGB color component and for their linear sum. The spatial frequency power spectrum was also extracted from a grayscale version of each image, and the obtained series of frequency coefficients was fitted with a power function of the form

Pair-wise correlations were computed between the obtained mean intensity, SD, and power coefficients of each image and its arousal and valence normative rating. Only mean intensity and power significantly correlate (p < .05) with valence (r_intensity=.12, r_power=–.16, N = 384), whereas none of the statistics covaries significantly with arousal.

For each of these image dimensions (mean intensity and power), two subsets were created comprising images with scores above and below 1 SD of the mean. A Weibull model was then fitted to the responses given within the RSVP task to each subset, yielding a comparative estimate of the model’s parameters across dimensions.

Pictures with high- and low-power exponent λ showed no difference in either Weibull’s parameter (α_low = 371, CI 319-423; β_low = 1.08, CI .87-1.29; α_high = 332, CI 287-377; β_high = 1.11, CI .9-1.32). High- and low-brightness images, on the other hand, did produce a difference in the α parameter, brighter images being recognized better at faster exposures than darker images (α_bright = 278, CI 255-301; β_bright = 1.02, CI .91-1.13; α_dark = 424, CI 383-465; β_dark = 1.17, CI 1-1.34). Correlation and RSVP data for the mean intensity partitions are shown in Figure 7.

A three-fold difference in mean-luminance corresponds to a difference of ≈150 ms in average speed of information accumulation in the RSVP task (α parameter), but does not significantly affect the steepness of the curves (β parameter).

In summary, no positive finding has emerged from the study of the interactions between image statistics and emotional responses. Basic luminance statistics correlate very weakly with the emotional dimensions. An appreciable effect on the RSVP responses can be observed only for large differences in mean pixel intensity, paralleling the effects observed with variances in arousal. Yet, there is no obvious correlation between arousal ratings and the mean luminance of images. While the role of low-level image properties needs to be explored more deeply, it is hard to avoid the conclusion that the image statistics analyzed so far do not drive the results obtained in the RSVP task, particularly as regards differences in the steepness of the curves.

In the RSVP task of Experiment 1, negative images were remembered less than positive or neutral images when presented at the shortest exposures. It is possible that negative images require longer times to be comprehended conceptually, thus explaining the reduced performance and the effect on the steepness of the psychometric curves. This conjecture was tested in the present experiment by examining the ability of observers to categorize images at all exposures.

The valence-rating task consisted of 384 consecutive, self-paced trials during which each subject was exposed to the entire IAPS set. On each trial subjects saw a single image followed by a nonsense, color, noise mask (see Figure 8). The mask consisted of a static checkerboard, where each square check had a size of 0.25 deg and a color extracted at random from the color palette of the test-mage. Images were presented for exposures of 13-1710 ms (counterbalanced across trials and across subjects), while the mask lasted 4 times longer than the test-image. After seeing each picture/mask combination, subjects rated the valence of the image on a 9-point scale and indicated the confidence of their rating on a 3-point scale. The instructions verbally given to the subjects on how to form their response were as follows: “After you have seen an image, you will be asked to indicate how pleasant or unpleasant is the situation represented. If the picture represents a very pleasant situation respond +4; if it represents an extremely unpleasant situation respond –4. You should rate neutral pictures that are neither positive nor negative as 0. For intermediate levels of pleasantness/unpleasantness use intermediate ratings (1 to 3 for pleasant, –1 to –3 for unpleasant). If the picture was presented so briefly that you couldn’t see much and therefore you couldn’t make up your judgment, respond 0. Do not think too long about the rating to give to any picture. You will then be asked to indicate how confident you are about your response. You can choose among completely sure, not so sure and totally unsure.” Each subject was tested on 4 exposures. A total of 48 subjects were tested over 8 exposures. The experimental design was counterbalanced so that 6 ratings were collected per image at each exposure from different subjects.

The subjects’ rating of any image was remapped to the range 1-9 (by adding 5 to each score) and assigned to a negative, neutral, or positive category, according to the normative valence score of that image published in the IAPS set. These three categories had valence < 4, 4-6, and > 6, respectively, averaged across arousal levels. At each exposure the ratings were averaged across subjects within each category. Hazard rates for categorization were calculated from the proportion assignment of each image to its asymptotic category at any image exposure.

The average ratings of the images given by the subjects are shown in Figure 9A. Affect builds up with increasing exposure: Positive and negative images are categorized as such increasingly more often as the pictures are seen for longer times. However, already with an exposure of one video-frame, the different categories are separately clustered, positive images being on average rated higher than neutral images (t = 5.39, p < .001, df = 1295) and neutral images rated higher than negative images (t = 7.08, p < .0001, df = 1294). Of specific interest here is the temporal dynamic of the categorization process: Is there a difference in the time-course of the semantic valuation of positive as opposed to negative images? Do negative images take longer to be categorized as such compared to positive images?

To answer these questions we coded as correct any response that had assigned an image to the same category as that to which it belonged, according to the normative, IAPS rating, and as incorrect those responses that did not correspond to the IAPS rating. For example, if picture Z was assigned by a subject a rating of 5 while the IAPS rating was 7 (i.e., a positive picture), the assigned score would be incorrect, because we took positive pictures to be rated as 6 or higher. We then calculated the proportion correct scores per exposure for positive and negative images and corrected the results for guessing (guessing rate was 33%). The final results of this calculation are shown in Figure 9B. This kind of analysis reveals no difference in the temporal dynamic of categorization for positive and negative images: The two curves overlap.

By analogy to the RSVP model, it is tempting to interpret the proportion correct score as a measure of the accumulation of stimulus information leading to the semantic categorization of the image. Following such interpretation, the hazard rate of the categorization process then indicates the probability that a bit of information not yet extracted from the stimulus is extracted in the next instant of time and contributes to forming a semantic valuation of the image. The hazard rates for categorization, calculated from the proportion correct scores, are shown in Figure 9C. The hazard curves for positive and negative images overlap, indicating similar temporal dynamics. Interestingly, both curves have a decreasing excursus and largely fade after 200 ms of exposure. Contrast this to the constant or increasing trend found in the RSVP task.

The valence-rating experiment was prompted by the conjecture that the conceptual meaning of negative images might be harder to grasp, particularly at the shortest exposures, thereby accounting for the steeper response curves found in the RSVP task. In favor of this hypothesis is the observation that images of negative valence tend to be encountered less frequently in everyday life, therefore lacking in familiarity to the subjects. Contrary to the hypothesis, we found that with an exposure of only one video frame both negative and positive pictures are categorized as such at a rate better than chance. Moreover, examination of the temporal dynamic of the semantic classification of the scenes revealed that the categorization process appears to proceed at the same rate for both positive and negative images. This implies that an early, selective categorization disadvantage for negative images is not a likely explanation for the effects observed in the RSVP task.

Recent studies (Keysers, Xiao, Foeldiak, & Perrett, 2001; Li, VanRullen, Koch, & Perona, 2002; Thorpe, Fize, & Marlot, 1996; VanRullen & Thorpe, 2001a) have demonstrated very clearly that several categorization tasks can be successfully performed with very brief exposures: The present finding extends this ability to the domain of emotional valence. Yet, it might be argued that the mask we used was not powerful enough to arrest processing. Indeed, the checkerboard mask contained most of the power at one spatial scale, was static, and its duration was proportional to the test-image exposure. However, in follow-up experiments to be described elsewhere (for a preliminary report, see Maljkovic et al., 2004), we have used multi-scale, dynamic masks presented for 2 s at all image exposures and have obtained virtually identical results. Finally, one widely accepted marker for mask effectiveness is the fact that performance decreases constantly with shorter stimulus onset asynchrony, as can be seen clearly in the data.

The examination of the temporal dynamic of the semantic classification revealed a surprising fact: Unlike the RSVP task, in which scene information keeps accumulating steadily up to 2 s of exposure, the scene categorization task seems to rely on a more transient response, where most of the information relevant to the task is extracted very early. This difference is evident in the pattern of the hazard rates: Hazards are constant or increasing with exposure for the RSVP task, but peak early, then decline and fade to a very small value by 200 ms of exposure in the categorization task. It is tempting to speculate that the different dynamics observed in the two tasks might be related to the different demands on memory, a conjecture further examined in Experiment 3.

The matching-to-sample task consisted of 192 consecutive, self-paced trials during which each subject saw 192 images from the IAPS set. On each trial subjects saw a single image presented in the top half of the screen for exposures of 13-100 ms (counterbalanced across trials and across subjects). Following the study exposure, the picture was immediately replaced by a display containing 3 images: The study image was replaced by a picture mask drawn from an additional set of 192 images of neutral emotional content; in the lower part of the display the study (“seen”) image reappeared to the left or right (chosen randomly) of a comparison (“unseen”) picture drawn from the IAPS set and having similar arousal and valence rating (see Figure 10). These three images remained on the screen until the subject indicated which picture (lower left or right) matched the “seen” image. Each picture (seen, unseen, and mask) was presented only once per subject. A total of 24 subjects were tested over 4 exposures.

Each image was assigned to a negative, neutral, or positive category, according to the valence score of that image published in the IAPS set. These three categories had valence < 4, 4-6, and > 6, respectively, averaged across arousal levels. Within each category, proportion correct matches were calculated at each exposure across all subjects. The scores were then corrected for guessing by subtracting chance performance from the observed fraction correct and dividing by 1 minus chance performance.

As can be seen in Figure 10, performance in this task is very good: On average, subjects are able to correctly match 40% of the images (discounting guesses) at only one video-frame exposure and show perfect recognition with an exposure of 100 ms. No difference can be observed between the different image categories at any exposure. As such, these results parallel those obtained in the rating task of Experiment 2. Thus the results of this very simple task and those of the rating task of Experiment 2 demonstrate that when memory interference is low the responses to images varying in emotional content are indistinguishable. This suggests that the differential performance observed in the RSVP task has to be ascribed to mechanisms that selectively affect short-term memory accumulation.

Performance in RSVP tasks is summarized remarkably well by Weibull models. Perhaps the best way of showing this is to consider Equations 1, 2, and 6 together, leading to the following expression:

The slope of the function (shape, or β parameter) is unity for positive and neutral pictures, corresponding to a constant hazard rate, but it is greater than unity for negative images, indicating that with negative pictures the hazard rate increases with exposure time. The difference in intercept is also dependent only on β, as can be demonstrated by constrained regression with α as a shared parameter. The scale parameter α, indicating the exposure at which 63% of images are remembered, thus does not vary as a function of valence; however, it does vary with arousal. Across arousal levels it can differ by as much as 150 ms, as can be seen in Figure 5, and this range of variation is comparable to what can be obtained with a three-fold difference in the overall brightness of images (see Figure 7). Therefore, the parameters of the Weibull model describing recognition memory performance in the RSVP task correlate orthogonally with valence and arousal: Shape correlates with valence and scale correlates with arousal, but not vice-versa. As such, we interpret these results to indicate that valence and arousal affect short-term memory performance independently.

Osgood et al. (1957) was among the first to demonstrate by factor analysis that valence and arousal are two independent dimensions of the emotional experience, a fact accepted by many modern psychological theories of emotions (Lang et al., 1990; Russell, 1980). More recently, several imaging and electrophysiological studies have demonstrated that dissociable anatomical substrates are responsible for the effects induced by these two dimensions (Anderson et al., 2003; Arana et al., 2003; Kensinger & Corkin, 2004; Small et al., 2003; Yeung & Sanfey, 2004). The amygdala is often indicated as the primary structure involved in arousal-dependent responses, whereas the prefrontal cortex has been implicated in responses that vary with valence. It is thus not so surprising that this independence carries also to short-term memory.

The separate influence on performance attributable to each dimension has intriguing implications. Arousal effects are similar to those elicited by varying the mean-luminance of images. These effects could be entirely automatic, perhaps feed-forward and dependent upon the magnitude of response in the early filters (Loftus & Ruthruff, 1994), or feed-back in nature, such as due to gain control mechanisms (Wilson & Humanski, 1993), but with very fast temporal dynamics. Either way, the net result is an essentially instantaneous modulation by arousal of information uptake, with virtually no evidence that the speed of processing changes with increasing exposure. However, when confronted with images of negative valence, subjects’ performance bears the signature of a more controlled process, demonstrating a clear temporal dynamic. Short-term memory is impaired at the shortest exposures, but improves faster than for positive images at longer exposures. The signature phenomenon of this effect is an accelerating hazard rate. We interpret this pattern of results to mean that information accumulation in short-term memory is a monitored process, whose gain is adjusted quickly and automatically as function of arousal, but also actively and vigorously modified by the realization that an image has negative content.

We propose that the difference observed in the RSVP task between positive and negative images is specific to VSTM encoding. Three converging lines of evidence support this claim: Different valence categories share the same image statistics (Image statistics), making it unlikely that the effect observed depends on low-level image properties; the temporal dynamics of emotional categorization does not differ across valence categories (Experiment 2), suggesting that a cognitive evaluation of the meaning of the scene is available equally early for all emotional categories; and, finally, tasks such as simultaneous matching-to-sample (Experiment 3), which minimize short-term memory load, do not elicit a behavioral effect specific to valence.

The luminance statistics of images that we have examined do not correlate significantly with the emotional dimensions, except for image brightness. There is a small tendency for positive images to be slightly brighter than the average in the IAPS set. Not surprisingly, the effect of increased brightness is to shift the scale of the performance curve in the RSVP task, without affecting its shape. This is expected from the fact that brighter images might elicit stronger and faster responses in the early filters, thus increasing the amount of information made available per unit time. More sophisticated analyses of low-level image statistics and their ability to support an automatic classification of emotional content in scenes will be discussed in a companion study (for a preliminary report, see Maljkovic et al., 2004). Suffice to say here that we are unaware of any statistical algorithm capable of mimicking the human in this respect. It is after all not so surprising that low-level image properties cannot separate the emotional categories: Unlike other categorization tasks, for example, cityscapes versus landscapes (Torralba & Oliva, 2003; Vailaya, Jain, & Zhang, 1998), where same category exemplars share substantial physical similarities, emotional categories contain material that is very heterogeneous, and, therefore, very hard to classify based on statistical, physical image properties.

In Experiment 2 we have shown that observers are able to categorize the emotional content of scenes with very brief exposures, already giving a better than chance performance with only one video frame of exposure. Many recent (Keysers et al., 2001; Li et al., 2002; Thorpe et al., 1996; VanRullen & Thorpe, 2001a, 2001b) and older (Biederman, Rabinowitz, Glass, & Stacy, 1974) studies have shown that a very brief, masked presentation of visual stimuli is sufficient to allow for a variety of categorization tasks. These observations have sparked a lively debate concerning the nature of the underlying neural processing. In contrast to popular accounts of object recognition attributing an essential role to feedback activity from higher level centers (Mumford, 1991, 1992), the results of these experiments seem more compatible with an essentially feed-forward mechanism (VanRullen & Koch, 2003). While the categorization in many previous studies, similar in spirit to our own, could be based on differences in low-level image statistics (Johnson & Olshausen, 2003), such an explanation does not seem to hold in our case, a point we return to below. Whichever the mechanistic basis of such categorization ability, its temporal course is transient in nature: Most of the information accrual necessary to perform the task happens in the first ≈200 ms after stimulus onset. This temporal pattern distinguishes the categorization task from the RSVP task, which instead proves to depend on a sustained accumulation of information over at least ≈2 s of exposure to the stimulus. Furthermore, in the rating task, positive and negative images display similar categorization dynamics, unlike in the RSVP task. Thus, an early comprehension of the emotional content of the scene could allow for an intelligent management of the limited resources of VSTM.

The matching-to-sample-task that we have used in Experiment 3 does not require encoding in short-term memory and has been used as a baseline comparison in many previous studies of working memory (Gaffan, 1974). Similarly, our aim was to obtain a baseline with which to compare the effects of the RSVP task. We have found a total absence of a specific effect of valence on performance in the matching task, which converges with the evidence discussed so far in suggesting that negative valence exerts a specific effect only on VSTM.

Differences in long-term memory encoding and retrieval of emotional material are well documented: It is generally found that emotional information is remembered better than neutral information (Hamann, 2001). Given that the average emotional charge of negative images might have been higher than positive images in arousal and/or in valence content, one may wonder whether long-term effects could have contaminated our task and been responsible for the observed pattern of behavior. Long-term influences can be studied in the RSVP task by examining the effect of serial testing position (i.e., of the order in which each picture was shown during the recognition stage). If there is memory leakage and other factors being equal, pictures tested later in the recognition stage should be remembered less than pictures tested earlier. A detailed analysis of long-term memory effects in the RSVP task is beyond the scope of the present study and will be presented in a separate paper. Suffice to say here that memory for positive images, but not for negative images, shows a small leakage (≈10% across all exposures). The critical question is whether this memory leakage can account for the pattern of temporal dynamics we have uncovered in the RSVP task. The analysis of performance with images shown early and late in the recognition stage reveals that the shape of the psychometric functions does not change with testing position. Thus, a selective long-term memory leakage does not seem to be the source of the effects we have documented.

A variety of other accounts based on serial position effects could also be entertained. We address here two accounts most commonly discussed in RSVP studies. First is the possibility that performance with the current image is better or worse when preceded by a negative image than a positive one. We have analyzed the performance with neutral images when preceded by negative or positive pictures and have found no evidence of serial presentation position effects in this case. The second account is the possible lack of “attentional blink” (AB) for negative stimuli. The attentional blink refers to the finding that a second target is frequently missed when it appears 200-500 ms after the onset of a first target. In a recent study, it was demonstrated that negative words presented within an RSVP stream do not suffer the AB (Anderson & Phelps, 2001). The present data are compatible with the concept of a reduced AB for negative material inasmuch as performance with negative images at exposures longer than ≈400 ms is at least as good or better than with positive images. However, if this type of effect is to explain the present data, then one should predict a stronger attentional blink for negative than positive pictures at very short exposures (<100 ms), an effect which would disappear or reverse after ≈250 ms. To our knowledge this pattern has never been demonstrated. Thus no known serial position effects account for the memory accumulation results we report here.

What then is the nature of the effects of valence on VSTM? The storage function of VSTM is limited and slow to be deployed. Thus it has the characteristics of a precious, limited capacity resource, which needs to be carefully managed. Selection and management mechanisms need to be in place to direct the appropriate deployment of such resource. One of the most widely studied of such mechanisms is attention: It ensures that task-relevant stimuli are processed better and faster at the expense of other, less relevant stimuli. It is the motivational system, however, that understands the wider goals of the organism in terms of approach and avoidance reactions. Thus it is reasonable to assume that optimal behavior needs the integration of both systems, the attentional and the emotional. Attention speeds up information accrual: This was clearly demonstrated by Carrasco and colleagues (Carrasco & McElree, 2001), who showed that the time constant of the exponential function describing accuracy of visual search as function of display exposure is faster when the search array is preceded by an exogenous informative cue. It is tempting to speculate that negative valence might act as an endogenous attentional cue, thus leading to a speed-up of information accrual similar in nature to that obtained by exogenous cues. However, unlike Carrasco, who proposes that transient covert attention increases the gain of early detectors (Carrasco, Ling, & Read, 2004), we envisage the effect of negative valence as a later process, acting at the level of short-term memory storage. The time-course of the effect we observe seems too slow to be compatible with a classic gain control mechanism, and no evidence of such effect is observed in tasks where short-term memory involvement is presumed to be minimal (Experiments 2 and 3). Furthermore, the effect appears complex: Recall that the hazard rate for negative images in the RSVP task not only accelerates over time, but also starts from a lower level than for positive or neutral images. Reluctant as we may be to put forward an explanation of such a feature at this stage, it is possible that image features most diagnostic of negative content attract most of the early processing resources at the expense of the wider context, thus hindering recognition memory (conjectures of similar kind have often been advanced in past studies and most recently by Kensinger & Corkin, 2003). Thus, a more appropriate metaphor for the effects of negative valence might be that of a gating system, decreasing then increasing information flow to visual short-term memory. From a teleological standpoint, neutral and positive images do not represent a challenge to the organism, and information accrual from such sources can be let to follow a free-flow, random sampling regime. Conversely, negative images might represent a threat that requires a fast and costly reaction, thus demanding a more accurate and controlled flow of information.

This research was supported by National Institutes of Health Grant R01-EY13155.

We wish to thank Hany Farid for consulting on image statistics, Mari Sian Davies for piloting the memory experiment, and Julie Berger for her invaluable contributions to all aspects of experimental design. We thank the three anonymous reviewers for their very constructive criticism and helpful suggestions.