Viewers can perceive the gist of a scene in a single eye fixation. What visual information contributes to such rapid gist perception? Previous work suggests that with brief stimulus durations, low spatial frequency information contributes more to gist perception than does high spatial frequency information (Schyns & Oliva, 1994). Color information also contributes to scene recognition (Oliva & Schyns, 2000), and given the low-pass nature of color vision (Anderson, Mullen, & Hess, 1991), low-frequency color information may be sufficient for normal gist detection at brief durations. We investigated gist perception using color and monochrome images with and without low-pass filtering to determine whether high-spatial frequency information is necessary for rapid gist perception and whether color information is necessary and/or sufficient. In an RSVP scene detection task, viewers saw a rapid sequence of 3 scene images and judged whether or not the middle image in the sequence matched a pre-cued scene label. The label matched on half the trials, and the images were drawn from a set of 10 scene categories. Performance for color all-pass, monochrome all-pass, color low-pass, and monochrome low-pass images was compared across groups of subjects for a range of display times (22-121 ms). Low-pass filtering disrupted scene gist detection, even at the shortest stimulus durations (22-55 ms), suggesting that low-spatial frequency information is not sufficient for rapid gist detection; high-spatial frequency information is necessary even at short durations. Removing color produced slightly better gist detection when the image included both low and high-spatial frequency information. However, removing color from low-pass filtered images greatly reduced scene gist detection. Apparently, color information is neither necessary nor sufficient for gist perception, but it can contribute when high spatial frequency information is absent.