Identifying a facial feature (e.g. the eyes) is influenced by the position and identity of other features (e.g. the mouth) of the face, supporting the view that an individual face is represented as a whole in the human brain. To clarify how early in the time-course of face processing this holistic individual representation is accessed we recorded event-related potentials during an adaptation paradigm of the composite face illusion (CFI). Observers performed a matching task on top halves of two faces presented sequentially. For each face pair, top and bottom face halves could be both identical, both different, or only the bottom half differed. The signal was larger over the right occipito-temporal cortex at about 160 ms (N170) when the attended top half differed between the two faces than when identical top halves were repeated. Crucially, a larger N170 was also found when the top halves of the two faces were the same, yet the observers had the illusion that they differed (CFI). This effect was not found when the two face halves were spatially misaligned. These observations indicate that the earliest perceptual representation of an individual face in the human brain is holistic rather than based on independent face parts.