There is no consensus on whether luminance-modulated (LM) and contrast-modulated (CM) stimuli are processed by common or separate mechanisms. To investigate this, the sensitivity variations to these stimuli are generally compared as a function of different parameters (e.g., sensitivity as a function of the spatial or temporal window sizes) and similar properties have been observed. The present study targets the sensitivity difference between LM and CM stimuli processing. Therefore, instead of studying the variation of sensitivity in different conditions, we propose to decompose the sensitivities in internal equivalent noise (IEN) and calculation efficiency (CE) to evaluate at which processing level the two mechanisms differ. For each stimulus type, the IEN and CE of four observers were evaluated using three different carriers (plaid, checkerboard, and binary noise). No significant CE differences were noted in all six conditions (3 carriers × 2 modulation types), but important differences were found between the IEN of the two stimulus types. These data support the hypothesis that the two pathways are initially separate and that the two stimuli may be treated by common mechanisms at a later processing stage. Based on ideal observer analysis, pre-rectification internal noise could explain the difference of IEN between LM and CM stimuli detection when using binary noise as a carrier but not when using a plaid or a checkerboard. We conclude that a suboptimal rectification process causes higher IEN for CM stimuli detection compared with LM stimuli detection and that the intrinsic noise of the binary carrier had a greater impact on the IEN than the suboptimal rectification.