Error-driven channel re-weighting of early sensory representations accounts for temporal dynamics and switch costs of perceptual learning in a non-stationary environment. Learning was evaluated for orientation discrimination of peripheral Gabor targets (+/-10 deg) in two filtered noise "contexts" with predominate orientations at either +/-15 deg. The training schedule alternated two-day blocks of each context. We tested 3 target contrast levels. Training with feedback improved both discriminability and speed within and across blocks. However, there was a cost at each context switch. Cost magnitude (about 0.3 d') remained constant over 5 switches (9600 trials). For context-congruent targets, accuracy paradoxically decreased slightly with increasing Gabor contrast; for context-incongruent targets, accuracy increased substantially with Gabor contrast. A computational model accounts for all these results. Visual stimuli are first processed by standard orientation and frequency tuned units that incorporate contrast gain control via divisive normalization. Learning occurs only in the connections to decision units; the stimulus representations never change. Weights are updated by an incremental error-correcting rule that tracks the statistics of the environment. Task-correlated units gain strength while irrelevant frequencies and orientations are suppressed, producing a gradual learning curve. The optimal weight vectors are impacted by context because the background noise corrupts the predictive value of congruent channels. If the context shifts abruptly, the system lags behind as it works with suboptimal weights until it readapts, creating switch costs of approximately equal magnitude across successive changes in context. The normalization and nonlinearities in the system cause greater damage to the congruent channels, making the incongruent ones more predictive. This accounts for the counterintuitive congruence-by-difficulty interaction.