In the cocktail party situation, people with normal hearing can usually follow a single speaker among multiple concurrent ones. To this end, the auditory system must decompose the mixture of sounds into meaningful streams (auditory scene analysis) and select the one with the behaviorally relevant information. At the same time, processing the irrelevant streams should be suppressed to some degree in order to conserve capacities and prevent distraction. However, there is no agreement in the literature as to whether the background is segregated into multiple streams/speakers. The current study varied the number of concurrent speech streams and investigated target detection and memory for the contents of a target stream as well as the processing of distractors. A male-spoken target stream was either presented alone (single-speech), in parallel with one male-spoken (one-distractor), or with a male- and a female-spoken distractor (two-distractor). Behavioral measures of target detection and content tracking performance as well as target- and distractor detection-related ERPs were assessed. We found that the detection sensitivity and the target N2b amplitude decreased whereas the P3b amplitude increased from the single-speech to the concurrent speech streams conditions. Importantly, the behavioral distractor effect differed between the conditions with one- vs. two-distractor (distraction by the female speaker was lower than that of the male speaker in either condition) and the target N2b elicited in the presence of two distractors was significantly smaller than that elicited in the presence of one distractor. Further, the voltage in the N2b time window significantly differed between the one- and two-distractor conditions for the same (M2) speaker. These results show that speech processing was different in the presence of one vs. two distractors, and thus, the current data suggest that the two background speech streams were segregated from each other.