Two acoustic voice cues primarily define the perception of voice gender: the mean fundamental frequency (F0), related to the glottal pulse rate and perceived as a speaker’s pitch, and the formant frequency distribution defined by the vocal tract length (VTL), related to speaker’s size and adding to the characteristic vocal timbre. F0 and VTL are used beyond voice gender categorization and can enhance intelligibility in conditions where multiple talkers speak at the same time. Previous research has shown that voice perception, tested in speaker discrimination or identification tasks, could be altered by linguistic processing. Related to this, the advantage of the familiar language in voice perception tasks has been described as a robust effect. However, it is not clear at which linguistic level this familiarity effect arises, and how this effect influences the perceptual use of certain vocal parameters such as F0 and VTL for recognizing characteristics of a speaker. Here, we studied the effects of lexical and phonological processing on the weighting of F0 and VTL cues for perceived voice gender categorization in normal-hearing native speakers of Dutch. In a two-alternative forced choice paradigm, participants listened to monosyllabic words and matching nonwords and performed a voice gender categorization task. Responses were limited to “woman” or “man”. F0 and VTL in three female reference voices were manipulated to produce different voice samples on a female-male continuum. The perceptual weighting of F0 and VTL cues for these categorizations was estimated by the coefficients for F0 and VTL factors in a logistic regression model. Linguistic effects were tested by comparing F0 and VTL weighting coefficients in three linguistic conditions: Effects of lexical processing were tested by comparing words with phonotactically plausible nonwords and for effects of phonological processing, nonwords were presented forward- and time-reversed, so that nonwords did not adhere to the phonotactic rules anymore. Listeners gave significantly more perceptual weight to F0 and VTL when listening to words and phonotactically plausible nonwords compared to time-reversed implausible nonwords, while there was no difference in weighting these cues between words and plausible nonwords. This advantage of linguistic stimuli that adhere to the phonotactic rules, regardless of whether they are meaningful words or not, indicates that the weighting of F0 and VTL cues is altered by phonological, but not by lexical processing when using these voice cues for categorizing the speaker’s perceived voice gender.