A model of cross-linguistic speech perception is presented in this paper. Two key characteristics of the model are that (1) it makes use of a universal auditory perceptual map, which is unchanged by linguistic experience, and (2) cross-linguistic perceptual differences arise from the lexicon. Two simulations illustrating how such a model can produce language-specific perceptual patterns are presented. The first simulation derives the Japanese listener's perceptual merger of American English [r] and [l], and shows how this effect can be sensitive to experimental task, by reducing or enhancing the influence of the lexicon in the perceptual task. The second simulation illustrates the impact of different types of perceptual assimilation of foreign sounds. American English listeners' discrimination responses for three Zulu consonant contrasts (Best, et al., 2001) are simulated. Finally, results of an empirical test of this model for fricative perception by Dutch and English listeners shows that results for the two groups of listeners do not differ in a speeded discrimination task, while linguistic differences are apparent in a subjective similarity rating task.