Is there some survey of vowels and diphthongs of how some might be the most distinguishing sounds of a language ? For English, it must be the [Ä«] sound (as in five), even though it's not even the most frequent. Only a feeling. Though the idea fits rather well with the 'stiff upper lip' attribution.
Scanning the chart of vowels and diphthongs quickly the ones that stick out as characteristically English are the "short a" as in cat, the short "oo" sound as in book, and the "aw" sound as in caught. On another forum where we talk a lot about pronunciation there's a lot of haggling over the way vowels are pronounced in British vs American English, and one that's always part of the argument is one of the long "ah" sounds of UK English that some claim doesn't even occur in the American version.
Diphthongs like "ow" and "oi" seem to be as common in other languages as they are in English, so nothing to add from that category.
The most characteristic sounds of English may be the voiced and unvoiced "th" sounds, heard in then and thin respectively.
The most unique sound in English is the "r". It's the one that speakers of other languages just can't get exactly right.
Perhaps more so when r is deep inside, modifies something before it and then blends into something else again, as in Charles, gorge ? I bet there is whole science behind that.
To me, the 'r' and the flap t in American English are among the distinguishing factors of English. I like to mention 'l' as well, especially at the end of the words, and I think, just in American English. In an episode where Martha and Grant were talking about some (I think, old-fashioned) 'names', they mentioned 'nyargle'. I would hear it as nyargoe until I looked it up. This is more tricky in Estuary English, where 'milk' is pronounced as mewk!
The voiced and unvoiced th also happen in Arabic.