Discussion Forum—A Way with Words, a fun radio show and podcast about language

A person's perspective depends on the implied relationships involved. Water heater might be paraphrased as Heater of water. Hot water heater might rather be paraphrased as Heater for hot water.

It's not an uncommon construction for a noun phrase to contain the result of the action or even the final product in mind, i.e. X to make/produce Y. Some examples:

popcorn popper
tea kettle
candle mould
cake batter
pie filling
lightbulb
cookie sheet

hot water heater is another example of a noun phrase focused on the outcome, rather than the ingredient. Perhaps the emphasis on the product came about to distinguish between the heater that produces the house's hot tap water as opposed to the heater of water that heats the house through radiators.

Hot water heater

Here in Mexico it's called a boiler, even though there's something wrong with yours if it boils your water. Ironically, the "proper" Spanish term for it simply means water heater.