While adding a resistor to get an approximate 100 ohm impedance will protect the decoder from overload, you may not hear much if any sound from the speaker. I'm going to use round numbers to explain. Pretend the speaker is 10 ohms, the resistor would then be 90 ohms to get a total 100 ohm load. The voltage put out by the decoder will split across the two devices proportionally. The 10 ohm device gets 10% of the voltage, the 90 ohm device gets 90%. At full volume nearly all of the voltage (and thus power) is going to the resistor. Only 10% of the voltage is going to the speaker. This would result in low volume, maybe not heard at all. To get any appreciable volume, you want the speaker to get most (all) of the output voltage. You really want something close to 100 ohm for a speaker.
Lyle