Jeff: I agree. Once this thing is in position, it can't move, because the rail ends interlock. But all it would take is a stray piece of ballast to keep the rail from dropping in to place. The train might not derail, but the vertical misalignment would cause quit an impact, probably breaking parts on both the track and train.
As several of you have said, at least you can run through a misaligned point-type switch and all you wreck is the switch!