I've mentioned it before, but if you want CHEAP rock ballast, and aren't concerned with color, "standard" mortar sand is very close to N scale. A building supply or masonry dealer should have it. And, yes, there is a standard for mortar sand, it isn't just common sand poured into a bag!
I bought mine about 15 years ago, but at that time, it sold for $20 a ton here in northeast Kentucky. The concrete block dealer I went to sold it by the truckload, and trying to measure a cardboard box full was a little hard. Based on an old NP MOW guide for crane operators, and the volume of the box, we decided it was probably about 80 pounds of sand, so called it a dollar and we were both satisfied. I've been using it for scenery base and fills ever since. The only problem is that the only color they had was yellow-brown, the same as common dirt in this area, so certainly NOT what I wanted for ballast. The ground in the part of Montana I model is the same color, though, so sifting on some fine dirt/dust from our yard makes great ground cover.