If you can find the right color, "standard" mortar sand, from a masonry supply, is very close to the right size for N scale ballast. If I'd known that at the time, I;d have used it for my MILW branch, instead of buying ballast.
I got my first supply when the remodeled the building next to my then employer, probably 20 pounds, for free, as it was left over - in the middle of OUR parking lot. The rest I bought form a concrete block dealer, for $20/ton. Since I only wanted a box full, probably 75 or 80 pounds, and their scale was designed for large trucks, we guestimated it to be a dollar's worth!