If you put the real dirt down first, both fine & super fine, then the grout, it "fills in" the coarser real dirt and makes for almost-N-scale dirt.
I had the same thing happen first with the sifter: gave it a shake and clumps fell down.
The trick is to place the grout in the sifter while over the container, move the sifter gently to the place you want to grout, and then lightly hit the inside rim of the sifter with a spoon:
Combining different colors of grout (haystack & sandstone, haystack and nutmeg, nutmeg & charcol, grey & blah blah blah) helps give some variety, as does combining sanded and non-sanded grout:
Then, to get a dirt road effect, take your fingers and "sand" the grout by rubbing tracks & smoothing the surface a bit.
Ed: when I got clumps and tried to tamp it down, it cracked later after the diluted glue dried.
But maybe I did that wrong.
Oops: I guess this is the grout thread now