I've developed a couple of techniques for using UNSANDED grout for dirt.
I guess I go overkill with it, but here's what I do.
First I paint on a layer of paint and glue (craft paint and white glue). I usually jus squirt it on in big loops, and mix it around together with my brush. I usually mist this with some water to thin it out a bit to make it spread easier.
Then I sift the grout down on top of this wet mess.
Finally, depending on the circumstances, I do one of the following:
Brush the unglued stuff into some sort of containment vessel to collect and re-use.
Vacuum the unglued stuff up (use a vacuum with a really good filter, otherwise, dust everywhere).
Use a piece of broken off foam to pat the unglued stuff down flat, mist it with a water/alcohol mix, and then drip some glue into it. If it looks bumpy, you can sprinkle some more down too.
All of this takes some practice to get right, so I'd recommend using some scrap foam or something and practicing before you commit to the layout.