I think the problem is with the material you're carving. Sheet rock spackle would not be my first choice. Actually, my first choice is simple, old Plaster Of Paris. I don't need no steenking Hydrocal to carve rocks!
Good to see your reference photo. Looks like a lot of the detail we're looking at is pretty small and might not be too visible in a "cliff" or a rock cut. Yup, a rock cut, which is most likely what the second photo shows on your layout. The railroads aligned as much as they could by bypassing large rock formations, but in canyons, following rivers and streams, they blasted a lot of rock, and most rock faces that close to your mainline would be a blasted face, not naturally occurring.
Blasted faces are characterized by the vertical (or near vertical) remnants of the holes the construction crews would stuff Nitro Glycerin or (later) Dynamite into. These look like roughly vertical channels scoring the rock faces. Here's a photo I found online somewhere that graphically illustrates this:
Although this photo illustrates a big blasted face, sometimes the blasted face wasn't that big, and was only part of the cliff, so both natural and blasted surfaces would be present alongside the mainlines. There are a lot of examples of that in the prototype I'm modeling.
What I'm saying is that there's room for both natural and man-altered rock faces in model railroading scenery. But, ya gotta think of what actually went on during the building of your railroad, and model THAT, rather than just pick a nice piece of cliff by a stream and carve that.
The reason I love plain ol' plaster is because it's easy to control and color and texture. You can color it in a lot of ways, but powdered tempera colors are really easy to use, as is mixing in dirt (after you've baked it at 400+ degrees for an hour to kill the fauna and flora). Dirt and sand also add texture. Plaster needs to be somewhere between runny and cured to carve efficiently. To speed things up, you add salt, and it really sets off quick with a little sodium chloride. I like my plaster to set up quick so I mix in salt. My buddy Kelly likes to take his time, so he mixes in vinegar to slow it down.
When you've got your plaster in the right "zone" (somewhere between runny and rock hard) you carve in the the big features first using screwdrivers, sharpened popsicle sticks, Xacto knives, artist's spatulas...whatever. Splop blobs of semi-hard plaster onto the thinner plaster surface and carve the big rocks and cracks. Then, texture it using hairbrushes, boar bristle paint brushes, wire brushes of various stiffnesses, toothbrushes, sandpaper, damp (almost wet) natural sponges or whatever else you have on hand, until you get the effect you want.
Did I mention old file cards?? They're great for certain applications, as are bigger wire brushes.
When the plaster is semi-hard, the coarse details you've initially carved will be softened somewhat (or a lot) by using brushes. For rock surfaces that have lots of sharp edges, now's the time to start inserting and popping the plaster away. Squarish surfaces can be carved with scrapers, carving tools inserted into the semi-hard plaster then flicked outward for random edges. Do this over and over either horizontally or vertically depending on your reference photos.
Rock has "grain" and most of the time it's pretty clear what it is and what direction it's going. Slate almost always lies somewhat horizontally, and the fine lines can be easily scribed using a stiff fine wire brush. If the exposed slate faces are squared off, as they often are, scribe the lines heavily, then use a scraper, flat knife blade or artist's pallette knife to cut away the outermost surface to get that flattened, squarish look.
If your rocks are sandstone or weathered granite, then use a damp (almost wet) sponge to round off your carved pieces, then go back lightly with fine brushes and dental picks to make cracks and sharp edges where the rocks have cracked and broken, or are just about to break.
You can texture the plaster rock surfaces with coarse sandpaper too, or a ripped up foam sanding block by pressing these materials against the semi-hard surfaces.
After it's all set up and hard, you can continue to create sharp corners, cracks and breaks.
All of this goes really fast for me, and I rarely work on a section that's bigger than 8" X 8" because of my salt additive and setting up quickly. My buddy Kelly, on the other hand, will work for an hour or more on a piece, taking it slowly because he's added vinegar, so he's got more time.
Although I can't find a clear photo of my blasted rock surfaces in my cuts at Wilhemina pass, here's a good example of a rock cut and natural surface that I carved from several reference photos I'd taken of the actual place. This was done using salt as an accelerator, with black and brown tempera added to gray up the plaster. Washed with India Ink and various washes and dry brushings of color using fine artist's acrylic (so it won't fade) tube paints:
Here's a shot of Kelly's slow rock carving depicting Weber Canyon. You'll note that although in Weber Canyon, there are plenty of naturally occurring vertical lines, the vertical lines near the track are REALLY vertical, which indicates blast hole remnants. Since this area was blasted out in 1869, they're not as prominent as in the first photo:
Once again, Kelly's work on the rock carving. Note that he's got both slate and sandstone, which are common in the Weber/Echo Canyons and are extremely folded. You can see in the cut on the right side of the photo, Kelly has once again included blast hole remnants. His shale is very well done, and is almost exclusively accomplished using fine welder's wire brushes, with additional scoring being done with a pick or Xacto knife:
Hope this helps. Use plaster, not dry wall spackle. You'll have way more control.
Anyway, have fun. I love carving rocks and building scenery. It's by far the most rewarding part of the layout, especially when non-railfans see something they can identify with...rocks, mountains, cuts, tunnels, trees, grass...while all of us model railroaders are gawking at the superdetailed engines coming through!
