Something has been feeling off about the feel of this layout and I've finally figured out what it is! I wasn't following the prototype closely enough. While this isn't a prototype location, there are plenty of mill towns for me to draw inspiration from and I had the basic layout wrong. I had the river, the road, and the mill in that order. It should be river, mill, road. The mills are all built on the rivers because they're all water powered. I had that realization as I was going to bed and the next day I rearranged things substantially. I've had two buildings glued together for years, but now I've separated them. One will go on the other side of the river. I'm adding a stone foundation to blend it into the riverbank.
I will relay the spur track with more of a curve to reach the new location of the Middlesex Manufacturing kit. Its not on the river, separated by a road and siding from the mill that will be on the river. I think I'll connect the two with an elevated walkway.
The future atmosphere may feel something like this:
I'm left with a large open space in the middle of the layout. I'm planning on scratchbuilding two large footprint mills, one tall one by the river and another two story one up against the track. There will also be a building by the dam to control water in the sluceway, which I will also model. There will also be a road bridge just downstream on the dam, represented here by the Evergreen package. Here's an overhead shot.