I'm perfectly fine with strangers coming to the trails, on a dry day, whilst being glass smooth, not casing, putting down some magic clay and doing some watering. Basically doing the same as the locals.
If they live locally, not lent a hand over winter, come down a couple times, i'm fine with that. If they live locally and come down all the time, i tell them to clear off.
I'm not cool with people coming down on a wet day, just expecting to ride, no offer to dig or do anything, then progress onto riding even though it's too wet and tearing up the top surface.
It's normally only kids that can be a bit reckless, they don't seem to have the common sense to see if something they're doing is ruining something. That or don't care.
Forgot about litter, it's not hard to take it home with you. that's one thing land owners/councils/busy bodies will pick up on.
I would have been fine with someone like eggit riding them, in those conditions.
However, i do think builders do have the right to be against it if they so wish.