maybe a few clarifying questions can help:
1. these metal shavings that you see, its from the races because the whole thing has fallen into pieces whenever you take out the fork? the few times that it happened, i assume they looked like they failed in a similar fashion? this itself points to the races not sitting flush on a perfectly flat surface. if the surface that the race sits on is "wobbly" or crooked, all the pressure from riding puts stress on to it. maybe you can check the fork/lower race of headtube surfaces? using a ream and face tool on them may solve the issue.
2. you said the ball bearings are fine, do they have any flat spots on them? look very closely. using bearing balls that're of a lesser quality (less hard, less round etc) shouldn't really destroy your bearing races, but is worth a check over.
3. i'd assume you use the usual bearings where there're about 8 balls held in equal spacing with a crown? is it distorted in any way when you take it out? damaged? it may shred some light on the issue. personally i've never had a headset as smooth as a FSA unsealed pig with looseball bearings (i.e. without crown, just more balls) that Sheepdog taught us to do wayyyyyy back in the day. not even a properly installed chris king non-integrated/internal.
4. a headset that's too loose usually results in flaring, not race damage. as for too tight, race damage as you described is possible but very unlikely, since it'd have to be noticiably over-tightened for that to happen. just for information sharing tho, sometimes a headset becomes tighter/looser than how you set it with the compression bolt when you tighten the stem's pinch bolts, due to minor distortion in things. you can test how self-tightening your particular set up is by only tightening the compression bolt to where the wobble is just gone, then check the resistance in turning the bars. you then tighten the stem pinch bolts and compare how much the resistance has increased. this may give you a better idea of how much preloading you need in your compression bolt.
5. how often does this happen? if the race is shattered everytime you loosen the forks, how often do you do that? every 2 weeks? 2 months?