The fly bolt is bigger than normal, so it doesn't work properly with a collet stem. Instead of pressing on the actual collet piece like it should, it rests on the very edge of the top half of the stem, and thus compresses the collet slightly while you're adjusting the bearings. It's my own fault for not simply turning the bolt down a little, but as I said, it still seems pretty stupid to use such a soft material for a bolt.