Ok, now I am going to sound contradictory, I don’t mind racing when it’s wet! It’s the one time where the protocol is ok to ride on wet trails and really get a macho muddy moment!
Having been the race organizer last year and a volunteer for several prior years for the Sizzler XC race in Greensboro, I understand why you have to have a "race no matter the weather" policy. The schedule dates are few and far between, scheduling with sanctioning bodies, racers traveling from out of town. Rain storms can pop up just minutes before races and to cancel due to a shower would be chaotic. It is also one of the biggest revenue generators for the local club sponsoring the race so if you cancel, it can really hurt the local club financially.
Yes, 200 racers on a muddy track can do some real damage, but the organizing club accepts that risks and is prepared to rally the volunteers afterward. Worth the price to have 200 racers from across the state and more to experience your trails and drive revenue for future trail maintenance.
The difference is that racing if wet is just one weekend out of the year. Bandit riders that continue to ride in all weather at their will year round, do continual damage year round that makes it difficult to keep up with and frustrate local trail volunteers.