I think my reasons for disliking him are pretty reasonable.
1- He was a defensive coach that couldn't coach his position. I don't make that comment without the supporting comments from Houston and his staff. Many times it was said how little the defensive players truly knew about playing defense. "Gap integrity" wasn't being taught, and if it was they didn't know what it was.
2- He was burning bridges in recruiting circles. He was giving offers to players and pulling them. There was an article (the RTD?) that expressed this and JMU was getting a bad reputation among HS coaches. Imagine what may have happened in a few more years.
3- Attitude to those outside his circle/sore loser. We all didn't care for our beat writer, but that doesn't mean that you belittle the guy. He screwed up by interviewing players without permission (not cool) and Withers ran him over publicly for it. When your spouse goes on public media after leaving JMU, it was a lack of class and not only made her look bad, but him as well.
Many of you seem to have forgotten his post game conferences after a loss. Instead of taking the loss like a man, he acted like a child.

There is no doubt Withers did some great things for us. He was a guy that gave JMU a shot in the arm and brought excitement back to the program. But you can only hold on to that for so long before the results prove otherwise. I don't just dislike him for what he did, but what he might have done to this program if he had stayed longer. Yes, we will never know (thank god), but there were too many red flags to suggest this program would have been hurting.
He left at the perfect time for him. His stock was high and he struck. I don't blame him for that.