Someone I know just quit their job because they refused to spend any more time working with an asshole manager. They had been at the company for many years, and would have been what I would consider a reasonably loyal employee. Ultimately, that turned out to be completely over powered by the pain felt in dealing with the asshole manager.
Working in an environment that has no assholes is a good goal to have, so I'm supportive of the decision. Loyalty to people is far more important in my mind than loyalty to organisations. Its nice to find a company that has beliefs in line with your own, but those beliefs only come about because of the people that created them. Find those people, and work with them, regardless of what company they happen to be in.