The reason those with faith are seen as ignorant should they not sway from their views whereas the atheist is seem as making an 'intelligent decision' for standing firm is that religious beliefs have very little convincing evidence on their side. It is much easier for many to put suffering down to natural causes than it is to consider it being the work of a god, as this would lead to the issue of why a god would cause suffering. Similarly, it is easier to say "the world was created by chance" than to believe it was created by an omnipotent being. This is because the latter explanation requires faith in something we have very little knowledge of, whereas chance is something that occurs in everyday life and so it is easier for us to believe it.
In short, atheism is easier to explain with what knowledge we have of the world around us than faith, which requires trust in things we do not understand.
However, I think you will find that many with faith will see an athiest not swaying in their views after hearing a religious view as being ignorant; it really works both ways. I agree with you that many of those who describe themselves as atheistic do listen to those with faith with ignorance, though regretfully this problem exists in the opposite way round also.