Does going to a graduate school with less prestige than your undergraduate school hurt you?

I went to a prestigious undergraduate school and finished strongly. In my search for graduate schools, the ones specializing in the field I want are less prestigious than my undergraduate school.

The average starting salary with a graduate degree from one of them is actually $30k less than my starting salary with just an undergraduate degree.

If I attend a less prestigious graduate school, will that cancel out my undergraduate degree? Could I end up making less money with my graduate degree from one of these less prestigious school? Will employers make the false assumption that I attended a lesser graduate school because I did poorly as an undergraduate?