Conflicts with colleagues as a postdoc - reason enough to start looking for the next job?

I have recently started as a postdoc at a well-regarded foreign university. I like the place and I have no particular problems with my professor (even though his style isn't exactly to my taste, but I could live with that and take the difference as a learning opportunity).

However, I have a major problem with my fellow postdocs and other junior colleagues. I get the feeling they're just too full of themselves -- but whatever the reason, the situation is that I simply want nothing to do with them. Given that our research group is the only group at the department whose research I could plausibly participate in at the moment, this leaves me with practically no opportunities for fruitful daily scientific interaction.

I'd like to return to my home country after a couple of years of postdoc positions abroad, so I'm feeling some pressure to get the most of my time. I have often heard that the most valuable thing you get out of postdoc positions is the contacts you make. If this is the case, I'm probably wasting my time at my current job. On the other hand, my current institution has much more resources, scientific activities and opportunities to meet distinguished visiting researchers than pretty much any place else I might get to.

So the question is: what should I prioritize when deciding whether to continue at my current job? Is it worth tolerating intolerable colleagues for the opportunity of working at a prestigious and well-endowed university?

I'm sure a recommendation from my current professor would weigh a lot when looking for the next job, but I'm also quite sure I haven't been able to prove myself to him yet.