Associate editor's dilemma with inconsistent first/second round review

Say I work as an AE for an international journal. I managed to find only two reviewers for a manuscript submitted to the journal lately. Personally, I don't like the manuscript at all (research taste involved...), but it is not fair to not base my final decision on the reviewers' comments.

For the first round review, both reviewers recommended a minor revision of the manuscript. Reviewer A appears to be more impartial than Reviewer B (who is positively biased). The same decision is delivered to the authors. However, during the second round, Reviewer A suddenly changed his mind to a major revision, which according to himself/herself is due to the failure of the authors to address his/her comments.

For this particular journal, there is usually only two rounds of review, and I don't think a third round will help improve the manuscript anymore. I am inclined to agree with Reviewer A and thereby reject the manuscript. However, it seems unfair to the authors and is hard to justify Reviewer A's change of mind in the second round.

Now I am inclined to reject the manuscript, but it has to be based on solid grounds. Solid grounds can be found (if I write the review myself, I can certainly make one), but the reviewers did not give me one: if the recommendation is minor revision for the first time, how can it be major revision for the second time? Therefore, it seems to me both the authors and the reviewers did a poor job.

So, how should I make the decision here? If I choose to reject, how do I appear fair to the authors?