How harmful is it for a PhD student to work with a supervisor whose ambition is greater or lesser than that of the student?

How harmful is it for a PhD student to work with a supervisor whose ambition is greater or lesser than that of the student?

I can imagine that if the supervisor aims much higher than the PhD student the experience could be nightmarish for the student (and frustrating for the supervisor). Similarly if the supervisor aims too low (something like: "Just generalize this theory in way X, that should be enough for a PhD") the work could feel meaningless and the time spent on it wasted, when more challenging and maybe even more important questions could deserve the attention and resources.

Background: I am interested in a career in research. I am worried that if I aim too low in my PhD project it will hurt my career afterwards. PhD takes years and so one would ideally make most of the years spent doing it. Some supervisors are more ambitious and knowledgeable than others. I would imagine that a PhD with a more ambitious researcher (with talent and education to match) would improve one's chances of a career in academia than with a less ambitious one. I have read that the best thesis is a finished one, but surely one should think about how the PhD helps to further a post-PhD research career.

Could the difference be great enough to justify changing programs? Even if the funding is good?