I can't attest to the accuracy, or otherwise, of the actual Pasteur quote (usually some variant of "Bernard was right, the microbe is nothing, the millieu is everything"), but even without knowing whether he even said it, it is remarkably still possible to comment on their interpretation.
In 1878, Louis Pasteur found his germ theory challenged by Claude Bernard in a posthumously published paper that disputed Pasteur's claims over the nature of fermentation. Pasteur had claimed fermentation was caused by micro-organisms and if a medium was first sterilised and kept from reinfection, fermentation would not occur. Bernard meanwhile had claimed to have observed fermentation is just such conditions and claimed it was a property of the medium that brought about fermentation, not the action of some fanciful "microbe".
Pasteur immediately began a lengthy series of experiments to rebut Bernard's results, and accused the people responsible for the posthumous publication of Bernard's work, and Bernard himself, of shoddy experimentation, seeing only what they expected to see, etc.
Even so it is clear that Pasteur did not have as easy a time of it as he hoped and the issue was never settled to either party's satisfaction before Pasteur's own death in 1895.
It was only several years afterwards that the correct explanation was determined. Fermentation occurs by the action of enzymes on the sugar in the medium. These enzymes come from a living source, but that source need not be a microorganism. In many cases of fermentation the enzyme is derived from the cellular chemistry of the fruit base of the fermenting medium.
So in a sense both Pasteur and Bernard were right, and even today the fermentation of wines are routinely achieved through the addition of yeast, while the fermentation of cider traditionally derives from the maceration of the cells of the apples.
Well this was a fun project, even though it leads to a distinctly unsatisfying conclusion. I cannot say whether Pasteur ever admitted Bernard was right and he was wrong, but I can see that he very well might have, particularly if plagued with illness and doubts and faced with uncomfortable instances of "autofermentation". But the quote relates to a particular squabble over the process of fermentation, that can happen either through microbial or metabolic processes, to extend it to a general comment on the whole of germ theory is just wishful thinking.
But then, what else would you expect from people who's whole medicine cabinet contains nothing more than bottled "wishful thinking".