Think for a moment how challenging it must be to review a product for one of the publications that do so. You're sent a new product, perhaps a DAC, and you're asked to let the world know how it sounds. You replace the DAC you use as a reference and put in the new one. It's the same process we would do if we bought one and took it home to try.
The DAC sounds better, worse or the same as the one you're comparing it to. A thorough reviewer would try different cables, sources, musical selections etc. and finally arrive at a decision and write it up. It's a tough job, one I have a great deal of respect for those who are really good at it.
But consider that what's really happening is a single piece of equipment is being tested in a complex system of many other variables. The reviewer or the Audiophile is unlikely to try 10 different pair of speakers, 10 different amplifiers, cables, sources and rooms. No, of course not, that would be silly. Yet, changing one thing in a complex system tells you only how that piece interacts within that system. You know more, but you don't know everything and cannot.
The same is true for any manufacturer trying to evaluate the next design change.
For example I am in the middle of testing some new designs our engineering group is working on and I do the same thing: i replace what is with what's new and evaluate. My speakers are still the big Magnepans and I know, as do you, that not everything sounds the same on Maggies as they do on Wilson's or Avalons or Martin Logans. Yet I can't change my speakers to test for all variations.
So how does one make an informed decision on variables within a complex system? Is it possible? Sure it is because we all have our favorite reviewers whom we trust and they face those challenges daily.
Tomorrow let's find out how to start applying the art of listening.
