Part 2: Testing 1, 2, 3.
Forget the stories and debates about “objective testing,” or its failings, that you read about in consumer magazines or hear about on audio internet sites. 99% of those discussions are meaningless, based on equally meaningless tests done too casually by people who are neither unbiased nor true experts; people trying to prove a point, one way or the other. Good objective testing takes time and care, and is used to understand, not advocate. Similarly, real audio science is conducted at research laboratories and universities, not magazines and trade shows. It is done by researchers trained in experimental design and dedicated to the fields of hearing and perception, people who have no stake in any particular outcome, publishing in journals that would be totally obscure to the vast majority of audiophiles.
It is important to recognize that objective testing is not the same as, “meter reading.” Meters may, of course, be involved in specific types of objective audio testing, but their use in modern audio research work is rather limited. Aside from routine, test-bench engineering tasks, the majority of objective tests and experiments include listening panels working with excellent equipment. Good experiments, the ones that other professionals put any stock in, are sincere, carefully designed, low stress and make every effort to be unbiased. They take into consideration the range of individual human hearing capabilities, from limited and naive, to Golden Eared and expert, and they don’t assume the outcome. Generally, these experiments fall into two different categories: experiments which attempt to understand the limits and abilities of hearing (in other words, can we hear some sound or detect some slight sonic difference or not?), and experiments which attempt to understand what people prefer. These are very different matters, each presenting its own unique challenges, and so the kind of experimental protocols used to conduct them also are different.
The first kind of experiment might ask, “What is the highest frequency that is audible to any listener, or which affects the sound in any way.” In contrast, the second kind of experiment might instead ask, “Do listeners prefer amplifiers with response over 20KHz, or not?” These are obviously very different questions, yet such questions are constantly being confused in the popular debate over objective testing. “Can anyone tell the sound of these amps apart?” Type 1. “Which of these amps do most listeners, prefer?” Type 2. “Can people taste any difference between Coke and Pepsi?” Type 1. “Do most people prefer Coke or Pepsi?” Type 2. And, while it is true that preference itself is complex and can not be measured or explained unambiguously, the testing and documenting of preferences can be done objectively.
OK, so what does this objective testing say about the current state of high end audio, once you toss out the dross? Frankly, it says that some of the things that audiophiles have long believed are no longer true, if they ever were. At least a few of the assertions audiophiles often make about the differences in sound between different pieces of gear seem to disappear in even the most careful and sympathetic experiments. Likewise, some people who insist they prefer Coke, actually choose Pepsi. Hey, don’t shoot the messenger, and don’t assume the all the experiments are junk. As I said in Part 1, some of us work very hard to figure out what is really going on, not what one “side” or the other wants to think is really going on.
These days, when it comes to purely electronic components like preamps and amps, the differences between competently designed and properly functioning models are vanishingly small, or even nonexistent. Even the differences between transports and converters are getting to be less than the variations you find between any two speakers with sequential serial numbers. Turntables and cartridges, well, we are still back in the 60’s when it comes to sonic neutrality, for better or worse. I’m intentionally leaving loudspeakers out of this assessment, since we are just beginning to understand how to even think about how they should ideally perform and, therefore, how to test them in a truly meaningful way.
Don’t bother telling me that I am a fool for saying, “All amps sound alike.” Yawn. I am no fool, and that is not what I am saying. No audio engineer has ever seriously claimed that amps in clipping or with excessive output impedances sounded even similar. None that I know of believe that a 8W tube amp exhibiting output transformer distortions sounds like a 100W solid state amp, or that two solid state amps with different damping factors will sound the same into all different speakers. Everyone knows that weird loads affect different circuits in different ways. Everyone knows that it is possible to create artificial, low-level test signals that reveal converter differences—and so on, and so on. There are plenty of good reasons to choose one product over another.
Nevertheless, I repeat what I said: thousands of carefully conducted tests indicate that competently designed electronic components are most often close to indistinguishable when used within their proper operating range, even to the most critical ears. —unless, of course, they are intentionally designed to sound otherwise, as many expensive cables and interconnects are. But that is a coil of worms I will save for another day.
It is very clear to audio scientists that audiophiles and their opinion shapers in the press and in company marketing departments vastly overstate the audible differences between models and brands. And, what you believe is often what you hear. It’s a positive feedback loop that can reinforce the prevailing high end belief system and obscure the facts. So, who in the industry accepts this position?
Having been in this business for approaching forty years, I’ve known a lot of people at a lot of different companies. There are few important reviewers whose home systems I haven’t listened to at length and with whom I have not shared a few too many drinks. I am convinced that most high end reviewers are quite sincere, if not always technically rigorous or correct in all their assertions. It is likewise clear to me that some of the more experienced reviewers have at least an inkling that the emperor is missing some clothes, even as they may dissemble about this and that listening impression.
Manufacturers, however, I don’t consider as quite as kindly. I think many of the design engineers working at high end brands know full well that some of the stuff their marketing department spits out is simply not justified or correct. Indeed, some of those very marketing people themselves probably realize that they are saying things that they could never back up, either in the lab or the listening room. But, such is the game, and it is unlikely to end any time soon. Why should it? What high end brand would gain anything from such an admission? No, it is much easier to convince oneself that there is some esoteric detail about power supply noise or negative feedback or how capacitor dielectrics really work that invalidates all the objective testing results, as if we design engineers and audio scientists have somehow overlooked such things. (We haven’t.)
Of course, this is not a black and white issue. I am well aware that there are many high end beliefs that are absolutely correct. There are even several that science initially disbelieved, but which proved to be true over time. I know this; it is my job to know this. I’ve even taken on the technical establishment from time to time over matters once considered settled and done. Further, as a business person, I totally get it that customer satisfaction is what matters in the end, whatever the reason for that satisfaction. Still, truth matters, too. Customer satisfaction doesn’t change the fact some widely accepted high end beliefs simply contradict what objective testing shows. It is what it is.
Please, try to remember that behind the high end gurus and brand evangelists waxing rhapsodic in magazine interviews and marketing white papers, behind the star designers and the legendary inventors, there are hundreds of skilled engineers working to crank out quality designs month after month, year after year, with little room for error, cost/schedule inefficiency, or under-performance. There are a lot easier and more lucrative ways for an engineer to earn a living than spending the best years of one’s life chasing sonic nuances at the borderlines of human perception in order to sell a few thousand preamps or subwoofers.
By and large, audio design engineers, and audio research scientists, do this because we love music… we love listening to it, we love being around people who create and enjoy it, we love the quest for sonic perfection and we take our sound seriously. Seriously. We want the truth because we sincerely believe that seeking it is the best path to audio perfection. Don’t assume we are always wrong or that the prevailing wisdom is always right. And, when it comes down to just plain listening with your own ears, remember to grudgingly invoke the spirit of Ronald Reagan: “Trust, but verify!”
