Absence of evidence

February 9, 2022
 by Paul McGowan

Here is a wonderful aphorism I am fond of:

“Absence of evidence is not evidence of absence.”

This can have multiple meanings but the one that rings true for me concerns our age old debate about measurements and the fact that not everyone hears the same things.

If you don’t hear any difference between two products we can draw multiple conclusions. Among the many are:

  1. There are no differences
  2. Your system cannot resolve those differences
  3. You’re not listening to program material that contains musical information where those differences are most pronounced.
  4. You’re not an experienced listener.

That you hear no difference is not definitive proof it does not exist. It is merely an observation that in this particular set of circumstances you do not register a difference.

The same holds true for measurements. If we’re only looking for a fixed set of variables (FR,THD,IM,N) the fact we missed a circuit’s tendency to ring when stimulated by a specific square wave does not mean it didn’t ring.

It merely meant we missed it.

There’s nothing wrong with a good measure of hubris unless it is so strong it blinds us.

Subscribe to Paul's Posts

44 comments on “Absence of evidence”

  1. It really depends on how you go about buying hifi and whether you have the patience to compare products side by side, at the same time, and not one after another. I once did this, with four comparable products lined up on the floor at home and plugged and unplugged frequently. Really boring and frustrating.Over the years I’ve got an appreciation how certain types of products and brands perform, and what I want, so I’ve chosen products without much in the way of comparison. I think the problem is that many products will sound the same and you can drive yourself mad not hearing any difference.

  2. I have always maintained, & still do, that measurements, per se, are important in the design, prototyping & manufacture of home-audio gear & loudspeakers; along with extensive listening sessions by at least 6 trained & critical listeners…& including some of the work done by Toole & Olive…only some of their work…not all.
    However, once said gear & loudspeakers are sent completed & packaged from the factory to the warehouse, dealer or buyer’s home (listening room) it’s too late to really concern yourself with measurements, unless the item is not performing within specification.

    This description of a high-end home-audio rig is very enlightening.
    Note that he has separate & external power supplies for the electronics, 11 pieces in all, & that he employs a 4-way, 2 channel active crossover that is wired to 4 different amplifiers that have 4 separate & individual sections of the entire frequency range (20Hz – 25kHz) to deal with, this application keeping distortion as low as is possible…this is a serious high-end rig!

    https://www.youtube.com/watch?v=n5UxESCVdQ4

    We often discuss here about a high-end home-audio rig that will create a ‘live’ soundstage & that we can listen to for hours on end without any fatigue…well it’s often not quite as simple as plugging a couple of 3-way floorstanders into a couple of power amps.

  3. I’m reading the post today and all is going along swimmingly until the last line. Then I become stuck. Specifically on the word hubris. Which definition do I use? Arrogance or self confidence and pride?

    One is so confident, they are right. The other is so confident, you are wrong.

    In the mean time concerning audio, all I can do is enjoy what I have, and strive for better sounding to me within the budget and other confines I set. Occasionally smile, (when an extremely small amount of hubris wells up inside), that I’ve been able to get to the point I’m at now.

    1. I think Paul’s reference to hubris was a mea culpa on his part. It seems to have no relevance to the rest of the post. I can’t image Paul flagellating himself (in the Catholic sense) over some of his hubristic marketing copy, which is noticeably bereft of measurements.

      You can’t be hubristic when enjoying your hifi system. There is no limit to how much pride one can take in such baubles, and **** what anyone else thinks.

        1. What are you talking about?

          It is YOU Paul that has absence of evidence. You “claim” this magic hearing and physiologically it is highly unlikely (You are past 70, aren’t you?). You complain that DB studies are stressful as an excuse for not doing it.

          If you are so convinced of the superiority of your hearing or your equipment then it should be easy for you to demonstrate that “blindly”.

          If YOU think that something is missing, then say specifically what it is and then demonstrate it. How do you think that they figured out that the sun and the moon didn’t rotate around the world. Everyone knew this! Everyone could see it with their own eyes! And the moon even changes have, darned it.

          1. I love it.

            Stereophile’s Jim Austin writes: “There are as many opinions as there are experts.”

            I have tried to help you understand that when we do these tests they are indeed blind. Somehow that escapes you.

            In any case, I think it’s no longer of value to our community for you and me to bicker about this.

            Please feel free to continue sharing your thoughts. I will, for now, be in the background reading them. Thanks.

      1. You’re so much better at reading between the lines than I Steven. 😉

        The only context I could find was ‘using a good measure of’ 😀 – and of course that becomes an individual ‘standard’

        Oops – I hit reply and Paul’s explanation showed up. Mystery solved.

  4. A fifth possible reason for being unable to hear a difference is hearing loss. It is well known that the audiophile community consists of many older persons, myself included. I wonder if the brightness of many systems and particularly of loudspeakers now being made is a consequence of the inevitable loss of high frequency acuity that comes with age. Manufacturers are simply catering to the needs and deficiencies of their customers.

  5. I doubt anyone who has heard the difference of utilizing a quality film capacitor in the signal chain in comparison to an electrolytic, even though rated at the same voltage and measuring the same capacitance, would discount the benefits of using the film rather than the electrolytic from a sonic perspective.

    So what does the measurement of it’s capacitance rating tell us about it’s sonic difference in materials utilized? Not much. Which means that while the capacitance value gives a good benchmark to what frequencies can pass, it does very little to expound on it’s attack and release characteristics.

    Point being, you may have to realize that you are using the wrong measurement to evaluate your desired result.

      1. “That you hear no difference is not definitive proof it does not exist. It is merely an observation that in this particular set of circumstances you do not register a difference.” – Paul McGowan

        1. Hence the requirement for at least half a dozen critical listeners when finalising the development of a loudspeaker or other piece of home-audio electronics.

    1. In the comparison of film vs aluminum electronic, which sonicly is detectable even to the untrained ear, there is more to the measurement than just capacitance and voltage, like ESR and dielectric absorption. When more measurements are taken into account the ‘black magic’ goes away.

  6. This is something i agree with totally. Measurements are the basic’s off course. A place from where we can start in our journey. After that, we also have to rely on trusting our ears…

    1. That is absolutely a valid point. But even more so, is that a person’s preference also plays a vital role in the subjective results. And this is where the clarity of the water, (the results) gets real muddy, real quick.

  7. I think that comparing audio gear by datasheet is very dangerous, in this case none would buy tube amplifiers nor expensive high end audio gear at all. Generally the official data provided by manufacturers are just a rough clue for orientation. Which maker publishes waterfall diagrams with their loudspeaker for example ?
    Decades ago I went to an audio lab to measure my home brewed power amp with their priceless equipment. In the result I got >> 19 pages << of data including even the dirt that my amp leaves with different loads on the power line. Then a specialist similar to Paul's Mr. Stroebel gave me precious ideas how to improve the design with a few hacks.
    Compared by common given data of a $ 600 firecracker this amplifier will lose in many aspects, that's for sure.

    To find good material I can only repeat similar to Paul's suggestions:
    1. Make a CD with a wide span of diversity of your preferred music.
    2. Go to different dealers and test the gear under equal conditions, take your time and don't let the salesman talk to much !.
    3. Test the equipment at home with a little intention not to buy it.
    4. If you are convinced that the new part is a "must have" then buy it.

  8. ZTPIII and m3 have already stated the “5th Element”…Your -Hearing- (huh)! All audiophiles (and everyone) have unique auditory physiology that impacts how we interpret sounds. Even if sensitivity and frequency response are equal between individuals, it certainly could be that we hear/interpret sounds differently!

    If Paul is including “listening room” in #2 System, great! If not, then add a 6th element…your listening environment!

  9. “Absence of evidence is not evidence of absence.”

    Sadly, based on the current prevalence of science denial, and rising political tribalism in the world, in 2022:

    “Evidence of evidence is not evidence.”

  10. I always have a problem with the fourth conclusion. In my opinion, not being an “experienced listener” does not mean you don’t hear the difference. It just means you may not recognize or be able to articulate the difference. You may only be able to say, “For some reason I like that sound better.”

  11. “Absence of evidence is not evidence of absence.”

    On a (way, way outside) tangent, this reminds me of a lawyer in these parts whose road signs carry his motto: “Just because you did it doesn’t mean you’re guilty”. LOL

  12. When I comment here I often close my comment with he following: ” As always in audio YMMV ( your mileage may vary ). That is a corollary to Paul’s post today. There are people in audio who are entrenched in their beliefs and are not going to change their beliefs no matter how much you try to change them. There are people in audio who are open to new ideas and new things and willing to try them. There are people in audio who believe only in measurements and people in audio who do not believe in measurements at all. There are people in audio who understand measurements and use them for what they are and realize what the limits of measurements are. There are people in audio who are willing to critically listen to different gear and do shoot-outs between gear to determine which sounds best to them and there are people in audio who really on dealers and perhaps other people to choose the gear for them. There are old people in audio who do not hear so well and there old people in audio who still do hear very well. And the same can be said for young people in audio.

    Audio is what I call a free style event. There is no right or wrong and the rules are very flexible. The only thing I think we should all do is enjoy the music as much as possible.

    1. Oh my gawd – audio rules are like ART!
      Some folk like a precise painted replication that looks identical to the actual scene, some prefer a distorted barely recognizable side-slanted interpretation, and some will duct tape a banana to their integrated amplifier and proclaim to the world it is worth $120,000…
      Me? When I see a ship at sea or an old barn in a field, I’m impressed if have to really observe closely and ask myself – Is that a photo or a painting? A blue circle with a red splatter on a white canvas? It’s HOW MUCH? Ah, I see…it’s a Bose. (Pronounced Bosee (accent aigu) – as in overpriced ‘Rose’ wine)
      I know – it’s hard for me to get thru the day without a badly butchered metaphor… Or perhaps I’m day drinking… you really don’t know – Do you?
      But get it because YOU like it. Not because it’s worth $XXX. And SOME rules do apply; I’ve seen too many online photos of Richie Rich’s soundroom with the absolute top-of-the-line dipoles on either side of the 22″ cabinet and all crammed RIGHT up against the wall with the caption “Sounds awesome!”
      Uh…ya, I can tell from here – no it DOESN’T… Here’s your banana… Get your own duct tape.

  13. Indeed, I could write a book along these lines. Years ago, I took classes in ear training (yes, it’s a thing). After much practice, I’ll never forget when a live symphony orchestra suddenly resolved itself from a homogeneous wash of sound into distinguishable instruments. Exciting as that newfound skill was, it had its downside – now I could hear individual bungles, and nearly every concert has some. It took several more years before I progressed from celebrating any high-frequency extension in audio playback to noticing whether those highs actually correlated with the original program material.

    It’s unsurprising to me, therefore, that many folks – even within the professional audio community – can’t hear certain details in the music or in the playback chain. I’ve seen audiophile salesmen demonstrate amps with horrible levels of glaze and grunge, or a severely worn LP that made the Philadelphia Orchestra strings sound like sheer fuzz, while proudly proclaiming “doesn’t it sound great!”

    Worst of all, though, is when someone tries to promulgate ignorance in the guise of authoritative expertise, as when engineers in the mid-1980s accused listeners of being delusional because we could hear the problems caused by jitter before the engineers had figured it out. Or the author of a physics book who devised home experiments designed to “prove” that people can’t hear phase differences, complete with snide comments like “you would need to have a physical connection between your ears – essentially a hole in your head.” FYI, the only valid such experiment in that book – taking two tuning forks of the same nominal pitch and listening for the beat frequency caused by production tolerances, one fork at each ear – produced about a 15-degree oscillation of the stereo image for me, without any amplitude variation. I’m still enjoying that “hole in my head.”

  14. Hello Tiburd,
    I am fortunate to be able to walk about 6 minutes from our front door to our seats at The Kimmel Center for a Philadelphia Orchestra concert. Depending on the aural and visual perspective one prefers, most of the hall (by Rienzo Piano) sounds glorious for a given location.

    If you wish to replicate that fuzz experiment for real, I have discovered one section of about 16 seats that makes the Philadelphia sound like a fuzzy Bose’ table radio- live! Both enlightening and excremental. Hint: They are not the cheapest seats.

    Stay Healthy.
    – Jeffrey

  15. So my shoe size is UK 9. If I go into a shoe shop, I would not expect every pair of size 9 shoes to fit properly. Shoes used to have width measurements (I’m a 9F), but they no longer bother. Suits and audio are the same. A few basic measurements to get started, the rest is fit and styling and sometimes bespoke.

Leave a Reply

© 2022 PS Audio, Inc.

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram