Can’t vs. has

October 20, 2021
 by Paul McGowan

If you line up three blindfolded food tasters and rate their responses on the difference in taste between arugula and parsley, would it make sense to proclaim there are no taste differences if that should be their conclusion?

More to the point, if in blind testing on someone’s HiFi system there are no differences heard between cables or equipment, do we simply then state there are no differences to be had?

When does it make sense to first set up a system that testers do hear differences and then begin testing it on differing levels of listeners?

If you want to discover how many people can tell the difference between something you first need to make sure the experiment is set up in such a way that at a minimum the proponents of the experiment agree it’s working.

If we want to learn the truth about falling trees and the sounds they do or do not make, it’s probably best to set up the experiment in a forest of trees.

Subscribe to Paul's Posts

33 comments on “Can’t vs. has”

  1. It is very easy to set up blind A/B tests. The presumption is not that skilled listeners will hear differences, but that the number of listeners that do say they hear a difference is statistically significant. This is because some people listening to A/A and B/B will say they hear a difference. You need A/B and B/A to have statistically significant differences.

    I suspect this is rarely done in the trade, and when done externally is ignored, but would rather rely on a review, which is a biased sample of 1. Anyone who’s done Statistics 101 wouldn’t give a biased sample of 1 the time of day.

  2. Or in a garden full of tomato vines with an electrified (bear-proof) fence 😉
    I don’t care what three blind mice can or can’t hear…or taste for that matter.
    All that matters to me, the listener, is what I can hear & whether it will keep me
    from wanting to get up & go to the toilet when that second glass of red wine is empty.

  3. I’d never heard of arugula, until wiki told me it’s what over here we call rocket. It’s a staple ingredient in the summer, if you can’t tell the peppery difference between rocket and parsley you need your ears tested.

    I was reminded of a foodie example that sometimes substituting a cheaper product has its benefits. Audiophiles go on about different sounding capacitors. Napoleon could not source enough chocolate to keep his troops happy in Piedmonte in the early 19th century, so someone had the bright idea of mixing in hazelnuts to bulk it up. Thus was born Gianduja, one of the great joys of life. The British can of course claim credit because it arose from a blockade of French ports. As they say, necessity is the mother of invention.

  4. Let’s imagine a forum, where Formula 1 race car drivers down to race kart drivers, down to normal people with BMW’s or Toyota’s participate in common. Let’s further imagine, the topic “influence of rear spoilers in connection with roadholding” comes up.

    I’d say most folks below a Porsche or similar with a large tail plane are lucky not to have the problem and don’t join the discussion or just read and ask questions for pure interest.

    The audio forum behavior would probably be, that a bunch of family car drivers would first doubt that rear spoilers have any influence, exist at all or ask for valid measurements of their efficiency. If they tried one on their city car and it had no effect, they’d transfer this 1:1 on Formula 1 cars and claim, they must have no effect generally. In the following they’d declare manufacturers of rear spoilers as voodoo preaching, selfish profiteers and their customers as money wasting idiots. Subsequently they’d join multiple race car forums to tell the surprised members how meaningless rear spoilers are and that at least a double blind driving test using ordinary cars must be initiated. 😉

    1. I’ve never been in a racing car, but I know spoilers work, because I’ve been in an aeroplane.

      A spoiler is an aeroplane wing upside down that creates a force downwards, increases friction, increases grip. It is the opposite of a wing on an aeroplane that creates a force upwards and keeps you and 300 other people and their luggage in the air for 6 hours. Friction? Try walking on ice. These are things that most people can understand because they have experienced them, even if not in a racing car, and you could probably explain an aerofoil to a 4-year-old, without having to go into differences in air pressure.

      If you did an A/B test of the effectiveness of an aerofoil by putting 300 people in an Airbus and flying it at (a) 500mph and then (b) 50mph, I think they’d notice the difference.

      The other difference is that if you fit two different spoilers to a racing car, the difference is likely to be evident in different lap times. If the lap times are the same then the conclusion is that the difference, if there is any, has no practical effect. This is the opposite of what Paul often says that everything makes a difference, even if you can’t measure it.

      Just as explaining differences in air pressure over a wing may be the point at which many people would switch off, I have the same reaction with electronics explanations.

    2. I’m glad we we’re imagining in this example that you make. I find it highly doubtful that anyone who follows Formula One racing would ever compare a rear spoiler’s ability to stabilize their personal vehicle with an adjustable spoiler on a Formula One monster. I don’t think that you’d find one percent of this population would even bring the subject up.

      You gave a good example though in your post since we’re imagining.

  5. Any test that tests a hypothesis proves or does not prove this hypothesis at a given level of probability. And it is important to understand what the wording of the test result means. I guess that 95% (or more) of the population is unable to understand the exact formulation of the test result. When these 95% start discussing later, it’s to cry or laugh …
    In the end I agree with Fat Rat – I belive my ears, no matter the drink … 😀

  6. “When does it make sense to first set up a system that testers do hear differences and then begin testing it on differing levels of listeners?”

    To answer the question proposed today. It only makes sense from a marketing point IMO. Most reviews are only done with 1 person or so and with non standardized systems. Those 1 person results are what many use as a guide in making choices. It would seem to me that if things are to be more objective than standards need to be set. Once a ‘standardized’ system and room are defined then testing can begin. PSA (and other manufacturers) may be the closest to achieving that goal. Take MR2, when new equipment needs to be evaluated it’s swapped in and everything else from power sources and cables stays the same. I assume you (PSA) don’t happen to have a gain cell Dac in the chain and then decide to swap in a new preamp to determine its sonic viability. You go back to the original reference set up.

    Even in the case of information dissemination from Paul we only have a reference of one.

    So we’re back to the individual. Their individual ears, the individual room, and their individual set-up.
    If that is the case it’s a solo hobby With the occasional influence / demands of others having an effect on the choices. In the end it doesn’t matter if 99 out of a 100 people like something better if you’re the one who doesn’t like that something. ✌️

      1. I agree, and this is why I want every audio magazine ( print or online ) to require their reviewers to list their home audio system in detail including the listening room dimensions.

  7. Why oh why are we discussing A/B testing? There are audio forums out there that have band discussion of A/B testing. It is impossible to get consensus on the value of A/B testing with respect to hearing a difference in audio. It is like debating if a cables sound the same, if all power amps that measure the same sound the same, if the 0’s and 1’s are the same than digital audio sounds the same. People have fixed opinions on this stuff and they will not change their minds on it.

    Here is an example of how some people do not understand how to use information that is available to them. Steven, I am going to disagree with you again, it is not personal. You stated above that a review is all but worthless because it is a sample of one. A few years ago I discovered that two of the pieces of gear in my audio system are also in the home system of one of the reviewers at TAS ( The Absolute Sound ). I dug out my back issues of TAS and read the reviews that this reviewer had done on those two pieces of gear several years ago. I was impressed that his reviews were almost exactly how I would have described the gear if I wrote the review. Now last year I was lucky enough to be chosen to be a beta tester fro the new PS Audio Perfectwave SACD Transport ( PST ). I did an A/B test between the PST and to older PS Audio DMP ( Directstream Memory Player ) transport that I was currently using. I had my wife ( a totally unbiased participant ) also do the A/B test. It took both of us about five seconds of listening to realize that the PST sound noticeably better than the older DMP. I wrote a very positive review of the PST and you can find it on the PS Audio Forums in a thread about the PST. About 4 or 5 months later the TAS reviewer mentioned above gave the PST a Golden Ear Award.

    So, if I am in the market for a new piece of gear and I discover that the TAS reviewer that I have agreed with three times has reviewed that new gear do you think I should follow his advice?

    1. Hi Tony,
      I’ve always found that the units that are within my price range that get a great review go on the short-list for me to audition & then I can make my final decision on what to purchase.

    2. Tony, I read as many reviews as possible on a particular component that I may want to purchase and look for the similarities and the differences in these reviews if there are any. Then, if I can’t find an audio salon to have an extended listen I’ll buy a 30 day moneyback guarantee item and move forward because most of the reviewers I trust are in line with my listening experiences after I make the purchase and the unit burns in.

    3. Hi Tony,

      There was a time when I was in deeper where I finally was able to read what a certain reviewer wrote and when I auditioned for my self I could hear what was described. So I fully understand what you’re saying.

      Those days are not so prominent anymore for me.

      As far as A/B in audio discussions it’s sometimes it’s like Twix. Right side / left side, but turn it around and the sides change. (In name only)

    4. I, with you on that, Tony. I’m happy to be a biased sample of 1 with regard to my own audio system. I found that Alan Sircon raved lyrical about the products I like, although I did not check if he raved equally lyrically about other things.

      Using my non-statistical biased sampling technique (= listening to things) works for me. The extent of my research is usually embarrassingly bad. I bought Shunyata without reading anything and assumed they came from Japan. It transpired they come from the USA and Shunyata is a Buddhist word for nothingness, which is not something that applied to the price.

      I’ve mentioned before I read reviews to keep up with what’s about, what things do and how they do it. I lose interest when the reviewer starts listing their record collection and I certainly don’t hang around for their opinion.

  8. I read an article in Stereo Review, in August of 1998.
    The article was in titled, “Ggeak Or Tweak.”
    What they had and done, was sat up two diffrent stereo systems.
    But they only used one pare of speakers.
    One of the sistems, used a Sunfire Classic Tube preamp that was left on, for a week.
    The other system, was more then 20 years old.
    The only part that could be seen, were the speakers.
    Both of the systems, were hidden behind a kirtan.
    It was about 300 people that participated in the blind listening test.
    And this blind listening test, went on for hours.
    But at the end of it, it was 70% of the people that identified the ggeak system, and 30% of the people that identified the tweak system.
    The speakers that were used, were Peridine speakers that costed about 20,000USD.
    Go figure this one!

    1. JP, This is not surprising at all. If there is one thing we should all know about audio is that everyone’s preference for how things sound is different. There are people who love the sound of tube amps and there are people who love the sound of solid state amps. I have compared tube amps to one another and all tube amps do not sound the same. I personally think that speakers determine the sound of your system more than anything else, however, that does not mean that everything upstream of the speakers sounds the same. If that were the case then everything but speakers would become very cheap to buy because people would buy on price only because everything sounds the same. In fact, people who think to all audio gear sounds the same tend to have very cheap audio systems since all the care about is to the hear the music. My mother spent her entire life listening to music on an AM radio that she had in the kitchen. She heard the music and that was all she wanted.

      1. Good morning Tony!
        This kind of reminds me of what happened when I was 16.
        I had just bought myself a really nice spatial house to live in.
        And everything I had at my mom’s and dad’s house, all got put in to storage.
        But I was without a stereo system to listen to my music on for awhile.
        And so, one Saturday afternoon, my hole family got in to my car with me, to take a drive out to Walmart.
        Everybody needed to get something from that department store.
        I had a funny feeling that, I would have to take everybody back home, and drive back out to Walmart by myself.
        But before that happened, my sister and I were checking out what Walmart had to offer in the way of stereo systems.
        I was looking at a Pioneer system that had a $500 price tag on it.
        But my sister was trying to push me towards a very cheep stereo system, that was made by Sound Design.
        If you know anything about that company, then you automatically know that any of the stuff they made, was no good.
        My sister didn’t understand that at all.
        But what she said to me was, “a stereo is a stereo.”
        But me knowing my own ears, I would have been much happier with the Pioneer then the Sound Design.
        It was the high quality sound is what I was looking for.
        And also, if I had to spend a hole entire year listening to it, I wanted my system to be one that I could rely on.
        Those Sound Design systems, would brake on you, in 6 months time.
        So, ya, I agree with you, hole heartedly!
        Not every speaker pare, sounds exactly the same.
        A tube amp, is not gonna sound like a transistor amp.
        And a 6inch woofer, will not give you the same sound, that you can get from either a 15 or 18inch woofer.
        So, yes, you’re correct about everything man!

  9. I keep remembering that certain manufacturer’s audio show demonstration where the guy starts with their cheapest cable and ends with with their most expensive. Each time that he climbs the cost ladder he tells the assembled listeners which cable they will hear next and that they will hear a dramatic improvement. After each A-B he asks for a show of hands which cable sounded better. Hands fly up for the most expensive cables–except mine. In that demonstration I hear absolutely NO difference between the middle line and the highest-end model.

    I do hear distinct differences between different cables in my home system. The difference is typically in the weights of the bass and treble relative to the midrange, the degree of sibilance, the thickness of the sound, holography and the quality of reverb tails. The duration, depth and clarity of the reverb tails is an easily heard characteristic of the best cables and power conditioners.

  10. I guess this post of Paul’s can be a highly relatable set up for the existence of “cable burn in.” 😉
    I mean how else are you gonna do it.

  11. “If we want to learn the truth about falling trees and the sounds they do or do not make…” There is no question that the sound is produced, whether you are there to listen or not. That’s a given. We are not speaking of quantum theory.

    The question is not “does cable A sound different than cable B.” That’s a given. The real question is what changes actually happen within the synergy of the system’s response, and what is actually responsible. You muddy the water with the “observer effect.”

Leave a Reply

Stop by for a tour:
Mon-Fri, 8:30am-5pm MST

4865 Sterling Dr.
Boulder, CO 80301

Join the hi-fi family

Stop by for a tour:
4865 Sterling Dr.
Boulder, CO 80301

Join the hi-fi family

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram