Listening to your meters

May 22, 2016
 by Paul McGowan

We spend lots of money on measurement equipment. Hundreds of thousands of dollars in fact. These instruments help us ensure the products you get measure correctly and perform as they were designed. And these same measurement tools aid our engineers in developing new products, quantifying changes made in their design. Yet there is much we hear but cannot yet measure.

If you ever wanted proof of this, you'd get it on our forums. We've recently launched a beta test of the new DirectStream firmware called Torreys (named after one of Colorado's many 14,000 foot peaks). The praise for its improvement to sound has been near universal. Raves in fact. But we did discover some bugs.

The bugs were not hard to fix, required no changes to the basic architecture designer Smith had built, and we released a fixed version to a handful of users with a simple caveat. "Pay no attention to the sound quality, just let us know if the bugs are fixed."

They did, and the bugs were fixed. But they could not restrain themselves from commenting on the sound, which they were told to ignore, but hated none the less. Not one of them could listen for long and they all had the same comments: dull, flat, lifeless, etc. None of the testers know each other, they are scattered throughout the world, their equipment as varied as strangers in a crowd.

Whenever changes–even small ones–are made to the firmware we have to voice the DAC anew. Voicing does not involve changes to the firmware itself, we compile multiple versions and go through a day-long process of choosing the right sound.

Try and measure differences between versions and the meters just cough and die. There are no measurable differences. Yet the ears hear them easily.

Of course this means we're better measuring devices than our meters. It doesn't mean meters can never measure something.

This post isn't about meter bashing.

It's about levels of acuity.

Subscribe to Paul's Posts

32 comments on “Listening to your meters”

  1. I am totally convinced, Paul, that it should be possible to measure the changes in the DAC's output. Changes intended by the software modification the designer made.
    However you won't be able to predict the various interactions of the DAC with the specific type of amp which is to be fed. And you won't be able to predict the final sound quality highly depending on the design of the listening room and speakers. And of the sound recognition patterns of the individual listener of course! Finally experienced speaker designers always told me that most often speakers showing the worst measured data sounded better than near perfect speakers with rule flat frequency plots. This proves imho that the complexity of a stereo system yet cannot be simulated mathematically and the parameters most relevant for the perceived sound quality are not fully understood! If NASA know-how would be that incomplete no missile would ever reach it's target!

    1. Well, it would be nice, but despite all our fancy equipment, FFTs, jitter measurements, we cannot detect any difference. Either we're looking in the wrong place or the changes are below perceptibility for a machine - at least the machines we possess, and we possess the best our industry has to offer.

      Now, we're not NASA.

      1. Yes, either looking on the wrong place or not using the data correctly. With wine, it turns out there are specific molecules to look for, but it's the ratios that matter, more than the absolute levels. The human sensory system (including the easy or brains process the data) has structure and you have to discover how it works.

  2. Hi Paul
    What about to run an experiment? Announce Yale as a new update project.
    I'm curious how many will find out, that it sounds like Yale.
    That's the dillemma of emotion and conditioning. I'm sure that your advice, not to evaluate sound quality, has led to the opposite reaction, ie "there must have happened something in sound quality".
    Humans are strange in their perception. Me too.
    Best wishes from Germany.

    1. I love this idea, although Paul would suffer some credibility damage if he did do that. I was one of the first to report that I heard sound differences with firmware changes to the PerfectWave DAC and had to endure some criticism initially. A herd mentality does develop, I fear, and although most of us do not have proximate contact with each other, there is this WWW thing that makes our impressions immediately available. There certainly are differences in the sounds of the firmware versions and personally, I find them subtle. Many describe them as astounding, but that's just our way I guess. They have been improvements that many would spend money to have, I might add.
      One thing about the idea of listening that I don't understand is that when we listen for the same sort of variances in sound when mixing audio at the console there is never any criticism but when we do the same thing at home with our stereos there arises a chorus of incredulity. Go figure.

    2. Let me add a little historical note. The initial PWD firmware changes were for functionality only. I think that the first one we noted as sounding different was just an update for the front panel screen. After several new versions, some thought the sound was not as good as previous versions and many stayed with an earlier iteration. When this firmware thing started, PS Audio thought we were nuts (well, OK, mea culpa) but pursued the idea anyway. The current changes to the DS DAC are for both functionality and sound, so they have a very different focus. Paul's nerd army marches on...

    3. I disagree. While interesting in principal, and while many may be tricked, there's a dedicated group of our customers that would uncover it quickly, but would be able to accurately point out what was changed and how. I have seen it too many times.

      A few years ago we just did what you suggested. We presented bug fixes with different versions and did our best to convince people it wouldn't change the sound. They weren't pleased. For many, the changes were inaudible. For the discerning few, the changes were immediate, obvious and commented on without prompting.

  3. To piggy-back on Adi's thought . . . While the acuity of those measuring devices on the sides of our heads is excellent, that computer inside our heads has more than a few software bugs that hamper our ability to accurately interpret those measurements.

  4. Paul, I recognize and appreciate your dilemma - the measurements you are using provide you little useful information about sound quality. With all due respect, sincerely, your engineers must be measuring the wrong things, or at least not a full set of the right things. Measuring wine quality is an extraordinarily controversial thing. Yet some winemakers (sensory, chemical, and economic data scientists, really) after decades of research, have found techniques that work. They have become practical and reasonably inexpensive to do. It's complex and not widely accepted. Part of the answer is discovering the hidden segmentation of consumer preference. Perhaps if you put some effort into finding those secret rubrics for musical audio quality you will have a tremendous competitive advantage and speed your development process.

    1. MediumRare, while that is an admirable goal, I think it is virtually impossible to accomplish in this field. Different listeners will focus on, hear, and prefer different things. The same listener will have different preferences on different days or with different types of music. One system and room will reproduce the same audio signal differently than another system in a different room. Not to mention the difficulty in finding ways to accurately and objectively measure things that audio enthusiasts listen for, like soundstage width and depth, timbral accuracy, imaging, pace, rhythm and timing, etc. etc. In short, there are just too many variables presented by the end user to come up with a system of measurement that will guide engineering decisions and result in meaningful improvement of sound for everyone.

      1. I couldn't agree more, but we're not that big of a company. We have 33 employees–big by high end audio standards, small compared to the task– and of those only a handful in engineering. They're busier than one legged men at an ass kicking contest already. While we might be able to spend quality time figuring it out, the problem would be everything else would collapse.

        Wish we had the resources.

        1. I bet you have the data already but haven't used it. Maybe a grad student would be interested to give it a shot as a research project?

  5. If voicing doesn't involve changes to the firmware, than what is it that you are changing / doing when you are voicing the dac?

    On a far less serious note, Adi's suggestion of releasing Yale as a new release (let's call it "Maroon") appeals to my sense of humour. Release Maroon with great fanfare and see how many people wax eloquent about all of the great improvements of Maroon over Yale. Maybe something to consider for next April 1.

    1. MurrayB said: "On a far less serious note, Adi’s suggestion of releasing Yale as a new release (let’s call it “Maroon”) appeals to my sense of humour. Release Maroon with great fanfare and see how many people wax eloquent about all of the great improvements of Maroon over Yale. Maybe something to consider for next April 1."

      The thing is, even without performing this little experiment, we all know that this is exactly what would happen. The power of suggestion is strong and our minds are completely susceptible to it.

    2. If you have the time, head to the forums and read up on ted Smith's comments. He explains in painful detail what is changing and why. NASA even understands what we go through, as does the designers of the Xilinx FPGA set. Understanding the issues and solving them are too different things.

  6. I have a theory about why these measurements fail: we are typically trying to compare two absolute measurements rather perform a single differential measurement.

    Consider determining the extraordinarily small difference between two 10 VDC reference standards, which is typically on the order of single-digit parts per million. One could obviously measure each one with a long-scale meter and calculate the difference. But, it would be much better to use a null meter, which essentially acts as a voltage balance. The error of the measurements made by the long-scale DMM are going to be proportional to the measurement itself, roughly 10 VDC. But the error of the measurement made by the null meter will be proportional to just the difference between the two voltages, which is many orders of magnitude smaller than the nominal voltage.

    So, to compare two DirectStream units running different firmware, we could synchronize them to a master clock and have them play a test track; set one unit to IN phase and the other to OUT and then sum their outputs. Since their outputs would largely cancel, this would leave just the difference to be measured. We could then set our oscilloscope or signal analyzer to a much smaller voltage range, affording orders of magnitude better resolution.

    Has anyone tried anything like this?

    1. Great idea! However finally a model or a theory is lacking that would transfer the differences measured to differences in perceived "sound quality"! The funny thing is our ears can even detect the changes in mains quality varying during 25 hours. And even more incomprehensible: a speaker with non-flat frequency response can sound much better than another speaker showing heavy peaks and dips. Thus at least you should apply your differential measurement comparing the mastered output with the input received by a microphone at ear level! 🙂

      1. I gave this idea more thought after I posted it and understand that it may be difficult to correlate the measurment results to software / hardware changes or actual sonics qualities. But, I figured it could at least be a starting point. From what I understand, the current issue is that literally no measurements can be had that tell us anything- if there is any relevant information, it buried below the limits of truncation or sensitivity.

        Another flaw in my theory that occurred to me is the Directstream systems may have variance among the population. So, before comparing two different versions of firmware on two different Directstream units, we probably should first compare the two playing the same firmware. If this measurement is indeed ultra-sensitive, it may actually measure the physical differences between the two players themselves. But, hopefully the variation among the unique components comprising the two systems would average out, making them effectively identical.

        1. I am confident you are on the right track. Remember too, that perception is personal, yet there are likely groups of people with similar preferences. Unlikely for one product to please them all, but it must please one group (segment) extremely well.

          1. Agreed. I see these yet-to-be-developed measurement techniques as just an improved tool. But the real question of how to best voice the product will always remain. I think the best-case scenario we could ever hope for is that we identify a set of measurement results that closely correlate to various sonic characteristics and then leave it in the hands of the product designers to decided how to best tweak the system.

  7. I believe that, in general, if only you knew what to measure - and could afford the equipment - you would be able to measure it.

  8. So true. Measurements do not indicate the quality of sound always and in many cases,after a point, not at all. The human ear is exceedingly more sensitive. There are things we cannot measure. I am sure there things about sound we do not even know exist. Yet the human mind is a wonderful thing. There are people who are convinced they can hear numbers. Maybe they can. After all we all do not hear the same. Remember the magazine called Stereo Review ? Gosh did I swallow the number game hook, line and sinker at one point. The amount of money I could have saved not upgrading equipment to get my system to sound musical. Now when I think of it I wonder how naive I was for not listening to what my ears were trying to tell me. So the lessen learned is, always listen to your ears. Regards.

  9. Is there any equipment that can measure phase shift across a range of frequencies?
    At any point in time, music contains a complex mix of frequencies, even if it is a solo instrument. In junior high band, I learned that the thing that makes each instrument sound unique is a series of overtones that differ in frequency from the fundamental tone and the relative amplitude of those overtone frequencies. The band director told us that the local university had equipment that could take a recording of one instrument and make it sound like another instrument by changing the set of overtones relative to the fundamental frequency.
    So, I would think that if the phase (arrival time, not polarity) of different frequencies across the spectrum is not the same (all overtones do not arrive at the output at the same time as the fundamental), then that could account for a difference in the sound that would not show up by measured frequency response (amplitude) alone. So, a frequency response curve (amplitude) of two different sources that is identical, might not necessarily sound the same if the phase shift across frequencies is different.
    I have never read a study on this, but it is something that I have often wondered about when it comes to discussions about two different pieces of equipment measure the same.
    From the Belden Iconoclast cable patent there is this statement.
    As with self inductance or impedance effects, propagation velocity is similarly frequency dependent and, with wide differences between arrival times of low frequency components and high frequency components of an audio signal, can result in audible phase distortion and "smearing".

  10. What is the end goal? Subjective, individual, variable. What are the system variables from microphone to room acoustics? Variable and infinite combinations. What are the prospects that perfecting each element in a playback system independently of each other will end in satisfactory results? Zero. You cannot understand a forest by just understanding individual trees. This is especially true when there are many elements in the forest you have no control over like what recording process was used, what are the playback room acoustics. You can't even be sure that the tree you just planted successfully in isolation will thrive in the milieu of the forest. This is the result of naive intuitive thinking that looks only at the individual trees but doesn't understand the forest. This is what systems engineering does not do. If it did, it would always fail. The mathematical modeling called the overall transfer function is not up to it. The measurements based on the simplistic approach don't work. The proof is as easy as the power output meters on a amplifier swinging back and forth wildly telling you hundreds of watts are being delivered to a load when there are no speakers connected and no watts are being delivered. They don't measure power at all, they measure voltage but are calibrated into a presumed 8 ohm load. What exactly are you measuring?

    I feel like the kid in the movie "The Shining" who led Jack Nicholson into the maze in the snow chasing him trying to kill him. The kid knew the maze and how to get out. Nicholson's character didn't. Presumably he died in that maze lost in the freezing cold.

    Hint, the maze is about sound fields and hearing. each element in the maze, each path down each branch point takes you somewhere else. There are mazes within mazes within mazes. What is the alternative audiophiles and those who create products for them use instead of competent systems engineering? Trial and error. Try everything in every combination hoping one day you'll find your way out of the maze. And every day someone invents a new branch down a new path.

    1. You may not understand the forest by looking at trees, but a true arboreal scientist might be able to. And he certainly can't know the forest without knowing the trees. As an engineer I'm sure you will agree after giving it more thought. As long as this discussion is about opinions it will dance around the truth. The truth is in the numbers.

      1. "The truth is in the numbers." As one who is, I would assume, a Man of Science, one should really question that, as it is not terribly different from saying, "the truth is in the colors" or "the sounds", etc. If you think that numbers arrived at by Science over time remain absolute, you should perhaps have another think.

  11. The ears measure, and "praise" what sounds more "real", and this is an intricate process within the brain - a field of research called Auditory Scene Analysis deals with the fact that the mind is far cleverer than mere blind instrument measuring would indicate - where the cues and clues in the recording are more usefully presented is what counts - but the audio industry hasn't taken this on board so far. So, just about everyone is wandering blind, blundering into obstacles left, right and centre - trying to discover "good sound", which they know must be out there, somewhere ...

    1. I'd argue that most prefer what sounds "best" to them. It would seem that it's scientists who are concerned more with the evidence of numbers than the evidence of their own senses and tastes. I mean, we're not talking about stuff that affects world politics here.

      1. Where the scientists get it 'wrong' is that they haven't correlated what people prefer, to what the numbers say. Because they can't easily measure what it is that people are hearing in playback they like more, they frequently come up with lame excuses, pointing the finger at the listener himself as something "to blame" - ego wars say this is the easy way to win the battle, and they're content with that ...

        1. For I'm also a scientist I'd like to link both worlds of senses and numbers. Both are missing a crucial property.
          Senses are missing objektive reproduction while numbers or measures are missing sensitivity. The dillemma is that senses are much more complex and sensitive than measurement equipment and messurement equipment is more or less repeatable. But both may be correct within its limitations. That's the challenge to mention the outcome of any finding either as sensual or number finding together with the scale of confidence.
          I agree that hearing beats measuring. However we should give the sense (ears) a try to get it as objective as possible because then it will be masterpiece.
          We should be brave to have double blind experiments. It's the only way of getting truthful results.
          Best wishes from Germany

  12. The problem is that machines are far better at measuring steady state signals, and de-compose the signal into time quanta and level quanta in linear Cartesian fashion. Perception is far better at decoding complexity like music and its reverberations in a room in the analog domain.

    Think about how MLS signals work so you can separate direct signal from echoes with an omni-directional microphone. The waveform mathematically designed to have a distinct signature of all frequencies in phased sequences so they can be recognized when they are swamped by echoes. The same principle is used for space communications when the signal can be 90dB BELOW the noise floor.

    Human hearing can utilize any and all direct wavefronts as a signature, looking for reflections at lower levels according to the inverse square law because the pressure wavefront spreads with time. The comparison also incorporates all possible frequency dependent phase shifts which are caused by the direction of arrival and the shape of the ears.

    Because the brain is looking for correlated signals, they can be detected well below the noise floor. The processing is needed to produce useful information about the acoustics of the surroundings which is useful for survival. This requires phase discrimination on the order of the smallest features of the pinnae and left-right time differences of one degree, so it is done to a precision of a couple of microseconds.

    In conversion from analog to digital and back, filters are employed that shift phase well over these thresholds. In a null test, you would have to look for correlated signal echo differences 20dB or more below the noise floor.

Leave a Reply

Stop by for a tour:
Mon-Fri, 8:30am-5pm MST

4865 Sterling Dr.
Boulder, CO 80301

Join the hi-fi family

Stop by for a tour:
4865 Sterling Dr.
Boulder, CO 80301

Join the hi-fi family

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram