Listening to improvisation affords me the occasion to observe my ear at work / play. Last night I went to hear a pianist who specializes in improvisatory performances of considerable duration.
Unlike other performances I had heard of this same artist, this one began with transparent pitch specificity. I do not have perfect pitch, but I do respond to the different resonant colors of black vs. white keys and I relate what I hear to my singing voice, so I really do know where I am when pitch is delivered with such clarity.
Once heard, the memory of these sensations is simply there to be referred to, contrasted with, changed -- the process doesn't matter. What matters is the hunger to return to something recalling that initial grounding.
In that observation various musical thinkers got it right - Schencker, Zuckerkandl, etc. -- though in no way does it require or assume formal structure in order to work. The ear simply wants it and that is quite enough.
I do agree with this performer that we have been over-structured in our awareness of what we hear so that it is harder and harder for us simply to open ourselves to the resonance of Mozart or Haydn or Beethoven composing for the new (i.e., no longer pure in the sense of vocal overtones) sound medium that is the piano.