Just discovered that there is an accompanying app to Soundflower called Soundflowerbed which allows you to ’listen in’ to whatever gets put in to Soundflower. It can send some signals to the headphone for example.
(No idea why I read over that when I downloaded Soundflower.)
Anyway, I don’t think it will make a difference for my setup but it might be nice for debugging. Next to that it has a ‘buffer size’ setting to experiment with in case of trouble.
Update 041027: See Minimal latency to headphones with Soundflowerbed.