I agree. Bryter Layer is a very unique album - sometimes I love it to bits and sometimes I hate it. Very strange. But I think it's one of those albums that grows on you after a while - it becomes 'brighter later' (ha ha). It was the last Nick Drake album I bought and I was disappointed at first because of the sheer atmospheric difference to the other two. I guess it was just a little unexpected.
However, last night I couldn't sleep and I wanted to listen to a CD I hadn't played in a while. So I put Bryter Layter on and suddenly everything made sense - the arrangements, the vocals, the instruments... I don't know why but just the combination of it being pitch black and having these wonderfully 'visual' songs playing quietly in the background made me really 'experience' the positives and subtleties of Bryter Layter a lot of people probably miss. Also, the city clock outside my house chimed during 'At The Chime of a City Clock,' which was funny!
Anyway, my recommendation - listen to it with your eyes closed!