I'll admit, sometimes I have started to read a story solely for the sex scenes but as I read, found it intriguing and end up reading the entire thing, which makes the sex scene all the more gratifying as was previously mentioned.
I personally depend the build up on the characters I'm writing. Is the character meek and/or shy...a virgin even? Or is the character sexually confident by nature? The answer to this is usually what determines how much of a build up you have, I have found.