Robin, no you are not so screwed up (from what I've read in your journal). I still remember a co-worker from when I was 16....he had fat fingers and big hands---I just hated it when he'd walk by me and lay one of those big paws on my shoulder or back. Gave me the hebegebees. I also went out on a date with a very sweet guy when I was about 18, he also had big sweaty hands and fat fingers......I couldn't hold hands with him and was praying for the date to be over---it just bugged me that much. He was (and still is) a great guy and a lovely friend -- but there was no way in the world that I could ever have thought of him romantically or seriously have him in my life, because of those hands.
Sure sounds shallow when I read what I've wrote, but visualizing those two instances, from many, many years ago (and a few more many-s added in) and I get that same feeling.
The only reason I'm sharing that is so that you'll know you aren't alone in noticing (and being bothered) by something that sounds pretty trite and insignificant.
For some people teeth, noses, body hair (too much or not enough) and a multitude of other things are a turn off.. no odder or less important than 'hands'.
Don't force yourself to feel feelings or to do things that just don't feel right or make you happy. He may be a good friend, but nothing says he has to be a hand holding, kissing or more intimate friend.
Hope something I've shared will help underscore what you are already thinking or feeling.
My best to you!