The line "technically it's ephebophilia" is like "I'm not racist, but". I mean, I've uttered both, but they're both lines that are considered pretty clear indicators, even if they aren't always.
Yes, there's a very big difference between
A) being sexually attracted to and/or abusing toddlers/babies
B) being sexually attracted and/or abusing preteens
C) being attracted to and/or abusing girls who have reached a physical age where they're sexually/physically adult, even if they are far younger than yourself.
A lot of our culture is based on the adulation/worship of youth as beauty. Compare beauty ideals from years/generations ago to now and there are clear differences. Not a straight line, though - the 70s and 90s had a clear thing for youth, with the ideal being pretty much a 16 to 18 year old girl. These days we're skewing more towards 24-25 year old and somewhat more feminine - ads and books are back in vogue. Some periods (and cultures) had beauty ideals skewing more towards the motherly figure - wide hips, big breasts, fertile, etc.
You may think a 50 year old guy fancying a 18 year old girl is wrong, and I agree - but it's a whole different thing to a guy of any age being attracted to a 4 year old. It's not a matter of degrees, it's a whole different thing.