Oh c'mon, 20k is a big village... King's Landing alone is supposed to have 500k people...From a description of the islands they should number no more than 20,000 from the description of their military they are almost 500,000.
Sure, the islands don't do crops well, but seeing how the population is almost all sailors (slaves doing most of the menial work), they probably just need to import a lot of lemons, and just eat a lot of fish.
And their ships probably don't have really big crews either, considering that Cersei mentions a 10th century ship that held about 100+ people as the biggest one they make, and the ironmen dont seem to use those, and mostly have viking longships.