Here is what I have learned. If I play games on my phone and ignore everyone, people comment on how I spend all my time on my phone. If I COLOR and ignore everyone, no one cares.
I wonder about that reaction. A coloring book is an act of creation, reading is gaining knowledge or entertainment.
The phone can be anything, but I wonder if the people you're physically with think you're socializing with others. Facebook and texting, instagram, tinder - these are signals that the you're choosing other people over the person right in front of you.
Coloring and reading - you're choosing an activity while someone is here, but you're not necessarily switching to another social situation.
I've had more than one business meeting where participants were on their phones throughout, though, and whipping out a coloring book or piece of literature and doing that during the meeting would have been unacceptable. In these cases it's assumed that the participants are using their phones for meeting business, but everyone knows they are more often than not tuned out - they just can't call them out on it without risking being embarrassed if they actually are doing something meeting related.
Social situations and expectations are really fascinating...