Kae was bullied at school because his intelligence shoved such a huge wedge between him and his peers. He didn’t understand them (mostly why they were so slow) and they didn’t appreciate his nonchalance. Fortunately, it never progressed quite to the point where he came home injured, but it still made school miserable.
Probably the only person who knew how to treat him well was his mother, and while his father didn’t, the man was at least nice about it at the time. His mother was the only person he didn’t feel ostracised from in some way.
The reason he enjoys smart people (note: people smarter than him) so much is because it makes him feel like less of an odd-man-out. Very few people, however, measure up to that standard.
While his intentions are far from malicious, he’d still unwittingly destroy (at least part of) the world in order to accomplish his goal, which is the main reason why no one trusts him with his ability.