We could place all smilies in a dropdown box that shows up by clicking a smiley button. Similar to how it works on Facebook, Viber and on keyboard layouts for most smartphones and tablets.
Plus, we could get rid of the popup for the extra smilies if we utilize some sort of pagination inside the box.
P.S.: Somewhat related to this RFC and already mentioned before - we could use a web-safe color picker for the font color button.