Jquery scrolls up after messages are sent in chat page

The auto scroll works for a couple messages on it's own. Then it randomly gives out and starts scrolling up when the user sends a message.

Here's the applicable Javascript. And the limited styles (Haven't really gotten to styling yet.) Also the site if you want a reference as to what it looks like. http://ift.tt/1cjVJCd

  //send message
    $messageForm.submit(function(e){
        e.preventDefault();
        socket.emit('send message', $messageBox.val());
        $messageBox.val("");
        $("#chat").animate({ scrollTop: $("#chat").height() }, "fast");
        return false;


    });


    //Add the users message to the chat.
    socket.on('pushMessage', function(data){

        $chat.append("<li><b>" + data.nick+ ": " + "</b>" + data.msg + " </br>" + "</li>");
        $("#chat").animate({ scrollTop: $("#chat").height() }, "fast");



    });

Styling:

 #chat{
    height:500px;
    width:300px;
    overflow-y:scroll;
    overflow-x:auto;

}
#mainWrap{
    display:none;

}
#chatWrap{
    float: left;
    border:1px #000 solid;
    overflow-y:scroll;
    width:302px;