To have a decent experience when ssh-ing to other servers and using not "well known" xterms, it's important to set up your local SSH so that it specifies a standard.

In your ~/.ssh.config file (create it if it doesn't exist), add this :

Host *
     SetEnv TERM=xterm-256color

Now, you should have some colors and backspace working normally.