Have you tried running doom emacs in tmux on the remote server and accessing it with ssh? Doom emacs is all the good of an emacs environment, all the good of vim keybinds, and they worked in a decent amount of optimizations so it only loads the necessary stuff on demand (mine has a startup time of just over 1 second, slower than vim but barely an inconvenience). Can write a quick script to ssh copy (or git pull) your current configs on the server so you only have to maintain one set of configs if you want

scp ~/.config/doom/config.el username@server:~/.config/doom/config.el

Run emacs in tmux if you want to keep the emacs session open across multiple ssh sessions