SSH Login to a New Server: From Password to Disabling Root

You've received the server, the root password is in the panel, and the first thing you do is:

ssh root@185.10.20.30

And a few seconds later you see this:

Permission denied (publickey).

Or worse: you log in, do your work, and the next morning the same command doesn't respond because someone out there has tried the password thousands of times and fail2ban has blocked your IP. This article is about that first hour; the work that, if done right, makes the rest of the server's life easier.

First, diagnose why SSH login is being rejected

Before making any changes, read the error message. There are three common cases, and each has a different remedy.

• Connection refused means the sshd service isn't up or is listening on a different port. Log in through the server panel console (VNC) and run systemctl status sshd.
• Permission denied (publickey) means the server only accepts keys and you haven't sent one. Use ssh -v root@IP to see which keys were offered.
• Connection timed out is almost always the firewall, not SSH. Either the port is closed in the cloud panel, or ufw on the server hasn't left port 22 open.

If none of these work and you don't have console access either, the usual rescue path is booting into rescue mode; the procedure is laid out step by step in Reinstalling the server and working with rescue mode. Read this before you break access, not after.

Create a non-root user before anything else

Working as root is tempting and wrong. Create a user, add them to the sudo group, and log in as them from that moment on:

adduser ali
usermod -aG sudo ali
rsync --archive --chown=ali:ali ~/.ssh /home/ali

That rsync line matters. If you copy the public key manually and set the file permissions wrong, SSH will silently reject you. The correct permissions are:

chmod 700 /home/ali/.ssh
chmod 600 /home/ali/.ssh/authorized_keys
chown -R ali:ali /home/ali/.ssh

Now open a new terminal and run ssh ali@IP. As long as this window is open, don't close the root window. This is your only real insurance policy.

Transferring the key, and why we keep the password

On your own laptop:

ssh-keygen -t ed25519 -C "ali@laptop"
ssh-copy-id ali@185.10.20.30

I prefer ed25519 over rsa: shorter key, faster signing, and secure by default on OpenSSH version 8 and above. If you have an old client that doesn't recognize ed25519, run ssh-keygen -t rsa -b 4096 and that's it.

Here's where people make a mistake: many set PasswordAuthentication no that same night and don't change the root password either. Three weeks later the laptop is replaced, the old key goes off with the backup files, and there's no way in except rescue. I keep password authentication open for the non-root user and only disable root. The cost is a slightly larger attack surface; in return you have a legitimate back door protected by fail2ban and a long password.

Disabling root login: a small change, a big effect

Create the file /etc/ssh/sshd_config.d/99-hardening.conf so you don't touch the distribution's default settings:

PermitRootLogin no
PasswordAuthentication yes
PubkeyAuthentication yes
MaxAuthTries 3
LoginGraceTime 20

Then sshd -t to test the syntax and systemctl reload sshd. Never run restart when you're connected over that same SSH session; reload doesn't drop the current connection, restart might.

Don't underestimate MaxAuthTries 3. On a server I set up last month, /var/log/auth.log showed 41,000 failed attempts in 24 hours; with this setting each connection closes after three attempts and the load on sshd drops noticeably.

fail2ban and the port: two decisions that are often confused

Changing the SSH port from 22 to something else doesn't bring real security; it just reduces log noise. If you find the scanners annoying, change it, but invest in fail2ban instead:

apt install fail2ban
# /etc/fail2ban/jail.local
[sshd]
enabled = true
maxretry = 4
bantime = 1h

A point that's rarely mentioned: if you changed the SSH port, you must also update port in the jail, otherwise fail2ban listens on the wrong port and never bans anyone. The sign is that the log is full of failed attempts but fail2ban-client status sshd shows zero.

Things that should be finished in that first hour

1. Full update: apt update && apt full-upgrade -y and then a reboot so the new kernel takes effect.
2. Firewall: ufw allow 22/tcp && ufw allow 80,443/tcp && ufw enable. Order matters; if you enable first, you lock yourself out.
3. Server time: timedatectl set-timezone Asia/Tehran so logs read in local time.
4. Swap if RAM is low, and check free -h and df -h.
5. An initial snapshot or backup from the panel, before installing anything heavy.

If the server becomes slow after all this, the problem probably isn't SSH; follow the CPU and I/O usage pattern in Diagnosing the cause of high server load.

Transferring files and the road ahead

To upload a project from your laptop, scp -r ./app ali@IP:/var/www/ is enough, but for anything larger than a few hundred megabytes use rsync so it resumes from where it left off if the connection drops. The details of the flags are in Transferring files with scp and rsync.

If your application is containerized, bring up Docker before anything else so migration is easier later; the guide Setting up Docker on a VPS covers everything from installation to the first container. And if MySQL is going to live on this same server, remove the anonymous user that same night; the procedure is in Securing MySQL.

For small and medium servers, this order is enough. If your workload is heavy and you need dedicated CPU and NVMe disk resources, Linux hosting is a more sensible starting point than manually configuring everything on a shared VPS.

Frequently asked questions

Why can't I log in via SSH anymore after disabling root login?

Because you didn't test the non-root user before disabling root, or the permissions on ~/.ssh aren't right. If the directory permission is 755 or the key file is 644, SSH rejects the login without any clear message. Log in through the panel console and apply chmod 700 and chmod 600.

Does changing the SSH port make it more secure?

Not as much as people imagine. Scanners search the full port range and just spend a few extra seconds. Its real effect is reducing log noise. If you want to stop brute-force, fail2ban and public keys have a bigger impact.

Is an ed25519 key better or RSA?

For any server running OpenSSH version 6.5 or above, ed25519 is the better choice: shorter, faster, and not dependent on key length. Only if you're working with old network equipment or very old clients, stick with RSA at 4096 bits.

What happens if I lose the private key?

If you've left password authentication open for the non-root user, you log in with that and generate a new key. If it's closed, only rescue mode remains. This one small decision is what makes the difference between five minutes and an hour of downtime.