Yes, I realize I'm using a relatively immature term to refer to this event: I got hacked. In more sophisticated terms, somebody managed to install bonet on my server (presumably)
The Story and Deductions
So yesterday I logged in to my server (at home) and noticed that the "last login" was from a domain I didn't recognize. Turns out that it was me, logging in from school, and another odd domain that I caught was from 3G from my phone.
Anyway, I do remember from quite a while back Google kept captchaing me, and Pokémon Showdown was accusing me of spamming, so I connected the dots and assumed that someone managed to get into my server and install botnet. Also I recently read a story about a guy setting up an ssh honeypot and checking out what people do after they get in https://sysdig.com/blog/fishing-for-hackers/. I wasn't able to locate the botnet program myself (as I could see from that guy's post attackers are pretty good at hiding their stuff), but I was able to deduce its presence from these clues. I'm not too experienced with network monitoring tools.
I checked the ssh log (/var/log/auth.log) and I was surprised at how many login attempts there were. I knew the honeypot guy set up his server in a pretty popular IP space to ensure that he would "fish a hacker" (I hate myself for using this term in this way, thanks a lot society), but I was a nobody in Japan, why would anybody try to break into my home server? -- yeah to install botnet, of course, duh.
Then I also remembered that I left a VNC server open for quite some time... great idea, I know. (Remember how the pi user on a Raspberry Pi can sudo without a password?) I closed it off earlier this year, but it was open long enough that I guess resourceful hackers scanning IPs could eventually find it and put botnet on my server. Especially with all those "insecure IoT webcam" scanners out there on the web, I don't know why I didn't expect anybody to get to my VNC server.
I remember patching up Heartbeat and Shell Shock when they came out, so I doubt the attacker (there we go that's the word) could have gotten in with those methods. Pretty sure it was the VNC that opened the front door.
The Root of the Problem and The Solution
To be honest, prior to this I always thought that Linux systems were invincible (to a certain extent), which I suppose they are if the administrator sets them up properly. If you're good, you could put a server in the DMZ, whereas if you're not, let your commercial router be the firewall.
I guess the problem here was my error of leaving the VNC server open. (I'm just imagining the attacker checking out port 80 to see what's cool and he reads this and is like "lol I got in by brute-forcing your ssh and editing the log lol n00b")
So for the solution I did pretty simple stuff: randomizing the ssh port and closing off all ports but ssh, HTTP and Minecraft on the router. I decided the router is probably better at firewalling then I am. I don't think posting the solutions would be a problem since if an attacker were to come again he/she should be able to deduce these simple measures easily.
What I Learned
Yeah getting hacked is pretty bad. At least I didn't have any user data on it (even if I did I would encrypt it like sane people would, although I can't afford an SSL certificate from GoDaddy for HTTPS). I got turned into botnet so I suppose it's not as bad as losing cash or sensitive information, but I thought it was a good wake-up call to the perils of the Internet.
Prior to this, like I mentioned earlier, I disregarded security quite a bit due to being in the safe haven of Linux, whose image was just shattered by this event. I better start learning network monitoring tools and security stuff. I suppose I could start with ipstat, iptables, and sysdig.