What I learned from Russian students: logging is important
When I published my blog about openSUSE a couple of weeks ago, most questions I received in private were about the Russian students I mentioned. In that blog I quickly described how my interest in information security started, about 25 years ago. This blog gives you a bit of historical background and a few more details.
Historical background
It was 1995. I was studying at a university, but I was already running one of the servers of the faculty. It was a Linux box, and I also helped to run a FreeBSD server hosting the faculty web server. It was just three years after the Soviet army finally left Hungary. Our university had many students from Russia. While Hungarian students could attend the university for free, Russian students had to pay for their studies. As they were paying a lot, they could do anything, nobody punished their activities. And they did a lot of things, as they felt that they can still do anything in the ‘colonies’.
It was 1995, there was no Internet yet in the student dormitories. There was no Gmail, or any similar provider yet. Not even teachers received e-mail addresses automatically. Even if some people had computers at home, there was no Internet access yet from homes. Students could access servers from computer labs at the university. The Russian students had their own computer lab, where nobody else was allowed to enter.
It is 1995, the fifth consecutive year that funding was taken away from higher education. Which meant that faculties started to ask money from other faculties for their services. Russian students belonged to another faculty, so they could not get a user name on our servers.
Infosec is overrated
By that time even if I was running a couple of servers, I was just the same as the vast majority users even today. I mean, I thought that information security is overrated, ease of use, comfort are a way lot more important. It did not help either, that most of the commonly used protocols were not encrypted, like telnet, ftp, rsh and others. Even these protocols were often difficult to use from Windows machines. I was learning Linux and FreeBSD, and I was enabling all kinds of services. Using rsh between the two faculty UNIX servers was fun.
Logging is important
I checked the logfiles of the servers I managed occasionally, but mostly only to check if the hard drives were showing any signs of failure. While browsing the logs for hard drive errors, I came across some suspicious login messages. Logins from previously not seen unknown IP addresses. I knew that the addresses were from campus, so I asked around. It turned out, that they belonged to the Russian students laboratory. And talking to the user it turned out, that he was unaware that his account was used also by someone else.
The exact order of events is a kind of blurry, it was a quarter of a century ago. I started to check log messages not just for hard drive problems but also for security related events. I could see more and more logins from the Russian students laboratory. It was a kind of cat and mouse game, I was trying to keep unauthorized users out of the system. They kept coming back and started to do nasty things. Along the way I learned a lot about security:
- Network sniffing: most of the university had a BNC network and was using hubs instead of switches. Combine these with non-encrypted protocols…
- Keyboard loggers
- Black market. Access for students of our faculty was free, they just had to ask for it. Sometimes minutes after they received access, there was a login from the Russian lab. Accounts on my servers had a good price…
- Denial of Service: they tried all kinds of DoS attacks, like fork bombs, too many logins, etc.
- Stepping stone for further attacks, so I got some not so kind e-mails asking for explanation
Turning on a firewall could have been an easy way out, but seeing the IP addresses of the Russian lab in the system logs was the perfect indicator of compromise for an account. The account got quickly disabled, either for life (see black market) or until a password change. In the second case I tried to investigate, how the password was stolen. And of course gave a quick education on security awareness. Showing my log messages I tried to ask for some help to stop the Russian students, but as I was just a first year student and Russian students were paying: nobody cared.
Next steps
After so many years I do not recall any more how I got the hint, but I was suggested that I visit the Russian students computer lab. I was not supposed to enter there, but as they were messing with my servers, I did not care. The door was open, I walked in and looked around. The /etc/passwd file of my Linux box was printed on the wall. Even if encrypted, but it contained the passwords. As also described in my openSUSE blog, this was a final push towards information security.
FreeBSD already had passwords separate from the user readable passwd file, so I knew the concept. I looked around and found that the Linux distribution called Jurix had shadow passwords. It was a brand new thing in the Linux world at that time. I quickly migrated my Linux server to Jurix and did all kinds of hardening along the way. I removed all non-essential services, like rsh. Even if most users kept using telnet and other insecure services, I started to use SSH, which was just released.
When Russian students realized that they cannot get into my servers easily any more, they even tried to bribe me for access – with a counterfeit gaming CD for Windows :-)
Epilogue
As you can see, I ended up on the defender side. I did lots of security hardening and built systems that ran securely even years after I abandoned them. Logging still takes an important role in my life: I work with syslog-ng. Russian students were a major PITA at that time, but I learned a lot about security while I was trying to keep them out of the servers I managed.