On Thursday, Jan 23, Datadog reported high CPU usage for Jensen. I was running some stuff on it at the time so was wondering if my job was causing the high usage.
htop showed some impact by my job, with the majority if CPU usage being a script or executable called
./cron run by user
testing. I remarked on the high CPU usage at the time but assumed it was an RTP testing something. This was later recognized as being not the case, leading to jensen being shut down and wiped.
Nine remote connections to the testing account on jensen were logged. The primary attack, which we noticed and was the last logged connection, was a 3 part attack: A bitcoin miner (which threw the CPU usage alert), a perl script, and an ssh bruteforcing binary. This attack likely had no impact on user data. The previous connection, which was from the same IP three minutes prior, left a file named
/tmp/up.txt that contained what intially appeared to be the username and password of the testing user, however whether or not this is correct is unknown. It then attempted to change the password with
passwd, which failed because we use Kerberos/FreeIPA. The impacts of the other connections are unknown at this time.
The malware left several files behind in the
/users/u99/testing/ (which will be referred to as
/tmp/, as well as altering the testing user's crontab. Unsurprisingly,
~/.bash_history is empty, and won't provide any clues as to what happened. The first notable modification by the malware is adding a public key to ~/.ssh/authorized_keys. This key had the comment
mdrfckr. When googled, the first result for this word is a post on the Ubuntu forums with the same public key as we had. The second result is about a cron job taking up 100% of CPU usage on an Ubuntu 18 server. Both posts were several months to a year old but matched what happened. There was also a
cron.d file in
~/bashtemp/, which contained some odd lines. These lines were very similar to ones found in both posts and pointed to a few files and folders. Looking into these folders provided some more clues
The first folder pointed to by the crontab on our machine was
.bashtemp, starting with a
. (period), is hidden from normal ls, and we see this being a common trend with other folders used by this malware.
ls -a will show these hidden folders. Inspecting
~/.bashtemp/a/ shows two ELF binaries and four shell scripts. One of those binaries is named
cron, the same name as the original suspicious process. Seeing as this was a common infection, I uploaded the cron binary to VirusTotal.com. VirusTotal quickly recognized it as a Bitcoin miner. Some also recognized it as a trojan. VirusTotal also linked to a report from joesandbox.com, which also classified it as a miner/trojan. The other binary in the folder, named anacron, was classified the same, with the other files being shell scripts used to set up and run the miner.
The next folder to look at was
~/.bashtemp/b/. This contained 4 shells scripts, 3 of which looked similar with the purpose of setting up and running the file run. This file was a bit of an anomaly. The last line is the one that set the ssh key we found by forcibly removing the
~/.ssh folder and adding the mdrfckr key as the only allowed key. This is presumably to make it harder for administrators to access the server and would be effective on a single-user machine that only allows public key login, however that wasn't an issue given the nature of jensen. It also contained a large block of base64 text, which it then decoded and passed to perl. Decoding this base64 showed an exec call to a obfuscated perl script. VirusTotal and JoeSandbox both recognize it as a trojan/shellbot, with one engine labeling it an IRC bot. Curiously, it appears to be written in Portugese.
The last place pointed to by the crontab was
/tmp/.X19-unix/. This folder name varies the most among reports of similar malware but the
/tmp/.X[NUBMER]-unix trend matches. This contained the hidden folder
.rsync as well as a file called
dota3.tar.gz, which just contained the compressed contents of the
.rsync folder. The
.rsync folder contained a few setup scripts similar to what we saw earlier as well as 3 folders, named
b had similar contents to the ones we saw in testing's homedir, but the
c folder is new. The contents are familar, several setup scripts and some binaries. These binaries are new and follow a naming trend. The setup scripts show that each binary is the same program, just compiled for different architectures. The setup scripts select and run the one for the current architecture. VirusTotal reports these binaries as backdoor/bruteforce programs, which matches a report by trendmicro.com, who report similar file structures and names.
Once I had a decent understanding of the files on jensen, I looked through the authentication logs to try to find where the attacker was coming from. There was a theory that the public key we found was left over from when 24ball was compromized, but several things indicate this to not be the case. First off, the scripts found in
~/.bashtemp/b/ are responsible for overwriting all keys in
~/.ssh/authorized_keys and replacing them with its own. The key found in
~/.ssh/authorized_keys matches the key found in the script. Furthermore, I inspected the authentication logs and found nine password logins for the user testing in the past 14 days (Since January 13). There are no logins since Jan. 13 to the testing user using an SSH key. The exploitation of a weak password is further backed up by the fact jensen doesn't mount ceph - even if a publickey was dropped on 24ball when it was compromised, jensen wouldn't have that key.
Out of the nine connections, there was one that seems likely to be the one that introduced the malware we noticed. I was able to correlate these by checking the file creation time of the
dota3.tar.gz file, which
ls -l reported to be
Jan 19 23:53. The last connection we logged was initiated at
Jan 19 23:53:29 and lasted for under 1 second. The originating IP points to a server from a small provider in the Netherlands. The server is not responding on any port now. The other IPs point to Indonesia, Vietnam, Singapore, and New Jersey and all respond on various standard ports. Some of those connections attempted to change the password (with
passwd) but because we use Kerberos/FreeIPA, these attempts failed. Whether or not any of these connections did anything else is unknown.