Unexplained outbound connection to port 443

[HughA]

Fellow Techos,

I was checking the integrity of my FC4 system using AIDE, and noticed that the checksum signatures of a lot of files have changed inexplicably, for example /bin/rpm and usr/bin/php. In the course of poking around I ran a 'netstat -lan | head -15' command and saw an outbound connection to port 443 on a remote server. Since I didn't have a browser open at the time, this caused me some concern. See the output below:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 198 xx.xx.xx.xx:46657 209.132.177.100:443 ESTABLISHED
tcp 1 1 xx.xx.xx.xx:44315 209.132.177.100:443 LAST_ACK
tcp 0 0 :::65480 :::* LISTEN
tcp 0 0 :::65522 :::* LISTEN
Active UNIX domain sockets (servers and established)

[root@localhost hugh]# nslookup 209.132.177.100
Server: 202.63.43.130
Address: 202.63.43.130#53

Non-authoritative answer:
100.177.132.209.in-addr.arpa name = xmlrpc.rhn.redhat.com.

Authoritative answers can be found from:
177.132.209.in-addr.arpa nameserver = ns3.redhat.com.
177.132.209.in-addr.arpa nameserver = ns1.redhat.com.
177.132.209.in-addr.arpa nameserver = ns2.redhat.com.

[root@localhost hugh]#

I visited the web site as indicated above, and my server complained that it didn't like the certificate I was using. When I opted to accepted the certificate temporarily for the session, the web page simply comes up as 'Red Hat Network'. Is this an indication that I have been hacked and am leaking information? (Note that 'rkhunter' and 'chkrootkit' both indicate that my server is clean).

Thanks and regards,
Hugh

[bwkaz]

First: Fedora Core is Red Hat under a different name. (After RH9, they went to officially supporting only companies that bought Red Hat Enterprise Server; Fedora Core is for home users that want a free download without official support.)

Second: Since it's xmlrpc.rhn.whatever, I'm assuming that it's the server that handles XML-RPC calls having something to do with the Red Hat Network (which is how RH/FC automatically distribute patches). XML-RPC is a way for programs to "call functions" on a server, using an XML based transport. It acts similarly to function calls inside a process.

Third: the only thing a program like AIDE does is check that the signatures of various files on your system haven't changed from the last time its database was updated. ANY kind of update (for instance, to the PHP package or the RPM package, or any other package) will invalidate the signatures that AIDE collected when it built its database, because the update will change files' contents.

The only way to successfully use an intrusion-detection system is to run it just before applying any vendor update(s), make sure everything's OK, then apply the update(s), then rebuild the AIDE (or whatever) database. Then next time, run it in detect mode first again. It's important that you do not just blindly update the AIDE database; it is possible (though unlikely) that not every change on your system was made by an update. This is why you have to check AIDE before applying the updates, not after.

But you've got something looking for vendor updates automatically, so you can't very easily do that. (I don't want to say that looking for updates automatically is bad, though; I have an "apt-get update && apt-get upgrade" that runs every day from cron on a Debian box at work.)

Bottom line: That SSL connection is getting updates from Fedora. You might as well ignore AIDE from this point forward if you're going to keep applying vendor updates automatically. There's no way to tell the difference when using AIDE between an update changing a file versus an attacker changing the file. (Unless you always run the detection just before applying the update, and then update the AIDE database after applying the update, as above.)

[HughA]

Thanks for the feedback bwkaz,

I am aware the FC is RH by another name, I do understand how AIDE works, and I am well aware of the potential pitfalls of automatic software updates ;-)

Now that we have that out of the way, I myself came to the same conclusion as to a possible cause of the symptoms I am observing - that is, automatic updates of some sort. I have deliberately avoided setting up any automatic updates so that I could co-ordinate software version control with intrusion detection and with backups / media cycling / DR strategies (after all, I don't want to back up a compromised server, do I?). I have checked root crontab (nothing there), and it seems that anacrontab only runs a 'yum' update if the file /var/lock/subsys/yum exists...so the hunt continues in this regard. In the meantime, I have set up iptables to block outbound traffic except for ports 53 and 80, with dropped packets being logged so I can chase down the culprit.

If I may be so bold as to ask a related question in regard to this thread, what does anyone make of the fact that some of the files that have been changed show (via AIDE):

a) A different MD5 checksum signature
b) A different SHA1 checksum signature
c) A differerence in ctime
d) NO difference in mtime
e) NO change in file size

To my mind, this looks suspicious - the checksums have changed, but file sizes and modification times are the same...see below for an example:

File: /usr/bin/php
Ctime : 2005-11-27 10:49:44 , 2006-03-13 04:02:55
Inode : 922985 , 920389
MD5 : r+2tPfCVtUrm7tLDiom0Tw== , Ulm0vXVUa8wAxDHIYgVuDA==
SHA1 : Jw7LpY/SmI0aic5j4gBco52gUlE= , soM9t/e+rRjUIqdH4Mdms3gjTec=

I am going to set up a DR server with a fresh installation of FC and do some manual checks of RPM versions and file sizes and checksums, just to be sure. I'll let you all know how I go with this.

Regards,
Hugh
Senior UNIX Technical Consultant (AIX Certified Advanced Technical Expert)

[bwkaz]

what does anyone make of the fact that some of the files that have been changed show (via AIDE):

a) A different MD5 checksum signature The contents of the file named <whatever> have changed.

b) A different SHA1 checksum signature The contents of the file named <whatever> have changed.

c) A differerence in ctime The inode of the file named <whatever> has changed. This can happen because of changes in the mtime, ctime (and perhaps atime) fields, the file's size, or the list of blocks used to store the file. (Or perhaps a few other fields.)

It can also happen if the file gets removed and recreated. (This is the standard method of upgrading a package -- if you don't remove the old version, then programs using the file will segfault when you start writing over the place on disk where their code is coming from. So installers will remove the old binary/library/whatever first, then copy the new one into place. This way, since unlink()ed files don't get physically removed until the last handle to them is closed, current programs can keep running under the old version until they get restarted. And so you don't have to reboot the machine just because you applied a patch to IE.)

d) NO difference in mtime Someone used "touch --date=whatever /path/to/file" after changing the file's inode. (Or they called the glibc equivalent function.) It's possible that package installations will do this, and it would make sense if the package's version didn't increment (but just the patch-set applied by the distro).

e) NO change in file size Not all bugfixes change the file's size. Changing one opcode to another in the binary (e.g. changing a JL (jump if less than) to a JGE (jump if greater than or equal to)), or changing a data structure to another structure of the same size, won't change the file's size.

It is possible that something nefarious is going on. However, I highly doubt it -- you've caught your own system's autoupdate in the act, and you're wondering why files have changed. That just seems like a "duh" to me. ;)

[HughA]

Thanks bwkaz,

If software updates are given to re-setting mtimes, then my fears are allayed. I will concede that this is indeed a 'duh', and will consider changing my nom-de-techo to Homer Simpson :)

Regards,
Hugh