Investigating Unknown Ports and Services on Linux System

17 Nov 2022 (1760 Words, 10 Minutes)

Today I will walk you through a scenario which most probably you have ran into while doing nmap scans on your localhost machine, or in the subnet itself.

NOTE: The demonstrated techniques could be applied to numerous different scenarios, for instance doing the same on any remote machine or in the enterprise environment but - word of caution here, please be very careful because the uptime of services and the dependency of applications on them matter a lot in the production environment. That’s the reason why I am moving ahead with the case of localhost.

It’ll help you cultivate your Threat Hunting skills and make you even better at managing network and system services, processes and state of ports in Linux.

The Scenario

Imagine you scanned your localhost (nmap localhost -p- -T4 -sV) and found “unknown” services running on some unusual port.

I run nmap scans on my localhost a lot. I keep unnecessary services disabled. And when I run any application or tool which enables the disabled services while I am using it, I ensure that it’s disabled again once I am done with it.

I will brief upon how to tackle this scenario, what to do and what to avoid and things to consider before acting upon it.

Unknown services & unusual ports!

Let’s begin with our beloved nmap scan :

nmap -T4 -p- localhost -sV

Results :

Clearly one can see 4 important details :

1. Port
2. State
3. Service
4. Version

The first two services are known and good for me, because I know I have used xrdp recently and also PostgreSQL Database which was a dependency for an application I used.

But the 3rd one sticks out and warrants an investigation. So I used netstat utility to list all the sockets listening, established or in any state of connection with any local or remote host on either IPv4 or IPv6 and disabled to resolve names.

So two of my favorite set of switches I often use with netstat is :

netstat -punta
netstat -tulnp
# use netstat --help or man netstat for more info on switches. 

It doesn’t look good because the Local Address (127.0.0.1) on port 41699 is in LISTEN state for a foreign address 0.0.0.0:* which means all IPs and all ports.

NOTE: We can also use ss -tulnp / ss -punta, It’s more verbose and beautifully outputted.

I listed all the sockets again and grepped for the port 41699 -

It’s interesting to note that the communication was happening over my localhost only, and not with any shady remote host. But digging in deeper using utility tools such as fuser and lsof reveals much insight into what’s happening. Here fuser utility wasn’t helpful but these command come in handy.

Notice that -

lsof -i TCP:41699

reveals that the command used is chrome and the sockets are bound in an established communication - localhost:54590->localhost:41699 (ESTABLISHED)

again, fuser utility couldn’t help with anything, meanwhile ss displayed generic network connections we already discovered via netstat.

Process / PID / Program name field in both the command (netstat and ss) are either going blank whitespace or with a (-)

This indicates that I have to use root privileges to pull out more information for those fields.

Running netstat with root privileges revealed interesting info regarding the process name, which is responsible for opening that port in first place and going into LISTENing state -

So it’s containerd - the daemon for container runtime.

netstat displays the PID / Program name associated with each socket, and here we can notice that PID = 100198, so let’s kill it and close that open port.

sudo kill -9 100198 # Executed successfully. Killed that process.
sudo netstat -ap | grep 41699 # NO OUTPUT. Means its closed port yay!!
ss -tlnp # TF?! a new higher port respawned! 43927

We just killed the process (100198) responsible for opening the port but then the service (containerd) respawned another higher port in LISTENing mode… Service(s) use a bunch of processes underneath to carry out various tasks, like sync, monitoring, booting up and so on. We can definitely control the services, I will show in a moment, but the key thing to note here is - Do not rely on just killing the processes. Sometimes those process can be standalone which implies that once you kill it, it won’t appear again until you reboot your system (also depending upon the configurations). In other cases (majority) you have to take down the service itself so that the process doesn’t appear again.

NOTE: Here you can see an example of persistent process, which will respawn everytime it’s (Process Identifier) PID is killed by us.

systemctl utility helps us inquire about any service.

sudo systemctl status containerd

Running the command - (to stop the service)

sudo systemctl stop containerd

you can disable it as well, via just replacing stop with disable, and vice-versa you can start or enable it at your wish.

nmap reveals that there are only 2 open ports available now, services of which are known and trivial to put them down if I wished.

1. 3389/tcp xrd
2. 5432/tcp postgresql

sudo systemctl stop postgresql
sudo systemctl disable postgresql
sudo systemctl stop xrdp

Additional Commands

Environments differ and so does the needs. Hence I am listing some more commands which you can use to enumerate the system better. Hopefully these will give you insight into the existence of a process related to any service and its associated open ports. I highly encourage you to try them out in your environment. This can help in local enumeration after post-exploitation as well.

ps aux

It will enumerate all the running processes in the system, it’ll display you the user whom under the process is running as (security context), the PID, its start time, and most importantly the command itself. There are lots of other fields and lots of other ways of display / output we can customize it.

sudo cat /etc/services

It has network services mapped to its port number respectively.

systemctl list-unit-files --type service --all

It will display all the services with their state in the system. Enabled, disabled, masked, static, alias, indirect, generated… are some of the states of services mentioned.

for x in {0..65535}; do sudo lsof -i:$x; done

We can iterate over lsof utility to find more information on processes.

Next is the /proc/ directory in linux, it’s kinda special in the sense that it contains all the running processes details in the form of files and it does include the processes owned by root user and system (kernel level) processes. Few basic commands to begin with are -

ls -al /proc/1026/exe # executable of the process
ls -al /proc/1026/cwd # current working directory of the process
cat /proc/1026/cmdline # command used to run the process / command line argument

Let’s say we wished to inspect the process with PID = 1026

By now you’ll know what is the current working directory of that process, along with the exact executable and any command line arguments if it has any. This kind of information is very critical to obtain while doing Threat Hunting in live environments.

DO's & DONT's

Tips on what to do :

1. You can always look up on the Internet what the higher port number stands for, which network service it maps to and so on…
2. You should always be mindful about your activities, because if you took down some service inadvertently in a production / live environment, it cost a lot in business terms and JIRA tickets.
3. Always follow a methodological approach to begin with, please avoid / be very careful while automating this step. Administrators can host services on non standard ports as well.
4. Do extensive research at each step, because it would lead you to making wise strategic decisions.

Tips on what NOT to do :

1. Do not rush into any conclusion and start killing processes / services in any stretch of your imagination, capability or capacity. Remain calm and composed throughout the investigation and move ahead based on that analysis.
2. Do not panick or attribute unusual higher ports to any threat actor. Because rootkits, backdoors, C2s, malwares often use higher port numbers to communicate in a while. Based on the investigation and analysis, issue Incident Response or full fledged Threat Hunting if needed.

Concluding remarks

Although malicious threat actors have better ways to hide in the system and establish privileged persistence, it’s healthy to keep a proactive stance for your digital assets. Doing these mini Threat Hunts will ensure nothing goes past your defense solutions and still continues to damamge your Confidentiality, Integrity and Availability. I will get this all covered in the upcoming blogs. This post wasn’t greared towards full fledged Threat Hunting, but it keeps paranoia in control.

This particular topic was something which haunted me a lot when I was getting started in the realms of offensive security. Usually during reconnaissance and footprinting, I would nmap scan the target server and fetch information, based upon that I can plan and mount my initial set of attacks. My senses won’t rest until I make sure that my own localhost has the least number of services running which interacts with the broader internet or network in general. These services would be listening for active connections on all the ports on all the IP, whether local or remote.

The very idea of being ignorant and living in oblivion, of not having visibility into your own localhost haunts me till date. I am progressing and will keep sharing my knowledge and experience. Coming next would be nmap cheat sheet (my custom version), and how to detect and uncover Persistence in your Linux system, there I will deal with persistent services and much more.

If you want me to blog about anything related to cybersecurity in general then get in contact with me. I will try my best to blog on it.