[Day 2] Log Analysis Santa’s Naughty & Nice Log¶
Ok, on attaque cette seconde journée du calendrier de l’avent de Noël. Le thème d’aujourd’hui est simple : Lecture de log. Cette fois-ci, il faut spawn une machine (et donc se connecter dessus). Je relance donc ma kali. Bon, les explications sont super une fois encore, alors je les garde (mais en anglais). je les traduirais -peut-être- un jour.
Le cours¶
What Are Log Files and Why Are They Useful¶
Log files are files that contain historical records of events and other data from an application. Some common examples of events that you may find in a log file:
Login attempts or failures
Traffic on a network
Things (website URLs, files, etc.) that have been accessed
Password changes
Application errors (used in debugging)
and many, many more
By making a historical record of events that have happened, log files are extremely important pieces of evidence when investigating:
What has happened?
When has it happened?
Where has it happened?
Who did it? Were they successful?
What is the result of this action?
For example, a systems administrator may want to log the traffic happening on a network. We can use logging to answer the questions above in a given scenario:
A user has reportedly accessed inappropriate material on a University network. With logging in place, a systems administrator could determine the following:
Question |
Answer |
|---|---|
What has happened? |
A user is confirmed to have accessed inappropriate material on the University network. |
When has it happened? |
It happened at 12:08 on Tuesday, 01/10/2022. |
Where has it happened? |
It happened from a device with an IP address (an identifier on the network) of 10.3.24.51. |
Who did it? Were they successful? |
The user was logged into the university network with their student account. |
What is the result of the action? |
The user was able to access inappropriatecontent.thm. |
What Does a Log File Look Like?¶
Log files come in all shapes and sizes. However, a useful log will contain at least some of the following attributes:
A timestamp of the event (I.e. Date & Time)
The name of the service that is generating the logfile (I.e. SSH is a remote device management protocol that allows a user to login into a system remotely)
The actual event the service logs (i.e., in the event of a failed authentication, what credentials were tried, and by whom? (IP address)).
Common Locations of Log Files¶
Windows¶
Windows features an in-built application that allows us to access historical records of events that happen. These events are usually categorised into the following:
Category |
Description |
Example |
|---|---|---|
Application |
This category contains all the events related to applications on the system. |
For example, you can determine when services or applications are stopped and started and why the service « tryhackme.exe » was restarted. |
Security |
This category contains all of the events related to the system’s security. |
For example, you can see when a user logs in to a system or accesses the credential manager for passwords. User « cmnatic » successfully logged in. |
Setup |
This category contains all of the events related to the system’s maintenance. |
For example, Windows update logs are stored here. The system must be restarted before « KB10134 » can be installed. |
System |
This category contains all the events related to the system itself. This category of events contains logs that relate to changes in the system itself. |
For example, when the system is powered on or off or when devices such as USB drives are plugged-in or removed. The system unexpectedly shutdown due to power issues. |
Linux (Ubuntu/Debian)¶
On this flavour of Linux, operating system log files (and often software-specific such as apache2) are located within the /var/log directory. We can use the ls in the /var/log directory to list all the log files located on the system:
cmnatic@aoc2022-day-2:/var/log$ ls -lah
total 724K
drwxrwxr-x 9 root syslog 4.0K Nov 14 10:59 .
drwxr-xr-x 13 root root 4.0K Oct 26 2020 ..
drwxr--r-x 3 root root 4.0K Nov 14 10:56 amazon
drwxr-xr-x 2 root root 4.0K Oct 26 2020 apt
-rw-r----- 1 syslog adm 11K Nov 14 11:03 auth.log
-rw-rw---- 1 root utmp 0 Oct 26 2020 btmp
-rw-r--r-- 1 root root 7.3K Nov 14 10:59 cloud-init-output.log
-rw-r--r-- 1 syslog adm 251K Nov 14 10:59 cloud-init.log
drwxr-xr-x 2 root root 4.0K Oct 7 2020 dist-upgrade
-rw-r--r-- 1 root adm 36K Nov 14 10:59 dmesg
-rw-r--r-- 1 root adm 36K Nov 14 10:56 dmesg.0
-rw-r--r-- 1 root root 12K Oct 26 2020 dpkg.log
drwxr-sr-x+ 3 root systemd-journal 4.0K Nov 14 10:55 journal
-rw-r----- 1 syslog adm 98K Nov 14 10:59 kern.log
drwxr-xr-x 2 landscape landscape 4.0K Nov 14 10:57 landscape
-rw-rw-r-- 1 root utmp 286K Nov 14 11:03 lastlog
drwx------ 2 root root 4.0K Nov 14 10:55 private
-rw-r----- 1 syslog adm 207K Nov 14 11:03 syslog
drwxr-x--- 2 root adm 4.0K Nov 14 10:55 unattended-upgrades
-rw-rw-r-- 1 root utmp 8.3K Nov 14 11:03 wtmp
The following table highlights some important log files:
Category |
Description |
File (Ubuntu) |
Example |
|---|---|---|---|
Authentication |
This log file contains all authentication (log in). This is usually attempted either remotely or on the system itself (i.e., accessing another user after logging in). |
auth.log |
Failed password for root from 192.168.1.35 port 22 ssh2. |
Package Management |
This log file contains all events related to package management on the system. When installing a new software (a package), this is logged in this file. This is useful for debugging or reverting changes in case this installation causes unintended behaviour on the system. |
dpkg.log |
2022-06-03 21:45:59 installed neofetch. |
Syslog |
This log file contains all events related to things happening in the system’s background. For example, crontabs executing, services starting and stopping, or other automatic behaviours such as log rotation. This file can help debug problems. |
syslog |
2022-06-03 13:33:7 Finished Daily apt download activities.. |
Kernel |
This log file contains all events related to kernel events on the system. For example, changes to the kernel, or output from devices such as networking equipment or physical devices such as USB devices. |
kern.log |
2022-06-03 10:10:01 Firewalling registered |
Looking Through Log Files¶
Log files can quickly contain many events and hundreds, if not thousands, of entries. The difficulty in analysing log files is separating useful information from useless. Tools such as Splunk are software solutions known as Security Information and Event Management (SIEM) is dedicated to aggregating logs for analysis. Listed in the table below are some of the advantages and disadvantages of these platforms:
Advantage |
Disadvantage |
|---|---|
SIEM platforms are dedicated services for log analysis. |
Commercial SIEM platforms are expensive to license and run. |
SIEM platforms can collect a wide variety of logs - from devices to networking equipment. |
SIEM platforms take considerable time to properly set up and configure. |
SIEM platforms allow for advanced, in-depth analysis of many log files at once. |
SIEM platforms require training to be properly used. |
Luckily for us, most operating systems already come with a set of tools that allow us to search through log files. In this room, we will be using the grep command on Linux.
Grep 101¶
Grep is a command dedicated to searching for a given text in a file. Grep takes a given input (a text or value) and searches the entire file for any text that matches our input.
Before using grep, we have to find the location of the log file that we want to search for. By default, grep will use your current working directory. You can find out what your current working directory is by using pwd. For example, in the terminal below, we are in the working directory /home/cmnatic/aoc2022/day2/:
cmnatic@thm:~/aoc2022/day2 pwd
/home/cmnatic/aoc2022/day2/
If we wish to change our current working directory, you can use cd followed by the new path you wish to change to. For example,cd /my/path/here. Once we’ve determined that we are in the correct directory, we can use ls to list the files and directories in our current working path. An example of this has been put into the terminal below:
Using ls to list the files and directories in our current directory
cmnatic@aoc2022-day-2:~$ ls -lah
webserver.log helloworld.txt mydirectory
Now that we know where our log files are, we can begin to proceed with learning how to use grep. To use grep, we need to do three things:
Call the command.
Specify any options that we wish to use (this will later be explained), but for now, we can ignore this.
Specify the location of the file we wish to search through (grep will first assume the file is in your current directory unless you tell it otherwise by providing the path to the file i.e.
/path/to/our/logfile.log).
For example, in the terminal below, we are using grep to look through the log file for an IP address. The log file is located in our current working directory, so we do not need to provide a path to the log file - just the name of the log file. Using grep to look in a log file for activity from an IP address
ubuntu@thm:~ grep "192.168.1.30" access.log
192.168.1.30 - - [14/Nov/2022:00:53:07 +0000] "GET / HTTP/1.1" 200 13742
192.168.1.30 - - [14/Nov/2022:00:53:43 +0000] "HEAD"
In the terminal above, we can see two entries in this log file (access.log) for the IP address « 192.168.1.30 ». For reference, we’ve narrowed down two entries from a log file with 469 entries. Our life has already been made easier! Here are some ideas for things you may want to use grep to search a log file for:
A name of a computer.
A name of a file.
A name of a user account.
An IP address.
A certain timestamp or date.
As previously mentioned, we can provide some options to grep to enable us to have more control over the results of grep. The table below contains some of the common options that you may wish to use with grep.
Option |
Description |
Example |
|---|---|---|
-i |
Perform a case insensitive search. For example, « helloworld » and « HELLOWORLD » will return the same results |
|
-E |
Searches using regex (regular expressions). For example, we can search for lines that contain either « thm » or « tryhackme » |
|
-r |
Search recursively. For example, search all of the files in a directory for this value. |
|
Further options available in grep can be searched within grep’s manual page via man grep
Le chall¶
IP address: 10.10.234.225
Username: elfmcblue
Password: tryhackme!
┌──(kali㉿kali)-[~]
└─$ ssh elfmcblue@10.10.234.225
The authenticity of host '10.10.234.225 (10.10.234.225)' can't be established.
ED25519 key fingerprint is SHA256:grRb9iDXKGPHTGW/v6HCIyGjApJfSTFBWpLwrSLfuFs.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.10.234.225' (ED25519) to the list of known hosts.
elfmcblue@10.10.234.225's password:
Bon, comme celle-ci devrait être assez simple et très guidée, on va répondre aux questions.
Question 1 : Use the ls command to list the files present in the current directory. How many log files are present?¶
elfmcblue@day-2-log-analysis:~$ ls -l
total 19396
-rw-r--r-- 1 elfmcblue elfmcblue 223240 Nov 30 13:27 SSHD.log
-rw-r--r-- 1 elfmcblue elfmcblue 19634860 Nov 21 14:56 webserver.log
Question 2 :Elf McSkidy managed to capture the logs generated by the web server. What is the name of this log file?¶
elfmcblue@day-2-log-analysis:~$ ls /var/log/
amazon/ auth.log.1 cloud-init-output.log dmesg dmesg.2.gz dpkg.log kern.log lastlog syslog.1 wtmp
apt/ btmp cloud-init.log dmesg.0 dmesg.3.gz dpkg.log.1 kern.log.1 private/ syslog.2.gz
auth.log btmp.1 dist-upgrade/ dmesg.1.gz dmesg.4.gz journal/ landscape/ syslog unattended-upgrades/
Euh attends, je vois rien là ? J’ai du regarder l’aide : Vous pouvez vous servir de la commande précédente pour trouver le fichier. Ah oui. Il y a deux fichiers de logs dans le répertoire home, pas besoin d’aller aussi loin.
Question 3 :On what day was Santa’s naughty and nice list stolen?¶
N.B Il y avait une validation avant qui indique de regarder dans le fichier de log webserver.
elfmcblue@day-2-log-analysis:~$ cat ./webserver.log
10.9.12.30 - - [18/Nov/2022:12:18:23 +0000] "GET / HTTP/1.1" 200 3036 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
10.9.12.30 - - [18/Nov/2022:12:18:23 +0000] "GET /assets/css/stylesheet.e534de95c45f12e712642d4891fdc622837d0270dd58b129282e0e4b65b5df1a.css HTTP/1.1" 200 4526 "http://10.10.60.160/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
10.10.249.191 - - [18/Nov/2022:12:28:15 +0000] "GET / HTTP/1.1" 200 2980 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:28:15 +0000] "GET /d30f0e6a-9e9c-465a-b4d2-279e8785efde HTTP/1.1" 404 437 "-" "gobuster/3.0.1"
Ok bon, on voit un traînée de requêtes gobuster de la même date qui suivent. D’ailleurs, c’est très intéressant de remarquer à quel point on est visible avec un gobuster. Dans le cas où l’on cherche à être discret, il serait utile de changer ça.
Question 4 :What is the IP address of the attacker?¶
On a la réponse juste au dessus, il faut juste comprendre le formating des logs web. Bien souvent, on a
ip ?? ?? date commande code_retour Taille Referer(?) User-Agent
Question 5 :What is the name of the important list that the attacker stole from Santa?¶
Pour ça, il va falloir se débarsser du lot d’évènements lié à gobuster qui nous flood. On sait un truc : si la liste a fuitée, alors c’est qu’il a pu le récupéré… Donc il doit y avoir un code 200 (ou au moins pas de 404).
elfmcblue@day-2-log-analysis:~$ cat webserver.log | grep -v "404"
10.9.12.30 - - [18/Nov/2022:12:18:23 +0000] "GET / HTTP/1.1" 200 3036 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
10.9.12.30 - - [18/Nov/2022:12:18:23 +0000] "GET /assets/css/stylesheet.e534de95c45f12e712642d4891fdc622837d0270dd58b129282e0e4b65b5df1a.css HTTP/1.1" 200 4526 "http://10.10.60.160/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
10.10.249.191 - - [18/Nov/2022:12:28:15 +0000] "GET / HTTP/1.1" 200 2980 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:28:15 +0000] "GET /assets HTTP/1.1" 301 527 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:28:15 +0000] "GET /categories HTTP/1.1" 301 535 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:28:15 +0000] "GET /tags HTTP/1.1" 301 523 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:34:39 +0000] "GET /santaslist.txt HTTP/1.1" 200 133872 "-" "Wget/1.19.4 (linux-gnu)"
10.10.249.191 - - [18/Nov/2022:12:35:16 +0000] "GET / HTTP/1.1" 200 2980 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:35:16 +0000] "GET /categories HTTP/1.1" 301 535 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:35:16 +0000] "GET /assets HTTP/1.1" 301 527 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:35:16 +0000] "GET /tags HTTP/1.1" 301 523 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:38:05 +0000] "GET / HTTP/1.1" 200 2980 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:38:10 +0000] "GET / HTTP/1.1" 200 2980 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:38:10 +0000] "GET /.htpasswd HTTP/1.1" 403 440 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:38:10 +0000] "GET /.htaccess HTTP/1.1" 403 440 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:38:10 +0000] "GET /index.html HTTP/1.1" 200 2980 "-" "gobuster/3.0.1"
10.10.249.191 - - [18/Nov/2022:12:38:10 +0000] "GET /server-status HTTP/1.1" 403 440 "-" "gobuster/3.0.1"
J’ai filtré sur le 404 car il y avait trop de retour avec le 200.
Question 6 :Look through the log files for the flag. The format of the flag is: THM{}¶
On pouvait s’en douter, ils ont caché un flag pour nous faire tester d’autres commandes de grep. Ici, on peut donc deviner qu’il y a un flag dans un autre fichier. On va grep récursivement !
elfmcblue@day-2-log-analysis:~$ grep -rn THM
webserver.log:99082:10.10.249.191 - - [18/Nov/2022:12:35:20 +0000] "GET /AU7VTHM1YVYV8 HTTP/1.1" 404 437 "-" "gobuster/3.0.1"
SSHD.log:1832:THM{STOLENSANTASLIST}
Et voilà ! Retour demain pour la suite !