Troubleshooting Containers with Sysdig Inspect

This is the third part of our series about Sysdig. The first tutorial walked you through the process of capturing and analyzing data with the Sysdig command-line utility. The second tutorial showed how to use Csysdig, the ncurses based GUI. Now, we’ll focus on explaining how you can troubleshoot your containers with Sysdig Inspect.

Prerequisites

Before starting this tutorial, you should follow the first part of the series. We assume that Docker Compose and Sysdig are installed on your system. Also, you should have a running WordPress installation. Enter the following command to verify that everything is properly set up:

docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                  NAMES
f2a5bf941eba        wordpress:latest    "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:8000->80/tcp   wordpress-sysdig_wordpress_1
95c44dd63ba9        mysql:5.7           "docker-entrypoint.s…"   About a minute ago   Up About a minute   3306/tcp, 33060/tcp    wordpress-sysdig_db_1
8405c2c87f59        sysdig/sysdig       "/docker-entrypoint.…"   About a minute ago   Up About a minute                          sysdig

Make sure the following three Docker containers are running on your system:

• WordPress
• MySQL
• Sysdig

Install Sysdig Inspect

There are two ways in which you can install Sysdig inspect:

• Download the pre-built binary for your operating system from the project’s GitHub repository.
• Run it as a Docker container.

For this tutorial, you will install Sysdig Inspect inside of a Docker container.

In this section, you’ll install Sysdig Inspect and make it so that the trace files captured with sysdig can be accessed from the Sysdig Inspect container.

1. In a new terminal window (on the host), run the following command to create a directory called captures:

mkdir captures

2. You can install Sysdig Inspect in a container with the following docker run command:

docker run -d -v /home/vagrant/captures/:/captures -p8080:3000 sysdig/sysdig-inspect:latest

Unable to find image 'sysdig/sysdig-inspect:latest' locally
latest: Pulling from sysdig/sysdig-inspect
f661dfa5eb7d: Pull complete 
572ec58ac70f: Pull complete 
c1d2c84e7140: Pull complete 
1ebe1ad7056b: Pull complete 
e5e1416d36d0: Pull complete 
9b379815df57: Pull complete 
9034f4cfeb96: Pull complete 
91638da20eef: Pull complete 
a4105e889626: Pull complete 
b7359900512b: Pull complete 
2daad7eb7645: Pull complete 
d0fb8981639d: Pull complete 
ed8bc4c05912: Pull complete 
29d1d8034bab: Pull complete 
fead5f9a4603: Pull complete 
Digest: sha256:0c6270a09cfb28f2c248761f4783d0bc53d3dfd9b83d27f03b98039d5366cf45
Status: Downloaded newer image for sysdig/sysdig-inspect:latest
ee88efbc56205164a269102d266509ce5a677d024071d29774b12cd3414a7a69

A few things to note about the above command:

• The -v flag specifies the files on the host that Sysdig Inspect can access. In this example, the files placed inside of the /home/vagrant/captures/ directory will be available in the captures folder in the container running Sysdig Inspect.
• The -d flag starts the container in detached mode
• The -p flag publishes the 3000 port to the 8000 port on the host.

3. At this point you can inspect the status of the Sysdig Inspect container with:

docker ps

Note that the status is currently set to starting. Wait a bit until it changes to healthy:

4. Now you can point your browser to

   http://localhost:8080

   :

Capture Data

As you can see in the screenshot above, you can specify the path to a capture file taken with Sysdig. In this section, you’ll create a new trace file, simulate an issue by deleting a file, and then load the data into Csysdig Inspect for inspection.

1. In a new terminal window (on the host), display the list of your active containers:

docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
7da3fee501db        sysdig/sysdig-inspect:latest   "/docker-entrypoint.…"   2 minutes ago       Up 8 minutes (healthy)   0.0.0.0:8080->3000/tcp   agitated_leakey
f2a5bf941eba        wordpress:latest    "docker-entrypoint.s…"   8 minutes ago       Up 8 minutes        0.0.0.0:8000->80/tcp   wordpress-sysdig_wordpress_1
95c44dd63ba9        mysql:5.7           "docker-entrypoint.s…"   8 minutes ago       Up 8 minutes        3306/tcp, 33060/tcp    wordpress-sysdig_db_1
8405c2c87f59        sysdig/sysdig       "/docker-entrypoint.…"   9 minutes ago       Up 9 minutes                               sysdig

2. Start a shell inside of the wordpress container by entering the following docker exec command and passing the identifier of the container:

docker exec -ti f2a5bf941eba /bin/bash

☞ In the above example, the identifier of our container is f2a5bf941eba but yours will be different.

3. Similarly, start a shell inside of the csysdig container with:

docker exec -ti 8405c2c87f59 /bin/bash

4. Initiate a new capture by entering the following command inside of the sysdig container:

sysdig -w monitoring-wordpress-error.scap

5. Open a new terminal window (on the host), and use ApacheBench to make 10000 requests with a maximum of 100 requests running concurrently:

ab -n 10000 -c 100 http://localhost:8000/

6. While ApacheBench is running, move back to the wordpress container and delete the index.php file:

rm -rf index.php

7. When the benchmark is finished, move to the sysdig container and stop capturing data by entering “CTRL+C”.

8. If everything went well, a file named monitoring-wordpress-error.scap should have been created:

ls -la monitoring-wordpress-error.scap

-rw-r--r--. 1 root root 146795720 Jan 16 20:56 monitoring-wordpress-error.scap

9. Switch back to the host, and use the docker cp command to copy the capture file to the captures directory:

docker cp 8405c2c87f59:/monitoring-wordpress-error.scap captures/

10. Verify whether the file has been successfully copied to the host with:

ls -la captures/

total 143356
drwxrwxr-x. 2 vagrant vagrant        45 Jan 16 21:02 .
drwx------. 8 vagrant vagrant       208 Jan 16 18:27 ..
-rw-r--r--. 1 vagrant vagrant 146795720 Jan 16 20:56 monitoring-wordpress-error.scap

11. Point your browser to

    http://localhost:8080

    and type the path to the trace file (that is /captures/monitoring-wordpress-error.scap) in the “capture file path” text area:

12. Press the Enter key and then wait a bit while Sysdig Inspect loads the trace file:

13. When everything is ready, you will see the overview page:

How to Navigate the Interface

In this section, we’ll explain the basic concepts behind Sysdig’s Inspect interface. Once the trace file is loaded, you’ll be presented with a screen similar to the one below (the annotations are ours):

There’s a lot of information packed in here. Let’s break this down:

(A) At the top of the screen, Sysdig Inspect shows the name of the currently open trace file.

(B) The drill-down section shows the list of active filters. Select the “Containers” view and then navigate to the “Processes view”:

☞ You can customize what data is displayed by manually editing the “Sysdig Filter” textbox.

(C) Similar to how Csysdig works, Sysdig Inspect lets you filter the data using several built-in views.

(D) The main window contains several tiles. Each tile shows the values of a particular metric and its evolution in the period of time in which the capture was taken. Tiles are grouped together to make navigation easier.

(E) The timeline allows you to use two sliders to narrow down the period during which the data is analyzed. Continuing our previous example in which we selected the “Container” view and then “Navigated” to the process view, we used the sliders to narrow down the period of time for which the data is displayed:

We’ve covered the basics of using the interface! In the next section, we’ll show how you can use Sysdig Inspect to troubleshoot your containers.

Put It to the Test

Remember that, in the “Capture Data” section, we’ve purposely introduced an error. Now, we’ll make use of that to showcase how you can use Sysdig Inspect to analyze data and troubleshoot your containers.

1. First, navigate to the overview page:

2. From the “Views” section, select “Errors”. You’ll see a high number of ENOENT (which stands for Error NO ENTry) errors:

3. Highlight the ENOENT line by clicking on it. This will display three round buttons:

• Drill-Down
• I/O Streams
• Syscalls

4. Select “Drill-down’ and Sysdig Inspect will show the list of processes that raised this error:

5. Pick the one from top of the list. Then, select “Syscalls”. Since we’ve deleted the index.php file a bit later in the capture, you must scroll down a few lines until you see lots of error messages similar to the ones below:

☞ Note that the Sysdig Filter checkbox has been automatically updated.

Now, this sheds some light, but it doesn’t tell you what has happened.

6. Let’s clear all the filters and navigate to the default view by selecting “Overview”:

7. See the “Security” section. It shows that two commands were executed, and one file was deleted.

8. To get more details, select the Deleted Files tile:

Note that the Drill-down button is now displayed. Also, the moment when the file was deleted is shown at the bottom of the page.

9. Select the Drill-down button, and Sysdig Inspect will show that the index.php file was deleted:

10. If you select the Executed commands -> Drill-down tile, then you’ll see that the root user deleted the index.php file by running the rm -f command:

As you can see, Sysdig Inspect displays a host of useful information you can rely on to troubleshoot your containers.

---

In this example, it wasn’t overly difficult to figure out what caused the issue, but there may be cases in which you’ll have to dig a bit deeper. The knowledge you acquired is a complete tool belt that’ll help you get through in any situation that requires troubleshooting your container-based applications.

Thanks for reading!