Logging and Debugging QEMU Virtual Machines

Last updated: June 22, 2024

Reviewed by: Korbin Brown

1. Introduction

QEMU (Quick Emulator) is a powerful open-source emulator and virtualizer widely used to run virtual machines (VMs). Whether we’re managing virtual servers, testing software, or simulating different environments, QEMU’s versatility is unmatched. However, like any virtual environment, VMs can experience issues. This is where logging and debugging become crucial.

In this tutorial, we’ll explore various logging options available in QEMU. We’ll also look into debugging techniques to identify and resolve problems efficiently, from basic logging setups to advanced debugging methods. Let’s get started!

2. Setting up QEMU Logging

Logging is a vital aspect of virtual machine management. It helps us track the VM’s behavior, diagnose issues, and maintain a stable environment. QEMU provides several logging mechanisms that we can use to capture valuable information about the VM’s operation.

Specifically, QEMU offers various command-line options to enable logging. These options allow us to specify the type and level of information we want to log.

Let’s explore some logging setups.

2.1. Using the -D Option for Logging

The -D option is the simplest way to enable logging in QEMU. This option lets us specify a file where QEMU will write its log messages.

With the -D option, we start a QEMU VM with basic logging enabled:

$ qemu-system-i386 -D ./qemu-debug-log -monitor pty -device e1000,netdev=tap0 -netdev tap,id=tap0 -m 512M -display vnc=:1 -drive file=FreeBSD10.2

Let’s understand our interaction here:

  • qemu-system-i386 – launches the QEMU emulator for an i386 architecture
  • -D ./qemu-debug-log – directs QEMU to write log messages to the file qemu-debug-log in the current directory
  • -monitor pty – sets up a monitor using a pseudo-terminal
  • -device e1000,netdev=tap0 – adds an e1000 network device connected to tap0
  • -netdev tap,id=tap0 – configures a TAP network backend with ID tap0
  • -m 512M – allocates 512 MB of memory to the VM
  • -display vnc=:1 – enables VNC (Virtual Network Computing) display on port 5901
  • -drive file=FreeBSD10.2 – specifies the disk image file to use

When we run the above command, QEMU creates a file qemu-debug-log in the current directory. This file contains log messages generated by QEMU during the VM’s operation.

2.2. Viewing the Log File

Let’s see what the contents of our sample qemu-debug-log file might look like:

# Viewing the qemu-debug-log File

qemu-system-i386: -drive file=FreeBSD10.2: could not open disk image FreeBSD10.2: No such file or directory
qemu-system-i386: -netdev tap,id=tap0: could not configure /dev/net/tun (tap0): No such device
qemu-system-i386: -device e1000,netdev=tap0: Device 'e1000' could not be initialized
VNC server running on ::1:5901

Let’s break down this log file’s content:

  • qemu-system-i386: -drive … – indicates that the specified disk image file is missing or the path is incorrect
  • qemu-system-i386: -netdev … – QEMU attempted to configure a TAP network device with the ID tap0, but it failed because the TAP device doesn’t exist, which may happen if the TAP network interface isn’t properly set up on the host machine
  • qemu-system-i386: -device … – QEMU tried to add an e1000 network device linked to tap0, but initialization failed, likely due to the previous network configuration error
  • VNC server … – despite the previous errors, QEMU successfully started a VNC server on port 5901, which is accessible using a VNC viewer

Reviewing these log entries, we can diagnose and address the issues preventing the VM from starting correctly. For instance, we need to ensure the disk image file exists at the specified path and that we configure the TAP network device properly.

2.3. Specifying Log File Location

We can also specify a different location for the log file:

$ qemu-system-i386 -D /tmp/qemu-debug-log -monitor...

In this case, QEMU writes the log messages to /tmp/qemu-debug-log.

2.4. Using the -d Option for Component-Specific Logging

For more detailed logging, QEMU provides the -d option, which enables logging for specific virtual machine components. This logging level is useful when we need to diagnose specific issues or gain deeper insights into the VM’s operations.

Let’s see an example of logging CPU resets:

$ qemu-system-i386 -d cpu_reset -D ./cpu_reset.log -monitor pty -device e1000,netdev=tap0 -netdev tap,id=tap0 -m 512M -display vnc=:1 -drive file=FreeBSD10.2

Here, -d cpu_reset enables logging for CPU reset events. Then, -D ./cpu_reset.log – specifies the log file for the detailed log messages.

In short, this command generates a cpu_reset.log file in the current directory containing detailed logs of CPU reset events.

2.5. Combining Logging Options for Detailed Debugging

Importantly, we can use the -d option to combine various other components:

  • in_asm – logs each instruction executed by the CPU
  • exec – logs code translations
  • int – logs interrupts
  • op – logs micro-operations

For instance, we can log CPU instructions and interrupts together:

$ qemu-system-i386 -d in_asm,int -D ./detailed.log -monitor pty -device e1000,netdev=tap0 -netdev tap,id=tap0 -m 512M -display vnc=:1 -drive file=FreeBSD10.2

Here, we log CPU instructions and interrupts into the ./detailed.log file.

Combining logging options is particularly useful when diagnosing complex issues involving multiple components. For instance, if we’re troubleshooting performance problems, logging instructions (in_asm) and interrupts (int) together can provide a clearer picture of the VM’s behavior.

3. Utilizing QEMU Monitor for Logging and Debugging

The QEMU Monitor is a powerful tool that allows us to interact with the VM while it’s running. We can use it to enable additional logging, manage the VM’s state, and execute various commands for debugging purposes.

Specifically, the QEMU Monitor provides a command-line interface to control the VM.

To start QEMU with the monitor, we use the -monitor stdio option:

$ qemu-system-i386 -monitor stdio -device e1000,netdev=tap0 -netdev tap,id=tap0 -m 512M -display vnc=:1 -drive file=FreeBSD10.2

When we start QEMU with -monitor stdio, the QEMU Monitor interface appears in the terminal.

Here’s what the QEMU Monitor interface looks like:

QEMU 6.0.0 monitor - type 'help' for more information
(qemu)

This prompt indicates that the QEMU Monitor is ready to accept commands.

3.1. Enabling Logging

Once the VM is running, we can use the QEMU Monitor to enable logging.

For example, to enable CPU logging, we use:

(qemu) log_cpu on

This command enables the logging of CPU events.

3.2. Monitor Commands for Logging

Here are some useful monitor commands for logging:

  • log_cpu on – enables logging of CPU events
  • info log – displays current logging settings
  • logfile <filename> – changes the log file

With these commands, we can easily control and manage logging settings, which helps us effectively debug and analyze our virtual machine’s behavior.

3.3. Viewing and Managing Logs Through the Monitor

The QEMU Monitor also allows us to manage logs. We can change the log file or view the current logging configuration without restarting the VM. This flexibility is invaluable for on-the-fly debugging and analysis.

For instance, we can decide to change the log file while the VM is running:

(qemu) logfile /tmp/new-logfile.log

This command redirects all future log messages to /tmp/new-logfile.log.

4. Analyzing and Interpreting QEMU Logs

After setting up logging, we can analyze and interpret the log files to help us understand the VM’s behavior and identify potential issues.

QEMU log files contain various types of entries, each providing different information about the VM’s operations.

Let’s review some common types of log entries:

  • CPU instructions: logs of each instruction the CPU executes when in_asm logging is enabled
  • Interrupts: details of interrupt events when int logging is enabled
  • Resets: information about CPU resets when cpu_reset logging is enabled

To diagnose issues, we need to correlate log entries with the problems we’re experiencing.

For example, if a VM crashes frequently, we’ll want to look for patterns in the log entries before each crash.

4.1. Diagnosing a VM Crash

Let’s walk through a practical step-by-step example of diagnosing a VM crash:

  1. Identifying the problem: Determine the symptoms, such as frequent crashes or performance issues.
  2. Enabling detailed logging: Use specific logging options like -d cpu_reset, int.
  3. Reproducing the issue: Run the VM and trigger the problem.
  4. Analyzing the logs: Look for abnormal log entries around the time of the issue.
  5. Identifying patterns: Check for recurring entries or specific events leading to the problem.

Let’s see a sample log file demonstrating a VM crash:

# qemu-debug-log File

2024-06-04 10:15:30.123456: CPU Reset requested.
2024-06-04 10:15:30.123789: CPU Reset complete.
2024-06-04 10:15:31.987654: Interrupt 0x20 received.
2024-06-04 10:15:31.987678: Interrupt 0x21 received.
2024-06-04 10:15:32.123456: CPU Reset requested.
2024-06-04 10:15:32.123789: CPU Reset complete.
2024-06-04 10:15:33.654321: Interrupt 0x20 received.
2024-06-04 10:15:34.000000: VM crashed.

In this sample log file, we observe multiple CPU resets and interrupt entries just before the VM crash.

4.2. What to Do Next

After identifying patterns in the log files that correlate with the VM crashes or other issues, we can take steps to resolve the problem:

  1. Research and troubleshoot: We can use the information from the logs to research potential causes or look up the error messages.
  2. Update configuration: We can adjust the VM’s configuration to address the identified issues by changing hardware emulation settings, updating network configurations, or increasing memory allocation.
  3. Test changes: After making configuration changes, we should restart the VM and attempt to reproduce the issue. We should also monitor the logs to verify whether the problem persists or if the changes have resolved it.
  4. Apply patches and updates: We should ensure that QEMU and any related software (such as the guest operating system) are up to date.

To address future issues quickly, we should implement regular monitoring and logging. Consistent log analysis helps maintain the virtual environment’s stability and performance.

5. Debugging Techniques in QEMU

Debugging involves more than just reading logs. We need to actively diagnose and resolve issues using various tools and techniques.

Typically, debugging scenarios include VM crashes, performance issues, and hardware emulation problems. Each requires a different approach and set of tools.

5.1. Using GDB With QEMU for In-Depth Debugging

The GNU Debugger (GDB) is a powerful tool we can use to debug QEMU VMs. We can attach GDB to a running VM to inspect its state and control its execution.

To start QEMU with GDB support, we use the -s -S options:

$ qemu-system-i386 -s -S -monitor stdio -device e1000,netdev=tap0 -netdev tap,id=tap0 -m 512M -display vnc=:1 -drive file=FreeBSD10.2

Here, -s starts QEMU in GDB server mode, listening on port 1234. Then, -S freezes the CPU at startup until GDB attaches.

5.2. Attaching GDB to the QEMU VM

After starting QEMU with GDB server mode, we can now attach GDB to the VM:

$ gdb

This opens the GDB interface.

Next, we connect to the QEMU instance:

(gdb) target remote localhost:1234

Here, we connect GDB to the QEMU instance running on localhost at port 1234.

Notably, here are some basic GDB commands we can execute:

  • continue – resumes execution of the VM
  • break <location> – sets a breakpoint at the specified location
  • info registers – displays the current state of the CPU registers
  • step or next – executes the next instruction

By effectively analyzing logs and employing debugging techniques, we can resolve various issues affecting our QEMU VMs.

6. Integrating QEMU Logs With Syslog

Syslog is a standard for logging program messages, and we can use it to collect logs from various sources, including QEMU.

To enhance our logging and monitoring capabilities, we should consider integrating QEMU logs with an external system like Syslog. This would allow centralized logging, easier management, and better data analysis.

Let’s see an example setup with rsyslog for FreeBSD.

6.1. Installing rsyslog

First, we need to install rsyslog.

On FreeBSD, we use the pkg command:

$ pkg install rsyslog

This command installs the rsyslog package on FreeBSD.

6.2. Configuring rsyslog to Accept Remote Logs

After installation, we configure rsyslog to accept remote logs.

To do this, we need to edit the /usr/local/etc/rsyslog.conf file and add our configuration:

# Editing the /usr/local/etc/rsyslog.conf FIle
...
module(load="imudp")
input(type="imudp" port="514")

In this configuration, module(load=”imudp”) loads the UDP input module. Then, input(type=”imudp” port=”514″) configures rsyslog to listen for UDP logs on port 514.

We add these lines to the configuration file to enable rsyslog to accept remote logs.

6.3. Configuring QEMU to Send Logs to the Syslog Server

Now, we configure QEMU to send logs to the Syslog server.

For this, we edit the QEMU command line to include -D syslog:local1:

$ qemu-system-i386 -D syslog:local1 -monitor pty -device e1000,netdev=tap0 -netdev tap,id=tap0 -m 512M -display vnc=:1 -drive file=FreeBSD10.2

Here, -D syslog:local1 sends log messages to the local Syslog facility local1.

Notably, local1 is one of the local Syslog facilities (ranging from local0 to local7) we can use for custom logging purposes.

Finally, we restart rsyslog to apply changes:

$ service rsyslog restart

We can now check the Syslog log file, typically at /var/log/messages, to verify that logs are being sent to Syslog.

With this, we have a central source for logging, facilitating easier management and better data analysis.

7. Conclusion

In this article, we explored various mechanisms for logging and debugging QEMU virtual machines. We covered setting up basic logging with the -D option, the -d option for component-specific logging, and the QEMU Monitor for dynamic logging.

Also, we discussed analyzing and interpreting QEMU logs, providing practical examples for diagnosing issues. Additionally, we looked into debugging techniques using GDB to gain deeper insights into VM operations.

Finally, we covered integrating QEMU logs with an external system like Syslog for centralized logging and monitoring.

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments