Posted on

Run a job in the background on Linux instance

When we are working on a remote Linux instance, there are a lot of times that we need to run some intensive computation Job/Process on the instance, which are time-consuming. The Job/Process can be run for more than an hour or even a day, and the struggle connecting can ruin the whole process. I had that issue a few years ago that I need to run some experiments for more than a day and I always stuck with the message “Write Failed: broken pipe” by unstable connection. Therefore, in this post, I want to share a solution to that issue. On the one hand, you can increase the timeout of ssh to avoid the connection lost issue. However, that is not a good solution to the problem, power or the Internet outage can still ruin your process. There is a special command for this task.

On the Linux system, there is a command called “nohup“, which allows Job/Process to able to run in the background (even we are logged off). The usage of this command is very simple. We just need to run “nohup” with our regular command. 

$nohup ./testRunning "argv1" "argv2" ... &

Note that the last symbol (&) is required. This will set the system to run the process even we are logged off. However, we might want to keep the output (stdout and stderr) from the process. We can also save the result to output file by the following command

$nohup ./testRunning "argv1" "argv2" ... > out.txt 2>&1 &

The command will redirect the output from both stdout and stderr to a filename “out.txt”. The first “>” symbol pipeline stdout to filename out.txt and the second symbol “2>&1” append stderr (“2”) to stdout (“1”).

With nohup command, you don’t need to keep worrying about the connection to the instant and you don’t need to keep your machine login to the instant ( Safe some energy ( ͡❛ ͜ʖ ͡❛) ).

Posted on

Segmentation Fault? Valgrind can detect that

Since this is the first post on my blog, I want to start with a simple thing, but it is an annoying problem in the software development process. An error knows as a segmentation fault or memory corruption. Recently, I had found a project that has memory corruption in linux system, and the software was randomly crash. The segmentation fault is not a difficult bug to solve if we can identify where it is. However, in my case, the crash was random, and the project was large (line by line checking is out of an option).

I did some research online and found an interesting tool on a Linux system, which is called “Valgrind” (https://valgrind.org). Valgrind is a dynamic analysis tool that can detect memory management and threading bugs. Valgrind can also generate deep profile programs for performance analysis. However, in this post, we will focus on the debugging part.

By simply call Valgrind with the program, the tool can identify the line of code that has illegal access to a memory address. Valgrind can detect all of the major memory access issues; e.g., out of range read, out of range write, uninitialized access memory.

Let’s look into an example of using Valgrind, here is the example of a c++ program that has memory corruption.

//Filename Test.cpp
#include <iostream>

int main() {
    int *x = new int[10];
    x[12] = 0; // out of range write
}

As we can see, the program contains out of range writing error. Then we compile the program with the “-g” flag and run the program by Valgrind. Note that the “-g” parameter will generate source-level debugging information.

$gcc -g test.cpp 
$valgrind ./a.out
==23101== Memcheck, a memory error detector
==23101== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==23101== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==23101== Command: ./a.out
==23101== 
==23101== Invalid write of size 4
==23101==    at 0x1087A8: main (test.cpp:6)
==23101==  Address 0x5b7dcb0 is 8 bytes after a block of size 40 alloc'd
==23101==    at 0x4C3089F: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23101==    by 0x10879B: main (test.cpp:5)
==23101== 
.
.
.

Valgrind will show the output of the analysis. As we can see from our example, Valgrind detects Invalid write of 4 bytes and at line number 6 (test.cpp:6).

Bonus point, Valgrind can also search for memory leak in the program. We just need to add an option of “—leak-chek=yes” as an example below.

$valgrind --leak-check=yes ./a.out
.
.
.
==23152== HEAP SUMMARY:
==23152==     in use at exit: 40 bytes in 1 blocks
==23152==   total heap usage: 2 allocs, 1 frees, 72,744 bytes allocated
==23152== 
==23152== 40 bytes in 1 blocks are definitely lost in loss record 1 of 1
==23152==    at 0x4C3089F: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23152==    by 0x10879B: main (test.cpp:5)
==23152== 
==23152== LEAK SUMMARY:
==23152==    definitely lost: 40 bytes in 1 blocks
==23152==    indirectly lost: 0 bytes in 0 blocks
==23152==      possibly lost: 0 bytes in 0 blocks
==23152==    still reachable: 0 bytes in 0 blocks
==23152==         suppressed: 0 bytes in 0 blocks

As we can see Valgrind is a great tool for analyzing the memory and detect the segmentation fault. It can detect most of the memory errors/bugs with a little effort from the developer.