Thursday, April 4, 2013

Parallel make utility to leverage from multi core processors

By default make utility, which is used for building native code is single threaded, which means that though there can be different targets compiled in parallel from the makefiles, make utility will go in a serial fashion and build targets mentioned in the makefiles.

But if you have multi core processors, you can utilize a handy switch in the make utility to build different targets in parallel.

To get the number of CPUs you have in your machine, use: grep 'processor.*:' /proc/cpuinfo | wc –l

Suppose you have two CPUs, you can command make utility to utilize both the CPUs at the same time and perform two different compilations in parallel. Make utility takes a parameter, -j, which specifies the number of parallel make threads you need. You can say make -j 2 makefiles, which will performs compilations in two parallel make threads.

Now, to tie the make threads to the number of CPUs, we can use:

make -f makefile -j `grep 'processor.*:' /proc/cpuinfo | wc -l` -k buildall

This can utilize the full true computation power of your machine and complete compilation in less time.


Drawback:

All separate make threads write output messages to the same output stream, so in your terminal, you will see jumbled messages from all parallel make threads. This could be a problem for logs, but for developers, who just want only the compilation to be done fast and not worried about the logs, this is cool.