Parallelization – is it faster?

I have earlier written about parallelizing I/O. It is not that surprising that you can overload your disk with parallel requests and get worse I/O than serial access. What may be more surprising is that parallelizing CPU intensive tasks may not benefit from hyperthreading.

# Compress /dev/zero for 5 seconds
doit() { seq $1 |parallel -N0 –timeout 5 -j$1 -u zstd ‘<‘ /dev/zero | wc -c ; }
export -f doit
seq $(parallel –number-of-threads) | parallel -j1 –shuf –tag doit :::: – ::: {1..5} 2>/dev/null

This computes how much data `zstd` generates when run for 5 seconds. It does this with different number of jobs in parallel. On my system with 4 cores/8 threads you might expect a big increase from 1 to 4 jobs in parallel and a small increase from 5 to 8 jobs.

That is not what is happening.

I see the big increase from 1 to 4 jobs in parallel (as expected), but more than 4 jobs in parallel is actually slower than running 4 in parallel.

On my 2 core/4 thread machine the performance increases until 3 jobs are run in parallel – both 2 and 4 jobs in parallel are slower.

`zstd` is not a representative tool – normally you will get an increase in performance up to the number of threads – less and less increase for each thread. `gzip` behaves more like one would expect:  On my 2 core/4 thread machine gives 1 job = 50%, 2 jobs = 75%, 3 jobs = 90%, 4 jobs in parallel = 100%.

 

Advertisements
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s