I already said about the wonders of xargs before
I recovered a discussion I had with a friend about it, and back them I provided him with a neat way to see parallelism. I thought to annotate it here.
Consider the following terminal session, in which the consumer.sh script
will emulate some processing. I pipe seq 1 10 in xargs with and without
the -L flag. Look for the timestamps :-)
$ cat <<EOF >consumer.sh
#!/bin/sh
for arg; do
printf '%s %s: %s\n' "$(date)" $$ "$arg"
sleep 1 # fake processing
done
EOF
$ # Using xargs without -L1
$ seq 1 100 | xargs -P3 sh ./consumer.sh
Wed Jul 31 19:16:59 CEST 2019 19972: 1
Wed Jul 31 19:17:00 CEST 2019 19972: 2
Wed Jul 31 19:17:01 CEST 2019 19972: 3
Wed Jul 31 19:17:02 CEST 2019 19972: 4
Wed Jul 31 19:17:03 CEST 2019 19972: 5
^C
$ # Using xargs with -L1
$ seq 1 100 | xargs -L1 -P3 sh ./consumer.sh
Wed Jul 31 19:17:17 CEST 2019 19998: 2
Wed Jul 31 19:17:17 CEST 2019 19997: 1
Wed Jul 31 19:17:17 CEST 2019 19999: 3
Wed Jul 31 19:17:18 CEST 2019 20006: 4
Wed Jul 31 19:17:18 CEST 2019 20007: 5
Wed Jul 31 19:17:18 CEST 2019 20008: 6
Wed Jul 31 19:17:19 CEST 2019 20015: 7
Wed Jul 31 19:17:19 CEST 2019 20017: 8
Wed Jul 31 19:17:19 CEST 2019 20018: 9
Wed Jul 31 19:17:20 CEST 2019 20024: 10
Wed Jul 31 19:17:20 CEST 2019 20026: 11
Wed Jul 31 19:17:20 CEST 2019 20027: 12
Wed Jul 31 19:17:21 CEST 2019 20033: 13
Wed Jul 31 19:17:21 CEST 2019 20034: 14
Wed Jul 31 19:17:21 CEST 2019 20036: 15
^C
If we don't add -L1 we use only one process, since there's no obvious way
to split the (possibly infinite) output for the three (-P3) processes.
With -L1 we tell xargs to send one line per invocation, so we always
have three processes at the time, as demonstrated by pids and timestamps)