How to run command or code in parallel in bash shell under Linux or Unix
http://ift.tt/2q8BulH
ow do I run commands in parallel in a bash shell script running under Linux or Unix-like operating system? How can I run multiple programs in parallel from a bash script?
You have various options to run programs or commands in parallel:
=> Use GNU/parallel or xargs command.
=> Use wait built-in command with &.
=> Use xargs command.
Putting jobs in background
The syntax is:
command &
command arg1 arg2 &
custom_function &
OR
prog1 &
prog2 &
wait
prog3
In above code sample, prog1, and prog2 would be started in the background, and the shell would wait until those are completed before starting the next program named progr3.
Examples
In this following example run sleep command in the background:
$ sleep 60 &
$ sleep 90 &
$ sleep 120 &
To displays status of jobs in the current shell session run jobs command as follows:
$ jobs
Sample outputs:
[1] Running sleep 60 & [2]- Running sleep 90 & [3]+ Running sleep 120 &
Let us write a simple bash shell script:
#!/bin/bash # Our custom function cust_func(){ echo "Do something $1 times..." sleep 1 } # For loop 5 times for i in {1..5} do cust_func $i & # Put a function in the background done ## Put all cust_func in the background and bash ## would wait until those are completed ## before displaying all done message wait echo "All done" |
#!/bin/bash
# Our custom function
cust_func(){
echo “Do something $1 times…”
sleep 1
}
# For loop 5 times
for i in {1..5}
do
cust_func $i & # Put a function in the background
done## Put all cust_func in the background and bash
## would wait until those are completed
## before displaying all done message
wait
echo “All done”
Let us say you have a text file as follows:
$ cat list.txt
Sample outputs:
http://ift.tt/2piFZq2 http://ift.tt/2pKEy69 http://ift.tt/2pj1KWo http://ift.tt/2pKCco9 http://ift.tt/2piQTMf http://ift.tt/2pKvOgu http://ift.tt/2piKKjd http://ift.tt/2pKAL96 http://ift.tt/2piLzIT http://ift.tt/2pKlQfe http://ift.tt/2piXVAI http://ift.tt/2pKvtKT
To download all files in parallel using wget:
#!/bin/bash # Our custom function cust_func(){ wget -q "$1" } while IFS= read -r url do cust_func "$url" & done < list.txt wait echo "All files are downloaded." |
#!/bin/bash
# Our custom function
cust_func(){
wget -q “$1”
}while IFS= read -r url
do
cust_func “$url” &
done < list.txtwait
echo “All files are downloaded.”
GNU parallel examples
From the GNU project site:
GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables.
The syntax is pretty simple:
parallel ::: prog1 prog2
For example, you can find all *.doc files and gzip (compress) it using the following syntax:
$ find . -type f -name '*.doc' | parallel gzip --best
$ find . -type f -name '*.doc.gz'
Our above wget example can be simplified using GNU parallel as follows:
$ cat list.txt | parallel -j 4 wget -q {}
OR
$ parallel -j 4 wget -q {} < list.txt
See also
Linux
Linux
via [RSS/Feed] nixCraft: Linux Tips, Hacks, Tutorials, And Ideas In Blog Format http://ift.tt/Awfi7s
May 5, 2017 at 06:02PM
Deprecated: Function get_magic_quotes_gpc() is deprecated in /srv/marcgottlieb.com/wp-includes/formatting.php on line 4826
Deprecated: Function get_magic_quotes_gpc() is deprecated in /srv/marcgottlieb.com/wp-includes/formatting.php on line 4826
Deprecated: Function get_magic_quotes_gpc() is deprecated in /srv/marcgottlieb.com/wp-includes/formatting.php on line 4826