Linux multiple threads download from file with wget
Assume we have a file that have all the urls to download. sed 's|^|https://data.commoncrawl.org/|' wet.paths | xargs -n 1 -P 15 wget --timeout=1800 --retry-connrefused --waitretry=10 -nc &