I've got a whole heap of files on a server, and I want to upload these onto S3. The files are stored with a .data extension, but really they're just a bunch of jpegs,pngs,zips or pdfs.
I've already written a short script which finds the mime type and uploads them onto S3 and that works but it's slow. Is there any way to make the below run using gnu parallel?
#!/bin/bash
for n in $(find -name "*.data")
do
data=".data"
extension=`file $n | cut -d ' ' -f2 | awk '{print tolower($0)}'`
mimetype=`file --mime-type $n | cut -d ' ' -f2`
fullpath=`readlink -f $n`
changed="${fullpath/.data/.$extension}"
filePathWithExtensionChanged=${changed#*internal_data}
s3upload="s3cmd put -m $mimetype --acl-public $fullpath s3://tff-xenforo-data"$filePathWithExtensionChanged
response=`$s3upload`
echo $response
done
Also I'm sure this code could be greatly improved in general :) Feedback tips would be greatly appreciated.
You are clearly skilled in writing shell, and extremely close to a solution:
s3upload_single() {
n=$1
data=".data"
extension=`file $n | cut -d ' ' -f2 | awk '{print tolower($0)}'`
mimetype=`file --mime-type $n | cut -d ' ' -f2`
fullpath=`readlink -f $n`
changed="${fullpath/.data/.$extension}"
filePathWithExtensionChanged=${changed#*internal_data}
s3upload="s3cmd put -m $mimetype --acl-public $fullpath s3://tff-xenforo-data"$filePathWithExtensionChanged
response=`$s3upload`
echo $response
}
export -f s3upload_single
find -name "*.data" | parallel s3upload_single