Details
-
Bug
-
Status: Open
-
Blocker
-
Resolution: Unresolved
-
1.2.1
-
None
-
None
Description
Hello,
I am currently working on a tool which allows to:
- Take a csv file which contains a table list
- Make a loop which will run through the list of tables and send it to beeline to retrieve the ddl of the tables which will be sent to a file
in my example there are 8 tables and I would like to send 5 beeline requests in the background. As soon as one of the tasks ends, the 6th request is sent to beeline etc etc. As soon as a new request ends, it sends a new request in the background.
There must always be 5 requests at the same time.
When I send the requests, I program well but the terminal bug at the end of the program. I write for example ls, nothing is displayed, i press enter, the command executes and print output.
Do you have an idea?
The code
#!/bin/bash
func_try(){
beelineUsername="xxx"
beelinePassword="xxx"
jdcb_url='xxx'
extra_param='hive.support.concurrency=true;hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;hive.compactor.initiator.on=true;hive.compactor.worker.threads=1;'
request="SHOW CREATE TABLE base.${$1}"
ref_pid=$BASHPID
change=$2
ddl=$( beeline -n "$beelineUsername" -p "$beelinePassword" --fastConnect=true -u "$jdcb_url$extra_param" --silent=true --outputformat="csv2" -f "./resultats/$change.hql" 2>&1)
ddlTable=$(echo "$ddl" > "./pid/${change}_${ref_pid}_ref.txt")
rm "${folder}/${changer}.hql"
}
ARG_PAR=5
folder="./resultats"
rm -f ."/resultats/*.hql"
count=1
while read -r Table;
do
- Get Table to send in beeline
if [ ! -z "$Table" ]
then
nb_file=$(ls ${folder} | wc -l) - This loop is bloking when $ARG_PAR=5 and wait one of beeline request end
- for lauch the next request
while [ $nb_file -eq ${ARG_PAR} ]
do
nb_file=$(ls ${folder} | wc -l)
sleep 5
done
- Create token. This token contain the request to send in beeline
echo "SHOW CREATE TABLE braff00.$Table" > "${folder}/${count}.hql"
func_try "${Table}" "${count}" &
fi
let "count+=1"
done < "./table.txt"