更容易的方法来比较变量

Question

我需要比较文件'tmpcsv2'中的一组变量与'uniq_id'中的变量，我详细说明下面的文件。

tmpcsv2 - >该文件被另一个脚本'script1'更新，'script1'的每次运行都会更新（不附加）'tmpcsv2'中的新变量。没有。变量可能是1，可以去钱柜200

eg: 
2042344352 
2470697747 
2635527510 
3667769962 

uniq的-ID - >这是一个固定的变量（约100K中没有。）

(Business Name,Job ID,Job Size) 
biz,1000036446,225210640 
biz,100006309,6710840 
biz,1000069211,2084019000 
biz,1000118720,34194040 
biz,1000150241,212322636 

我使用“的'循环+'如果'比较它们如下所示，是否有这样做更容易或更快（影响更小）的方式？当我运行它时，需要很长时间才能输出结果。打印命令仅用于测试，稍后将被删除！

****Part of a bigger script**** 
amt=0 
mjc=0 
for jbid in `cat tmpcsv2` #Pick ID for match & calculation 
do 
    printf "Checking ID $jbid\n" >> Acsv3.tmp 
    for bsid in `cat uniq_id` #Matching jobs & size calulation 
    do 
     ckid=`echo $bsid | cut -d "," -f2` #ckid is the ID to check 
     jbsiz=`echo $bsid | cut -d "," -f3` #size of the ID 
     if [ $jbid == $ckid ] 
     then 
      printf "Matched at $ckid\n" #Print on Match found 
      printf "Valid -> $jbid\n" >> Bcsv3.tmp 
      ((mjc++)) #Increment Matched Job Count 
      amt=$((amt+jbsiz)) #Add size of matched jobs 
      break 
     else 
      printf "No Match at $cksid\n" #No matches 
     fi 
    done 
    printf "Check for ID $jbid done\n" >> Acsv3.tmp 
    printf "Matched $mjc jobs with combined size of $amt\n" >> Acsv3.tmp 
done 
****End of Comparision**** 

Answer 1

shell是用于处理这么多数据的错误工具，但它是可行的。这里最基本的错误是reading lines with for。通过在每次迭代中不重新打开文件可以显着提高性能。

function main { 
    # Variables used elsewhere should be initialized there, not localized here. 
    typeset amt=0 mjc=0 jbid ckid jbsiz 

    while IFS= read -r jbid; do 
     printf 'Checking ID %s\n' "$jbid" >&3 
     while IFS=, read -r _ ckid jbsiz _; do 
      case $jbsiz in 
       *[^[:digit:]]*|'') 
        # validation is important for subsequent arithmetic. 
        return 1 
        ;; 
       "$ckid") # Assuming "cksid" was a typo. Replace if not. 
        printf 'Matched at %s\n' "$ckid" 
        printf 'Valid -> %s\n' "$jbid" >&4 
        ((mjc++, amt += jbsiz)) 
        break 
        ;; 
       *) 
        printf 'No match at %s\n' "$ckid" 
      esac 
     done <uniqid 
     { 
      printf 'Check for ID %s done\n' "$jbid" 
      printf 'Matched %s jobs with combined size of %s\n' "$mjc" "$amt" 
     } >&3 
    done <tmpcsv2 3>>Acsv3.tmp 4>>Bcsv3.tmp 
} 

最后，与几乎任何其他语言一样，最后，相当于awk脚本的性能将明显优于此Bash脚本。通过使用mapfile而不是读取循环，您还可以通过使用mapfile回调来模拟使用嵌套的读取循环逻辑。

Answer 2

我想出了这个，不知道这是否可以缩短，但它确实运行得更快！任何帮助都感激不尽 ！

************ 
while read -r line #File read start 
do 
IFS=$"," 
val=$line 
amt=0 
mjc=0 
cjc=0 
for lsid in $val 
do 
    cksid=`echo $lsid | sed -e 's/*//g' -e 's/"//g'` 
    printf "Checking for $cksid\n" 
    ((cjc++)) #Count of jobs to check 
    prsnt=`grep -w $cksid uniq_id` 
    if [ $? -eq 0 ] 
    then 
     printf "Valid -> $cksid\n" 
     jbsiz=`grep -w $prsnt | cut -d, -f2` 
     ((mjc++, amt += jbsiz)) 
     break 
    else 
     printf "No Data for $cksid\n" 
    fi 

done 
done < tmpcsv2 
***********