Set-S3Acl Powershell cmdlet无法正常工作20-30k对象

我的客户端遇到了问题，即他们意外地将1300万个对象（文件）复制到了具有错误权限的S3存储桶中。他们已经要求我的团队修复它。我们必须使用正确的ACL更新S3存储桶中的每一千三百万个文件。我们正在使用下面的PowerShell脚本来解决它。但是，如果脚本在具有超过20-30k个对象的文件夹上运行，则无法设置ACL。 [它通过循环迭代，但它不会设置权限发布20-30k对象，也不例外]Set-S3Acl Powershell cmdlet无法正常工作20-30k对象

我怀疑请求可能会被限制。有没有人遇到过这样的问题。请帮助我如何继续。

我在寻找下列问题的答案： 1.如果API调用被拦截@ 20-30k对象，我该如何修改脚本以克服它。 2.对于数百万个对象来说，修改AWS资源的脚本编制方式（如为S3对象设置ACL权限）的最佳做法是什么

（我不是在寻找“BucketPolicy”方法，因为我们有用一个脚本来做到这一点，并将ACL应用到每个S3对象）

Param (
    [Parameter(Position=0,Mandatory=$true)] 
    [string]$profile, 
    [Parameter(Position=1,Mandatory=$true)] 
    [string]$switchToAccount, 
    [Parameter(Position=2,Mandatory=$true)] 
    [string]$roleName, 
    [Parameter(Position=3,Mandatory=$true)] 
    [string]$keyPrefix 
) 

#Set base AWS credentials 
Set-AWSCredentials -ProfileName $profile 
Set-DefaultAWSRegion -Region $region 

#Get and set MFA device ARN 
$userName = (Get-IAMUser).UserName 
$mfaArn = "arn:aws:iam::xxxxxxxxx:mfa/" + "$userName" 

#Configure CAA roles 
$roleArn = "arn:aws:iam::" + "$switchToAccount" + ":role/" + "$roleName" 
$roleSessionName = "xxxxxxxxxxxx" 

#Prompt for MFA token and perform CAA request 
$tokenCode = Read-Host -Prompt "Enter MFA token for $accountNumber" 
$switchRole = Use-STSRole -RoleSessionName $roleSessionName -RoleArn $roleArn -TokenCode $tokenCode -SerialNumber $mfaArn 

#Set new role for CAA 
Set-AWSCredentials -Credential $switchRole.Credentials 

#Declare access level for S3 Object ACL grantees 
$FULL_CONTROL = [Amazon.S3.S3Permission]::FULL_CONTROL 
$grants = @(); 

#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxxxxxx 
$grantee1 = New-Object -TypeName Amazon.S3.Model.S3Grantee 
$grantee1.EmailAddress = "xxxxxxxxxxxxxxxxxxx" 

#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxx 
$grantee2 = New-Object -TypeName Amazon.S3.Model.S3Grantee 
$grantee2.EmailAddress = "xxxxxxxxxxxxxxxxxxx" 

#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxxxxx 
$grantee3 = New-Object -TypeName Amazon.S3.Model.S3Grantee 
$grantee3.EmailAddress = "xxxxxxxxxxxxxxxxxxxxx" 

#Create grant and add to grant list 
$grant1 = New-Object -TypeName Amazon.S3.Model.S3Grant 
$grant1.Grantee = $grantee1 
$grant1.Permission = $FULL_CONTROL 
$grants += $grant1 

#Create grant and add to grant list 
$grant2 = New-Object -TypeName Amazon.S3.Model.S3Grant 
$grant2.Grantee = $grantee2 
$grant2.Permission = $FULL_CONTROL 
$grants += $grant2 

#Create grant and add to grant list 
$grant3 = New-Object -TypeName Amazon.S3.Model.S3Grant 
$grant3.Grantee = $grantee3 
$grant3.Permission = $FULL_CONTROL 
$grants += $grant3 



#Set bucket name for S3 objects 
$bucketName = "xxxxxxxxxxxxxxxxxxxxxxxxx" 
#Get all S3 Objects in specified bucket 
$s3Objects = Get-S3Object -BucketName $bucketName -KeyPrefix $keyPrefix 




#Count for progress bar 
$totalObjects = $s3Objects.length 
$i = 1 
$fail_count = 0 
$current_count = 0 
$file_path = "C:\Users\Administrator\Desktop\Failed_Objects_new\" + $keyPrefix.Replace("/","_") + ".txt" 
$file_path_retry = "C:\Users\Administrator\Desktop\Failed_Objects_new_retry\" + $keyPrefix.Replace("/","_") + ".txt" 

     new-item $file_path -ItemType file 
     new-item $file_path_retry -ItemType file 

"Total Object Count:" + $totalObjects + "`n" | Out-File $file_path -Append 


foreach($s3Object in $s3Objects){ 

    $owner = $s3Object.owner.id 
    $s3Object.name | Write-Output 
    $current_count++ 
    #Extracts Key for each S3 object in bucket 
    $key = $s3Object.Key 

    #Logging 
    Write-Host "Setting $bucketName | $key | $grants" 

    # Pick objects that were modified on or before July 15th 

    try { 

     if (($s3Object.LastModified.month -lt 7)) { 
      Set-S3ACL -BucketName $bucketName -Key $key -Grant $grants -OwnerId $owner 
      $owner | Write-Host 
     } 

     elseif(($s3Object.LastModified.month -eq 7) -and ($s3Object.LastModified.day -le 15)) { 

      Set-S3ACL -BucketName $bucketName -Key $key -Grant $grants -OwnerId $owner 
      $owner | Write-Host 
     } 

    }catch{ 

     "Failed $bucketName | $key | $grants" | out-file $file_path -Append 
     $key | Out-File $file_path_retry -Append 
     $fail_count++ 
    } 

    Write-Host "progress: " $current_count "/" $totalObjects 
    #Update progress bar 
    $percentComplete = $i/$totalObjects 
    Write-Progress -Activity "Setting S3 Object ACL's" -Status "$i% complete" -PercentComplete $percentComplete 
    $i++ 

} 



"`n`n Total Fail Count:" + $fail_count | Out-File $file_path -Append

来源

2017-07-24 Manjunath Rao

如何用正确的权限再次复制所有文件！这可能是最快的解决方案..因为复制将迭代发生 – Deepak

我确实推荐它。目前管理层不想去客户端，而是希望我们通过脚本修复它。 –

步骤来调试问题：

确保如果节流问题。在for循环中;打破10k个对象后，看看是否一切正常。
另外，把try语句里面的print语句放在if和else里面，以确定它是否到达那里;何时失败？

来源

2017-07-24 17:36:04 Deepak

该脚本工作到20-30k对象。它会停止授予权限后的权限。我在try块内有一个命令“$ owner | Write-Host”来测试它是否到达。它正在达到TRY块。但是，它不授予权限。 –

Set-S3Acl Powershell cmdlet无法正常工作20-30k对象

回答

相关问题