0
更新到最新的SDK版本0.3.150326，我们有工作不能因这个错误：错误：“不能缩小不失活跃洗牌数据”
(d0f58ccaf368cf1f): Workflow failed. Causes: (539037ea87656484): Cannot downsize without losing active shuffle data. old_size = 10, new_size = 8.
工作ID：2015-04-02_21_26_53-11930390736602232537
还没有能够重现，但我想我应该问这是否是一个已知的问题？
看着docs，看起来自动缩放目前只是“实验性”的，但我会想象这是Cloud Dataflow的核心功能，因此应该完全支持。
1087 [main] INFO com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner - Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
1103 [main] INFO com.google.cloud.dataflow.sdk.util.PackageUtil - Uploading 79 files from PipelineOptions.filesToStage to GCS to prepare for execution in the cloud.
43086 [main] INFO com.google.cloud.dataflow.sdk.util.PackageUtil - Uploading PipelineOptions.filesToStage complete: 2 files newly uploaded, 77 files cached
Dataflow SDK version: 0.3.150326
57718 [main] INFO com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner - To access the Dataflow monitoring console, please navigate to https://console.developers.google.com/project/gdfp-7414/dataflow/job/2015-04-02_21_26_53-11930390736602232537
Submitted job: 2015-04-02_21_26_53-11930390736602232537
2015-04-03T04:26:54.710Z: (3a5437c7f9c6e33f): Expanding GroupByKey operations into optimizable parts.
2015-04-03T04:26:54.714Z: (3a5437c7f9c6e8dd): Annotating graph with Autotuner information.
2015-04-03T04:26:55.436Z: (3a5437c7f9c6e85b): Fusing adjacent ParDo, Read, Write, and Flatten operations
2015-04-03T04:26:55.453Z: (3a5437c7f9c6efad): Fusing consumer denormalized-write-to-BQ into events-denormalize
2015-04-03T04:26:55.455Z: (3a5437c7f9c6e54b): Fusing consumer events-denormalize into events-read-from-BQ
2015-04-03T04:26:55.457Z: (3a5437c7f9c6eae9): Fusing consumer unmapped-write-to-BQ into events-denormalize
2015-04-03T04:26:55.504Z: (3a5437c7f9c6e67d): Adding StepResource setup and teardown to workflow graph.
2015-04-03T04:26:55.525Z: (971aceaf96c03b86): Starting the input generators.
2015-04-03T04:26:55.546Z: (ea598353613cc1d3): Adding workflow start and stop steps.
2015-04-03T04:26:55.548Z: (ea598353613ccd39): Assigning stage ids.
2015-04-03T04:26:56.017Z: S07: (fb31ac3e5c3be05a): Executing operation WeightingFactor
2015-04-03T04:26:56.024Z: S09: (ee7049b2bfe3f48c): Executing operation Name_Community
2015-04-03T04:26:56.037Z: (3a5437c7f9c6e293): Starting worker pool setup.
2015-04-03T04:26:56.042Z: (3a5437c7f9c6edcf): Starting 5 workers...
2015-04-03T04:26:56.047Z: S01: (a25730bd9d25e5ed): Executing operation Browser_mapping
2015-04-03T04:26:56.049Z: S11: (fb31ac3e5c3beb06): Executing operation WebsiteVHH
2015-04-03T04:26:56.051Z: (30eb1307dfc8372f): Value "Name_Community.out" materialized.
2015-04-03T04:26:56.065Z: (52e655ceeab44257): Value "WeightingFactor.out" materialized.
2015-04-03T04:26:56.072Z: S03: (c024e27994951718): Executing operation OS_mapping
2015-04-03T04:26:56.076Z: S10: (a3947955b25f3830): Executing operation AsIterable3/CreatePCollectionView
2015-04-03T04:26:56.087Z: (4c9eb5a54721c4f7): Value "WebsiteVHH.out" materialized.
2015-04-03T04:26:56.094Z: S05: (52e655ceeab4458a): Executing operation SA1_Area_Metro
2015-04-03T04:26:56.103Z: S08: (c024e279949513f4): Executing operation AsIterable2/CreatePCollectionView
2015-04-03T04:26:56.106Z: (4c9eb5a54721cd78): Value "AsIterable3/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.107Z: (58b58f637f29b69a): Value "OS_mapping.out" materialized.
2015-04-03T04:26:56.115Z: (f0587ec8b1f9f69f): Value "Browser_mapping.out" materialized.
2015-04-03T04:26:56.126Z: (a277f34c719a133): Value "SA1_Area_Metro.out" materialized.
2015-04-03T04:26:56.127Z: S12: (c024e279949510d0): Executing operation AsIterable4/CreatePCollectionView
2015-04-03T04:26:56.132Z: S04: (52e655ceeab44adf): Executing operation AsIterable6/CreatePCollectionView
2015-04-03T04:26:56.136Z: (f0587ec8b1f9fd86): Value "AsIterable2/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.141Z: S02: (eb97fca639a2101b): Executing operation AsIterable5/CreatePCollectionView
2015-04-03T04:26:56.151Z: S06: (8cc6100045f0af9b): Executing operation AsIterable/CreatePCollectionView
2015-04-03T04:26:56.159Z: (6da6e59d099e8c60): Value "AsIterable4/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.163Z: (4c9eb5a54721c5f9): Value "AsIterable6/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.173Z: (a3947955b25f3701): Value "AsIterable5/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.178Z: (58b58f637f29b853): Value "AsIterable/CreatePCollectionView.out" materialized.
2015-04-03T04:26:56.199Z: S13: (8cc6100045f0ac67): Executing operation events-read-from-BQ+events-denormalize+denormalized-write-to-BQ+unmapped-write-to-BQ
2015-04-03T04:26:56.653Z: (6153d4cd276be2a0): Autoscaling: Enabled for job /workflows/wf-2015-04-02_21_26_53-11930390736602232537
2015-04-03T04:30:31.754Z: (a94b4f451005c934): Autoscaling: Resizing worker pool from 5 to 10.
2015-04-03T04:31:01.754Z: (a94b4f451005c38e): Autoscaling: Resizing worker pool from 10 to 8.
2015-04-03T04:31:02.363Z: (d0f58ccaf368cf1f): Workflow failed. Causes: (539037ea87656484): Cannot downsize without losing active shuffle data. old_size = 10, new_size = 8.
2015-04-03T04:31:02.396Z: (7f503ea3d5c37a55): Stopping the input generators.
2015-04-03T04:31:02.411Z: (58b58f637f29ba9f): Cleaning up.
2015-04-03T04:31:02.442Z: (58b58f637f29bc58): Tearing down pending resources...
2015-04-03T04:31:02.447Z: (58b58f637f29be11): Starting worker pool teardown.
2015-04-03T04:31:02.453Z: (58b58f637f29b05d): Stopping worker pool...
2015-04-03T04:31:03.062Z: (a1f260e16fea5b6): Workflow failed. Causes: (539037ea87656484): Cannot downsize without losing active shuffle data. old_size = 10, new_size = 8.
458752 [main] INFO com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner - Job finished with status FAILED
458755 [main] INFO com.<removed>.cdf.job.AbstractCloudDataFlowJob - com.google.cloud.dataf[email protected]27a7ef08
458755 [main] INFO com.<removed>.cdf.job.AbstractCloudDataFlowJob - Cleaning up after <removed> job. At the moment nothing to do.
Disconnected from the target VM, address: '127.0.0.1:57739', transport: 'socket'