尝试从密钥空间读取时Cassandra复制节点失败

Question

我的Cassandra（3.0.2）复制集中有3个节点。我的一致性级别是“一个”。在开始时，我的所有密钥空间的复制因子都等于1.我通过更改表进行更改，并在所有节点上运行“nodetool修复”。现在，当我试图选择一些数据（而不是在每一个密钥空间），我得到这样的事情（选择keyspace.table *）：

Traceback (most recent call last): File "/usr/bin/cqlsh.py", line 1258, in perform_simple_statement result = future.result() File "cassandra/cluster.py", line 3781, in cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:73073) raise self._final_exception ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

在“/var/log/cassandra/system.log “我得到：

WARN [SharedPool-Worker-2] 2017-04-07 12:46:20,036 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-2,5,main]: {} java.lang.AssertionError: null at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$IndexState.updateBlock(AbstractSSTableIterator.java:463) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardIndexedReader.computeNext(SSTableIterator.java:268) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:158) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:352) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:219) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:426) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:286) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:298) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:136) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:128) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1721) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2375) ~[apache-cassandra-3.0.2.jar:3.0.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.2.jar:3.0.2] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] DEBUG [SharedPool-Worker-1] 2017-04-07 12:46:20,037 ReadCallback.java:126 - Failed; received 0 of 1 responses

而且我得到：

DEBUG [SharedPool-Worker-1] 2017-04-07 13:20:30,002 ReadCallback.java:126 - Timed out; received 0 of 1 responses

我检查有没有关于端口9042和7000。我改变选项中的节点之间的连接‘/etc/cassandra/cassandra.yml’像“read_request_timeout_in_ms”，“range_request_timeout_in_m s“，”write_request_timeout_in_ms“或”truncate_request_timeout_in_ms“。我更改了文件“〜/ .cassandra/cqlshrc”和选项“client_timeout = 3600”。 Additonally当我如做“从keyspace.table选择*其中列1 =‘值’和列2 =价值”，我得到：

ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

任何想法？

Answer 1

这或多或少是一条评论，但因为有这么多话要说它不适合评论。

这将是非常好的说什么复制因素你改变了价值。我只是假设它是3，因为它非常标准。然后再次，因为你只有一群3人有时将RF设置为2.你还提到你更新了表上的复制因子。据我所知，复制因子是在密钥空间级别上设置的。

如果您发布了发生错误的密钥空间的描述，这将非常有帮助。

考虑到select * from something可能在您的集群中非常密集，特别是如果您有大量数据。如果你要在cqlsh中执行这个查询，你可能会得到10 000，然后你再次只提到了cqlsh，并且没有应用程序代码，所以我只是注意到这个问题。

能否请你提供nodetool status只是为了确保你实际上没有运行一些节点下的查询。因为第一个错误看起来像这样。

第二个错误，你发布了一个堆栈跟踪，它看起来像你缺少磁盘上的一些sstables？是否有某种其他过程以某种方式操纵sstables的可能性？

你也改变了cassandra.yaml中的很多属性，基本上你将预期的响应时间减少了近50％，我想这也难怪节点没有时间来响应......在整个表上计数通常需要超过3.6秒

推理这些值改变的原因简直就是缺失。

Answer 2

MarkoŠvaljek，是的，我将复制因子从1更改为3（因为我的复制中有3个节点）。你是对的;你改变了密钥空间的复制因子，这就是我所做的。在这里，你有我的密钥空间的描述通常在那里我得到了一些错误（当然它与其他keyspaces发生）：

soi@cqlsh> desc keyspace engine; 

CREATE KEYSPACE engine WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true; 

CREATE TABLE engine.messages (
    persistence_id text, 
    partition_nr bigint, 
    sequence_nr bigint, 
    timestamp timeuuid, 
    timebucket text, 
    message blob, 
    tag1 text, 
    tag2 text, 
    tag3 text, 
    used boolean static, 
    PRIMARY KEY ((persistence_id, partition_nr), sequence_nr, timestamp, timebucket) 
) WITH CLUSTERING ORDER BY (sequence_nr ASC, timestamp ASC, timebucket ASC) 
    AND bloom_filter_fp_chance = 0.01 
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} 
    AND comment = '' 
    AND compaction = {'bucket_high': '1.5', 'bucket_low': '0.5', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'enabled': 'true', 'max_threshold': '32', 'min_sstable_size': '50', 'min_threshold': '4', 'tombstone_compaction_interval': '86400', 'tombstone_threshold': '0.2', 'unchecked_tombstone_compaction': 'false'} 
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} 
    AND crc_check_chance = 1.0 
    AND dclocal_read_repair_chance = 0.1 
    AND default_time_to_live = 0 
    AND gc_grace_seconds = 864000 
    AND max_index_interval = 2048 
    AND memtable_flush_period_in_ms = 0 
    AND min_index_interval = 128 
    AND read_repair_chance = 0.0 
    AND speculative_retry = '99PERCENTILE'; 

CREATE MATERIALIZED VIEW engine.eventsbytag1 AS 
    SELECT tag1, timebucket, timestamp, persistence_id, partition_nr, sequence_nr, message 
    FROM engine.messages 
    WHERE persistence_id IS NOT NULL AND partition_nr IS NOT NULL AND sequence_nr IS NOT NULL AND tag1 IS NOT NULL AND timestamp IS NOT NULL AND timebucket IS NOT NULL 
    PRIMARY KEY ((tag1, timebucket), timestamp, persistence_id, partition_nr, sequence_nr) 
    WITH CLUSTERING ORDER BY (timestamp ASC, persistence_id ASC, partition_nr ASC, sequence_nr ASC) 
    AND bloom_filter_fp_chance = 0.01 
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} 
    AND comment = '' 
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'} 
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} 
    AND crc_check_chance = 1.0 
    AND dclocal_read_repair_chance = 0.1 
    AND default_time_to_live = 0 
    AND gc_grace_seconds = 864000 
    AND max_index_interval = 2048 
    AND memtable_flush_period_in_ms = 0 
    AND min_index_interval = 128 
    AND read_repair_chance = 0.0 
    AND speculative_retry = '99PERCENTILE'; 

CREATE TABLE engine.config (
    property text PRIMARY KEY, 
    value text 
) WITH bloom_filter_fp_chance = 0.01 
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} 
    AND comment = '' 
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'} 
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} 
    AND crc_check_chance = 1.0 
    AND dclocal_read_repair_chance = 0.1 
    AND default_time_to_live = 0 
    AND gc_grace_seconds = 864000 
    AND max_index_interval = 2048 
    AND memtable_flush_period_in_ms = 0 
    AND min_index_interval = 128 
    AND read_repair_chance = 0.0 
    AND speculative_retry = '99PERCENTILE'; 

CREATE TABLE engine.metadata (
    persistence_id text PRIMARY KEY, 
    deleted_to bigint, 
    properties map<text, text> 
) WITH bloom_filter_fp_chance = 0.01 
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} 
    AND comment = '' 
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'} 
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} 
    AND crc_check_chance = 1.0 
    AND dclocal_read_repair_chance = 0.1 
    AND default_time_to_live = 0 
    AND gc_grace_seconds = 864000 
    AND max_index_interval = 2048 
    AND memtable_flush_period_in_ms = 0 
    AND min_index_interval = 128 
    AND read_repair_chance = 0.0 
    AND speculative_retry = '99PERCENTILE'; 

我平时代码错误没有。 1200或1300，你可以在第一篇文章中看到。在这里你有我的“节点工具状态”：

ubuntu@cassandra-db1:~$ nodetool status 
Datacenter: datacenter1 
======================= 
Status=Up/Down 
|/ State=Normal/Leaving/Joining/Moving 
-- Address  Load  Tokens  Owns Host ID        Rack 
UN 192.168.1.13 3.94 MB 256   ?  8ebcc3fe-9869-44c5-b7a5-e4f0f5a0beb1 rack1 
UN 192.168.1.14 4.26 MB 256   ?  977831cb-98fe-4170-ab15-2b4447559003 rack1 
UN 192.168.1.15 4.94 MB 256   ?  7515a967-cbdc-4d89-989b-c0a2f124173f rack1 

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless 

我不这么认为一些其他进程可以操纵磁盘上的一些数据。我会补充说，我有类似的复制，我有更多的数据，我没有这样的问题。

Answer 3

固定！我将Cassandra版本从3.0.2更改为3.0.9，并解决了问题。