Job Failed #617448
Job #617448 failed for b78bcb5c:
This failure looks really nasty... ouch. Might even be a bug in rocksdb, investigate.
[ RUN ] Raft_e2e.simultaneous_clients
[1507134612970] INFO: Spinning up node #1
[1507134612988] INFO: Spinning up node #2
[1507134614123] EVENT: server2:23458: TIMEOUT after 1130ms, I am not receiving heartbeats. Attempting to start election.
[1507134614123] EVENT: Progressing raft term: 0 ==> 1
[1507134614123] EVENT: Status transition: FOLLOWER ==> CANDIDATE
[1507134614123] INFO: server2:23458: Starting election round for term 1
[1507134614128] EVENT: Progressing raft term: 0 ==> 1
[1507134614129] EVENT: Granting vote for term 1 to server2:23458
[1507134614129] INFO: server2:23458: Vote requests have been sent off, will allow a window of 1000ms to receive replies.
[1507134614129] EVENT: Progressing raft term: 0 ==> 1
[1507134614130] EVENT: Granting vote for term 1 to server2:23458
[1507134614130] INFO: No longer accepting replies to vote requests, time to make a tally.
[1507134614130] EVENT: Election round successful for term 1. Contacted 2 nodes, received 2 replies with a tally of 2 positive votes, 0 refused votes, and 0 vetoes.
[1507134614131] EVENT: Status transition: CANDIDATE ==> LEADER
[1507134614131] EVENT: Ascending as leader for term 1. Long may I reign.
[1507134614131] INFO: Starting replicator for term 1
[1507134614131] EVENT: Activating replicator for term 1
[1507134614131] INFO: Reconfiguring replicator for membership epoch 0
[1507134614137] EVENT: Recognizing leader server2:23458 for term 1
[1507134614137] EVENT: Recognizing leader server2:23458 for term 1
[1507134614138] EVENT: Replication target server0:23456 came back online. Log size: 1, lagging 1 entries behind me. (approximate)
[1507134614138] EVENT: Replication target server1:23457 came back online. Log size: 1, lagging 1 entries behind me. (approximate)
[1507134614145] INFO: Achieved state consensus for term 1 with leader server2:23458
[1507134614145] INFO: Condition 'checkStateConsensus(0, 1, 2)' is true after 113 attempts
[1507134614145] INFO: Achieved state consensus for term 1 with leader server2:23458
[1507134614403] INFO: Condition 'stateMachine(i)->get("asdf", value).ok() && value == "3456"' is true after 24 attempts
[1507134614414] INFO: Condition 'stateMachine(i)->get("asdf", value).ok() && value == "3456"' is true after 1 attempts
[1507134614424] INFO: Condition 'stateMachine(i)->get("asdf", value).ok() && value == "3456"' is true after 1 attempts
[1507134614477] INFO: Race-y request: GET client3 ==>
[1507134615081] INFO: Openning state machine '/tmp/quarkdb-tests/checkpoint/state-machine'.
[1507134615245] INFO: Closing state machine '/tmp/quarkdb-tests/checkpoint/state-machine'
/root/rpmbuild/BUILD/quarkdb-0.0.5.70.b78bcb5/test/e2e.cc:160: Failure
Value of: checkpointSM.get("client3", tmp).ok()
Actual: false
Expected: true
[1507134615248] EVENT: Spinning until all requests being dispatched (0) have been processed.
[1507134615248] INFO: All requests processed, detaching.
[1507134615248] INFO: Shutting down the raft machinery.
[1507134615248] EVENT: Status transition: FOLLOWER ==> SHUTDOWN
[1507134615248] EVENT: De-activating replicator
[1507134615249] INFO: Backend has been detached from this quarkdb shard.
[1507134615249] EVENT: Spinning until all requests being dispatched (0) have been processed.
[1507134615249] INFO: All requests processed, detaching.
[1507134615249] INFO: Shutting down the raft machinery.
[1507134615250] EVENT: Status transition: FOLLOWER ==> SHUTDOWN
[1507134615250] EVENT: De-activating replicator
[1507134615250] INFO: Backend has been detached from this quarkdb shard.
[1507134615252] EVENT: Spinning until all requests being dispatched (0) have been processed.
[1507134615252] INFO: All requests processed, detaching.
[1507134615252] INFO: Shutting down the raft machinery.
[1507134615252] EVENT: Status transition: LEADER ==> SHUTDOWN
[1507134615252] EVENT: De-activating replicator
[1507134615252] EVENT: Shutting down replicator tracker for server0:23456
[1507134615252] EVENT: Shutting down replicator tracker for server1:23457
[1507134615254] EVENT: De-activating replicator
[1507134615254] INFO: Backend has been detached from this quarkdb shard.
[1507134615255] INFO: Spinning up node #0
[ FAILED ] Raft_e2e.simultaneous_clients (2288 ms)