Commit 4b6d4bd5 authored by Georgios Bitzes's avatar Georgios Bitzes
Browse files

doc: add production checklist, improving writing in upgrading.md

parent 2ed7d54f
Pipeline #1504363 passed with stages
in 99 minutes and 51 seconds
# Checklist for production
You've decided to run a cluster in production — great! Before hitting the red button, here's a list of recommendations
for your setup.
1. Ensure the cluster is secure — redis is a popular protocol, and there are bots scanning the
entire internet looking to attack unsecured redis instances. _This is not theoretical_ and we
have seen it happen.
* **Essential:** Configure your instance with [password authentication](authentication.md)
* _Recommended:_ Ensure the relevant ports are blocked from the open internet with a firewall, only
available within your internal network.
* Optional: Use a script that periodically verifies the cluster is inaccessible without
a password, and its ports shut from the open internet.
2. Ensure backups are taken at regular intervals — even though QuarkDB is replicated, _you still need backups_.
* **Essential:** Set-up a script to take periodic [backups](backup.md).
* **Essential:** Ensure your backup script will not silently fail. In case of failure, an alarm should be generated.
* _Recommended:_ Do basic sanity checking of the generated backup using `quarkdb-validate-checkpoint` tool before putting
into long-term storage.
document.addEventListener("DOMContentLoaded", function() {
load_navpane();
});
function load_navpane() {
var width = window.innerWidth;
if (width <= 1200) {
return;
}
var nav = document.getElementsByClassName("md-nav");
for(var i = 0; i < nav.length; i++) {
if (typeof nav.item(i).style === "undefined") {
continue;
}
if (nav.item(i).getAttribute("data-md-level") && nav.item(i).getAttribute("data-md-component")) {
nav.item(i).style.display = 'block';
nav.item(i).style.overflow = 'visible';
}
}
var nav = document.getElementsByClassName("md-nav__toggle");
for(var i = 0; i < nav.length; i++) {
nav.item(i).checked = true;
}
}
......@@ -2,7 +2,7 @@
## Introduction
QuarkDB is a highly available datastore speaking a redis-compatible protocol, being
QuarkDB is a highly available datastore speaking the redis wire protocol (RESP2), being
developed by IT-ST at CERN.
Highlights:
......@@ -17,7 +17,8 @@ storing metadata for billions of files.
## Getting started
Visit [this chapter](installation.md) for instructions on how to get a
QuarkDB cluster up and running.
QuarkDB cluster up and running. Check out the [production checklist](checklist.md) before
running in production.
There's also a short [screencast demo](https://asciinema.org/a/NdX791Ah4JVkGQnUQkBVm3dDJ),
which shows how to set up a test cluster on localhost.
......@@ -2,34 +2,34 @@
* QuarkDB does not start: `Cannot open '/var/lib/quarkdb/node-01/current/state-machine':IO error: while open a file for lock: /var/lib/quarkdb/node-01/current/state-machine/LOCK: Permission denied`
Most likely, the database owner permissions are not correct. If running
with the default systemd init scripts, you need run ``chown -R xrootd:xrootd /var/lib/quarkdb/node-01/``.
Most likely, the database owner permissions are not correct. If running
with the default systemd init scripts, you need run ``chown -R xrootd:xrootd /var/lib/quarkdb/node-01/``.
* I get this error in the log: `MISCONFIGURATION: received handshake with wrong cluster id: "..." (mine is "...")`
All members in a cluster have to have the same cluster ID, as provided in `quarkdb-create`.
You probably assigned a different one to each member.
All members in a cluster have to have the same cluster ID, as provided in `quarkdb-create`.
You probably assigned a different one to each member.
Each cluster has a unique identifier assigned to it, which each member node has to know.
During the initial handshake between two nodes, the cluster ID is exchanged - if
it does not match, the connection is shut down, and they refuse to communicate.
Each cluster has a unique identifier assigned to it, which each member node has to know.
During the initial handshake between two nodes, the cluster ID is exchanged - if
it does not match, the connection is shut down, and they refuse to communicate.
This is to prevent damage from a misconfiguration where two nodes from different
instances accidentally cross paths.
This is to prevent damage from a misconfiguration where two nodes from different
instances accidentally cross paths.
* I get this error in the log: `WARNING: I am not receiving heartbeats - not running for leader since in membership epoch 0 I am not a full node. Will keep on waiting.`
The node identifier, as provided in `redis.myself`, is not part of the QuarkDB cluster, as
initialized in ``quarkdb-create``. Either change `redis.myself` to match one of
the nodes provided in ``quarkdb-create``, or re-run ``quarkdb-create`` with the correct
nodes.
The node identifier, as provided in `redis.myself`, is not part of the QuarkDB cluster, as
initialized in ``quarkdb-create``. Either change `redis.myself` to match one of
the nodes provided in ``quarkdb-create``, or re-run ``quarkdb-create`` with the correct
nodes.
Run `redis-cli -p 7777 raft-info` to check the current cluster membership. Each
node expects to find `redis.myself` within `NODES`, to decide whether it's capable
of starting an election.
Run `redis-cli -p 7777 raft-info` to check the current cluster membership. Each
node expects to find `redis.myself` within `NODES`, to decide whether it's capable
of starting an election.
* The nodes are holding elections indefinitely, even though I've started all of them!
It's likely a firewall issue. Try running
`redis-cli -h qdb-test-2.cern.ch -p 7777 raft-info` from a different node, for example -
if you can't connect, it's a firewall issue.
It's likely a firewall issue. Try running
`redis-cli -h qdb-test-2.cern.ch -p 7777 raft-info` from a different node, for example -
if you can't connect, it's a firewall issue.
......@@ -2,41 +2,42 @@
## Can I upgrade without downtime?
**Yes**. It's possible to upgrade a QuarkDB cluster with minimal impact on availability:
The time needed to perform a single leader election, which is typically a couple
of seconds.
_Yes_, a routine version upgrade should require only a single leader election.
The recommended steps are the following:
The recommended upgrade procedure is the following:
1. Use ``raft-info`` to find out the current cluster leader.
* Find out the current cluster leader by running ``raft-info`` command.
2. For each of the _followers_, do the following _one by one_:
* For each follower node, upgrade the system packages, restart the QuarkDB
service, and wait until the leader declares the node online, and having
an up-to-date journal.
Run ``raft-info`` on the leader node, check ``REPLICA`` section for this
information.
* Upgrade system packages.
* Make sure to upgrade the followers _one by one_, not all at once. If you
simultaneously upgrade both followers on a 3-node cluster, for example, the
cluster will become unavailable due to loss of quorum until the nodes come
back online. This could take a couple of minutes.
* Restart the QuarkDB service.
* Finally, upgrade and restart the leader. The followers will detect the
* Wait until the leader declares the node as back online, and as having an up-to-date
journal. Run ``raft-info`` on the leader node, check ``REPLICA`` section for this
information.
3. Finally, upgrade and restart the leader. The followers will detect the
absence of heartbeats, and elect a new leader among themselves within a
few seconds.
## Sub-optimal ways of upgrading QuarkDB
Upgrading the followers _one by one_ and not all at once is **important**: If you
simultaneously upgrade both followers on a 3-node cluster, the cluster could become
unavailable for a couple of minutes due to loss of quorum until the nodes come back
online.
## What not to do
All following methods are worse than the above, since they cause longer downtime
than a single leader election:
than a single leader election.
* Restart the leader first. Inevitably, leadership will go to one of the followers,
which will have to be restarted too at some point for an upgrade, resulting
* Upgrade the leader first. Inevitably, leadership will jump to one of the followers,
which will have to be restarted too at some point for the upgrade, resulting
in 2 or more elections for the upgrade in total.
* Restart all QuarkDB daemons at the same time: The cluster could potentially
go down for long, depending on how quickly the processes are able to come back
online. For large databases, this could take a couple of minutes.
* Upgrade all QuarkDB daemons at the same time: The cluster could potentially
go down for minutes, depending on how quickly the processes are able to come back
online.
* An upgrade when quorum is shaky: For example, if only 2 out of 3 nodes are
available (maybe the third died from a broken hard drive), an upgrade of any
......
......@@ -33,17 +33,15 @@ theme:
feature:
tabs: true
extra_javascript:
- expand-always.js
nav:
- Home:
- Home: index.md
- Introduction: index.md
- Getting started:
- Installation: installation.md
- Configuration: configuration.md
- Troubleshooting: troubleshooting.md
- Operator's manual:
- Checklist for production: checklist.md
- Version upgrades: upgrading.md
- Backup & restore: backup.md
- Configuration:
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment