Bug: Investigate high number of DLQ Messages

Reason: Failure of email consumer.

Error in Sentry: https://push-notifications-sentry.web.cern.ch/sentry/consumers/issues/228/?query=is%3Aunresolved

Investigate:

• reason for "broken pipe" in stomp connection
• ways to mitigate - healthchecks to kill the consumer, or a way to propagate the consumer crash and let it be properly restarted

| | Traceback (most recent call last):|
| | File "/opt/notifications_consumer/consumer.py", line 44, in on_message|
| | self.ack_message(headers["message-id"])|
| | File "/opt/notifications_consumer/consumer.py", line 28, in ack_message|
| | connection.ack(message_id, 1)|
| | File "/usr/local/lib/python3.6/site-packages/stomp/protocol.py", line 275, in ack|
| | self.send_frame(CMD_ACK, headers)|
| | File "/usr/local/lib/python3.6/site-packages/stomp/protocol.py", line 244, in send_frame|
| | self.transport.transmit(frame)|
| | File "/usr/local/lib/python3.6/site-packages/stomp/transport.py", line 272, in transmit|
| | self.send(packed_frame)|
| | File "/usr/local/lib/python3.6/site-packages/stomp/transport.py", line 644, in send|
| | raise e|
| | File "/usr/local/lib/python3.6/site-packages/stomp/transport.py", line 640, in send|
| | self.socket.sendall(encoded_frame)|
| | BrokenPipeError: [Errno 32] Broken pipe|
| | |
| | During handling of the above exception, another exception occurred:|
| | |
| | Traceback (most recent call last):|
| | File "/usr/local/lib/python3.6/site-packages/stomp/transport.py", line 640, in send|
| | self.socket.sendall(encoded_frame)|
| | BrokenPipeError: [Errno 32] Broken pipe|
| | 2021-06-04 14:02:13,229 ERROR stomp.py Error sending frame|
| | Traceback (most recent call last):|
| | File "/opt/notifications_consumer/consumer.py", line 44, in on_message|
| | self.ack_message(headers["message-id"])|
| | File "/opt/notifications_consumer/consumer.py", line 28, in ack_message|
| | connection.ack(message_id, 1)|
| | File "/usr/local/lib/python3.6/site-packages/stomp/protocol.py", line 275, in ack|
| | self.send_frame(CMD_ACK, headers)|
| | File "/usr/local/lib/python3.6/site-packages/stomp/protocol.py", line 244, in send_frame|
| | self.transport.transmit(frame)|
| | File "/usr/local/lib/python3.6/site-packages/stomp/transport.py", line 272, in transmit|
| | self.send(packed_frame)|
| | File "/usr/local/lib/python3.6/site-packages/stomp/transport.py", line 644, in send|
| | raise e|
| | File "/usr/local/lib/python3.6/site-packages/stomp/transport.py", line 640, in send|
| | self.socket.sendall(encoded_frame)|
| | BrokenPipeError: [Errno 32] Broken pipe|