Scenario

When deploying Sourcegraph, more so after an upgrade, you could encounter Redis container(s) or pod(s) restarting frequently.

Looking at the logs, you may come across something similar to the below:

# oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

# Redis version=5.0.13, bits=64, commit=00000000, modified=0, pid=7, just started

# Configuration loaded

# Not listening to IPv6: unsupported

* Running mode=standalone, port=6379.

# WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

# Server initialized

# WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

# WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.

What does this mean?

This potentially indicates that is likely Redis is hitting an out of memory issue. 

Troubleshooting

First, you could start by checking if all the services are healthy. 
More so paying attention to check if any strange logs from the frontend or gitserver.

Also, you may need to check if there are any alerts firing in the Redis grafana dashboard.

If nothing stands out, you could do the following:

1. Make sure you have given the Redis container enough memory. The minimum recommended is 100MB, but for production use you'll want 1g or more. In your case, you're running the standard numbers. Perhaps we could increase this to see if this helps. Going 2X on current could be a great start.
    
2. Check that overcommit_memory is actually set to 0. You can check this on the host machine with: 

cat /proc/sys/vm/overcommit_memory

If it is 0, you'll want to increase it to 1 or 2. You can do this temporarily with:
 

sysctl vm.overcommit_memory=1

Or permanently by adding it to /etc/sysctl.conf .

1. Increase overcommit_ratio. This controls how much memory can be allocated beyond the amount of swap and RAM. You can check it with:

cat /proc/sys/vm/overcommit_ratio

And increase it in the same way as overcommit_memory. A value of 50 or higher is recommended for Redis.

1. Make sure you don't have too many background processes running on the host that could be consuming memory.
2. If possible, isolate the Redis container by running it on a separate machine. This can prevent it from competing for memory with other processes.
3. Consider using Redis persistence to avoid the container constantly restarting and reloading data into memory.
4. As a last resort, you may need to switch to a larger machine with more memory to properly run your Redis container.