Clearing OOB (Out Of Band) Migrations

Scenario

Below are two examples of OOB migrations in the Site Admin > Migrations

screen.

It some cases OOB migrations will get stuck working in the background. 

The one below at 56% was stuck for about 3 hours as you can see. 

When a migrations has been stuck for this long it is likely that it has to be manually run.

Troubleshooting Steps

1. Observe the OOB migration is stuck in an unfinished state. (Anything longer than 3+ hours is a safe assumption that the migration is stuck)
2. Spin down the databases in the Sourcegraph deployment. VERY IMPORTANT!
3. Use the migrator container in the Sourcegraph deployment with the following flag:

   migrator -run-out-of-band-migrations 

4. It's important to note that this can be done in a number of ways, depending on deployment. Below are the most common methods.

Running migrator in a Docker Compose Deployment 

1. 1. 1. In a Docker Compose deployment, update the migrator YAML node in the docker-compose.yaml file by updating the child command key. 
      2. Change the default value of the command key from up to run-out-of-band-migrations.
      3. Run docker compose up to apply the migrator container change and migrator will begin to run the out of band migrations manually.
      4. You will want to change the docker-compose.yaml file back otherwise the migrator will perform this operation every time it is spun up.

Running migrator in a K8s Deployment 

1. 1. 1. In a Kubernetes deployment, migrator runs as an initContainer. Update the sourcegraph-frontend.Deployment.yaml so that the migrator argument is -run-out-of-band-migrations.
      2. Apply the new change with:

      3.  kubectl apply -f frontend.Deployment.yaml

      4. You will want to change the frontend.Deployment.yaml back otherwise the migrator will perform this operation every time it is spun up.

Solution

With the above troubleshooting steps, the pending migrations should be cleared. 

If they are not, you are not dealing with a routine stuck migration issue and it will need more advanced troubleshooting to see why the issue is occurring. 

It is suggested that you approach the team that is responsible for the migration after performing a code search on the sourcegraph/sourcegraph repo to see what the migration is actually doing/changing in the targeted database.