Scenario
Blobstore replaced minio in v4.2.1. For customers running Kustomize patches for minio, chances are, that they can and do forget to apply the correct folder permissions for blobstore. What this causes then is either the precise-code-intel-worker pod or container constantly restarting or falling into a CrashLoop state.
Description
You'll see log entries in precise-code-intel worker that look like:
Message: {"SeverityText":"FATAL","Timestamp":1681494676729380830,"InstrumentationScope":"codeintel-worker","Caller":"shared/shared.go:108","Function":"github.com/sourcegraph/sourcegraph/enterprise/cmd/precise-code-intel-worker/shared.Main","Body":"Failed to initialize upload store","Resource":{"service.name":"precise-code-intel-worker","service.version":"4.3.1","service.instance.id":"precise-code-intel-worker-7697f89fb8-wb7cg"},"Attributes":{"error":"failed to create bucket: operation error S3: CreateBucket, https response error StatusCode: 403, RequestID: 4442587FB7D0A2F9, HostID: , api error AccessDenied: Could not create bucket"}}
Workaround
First, you need to confirm if it's indeed a permissions issue by performing the following:
Shell into the blobstore pod:
kubectl exec -it blobstore sh
Test if you can create a bucket:
/opt/s3proxy $ curl -i -X PUT http://127.0.0.1:9000/create-bucket-test
Output should look like:
HTTP/1.1 200 OK Date: Mon, 17 Apr 2023 12:41:57 GMT x-amz-request-id: 4442587FB7D0A2F9 Location: /create-bucket-test Content-Length: 0 Server: Jetty(11.0.11)
Confirm that file permissions are correct (should show
sourcegraph for the /data directory and all directories below:
/opt/s3proxy $ ls -lah /data total 36K drwxrwsr-x 6 sourcegr sourcegr 4.0K Apr 17 12:41 . drwxr-xr-x 1 root root 4.0K Apr 13 11:36 .. drwxr-x--x 2 sourcegr sourcegr 4.0K Apr 17 12:41 create-bucket-test drwxrws--- 2 sourcegr sourcegr 16.0K Dec 21 21:38 lost+found drwxrws--x 2 sourcegr sourcegr 4.0K Apr 14 19:36 lsif-uploads
Confirm that the frontend can reach blobstore:
kubectl exec -it $frontend_pod sh curl -i http://blobstore:9000/
You should see a 200 response like:
HTTP/1.1 200 OK Date: Mon, 17 Apr 2023 12:44:36 GMT x-amz-request-id: 4442587FB7D0A2F9 Content-Type: application/xml;charset=utf-8 Transfer-Encoding: chunked Server: Jetty(11.0.11)
If this works as above, then the customer needs to change the permissions in blobstore to give it w+x rights using either 775/777.