Troubleshooting blobstore blocking bucket creation in precise-code-intel-worker


Blobstore replaced minio in v4.2.1. For customers running Kustomize patches for minio, chances are, that they can and do forget to apply the correct folder permissions for blobstore. What this causes then is either the precise-code-intel-worker pod or container constantly restarting or falling into a CrashLoop state.



You'll see log entries in precise-code-intel worker that look like:

      Message:     {"SeverityText":"FATAL","Timestamp":1681494676729380830,"InstrumentationScope":"codeintel-worker","Caller":"shared/shared.go:108","Function":"","Body":"Failed to initialize upload store","Resource":{"":"precise-code-intel-worker","service.version":"4.3.1","":"precise-code-intel-worker-7697f89fb8-wb7cg"},"Attributes":{"error":"failed to create bucket: operation error S3: CreateBucket, https response error StatusCode: 403, RequestID: 4442587FB7D0A2F9, HostID: , api error AccessDenied: Could not create bucket"}}


First, you need to confirm if it's indeed a permissions issue by performing the following:


Shell into the blobstore pod:
kubectl exec -it blobstore sh
Test if you can create a bucket:
/opt/s3proxy $ curl -i -X PUT
Output should look like:
HTTP/1.1 200 OK
Date: Mon, 17 Apr 2023 12:41:57 GMT
x-amz-request-id: 4442587FB7D0A2F9
Location: /create-bucket-test
Content-Length: 0
Server: Jetty(11.0.11)
Confirm that file permissions are correct (should show sourcegraph for the /data directory and all directories below:
/opt/s3proxy $ ls -lah /data
total 36K
drwxrwsr-x    6 sourcegr sourcegr    4.0K Apr 17 12:41 .
drwxr-xr-x    1 root     root        4.0K Apr 13 11:36 ..
drwxr-x--x    2 sourcegr sourcegr    4.0K Apr 17 12:41 create-bucket-test
drwxrws---    2 sourcegr sourcegr   16.0K Dec 21 21:38 lost+found
drwxrws--x    2 sourcegr sourcegr    4.0K Apr 14 19:36 lsif-uploads
Confirm that the frontend can reach blobstore:
kubectl exec -it $frontend_pod sh
curl -i http://blobstore:9000/
You should see a 200 response like:
HTTP/1.1 200 OK
Date: Mon, 17 Apr 2023 12:44:36 GMT
x-amz-request-id: 4442587FB7D0A2F9
Content-Type: application/xml;charset=utf-8
Transfer-Encoding: chunked
Server: Jetty(11.0.11)

If this works as above, then the customer needs to change the permissions in blobstore to give it w+x rights using either 775/777.

Was this article helpful?
0 out of 0 found this helpful

Articles in this section